diff --git a/.agents/skills/rar-format/SKILL.md b/.agents/skills/rar-format/SKILL.md new file mode 100644 index 00000000..b5b91d58 --- /dev/null +++ b/.agents/skills/rar-format/SKILL.md @@ -0,0 +1,22 @@ +--- +name: rar-format +description: Reference the RAR/RAR5 archive container format and UnRAR source behavior. Use when an AI agent needs to answer questions or make code changes involving RAR signatures, RAR5 vint fields, block headers, file/service headers, extra records, solid archives, multivolume parts, encryption headers, recovery records, quick-open data, or SharpCompress Rar parsing and extraction behavior. +--- + +# Rar Format + +Use this skill for RAR container-format work. It provides a local, SharpCompress-oriented reference for RAR 5.0 archive blocks, older RAR header compatibility, UnRAR source behavior, and current SharpCompress Rar implementation boundaries. + +## Reference + +- Read [references/rar-format.md](references/rar-format.md) when the task depends on RAR binary layout, RAR4 vs RAR5 signatures, RAR5 vint encoding, block/header flags, file and service header fields, extra records, encrypted headers, solid archives, split volumes, redirection records, or current SharpCompress Rar support boundaries. +- Treat the reference as an implementation guide, not a standards replacement. It summarizes `reference/RAR 5.0 archive format.htm` and selected files under `reference/unrar/`, especially `headers5.hpp`, `headers.hpp`, `arcread.cpp`, `rawread.*`, `crypt*.cpp`, `qopen.*`, `volume.*`, `recvol*.cpp`, `unpack*.cpp`, and `blake2sp.*`. +- Prefer the SharpCompress support matrix in the reference over generic RAR assumptions when changing code. RAR is a read-only format in SharpCompress, and metadata/service-header support is intentionally partial. + +## Workflow + +1. Identify which layer is involved: signature detection, RAR4 headers, RAR5 headers, vint parsing, block flags, extra records, file/service metadata, encrypted headers, multivolume handling, decompression, reader/archive API behavior, or tests. +2. Open the relevant section in `references/rar-format.md` and use the source-file pointers before changing code. +3. For header parsing changes, cross-check sync and async implementations: `MarkHeader.cs`, `RarHeader.cs`, `RarHeaderFactory.cs`, `FileHeader.cs`, `Flags.cs`, and their async counterparts. +4. For extraction changes, verify split and solid archive behavior with `RarArchive`, `RarReader`, `RarStream`, and the unpacker files under `src/SharpCompress/Compressors/Rar/`. +5. For support claims, keep unsupported or partial features explicit: RAR writing, pre-RAR4 archives, complete service-header semantics, quick-open use, recovery reconstruction, and encryption/version constraints. diff --git a/.agents/skills/rar-format/references/rar-format.md b/.agents/skills/rar-format/references/rar-format.md new file mode 100644 index 00000000..47fd01fa --- /dev/null +++ b/.agents/skills/rar-format/references/rar-format.md @@ -0,0 +1,459 @@ +# RAR Format Reference + +This reference summarizes the RAR archive container format for SharpCompress work. It is locally authored from the RAR 5.0 format document, UnRAR source code, and the current SharpCompress implementation. + +Primary local references: + +- `reference/RAR 5.0 archive format.htm` +- `reference/unrar/headers5.hpp` +- `reference/unrar/headers.hpp` +- `reference/unrar/arcread.cpp` +- `reference/unrar/rawread.cpp` +- `reference/unrar/rawread.hpp` +- `reference/unrar/crypt.cpp` +- `reference/unrar/crypt5.cpp` +- `reference/unrar/qopen.cpp` +- `reference/unrar/qopen.hpp` +- `reference/unrar/volume.cpp` +- `reference/unrar/volume.hpp` +- `reference/unrar/recvol.cpp` +- `reference/unrar/recvol5.cpp` +- `reference/unrar/unpack.cpp` +- `reference/unrar/unpack15.cpp` +- `reference/unrar/unpack20.cpp` +- `reference/unrar/unpack30.cpp` +- `reference/unrar/unpack50.cpp` +- `reference/unrar/blake2sp.cpp` +- `reference/unrar/blake2sp.hpp` + +Primary SharpCompress references: + +- `docs/FORMATS.md` +- `src/SharpCompress/Factories/RarFactory.cs` +- `src/SharpCompress/Archives/Rar/RarArchive.cs` +- `src/SharpCompress/Archives/Rar/RarArchiveEntry.cs` +- `src/SharpCompress/Archives/Rar/RarArchiveVolumeFactory.cs` +- `src/SharpCompress/Readers/Rar/RarReader.cs` +- `src/SharpCompress/Readers/Rar/MultiVolumeRarReader.cs` +- `src/SharpCompress/Common/Rar/RarEntry.cs` +- `src/SharpCompress/Common/Rar/RarFilePart.cs` +- `src/SharpCompress/Common/Rar/Rar5CryptoInfo.cs` +- `src/SharpCompress/Common/Rar/RarCryptoBinaryReader.cs` +- `src/SharpCompress/Common/Rar/RarCryptoWrapper.cs` +- `src/SharpCompress/Common/Rar/Headers/MarkHeader.cs` +- `src/SharpCompress/Common/Rar/Headers/RarHeader.cs` +- `src/SharpCompress/Common/Rar/Headers/RarHeaderFactory.cs` +- `src/SharpCompress/Common/Rar/Headers/FileHeader.cs` +- `src/SharpCompress/Common/Rar/Headers/Flags.cs` +- `src/SharpCompress/Compressors/Rar/` +- `tests/SharpCompress.Test/Rar/` + +## Contents + +- [Format Overview](#format-overview) +- [Signatures And SFX](#signatures-and-sfx) +- [RAR5 Vint Encoding](#rar5-vint-encoding) +- [RAR5 General Block Format](#rar5-general-block-format) +- [RAR5 Header Types](#rar5-header-types) +- [RAR5 Common Header Flags](#rar5-common-header-flags) +- [RAR5 Main Archive Header](#rar5-main-archive-header) +- [RAR5 File And Service Headers](#rar5-file-and-service-headers) +- [RAR5 Extra Area Records](#rar5-extra-area-records) +- [RAR5 Compression Info](#rar5-compression-info) +- [Encryption](#encryption) +- [Checksums And Hashes](#checksums-and-hashes) +- [Service Headers](#service-headers) +- [RAR4 Compatibility](#rar4-compatibility) +- [Multivolume And Solid Archives](#multivolume-and-solid-archives) +- [SharpCompress Support Matrix](#sharpcompress-support-matrix) +- [SharpCompress Read Behavior](#sharpcompress-read-behavior) +- [Known Limitations](#known-limitations) +- [Test Fixtures](#test-fixtures) + +## Format Overview + +RAR is a block-oriented archive format. SharpCompress supports RAR as a read-only archive/reader format and delegates decompression to its RAR unpacker implementation under `src/SharpCompress/Compressors/Rar/`. + +RAR 5.0 general layout: + +```text +self-extracting module (optional) +RAR 5.0 signature +archive encryption header (optional) +main archive header +archive comment service header (optional) +file header 1 +service headers for preceding file (optional) +... +file header N +service headers for preceding file (optional) +recovery record (optional) +end of archive header +``` + +RAR has no ZIP-style central directory. Readers scan blocks in sequence. Random access is limited by solid compression, split volumes, and the need to process preceding compressed data for solid archives. + +## Signatures And SFX + +RAR signatures: + +| Format | Bytes | SharpCompress behavior | +| --- | --- | --- | +| Old pre-RAR4 marker | `52 45 7e 5e` | Explicitly unsupported | +| RAR4 marker | `52 61 72 21 1a 07 00` | Supported read path | +| RAR5 marker | `52 61 72 21 1a 07 01 00` | Supported read path | + +RAR archives can be preceded by a self-extracting module. `MarkHeader.Read` scans for the marker when `ReaderOptions.LookForHeader` is enabled. SharpCompress uses a maximum SFX scan size from the UnRAR implementation notes. + +## RAR5 Vint Encoding + +RAR5 uses variable-length integers, called `vint` in the format document. + +Rules: + +- Each byte contributes 7 data bits. +- The high bit is the continuation flag. +- If the high bit is `0`, the byte is the last byte in the sequence. +- The first byte contains the least significant 7 bits. +- RAR currently uses vint for up to 64-bit integers, so values can occupy up to 10 bytes. +- Writers can preallocate more bytes than needed by using leading `0x80` bytes, which encode zero with continuation set. + +SharpCompress reads these through `ReadRarVInt*` helpers on marking readers. For RAR5 header size, SharpCompress limits the size field to 3 vint bytes to match the current format implementation limit of a 2 MB maximum header size. + +## RAR5 General Block Format + +RAR5 blocks share a common header shape: + +| Field | Size | Notes | +| --- | --- | --- | +| Header CRC32 | `uint32` | CRC32 of header data starting at header size through optional extra area | +| Header size | `vint` | Size from header type through optional extra area; current max 3 bytes for 2 MB headers | +| Header type | `vint` | See [RAR5 Header Types](#rar5-header-types) | +| Header flags | `vint` | Common flags for all headers | +| Extra area size | `vint` | Present only when common flag `0x0001` is set | +| Data size | `vint` | Present only when common flag `0x0002` is set | +| Type-specific fields | variable | Depends on header type | +| Extra area | variable | Present only when common flag `0x0001` is set | +| Data area | variable | Present only when common flag `0x0002` is set; not included in header CRC/size | + +SharpCompress reads the common fields in `RarHeader.Initialize`, then creates typed headers in `RarHeaderFactory`. + +## RAR5 Header Types + +RAR5 header type values: + +| Type | Meaning | SharpCompress code path | +| --- | --- | --- | +| `1` | Main archive header | `ArchiveHeader` | +| `2` | File header | `FileHeader` with `HeaderType.File` | +| `3` | Service header | `FileHeader` with `HeaderType.Service` | +| `4` | Archive encryption header | `ArchiveCryptHeader` | +| `5` | End of archive header | `EndArchiveHeader` | + +SharpCompress constants live in `HeaderCodeV` in `Flags.cs`. + +## RAR5 Common Header Flags + +Common RAR5 header flags from `headers5.hpp` and `Flags.cs`: + +| Flag | Meaning | +| --- | --- | +| `0x0001` | Extra area is present | +| `0x0002` | Data area is present | +| `0x0004` | Unknown blocks with this flag must be skipped when updating | +| `0x0008` | Data area continues from previous volume | +| `0x0010` | Data area continues in next volume | +| `0x0020` | Block depends on preceding file block | +| `0x0040` | Preserve child block if host block is modified | + +SharpCompress exposes split status through `FileHeader.IsSplitBefore`, `FileHeader.IsSplitAfter`, and `RarEntry.IsSplitAfter`. + +## RAR5 Main Archive Header + +Main archive header fields after the common block fields: + +| Field | Size | Notes | +| --- | --- | --- | +| Archive flags | `vint` | Volume, solid, recovery, locked flags | +| Volume number | `vint` | Present only when archive flag `0x0002` is set | +| Extra area | variable | Optional records, currently locator is defined | + +Main archive flags: + +| Flag | Meaning | +| --- | --- | +| `0x0001` | Volume, archive is part of a multivolume set | +| `0x0002` | Volume number field is present | +| `0x0004` | Solid archive | +| `0x0008` | Recovery record is present | +| `0x0010` | Locked archive | + +Main header extra record types: + +| Type | Name | Meaning | +| --- | --- | --- | +| `0x01` | Locator | Optional offsets to quick-open and recovery-record blocks | + +SharpCompress parses archive flags in `ArchiveHeader` and uses volume/solid information in archive and reader flows. + +## RAR5 File And Service Headers + +File and service headers share the same base layout. Header type `2` is a file header and header type `3` is a service header. + +Fields after common block fields: + +| Field | Size | Notes | +| --- | --- | --- | +| File flags | `vint` | Directory, time, CRC, unknown unpacked size | +| Unpacked size | `vint` | Present even when unknown-size flag is set, but ignored then | +| Attributes | `vint` | OS-specific file attributes | +| mtime | `uint32` | Unix time, present only when file flag `0x0002` is set | +| Data CRC32 | `uint32` | Present only when file flag `0x0004` is set | +| Compression information | `vint` | Algorithm version, solid flag, method, dictionary size | +| Host OS | `vint` | `0` Windows, `1` Unix | +| Name length | `vint` | Byte count | +| Name | variable | UTF-8, no trailing zero | +| Extra area | variable | Optional file/service extra records | +| Data area | variable | File data or service data | + +RAR5 file flags: + +| Flag | Meaning | +| --- | --- | +| `0x0001` | Directory filesystem object | +| `0x0002` | Unix mtime field is present | +| `0x0004` | CRC32 field is present | +| `0x0008` | Unpacked size is unknown; extract until compression stream ends | + +SharpCompress parses these in `FileHeader.ReadFromReaderV5`. + +## RAR5 Extra Area Records + +Each extra record has this shape: + +```text +record size vint size from type through record data +record type vint +record data variable +``` + +File and service header extra record types: + +| Type | Name | SharpCompress behavior | +| --- | --- | --- | +| `0x01` | File encryption | Parses `Rar5CryptoInfo` | +| `0x02` | File hash | Reads BLAKE2sp digest when present | +| `0x03` | File time | Reads high precision mtime/ctime/atime | +| `0x04` | File version | Currently skipped/drained | +| `0x05` | Redirection | Parses symlink/junction/hard link/file copy metadata | +| `0x06` | Unix owner | Currently skipped/drained | +| `0x07` | Service data | Currently skipped/drained except service header data handling | + +Unknown records must be skipped without interrupting normal operation. SharpCompress drains unhandled extra record bytes after each record. + +Redirection types: + +| Value | Meaning | +| --- | --- | +| `0x0001` | Unix symlink | +| `0x0002` | Windows symlink | +| `0x0003` | Windows junction | +| `0x0004` | Hard link | +| `0x0005` | File copy | + +`RarEntry.IsRedir` and `RarEntry.RedirTargetName` expose part of this metadata. + +## RAR5 Compression Info + +RAR5 file compression information is a packed vint bit field: + +| Bits/mask | Meaning | +| --- | --- | +| `0x003f` | Compression algorithm version, currently 0 in RAR5; SharpCompress stores it as value + 50 | +| `0x0040` | Solid flag; dictionary continues from preceding files | +| `0x0380` | Compression method, values 0-5 are used; 0 is store/no compression | +| `0x3c00` | Minimum dictionary size: 0 means 128 KB, 1 means 256 KB, through 15 meaning 4096 MB | + +SharpCompress maps these to `FileHeader.CompressionAlgorithm`, `FileHeader.IsSolid`, `FileHeader.CompressionMethod`, and `FileHeader.WindowSize`. + +RAR4 and older algorithm values are distinct. `RarEntry.IsRarV3` currently treats algorithm values `15`, `20`, `26`, `29`, and `36` as legacy RAR code paths. + +## Encryption + +RAR5 archive encryption header fields: + +| Field | Size | Notes | +| --- | --- | --- | +| Encryption version | `vint` | Current supported version is 0, AES-256 | +| Encryption flags | `vint` | `0x0001` means password check data is present | +| KDF count | 1 byte | Binary logarithm of PBKDF2 iteration count | +| Salt | 16 bytes | Header encryption salt | +| Check value | 12 bytes | Optional password check plus checksum | + +When archive headers are encrypted, each following header starts with a 16-byte AES-256 initialization vector followed by encrypted header data aligned to 16 bytes. + +RAR5 file encryption extra record fields include encryption version, flags, KDF count, 16-byte salt, 16-byte IV, and optional 12-byte password check data. + +SharpCompress behavior: + +- Archive encryption header is parsed by `ArchiveCryptHeader`. +- File encryption extra records are parsed into `Rar5CryptoInfo`. +- Header decryption uses `RarCryptoBinaryReader` with `CryptKey5`. +- File data decryption uses `RarCryptoWrapper` around the packed stream. +- Password is required through `ReaderOptions.Password`; missing password throws `CryptographicException`. +- RAR4 encrypted file data uses RAR3 crypto classes and salt handling. + +## Checksums And Hashes + +RAR5 checksum/hash locations: + +- Header CRC32 covers header data starting at the header size field through optional extra area. It does not include the data area. +- File header can contain CRC32 of unpacked data when file flag `0x0004` is set. +- File hash extra record type `0x02` can contain BLAKE2sp hash data. The defined RAR5 hash type value is `0x00` for BLAKE2sp. +- For split files, hashes can apply to packed data for non-final volume parts. + +SharpCompress verifies header CRC in `RarHeader.VerifyHeaderCrc`. RAR CRC logic is in `RarCrcBinaryReader`, `AsyncRarCrcBinaryReader`, and `RarCRC`. + +## Service Headers + +RAR5 service headers use the file-header structure with header type `3`. Known service names from the RAR5 document and UnRAR source include: + +| Name | Meaning | +| --- | --- | +| `CMT` | Archive comment | +| `QO` | Quick-open data | +| `ACL` | NTFS permissions | +| `STM` | NTFS alternate data stream | +| `RR` | Recovery record | + +SharpCompress handles `CMT` specially by exposing its packed stream in `RarHeaderFactory`; most other service data is skipped or only partially modeled. + +Quick-open caution from the RAR5 format document: + +- Quick-open data can store copies of headers for faster listing. +- If quick-open data is used to display names, extraction must use the same source. Otherwise malicious archives could show one name and extract another. +- SharpCompress should not use quick-open data for one path and ordinary headers for another without carefully preserving this invariant. + +## RAR4 Compatibility + +SharpCompress supports RAR4-style headers as well as RAR5. + +RAR4 header codes from UnRAR and `Flags.cs`: + +| Code | Meaning | +| --- | --- | +| `0x72` | Mark header | +| `0x73` | Archive/main header | +| `0x74` | File header | +| `0x75` | Comment header | +| `0x76` | AV header | +| `0x77` | Old subheader | +| `0x78` | Protect/recovery header | +| `0x79` | Sign header | +| `0x7a` | New subheader/service header | +| `0x7b` | End archive header | + +RAR4 file-header behavior differs from RAR5: + +- Fixed-size base fields rather than RAR5 vint-heavy layout. +- Optional large-file high-size fields when `LARGE` flag is set. +- File names can use older RAR Unicode name encoding. +- DOS timestamps and RAR4 extended time flags are used. +- Directory status is encoded through the window mask. +- Path separators differ: RAR4 can use backslashes as separators, while RAR5 uses `/` as the universal separator. + +SharpCompress parses this in `FileHeader.ReadFromReaderV4` and maps paths through `ConvertPathV4`. + +## Multivolume And Solid Archives + +RAR supports multivolume archives and solid compression. + +Relevant flags: + +- Main archive `0x0001`: archive is part of a volume set. +- Main archive `0x0002`: RAR5 volume number field is present. +- Common block `0x0008`: data continues from previous volume. +- Common block `0x0010`: data continues in next volume. +- RAR4 file flags `SPLIT_BEFORE` and `SPLIT_AFTER` indicate split file data. +- RAR5 compression info `0x0040` indicates solid compression. + +SharpCompress behavior: + +- `RarFactory` implements `IMultiArchiveFactory`. +- `RarArchiveVolumeFactory` resolves volume file parts. +- `RarEntry.IsSplitAfter` exposes split status. +- Solid archives should be extracted sequentially. Prefer `ExtractAllEntries()` for solid RAR archives rather than extracting arbitrary entries independently. + +## SharpCompress Support Matrix + +| Feature | Support | Notes | +| --- | --- | --- | +| RAR read | Yes | Archive and Reader APIs | +| RAR write | No | RAR is read-only | +| RAR4 headers | Yes | Supported parsing path | +| RAR5 headers | Yes | Supported parsing path | +| Pre-RAR4 marker | No | `MarkHeader` throws unsupported format | +| RAR5 vint fields | Yes | Reader helpers parse vint values | +| File/service extra records | Partial | Encryption/hash/time/redirection modeled; others mostly skipped | +| Solid archives | Yes, with sequencing constraints | Use sequential extraction | +| Multivolume archives | Yes | Archive/multi-volume factory paths | +| Encrypted archives | Partial | Password required; format/version constraints apply | +| Archive comments | Partial | `CMT` service header handled specially | +| Quick-open records | Not a public feature | Avoid inconsistent listing/extraction behavior | +| Recovery records | Skipped/partial | Not recovery reconstruction API | + +## SharpCompress Read Behavior + +Header flow: + +1. `MarkHeader.Read` scans for RAR4 or RAR5 signature and rejects old pre-RAR4 signatures. +2. `RarHeaderFactory.ReadHeaders` records whether the archive is RAR5. +3. `RarHeader.TryReadBase` reads common header fields and CRC state. +4. `RarHeaderFactory` creates typed headers based on header code. +5. File data is either skipped, wrapped in `ReadOnlySubStream`, or wrapped in decryption stream depending on mode and encryption metadata. + +Seekable mode: + +- File data positions are recorded in `FileHeader.DataStartPosition`. +- Streams can seek over packed data while collecting headers. + +Streaming mode: + +- File data is exposed as a substream for file headers. +- Non-file service data is skipped unless specially handled. +- Consumers must process entries in archive order. + +Decompression: + +- `RarStream` wraps the packed stream and an `IRarUnpack` implementation. +- `RarStream` is non-seekable. +- `RarStream.Initialize` starts unpacking based on the parsed `FileHeader`. + +## Known Limitations + +Keep these limitations explicit in code comments, docs, and tests: + +- No RAR writing support. +- No support for pre-RAR4 archives. +- Service headers are not fully modeled by public APIs. +- Quick-open records are not a general public feature and have security-sensitive listing/extraction consistency requirements. +- Recovery record reconstruction is not exposed as a full repair feature. +- RAR5 Unix owner records and service data records are mostly skipped unless a specific behavior is implemented. +- Redirection metadata is surfaced, but extraction semantics for all link types should be treated carefully and tested for security. +- Encrypted archive support depends on password, encryption version, and KDF limits. +- Solid and multivolume archives must be tested with sequential extraction scenarios. + +## Test Fixtures + +Representative RAR test files live under `tests/TestArchives/Archives/` and related test archive folders. Use existing fixtures when possible instead of adding new binary archives. + +Representative test files: + +- `tests/SharpCompress.Test/Rar/RarArchiveTests.cs` +- `tests/SharpCompress.Test/Rar/RarArchiveAsyncTests.cs` +- `tests/SharpCompress.Test/Rar/RarReaderTests.cs` +- `tests/SharpCompress.Test/Rar/RarReaderAsyncTests.cs` +- `tests/SharpCompress.Test/Rar/RarHeaderFactoryTest.cs` +- `tests/SharpCompress.Test/Rar/RarCRCTest.cs` + +When changing RAR behavior, include both Archive and Reader API coverage where applicable. For solid or multivolume behavior, prefer tests that extract entries sequentially and verify stream ownership and volume transitions. diff --git a/.agents/skills/sevenzip-format/SKILL.md b/.agents/skills/sevenzip-format/SKILL.md new file mode 100644 index 00000000..20635b38 --- /dev/null +++ b/.agents/skills/sevenzip-format/SKILL.md @@ -0,0 +1,21 @@ +--- +name: sevenzip-format +description: Reference the 7z/7zip archive container format. Use when an AI agent needs to answer questions or make code changes involving 7z signatures, headers, encoded headers, NID/property IDs, packed streams, folders/coders, bind pairs, substreams, file metadata properties, or SharpCompress 7Zip parsing behavior. +--- + +# Sevenzip Format + +Use this skill for 7z container-format work. It provides a local Markdown conversion of the LZMA SDK `7zFormat.txt` reference. + +## Reference + +- Read [references/7z-format.md](references/7z-format.md) when the task depends on 7z binary layout, header property IDs, stream/folder relationships, metadata fields, or encoded headers. +- Treat the reference as the LZMA SDK 7z format description version 4.59. It describes the container grammar, not compression method internals; method-specific codec details are outside this skill. +- Preserve source field names and numeric IDs when mapping the spec to code. The converted reference keeps source spelling inside syntax blocks where exact matching may matter. + +## Workflow + +1. Identify which part of the 7z container is involved: signature/start header, packed streams, coders/folders, substreams, files info, or encoded headers. +2. Open the relevant section in `references/7z-format.md` and use the table of contents to avoid loading unrelated details. +3. When implementing or reviewing parsing logic, pay special attention to optional blocks marked with `[]`, 7z's variable-length `UINT64` encoding, and little-endian `REAL_UINT64` fields. +4. Cross-check behavior against SharpCompress tests and existing parser conventions before changing public API or stream behavior. diff --git a/.agents/skills/sevenzip-format/references/7z-format.md b/.agents/skills/sevenzip-format/references/7z-format.md new file mode 100644 index 00000000..0cd056a3 --- /dev/null +++ b/.agents/skills/sevenzip-format/references/7z-format.md @@ -0,0 +1,478 @@ +# 7z Format Description (4.59) + +Source: https://github.com/jljusten/LZMA-SDK/blob/master/DOC/7zFormat.txt + +Raw download used for this conversion: https://raw.githubusercontent.com/jljusten/LZMA-SDK/master/DOC/7zFormat.txt + +Downloaded and converted on 2026-05-23. + +This is a Markdown conversion of the LZMA SDK plaintext 7z archive format description. Pseudo-grammar blocks preserve the source field names and spelling. + +## Contents + +- [Overview](#overview) +- [Format Structure Overview](#format-structure-overview) +- [Notes About Notation and Encoding](#notes-about-notation-and-encoding) +- [Property IDs](#property-ids) +- [7z Format Headers](#7z-format-headers) + - [SignatureHeader](#signatureheader) + - [ArchiveProperties](#archiveproperties) + - [Digests](#digests-numstreams) + - [PackInfo](#packinfo) + - [Folder](#folder) + - [Coders Info](#coders-info) + - [SubStreams Info](#substreams-info) + - [Streams Info](#streams-info) + - [FilesInfo](#filesinfo) + - [Header](#header) + - [HeaderInfo](#headerinfo) + +## Overview + +This file contains a description of the 7z archive format. + +A 7z archive can contain files compressed with any method. See `Methods.txt` in the LZMA SDK for descriptions of defined compression methods. + +## Format Structure Overview + +Some fields can be optional. + +### Archive Structure + +```text +SignatureHeader +[PackedStreams] +[PackedStreamsForHeaders] +[ + Header + or + { + Packed Header + HeaderInfo + } +] +``` + +### Header Structure + +```text +{ + ArchiveProperties + AdditionalStreams + { + PackInfo + { + PackPos + NumPackStreams + Sizes[NumPackStreams] + CRCs[NumPackStreams] + } + CodersInfo + { + NumFolders + Folders[NumFolders] + { + NumCoders + CodersInfo[NumCoders] + { + ID + NumInStreams; + NumOutStreams; + PropertiesSize + Properties[PropertiesSize] + } + NumBindPairs + BindPairsInfo[NumBindPairs] + { + InIndex; + OutIndex; + } + PackedIndices + } + UnPackSize[Folders][Folders.NumOutstreams] + CRCs[NumFolders] + } + SubStreamsInfo + { + NumUnPackStreamsInFolders[NumFolders]; + UnPackSizes[] + CRCs[] + } + } + MainStreamsInfo + { + (Same as in AdditionalStreams) + } + FilesInfo + { + NumFiles + Properties[] + { + ID + Size + Data + } + } +} +``` + +### HeaderInfo Structure + +```text +{ + (Same as in AdditionalStreams) +} +``` + +## Notes About Notation and Encoding + +7z uses little-endian encoding. + +Optional headers are marked as: + +```text +[] +Header +[] +``` + +`REAL_UINT64` means a real `UINT64`. + +`UINT64` means a real `UINT64` encoded with the following scheme. The size of the encoding sequence depends on the first byte: + +| First byte (binary) | Extra bytes | Value | +| --- | --- | --- | +| `0xxxxxxx` | none | `( xxxxxxx )` | +| `10xxxxxx` | `BYTE y[1]` | `( xxxxxx << (8 * 1)) + y` | +| `110xxxxx` | `BYTE y[2]` | `( xxxxx << (8 * 2)) + y` | +| `...` | `...` | `...` | +| `1111110x` | `BYTE y[6]` | `( x << (8 * 6)) + y` | +| `11111110` | `BYTE y[7]` | `y` | +| `11111111` | `BYTE y[8]` | `y` | + +## Property IDs + +| ID | Name | +| --- | --- | +| `0x00` | `kEnd` | +| `0x01` | `kHeader` | +| `0x02` | `kArchiveProperties` | +| `0x03` | `kAdditionalStreamsInfo` | +| `0x04` | `kMainStreamsInfo` | +| `0x05` | `kFilesInfo` | +| `0x06` | `kPackInfo` | +| `0x07` | `kUnPackInfo` | +| `0x08` | `kSubStreamsInfo` | +| `0x09` | `kSize` | +| `0x0A` | `kCRC` | +| `0x0B` | `kFolder` | +| `0x0C` | `kCodersUnPackSize` | +| `0x0D` | `kNumUnPackStream` | +| `0x0E` | `kEmptyStream` | +| `0x0F` | `kEmptyFile` | +| `0x10` | `kAnti` | +| `0x11` | `kName` | +| `0x12` | `kCTime` | +| `0x13` | `kATime` | +| `0x14` | `kMTime` | +| `0x15` | `kWinAttributes` | +| `0x16` | `kComment` | +| `0x17` | `kEncodedHeader` | +| `0x18` | `kStartPos` | +| `0x19` | `kDummy` | + +## 7z Format Headers + +### SignatureHeader + +```text +BYTE kSignature[6] = {'7', 'z', 0xBC, 0xAF, 0x27, 0x1C}; + +ArchiveVersion +{ + BYTE Major; // now = 0 + BYTE Minor; // now = 2 +}; + +UINT32 StartHeaderCRC; + +StartHeader +{ + REAL_UINT64 NextHeaderOffset + REAL_UINT64 NextHeaderSize + UINT32 NextHeaderCRC +} +``` + +### ArchiveProperties + +```text +BYTE NID::kArchiveProperties (0x02) +for (;;) +{ + BYTE PropertyType; + if (aType == 0) + break; + UINT64 PropertySize; + BYTE PropertyData[PropertySize]; +} +``` + +### Digests (NumStreams) + +```text +BYTE AllAreDefined +if (AllAreDefined == 0) +{ + for(NumStreams) + BIT Defined +} +UINT32 CRCs[NumDefined] +``` + +### PackInfo + +```text +BYTE NID::kPackInfo (0x06) +UINT64 PackPos +UINT64 NumPackStreams + +[] +BYTE NID::kSize (0x09) +UINT64 PackSizes[NumPackStreams] +[] + +[] +BYTE NID::kCRC (0x0A) +PackStreamDigests[NumPackStreams] +[] + +BYTE NID::kEnd +``` + +### Folder + +```text +UINT64 NumCoders; +for (NumCoders) +{ + BYTE + { + 0:3 CodecIdSize + 4: Is Complex Coder + 5: There Are Attributes + 6: Reserved + 7: There are more alternative methods. (Not used anymore, must be 0). + } + BYTE CodecId[CodecIdSize] + if (Is Complex Coder) + { + UINT64 NumInStreams; + UINT64 NumOutStreams; + } + if (There Are Attributes) + { + UINT64 PropertiesSize + BYTE Properties[PropertiesSize] + } +} + +NumBindPairs = NumOutStreamsTotal - 1; + +for (NumBindPairs) +{ + UINT64 InIndex; + UINT64 OutIndex; +} + +NumPackedStreams = NumInStreamsTotal - NumBindPairs; +if (NumPackedStreams > 1) + for(NumPackedStreams) + { + UINT64 Index; + }; +``` + +### Coders Info + +```text +BYTE NID::kUnPackInfo (0x07) + +BYTE NID::kFolder (0x0B) +UINT64 NumFolders +BYTE External +switch(External) +{ + case 0: + Folders[NumFolders] + case 1: + UINT64 DataStreamIndex +} + +BYTE ID::kCodersUnPackSize (0x0C) +for(Folders) + for(Folder.NumOutStreams) + UINT64 UnPackSize; + +[] +BYTE NID::kCRC (0x0A) +UnPackDigests[NumFolders] +[] + +BYTE NID::kEnd +``` + +### SubStreams Info + +```text +BYTE NID::kSubStreamsInfo; (0x08) + +[] +BYTE NID::kNumUnPackStream; (0x0D) +UINT64 NumUnPackStreamsInFolders[NumFolders]; +[] + +[] +BYTE NID::kSize (0x09) +UINT64 UnPackSizes[] +[] + +[] +BYTE NID::kCRC (0x0A) +Digests[Number of streams with unknown CRC] +[] + +BYTE NID::kEnd +``` + +### Streams Info + +```text +[] +PackInfo +[] + +[] +CodersInfo +[] + +[] +SubStreamsInfo +[] + +BYTE NID::kEnd +``` + +### FilesInfo + +```text +BYTE NID::kFilesInfo; (0x05) +UINT64 NumFiles + +for (;;) +{ + BYTE PropertyType; + if (aType == 0) + break; + + UINT64 Size; + + switch(PropertyType) + { + kEmptyStream: (0x0E) + for(NumFiles) + BIT IsEmptyStream + + kEmptyFile: (0x0F) + for(EmptyStreams) + BIT IsEmptyFile + + kAnti: (0x10) + for(EmptyStreams) + BIT IsAntiFile + + case kCTime: (0x12) + case kATime: (0x13) + case kMTime: (0x14) + BYTE AllAreDefined + if (AllAreDefined == 0) + { + for(NumFiles) + BIT TimeDefined + } + BYTE External; + if(External != 0) + UINT64 DataIndex + [] + for(Definded Items) + UINT64 Time + [] + + kNames: (0x11) + BYTE External; + if(External != 0) + UINT64 DataIndex + [] + for(Files) + { + wchar_t Names[NameSize]; + wchar_t 0; + } + [] + + kAttributes: (0x15) + BYTE AllAreDefined + if (AllAreDefined == 0) + { + for(NumFiles) + BIT AttributesAreDefined + } + BYTE External; + if(External != 0) + UINT64 DataIndex + [] + for(Definded Attributes) + UINT32 Attributes + [] + } +} +``` + +### Header + +```text +BYTE NID::kHeader (0x01) + +[] +ArchiveProperties +[] + +[] +BYTE NID::kAdditionalStreamsInfo; (0x03) +StreamsInfo +[] + +[] +BYTE NID::kMainStreamsInfo; (0x04) +StreamsInfo +[] + +[] +FilesInfo +[] + +BYTE NID::kEnd +``` + +### HeaderInfo + +```text +[] +BYTE NID::kEncodedHeader; (0x17) +StreamsInfo for Encoded Header +[] +``` + +--- + +End of document. diff --git a/.agents/skills/tar-format/SKILL.md b/.agents/skills/tar-format/SKILL.md new file mode 100644 index 00000000..eb622faa --- /dev/null +++ b/.agents/skills/tar-format/SKILL.md @@ -0,0 +1,22 @@ +--- +name: tar-format +description: Reference the Tar/USTAR/PAX/GNU tar archive container format. Use when an AI agent needs to answer questions or make code changes involving tar headers, 512-byte blocks, checksums, typeflags, USTAR prefixes, PAX local/global extended headers, GNU long names/links, sparse entries, wrapper compression, or SharpCompress Tar parsing and writing behavior. +--- + +# Tar Format + +Use this skill for tar container-format work. It provides a local, SharpCompress-oriented reference for POSIX USTAR, POSIX PAX, GNU tar extensions, and the current SharpCompress Tar implementation. + +## Reference + +- Read [references/tar-format.md](references/tar-format.md) when the task depends on tar binary layout, header field offsets, typeflag behavior, checksum rules, PAX records, GNU long-name/link records, wrapper compression support, or current SharpCompress Tar support boundaries. +- Treat the reference as an implementation guide, not a standards replacement. It cites POSIX and GNU tar sources, but it also documents SharpCompress-specific behavior and limitations. +- Prefer the SharpCompress support matrix in the reference over generic tar assumptions when changing code. Tar dialect support is intentionally partial in some areas. + +## Workflow + +1. Identify which layer is involved: raw tar block/header parsing, POSIX USTAR fields, POSIX PAX metadata, GNU tar extensions, wrapper compression, reader/archive/writer API behavior, or tests. +2. Open the relevant section in `references/tar-format.md` and use the source-file pointers before changing code. +3. For parsing changes, cross-check `TarHeader.cs`, `TarHeader.Async.cs`, `EntryType.cs`, and matching sync/async tests. +4. For writer changes, verify both sync and async file/directory paths and `TarWriterOptions.HeaderFormat` behavior. +5. For support claims, keep unsupported features explicit: PAX write, sparse reconstruction, device/FIFO semantics, link-writing APIs, and writing `tar.xz`, `tar.zst`, or `tar.Z`. diff --git a/.agents/skills/tar-format/references/tar-format.md b/.agents/skills/tar-format/references/tar-format.md new file mode 100644 index 00000000..1ee10c1a --- /dev/null +++ b/.agents/skills/tar-format/references/tar-format.md @@ -0,0 +1,360 @@ +# Tar Format Reference + +This reference summarizes the Tar archive container format for SharpCompress work. It is locally authored from public format references and the current SharpCompress implementation. + +Primary external references: + +- POSIX `pax` and `ustar`: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html +- GNU tar basic format: https://www.gnu.org/software/tar/manual/html_node/Standard.html +- GNU tar extensions: https://www.gnu.org/software/tar/manual/html_node/Extensions.html + +Primary SharpCompress references: + +- `docs/TAR_SPEC.md` +- `docs/TAR_GAP_ANALYSIS.md` +- `src/SharpCompress/Factories/TarFactory.cs` +- `src/SharpCompress/Factories/TarWrapper.cs` +- `src/SharpCompress/Common/Tar/Headers/TarHeader.cs` +- `src/SharpCompress/Common/Tar/Headers/TarHeader.Async.cs` +- `src/SharpCompress/Common/Tar/Headers/EntryType.cs` +- `src/SharpCompress/Writers/Tar/TarWriter.cs` +- `src/SharpCompress/Writers/Tar/TarWriter.Async.cs` +- `src/SharpCompress/Writers/Tar/TarWriterOptions.cs` +- `tests/SharpCompress.Test/Tar/` + +## Contents + +- [Format Overview](#format-overview) +- [USTAR Header Layout](#ustar-header-layout) +- [Entry Type Flags](#entry-type-flags) +- [Numeric Fields](#numeric-fields) +- [Checksum](#checksum) +- [Path Names](#path-names) +- [PAX Extended Headers](#pax-extended-headers) +- [GNU Tar Extensions](#gnu-tar-extensions) +- [SharpCompress Support Matrix](#sharpcompress-support-matrix) +- [SharpCompress Read Behavior](#sharpcompress-read-behavior) +- [SharpCompress Write Behavior](#sharpcompress-write-behavior) +- [Known Limitations](#known-limitations) +- [Test Fixtures](#test-fixtures) + +## Format Overview + +A tar archive is a sequence of 512-byte blocks. Each archive member is represented by: + +```text +header block (512 bytes) +payload blocks, padded to a 512-byte boundary +``` + +The archive should end with two 512-byte blocks filled with zero bytes. Readers should be tolerant of missing end markers because real-world tar tools may produce archives without them. + +The header contains file metadata and the payload size. Tar has no central directory. Streaming readers must consume or skip each payload and its padding before the next header can be parsed. + +SharpCompress relies on this in `TarReadOnlySubStream`: disposing an entry stream consumes unread entry bytes plus 512-byte padding so the next header remains aligned. + +## USTAR Header Layout + +The POSIX USTAR header is exactly 512 bytes. Field offsets are byte offsets from the beginning of the header block. + +| Field | Offset | Length | Notes | +| ----- | ------ | ------ | ----- | +| `name` | 0 | 100 | File name or final path component | +| `mode` | 100 | 8 | Octal file mode | +| `uid` | 108 | 8 | Octal owner id | +| `gid` | 116 | 8 | Octal group id | +| `size` | 124 | 12 | Octal payload size, or GNU base-256 in some archives | +| `mtime` | 136 | 12 | Octal seconds since Unix epoch | +| `chksum` | 148 | 8 | Header checksum | +| `typeflag` | 156 | 1 | Entry type | +| `linkname` | 157 | 100 | Link target for hard/symbolic links | +| `magic` | 257 | 6 | Usually `ustar` followed by NUL | +| `version` | 263 | 2 | Usually `00` | +| `uname` | 265 | 32 | Owner name | +| `gname` | 297 | 32 | Group name | +| `devmajor` | 329 | 8 | Character/block device major number | +| `devminor` | 337 | 8 | Character/block device minor number | +| `prefix` | 345 | 155 | USTAR path prefix | +| padding | 500 | 12 | Unused padding to 512 bytes | + +SharpCompress parses the core fields in `TarHeader.Read` and `TarHeader.ReadAsync`. It reconstructs USTAR paths as `prefix + "/" + name` when `magic` is exactly `ustar` and `prefix` is non-empty. + +## Entry Type Flags + +Common POSIX typeflags: + +| Typeflag | Meaning | +| -------- | ------- | +| NUL | Regular file, older tar form | +| `0` | Regular file | +| `1` | Hard link | +| `2` | Symbolic link | +| `3` | Character device | +| `4` | Block device | +| `5` | Directory | +| `6` | FIFO | +| `7` | Contiguous file, reserved by POSIX historical usage | +| `x` | POSIX PAX local extended header for the following file | +| `g` | POSIX PAX global extended header for following files | + +GNU and other extension typeflags relevant to SharpCompress: + +| Typeflag | Meaning | +| -------- | ------- | +| `K` | GNU long link target for the next real entry | +| `L` | GNU long path name for the next real entry | +| `S` | GNU sparse file | +| `V` | GNU volume header | + +SharpCompress declares these in `EntryType.cs`: + +```text +File = 0 +OldFile = '0' +HardLink = '1' +SymLink = '2' +CharDevice = '3' +BlockDevice = '4' +Directory = '5' +Fifo = '6' +LongLink = 'K' +LongName = 'L' +SparseFile = 'S' +VolumeHeader = 'V' +LocalExtendedHeader = 'x' +GlobalExtendedHeader = 'g' +``` + +Declaration does not mean full semantic support. Sparse, device, FIFO, and volume-header semantics are not fully modeled by the public API. + +## Numeric Fields + +Standard tar numeric fields are ASCII octal values, usually NUL-terminated or space-padded depending on writer. Important fields include `mode`, `uid`, `gid`, `size`, `mtime`, `chksum`, `devmajor`, and `devminor`. + +GNU tar can use base-256 binary encoding for values that exceed the octal field range: + +- A leading byte with bit `0x80` indicates a positive binary value. +- GNU documentation also describes `0xff` as a negative two's-complement marker. +- The value bytes are big-endian. + +SharpCompress currently handles binary `size` fields when bit `0x80` is set and writes large sizes in GNU long-link mode using a base-256-style binary `size` field. It also has an old GNU uid/gid quirk reader for fields beginning with `0x80 0x00`. + +## Checksum + +The checksum is the simple sum of all 512 header bytes, treating the 8-byte checksum field at offset 148 as spaces (`0x20`) during calculation. + +SharpCompress checksum behavior: + +- `RecalculateChecksum` fills the checksum field with eight spaces and sums bytes as unsigned values. +- `checkChecksum` accepts both POSIX unsigned sums and signed-byte sums used by some historical tar implementations. +- An all-zero block is treated as an empty/end marker case. + +When editing parser code, keep checksum compatibility broad enough for old archives. When editing writer code, use POSIX unsigned checksum output. + +## Path Names + +Classic tar has a 100-byte `name` field. POSIX USTAR extends this with a 155-byte `prefix` field. + +USTAR path reconstruction: + +```text +full path = prefix + "/" + name +``` + +USTAR writer constraints: + +- `name` must fit within the 100-byte name field. +- `prefix` must fit within the 155-byte prefix field. +- Splitting is normally done at a directory separator. + +SharpCompress write behavior: + +- `TarHeaderWriteFormat.USTAR` tries to split long paths into `prefix` and `name`. +- If a path cannot fit, SharpCompress throws `InvalidFormatException` and tells callers to use GNU tar format. +- `TarHeaderWriteFormat.GNU_TAR_LONG_LINK` writes GNU long-name metadata for names over 100 bytes. + +SharpCompress path normalization in `TarWriter`: + +- Converts backslashes to `/`. +- Removes drive prefixes before `:`. +- Trims leading and trailing `/` for file entries. +- Ensures directory entries end with `/`. +- Skips empty or root-equivalent directory names. + +## PAX Extended Headers + +POSIX PAX uses regular tar header blocks with special typeflags and a payload of UTF-8 key/value records. + +PAX header typeflags: + +- `x`: local extended header, applies to the next real file entry. +- `g`: global extended header, applies to subsequent entries until overridden by another global or local header. + +Each record has this form: + +```text + =\n +``` + +The decimal length includes every byte in the record, including the digits of the length itself, the space, key, equals sign, value, and newline. + +SharpCompress PAX read support is intentionally limited to selected keys: + +| Key | Effect | +| --- | ------ | +| `path` | Overrides entry path/name | +| `linkpath` | Overrides hard/symbolic link target | +| `size` | Overrides payload size | +| `mtime` | Overrides modification time | +| `uid` | Overrides owner id | +| `gid` | Overrides group id | +| `mode` | Overrides mode | + +Local metadata overrides global metadata. Unknown PAX keys are ignored. PAX payload reads are capped at 65536 bytes to avoid memory exhaustion from malformed archives. + +SharpCompress does not currently write PAX headers. + +## GNU Tar Extensions + +SharpCompress supports the most common GNU extensions needed for interoperability. + +### Long Name and Long Link + +GNU long-name and long-link records are synthetic entries that apply to the next real entry: + +| Typeflag | Purpose | +| -------- | ------- | +| `L` | Long file name/path for next entry | +| `K` | Long link target for next entry | + +The payload contains the long name or link target, padded to a 512-byte boundary. SharpCompress caps long-name payload reads at 32768 bytes. + +Writer behavior in `GNU_TAR_LONG_LINK` mode: + +1. Write a synthetic `././@LongLink` header with `typeflag = 'L'` when the name exceeds 100 bytes. +2. Write the long-name payload and 512-byte padding. +3. Write the actual file or directory header. + +### Sparse Files + +GNU sparse tar uses `typeflag = 'S'` and additional sparse-map metadata. POSIX PAX sparse variants use keys such as `GNU.sparse.*`. + +SharpCompress currently recognizes the sparse entry type enum value but does not reconstruct sparse holes or semantically expose sparse maps. Treat sparse support as unsupported unless implementing full reconstruction and tests. + +### Base-256 Numeric Fields + +GNU tar uses base-256 binary fields for out-of-range numeric values. SharpCompress reads binary `size` fields and writes large sizes in GNU mode. + +## SharpCompress Support Matrix + +Wrapper detection is defined in `TarWrapper.Wrappers`. Detection is content-based: wrapper detection is followed by a tar-header probe of the decompressed payload. + +| Wrapper | Extensions | Read | Write | +| ------- | ---------- | ---- | ----- | +| Plain tar | `tar` | Yes | Yes | +| Tar + GZip | `tar.gz`, `taz`, `tgz` | Yes | Yes | +| Tar + BZip2 | `tar.bz2`, `tb2`, `tbz`, `tbz2`, `tz2` | Yes | Yes | +| Tar + LZip | `tar.lz` | Yes | Yes | +| Tar + XZ | `tar.xz`, `txz` | Yes | No | +| Tar + ZStandard | `tar.zst`, `tar.zstd`, `tzst`, `tzstd` | Yes | No | +| Tar + LZW compress | `tar.Z`, `tZ`, `taZ` | Yes | No | + +Writer support currently accepts only these compression types: + +- `CompressionType.None` +- `CompressionType.GZip` +- `CompressionType.BZip2` +- `CompressionType.LZip` + +Other compression types throw `InvalidFormatException`. + +## SharpCompress Read Behavior + +Reader API: + +- `TarReader` is forward-only and supports non-seekable streams. +- `ReaderFactory.OpenReader` can auto-detect tar and wrapper compression. +- Entry streams must be consumed or disposed so the next header can be aligned. + +Archive API: + +- `TarArchive.OpenArchive(Stream)` and `TarArchive.OpenAsyncArchive(Stream)` require seekable streams. +- File/path overloads own the opened file stream. +- Compressed tar archive access follows streaming semantics over the decompressed stream rather than full random-access semantics. + +Parsed metadata surfaced through entries includes: + +- `Key` +- `LinkTarget` +- `Size` +- `CompressedSize` +- `LastModifiedTime` +- `IsDirectory` +- `Mode` +- `UserID` +- `GroupId` + +Tar entries are always reported as unencrypted and CRC is always `0`. + +## SharpCompress Write Behavior + +`TarWriter` is forward-only. It writes a header, payload, payload padding, and finally two zero blocks on dispose when `FinalizeArchiveOnClose` is true. + +Important writer rules: + +- Tar requires payload size in the header. +- If the source stream is non-seekable and no `size` is supplied, `TarWriter` throws `ArgumentException`. +- `TarWriterOptions.HeaderFormat` defaults to `GNU_TAR_LONG_LINK`. +- Current sync and async file and directory write paths honor `HeaderFormat`. +- `USTAR` mode writes USTAR headers and throws when paths cannot fit. +- `GNU_TAR_LONG_LINK` mode writes GNU long-name records for long paths. +- The public writer supports regular files and directories, not links, devices, FIFOs, sparse maps, or PAX metadata. + +Writer metadata is narrower than reader metadata. It writes name, size, last modified time, and file/directory type, and uses fixed mode/user/group defaults in the header. + +## Known Limitations + +Keep these limitations explicit in code comments, docs, and tests: + +- No PAX write support. +- No sparse-file reconstruction or sparse write support. +- No public API for writing symbolic links or hard links. +- No device or FIFO metadata object model beyond internal type recognition. +- No write support for `tar.xz`, `tar.zst`, or `tar.Z`. +- PAX read support is limited to `path`, `linkpath`, `size`, `mtime`, `uid`, `gid`, and `mode`. +- Unknown PAX keys are ignored. +- Stream-based `TarArchive` open requires seekable input. + +## Test Fixtures + +Representative fixtures in `tests/TestArchives/Archives/`: + +- `Tar.tar` +- `Tar.tar.gz` +- `Tar.tar.bz2` +- `Tar.tar.lz` +- `Tar.tar.xz` +- `Tar.tar.zst` +- `Tar.tar.Z` +- `Tar.oldgnu.tar.gz` +- `very long filename.tar` +- `ustar with long names.tar` +- `Tar.LongPathsWithLongNameExtension.tar` +- `Tar.PaxLocalHeader.tar` +- `Tar.PaxLocalHeader.Link.tar` +- `Tar.PaxGlobalHeader.tar` +- `Tar.PaxGlobalHeader.Link.tar` +- `Tar.Empty.tar` +- `TarCorrupted.tar` +- `TarWithSymlink.tar.gz` + +Representative test files: + +- `tests/SharpCompress.Test/Tar/TarReaderTests.cs` +- `tests/SharpCompress.Test/Tar/TarReaderAsyncTests.cs` +- `tests/SharpCompress.Test/Tar/TarArchiveTests.cs` +- `tests/SharpCompress.Test/Tar/TarArchiveAsyncTests.cs` +- `tests/SharpCompress.Test/Tar/TarWriterTests.cs` +- `tests/SharpCompress.Test/Tar/TarWriterAsyncTests.cs` +- `tests/SharpCompress.Test/Tar/TarWriterDirectoryTests.cs` +- `tests/SharpCompress.Test/Tar/TarArchiveDirectoryTests.cs` diff --git a/.agents/skills/xz-lzma-format/SKILL.md b/.agents/skills/xz-lzma-format/SKILL.md new file mode 100644 index 00000000..7defd7e9 --- /dev/null +++ b/.agents/skills/xz-lzma-format/SKILL.md @@ -0,0 +1,22 @@ +--- +name: xz-lzma-format +description: Reference the XZ container format and LZMA/LZMA2 decoder behavior. Use when an AI agent needs to answer questions or make code changes involving XZ headers, blocks, indexes, checks, CRC64/XZ, LZMA2 chunks, LZMA end markers, corrupt .xz test files, or SharpCompress XZ/LZMA parsing and decompression behavior. +--- + +# XZ and LZMA Format + +Use this skill for work at the boundary between the XZ container and the LZMA/LZMA2 compression streams. It captures the key details needed for SharpCompress XZ block parsing, XZ integrity checks, and LZMA2 decoder corruption handling. + +## Reference + +- Read [references/xz-lzma-format.md](references/xz-lzma-format.md) when the task depends on XZ binary layout, XZ block checks, CRC64/XZ parameters, LZMA2 chunk control bytes, LZMA end-of-payload markers, or XZ Utils bad-file expectations. +- Treat XZ as a container around filter chains. Do not assume raw LZMA/LZMA2 behavior is equivalent to XZ stream validation. +- Use the linked XZ Utils/liblzma sources in the reference when matching corruption behavior. The test corpus contains intentionally bad files that must throw even when they can produce all expected output bytes. + +## Workflow + +1. Identify the layer involved: XZ stream header/footer, block header, compressed data padding, block check, index, filter chain, LZMA2 chunk framing, or raw LZMA range decoding. +2. Open `references/xz-lzma-format.md` and cross-check the relevant spec/source section before changing parser or decoder code. +3. For XZ checksum work, verify the stream check type from the XZ header. Use CRC32, CRC64/XZ, SHA-256, or no check according to the header, not according to a test fixture assumption. +4. For LZMA2 corruption work, compare SharpCompress behavior against the XZ Utils test corpus notes and liblzma decoder state model. +5. Test both sync and async paths. Relevant files are `XZBlock.cs`, `XZBlock.Async.cs`, `XZStream.cs`, `XZStream.Async.cs`, `LzmaStream.cs`, `LzmaStream.Async.cs`, `LzmaDecoder.cs`, and `LzmaDecoder.Async.cs`. diff --git a/.agents/skills/xz-lzma-format/references/xz-lzma-format.md b/.agents/skills/xz-lzma-format/references/xz-lzma-format.md new file mode 100644 index 00000000..60538f6f --- /dev/null +++ b/.agents/skills/xz-lzma-format/references/xz-lzma-format.md @@ -0,0 +1,174 @@ +# XZ and LZMA/LZMA2 Reference + +This reference summarizes the XZ container and LZMA/LZMA2 decoder facts that matter for SharpCompress maintenance. It is a local guide, not a full copy of the specs. + +## Upstream References + +- XZ file format specification: `https://raw.githubusercontent.com/tukaani-project/xz/master/doc/xz-file-format.txt` +- liblzma LZMA2 decoder: `https://raw.githubusercontent.com/tukaani-project/xz/master/src/liblzma/lzma/lzma2_decoder.c` +- liblzma LZMA decoder: `https://raw.githubusercontent.com/tukaani-project/xz/master/src/liblzma/lzma/lzma_decoder.c` +- XZ Utils test file descriptions: `https://raw.githubusercontent.com/tukaani-project/xz/master/tests/files/README` +- XZ range decoder reference, useful for `rc_is_finished` and normalization behavior: `https://raw.githubusercontent.com/tukaani-project/xz/master/src/liblzma/rangecoder/range_decoder.h` + +## SharpCompress Pointers + +- XZ container stream: `src/SharpCompress/Compressors/Xz/XZStream.cs`, `src/SharpCompress/Compressors/Xz/XZStream.Async.cs` +- XZ block parsing/checks: `src/SharpCompress/Compressors/Xz/XZBlock.cs`, `src/SharpCompress/Compressors/Xz/XZBlock.Async.cs` +- XZ header/footer/index: `XZHeader.cs`, `XZFooter.cs`, `XZIndex.cs`, `XZIndexRecord.cs`, and async counterparts. +- XZ LZMA2 filter wrapper: `src/SharpCompress/Compressors/Xz/Filters/Lzma2Filter.cs`, `Lzma2Filter.Async.cs` +- LZMA/LZMA2 stream and decoder: `src/SharpCompress/Compressors/LZMA/LzmaStream.cs`, `LzmaStream.Async.cs`, `LzmaDecoder.cs`, `LzmaDecoder.Async.cs`, `RangeCoder/RangeCoder.cs`, `RangeCoder/RangeCoder.Async.cs` +- Core tests: `tests/SharpCompress.Test/Xz/*`, `tests/SharpCompress.Test/Streams/LzmaStreamTests.cs`, `tests/SharpCompress.Test/Streams/LzmaStreamAsyncTests.cs` +- Corruption fixture discussed here: `tests/TestArchives/Archives/bad-1-lzma2-7.xz` + +## XZ Container Structure + +An XZ file is one or more XZ streams. A typical single stream is: + +```text +Stream Header -> Block(s) -> Index -> Stream Footer +``` + +Important layout rules: + +- XZ files and XZ streams are aligned to four-byte boundaries. +- Stream header magic is `FD 37 7A 58 5A 00`. +- Stream footer magic is `59 5A` (`YZ`). +- Stream header/footer flags include the check type used for every block in the stream. +- A block consists of `Block Header`, `Compressed Data`, `Block Padding`, and `Check`. +- Block padding is 0-3 null bytes and makes the block size a multiple of four. +- The Index contains one record per block: `Unpadded Size` and `Uncompressed Size`. + +## Variable-Length Integers + +XZ variable-length integers encode seven data bits per byte. The high bit means continuation. Current XZ limits the encoded integer to nine bytes/63 bits. + +SharpCompress implementation: + +- `MultiByteIntegers.ReadXZInteger` reads these values. +- It rejects overlong encodings when a continuation byte is `0x00`. +- Use this for block sizes, filter IDs, filter property sizes, index counts, and index record fields. + +## XZ Checks Versus Raw LZMA + +XZ checks are container-level integrity checks over uncompressed block data. Raw LZMA/LZMA2 decoding does not provide the same container-level CRC validation. + +Supported XZ check IDs in SharpCompress: + +- `0x00`: none, 0 bytes. +- `0x01`: CRC32, 4 bytes. +- `0x04`: CRC64/XZ, 8 bytes. +- `0x0A`: SHA-256, 32 bytes. + +CRC64/XZ details: + +- Polynomial: reflected ECMA polynomial `0xC96C5795D7870F42`. +- Initial value: `0xffffffffffffffff`. +- Final XOR: `0xffffffffffffffff`. +- Stored little-endian in the block check field. +- Test vector for `"123456789"`: `0x995DC9BBDF1939FA`. + +Common pitfall: + +- CRC64/XZ is not the older `Iso3309Polynomial = 0xD800000000000000` path that existed in SharpCompress's generic `Crc64` helper. Using that produces wrong XZ block check values. + +## XZ Block Check Handling + +When reading an `XZBlock`: + +1. Parse and CRC-validate the block header. +2. Build the filter chain in reverse order from the List of Filter Flags. +3. Read uncompressed bytes through the filter chain and update the selected check over the uncompressed bytes. +4. At block end, skip/validate block padding. +5. Read the check field and compare to the computed value. + +Important behavior: + +- A short `Stream.Read` result does not universally mean EOF. Be careful when using `bytesRead != count` as an end-of-block signal. +- Tests using `StreamReader.ReadToEnd()` often expose end-of-block behavior because they force padding/check validation. +- The check type must match the XZ stream header. For example, a fixture may have one XZ stream using CRC32 and another using CRC64; do not hard-code CRC64 in block tests. + +## LZMA2 Chunks + +LZMA2 is the only LZMA-family filter defined for XZ (`Filter ID 0x21`). Raw LZMA is not an XZ filter. + +LZMA2 control byte categories from liblzma: + +- `0x00`: LZMA2 end marker. +- `0x01` or `>= 0xE0`: dictionary reset; the next LZMA chunk must set new properties. +- `>= 0x80`: LZMA chunk. The control byte and following two bytes encode uncompressed chunk size. The next two bytes encode compressed chunk size. Some control values also provide new LZMA properties. +- `0x02`: uncompressed chunk without dictionary reset. +- `0x01`: uncompressed chunk with dictionary reset. +- `0x03..0x7F`: invalid/reserved control values. + +For LZMA chunks, the LZMA2 decoder must track: + +- Exact uncompressed chunk size. +- Exact compressed chunk size. +- Whether LZMA properties are needed. +- Whether dictionary reset is required. +- Whether the inner LZMA stream saw an LZMA end-of-payload marker. + +## LZMA End-Of-Payload Marker In LZMA2 + +The XZ Utils bad-file corpus describes `bad-1-lzma2-7.xz` as: + +```text +bad-1-lzma2-7.xz has EOPM at LZMA level. +``` + +Meaning: + +- The outer XZ container can be parsed. +- The LZMA2 stream can produce all advertised uncompressed bytes. +- The inner raw LZMA decoder still reaches an LZMA end-of-payload marker (`rep0 == uint.MaxValue`). +- LZMA2 must reject this. End-of-payload markers are for raw LZMA cases with unknown size; they are not valid as an LZMA-level terminator inside an LZMA2 chunk. + +liblzma behavior: + +- `lzma2_decoder.c` calls the inner LZMA decoder with a known `uncompressed_size` and `allow_eopm = false`. +- `lzma_decoder.c` treats EOPM as data error when EOPM is not valid. +- The XZ Utils test README says all `bad-*` files must cause decoder errors. + +SharpCompress maintenance guidance: + +- If a bad LZMA2 fixture produces all output bytes but `xz --test` reports corrupt data, inspect the inner decoder state, not just output length or XZ block check. +- In SharpCompress, `Decoder.HasEndMarker => _rep0 == uint.MaxValue` is the useful signal for the `bad-1-lzma2-7.xz` case. +- Validate both sync and async paths; `Stream.CopyToAsync` can use byte-array or `Memory` read overloads depending on target framework and wrapper stream. + +## Exception Expectations + +`DataErrorException` is internal and derives from `SharpCompressException`. Some public XZ parsing failures throw `InvalidFormatException`; raw decoder corruption may surface as `SharpCompressException` via `DataErrorException` unless the wrapper maps it. + +Testing guidance: + +- Use exact `InvalidFormatException` when the code path is XZ header/footer/block/index/check validation. +- Use `Assert.ThrowsAnyAsync` or equivalent when the desired behavior is simply that corrupt LZMA/LZMA2 data is rejected. +- Do not weaken tests to accept no exception for XZ Utils `bad-*` fixtures. + +## Useful Commands + +Use the system `xz` tool as an oracle when available: + +```bash +xz --test --verbose tests/TestArchives/Archives/bad-1-lzma2-7.xz +xz --robot --list --verbose tests/TestArchives/Archives/bad-1-lzma2-7.xz +``` + +Expected for `bad-1-lzma2-7.xz`: + +```text +xz: tests/TestArchives/Archives/bad-1-lzma2-7.xz: Compressed data is corrupt +``` + +Targeted SharpCompress tests: + +```bash +dotnet test tests/SharpCompress.Test/SharpCompress.Test.csproj --framework net10.0 --filter "FullyQualifiedName~SharpCompress.Test.Streams.LzmaStream|FullyQualifiedName~SharpCompress.Test.Xz" +``` + +## Current SharpCompress Gotchas + +- XZ index CRC32 verification may still be incomplete; check `XZIndex.VerifyCrc32` before relying on index corruption detection. +- XZ block/index size semantics are easy to confuse. `Unpadded Size` excludes block padding but includes header, compressed data, and check. `Uncompressed Size` is raw output size. +- LZMA2 chunks include their own compressed and uncompressed chunk sizes; these are separate from XZ block header/index sizes. +- `StreamReader.ReadToEnd()` and `TransferToAsync(Stream.Null, long.MaxValue)` are useful for forcing full-stream validation. diff --git a/.agents/skills/zip-format/SKILL.md b/.agents/skills/zip-format/SKILL.md new file mode 100644 index 00000000..04bc1e9d --- /dev/null +++ b/.agents/skills/zip-format/SKILL.md @@ -0,0 +1,22 @@ +--- +name: zip-format +description: Reference the ZIP/ZIP64/PKWARE APPNOTE archive container format. Use when an AI agent needs to answer questions or make code changes involving ZIP local headers, central directory records, EOCD/Zip64 records, data descriptors, general purpose bit flags, compression method IDs, extra fields, encryption markers, split archives, or SharpCompress Zip parsing and writing behavior. +--- + +# Zip Format + +Use this skill for ZIP container-format work. It provides a local, SharpCompress-oriented reference for PKWARE APPNOTE ZIP records, ZIP64, compression method IDs, extra fields, and the current SharpCompress Zip implementation. + +## Reference + +- Read [references/zip-format.md](references/zip-format.md) when the task depends on ZIP binary layout, record signatures, local vs central directory metadata, data descriptor rules, Zip64 sentinel values, extra field parsing, compression method IDs, encryption markers, split archive handling, or current SharpCompress Zip support boundaries. +- Treat the reference as an implementation guide, not a standards replacement. It summarizes PKWARE APPNOTE 6.3.10 from `https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT` and documents SharpCompress-specific behavior and limitations. +- Prefer the SharpCompress support matrix in the reference over generic ZIP assumptions when changing code. ZIP is highly extensible, and SharpCompress intentionally supports only selected records, methods, and extra fields. + +## Workflow + +1. Identify which layer is involved: header discovery, local file headers, central directory headers, EOCD/Zip64 records, data descriptors, extra fields, compression methods, encryption, reader/archive/writer API behavior, or tests. +2. Open the relevant section in `references/zip-format.md` and use the source-file pointers before changing code. +3. For parsing changes, cross-check sync and async header readers: `ZipHeaderFactory.cs`, `ZipHeaderFactory.Async.cs`, `SeekableZipHeaderFactory.cs`, `StreamingZipHeaderFactory.cs`, `ZipFileEntry.cs`, `LocalEntryHeader.cs`, and `DirectoryEntryHeader.cs`. +4. For writer changes, verify local header, post-data descriptor, central directory, Zip64, and sync/async paths together. +5. For support claims, keep unsupported features explicit: central directory encryption, strong encryption records, XZ writing, non-standard compression methods, broad extra-field semantics, and non-seekable Zip64 writing. diff --git a/.agents/skills/zip-format/references/zip-format.md b/.agents/skills/zip-format/references/zip-format.md new file mode 100644 index 00000000..defafa56 --- /dev/null +++ b/.agents/skills/zip-format/references/zip-format.md @@ -0,0 +1,456 @@ +# ZIP Format Reference + +This reference summarizes the ZIP archive container format for SharpCompress work. It is locally authored from PKWARE APPNOTE and the current SharpCompress implementation. + +Primary external reference: + +- PKWARE APPNOTE.TXT - ZIP File Format Specification, version 6.3.10: https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT + +Primary SharpCompress references: + +- `docs/FORMATS.md` +- `src/SharpCompress/Factories/ZipFactory.cs` +- `src/SharpCompress/Archives/Zip/ZipArchive.cs` +- `src/SharpCompress/Readers/Zip/ZipReader.cs` +- `src/SharpCompress/Writers/Zip/ZipWriter.cs` +- `src/SharpCompress/Writers/Zip/ZipWritingStream.cs` +- `src/SharpCompress/Writers/Zip/ZipCentralDirectoryEntry.cs` +- `src/SharpCompress/Common/Zip/ZipCompressionMethod.cs` +- `src/SharpCompress/Common/Zip/ZipEntry.cs` +- `src/SharpCompress/Common/Zip/ZipFilePart.cs` +- `src/SharpCompress/Common/Zip/ZipHeaderFactory.cs` +- `src/SharpCompress/Common/Zip/ZipHeaderFactory.Async.cs` +- `src/SharpCompress/Common/Zip/SeekableZipHeaderFactory.cs` +- `src/SharpCompress/Common/Zip/SeekableZipHeaderFactory.Async.cs` +- `src/SharpCompress/Common/Zip/StreamingZipHeaderFactory.cs` +- `src/SharpCompress/Common/Zip/StreamingZipHeaderFactory.Async.cs` +- `src/SharpCompress/Common/Zip/Headers/ZipFileEntry.cs` +- `src/SharpCompress/Common/Zip/Headers/LocalEntryHeader.cs` +- `src/SharpCompress/Common/Zip/Headers/DirectoryEntryHeader.cs` +- `src/SharpCompress/Common/Zip/Headers/LocalEntryHeaderExtraFactory.cs` +- `tests/SharpCompress.Test/Zip/` + +## Contents + +- [Format Overview](#format-overview) +- [Record Signatures](#record-signatures) +- [Local File Header](#local-file-header) +- [Central Directory Header](#central-directory-header) +- [End Of Central Directory](#end-of-central-directory) +- [Data Descriptors](#data-descriptors) +- [General Purpose Bit Flags](#general-purpose-bit-flags) +- [Compression Methods](#compression-methods) +- [Extra Fields](#extra-fields) +- [Zip64](#zip64) +- [Names, Comments, And Encoding](#names-comments-and-encoding) +- [Encryption](#encryption) +- [Seekable And Streaming Reads](#seekable-and-streaming-reads) +- [SharpCompress Support Matrix](#sharpcompress-support-matrix) +- [SharpCompress Write Behavior](#sharpcompress-write-behavior) +- [Known Limitations](#known-limitations) +- [Test Fixtures](#test-fixtures) + +## Format Overview + +A ZIP archive stores each file as a local file record followed by compressed or stored payload bytes. Metadata is repeated in a central directory near the end of the archive, followed by the end of central directory record. + +High-level APPNOTE layout: + +```text +[local file header 1] +[encryption header 1] +[file data 1] +[data descriptor 1] +... +[local file header n] +[encryption header n] +[file data n] +[data descriptor n] +[archive decryption header] +[archive extra data record] +[central directory header 1] +... +[central directory header n] +[zip64 end of central directory record] +[zip64 end of central directory locator] +[end of central directory record] +``` + +All ordinary multi-byte ZIP fields are little-endian unless APPNOTE says otherwise. ZIP readers must identify records by signatures rather than by extension. + +SharpCompress has two important read modes: + +- Seekable Archive API reads the central directory first and then seeks to local headers for entry data. +- Streaming Reader API processes local headers and payloads in order and cannot rely on central directory data unless it is already supplied by the caller/path flow. + +## Record Signatures + +Common ZIP signatures used by SharpCompress: + +| Record | Signature | SharpCompress constant | +| --- | --- | --- | +| Local file header | `0x04034b50` | `ENTRY_HEADER_BYTES` | +| Data descriptor | `0x08074b50` | `POST_DATA_DESCRIPTOR` | +| Central directory file header | `0x02014b50` | `DIRECTORY_START_HEADER_BYTES` | +| End of central directory | `0x06054b50` | `DIRECTORY_END_HEADER_BYTES` | +| Digital signature | `0x05054b50` | `DIGITAL_SIGNATURE` | +| Split archive marker | `0x30304b50` | `SPLIT_ARCHIVE_HEADER_BYTES` | +| Zip64 end of central directory | `0x06064b50` | `ZIP64_END_OF_CENTRAL_DIRECTORY` | +| Zip64 end of central directory locator | `0x07064b50` | `ZIP64_END_OF_CENTRAL_DIRECTORY_LOCATOR` | + +`ZipHeaderFactory.IsHeader` recognizes these signatures while streaming. + +## Local File Header + +The local file header immediately precedes optional encryption metadata and file data. + +| Field | Size | Notes | +| --- | --- | --- | +| Signature | 4 | `0x04034b50` | +| Version needed to extract | 2 | Feature/version indicator | +| General purpose bit flag | 2 | Encryption, data descriptor, EFS, method-specific bits | +| Compression method | 2 | See [Compression Methods](#compression-methods) | +| Last mod file time | 2 | MS-DOS time | +| Last mod file date | 2 | MS-DOS date | +| CRC-32 | 4 | Zero when bit 3 defers values to data descriptor | +| Compressed size | 4 | `0xffffffff` sentinel when Zip64 extra carries value | +| Uncompressed size | 4 | `0xffffffff` sentinel when Zip64 extra carries value | +| File name length | 2 | Byte count | +| Extra field length | 2 | Byte count | +| File name | variable | Not NUL-terminated | +| Extra field | variable | Sequence of ID/length/data blocks | + +SharpCompress reads this in `LocalEntryHeader.Read` and then decodes names, loads extra fields, applies Unicode path extra data, applies Zip64 extra data, and applies selected Unix time data. + +## Central Directory Header + +The central directory file header repeats entry metadata and adds fields needed for random access. + +| Field | Size | Notes | +| --- | --- | --- | +| Signature | 4 | `0x02014b50` | +| Version made by | 2 | Host/version metadata | +| Version needed to extract | 2 | Feature/version indicator | +| General purpose bit flag | 2 | Same semantic space as local header | +| Compression method | 2 | See [Compression Methods](#compression-methods) | +| Last mod file time | 2 | MS-DOS time | +| Last mod file date | 2 | MS-DOS date | +| CRC-32 | 4 | Entry CRC | +| Compressed size | 4 | Zip64 sentinel possible | +| Uncompressed size | 4 | Zip64 sentinel possible | +| File name length | 2 | Byte count | +| Extra field length | 2 | Byte count | +| File comment length | 2 | Byte count | +| Disk number start | 2 | Zip64 sentinel possible | +| Internal file attributes | 2 | Host/application metadata | +| External file attributes | 4 | Host-dependent file attributes | +| Relative offset of local header | 4 | Zip64 sentinel possible | +| File name | variable | Not NUL-terminated | +| Extra field | variable | Sequence of ID/length/data blocks | +| File comment | variable | Not NUL-terminated | + +SharpCompress reads this in `DirectoryEntryHeader.Read`. Seekable ZIP reads use central directory entries to discover the archive contents, then `SeekableZipHeaderFactory.GetLocalHeader` seeks to local headers and copies central-directory-only metadata onto the local entry. + +## End Of Central Directory + +Every normal ZIP archive ends with exactly one EOCD record. The minimum EOCD length is 22 bytes, plus an optional ZIP file comment up to 65535 bytes. + +EOCD fields: + +| Field | Size | Notes | +| --- | --- | --- | +| Signature | 4 | `0x06054b50` | +| Number of this disk | 2 | Split/spanned metadata | +| Central directory start disk | 2 | Split/spanned metadata | +| Entries on this disk | 2 | `0xffff` sentinel when Zip64 is needed | +| Total entries | 2 | `0xffff` sentinel when Zip64 is needed | +| Central directory size | 4 | `0xffffffff` sentinel when Zip64 is needed | +| Central directory offset | 4 | `0xffffffff` sentinel when Zip64 is needed | +| ZIP file comment length | 2 | Byte count | +| ZIP file comment | variable | Archive-level comment | + +`SeekableZipHeaderFactory.SeekBackToHeader` searches backwards from the end of the stream across the maximum EOCD/comment search window. + +## Data Descriptors + +When general purpose bit 3 is set, the local header CRC and size fields are placeholders. The actual values follow file data in a data descriptor. + +Descriptor forms: + +```text +[optional signature 0x08074b50] +crc-32 4 bytes +compressed size 4 or 8 bytes +uncompressed size 4 or 8 bytes +``` + +APPNOTE says the signature was not originally assigned but is commonly used. SharpCompress handles descriptors with and without the signature in streaming reads. + +Important SharpCompress behavior: + +- `StreamingZipHeaderFactory` reads post-data descriptors after the previous entry stream has been consumed. +- Streaming descriptor parsing has compatibility logic for 32-bit and 64-bit sizes. +- `ZipWriter` writes a descriptor with signature for non-seekable output when Zip64 is not required. +- `ZipWriter` intentionally rejects Zip64 on non-seekable streams. + +## General Purpose Bit Flags + +Selected flags relevant to SharpCompress: + +| Bit | Meaning | +| --- | --- | +| 0 | Entry is encrypted | +| 1 | Method-specific. For LZMA method 14, set means EOS marker is used. For Implode it has dictionary-size meaning. | +| 2 | Method-specific. For Deflate/Deflate64 it encodes compression option; for Implode it has tree-count meaning. | +| 3 | CRC and sizes are deferred to a data descriptor after file data | +| 6 | Strong encryption | +| 11 | Language encoding flag (EFS): file name and comment are UTF-8 | +| 13 | Central directory encryption masks selected local header values | + +SharpCompress decodes names/comments as UTF-8 when EFS is set. For ZIP LZMA writing on non-seekable output, it sets bit 1 for EOS marker behavior. + +## Compression Methods + +APPNOTE compression method IDs relevant to SharpCompress and nearby unsupported methods: + +| ID | APPNOTE method | SharpCompress status | +| --- | --- | --- | +| 0 | Stored | Read/write | +| 1 | Shrunk | Read | +| 2 | Reduced factor 1 | Read | +| 3 | Reduced factor 2 | Read | +| 4 | Reduced factor 3 | Read | +| 5 | Reduced factor 4 | Read | +| 6 | Imploded | Read | +| 8 | Deflated | Read/write | +| 9 | Deflate64 | Read | +| 12 | BZIP2 | Read/write | +| 14 | LZMA | Read/write | +| 93 | Zstandard | Read/write | +| 95 | XZ | Read | +| 98 | PPMd version I, Rev 1 | Read/write | +| 99 | AE-x encryption marker | Read for WinZip AES handling | + +SharpCompress declares these in `ZipCompressionMethod.cs`: + +```text +None = 0 +Shrink = 1 +Reduce1 = 2 +Reduce2 = 3 +Reduce3 = 4 +Reduce4 = 5 +Explode = 6 +Deflate = 8 +Deflate64 = 9 +BZip2 = 12 +LZMA = 14 +ZStandard = 93 +Xz = 95 +PPMd = 98 +WinzipAes = 0x63 +``` + +ZIP has method 14 for LZMA and method 95 for XZ. There is no separate APPNOTE ZIP compression method named LZMA2 in the method table. XZ commonly uses LZMA2 internally, but a ZIP entry using APPNOTE method 95 is an XZ-compressed ZIP entry, not a separate LZMA2 ZIP method. + +`ZipFilePart.ToCompressionType` maps supported read methods to public `CompressionType` values and throws for unsupported methods before decompression. `ZipEntry.CompressionType` reports the public entry compression type, including `CompressionType.Xz` for method 95. + +## Extra Fields + +APPNOTE extra fields use this generic structure: + +```text +header id 2 bytes +data size 2 bytes +data variable +``` + +SharpCompress parses extra fields in `ZipFileEntry.LoadExtra`. Unknown or unsupported extra fields become `NotImplementedExtraData` and are preserved only as raw data for internal parsing decisions. + +Recognized extra fields: + +| Header ID | Meaning | SharpCompress type | +| --- | --- | --- | +| `0x0001` | Zip64 extended information | `Zip64ExtendedInformationExtraField` | +| `0x5455` | Extended timestamp / Unix time | `UnixTimeExtraField` | +| `0x7075` | Info-ZIP Unicode path | `ExtraUnicodePathExtraField` | +| `0x9901` | WinZip AES | Raw `ExtraData` used by AES logic | + +Zip64 extra values appear only when the corresponding 16-bit or 32-bit field in the local or central directory is set to its maximum sentinel. Values must appear in APPNOTE order: + +1. Original/uncompressed size +2. Compressed size +3. Relative header offset +4. Disk start number + +`Zip64ExtendedInformationExtraField.Process` enforces the required byte count for the sentinel fields being resolved. + +## Zip64 + +Zip64 extends size, count, and offset fields beyond classic ZIP limits. Classic fields use sentinel values when the real value is stored elsewhere: + +| Classic field size | Sentinel | +| --- | --- | +| 2 bytes | `0xffff` | +| 4 bytes | `0xffffffff` | + +Zip64 structures: + +- Zip64 extended information extra field (`0x0001`) carries per-entry sizes and offsets. +- Zip64 end of central directory record (`0x06064b50`) carries archive-level counts, central directory size, and central directory offset. +- Zip64 end of central directory locator (`0x07064b50`) points to the Zip64 EOCD record. + +SharpCompress read behavior: + +- Seekable reads detect Zip64 through EOCD sentinel values and then locate the Zip64 EOCD locator and record. +- Local and central entry readers apply Zip64 extra data when size/offset fields contain sentinels. + +SharpCompress write behavior: + +- `ZipWriterOptions.UseZip64` controls whether local headers reserve Zip64 extra data for entries. +- `ZipWriter` emits Zip64 EOCD and locator when entry counts, central directory size, or offsets require them. +- Non-seekable Zip64 writing is rejected because current post-data descriptor handling cannot safely represent the required Zip64 values in all cases. + +## Names, Comments, And Encoding + +APPNOTE file name and comment fields are length-prefixed byte sequences, not NUL-terminated strings. + +Rules relevant to SharpCompress: + +- If general purpose bit 11 (EFS) is set, file names and comments must be UTF-8. +- If EFS is not set, SharpCompress uses the configured archive encoding. +- If Info-ZIP Unicode path extra field `0x7075` is present and the caller did not force an encoding, SharpCompress uses the Unicode name from that extra field. +- `ZipWriter.NormalizeFilename` converts backslashes to `/`, removes drive prefixes before `:`, and trims leading/trailing `/` for file entries. +- `ZipWriter` ensures directory entries end with `/`. + +## Encryption + +APPNOTE defines traditional PKWARE encryption, strong encryption features, central directory encryption, and method 99 as an AE-x encryption marker. + +SharpCompress behavior: + +- Traditional PKWARE-encrypted ZIP entries can be read when a password is supplied. +- WinZip AES entries use method 99 (`WinzipAes`) and extra field `0x9901`; SharpCompress reads the actual compression method from that extra data. +- `ZipHeaderFactory.LoadHeader` rejects encrypted ZIP data that requires unsupported non-seekable handling. +- Central directory encryption and broad strong encryption records are not general-purpose supported features. Keep this explicit in support claims. + +## Seekable And Streaming Reads + +Seekable read path: + +- `SeekableZipHeaderFactory` searches backward for EOCD. +- If EOCD indicates Zip64, it reads the Zip64 locator and Zip64 EOCD. +- It iterates central directory file headers. +- `GetLocalHeader` seeks to each entry's local header when entry data is needed. + +Streaming read path: + +- `StreamingZipHeaderFactory` reads local headers in archive order. +- Entry data must be consumed or skipped before the next header can be parsed. +- If bit 3 is set, descriptor values are read after entry data. +- Directory entries may be inferred from names ending in `/`, or from zero-size names ending in `\` for older .NET-produced archives. + +Archive API vs Reader API: + +- `ZipArchive` is appropriate for seekable streams and multi-volume/split archives. +- `ZipReader` is forward-only and does not seek across volume files. + +## SharpCompress Support Matrix + +Current ZIP support summary from `docs/FORMATS.md` and implementation: + +| Feature | Read | Write | Notes | +| --- | --- | --- | --- | +| Stored | Yes | Yes | Method 0 | +| Deflate | Yes | Yes | Method 8 | +| Deflate64 | Yes | No | Method 9 | +| BZip2 | Yes | Yes | Method 12 | +| LZMA | Yes | Yes | Method 14 | +| PPMd | Yes | Yes | Method 98 | +| ZStandard | Yes | Yes | Method 93 | +| XZ | Yes | No | Method 95 | +| Shrink | Yes | No | Legacy method 1 | +| Reduce | Yes | No | Legacy methods 2-5 | +| Implode | Yes | No | Legacy method 6 | +| Zip64 | Yes | Yes | Writing requires seekable stream for Zip64 | +| Data descriptors | Yes | Yes | Writer uses them for non-seekable non-Zip64 output | +| Traditional PKWARE encryption | Yes | No | Password required | +| WinZip AES | Yes | No | Method 99 + extra field `0x9901` | +| Split/multi-volume ZIP | Yes | No | Use Archive API | + +## SharpCompress Write Behavior + +`ZipWriter` is forward-only for entry creation but writes a central directory on dispose. It tracks local header offsets and entry sizes as data is written. + +Writer compression mapping in `ZipWriter.ToZipCompressionMethod` currently accepts: + +- `CompressionType.None` +- `CompressionType.Deflate` +- `CompressionType.BZip2` +- `CompressionType.LZMA` +- `CompressionType.PPMd` +- `CompressionType.ZStandard` + +Other compression types throw `InvalidFormatException`, including `CompressionType.Xz`. + +Important writer rules: + +- Local headers are written before payload data. +- Seekable output lets SharpCompress patch CRC and size fields after compression. +- Non-seekable output uses post-data descriptors for non-Zip64 entries. +- Zip64 on non-seekable output is rejected. +- Zero-byte file entries are normalized to stored/no compression in the central directory. +- Central directory entries are written on `Dispose` / `DisposeAsync`; callers must dispose writers to finalize archives. + +## Known Limitations + +Keep these limitations explicit in code comments, docs, and tests: + +- No separate ZIP LZMA2 method support. APPNOTE method 95 is XZ; XZ may use LZMA2 internally. +- No XZ writing for ZIP entries. +- No Deflate64, Shrink, Reduce, or Implode writing. +- No general-purpose central directory encryption support. +- No broad APPNOTE strong encryption record support beyond current traditional PKWARE and WinZip AES read paths. +- No non-seekable Zip64 writing. +- Unknown extra fields are not semantically modeled unless explicitly implemented. +- ZipReader cannot seek across multi-volume/split archive parts; use ZipArchive for split archives. + +## Test Fixtures + +Representative fixtures in `tests/TestArchives/Archives/`: + +- `Zip.none.zip` +- `Zip.deflate.zip` +- `Zip.deflate.dd.zip` +- `Zip.deflate64.zip` +- `Zip.bzip2.zip` +- `Zip.lzma.zip` +- `Zip.lzma.dd.zip` +- `Zip.ppmd.zip` +- `Zip.shrink.zip` +- `Zip.reduce1.zip` +- `Zip.reduce2.zip` +- `Zip.reduce3.zip` +- `Zip.reduce4.zip` +- `Zip.implode.zip` +- `Zip.zip64.zip` +- `Zip.zstd.WinzipAES.mixed.zip` +- `WinZip27_XZ.zipx` +- `WinZip27_ZSTD.zipx` +- `WinZip26.nocomp.multi.zip` +- `WinZip26.nocomp.multi.zipx` +- `Zip.UnicodePathExtra.zip` +- `Zip.EntryComment.zip` + +Representative test files: + +- `tests/SharpCompress.Test/Zip/ZipArchiveTests.cs` +- `tests/SharpCompress.Test/Zip/ZipArchiveAsyncTests.cs` +- `tests/SharpCompress.Test/Zip/ZipReaderTests.cs` +- `tests/SharpCompress.Test/Zip/ZipReaderAsyncTests.cs` +- `tests/SharpCompress.Test/Zip/ZipWriterTests.cs` +- `tests/SharpCompress.Test/Zip/ZipWriterAsyncTests.cs` +- `tests/SharpCompress.Test/Zip/Zip64Tests.cs` +- `tests/SharpCompress.Test/Zip/Zip64AsyncTests.cs` +- `tests/SharpCompress.Test/Zip/Zip64VersionConsistencyTests.cs` +- `tests/SharpCompress.Test/Zip/ZipFilePartTests.cs` diff --git a/.config/dotnet-tools.json b/.config/dotnet-tools.json index ef84a272..9e028ef5 100644 --- a/.config/dotnet-tools.json +++ b/.config/dotnet-tools.json @@ -3,10 +3,11 @@ "isRoot": true, "tools": { "csharpier": { - "version": "0.25.0", + "version": "1.3.0", "commands": [ - "dotnet-csharpier" - ] + "csharpier" + ], + "rollForward": false } } -} \ No newline at end of file +} diff --git a/.editorconfig b/.editorconfig index 1903a97c..e94ae1dd 100644 --- a/.editorconfig +++ b/.editorconfig @@ -70,7 +70,7 @@ indent_style = tab [*.{cs,csx,cake,vb,vbx}] # Default Severity for all .NET Code Style rules below -dotnet_analyzer_diagnostic.severity = warning +dotnet_analyzer_diagnostic.severity = silent ########################################## # File Header (Uncomment to support file headers) @@ -257,59 +257,73 @@ csharp_style_namespace_declarations = file_scoped ########################################## [*.{cs,csx,cake,vb,vbx}] -dotnet_diagnostic.CA1000.severity = suggestion -dotnet_diagnostic.CA1001.severity = error -dotnet_diagnostic.CA1018.severity = error +dotnet_diagnostic.CA1000.severity = error # do not declare static members on generic types +dotnet_diagnostic.CA1001.severity = error # disposable field owners should be disposable +dotnet_diagnostic.CA1018.severity = error # mark custom attributes with AttributeUsage dotnet_diagnostic.CA1036.severity = silent -dotnet_diagnostic.CA1051.severity = suggestion -dotnet_diagnostic.CA1068.severity = error -dotnet_diagnostic.CA1069.severity = error -dotnet_diagnostic.CA1304.severity = error -dotnet_diagnostic.CA1305.severity = suggestion -dotnet_diagnostic.CA1307.severity = suggestion -dotnet_diagnostic.CA1309.severity = suggestion -dotnet_diagnostic.CA1310.severity = error -dotnet_diagnostic.CA1707.severity = suggestion -dotnet_diagnostic.CA1708.severity = suggestion -dotnet_diagnostic.CA1711.severity = suggestion -dotnet_diagnostic.CA1716.severity = suggestion -dotnet_diagnostic.CA1720.severity = suggestion -dotnet_diagnostic.CA1725.severity = suggestion -dotnet_diagnostic.CA1805.severity = suggestion -dotnet_diagnostic.CA1816.severity = suggestion -dotnet_diagnostic.CA1822.severity = suggestion -dotnet_diagnostic.CA1825.severity = error +dotnet_diagnostic.CA1051.severity = suggestion # do not declare visible instance fields +dotnet_diagnostic.CA1068.severity = error # cancellation token parameters must come last +dotnet_diagnostic.CA1069.severity = error # enums should not have duplicate values +dotnet_diagnostic.CA1304.severity = error # specify CultureInfo for culture-sensitive operations +dotnet_diagnostic.CA1305.severity = error # specify IFormatProvider +dotnet_diagnostic.CA1307.severity = error # specify StringComparison for clarity +dotnet_diagnostic.CA1309.severity = error # use ordinal StringComparison +dotnet_diagnostic.CA1310.severity = error # specify StringComparison for correctness +dotnet_diagnostic.CA1507.severity = error # use nameof in place of string literals +dotnet_diagnostic.CA1513.severity = suggestion # use ObjectDisposedException throw helper +dotnet_diagnostic.CA1707.severity = suggestion # identifiers should not contain underscores +dotnet_diagnostic.CA1708.severity = suggestion # identifiers should differ by more than case +dotnet_diagnostic.CA1711.severity = suggestion # identifiers should not have incorrect suffixes +dotnet_diagnostic.CA1716.severity = suggestion # identifiers should not match language keywords +dotnet_diagnostic.CA1720.severity = suggestion # identifiers should not contain type names +dotnet_diagnostic.CA1725.severity = error # parameter names should match base declaration +dotnet_diagnostic.CA1805.severity = suggestion # avoid unnecessary default value initialization +dotnet_diagnostic.CA1816.severity = suggestion # call GC.SuppressFinalize correctly +dotnet_diagnostic.CA1822.severity = suggestion # mark members static when possible +dotnet_diagnostic.CA1825.severity = error # avoid zero-length array allocations dotnet_diagnostic.CA1826.severity = silent -dotnet_diagnostic.CA1827.severity = error -dotnet_diagnostic.CA1829.severity = suggestion -dotnet_diagnostic.CA1834.severity = error -dotnet_diagnostic.CA1845.severity = suggestion -dotnet_diagnostic.CA1848.severity = suggestion -dotnet_diagnostic.CA1852.severity = suggestion -dotnet_diagnostic.CA2016.severity = suggestion -dotnet_diagnostic.CA2201.severity = error -dotnet_diagnostic.CA2206.severity = error -dotnet_diagnostic.CA2208.severity = error -dotnet_diagnostic.CA2211.severity = error -dotnet_diagnostic.CA2249.severity = error -dotnet_diagnostic.CA2251.severity = error +dotnet_diagnostic.CA1827.severity = error # use Any() instead of Count()/LongCount() checks +dotnet_diagnostic.CA1829.severity = error # use Length or Count property instead of LINQ Count() +dotnet_diagnostic.CA1834.severity = error # prefer StringBuilder.Append(char) for single chars +dotnet_diagnostic.CA1845.severity = error # use span-based string.Concat overloads +dotnet_diagnostic.CA1848.severity = error # use LoggerMessage for high-performance logging +dotnet_diagnostic.CA1852.severity = suggestion # seal types that are not intended for inheritance +dotnet_diagnostic.CA1860.severity = silent +dotnet_diagnostic.CA2016.severity = error # forward CancellationToken to invoked methods +dotnet_diagnostic.CA2201.severity = error # do not throw reserved or overly general exceptions +dotnet_diagnostic.CA2206.severity = error # enforce CA2206 usage guidance +dotnet_diagnostic.CA2208.severity = error # instantiate ArgumentException types correctly +dotnet_diagnostic.CA2211.severity = error # non-constant fields should not be visible +dotnet_diagnostic.CA2249.severity = error # prefer string.Contains over string.IndexOf checks +dotnet_diagnostic.CA2251.severity = error # use string.Equals over string.Compare equality checks dotnet_diagnostic.CA2252.severity = none -dotnet_diagnostic.CA2254.severity = suggestion +dotnet_diagnostic.CA2254.severity = error # logging message templates should be static expressions -dotnet_diagnostic.CS0169.severity = error -dotnet_diagnostic.CS0219.severity = error -dotnet_diagnostic.CS0649.severity = suggestion -dotnet_diagnostic.CS1998.severity = error -dotnet_diagnostic.CS8602.severity = error -dotnet_diagnostic.CS8604.severity = error -dotnet_diagnostic.CS8618.severity = error -dotnet_diagnostic.CS0618.severity = error -dotnet_diagnostic.CS1998.severity = error -dotnet_diagnostic.CS4014.severity = error -dotnet_diagnostic.CS8600.severity = error -dotnet_diagnostic.CS8603.severity = error -dotnet_diagnostic.CS8625.severity = error -dotnet_diagnostic.CS8981.severity = suggestion +; High volume analyzers requiring extensive refactoring - set to suggestion temporarily +dotnet_diagnostic.CA1835.severity = suggestion # prefer Memory-based async overloads +dotnet_diagnostic.CA1510.severity = error # use ArgumentNullException.ThrowIfNull +dotnet_diagnostic.CA1512.severity = error # use ArgumentOutOfRangeException throw helpers +dotnet_diagnostic.CA1844.severity = suggestion # provide memory-based async stream overrides +dotnet_diagnostic.CA1825.severity = error # avoid zero-length array allocations +dotnet_diagnostic.CA1712.severity = suggestion # do not prefix enum values with type name +dotnet_diagnostic.CA2022.severity = suggestion # avoid inexact reads with Stream.Read +dotnet_diagnostic.CA1850.severity = error # prefer static HashData over ComputeHash +dotnet_diagnostic.CA2263.severity = error # prefer generic overload when type is known +dotnet_diagnostic.CA2012.severity = error # use ValueTasks correctly +dotnet_diagnostic.CA1001.severity = error # disposable field owners should be disposable + +dotnet_diagnostic.CS0169.severity = error # field is never used +dotnet_diagnostic.CS0219.severity = error # variable assigned but never used +dotnet_diagnostic.CS0649.severity = error # field is never assigned and remains default +dotnet_diagnostic.CS1998.severity = error # async method lacks await operators +dotnet_diagnostic.CS8602.severity = error # possible null reference dereference +dotnet_diagnostic.CS8604.severity = error # possible null reference argument +dotnet_diagnostic.CS8618.severity = error # non-nullable member is uninitialized +dotnet_diagnostic.CS0618.severity = error # obsolete member usage +dotnet_diagnostic.CS4014.severity = error # unawaited task call +dotnet_diagnostic.CS8600.severity = error # possible null to non-nullable conversion +dotnet_diagnostic.CS8603.severity = error # possible null reference return +dotnet_diagnostic.CS8625.severity = error # cannot assign null to non-nullable reference dotnet_diagnostic.BL0005.severity = suggestion @@ -317,9 +331,9 @@ dotnet_diagnostic.MVC1000.severity = suggestion dotnet_diagnostic.RZ10012.severity = error -dotnet_diagnostic.IDE0004.severity = error # redundant cast -dotnet_diagnostic.IDE0005.severity = error -dotnet_diagnostic.IDE0007.severity = error # Use var +dotnet_diagnostic.IDE0004.severity = suggestion # redundant cast +dotnet_diagnostic.IDE0005.severity = suggestion +dotnet_diagnostic.IDE0007.severity = suggestion # Use var dotnet_diagnostic.IDE0011.severity = error # Use braces on if statements dotnet_diagnostic.IDE0010.severity = silent # populate switch dotnet_diagnostic.IDE0017.severity = suggestion # initialization can be simplified @@ -329,15 +343,15 @@ dotnet_diagnostic.IDE0023.severity = suggestion # use expression body for operat dotnet_diagnostic.IDE0024.severity = silent # expression body for operators dotnet_diagnostic.IDE0025.severity = suggestion # use expression body for properties dotnet_diagnostic.IDE0027.severity = suggestion # Use expression body for accessors -dotnet_diagnostic.IDE0028.severity = silent +dotnet_diagnostic.IDE0028.severity = silent # expression body for accessors dotnet_diagnostic.IDE0032.severity = suggestion # Use auto property dotnet_diagnostic.IDE0033.severity = error # prefer tuple name dotnet_diagnostic.IDE0037.severity = suggestion # simplify anonymous type -dotnet_diagnostic.IDE0040.severity = error # modifiers required +dotnet_diagnostic.IDE0040.severity = suggestion # modifiers required dotnet_diagnostic.IDE0041.severity = error # simplify null dotnet_diagnostic.IDE0042.severity = error # deconstruct variable dotnet_diagnostic.IDE0044.severity = suggestion # make field only when possible -dotnet_diagnostic.IDE0047.severity = suggestion # paratemeter name +dotnet_diagnostic.IDE0047.severity = suggestion # parameter name dotnet_diagnostic.IDE0051.severity = error # unused field dotnet_diagnostic.IDE0052.severity = error # unused member dotnet_diagnostic.IDE0053.severity = suggestion # lambda not needed @@ -347,15 +361,76 @@ dotnet_diagnostic.IDE0060.severity = suggestion # unused parameters dotnet_diagnostic.IDE0061.severity = suggestion # local expression body dotnet_diagnostic.IDE0062.severity = suggestion # local to static dotnet_diagnostic.IDE0063.severity = error # simplify using + +[src/**/*.cs] +dotnet_diagnostic.VSTHRD002.severity = error # avoid sync waits on async operations +dotnet_diagnostic.VSTHRD100.severity = error # avoid async void methods +dotnet_diagnostic.VSTHRD101.severity = error # avoid unsupported async delegates +dotnet_diagnostic.VSTHRD102.severity = error # implement internal logic asynchronously +dotnet_diagnostic.VSTHRD103.severity = error # use async methods from async methods +dotnet_diagnostic.VSTHRD104.severity = error # offer async alternatives when possible +dotnet_diagnostic.VSTHRD107.severity = error # await task within using expression +dotnet_diagnostic.VSTHRD110.severity = error # observe result of async calls +dotnet_diagnostic.VSTHRD111.severity = error # use ConfigureAwait(bool) +dotnet_diagnostic.VSTHRD112.severity = error # implement System.IAsyncDisposable +dotnet_diagnostic.VSTHRD113.severity = error # check for System.IAsyncDisposable +dotnet_diagnostic.VSTHRD114.severity = error # avoid returning null from Task methods +dotnet_diagnostic.VSTHRD200.severity = suggestion # use Async suffix naming convention + +[build/**/*.cs] +dotnet_diagnostic.VSTHRD001.severity = none # avoid legacy thread switching methods (disabled for build scripts) +dotnet_diagnostic.VSTHRD002.severity = none # avoid sync waits on async operations (disabled for build scripts) +dotnet_diagnostic.VSTHRD003.severity = none # avoid awaiting foreign tasks (disabled for build scripts) +dotnet_diagnostic.VSTHRD004.severity = none # await SwitchToMainThreadAsync (disabled for build scripts) +dotnet_diagnostic.VSTHRD010.severity = none # invoke single-threaded types on main thread (disabled for build scripts) +dotnet_diagnostic.VSTHRD011.severity = none # use AsyncLazy (disabled for build scripts) +dotnet_diagnostic.VSTHRD012.severity = none # provide JoinableTaskFactory where allowed (disabled for build scripts) +dotnet_diagnostic.VSTHRD100.severity = none # avoid async void methods (disabled for build scripts) +dotnet_diagnostic.VSTHRD101.severity = none # avoid unsupported async delegates (disabled for build scripts) +dotnet_diagnostic.VSTHRD102.severity = none # implement internal logic asynchronously (disabled for build scripts) +dotnet_diagnostic.VSTHRD103.severity = none # use async methods from async methods (disabled for build scripts) +dotnet_diagnostic.VSTHRD104.severity = none # offer async alternatives when possible (disabled for build scripts) +dotnet_diagnostic.VSTHRD105.severity = none # avoid TaskScheduler.Current assumptions (disabled for build scripts) +dotnet_diagnostic.VSTHRD106.severity = none # use InvokeAsync for async events (disabled for build scripts) +dotnet_diagnostic.VSTHRD107.severity = none # await task within using expression (disabled for build scripts) +dotnet_diagnostic.VSTHRD108.severity = none # assert thread affinity unconditionally (disabled for build scripts) +dotnet_diagnostic.VSTHRD109.severity = none # switch instead of assert in async methods (disabled for build scripts) +dotnet_diagnostic.VSTHRD110.severity = none # observe result of async calls (disabled for build scripts) +dotnet_diagnostic.VSTHRD111.severity = none # use ConfigureAwait(bool) (disabled for build scripts) +dotnet_diagnostic.VSTHRD112.severity = none # implement System.IAsyncDisposable (disabled for build scripts) +dotnet_diagnostic.VSTHRD113.severity = none # check for System.IAsyncDisposable (disabled for build scripts) +dotnet_diagnostic.VSTHRD114.severity = none # avoid returning null from Task methods (disabled for build scripts) +dotnet_diagnostic.VSTHRD115.severity = none # avoid explicit null SynchronizationContext in JTC (disabled for build scripts) +dotnet_diagnostic.VSTHRD200.severity = none # use Async suffix naming convention (disabled for build scripts) + +[tests/**/*.cs] +dotnet_diagnostic.CA1861.severity = suggestion # avoid constant arrays as arguments +dotnet_diagnostic.CA1305.severity = suggestion # specify IFormatProvider +dotnet_diagnostic.CA1307.severity = suggestion # specify StringComparison for clarity +dotnet_diagnostic.IDE0042.severity = suggestion +dotnet_diagnostic.IDE0051.severity = suggestion +dotnet_diagnostic.IDE0063.severity = suggestion dotnet_diagnostic.IDE0066.severity = suggestion # switch expression dotnet_diagnostic.IDE0072.severity = suggestion # Populate switch - forces population of all cases even when default specified dotnet_diagnostic.IDE0078.severity = suggestion # use pattern matching dotnet_diagnostic.IDE0090.severity = suggestion # new can be simplified -dotnet_diagnostic.IDE0130.severity = error # namespace folder structure +dotnet_diagnostic.IDE0130.severity = suggestion # namespace folder structure dotnet_diagnostic.IDE0160.severity = silent # Use block namespaces ARE NOT required dotnet_diagnostic.IDE0161.severity = error # Please use file namespaces dotnet_diagnostic.IDE0200.severity = suggestion # lambda not needed dotnet_diagnostic.IDE1006.severity = suggestion # Naming rule violation: These words cannot contain lower case characters +dotnet_diagnostic.IDE0260.severity = suggestion # Use pattern matching +dotnet_diagnostic.IDE0270.severity = suggestion # Null check simplifcation +dotnet_diagnostic.IDE0290.severity = suggestion # Primary Constructor +dotnet_diagnostic.IDE0300.severity = suggestion # Collection +dotnet_diagnostic.IDE0305.severity = suggestion # Collection ToList + +dotnet_diagnostic.NX0001.severity = error +dotnet_diagnostic.NX0002.severity = silent +dotnet_diagnostic.NX0003.severity = silent + +dotnet_diagnostic.VSTHRD110.severity = error # observe result of async calls +dotnet_diagnostic.VSTHRD107.severity = error # await task within using expression ########################################## # Styles diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 100644 index 00000000..96094115 --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1,27 @@ + +# RTK — Token-Optimized CLI + +**rtk** is a CLI proxy that filters and compresses command outputs, saving 60-90% tokens. + +## Rule + +Always prefix shell commands with `rtk`: + +```bash +# Instead of: Use: +git status rtk git status +git log -10 rtk git log -10 +cargo test rtk cargo test +docker ps rtk docker ps +kubectl get pods rtk kubectl pods +``` + +## Meta commands (use directly) + +```bash +rtk gain # Token savings dashboard +rtk gain --history # Per-command savings history +rtk discover # Find missed rtk opportunities +rtk proxy # Run raw (no filtering) but track usage +``` + diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 67534058..c6695bb2 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -1,6 +1,19 @@ version: 2 + updates: - - package-ecosystem: "github-actions" # search for actions - there are other options available - directory: "/" # search in .github/workflows under root `/` + - package-ecosystem: "nuget" + directory: "/" schedule: - interval: "weekly" # check for action update every week + interval: "weekly" + + open-pull-requests-limit: 2 + + groups: + all-dependencies: + patterns: + - "*" + + ignore: + - dependency-name: "*" + update-types: + - "version-update:semver-major" diff --git a/.github/hooks/rtk-rewrite.json b/.github/hooks/rtk-rewrite.json new file mode 100644 index 00000000..eb2a5a7f --- /dev/null +++ b/.github/hooks/rtk-rewrite.json @@ -0,0 +1,12 @@ +{ + "hooks": { + "PreToolUse": [ + { + "type": "command", + "command": "rtk hook copilot", + "cwd": ".", + "timeout": 5 + } + ] + } +} diff --git a/.github/workflows/NUGET_RELEASE.md b/.github/workflows/NUGET_RELEASE.md new file mode 100644 index 00000000..42375089 --- /dev/null +++ b/.github/workflows/NUGET_RELEASE.md @@ -0,0 +1,155 @@ +# NuGet Release Workflow + +This document describes the automated NuGet release workflow for SharpCompress. + +## Overview + +The `nuget-release.yml` workflow automatically builds, tests, and publishes SharpCompress packages to NuGet.org when: +- Changes are pushed to the `master` or `release` branch +- A version tag (format: `MAJOR.MINOR.PATCH`) is pushed + +The workflow runs on both Windows and Ubuntu, but only the Windows build publishes to NuGet. + +## How It Works + +### Version Determination + +The workflow automatically determines the version based on whether the commit is tagged using C# code in the build project: + +1. **Tagged Release (Stable)**: + - If the current commit has a version tag (e.g., `0.42.1`) + - Uses the tag as the version number + - Published as a stable release + +2. **Untagged Release (Prerelease)**: + - If the current commit is NOT tagged + - Creates a prerelease version based on the next minor version + - Format: `{NEXT_MINOR_VERSION}-beta.{COMMIT_COUNT}` + - Example: `0.43.0-beta.123` (if last tag is 0.42.x) + - Published as a prerelease to NuGet.org (Windows build only) + +### Workflow Steps + +The workflow runs on a matrix of operating systems (Windows and Ubuntu): + +1. **Checkout**: Fetches the repository with full history for version detection +2. **Setup .NET**: Installs .NET 10.0 +3. **Determine Version**: Runs `determine-version` build target to check for tags and determine version +4. **Update Version**: Runs `update-version` build target to update the version in the project file +5. **Build and Test**: Runs the full build and test suite on both platforms +6. **Upload Artifacts**: Uploads the generated `.nupkg` files as workflow artifacts (separate for each OS) +7. **Push to NuGet**: (Windows only) Runs `push-to-nuget` build target to publish the package to NuGet.org using the API key + +All version detection, file updates, and publishing logic is implemented in C# in the `build/Program.cs` file using build targets. + +## Setup Requirements + +### 1. NuGet API Key Secret + +The workflow requires a `NUGET_API_KEY` secret to be configured in the repository settings: + +1. Go to https://www.nuget.org/account/apikeys +2. Create a new API key with "Push" permission for the SharpCompress package +3. In GitHub, go to: **Settings** → **Secrets and variables** → **Actions** +4. Create a new secret named `NUGET_API_KEY` with the API key value + +### 2. Branch Protection (Recommended) + +Consider enabling branch protection rules for the `release` branch to ensure: +- Code reviews are required before merging +- Status checks pass before merging +- Only authorized users can push to the branch + +## Usage + +### Creating a Stable Release + +There are two ways to trigger a stable release: + +**Method 1: Push tag to trigger workflow** +1. Ensure all changes are committed on the `master` or `release` branch +2. Create and push a version tag: + ```bash + git checkout master # or release + git tag 0.43.0 + git push origin 0.43.0 + ``` +3. The workflow will automatically trigger, build, test, and publish `SharpCompress 0.43.0` to NuGet.org (Windows build) + +**Method 2: Tag after pushing to branch** +1. Ensure all changes are merged and pushed to the `master` or `release` branch +2. Create and push a version tag on the already-pushed commit: + ```bash + git checkout master # or release + git tag 0.43.0 + git push origin 0.43.0 + ``` +3. The workflow will automatically trigger, build, test, and publish `SharpCompress 0.43.0` to NuGet.org (Windows build) + +### Creating a Prerelease + +1. Push changes to the `master` or `release` branch without tagging: + ```bash + git checkout master # or release + git push origin master # or release + ``` +2. The workflow will automatically: + - Build and test the project on both Windows and Ubuntu + - Publish a prerelease version like `0.43.0-beta.456` to NuGet.org (Windows build) + +## Troubleshooting + +### Workflow Fails to Push to NuGet + +- **Check the API Key**: Ensure `NUGET_API_KEY` is set correctly in repository secrets +- **Check API Key Permissions**: Verify the API key has "Push" permission for SharpCompress +- **Check API Key Expiration**: NuGet API keys may expire; create a new one if needed + +### Version Conflict + +If you see "Package already exists" errors: +- The workflow uses `--skip-duplicate` flag to handle this gracefully +- If you need to republish the same version, delete it from NuGet.org first (if allowed) + +### Build or Test Failures + +- The workflow will not push to NuGet if build or tests fail +- Check the workflow logs in GitHub Actions for details +- Fix the issues and push again + +## Manual Package Creation + +If you need to create a package manually without publishing: + +```bash +dotnet run --project build/build.csproj -- publish +``` + +The package will be created in the `artifacts/` directory. + +## Build Targets + +The workflow uses the following C# build targets defined in `build/Program.cs`: + +- **determine-version**: Detects version from git tags and outputs VERSION and PRERELEASE variables +- **update-version**: Updates VersionPrefix, AssemblyVersion, and FileVersion in the project file +- **push-to-nuget**: Pushes the generated NuGet packages to NuGet.org (requires NUGET_API_KEY) + +These targets can be run manually for testing: + +```bash +# Determine the version +dotnet run --project build/build.csproj -- determine-version + +# Update version in project file +VERSION=0.43.0 dotnet run --project build/build.csproj -- update-version + +# Push to NuGet (requires NUGET_API_KEY environment variable) +NUGET_API_KEY=your-key dotnet run --project build/build.csproj -- push-to-nuget +``` + +## Related Files + +- `.github/workflows/nuget-release.yml` - The workflow definition +- `build/Program.cs` - Build script with version detection and publishing logic +- `src/SharpCompress/SharpCompress.csproj` - Project file with version information diff --git a/.github/workflows/TESTING.md b/.github/workflows/TESTING.md new file mode 100644 index 00000000..6afaa8aa --- /dev/null +++ b/.github/workflows/TESTING.md @@ -0,0 +1,120 @@ +# Testing Guide for NuGet Release Workflow + +This document describes how to test the NuGet release workflow. + +## Testing Strategy + +Since this workflow publishes to NuGet.org and requires repository secrets, testing should be done carefully. The workflow runs on both Windows and Ubuntu, but only the Windows build publishes to NuGet. + +## Pre-Testing Checklist + +- [x] Workflow YAML syntax validated +- [x] Version determination logic tested locally +- [x] Version update logic tested locally +- [x] Build script works (`dotnet run --project build/build.csproj`) + +## Manual Testing Steps + +### 1. Test Prerelease Publishing (Recommended First Test) + +This tests the workflow on untagged commits to the master or release branch. + +**Steps:** +1. Ensure `NUGET_API_KEY` secret is configured in repository settings +2. Create a test commit on the `master` or `release` branch (e.g., update a comment or README) +3. Push to the `master` or `release` branch +4. Monitor the GitHub Actions workflow at: https://github.com/adamhathcock/sharpcompress/actions +5. Verify: + - Workflow triggers and runs successfully on both Windows and Ubuntu + - Version is determined correctly (e.g., `0.43.0-beta.XXX` if last tag is 0.42.x) + - Build and tests pass on both platforms + - Package artifacts are uploaded for both platforms + - Package is pushed to NuGet.org as prerelease (Windows build only) + +**Expected Outcome:** +- A new prerelease package appears on NuGet.org: https://www.nuget.org/packages/SharpCompress/ +- Package version follows pattern: `{NEXT_MINOR_VERSION}-beta.{COMMIT_COUNT}` + +### 2. Test Tagged Release Publishing + +This tests the workflow when a version tag is pushed. + +**Steps:** +1. Prepare the `master` or `release` branch with all desired changes +2. Create a version tag (must be a pure semantic version like `MAJOR.MINOR.PATCH`): + ```bash + git checkout master # or release + git tag 0.42.2 + git push origin 0.42.2 + ``` +3. Monitor the GitHub Actions workflow +4. Verify: + - Workflow triggers and runs successfully on both Windows and Ubuntu + - Version is determined as the tag (e.g., `0.42.2`) + - Build and tests pass on both platforms + - Package artifacts are uploaded for both platforms + - Package is pushed to NuGet.org as stable release (Windows build only) + +**Expected Outcome:** +- A new stable release package appears on NuGet.org +- Package version matches the tag + +### 3. Test Duplicate Package Handling + +This tests the `--skip-duplicate` flag behavior. + +**Steps:** +1. Push to the `release` branch without making changes +2. Monitor the workflow +3. Verify: + - Workflow runs but NuGet push is skipped with "duplicate" message + - No errors occur + +### 4. Test Build Failure Handling + +This tests that failed builds don't publish packages. + +**Steps:** +1. Introduce a breaking change in a test or code +2. Push to the `release` branch +3. Verify: + - Workflow runs and detects the failure + - Build or test step fails + - NuGet push step is skipped + - No package is published + +## Verification + +After each test, verify: + +1. **GitHub Actions Logs**: Check the workflow logs for any errors or warnings +2. **NuGet.org**: Verify the package appears with correct version and metadata +3. **Artifacts**: Download and inspect the uploaded artifacts + +## Rollback/Cleanup + +If testing produces unwanted packages: + +1. **Prerelease packages**: Can be unlisted on NuGet.org (Settings → Unlist) +2. **Stable packages**: Cannot be deleted, only unlisted (use test versions) +3. **Tags**: Can be deleted with: + ```bash + git tag -d 0.42.2 + git push origin :refs/tags/0.42.2 + ``` + +## Known Limitations + +- NuGet.org does not allow re-uploading the same version +- Deleted packages on NuGet.org reserve the version number +- The workflow requires the `NUGET_API_KEY` secret to be set + +## Success Criteria + +The workflow is considered successful if: + +- ✅ Prerelease versions are published correctly with beta suffix +- ✅ Tagged versions are published as stable releases +- ✅ Build and test failures prevent publishing +- ✅ Duplicate packages are handled gracefully +- ✅ Workflow logs are clear and informative diff --git a/.github/workflows/dotnetcore.yml b/.github/workflows/dotnetcore.yml deleted file mode 100644 index 4486b23f..00000000 --- a/.github/workflows/dotnetcore.yml +++ /dev/null @@ -1,32 +0,0 @@ -name: SharpCompress -on: - push: - branches: - - 'master' - pull_request: - types: [ opened, synchronize, reopened, ready_for_review ] - -jobs: - build: - runs-on: ${{ matrix.os }} - strategy: - matrix: - os: [windows-latest, ubuntu-latest] - - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-dotnet@v3 - with: - dotnet-version: 7.0.x - - name: NuGet Caching - uses: actions/cache@v3 - with: - path: ~/.nuget/packages - key: ${{ runner.os }}-nuget-${{ hashFiles('packages.lock.json', '*/packages.lock.json') }} - restore-keys: | - ${{ runner.os }}-nuget- - - run: dotnet run --project build/build.csproj - - uses: actions/upload-artifact@v3 - with: - name: ${{ matrix.os }}-sharpcompress.nupkg - path: artifacts/* diff --git a/.github/workflows/nuget-release.yml b/.github/workflows/nuget-release.yml new file mode 100644 index 00000000..345b82c4 --- /dev/null +++ b/.github/workflows/nuget-release.yml @@ -0,0 +1,67 @@ +name: NuGet Release + +on: + push: + branches: + - 'master' + - 'release' + tags: + - '[0-9]+.[0-9]+.[0-9]+' + pull_request: + branches: + - 'master' + - 'release' + +permissions: + contents: read + +jobs: + build-and-publish: + runs-on: ${{ matrix.os }} + strategy: + matrix: + os: [windows-latest, ubuntu-latest] + + steps: + - uses: actions/checkout@v6 + with: + fetch-depth: 0 # Fetch all history for versioning + + - uses: actions/setup-dotnet@v5 + with: + global-json-file: global.json + + # Determine version using C# build target + - name: Determine Version + id: version + run: dotnet run --project build/build.csproj -- determine-version + + # Update version in project file using C# build target + - name: Update Version in Project + run: dotnet run --project build/build.csproj -- update-version + env: + VERSION: ${{ steps.version.outputs.version }} + + # Build and test + - name: Build and Test + run: dotnet run --project build/build.csproj + + - name: Validate AOT Smoke Test + if: matrix.os == 'ubuntu-latest' + run: | + dotnet publish tests/SharpCompress.AotSmoke/SharpCompress.AotSmoke.csproj --configuration Release --runtime linux-x64 --self-contained true --output artifacts/aot-smoke + ./artifacts/aot-smoke/SharpCompress.AotSmoke + + # Upload artifacts for verification + - name: Upload NuGet Package + uses: actions/upload-artifact@v7 + with: + name: ${{ matrix.os }}-nuget-package + path: artifacts/*.nupkg + + # Push to NuGet.org only for version tag pushes (Windows only) + - name: Push to NuGet + if: success() && matrix.os == 'windows-latest' && startsWith(github.ref, 'refs/tags/') + run: dotnet run --project build/build.csproj -- push-to-nuget + env: + NUGET_API_KEY: ${{ secrets.NUGET_API_KEY }} diff --git a/.github/workflows/performance-benchmarks.yml b/.github/workflows/performance-benchmarks.yml new file mode 100644 index 00000000..3adf01aa --- /dev/null +++ b/.github/workflows/performance-benchmarks.yml @@ -0,0 +1,50 @@ +name: Performance Benchmarks + +on: + push: + branches: + - 'master' + - 'release' + pull_request: + branches: + - 'master' + - 'release' + workflow_dispatch: + +permissions: + contents: read + +jobs: + benchmark: + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v6 + with: + fetch-depth: 0 + + - uses: actions/setup-dotnet@v5 + with: + global-json-file: global.json + + - name: Build Performance Project + run: dotnet build tests/SharpCompress.Performance/SharpCompress.Performance.csproj --configuration Release + + - name: Run Benchmarks + run: dotnet run --project tests/SharpCompress.Performance/SharpCompress.Performance.csproj --configuration Release --no-build -- --filter "*" --exporters json markdown --artifacts benchmark-results + continue-on-error: true + + - name: Display Benchmark Results + if: always() + run: dotnet run --project build/build.csproj -- display-benchmark-results + + - name: Compare with Baseline + if: always() + run: dotnet run --project build/build.csproj -- compare-benchmark-results + + - name: Upload Benchmark Results + if: always() + uses: actions/upload-artifact@v7 + with: + name: benchmark-results + path: benchmark-results/ diff --git a/.gitignore b/.gitignore index d6a5a1d1..6a1f0613 100644 --- a/.gitignore +++ b/.gitignore @@ -4,18 +4,24 @@ _ReSharper.SharpCompress/ bin/ *.suo *.user -TestArchives/Scratch/ -TestArchives/Scratch2/ +tests/TestArchives/Scratch/ +tests/TestArchives/Scratch2/ TestResults/ *.nupkg packages/*/ project.lock.json tests/TestArchives/Scratch +tests/TestArchives/*/Scratch +tests/TestArchives/*/Scratch2 .vs tools -.vscode .idea/ +artifacts/ +BenchmarkDotNet.Artifacts/ +baseline-artifacts/ +profiler-snapshots/ .DS_Store *.snupkg -/tests/TestArchives/6d23a38c-f064-4ef1-ad89-b942396f53b9/Scratch +benchmark-results/ +.opencode/ diff --git a/.vscode/extensions.json b/.vscode/extensions.json new file mode 100644 index 00000000..0c9d8a37 --- /dev/null +++ b/.vscode/extensions.json @@ -0,0 +1,9 @@ +{ + "recommendations": [ + "ms-dotnettools.csdevkit", + "ms-dotnettools.csharp", + "ms-dotnettools.vscode-dotnet-runtime", + "csharpier.csharpier-vscode", + "formulahendry.dotnet-test-explorer" + ] +} diff --git a/.vscode/launch.json b/.vscode/launch.json new file mode 100644 index 00000000..8171a42b --- /dev/null +++ b/.vscode/launch.json @@ -0,0 +1,97 @@ +{ + "version": "0.2.0", + "configurations": [ + { + "name": "Debug Tests (net10.0)", + "type": "coreclr", + "request": "launch", + "preLaunchTask": "build", + "program": "dotnet", + "args": [ + "test", + "${workspaceFolder}/tests/SharpCompress.Test/SharpCompress.Test.csproj", + "-f", + "net10.0", + "--no-build", + "--verbosity=normal" + ], + "cwd": "${workspaceFolder}", + "console": "internalConsole", + "stopAtEntry": false + }, + { + "name": "Debug Specific Test (net10.0)", + "type": "coreclr", + "request": "launch", + "preLaunchTask": "build", + "program": "dotnet", + "args": [ + "test", + "${workspaceFolder}/tests/SharpCompress.Test/SharpCompress.Test.csproj", + "-f", + "net10.0", + "--no-build", + "--filter", + "FullyQualifiedName~${input:testName}" + ], + "cwd": "${workspaceFolder}", + "console": "internalConsole", + "stopAtEntry": false + }, + { + "name": "Debug Performance Tests", + "type": "coreclr", + "request": "launch", + "preLaunchTask": "build", + "program": "dotnet", + "args": [ + "run", + "--project", + "${workspaceFolder}/tests/SharpCompress.Performance/SharpCompress.Performance.csproj", + "--no-build" + ], + "cwd": "${workspaceFolder}", + "console": "internalConsole", + "stopAtEntry": false + }, + { + "name": "Debug Build Script", + "type": "coreclr", + "request": "launch", + "program": "dotnet", + "args": [ + "run", + "--project", + "${workspaceFolder}/build/build.csproj", + "--", + "${input:buildTarget}" + ], + "cwd": "${workspaceFolder}", + "console": "internalConsole", + "stopAtEntry": false + } + ], + "inputs": [ + { + "id": "testName", + "type": "promptString", + "description": "Enter test name or pattern (e.g., TestMethodName or ClassName)", + "default": "" + }, + { + "id": "buildTarget", + "type": "pickString", + "description": "Select build target", + "options": [ + "clean", + "restore", + "build", + "test", + "format", + "publish", + "default" + ], + "default": "build" + } + ] +} diff --git a/.vscode/settings.json b/.vscode/settings.json new file mode 100644 index 00000000..96bc4d85 --- /dev/null +++ b/.vscode/settings.json @@ -0,0 +1,32 @@ +{ + "dotnet.defaultSolution": "SharpCompress.slnx", + "files.exclude": { + "**/bin": true, + "**/obj": true + }, + "files.watcherExclude": { + "**/bin/**": true, + "**/obj/**": true, + "**/artifacts/**": true + }, + "search.exclude": { + "**/bin": true, + "**/obj": true, + "**/artifacts": true + }, + "editor.formatOnSave": false, + "[csharp]": { + "editor.defaultFormatter": "csharpier.csharpier-vscode", + "editor.formatOnSave": true, + "editor.codeActionsOnSave": { + "source.fixAll": "explicit" + } + }, + "csharpier.enableDebugLogs": false, + "omnisharp.enableRoslynAnalyzers": true, + "omnisharp.enableEditorConfigSupport": true, + "dotnet-test-explorer.testProjectPath": "tests/**/*.csproj", + "chat.tools.terminal.autoApprove": { + "dotnet csharpier": true + } +} diff --git a/.vscode/tasks.json b/.vscode/tasks.json new file mode 100644 index 00000000..c17d4d04 --- /dev/null +++ b/.vscode/tasks.json @@ -0,0 +1,178 @@ +{ + "version": "2.0.0", + "tasks": [ + { + "label": "build", + "command": "dotnet", + "type": "process", + "args": [ + "build", + "${workspaceFolder}/SharpCompress.slnx", + "/property:GenerateFullPaths=true", + "/consoleloggerparameters:NoSummary;ForceNoAlign" + ], + "problemMatcher": "$msCompile", + "group": { + "kind": "build", + "isDefault": true + } + }, + { + "label": "build-release", + "command": "dotnet", + "type": "process", + "args": [ + "build", + "${workspaceFolder}/SharpCompress.slnx", + "-c", + "Release", + "/property:GenerateFullPaths=true", + "/consoleloggerparameters:NoSummary;ForceNoAlign" + ], + "problemMatcher": "$msCompile", + "group": "build" + }, + { + "label": "build-library", + "command": "dotnet", + "type": "process", + "args": [ + "build", + "${workspaceFolder}/src/SharpCompress/SharpCompress.csproj", + "/property:GenerateFullPaths=true", + "/consoleloggerparameters:NoSummary;ForceNoAlign" + ], + "problemMatcher": "$msCompile", + "group": "build" + }, + { + "label": "restore", + "command": "dotnet", + "type": "process", + "args": [ + "restore", + "${workspaceFolder}/SharpCompress.slnx" + ], + "problemMatcher": "$msCompile" + }, + { + "label": "clean", + "command": "dotnet", + "type": "process", + "args": [ + "clean", + "${workspaceFolder}/SharpCompress.slnx" + ], + "problemMatcher": "$msCompile" + }, + { + "label": "test", + "command": "dotnet", + "type": "process", + "args": [ + "test", + "${workspaceFolder}/tests/SharpCompress.Test/SharpCompress.Test.csproj", + "--no-build", + "--verbosity=normal" + ], + "problemMatcher": "$msCompile", + "group": { + "kind": "test", + "isDefault": true + }, + "dependsOn": "build" + }, + { + "label": "test-net10", + "command": "dotnet", + "type": "process", + "args": [ + "test", + "${workspaceFolder}/tests/SharpCompress.Test/SharpCompress.Test.csproj", + "-f", + "net10.0", + "--no-build", + "--verbosity=normal" + ], + "problemMatcher": "$msCompile", + "group": "test", + "dependsOn": "build" + }, + { + "label": "test-net48", + "command": "dotnet", + "type": "process", + "args": [ + "test", + "${workspaceFolder}/tests/SharpCompress.Test/SharpCompress.Test.csproj", + "-f", + "net48", + "--no-build", + "--verbosity=normal" + ], + "problemMatcher": "$msCompile", + "group": "test", + "dependsOn": "build" + }, + { + "label": "format", + "command": "dotnet", + "type": "process", + "args": [ + "csharpier", + "." + ], + "problemMatcher": [] + }, + { + "label": "format-check", + "command": "dotnet", + "type": "process", + "args": [ + "csharpier", + "check", + "." + ], + "problemMatcher": [] + }, + { + "label": "run-build-script", + "command": "dotnet", + "type": "process", + "args": [ + "run", + "--project", + "${workspaceFolder}/build/build.csproj" + ], + "problemMatcher": "$msCompile" + }, + { + "label": "pack", + "command": "dotnet", + "type": "process", + "args": [ + "pack", + "${workspaceFolder}/src/SharpCompress/SharpCompress.csproj", + "-c", + "Release", + "-o", + "${workspaceFolder}/artifacts/" + ], + "problemMatcher": "$msCompile", + "dependsOn": "build-release" + }, + { + "label": "performance-tests", + "command": "dotnet", + "type": "process", + "args": [ + "run", + "--project", + "${workspaceFolder}/tests/SharpCompress.Performance/SharpCompress.Performance.csproj", + "-c", + "Release" + ], + "problemMatcher": "$msCompile" + } + ] +} diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..eaa3e326 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,231 @@ +--- +description: 'Guidelines for building SharpCompress - A C# compression library' +applyTo: '**/*.cs' +--- + +# SharpCompress Development + +## About SharpCompress +SharpCompress is a pure C# compression library supporting multiple archive formats (Zip, Tar, GZip, BZip2, 7Zip, Rar, LZip, XZ, ZStandard, Arc, Arj, Ace, LZW). The project currently targets .NET Framework 4.8, .NET Standard 2.0/2.1, .NET 6.0, .NET 8.0, and .NET 10.0. The library provides both seekable Archive APIs and forward-only Reader/Writer APIs for streaming scenarios. + +## C# Instructions +- Use language features supported by the current project toolchain (`LangVersion=latest`) and existing codebase patterns. +- Add comments for non-obvious logic and important design decisions; avoid redundant comments. +- Follow the existing code style and patterns in the codebase. + +## General Instructions +- **Do not commit or stage changes unless the user explicitly asks for it.** +- Make only high confidence suggestions when reviewing code changes. +- Write code with good maintainability practices, including comments on why certain design decisions were made. +- Handle edge cases and write clear exception handling. +- For libraries or external dependencies, mention their usage and purpose in comments. +- Preserve backward compatibility when making changes to public APIs. + +### Workspace Hygiene +- Do not edit generated or machine-local files unless required for the task (for example: `bin/`, `obj/`, `*.csproj.user`). +- Avoid broad formatting-only diffs in unrelated files. + +## Naming Conventions + +- Follow PascalCase for component names, method names, and public members. +- Use camelCase for private fields and local variables. +- Prefix interface names with "I" (e.g., IUserService). + +## Code Formatting + +**Copilot agents: You MUST run the `format` task after making code changes to ensure consistency.** + +- Use CSharpier for code formatting to ensure consistent style across the project +- CSharpier is configured as a local tool in `.config/dotnet-tools.json` + +### Commands + +1. **Restore tools** (first time only): + ```bash + dotnet tool restore + ``` + +2. **Check if files are formatted correctly** (doesn't modify files): + ```bash + dotnet csharpier check . + ``` + - Exit code 0: All files are properly formatted + - Exit code 1: Some files need formatting (will show which files and differences) + +3. **Format files** (modifies files): + ```bash + dotnet csharpier format . + ``` + - Formats all files in the project to match CSharpier style + - Run from project root directory + +4. **Configure your IDE** to format on save using CSharpier for the best experience + +### Additional Notes +- The project also uses `.editorconfig` for editor settings (indentation, encoding, etc.) +- Let CSharpier handle code style while `.editorconfig` handles editor behavior +- Always run `dotnet csharpier check .` before committing to verify formatting + +## Project Setup and Structure + +- The project targets multiple frameworks: .NET Framework 4.8, .NET Standard 2.0/2.1, .NET 6.0, .NET 8.0, and .NET 10.0 +- Main library is in `src/SharpCompress/` +- Tests are in `tests/SharpCompress.Test/` +- Performance tests are in `tests/SharpCompress.Performance/` +- Test archives are in `tests/TestArchives/` +- Build project is in `build/` +- Use `dotnet build` to build the solution +- Use `dotnet test` to run tests +- Solution file: `SharpCompress.slnx` + +### Directory Structure +``` +src/SharpCompress/ + ├── Archives/ # IArchive implementations (Zip, Tar, Rar, 7Zip, GZip) + ├── Readers/ # IReader implementations (forward-only) + ├── Writers/ # IWriter implementations (forward-only) + ├── Compressors/ # Low-level compression streams (BZip2, Deflate, LZMA, etc.) + ├── Factories/ # Format detection and factory pattern + ├── Common/ # Shared types (ArchiveType, Entry, Options) + ├── Crypto/ # Encryption implementations + └── IO/ # Stream utilities and wrappers + +tests/SharpCompress.Test/ + ├── Zip/, Tar/, Rar/, SevenZip/, GZip/, BZip2/ # Format-specific tests + ├── TestBase.cs # Base test class with helper methods + +tests/ + ├── SharpCompress.Test/ # Unit/integration tests + ├── SharpCompress.Performance/ # Benchmark tests + └── TestArchives/ # Test data archives +``` + +### Factory Pattern +Factory implementations can implement one or more interfaces (`IArchiveFactory`, `IReaderFactory`, `IWriterFactory`) depending on format capabilities: +- `ArchiveFactory.OpenArchive()` - Opens archive API objects from seekable streams/files +- `ArchiveFactory.OpenAsyncArchive()` - Opens async archive API objects for async archive use cases +- `ReaderFactory.OpenReader()` - Auto-detects and opens forward-only readers +- `ReaderFactory.OpenAsyncReader()` - Auto-detects and opens forward-only async readers +- `WriterFactory.OpenWriter()` - Creates a writer for a specified `ArchiveType` +- `WriterFactory.OpenAsyncWriter()` - Creates an async writer for async write scenarios +- Factories located in: `src/SharpCompress/Factories/` + +## Nullable Reference Types + +- Declare variables non-nullable, and check for `null` at entry points. +- Always use `is null` or `is not null` instead of `== null` or `!= null`. +- Trust the C# null annotations and don't add null checks when the type system says a value cannot be null. + +## SharpCompress-Specific Guidelines + +### Supported Formats +SharpCompress supports multiple archive and compression formats: +- **Archive Formats**: Zip, Tar, 7Zip, Rar (read-only), Ace (read-only), Arc (read-only), Arj (read-only), LZW (read-only) +- **Compression**: DEFLATE, BZip2, LZMA/LZMA2, PPMd, ZStandard, LZip, XZ (decompress only), Deflate64 (decompress only), legacy Zip/Arc/Arj/Ace methods (read-only as applicable) +- **Combined Formats**: Tar.GZip, Tar.BZip2, Tar.LZip, Tar.XZ (decompress only), Tar.ZStandard (decompress only), Tar.LZW (decompress only) +- **ZIP ZStandard**: ZIP supports ZStandard reading and writing; Tar.ZStandard is decompress-only. +- See [docs/FORMATS.md](docs/FORMATS.md) for complete format support matrix + +### Stream Handling Rules +- **Disposal semantics**: The default `ReaderOptions.LeaveStreamOpen` value is `false`, but effective stream ownership depends on which API overload you call + - File-based overloads (e.g., `OpenArchive(string filePath)`) open the file internally and own that stream, so it is closed by default with the archive/reader + - Do **not** rely on a specific `ReaderOptions` preset being used internally; some implementations may use `ReaderOptions.ForFilePath`, while others may use default `ReaderOptions` with the same ownership semantics + - Several high-level overloads that accept a caller-provided `Stream` use external-stream semantics by default (for example, `ReaderFactory.OpenReader(Stream)` / `ArchiveFactory.OpenArchive(Stream)`), so the caller's stream is typically left open unless you opt into different ownership behavior + - Do **not** assume every stream-based overload behaves identically; some APIs require you to pass stream ownership options explicitly +- **For caller-provided streams**: When the overload accepts `ReaderOptions`, pass `ReaderOptions.ForExternalStream` or use `ReaderOptions` with `LeaveStreamOpen = true` whenever the caller must retain ownership of the stream + - Example: `var options = new ReaderOptions { LeaveStreamOpen = true };` + - Or: `var options = ReaderOptions.ForExternalStream;` +- **For file paths**: SharpCompress manages the stream lifecycle for the internally opened file stream; no manual disposal is needed beyond the archive/reader itself +- Use `NonDisposingStream` wrapper when working with compression streams directly to prevent disposal +- Always dispose of readers, writers, and archives in `using` / `await using` blocks +- For forward-only operations, use Reader/Writer APIs; for random access, use Archive APIs + +### Async/Await Patterns +- All I/O operations support async/await with `CancellationToken` +- Async methods follow the naming convention: `MethodNameAsync` +- For async archive scenarios, prefer `ArchiveFactory.OpenAsyncArchive(...)` over sync `OpenArchive(...)`. +- For async forward-only read scenarios, prefer `ReaderFactory.OpenAsyncReader(...)` over sync `OpenReader(...)`. +- For async write scenarios, prefer `WriterFactory.OpenAsyncWriter(...)` over sync `OpenWriter(...)`. +- Key async methods: + - `WriteEntryToAsync` - Extract entry asynchronously + - `WriteAllToDirectoryAsync` - Extract all entries asynchronously + - `WriteAsync` - Write entry asynchronously + - `WriteAllAsync` - Write directory asynchronously + - `OpenEntryStreamAsync` - Open entry stream asynchronously +- Always provide `CancellationToken` parameter in async methods + +### Archive APIs vs Reader/Writer APIs +- **Archive API**: Use for random access with seekable streams (e.g., `ZipArchive`, `TarArchive`) +- **Reader API**: Use for forward-only reading on non-seekable streams (e.g., `ZipReader`, `TarReader`) +- **Writer API**: Use for forward-only writing on streams (e.g., `ZipWriter`, `TarWriter`) +- 7Zip only supports Archive API due to format limitations + +### Tar-Specific Considerations +- Tar format requires file size in the header +- If no size is specified to TarWriter and the stream is not seekable, an exception will be thrown +- Tar combined with compression is supported for reading with GZip, BZip2, LZip, XZ, ZStandard, and LZW +- Tar writing supports uncompressed Tar, GZip, BZip2, and LZip wrappers + +### Zip-Specific Considerations +- Supports Zip64 for large files (seekable streams only) +- Supports PKWare and WinZip AES encryption +- Multiple compression methods: None, Shrink, Reduce, Implode, DEFLATE, Deflate64, BZip2, LZMA, PPMd +- Encrypted LZMA is not supported + +### Performance Considerations +- For large files, use Reader/Writer APIs with non-seekable streams to avoid loading entire file in memory +- Leverage async I/O for better scalability +- Consider compression level trade-offs (speed vs. size) +- Use appropriate buffer sizes for stream operations + +## Testing + +- Always include test cases for critical paths of the application. +- Test with multiple archive formats when making changes to core functionality. +- Include tests for both Archive and Reader/Writer APIs when applicable. +- Test async operations with cancellation tokens. +- Do not emit "Act", "Arrange" or "Assert" comments. +- Copy existing style in nearby files for test method names and capitalization. +- Use test archives from `tests/TestArchives` directory for consistency. +- Test stream disposal and `LeaveStreamOpen` behavior. +- Test edge cases: empty archives, large files, corrupted archives, encrypted archives. + +### Validation Expectations +- Run targeted tests for the changed area first. +- On non-Windows machines, avoid net48 test runs unless Mono is installed; use framework-specific validation such as `--framework net10.0` instead. +- Run `dotnet csharpier format .` after code edits. +- Run `dotnet csharpier check .` before handing off changes. + +### Test Organization +- Base class: `TestBase` - Provides `TEST_ARCHIVES_PATH`, `SCRATCH_FILES_PATH`, temp directory management +- Framework: xUnit with AwesomeAssertions +- Test archives: `tests/TestArchives/` - Use existing archives, don't create new ones unnecessarily +- Match naming style of nearby test files + +### Public API Change Checklist +- Preserve existing public method signatures and behavior when possible. +- If a breaking change is unavoidable, document it and provide a migration path. +- Add or update tests that cover backward compatibility expectations. +- Avoid exposing public `init` setters, positional records, `required` members, or other metadata that forces consumers onto newer C# language versions; validate older-consumer compatibility with tests when changing exported APIs. + +### Public API Documentation Checklist +- When adding, removing, renaming, or changing public APIs, update the public docs in the same change. +- Check `docs/API.md` for new factory methods, options, interfaces, extension methods, enums, archive/reader/writer APIs, compression provider APIs, and examples. +- Check `docs/USAGE.md` when the API change affects recommended usage patterns or requires a new example. +- Check `docs/FORMATS.md` when the API change affects supported archive formats, compression methods, reader/archive/writer availability, or detection behavior. +- Check `README.md` when the change affects the top-level support summary, target frameworks, major capabilities, or user-facing feature list. +- For public API changes, verify examples compile conceptually against the actual public signatures and avoid documenting internal-only types. + +### Stream Ownership and Position Checklist +- Verify `LeaveStreamOpen` behavior for externally owned streams. +- Validate behavior for both seekable and non-seekable streams. +- Ensure stream position assumptions are explicit and tested. + +## Common Pitfalls + +1. **Don't mix Archive and Reader APIs** - Archive needs seekable stream, Reader doesn't +2. **Don't mix sync and async open paths** - For async workflows use `OpenAsyncArchive`/`OpenAsyncReader`/`OpenAsyncWriter`, not `OpenArchive`/`OpenReader`/`OpenWriter` +3. **Solid archives (Rar, 7Zip)** - Use `ExtractAllEntries()` for best performance, not individual entry extraction +4. **Stream disposal** - Always set `LeaveStreamOpen` explicitly when needed (default is to close) +5. **Tar + non-seekable stream** - Must provide file size or it will throw +6. **Format detection** - Use `ReaderFactory.OpenReader()` / `ReaderFactory.OpenAsyncReader()` for auto-detection, test with actual archive files diff --git a/Directory.Build.props b/Directory.Build.props index 4b343bf8..6ff916b6 100644 --- a/Directory.Build.props +++ b/Directory.Build.props @@ -8,7 +8,9 @@ true true true - False - False + true + true + true + ${NoWarn};IDE0051 diff --git a/Directory.Packages.props b/Directory.Packages.props new file mode 100644 index 00000000..7559f05e --- /dev/null +++ b/Directory.Packages.props @@ -0,0 +1,23 @@ + + + + + + + + + + + + + + + + + + + + diff --git a/FORMATS.md b/FORMATS.md deleted file mode 100644 index cca190ad..00000000 --- a/FORMATS.md +++ /dev/null @@ -1,59 +0,0 @@ -# Formats - -## Accessing Archives - -* Archive classes allow random access to a seekable stream. -* Reader classes allow forward-only reading on a stream. -* Writer classes allow forward-only Writing on a stream. - -## Supported Format Table - -| Archive Format | Compression Format(s) | Compress/Decompress | Archive API | Reader API | Writer API | -| ---------------------- | ------------------------------------------------- | ------------------- | --------------- | ---------- | ------------- | -| Rar | Rar | Decompress (1) | RarArchive | RarReader | N/A | -| Zip (2) | None, DEFLATE, Deflate64, BZip2, LZMA/LZMA2, PPMd | Both | ZipArchive | ZipReader | ZipWriter | -| Tar | None | Both | TarArchive | TarReader | TarWriter (3) | -| Tar.GZip | DEFLATE | Both | TarArchive | TarReader | TarWriter (3) | -| Tar.BZip2 | BZip2 | Both | TarArchive | TarReader | TarWriter (3) | -| Tar.LZip | LZMA | Both | TarArchive | TarReader | TarWriter (3) | -| Tar.XZ | LZMA2 | Decompress | TarArchive | TarReader | TarWriter (3) | -| GZip (single file) | DEFLATE | Both | GZipArchive | GZipReader | GZipWriter | -| 7Zip (4) | LZMA, LZMA2, BZip2, PPMd, BCJ, BCJ2, Deflate | Decompress | SevenZipArchive | N/A | N/A | - -1. SOLID Rars are only supported in the RarReader API. -2. Zip format supports pkware and WinzipAES encryption. However, encrypted LZMA is not supported. Zip64 reading/writing is supported but only with seekable streams as the Zip spec doesn't support Zip64 data in post data descriptors. Deflate64 is only supported for reading. -3. The Tar format requires a file size in the header. If no size is specified to the TarWriter and the stream is not seekable, then an exception will be thrown. -4. The 7Zip format doesn't allow for reading as a forward-only stream so 7Zip is only supported through the Archive API -5. LZip has no support for extra data like the file name or timestamp. There is a default filename used when looking at the entry Key on the archive. - -## Compression Streams - -For those who want to directly compress/decompress bits. The single file formats are represented here as well. However, BZip2, LZip and XZ have no metadata (GZip has a little) so using them without something like a Tar file makes little sense. - -| Compressor | Compress/Decompress | -| --------------- | ------------------- | -| BZip2Stream | Both | -| GZipStream | Both | -| DeflateStream | Both | -| Deflate64Stream | Decompress | -| LZMAStream | Both | -| PPMdStream | Both | -| ADCStream | Decompress | -| LZipStream | Both | -| XZStream | Decompress | - -## Archive Formats vs Compression - -Sometimes the terminology gets mixed. - -### Compression - -DEFLATE, LZMA are pure compression algorithms - -### Formats - -Formats like Zip, 7Zip, Rar are archive formats only. They use other compression methods (e.g. DEFLATE, LZMA, etc.) or propriatory (e.g RAR) - -### Overlap - -GZip, BZip2 and LZip are single file archival formats. The overlap in the API happens because Tar uses the single file formats as "compression" methods and the API tries to hide this a bit. diff --git a/NuGet.config b/NuGet.config new file mode 100644 index 00000000..48259210 --- /dev/null +++ b/NuGet.config @@ -0,0 +1,9 @@ + + + + + + + + + diff --git a/README.md b/README.md index 20681105..1908272f 100644 --- a/README.md +++ b/README.md @@ -1,18 +1,24 @@ # SharpCompress -SharpCompress is a compression library in pure C# for .NET Standard 2.0, 2.1, .NET Core 3.1 and .NET 5.0 that can unrar, un7zip, unzip, untar unbzip2, ungzip, unlzip with forward-only reading and file random access APIs. Write support for zip/tar/bzip2/gzip/lzip are implemented. +SharpCompress is a compression library in pure C# for .NET Framework 4.8, .NET Standard 2.0/2.1, .NET 6.0, .NET 8.0, and .NET 10.0 that can unrar, un7zip, unzip, untar, unbzip2, ungzip, unlzip, unxz, unzstd, unarc, unarj, unace, and unlzw with forward-only reading and file random access APIs. Write support for zip, tar, bzip2, gzip, lzip, zstandard compression streams, and 7zip archives is implemented. The major feature is support for non-seekable streams so large files can be processed on the fly (i.e. download stream). +**NEW:** All I/O operations now support async/await for improved performance and scalability. See the [USAGE.md](docs/USAGE.md#async-examples) for examples. + GitHub Actions Build - -[![SharpCompress](https://github.com/adamhathcock/sharpcompress/actions/workflows/dotnetcore.yml/badge.svg)](https://github.com/adamhathcock/sharpcompress/actions/workflows/dotnetcore.yml) -[![Static Badge](https://img.shields.io/badge/API%20Documentation-RobiniaDocs-43bc00?logo=readme&logoColor=white)](https://www.robiniadocs.com/d/sharpcompress/api/SharpCompress.html) +[![SharpCompress](https://github.com/adamhathcock/sharpcompress/actions/workflows/nuget-release.yml/badge.svg)](https://github.com/adamhathcock/sharpcompress/actions/workflows/nuget-release.yml) +[![Static Badge](https://img.shields.io/badge/API%20Docs-DNDocs-190088?logo=readme&logoColor=white)](https://dndocs.com/d/sharpcompress/api/index.html) ## Need Help? Post Issues on Github! -Check the [Supported Formats](FORMATS.md) and [Basic Usage.](USAGE.md) +Check the [Supported Formats](docs/FORMATS.md), [API Reference](docs/API.md), and [Basic Usage](docs/USAGE.md). + +## Custom Compression Providers + +If you need to swap out SharpCompress’s built-in codecs, the `Providers` property (and `WithProviders(...)` extensions) on `ReaderOptions` and `WriterOptions` lets you supply a `CompressionProviderRegistry`. The selected registry is used by Reader/Writer APIs, Archive APIs, and async extraction paths, so the same provider choice is applied consistently across open/read/write flows. The default registry is already wired up, so customization is only necessary when you want to plug in alternatives such as `SystemGZipCompressionProvider` or a third-party `CompressionProvider`. See [docs/USAGE.md#custom-compression-providers](docs/USAGE.md#custom-compression-providers) for guided examples. ## Recommended Formats @@ -20,10 +26,12 @@ In general, I recommend GZip (Deflate)/BZip2 (BZip)/LZip (LZMA) as the simplicit Zip is okay, but it's a very hap-hazard format and the variation in headers and implementations makes it hard to get correct. Uses Deflate by default but supports a lot of compression methods. -RAR is not recommended as it's a propriatory format and the compression is closed source. Use Tar/LZip for LZMA +RAR is not recommended as it's a proprietary format and the compression is closed source. Use Tar/LZip for LZMA 7Zip and XZ both are overly complicated. 7Zip does not support streamable formats. XZ has known holes explained here: (http://www.nongnu.org/lzip/xz_inadequate.html) Use Tar/LZip for LZMA compression instead. +ZStandard is an efficient format that works well for streaming with a flexible compression level to tweak the speed/performance trade off you are looking for. + ## A Simple Request Hi everyone. I hope you're using SharpCompress and finding it useful. Please give me feedback on what you'd like to see changed especially as far as usability goes. New feature suggestions are always welcome as well. I would also like to know what projects SharpCompress is being used in. I like seeing how it is used to give me ideas for future versions. Thanks! @@ -34,154 +42,7 @@ Please do not email me directly to ask for help. If you think there is a real is I'm always looking for help or ideas. Please submit code or email with ideas. Unfortunately, just letting me know you'd like to help is not enough because I really have no overall plan of what needs to be done. I'll definitely accept code submissions and add you as a member of the project! -## TODOs (always lots) - -* RAR 5 decryption support -* 7Zip writing -* Zip64 (Need writing and extend Reading) -* Multi-volume Zip support. - -## Version Log - -* [Releases](https://github.com/adamhathcock/sharpcompress/releases) - -### Version 0.18 - -* [Now on Github releases](https://github.com/adamhathcock/sharpcompress/releases/tag/0.18) - -### Version 0.17.1 - -* Fix - [Bug Fix for .NET Core on Windows](https://github.com/adamhathcock/sharpcompress/pull/257) - -### Version 0.17.0 - -* New - Full LZip support! Can read and write LZip files and Tars inside LZip files. [Make LZip a first class citizen. #241](https://github.com/adamhathcock/sharpcompress/issues/241) -* New - XZ read support! Can read XZ files and Tars inside XZ files. [XZ in SharpCompress #91](https://github.com/adamhathcock/sharpcompress/issues/94) -* Fix - [Regression - zip file writing on seekable streams always assumed stream start was 0. Introduced with Zip64 writing.](https://github.com/adamhathcock/sharpcompress/issues/244) -* Fix - [Zip files with post-data descriptors can be properly skipped via decompression](https://github.com/adamhathcock/sharpcompress/issues/162) - -### Version 0.16.2 - -* Fix [.NET 3.5 should support files and cryptography (was a regression from 0.16.0)](https://github.com/adamhathcock/sharpcompress/pull/251) -* Fix [Zip per entry compression customization wrote the wrong method into the zip archive](https://github.com/adamhathcock/sharpcompress/pull/249) - -### Version 0.16.1 - -* Fix [Preserve compression method when getting a compressed stream](https://github.com/adamhathcock/sharpcompress/pull/235) -* Fix [RAR entry key normalization fix](https://github.com/adamhathcock/sharpcompress/issues/201) - -### Version 0.16.0 - -* Breaking - [Progress Event Tracking rethink](https://github.com/adamhathcock/sharpcompress/pull/226) -* Update to VS2017 - [VS2017](https://github.com/adamhathcock/sharpcompress/pull/231) - Framework targets have been changed. -* New - [Add Zip64 writing](https://github.com/adamhathcock/sharpcompress/pull/211) -* [Fix invalid/mismatching Zip version flags.](https://github.com/adamhathcock/sharpcompress/issues/164) - This allows nuget/System.IO.Packaging to read zip files generated by SharpCompress -* [Fix 7Zip directory hiding](https://github.com/adamhathcock/sharpcompress/pull/215/files) -* [Verify RAR CRC headers](https://github.com/adamhathcock/sharpcompress/pull/220) - -### Version 0.15.2 - -* [Fix invalid headers](https://github.com/adamhathcock/sharpcompress/pull/210) - fixes an issue creating large-ish zip archives that was introduced with zip64 reading. - -### Version 0.15.1 - -* [Zip64 extending information and ZipReader](https://github.com/adamhathcock/sharpcompress/pull/206) - -### Version 0.15.0 - -* [Add zip64 support for ZipArchive extraction](https://github.com/adamhathcock/sharpcompress/pull/205) - -### Version 0.14.1 - -* [.NET Assemblies aren't strong named](https://github.com/adamhathcock/sharpcompress/issues/158) -* [Pkware encryption for Zip files didn't allow for multiple reads of an entry](https://github.com/adamhathcock/sharpcompress/issues/197) -* [GZip Entry couldn't be read multiple times](https://github.com/adamhathcock/sharpcompress/issues/198) - -### Version 0.14.0 - -* [Support for LZip reading in for Tars](https://github.com/adamhathcock/sharpcompress/pull/191) - -### Version 0.13.1 - -* [Fix null password on ReaderFactory. Fix null options on SevenZipArchive](https://github.com/adamhathcock/sharpcompress/pull/188) -* [Make PpmdProperties lazy to avoid unnecessary allocations.](https://github.com/adamhathcock/sharpcompress/pull/185) - -### Version 0.13.0 - -* Breaking change: Big refactor of Options on API. -* 7Zip supports Deflate - -### Version 0.12.4 - -* Forward only zip issue fix https://github.com/adamhathcock/sharpcompress/issues/160 -* Try to fix frameworks again by copying targets from JSON.NET - -### Version 0.12.3 - -* 7Zip fixes https://github.com/adamhathcock/sharpcompress/issues/73 -* Maybe all profiles will work with project.json now - -### Version 0.12.2 - -* Support Profile 259 again - -### Version 0.12.1 - -* Support Silverlight 5 - -### Version 0.12.0 - -* .NET Core RTM! -* Bug fix for Tar long paths - -### Version 0.11.6 - -* Bug fix for global header in Tar -* Writers now have a leaveOpen `bool` overload. They won't close streams if not-requested to. - -### Version 0.11.5 - -* Bug fix in Skip method - -### Version 0.11.4 - -* SharpCompress is now endian neutral (matters for Mono platforms) -* Fix for Inflate (need to change implementation) -* Fixes for RAR detection - -### Version 0.11.1 - -* Added Cancel on IReader -* Removed .NET 2.0 support and LinqBridge dependency - -### Version 0.11 - -* Been over a year, contains mainly fixes from contributors! -* Possible breaking change: ArchiveEncoding is UTF8 by default now. -* TAR supports writing long names using longlink -* RAR Protect Header added - -### Version 0.10.3 - -* Finally fixed Disposal issue when creating a new archive with the Archive API - -### Version 0.10.2 - -* Fixed Rar Header reading for invalid extended time headers. -* Windows Store assembly is now strong named -* Known issues with Long Tar names being worked on -* Updated to VS2013 -* Portable targets SL5 and Windows Phone 8 (up from SL4 and WP7) - -### Version 0.10.1 - -* Fixed 7Zip extraction performance problem - -### Version 0.10: - -* Added support for RAR Decryption (thanks to https://github.com/hrasyid) -* Embedded some BouncyCastle crypto classes to allow RAR Decryption and Winzip AES Decryption in Portable and Windows Store DLLs -* Built in Release (I think) +## Notes XZ implementation based on: https://github.com/sambott/XZ.NET by @sambott @@ -189,6 +50,8 @@ XZ BCJ filters support contributed by Louis-Michel Bergeron, on behalf of aDolus 7Zip implementation based on: https://code.google.com/p/managed-lzma/ +Zstandard implementation from: https://github.com/oleg-st/ZstdSharp + LICENSE Copyright (c) 2000 - 2011 The Legion Of The Bouncy Castle (http://www.bouncycastle.org) diff --git a/SharpCompress.sln b/SharpCompress.sln deleted file mode 100644 index 71ec294e..00000000 --- a/SharpCompress.sln +++ /dev/null @@ -1,48 +0,0 @@ - -Microsoft Visual Studio Solution File, Format Version 12.00 -# Visual Studio 15 -VisualStudioVersion = 15.0.26430.6 -MinimumVisualStudioVersion = 10.0.40219.1 -Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "src", "src", "{3C5BE746-03E5-4895-9988-0B57F162F86C}" -EndProject -Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "tests", "tests", "{0F0901FF-E8D9-426A-B5A2-17C7F47C1529}" -EndProject -Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "SharpCompress", "src\SharpCompress\SharpCompress.csproj", "{FD19DDD8-72B2-4024-8665-0D1F7A2AA998}" -EndProject -Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "SharpCompress.Test", "tests\SharpCompress.Test\SharpCompress.Test.csproj", "{F2B1A1EB-0FA6-40D0-8908-E13247C7226F}" -EndProject -Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "build", "build\build.csproj", "{D4D613CB-5E94-47FB-85BE-B8423D20C545}" -EndProject -Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Config", "Config", "{CDB42573-7D22-4490-BA12-1B7FB99CE7FB}" - ProjectSection(SolutionItems) = preProject - Directory.Build.props = Directory.Build.props - global.json = global.json - EndProjectSection -EndProject -Global - GlobalSection(SolutionConfigurationPlatforms) = preSolution - Debug|Any CPU = Debug|Any CPU - Release|Any CPU = Release|Any CPU - EndGlobalSection - GlobalSection(ProjectConfigurationPlatforms) = postSolution - {FD19DDD8-72B2-4024-8665-0D1F7A2AA998}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {FD19DDD8-72B2-4024-8665-0D1F7A2AA998}.Debug|Any CPU.Build.0 = Debug|Any CPU - {FD19DDD8-72B2-4024-8665-0D1F7A2AA998}.Release|Any CPU.ActiveCfg = Release|Any CPU - {FD19DDD8-72B2-4024-8665-0D1F7A2AA998}.Release|Any CPU.Build.0 = Release|Any CPU - {F2B1A1EB-0FA6-40D0-8908-E13247C7226F}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {F2B1A1EB-0FA6-40D0-8908-E13247C7226F}.Debug|Any CPU.Build.0 = Debug|Any CPU - {F2B1A1EB-0FA6-40D0-8908-E13247C7226F}.Release|Any CPU.ActiveCfg = Release|Any CPU - {F2B1A1EB-0FA6-40D0-8908-E13247C7226F}.Release|Any CPU.Build.0 = Release|Any CPU - {D4D613CB-5E94-47FB-85BE-B8423D20C545}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {D4D613CB-5E94-47FB-85BE-B8423D20C545}.Debug|Any CPU.Build.0 = Debug|Any CPU - {D4D613CB-5E94-47FB-85BE-B8423D20C545}.Release|Any CPU.ActiveCfg = Release|Any CPU - {D4D613CB-5E94-47FB-85BE-B8423D20C545}.Release|Any CPU.Build.0 = Release|Any CPU - EndGlobalSection - GlobalSection(SolutionProperties) = preSolution - HideSolutionNode = FALSE - EndGlobalSection - GlobalSection(NestedProjects) = preSolution - {FD19DDD8-72B2-4024-8665-0D1F7A2AA998} = {3C5BE746-03E5-4895-9988-0B57F162F86C} - {F2B1A1EB-0FA6-40D0-8908-E13247C7226F} = {0F0901FF-E8D9-426A-B5A2-17C7F47C1529} - EndGlobalSection -EndGlobal diff --git a/SharpCompress.sln.DotSettings b/SharpCompress.sln.DotSettings deleted file mode 100644 index 248b4c35..00000000 --- a/SharpCompress.sln.DotSettings +++ /dev/null @@ -1,132 +0,0 @@ - - DO_NOT_SHOW - ERROR - ERROR - ERROR - ERROR - ERROR - ERROR - ERROR - ERROR - ERROR - DO_NOT_SHOW - - <?xml version="1.0" encoding="utf-16"?><Profile name="Basic Clean"><CSOptimizeUsings><OptimizeUsings>True</OptimizeUsings><EmbraceInRegion>False</EmbraceInRegion><RegionName></RegionName></CSOptimizeUsings><CSShortenReferences>True</CSShortenReferences><CSRemoveCodeRedundancies>True</CSRemoveCodeRedundancies><CSMakeFieldReadonly>True</CSMakeFieldReadonly><CSCodeStyleAttributes ArrangeTypeAccessModifier="False" ArrangeTypeMemberAccessModifier="False" SortModifiers="False" RemoveRedundantParentheses="False" AddMissingParentheses="False" ArrangeBraces="True" ArrangeAttributes="False" ArrangeArgumentsStyle="False" /><RemoveCodeRedundancies>True</RemoveCodeRedundancies><CSUseAutoProperty>True</CSUseAutoProperty><CSMakeAutoPropertyGetOnly>True</CSMakeAutoPropertyGetOnly><CSReformatCode>True</CSReformatCode></Profile> - - Basic Clean - True - Named - Required - Required - Required - Required - True - True - True - True - True - True - True - True - True - True - True - True - 0 - 1 - - SEPARATE - ALWAYS_ADD - ALWAYS_ADD - ALWAYS_ADD - ALWAYS_ADD - ALWAYS_ADD - ALWAYS_ADD - True - 1 - 1 - NEVER - NEVER - False - False - NEVER - True - False - NEVER - True - - True - - LINE_BREAK - False - True - True - - - False - False - False - CHOP_IF_LONG - CHOP_IF_LONG - CHOP_IF_LONG - False - CHOP_IF_LONG - UseVarWhenEvident - UseVarWhenEvident - UseVarWhenEvident - - <Policy Inspect="True" Prefix="_" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="AA_BB" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="AA_BB" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="AA_BB" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="AaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="AaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="AaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="AaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="AaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="AaBb" /> - <Policy Inspect="True" Prefix="I" Suffix="" Style="AaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="AaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="T" Suffix="" Style="AaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="AaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="AaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="AaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="AaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="aaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="AaBb" /> - <Policy Inspect="True" Prefix="" Suffix="" Style="AaBb" /> - True - True - True - True - True - True - True - True - True - <SessionState ContinuousTestingIsOn="False" ContinuousTestingMode="0" FrameworkVersion="{x:Null}" IsLocked="False" Name="All tests from Solution" PlatformMonoPreference="{x:Null}" PlatformType="{x:Null}" xmlns="urn:schemas-jetbrains-com:jetbrains-ut-session" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"> - <Solution /> -</SessionState> diff --git a/SharpCompress.slnx b/SharpCompress.slnx new file mode 100644 index 00000000..0f3880c9 --- /dev/null +++ b/SharpCompress.slnx @@ -0,0 +1,24 @@ + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/USAGE.md b/USAGE.md deleted file mode 100644 index c96f9c1b..00000000 --- a/USAGE.md +++ /dev/null @@ -1,158 +0,0 @@ -# SharpCompress Usage - -## Stream Rules (changed with 0.21) - -When dealing with Streams, the rule should be that you don't close a stream you didn't create. This, in effect, should mean you should always put a Stream in a using block to dispose it. - -However, the .NET Framework often has classes that will dispose streams by default to make things "easy" like the following: - -```C# -using (var reader = new StreamReader(File.Open("foo"))) -{ - ... -} -``` - -In this example, reader should get disposed. However, stream rules should say the the `FileStream` created by `File.Open` should remain open. However, the .NET Framework closes it for you by default unless you override the constructor. In general, you should be writing Stream code like this: - -```C# -using (var fileStream = File.Open("foo")) -using (var reader = new StreamReader(fileStream)) -{ - ... -} -``` - -To deal with the "correct" rules as well as the expectations of users, I've decided to always close wrapped streams as of 0.21. - -To be explicit though, consider always using the overloads that use `ReaderOptions` or `WriterOptions` and explicitly set `LeaveStreamOpen` the way you want. - -If using Compression Stream classes directly and you don't want the wrapped stream to be closed. Use the `NonDisposingStream` as a wrapped to prevent the stream being disposed. The change in 0.21 simplified a lot even though the usage is a bit more convoluted. - -## Samples - -Also, look over the tests for more thorough [examples](https://github.com/adamhathcock/sharpcompress/tree/master/tests/SharpCompress.Test) - -### Create Zip Archive from multiple files -```C# -using(var archive = ZipArchive.Create()) -{ - archive.AddEntry("file01.txt", "C:\\file01.txt"); - archive.AddEntry("file02.txt", "C:\\file02.txt"); - ... - - archive.SaveTo("C:\\temp.zip", CompressionType.Deflate); -} -``` - -### Create Zip Archive from all files in a directory to a file - -```C# -using (var archive = ZipArchive.Create()) -{ - archive.AddAllFromDirectory("D:\\temp"); - archive.SaveTo("C:\\temp.zip", CompressionType.Deflate); -} -``` - -### Create Zip Archive from all files in a directory and save in memory - -```C# -var memoryStream = new MemoryStream(); -using (var archive = ZipArchive.Create()) -{ - archive.AddAllFromDirectory("D:\\temp"); - archive.SaveTo(memoryStream, new WriterOptions(CompressionType.Deflate) - { - LeaveStreamOpen = true - }); -} -//reset memoryStream to be usable now -memoryStream.Position = 0; -``` - -### Extract all files from a Rar file to a directory using RarArchive - -```C# -using (var archive = RarArchive.Open("Test.rar")) -{ - foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) - { - entry.WriteToDirectory("D:\\temp", new ExtractionOptions() - { - ExtractFullPath = true, - Overwrite = true - }); - } -} -``` - -### Use ReaderFactory to autodetect archive type and Open the entry stream - -```C# -using (Stream stream = File.OpenRead("Tar.tar.bz2")) -using (var reader = ReaderFactory.Open(stream)) -{ - while (reader.MoveToNextEntry()) - { - if (!reader.Entry.IsDirectory) - { - Console.WriteLine(reader.Entry.Key); - reader.WriteEntryToDirectory(@"C:\temp", new ExtractionOptions() - { - ExtractFullPath = true, - Overwrite = true - }); - } - } -} -``` - -### Use ReaderFactory to autodetect archive type and Open the entry stream - -```C# -using (Stream stream = File.OpenRead("Tar.tar.bz2")) -using (var reader = ReaderFactory.Open(stream)) -{ - while (reader.MoveToNextEntry()) - { - if (!reader.Entry.IsDirectory) - { - using (var entryStream = reader.OpenEntryStream()) - { - entryStream.CopyTo(...); - } - } - } -} -``` - -### Use WriterFactory to write all files from a directory in a streaming manner. - -```C# -using (Stream stream = File.OpenWrite("C:\\temp.tgz")) -using (var writer = WriterFactory.Open(stream, ArchiveType.Tar, new WriterOptions(CompressionType.GZip) - { - LeaveOpenStream = true - })) -{ - writer.WriteAll("D:\\temp", "*", SearchOption.AllDirectories); -} -``` - -### Extract zip which has non-utf8 encoded filename(cp932) - -```C# -var opts = new SharpCompress.Readers.ReaderOptions(); -var encoding = Encoding.GetEncoding(932); -opts.ArchiveEncoding = new SharpCompress.Common.ArchiveEncoding(); -opts.ArchiveEncoding.CustomDecoder = (data, x, y) => -{ - return encoding.GetString(data); -}; -var tr = SharpCompress.Archives.Zip.ZipArchive.Open("test.zip", opts); -foreach(var entry in tr.Entries) -{ - Console.WriteLine($"{entry.Key}"); -} -``` diff --git a/build/Program.cs b/build/Program.cs index 9475a66b..4f9d313c 100644 --- a/build/Program.cs +++ b/build/Program.cs @@ -1,21 +1,33 @@ using System; using System.Collections.Generic; +using System.Globalization; using System.IO; +using System.Linq; using System.Runtime.InteropServices; +using System.Text.RegularExpressions; +using System.Threading.Tasks; using GlobExpressions; using static Bullseye.Targets; using static SimpleExec.Command; const string Clean = "clean"; const string Restore = "restore"; +const string UpdateLocks = "update-locks"; const string Build = "build"; const string Test = "test"; const string Format = "format"; +const string CheckFormat = "check-format"; const string Publish = "publish"; +const string DetermineVersion = "determine-version"; +const string UpdateVersion = "update-version"; +const string PushToNuGet = "push-to-nuget"; +const string DisplayBenchmarkResults = "display-benchmark-results"; +const string CompareBenchmarkResults = "compare-benchmark-results"; +const string GenerateBaseline = "generate-baseline"; Target( Clean, - ForEach("**/bin", "**/obj"), + ["**/bin", "**/obj"], dir => { IEnumerable GetDirectories(string d) @@ -44,14 +56,23 @@ Target( () => { Run("dotnet", "tool restore"); - Run("dotnet", "csharpier --check ."); + Run("dotnet", "csharpier format ."); } ); -Target(Restore, DependsOn(Format), () => Run("dotnet", "restore")); +Target( + CheckFormat, + () => + { + Run("dotnet", "tool restore"); + Run("dotnet", "csharpier check ."); + } +); +Target(Restore, [CheckFormat], () => Run("dotnet", "restore --locked-mode")); +Target(UpdateLocks, [CheckFormat], () => Run("dotnet", "restore --force-evaluate")); Target( Build, - DependsOn(Restore), + [Restore], () => { Run("dotnet", "build src/SharpCompress/SharpCompress.csproj -c Release --no-restore"); @@ -60,8 +81,8 @@ Target( Target( Test, - DependsOn(Build), - ForEach("net7.0", "net462"), + [Build], + ["net10.0", "net48"], framework => { IEnumerable GetFiles(string d) @@ -69,7 +90,7 @@ Target( return Glob.Files(".", d); } - if (!RuntimeInformation.IsOSPlatform(OSPlatform.Windows) && framework == "net462") + if (!RuntimeInformation.IsOSPlatform(OSPlatform.Windows) && framework == "net48") { return; } @@ -83,13 +104,611 @@ Target( Target( Publish, - DependsOn(Test), + [Test], () => { Run("dotnet", "pack src/SharpCompress/SharpCompress.csproj -c Release -o artifacts/"); } ); -Target("default", DependsOn(Publish), () => Console.WriteLine("Done!")); +Target( + DetermineVersion, + async () => + { + var (version, isPrerelease) = await GetVersion(); + Console.WriteLine($"VERSION={version}"); + Console.WriteLine( + $"PRERELEASE={isPrerelease.ToString().ToLower(CultureInfo.InvariantCulture)}" + ); + + // Write to environment file for GitHub Actions + var githubOutput = Environment.GetEnvironmentVariable("GITHUB_OUTPUT"); + if (!string.IsNullOrEmpty(githubOutput)) + { + File.AppendAllText(githubOutput, $"version={version}\n"); + File.AppendAllText( + githubOutput, + $"prerelease={isPrerelease.ToString().ToLower(CultureInfo.InvariantCulture)}\n" + ); + } + } +); + +Target( + UpdateVersion, + async () => + { + var version = Environment.GetEnvironmentVariable("VERSION"); + if (string.IsNullOrEmpty(version)) + { + var (detectedVersion, _) = await GetVersion(); + version = detectedVersion; + } + + Console.WriteLine($"Updating project file with version: {version}"); + + var projectPath = "src/SharpCompress/SharpCompress.csproj"; + var content = File.ReadAllText(projectPath); + + // Get base version (without prerelease suffix) + var baseVersion = version.Split('-')[0]; + + // Update VersionPrefix + content = Regex.Replace( + content, + @"[^<]*", + $"{version}" + ); + + // Update AssemblyVersion + content = Regex.Replace( + content, + @"[^<]*", + $"{baseVersion}" + ); + + // Update FileVersion + content = Regex.Replace( + content, + @"[^<]*", + $"{baseVersion}" + ); + + File.WriteAllText(projectPath, content); + Console.WriteLine($"Updated VersionPrefix to: {version}"); + Console.WriteLine($"Updated AssemblyVersion and FileVersion to: {baseVersion}"); + } +); + +Target( + PushToNuGet, + () => + { + var apiKey = Environment.GetEnvironmentVariable("NUGET_API_KEY"); + if (string.IsNullOrEmpty(apiKey)) + { + Console.WriteLine( + "NUGET_API_KEY environment variable is not set. Skipping NuGet push." + ); + return; + } + + var packages = Directory.GetFiles("artifacts", "*.nupkg"); + if (packages.Length == 0) + { + Console.WriteLine("No packages found in artifacts directory."); + return; + } + + foreach (var package in packages) + { + Console.WriteLine($"Pushing {package} to NuGet.org"); + try + { + // Note: API key is passed via command line argument which is standard practice for dotnet nuget push + // The key is already in an environment variable and not displayed in normal output + Run( + "dotnet", + $"nuget push \"{package}\" --api-key {apiKey} --source https://api.nuget.org/v3/index.json --skip-duplicate" + ); + } + catch (Exception ex) + { + Console.WriteLine($"Failed to push {package}: {ex.Message}"); + throw; + } + } + } +); + +Target( + DisplayBenchmarkResults, + () => + { + var githubStepSummary = Environment.GetEnvironmentVariable("GITHUB_STEP_SUMMARY"); + var resultsDir = "benchmark-results/results"; + + if (!Directory.Exists(resultsDir)) + { + Console.WriteLine("No benchmark results found."); + return; + } + + var markdownFiles = Directory + .GetFiles(resultsDir, "*-report-github.md") + .OrderBy(f => f) + .ToList(); + + if (markdownFiles.Count == 0) + { + Console.WriteLine("No benchmark markdown reports found."); + return; + } + + var output = new List { "## Benchmark Results", "" }; + + foreach (var file in markdownFiles) + { + Console.WriteLine($"Processing {Path.GetFileName(file)}"); + var content = File.ReadAllText(file); + output.Add(content); + output.Add(""); + } + + // Write to GitHub Step Summary if available + if (!string.IsNullOrEmpty(githubStepSummary)) + { + File.AppendAllLines(githubStepSummary, output); + Console.WriteLine($"Benchmark results written to GitHub Step Summary"); + } + else + { + // Write to console if not in GitHub Actions + foreach (var line in output) + { + Console.WriteLine(line); + } + } + } +); + +Target( + CompareBenchmarkResults, + () => + { + var githubStepSummary = Environment.GetEnvironmentVariable("GITHUB_STEP_SUMMARY"); + var baselinePath = "tests/SharpCompress.Performance/baseline-results.md"; + var resultsDir = "benchmark-results/results"; + + var output = new List { "## Comparison with Baseline", "" }; + + if (!File.Exists(baselinePath)) + { + Console.WriteLine("Baseline file not found"); + output.Add("⚠️ Baseline file not found. Run `generate-baseline` to create it."); + WriteOutput(output, githubStepSummary); + return; + } + + if (!Directory.Exists(resultsDir)) + { + Console.WriteLine("No current benchmark results found."); + output.Add("⚠️ No current benchmark results found. Showing baseline only."); + output.Add(""); + output.Add("### Baseline Results"); + output.AddRange(File.ReadAllLines(baselinePath)); + WriteOutput(output, githubStepSummary); + return; + } + + var markdownFiles = Directory + .GetFiles(resultsDir, "*-report-github.md") + .OrderBy(f => f) + .ToList(); + + if (markdownFiles.Count == 0) + { + Console.WriteLine("No current benchmark markdown reports found."); + output.Add("⚠️ No current benchmark results found. Showing baseline only."); + output.Add(""); + output.Add("### Baseline Results"); + output.AddRange(File.ReadAllLines(baselinePath)); + WriteOutput(output, githubStepSummary); + return; + } + + Console.WriteLine("Parsing baseline results..."); + var baselineMetrics = ParseBenchmarkResults(File.ReadAllText(baselinePath)); + + Console.WriteLine("Parsing current results..."); + var currentText = string.Join("\n", markdownFiles.Select(f => File.ReadAllText(f))); + var currentMetrics = ParseBenchmarkResults(currentText); + + Console.WriteLine("Comparing results..."); + output.Add("### Performance Comparison"); + output.Add(""); + output.Add( + "| Benchmark | Baseline Mean | Current Mean | Change | Baseline Memory | Current Memory | Change |" + ); + output.Add( + "|-----------|---------------|--------------|--------|-----------------|----------------|--------|" + ); + + var hasRegressions = false; + var hasImprovements = false; + + foreach (var method in currentMetrics.Keys.Union(baselineMetrics.Keys).OrderBy(k => k)) + { + var hasCurrent = currentMetrics.TryGetValue(method, out var current); + var hasBaseline = baselineMetrics.TryGetValue(method, out var baseline); + + if (!hasCurrent) + { + output.Add( + $"| {method} | {baseline!.Mean} | ❌ Missing | N/A | {baseline.Memory} | N/A | N/A |" + ); + continue; + } + + if (!hasBaseline) + { + output.Add( + $"| {method} | ❌ New | {current!.Mean} | N/A | N/A | {current.Memory} | N/A |" + ); + continue; + } + + var timeChange = CalculateChange(baseline!.MeanValue, current!.MeanValue); + var memChange = CalculateChange(baseline.MemoryValue, current.MemoryValue); + + var timeIcon = + timeChange > 25 ? "🔴" + : timeChange < -25 ? "🟢" + : "⚪"; + var memIcon = + memChange > 25 ? "🔴" + : memChange < -25 ? "🟢" + : "⚪"; + + if (timeChange > 25 || memChange > 25) + { + hasRegressions = true; + } + if (timeChange < -25 || memChange < -25) + { + hasImprovements = true; + } + + output.Add( + $"| {method} | {baseline.Mean} | {current.Mean} | {timeIcon} {timeChange:+0.0;-0.0;0}% | {baseline.Memory} | {current.Memory} | {memIcon} {memChange:+0.0;-0.0;0}% |" + ); + } + + output.Add(""); + output.Add("**Legend:**"); + output.Add("- 🔴 Regression (>25% slower/more memory)"); + output.Add("- 🟢 Improvement (>25% faster/less memory)"); + output.Add("- ⚪ No significant change"); + + if (hasRegressions) + { + output.Add(""); + output.Add( + "⚠️ **Warning**: Performance regressions detected. Review the changes carefully." + ); + } + else if (hasImprovements) + { + output.Add(""); + output.Add("✅ Performance improvements detected!"); + } + else + { + output.Add(""); + output.Add("✅ Performance is stable compared to baseline."); + } + + WriteOutput(output, githubStepSummary); + } +); + +Target( + GenerateBaseline, + () => + { + var perfProject = "tests/SharpCompress.Performance/SharpCompress.Performance.csproj"; + var baselinePath = "tests/SharpCompress.Performance/baseline-results.md"; + var artifactsDir = "baseline-artifacts"; + + Console.WriteLine("Building performance project..."); + Run("dotnet", $"build {perfProject} --configuration Release"); + + Console.WriteLine("Running benchmarks to generate baseline..."); + Run( + "dotnet", + $"run --project {perfProject} --configuration Release --no-build -- --filter \"*\" --exporters markdown --artifacts {artifactsDir}" + ); + + var resultsDir = Path.Combine(artifactsDir, "results"); + if (!Directory.Exists(resultsDir)) + { + Console.WriteLine("ERROR: No benchmark results generated."); + return; + } + + var markdownFiles = Directory + .GetFiles(resultsDir, "*-report-github.md") + .OrderBy(f => f) + .ToList(); + + if (markdownFiles.Count == 0) + { + Console.WriteLine("ERROR: No markdown reports found."); + return; + } + + Console.WriteLine($"Combining {markdownFiles.Count} benchmark reports..."); + var baselineContent = new List(); + + foreach (var file in markdownFiles) + { + var lines = File.ReadAllLines(file); + baselineContent.AddRange(lines.Select(l => l.Trim()).Where(l => l.StartsWith('|'))); + } + + File.WriteAllText(baselinePath, string.Join(Environment.NewLine, baselineContent)); + Console.WriteLine($"Baseline written to {baselinePath}"); + + // Clean up artifacts directory + if (Directory.Exists(artifactsDir)) + { + Directory.Delete(artifactsDir, true); + Console.WriteLine("Cleaned up artifacts directory."); + } + } +); + +Target("default", [Publish], () => Console.WriteLine("Done!")); await RunTargetsAndExitAsync(args); + +static async Task<(string version, bool isPrerelease)> GetVersion() +{ + // Check if current commit has a version tag + var currentTag = (await GetGitOutput("tag", "--points-at HEAD")) + .Split('\n', StringSplitOptions.RemoveEmptyEntries) + .FirstOrDefault(tag => Regex.IsMatch(tag.Trim(), @"^\d+\.\d+\.\d+$")); + + if (!string.IsNullOrEmpty(currentTag)) + { + // Tagged release - use the tag as version + var version = currentTag.Trim(); + Console.WriteLine($"Building tagged release version: {version}"); + return (version, false); + } + else + { + // Not tagged - create prerelease version + var allTags = (await GetGitOutput("tag", "--list")) + .Split('\n', StringSplitOptions.RemoveEmptyEntries) + .Where(tag => Regex.IsMatch(tag.Trim(), @"^\d+\.\d+\.\d+$")) + .Select(tag => tag.Trim()) + .ToList(); + + var lastTag = allTags.OrderBy(tag => Version.Parse(tag)).LastOrDefault() ?? "0.0.0"; + var lastVersion = Version.Parse(lastTag); + + // Determine version increment based on branch + var currentBranch = await GetCurrentBranch(); + Version nextVersion; + + if (currentBranch == "release") + { + // Release branch: increment patch version + nextVersion = new Version(lastVersion.Major, lastVersion.Minor, lastVersion.Build + 1); + Console.WriteLine($"Building prerelease for release branch (patch increment)"); + } + else + { + // Master or other branches: increment minor version + nextVersion = new Version(lastVersion.Major, lastVersion.Minor + 1, 0); + Console.WriteLine($"Building prerelease for {currentBranch} branch (minor increment)"); + } + + // Use commit count since the last version tag if available; otherwise, fall back to total count + var revListArgs = allTags.Any() ? $"--count {lastTag}..HEAD" : "--count HEAD"; + var commitCount = (await GetGitOutput("rev-list", revListArgs)).Trim(); + + var version = $"{nextVersion}-beta.{commitCount}"; + Console.WriteLine($"Building prerelease version: {version}"); + return (version, true); + } +} + +static async Task GetCurrentBranch() +{ + // In GitHub Actions, GITHUB_REF_NAME contains the branch name + var githubRefName = Environment.GetEnvironmentVariable("GITHUB_REF_NAME"); + if (!string.IsNullOrEmpty(githubRefName)) + { + return githubRefName; + } + + // Fallback to git command for local builds + try + { + var (output, _) = await ReadAsync("git", "branch --show-current"); + return output.Trim(); + } + catch (Exception ex) + { + Console.WriteLine($"Warning: Could not determine current branch: {ex.Message}"); + return "unknown"; + } +} + +static async Task GetGitOutput(string command, string args) +{ + try + { + // Use SimpleExec's Read to execute git commands in a cross-platform way + var (output, _) = await ReadAsync("git", $"{command} {args}"); + return output; + } + catch (Exception ex) + { + throw new InvalidOperationException( + $"Git command failed: git {command} {args}\n{ex.Message}", + ex + ); + } +} + +static void WriteOutput(List output, string? githubStepSummary) +{ + if (!string.IsNullOrEmpty(githubStepSummary)) + { + File.AppendAllLines(githubStepSummary, output); + Console.WriteLine("Comparison written to GitHub Step Summary"); + } + else + { + foreach (var line in output) + { + Console.WriteLine(line); + } + } +} + +static Dictionary ParseBenchmarkResults(string markdown) +{ + var metrics = new Dictionary(); + var lines = markdown.Split('\n'); + + for (int i = 0; i < lines.Length; i++) + { + var line = lines[i].Trim(); + + // Look for table rows with benchmark data + if (line.StartsWith('|') && line.Contains("'", StringComparison.Ordinal) && i > 0) + { + var parts = line.Split('|', StringSplitOptions.TrimEntries); + if (parts.Length >= 5) + { + var method = parts[1].Replace("'", "'", StringComparison.Ordinal); + var meanStr = parts[2]; + + // Find Allocated column - it's usually the last column or labeled "Allocated" + string memoryStr = "N/A"; + for (int j = parts.Length - 2; j >= 2; j--) + { + if ( + parts[j].Contains("KB", StringComparison.Ordinal) + || parts[j].Contains("MB", StringComparison.Ordinal) + || parts[j].Contains("GB", StringComparison.Ordinal) + || parts[j].Contains('B', StringComparison.Ordinal) + ) + { + memoryStr = parts[j]; + break; + } + } + + if ( + !method.Equals("Method", StringComparison.OrdinalIgnoreCase) + && !string.IsNullOrWhiteSpace(method) + ) + { + var metric = new BenchmarkMetric + { + Method = method, + Mean = meanStr, + MeanValue = ParseTimeValue(meanStr), + Memory = memoryStr, + MemoryValue = ParseMemoryValue(memoryStr), + }; + metrics[method] = metric; + } + } + } + } + + return metrics; +} + +static double ParseTimeValue(string timeStr) +{ + if (string.IsNullOrWhiteSpace(timeStr) || timeStr == "N/A" || timeStr == "NA") + { + return 0; + } + + // Remove thousands separators and parse + timeStr = timeStr.Replace(",", "", StringComparison.Ordinal).Trim(); + + var match = Regex.Match(timeStr, @"([\d.]+)\s*(\w+)"); + if (!match.Success) + { + return 0; + } + + var value = double.Parse(match.Groups[1].Value, CultureInfo.InvariantCulture); + var unit = match.Groups[2].Value.ToLower(CultureInfo.InvariantCulture); + + // Convert to microseconds for comparison + return unit switch + { + "s" => value * 1_000_000, + "ms" => value * 1_000, + "μs" or "us" => value, + "ns" => value / 1_000, + _ => value, + }; +} + +static double ParseMemoryValue(string memStr) +{ + if (string.IsNullOrWhiteSpace(memStr) || memStr == "N/A" || memStr == "NA") + { + return 0; + } + + memStr = memStr.Replace(",", "", StringComparison.Ordinal).Trim(); + + var match = Regex.Match(memStr, @"([\d.]+)\s*(\w+)"); + if (!match.Success) + { + return 0; + } + + var value = double.Parse(match.Groups[1].Value, CultureInfo.InvariantCulture); + var unit = match.Groups[2].Value.ToUpper(CultureInfo.InvariantCulture); + + // Convert to KB for comparison + return unit switch + { + "GB" => value * 1_024 * 1_024, + "MB" => value * 1_024, + "KB" => value, + "B" => value / 1_024, + _ => value, + }; +} + +static double CalculateChange(double baseline, double current) +{ + if (baseline == 0) + { + return 0; + } + return ((current - baseline) / baseline) * 100; +} + +record BenchmarkMetric +{ + public string Method { get; init; } = ""; + public string Mean { get; init; } = ""; + public double MeanValue { get; init; } + public string Memory { get; init; } = ""; + public double MemoryValue { get; init; } +} diff --git a/build/build.csproj b/build/build.csproj index 7b478cf3..8f5b6f3c 100644 --- a/build/build.csproj +++ b/build/build.csproj @@ -1,14 +1,11 @@ - Exe - net7.0 + net10.0 - - - - + + + - diff --git a/build/packages.lock.json b/build/packages.lock.json new file mode 100644 index 00000000..8cc0db72 --- /dev/null +++ b/build/packages.lock.json @@ -0,0 +1,80 @@ +{ + "version": 2, + "dependencies": { + "net10.0": { + "Bullseye": { + "type": "Direct", + "requested": "[6.1.0, )", + "resolved": "6.1.0", + "contentHash": "fltnAJDe0BEX5eymXGUq+il2rSUA0pHqUonNDRH2TrvRu8SkU17mYG0IVpdmG2ibtfhdjNrv4CuTCxHOwcozCA==" + }, + "Glob": { + "type": "Direct", + "requested": "[1.1.9, )", + "resolved": "1.1.9", + "contentHash": "AfK5+ECWYTP7G3AAdnU8IfVj+QpGjrh9GC2mpdcJzCvtQ4pnerAGwHsxJ9D4/RnhDUz2DSzd951O/lQjQby2Sw==" + }, + "Microsoft.NETFramework.ReferenceAssemblies": { + "type": "Direct", + "requested": "[1.0.3, )", + "resolved": "1.0.3", + "contentHash": "vUc9Npcs14QsyOD01tnv/m8sQUnGTGOw1BCmKcv77LBJY7OxhJ+zJF7UD/sCL3lYNFuqmQEVlkfS4Quif6FyYg==", + "dependencies": { + "Microsoft.NETFramework.ReferenceAssemblies.net461": "1.0.3" + } + }, + "Microsoft.SourceLink.GitHub": { + "type": "Direct", + "requested": "[10.0.300, )", + "resolved": "10.0.300", + "contentHash": "QzCtLkXVb3l4IxcpvJCbzUwMLihAmLN6vVLjQGSzYSF8d2dvXxqJAZk83RV3gYnp2egz8jRMgSR2woY3vOahTA==", + "dependencies": { + "Microsoft.Build.Tasks.Git": "10.0.300", + "Microsoft.SourceLink.Common": "10.0.300", + "System.IO.Hashing": "10.0.8" + } + }, + "Microsoft.VisualStudio.Threading.Analyzers": { + "type": "Direct", + "requested": "[17.14.15, )", + "resolved": "17.14.15", + "contentHash": "mXQPJsbuUD2ydq4/ffd8h8tSOFCXec+2xJOVNCvXjuMOq/+5EKHq3D2m2MC2+nUaXeFMSt66VS/J4HdKBixgcw==" + }, + "PolySharp": { + "type": "Direct", + "requested": "[1.16.0, )", + "resolved": "1.16.0", + "contentHash": "3kdIIceBPumwjw279FuiVMfVENT2cGASXJgcigdySsbX2dJB8ofUgG6i47yqF/k1qu6fvNR3csrSekZPviR6kQ==" + }, + "SimpleExec": { + "type": "Direct", + "requested": "[13.0.0, )", + "resolved": "13.0.0", + "contentHash": "zcCR1pupa1wI1VqBULRiQKeHKKZOuJhi/K+4V5oO+rHJZlaOD53ViFo1c3PavDoMAfSn/FAXGAWpPoF57rwhYg==" + }, + "Microsoft.Build.Tasks.Git": { + "type": "Transitive", + "resolved": "10.0.300", + "contentHash": "P0kaQwVZx4xIUe2FtrLyBadYNXuAljttJUPvjBYRuHhPE8L77L42KakLDkaADRiUrGspoLcMwayjrbQhYTr0zA==", + "dependencies": { + "System.IO.Hashing": "10.0.8" + } + }, + "Microsoft.NETFramework.ReferenceAssemblies.net461": { + "type": "Transitive", + "resolved": "1.0.3", + "contentHash": "AmOJZwCqnOCNp6PPcf9joyogScWLtwy0M1WkqfEQ0M9nYwyDD7EX9ZjscKS5iYnyvteX7kzSKFCKt9I9dXA6mA==" + }, + "Microsoft.SourceLink.Common": { + "type": "Transitive", + "resolved": "10.0.300", + "contentHash": "0jlkXaUGjYlWTIVPve5MftjKHnT3SlAtq9BCLV4J9IjdPrxV/+4rMlBSjfr1khG8/GC6KGojjola8E1VvWF0qQ==" + }, + "System.IO.Hashing": { + "type": "Transitive", + "resolved": "10.0.8", + "contentHash": "+dJsbPJ3FyUbTZNplFj0RCKePFizmv6ewDV46JE9q/IVH4c3xTCftHfHelLsAKf0jryIPqgMb5GpS0x7TAY3mg==" + } + } + } +} \ No newline at end of file diff --git a/docs/API.md b/docs/API.md new file mode 100644 index 00000000..db807f55 --- /dev/null +++ b/docs/API.md @@ -0,0 +1,810 @@ +# API Quick Reference + +Quick reference for commonly used SharpCompress APIs. + +## Factory Methods + +### Opening Archives + +```csharp +// Auto-detect format +using (var reader = ReaderFactory.OpenReader(stream)) +{ + // Works with Zip, Tar, GZip, Rar, 7Zip, etc. +} + +// Specific format - Archive API +using (var archive = ZipArchive.OpenArchive("file.zip")) +using (var archive = TarArchive.OpenArchive("file.tar")) +using (var archive = RarArchive.OpenArchive("file.rar")) +using (var archive = SevenZipArchive.OpenArchive("file.7z")) +using (var archive = GZipArchive.OpenArchive("file.gz")) + +// With fluent options (preferred) +var options = ReaderOptions.ForEncryptedArchive("password") + .WithArchiveEncoding(new ArchiveEncoding { Default = Encoding.GetEncoding(932) }); +using (var archive = ZipArchive.OpenArchive("encrypted.zip", options)) + +// Alternative: object initializer +var options2 = new ReaderOptions +{ + Password = "password", + LeaveStreamOpen = true, + ArchiveEncoding = new ArchiveEncoding { Default = Encoding.GetEncoding(932) } +}; +``` + +```csharp +// Auto-detect and open with the Archive API +using (var archive = ArchiveFactory.OpenArchive("archive.zip")) +{ + foreach (var entry in archive.Entries) + { + Console.WriteLine(entry.Key); + } +} + +// Detect before opening +if (ArchiveFactory.IsArchive("archive.zip", out var archiveType)) +{ + Console.WriteLine($"Detected {archiveType}"); +} + +// Detect capabilities before choosing Archive API vs Reader API +var info = ArchiveFactory.GetArchiveInformation("archive.arc"); +if (info is not null) +{ + Console.WriteLine($"Type: {info.Type}"); + Console.WriteLine($"Supports random access: {info.SupportsRandomAccess}"); +} + +var asyncInfo = await ArchiveFactory.GetArchiveInformationAsync( + "archive.zip", + cancellationToken +); + +// Multi-volume archives +var parts = ArchiveFactory.GetFileParts("archive.part1.rar") + .Select(path => new FileInfo(path)) + .ToArray(); +using (var archive = ArchiveFactory.OpenArchive(parts)) +{ + archive.WriteToDirectory(@"C:\output"); +} +``` + +`ArchiveInformation.SupportsRandomAccess` is `true` when the detected format supports `IArchive` random access. It is `false` for reader-only formats such as Ace, Arc, Arj, and standalone LZW, where `ReaderFactory.OpenReader` should be used instead. + +### Creating Archives + +```csharp +// Writer Factory +using (var writer = WriterFactory.OpenWriter(stream, ArchiveType.Zip, CompressionType.Deflate)) +{ + // Write entries +} + +// Specific writer +using (var archive = ZipArchive.CreateArchive()) +using (var archive = TarArchive.CreateArchive()) +using (var archive = GZipArchive.CreateArchive()) + +// With fluent options (preferred) +var options = WriterOptions.ForZip() + .WithCompressionLevel(9) + .WithLeaveStreamOpen(false) + .WithBufferSize(131072); +using (var archive = ZipArchive.CreateArchive()) +{ + archive.SaveTo("output.zip", options); +} + +// Alternative: constructor with object initializer +var options2 = new WriterOptions(CompressionType.Deflate) +{ + CompressionLevel = 9, + LeaveStreamOpen = false, + BufferSize = 131072 +}; +``` + +`WriterOptions.BufferSize` controls stream copy buffers used while writing archive entries. If it is not set, SharpCompress falls back to `Constants.BufferSize`. + +--- + +## Archive API Methods + +### Reading/Extracting + +```csharp +using (var archive = ZipArchive.OpenArchive("file.zip")) +{ + // Get all entries + IEnumerable entries = archive.Entries; + + // Find specific entry + var entry = archive.Entries.FirstOrDefault(e => e.Key == "file.txt"); + + // Extract all + archive.WriteToDirectory(@"C:\output"); + + // Extract single entry + var firstEntry = archive.Entries.First(); + firstEntry.WriteToFile(@"C:\output\file.txt"); + + // Extract single entry to a stream with extraction options + using (var outputStream = File.Create(@"C:\output\file.txt")) + { + firstEntry.WriteTo( + outputStream, + new ExtractionOptions { CheckCrc = false } + ); + } + + // Get entry stream + using (var stream = entry.OpenEntryStream()) + { + stream.CopyTo(outputStream); + } +} + +// Async extraction (requires IAsyncArchive) +await using (var asyncArchive = await ZipArchive.OpenAsyncArchive("file.zip")) +{ + // Extract all entries asynchronously + await asyncArchive.WriteToDirectoryAsync( + @"C:\output", + cancellationToken: cancellationToken + ); +} + +// Open a specific entry stream asynchronously +await using (var asyncArchive = await ZipArchive.OpenAsyncArchive("file.zip")) +{ + await foreach (var entry in asyncArchive.EntriesAsync) + { + await using (var outputStream = File.Create(@"C:\output\" + entry.Key)) + { + await entry.WriteToAsync( + outputStream, + new ExtractionOptions { CheckCrc = false }, + cancellationToken: cancellationToken + ); + } + + using (var stream = await entry.OpenEntryStreamAsync(cancellationToken)) + { + // ... + } + } +} +``` + +### Entry Properties + +```csharp +foreach (var entry in archive.Entries) +{ + string name = entry.Key; // Entry name/path + long size = entry.Size; // Uncompressed size + long compressedSize = entry.CompressedSize; + bool isDir = entry.IsDirectory; + DateTime? modTime = entry.LastModifiedTime; + CompressionType compression = entry.CompressionType; +} +``` + +### Creating Archives + +```csharp +using (var archive = ZipArchive.CreateArchive()) +{ + // Add file + archive.AddEntry("file.txt", @"C:\source\file.txt"); + + // Add multiple files + archive.AddAllFromDirectory(@"C:\source"); + archive.AddAllFromDirectory(@"C:\source", "*.txt"); // Pattern + + // Save to file + archive.SaveTo("output.zip", CompressionType.Deflate); + + // Save to stream + archive.SaveTo(outputStream, new WriterOptions(CompressionType.Deflate) + { + CompressionLevel = 9, + LeaveStreamOpen = true + }); +} +``` + +--- + +## Reader API Methods + +### Forward-Only Reading + +```csharp +using (var stream = File.OpenRead("file.zip")) +using (var reader = ReaderFactory.OpenReader(stream)) +{ + while (reader.MoveToNextEntry()) + { + IArchiveEntry entry = reader.Entry; + + if (!entry.IsDirectory) + { + // Extract entry + reader.WriteEntryToDirectory(@"C:\output"); + reader.WriteEntryToFile(@"C:\output\file.txt"); + + // Or get stream + using (var entryStream = reader.OpenEntryStream()) + { + entryStream.CopyTo(outputStream); + } + } + } +} + +// Async variants (use OpenAsyncReader to get IAsyncReader) +using (var stream = File.OpenRead("file.zip")) +await using (var reader = await ReaderFactory.OpenAsyncReader(stream)) +{ + while (await reader.MoveToNextEntryAsync()) + { + await reader.WriteEntryToFileAsync( + @"C:\output\" + reader.Entry.Key, + cancellationToken: cancellationToken + ); + } + + // Async extraction of all entries + await reader.WriteAllToDirectoryAsync( + @"C:\output", + cancellationToken: cancellationToken + ); +} +``` + +--- + +## Writer API Methods + +### Creating Archives (Streaming) + +```csharp +using (var stream = File.Create("output.zip")) +using (var writer = WriterFactory.OpenWriter(stream, ArchiveType.Zip, CompressionType.Deflate)) +{ + // Write single file + using (var fileStream = File.OpenRead("source.txt")) + { + writer.Write("entry.txt", fileStream, DateTime.Now); + } + + // Write directory + writer.WriteAll("C:\\source", "*", SearchOption.AllDirectories); + writer.WriteAll("C:\\source", "*.txt", SearchOption.TopDirectoryOnly); +} + +// Async variants: use OpenAsyncWriter to get IAsyncWriter +await using var stream = File.Create("output.zip"); +await using var writer = await WriterFactory.OpenAsyncWriter(stream, ArchiveType.Zip, new WriterOptions(CompressionType.Deflate), cancellationToken); + +using (var fileStream = File.OpenRead("source.txt")) +{ + await writer.WriteAsync("entry.txt", fileStream, DateTime.Now, cancellationToken); +} + +await writer.WriteAllAsync("C:\\source", "*", SearchOption.AllDirectories, cancellationToken); +``` + +--- + +## Common Options + +### ReaderOptions + +Use preset properties and fluent helpers for common configurations: + +```csharp +// External stream with password and custom encoding +var options = ReaderOptions.ForExternalStream + .WithPassword("password") + .WithArchiveEncoding(new ArchiveEncoding { Default = Encoding.GetEncoding(932) }); + +using (var archive = ZipArchive.OpenArchive("file.zip", options)) +{ + // ... +} + +// Open-time presets +var external = ReaderOptions.ForExternalStream; +var owned = ReaderOptions.ForFilePath; +var encrypted = ReaderOptions.ForEncryptedArchive("password"); +var encoded = ReaderOptions.ForEncoding(new ArchiveEncoding { Default = Encoding.UTF8 }); +var sfx = ReaderOptions.ForSelfExtractingArchive("password"); + +// Faster detection when the container is known +var hinted = ReaderOptions.ForExternalStream.WithExtensionHint("tar.gz"); + +// Increase for non-seekable streams with large detection probes, such as SFX RAR +var buffered = ReaderOptions.ForExternalStream.WithRewindableBufferSize(1_048_576); + +// Extraction presets +var safeOptions = ExtractionOptions.SafeExtract; // No overwrite +var flatOptions = ExtractionOptions.FlatExtract; // No directory structure +var metadataOptions = ExtractionOptions.PreserveMetadata; // Keep timestamps and attributes + +// Tune extraction copy buffering +var extractionOptions = new ExtractionOptions +{ + BufferSize = 131072, + CheckCrc = true, // Default: validate entry checksums when archive metadata provides one +}; + +// Factory defaults: +// - file path / FileInfo overloads use LeaveStreamOpen = false +// - stream overloads use LeaveStreamOpen = true +``` + +Alternative: traditional object initializer: + +```csharp +var options = new ReaderOptions +{ + Password = "password", + LeaveStreamOpen = true, + ArchiveEncoding = new ArchiveEncoding { Default = Encoding.GetEncoding(932) }, + ExtensionHint = "zip", + LookForHeader = true, + DisableCheckIncomplete = false, + BufferSize = 81920, + RewindableBufferSize = 1_048_576, +}; + +var extractionOptions = new ExtractionOptions +{ + ExtractFullPath = true, + Overwrite = true, + BufferSize = 131072, +}; +``` + +### WriterOptions + +Factory methods provide a clean, discoverable way to create writer options: + +```csharp +// Factory methods for common archive types +var zipOptions = WriterOptions.ForZip() // ZIP with Deflate + .WithCompressionLevel(9) // 0-9 for Deflate + .WithLeaveStreamOpen(false); // Close stream when done + +var tarOptions = WriterOptions.ForTar(CompressionType.GZip) // TAR with GZip + .WithLeaveStreamOpen(false); + +var gzipOptions = WriterOptions.ForGZip() // GZip file + .WithCompressionLevel(6); + +archive.SaveTo("output.zip", zipOptions); +``` + +Use typed writer options when a format exposes extra settings: + +```csharp +// ZIP archive-level options +var zipWriterOptions = new ZipWriterOptions(CompressionType.Deflate) +{ + ArchiveComment = "Created by SharpCompress", + UseZip64 = true, + CompressionLevel = 9, +}; + +// ZIP per-entry options +using (var writer = new ZipWriter(outputStream, zipWriterOptions)) +using (var source = File.OpenRead("source.txt")) +{ + writer.Write("entry.txt", source, new ZipWriterEntryOptions + { + CompressionType = CompressionType.ZStandard, + CompressionLevel = 3, + EntryComment = "per-entry comment", + ModificationDateTime = DateTime.UtcNow, + EnableZip64 = true, + }); +} + +// TAR-specific options +var tarOptions = new TarWriterOptions(CompressionType.GZip, finalizeArchiveOnClose: true) +{ + HeaderFormat = TarHeaderWriteFormat.GNU_TAR_LONG_LINK, +}; + +// GZip-specific options +var gzipOptions = new GZipWriterOptions(compressionLevel: 9); + +// 7z writing requires a seekable output stream and writes non-solid archives +var sevenZipOptions = new SevenZipWriterOptions(CompressionType.LZMA2) +{ + CompressHeader = true, + LzmaProperties = new LzmaEncoderProperties(), +}; +``` + +Alternative: traditional constructor with object initializer: + +```csharp +var options = new WriterOptions(CompressionType.Deflate) +{ + CompressionLevel = 9, + LeaveStreamOpen = true, +}; +archive.SaveTo("output.zip", options); +``` + +### Extraction behavior + +```csharp +var options = new ExtractionOptions +{ + ExtractFullPath = true, // Recreate directory structure + Overwrite = true, // Overwrite existing files + PreserveFileTime = true, // Keep original timestamps + CheckCrc = true // Validate payload checksums when available +}; + +using (var archive = ZipArchive.OpenArchive("file.zip")) +{ + archive.WriteToDirectory(@"C:\output", options); +} +``` + +`CheckCrc` validates archive-level payload checksums when the format stores reliable metadata, such as ZIP CRC32 values. Formats without payload checksums skip this validation. Decompressor integrity checks that are required to decode a stream may still fail even when `CheckCrc` is disabled. + +### Options matrix + +```text +ReaderOptions: open-time behavior (password, encoding, stream ownership) +ExtractionOptions: extract-time behavior (overwrite, paths, timestamps, attributes, symlinks) +WriterOptions: write-time behavior (compression type/level, encoding, stream ownership) +ZipWriterEntryOptions: per-entry ZIP overrides (compression, level, timestamps, comments, zip64) +``` + +### Compression Providers + +`ReaderOptions` and `WriterOptions` expose a `Providers` registry that controls which `ICompressionProvider` implementations are used for each `CompressionType`. The registry defaults to `CompressionProviderRegistry.Default`, so you only need to set it if you want to swap in a custom provider (for example the `SystemGZipCompressionProvider` or `SystemDeflateCompressionProvider`). The selected registry is honored by Reader/Writer APIs, Archive APIs, and async entry-stream extraction paths. + +```csharp +var registry = CompressionProviderRegistry.Default + .With(new SystemGZipCompressionProvider()) + .With(new SystemDeflateCompressionProvider()); +var readerOptions = ReaderOptions.ForFilePath.WithProviders(registry); +var writerOptions = new WriterOptions(CompressionType.GZip) +{ + CompressionLevel = 6, +}.WithProviders(registry); + +using var reader = ReaderFactory.OpenReader(input, readerOptions); +using var writer = WriterFactory.OpenWriter(output, ArchiveType.GZip, writerOptions); +``` + +Registry API summary: + +```text +CompressionProviderRegistry.Default: built-in provider registry +CompressionProviderRegistry.Empty: empty registry, primarily useful for tests +With(provider): returns a new registry with that provider added or replaced +GetProvider(type): returns the registered ICompressionProvider, or null +CreateCompressStream(...): creates a compression stream or throws if unsupported +CreateDecompressStream(...): creates a decompression stream or throws if unsupported +CreateCompressStreamAsync(...): async stream creation counterpart +CreateDecompressStreamAsync(...): async stream creation counterpart +GetCompressingProvider(type): returns ICompressionProviderHooks when available +``` + +`ICompressionProvider` implementations declare their `CompressionType`, whether they support compression/decompression, and create sync/async streams. For simple providers, derive from `CompressionProviderBase`; it supplies default async implementations that delegate to the synchronous methods. For read-only codecs, derive from `DecompressionOnlyProviderBase`. + +```csharp +public sealed class CustomGZipProvider : CompressionProviderBase +{ + public override CompressionType CompressionType => CompressionType.GZip; + public override bool SupportsCompression => true; + public override bool SupportsDecompression => true; + + public override Stream CreateCompressStream(Stream destination, int compressionLevel) => + new GZipStream(destination, CompressionMode.Compress, leaveOpen: false); + + public override Stream CreateDecompressStream(Stream source) => + new GZipStream(source, CompressionMode.Decompress, leaveOpen: false); +} +``` + +`CompressionContext` carries metadata that providers may need when creating streams, including `InputSize`, `OutputSize`, `Properties`, `CanSeek`, `FormatOptions`, and `ReaderOptions`. Use `CompressionContext.FromStream(stream)` to populate stream-derived values, `WithReaderOptions(...)` to attach reader metadata, and `ResolveArchiveEncoding()` to get the archive header encoding from the context. + +When a format needs additional initialization/finalization data (LZMA, PPMd, etc.) the registry exposes `GetCompressingProvider` which returns the `ICompressionProviderHooks` contract; the rest of the API continues to flow through `Providers`, including pre/properties/post compression hook data. + +--- + +## Compression Types + +### Available Compressions + +```csharp +// For creating archives +CompressionType.None // No compression (store) +CompressionType.GZip // GZip wrapper/combined tar compression +CompressionType.Deflate // DEFLATE (default for ZIP/GZip) +CompressionType.Deflate64 // Deflate64 +CompressionType.BZip2 // BZip2 +CompressionType.LZMA // LZMA (for 7Zip, LZip, XZ) +CompressionType.LZMA2 // LZMA2 (for 7Zip/XZ) +CompressionType.BCJ // 7Zip branch converter filter +CompressionType.BCJ2 // 7Zip branch converter filter +CompressionType.LZip // LZip wrapper/combined tar compression +CompressionType.Xz // XZ wrapper/combined tar compression +CompressionType.PPMd // PPMd (for ZIP) +CompressionType.Rar // RAR compression (read-only) +CompressionType.Lzw // LZW, including .Z/tar.Z reading +CompressionType.Shrink // ZIP shrink (read-only) +CompressionType.Reduce1 // ZIP reduce methods (read-only) +CompressionType.Reduce2 +CompressionType.Reduce3 +CompressionType.Reduce4 +CompressionType.Explode // ZIP implode/explode (read-only) +CompressionType.Squeezed // ARC/ARJ legacy compression (read-only) +CompressionType.Packed // ARC legacy compression (read-only) +CompressionType.Crunched // ARC legacy compression (read-only) +CompressionType.Squashed // Legacy compression (read-only) +CompressionType.Crushed // Legacy compression (read-only) +CompressionType.Distilled // Legacy compression (read-only) +CompressionType.ZStandard // ZStandard +CompressionType.ArjLZ77 // ARJ compression (read-only) +CompressionType.AceLZ77 // ACE compression (read-only) +CompressionType.Unknown + +// For Tar archives with compression +// Use WriterFactory to create compressed tar archives +using (var writer = WriterFactory.OpenWriter(stream, ArchiveType.Tar, CompressionType.GZip)) // Tar.GZip +using (var writer = WriterFactory.OpenWriter(stream, ArchiveType.Tar, CompressionType.BZip2)) // Tar.BZip2 +``` + +### Archive Types + +```csharp +ArchiveType.Zip +ArchiveType.Tar +ArchiveType.GZip +ArchiveType.BZip2 +ArchiveType.Rar +ArchiveType.SevenZip +ArchiveType.Arc +ArchiveType.Arj +ArchiveType.Ace +ArchiveType.Lzw +``` + +--- + +## Patterns & Examples + +### Extract with Error Handling + +```csharp +try +{ + using (var archive = ZipArchive.OpenArchive("archive.zip", + ReaderOptions.ForEncryptedArchive("password"))) + { + archive.WriteToDirectory(@"C:\output"); + } +} +catch (PasswordRequiredException) +{ + Console.WriteLine("Password required"); +} +catch (InvalidArchiveException) +{ + Console.WriteLine("Archive is invalid"); +} +catch (SharpCompressException ex) +{ + Console.WriteLine($"Error: {ex.Message}"); +} +``` + +### Extract with Progress + +```csharp +var progress = new Progress(report => +{ + Console.WriteLine($"Extracting {report.EntryPath}: {report.PercentComplete}%"); +}); + +var options = ReaderOptions.ForFilePath.WithProgress(progress); +using (var archive = ZipArchive.OpenArchive("archive.zip", options)) +{ + archive.WriteToDirectory(@"C:\output"); +} +``` + +### Async Extract with Cancellation + +```csharp +var cts = new CancellationTokenSource(); +cts.CancelAfter(TimeSpan.FromMinutes(5)); + +try +{ + await using (var archive = await ZipArchive.OpenAsyncArchive("archive.zip")) + { + await archive.WriteToDirectoryAsync( + @"C:\output", + cancellationToken: cts.Token + ); + } +} +catch (OperationCanceledException) +{ + Console.WriteLine("Extraction cancelled"); +} +``` + +### Create with Custom Compression + +```csharp +using (var archive = ZipArchive.CreateArchive()) +{ + archive.AddAllFromDirectory(@"D:\source"); + + // Fastest + archive.SaveTo("fast.zip", new WriterOptions(CompressionType.Deflate) + { + CompressionLevel = 1 + }); + + // Balanced (default) + archive.SaveTo("normal.zip", CompressionType.Deflate); + + // Best compression + archive.SaveTo("best.zip", new WriterOptions(CompressionType.Deflate) + { + CompressionLevel = 9 + }); +} +``` + +### Stream Processing (No File I/O) + +```csharp +using (var outputStream = new MemoryStream()) +using (var archive = ZipArchive.CreateArchive()) +{ + // Add content from memory + using (var contentStream = new MemoryStream(Encoding.UTF8.GetBytes("Hello"))) + { + archive.AddEntry("file.txt", contentStream); + } + + // Save to memory + archive.SaveTo(outputStream, CompressionType.Deflate); + + // Get bytes + byte[] archiveBytes = outputStream.ToArray(); +} +``` + +### Buffered Forward-Only Streams + +`SharpCompressStream` can wrap streams with buffering for forward-only scenarios: + +```csharp +// Wrap a non-seekable stream with buffering +using (var bufferedStream = new SharpCompressStream(rawStream)) +{ + // Provides ring buffer functionality for reading ahead + // and seeking within buffered data + using (var reader = ReaderFactory.OpenReader(bufferedStream)) + { + while (reader.MoveToNextEntry()) + { + reader.WriteEntryToDirectory(@"C:\output"); + } + } +} +``` + +Useful for: +- Non-seekable streams (network streams, pipes) +- Forward-only reading with limited look-ahead +- Buffering unbuffered streams for better performance + +### Extract Specific Files + +```csharp +using (var archive = ZipArchive.OpenArchive("archive.zip")) +{ + var filesToExtract = new[] { "file1.txt", "file2.txt" }; + + foreach (var entry in archive.Entries.Where(e => filesToExtract.Contains(e.Key))) + { + entry.WriteToFile(@"C:\output\" + entry.Key); + } +} +``` + +### List Archive Contents + +```csharp +using (var archive = ZipArchive.OpenArchive("archive.zip")) +{ + foreach (var entry in archive.Entries) + { + if (entry.IsDirectory) + Console.WriteLine($"[DIR] {entry.Key}"); + else + Console.WriteLine($"[FILE] {entry.Key} ({entry.Size} bytes)"); + } +} +``` + +--- + +## Common Mistakes + +### ✗ Wrong - Stream not disposed + +```csharp +var stream = File.OpenRead("archive.zip"); +var archive = ZipArchive.OpenArchive(stream); +archive.WriteToDirectory(@"C:\output"); +// stream not disposed - leaked resource +``` + +### ✓ Correct - Using blocks + +```csharp +using (var stream = File.OpenRead("archive.zip")) +using (var archive = ZipArchive.OpenArchive(stream)) +{ + archive.WriteToDirectory(@"C:\output"); +} +// Both properly disposed +``` + +### ✗ Wrong - Mixing API styles + +```csharp +// Loading entire archive then iterating +using (var archive = ZipArchive.OpenArchive("large.zip")) +{ + var entries = archive.Entries.ToList(); // Loads all in memory + foreach (var e in entries) + { + e.WriteToFile(...); // Then extracts each + } +} +``` + +### ✓ Correct - Use Reader for large files + +```csharp +// Streaming iteration +using (var stream = File.OpenRead("large.zip")) +using (var reader = ReaderFactory.OpenReader(stream)) +{ + while (reader.MoveToNextEntry()) + { + reader.WriteEntryToDirectory(@"C:\output"); + } +} +``` + +--- + +## Related Documentation + +- [USAGE.md](USAGE.md) - Complete code examples +- [FORMATS.md](FORMATS.md) - Supported formats +- [PERFORMANCE.md](PERFORMANCE.md) - API selection guide diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 00000000..c463bf8e --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,658 @@ +# SharpCompress Architecture Guide + +This guide explains the internal architecture and design patterns of SharpCompress for contributors. + +## Overview + +SharpCompress is organized into three main layers: + +``` +┌─────────────────────────────────────────┐ +│ User-Facing APIs (Top Layer) │ +│ Archive, Reader, Writer Factories │ +├─────────────────────────────────────────┤ +│ Format-Specific Implementations │ +│ ZipArchive, TarReader, GZipWriter, │ +│ RarArchive, SevenZipArchive, etc. │ +├─────────────────────────────────────────┤ +│ Compression & Crypto (Bottom Layer) │ +│ Deflate, LZMA, BZip2, AES, CRC32 │ +└─────────────────────────────────────────┘ +``` + +--- + +## Directory Structure + +### `src/SharpCompress/` + +#### `Archives/` - Archive Implementations +Contains `IArchive` implementations for seekable, random-access APIs. + +**Key Files:** +- `AbstractArchive.cs` - Base class for all archives +- `IArchive.cs` - Archive interface definition +- `ArchiveFactory.cs` - Factory for opening archives +- Format-specific: `ZipArchive.cs`, `TarArchive.cs`, `RarArchive.cs`, `SevenZipArchive.cs`, `GZipArchive.cs` + +**Use Archive API when:** +- Stream is seekable (file, memory) +- Need random access to entries +- Archive fits in memory +- Simplicity is important + +#### `Readers/` - Reader Implementations +Contains `IReader` implementations for forward-only, non-seekable APIs. + +**Key Files:** +- `AbstractReader.cs` - Base reader class +- `IReader.cs` - Reader interface +- `ReaderFactory.cs` - Auto-detection factory +- `ReaderOptions.cs` - Configuration for readers +- Format-specific: `ZipReader.cs`, `TarReader.cs`, `GZipReader.cs`, `RarReader.cs`, etc. + +**Use Reader API when:** +- Stream is non-seekable (network, pipe, compressed) +- Processing large files +- Memory is limited +- Forward-only processing is acceptable + +#### `Writers/` - Writer Implementations +Contains `IWriter` implementations for forward-only writing. + +**Key Files:** +- `AbstractWriter.cs` - Base writer class +- `IWriter.cs` - Writer interface +- `WriterFactory.cs` - Factory for creating writers +- `WriterOptions.cs` - Configuration for writers +- Format-specific: `ZipWriter.cs`, `TarWriter.cs`, `GZipWriter.cs` + +#### `Factories/` - Format Detection +Factory classes for auto-detecting archive format and creating appropriate readers/writers. + +**Key Files:** +- `Factory.cs` - Base factory class +- `IFactory.cs` - Factory interface +- Format-specific: `ZipFactory.cs`, `TarFactory.cs`, `RarFactory.cs`, etc. + +**How It Works:** +1. `ReaderFactory.OpenReader(stream)` probes stream signatures +2. Identifies format by magic bytes +3. Creates appropriate reader instance +4. Returns generic `IReader` interface + +#### `Common/` - Shared Types +Common types, options, and enumerations used across formats. + +**Key Files:** +- `IEntry.cs` - Entry interface (file within archive) +- `Entry.cs` - Entry implementation +- `ArchiveType.cs` - Enum for archive formats +- `CompressionType.cs` - Enum for compression methods +- `ArchiveEncoding.cs` - Character encoding configuration +- `IExtractionOptions.cs` - Interface for extraction configuration +- `ExtractionOptions.cs` - Extraction behavior options for file extraction APIs +- Format-specific headers: `Zip/Headers/`, `Tar/Headers/`, `Rar/Headers/`, etc. + +#### `Compressors/` - Compression Algorithms +Low-level compression streams implementing specific algorithms. + +**Algorithms:** +- `Deflate/` - DEFLATE compression (Zip default) +- `BZip2/` - BZip2 compression +- `LZMA/` - LZMA compression (7Zip, XZ, LZip) +- `PPMd/` - Prediction by Partial Matching (Zip, 7Zip) +- `ZStandard/` - ZStandard compression (decompression only) +- `Xz/` - XZ format (decompression only) +- `Rar/` - RAR-specific unpacking +- `Arj/`, `Arc/`, `Ace/` - Legacy format decompression +- `Filters/` - BCJ/BCJ2 filters for executable compression + +**Each Compressor:** +- Implements a `Stream` subclass +- Provides both compression and decompression +- Some are read-only (decompression only) + +#### `Crypto/` - Encryption & Hashing +Cryptographic functions and stream wrappers. + +**Key Files:** +- `Crc32Stream.cs` - CRC32 calculation wrapper +- `BlockTransformer.cs` - Block cipher transformations +- AES, PKWare, WinZip encryption implementations + +#### `IO/` - Stream Utilities +Stream wrappers and utilities. + +**Key Classes:** +- `SharpCompressStream` - Base stream class +- `ProgressReportingStream` - Progress tracking wrapper +- `MarkingBinaryReader` - Binary reader with position marks +- `BufferedSubStream` - Buffered read-only substream +- `ReadOnlySubStream` - Read-only view of parent stream +- `NonDisposingStream` - Prevents wrapped stream disposal + +--- + +## Design Patterns + +### 1. Factory Pattern + +**Purpose:** Auto-detect format and create appropriate reader/writer. + +**Example:** +```csharp +// User calls factory +using (var reader = ReaderFactory.OpenReader(stream)) // Returns IReader +{ + while (reader.MoveToNextEntry()) + { + // Process entry + } +} + +// Behind the scenes: +// 1. Factory.Open() probes stream signatures +// 2. Detects format (Zip, Tar, Rar, etc.) +// 3. Creates appropriate reader (ZipReader, TarReader, etc.) +// 4. Returns as generic IReader interface +``` + +**Files:** +- `src/SharpCompress/Factories/ReaderFactory.cs` +- `src/SharpCompress/Factories/WriterFactory.cs` +- `src/SharpCompress/Factories/ArchiveFactory.cs` + +### 2. Strategy Pattern + +**Purpose:** Encapsulate compression algorithms as swappable strategies. + +**Example:** +```csharp +// Different compression strategies +CompressionType.Deflate // DEFLATE +CompressionType.BZip2 // BZip2 +CompressionType.LZMA // LZMA +CompressionType.PPMd // PPMd + +// Writer uses strategy pattern +var archive = ZipArchive.CreateArchive(); +archive.SaveTo("output.zip", CompressionType.Deflate); // Use Deflate +archive.SaveTo("output.bz2", CompressionType.BZip2); // Use BZip2 +``` + +**Files:** +- `src/SharpCompress/Compressors/` - Strategy implementations + +### 3. Decorator Pattern + +**Purpose:** Wrap streams with additional functionality. + +**Example:** +```csharp +// Progress reporting decorator +var progressStream = new ProgressReportingStream(baseStream, progressReporter); +progressStream.Read(buffer, 0, buffer.Length); // Reports progress + +// Non-disposing decorator +var nonDisposingStream = new NonDisposingStream(baseStream); +using (var compressor = new DeflateStream(nonDisposingStream)) +{ + // baseStream won't be disposed when compressor is disposed +} +``` + +**Files:** +- `src/SharpCompress/IO/ProgressReportingStream.cs` +- `src/SharpCompress/IO/NonDisposingStream.cs` + +### 4. Template Method Pattern + +**Purpose:** Define algorithm skeleton in base class, let subclasses fill details. + +**Example:** +```csharp +// AbstractArchive defines common archive operations +public abstract class AbstractArchive : IArchive +{ + // Template methods + public virtual void WriteToDirectory(string destinationDirectory) + { + // Common extraction logic + foreach (var entry in Entries) + { + // Call subclass method + entry.WriteToFile(destinationPath); + } + } + + // Subclasses override format-specific details + protected abstract Entry CreateEntry(EntryData data); +} +``` + +**Files:** +- `src/SharpCompress/Archives/AbstractArchive.cs` +- `src/SharpCompress/Readers/AbstractReader.cs` + +### 5. Iterator Pattern + +**Purpose:** Provide sequential access to entries. + +**Example:** +```csharp +// Archive API - provides collection +IEnumerable entries = archive.Entries; +foreach (var entry in entries) +{ + // Random access - entries already in memory +} + +// Reader API - provides iterator +IReader reader = ReaderFactory.OpenReader(stream); +while (reader.MoveToNextEntry()) +{ + // Forward-only iteration - one entry at a time + var entry = reader.Entry; +} +``` + +--- + +## Key Interfaces + +### IArchive - Random Access API + +```csharp +public interface IArchive : IDisposable +{ + IEnumerable Entries { get; } + + void WriteToDirectory(string destinationDirectory); + + IEntry FirstOrDefault(Func predicate); + + // ... format-specific methods +} +``` + +**Implementations:** `ZipArchive`, `TarArchive`, `RarArchive`, `SevenZipArchive`, `GZipArchive` + +### IReader - Forward-Only API + +```csharp +public interface IReader : IDisposable +{ + IEntry Entry { get; } + + bool MoveToNextEntry(); + + void WriteEntryToDirectory(string destinationDirectory); + + Stream OpenEntryStream(); + + // ... async variants +} +``` + +**Implementations:** `ZipReader`, `TarReader`, `RarReader`, `GZipReader`, etc. + +### IWriter - Writing API + +```csharp +public interface IWriter : IDisposable +{ + void Write(string entryPath, Stream source, + DateTime? modificationTime = null); + + void WriteAll(string sourceDirectory, string searchPattern, + SearchOption searchOption); + + // ... async variants +} +``` + +**Implementations:** `ZipWriter`, `TarWriter`, `GZipWriter` + +### IEntry - Archive Entry + +```csharp +public interface IEntry +{ + string Key { get; } + uint Size { get; } + uint CompressedSize { get; } + bool IsDirectory { get; } + DateTime? LastModifiedTime { get; } + CompressionType CompressionType { get; } + + void WriteToFile(string fullPath); + void WriteToStream(Stream destinationStream); + Stream OpenEntryStream(); + + // ... async variants +} +``` + +--- + +## Adding Support for a New Format + +### Step 1: Understand the Format +- Research format specification +- Understand compression/encryption used +- Study existing similar formats in codebase + +### Step 2: Create Format Structure Classes + +**Create:** `src/SharpCompress/Common/NewFormat/` + +```csharp +// Headers and data structures +public class NewFormatHeader +{ + public uint Magic { get; set; } + public ushort Version { get; set; } + // ... other fields + + public static NewFormatHeader Read(BinaryReader reader) + { + // Deserialize from binary + } +} + +public class NewFormatEntry +{ + public string FileName { get; set; } + public uint CompressedSize { get; set; } + public uint UncompressedSize { get; set; } + // ... other fields +} +``` + +### Step 3: Create Archive Implementation + +**Create:** `src/SharpCompress/Archives/NewFormat/NewFormatArchive.cs` + +```csharp +public class NewFormatArchive : AbstractArchive +{ + private NewFormatHeader _header; + private List _entries; + + public static NewFormatArchive OpenArchive(Stream stream) + { + var archive = new NewFormatArchive(); + archive._header = NewFormatHeader.Read(stream); + archive.LoadEntries(stream); + return archive; + } + + public override IEnumerable Entries => _entries.Select(e => new Entry(e)); + + protected override Stream OpenEntryStream(Entry entry) + { + // Return decompressed stream for entry + } + + // ... other abstract method implementations +} +``` + +### Step 4: Create Reader Implementation + +**Create:** `src/SharpCompress/Readers/NewFormat/NewFormatReader.cs` + +```csharp +public class NewFormatReader : AbstractReader +{ + private NewFormatHeader _header; + private BinaryReader _reader; + + public NewFormatReader(Stream stream) + { + _reader = new BinaryReader(stream); + _header = NewFormatHeader.Read(_reader); + } + + public override bool MoveToNextEntry() + { + // Read next entry header + if (!_reader.BaseStream.CanRead) return false; + + var entryData = NewFormatEntry.Read(_reader); + // ... set this.Entry + return entryData != null; + } + + // ... other abstract method implementations +} +``` + +### Step 5: Create Factory + +**Create:** `src/SharpCompress/Factories/NewFormatFactory.cs` + +```csharp +public class NewFormatFactory : Factory, IArchiveFactory, IReaderFactory +{ + // Archive format magic bytes (signature) + private static readonly byte[] NewFormatSignature = new byte[] { 0x4E, 0x46 }; // "NF" + + public static NewFormatFactory Instance { get; } = new(); + + public IArchive CreateArchive(Stream stream) + => NewFormatArchive.OpenArchive(stream); + + public IReader CreateReader(Stream stream, ReaderOptions options) + => new NewFormatReader(stream) { Options = options }; + + public bool Matches(Stream stream, ReadOnlySpan signature) + => signature.StartsWith(NewFormatSignature); +} +``` + +### Step 6: Register Factory + +**Update:** `src/SharpCompress/Factories/ArchiveFactory.cs` + +```csharp +private static readonly IFactory[] Factories = +{ + ZipFactory.Instance, + TarFactory.Instance, + RarFactory.Instance, + SevenZipFactory.Instance, + GZipFactory.Instance, + NewFormatFactory.Instance, // Add here + // ... other factories +}; +``` + +### Step 7: Add Tests + +**Create:** `tests/SharpCompress.Test/NewFormat/NewFormatTests.cs` + +```csharp +public class NewFormatTests : TestBase +{ + [Fact] + public void NewFormat_Extracts_Successfully() + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, "archive.newformat"); + using (var archive = NewFormatArchive.OpenArchive(archivePath)) + { + archive.WriteToDirectory(SCRATCH_FILES_PATH); + // Assert extraction + } + } + + [Fact] + public void NewFormat_Reader_Works() + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, "archive.newformat"); + using (var stream = File.OpenRead(archivePath)) + using (var reader = new NewFormatReader(stream)) + { + Assert.True(reader.MoveToNextEntry()); + Assert.NotNull(reader.Entry); + } + } +} +``` + +### Step 8: Add Test Archives + +Place test files in `tests/TestArchives/Archives/NewFormat/` directory. + +### Step 9: Document + +Update `docs/FORMATS.md` with format support information. + +--- + +## Compression Algorithm Implementation + +### Creating a New Compression Stream + +**Example:** Creating `CustomStream` for a custom compression algorithm + +```csharp +public class CustomStream : Stream +{ + private readonly Stream _baseStream; + private readonly bool _leaveOpen; + + public CustomStream(Stream baseStream, bool leaveOpen = false) + { + _baseStream = baseStream; + _leaveOpen = leaveOpen; + } + + public override int Read(byte[] buffer, int offset, int count) + { + // Decompress data from _baseStream into buffer + // Return number of decompressed bytes + } + + public override void Write(byte[] buffer, int offset, int count) + { + // Compress data from buffer into _baseStream + } + + protected override void Dispose(bool disposing) + { + if (disposing && !_leaveOpen) + { + _baseStream?.Dispose(); + } + base.Dispose(disposing); + } +} +``` + +--- + +## Stream Handling Best Practices + +### Disposal Pattern + +```csharp +// Correct: Nested using blocks +using (var fileStream = File.OpenRead("archive.zip")) +using (var archive = ZipArchive.OpenArchive(fileStream)) +{ + archive.WriteToDirectory(@"C:\output"); +} +// Both archive and fileStream properly disposed + +// Correct: Using with options +var options = new ReaderOptions { LeaveStreamOpen = true }; +var stream = File.OpenRead("archive.zip"); +using (var archive = ZipArchive.OpenArchive(stream, options)) +{ + archive.WriteToDirectory(@"C:\output"); +} +stream.Dispose(); // Manually dispose if LeaveStreamOpen = true +``` + +### NonDisposingStream Wrapper + +```csharp +// Prevent unwanted stream closure +var baseStream = File.OpenRead("data.bin"); +var nonDisposing = new NonDisposingStream(baseStream); + +using (var compressor = new DeflateStream(nonDisposing)) +{ + // Compressor won't close baseStream when disposed +} + +// baseStream still usable +baseStream.Position = 0; // Works +baseStream.Dispose(); // Manual disposal +``` + +--- + +## Performance Considerations + +### Memory Efficiency + +1. **Avoid loading entire archive in memory** - Use Reader API for large files +2. **Process entries sequentially** - Especially for solid archives +3. **Use appropriate buffer sizes** - Larger buffers for network I/O +4. **Dispose streams promptly** - Free resources when done + +### Algorithm Selection + +1. **Archive API** - Fast for small archives with random access +2. **Reader API** - Efficient for large files or streaming +3. **Solid archives** - Sequential extraction much faster +4. **Compression levels** - Trade-off between speed and size + +--- + +## Testing Guidelines + +### Test Coverage + +1. **Happy path** - Normal extraction works +2. **Edge cases** - Empty archives, single file, many files +3. **Corrupted data** - Handle gracefully +4. **Error cases** - Missing passwords, unsupported compression +5. **Async operations** - Both sync and async code paths + +### Test Archives + +- Use `tests/TestArchives/` for test data +- Create format-specific subdirectories +- Include encrypted, corrupted, and edge case archives +- Don't recreate existing archives + +### Test Patterns + +```csharp +[Fact] +public void Archive_Extraction_Works() +{ + // Arrange + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "test.zip"); + + // Act + using (var archive = ZipArchive.OpenArchive(testArchive)) + { + archive.WriteToDirectory(SCRATCH_FILES_PATH); + } + + // Assert + Assert.True(File.Exists(Path.Combine(SCRATCH_FILES_PATH, "file.txt"))); +} +``` + +--- + +## Related Documentation + +- [AGENTS.md](../AGENTS.md) - Development guidelines +- [FORMATS.md](FORMATS.md) - Supported formats diff --git a/docs/ENCODING.md b/docs/ENCODING.md new file mode 100644 index 00000000..5ae0de09 --- /dev/null +++ b/docs/ENCODING.md @@ -0,0 +1,603 @@ +# SharpCompress Character Encoding Guide + +This guide explains how SharpCompress handles character encoding for archive entries (filenames, comments, etc.). + +## Overview + +Most archive formats store filenames and metadata as bytes. SharpCompress must convert these bytes to strings using the appropriate character encoding. + +**Common Problem:** Archives created on systems with non-UTF8 encodings (especially Japanese, Chinese systems) appear with corrupted filenames when extracted on systems that assume UTF8. + +--- + +## ArchiveEncoding Class + +### Basic Usage + +```csharp +using SharpCompress.Common; +using SharpCompress.Readers; + +// Configure encoding using fluent factory method (preferred) +var options = ReaderOptions.ForEncoding( + new ArchiveEncoding { Default = Encoding.GetEncoding(932) }); // cp932 for Japanese + +using (var archive = ZipArchive.OpenArchive("japanese.zip", options)) +{ + foreach (var entry in archive.Entries) + { + Console.WriteLine(entry.Key); // Now shows correct characters + } +} + +// Alternative: object initializer +var options2 = new ReaderOptions +{ + ArchiveEncoding = new ArchiveEncoding { Default = Encoding.GetEncoding(932) } +}; +``` + +### ArchiveEncoding Properties + +| Property | Purpose | +|----------|---------| +| `Default` | Default encoding for filenames (fallback) | +| `CustomDecoder` | Custom decoding function for special cases | + +### Setting for Different APIs + +**Archive API:** +```csharp +var options = ReaderOptions.ForEncoding( + new ArchiveEncoding { Default = Encoding.GetEncoding(932) }); +using (var archive = ZipArchive.OpenArchive("file.zip", options)) +{ + // Use archive with correct encoding +} +``` + +**Reader API:** +```csharp +var options = ReaderOptions.ForEncoding( + new ArchiveEncoding { Default = Encoding.GetEncoding(932) }); +using (var stream = File.OpenRead("file.zip")) +using (var reader = ReaderFactory.OpenReader(stream, options)) +{ + while (reader.MoveToNextEntry()) + { + // Filenames decoded correctly + } +} +``` + +--- + +## Common Encodings + +### Asian Encodings + +#### cp932 (Japanese) +```csharp +// Windows-31J, Shift-JIS variant used on Japanese Windows +var options = new ReaderOptions +{ + ArchiveEncoding = new ArchiveEncoding + { + Default = Encoding.GetEncoding(932) + } +}; +using (var archive = ZipArchive.OpenArchive("japanese.zip", options)) +{ + // Correctly decodes Japanese filenames +} +``` + +**When to use:** +- Archives from Japanese Windows systems +- Files with Japanese characters in names + +#### gb2312 (Simplified Chinese) +```csharp +// Simplified Chinese +var options = new ReaderOptions +{ + ArchiveEncoding = new ArchiveEncoding + { + Default = Encoding.GetEncoding("gb2312") + } +}; +``` + +#### gbk (Extended Simplified Chinese) +```csharp +// Extended Simplified Chinese (more characters than gb2312) +var options = new ReaderOptions +{ + ArchiveEncoding = new ArchiveEncoding + { + Default = Encoding.GetEncoding("gbk") + } +}; +``` + +#### big5 (Traditional Chinese) +```csharp +// Traditional Chinese (Taiwan, Hong Kong) +var options = new ReaderOptions +{ + ArchiveEncoding = new ArchiveEncoding + { + Default = Encoding.GetEncoding("big5") + } +}; +``` + +#### euc-jp (Japanese, Unix) +```csharp +// Extended Unix Code for Japanese +var options = new ReaderOptions +{ + ArchiveEncoding = new ArchiveEncoding + { + Default = Encoding.GetEncoding("eucjp") + } +}; +``` + +#### euc-kr (Korean) +```csharp +// Extended Unix Code for Korean +var options = new ReaderOptions +{ + ArchiveEncoding = new ArchiveEncoding + { + Default = Encoding.GetEncoding("euc-kr") + } +}; +``` + +### Western European Encodings + +#### iso-8859-1 (Latin-1) +```csharp +// Western European (includes accented characters) +var options = new ReaderOptions +{ + ArchiveEncoding = new ArchiveEncoding + { + Default = Encoding.GetEncoding("iso-8859-1") + } +}; +``` + +**When to use:** +- Archives from French, German, Spanish systems +- Files with accented characters (é, ñ, ü, etc.) + +#### cp1252 (Windows-1252) +```csharp +// Windows Western European +// Very similar to iso-8859-1 but with additional printable characters +var options = new ReaderOptions +{ + ArchiveEncoding = new ArchiveEncoding + { + Default = Encoding.GetEncoding("cp1252") + } +}; +``` + +**When to use:** +- Archives from older Western European Windows systems +- Files with smart quotes and other Windows-specific characters + +#### iso-8859-15 (Latin-9) +```csharp +// Western European with Euro symbol support +var options = new ReaderOptions +{ + ArchiveEncoding = new ArchiveEncoding + { + Default = Encoding.GetEncoding("iso-8859-15") + } +}; +``` + +### Cyrillic Encodings + +#### cp1251 (Windows Cyrillic) +```csharp +// Russian, Serbian, Bulgarian, etc. +var options = new ReaderOptions +{ + ArchiveEncoding = new ArchiveEncoding + { + Default = Encoding.GetEncoding("cp1251") + } +}; +``` + +#### koi8-r (KOI8 Russian) +```csharp +// Russian (Unix standard) +var options = new ReaderOptions +{ + ArchiveEncoding = new ArchiveEncoding + { + Default = Encoding.GetEncoding("koi8-r") + } +}; +``` + +### UTF Encodings (Modern) + +#### UTF-8 (Default) +```csharp +// Modern standard - usually correct for new archives +var options = new ReaderOptions +{ + ArchiveEncoding = new ArchiveEncoding + { + Default = Encoding.UTF8 + } +}; +``` + +#### UTF-16 +```csharp +// Unicode - rarely used in archives +var options = new ReaderOptions +{ + ArchiveEncoding = new ArchiveEncoding + { + Default = Encoding.Unicode + } +}; +``` + +--- + +## Encoding Auto-Detection + +SharpCompress attempts to auto-detect encoding, but this isn't always reliable: + +```csharp +// Auto-detection (default) +using (var archive = ZipArchive.OpenArchive("file.zip")) // Uses UTF8 by default +{ + // May show corrupted characters if archive uses different encoding +} + +// Explicit encoding (more reliable) +var options = new ReaderOptions +{ + ArchiveEncoding = new ArchiveEncoding { Default = Encoding.GetEncoding(932) } +}; +using (var archive = ZipArchive.OpenArchive("file.zip", options)) +{ + // Correct characters displayed +} +``` + +### When Manual Override is Needed + +| Situation | Solution | +|-----------|----------| +| Archive shows corrupted characters | Specify the encoding explicitly | +| Archives from specific region | Use that region's encoding | +| Mixed encodings in archive | Use CustomDecoder | +| Testing with international files | Try different encodings | + +--- + +## Custom Decoder + +For complex scenarios where a single encoding isn't sufficient: + +### Basic Custom Decoder + +```csharp +var options = new ReaderOptions +{ + ArchiveEncoding = new ArchiveEncoding + { + CustomDecoder = (data, offset, length) => + { + // Custom decoding logic + var bytes = new byte[length]; + Array.Copy(data, offset, bytes, 0, length); + + // Try UTF8 first + try + { + return Encoding.UTF8.GetString(bytes); + } + catch + { + // Fallback to cp932 if UTF8 fails + return Encoding.GetEncoding(932).GetString(bytes); + } + } + } +}; + +using (var archive = ZipArchive.OpenArchive("mixed.zip", options)) +{ + foreach (var entry in archive.Entries) + { + Console.WriteLine(entry.Key); // Uses custom decoder + } +} +``` + +### Advanced: Detect Encoding by Content + +```csharp +var options = new ReaderOptions +{ + ArchiveEncoding = new ArchiveEncoding + { + CustomDecoder = DetectAndDecode + } +}; + +private static string DetectAndDecode(byte[] data, int offset, int length) +{ + var bytes = new byte[length]; + Array.Copy(data, offset, bytes, 0, length); + + // Try UTF8 (most modern archives) + try + { + var str = Encoding.UTF8.GetString(bytes); + // Verify it decoded correctly (no replacement characters) + if (!str.Contains('\uFFFD')) + return str; + } + catch { } + + // Try cp932 (Japanese) + try + { + var str = Encoding.GetEncoding(932).GetString(bytes); + if (!str.Contains('\uFFFD')) + return str; + } + catch { } + + // Fallback to iso-8859-1 (always succeeds) + return Encoding.GetEncoding("iso-8859-1").GetString(bytes); +} +``` + +--- + +## Code Examples + +### Extract Archive with Japanese Filenames + +```csharp +var options = new ReaderOptions +{ + ArchiveEncoding = new ArchiveEncoding + { + Default = Encoding.GetEncoding(932) // cp932 + } +}; + +using (var archive = ZipArchive.OpenArchive("japanese_files.zip", options)) +{ + archive.WriteToDirectory(@"C:\output"); +} +// Files extracted with correct Japanese names +``` + +### Extract Archive with Western European Filenames + +```csharp +var options = new ReaderOptions +{ + ArchiveEncoding = new ArchiveEncoding + { + Default = Encoding.GetEncoding("iso-8859-1") + } +}; + +using (var archive = ZipArchive.OpenArchive("french_files.zip", options)) +{ + archive.WriteToDirectory(@"C:\output"); +} +// Accented characters (é, è, ê, etc.) display correctly +``` + +### Extract Archive with Chinese Filenames + +```csharp +var options = new ReaderOptions +{ + ArchiveEncoding = new ArchiveEncoding + { + Default = Encoding.GetEncoding("gbk") // Simplified Chinese + } +}; + +using (var archive = ZipArchive.OpenArchive("chinese_files.zip", options)) +{ + archive.WriteToDirectory(@"C:\output"); +} +``` + +### Extract Archive with Russian Filenames + +```csharp +var options = new ReaderOptions +{ + ArchiveEncoding = new ArchiveEncoding + { + Default = Encoding.GetEncoding("cp1251") // Windows Cyrillic + } +}; + +using (var archive = ZipArchive.OpenArchive("russian_files.zip", options)) +{ + archive.WriteToDirectory(@"C:\output"); +} +``` + +### Reader API with Encoding + +```csharp +var options = new ReaderOptions +{ + ArchiveEncoding = new ArchiveEncoding + { + Default = Encoding.GetEncoding(932) + } +}; + +using (var stream = File.OpenRead("japanese.zip")) +using (var reader = ReaderFactory.OpenReader(stream, options)) +{ + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + Console.WriteLine(reader.Entry.Key); // Correct characters + reader.WriteEntryToDirectory(@"C:\output"); + } + } +} +``` + +--- + +## Creating Archives with Correct Encoding + +When creating archives, SharpCompress uses UTF8 by default (recommended): + +```csharp +// Create with UTF8 (default, recommended) +using (var archive = ZipArchive.CreateArchive()) +{ + archive.AddAllFromDirectory(@"D:\my_files"); + archive.SaveTo("output.zip", CompressionType.Deflate); + // Archives created with UTF8 encoding +} +``` + +If you need to create archives for systems that expect specific encodings: + +```csharp +// Note: SharpCompress Writer API uses UTF8 for encoding +// To create archives with other encodings, consider: +// 1. Let users on those systems create archives +// 2. Use system tools (7-Zip, WinRAR) with desired encoding +// 3. Post-process archives if absolutely necessary + +// For now, recommend modern UTF8-based archives +``` + +--- + +## Troubleshooting Encoding Issues + +### Filenames Show Question Marks (?) + +``` +✗ Wrong encoding detected +test文件.txt → test???.txt +``` + +**Solution:** Specify correct encoding explicitly + +```csharp +var options = new ReaderOptions +{ + ArchiveEncoding = new ArchiveEncoding + { + Default = Encoding.GetEncoding("gbk") // Try different encodings + } +}; +``` + +### Filenames Show Replacement Character (￿) + +``` +✗ Invalid bytes for selected encoding +café.txt → caf￿.txt +``` + +**Solution:** +1. Try a different encoding (see Common Encodings table) +2. Use CustomDecoder with fallback encoding +3. Archive might be corrupted + +### Mixed Encodings in Single Archive + +```csharp +// Use CustomDecoder to handle mixed encodings +var options = new ReaderOptions +{ + ArchiveEncoding = new ArchiveEncoding + { + CustomDecoder = (data, offset, length) => + { + // Try multiple encodings in priority order + var bytes = new byte[length]; + Array.Copy(data, offset, bytes, 0, length); + + foreach (var encoding in new[] + { + Encoding.UTF8, + Encoding.GetEncoding(932), + Encoding.GetEncoding("iso-8859-1") + }) + { + try + { + var str = encoding.GetString(bytes); + if (!str.Contains('\uFFFD')) + return str; + } + catch { } + } + + // Final fallback + return Encoding.GetEncoding("iso-8859-1").GetString(bytes); + } + } +}; +``` + +--- + +## Encoding Reference Table + +| Encoding | Code | Use Case | +|----------|------|----------| +| UTF-8 | (default) | Modern archives, recommended | +| cp932 | 932 | Japanese Windows | +| gb2312 | "gb2312" | Simplified Chinese | +| gbk | "gbk" | Extended Simplified Chinese | +| big5 | "big5" | Traditional Chinese | +| iso-8859-1 | "iso-8859-1" | Western European | +| cp1252 | "cp1252" | Windows Western European | +| cp1251 | "cp1251" | Russian/Cyrillic | +| euc-jp | "euc-jp" | Japanese Unix | +| euc-kr | "euc-kr" | Korean | + +--- + +## Best Practices + +1. **Use UTF-8 for new archives** - Most modern systems support it +2. **Ask the archive creator** - When receiving archives with corrupted names +3. **Provide encoding options** - If your app handles user archives +4. **Document your assumption** - Tell users what encoding you're using +5. **Test with international files** - Before releasing production code + +--- + +## Related Documentation + +- [USAGE.md](USAGE.md#extract-zip-which-has-non-utf8-encoded-filenamycp932) - Usage examples diff --git a/docs/FORMATS.md b/docs/FORMATS.md new file mode 100644 index 00000000..3504d659 --- /dev/null +++ b/docs/FORMATS.md @@ -0,0 +1,94 @@ +# Formats + +## Accessing Archives + +* Archive classes allow random access to a seekable stream. +* Reader classes allow forward-only reading on a stream. +* Writer classes allow forward-only Writing on a stream. + +## Supported Format Table + +| Archive Format | Compression Format(s) | Compress/Decompress | Archive API | Reader API | Writer API | +| ------------------ | ------------------------------------------------------------------- | ------------------- | --------------- | ---------- | --------------- | +| Ace | None | Decompress | N/A | AceReader | N/A | +| Arc | None, Packed, Squeezed, Crunched | Decompress | N/A | ArcReader | N/A | +| Arj | None | Decompress | N/A | ArjReader | N/A | +| Rar | Rar | Decompress | RarArchive | RarReader | N/A | +| Zip (2) | None, Shrink, Reduce, Implode, DEFLATE, Deflate64, BZip2, LZMA, PPMd, ZStandard, XZ | Both | ZipArchive | ZipReader | ZipWriter | +| Tar | None | Both | TarArchive | TarReader | TarWriter (3) | +| Tar.GZip | DEFLATE | Both | TarArchive | TarReader | TarWriter (3) | +| Tar.BZip2 | BZip2 | Both | TarArchive | TarReader | TarWriter (3) | +| Tar.Zstandard | ZStandard | Decompress | TarArchive | TarReader | N/A | +| Tar.LZip | LZMA | Both | TarArchive | TarReader | TarWriter (3) | +| Tar.XZ | LZMA2 | Decompress | TarArchive | TarReader | N/A | +| Tar.LZW | LZW | Decompress | TarArchive | TarReader | N/A | +| GZip (single file) | DEFLATE | Both | GZipArchive | GZipReader | GZipWriter | +| 7Zip (4) | LZMA, LZMA2, BZip2, PPMd, BCJ, BCJ2, Deflate | Both | SevenZipArchive | N/A | SevenZipWriter | + +1. SOLID Rars are only supported in the RarReader API. +2. Zip format supports pkware and WinzipAES encryption. However, encrypted LZMA is not supported. Zip64 reading/writing is supported but only with seekable streams as the Zip spec doesn't support Zip64 data in post data descriptors. Deflate64, Shrink, Reduce, Implode, and XZ are only supported for reading. ZStandard is supported for reading and writing. See [Zip Format Notes](#zip-format-notes) for details on multi-volume archives and streaming behavior. +3. The Tar format requires a file size in the header. If no size is specified to the TarWriter and the stream is not seekable, then an exception will be thrown. +4. The 7Zip format doesn't allow for reading as a forward-only stream, so 7Zip read support is only through the Archive API. Writing is supported through SevenZipWriter for non-solid archives with LZMA/LZMA2 and requires a seekable output stream. See [7Zip Format Notes](#7zip-format-notes) for details on async extraction behavior. +5. LZip has no support for extra data like the file name or timestamp. There is a default filename used when looking at the entry Key on the archive. + +`ArchiveFactory.GetArchiveInformation(...).SupportsRandomAccess` is `true` when the detected format has an Archive API in this table. It is `false` for reader-only formats such as Ace, Arc, Arj, and standalone LZW. + +### Zip Format Notes + +- Multi-volume/split ZIP archives require ZipArchive (seekable streams) as ZipReader cannot seek across volume files. +- ZipReader processes entries from LocalEntry headers (which include directory entries ending with `/`) and intentionally skips DirectoryEntry headers from the central directory, as they are redundant in streaming mode - all entry data comes from LocalEntry headers which ZipReader has already processed. +- ZIP supports ZStandard for reading and writing. Tar.Zstandard support is read/decompress only. + +### 7Zip Format Notes + +- **Async Extraction Performance**: When using async extraction methods (e.g., `ExtractAllEntries()` with `MoveToNextEntryAsync()`), each file creates its own decompression stream to avoid state corruption in the LZMA decoder. This is less efficient than synchronous extraction, which can reuse a single decompression stream for multiple files in the same folder. + + **Performance Impact**: For archives with many small files in the same compression folder, async extraction will be slower than synchronous extraction because it must: + 1. Create a new LZMA decoder for each file + 2. Skip through the decompressed data to reach each file's starting position + + **Recommendation**: For best performance with 7Zip archives, use synchronous extraction methods (`MoveToNextEntry()` and `WriteEntryToDirectory()`) when possible. Use async methods only when you need to avoid blocking the thread (e.g., in UI applications or async-only contexts). + + **Technical Details**: 7Zip archives group files into "folders" (compression units), where all files in a folder share one continuous LZMA-compressed stream. The LZMA decoder maintains internal state (dictionary window, decoder positions) that assumes sequential, non-interruptible processing. Async operations can yield control during awaits, which would corrupt this shared state. To avoid this, async extraction creates a fresh decoder stream for each file. + +### XZ Format Notes + +- XZ is a container format around LZMA2-compressed blocks, not just raw LZMA/LZMA2 data. +- XZ streams can include per-block integrity checks selected by the stream header: CRC32, CRC64/XZ, SHA-256, or none. SharpCompress validates these checks while reading XZ blocks. +- Raw LZMA/LZMA2 decoding does not provide the same container-level CRC validation; it only validates what the decoder format itself can detect, such as malformed compressed data or invalid end markers. + +## Compression Streams + +For those who want to directly compress/decompress bits. The single file formats are represented here as well. However, BZip2, LZip and XZ have no metadata (GZip has a little) so using them without something like a Tar file makes little sense. + +| Compressor | Compress/Decompress | +| --------------- | ------------------- | +| BZip2Stream | Both | +| GZipStream | Both | +| DeflateStream | Both | +| Deflate64Stream | Decompress | +| LZMAStream | Both | +| PPMdStream | Both | +| LzwStream | Decompress | +| ADCStream | Decompress | +| LZipStream | Both | +| XZStream | Decompress | +| ZStandard CompressionStream/DecompressionStream | Both | + +## Archive Formats vs Compression + +Sometimes the terminology gets mixed. + +### Compression + +DEFLATE, LZMA are pure compression algorithms + +### Formats + +Formats like Zip, 7Zip, Rar are archive formats only. They use other compression methods (e.g. DEFLATE, LZMA, etc.) or propriatory (e.g RAR) + +### Overlap + +GZip, BZip2, LZip, XZ, LZW, and ZStandard are single file or compression wrapper formats. The overlap in the API happens because Tar uses these formats as "compression" methods and the API tries to hide this a bit. + +`ArchiveType` represents archive containers exposed by the high-level APIs (`Rar`, `Zip`, `Tar`, `SevenZip`, `GZip`, `Arc`, `Arj`, `Ace`, and `Lzw`). `XZ` and `ZStandard` are represented as `CompressionType` values rather than `ArchiveType` values. diff --git a/docs/OLD_CHANGELOG.md b/docs/OLD_CHANGELOG.md new file mode 100644 index 00000000..fbbdd7ae --- /dev/null +++ b/docs/OLD_CHANGELOG.md @@ -0,0 +1,142 @@ + +# Version Log + +* [Releases](https://github.com/adamhathcock/sharpcompress/releases) + +## Version 0.18 + +* [Now on Github releases](https://github.com/adamhathcock/sharpcompress/releases/tag/0.18) + +## Version 0.17.1 + +* Fix - [Bug Fix for .NET Core on Windows](https://github.com/adamhathcock/sharpcompress/pull/257) + +## Version 0.17.0 + +* New - Full LZip support! Can read and write LZip files and Tars inside LZip files. [Make LZip a first class citizen. #241](https://github.com/adamhathcock/sharpcompress/issues/241) +* New - XZ read support! Can read XZ files and Tars inside XZ files. [XZ in SharpCompress #91](https://github.com/adamhathcock/sharpcompress/issues/94) +* Fix - [Regression - zip file writing on seekable streams always assumed stream start was 0. Introduced with Zip64 writing.](https://github.com/adamhathcock/sharpcompress/issues/244) +* Fix - [Zip files with post-data descriptors can be properly skipped via decompression](https://github.com/adamhathcock/sharpcompress/issues/162) + +## Version 0.16.2 + +* Fix [.NET 3.5 should support files and cryptography (was a regression from 0.16.0)](https://github.com/adamhathcock/sharpcompress/pull/251) +* Fix [Zip per entry compression customization wrote the wrong method into the zip archive](https://github.com/adamhathcock/sharpcompress/pull/249) + +## Version 0.16.1 + +* Fix [Preserve compression method when getting a compressed stream](https://github.com/adamhathcock/sharpcompress/pull/235) +* Fix [RAR entry key normalization fix](https://github.com/adamhathcock/sharpcompress/issues/201) + +## Version 0.16.0 + +* Breaking - [Progress Event Tracking rethink](https://github.com/adamhathcock/sharpcompress/pull/226) +* Update to VS2017 - [VS2017](https://github.com/adamhathcock/sharpcompress/pull/231) - Framework targets have been changed. +* New - [Add Zip64 writing](https://github.com/adamhathcock/sharpcompress/pull/211) +* [Fix invalid/mismatching Zip version flags.](https://github.com/adamhathcock/sharpcompress/issues/164) - This allows nuget/System.IO.Packaging to read zip files generated by SharpCompress +* [Fix 7Zip directory hiding](https://github.com/adamhathcock/sharpcompress/pull/215/files) +* [Verify RAR CRC headers](https://github.com/adamhathcock/sharpcompress/pull/220) + +## Version 0.15.2 + +* [Fix invalid headers](https://github.com/adamhathcock/sharpcompress/pull/210) - fixes an issue creating large-ish zip archives that was introduced with zip64 reading. + +## Version 0.15.1 + +* [Zip64 extending information and ZipReader](https://github.com/adamhathcock/sharpcompress/pull/206) + +## Version 0.15.0 + +* [Add zip64 support for ZipArchive extraction](https://github.com/adamhathcock/sharpcompress/pull/205) + +## Version 0.14.1 + +* [.NET Assemblies aren't strong named](https://github.com/adamhathcock/sharpcompress/issues/158) +* [Pkware encryption for Zip files didn't allow for multiple reads of an entry](https://github.com/adamhathcock/sharpcompress/issues/197) +* [GZip Entry couldn't be read multiple times](https://github.com/adamhathcock/sharpcompress/issues/198) + +## Version 0.14.0 + +* [Support for LZip reading in for Tars](https://github.com/adamhathcock/sharpcompress/pull/191) + +## Version 0.13.1 + +* [Fix null password on ReaderFactory. Fix null options on SevenZipArchive](https://github.com/adamhathcock/sharpcompress/pull/188) +* [Make PpmdProperties lazy to avoid unnecessary allocations.](https://github.com/adamhathcock/sharpcompress/pull/185) + +## Version 0.13.0 + +* Breaking change: Big refactor of Options on API. +* 7Zip supports Deflate + +## Version 0.12.4 + +* Forward only zip issue fix https://github.com/adamhathcock/sharpcompress/issues/160 +* Try to fix frameworks again by copying targets from JSON.NET + +## Version 0.12.3 + +* 7Zip fixes https://github.com/adamhathcock/sharpcompress/issues/73 +* Maybe all profiles will work with project.json now + +## Version 0.12.2 + +* Support Profile 259 again + +## Version 0.12.1 + +* Support Silverlight 5 + +## Version 0.12.0 + +* .NET Core RTM! +* Bug fix for Tar long paths + +## Version 0.11.6 + +* Bug fix for global header in Tar +* Writers now have a leaveOpen `bool` overload. They won't close streams if not-requested to. + +## Version 0.11.5 + +* Bug fix in Skip method + +## Version 0.11.4 + +* SharpCompress is now endian neutral (matters for Mono platforms) +* Fix for Inflate (need to change implementation) +* Fixes for RAR detection + +## Version 0.11.1 + +* Added Cancel on IReader +* Removed .NET 2.0 support and LinqBridge dependency + +## Version 0.11 + +* Been over a year, contains mainly fixes from contributors! +* Possible breaking change: ArchiveEncoding is UTF8 by default now. +* TAR supports writing long names using longlink +* RAR Protect Header added + +## Version 0.10.3 + +* Finally fixed Disposal issue when creating a new archive with the Archive API + +## Version 0.10.2 + +* Fixed Rar Header reading for invalid extended time headers. +* Windows Store assembly is now strong named +* Known issues with Long Tar names being worked on +* Updated to VS2013 +* Portable targets SL5 and Windows Phone 8 (up from SL4 and WP7) + +## Version 0.10.1 + +* Fixed 7Zip extraction performance problem + +## Version 0.10: + +* Added support for RAR Decryption (thanks to https://github.com/hrasyid) +* Embedded some BouncyCastle crypto classes to allow RAR Decryption and Winzip AES Decryption in Portable and Windows Store DLLs +* Built in Release (I think) diff --git a/docs/PERFORMANCE.md b/docs/PERFORMANCE.md new file mode 100644 index 00000000..05b12d16 --- /dev/null +++ b/docs/PERFORMANCE.md @@ -0,0 +1,465 @@ +# SharpCompress Performance Guide + +This guide helps you optimize SharpCompress for performance in various scenarios. + +## API Selection Guide + +### Archive API vs Reader API + +Choose the right API based on your use case: + +| Aspect | Archive API | Reader API | +|--------|------------|-----------| +| **Stream Type** | Seekable only | Non-seekable OK | +| **Memory Usage** | All entries in memory | One entry at a time | +| **Random Access** | ✓ Yes | ✗ No | +| **Best For** | Small-to-medium archives | Large or streaming data | +| **Performance** | Fast for random access | Better for large files | + +### Archive API (Fast for Random Access) + +```csharp +// Use when: +// - Archive fits in memory +// - You need random access to entries +// - Stream is seekable (file, MemoryStream) + +using (var archive = ZipArchive.OpenArchive("archive.zip")) +{ + // Random access - all entries available + var specific = archive.Entries.FirstOrDefault(e => e.Key == "file.txt"); + if (specific != null) + { + specific.WriteToFile(@"C:\output\file.txt"); + } +} +``` + +**Performance Characteristics:** +- ✓ Instant entry lookup +- ✓ Parallel extraction possible +- ✗ Entire archive in memory +- ✗ Can't process while downloading + +### Reader API (Best for Large Files) + +```csharp +// Use when: +// - Processing large archives (>100 MB) +// - Streaming from network/pipe +// - Memory is constrained +// - Forward-only processing is acceptable + +using (var stream = File.OpenRead("large.zip")) +using (var reader = ReaderFactory.OpenReader(stream)) +{ + while (reader.MoveToNextEntry()) + { + // Process one entry at a time + reader.WriteEntryToDirectory(@"C:\output"); + } +} +``` + +**Performance Characteristics:** +- ✓ Minimal memory footprint +- ✓ Works with non-seekable streams +- ✓ Can process while downloading +- ✗ Forward-only (no random access) +- ✗ Entry lookup requires iteration + +--- + +## Buffer Sizing + +### Understanding Buffers + +SharpCompress uses internal buffers for reading compressed data. Buffer size affects: +- **Speed:** Larger buffers = fewer I/O operations = faster +- **Memory:** Larger buffers = higher memory usage + +### Recommended Buffer Sizes + +| Scenario | Size | Notes | +|----------|------|-------| +| Embedded/IoT devices | 4-8 KB | Minimal memory usage | +| Memory-constrained | 16-32 KB | Conservative default | +| Standard use (default) | 64 KB | Recommended default | +| Large file streaming | 256 KB | Better throughput | +| High-speed SSD | 512 KB - 1 MB | Maximum throughput | + +### How Buffer Size Affects Performance + +```csharp +// SharpCompress manages buffers internally +// You can't directly set buffer size, but you can: + +// 1. Use Stream.CopyTo with explicit buffer size +using (var entryStream = reader.OpenEntryStream()) +using (var fileStream = File.Create(@"C:\output\file.txt")) +{ + // 64 KB buffer (default) + entryStream.CopyTo(fileStream); + + // Or specify larger buffer for faster copy + entryStream.CopyTo(fileStream, bufferSize: 262144); // 256 KB +} + +// 2. Use custom buffer for writing +using (var entryStream = reader.OpenEntryStream()) +using (var fileStream = File.Create(@"C:\output\file.txt")) +{ + byte[] buffer = new byte[262144]; // 256 KB + int bytesRead; + while ((bytesRead = entryStream.Read(buffer, 0, buffer.Length)) > 0) + { + fileStream.Write(buffer, 0, bytesRead); + } +} +``` + +--- + +## Streaming Large Files + +### Non-Seekable Stream Patterns + +For processing archives from downloads or pipes: + +```csharp +// Download stream (non-seekable) +using (var httpStream = await httpClient.GetStreamAsync(url)) +using (var reader = ReaderFactory.OpenReader(httpStream)) +{ + // Process entries as they arrive + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + reader.WriteEntryToDirectory(@"C:\output"); + } + } +} +``` + +**Performance Tips:** +- Don't try to buffer the entire stream +- Process entries immediately +- Use async APIs for better responsiveness + +### Download-Then-Extract vs Streaming + +Choose based on your constraints: + +| Approach | When to Use | +|----------|------------| +| **Download then extract** | Moderate size, need random access | +| **Stream during download** | Large files, bandwidth limited, memory constrained | + +```csharp +// Download then extract (requires disk space) +var archivePath = await DownloadFile(url, @"C:\temp\archive.zip"); +using (var archive = ZipArchive.OpenArchive(archivePath)) +{ + archive.WriteToDirectory(@"C:\output"); +} + +// Stream during download (on-the-fly extraction) +using (var httpStream = await httpClient.GetStreamAsync(url)) +using (var reader = ReaderFactory.OpenReader(httpStream)) +{ + while (reader.MoveToNextEntry()) + { + reader.WriteEntryToDirectory(@"C:\output"); + } +} +``` + +--- + +## Solid Archive Optimization + +### Why Solid Archives Are Slow + +Solid archives (Rar, 7Zip) group files together in a single compressed stream: + +``` +Solid Archive Layout: +[Header] [Compressed Stream] [Footer] + ├─ File1 compressed data + ├─ File2 compressed data + ├─ File3 compressed data + └─ File4 compressed data +``` + +Extracting File3 requires decompressing File1 and File2 first. + +### Sequential vs Random Extraction + +**Random Extraction (Slow):** +```csharp +using (var archive = RarArchive.OpenArchive("solid.rar")) +{ + foreach (var entry in archive.Entries) + { + entry.WriteToFile(@"C:\output\" + entry.Key); // ✗ Slow! + // Each entry triggers full decompression from start + } +} +``` + +**Sequential Extraction (Fast):** +```csharp +using (var archive = RarArchive.OpenArchive("solid.rar")) +{ + // Method 1: Use WriteToDirectory (recommended) + archive.WriteToDirectory(@"C:\output"); + + // Method 2: Use ExtractAllEntries + archive.ExtractAllEntries(); + + // Method 3: Use Reader API (also sequential) + using (var reader = RarReader.Open(File.OpenRead("solid.rar"))) + { + while (reader.MoveToNextEntry()) + { + reader.WriteEntryToDirectory(@"C:\output"); + } + } +} +``` + +**Performance Impact:** +- Random extraction: O(n²) - very slow for many files +- Sequential extraction: O(n) - 10-100x faster + +### Best Practices for Solid Archives + +1. **Always extract sequentially** when possible +2. **Use Reader API** for large solid archives +3. **Process entries in order** from the archive +4. **Consider using 7Zip command-line** for scripted extractions + +--- + +## Compression Level Trade-offs + +### Deflate/GZip Levels + +```csharp +// Level 1 = Fastest, largest size +// Level 6 = Default (balanced) +// Level 9 = Slowest, best compression + +// Write with different compression levels +using (var archive = ZipArchive.CreateArchive()) +{ + archive.AddAllFromDirectory(@"D:\data"); + + // Fast compression (level 1) + archive.SaveTo("fast.zip", new WriterOptions(CompressionType.Deflate) + { + CompressionLevel = 1 + }); + + // Default compression (level 6) + archive.SaveTo("default.zip", CompressionType.Deflate); + + // Best compression (level 9) + archive.SaveTo("best.zip", new WriterOptions(CompressionType.Deflate) + { + CompressionLevel = 9 + }); +} +``` + +**Speed vs Size:** +| Level | Speed | Size | Use Case | +|-------|-------|------|----------| +| 1 | 10x | 90% | Network, streaming | +| 6 | 1x | 75% | Default (good balance) | +| 9 | 0.1x | 65% | Archival, static storage | + +### BZip2 Block Size + +```csharp +// BZip2 block size affects memory and compression +// 100K to 900K (default 900K) + +// Smaller block size = lower memory, faster +// Larger block size = better compression, slower + +using (var archive = TarArchive.CreateArchive()) +{ + archive.AddAllFromDirectory(@"D:\data"); + + // These are preset in WriterOptions via CompressionLevel + archive.SaveTo("archive.tar.bz2", CompressionType.BZip2); +} +``` + +### LZMA Settings + +LZMA compression is very powerful but memory-intensive: + +```csharp +// LZMA (7Zip, .tar.lzma): +// - Dictionary size: 16 KB to 1 GB (default 32 MB) +// - Faster preset: smaller dictionary +// - Better compression: larger dictionary + +// Preset via CompressionType +using (var archive = TarArchive.CreateArchive()) +{ + archive.AddAllFromDirectory(@"D:\data"); + archive.SaveTo("archive.tar.xz", CompressionType.LZMA); // Default settings +} +``` + +--- + +## Async Performance + +### When Async Helps + +Async is beneficial when: +- **Long I/O operations** (network, slow disks) +- **UI responsiveness** needed (Windows Forms, WPF, Blazor) +- **Server applications** (ASP.NET, multiple concurrent operations) + +```csharp +// Async extraction (non-blocking) +using (var archive = ZipArchive.OpenArchive("archive.zip")) +{ + await archive.WriteToDirectoryAsync( + @"C:\output", + cancellationToken: cancellationToken + ); +} +// Thread can handle other work while I/O happens +``` + +### When Async Doesn't Help + +Async doesn't improve performance for: +- **CPU-bound operations** (already fast) +- **Local SSD I/O** (I/O is fast enough) +- **Single-threaded scenarios** (no parallelism benefit) + +```csharp +// Sync extraction (simpler, same performance on fast I/O) +using (var archive = ZipArchive.OpenArchive("archive.zip")) +{ + archive.WriteToDirectory(@"C:\output"); +} +// Simple and fast - no async needed +``` + +### Cancellation Pattern + +```csharp +var cts = new CancellationTokenSource(); + +// Cancel after 5 minutes +cts.CancelAfter(TimeSpan.FromMinutes(5)); + +try +{ + using (var archive = ZipArchive.OpenArchive("archive.zip")) + { + await archive.WriteToDirectoryAsync( + @"C:\output", + cancellationToken: cts.Token + ); + } +} +catch (OperationCanceledException) +{ + Console.WriteLine("Extraction cancelled"); + // Clean up partial extraction if needed +} +``` + +--- + +## Practical Performance Tips + +### 1. Choose the Right API + +| Scenario | API | Why | +|----------|-----|-----| +| Small archives | Archive | Faster random access | +| Large archives | Reader | Lower memory | +| Streaming | Reader | Works on non-seekable streams | +| Download streams | Reader | Async extraction while downloading | + +### 2. Batch Operations + +```csharp +// ✗ Slow - opens each archive separately +foreach (var file in files) +{ + using (var archive = ZipArchive.OpenArchive("archive.zip")) + { + archive.WriteToDirectory(@"C:\output"); + } +} + +// ✓ Better - process multiple entries at once +using (var archive = ZipArchive.OpenArchive("archive.zip")) +{ + archive.WriteToDirectory(@"C:\output"); +} +``` + +### 3. Profile Your Code + +```csharp +var sw = Stopwatch.StartNew(); +using (var archive = ZipArchive.OpenArchive("large.zip")) +{ + archive.WriteToDirectory(@"C:\output"); +} +sw.Stop(); + +Console.WriteLine($"Extraction took {sw.ElapsedMilliseconds}ms"); + +// Measure memory before/after +var beforeMem = GC.GetTotalMemory(true); +// ... do work ... +var afterMem = GC.GetTotalMemory(true); +Console.WriteLine($"Memory used: {(afterMem - beforeMem) / 1024 / 1024}MB"); +``` + +--- + +## Troubleshooting Performance + +### Extraction is Slow + +1. **Check if solid archive** → Use sequential extraction +2. **Check API** → Reader API might be faster for large files +3. **Check compression level** → Higher levels are slower to decompress +4. **Check I/O** → Network drives are much slower than SSD +5. **Check buffer size** → May need larger buffers for network + +### High Memory Usage + +1. **Use Reader API** instead of Archive API +2. **Process entries immediately** rather than buffering +3. **Reduce compression level** if writing +4. **Check for memory leaks** in your code + +### CPU Usage at 100% + +1. **Normal for compression** - especially with high compression levels +2. **Consider lower level** for faster processing +3. **Reduce parallelism** if processing multiple archives +4. **Check if awaiting properly** in async code + +--- + +## Related Documentation + +- [PERFORMANCE.md](USAGE.md) - Usage examples with performance considerations +- [FORMATS.md](FORMATS.md) - Format-specific performance notes diff --git a/docs/TAR_GAP_ANALYSIS.md b/docs/TAR_GAP_ANALYSIS.md new file mode 100644 index 00000000..0b621261 --- /dev/null +++ b/docs/TAR_GAP_ANALYSIS.md @@ -0,0 +1,291 @@ +# Tar Gap Analysis + +## Scope + +This document compares the current Tar documentation, tests, and code paths in SharpCompress. + +It is intentionally implementation-focused. The goal is to identify mismatches, omissions, and incomplete areas in the current SharpCompress Tar support. + +Primary references: + +- `docs/FORMATS.md` +- `src/SharpCompress/Factories/TarFactory.cs` +- `src/SharpCompress/Factories/TarWrapper.cs` +- `src/SharpCompress/Archives/Tar/` +- `src/SharpCompress/Readers/Tar/` +- `src/SharpCompress/Writers/Tar/` +- `src/SharpCompress/Common/Tar/` +- `tests/SharpCompress.Test/Tar/` + +## Implemented Since Baseline + +- `Tar.XZ` is now documented as read-only (`Writer API = N/A`) in `docs/FORMATS.md`. +- Local PAX extended headers (`x`) are now implemented on the read path for selected keys. +- Global PAX extended headers (`g`) are now implemented on the read path for selected keys. +- Tar tests now include local PAX coverage for reader/archive sync and async paths. +- Tar tests now include global PAX coverage for reader/archive sync and async paths. +- `TarWriterOptions.HeaderFormat` is now honored in sync and async file and directory write paths. +- Tar tests now cover `USTAR` and `GNU_TAR_LONG_LINK`, including USTAR long-name failure scenarios. +- Symlink coverage now includes `TarWithSymlink.tar.gz` for reader sync and async paths. +- Tar tests now explicitly cover unsupported tar wrapper compression writes (`Xz`, `ZStandard`, `Lzw`) for sync and async writer paths. +- `TarArchive.OpenAsyncArchive(Stream)` now enforces the same seekable-stream contract as `TarArchive.OpenArchive(Stream)`. +- Sparse handling remains explicitly unsupported. +- Non-modeled PAX keys remain explicitly unsupported. + +## Claimed vs Actual Support + +### `Tar.XZ` is read-only + +`tar.xz` is supported for reading, but not for writing. + +Actual implementation in `src/SharpCompress/Writers/Tar/TarWriter.cs` does not support `CompressionType.Xz`. The writer throws `InvalidFormatException` for any compression type outside: + +- `None` +- `GZip` +- `BZip2` +- `LZip` + +Impact: + +- Tar write support is narrower than Tar read support +- `tar.xz` creation is not available through the built-in Tar writer + +Recommended action: + +- keep the format table marked `N/A` for Tar.XZ writer support + +## Read-Path Gaps + +### Local and global PAX headers are implemented for selected keys + +Local (`x`) and global (`g`) POSIX PAX extended headers are now supported on the read path. + +Supported keys in the current implementation: + +- `path` +- `linkpath` +- `size` +- `mtime` +- `uid` +- `gid` +- `mode` + +Remaining gap: + +- non-modeled PAX keys are still ignored +- PAX sparse extensions are still unsupported + +Recommended action: + +- keep supported-key boundaries documented and test-covered +- keep unsupported-key behavior explicit in docs + +### Sparse files are not semantically implemented + +`EntryType` defines `SparseFile`, but the read path does not contain sparse map handling or sparse reconstruction logic. + +PAX sparse extensions are also unsupported (for example `GNU.sparse.*` and similar sparse metadata keys). + +Evidence: + +- `src/SharpCompress/Common/Tar/Headers/EntryType.cs` +- no sparse-specific code in `TarHeader`, `TarEntry`, `TarFilePart`, or `TarArchive` +- no sparse tests + +Impact: + +- sparse entries may be treated as ordinary entries rather than sparse files with holes + +Recommended action: + +- keep sparse support explicitly documented as unsupported +- add sparse fixtures and tests only when sparse reconstruction is implemented + +### Non-modeled PAX keys are still unsupported + +PAX parsing is intentionally limited to modeled keys (`path`, `linkpath`, `size`, `mtime`, `uid`, `gid`, `mode`). + +Not currently modeled/supported: + +- `uname` +- `gname` +- `atime` +- `ctime` +- device-specific values and vendor keys + +Recommended action: + +- keep unsupported-key behavior documented as ignored +- add support only when there is a consumer-facing object model for it + +### Device and FIFO semantics are not surfaced + +The entry type enum includes `CharDevice`, `BlockDevice`, and `Fifo`, but the public tar model does not expose device metadata semantics. + +Impact: + +- such entries may not round-trip meaningfully through the API +- behavior is undocumented and untested + +Recommended action: + +- either document them as raw/unmodeled entry types or add dedicated support + +## Write-Path Gaps + +### `HeaderFormat` consistency is resolved + +`TarWriterOptions.HeaderFormat` is now applied across: + +- sync file writes +- sync directory writes +- async file writes +- async directory writes + +Regression tests now cover both `USTAR` and `GNU_TAR_LONG_LINK` behavior. + +### No public link-writing support + +The read path supports symbolic and hard link targets through `TarEntry.LinkTarget`, but the write API exposes only regular file and directory creation. + +Impact: + +- symlink and hardlink tar archives cannot be created through the current public Tar writer API + +Recommended action: + +- either document this as a deliberate limitation or add link-writing APIs + +### Metadata round-trip support is incomplete + +The writer does not round-trip rich tar metadata beyond the basic fields needed for file and directory entries. + +Current write behavior sets fixed defaults for some fields such as mode, owner id, and group id. + +Impact: + +- modified or newly created tar archives may lose metadata fidelity relative to the original archive + +Recommended action: + +- document current metadata write behavior clearly +- expand metadata support only if needed by consumers + +### No write support for some detected wrappers + +The detection and read path supports wrappers that the write path does not support. + +| Wrapper | Read support | Write support | +| ------- | ------------ | ------------- | +| `tar.xz` | Yes | No | +| `tar.zst` | Yes | No | +| `tar.Z` | Yes | No | + +This is not inherently wrong, but it should be clearly documented everywhere support is summarized. + +## Sync and Async API Inconsistencies + +### Seekability contract alignment is resolved + +`TarArchive.OpenArchive(Stream)` and `TarArchive.OpenAsyncArchive(Stream)` now both enforce the same seekable-stream contract and throw `ArgumentException` for non-seekable input. + +Tar tests include an async regression case for non-seekable stream open. + +### Header format alignment between sync and async is resolved + +Sync and async Tar writer paths now both honor `TarWriterOptions.HeaderFormat`, and matching tests are present for both paths. + +## Test Coverage Gaps + +### Symlink coverage is now present for reader paths + +Symlink behavior is now asserted for sync and async reader paths using: + +- `tests/TestArchives/Archives/TarWithSymlink.tar.gz` + +Archive-path symlink assertions currently rely on small tar fixtures rather than this large compressed sample. + +### Header format coverage is now present + +Tar tests now cover: + +- `TarWriterOptions.HeaderFormat = USTAR` +- `TarWriterOptions.HeaderFormat = GNU_TAR_LONG_LINK` +- long-name failure in USTAR mode +- long-name success in GNU mode through sync and async writer paths + +### No tests for sparse tar semantics + +Local and global PAX coverage now exists, but there is still no evidence of coverage for: + +- sparse tar entries +- sparse PAX extensions + +Impact: + +- unsupported or partial behavior is neither documented by tests nor protected from regression + +Recommended action: + +- either add fixtures and tests or document these as unsupported with no test coverage + +### Unsupported-wrapper writer coverage is now present + +Tar writer tests now explicitly verify `InvalidFormatException` for unsupported tar wrapper compression types: + +- `CompressionType.Xz` +- `CompressionType.ZStandard` +- `CompressionType.Lzw` + +Coverage exists in both sync and async writer test paths. + +## Documentation Gaps + +### Current format documentation is too coarse for Tar + +`docs/FORMATS.md` summarizes support at the wrapper level, but Tar behavior depends on more than wrapper compression. + +Missing implementation-specific details include: + +- GNU long-name and long-link support +- USTAR prefix handling +- oldgnu numeric quirk handling +- partial PAX support boundaries (selected local/global keys supported) +- missing sparse support +- reader vs archive behavior differences for compressed tar +- file-size requirements for writing from non-seekable sources + +Recommended action: + +- keep `docs/FORMATS.md` high-level +- add and maintain a dedicated Tar spec document for details + +### The current docs do not call out partial support clearly + +The codebase supports some tar dialect features and not others, but the docs do not separate: + +- fully supported +- partially supported +- unsupported + +Recommended action: + +- use an explicit feature matrix in the Tar documentation + +## Recommended Follow-Ups + +### Priority 1 + +- Improve metadata round-trip behavior only if there is a consumer need +- Evaluate whether non-modeled PAX keys should remain ignored or be surfaced in a future metadata API + +## Summary + +The SharpCompress Tar implementation is strong on common read scenarios and basic write scenarios, but the current gaps fall into four categories: + +- documentation overstating or under-describing support +- incomplete feature coverage for less common tar dialect features +- intentionally deferred metadata and API-surface decisions +- test coverage holes around advanced tar metadata features + +`docs/TAR_SPEC.md` should be treated as the implementation baseline. This document identifies where that baseline is incomplete, inconsistent, or incorrectly reflected elsewhere in the repository. diff --git a/docs/TAR_SPEC.md b/docs/TAR_SPEC.md new file mode 100644 index 00000000..b1870004 --- /dev/null +++ b/docs/TAR_SPEC.md @@ -0,0 +1,467 @@ +# Tar Spec + +## Scope + +This document describes the Tar implementation that exists in SharpCompress today. + +It is intentionally SharpCompress-specific. It documents actual behavior in the current codebase, including partial support and limitations. It is not a general tar format reference. + +Primary implementation files: + +- `src/SharpCompress/Factories/TarFactory.cs` +- `src/SharpCompress/Factories/TarWrapper.cs` +- `src/SharpCompress/Archives/Tar/TarArchive.cs` +- `src/SharpCompress/Archives/Tar/TarArchive.Async.cs` +- `src/SharpCompress/Archives/Tar/TarArchive.Factory.cs` +- `src/SharpCompress/Readers/Tar/TarReader.cs` +- `src/SharpCompress/Readers/Tar/TarReader.Async.cs` +- `src/SharpCompress/Writers/Tar/TarWriter.cs` +- `src/SharpCompress/Writers/Tar/TarWriter.Async.cs` +- `src/SharpCompress/Writers/Tar/TarWriterOptions.cs` +- `src/SharpCompress/Common/Tar/Headers/TarHeader.cs` +- `src/SharpCompress/Common/Tar/Headers/TarHeader.Async.cs` +- `src/SharpCompress/Common/Tar/TarHeaderFactory.cs` +- `src/SharpCompress/Common/Tar/TarHeaderFactory.Async.cs` + +## API Surface + +SharpCompress exposes Tar support through four main entry points. + +| Type | Role | +| ---- | ---- | +| `TarFactory` | Format detection and factory entry point for archive, reader, and writer APIs | +| `TarArchive` | Archive API for enumerating and rewriting tar archives | +| `TarReader` | Forward-only reader API for streaming tar extraction | +| `TarWriter` | Forward-only writer API for creating tar archives | + +`TarWriterOptions` controls output compression, stream ownership, archive finalization, encoding, and header write format. + +## Supported Wrapper Formats + +Tar wrapper detection is defined by `TarWrapper.Wrappers` in `src/SharpCompress/Factories/TarWrapper.cs`. + +### Supported Extensions + +| Wrapper | Extensions | +| ------- | ---------- | +| Plain tar | `tar` | +| Tar + BZip2 | `tar.bz2`, `tb2`, `tbz`, `tbz2`, `tz2` | +| Tar + GZip | `tar.gz`, `taz`, `tgz` | +| Tar + ZStandard | `tar.zst`, `tar.zstd`, `tzst`, `tzstd` | +| Tar + LZip | `tar.lz` | +| Tar + XZ | `tar.xz`, `txz` | +| Tar + LZW compress | `tar.Z`, `tZ`, `taZ` | + +### API Support Matrix + +| Wrapper | Detection | `TarArchive` read | `TarReader` read | `TarWriter` write | +| ------- | --------- | ----------------- | ---------------- | ----------------- | +| Plain tar | Yes | Yes | Yes | Yes | +| Tar + GZip | Yes | Yes | Yes | Yes | +| Tar + BZip2 | Yes | Yes | Yes | Yes | +| Tar + LZip | Yes | Yes | Yes | Yes | +| Tar + XZ | Yes | Yes | Yes | No | +| Tar + ZStandard | Yes | Yes | Yes | No | +| Tar + LZW compress | Yes | Yes | Yes | No | + +Write support is implemented in `src/SharpCompress/Writers/Tar/TarWriter.cs` and currently accepts only `CompressionType.None`, `CompressionType.GZip`, `CompressionType.BZip2`, and `CompressionType.LZip`. + +## Detection Behavior + +Tar detection is implemented in `TarFactory.IsArchive`, `TarFactory.IsArchiveAsync`, `TarFactory.GetCompressionType`, and `TarFactory.GetCompressionTypeAsync`. + +Detection behavior is: + +1. Wrap the incoming stream in `SharpCompressStream`. +2. Start recording with a rewind buffer sized from `TarWrapper.MaximumRewindBufferSize`. +3. Probe each registered wrapper in order. +4. If a wrapper matches, create a decompression stream for that wrapper. +5. Call `TarArchive.IsTarFile` or `TarArchive.IsTarFileAsync` on the decompressed stream. +6. If the tar probe succeeds, treat the stream as tar with that wrapper compression. + +Implications: + +- Tar detection is content-based, not extension-based. +- Wrapper detection is not sufficient by itself. The decompressed payload must also parse as tar. +- Non-seekable detection is supported through the recording and rewind mechanism. +- The largest rewind requirement currently comes from BZip2, which declares a larger minimum probe buffer in `TarWrapper`. + +`TarArchive.IsTarFile` and `TarArchive.IsTarFileAsync` attempt to read a single tar header and return `false` on any exception. They also treat an all-zero empty archive block as a valid empty tar archive when the entry type is defined. + +## Reader Behavior + +`TarReader` is the forward-only streaming API. + +Implementation files: + +- `src/SharpCompress/Readers/Tar/TarReader.cs` +- `src/SharpCompress/Readers/Tar/TarReader.Async.cs` +- `src/SharpCompress/Common/Tar/TarEntry.cs` +- `src/SharpCompress/Common/Tar/TarEntry.Async.cs` +- `src/SharpCompress/Common/Tar/TarReadOnlySubStream.cs` + +Reader behavior: + +- The reader always enumerates entries in streaming mode. +- It works with non-seekable input streams. +- It applies decompression based on the detected wrapper compression type before parsing tar headers. +- Entry streams are backed by `TarReadOnlySubStream`. + +`TarReadOnlySubStream` has an important behavior: disposing an entry stream consumes any unread entry bytes and any required 512-byte padding so that the next header can be read correctly. This is what makes skipping entries work in streaming mode. + +### Reader Compression Mapping + +`TarReader.RequestInitialStream` and `RequestInitialStreamAsync` map the detected wrapper to the corresponding decompression stream: + +- `None` +- `BZip2` +- `GZip` +- `ZStandard` +- `LZip` +- `Xz` +- `Lzw` + +### Reader Entry Semantics + +For each entry, SharpCompress exposes: + +- `Key` from the parsed tar name +- `LinkTarget` for symbolic and hard links +- `Size` +- `CompressedSize` +- `LastModifiedTime` +- `IsDirectory` +- `Mode` +- `UserID` +- `GroupId` + +Tar entries are always reported as unencrypted and CRC is always `0`. + +## Archive Behavior + +`TarArchive` is the archive API. + +Implementation files: + +- `src/SharpCompress/Archives/Tar/TarArchive.cs` +- `src/SharpCompress/Archives/Tar/TarArchive.Async.cs` +- `src/SharpCompress/Archives/Tar/TarArchive.Factory.cs` + +### Open Behavior + +`TarArchive.OpenArchive(Stream)` and `TarArchive.OpenAsyncArchive(Stream)` require a seekable stream and throw `ArgumentException` when `CanSeek` is `false`. + +`TarArchive.OpenArchive(FileInfo)` and the list-based overloads use `SourceStream` and determine wrapper compression by calling `TarFactory.GetCompressionType`. + +Asynchronous `OpenAsyncArchive` overloads use `TarFactory.GetCompressionTypeAsync` for wrapper detection. + +### Entry Loading + +`TarArchive.LoadEntries` and `LoadEntriesAsync` parse entries differently depending on wrapper compression: + +- Uncompressed tar uses `StreamingMode.Seekable`. +- Wrapped tar uses `StreamingMode.Streaming` because the decompressed stream is not treated as random-access. + +When seekable mode is used, the header stores `DataStartPosition`, and entries reopen data through `TarFilePart` by seeking back to the data position. + +When streaming mode is used, the header stores a `PackedStream`, and entry access follows streaming semantics over the decompressed stream. + +### Archive Rewrite Behavior + +`TarArchive` supports creating and modifying archives through `AbstractWritableArchive`: + +- add file entries +- add directory entries +- remove entries +- save to a new stream or path + +Archive rewrite is implemented by enumerating the existing and new entries and writing them back out through `TarWriter`. + +## Writer Behavior + +`TarWriter` is the forward-only tar writer. + +Implementation files: + +- `src/SharpCompress/Writers/Tar/TarWriter.cs` +- `src/SharpCompress/Writers/Tar/TarWriter.Async.cs` +- `src/SharpCompress/Writers/Tar/TarWriterOptions.cs` + +### Supported Output Compression + +The writer supports these output compression types: + +- `CompressionType.None` +- `CompressionType.GZip` +- `CompressionType.BZip2` +- `CompressionType.LZip` + +Any other compression type causes `InvalidFormatException`. + +### Stream Ownership + +If `LeaveStreamOpen` is `true`, `TarWriter` wraps the destination in a non-disposing stream. + +### File Writing + +`TarWriter.Write` and `WriteAsync` write a tar header followed by file contents, then pad the payload to the next 512-byte boundary. + +If the source stream is non-seekable and the caller does not supply `size`, the writer throws `ArgumentException` because tar requires the file size in the header. + +### Directory Writing + +`WriteDirectory` and `WriteDirectoryAsync` normalize the directory name to use forward slashes and ensure the key ends with `/`. + +Empty or root-equivalent directory names are skipped. + +### Archive Finalization + +If `FinalizeArchiveOnClose` is `true`, disposing the writer writes two 512-byte zero blocks to terminate the archive. + +If the output stream implements `IFinishable`, dispose also calls `Finish()`. + +## Header Write Formats + +`TarHeaderWriteFormat` is defined in `src/SharpCompress/Common/Tar/Headers/TarHeaderWriteFormat.cs`. + +Supported write formats: + +- `GNU_TAR_LONG_LINK` +- `USTAR` + +`TarWriterOptions.HeaderFormat` defaults to `GNU_TAR_LONG_LINK`. + +Current implementation behavior is narrower than the option surface suggests: + +- sync file writes use the configured `HeaderFormat` +- sync directory writes currently construct the default tar header format +- async file writes currently construct the default tar header format +- async directory writes currently construct the default tar header format + +In practice, this means the configured `HeaderFormat` is currently honored only by the synchronous file write path. + +### GNU Long Name Write Behavior + +In GNU mode, when a file name exceeds the 100-byte field, `TarHeader.WriteGnuTarLongLink` writes a synthetic long-name header using `././@LongLink` and `EntryType.LongName`, then writes the long name payload, and finally writes the actual file entry. + +GNU mode also writes large file sizes using binary size encoding when the size does not fit the standard octal field. + +### USTAR Write Behavior + +When the synchronous file write path is configured for `USTAR`, `TarHeader.WriteUstar` attempts to split a long path into: + +- the main `name` field +- the `prefix` field + +If the name cannot be represented in USTAR field limits, the writer throws `InvalidFormatException` and instructs the caller to use GNU Tar format instead. + +## Header Read Behavior + +Tar header parsing is implemented in `TarHeader.Read` and `TarHeader.ReadAsync`. + +### Implemented Read Features + +| Feature | Read support | +| ------- | ------------ | +| Regular file entries | Yes | +| Directory entries | Yes | +| Symbolic link target reading | Yes | +| Hard link target reading | Yes | +| GNU long name (`L`) | Yes | +| GNU long link (`K`) | Yes | +| PAX local extended header (`x`) | Yes (selected keys) | +| PAX global extended header (`g`) | Yes (selected keys) | +| USTAR prefix reconstruction | Yes | +| Binary size field parsing | Yes | +| oldgnu uid/gid numeric quirk parsing | Yes | +| POSIX and signed checksum validation | Yes | + +### Entry Types Recognized by the Code + +`EntryType` currently declares these values in `src/SharpCompress/Common/Tar/Headers/EntryType.cs`: + +- `File` +- `OldFile` +- `HardLink` +- `SymLink` +- `CharDevice` +- `BlockDevice` +- `Directory` +- `Fifo` +- `LongLink` +- `LongName` +- `SparseFile` +- `VolumeHeader` +- `LocalExtendedHeader` +- `GlobalExtendedHeader` + +SharpCompress currently has explicit handling for only a subset of those values during read and write. + +### Long Name and Long Link Reads + +When `TarHeader.Read` encounters `EntryType.LongName` or `EntryType.LongLink`, it reads the payload and applies it to the next real header. + +Long-name payload reads are capped at `32768` bytes to avoid memory exhaustion from malformed archives. + +### PAX Local Header Reads + +SharpCompress now consumes local PAX extended headers (`x`) and applies supported key overrides to the next real entry. + +Currently supported keys: + +- `path` +- `linkpath` +- `size` +- `mtime` +- `uid` +- `gid` +- `mode` + +Unknown PAX keys are ignored. + +### PAX Global Header Reads + +SharpCompress consumes global PAX extended headers (`g`) and applies supported key overrides to subsequent entries. + +Supported keys match local PAX support: + +- `path` +- `linkpath` +- `size` +- `mtime` +- `uid` +- `gid` +- `mode` + +Global metadata is overridden by local per-entry metadata when both are present. + +### Name Reconstruction + +For USTAR headers, if the magic field is `ustar` and the prefix field is populated, SharpCompress reconstructs the entry name as `prefix + "/" + name`. + +## Name and Metadata Handling + +### Path Normalization + +Writer path normalization is implemented in `TarWriter.NormalizeFilename` and `NormalizeDirectoryName`. + +Behavior: + +- backslashes are converted to `/` +- drive prefixes before `:` are removed +- leading and trailing `/` are trimmed for file entries +- directory entries are normalized to end with `/` + +### Encoding + +Tar name encoding and decoding is controlled by `IArchiveEncoding`. + +- reader APIs decode names with `ReaderOptions.ArchiveEncoding` +- writer APIs encode names with `TarWriterOptions.ArchiveEncoding` + +The tests include UTF-8 and code page coverage for tar name handling. + +### Metadata Surface + +Tar metadata currently surfaced through `TarEntry` includes: + +- name +- link target +- mode +- uid +- gid +- size +- last modified time + +Writer metadata is narrower. The writer sets: + +- `LastModifiedTime` +- `Name` +- `Size` +- entry type for file or directory + +The current writer writes fixed mode, owner id, and group id defaults rather than round-tripping full metadata. + +## Async Behavior + +Async tar support is provided by: + +- `TarArchive.OpenAsyncArchive` +- `TarReader.OpenAsyncReader` +- `TarWriter.WriteAsync` +- `TarWriter.WriteDirectoryAsync` +- `TarHeader.ReadAsync` +- `TarHeader.WriteAsync` + +The async implementations generally mirror the sync implementations while using async header parsing, decompression, and stream copy paths. + +## Known Limitations + +This section documents current implementation limits, not desired future behavior. + +### Write limitations + +- No write support for `tar.xz` +- No write support for `tar.zst` +- No write support for `tar.Z` +- No public API for writing symbolic links or hard links +- No PAX write support +- No sparse file write support +- No device or FIFO write support + +### Read limitations or partial support + +- PAX support is limited to selected keys (`path`, `linkpath`, `size`, `mtime`, `uid`, `gid`, `mode`) +- No semantic sparse file handling beyond recognizing the entry type enum value +- No special device or FIFO object model beyond the raw entry type information available internally + +### Archive behavior limitations + +- Stream-based archive open requires a seekable input stream +- Compressed tar archive access is not full random-access in the same sense as uncompressed seekable tar + +## Test Coverage Map + +Tar tests live in `tests/SharpCompress.Test/Tar/`. + +Representative coverage: + +| Area | Tests | +| ---- | ----- | +| Wrapper detection and reading | `TarReaderTests.cs`, `TarReaderAsyncTests.cs` | +| Archive open and rewrite | `TarArchiveTests.cs`, `TarArchiveAsyncTests.cs` | +| Writer behavior | `TarWriterTests.cs`, `TarWriterAsyncTests.cs` | +| Directory entry behavior | `TarWriterDirectoryTests.cs`, `TarArchiveDirectoryTests.cs` | +| Long-name behavior | `TarArchiveTests.cs`, `TarReaderTests.cs` | +| Corruption and broken stream handling | `TarReaderTests.cs`, `TarReaderAsyncTests.cs` | + +Representative tar test archives in `tests/TestArchives/Archives/`: + +- `Tar.tar` +- `Tar.tar.gz` +- `Tar.tar.bz2` +- `Tar.tar.lz` +- `Tar.tar.xz` +- `Tar.tar.zst` +- `Tar.tar.Z` +- `Tar.oldgnu.tar.gz` +- `very long filename.tar` +- `ustar with long names.tar` +- `Tar.LongPathsWithLongNameExtension.tar` +- `Tar.PaxGlobalHeader.tar` +- `Tar.PaxGlobalHeader.Link.tar` +- `Tar.Empty.tar` +- `TarCorrupted.tar` +- `TarWithSymlink.tar.gz` + +## Summary + +SharpCompress Tar support is centered around: + +- broad read support for common tar wrappers +- forward-only reader behavior for streamed extraction +- seekable archive support for uncompressed tar and archive rewrite workflows +- narrower write support than read support +- GNU long-name and USTAR write support +- PAX local header (`x`) read support for selected metadata keys +- partial coverage for less common tar dialect features diff --git a/docs/USAGE.md b/docs/USAGE.md new file mode 100644 index 00000000..9666d43f --- /dev/null +++ b/docs/USAGE.md @@ -0,0 +1,474 @@ +# SharpCompress Usage + +## Async/Await Support + +SharpCompress now provides full async/await support for all I/O operations. All `Read`, `Write`, and extraction operations have async equivalents ending in `Async` that accept an optional `CancellationToken`. This enables better performance and scalability for I/O-bound operations. + +**Key Async Methods:** +- `reader.WriteEntryToAsync(stream, cancellationToken)` - Extract entry asynchronously +- `reader.WriteAllToDirectoryAsync(path, cancellationToken: cancellationToken)` - Extract all asynchronously +- `writer.WriteAsync(filename, stream, modTime, cancellationToken)` - Write entry asynchronously +- `writer.WriteAllAsync(directory, pattern, searchOption, cancellationToken)` - Write directory asynchronously +- `entry.OpenEntryStreamAsync(cancellationToken)` - Open entry stream asynchronously + +See [Async Examples](#async-examples) section below for usage patterns. + +## Stream Rules + +When dealing with Streams, the rule should be that you don't close a stream you didn't create. This, in effect, should mean you should always put a Stream in a using block to dispose it. + +However, the .NET Framework often has classes that will dispose streams by default to make things "easy" like the following: + +```C# +using (var reader = new StreamReader(File.Open("foo"))) +{ + ... +} +``` + +In this example, reader should get disposed. However, stream rules should say the the `FileStream` created by `File.Open` should remain open. However, the .NET Framework closes it for you by default unless you override the constructor. In general, you should be writing Stream code like this: + +```C# +using (var fileStream = File.Open("foo")) +using (var reader = new StreamReader(fileStream)) +{ + ... +} +``` + +To deal with the "correct" rules as well as the expectations of users, I've decided to always close wrapped streams as of 0.21. + +To be explicit though, consider always using the overloads that use `ReaderOptions` or `WriterOptions` and explicitly set `LeaveStreamOpen` the way you want. + +Default behavior in factory APIs: +- File path / `FileInfo` overloads set `LeaveStreamOpen = false`. +- Caller-provided `Stream` overloads set `LeaveStreamOpen = true`. + +If using Compression Stream classes directly and you don't want the wrapped stream to be closed. Use the `NonDisposingStream` as a wrapper to prevent the stream being disposed. The change in 0.21 simplified a lot even though the usage is a bit more convoluted. + +## Samples + +Also, look over the tests for more thorough [examples](https://github.com/adamhathcock/sharpcompress/tree/master/tests/SharpCompress.Test) + +### Create Zip Archive from multiple files +```C# +using(var archive = ZipArchive.CreateArchive()) +{ + archive.AddEntry("file01.txt", "C:\\file01.txt"); + archive.AddEntry("file02.txt", "C:\\file02.txt"); + ... + + archive.SaveTo("C:\\temp.zip", CompressionType.Deflate); +} +``` + +### Create Zip Archive from all files in a directory to a file + +```C# +using (var archive = ZipArchive.CreateArchive()) +{ + archive.AddAllFromDirectory("D:\\temp"); + archive.SaveTo("C:\\temp.zip", CompressionType.Deflate); +} +``` + +### Create Zip Archive from all files in a directory and save in memory + +```C# +var memoryStream = new MemoryStream(); +using (var archive = ZipArchive.CreateArchive()) +{ + archive.AddAllFromDirectory("D:\\temp"); + archive.SaveTo(memoryStream, new WriterOptions(CompressionType.Deflate) + { + LeaveStreamOpen = true + }); +} +//reset memoryStream to be usable now +memoryStream.Position = 0; +``` + +### Extract all files from a rar file to a directory using RarArchive + +Note: Extracting a solid rar or 7z file needs to be done in sequential order to get acceptable decompression speed. +`ExtractAllEntries` is primarily intended for solid archives (like solid Rar) or 7Zip archives, where sequential extraction provides the best performance. For general/simple extraction with any supported archive type, use `archive.WriteToDirectory()` instead. + +```C# +// Use ReaderOptions for open-time behavior and ExtractionOptions for extract-time behavior +using (var archive = RarArchive.OpenArchive("Test.rar", ReaderOptions.ForFilePath)) +{ + // Simple extraction with RarArchive; this WriteToDirectory pattern works for all archive types + archive.WriteToDirectory( + @"D:\temp", + new ExtractionOptions + { + ExtractFullPath = true, + Overwrite = true, + BufferSize = 131072, + CheckCrc = true, // Default: validate payload checksums when available + } + ); +} +``` + +### Iterate over all files from a Rar file using RarArchive + +```C# +using (var archive = RarArchive.OpenArchive("Test.rar")) +{ + foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) + { + Console.WriteLine($"{entry.Key}: {entry.Size} bytes"); + } +} +``` + +### Extract solid Rar or 7Zip archives with progress reporting + +For optimal performance with solid Rar and 7Zip archives, extract entries sequentially. `WriteToDirectory` handles that internally for simple extraction. Use `ExtractAllEntries` when you need to manually iterate in sequential order. + +```C# +using SharpCompress.Common; +using SharpCompress.Readers; + +var progress = new Progress(report => +{ + Console.WriteLine($"Extracting {report.EntryPath}: {report.PercentComplete}%"); +}); + +using (var archive = RarArchive.OpenArchive("archive.rar", + ReaderOptions.ForFilePath + .WithProgress(progress))) +{ + archive.WriteToDirectory( + @"D:\output", + new ExtractionOptions { ExtractFullPath = true, Overwrite = true, CheckCrc = true } + ); +} +``` + +Manual sequential extraction: + +```C# +using (var archive = RarArchive.OpenArchive("archive.rar", + ReaderOptions.ForFilePath.WithProgress(progress))) +using (var reader = archive.ExtractAllEntries()) +{ + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + reader.WriteEntryToDirectory(@"D:\output"); + } + } +} +``` + +### Use ReaderFactory to autodetect archive type and Open the entry stream + +```C# +using (Stream stream = File.OpenRead("Tar.tar.bz2")) +using (var reader = ReaderFactory.OpenReader(stream)) +{ + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + Console.WriteLine(reader.Entry.Key); + reader.WriteEntryToDirectory(@"C:\temp"); + } + } +} +``` + +### Use ReaderFactory to autodetect archive type and Open the entry stream + +```C# +using (Stream stream = File.OpenRead("Tar.tar.bz2")) +using (var reader = ReaderFactory.OpenReader(stream)) +{ + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + using (var entryStream = reader.OpenEntryStream()) + { + entryStream.CopyTo(...); + } + } + } +} +``` + +### Use WriterFactory to write all files from a directory in a streaming manner. + +```C# +using (Stream stream = File.OpenWrite("C:\\temp.tgz")) +using (var writer = WriterFactory.OpenWriter( + stream, + ArchiveType.Tar, + WriterOptions.ForTar(CompressionType.GZip).WithLeaveStreamOpen(true))) +{ + writer.WriteAll("D:\\temp", "*", SearchOption.AllDirectories); +} +``` + +### Use ArchiveFactory to autodetect and open archives + +```C# +if (ArchiveFactory.IsArchive("archive.zip", out var archiveType)) +{ + Console.WriteLine($"Detected {archiveType}"); +} + +using (var archive = ArchiveFactory.OpenArchive("archive.zip")) +{ + archive.WriteToDirectory(@"D:\output"); +} +``` + +### Use ArchiveInformation to choose the right API + +```C# +var archivePath = "archive.arc"; +var info = ArchiveFactory.GetArchiveInformation(archivePath); +if (info is null) +{ + Console.WriteLine("Not a supported archive"); +} +else if (info.SupportsRandomAccess) +{ + using var archive = ArchiveFactory.OpenArchive(archivePath); + archive.WriteToDirectory(@"D:\output"); +} +else +{ + using var reader = ReaderFactory.OpenReader(archivePath); + reader.WriteAllToDirectory(@"D:\output"); +} +``` + +`SupportsRandomAccess` is `false` for reader-only formats such as Ace, Arc, Arj, and standalone LZW. Use the Reader API for those formats. + +### Open multi-volume archives + +```C# +var parts = ArchiveFactory.GetFileParts("archive.part1.rar") + .Select(path => new FileInfo(path)) + .ToArray(); + +using (var archive = ArchiveFactory.OpenArchive(parts)) +{ + archive.WriteToDirectory(@"D:\output"); +} +``` + +### Use ReaderOptions for self-extracting archives and detection hints + +```C# +var sfxOptions = ReaderOptions.ForSelfExtractingArchive("password"); +using (var archive = RarArchive.OpenArchive("setup.exe", sfxOptions)) +{ + archive.WriteToDirectory(@"D:\output"); +} + +using (Stream stream = File.OpenRead("backup")) +using (var reader = ReaderFactory.OpenReader( + stream, + ReaderOptions.ForExternalStream.WithExtensionHint("tar.gz"))) +{ + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + reader.WriteEntryToDirectory(@"D:\output"); + } + } +} +``` + +### Write ZIP entries with per-entry options + +```C# +using Stream archiveStream = File.Create("output.zip"); +using var writer = new ZipWriter( + archiveStream, + new ZipWriterOptions(CompressionType.Deflate) + { + ArchiveComment = "Archive comment", + UseZip64 = true, + }); + +using Stream source = File.OpenRead("input.txt"); +writer.Write("entry.txt", source, new ZipWriterEntryOptions +{ + CompressionType = CompressionType.ZStandard, + CompressionLevel = 3, + EntryComment = "Entry comment", + ModificationDateTime = DateTime.UtcNow, + EnableZip64 = true, +}); +``` + +### Write a 7z archive + +```C# +using Stream stream = File.Create("output.7z"); +using var writer = WriterFactory.OpenWriter( + stream, + ArchiveType.SevenZip, + new SevenZipWriterOptions(CompressionType.LZMA2) + { + CompressHeader = true, + }); + +using Stream source = File.OpenRead("input.txt"); +writer.Write("input.txt", source, DateTime.UtcNow); +``` + +### Extract zip which has non-utf8 encoded filename(cp932) + +```C# +var encoding = Encoding.GetEncoding(932); +var opts = new ReaderOptions() + .WithArchiveEncoding(new ArchiveEncoding + { + CustomDecoder = (data, x, y) => encoding.GetString(data) + }); + +using var archive = ZipArchive.OpenArchive("test.zip", opts); +foreach(var entry in archive.Entries) +{ + Console.WriteLine($"{entry.Key}"); +} +``` + +## Custom Compression Providers + +By default `ReaderOptions` and `WriterOptions` already include `CompressionProviderRegistry.Default` via their `Providers` property, so you can read and write without touching the registry yet still get SharpCompress’s built-in implementations. + +The configured registry is used consistently across Reader APIs, Writer APIs, Archive APIs, and async entry-stream extraction, including compressed TAR wrappers and ZIP async decompression. + +To replace specific algorithms (for example to use `System.IO.Compression` for GZip or Deflate), create a modified registry and pass it through the same options: + +```C# +var customRegistry = CompressionProviderRegistry.Default + .With(new SystemGZipCompressionProvider()) + .With(new SystemDeflateCompressionProvider()); + +var readerOptions = ReaderOptions.ForFilePath + .WithProviders(customRegistry); +using var reader = ReaderFactory.OpenReader(stream, readerOptions); + +var writerOptions = new WriterOptions(CompressionType.GZip) + .WithProviders(customRegistry); +using var writer = WriterFactory.OpenWriter(outputStream, ArchiveType.GZip, writerOptions); +``` + +The registry is immutable. `With(provider)` returns a new registry and replaces any existing provider for that provider's `CompressionType`. You can inspect or use providers directly with `GetProvider`, `CreateCompressStream`, `CreateDecompressStream`, and their async/context overloads, but most application code should flow the registry through `ReaderOptions` or `WriterOptions` so archive readers and writers can supply the right `CompressionContext`. + +To implement a custom provider, implement `ICompressionProvider` directly or derive from `CompressionProviderBase` for default async methods. Derive from `DecompressionOnlyProviderBase` for read-only codecs. Providers can use `CompressionContext` for stream size, seekability, reader options, compression properties, and format-specific metadata. + +The registry also exposes `GetCompressingProvider` (now returning `ICompressionProviderHooks`) when a compression format needs pre- or post-stream data (e.g., LZMA/PPMd). Implementations that need extra headers can supply those bytes through the `ICompressionProviderHooks` members while the rest of the API still works through the `Providers` property. + +## Async Examples + +### Async Reader Examples + +**Extract single entry asynchronously:** +```C# +using Stream stream = File.OpenRead("archive.zip"); +await using var reader = await ReaderFactory.OpenAsyncReader(stream, cancellationToken: cancellationToken); +while (await reader.MoveToNextEntryAsync(cancellationToken)) +{ + if (!reader.Entry.IsDirectory) + { + using var outputStream = File.Create("output.bin"); + await reader.WriteEntryToAsync(outputStream, cancellationToken); + } +} +``` + +**Extract all entries asynchronously:** +```C# +using Stream stream = File.OpenRead("archive.tar.gz"); +await using var reader = await ReaderFactory.OpenAsyncReader(stream, cancellationToken: cancellationToken); +await reader.WriteAllToDirectoryAsync( + @"D:\temp", + cancellationToken: cancellationToken +); +``` + +**Open and process entry stream asynchronously:** +```C# +await using var archive = await ZipArchive.OpenAsyncArchive("archive.zip", cancellationToken: cancellationToken); +await foreach (var entry in archive.EntriesAsync) +{ + if (!entry.IsDirectory) + { + using var entryStream = await entry.OpenEntryStreamAsync(cancellationToken); + // Process the decompressed stream asynchronously + await ProcessStreamAsync(entryStream, cancellationToken); + } +} +``` + +### Async Writer Examples + +**Write single file asynchronously:** +```C# +using Stream archiveStream = File.OpenWrite("output.zip"); +await using var writer = await WriterFactory.OpenAsyncWriter(archiveStream, ArchiveType.Zip, new WriterOptions(CompressionType.Deflate), cancellationToken); +using Stream fileStream = File.OpenRead("input.txt"); +await writer.WriteAsync("entry.txt", fileStream, DateTime.Now, cancellationToken); +``` + +**Write entire directory asynchronously:** +```C# +using Stream stream = File.OpenWrite("backup.tar.gz"); +await using var writer = await WriterFactory.OpenAsyncWriter(stream, ArchiveType.Tar, new WriterOptions(CompressionType.GZip), cancellationToken); +await writer.WriteAllAsync( + @"D:\files", + "*", + SearchOption.AllDirectories, + cancellationToken +); +``` + +**Write with progress tracking and cancellation:** +```C# +var cts = new CancellationTokenSource(); + +// Set timeout or cancel from UI +cts.CancelAfter(TimeSpan.FromMinutes(5)); + +using Stream stream = File.OpenWrite("archive.zip"); +await using var writer = await WriterFactory.OpenAsyncWriter(stream, ArchiveType.Zip, new WriterOptions(CompressionType.Deflate), cts.Token); +try +{ + await writer.WriteAllAsync(@"D:\data", "*", SearchOption.AllDirectories, cts.Token); +} +catch (OperationCanceledException) +{ + Console.WriteLine("Operation was cancelled"); +} +``` + +### Archive Async Examples + +**Extract from archive asynchronously:** +```C# +await using var archive = await ZipArchive.OpenAsyncArchive("archive.zip", cancellationToken: cancellationToken); +// Simple async extraction - works for all archive types +await archive.WriteToDirectoryAsync( + @"C:\output", + cancellationToken: cancellationToken +); +``` + +**Benefits of Async Operations:** +- Non-blocking I/O for better application responsiveness +- Improved scalability for server applications +- Support for cancellation via CancellationToken +- Better resource utilization in async/await contexts +- Compatible with modern .NET async patterns diff --git a/global.json b/global.json index e6e67e4e..a6dc747f 100644 --- a/global.json +++ b/global.json @@ -1,6 +1,6 @@ { "sdk": { - "version": "7.0.101", - "rollForward": "latestFeature" + "version": "10.0.300", + "rollForward": "disable" } } diff --git a/opencode.json b/opencode.json new file mode 100644 index 00000000..e38cc010 --- /dev/null +++ b/opencode.json @@ -0,0 +1,6 @@ +{ + "$schema": "https://opencode.ai/config.json", + "skills": { + "paths": [".agents/skills"] + } +} diff --git a/reference/APPNOTE.TXT b/reference/APPNOTE.TXT index c89d8a8e..e74b9e0e 100644 --- a/reference/APPNOTE.TXT +++ b/reference/APPNOTE.TXT @@ -1,3497 +1,6 @@ -File: APPNOTE.TXT - .ZIP File Format Specification -Version: 6.3.4 -Status: Final - replaces version 6.3.3 -Revised: October 1, 2014 -Copyright (c) 1989 - 2014 PKWARE Inc., All Rights Reserved. +APPNOTE.TXT is the ZIP file format specification. +Section 1.4 states that PKWARE permits use of this document only for creating products, programs, and processes that read and write ZIP files, and that any reproduction or distribution of the document in whole or in part without prior written permission from PKWARE is strictly prohibited. +This file has therefore been replaced with a reference link. -1.0 Introduction ---------------- - -1.1 Purpose ------------ - - 1.1.1 This specification is intended to define a cross-platform, - interoperable file storage and transfer format. Since its - first publication in 1989, PKWARE, Inc. ("PKWARE") has remained - committed to ensuring the interoperability of the .ZIP file - format through periodic publication and maintenance of this - specification. We trust that all .ZIP compatible vendors and - application developers that use and benefit from this format - will share and support this commitment to interoperability. - -1.2 Scope ---------- - - 1.2.1 ZIP is one of the most widely used compressed file formats. It is - universally used to aggregate, compress, and encrypt files into a single - interoperable container. No specific use or application need is - defined by this format and no specific implementation guidance is - provided. This document provides details on the storage format for - creating ZIP files. Information is provided on the records and - fields that describe what a ZIP file is. - -1.3 Trademarks --------------- - - 1.3.1 PKWARE, PKZIP, SecureZIP, and PKSFX are registered trademarks of - PKWARE, Inc. in the United States and elsewhere. PKPatchMaker, - Deflate64, and ZIP64 are trademarks of PKWARE, Inc. Other marks - referenced within this document appear for identification - purposes only and are the property of their respective owners. - - -1.4 Permitted Use ------------------ - - 1.4.1 This document, "APPNOTE.TXT - .ZIP File Format Specification" is the - exclusive property of PKWARE. Use of the information contained in this - document is permitted solely for the purpose of creating products, - programs and processes that read and write files in the ZIP format - subject to the terms and conditions herein. - - 1.4.2 Use of the content of this document within other publications is - permitted only through reference to this document. Any reproduction - or distribution of this document in whole or in part without prior - written permission from PKWARE is strictly prohibited. - - 1.4.3 Certain technological components provided in this document are the - patented proprietary technology of PKWARE and as such require a - separate, executed license agreement from PKWARE. Applicable - components are marked with the following, or similar, statement: - 'Refer to the section in this document entitled "Incorporating - PKWARE Proprietary Technology into Your Product" for more information'. - -1.5 Contacting PKWARE ---------------------- - - 1.5.1 If you have questions on this format, its use, or licensing, or if you - wish to report defects, request changes or additions, please contact: - - PKWARE, Inc. - 201 E. Pittsburgh Avenue, Suite 400 - Milwaukee, WI 53204 - +1-414-289-9788 - +1-414-289-9789 FAX - zipformat@pkware.com - - 1.5.2 Information about this format and copies of this document are publicly - available at: - - http://www.pkware.com/appnote - -1.6 Disclaimer --------------- - - 1.6.1 Although PKWARE will attempt to supply current and accurate - information relating to its file formats, algorithms, and the - subject programs, the possibility of error or omission cannot - be eliminated. PKWARE therefore expressly disclaims any warranty - that the information contained in the associated materials relating - to the subject programs and/or the format of the files created or - accessed by the subject programs and/or the algorithms used by - the subject programs, or any other matter, is current, correct or - accurate as delivered. Any risk of damage due to any possible - inaccurate information is assumed by the user of the information. - Furthermore, the information relating to the subject programs - and/or the file formats created or accessed by the subject - programs and/or the algorithms used by the subject programs is - subject to change without notice. - -2.0 Revisions --------------- - -2.1 Document Status --------------------- - - 2.1.1 If the STATUS of this file is marked as DRAFT, the content - defines proposed revisions to this specification which may consist - of changes to the ZIP format itself, or that may consist of other - content changes to this document. Versions of this document and - the format in DRAFT form may be subject to modification prior to - publication STATUS of FINAL. DRAFT versions are published periodically - to provide notification to the ZIP community of pending changes and to - provide opportunity for review and comment. - - 2.1.2 Versions of this document having a STATUS of FINAL are - considered to be in the final form for that version of the document - and are not subject to further change until a new, higher version - numbered document is published. Newer versions of this format - specification are intended to remain interoperable with with all prior - versions whenever technically possible. - -2.2 Change Log --------------- - - Version Change Description Date - ------- ------------------ ---------- - 5.2 -Single Password Symmetric Encryption 07/16/2003 - storage - - 6.1.0 -Smartcard compatibility 01/20/2004 - -Documentation on certificate storage - - 6.2.0 -Introduction of Central Directory 04/26/2004 - Encryption for encrypting metadata - -Added OS X to Version Made By values - - 6.2.1 -Added Extra Field placeholder for 04/01/2005 - POSZIP using ID 0x4690 - - -Clarified size field on - "zip64 end of central directory record" - - 6.2.2 -Documented Final Feature Specification 01/06/2006 - for Strong Encryption - - -Clarifications and typographical - corrections - - 6.3.0 -Added tape positioning storage 09/29/2006 - parameters - - -Expanded list of supported hash algorithms - - -Expanded list of supported compression - algorithms - - -Expanded list of supported encryption - algorithms - - -Added option for Unicode filename - storage - - -Clarifications for consistent use - of Data Descriptor records - - -Added additional "Extra Field" - definitions - - 6.3.1 -Corrected standard hash values for 04/11/2007 - SHA-256/384/512 - - 6.3.2 -Added compression method 97 09/28/2007 - - -Documented InfoZIP "Extra Field" - values for UTF-8 file name and - file comment storage - - 6.3.3 -Formatting changes to support 09/01/2012 - easier referencing of this APPNOTE - from other documents and standards - - 6.3.4 -Address change 10/01/2014 - - -3.0 Notations -------------- - - 3.1 Use of the term MUST or SHALL indicates a required element. - - 3.2 MAY NOT or SHALL NOT indicates an element is prohibited from use. - - 3.3 SHOULD indicates a RECOMMENDED element. - - 3.4 SHOULD NOT indicates an element NOT RECOMMENDED for use. - - 3.5 MAY indicates an OPTIONAL element. - - -4.0 ZIP Files -------------- - -4.1 What is a ZIP file ----------------------- - - 4.1.1 ZIP files MAY be identified by the standard .ZIP file extension - although use of a file extension is not required. Use of the - extension .ZIPX is also recognized and MAY be used for ZIP files. - Other common file extensions using the ZIP format include .JAR, .WAR, - .DOCX, .XLXS, .PPTX, .ODT, .ODS, .ODP and others. Programs reading or - writing ZIP files SHOULD rely on internal record signatures described - in this document to identify files in this format. - - 4.1.2 ZIP files SHOULD contain at least one file and MAY contain - multiple files. - - 4.1.3 Data compression MAY be used to reduce the size of files - placed into a ZIP file, but is not required. This format supports the - use of multiple data compression algorithms. When compression is used, - one of the documented compression algorithms MUST be used. Implementors - are advised to experiment with their data to determine which of the - available algorithms provides the best compression for their needs. - Compression method 8 (Deflate) is the method used by default by most - ZIP compatible application programs. - - - 4.1.4 Data encryption MAY be used to protect files within a ZIP file. - Keying methods supported for encryption within this format include - passwords and public/private keys. Either MAY be used individually - or in combination. Encryption MAY be applied to individual files. - Additional security MAY be used through the encryption of ZIP file - metadata stored within the Central Directory. See the section on the - Strong Encryption Specification for information. Refer to the section - in this document entitled "Incorporating PKWARE Proprietary Technology - into Your Product" for more information. - - 4.1.5 Data integrity MUST be provided for each file using CRC32. - - 4.1.6 Additional data integrity MAY be included through the use of - digital signatures. Individual files MAY be signed with one or more - digital signatures. The Central Directory, if signed, MUST use a - single signature. - - 4.1.7 Files MAY be placed within a ZIP file uncompressed or stored. - The term "stored" as used in the context of this document means the file - is copied into the ZIP file uncompressed. - - 4.1.8 Each data file placed into a ZIP file MAY be compressed, stored, - encrypted or digitally signed independent of how other data files in the - same ZIP file are archived. - - 4.1.9 ZIP files MAY be streamed, split into segments (on fixed or on - removable media) or "self-extracting". Self-extracting ZIP - files MUST include extraction code for a target platform within - the ZIP file. - - 4.1.10 Extensibility is provided for platform or application specific - needs through extra data fields that MAY be defined for custom - purposes. Extra data definitions MUST NOT conflict with existing - documented record definitions. - - 4.1.11 Common uses for ZIP MAY also include the use of manifest files. - Manifest files store application specific information within a file stored - within the ZIP file. This manifest file SHOULD be the first file in the - ZIP file. This specification does not provide any information or guidance on - the use of manifest files within ZIP files. Refer to the application developer - for information on using manifest files and for any additional profile - information on using ZIP within an application. - - 4.1.12 ZIP files MAY be placed within other ZIP files. - -4.2 ZIP Metadata ----------------- - - 4.2.1 ZIP files are identified by metadata consisting of defined record types - containing the storage information necessary for maintaining the files - placed into a ZIP file. Each record type MUST be identified using a header - signature that identifies the record type. Signature values begin with the - two byte constant marker of 0x4b50, representing the characters "PK". - - -4.3 General Format of a .ZIP file ---------------------------------- - - 4.3.1 A ZIP file MUST contain an "end of central directory record". A ZIP - file containing only an "end of central directory record" is considered an - empty ZIP file. Files may be added or replaced within a ZIP file, or deleted. - A ZIP file MUST have only one "end of central directory record". Other - records defined in this specification MAY be used as needed to support - storage requirements for individual ZIP files. - - 4.3.2 Each file placed into a ZIP file MUST be preceeded by a "local - file header" record for that file. Each "local file header" MUST be - accompanied by a corresponding "central directory header" record within - the central directory section of the ZIP file. - - 4.3.3 Files MAY be stored in arbitrary order within a ZIP file. A ZIP - file MAY span multiple volumes or it MAY be split into user-defined - segment sizes. All values MUST be stored in little-endian byte order unless - otherwise specified in this document for a specific data element. - - 4.3.4 Compression MUST NOT be applied to a "local file header", an "encryption - header", or an "end of central directory record". Individual "central - directory records" must not be compressed, but the aggregate of all central - directory records MAY be compressed. - - 4.3.5 File data MAY be followed by a "data descriptor" for the file. Data - descriptors are used to facilitate ZIP file streaming. - - - 4.3.6 Overall .ZIP file format: - - [local file header 1] - [encryption header 1] - [file data 1] - [data descriptor 1] - . - . - . - [local file header n] - [encryption header n] - [file data n] - [data descriptor n] - [archive decryption header] - [archive extra data record] - [central directory header 1] - . - . - . - [central directory header n] - [zip64 end of central directory record] - [zip64 end of central directory locator] - [end of central directory record] - - - 4.3.7 Local file header: - - local file header signature 4 bytes (0x04034b50) - version needed to extract 2 bytes - general purpose bit flag 2 bytes - compression method 2 bytes - last mod file time 2 bytes - last mod file date 2 bytes - crc-32 4 bytes - compressed size 4 bytes - uncompressed size 4 bytes - file name length 2 bytes - extra field length 2 bytes - - file name (variable size) - extra field (variable size) - - 4.3.8 File data - - Immediately following the local header for a file - SHOULD be placed the compressed or stored data for the file. - If the file is encrypted, the encryption header for the file - SHOULD be placed after the local header and before the file - data. The series of [local file header][encryption header] - [file data][data descriptor] repeats for each file in the - .ZIP archive. - - Zero-byte files, directories, and other file types that - contain no content MUST not include file data. - - 4.3.9 Data descriptor: - - crc-32 4 bytes - compressed size 4 bytes - uncompressed size 4 bytes - - 4.3.9.1 This descriptor MUST exist if bit 3 of the general - purpose bit flag is set (see below). It is byte aligned - and immediately follows the last byte of compressed data. - This descriptor SHOULD be used only when it was not possible to - seek in the output .ZIP file, e.g., when the output .ZIP file - was standard output or a non-seekable device. For ZIP64(tm) format - archives, the compressed and uncompressed sizes are 8 bytes each. - - 4.3.9.2 When compressing files, compressed and uncompressed sizes - should be stored in ZIP64 format (as 8 byte values) when a - file's size exceeds 0xFFFFFFFF. However ZIP64 format may be - used regardless of the size of a file. When extracting, if - the zip64 extended information extra field is present for - the file the compressed and uncompressed sizes will be 8 - byte values. - - 4.3.9.3 Although not originally assigned a signature, the value - 0x08074b50 has commonly been adopted as a signature value - for the data descriptor record. Implementers should be - aware that ZIP files may be encountered with or without this - signature marking data descriptors and SHOULD account for - either case when reading ZIP files to ensure compatibility. - - 4.3.9.4 When writing ZIP files, implementors SHOULD include the - signature value marking the data descriptor record. When - the signature is used, the fields currently defined for - the data descriptor record will immediately follow the - signature. - - 4.3.9.5 An extensible data descriptor will be released in a - future version of this APPNOTE. This new record is intended to - resolve conflicts with the use of this record going forward, - and to provide better support for streamed file processing. - - 4.3.9.6 When the Central Directory Encryption method is used, - the data descriptor record is not required, but MAY be used. - If present, and bit 3 of the general purpose bit field is set to - indicate its presence, the values in fields of the data descriptor - record MUST be set to binary zeros. See the section on the Strong - Encryption Specification for information. Refer to the section in - this document entitled "Incorporating PKWARE Proprietary Technology - into Your Product" for more information. - - - 4.3.10 Archive decryption header: - - 4.3.10.1 The Archive Decryption Header is introduced in version 6.2 - of the ZIP format specification. This record exists in support - of the Central Directory Encryption Feature implemented as part of - the Strong Encryption Specification as described in this document. - When the Central Directory Structure is encrypted, this decryption - header MUST precede the encrypted data segment. - - 4.3.10.2 The encrypted data segment SHALL consist of the Archive - extra data record (if present) and the encrypted Central Directory - Structure data. The format of this data record is identical to the - Decryption header record preceding compressed file data. If the - central directory structure is encrypted, the location of the start of - this data record is determined using the Start of Central Directory - field in the Zip64 End of Central Directory record. See the - section on the Strong Encryption Specification for information - on the fields used in the Archive Decryption Header record. - Refer to the section in this document entitled "Incorporating - PKWARE Proprietary Technology into Your Product" for more information. - - - 4.3.11 Archive extra data record: - - archive extra data signature 4 bytes (0x08064b50) - extra field length 4 bytes - extra field data (variable size) - - 4.3.11.1 The Archive Extra Data Record is introduced in version 6.2 - of the ZIP format specification. This record MAY be used in support - of the Central Directory Encryption Feature implemented as part of - the Strong Encryption Specification as described in this document. - When present, this record MUST immediately precede the central - directory data structure. - - 4.3.11.2 The size of this data record SHALL be included in the - Size of the Central Directory field in the End of Central - Directory record. If the central directory structure is compressed, - but not encrypted, the location of the start of this data record is - determined using the Start of Central Directory field in the Zip64 - End of Central Directory record. Refer to the section in this document - entitled "Incorporating PKWARE Proprietary Technology into Your - Product" for more information. - - 4.3.12 Central directory structure: - - [central directory header 1] - . - . - . - [central directory header n] - [digital signature] - - File header: - - central file header signature 4 bytes (0x02014b50) - version made by 2 bytes - version needed to extract 2 bytes - general purpose bit flag 2 bytes - compression method 2 bytes - last mod file time 2 bytes - last mod file date 2 bytes - crc-32 4 bytes - compressed size 4 bytes - uncompressed size 4 bytes - file name length 2 bytes - extra field length 2 bytes - file comment length 2 bytes - disk number start 2 bytes - internal file attributes 2 bytes - external file attributes 4 bytes - relative offset of local header 4 bytes - - file name (variable size) - extra field (variable size) - file comment (variable size) - - 4.3.13 Digital signature: - - header signature 4 bytes (0x05054b50) - size of data 2 bytes - signature data (variable size) - - With the introduction of the Central Directory Encryption - feature in version 6.2 of this specification, the Central - Directory Structure MAY be stored both compressed and encrypted. - Although not required, it is assumed when encrypting the - Central Directory Structure, that it will be compressed - for greater storage efficiency. Information on the - Central Directory Encryption feature can be found in the section - describing the Strong Encryption Specification. The Digital - Signature record will be neither compressed nor encrypted. - - 4.3.14 Zip64 end of central directory record - - zip64 end of central dir - signature 4 bytes (0x06064b50) - size of zip64 end of central - directory record 8 bytes - version made by 2 bytes - version needed to extract 2 bytes - number of this disk 4 bytes - number of the disk with the - start of the central directory 4 bytes - total number of entries in the - central directory on this disk 8 bytes - total number of entries in the - central directory 8 bytes - size of the central directory 8 bytes - offset of start of central - directory with respect to - the starting disk number 8 bytes - zip64 extensible data sector (variable size) - - 4.3.14.1 The value stored into the "size of zip64 end of central - directory record" should be the size of the remaining - record and should not include the leading 12 bytes. - - Size = SizeOfFixedFields + SizeOfVariableData - 12. - - 4.3.14.2 The above record structure defines Version 1 of the - zip64 end of central directory record. Version 1 was - implemented in versions of this specification preceding - 6.2 in support of the ZIP64 large file feature. The - introduction of the Central Directory Encryption feature - implemented in version 6.2 as part of the Strong Encryption - Specification defines Version 2 of this record structure. - Refer to the section describing the Strong Encryption - Specification for details on the version 2 format for - this record. Refer to the section in this document entitled - "Incorporating PKWARE Proprietary Technology into Your Product" - for more information applicable to use of Version 2 of this - record. - - 4.3.14.3 Special purpose data MAY reside in the zip64 extensible - data sector field following either a V1 or V2 version of this - record. To ensure identification of this special purpose data - it must include an identifying header block consisting of the - following: - - Header ID - 2 bytes - Data Size - 4 bytes - - The Header ID field indicates the type of data that is in the - data block that follows. - - Data Size identifies the number of bytes that follow for this - data block type. - - 4.3.14.4 Multiple special purpose data blocks MAY be present. - Each MUST be preceded by a Header ID and Data Size field. Current - mappings of Header ID values supported in this field are as - defined in APPENDIX C. - - 4.3.15 Zip64 end of central directory locator - - zip64 end of central dir locator - signature 4 bytes (0x07064b50) - number of the disk with the - start of the zip64 end of - central directory 4 bytes - relative offset of the zip64 - end of central directory record 8 bytes - total number of disks 4 bytes - - 4.3.16 End of central directory record: - - end of central dir signature 4 bytes (0x06054b50) - number of this disk 2 bytes - number of the disk with the - start of the central directory 2 bytes - total number of entries in the - central directory on this disk 2 bytes - total number of entries in - the central directory 2 bytes - size of the central directory 4 bytes - offset of start of central - directory with respect to - the starting disk number 4 bytes - .ZIP file comment length 2 bytes - .ZIP file comment (variable size) - -4.4 Explanation of fields --------------------------- - - 4.4.1 General notes on fields - - 4.4.1.1 All fields unless otherwise noted are unsigned and stored - in Intel low-byte:high-byte, low-word:high-word order. - - 4.4.1.2 String fields are not null terminated, since the length - is given explicitly. - - 4.4.1.3 The entries in the central directory may not necessarily - be in the same order that files appear in the .ZIP file. - - 4.4.1.4 If one of the fields in the end of central directory - record is too small to hold required data, the field should be - set to -1 (0xFFFF or 0xFFFFFFFF) and the ZIP64 format record - should be created. - - 4.4.1.5 The end of central directory record and the Zip64 end - of central directory locator record MUST reside on the same - disk when splitting or spanning an archive. - - 4.4.2 version made by (2 bytes) - - 4.4.2.1 The upper byte indicates the compatibility of the file - attribute information. If the external file attributes - are compatible with MS-DOS and can be read by PKZIP for - DOS version 2.04g then this value will be zero. If these - attributes are not compatible, then this value will - identify the host system on which the attributes are - compatible. Software can use this information to determine - the line record format for text files etc. - - 4.4.2.2 The current mappings are: - - 0 - MS-DOS and OS/2 (FAT / VFAT / FAT32 file systems) - 1 - Amiga 2 - OpenVMS - 3 - UNIX 4 - VM/CMS - 5 - Atari ST 6 - OS/2 H.P.F.S. - 7 - Macintosh 8 - Z-System - 9 - CP/M 10 - Windows NTFS - 11 - MVS (OS/390 - Z/OS) 12 - VSE - 13 - Acorn Risc 14 - VFAT - 15 - alternate MVS 16 - BeOS - 17 - Tandem 18 - OS/400 - 19 - OS X (Darwin) 20 thru 255 - unused - - 4.4.2.3 The lower byte indicates the ZIP specification version - (the version of this document) supported by the software - used to encode the file. The value/10 indicates the major - version number, and the value mod 10 is the minor version - number. - - 4.4.3 version needed to extract (2 bytes) - - 4.4.3.1 The minimum supported ZIP specification version needed - to extract the file, mapped as above. This value is based on - the specific format features a ZIP program MUST support to - be able to extract the file. If multiple features are - applied to a file, the minimum version MUST be set to the - feature having the highest value. New features or feature - changes affecting the published format specification will be - implemented using higher version numbers than the last - published value to avoid conflict. - - 4.4.3.2 Current minimum feature versions are as defined below: - - 1.0 - Default value - 1.1 - File is a volume label - 2.0 - File is a folder (directory) - 2.0 - File is compressed using Deflate compression - 2.0 - File is encrypted using traditional PKWARE encryption - 2.1 - File is compressed using Deflate64(tm) - 2.5 - File is compressed using PKWARE DCL Implode - 2.7 - File is a patch data set - 4.5 - File uses ZIP64 format extensions - 4.6 - File is compressed using BZIP2 compression* - 5.0 - File is encrypted using DES - 5.0 - File is encrypted using 3DES - 5.0 - File is encrypted using original RC2 encryption - 5.0 - File is encrypted using RC4 encryption - 5.1 - File is encrypted using AES encryption - 5.1 - File is encrypted using corrected RC2 encryption** - 5.2 - File is encrypted using corrected RC2-64 encryption** - 6.1 - File is encrypted using non-OAEP key wrapping*** - 6.2 - Central directory encryption - 6.3 - File is compressed using LZMA - 6.3 - File is compressed using PPMd+ - 6.3 - File is encrypted using Blowfish - 6.3 - File is encrypted using Twofish - - 4.4.3.3 Notes on version needed to extract - - * Early 7.x (pre-7.2) versions of PKZIP incorrectly set the - version needed to extract for BZIP2 compression to be 50 - when it should have been 46. - - ** Refer to the section on Strong Encryption Specification - for additional information regarding RC2 corrections. - - *** Certificate encryption using non-OAEP key wrapping is the - intended mode of operation for all versions beginning with 6.1. - Support for OAEP key wrapping MUST only be used for - backward compatibility when sending ZIP files to be opened by - versions of PKZIP older than 6.1 (5.0 or 6.0). - - + Files compressed using PPMd MUST set the version - needed to extract field to 6.3, however, not all ZIP - programs enforce this and may be unable to decompress - data files compressed using PPMd if this value is set. - - When using ZIP64 extensions, the corresponding value in the - zip64 end of central directory record MUST also be set. - This field should be set appropriately to indicate whether - Version 1 or Version 2 format is in use. - - - 4.4.4 general purpose bit flag: (2 bytes) - - Bit 0: If set, indicates that the file is encrypted. - - (For Method 6 - Imploding) - Bit 1: If the compression method used was type 6, - Imploding, then this bit, if set, indicates - an 8K sliding dictionary was used. If clear, - then a 4K sliding dictionary was used. - - Bit 2: If the compression method used was type 6, - Imploding, then this bit, if set, indicates - 3 Shannon-Fano trees were used to encode the - sliding dictionary output. If clear, then 2 - Shannon-Fano trees were used. - - (For Methods 8 and 9 - Deflating) - Bit 2 Bit 1 - 0 0 Normal (-en) compression option was used. - 0 1 Maximum (-exx/-ex) compression option was used. - 1 0 Fast (-ef) compression option was used. - 1 1 Super Fast (-es) compression option was used. - - (For Method 14 - LZMA) - Bit 1: If the compression method used was type 14, - LZMA, then this bit, if set, indicates - an end-of-stream (EOS) marker is used to - mark the end of the compressed data stream. - If clear, then an EOS marker is not present - and the compressed data size must be known - to extract. - - Note: Bits 1 and 2 are undefined if the compression - method is any other. - - Bit 3: If this bit is set, the fields crc-32, compressed - size and uncompressed size are set to zero in the - local header. The correct values are put in the - data descriptor immediately following the compressed - data. (Note: PKZIP version 2.04g for DOS only - recognizes this bit for method 8 compression, newer - versions of PKZIP recognize this bit for any - compression method.) - - Bit 4: Reserved for use with method 8, for enhanced - deflating. - - Bit 5: If this bit is set, this indicates that the file is - compressed patched data. (Note: Requires PKZIP - version 2.70 or greater) - - Bit 6: Strong encryption. If this bit is set, you MUST - set the version needed to extract value to at least - 50 and you MUST also set bit 0. If AES encryption - is used, the version needed to extract value MUST - be at least 51. See the section describing the Strong - Encryption Specification for details. Refer to the - section in this document entitled "Incorporating PKWARE - Proprietary Technology into Your Product" for more - information. - - Bit 7: Currently unused. - - Bit 8: Currently unused. - - Bit 9: Currently unused. - - Bit 10: Currently unused. - - Bit 11: Language encoding flag (EFS). If this bit is set, - the filename and comment fields for this file - MUST be encoded using UTF-8. (see APPENDIX D) - - Bit 12: Reserved by PKWARE for enhanced compression. - - Bit 13: Set when encrypting the Central Directory to indicate - selected data values in the Local Header are masked to - hide their actual values. See the section describing - the Strong Encryption Specification for details. Refer - to the section in this document entitled "Incorporating - PKWARE Proprietary Technology into Your Product" for - more information. - - Bit 14: Reserved by PKWARE. - - Bit 15: Reserved by PKWARE. - - 4.4.5 compression method: (2 bytes) - - 0 - The file is stored (no compression) - 1 - The file is Shrunk - 2 - The file is Reduced with compression factor 1 - 3 - The file is Reduced with compression factor 2 - 4 - The file is Reduced with compression factor 3 - 5 - The file is Reduced with compression factor 4 - 6 - The file is Imploded - 7 - Reserved for Tokenizing compression algorithm - 8 - The file is Deflated - 9 - Enhanced Deflating using Deflate64(tm) - 10 - PKWARE Data Compression Library Imploding (old IBM TERSE) - 11 - Reserved by PKWARE - 12 - File is compressed using BZIP2 algorithm - 13 - Reserved by PKWARE - 14 - LZMA (EFS) - 15 - Reserved by PKWARE - 16 - Reserved by PKWARE - 17 - Reserved by PKWARE - 18 - File is compressed using IBM TERSE (new) - 19 - IBM LZ77 z Architecture (PFS) - 97 - WavPack compressed data - 98 - PPMd version I, Rev 1 - - - 4.4.6 date and time fields: (2 bytes each) - - The date and time are encoded in standard MS-DOS format. - If input came from standard input, the date and time are - those at which compression was started for this data. - If encrypting the central directory and general purpose bit - flag 13 is set indicating masking, the value stored in the - Local Header will be zero. - - 4.4.7 CRC-32: (4 bytes) - - The CRC-32 algorithm was generously contributed by - David Schwaderer and can be found in his excellent - book "C Programmers Guide to NetBIOS" published by - Howard W. Sams & Co. Inc. The 'magic number' for - the CRC is 0xdebb20e3. The proper CRC pre and post - conditioning is used, meaning that the CRC register - is pre-conditioned with all ones (a starting value - of 0xffffffff) and the value is post-conditioned by - taking the one's complement of the CRC residual. - If bit 3 of the general purpose flag is set, this - field is set to zero in the local header and the correct - value is put in the data descriptor and in the central - directory. When encrypting the central directory, if the - local header is not in ZIP64 format and general purpose - bit flag 13 is set indicating masking, the value stored - in the Local Header will be zero. - - 4.4.8 compressed size: (4 bytes) - 4.4.9 uncompressed size: (4 bytes) - - The size of the file compressed (4.4.8) and uncompressed, - (4.4.9) respectively. When a decryption header is present it - will be placed in front of the file data and the value of the - compressed file size will include the bytes of the decryption - header. If bit 3 of the general purpose bit flag is set, - these fields are set to zero in the local header and the - correct values are put in the data descriptor and - in the central directory. If an archive is in ZIP64 format - and the value in this field is 0xFFFFFFFF, the size will be - in the corresponding 8 byte ZIP64 extended information - extra field. When encrypting the central directory, if the - local header is not in ZIP64 format and general purpose bit - flag 13 is set indicating masking, the value stored for the - uncompressed size in the Local Header will be zero. - - 4.4.10 file name length: (2 bytes) - 4.4.11 extra field length: (2 bytes) - 4.4.12 file comment length: (2 bytes) - - The length of the file name, extra field, and comment - fields respectively. The combined length of any - directory record and these three fields should not - generally exceed 65,535 bytes. If input came from standard - input, the file name length is set to zero. - - - 4.4.13 disk number start: (2 bytes) - - The number of the disk on which this file begins. If an - archive is in ZIP64 format and the value in this field is - 0xFFFF, the size will be in the corresponding 4 byte zip64 - extended information extra field. - - 4.4.14 internal file attributes: (2 bytes) - - Bits 1 and 2 are reserved for use by PKWARE. - - 4.4.14.1 The lowest bit of this field indicates, if set, - that the file is apparently an ASCII or text file. If not - set, that the file apparently contains binary data. - The remaining bits are unused in version 1.0. - - 4.4.14.2 The 0x0002 bit of this field indicates, if set, that - a 4 byte variable record length control field precedes each - logical record indicating the length of the record. The - record length control field is stored in little-endian byte - order. This flag is independent of text control characters, - and if used in conjunction with text data, includes any - control characters in the total length of the record. This - value is provided for mainframe data transfer support. - - 4.4.15 external file attributes: (4 bytes) - - The mapping of the external attributes is - host-system dependent (see 'version made by'). For - MS-DOS, the low order byte is the MS-DOS directory - attribute byte. If input came from standard input, this - field is set to zero. - - 4.4.16 relative offset of local header: (4 bytes) - - This is the offset from the start of the first disk on - which this file appears, to where the local header should - be found. If an archive is in ZIP64 format and the value - in this field is 0xFFFFFFFF, the size will be in the - corresponding 8 byte zip64 extended information extra field. - - 4.4.17 file name: (Variable) - - 4.4.17.1 The name of the file, with optional relative path. - The path stored MUST not contain a drive or - device letter, or a leading slash. All slashes - MUST be forward slashes '/' as opposed to - backwards slashes '\' for compatibility with Amiga - and UNIX file systems etc. If input came from standard - input, there is no file name field. - - 4.4.17.2 If using the Central Directory Encryption Feature and - general purpose bit flag 13 is set indicating masking, the file - name stored in the Local Header will not be the actual file name. - A masking value consisting of a unique hexadecimal value will - be stored. This value will be sequentially incremented for each - file in the archive. See the section on the Strong Encryption - Specification for details on retrieving the encrypted file name. - Refer to the section in this document entitled "Incorporating PKWARE - Proprietary Technology into Your Product" for more information. - - - 4.4.18 file comment: (Variable) - - The comment for this file. - - 4.4.19 number of this disk: (2 bytes) - - The number of this disk, which contains central - directory end record. If an archive is in ZIP64 format - and the value in this field is 0xFFFF, the size will - be in the corresponding 4 byte zip64 end of central - directory field. - - - 4.4.20 number of the disk with the start of the central - directory: (2 bytes) - - The number of the disk on which the central - directory starts. If an archive is in ZIP64 format - and the value in this field is 0xFFFF, the size will - be in the corresponding 4 byte zip64 end of central - directory field. - - 4.4.21 total number of entries in the central dir on - this disk: (2 bytes) - - The number of central directory entries on this disk. - If an archive is in ZIP64 format and the value in - this field is 0xFFFF, the size will be in the - corresponding 8 byte zip64 end of central - directory field. - - 4.4.22 total number of entries in the central dir: (2 bytes) - - The total number of files in the .ZIP file. If an - archive is in ZIP64 format and the value in this field - is 0xFFFF, the size will be in the corresponding 8 byte - zip64 end of central directory field. - - 4.4.23 size of the central directory: (4 bytes) - - The size (in bytes) of the entire central directory. - If an archive is in ZIP64 format and the value in - this field is 0xFFFFFFFF, the size will be in the - corresponding 8 byte zip64 end of central - directory field. - - 4.4.24 offset of start of central directory with respect to - the starting disk number: (4 bytes) - - Offset of the start of the central directory on the - disk on which the central directory starts. If an - archive is in ZIP64 format and the value in this - field is 0xFFFFFFFF, the size will be in the - corresponding 8 byte zip64 end of central - directory field. - - 4.4.25 .ZIP file comment length: (2 bytes) - - The length of the comment for this .ZIP file. - - 4.4.26 .ZIP file comment: (Variable) - - The comment for this .ZIP file. ZIP file comment data - is stored unsecured. No encryption or data authentication - is applied to this area at this time. Confidential information - should not be stored in this section. - - 4.4.27 zip64 extensible data sector (variable size) - - (currently reserved for use by PKWARE) - - - 4.4.28 extra field: (Variable) - - This SHOULD be used for storage expansion. If additional - information needs to be stored within a ZIP file for special - application or platform needs, it SHOULD be stored here. - Programs supporting earlier versions of this specification can - then safely skip the file, and find the next file or header. - This field will be 0 length in version 1.0. - - Existing extra fields are defined in the section - Extensible data fields that follows. - -4.5 Extensible data fields --------------------------- - - 4.5.1 In order to allow different programs and different types - of information to be stored in the 'extra' field in .ZIP - files, the following structure MUST be used for all - programs storing data in this field: - - header1+data1 + header2+data2 . . . - - Each header should consist of: - - Header ID - 2 bytes - Data Size - 2 bytes - - Note: all fields stored in Intel low-byte/high-byte order. - - The Header ID field indicates the type of data that is in - the following data block. - - Header IDs of 0 thru 31 are reserved for use by PKWARE. - The remaining IDs can be used by third party vendors for - proprietary usage. - - 4.5.2 The current Header ID mappings defined by PKWARE are: - - 0x0001 Zip64 extended information extra field - 0x0007 AV Info - 0x0008 Reserved for extended language encoding data (PFS) - (see APPENDIX D) - 0x0009 OS/2 - 0x000a NTFS - 0x000c OpenVMS - 0x000d UNIX - 0x000e Reserved for file stream and fork descriptors - 0x000f Patch Descriptor - 0x0014 PKCS#7 Store for X.509 Certificates - 0x0015 X.509 Certificate ID and Signature for - individual file - 0x0016 X.509 Certificate ID for Central Directory - 0x0017 Strong Encryption Header - 0x0018 Record Management Controls - 0x0019 PKCS#7 Encryption Recipient Certificate List - 0x0065 IBM S/390 (Z390), AS/400 (I400) attributes - - uncompressed - 0x0066 Reserved for IBM S/390 (Z390), AS/400 (I400) - attributes - compressed - 0x4690 POSZIP 4690 (reserved) - - - 4.5.3 -Zip64 Extended Information Extra Field (0x0001): - - The following is the layout of the zip64 extended - information "extra" block. If one of the size or - offset fields in the Local or Central directory - record is too small to hold the required data, - a Zip64 extended information record is created. - The order of the fields in the zip64 extended - information record is fixed, but the fields MUST - only appear if the corresponding Local or Central - directory record field is set to 0xFFFF or 0xFFFFFFFF. - - Note: all fields stored in Intel low-byte/high-byte order. - - Value Size Description - ----- ---- ----------- -(ZIP64) 0x0001 2 bytes Tag for this "extra" block type - Size 2 bytes Size of this "extra" block - Original - Size 8 bytes Original uncompressed file size - Compressed - Size 8 bytes Size of compressed data - Relative Header - Offset 8 bytes Offset of local header record - Disk Start - Number 4 bytes Number of the disk on which - this file starts - - This entry in the Local header MUST include BOTH original - and compressed file size fields. If encrypting the - central directory and bit 13 of the general purpose bit - flag is set indicating masking, the value stored in the - Local Header for the original file size will be zero. - - - 4.5.4 -OS/2 Extra Field (0x0009): - - The following is the layout of the OS/2 attributes "extra" - block. (Last Revision 09/05/95) - - Note: all fields stored in Intel low-byte/high-byte order. - - Value Size Description - ----- ---- ----------- -(OS/2) 0x0009 2 bytes Tag for this "extra" block type - TSize 2 bytes Size for the following data block - BSize 4 bytes Uncompressed Block Size - CType 2 bytes Compression type - EACRC 4 bytes CRC value for uncompress block - (var) variable Compressed block - - The OS/2 extended attribute structure (FEA2LIST) is - compressed and then stored in its entirety within this - structure. There will only ever be one "block" of data in - VarFields[]. - - 4.5.5 -NTFS Extra Field (0x000a): - - The following is the layout of the NTFS attributes - "extra" block. (Note: At this time the Mtime, Atime - and Ctime values MAY be used on any WIN32 system.) - - Note: all fields stored in Intel low-byte/high-byte order. - - Value Size Description - ----- ---- ----------- -(NTFS) 0x000a 2 bytes Tag for this "extra" block type - TSize 2 bytes Size of the total "extra" block - Reserved 4 bytes Reserved for future use - Tag1 2 bytes NTFS attribute tag value #1 - Size1 2 bytes Size of attribute #1, in bytes - (var) Size1 Attribute #1 data - . - . - . - TagN 2 bytes NTFS attribute tag value #N - SizeN 2 bytes Size of attribute #N, in bytes - (var) SizeN Attribute #N data - - For NTFS, values for Tag1 through TagN are as follows: - (currently only one set of attributes is defined for NTFS) - - Tag Size Description - ----- ---- ----------- - 0x0001 2 bytes Tag for attribute #1 - Size1 2 bytes Size of attribute #1, in bytes - Mtime 8 bytes File last modification time - Atime 8 bytes File last access time - Ctime 8 bytes File creation time - - 4.5.6 -OpenVMS Extra Field (0x000c): - - The following is the layout of the OpenVMS attributes - "extra" block. - - Note: all fields stored in Intel low-byte/high-byte order. - - Value Size Description - ----- ---- ----------- - (VMS) 0x000c 2 bytes Tag for this "extra" block type - TSize 2 bytes Size of the total "extra" block - CRC 4 bytes 32-bit CRC for remainder of the block - Tag1 2 bytes OpenVMS attribute tag value #1 - Size1 2 bytes Size of attribute #1, in bytes - (var) Size1 Attribute #1 data - . - . - . - TagN 2 bytes OpenVMS attribute tag value #N - SizeN 2 bytes Size of attribute #N, in bytes - (var) SizeN Attribute #N data - - OpenVMS Extra Field Rules: - - 4.5.6.1. There will be one or more attributes present, which - will each be preceded by the above TagX & SizeX values. - These values are identical to the ATR$C_XXXX and ATR$S_XXXX - constants which are defined in ATR.H under OpenVMS C. Neither - of these values will ever be zero. - - 4.5.6.2. No word alignment or padding is performed. - - 4.5.6.3. A well-behaved PKZIP/OpenVMS program should never produce - more than one sub-block with the same TagX value. Also, there will - never be more than one "extra" block of type 0x000c in a particular - directory record. - - 4.5.7 -UNIX Extra Field (0x000d): - - The following is the layout of the UNIX "extra" block. - Note: all fields are stored in Intel low-byte/high-byte - order. - - Value Size Description - ----- ---- ----------- -(UNIX) 0x000d 2 bytes Tag for this "extra" block type - TSize 2 bytes Size for the following data block - Atime 4 bytes File last access time - Mtime 4 bytes File last modification time - Uid 2 bytes File user ID - Gid 2 bytes File group ID - (var) variable Variable length data field - - The variable length data field will contain file type - specific data. Currently the only values allowed are - the original "linked to" file names for hard or symbolic - links, and the major and minor device node numbers for - character and block device nodes. Since device nodes - cannot be either symbolic or hard links, only one set of - variable length data is stored. Link files will have the - name of the original file stored. This name is NOT NULL - terminated. Its size can be determined by checking TSize - - 12. Device entries will have eight bytes stored as two 4 - byte entries (in little endian format). The first entry - will be the major device number, and the second the minor - device number. - - 4.5.8 -PATCH Descriptor Extra Field (0x000f): - - 4.5.8.1 The following is the layout of the Patch Descriptor - "extra" block. - - Note: all fields stored in Intel low-byte/high-byte order. - - Value Size Description - ----- ---- ----------- -(Patch) 0x000f 2 bytes Tag for this "extra" block type - TSize 2 bytes Size of the total "extra" block - Version 2 bytes Version of the descriptor - Flags 4 bytes Actions and reactions (see below) - OldSize 4 bytes Size of the file about to be patched - OldCRC 4 bytes 32-bit CRC of the file to be patched - NewSize 4 bytes Size of the resulting file - NewCRC 4 bytes 32-bit CRC of the resulting file - - 4.5.8.2 Actions and reactions - - Bits Description - ---- ---------------- - 0 Use for auto detection - 1 Treat as a self-patch - 2-3 RESERVED - 4-5 Action (see below) - 6-7 RESERVED - 8-9 Reaction (see below) to absent file - 10-11 Reaction (see below) to newer file - 12-13 Reaction (see below) to unknown file - 14-15 RESERVED - 16-31 RESERVED - - 4.5.8.2.1 Actions - - Action Value - ------ ----- - none 0 - add 1 - delete 2 - patch 3 - - 4.5.8.2.2 Reactions - - Reaction Value - -------- ----- - ask 0 - skip 1 - ignore 2 - fail 3 - - 4.5.8.3 Patch support is provided by PKPatchMaker(tm) technology - and is covered under U.S. Patents and Patents Pending. The use or - implementation in a product of certain technological aspects set - forth in the current APPNOTE, including those with regard to - strong encryption or patching requires a license from PKWARE. - Refer to the section in this document entitled "Incorporating - PKWARE Proprietary Technology into Your Product" for more - information. - - 4.5.9 -PKCS#7 Store for X.509 Certificates (0x0014): - - This field MUST contain information about each of the certificates - files may be signed with. When the Central Directory Encryption - feature is enabled for a ZIP file, this record will appear in - the Archive Extra Data Record, otherwise it will appear in the - first central directory record and will be ignored in any - other record. - - - Note: all fields stored in Intel low-byte/high-byte order. - - Value Size Description - ----- ---- ----------- -(Store) 0x0014 2 bytes Tag for this "extra" block type - TSize 2 bytes Size of the store data - TData TSize Data about the store - - - 4.5.10 -X.509 Certificate ID and Signature for individual file (0x0015): - - This field contains the information about which certificate in - the PKCS#7 store was used to sign a particular file. It also - contains the signature data. This field can appear multiple - times, but can only appear once per certificate. - - Note: all fields stored in Intel low-byte/high-byte order. - - Value Size Description - ----- ---- ----------- -(CID) 0x0015 2 bytes Tag for this "extra" block type - TSize 2 bytes Size of data that follows - TData TSize Signature Data - - 4.5.11 -X.509 Certificate ID and Signature for central directory (0x0016): - - This field contains the information about which certificate in - the PKCS#7 store was used to sign the central directory structure. - When the Central Directory Encryption feature is enabled for a - ZIP file, this record will appear in the Archive Extra Data Record, - otherwise it will appear in the first central directory record. - - Note: all fields stored in Intel low-byte/high-byte order. - - Value Size Description - ----- ---- ----------- -(CDID) 0x0016 2 bytes Tag for this "extra" block type - TSize 2 bytes Size of data that follows - TData TSize Data - - 4.5.12 -Strong Encryption Header (0x0017): - - Value Size Description - ----- ---- ----------- - 0x0017 2 bytes Tag for this "extra" block type - TSize 2 bytes Size of data that follows - Format 2 bytes Format definition for this record - AlgID 2 bytes Encryption algorithm identifier - Bitlen 2 bytes Bit length of encryption key - Flags 2 bytes Processing flags - CertData TSize-8 Certificate decryption extra field data - (refer to the explanation for CertData - in the section describing the - Certificate Processing Method under - the Strong Encryption Specification) - - See the section describing the Strong Encryption Specification - for details. Refer to the section in this document entitled - "Incorporating PKWARE Proprietary Technology into Your Product" - for more information. - - 4.5.13 -Record Management Controls (0x0018): - - Value Size Description - ----- ---- ----------- -(Rec-CTL) 0x0018 2 bytes Tag for this "extra" block type - CSize 2 bytes Size of total extra block data - Tag1 2 bytes Record control attribute 1 - Size1 2 bytes Size of attribute 1, in bytes - Data1 Size1 Attribute 1 data - . - . - . - TagN 2 bytes Record control attribute N - SizeN 2 bytes Size of attribute N, in bytes - DataN SizeN Attribute N data - - - 4.5.14 -PKCS#7 Encryption Recipient Certificate List (0x0019): - - This field MAY contain information about each of the certificates - used in encryption processing and it can be used to identify who is - allowed to decrypt encrypted files. This field should only appear - in the archive extra data record. This field is not required and - serves only to aid archive modifications by preserving public - encryption key data. Individual security requirements may dictate - that this data be omitted to deter information exposure. - - Note: all fields stored in Intel low-byte/high-byte order. - - Value Size Description - ----- ---- ----------- -(CStore) 0x0019 2 bytes Tag for this "extra" block type - TSize 2 bytes Size of the store data - TData TSize Data about the store - - TData: - - Value Size Description - ----- ---- ----------- - Version 2 bytes Format version number - must 0x0001 at this time - CStore (var) PKCS#7 data blob - - See the section describing the Strong Encryption Specification - for details. Refer to the section in this document entitled - "Incorporating PKWARE Proprietary Technology into Your Product" - for more information. - - 4.5.15 -MVS Extra Field (0x0065): - - The following is the layout of the MVS "extra" block. - Note: Some fields are stored in Big Endian format. - All text is in EBCDIC format unless otherwise specified. - - Value Size Description - ----- ---- ----------- -(MVS) 0x0065 2 bytes Tag for this "extra" block type - TSize 2 bytes Size for the following data block - ID 4 bytes EBCDIC "Z390" 0xE9F3F9F0 or - "T4MV" for TargetFour - (var) TSize-4 Attribute data (see APPENDIX B) - - - 4.5.16 -OS/400 Extra Field (0x0065): - - The following is the layout of the OS/400 "extra" block. - Note: Some fields are stored in Big Endian format. - All text is in EBCDIC format unless otherwise specified. - - Value Size Description - ----- ---- ----------- -(OS400) 0x0065 2 bytes Tag for this "extra" block type - TSize 2 bytes Size for the following data block - ID 4 bytes EBCDIC "I400" 0xC9F4F0F0 or - "T4MV" for TargetFour - (var) TSize-4 Attribute data (see APPENDIX A) - -4.6 Third Party Mappings ------------------------- - - 4.6.1 Third party mappings commonly used are: - - 0x07c8 Macintosh - 0x2605 ZipIt Macintosh - 0x2705 ZipIt Macintosh 1.3.5+ - 0x2805 ZipIt Macintosh 1.3.5+ - 0x334d Info-ZIP Macintosh - 0x4341 Acorn/SparkFS - 0x4453 Windows NT security descriptor (binary ACL) - 0x4704 VM/CMS - 0x470f MVS - 0x4b46 FWKCS MD5 (see below) - 0x4c41 OS/2 access control list (text ACL) - 0x4d49 Info-ZIP OpenVMS - 0x4f4c Xceed original location extra field - 0x5356 AOS/VS (ACL) - 0x5455 extended timestamp - 0x554e Xceed unicode extra field - 0x5855 Info-ZIP UNIX (original, also OS/2, NT, etc) - 0x6375 Info-ZIP Unicode Comment Extra Field - 0x6542 BeOS/BeBox - 0x7075 Info-ZIP Unicode Path Extra Field - 0x756e ASi UNIX - 0x7855 Info-ZIP UNIX (new) - 0xa220 Microsoft Open Packaging Growth Hint - 0xfd4a SMS/QDOS - - Detailed descriptions of Extra Fields defined by third - party mappings will be documented as information on - these data structures is made available to PKWARE. - PKWARE does not guarantee the accuracy of any published - third party data. - - 4.6.2 Third-party Extra Fields must include a Header ID using - the format defined in the section of this document - titled Extensible Data Fields (section 4.5). - - The Data Size field indicates the size of the following - data block. Programs can use this value to skip to the - next header block, passing over any data blocks that are - not of interest. - - Note: As stated above, the size of the entire .ZIP file - header, including the file name, comment, and extra - field should not exceed 64K in size. - - 4.6.3 In case two different programs should appropriate the same - Header ID value, it is strongly recommended that each - program SHOULD place a unique signature of at least two bytes in - size (and preferably 4 bytes or bigger) at the start of - each data area. Every program SHOULD verify that its - unique signature is present, in addition to the Header ID - value being correct, before assuming that it is a block of - known type. - - Third-party Mappings: - - 4.6.4 -ZipIt Macintosh Extra Field (long) (0x2605): - - The following is the layout of the ZipIt extra block - for Macintosh. The local-header and central-header versions - are identical. This block must be present if the file is - stored MacBinary-encoded and it should not be used if the file - is not stored MacBinary-encoded. - - Value Size Description - ----- ---- ----------- - (Mac2) 0x2605 Short tag for this extra block type - TSize Short total data size for this block - "ZPIT" beLong extra-field signature - FnLen Byte length of FileName - FileName variable full Macintosh filename - FileType Byte[4] four-byte Mac file type string - Creator Byte[4] four-byte Mac creator string - - - 4.6.5 -ZipIt Macintosh Extra Field (short, for files) (0x2705): - - The following is the layout of a shortened variant of the - ZipIt extra block for Macintosh (without "full name" entry). - This variant is used by ZipIt 1.3.5 and newer for entries of - files (not directories) that do not have a MacBinary encoded - file. The local-header and central-header versions are identical. - - Value Size Description - ----- ---- ----------- - (Mac2b) 0x2705 Short tag for this extra block type - TSize Short total data size for this block (12) - "ZPIT" beLong extra-field signature - FileType Byte[4] four-byte Mac file type string - Creator Byte[4] four-byte Mac creator string - fdFlags beShort attributes from FInfo.frFlags, - may be omitted - 0x0000 beShort reserved, may be omitted - - - 4.6.6 -ZipIt Macintosh Extra Field (short, for directories) (0x2805): - - The following is the layout of a shortened variant of the - ZipIt extra block for Macintosh used only for directory - entries. This variant is used by ZipIt 1.3.5 and newer to - save some optional Mac-specific information about directories. - The local-header and central-header versions are identical. - - Value Size Description - ----- ---- ----------- - (Mac2c) 0x2805 Short tag for this extra block type - TSize Short total data size for this block (12) - "ZPIT" beLong extra-field signature - frFlags beShort attributes from DInfo.frFlags, may - be omitted - View beShort ZipIt view flag, may be omitted - - - The View field specifies ZipIt-internal settings as follows: - - Bits of the Flags: - bit 0 if set, the folder is shown expanded (open) - when the archive contents are viewed in ZipIt. - bits 1-15 reserved, zero; - - - 4.6.7 -FWKCS MD5 Extra Field (0x4b46): - - The FWKCS Contents_Signature System, used in - automatically identifying files independent of file name, - optionally adds and uses an extra field to support the - rapid creation of an enhanced contents_signature: - - Header ID = 0x4b46 - Data Size = 0x0013 - Preface = 'M','D','5' - followed by 16 bytes containing the uncompressed file's - 128_bit MD5 hash(1), low byte first. - - When FWKCS revises a .ZIP file central directory to add - this extra field for a file, it also replaces the - central directory entry for that file's uncompressed - file length with a measured value. - - FWKCS provides an option to strip this extra field, if - present, from a .ZIP file central directory. In adding - this extra field, FWKCS preserves .ZIP file Authenticity - Verification; if stripping this extra field, FWKCS - preserves all versions of AV through PKZIP version 2.04g. - - FWKCS, and FWKCS Contents_Signature System, are - trademarks of Frederick W. Kantor. - - (1) R. Rivest, RFC1321.TXT, MIT Laboratory for Computer - Science and RSA Data Security, Inc., April 1992. - ll.76-77: "The MD5 algorithm is being placed in the - public domain for review and possible adoption as a - standard." - - - 4.6.8 -Info-ZIP Unicode Comment Extra Field (0x6375): - - Stores the UTF-8 version of the file comment as stored in the - central directory header. (Last Revision 20070912) - - Value Size Description - ----- ---- ----------- - (UCom) 0x6375 Short tag for this extra block type ("uc") - TSize Short total data size for this block - Version 1 byte version of this extra field, currently 1 - ComCRC32 4 bytes Comment Field CRC32 Checksum - UnicodeCom Variable UTF-8 version of the entry comment - - Currently Version is set to the number 1. If there is a need - to change this field, the version will be incremented. Changes - may not be backward compatible so this extra field should not be - used if the version is not recognized. - - The ComCRC32 is the standard zip CRC32 checksum of the File Comment - field in the central directory header. This is used to verify that - the comment field has not changed since the Unicode Comment extra field - was created. This can happen if a utility changes the File Comment - field but does not update the UTF-8 Comment extra field. If the CRC - check fails, this Unicode Comment extra field should be ignored and - the File Comment field in the header should be used instead. - - The UnicodeCom field is the UTF-8 version of the File Comment field - in the header. As UnicodeCom is defined to be UTF-8, no UTF-8 byte - order mark (BOM) is used. The length of this field is determined by - subtracting the size of the previous fields from TSize. If both the - File Name and Comment fields are UTF-8, the new General Purpose Bit - Flag, bit 11 (Language encoding flag (EFS)), can be used to indicate - both the header File Name and Comment fields are UTF-8 and, in this - case, the Unicode Path and Unicode Comment extra fields are not - needed and should not be created. Note that, for backward - compatibility, bit 11 should only be used if the native character set - of the paths and comments being zipped up are already in UTF-8. It is - expected that the same file comment storage method, either general - purpose bit 11 or extra fields, be used in both the Local and Central - Directory Header for a file. - - - 4.6.9 -Info-ZIP Unicode Path Extra Field (0x7075): - - Stores the UTF-8 version of the file name field as stored in the - local header and central directory header. (Last Revision 20070912) - - Value Size Description - ----- ---- ----------- - (UPath) 0x7075 Short tag for this extra block type ("up") - TSize Short total data size for this block - Version 1 byte version of this extra field, currently 1 - NameCRC32 4 bytes File Name Field CRC32 Checksum - UnicodeName Variable UTF-8 version of the entry File Name - - Currently Version is set to the number 1. If there is a need - to change this field, the version will be incremented. Changes - may not be backward compatible so this extra field should not be - used if the version is not recognized. - - The NameCRC32 is the standard zip CRC32 checksum of the File Name - field in the header. This is used to verify that the header - File Name field has not changed since the Unicode Path extra field - was created. This can happen if a utility renames the File Name but - does not update the UTF-8 path extra field. If the CRC check fails, - this UTF-8 Path Extra Field should be ignored and the File Name field - in the header should be used instead. - - The UnicodeName is the UTF-8 version of the contents of the File Name - field in the header. As UnicodeName is defined to be UTF-8, no UTF-8 - byte order mark (BOM) is used. The length of this field is determined - by subtracting the size of the previous fields from TSize. If both - the File Name and Comment fields are UTF-8, the new General Purpose - Bit Flag, bit 11 (Language encoding flag (EFS)), can be used to - indicate that both the header File Name and Comment fields are UTF-8 - and, in this case, the Unicode Path and Unicode Comment extra fields - are not needed and should not be created. Note that, for backward - compatibility, bit 11 should only be used if the native character set - of the paths and comments being zipped up are already in UTF-8. It is - expected that the same file name storage method, either general - purpose bit 11 or extra fields, be used in both the Local and Central - Directory Header for a file. - - - 4.6.10 -Microsoft Open Packaging Growth Hint (0xa220): - - Value Size Description - ----- ---- ----------- - 0xa220 Short tag for this extra block type - TSize Short size of Sig + PadVal + Padding - Sig Short verification signature (A028) - PadVal Short Initial padding value - Padding variable filled with NULL characters - -4.7 Manifest Files ------------------- - - 4.7.1 Applications using ZIP files may have a need for additional - information that must be included with the files placed into - a ZIP file. Application specific information that cannot be - stored using the defined ZIP storage records SHOULD be stored - using the extensible Extra Field convention defined in this - document. However, some applications may use a manifest - file as a means for storing additional information. One - example is the META-INF/MANIFEST.MF file used in ZIP formatted - files having the .JAR extension (JAR files). - - 4.7.2 A manifest file is a file created for the application process - that requires this information. A manifest file MAY be of any - file type required by the defining application process. It is - placed within the same ZIP file as files to which this information - applies. By convention, this file is typically the first file placed - into the ZIP file and it may include a defined directory path. - - 4.7.3 Manifest files may be compressed or encrypted as needed for - application processing of the files inside the ZIP files. - - Manifest files are outside of the scope of this specification. - - -5.0 Explanation of compression methods --------------------------------------- - - -5.1 UnShrinking - Method 1 --------------------------- - - 5.1.1 Shrinking is a Dynamic Ziv-Lempel-Welch compression algorithm - with partial clearing. The initial code size is 9 bits, and the - maximum code size is 13 bits. Shrinking differs from conventional - Dynamic Ziv-Lempel-Welch implementations in several respects: - - 5.1.2 The code size is controlled by the compressor, and is - not automatically increased when codes larger than the current - code size are created (but not necessarily used). When - the decompressor encounters the code sequence 256 - (decimal) followed by 1, it should increase the code size - read from the input stream to the next bit size. No - blocking of the codes is performed, so the next code at - the increased size should be read from the input stream - immediately after where the previous code at the smaller - bit size was read. Again, the decompressor should not - increase the code size used until the sequence 256,1 is - encountered. - - 5.1.3 When the table becomes full, total clearing is not - performed. Rather, when the compressor emits the code - sequence 256,2 (decimal), the decompressor should clear - all leaf nodes from the Ziv-Lempel tree, and continue to - use the current code size. The nodes that are cleared - from the Ziv-Lempel tree are then re-used, with the lowest - code value re-used first, and the highest code value - re-used last. The compressor can emit the sequence 256,2 - at any time. - -5.2 Expanding - Methods 2-5 ---------------------------- - - 5.2.1 The Reducing algorithm is actually a combination of two - distinct algorithms. The first algorithm compresses repeated - byte sequences, and the second algorithm takes the compressed - stream from the first algorithm and applies a probabilistic - compression method. - - 5.2.2 The probabilistic compression stores an array of 'follower - sets' S(j), for j=0 to 255, corresponding to each possible - ASCII character. Each set contains between 0 and 32 - characters, to be denoted as S(j)[0],...,S(j)[m], where m<32. - The sets are stored at the beginning of the data area for a - Reduced file, in reverse order, with S(255) first, and S(0) - last. - - 5.2.3 The sets are encoded as { N(j), S(j)[0],...,S(j)[N(j)-1] }, - where N(j) is the size of set S(j). N(j) can be 0, in which - case the follower set for S(j) is empty. Each N(j) value is - encoded in 6 bits, followed by N(j) eight bit character values - corresponding to S(j)[0] to S(j)[N(j)-1] respectively. If - N(j) is 0, then no values for S(j) are stored, and the value - for N(j-1) immediately follows. - - 5.2.4 Immediately after the follower sets, is the compressed data - stream. The compressed data stream can be interpreted for the - probabilistic decompression as follows: - - let Last-Character <- 0. - loop until done - if the follower set S(Last-Character) is empty then - read 8 bits from the input stream, and copy this - value to the output stream. - otherwise if the follower set S(Last-Character) is non-empty then - read 1 bit from the input stream. - if this bit is not zero then - read 8 bits from the input stream, and copy this - value to the output stream. - otherwise if this bit is zero then - read B(N(Last-Character)) bits from the input - stream, and assign this value to I. - Copy the value of S(Last-Character)[I] to the - output stream. - - assign the last value placed on the output stream to - Last-Character. - end loop - - B(N(j)) is defined as the minimal number of bits required to - encode the value N(j)-1. - - 5.2.5 The decompressed stream from above can then be expanded to - re-create the original file as follows: - - let State <- 0. - - loop until done - read 8 bits from the input stream into C. - case State of - 0: if C is not equal to DLE (144 decimal) then - copy C to the output stream. - otherwise if C is equal to DLE then - let State <- 1. - - 1: if C is non-zero then - let V <- C. - let Len <- L(V) - let State <- F(Len). - otherwise if C is zero then - copy the value 144 (decimal) to the output stream. - let State <- 0 - - 2: let Len <- Len + C - let State <- 3. - - 3: move backwards D(V,C) bytes in the output stream - (if this position is before the start of the output - stream, then assume that all the data before the - start of the output stream is filled with zeros). - copy Len+3 bytes from this position to the output stream. - let State <- 0. - end case - end loop - - The functions F,L, and D are dependent on the 'compression - factor', 1 through 4, and are defined as follows: - - For compression factor 1: - L(X) equals the lower 7 bits of X. - F(X) equals 2 if X equals 127 otherwise F(X) equals 3. - D(X,Y) equals the (upper 1 bit of X) * 256 + Y + 1. - For compression factor 2: - L(X) equals the lower 6 bits of X. - F(X) equals 2 if X equals 63 otherwise F(X) equals 3. - D(X,Y) equals the (upper 2 bits of X) * 256 + Y + 1. - For compression factor 3: - L(X) equals the lower 5 bits of X. - F(X) equals 2 if X equals 31 otherwise F(X) equals 3. - D(X,Y) equals the (upper 3 bits of X) * 256 + Y + 1. - For compression factor 4: - L(X) equals the lower 4 bits of X. - F(X) equals 2 if X equals 15 otherwise F(X) equals 3. - D(X,Y) equals the (upper 4 bits of X) * 256 + Y + 1. - -5.3 Imploding - Method 6 ------------------------- - - 5.3.1 The Imploding algorithm is actually a combination of two - distinct algorithms. The first algorithm compresses repeated byte - sequences using a sliding dictionary. The second algorithm is - used to compress the encoding of the sliding dictionary output, - using multiple Shannon-Fano trees. - - 5.3.2 The Imploding algorithm can use a 4K or 8K sliding dictionary - size. The dictionary size used can be determined by bit 1 in the - general purpose flag word; a 0 bit indicates a 4K dictionary - while a 1 bit indicates an 8K dictionary. - - 5.3.3 The Shannon-Fano trees are stored at the start of the - compressed file. The number of trees stored is defined by bit 2 in - the general purpose flag word; a 0 bit indicates two trees stored, - a 1 bit indicates three trees are stored. If 3 trees are stored, - the first Shannon-Fano tree represents the encoding of the - Literal characters, the second tree represents the encoding of - the Length information, the third represents the encoding of the - Distance information. When 2 Shannon-Fano trees are stored, the - Length tree is stored first, followed by the Distance tree. - - 5.3.4 The Literal Shannon-Fano tree, if present is used to represent - the entire ASCII character set, and contains 256 values. This - tree is used to compress any data not compressed by the sliding - dictionary algorithm. When this tree is present, the Minimum - Match Length for the sliding dictionary is 3. If this tree is - not present, the Minimum Match Length is 2. - - 5.3.5 The Length Shannon-Fano tree is used to compress the Length - part of the (length,distance) pairs from the sliding dictionary - output. The Length tree contains 64 values, ranging from the - Minimum Match Length, to 63 plus the Minimum Match Length. - - 5.3.6 The Distance Shannon-Fano tree is used to compress the Distance - part of the (length,distance) pairs from the sliding dictionary - output. The Distance tree contains 64 values, ranging from 0 to - 63, representing the upper 6 bits of the distance value. The - distance values themselves will be between 0 and the sliding - dictionary size, either 4K or 8K. - - 5.3.7 The Shannon-Fano trees themselves are stored in a compressed - format. The first byte of the tree data represents the number of - bytes of data representing the (compressed) Shannon-Fano tree - minus 1. The remaining bytes represent the Shannon-Fano tree - data encoded as: - - High 4 bits: Number of values at this bit length + 1. (1 - 16) - Low 4 bits: Bit Length needed to represent value + 1. (1 - 16) - - 5.3.8 The Shannon-Fano codes can be constructed from the bit lengths - using the following algorithm: - - 1) Sort the Bit Lengths in ascending order, while retaining the - order of the original lengths stored in the file. - - 2) Generate the Shannon-Fano trees: - - Code <- 0 - CodeIncrement <- 0 - LastBitLength <- 0 - i <- number of Shannon-Fano codes - 1 (either 255 or 63) - - loop while i >= 0 - Code = Code + CodeIncrement - if BitLength(i) <> LastBitLength then - LastBitLength=BitLength(i) - CodeIncrement = 1 shifted left (16 - LastBitLength) - ShannonCode(i) = Code - i <- i - 1 - end loop - - 3) Reverse the order of all the bits in the above ShannonCode() - vector, so that the most significant bit becomes the least - significant bit. For example, the value 0x1234 (hex) would - become 0x2C48 (hex). - - 4) Restore the order of Shannon-Fano codes as originally stored - within the file. - - Example: - - This example will show the encoding of a Shannon-Fano tree - of size 8. Notice that the actual Shannon-Fano trees used - for Imploding are either 64 or 256 entries in size. - - Example: 0x02, 0x42, 0x01, 0x13 - - The first byte indicates 3 values in this table. Decoding the - bytes: - 0x42 = 5 codes of 3 bits long - 0x01 = 1 code of 2 bits long - 0x13 = 2 codes of 4 bits long - - This would generate the original bit length array of: - (3, 3, 3, 3, 3, 2, 4, 4) - - There are 8 codes in this table for the values 0 thru 7. Using - the algorithm to obtain the Shannon-Fano codes produces: - - Reversed Order Original - Val Sorted Constructed Code Value Restored Length - --- ------ ----------------- -------- -------- ------ - 0: 2 1100000000000000 11 101 3 - 1: 3 1010000000000000 101 001 3 - 2: 3 1000000000000000 001 110 3 - 3: 3 0110000000000000 110 010 3 - 4: 3 0100000000000000 010 100 3 - 5: 3 0010000000000000 100 11 2 - 6: 4 0001000000000000 1000 1000 4 - 7: 4 0000000000000000 0000 0000 4 - - The values in the Val, Order Restored and Original Length columns - now represent the Shannon-Fano encoding tree that can be used for - decoding the Shannon-Fano encoded data. How to parse the - variable length Shannon-Fano values from the data stream is beyond - the scope of this document. (See the references listed at the end of - this document for more information.) However, traditional decoding - schemes used for Huffman variable length decoding, such as the - Greenlaw algorithm, can be successfully applied. - - 5.3.9 The compressed data stream begins immediately after the - compressed Shannon-Fano data. The compressed data stream can be - interpreted as follows: - - loop until done - read 1 bit from input stream. - - if this bit is non-zero then (encoded data is literal data) - if Literal Shannon-Fano tree is present - read and decode character using Literal Shannon-Fano tree. - otherwise - read 8 bits from input stream. - copy character to the output stream. - otherwise (encoded data is sliding dictionary match) - if 8K dictionary size - read 7 bits for offset Distance (lower 7 bits of offset). - otherwise - read 6 bits for offset Distance (lower 6 bits of offset). - - using the Distance Shannon-Fano tree, read and decode the - upper 6 bits of the Distance value. - - using the Length Shannon-Fano tree, read and decode - the Length value. - - Length <- Length + Minimum Match Length - - if Length = 63 + Minimum Match Length - read 8 bits from the input stream, - add this value to Length. - - move backwards Distance+1 bytes in the output stream, and - copy Length characters from this position to the output - stream. (if this position is before the start of the output - stream, then assume that all the data before the start of - the output stream is filled with zeros). - end loop - -5.4 Tokenizing - Method 7 -------------------------- - - 5.4.1 This method is not used by PKZIP. - -5.5 Deflating - Method 8 ------------------------- - - 5.5.1 The Deflate algorithm is similar to the Implode algorithm using - a sliding dictionary of up to 32K with secondary compression - from Huffman/Shannon-Fano codes. - - 5.5.2 The compressed data is stored in blocks with a header describing - the block and the Huffman codes used in the data block. The header - format is as follows: - - Bit 0: Last Block bit This bit is set to 1 if this is the last - compressed block in the data. - Bits 1-2: Block type - 00 (0) - Block is stored - All stored data is byte aligned. - Skip bits until next byte, then next word = block - length, followed by the ones compliment of the block - length word. Remaining data in block is the stored - data. - - 01 (1) - Use fixed Huffman codes for literal and distance codes. - Lit Code Bits Dist Code Bits - --------- ---- --------- ---- - 0 - 143 8 0 - 31 5 - 144 - 255 9 - 256 - 279 7 - 280 - 287 8 - - Literal codes 286-287 and distance codes 30-31 are - never used but participate in the huffman construction. - - 10 (2) - Dynamic Huffman codes. (See expanding Huffman codes) - - 11 (3) - Reserved - Flag a "Error in compressed data" if seen. - - 5.5.3 Expanding Huffman Codes - - If the data block is stored with dynamic Huffman codes, the Huffman - codes are sent in the following compressed format: - - 5 Bits: # of Literal codes sent - 256 (256 - 286) - All other codes are never sent. - 5 Bits: # of Dist codes - 1 (1 - 32) - 4 Bits: # of Bit Length codes - 3 (3 - 19) - - The Huffman codes are sent as bit lengths and the codes are built as - described in the implode algorithm. The bit lengths themselves are - compressed with Huffman codes. There are 19 bit length codes: - - 0 - 15: Represent bit lengths of 0 - 15 - 16: Copy the previous bit length 3 - 6 times. - The next 2 bits indicate repeat length (0 = 3, ... ,3 = 6) - Example: Codes 8, 16 (+2 bits 11), 16 (+2 bits 10) will - expand to 12 bit lengths of 8 (1 + 6 + 5) - 17: Repeat a bit length of 0 for 3 - 10 times. (3 bits of length) - 18: Repeat a bit length of 0 for 11 - 138 times (7 bits of length) - - The lengths of the bit length codes are sent packed 3 bits per value - (0 - 7) in the following order: - - 16, 17, 18, 0, 8, 7, 9, 6, 10, 5, 11, 4, 12, 3, 13, 2, 14, 1, 15 - - The Huffman codes should be built as described in the Implode algorithm - except codes are assigned starting at the shortest bit length, i.e. the - shortest code should be all 0's rather than all 1's. Also, codes with - a bit length of zero do not participate in the tree construction. The - codes are then used to decode the bit lengths for the literal and - distance tables. - - The bit lengths for the literal tables are sent first with the number - of entries sent described by the 5 bits sent earlier. There are up - to 286 literal characters; the first 256 represent the respective 8 - bit character, code 256 represents the End-Of-Block code, the remaining - 29 codes represent copy lengths of 3 thru 258. There are up to 30 - distance codes representing distances from 1 thru 32k as described - below. - - Length Codes - ------------ - Extra Extra Extra Extra - Code Bits Length Code Bits Lengths Code Bits Lengths Code Bits Length(s) - ---- ---- ------ ---- ---- ------- ---- ---- ------- ---- ---- --------- - 257 0 3 265 1 11,12 273 3 35-42 281 5 131-162 - 258 0 4 266 1 13,14 274 3 43-50 282 5 163-194 - 259 0 5 267 1 15,16 275 3 51-58 283 5 195-226 - 260 0 6 268 1 17,18 276 3 59-66 284 5 227-257 - 261 0 7 269 2 19-22 277 4 67-82 285 0 258 - 262 0 8 270 2 23-26 278 4 83-98 - 263 0 9 271 2 27-30 279 4 99-114 - 264 0 10 272 2 31-34 280 4 115-130 - - Distance Codes - -------------- - Extra Extra Extra Extra - Code Bits Dist Code Bits Dist Code Bits Distance Code Bits Distance - ---- ---- ---- ---- ---- ------ ---- ---- -------- ---- ---- -------- - 0 0 1 8 3 17-24 16 7 257-384 24 11 4097-6144 - 1 0 2 9 3 25-32 17 7 385-512 25 11 6145-8192 - 2 0 3 10 4 33-48 18 8 513-768 26 12 8193-12288 - 3 0 4 11 4 49-64 19 8 769-1024 27 12 12289-16384 - 4 1 5,6 12 5 65-96 20 9 1025-1536 28 13 16385-24576 - 5 1 7,8 13 5 97-128 21 9 1537-2048 29 13 24577-32768 - 6 2 9-12 14 6 129-192 22 10 2049-3072 - 7 2 13-16 15 6 193-256 23 10 3073-4096 - - 5.5.4 The compressed data stream begins immediately after the - compressed header data. The compressed data stream can be - interpreted as follows: - - do - read header from input stream. - - if stored block - skip bits until byte aligned - read count and 1's compliment of count - copy count bytes data block - otherwise - loop until end of block code sent - decode literal character from input stream - if literal < 256 - copy character to the output stream - otherwise - if literal = end of block - break from loop - otherwise - decode distance from input stream - - move backwards distance bytes in the output stream, and - copy length characters from this position to the output - stream. - end loop - while not last block - - if data descriptor exists - skip bits until byte aligned - read crc and sizes - endif - -5.6 Enhanced Deflating - Method 9 ---------------------------------- - - 5.6.1 The Enhanced Deflating algorithm is similar to Deflate but uses - a sliding dictionary of up to 64K. Deflate64(tm) is supported - by the Deflate extractor. - -5.7 BZIP2 - Method 12 ---------------------- - - 5.7.1 BZIP2 is an open-source data compression algorithm developed by - Julian Seward. Information and source code for this algorithm - can be found on the internet. - -5.8 LZMA - Method 14 ---------------------- - - 5.8.1 LZMA is a block-oriented, general purpose data compression - algorithm developed and maintained by Igor Pavlov. It is a derivative - of LZ77 that utilizes Markov chains and a range coder. Information and - source code for this algorithm can be found on the internet. Consult - with the author of this algorithm for information on terms or - restrictions on use. - - Support for LZMA within the ZIP format is defined as follows: - - 5.8.2 The Compression method field within the ZIP Local and Central - Header records will be set to the value 14 to indicate data was - compressed using LZMA. - - 5.8.3 The Version needed to extract field within the ZIP Local and - Central Header records will be set to 6.3 to indicate the minimum - ZIP format version supporting this feature. - - 5.8.4 File data compressed using the LZMA algorithm must be placed - immediately following the Local Header for the file. If a standard - ZIP encryption header is required, it will follow the Local Header - and will precede the LZMA compressed file data segment. The location - of LZMA compressed data segment within the ZIP format will be as shown: - - [local header file 1] - [encryption header file 1] - [LZMA compressed data segment for file 1] - [data descriptor 1] - [local header file 2] - - 5.8.5 The encryption header and data descriptor records may - be conditionally present. The LZMA Compressed Data Segment - will consist of an LZMA Properties Header followed by the - LZMA Compressed Data as shown: - - [LZMA properties header for file 1] - [LZMA compressed data for file 1] - - 5.8.6 The LZMA Compressed Data will be stored as provided by the - LZMA compression library. Compressed size, uncompressed size and - other file characteristics about the file being compressed must be - stored in standard ZIP storage format. - - 5.8.7 The LZMA Properties Header will store specific data required - to decompress the LZMA compressed Data. This data is set by the - LZMA compression engine using the function WriteCoderProperties() - as documented within the LZMA SDK. - - 5.8.8 Storage fields for the property information within the LZMA - Properties Header are as follows: - - LZMA Version Information 2 bytes - LZMA Properties Size 2 bytes - LZMA Properties Data variable, defined by "LZMA Properties Size" - - 5.8.8.1 LZMA Version Information - this field identifies which version - of the LZMA SDK was used to compress a file. The first byte will - store the major version number of the LZMA SDK and the second - byte will store the minor number. - - 5.8.8.2 LZMA Properties Size - this field defines the size of the - remaining property data. Typically this size should be determined by - the version of the SDK. This size field is included as a convenience - and to help avoid any ambiguity should it arise in the future due - to changes in this compression algorithm. - - 5.8.8.3 LZMA Property Data - this variable sized field records the - required values for the decompressor as defined by the LZMA SDK. - The data stored in this field should be obtained using the - WriteCoderProperties() in the version of the SDK defined by - the "LZMA Version Information" field. - - 5.8.8.4 The layout of the "LZMA Properties Data" field is a function of - the LZMA compression algorithm. It is possible that this layout may be - changed by the author over time. The data layout in version 4.3 of the - LZMA SDK defines a 5 byte array that uses 4 bytes to store the dictionary - size in little-endian order. This is preceded by a single packed byte as - the first element of the array that contains the following fields: - - PosStateBits - LiteralPosStateBits - LiteralContextBits - - Refer to the LZMA documentation for a more detailed explanation of - these fields. - - 5.8.9 Data compressed with method 14, LZMA, may include an end-of-stream - (EOS) marker ending the compressed data stream. This marker is not - required, but its use is highly recommended to facilitate processing - and implementers should include the EOS marker whenever possible. - When the EOS marker is used, general purpose bit 1 must be set. If - general purpose bit 1 is not set, the EOS marker is not present. - -5.9 WavPack - Method 97 ------------------------ - - 5.9.1 Information describing the use of compression method 97 is - provided by WinZIP International, LLC. This method relies on the - open source WavPack audio compression utility developed by David Bryant. - Information on WavPack is available at www.wavpack.com. Please consult - with the author of this algorithm for information on terms and - restrictions on use. - - 5.9.2 WavPack data for a file begins immediately after the end of the - local header data. This data is the output from WavPack compression - routines. Within the ZIP file, the use of WavPack compression is - indicated by setting the compression method field to a value of 97 - in both the local header and the central directory header. The Version - needed to extract and version made by fields use the same values as are - used for data compressed using the Deflate algorithm. - - 5.9.3 An implementation note for storing digital sample data when using - WavPack compression within ZIP files is that all of the bytes of - the sample data should be compressed. This includes any unused - bits up to the byte boundary. An example is a 2 byte sample that - uses only 12 bits for the sample data with 4 unused bits. If only - 12 bits are passed as the sample size to the WavPack routines, the 4 - unused bits will be set to 0 on extraction regardless of their original - state. To avoid this, the full 16 bits of the sample data size - should be provided. - -5.10 PPMd - Method 98 ---------------------- - - 5.10.1 PPMd is a data compression algorithm developed by Dmitry Shkarin - which includes a carryless rangecoder developed by Dmitry Subbotin. - This algorithm is based on predictive phrase matching on multiple - order contexts. Information and source code for this algorithm - can be found on the internet. Consult with the author of this - algorithm for information on terms or restrictions on use. - - 5.10.2 Support for PPMd within the ZIP format currently is provided only - for version I, revision 1 of the algorithm. Storage requirements - for using this algorithm are as follows: - - 5.10.3 Parameters needed to control the algorithm are stored in the two - bytes immediately preceding the compressed data. These bytes are - used to store the following fields: - - Model order - sets the maximum model order, default is 8, possible - values are from 2 to 16 inclusive - - Sub-allocator size - sets the size of sub-allocator in MB, default is 50, - possible values are from 1MB to 256MB inclusive - - Model restoration method - sets the method used to restart context - model at memory insufficiency, values are: - - 0 - restarts model from scratch - default - 1 - cut off model - decreases performance by as much as 2x - 2 - freeze context tree - not recommended - - 5.10.4 An example for packing these fields into the 2 byte storage field is - illustrated below. These values are stored in Intel low-byte/high-byte - order. - - wPPMd = (Model order - 1) + - ((Sub-allocator size - 1) << 4) + - (Model restoration method << 12) - - -6.0 Traditional PKWARE Encryption ----------------------------------- - - 6.0.1 The following information discusses the decryption steps - required to support traditional PKWARE encryption. This - form of encryption is considered weak by today's standards - and its use is recommended only for situations with - low security needs or for compatibility with older .ZIP - applications. - -6.1 Traditional PKWARE Decryption ---------------------------------- - - 6.1.1 PKWARE is grateful to Mr. Roger Schlafly for his expert - contribution towards the development of PKWARE's traditional - encryption. - - 6.1.2 PKZIP encrypts the compressed data stream. Encrypted files - must be decrypted before they can be extracted to their original - form. - - 6.1.3 Each encrypted file has an extra 12 bytes stored at the start - of the data area defining the encryption header for that file. The - encryption header is originally set to random values, and then - itself encrypted, using three, 32-bit keys. The key values are - initialized using the supplied encryption password. After each byte - is encrypted, the keys are then updated using pseudo-random number - generation techniques in combination with the same CRC-32 algorithm - used in PKZIP and described elsewhere in this document. - - 6.1.4 The following are the basic steps required to decrypt a file: - - 1) Initialize the three 32-bit keys with the password. - 2) Read and decrypt the 12-byte encryption header, further - initializing the encryption keys. - 3) Read and decrypt the compressed data stream using the - encryption keys. - - 6.1.5 Initializing the encryption keys - - Key(0) <- 305419896 - Key(1) <- 591751049 - Key(2) <- 878082192 - - loop for i <- 0 to length(password)-1 - update_keys(password(i)) - end loop - - Where update_keys() is defined as: - - update_keys(char): - Key(0) <- crc32(key(0),char) - Key(1) <- Key(1) + (Key(0) & 000000ffH) - Key(1) <- Key(1) * 134775813 + 1 - Key(2) <- crc32(key(2),key(1) >> 24) - end update_keys - - Where crc32(old_crc,char) is a routine that given a CRC value and a - character, returns an updated CRC value after applying the CRC-32 - algorithm described elsewhere in this document. - - 6.1.6 Decrypting the encryption header - - The purpose of this step is to further initialize the encryption - keys, based on random data, to render a plaintext attack on the - data ineffective. - - Read the 12-byte encryption header into Buffer, in locations - Buffer(0) thru Buffer(11). - - loop for i <- 0 to 11 - C <- buffer(i) ^ decrypt_byte() - update_keys(C) - buffer(i) <- C - end loop - - Where decrypt_byte() is defined as: - - unsigned char decrypt_byte() - local unsigned short temp - temp <- Key(2) | 2 - decrypt_byte <- (temp * (temp ^ 1)) >> 8 - end decrypt_byte - - After the header is decrypted, the last 1 or 2 bytes in Buffer - should be the high-order word/byte of the CRC for the file being - decrypted, stored in Intel low-byte/high-byte order. Versions of - PKZIP prior to 2.0 used a 2 byte CRC check; a 1 byte CRC check is - used on versions after 2.0. This can be used to test if the password - supplied is correct or not. - - 6.1.7 Decrypting the compressed data stream - - The compressed data stream can be decrypted as follows: - - loop until done - read a character into C - Temp <- C ^ decrypt_byte() - update_keys(temp) - output Temp - end loop - - -7.0 Strong Encryption Specification ------------------------------------ - - 7.0.1 Portions of the Strong Encryption technology defined in this - specification are covered under patents and pending patent applications. - Refer to the section in this document entitled "Incorporating - PKWARE Proprietary Technology into Your Product" for more information. - -7.1 Strong Encryption Overview ------------------------------- - - 7.1.1 Version 5.x of this specification introduced support for strong - encryption algorithms. These algorithms can be used with either - a password or an X.509v3 digital certificate to encrypt each file. - This format specification supports either password or certificate - based encryption to meet the security needs of today, to enable - interoperability between users within both PKI and non-PKI - environments, and to ensure interoperability between different - computing platforms that are running a ZIP program. - - 7.1.2 Password based encryption is the most common form of encryption - people are familiar with. However, inherent weaknesses with - passwords (e.g. susceptibility to dictionary/brute force attack) - as well as password management and support issues make certificate - based encryption a more secure and scalable option. Industry - efforts and support are defining and moving towards more advanced - security solutions built around X.509v3 digital certificates and - Public Key Infrastructures(PKI) because of the greater scalability, - administrative options, and more robust security over traditional - password based encryption. - - 7.1.3 Most standard encryption algorithms are supported with this - specification. Reference implementations for many of these - algorithms are available from either commercial or open source - distributors. Readily available cryptographic toolkits make - implementation of the encryption features straight-forward. - This document is not intended to provide a treatise on data - encryption principles or theory. Its purpose is to document the - data structures required for implementing interoperable data - encryption within the .ZIP format. It is strongly recommended that - you have a good understanding of data encryption before reading - further. - - 7.1.4 The algorithms introduced in Version 5.0 of this specification - include: - - RC2 40 bit, 64 bit, and 128 bit - RC4 40 bit, 64 bit, and 128 bit - DES - 3DES 112 bit and 168 bit - - Version 5.1 adds support for the following: - - AES 128 bit, 192 bit, and 256 bit - - - 7.1.5 Version 6.1 introduces encryption data changes to support - interoperability with Smartcard and USB Token certificate storage - methods which do not support the OAEP strengthening standard. - - 7.1.6 Version 6.2 introduces support for encrypting metadata by compressing - and encrypting the central directory data structure to reduce information - leakage. Information leakage can occur in legacy ZIP applications - through exposure of information about a file even though that file is - stored encrypted. The information exposed consists of file - characteristics stored within the records and fields defined by this - specification. This includes data such as a file's name, its original - size, timestamp and CRC32 value. - - 7.1.7 Version 6.3 introduces support for encrypting data using the Blowfish - and Twofish algorithms. These are symmetric block ciphers developed - by Bruce Schneier. Blowfish supports using a variable length key from - 32 to 448 bits. Block size is 64 bits. Implementations should use 16 - rounds and the only mode supported within ZIP files is CBC. Twofish - supports key sizes 128, 192 and 256 bits. Block size is 128 bits. - Implementations should use 16 rounds and the only mode supported within - ZIP files is CBC. Information and source code for both Blowfish and - Twofish algorithms can be found on the internet. Consult with the author - of these algorithms for information on terms or restrictions on use. - - 7.1.8 Central Directory Encryption provides greater protection against - information leakage by encrypting the Central Directory structure and - by masking key values that are replicated in the unencrypted Local - Header. ZIP compatible programs that cannot interpret an encrypted - Central Directory structure cannot rely on the data in the corresponding - Local Header for decompression information. - - 7.1.9 Extra Field records that may contain information about a file that should - not be exposed should not be stored in the Local Header and should only - be written to the Central Directory where they can be encrypted. This - design currently does not support streaming. Information in the End of - Central Directory record, the Zip64 End of Central Directory Locator, - and the Zip64 End of Central Directory records are not encrypted. Access - to view data on files within a ZIP file with an encrypted Central Directory - requires the appropriate password or private key for decryption prior to - viewing any files, or any information about the files, in the archive. - - 7.1.10 Older ZIP compatible programs not familiar with the Central Directory - Encryption feature will no longer be able to recognize the Central - Directory and may assume the ZIP file is corrupt. Programs that - attempt streaming access using Local Headers will see invalid - information for each file. Central Directory Encryption need not be - used for every ZIP file. Its use is recommended for greater security. - ZIP files not using Central Directory Encryption should operate as - in the past. - - 7.1.11 This strong encryption feature specification is intended to provide for - scalable, cross-platform encryption needs ranging from simple password - encryption to authenticated public/private key encryption. - - 7.1.12 Encryption provides data confidentiality and privacy. It is - recommended that you combine X.509 digital signing with encryption - to add authentication and non-repudiation. - - -7.2 Single Password Symmetric Encryption Method ------------------------------------------------ - - 7.2.1 The Single Password Symmetric Encryption Method using strong - encryption algorithms operates similarly to the traditional - PKWARE encryption defined in this format. Additional data - structures are added to support the processing needs of the - strong algorithms. - - The Strong Encryption data structures are: - - 7.2.2 General Purpose Bits - Bits 0 and 6 of the General Purpose bit - flag in both local and central header records. Both bits set - indicates strong encryption. Bit 13, when set indicates the Central - Directory is encrypted and that selected fields in the Local Header - are masked to hide their actual value. - - - 7.2.3 Extra Field 0x0017 in central header only. - - Fields to consider in this record are: - - 7.2.3.1 Format - the data format identifier for this record. The only - value allowed at this time is the integer value 2. - - 7.2.3.2 AlgId - integer identifier of the encryption algorithm from the - following range - - 0x6601 - DES - 0x6602 - RC2 (version needed to extract < 5.2) - 0x6603 - 3DES 168 - 0x6609 - 3DES 112 - 0x660E - AES 128 - 0x660F - AES 192 - 0x6610 - AES 256 - 0x6702 - RC2 (version needed to extract >= 5.2) - 0x6720 - Blowfish - 0x6721 - Twofish - 0x6801 - RC4 - 0xFFFF - Unknown algorithm - - 7.2.3.3 Bitlen - Explicit bit length of key - - 32 - 448 bits - - 7.2.3.4 Flags - Processing flags needed for decryption - - 0x0001 - Password is required to decrypt - 0x0002 - Certificates only - 0x0003 - Password or certificate required to decrypt - - Values > 0x0003 reserved for certificate processing - - - 7.2.4 Decryption header record preceding compressed file data. - - -Decryption Header: - - Value Size Description - ----- ---- ----------- - IVSize 2 bytes Size of initialization vector (IV) - IVData IVSize Initialization vector for this file - Size 4 bytes Size of remaining decryption header data - Format 2 bytes Format definition for this record - AlgID 2 bytes Encryption algorithm identifier - Bitlen 2 bytes Bit length of encryption key - Flags 2 bytes Processing flags - ErdSize 2 bytes Size of Encrypted Random Data - ErdData ErdSize Encrypted Random Data - Reserved1 4 bytes Reserved certificate processing data - Reserved2 (var) Reserved for certificate processing data - VSize 2 bytes Size of password validation data - VData VSize-4 Password validation data - VCRC32 4 bytes Standard ZIP CRC32 of password validation data - - 7.2.4.1 IVData - The size of the IV should match the algorithm block size. - The IVData can be completely random data. If the size of - the randomly generated data does not match the block size - it should be complemented with zero's or truncated as - necessary. If IVSize is 0,then IV = CRC32 + Uncompressed - File Size (as a 64 bit little-endian, unsigned integer value). - - 7.2.4.2 Format - the data format identifier for this record. The only - value allowed at this time is the integer value 3. - - 7.2.4.3 AlgId - integer identifier of the encryption algorithm from the - following range - - 0x6601 - DES - 0x6602 - RC2 (version needed to extract < 5.2) - 0x6603 - 3DES 168 - 0x6609 - 3DES 112 - 0x660E - AES 128 - 0x660F - AES 192 - 0x6610 - AES 256 - 0x6702 - RC2 (version needed to extract >= 5.2) - 0x6720 - Blowfish - 0x6721 - Twofish - 0x6801 - RC4 - 0xFFFF - Unknown algorithm - - 7.2.4.4 Bitlen - Explicit bit length of key - - 32 - 448 bits - - 7.2.4.5 Flags - Processing flags needed for decryption - - 0x0001 - Password is required to decrypt - 0x0002 - Certificates only - 0x0003 - Password or certificate required to decrypt - - Values > 0x0003 reserved for certificate processing - - 7.2.4.6 ErdData - Encrypted random data is used to store random data that - is used to generate a file session key for encrypting - each file. SHA1 is used to calculate hash data used to - derive keys. File session keys are derived from a master - session key generated from the user-supplied password. - If the Flags field in the decryption header contains - the value 0x4000, then the ErdData field must be - decrypted using 3DES. If the value 0x4000 is not set, - then the ErdData field must be decrypted using AlgId. - - - 7.2.4.7 Reserved1 - Reserved for certificate processing, if value is - zero, then Reserved2 data is absent. See the explanation - under the Certificate Processing Method for details on - this data structure. - - 7.2.4.8 Reserved2 - If present, the size of the Reserved2 data structure - is located by skipping the first 4 bytes of this field - and using the next 2 bytes as the remaining size. See - the explanation under the Certificate Processing Method - for details on this data structure. - - 7.2.4.9 VSize - This size value will always include the 4 bytes of the - VCRC32 data and will be greater than 4 bytes. - - 7.2.4.10 VData - Random data for password validation. This data is VSize - in length and VSize must be a multiple of the encryption - block size. VCRC32 is a checksum value of VData. - VData and VCRC32 are stored encrypted and start the - stream of encrypted data for a file. - - - 7.2.5 Useful Tips - - 7.2.5.1 Strong Encryption is always applied to a file after compression. The - block oriented algorithms all operate in Cypher Block Chaining (CBC) - mode. The block size used for AES encryption is 16. All other block - algorithms use a block size of 8. Two IDs are defined for RC2 to - account for a discrepancy found in the implementation of the RC2 - algorithm in the cryptographic library on Windows XP SP1 and all - earlier versions of Windows. It is recommended that zero length files - not be encrypted, however programs should be prepared to extract them - if they are found within a ZIP file. - - 7.2.5.2 A pseudo-code representation of the encryption process is as follows: - - Password = GetUserPassword() - MasterSessionKey = DeriveKey(SHA1(Password)) - RD = CryptographicStrengthRandomData() - For Each File - IV = CryptographicStrengthRandomData() - VData = CryptographicStrengthRandomData() - VCRC32 = CRC32(VData) - FileSessionKey = DeriveKey(SHA1(IV + RD) - ErdData = Encrypt(RD,MasterSessionKey,IV) - Encrypt(VData + VCRC32 + FileData, FileSessionKey,IV) - Done - - 7.2.5.3 The function names and parameter requirements will depend on - the choice of the cryptographic toolkit selected. Almost any - toolkit supporting the reference implementations for each - algorithm can be used. The RSA BSAFE(r), OpenSSL, and Microsoft - CryptoAPI libraries are all known to work well. - - - 7.3 Single Password - Central Directory Encryption - -------------------------------------------------- - - 7.3.1 Central Directory Encryption is achieved within the .ZIP format by - encrypting the Central Directory structure. This encapsulates the metadata - most often used for processing .ZIP files. Additional metadata is stored for - redundancy in the Local Header for each file. The process of concealing - metadata by encrypting the Central Directory does not protect the data within - the Local Header. To avoid information leakage from the exposed metadata - in the Local Header, the fields containing information about a file are masked. - - 7.3.2 Local Header - - Masking replaces the true content of the fields for a file in the Local - Header with false information. When masked, the Local Header is not - suitable for streaming access and the options for data recovery of damaged - archives is reduced. Extra Data fields that may contain confidential - data should not be stored within the Local Header. The value set into - the Version needed to extract field should be the correct value needed to - extract the file without regard to Central Directory Encryption. The fields - within the Local Header targeted for masking when the Central Directory is - encrypted are: - - Field Name Mask Value - ------------------ --------------------------- - compression method 0 - last mod file time 0 - last mod file date 0 - crc-32 0 - compressed size 0 - uncompressed size 0 - file name (variable size) Base 16 value from the - range 1 - 0xFFFFFFFFFFFFFFFF - represented as a string whose - size will be set into the - file name length field - - The Base 16 value assigned as a masked file name is simply a sequentially - incremented value for each file starting with 1 for the first file. - Modifications to a ZIP file may cause different values to be stored for - each file. For compatibility, the file name field in the Local Header - should never be left blank. As of Version 6.2 of this specification, - the Compression Method and Compressed Size fields are not yet masked. - Fields having a value of 0xFFFF or 0xFFFFFFFF for the ZIP64 format - should not be masked. - - 7.3.3 Encrypting the Central Directory - - Encryption of the Central Directory does not include encryption of the - Central Directory Signature data, the Zip64 End of Central Directory - record, the Zip64 End of Central Directory Locator, or the End - of Central Directory record. The ZIP file comment data is never - encrypted. - - Before encrypting the Central Directory, it may optionally be compressed. - Compression is not required, but for storage efficiency it is assumed - this structure will be compressed before encrypting. Similarly, this - specification supports compressing the Central Directory without - requiring that it also be encrypted. Early implementations of this - feature will assume the encryption method applied to files matches the - encryption applied to the Central Directory. - - Encryption of the Central Directory is done in a manner similar to - that of file encryption. The encrypted data is preceded by a - decryption header. The decryption header is known as the Archive - Decryption Header. The fields of this record are identical to - the decryption header preceding each encrypted file. The location - of the Archive Decryption Header is determined by the value in the - Start of the Central Directory field in the Zip64 End of Central - Directory record. When the Central Directory is encrypted, the - Zip64 End of Central Directory record will always be present. - - The layout of the Zip64 End of Central Directory record for all - versions starting with 6.2 of this specification will follow the - Version 2 format. The Version 2 format is as follows: - - The leading fixed size fields within the Version 1 format for this - record remain unchanged. The record signature for both Version 1 - and Version 2 will be 0x06064b50. Immediately following the last - byte of the field known as the Offset of Start of Central - Directory With Respect to the Starting Disk Number will begin the - new fields defining Version 2 of this record. - - 7.3.4 New fields for Version 2 - - Note: all fields stored in Intel low-byte/high-byte order. - - Value Size Description - ----- ---- ----------- - Compression Method 2 bytes Method used to compress the - Central Directory - Compressed Size 8 bytes Size of the compressed data - Original Size 8 bytes Original uncompressed size - AlgId 2 bytes Encryption algorithm ID - BitLen 2 bytes Encryption key length - Flags 2 bytes Encryption flags - HashID 2 bytes Hash algorithm identifier - Hash Length 2 bytes Length of hash data - Hash Data (variable) Hash data - - The Compression Method accepts the same range of values as the - corresponding field in the Central Header. - - The Compressed Size and Original Size values will not include the - data of the Central Directory Signature which is compressed or - encrypted. - - The AlgId, BitLen, and Flags fields accept the same range of values - the corresponding fields within the 0x0017 record. - - Hash ID identifies the algorithm used to hash the Central Directory - data. This data does not have to be hashed, in which case the - values for both the HashID and Hash Length will be 0. Possible - values for HashID are: - - Value Algorithm - ------ --------- - 0x0000 none - 0x0001 CRC32 - 0x8003 MD5 - 0x8004 SHA1 - 0x8007 RIPEMD160 - 0x800C SHA256 - 0x800D SHA384 - 0x800E SHA512 - - 7.3.5 When the Central Directory data is signed, the same hash algorithm - used to hash the Central Directory for signing should be used. - This is recommended for processing efficiency, however, it is - permissible for any of the above algorithms to be used independent - of the signing process. - - The Hash Data will contain the hash data for the Central Directory. - The length of this data will vary depending on the algorithm used. - - The Version Needed to Extract should be set to 62. - - The value for the Total Number of Entries on the Current Disk will - be 0. These records will no longer support random access when - encrypting the Central Directory. - - 7.3.6 When the Central Directory is compressed and/or encrypted, the - End of Central Directory record will store the value 0xFFFFFFFF - as the value for the Total Number of Entries in the Central - Directory. The value stored in the Total Number of Entries in - the Central Directory on this Disk field will be 0. The actual - values will be stored in the equivalent fields of the Zip64 - End of Central Directory record. - - 7.3.7 Decrypting and decompressing the Central Directory is accomplished - in the same manner as decrypting and decompressing a file. - - 7.4 Certificate Processing Method - --------------------------------- - - The Certificate Processing Method for ZIP file encryption - defines the following additional data fields: - - 7.4.1 Certificate Flag Values - - Additional processing flags that can be present in the Flags field of both - the 0x0017 field of the central directory Extra Field and the Decryption - header record preceding compressed file data are: - - 0x0007 - reserved for future use - 0x000F - reserved for future use - 0x0100 - Indicates non-OAEP key wrapping was used. If this - this field is set, the version needed to extract must - be at least 61. This means OAEP key wrapping is not - used when generating a Master Session Key using - ErdData. - 0x4000 - ErdData must be decrypted using 3DES-168, otherwise use the - same algorithm used for encrypting the file contents. - 0x8000 - reserved for future use - - - 7.4.2 CertData - Extra Field 0x0017 record certificate data structure - - The data structure used to store certificate data within the section - of the Extra Field defined by the CertData field of the 0x0017 - record are as shown: - - Value Size Description - ----- ---- ----------- - RCount 4 bytes Number of recipients. - HashAlg 2 bytes Hash algorithm identifier - HSize 2 bytes Hash size - SRList (var) Simple list of recipients hashed public keys - - - RCount This defines the number intended recipients whose - public keys were used for encryption. This identifies - the number of elements in the SRList. - - HashAlg This defines the hash algorithm used to calculate - the public key hash of each public key used - for encryption. This field currently supports - only the following value for SHA-1 - - 0x8004 - SHA1 - - HSize This defines the size of a hashed public key. - - SRList This is a variable length list of the hashed - public keys for each intended recipient. Each - element in this list is HSize. The total size of - SRList is determined using RCount * HSize. - - - 7.4.3 Reserved1 - Certificate Decryption Header Reserved1 Data - - Value Size Description - ----- ---- ----------- - RCount 4 bytes Number of recipients. - - RCount This defines the number intended recipients whose - public keys were used for encryption. This defines - the number of elements in the REList field defined below. - - - 7.4.4 Reserved2 - Certificate Decryption Header Reserved2 Data Structures - - - Value Size Description - ----- ---- ----------- - HashAlg 2 bytes Hash algorithm identifier - HSize 2 bytes Hash size - REList (var) List of recipient data elements - - - HashAlg This defines the hash algorithm used to calculate - the public key hash of each public key used - for encryption. This field currently supports - only the following value for SHA-1 - - 0x8004 - SHA1 - - HSize This defines the size of a hashed public key - defined in REHData. - - REList This is a variable length of list of recipient data. - Each element in this list consists of a Recipient - Element data structure as follows: - - - Recipient Element (REList) Data Structure: - - Value Size Description - ----- ---- ----------- - RESize 2 bytes Size of REHData + REKData - REHData HSize Hash of recipients public key - REKData (var) Simple key blob - - - RESize This defines the size of an individual REList - element. This value is the combined size of the - REHData field + REKData field. REHData is defined by - HSize. REKData is variable and can be calculated - for each REList element using RESize and HSize. - - REHData Hashed public key for this recipient. - - REKData Simple Key Blob. The format of this data structure - is identical to that defined in the Microsoft - CryptoAPI and generated using the CryptExportKey() - function. The version of the Simple Key Blob - supported at this time is 0x02 as defined by - Microsoft. - -7.5 Certificate Processing - Central Directory Encryption ---------------------------------------------------------- - - 7.5.1 Central Directory Encryption using Digital Certificates will - operate in a manner similar to that of Single Password Central - Directory Encryption. This record will only be present when there - is data to place into it. Currently, data is placed into this - record when digital certificates are used for either encrypting - or signing the files within a ZIP file. When only password - encryption is used with no certificate encryption or digital - signing, this record is not currently needed. When present, this - record will appear before the start of the actual Central Directory - data structure and will be located immediately after the Archive - Decryption Header if the Central Directory is encrypted. - - 7.5.2 The Archive Extra Data record will be used to store the following - information. Additional data may be added in future versions. - - Extra Data Fields: - - 0x0014 - PKCS#7 Store for X.509 Certificates - 0x0016 - X.509 Certificate ID and Signature for central directory - 0x0019 - PKCS#7 Encryption Recipient Certificate List - - The 0x0014 and 0x0016 Extra Data records that otherwise would be - located in the first record of the Central Directory for digital - certificate processing. When encrypting or compressing the Central - Directory, the 0x0014 and 0x0016 records must be located in the - Archive Extra Data record and they should not remain in the first - Central Directory record. The Archive Extra Data record will also - be used to store the 0x0019 data. - - 7.5.3 When present, the size of the Archive Extra Data record will be - included in the size of the Central Directory. The data of the - Archive Extra Data record will also be compressed and encrypted - along with the Central Directory data structure. - -7.6 Certificate Processing Differences --------------------------------------- - - 7.6.1 The Certificate Processing Method of encryption differs from the - Single Password Symmetric Encryption Method as follows. Instead - of using a user-defined password to generate a master session key, - cryptographically random data is used. The key material is then - wrapped using standard key-wrapping techniques. This key material - is wrapped using the public key of each recipient that will need - to decrypt the file using their corresponding private key. - - 7.6.2 This specification currently assumes digital certificates will follow - the X.509 V3 format for 1024 bit and higher RSA format digital - certificates. Implementation of this Certificate Processing Method - requires supporting logic for key access and management. This logic - is outside the scope of this specification. - -7.7 OAEP Processing with Certificate-based Encryption ------------------------------------------------------ - - 7.7.1 OAEP stands for Optimal Asymmetric Encryption Padding. It is a - strengthening technique used for small encoded items such as decryption - keys. This is commonly applied in cryptographic key-wrapping techniques - and is supported by PKCS #1. Versions 5.0 and 6.0 of this specification - were designed to support OAEP key-wrapping for certificate-based - decryption keys for additional security. - - 7.7.2 Support for private keys stored on Smartcards or Tokens introduced - a conflict with this OAEP logic. Most card and token products do - not support the additional strengthening applied to OAEP key-wrapped - data. In order to resolve this conflict, versions 6.1 and above of this - specification will no longer support OAEP when encrypting using - digital certificates. - - 7.7.3 Versions of PKZIP available during initial development of the - certificate processing method set a value of 61 into the - version needed to extract field for a file. This indicates that - non-OAEP key wrapping is used. This affects certificate encryption - only, and password encryption functions should not be affected by - this value. This means values of 61 may be found on files encrypted - with certificates only, or on files encrypted with both password - encryption and certificate encryption. Files encrypted with both - methods can safely be decrypted using the password methods documented. - -8.0 Splitting and Spanning ZIP files -------------------------------------- - - 8.1 Spanned ZIP files - - 8.1.1 Spanning is the process of segmenting a ZIP file across - multiple removable media. This support has typically only - been provided for DOS formatted floppy diskettes. - - 8.2 Split ZIP files - - 8.2.1 File splitting is a newer derivation of spanning. - Splitting follows the same segmentation process as - spanning, however, it does not require writing each - segment to a unique removable medium and instead supports - placing all pieces onto local or non-removable locations - such as file systems, local drives, folders, etc. - - 8.3 File Naming Differences - - 8.3.1 A key difference between spanned and split ZIP files is - that all pieces of a spanned ZIP file have the same name. - Since each piece is written to a separate volume, no name - collisions occur and each segment can reuse the original - .ZIP file name given to the archive. - - 8.3.2 Sequence ordering for DOS spanned archives uses the DOS - volume label to determine segment numbers. Volume labels - for each segment are written using the form PKBACK#xxx, - where xxx is the segment number written as a decimal - value from 001 - nnn. - - 8.3.3 Split ZIP files are typically written to the same location - and are subject to name collisions if the spanned name - format is used since each segment will reside on the same - drive. To avoid name collisions, split archives are named - as follows. - - Segment 1 = filename.z01 - Segment n-1 = filename.z(n-1) - Segment n = filename.zip - - 8.3.4 The .ZIP extension is used on the last segment to support - quickly reading the central directory. The segment number - n should be a decimal value. - - 8.4 Spanned Self-extracting ZIP Files - - 8.4.1 Spanned ZIP files may be PKSFX Self-extracting ZIP files. - PKSFX files may also be split, however, in this case - the first segment must be named filename.exe. The first - segment of a split PKSFX archive must be large enough to - include the entire executable program. - - 8.5 Capacities and Markers - - 8.5.1 Capacities for split archives are as follows: - - Maximum number of segments = 4,294,967,295 - 1 - Maximum .ZIP segment size = 4,294,967,295 bytes - Minimum segment size = 64K - Maximum PKSFX segment size = 2,147,483,647 bytes - - 8.5.2 Segment sizes may be different however by convention, all - segment sizes should be the same with the exception of the - last, which may be smaller. Local and central directory - header records must never be split across a segment boundary. - When writing a header record, if the number of bytes remaining - within a segment is less than the size of the header record, - end the current segment and write the header at the start - of the next segment. The central directory may span segment - boundaries, but no single record in the central directory - should be split across segments. - - 8.5.3 Spanned/Split archives created using PKZIP for Windows - (V2.50 or greater), PKZIP Command Line (V2.50 or greater), - or PKZIP Explorer will include a special spanning - signature as the first 4 bytes of the first segment of - the archive. This signature (0x08074b50) will be - followed immediately by the local header signature for - the first file in the archive. - - 8.5.4 A special spanning marker may also appear in spanned/split - archives if the spanning or splitting process starts but - only requires one segment. In this case the 0x08074b50 - signature will be replaced with the temporary spanning - marker signature of 0x30304b50. Split archives can - only be uncompressed by other versions of PKZIP that - know how to create a split archive. - - 8.5.5 The signature value 0x08074b50 is also used by some - ZIP implementations as a marker for the Data Descriptor - record. Conflict in this alternate assignment can be - avoided by ensuring the position of the signature - within the ZIP file to determine the use for which it - is intended. - -9.0 Change Process ------------------- - - 9.1 In order for the .ZIP file format to remain a viable technology, this - specification should be considered as open for periodic review and - revision. Although this format was originally designed with a - certain level of extensibility, not all changes in technology - (present or future) were or will be necessarily considered in its - design. - - 9.2 If your application requires new definitions to the - extensible sections in this format, or if you would like to - submit new data structures or new capabilities, please forward - your request to zipformat@pkware.com. All submissions will be - reviewed by the ZIP File Specification Committee for possible - inclusion into future versions of this specification. - - 9.3 Periodic revisions to this specification will be published as - DRAFT or as FINAL status to ensure interoperability. We encourage - comments and feedback that may help improve clarity or content. - - -10.0 Incorporating PKWARE Proprietary Technology into Your Product ------------------------------------------------------------------- - - 10.1 The Use or Implementation in a product of APPNOTE technological - components pertaining to either strong encryption or patching requires - a separate, executed license agreement from PKWARE. Please contact - PKWARE at zipformat@pkware.com or +1-414-289-9788 with regard to - acquiring such a license. - - 10.2 Additional information regarding PKWARE proprietray technology is - available at http://www.pkware.com/appnote. - -11.0 Acknowledgements ---------------------- - - In addition to the above mentioned contributors to PKZIP and PKUNZIP, - PKWARE would like to extend special thanks to Robert Mahoney for - suggesting the extension .ZIP for this software. - -12.0 References ---------------- - - Fiala, Edward R., and Greene, Daniel H., "Data compression with - finite windows", Communications of the ACM, Volume 32, Number 4, - April 1989, pages 490-505. - - Held, Gilbert, "Data Compression, Techniques and Applications, - Hardware and Software Considerations", John Wiley & Sons, 1987. - - Huffman, D.A., "A method for the construction of minimum-redundancy - codes", Proceedings of the IRE, Volume 40, Number 9, September 1952, - pages 1098-1101. - - Nelson, Mark, "LZW Data Compression", Dr. Dobbs Journal, Volume 14, - Number 10, October 1989, pages 29-37. - - Nelson, Mark, "The Data Compression Book", M&T Books, 1991. - - Storer, James A., "Data Compression, Methods and Theory", - Computer Science Press, 1988 - - Welch, Terry, "A Technique for High-Performance Data Compression", - IEEE Computer, Volume 17, Number 6, June 1984, pages 8-19. - - Ziv, J. and Lempel, A., "A universal algorithm for sequential data - compression", Communications of the ACM, Volume 30, Number 6, - June 1987, pages 520-540. - - Ziv, J. and Lempel, A., "Compression of individual sequences via - variable-rate coding", IEEE Transactions on Information Theory, - Volume 24, Number 5, September 1978, pages 530-536. - - -APPENDIX A - AS/400 Extra Field (0x0065) Attribute Definitions --------------------------------------------------------------- - -A.1 Field Definition Structure: - - a. field length including length 2 bytes - b. field code 2 bytes - c. data x bytes - -A.2 Field Code Description - - 4001 Source type i.e. CLP etc - 4002 The text description of the library - 4003 The text description of the file - 4004 The text description of the member - 4005 x'F0' or 0 is PF-DTA, x'F1' or 1 is PF_SRC - 4007 Database Type Code 1 byte - 4008 Database file and fields definition - 4009 GZIP file type 2 bytes - 400B IFS code page 2 bytes - 400C IFS Creation Time 4 bytes - 400D IFS Access Time 4 bytes - 400E IFS Modification time 4 bytes - 005C Length of the records in the file 2 bytes - 0068 GZIP two words 8 bytes - -APPENDIX B - z/OS Extra Field (0x0065) Attribute Definitions ------------------------------------------------------------- - -B.1 Field Definition Structure: - - a. field length including length 2 bytes - b. field code 2 bytes - c. data x bytes - -B.2 Field Code Description - - 0001 File Type 2 bytes - 0002 NonVSAM Record Format 1 byte - 0003 Reserved - 0004 NonVSAM Block Size 2 bytes Big Endian - 0005 Primary Space Allocation 3 bytes Big Endian - 0006 Secondary Space Allocation 3 bytes Big Endian - 0007 Space Allocation Type1 byte flag - 0008 Modification Date Retired with PKZIP 5.0 + - 0009 Expiration Date Retired with PKZIP 5.0 + - 000A PDS Directory Block Allocation 3 bytes Big Endian binary value - 000B NonVSAM Volume List variable - 000C UNIT Reference Retired with PKZIP 5.0 + - 000D DF/SMS Management Class 8 bytes EBCDIC Text Value - 000E DF/SMS Storage Class 8 bytes EBCDIC Text Value - 000F DF/SMS Data Class 8 bytes EBCDIC Text Value - 0010 PDS/PDSE Member Info. 30 bytes - 0011 VSAM sub-filetype 2 bytes - 0012 VSAM LRECL 13 bytes EBCDIC "(num_avg num_max)" - 0013 VSAM Cluster Name Retired with PKZIP 5.0 + - 0014 VSAM KSDS Key Information 13 bytes EBCDIC "(num_length num_position)" - 0015 VSAM Average LRECL 5 bytes EBCDIC num_value padded with blanks - 0016 VSAM Maximum LRECL 5 bytes EBCDIC num_value padded with blanks - 0017 VSAM KSDS Key Length 5 bytes EBCDIC num_value padded with blanks - 0018 VSAM KSDS Key Position 5 bytes EBCDIC num_value padded with blanks - 0019 VSAM Data Name 1-44 bytes EBCDIC text string - 001A VSAM KSDS Index Name 1-44 bytes EBCDIC text string - 001B VSAM Catalog Name 1-44 bytes EBCDIC text string - 001C VSAM Data Space Type 9 bytes EBCDIC text string - 001D VSAM Data Space Primary 9 bytes EBCDIC num_value left-justified - 001E VSAM Data Space Secondary 9 bytes EBCDIC num_value left-justified - 001F VSAM Data Volume List variable EBCDIC text list of 6-character Volume IDs - 0020 VSAM Data Buffer Space 8 bytes EBCDIC num_value left-justified - 0021 VSAM Data CISIZE 5 bytes EBCDIC num_value left-justified - 0022 VSAM Erase Flag 1 byte flag - 0023 VSAM Free CI % 3 bytes EBCDIC num_value left-justified - 0024 VSAM Free CA % 3 bytes EBCDIC num_value left-justified - 0025 VSAM Index Volume List variable EBCDIC text list of 6-character Volume IDs - 0026 VSAM Ordered Flag 1 byte flag - 0027 VSAM REUSE Flag 1 byte flag - 0028 VSAM SPANNED Flag 1 byte flag - 0029 VSAM Recovery Flag 1 byte flag - 002A VSAM WRITECHK Flag 1 byte flag - 002B VSAM Cluster/Data SHROPTS 3 bytes EBCDIC "n,y" - 002C VSAM Index SHROPTS 3 bytes EBCDIC "n,y" - 002D VSAM Index Space Type 9 bytes EBCDIC text string - 002E VSAM Index Space Primary 9 bytes EBCDIC num_value left-justified - 002F VSAM Index Space Secondary 9 bytes EBCDIC num_value left-justified - 0030 VSAM Index CISIZE 5 bytes EBCDIC num_value left-justified - 0031 VSAM Index IMBED 1 byte flag - 0032 VSAM Index Ordered Flag 1 byte flag - 0033 VSAM REPLICATE Flag 1 byte flag - 0034 VSAM Index REUSE Flag 1 byte flag - 0035 VSAM Index WRITECHK Flag 1 byte flag Retired with PKZIP 5.0 + - 0036 VSAM Owner 8 bytes EBCDIC text string - 0037 VSAM Index Owner 8 bytes EBCDIC text string - 0038 Reserved - 0039 Reserved - 003A Reserved - 003B Reserved - 003C Reserved - 003D Reserved - 003E Reserved - 003F Reserved - 0040 Reserved - 0041 Reserved - 0042 Reserved - 0043 Reserved - 0044 Reserved - 0045 Reserved - 0046 Reserved - 0047 Reserved - 0048 Reserved - 0049 Reserved - 004A Reserved - 004B Reserved - 004C Reserved - 004D Reserved - 004E Reserved - 004F Reserved - 0050 Reserved - 0051 Reserved - 0052 Reserved - 0053 Reserved - 0054 Reserved - 0055 Reserved - 0056 Reserved - 0057 Reserved - 0058 PDS/PDSE Member TTR Info. 6 bytes Big Endian - 0059 PDS 1st LMOD Text TTR 3 bytes Big Endian - 005A PDS LMOD EP Rec # 4 bytes Big Endian - 005B Reserved - 005C Max Length of records 2 bytes Big Endian - 005D PDSE Flag 1 byte flag - 005E Reserved - 005F Reserved - 0060 Reserved - 0061 Reserved - 0062 Reserved - 0063 Reserved - 0064 Reserved - 0065 Last Date Referenced 4 bytes Packed Hex "yyyymmdd" - 0066 Date Created 4 bytes Packed Hex "yyyymmdd" - 0068 GZIP two words 8 bytes - 0071 Extended NOTE Location 12 bytes Big Endian - 0072 Archive device UNIT 6 bytes EBCDIC - 0073 Archive 1st Volume 6 bytes EBCDIC - 0074 Archive 1st VOL File Seq# 2 bytes Binary - -APPENDIX C - Zip64 Extensible Data Sector Mappings ---------------------------------------------------- - - -Z390 Extra Field: - - The following is the general layout of the attributes for the - ZIP 64 "extra" block for extended tape operations. - - Note: some fields stored in Big Endian format. All text is - in EBCDIC format unless otherwise specified. - - Value Size Description - ----- ---- ----------- - (Z390) 0x0065 2 bytes Tag for this "extra" block type - Size 4 bytes Size for the following data block - Tag 4 bytes EBCDIC "Z390" - Length71 2 bytes Big Endian - Subcode71 2 bytes Enote type code - FMEPos 1 byte - Length72 2 bytes Big Endian - Subcode72 2 bytes Unit type code - Unit 1 byte Unit - Length73 2 bytes Big Endian - Subcode73 2 bytes Volume1 type code - FirstVol 1 byte Volume - Length74 2 bytes Big Endian - Subcode74 2 bytes FirstVol file sequence - FileSeq 2 bytes Sequence - -APPENDIX D - Language Encoding (EFS) ------------------------------------- - -D.1 The ZIP format has historically supported only the original IBM PC character -encoding set, commonly referred to as IBM Code Page 437. This limits storing -file name characters to only those within the original MS-DOS range of values -and does not properly support file names in other character encodings, or -languages. To address this limitation, this specification will support the -following change. - -D.2 If general purpose bit 11 is unset, the file name and comment should conform -to the original ZIP character encoding. If general purpose bit 11 is set, the -filename and comment must support The Unicode Standard, Version 4.1.0 or -greater using the character encoding form defined by the UTF-8 storage -specification. The Unicode Standard is published by the The Unicode -Consortium (www.unicode.org). UTF-8 encoded data stored within ZIP files -is expected to not include a byte order mark (BOM). - -D.3 Applications may choose to supplement this file name storage through the use -of the 0x0008 Extra Field. Storage for this optional field is currently -undefined, however it will be used to allow storing extended information -on source or target encoding that may further assist applications with file -name, or file content encoding tasks. Please contact PKWARE with any -requirements on how this field should be used. - -D.4 The 0x0008 Extra Field storage may be used with either setting for general -purpose bit 11. Examples of the intended usage for this field is to store -whether "modified-UTF-8" (JAVA) is used, or UTF-8-MAC. Similarly, other -commonly used character encoding (code page) designations can be indicated -through this field. Formalized values for use of the 0x0008 record remain -undefined at this time. The definition for the layout of the 0x0008 field -will be published when available. Use of the 0x0008 Extra Field provides -for storing data within a ZIP file in an encoding other than IBM Code -Page 437 or UTF-8. - -D.5 General purpose bit 11 will not imply any encoding of file content or -password. Values defining character encoding for file content or -password must be stored within the 0x0008 Extended Language Encoding -Extra Field. - -D.6 Ed Gordon of the Info-ZIP group has defined a pair of "extra field" records -that can be used to store UTF-8 file name and file comment fields. These -records can be used for cases when the general purpose bit 11 method -for storing UTF-8 data in the standard file name and comment fields is -not desirable. A common case for this alternate method is if backward -compatibility with older programs is required. - -D.7 Definitions for the record structure of these fields are included above -in the section on 3rd party mappings for "extra field" records. These -records are identified by Header ID's 0x6375 (Info-ZIP Unicode Comment -Extra Field) and 0x7075 (Info-ZIP Unicode Path Extra Field). - -D.8 The choice of which storage method to use when writing a ZIP file is left -to the implementation. Developers should expect that a ZIP file may -contain either method and should provide support for reading data in -either format. Use of general purpose bit 11 reduces storage requirements -for file name data by not requiring additional "extra field" data for -each file, but can result in older ZIP programs not being able to extract -files. Use of the 0x6375 and 0x7075 records will result in a ZIP file -that should always be readable by older ZIP programs, but requires more -storage per file to write file name and/or file comment fields. +Specification Link: +https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT diff --git a/src/SharpCompress/Algorithms/Adler32.cs b/src/SharpCompress/Algorithms/Adler32.cs index 1f094c13..d7645327 100644 --- a/src/SharpCompress/Algorithms/Adler32.cs +++ b/src/SharpCompress/Algorithms/Adler32.cs @@ -1,7 +1,7 @@ // Copyright (c) Six Labors. // Licensed under the Apache License, Version 2.0. -#if !NETSTANDARD2_0 && !NETSTANDARD2_1 && !NETFRAMEWORK +#if !LEGACY_DOTNET #define SUPPORTS_RUNTIME_INTRINSICS #define SUPPORTS_HOTPATH #endif @@ -62,7 +62,7 @@ internal static class Adler32 // From https://github.com/SixLabors/ImageSharp/bl public static int ReduceSum(Vector256 accumulator) { // Add upper lane to lower lane. - Vector128 vsum = Sse2.Add(accumulator.GetLower(), accumulator.GetUpper()); + var vsum = Sse2.Add(accumulator.GetLower(), accumulator.GetUpper()); // Add odd to even. vsum = Sse2.Add(vsum, Sse2.Shuffle(vsum, 0b_11_11_01_01)); @@ -81,7 +81,7 @@ internal static class Adler32 // From https://github.com/SixLabors/ImageSharp/bl [MethodImpl(MethodImplOptions.AggressiveInlining)] public static int EvenReduceSum(Vector256 accumulator) { - Vector128 vsum = Sse2.Add(accumulator.GetLower(), accumulator.GetUpper()); // add upper lane to lower lane + var vsum = Sse2.Add(accumulator.GetLower(), accumulator.GetUpper()); // add upper lane to lower lane vsum = Sse2.Add(vsum, Sse2.Shuffle(vsum, 0b_11_10_11_10)); // add high to low // Vector128.ToScalar() isn't optimized pre-net5.0 https://github.com/dotnet/runtime/pull/37882 @@ -141,7 +141,7 @@ internal static class Adler32 // From https://github.com/SixLabors/ImageSharp/bl 4, 3, 2, - 1 // tap2 + 1, // tap2 }; #endif @@ -189,29 +189,29 @@ internal static class Adler32 // From https://github.com/SixLabors/ImageSharp/bl [MethodImpl(InliningOptions.HotPath | InliningOptions.ShortMethod)] private static unsafe uint CalculateSse(uint adler, ReadOnlySpan buffer) { - uint s1 = adler & 0xFFFF; - uint s2 = (adler >> 16) & 0xFFFF; + var s1 = adler & 0xFFFF; + var s2 = (adler >> 16) & 0xFFFF; // Process the data in blocks. - uint length = (uint)buffer.Length; - uint blocks = length / BlockSize; + var length = (uint)buffer.Length; + var blocks = length / BlockSize; length -= blocks * BlockSize; fixed (byte* bufferPtr = &MemoryMarshal.GetReference(buffer)) { fixed (byte* tapPtr = &MemoryMarshal.GetReference(Tap1Tap2)) { - byte* localBufferPtr = bufferPtr; + var localBufferPtr = bufferPtr; // _mm_setr_epi8 on x86 - Vector128 tap1 = Sse2.LoadVector128((sbyte*)tapPtr); - Vector128 tap2 = Sse2.LoadVector128((sbyte*)(tapPtr + 0x10)); - Vector128 zero = Vector128.Zero; + var tap1 = Sse2.LoadVector128((sbyte*)tapPtr); + var tap2 = Sse2.LoadVector128((sbyte*)(tapPtr + 0x10)); + var zero = Vector128.Zero; var ones = Vector128.Create((short)1); while (blocks > 0) { - uint n = NMAX / BlockSize; /* The NMAX constraint. */ + var n = NMAX / BlockSize; /* The NMAX constraint. */ if (n > blocks) { n = blocks; @@ -221,15 +221,15 @@ internal static class Adler32 // From https://github.com/SixLabors/ImageSharp/bl // Process n blocks of data. At most NMAX data bytes can be // processed before s2 must be reduced modulo BASE. - Vector128 v_ps = Vector128.CreateScalar(s1 * n); - Vector128 v_s2 = Vector128.CreateScalar(s2); - Vector128 v_s1 = Vector128.Zero; + var v_ps = Vector128.CreateScalar(s1 * n); + var v_s2 = Vector128.CreateScalar(s2); + var v_s1 = Vector128.Zero; do { // Load 32 input bytes. - Vector128 bytes1 = Sse3.LoadDquVector128(localBufferPtr); - Vector128 bytes2 = Sse3.LoadDquVector128(localBufferPtr + 0x10); + var bytes1 = Sse3.LoadDquVector128(localBufferPtr); + var bytes2 = Sse3.LoadDquVector128(localBufferPtr + 0x10); // Add previous block byte sum to v_ps. v_ps = Sse2.Add(v_ps, v_s1); @@ -237,11 +237,11 @@ internal static class Adler32 // From https://github.com/SixLabors/ImageSharp/bl // Horizontally add the bytes for s1, multiply-adds the // bytes by [ 32, 31, 30, ... ] for s2. v_s1 = Sse2.Add(v_s1, Sse2.SumAbsoluteDifferences(bytes1, zero).AsUInt32()); - Vector128 mad1 = Ssse3.MultiplyAddAdjacent(bytes1, tap1); + var mad1 = Ssse3.MultiplyAddAdjacent(bytes1, tap1); v_s2 = Sse2.Add(v_s2, Sse2.MultiplyAddAdjacent(mad1, ones).AsUInt32()); v_s1 = Sse2.Add(v_s1, Sse2.SumAbsoluteDifferences(bytes2, zero).AsUInt32()); - Vector128 mad2 = Ssse3.MultiplyAddAdjacent(bytes2, tap2); + var mad2 = Ssse3.MultiplyAddAdjacent(bytes2, tap2); v_s2 = Sse2.Add(v_s2, Sse2.MultiplyAddAdjacent(mad2, ones).AsUInt32()); localBufferPtr += BlockSize; @@ -281,15 +281,15 @@ internal static class Adler32 // From https://github.com/SixLabors/ImageSharp/bl [MethodImpl(InliningOptions.HotPath | InliningOptions.ShortMethod)] public static unsafe uint CalculateAvx2(uint adler, ReadOnlySpan buffer) { - uint s1 = adler & 0xFFFF; - uint s2 = (adler >> 16) & 0xFFFF; - uint length = (uint)buffer.Length; + var s1 = adler & 0xFFFF; + var s2 = (adler >> 16) & 0xFFFF; + var length = (uint)buffer.Length; fixed (byte* bufferPtr = &MemoryMarshal.GetReference(buffer)) { - byte* localBufferPtr = bufferPtr; + var localBufferPtr = bufferPtr; - Vector256 zero = Vector256.Zero; + var zero = Vector256.Zero; var dot3v = Vector256.Create((short)1); var dot2v = Vector256.Create( 32, @@ -333,29 +333,29 @@ internal static class Adler32 // From https://github.com/SixLabors/ImageSharp/bl while (length >= 32) { - int k = length < NMAX ? (int)length : (int)NMAX; + var k = length < NMAX ? (int)length : (int)NMAX; k -= k % 32; length -= (uint)k; - Vector256 vs10 = vs1; - Vector256 vs3 = Vector256.Zero; + var vs10 = vs1; + var vs3 = Vector256.Zero; while (k >= 32) { // Load 32 input bytes. - Vector256 block = Avx.LoadVector256(localBufferPtr); + var block = Avx.LoadVector256(localBufferPtr); // Sum of abs diff, resulting in 2 x int32's - Vector256 vs1sad = Avx2.SumAbsoluteDifferences(block, zero); + var vs1sad = Avx2.SumAbsoluteDifferences(block, zero); vs1 = Avx2.Add(vs1, vs1sad.AsUInt32()); vs3 = Avx2.Add(vs3, vs10); // sum 32 uint8s to 16 shorts. - Vector256 vshortsum2 = Avx2.MultiplyAddAdjacent(block, dot2v); + var vshortsum2 = Avx2.MultiplyAddAdjacent(block, dot2v); // sum 16 shorts to 8 uint32s. - Vector256 vsum2 = Avx2.MultiplyAddAdjacent(vshortsum2, dot3v); + var vsum2 = Avx2.MultiplyAddAdjacent(vshortsum2, dot3v); vs2 = Avx2.Add(vsum2.AsUInt32(), vs2); vs10 = vs1; @@ -434,14 +434,14 @@ internal static class Adler32 // From https://github.com/SixLabors/ImageSharp/bl [MethodImpl(InliningOptions.HotPath | InliningOptions.ShortMethod)] private static unsafe uint CalculateScalar(uint adler, ReadOnlySpan buffer) { - uint s1 = adler & 0xFFFF; - uint s2 = (adler >> 16) & 0xFFFF; + var s1 = adler & 0xFFFF; + var s2 = (adler >> 16) & 0xFFFF; uint k; fixed (byte* bufferPtr = buffer) { var localBufferPtr = bufferPtr; - uint length = (uint)buffer.Length; + var length = (uint)buffer.Length; while (length > 0) { diff --git a/src/SharpCompress/Archives/AbstractArchive.Async.cs b/src/SharpCompress/Archives/AbstractArchive.Async.cs new file mode 100644 index 00000000..7c68342f --- /dev/null +++ b/src/SharpCompress/Archives/AbstractArchive.Async.cs @@ -0,0 +1,105 @@ +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Readers; + +namespace SharpCompress.Archives; + +public abstract partial class AbstractArchive + where TEntry : IArchiveEntry + where TVolume : IVolume +{ + #region Async Support + + // Async properties + public virtual IAsyncEnumerable EntriesAsync => _lazyEntriesAsync; + + public IAsyncEnumerable VolumesAsync => _lazyVolumesAsync; + + protected virtual async IAsyncEnumerable LoadEntriesAsync( + IAsyncEnumerable volumes + ) + { + foreach (var item in LoadEntries(await volumes.ToListAsync().ConfigureAwait(false))) + { + yield return item; + } + } + + public virtual async ValueTask DisposeAsync() + { + if (!_disposed) + { + await foreach (var v in _lazyVolumesAsync.ConfigureAwait(false)) + { + v.Dispose(); + } + foreach (var v in _lazyEntriesAsync.GetLoaded().Cast()) + { + v.Close(); + } + _sourceStream?.Dispose(); + + _disposed = true; + } + } + + private async ValueTask EnsureEntriesLoadedAsync() + { + await _lazyEntriesAsync.EnsureFullyLoaded().ConfigureAwait(false); + await _lazyVolumesAsync.EnsureFullyLoaded().ConfigureAwait(false); + } + + private async IAsyncEnumerable EntriesAsyncCast() + { + await foreach (var entry in EntriesAsync.ConfigureAwait(false)) + { + yield return entry; + } + } + + IAsyncEnumerable IAsyncArchive.EntriesAsync => EntriesAsyncCast(); + + IAsyncEnumerable IAsyncArchive.VolumesAsync => VolumesAsyncCast(); + + private async IAsyncEnumerable VolumesAsyncCast() + { + await foreach (var volume in _lazyVolumesAsync.ConfigureAwait(false)) + { + yield return volume; + } + } + + public async ValueTask ExtractAllEntriesAsync() + { + if (!await IsSolidAsync().ConfigureAwait(false) && Type != ArchiveType.SevenZip) + { + throw new SharpCompressException( + "ExtractAllEntries can only be used on solid archives or 7Zip archives (which require random access)." + ); + } + await EnsureEntriesLoadedAsync().ConfigureAwait(false); + return await CreateReaderForSolidExtractionAsync().ConfigureAwait(false); + } + + public virtual ValueTask IsSolidAsync() => new(false); + + public async ValueTask IsCompleteAsync() + { + await EnsureEntriesLoadedAsync().ConfigureAwait(false); + return await EntriesAsync.AllAsync(x => x.IsComplete).ConfigureAwait(false); + } + + public async ValueTask TotalSizeAsync() => + await EntriesAsync + .AggregateAsync(0L, (total, cf) => total + cf.CompressedSize) + .ConfigureAwait(false); + + public async ValueTask TotalUncompressedSizeAsync() => + await EntriesAsync.AggregateAsync(0L, (total, cf) => total + cf.Size).ConfigureAwait(false); + + public ValueTask IsEncryptedAsync() => new(IsEncrypted); + + #endregion +} diff --git a/src/SharpCompress/Archives/AbstractArchive.cs b/src/SharpCompress/Archives/AbstractArchive.cs index cfac55f2..1e4b353b 100644 --- a/src/SharpCompress/Archives/AbstractArchive.cs +++ b/src/SharpCompress/Archives/AbstractArchive.cs @@ -1,76 +1,67 @@ -using System; using System.Collections.Generic; -using System.IO; using System.Linq; +using System.Threading.Tasks; using SharpCompress.Common; using SharpCompress.IO; using SharpCompress.Readers; namespace SharpCompress.Archives; -public abstract class AbstractArchive : IArchive, IArchiveExtractionListener +public abstract partial class AbstractArchive : IArchive, IAsyncArchive where TEntry : IArchiveEntry where TVolume : IVolume { - private readonly LazyReadOnlyCollection lazyVolumes; - private readonly LazyReadOnlyCollection lazyEntries; + private readonly LazyReadOnlyCollection _lazyVolumes; + private readonly LazyReadOnlyCollection _lazyEntries; + private bool _disposed; + private readonly SourceStream? _sourceStream; - public event EventHandler>? EntryExtractionBegin; - public event EventHandler>? EntryExtractionEnd; + // Async fields - kept in original file per refactoring rules + private readonly LazyAsyncReadOnlyCollection _lazyVolumesAsync; + private readonly LazyAsyncReadOnlyCollection _lazyEntriesAsync; - public event EventHandler? CompressedBytesRead; - public event EventHandler? FilePartExtractionBegin; + public ReaderOptions ReaderOptions { get; protected set; } - protected ReaderOptions ReaderOptions { get; } - - private bool disposed; - protected SourceStream SrcStream; - - internal AbstractArchive(ArchiveType type, SourceStream srcStream) + internal AbstractArchive(ArchiveType type, SourceStream sourceStream) { Type = type; - ReaderOptions = srcStream.ReaderOptions; - SrcStream = srcStream; - lazyVolumes = new LazyReadOnlyCollection(LoadVolumes(SrcStream)); - lazyEntries = new LazyReadOnlyCollection(LoadEntries(Volumes)); + ReaderOptions = sourceStream.ReaderOptions; + _sourceStream = sourceStream; + _lazyVolumes = new LazyReadOnlyCollection(LoadVolumes(_sourceStream)); + _lazyEntries = new LazyReadOnlyCollection(LoadEntries(Volumes)); + _lazyVolumesAsync = new LazyAsyncReadOnlyCollection( + LoadVolumesAsync(_sourceStream) + ); + _lazyEntriesAsync = new LazyAsyncReadOnlyCollection( + LoadEntriesAsync(_lazyVolumesAsync) + ); } -#nullable disable internal AbstractArchive(ArchiveType type) { Type = type; - lazyVolumes = new LazyReadOnlyCollection(Enumerable.Empty()); - lazyEntries = new LazyReadOnlyCollection(Enumerable.Empty()); + ReaderOptions = ReaderOptions.Default; + _lazyVolumes = new LazyReadOnlyCollection(Enumerable.Empty()); + _lazyEntries = new LazyReadOnlyCollection(Enumerable.Empty()); + _lazyVolumesAsync = new LazyAsyncReadOnlyCollection( + AsyncEnumerableEx.Empty() + ); + _lazyEntriesAsync = new LazyAsyncReadOnlyCollection( + AsyncEnumerableEx.Empty() + ); } -#nullable enable - public ArchiveType Type { get; } - void IArchiveExtractionListener.FireEntryExtractionBegin(IArchiveEntry entry) => - EntryExtractionBegin?.Invoke(this, new ArchiveExtractionEventArgs(entry)); - - void IArchiveExtractionListener.FireEntryExtractionEnd(IArchiveEntry entry) => - EntryExtractionEnd?.Invoke(this, new ArchiveExtractionEventArgs(entry)); - - private static Stream CheckStreams(Stream stream) - { - if (!stream.CanSeek || !stream.CanRead) - { - throw new ArgumentException("Archive streams must be Readable and Seekable"); - } - return stream; - } - /// /// Returns an ReadOnlyCollection of all the RarArchiveEntries across the one or many parts of the RarArchive. /// - public virtual ICollection Entries => lazyEntries; + public virtual ICollection Entries => _lazyEntries; /// /// Returns an ReadOnlyCollection of all the RarArchiveVolumes across the one or many parts of the RarArchive. /// - public ICollection Volumes => lazyVolumes; + public ICollection Volumes => _lazyVolumes; /// /// The total size of the files compressed in the archive. @@ -81,60 +72,37 @@ public abstract class AbstractArchive : IArchive, IArchiveExtra /// /// The total size of the files as uncompressed in the archive. /// - public virtual long TotalUncompressSize => + public virtual long TotalUncompressedSize => Entries.Aggregate(0L, (total, cf) => total + cf.Size); - protected abstract IEnumerable LoadVolumes(SourceStream srcStream); + protected abstract IEnumerable LoadVolumes(SourceStream sourceStream); protected abstract IEnumerable LoadEntries(IEnumerable volumes); + protected virtual IAsyncEnumerable LoadVolumesAsync(SourceStream sourceStream) => + LoadVolumes(sourceStream).ToAsyncEnumerable(); + IEnumerable IArchive.Entries => Entries.Cast(); - IEnumerable IArchive.Volumes => lazyVolumes.Cast(); + IEnumerable IArchive.Volumes => _lazyVolumes.Cast(); public virtual void Dispose() { - if (!disposed) + if (!_disposed) { - lazyVolumes.ForEach(v => v.Dispose()); - lazyEntries.GetLoaded().Cast().ForEach(x => x.Close()); - SrcStream?.Dispose(); + _lazyVolumes.ForEach(v => v.Dispose()); + _lazyEntries.GetLoaded().Cast().ForEach(x => x.Close()); + _sourceStream?.Dispose(); - disposed = true; + _disposed = true; } } - void IArchiveExtractionListener.EnsureEntriesLoaded() + private void EnsureEntriesLoaded() { - lazyEntries.EnsureFullyLoaded(); - lazyVolumes.EnsureFullyLoaded(); + _lazyEntries.EnsureFullyLoaded(); + _lazyVolumes.EnsureFullyLoaded(); } - void IExtractionListener.FireCompressedBytesRead( - long currentPartCompressedBytes, - long compressedReadBytes - ) => - CompressedBytesRead?.Invoke( - this, - new CompressedBytesReadEventArgs( - currentFilePartCompressedBytesRead: currentPartCompressedBytes, - compressedBytesRead: compressedReadBytes - ) - ); - - void IExtractionListener.FireFilePartExtractionBegin( - string name, - long size, - long compressedSize - ) => - FilePartExtractionBegin?.Invoke( - this, - new FilePartExtractionBeginEventArgs( - compressedSize: compressedSize, - size: size, - name: name - ) - ); - /// /// Use this method to extract all entries in an archive in order. /// This is primarily for SOLID Rar Archives or 7Zip Archives as they need to be @@ -148,17 +116,29 @@ public abstract class AbstractArchive : IArchive, IArchiveExtra /// public IReader ExtractAllEntries() { - ((IArchiveExtractionListener)this).EnsureEntriesLoaded(); + if (!IsSolid && Type != ArchiveType.SevenZip) + { + throw new SharpCompressException( + "ExtractAllEntries can only be used on solid archives or 7Zip archives (which require random access)." + ); + } + EnsureEntriesLoaded(); return CreateReaderForSolidExtraction(); } protected abstract IReader CreateReaderForSolidExtraction(); + protected abstract ValueTask CreateReaderForSolidExtractionAsync(); /// /// Archive is SOLID (this means the Archive saved bytes by reusing information which helps for archives containing many small files). /// public virtual bool IsSolid => false; + /// + /// Archive is ENCRYPTED (this means the Archive has password-protected files). + /// + public virtual bool IsEncrypted => false; + /// /// The archive can find all the parts of the archive needed to fully extract the archive. This forces the parsing of the entire archive. /// @@ -166,7 +146,7 @@ public abstract class AbstractArchive : IArchive, IArchiveExtra { get { - ((IArchiveExtractionListener)this).EnsureEntriesLoaded(); + EnsureEntriesLoaded(); return Entries.All(x => x.IsComplete); } } diff --git a/src/SharpCompress/Archives/AbstractWritableArchive.Async.cs b/src/SharpCompress/Archives/AbstractWritableArchive.Async.cs new file mode 100644 index 00000000..67b8dffc --- /dev/null +++ b/src/SharpCompress/Archives/AbstractWritableArchive.Async.cs @@ -0,0 +1,140 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Options; + +namespace SharpCompress.Archives; + +public abstract partial class AbstractWritableArchive + where TEntry : IArchiveEntry + where TVolume : IVolume + where TOptions : IWriterOptions +{ + // Async property moved from main file + private IAsyncEnumerable OldEntriesAsync => + base.EntriesAsync.Where(x => !removedEntries.Contains(x)); + + private async ValueTask RebuildModifiedCollectionAsync() + { + if (pauseRebuilding) + { + return; + } + hasModifications = true; + newEntries.RemoveAll(v => removedEntries.Contains(v)); + modifiedEntries.Clear(); + await foreach (var entry in OldEntriesAsync.ConfigureAwait(false)) + { + modifiedEntries.Add(entry); + } + modifiedEntries.AddRange(newEntries); + } + + public async ValueTask RemoveEntryAsync(TEntry entry) + { + if (!removedEntries.Contains(entry)) + { + removedEntries.Add(entry); + await RebuildModifiedCollectionAsync().ConfigureAwait(false); + } + } + + private async ValueTask DoesKeyMatchExistingAsync( + string key, + CancellationToken cancellationToken + ) + { + await foreach ( + var entry in EntriesAsync.WithCancellation(cancellationToken).ConfigureAwait(false) + ) + { + var path = entry.Key; + if (path is null) + { + continue; + } + var p = path.Replace('/', '\\'); + if (p.Length > 0 && p[0] == '\\') + { + p = p.Substring(1); + } + return string.Equals(p, key, StringComparison.OrdinalIgnoreCase); + } + return false; + } + + public async ValueTask AddEntryAsync( + string key, + Stream source, + bool closeStream, + long size = 0, + DateTime? modified = null, + CancellationToken cancellationToken = default + ) + { + if (key.Length > 0 && key[0] is '/' or '\\') + { + key = key.Substring(1); + } + if (await DoesKeyMatchExistingAsync(key, cancellationToken).ConfigureAwait(false)) + { + throw new ArchiveException("Cannot add entry with duplicate key: " + key); + } + var entry = CreateEntry(key, source, size, modified, closeStream); + newEntries.Add(entry); + await RebuildModifiedCollectionAsync().ConfigureAwait(false); + return entry; + } + + public async ValueTask AddDirectoryEntryAsync( + string key, + DateTime? modified = null, + CancellationToken cancellationToken = default + ) + { + if (key.Length > 0 && key[0] is '/' or '\\') + { + key = key.Substring(1); + } + if (await DoesKeyMatchExistingAsync(key, cancellationToken).ConfigureAwait(false)) + { + throw new ArchiveException("Cannot add entry with duplicate key: " + key); + } + var entry = CreateDirectoryEntry(key, modified); + newEntries.Add(entry); + await RebuildModifiedCollectionAsync().ConfigureAwait(false); + return entry; + } + + public async ValueTask SaveToAsync( + Stream stream, + TOptions options, + CancellationToken cancellationToken = default + ) + { + //reset streams of new entries + newEntries.Cast().ForEach(x => x.Stream.Seek(0, SeekOrigin.Begin)); + await SaveToAsync(stream, options, OldEntriesAsync, newEntries, cancellationToken) + .ConfigureAwait(false); + } + + public override async ValueTask DisposeAsync() + { + await base.DisposeAsync().ConfigureAwait(false); + newEntries.Cast().ForEach(x => x.Close()); + removedEntries.Cast().ForEach(x => x.Close()); + modifiedEntries.Cast().ForEach(x => x.Close()); + } + + protected abstract ValueTask SaveToAsync( + Stream stream, + TOptions options, + IAsyncEnumerable oldEntries, + IEnumerable newEntries, + CancellationToken cancellationToken = default + ); +} diff --git a/src/SharpCompress/Archives/AbstractWritableArchive.cs b/src/SharpCompress/Archives/AbstractWritableArchive.cs index 30083ec7..59d55771 100644 --- a/src/SharpCompress/Archives/AbstractWritableArchive.cs +++ b/src/SharpCompress/Archives/AbstractWritableArchive.cs @@ -2,23 +2,28 @@ using System; using System.Collections.Generic; using System.IO; using System.Linq; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common; +using SharpCompress.Common.Options; using SharpCompress.IO; using SharpCompress.Writers; namespace SharpCompress.Archives; -public abstract class AbstractWritableArchive +public abstract partial class AbstractWritableArchive : AbstractArchive, - IWritableArchive + IWritableArchive, + IWritableAsyncArchive where TEntry : IArchiveEntry where TVolume : IVolume + where TOptions : IWriterOptions { private class RebuildPauseDisposable : IDisposable { - private readonly AbstractWritableArchive archive; + private readonly AbstractWritableArchive archive; - public RebuildPauseDisposable(AbstractWritableArchive archive) + public RebuildPauseDisposable(AbstractWritableArchive archive) { this.archive = archive; archive.pauseRebuilding = true; @@ -31,18 +36,18 @@ public abstract class AbstractWritableArchive } } - private readonly List newEntries = new List(); - private readonly List removedEntries = new List(); + private readonly List newEntries = new(); + private readonly List removedEntries = new(); - private readonly List modifiedEntries = new List(); + private readonly List modifiedEntries = new(); private bool hasModifications; private bool pauseRebuilding; internal AbstractWritableArchive(ArchiveType type) : base(type) { } - internal AbstractWritableArchive(ArchiveType type, SourceStream srcStream) - : base(type, srcStream) { } + internal AbstractWritableArchive(ArchiveType type, SourceStream sourceStream) + : base(type, sourceStream) { } public override ICollection Entries { @@ -94,6 +99,9 @@ public abstract class AbstractWritableArchive DateTime? modified ) => AddEntry(key, source, closeStream, size, modified); + IArchiveEntry IWritableArchive.AddDirectoryEntry(string key, DateTime? modified) => + AddDirectoryEntry(key, modified); + public TEntry AddEntry( string key, Stream source, @@ -120,6 +128,10 @@ public abstract class AbstractWritableArchive { foreach (var path in Entries.Select(x => x.Key)) { + if (path is null) + { + continue; + } var p = path.Replace('/', '\\'); if (p.Length > 0 && p[0] == '\\') { @@ -130,7 +142,43 @@ public abstract class AbstractWritableArchive return false; } - public void SaveTo(Stream stream, WriterOptions options) + ValueTask IWritableAsyncArchive.RemoveEntryAsync(IArchiveEntry entry) => + RemoveEntryAsync((TEntry)entry); + + async ValueTask IWritableAsyncArchive.AddEntryAsync( + string key, + Stream source, + bool closeStream, + long size, + DateTime? modified, + CancellationToken cancellationToken + ) => + await AddEntryAsync(key, source, closeStream, size, modified, cancellationToken) + .ConfigureAwait(false); + + async ValueTask IWritableAsyncArchive.AddDirectoryEntryAsync( + string key, + DateTime? modified, + CancellationToken cancellationToken + ) => await AddDirectoryEntryAsync(key, modified, cancellationToken).ConfigureAwait(false); + + public TEntry AddDirectoryEntry(string key, DateTime? modified = null) + { + if (key.Length > 0 && key[0] is '/' or '\\') + { + key = key.Substring(1); + } + if (DoesKeyMatchExisting(key)) + { + throw new ArchiveException("Cannot add entry with duplicate key: " + key); + } + var entry = CreateDirectoryEntry(key, modified); + newEntries.Add(entry); + RebuildModifiedCollection(); + return entry; + } + + public void SaveTo(Stream stream, TOptions options) { //reset streams of new entries newEntries.Cast().ForEach(x => x.Stream.Seek(0, SeekOrigin.Begin)); @@ -147,7 +195,7 @@ public abstract class AbstractWritableArchive { if (!source.CanRead || !source.CanSeek) { - throw new ArgumentException( + throw new ArchiveException( "Streams must be readable and seekable to use the Writing Archive API" ); } @@ -162,9 +210,11 @@ public abstract class AbstractWritableArchive bool closeStream ); + protected abstract TEntry CreateDirectoryEntry(string key, DateTime? modified); + protected abstract void SaveTo( Stream stream, - WriterOptions options, + TOptions options, IEnumerable oldEntries, IEnumerable newEntries ); diff --git a/src/SharpCompress/Archives/ArchiveFactory.Async.cs b/src/SharpCompress/Archives/ArchiveFactory.Async.cs new file mode 100644 index 00000000..6c45cb1f --- /dev/null +++ b/src/SharpCompress/Archives/ArchiveFactory.Async.cs @@ -0,0 +1,110 @@ +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Readers; + +namespace SharpCompress.Archives; + +public static partial class ArchiveFactory +{ + public static async ValueTask OpenAsyncArchive( + Stream stream, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + readerOptions ??= ReaderOptions.ForExternalStream; + var factory = await FindFactoryAsync(stream, cancellationToken) + .ConfigureAwait(false); + return await factory + .OpenAsyncArchive(stream, readerOptions, cancellationToken) + .ConfigureAwait(false); + } + + public static ValueTask OpenAsyncArchive( + string filePath, + ReaderOptions? options = null, + CancellationToken cancellationToken = default + ) + { + filePath.NotNullOrEmpty(nameof(filePath)); + return OpenAsyncArchive( + new FileInfo(filePath), + options ?? ReaderOptions.ForFilePath, + cancellationToken + ); + } + + public static async ValueTask OpenAsyncArchive( + FileInfo fileInfo, + ReaderOptions? options = null, + CancellationToken cancellationToken = default + ) + { + options ??= ReaderOptions.ForFilePath; + + var factory = await FindFactoryAsync(fileInfo, cancellationToken) + .ConfigureAwait(false); + return await factory + .OpenAsyncArchive(fileInfo, options, cancellationToken) + .ConfigureAwait(false); + } + + public static async ValueTask OpenAsyncArchive( + IReadOnlyList fileInfos, + ReaderOptions? options = null, + CancellationToken cancellationToken = default + ) + { + fileInfos.NotNull(nameof(fileInfos)); + var filesArray = fileInfos; + if (filesArray.Count == 0) + { + throw new ArchiveOperationException("No files to open"); + } + + var fileInfo = filesArray[0]; + if (filesArray.Count == 1) + { + return await OpenAsyncArchive(fileInfo, options, cancellationToken) + .ConfigureAwait(false); + } + + fileInfo.NotNull(nameof(fileInfo)); + options ??= ReaderOptions.ForFilePath; + + var factory = await FindFactoryAsync(fileInfo, cancellationToken) + .ConfigureAwait(false); + return await factory + .OpenAsyncArchive(filesArray, options, cancellationToken) + .ConfigureAwait(false); + } + + public static async ValueTask OpenAsyncArchive( + IReadOnlyList streams, + ReaderOptions? options = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + var streamsArray = streams.RequireReadable().RequireSeekable().ToList(); + var firstStream = streamsArray[0]; + if (streamsArray.Count == 1) + { + return await OpenAsyncArchive(firstStream, options, cancellationToken) + .ConfigureAwait(false); + } + + firstStream.NotNull(nameof(firstStream)); + options ??= ReaderOptions.ForExternalStream; + + var factory = await FindFactoryAsync(firstStream, cancellationToken) + .ConfigureAwait(false); + return await factory + .OpenAsyncArchive(streamsArray, options, cancellationToken) + .ConfigureAwait(false); + } +} diff --git a/src/SharpCompress/Archives/ArchiveFactory.Detection.cs b/src/SharpCompress/Archives/ArchiveFactory.Detection.cs new file mode 100644 index 00000000..a0084914 --- /dev/null +++ b/src/SharpCompress/Archives/ArchiveFactory.Detection.cs @@ -0,0 +1,264 @@ +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Factories; +using SharpCompress.Readers; + +namespace SharpCompress.Archives; + +public static partial class ArchiveFactory +{ + /// + /// Returns information about the archive at the given file path asynchronously, + /// or if the file is not a recognized archive. + /// + /// Path to the archive file. + /// Cancellation token. + public static async ValueTask GetArchiveInformationAsync( + string filePath, + CancellationToken cancellationToken = default + ) => + await GetArchiveInformationAsync(filePath, ReaderOptions.ForFilePath, cancellationToken) + .ConfigureAwait(false); + + /// + /// Returns information about the archive at the given file path asynchronously, + /// or if the file is not a recognized archive. + /// + /// Path to the archive file. + /// Options controlling archive detection. + /// Cancellation token. + public static async ValueTask GetArchiveInformationAsync( + string filePath, + ReaderOptions? readerOptions, + CancellationToken cancellationToken = default + ) + { + filePath.NotNullOrEmpty(nameof(filePath)); + using Stream stream = File.OpenRead(filePath); + return await GetArchiveInformationAsync( + stream, + readerOptions ?? ReaderOptions.ForFilePath, + cancellationToken + ) + .ConfigureAwait(false); + } + + /// + /// Returns information about the archive in the given stream asynchronously, + /// or if the stream is not a recognized archive. + /// + /// A readable and seekable stream positioned at the start of the archive. + /// Cancellation token. + public static async ValueTask GetArchiveInformationAsync( + Stream stream, + CancellationToken cancellationToken = default + ) => + await GetArchiveInformationAsync(stream, ReaderOptions.ForExternalStream, cancellationToken) + .ConfigureAwait(false); + + /// + /// Returns information about the archive in the given stream asynchronously, + /// or if the stream is not a recognized archive. + /// + /// A readable and seekable stream positioned at the start of the archive. + /// Options controlling archive detection. + /// Cancellation token. + public static async ValueTask GetArchiveInformationAsync( + Stream stream, + ReaderOptions? readerOptions, + CancellationToken cancellationToken = default + ) + { + stream.RequireReadable(); + stream.RequireSeekable(); + + var factory = await TryFindFactoryAsync( + stream, + readerOptions ?? ReaderOptions.ForExternalStream, + cancellationToken + ) + .ConfigureAwait(false); + return factory is null + ? null + : new ArchiveInformation(factory.KnownArchiveType, factory is IArchiveFactory); + } + + internal static ValueTask FindFactoryAsync( + string filePath, + CancellationToken cancellationToken = default + ) + where T : IFactory + { + filePath.NotNullOrEmpty(nameof(filePath)); + return FindFactoryAsync(new FileInfo(filePath), cancellationToken); + } + + internal static async ValueTask FindFactoryAsync( + FileInfo fileInfo, + CancellationToken cancellationToken = default + ) + where T : IFactory + { + fileInfo.NotNull(nameof(fileInfo)); + using Stream stream = fileInfo.OpenRead(); + return await FindFactoryAsync(stream, cancellationToken).ConfigureAwait(false); + } + + internal static async ValueTask FindFactoryAsync( + Stream stream, + CancellationToken cancellationToken = default + ) + where T : IFactory + { + stream.RequireReadable(); + stream.RequireSeekable(); + + // Use the shared async detection loop over all factories. If the matched factory + // implements T we return it; otherwise (or if nothing matched) we fall through + // to the same "unsupported format" exception that the original code produced, + // listing the T-typed factories as the hint for the caller. + var factory = await TryFindFactoryAsync(stream, cancellationToken).ConfigureAwait(false); + if (factory is T typedFactory) + { + return typedFactory; + } + + var extensions = string.Join(", ", Factory.Factories.OfType().Select(item => item.Name)); + + throw new ArchiveOperationException( + $"Cannot determine compressed stream type. Supported Archive Formats: {extensions}" + ); + } + + /// + /// Async counterpart of . + /// Iterates all registered factories and returns the first one whose + /// recognises the stream, or . + /// Stream position is restored to its value at entry on both success and failure. + /// + private static async ValueTask TryFindFactoryAsync( + Stream stream, + CancellationToken cancellationToken + ) => + await TryFindFactoryAsync(stream, ReaderOptions.ForExternalStream, cancellationToken) + .ConfigureAwait(false); + + private static async ValueTask TryFindFactoryAsync( + Stream stream, + ReaderOptions readerOptions, + CancellationToken cancellationToken + ) + { + var startPosition = stream.Position; + + foreach (var factory in Factory.Factories) + { + stream.Seek(startPosition, SeekOrigin.Begin); + var isArchive = await factory + .IsArchiveAsync(stream, readerOptions, cancellationToken) + .ConfigureAwait(false); + + if (isArchive) + { + stream.Seek(startPosition, SeekOrigin.Begin); + return factory; + } + } + + stream.Seek(startPosition, SeekOrigin.Begin); + return null; + } + + /// + /// Returns information about the archive at the given file path, + /// or if the file is not a recognized archive. + /// + /// Path to the archive file. + public static ArchiveInformation? GetArchiveInformation(string filePath) => + GetArchiveInformation(filePath, ReaderOptions.ForFilePath); + + /// + /// Returns information about the archive at the given file path, + /// or if the file is not a recognized archive. + /// + /// Path to the archive file. + /// Options controlling archive detection. + public static ArchiveInformation? GetArchiveInformation( + string filePath, + ReaderOptions? readerOptions + ) + { + filePath.NotNullOrEmpty(nameof(filePath)); + using Stream stream = File.OpenRead(filePath); + return GetArchiveInformation(stream, readerOptions ?? ReaderOptions.ForFilePath); + } + + /// + /// Returns information about the archive in the given stream, + /// or if the stream is not a recognized archive. + /// + /// A readable and seekable stream positioned at the start of the archive. + public static ArchiveInformation? GetArchiveInformation(Stream stream) => + GetArchiveInformation(stream, ReaderOptions.ForExternalStream); + + /// + /// Returns information about the archive in the given stream, + /// or if the stream is not a recognized archive. + /// + /// A readable and seekable stream positioned at the start of the archive. + /// Options controlling archive detection. + public static ArchiveInformation? GetArchiveInformation( + Stream stream, + ReaderOptions? readerOptions + ) + { + stream.RequireReadable(); + stream.RequireSeekable(); + + var factory = TryFindFactory(stream, readerOptions ?? ReaderOptions.ForExternalStream); + return factory is null + ? null + : new ArchiveInformation(factory.KnownArchiveType, factory is IArchiveFactory); + } + + /// + /// Iterates all registered factories and returns the first one whose + /// recognises the stream, or . + /// Stream position is restored to its value at entry on both success and failure. + /// + /// + /// This is the shared, seekable-stream detection core used by + /// , , + /// and . + /// + /// uses a separate code path + /// based on rewindable buffering, which supports + /// non-seekable streams and is therefore not unified with this helper. + /// + /// + private static IFactory? TryFindFactory(Stream stream) => + TryFindFactory(stream, ReaderOptions.ForExternalStream); + + private static IFactory? TryFindFactory(Stream stream, ReaderOptions readerOptions) + { + var startPosition = stream.Position; + + foreach (var factory in Factory.Factories) + { + stream.Seek(startPosition, SeekOrigin.Begin); + var isArchive = factory.IsArchive(stream, readerOptions); + + if (isArchive) + { + stream.Seek(startPosition, SeekOrigin.Begin); + return factory; + } + } + + stream.Seek(startPosition, SeekOrigin.Begin); + return null; + } +} diff --git a/src/SharpCompress/Archives/ArchiveFactory.cs b/src/SharpCompress/Archives/ArchiveFactory.cs index cd0952f1..49e2f9b4 100644 --- a/src/SharpCompress/Archives/ArchiveFactory.cs +++ b/src/SharpCompress/Archives/ArchiveFactory.cs @@ -2,224 +2,232 @@ using System; using System.Collections.Generic; using System.IO; using System.Linq; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common; +using SharpCompress.Common.Options; using SharpCompress.Factories; using SharpCompress.Readers; namespace SharpCompress.Archives; -public static class ArchiveFactory +public static partial class ArchiveFactory { - /// - /// Opens an Archive for random access - /// - /// - /// - /// - public static IArchive Open(Stream stream, ReaderOptions? readerOptions = null) + public static IArchive OpenArchive(Stream stream, ReaderOptions? readerOptions = null) { - readerOptions ??= new ReaderOptions(); - - return FindFactory(stream).Open(stream, readerOptions); + readerOptions ??= ReaderOptions.ForExternalStream; + return FindFactory(stream).OpenArchive(stream, readerOptions); } - public static IWritableArchive Create(ArchiveType type) + public static IWritableArchive CreateArchive() + where TOptions : IWriterOptions { - var factory = Factory.Factories - .OfType() - .FirstOrDefault(item => item.KnownArchiveType == type); + var factory = Factory + .Factories.OfType>() + .FirstOrDefault(); if (factory != null) { - return factory.CreateWriteableArchive(); + return factory.CreateArchive(); } - throw new NotSupportedException("Cannot create Archives of type: " + type); + throw new NotSupportedException("Cannot create Archives of type: " + typeof(TOptions)); } - /// - /// Constructor expects a filepath to an existing file. - /// - /// - /// - public static IArchive Open(string filePath, ReaderOptions? options = null) + public static IArchive OpenArchive(string filePath, ReaderOptions? options = null) { - filePath.CheckNotNullOrEmpty(nameof(filePath)); - return Open(new FileInfo(filePath), options); + filePath.NotNullOrEmpty(nameof(filePath)); + return OpenArchive(new FileInfo(filePath), options ?? ReaderOptions.ForFilePath); } - /// - /// Constructor with a FileInfo object to an existing file. - /// - /// - /// - public static IArchive Open(FileInfo fileInfo, ReaderOptions? options = null) + public static IArchive OpenArchive(FileInfo fileInfo, ReaderOptions? options = null) { - options ??= new ReaderOptions { LeaveStreamOpen = false }; + options ??= ReaderOptions.ForFilePath; - return FindFactory(fileInfo).Open(fileInfo, options); + return FindFactory(fileInfo).OpenArchive(fileInfo, options); } - /// - /// Constructor with IEnumerable FileInfo objects, multi and split support. - /// - /// - /// - public static IArchive Open(IEnumerable fileInfos, ReaderOptions? options = null) + public static IArchive OpenArchive( + IReadOnlyList fileInfos, + ReaderOptions? options = null + ) { - fileInfos.CheckNotNull(nameof(fileInfos)); - var filesArray = fileInfos.ToArray(); - if (filesArray.Length == 0) + fileInfos.NotNull(nameof(fileInfos)); + var filesArray = fileInfos; + if (filesArray.Count == 0) { - throw new InvalidOperationException("No files to open"); + throw new ArchiveOperationException("No files to open"); } var fileInfo = filesArray[0]; - if (filesArray.Length == 1) + if (filesArray.Count == 1) { - return Open(fileInfo, options); + return OpenArchive(fileInfo, options); } - fileInfo.CheckNotNull(nameof(fileInfo)); - options ??= new ReaderOptions { LeaveStreamOpen = false }; + fileInfo.NotNull(nameof(fileInfo)); + options ??= ReaderOptions.ForFilePath; - return FindFactory(fileInfo).Open(filesArray, options); + return FindFactory(fileInfo).OpenArchive(filesArray, options); } - /// - /// Constructor with IEnumerable FileInfo objects, multi and split support. - /// - /// - /// - public static IArchive Open(IEnumerable streams, ReaderOptions? options = null) + public static IArchive OpenArchive(IReadOnlyList streams, ReaderOptions? options = null) { - streams.CheckNotNull(nameof(streams)); - var streamsArray = streams.ToArray(); - if (streamsArray.Length == 0) + var streamsArray = streams.RequireReadable().RequireSeekable().ToList(); + if (streamsArray.Count == 0) { - throw new InvalidOperationException("No streams"); + throw new ArchiveOperationException("No streams"); } var firstStream = streamsArray[0]; - if (streamsArray.Length == 1) + if (streamsArray.Count == 1) { - return Open(firstStream, options); + return OpenArchive(firstStream, options); } - firstStream.CheckNotNull(nameof(firstStream)); - options ??= new ReaderOptions(); + firstStream.NotNull(nameof(firstStream)); + options ??= ReaderOptions.ForExternalStream; - return FindFactory(firstStream).Open(streamsArray, options); + return FindFactory(firstStream).OpenArchive(streamsArray, options); } - /// - /// Extract to specific directory, retaining filename - /// public static void WriteToDirectory( string sourceArchive, string destinationDirectory, ExtractionOptions? options = null ) { - using var archive = Open(sourceArchive); - foreach (var entry in archive.Entries) - { - entry.WriteToDirectory(destinationDirectory, options); - } + using var archive = OpenArchive(sourceArchive); + archive.WriteToDirectory(destinationDirectory, options); } - private static T FindFactory(FileInfo finfo) + public static T FindFactory(string filePath) where T : IFactory { - finfo.CheckNotNull(nameof(finfo)); + filePath.NotNullOrEmpty(nameof(filePath)); + using Stream stream = File.OpenRead(filePath); + return FindFactory(stream); + } + + public static T FindFactory(FileInfo finfo) + where T : IFactory + { + finfo.NotNull(nameof(finfo)); using Stream stream = finfo.OpenRead(); return FindFactory(stream); } - private static T FindFactory(Stream stream) + public static T FindFactory(Stream stream) where T : IFactory { - stream.CheckNotNull(nameof(stream)); - if (!stream.CanRead || !stream.CanSeek) + stream.RequireReadable(); + stream.RequireSeekable(); + + // Use the shared detection loop over all factories. If the matched factory + // implements T we return it; otherwise (or if nothing matched) we fall through + // to the same "unsupported format" exception that the original code produced, + // listing the T-typed factories as the hint for the caller. + var factory = TryFindFactory(stream); + if (factory is T typedFactory) { - throw new ArgumentException("Stream should be readable and seekable"); + return typedFactory; } - var factories = Factory.Factories.OfType(); + var extensions = string.Join(", ", Factory.Factories.OfType().Select(item => item.Name)); - var startPosition = stream.Position; - - foreach (var factory in factories) - { - stream.Seek(startPosition, SeekOrigin.Begin); - - if (factory.IsArchive(stream)) - { - stream.Seek(startPosition, SeekOrigin.Begin); - - return factory; - } - } - - var extensions = string.Join(", ", factories.Select(item => item.Name)); - - throw new InvalidOperationException( + throw new ArchiveOperationException( $"Cannot determine compressed stream type. Supported Archive Formats: {extensions}" ); } public static bool IsArchive(string filePath, out ArchiveType? type) { - filePath.CheckNotNullOrEmpty(nameof(filePath)); + return IsArchive(filePath, ReaderOptions.ForFilePath, out type); + } + + public static bool IsArchive( + string filePath, + ReaderOptions? readerOptions, + out ArchiveType? type + ) + { + filePath.NotNullOrEmpty(nameof(filePath)); using Stream s = File.OpenRead(filePath); - return IsArchive(s, out type); + return IsArchive(s, readerOptions ?? ReaderOptions.ForFilePath, out type); } public static bool IsArchive(Stream stream, out ArchiveType? type) { - type = null; - stream.CheckNotNull(nameof(stream)); - - if (!stream.CanRead || !stream.CanSeek) - { - throw new ArgumentException("Stream should be readable and seekable"); - } - - var startPosition = stream.Position; - - foreach (var factory in Factory.Factories) - { - stream.Position = startPosition; - - if (factory.IsArchive(stream, null)) - { - type = factory.KnownArchiveType; - return true; - } - } - - return false; + return IsArchive(stream, ReaderOptions.ForExternalStream, out type); + } + + public static bool IsArchive(Stream stream, ReaderOptions? readerOptions, out ArchiveType? type) + { + stream.RequireReadable(); + stream.RequireSeekable(); + + var factory = TryFindFactory(stream, readerOptions ?? ReaderOptions.ForExternalStream); + type = factory?.KnownArchiveType; + return factory is not null; + } + + public static async ValueTask<(bool IsArchive, ArchiveType? Type)> IsArchiveAsync( + string filePath, + CancellationToken cancellationToken = default + ) => + await IsArchiveAsync(filePath, ReaderOptions.ForFilePath, cancellationToken) + .ConfigureAwait(false); + + public static async ValueTask<(bool IsArchive, ArchiveType? Type)> IsArchiveAsync( + string filePath, + ReaderOptions? readerOptions, + CancellationToken cancellationToken = default + ) + { + filePath.NotNullOrEmpty(nameof(filePath)); + using Stream stream = File.OpenRead(filePath); + return await IsArchiveAsync( + stream, + readerOptions ?? ReaderOptions.ForFilePath, + cancellationToken + ) + .ConfigureAwait(false); + } + + public static async ValueTask<(bool IsArchive, ArchiveType? Type)> IsArchiveAsync( + Stream stream, + CancellationToken cancellationToken = default + ) => + await IsArchiveAsync(stream, ReaderOptions.ForExternalStream, cancellationToken) + .ConfigureAwait(false); + + public static async ValueTask<(bool IsArchive, ArchiveType? Type)> IsArchiveAsync( + Stream stream, + ReaderOptions? readerOptions, + CancellationToken cancellationToken = default + ) + { + stream.RequireReadable(); + stream.RequireSeekable(); + + var factory = await TryFindFactoryAsync( + stream, + readerOptions ?? ReaderOptions.ForExternalStream, + cancellationToken + ) + .ConfigureAwait(false); + return (factory is not null, factory?.KnownArchiveType); } - /// - /// From a passed in archive (zip, rar, 7z, 001), return all parts. - /// - /// - /// public static IEnumerable GetFileParts(string part1) { - part1.CheckNotNullOrEmpty(nameof(part1)); + part1.NotNullOrEmpty(nameof(part1)); return GetFileParts(new FileInfo(part1)).Select(a => a.FullName); } - /// - /// From a passed in archive (zip, rar, 7z, 001), return all parts. - /// - /// - /// public static IEnumerable GetFileParts(FileInfo part1) { - part1.CheckNotNull(nameof(part1)); + part1.NotNull(nameof(part1)); yield return part1; foreach (var factory in Factory.Factories.OfType()) @@ -230,7 +238,7 @@ public static class ArchiveFactory if (part != null) { yield return part; - while ((part = factory.GetFilePart(i++, part1)) != null) //tests split too + while ((part = factory.GetFilePart(i++, part1)) != null) { yield return part; } diff --git a/src/SharpCompress/Archives/ArchiveInformation.cs b/src/SharpCompress/Archives/ArchiveInformation.cs new file mode 100644 index 00000000..75d3cb5a --- /dev/null +++ b/src/SharpCompress/Archives/ArchiveInformation.cs @@ -0,0 +1,38 @@ +using SharpCompress.Common; + +namespace SharpCompress.Archives; + +/// +/// Contains information about a detected archive, including its type and supported capabilities. +/// +/// +/// Use or +/// +/// to obtain an instance of this record. +/// +public record ArchiveInformation +{ + /// + /// The type of archive detected, or when the format is not a registered well-known type. + /// + public ArchiveType? Type { get; set; } + + /// + /// when this archive format supports random access via the API, + /// meaning the full file listing can be retrieved without decompressing the entire archive. + /// when only the API is available, + /// which reads entries sequentially and can only report per-entry progress. + /// + public bool SupportsRandomAccess { get; set; } + + /// + /// Creates a new archive information instance. + /// + /// The detected archive type. + /// Whether the detected format supports random access. + public ArchiveInformation(ArchiveType? type, bool supportsRandomAccess) + { + Type = type; + SupportsRandomAccess = supportsRandomAccess; + } +} diff --git a/src/SharpCompress/Archives/ArchiveVolumeFactory.cs b/src/SharpCompress/Archives/ArchiveVolumeFactory.cs index d8f5535f..92ed6818 100644 --- a/src/SharpCompress/Archives/ArchiveVolumeFactory.cs +++ b/src/SharpCompress/Archives/ArchiveVolumeFactory.cs @@ -1,6 +1,7 @@ using System; using System.IO; using System.Text.RegularExpressions; +using SharpCompress.Common; namespace SharpCompress.Archives; @@ -11,20 +12,27 @@ internal abstract class ArchiveVolumeFactory FileInfo? item = null; //split 001, 002 ... - Match m = Regex.Match(part1.Name, @"^(.*\.)([0-9]+)$", RegexOptions.IgnoreCase); + var m = Regex.Match(part1.Name, @"^(.*\.)([0-9]+)$", RegexOptions.IgnoreCase); if (m.Success) + { item = new FileInfo( Path.Combine( part1.DirectoryName!, String.Concat( m.Groups[1].Value, - (index + 1).ToString().PadLeft(m.Groups[2].Value.Length, '0') + (index + 1) + .ToString(Constants.DefaultCultureInfo) + .PadLeft(m.Groups[2].Value.Length, '0') ) ) ); + } if (item != null && item.Exists) + { return item; + } + return null; } } diff --git a/src/SharpCompress/Archives/GZip/GZipArchive.Async.cs b/src/SharpCompress/Archives/GZip/GZipArchive.Async.cs new file mode 100644 index 00000000..49f03937 --- /dev/null +++ b/src/SharpCompress/Archives/GZip/GZipArchive.Async.cs @@ -0,0 +1,93 @@ +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.GZip; +using SharpCompress.Common.Options; +using SharpCompress.Readers; +using SharpCompress.Readers.GZip; +using SharpCompress.Writers; +using SharpCompress.Writers.GZip; + +namespace SharpCompress.Archives.GZip; + +public partial class GZipArchive +{ + public ValueTask SaveToAsync(string filePath, CancellationToken cancellationToken = default) => + SaveToAsync(new FileInfo(filePath), cancellationToken); + + public async ValueTask SaveToAsync( + FileInfo fileInfo, + CancellationToken cancellationToken = default + ) + { + using var stream = fileInfo.Open(FileMode.Create, FileAccess.Write); + await SaveToAsync(stream, new GZipWriterOptions(CompressionType.GZip), cancellationToken) + .ConfigureAwait(false); + } + + protected override async ValueTask SaveToAsync( + Stream stream, + GZipWriterOptions options, + IAsyncEnumerable oldEntries, + IEnumerable newEntries, + CancellationToken cancellationToken = default + ) + { + if (Entries.Count > 1) + { + throw new InvalidFormatException("Only one entry is allowed in a GZip Archive"); + } + await using var writer = new GZipWriter(stream, options); + await foreach ( + var entry in oldEntries.WithCancellation(cancellationToken).ConfigureAwait(false) + ) + { + if (!entry.IsDirectory) + { + using var entryStream = await entry + .OpenEntryStreamAsync(cancellationToken) + .ConfigureAwait(false); + await writer + .WriteAsync( + entry.Key.NotNull("Entry Key is null"), + entryStream, + cancellationToken + ) + .ConfigureAwait(false); + } + } + foreach (var entry in newEntries.Where(x => !x.IsDirectory)) + { + using var entryStream = await entry + .OpenEntryStreamAsync(cancellationToken) + .ConfigureAwait(false); + await writer + .WriteAsync(entry.Key.NotNull("Entry Key is null"), entryStream, cancellationToken) + .ConfigureAwait(false); + } + } + + protected override ValueTask CreateReaderForSolidExtractionAsync() + { + var stream = Volumes.Single().Stream; + stream.Position = 0; + return new((IAsyncReader)GZipReader.OpenReader(stream, ReaderOptions)); + } + + protected override async IAsyncEnumerable LoadEntriesAsync( + IAsyncEnumerable volumes + ) + { + var stream = (await volumes.SingleAsync().ConfigureAwait(false)).Stream; + yield return new GZipArchiveEntry( + this, + await GZipFilePart + .CreateAsync(stream, ReaderOptions.ArchiveEncoding, ReaderOptions.Providers) + .ConfigureAwait(false), + ReaderOptions + ); + } +} diff --git a/src/SharpCompress/Archives/GZip/GZipArchive.Factory.cs b/src/SharpCompress/Archives/GZip/GZipArchive.Factory.cs new file mode 100644 index 00000000..2f5e5e33 --- /dev/null +++ b/src/SharpCompress/Archives/GZip/GZipArchive.Factory.cs @@ -0,0 +1,199 @@ +using System; +using System.Buffers; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.IO; +using SharpCompress.Readers; +using SharpCompress.Writers.GZip; + +namespace SharpCompress.Archives.GZip; + +public partial class GZipArchive +#if NET8_0_OR_GREATER + : IWritableArchiveOpenable, + IMultiArchiveOpenable< + IWritableArchive, + IWritableAsyncArchive + > +#endif +{ + public static ValueTask> OpenAsyncArchive( + string filePath, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + filePath.NotNullOrEmpty(nameof(filePath)); + return OpenAsyncArchive(new FileInfo(filePath), readerOptions, cancellationToken); + } + + public static IWritableArchive OpenArchive( + string filePath, + ReaderOptions? readerOptions = null + ) + { + filePath.NotNullOrEmpty(nameof(filePath)); + return OpenArchive(new FileInfo(filePath), readerOptions ?? ReaderOptions.ForFilePath); + } + + public static IWritableArchive OpenArchive( + FileInfo fileInfo, + ReaderOptions? readerOptions = null + ) + { + fileInfo.NotNull(nameof(fileInfo)); + return new GZipArchive( + new SourceStream( + fileInfo, + i => ArchiveVolumeFactory.GetFilePart(i, fileInfo), + readerOptions ?? ReaderOptions.ForFilePath + ) + ); + } + + public static IWritableArchive OpenArchive( + IReadOnlyList fileInfos, + ReaderOptions? readerOptions = null + ) + { + fileInfos.NotNull(nameof(fileInfos)); + var files = fileInfos; + return new GZipArchive( + new SourceStream( + files[0], + i => i < files.Count ? files[i] : null, + readerOptions ?? ReaderOptions.ForFilePath + ) + ); + } + + public static IWritableArchive OpenArchive( + IReadOnlyList streams, + ReaderOptions? readerOptions = null + ) + { + var strms = streams.RequireReadable().RequireSeekable().ToList(); + return new GZipArchive( + new SourceStream( + strms[0], + i => i < strms.Count ? strms[i] : null, + readerOptions ?? ReaderOptions.ForExternalStream + ) + ); + } + + public static IWritableArchive OpenArchive( + Stream stream, + ReaderOptions? readerOptions = null + ) + { + stream.RequireReadable(); + stream.RequireSeekable(); + + return new GZipArchive( + new SourceStream(stream, _ => null, readerOptions ?? ReaderOptions.ForExternalStream) + ); + } + + public static ValueTask> OpenAsyncArchive( + Stream stream, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IWritableAsyncArchive)OpenArchive(stream, readerOptions)); + } + + public static ValueTask> OpenAsyncArchive( + FileInfo fileInfo, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IWritableAsyncArchive)OpenArchive(fileInfo, readerOptions)); + } + + public static ValueTask> OpenAsyncArchive( + IReadOnlyList streams, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IWritableAsyncArchive)OpenArchive(streams, readerOptions)); + } + + public static ValueTask> OpenAsyncArchive( + IReadOnlyList fileInfos, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IWritableAsyncArchive)OpenArchive(fileInfos, readerOptions)); + } + + public static IWritableArchive CreateArchive() => new GZipArchive(); + + public static ValueTask> CreateAsyncArchive() => + new(new GZipArchive()); + + public static bool IsGZipFile(string filePath) => IsGZipFile(new FileInfo(filePath)); + + public static bool IsGZipFile(FileInfo fileInfo) + { + if (!fileInfo.Exists) + { + return false; + } + + using Stream stream = fileInfo.OpenRead(); + return IsGZipFile(stream); + } + + public static bool IsGZipFile(Stream stream) + { + Span header = stackalloc byte[10]; + + if (!stream.ReadFully(header)) + { + return false; + } + + if (header[0] != 0x1F || header[1] != 0x8B || header[2] != 8) + { + return false; + } + + return true; + } + + public static async ValueTask IsGZipFileAsync( + Stream stream, + CancellationToken cancellationToken = default + ) + { + var header = ArrayPool.Shared.Rent(10); + try + { + await stream.ReadFullyAsync(header, 0, 10, cancellationToken).ConfigureAwait(false); + + if (header[0] != 0x1F || header[1] != 0x8B || header[2] != 8) + { + return false; + } + + return true; + } + finally + { + ArrayPool.Shared.Return(header); + } + } +} diff --git a/src/SharpCompress/Archives/GZip/GZipArchive.cs b/src/SharpCompress/Archives/GZip/GZipArchive.cs index fb2f7691..e47bf7c6 100644 --- a/src/SharpCompress/Archives/GZip/GZipArchive.cs +++ b/src/SharpCompress/Archives/GZip/GZipArchive.cs @@ -4,6 +4,7 @@ using System.IO; using System.Linq; using SharpCompress.Common; using SharpCompress.Common.GZip; +using SharpCompress.Common.Options; using SharpCompress.IO; using SharpCompress.Readers; using SharpCompress.Readers.GZip; @@ -12,116 +13,19 @@ using SharpCompress.Writers.GZip; namespace SharpCompress.Archives.GZip; -public class GZipArchive : AbstractWritableArchive +public partial class GZipArchive + : AbstractWritableArchive { - /// - /// Constructor expects a filepath to an existing file. - /// - /// - /// - public static GZipArchive Open(string filePath, ReaderOptions? readerOptions = null) + private GZipArchive(SourceStream sourceStream) + : base(ArchiveType.GZip, sourceStream) { } + + internal GZipArchive() + : base(ArchiveType.GZip) { } + + protected override IEnumerable LoadVolumes(SourceStream sourceStream) { - filePath.CheckNotNullOrEmpty(nameof(filePath)); - return Open(new FileInfo(filePath), readerOptions ?? new ReaderOptions()); - } - - /// - /// Constructor with a FileInfo object to an existing file. - /// - /// - /// - public static GZipArchive Open(FileInfo fileInfo, ReaderOptions? readerOptions = null) - { - fileInfo.CheckNotNull(nameof(fileInfo)); - return new GZipArchive( - new SourceStream( - fileInfo, - i => ArchiveVolumeFactory.GetFilePart(i, fileInfo), - readerOptions ?? new ReaderOptions() - ) - ); - } - - /// - /// Constructor with all file parts passed in - /// - /// - /// - public static GZipArchive Open( - IEnumerable fileInfos, - ReaderOptions? readerOptions = null - ) - { - fileInfos.CheckNotNull(nameof(fileInfos)); - var files = fileInfos.ToArray(); - return new GZipArchive( - new SourceStream( - files[0], - i => i < files.Length ? files[i] : null, - readerOptions ?? new ReaderOptions() - ) - ); - } - - /// - /// Constructor with all stream parts passed in - /// - /// - /// - public static GZipArchive Open(IEnumerable streams, ReaderOptions? readerOptions = null) - { - streams.CheckNotNull(nameof(streams)); - var strms = streams.ToArray(); - return new GZipArchive( - new SourceStream( - strms[0], - i => i < strms.Length ? strms[i] : null, - readerOptions ?? new ReaderOptions() - ) - ); - } - - /// - /// Takes a seekable Stream as a source - /// - /// - /// - public static GZipArchive Open(Stream stream, ReaderOptions? readerOptions = null) - { - stream.CheckNotNull(nameof(stream)); - return new GZipArchive( - new SourceStream(stream, i => null, readerOptions ?? new ReaderOptions()) - ); - } - - public static GZipArchive Create() => new GZipArchive(); - - /// - /// Constructor with a SourceStream able to handle FileInfo and Streams. - /// - /// - /// - internal GZipArchive(SourceStream srcStream) - : base(ArchiveType.Tar, srcStream) { } - - protected override IEnumerable LoadVolumes(SourceStream srcStream) - { - srcStream.LoadAllParts(); - var idx = 0; - return srcStream.Streams.Select(a => new GZipVolume(a, ReaderOptions, idx++)); - } - - public static bool IsGZipFile(string filePath) => IsGZipFile(new FileInfo(filePath)); - - public static bool IsGZipFile(FileInfo fileInfo) - { - if (!fileInfo.Exists) - { - return false; - } - - using Stream stream = fileInfo.OpenRead(); - return IsGZipFile(stream); + sourceStream.LoadAllParts(); + return sourceStream.Streams.Select(a => new GZipVolume(a, ReaderOptions, 0)); } public void SaveTo(string filePath) => SaveTo(new FileInfo(filePath)); @@ -129,33 +33,11 @@ public class GZipArchive : AbstractWritableArchive public void SaveTo(FileInfo fileInfo) { using var stream = fileInfo.Open(FileMode.Create, FileAccess.Write); - SaveTo(stream, new WriterOptions(CompressionType.GZip)); + SaveTo(stream, new GZipWriterOptions(CompressionType.GZip)); } - public static bool IsGZipFile(Stream stream) - { - // read the header on the first read - Span header = stackalloc byte[10]; - - // workitem 8501: handle edge case (decompress empty stream) - if (!stream.ReadFully(header)) - { - return false; - } - - if (header[0] != 0x1F || header[1] != 0x8B || header[2] != 8) - { - return false; - } - - return true; - } - - internal GZipArchive() - : base(ArchiveType.GZip) { } - protected override GZipArchiveEntry CreateEntryInternal( - string filePath, + string key, Stream source, long size, DateTime? modified, @@ -164,27 +46,34 @@ public class GZipArchive : AbstractWritableArchive { if (Entries.Any()) { - throw new InvalidOperationException("Only one entry is allowed in a GZip Archive"); + throw new InvalidFormatException("Only one entry is allowed in a GZip Archive"); } - return new GZipWritableArchiveEntry(this, source, filePath, size, modified, closeStream); + return new GZipWritableArchiveEntry(this, source, key, size, modified, closeStream); } + protected override GZipArchiveEntry CreateDirectoryEntry(string key, DateTime? modified) => + throw new NotSupportedException("GZip archives do not support directory entries."); + protected override void SaveTo( Stream stream, - WriterOptions options, + GZipWriterOptions options, IEnumerable oldEntries, IEnumerable newEntries ) { if (Entries.Count > 1) { - throw new InvalidOperationException("Only one entry is allowed in a GZip Archive"); + throw new InvalidFormatException("Only one entry is allowed in a GZip Archive"); } - using var writer = new GZipWriter(stream, new GZipWriterOptions(options)); + using var writer = new GZipWriter(stream, options); foreach (var entry in oldEntries.Concat(newEntries).Where(x => !x.IsDirectory)) { using var entryStream = entry.OpenEntryStream(); - writer.Write(entry.Key, entryStream, entry.LastModifiedTime); + writer.Write( + entry.Key.NotNull("Entry Key is null"), + entryStream, + entry.LastModifiedTime + ); } } @@ -193,7 +82,8 @@ public class GZipArchive : AbstractWritableArchive var stream = volumes.Single().Stream; yield return new GZipArchiveEntry( this, - new GZipFilePart(stream, ReaderOptions.ArchiveEncoding) + GZipFilePart.Create(stream, ReaderOptions.ArchiveEncoding, ReaderOptions.Providers), + ReaderOptions ); } @@ -201,6 +91,6 @@ public class GZipArchive : AbstractWritableArchive { var stream = Volumes.Single().Stream; stream.Position = 0; - return GZipReader.Open(stream); + return GZipReader.OpenReader(stream, ReaderOptions); } } diff --git a/src/SharpCompress/Archives/GZip/GZipArchiveEntry.cs b/src/SharpCompress/Archives/GZip/GZipArchiveEntry.cs index be872e80..9d975788 100644 --- a/src/SharpCompress/Archives/GZip/GZipArchiveEntry.cs +++ b/src/SharpCompress/Archives/GZip/GZipArchiveEntry.cs @@ -1,23 +1,43 @@ -using System.IO; +using System.IO; using System.Linq; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common.GZip; +using SharpCompress.Common.Options; namespace SharpCompress.Archives.GZip; public class GZipArchiveEntry : GZipEntry, IArchiveEntry { - internal GZipArchiveEntry(GZipArchive archive, GZipFilePart part) - : base(part) => Archive = archive; + internal GZipArchiveEntry(GZipArchive archive, GZipFilePart? part, IReaderOptions readerOptions) + : base(part, readerOptions) => Archive = archive; public virtual Stream OpenEntryStream() { //this is to reset the stream to be read multiple times var part = (GZipFilePart)Parts.Single(); - if (part.GetRawStream().Position != part.EntryStartPosition) + var rawStream = part.GetRawStream(); + if (rawStream.CanSeek && rawStream.Position != part.EntryStartPosition) { - part.GetRawStream().Position = part.EntryStartPosition; + rawStream.Position = part.EntryStartPosition; } - return Parts.Single().GetCompressedStream(); + return Parts.Single().GetCompressedStream().NotNull(); + } + + public virtual async ValueTask OpenEntryStreamAsync( + CancellationToken cancellationToken = default + ) + { + // Reset the stream position if seekable + var part = (GZipFilePart)Parts.Single(); + var rawStream = part.GetRawStream(); + if (rawStream.CanSeek && rawStream.Position != part.EntryStartPosition) + { + rawStream.Position = part.EntryStartPosition; + } + return ( + await Parts.Single().GetCompressedStreamAsync(cancellationToken).ConfigureAwait(false) + ).NotNull(); } #region IArchiveEntry Members diff --git a/src/SharpCompress/Archives/GZip/GZipWritableArchiveEntry.cs b/src/SharpCompress/Archives/GZip/GZipWritableArchiveEntry.cs index 27dfc2bf..740ef0ed 100644 --- a/src/SharpCompress/Archives/GZip/GZipWritableArchiveEntry.cs +++ b/src/SharpCompress/Archives/GZip/GZipWritableArchiveEntry.cs @@ -1,8 +1,8 @@ -#nullable disable - using System; using System.Collections.Generic; using System.IO; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common; using SharpCompress.IO; @@ -21,7 +21,7 @@ internal sealed class GZipWritableArchiveEntry : GZipArchiveEntry, IWritableArch DateTime? lastModified, bool closeStream ) - : base(archive, null) + : base(archive, null, archive.ReaderOptions) { this.stream = stream; Key = path; @@ -32,7 +32,7 @@ internal sealed class GZipWritableArchiveEntry : GZipArchiveEntry, IWritableArch public override long Crc => 0; - public override string Key { get; } + public override string? Key { get; } public override long CompressedSize => 0; @@ -60,7 +60,15 @@ internal sealed class GZipWritableArchiveEntry : GZipArchiveEntry, IWritableArch { //ensure new stream is at the start, this could be reset stream.Seek(0, SeekOrigin.Begin); - return NonDisposingStream.Create(stream); + return SharpCompressStream.CreateNonDisposing(stream); + } + + public override ValueTask OpenEntryStreamAsync( + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new(OpenEntryStream()); } internal override void Close() diff --git a/src/SharpCompress/Archives/IArchive.cs b/src/SharpCompress/Archives/IArchive.cs index 154529bf..ba0f74a5 100644 --- a/src/SharpCompress/Archives/IArchive.cs +++ b/src/SharpCompress/Archives/IArchive.cs @@ -7,17 +7,16 @@ namespace SharpCompress.Archives; public interface IArchive : IDisposable { - event EventHandler> EntryExtractionBegin; - event EventHandler> EntryExtractionEnd; - - event EventHandler CompressedBytesRead; - event EventHandler FilePartExtractionBegin; - IEnumerable Entries { get; } IEnumerable Volumes { get; } ArchiveType Type { get; } + /// + /// The options used when opening this archive. + /// + ReaderOptions ReaderOptions { get; } + /// /// Use this method to extract all entries in an archive in order. /// This is primarily for SOLID Rar Archives or 7Zip Archives as they need to be @@ -44,5 +43,10 @@ public interface IArchive : IDisposable /// /// The total size of the files as uncompressed in the archive. /// - long TotalUncompressSize { get; } + long TotalUncompressedSize { get; } + + /// + /// Returns whether the archive is encrypted. + /// + bool IsEncrypted { get; } } diff --git a/src/SharpCompress/Archives/IArchiveEntry.cs b/src/SharpCompress/Archives/IArchiveEntry.cs index 708753cb..a38e65a0 100644 --- a/src/SharpCompress/Archives/IArchiveEntry.cs +++ b/src/SharpCompress/Archives/IArchiveEntry.cs @@ -1,4 +1,6 @@ using System.IO; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common; namespace SharpCompress.Archives; @@ -11,6 +13,12 @@ public interface IArchiveEntry : IEntry /// Stream OpenEntryStream(); + /// + /// Opens the current entry as a stream that will decompress as it is read asynchronously. + /// Read the entire stream or use SkipEntry on EntryStream. + /// + ValueTask OpenEntryStreamAsync(CancellationToken cancellationToken = default); + /// /// The archive can find all the parts of the archive needed to extract this entry. /// diff --git a/src/SharpCompress/Archives/IArchiveEntryExtensions.cs b/src/SharpCompress/Archives/IArchiveEntryExtensions.cs index 0992f152..502a419e 100644 --- a/src/SharpCompress/Archives/IArchiveEntryExtensions.cs +++ b/src/SharpCompress/Archives/IArchiveEntryExtensions.cs @@ -1,4 +1,7 @@ +using System; using System.IO; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common; using SharpCompress.IO; @@ -6,65 +9,233 @@ namespace SharpCompress.Archives; public static class IArchiveEntryExtensions { - public static void WriteTo(this IArchiveEntry archiveEntry, Stream streamToWriteTo) + /// The archive entry to extract. + extension(IArchiveEntry archiveEntry) { - if (archiveEntry.IsDirectory) + /// + /// Extract entry to the specified stream. + /// + /// The stream to write the entry content to. + /// Optional progress reporter for tracking extraction progress. + public void WriteTo(Stream streamToWriteTo, IProgress? progress = null) => + archiveEntry.WriteTo(streamToWriteTo, bufferSize: null, progress: progress); + + /// + /// Extract entry to the specified stream. + /// + /// The stream to write the entry content to. + /// Options for configuring extraction behavior. + /// Optional progress reporter for tracking extraction progress. + public void WriteTo( + Stream streamToWriteTo, + ExtractionOptions options, + IProgress? progress = null + ) => archiveEntry.WriteTo(streamToWriteTo, options.BufferSize, options, progress); + + private void WriteTo( + Stream streamToWriteTo, + int? bufferSize, + ExtractionOptions? options = null, + IProgress? progress = null + ) { - throw new ExtractionException("Entry is a file directory and cannot be extracted."); + if (archiveEntry.IsDirectory) + { + throw new ExtractionException("Entry is a file directory and cannot be extracted."); + } + + using var entryStream = archiveEntry.OpenEntryStream(); + var checkedStream = options is null + ? entryStream + : IEntryExtensions.WrapWithChecksumValidation(archiveEntry, entryStream, options); + var sourceStream = WrapWithProgress(checkedStream, archiveEntry, progress); + sourceStream.CopyTo(streamToWriteTo, bufferSize ?? Constants.BufferSize); } - var streamListener = (IArchiveExtractionListener)archiveEntry.Archive; - streamListener.EnsureEntriesLoaded(); - streamListener.FireEntryExtractionBegin(archiveEntry); - streamListener.FireFilePartExtractionBegin( - archiveEntry.Key, - archiveEntry.Size, - archiveEntry.CompressedSize - ); - var entryStream = archiveEntry.OpenEntryStream(); - if (entryStream is null) + /// + /// Extract entry to the specified stream asynchronously. + /// + /// The stream to write the entry content to. + /// Cancellation token. + /// Optional progress reporter for tracking extraction progress. + public async ValueTask WriteToAsync( + Stream streamToWriteTo, + IProgress? progress = null, + CancellationToken cancellationToken = default + ) => + await archiveEntry + .WriteToAsync( + streamToWriteTo, + Constants.BufferSize, + progress: progress, + cancellationToken: cancellationToken + ) + .ConfigureAwait(false); + + /// + /// Extract entry to the specified stream asynchronously. + /// + /// The stream to write the entry content to. + /// Options for configuring extraction behavior. + /// Optional progress reporter for tracking extraction progress. + /// Cancellation token. + public async ValueTask WriteToAsync( + Stream streamToWriteTo, + ExtractionOptions options, + IProgress? progress = null, + CancellationToken cancellationToken = default + ) => + await archiveEntry + .WriteToAsync( + streamToWriteTo, + options.BufferSize, + options, + progress, + cancellationToken + ) + .ConfigureAwait(false); + + private async ValueTask WriteToAsync( + Stream streamToWriteTo, + int? bufferSize, + ExtractionOptions? options = null, + IProgress? progress = null, + CancellationToken cancellationToken = default + ) { - return; + if (archiveEntry.IsDirectory) + { + throw new ExtractionException("Entry is a file directory and cannot be extracted."); + } + +#if LEGACY_DOTNET + using var entryStream = await archiveEntry + .OpenEntryStreamAsync(cancellationToken) + .ConfigureAwait(false); +#else + await using var entryStream = await archiveEntry + .OpenEntryStreamAsync(cancellationToken) + .ConfigureAwait(false); +#endif + var checkedStream = options is null + ? entryStream + : IEntryExtensions.WrapWithChecksumValidation(archiveEntry, entryStream, options); + var sourceStream = WrapWithProgress(checkedStream, archiveEntry, progress); + await sourceStream + .CopyToAsync(streamToWriteTo, bufferSize ?? Constants.BufferSize, cancellationToken) + .ConfigureAwait(false); } - using (entryStream) - { - using Stream s = new ListeningStream(streamListener, entryStream); - s.TransferTo(streamToWriteTo); - } - streamListener.FireEntryExtractionEnd(archiveEntry); } - /// - /// Extract to specific directory, retaining filename - /// - public static void WriteToDirectory( - this IArchiveEntry entry, - string destinationDirectory, - ExtractionOptions? options = null - ) => - ExtractionMethods.WriteEntryToDirectory( - entry, - destinationDirectory, - options, - entry.WriteToFile - ); + private static Stream WrapWithProgress( + Stream source, + IArchiveEntry entry, + IProgress? progress + ) + { + if (progress is null) + { + return source; + } - /// - /// Extract to specific file - /// - public static void WriteToFile( - this IArchiveEntry entry, - string destinationFileName, - ExtractionOptions? options = null - ) => - ExtractionMethods.WriteEntryToFile( - entry, - destinationFileName, - options, - (x, fm) => - { - using var fs = File.Open(destinationFileName, fm); - entry.WriteTo(fs); - } + var entryPath = entry.Key ?? string.Empty; + var totalBytes = GetEntrySizeSafe(entry); + return new ProgressReportingStream( + source, + progress, + entryPath, + totalBytes, + leaveOpen: true ); + } + + private static long? GetEntrySizeSafe(IArchiveEntry entry) + { + try + { + var size = entry.Size; + return size >= 0 ? size : null; + } + catch (NotImplementedException) + { + return null; + } + } + + extension(IArchiveEntry entry) + { + /// + /// Extract to specific directory, retaining filename + /// + public void WriteToDirectory( + string destinationDirectory, + ExtractionOptions? options = null + ) => + entry.WriteEntryToDirectory( + destinationDirectory, + options, + (path) => entry.WriteToFile(path, options) + ); + + /// + /// Extract to specific directory asynchronously, retaining filename + /// + public async ValueTask WriteToDirectoryAsync( + string destinationDirectory, + ExtractionOptions? options = null, + CancellationToken cancellationToken = default + ) => + await entry + .WriteEntryToDirectoryAsync( + destinationDirectory, + options, + async (path, ct) => + await entry.WriteToFileAsync(path, options, ct).ConfigureAwait(false), + cancellationToken + ) + .ConfigureAwait(false); + + /// + /// Extract to specific file + /// + public void WriteToFile(string destinationFileName, ExtractionOptions? options = null) + { + options ??= new ExtractionOptions(); + entry.WriteEntryToFile( + destinationFileName, + options, + (x, fm) => + { + using var fs = File.Open(x, fm); + entry.WriteTo(fs, options?.BufferSize ?? Constants.BufferSize, options, null); + } + ); + } + + /// + /// Extract to specific file asynchronously + /// + public async ValueTask WriteToFileAsync( + string destinationFileName, + ExtractionOptions? options = null, + CancellationToken cancellationToken = default + ) + { + options ??= new ExtractionOptions(); + await entry + .WriteEntryToFileAsync( + destinationFileName, + options, + async (x, fm, ct) => + { + using var fs = File.Open(x, fm); + await entry + .WriteToAsync(fs, options.BufferSize, options, null, ct) + .ConfigureAwait(false); + }, + cancellationToken + ) + .ConfigureAwait(false); + } + } } diff --git a/src/SharpCompress/Archives/IArchiveExtensions.cs b/src/SharpCompress/Archives/IArchiveExtensions.cs index 56ea0d9f..d99a7afd 100644 --- a/src/SharpCompress/Archives/IArchiveExtensions.cs +++ b/src/SharpCompress/Archives/IArchiveExtensions.cs @@ -1,77 +1,69 @@ using System; -using System.Collections.Generic; -using System.IO; -using System.Linq; -using System.Threading; -using System.Threading.Tasks; using SharpCompress.Common; +using SharpCompress.Readers; namespace SharpCompress.Archives; public static class IArchiveExtensions { - /// - /// Extract to specific directory, retaining filename - /// - public static void WriteToDirectory( - this IArchive archive, - string destinationDirectory, - ExtractionOptions? options = null - ) + extension(IArchive archive) { - foreach (var entry in archive.Entries.Where(x => !x.IsDirectory)) + /// + /// Extract to specific directory with progress reporting + /// + /// The folder to extract into. + /// Extraction options. + /// Optional progress reporter for tracking extraction progress. + public void WriteToDirectory( + string destinationDirectory, + ExtractionOptions? options = null, + IProgress? progress = null + ) { - entry.WriteToDirectory(destinationDirectory, options); + if (archive.IsSolid || archive.Type == ArchiveType.SevenZip) + { + using var reader = archive.ExtractAllEntries(); + reader.WriteAllToDirectory(destinationDirectory, options); + } + else + { + archive.WriteToDirectoryInternal(destinationDirectory, options, progress); + } } - } - /// - /// Extracts the archive to the destination directory. Directories will be created as needed. - /// - /// The archive to extract. - /// The folder to extract into. - /// Optional progress report callback. - /// Optional cancellation token. - public static void ExtractToDirectory( - this IArchive archive, - string destination, - Action? progressReport = null, - CancellationToken cancellationToken = default - ) - { - // Prepare for progress reporting - var totalBytes = archive.TotalUncompressSize; - var bytesRead = 0L; - - // Tracking for created directories. - var seenDirectories = new HashSet(); - - // Extract - var entries = archive.ExtractAllEntries(); - while (entries.MoveToNextEntry()) + private void WriteToDirectoryInternal( + string destinationDirectory, + ExtractionOptions? options, + IProgress? progress + ) { - cancellationToken.ThrowIfCancellationRequested(); + options ??= new ExtractionOptions(); + var fullDestinationDirectoryPath = DirectoryManagement.GetFullDestinationDirectoryPath( + destinationDirectory + ); - var entry = entries.Entry; - if (entry.IsDirectory) + var totalBytes = archive.TotalUncompressedSize; + var bytesRead = 0L; + + foreach (var entry in archive.Entries) { - continue; + if (entry.IsDirectory) + { + entry.WriteEntryToDirectoryCore(fullDestinationDirectoryPath, options, null); + continue; + } + + entry.WriteEntryToDirectoryCore( + fullDestinationDirectoryPath, + options, + path => entry.WriteToFile(path, options) + ); + + bytesRead += entry.Size; + progress?.Report( + new ProgressReport(entry.Key ?? string.Empty, bytesRead, totalBytes) + ); } - - // Create each directory - var path = Path.Combine(destination, entry.Key); - if (Path.GetDirectoryName(path) is { } directory && seenDirectories.Add(path)) - { - Directory.CreateDirectory(directory); - } - - // Write file - using var fs = File.OpenWrite(path); - entries.WriteEntryTo(fs); - - // Update progress - bytesRead += entry.Size; - progressReport?.Invoke(bytesRead / (double)totalBytes); } } } diff --git a/src/SharpCompress/Archives/IArchiveExtractionListener.cs b/src/SharpCompress/Archives/IArchiveExtractionListener.cs deleted file mode 100644 index 7bc2ef34..00000000 --- a/src/SharpCompress/Archives/IArchiveExtractionListener.cs +++ /dev/null @@ -1,10 +0,0 @@ -using SharpCompress.Common; - -namespace SharpCompress.Archives; - -internal interface IArchiveExtractionListener : IExtractionListener -{ - void EnsureEntriesLoaded(); - void FireEntryExtractionBegin(IArchiveEntry entry); - void FireEntryExtractionEnd(IArchiveEntry entry); -} diff --git a/src/SharpCompress/Archives/IArchiveFactory.cs b/src/SharpCompress/Archives/IArchiveFactory.cs index 370e5c9f..03185424 100644 --- a/src/SharpCompress/Archives/IArchiveFactory.cs +++ b/src/SharpCompress/Archives/IArchiveFactory.cs @@ -1,4 +1,6 @@ using System.IO; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Factories; using SharpCompress.Readers; @@ -24,12 +26,38 @@ public interface IArchiveFactory : IFactory /// /// An open, readable and seekable stream. /// reading options. - IArchive Open(Stream stream, ReaderOptions? readerOptions = null); + IArchive OpenArchive(Stream stream, ReaderOptions? readerOptions = null); + + /// + /// Opens an Archive for random access asynchronously. + /// + /// An open, readable and seekable stream. + /// reading options. + /// Cancellation token. + /// A containing the opened async archive. + ValueTask OpenAsyncArchive( + Stream stream, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ); /// /// Constructor with a FileInfo object to an existing file. /// /// the file to open. /// reading options. - IArchive Open(FileInfo fileInfo, ReaderOptions? readerOptions = null); + IArchive OpenArchive(FileInfo fileInfo, ReaderOptions? readerOptions = null); + + /// + /// Opens an Archive from a FileInfo object asynchronously. + /// + /// the file to open. + /// reading options. + /// Cancellation token. + /// A containing the opened async archive. + ValueTask OpenAsyncArchive( + FileInfo fileInfo, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ); } diff --git a/src/SharpCompress/Archives/IArchiveOpenable.cs b/src/SharpCompress/Archives/IArchiveOpenable.cs new file mode 100644 index 00000000..22c0ed34 --- /dev/null +++ b/src/SharpCompress/Archives/IArchiveOpenable.cs @@ -0,0 +1,41 @@ +#if NET8_0_OR_GREATER +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Readers; + +namespace SharpCompress.Archives; + +public interface IArchiveOpenable + where TSync : IArchive + where TASync : IAsyncArchive +{ + public static abstract TSync OpenArchive(string filePath, ReaderOptions? readerOptions = null); + + public static abstract TSync OpenArchive( + FileInfo fileInfo, + ReaderOptions? readerOptions = null + ); + + public static abstract TSync OpenArchive(Stream stream, ReaderOptions? readerOptions = null); + + public static abstract ValueTask OpenAsyncArchive( + string filePath, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ); + + public static abstract ValueTask OpenAsyncArchive( + Stream stream, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ); + + public static abstract ValueTask OpenAsyncArchive( + FileInfo fileInfo, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ); +} + +#endif diff --git a/src/SharpCompress/Archives/IAsyncArchive.cs b/src/SharpCompress/Archives/IAsyncArchive.cs new file mode 100644 index 00000000..2994e8b8 --- /dev/null +++ b/src/SharpCompress/Archives/IAsyncArchive.cs @@ -0,0 +1,48 @@ +using System; +using System.Collections.Generic; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Readers; + +namespace SharpCompress.Archives; + +public interface IAsyncArchive : IAsyncDisposable +{ + IAsyncEnumerable EntriesAsync { get; } + IAsyncEnumerable VolumesAsync { get; } + + ArchiveType Type { get; } + + /// + /// Use this method to extract all entries in an archive in order. + /// This is primarily for SOLID Rar Archives or 7Zip Archives as they need to be + /// extracted sequentially for the best performance. + /// + ValueTask ExtractAllEntriesAsync(); + + /// + /// Archive is SOLID (this means the Archive saved bytes by reusing information which helps for archives containing many small files). + /// Rar Archives can be SOLID while all 7Zip archives are considered SOLID. + /// + ValueTask IsSolidAsync(); + + /// + /// This checks to see if all the known entries have IsComplete = true + /// + ValueTask IsCompleteAsync(); + + /// + /// The total size of the files compressed in the archive. + /// + ValueTask TotalSizeAsync(); + + /// + /// The total size of the files as uncompressed in the archive. + /// + ValueTask TotalUncompressedSizeAsync(); + + /// + /// Returns whether the archive is encrypted. + /// + ValueTask IsEncryptedAsync(); +} diff --git a/src/SharpCompress/Archives/IAsyncArchiveExtensions.cs b/src/SharpCompress/Archives/IAsyncArchiveExtensions.cs new file mode 100644 index 00000000..e9fd9723 --- /dev/null +++ b/src/SharpCompress/Archives/IAsyncArchiveExtensions.cs @@ -0,0 +1,126 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Readers; + +namespace SharpCompress.Archives; + +public static class IAsyncArchiveExtensions +{ + extension(IAsyncArchive archive) + { + /// + /// Extract to specific directory asynchronously with progress reporting and cancellation support + /// + /// The folder to extract into. + /// Extraction options. + /// Optional progress reporter for tracking extraction progress. + /// Optional cancellation token. + public async ValueTask WriteToDirectoryAsync( + string destinationDirectory, + ExtractionOptions? options = null, + IProgress? progress = null, + CancellationToken cancellationToken = default + ) + { + if ( + await archive.IsSolidAsync().ConfigureAwait(false) + || archive.Type == ArchiveType.SevenZip + ) + { + var totalBytes = await archive.TotalUncompressedSizeAsync().ConfigureAwait(false); + var bytesRead = 0L; + await using var reader = await archive + .ExtractAllEntriesAsync() + .ConfigureAwait(false); + while (await reader.MoveToNextEntryAsync(cancellationToken).ConfigureAwait(false)) + { + cancellationToken.ThrowIfCancellationRequested(); + + await reader + .WriteEntryToDirectoryAsync( + destinationDirectory, + options, + cancellationToken + ) + .ConfigureAwait(false); + + if (reader.Entry.IsDirectory) + { + continue; + } + + bytesRead += reader.Entry.Size; + progress?.Report( + new ProgressReport(reader.Entry.Key ?? string.Empty, bytesRead, totalBytes) + ); + } + } + else + { + await archive + .WriteToDirectoryAsyncInternal( + destinationDirectory, + options, + progress, + cancellationToken + ) + .ConfigureAwait(false); + } + } + + private async ValueTask WriteToDirectoryAsyncInternal( + string destinationDirectory, + ExtractionOptions? options, + IProgress? progress, + CancellationToken cancellationToken + ) + { + options ??= new ExtractionOptions(); + var fullDestinationDirectoryPath = DirectoryManagement.GetFullDestinationDirectoryPath( + destinationDirectory + ); + + var totalBytes = await archive.TotalUncompressedSizeAsync().ConfigureAwait(false); + var bytesRead = 0L; + + await foreach ( + var entry in archive + .EntriesAsync.WithCancellation(cancellationToken) + .ConfigureAwait(false) + ) + { + cancellationToken.ThrowIfCancellationRequested(); + + if (entry.IsDirectory) + { + await entry + .WriteEntryToDirectoryAsyncCore( + fullDestinationDirectoryPath, + options, + null, + cancellationToken + ) + .ConfigureAwait(false); + continue; + } + + await entry + .WriteEntryToDirectoryAsyncCore( + fullDestinationDirectoryPath, + options, + async (path, ct) => + await entry.WriteToFileAsync(path, options, ct).ConfigureAwait(false), + cancellationToken + ) + .ConfigureAwait(false); + + bytesRead += entry.Size; + progress?.Report( + new ProgressReport(entry.Key ?? string.Empty, bytesRead, totalBytes) + ); + } + } + } +} diff --git a/src/SharpCompress/Archives/IMultiArchiveFactory.cs b/src/SharpCompress/Archives/IMultiArchiveFactory.cs index c26b649f..3f9e0ec1 100644 --- a/src/SharpCompress/Archives/IMultiArchiveFactory.cs +++ b/src/SharpCompress/Archives/IMultiArchiveFactory.cs @@ -1,5 +1,7 @@ using System.Collections.Generic; using System.IO; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Factories; using SharpCompress.Readers; @@ -25,12 +27,38 @@ public interface IMultiArchiveFactory : IFactory /// /// /// reading options. - IArchive Open(IReadOnlyList streams, ReaderOptions? readerOptions = null); + IArchive OpenArchive(IReadOnlyList streams, ReaderOptions? readerOptions = null); + + /// + /// Opens a multi-part archive from streams asynchronously. + /// + /// + /// reading options. + /// Cancellation token. + /// A containing the opened async archive. + ValueTask OpenAsyncArchive( + IReadOnlyList streams, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ); /// /// Constructor with IEnumerable Stream objects, multi and split support. /// /// /// reading options. - IArchive Open(IReadOnlyList fileInfos, ReaderOptions? readerOptions = null); + IArchive OpenArchive(IReadOnlyList fileInfos, ReaderOptions? readerOptions = null); + + /// + /// Opens a multi-part archive from files asynchronously. + /// + /// + /// reading options. + /// Cancellation token. + /// A containing the opened async archive. + ValueTask OpenAsyncArchive( + IReadOnlyList fileInfos, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ); } diff --git a/src/SharpCompress/Archives/IMultiArchiveOpenable.cs b/src/SharpCompress/Archives/IMultiArchiveOpenable.cs new file mode 100644 index 00000000..4c208ec1 --- /dev/null +++ b/src/SharpCompress/Archives/IMultiArchiveOpenable.cs @@ -0,0 +1,36 @@ +#if NET8_0_OR_GREATER +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Readers; + +namespace SharpCompress.Archives; + +public interface IMultiArchiveOpenable + where TSync : IArchive + where TASync : IAsyncArchive +{ + public static abstract TSync OpenArchive( + IReadOnlyList fileInfos, + ReaderOptions? readerOptions = null + ); + + public static abstract TSync OpenArchive( + IReadOnlyList streams, + ReaderOptions? readerOptions = null + ); + + public static abstract ValueTask OpenAsyncArchive( + IReadOnlyList streams, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ); + + public static abstract ValueTask OpenAsyncArchive( + IReadOnlyList fileInfos, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ); +} +#endif diff --git a/src/SharpCompress/Archives/IWritableArchive.cs b/src/SharpCompress/Archives/IWritableArchive.cs index 37b84aa0..0d17a937 100644 --- a/src/SharpCompress/Archives/IWritableArchive.cs +++ b/src/SharpCompress/Archives/IWritableArchive.cs @@ -1,13 +1,23 @@ using System; using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Options; using SharpCompress.Writers; namespace SharpCompress.Archives; -public interface IWritableArchive : IArchive +public interface IWritableArchiveCommon { - void RemoveEntry(IArchiveEntry entry); + /// + /// Use this to pause entry rebuilding when adding large collections of entries. Dispose when complete. A using statement is recommended. + /// + /// IDisposeable to resume entry rebuilding + IDisposable PauseEntryRebuilding(); +} +public interface IWritableArchive : IArchive, IWritableArchiveCommon +{ IArchiveEntry AddEntry( string key, Stream source, @@ -16,11 +26,61 @@ public interface IWritableArchive : IArchive DateTime? modified = null ); - void SaveTo(Stream stream, WriterOptions options); + IArchiveEntry AddDirectoryEntry(string key, DateTime? modified = null); /// - /// Use this to pause entry rebuilding when adding large collections of entries. Dispose when complete. A using statement is recommended. + /// Removes the specified entry from the archive. /// - /// IDisposeable to resume entry rebuilding - IDisposable PauseEntryRebuilding(); + void RemoveEntry(IArchiveEntry entry); +} + +public interface IWritableArchive : IWritableArchive + where TOptions : IWriterOptions +{ + /// + /// Saves the archive to the specified stream using the given writer options. + /// + void SaveTo(Stream stream, TOptions options); +} + +public interface IWritableAsyncArchive : IAsyncArchive, IWritableArchiveCommon +{ + /// + /// Asynchronously adds an entry to the archive with the specified key, source stream, and options. + /// + ValueTask AddEntryAsync( + string key, + Stream source, + bool closeStream, + long size = 0, + DateTime? modified = null, + CancellationToken cancellationToken = default + ); + + /// + /// Asynchronously adds a directory entry to the archive with the specified key and modification time. + /// + ValueTask AddDirectoryEntryAsync( + string key, + DateTime? modified = null, + CancellationToken cancellationToken = default + ); + + /// + /// Removes the specified entry from the archive. + /// + ValueTask RemoveEntryAsync(IArchiveEntry entry); +} + +public interface IWritableAsyncArchive : IWritableAsyncArchive + where TOptions : IWriterOptions +{ + /// + /// Asynchronously saves the archive to the specified stream using the given writer options. + /// + ValueTask SaveToAsync( + Stream stream, + TOptions options, + CancellationToken cancellationToken = default + ); } diff --git a/src/SharpCompress/Archives/IWritableArchiveExtensions.cs b/src/SharpCompress/Archives/IWritableArchiveExtensions.cs index 8f531d41..f9f35018 100644 --- a/src/SharpCompress/Archives/IWritableArchiveExtensions.cs +++ b/src/SharpCompress/Archives/IWritableArchiveExtensions.cs @@ -1,86 +1,82 @@ -using System; +using System; using System.IO; -using SharpCompress.Writers; +using SharpCompress.Common.Options; namespace SharpCompress.Archives; public static class IWritableArchiveExtensions { - public static void AddEntry( - this IWritableArchive writableArchive, - string entryPath, - string filePath - ) + extension(IWritableArchive writableArchive) { - var fileInfo = new FileInfo(filePath); - if (!fileInfo.Exists) + public void AddAllFromDirectory( + string directoryPath, + string searchPattern = "*.*", + SearchOption searchOption = SearchOption.AllDirectories + ) { - throw new FileNotFoundException("Could not AddEntry: " + filePath); + using (writableArchive.PauseEntryRebuilding()) + { + foreach ( + var filePath in Directory.EnumerateFiles( + directoryPath, + searchPattern, + searchOption + ) + ) + { + var fileInfo = new FileInfo(filePath); + writableArchive.AddEntry( + filePath.Substring(directoryPath.Length), + fileInfo.OpenRead(), + true, + fileInfo.Length, + fileInfo.LastWriteTime + ); + } + } + } + + public IArchiveEntry AddEntry(string key, string file) => + writableArchive.AddEntry(key, new FileInfo(file)); + + public IArchiveEntry AddEntry( + string key, + Stream source, + long size = 0, + DateTime? modified = null + ) => writableArchive.AddEntry(key, source, false, size, modified); + + public IArchiveEntry AddEntry(string key, FileInfo fileInfo) + { + if (!fileInfo.Exists) + { + throw new ArgumentException("FileInfo does not exist."); + } + return writableArchive.AddEntry( + key, + fileInfo.OpenRead(), + true, + fileInfo.Length, + fileInfo.LastWriteTime + ); } - writableArchive.AddEntry( - entryPath, - new FileInfo(filePath).OpenRead(), - true, - fileInfo.Length, - fileInfo.LastWriteTime - ); } - public static void SaveTo( - this IWritableArchive writableArchive, + public static void SaveTo( + this IWritableArchive writableArchive, string filePath, - WriterOptions options - ) => writableArchive.SaveTo(new FileInfo(filePath), options); - - public static void SaveTo( - this IWritableArchive writableArchive, - FileInfo fileInfo, - WriterOptions options + TOptions options ) + where TOptions : IWriterOptions => writableArchive.SaveTo(new FileInfo(filePath), options); + + public static void SaveTo( + this IWritableArchive writableArchive, + FileInfo fileInfo, + TOptions options + ) + where TOptions : IWriterOptions { using var stream = fileInfo.Open(FileMode.Create, FileAccess.Write); writableArchive.SaveTo(stream, options); } - - public static void AddAllFromDirectory( - this IWritableArchive writableArchive, - string filePath, - string searchPattern = "*.*", - SearchOption searchOption = SearchOption.AllDirectories - ) - { - using (writableArchive.PauseEntryRebuilding()) - { - foreach (var path in Directory.EnumerateFiles(filePath, searchPattern, searchOption)) - { - var fileInfo = new FileInfo(path); - writableArchive.AddEntry( - path.Substring(filePath.Length), - fileInfo.OpenRead(), - true, - fileInfo.Length, - fileInfo.LastWriteTime - ); - } - } - } - - public static IArchiveEntry AddEntry( - this IWritableArchive writableArchive, - string key, - FileInfo fileInfo - ) - { - if (!fileInfo.Exists) - { - throw new ArgumentException("FileInfo does not exist."); - } - return writableArchive.AddEntry( - key, - fileInfo.OpenRead(), - true, - fileInfo.Length, - fileInfo.LastWriteTime - ); - } } diff --git a/src/SharpCompress/Archives/IWriteableArchiveFactory.cs b/src/SharpCompress/Archives/IWritableArchiveFactory.cs similarity index 64% rename from src/SharpCompress/Archives/IWriteableArchiveFactory.cs rename to src/SharpCompress/Archives/IWritableArchiveFactory.cs index 4fae9f55..7c7ace2c 100644 --- a/src/SharpCompress/Archives/IWriteableArchiveFactory.cs +++ b/src/SharpCompress/Archives/IWritableArchiveFactory.cs @@ -1,7 +1,9 @@ +using SharpCompress.Common.Options; + namespace SharpCompress.Archives; /// -/// Decorator for used to declare an archive format as able to create writeable archives +/// Decorator for used to declare an archive format as able to create writable archives. /// /// /// Implemented by:
@@ -10,11 +12,13 @@ namespace SharpCompress.Archives; /// /// /// -public interface IWriteableArchiveFactory : Factories.IFactory +/// +public interface IWritableArchiveFactory : Factories.IFactory + where TOptions : IWriterOptions { /// /// Creates a new, empty archive, ready to be written. /// /// - IWritableArchive CreateWriteableArchive(); + IWritableArchive CreateArchive(); } diff --git a/src/SharpCompress/Archives/IWritableArchiveOpenable.cs b/src/SharpCompress/Archives/IWritableArchiveOpenable.cs new file mode 100644 index 00000000..2e5c6855 --- /dev/null +++ b/src/SharpCompress/Archives/IWritableArchiveOpenable.cs @@ -0,0 +1,14 @@ +using System.Threading.Tasks; +using SharpCompress.Common.Options; + +#if NET8_0_OR_GREATER +namespace SharpCompress.Archives; + +public interface IWritableArchiveOpenable + : IArchiveOpenable, IWritableAsyncArchive> + where TOptions : IWriterOptions +{ + public static abstract IWritableArchive CreateArchive(); + public static abstract ValueTask> CreateAsyncArchive(); +} +#endif diff --git a/src/SharpCompress/Archives/IWritableAsyncArchiveExtensions.cs b/src/SharpCompress/Archives/IWritableAsyncArchiveExtensions.cs new file mode 100644 index 00000000..dcfee164 --- /dev/null +++ b/src/SharpCompress/Archives/IWritableAsyncArchiveExtensions.cs @@ -0,0 +1,89 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Options; + +namespace SharpCompress.Archives; + +public static class IWritableAsyncArchiveExtensions +{ + extension(IWritableAsyncArchive writableArchive) + { + public async ValueTask AddAllFromDirectoryAsync( + string directoryPath, + string searchPattern = "*.*", + SearchOption searchOption = SearchOption.AllDirectories + ) + { + using (writableArchive.PauseEntryRebuilding()) + { + foreach ( + var filePath in Directory.EnumerateFiles( + directoryPath, + searchPattern, + searchOption + ) + ) + { + var fileInfo = new FileInfo(filePath); + await writableArchive + .AddEntryAsync( + filePath.Substring(directoryPath.Length), + fileInfo.OpenRead(), + true, + fileInfo.Length, + fileInfo.LastWriteTime + ) + .ConfigureAwait(false); + } + } + } + + public ValueTask AddEntryAsync(string key, string file) => + writableArchive.AddEntryAsync(key, new FileInfo(file)); + + public ValueTask AddEntryAsync( + string key, + Stream source, + long size = 0, + DateTime? modified = null + ) => writableArchive.AddEntryAsync(key, source, false, size, modified); + + public ValueTask AddEntryAsync(string key, FileInfo fileInfo) + { + if (!fileInfo.Exists) + { + throw new ArgumentException("FileInfo does not exist."); + } + return writableArchive.AddEntryAsync( + key, + fileInfo.OpenRead(), + true, + fileInfo.Length, + fileInfo.LastWriteTime + ); + } + } + + public static ValueTask SaveToAsync( + this IWritableAsyncArchive writableArchive, + string filePath, + TOptions options, + CancellationToken cancellationToken = default + ) + where TOptions : IWriterOptions => + writableArchive.SaveToAsync(new FileInfo(filePath), options, cancellationToken); + + public static async ValueTask SaveToAsync( + this IWritableAsyncArchive writableArchive, + FileInfo fileInfo, + TOptions options, + CancellationToken cancellationToken = default + ) + where TOptions : IWriterOptions + { + using var stream = fileInfo.Open(FileMode.Create, FileAccess.Write); + await writableArchive.SaveToAsync(stream, options, cancellationToken).ConfigureAwait(false); + } +} diff --git a/src/SharpCompress/Archives/Rar/FileInfoRarArchiveVolume.cs b/src/SharpCompress/Archives/Rar/FileInfoRarArchiveVolume.cs index 3e576db3..54c99a93 100644 --- a/src/SharpCompress/Archives/Rar/FileInfoRarArchiveVolume.cs +++ b/src/SharpCompress/Archives/Rar/FileInfoRarArchiveVolume.cs @@ -1,4 +1,5 @@ using System.Collections.Generic; +using System.Collections.ObjectModel; using System.IO; using System.Linq; using SharpCompress.Common.Rar; @@ -13,20 +14,18 @@ namespace SharpCompress.Archives.Rar; /// internal class FileInfoRarArchiveVolume : RarVolume { - internal FileInfoRarArchiveVolume(FileInfo fileInfo, ReaderOptions options, int index = 0) - : base(StreamingMode.Seekable, fileInfo.OpenRead(), FixOptions(options), index) + internal FileInfoRarArchiveVolume(FileInfo fileInfo, ReaderOptions options, int index) + : base( + StreamingMode.Seekable, + fileInfo.OpenRead(), + options.WithLeaveStreamOpen(false), + index + ) { FileInfo = fileInfo; FileParts = GetVolumeFileParts().ToArray().ToReadOnly(); } - private static ReaderOptions FixOptions(ReaderOptions options) - { - //make sure we're closing streams with fileinfo - options.LeaveStreamOpen = false; - return options; - } - internal ReadOnlyCollection FileParts { get; } internal FileInfo FileInfo { get; } @@ -35,4 +34,7 @@ internal class FileInfoRarArchiveVolume : RarVolume new FileInfoRarFilePart(this, ReaderOptions.Password, markHeader, fileHeader, FileInfo); internal override IEnumerable ReadFileParts() => FileParts; + + internal override IAsyncEnumerable ReadFilePartsAsync() => + FileParts.ToAsyncEnumerable(); } diff --git a/src/SharpCompress/Archives/Rar/RarArchive.Async.cs b/src/SharpCompress/Archives/Rar/RarArchive.Async.cs new file mode 100644 index 00000000..ce74a3c3 --- /dev/null +++ b/src/SharpCompress/Archives/Rar/RarArchive.Async.cs @@ -0,0 +1,56 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Archives.Rar; +using SharpCompress.Common; +using SharpCompress.Common.Rar; +using SharpCompress.IO; +using SharpCompress.Readers; +using SharpCompress.Readers.Rar; + +namespace SharpCompress.Archives.Rar; + +public partial class RarArchive +{ + public override async ValueTask DisposeAsync() + { + if (!_disposed) + { + if (UnpackV1.IsValueCreated && UnpackV1.Value is IDisposable unpackV1) + { + unpackV1.Dispose(); + } + + _disposed = true; + await base.DisposeAsync().ConfigureAwait(false); + } + } + + protected override async ValueTask CreateReaderForSolidExtractionAsync() + { + if (await this.IsMultipartVolumeAsync().ConfigureAwait(false)) + { + var streams = await VolumesAsync + .Select(volume => + { + volume.Stream.Position = 0; + return volume.Stream; + }) + .ToListAsync() + .ConfigureAwait(false); + return (RarReader)RarReader.OpenReader(streams, ReaderOptions); + } + + var stream = (await VolumesAsync.FirstAsync().ConfigureAwait(false)).Stream; + stream.Position = 0; + return (RarReader)RarReader.OpenReader(stream, ReaderOptions); + } + + public override async ValueTask IsSolidAsync() => + await (await VolumesAsync.CastAsync().FirstAsync().ConfigureAwait(false)) + .IsSolidArchiveAsync() + .ConfigureAwait(false); +} diff --git a/src/SharpCompress/Archives/Rar/RarArchive.Extensions.cs b/src/SharpCompress/Archives/Rar/RarArchive.Extensions.cs index bf6f0cd4..aaca25d1 100644 --- a/src/SharpCompress/Archives/Rar/RarArchive.Extensions.cs +++ b/src/SharpCompress/Archives/Rar/RarArchive.Extensions.cs @@ -1,18 +1,40 @@ -using System.Linq; +using System.Linq; +using System.Threading.Tasks; +using SharpCompress.Common.Rar; namespace SharpCompress.Archives.Rar; public static class RarArchiveExtensions { - /// - /// RarArchive is the first volume of a multi-part archive. If MultipartVolume is true and IsFirstVolume is false then the first volume file must be missing. - /// - public static bool IsFirstVolume(this RarArchive archive) => - archive.Volumes.First().IsFirstVolume; + extension(IRarArchive archive) + { + /// + /// RarArchive is the first volume of a multi-part archive. If MultipartVolume is true and IsFirstVolume is false then the first volume file must be missing. + /// + public bool IsFirstVolume() => archive.Volumes.Cast().First().IsFirstVolume; - /// - /// RarArchive is part of a multi-part archive. - /// - public static bool IsMultipartVolume(this RarArchive archive) => - archive.Volumes.First().IsMultiVolume; + /// + /// RarArchive is part of a multi-part archive. + /// + public bool IsMultipartVolume() => archive.Volumes.Cast().First().IsMultiVolume; + } + + extension(IRarAsyncArchive archive) + { + /// + /// RarArchive is the first volume of a multi-part archive. If MultipartVolume is true and IsFirstVolume is false then the first volume file must be missing. + /// + public async ValueTask IsFirstVolumeAsync() => + ( + await archive.VolumesAsync.CastAsync().FirstAsync().ConfigureAwait(false) + ).IsFirstVolume; + + /// + /// RarArchive is part of a multi-part archive. + /// + public async ValueTask IsMultipartVolumeAsync() => + ( + await archive.VolumesAsync.CastAsync().FirstAsync().ConfigureAwait(false) + ).IsMultiVolume; + } } diff --git a/src/SharpCompress/Archives/Rar/RarArchive.Factory.cs b/src/SharpCompress/Archives/Rar/RarArchive.Factory.cs new file mode 100644 index 00000000..a8a0448a --- /dev/null +++ b/src/SharpCompress/Archives/Rar/RarArchive.Factory.cs @@ -0,0 +1,184 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Rar; +using SharpCompress.Common.Rar.Headers; +using SharpCompress.Compressors.Rar; +using SharpCompress.IO; +using SharpCompress.Readers; +using SharpCompress.Readers.Rar; + +namespace SharpCompress.Archives.Rar; + +public partial class RarArchive +#if NET8_0_OR_GREATER + : IArchiveOpenable, + IMultiArchiveOpenable +#endif +{ + public static ValueTask OpenAsyncArchive( + string filePath, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + filePath.NotNullOrEmpty(nameof(filePath)); + return new((IRarAsyncArchive)OpenArchive(new FileInfo(filePath), readerOptions)); + } + + public static IRarArchive OpenArchive(string filePath, ReaderOptions? readerOptions = null) + { + filePath.NotNullOrEmpty(nameof(filePath)); + var fileInfo = new FileInfo(filePath); + return new RarArchive( + new SourceStream( + fileInfo, + i => RarArchiveVolumeFactory.GetFilePart(i, fileInfo), + readerOptions ?? ReaderOptions.ForFilePath + ) + ); + } + + public static IRarArchive OpenArchive(FileInfo fileInfo, ReaderOptions? readerOptions = null) + { + fileInfo.NotNull(nameof(fileInfo)); + return new RarArchive( + new SourceStream( + fileInfo, + i => RarArchiveVolumeFactory.GetFilePart(i, fileInfo), + readerOptions ?? ReaderOptions.ForFilePath + ) + ); + } + + public static IRarArchive OpenArchive(Stream stream, ReaderOptions? readerOptions = null) + { + stream.RequireReadable(); + stream.RequireSeekable(); + + return new RarArchive( + new SourceStream(stream, _ => null, readerOptions ?? ReaderOptions.ForExternalStream) + ); + } + + public static IRarArchive OpenArchive( + IReadOnlyList fileInfos, + ReaderOptions? readerOptions = null + ) + { + fileInfos.NotNull(nameof(fileInfos)); + var files = fileInfos; + return new RarArchive( + new SourceStream( + files[0], + i => i < files.Count ? files[i] : null, + readerOptions ?? ReaderOptions.ForFilePath + ) + ); + } + + public static IRarArchive OpenArchive( + IReadOnlyList streams, + ReaderOptions? readerOptions = null + ) + { + var strms = streams.RequireReadable().RequireSeekable().ToList(); + return new RarArchive( + new SourceStream( + strms[0], + i => i < strms.Count ? strms[i] : null, + readerOptions ?? ReaderOptions.ForExternalStream + ) + ); + } + + public static ValueTask OpenAsyncArchive( + Stream stream, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IRarAsyncArchive)OpenArchive(stream, readerOptions)); + } + + public static ValueTask OpenAsyncArchive( + FileInfo fileInfo, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IRarAsyncArchive)OpenArchive(fileInfo, readerOptions)); + } + + public static ValueTask OpenAsyncArchive( + IReadOnlyList streams, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IRarAsyncArchive)OpenArchive(streams, readerOptions)); + } + + public static ValueTask OpenAsyncArchive( + IReadOnlyList fileInfos, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IRarAsyncArchive)OpenArchive(fileInfos, readerOptions)); + } + + public static bool IsRarFile(string filePath) => IsRarFile(new FileInfo(filePath)); + + public static bool IsRarFile(FileInfo fileInfo) + { + if (!fileInfo.Exists) + { + return false; + } + using Stream stream = fileInfo.OpenRead(); + return IsRarFile(stream); + } + + public static bool IsRarFile(Stream stream, ReaderOptions? options = null) + { + try + { + MarkHeader.Read(stream, true, options?.LookForHeader ?? false); + return true; + } + catch + { + return false; + } + } + + public static async ValueTask IsRarFileAsync( + Stream stream, + ReaderOptions? options = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + try + { + await MarkHeader + .ReadAsync(stream, true, options?.LookForHeader ?? false, cancellationToken) + .ConfigureAwait(false); + return true; + } + catch + { + return false; + } + } +} diff --git a/src/SharpCompress/Archives/Rar/RarArchive.cs b/src/SharpCompress/Archives/Rar/RarArchive.cs index 29d733f6..14d8aaa3 100644 --- a/src/SharpCompress/Archives/Rar/RarArchive.cs +++ b/src/SharpCompress/Archives/Rar/RarArchive.cs @@ -1,6 +1,9 @@ +using System; using System.Collections.Generic; using System.IO; using System.Linq; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common; using SharpCompress.Common.Rar; using SharpCompress.Common.Rar.Headers; @@ -11,167 +14,98 @@ using SharpCompress.Readers.Rar; namespace SharpCompress.Archives.Rar; -public class RarArchive : AbstractArchive +public interface IRarArchiveCommon { - internal Lazy UnpackV2017 { get; } = - new Lazy(() => new Compressors.Rar.UnpackV2017.Unpack()); - internal Lazy UnpackV1 { get; } = - new Lazy(() => new Compressors.Rar.UnpackV1.Unpack()); + int MinVersion { get; } + int MaxVersion { get; } +} - /// - /// Constructor with a SourceStream able to handle FileInfo and Streams. - /// - /// - /// - internal RarArchive(SourceStream srcStream) - : base(ArchiveType.Rar, srcStream) { } +public interface IRarArchive : IArchive, IRarArchiveCommon { } + +public interface IRarAsyncArchive : IAsyncArchive, IRarArchiveCommon { } + +public partial class RarArchive + : AbstractArchive, + IRarArchive, + IRarAsyncArchive +{ + private bool _disposed; + internal Lazy UnpackV2017 { get; } = + new(() => new Compressors.Rar.UnpackV2017.Unpack()); + internal Lazy UnpackV1 { get; } = new(() => new Compressors.Rar.UnpackV1.Unpack()); + + private RarArchive(SourceStream sourceStream) + : base(ArchiveType.Rar, sourceStream) { } + + public override void Dispose() + { + if (!_disposed) + { + if (UnpackV1.IsValueCreated && UnpackV1.Value is IDisposable unpackV1) + { + unpackV1.Dispose(); + } + if (UnpackV2017.IsValueCreated && UnpackV2017.Value is IDisposable unpackV2017) + { + unpackV2017.Dispose(); + } + + _disposed = true; + base.Dispose(); + } + } protected override IEnumerable LoadEntries(IEnumerable volumes) => RarArchiveEntryFactory.GetEntries(this, volumes, ReaderOptions); - protected override IEnumerable LoadVolumes(SourceStream srcStream) - { - SrcStream.LoadAllParts(); //request all streams - var streams = SrcStream.Streams.ToArray(); - var idx = 0; - if (streams.Length > 1 && IsRarFile(streams[1], ReaderOptions)) //test part 2 - true = multipart not split - { - SrcStream.IsVolumes = true; - streams[1].Position = 0; - SrcStream.Position = 0; + // Simple async property - kept in original file + protected override IAsyncEnumerable LoadEntriesAsync( + IAsyncEnumerable volumes + ) => RarArchiveEntryFactory.GetEntriesAsync(this, volumes, ReaderOptions); - return srcStream.Streams.Select( - a => new StreamRarArchiveVolume(a, ReaderOptions, idx++) - ); - } - else //split mode or single file + protected override IEnumerable LoadVolumes(SourceStream sourceStream) + { + sourceStream.LoadAllParts(); + var streams = sourceStream.Streams.ToArray(); + var i = 0; + if (streams.Length > 1 && IsRarFile(streams[1], ReaderOptions)) { - return new StreamRarArchiveVolume(SrcStream, ReaderOptions, idx++).AsEnumerable(); + sourceStream.IsVolumes = true; + streams[1].Position = 0; + sourceStream.Position = 0; + + return sourceStream.Streams.Select(a => new StreamRarArchiveVolume( + a, + ReaderOptions, + i++ + )); } + + return new StreamRarArchiveVolume(sourceStream, ReaderOptions, i++).AsEnumerable(); } protected override IReader CreateReaderForSolidExtraction() { + if (this.IsMultipartVolume()) + { + var streams = Volumes.Select(volume => + { + volume.Stream.Position = 0; + return volume.Stream; + }); + return (RarReader)RarReader.OpenReader(streams, ReaderOptions); + } + var stream = Volumes.First().Stream; stream.Position = 0; - return RarReader.Open(stream, ReaderOptions); + return (RarReader)RarReader.OpenReader(stream, ReaderOptions); } public override bool IsSolid => Volumes.First().IsSolidArchive; + public override bool IsEncrypted => Entries.First(x => !x.IsDirectory).IsEncrypted; + public virtual int MinVersion => Volumes.First().MinVersion; + public virtual int MaxVersion => Volumes.First().MaxVersion; - - #region Creation - /// - /// Constructor with a FileInfo object to an existing file. - /// - /// - /// - public static RarArchive Open(string filePath, ReaderOptions? options = null) - { - filePath.CheckNotNullOrEmpty(nameof(filePath)); - var fileInfo = new FileInfo(filePath); - return new RarArchive( - new SourceStream( - fileInfo, - i => RarArchiveVolumeFactory.GetFilePart(i, fileInfo), - options ?? new ReaderOptions() - ) - ); - } - - /// - /// Constructor with a FileInfo object to an existing file. - /// - /// - /// - public static RarArchive Open(FileInfo fileInfo, ReaderOptions? options = null) - { - fileInfo.CheckNotNull(nameof(fileInfo)); - return new RarArchive( - new SourceStream( - fileInfo, - i => RarArchiveVolumeFactory.GetFilePart(i, fileInfo), - options ?? new ReaderOptions() - ) - ); - } - - /// - /// Takes a seekable Stream as a source - /// - /// - /// - public static RarArchive Open(Stream stream, ReaderOptions? options = null) - { - stream.CheckNotNull(nameof(stream)); - return new RarArchive(new SourceStream(stream, i => null, options ?? new ReaderOptions())); - } - - /// - /// Constructor with all file parts passed in - /// - /// - /// - public static RarArchive Open( - IEnumerable fileInfos, - ReaderOptions? readerOptions = null - ) - { - fileInfos.CheckNotNull(nameof(fileInfos)); - var files = fileInfos.ToArray(); - return new RarArchive( - new SourceStream( - files[0], - i => i < files.Length ? files[i] : null, - readerOptions ?? new ReaderOptions() - ) - ); - } - - /// - /// Constructor with all stream parts passed in - /// - /// - /// - public static RarArchive Open(IEnumerable streams, ReaderOptions? readerOptions = null) - { - streams.CheckNotNull(nameof(streams)); - var strms = streams.ToArray(); - return new RarArchive( - new SourceStream( - strms[0], - i => i < strms.Length ? strms[i] : null, - readerOptions ?? new ReaderOptions() - ) - ); - } - - public static bool IsRarFile(string filePath) => IsRarFile(new FileInfo(filePath)); - - public static bool IsRarFile(FileInfo fileInfo) - { - if (!fileInfo.Exists) - { - return false; - } - using Stream stream = fileInfo.OpenRead(); - return IsRarFile(stream); - } - - public static bool IsRarFile(Stream stream, ReaderOptions? options = null) - { - try - { - MarkHeader.Read(stream, true, false); - return true; - } - catch - { - return false; - } - } - - #endregion } diff --git a/src/SharpCompress/Archives/Rar/RarArchiveEntry.Async.cs b/src/SharpCompress/Archives/Rar/RarArchiveEntry.Async.cs new file mode 100644 index 00000000..471609aa --- /dev/null +++ b/src/SharpCompress/Archives/Rar/RarArchiveEntry.Async.cs @@ -0,0 +1,43 @@ +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Rar; +using SharpCompress.Common.Rar.Headers; +using SharpCompress.Compressors.Rar; +using SharpCompress.Readers; + +namespace SharpCompress.Archives.Rar; + +public partial class RarArchiveEntry +{ + public async ValueTask OpenEntryStreamAsync( + CancellationToken cancellationToken = default + ) + { + RarStream stream; + if (IsRarV3) + { + stream = new RarStream( + archive.UnpackV1.Value, + FileHeader, + await MultiVolumeReadOnlyAsyncStream + .Create(Parts.ToAsyncEnumerable().CastAsync()) + .ConfigureAwait(false) + ); + } + else + { + stream = new RarStream( + archive.UnpackV2017.Value, + FileHeader, + await MultiVolumeReadOnlyAsyncStream + .Create(Parts.ToAsyncEnumerable().CastAsync()) + .ConfigureAwait(false) + ); + } + + await stream.InitializeAsync(cancellationToken).ConfigureAwait(false); + return stream; + } +} diff --git a/src/SharpCompress/Archives/Rar/RarArchiveEntry.cs b/src/SharpCompress/Archives/Rar/RarArchiveEntry.cs index 5885210a..021381c3 100644 --- a/src/SharpCompress/Archives/Rar/RarArchiveEntry.cs +++ b/src/SharpCompress/Archives/Rar/RarArchiveEntry.cs @@ -1,6 +1,9 @@ +using System; using System.Collections.Generic; using System.IO; using System.Linq; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common; using SharpCompress.Common.Rar; using SharpCompress.Common.Rar.Headers; @@ -9,7 +12,7 @@ using SharpCompress.Readers; namespace SharpCompress.Archives.Rar; -public class RarArchiveEntry : RarEntry, IArchiveEntry +public partial class RarArchiveEntry : RarEntry, IArchiveEntry { private readonly ICollection parts; private readonly RarArchive archive; @@ -20,6 +23,7 @@ public class RarArchiveEntry : RarEntry, IArchiveEntry IEnumerable parts, ReaderOptions readerOptions ) + : base(readerOptions) { this.parts = parts.ToList(); this.archive = archive; @@ -40,7 +44,10 @@ public class RarArchiveEntry : RarEntry, IArchiveEntry get { CheckIncomplete(); - return parts.Select(fp => fp.FileHeader).Single(fh => !fh.IsSplitAfter).FileCrc; + return BitConverter.ToUInt32( + parts.Select(fp => fp.FileHeader).Single(fh => !fh.IsSplitAfter).FileCrc.NotNull(), + 0 + ); } } @@ -64,20 +71,23 @@ public class RarArchiveEntry : RarEntry, IArchiveEntry public Stream OpenEntryStream() { + var readStream = new MultiVolumeReadOnlyStream(Parts.Cast()); + RarStream stream; if (IsRarV3) { - return new RarStream( - archive.UnpackV1.Value, - FileHeader, - new MultiVolumeReadOnlyStream(Parts.Cast(), archive) - ); + stream = RarCrcStream.Create(archive.UnpackV1.Value, FileHeader, readStream); + } + else if (FileHeader.FileCrc?.Length > 5) + { + stream = RarBLAKE2spStream.Create(archive.UnpackV2017.Value, FileHeader, readStream); + } + else + { + stream = RarCrcStream.Create(archive.UnpackV2017.Value, FileHeader, readStream); } - return new RarStream( - archive.UnpackV2017.Value, - FileHeader, - new MultiVolumeReadOnlyStream(Parts.Cast(), archive) - ); + stream.Initialize(); + return stream; } public bool IsComplete diff --git a/src/SharpCompress/Archives/Rar/RarArchiveEntryFactory.cs b/src/SharpCompress/Archives/Rar/RarArchiveEntryFactory.cs index ec4ace7c..f4a000a1 100644 --- a/src/SharpCompress/Archives/Rar/RarArchiveEntryFactory.cs +++ b/src/SharpCompress/Archives/Rar/RarArchiveEntryFactory.cs @@ -1,4 +1,5 @@ using System.Collections.Generic; +using System.Threading.Tasks; using SharpCompress.Common.Rar; using SharpCompress.Readers; @@ -17,6 +18,19 @@ internal static class RarArchiveEntryFactory } } + private static async IAsyncEnumerable GetFilePartsAsync( + IAsyncEnumerable parts + ) + { + await foreach (var rarPart in parts.ConfigureAwait(false)) + { + await foreach (var fp in rarPart.ReadFilePartsAsync().ConfigureAwait(false)) + { + yield return fp; + } + } + } + private static IEnumerable> GetMatchedFileParts( IEnumerable parts ) @@ -38,6 +52,27 @@ internal static class RarArchiveEntryFactory } } + private static async IAsyncEnumerable> GetMatchedFilePartsAsync( + IAsyncEnumerable parts + ) + { + var groupedParts = new List(); + await foreach (var fp in GetFilePartsAsync(parts).ConfigureAwait(false)) + { + groupedParts.Add(fp); + + if (!fp.FileHeader.IsSplitAfter) + { + yield return groupedParts; + groupedParts = new List(); + } + } + if (groupedParts.Count > 0) + { + yield return groupedParts; + } + } + internal static IEnumerable GetEntries( RarArchive archive, IEnumerable rarParts, @@ -49,4 +84,16 @@ internal static class RarArchiveEntryFactory yield return new RarArchiveEntry(archive, groupedParts, readerOptions); } } + + internal static async IAsyncEnumerable GetEntriesAsync( + RarArchive archive, + IAsyncEnumerable rarParts, + ReaderOptions readerOptions + ) + { + await foreach (var groupedParts in GetMatchedFilePartsAsync(rarParts).ConfigureAwait(false)) + { + yield return new RarArchiveEntry(archive, groupedParts, readerOptions); + } + } } diff --git a/src/SharpCompress/Archives/Rar/RarArchiveVolumeFactory.cs b/src/SharpCompress/Archives/Rar/RarArchiveVolumeFactory.cs index 72ccf60f..1e464876 100644 --- a/src/SharpCompress/Archives/Rar/RarArchiveVolumeFactory.cs +++ b/src/SharpCompress/Archives/Rar/RarArchiveVolumeFactory.cs @@ -1,6 +1,7 @@ using System; using System.IO; using System.Text.RegularExpressions; +using SharpCompress.Common; namespace SharpCompress.Archives.Rar; @@ -11,23 +12,28 @@ internal static class RarArchiveVolumeFactory FileInfo? item = null; //new style rar - ..part1 | /part01 | part001 .... - Match m = Regex.Match(part1.Name, @"^(.*\.part)([0-9]+)(\.rar)$", RegexOptions.IgnoreCase); + var m = Regex.Match(part1.Name, @"^(.*\.part)([0-9]+)(\.rar)$", RegexOptions.IgnoreCase); if (m.Success) + { item = new FileInfo( Path.Combine( part1.DirectoryName!, String.Concat( m.Groups[1].Value, - (index + 1).ToString().PadLeft(m.Groups[2].Value.Length, '0'), + (index + 1) + .ToString(Constants.DefaultCultureInfo) + .PadLeft(m.Groups[2].Value.Length, '0'), m.Groups[3].Value ) ) ); + } else { //old style - ...rar, .r00, .r01 ... m = Regex.Match(part1.Name, @"^(.*\.)([r-z{])(ar|[0-9]+)$", RegexOptions.IgnoreCase); if (m.Success) + { item = new FileInfo( Path.Combine( part1.DirectoryName!, @@ -36,16 +42,29 @@ internal static class RarArchiveVolumeFactory index == 0 ? m.Groups[2].Value + m.Groups[3].Value : (char)(m.Groups[2].Value[0] + ((index - 1) / 100)) - + (index - 1).ToString("D4").Substring(2) + + (index - 1) + .ToString( + "D4", + global::SharpCompress + .Common + .Constants + .DefaultCultureInfo + ) + .Substring(2) ) ) ); + } else //split .001, .002 .... + { return ArchiveVolumeFactory.GetFilePart(index, part1); + } } if (item != null && item.Exists) + { return item; + } return null; //no more items } diff --git a/src/SharpCompress/Archives/Rar/SeekableFilePart.cs b/src/SharpCompress/Archives/Rar/SeekableFilePart.cs index 08aceb13..97822d90 100644 --- a/src/SharpCompress/Archives/Rar/SeekableFilePart.cs +++ b/src/SharpCompress/Archives/Rar/SeekableFilePart.cs @@ -6,8 +6,8 @@ namespace SharpCompress.Archives.Rar; internal class SeekableFilePart : RarFilePart { - private readonly Stream stream; - private readonly string? password; + private readonly Stream _stream; + private readonly string? _password; internal SeekableFilePart( MarkHeader mh, @@ -18,18 +18,27 @@ internal class SeekableFilePart : RarFilePart ) : base(mh, fh, index) { - this.stream = stream; - this.password = password; + _stream = stream; + _password = password; } internal override Stream GetCompressedStream() { - stream.Position = FileHeader.DataStartPosition; + _stream.Position = FileHeader.DataStartPosition; + if (FileHeader.R4Salt != null) { - return new RarCryptoWrapper(stream, password!, FileHeader.R4Salt); + var cryptKey = new CryptKey3(_password!); + return new RarCryptoWrapper(_stream, FileHeader.R4Salt, cryptKey); } - return stream; + + if (FileHeader.Rar5CryptoInfo != null) + { + var cryptKey = new CryptKey5(_password!, FileHeader.Rar5CryptoInfo); + return new RarCryptoWrapper(_stream, FileHeader.Rar5CryptoInfo.Salt, cryptKey); + } + + return _stream; } internal override string FilePartName => "Unknown Stream - File Entry: " + FileHeader.FileName; diff --git a/src/SharpCompress/Archives/Rar/StreamRarArchiveVolume.cs b/src/SharpCompress/Archives/Rar/StreamRarArchiveVolume.cs index 3eb5095c..b7ffc67e 100644 --- a/src/SharpCompress/Archives/Rar/StreamRarArchiveVolume.cs +++ b/src/SharpCompress/Archives/Rar/StreamRarArchiveVolume.cs @@ -9,11 +9,14 @@ namespace SharpCompress.Archives.Rar; internal class StreamRarArchiveVolume : RarVolume { - internal StreamRarArchiveVolume(Stream stream, ReaderOptions options, int index = 0) + internal StreamRarArchiveVolume(Stream stream, ReaderOptions options, int index) : base(StreamingMode.Seekable, stream, options, index) { } internal override IEnumerable ReadFileParts() => GetVolumeFileParts(); + internal override IAsyncEnumerable ReadFilePartsAsync() => + GetVolumeFilePartsAsync(); + internal override RarFilePart CreateFilePart(MarkHeader markHeader, FileHeader fileHeader) => new SeekableFilePart(markHeader, fileHeader, Index, Stream, ReaderOptions.Password); } diff --git a/src/SharpCompress/Archives/SevenZip/SevenZipArchive.Async.cs b/src/SharpCompress/Archives/SevenZip/SevenZipArchive.Async.cs new file mode 100644 index 00000000..ce00f20d --- /dev/null +++ b/src/SharpCompress/Archives/SevenZip/SevenZipArchive.Async.cs @@ -0,0 +1,80 @@ +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.SevenZip; +using SharpCompress.IO; +using SharpCompress.Readers; + +namespace SharpCompress.Archives.SevenZip; + +public partial class SevenZipArchive +{ + private async ValueTask LoadFactoryAsync( + Stream stream, + CancellationToken cancellationToken = default + ) + { + if (_database is null) + { + stream.Position = 0; + var reader = new ArchiveReader(); + await reader + .OpenAsync(stream, lookForHeader: ReaderOptions.LookForHeader, cancellationToken) + .ConfigureAwait(false); + _database = await reader + .ReadDatabaseAsync(new PasswordProvider(ReaderOptions.Password), cancellationToken) + .ConfigureAwait(false); + } + } + + protected override async IAsyncEnumerable LoadEntriesAsync( + IAsyncEnumerable volumes + ) + { + var stream = (await volumes.SingleAsync().ConfigureAwait(false)).Stream; + await LoadFactoryAsync(stream).ConfigureAwait(false); + if (_database is null) + { + yield break; + } + var entries = new SevenZipArchiveEntry[_database._files.Count]; + for (var i = 0; i < _database._files.Count; i++) + { + var file = _database._files[i]; + entries[i] = new SevenZipArchiveEntry( + this, + new SevenZipFilePart(stream, _database, i, file, ReaderOptions.ArchiveEncoding), + ReaderOptions + ); + } + foreach (var group in entries.Where(x => !x.IsDirectory).GroupBy(x => x.FilePart.Folder)) + { + var isSolid = false; + foreach (var entry in group) + { + entry.IsSolid = isSolid; + isSolid = true; + } + } + + foreach (var entry in entries) + { + yield return entry; + } + } + + protected override ValueTask CreateReaderForSolidExtractionAsync() => + new(new SevenZipReader(ReaderOptions, this)); + + public override async ValueTask IsSolidAsync() + { + var entries = await EntriesAsync + .Where(x => !x.IsDirectory) + .ToListAsync() + .ConfigureAwait(false); + return entries.GroupBy(x => x.FilePart.Folder).Any(folder => folder.Skip(1).Any()); + } +} diff --git a/src/SharpCompress/Archives/SevenZip/SevenZipArchive.Factory.cs b/src/SharpCompress/Archives/SevenZip/SevenZipArchive.Factory.cs new file mode 100644 index 00000000..8c177a0f --- /dev/null +++ b/src/SharpCompress/Archives/SevenZip/SevenZipArchive.Factory.cs @@ -0,0 +1,263 @@ +using System; +using System.Buffers; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.IO; +using SharpCompress.Readers; + +namespace SharpCompress.Archives.SevenZip; + +public partial class SevenZipArchive +#if NET8_0_OR_GREATER + : IArchiveOpenable, + IMultiArchiveOpenable +#endif +{ + public static ValueTask OpenAsyncArchive( + string filePath, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + filePath.NotNullOrEmpty(nameof(filePath)); + return new( + (IAsyncArchive)OpenArchive( + new FileInfo(filePath), + readerOptions ?? ReaderOptions.ForFilePath + ) + ); + } + + public static IArchive OpenArchive(string filePath, ReaderOptions? readerOptions = null) + { + filePath.NotNullOrEmpty(nameof(filePath)); + return OpenArchive(new FileInfo(filePath), readerOptions ?? ReaderOptions.ForFilePath); + } + + public static IArchive OpenArchive(FileInfo fileInfo, ReaderOptions? readerOptions = null) + { + fileInfo.NotNull(nameof(fileInfo)); + return new SevenZipArchive( + new SourceStream( + fileInfo, + i => ArchiveVolumeFactory.GetFilePart(i, fileInfo), + readerOptions ?? ReaderOptions.ForFilePath + ) + ); + } + + public static IArchive OpenArchive( + IReadOnlyList fileInfos, + ReaderOptions? readerOptions = null + ) + { + fileInfos.NotNull(nameof(fileInfos)); + var files = fileInfos; + return new SevenZipArchive( + new SourceStream( + files[0], + i => i < files.Count ? files[i] : null, + readerOptions ?? ReaderOptions.ForFilePath + ) + ); + } + + public static IArchive OpenArchive( + IReadOnlyList streams, + ReaderOptions? readerOptions = null + ) + { + var strms = streams.RequireReadable().RequireSeekable().ToList(); + return new SevenZipArchive( + new SourceStream( + strms[0], + i => i < strms.Count ? strms[i] : null, + readerOptions ?? ReaderOptions.ForExternalStream + ) + ); + } + + public static IArchive OpenArchive(Stream stream, ReaderOptions? readerOptions = null) + { + stream.RequireReadable(); + stream.RequireSeekable(); + + return new SevenZipArchive( + new SourceStream(stream, _ => null, readerOptions ?? ReaderOptions.ForExternalStream) + ); + } + + public static ValueTask OpenAsyncArchive( + Stream stream, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncArchive)OpenArchive(stream, readerOptions)); + } + + public static ValueTask OpenAsyncArchive( + FileInfo fileInfo, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncArchive)OpenArchive(fileInfo, readerOptions)); + } + + public static ValueTask OpenAsyncArchive( + IReadOnlyList streams, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncArchive)OpenArchive(streams, readerOptions)); + } + + public static ValueTask OpenAsyncArchive( + IReadOnlyList fileInfos, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncArchive)OpenArchive(fileInfos, readerOptions)); + } + + public static bool IsSevenZipFile(string filePath) => IsSevenZipFile(new FileInfo(filePath)); + + public static bool IsSevenZipFile(FileInfo fileInfo) + { + if (!fileInfo.Exists) + { + return false; + } + using Stream stream = fileInfo.OpenRead(); + return IsSevenZipFile(stream); + } + + public static bool IsSevenZipFile(Stream stream) => + IsSevenZipFile(stream, ReaderOptions.ForExternalStream); + + public static bool IsSevenZipFile(Stream stream, ReaderOptions? readerOptions) + { + try + { + return SignatureMatch(stream, readerOptions?.LookForHeader ?? false); + } + catch + { + return false; + } + } + + public static async ValueTask IsSevenZipFileAsync( + Stream stream, + CancellationToken cancellationToken = default + ) => + await IsSevenZipFileAsync(stream, ReaderOptions.ForExternalStream, cancellationToken) + .ConfigureAwait(false); + + public static async ValueTask IsSevenZipFileAsync( + Stream stream, + ReaderOptions? readerOptions, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + try + { + return await SignatureMatchAsync( + stream, + readerOptions?.LookForHeader ?? false, + cancellationToken + ) + .ConfigureAwait(false); + } + catch + { + return false; + } + } + + private static ReadOnlySpan Signature => [(byte)'7', (byte)'z', 0xBC, 0xAF, 0x27, 0x1C]; + + private static bool SignatureMatch(Stream stream, bool lookForHeader) + { + var buffer = ArrayPool.Shared.Rent(6); + try + { + var maxScanOffset = lookForHeader ? 0x80000 - 20 : 0; + for (var offset = 0; offset <= maxScanOffset; offset++) + { + stream.ReadExact(buffer, 0, 6); + if (buffer.AsSpan().Slice(0, 6).SequenceEqual(Signature)) + { + return true; + } + + if (!lookForHeader || !stream.CanSeek || stream.Length - stream.Position < 6) + { + return false; + } + + stream.Position -= 5; + } + + return false; + } + finally + { + ArrayPool.Shared.Return(buffer); + } + } + + private static async ValueTask SignatureMatchAsync( + Stream stream, + bool lookForHeader, + CancellationToken cancellationToken + ) + { + var buffer = ArrayPool.Shared.Rent(6); + try + { + var maxScanOffset = lookForHeader ? 0x80000 - 20 : 0; + for (var offset = 0; offset <= maxScanOffset; offset++) + { + if ( + !await stream + .ReadFullyAsync(buffer, 0, 6, cancellationToken) + .ConfigureAwait(false) + ) + { + return false; + } + + if (buffer.AsSpan().Slice(0, 6).SequenceEqual(Signature)) + { + return true; + } + + if (!lookForHeader || !stream.CanSeek || stream.Length - stream.Position < 6) + { + return false; + } + + stream.Position -= 5; + } + + return false; + } + finally + { + ArrayPool.Shared.Return(buffer); + } + } +} diff --git a/src/SharpCompress/Archives/SevenZip/SevenZipArchive.cs b/src/SharpCompress/Archives/SevenZip/SevenZipArchive.cs index 7577b43d..27822313 100644 --- a/src/SharpCompress/Archives/SevenZip/SevenZipArchive.cs +++ b/src/SharpCompress/Archives/SevenZip/SevenZipArchive.cs @@ -1,129 +1,32 @@ -#nullable disable - using System; using System.Collections.Generic; using System.IO; using System.Linq; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common; using SharpCompress.Common.SevenZip; -using SharpCompress.Compressors.LZMA.Utilites; +using SharpCompress.Compressors.LZMA.Utilities; using SharpCompress.IO; using SharpCompress.Readers; namespace SharpCompress.Archives.SevenZip; -public class SevenZipArchive : AbstractArchive +public partial class SevenZipArchive : AbstractArchive { - private ArchiveDatabase database; - - /// - /// Constructor expects a filepath to an existing file. - /// - /// - /// - public static SevenZipArchive Open(string filePath, ReaderOptions readerOptions = null) - { - filePath.CheckNotNullOrEmpty("filePath"); - return Open(new FileInfo(filePath), readerOptions ?? new ReaderOptions()); - } - - /// - /// Constructor with a FileInfo object to an existing file. - /// - /// - /// - public static SevenZipArchive Open(FileInfo fileInfo, ReaderOptions readerOptions = null) - { - fileInfo.CheckNotNull("fileInfo"); - return new SevenZipArchive( - new SourceStream( - fileInfo, - i => ArchiveVolumeFactory.GetFilePart(i, fileInfo), - readerOptions ?? new ReaderOptions() - ) - ); - } - - /// - /// Constructor with all file parts passed in - /// - /// - /// - public static SevenZipArchive Open( - IEnumerable fileInfos, - ReaderOptions readerOptions = null - ) - { - fileInfos.CheckNotNull(nameof(fileInfos)); - var files = fileInfos.ToArray(); - return new SevenZipArchive( - new SourceStream( - files[0], - i => i < files.Length ? files[i] : null, - readerOptions ?? new ReaderOptions() - ) - ); - } - - /// - /// Constructor with all stream parts passed in - /// - /// - /// - public static SevenZipArchive Open( - IEnumerable streams, - ReaderOptions readerOptions = null - ) - { - streams.CheckNotNull(nameof(streams)); - var strms = streams.ToArray(); - return new SevenZipArchive( - new SourceStream( - strms[0], - i => i < strms.Length ? strms[i] : null, - readerOptions ?? new ReaderOptions() - ) - ); - } - - /// - /// Takes a seekable Stream as a source - /// - /// - /// - public static SevenZipArchive Open(Stream stream, ReaderOptions readerOptions = null) - { - stream.CheckNotNull("stream"); - return new SevenZipArchive( - new SourceStream(stream, i => null, readerOptions ?? new ReaderOptions()) - ); - } + private ArchiveDatabase? _database; /// /// Constructor with a SourceStream able to handle FileInfo and Streams. /// - /// - /// - internal SevenZipArchive(SourceStream srcStream) - : base(ArchiveType.SevenZip, srcStream) { } + /// + private SevenZipArchive(SourceStream sourceStream) + : base(ArchiveType.SevenZip, sourceStream) { } - protected override IEnumerable LoadVolumes(SourceStream srcStream) + protected override IEnumerable LoadVolumes(SourceStream sourceStream) { - SrcStream.LoadAllParts(); //request all streams - var idx = 0; - return new SevenZipVolume(srcStream, ReaderOptions, idx++).AsEnumerable(); //simple single volume or split, multivolume not supported - } - - public static bool IsSevenZipFile(string filePath) => IsSevenZipFile(new FileInfo(filePath)); - - public static bool IsSevenZipFile(FileInfo fileInfo) - { - if (!fileInfo.Exists) - { - return false; - } - using Stream stream = fileInfo.OpenRead(); - return IsSevenZipFile(stream); + sourceStream.NotNull("SourceStream is null").LoadAllParts(); //request all streams + return new SevenZipVolume(sourceStream, ReaderOptions, 0).AsEnumerable(); //simple single volume or split, multivolume not supported } internal SevenZipArchive() @@ -133,133 +36,281 @@ public class SevenZipArchive : AbstractArchive volumes ) { - var stream = volumes.Single().Stream; - LoadFactory(stream); - var entries = new SevenZipArchiveEntry[database._files.Count]; - for (var i = 0; i < database._files.Count; i++) + foreach (var volume in volumes) { - var file = database._files[i]; - entries[i] = new SevenZipArchiveEntry( - this, - new SevenZipFilePart(stream, database, i, file, ReaderOptions.ArchiveEncoding) - ); - } - foreach (var group in entries.Where(x => !x.IsDirectory).GroupBy(x => x.FilePart.Folder)) - { - var isSolid = false; - foreach (var entry in group) + LoadFactory(volume.Stream); + if (_database is null) { - entry.IsSolid = isSolid; - isSolid = true; //mark others in this group as solid - same as rar behaviour. + yield break; + } + var entries = new SevenZipArchiveEntry[_database._files.Count]; + for (var i = 0; i < _database._files.Count; i++) + { + var file = _database._files[i]; + entries[i] = new SevenZipArchiveEntry( + this, + new SevenZipFilePart( + volume.Stream, + _database, + i, + file, + ReaderOptions.ArchiveEncoding + ), + ReaderOptions + ); + } + foreach ( + var group in entries.Where(x => !x.IsDirectory).GroupBy(x => x.FilePart.Folder) + ) + { + var isSolid = false; + foreach (var entry in group) + { + entry.IsSolid = isSolid; + isSolid = true; + } + } + + foreach (var entry in entries) + { + yield return entry; } } - - return entries; } private void LoadFactory(Stream stream) { - if (database is null) + if (_database is null) { stream.Position = 0; var reader = new ArchiveReader(); - reader.Open(stream); - database = reader.ReadDatabase(new PasswordProvider(ReaderOptions.Password)); + reader.Open(stream, lookForHeader: ReaderOptions.LookForHeader); + _database = reader.ReadDatabase(new PasswordProvider(ReaderOptions.Password)); } } - public static bool IsSevenZipFile(Stream stream) - { - try - { - return SignatureMatch(stream); - } - catch - { - return false; - } - } - - private static ReadOnlySpan SIGNATURE => - new byte[] { (byte)'7', (byte)'z', 0xBC, 0xAF, 0x27, 0x1C }; - - private static bool SignatureMatch(Stream stream) - { - var reader = new BinaryReader(stream); - ReadOnlySpan signatureBytes = reader.ReadBytes(6); - return signatureBytes.SequenceEqual(SIGNATURE); - } - protected override IReader CreateReaderForSolidExtraction() => new SevenZipReader(ReaderOptions, this); public override bool IsSolid => - Entries.Where(x => !x.IsDirectory).GroupBy(x => x.FilePart.Folder).Count() > 1; + Entries + .Where(x => !x.IsDirectory) + .GroupBy(x => x.FilePart.Folder) + .Any(folder => folder.Skip(1).Any()); - public override long TotalSize - { - get - { - var i = Entries.Count; - return database._packSizes.Aggregate(0L, (total, packSize) => total + packSize); - } - } + public override bool IsEncrypted => Entries.First(x => !x.IsDirectory).IsEncrypted; - private sealed class SevenZipReader : AbstractReader + public override long TotalSize => + _database?._packSizes.Aggregate(0L, (total, packSize) => total + packSize) ?? 0; + + internal sealed class SevenZipReader : AbstractReader { - private readonly SevenZipArchive archive; - private CFolder currentFolder; - private Stream currentStream; - private CFileItem currentItem; + private readonly SevenZipArchive _archive; + private SevenZipEntry? _currentEntry; + private Stream? _currentFolderStream; + private CFolder? _currentFolder; + + /// + /// Enables internal diagnostics for tests. + /// When disabled (default), diagnostics properties return null to avoid exposing internal state. + /// + internal bool DiagnosticsEnabled { get; set; } + + /// + /// Current folder instance used to decide whether the solid folder stream should be reused. + /// Only available when is true. + /// + internal object? DiagnosticsCurrentFolder => DiagnosticsEnabled ? _currentFolder : null; + + /// + /// Current shared folder stream instance. + /// Only available when is true. + /// + internal Stream? DiagnosticsCurrentFolderStream => + DiagnosticsEnabled ? _currentFolderStream : null; internal SevenZipReader(ReaderOptions readerOptions, SevenZipArchive archive) - : base(readerOptions, ArchiveType.SevenZip) => this.archive = archive; + : base(readerOptions, ArchiveType.SevenZip, false) => this._archive = archive; - public override SevenZipVolume Volume => archive.Volumes.Single(); + public override SevenZipVolume Volume => _archive.Volumes.Single(); protected override IEnumerable GetEntries(Stream stream) { - var entries = archive.Entries.ToList(); + var entries = _archive.Entries.ToList(); stream.Position = 0; foreach (var dir in entries.Where(x => x.IsDirectory)) { + _currentEntry = dir; yield return dir; } - foreach ( - var group in entries.Where(x => !x.IsDirectory).GroupBy(x => x.FilePart.Folder) - ) + // For solid archives (entries in the same folder share a compressed stream), + // we must iterate entries sequentially and maintain the folder stream state + // across entries in the same folder to avoid recreating the decompression + // stream for each file, which breaks contiguous streaming. + foreach (var entry in entries.Where(x => !x.IsDirectory)) { - currentFolder = group.Key; - if (group.Key is null) - { - currentStream = Stream.Null; - } - else - { - currentStream = archive.database.GetFolderStream( - stream, - currentFolder, - new PasswordProvider(Options.Password) - ); - } - foreach (var entry in group) - { - currentItem = entry.FilePart.Header; - yield return entry; - } + _currentEntry = entry; + yield return entry; } } - protected override EntryStream GetEntryStream() => - CreateEntryStream(new ReadOnlySubStream(currentStream, currentItem.Size)); + protected override EntryStream GetEntryStream() + { + var entry = _currentEntry.NotNull("currentEntry is not null"); + if (entry.IsDirectory) + { + return CreateEntryStream(Stream.Null); + } + + var folder = entry.FilePart.Folder; + + // If folder is null (empty stream entry), return empty stream + if (folder is null) + { + return CreateEntryStream(Stream.Null); + } + + // Check if we're starting a new folder - dispose old folder stream if needed + if (folder != _currentFolder) + { + _currentFolderStream?.Dispose(); + _currentFolderStream = null; + _currentFolder = folder; + } + + // Create the folder stream once per folder + if (_currentFolderStream is null) + { + _currentFolderStream = _archive._database!.GetFolderStream( + _archive.Volumes.Single().Stream, + folder!, + _archive._database.PasswordProvider + ); + } + + return CreateEntryStream( + new ReadOnlySubStream(_currentFolderStream, entry.Size, leaveOpen: true) + ); + } + + protected override ValueTask GetEntryStreamAsync( + CancellationToken cancellationToken = default + ) => new(GetEntryStream()); + + public override void Dispose() + { + _currentFolderStream?.Dispose(); + _currentFolderStream = null; + base.Dispose(); + } + } + + /// + /// WORKAROUND: Forces async operations to use synchronous equivalents. + /// This is necessary because the LZMA decoder has bugs in its async implementation + /// that cause state corruption (IndexOutOfRangeException, DataErrorException). + /// + /// The proper fix would be to repair the LZMA decoder's async methods + /// (LzmaStream.ReadAsync, Decoder.CodeAsync, OutWindow async operations), + /// but that requires deep changes to the decoder state machine. + /// + private sealed class SyncOnlyStream : Stream + { + private readonly Stream _baseStream; + + public SyncOnlyStream(Stream baseStream) => _baseStream = baseStream; + + public override bool CanRead => _baseStream.CanRead; + public override bool CanSeek => _baseStream.CanSeek; + public override bool CanWrite => _baseStream.CanWrite; + public override long Length => _baseStream.Length; + public override long Position + { + get => _baseStream.Position; + set => _baseStream.Position = value; + } + + public override void Flush() => _baseStream.Flush(); + + public override int Read(byte[] buffer, int offset, int count) => + _baseStream.Read(buffer, offset, count); + + public override long Seek(long offset, SeekOrigin origin) => + _baseStream.Seek(offset, origin); + + public override void SetLength(long value) => _baseStream.SetLength(value); + + public override void Write(byte[] buffer, int offset, int count) => + _baseStream.Write(buffer, offset, count); + + // Force async operations to use sync equivalents to avoid LZMA decoder bugs + public override Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return Task.FromResult(_baseStream.Read(buffer, offset, count)); + } + + public override Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + cancellationToken.ThrowIfCancellationRequested(); + _baseStream.Write(buffer, offset, count); + return Task.CompletedTask; + } + + public override Task FlushAsync(CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + _baseStream.Flush(); + return Task.CompletedTask; + } + +#if !LEGACY_DOTNET + public override ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new ValueTask(_baseStream.Read(buffer.Span)); + } + + public override ValueTask WriteAsync( + ReadOnlyMemory buffer, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + _baseStream.Write(buffer.Span); + return ValueTask.CompletedTask; + } +#endif + + protected override void Dispose(bool disposing) + { + if (disposing) + { + _baseStream.Dispose(); + } + base.Dispose(disposing); + } } private class PasswordProvider : IPasswordProvider { - private readonly string _password; + private readonly string? _password; - public PasswordProvider(string password) => _password = password; + public PasswordProvider(string? password) => _password = password; - public string CryptoGetTextPassword() => _password; + public string? CryptoGetTextPassword() => _password; } } diff --git a/src/SharpCompress/Archives/SevenZip/SevenZipArchiveEntry.cs b/src/SharpCompress/Archives/SevenZip/SevenZipArchiveEntry.cs index 9824ca47..36e3542d 100644 --- a/src/SharpCompress/Archives/SevenZip/SevenZipArchiveEntry.cs +++ b/src/SharpCompress/Archives/SevenZip/SevenZipArchiveEntry.cs @@ -1,15 +1,29 @@ -using System.IO; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Options; using SharpCompress.Common.SevenZip; namespace SharpCompress.Archives.SevenZip; public class SevenZipArchiveEntry : SevenZipEntry, IArchiveEntry { - internal SevenZipArchiveEntry(SevenZipArchive archive, SevenZipFilePart part) - : base(part) => Archive = archive; + internal SevenZipArchiveEntry( + SevenZipArchive archive, + SevenZipFilePart part, + IReaderOptions readerOptions + ) + : base(part, readerOptions) => Archive = archive; public Stream OpenEntryStream() => FilePart.GetCompressedStream(); + public async ValueTask OpenEntryStreamAsync( + CancellationToken cancellationToken = default + ) => + ( + await FilePart.GetCompressedStreamAsync(cancellationToken).ConfigureAwait(false) + ).NotNull(); + public IArchive Archive { get; } public bool IsComplete => true; diff --git a/src/SharpCompress/Archives/Tar/TarArchive.Async.cs b/src/SharpCompress/Archives/Tar/TarArchive.Async.cs new file mode 100644 index 00000000..5d8833af --- /dev/null +++ b/src/SharpCompress/Archives/Tar/TarArchive.Async.cs @@ -0,0 +1,133 @@ +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Options; +using SharpCompress.Common.Tar; +using SharpCompress.IO; +using SharpCompress.Readers; +using SharpCompress.Readers.Tar; +using SharpCompress.Writers; +using SharpCompress.Writers.Tar; + +namespace SharpCompress.Archives.Tar; + +public partial class TarArchive +{ + protected override async ValueTask SaveToAsync( + Stream stream, + TarWriterOptions options, + IAsyncEnumerable oldEntries, + IEnumerable newEntries, + CancellationToken cancellationToken = default + ) + { + using var writer = new TarWriter(stream, options); + await foreach ( + var entry in oldEntries.WithCancellation(cancellationToken).ConfigureAwait(false) + ) + { + if (entry.IsDirectory) + { + await writer + .WriteDirectoryAsync( + entry.Key.NotNull("Entry Key is null"), + entry.LastModifiedTime, + cancellationToken + ) + .ConfigureAwait(false); + } + else + { + using var entryStream = entry.OpenEntryStream(); + await writer + .WriteAsync( + entry.Key.NotNull("Entry Key is null"), + entryStream, + entry.LastModifiedTime, + entry.Size, + cancellationToken + ) + .ConfigureAwait(false); + } + } + foreach (var entry in newEntries) + { + if (entry.IsDirectory) + { + await writer + .WriteDirectoryAsync( + entry.Key.NotNull("Entry Key is null"), + entry.LastModifiedTime, + cancellationToken + ) + .ConfigureAwait(false); + } + else + { + using var entryStream = entry.OpenEntryStream(); + await writer + .WriteAsync( + entry.Key.NotNull("Entry Key is null"), + entryStream, + entry.LastModifiedTime, + entry.Size, + cancellationToken + ) + .ConfigureAwait(false); + } + } + } + + protected override ValueTask CreateReaderForSolidExtractionAsync() + { + var stream = Volumes.Single().Stream; + stream.Position = 0; + return new((IAsyncReader)new TarReader(stream, ReaderOptions, _compressionType)); + } + + protected override async IAsyncEnumerable LoadEntriesAsync( + IAsyncEnumerable volumes + ) + { + var sourceStream = (await volumes.SingleAsync().ConfigureAwait(false)).Stream; + var stream = await GetStreamAsync(sourceStream).ConfigureAwait(false); + if (stream.CanSeek) + { + stream.Position = 0; + } + + var streamingMode = + _compressionType == CompressionType.None + ? StreamingMode.Seekable + : StreamingMode.Streaming; + + await foreach ( + var header in TarHeaderFactory.ReadHeaderAsync( + streamingMode, + stream, + ReaderOptions.ArchiveEncoding + ) + ) + { + if (header != null) + { + yield return new TarArchiveEntry( + this, + new TarFilePart( + header, + _compressionType == CompressionType.None ? stream : null + ), + CompressionType.None, + ReaderOptions + ); + } + else + { + throw new IncompleteArchiveException("Failed to read TAR header"); + } + } + } +} diff --git a/src/SharpCompress/Archives/Tar/TarArchive.Factory.cs b/src/SharpCompress/Archives/Tar/TarArchive.Factory.cs new file mode 100644 index 00000000..b63a6c22 --- /dev/null +++ b/src/SharpCompress/Archives/Tar/TarArchive.Factory.cs @@ -0,0 +1,257 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Tar.Headers; +using SharpCompress.Factories; +using SharpCompress.IO; +using SharpCompress.Readers; +using SharpCompress.Writers.Tar; + +namespace SharpCompress.Archives.Tar; + +public partial class TarArchive +#if NET8_0_OR_GREATER + : IWritableArchiveOpenable, + IMultiArchiveOpenable< + IWritableArchive, + IWritableAsyncArchive + > +#endif +{ + public static IWritableArchive OpenArchive( + string filePath, + ReaderOptions? readerOptions = null + ) + { + filePath.NotNullOrEmpty(nameof(filePath)); + return OpenArchive(new FileInfo(filePath), readerOptions); + } + + public static IWritableArchive OpenArchive( + FileInfo fileInfo, + ReaderOptions? readerOptions = null + ) + { + fileInfo.NotNull(nameof(fileInfo)); + return OpenArchive([fileInfo], readerOptions ?? ReaderOptions.ForFilePath); + } + + public static IWritableArchive OpenArchive( + IReadOnlyList fileInfos, + ReaderOptions? readerOptions = null + ) + { + fileInfos.NotNull(nameof(fileInfos)); + var files = fileInfos; + var sourceStream = new SourceStream( + files[0], + i => i < files.Count ? files[i] : null, + readerOptions ?? ReaderOptions.ForFilePath + ); + var compressionType = TarFactory.GetCompressionType( + sourceStream, + sourceStream.ReaderOptions + ); + sourceStream.Seek(0, SeekOrigin.Begin); + return new TarArchive(sourceStream, compressionType); + } + + public static IWritableArchive OpenArchive( + IReadOnlyList streams, + ReaderOptions? readerOptions = null + ) + { + var strms = streams.RequireReadable().RequireSeekable().ToList(); + var sourceStream = new SourceStream( + strms[0], + i => i < strms.Count ? strms[i] : null, + readerOptions ?? ReaderOptions.ForExternalStream + ); + var compressionType = TarFactory.GetCompressionType( + sourceStream, + sourceStream.ReaderOptions + ); + sourceStream.Seek(0, SeekOrigin.Begin); + return new TarArchive(sourceStream, compressionType); + } + + public static IWritableArchive OpenArchive( + Stream stream, + ReaderOptions? readerOptions = null + ) + { + stream.RequireReadable(); + stream.RequireSeekable(); + + return OpenArchive([stream], readerOptions); + } + + public static async ValueTask> OpenAsyncArchive( + Stream stream, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + stream.RequireReadable(); + stream.RequireSeekable(); + var sourceStream = new SourceStream( + stream, + i => null, + readerOptions ?? ReaderOptions.ForExternalStream + ); + var compressionType = await TarFactory + .GetCompressionTypeAsync(sourceStream, sourceStream.ReaderOptions, cancellationToken) + .ConfigureAwait(false); + sourceStream.Seek(0, SeekOrigin.Begin); + return new TarArchive(sourceStream, compressionType); + } + + public static ValueTask> OpenAsyncArchive( + string filePath, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + filePath.NotNullOrEmpty(nameof(filePath)); + return OpenAsyncArchive(new FileInfo(filePath), readerOptions, cancellationToken); + } + + public static async ValueTask> OpenAsyncArchive( + FileInfo fileInfo, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + fileInfo.NotNull(nameof(fileInfo)); + readerOptions ??= ReaderOptions.ForFilePath; + var sourceStream = new SourceStream(fileInfo, i => null, readerOptions); + var compressionType = await TarFactory + .GetCompressionTypeAsync(sourceStream, sourceStream.ReaderOptions, cancellationToken) + .ConfigureAwait(false); + sourceStream.Seek(0, SeekOrigin.Begin); + return new TarArchive(sourceStream, compressionType); + } + + public static async ValueTask> OpenAsyncArchive( + IReadOnlyList streams, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + var strms = streams.RequireReadable().RequireSeekable().ToList(); + var sourceStream = new SourceStream( + strms[0], + i => i < strms.Count ? strms[i] : null, + readerOptions ?? ReaderOptions.ForExternalStream + ); + var compressionType = await TarFactory + .GetCompressionTypeAsync(sourceStream, sourceStream.ReaderOptions, cancellationToken) + .ConfigureAwait(false); + sourceStream.Seek(0, SeekOrigin.Begin); + return new TarArchive(sourceStream, compressionType); + } + + public static async ValueTask> OpenAsyncArchive( + IReadOnlyList fileInfos, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + fileInfos.NotNull(nameof(fileInfos)); + var files = fileInfos; + var sourceStream = new SourceStream( + files[0], + i => i < files.Count ? files[i] : null, + readerOptions ?? ReaderOptions.ForFilePath + ); + var compressionType = await TarFactory + .GetCompressionTypeAsync(sourceStream, sourceStream.ReaderOptions, cancellationToken) + .ConfigureAwait(false); + sourceStream.Seek(0, SeekOrigin.Begin); + return new TarArchive(sourceStream, compressionType); + } + + public static bool IsTarFile(string filePath) => IsTarFile(new FileInfo(filePath)); + + public static bool IsTarFile(FileInfo fileInfo) + { + if (!fileInfo.Exists) + { + return false; + } + using Stream stream = fileInfo.OpenRead(); + return IsTarFile(stream); + } + + public static bool IsTarFile(Stream stream) + { + try + { + var tarHeader = new TarHeader(new ArchiveEncoding()); + var reader = new BinaryReader(stream, Encoding.UTF8, false); + var readSucceeded = tarHeader.Read(reader); + var isEmptyArchive = + tarHeader.Name?.Length == 0 + && tarHeader.Size == 0 + && IsDefined(tarHeader.EntryType); + return readSucceeded || isEmptyArchive; + } + catch (Exception) + { + // Catch all exceptions during tar header reading to determine if this is a valid tar file + // Invalid tar files or corrupted streams will throw various exceptions + return false; + } + } + + public static async ValueTask IsTarFileAsync( + Stream stream, + CancellationToken cancellationToken = default + ) + { + try + { + var tarHeader = new TarHeader(new ArchiveEncoding()); +#if NET8_0_OR_GREATER + await using var reader = new AsyncBinaryReader(stream, leaveOpen: true); +#else + using var reader = new AsyncBinaryReader(stream, leaveOpen: true); +#endif + var readSucceeded = await tarHeader.ReadAsync(reader).ConfigureAwait(false); + var isEmptyArchive = + tarHeader.Name?.Length == 0 + && tarHeader.Size == 0 + && IsDefined(tarHeader.EntryType); + return readSucceeded || isEmptyArchive; + } + catch (Exception) + { + // Catch all exceptions during tar header reading to determine if this is a valid tar file + // Invalid tar files or corrupted streams will throw various exceptions + return false; + } + } + + public static IWritableArchive CreateArchive() => new TarArchive(); + + public static ValueTask> CreateAsyncArchive() => + new(new TarArchive()); + + private static bool IsDefined(EntryType value) + { +#if LEGACY_DOTNET + return Enum.IsDefined(typeof(EntryType), value); +#else + return Enum.IsDefined(value); +#endif + } +} diff --git a/src/SharpCompress/Archives/Tar/TarArchive.cs b/src/SharpCompress/Archives/Tar/TarArchive.cs index 43184b66..038fd32d 100644 --- a/src/SharpCompress/Archives/Tar/TarArchive.cs +++ b/src/SharpCompress/Archives/Tar/TarArchive.cs @@ -2,152 +2,123 @@ using System; using System.Collections.Generic; using System.IO; using System.Linq; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common; using SharpCompress.Common.Tar; -using SharpCompress.Common.Tar.Headers; using SharpCompress.IO; +using SharpCompress.Providers; using SharpCompress.Readers; using SharpCompress.Readers.Tar; -using SharpCompress.Writers; using SharpCompress.Writers.Tar; namespace SharpCompress.Archives.Tar; -public class TarArchive : AbstractWritableArchive +public partial class TarArchive + : AbstractWritableArchive { - /// - /// Constructor expects a filepath to an existing file. - /// - /// - /// - public static TarArchive Open(string filePath, ReaderOptions? readerOptions = null) + private readonly CompressionType _compressionType; + + protected override IEnumerable LoadVolumes(SourceStream sourceStream) { - filePath.CheckNotNullOrEmpty(nameof(filePath)); - return Open(new FileInfo(filePath), readerOptions ?? new ReaderOptions()); + sourceStream.NotNull("SourceStream is null").LoadAllParts(); + return new TarVolume(sourceStream, ReaderOptions, 1).AsEnumerable(); } - /// - /// Constructor with a FileInfo object to an existing file. - /// - /// - /// - public static TarArchive Open(FileInfo fileInfo, ReaderOptions? readerOptions = null) + internal TarArchive(SourceStream sourceStream, CompressionType compressionType) + : base(ArchiveType.Tar, sourceStream) { - fileInfo.CheckNotNull(nameof(fileInfo)); - return new TarArchive( - new SourceStream( - fileInfo, - i => ArchiveVolumeFactory.GetFilePart(i, fileInfo), - readerOptions ?? new ReaderOptions() - ) - ); + _compressionType = compressionType; } - /// - /// Constructor with all file parts passed in - /// - /// - /// - public static TarArchive Open( - IEnumerable fileInfos, - ReaderOptions? readerOptions = null - ) - { - fileInfos.CheckNotNull(nameof(fileInfos)); - var files = fileInfos.ToArray(); - return new TarArchive( - new SourceStream( - files[0], - i => i < files.Length ? files[i] : null, - readerOptions ?? new ReaderOptions() - ) - ); - } - - /// - /// Constructor with all stream parts passed in - /// - /// - /// - public static TarArchive Open(IEnumerable streams, ReaderOptions? readerOptions = null) - { - streams.CheckNotNull(nameof(streams)); - var strms = streams.ToArray(); - return new TarArchive( - new SourceStream( - strms[0], - i => i < strms.Length ? strms[i] : null, - readerOptions ?? new ReaderOptions() - ) - ); - } - - /// - /// Takes a seekable Stream as a source - /// - /// - /// - public static TarArchive Open(Stream stream, ReaderOptions? readerOptions = null) - { - stream.CheckNotNull(nameof(stream)); - return new TarArchive( - new SourceStream(stream, i => null, readerOptions ?? new ReaderOptions()) - ); - } - - public static bool IsTarFile(string filePath) => IsTarFile(new FileInfo(filePath)); - - public static bool IsTarFile(FileInfo fileInfo) - { - if (!fileInfo.Exists) - { - return false; - } - using Stream stream = fileInfo.OpenRead(); - return IsTarFile(stream); - } - - public static bool IsTarFile(Stream stream) - { - try - { - var tarHeader = new TarHeader(new ArchiveEncoding()); - var readSucceeded = tarHeader.Read(new BinaryReader(stream)); - var isEmptyArchive = - tarHeader.Name.Length == 0 - && tarHeader.Size == 0 - && Enum.IsDefined(typeof(EntryType), tarHeader.EntryType); - return readSucceeded || isEmptyArchive; - } - catch { } - return false; - } - - protected override IEnumerable LoadVolumes(SourceStream srcStream) - { - SrcStream.LoadAllParts(); //request all streams - var idx = 0; - return new TarVolume(srcStream, ReaderOptions, idx++).AsEnumerable(); //simple single volume or split, multivolume not supported - } - - /// - /// Constructor with a SourceStream able to handle FileInfo and Streams. - /// - /// - /// - internal TarArchive(SourceStream srcStream) - : base(ArchiveType.Tar, srcStream) { } - - internal TarArchive() + private TarArchive() : base(ArchiveType.Tar) { } + private Stream GetStream(Stream stream) => + _compressionType switch + { + CompressionType.BZip2 => ReaderOptions.Providers.CreateDecompressStream( + CompressionType.BZip2, + stream + ), + CompressionType.GZip => ReaderOptions.Providers.CreateDecompressStream( + CompressionType.GZip, + stream, + CompressionContext.FromStream(stream).WithReaderOptions(ReaderOptions) + ), + CompressionType.ZStandard => ReaderOptions.Providers.CreateDecompressStream( + CompressionType.ZStandard, + stream + ), + CompressionType.LZip => ReaderOptions.Providers.CreateDecompressStream( + CompressionType.LZip, + stream + ), + CompressionType.Xz => ReaderOptions.Providers.CreateDecompressStream( + CompressionType.Xz, + stream + ), + CompressionType.Lzw => ReaderOptions.Providers.CreateDecompressStream( + CompressionType.Lzw, + stream + ), + CompressionType.None => stream, + _ => throw new NotSupportedException("Invalid compression type: " + _compressionType), + }; + + private ValueTask GetStreamAsync( + Stream stream, + CancellationToken cancellationToken = default + ) => + _compressionType switch + { + CompressionType.BZip2 => ReaderOptions.Providers.CreateDecompressStreamAsync( + CompressionType.BZip2, + stream, + cancellationToken + ), + CompressionType.GZip => ReaderOptions.Providers.CreateDecompressStreamAsync( + CompressionType.GZip, + stream, + CompressionContext.FromStream(stream).WithReaderOptions(ReaderOptions), + cancellationToken + ), + CompressionType.ZStandard => ReaderOptions.Providers.CreateDecompressStreamAsync( + CompressionType.ZStandard, + stream, + cancellationToken + ), + CompressionType.LZip => ReaderOptions.Providers.CreateDecompressStreamAsync( + CompressionType.LZip, + stream, + cancellationToken + ), + CompressionType.Xz => ReaderOptions.Providers.CreateDecompressStreamAsync( + CompressionType.Xz, + stream, + cancellationToken + ), + CompressionType.Lzw => ReaderOptions.Providers.CreateDecompressStreamAsync( + CompressionType.Lzw, + stream, + cancellationToken + ), + CompressionType.None => new ValueTask(stream), + _ => throw new NotSupportedException("Invalid compression type: " + _compressionType), + }; + protected override IEnumerable LoadEntries(IEnumerable volumes) { - var stream = volumes.Single().Stream; - TarHeader? previousHeader = null; + var stream = GetStream(volumes.Single().Stream); + if (stream.CanSeek) + { + stream.Position = 0; + } foreach ( var header in TarHeaderFactory.ReadHeader( - StreamingMode.Seekable, + _compressionType == CompressionType.None + ? StreamingMode.Seekable + : StreamingMode.Streaming, stream, ReaderOptions.ArchiveEncoding ) @@ -155,50 +126,25 @@ public class TarArchive : AbstractWritableArchive { if (header != null) { - if (header.EntryType == EntryType.LongName) - { - previousHeader = header; - } - else - { - if (previousHeader != null) - { - var entry = new TarArchiveEntry( - this, - new TarFilePart(previousHeader, stream), - CompressionType.None - ); - - var oldStreamPos = stream.Position; - - using (var entryStream = entry.OpenEntryStream()) - { - using var memoryStream = new MemoryStream(); - entryStream.TransferTo(memoryStream); - memoryStream.Position = 0; - var bytes = memoryStream.ToArray(); - - header.Name = ReaderOptions.ArchiveEncoding.Decode(bytes).TrimNulls(); - } - - stream.Position = oldStreamPos; - - previousHeader = null; - } - yield return new TarArchiveEntry( - this, - new TarFilePart(header, stream), - CompressionType.None - ); - } + yield return new TarArchiveEntry( + this, + new TarFilePart( + header, + _compressionType == CompressionType.None ? stream : null + ), + CompressionType.None, + ReaderOptions + ); + } + else + { + throw new IncompleteArchiveException("Failed to read TAR header"); } } } - public static TarArchive Create() => new TarArchive(); - protected override TarArchiveEntry CreateEntryInternal( - string filePath, + string key, Stream source, long size, DateTime? modified, @@ -208,24 +154,42 @@ public class TarArchive : AbstractWritableArchive this, source, CompressionType.Unknown, - filePath, + key, size, modified, closeStream ); + protected override TarArchiveEntry CreateDirectoryEntry(string key, DateTime? modified) => + new TarWritableArchiveEntry(this, key, modified); + protected override void SaveTo( Stream stream, - WriterOptions options, + TarWriterOptions options, IEnumerable oldEntries, IEnumerable newEntries ) { - using var writer = new TarWriter(stream, new TarWriterOptions(options)); - foreach (var entry in oldEntries.Concat(newEntries).Where(x => !x.IsDirectory)) + using var writer = new TarWriter(stream, options); + foreach (var entry in oldEntries.Concat(newEntries)) { - using var entryStream = entry.OpenEntryStream(); - writer.Write(entry.Key, entryStream, entry.LastModifiedTime, entry.Size); + if (entry.IsDirectory) + { + writer.WriteDirectory( + entry.Key.NotNull("Entry Key is null"), + entry.LastModifiedTime + ); + } + else + { + using var entryStream = entry.OpenEntryStream(); + writer.Write( + entry.Key.NotNull("Entry Key is null"), + entryStream, + entry.LastModifiedTime, + entry.Size + ); + } } } @@ -233,6 +197,6 @@ public class TarArchive : AbstractWritableArchive { var stream = Volumes.Single().Stream; stream.Position = 0; - return TarReader.Open(stream); + return new TarReader(stream, ReaderOptions, _compressionType); } } diff --git a/src/SharpCompress/Archives/Tar/TarArchiveEntry.cs b/src/SharpCompress/Archives/Tar/TarArchiveEntry.cs index 2da84d71..c09d4755 100644 --- a/src/SharpCompress/Archives/Tar/TarArchiveEntry.cs +++ b/src/SharpCompress/Archives/Tar/TarArchiveEntry.cs @@ -1,16 +1,31 @@ -using System.IO; +using System.IO; using System.Linq; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common; +using SharpCompress.Common.Options; using SharpCompress.Common.Tar; namespace SharpCompress.Archives.Tar; public class TarArchiveEntry : TarEntry, IArchiveEntry { - internal TarArchiveEntry(TarArchive archive, TarFilePart part, CompressionType compressionType) - : base(part, compressionType) => Archive = archive; + internal TarArchiveEntry( + TarArchive archive, + TarFilePart? part, + CompressionType compressionType, + IReaderOptions readerOptions + ) + : base(part, compressionType, readerOptions) => Archive = archive; - public virtual Stream OpenEntryStream() => Parts.Single().GetCompressedStream(); + public virtual Stream OpenEntryStream() => Parts.Single().GetCompressedStream().NotNull(); + + public async ValueTask OpenEntryStreamAsync( + CancellationToken cancellationToken = default + ) => + ( + await Parts.Single().GetCompressedStreamAsync(cancellationToken).ConfigureAwait(false) + ).NotNull(); #region IArchiveEntry Members diff --git a/src/SharpCompress/Archives/Tar/TarWritableArchiveEntry.cs b/src/SharpCompress/Archives/Tar/TarWritableArchiveEntry.cs index 0bbea709..84f8ff44 100644 --- a/src/SharpCompress/Archives/Tar/TarWritableArchiveEntry.cs +++ b/src/SharpCompress/Archives/Tar/TarWritableArchiveEntry.cs @@ -1,5 +1,3 @@ -#nullable disable - using System; using System.Collections.Generic; using System.IO; @@ -11,7 +9,8 @@ namespace SharpCompress.Archives.Tar; internal sealed class TarWritableArchiveEntry : TarArchiveEntry, IWritableArchiveEntry { private readonly bool closeStream; - private readonly Stream stream; + private readonly Stream? stream; + private readonly bool isDirectory; internal TarWritableArchiveEntry( TarArchive archive, @@ -22,13 +21,29 @@ internal sealed class TarWritableArchiveEntry : TarArchiveEntry, IWritableArchiv DateTime? lastModified, bool closeStream ) - : base(archive, null, compressionType) + : base(archive, null, compressionType, archive.ReaderOptions) { this.stream = stream; Key = path; Size = size; LastModifiedTime = lastModified; this.closeStream = closeStream; + isDirectory = false; + } + + internal TarWritableArchiveEntry( + TarArchive archive, + string directoryPath, + DateTime? lastModified + ) + : base(archive, null, CompressionType.None, archive.ReaderOptions) + { + stream = null; + Key = directoryPath; + Size = 0; + LastModifiedTime = lastModified; + closeStream = false; + isDirectory = true; } public override long Crc => 0; @@ -49,23 +64,27 @@ internal sealed class TarWritableArchiveEntry : TarArchiveEntry, IWritableArchiv public override bool IsEncrypted => false; - public override bool IsDirectory => false; + public override bool IsDirectory => isDirectory; public override bool IsSplitAfter => false; internal override IEnumerable Parts => throw new NotImplementedException(); - Stream IWritableArchiveEntry.Stream => stream; + Stream IWritableArchiveEntry.Stream => stream ?? Stream.Null; public override Stream OpenEntryStream() { + if (stream is null) + { + return Stream.Null; + } //ensure new stream is at the start, this could be reset stream.Seek(0, SeekOrigin.Begin); - return NonDisposingStream.Create(stream); + return SharpCompressStream.CreateNonDisposing(stream); } internal override void Close() { - if (closeStream) + if (closeStream && stream is not null) { stream.Dispose(); } diff --git a/src/SharpCompress/Archives/Zip/ZipArchive.Async.cs b/src/SharpCompress/Archives/Zip/ZipArchive.Async.cs new file mode 100644 index 00000000..97c412e6 --- /dev/null +++ b/src/SharpCompress/Archives/Zip/ZipArchive.Async.cs @@ -0,0 +1,139 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Options; +using SharpCompress.Common.Zip; +using SharpCompress.Common.Zip.Headers; +using SharpCompress.IO; +using SharpCompress.Readers; +using SharpCompress.Writers; +using SharpCompress.Writers.Zip; + +namespace SharpCompress.Archives.Zip; + +public partial class ZipArchive +{ + protected override async IAsyncEnumerable LoadEntriesAsync( + IAsyncEnumerable volumes + ) + { + var vols = await volumes.ToListAsync().ConfigureAwait(false); + var volsArray = vols.ToArray(); + + await foreach ( + var h in headerFactory.NotNull().ReadSeekableHeaderAsync(volsArray.Last().Stream) + ) + { + if (h != null) + { + switch (h.ZipHeaderType) + { + case ZipHeaderType.DirectoryEntry: + { + var deh = (DirectoryEntryHeader)h; + Stream s; + if ( + deh.RelativeOffsetOfEntryHeader + deh.CompressedSize + > volsArray[deh.DiskNumberStart].Stream.Length + ) + { + var v = volsArray.Skip(deh.DiskNumberStart).ToArray(); + s = new SourceStream( + v[0].Stream, + i => i < v.Length ? v[i].Stream : null, + ReaderOptions.ForExternalStream + ); + } + else + { + s = volsArray[deh.DiskNumberStart].Stream; + } + + yield return new ZipArchiveEntry( + this, + new SeekableZipFilePart( + headerFactory.NotNull(), + deh, + s, + ReaderOptions.Providers + ), + ReaderOptions + ); + } + break; + case ZipHeaderType.DirectoryEnd: + { + var bytes = ((DirectoryEndHeader)h).Comment ?? Array.Empty(); + volsArray.Last().Comment = ReaderOptions.ArchiveEncoding.Decode(bytes); + yield break; + } + } + } + } + } + + protected override async ValueTask SaveToAsync( + Stream stream, + ZipWriterOptions options, + IAsyncEnumerable oldEntries, + IEnumerable newEntries, + CancellationToken cancellationToken = default + ) + { + using var writer = new ZipWriter(stream, options); + await foreach ( + var entry in oldEntries.WithCancellation(cancellationToken).ConfigureAwait(false) + ) + { + if (entry.IsDirectory) + { + await writer + .WriteDirectoryAsync( + entry.Key.NotNull("Entry Key is null"), + entry.LastModifiedTime, + cancellationToken + ) + .ConfigureAwait(false); + } + else + { + using var entryStream = entry.OpenEntryStream(); + await writer + .WriteAsync( + entry.Key.NotNull("Entry Key is null"), + entryStream, + cancellationToken + ) + .ConfigureAwait(false); + } + } + foreach (var entry in newEntries) + { + if (entry.IsDirectory) + { + await writer + .WriteDirectoryAsync( + entry.Key.NotNull("Entry Key is null"), + entry.LastModifiedTime, + cancellationToken + ) + .ConfigureAwait(false); + } + else + { + using var entryStream = entry.OpenEntryStream(); + await writer + .WriteAsync( + entry.Key.NotNull("Entry Key is null"), + entryStream, + cancellationToken + ) + .ConfigureAwait(false); + } + } + } +} diff --git a/src/SharpCompress/Archives/Zip/ZipArchive.Factory.cs b/src/SharpCompress/Archives/Zip/ZipArchive.Factory.cs new file mode 100644 index 00000000..1e02ad6f --- /dev/null +++ b/src/SharpCompress/Archives/Zip/ZipArchive.Factory.cs @@ -0,0 +1,306 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Zip; +using SharpCompress.Common.Zip.Headers; +using SharpCompress.IO; +using SharpCompress.Readers; +using SharpCompress.Writers.Zip; + +namespace SharpCompress.Archives.Zip; + +public partial class ZipArchive +#if NET8_0_OR_GREATER + : IWritableArchiveOpenable, + IMultiArchiveOpenable< + IWritableArchive, + IWritableAsyncArchive + > +#endif +{ + public static IWritableArchive OpenArchive( + string filePath, + ReaderOptions? readerOptions = null + ) + { + filePath.NotNullOrEmpty(nameof(filePath)); + return OpenArchive(new FileInfo(filePath), readerOptions); + } + + public static IWritableArchive OpenArchive( + FileInfo fileInfo, + ReaderOptions? readerOptions = null + ) + { + fileInfo.NotNull(nameof(fileInfo)); + return new ZipArchive( + new SourceStream( + fileInfo, + i => ZipArchiveVolumeFactory.GetFilePart(i, fileInfo), + readerOptions ?? ReaderOptions.ForFilePath + ) + ); + } + + public static IWritableArchive OpenArchive( + IReadOnlyList fileInfos, + ReaderOptions? readerOptions = null + ) + { + fileInfos.NotNull(nameof(fileInfos)); + var files = fileInfos; + return new ZipArchive( + new SourceStream( + files[0], + i => i < files.Count ? files[i] : null, + readerOptions ?? ReaderOptions.ForFilePath + ) + ); + } + + public static IWritableArchive OpenArchive( + IReadOnlyList streams, + ReaderOptions? readerOptions = null + ) + { + var strms = streams.RequireReadable().RequireSeekable().ToList(); + return new ZipArchive( + new SourceStream( + strms[0], + i => i < strms.Count ? strms[i] : null, + readerOptions ?? ReaderOptions.ForExternalStream + ) + ); + } + + public static IWritableArchive OpenArchive( + Stream stream, + ReaderOptions? readerOptions = null + ) + { + stream.RequireReadable(); + stream.RequireSeekable(); + + return new ZipArchive( + new SourceStream(stream, i => null, readerOptions ?? ReaderOptions.ForExternalStream) + ); + } + + public static ValueTask> OpenAsyncArchive( + string filePath, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IWritableAsyncArchive)OpenArchive(filePath, readerOptions)); + } + + public static ValueTask> OpenAsyncArchive( + Stream stream, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IWritableAsyncArchive)OpenArchive(stream, readerOptions)); + } + + public static ValueTask> OpenAsyncArchive( + FileInfo fileInfo, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IWritableAsyncArchive)OpenArchive(fileInfo, readerOptions)); + } + + public static ValueTask> OpenAsyncArchive( + IReadOnlyList streams, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IWritableAsyncArchive)OpenArchive(streams, readerOptions)); + } + + public static ValueTask> OpenAsyncArchive( + IReadOnlyList fileInfos, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IWritableAsyncArchive)OpenArchive(fileInfos, readerOptions)); + } + + public static bool IsZipFile(string filePath, string? password = null) => + IsZipFile(new FileInfo(filePath), password); + + public static bool IsZipFile(FileInfo fileInfo, string? password = null) + { + if (!fileInfo.Exists) + { + return false; + } + using Stream stream = fileInfo.OpenRead(); + return IsZipFile(stream, password); + } + + public static bool IsZipFile(Stream stream, string? password = null) + { + var headerFactory = new StreamingZipHeaderFactory(password, new ArchiveEncoding(), null); + try + { + var header = headerFactory + .ReadStreamHeader(stream) + .FirstOrDefault(x => x.ZipHeaderType != ZipHeaderType.Split); + if (header is null) + { + return false; + } + return IsDefined(header.ZipHeaderType); + } + catch (CryptographicException) + { + return true; + } + catch + { + return false; + } + } + + public static bool IsZipMulti(Stream stream, string? password = null) + { + var headerFactory = new StreamingZipHeaderFactory(password, new ArchiveEncoding(), null); + try + { + var header = headerFactory + .ReadStreamHeader(stream) + .FirstOrDefault(x => x.ZipHeaderType != ZipHeaderType.Split); + if (header is null) + { + if (stream.CanSeek) + { + var z = new SeekableZipHeaderFactory(password, new ArchiveEncoding()); + var x = z.ReadSeekableHeader(stream).FirstOrDefault(); + return x?.ZipHeaderType == ZipHeaderType.DirectoryEntry; + } + else + { + return false; + } + } + return IsDefined(header.ZipHeaderType); + } + catch (CryptographicException) + { + return true; + } + catch + { + return false; + } + } + + public static async ValueTask IsZipFileAsync( + Stream stream, + string? password = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + var headerFactory = new StreamingZipHeaderFactory(password, new ArchiveEncoding(), null); + try + { + var header = await headerFactory + .ReadStreamHeaderAsync(stream) + .Where(x => x.ZipHeaderType != ZipHeaderType.Split) + .FirstOrDefaultAsync(cancellationToken) + .ConfigureAwait(false); + if (header is null) + { + return false; + } + return IsDefined(header.ZipHeaderType); + } + catch (CryptographicException) + { + return true; + } + catch + { + return false; + } + } + + public static IWritableArchive CreateArchive() => new ZipArchive(); + + public static ValueTask> CreateAsyncArchive() => + new(new ZipArchive()); + + public static async ValueTask IsZipMultiAsync( + Stream stream, + string? password = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + var headerFactory = new StreamingZipHeaderFactory(password, new ArchiveEncoding(), null); + try + { + var header = await headerFactory + .ReadStreamHeaderAsync(stream) + .Where(x => x.ZipHeaderType != ZipHeaderType.Split) + .FirstOrDefaultAsync(cancellationToken) + .ConfigureAwait(false); + if (header is null) + { + if (stream.CanSeek) + { + var z = new SeekableZipHeaderFactory(password, new ArchiveEncoding()); + ZipHeader? x = null; + await foreach ( + var h in z.ReadSeekableHeaderAsync(stream) + .WithCancellation(cancellationToken) + .ConfigureAwait(false) + ) + { + x = h; + break; + } + return x?.ZipHeaderType == ZipHeaderType.DirectoryEntry; + } + else + { + return false; + } + } + return IsDefined(header.ZipHeaderType); + } + catch (CryptographicException) + { + return true; + } + catch + { + return false; + } + } + + private static bool IsDefined(ZipHeaderType value) + { +#if LEGACY_DOTNET + return Enum.IsDefined(typeof(ZipHeaderType), value); +#else + return Enum.IsDefined(value); +#endif + } +} diff --git a/src/SharpCompress/Archives/Zip/ZipArchive.cs b/src/SharpCompress/Archives/Zip/ZipArchive.cs index 14e4a93e..3b5fd5b2 100644 --- a/src/SharpCompress/Archives/Zip/ZipArchive.cs +++ b/src/SharpCompress/Archives/Zip/ZipArchive.cs @@ -2,7 +2,10 @@ using System; using System.Collections.Generic; using System.IO; using System.Linq; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common; +using SharpCompress.Common.Options; using SharpCompress.Common.Zip; using SharpCompress.Common.Zip.Headers; using SharpCompress.Compressors.Deflate; @@ -14,224 +17,63 @@ using SharpCompress.Writers.Zip; namespace SharpCompress.Archives.Zip; -public class ZipArchive : AbstractWritableArchive +public partial class ZipArchive + : AbstractWritableArchive { -#nullable disable - private readonly SeekableZipHeaderFactory headerFactory; + private readonly SeekableZipHeaderFactory? headerFactory; -#nullable enable - - /// - /// Gets or sets the compression level applied to files added to the archive, - /// if the compression method is set to deflate - /// public CompressionLevel DeflateCompressionLevel { get; set; } - /// - /// Constructor with a SourceStream able to handle FileInfo and Streams. - /// - /// - /// - internal ZipArchive(SourceStream srcStream) - : base(ArchiveType.Zip, srcStream) => + internal ZipArchive(SourceStream sourceStream) + : base(ArchiveType.Zip, sourceStream) => headerFactory = new SeekableZipHeaderFactory( - srcStream.ReaderOptions.Password, - srcStream.ReaderOptions.ArchiveEncoding + sourceStream.ReaderOptions.Password, + sourceStream.ReaderOptions.ArchiveEncoding ); - /// - /// Constructor expects a filepath to an existing file. - /// - /// - /// - public static ZipArchive Open(string filePath, ReaderOptions? readerOptions = null) - { - filePath.CheckNotNullOrEmpty(nameof(filePath)); - return Open(new FileInfo(filePath), readerOptions ?? new ReaderOptions()); - } - - /// - /// Constructor with a FileInfo object to an existing file. - /// - /// - /// - public static ZipArchive Open(FileInfo fileInfo, ReaderOptions? readerOptions = null) - { - fileInfo.CheckNotNull(nameof(fileInfo)); - return new ZipArchive( - new SourceStream( - fileInfo, - i => ZipArchiveVolumeFactory.GetFilePart(i, fileInfo), - readerOptions ?? new ReaderOptions() - ) - ); - } - - /// - /// Constructor with all file parts passed in - /// - /// - /// - public static ZipArchive Open( - IEnumerable fileInfos, - ReaderOptions? readerOptions = null - ) - { - fileInfos.CheckNotNull(nameof(fileInfos)); - var files = fileInfos.ToArray(); - return new ZipArchive( - new SourceStream( - files[0], - i => i < files.Length ? files[i] : null, - readerOptions ?? new ReaderOptions() - ) - ); - } - - /// - /// Constructor with all stream parts passed in - /// - /// - /// - public static ZipArchive Open(IEnumerable streams, ReaderOptions? readerOptions = null) - { - streams.CheckNotNull(nameof(streams)); - var strms = streams.ToArray(); - return new ZipArchive( - new SourceStream( - strms[0], - i => i < strms.Length ? strms[i] : null, - readerOptions ?? new ReaderOptions() - ) - ); - } - - /// - /// Takes a seekable Stream as a source - /// - /// - /// - public static ZipArchive Open(Stream stream, ReaderOptions? readerOptions = null) - { - stream.CheckNotNull(nameof(stream)); - return new ZipArchive( - new SourceStream(stream, i => null, readerOptions ?? new ReaderOptions()) - ); - } - - public static bool IsZipFile(string filePath, string? password = null) => - IsZipFile(new FileInfo(filePath), password); - - public static bool IsZipFile(FileInfo fileInfo, string? password = null) - { - if (!fileInfo.Exists) - { - return false; - } - using Stream stream = fileInfo.OpenRead(); - return IsZipFile(stream, password); - } - - public static bool IsZipFile(Stream stream, string? password = null) - { - var headerFactory = new StreamingZipHeaderFactory(password, new ArchiveEncoding(), null); - try - { - var header = headerFactory - .ReadStreamHeader(stream) - .FirstOrDefault(x => x.ZipHeaderType != ZipHeaderType.Split); - if (header is null) - { - return false; - } - return Enum.IsDefined(typeof(ZipHeaderType), header.ZipHeaderType); - } - catch (CryptographicException) - { - return true; - } - catch - { - return false; - } - } - - public static bool IsZipMulti(Stream stream, string? password = null) - { - var headerFactory = new StreamingZipHeaderFactory(password, new ArchiveEncoding(), null); - try - { - var header = headerFactory - .ReadStreamHeader(stream) - .FirstOrDefault(x => x.ZipHeaderType != ZipHeaderType.Split); - if (header is null) - { - if (stream.CanSeek) //could be multipart. Test for central directory - might not be z64 safe - { - var z = new SeekableZipHeaderFactory(password, new ArchiveEncoding()); - var x = z.ReadSeekableHeader(stream).FirstOrDefault(); - return x?.ZipHeaderType == ZipHeaderType.DirectoryEntry; - } - else - { - return false; - } - } - return Enum.IsDefined(typeof(ZipHeaderType), header.ZipHeaderType); - } - catch (CryptographicException) - { - return true; - } - catch - { - return false; - } - } - - protected override IEnumerable LoadVolumes(SourceStream srcStream) - { - SrcStream.LoadAllParts(); //request all streams - SrcStream.Position = 0; - - var streams = SrcStream.Streams.ToList(); - var idx = 0; - if (streams.Count > 1) //test part 2 - true = multipart not split - { - streams[1].Position += 4; //skip the POST_DATA_DESCRIPTOR to prevent an exception - var isZip = IsZipFile(streams[1], ReaderOptions.Password); - streams[1].Position -= 4; - if (isZip) - { - SrcStream.IsVolumes = true; - - var tmp = streams[0]; //arcs as zip, z01 ... swap the zip the end - streams.RemoveAt(0); - streams.Add(tmp); - - //streams[0].Position = 4; //skip the POST_DATA_DESCRIPTOR to prevent an exception - return streams.Select(a => new ZipVolume(a, ReaderOptions, idx++)); - } - } - - //split mode or single file - return new ZipVolume(SrcStream, ReaderOptions, idx++).AsEnumerable(); - } - internal ZipArchive() : base(ArchiveType.Zip) { } + protected override IEnumerable LoadVolumes(SourceStream sourceStream) + { + sourceStream.LoadAllParts(); + //stream.Position = 0; + + var streams = sourceStream.Streams.ToList(); + var idx = 0; + if (streams.Count > 1) + { + //check if second stream is zip header without changing position + var headerProbeStream = streams[1]; + var startPosition = headerProbeStream.Position; + headerProbeStream.Position = startPosition + 4; + var isZip = IsZipFile(headerProbeStream, ReaderOptions.Password); + headerProbeStream.Position = startPosition; + if (isZip) + { + sourceStream.IsVolumes = true; + + var tmp = streams[0]; + streams.RemoveAt(0); + streams.Add(tmp); + + return streams.Select(a => new ZipVolume(a, ReaderOptions, idx++)); + } + } + + return new ZipVolume(sourceStream, ReaderOptions, idx++).AsEnumerable(); + } + protected override IEnumerable LoadEntries(IEnumerable volumes) { var vols = volumes.ToArray(); - foreach (var h in headerFactory.ReadSeekableHeader(vols.Last().Stream)) + foreach (var h in headerFactory.NotNull().ReadSeekableHeader(vols.Last().Stream)) { if (h != null) { switch (h.ZipHeaderType) { case ZipHeaderType.DirectoryEntry: - { var deh = (DirectoryEntryHeader)h; Stream s; @@ -244,7 +86,7 @@ public class ZipArchive : AbstractWritableArchive s = new SourceStream( v[0].Stream, i => i < v.Length ? v[i].Stream : null, - new ReaderOptions() { LeaveStreamOpen = true } + ReaderOptions.ForExternalStream ); } else @@ -254,14 +96,20 @@ public class ZipArchive : AbstractWritableArchive yield return new ZipArchiveEntry( this, - new SeekableZipFilePart(headerFactory, deh, s) + new SeekableZipFilePart( + headerFactory.NotNull(), + deh, + s, + ReaderOptions.Providers + ), + ReaderOptions ); } break; case ZipHeaderType.DirectoryEnd: { var bytes = ((DirectoryEndHeader)h).Comment ?? Array.Empty(); - volumes.Last().Comment = ReaderOptions.ArchiveEncoding.Decode(bytes); + vols.Last().Comment = ReaderOptions.ArchiveEncoding.Decode(bytes); yield break; } } @@ -269,37 +117,60 @@ public class ZipArchive : AbstractWritableArchive } } - public void SaveTo(Stream stream) => SaveTo(stream, new WriterOptions(CompressionType.Deflate)); + public void SaveTo(Stream stream) => + SaveTo(stream, new ZipWriterOptions(CompressionType.Deflate)); protected override void SaveTo( Stream stream, - WriterOptions options, + ZipWriterOptions options, IEnumerable oldEntries, IEnumerable newEntries ) { - using var writer = new ZipWriter(stream, new ZipWriterOptions(options)); - foreach (var entry in oldEntries.Concat(newEntries).Where(x => !x.IsDirectory)) + using var writer = new ZipWriter(stream, options); + foreach (var entry in oldEntries.Concat(newEntries)) { - using var entryStream = entry.OpenEntryStream(); - writer.Write(entry.Key, entryStream, entry.LastModifiedTime); + if (entry.IsDirectory) + { + writer.WriteDirectory( + entry.Key.NotNull("Entry Key is null"), + entry.LastModifiedTime + ); + } + else + { + using var entryStream = entry.OpenEntryStream(); + writer.Write( + entry.Key.NotNull("Entry Key is null"), + entryStream, + entry.LastModifiedTime + ); + } } } protected override ZipArchiveEntry CreateEntryInternal( - string filePath, + string key, Stream source, long size, DateTime? modified, bool closeStream - ) => new ZipWritableArchiveEntry(this, source, filePath, size, modified, closeStream); + ) => new ZipWritableArchiveEntry(this, source, key, size, modified, closeStream); - public static ZipArchive Create() => new ZipArchive(); + protected override ZipArchiveEntry CreateDirectoryEntry(string key, DateTime? modified) => + new ZipWritableArchiveEntry(this, key, modified); protected override IReader CreateReaderForSolidExtraction() + { + var stream = Volumes.Single().Stream; + //stream.Position = 0; + return ZipReader.OpenReader(stream, ReaderOptions, Entries); + } + + protected override ValueTask CreateReaderForSolidExtractionAsync() { var stream = Volumes.Single().Stream; stream.Position = 0; - return ZipReader.Open(stream, ReaderOptions, Entries); + return new((IAsyncReader)ZipReader.OpenReader(stream, ReaderOptions, Entries)); } } diff --git a/src/SharpCompress/Archives/Zip/ZipArchiveEntry.Async.cs b/src/SharpCompress/Archives/Zip/ZipArchiveEntry.Async.cs new file mode 100644 index 00000000..308727a6 --- /dev/null +++ b/src/SharpCompress/Archives/Zip/ZipArchiveEntry.Async.cs @@ -0,0 +1,24 @@ +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Zip; + +namespace SharpCompress.Archives.Zip; + +public partial class ZipArchiveEntry +{ + public async ValueTask OpenEntryStreamAsync( + CancellationToken cancellationToken = default + ) + { + var part = Parts.Single(); + if (part is SeekableZipFilePart seekablePart) + { + return ( + await seekablePart.GetCompressedStreamAsync(cancellationToken).ConfigureAwait(false) + ).NotNull(); + } + return OpenEntryStream(); + } +} diff --git a/src/SharpCompress/Archives/Zip/ZipArchiveEntry.cs b/src/SharpCompress/Archives/Zip/ZipArchiveEntry.cs index a94ed2c6..b4f9e8bd 100644 --- a/src/SharpCompress/Archives/Zip/ZipArchiveEntry.cs +++ b/src/SharpCompress/Archives/Zip/ZipArchiveEntry.cs @@ -1,15 +1,22 @@ -using System.IO; +using System.IO; using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Options; using SharpCompress.Common.Zip; namespace SharpCompress.Archives.Zip; -public class ZipArchiveEntry : ZipEntry, IArchiveEntry +public partial class ZipArchiveEntry : ZipEntry, IArchiveEntry { - internal ZipArchiveEntry(ZipArchive archive, SeekableZipFilePart? part) - : base(part) => Archive = archive; + internal ZipArchiveEntry( + ZipArchive archive, + SeekableZipFilePart? part, + IReaderOptions readerOptions + ) + : base(part, readerOptions) => Archive = archive; - public virtual Stream OpenEntryStream() => Parts.Single().GetCompressedStream(); + public virtual Stream OpenEntryStream() => Parts.Single().GetCompressedStream().NotNull(); #region IArchiveEntry Members @@ -18,6 +25,4 @@ public class ZipArchiveEntry : ZipEntry, IArchiveEntry public bool IsComplete => true; #endregion - - public string? Comment => ((SeekableZipFilePart)Parts.Single()).Comment; } diff --git a/src/SharpCompress/Archives/Zip/ZipArchiveVolumeFactory.cs b/src/SharpCompress/Archives/Zip/ZipArchiveVolumeFactory.cs index fdaa85b9..ca2f0304 100644 --- a/src/SharpCompress/Archives/Zip/ZipArchiveVolumeFactory.cs +++ b/src/SharpCompress/Archives/Zip/ZipArchiveVolumeFactory.cs @@ -1,6 +1,7 @@ using System; using System.IO; using System.Text.RegularExpressions; +using SharpCompress.Common; namespace SharpCompress.Archives.Zip; @@ -12,23 +13,29 @@ internal static class ZipArchiveVolumeFactory //load files with zip/zipx first. Swapped to end once loaded in ZipArchive //new style .zip, z01.. | .zipx, zx01 - if the numbers go beyond 99 then they use 100 ...1000 etc - Match m = Regex.Match(part1.Name, @"^(.*\.)(zipx?|zx?[0-9]+)$", RegexOptions.IgnoreCase); + var m = Regex.Match(part1.Name, @"^(.*\.)(zipx?|zx?[0-9]+)$", RegexOptions.IgnoreCase); if (m.Success) + { item = new FileInfo( Path.Combine( part1.DirectoryName!, String.Concat( m.Groups[1].Value, Regex.Replace(m.Groups[2].Value, @"[^xz]", ""), - index.ToString().PadLeft(2, '0') + index.ToString(Constants.DefaultCultureInfo).PadLeft(2, '0') ) ) ); + } else //split - 001, 002 ... + { return ArchiveVolumeFactory.GetFilePart(index, part1); + } if (item != null && item.Exists) + { return item; + } return null; //no more items } diff --git a/src/SharpCompress/Archives/Zip/ZipWritableArchiveEntry.cs b/src/SharpCompress/Archives/Zip/ZipWritableArchiveEntry.cs index 628e505f..b74fea04 100644 --- a/src/SharpCompress/Archives/Zip/ZipWritableArchiveEntry.cs +++ b/src/SharpCompress/Archives/Zip/ZipWritableArchiveEntry.cs @@ -9,7 +9,8 @@ namespace SharpCompress.Archives.Zip; internal class ZipWritableArchiveEntry : ZipArchiveEntry, IWritableArchiveEntry { private readonly bool closeStream; - private readonly Stream stream; + private readonly Stream? stream; + private readonly bool isDirectory; private bool isDisposed; internal ZipWritableArchiveEntry( @@ -20,13 +21,29 @@ internal class ZipWritableArchiveEntry : ZipArchiveEntry, IWritableArchiveEntry DateTime? lastModified, bool closeStream ) - : base(archive, null) + : base(archive, null, archive.ReaderOptions) { this.stream = stream; Key = path; Size = size; LastModifiedTime = lastModified; this.closeStream = closeStream; + isDirectory = false; + } + + internal ZipWritableArchiveEntry( + ZipArchive archive, + string directoryPath, + DateTime? lastModified + ) + : base(archive, null, archive.ReaderOptions) + { + stream = null; + Key = directoryPath; + Size = 0; + LastModifiedTime = lastModified; + closeStream = false; + isDirectory = true; } public override long Crc => 0; @@ -47,24 +64,28 @@ internal class ZipWritableArchiveEntry : ZipArchiveEntry, IWritableArchiveEntry public override bool IsEncrypted => false; - public override bool IsDirectory => false; + public override bool IsDirectory => isDirectory; public override bool IsSplitAfter => false; internal override IEnumerable Parts => throw new NotImplementedException(); - Stream IWritableArchiveEntry.Stream => stream; + Stream IWritableArchiveEntry.Stream => stream ?? Stream.Null; public override Stream OpenEntryStream() { + if (stream is null) + { + return Stream.Null; + } //ensure new stream is at the start, this could be reset stream.Seek(0, SeekOrigin.Begin); - return NonDisposingStream.Create(stream); + return SharpCompressStream.CreateNonDisposing(stream); } internal override void Close() { - if (closeStream && !isDisposed) + if (closeStream && !isDisposed && stream is not null) { stream.Dispose(); isDisposed = true; diff --git a/src/SharpCompress/AssemblyInfo.cs b/src/SharpCompress/AssemblyInfo.cs index 0270020d..c11eb8e0 100644 --- a/src/SharpCompress/AssemblyInfo.cs +++ b/src/SharpCompress/AssemblyInfo.cs @@ -1,3 +1,8 @@ using System; +using System.Runtime.CompilerServices; -[assembly: CLSCompliant(true)] +// CLSCompliant(false) is required because ZStandard integration uses unsafe code +[assembly: CLSCompliant(false)] +[assembly: InternalsVisibleTo( + "SharpCompress.Test,PublicKey=0024000004800000940000000602000000240000525341310004000001000100158bebf1433f76dffc356733c138babea7a47536c65ed8009b16372c6f4edbb20554db74a62687f56b97c20a6ce8c4b123280279e33c894e7b3aa93ab3c573656fde4db576cfe07dba09619ead26375b25d2c4a8e43f7be257d712b0dd2eb546f67adb09281338618a58ac834fc038dd7e2740a7ab3591826252e4f4516306dc" +)] diff --git a/src/SharpCompress/Common/Ace/AceCrc.cs b/src/SharpCompress/Common/Ace/AceCrc.cs new file mode 100644 index 00000000..bbd51d92 --- /dev/null +++ b/src/SharpCompress/Common/Ace/AceCrc.cs @@ -0,0 +1,64 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text; +using System.Threading.Tasks; + +namespace SharpCompress.Common.Ace; + +public class AceCrc +{ + // CRC-32 lookup table (standard polynomial 0xEDB88320, reflected) + private static readonly uint[] Crc32Table = GenerateTable(); + + private static uint[] GenerateTable() + { + var table = new uint[256]; + + for (int i = 0; i < 256; i++) + { + uint crc = (uint)i; + + for (int j = 0; j < 8; j++) + { + if ((crc & 1) != 0) + { + crc = (crc >> 1) ^ 0xEDB88320u; + } + else + { + crc >>= 1; + } + } + + table[i] = crc; + } + + return table; + } + + /// + /// Calculate ACE CRC-32 checksum. + /// ACE CRC-32 uses standard CRC-32 polynomial (0xEDB88320, reflected) + /// with init=0xFFFFFFFF but NO final XOR. + /// + public static uint AceCrc32(ReadOnlySpan data) + { + uint crc = 0xFFFFFFFFu; + + foreach (byte b in data) + { + crc = (crc >> 8) ^ Crc32Table[(crc ^ b) & 0xFF]; + } + + return crc; // No final XOR for ACE + } + + /// + /// ACE CRC-16 is the lower 16 bits of the ACE CRC-32. + /// + public static ushort AceCrc16(ReadOnlySpan data) + { + return (ushort)(AceCrc32(data) & 0xFFFF); + } +} diff --git a/src/SharpCompress/Common/Ace/AceEntry.cs b/src/SharpCompress/Common/Ace/AceEntry.cs new file mode 100644 index 00000000..0164d229 --- /dev/null +++ b/src/SharpCompress/Common/Ace/AceEntry.cs @@ -0,0 +1,77 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using SharpCompress.Common.Ace.Headers; +using SharpCompress.Common.Options; + +namespace SharpCompress.Common.Ace; + +public class AceEntry : Entry +{ + private readonly AceFilePart _filePart; + + internal AceEntry(AceFilePart filePart, IReaderOptions readerOptions) + : base(readerOptions) + { + _filePart = filePart; + } + + public override long Crc + { + get + { + if (_filePart == null) + { + return 0; + } + return _filePart.Header.Crc32; + } + } + + internal override ChecksumDescriptor Checksum => + !IsDirectory + && !IsEncrypted + && !_filePart.Header.IsContinuedFromPrev + && !_filePart.Header.IsContinuedToNext + ? new ChecksumDescriptor(ChecksumKind.Crc32NoFinalXor, _filePart.Header.Crc32, true) + : default; + + public override string? Key => _filePart?.Header.Filename; + + public override string? LinkTarget => null; + + public override long CompressedSize => _filePart?.Header.PackedSize ?? 0; + + public override CompressionType CompressionType + { + get + { + if (_filePart.Header.CompressionType == Headers.CompressionType.Stored) + { + return CompressionType.None; + } + return CompressionType.AceLZ77; + } + } + + public override long Size => _filePart?.Header.OriginalSize ?? 0; + + public override DateTime? LastModifiedTime => _filePart.Header.DateTime; + + public override DateTime? CreatedTime => null; + + public override DateTime? LastAccessedTime => null; + + public override DateTime? ArchivedTime => null; + + public override bool IsEncrypted => _filePart.Header.IsFileEncrypted; + + public override bool IsDirectory => _filePart.Header.IsDirectory; + + public override bool IsSplitAfter => false; + + internal override IEnumerable Parts => _filePart.Empty(); +} diff --git a/src/SharpCompress/Common/Ace/AceFilePart.cs b/src/SharpCompress/Common/Ace/AceFilePart.cs new file mode 100644 index 00000000..a56c17f6 --- /dev/null +++ b/src/SharpCompress/Common/Ace/AceFilePart.cs @@ -0,0 +1,51 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using SharpCompress.Common.Ace.Headers; +using SharpCompress.IO; + +namespace SharpCompress.Common.Ace; + +public class AceFilePart : FilePart +{ + private readonly Stream _stream; + internal AceFileHeader Header { get; set; } + + internal AceFilePart(AceFileHeader localAceHeader, Stream seekableStream) + : base(localAceHeader.ArchiveEncoding) + { + _stream = seekableStream; + Header = localAceHeader; + } + + internal override string? FilePartName => Header.Filename; + + internal override Stream GetCompressedStream() + { + if (_stream != null) + { + Stream compressedStream; + switch (Header.CompressionType) + { + case Headers.CompressionType.Stored: + compressedStream = new ReadOnlySubStream( + _stream, + Header.DataStartPosition, + Header.PackedSize + ); + break; + default: + throw new NotSupportedException( + "CompressionMethod: " + Header.CompressionQuality + ); + } + return compressedStream; + } + return _stream.NotNull(); + } + + internal override Stream? GetRawStream() => _stream; +} diff --git a/src/SharpCompress/Common/Ace/AceVolume.cs b/src/SharpCompress/Common/Ace/AceVolume.cs new file mode 100644 index 00000000..f3931d2c --- /dev/null +++ b/src/SharpCompress/Common/Ace/AceVolume.cs @@ -0,0 +1,34 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using SharpCompress.Common.Arj; +using SharpCompress.Readers; + +namespace SharpCompress.Common.Ace; + +public class AceVolume : Volume +{ + public AceVolume(Stream stream, ReaderOptions readerOptions, int index = 0) + : base(stream, readerOptions, index) { } + + public override bool IsFirstVolume + { + get { return true; } + } + + /// + /// ArjArchive is part of a multi-part archive. + /// + public override bool IsMultiVolume + { + get { return false; } + } + + internal IEnumerable GetVolumeFileParts() + { + return new List(); + } +} diff --git a/src/SharpCompress/Common/Ace/Headers/AceFileHeader.Async.cs b/src/SharpCompress/Common/Ace/Headers/AceFileHeader.Async.cs new file mode 100644 index 00000000..4f4aecbf --- /dev/null +++ b/src/SharpCompress/Common/Ace/Headers/AceFileHeader.Async.cs @@ -0,0 +1,111 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Arc; + +namespace SharpCompress.Common.Ace.Headers; + +public sealed partial class AceFileHeader +{ + /// + /// Asynchronously reads the next file entry header from the stream. + /// Returns null if no more entries or end of archive. + /// Supports both ACE 1.0 and ACE 2.0 formats. + /// + public override async ValueTask ReadAsync( + Stream reader, + CancellationToken cancellationToken = default + ) + { + var headerData = await ReadHeaderAsync(reader, cancellationToken).ConfigureAwait(false); + if (headerData.Length == 0) + { + return null; + } + int offset = 0; + + // Header type (1 byte) + HeaderType = headerData[offset++]; + + // Skip recovery record headers (ACE 2.0 feature) + if (HeaderType == (byte)SharpCompress.Common.Ace.Headers.AceHeaderType.RECOVERY32) + { + // Skip to next header + return null; + } + + if (HeaderType != (byte)SharpCompress.Common.Ace.Headers.AceHeaderType.FILE) + { + // Unknown header type - skip + return null; + } + + // Header flags (2 bytes) + HeaderFlags = BitConverter.ToUInt16(headerData, offset); + offset += 2; + + // Packed size (4 bytes) + PackedSize = BitConverter.ToUInt32(headerData, offset); + offset += 4; + + // Original size (4 bytes) + OriginalSize = BitConverter.ToUInt32(headerData, offset); + offset += 4; + + // File date/time in DOS format (4 bytes) + var dosDateTime = BitConverter.ToUInt32(headerData, offset); + DateTime = ConvertDosDateTime(dosDateTime); + offset += 4; + + // File attributes (4 bytes) + Attributes = (int)BitConverter.ToUInt32(headerData, offset); + offset += 4; + + // CRC32 (4 bytes) + Crc32 = BitConverter.ToUInt32(headerData, offset); + offset += 4; + + // Compression type (1 byte) + byte compressionType = headerData[offset++]; + CompressionType = GetCompressionType(compressionType); + + // Compression quality/parameter (1 byte) + byte compressionQuality = headerData[offset++]; + CompressionQuality = GetCompressionQuality(compressionQuality); + + // Parameters (2 bytes) + Parameters = BitConverter.ToUInt16(headerData, offset); + offset += 2; + + // Reserved (2 bytes) - skip + offset += 2; + + // Filename length (2 bytes) + var filenameLength = BitConverter.ToUInt16(headerData, offset); + offset += 2; + + // Filename + if (offset + filenameLength <= headerData.Length) + { + Filename = ArchiveEncoding.Decode(headerData, offset, filenameLength); + offset += filenameLength; + } + + // Handle comment if present + if ((HeaderFlags & SharpCompress.Common.Ace.Headers.HeaderFlags.COMMENT) != 0) + { + // Comment length (2 bytes) + if (offset + 2 <= headerData.Length) + { + ushort commentLength = BitConverter.ToUInt16(headerData, offset); + offset += 2 + commentLength; // Skip comment + } + } + + // Store the data start position + DataStartPosition = reader.Position; + + return this; + } +} diff --git a/src/SharpCompress/Common/Ace/Headers/AceFileHeader.cs b/src/SharpCompress/Common/Ace/Headers/AceFileHeader.cs new file mode 100644 index 00000000..1c71d18d --- /dev/null +++ b/src/SharpCompress/Common/Ace/Headers/AceFileHeader.cs @@ -0,0 +1,174 @@ +using System; +using System.Buffers.Binary; +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using System.Xml.Linq; +using SharpCompress.Common.Arc; + +namespace SharpCompress.Common.Ace.Headers; + +/// +/// ACE file entry header +/// +public sealed partial class AceFileHeader : AceHeader +{ + public long DataStartPosition { get; private set; } + public long PackedSize { get; set; } + public long OriginalSize { get; set; } + public DateTime DateTime { get; set; } + public int Attributes { get; set; } + public uint Crc32 { get; set; } + public CompressionType CompressionType { get; set; } + public CompressionQuality CompressionQuality { get; set; } + public ushort Parameters { get; set; } + public string Filename { get; set; } = string.Empty; + public List Comment { get; set; } = new(); + + /// + /// File data offset in the archive + /// + public ulong DataOffset { get; set; } + + public bool IsDirectory => (Attributes & 0x10) != 0; + + public bool IsContinuedFromPrev => + (HeaderFlags & SharpCompress.Common.Ace.Headers.HeaderFlags.CONTINUED_PREV) != 0; + + public bool IsContinuedToNext => + (HeaderFlags & SharpCompress.Common.Ace.Headers.HeaderFlags.CONTINUED_NEXT) != 0; + + public int DictionarySize + { + get + { + int bits = Parameters & 0x0F; + return bits < 10 ? 1024 : 1 << bits; + } + } + + public AceFileHeader(IArchiveEncoding archiveEncoding) + : base(archiveEncoding, AceHeaderType.FILE) { } + + /// + /// Reads the next file entry header from the stream. + /// Returns null if no more entries or end of archive. + /// Supports both ACE 1.0 and ACE 2.0 formats. + /// + public override AceHeader? Read(Stream reader) + { + var headerData = ReadHeader(reader); + if (headerData.Length == 0) + { + return null; + } + int offset = 0; + + // Header type (1 byte) + HeaderType = headerData[offset++]; + + // Skip recovery record headers (ACE 2.0 feature) + if (HeaderType == (byte)SharpCompress.Common.Ace.Headers.AceHeaderType.RECOVERY32) + { + // Skip to next header + return null; + } + + if (HeaderType != (byte)SharpCompress.Common.Ace.Headers.AceHeaderType.FILE) + { + // Unknown header type - skip + return null; + } + + // Header flags (2 bytes) + HeaderFlags = BitConverter.ToUInt16(headerData, offset); + offset += 2; + + // Packed size (4 bytes) + PackedSize = BitConverter.ToUInt32(headerData, offset); + offset += 4; + + // Original size (4 bytes) + OriginalSize = BitConverter.ToUInt32(headerData, offset); + offset += 4; + + // File date/time in DOS format (4 bytes) + var dosDateTime = BitConverter.ToUInt32(headerData, offset); + DateTime = ConvertDosDateTime(dosDateTime); + offset += 4; + + // File attributes (4 bytes) + Attributes = (int)BitConverter.ToUInt32(headerData, offset); + offset += 4; + + // CRC32 (4 bytes) + Crc32 = BitConverter.ToUInt32(headerData, offset); + offset += 4; + + // Compression type (1 byte) + byte compressionType = headerData[offset++]; + CompressionType = GetCompressionType(compressionType); + + // Compression quality/parameter (1 byte) + byte compressionQuality = headerData[offset++]; + CompressionQuality = GetCompressionQuality(compressionQuality); + + // Parameters (2 bytes) + Parameters = BitConverter.ToUInt16(headerData, offset); + offset += 2; + + // Reserved (2 bytes) - skip + offset += 2; + + // Filename length (2 bytes) + var filenameLength = BitConverter.ToUInt16(headerData, offset); + offset += 2; + + // Filename + if (offset + filenameLength <= headerData.Length) + { + Filename = ArchiveEncoding.Decode(headerData, offset, filenameLength); + offset += filenameLength; + } + + // Handle comment if present + if ((HeaderFlags & SharpCompress.Common.Ace.Headers.HeaderFlags.COMMENT) != 0) + { + // Comment length (2 bytes) + if (offset + 2 <= headerData.Length) + { + ushort commentLength = BitConverter.ToUInt16(headerData, offset); + offset += 2 + commentLength; // Skip comment + } + } + + // Store the data start position + DataStartPosition = reader.Position; + + return this; + } + + // ReadAsync moved to AceFileHeader.Async.cs + + public CompressionType GetCompressionType(byte value) => + value switch + { + 0 => CompressionType.Stored, + 1 => CompressionType.Lz77, + 2 => CompressionType.Blocked, + _ => CompressionType.Unknown, + }; + + public CompressionQuality GetCompressionQuality(byte value) => + value switch + { + 0 => CompressionQuality.None, + 1 => CompressionQuality.Fastest, + 2 => CompressionQuality.Fast, + 3 => CompressionQuality.Normal, + 4 => CompressionQuality.Good, + 5 => CompressionQuality.Best, + _ => CompressionQuality.Unknown, + }; +} diff --git a/src/SharpCompress/Common/Ace/Headers/AceHeader.Async.cs b/src/SharpCompress/Common/Ace/Headers/AceHeader.Async.cs new file mode 100644 index 00000000..86cf2d01 --- /dev/null +++ b/src/SharpCompress/Common/Ace/Headers/AceHeader.Async.cs @@ -0,0 +1,75 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Common.Ace.Headers; + +public abstract partial class AceHeader +{ + public abstract ValueTask ReadAsync( + Stream reader, + CancellationToken cancellationToken = default + ); + + public async ValueTask ReadHeaderAsync( + Stream stream, + CancellationToken cancellationToken = default + ) + { + // Read header CRC (2 bytes) and header size (2 bytes) + var headerBytes = new byte[4]; + if ( + !await stream.ReadFullyAsync(headerBytes, 0, 4, cancellationToken).ConfigureAwait(false) + ) + { + return Array.Empty(); + } + + HeaderCrc = BitConverter.ToUInt16(headerBytes, 0); // CRC for validation + HeaderSize = BitConverter.ToUInt16(headerBytes, 2); + if (HeaderSize == 0) + { + return Array.Empty(); + } + + // Read the header data + var body = new byte[HeaderSize]; + if ( + !await stream + .ReadFullyAsync(body, 0, HeaderSize, cancellationToken) + .ConfigureAwait(false) + ) + { + return Array.Empty(); + } + + // Verify crc + var checksum = AceCrc.AceCrc16(body); + if (checksum != HeaderCrc) + { + throw new InvalidFormatException("Header checksum is invalid"); + } + return body; + } + + /// + /// Asynchronously checks if the stream is an ACE archive + /// + /// The stream to read from + /// Cancellation token + /// True if the stream is an ACE archive, false otherwise + public static async ValueTask IsArchiveAsync( + Stream stream, + CancellationToken cancellationToken = default + ) + { + var bytes = new byte[14]; + if (!await stream.ReadFullyAsync(bytes, 0, 14, cancellationToken).ConfigureAwait(false)) + { + return false; + } + + return CheckMagicBytes(bytes, 7); + } +} diff --git a/src/SharpCompress/Common/Ace/Headers/AceHeader.cs b/src/SharpCompress/Common/Ace/Headers/AceHeader.cs new file mode 100644 index 00000000..8d270558 --- /dev/null +++ b/src/SharpCompress/Common/Ace/Headers/AceHeader.cs @@ -0,0 +1,156 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Arj.Headers; +using SharpCompress.Crypto; + +namespace SharpCompress.Common.Ace.Headers; + +/// +/// Header type constants +/// +public enum AceHeaderType +{ + MAIN = 0, + FILE = 1, + RECOVERY32 = 2, + RECOVERY64A = 3, + RECOVERY64B = 4, +} + +public abstract partial class AceHeader +{ + // ACE signature: bytes at offset 7 should be "**ACE**" + private static readonly byte[] AceSignature = + [ + (byte)'*', + (byte)'*', + (byte)'A', + (byte)'C', + (byte)'E', + (byte)'*', + (byte)'*', + ]; + + public AceHeader(IArchiveEncoding archiveEncoding, AceHeaderType type) + { + AceHeaderType = type; + ArchiveEncoding = archiveEncoding; + } + + public IArchiveEncoding ArchiveEncoding { get; } + public AceHeaderType AceHeaderType { get; } + + public ushort HeaderFlags { get; set; } + public ushort HeaderCrc { get; set; } + public ushort HeaderSize { get; set; } + public byte HeaderType { get; set; } + + public bool IsFileEncrypted => + (HeaderFlags & SharpCompress.Common.Ace.Headers.HeaderFlags.FILE_ENCRYPTED) != 0; + public bool Is64Bit => + (HeaderFlags & SharpCompress.Common.Ace.Headers.HeaderFlags.MEMORY_64BIT) != 0; + + public bool IsSolid => + (HeaderFlags & SharpCompress.Common.Ace.Headers.HeaderFlags.SOLID_MAIN) != 0; + + public bool IsMultiVolume => + (HeaderFlags & SharpCompress.Common.Ace.Headers.HeaderFlags.MULTIVOLUME) != 0; + + public abstract AceHeader? Read(Stream reader); + + // Async methods moved to AceHeader.Async.cs + + public byte[] ReadHeader(Stream stream) + { + // Read header CRC (2 bytes) and header size (2 bytes) + var headerBytes = new byte[4]; + if (!stream.ReadFully(headerBytes)) + { + return Array.Empty(); + } + + HeaderCrc = BitConverter.ToUInt16(headerBytes, 0); // CRC for validation + HeaderSize = BitConverter.ToUInt16(headerBytes, 2); + if (HeaderSize == 0) + { + return Array.Empty(); + } + + // Read the header data + var body = new byte[HeaderSize]; + if (!stream.ReadFully(body)) + { + return Array.Empty(); + } + + // Verify crc + var checksum = AceCrc.AceCrc16(body); + if (checksum != HeaderCrc) + { + throw new InvalidFormatException("Header checksum is invalid"); + } + return body; + } + + public static bool IsArchive(Stream stream) + { + // ACE files have a specific signature + // First two bytes are typically 0x60 0xEA (signature bytes) + // At offset 7, there should be "**ACE**" (7 bytes) + var bytes = new byte[14]; + if (stream.Read(bytes, 0, 14) != 14) + { + return false; + } + + // Check for "**ACE**" at offset 7 + return CheckMagicBytes(bytes, 7); + } + + protected static bool CheckMagicBytes(byte[] headerBytes, int offset) + { + // Check for "**ACE**" at specified offset + for (int i = 0; i < AceSignature.Length; i++) + { + if (headerBytes[offset + i] != AceSignature[i]) + { + return false; + } + } + return true; + } + + protected DateTime ConvertDosDateTime(uint dosDateTime) + { + try + { + int second = (int)(dosDateTime & 0x1F) * 2; + int minute = (int)((dosDateTime >> 5) & 0x3F); + int hour = (int)((dosDateTime >> 11) & 0x1F); + int day = (int)((dosDateTime >> 16) & 0x1F); + int month = (int)((dosDateTime >> 21) & 0x0F); + int year = (int)((dosDateTime >> 25) & 0x7F) + 1980; + + if ( + day < 1 + || day > 31 + || month < 1 + || month > 12 + || hour > 23 + || minute > 59 + || second > 59 + ) + { + return DateTime.MinValue; + } + + return new DateTime(year, month, day, hour, minute, second); + } + catch + { + return DateTime.MinValue; + } + } +} diff --git a/src/SharpCompress/Common/Ace/Headers/AceMainHeader.Async.cs b/src/SharpCompress/Common/Ace/Headers/AceMainHeader.Async.cs new file mode 100644 index 00000000..39205945 --- /dev/null +++ b/src/SharpCompress/Common/Ace/Headers/AceMainHeader.Async.cs @@ -0,0 +1,83 @@ +using System; +using System.Buffers.Binary; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Crypto; + +namespace SharpCompress.Common.Ace.Headers; + +public sealed partial class AceMainHeader +{ + /// + /// Asynchronously reads the main archive header from the stream. + /// Returns header if this is a valid ACE archive. + /// Supports both ACE 1.0 and ACE 2.0 formats. + /// + public override async ValueTask ReadAsync( + Stream reader, + CancellationToken cancellationToken = default + ) + { + var headerData = await ReadHeaderAsync(reader, cancellationToken).ConfigureAwait(false); + if (headerData.Length == 0) + { + return null; + } + int offset = 0; + + // Header type should be 0 for main header + if (headerData[offset++] != HeaderType) + { + return null; + } + + // Header flags (2 bytes) + HeaderFlags = BitConverter.ToUInt16(headerData, offset); + offset += 2; + + // Skip signature "**ACE**" (7 bytes) + if (!CheckMagicBytes(headerData, offset)) + { + throw new InvalidFormatException("Invalid ACE archive signature."); + } + offset += 7; + + // ACE version (1 byte) - 10 for ACE 1.0, 20 for ACE 2.0 + AceVersion = headerData[offset++]; + ExtractVersion = headerData[offset++]; + + // Host OS (1 byte) + if (offset < headerData.Length) + { + var hostOsByte = headerData[offset++]; + HostOS = hostOsByte <= 11 ? (HostOS)hostOsByte : HostOS.Unknown; + } + // Volume number (1 byte) + VolumeNumber = headerData[offset++]; + + // Creation date/time (4 bytes) + var dosDateTime = BitConverter.ToUInt32(headerData, offset); + DateTime = ConvertDosDateTime(dosDateTime); + offset += 4; + + // Reserved fields (8 bytes) + if (offset + 8 <= headerData.Length) + { + offset += 8; + } + + // Skip additional fields based on flags + // Handle comment if present + if ((HeaderFlags & SharpCompress.Common.Ace.Headers.HeaderFlags.COMMENT) != 0) + { + if (offset + 2 <= headerData.Length) + { + ushort commentLength = BitConverter.ToUInt16(headerData, offset); + offset += 2 + commentLength; + } + } + + return this; + } +} diff --git a/src/SharpCompress/Common/Ace/Headers/AceMainHeader.cs b/src/SharpCompress/Common/Ace/Headers/AceMainHeader.cs new file mode 100644 index 00000000..e0083c22 --- /dev/null +++ b/src/SharpCompress/Common/Ace/Headers/AceMainHeader.cs @@ -0,0 +1,100 @@ +using System; +using System.Buffers.Binary; +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Ace.Headers; +using SharpCompress.Common.Zip.Headers; +using SharpCompress.Crypto; + +namespace SharpCompress.Common.Ace.Headers; + +/// +/// ACE main archive header +/// +public sealed partial class AceMainHeader : AceHeader +{ + public byte ExtractVersion { get; set; } + public byte CreatorVersion { get; set; } + public HostOS HostOS { get; set; } + public byte VolumeNumber { get; set; } + public DateTime DateTime { get; set; } + public string Advert { get; set; } = string.Empty; + public List Comment { get; set; } = new(); + public byte AceVersion { get; private set; } + + public AceMainHeader(IArchiveEncoding archiveEncoding) + : base(archiveEncoding, AceHeaderType.MAIN) { } + + /// + /// Reads the main archive header from the stream. + /// Returns header if this is a valid ACE archive. + /// Supports both ACE 1.0 and ACE 2.0 formats. + /// + public override AceHeader? Read(Stream reader) + { + var headerData = ReadHeader(reader); + if (headerData.Length == 0) + { + return null; + } + int offset = 0; + + // Header type should be 0 for main header + if (headerData[offset++] != HeaderType) + { + return null; + } + + // Header flags (2 bytes) + HeaderFlags = BitConverter.ToUInt16(headerData, offset); + offset += 2; + + // Skip signature "**ACE**" (7 bytes) + if (!CheckMagicBytes(headerData, offset)) + { + throw new InvalidFormatException("Invalid ACE archive signature."); + } + offset += 7; + + // ACE version (1 byte) - 10 for ACE 1.0, 20 for ACE 2.0 + AceVersion = headerData[offset++]; + ExtractVersion = headerData[offset++]; + + // Host OS (1 byte) + if (offset < headerData.Length) + { + var hostOsByte = headerData[offset++]; + HostOS = hostOsByte <= 11 ? (HostOS)hostOsByte : HostOS.Unknown; + } + // Volume number (1 byte) + VolumeNumber = headerData[offset++]; + + // Creation date/time (4 bytes) + var dosDateTime = BitConverter.ToUInt32(headerData, offset); + DateTime = ConvertDosDateTime(dosDateTime); + offset += 4; + + // Reserved fields (8 bytes) + if (offset + 8 <= headerData.Length) + { + offset += 8; + } + + // Skip additional fields based on flags + // Handle comment if present + if ((HeaderFlags & SharpCompress.Common.Ace.Headers.HeaderFlags.COMMENT) != 0) + { + if (offset + 2 <= headerData.Length) + { + ushort commentLength = BitConverter.ToUInt16(headerData, offset); + offset += 2 + commentLength; + } + } + + return this; + } + + // ReadAsync moved to AceMainHeader.Async.cs +} diff --git a/src/SharpCompress/Common/Ace/Headers/CompressionQuality.cs b/src/SharpCompress/Common/Ace/Headers/CompressionQuality.cs new file mode 100644 index 00000000..eb53d639 --- /dev/null +++ b/src/SharpCompress/Common/Ace/Headers/CompressionQuality.cs @@ -0,0 +1,15 @@ +namespace SharpCompress.Common.Ace.Headers; + +/// +/// Compression quality +/// +public enum CompressionQuality +{ + None, + Fastest, + Fast, + Normal, + Good, + Best, + Unknown, +} diff --git a/src/SharpCompress/Common/Ace/Headers/CompressionType.cs b/src/SharpCompress/Common/Ace/Headers/CompressionType.cs new file mode 100644 index 00000000..f86fa080 --- /dev/null +++ b/src/SharpCompress/Common/Ace/Headers/CompressionType.cs @@ -0,0 +1,12 @@ +namespace SharpCompress.Common.Ace.Headers; + +/// +/// Compression types +/// +public enum CompressionType +{ + Stored, + Lz77, + Blocked, + Unknown, +} diff --git a/src/SharpCompress/Common/Ace/Headers/HeaderFlags.cs b/src/SharpCompress/Common/Ace/Headers/HeaderFlags.cs new file mode 100644 index 00000000..b3d67898 --- /dev/null +++ b/src/SharpCompress/Common/Ace/Headers/HeaderFlags.cs @@ -0,0 +1,32 @@ +namespace SharpCompress.Common.Ace.Headers; + +/// +/// Header flags (main + file, overlapping meanings) +/// +public static class HeaderFlags +{ + // Shared / low bits + public const ushort ADDSIZE = 0x0001; // extra size field present + public const ushort COMMENT = 0x0002; // comment present + public const ushort MEMORY_64BIT = 0x0004; + public const ushort AV_STRING = 0x0008; // AV string present + public const ushort SOLID = 0x0010; // solid file + public const ushort LOCKED = 0x0020; + public const ushort PROTECTED = 0x0040; + + // Main header specific + public const ushort V20FORMAT = 0x0100; + public const ushort SFX = 0x0200; + public const ushort LIMITSFXJR = 0x0400; + public const ushort MULTIVOLUME = 0x0800; + public const ushort ADVERT = 0x1000; + public const ushort RECOVERY = 0x2000; + public const ushort LOCKED_MAIN = 0x4000; + public const ushort SOLID_MAIN = 0x8000; + + // File header specific (same bits, different meaning) + public const ushort NTSECURITY = 0x0400; + public const ushort CONTINUED_PREV = 0x1000; + public const ushort CONTINUED_NEXT = 0x2000; + public const ushort FILE_ENCRYPTED = 0x4000; // file encrypted (file header) +} diff --git a/src/SharpCompress/Common/Ace/Headers/HostOS.cs b/src/SharpCompress/Common/Ace/Headers/HostOS.cs new file mode 100644 index 00000000..d58d30c0 --- /dev/null +++ b/src/SharpCompress/Common/Ace/Headers/HostOS.cs @@ -0,0 +1,21 @@ +namespace SharpCompress.Common.Ace.Headers; + +/// +/// Host OS type +/// +public enum HostOS +{ + MsDos = 0, + Os2, + Windows, + Unix, + MacOs, + WinNt, + Primos, + AppleGs, + Atari, + Vax, + Amiga, + Next, + Unknown, +} diff --git a/src/SharpCompress/Common/Arc/ArcEntry.cs b/src/SharpCompress/Common/Arc/ArcEntry.cs new file mode 100644 index 00000000..9627606d --- /dev/null +++ b/src/SharpCompress/Common/Arc/ArcEntry.cs @@ -0,0 +1,66 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using SharpCompress.Common.GZip; +using SharpCompress.Common.Options; +using SharpCompress.Common.Tar; + +namespace SharpCompress.Common.Arc; + +public class ArcEntry : Entry +{ + private readonly ArcFilePart? _filePart; + + internal ArcEntry(ArcFilePart? filePart, IReaderOptions readerOptions) + : base(readerOptions) + { + _filePart = filePart; + } + + public override long Crc + { + get + { + if (_filePart == null) + { + return 0; + } + return _filePart.Header.Crc16; + } + } + + internal override ChecksumDescriptor Checksum => + _filePart is not null && _filePart.Header.CompressionMethod != CompressionType.Unknown + ? new ChecksumDescriptor(ChecksumKind.Crc16Arc, _filePart.Header.Crc16, true) + : default; + + public override string? Key => _filePart?.Header.Name; + + public override string? LinkTarget => null; + + public override long CompressedSize => _filePart?.Header.CompressedSize ?? 0; + + public override CompressionType CompressionType => + _filePart?.Header.CompressionMethod ?? CompressionType.Unknown; + + public override long Size => throw new NotImplementedException(); + + public override DateTime? LastModifiedTime => null; + + public override DateTime? CreatedTime => null; + + public override DateTime? LastAccessedTime => null; + + public override DateTime? ArchivedTime => null; + + public override bool IsEncrypted => false; + + public override bool IsDirectory => false; + + public override bool IsSplitAfter => false; + + internal override IEnumerable Parts => _filePart.Empty(); +} diff --git a/src/SharpCompress/Common/Arc/ArcEntryHeader.cs b/src/SharpCompress/Common/Arc/ArcEntryHeader.cs new file mode 100644 index 00000000..983b1b78 --- /dev/null +++ b/src/SharpCompress/Common/Arc/ArcEntryHeader.cs @@ -0,0 +1,95 @@ +using System; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Common.Arc; + +public class ArcEntryHeader +{ + public IArchiveEncoding ArchiveEncoding { get; } + public CompressionType CompressionMethod { get; private set; } + public string? Name { get; private set; } + public long CompressedSize { get; private set; } + public DateTime DateTime { get; private set; } + public int Crc16 { get; private set; } + public long OriginalSize { get; private set; } + public long DataStartPosition { get; private set; } + + public ArcEntryHeader(IArchiveEncoding archiveEncoding) + { + this.ArchiveEncoding = archiveEncoding; + } + + public ArcEntryHeader? ReadHeader(Stream stream) + { + byte[] headerBytes = new byte[29]; + if (stream.Read(headerBytes, 0, headerBytes.Length) != headerBytes.Length) + { + return null; + } + DataStartPosition = stream.Position; + return LoadFrom(headerBytes); + } + + public async ValueTask ReadHeaderAsync( + Stream stream, + CancellationToken cancellationToken = default + ) + { + byte[] headerBytes = new byte[29]; + if ( + await stream + .ReadAsync(headerBytes, 0, headerBytes.Length, cancellationToken) + .ConfigureAwait(false) != headerBytes.Length + ) + { + return null; + } + DataStartPosition = stream.Position; + return LoadFrom(headerBytes); + } + + public ArcEntryHeader LoadFrom(byte[] headerBytes) + { + CompressionMethod = GetCompressionType(headerBytes[1]); + + // Read name + int nameEnd = Array.IndexOf(headerBytes, (byte)0, 1); // Find null terminator + Name = Encoding.UTF8.GetString(headerBytes, 2, nameEnd > 0 ? nameEnd - 2 : 12); + + int offset = 15; + CompressedSize = BitConverter.ToUInt32(headerBytes, offset); + offset += 4; + uint rawDateTime = BitConverter.ToUInt32(headerBytes, offset); + DateTime = ConvertToDateTime(rawDateTime); + offset += 4; + Crc16 = BitConverter.ToUInt16(headerBytes, offset); + offset += 2; + OriginalSize = BitConverter.ToUInt32(headerBytes, offset); + return this; + } + + private CompressionType GetCompressionType(byte value) + { + return value switch + { + 1 or 2 => CompressionType.None, + 3 => CompressionType.Packed, + 4 => CompressionType.Squeezed, + 5 or 6 or 7 or 8 => CompressionType.Crunched, + 9 => CompressionType.Squashed, + 10 => CompressionType.Crushed, + 11 => CompressionType.Distilled, + _ => CompressionType.Unknown, + }; + } + + public static DateTime ConvertToDateTime(long rawDateTime) + { + // Convert Unix timestamp to DateTime (UTC) + return DateTimeOffset.FromUnixTimeSeconds(rawDateTime).UtcDateTime; + } +} diff --git a/src/SharpCompress/Common/Arc/ArcFilePart.Async.cs b/src/SharpCompress/Common/Arc/ArcFilePart.Async.cs new file mode 100644 index 00000000..4af92288 --- /dev/null +++ b/src/SharpCompress/Common/Arc/ArcFilePart.Async.cs @@ -0,0 +1,57 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Compressors.ArcLzw; +using SharpCompress.Compressors.Lzw; +using SharpCompress.Compressors.RLE90; +using SharpCompress.Compressors.Squeezed; +using SharpCompress.IO; + +namespace SharpCompress.Common.Arc; + +public partial class ArcFilePart +{ + internal override async ValueTask GetCompressedStreamAsync( + CancellationToken cancellationToken = default + ) + { + if (_stream != null) + { + Stream compressedStream; + switch (Header.CompressionMethod) + { + case CompressionType.None: + compressedStream = new ReadOnlySubStream( + _stream, + Header.DataStartPosition, + Header.CompressedSize + ); + break; + case CompressionType.Packed: + compressedStream = new RunLength90Stream(_stream, (int)Header.CompressedSize); + break; + case CompressionType.Squeezed: + compressedStream = await SqueezeStream + .CreateAsync(_stream, (int)Header.CompressedSize, cancellationToken) + .ConfigureAwait(false); + break; + case CompressionType.Crunched: + if (Header.OriginalSize > 128 * 1024) + { + throw new NotSupportedException( + "CompressionMethod: " + Header.CompressionMethod + " with size > 128KB" + ); + } + compressedStream = new ArcLzwStream(_stream, (int)Header.CompressedSize, true); + break; + default: + throw new NotSupportedException( + "CompressionMethod: " + Header.CompressionMethod + ); + } + return compressedStream; + } + return _stream; + } +} diff --git a/src/SharpCompress/Common/Arc/ArcFilePart.cs b/src/SharpCompress/Common/Arc/ArcFilePart.cs new file mode 100644 index 00000000..d104396e --- /dev/null +++ b/src/SharpCompress/Common/Arc/ArcFilePart.cs @@ -0,0 +1,74 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using SharpCompress.Common.GZip; +using SharpCompress.Common.Tar; +using SharpCompress.Common.Tar.Headers; +using SharpCompress.Common.Zip.Headers; +using SharpCompress.Compressors.ArcLzw; +using SharpCompress.Compressors.Lzw; +using SharpCompress.Compressors.RLE90; +using SharpCompress.Compressors.Squeezed; +using SharpCompress.IO; + +namespace SharpCompress.Common.Arc; + +public partial class ArcFilePart : FilePart +{ + private readonly Stream? _stream; + + internal ArcFilePart(ArcEntryHeader localArcHeader, Stream? seekableStream) + : base(localArcHeader.ArchiveEncoding) + { + _stream = seekableStream; + Header = localArcHeader; + } + + internal ArcEntryHeader Header { get; set; } + + internal override string? FilePartName => Header.Name; + + internal override Stream GetCompressedStream() + { + if (_stream != null) + { + Stream compressedStream; + switch (Header.CompressionMethod) + { + case CompressionType.None: + compressedStream = new ReadOnlySubStream( + _stream, + Header.DataStartPosition, + Header.CompressedSize + ); + break; + case CompressionType.Packed: + compressedStream = new RunLength90Stream(_stream, (int)Header.CompressedSize); + break; + case CompressionType.Squeezed: + compressedStream = SqueezeStream.Create(_stream, (int)Header.CompressedSize); + break; + case CompressionType.Crunched: + if (Header.OriginalSize > 128 * 1024) + { + throw new NotSupportedException( + "CompressionMethod: " + Header.CompressionMethod + " with size > 128KB" + ); + } + compressedStream = new ArcLzwStream(_stream, (int)Header.CompressedSize, true); + break; + default: + throw new NotSupportedException( + "CompressionMethod: " + Header.CompressionMethod + ); + } + return compressedStream; + } + return _stream.NotNull(); + } + + internal override Stream? GetRawStream() => _stream; +} diff --git a/src/SharpCompress/Common/Arc/ArcVolume.cs b/src/SharpCompress/Common/Arc/ArcVolume.cs new file mode 100644 index 00000000..99fb56ee --- /dev/null +++ b/src/SharpCompress/Common/Arc/ArcVolume.cs @@ -0,0 +1,15 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using SharpCompress.Readers; + +namespace SharpCompress.Common.Arc; + +public class ArcVolume : Volume +{ + public ArcVolume(Stream stream, ReaderOptions readerOptions, int index = 0) + : base(stream, readerOptions, index) { } +} diff --git a/src/SharpCompress/Common/ArchiveEncoding.cs b/src/SharpCompress/Common/ArchiveEncoding.cs index f66044d3..2fd4a493 100644 --- a/src/SharpCompress/Common/ArchiveEncoding.cs +++ b/src/SharpCompress/Common/ArchiveEncoding.cs @@ -3,53 +3,11 @@ using System.Text; namespace SharpCompress.Common; -public class ArchiveEncoding +public class ArchiveEncoding : IArchiveEncoding { - /// - /// Default encoding to use when archive format doesn't specify one. - /// - public Encoding Default { get; set; } - - /// - /// ArchiveEncoding used by encryption schemes which don't comply with RFC 2898. - /// - public Encoding Password { get; set; } - - /// - /// Set this encoding when you want to force it for all encoding operations. - /// + public Encoding Default { get; set; } = Encoding.Default; + public Encoding Password { get; set; } = Encoding.Default; + public Encoding UTF8 { get; set; } = Encoding.UTF8; public Encoding? Forced { get; set; } - - /// - /// Set this when you want to use a custom method for all decoding operations. - /// - /// string Func(bytes, index, length) - public Func? CustomDecoder { get; set; } - - public ArchiveEncoding() - : this(Encoding.Default, Encoding.Default) { } - - public ArchiveEncoding(Encoding def, Encoding password) - { - Default = def; - Password = password; - } - -#if !NETFRAMEWORK - static ArchiveEncoding() => Encoding.RegisterProvider(CodePagesEncodingProvider.Instance); -#endif - - public string Decode(byte[] bytes) => Decode(bytes, 0, bytes.Length); - - public string Decode(byte[] bytes, int start, int length) => - GetDecoder().Invoke(bytes, start, length); - - public string DecodeUTF8(byte[] bytes) => Encoding.UTF8.GetString(bytes, 0, bytes.Length); - - public byte[] Encode(string str) => GetEncoding().GetBytes(str); - - public Encoding GetEncoding() => Forced ?? Default ?? Encoding.UTF8; - - public Func GetDecoder() => - CustomDecoder ?? ((bytes, index, count) => GetEncoding().GetString(bytes, index, count)); + public Func? CustomDecoder { get; set; } } diff --git a/src/SharpCompress/Common/ArchiveEncodingExtensions.cs b/src/SharpCompress/Common/ArchiveEncodingExtensions.cs new file mode 100644 index 00000000..88dc35b4 --- /dev/null +++ b/src/SharpCompress/Common/ArchiveEncodingExtensions.cs @@ -0,0 +1,87 @@ +using System; +using System.Text; + +namespace SharpCompress.Common; + +/// +/// Specifies the type of encoding to use. +/// +public enum EncodingType +{ + /// + /// Uses the default encoding. + /// + Default, + + /// + /// Uses UTF-8 encoding. + /// + UTF8, +} + +/// +/// Provides extension methods for archive encoding. +/// +public static class ArchiveEncodingExtensions +{ +#if !NETFRAMEWORK + /// + /// Registers the code pages encoding provider. + /// + static ArchiveEncodingExtensions() => + Encoding.RegisterProvider(CodePagesEncodingProvider.Instance); +#endif + + extension(IArchiveEncoding encoding) + { + /// + /// Gets the encoding based on the archive encoding settings. + /// + /// Whether to use UTF-8. + /// The encoding. + public Encoding GetEncoding(bool useUtf8 = false) => + encoding.Forced ?? (useUtf8 ? encoding.UTF8 : encoding.Default); + + /// + /// Gets the decoder function for the archive encoding. + /// + /// The decoder function. + public Func GetDecoder() => + encoding.CustomDecoder + ?? ( + (bytes, index, count, type) => + encoding.GetEncoding(type == EncodingType.UTF8).GetString(bytes, index, count) + ); + + /// + /// Encodes a string using the default encoding. + /// + /// The string to encode. + /// The encoded bytes. + public byte[] Encode(string str) => encoding.Default.GetBytes(str); + + /// + /// Decodes bytes using the specified encoding type. + /// + /// The bytes to decode. + /// The encoding type. + /// The decoded string. + public string Decode(byte[] bytes, EncodingType type = EncodingType.Default) => + encoding.Decode(bytes, 0, bytes.Length, type); + + /// + /// Decodes a portion of bytes using the specified encoding type. + /// + /// The bytes to decode. + /// The start index. + /// The length. + /// The encoding type. + /// The decoded string. + public string Decode( + byte[] bytes, + int start, + int length, + EncodingType type = EncodingType.Default + ) => encoding.GetDecoder()(bytes, start, length, type); + } +} diff --git a/src/SharpCompress/Common/ArchiveException.cs b/src/SharpCompress/Common/ArchiveException.cs deleted file mode 100644 index 507d5fd8..00000000 --- a/src/SharpCompress/Common/ArchiveException.cs +++ /dev/null @@ -1,9 +0,0 @@ -using System; - -namespace SharpCompress.Common; - -public class ArchiveException : Exception -{ - public ArchiveException(string message) - : base(message) { } -} diff --git a/src/SharpCompress/Common/ArchiveExtractionEventArgs.cs b/src/SharpCompress/Common/ArchiveExtractionEventArgs.cs deleted file mode 100644 index 80817748..00000000 --- a/src/SharpCompress/Common/ArchiveExtractionEventArgs.cs +++ /dev/null @@ -1,10 +0,0 @@ -using System; - -namespace SharpCompress.Common; - -public class ArchiveExtractionEventArgs : EventArgs -{ - internal ArchiveExtractionEventArgs(T entry) => Item = entry; - - public T Item { get; } -} diff --git a/src/SharpCompress/Common/ArchiveType.cs b/src/SharpCompress/Common/ArchiveType.cs index 49b35387..ae95af76 100644 --- a/src/SharpCompress/Common/ArchiveType.cs +++ b/src/SharpCompress/Common/ArchiveType.cs @@ -6,5 +6,9 @@ public enum ArchiveType Zip, Tar, SevenZip, - GZip + GZip, + Arc, + Arj, + Ace, + Lzw, } diff --git a/src/SharpCompress/Common/Arj/ArjEntry.cs b/src/SharpCompress/Common/Arj/ArjEntry.cs new file mode 100644 index 00000000..f0159a66 --- /dev/null +++ b/src/SharpCompress/Common/Arj/ArjEntry.cs @@ -0,0 +1,66 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using SharpCompress.Common.Arc; +using SharpCompress.Common.Arj.Headers; +using SharpCompress.Common.Options; + +namespace SharpCompress.Common.Arj; + +public class ArjEntry : Entry +{ + private readonly ArjFilePart _filePart; + + internal ArjEntry(ArjFilePart filePart, IReaderOptions readerOptions) + : base(readerOptions) + { + _filePart = filePart; + } + + public override long Crc => _filePart.Header.OriginalCrc32; + + internal override ChecksumDescriptor Checksum => + !IsDirectory + && _filePart.Header.CompressionMethod != CompressionMethod.NoDataNoCrc + && _filePart.Header.CompressionMethod != CompressionMethod.NoData + ? new ChecksumDescriptor(ChecksumKind.Crc32, _filePart.Header.OriginalCrc32, true) + : default; + + public override string? Key => _filePart?.Header.Name; + + public override string? LinkTarget => null; + + public override long CompressedSize => _filePart?.Header.CompressedSize ?? 0; + + public override CompressionType CompressionType + { + get + { + if (_filePart.Header.CompressionMethod == CompressionMethod.Stored) + { + return CompressionType.None; + } + return CompressionType.ArjLZ77; + } + } + + public override long Size => _filePart?.Header.OriginalSize ?? 0; + + public override DateTime? LastModifiedTime => _filePart.Header.DateTimeModified.DateTime; + + public override DateTime? CreatedTime => _filePart.Header.DateTimeCreated?.DateTime; + + public override DateTime? LastAccessedTime => _filePart.Header.DateTimeAccessed?.DateTime; + + public override DateTime? ArchivedTime => null; + + public override bool IsEncrypted => false; + + public override bool IsDirectory => _filePart.Header.FileType == FileType.Directory; + + public override bool IsSplitAfter => false; + + internal override IEnumerable Parts => _filePart.Empty(); +} diff --git a/src/SharpCompress/Common/Arj/ArjFilePart.cs b/src/SharpCompress/Common/Arj/ArjFilePart.cs new file mode 100644 index 00000000..d4f54067 --- /dev/null +++ b/src/SharpCompress/Common/Arj/ArjFilePart.cs @@ -0,0 +1,60 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using SharpCompress.Common.Arj.Headers; +using SharpCompress.Compressors.Arj; +using SharpCompress.IO; + +namespace SharpCompress.Common.Arj; + +public class ArjFilePart : FilePart +{ + private readonly Stream _stream; + internal ArjLocalHeader Header { get; set; } + + internal ArjFilePart(ArjLocalHeader localArjHeader, Stream seekableStream) + : base(localArjHeader.ArchiveEncoding) + { + _stream = seekableStream; + Header = localArjHeader; + } + + internal override string? FilePartName => Header.Name; + + internal override Stream GetCompressedStream() + { + Stream compressedStream; + switch (Header.CompressionMethod) + { + case CompressionMethod.Stored: + compressedStream = new ReadOnlySubStream( + _stream, + Header.DataStartPosition, + Header.CompressedSize + ); + break; + case CompressionMethod.CompressedMost: + case CompressionMethod.Compressed: + case CompressionMethod.CompressedFaster: + if (Header.OriginalSize > 128 * 1024) + { + throw new NotSupportedException( + "CompressionMethod: " + Header.CompressionMethod + " with size > 128KB" + ); + } + compressedStream = new LhaStream(_stream, (int)Header.OriginalSize); + break; + case CompressionMethod.CompressedFastest: + compressedStream = new LHDecoderStream(_stream, (int)Header.OriginalSize); + break; + default: + throw new NotSupportedException("CompressionMethod: " + Header.CompressionMethod); + } + return compressedStream; + } + + internal override Stream GetRawStream() => _stream; +} diff --git a/src/SharpCompress/Common/Arj/ArjVolume.cs b/src/SharpCompress/Common/Arj/ArjVolume.cs new file mode 100644 index 00000000..d65d56c9 --- /dev/null +++ b/src/SharpCompress/Common/Arj/ArjVolume.cs @@ -0,0 +1,35 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using SharpCompress.Common.Rar; +using SharpCompress.Common.Rar.Headers; +using SharpCompress.Readers; + +namespace SharpCompress.Common.Arj; + +public class ArjVolume : Volume +{ + public ArjVolume(Stream stream, ReaderOptions readerOptions, int index = 0) + : base(stream, readerOptions, index) { } + + public override bool IsFirstVolume + { + get { return true; } + } + + /// + /// ArjArchive is part of a multi-part archive. + /// + public override bool IsMultiVolume + { + get { return false; } + } + + internal IEnumerable GetVolumeFileParts() + { + return new List(); + } +} diff --git a/src/SharpCompress/Common/Arj/Headers/ArjHeader.Async.cs b/src/SharpCompress/Common/Arj/Headers/ArjHeader.Async.cs new file mode 100644 index 00000000..660fbc53 --- /dev/null +++ b/src/SharpCompress/Common/Arj/Headers/ArjHeader.Async.cs @@ -0,0 +1,140 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Crypto; + +namespace SharpCompress.Common.Arj.Headers; + +public abstract partial class ArjHeader +{ + public abstract ValueTask ReadAsync( + Stream reader, + CancellationToken cancellationToken = default + ); + + public async ValueTask ReadHeaderAsync( + Stream stream, + CancellationToken cancellationToken = default + ) + { + // check for magic bytes + var magic = new byte[2]; + if (await stream.ReadAsync(magic, 0, 2, cancellationToken).ConfigureAwait(false) != 2) + { + return Array.Empty(); + } + + if (!CheckMagicBytes(magic)) + { + throw new InvalidFormatException("Not an ARJ file (wrong magic bytes)"); + } + + // read header_size + byte[] headerBytes = new byte[2]; + await stream.ReadAsync(headerBytes, 0, 2, cancellationToken).ConfigureAwait(false); + var headerSize = (ushort)(headerBytes[0] | headerBytes[1] << 8); + if (headerSize < 1) + { + return Array.Empty(); + } + + var body = new byte[headerSize]; + var read = await stream + .ReadAsync(body, 0, headerSize, cancellationToken) + .ConfigureAwait(false); + if (read < headerSize) + { + return Array.Empty(); + } + + byte[] crc = new byte[4]; + await stream.ReadFullyAsync(crc, 0, 4, cancellationToken).ConfigureAwait(false); + var checksum = Crc32Stream.Compute(body); + // Compute the hash value + if (checksum != BitConverter.ToUInt32(crc, 0)) + { + throw new InvalidFormatException("Header checksum is invalid"); + } + return body; + } + + protected async ValueTask> ReadExtendedHeadersAsync( + Stream reader, + CancellationToken cancellationToken = default + ) + { + List extendedHeader = new List(); + byte[] buffer = new byte[2]; + + while (true) + { + int bytesRead = await reader + .ReadAsync(buffer, 0, 2, cancellationToken) + .ConfigureAwait(false); + if (bytesRead < 2) + { + throw new IncompleteArchiveException( + "Unexpected end of stream while reading extended header size." + ); + } + + var extHeaderSize = (ushort)(buffer[0] | (buffer[1] << 8)); + if (extHeaderSize == 0) + { + return extendedHeader; + } + + byte[] header = new byte[extHeaderSize]; + bytesRead = await reader + .ReadAsync(header, 0, extHeaderSize, cancellationToken) + .ConfigureAwait(false); + if (bytesRead < extHeaderSize) + { + throw new IncompleteArchiveException( + "Unexpected end of stream while reading extended header data." + ); + } + + byte[] crcextended = new byte[4]; + bytesRead = await reader + .ReadAsync(crcextended, 0, 4, cancellationToken) + .ConfigureAwait(false); + if (bytesRead < 4) + { + throw new IncompleteArchiveException( + "Unexpected end of stream while reading extended header CRC." + ); + } + + var checksum = Crc32Stream.Compute(header); + if (checksum != BitConverter.ToUInt32(crcextended, 0)) + { + throw new InvalidFormatException("Extended header checksum is invalid"); + } + + extendedHeader.Add(header); + } + } + + /// + /// Asynchronously checks if the stream is an ARJ archive + /// + /// The stream to read from + /// Cancellation token + /// True if the stream is an ARJ archive, false otherwise + public static async ValueTask IsArchiveAsync( + Stream stream, + CancellationToken cancellationToken = default + ) + { + var bytes = new byte[2]; + if (await stream.ReadAsync(bytes, 0, 2, cancellationToken).ConfigureAwait(false) != 2) + { + return false; + } + + return CheckMagicBytes(bytes); + } +} diff --git a/src/SharpCompress/Common/Arj/Headers/ArjHeader.cs b/src/SharpCompress/Common/Arj/Headers/ArjHeader.cs new file mode 100644 index 00000000..f974e44a --- /dev/null +++ b/src/SharpCompress/Common/Arj/Headers/ArjHeader.cs @@ -0,0 +1,163 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Zip.Headers; +using SharpCompress.Crypto; + +namespace SharpCompress.Common.Arj.Headers; + +public enum ArjHeaderType +{ + MainHeader, + LocalHeader, +} + +public abstract partial class ArjHeader +{ + private const ushort ARJ_MAGIC = 0xEA60; + + public ArjHeader(ArjHeaderType type) + { + ArjHeaderType = type; + } + + public ArjHeaderType ArjHeaderType { get; } + public byte Flags { get; set; } + public FileType FileType { get; set; } + + public abstract ArjHeader? Read(Stream reader); + + // Async methods moved to ArjHeader.Async.cs + + public byte[] ReadHeader(Stream stream) + { + // check for magic bytes + var magic = new byte[2]; + if (stream.Read(magic) != 2) + { + return Array.Empty(); + } + + if (!CheckMagicBytes(magic)) + { + throw new InvalidFormatException("Not an ARJ file (wrong magic bytes)"); + } + + // read header_size + byte[] headerBytes = new byte[2]; + stream.Read(headerBytes, 0, 2); + var headerSize = (ushort)(headerBytes[0] | headerBytes[1] << 8); + if (headerSize < 1) + { + return Array.Empty(); + } + + var body = new byte[headerSize]; + var read = stream.Read(body, 0, headerSize); + if (read < headerSize) + { + return Array.Empty(); + } + + byte[] crc = new byte[4]; + read = stream.Read(crc, 0, 4); + var checksum = Crc32Stream.Compute(body); + // Compute the hash value + if (checksum != BitConverter.ToUInt32(crc, 0)) + { + throw new InvalidFormatException("Header checksum is invalid"); + } + return body; + } + + // ReadHeaderAsync moved to ArjHeader.Async.cs + + protected List ReadExtendedHeaders(Stream reader) + { + List extendedHeader = new List(); + byte[] buffer = new byte[2]; + + while (true) + { + int bytesRead = reader.Read(buffer, 0, 2); + if (bytesRead < 2) + { + throw new IncompleteArchiveException( + "Unexpected end of stream while reading extended header size." + ); + } + + var extHeaderSize = (ushort)(buffer[0] | (buffer[1] << 8)); + if (extHeaderSize == 0) + { + return extendedHeader; + } + + byte[] header = new byte[extHeaderSize]; + bytesRead = reader.Read(header, 0, extHeaderSize); + if (bytesRead < extHeaderSize) + { + throw new IncompleteArchiveException( + "Unexpected end of stream while reading extended header data." + ); + } + + byte[] crc = new byte[4]; + bytesRead = reader.Read(crc, 0, 4); + if (bytesRead < 4) + { + throw new IncompleteArchiveException( + "Unexpected end of stream while reading extended header CRC." + ); + } + + var checksum = Crc32Stream.Compute(header); + if (checksum != BitConverter.ToUInt32(crc, 0)) + { + throw new InvalidFormatException("Extended header checksum is invalid"); + } + + extendedHeader.Add(header); + } + } + + // Flag helpers + public bool IsGabled => (Flags & 0x01) != 0; + public bool IsAnsiPage => (Flags & 0x02) != 0; + public bool IsVolume => (Flags & 0x04) != 0; + public bool IsArjProtected => (Flags & 0x08) != 0; + public bool IsPathSym => (Flags & 0x10) != 0; + public bool IsBackup => (Flags & 0x20) != 0; + public bool IsSecured => (Flags & 0x40) != 0; + public bool IsAltName => (Flags & 0x80) != 0; + + public static FileType FileTypeFromByte(byte value) + { +#if LEGACY_DOTNET + return Enum.IsDefined(typeof(FileType), value) ? (FileType)value : Headers.FileType.Unknown; +#else + return Enum.IsDefined((FileType)value) ? (FileType)value : Headers.FileType.Unknown; +#endif + } + + public static bool IsArchive(Stream stream) + { + var bytes = new byte[2]; + if (stream.Read(bytes, 0, 2) != 2) + { + return false; + } + + return CheckMagicBytes(bytes); + } + + protected static bool CheckMagicBytes(byte[] headerBytes) + { + var magicValue = (ushort)(headerBytes[0] | headerBytes[1] << 8); + return magicValue == ARJ_MAGIC; + } +} diff --git a/src/SharpCompress/Common/Arj/Headers/ArjLocalHeader.Async.cs b/src/SharpCompress/Common/Arj/Headers/ArjLocalHeader.Async.cs new file mode 100644 index 00000000..5b5dab70 --- /dev/null +++ b/src/SharpCompress/Common/Arj/Headers/ArjLocalHeader.Async.cs @@ -0,0 +1,24 @@ +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Common.Arj.Headers; + +public partial class ArjLocalHeader +{ + public override async ValueTask ReadAsync( + Stream reader, + CancellationToken cancellationToken = default + ) + { + var body = await ReadHeaderAsync(reader, cancellationToken).ConfigureAwait(false); + if (body.Length > 0) + { + await ReadExtendedHeadersAsync(reader, cancellationToken).ConfigureAwait(false); + var header = LoadFrom(body); + header.DataStartPosition = reader.Position; + return header; + } + return null; + } +} diff --git a/src/SharpCompress/Common/Arj/Headers/ArjLocalHeader.cs b/src/SharpCompress/Common/Arj/Headers/ArjLocalHeader.cs new file mode 100644 index 00000000..1a91a77f --- /dev/null +++ b/src/SharpCompress/Common/Arj/Headers/ArjLocalHeader.cs @@ -0,0 +1,160 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Runtime.CompilerServices; +using System.Text; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Common.Arj.Headers; + +public partial class ArjLocalHeader : ArjHeader +{ + public ArchiveEncoding ArchiveEncoding { get; } + public long DataStartPosition { get; protected set; } + + public byte ArchiverVersionNumber { get; set; } + public byte MinVersionToExtract { get; set; } + public HostOS HostOS { get; set; } + public CompressionMethod CompressionMethod { get; set; } + public DosDateTime DateTimeModified { get; set; } = new DosDateTime(0); + public long CompressedSize { get; set; } + public long OriginalSize { get; set; } + public long OriginalCrc32 { get; set; } + public int FileSpecPosition { get; set; } + public int FileAccessMode { get; set; } + public byte FirstChapter { get; set; } + public byte LastChapter { get; set; } + public long ExtendedFilePosition { get; set; } + public DosDateTime? DateTimeAccessed { get; set; } + public DosDateTime? DateTimeCreated { get; set; } + public long OriginalSizeEvenForVolumes { get; set; } + public string Name { get; set; } = string.Empty; + public string Comment { get; set; } = string.Empty; + + private const byte StdHdrSize = 30; + private const byte R9HdrSize = 46; + + public ArjLocalHeader(ArchiveEncoding archiveEncoding) + : base(ArjHeaderType.LocalHeader) + { + ArchiveEncoding = + archiveEncoding ?? throw new ArgumentNullException(nameof(archiveEncoding)); + } + + public override ArjHeader? Read(Stream reader) + { + var body = ReadHeader(reader); + if (body.Length > 0) + { + ReadExtendedHeaders(reader); + var header = LoadFrom(body); + header.DataStartPosition = reader.Position; + return header; + } + return null; + } + + // ReadAsync moved to ArjLocalHeader.Async.cs + + public ArjLocalHeader LoadFrom(byte[] headerBytes) + { + int offset = 0; + + int ReadInt16() + { + if (offset + 1 >= headerBytes.Length) + { + throw new IncompleteArchiveException("Unexpected end of stream."); + } + var v = headerBytes[offset] & 0xFF | (headerBytes[offset + 1] & 0xFF) << 8; + offset += 2; + return v; + } + long ReadInt32() + { + if (offset + 3 >= headerBytes.Length) + { + throw new IncompleteArchiveException("Unexpected end of stream."); + } + long v = + headerBytes[offset] & 0xFF + | (headerBytes[offset + 1] & 0xFF) << 8 + | (headerBytes[offset + 2] & 0xFF) << 16 + | (headerBytes[offset + 3] & 0xFF) << 24; + offset += 4; + return v; + } + + byte headerSize = headerBytes[offset++]; + ArchiverVersionNumber = headerBytes[offset++]; + MinVersionToExtract = headerBytes[offset++]; + HostOS hostOS = (HostOS)headerBytes[offset++]; + Flags = headerBytes[offset++]; + CompressionMethod = CompressionMethodFromByte(headerBytes[offset++]); + FileType = FileTypeFromByte(headerBytes[offset++]); + + offset++; // Skip 1 byte + + var rawTimestamp = ReadInt32(); + DateTimeModified = rawTimestamp != 0 ? new DosDateTime(rawTimestamp) : new DosDateTime(0); + + CompressedSize = ReadInt32(); + OriginalSize = ReadInt32(); + OriginalCrc32 = ReadInt32(); + FileSpecPosition = ReadInt16(); + FileAccessMode = ReadInt16(); + + FirstChapter = headerBytes[offset++]; + LastChapter = headerBytes[offset++]; + + ExtendedFilePosition = 0; + OriginalSizeEvenForVolumes = 0; + + if (headerSize > StdHdrSize) + { + ExtendedFilePosition = ReadInt32(); + + if (headerSize >= R9HdrSize) + { + rawTimestamp = ReadInt32(); + DateTimeAccessed = rawTimestamp != 0 ? new DosDateTime(rawTimestamp) : null; + rawTimestamp = ReadInt32(); + DateTimeCreated = rawTimestamp != 0 ? new DosDateTime(rawTimestamp) : null; + OriginalSizeEvenForVolumes = ReadInt32(); + } + } + + Name = Encoding.ASCII.GetString( + headerBytes, + offset, + Array.IndexOf(headerBytes, (byte)0, offset) - offset + ); + offset += Name.Length + 1; + + Comment = Encoding.ASCII.GetString( + headerBytes, + offset, + Array.IndexOf(headerBytes, (byte)0, offset) - offset + ); + offset += Comment.Length + 1; + + return this; + } + + public static CompressionMethod CompressionMethodFromByte(byte value) + { + return value switch + { + 0 => CompressionMethod.Stored, + 1 => CompressionMethod.CompressedMost, + 2 => CompressionMethod.Compressed, + 3 => CompressionMethod.CompressedFaster, + 4 => CompressionMethod.CompressedFastest, + 8 => CompressionMethod.NoDataNoCrc, + 9 => CompressionMethod.NoData, + _ => CompressionMethod.Unknown, + }; + } +} diff --git a/src/SharpCompress/Common/Arj/Headers/ArjMainHeader.Async.cs b/src/SharpCompress/Common/Arj/Headers/ArjMainHeader.Async.cs new file mode 100644 index 00000000..2271877c --- /dev/null +++ b/src/SharpCompress/Common/Arj/Headers/ArjMainHeader.Async.cs @@ -0,0 +1,18 @@ +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Common.Arj.Headers; + +public partial class ArjMainHeader +{ + public override async ValueTask ReadAsync( + Stream reader, + CancellationToken cancellationToken = default + ) + { + var body = await ReadHeaderAsync(reader, cancellationToken).ConfigureAwait(false); + await ReadExtendedHeadersAsync(reader, cancellationToken).ConfigureAwait(false); + return LoadFrom(body); + } +} diff --git a/src/SharpCompress/Common/Arj/Headers/ArjMainHeader.cs b/src/SharpCompress/Common/Arj/Headers/ArjMainHeader.cs new file mode 100644 index 00000000..74fb0542 --- /dev/null +++ b/src/SharpCompress/Common/Arj/Headers/ArjMainHeader.cs @@ -0,0 +1,138 @@ +using System; +using System.IO; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Compressors.Deflate; +using SharpCompress.Crypto; + +namespace SharpCompress.Common.Arj.Headers; + +public partial class ArjMainHeader : ArjHeader +{ + public ArchiveEncoding ArchiveEncoding { get; } + + public int ArchiverVersionNumber { get; private set; } + public int MinVersionToExtract { get; private set; } + public HostOS HostOs { get; private set; } + public int SecurityVersion { get; private set; } + public DosDateTime CreationDateTime { get; private set; } = new DosDateTime(0); + public long CompressedSize { get; private set; } + public long ArchiveSize { get; private set; } + public long SecurityEnvelope { get; private set; } + public int FileSpecPosition { get; private set; } + public int SecurityEnvelopeLength { get; private set; } + public int EncryptionVersion { get; private set; } + public int LastChapter { get; private set; } + + public int ArjProtectionFactor { get; private set; } + public int Flags2 { get; private set; } + public string Name { get; private set; } = string.Empty; + public string Comment { get; private set; } = string.Empty; + + public ArjMainHeader(ArchiveEncoding archiveEncoding) + : base(ArjHeaderType.MainHeader) + { + ArchiveEncoding = + archiveEncoding ?? throw new ArgumentNullException(nameof(archiveEncoding)); + } + + public override ArjHeader? Read(Stream reader) + { + var body = ReadHeader(reader); + ReadExtendedHeaders(reader); + return LoadFrom(body); + } + + // ReadAsync moved to ArjMainHeader.Async.cs + + public ArjMainHeader LoadFrom(byte[] headerBytes) + { + var offset = 1; + + byte ReadByte() + { + if (offset >= headerBytes.Length) + { + throw new IncompleteArchiveException("Unexpected end of stream."); + } + return (byte)(headerBytes[offset++] & 0xFF); + } + + int ReadInt16() + { + if (offset + 1 >= headerBytes.Length) + { + throw new IncompleteArchiveException("Unexpected end of stream."); + } + var v = headerBytes[offset] & 0xFF | (headerBytes[offset + 1] & 0xFF) << 8; + offset += 2; + return v; + } + + long ReadInt32() + { + if (offset + 3 >= headerBytes.Length) + { + throw new IncompleteArchiveException("Unexpected end of stream."); + } + long v = + headerBytes[offset] & 0xFF + | (headerBytes[offset + 1] & 0xFF) << 8 + | (headerBytes[offset + 2] & 0xFF) << 16 + | (headerBytes[offset + 3] & 0xFF) << 24; + offset += 4; + return v; + } + string ReadNullTerminatedString(byte[] x, int startIndex) + { + var result = new StringBuilder(); + int i = startIndex; + + while (i < x.Length && x[i] != 0) + { + result.Append((char)x[i]); + i++; + } + + // Skip the null terminator + i++; + if (i < x.Length) + { + byte[] remainder = new byte[x.Length - i]; + Array.Copy(x, i, remainder, 0, remainder.Length); + x = remainder; + } + + return result.ToString(); + } + + ArchiverVersionNumber = ReadByte(); + MinVersionToExtract = ReadByte(); + + var hostOsByte = ReadByte(); + HostOs = hostOsByte <= 11 ? (HostOS)hostOsByte : HostOS.Unknown; + + Flags = ReadByte(); + SecurityVersion = ReadByte(); + FileType = FileTypeFromByte(ReadByte()); + + offset++; // skip reserved + + CreationDateTime = new DosDateTime((int)ReadInt32()); + CompressedSize = ReadInt32(); + ArchiveSize = ReadInt32(); + + SecurityEnvelope = ReadInt32(); + FileSpecPosition = ReadInt16(); + SecurityEnvelopeLength = ReadInt16(); + + EncryptionVersion = ReadByte(); + LastChapter = ReadByte(); + + Name = ReadNullTerminatedString(headerBytes, offset); + Comment = ReadNullTerminatedString(headerBytes, offset + 1 + Name.Length); + + return this; + } +} diff --git a/src/SharpCompress/Common/Arj/Headers/CompressionMethod.cs b/src/SharpCompress/Common/Arj/Headers/CompressionMethod.cs new file mode 100644 index 00000000..e4423e2a --- /dev/null +++ b/src/SharpCompress/Common/Arj/Headers/CompressionMethod.cs @@ -0,0 +1,19 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text; +using System.Threading.Tasks; + +namespace SharpCompress.Common.Arj.Headers; + +public enum CompressionMethod +{ + Stored = 0, + CompressedMost = 1, + Compressed = 2, + CompressedFaster = 3, + CompressedFastest = 4, + NoDataNoCrc = 8, + NoData = 9, + Unknown, +} diff --git a/src/SharpCompress/Common/Arj/Headers/DosDateTime.cs b/src/SharpCompress/Common/Arj/Headers/DosDateTime.cs new file mode 100644 index 00000000..3f913faa --- /dev/null +++ b/src/SharpCompress/Common/Arj/Headers/DosDateTime.cs @@ -0,0 +1,37 @@ +using System; + +namespace SharpCompress.Common.Arj.Headers; + +public class DosDateTime +{ + public DateTime DateTime { get; } + + public DosDateTime(long dosValue) + { + // Ensure only the lower 32 bits are used + int value = unchecked((int)(dosValue & 0xFFFFFFFF)); + + var date = (value >> 16) & 0xFFFF; + var time = value & 0xFFFF; + + var day = date & 0x1F; + var month = (date >> 5) & 0x0F; + var year = ((date >> 9) & 0x7F) + 1980; + + var second = (time & 0x1F) * 2; + var minute = (time >> 5) & 0x3F; + var hour = (time >> 11) & 0x1F; + + try + { + DateTime = new DateTime(year, month, day, hour, minute, second); + } + catch + { + DateTime = DateTime.MinValue; + } + } + + public override string ToString() => + DateTime.ToString("yyyy-MM-dd HH:mm:ss", Constants.DefaultCultureInfo); +} diff --git a/src/SharpCompress/Common/Arj/Headers/FileType.cs b/src/SharpCompress/Common/Arj/Headers/FileType.cs new file mode 100644 index 00000000..18b4535b --- /dev/null +++ b/src/SharpCompress/Common/Arj/Headers/FileType.cs @@ -0,0 +1,12 @@ +namespace SharpCompress.Common.Arj.Headers; + +public enum FileType : byte +{ + Binary = 0, + Text7Bit = 1, + CommentHeader = 2, + Directory = 3, + VolumeLabel = 4, + ChapterLabel = 5, + Unknown = 255, +} diff --git a/src/SharpCompress/Common/Arj/Headers/HostOS.cs b/src/SharpCompress/Common/Arj/Headers/HostOS.cs new file mode 100644 index 00000000..ded53a94 --- /dev/null +++ b/src/SharpCompress/Common/Arj/Headers/HostOS.cs @@ -0,0 +1,18 @@ +namespace SharpCompress.Common.Arj.Headers; + +public enum HostOS +{ + MsDos = 0, + PrimOS = 1, + Unix = 2, + Amiga = 3, + MacOs = 4, + OS2 = 5, + AppleGS = 6, + AtariST = 7, + NeXT = 8, + VaxVMS = 9, + Win95 = 10, + Win32 = 11, + Unknown = 255, +} diff --git a/src/SharpCompress/Common/ChecksumDescriptor.cs b/src/SharpCompress/Common/ChecksumDescriptor.cs new file mode 100644 index 00000000..b418df2b --- /dev/null +++ b/src/SharpCompress/Common/ChecksumDescriptor.cs @@ -0,0 +1,14 @@ +namespace SharpCompress.Common; + +internal enum ChecksumKind +{ + Crc32, + Crc32NoFinalXor, + Crc16Arc, +} + +internal readonly record struct ChecksumDescriptor( + ChecksumKind Kind, + long ExpectedValue, + bool IsAvailable +); diff --git a/src/SharpCompress/Common/ChecksumValidationStream.cs b/src/SharpCompress/Common/ChecksumValidationStream.cs new file mode 100644 index 00000000..1b14cc9d --- /dev/null +++ b/src/SharpCompress/Common/ChecksumValidationStream.cs @@ -0,0 +1,178 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Crypto; + +namespace SharpCompress.Common; + +internal sealed class ChecksumValidationStream : Stream +{ + private readonly Stream _stream; + private readonly ChecksumDescriptor _checksum; + private readonly string _entryName; + private readonly uint[] _crc32Table; + private uint _seed = Crc32Stream.DEFAULT_SEED; + private ushort _crc16; + private bool _validated; + + internal ChecksumValidationStream(Stream stream, ChecksumDescriptor checksum, string? entryName) + { + _stream = stream; + _checksum = checksum; + _entryName = string.IsNullOrEmpty(entryName) ? "Entry" : entryName!; + _crc32Table = Crc32Stream.InitializeTable(Crc32Stream.DEFAULT_POLYNOMIAL); + } + + public override bool CanRead => _stream.CanRead; + public override bool CanSeek => false; + public override bool CanWrite => false; + public override long Length => _stream.Length; + + public override long Position + { + get => _stream.Position; + set => throw new NotSupportedException(); + } + + public override void Flush() => _stream.Flush(); + + public override Task FlushAsync(CancellationToken cancellationToken) => + _stream.FlushAsync(cancellationToken); + + public override int Read(byte[] buffer, int offset, int count) + { + var read = _stream.Read(buffer, offset, count); + UpdateAndValidateAtEof(buffer.AsSpan(offset, read), read); + return read; + } + +#if !LEGACY_DOTNET + public override int Read(Span buffer) + { + var read = _stream.Read(buffer); + UpdateAndValidateAtEof(buffer[..read], read); + return read; + } +#endif + + public override int ReadByte() => throw new NotSupportedException(); + + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + var read = await _stream + .ReadAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + UpdateAndValidateAtEof(buffer.AsSpan(offset, read), read); + return read; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + var read = await _stream.ReadAsync(buffer, cancellationToken).ConfigureAwait(false); + UpdateAndValidateAtEof(buffer.Span[..read], read); + return read; + } +#endif + + public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); + + public override void SetLength(long value) => throw new NotSupportedException(); + + public override void Write(byte[] buffer, int offset, int count) => + throw new NotSupportedException(); + + private void UpdateAndValidateAtEof(ReadOnlySpan buffer, int read) + { + if (read > 0) + { + UpdateChecksum(buffer); + return; + } + + Validate(); + } + + private void UpdateChecksum(ReadOnlySpan buffer) + { + switch (_checksum.Kind) + { + case ChecksumKind.Crc32: + case ChecksumKind.Crc32NoFinalXor: + _seed = Crc32Stream.CalculateCrc(_crc32Table, _seed, buffer); + break; + case ChecksumKind.Crc16Arc: + _crc16 = CalculateCrc16Arc(_crc16, buffer); + break; + } + } + + private void Validate() + { + if (_validated) + { + return; + } + + _validated = true; + + switch (_checksum.Kind) + { + case ChecksumKind.Crc32: + ValidateCrc32(finalXor: true); + break; + case ChecksumKind.Crc32NoFinalXor: + ValidateCrc32(finalXor: false); + break; + case ChecksumKind.Crc16Arc: + ValidateCrc16Arc(); + break; + } + } + + private void ValidateCrc32(bool finalXor) + { + var actual = finalXor ? ~_seed : _seed; + var expected = unchecked((uint)_checksum.ExpectedValue); + if (actual != expected) + { + throw new InvalidFormatException( + $"CRC mismatch for entry '{_entryName}'. Expected 0x{expected:X8}, actual 0x{actual:X8}." + ); + } + } + + private void ValidateCrc16Arc() + { + var expected = unchecked((ushort)_checksum.ExpectedValue); + if (_crc16 != expected) + { + throw new InvalidFormatException( + $"CRC mismatch for entry '{_entryName}'. Expected 0x{expected:X4}, actual 0x{_crc16:X4}." + ); + } + } + + private static ushort CalculateCrc16Arc(ushort crc, ReadOnlySpan buffer) + { + foreach (var value in buffer) + { + crc ^= value; + for (var i = 0; i < 8; i++) + { + crc = (crc & 1) != 0 ? (ushort)((crc >> 1) ^ 0xA001) : (ushort)(crc >> 1); + } + } + + return crc; + } +} diff --git a/src/SharpCompress/Common/CompressedBytesReadEventArgs.cs b/src/SharpCompress/Common/CompressedBytesReadEventArgs.cs deleted file mode 100644 index 34ca461f..00000000 --- a/src/SharpCompress/Common/CompressedBytesReadEventArgs.cs +++ /dev/null @@ -1,25 +0,0 @@ -using System; - -namespace SharpCompress.Common; - -public sealed class CompressedBytesReadEventArgs : EventArgs -{ - public CompressedBytesReadEventArgs( - long compressedBytesRead, - long currentFilePartCompressedBytesRead - ) - { - CompressedBytesRead = compressedBytesRead; - CurrentFilePartCompressedBytesRead = currentFilePartCompressedBytesRead; - } - - /// - /// Compressed bytes read for the current entry - /// - public long CompressedBytesRead { get; } - - /// - /// Current file part read for Multipart files (e.g. Rar) - /// - public long CurrentFilePartCompressedBytesRead { get; } -} diff --git a/src/SharpCompress/Common/CompressionType.cs b/src/SharpCompress/Common/CompressionType.cs index 11263245..f9b63855 100644 --- a/src/SharpCompress/Common/CompressionType.cs +++ b/src/SharpCompress/Common/CompressionType.cs @@ -9,10 +9,27 @@ public enum CompressionType Deflate, Rar, LZMA, + LZMA2, BCJ, BCJ2, LZip, Xz, Unknown, - Deflate64 + Deflate64, + Shrink, + Lzw, + Reduce1, + Reduce2, + Reduce3, + Reduce4, + Explode, + Squeezed, + Packed, + Crunched, + Squashed, + Crushed, + Distilled, + ZStandard, + ArjLZ77, + AceLZ77, } diff --git a/src/SharpCompress/Common/Constants.cs b/src/SharpCompress/Common/Constants.cs new file mode 100644 index 00000000..44909ef9 --- /dev/null +++ b/src/SharpCompress/Common/Constants.cs @@ -0,0 +1,50 @@ +using System.Globalization; + +namespace SharpCompress.Common; + +public static class Constants +{ + /// + /// The default buffer size for stream operations, matching .NET's Stream.CopyTo default of 81920 bytes. + /// This can be modified globally at runtime. + /// + // TODO: Revisit remaining non-extraction usages after extraction buffering moves to ExtractionOptions. + public static int BufferSize { get; set; } = 81920; + + /// + /// The default size for rewindable buffers in SharpCompressStream. + /// Used for format detection on non-seekable streams. + /// + /// + /// + /// When opening archives from non-seekable streams (network streams, pipes, + /// compressed streams), SharpCompress uses a ring buffer to enable format + /// auto-detection. This buffer allows the library to try multiple decoders + /// by rewinding and re-reading the same data. + /// + /// + /// Default: 81920 bytes (80KB) — sufficient for most formats. + /// Formats that require larger buffers (e.g. BZip2, ZStandard) declare their + /// own minimum via TarWrapper.MinimumRewindBufferSize, and + /// TarWrapper.MaximumRewindBufferSize is used at stream construction + /// to ensure the correct capacity is allocated upfront. + /// + /// + /// Typical usage: 500-1000 bytes for most archives + /// + /// + /// Can be overridden per-stream via ReaderOptions.RewindableBufferSize. + /// + /// + /// Increase if: + /// + /// Handling self-extracting archives (may need 512KB+) + /// Format detection fails with buffer overflow errors + /// Using custom formats with large headers + /// + /// + /// + public static int RewindableBufferSize { get; set; } = 81920; + + public static CultureInfo DefaultCultureInfo { get; set; } = CultureInfo.InvariantCulture; +} diff --git a/src/SharpCompress/Common/CryptographicException.cs b/src/SharpCompress/Common/CryptographicException.cs deleted file mode 100644 index 6127524a..00000000 --- a/src/SharpCompress/Common/CryptographicException.cs +++ /dev/null @@ -1,9 +0,0 @@ -using System; - -namespace SharpCompress.Common; - -public class CryptographicException : Exception -{ - public CryptographicException(string message) - : base(message) { } -} diff --git a/src/SharpCompress/Common/DirectoryManagement.cs b/src/SharpCompress/Common/DirectoryManagement.cs new file mode 100644 index 00000000..4bd4585d --- /dev/null +++ b/src/SharpCompress/Common/DirectoryManagement.cs @@ -0,0 +1,77 @@ +using System.IO; + +namespace SharpCompress.Common; + +internal static class DirectoryManagement +{ + internal const string CreateDirectoryOutsideDestinationMessage = + "Entry is trying to create a directory outside of the destination directory."; + internal const string WriteFileOutsideDestinationMessage = + "Entry is trying to write a file outside of the destination directory."; + + internal static string GetFullDestinationDirectoryPath(string destinationDirectory) + { + var fullDestinationDirectoryPath = Path.GetFullPath(destinationDirectory); + + // Keep the trailing separator so prefix checks cannot match sibling directories. + if ( + !IsDirectorySeparator( + fullDestinationDirectoryPath[fullDestinationDirectoryPath.Length - 1] + ) + ) + { + fullDestinationDirectoryPath += Path.DirectorySeparatorChar; + } + + if (!Directory.Exists(fullDestinationDirectoryPath)) + { + throw new ExtractionException( + $"Directory does not exist to extract to: {fullDestinationDirectoryPath}" + ); + } + + return fullDestinationDirectoryPath; + } + + internal static void EnsurePathInDestinationDirectory( + string destinationPath, + string fullDestinationDirectoryPath, + string exceptionMessage + ) + { + if (destinationPath.StartsWith(fullDestinationDirectoryPath, Utility.PathComparison)) + { + return; + } + + if ( + string.Equals( + destinationPath, + TrimTrailingDirectorySeparators(fullDestinationDirectoryPath), + Utility.PathComparison + ) + ) + { + return; + } + + throw new ExtractionException(exceptionMessage); + } + + private static bool IsDirectorySeparator(char value) => + value == Path.DirectorySeparatorChar || value == Path.AltDirectorySeparatorChar; + + private static string TrimTrailingDirectorySeparators(string path) + { + var root = Path.GetPathRoot(path); + var rootLength = root?.Length ?? 0; + var end = path.Length; + + while (end > rootLength && IsDirectorySeparator(path[end - 1])) + { + end--; + } + + return end == path.Length ? path : path.Substring(0, end); + } +} diff --git a/src/SharpCompress/Common/Entry.cs b/src/SharpCompress/Common/Entry.cs index 85219a43..43095b71 100644 --- a/src/SharpCompress/Common/Entry.cs +++ b/src/SharpCompress/Common/Entry.cs @@ -1,6 +1,8 @@ using System; using System.Collections.Generic; +using System.IO; using System.Linq; +using SharpCompress.Common.Options; namespace SharpCompress.Common; @@ -14,7 +16,7 @@ public abstract class Entry : IEntry /// /// The string key of the file internal to the Archive. /// - public abstract string Key { get; } + public abstract string? Key { get; } /// /// The target of a symlink entry internal to the Archive. Will be null if not a symlink. @@ -71,11 +73,11 @@ public abstract class Entry : IEntry /// public abstract bool IsSplitAfter { get; } - public int VolumeIndexFirst => Parts?.FirstOrDefault()?.Index ?? 0; - public int VolumeIndexLast => Parts?.LastOrDefault()?.Index ?? 0; + public int VolumeIndexFirst => Parts.FirstOrDefault()?.Index ?? 0; + public int VolumeIndexLast => Parts.LastOrDefault()?.Index ?? 0; /// - public override string ToString() => Key; + public override string ToString() => Key ?? "Entry"; internal abstract IEnumerable Parts { get; } @@ -83,8 +85,31 @@ public abstract class Entry : IEntry internal virtual void Close() { } + internal virtual ChecksumDescriptor Checksum => default; + + internal virtual Stream WrapWithChecksumValidation(Stream source, ExtractionOptions options) + { + var checksum = Checksum; + if (!checksum.IsAvailable) + { + return source; + } + + return new ChecksumValidationStream(source, checksum, Key); + } + /// /// Entry file attribute. /// public virtual int? Attrib => throw new NotImplementedException(); + + /// + /// The options used when opening this entry's source (reader or archive). + /// + public IReaderOptions Options { get; protected set; } + + protected Entry(IReaderOptions readerOptions) + { + Options = readerOptions; + } } diff --git a/src/SharpCompress/Common/EntryStream.Async.cs b/src/SharpCompress/Common/EntryStream.Async.cs new file mode 100644 index 00000000..a58afe70 --- /dev/null +++ b/src/SharpCompress/Common/EntryStream.Async.cs @@ -0,0 +1,81 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.IO; + +namespace SharpCompress.Common; + +public partial class EntryStream +{ + /// + /// Asynchronously skip the rest of the entry stream. + /// + public async ValueTask SkipEntryAsync(CancellationToken cancellationToken = default) + { + await this.SkipAsync(cancellationToken).ConfigureAwait(false); + _completed = true; + } + +#if !LEGACY_DOTNET + public override async ValueTask DisposeAsync() + { + if (_isDisposed) + { + return; + } + _isDisposed = true; + if (!(_completed || _reader.Cancelled)) + { + await SkipEntryAsync().ConfigureAwait(false); + } + + //Need a safe standard approach to this - it's okay for compression to overreads. Handling needs to be standardised + if (_stream is IStreamStack ss) + { + if (ss.BaseStream() is SharpCompress.Compressors.Deflate.DeflateStream deflateStream) + { + await deflateStream.FlushAsync().ConfigureAwait(false); + } + else if (ss.BaseStream() is SharpCompress.Compressors.LZMA.LzmaStream lzmaStream) + { + await lzmaStream.FlushAsync().ConfigureAwait(false); + } + } + await base.DisposeAsync().ConfigureAwait(false); + await _stream.DisposeAsync().ConfigureAwait(false); + } +#endif + + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + var read = await _stream + .ReadAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + if (read <= 0) + { + _completed = true; + } + return read; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + var read = await _stream.ReadAsync(buffer, cancellationToken).ConfigureAwait(false); + if (read <= 0) + { + _completed = true; + } + return read; + } +#endif +} diff --git a/src/SharpCompress/Common/EntryStream.cs b/src/SharpCompress/Common/EntryStream.cs index b3e7edea..b8ccf339 100644 --- a/src/SharpCompress/Common/EntryStream.cs +++ b/src/SharpCompress/Common/EntryStream.cs @@ -1,10 +1,14 @@ using System; using System.IO; +using System.IO.Compression; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.IO; using SharpCompress.Readers; namespace SharpCompress.Common; -public class EntryStream : Stream +public partial class EntryStream : Stream { private readonly IReader _reader; private readonly Stream _stream; @@ -28,15 +32,45 @@ public class EntryStream : Stream protected override void Dispose(bool disposing) { - if (!(_completed || _reader.Cancelled)) - { - SkipEntry(); - } if (_isDisposed) { return; } _isDisposed = true; + if (!(_completed || _reader.Cancelled)) + { + if (Utility.UseSyncOverAsyncDispose()) + { +#pragma warning disable VSTHRD002 // Avoid problematic synchronous waits +#pragma warning disable CA2012 + SkipEntryAsync().GetAwaiter().GetResult(); +#pragma warning restore CA2012 +#pragma warning restore VSTHRD002 // Avoid problematic synchronous waits + } + else + { + SkipEntry(); + } + } + + //Need a safe standard approach to this - it's okay for compression to overreads. Handling needs to be standardised + if (_stream is IStreamStack ss) + { + if ( + ss.GetStream() + is SharpCompress.Compressors.Deflate.DeflateStream deflateStream + ) + { + deflateStream.Flush(); //Deflate over reads. Knock it back + } + else if ( + ss.GetStream() + is SharpCompress.Compressors.LZMA.LzmaStream lzmaStream + ) + { + lzmaStream.Flush(); //Lzma over reads. Knock it back + } + } base.Dispose(disposing); _stream.Dispose(); } @@ -49,11 +83,13 @@ public class EntryStream : Stream public override void Flush() { } + public override Task FlushAsync(CancellationToken cancellationToken) => Task.CompletedTask; + public override long Length => _stream.Length; public override long Position { - get => throw new NotSupportedException(); + get => _stream.Position; //throw new NotSupportedException(); set => throw new NotSupportedException(); } @@ -83,4 +119,11 @@ public class EntryStream : Stream public override void Write(byte[] buffer, int offset, int count) => throw new NotSupportedException(); + + public override Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) => throw new NotSupportedException(); } diff --git a/src/SharpCompress/Common/ExtractionException.cs b/src/SharpCompress/Common/ExtractionException.cs deleted file mode 100644 index 4bc4f00c..00000000 --- a/src/SharpCompress/Common/ExtractionException.cs +++ /dev/null @@ -1,12 +0,0 @@ -using System; - -namespace SharpCompress.Common; - -public class ExtractionException : Exception -{ - public ExtractionException(string message) - : base(message) { } - - public ExtractionException(string message, Exception inner) - : base(message, inner) { } -} diff --git a/src/SharpCompress/Common/ExtractionMethods.cs b/src/SharpCompress/Common/ExtractionMethods.cs deleted file mode 100644 index c8e5896e..00000000 --- a/src/SharpCompress/Common/ExtractionMethods.cs +++ /dev/null @@ -1,117 +0,0 @@ -using System; -using System.IO; - -namespace SharpCompress.Common; - -internal static class ExtractionMethods -{ - /// - /// Extract to specific directory, retaining filename - /// - public static void WriteEntryToDirectory( - IEntry entry, - string destinationDirectory, - ExtractionOptions? options, - Action write - ) - { - string destinationFileName; - string fullDestinationDirectoryPath = Path.GetFullPath(destinationDirectory); - - //check for trailing slash. - if ( - fullDestinationDirectoryPath[fullDestinationDirectoryPath.Length - 1] - != Path.DirectorySeparatorChar - ) - { - fullDestinationDirectoryPath += Path.DirectorySeparatorChar; - } - - if (!Directory.Exists(fullDestinationDirectoryPath)) - { - throw new ExtractionException( - $"Directory does not exist to extract to: {fullDestinationDirectoryPath}" - ); - } - - options ??= new ExtractionOptions() { Overwrite = true }; - - string file = Path.GetFileName(entry.Key); - if (options.ExtractFullPath) - { - string folder = Path.GetDirectoryName(entry.Key)!; - string destdir = Path.GetFullPath(Path.Combine(fullDestinationDirectoryPath, folder)); - - if (!Directory.Exists(destdir)) - { - if (!destdir.StartsWith(fullDestinationDirectoryPath, StringComparison.Ordinal)) - { - throw new ExtractionException( - "Entry is trying to create a directory outside of the destination directory." - ); - } - - Directory.CreateDirectory(destdir); - } - destinationFileName = Path.Combine(destdir, file); - } - else - { - destinationFileName = Path.Combine(fullDestinationDirectoryPath, file); - } - - if (!entry.IsDirectory) - { - destinationFileName = Path.GetFullPath(destinationFileName); - - if ( - !destinationFileName.StartsWith( - fullDestinationDirectoryPath, - StringComparison.Ordinal - ) - ) - { - throw new ExtractionException( - "Entry is trying to write a file outside of the destination directory." - ); - } - write(destinationFileName, options); - } - else if (options.ExtractFullPath && !Directory.Exists(destinationFileName)) - { - Directory.CreateDirectory(destinationFileName); - } - } - - public static void WriteEntryToFile( - IEntry entry, - string destinationFileName, - ExtractionOptions? options, - Action openAndWrite - ) - { - if (entry.LinkTarget != null) - { - if (options?.WriteSymbolicLink is null) - { - throw new ExtractionException( - "Entry is a symbolic link but ExtractionOptions.WriteSymbolicLink delegate is null" - ); - } - options.WriteSymbolicLink(destinationFileName, entry.LinkTarget); - } - else - { - FileMode fm = FileMode.Create; - options ??= new ExtractionOptions() { Overwrite = true }; - - if (!options.Overwrite) - { - fm = FileMode.CreateNew; - } - - openAndWrite(destinationFileName, fm); - entry.PreserveExtractionOptions(destinationFileName, options); - } - } -} diff --git a/src/SharpCompress/Common/ExtractionOptions.cs b/src/SharpCompress/Common/ExtractionOptions.cs index 18c0f0ac..04b33416 100644 --- a/src/SharpCompress/Common/ExtractionOptions.cs +++ b/src/SharpCompress/Common/ExtractionOptions.cs @@ -1,40 +1,116 @@ using System; +using SharpCompress.Common.Options; namespace SharpCompress.Common; -public class ExtractionOptions +/// +/// Options for configuring extraction behavior when extracting archive entries. +/// +/// +/// Configure extraction behavior with constructors, property setters, or the with expression: +/// +/// var options = new ExtractionOptions { Overwrite = false }; +/// options = options with { PreserveFileTime = true }; +/// +/// +public sealed record ExtractionOptions : IExtractionOptions { /// - /// overwrite target if it exists + /// Overwrite target if it exists. + /// Breaking change: Default changed from false to true in version 0.40.0. /// - public bool Overwrite { get; set; } + public bool Overwrite { get; set; } = true; /// - /// extract with internal directory structure + /// Extract with internal directory structure. + /// Breaking change: Default changed from false to true in version 0.40.0. /// - public bool ExtractFullPath { get; set; } + public bool ExtractFullPath { get; set; } = true; /// - /// preserve file time + /// Preserve file time. + /// Breaking change: Default changed from false to true in version 0.40.0. /// - public bool PreserveFileTime { get; set; } + public bool PreserveFileTime { get; set; } = true; /// - /// preserve windows file attributes + /// Preserve windows file attributes. /// public bool PreserveAttributes { get; set; } /// - /// Delegate for writing symbolic links to disk. - /// sourcePath is where the symlink is created. - /// targetPath is what the symlink refers to. + /// Buffer size for extraction stream copy operations. /// - public delegate void SymbolicLinkWriterDelegate(string sourcePath, string targetPath); + public int BufferSize { get; set; } = Constants.BufferSize; - public SymbolicLinkWriterDelegate WriteSymbolicLink = (sourcePath, targetPath) => + /// + /// Validate archive entry checksums during extraction when checksum metadata is available. + /// + /// + /// Formats without payload checksums skip this validation. Compression-format integrity + /// checks that are required to decode data may still fail even when this is disabled. + /// + public bool CheckCrc { get; set; } = true; + + /// + /// Delegate for writing symbolic links to disk. + /// The first parameter is the source path (where the symlink is created). + /// The second parameter is the target path (what the symlink refers to). + /// + /// + /// Breaking change: Changed from field to property in version 0.40.0. + /// If no handler is provided, symbolic links are silently skipped during extraction. + /// + public Action? SymbolicLinkHandler { get; set; } + + /// + /// Creates a new ExtractionOptions instance with default values. + /// + public ExtractionOptions() { } + + /// + /// Creates a new ExtractionOptions instance with the specified overwrite behavior. + /// + /// Whether to overwrite existing files. + public ExtractionOptions(bool overwrite) => Overwrite = overwrite; + + /// + /// Creates a new ExtractionOptions instance with the specified extraction path and overwrite behavior. + /// + /// Whether to preserve directory structure. + /// Whether to overwrite existing files. + public ExtractionOptions(bool extractFullPath, bool overwrite) { - Console.WriteLine( - $"Could not write symlink {sourcePath} -> {targetPath}, for more information please see https://github.com/dotnet/runtime/issues/24271" - ); - }; + ExtractFullPath = extractFullPath; + Overwrite = overwrite; + } + + /// + /// Creates a new ExtractionOptions instance with the specified extraction path, overwrite behavior, and file time preservation. + /// + /// Whether to preserve directory structure. + /// Whether to overwrite existing files. + /// Whether to preserve file modification times. + public ExtractionOptions(bool extractFullPath, bool overwrite, bool preserveFileTime) + { + ExtractFullPath = extractFullPath; + Overwrite = overwrite; + PreserveFileTime = preserveFileTime; + } + + /// + /// Gets an ExtractionOptions instance configured for safe extraction (no overwrite). + /// + public static ExtractionOptions SafeExtract => new(overwrite: false); + + /// + /// Gets an ExtractionOptions instance configured for flat extraction (no directory structure). + /// + public static ExtractionOptions FlatExtract => new(extractFullPath: false, overwrite: true); + + /// + /// Gets an ExtractionOptions instance configured to preserve timestamps and attributes. + /// + public static ExtractionOptions PreserveMetadata => + new() { PreserveFileTime = true, PreserveAttributes = true }; } diff --git a/src/SharpCompress/Common/FilePart.cs b/src/SharpCompress/Common/FilePart.cs index 3c286d54..583dbf4b 100644 --- a/src/SharpCompress/Common/FilePart.cs +++ b/src/SharpCompress/Common/FilePart.cs @@ -1,17 +1,23 @@ using System.IO; +using System.Threading; +using System.Threading.Tasks; namespace SharpCompress.Common; public abstract class FilePart { - protected FilePart(ArchiveEncoding archiveEncoding) => ArchiveEncoding = archiveEncoding; + protected FilePart(IArchiveEncoding archiveEncoding) => ArchiveEncoding = archiveEncoding; - internal ArchiveEncoding ArchiveEncoding { get; } + internal IArchiveEncoding ArchiveEncoding { get; } - internal abstract string FilePartName { get; } + internal abstract string? FilePartName { get; } public int Index { get; set; } - internal abstract Stream GetCompressedStream(); + internal abstract Stream? GetCompressedStream(); internal abstract Stream? GetRawStream(); internal bool Skipped { get; set; } + + internal virtual ValueTask GetCompressedStreamAsync( + CancellationToken cancellationToken = default + ) => new(GetCompressedStream()); } diff --git a/src/SharpCompress/Common/FilePartExtractionBeginEventArgs.cs b/src/SharpCompress/Common/FilePartExtractionBeginEventArgs.cs deleted file mode 100644 index d5b8328c..00000000 --- a/src/SharpCompress/Common/FilePartExtractionBeginEventArgs.cs +++ /dev/null @@ -1,28 +0,0 @@ -using System; - -namespace SharpCompress.Common; - -public sealed class FilePartExtractionBeginEventArgs : EventArgs -{ - public FilePartExtractionBeginEventArgs(string name, long size, long compressedSize) - { - Name = name; - Size = size; - CompressedSize = compressedSize; - } - - /// - /// File name for the part for the current entry - /// - public string Name { get; } - - /// - /// Uncompressed size of the current entry in the part - /// - public long Size { get; } - - /// - /// Compressed size of the current entry in the part - /// - public long CompressedSize { get; } -} diff --git a/src/SharpCompress/Common/FlagUtility.cs b/src/SharpCompress/Common/FlagUtility.cs index 8f272b9d..47b7f7ea 100644 --- a/src/SharpCompress/Common/FlagUtility.cs +++ b/src/SharpCompress/Common/FlagUtility.cs @@ -46,7 +46,11 @@ internal static class FlagUtility /// Flag to test /// public static bool HasFlag(T bitField, T flag) - where T : struct => HasFlag(Convert.ToInt64(bitField), Convert.ToInt64(flag)); + where T : struct => + HasFlag( + Convert.ToInt64(bitField, Constants.DefaultCultureInfo), + Convert.ToInt64(flag, Constants.DefaultCultureInfo) + ); /// /// Returns true if the flag is set on the specified bit field. @@ -82,5 +86,10 @@ internal static class FlagUtility /// bool /// The flagged variable with the flag changed public static long SetFlag(T bitField, T flag, bool on) - where T : struct => SetFlag(Convert.ToInt64(bitField), Convert.ToInt64(flag), on); + where T : struct => + SetFlag( + Convert.ToInt64(bitField, Constants.DefaultCultureInfo), + Convert.ToInt64(flag, Constants.DefaultCultureInfo), + on + ); } diff --git a/src/SharpCompress/Common/GZip/GZipChecksumValidationStream.cs b/src/SharpCompress/Common/GZip/GZipChecksumValidationStream.cs new file mode 100644 index 00000000..512d7cc2 --- /dev/null +++ b/src/SharpCompress/Common/GZip/GZipChecksumValidationStream.cs @@ -0,0 +1,165 @@ +using System; +using System.Buffers.Binary; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Crypto; + +namespace SharpCompress.Common.GZip; + +internal sealed class GZipChecksumValidationStream : Stream +{ + private readonly Stream _source; + private readonly Stream _rawStream; + private readonly string _entryName; + private readonly uint? _expectedCrc; + private readonly uint? _expectedSize; + private readonly uint[] _crc32Table; + private uint _seed = Crc32Stream.DEFAULT_SEED; + private uint _size; + private bool _validated; + + internal GZipChecksumValidationStream( + Stream source, + Stream rawStream, + string? entryName, + uint? expectedCrc, + uint? expectedSize + ) + { + _source = source; + _rawStream = rawStream; + _entryName = string.IsNullOrEmpty(entryName) ? "Entry" : entryName!; + _expectedCrc = expectedCrc; + _expectedSize = expectedSize; + _crc32Table = Crc32Stream.InitializeTable(Crc32Stream.DEFAULT_POLYNOMIAL); + } + + public override bool CanRead => _source.CanRead; + public override bool CanSeek => false; + public override bool CanWrite => false; + public override long Length => _source.Length; + + public override long Position + { + get => _source.Position; + set => throw new NotSupportedException(); + } + + public override void Flush() => _source.Flush(); + + public override Task FlushAsync(CancellationToken cancellationToken) => + _source.FlushAsync(cancellationToken); + + public override int Read(byte[] buffer, int offset, int count) + { + var read = _source.Read(buffer, offset, count); + UpdateAndValidateAtEof(buffer.AsSpan(offset, read), read); + return read; + } + +#if !LEGACY_DOTNET + public override int Read(Span buffer) + { + var read = _source.Read(buffer); + UpdateAndValidateAtEof(buffer[..read], read); + return read; + } +#endif + + public override int ReadByte() + { + var value = _source.ReadByte(); + if (value == -1) + { + Validate(); + } + else + { + _seed = Crc32Stream.CalculateCrc(_crc32Table, _seed, (byte)value); + _size++; + } + + return value; + } + + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + var read = await _source + .ReadAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + UpdateAndValidateAtEof(buffer.AsSpan(offset, read), read); + return read; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + var read = await _source.ReadAsync(buffer, cancellationToken).ConfigureAwait(false); + UpdateAndValidateAtEof(buffer.Span[..read], read); + return read; + } +#endif + + public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); + + public override void SetLength(long value) => throw new NotSupportedException(); + + public override void Write(byte[] buffer, int offset, int count) => + throw new NotSupportedException(); + + private void UpdateAndValidateAtEof(ReadOnlySpan buffer, int read) + { + if (read > 0) + { + _seed = Crc32Stream.CalculateCrc(_crc32Table, _seed, buffer); + _size += unchecked((uint)read); + return; + } + + Validate(); + } + + private void Validate() + { + if (_validated) + { + return; + } + + _validated = true; + + var expectedCrc = _expectedCrc; + var expectedSize = _expectedSize; + if (!expectedCrc.HasValue || !expectedSize.HasValue) + { + Span trailer = stackalloc byte[8]; + _rawStream.ReadFully(trailer); + expectedCrc = BinaryPrimitives.ReadUInt32LittleEndian(trailer); + expectedSize = BinaryPrimitives.ReadUInt32LittleEndian(trailer[4..]); + } + + var actualCrc = ~_seed; + if (actualCrc != expectedCrc.Value) + { + throw new InvalidFormatException( + $"CRC mismatch for entry '{_entryName}'. Expected 0x{expectedCrc.Value:X8}, actual 0x{actualCrc:X8}." + ); + } + + if (_size != expectedSize.Value) + { + throw new InvalidFormatException( + $"Size mismatch for entry '{_entryName}'. Expected {expectedSize.Value}, actual {_size}." + ); + } + } +} diff --git a/src/SharpCompress/Common/GZip/GZipEntry.Async.cs b/src/SharpCompress/Common/GZip/GZipEntry.Async.cs new file mode 100644 index 00000000..7bb5f983 --- /dev/null +++ b/src/SharpCompress/Common/GZip/GZipEntry.Async.cs @@ -0,0 +1,21 @@ +using System.Collections.Generic; +using System.IO; +using SharpCompress.Readers; + +namespace SharpCompress.Common.GZip; + +public partial class GZipEntry +{ + internal static async IAsyncEnumerable GetEntriesAsync( + Stream stream, + ReaderOptions options + ) + { + yield return new GZipEntry( + await GZipFilePart + .CreateAsync(stream, options.ArchiveEncoding, options.Providers) + .ConfigureAwait(false), + options + ); + } +} diff --git a/src/SharpCompress/Common/GZip/GZipEntry.cs b/src/SharpCompress/Common/GZip/GZipEntry.cs index bb9a22da..48f08d46 100644 --- a/src/SharpCompress/Common/GZip/GZipEntry.cs +++ b/src/SharpCompress/Common/GZip/GZipEntry.cs @@ -1,28 +1,37 @@ -using System; +using System; using System.Collections.Generic; using System.IO; +using SharpCompress.Common.Options; +using SharpCompress.Readers; namespace SharpCompress.Common.GZip; -public class GZipEntry : Entry +public partial class GZipEntry : Entry { - private readonly GZipFilePart _filePart; + private readonly GZipFilePart? _filePart; - internal GZipEntry(GZipFilePart filePart) => _filePart = filePart; + internal GZipEntry(GZipFilePart? filePart, IReaderOptions readerOptions) + : base(readerOptions) + { + _filePart = filePart; + } public override CompressionType CompressionType => CompressionType.GZip; - public override long Crc => _filePart.Crc ?? 0; + public override long Crc => _filePart?.Crc ?? 0; - public override string Key => _filePart.FilePartName; + internal override Stream WrapWithChecksumValidation(Stream source, ExtractionOptions options) => + _filePart?.WrapWithChecksumValidation(source, Key) ?? source; + + public override string? Key => _filePart?.FilePartName; public override string? LinkTarget => null; public override long CompressedSize => 0; - public override long Size => _filePart.UncompressedSize ?? 0; + public override long Size => _filePart?.UncompressedSize ?? 0; - public override DateTime? LastModifiedTime => _filePart.DateModified; + public override DateTime? LastModifiedTime => _filePart?.DateModified; public override DateTime? CreatedTime => null; @@ -36,10 +45,15 @@ public class GZipEntry : Entry public override bool IsSplitAfter => false; - internal override IEnumerable Parts => _filePart.AsEnumerable(); + internal override IEnumerable Parts => _filePart.Empty(); - internal static IEnumerable GetEntries(Stream stream, OptionsBase options) + internal static IEnumerable GetEntries(Stream stream, ReaderOptions options) { - yield return new GZipEntry(new GZipFilePart(stream, options.ArchiveEncoding)); + yield return new GZipEntry( + GZipFilePart.Create(stream, options.ArchiveEncoding, options.Providers), + options + ); } + + // Async methods moved to GZipEntry.Async.cs } diff --git a/src/SharpCompress/Common/GZip/GZipFilePart.Async.cs b/src/SharpCompress/Common/GZip/GZipFilePart.Async.cs new file mode 100644 index 00000000..ae3e118a --- /dev/null +++ b/src/SharpCompress/Common/GZip/GZipFilePart.Async.cs @@ -0,0 +1,147 @@ +using System; +using System.Buffers.Binary; +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Tar.Headers; +using SharpCompress.Compressors; +using SharpCompress.Compressors.Deflate; +using SharpCompress.Providers; + +namespace SharpCompress.Common.GZip; + +internal sealed partial class GZipFilePart +{ + internal static async ValueTask CreateAsync( + Stream stream, + IArchiveEncoding archiveEncoding, + CompressionProviderRegistry compressionProviders, + CancellationToken cancellationToken = default + ) + { + var part = new GZipFilePart(stream, archiveEncoding, compressionProviders); + + await part.ReadAndValidateGzipHeaderAsync(cancellationToken).ConfigureAwait(false); + if (stream.CanSeek) + { + var position = stream.Position; + stream.Position = stream.Length - 8; + await part.ReadTrailerAsync(cancellationToken).ConfigureAwait(false); + stream.Position = position; + part.EntryStartPosition = position; + } + else + { + // For non-seekable streams, we can't read the trailer or track position. + // Set to 0 since the stream will be read sequentially from its current position. + part.EntryStartPosition = 0; + } + return part; + } + + private async ValueTask ReadTrailerAsync(CancellationToken cancellationToken = default) + { + // Read and potentially verify the GZIP trailer: CRC32 and size mod 2^32 + var trailer = new byte[8]; + _ = await _stream.ReadFullyAsync(trailer, 0, 8, cancellationToken).ConfigureAwait(false); + + Crc = BinaryPrimitives.ReadUInt32LittleEndian(trailer); + UncompressedSize = BinaryPrimitives.ReadUInt32LittleEndian(trailer.AsSpan().Slice(4)); + } + + private async ValueTask ReadAndValidateGzipHeaderAsync( + CancellationToken cancellationToken = default + ) + { + // read the header on the first read + var header = new byte[10]; + var n = await _stream.ReadAsync(header, 0, 10, cancellationToken).ConfigureAwait(false); + + // workitem 8501: handle edge case (decompress empty stream) + if (n == 0) + { + return; + } + + if (n != 10) + { + throw new ZlibException("Not a valid GZIP stream."); + } + + if (header[0] != 0x1F || header[1] != 0x8B || header[2] != 8) + { + throw new ZlibException("Bad GZIP header."); + } + + var timet = BinaryPrimitives.ReadInt32LittleEndian(header.AsSpan().Slice(4)); + DateModified = TarHeader.EPOCH.AddSeconds(timet); + if ((header[3] & 0x04) == 0x04) + { + // read and discard extra field + var lengthField = new byte[2]; + _ = await _stream.ReadAsync(lengthField, 0, 2, cancellationToken).ConfigureAwait(false); + + var extraLength = (short)(lengthField[0] + (lengthField[1] * 256)); + var extra = new byte[extraLength]; + + if (!await _stream.ReadFullyAsync(extra, cancellationToken).ConfigureAwait(false)) + { + throw new ZlibException("Unexpected end-of-file reading GZIP header."); + } + } + if ((header[3] & 0x08) == 0x08) + { + _name = await ReadZeroTerminatedStringAsync(_stream, cancellationToken) + .ConfigureAwait(false); + } + if ((header[3] & 0x10) == 0x010) + { + await ReadZeroTerminatedStringAsync(_stream, cancellationToken).ConfigureAwait(false); + } + if ((header[3] & 0x02) == 0x02) + { + var buf = new byte[1]; + _ = await _stream.ReadAsync(buf, 0, 1, cancellationToken).ConfigureAwait(false); // CRC16, ignore + } + } + + private async ValueTask ReadZeroTerminatedStringAsync( + Stream stream, + CancellationToken cancellationToken = default + ) + { + var buf1 = new byte[1]; + var list = new List(); + var done = false; + do + { + // workitem 7740 + var n = await stream.ReadAsync(buf1, 0, 1, cancellationToken).ConfigureAwait(false); + if (n != 1) + { + throw new ZlibException("Unexpected EOF reading GZIP header."); + } + if (buf1[0] == 0) + { + done = true; + } + else + { + list.Add(buf1[0]); + } + } while (!done); + var buffer = list.ToArray(); + return ArchiveEncoding.Decode(buffer); + } + + internal override async ValueTask GetCompressedStreamAsync( + CancellationToken cancellationToken = default + ) + { + // GZip uses Deflate compression + return await _compressionProviders + .CreateDecompressStreamAsync(CompressionType.Deflate, _stream, cancellationToken) + .ConfigureAwait(false); + } +} diff --git a/src/SharpCompress/Common/GZip/GZipFilePart.cs b/src/SharpCompress/Common/GZip/GZipFilePart.cs index fbf4ee45..13a1c968 100644 --- a/src/SharpCompress/Common/GZip/GZipFilePart.cs +++ b/src/SharpCompress/Common/GZip/GZipFilePart.cs @@ -5,39 +5,70 @@ using System.IO; using SharpCompress.Common.Tar.Headers; using SharpCompress.Compressors; using SharpCompress.Compressors.Deflate; +using SharpCompress.IO; +using SharpCompress.Providers; namespace SharpCompress.Common.GZip; -internal sealed class GZipFilePart : FilePart +internal sealed partial class GZipFilePart : FilePart { private string? _name; private readonly Stream _stream; + private readonly CompressionProviderRegistry _compressionProviders; - internal GZipFilePart(Stream stream, ArchiveEncoding archiveEncoding) - : base(archiveEncoding) + internal static GZipFilePart Create( + Stream stream, + IArchiveEncoding archiveEncoding, + CompressionProviderRegistry compressionProviders + ) { - _stream = stream; - ReadAndValidateGzipHeader(); + var part = new GZipFilePart(stream, archiveEncoding, compressionProviders); + + part.ReadAndValidateGzipHeader(); if (stream.CanSeek) { var position = stream.Position; stream.Position = stream.Length - 8; - ReadTrailer(); + part.ReadTrailer(); stream.Position = position; + part.EntryStartPosition = position; } - EntryStartPosition = stream.Position; + else + { + // For non-seekable streams, we can't read the trailer or track position. + // Set to 0 since the stream will be read sequentially from its current position. + part.EntryStartPosition = 0; + } + return part; } - internal long EntryStartPosition { get; } + private GZipFilePart( + Stream stream, + IArchiveEncoding archiveEncoding, + CompressionProviderRegistry compressionProviders + ) + : base(archiveEncoding) + { + _stream = SharpCompressStream.Create(stream); + _compressionProviders = compressionProviders; + } + + internal long EntryStartPosition { get; private set; } internal DateTime? DateModified { get; private set; } internal uint? Crc { get; private set; } internal uint? UncompressedSize { get; private set; } - internal override string FilePartName => _name!; + internal override string? FilePartName => _name; - internal override Stream GetCompressedStream() => - new DeflateStream(_stream, CompressionMode.Decompress, CompressionLevel.Default); + internal override Stream GetCompressedStream() + { + //GZip uses Deflate compression, at this point we need a deflate stream + return _compressionProviders.CreateDecompressStream(CompressionType.Deflate, _stream); + } + + internal Stream WrapWithChecksumValidation(Stream source, string? entryName) => + new GZipChecksumValidationStream(source, _stream, entryName, Crc, UncompressedSize); internal override Stream GetRawStream() => _stream; @@ -45,7 +76,7 @@ internal sealed class GZipFilePart : FilePart { // Read and potentially verify the GZIP trailer: CRC32 and size mod 2^32 Span trailer = stackalloc byte[8]; - var n = _stream.Read(trailer); + _stream.ReadFully(trailer); Crc = BinaryPrimitives.ReadUInt32LittleEndian(trailer); UncompressedSize = BinaryPrimitives.ReadUInt32LittleEndian(trailer.Slice(4)); diff --git a/src/SharpCompress/Common/GZip/GZipVolume.cs b/src/SharpCompress/Common/GZip/GZipVolume.cs index 0dd9b8d9..b753c3ad 100644 --- a/src/SharpCompress/Common/GZip/GZipVolume.cs +++ b/src/SharpCompress/Common/GZip/GZipVolume.cs @@ -5,11 +5,11 @@ namespace SharpCompress.Common.GZip; public class GZipVolume : Volume { - public GZipVolume(Stream stream, ReaderOptions options, int index = 0) + public GZipVolume(Stream stream, ReaderOptions options, int index) : base(stream, options, index) { } public GZipVolume(FileInfo fileInfo, ReaderOptions options) - : base(fileInfo.OpenRead(), options) => options.LeaveStreamOpen = false; + : base(fileInfo.OpenRead(), options.WithLeaveStreamOpen(false)) { } public override bool IsFirstVolume => true; diff --git a/src/SharpCompress/Common/IArchiveEncoding.cs b/src/SharpCompress/Common/IArchiveEncoding.cs new file mode 100644 index 00000000..92fe0093 --- /dev/null +++ b/src/SharpCompress/Common/IArchiveEncoding.cs @@ -0,0 +1,36 @@ +using System; +using System.Text; + +namespace SharpCompress.Common; + +/// +/// Defines the encoding settings for archives. +/// +public interface IArchiveEncoding +{ + /// + /// Default encoding to use when archive format doesn't specify one. Required and defaults to Encoding.Default. + /// + public Encoding Default { get; set; } + + /// + /// ArchiveEncoding used by encryption schemes which don't comply with RFC 2898. Required and defaults to Encoding.Default. + /// + public Encoding Password { get; set; } + + /// + /// Default encoding to use when archive format specifies UTF-8 encoding. Required and defaults to Encoding.UTF8. + /// + public Encoding UTF8 { get; set; } + + /// + /// Set this encoding when you want to force it for all encoding operations. + /// + public Encoding? Forced { get; set; } + + /// + /// Set this when you want to use a custom method for all decoding operations. + /// + /// string Func(bytes, index, length, EncodingType) + public Func? CustomDecoder { get; set; } +} diff --git a/src/SharpCompress/Common/IEntry.Extensions.cs b/src/SharpCompress/Common/IEntry.Extensions.cs deleted file mode 100644 index 7e9b79a3..00000000 --- a/src/SharpCompress/Common/IEntry.Extensions.cs +++ /dev/null @@ -1,50 +0,0 @@ -using System.IO; - -namespace SharpCompress.Common; - -internal static class EntryExtensions -{ - internal static void PreserveExtractionOptions( - this IEntry entry, - string destinationFileName, - ExtractionOptions options - ) - { - if (options.PreserveFileTime || options.PreserveAttributes) - { - var nf = new FileInfo(destinationFileName); - if (!nf.Exists) - { - return; - } - - // update file time to original packed time - if (options.PreserveFileTime) - { - if (entry.CreatedTime.HasValue) - { - nf.CreationTime = entry.CreatedTime.Value; - } - - if (entry.LastModifiedTime.HasValue) - { - nf.LastWriteTime = entry.LastModifiedTime.Value; - } - - if (entry.LastAccessedTime.HasValue) - { - nf.LastAccessTime = entry.LastAccessedTime.Value; - } - } - - if (options.PreserveAttributes) - { - if (entry.Attrib.HasValue) - { - nf.Attributes = (FileAttributes) - System.Enum.ToObject(typeof(FileAttributes), entry.Attrib.Value); - } - } - } - } -} diff --git a/src/SharpCompress/Common/IEntry.cs b/src/SharpCompress/Common/IEntry.cs index df1fa603..741b4825 100644 --- a/src/SharpCompress/Common/IEntry.cs +++ b/src/SharpCompress/Common/IEntry.cs @@ -1,4 +1,5 @@ using System; +using SharpCompress.Common.Options; namespace SharpCompress.Common; @@ -9,7 +10,7 @@ public interface IEntry long CompressedSize { get; } long Crc { get; } DateTime? CreatedTime { get; } - string Key { get; } + string? Key { get; } string? LinkTarget { get; } bool IsDirectory { get; } bool IsEncrypted { get; } @@ -21,4 +22,9 @@ public interface IEntry DateTime? LastModifiedTime { get; } long Size { get; } int? Attrib { get; } + + /// + /// The options used when opening this entry's source (reader or archive). + /// + IReaderOptions Options { get; } } diff --git a/src/SharpCompress/Common/IEntryExtensions.Async.cs b/src/SharpCompress/Common/IEntryExtensions.Async.cs new file mode 100644 index 00000000..4ef697e0 --- /dev/null +++ b/src/SharpCompress/Common/IEntryExtensions.Async.cs @@ -0,0 +1,106 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Common; + +internal static partial class IEntryExtensions +{ + extension(IEntry entry) + { + internal async ValueTask WriteEntryToDirectoryAsync( + string destinationDirectory, + ExtractionOptions? options, + Func writeAsync, + CancellationToken cancellationToken = default + ) + { + options ??= new ExtractionOptions(); + var fullDestinationDirectoryPath = DirectoryManagement.GetFullDestinationDirectoryPath( + destinationDirectory + ); + + await WriteEntryToDirectoryAsyncCore( + entry, + fullDestinationDirectoryPath, + options, + writeAsync, + cancellationToken + ) + .ConfigureAwait(false); + } + + internal async ValueTask WriteEntryToDirectoryAsyncCore( + string fullDestinationDirectoryPath, + ExtractionOptions options, + Func? writeAsync, + CancellationToken cancellationToken = default + ) + { + var destinationFileName = GetEntryDestinationFileName( + entry, + fullDestinationDirectoryPath, + options + ); + + if (!entry.IsDirectory) + { + destinationFileName = Path.GetFullPath(destinationFileName); + + DirectoryManagement.EnsurePathInDestinationDirectory( + destinationFileName, + fullDestinationDirectoryPath, + DirectoryManagement.WriteFileOutsideDestinationMessage + ); + + if (writeAsync != null) + { + await writeAsync(destinationFileName, cancellationToken).ConfigureAwait(false); + } + } + else if (options.ExtractFullPath) + { + destinationFileName = Path.GetFullPath(destinationFileName); + + DirectoryManagement.EnsurePathInDestinationDirectory( + destinationFileName, + fullDestinationDirectoryPath, + DirectoryManagement.CreateDirectoryOutsideDestinationMessage + ); + + if (!Directory.Exists(destinationFileName)) + { + Directory.CreateDirectory(destinationFileName); + } + } + } + + public async ValueTask WriteEntryToFileAsync( + string destinationFileName, + ExtractionOptions? options, + Func openAndWriteAsync, + CancellationToken cancellationToken = default + ) + { + options ??= new ExtractionOptions(); + if (entry.LinkTarget != null) + { + options.SymbolicLinkHandler?.Invoke(destinationFileName, entry.LinkTarget); + } + else + { + var fm = FileMode.Create; + + if (!options.Overwrite) + { + fm = FileMode.CreateNew; + } + + await openAndWriteAsync(destinationFileName, fm, cancellationToken) + .ConfigureAwait(false); + entry.PreserveExtractionOptions(destinationFileName, options); + } + } + } +} diff --git a/src/SharpCompress/Common/IEntryExtensions.cs b/src/SharpCompress/Common/IEntryExtensions.cs new file mode 100644 index 00000000..492ad85d --- /dev/null +++ b/src/SharpCompress/Common/IEntryExtensions.cs @@ -0,0 +1,205 @@ +using System; +using System.IO; + +namespace SharpCompress.Common; + +internal static partial class IEntryExtensions +{ + internal static Stream WrapWithChecksumValidation( + IEntry entry, + Stream source, + ExtractionOptions? options + ) + { + options ??= new ExtractionOptions(); + if (options.CheckCrc && entry is Entry typedEntry) + { + return typedEntry.WrapWithChecksumValidation(source, options); + } + + return source; + } + + extension(IEntry entry) + { + /// + /// Extract to specific directory, retaining filename + /// + internal void WriteEntryToDirectory( + string destinationDirectory, + ExtractionOptions? options, + Action write + ) + { + options ??= new ExtractionOptions(); + var fullDestinationDirectoryPath = DirectoryManagement.GetFullDestinationDirectoryPath( + destinationDirectory + ); + + WriteEntryToDirectoryCore(entry, fullDestinationDirectoryPath, options, write); + } + + internal void WriteEntryToDirectoryCore( + string fullDestinationDirectoryPath, + ExtractionOptions options, + Action? write + ) + { + var destinationFileName = GetEntryDestinationFileName( + entry, + fullDestinationDirectoryPath, + options + ); + + if (!entry.IsDirectory) + { + destinationFileName = Path.GetFullPath(destinationFileName); + + DirectoryManagement.EnsurePathInDestinationDirectory( + destinationFileName, + fullDestinationDirectoryPath, + DirectoryManagement.WriteFileOutsideDestinationMessage + ); + write?.Invoke(destinationFileName); + } + else if (options.ExtractFullPath) + { + destinationFileName = Path.GetFullPath(destinationFileName); + + DirectoryManagement.EnsurePathInDestinationDirectory( + destinationFileName, + fullDestinationDirectoryPath, + DirectoryManagement.CreateDirectoryOutsideDestinationMessage + ); + + if (!Directory.Exists(destinationFileName)) + { + Directory.CreateDirectory(destinationFileName); + } + } + } + + private string GetEntryDestinationFileName( + string fullDestinationDirectoryPath, + ExtractionOptions options + ) + { + var file = Path.GetFileName(entry.Key.NotNull("Entry Key is null")) + .NotNull("File is null"); + file = Utility.ReplaceInvalidFileNameChars(file); + + if (options.ExtractFullPath) + { + var folder = Path.GetDirectoryName(entry.Key.NotNull("Entry Key is null")) + .NotNull("Directory is null"); + var destdir = Path.GetFullPath(Path.Combine(fullDestinationDirectoryPath, folder)); + + DirectoryManagement.EnsurePathInDestinationDirectory( + destdir, + fullDestinationDirectoryPath, + entry.IsDirectory + ? DirectoryManagement.CreateDirectoryOutsideDestinationMessage + : DirectoryManagement.WriteFileOutsideDestinationMessage + ); + + if (!Directory.Exists(destdir)) + { + Directory.CreateDirectory(destdir); + } + + return Path.Combine(destdir, file); + } + + return Path.Combine(fullDestinationDirectoryPath, file); + } + + public void WriteEntryToFile( + string destinationFileName, + ExtractionOptions? options, + Action openAndWrite + ) + { + options ??= new ExtractionOptions(); + if (entry.LinkTarget != null) + { + options.SymbolicLinkHandler?.Invoke(destinationFileName, entry.LinkTarget); + } + else + { + var fm = FileMode.Create; + + if (!options.Overwrite) + { + fm = FileMode.CreateNew; + } + + openAndWrite(destinationFileName, fm); + entry.PreserveExtractionOptions(destinationFileName, options); + } + } + + internal void PreserveExtractionOptions( + string destinationFileName, + ExtractionOptions options + ) + { + if (options.PreserveFileTime || options.PreserveAttributes) + { + var nf = new FileInfo(destinationFileName); + if (!nf.Exists) + { + return; + } + + // update file time to original packed time + if (options.PreserveFileTime) + { + if (entry.CreatedTime.HasValue) + { + try + { + nf.CreationTime = entry.CreatedTime.Value; + } + catch + { + // Invalid time or the OS rejected + } + } + + if (entry.LastModifiedTime.HasValue) + { + try + { + nf.LastWriteTime = entry.LastModifiedTime.Value; + } + catch + { + // Invalid time or the OS rejected + } + } + + if (entry.LastAccessedTime.HasValue) + { + try + { + nf.LastAccessTime = entry.LastAccessedTime.Value; + } + catch + { + // Invalid time or the OS rejected + } + } + } + + if (options.PreserveAttributes) + { + if (entry.Attrib.HasValue) + { + nf.Attributes = (FileAttributes) + Enum.ToObject(typeof(FileAttributes), entry.Attrib.Value); + } + } + } + } + } +} diff --git a/src/SharpCompress/Common/IExtractionListener.cs b/src/SharpCompress/Common/IExtractionListener.cs deleted file mode 100644 index e1389810..00000000 --- a/src/SharpCompress/Common/IExtractionListener.cs +++ /dev/null @@ -1,7 +0,0 @@ -namespace SharpCompress.Common; - -public interface IExtractionListener -{ - void FireFilePartExtractionBegin(string name, long size, long compressedSize); - void FireCompressedBytesRead(long currentPartCompressedBytes, long compressedReadBytes); -} diff --git a/src/SharpCompress/Common/IVolume.cs b/src/SharpCompress/Common/IVolume.cs index abbc0406..7b0e5a2c 100644 --- a/src/SharpCompress/Common/IVolume.cs +++ b/src/SharpCompress/Common/IVolume.cs @@ -2,9 +2,9 @@ using System; namespace SharpCompress.Common; -public interface IVolume : IDisposable +public interface IVolume : IDisposable, IAsyncDisposable { int Index { get; } - string FileName { get; } + string? FileName { get; } } diff --git a/src/SharpCompress/Common/IncompleteArchiveException.cs b/src/SharpCompress/Common/IncompleteArchiveException.cs deleted file mode 100644 index a033001a..00000000 --- a/src/SharpCompress/Common/IncompleteArchiveException.cs +++ /dev/null @@ -1,7 +0,0 @@ -namespace SharpCompress.Common; - -public class IncompleteArchiveException : ArchiveException -{ - public IncompleteArchiveException(string message) - : base(message) { } -} diff --git a/src/SharpCompress/Common/InvalidFormatException.cs b/src/SharpCompress/Common/InvalidFormatException.cs deleted file mode 100644 index 8f14df14..00000000 --- a/src/SharpCompress/Common/InvalidFormatException.cs +++ /dev/null @@ -1,12 +0,0 @@ -using System; - -namespace SharpCompress.Common; - -public class InvalidFormatException : ExtractionException -{ - public InvalidFormatException(string message) - : base(message) { } - - public InvalidFormatException(string message, Exception inner) - : base(message, inner) { } -} diff --git a/src/SharpCompress/Common/IsExternalInit.cs b/src/SharpCompress/Common/IsExternalInit.cs new file mode 100644 index 00000000..d6d1f7c2 --- /dev/null +++ b/src/SharpCompress/Common/IsExternalInit.cs @@ -0,0 +1,18 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +// This file is required for init-only properties to work on older target frameworks (.NET Framework 4.8, .NET Standard 2.0) +// The IsExternalInit type is used by the compiler for records and init-only properties + +#if NETFRAMEWORK || NETSTANDARD2_0 || NETSTANDARD2_1 +using System.ComponentModel; + +namespace System.Runtime.CompilerServices; + +/// +/// Reserved to be used by the compiler for tracking metadata. +/// This class should not be used by developers in source code. +/// +[EditorBrowsable(EditorBrowsableState.Never)] +internal static class IsExternalInit { } +#endif diff --git a/src/SharpCompress/Common/Lzw/LzwEntry.Async.cs b/src/SharpCompress/Common/Lzw/LzwEntry.Async.cs new file mode 100644 index 00000000..d42ceb41 --- /dev/null +++ b/src/SharpCompress/Common/Lzw/LzwEntry.Async.cs @@ -0,0 +1,24 @@ +using System.Collections.Generic; +using System.IO; +using System.Runtime.CompilerServices; +using System.Threading; +using SharpCompress.Readers; + +namespace SharpCompress.Common.Lzw; + +public partial class LzwEntry +{ + internal static async IAsyncEnumerable GetEntriesAsync( + Stream stream, + ReaderOptions options, + [EnumeratorCancellation] CancellationToken cancellationToken = default + ) + { + yield return new LzwEntry( + await LzwFilePart + .CreateAsync(stream, options.ArchiveEncoding, options.Providers, cancellationToken) + .ConfigureAwait(false), + options + ); + } +} diff --git a/src/SharpCompress/Common/Lzw/LzwEntry.cs b/src/SharpCompress/Common/Lzw/LzwEntry.cs new file mode 100644 index 00000000..ba795368 --- /dev/null +++ b/src/SharpCompress/Common/Lzw/LzwEntry.cs @@ -0,0 +1,56 @@ +using System; +using System.Collections.Generic; +using System.IO; +using SharpCompress.Common.Options; +using SharpCompress.Readers; + +namespace SharpCompress.Common.Lzw; + +public partial class LzwEntry : Entry +{ + private readonly LzwFilePart? _filePart; + + internal LzwEntry(LzwFilePart? filePart, IReaderOptions readerOptions) + : base(readerOptions) + { + _filePart = filePart; + } + + public override CompressionType CompressionType => CompressionType.Lzw; + + public override long Crc => 0; + + public override string? Key => _filePart?.FilePartName; + + public override string? LinkTarget => null; + + public override long CompressedSize => 0; + + public override long Size => 0; + + public override DateTime? LastModifiedTime => null; + + public override DateTime? CreatedTime => null; + + public override DateTime? LastAccessedTime => null; + + public override DateTime? ArchivedTime => null; + + public override bool IsEncrypted => false; + + public override bool IsDirectory => false; + + public override bool IsSplitAfter => false; + + internal override IEnumerable Parts => _filePart.Empty(); + + internal static IEnumerable GetEntries(Stream stream, ReaderOptions options) + { + yield return new LzwEntry( + LzwFilePart.Create(stream, options.ArchiveEncoding, options.Providers), + options + ); + } + + // Async methods moved to LzwEntry.Async.cs +} diff --git a/src/SharpCompress/Common/Lzw/LzwFilePart.Async.cs b/src/SharpCompress/Common/Lzw/LzwFilePart.Async.cs new file mode 100644 index 00000000..f0dd25b6 --- /dev/null +++ b/src/SharpCompress/Common/Lzw/LzwFilePart.Async.cs @@ -0,0 +1,35 @@ +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Providers; + +namespace SharpCompress.Common.Lzw; + +internal sealed partial class LzwFilePart +{ + internal static async ValueTask CreateAsync( + Stream stream, + IArchiveEncoding archiveEncoding, + CompressionProviderRegistry compressionProviders, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + var part = new LzwFilePart(stream, archiveEncoding, compressionProviders); + + // For non-seekable streams, we can't track position, so use 0 since the stream will be + // read sequentially from its current position. + part.EntryStartPosition = stream.CanSeek ? stream.Position : 0; + return part; + } + + internal override async ValueTask GetCompressedStreamAsync( + CancellationToken cancellationToken = default + ) + { + return await _compressionProviders + .CreateDecompressStreamAsync(CompressionType.Lzw, _stream, cancellationToken) + .ConfigureAwait(false); + } +} diff --git a/src/SharpCompress/Common/Lzw/LzwFilePart.cs b/src/SharpCompress/Common/Lzw/LzwFilePart.cs new file mode 100644 index 00000000..08802d2f --- /dev/null +++ b/src/SharpCompress/Common/Lzw/LzwFilePart.cs @@ -0,0 +1,71 @@ +using System.IO; +using SharpCompress.Common; +using SharpCompress.Providers; + +namespace SharpCompress.Common.Lzw; + +internal sealed partial class LzwFilePart : FilePart +{ + private readonly Stream _stream; + private readonly string? _name; + private readonly CompressionProviderRegistry _compressionProviders; + + internal static LzwFilePart Create( + Stream stream, + IArchiveEncoding archiveEncoding, + CompressionProviderRegistry compressionProviders + ) + { + var part = new LzwFilePart(stream, archiveEncoding, compressionProviders); + + // For non-seekable streams, we can't track position, so use 0 since the stream will be + // read sequentially from its current position. + part.EntryStartPosition = stream.CanSeek ? stream.Position : 0; + return part; + } + + private LzwFilePart( + Stream stream, + IArchiveEncoding archiveEncoding, + CompressionProviderRegistry compressionProviders + ) + : base(archiveEncoding) + { + _stream = stream; + _name = DeriveFileName(stream); + _compressionProviders = compressionProviders; + } + + internal long EntryStartPosition { get; private set; } + + internal override string? FilePartName => _name; + + internal override Stream GetCompressedStream() => + _compressionProviders.CreateDecompressStream(CompressionType.Lzw, _stream); + + internal override Stream GetRawStream() => _stream; + + private static string? DeriveFileName(Stream stream) + { + // Unwrap SharpCompressStream to get to the underlying FileStream + var unwrappedStream = stream; + if (stream is SharpCompress.IO.IStreamStack streamStack) + { + unwrappedStream = streamStack.BaseStream(); + } + + // Try to derive filename from FileStream + if (unwrappedStream is FileStream fileStream && !string.IsNullOrEmpty(fileStream.Name)) + { + var fileName = Path.GetFileName(fileStream.Name); + // Strip .Z extension if present + if (fileName.EndsWith(".Z", System.StringComparison.OrdinalIgnoreCase)) + { + return fileName.Substring(0, fileName.Length - 2); + } + return fileName; + } + // Default name for non-file streams + return "data"; + } +} diff --git a/src/SharpCompress/Common/Lzw/LzwVolume.cs b/src/SharpCompress/Common/Lzw/LzwVolume.cs new file mode 100644 index 00000000..be50f106 --- /dev/null +++ b/src/SharpCompress/Common/Lzw/LzwVolume.cs @@ -0,0 +1,17 @@ +using System.IO; +using SharpCompress.Readers; + +namespace SharpCompress.Common.Lzw; + +public class LzwVolume : Volume +{ + public LzwVolume(Stream stream, ReaderOptions options, int index) + : base(stream, options, index) { } + + public LzwVolume(FileInfo fileInfo, ReaderOptions options) + : base(fileInfo.OpenRead(), options.WithLeaveStreamOpen(false)) { } + + public override bool IsFirstVolume => true; + + public override bool IsMultiVolume => false; +} diff --git a/src/SharpCompress/Common/MemberNotNullAttribute.cs b/src/SharpCompress/Common/MemberNotNullAttribute.cs new file mode 100644 index 00000000..0fab0157 --- /dev/null +++ b/src/SharpCompress/Common/MemberNotNullAttribute.cs @@ -0,0 +1,18 @@ +// Compatibility shim for frameworks that don't expose MemberNotNullAttribute +#if NETSTANDARD2_1 || LEGACY_DOTNET +using System; + +namespace System.Diagnostics.CodeAnalysis +{ + [AttributeUsage( + AttributeTargets.Method | AttributeTargets.Property | AttributeTargets.Constructor, + AllowMultiple = true + )] + internal sealed class MemberNotNullAttribute : Attribute + { + public MemberNotNullAttribute(string member) { } + + public MemberNotNullAttribute(params string[] members) { } + } +} +#endif diff --git a/src/SharpCompress/Common/MultiVolumeExtractionException.cs b/src/SharpCompress/Common/MultiVolumeExtractionException.cs deleted file mode 100644 index 764ac808..00000000 --- a/src/SharpCompress/Common/MultiVolumeExtractionException.cs +++ /dev/null @@ -1,12 +0,0 @@ -using System; - -namespace SharpCompress.Common; - -public class MultiVolumeExtractionException : ExtractionException -{ - public MultiVolumeExtractionException(string message) - : base(message) { } - - public MultiVolumeExtractionException(string message, Exception inner) - : base(message, inner) { } -} diff --git a/src/SharpCompress/Common/MultipartStreamRequiredException.cs b/src/SharpCompress/Common/MultipartStreamRequiredException.cs deleted file mode 100644 index 33a842d9..00000000 --- a/src/SharpCompress/Common/MultipartStreamRequiredException.cs +++ /dev/null @@ -1,7 +0,0 @@ -namespace SharpCompress.Common; - -public class MultipartStreamRequiredException : ExtractionException -{ - public MultipartStreamRequiredException(string message) - : base(message) { } -} diff --git a/src/SharpCompress/Common/Options/IEncodingOptions.cs b/src/SharpCompress/Common/Options/IEncodingOptions.cs new file mode 100644 index 00000000..47fb56f8 --- /dev/null +++ b/src/SharpCompress/Common/Options/IEncodingOptions.cs @@ -0,0 +1,6 @@ +namespace SharpCompress.Common.Options; + +public interface IEncodingOptions +{ + IArchiveEncoding ArchiveEncoding { get; set; } +} diff --git a/src/SharpCompress/Common/Options/IExtractionOptions.cs b/src/SharpCompress/Common/Options/IExtractionOptions.cs new file mode 100644 index 00000000..767bfb93 --- /dev/null +++ b/src/SharpCompress/Common/Options/IExtractionOptions.cs @@ -0,0 +1,44 @@ +using System; + +namespace SharpCompress.Common.Options; + +/// +/// Options for configuring extraction behavior when extracting archive entries to the filesystem. +/// +public interface IExtractionOptions +{ + /// + /// Overwrite target if it exists. + /// Breaking change: Default changed from false to true in version 0.40.0. + /// + bool Overwrite { get; set; } + + /// + /// Extract with internal directory structure. + /// Breaking change: Default changed from false to true in version 0.40.0. + /// + bool ExtractFullPath { get; set; } + + /// + /// Preserve file time. + /// Breaking change: Default changed from false to true in version 0.40.0. + /// + bool PreserveFileTime { get; set; } + + /// + /// Preserve windows file attributes. + /// + bool PreserveAttributes { get; set; } + + /// + /// Buffer size for extraction stream copy operations. + /// + int BufferSize { get; set; } + + /// + /// Delegate for writing symbolic links to disk. + /// The first parameter is the source path (where the symlink is created). + /// The second parameter is the target path (what the symlink refers to). + /// + Action? SymbolicLinkHandler { get; set; } +} diff --git a/src/SharpCompress/Common/Options/IProgressOptions.cs b/src/SharpCompress/Common/Options/IProgressOptions.cs new file mode 100644 index 00000000..80e9dd16 --- /dev/null +++ b/src/SharpCompress/Common/Options/IProgressOptions.cs @@ -0,0 +1,8 @@ +using System; + +namespace SharpCompress.Common.Options; + +public interface IProgressOptions +{ + IProgress? Progress { get; set; } +} diff --git a/src/SharpCompress/Common/Options/IReaderOptions.cs b/src/SharpCompress/Common/Options/IReaderOptions.cs new file mode 100644 index 00000000..c25f4825 --- /dev/null +++ b/src/SharpCompress/Common/Options/IReaderOptions.cs @@ -0,0 +1,44 @@ +using SharpCompress.Compressors; +using SharpCompress.Providers; + +namespace SharpCompress.Common.Options; + +public interface IReaderOptions : IStreamOptions, IEncodingOptions, IProgressOptions +{ + /// + /// Look for RarArchive (Check for self-extracting archives or cases where RarArchive isn't at the start of the file) + /// + bool LookForHeader { get; set; } + + /// + /// Password for encrypted archives. + /// + string? Password { get; set; } + + /// + /// Disable checking for incomplete archives. + /// + bool DisableCheckIncomplete { get; set; } + + /// + /// Buffer size for stream operations. + /// + int BufferSize { get; set; } + + /// + /// Provide a hint for the extension of the archive being read, can speed up finding the correct decoder. + /// + string? ExtensionHint { get; set; } + + /// + /// Size of the rewindable buffer for non-seekable streams. + /// + int? RewindableBufferSize { get; set; } + + /// + /// Registry of compression providers. + /// Defaults to but can be replaced with custom providers. + /// Use this to provide alternative decompression implementations. + /// + CompressionProviderRegistry Providers { get; set; } +} diff --git a/src/SharpCompress/Common/Options/IStreamOptions.cs b/src/SharpCompress/Common/Options/IStreamOptions.cs new file mode 100644 index 00000000..94a60790 --- /dev/null +++ b/src/SharpCompress/Common/Options/IStreamOptions.cs @@ -0,0 +1,6 @@ +namespace SharpCompress.Common.Options; + +public interface IStreamOptions +{ + bool LeaveStreamOpen { get; set; } +} diff --git a/src/SharpCompress/Common/Options/IWriterOptions.cs b/src/SharpCompress/Common/Options/IWriterOptions.cs new file mode 100644 index 00000000..37fbb21e --- /dev/null +++ b/src/SharpCompress/Common/Options/IWriterOptions.cs @@ -0,0 +1,33 @@ +using SharpCompress.Common; +using SharpCompress.Compressors; +using SharpCompress.Providers; + +namespace SharpCompress.Common.Options; + +/// +/// Options for configuring writer behavior when creating archives. +/// +public interface IWriterOptions : IStreamOptions, IEncodingOptions, IProgressOptions +{ + /// + /// The compression type to use for the archive. + /// + CompressionType CompressionType { get; set; } + + /// + /// The compression level to be used when the compression type supports variable levels. + /// + int CompressionLevel { get; set; } + + /// + /// Buffer size for writer stream copy operations. + /// + int BufferSize { get; set; } + + /// + /// Registry of compression providers. + /// Defaults to but can be replaced with custom providers, such as + /// System.IO.Compression for Deflate/GZip on modern .NET. + /// + CompressionProviderRegistry Providers { get; set; } +} diff --git a/src/SharpCompress/Common/OptionsBase.cs b/src/SharpCompress/Common/OptionsBase.cs deleted file mode 100644 index 61a6cb55..00000000 --- a/src/SharpCompress/Common/OptionsBase.cs +++ /dev/null @@ -1,11 +0,0 @@ -namespace SharpCompress.Common; - -public class OptionsBase -{ - /// - /// SharpCompress will keep the supplied streams open. Default is true. - /// - public bool LeaveStreamOpen { get; set; } = true; - - public ArchiveEncoding ArchiveEncoding { get; set; } = new ArchiveEncoding(); -} diff --git a/src/SharpCompress/Common/PasswordProtectedException.cs b/src/SharpCompress/Common/PasswordProtectedException.cs deleted file mode 100644 index 9ebe3d65..00000000 --- a/src/SharpCompress/Common/PasswordProtectedException.cs +++ /dev/null @@ -1,12 +0,0 @@ -using System; - -namespace SharpCompress.Common; - -public class PasswordProtectedException : ExtractionException -{ - public PasswordProtectedException(string message) - : base(message) { } - - public PasswordProtectedException(string message, Exception inner) - : base(message, inner) { } -} diff --git a/src/SharpCompress/Common/ProgressReport.cs b/src/SharpCompress/Common/ProgressReport.cs new file mode 100644 index 00000000..2d836867 --- /dev/null +++ b/src/SharpCompress/Common/ProgressReport.cs @@ -0,0 +1,43 @@ +namespace SharpCompress.Common; + +/// +/// Represents progress information for compression or extraction operations. +/// +public sealed class ProgressReport +{ + /// + /// Initializes a new instance of the class. + /// + /// The path of the entry being processed. + /// Number of bytes transferred so far. + /// Total bytes to be transferred, or null if unknown. + public ProgressReport(string entryPath, long bytesTransferred, long? totalBytes) + { + EntryPath = entryPath; + BytesTransferred = bytesTransferred; + TotalBytes = totalBytes; + } + + /// + /// Gets the path of the entry being processed. + /// + public string EntryPath { get; } + + /// + /// Gets the number of bytes transferred so far. + /// + public long BytesTransferred { get; } + + /// + /// Gets the total number of bytes to be transferred, or null if unknown. + /// + public long? TotalBytes { get; } + + /// + /// Gets the progress percentage (0-100), or null if total bytes is unknown. + /// + public double? PercentComplete => + TotalBytes.HasValue && TotalBytes.Value > 0 + ? (double)BytesTransferred / TotalBytes.Value * 100 + : null; +} diff --git a/src/SharpCompress/Common/Rar/AsyncMarkingBinaryReader.cs b/src/SharpCompress/Common/Rar/AsyncMarkingBinaryReader.cs new file mode 100644 index 00000000..57aa20a5 --- /dev/null +++ b/src/SharpCompress/Common/Rar/AsyncMarkingBinaryReader.cs @@ -0,0 +1,213 @@ +using System; +using System.Buffers.Binary; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.IO; + +namespace SharpCompress.Common.Rar; + +internal class AsyncMarkingBinaryReader : IDisposable +#if NET8_0_OR_GREATER + , IAsyncDisposable +#endif +{ + private readonly AsyncBinaryReader _reader; + + public AsyncMarkingBinaryReader(Stream stream) + { + _reader = new AsyncBinaryReader(stream, leaveOpen: true); + } + + public Stream BaseStream => _reader.BaseStream; + + public virtual long CurrentReadByteCount { get; protected set; } + + public virtual void Mark() => CurrentReadByteCount = 0; + + public virtual async ValueTask ReadBooleanAsync( + CancellationToken cancellationToken = default + ) => await ReadByteAsync(cancellationToken).ConfigureAwait(false) != 0; + + public virtual async ValueTask ReadByteAsync( + CancellationToken cancellationToken = default + ) + { + CurrentReadByteCount++; + return await _reader.ReadByteAsync(cancellationToken).ConfigureAwait(false); + } + + public virtual async ValueTask ReadBytesAsync( + int count, + CancellationToken cancellationToken = default + ) + { + CurrentReadByteCount += count; + var bytes = new byte[count]; + try + { + await _reader.ReadBytesAsync(bytes, 0, count, cancellationToken).ConfigureAwait(false); + } + catch (IncompleteArchiveException ex) + { + throw new InvalidFormatException( + string.Format( + Constants.DefaultCultureInfo, + "Could not read the requested amount of bytes. End of stream reached. Requested: {0}", + count + ), + ex + ); + } + return bytes; + } + + public async ValueTask ReadUInt16Async(CancellationToken cancellationToken = default) + { + var bytes = await ReadBytesAsync(2, cancellationToken).ConfigureAwait(false); + return BinaryPrimitives.ReadUInt16LittleEndian(bytes); + } + + public async ValueTask ReadUInt32Async(CancellationToken cancellationToken = default) + { + var bytes = await ReadBytesAsync(4, cancellationToken).ConfigureAwait(false); + return BinaryPrimitives.ReadUInt32LittleEndian(bytes); + } + + public virtual async ValueTask ReadUInt64Async( + CancellationToken cancellationToken = default + ) + { + var bytes = await ReadBytesAsync(8, cancellationToken).ConfigureAwait(false); + return BinaryPrimitives.ReadUInt64LittleEndian(bytes); + } + + public virtual async ValueTask ReadInt16Async( + CancellationToken cancellationToken = default + ) + { + var bytes = await ReadBytesAsync(2, cancellationToken).ConfigureAwait(false); + return BinaryPrimitives.ReadInt16LittleEndian(bytes); + } + + public virtual async ValueTask ReadInt32Async( + CancellationToken cancellationToken = default + ) + { + var bytes = await ReadBytesAsync(4, cancellationToken).ConfigureAwait(false); + return BinaryPrimitives.ReadInt32LittleEndian(bytes); + } + + public virtual async ValueTask ReadInt64Async( + CancellationToken cancellationToken = default + ) + { + var bytes = await ReadBytesAsync(8, cancellationToken).ConfigureAwait(false); + return BinaryPrimitives.ReadInt64LittleEndian(bytes); + } + + public async ValueTask ReadRarVIntAsync( + int maxBytes = 10, + CancellationToken cancellationToken = default + ) => await DoReadRarVIntAsync((maxBytes - 1) * 7, cancellationToken).ConfigureAwait(false); + + private async ValueTask DoReadRarVIntAsync( + int maxShift, + CancellationToken cancellationToken + ) + { + var shift = 0; + ulong result = 0; + do + { + var b0 = await ReadByteAsync(cancellationToken).ConfigureAwait(false); + var b1 = ((uint)b0) & 0x7f; + ulong n = b1; + var shifted = n << shift; + if (n != shifted >> shift) + { + // overflow + break; + } + result |= shifted; + if (b0 == b1) + { + return result; + } + shift += 7; + } while (shift <= maxShift); + + throw new InvalidFormatException("malformed vint"); + } + + public async ValueTask ReadRarVIntUInt32Async( + int maxBytes = 5, + CancellationToken cancellationToken = default + ) => + // hopefully this gets inlined + await DoReadRarVIntUInt32Async((maxBytes - 1) * 7, cancellationToken).ConfigureAwait(false); + + public async ValueTask ReadRarVIntUInt16Async( + int maxBytes = 3, + CancellationToken cancellationToken = default + ) => + // hopefully this gets inlined + checked( + (ushort) + await DoReadRarVIntUInt32Async((maxBytes - 1) * 7, cancellationToken) + .ConfigureAwait(false) + ); + + public async ValueTask ReadRarVIntByteAsync( + int maxBytes = 2, + CancellationToken cancellationToken = default + ) => + // hopefully this gets inlined + checked( + (byte) + await DoReadRarVIntUInt32Async((maxBytes - 1) * 7, cancellationToken) + .ConfigureAwait(false) + ); + + public async ValueTask SkipAsync(int count, CancellationToken cancellationToken = default) + { + CurrentReadByteCount += count; + await _reader.SkipAsync(count, cancellationToken).ConfigureAwait(false); + } + + private async ValueTask DoReadRarVIntUInt32Async( + int maxShift, + CancellationToken cancellationToken = default + ) + { + var shift = 0; + uint result = 0; + do + { + var b0 = await ReadByteAsync(cancellationToken).ConfigureAwait(false); + var b1 = ((uint)b0) & 0x7f; + var n = b1; + var shifted = n << shift; + if (n != shifted >> shift) + { + // overflow + break; + } + result |= shifted; + if (b0 == b1) + { + return result; + } + shift += 7; + } while (shift <= maxShift); + + throw new InvalidFormatException("malformed vint"); + } + + public virtual void Dispose() => _reader.Dispose(); + +#if NET8_0_OR_GREATER + public virtual ValueTask DisposeAsync() => _reader.DisposeAsync(); +#endif +} diff --git a/src/SharpCompress/Common/Rar/AsyncRarCrcBinaryReader.cs b/src/SharpCompress/Common/Rar/AsyncRarCrcBinaryReader.cs new file mode 100644 index 00000000..c4c440a1 --- /dev/null +++ b/src/SharpCompress/Common/Rar/AsyncRarCrcBinaryReader.cs @@ -0,0 +1,45 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Compressors.Rar; + +namespace SharpCompress.Common.Rar; + +internal class AsyncRarCrcBinaryReader(Stream stream) : AsyncMarkingBinaryReader(stream) +{ + private uint _currentCrc; + + public uint GetCrc32() => ~_currentCrc; + + public void ResetCrc() => _currentCrc = 0xffffffff; + + protected void UpdateCrc(byte b) => _currentCrc = RarCRC.CheckCrc(_currentCrc, b); + + protected async ValueTask ReadBytesNoCrcAsync( + int count, + CancellationToken cancellationToken = default + ) + { + return await base.ReadBytesAsync(count, cancellationToken).ConfigureAwait(false); + } + + public override async ValueTask ReadByteAsync( + CancellationToken cancellationToken = default + ) + { + var b = await base.ReadByteAsync(cancellationToken).ConfigureAwait(false); + _currentCrc = RarCRC.CheckCrc(_currentCrc, b); + return b; + } + + public override async ValueTask ReadBytesAsync( + int count, + CancellationToken cancellationToken = default + ) + { + var result = await base.ReadBytesAsync(count, cancellationToken).ConfigureAwait(false); + _currentCrc = RarCRC.CheckCrc(_currentCrc, result, 0, result.Length); + return result; + } +} diff --git a/src/SharpCompress/Common/Rar/AsyncRarCryptoBinaryReader.cs b/src/SharpCompress/Common/Rar/AsyncRarCryptoBinaryReader.cs new file mode 100644 index 00000000..c3f5d005 --- /dev/null +++ b/src/SharpCompress/Common/Rar/AsyncRarCryptoBinaryReader.cs @@ -0,0 +1,111 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Rar.Headers; +using SharpCompress.Crypto; + +namespace SharpCompress.Common.Rar; + +internal sealed class AsyncRarCryptoBinaryReader : AsyncRarCrcBinaryReader +{ + private BlockTransformer _rijndael = default!; + private readonly Queue _data = new(); + private long _readCount; + + private AsyncRarCryptoBinaryReader(Stream stream) + : base(stream) { } + + public static async ValueTask Create( + Stream stream, + ICryptKey cryptKey, + byte[]? salt = null + ) + { + var binary = new AsyncRarCryptoBinaryReader(stream); + if (salt == null) + { + salt = await binary + .ReadBytesAsyncBase(EncryptionConstV5.SIZE_SALT30) + .ConfigureAwait(false); + binary._readCount += EncryptionConstV5.SIZE_SALT30; + } + binary._rijndael = new BlockTransformer(cryptKey.Transformer(salt)); + return binary; + } + + public override long CurrentReadByteCount + { + get => _readCount; + protected set + { + // ignore + } + } + + public override void Mark() => _readCount = 0; + + public override async ValueTask ReadByteAsync( + CancellationToken cancellationToken = default + ) + { + var bytes = await ReadAndDecryptBytesAsync(1, cancellationToken).ConfigureAwait(false); + return bytes[0]; + } + + private ValueTask ReadBytesAsyncBase(int count) => base.ReadBytesAsync(count); + + public override async ValueTask ReadBytesAsync( + int count, + CancellationToken cancellationToken = default + ) + { + return await ReadAndDecryptBytesAsync(count, cancellationToken).ConfigureAwait(false); + } + + private async ValueTask ReadAndDecryptBytesAsync( + int count, + CancellationToken cancellationToken + ) + { + var queueSize = _data.Count; + var sizeToRead = count - queueSize; + + if (sizeToRead > 0) + { + var alignedSize = sizeToRead + ((~sizeToRead + 1) & 0xf); + for (var i = 0; i < alignedSize / 16; i++) + { + var cipherText = await ReadBytesNoCrcAsync(16, cancellationToken) + .ConfigureAwait(false); + var readBytes = _rijndael.ProcessBlock(cipherText); + foreach (var readByte in readBytes) + { + _data.Enqueue(readByte); + } + } + } + + var decryptedBytes = new byte[count]; + + for (var i = 0; i < count; i++) + { + var b = _data.Dequeue(); + decryptedBytes[i] = b; + UpdateCrc(b); + } + + _readCount += count; + return decryptedBytes; + } + + public void ClearQueue() => _data.Clear(); + + public void SkipQueue() + { + var position = BaseStream.Position; + BaseStream.Position = position + _data.Count; + ClearQueue(); + } +} diff --git a/src/SharpCompress/Common/Rar/CryptKey3.cs b/src/SharpCompress/Common/Rar/CryptKey3.cs new file mode 100644 index 00000000..950f3ad6 --- /dev/null +++ b/src/SharpCompress/Common/Rar/CryptKey3.cs @@ -0,0 +1,99 @@ +using System; +using System.Diagnostics.CodeAnalysis; +using System.Security.Cryptography; +using System.Text; +using SharpCompress.Common.Rar.Headers; + +namespace SharpCompress.Common.Rar; + +[SuppressMessage( + "Security", + "CA5350:Do Not Use Weak Cryptographic Algorithms", + Justification = "RAR3 key derivation is SHA-1 based by format definition." +)] +internal class CryptKey3 : ICryptKey +{ + const int AES_128 = 128; + + private readonly string _password; + + public CryptKey3(string? password) => _password = password ?? string.Empty; + + public ICryptoTransform Transformer(byte[] salt) + { + var aesIV = new byte[EncryptionConstV5.SIZE_INITV]; + + var rawLength = 2 * _password.Length; + var rawPassword = new byte[rawLength + EncryptionConstV5.SIZE_SALT30]; + var passwordBytes = Encoding.UTF8.GetBytes(_password); + for (var i = 0; i < _password.Length; i++) + { + rawPassword[i * 2] = passwordBytes[i]; + rawPassword[(i * 2) + 1] = 0; + } + + for (var i = 0; i < salt.Length; i++) + { + rawPassword[i + rawLength] = salt[i]; + } + +#if LEGACY_DOTNET + var msgDigest = SHA1.Create(); +#endif + const int noOfRounds = (1 << 18); + const int iblock = 3; + + byte[] digest; + var data = new byte[(rawPassword.Length + iblock) * noOfRounds]; + + //TODO slow code below, find ways to optimize + for (var i = 0; i < noOfRounds; i++) + { + rawPassword.CopyTo(data, i * (rawPassword.Length + iblock)); + + data[(i * (rawPassword.Length + iblock)) + rawPassword.Length + 0] = (byte)i; + data[(i * (rawPassword.Length + iblock)) + rawPassword.Length + 1] = (byte)(i >> 8); + data[(i * (rawPassword.Length + iblock)) + rawPassword.Length + 2] = (byte)(i >> 16); + + if (i % (noOfRounds / EncryptionConstV5.SIZE_INITV) == 0) + { +#if LEGACY_DOTNET + digest = msgDigest.ComputeHash(data, 0, (i + 1) * (rawPassword.Length + iblock)); +#else + digest = SHA1.HashData(data.AsSpan(0, (i + 1) * (rawPassword.Length + iblock))); +#endif + aesIV[i / (noOfRounds / EncryptionConstV5.SIZE_INITV)] = digest[19]; + } + } +#if LEGACY_DOTNET + digest = msgDigest.ComputeHash(data); +#else + digest = SHA1.HashData(data); +#endif + //slow code ends + + var aesKey = new byte[EncryptionConstV5.SIZE_INITV]; + for (var i = 0; i < 4; i++) + { + for (var j = 0; j < 4; j++) + { + aesKey[(i * 4) + j] = (byte)( + ( + ((digest[i * 4] * 0x1000000) & 0xff000000) + | (uint)((digest[(i * 4) + 1] * 0x10000) & 0xff0000) + | (uint)((digest[(i * 4) + 2] * 0x100) & 0xff00) + | (uint)(digest[(i * 4) + 3] & 0xff) + ) >> (j * 8) + ); + } + } + + var aes = Aes.Create(); + aes.KeySize = AES_128; + aes.Mode = CipherMode.CBC; + aes.Padding = PaddingMode.None; + aes.Key = aesKey; + aes.IV = aesIV; + return aes.CreateDecryptor(); + } +} diff --git a/src/SharpCompress/Common/Rar/CryptKey5.cs b/src/SharpCompress/Common/Rar/CryptKey5.cs new file mode 100644 index 00000000..490a3c54 --- /dev/null +++ b/src/SharpCompress/Common/Rar/CryptKey5.cs @@ -0,0 +1,104 @@ +using System.Collections.Generic; +using System.Linq; +using System.Security.Cryptography; +using System.Text; +using SharpCompress.Common.Rar.Headers; + +namespace SharpCompress.Common.Rar; + +internal class CryptKey5 : ICryptKey +{ + const int AES_256 = 256; + const int DERIVED_KEY_LENGTH = 0x10; + const int SHA256_DIGEST_SIZE = 32; + + private string _password; + private Rar5CryptoInfo _cryptoInfo; + private byte[] _pswCheck = []; + private byte[] _hashKey = []; + + public CryptKey5(string? password, Rar5CryptoInfo rar5CryptoInfo) + { + _password = password ?? ""; + _cryptoInfo = rar5CryptoInfo; + } + + public byte[] PswCheck => _pswCheck; + + public byte[] HashKey => _hashKey; + + private static List GenerateRarPBKDF2Key( + string password, + byte[] salt, + int iterations, + int keyLength + ) + { + var passwordBytes = Encoding.UTF8.GetBytes(password); +#if LEGACY_DOTNET + using var hmac = new HMACSHA256(passwordBytes); + var block = hmac.ComputeHash(salt); +#else + var block = HMACSHA256.HashData(passwordBytes, salt); +#endif + var finalHash = (byte[])block.Clone(); + + var loop = new int[] { iterations, 17, 17 }; + var res = new List { }; + + for (var x = 0; x < 3; x++) + { + for (var i = 1; i < loop[x]; i++) + { +#if LEGACY_DOTNET + block = hmac.ComputeHash(block); +#else + block = HMACSHA256.HashData(passwordBytes, block); +#endif + for (var j = 0; j < finalHash.Length; j++) + { + finalHash[j] ^= block[j]; + } + } + + res.Add((byte[])finalHash.Clone()); + } + + return res; + } + + public ICryptoTransform Transformer(byte[] salt) + { + var iterations = (1 << _cryptoInfo.LG2Count); // Adjust the number of iterations as needed + + var salt_rar5 = salt.Concat(new byte[] { 0, 0, 0, 1 }); + var derivedKey = GenerateRarPBKDF2Key( + _password, + salt_rar5.ToArray(), + iterations, + DERIVED_KEY_LENGTH + ); + + _hashKey = derivedKey[1]; + + _pswCheck = new byte[EncryptionConstV5.SIZE_PSWCHECK]; + + for (var i = 0; i < SHA256_DIGEST_SIZE; i++) + { + _pswCheck[i % EncryptionConstV5.SIZE_PSWCHECK] ^= derivedKey[2][i]; + } + + if (_cryptoInfo.UsePswCheck && !_cryptoInfo.PswCheck.SequenceEqual(_pswCheck)) + { + throw new CryptographicException("The password did not match."); + } + + var aes = Aes.Create(); + aes.KeySize = AES_256; + aes.Mode = CipherMode.CBC; + aes.Padding = PaddingMode.None; + aes.Key = derivedKey[0]; + aes.IV = _cryptoInfo.InitV; + return aes.CreateDecryptor(); + } +} diff --git a/src/SharpCompress/Common/Rar/Headers/AVHeader.cs b/src/SharpCompress/Common/Rar/Headers/AVHeader.cs index cd51d80b..fafab9df 100644 --- a/src/SharpCompress/Common/Rar/Headers/AVHeader.cs +++ b/src/SharpCompress/Common/Rar/Headers/AVHeader.cs @@ -4,13 +4,14 @@ namespace SharpCompress.Common.Rar.Headers; internal class AvHeader : RarHeader { - public AvHeader(RarHeader header, RarCrcBinaryReader reader) - : base(header, reader, HeaderType.Av) + public static AvHeader Create(RarHeader header, RarCrcBinaryReader reader) { - if (IsRar5) + var c = CreateChild(header, reader, HeaderType.Av); + if (c.IsRar5) { throw new InvalidFormatException("unexpected rar5 record"); } + return c; } protected override void ReadFinish(MarkingBinaryReader reader) diff --git a/src/SharpCompress/Common/Rar/Headers/ArchiveCryptHeader.Async.cs b/src/SharpCompress/Common/Rar/Headers/ArchiveCryptHeader.Async.cs new file mode 100644 index 00000000..eb34ec71 --- /dev/null +++ b/src/SharpCompress/Common/Rar/Headers/ArchiveCryptHeader.Async.cs @@ -0,0 +1,30 @@ +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Rar; +using SharpCompress.IO; + +namespace SharpCompress.Common.Rar.Headers; + +internal sealed partial class ArchiveCryptHeader +{ + public static async ValueTask CreateAsync( + RarHeader header, + AsyncRarCrcBinaryReader reader, + CancellationToken cancellationToken = default + ) => + await CreateChildAsync( + header, + reader, + HeaderType.Crypt, + cancellationToken + ) + .ConfigureAwait(false); + + protected sealed override async ValueTask ReadFinishAsync( + AsyncMarkingBinaryReader reader, + CancellationToken cancellationToken = default + ) + { + CryptInfo = await Rar5CryptoInfo.CreateAsync(reader, false).ConfigureAwait(false); + } +} diff --git a/src/SharpCompress/Common/Rar/Headers/ArchiveCryptHeader.cs b/src/SharpCompress/Common/Rar/Headers/ArchiveCryptHeader.cs index 619e8fc9..bdb107fc 100644 --- a/src/SharpCompress/Common/Rar/Headers/ArchiveCryptHeader.cs +++ b/src/SharpCompress/Common/Rar/Headers/ArchiveCryptHeader.cs @@ -1,50 +1,15 @@ -#nullable disable - +using SharpCompress.Common.Rar; using SharpCompress.IO; namespace SharpCompress.Common.Rar.Headers; -internal class ArchiveCryptHeader : RarHeader +internal sealed partial class ArchiveCryptHeader : RarHeader { - private const int CRYPT_VERSION = 0; // Supported encryption version. - private const int SIZE_SALT50 = 16; - private const int SIZE_PSWCHECK = 8; - private const int SIZE_PSWCHECK_CSUM = 4; - private const int CRYPT5_KDF_LG2_COUNT_MAX = 24; // LOG2 of maximum accepted iteration count. + public static ArchiveCryptHeader Create(RarHeader header, RarCrcBinaryReader reader) => + CreateChild(header, reader, HeaderType.Crypt); - private bool _usePswCheck; - private uint _lg2Count; // Log2 of PBKDF2 repetition count. - private byte[] _salt; - private byte[] _pswCheck; - private byte[] _pswCheckCsm; + public Rar5CryptoInfo CryptInfo = default!; - public ArchiveCryptHeader(RarHeader header, RarCrcBinaryReader reader) - : base(header, reader, HeaderType.Crypt) { } - - protected override void ReadFinish(MarkingBinaryReader reader) - { - var cryptVersion = reader.ReadRarVIntUInt32(); - if (cryptVersion > CRYPT_VERSION) - { - //error? - return; - } - var encryptionFlags = reader.ReadRarVIntUInt32(); - _usePswCheck = FlagUtility.HasFlag(encryptionFlags, EncryptionFlagsV5.CHFL_CRYPT_PSWCHECK); - _lg2Count = reader.ReadRarVIntByte(1); - - //UsePswCheck = HasHeaderFlag(EncryptionFlagsV5.CHFL_CRYPT_PSWCHECK); - if (_lg2Count > CRYPT5_KDF_LG2_COUNT_MAX) - { - //error? - return; - } - - _salt = reader.ReadBytes(SIZE_SALT50); - if (_usePswCheck) - { - _pswCheck = reader.ReadBytes(SIZE_PSWCHECK); - _pswCheckCsm = reader.ReadBytes(SIZE_PSWCHECK_CSUM); - } - } + protected sealed override void ReadFinish(MarkingBinaryReader reader) => + CryptInfo = Rar5CryptoInfo.Create(reader, false); } diff --git a/src/SharpCompress/Common/Rar/Headers/ArchiveHeader.Async.cs b/src/SharpCompress/Common/Rar/Headers/ArchiveHeader.Async.cs new file mode 100644 index 00000000..9a9b3065 --- /dev/null +++ b/src/SharpCompress/Common/Rar/Headers/ArchiveHeader.Async.cs @@ -0,0 +1,51 @@ +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Rar; +using SharpCompress.IO; + +namespace SharpCompress.Common.Rar.Headers; + +internal sealed partial class ArchiveHeader +{ + public static async ValueTask CreateAsync( + RarHeader header, + AsyncRarCrcBinaryReader reader, + CancellationToken cancellationToken = default + ) => + await CreateChildAsync(header, reader, HeaderType.Archive, cancellationToken) + .ConfigureAwait(false); + + protected sealed override async ValueTask ReadFinishAsync( + AsyncMarkingBinaryReader reader, + CancellationToken cancellationToken = default + ) + { + if (IsRar5) + { + Flags = await reader + .ReadRarVIntUInt16Async(cancellationToken: cancellationToken) + .ConfigureAwait(false); + if (HasFlag(ArchiveFlagsV5.HAS_VOLUME_NUMBER)) + { + VolumeNumber = (int) + await reader + .ReadRarVIntUInt32Async(cancellationToken: cancellationToken) + .ConfigureAwait(false); + } + // later: we may have a locator record if we need it + //if (ExtraSize != 0) { + // ReadLocator(reader); + //} + } + else + { + Flags = HeaderFlags; + HighPosAv = await reader.ReadInt16Async(cancellationToken).ConfigureAwait(false); + PosAv = await reader.ReadInt32Async(cancellationToken).ConfigureAwait(false); + if (HasFlag(ArchiveFlagsV4.ENCRYPT_VER)) + { + _ = await reader.ReadByteAsync(cancellationToken).ConfigureAwait(false); + } + } + } +} diff --git a/src/SharpCompress/Common/Rar/Headers/ArchiveHeader.cs b/src/SharpCompress/Common/Rar/Headers/ArchiveHeader.cs index e8f344ad..1f6f147d 100644 --- a/src/SharpCompress/Common/Rar/Headers/ArchiveHeader.cs +++ b/src/SharpCompress/Common/Rar/Headers/ArchiveHeader.cs @@ -1,13 +1,14 @@ +using SharpCompress.Common.Rar; using SharpCompress.IO; namespace SharpCompress.Common.Rar.Headers; -internal sealed class ArchiveHeader : RarHeader +internal sealed partial class ArchiveHeader : RarHeader { - public ArchiveHeader(RarHeader header, RarCrcBinaryReader reader) - : base(header, reader, HeaderType.Archive) { } + public static ArchiveHeader Create(RarHeader header, RarCrcBinaryReader reader) => + CreateChild(header, reader, HeaderType.Archive); - protected override void ReadFinish(MarkingBinaryReader reader) + protected sealed override void ReadFinish(MarkingBinaryReader reader) { if (IsRar5) { @@ -28,7 +29,7 @@ internal sealed class ArchiveHeader : RarHeader PosAv = reader.ReadInt32(); if (HasFlag(ArchiveFlagsV4.ENCRYPT_VER)) { - EncryptionVersion = reader.ReadByte(); + _ = reader.ReadByte(); } } } @@ -43,8 +44,6 @@ internal sealed class ArchiveHeader : RarHeader internal int? PosAv { get; private set; } - private byte? EncryptionVersion { get; set; } - public bool? IsEncrypted => IsRar5 ? null : HasFlag(ArchiveFlagsV4.PASSWORD); public bool OldNumberingFormat => !IsRar5 && !HasFlag(ArchiveFlagsV4.NEW_NUMBERING); diff --git a/src/SharpCompress/Common/Rar/Headers/CommentHeader.cs b/src/SharpCompress/Common/Rar/Headers/CommentHeader.cs index 54abf9cc..14f0a799 100644 --- a/src/SharpCompress/Common/Rar/Headers/CommentHeader.cs +++ b/src/SharpCompress/Common/Rar/Headers/CommentHeader.cs @@ -4,13 +4,14 @@ namespace SharpCompress.Common.Rar.Headers; internal class CommentHeader : RarHeader { - protected CommentHeader(RarHeader header, RarCrcBinaryReader reader) - : base(header, reader, HeaderType.Comment) + public static CommentHeader Create(RarHeader header, RarCrcBinaryReader reader) { - if (IsRar5) + var c = CreateChild(header, reader, HeaderType.Comment); + if (c.IsRar5) { throw new InvalidFormatException("unexpected rar5 record"); } + return c; } protected override void ReadFinish(MarkingBinaryReader reader) diff --git a/src/SharpCompress/Common/Rar/Headers/EndArchiveHeader.Async.cs b/src/SharpCompress/Common/Rar/Headers/EndArchiveHeader.Async.cs new file mode 100644 index 00000000..8311c796 --- /dev/null +++ b/src/SharpCompress/Common/Rar/Headers/EndArchiveHeader.Async.cs @@ -0,0 +1,47 @@ +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Rar; +using SharpCompress.IO; + +namespace SharpCompress.Common.Rar.Headers; + +internal sealed partial class EndArchiveHeader +{ + public static async ValueTask CreateAsync( + RarHeader header, + AsyncRarCrcBinaryReader reader, + CancellationToken cancellationToken = default + ) => + await CreateChildAsync( + header, + reader, + HeaderType.EndArchive, + cancellationToken + ) + .ConfigureAwait(false); + + protected sealed override async ValueTask ReadFinishAsync( + AsyncMarkingBinaryReader reader, + CancellationToken cancellationToken = default + ) + { + if (IsRar5) + { + Flags = await reader + .ReadRarVIntUInt16Async(cancellationToken: cancellationToken) + .ConfigureAwait(false); + } + else + { + Flags = HeaderFlags; + if (HasFlag(EndArchiveFlagsV4.DATA_CRC)) + { + ArchiveCrc = await reader.ReadInt32Async(cancellationToken).ConfigureAwait(false); + } + if (HasFlag(EndArchiveFlagsV4.VOLUME_NUMBER)) + { + VolumeNumber = await reader.ReadInt16Async(cancellationToken).ConfigureAwait(false); + } + } + } +} diff --git a/src/SharpCompress/Common/Rar/Headers/EndArchiveHeader.cs b/src/SharpCompress/Common/Rar/Headers/EndArchiveHeader.cs index f5bc4523..def87962 100644 --- a/src/SharpCompress/Common/Rar/Headers/EndArchiveHeader.cs +++ b/src/SharpCompress/Common/Rar/Headers/EndArchiveHeader.cs @@ -1,13 +1,14 @@ -using SharpCompress.IO; +using SharpCompress.Common.Rar; +using SharpCompress.IO; namespace SharpCompress.Common.Rar.Headers; -internal class EndArchiveHeader : RarHeader +internal sealed partial class EndArchiveHeader : RarHeader { - public EndArchiveHeader(RarHeader header, RarCrcBinaryReader reader) - : base(header, reader, HeaderType.EndArchive) { } + public static EndArchiveHeader Create(RarHeader header, RarCrcBinaryReader reader) => + CreateChild(header, reader, HeaderType.EndArchive); - protected override void ReadFinish(MarkingBinaryReader reader) + protected sealed override void ReadFinish(MarkingBinaryReader reader) { if (IsRar5) { diff --git a/src/SharpCompress/Common/Rar/Headers/FileHeader.Async.cs b/src/SharpCompress/Common/Rar/Headers/FileHeader.Async.cs new file mode 100644 index 00000000..31024882 --- /dev/null +++ b/src/SharpCompress/Common/Rar/Headers/FileHeader.Async.cs @@ -0,0 +1,439 @@ +using System; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Rar; +using SharpCompress.IO; +using size_t = System.UInt32; + +namespace SharpCompress.Common.Rar.Headers; + +internal partial class FileHeader +{ + public static async ValueTask CreateAsync( + RarHeader header, + AsyncRarCrcBinaryReader reader, + HeaderType headerType, + CancellationToken cancellationToken = default + ) => + await CreateChildAsync(header, reader, headerType, cancellationToken) + .ConfigureAwait(false); + + protected override async ValueTask ReadFinishAsync( + AsyncMarkingBinaryReader reader, + CancellationToken cancellationToken + ) + { + if (IsRar5) + { + await ReadFromReaderV5Async(reader, cancellationToken).ConfigureAwait(false); + } + else + { + await ReadFromReaderV4Async(reader, cancellationToken).ConfigureAwait(false); + } + } + + private async ValueTask ReadFromReaderV5Async( + AsyncMarkingBinaryReader reader, + CancellationToken cancellationToken + ) + { + Flags = await reader + .ReadRarVIntUInt16Async(cancellationToken: cancellationToken) + .ConfigureAwait(false); + + var lvalue = checked( + (long) + await reader + .ReadRarVIntAsync(cancellationToken: cancellationToken) + .ConfigureAwait(false) + ); + + UncompressedSize = HasFlag(FileFlagsV5.UNPACKED_SIZE_UNKNOWN) ? long.MaxValue : lvalue; + + FileAttributes = await reader + .ReadRarVIntUInt32Async(cancellationToken: cancellationToken) + .ConfigureAwait(false); + + if (HasFlag(FileFlagsV5.HAS_MOD_TIME)) + { + FileLastModifiedTime = Utility.UnixTimeToDateTime( + await reader.ReadUInt32Async(cancellationToken).ConfigureAwait(false) + ); + } + + if (HasFlag(FileFlagsV5.HAS_CRC32)) + { + FileCrc = await reader.ReadBytesAsync(4, cancellationToken).ConfigureAwait(false); + } + + var compressionInfo = await reader + .ReadRarVIntUInt16Async(cancellationToken: cancellationToken) + .ConfigureAwait(false); + + CompressionAlgorithm = (byte)((compressionInfo & 0x3f) + 50); + IsSolid = (compressionInfo & 0x40) == 0x40; + CompressionMethod = (byte)((compressionInfo >> 7) & 0x7); + WindowSize = IsDirectory ? 0 : ((size_t)0x20000) << ((compressionInfo >> 10) & 0xf); + + _ = await reader + .ReadRarVIntByteAsync(cancellationToken: cancellationToken) + .ConfigureAwait(false); + + var nameSize = await reader + .ReadRarVIntUInt16Async(cancellationToken: cancellationToken) + .ConfigureAwait(false); + + var b = await reader.ReadBytesAsync(nameSize, cancellationToken).ConfigureAwait(false); + FileName = ConvertPathV5(Encoding.UTF8.GetString(b, 0, b.Length)); + + if (ExtraSize != (uint)RemainingHeaderBytesAsync(reader)) + { + throw new InvalidFormatException("rar5 header size / extra size inconsistency"); + } + + const ushort FHEXTRA_CRYPT = 0x01; + const ushort FHEXTRA_HASH = 0x02; + const ushort FHEXTRA_HTIME = 0x03; + const ushort FHEXTRA_REDIR = 0x05; + + while (reader.CurrentReadByteCount < HeaderSize) + { + var size = await reader + .ReadRarVIntUInt16Async(cancellationToken: cancellationToken) + .ConfigureAwait(false); + var n = HeaderSize - reader.CurrentReadByteCount; + var type = await reader + .ReadRarVIntUInt16Async(cancellationToken: cancellationToken) + .ConfigureAwait(false); + switch (type) + { + case FHEXTRA_CRYPT: + { + Rar5CryptoInfo = await Rar5CryptoInfo + .CreateAsync(reader, true) + .ConfigureAwait(false); + if (Rar5CryptoInfo.PswCheck.All(singleByte => singleByte == 0)) + { + Rar5CryptoInfo = null; + } + } + break; + case FHEXTRA_HASH: + { + const uint FHEXTRA_HASH_BLAKE2 = 0x0; + const int BLAKE2_DIGEST_SIZE = 0x20; + if ( + await reader + .ReadRarVIntUInt32Async(cancellationToken: cancellationToken) + .ConfigureAwait(false) == FHEXTRA_HASH_BLAKE2 + ) + { + _hash = await reader + .ReadBytesAsync(BLAKE2_DIGEST_SIZE, cancellationToken) + .ConfigureAwait(false); + } + } + break; + case FHEXTRA_HTIME: + { + var flags = await reader + .ReadRarVIntUInt16Async(cancellationToken: cancellationToken) + .ConfigureAwait(false); + var isWindowsTime = (flags & 1) == 0; + if ((flags & 0x2) == 0x2) + { + FileLastModifiedTime = await ReadExtendedTimeV5Async( + reader, + isWindowsTime, + cancellationToken + ) + .ConfigureAwait(false); + } + if ((flags & 0x4) == 0x4) + { + FileCreatedTime = await ReadExtendedTimeV5Async( + reader, + isWindowsTime, + cancellationToken + ) + .ConfigureAwait(false); + } + if ((flags & 0x8) == 0x8) + { + FileLastAccessedTime = await ReadExtendedTimeV5Async( + reader, + isWindowsTime, + cancellationToken + ) + .ConfigureAwait(false); + } + } + break; + case FHEXTRA_REDIR: + { + RedirType = await reader + .ReadRarVIntByteAsync(cancellationToken: cancellationToken) + .ConfigureAwait(false); + RedirFlags = await reader + .ReadRarVIntByteAsync(cancellationToken: cancellationToken) + .ConfigureAwait(false); + var nn = await reader + .ReadRarVIntUInt16Async(cancellationToken: cancellationToken) + .ConfigureAwait(false); + var bb = await reader + .ReadBytesAsync(nn, cancellationToken) + .ConfigureAwait(false); + RedirTargetName = ConvertPathV5(Encoding.UTF8.GetString(bb, 0, bb.Length)); + } + break; + default: + break; + } + var did = (int)(n - (HeaderSize - reader.CurrentReadByteCount)); + var drain = size - did; + if (drain > 0) + { + await reader.ReadBytesAsync(drain, cancellationToken).ConfigureAwait(false); + } + } + + if (AdditionalDataSize != 0) + { + CompressedSize = AdditionalDataSize; + } + } + + private async ValueTask ReadFromReaderV4Async( + AsyncMarkingBinaryReader reader, + CancellationToken cancellationToken + ) + { + Flags = HeaderFlags; + IsSolid = HasFlag(FileFlagsV4.SOLID); + WindowSize = IsDirectory + ? 0U + : ((size_t)0x10000) << ((Flags & FileFlagsV4.WINDOW_MASK) >> 5); + + var lowUncompressedSize = await reader + .ReadUInt32Async(cancellationToken) + .ConfigureAwait(false); + + _ = await reader.ReadByteAsync(cancellationToken).ConfigureAwait(false); + + FileCrc = await reader.ReadBytesAsync(4, cancellationToken).ConfigureAwait(false); + + FileLastModifiedTime = Utility.DosDateToDateTime( + await reader.ReadUInt32Async(cancellationToken).ConfigureAwait(false) + ); + + CompressionAlgorithm = await reader.ReadByteAsync(cancellationToken).ConfigureAwait(false); + CompressionMethod = (byte)( + (await reader.ReadByteAsync(cancellationToken).ConfigureAwait(false)) - 0x30 + ); + + var nameSize = await reader.ReadInt16Async(cancellationToken).ConfigureAwait(false); + + FileAttributes = await reader.ReadUInt32Async(cancellationToken).ConfigureAwait(false); + + uint highCompressedSize = 0; + uint highUncompressedkSize = 0; + if (HasFlag(FileFlagsV4.LARGE)) + { + highCompressedSize = await reader + .ReadUInt32Async(cancellationToken) + .ConfigureAwait(false); + highUncompressedkSize = await reader + .ReadUInt32Async(cancellationToken) + .ConfigureAwait(false); + } + else + { + if (lowUncompressedSize == 0xffffffff) + { + lowUncompressedSize = 0xffffffff; + highUncompressedkSize = int.MaxValue; + } + } + CompressedSize = UInt32To64(highCompressedSize, checked((uint)AdditionalDataSize)); + UncompressedSize = UInt32To64(highUncompressedkSize, lowUncompressedSize); + + nameSize = nameSize > 4 * 1024 ? (short)(4 * 1024) : nameSize; + + var fileNameBytes = await reader + .ReadBytesAsync(nameSize, cancellationToken) + .ConfigureAwait(false); + + const int newLhdSize = 32; + + switch (HeaderCode) + { + case HeaderCodeV.RAR4_FILE_HEADER: + { + if (HasFlag(FileFlagsV4.UNICODE)) + { + var length = 0; + while (length < fileNameBytes.Length && fileNameBytes[length] != 0) + { + length++; + } + if (length != nameSize) + { + length++; + FileName = FileNameDecoder.Decode(fileNameBytes, length); + } + else + { + FileName = ArchiveEncoding.Decode(fileNameBytes); + } + } + else + { + FileName = ArchiveEncoding.Decode(fileNameBytes); + } + FileName = ConvertPathV4(FileName); + } + break; + case HeaderCodeV.RAR4_NEW_SUB_HEADER: + { + var datasize = HeaderSize - newLhdSize - nameSize; + if (HasFlag(FileFlagsV4.SALT)) + { + datasize -= EncryptionConstV5.SIZE_SALT30; + } + if (datasize > 0) + { + SubData = await reader + .ReadBytesAsync(datasize, cancellationToken) + .ConfigureAwait(false); + } + + if (NewSubHeaderType.SUBHEAD_TYPE_RR.Equals(fileNameBytes.Take(4).ToArray())) + { + if (SubData is null) + { + throw new InvalidFormatException(); + } + RecoverySectors = + SubData[8] + + (SubData[9] << 8) + + (SubData[10] << 16) + + (SubData[11] << 24); + } + } + break; + } + + if (HasFlag(FileFlagsV4.SALT)) + { + R4Salt = await reader + .ReadBytesAsync(EncryptionConstV5.SIZE_SALT30, cancellationToken) + .ConfigureAwait(false); + } + if (HasFlag(FileFlagsV4.EXT_TIME)) + { + if (reader.CurrentReadByteCount >= 2) + { + var extendedFlags = await reader + .ReadUInt16Async(cancellationToken) + .ConfigureAwait(false); + if (FileLastModifiedTime is not null) + { + FileLastModifiedTime = await ProcessExtendedTimeV4Async( + extendedFlags, + FileLastModifiedTime, + reader, + 0, + cancellationToken + ) + .ConfigureAwait(false); + } + + FileCreatedTime = await ProcessExtendedTimeV4Async( + extendedFlags, + null, + reader, + 1, + cancellationToken + ) + .ConfigureAwait(false); + FileLastAccessedTime = await ProcessExtendedTimeV4Async( + extendedFlags, + null, + reader, + 2, + cancellationToken + ) + .ConfigureAwait(false); + FileArchivedTime = await ProcessExtendedTimeV4Async( + extendedFlags, + null, + reader, + 3, + cancellationToken + ) + .ConfigureAwait(false); + } + } + } + + private static async ValueTask ReadExtendedTimeV5Async( + AsyncMarkingBinaryReader reader, + bool isWindowsTime, + CancellationToken cancellationToken + ) + { + if (isWindowsTime) + { + return DateTime.FromFileTime( + await reader.ReadInt64Async(cancellationToken).ConfigureAwait(false) + ); + } + else + { + return Utility.UnixTimeToDateTime( + await reader.ReadUInt32Async(cancellationToken).ConfigureAwait(false) + ); + } + } + + private static async ValueTask ProcessExtendedTimeV4Async( + ushort extendedFlags, + DateTime? time, + AsyncMarkingBinaryReader reader, + int i, + CancellationToken cancellationToken + ) + { + var rmode = (uint)extendedFlags >> ((3 - i) * 4); + if ((rmode & 8) == 0) + { + return null; + } + if (i != 0) + { + var dosTime = await reader.ReadUInt32Async(cancellationToken).ConfigureAwait(false); + time = Utility.DosDateToDateTime(dosTime); + } + if ((rmode & 4) == 0 && time is not null) + { + time = time.Value.AddSeconds(1); + } + uint nanosecondHundreds = 0; + var count = (int)rmode & 3; + for (var j = 0; j < count; j++) + { + var b = await reader.ReadByteAsync(cancellationToken).ConfigureAwait(false); + nanosecondHundreds |= (((uint)b) << ((j + 3 - count) * 8)); + } + + if (time is not null) + { + return time.Value.AddMilliseconds(nanosecondHundreds * Math.Pow(10, -4)); + } + return null; + } +} diff --git a/src/SharpCompress/Common/Rar/Headers/FileHeader.cs b/src/SharpCompress/Common/Rar/Headers/FileHeader.cs index e95052ef..cbf78503 100644 --- a/src/SharpCompress/Common/Rar/Headers/FileHeader.cs +++ b/src/SharpCompress/Common/Rar/Headers/FileHeader.cs @@ -1,26 +1,24 @@ -#nullable disable - -#if !Rar2017_64bit -using size_t = System.UInt32; -#else -using nint = System.Int64; -using nuint = System.UInt64; -using size_t = System.UInt64; -#endif - -using SharpCompress.IO; using System; using System.IO; +using System.Linq; using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Rar; +using SharpCompress.IO; +using size_t = System.UInt32; namespace SharpCompress.Common.Rar.Headers; -internal class FileHeader : RarHeader +internal partial class FileHeader : RarHeader { - private uint _fileCrc; + private byte[]? _hash; - public FileHeader(RarHeader header, RarCrcBinaryReader reader, HeaderType headerType) - : base(header, reader, headerType) { } + public static FileHeader Create( + RarHeader header, + RarCrcBinaryReader reader, + HeaderType headerType + ) => CreateChild(header, reader, headerType); protected override void ReadFinish(MarkingBinaryReader reader) { @@ -52,7 +50,7 @@ internal class FileHeader : RarHeader if (HasFlag(FileFlagsV5.HAS_CRC32)) { - FileCrc = reader.ReadUInt32(); + FileCrc = reader.ReadBytes(4); } var compressionInfo = reader.ReadRarVIntUInt16(); @@ -74,27 +72,10 @@ internal class FileHeader : RarHeader // Bits 11 - 14 (0x3c00) define the minimum size of dictionary size required to extract data. Value 0 means 128 KB, 1 - 256 KB, ..., 14 - 2048 MB, 15 - 4096 MB. WindowSize = IsDirectory ? 0 : ((size_t)0x20000) << ((compressionInfo >> 10) & 0xf); - HostOs = reader.ReadRarVIntByte(); + _ = reader.ReadRarVIntByte(); var nameSize = reader.ReadRarVIntUInt16(); - // Variable length field containing Name length bytes in UTF-8 format without trailing zero. - // For file header this is a name of archived file. Forward slash character is used as the path separator both for Unix and Windows names. - // Backslashes are treated as a part of name for Unix names and as invalid character for Windows file names. Type of name is defined by Host OS field. - // - // TODO: not sure if anything needs to be done to handle the following: - // If Unix file name contains any high ASCII characters which cannot be correctly converted to Unicode and UTF-8 - // we map such characters to to 0xE080 - 0xE0FF private use Unicode area and insert 0xFFFE Unicode non-character - // to resulting string to indicate that it contains mapped characters, which need to be converted back when extracting. - // Concrete position of 0xFFFE is not defined, we need to search the entire string for it. Such mapped names are not - // portable and can be correctly unpacked only on the same system where they were created. - // - // For service header this field contains a name of service header. Now the following names are used: - // CMT Archive comment - // QO Archive quick open data - // ACL NTFS file permissions - // STM NTFS alternate data stream - // RR Recovery record var b = reader.ReadBytes(nameSize); FileName = ConvertPathV5(Encoding.UTF8.GetString(b, 0, b.Length)); @@ -104,7 +85,13 @@ internal class FileHeader : RarHeader throw new InvalidFormatException("rar5 header size / extra size inconsistency"); } - isEncryptedRar5 = false; + const ushort FHEXTRA_CRYPT = 0x01; + const ushort FHEXTRA_HASH = 0x02; + const ushort FHEXTRA_HTIME = 0x03; + // const ushort FHEXTRA_VERSION = 0x04; + const ushort FHEXTRA_REDIR = 0x05; + // const ushort FHEXTRA_UOWNER = 0x06; + // const ushort FHEXTRA_SUBDATA = 0x07; while (RemainingHeaderBytes(reader) > 0) { @@ -113,23 +100,27 @@ internal class FileHeader : RarHeader var type = reader.ReadRarVIntUInt16(); switch (type) { - //TODO - case 1: // file encryption - + case FHEXTRA_CRYPT: // file encryption { - isEncryptedRar5 = true; + Rar5CryptoInfo = Rar5CryptoInfo.Create(reader, true); - //var version = reader.ReadRarVIntByte(); - //if (version != 0) throw new InvalidFormatException("unknown encryption algorithm " + version); + if (Rar5CryptoInfo.PswCheck.All(singleByte => singleByte == 0)) + { + Rar5CryptoInfo = null; + } } break; - // case 2: // file hash - // { - // - // } - // break; - case 3: // file time - + case FHEXTRA_HASH: + { + const uint FHEXTRA_HASH_BLAKE2 = 0x0; + const int BLAKE2_DIGEST_SIZE = 0x20; + if ((uint)reader.ReadRarVInt() == FHEXTRA_HASH_BLAKE2) + { + _hash = reader.ReadBytes(BLAKE2_DIGEST_SIZE); + } + } + break; + case FHEXTRA_HTIME: // file time { var flags = reader.ReadRarVIntUInt16(); var isWindowsTime = (flags & 1) == 0; @@ -147,30 +138,16 @@ internal class FileHeader : RarHeader } } break; - //TODO - // case 4: // file version - // { - // - // } - // break; - // case 5: // file system redirection - // { - // - // } - // break; - // case 6: // unix owner - // { - // - // } - // break; - // case 7: // service data - // { - // - // } - // break; - + case FHEXTRA_REDIR: // file system redirection + { + RedirType = reader.ReadRarVIntByte(); + RedirFlags = reader.ReadRarVIntByte(); + var nn = reader.ReadRarVIntUInt16(); + var bb = reader.ReadBytes(nn); + RedirTargetName = ConvertPathV5(Encoding.UTF8.GetString(bb, 0, bb.Length)); + } + break; default: - // skip unknown record types to allow new record types to be added in the future break; } // drain any trailing bytes of extra record @@ -220,9 +197,9 @@ internal class FileHeader : RarHeader var lowUncompressedSize = reader.ReadUInt32(); - HostOs = reader.ReadByte(); + _ = reader.ReadByte(); - FileCrc = reader.ReadUInt32(); + FileCrc = reader.ReadBytes(4); FileLastModifiedTime = Utility.DosDateToDateTime(reader.ReadUInt32()); @@ -255,13 +232,11 @@ internal class FileHeader : RarHeader var fileNameBytes = reader.ReadBytes(nameSize); - const int saltSize = 8; const int newLhdSize = 32; switch (HeaderCode) { case HeaderCodeV.RAR4_FILE_HEADER: - { if (HasFlag(FileFlagsV4.UNICODE)) { @@ -288,12 +263,11 @@ internal class FileHeader : RarHeader } break; case HeaderCodeV.RAR4_NEW_SUB_HEADER: - { var datasize = HeaderSize - newLhdSize - nameSize; if (HasFlag(FileFlagsV4.SALT)) { - datasize -= saltSize; + datasize -= EncryptionConstV5.SIZE_SALT30; } if (datasize > 0) { @@ -302,6 +276,10 @@ internal class FileHeader : RarHeader if (NewSubHeaderType.SUBHEAD_TYPE_RR.Equals(fileNameBytes)) { + if (SubData is null) + { + throw new InvalidFormatException(); + } RecoverySectors = SubData[8] + (SubData[9] << 8) @@ -314,21 +292,23 @@ internal class FileHeader : RarHeader if (HasFlag(FileFlagsV4.SALT)) { - R4Salt = reader.ReadBytes(saltSize); + R4Salt = reader.ReadBytes(EncryptionConstV5.SIZE_SALT30); } if (HasFlag(FileFlagsV4.EXT_TIME)) { - // verify that the end of the header hasn't been reached before reading the Extended Time. - // some tools incorrectly omit Extended Time despite specifying FileFlags.EXTTIME, which most parsers tolerate. if (RemainingHeaderBytes(reader) >= 2) { var extendedFlags = reader.ReadUInt16(); - FileLastModifiedTime = ProcessExtendedTimeV4( - extendedFlags, - FileLastModifiedTime, - reader, - 0 - ); + if (FileLastModifiedTime is not null) + { + FileLastModifiedTime = ProcessExtendedTimeV4( + extendedFlags, + FileLastModifiedTime, + reader, + 0 + ); + } + FileCreatedTime = ProcessExtendedTimeV4(extendedFlags, null, reader, 1); FileLastAccessedTime = ProcessExtendedTimeV4(extendedFlags, null, reader, 2); FileArchivedTime = ProcessExtendedTimeV4(extendedFlags, null, reader, 3); @@ -360,7 +340,7 @@ internal class FileHeader : RarHeader var dosTime = reader.ReadUInt32(); time = Utility.DosDateToDateTime(dosTime); } - if ((rmode & 4) == 0) + if ((rmode & 4) == 0 && time is not null) { time = time.Value.AddSeconds(1); } @@ -373,7 +353,11 @@ internal class FileHeader : RarHeader } //10^-7 to 10^-3 - return time.Value.AddMilliseconds(nanosecondHundreds * Math.Pow(10, -4)); + if (time is not null) + { + return time.Value.AddMilliseconds(nanosecondHundreds * Math.Pow(10, -4)); + } + return null; } private static string ConvertPathV4(string path) @@ -389,24 +373,16 @@ internal class FileHeader : RarHeader return path; } - public override string ToString() => FileName; + public override string ToString() => FileName ?? "FileHeader"; private ushort Flags { get; set; } private bool HasFlag(ushort flag) => (Flags & flag) == flag; - internal uint FileCrc + internal byte[]? FileCrc { - get - { - if (IsRar5 && !HasFlag(FileFlagsV5.HAS_CRC32)) - { - //!!! rar5: - throw new InvalidOperationException("TODO rar5"); - } - return _fileCrc; - } - private set => _fileCrc = value; + get => _hash; + private set => _hash = value; } // 0 - storing @@ -428,20 +404,25 @@ internal class FileHeader : RarHeader public bool IsSolid { get; private set; } + public byte RedirType { get; private set; } + public bool IsRedir => RedirType != 0; + public byte RedirFlags { get; private set; } + public bool IsRedirDirectory => (RedirFlags & RedirFlagV5.DIRECTORY) != 0; + public string? RedirTargetName { get; private set; } + // unused for UnpackV1 implementation (limitation) internal size_t WindowSize { get; private set; } - internal byte[] R4Salt { get; private set; } - - private byte HostOs { get; set; } + internal byte[]? R4Salt { get; private set; } + internal Rar5CryptoInfo? Rar5CryptoInfo { get; private set; } internal uint FileAttributes { get; private set; } internal long CompressedSize { get; private set; } internal long UncompressedSize { get; private set; } - internal string FileName { get; private set; } - internal byte[] SubData { get; private set; } + internal string? FileName { get; private set; } + internal byte[]? SubData { get; private set; } internal int RecoverySectors { get; private set; } internal long DataStartPosition { get; set; } - public Stream PackedStream { get; set; } + public Stream? PackedStream { get; set; } public bool IsSplitBefore => IsRar5 ? HasHeaderFlag(HeaderFlagsV5.SPLIT_BEFORE) : HasFlag(FileFlagsV4.SPLIT_BEFORE); @@ -450,8 +431,7 @@ internal class FileHeader : RarHeader public bool IsDirectory => HasFlag(IsRar5 ? FileFlagsV5.DIRECTORY : FileFlagsV4.DIRECTORY); - private bool isEncryptedRar5 = false; - public bool IsEncrypted => IsRar5 ? isEncryptedRar5 : HasFlag(FileFlagsV4.PASSWORD); + public bool IsEncrypted => IsRar5 ? Rar5CryptoInfo != null : HasFlag(FileFlagsV4.PASSWORD); internal DateTime? FileLastModifiedTime { get; private set; } diff --git a/src/SharpCompress/Common/Rar/Headers/Flags.cs b/src/SharpCompress/Common/Rar/Headers/Flags.cs index 5a52f001..ef654ad5 100644 --- a/src/SharpCompress/Common/Rar/Headers/Flags.cs +++ b/src/SharpCompress/Common/Rar/Headers/Flags.cs @@ -13,7 +13,7 @@ public enum HeaderType : byte Sign, NewSub, EndArchive, - Crypt + Crypt, } internal static class HeaderCodeV @@ -50,6 +50,17 @@ internal static class EncryptionFlagsV5 public const uint FHEXTRA_CRYPT_HASHMAC = 0x02; } +internal static class EncryptionConstV5 +{ + public const int VERSION = 0; + public const uint CRYPT5_KDF_LG2_COUNT_MAX = 0x24; + public const int SIZE_SALT30 = 0x08; + public const int SIZE_SALT50 = 0x10; + public const int SIZE_INITV = 0x10; + public const int SIZE_PSWCHECK = 0x08; + public const int SIZE_PSWCHECK_CSUM = 0x04; +} + internal static class HeaderFlagsV5 { public const ushort HAS_EXTRA = 0x0001; @@ -146,3 +157,17 @@ internal static class EndArchiveFlagsV5 { public const ushort HAS_NEXT_VOLUME = 0x0001; } + +internal static class RedirTypeV5 +{ + public const byte UNIX_SYMLINK = 0x0001; + public const byte WIN_SYMLINK = 0x0002; + public const byte WIN_JUNCTION = 0x0003; + public const byte HARD_LINK = 0x0004; + public const byte FILE_COPY = 0x0005; +} + +internal static class RedirFlagV5 +{ + public const byte DIRECTORY = 0x0001; +} diff --git a/src/SharpCompress/Common/Rar/Headers/MarkHeader.Async.cs b/src/SharpCompress/Common/Rar/Headers/MarkHeader.Async.cs new file mode 100644 index 00000000..77cf4476 --- /dev/null +++ b/src/SharpCompress/Common/Rar/Headers/MarkHeader.Async.cs @@ -0,0 +1,136 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Common.Rar.Headers; + +internal partial class MarkHeader +{ + private static async ValueTask GetByteAsync( + Stream stream, + CancellationToken cancellationToken + ) + { + var buffer = new byte[1]; + var bytesRead = await stream + .ReadAsync(buffer, 0, 1, cancellationToken) + .ConfigureAwait(false); + if (bytesRead == 1) + { + return buffer[0]; + } + throw new IncompleteArchiveException("Unexpected end of stream."); + } + + public static async ValueTask ReadAsync( + Stream stream, + bool leaveStreamOpen, + bool lookForHeader, + CancellationToken cancellationToken = default + ) + { + var maxScanIndex = lookForHeader ? MAX_SFX_SIZE : 0; + try + { + var start = -1; + var b = await GetByteAsync(stream, cancellationToken).ConfigureAwait(false); + start++; + while (start <= maxScanIndex) + { + if (b == 0x52) + { + b = await GetByteAsync(stream, cancellationToken).ConfigureAwait(false); + start++; + if (b == 0x61) + { + b = await GetByteAsync(stream, cancellationToken).ConfigureAwait(false); + start++; + if (b != 0x72) + { + continue; + } + + b = await GetByteAsync(stream, cancellationToken).ConfigureAwait(false); + start++; + if (b != 0x21) + { + continue; + } + + b = await GetByteAsync(stream, cancellationToken).ConfigureAwait(false); + start++; + if (b != 0x1a) + { + continue; + } + + b = await GetByteAsync(stream, cancellationToken).ConfigureAwait(false); + start++; + if (b != 0x07) + { + continue; + } + + b = await GetByteAsync(stream, cancellationToken).ConfigureAwait(false); + start++; + if (b == 1) + { + b = await GetByteAsync(stream, cancellationToken).ConfigureAwait(false); + start++; + if (b != 0) + { + continue; + } + + return new MarkHeader(true); // Rar5 + } + else if (b == 0) + { + return new MarkHeader(false); // Rar4 + } + } + else if (b == 0x45) + { + b = await GetByteAsync(stream, cancellationToken).ConfigureAwait(false); + start++; + if (b != 0x7e) + { + continue; + } + + b = await GetByteAsync(stream, cancellationToken).ConfigureAwait(false); + start++; + if (b != 0x5e) + { + continue; + } + + throw new InvalidFormatException( + "Rar format version pre-4 is unsupported." + ); + } + } + else + { + b = await GetByteAsync(stream, cancellationToken).ConfigureAwait(false); + start++; + } + } + } + catch (Exception e) + { + if (!leaveStreamOpen) + { +#if LEGACY_DOTNET && !NETSTANDARD2_1 + stream.Dispose(); +#else + await stream.DisposeAsync().ConfigureAwait(false); +#endif + } + throw new InvalidFormatException("Error trying to read rar signature.", e); + } + + throw new InvalidFormatException("Rar signature not found"); + } +} diff --git a/src/SharpCompress/Common/Rar/Headers/MarkHeader.cs b/src/SharpCompress/Common/Rar/Headers/MarkHeader.cs index 9ca0f9b2..28b27cdb 100644 --- a/src/SharpCompress/Common/Rar/Headers/MarkHeader.cs +++ b/src/SharpCompress/Common/Rar/Headers/MarkHeader.cs @@ -1,9 +1,11 @@ using System; using System.IO; +using System.Threading; +using System.Threading.Tasks; namespace SharpCompress.Common.Rar.Headers; -internal class MarkHeader : IRarHeader +internal partial class MarkHeader : IRarHeader { private const int MAX_SFX_SIZE = 0x80000 - 16; //archive.cpp line 136 @@ -22,7 +24,7 @@ internal class MarkHeader : IRarHeader { return (byte)b; } - throw new EndOfStreamException(); + throw new IncompleteArchiveException("Unexpected end of stream."); } public static MarkHeader Read(Stream stream, bool leaveStreamOpen, bool lookForHeader) diff --git a/src/SharpCompress/Common/Rar/Headers/NewSubHeaderType.cs b/src/SharpCompress/Common/Rar/Headers/NewSubHeaderType.cs index b45f98bb..6285c4d2 100644 --- a/src/SharpCompress/Common/Rar/Headers/NewSubHeaderType.cs +++ b/src/SharpCompress/Common/Rar/Headers/NewSubHeaderType.cs @@ -42,4 +42,20 @@ internal sealed class NewSubHeaderType : IEquatable } public bool Equals(NewSubHeaderType? other) => other is not null && Equals(other._bytes); + + public override bool Equals(object? obj) => obj is NewSubHeaderType other && Equals(other); + + public override int GetHashCode() + { + unchecked + { + var hash = 17; + foreach (byte value in _bytes) + { + hash = (hash * 31) + value; + } + + return hash; + } + } } diff --git a/src/SharpCompress/Common/Rar/Headers/ProtectHeader.Async.cs b/src/SharpCompress/Common/Rar/Headers/ProtectHeader.Async.cs new file mode 100644 index 00000000..0fe33a38 --- /dev/null +++ b/src/SharpCompress/Common/Rar/Headers/ProtectHeader.Async.cs @@ -0,0 +1,40 @@ +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Rar; +using SharpCompress.IO; + +namespace SharpCompress.Common.Rar.Headers; + +internal sealed partial class ProtectHeader +{ + public static async ValueTask CreateAsync( + RarHeader header, + AsyncRarCrcBinaryReader reader, + CancellationToken cancellationToken = default + ) + { + var c = await CreateChildAsync( + header, + reader, + HeaderType.Protect, + cancellationToken + ) + .ConfigureAwait(false); + if (c.IsRar5) + { + throw new InvalidFormatException("unexpected rar5 record"); + } + return c; + } + + protected sealed override async ValueTask ReadFinishAsync( + AsyncMarkingBinaryReader reader, + CancellationToken cancellationToken = default + ) + { + Version = await reader.ReadByteAsync(cancellationToken).ConfigureAwait(false); + RecSectors = await reader.ReadUInt16Async(cancellationToken).ConfigureAwait(false); + TotalBlocks = await reader.ReadUInt32Async(cancellationToken).ConfigureAwait(false); + Mark = await reader.ReadBytesAsync(8, cancellationToken).ConfigureAwait(false); + } +} diff --git a/src/SharpCompress/Common/Rar/Headers/ProtectHeader.cs b/src/SharpCompress/Common/Rar/Headers/ProtectHeader.cs index cf8db3c0..71b9c420 100644 --- a/src/SharpCompress/Common/Rar/Headers/ProtectHeader.cs +++ b/src/SharpCompress/Common/Rar/Headers/ProtectHeader.cs @@ -1,20 +1,21 @@ +using SharpCompress.Common.Rar; using SharpCompress.IO; namespace SharpCompress.Common.Rar.Headers; -// ProtectHeader is part of the Recovery Record feature -internal sealed class ProtectHeader : RarHeader +internal sealed partial class ProtectHeader : RarHeader { - public ProtectHeader(RarHeader header, RarCrcBinaryReader reader) - : base(header, reader, HeaderType.Protect) + public static ProtectHeader Create(RarHeader header, RarCrcBinaryReader reader) { - if (IsRar5) + var c = CreateChild(header, reader, HeaderType.Protect); + if (c.IsRar5) { throw new InvalidFormatException("unexpected rar5 record"); } + return c; } - protected override void ReadFinish(MarkingBinaryReader reader) + protected sealed override void ReadFinish(MarkingBinaryReader reader) { Version = reader.ReadByte(); RecSectors = reader.ReadUInt16(); diff --git a/src/SharpCompress/Common/Rar/Headers/RarHeader.Async.cs b/src/SharpCompress/Common/Rar/Headers/RarHeader.Async.cs new file mode 100644 index 00000000..e4692a97 --- /dev/null +++ b/src/SharpCompress/Common/Rar/Headers/RarHeader.Async.cs @@ -0,0 +1,115 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Rar; +using SharpCompress.IO; + +namespace SharpCompress.Common.Rar.Headers; + +internal partial class RarHeader +{ + internal static async ValueTask TryReadBaseAsync( + AsyncRarCrcBinaryReader reader, + bool isRar5, + IArchiveEncoding archiveEncoding, + CancellationToken cancellationToken = default + ) + { + try + { + var header = new RarHeader(); + await header + .InitializeAsync(reader, isRar5, archiveEncoding, cancellationToken) + .ConfigureAwait(false); + return header; + } + catch (InvalidFormatException) + { + return null; + } + } + + private async ValueTask InitializeAsync( + AsyncRarCrcBinaryReader reader, + bool isRar5, + IArchiveEncoding archiveEncoding, + CancellationToken cancellationToken + ) + { + _headerType = HeaderType.Null; + _isRar5 = isRar5; + ArchiveEncoding = archiveEncoding; + if (IsRar5) + { + HeaderCrc = await reader.ReadUInt32Async(cancellationToken).ConfigureAwait(false); + reader.ResetCrc(); + HeaderSize = (int) + await reader.ReadRarVIntUInt32Async(3, cancellationToken).ConfigureAwait(false); + reader.Mark(); + HeaderCode = await reader + .ReadRarVIntByteAsync(cancellationToken: cancellationToken) + .ConfigureAwait(false); + HeaderFlags = await reader + .ReadRarVIntUInt16Async(2, cancellationToken) + .ConfigureAwait(false); + + if (HasHeaderFlag(HeaderFlagsV5.HAS_EXTRA)) + { + ExtraSize = await reader + .ReadRarVIntUInt32Async(cancellationToken: cancellationToken) + .ConfigureAwait(false); + } + if (HasHeaderFlag(HeaderFlagsV5.HAS_DATA)) + { + AdditionalDataSize = (long) + await reader + .ReadRarVIntAsync(cancellationToken: cancellationToken) + .ConfigureAwait(false); + } + } + else + { + reader.Mark(); + HeaderCrc = await reader.ReadUInt16Async(cancellationToken).ConfigureAwait(false); + reader.ResetCrc(); + HeaderCode = await reader.ReadByteAsync(cancellationToken).ConfigureAwait(false); + HeaderFlags = await reader.ReadUInt16Async(cancellationToken).ConfigureAwait(false); + HeaderSize = await reader.ReadInt16Async(cancellationToken).ConfigureAwait(false); + if (HasHeaderFlag(HeaderFlagsV4.HAS_DATA)) + { + AdditionalDataSize = await reader + .ReadUInt32Async(cancellationToken) + .ConfigureAwait(false); + } + } + } + + internal static async ValueTask CreateChildAsync( + RarHeader header, + AsyncRarCrcBinaryReader reader, + HeaderType headerType, + CancellationToken cancellationToken = default + ) + where T : RarHeader, new() + { + var child = new T() { ArchiveEncoding = header.ArchiveEncoding }; + child._headerType = headerType; + child._isRar5 = header.IsRar5; + child.HeaderCrc = header.HeaderCrc; + child.HeaderCode = header.HeaderCode; + child.HeaderFlags = header.HeaderFlags; + child.HeaderSize = header.HeaderSize; + child.ExtraSize = header.ExtraSize; + child.AdditionalDataSize = header.AdditionalDataSize; + await child.ReadFinishAsync(reader, cancellationToken).ConfigureAwait(false); + + var n = child.RemainingHeaderBytesAsync(reader); + if (n > 0) + { + await reader.ReadBytesAsync(n, cancellationToken).ConfigureAwait(false); + } + + child.VerifyHeaderCrc(reader.GetCrc32()); + return child; + } +} diff --git a/src/SharpCompress/Common/Rar/Headers/RarHeader.cs b/src/SharpCompress/Common/Rar/Headers/RarHeader.cs index 81002fc4..23e08060 100644 --- a/src/SharpCompress/Common/Rar/Headers/RarHeader.cs +++ b/src/SharpCompress/Common/Rar/Headers/RarHeader.cs @@ -1,33 +1,46 @@ -using System; -using System.IO; +using System; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Rar; using SharpCompress.IO; namespace SharpCompress.Common.Rar.Headers; // http://www.forensicswiki.org/w/images/5/5b/RARFileStructure.txt // https://www.rarlab.com/technote.htm -internal class RarHeader : IRarHeader +internal partial class RarHeader : IRarHeader { - private readonly HeaderType _headerType; - private readonly bool _isRar5; + private HeaderType _headerType; + private bool _isRar5; + + protected RarHeader() + { + ArchiveEncoding = new ArchiveEncoding(); + } internal static RarHeader? TryReadBase( RarCrcBinaryReader reader, bool isRar5, - ArchiveEncoding archiveEncoding + IArchiveEncoding archiveEncoding ) { try { - return new RarHeader(reader, isRar5, archiveEncoding); + var header = new RarHeader(); + header.Initialize(reader, isRar5, archiveEncoding); + return header; } - catch (EndOfStreamException) + catch (InvalidFormatException) { return null; } } - private RarHeader(RarCrcBinaryReader reader, bool isRar5, ArchiveEncoding archiveEncoding) + private void Initialize( + RarCrcBinaryReader reader, + bool isRar5, + IArchiveEncoding archiveEncoding + ) { _headerType = HeaderType.Null; _isRar5 = isRar5; @@ -65,34 +78,48 @@ internal class RarHeader : IRarHeader } } - protected RarHeader(RarHeader header, RarCrcBinaryReader reader, HeaderType headerType) + internal static T CreateChild( + RarHeader header, + RarCrcBinaryReader reader, + HeaderType headerType + ) + where T : RarHeader, new() { - _headerType = headerType; - _isRar5 = header.IsRar5; - HeaderCrc = header.HeaderCrc; - HeaderCode = header.HeaderCode; - HeaderFlags = header.HeaderFlags; - HeaderSize = header.HeaderSize; - ExtraSize = header.ExtraSize; - AdditionalDataSize = header.AdditionalDataSize; - ArchiveEncoding = header.ArchiveEncoding; - ReadFinish(reader); + var child = new T() { ArchiveEncoding = header.ArchiveEncoding }; + child._headerType = headerType; + child._isRar5 = header.IsRar5; + child.HeaderCrc = header.HeaderCrc; + child.HeaderCode = header.HeaderCode; + child.HeaderFlags = header.HeaderFlags; + child.HeaderSize = header.HeaderSize; + child.ExtraSize = header.ExtraSize; + child.AdditionalDataSize = header.AdditionalDataSize; + child.ReadFinish(reader); - var n = RemainingHeaderBytes(reader); + var n = child.RemainingHeaderBytes(reader); if (n > 0) { reader.ReadBytes(n); } - VerifyHeaderCrc(reader.GetCrc32()); + child.VerifyHeaderCrc(reader.GetCrc32()); + return child; } protected int RemainingHeaderBytes(MarkingBinaryReader reader) => checked(HeaderSize - (int)reader.CurrentReadByteCount); + protected int RemainingHeaderBytesAsync(AsyncMarkingBinaryReader reader) => + checked(HeaderSize - (int)reader.CurrentReadByteCount); + protected virtual void ReadFinish(MarkingBinaryReader reader) => throw new NotImplementedException(); + protected virtual ValueTask ReadFinishAsync( + AsyncMarkingBinaryReader reader, + CancellationToken cancellationToken = default + ) => throw new NotImplementedException(); + private void VerifyHeaderCrc(uint crc32) { var b = (IsRar5 ? crc32 : (ushort)crc32) == HeaderCrc; @@ -104,27 +131,27 @@ internal class RarHeader : IRarHeader public HeaderType HeaderType => _headerType; - protected bool IsRar5 => _isRar5; + internal bool IsRar5 => _isRar5; - protected uint HeaderCrc { get; } + protected uint HeaderCrc { get; private set; } - internal byte HeaderCode { get; } + internal byte HeaderCode { get; private set; } - protected ushort HeaderFlags { get; } + protected ushort HeaderFlags { get; private set; } protected bool HasHeaderFlag(ushort flag) => (HeaderFlags & flag) == flag; - protected int HeaderSize { get; } + protected int HeaderSize { get; private set; } - internal ArchiveEncoding ArchiveEncoding { get; } + internal IArchiveEncoding ArchiveEncoding { get; private set; } /// /// Extra header size. /// - protected uint ExtraSize { get; } + protected uint ExtraSize { get; private set; } /// /// Size of additional data (eg file contents) /// - protected long AdditionalDataSize { get; } + protected long AdditionalDataSize { get; private set; } } diff --git a/src/SharpCompress/Common/Rar/Headers/RarHeaderFactory.Async.cs b/src/SharpCompress/Common/Rar/Headers/RarHeaderFactory.Async.cs new file mode 100644 index 00000000..d020add9 --- /dev/null +++ b/src/SharpCompress/Common/Rar/Headers/RarHeaderFactory.Async.cs @@ -0,0 +1,256 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Rar; +using SharpCompress.IO; +using SharpCompress.Readers; + +namespace SharpCompress.Common.Rar.Headers; + +public partial class RarHeaderFactory +{ + public async IAsyncEnumerable ReadHeadersAsync(Stream stream) + { + var markHeader = await MarkHeader + .ReadAsync( + stream, + Options.LeaveStreamOpen, + Options.LookForHeader, + CancellationToken.None + ) + .ConfigureAwait(false); + _isRar5 = markHeader.IsRar5; + yield return markHeader; + + RarHeader? header; + while ( + ( + header = await TryReadNextHeaderAsync(stream, CancellationToken.None) + .ConfigureAwait(false) + ) != null + ) + { + yield return header; + if (header.HeaderType == HeaderType.EndArchive) + { + // End of archive marker. RAR does not read anything after this header letting to use third + // party tools to add extra information such as a digital signature to archive. + yield break; + } + } + } + + private async ValueTask TryReadNextHeaderAsync( + Stream stream, + CancellationToken cancellationToken + ) + { + AsyncRarCrcBinaryReader reader; + if (!IsEncrypted) + { + reader = new AsyncRarCrcBinaryReader(stream); + } + else + { + if (Options.Password is null) + { + throw new CryptographicException( + "Encrypted Rar archive has no password specified." + ); + } + + if (_isRar5 && _cryptInfo != null) + { + await _cryptInfo + .ReadInitVAsync(new AsyncMarkingBinaryReader(stream)) + .ConfigureAwait(false); + var _headerKey = new CryptKey5(Options.Password!, _cryptInfo); + + reader = await AsyncRarCryptoBinaryReader + .Create(stream, _headerKey, _cryptInfo.Salt) + .ConfigureAwait(false); + } + else + { + var key = new CryptKey3(Options.Password); + reader = await AsyncRarCryptoBinaryReader.Create(stream, key).ConfigureAwait(false); + } + } + + var header = await RarHeader + .TryReadBaseAsync(reader, _isRar5, Options.ArchiveEncoding, cancellationToken) + .ConfigureAwait(false); + if (header is null) + { + return null; + } + switch (header.HeaderCode) + { + case HeaderCodeV.RAR5_ARCHIVE_HEADER: + case HeaderCodeV.RAR4_ARCHIVE_HEADER: + { + var ah = await ArchiveHeader + .CreateAsync(header, reader, cancellationToken) + .ConfigureAwait(false); + if (ah.IsEncrypted == true) + { + //!!! rar5 we don't know yet + IsEncrypted = true; + } + return ah; + } + + case HeaderCodeV.RAR4_PROTECT_HEADER: + { + var ph = await ProtectHeader + .CreateAsync(header, reader, cancellationToken) + .ConfigureAwait(false); + // skip the recovery record data, we do not use it. + switch (StreamingMode) + { + case StreamingMode.Seekable: + { + reader.BaseStream.Position += ph.DataSize; + } + break; + case StreamingMode.Streaming: + { + await reader + .BaseStream.SkipAsync(ph.DataSize, cancellationToken) + .ConfigureAwait(false); + } + break; + default: + { + throw new InvalidFormatException("Invalid StreamingMode"); + } + } + return ph; + } + + case HeaderCodeV.RAR5_SERVICE_HEADER: + { + var fh = await FileHeader + .CreateAsync(header, reader, HeaderType.Service, cancellationToken) + .ConfigureAwait(false); + if (fh.FileName == "CMT") + { + fh.PackedStream = new ReadOnlySubStream(reader.BaseStream, fh.CompressedSize); + } + else + { + await SkipDataAsync(fh, reader, cancellationToken).ConfigureAwait(false); + } + return fh; + } + + case HeaderCodeV.RAR4_NEW_SUB_HEADER: + { + var fh = await FileHeader + .CreateAsync(header, reader, HeaderType.NewSub, cancellationToken) + .ConfigureAwait(false); + await SkipDataAsync(fh, reader, cancellationToken).ConfigureAwait(false); + return fh; + } + + case HeaderCodeV.RAR5_FILE_HEADER: + case HeaderCodeV.RAR4_FILE_HEADER: + { + var fh = await FileHeader + .CreateAsync(header, reader, HeaderType.File, cancellationToken) + .ConfigureAwait(false); + switch (StreamingMode) + { + case StreamingMode.Seekable: + { + fh.DataStartPosition = reader.BaseStream.Position; + reader.BaseStream.Position += fh.CompressedSize; + } + break; + case StreamingMode.Streaming: + { + var ms = new ReadOnlySubStream(reader.BaseStream, fh.CompressedSize); + if (fh.R4Salt is null && fh.Rar5CryptoInfo is null) + { + fh.PackedStream = ms; + } + else + { + fh.PackedStream = new RarCryptoWrapper( + ms, + fh.R4Salt is null + ? fh.Rar5CryptoInfo.NotNull().Salt + : fh.R4Salt, + fh.R4Salt is null + ? new CryptKey5( + Options.Password, + fh.Rar5CryptoInfo.NotNull() + ) + : new CryptKey3(Options.Password) + ); + } + } + break; + default: + { + throw new InvalidFormatException("Invalid StreamingMode"); + } + } + return fh; + } + case HeaderCodeV.RAR5_END_ARCHIVE_HEADER: + case HeaderCodeV.RAR4_END_ARCHIVE_HEADER: + { + return await EndArchiveHeader + .CreateAsync(header, reader, cancellationToken) + .ConfigureAwait(false); + } + case HeaderCodeV.RAR5_ARCHIVE_ENCRYPTION_HEADER: + { + var cryptoHeader = await ArchiveCryptHeader + .CreateAsync(header, reader, cancellationToken) + .ConfigureAwait(false); + IsEncrypted = true; + _cryptInfo = cryptoHeader.CryptInfo; + + return cryptoHeader; + } + default: + { + throw new InvalidFormatException("Unknown Rar Header: " + header.HeaderCode); + } + } + } + + private async ValueTask SkipDataAsync( + FileHeader fh, + AsyncRarCrcBinaryReader reader, + CancellationToken cancellationToken + ) + { + switch (StreamingMode) + { + case StreamingMode.Seekable: + { + fh.DataStartPosition = reader.BaseStream.Position; + reader.BaseStream.Position += fh.CompressedSize; + } + break; + case StreamingMode.Streaming: + { + //skip the data because it's useless? + await reader + .BaseStream.SkipAsync(fh.CompressedSize, cancellationToken) + .ConfigureAwait(false); + } + break; + default: + { + throw new InvalidFormatException("Invalid StreamingMode"); + } + } + } +} diff --git a/src/SharpCompress/Common/Rar/Headers/RarHeaderFactory.cs b/src/SharpCompress/Common/Rar/Headers/RarHeaderFactory.cs index d371166d..e9694c2a 100644 --- a/src/SharpCompress/Common/Rar/Headers/RarHeaderFactory.cs +++ b/src/SharpCompress/Common/Rar/Headers/RarHeaderFactory.cs @@ -1,14 +1,19 @@ +using System; using System.Collections.Generic; using System.IO; +using System.Linq; +using SharpCompress.Common.Rar; using SharpCompress.IO; using SharpCompress.Readers; namespace SharpCompress.Common.Rar.Headers; -public class RarHeaderFactory +public partial class RarHeaderFactory { private bool _isRar5; + private Rar5CryptoInfo? _cryptInfo; + public RarHeaderFactory(StreamingMode mode, ReaderOptions options) { StreamingMode = mode; @@ -53,7 +58,19 @@ public class RarHeaderFactory "Encrypted Rar archive has no password specified." ); } - reader = new RarCryptoBinaryReader(stream, Options.Password); + + if (_isRar5 && _cryptInfo != null) + { + _cryptInfo.ReadInitV(new MarkingBinaryReader(stream)); + var _headerKey = new CryptKey5(Options.Password!, _cryptInfo); + + reader = RarCryptoBinaryReader.Create(stream, _headerKey, _cryptInfo.Salt); + } + else + { + var key = new CryptKey3(Options.Password); + reader = RarCryptoBinaryReader.Create(stream, key); + } } var header = RarHeader.TryReadBase(reader, _isRar5, Options.ArchiveEncoding); @@ -66,7 +83,7 @@ public class RarHeaderFactory case HeaderCodeV.RAR5_ARCHIVE_HEADER: case HeaderCodeV.RAR4_ARCHIVE_HEADER: { - var ah = new ArchiveHeader(header, reader); + var ah = ArchiveHeader.Create(header, reader); if (ah.IsEncrypted == true) { //!!! rar5 we don't know yet @@ -77,18 +94,16 @@ public class RarHeaderFactory case HeaderCodeV.RAR4_PROTECT_HEADER: { - var ph = new ProtectHeader(header, reader); + var ph = ProtectHeader.Create(header, reader); // skip the recovery record data, we do not use it. switch (StreamingMode) { case StreamingMode.Seekable: - { reader.BaseStream.Position += ph.DataSize; } break; case StreamingMode.Streaming: - { reader.BaseStream.Skip(ph.DataSize); } @@ -104,14 +119,21 @@ public class RarHeaderFactory case HeaderCodeV.RAR5_SERVICE_HEADER: { - var fh = new FileHeader(header, reader, HeaderType.Service); - SkipData(fh, reader); + var fh = FileHeader.Create(header, reader, HeaderType.Service); + if (fh.FileName == "CMT") + { + fh.PackedStream = new ReadOnlySubStream(reader.BaseStream, fh.CompressedSize); + } + else + { + SkipData(fh, reader); + } return fh; } case HeaderCodeV.RAR4_NEW_SUB_HEADER: { - var fh = new FileHeader(header, reader, HeaderType.NewSub); + var fh = FileHeader.Create(header, reader, HeaderType.NewSub); SkipData(fh, reader); return fh; } @@ -119,21 +141,19 @@ public class RarHeaderFactory case HeaderCodeV.RAR5_FILE_HEADER: case HeaderCodeV.RAR4_FILE_HEADER: { - var fh = new FileHeader(header, reader, HeaderType.File); + var fh = FileHeader.Create(header, reader, HeaderType.File); switch (StreamingMode) { case StreamingMode.Seekable: - { fh.DataStartPosition = reader.BaseStream.Position; reader.BaseStream.Position += fh.CompressedSize; } break; case StreamingMode.Streaming: - { var ms = new ReadOnlySubStream(reader.BaseStream, fh.CompressedSize); - if (fh.R4Salt is null) + if (fh.R4Salt is null && fh.Rar5CryptoInfo is null) { fh.PackedStream = ms; } @@ -141,8 +161,15 @@ public class RarHeaderFactory { fh.PackedStream = new RarCryptoWrapper( ms, - Options.Password!, - fh.R4Salt + fh.R4Salt is null + ? fh.Rar5CryptoInfo.NotNull().Salt + : fh.R4Salt, + fh.R4Salt is null + ? new CryptKey5( + Options.Password, + fh.Rar5CryptoInfo.NotNull() + ) + : new CryptKey3(Options.Password) ); } } @@ -157,13 +184,15 @@ public class RarHeaderFactory case HeaderCodeV.RAR5_END_ARCHIVE_HEADER: case HeaderCodeV.RAR4_END_ARCHIVE_HEADER: { - return new EndArchiveHeader(header, reader); + return EndArchiveHeader.Create(header, reader); } case HeaderCodeV.RAR5_ARCHIVE_ENCRYPTION_HEADER: { - var ch = new ArchiveCryptHeader(header, reader); + var cryptoHeader = ArchiveCryptHeader.Create(header, reader); IsEncrypted = true; - return ch; + _cryptInfo = cryptoHeader.CryptInfo; + + return cryptoHeader; } default: { @@ -177,14 +206,12 @@ public class RarHeaderFactory switch (StreamingMode) { case StreamingMode.Seekable: - { fh.DataStartPosition = reader.BaseStream.Position; reader.BaseStream.Position += fh.CompressedSize; } break; case StreamingMode.Streaming: - { //skip the data because it's useless? reader.BaseStream.Skip(fh.CompressedSize); diff --git a/src/SharpCompress/Common/Rar/Headers/SignHeader.cs b/src/SharpCompress/Common/Rar/Headers/SignHeader.cs index 837ff3dc..f5487089 100644 --- a/src/SharpCompress/Common/Rar/Headers/SignHeader.cs +++ b/src/SharpCompress/Common/Rar/Headers/SignHeader.cs @@ -4,13 +4,14 @@ namespace SharpCompress.Common.Rar.Headers; internal class SignHeader : RarHeader { - protected SignHeader(RarHeader header, RarCrcBinaryReader reader) - : base(header, reader, HeaderType.Sign) + public static SignHeader Create(RarHeader header, RarCrcBinaryReader reader) { - if (IsRar5) + var c = CreateChild(header, reader, HeaderType.Sign); + if (c.IsRar5) { throw new InvalidFormatException("unexpected rar5 record"); } + return c; } protected override void ReadFinish(MarkingBinaryReader reader) diff --git a/src/SharpCompress/Common/Rar/ICryptKey.cs b/src/SharpCompress/Common/Rar/ICryptKey.cs new file mode 100644 index 00000000..94f068f5 --- /dev/null +++ b/src/SharpCompress/Common/Rar/ICryptKey.cs @@ -0,0 +1,8 @@ +using System.Security.Cryptography; + +namespace SharpCompress.Common.Rar; + +internal interface ICryptKey +{ + ICryptoTransform Transformer(byte[] salt); +} diff --git a/src/SharpCompress/Common/Rar/Rar5CryptoInfo.cs b/src/SharpCompress/Common/Rar/Rar5CryptoInfo.cs new file mode 100644 index 00000000..2456e18a --- /dev/null +++ b/src/SharpCompress/Common/Rar/Rar5CryptoInfo.cs @@ -0,0 +1,140 @@ +using System; +using System.Security.Cryptography; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Rar.Headers; +using SharpCompress.IO; + +namespace SharpCompress.Common.Rar; + +internal class Rar5CryptoInfo +{ + private Rar5CryptoInfo() { } + + public static Rar5CryptoInfo Create(MarkingBinaryReader reader, bool readInitV) + { + var cryptoInfo = new Rar5CryptoInfo(); + var cryptVersion = reader.ReadRarVIntUInt32(); + if (cryptVersion > EncryptionConstV5.VERSION) + { + throw new CryptographicException($"Unsupported crypto version of {cryptVersion}"); + } + var encryptionFlags = reader.ReadRarVIntUInt32(); + cryptoInfo.UsePswCheck = FlagUtility.HasFlag( + encryptionFlags, + EncryptionFlagsV5.CHFL_CRYPT_PSWCHECK + ); + cryptoInfo.LG2Count = reader.ReadRarVIntByte(1); + + if (cryptoInfo.LG2Count > EncryptionConstV5.CRYPT5_KDF_LG2_COUNT_MAX) + { + throw new CryptographicException($"Unsupported LG2 count of {cryptoInfo.LG2Count}."); + } + + cryptoInfo.Salt = reader.ReadBytes(EncryptionConstV5.SIZE_SALT50); + + if (readInitV) // File header needs to read IV here + { + cryptoInfo.ReadInitV(reader); + } + + if (cryptoInfo.UsePswCheck) + { + cryptoInfo.PswCheck = reader.ReadBytes(EncryptionConstV5.SIZE_PSWCHECK); + var _pswCheckCsm = reader.ReadBytes(EncryptionConstV5.SIZE_PSWCHECK_CSUM); + +#if LEGACY_DOTNET + var sha = SHA256.Create(); + cryptoInfo.UsePswCheck = sha.ComputeHash(cryptoInfo.PswCheck) + .AsSpan() + .StartsWith(_pswCheckCsm.AsSpan()); +#else + cryptoInfo.UsePswCheck = SHA256 + .HashData(cryptoInfo.PswCheck) + .AsSpan() + .StartsWith(_pswCheckCsm.AsSpan()); +#endif + } + return cryptoInfo; + } + + public static async ValueTask CreateAsync( + AsyncMarkingBinaryReader reader, + bool readInitV + ) + { + var cryptoInfo = new Rar5CryptoInfo(); + var cryptVersion = await reader + .ReadRarVIntUInt32Async(cancellationToken: CancellationToken.None) + .ConfigureAwait(false); + if (cryptVersion > EncryptionConstV5.VERSION) + { + throw new CryptographicException($"Unsupported crypto version of {cryptVersion}"); + } + var encryptionFlags = await reader + .ReadRarVIntUInt32Async(cancellationToken: CancellationToken.None) + .ConfigureAwait(false); + cryptoInfo.UsePswCheck = FlagUtility.HasFlag( + encryptionFlags, + EncryptionFlagsV5.CHFL_CRYPT_PSWCHECK + ); + cryptoInfo.LG2Count = (int) + await reader + .ReadRarVIntUInt32Async(cancellationToken: CancellationToken.None) + .ConfigureAwait(false); + if (cryptoInfo.LG2Count > EncryptionConstV5.CRYPT5_KDF_LG2_COUNT_MAX) + { + throw new CryptographicException($"Unsupported LG2 count of {cryptoInfo.LG2Count}."); + } + + cryptoInfo.Salt = await reader + .ReadBytesAsync(EncryptionConstV5.SIZE_SALT50, CancellationToken.None) + .ConfigureAwait(false); + + if (readInitV) + { + await cryptoInfo.ReadInitVAsync(reader).ConfigureAwait(false); + } + + if (cryptoInfo.UsePswCheck) + { + cryptoInfo.PswCheck = await reader + .ReadBytesAsync(EncryptionConstV5.SIZE_PSWCHECK, CancellationToken.None) + .ConfigureAwait(false); + var _pswCheckCsm = await reader + .ReadBytesAsync(EncryptionConstV5.SIZE_PSWCHECK_CSUM, CancellationToken.None) + .ConfigureAwait(false); + +#if LEGACY_DOTNET + var sha = SHA256.Create(); + cryptoInfo.UsePswCheck = sha.ComputeHash(cryptoInfo.PswCheck) + .AsSpan() + .StartsWith(_pswCheckCsm.AsSpan()); +#else + cryptoInfo.UsePswCheck = SHA256 + .HashData(cryptoInfo.PswCheck) + .AsSpan() + .StartsWith(_pswCheckCsm.AsSpan()); +#endif + } + return cryptoInfo; + } + + public void ReadInitV(MarkingBinaryReader reader) => + InitV = reader.ReadBytes(EncryptionConstV5.SIZE_INITV); + + public async ValueTask ReadInitVAsync(AsyncMarkingBinaryReader reader) => + InitV = await reader + .ReadBytesAsync(EncryptionConstV5.SIZE_INITV, CancellationToken.None) + .ConfigureAwait(false); + + public bool UsePswCheck = false; + + public int LG2Count = 0; + + public byte[] InitV = []; + + public byte[] Salt = []; + + public byte[] PswCheck = []; +} diff --git a/src/SharpCompress/Common/Rar/RarCryptoBinaryReader.cs b/src/SharpCompress/Common/Rar/RarCryptoBinaryReader.cs index 0be6e74e..319cf4c5 100644 --- a/src/SharpCompress/Common/Rar/RarCryptoBinaryReader.cs +++ b/src/SharpCompress/Common/Rar/RarCryptoBinaryReader.cs @@ -1,27 +1,33 @@ -using System.Collections.Generic; +using System.Collections.Generic; using System.IO; +using SharpCompress.Common.Rar.Headers; +using SharpCompress.Crypto; namespace SharpCompress.Common.Rar; internal sealed class RarCryptoBinaryReader : RarCrcBinaryReader { - private RarRijndael _rijndael; - private byte[] _salt; - private readonly string _password; - private readonly Queue _data = new Queue(); + private BlockTransformer _rijndael = default!; + private readonly Queue _data = new(); private long _readCount; - public RarCryptoBinaryReader(Stream stream, string password) - : base(stream) + private RarCryptoBinaryReader(Stream stream) + : base(stream) { } + + public static RarCryptoBinaryReader Create( + Stream stream, + ICryptKey cryptKey, + byte[]? salt = null + ) { - _password = password; - - // coderb: not sure why this was being done at this logical point - //SkipQueue(); - var salt = ReadBytes(8); - - _salt = salt; - _rijndael = RarRijndael.InitializeFrom(_password, salt); + var binary = new RarCryptoBinaryReader(stream); + if (salt == null) + { + salt = binary.ReadBytesBase(EncryptionConstV5.SIZE_SALT30); + binary._readCount += EncryptionConstV5.SIZE_SALT30; + } + binary._rijndael = new BlockTransformer(cryptKey.Transformer(salt)); + return binary; } // track read count ourselves rather than using the underlying stream since we buffer @@ -36,29 +42,11 @@ internal sealed class RarCryptoBinaryReader : RarCrcBinaryReader public override void Mark() => _readCount = 0; - private bool UseEncryption => _salt != null; + public override byte ReadByte() => ReadAndDecryptBytes(1)[0]; - public override byte ReadByte() - { - if (UseEncryption) - { - return ReadAndDecryptBytes(1)[0]; - } + public override byte[] ReadBytes(int count) => ReadAndDecryptBytes(count); - _readCount++; - return base.ReadByte(); - } - - public override byte[] ReadBytes(int count) - { - if (UseEncryption) - { - return ReadAndDecryptBytes(count); - } - - _readCount += count; - return base.ReadBytes(count); - } + private byte[] ReadBytesBase(int count) => base.ReadBytes(count); private byte[] ReadAndDecryptBytes(int count) { diff --git a/src/SharpCompress/Common/Rar/RarCryptoWrapper.cs b/src/SharpCompress/Common/Rar/RarCryptoWrapper.cs index a1cee046..6e1ed222 100644 --- a/src/SharpCompress/Common/Rar/RarCryptoWrapper.cs +++ b/src/SharpCompress/Common/Rar/RarCryptoWrapper.cs @@ -1,37 +1,33 @@ using System; using System.Collections.Generic; using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Crypto; namespace SharpCompress.Common.Rar; internal sealed class RarCryptoWrapper : Stream { private readonly Stream _actualStream; - private readonly byte[] _salt; - private RarRijndael _rijndael; - private readonly Queue _data = new Queue(); + private BlockTransformer _rijndael; + private readonly Queue _data = new(); - public RarCryptoWrapper(Stream actualStream, string password, byte[] salt) + public RarCryptoWrapper(Stream actualStream, byte[] salt, ICryptKey key) { _actualStream = actualStream; - _salt = salt; - _rijndael = RarRijndael.InitializeFrom(password ?? "", salt); + _rijndael = new BlockTransformer(key.Transformer(salt)); } - public override void Flush() => throw new NotSupportedException(); + public override void Flush() { } public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); public override void SetLength(long value) => throw new NotSupportedException(); - public override int Read(byte[] buffer, int offset, int count) - { - if (_salt is null) - { - return _actualStream.Read(buffer, offset, count); - } - return ReadAndDecrypt(buffer, offset, count); - } + public override int Read(byte[] buffer, int offset, int count) => + ReadAndDecrypt(buffer, offset, count); public int ReadAndDecrypt(byte[] buffer, int offset, int count) { @@ -41,7 +37,7 @@ internal sealed class RarCryptoWrapper : Stream if (sizeToRead > 0) { var alignedSize = sizeToRead + ((~sizeToRead + 1) & 0xf); - Span cipherText = stackalloc byte[RarRijndael.CRYPTO_BLOCK_SIZE]; + Span cipherText = stackalloc byte[16]; for (var i = 0; i < alignedSize / 16; i++) { //long ax = System.currentTimeMillis(); @@ -62,6 +58,83 @@ internal sealed class RarCryptoWrapper : Stream return count; } + public override Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) => ReadAndDecryptAsync(buffer, offset, count, cancellationToken).AsTask(); + + private async ValueTask ReadAndDecryptAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + cancellationToken.ThrowIfCancellationRequested(); + + var queueSize = _data.Count; + var sizeToRead = count - queueSize; + + if (sizeToRead > 0) + { + var alignedSize = sizeToRead + ((~sizeToRead + 1) & 0xf); + byte[] cipherText = new byte[16]; + + try + { + for (var i = 0; i < alignedSize / 16; i++) + { + await _actualStream + .ReadExactAsync(cipherText, 0, 16, cancellationToken) + .ConfigureAwait(false); + + var readBytes = _rijndael.ProcessBlock(cipherText); + foreach (var readByte in readBytes) + { + _data.Enqueue(readByte); + } + } + } + catch (EndOfStreamException e) + { + throw new InvalidFormatException("Unexpected end of encrypted stream", e); + } + } + + var bytesToReturn = Math.Min(count, _data.Count); + for (var i = 0; i < bytesToReturn; i++) + { + buffer[offset + i] = _data.Dequeue(); + } + + return bytesToReturn; + } + +#if NETCOREAPP2_1_OR_GREATER || NETSTANDARD2_1_OR_GREATER + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + + var array = System.Buffers.ArrayPool.Shared.Rent(buffer.Length); + try + { + var bytesRead = await ReadAndDecryptAsync(array, 0, buffer.Length, cancellationToken) + .ConfigureAwait(false); + new ReadOnlySpan(array, 0, bytesRead).CopyTo(buffer.Span); + return bytesRead; + } + finally + { + System.Buffers.ArrayPool.Shared.Return(array); + } + } +#endif + public override void Write(byte[] buffer, int offset, int count) => throw new NotSupportedException(); @@ -77,11 +150,11 @@ internal sealed class RarCryptoWrapper : Stream protected override void Dispose(bool disposing) { - if (_rijndael != null) + if (disposing) { _rijndael.Dispose(); - _rijndael = null!; } + base.Dispose(disposing); } } diff --git a/src/SharpCompress/Common/Rar/RarEntry.cs b/src/SharpCompress/Common/Rar/RarEntry.cs index 7f280e4a..506b69e6 100644 --- a/src/SharpCompress/Common/Rar/RarEntry.cs +++ b/src/SharpCompress/Common/Rar/RarEntry.cs @@ -1,4 +1,5 @@ using System; +using SharpCompress.Common.Options; using SharpCompress.Common.Rar.Headers; namespace SharpCompress.Common.Rar; @@ -7,6 +8,9 @@ public abstract class RarEntry : Entry { internal abstract FileHeader FileHeader { get; } + protected RarEntry(IReaderOptions readerOptions) + : base(readerOptions) { } + /// /// As the V2017 port isn't complete, add this check to use the legacy Rar code. /// @@ -20,12 +24,12 @@ public abstract class RarEntry : Entry /// /// The File's 32 bit CRC Hash /// - public override long Crc => FileHeader.FileCrc; + public override long Crc => BitConverter.ToUInt32(FileHeader.FileCrc.NotNull(), 0); /// /// The path of the file internal to the Rar Archive. /// - public override string Key => FileHeader.FileName; + public override string? Key => FileHeader.FileName; public override string? LinkTarget => null; @@ -55,14 +59,24 @@ public abstract class RarEntry : Entry public override bool IsEncrypted => FileHeader.IsEncrypted; /// - /// Entry is password protected and encrypted and cannot be extracted. + /// Entry Windows file attributes + /// + public override int? Attrib => (int)FileHeader.FileAttributes; + + /// + /// Entry is a directory /// public override bool IsDirectory => FileHeader.IsDirectory; public override bool IsSplitAfter => FileHeader.IsSplitAfter; + public bool IsRedir => FileHeader.IsRedir; + + public string? RedirTargetName => FileHeader.RedirTargetName; + public override string ToString() => string.Format( + Constants.DefaultCultureInfo, "Entry Path: {0} Compressed Size: {1} Uncompressed Size: {2} CRC: {3}", Key, CompressedSize, diff --git a/src/SharpCompress/Common/Rar/RarRijndael.cs b/src/SharpCompress/Common/Rar/RarRijndael.cs deleted file mode 100644 index 94592639..00000000 --- a/src/SharpCompress/Common/Rar/RarRijndael.cs +++ /dev/null @@ -1,114 +0,0 @@ -#nullable disable - -using System; -using System.Security.Cryptography; -using System.Text; -using SharpCompress.Crypto; - -namespace SharpCompress.Common.Rar; - -internal class RarRijndael : IDisposable -{ - internal const int CRYPTO_BLOCK_SIZE = 16; - - private readonly string _password; - private readonly byte[] _salt; - private byte[] _aesInitializationVector; - private RijndaelEngine _rijndael; - - private RarRijndael(string password, byte[] salt) - { - _password = password; - _salt = salt; - } - - private void Initialize() - { - _rijndael = new RijndaelEngine(); - _aesInitializationVector = new byte[CRYPTO_BLOCK_SIZE]; - var rawLength = 2 * _password.Length; - var rawPassword = new byte[rawLength + 8]; - var passwordBytes = Encoding.UTF8.GetBytes(_password); - for (var i = 0; i < _password.Length; i++) - { - rawPassword[i * 2] = passwordBytes[i]; - rawPassword[(i * 2) + 1] = 0; - } - for (var i = 0; i < _salt.Length; i++) - { - rawPassword[i + rawLength] = _salt[i]; - } - - const int noOfRounds = (1 << 18); - const int iblock = 3; - byte[] digest; - var data = new byte[(rawPassword.Length + iblock) * noOfRounds]; - - //TODO slow code below, find ways to optimize - for (var i = 0; i < noOfRounds; i++) - { - rawPassword.CopyTo(data, i * (rawPassword.Length + iblock)); - - data[(i * (rawPassword.Length + iblock)) + rawPassword.Length + 0] = (byte)i; - data[(i * (rawPassword.Length + iblock)) + rawPassword.Length + 1] = (byte)(i >> 8); - data[(i * (rawPassword.Length + iblock)) + rawPassword.Length + 2] = (byte)( - i >> CRYPTO_BLOCK_SIZE - ); - - if (i % (noOfRounds / CRYPTO_BLOCK_SIZE) == 0) - { - digest = SHA1.Create() - .ComputeHash(data, 0, (i + 1) * (rawPassword.Length + iblock)); - _aesInitializationVector[i / (noOfRounds / CRYPTO_BLOCK_SIZE)] = digest[19]; - } - } - digest = SHA1.Create().ComputeHash(data); - //slow code ends - - var aesKey = new byte[CRYPTO_BLOCK_SIZE]; - for (var i = 0; i < 4; i++) - { - for (var j = 0; j < 4; j++) - { - aesKey[(i * 4) + j] = (byte)( - ( - ((digest[i * 4] * 0x1000000) & 0xff000000) - | (uint)((digest[(i * 4) + 1] * 0x10000) & 0xff0000) - | (uint)((digest[(i * 4) + 2] * 0x100) & 0xff00) - | (uint)(digest[(i * 4) + 3] & 0xff) - ) >> (j * 8) - ); - } - } - - _rijndael.Init(false, new KeyParameter(aesKey)); - } - - public static RarRijndael InitializeFrom(string password, byte[] salt) - { - var rijndael = new RarRijndael(password, salt); - rijndael.Initialize(); - return rijndael; - } - - public byte[] ProcessBlock(ReadOnlySpan cipherText) - { - Span plainText = stackalloc byte[CRYPTO_BLOCK_SIZE]; // 16 bytes - var decryptedBytes = new byte[CRYPTO_BLOCK_SIZE]; - _rijndael.ProcessBlock(cipherText, plainText); - - for (var j = 0; j < CRYPTO_BLOCK_SIZE; j++) - { - decryptedBytes[j] = (byte)(plainText[j] ^ _aesInitializationVector[j % 16]); //32:114, 33:101 - } - - for (var j = 0; j < _aesInitializationVector.Length; j++) - { - _aesInitializationVector[j] = cipherText[j]; - } - - return decryptedBytes; - } - - public void Dispose() { } -} diff --git a/src/SharpCompress/Common/Rar/RarVolume.cs b/src/SharpCompress/Common/Rar/RarVolume.cs index 7be719ec..1b7de043 100644 --- a/src/SharpCompress/Common/Rar/RarVolume.cs +++ b/src/SharpCompress/Common/Rar/RarVolume.cs @@ -2,6 +2,10 @@ using System; using System.Collections.Generic; using System.IO; using System.Linq; +using System.Runtime.CompilerServices; +using System.Text; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common.Rar.Headers; using SharpCompress.IO; using SharpCompress.Readers; @@ -14,20 +18,19 @@ namespace SharpCompress.Common.Rar; public abstract class RarVolume : Volume { private readonly RarHeaderFactory _headerFactory; - internal int _maxCompressionAlgorithm; + private int _maxCompressionAlgorithm; - internal RarVolume(StreamingMode mode, Stream stream, ReaderOptions options, int index = 0) + internal RarVolume(StreamingMode mode, Stream stream, ReaderOptions options, int index) : base(stream, options, index) => _headerFactory = new RarHeaderFactory(mode, options); -#nullable disable - internal ArchiveHeader ArchiveHeader { get; private set; } + private ArchiveHeader? ArchiveHeader { get; set; } -#nullable enable - - internal StreamingMode Mode => _headerFactory.StreamingMode; + private StreamingMode Mode => _headerFactory.StreamingMode; internal abstract IEnumerable ReadFileParts(); + internal abstract IAsyncEnumerable ReadFilePartsAsync(); + internal abstract RarFilePart CreateFilePart(MarkHeader markHeader, FileHeader fileHeader); internal IEnumerable GetVolumeFileParts() @@ -38,19 +41,16 @@ public abstract class RarVolume : Volume switch (header.HeaderType) { case HeaderType.Mark: - { lastMarkHeader = (MarkHeader)header; } break; case HeaderType.Archive: - { ArchiveHeader = (ArchiveHeader)header; } break; case HeaderType.File: - { var fh = (FileHeader)header; if (_maxCompressionAlgorithm < fh.CompressionAlgorithm) @@ -62,19 +62,65 @@ public abstract class RarVolume : Volume } break; case HeaderType.Service: - { var fh = (FileHeader)header; if (fh.FileName == "CMT") { - var part = CreateFilePart(lastMarkHeader!, fh); var buffer = new byte[fh.CompressedSize]; - part.GetCompressedStream().Read(buffer, 0, buffer.Length); - Comment = System.Text.Encoding.UTF8.GetString( - buffer, - 0, - buffer.Length - 1 - ); + fh.PackedStream.NotNull().ReadFully(buffer); + Comment = Encoding.UTF8.GetString(buffer, 0, buffer.Length - 1); + } + } + break; + } + } + } + + internal async IAsyncEnumerable GetVolumeFilePartsAsync( + [EnumeratorCancellation] CancellationToken cancellationToken = default + ) + { + MarkHeader? lastMarkHeader = null; + await foreach ( + var header in _headerFactory + .ReadHeadersAsync(Stream) + .WithCancellation(cancellationToken) + ) + { + switch (header.HeaderType) + { + case HeaderType.Mark: + { + lastMarkHeader = (MarkHeader)header; + } + break; + case HeaderType.Archive: + { + ArchiveHeader = (ArchiveHeader)header; + } + break; + case HeaderType.File: + { + var fh = (FileHeader)header; + if (_maxCompressionAlgorithm < fh.CompressionAlgorithm) + { + _maxCompressionAlgorithm = fh.CompressionAlgorithm; + } + + yield return CreateFilePart(lastMarkHeader!, fh); + } + break; + case HeaderType.Service: + { + var fh = (FileHeader)header; + if (fh.FileName == "CMT") + { + var buffer = new byte[fh.CompressedSize]; + await fh + .PackedStream.NotNull() + .ReadFullyAsync(buffer, cancellationToken) + .ConfigureAwait(false); + Comment = Encoding.UTF8.GetString(buffer, 0, buffer.Length - 1); } } break; @@ -88,13 +134,13 @@ public abstract class RarVolume : Volume { if (Mode == StreamingMode.Streaming) { - throw new InvalidOperationException( + throw new ArchiveOperationException( "ArchiveHeader should never been null in a streaming read." ); } // we only want to load the archive header to avoid overhead but have to do the nasty thing and reset the stream - GetVolumeFileParts().First(); + _ = GetVolumeFileParts().First(); Stream.Position = 0; } } @@ -108,7 +154,7 @@ public abstract class RarVolume : Volume get { EnsureArchiveHeaderLoaded(); - return ArchiveHeader.IsFirstVolume; + return ArchiveHeader?.IsFirstVolume ?? false; } } @@ -120,7 +166,7 @@ public abstract class RarVolume : Volume get { EnsureArchiveHeaderLoaded(); - return ArchiveHeader.IsVolume; + return ArchiveHeader?.IsVolume ?? false; } } @@ -133,10 +179,16 @@ public abstract class RarVolume : Volume get { EnsureArchiveHeaderLoaded(); - return ArchiveHeader.IsSolid; + return ArchiveHeader?.IsSolid ?? false; } } + public async ValueTask IsSolidArchiveAsync(CancellationToken cancellationToken = default) + { + await EnsureArchiveHeaderLoadedAsync(cancellationToken).ConfigureAwait(false); + return ArchiveHeader?.IsSolid ?? false; + } + public int MinVersion { get @@ -185,5 +237,70 @@ public abstract class RarVolume : Volume } } + private async ValueTask EnsureArchiveHeaderLoadedAsync(CancellationToken cancellationToken) + { + if (ArchiveHeader is null) + { + if (Mode == StreamingMode.Streaming) + { + throw new ArchiveOperationException( + "ArchiveHeader should never been null in a streaming read." + ); + } + + // we only want to load the archive header to avoid overhead but have to do the nasty thing and reset the stream +#pragma warning disable CA2016 // Forward token if available; polyfill FirstAsync has no token overload + await GetVolumeFilePartsAsync(cancellationToken).FirstAsync().ConfigureAwait(false); +#pragma warning restore CA2016 + Stream.Position = 0; + } + } + + public virtual async ValueTask MinVersionAsync( + CancellationToken cancellationToken = default + ) + { + await EnsureArchiveHeaderLoadedAsync(cancellationToken).ConfigureAwait(false); + if (_maxCompressionAlgorithm >= 50) + { + return 5; //5-6 + } + else if (_maxCompressionAlgorithm >= 29) + { + return 3; //3-4 + } + else if (_maxCompressionAlgorithm >= 20) + { + return 2; //2 + } + else + { + return 1; + } + } + + public virtual async ValueTask MaxVersionAsync( + CancellationToken cancellationToken = default + ) + { + await EnsureArchiveHeaderLoadedAsync(cancellationToken).ConfigureAwait(false); + if (_maxCompressionAlgorithm >= 50) + { + return 6; //5-6 + } + else if (_maxCompressionAlgorithm >= 29) + { + return 4; //3-4 + } + else if (_maxCompressionAlgorithm >= 20) + { + return 2; //2 + } + else + { + return 1; + } + } + public string? Comment { get; internal set; } } diff --git a/src/SharpCompress/Common/ReaderExtractionEventArgs.cs b/src/SharpCompress/Common/ReaderExtractionEventArgs.cs deleted file mode 100644 index 7c4363e2..00000000 --- a/src/SharpCompress/Common/ReaderExtractionEventArgs.cs +++ /dev/null @@ -1,17 +0,0 @@ -using System; -using SharpCompress.Readers; - -namespace SharpCompress.Common; - -public sealed class ReaderExtractionEventArgs : EventArgs -{ - internal ReaderExtractionEventArgs(T entry, ReaderProgress? readerProgress = null) - { - Item = entry; - ReaderProgress = readerProgress; - } - - public T Item { get; } - - public ReaderProgress? ReaderProgress { get; } -} diff --git a/src/SharpCompress/Common/SevenZip/ArchiveDatabase.Async.cs b/src/SharpCompress/Common/SevenZip/ArchiveDatabase.Async.cs new file mode 100644 index 00000000..ae2ff232 --- /dev/null +++ b/src/SharpCompress/Common/SevenZip/ArchiveDatabase.Async.cs @@ -0,0 +1,38 @@ +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Compressors.LZMA; +using SharpCompress.Compressors.LZMA.Utilities; + +namespace SharpCompress.Common.SevenZip; + +internal sealed partial class ArchiveDatabase +{ + internal async ValueTask GetFolderStreamAsync( + Stream stream, + CFolder folder, + IPasswordProvider pw, + CancellationToken cancellationToken + ) + { + var packStreamIndex = folder._firstPackStreamId; + var folderStartPackPos = GetFolderStreamPos(folder, 0); + var count = folder._packStreams.Count; + var packSizes = new long[count]; + for (var j = 0; j < count; j++) + { + packSizes[j] = _packSizes[packStreamIndex + j]; + } + + return await DecoderStreamHelper + .CreateDecoderStreamAsync( + stream, + folderStartPackPos, + packSizes, + folder, + pw, + cancellationToken + ) + .ConfigureAwait(false); + } +} diff --git a/src/SharpCompress/Common/SevenZip/ArchiveDatabase.cs b/src/SharpCompress/Common/SevenZip/ArchiveDatabase.cs index 5e42494a..dbad5353 100644 --- a/src/SharpCompress/Common/SevenZip/ArchiveDatabase.cs +++ b/src/SharpCompress/Common/SevenZip/ArchiveDatabase.cs @@ -1,29 +1,27 @@ -#nullable disable - using System; using System.Collections.Generic; using System.IO; using SharpCompress.Compressors.LZMA; -using SharpCompress.Compressors.LZMA.Utilites; +using SharpCompress.Compressors.LZMA.Utilities; namespace SharpCompress.Common.SevenZip; -internal class ArchiveDatabase +internal partial class ArchiveDatabase { internal byte _majorVersion; internal byte _minorVersion; internal long _startPositionAfterHeader; internal long _dataStartPosition; - internal List _packSizes = new List(); - internal List _packCrCs = new List(); - internal List _folders = new List(); - internal List _numUnpackStreamsVector; - internal List _files = new List(); + internal List _packSizes = new(); + internal List _packCrCs = new(); + internal List _folders = new(); + internal List _numUnpackStreamsVector = null!; + internal List _files = new(); - internal List _packStreamStartPositions = new List(); - internal List _folderStartFileIndex = new List(); - internal List _fileIndexToFolderIndexMap = new List(); + internal List _packStreamStartPositions = new(); + internal List _folderStartFileIndex = new(); + internal List _fileIndexToFolderIndexMap = new(); internal IPasswordProvider PasswordProvider { get; } @@ -89,7 +87,7 @@ internal class ArchiveDatabase { if (folderIndex >= _folders.Count) { - throw new InvalidOperationException(); + throw new ArchiveOperationException(); } _folderStartFileIndex.Add(i); // check it diff --git a/src/SharpCompress/Common/SevenZip/ArchiveReader.Async.cs b/src/SharpCompress/Common/SevenZip/ArchiveReader.Async.cs new file mode 100644 index 00000000..52964077 --- /dev/null +++ b/src/SharpCompress/Common/SevenZip/ArchiveReader.Async.cs @@ -0,0 +1,469 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Compressors.Deflate64; +using SharpCompress.Compressors.LZMA; +using SharpCompress.Compressors.LZMA.Utilities; +using SharpCompress.IO; +using BlockType = SharpCompress.Compressors.LZMA.Utilities.BlockType; + +namespace SharpCompress.Common.SevenZip; + +internal sealed partial class ArchiveReader +{ + public async ValueTask OpenAsync( + Stream stream, + bool lookForHeader, + CancellationToken cancellationToken = default + ) + { + Close(); + + _streamOrigin = stream.Position; + _streamEnding = stream.Length; + + var canScan = lookForHeader ? 0x80000 - 20 : 0; + while (true) + { + // TODO: Check Signature! + _header = new byte[0x20]; + await stream.ReadExactAsync(_header, 0, 0x20, cancellationToken).ConfigureAwait(false); + + if ( + !lookForHeader + || _header + .AsSpan(0, length: 6) + .SequenceEqual([0x37, 0x7A, 0xBC, 0xAF, 0x27, 0x1C]) + ) + { + break; + } + + if (canScan == 0) + { + throw new InvalidFormatException("Unable to find 7z signature"); + } + + canScan--; + stream.Position = ++_streamOrigin; + } + + _stream = stream; + } + + public async ValueTask ReadDatabaseAsync( + IPasswordProvider pass, + CancellationToken cancellationToken = default + ) + { + var db = new ArchiveDatabase(pass); + db.Clear(); + + db._majorVersion = _header[6]; + db._minorVersion = _header[7]; + + if (db._majorVersion != 0) + { + throw new ArchiveOperationException(); + } + + var crcFromArchive = DataReader.Get32(_header, 8); + var nextHeaderOffset = (long)DataReader.Get64(_header, 0xC); + var nextHeaderSize = (long)DataReader.Get64(_header, 0x14); + var nextHeaderCrc = DataReader.Get32(_header, 0x1C); + + var crc = Crc.INIT_CRC; + crc = Crc.Update(crc, nextHeaderOffset); + crc = Crc.Update(crc, nextHeaderSize); + crc = Crc.Update(crc, nextHeaderCrc); + crc = Crc.Finish(crc); + + if (crc != crcFromArchive) + { + throw new ArchiveOperationException(); + } + + db._startPositionAfterHeader = _streamOrigin + 0x20; + + // empty header is ok + if (nextHeaderSize == 0) + { + db.Fill(); + return db; + } + + if (nextHeaderOffset < 0 || nextHeaderSize < 0 || nextHeaderSize > int.MaxValue) + { + throw new ArchiveOperationException(); + } + + if (nextHeaderOffset > _streamEnding - db._startPositionAfterHeader) + { + throw new ArchiveOperationException("nextHeaderOffset is invalid"); + } + + _stream.Seek(nextHeaderOffset, SeekOrigin.Current); + + var header = new byte[nextHeaderSize]; + await _stream + .ReadExactAsync(header, 0, header.Length, cancellationToken) + .ConfigureAwait(false); + + if (Crc.Finish(Crc.Update(Crc.INIT_CRC, header, 0, header.Length)) != nextHeaderCrc) + { + throw new ArchiveOperationException(); + } + + using (var streamSwitch = new CStreamSwitch()) + { + streamSwitch.Set(this, header); + + var type = ReadId(); + if (type != BlockType.Header) + { + if (type != BlockType.EncodedHeader) + { + throw new ArchiveOperationException(); + } + + var dataVector = await ReadAndDecodePackedStreamsAsync( + db._startPositionAfterHeader, + db.PasswordProvider, + cancellationToken + ) + .ConfigureAwait(false); + + // compressed header without content is odd but ok + if (dataVector.Count == 0) + { + db.Fill(); + return db; + } + + if (dataVector.Count != 1) + { + throw new ArchiveOperationException(); + } + + streamSwitch.Set(this, dataVector[0]); + + if (ReadId() != BlockType.Header) + { + throw new ArchiveOperationException(); + } + } + + await ReadHeaderAsync(db, db.PasswordProvider, cancellationToken).ConfigureAwait(false); + } + db.Fill(); + return db; + } + + private async ValueTask> ReadAndDecodePackedStreamsAsync( + long baseOffset, + IPasswordProvider pass, + CancellationToken cancellationToken + ) + { + try + { + ReadStreamsInfo( + null, + out var dataStartPos, + out var packSizes, + out var packCrCs, + out var folders, + out var numUnpackStreamsInFolders, + out var unpackSizes, + out var digests + ); + + dataStartPos += baseOffset; + + var dataVector = new List(folders.Count); + var packIndex = 0; + foreach (var folder in folders) + { + var oldDataStartPos = dataStartPos; + var myPackSizes = new long[folder._packStreams.Count]; + for (var i = 0; i < myPackSizes.Length; i++) + { + var packSize = packSizes[packIndex + i]; + myPackSizes[i] = packSize; + dataStartPos += packSize; + } + + var outStream = await DecoderStreamHelper + .CreateDecoderStreamAsync( + _stream, + oldDataStartPos, + myPackSizes, + folder, + pass, + cancellationToken + ) + .ConfigureAwait(false); + + var unpackSize = checked((int)folder.GetUnpackSize()); + var data = new byte[unpackSize]; + await outStream + .ReadExactAsync(data, 0, data.Length, cancellationToken) + .ConfigureAwait(false); + if (outStream.ReadByte() >= 0) + { + throw new InvalidFormatException("Decoded stream is longer than expected."); + } + dataVector.Add(data); + + if (folder.UnpackCrcDefined) + { + if ( + Crc.Finish(Crc.Update(Crc.INIT_CRC, data, 0, unpackSize)) + != folder._unpackCrc + ) + { + throw new InvalidFormatException( + "Decoded stream does not match expected CRC." + ); + } + } + } + return dataVector; + } + finally { } + } + + private async ValueTask ReadHeaderAsync( + ArchiveDatabase db, + IPasswordProvider getTextPassword, + CancellationToken cancellationToken + ) + { + try + { + var type = ReadId(); + + if (type == BlockType.ArchiveProperties) + { + ReadArchiveProperties(); + type = ReadId(); + } + + List? dataVector = null; + if (type == BlockType.AdditionalStreamsInfo) + { + dataVector = await ReadAndDecodePackedStreamsAsync( + db._startPositionAfterHeader, + getTextPassword, + cancellationToken + ) + .ConfigureAwait(false); + type = ReadId(); + } + + List unpackSizes; + List digests; + + if (type == BlockType.MainStreamsInfo) + { + ReadStreamsInfo( + dataVector, + out db._dataStartPosition, + out db._packSizes, + out db._packCrCs, + out db._folders, + out db._numUnpackStreamsVector, + out unpackSizes, + out digests + ); + + db._dataStartPosition += db._startPositionAfterHeader; + type = ReadId(); + } + else + { + unpackSizes = new List(db._folders.Count); + digests = new List(db._folders.Count); + db._numUnpackStreamsVector = new List(db._folders.Count); + for (var i = 0; i < db._folders.Count; i++) + { + var folder = db._folders[i]; + unpackSizes.Add(folder.GetUnpackSize()); + digests.Add(folder._unpackCrc); + db._numUnpackStreamsVector.Add(1); + } + } + + db._files.Clear(); + + if (type == BlockType.End) + { + return; + } + + if (type != BlockType.FilesInfo) + { + throw new ArchiveOperationException(); + } + + var numFiles = ReadNum(); + db._files = new List(numFiles); + for (var i = 0; i < numFiles; i++) + { + db._files.Add(new CFileItem()); + } + + var emptyStreamVector = new BitVector(numFiles); + BitVector emptyFileVector = null!; + BitVector antiFileVector = null!; + var numEmptyStreams = 0; + + for (; ; ) + { + type = ReadId(); + if (type == BlockType.End) + { + break; + } + + var size = checked((long)ReadNumber()); + var oldPos = _currentReader.Offset; + switch (type) + { + case BlockType.Name: + using (var streamSwitch = new CStreamSwitch()) + { + streamSwitch.Set(this, dataVector ?? []); + for (var i = 0; i < db._files.Count; i++) + { + db._files[i].Name = _currentReader.ReadString(); + } + } + break; + case BlockType.WinAttributes: + ReadAttributeVector( + dataVector, + numFiles, + delegate(int i, uint? attr) + { + db._files[i].ExtendedAttrib = attr; + + if (attr.HasValue && (attr.Value >> 16) != 0) + { + attr = attr.Value & 0x7FFFu; + } + + db._files[i].Attrib = attr; + } + ); + break; + case BlockType.EmptyStream: + emptyStreamVector = ReadBitVector(numFiles); + for (var i = 0; i < emptyStreamVector.Length; i++) + { + if (emptyStreamVector[i]) + { + numEmptyStreams++; + } + else { } + } + + emptyFileVector = new BitVector(numEmptyStreams); + antiFileVector = new BitVector(numEmptyStreams); + break; + case BlockType.EmptyFile: + emptyFileVector = ReadBitVector(numEmptyStreams); + break; + case BlockType.Anti: + antiFileVector = ReadBitVector(numEmptyStreams); + break; + case BlockType.StartPos: + ReadNumberVector( + dataVector, + numFiles, + delegate(int i, long? startPos) + { + db._files[i].StartPos = startPos; + } + ); + break; + case BlockType.CTime: + ReadDateTimeVector( + dataVector, + numFiles, + delegate(int i, DateTime? time) + { + db._files[i].CTime = time; + } + ); + break; + case BlockType.ATime: + ReadDateTimeVector( + dataVector, + numFiles, + delegate(int i, DateTime? time) + { + db._files[i].ATime = time; + } + ); + break; + case BlockType.MTime: + ReadDateTimeVector( + dataVector, + numFiles, + delegate(int i, DateTime? time) + { + db._files[i].MTime = time; + } + ); + break; + case BlockType.Dummy: + for (long j = 0; j < size; j++) + { + if (ReadByte() != 0) + { + throw new ArchiveOperationException(); + } + } + break; + default: + SkipData(size); + break; + } + + var checkRecordsSize = (db._majorVersion > 0 || db._minorVersion > 2); + if (checkRecordsSize && _currentReader.Offset - oldPos != size) + { + throw new ArchiveOperationException(); + } + } + + var emptyFileIndex = 0; + var sizeIndex = 0; + for (var i = 0; i < numFiles; i++) + { + var file = db._files[i]; + file.HasStream = !emptyStreamVector[i]; + if (file.HasStream) + { + file.IsDir = false; + file.IsAnti = false; + file.Size = unpackSizes[sizeIndex]; + file.Crc = digests[sizeIndex]; + sizeIndex++; + } + else + { + file.IsDir = !emptyFileVector[emptyFileIndex]; + file.IsAnti = antiFileVector[emptyFileIndex]; + emptyFileIndex++; + file.Size = 0; + file.Crc = null; + } + } + } + finally { } + } +} diff --git a/src/SharpCompress/Common/SevenZip/ArchiveReader.cs b/src/SharpCompress/Common/SevenZip/ArchiveReader.cs index 42b5b6b2..b75d123c 100644 --- a/src/SharpCompress/Common/SevenZip/ArchiveReader.cs +++ b/src/SharpCompress/Common/SevenZip/ArchiveReader.cs @@ -1,26 +1,28 @@ -#nullable disable - using System; using System.Collections.Generic; using System.Diagnostics; using System.IO; using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Compressors.Deflate64; using SharpCompress.Compressors.LZMA; -using SharpCompress.Compressors.LZMA.Utilites; +using SharpCompress.Compressors.LZMA.Utilities; using SharpCompress.IO; +using BlockType = SharpCompress.Compressors.LZMA.Utilities.BlockType; namespace SharpCompress.Common.SevenZip; -internal class ArchiveReader +internal partial class ArchiveReader { - internal Stream _stream; - internal Stack _readerStack = new Stack(); - internal DataReader _currentReader; + internal Stream _stream = null!; + internal Stack _readerStack = new(); + internal DataReader _currentReader = null!; internal long _streamOrigin; internal long _streamEnding; - internal byte[] _header; + internal byte[] _header = null!; - private readonly Dictionary _cachedStreams = new Dictionary(); + private readonly Dictionary _cachedStreams = new(); internal void AddByteStream(byte[] buffer, int offset, int length) { @@ -52,9 +54,6 @@ internal class ArchiveReader { return null; } -#if DEBUG - Log.WriteLine("ReadId: {0}", (BlockType)id); -#endif return (BlockType)id; } @@ -73,7 +72,7 @@ internal class ArchiveReader } if (type == BlockType.End) { - throw new InvalidOperationException(); + throw new ArchiveOperationException(); } SkipData(); } @@ -128,12 +127,12 @@ internal class ArchiveReader return ReadBitVector(length); } - private void ReadNumberVector(List dataVector, int numFiles, Action action) + private void ReadNumberVector(List? dataVector, int numFiles, Action action) { var defined = ReadOptionalBitVector(numFiles); using var streamSwitch = new CStreamSwitch(); - streamSwitch.Set(this, dataVector); + streamSwitch.Set(this, dataVector ?? []); for (var i = 0; i < numFiles; i++) { @@ -162,7 +161,7 @@ internal class ArchiveReader } private void ReadDateTimeVector( - List dataVector, + List? dataVector, int numFiles, Action action ) => @@ -173,14 +172,14 @@ internal class ArchiveReader ); private void ReadAttributeVector( - List dataVector, + List? dataVector, int numFiles, Action action ) { var boolVector = ReadOptionalBitVector(numFiles); using var streamSwitch = new CStreamSwitch(); - streamSwitch.Set(this, dataVector); + streamSwitch.Set(this, dataVector ?? []); for (var i = 0; i < numFiles; i++) { if (boolVector[i]) @@ -200,25 +199,14 @@ internal class ArchiveReader private void GetNextFolderItem(CFolder folder) { -#if DEBUG - Log.WriteLine("-- GetNextFolderItem --"); - Log.PushIndent(); -#endif try { var numCoders = ReadNum(); -#if DEBUG - Log.WriteLine("NumCoders: " + numCoders); -#endif folder._coders = new List(numCoders); var numInStreams = 0; var numOutStreams = 0; for (var i = 0; i < numCoders; i++) { -#if DEBUG - Log.WriteLine("-- Coder --"); - Log.PushIndent(); -#endif try { var coder = new CCoderInfo(); @@ -228,18 +216,6 @@ internal class ArchiveReader var idSize = (mainByte & 0xF); var longId = new byte[idSize]; ReadBytes(longId, 0, idSize); -#if DEBUG - Log.WriteLine( - "MethodId: " - + string.Join( - "", - Enumerable - .Range(0, idSize) - .Select(x => longId[x].ToString("x2")) - .ToArray() - ) - ); -#endif if (idSize > 8) { throw new NotSupportedException(); @@ -255,21 +231,9 @@ internal class ArchiveReader { coder._numInStreams = ReadNum(); coder._numOutStreams = ReadNum(); -#if DEBUG - Log.WriteLine( - "Complex Stream (In: " - + coder._numInStreams - + " - Out: " - + coder._numOutStreams - + ")" - ); -#endif } else { -#if DEBUG - Log.WriteLine("Simple Stream (In: 1 - Out: 1)"); -#endif coder._numInStreams = 1; coder._numOutStreams = 1; } @@ -279,15 +243,6 @@ internal class ArchiveReader var propsSize = ReadNum(); coder._props = new byte[propsSize]; ReadBytes(coder._props, 0, propsSize); -#if DEBUG - Log.WriteLine( - "Settings: " - + string.Join( - "", - coder._props.Select(bt => bt.ToString("x2")).ToArray() - ) - ); -#endif } if ((mainByte & 0x80) != 0) @@ -298,33 +253,18 @@ internal class ArchiveReader numInStreams += coder._numInStreams; numOutStreams += coder._numOutStreams; } - finally - { -#if DEBUG - Log.PopIndent(); -#endif - } + finally { } } var numBindPairs = numOutStreams - 1; folder._bindPairs = new List(numBindPairs); -#if DEBUG - Log.WriteLine("BindPairs: " + numBindPairs); - Log.PushIndent(); -#endif for (var i = 0; i < numBindPairs; i++) { var bp = new CBindPair(); bp._inIndex = ReadNum(); bp._outIndex = ReadNum(); folder._bindPairs.Add(bp); -#if DEBUG - Log.WriteLine("#" + i + " - In: " + bp._inIndex + " - Out: " + bp._outIndex); -#endif } -#if DEBUG - Log.PopIndent(); -#endif if (numInStreams < numBindPairs) { @@ -340,9 +280,6 @@ internal class ArchiveReader { if (folder.FindBindPairForInStream(i) < 0) { -#if DEBUG - Log.WriteLine("Single PackStream: #" + i); -#endif folder._packStreams.Add(i); break; } @@ -355,37 +292,18 @@ internal class ArchiveReader } else { -#if DEBUG - Log.WriteLine("Multiple PackStreams ..."); - Log.PushIndent(); -#endif for (var i = 0; i < numPackStreams; i++) { var num = ReadNum(); -#if DEBUG - Log.WriteLine("#" + i + " - " + num); -#endif folder._packStreams.Add(num); } -#if DEBUG - Log.PopIndent(); -#endif } } - finally - { -#if DEBUG - Log.PopIndent(); -#endif - } + finally { } } private List ReadHashDigests(int count) { -#if DEBUG - Log.Write("ReadHashDigests:"); -#endif - var defined = ReadOptionalBitVector(count); var digests = new List(count); for (var i = 0; i < count; i++) @@ -393,23 +311,13 @@ internal class ArchiveReader if (defined[i]) { var crc = ReadUInt32(); -#if DEBUG - Log.Write(" " + crc.ToString("x8")); -#endif digests.Add(crc); } else { -#if DEBUG - Log.Write(" ########"); -#endif digests.Add(null); } } -#if DEBUG - - Log.WriteLine(); -#endif return digests; } @@ -419,40 +327,21 @@ internal class ArchiveReader out List packCrCs ) { -#if DEBUG - Log.WriteLine("-- ReadPackInfo --"); - Log.PushIndent(); -#endif try { - packCrCs = null; + packCrCs = null!; dataOffset = checked((long)ReadNumber()); -#if DEBUG - Log.WriteLine("DataOffset: " + dataOffset); -#endif var numPackStreams = ReadNum(); -#if DEBUG - Log.WriteLine("NumPackStreams: " + numPackStreams); -#endif WaitAttribute(BlockType.Size); packSizes = new List(numPackStreams); -#if DEBUG - Log.Write("Sizes:"); -#endif for (var i = 0; i < numPackStreams; i++) { var size = checked((long)ReadNumber()); -#if DEBUG - Log.Write(" " + size); -#endif packSizes.Add(size); } -#if DEBUG - Log.WriteLine(); -#endif BlockType? type; for (; ; ) @@ -479,31 +368,19 @@ internal class ArchiveReader } } } - finally - { -#if DEBUG - Log.PopIndent(); -#endif - } + finally { } } - private void ReadUnpackInfo(List dataVector, out List folders) + private void ReadUnpackInfo(List? dataVector, out List folders) { -#if DEBUG - Log.WriteLine("-- ReadUnpackInfo --"); - Log.PushIndent(); -#endif try { WaitAttribute(BlockType.Folder); var numFolders = ReadNum(); -#if DEBUG - Log.WriteLine("NumFolders: {0}", numFolders); -#endif using (var streamSwitch = new CStreamSwitch()) { - streamSwitch.Set(this, dataVector); + streamSwitch.Set(this, dataVector ?? []); //folders.Clear(); //folders.Reserve(numFolders); @@ -519,27 +396,15 @@ internal class ArchiveReader } WaitAttribute(BlockType.CodersUnpackSize); -#if DEBUG - Log.WriteLine("UnpackSizes:"); -#endif for (var i = 0; i < numFolders; i++) { var folder = folders[i]; -#if DEBUG - Log.Write(" #" + i + ":"); -#endif var numOutStreams = folder.GetNumOutStreams(); for (var j = 0; j < numOutStreams; j++) { var size = checked((long)ReadNumber()); -#if DEBUG - Log.Write(" " + size); -#endif folder._unpackSizes.Add(size); } -#if DEBUG - Log.WriteLine(); -#endif } for (; ; ) @@ -563,12 +428,7 @@ internal class ArchiveReader SkipData(); } } - finally - { -#if DEBUG - Log.PopIndent(); -#endif - } + finally { } } private void ReadSubStreamsInfo( @@ -578,13 +438,9 @@ internal class ArchiveReader out List digests ) { -#if DEBUG - Log.WriteLine("-- ReadSubStreamsInfo --"); - Log.PushIndent(); -#endif try { - numUnpackStreamsInFolders = null; + numUnpackStreamsInFolders = null!; BlockType? type; for (; ; ) @@ -593,20 +449,11 @@ internal class ArchiveReader if (type == BlockType.NumUnpackStream) { numUnpackStreamsInFolders = new List(folders.Count); -#if DEBUG - Log.Write("NumUnpackStreams:"); -#endif for (var i = 0; i < folders.Count; i++) { var num = ReadNum(); -#if DEBUG - Log.Write(" " + num); -#endif numUnpackStreamsInFolders.Add(num); } -#if DEBUG - Log.WriteLine(); -#endif continue; } if (type is BlockType.Crc or BlockType.Size) @@ -639,26 +486,17 @@ internal class ArchiveReader { continue; } -#if DEBUG - Log.Write("#{0} StreamSizes:", i); -#endif long sum = 0; for (var j = 1; j < numSubstreams; j++) { if (type == BlockType.Size) { var size = checked((long)ReadNumber()); -#if DEBUG - Log.Write(" " + size); -#endif unpackSizes.Add(size); sum += size; } } unpackSizes.Add(folders[i].GetUnpackSize() - sum); -#if DEBUG - Log.WriteLine(" - rest: " + unpackSizes.Last()); -#endif } if (type == BlockType.Size) { @@ -677,7 +515,7 @@ internal class ArchiveReader numDigestsTotal += numSubstreams; } - digests = null; + digests = null!; for (; ; ) { @@ -694,7 +532,7 @@ internal class ArchiveReader var folder = folders[i]; if (numSubstreams == 1 && folder.UnpackCrcDefined) { - digests.Add(folder._unpackCrc.Value); + digests.Add(folder._unpackCrc!.Value); } else { @@ -730,16 +568,11 @@ internal class ArchiveReader type = ReadId(); } } - finally - { -#if DEBUG - Log.PopIndent(); -#endif - } + finally { } } private void ReadStreamsInfo( - List dataVector, + List? dataVector, out long dataOffset, out List packSizes, out List packCrCs, @@ -749,19 +582,15 @@ internal class ArchiveReader out List digests ) { -#if DEBUG - Log.WriteLine("-- ReadStreamsInfo --"); - Log.PushIndent(); -#endif try { dataOffset = long.MinValue; - packSizes = null; - packCrCs = null; - folders = null; - numUnpackStreamsInFolders = null; - unpackSizes = null; - digests = null; + packSizes = null!; + packCrCs = null!; + folders = null!; + numUnpackStreamsInFolders = null!; + unpackSizes = null!; + digests = null!; for (; ; ) { @@ -777,31 +606,22 @@ internal class ArchiveReader break; case BlockType.SubStreamsInfo: ReadSubStreamsInfo( - folders, + folders!, out numUnpackStreamsInFolders, out unpackSizes, out digests ); break; default: - throw new InvalidOperationException(); + throw new InvalidFormatException(); } } } - finally - { -#if DEBUG - Log.PopIndent(); -#endif - } + finally { } } private List ReadAndDecodePackedStreams(long baseOffset, IPasswordProvider pass) { -#if DEBUG - Log.WriteLine("-- ReadAndDecodePackedStreams --"); - Log.PushIndent(); -#endif try { ReadStreamsInfo( @@ -843,7 +663,7 @@ internal class ArchiveReader outStream.ReadExact(data, 0, data.Length); if (outStream.ReadByte() >= 0) { - throw new InvalidOperationException("Decoded stream is longer than expected."); + throw new InvalidFormatException("Decoded stream is longer than expected."); } dataVector.Add(data); @@ -854,7 +674,7 @@ internal class ArchiveReader != folder._unpackCrc ) { - throw new InvalidOperationException( + throw new InvalidFormatException( "Decoded stream does not match expected CRC." ); } @@ -862,20 +682,11 @@ internal class ArchiveReader } return dataVector; } - finally - { -#if DEBUG - Log.PopIndent(); -#endif - } + finally { } } private void ReadHeader(ArchiveDatabase db, IPasswordProvider getTextPassword) { -#if DEBUG - Log.WriteLine("-- ReadHeader --"); - Log.PushIndent(); -#endif try { var type = ReadId(); @@ -886,7 +697,7 @@ internal class ArchiveReader type = ReadId(); } - List dataVector = null; + List? dataVector = null; if (type == BlockType.AdditionalStreamsInfo) { dataVector = ReadAndDecodePackedStreams( @@ -938,13 +749,10 @@ internal class ArchiveReader if (type != BlockType.FilesInfo) { - throw new InvalidOperationException(); + throw new ArchiveOperationException(); } var numFiles = ReadNum(); -#if DEBUG - Log.WriteLine("NumFiles: " + numFiles); -#endif db._files = new List(numFiles); for (var i = 0; i < numFiles; i++) { @@ -952,8 +760,8 @@ internal class ArchiveReader } var emptyStreamVector = new BitVector(numFiles); - BitVector emptyFileVector = null; - BitVector antiFileVector = null; + BitVector emptyFileVector = null!; + BitVector antiFileVector = null!; var numEmptyStreams = 0; for (; ; ) @@ -971,31 +779,24 @@ internal class ArchiveReader case BlockType.Name: using (var streamSwitch = new CStreamSwitch()) { - streamSwitch.Set(this, dataVector); -#if DEBUG - Log.Write("FileNames:"); -#endif + streamSwitch.Set(this, dataVector ?? []); for (var i = 0; i < db._files.Count; i++) { db._files[i].Name = _currentReader.ReadString(); -#if DEBUG - Log.Write(" " + db._files[i].Name); -#endif } -#if DEBUG - Log.WriteLine(); -#endif } break; case BlockType.WinAttributes: -#if DEBUG - Log.Write("WinAttributes:"); -#endif ReadAttributeVector( dataVector, numFiles, delegate(int i, uint? attr) { + // Keep the original attribute value because it could potentially get + // modified in the logic that follows. Some callers of the library may + // find the original value useful. + db._files[i].ExtendedAttrib = attr; + // Some third party implementations established an unofficial extension // of the 7z archive format by placing posix file attributes in the high // bits of the windows file attributes. This makes use of the fact that @@ -1019,155 +820,75 @@ internal class ArchiveReader } db._files[i].Attrib = attr; -#if DEBUG - Log.Write( - " " + (attr.HasValue ? attr.Value.ToString("x8") : "n/a") - ); -#endif } ); -#if DEBUG - Log.WriteLine(); -#endif break; case BlockType.EmptyStream: emptyStreamVector = ReadBitVector(numFiles); -#if DEBUG - - Log.Write("EmptyStream: "); -#endif for (var i = 0; i < emptyStreamVector.Length; i++) { if (emptyStreamVector[i]) { -#if DEBUG - Log.Write("x"); -#endif numEmptyStreams++; } - else - { -#if DEBUG - Log.Write("."); -#endif - } + else { } } -#if DEBUG - Log.WriteLine(); -#endif emptyFileVector = new BitVector(numEmptyStreams); antiFileVector = new BitVector(numEmptyStreams); break; case BlockType.EmptyFile: emptyFileVector = ReadBitVector(numEmptyStreams); -#if DEBUG - Log.Write("EmptyFile: "); - for (var i = 0; i < numEmptyStreams; i++) - { - Log.Write(emptyFileVector[i] ? "x" : "."); - } - Log.WriteLine(); -#endif break; case BlockType.Anti: antiFileVector = ReadBitVector(numEmptyStreams); -#if DEBUG - Log.Write("Anti: "); - for (var i = 0; i < numEmptyStreams; i++) - { - Log.Write(antiFileVector[i] ? "x" : "."); - } - Log.WriteLine(); -#endif break; case BlockType.StartPos: -#if DEBUG - Log.Write("StartPos:"); -#endif ReadNumberVector( dataVector, numFiles, delegate(int i, long? startPos) { db._files[i].StartPos = startPos; -#if DEBUG - Log.Write( - " " + (startPos.HasValue ? startPos.Value.ToString() : "n/a") - ); -#endif } ); -#if DEBUG - Log.WriteLine(); -#endif break; case BlockType.CTime: -#if DEBUG - Log.Write("CTime:"); -#endif ReadDateTimeVector( dataVector, numFiles, delegate(int i, DateTime? time) { db._files[i].CTime = time; -#if DEBUG - Log.Write(" " + (time.HasValue ? time.Value.ToString() : "n/a")); -#endif } ); -#if DEBUG - Log.WriteLine(); -#endif break; case BlockType.ATime: -#if DEBUG - Log.Write("ATime:"); -#endif ReadDateTimeVector( dataVector, numFiles, delegate(int i, DateTime? time) { db._files[i].ATime = time; -#if DEBUG - Log.Write(" " + (time.HasValue ? time.Value.ToString() : "n/a")); -#endif } ); -#if DEBUG - Log.WriteLine(); -#endif break; case BlockType.MTime: -#if DEBUG - Log.Write("MTime:"); -#endif ReadDateTimeVector( dataVector, numFiles, delegate(int i, DateTime? time) { db._files[i].MTime = time; -#if DEBUG - Log.Write(" " + (time.HasValue ? time.Value.ToString() : "n/a")); -#endif } ); -#if DEBUG - Log.WriteLine(); -#endif break; case BlockType.Dummy: -#if DEBUG - Log.Write("Dummy: " + size); -#endif for (long j = 0; j < size; j++) { if (ReadByte() != 0) { - throw new InvalidOperationException(); + throw new ArchiveOperationException(); } } break; @@ -1180,7 +901,7 @@ internal class ArchiveReader var checkRecordsSize = (db._majorVersion > 0 || db._minorVersion > 2); if (checkRecordsSize && _currentReader.Offset - oldPos != size) { - throw new InvalidOperationException(); + throw new ArchiveOperationException(); } } @@ -1208,40 +929,60 @@ internal class ArchiveReader } } } - finally - { -#if DEBUG - Log.PopIndent(); -#endif - } + finally { } } #endregion #region Public Methods - public void Open(Stream stream) + public void Open(Stream stream, bool lookForHeader) { Close(); _streamOrigin = stream.Position; _streamEnding = stream.Length; - // TODO: Check Signature! - _header = new byte[0x20]; - for (var offset = 0; offset < 0x20; ) + var canScan = lookForHeader ? 0x80000 - 20 : 0; + while (true) { - var delta = stream.Read(_header, offset, 0x20 - offset); - if (delta == 0) + // TODO: Check Signature! + _header = new byte[0x20]; + for (var offset = 0; offset < 0x20; ) { - throw new EndOfStreamException(); + var delta = stream.Read(_header, offset, 0x20 - offset); + if (delta == 0) + { + throw new IncompleteArchiveException("Unexpected end of stream."); + } + + offset += delta; } - offset += delta; + + if ( + !lookForHeader + || _header + .AsSpan(0, length: 6) + .SequenceEqual([0x37, 0x7A, 0xBC, 0xAF, 0x27, 0x1C]) + ) + { + break; + } + + if (canScan == 0) + { + throw new InvalidFormatException("Unable to find 7z signature"); + } + + canScan--; + stream.Position = ++_streamOrigin; } _stream = stream; } + // OpenAsync moved to ArchiveReader.Async.cs + public void Close() { _stream?.Dispose(); @@ -1264,7 +1005,7 @@ internal class ArchiveReader if (db._majorVersion != 0) { - throw new InvalidOperationException(); + throw new ArchiveOperationException(); } var crcFromArchive = DataReader.Get32(_header, 8); @@ -1280,7 +1021,7 @@ internal class ArchiveReader if (crc != crcFromArchive) { - throw new InvalidOperationException(); + throw new ArchiveOperationException(); } db._startPositionAfterHeader = _streamOrigin + 0x20; @@ -1294,12 +1035,12 @@ internal class ArchiveReader if (nextHeaderOffset < 0 || nextHeaderSize < 0 || nextHeaderSize > int.MaxValue) { - throw new InvalidOperationException(); + throw new ArchiveOperationException(); } if (nextHeaderOffset > _streamEnding - db._startPositionAfterHeader) { - throw new InvalidOperationException("nextHeaderOffset is invalid"); + throw new ArchiveOperationException("nextHeaderOffset is invalid"); } _stream.Seek(nextHeaderOffset, SeekOrigin.Current); @@ -1309,7 +1050,7 @@ internal class ArchiveReader if (Crc.Finish(Crc.Update(Crc.INIT_CRC, header, 0, header.Length)) != nextHeaderCrc) { - throw new InvalidOperationException(); + throw new ArchiveOperationException(); } using (var streamSwitch = new CStreamSwitch()) @@ -1321,7 +1062,7 @@ internal class ArchiveReader { if (type != BlockType.EncodedHeader) { - throw new InvalidOperationException(); + throw new ArchiveOperationException(); } var dataVector = ReadAndDecodePackedStreams( @@ -1338,14 +1079,14 @@ internal class ArchiveReader if (dataVector.Count != 1) { - throw new InvalidOperationException(); + throw new ArchiveOperationException(); } streamSwitch.Set(this, dataVector[0]); if (ReadId() != BlockType.Header) { - throw new InvalidOperationException(); + throw new ArchiveOperationException(); } } @@ -1355,11 +1096,13 @@ internal class ArchiveReader return db; } + // ReadDatabaseAsync moved to ArchiveReader.Async.cs + internal class CExtractFolderInfo { internal int _fileIndex; internal int _folderIndex; - internal List _extractStatuses = new List(); + internal List _extractStatuses = new(); internal CExtractFolderInfo(int fileIndex, int folderIndex) { @@ -1393,7 +1136,7 @@ internal class ArchiveReader public override bool CanWrite => false; - public override void Flush() => throw new NotSupportedException(); + public override void Flush() { } public override long Length => throw new NotSupportedException(); @@ -1411,7 +1154,7 @@ internal class ArchiveReader public override void SetLength(long value) => throw new NotSupportedException(); - private Stream _stream; + private Stream? _stream; private long _rem; private int _currentIndex; @@ -1423,7 +1166,7 @@ internal class ArchiveReader ) { OpenFile(); - _stream.Dispose(); + _stream.NotNull().Dispose(); _stream = null; _currentIndex++; } @@ -1432,16 +1175,14 @@ internal class ArchiveReader private void OpenFile() { var index = _startIndex + _currentIndex; -#if DEBUG - Log.WriteLine(_db._files[index].Name); -#endif - if (_db._files[index].CrcDefined) + var crc = _db._files[index].Crc; + if (crc.HasValue) { - _stream = new CrcCheckStream(_db._files[index].Crc.Value); + _stream = new CrcCheckStream(crc.Value); } else { - _stream = new MemoryStream(); + _stream = new PooledMemoryStream(); } _rem = _db._files[index].Size; } @@ -1519,7 +1260,7 @@ internal class ArchiveReader var firstFileIndex = db._folderStartFileIndex[folderIndex]; if (firstFileIndex > fileIndex || fileIndex - firstFileIndex >= numFilesInFolder) { - throw new InvalidOperationException(); + throw new ArchiveOperationException(); } var skipCount = fileIndex - firstFileIndex; @@ -1534,11 +1275,10 @@ internal class ArchiveReader return new ReadOnlySubStream(s, db._files[fileIndex].Size); } - public void Extract(ArchiveDatabase db, int[] indices) + public void Extract(ArchiveDatabase db, int[]? indices) { - var allFilesMode = (indices is null); - - var numItems = allFilesMode ? db._files.Count : indices.Length; + var allFilesMode = indices is null; + var numItems = allFilesMode ? db._files.Count : indices!.Length; if (numItems == 0) { @@ -1548,7 +1288,7 @@ internal class ArchiveReader var extractFolderInfoVector = new List(); for (var i = 0; i < numItems; i++) { - var fileIndex = allFilesMode ? i : indices[i]; + var fileIndex = allFilesMode ? i : indices![i]; var folderIndex = db._fileIndexToFolderIndexMap[fileIndex]; if (folderIndex == -1) @@ -1574,7 +1314,7 @@ internal class ArchiveReader } } - byte[] buffer = null; + byte[] buffer = null!; foreach (var efi in extractFolderInfoVector) { int startIndex; diff --git a/src/SharpCompress/Common/SevenZip/ArchiveWriter.cs b/src/SharpCompress/Common/SevenZip/ArchiveWriter.cs new file mode 100644 index 00000000..d9f90e58 --- /dev/null +++ b/src/SharpCompress/Common/SevenZip/ArchiveWriter.cs @@ -0,0 +1,52 @@ +using System.IO; +using SharpCompress.Compressors.LZMA.Utilities; + +namespace SharpCompress.Common.SevenZip; + +/// +/// Top-level orchestrator for writing 7z archive headers. +/// Assembles the complete header from StreamsInfo and FilesInfo, +/// and supports writing either a raw header (kHeader) or an +/// encoded/compressed header (kEncodedHeader). +/// +internal static class ArchiveHeaderWriter +{ + /// + /// Writes a raw (uncompressed) header containing MainStreamsInfo and FilesInfo. + /// + public static void WriteRawHeader( + Stream stream, + SevenZipStreamsInfoWriter? mainStreamsInfo, + SevenZipFilesInfoWriter? filesInfo + ) + { + stream.WriteByte((byte)BlockType.Header); + + if (mainStreamsInfo != null) + { + stream.WriteByte((byte)BlockType.MainStreamsInfo); + mainStreamsInfo.Write(stream); + } + + if (filesInfo != null) + { + stream.WriteByte((byte)BlockType.FilesInfo); + filesInfo.Write(stream); + } + + stream.WriteByte((byte)BlockType.End); + } + + /// + /// Writes an encoded header - a StreamsInfo block that describes + /// how to decompress the actual header data. + /// + public static void WriteEncodedHeader( + Stream stream, + SevenZipStreamsInfoWriter headerStreamsInfo + ) + { + stream.WriteByte((byte)BlockType.EncodedHeader); + headerStreamsInfo.Write(stream); + } +} diff --git a/src/SharpCompress/Common/SevenZip/CCoderInfo.cs b/src/SharpCompress/Common/SevenZip/CCoderInfo.cs index d53c44f0..ce1a8808 100644 --- a/src/SharpCompress/Common/SevenZip/CCoderInfo.cs +++ b/src/SharpCompress/Common/SevenZip/CCoderInfo.cs @@ -1,11 +1,9 @@ -#nullable disable - namespace SharpCompress.Common.SevenZip; internal class CCoderInfo { internal CMethodId _methodId; - internal byte[] _props; + internal byte[]? _props; internal int _numInStreams; internal int _numOutStreams; } diff --git a/src/SharpCompress/Common/SevenZip/CFileItem.cs b/src/SharpCompress/Common/SevenZip/CFileItem.cs index c6509ee0..5305bc5d 100644 --- a/src/SharpCompress/Common/SevenZip/CFileItem.cs +++ b/src/SharpCompress/Common/SevenZip/CFileItem.cs @@ -1,5 +1,3 @@ -#nullable disable - using System; namespace SharpCompress.Common.SevenZip; @@ -8,18 +6,13 @@ internal class CFileItem { public long Size { get; internal set; } public uint? Attrib { get; internal set; } + public uint? ExtendedAttrib { get; internal set; } public uint? Crc { get; internal set; } - public string Name { get; internal set; } + public string Name { get; internal set; } = string.Empty; public bool HasStream { get; internal set; } public bool IsDir { get; internal set; } - public bool CrcDefined => Crc != null; - - public bool AttribDefined => Attrib != null; - - public void SetAttrib(uint attrib) => Attrib = attrib; - public DateTime? CTime { get; internal set; } public DateTime? ATime { get; internal set; } public DateTime? MTime { get; internal set; } diff --git a/src/SharpCompress/Common/SevenZip/CFolder.cs b/src/SharpCompress/Common/SevenZip/CFolder.cs index 9d3516b1..683c27ca 100644 --- a/src/SharpCompress/Common/SevenZip/CFolder.cs +++ b/src/SharpCompress/Common/SevenZip/CFolder.cs @@ -6,11 +6,11 @@ namespace SharpCompress.Common.SevenZip; internal class CFolder { - internal List _coders = new List(); - internal List _bindPairs = new List(); - internal List _packStreams = new List(); + internal List _coders = new(); + internal List _bindPairs = new(); + internal List _packStreams = new(); internal int _firstPackStreamId; - internal List _unpackSizes = new List(); + internal List _unpackSizes = new(); internal uint? _unpackCrc; internal bool UnpackCrcDefined => _unpackCrc != null; @@ -30,7 +30,7 @@ internal class CFolder } } - throw new InvalidOperationException(); + throw new ArchiveOperationException(); } public int GetNumOutStreams() diff --git a/src/SharpCompress/Common/SevenZip/CMethodId.cs b/src/SharpCompress/Common/SevenZip/CMethodId.cs index 6dce5773..8494aad5 100644 --- a/src/SharpCompress/Common/SevenZip/CMethodId.cs +++ b/src/SharpCompress/Common/SevenZip/CMethodId.cs @@ -7,10 +7,10 @@ internal readonly struct CMethodId public const ulong K_LZMA2_ID = 0x21; public const ulong K_AES_ID = 0x06F10701; - public static readonly CMethodId K_COPY = new CMethodId(K_COPY_ID); - public static readonly CMethodId K_LZMA = new CMethodId(K_LZMA_ID); - public static readonly CMethodId K_LZMA2 = new CMethodId(K_LZMA2_ID); - public static readonly CMethodId K_AES = new CMethodId(K_AES_ID); + public static readonly CMethodId K_COPY = new(K_COPY_ID); + public static readonly CMethodId K_LZMA = new(K_LZMA_ID); + public static readonly CMethodId K_LZMA2 = new(K_LZMA2_ID); + public static readonly CMethodId K_AES = new(K_AES_ID); public readonly ulong _id; diff --git a/src/SharpCompress/Common/SevenZip/CStreamSwitch.cs b/src/SharpCompress/Common/SevenZip/CStreamSwitch.cs index 9238a5a1..8b429b19 100644 --- a/src/SharpCompress/Common/SevenZip/CStreamSwitch.cs +++ b/src/SharpCompress/Common/SevenZip/CStreamSwitch.cs @@ -15,9 +15,6 @@ internal struct CStreamSwitch : IDisposable if (_active) { _active = false; -#if DEBUG - Log.WriteLine("[end of switch]"); -#endif } if (_needRemove) @@ -47,22 +44,14 @@ internal struct CStreamSwitch : IDisposable var dataIndex = archive.ReadNum(); if (dataIndex < 0 || dataIndex >= dataVector.Count) { - throw new InvalidOperationException(); + throw new ArchiveOperationException(); } -#if DEBUG - Log.WriteLine("[switch to stream {0}]", dataIndex); -#endif _archive = archive; _archive.AddByteStream(dataVector[dataIndex], 0, dataVector[dataIndex].Length); _needRemove = true; _active = true; } - else - { -#if DEBUG - Log.WriteLine("[inline data]"); -#endif - } + else { } } } diff --git a/src/SharpCompress/Common/SevenZip/DataReader.cs b/src/SharpCompress/Common/SevenZip/DataReader.cs index f280a3c1..8788192b 100644 --- a/src/SharpCompress/Common/SevenZip/DataReader.cs +++ b/src/SharpCompress/Common/SevenZip/DataReader.cs @@ -49,7 +49,7 @@ internal class DataReader { if (Offset >= _ending) { - throw new EndOfStreamException(); + throw new IncompleteArchiveException("Unexpected end of stream."); } return _buffer[Offset++]; @@ -59,7 +59,7 @@ internal class DataReader { if (length > _ending - Offset) { - throw new EndOfStreamException(); + throw new IncompleteArchiveException("Unexpected end of stream."); } while (length-- > 0) @@ -72,13 +72,10 @@ internal class DataReader { if (size > _ending - Offset) { - throw new EndOfStreamException(); + throw new IncompleteArchiveException("Unexpected end of stream."); } Offset += (int)size; -#if DEBUG - Log.WriteLine("SkipData {0}", size); -#endif } public void SkipData() => SkipData(checked((long)ReadNumber())); @@ -87,7 +84,7 @@ internal class DataReader { if (Offset >= _ending) { - throw new EndOfStreamException(); + throw new IncompleteArchiveException("Unexpected end of stream."); } var firstByte = _buffer[Offset++]; @@ -105,7 +102,7 @@ internal class DataReader if (Offset >= _ending) { - throw new EndOfStreamException(); + throw new IncompleteArchiveException("Unexpected end of stream."); } value |= (ulong)_buffer[Offset++] << (8 * i); @@ -130,7 +127,7 @@ internal class DataReader { if (Offset + 4 > _ending) { - throw new EndOfStreamException(); + throw new IncompleteArchiveException("Unexpected end of stream."); } var res = Get32(_buffer, Offset); @@ -142,7 +139,7 @@ internal class DataReader { if (Offset + 8 > _ending) { - throw new EndOfStreamException(); + throw new IncompleteArchiveException("Unexpected end of stream."); } var res = Get64(_buffer, Offset); @@ -158,7 +155,7 @@ internal class DataReader { if (ending + 2 > _ending) { - throw new EndOfStreamException(); + throw new IncompleteArchiveException("Unexpected end of stream."); } if (_buffer[ending] == 0 && _buffer[ending + 1] == 0) diff --git a/src/SharpCompress/Common/SevenZip/SevenZipEntry.cs b/src/SharpCompress/Common/SevenZip/SevenZipEntry.cs index 66170ba8..576c1d16 100644 --- a/src/SharpCompress/Common/SevenZip/SevenZipEntry.cs +++ b/src/SharpCompress/Common/SevenZip/SevenZipEntry.cs @@ -1,11 +1,16 @@ -using System; +using System; using System.Collections.Generic; +using SharpCompress.Common.Options; namespace SharpCompress.Common.SevenZip; public class SevenZipEntry : Entry { - internal SevenZipEntry(SevenZipFilePart filePart) => FilePart = filePart; + internal SevenZipEntry(SevenZipFilePart filePart, IReaderOptions readerOptions) + : base(readerOptions) + { + FilePart = filePart; + } internal SevenZipFilePart FilePart { get; } @@ -13,7 +18,29 @@ public class SevenZipEntry : Entry public override long Crc => FilePart.Header.Crc ?? 0; - public override string Key => FilePart.Header.Name; + internal override ChecksumDescriptor Checksum + { + get + { + if ( + IsDirectory + || FilePart.Header.IsAnti + || !FilePart.Header.HasStream + || !FilePart.Header.Crc.HasValue + ) + { + return default; + } + + return new ChecksumDescriptor( + ChecksumKind.Crc32, + FilePart.Header.Crc.Value, + IsAvailable: true + ); + } + } + + public override string? Key => FilePart.Header.Name; public override string? LinkTarget => null; @@ -38,5 +65,8 @@ public class SevenZipEntry : Entry public override int? Attrib => FilePart.Header.Attrib.HasValue ? (int?)FilePart.Header.Attrib.Value : null; + public int? ExtendedAttrib => + FilePart.Header.ExtendedAttrib.HasValue ? (int?)FilePart.Header.ExtendedAttrib.Value : null; + internal override IEnumerable Parts => FilePart.AsEnumerable(); } diff --git a/src/SharpCompress/Common/SevenZip/SevenZipFilePart.cs b/src/SharpCompress/Common/SevenZip/SevenZipFilePart.cs index aad191c9..65bb7503 100644 --- a/src/SharpCompress/Common/SevenZip/SevenZipFilePart.cs +++ b/src/SharpCompress/Common/SevenZip/SevenZipFilePart.cs @@ -1,6 +1,7 @@ -using System; using System.IO; using System.Linq; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.IO; namespace SharpCompress.Common.SevenZip; @@ -16,7 +17,7 @@ internal class SevenZipFilePart : FilePart ArchiveDatabase database, int index, CFileItem fileEntry, - ArchiveEncoding archiveEncoding + IArchiveEncoding archiveEncoding ) : base(archiveEncoding) { @@ -41,7 +42,7 @@ internal class SevenZipFilePart : FilePart { if (!Header.HasStream) { - return null!; + return Stream.Null; } var folderStream = _database.GetFolderStream(_stream, Folder!, _database.PasswordProvider); @@ -56,7 +57,33 @@ internal class SevenZipFilePart : FilePart { folderStream.Skip(skipSize); } - return new ReadOnlySubStream(folderStream, Header.Size); + return new ReadOnlySubStream(folderStream, Header.Size, leaveOpen: false); + } + + internal override async ValueTask GetCompressedStreamAsync( + CancellationToken cancellationToken = default + ) + { + if (!Header.HasStream) + { + return Stream.Null; + } + var folderStream = await _database + .GetFolderStreamAsync(_stream, Folder!, _database.PasswordProvider, cancellationToken) + .ConfigureAwait(false); + + var firstFileIndex = _database._folderStartFileIndex[_database._folders.IndexOf(Folder!)]; + var skipCount = Index - firstFileIndex; + long skipSize = 0; + for (var i = 0; i < skipCount; i++) + { + skipSize += _database._files[firstFileIndex + i].Size; + } + if (skipSize > 0) + { + await folderStream.SkipAsync(skipSize, cancellationToken).ConfigureAwait(false); + } + return new ReadOnlySubStream(folderStream, Header.Size, leaveOpen: false); } public CompressionType CompressionType @@ -68,39 +95,31 @@ internal class SevenZipFilePart : FilePart } } + private const uint K_COPY = 0x0; private const uint K_LZMA2 = 0x21; private const uint K_LZMA = 0x030101; private const uint K_PPMD = 0x030401; private const uint K_B_ZIP2 = 0x040202; - internal CompressionType GetCompression() + private CompressionType GetCompression() { if (Header.IsDir) - return CompressionType.None; - - var coder = Folder!._coders.First(); - switch (coder._methodId._id) { - case K_LZMA: - case K_LZMA2: - { - return CompressionType.LZMA; - } - case K_PPMD: - { - return CompressionType.PPMd; - } - case K_B_ZIP2: - { - return CompressionType.BZip2; - } - default: - throw new NotImplementedException(); + return CompressionType.None; } + + var coder = Folder.NotNull()._coders.First(); + return coder._methodId._id switch + { + K_COPY => CompressionType.None, + K_LZMA or K_LZMA2 => CompressionType.LZMA, + K_PPMD => CompressionType.PPMd, + K_B_ZIP2 => CompressionType.BZip2, + _ => throw new InvalidFormatException(), + }; } internal bool IsEncrypted => - Header.IsDir - ? false - : Folder!._coders.FindIndex(c => c._methodId._id == CMethodId.K_AES_ID) != -1; + !Header.IsDir + && Folder?._coders.FindIndex(c => c._methodId._id == CMethodId.K_AES_ID) != -1; } diff --git a/src/SharpCompress/Common/SevenZip/SevenZipFilesInfo.cs b/src/SharpCompress/Common/SevenZip/SevenZipFilesInfo.cs new file mode 100644 index 00000000..9162474b --- /dev/null +++ b/src/SharpCompress/Common/SevenZip/SevenZipFilesInfo.cs @@ -0,0 +1,227 @@ +using System; +using System.IO; +using System.Text; +using SharpCompress.Compressors.LZMA.Utilities; +using SharpCompress.IO; + +namespace SharpCompress.Common.SevenZip; + +/// +/// Entry metadata collected during writing, used to build FilesInfo header. +/// +internal sealed class SevenZipWriteEntry +{ + public string Name { get; init; } = string.Empty; + public DateTime? ModificationTime { get; init; } + public uint? Attributes { get; init; } + public bool IsDirectory { get; init; } + public bool IsEmpty { get; init; } +} + +/// +/// Writes the FilesInfo section of a 7z header, including all file properties +/// (names, timestamps, attributes, empty stream/file markers). +/// +internal sealed class SevenZipFilesInfoWriter +{ + public SevenZipWriteEntry[] Entries { get; init; } = []; + + public void Write(Stream stream) + { + var numFiles = (ulong)Entries.Length; + stream.WriteEncodedUInt64(numFiles); + + // Count empty streams (directories + zero-length files) + var emptyStreamCount = 0; + for (var i = 0; i < Entries.Length; i++) + { + if (Entries[i].IsEmpty || Entries[i].IsDirectory) + { + emptyStreamCount++; + } + } + + // EmptyStream property + if (emptyStreamCount > 0) + { + WriteEmptyStreamProperty(stream, emptyStreamCount); + } + + // Names property + WriteNameProperty(stream); + + // MTime property + WriteMTimeProperty(stream); + + // Attributes property + WriteAttributesProperty(stream); + + stream.WriteByte((byte)BlockType.End); + } + + private void WriteEmptyStreamProperty(Stream stream, int emptyStreamCount) + { + var emptyStreams = new bool[Entries.Length]; + var emptyFiles = new bool[emptyStreamCount]; + var hasEmptyFile = false; + var emptyIndex = 0; + + for (var i = 0; i < Entries.Length; i++) + { + if (Entries[i].IsEmpty || Entries[i].IsDirectory) + { + emptyStreams[i] = true; + var isEmptyFile = !Entries[i].IsDirectory; + emptyFiles[emptyIndex++] = isEmptyFile; + if (isEmptyFile) + { + hasEmptyFile = true; + } + } + } + + // kEmptyStream + WriteFileProperty(stream, BlockType.EmptyStream, s => s.WriteBoolVector(emptyStreams)); + + // kEmptyFile (only if there are actual empty files, not just directories) + if (hasEmptyFile) + { + WriteFileProperty(stream, BlockType.EmptyFile, s => s.WriteBoolVector(emptyFiles)); + } + } + + private void WriteNameProperty(Stream stream) + { + WriteFileProperty( + stream, + BlockType.Name, + s => + { + // External = 0 (inline) + s.WriteByte(0); + + for (var i = 0; i < Entries.Length; i++) + { + var nameBytes = Encoding.Unicode.GetBytes(Entries[i].Name); + s.Write(nameBytes); + // null terminator (2 bytes for UTF-16) + s.WriteByte(0); + s.WriteByte(0); + } + } + ); + } + + private void WriteMTimeProperty(Stream stream) + { + var hasTimes = false; + for (var i = 0; i < Entries.Length; i++) + { + if (Entries[i].ModificationTime != null) + { + hasTimes = true; + break; + } + } + + if (!hasTimes) + { + return; + } + + WriteFileProperty( + stream, + BlockType.MTime, + s => + { + var defined = new bool[Entries.Length]; + for (var i = 0; i < Entries.Length; i++) + { + defined[i] = Entries[i].ModificationTime != null; + } + s.WriteOptionalBoolVector(defined); + + // External = 0 (inline) + s.WriteByte(0); + + var buf = new byte[8]; + for (var i = 0; i < Entries.Length; i++) + { + if (Entries[i].ModificationTime is { } mtime) + { + var fileTime = (ulong)mtime.ToUniversalTime().ToFileTimeUtc(); + System.Buffers.Binary.BinaryPrimitives.WriteUInt64LittleEndian( + buf, + fileTime + ); + s.Write(buf, 0, 8); + } + } + } + ); + } + + private void WriteAttributesProperty(Stream stream) + { + var hasAttrs = false; + for (var i = 0; i < Entries.Length; i++) + { + if (Entries[i].Attributes != null) + { + hasAttrs = true; + break; + } + } + + if (!hasAttrs) + { + return; + } + + WriteFileProperty( + stream, + BlockType.WinAttributes, + s => + { + var defined = new bool[Entries.Length]; + for (var i = 0; i < Entries.Length; i++) + { + defined[i] = Entries[i].Attributes != null; + } + s.WriteOptionalBoolVector(defined); + + // External = 0 (inline) + s.WriteByte(0); + + var buf = new byte[4]; + for (var i = 0; i < Entries.Length; i++) + { + if (Entries[i].Attributes is { } attrs) + { + System.Buffers.Binary.BinaryPrimitives.WriteUInt32LittleEndian(buf, attrs); + s.Write(buf, 0, 4); + } + } + } + ); + } + + /// + /// Writes a file property block: PropertyID + size + data. + /// Size is computed by writing to a temporary buffer first. + /// + private static void WriteFileProperty( + Stream stream, + BlockType propertyId, + Action writeData + ) + { + using var dataStream = new PooledMemoryStream(); + writeData(dataStream); + + stream.WriteByte((byte)propertyId); + stream.WriteEncodedUInt64((ulong)dataStream.Length); + dataStream.Position = 0; + dataStream.CopyTo(stream); + } +} diff --git a/src/SharpCompress/Common/SevenZip/SevenZipHeaderStructures.cs b/src/SharpCompress/Common/SevenZip/SevenZipHeaderStructures.cs new file mode 100644 index 00000000..24e86bfd --- /dev/null +++ b/src/SharpCompress/Common/SevenZip/SevenZipHeaderStructures.cs @@ -0,0 +1,306 @@ +using System; +using System.IO; +using SharpCompress.Compressors.LZMA.Utilities; + +namespace SharpCompress.Common.SevenZip; + +/// +/// Writes Digests (CRC32 arrays with optional-defined-vector) for 7z headers. +/// +internal sealed class SevenZipDigestsWriter(uint?[] crcs) +{ + public uint?[] CRCs { get; } = crcs; + + public void Write(Stream stream) + { + var defined = new bool[CRCs.Length]; + for (var i = 0; i < CRCs.Length; i++) + { + defined[i] = CRCs[i] != null; + } + + stream.WriteOptionalBoolVector(defined); + + var buf = new byte[4]; + for (var i = 0; i < CRCs.Length; i++) + { + if (CRCs[i] is { } crcValue) + { + System.Buffers.Binary.BinaryPrimitives.WriteUInt32LittleEndian(buf, crcValue); + stream.Write(buf, 0, 4); + } + } + } + + public bool HasAnyDefined() + { + for (var i = 0; i < CRCs.Length; i++) + { + if (CRCs[i] != null) + { + return true; + } + } + return false; + } +} + +/// +/// Writes PackInfo section: packed stream positions, sizes, and CRCs. +/// +internal sealed class SevenZipPackInfoWriter +{ + public ulong PackPos { get; init; } + public ulong[] Sizes { get; init; } = []; + public uint?[] CRCs { get; init; } = []; + + public void Write(Stream stream) + { + stream.WriteEncodedUInt64(PackPos); + stream.WriteEncodedUInt64((ulong)Sizes.Length); + + // Sizes + stream.WriteByte((byte)BlockType.Size); + for (var i = 0; i < Sizes.Length; i++) + { + stream.WriteEncodedUInt64(Sizes[i]); + } + + // CRCs (optional) + var digests = new SevenZipDigestsWriter(CRCs); + if (digests.HasAnyDefined()) + { + stream.WriteByte((byte)BlockType.Crc); + digests.Write(stream); + } + + stream.WriteByte((byte)BlockType.End); + } +} + +/// +/// Writes UnPackInfo section: folder definitions (coders, bind pairs, unpack sizes, CRCs). +/// +internal sealed class SevenZipUnPackInfoWriter +{ + public CFolder[] Folders { get; init; } = []; + + public void Write(Stream stream) + { + stream.WriteByte((byte)BlockType.Folder); + + // Number of folders + stream.WriteEncodedUInt64((ulong)Folders.Length); + + // External = 0 (inline) + stream.WriteByte(0); + + // Write each folder's coder definitions + for (var i = 0; i < Folders.Length; i++) + { + WriteFolder(stream, Folders[i]); + } + + // CodersUnPackSize + stream.WriteByte((byte)BlockType.CodersUnpackSize); + for (var i = 0; i < Folders.Length; i++) + { + for (var j = 0; j < Folders[i]._unpackSizes.Count; j++) + { + stream.WriteEncodedUInt64((ulong)Folders[i]._unpackSizes[j]); + } + } + + // UnPackDigests (CRCs per folder) + var hasCrc = false; + for (var i = 0; i < Folders.Length; i++) + { + if (Folders[i]._unpackCrc != null) + { + hasCrc = true; + break; + } + } + + if (hasCrc) + { + stream.WriteByte((byte)BlockType.Crc); + var crcs = new uint?[Folders.Length]; + for (var i = 0; i < Folders.Length; i++) + { + crcs[i] = Folders[i]._unpackCrc; + } + new SevenZipDigestsWriter(crcs).Write(stream); + } + + stream.WriteByte((byte)BlockType.End); + } + + private static void WriteFolder(Stream stream, CFolder folder) + { + // NumCoders + stream.WriteEncodedUInt64((ulong)folder._coders.Count); + + for (var i = 0; i < folder._coders.Count; i++) + { + WriteCoder(stream, folder._coders[i]); + } + + // BindPairs + for (var i = 0; i < folder._bindPairs.Count; i++) + { + stream.WriteEncodedUInt64((ulong)folder._bindPairs[i]._inIndex); + stream.WriteEncodedUInt64((ulong)folder._bindPairs[i]._outIndex); + } + + // PackedIndices (only if > 1 packed stream) + var numPackStreams = folder._packStreams.Count; + if (numPackStreams > 1) + { + for (var i = 0; i < numPackStreams; i++) + { + stream.WriteEncodedUInt64((ulong)folder._packStreams[i]); + } + } + } + + private static void WriteCoder(Stream stream, CCoderInfo coder) + { + var codecIdLength = coder._methodId.GetLength(); + byte attributes = (byte)(codecIdLength & 0x0F); + + var isComplex = coder._numInStreams != 1 || coder._numOutStreams != 1; + if (isComplex) + { + attributes |= 0x10; + } + + var hasProperties = coder._props != null && coder._props.Length > 0; + if (hasProperties) + { + attributes |= 0x20; + } + + stream.WriteByte(attributes); + + // Codec ID bytes (big-endian, most significant byte first) + var codecId = new byte[codecIdLength]; + var id = coder._methodId._id; + for (var i = codecIdLength - 1; i >= 0; i--) + { + codecId[i] = (byte)(id & 0xFF); + id >>= 8; + } + stream.Write(codecId, 0, codecIdLength); + + if (isComplex) + { + stream.WriteEncodedUInt64((ulong)coder._numInStreams); + stream.WriteEncodedUInt64((ulong)coder._numOutStreams); + } + + if (hasProperties) + { + stream.WriteEncodedUInt64((ulong)coder._props!.Length); + stream.Write(coder._props); + } + } +} + +/// +/// Writes SubStreamsInfo section: per-file unpack sizes and CRCs within folders. +/// +internal sealed class SevenZipSubStreamsInfoWriter +{ + public CFolder[] Folders { get; init; } = []; + public ulong[] NumUnPackStreamsInFolders { get; init; } = []; + public ulong[] UnPackSizes { get; init; } = []; + public uint?[] CRCs { get; init; } = []; + + public void Write(Stream stream) + { + var numFolders = (ulong)Folders.Length; + + // NumUnPackStream per folder (skip if all folders have exactly 1 stream) + var totalStreams = 0UL; + var allSingle = true; + for (var i = 0; i < NumUnPackStreamsInFolders.Length; i++) + { + totalStreams += NumUnPackStreamsInFolders[i]; + if (NumUnPackStreamsInFolders[i] != 1) + { + allSingle = false; + } + } + + if (!allSingle) + { + stream.WriteByte((byte)BlockType.NumUnpackStream); + for (var i = 0; i < NumUnPackStreamsInFolders.Length; i++) + { + stream.WriteEncodedUInt64(NumUnPackStreamsInFolders[i]); + } + } + + // UnPackSizes - write all except the last per folder (it's implicit from folder unpack size). + // Only emit the Size block when at least one folder has multiple substreams. + if (UnPackSizes.Length > 0 && !allSingle) + { + stream.WriteByte((byte)BlockType.Size); + + var sizeIndex = 0; + for (var i = 0; i < NumUnPackStreamsInFolders.Length; i++) + { + var numStreams = NumUnPackStreamsInFolders[i]; + for (var j = 1UL; j < numStreams; j++) + { + stream.WriteEncodedUInt64(UnPackSizes[sizeIndex++]); + } + sizeIndex++; // skip the last (implicit) + } + } + + // Digests for streams with unknown CRCs + var digests = new SevenZipDigestsWriter(CRCs); + if (digests.HasAnyDefined()) + { + stream.WriteByte((byte)BlockType.Crc); + digests.Write(stream); + } + + stream.WriteByte((byte)BlockType.End); + } +} + +/// +/// Writes the complete StreamsInfo section (PackInfo + UnPackInfo + SubStreamsInfo). +/// +internal sealed class SevenZipStreamsInfoWriter +{ + public SevenZipPackInfoWriter? PackInfo { get; init; } + public SevenZipUnPackInfoWriter? UnPackInfo { get; init; } + public SevenZipSubStreamsInfoWriter? SubStreamsInfo { get; init; } + + public void Write(Stream stream) + { + if (PackInfo != null) + { + stream.WriteByte((byte)BlockType.PackInfo); + PackInfo.Write(stream); + } + + if (UnPackInfo != null) + { + stream.WriteByte((byte)BlockType.UnpackInfo); + UnPackInfo.Write(stream); + } + + if (SubStreamsInfo != null) + { + stream.WriteByte((byte)BlockType.SubStreamsInfo); + SubStreamsInfo.Write(stream); + } + + stream.WriteByte((byte)BlockType.End); + } +} diff --git a/src/SharpCompress/Common/SevenZip/SevenZipSignatureHeader.cs b/src/SharpCompress/Common/SevenZip/SevenZipSignatureHeader.cs new file mode 100644 index 00000000..060bc654 --- /dev/null +++ b/src/SharpCompress/Common/SevenZip/SevenZipSignatureHeader.cs @@ -0,0 +1,145 @@ +using System; +using System.Buffers.Binary; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Crypto; + +namespace SharpCompress.Common.SevenZip; + +/// +/// Handles writing the 7z signature header (32 bytes at position 0 of the archive). +/// Layout: [6 bytes magic] [2 bytes version] [4 bytes StartHeaderCRC] [20 bytes StartHeader] +/// +internal static class SevenZipSignatureHeaderWriter +{ + /// + /// 7z file magic signature bytes. + /// + private static readonly byte[] Signature = [(byte)'7', (byte)'z', 0xBC, 0xAF, 0x27, 0x1C]; + + /// + /// Total size of the signature header in bytes (6+2+4+8+8+4 = 32). + /// + public const int HeaderSize = 32; + + /// + /// Writes a placeholder signature header (all zeros for CRC/offset fields). + /// Call this at the start of archive creation to reserve space. + /// + public static void WritePlaceholder(Stream stream) + { + var header = new byte[HeaderSize]; + + // magic signature + Array.Copy(Signature, 0, header, 0, Signature.Length); + + // version: major=0, minor=2 (standard 7z format) + header[6] = 0; + header[7] = 2; + + // remaining 24 bytes are zero (placeholder for CRC and StartHeader) + stream.Write(header, 0, header.Length); + } + + /// + /// Asynchronously writes a placeholder signature header (all zeros for CRC/offset fields). + /// Call this at the start of archive creation to reserve space. + /// + public static async ValueTask WritePlaceholderAsync( + Stream stream, + CancellationToken cancellationToken = default + ) + { + var header = new byte[HeaderSize]; + + // magic signature + Array.Copy(Signature, 0, header, 0, Signature.Length); + + // version: major=0, minor=2 (standard 7z format) + header[6] = 0; + header[7] = 2; + + // remaining 24 bytes are zero (placeholder for CRC and StartHeader) + await stream.WriteAsync(header, 0, header.Length, cancellationToken).ConfigureAwait(false); + } + + /// + /// Writes the final signature header with correct offsets and CRCs. + /// The stream must be seekable; this method seeks to position 0. + /// + /// The archive output stream (seekable). + /// Offset from end of signature header to start of metadata header. + /// Size of the metadata header in bytes. + /// CRC32 of the metadata header bytes. + public static void WriteFinal( + Stream stream, + ulong nextHeaderOffset, + ulong nextHeaderSize, + uint nextHeaderCrc + ) + { + var header = BuildFinalHeader(nextHeaderOffset, nextHeaderSize, nextHeaderCrc); + + // Write at position 0 + stream.Position = 0; + stream.Write(header, 0, header.Length); + } + + /// + /// Asynchronously writes the final signature header with correct offsets and CRCs. + /// The stream must be seekable; this method seeks to position 0. + /// + public static async ValueTask WriteFinalAsync( + Stream stream, + ulong nextHeaderOffset, + ulong nextHeaderSize, + uint nextHeaderCrc, + CancellationToken cancellationToken = default + ) + { + var header = BuildFinalHeader(nextHeaderOffset, nextHeaderSize, nextHeaderCrc); + + // Write at position 0 + stream.Position = 0; + await stream.WriteAsync(header, 0, header.Length, cancellationToken).ConfigureAwait(false); + } + + private static byte[] BuildFinalHeader( + ulong nextHeaderOffset, + ulong nextHeaderSize, + uint nextHeaderCrc + ) + { + // Build StartHeader (20 bytes): NextHeaderOffset(8) + NextHeaderSize(8) + NextHeaderCRC(4) + var startHeader = new byte[20]; + BinaryPrimitives.WriteUInt64LittleEndian(startHeader.AsSpan(0, 8), nextHeaderOffset); + BinaryPrimitives.WriteUInt64LittleEndian(startHeader.AsSpan(8, 8), nextHeaderSize); + BinaryPrimitives.WriteUInt32LittleEndian(startHeader.AsSpan(16, 4), nextHeaderCrc); + + // CRC32 of StartHeader + var startHeaderCrc = Crc32Stream.Compute( + Crc32Stream.DEFAULT_POLYNOMIAL, + Crc32Stream.DEFAULT_SEED, + startHeader + ); + + // Assemble full 32-byte header + var header = new byte[HeaderSize]; + + // magic signature + Array.Copy(Signature, 0, header, 0, Signature.Length); + + // version + header[6] = 0; + header[7] = 2; + + // StartHeaderCRC + BinaryPrimitives.WriteUInt32LittleEndian(header.AsSpan(8, 4), startHeaderCrc); + + // StartHeader + Array.Copy(startHeader, 0, header, 12, startHeader.Length); + + return header; + } +} diff --git a/src/SharpCompress/Common/SevenZip/SevenZipStreamsCompressor.cs b/src/SharpCompress/Common/SevenZip/SevenZipStreamsCompressor.cs new file mode 100644 index 00000000..1b704e0c --- /dev/null +++ b/src/SharpCompress/Common/SevenZip/SevenZipStreamsCompressor.cs @@ -0,0 +1,278 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Compressors.LZMA; +using SharpCompress.Crypto; + +namespace SharpCompress.Common.SevenZip; + +/// +/// Result of compressing a stream - contains folder metadata, compressed sizes, and CRCs. +/// +internal sealed class PackedStream +{ + public CFolder Folder { get; init; } = new(); + public ulong[] Sizes { get; init; } = []; + public uint?[] CRCs { get; init; } = []; +} + +/// +/// Compresses a single input stream using LZMA or LZMA2, writing compressed output +/// to the archive stream. Builds the CFolder metadata describing the compression. +/// Uses SharpCompress's existing LzmaStream encoder. +/// +internal sealed class SevenZipStreamsCompressor(Stream outputStream) +{ + /// + /// Compresses the input stream to the output stream using the specified method. + /// Returns a PackedStream containing folder metadata, compressed size, and CRCs. + /// + /// Uncompressed data to compress. + /// Compression method (LZMA or LZMA2). + /// LZMA encoder properties (null for defaults). + public PackedStream Compress( + Stream inputStream, + CompressionType compressionType, + LzmaEncoderProperties? encoderProperties = null + ) + { + var isLzma2 = compressionType == CompressionType.LZMA2; + encoderProperties ??= new LzmaEncoderProperties(eos: !isLzma2); + + var outStartOffset = outputStream.Position; + + // Wrap the output stream in CRC calculator + using var outCrcStream = new Crc32Stream(outputStream); + + byte[] properties; + + if (isLzma2) + { + // LZMA2: use Lzma2EncoderStream for chunk-based framing + using var lzma2Stream = new Lzma2EncoderStream( + outCrcStream, + encoderProperties.DictionarySize, + encoderProperties.NumFastBytes + ); + + CopyWithCrc(inputStream, lzma2Stream, out var inputCrc2, out var inputSize2); + lzma2Stream.Dispose(); + + properties = lzma2Stream.Properties; + + return BuildPackedStream( + isLzma2: true, + properties, + (ulong)(outputStream.Position - outStartOffset), + (ulong)inputSize2, + inputCrc2, + outCrcStream.Crc + ); + } + + // LZMA + using var lzmaStream = LzmaStream.Create(encoderProperties, false, outCrcStream); + properties = lzmaStream.Properties; + + CopyWithCrc(inputStream, lzmaStream, out var inputCrc, out var inputSize); + lzmaStream.Dispose(); + + return BuildPackedStream( + isLzma2: false, + properties, + (ulong)(outputStream.Position - outStartOffset), + (ulong)inputSize, + inputCrc, + outCrcStream.Crc + ); + } + + /// + /// Asynchronously compresses the input stream to the output stream using the specified method. + /// Returns a PackedStream containing folder metadata, compressed size, and CRCs. + /// + /// Uncompressed data to compress. + /// Compression method (LZMA or LZMA2). + /// LZMA encoder properties (null for defaults). + /// Cancellation token. + public async ValueTask CompressAsync( + Stream inputStream, + CompressionType compressionType, + LzmaEncoderProperties? encoderProperties = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + + var isLzma2 = compressionType == CompressionType.LZMA2; + encoderProperties ??= new LzmaEncoderProperties(eos: !isLzma2); + + var outStartOffset = outputStream.Position; + + // Wrap the output stream in CRC calculator + using var outCrcStream = new Crc32Stream(outputStream); + + byte[] properties; + + if (isLzma2) + { + // LZMA2: use Lzma2EncoderStream for chunk-based framing + uint inputCrc2; + long inputSize2; + { + await using var lzma2Stream = new Lzma2EncoderStream( + outCrcStream, + encoderProperties.DictionarySize, + encoderProperties.NumFastBytes + ); + + (inputCrc2, inputSize2) = await CopyWithCrcAsync( + inputStream, + lzma2Stream, + cancellationToken + ) + .ConfigureAwait(false); + + properties = lzma2Stream.Properties; + } + + return BuildPackedStream( + isLzma2: true, + properties, + (ulong)(outputStream.Position - outStartOffset), + (ulong)inputSize2, + inputCrc2, + outCrcStream.Crc + ); + } + + // LZMA + uint inputCrc; + long inputSize; + { + await using var lzmaStream = LzmaStream.Create(encoderProperties, false, outCrcStream); + properties = lzmaStream.Properties; + + (inputCrc, inputSize) = await CopyWithCrcAsync( + inputStream, + lzmaStream, + cancellationToken + ) + .ConfigureAwait(false); + } + + return BuildPackedStream( + isLzma2: false, + properties, + (ulong)(outputStream.Position - outStartOffset), + (ulong)inputSize, + inputCrc, + outCrcStream.Crc + ); + } + + /// + /// Copies data from source to destination while computing CRC32 of the source data. + /// Uses Crc32Stream.Compute for CRC calculation to avoid duplicating the table/algorithm. + /// + private static void CopyWithCrc( + Stream source, + Stream destination, + out uint crc, + out long bytesRead + ) + { + var seed = Crc32Stream.DEFAULT_SEED; + var buffer = new byte[81920]; + long totalRead = 0; + + int read; + while ((read = source.Read(buffer, 0, buffer.Length)) > 0) + { + // Crc32Stream.Compute returns ~CalculateCrc(table, seed, data), + // so passing ~result as next seed chains correctly. + seed = ~Crc32Stream.Compute( + Crc32Stream.DEFAULT_POLYNOMIAL, + seed, + buffer.AsSpan(0, read) + ); + destination.Write(buffer, 0, read); + totalRead += read; + } + + crc = ~seed; + bytesRead = totalRead; + } + + /// + /// Asynchronously copies data from source to destination while computing CRC32 of source data. + /// Uses Crc32Stream.Compute for CRC calculation to avoid duplicating the table/algorithm. + /// + private static async ValueTask<(uint crc, long bytesRead)> CopyWithCrcAsync( + Stream source, + Stream destination, + CancellationToken cancellationToken + ) + { + var seed = Crc32Stream.DEFAULT_SEED; + var buffer = new byte[81920]; + long totalRead = 0; + + int read; + while ( + ( + read = await source + .ReadAsync(buffer, 0, buffer.Length, cancellationToken) + .ConfigureAwait(false) + ) > 0 + ) + { + // Crc32Stream.Compute returns ~CalculateCrc(table, seed, data), + // so passing ~result as next seed chains correctly. + seed = ~Crc32Stream.Compute( + Crc32Stream.DEFAULT_POLYNOMIAL, + seed, + buffer.AsSpan(0, read) + ); + await destination.WriteAsync(buffer, 0, read, cancellationToken).ConfigureAwait(false); + totalRead += read; + } + + return (~seed, totalRead); + } + + private static PackedStream BuildPackedStream( + bool isLzma2, + byte[] properties, + ulong compressedSize, + ulong uncompressedSize, + uint inputCrc, + uint? outputCrc + ) + { + var methodId = isLzma2 ? CMethodId.K_LZMA2 : CMethodId.K_LZMA; + + var folder = new CFolder(); + folder._coders.Add( + new CCoderInfo + { + _methodId = methodId, + _numInStreams = 1, + _numOutStreams = 1, + _props = properties, + } + ); + folder._packStreams.Add(0); + folder._unpackSizes.Add((long)uncompressedSize); + folder._unpackCrc = inputCrc; + + return new PackedStream + { + Folder = folder, + Sizes = [compressedSize], + CRCs = [outputCrc], + }; + } +} diff --git a/src/SharpCompress/Common/SevenZip/SevenZipWriteExtensions.cs b/src/SharpCompress/Common/SevenZip/SevenZipWriteExtensions.cs new file mode 100644 index 00000000..26b8692c --- /dev/null +++ b/src/SharpCompress/Common/SevenZip/SevenZipWriteExtensions.cs @@ -0,0 +1,97 @@ +using System; +using System.IO; + +namespace SharpCompress.Common.SevenZip; + +/// +/// Stream extension methods for writing 7z binary format primitives. +/// Mirrors the read-side encoding in DataReader.ReadNumber() and the reference +/// StreamExtensions (ReadDecodedUInt64/WriteEncodedUInt64/WriteBoolVector). +/// +internal static class SevenZipWriteExtensions +{ + /// + /// Writes a variable-length encoded 64-bit unsigned integer to the stream. + /// Uses the 7z VLQ format: the first byte has leading 1-bits indicating how many + /// extra bytes follow, with remaining bits holding the high part of the value. + /// + public static int WriteEncodedUInt64(this Stream stream, ulong value) + { + var data = new byte[9]; + data[0] = 0xFF; + byte mask = 0x80; + var length = 1; + + for (var i = 0; i < 8; i++) + { + if (value < mask) + { + var headerMask = (byte)((0xFF ^ mask) ^ (mask - 1u)); + data[0] = (byte)(value | headerMask); + break; + } + + data[length++] = (byte)(value & 0xFF); + value >>= 8; + mask >>= 1; + } + + stream.Write(data, 0, length); + return length; + } + + /// + /// Writes a boolean vector as a packed bitmask. + /// Each bool becomes one bit, MSB first, padded to byte boundary. + /// + public static ulong WriteBoolVector(this Stream stream, bool[] vector) + { + byte mask = 0x80; + byte b = 0; + ulong bytesWritten = 0; + + for (var i = 0L; i < vector.LongLength; i++) + { + if (vector[i]) + { + b |= mask; + } + + mask >>= 1; + if (mask == 0) + { + stream.WriteByte(b); + bytesWritten++; + mask = 0x80; + b = 0; + } + } + + if (mask != 0x80) + { + stream.WriteByte(b); + bytesWritten++; + } + + return bytesWritten; + } + + /// + /// Writes an optional bool vector. If all elements are true, writes a single 0x01 byte + /// (AllAreDefined marker). Otherwise writes 0x00 followed by the packed bitmask. + /// + public static void WriteOptionalBoolVector(this Stream stream, bool[] vector) + { + for (var i = 0L; i < vector.LongLength; i++) + { + if (!vector[i]) + { + stream.WriteByte(0); + stream.WriteBoolVector(vector); + return; + } + } + + stream.WriteByte(1); + } +} diff --git a/src/SharpCompress/Common/SharpCompressException.cs b/src/SharpCompress/Common/SharpCompressException.cs new file mode 100644 index 00000000..7dc7cdda --- /dev/null +++ b/src/SharpCompress/Common/SharpCompressException.cs @@ -0,0 +1,59 @@ +using System; + +namespace SharpCompress.Common; + +public class SharpCompressException : Exception +{ + public SharpCompressException() { } + + public SharpCompressException(string message) + : base(message) { } + + public SharpCompressException(string message, Exception inner) + : base(message, inner) { } +} + +public class ArchiveException(string message) : SharpCompressException(message); + +public class ArchiveOperationException : SharpCompressException +{ + public ArchiveOperationException() { } + + public ArchiveOperationException(string message) + : base(message) { } + + public ArchiveOperationException(string message, Exception inner) + : base(message, inner) { } +} + +public class IncompleteArchiveException(string message) : ArchiveException(message); + +public class CryptographicException(string message) : SharpCompressException(message); + +public class ReaderCancelledException(string message) : SharpCompressException(message); + +public class ExtractionException : SharpCompressException +{ + public ExtractionException() { } + + public ExtractionException(string message) + : base(message) { } + + public ExtractionException(string message, Exception inner) + : base(message, inner) { } +} + +public class MultipartStreamRequiredException(string message) : ExtractionException(message); + +public class MultiVolumeExtractionException(string message) : ExtractionException(message); + +public class InvalidFormatException : ExtractionException +{ + public InvalidFormatException() { } + + public InvalidFormatException(string message) + : base(message) { } + + public InvalidFormatException(string message, Exception inner) + : base(message, inner) { } +} diff --git a/src/SharpCompress/Common/Tar/Headers/EntryType.cs b/src/SharpCompress/Common/Tar/Headers/EntryType.cs index 3e6877c7..cb1d6535 100644 --- a/src/SharpCompress/Common/Tar/Headers/EntryType.cs +++ b/src/SharpCompress/Common/Tar/Headers/EntryType.cs @@ -14,5 +14,6 @@ internal enum EntryType : byte LongName = (byte)'L', SparseFile = (byte)'S', VolumeHeader = (byte)'V', - GlobalExtendedHeader = (byte)'g' + LocalExtendedHeader = (byte)'x', + GlobalExtendedHeader = (byte)'g', } diff --git a/src/SharpCompress/Common/Tar/Headers/TarHeader.Async.cs b/src/SharpCompress/Common/Tar/Headers/TarHeader.Async.cs new file mode 100644 index 00000000..42f6678d --- /dev/null +++ b/src/SharpCompress/Common/Tar/Headers/TarHeader.Async.cs @@ -0,0 +1,381 @@ +using System; +using System.Buffers; +using System.Buffers.Binary; +using System.Collections.Generic; +using System.IO; +using System.IO.Compression; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.IO; + +namespace SharpCompress.Common.Tar.Headers; + +internal sealed partial class TarHeader +{ + internal async ValueTask WriteAsync( + Stream output, + CancellationToken cancellationToken = default + ) + { + switch (WriteFormat) + { + case TarHeaderWriteFormat.GNU_TAR_LONG_LINK: + await WriteGnuTarLongLinkAsync(output, cancellationToken).ConfigureAwait(false); + break; + case TarHeaderWriteFormat.USTAR: + await WriteUstarAsync(output, cancellationToken).ConfigureAwait(false); + break; + default: + throw new ArchiveOperationException("This should be impossible..."); + } + } + + private async ValueTask WriteUstarAsync(Stream output, CancellationToken cancellationToken) + { + var buffer = new byte[BLOCK_SIZE]; + + WriteOctalBytes(511, buffer, 100, 8); + WriteOctalBytes(0, buffer, 108, 8); + WriteOctalBytes(0, buffer, 116, 8); + + var nameByteCount = ArchiveEncoding + .GetEncoding() + .GetByteCount(Name.NotNull("Name is null")); + + if (nameByteCount > 100) + { + string fullName = Name.NotNull("Name is null"); + + List dirSeps = new List(); + for (int i = 0; i < fullName.Length; i++) + { + if (fullName[i] == Path.DirectorySeparatorChar) + { + dirSeps.Add(i); + } + } + + int splitIndex = -1; + for (int i = 0; i < dirSeps.Count; i++) + { +#if NET6_0_OR_GREATER + int count = ArchiveEncoding + .GetEncoding() + .GetByteCount(fullName.AsSpan(0, dirSeps[i])); +#else + int count = ArchiveEncoding + .GetEncoding() + .GetByteCount(fullName.Substring(0, dirSeps[i])); +#endif + if (count < 155) + { + splitIndex = dirSeps[i]; + } + else + { + break; + } + } + + if (splitIndex == -1) + { + throw new InvalidFormatException( + $"Tar header USTAR format can not fit file name \"{fullName}\" of length {nameByteCount}! Directory separator not found! Try using GNU Tar format instead!" + ); + } + + string namePrefix = fullName.Substring(0, splitIndex); + string name = fullName.Substring(splitIndex + 1); + + if (this.ArchiveEncoding.GetEncoding().GetByteCount(namePrefix) >= 155) + { + throw new InvalidFormatException( + $"Tar header USTAR format can not fit file name \"{fullName}\" of length {nameByteCount}! Try using GNU Tar format instead!" + ); + } + + if (this.ArchiveEncoding.GetEncoding().GetByteCount(name) >= 100) + { + throw new InvalidFormatException( + $"Tar header USTAR format can not fit file name \"{fullName}\" of length {nameByteCount}! Try using GNU Tar format instead!" + ); + } + + WriteStringBytes(ArchiveEncoding.Encode(namePrefix), buffer, 345, 100); + WriteStringBytes(ArchiveEncoding.Encode(name), buffer, 100); + } + else + { + WriteStringBytes(ArchiveEncoding.Encode(Name.NotNull("Name is null")), buffer, 100); + } + + WriteOctalBytes(Size, buffer, 124, 12); + var time = (long)(LastModifiedTime.ToUniversalTime() - EPOCH).TotalSeconds; + WriteOctalBytes(time, buffer, 136, 12); + buffer[156] = (byte)EntryType; + + WriteStringBytes(Encoding.ASCII.GetBytes("ustar"), buffer, 257, 6); + buffer[263] = 0x30; + buffer[264] = 0x30; + + var crc = RecalculateChecksum(buffer); + WriteOctalBytes(crc, buffer, 148, 8); + + await output.WriteAsync(buffer, 0, buffer.Length, cancellationToken).ConfigureAwait(false); + } + + private async ValueTask WriteGnuTarLongLinkAsync( + Stream output, + CancellationToken cancellationToken + ) + { + var buffer = new byte[BLOCK_SIZE]; + + WriteOctalBytes(511, buffer, 100, 8); + WriteOctalBytes(0, buffer, 108, 8); + WriteOctalBytes(0, buffer, 116, 8); + + var nameByteCount = ArchiveEncoding + .GetEncoding() + .GetByteCount(Name.NotNull("Name is null")); + if (nameByteCount > 100) + { + WriteStringBytes("././@LongLink", buffer, 0, 100); + buffer[156] = (byte)EntryType.LongName; + WriteOctalBytes(nameByteCount + 1, buffer, 124, 12); + } + else + { + WriteStringBytes(ArchiveEncoding.Encode(Name.NotNull("Name is null")), buffer, 100); + WriteOctalBytes(Size, buffer, 124, 12); + var time = (long)(LastModifiedTime.ToUniversalTime() - EPOCH).TotalSeconds; + WriteOctalBytes(time, buffer, 136, 12); + buffer[156] = (byte)EntryType; + + if (Size >= 0x1FFFFFFFF) + { + Span bytes12 = stackalloc byte[12]; + BinaryPrimitives.WriteInt64BigEndian(bytes12.Slice(4), Size); + bytes12[0] |= 0x80; + bytes12.CopyTo(buffer.AsSpan(124)); + } + } + + var crc = RecalculateChecksum(buffer); + WriteOctalBytes(crc, buffer, 148, 8); + + await output.WriteAsync(buffer, 0, buffer.Length, cancellationToken).ConfigureAwait(false); + + if (nameByteCount > 100) + { + await WriteLongFilenameHeaderAsync(output, cancellationToken).ConfigureAwait(false); + Name = ArchiveEncoding.Decode( + ArchiveEncoding.Encode(Name.NotNull("Name is null")), + 0, + 100 - ArchiveEncoding.GetEncoding().GetMaxByteCount(1) + ); + await WriteGnuTarLongLinkAsync(output, cancellationToken).ConfigureAwait(false); + } + } + + private async ValueTask WriteLongFilenameHeaderAsync( + Stream output, + CancellationToken cancellationToken + ) + { + var nameBytes = ArchiveEncoding.Encode(Name.NotNull("Name is null")); + await output + .WriteAsync(nameBytes, 0, nameBytes.Length, cancellationToken) + .ConfigureAwait(false); + + var numPaddingBytes = BLOCK_SIZE - (nameBytes.Length % BLOCK_SIZE); + if (numPaddingBytes == 0) + { + numPaddingBytes = BLOCK_SIZE; + } + + await output + .WriteAsync(new byte[numPaddingBytes], 0, numPaddingBytes, cancellationToken) + .ConfigureAwait(false); + } + + internal async ValueTask ReadAsync( + AsyncBinaryReader reader, + PaxMetadata? globalPaxMetadata = null + ) + { + globalPaxMetadata ??= new PaxMetadata(); + var pendingMetadata = globalPaxMetadata.Clone(); + var buffer = ArrayPool.Shared.Rent(BLOCK_SIZE); + EntryType entryType; + try + { + while (true) + { + await reader.ReadBytesAsync(buffer, 0, BLOCK_SIZE).ConfigureAwait(false); + entryType = ReadEntryType(buffer); + + // LongName and LongLink headers can follow each other and need + // to apply to the header that follows them. + if (entryType == EntryType.LongName) + { + pendingMetadata.Name = await ReadLongNameAsync(reader, buffer) + .ConfigureAwait(false); + continue; + } + + if (entryType == EntryType.LongLink) + { + pendingMetadata.LinkName = await ReadLongNameAsync(reader, buffer) + .ConfigureAwait(false); + continue; + } + + if (entryType == EntryType.LocalExtendedHeader) + { + await ReadPaxMetadataAsync(reader, buffer, pendingMetadata) + .ConfigureAwait(false); + continue; + } + + if (entryType == EntryType.GlobalExtendedHeader) + { + await ReadPaxMetadataAsync(reader, buffer, globalPaxMetadata) + .ConfigureAwait(false); + pendingMetadata = globalPaxMetadata.Clone(); + continue; + } + + break; + } + } + finally + { + ArrayPool.Shared.Return(buffer); + } + + // Check header checksum + if (!checkChecksum(buffer)) + { + return false; + } + + Name = ArchiveEncoding.Decode(buffer, 0, 100).TrimNulls(); + EntryType = entryType; + Size = ReadSize(buffer); + LinkName = null; + + // for symlinks, additionally read the linkname + if (entryType == EntryType.SymLink || entryType == EntryType.HardLink) + { + LinkName = ArchiveEncoding.Decode(buffer, 157, 100).TrimNulls(); + } + + Mode = ReadAsciiInt64Base8(buffer, 100, 7); + UserId = ReadAsciiInt64Base8oldGnu(buffer, 108, 7); + GroupId = ReadAsciiInt64Base8oldGnu(buffer, 116, 7); + + var unixTimeStamp = ReadAsciiInt64Base8(buffer, 136, 11); + + LastModifiedTime = EPOCH.AddSeconds(unixTimeStamp).ToLocalTime(); + Magic = ArchiveEncoding.Decode(buffer, 257, 6).TrimNulls(); + + if (!string.IsNullOrEmpty(Magic) && "ustar".Equals(Magic, StringComparison.Ordinal)) + { + var namePrefix = ArchiveEncoding.Decode(buffer, 345, 157).TrimNulls(); + + if (!string.IsNullOrEmpty(namePrefix)) + { + Name = namePrefix + "/" + Name; + } + } + + pendingMetadata.ApplyTo(this); + + if (entryType == EntryType.Directory) + { + Mode |= 0b1_000_000_000; + } + + if (entryType != EntryType.LongName && Name.Length == 0) + { + return false; + } + + return true; + } + + private static async ValueTask ReadLengthAsync(AsyncBinaryReader reader, int length) + { + var buffer = ArrayPool.Shared.Rent(length); + try + { + await reader.ReadBytesAsync(buffer, 0, length).ConfigureAwait(false); + } + finally + { + ArrayPool.Shared.Return(buffer); + } + } + + private async ValueTask ReadLongNameAsync(AsyncBinaryReader reader, byte[] buffer) + { + var nameBytes = await ReadMetadataPayloadAsync( + reader, + buffer, + MAX_LONG_NAME_SIZE, + "Long name" + ) + .ConfigureAwait(false); + + return ArchiveEncoding.Decode(nameBytes, 0, nameBytes.Length).TrimNulls(); + } + + private async ValueTask ReadPaxMetadataAsync( + AsyncBinaryReader reader, + byte[] buffer, + PaxMetadata pendingMetadata + ) + { + var payload = await ReadMetadataPayloadAsync( + reader, + buffer, + MAX_PAX_HEADER_SIZE, + "PAX header" + ) + .ConfigureAwait(false); + + ParsePaxRecords(payload, pendingMetadata); + } + + private async ValueTask ReadMetadataPayloadAsync( + AsyncBinaryReader reader, + byte[] buffer, + int maxSize, + string payloadName + ) + { + var size = ReadSize(buffer); + + // Validate size to prevent memory exhaustion from malformed headers + if (size < 0 || size > maxSize) + { + throw new InvalidFormatException( + $"{payloadName} size {size} is invalid or exceeds maximum allowed size of {maxSize} bytes" + ); + } + + var payloadLength = (int)size; + var payload = new byte[payloadLength]; + await reader.ReadBytesAsync(payload, 0, payloadLength).ConfigureAwait(false); + + var paddingLength = GetPaddingLength(payloadLength); + if (paddingLength > 0) + { + await ReadLengthAsync(reader, paddingLength).ConfigureAwait(false); + } + + return payload; + } +} diff --git a/src/SharpCompress/Common/Tar/Headers/TarHeader.cs b/src/SharpCompress/Common/Tar/Headers/TarHeader.cs index ef86ab00..0239b118 100644 --- a/src/SharpCompress/Common/Tar/Headers/TarHeader.cs +++ b/src/SharpCompress/Common/Tar/Headers/TarHeader.cs @@ -1,20 +1,31 @@ -#nullable disable - using System; +using System.Buffers; using System.Buffers.Binary; +using System.Collections.Generic; +using System.Globalization; using System.IO; +using System.IO.Compression; using System.Text; +using System.Threading.Tasks; namespace SharpCompress.Common.Tar.Headers; -internal sealed class TarHeader +internal sealed partial class TarHeader { - internal static readonly DateTime EPOCH = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc); + internal static readonly DateTime EPOCH = new(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc); - public TarHeader(ArchiveEncoding archiveEncoding) => ArchiveEncoding = archiveEncoding; + public TarHeader( + IArchiveEncoding archiveEncoding, + TarHeaderWriteFormat writeFormat = TarHeaderWriteFormat.GNU_TAR_LONG_LINK + ) + { + ArchiveEncoding = archiveEncoding; + WriteFormat = writeFormat; + } - internal string Name { get; set; } - internal string LinkName { get; set; } + internal TarHeaderWriteFormat WriteFormat { get; set; } + internal string? Name { get; set; } + internal string? LinkName { get; set; } internal long Mode { get; set; } internal long UserId { get; set; } @@ -22,12 +33,93 @@ internal sealed class TarHeader internal long Size { get; set; } internal DateTime LastModifiedTime { get; set; } internal EntryType EntryType { get; set; } - internal Stream PackedStream { get; set; } - internal ArchiveEncoding ArchiveEncoding { get; } + internal Stream? PackedStream { get; set; } + internal IArchiveEncoding ArchiveEncoding { get; } internal const int BLOCK_SIZE = 512; + // Maximum size for long name/link headers to prevent memory exhaustion attacks + // This is generous enough for most real-world scenarios (32KB) + private const int MAX_LONG_NAME_SIZE = 32768; + private const int MAX_PAX_HEADER_SIZE = 65536; + + internal sealed class PaxMetadata + { + internal string? Name { get; set; } + internal string? LinkName { get; set; } + internal long? Mode { get; set; } + internal long? UserId { get; set; } + internal long? GroupId { get; set; } + internal long? Size { get; set; } + internal DateTime? LastModifiedTime { get; set; } + + internal PaxMetadata Clone() => + new() + { + Name = Name, + LinkName = LinkName, + Mode = Mode, + UserId = UserId, + GroupId = GroupId, + Size = Size, + LastModifiedTime = LastModifiedTime, + }; + + internal void ApplyTo(TarHeader header) + { + if (Name is not null) + { + header.Name = Name; + } + + if (LinkName is not null) + { + header.LinkName = LinkName; + } + + if (Size.HasValue) + { + header.Size = Size.Value; + } + + if (LastModifiedTime.HasValue) + { + header.LastModifiedTime = LastModifiedTime.Value; + } + + if (Mode.HasValue) + { + header.Mode = Mode.Value; + } + + if (UserId.HasValue) + { + header.UserId = UserId.Value; + } + + if (GroupId.HasValue) + { + header.GroupId = GroupId.Value; + } + } + } + internal void Write(Stream output) + { + switch (WriteFormat) + { + case TarHeaderWriteFormat.GNU_TAR_LONG_LINK: + WriteGnuTarLongLink(output); + break; + case TarHeaderWriteFormat.USTAR: + WriteUstar(output); + break; + default: + throw new ArchiveOperationException("This should be impossible..."); + } + } + + internal void WriteUstar(Stream output) { var buffer = new byte[BLOCK_SIZE]; @@ -36,7 +128,112 @@ internal sealed class TarHeader WriteOctalBytes(0, buffer, 116, 8); // group ID //ArchiveEncoding.UTF8.GetBytes("magic").CopyTo(buffer, 257); - var nameByteCount = ArchiveEncoding.GetEncoding().GetByteCount(Name); + var nameByteCount = ArchiveEncoding + .GetEncoding() + .GetByteCount(Name.NotNull("Name is null")); + + if (nameByteCount > 100) + { + // if name is longer, try to split it into name and namePrefix + + string fullName = Name.NotNull("Name is null"); + + // find all directory separators + List dirSeps = new List(); + for (int i = 0; i < fullName.Length; i++) + { + if (fullName[i] == Path.DirectorySeparatorChar) + { + dirSeps.Add(i); + } + } + + // find the right place to split the name + int splitIndex = -1; + for (int i = 0; i < dirSeps.Count; i++) + { +#if NET6_0_OR_GREATER + int count = ArchiveEncoding + .GetEncoding() + .GetByteCount(fullName.AsSpan(0, dirSeps[i])); +#else + int count = ArchiveEncoding + .GetEncoding() + .GetByteCount(fullName.Substring(0, dirSeps[i])); +#endif + if (count < 155) + { + splitIndex = dirSeps[i]; + } + else + { + break; + } + } + + if (splitIndex == -1) + { + throw new InvalidFormatException( + $"Tar header USTAR format can not fit file name \"{fullName}\" of length {nameByteCount}! Directory separator not found! Try using GNU Tar format instead!" + ); + } + + string namePrefix = fullName.Substring(0, splitIndex); + string name = fullName.Substring(splitIndex + 1); + + if (this.ArchiveEncoding.GetEncoding().GetByteCount(namePrefix) >= 155) + { + throw new InvalidFormatException( + $"Tar header USTAR format can not fit file name \"{fullName}\" of length {nameByteCount}! Try using GNU Tar format instead!" + ); + } + + if (this.ArchiveEncoding.GetEncoding().GetByteCount(name) >= 100) + { + throw new InvalidFormatException( + $"Tar header USTAR format can not fit file name \"{fullName}\" of length {nameByteCount}! Try using GNU Tar format instead!" + ); + } + + // write name prefix + WriteStringBytes(ArchiveEncoding.Encode(namePrefix), buffer, 345, 100); + // write partial name + WriteStringBytes(ArchiveEncoding.Encode(name), buffer, 100); + } + else + { + WriteStringBytes(ArchiveEncoding.Encode(Name.NotNull("Name is null")), buffer, 100); + } + + WriteOctalBytes(Size, buffer, 124, 12); + var time = (long)(LastModifiedTime.ToUniversalTime() - EPOCH).TotalSeconds; + WriteOctalBytes(time, buffer, 136, 12); + buffer[156] = (byte)EntryType; + + // write ustar magic field + WriteStringBytes(Encoding.ASCII.GetBytes("ustar"), buffer, 257, 6); + // write ustar version "00" + buffer[263] = 0x30; + buffer[264] = 0x30; + + var crc = RecalculateChecksum(buffer); + WriteOctalBytes(crc, buffer, 148, 8); + + output.Write(buffer, 0, buffer.Length); + } + + internal void WriteGnuTarLongLink(Stream output) + { + var buffer = new byte[BLOCK_SIZE]; + + WriteOctalBytes(511, buffer, 100, 8); // file mode + WriteOctalBytes(0, buffer, 108, 8); // owner ID + WriteOctalBytes(0, buffer, 116, 8); // group ID + + //ArchiveEncoding.UTF8.GetBytes("magic").CopyTo(buffer, 257); + var nameByteCount = ArchiveEncoding + .GetEncoding() + .GetByteCount(Name.NotNull("Name is null")); if (nameByteCount > 100) { // Set mock filename and filetype to indicate the next block is the actual name of the file @@ -46,7 +243,7 @@ internal sealed class TarHeader } else { - WriteStringBytes(ArchiveEncoding.Encode(Name), buffer, 100); + WriteStringBytes(ArchiveEncoding.Encode(Name.NotNull("Name is null")), buffer, 100); WriteOctalBytes(Size, buffer, 124, 12); var time = (long)(LastModifiedTime.ToUniversalTime() - EPOCH).TotalSeconds; WriteOctalBytes(time, buffer, 136, 12); @@ -77,17 +274,17 @@ internal sealed class TarHeader // // and then infinite recursion is occured in WriteLongFilenameHeader because truncated.Length is 102. Name = ArchiveEncoding.Decode( - ArchiveEncoding.Encode(Name), + ArchiveEncoding.Encode(Name.NotNull("Name is null")), 0, 100 - ArchiveEncoding.GetEncoding().GetMaxByteCount(1) ); - Write(output); + WriteGnuTarLongLink(output); } } private void WriteLongFilenameHeader(Stream output) { - var nameBytes = ArchiveEncoding.Encode(Name); + var nameBytes = ArchiveEncoding.Encode(Name.NotNull("Name is null")); output.Write(nameBytes, 0, nameBytes.Length); // pad to multiple of BlockSize bytes, and make sure a terminating null is added @@ -99,77 +296,339 @@ internal sealed class TarHeader output.Write(stackalloc byte[numPaddingBytes]); } - internal bool Read(BinaryReader reader) + internal bool Read(BinaryReader reader, PaxMetadata? globalPaxMetadata = null) { - var buffer = ReadBlock(reader); - if (buffer.Length == 0) + globalPaxMetadata ??= new PaxMetadata(); + var pendingMetadata = globalPaxMetadata.Clone(); + byte[] buffer; + EntryType entryType; + + while (true) + { + buffer = ReadBlock(reader); + + if (buffer.Length == 0) + { + return false; + } + + entryType = ReadEntryType(buffer); + + // LongName and LongLink headers can follow each other and need + // to apply to the header that follows them. + if (entryType == EntryType.LongName) + { + pendingMetadata.Name = ReadLongName(reader, buffer); + continue; + } + + if (entryType == EntryType.LongLink) + { + pendingMetadata.LinkName = ReadLongName(reader, buffer); + continue; + } + + if (entryType == EntryType.LocalExtendedHeader) + { + ReadPaxMetadata(reader, buffer, pendingMetadata); + continue; + } + + if (entryType == EntryType.GlobalExtendedHeader) + { + ReadPaxMetadata(reader, buffer, globalPaxMetadata); + pendingMetadata = globalPaxMetadata.Clone(); + continue; + } + + break; + } + + // Check header checksum + if (!checkChecksum(buffer)) { return false; } + Name = ArchiveEncoding.Decode(buffer, 0, 100).TrimNulls(); + EntryType = entryType; + Size = ReadSize(buffer); + LinkName = null; + // for symlinks, additionally read the linkname - if (ReadEntryType(buffer) == EntryType.SymLink) + if (entryType == EntryType.SymLink || entryType == EntryType.HardLink) { LinkName = ArchiveEncoding.Decode(buffer, 157, 100).TrimNulls(); } - if (ReadEntryType(buffer) == EntryType.LongName) - { - Name = ReadLongName(reader, buffer); - buffer = ReadBlock(reader); - } - else - { - Name = ArchiveEncoding.Decode(buffer, 0, 100).TrimNulls(); - } - - EntryType = ReadEntryType(buffer); - Size = ReadSize(buffer); - Mode = ReadAsciiInt64Base8(buffer, 100, 7); - if (EntryType == EntryType.Directory) - { - Mode |= 0b1_000_000_000; - } + UserId = ReadAsciiInt64Base8oldGnu(buffer, 108, 7); + GroupId = ReadAsciiInt64Base8oldGnu(buffer, 116, 7); - UserId = ReadAsciiInt64Base8(buffer, 108, 7); - GroupId = ReadAsciiInt64Base8(buffer, 116, 7); var unixTimeStamp = ReadAsciiInt64Base8(buffer, 136, 11); - LastModifiedTime = EPOCH.AddSeconds(unixTimeStamp).ToLocalTime(); + LastModifiedTime = EPOCH.AddSeconds(unixTimeStamp).ToLocalTime(); Magic = ArchiveEncoding.Decode(buffer, 257, 6).TrimNulls(); - if (!string.IsNullOrEmpty(Magic) && "ustar".Equals(Magic)) + if (!string.IsNullOrEmpty(Magic) && "ustar".Equals(Magic, StringComparison.Ordinal)) { - var namePrefix = ArchiveEncoding.Decode(buffer, 345, 157); - namePrefix = namePrefix.TrimNulls(); + var namePrefix = ArchiveEncoding.Decode(buffer, 345, 157).TrimNulls(); + if (!string.IsNullOrEmpty(namePrefix)) { Name = namePrefix + "/" + Name; } } - if (EntryType != EntryType.LongName && Name.Length == 0) + + pendingMetadata.ApplyTo(this); + + if (entryType == EntryType.Directory) + { + Mode |= 0b1_000_000_000; + } + + if (entryType != EntryType.LongName && Name.Length == 0) { return false; } + return true; } private string ReadLongName(BinaryReader reader, byte[] buffer) { - var size = ReadSize(buffer); - var nameLength = (int)size; - var nameBytes = reader.ReadBytes(nameLength); - var remainingBytesToRead = BLOCK_SIZE - (nameLength % BLOCK_SIZE); - - // Read the rest of the block and discard the data - if (remainingBytesToRead < BLOCK_SIZE) - { - reader.ReadBytes(remainingBytesToRead); - } + var nameBytes = ReadMetadataPayload(reader, buffer, MAX_LONG_NAME_SIZE, "Long name"); return ArchiveEncoding.Decode(nameBytes, 0, nameBytes.Length).TrimNulls(); } + private void ReadPaxMetadata(BinaryReader reader, byte[] buffer, PaxMetadata pendingMetadata) + { + var payload = ReadMetadataPayload(reader, buffer, MAX_PAX_HEADER_SIZE, "PAX header"); + ParsePaxRecords(payload, pendingMetadata); + } + + private byte[] ReadMetadataPayload( + BinaryReader reader, + byte[] buffer, + int maxSize, + string payloadName + ) + { + var size = ReadSize(buffer); + + // Validate size to prevent memory exhaustion from malformed headers + if (size < 0 || size > maxSize) + { + throw new InvalidFormatException( + $"{payloadName} size {size} is invalid or exceeds maximum allowed size of {maxSize} bytes" + ); + } + + var payloadLength = (int)size; + var payload = reader.ReadBytes(payloadLength); + + if (payload.Length != payloadLength) + { + throw new InvalidFormatException($"{payloadName} data is truncated."); + } + + SkipMetadataPadding(reader, payloadLength); + return payload; + } + + private static void SkipMetadataPadding(BinaryReader reader, int payloadLength) + { + var paddingLength = GetPaddingLength(payloadLength); + if (paddingLength == 0) + { + return; + } + + var padding = reader.ReadBytes(paddingLength); + if (padding.Length != paddingLength) + { + throw new InvalidFormatException("Metadata payload padding is truncated."); + } + } + + private static int GetPaddingLength(int payloadLength) + { + var remainder = payloadLength % BLOCK_SIZE; + return remainder == 0 ? 0 : BLOCK_SIZE - remainder; + } + + private static void ParsePaxRecords(byte[] payload, PaxMetadata pendingMetadata) + { + var index = 0; + while (index < payload.Length) + { + var spaceIndex = Array.IndexOf(payload, (byte)' ', index); + if (spaceIndex <= index) + { + throw new InvalidFormatException("Invalid PAX record: missing length separator."); + } + + var recordLength = ParsePaxRecordLength(payload, index, spaceIndex - index); + if (recordLength <= 0 || recordLength > payload.Length - index) + { + throw new InvalidFormatException( + "Invalid PAX record: record length exceeds payload." + ); + } + + var recordEnd = index + recordLength; + if (payload[recordEnd - 1] != (byte)'\n') + { + throw new InvalidFormatException( + "Invalid PAX record: record does not end with newline." + ); + } + + var keyValueStart = spaceIndex + 1; + var keyValueLength = recordEnd - keyValueStart - 1; + var equalsIndex = Array.IndexOf(payload, (byte)'=', keyValueStart, keyValueLength); + if (equalsIndex <= keyValueStart) + { + throw new InvalidFormatException( + "Invalid PAX record: missing key/value separator." + ); + } + + var key = Encoding.UTF8.GetString(payload, keyValueStart, equalsIndex - keyValueStart); + var valueStart = equalsIndex + 1; + var valueLength = recordEnd - valueStart - 1; + var value = Encoding.UTF8.GetString(payload, valueStart, valueLength); + + ApplyPaxKeyValue(pendingMetadata, key, value); + index = recordEnd; + } + } + + private static int ParsePaxRecordLength(byte[] payload, int offset, int length) + { + var lengthText = Encoding.ASCII.GetString(payload, offset, length); + if ( + !int.TryParse( + lengthText, + NumberStyles.None, + CultureInfo.InvariantCulture, + out var value + ) + ) + { + throw new InvalidFormatException($"Invalid PAX record length '{lengthText}'."); + } + + if (value <= 0) + { + throw new InvalidFormatException("Invalid PAX record length: value must be positive."); + } + + return value; + } + + private static void ApplyPaxKeyValue(PaxMetadata pendingMetadata, string key, string value) + { + switch (key) + { + case "path": + pendingMetadata.Name = value; + break; + case "linkpath": + pendingMetadata.LinkName = value; + break; + case "size": + pendingMetadata.Size = ParsePaxInt64(value, key, allowNegative: false); + break; + case "mtime": + pendingMetadata.LastModifiedTime = ParsePaxTimestamp(value, key); + break; + case "uid": + pendingMetadata.UserId = ParsePaxInt64(value, key); + break; + case "gid": + pendingMetadata.GroupId = ParsePaxInt64(value, key); + break; + case "mode": + pendingMetadata.Mode = ParsePaxMode(value); + break; + } + } + + private static long ParsePaxMode(string value) + { + if (string.IsNullOrWhiteSpace(value)) + { + throw new InvalidFormatException("Invalid PAX value for 'mode': value is empty."); + } + + if (IsOctalDigitsOnly(value)) + { + return Convert.ToInt64(value, 8); + } + + return ParsePaxInt64(value, "mode", allowNegative: false); + } + + private static bool IsOctalDigitsOnly(string value) + { + foreach (var ch in value) + { + if (ch < '0' || ch > '7') + { + return false; + } + } + + return value.Length > 0; + } + + private static long ParsePaxInt64(string value, string key, bool allowNegative = true) + { + if ( + !long.TryParse( + value, + NumberStyles.Integer, + CultureInfo.InvariantCulture, + out var parsed + ) + ) + { + throw new InvalidFormatException($"Invalid PAX value for '{key}': '{value}'."); + } + + if (!allowNegative && parsed < 0) + { + throw new InvalidFormatException($"Invalid PAX value for '{key}': '{value}'."); + } + + return parsed; + } + + private static DateTime ParsePaxTimestamp(string value, string key) + { + if ( + !double.TryParse( + value, + NumberStyles.Float, + CultureInfo.InvariantCulture, + out var seconds + ) + ) + { + throw new InvalidFormatException($"Invalid PAX value for '{key}': '{value}'."); + } + + try + { + return EPOCH.AddSeconds(seconds).ToLocalTime(); + } + catch (ArgumentOutOfRangeException ex) + { + throw new InvalidFormatException($"Invalid PAX value for '{key}': '{value}'.", ex); + } + } + private static EntryType ReadEntryType(byte[] buffer) => (EntryType)buffer[156]; private long ReadSize(byte[] buffer) @@ -188,7 +647,7 @@ internal sealed class TarHeader if (buffer.Length != 0 && buffer.Length < BLOCK_SIZE) { - throw new InvalidOperationException("Buffer is invalid size"); + throw new InvalidFormatException("Buffer is invalid size"); } return buffer; } @@ -200,6 +659,18 @@ internal sealed class TarHeader buffer.Slice(i, length - i).Clear(); } + private static void WriteStringBytes( + ReadOnlySpan name, + Span buffer, + int offset, + int length + ) + { + name.CopyTo(buffer.Slice(offset)); + var i = Math.Min(length, name.Length); + buffer.Slice(offset + i, length - i).Clear(); + } + private static void WriteStringBytes(string name, byte[] buffer, int offset, int length) { int i; @@ -249,6 +720,24 @@ internal sealed class TarHeader return Convert.ToInt64(s, 8); } + private static long ReadAsciiInt64Base8oldGnu(byte[] buffer, int offset, int count) + { + if (buffer[offset] == 0x80 && buffer[offset + 1] == 0x00) + { + return buffer[offset + 4] << 24 + | buffer[offset + 5] << 16 + | buffer[offset + 6] << 8 + | buffer[offset + 7]; + } + var s = Encoding.UTF8.GetString(buffer, offset, count).TrimNulls(); + + if (string.IsNullOrEmpty(s)) + { + return 0; + } + return Convert.ToInt64(s, 8); + } + private static long ReadAsciiInt64(byte[] buffer, int offset, int count) { var s = Encoding.UTF8.GetString(buffer, offset, count).TrimNulls(); @@ -256,7 +745,7 @@ internal sealed class TarHeader { return 0; } - return Convert.ToInt64(s); + return Convert.ToInt64(s, Constants.DefaultCultureInfo); } private static readonly byte[] eightSpaces = @@ -268,9 +757,45 @@ internal sealed class TarHeader (byte)' ', (byte)' ', (byte)' ', - (byte)' ' + (byte)' ', }; + internal static bool checkChecksum(byte[] buf) + { + const int eightSpacesChksum = 256; + var buffer = new Span(buf).Slice(0, 512); + int posix_sum = eightSpacesChksum; + int sun_sum = eightSpacesChksum; + + foreach (byte b in buffer) + { + posix_sum += b; + sun_sum += unchecked((sbyte)b); + } + + // Special case, empty file header + if (posix_sum == eightSpacesChksum) + { + return true; + } + + // Remove current checksum from calculation + foreach (byte b in buffer.Slice(148, 8)) + { + posix_sum -= b; + sun_sum -= unchecked((sbyte)b); + } + + // Read and compare checksum for header + var crc = ReadAsciiInt64Base8(buf, 148, 7); + if (crc != posix_sum && crc != sun_sum) + { + return false; + } + + return true; + } + internal static int RecalculateChecksum(byte[] buf) { // Set default value for checksum. That is 8 spaces. @@ -305,5 +830,5 @@ internal sealed class TarHeader public long? DataStartPosition { get; set; } - public string Magic { get; set; } + public string? Magic { get; set; } } diff --git a/src/SharpCompress/Common/Tar/Headers/TarHeaderWriteFormat.cs b/src/SharpCompress/Common/Tar/Headers/TarHeaderWriteFormat.cs new file mode 100644 index 00000000..3a3a434a --- /dev/null +++ b/src/SharpCompress/Common/Tar/Headers/TarHeaderWriteFormat.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Common.Tar.Headers; + +public enum TarHeaderWriteFormat +{ + GNU_TAR_LONG_LINK, + USTAR, +} diff --git a/src/SharpCompress/Common/Tar/TarEntry.Async.cs b/src/SharpCompress/Common/Tar/TarEntry.Async.cs new file mode 100644 index 00000000..c066da56 --- /dev/null +++ b/src/SharpCompress/Common/Tar/TarEntry.Async.cs @@ -0,0 +1,47 @@ +using System.Collections.Generic; +using System.IO; +using SharpCompress.Common.Options; +using SharpCompress.IO; + +namespace SharpCompress.Common.Tar; + +public partial class TarEntry +{ + internal static async IAsyncEnumerable GetEntriesAsync( + StreamingMode mode, + Stream stream, + CompressionType compressionType, + IArchiveEncoding archiveEncoding, + IReaderOptions readerOptions + ) + { + await foreach ( + var header in TarHeaderFactory.ReadHeaderAsync(mode, stream, archiveEncoding) + ) + { + if (header != null) + { + if (mode == StreamingMode.Seekable) + { + yield return new TarEntry( + new TarFilePart(header, stream), + compressionType, + readerOptions + ); + } + else + { + yield return new TarEntry( + new TarFilePart(header, null), + compressionType, + readerOptions + ); + } + } + else + { + throw new IncompleteArchiveException("Unexpected EOF reading tar file"); + } + } + } +} diff --git a/src/SharpCompress/Common/Tar/TarEntry.cs b/src/SharpCompress/Common/Tar/TarEntry.cs index 59743070..28e7d45f 100644 --- a/src/SharpCompress/Common/Tar/TarEntry.cs +++ b/src/SharpCompress/Common/Tar/TarEntry.cs @@ -1,18 +1,18 @@ -#nullable disable - using System; using System.Collections.Generic; using System.IO; +using SharpCompress.Common.Options; using SharpCompress.Common.Tar.Headers; using SharpCompress.IO; namespace SharpCompress.Common.Tar; -public class TarEntry : Entry +public partial class TarEntry : Entry { - private readonly TarFilePart _filePart; + private readonly TarFilePart? _filePart; - internal TarEntry(TarFilePart filePart, CompressionType type) + internal TarEntry(TarFilePart? filePart, CompressionType type, IReaderOptions readerOptions) + : base(readerOptions) { _filePart = filePart; CompressionType = type; @@ -22,15 +22,15 @@ public class TarEntry : Entry public override long Crc => 0; - public override string Key => _filePart.Header.Name; + public override string? Key => _filePart?.Header.Name; - public override string LinkTarget => _filePart.Header.LinkName; + public override string? LinkTarget => _filePart?.Header.LinkName; - public override long CompressedSize => _filePart.Header.Size; + public override long CompressedSize => _filePart?.Header.Size ?? 0; - public override long Size => _filePart.Header.Size; + public override long Size => _filePart?.Header.Size ?? 0; - public override DateTime? LastModifiedTime => _filePart.Header.LastModifiedTime; + public override DateTime? LastModifiedTime => _filePart?.Header.LastModifiedTime; public override DateTime? CreatedTime => null; @@ -40,36 +40,45 @@ public class TarEntry : Entry public override bool IsEncrypted => false; - public override bool IsDirectory => _filePart.Header.EntryType == EntryType.Directory; + public override bool IsDirectory => _filePart?.Header.EntryType == EntryType.Directory; public override bool IsSplitAfter => false; - public long Mode => _filePart.Header.Mode; + public long Mode => _filePart?.Header.Mode ?? 0; - public long UserID => _filePart.Header.UserId; + public long UserID => _filePart?.Header.UserId ?? 0; - public long GroupId => _filePart.Header.GroupId; + public long GroupId => _filePart?.Header.GroupId ?? 0; - internal override IEnumerable Parts => _filePart.AsEnumerable(); + internal override IEnumerable Parts => _filePart.Empty(); internal static IEnumerable GetEntries( StreamingMode mode, Stream stream, CompressionType compressionType, - ArchiveEncoding archiveEncoding + IArchiveEncoding archiveEncoding, + IReaderOptions readerOptions ) { - foreach (var h in TarHeaderFactory.ReadHeader(mode, stream, archiveEncoding)) + foreach (var header in TarHeaderFactory.ReadHeader(mode, stream, archiveEncoding)) { - if (h != null) + if (header != null) { if (mode == StreamingMode.Seekable) { - yield return new TarEntry(new TarFilePart(h, stream), compressionType); + yield return new TarEntry( + new TarFilePart(header, stream), + compressionType, + readerOptions + ); } else { - yield return new TarEntry(new TarFilePart(h, null), compressionType); + yield return new TarEntry( + new TarFilePart(header, null), + compressionType, + readerOptions + ); } } else @@ -78,4 +87,6 @@ public class TarEntry : Entry } } } + + // Async methods moved to TarEntry.Async.cs } diff --git a/src/SharpCompress/Common/Tar/TarFilePart.cs b/src/SharpCompress/Common/Tar/TarFilePart.cs index bfd2fe29..eb2f4f33 100644 --- a/src/SharpCompress/Common/Tar/TarFilePart.cs +++ b/src/SharpCompress/Common/Tar/TarFilePart.cs @@ -1,13 +1,15 @@ using System.IO; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common.Tar.Headers; namespace SharpCompress.Common.Tar; internal sealed class TarFilePart : FilePart { - private readonly Stream _seekableStream; + private readonly Stream? _seekableStream; - internal TarFilePart(TarHeader header, Stream seekableStream) + internal TarFilePart(TarHeader header, Stream? seekableStream) : base(header.ArchiveEncoding) { _seekableStream = seekableStream; @@ -16,16 +18,28 @@ internal sealed class TarFilePart : FilePart internal TarHeader Header { get; } - internal override string FilePartName => Header.Name; + internal override string? FilePartName => Header?.Name; internal override Stream GetCompressedStream() { if (_seekableStream != null) { - _seekableStream.Position = Header.DataStartPosition!.Value; + _seekableStream.Position = Header.DataStartPosition ?? 0; return new TarReadOnlySubStream(_seekableStream, Header.Size); } - return Header.PackedStream; + return Header.PackedStream.NotNull(); + } + + internal override ValueTask GetCompressedStreamAsync( + CancellationToken cancellationToken = default + ) + { + if (_seekableStream != null) + { + _seekableStream.Position = Header.DataStartPosition ?? 0; + return new ValueTask(new TarReadOnlySubStream(_seekableStream, Header.Size)); + } + return new ValueTask(Header.PackedStream.NotNull()); } internal override Stream? GetRawStream() => null; diff --git a/src/SharpCompress/Common/Tar/TarHeaderFactory.Async.cs b/src/SharpCompress/Common/Tar/TarHeaderFactory.Async.cs new file mode 100644 index 00000000..c011abd2 --- /dev/null +++ b/src/SharpCompress/Common/Tar/TarHeaderFactory.Async.cs @@ -0,0 +1,63 @@ +using System.Collections.Generic; +using System.IO; +using System.Threading.Tasks; +using SharpCompress.Common.Tar.Headers; +using SharpCompress.IO; + +namespace SharpCompress.Common.Tar; + +internal static partial class TarHeaderFactory +{ + internal static async IAsyncEnumerable ReadHeaderAsync( + StreamingMode mode, + Stream stream, + IArchiveEncoding archiveEncoding + ) + { +#if NET8_0_OR_GREATER + await using var reader = new AsyncBinaryReader(stream, leaveOpen: true); +#else + using var reader = new AsyncBinaryReader(stream, leaveOpen: true); +#endif + + var globalPaxMetadata = new TarHeader.PaxMetadata(); + + while (true) + { + TarHeader? header = null; + try + { + header = new TarHeader(archiveEncoding); + if (!await header.ReadAsync(reader, globalPaxMetadata).ConfigureAwait(false)) + { + yield break; + } + switch (mode) + { + case StreamingMode.Seekable: + { + header.DataStartPosition = stream.Position; + + //skip to nearest 512 + stream.Position += PadTo512(header.Size); + } + break; + case StreamingMode.Streaming: + { + header.PackedStream = new TarReadOnlySubStream(stream, header.Size); + } + break; + default: + { + throw new InvalidFormatException("Invalid StreamingMode"); + } + } + } + catch + { + header = null; + } + yield return header; + } + } +} diff --git a/src/SharpCompress/Common/Tar/TarHeaderFactory.cs b/src/SharpCompress/Common/Tar/TarHeaderFactory.cs index 85eec5b4..c95efaef 100644 --- a/src/SharpCompress/Common/Tar/TarHeaderFactory.cs +++ b/src/SharpCompress/Common/Tar/TarHeaderFactory.cs @@ -5,39 +5,38 @@ using SharpCompress.IO; namespace SharpCompress.Common.Tar; -internal static class TarHeaderFactory +internal static partial class TarHeaderFactory { internal static IEnumerable ReadHeader( StreamingMode mode, Stream stream, - ArchiveEncoding archiveEncoding + IArchiveEncoding archiveEncoding ) { + var globalPaxMetadata = new TarHeader.PaxMetadata(); while (true) { TarHeader? header = null; try { - var reader = new BinaryReader(stream); + var reader = new BinaryReader(stream, archiveEncoding.Default, leaveOpen: false); header = new TarHeader(archiveEncoding); - if (!header.Read(reader)) + if (!header.Read(reader, globalPaxMetadata)) { yield break; } switch (mode) { case StreamingMode.Seekable: - { - header.DataStartPosition = reader.BaseStream.Position; + header.DataStartPosition = stream.Position; //skip to nearest 512 - reader.BaseStream.Position += PadTo512(header.Size); + stream.Position += PadTo512(header.Size); } break; case StreamingMode.Streaming: - { header.PackedStream = new TarReadOnlySubStream(stream, header.Size); } @@ -56,6 +55,8 @@ internal static class TarHeaderFactory } } + // Async methods moved to TarHeaderFactory.Async.cs + private static long PadTo512(long size) { var zeros = (int)(size % 512); diff --git a/src/SharpCompress/Common/Tar/TarReadOnlySubStream.cs b/src/SharpCompress/Common/Tar/TarReadOnlySubStream.cs index 7ccfd238..86594f63 100644 --- a/src/SharpCompress/Common/Tar/TarReadOnlySubStream.cs +++ b/src/SharpCompress/Common/Tar/TarReadOnlySubStream.cs @@ -1,53 +1,128 @@ -using SharpCompress.IO; using System; using System.IO; +using System.Threading.Tasks; namespace SharpCompress.Common.Tar; -internal class TarReadOnlySubStream : NonDisposingStream +internal class TarReadOnlySubStream : Stream { + private readonly Stream _stream; + private bool _isDisposed; + private bool _isPositionedAtNextEntry; private long _amountRead; public TarReadOnlySubStream(Stream stream, long bytesToRead) - : base(stream, throwOnDispose: false) => BytesLeftToRead = bytesToRead; + { + _stream = stream; + BytesLeftToRead = bytesToRead; + } protected override void Dispose(bool disposing) { if (_isDisposed) { + base.Dispose(disposing); return; } _isDisposed = true; - if (disposing) { - // Ensure we read all remaining blocks for this entry. - Stream.Skip(BytesLeftToRead); - _amountRead += BytesLeftToRead; - - // If the last block wasn't a full 512 bytes, skip the remaining padding bytes. - var bytesInLastBlock = _amountRead % 512; - - if (bytesInLastBlock != 0) + if (Utility.UseSyncOverAsyncDispose()) { - Stream.Skip(512 - bytesInLastBlock); +#pragma warning disable VSTHRD002 // Avoid problematic synchronous waits +#pragma warning disable CA2012 + AdvanceToNextHeaderAsync().GetAwaiter().GetResult(); +#pragma warning restore CA2012 +#pragma warning restore VSTHRD002 // Avoid problematic synchronous waits + } + else + { + AdvanceToNextHeader(); } } - base.Dispose(disposing); } +#if !LEGACY_DOTNET + public override async System.Threading.Tasks.ValueTask DisposeAsync() + { + if (_isDisposed) + { + await base.DisposeAsync().ConfigureAwait(false); + return; + } + + _isDisposed = true; + await AdvanceToNextHeaderAsync().ConfigureAwait(false); + + GC.SuppressFinalize(this); + await base.DisposeAsync().ConfigureAwait(false); + } +#endif + private long BytesLeftToRead { get; set; } + private void AdvanceToNextHeader() + { + if (_isPositionedAtNextEntry) + { + return; + } + + if (BytesLeftToRead > 0) + { + _stream.Skip(BytesLeftToRead); + _amountRead += BytesLeftToRead; + BytesLeftToRead = 0; + } + + // Tar entry data is padded to 512-byte blocks, so callers that read to EOF + // should still leave the shared archive stream positioned at the next header. + var bytesInLastBlock = _amountRead % 512; + if (bytesInLastBlock != 0) + { + _stream.Skip(512 - bytesInLastBlock); + } + + _isPositionedAtNextEntry = true; + } + + private async ValueTask AdvanceToNextHeaderAsync() + { + if (_isPositionedAtNextEntry) + { + return; + } + + if (BytesLeftToRead > 0) + { + await _stream.SkipAsync(BytesLeftToRead).ConfigureAwait(false); + _amountRead += BytesLeftToRead; + BytesLeftToRead = 0; + } + + var bytesInLastBlock = _amountRead % 512; + if (bytesInLastBlock != 0) + { + await _stream.SkipAsync(512 - bytesInLastBlock).ConfigureAwait(false); + } + + _isPositionedAtNextEntry = true; + } + public override bool CanRead => true; public override bool CanSeek => false; public override bool CanWrite => false; - public override void Flush() => throw new NotSupportedException(); + public override void Flush() { } + + public override System.Threading.Tasks.Task FlushAsync( + System.Threading.CancellationToken cancellationToken + ) => System.Threading.Tasks.Task.CompletedTask; public override long Length => throw new NotSupportedException(); @@ -59,15 +134,24 @@ internal class TarReadOnlySubStream : NonDisposingStream public override int Read(byte[] buffer, int offset, int count) { + if (BytesLeftToRead <= 0) + { + AdvanceToNextHeader(); + return 0; + } if (BytesLeftToRead < count) { count = (int)BytesLeftToRead; } - int read = Stream.Read(buffer, offset, count); + var read = _stream.Read(buffer, offset, count); if (read > 0) { BytesLeftToRead -= read; _amountRead += read; + if (BytesLeftToRead == 0) + { + AdvanceToNextHeader(); + } } return read; } @@ -76,17 +160,82 @@ internal class TarReadOnlySubStream : NonDisposingStream { if (BytesLeftToRead <= 0) { + AdvanceToNextHeader(); return -1; } - int value = Stream.ReadByte(); + var value = _stream.ReadByte(); if (value != -1) { --BytesLeftToRead; ++_amountRead; + if (BytesLeftToRead == 0) + { + AdvanceToNextHeader(); + } } return value; } + public override async System.Threading.Tasks.Task ReadAsync( + byte[] buffer, + int offset, + int count, + System.Threading.CancellationToken cancellationToken + ) + { + if (BytesLeftToRead <= 0) + { + await AdvanceToNextHeaderAsync().ConfigureAwait(false); + return 0; + } + if (BytesLeftToRead < count) + { + count = (int)BytesLeftToRead; + } + var read = await _stream + .ReadAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + if (read > 0) + { + BytesLeftToRead -= read; + _amountRead += read; + if (BytesLeftToRead == 0) + { + await AdvanceToNextHeaderAsync().ConfigureAwait(false); + } + } + return read; + } + +#if !LEGACY_DOTNET + public override async System.Threading.Tasks.ValueTask ReadAsync( + System.Memory buffer, + System.Threading.CancellationToken cancellationToken = default + ) + { + if (BytesLeftToRead <= 0) + { + await AdvanceToNextHeaderAsync().ConfigureAwait(false); + return 0; + } + if (BytesLeftToRead < buffer.Length) + { + buffer = buffer.Slice(0, (int)BytesLeftToRead); + } + var read = await _stream.ReadAsync(buffer, cancellationToken).ConfigureAwait(false); + if (read > 0) + { + BytesLeftToRead -= read; + _amountRead += read; + if (BytesLeftToRead == 0) + { + await AdvanceToNextHeaderAsync().ConfigureAwait(false); + } + } + return read; + } +#endif + public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); public override void SetLength(long value) => throw new NotSupportedException(); diff --git a/src/SharpCompress/Common/Volume.Async.cs b/src/SharpCompress/Common/Volume.Async.cs new file mode 100644 index 00000000..3a07aece --- /dev/null +++ b/src/SharpCompress/Common/Volume.Async.cs @@ -0,0 +1,18 @@ +using System; +using System.Threading.Tasks; + +namespace SharpCompress.Common; + +public abstract partial class Volume +{ + public virtual async ValueTask DisposeAsync() + { +#if LEGACY_DOTNET + _actualStream.Dispose(); + await Task.CompletedTask.ConfigureAwait(false); +#else + await _actualStream.DisposeAsync().ConfigureAwait(false); +#endif + GC.SuppressFinalize(this); + } +} diff --git a/src/SharpCompress/Common/Volume.cs b/src/SharpCompress/Common/Volume.cs index 1f259257..6ada707b 100644 --- a/src/SharpCompress/Common/Volume.cs +++ b/src/SharpCompress/Common/Volume.cs @@ -1,22 +1,33 @@ using System; using System.IO; +using System.Linq; +using System.Threading.Tasks; using SharpCompress.IO; using SharpCompress.Readers; namespace SharpCompress.Common; -public abstract class Volume : IVolume +public abstract partial class Volume : IVolume, IAsyncDisposable { + private readonly Stream _baseStream; private readonly Stream _actualStream; internal Volume(Stream stream, ReaderOptions readerOptions, int index = 0) { Index = index; ReaderOptions = readerOptions; - if (readerOptions.LeaveStreamOpen) + _baseStream = stream; + + // Only rewind if it's a buffered SharpCompressStream (not passthrough) + if (stream is SharpCompressStream ss && !ss.IsPassthrough) { - stream = NonDisposingStream.Create(stream); + ss.Rewind(); } + if (ReaderOptions.LeaveStreamOpen) + { + stream = SharpCompressStream.CreateNonDisposing(stream); + } + _actualStream = stream; } @@ -32,7 +43,9 @@ public abstract class Volume : IVolume public virtual int Index { get; internal set; } - public string FileName => (_actualStream as FileStream)?.Name!; + public string? FileName => + (_baseStream as FileStream)?.Name + ?? (_baseStream as SourceStream)?.Files.FirstOrDefault()?.FullName; /// /// RarArchive is part of a multi-part archive. diff --git a/src/SharpCompress/Common/Zip/Headers/DirectoryEndHeader.Async.cs b/src/SharpCompress/Common/Zip/Headers/DirectoryEndHeader.Async.cs new file mode 100644 index 00000000..40b4ffc7 --- /dev/null +++ b/src/SharpCompress/Common/Zip/Headers/DirectoryEndHeader.Async.cs @@ -0,0 +1,21 @@ +using System.IO; +using System.Threading.Tasks; +using SharpCompress.IO; + +namespace SharpCompress.Common.Zip.Headers; + +internal partial class DirectoryEndHeader +{ + internal override async ValueTask Read(AsyncBinaryReader reader) + { + VolumeNumber = await reader.ReadUInt16Async().ConfigureAwait(false); + FirstVolumeWithDirectory = await reader.ReadUInt16Async().ConfigureAwait(false); + TotalNumberOfEntriesInDisk = await reader.ReadUInt16Async().ConfigureAwait(false); + TotalNumberOfEntries = await reader.ReadUInt16Async().ConfigureAwait(false); + DirectorySize = await reader.ReadUInt32Async().ConfigureAwait(false); + DirectoryStartOffsetRelativeToDisk = await reader.ReadUInt32Async().ConfigureAwait(false); + CommentLength = await reader.ReadUInt16Async().ConfigureAwait(false); + Comment = new byte[CommentLength]; + await reader.ReadBytesAsync(Comment, 0, CommentLength).ConfigureAwait(false); + } +} diff --git a/src/SharpCompress/Common/Zip/Headers/DirectoryEndHeader.cs b/src/SharpCompress/Common/Zip/Headers/DirectoryEndHeader.cs index 2e54a6dd..71da83af 100644 --- a/src/SharpCompress/Common/Zip/Headers/DirectoryEndHeader.cs +++ b/src/SharpCompress/Common/Zip/Headers/DirectoryEndHeader.cs @@ -2,7 +2,7 @@ using System.IO; namespace SharpCompress.Common.Zip.Headers; -internal class DirectoryEndHeader : ZipHeader +internal partial class DirectoryEndHeader : ZipHeader { public DirectoryEndHeader() : base(ZipHeaderType.DirectoryEnd) { } diff --git a/src/SharpCompress/Common/Zip/Headers/DirectoryEntryHeader.Async.cs b/src/SharpCompress/Common/Zip/Headers/DirectoryEntryHeader.Async.cs new file mode 100644 index 00000000..af978f64 --- /dev/null +++ b/src/SharpCompress/Common/Zip/Headers/DirectoryEntryHeader.Async.cs @@ -0,0 +1,44 @@ +using System.IO; +using System.Linq; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.IO; + +namespace SharpCompress.Common.Zip.Headers; + +internal partial class DirectoryEntryHeader +{ + internal override async ValueTask Read(AsyncBinaryReader reader) + { + Version = await reader.ReadUInt16Async().ConfigureAwait(false); + VersionNeededToExtract = await reader.ReadUInt16Async().ConfigureAwait(false); + Flags = (HeaderFlags)await reader.ReadUInt16Async().ConfigureAwait(false); + CompressionMethod = (ZipCompressionMethod) + await reader.ReadUInt16Async().ConfigureAwait(false); + OriginalLastModifiedTime = LastModifiedTime = await reader + .ReadUInt16Async() + .ConfigureAwait(false); + OriginalLastModifiedDate = LastModifiedDate = await reader + .ReadUInt16Async() + .ConfigureAwait(false); + Crc = await reader.ReadUInt32Async().ConfigureAwait(false); + IsCrcAvailable = true; + CompressedSize = await reader.ReadUInt32Async().ConfigureAwait(false); + UncompressedSize = await reader.ReadUInt32Async().ConfigureAwait(false); + var nameLength = await reader.ReadUInt16Async().ConfigureAwait(false); + var extraLength = await reader.ReadUInt16Async().ConfigureAwait(false); + var commentLength = await reader.ReadUInt16Async().ConfigureAwait(false); + DiskNumberStart = await reader.ReadUInt16Async().ConfigureAwait(false); + InternalFileAttributes = await reader.ReadUInt16Async().ConfigureAwait(false); + ExternalFileAttributes = await reader.ReadUInt32Async().ConfigureAwait(false); + RelativeOffsetOfEntryHeader = await reader.ReadUInt32Async().ConfigureAwait(false); + var name = new byte[nameLength]; + var extra = new byte[extraLength]; + var comment = new byte[commentLength]; + await reader.ReadBytesAsync(name, 0, nameLength).ConfigureAwait(false); + await reader.ReadBytesAsync(extra, 0, extraLength).ConfigureAwait(false); + await reader.ReadBytesAsync(comment, 0, commentLength).ConfigureAwait(false); + + ProcessReadData(name, extra, comment); + } +} diff --git a/src/SharpCompress/Common/Zip/Headers/DirectoryEntryHeader.cs b/src/SharpCompress/Common/Zip/Headers/DirectoryEntryHeader.cs index fc2cb262..f41c6047 100644 --- a/src/SharpCompress/Common/Zip/Headers/DirectoryEntryHeader.cs +++ b/src/SharpCompress/Common/Zip/Headers/DirectoryEntryHeader.cs @@ -3,9 +3,9 @@ using System.Linq; namespace SharpCompress.Common.Zip.Headers; -internal class DirectoryEntryHeader : ZipFileEntry +internal partial class DirectoryEntryHeader : ZipFileEntry { - public DirectoryEntryHeader(ArchiveEncoding archiveEncoding) + public DirectoryEntryHeader(IArchiveEncoding archiveEncoding) : base(ZipHeaderType.DirectoryEntry, archiveEncoding) { } internal override void Read(BinaryReader reader) @@ -14,9 +14,10 @@ internal class DirectoryEntryHeader : ZipFileEntry VersionNeededToExtract = reader.ReadUInt16(); Flags = (HeaderFlags)reader.ReadUInt16(); CompressionMethod = (ZipCompressionMethod)reader.ReadUInt16(); - LastModifiedTime = reader.ReadUInt16(); - LastModifiedDate = reader.ReadUInt16(); + OriginalLastModifiedTime = LastModifiedTime = reader.ReadUInt16(); + OriginalLastModifiedDate = LastModifiedDate = reader.ReadUInt16(); Crc = reader.ReadUInt32(); + IsCrcAvailable = true; CompressedSize = reader.ReadUInt32(); UncompressedSize = reader.ReadUInt32(); var nameLength = reader.ReadUInt16(); @@ -31,7 +32,11 @@ internal class DirectoryEntryHeader : ZipFileEntry var extra = reader.ReadBytes(extraLength); var comment = reader.ReadBytes(commentLength); - // According to .ZIP File Format Specification + ProcessReadData(name, extra, comment); + } + + private void ProcessReadData(byte[] name, byte[] extra, byte[] comment) + { // // For example: https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT // @@ -41,8 +46,8 @@ internal class DirectoryEntryHeader : ZipFileEntry if (Flags.HasFlag(HeaderFlags.Efs)) { - Name = ArchiveEncoding.DecodeUTF8(name); - Comment = ArchiveEncoding.DecodeUTF8(comment); + Name = ArchiveEncoding.Decode(name, EncodingType.UTF8); + Comment = ArchiveEncoding.Decode(comment, EncodingType.UTF8); } else { @@ -52,8 +57,8 @@ internal class DirectoryEntryHeader : ZipFileEntry LoadExtra(extra); - var unicodePathExtra = Extra.FirstOrDefault( - u => u.Type == ExtraDataType.UnicodePathExtraField + var unicodePathExtra = Extra.FirstOrDefault(u => + u.Type == ExtraDataType.UnicodePathExtraField ); if (unicodePathExtra != null && ArchiveEncoding.Forced == null) { @@ -85,6 +90,36 @@ internal class DirectoryEntryHeader : ZipFileEntry RelativeOffsetOfEntryHeader = zip64ExtraData.RelativeOffsetOfEntryHeader; } } + + var unixTimeExtra = Extra.FirstOrDefault(u => u.Type == ExtraDataType.UnixTimeExtraField); + + if (unixTimeExtra is not null) + { + // Tuple order is last modified time, last access time, and creation time. + var unixTimeTuple = ((UnixTimeExtraField)unixTimeExtra).UnicodeTimes; + + if (unixTimeTuple.Item1.HasValue) + { + var dosTime = Utility.DateTimeToDosTime(unixTimeTuple.Item1.Value); + + LastModifiedDate = (ushort)(dosTime >> 16); + LastModifiedTime = (ushort)(dosTime & 0x0FFFF); + } + else if (unixTimeTuple.Item2.HasValue) + { + var dosTime = Utility.DateTimeToDosTime(unixTimeTuple.Item2.Value); + + LastModifiedDate = (ushort)(dosTime >> 16); + LastModifiedTime = (ushort)(dosTime & 0x0FFFF); + } + else if (unixTimeTuple.Item3.HasValue) + { + var dosTime = Utility.DateTimeToDosTime(unixTimeTuple.Item3.Value); + + LastModifiedDate = (ushort)(dosTime >> 16); + LastModifiedTime = (ushort)(dosTime & 0x0FFFF); + } + } } internal ushort Version { get; private set; } @@ -93,11 +128,7 @@ internal class DirectoryEntryHeader : ZipFileEntry public long RelativeOffsetOfEntryHeader { get; set; } - public uint ExternalFileAttributes { get; set; } - public ushort InternalFileAttributes { get; set; } public ushort DiskNumberStart { get; set; } - - public string? Comment { get; private set; } } diff --git a/src/SharpCompress/Common/Zip/Headers/HeaderFlags.cs b/src/SharpCompress/Common/Zip/Headers/HeaderFlags.cs index 5aef3394..728f56b4 100644 --- a/src/SharpCompress/Common/Zip/Headers/HeaderFlags.cs +++ b/src/SharpCompress/Common/Zip/Headers/HeaderFlags.cs @@ -13,5 +13,5 @@ internal enum HeaderFlags : ushort EnhancedDeflate = 16, //Bit 11: Language encoding flag - Efs = 2048 + Efs = 2048, } diff --git a/src/SharpCompress/Common/Zip/Headers/IgnoreHeader.cs b/src/SharpCompress/Common/Zip/Headers/IgnoreHeader.cs index 5a587a7b..86b66a81 100644 --- a/src/SharpCompress/Common/Zip/Headers/IgnoreHeader.cs +++ b/src/SharpCompress/Common/Zip/Headers/IgnoreHeader.cs @@ -1,4 +1,6 @@ using System.IO; +using System.Threading.Tasks; +using SharpCompress.IO; namespace SharpCompress.Common.Zip.Headers; @@ -8,4 +10,6 @@ internal class IgnoreHeader : ZipHeader : base(type) { } internal override void Read(BinaryReader reader) { } + + internal override ValueTask Read(AsyncBinaryReader reader) => default; } diff --git a/src/SharpCompress/Common/Zip/Headers/LocalEntryHeader.Async.cs b/src/SharpCompress/Common/Zip/Headers/LocalEntryHeader.Async.cs new file mode 100644 index 00000000..2e96c942 --- /dev/null +++ b/src/SharpCompress/Common/Zip/Headers/LocalEntryHeader.Async.cs @@ -0,0 +1,35 @@ +using System.IO; +using System.Linq; +using System.Threading.Tasks; +using SharpCompress.IO; + +namespace SharpCompress.Common.Zip.Headers; + +internal partial class LocalEntryHeader +{ + internal override async ValueTask Read(AsyncBinaryReader reader) + { + Version = await reader.ReadUInt16Async().ConfigureAwait(false); + Flags = (HeaderFlags)await reader.ReadUInt16Async().ConfigureAwait(false); + CompressionMethod = (ZipCompressionMethod) + await reader.ReadUInt16Async().ConfigureAwait(false); + OriginalLastModifiedTime = LastModifiedTime = await reader + .ReadUInt16Async() + .ConfigureAwait(false); + OriginalLastModifiedDate = LastModifiedDate = await reader + .ReadUInt16Async() + .ConfigureAwait(false); + Crc = await reader.ReadUInt32Async().ConfigureAwait(false); + IsCrcAvailable = !Flags.HasFlag(HeaderFlags.UsePostDataDescriptor); + CompressedSize = await reader.ReadUInt32Async().ConfigureAwait(false); + UncompressedSize = await reader.ReadUInt32Async().ConfigureAwait(false); + var nameLength = await reader.ReadUInt16Async().ConfigureAwait(false); + var extraLength = await reader.ReadUInt16Async().ConfigureAwait(false); + var name = new byte[nameLength]; + var extra = new byte[extraLength]; + await reader.ReadBytesAsync(name, 0, nameLength).ConfigureAwait(false); + await reader.ReadBytesAsync(extra, 0, extraLength).ConfigureAwait(false); + + ProcessReadData(name, extra); + } +} diff --git a/src/SharpCompress/Common/Zip/Headers/LocalEntryHeader.cs b/src/SharpCompress/Common/Zip/Headers/LocalEntryHeader.cs index 93cc55ac..9d4512b1 100644 --- a/src/SharpCompress/Common/Zip/Headers/LocalEntryHeader.cs +++ b/src/SharpCompress/Common/Zip/Headers/LocalEntryHeader.cs @@ -3,9 +3,9 @@ using System.Linq; namespace SharpCompress.Common.Zip.Headers; -internal class LocalEntryHeader : ZipFileEntry +internal partial class LocalEntryHeader : ZipFileEntry { - public LocalEntryHeader(ArchiveEncoding archiveEncoding) + public LocalEntryHeader(IArchiveEncoding archiveEncoding) : base(ZipHeaderType.LocalEntry, archiveEncoding) { } internal override void Read(BinaryReader reader) @@ -13,9 +13,10 @@ internal class LocalEntryHeader : ZipFileEntry Version = reader.ReadUInt16(); Flags = (HeaderFlags)reader.ReadUInt16(); CompressionMethod = (ZipCompressionMethod)reader.ReadUInt16(); - LastModifiedTime = reader.ReadUInt16(); - LastModifiedDate = reader.ReadUInt16(); + OriginalLastModifiedTime = LastModifiedTime = reader.ReadUInt16(); + OriginalLastModifiedDate = LastModifiedDate = reader.ReadUInt16(); Crc = reader.ReadUInt32(); + IsCrcAvailable = !Flags.HasFlag(HeaderFlags.UsePostDataDescriptor); CompressedSize = reader.ReadUInt32(); UncompressedSize = reader.ReadUInt32(); var nameLength = reader.ReadUInt16(); @@ -23,7 +24,11 @@ internal class LocalEntryHeader : ZipFileEntry var name = reader.ReadBytes(nameLength); var extra = reader.ReadBytes(extraLength); - // According to .ZIP File Format Specification + ProcessReadData(name, extra); + } + + private void ProcessReadData(byte[] name, byte[] extra) + { // // For example: https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT // @@ -33,7 +38,7 @@ internal class LocalEntryHeader : ZipFileEntry if (Flags.HasFlag(HeaderFlags.Efs)) { - Name = ArchiveEncoding.DecodeUTF8(name); + Name = ArchiveEncoding.Decode(name, EncodingType.UTF8); } else { @@ -42,8 +47,8 @@ internal class LocalEntryHeader : ZipFileEntry LoadExtra(extra); - var unicodePathExtra = Extra.FirstOrDefault( - u => u.Type == ExtraDataType.UnicodePathExtraField + var unicodePathExtra = Extra.FirstOrDefault(u => + u.Type == ExtraDataType.UnicodePathExtraField ); if (unicodePathExtra != null && ArchiveEncoding.Forced == null) { @@ -64,6 +69,36 @@ internal class LocalEntryHeader : ZipFileEntry UncompressedSize = zip64ExtraData.UncompressedSize; } } + + var unixTimeExtra = Extra.FirstOrDefault(u => u.Type == ExtraDataType.UnixTimeExtraField); + + if (unixTimeExtra is not null) + { + // Tuple order is last modified time, last access time, and creation time. + var unixTimeTuple = ((UnixTimeExtraField)unixTimeExtra).UnicodeTimes; + + if (unixTimeTuple.Item1.HasValue) + { + var dosTime = Utility.DateTimeToDosTime(unixTimeTuple.Item1.Value); + + LastModifiedDate = (ushort)(dosTime >> 16); + LastModifiedTime = (ushort)(dosTime & 0x0FFFF); + } + else if (unixTimeTuple.Item2.HasValue) + { + var dosTime = Utility.DateTimeToDosTime(unixTimeTuple.Item2.Value); + + LastModifiedDate = (ushort)(dosTime >> 16); + LastModifiedTime = (ushort)(dosTime & 0x0FFFF); + } + else if (unixTimeTuple.Item3.HasValue) + { + var dosTime = Utility.DateTimeToDosTime(unixTimeTuple.Item3.Value); + + LastModifiedDate = (ushort)(dosTime >> 16); + LastModifiedTime = (ushort)(dosTime & 0x0FFFF); + } + } } internal ushort Version { get; private set; } diff --git a/src/SharpCompress/Common/Zip/Headers/LocalEntryHeaderExtraFactory.cs b/src/SharpCompress/Common/Zip/Headers/LocalEntryHeaderExtraFactory.cs index 80ec9b8f..daedd9a6 100644 --- a/src/SharpCompress/Common/Zip/Headers/LocalEntryHeaderExtraFactory.cs +++ b/src/SharpCompress/Common/Zip/Headers/LocalEntryHeaderExtraFactory.cs @@ -1,4 +1,4 @@ -using System; +using System; using System.Buffers.Binary; using System.Text; @@ -13,7 +13,8 @@ internal enum ExtraDataType : ushort // Third Party Mappings // -Info-ZIP Unicode Path Extra Field UnicodePathExtraField = 0x7075, - Zip64ExtendedInformationExtraField = 0x0001 + Zip64ExtendedInformationExtraField = 0x0001, + UnixTimeExtraField = 0x5455, } internal class ExtraData @@ -145,15 +146,106 @@ internal sealed class Zip64ExtendedInformationExtraField : ExtraData public uint VolumeNumber { get; private set; } } +internal sealed class UnixTimeExtraField : ExtraData +{ + public UnixTimeExtraField(ExtraDataType type, ushort length, byte[] dataBytes) + : base(type, length, dataBytes) { } + + /// + /// The unix modified time, last access time, and creation time, if set. + /// + /// Must return Tuple explicitly due to net462 support. + internal Tuple UnicodeTimes + { + get + { + // There has to be at least 5 byte for there to be a timestamp. + // 1 byte for flags and 4 bytes for a timestamp. + if (DataBytes is null || DataBytes.Length < 5) + { + return Tuple.Create(null, null, null); + } + + var flags = (RecordedTimeFlag)DataBytes[0]; + var isModifiedTimeSpecified = flags.HasFlag(RecordedTimeFlag.LastModified); + var isLastAccessTimeSpecified = flags.HasFlag(RecordedTimeFlag.LastAccessed); + var isCreationTimeSpecified = flags.HasFlag(RecordedTimeFlag.Created); + var currentIndex = 1; + DateTime? modifiedTime = null; + DateTime? lastAccessTime = null; + DateTime? creationTime = null; + + if (isModifiedTimeSpecified) + { + var modifiedEpochTime = BinaryPrimitives.ReadInt32LittleEndian( + DataBytes.AsSpan(currentIndex, 4) + ); + + currentIndex += 4; + modifiedTime = DateTimeOffset.FromUnixTimeSeconds(modifiedEpochTime).UtcDateTime; + } + + if (isLastAccessTimeSpecified) + { + if (currentIndex + 4 > DataBytes.Length) + { + return Tuple.Create(null, null, null); + } + + var lastAccessEpochTime = BinaryPrimitives.ReadInt32LittleEndian( + DataBytes.AsSpan(currentIndex, 4) + ); + + currentIndex += 4; + lastAccessTime = DateTimeOffset + .FromUnixTimeSeconds(lastAccessEpochTime) + .UtcDateTime; + } + + if (isCreationTimeSpecified) + { + if (currentIndex + 4 > DataBytes.Length) + { + return Tuple.Create(null, null, null); + } + + var creationTimeEpochTime = BinaryPrimitives.ReadInt32LittleEndian( + DataBytes.AsSpan(currentIndex, 4) + ); + + currentIndex += 4; + creationTime = DateTimeOffset + .FromUnixTimeSeconds(creationTimeEpochTime) + .UtcDateTime; + } + + return Tuple.Create(modifiedTime, lastAccessTime, creationTime); + } + } + + [Flags] + private enum RecordedTimeFlag + { + None = 0, + LastModified = 1, + LastAccessed = 2, + Created = 4, + } +} + internal static class LocalEntryHeaderExtraFactory { internal static ExtraData Create(ExtraDataType type, ushort length, byte[] extraData) => type switch { - ExtraDataType.UnicodePathExtraField - => new ExtraUnicodePathExtraField(type, length, extraData), - ExtraDataType.Zip64ExtendedInformationExtraField - => new Zip64ExtendedInformationExtraField(type, length, extraData), - _ => new ExtraData(type, length, extraData) + ExtraDataType.UnicodePathExtraField => new ExtraUnicodePathExtraField( + type, + length, + extraData + ), + ExtraDataType.Zip64ExtendedInformationExtraField => + new Zip64ExtendedInformationExtraField(type, length, extraData), + ExtraDataType.UnixTimeExtraField => new UnixTimeExtraField(type, length, extraData), + _ => new ExtraData(type, length, extraData), }; } diff --git a/src/SharpCompress/Common/Zip/Headers/SplitHeader.cs b/src/SharpCompress/Common/Zip/Headers/SplitHeader.cs index 4151a6cb..d5e68fee 100644 --- a/src/SharpCompress/Common/Zip/Headers/SplitHeader.cs +++ b/src/SharpCompress/Common/Zip/Headers/SplitHeader.cs @@ -1,5 +1,7 @@ using System; using System.IO; +using System.Threading.Tasks; +using SharpCompress.IO; namespace SharpCompress.Common.Zip.Headers; @@ -9,4 +11,7 @@ internal class SplitHeader : ZipHeader : base(ZipHeaderType.Split) { } internal override void Read(BinaryReader reader) => throw new NotImplementedException(); + + internal override ValueTask Read(AsyncBinaryReader reader) => + throw new NotImplementedException(); } diff --git a/src/SharpCompress/Common/Zip/Headers/Zip64DirectoryEndHeader.Async.cs b/src/SharpCompress/Common/Zip/Headers/Zip64DirectoryEndHeader.Async.cs new file mode 100644 index 00000000..9e510688 --- /dev/null +++ b/src/SharpCompress/Common/Zip/Headers/Zip64DirectoryEndHeader.Async.cs @@ -0,0 +1,27 @@ +using System.IO; +using System.Threading.Tasks; +using SharpCompress.IO; + +namespace SharpCompress.Common.Zip.Headers; + +internal partial class Zip64DirectoryEndHeader +{ + internal override async ValueTask Read(AsyncBinaryReader reader) + { + SizeOfDirectoryEndRecord = (long)await reader.ReadUInt64Async().ConfigureAwait(false); + VersionMadeBy = await reader.ReadUInt16Async().ConfigureAwait(false); + VersionNeededToExtract = await reader.ReadUInt16Async().ConfigureAwait(false); + VolumeNumber = await reader.ReadUInt32Async().ConfigureAwait(false); + FirstVolumeWithDirectory = await reader.ReadUInt32Async().ConfigureAwait(false); + TotalNumberOfEntriesInDisk = (long)await reader.ReadUInt64Async().ConfigureAwait(false); + TotalNumberOfEntries = (long)await reader.ReadUInt64Async().ConfigureAwait(false); + DirectorySize = (long)await reader.ReadUInt64Async().ConfigureAwait(false); + DirectoryStartOffsetRelativeToDisk = (long) + await reader.ReadUInt64Async().ConfigureAwait(false); + var size = (int)( + SizeOfDirectoryEndRecord - SIZE_OF_FIXED_HEADER_DATA_EXCEPT_SIGNATURE_AND_SIZE_FIELDS + ); + DataSector = new byte[size]; + await reader.ReadBytesAsync(DataSector, 0, size).ConfigureAwait(false); + } +} diff --git a/src/SharpCompress/Common/Zip/Headers/Zip64DirectoryEndHeader.cs b/src/SharpCompress/Common/Zip/Headers/Zip64DirectoryEndHeader.cs index a74b4d1f..3933b2e0 100644 --- a/src/SharpCompress/Common/Zip/Headers/Zip64DirectoryEndHeader.cs +++ b/src/SharpCompress/Common/Zip/Headers/Zip64DirectoryEndHeader.cs @@ -2,7 +2,7 @@ using System.IO; namespace SharpCompress.Common.Zip.Headers; -internal class Zip64DirectoryEndHeader : ZipHeader +internal partial class Zip64DirectoryEndHeader : ZipHeader { public Zip64DirectoryEndHeader() : base(ZipHeaderType.Zip64DirectoryEnd) { } diff --git a/src/SharpCompress/Common/Zip/Headers/Zip64DirectoryEndLocatorHeader.Async.cs b/src/SharpCompress/Common/Zip/Headers/Zip64DirectoryEndLocatorHeader.Async.cs new file mode 100644 index 00000000..e0095510 --- /dev/null +++ b/src/SharpCompress/Common/Zip/Headers/Zip64DirectoryEndLocatorHeader.Async.cs @@ -0,0 +1,16 @@ +using System.IO; +using System.Threading.Tasks; +using SharpCompress.IO; + +namespace SharpCompress.Common.Zip.Headers; + +internal partial class Zip64DirectoryEndLocatorHeader +{ + internal override async ValueTask Read(AsyncBinaryReader reader) + { + FirstVolumeWithDirectory = await reader.ReadUInt32Async().ConfigureAwait(false); + RelativeOffsetOfTheEndOfDirectoryRecord = (long) + await reader.ReadUInt64Async().ConfigureAwait(false); + TotalNumberOfVolumes = await reader.ReadUInt32Async().ConfigureAwait(false); + } +} diff --git a/src/SharpCompress/Common/Zip/Headers/Zip64DirectoryEndLocatorHeader.cs b/src/SharpCompress/Common/Zip/Headers/Zip64DirectoryEndLocatorHeader.cs index 3020d377..3477c804 100644 --- a/src/SharpCompress/Common/Zip/Headers/Zip64DirectoryEndLocatorHeader.cs +++ b/src/SharpCompress/Common/Zip/Headers/Zip64DirectoryEndLocatorHeader.cs @@ -2,11 +2,9 @@ using System.IO; namespace SharpCompress.Common.Zip.Headers; -internal class Zip64DirectoryEndLocatorHeader : ZipHeader +internal partial class Zip64DirectoryEndLocatorHeader() + : ZipHeader(ZipHeaderType.Zip64DirectoryEndLocator) { - public Zip64DirectoryEndLocatorHeader() - : base(ZipHeaderType.Zip64DirectoryEndLocator) { } - internal override void Read(BinaryReader reader) { FirstVolumeWithDirectory = reader.ReadUInt32(); diff --git a/src/SharpCompress/Common/Zip/Headers/ZipFileEntry.Async.cs b/src/SharpCompress/Common/Zip/Headers/ZipFileEntry.Async.cs new file mode 100644 index 00000000..f541fc8d --- /dev/null +++ b/src/SharpCompress/Common/Zip/Headers/ZipFileEntry.Async.cs @@ -0,0 +1,24 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Common.Zip.Headers; + +internal abstract partial class ZipFileEntry +{ + internal async ValueTask ComposeEncryptionDataAsync( + Stream archiveStream, + CancellationToken cancellationToken = default + ) + { + ThrowHelper.ThrowIfNull(archiveStream); + + var buffer = new byte[12]; + await archiveStream.ReadFullyAsync(buffer, 0, 12, cancellationToken).ConfigureAwait(false); + + var encryptionData = PkwareTraditionalEncryptionData.ForRead(Password!, this, buffer); + + return encryptionData; + } +} diff --git a/src/SharpCompress/Common/Zip/Headers/ZipFileEntry.cs b/src/SharpCompress/Common/Zip/Headers/ZipFileEntry.cs index 98d230f5..e7f0b042 100644 --- a/src/SharpCompress/Common/Zip/Headers/ZipFileEntry.cs +++ b/src/SharpCompress/Common/Zip/Headers/ZipFileEntry.cs @@ -1,5 +1,3 @@ -#nullable disable - using System; using System.Buffers.Binary; using System.Collections.Generic; @@ -7,34 +5,28 @@ using System.IO; namespace SharpCompress.Common.Zip.Headers; -internal abstract class ZipFileEntry : ZipHeader +internal abstract partial class ZipFileEntry(ZipHeaderType type, IArchiveEncoding archiveEncoding) + : ZipHeader(type) { - protected ZipFileEntry(ZipHeaderType type, ArchiveEncoding archiveEncoding) - : base(type) - { - Extra = new List(); - ArchiveEncoding = archiveEncoding; - } - internal bool IsDirectory { get { - if (Name.EndsWith('/')) + if (Name?.EndsWith('/') ?? false) { return true; } //.NET Framework 4.5 : System.IO.Compression::CreateFromDirectory() probably writes backslashes to headers - return CompressedSize == 0 && UncompressedSize == 0 && Name.EndsWith('\\'); + return CompressedSize == 0 && UncompressedSize == 0 && (Name?.EndsWith('\\') ?? false); } } - internal Stream PackedStream { get; set; } + internal Stream? PackedStream { get; set; } - internal ArchiveEncoding ArchiveEncoding { get; } + internal IArchiveEncoding ArchiveEncoding { get; } = archiveEncoding; - internal string Name { get; set; } + internal string? Name { get; set; } internal HeaderFlags Flags { get; set; } @@ -46,16 +38,13 @@ internal abstract class ZipFileEntry : ZipHeader internal long UncompressedSize { get; set; } - internal List Extra { get; set; } + internal List Extra { get; set; } = new(); - public string Password { get; set; } + public string? Password { get; set; } internal PkwareTraditionalEncryptionData ComposeEncryptionData(Stream archiveStream) { - if (archiveStream is null) - { - throw new ArgumentNullException(nameof(archiveStream)); - } + ThrowHelper.ThrowIfNull(archiveStream); var buffer = new byte[12]; archiveStream.ReadFully(buffer); @@ -65,20 +54,47 @@ internal abstract class ZipFileEntry : ZipHeader return encryptionData; } - internal WinzipAesEncryptionData WinzipAesEncryptionData { get; set; } + internal WinzipAesEncryptionData? WinzipAesEncryptionData { get; set; } + /// + /// The last modified date as read from the Local or Central Directory header. + /// + internal ushort OriginalLastModifiedDate { get; set; } + + /// + /// The last modified date from the UnixTimeExtraField, if present, or the + /// Local or Cental Directory header, if not. + /// internal ushort LastModifiedDate { get; set; } + /// + /// The last modified time as read from the Local or Central Directory header. + /// + internal ushort OriginalLastModifiedTime { get; set; } + + /// + /// The last modified time from the UnixTimeExtraField, if present, or the + /// Local or Cental Directory header, if not. + /// internal ushort LastModifiedTime { get; set; } internal uint Crc { get; set; } + internal bool IsCrcAvailable { get; set; } + protected void LoadExtra(byte[] extra) { - for (var i = 0; i < extra.Length - 4; ) + for (var i = 0; i < extra.Length; ) { + // Ensure we have at least a header (2-byte ID + 2-byte length) + if (i + 4 > extra.Length) + { + // Incomplete header — stop parsing extras + break; + } + var type = (ExtraDataType)BinaryPrimitives.ReadUInt16LittleEndian(extra.AsSpan(i)); - if (!Enum.IsDefined(typeof(ExtraDataType), type)) + if (!IsDefined(type)) { type = ExtraDataType.NotImplementedExtraData; } @@ -90,7 +106,17 @@ internal abstract class ZipFileEntry : ZipHeader if (length > extra.Length) { // bad extras block - return; + break; // allow processing optional other blocks + } + // Some ZIP files contain vendor-specific or malformed extra fields where the declared + // data length extends beyond the remaining buffer. This adjustment ensures that + // we only read data within bounds (i + 4 + length <= extra.Length) + // The example here is: 41 43 18 00 41 52 43 30 46 EB FF FF 51 29 03 C6 03 00 00 00 00 00 00 00 00 + // No existing zip utility uses 0x4341 ('AC') + if (i + 4 + length > extra.Length) + { + // incomplete or corrupt field + break; // allow processing other blocks } var data = new byte[length]; @@ -101,7 +127,20 @@ internal abstract class ZipFileEntry : ZipHeader } } - internal ZipFilePart Part { get; set; } + internal ZipFilePart? Part { get; set; } internal bool IsZip64 => CompressedSize >= uint.MaxValue; + + internal uint ExternalFileAttributes { get; set; } + + internal string? Comment { get; set; } + + private static bool IsDefined(ExtraDataType type) + { +#if LEGACY_DOTNET + return Enum.IsDefined(typeof(ExtraDataType), type); +#else + return Enum.IsDefined(type); +#endif + } } diff --git a/src/SharpCompress/Common/Zip/Headers/ZipHeader.cs b/src/SharpCompress/Common/Zip/Headers/ZipHeader.cs index 36d40a82..5daf0560 100644 --- a/src/SharpCompress/Common/Zip/Headers/ZipHeader.cs +++ b/src/SharpCompress/Common/Zip/Headers/ZipHeader.cs @@ -1,18 +1,15 @@ using System.IO; +using System.Threading.Tasks; +using SharpCompress.IO; namespace SharpCompress.Common.Zip.Headers; -internal abstract class ZipHeader +internal abstract class ZipHeader(ZipHeaderType type) { - protected ZipHeader(ZipHeaderType type) - { - ZipHeaderType = type; - HasData = true; - } - - internal ZipHeaderType ZipHeaderType { get; } + internal ZipHeaderType ZipHeaderType { get; } = type; internal abstract void Read(BinaryReader reader); + internal abstract ValueTask Read(AsyncBinaryReader reader); - internal bool HasData { get; set; } + internal bool HasData { get; set; } = true; } diff --git a/src/SharpCompress/Common/Zip/Headers/ZipHeaderType.cs b/src/SharpCompress/Common/Zip/Headers/ZipHeaderType.cs index 62c29d89..75cce8d8 100644 --- a/src/SharpCompress/Common/Zip/Headers/ZipHeaderType.cs +++ b/src/SharpCompress/Common/Zip/Headers/ZipHeaderType.cs @@ -8,5 +8,5 @@ internal enum ZipHeaderType DirectoryEnd, Split, Zip64DirectoryEnd, - Zip64DirectoryEndLocator + Zip64DirectoryEndLocator, } diff --git a/src/SharpCompress/Common/Zip/PkwareTraditionalCryptoStream.Async.cs b/src/SharpCompress/Common/Zip/PkwareTraditionalCryptoStream.Async.cs new file mode 100644 index 00000000..32a0a7bf --- /dev/null +++ b/src/SharpCompress/Common/Zip/PkwareTraditionalCryptoStream.Async.cs @@ -0,0 +1,129 @@ +using System; +using System.Buffers; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Common.Zip; + +internal partial class PkwareTraditionalCryptoStream +{ + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + if (_mode == CryptoMode.Encrypt) + { + throw new NotSupportedException("This stream does not encrypt via Read()"); + } + + ThrowHelper.ThrowIfNull(buffer); + + var temp = new byte[count]; + var readBytes = await _stream + .ReadAsync(temp, 0, count, cancellationToken) + .ConfigureAwait(false); + var decrypted = _encryptor.Decrypt(temp, readBytes); + Buffer.BlockCopy(decrypted, 0, buffer, offset, readBytes); + return readBytes; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + if (_mode == CryptoMode.Encrypt) + { + throw new NotSupportedException("This stream does not encrypt via Read()"); + } + + byte[] temp = ArrayPool.Shared.Rent(buffer.Length); + try + { + int readBytes = await _stream + .ReadAsync(temp.AsMemory(0, buffer.Length), cancellationToken) + .ConfigureAwait(false); + var decrypted = _encryptor.Decrypt(temp, readBytes); + decrypted.AsMemory(0, readBytes).CopyTo(buffer); + return readBytes; + } + finally + { + ArrayPool.Shared.Return(temp); + } + } +#endif + + public override async Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + if (_mode == CryptoMode.Decrypt) + { + throw new NotSupportedException("This stream does not Decrypt via Write()"); + } + + if (count == 0) + { + return; + } + + byte[] plaintext; + if (offset != 0) + { + plaintext = new byte[count]; + Buffer.BlockCopy(buffer, offset, plaintext, 0, count); + } + else + { + plaintext = buffer; + } + + var encrypted = _encryptor.Encrypt(plaintext, count); + await _stream + .WriteAsync(encrypted, 0, encrypted.Length, cancellationToken) + .ConfigureAwait(false); + } + +#if !LEGACY_DOTNET + public override async ValueTask WriteAsync( + ReadOnlyMemory buffer, + CancellationToken cancellationToken = default + ) + { + if (_mode == CryptoMode.Decrypt) + { + throw new NotSupportedException("This stream does not Decrypt via Write()"); + } + + if (buffer.Length == 0) + { + return; + } + + byte[] plaintext; + if (buffer.Span.Overlaps(buffer.Span)) + { + plaintext = buffer.ToArray(); + } + else + { + plaintext = new byte[buffer.Length]; + buffer.CopyTo(plaintext); + } + + var encrypted = _encryptor.Encrypt(plaintext, buffer.Length); + await _stream + .WriteAsync(encrypted.AsMemory(0, encrypted.Length), cancellationToken) + .ConfigureAwait(false); + } +#endif +} diff --git a/src/SharpCompress/Common/Zip/PkwareTraditionalCryptoStream.cs b/src/SharpCompress/Common/Zip/PkwareTraditionalCryptoStream.cs index 273a7a3c..2efd036e 100644 --- a/src/SharpCompress/Common/Zip/PkwareTraditionalCryptoStream.cs +++ b/src/SharpCompress/Common/Zip/PkwareTraditionalCryptoStream.cs @@ -1,4 +1,4 @@ -using System; +using System; using System.IO; namespace SharpCompress.Common.Zip; @@ -6,10 +6,10 @@ namespace SharpCompress.Common.Zip; internal enum CryptoMode { Encrypt, - Decrypt + Decrypt, } -internal class PkwareTraditionalCryptoStream : Stream +internal partial class PkwareTraditionalCryptoStream : Stream { private readonly PkwareTraditionalEncryptionData _encryptor; private readonly CryptoMode _mode; @@ -48,10 +48,7 @@ internal class PkwareTraditionalCryptoStream : Stream throw new NotSupportedException("This stream does not encrypt via Read()"); } - if (buffer is null) - { - throw new ArgumentNullException(nameof(buffer)); - } + ThrowHelper.ThrowIfNull(buffer); var temp = new byte[count]; var readBytes = _stream.Read(temp, 0, count); @@ -87,10 +84,7 @@ internal class PkwareTraditionalCryptoStream : Stream _stream.Write(encrypted, 0, encrypted.Length); } - public override void Flush() - { - //throw new NotSupportedException(); - } + public override void Flush() { } public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); diff --git a/src/SharpCompress/Common/Zip/PkwareTraditionalEncryptionData.cs b/src/SharpCompress/Common/Zip/PkwareTraditionalEncryptionData.cs index 353000d8..63d46bb6 100644 --- a/src/SharpCompress/Common/Zip/PkwareTraditionalEncryptionData.cs +++ b/src/SharpCompress/Common/Zip/PkwareTraditionalEncryptionData.cs @@ -6,11 +6,11 @@ namespace SharpCompress.Common.Zip; internal class PkwareTraditionalEncryptionData { - private static readonly CRC32 CRC32 = new CRC32(); + private static readonly CRC32 CRC32 = new(); private readonly uint[] _keys = { 0x12345678, 0x23456789, 0x34567890 }; - private readonly ArchiveEncoding _archiveEncoding; + private readonly IArchiveEncoding _archiveEncoding; - private PkwareTraditionalEncryptionData(string password, ArchiveEncoding archiveEncoding) + private PkwareTraditionalEncryptionData(string password, IArchiveEncoding archiveEncoding) { _archiveEncoding = archiveEncoding; Initialize(password); @@ -39,7 +39,7 @@ internal class PkwareTraditionalEncryptionData { throw new CryptographicException("The password did not match."); } - if (plainTextHeader[11] != (byte)((header.LastModifiedTime >> 8) & 0xff)) + if (plainTextHeader[11] != (byte)((header.OriginalLastModifiedTime >> 8) & 0xff)) { throw new CryptographicException("The password did not match."); } @@ -69,10 +69,7 @@ internal class PkwareTraditionalEncryptionData public byte[] Encrypt(byte[] plainText, int length) { - if (plainText is null) - { - throw new ArgumentNullException(nameof(plainText)); - } + ThrowHelper.ThrowIfNull(plainText); if (length > plainText.Length) { diff --git a/src/SharpCompress/Common/Zip/SeekableZipFilePart.Async.cs b/src/SharpCompress/Common/Zip/SeekableZipFilePart.Async.cs new file mode 100644 index 00000000..8fff8436 --- /dev/null +++ b/src/SharpCompress/Common/Zip/SeekableZipFilePart.Async.cs @@ -0,0 +1,26 @@ +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Zip.Headers; + +namespace SharpCompress.Common.Zip; + +internal partial class SeekableZipFilePart +{ + internal override async ValueTask GetCompressedStreamAsync( + CancellationToken cancellationToken = default + ) + { + if (!_isLocalHeaderLoaded) + { + await LoadLocalHeaderAsync(cancellationToken).ConfigureAwait(false); + _isLocalHeaderLoaded = true; + } + return await base.GetCompressedStreamAsync(cancellationToken).ConfigureAwait(false); + } + + private async ValueTask LoadLocalHeaderAsync(CancellationToken cancellationToken = default) => + Header = await _headerFactory + .GetLocalHeaderAsync(BaseStream, (DirectoryEntryHeader)Header) + .ConfigureAwait(false); +} diff --git a/src/SharpCompress/Common/Zip/SeekableZipFilePart.cs b/src/SharpCompress/Common/Zip/SeekableZipFilePart.cs index 63fdc933..f9c59dfd 100644 --- a/src/SharpCompress/Common/Zip/SeekableZipFilePart.cs +++ b/src/SharpCompress/Common/Zip/SeekableZipFilePart.cs @@ -1,25 +1,22 @@ using System.IO; using SharpCompress.Common.Zip.Headers; -using SharpCompress.IO; +using SharpCompress.Compressors; +using SharpCompress.Providers; namespace SharpCompress.Common.Zip; -internal class SeekableZipFilePart : ZipFilePart +internal partial class SeekableZipFilePart : ZipFilePart { private bool _isLocalHeaderLoaded; private readonly SeekableZipHeaderFactory _headerFactory; - private readonly DirectoryEntryHeader _directoryEntryHeader; internal SeekableZipFilePart( SeekableZipHeaderFactory headerFactory, DirectoryEntryHeader header, - Stream stream + Stream stream, + CompressionProviderRegistry compressionProviders ) - : base(header, stream) - { - _headerFactory = headerFactory; - _directoryEntryHeader = header; - } + : base(header, stream, compressionProviders) => _headerFactory = headerFactory; internal override Stream GetCompressedStream() { @@ -31,28 +28,12 @@ internal class SeekableZipFilePart : ZipFilePart return base.GetCompressedStream(); } - internal string? Comment => ((DirectoryEntryHeader)Header).Comment; - - private void LoadLocalHeader() - { - var hasData = Header.HasData; - Header = _headerFactory.GetLocalHeader(BaseStream, ((DirectoryEntryHeader)Header)); - Header.HasData = hasData; - } + private void LoadLocalHeader() => + Header = _headerFactory.GetLocalHeader(BaseStream, (DirectoryEntryHeader)Header); protected override Stream CreateBaseStream() { - BaseStream.Position = Header.DataStartPosition!.Value; - - if ( - (Header.CompressedSize == 0) - && FlagUtility.HasFlag(Header.Flags, HeaderFlags.UsePostDataDescriptor) - && (_directoryEntryHeader?.HasData == true) - && (_directoryEntryHeader?.CompressedSize != 0) - ) - { - return new ReadOnlySubStream(BaseStream, _directoryEntryHeader!.CompressedSize); - } + BaseStream.Position = Header.DataStartPosition.NotNull(); return BaseStream; } diff --git a/src/SharpCompress/Common/Zip/SeekableZipHeaderFactory.Async.cs b/src/SharpCompress/Common/Zip/SeekableZipHeaderFactory.Async.cs new file mode 100644 index 00000000..11e7eec4 --- /dev/null +++ b/src/SharpCompress/Common/Zip/SeekableZipHeaderFactory.Async.cs @@ -0,0 +1,162 @@ +using System; +using System.Buffers; +using System.Collections.Generic; +using System.IO; +using System.Threading.Tasks; +using SharpCompress.Common.Zip.Headers; +using SharpCompress.IO; + +namespace SharpCompress.Common.Zip; + +internal sealed partial class SeekableZipHeaderFactory +{ + internal async IAsyncEnumerable ReadSeekableHeaderAsync(Stream stream) + { +#if NET8_0_OR_GREATER + await using var reader = new AsyncBinaryReader(stream, leaveOpen: true); +#else + using var reader = new AsyncBinaryReader(stream, leaveOpen: true); +#endif + + await SeekBackToHeaderAsync(stream, reader).ConfigureAwait(false); + + var eocd_location = stream.Position; + var entry = new DirectoryEndHeader(); + await entry.Read(reader).ConfigureAwait(false); + + if (entry.IsZip64) + { + _zip64 = true; + + // ZIP64_END_OF_CENTRAL_DIRECTORY_LOCATOR should be before the EOCD + stream.Seek(eocd_location - ZIP64_EOCD_LENGTH - 4, SeekOrigin.Begin); + uint zip64_locator = await reader.ReadUInt32Async().ConfigureAwait(false); + if (zip64_locator != ZIP64_END_OF_CENTRAL_DIRECTORY_LOCATOR) + { + throw new ArchiveException("Failed to locate the Zip64 Directory Locator"); + } + + var zip64Locator = new Zip64DirectoryEndLocatorHeader(); + await zip64Locator.Read(reader).ConfigureAwait(false); + + stream.Seek(zip64Locator.RelativeOffsetOfTheEndOfDirectoryRecord, SeekOrigin.Begin); + var zip64Signature = await reader.ReadUInt32Async().ConfigureAwait(false); + if (zip64Signature != ZIP64_END_OF_CENTRAL_DIRECTORY) + { + throw new ArchiveException("Failed to locate the Zip64 Header"); + } + + var zip64Entry = new Zip64DirectoryEndHeader(); + await zip64Entry.Read(reader).ConfigureAwait(false); + stream.Seek(zip64Entry.DirectoryStartOffsetRelativeToDisk, SeekOrigin.Begin); + } + else + { + stream.Seek(entry.DirectoryStartOffsetRelativeToDisk, SeekOrigin.Begin); + } + + var position = stream.Position; + while (true) + { + stream.Position = position; + var signature = await reader.ReadUInt32Async().ConfigureAwait(false); + var nextHeader = await ReadHeader(signature, reader, _zip64).ConfigureAwait(false); + position = stream.Position; + + if (nextHeader is null) + { + yield break; + } + + if (nextHeader is DirectoryEntryHeader entryHeader) + { + //entry could be zero bytes so we need to know that. + entryHeader.HasData = entryHeader.CompressedSize != 0; + yield return entryHeader; + } + else if (nextHeader is DirectoryEndHeader endHeader) + { + yield return endHeader; + } + } + } + + private static async ValueTask SeekBackToHeaderAsync(Stream stream, AsyncBinaryReader reader) + { + // Minimum EOCD length + if (stream.Length < MINIMUM_EOCD_LENGTH) + { + throw new ArchiveException( + "Could not find Zip file Directory at the end of the file. File may be corrupted." + ); + } + + var len = + stream.Length < MAX_SEARCH_LENGTH_FOR_EOCD + ? (int)stream.Length + : MAX_SEARCH_LENGTH_FOR_EOCD; + + stream.Seek(-len, SeekOrigin.End); + var seek = ArrayPool.Shared.Rent(len); + + try + { + await reader.ReadBytesAsync(seek, 0, len, default).ConfigureAwait(false); + var memory = new Memory(seek, 0, len); + var span = memory.Span; + span.Reverse(); + + // don't exclude the minimum eocd region, otherwise you fail to locate the header in empty zip files + var max_search_area = len; // - MINIMUM_EOCD_LENGTH; + + for (var pos_from_end = 0; pos_from_end < max_search_area; ++pos_from_end) + { + if (IsMatch(span, pos_from_end, needle)) + { + stream.Seek(-pos_from_end, SeekOrigin.End); + return; + } + } + + throw new ArchiveException("Failed to locate the Zip Header"); + } + finally + { + ArrayPool.Shared.Return(seek); + } + } + + internal async ValueTask GetLocalHeaderAsync( + Stream stream, + DirectoryEntryHeader directoryEntryHeader + ) + { + stream.Seek(directoryEntryHeader.RelativeOffsetOfEntryHeader, SeekOrigin.Begin); +#if NET8_0_OR_GREATER + await using var reader = new AsyncBinaryReader(stream, leaveOpen: true); +#else + using var reader = new AsyncBinaryReader(stream, leaveOpen: true); +#endif + var signature = await reader.ReadUInt32Async().ConfigureAwait(false); + if ( + await ReadHeader(signature, reader, _zip64).ConfigureAwait(false) + is not LocalEntryHeader localEntryHeader + ) + { + throw new ArchiveOperationException(); + } + + // populate fields only known from the DirectoryEntryHeader + localEntryHeader.HasData = directoryEntryHeader.HasData; + localEntryHeader.ExternalFileAttributes = directoryEntryHeader.ExternalFileAttributes; + localEntryHeader.Comment = directoryEntryHeader.Comment; + + if (FlagUtility.HasFlag(localEntryHeader.Flags, HeaderFlags.UsePostDataDescriptor)) + { + localEntryHeader.Crc = directoryEntryHeader.Crc; + localEntryHeader.CompressedSize = directoryEntryHeader.CompressedSize; + localEntryHeader.UncompressedSize = directoryEntryHeader.UncompressedSize; + } + return localEntryHeader; + } +} diff --git a/src/SharpCompress/Common/Zip/SeekableZipHeaderFactory.cs b/src/SharpCompress/Common/Zip/SeekableZipHeaderFactory.cs index b99d0e2b..7a88a480 100644 --- a/src/SharpCompress/Common/Zip/SeekableZipHeaderFactory.cs +++ b/src/SharpCompress/Common/Zip/SeekableZipHeaderFactory.cs @@ -1,12 +1,14 @@ using System; +using System.Buffers; using System.Collections.Generic; using System.IO; +using System.Threading.Tasks; using SharpCompress.Common.Zip.Headers; using SharpCompress.IO; namespace SharpCompress.Common.Zip; -internal sealed class SeekableZipHeaderFactory : ZipHeaderFactory +internal sealed partial class SeekableZipHeaderFactory : ZipHeaderFactory { private const int MINIMUM_EOCD_LENGTH = 22; private const int ZIP64_EOCD_LENGTH = 20; @@ -15,7 +17,9 @@ internal sealed class SeekableZipHeaderFactory : ZipHeaderFactory private const int MAX_SEARCH_LENGTH_FOR_EOCD = 65557; private bool _zip64; - internal SeekableZipHeaderFactory(string? password, ArchiveEncoding archiveEncoding) + private static readonly byte[] needle = { 0x06, 0x05, 0x4b, 0x50 }; + + internal SeekableZipHeaderFactory(string? password, IArchiveEncoding archiveEncoding) : base(StreamingMode.Seekable, password, archiveEncoding) { } internal IEnumerable ReadSeekableHeader(Stream stream) @@ -85,7 +89,7 @@ internal sealed class SeekableZipHeaderFactory : ZipHeaderFactory } } - private static bool IsMatch(byte[] haystack, int position, byte[] needle) + private static bool IsMatch(Span haystack, int position, byte[] needle) { for (var i = 0; i < needle.Length; i++) { @@ -112,9 +116,6 @@ internal sealed class SeekableZipHeaderFactory : ZipHeaderFactory stream.Length < MAX_SEARCH_LENGTH_FOR_EOCD ? (int)stream.Length : MAX_SEARCH_LENGTH_FOR_EOCD; - // We search for marker in reverse to find the first occurance - byte[] needle = { 0x06, 0x05, 0x4b, 0x50 }; - stream.Seek(-len, SeekOrigin.End); var seek = reader.ReadBytes(len); @@ -147,7 +148,20 @@ internal sealed class SeekableZipHeaderFactory : ZipHeaderFactory var signature = reader.ReadUInt32(); if (ReadHeader(signature, reader, _zip64) is not LocalEntryHeader localEntryHeader) { - throw new InvalidOperationException(); + throw new ArchiveOperationException(); + } + + // populate fields only known from the DirectoryEntryHeader + localEntryHeader.HasData = directoryEntryHeader.HasData; + localEntryHeader.ExternalFileAttributes = directoryEntryHeader.ExternalFileAttributes; + localEntryHeader.Comment = directoryEntryHeader.Comment; + + if (FlagUtility.HasFlag(localEntryHeader.Flags, HeaderFlags.UsePostDataDescriptor)) + { + localEntryHeader.Crc = directoryEntryHeader.Crc; + localEntryHeader.IsCrcAvailable = true; + localEntryHeader.CompressedSize = directoryEntryHeader.CompressedSize; + localEntryHeader.UncompressedSize = directoryEntryHeader.UncompressedSize; } return localEntryHeader; } diff --git a/src/SharpCompress/Common/Zip/StreamingZipFilePart.Async.cs b/src/SharpCompress/Common/Zip/StreamingZipFilePart.Async.cs new file mode 100644 index 00000000..dbd62841 --- /dev/null +++ b/src/SharpCompress/Common/Zip/StreamingZipFilePart.Async.cs @@ -0,0 +1,31 @@ +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.IO; + +namespace SharpCompress.Common.Zip; + +internal sealed partial class StreamingZipFilePart +{ + internal override async ValueTask GetCompressedStreamAsync( + CancellationToken cancellationToken = default + ) + { + if (!Header.HasData) + { + return Stream.Null; + } + _decompressionStream = await CreateDecompressionStreamAsync( + await GetCryptoStreamAsync(CreateBaseStream(), cancellationToken) + .ConfigureAwait(false), + Header.CompressionMethod, + cancellationToken + ) + .ConfigureAwait(false); + if (LeaveStreamOpen) + { + return SharpCompressStream.CreateNonDisposing(_decompressionStream); + } + return _decompressionStream; + } +} diff --git a/src/SharpCompress/Common/Zip/StreamingZipFilePart.cs b/src/SharpCompress/Common/Zip/StreamingZipFilePart.cs index 1cd1f61f..e8ef7c55 100644 --- a/src/SharpCompress/Common/Zip/StreamingZipFilePart.cs +++ b/src/SharpCompress/Common/Zip/StreamingZipFilePart.cs @@ -1,19 +1,23 @@ using System.IO; -using System.Net.Sockets; using SharpCompress.Common.Zip.Headers; -using SharpCompress.Compressors.Deflate; +using SharpCompress.Compressors; using SharpCompress.IO; +using SharpCompress.Providers; namespace SharpCompress.Common.Zip; -internal sealed class StreamingZipFilePart : ZipFilePart +internal sealed partial class StreamingZipFilePart : ZipFilePart { private Stream? _decompressionStream; - internal StreamingZipFilePart(ZipFileEntry header, Stream stream) - : base(header, stream) { } + internal StreamingZipFilePart( + ZipFileEntry header, + Stream stream, + CompressionProviderRegistry compressionProviders + ) + : base(header, stream, compressionProviders) { } - protected override Stream CreateBaseStream() => Header.PackedStream; + protected override Stream CreateBaseStream() => Header.PackedStream.NotNull(); internal override Stream GetCompressedStream() { @@ -27,16 +31,16 @@ internal sealed class StreamingZipFilePart : ZipFilePart ); if (LeaveStreamOpen) { - return NonDisposingStream.Create(_decompressionStream); + return SharpCompressStream.CreateNonDisposing(_decompressionStream); } return _decompressionStream; } - internal BinaryReader FixStreamedFileLocation(ref RewindableStream rewindableStream) + internal BinaryReader FixStreamedFileLocation(ref Stream stream) { if (Header.IsDirectory) { - return new BinaryReader(rewindableStream); + return new BinaryReader(stream, System.Text.Encoding.Default, leaveOpen: true); } if (Header.HasData && !Skipped) @@ -48,14 +52,9 @@ internal sealed class StreamingZipFilePart : ZipFilePart // If we had TotalIn / TotalOut we could have used them Header.CompressedSize = _decompressionStream.Position; - if (_decompressionStream is DeflateStream deflateStream) - { - rewindableStream.Rewind(deflateStream.InputBuffer); - } - Skipped = true; } - var reader = new BinaryReader(rewindableStream); + var reader = new BinaryReader(stream, System.Text.Encoding.Default, leaveOpen: true); _decompressionStream = null; return reader; } diff --git a/src/SharpCompress/Common/Zip/StreamingZipHeaderFactory.Async.cs b/src/SharpCompress/Common/Zip/StreamingZipHeaderFactory.Async.cs new file mode 100644 index 00000000..f6ddfb2c --- /dev/null +++ b/src/SharpCompress/Common/Zip/StreamingZipHeaderFactory.Async.cs @@ -0,0 +1,346 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Zip.Headers; +using SharpCompress.IO; + +namespace SharpCompress.Common.Zip; + +internal sealed partial class StreamingZipHeaderFactory +{ + /// + /// Reads ZIP headers asynchronously for streams that do not support synchronous reads. + /// + internal IAsyncEnumerable ReadStreamHeaderAsync(Stream stream) => + new StreamHeaderAsyncEnumerable(this, stream); + + /// + /// Invokes the shared async header parsing logic on the base factory. + /// + private ValueTask ReadHeaderAsyncInternal( + uint headerBytes, + AsyncBinaryReader reader + ) => ReadHeader(headerBytes, reader); + + /// + /// Exposes the last parsed local entry header to the async enumerator so it can handle streaming data descriptors. + /// + private LocalEntryHeader? LastEntryHeader + { + get => _lastEntryHeader; + set => _lastEntryHeader = value; + } + + /// + /// Produces an async enumerator for streaming ZIP headers. + /// + private sealed class StreamHeaderAsyncEnumerable : IAsyncEnumerable + { + private readonly StreamingZipHeaderFactory _headerFactory; + private readonly Stream _stream; + + public StreamHeaderAsyncEnumerable(StreamingZipHeaderFactory headerFactory, Stream stream) + { + _headerFactory = headerFactory; + _stream = stream; + } + + public IAsyncEnumerator GetAsyncEnumerator( + CancellationToken cancellationToken = default + ) => new StreamHeaderAsyncEnumerator(_headerFactory, _stream, cancellationToken); + } + + /// + /// Async implementation of reading stream headers using to avoid sync reads. + /// + private sealed class StreamHeaderAsyncEnumerator : IAsyncEnumerator, IDisposable + { + private readonly StreamingZipHeaderFactory _headerFactory; + private readonly SharpCompressStream _sharpCompressStream; + private readonly AsyncBinaryReader _reader; + private readonly CancellationToken _cancellationToken; + private bool _completed; + + public StreamHeaderAsyncEnumerator( + StreamingZipHeaderFactory headerFactory, + Stream stream, + CancellationToken cancellationToken + ) + { + _headerFactory = headerFactory; + // Use Create to avoid double-wrapping if stream is already a SharpCompressStream, + // and to preserve seekability for DataDescriptorStream which needs to seek backward + _sharpCompressStream = SharpCompressStream.Create(stream); + _reader = new AsyncBinaryReader(_sharpCompressStream, leaveOpen: true); + _cancellationToken = cancellationToken; + } + + private ZipHeader? _current; + + public ZipHeader Current => + _current ?? throw new ArchiveOperationException("No current header is available."); + + /// + /// Advances to the next ZIP header in the stream, honoring streaming data descriptors where applicable. + /// + public async ValueTask MoveNextAsync() + { + if (_completed) + { + return false; + } + + while (true) + { + _cancellationToken.ThrowIfCancellationRequested(); + + uint headerBytes; + var lastEntryHeader = _headerFactory.LastEntryHeader; + if ( + lastEntryHeader != null + && FlagUtility.HasFlag(lastEntryHeader.Flags, HeaderFlags.UsePostDataDescriptor) + ) + { + if (lastEntryHeader.Part is null) + { + continue; + } + + var pos = _sharpCompressStream.CanSeek + ? (long?)_sharpCompressStream.Position + : null; + + var crc = await _reader + .ReadUInt32Async(_cancellationToken) + .ConfigureAwait(false); + if (crc == POST_DATA_DESCRIPTOR) + { + crc = await _reader + .ReadUInt32Async(_cancellationToken) + .ConfigureAwait(false); + } + lastEntryHeader.Crc = crc; + lastEntryHeader.IsCrcAvailable = true; + + //attempt 32bit read + ulong compressedSize = await _reader + .ReadUInt32Async(_cancellationToken) + .ConfigureAwait(false); + ulong uncompressedSize = await _reader + .ReadUInt32Async(_cancellationToken) + .ConfigureAwait(false); + headerBytes = await _reader + .ReadUInt32Async(_cancellationToken) + .ConfigureAwait(false); + + //check for zip64 sentinel or unexpected header + bool isSentinel = + compressedSize == 0xFFFFFFFF || uncompressedSize == 0xFFFFFFFF; + bool isHeader = headerBytes == 0x04034b50 || headerBytes == 0x02014b50; + + if (!isHeader && !isSentinel) + { + //reshuffle into 64-bit values + compressedSize = (uncompressedSize << 32) | compressedSize; + uncompressedSize = + ((ulong)headerBytes << 32) + | await _reader + .ReadUInt32Async(_cancellationToken) + .ConfigureAwait(false); + headerBytes = await _reader + .ReadUInt32Async(_cancellationToken) + .ConfigureAwait(false); + } + else if (isSentinel) + { + //standards-compliant zip64 descriptor + compressedSize = await _reader + .ReadUInt64Async(_cancellationToken) + .ConfigureAwait(false); + uncompressedSize = await _reader + .ReadUInt64Async(_cancellationToken) + .ConfigureAwait(false); + } + + lastEntryHeader.CompressedSize = (long)compressedSize; + lastEntryHeader.UncompressedSize = (long)uncompressedSize; + + if (pos.HasValue) + { + lastEntryHeader.DataStartPosition = pos - lastEntryHeader.CompressedSize; + } + } + else if (lastEntryHeader != null && lastEntryHeader.IsZip64) + { + if (lastEntryHeader.Part is null) + { + continue; + } + + var pos = _sharpCompressStream.CanSeek + ? (long?)_sharpCompressStream.Position + : null; + + headerBytes = await _reader + .ReadUInt32Async(_cancellationToken) + .ConfigureAwait(false); + + // A Zip64 entry that does not use a post-data descriptor stores its real CRC + // and sizes in the local header's Zip64 extra field, so a header signature + // (the next local entry, or the central directory) follows the data directly. + // In that case the entry's metadata is already correct and must not be + // overwritten with bytes read from the following header. We have only consumed + // the 4-byte signature, so fall through and parse this header normally. Because + // no seek-back is required here, this also works for non-seekable streams. + if (headerBytes == 0x04034b50 || headerBytes == 0x02014b50) + { + if (pos.HasValue) + { + lastEntryHeader.DataStartPosition = + pos - lastEntryHeader.CompressedSize; + } + } + else + { + // A data descriptor follows. Recover the CRC and sizes from it; the + // descriptor can carry either 32-bit or 64-bit sizes. + _ = await _reader.ReadUInt16Async(_cancellationToken).ConfigureAwait(false); // version + _ = await _reader.ReadUInt16Async(_cancellationToken).ConfigureAwait(false); // flags + _ = await _reader.ReadUInt16Async(_cancellationToken).ConfigureAwait(false); // compressionMethod + _ = await _reader.ReadUInt16Async(_cancellationToken).ConfigureAwait(false); // lastModifiedDate + _ = await _reader.ReadUInt16Async(_cancellationToken).ConfigureAwait(false); // lastModifiedTime + + var crc = await _reader + .ReadUInt32Async(_cancellationToken) + .ConfigureAwait(false); + + if (crc == POST_DATA_DESCRIPTOR) + { + crc = await _reader + .ReadUInt32Async(_cancellationToken) + .ConfigureAwait(false); + } + lastEntryHeader.Crc = crc; + lastEntryHeader.IsCrcAvailable = true; + + // The DataDescriptor can be either 64bit or 32bit + var compressedSize = await _reader + .ReadUInt32Async(_cancellationToken) + .ConfigureAwait(false); + var uncompressedSize = await _reader + .ReadUInt32Async(_cancellationToken) + .ConfigureAwait(false); + + var test64Bit = ((long)uncompressedSize << 32) | compressedSize; + if (test64Bit == lastEntryHeader.CompressedSize) + { + lastEntryHeader.UncompressedSize = + ( + (long) + await _reader + .ReadUInt32Async(_cancellationToken) + .ConfigureAwait(false) << 32 + ) | headerBytes; + headerBytes = await _reader + .ReadUInt32Async(_cancellationToken) + .ConfigureAwait(false); + } + else + { + lastEntryHeader.UncompressedSize = uncompressedSize; + } + + if (pos.HasValue) + { + lastEntryHeader.DataStartPosition = + pos - lastEntryHeader.CompressedSize; + + // 4 = First 4 bytes of the entry header (i.e. 50 4B 03 04) + _sharpCompressStream.Position = pos.Value + 4; + } + } + } + else + { + headerBytes = await _reader + .ReadUInt32Async(_cancellationToken) + .ConfigureAwait(false); + } + + _headerFactory.LastEntryHeader = null; + var header = await _headerFactory + .ReadHeaderAsyncInternal(headerBytes, _reader) + .ConfigureAwait(false); + if (header is null) + { + _completed = true; + return false; + } + + //entry could be zero bytes so we need to know that. + if (header.ZipHeaderType == ZipHeaderType.LocalEntry) + { + var localHeader = (LocalEntryHeader)header; + var directoryHeader = _headerFactory._entries?.FirstOrDefault(entry => + entry.Key == localHeader.Name + && localHeader.CompressedSize == 0 + && localHeader.UncompressedSize == 0 + && localHeader.Crc == 0 + && localHeader.IsDirectory == false + ); + + if (directoryHeader != null) + { + localHeader.UncompressedSize = directoryHeader.Size; + localHeader.CompressedSize = directoryHeader.CompressedSize; + localHeader.Crc = (uint)directoryHeader.Crc; + localHeader.IsCrcAvailable = true; + } + + // If we have CompressedSize, there is data to be read + if (localHeader.CompressedSize > 0) + { + header.HasData = true; + } // Check if zip is streaming ( Length is 0 and is declared in PostDataDescriptor ) + else if (localHeader.Flags.HasFlag(HeaderFlags.UsePostDataDescriptor)) + { + // Peek ahead to check if next data is a header or file data. + // Use the IStreamStack.Rewind mechanism to give back the peeked bytes. + var nextHeaderBytes = await _reader + .ReadUInt32Async(_cancellationToken) + .ConfigureAwait(false); + _sharpCompressStream.Rewind(sizeof(uint)); + + // Check if next data is PostDataDescriptor, streamed file with 0 length + header.HasData = !IsHeader(nextHeaderBytes); + } + else // We are not streaming and compressed size is 0, we have no data + { + header.HasData = false; + } + } + + _current = header; + return true; + } + } + + public ValueTask DisposeAsync() + { + Dispose(); + return default; + } + + /// + /// Disposes the underlying reader (without closing the archive stream). + /// + public void Dispose() + { + _reader.Dispose(); + } + } +} diff --git a/src/SharpCompress/Common/Zip/StreamingZipHeaderFactory.cs b/src/SharpCompress/Common/Zip/StreamingZipHeaderFactory.cs index e76a2856..8e775727 100644 --- a/src/SharpCompress/Common/Zip/StreamingZipHeaderFactory.cs +++ b/src/SharpCompress/Common/Zip/StreamingZipHeaderFactory.cs @@ -1,3 +1,4 @@ +using System; using System.Collections.Generic; using System.IO; using System.Linq; @@ -6,137 +7,235 @@ using SharpCompress.IO; namespace SharpCompress.Common.Zip; -internal class StreamingZipHeaderFactory : ZipHeaderFactory +internal partial class StreamingZipHeaderFactory : ZipHeaderFactory { private IEnumerable? _entries; internal StreamingZipHeaderFactory( string? password, - ArchiveEncoding archiveEncoding, + IArchiveEncoding archiveEncoding, IEnumerable? entries ) - : base(StreamingMode.Streaming, password, archiveEncoding) - { - _entries = entries; - } + : base(StreamingMode.Streaming, password, archiveEncoding) => _entries = entries; internal IEnumerable ReadStreamHeader(Stream stream) { - RewindableStream rewindableStream; + // Use Create to avoid double-wrapping if stream is already a SharpCompressStream, + // and to preserve seekability for DataDescriptorStream which needs to seek backward + var sharpCompressStream = SharpCompressStream.Create(stream); + var reader = new BinaryReader( + sharpCompressStream, + System.Text.Encoding.Default, + leaveOpen: true + ); - if (stream is RewindableStream rs) + try { - rewindableStream = rs; - } - else - { - rewindableStream = new RewindableStream(stream); - } - while (true) - { - ZipHeader? header; - var reader = new BinaryReader(rewindableStream); - uint headerBytes = 0; - if ( - _lastEntryHeader != null - && ( - FlagUtility.HasFlag(_lastEntryHeader.Flags, HeaderFlags.UsePostDataDescriptor) - || _lastEntryHeader.IsZip64 - ) - ) + while (true) { - reader = ((StreamingZipFilePart)_lastEntryHeader.Part).FixStreamedFileLocation( - ref rewindableStream - ); - var pos = rewindableStream.CanSeek ? (long?)rewindableStream.Position : null; - var crc = reader.ReadUInt32(); - if (crc == POST_DATA_DESCRIPTOR) + uint headerBytes = 0; + if ( + _lastEntryHeader != null + && FlagUtility.HasFlag( + _lastEntryHeader.Flags, + HeaderFlags.UsePostDataDescriptor + ) + ) { - crc = reader.ReadUInt32(); - } - _lastEntryHeader.Crc = crc; + if (_lastEntryHeader.Part is null) + { + continue; + } - // The DataDescriptor can be either 64bit or 32bit - var compressed_size = reader.ReadUInt32(); - var uncompressed_size = reader.ReadUInt32(); + // removed requirement for FixStreamedFileLocation() - // Check if we have header or 64bit DataDescriptor - headerBytes = reader.ReadUInt32(); - var test_header = !(headerBytes == 0x04034b50 || headerBytes == 0x02014b50); + var pos = sharpCompressStream.CanSeek + ? (long?)sharpCompressStream.Position + : null; - var test_64bit = ((long)uncompressed_size << 32) | compressed_size; - if (test_64bit == _lastEntryHeader.CompressedSize && test_header) - { - _lastEntryHeader.UncompressedSize = - ((long)reader.ReadUInt32() << 32) | headerBytes; + var crc = reader.ReadUInt32(); + if (crc == POST_DATA_DESCRIPTOR) + { + crc = reader.ReadUInt32(); + } + _lastEntryHeader.Crc = crc; + _lastEntryHeader.IsCrcAvailable = true; + + //attempt 32bit read + ulong compSize = reader.ReadUInt32(); + ulong uncompSize = reader.ReadUInt32(); headerBytes = reader.ReadUInt32(); + + //check for zip64 sentinel or unexpected header + bool isSentinel = compSize == 0xFFFFFFFF || uncompSize == 0xFFFFFFFF; + bool isHeader = headerBytes == 0x04034b50 || headerBytes == 0x02014b50; + + if (!isHeader && !isSentinel) + { + //reshuffle into 64-bit values + compSize = (uncompSize << 32) | compSize; + uncompSize = ((ulong)headerBytes << 32) | reader.ReadUInt32(); + headerBytes = reader.ReadUInt32(); + } + else if (isSentinel) + { + //standards-compliant zip64 descriptor + compSize = reader.ReadUInt64(); + uncompSize = reader.ReadUInt64(); + } + + _lastEntryHeader.CompressedSize = (long)compSize; + _lastEntryHeader.UncompressedSize = (long)uncompSize; + + if (pos.HasValue) + { + _lastEntryHeader.DataStartPosition = pos - _lastEntryHeader.CompressedSize; + } + } + else if (_lastEntryHeader != null && _lastEntryHeader.IsZip64) + { + if (_lastEntryHeader.Part is null) + { + continue; + } + + //reader = ((StreamingZipFilePart)_lastEntryHeader.Part).FixStreamedFileLocation( + // ref sharpCompressStream + //); + + var pos = sharpCompressStream.CanSeek + ? (long?)sharpCompressStream.Position + : null; + + headerBytes = reader.ReadUInt32(); + + // A Zip64 entry that does not use a post-data descriptor stores its real CRC + // and sizes in the local header's Zip64 extra field, so a header signature + // (the next local entry, or the central directory) follows the data directly. + // In that case the entry's metadata is already correct and must not be + // overwritten with bytes read from the following header. We have only consumed + // the 4-byte signature, so fall through and parse this header normally. + if (headerBytes == 0x04034b50 || headerBytes == 0x02014b50) + { + if (pos.HasValue) + { + _lastEntryHeader.DataStartPosition = + pos - _lastEntryHeader.CompressedSize; + } + } + else + { + // A data descriptor follows. Recover the CRC and sizes from it; the + // descriptor can carry either 32-bit or 64-bit sizes. + _ = reader.ReadUInt16(); // version + _ = reader.ReadUInt16(); // flags + _ = reader.ReadUInt16(); // compressionMethod + _ = reader.ReadUInt16(); // lastModifiedDate + _ = reader.ReadUInt16(); // lastModifiedTime + + var crc = reader.ReadUInt32(); + + if (crc == POST_DATA_DESCRIPTOR) + { + crc = reader.ReadUInt32(); + } + _lastEntryHeader.Crc = crc; + _lastEntryHeader.IsCrcAvailable = true; + + // The DataDescriptor can be either 64bit or 32bit + var compressed_size = reader.ReadUInt32(); + var uncompressed_size = reader.ReadUInt32(); + + var test_64bit = ((long)uncompressed_size << 32) | compressed_size; + if (test_64bit == _lastEntryHeader.CompressedSize) + { + _lastEntryHeader.UncompressedSize = + ((long)reader.ReadUInt32() << 32) | headerBytes; + headerBytes = reader.ReadUInt32(); + } + else + { + _lastEntryHeader.UncompressedSize = uncompressed_size; + } + + if (pos.HasValue) + { + _lastEntryHeader.DataStartPosition = + pos - _lastEntryHeader.CompressedSize; + + // 4 = First 4 bytes of the entry header (i.e. 50 4B 03 04) + sharpCompressStream.Position = pos.Value + 4; + } + } } else { - _lastEntryHeader.UncompressedSize = uncompressed_size; + try + { + headerBytes = reader.ReadUInt32(); + } + catch (EndOfStreamException ex) + { + throw new InvalidFormatException( + "Unexpected end of stream while reading ZIP archive", + ex + ); + } } - if (pos.HasValue) + _lastEntryHeader = null; + var header = ReadHeader(headerBytes, reader); + if (header is null) { - _lastEntryHeader.DataStartPosition = pos - _lastEntryHeader.CompressedSize; + yield break; } - } - else - { - headerBytes = reader.ReadUInt32(); - } - _lastEntryHeader = null; - header = ReadHeader(headerBytes, reader); - if (header is null) - { - yield break; - } - - //entry could be zero bytes so we need to know that. - if (header.ZipHeaderType == ZipHeaderType.LocalEntry) - { - var local_header = ((LocalEntryHeader)header); - var dir_header = _entries?.FirstOrDefault( - entry => + //entry could be zero bytes so we need to know that. + if (header.ZipHeaderType == ZipHeaderType.LocalEntry) + { + var local_header = ((LocalEntryHeader)header); + var dir_header = _entries?.FirstOrDefault(entry => entry.Key == local_header.Name && local_header.CompressedSize == 0 && local_header.UncompressedSize == 0 && local_header.Crc == 0 && local_header.IsDirectory == false - ); + ); - if (dir_header != null) - { - local_header.UncompressedSize = dir_header.Size; - local_header.CompressedSize = dir_header.CompressedSize; - local_header.Crc = (uint)dir_header.Crc; - } - - // If we have CompressedSize, there is data to be read - if (local_header.CompressedSize > 0) - { - header.HasData = true; - } // Check if zip is streaming ( Length is 0 and is declared in PostDataDescriptor ) - else if (local_header.Flags.HasFlag(HeaderFlags.UsePostDataDescriptor)) - { - var isRecording = rewindableStream.IsRecording; - if (!isRecording) + if (dir_header != null) { - rewindableStream.StartRecording(); + local_header.UncompressedSize = dir_header.Size; + local_header.CompressedSize = dir_header.CompressedSize; + local_header.Crc = (uint)dir_header.Crc; + local_header.IsCrcAvailable = true; } - var nextHeaderBytes = reader.ReadUInt32(); - // Check if next data is PostDataDescriptor, streamed file with 0 length - header.HasData = !IsHeader(nextHeaderBytes); - rewindableStream.Rewind(!isRecording); - } - else // We are not streaming and compressed size is 0, we have no data - { - header.HasData = false; + // If we have CompressedSize, there is data to be read + if (local_header.CompressedSize > 0) + { + header.HasData = true; + } // Check if zip is streaming ( Length is 0 and is declared in PostDataDescriptor ) + else if (local_header.Flags.HasFlag(HeaderFlags.UsePostDataDescriptor)) + { + // Peek ahead to check if next data is a header or file data. + // Use the IStreamStack.Rewind mechanism to give back the peeked bytes. + var nextHeaderBytes = reader.ReadUInt32(); + sharpCompressStream.Rewind(sizeof(uint)); + + // Check if next data is PostDataDescriptor, streamed file with 0 length + header.HasData = !IsHeader(nextHeaderBytes); + } + else // We are not streaming and compressed size is 0, we have no data + { + header.HasData = false; + } } + yield return header; } - yield return header; + } + finally + { + reader.Dispose(); } } } diff --git a/src/SharpCompress/Common/Zip/WinzipAesCryptoStream.Async.cs b/src/SharpCompress/Common/Zip/WinzipAesCryptoStream.Async.cs new file mode 100644 index 00000000..d68b8a87 --- /dev/null +++ b/src/SharpCompress/Common/Zip/WinzipAesCryptoStream.Async.cs @@ -0,0 +1,120 @@ +using System; +using System.Buffers; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Common.Zip; + +internal partial class WinzipAesCryptoStream +{ +#if !LEGACY_DOTNET + public override async ValueTask DisposeAsync() + { + if (_isDisposed) + { + await base.DisposeAsync().ConfigureAwait(false); + return; + } + _isDisposed = true; + // Read out last 10 auth bytes asynchronously + byte[] authBytes = ArrayPool.Shared.Rent(10); + try + { + await _stream.ReadFullyAsync(authBytes, 0, 10).ConfigureAwait(false); + } + finally + { + ArrayPool.Shared.Return(authBytes); + await _stream.DisposeAsync().ConfigureAwait(false); + } + await base.DisposeAsync().ConfigureAwait(false); + } +#endif + + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + if (_totalBytesLeftToRead == 0) + { + return 0; + } + var bytesToRead = count; + if (count > _totalBytesLeftToRead) + { + bytesToRead = (int)_totalBytesLeftToRead; + } + var read = await _stream + .ReadAsync(buffer, offset, bytesToRead, cancellationToken) + .ConfigureAwait(false); + _totalBytesLeftToRead -= read; + + ReadTransformBlocks(buffer, offset, read); + + return read; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + if (_totalBytesLeftToRead == 0) + { + return 0; + } + var bytesToRead = buffer.Length; + if (buffer.Length > _totalBytesLeftToRead) + { + bytesToRead = (int)_totalBytesLeftToRead; + } + var read = await _stream + .ReadAsync(buffer.Slice(0, bytesToRead), cancellationToken) + .ConfigureAwait(false); + _totalBytesLeftToRead -= read; + + ReadTransformBlocks(buffer.Span, read); + + return read; + } + + private void ReadTransformBlocks(Span buffer, int count) + { + var posn = 0; + var remaining = count; + + while (posn < buffer.Length && remaining > 0) + { + var n = ReadTransformOneBlock(buffer, posn, remaining); + posn += n; + remaining -= n; + } + } + + private int ReadTransformOneBlock(Span buffer, int offset, int remaining) + { + if (_counterOutOffset == BLOCK_SIZE_IN_BYTES) + { + FillCounterOut(); + } + + var bytesToXor = Math.Min(BLOCK_SIZE_IN_BYTES - _counterOutOffset, remaining); + XorInPlace(buffer, offset, bytesToXor, _counterOutOffset); + _counterOutOffset += bytesToXor; + return bytesToXor; + } + + private void XorInPlace(Span buffer, int offset, int count, int counterOffset) + { + for (var i = 0; i < count; i++) + { + buffer[offset + i] = (byte)(_counterOut[counterOffset + i] ^ buffer[offset + i]); + } + } +#endif +} diff --git a/src/SharpCompress/Common/Zip/WinzipAesCryptoStream.cs b/src/SharpCompress/Common/Zip/WinzipAesCryptoStream.cs index 093ac034..9d269473 100644 --- a/src/SharpCompress/Common/Zip/WinzipAesCryptoStream.cs +++ b/src/SharpCompress/Common/Zip/WinzipAesCryptoStream.cs @@ -1,20 +1,22 @@ using System; +using System.Buffers; using System.Buffers.Binary; using System.IO; using System.Security.Cryptography; +using System.Threading.Tasks; namespace SharpCompress.Common.Zip; -internal class WinzipAesCryptoStream : Stream +internal partial class WinzipAesCryptoStream : Stream { private const int BLOCK_SIZE_IN_BYTES = 16; - private readonly SymmetricAlgorithm _cipher; + private readonly Aes _cipher; private readonly byte[] _counter = new byte[BLOCK_SIZE_IN_BYTES]; private readonly Stream _stream; private readonly ICryptoTransform _transform; private int _nonce = 1; private byte[] _counterOut = new byte[BLOCK_SIZE_IN_BYTES]; - private bool _isFinalBlock; + private int _counterOutOffset = BLOCK_SIZE_IN_BYTES; private long _totalBytesLeftToRead; private bool _isDisposed; @@ -33,7 +35,7 @@ internal class WinzipAesCryptoStream : Stream _transform = _cipher.CreateEncryptor(winzipAesEncryptionData.KeyBytes, iv); } - private SymmetricAlgorithm CreateCipher(WinzipAesEncryptionData winzipAesEncryptionData) + private Aes CreateCipher(WinzipAesEncryptionData winzipAesEncryptionData) { var cipher = Aes.Create(); cipher.BlockSize = BLOCK_SIZE_IN_BYTES * 8; @@ -61,19 +63,46 @@ internal class WinzipAesCryptoStream : Stream { if (_isDisposed) { + base.Dispose(disposing); return; } _isDisposed = true; if (disposing) { - //read out last 10 auth bytes - Span ten = stackalloc byte[10]; - _stream.ReadFully(ten); + // Read out last 10 auth bytes - catch exceptions for async-only streams + if (Utility.UseSyncOverAsyncDispose()) + { + var ten = ArrayPool.Shared.Rent(10); + try + { +#pragma warning disable VSTHRD002 // Avoid problematic synchronous waits +#pragma warning disable CA2012 + _stream.ReadFullyAsync(ten, 0, 10).GetAwaiter().GetResult(); +#pragma warning restore CA2012 +#pragma warning restore VSTHRD002 // Avoid problematic synchronous waits + } + finally + { + ArrayPool.Shared.Return(ten); + } + } + else + { + Span ten = stackalloc byte[10]; + _stream.ReadFully(ten); + } _stream.Dispose(); } + base.Dispose(disposing); } - public override void Flush() => throw new NotSupportedException(); + private async ValueTask ReadAuthBytesAsync() + { + byte[] authBytes = new byte[10]; + await _stream.ReadFullyAsync(authBytes, 0, 10).ConfigureAwait(false); + } + + public override void Flush() { } public override int Read(byte[] buffer, int offset, int count) { @@ -94,58 +123,45 @@ internal class WinzipAesCryptoStream : Stream return read; } - private int ReadTransformOneBlock(byte[] buffer, int offset, int last) + private void FillCounterOut() { - if (_isFinalBlock) - { - throw new InvalidOperationException(); - } - - var bytesRemaining = last - offset; - var bytesToRead = - (bytesRemaining > BLOCK_SIZE_IN_BYTES) ? BLOCK_SIZE_IN_BYTES : bytesRemaining; - // update the counter BinaryPrimitives.WriteInt32LittleEndian(_counter, _nonce++); - - // Determine if this is the final block - if ((bytesToRead == bytesRemaining) && (_totalBytesLeftToRead == 0)) - { - _counterOut = _transform.TransformFinalBlock(_counter, 0, BLOCK_SIZE_IN_BYTES); - _isFinalBlock = true; - } - else - { - _transform.TransformBlock( - _counter, - 0, // offset - BLOCK_SIZE_IN_BYTES, - _counterOut, - 0 - ); // offset - } - - XorInPlace(buffer, offset, bytesToRead); - return bytesToRead; + _transform.TransformBlock( + _counter, + 0, // offset + BLOCK_SIZE_IN_BYTES, + _counterOut, + 0 + ); // offset + _counterOutOffset = 0; } - private void XorInPlace(byte[] buffer, int offset, int count) + private void XorInPlace(byte[] buffer, int offset, int count, int counterOffset) { for (var i = 0; i < count; i++) { - buffer[offset + i] = (byte)(_counterOut[i] ^ buffer[offset + i]); + buffer[offset + i] = (byte)(_counterOut[counterOffset + i] ^ buffer[offset + i]); } } private void ReadTransformBlocks(byte[] buffer, int offset, int count) { var posn = offset; - var last = count + offset; + var remaining = count; - while (posn < buffer.Length && posn < last) + while (posn < buffer.Length && remaining > 0) { - var n = ReadTransformOneBlock(buffer, posn, last); - posn += n; + if (_counterOutOffset == BLOCK_SIZE_IN_BYTES) + { + FillCounterOut(); + } + + var bytesToXor = Math.Min(BLOCK_SIZE_IN_BYTES - _counterOutOffset, remaining); + XorInPlace(buffer, posn, bytesToXor, _counterOutOffset); + _counterOutOffset += bytesToXor; + posn += bytesToXor; + remaining -= bytesToXor; } } diff --git a/src/SharpCompress/Common/Zip/WinzipAesEncryptionData.cs b/src/SharpCompress/Common/Zip/WinzipAesEncryptionData.cs index 251b919e..40309c3d 100644 --- a/src/SharpCompress/Common/Zip/WinzipAesEncryptionData.cs +++ b/src/SharpCompress/Common/Zip/WinzipAesEncryptionData.cs @@ -1,21 +1,21 @@ -#nullable disable - using System; using System.Buffers.Binary; +using System.Diagnostics.CodeAnalysis; using System.Security.Cryptography; +using System.Text; namespace SharpCompress.Common.Zip; +[SuppressMessage( + "Security", + "CA5379:Rfc2898DeriveBytes might be using a weak hash algorithm", + Justification = "WinZip AES specification requires PBKDF2 with SHA-1." +)] internal class WinzipAesEncryptionData { private const int RFC2898_ITERATIONS = 1000; - private readonly byte[] _salt; private readonly WinzipAesKeySize _keySize; - private readonly byte[] _passwordVerifyValue; - private readonly string _password; - - private byte[] _generatedVerifyValue; internal WinzipAesEncryptionData( WinzipAesKeySize keySize, @@ -25,10 +25,43 @@ internal class WinzipAesEncryptionData ) { _keySize = keySize; - _salt = salt; - _passwordVerifyValue = passwordVerifyValue; - _password = password; - Initialize(); + +#if LEGACY_DOTNET + var rfc2898 = new Rfc2898DeriveBytes(password, salt, RFC2898_ITERATIONS); + KeyBytes = rfc2898.GetBytes(KeySizeInBytes); + IvBytes = rfc2898.GetBytes(KeySizeInBytes); + var generatedVerifyValue = rfc2898.GetBytes(2); +#elif NET10_0_OR_GREATER + var derivedKeySize = (KeySizeInBytes * 2) + 2; + var passwordBytes = Encoding.UTF8.GetBytes(password); + var derivedKey = Rfc2898DeriveBytes.Pbkdf2( + passwordBytes, + salt, + RFC2898_ITERATIONS, + HashAlgorithmName.SHA1, + derivedKeySize + ); + KeyBytes = derivedKey.AsSpan(0, KeySizeInBytes).ToArray(); + IvBytes = derivedKey.AsSpan(KeySizeInBytes, KeySizeInBytes).ToArray(); + var generatedVerifyValue = derivedKey.AsSpan((KeySizeInBytes * 2), 2).ToArray(); +#else + var rfc2898 = new Rfc2898DeriveBytes( + password, + salt, + RFC2898_ITERATIONS, + HashAlgorithmName.SHA1 + ); + KeyBytes = rfc2898.GetBytes(KeySizeInBytes); + IvBytes = rfc2898.GetBytes(KeySizeInBytes); + var generatedVerifyValue = rfc2898.GetBytes(2); +#endif + + var verify = BinaryPrimitives.ReadInt16LittleEndian(passwordVerifyValue); + var generated = BinaryPrimitives.ReadInt16LittleEndian(generatedVerifyValue); + if (verify != generated) + { + throw new InvalidFormatException("bad password"); + } } internal byte[] IvBytes { get; set; } @@ -43,34 +76,6 @@ internal class WinzipAesEncryptionData WinzipAesKeySize.KeySize128 => 16, WinzipAesKeySize.KeySize192 => 24, WinzipAesKeySize.KeySize256 => 32, - _ => throw new InvalidOperationException(), + _ => throw new ArchiveOperationException(), }; - - private void Initialize() - { -#if NET7_0 - var rfc2898 = new Rfc2898DeriveBytes( - _password, - _salt, - RFC2898_ITERATIONS, - HashAlgorithmName.SHA1 - ); -#else - var rfc2898 = new Rfc2898DeriveBytes(_password, _salt, RFC2898_ITERATIONS); -#endif - - KeyBytes = rfc2898.GetBytes(KeySizeInBytes); // 16 or 24 or 32 ??? - IvBytes = rfc2898.GetBytes(KeySizeInBytes); - _generatedVerifyValue = rfc2898.GetBytes(2); - - var verify = BinaryPrimitives.ReadInt16LittleEndian(_passwordVerifyValue); - if (_password != null) - { - var generated = BinaryPrimitives.ReadInt16LittleEndian(_generatedVerifyValue); - if (verify != generated) - { - throw new InvalidFormatException("bad password"); - } - } - } } diff --git a/src/SharpCompress/Common/Zip/WinzipAesKeySize.cs b/src/SharpCompress/Common/Zip/WinzipAesKeySize.cs index 7772bcaa..276a3504 100644 --- a/src/SharpCompress/Common/Zip/WinzipAesKeySize.cs +++ b/src/SharpCompress/Common/Zip/WinzipAesKeySize.cs @@ -4,5 +4,5 @@ internal enum WinzipAesKeySize { KeySize128 = 1, KeySize192 = 2, - KeySize256 = 3 + KeySize256 = 3, } diff --git a/src/SharpCompress/Common/Zip/ZipCompressionMethod.cs b/src/SharpCompress/Common/Zip/ZipCompressionMethod.cs index 13c2dbe7..33eaf2b7 100644 --- a/src/SharpCompress/Common/Zip/ZipCompressionMethod.cs +++ b/src/SharpCompress/Common/Zip/ZipCompressionMethod.cs @@ -3,12 +3,18 @@ namespace SharpCompress.Common.Zip; internal enum ZipCompressionMethod { None = 0, + Shrink = 1, + Reduce1 = 2, + Reduce2 = 3, + Reduce3 = 4, + Reduce4 = 5, + Explode = 6, Deflate = 8, Deflate64 = 9, BZip2 = 12, LZMA = 14, - ZStd = 93, + ZStandard = 93, Xz = 95, PPMd = 98, - WinzipAes = 0x63 //http://www.winzip.com/aes_info.htm + WinzipAes = 0x63, //http://www.winzip.com/aes_info.htm } diff --git a/src/SharpCompress/Common/Zip/ZipEntry.cs b/src/SharpCompress/Common/Zip/ZipEntry.cs index 2c544b94..bfe1a50d 100644 --- a/src/SharpCompress/Common/Zip/ZipEntry.cs +++ b/src/SharpCompress/Common/Zip/ZipEntry.cs @@ -1,89 +1,171 @@ -#nullable disable - using System; using System.Collections.Generic; +using System.Linq; +using SharpCompress.Common.Options; using SharpCompress.Common.Zip.Headers; namespace SharpCompress.Common.Zip; public class ZipEntry : Entry { - private readonly ZipFilePart _filePart; + private readonly ZipFilePart? _filePart; - internal ZipEntry(ZipFilePart filePart) + // WinZip AES extra data constants + private const int MinimumWinZipAesExtraDataLength = 7; + private const int WinZipAesCompressionMethodOffset = 5; + + internal ZipEntry(ZipFilePart? filePart, IReaderOptions readerOptions) + : base(readerOptions) { - if (filePart != null) + if (filePart == null) { - _filePart = filePart; - LastModifiedTime = Utility.DosDateToDateTime( - filePart.Header.LastModifiedDate, - filePart.Header.LastModifiedTime - ); + return; } + _filePart = filePart; + + LastModifiedTime = Utility.DosDateToDateTime( + filePart.Header.LastModifiedDate, + filePart.Header.LastModifiedTime + ); + + var times = + filePart.Header.Extra.FirstOrDefault(header => + header.GetType() == typeof(UnixTimeExtraField) + ) as UnixTimeExtraField; + + LastAccessedTime = times?.UnicodeTimes.Item2; + CreatedTime = times?.UnicodeTimes.Item3; } public override CompressionType CompressionType { get { - switch (_filePart.Header.CompressionMethod) + var compressionMethod = GetActualCompressionMethod(); + return compressionMethod switch { - case ZipCompressionMethod.BZip2: - { - return CompressionType.BZip2; - } - case ZipCompressionMethod.Deflate: - { - return CompressionType.Deflate; - } - case ZipCompressionMethod.Deflate64: - { - return CompressionType.Deflate64; - } - case ZipCompressionMethod.LZMA: - { - return CompressionType.LZMA; - } - case ZipCompressionMethod.PPMd: - { - return CompressionType.PPMd; - } - case ZipCompressionMethod.None: - { - return CompressionType.None; - } - default: - { - return CompressionType.Unknown; - } - } + ZipCompressionMethod.BZip2 => CompressionType.BZip2, + ZipCompressionMethod.Deflate => CompressionType.Deflate, + ZipCompressionMethod.Deflate64 => CompressionType.Deflate64, + ZipCompressionMethod.LZMA => CompressionType.LZMA, + ZipCompressionMethod.PPMd => CompressionType.PPMd, + ZipCompressionMethod.None => CompressionType.None, + ZipCompressionMethod.Shrink => CompressionType.Shrink, + ZipCompressionMethod.Reduce1 => CompressionType.Reduce1, + ZipCompressionMethod.Reduce2 => CompressionType.Reduce2, + ZipCompressionMethod.Reduce3 => CompressionType.Reduce3, + ZipCompressionMethod.Reduce4 => CompressionType.Reduce4, + ZipCompressionMethod.Explode => CompressionType.Explode, + ZipCompressionMethod.ZStandard => CompressionType.ZStandard, + ZipCompressionMethod.Xz => CompressionType.Xz, + _ => CompressionType.Unknown, + }; } } - public override long Crc => _filePart.Header.Crc; + private ZipCompressionMethod GetActualCompressionMethod() + { + if (_filePart?.Header.CompressionMethod != ZipCompressionMethod.WinzipAes) + { + return _filePart?.Header.CompressionMethod ?? ZipCompressionMethod.None; + } - public override string Key => _filePart.Header.Name; + // For WinZip AES, the actual compression method is stored in the extra data + var aesExtraData = _filePart.Header.Extra.FirstOrDefault(x => + x.Type == ExtraDataType.WinZipAes + ); - public override string LinkTarget => null; + if (aesExtraData is null || aesExtraData.DataBytes.Length < MinimumWinZipAesExtraDataLength) + { + return ZipCompressionMethod.WinzipAes; + } - public override long CompressedSize => _filePart.Header.CompressedSize; + // The compression method is at offset 5 in the extra data + return (ZipCompressionMethod) + System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian( + aesExtraData.DataBytes.AsSpan(WinZipAesCompressionMethodOffset) + ); + } - public override long Size => _filePart.Header.UncompressedSize; + public override long Crc => _filePart?.Header.Crc ?? 0; + + internal override ChecksumDescriptor Checksum + { + get + { + if ( + _filePart is null + || IsDirectory + || !_filePart.Header.IsCrcAvailable + || !IsReliableCrcMetadata(_filePart.Header) + ) + { + return default; + } + + return new ChecksumDescriptor( + ChecksumKind.Crc32, + _filePart.Header.Crc, + IsAvailable: true + ); + } + } + + private static bool IsReliableCrcMetadata(ZipFileEntry header) + { + if (header.CompressionMethod != ZipCompressionMethod.WinzipAes) + { + return true; + } + + var aesExtraData = header.Extra.FirstOrDefault(x => x.Type == ExtraDataType.WinZipAes); + if (aesExtraData is null || aesExtraData.DataBytes.Length < MinimumWinZipAesExtraDataLength) + { + return false; + } + + var vendorVersion = System.Buffers.Binary.BinaryPrimitives.ReadUInt16LittleEndian( + aesExtraData.DataBytes + ); + + // WinZip AES AE-2 stores a zero CRC field by design and relies on AES authentication. + return vendorVersion == 0x0001; + } + + public override string? Key => _filePart?.Header.Name; + + public override string? LinkTarget => null; + + public override long CompressedSize => _filePart?.Header.CompressedSize ?? 0; + + public override long Size => _filePart?.Header.UncompressedSize ?? 0; public override DateTime? LastModifiedTime { get; } - public override DateTime? CreatedTime => null; + /// + /// + /// The returned time is UTC, not local. + /// + public override DateTime? CreatedTime { get; } - public override DateTime? LastAccessedTime => null; + /// + /// + /// The returned time is UTC, not local. + /// + public override DateTime? LastAccessedTime { get; } public override DateTime? ArchivedTime => null; public override bool IsEncrypted => - FlagUtility.HasFlag(_filePart.Header.Flags, HeaderFlags.Encrypted); + FlagUtility.HasFlag(_filePart?.Header.Flags ?? HeaderFlags.None, HeaderFlags.Encrypted); - public override bool IsDirectory => _filePart.Header.IsDirectory; + public override bool IsDirectory => _filePart?.Header.IsDirectory ?? false; public override bool IsSplitAfter => false; - internal override IEnumerable Parts => _filePart.AsEnumerable(); + internal override IEnumerable Parts => _filePart.Empty(); + + public override int? Attrib => (int?)_filePart?.Header.ExternalFileAttributes; + + public string? Comment => _filePart?.Header.Comment; } diff --git a/src/SharpCompress/Common/Zip/ZipFilePart.Async.cs b/src/SharpCompress/Common/Zip/ZipFilePart.Async.cs new file mode 100644 index 00000000..29f523a6 --- /dev/null +++ b/src/SharpCompress/Common/Zip/ZipFilePart.Async.cs @@ -0,0 +1,284 @@ +using System; +using System.Buffers.Binary; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Zip.Headers; +using SharpCompress.Compressors; +using SharpCompress.IO; +using SharpCompress.Providers; + +namespace SharpCompress.Common.Zip; + +internal abstract partial class ZipFilePart +{ + internal override async ValueTask GetCompressedStreamAsync( + CancellationToken cancellationToken = default + ) + { + if (!Header.HasData) + { + return Stream.Null; + } + var decompressionStream = await CreateDecompressionStreamAsync( + await GetCryptoStreamAsync(CreateBaseStream(), cancellationToken) + .ConfigureAwait(false), + Header.CompressionMethod, + cancellationToken + ) + .ConfigureAwait(false); + if (LeaveStreamOpen) + { + return SharpCompressStream.CreateNonDisposing(decompressionStream); + } + return decompressionStream; + } + + protected async ValueTask GetCryptoStreamAsync( + Stream plainStream, + CancellationToken cancellationToken = default + ) + { + var isFileEncrypted = FlagUtility.HasFlag(Header.Flags, HeaderFlags.Encrypted); + + if (Header.CompressedSize == 0 && isFileEncrypted) + { + throw new NotSupportedException("Cannot encrypt file with unknown size at start."); + } + + if ( + Header.CompressedSize == 0 + && FlagUtility.HasFlag(Header.Flags, HeaderFlags.UsePostDataDescriptor) + ) + { + plainStream = SharpCompressStream.CreateNonDisposing(plainStream); //make sure AES doesn't close + } + else + { + plainStream = new ReadOnlySubStream(plainStream, Header.CompressedSize); //make sure AES doesn't close + } + + if (isFileEncrypted) + { + switch (Header.CompressionMethod) + { + case ZipCompressionMethod.None: + case ZipCompressionMethod.Shrink: + case ZipCompressionMethod.Reduce1: + case ZipCompressionMethod.Reduce2: + case ZipCompressionMethod.Reduce3: + case ZipCompressionMethod.Reduce4: + case ZipCompressionMethod.Deflate: + case ZipCompressionMethod.Deflate64: + case ZipCompressionMethod.BZip2: + case ZipCompressionMethod.LZMA: + case ZipCompressionMethod.PPMd: + { + return new PkwareTraditionalCryptoStream( + plainStream, + await Header + .ComposeEncryptionDataAsync(plainStream, cancellationToken) + .ConfigureAwait(false), + CryptoMode.Decrypt + ); + } + + case ZipCompressionMethod.WinzipAes: + { + if (Header.WinzipAesEncryptionData != null) + { + return new WinzipAesCryptoStream( + plainStream, + Header.WinzipAesEncryptionData, + Header.CompressedSize - 10 + ); + } + return plainStream; + } + + default: + { + throw new ArchiveOperationException("Header.CompressionMethod is invalid"); + } + } + } + return plainStream; + } + + protected async ValueTask CreateDecompressionStreamAsync( + Stream stream, + ZipCompressionMethod method, + CancellationToken cancellationToken = default + ) + { + // Handle special cases first + switch (method) + { + case ZipCompressionMethod.None: + { + if (Header.CompressedSize is 0) + { + return new DataDescriptorStream(stream); + } + + return stream; + } + case ZipCompressionMethod.WinzipAes: + { + return await CreateWinzipAesDecompressionStreamAsync(stream, cancellationToken) + .ConfigureAwait(false); + } + } + + var compressionType = ToCompressionType(method); + var providers = GetProviders(); + var context = new CompressionContext + { + InputSize = Header.CompressedSize, + OutputSize = Header.UncompressedSize, + CanSeek = stream.CanSeek, + }; + + switch (method) + { + case ZipCompressionMethod.LZMA: + { + if (FlagUtility.HasFlag(Header.Flags, HeaderFlags.Encrypted)) + { + throw new NotSupportedException("LZMA with pkware encryption."); + } + // When the uncompressed size is known to be zero, skip remaining compressed + // bytes (required for streaming reads) and return an empty stream. + // Bit1 (EOS marker flag) means the output size is not stored in the header + // (the LZMA stream itself contains an end-of-stream marker instead), so we + // only short-circuit when the size is explicitly known to be zero. + if ( + !FlagUtility.HasFlag(Header.Flags, HeaderFlags.Bit1) + && Header.UncompressedSize == 0 + ) + { + await stream.SkipAsync(cancellationToken).ConfigureAwait(false); + return Stream.Null; + } + var buffer = new byte[4]; + await stream.ReadFullyAsync(buffer, 0, 4, cancellationToken).ConfigureAwait(false); + var propsSize = BinaryPrimitives.ReadUInt16LittleEndian(buffer.AsSpan(2, 2)); + var props = new byte[propsSize]; + await stream + .ReadFullyAsync(props, 0, propsSize, cancellationToken) + .ConfigureAwait(false); + + // When the uncompressed size is known to be zero, skip remaining compressed + // bytes (required for streaming reads) and return an empty stream. + // Bit1 (EOS marker flag) means the output size is not stored in the header + // (the LZMA stream itself contains an end-of-stream marker instead), so we + // only short-circuit when the size is explicitly known to be zero. + if ( + !FlagUtility.HasFlag(Header.Flags, HeaderFlags.Bit1) + && Header.UncompressedSize == 0 + ) + { + await stream.SkipAsync(cancellationToken).ConfigureAwait(false); + return Stream.Null; + } + + context = context with + { + Properties = props, + InputSize = + Header.CompressedSize > 0 ? Header.CompressedSize - 4 - props.Length : -1, + OutputSize = FlagUtility.HasFlag(Header.Flags, HeaderFlags.Bit1) + ? -1 + : Header.UncompressedSize, + }; + + return await providers + .CreateDecompressStreamAsync( + compressionType, + stream, + context, + cancellationToken + ) + .ConfigureAwait(false); + } + case ZipCompressionMethod.PPMd: + { + var props = new byte[2]; + await stream.ReadFullyAsync(props, 0, 2, cancellationToken).ConfigureAwait(false); + context = context with { Properties = props }; + return await providers + .CreateDecompressStreamAsync( + compressionType, + stream, + context, + cancellationToken + ) + .ConfigureAwait(false); + } + case ZipCompressionMethod.Explode: + { + context = context with { FormatOptions = Header.Flags }; + return await providers + .CreateDecompressStreamAsync( + compressionType, + stream, + context, + cancellationToken + ) + .ConfigureAwait(false); + } + default: + { + return await providers + .CreateDecompressStreamAsync( + compressionType, + stream, + context, + cancellationToken + ) + .ConfigureAwait(false); + } + } + } + + private async ValueTask CreateWinzipAesDecompressionStreamAsync( + Stream stream, + CancellationToken cancellationToken = default + ) + { + var data = Header.Extra.SingleOrDefault(x => x.Type == ExtraDataType.WinZipAes); + if (data is null) + { + throw new InvalidFormatException("No Winzip AES extra data found."); + } + + if (data.Length != 7) + { + throw new InvalidFormatException("Winzip data length is not 7."); + } + + var compressedMethod = BinaryPrimitives.ReadUInt16LittleEndian(data.DataBytes); + + if (compressedMethod != 0x01 && compressedMethod != 0x02) + { + throw new InvalidFormatException( + "Unexpected vendor version number for WinZip AES metadata" + ); + } + + var vendorId = BinaryPrimitives.ReadUInt16LittleEndian(data.DataBytes.AsSpan(2)); + if (vendorId != 0x4541) + { + throw new InvalidFormatException("Unexpected vendor ID for WinZip AES metadata"); + } + + return await CreateDecompressionStreamAsync( + stream, + (ZipCompressionMethod) + BinaryPrimitives.ReadUInt16LittleEndian(data.DataBytes.AsSpan(5)), + cancellationToken + ) + .ConfigureAwait(false); + } +} diff --git a/src/SharpCompress/Common/Zip/ZipFilePart.cs b/src/SharpCompress/Common/Zip/ZipFilePart.cs index faefdf15..de182f37 100644 --- a/src/SharpCompress/Common/Zip/ZipFilePart.cs +++ b/src/SharpCompress/Common/Zip/ZipFilePart.cs @@ -7,28 +7,39 @@ using SharpCompress.Compressors; using SharpCompress.Compressors.BZip2; using SharpCompress.Compressors.Deflate; using SharpCompress.Compressors.Deflate64; +using SharpCompress.Compressors.Explode; using SharpCompress.Compressors.LZMA; using SharpCompress.Compressors.PPMd; +using SharpCompress.Compressors.Reduce; +using SharpCompress.Compressors.Shrink; using SharpCompress.Compressors.Xz; +using SharpCompress.Compressors.ZStandard; using SharpCompress.IO; -using ZstdSharp; +using SharpCompress.Providers; namespace SharpCompress.Common.Zip; -internal abstract class ZipFilePart : FilePart +internal abstract partial class ZipFilePart : FilePart { - internal ZipFilePart(ZipFileEntry header, Stream stream) + private readonly CompressionProviderRegistry _compressionProviders; + + internal ZipFilePart( + ZipFileEntry header, + Stream stream, + CompressionProviderRegistry compressionProviders + ) : base(header.ArchiveEncoding) { Header = header; header.Part = this; BaseStream = stream; + _compressionProviders = compressionProviders; } internal Stream BaseStream { get; } internal ZipFileEntry Header { get; set; } - internal override string FilePartName => Header.Name; + internal override string? FilePartName => Header.Name; internal override Stream GetCompressedStream() { @@ -42,7 +53,7 @@ internal abstract class ZipFilePart : FilePart ); if (LeaveStreamOpen) { - return NonDisposingStream.Create(decompressionStream); + return SharpCompressStream.CreateNonDisposing(decompressionStream); } return decompressionStream; } @@ -61,107 +72,156 @@ internal abstract class ZipFilePart : FilePart protected bool LeaveStreamOpen => FlagUtility.HasFlag(Header.Flags, HeaderFlags.UsePostDataDescriptor) || Header.IsZip64; + /// + /// Gets the compression provider registry, falling back to default if not set. + /// + protected CompressionProviderRegistry GetProviders() => _compressionProviders; + + /// + /// Converts ZipCompressionMethod to CompressionType. + /// + protected static CompressionType ToCompressionType(ZipCompressionMethod method) => + method switch + { + ZipCompressionMethod.None => CompressionType.None, + ZipCompressionMethod.Deflate => CompressionType.Deflate, + ZipCompressionMethod.Deflate64 => CompressionType.Deflate64, + ZipCompressionMethod.BZip2 => CompressionType.BZip2, + ZipCompressionMethod.LZMA => CompressionType.LZMA, + ZipCompressionMethod.PPMd => CompressionType.PPMd, + ZipCompressionMethod.ZStandard => CompressionType.ZStandard, + ZipCompressionMethod.Xz => CompressionType.Xz, + ZipCompressionMethod.Shrink => CompressionType.Shrink, + ZipCompressionMethod.Reduce1 => CompressionType.Reduce1, + ZipCompressionMethod.Reduce2 => CompressionType.Reduce2, + ZipCompressionMethod.Reduce3 => CompressionType.Reduce3, + ZipCompressionMethod.Reduce4 => CompressionType.Reduce4, + ZipCompressionMethod.Explode => CompressionType.Explode, + _ => throw new NotSupportedException($"Unsupported compression method: {method}"), + }; + protected Stream CreateDecompressionStream(Stream stream, ZipCompressionMethod method) { + // Handle special cases first switch (method) { case ZipCompressionMethod.None: { - if (stream is ReadOnlySubStream) + if (Header.CompressedSize is 0) { - return stream; + return new DataDescriptorStream(stream); } + return stream; + } + case ZipCompressionMethod.WinzipAes: + { + return CreateWinzipAesDecompressionStream(stream); + } + } - if (Header.CompressedSize > 0) - { - return new ReadOnlySubStream(stream, Header.CompressedSize); - } + // Get the compression type and providers + var compressionType = ToCompressionType(method); + var providers = GetProviders(); - return new DataDescriptorStream(stream); - } - case ZipCompressionMethod.Deflate: - { - return new DeflateStream(stream, CompressionMode.Decompress); - } - case ZipCompressionMethod.Deflate64: - { - return new Deflate64Stream(stream, CompressionMode.Decompress); - } - case ZipCompressionMethod.BZip2: - { - return new BZip2Stream(stream, CompressionMode.Decompress, false); - } + // Build context with header information + var context = new CompressionContext + { + InputSize = Header.CompressedSize, + OutputSize = Header.UncompressedSize, + CanSeek = stream.CanSeek, + }; + + // Handle methods that need special context + switch (method) + { case ZipCompressionMethod.LZMA: { if (FlagUtility.HasFlag(Header.Flags, HeaderFlags.Encrypted)) { throw new NotSupportedException("LZMA with pkware encryption."); } - var reader = new BinaryReader(stream); - reader.ReadUInt16(); //LZMA version - var props = new byte[reader.ReadUInt16()]; - reader.Read(props, 0, props.Length); - return new LzmaStream( - props, + + using var reader = new BinaryReader( stream, - Header.CompressedSize > 0 ? Header.CompressedSize - 4 - props.Length : -1, - FlagUtility.HasFlag(Header.Flags, HeaderFlags.Bit1) - ? -1 - : Header.UncompressedSize + System.Text.Encoding.Default, + leaveOpen: true ); - } - case ZipCompressionMethod.Xz: - { - return new XZStream(stream); - } - case ZipCompressionMethod.ZStd: - { - return new DecompressionStream(stream); + reader.ReadUInt16(); // LZMA version + var propsLength = reader.ReadUInt16(); + var props = reader.ReadBytes(propsLength); + + // When the uncompressed size is known to be zero, skip remaining compressed + // bytes (required for streaming reads) and return an empty stream. + // Bit1 (EOS marker flag) means the output size is not stored in the header + // (the LZMA stream itself contains an end-of-stream marker instead), so we + // only short-circuit when the size is explicitly known to be zero. + if ( + !FlagUtility.HasFlag(Header.Flags, HeaderFlags.Bit1) + && Header.UncompressedSize == 0 + ) + { + stream.Skip(); + return Stream.Null; + } + + context = context with + { + Properties = props, + InputSize = + Header.CompressedSize > 0 ? Header.CompressedSize - 4 - props.Length : -1, + OutputSize = FlagUtility.HasFlag(Header.Flags, HeaderFlags.Bit1) + ? -1 + : Header.UncompressedSize, + }; + return providers.CreateDecompressStream(compressionType, stream, context); } case ZipCompressionMethod.PPMd: { Span props = stackalloc byte[2]; stream.ReadFully(props); - return new PpmdStream(new PpmdProperties(props), stream, false); + context = context with { Properties = props.ToArray() }; + return providers.CreateDecompressStream(compressionType, stream, context); } - case ZipCompressionMethod.WinzipAes: + case ZipCompressionMethod.Explode: { - var data = Header.Extra.SingleOrDefault(x => x.Type == ExtraDataType.WinZipAes); - if (data is null) - { - throw new InvalidFormatException("No Winzip AES extra data found."); - } - if (data.Length != 7) - { - throw new InvalidFormatException("Winzip data length is not 7."); - } - var compressedMethod = BinaryPrimitives.ReadUInt16LittleEndian(data.DataBytes); - - if (compressedMethod != 0x01 && compressedMethod != 0x02) - { - throw new InvalidFormatException( - "Unexpected vendor version number for WinZip AES metadata" - ); - } - - var vendorId = BinaryPrimitives.ReadUInt16LittleEndian(data.DataBytes.AsSpan(2)); - if (vendorId != 0x4541) - { - throw new InvalidFormatException( - "Unexpected vendor ID for WinZip AES metadata" - ); - } - return CreateDecompressionStream( - stream, - (ZipCompressionMethod) - BinaryPrimitives.ReadUInt16LittleEndian(data.DataBytes.AsSpan(5)) - ); - } - default: - { - throw new NotSupportedException("CompressionMethod: " + Header.CompressionMethod); + context = context with { FormatOptions = Header.Flags }; + return providers.CreateDecompressStream(compressionType, stream, context); } } + + // For simple methods, use the basic decompress + return providers.CreateDecompressStream(compressionType, stream, context); + } + + private Stream CreateWinzipAesDecompressionStream(Stream stream) + { + var data = Header.Extra.SingleOrDefault(x => x.Type == ExtraDataType.WinZipAes); + if (data is null) + { + throw new InvalidFormatException("No Winzip AES extra data found."); + } + if (data.Length != 7) + { + throw new InvalidFormatException("Winzip data length is not 7."); + } + var compressedMethod = BinaryPrimitives.ReadUInt16LittleEndian(data.DataBytes); + + if (compressedMethod != 0x01 && compressedMethod != 0x02) + { + throw new InvalidFormatException( + "Unexpected vendor version number for WinZip AES metadata" + ); + } + + var vendorId = BinaryPrimitives.ReadUInt16LittleEndian(data.DataBytes.AsSpan(2)); + if (vendorId != 0x4541) + { + throw new InvalidFormatException("Unexpected vendor ID for WinZip AES metadata"); + } + return CreateDecompressionStream( + stream, + (ZipCompressionMethod)BinaryPrimitives.ReadUInt16LittleEndian(data.DataBytes.AsSpan(5)) + ); } protected Stream GetCryptoStream(Stream plainStream) @@ -174,13 +234,11 @@ internal abstract class ZipFilePart : FilePart } if ( - ( - Header.CompressedSize == 0 - && FlagUtility.HasFlag(Header.Flags, HeaderFlags.UsePostDataDescriptor) - ) || Header.IsZip64 + Header.CompressedSize == 0 + && FlagUtility.HasFlag(Header.Flags, HeaderFlags.UsePostDataDescriptor) ) { - plainStream = NonDisposingStream.Create(plainStream); //make sure AES doesn't close + plainStream = SharpCompressStream.CreateNonDisposing(plainStream); //make sure AES doesn't close } else { @@ -192,6 +250,11 @@ internal abstract class ZipFilePart : FilePart switch (Header.CompressionMethod) { case ZipCompressionMethod.None: + case ZipCompressionMethod.Shrink: + case ZipCompressionMethod.Reduce1: + case ZipCompressionMethod.Reduce2: + case ZipCompressionMethod.Reduce3: + case ZipCompressionMethod.Reduce4: case ZipCompressionMethod.Deflate: case ZipCompressionMethod.Deflate64: case ZipCompressionMethod.BZip2: @@ -220,7 +283,7 @@ internal abstract class ZipFilePart : FilePart default: { - throw new InvalidOperationException("Header.CompressionMethod is invalid"); + throw new ArchiveOperationException("Header.CompressionMethod is invalid"); } } } diff --git a/src/SharpCompress/Common/Zip/ZipHeaderFactory.Async.cs b/src/SharpCompress/Common/Zip/ZipHeaderFactory.Async.cs new file mode 100644 index 00000000..bc5944b4 --- /dev/null +++ b/src/SharpCompress/Common/Zip/ZipHeaderFactory.Async.cs @@ -0,0 +1,167 @@ +using System; +using System.IO; +using System.Linq; +using System.Threading.Tasks; +using SharpCompress; +using SharpCompress.Common.Zip.Headers; +using SharpCompress.IO; + +namespace SharpCompress.Common.Zip; + +internal partial class ZipHeaderFactory +{ + protected async ValueTask ReadHeader( + uint headerBytes, + AsyncBinaryReader reader, + bool zip64 = false + ) + { + switch (headerBytes) + { + case ENTRY_HEADER_BYTES: + { + var entryHeader = new LocalEntryHeader(_archiveEncoding); + await entryHeader.Read(reader).ConfigureAwait(false); + await LoadHeaderAsync(entryHeader, reader.BaseStream).ConfigureAwait(false); + + _lastEntryHeader = entryHeader; + return entryHeader; + } + case DIRECTORY_START_HEADER_BYTES: + { + var entry = new DirectoryEntryHeader(_archiveEncoding); + await entry.Read(reader).ConfigureAwait(false); + return entry; + } + case POST_DATA_DESCRIPTOR: + { + if ( + _lastEntryHeader != null + && FlagUtility.HasFlag( + _lastEntryHeader.NotNull().Flags, + HeaderFlags.UsePostDataDescriptor + ) + ) + { + _lastEntryHeader.Crc = await reader.ReadUInt32Async().ConfigureAwait(false); + _lastEntryHeader.IsCrcAvailable = true; + _lastEntryHeader.CompressedSize = zip64 + ? (long)await reader.ReadUInt64Async().ConfigureAwait(false) + : await reader.ReadUInt32Async().ConfigureAwait(false); + _lastEntryHeader.UncompressedSize = zip64 + ? (long)await reader.ReadUInt64Async().ConfigureAwait(false) + : await reader.ReadUInt32Async().ConfigureAwait(false); + } + else + { + await reader.SkipAsync(zip64 ? 20 : 12).ConfigureAwait(false); + } + return null; + } + case DIGITAL_SIGNATURE: + return null; + case DIRECTORY_END_HEADER_BYTES: + { + var entry = new DirectoryEndHeader(); + await entry.Read(reader).ConfigureAwait(false); + return entry; + } + case SPLIT_ARCHIVE_HEADER_BYTES: + { + return new SplitHeader(); + } + case ZIP64_END_OF_CENTRAL_DIRECTORY: + { + var entry = new Zip64DirectoryEndHeader(); + await entry.Read(reader).ConfigureAwait(false); + return entry; + } + case ZIP64_END_OF_CENTRAL_DIRECTORY_LOCATOR: + { + var entry = new Zip64DirectoryEndLocatorHeader(); + await entry.Read(reader).ConfigureAwait(false); + return entry; + } + default: + return null; + } + } + + /// + /// Loads encryption metadata and stream positioning for a header using async reads where needed. + /// + private async ValueTask LoadHeaderAsync(ZipFileEntry entryHeader, Stream stream) + { + if (FlagUtility.HasFlag(entryHeader.Flags, HeaderFlags.Encrypted)) + { + if ( + !entryHeader.IsDirectory + && entryHeader.CompressedSize == 0 + && FlagUtility.HasFlag(entryHeader.Flags, HeaderFlags.UsePostDataDescriptor) + ) + { + throw new NotSupportedException( + "SharpCompress cannot currently read non-seekable Zip Streams with encrypted data that has been written in a non-seekable manner." + ); + } + + if (_password is null) + { + throw new CryptographicException("No password supplied for encrypted zip."); + } + + entryHeader.Password = _password; + + if (entryHeader.CompressionMethod == ZipCompressionMethod.WinzipAes) + { + var data = entryHeader.Extra.SingleOrDefault(x => + x.Type == ExtraDataType.WinZipAes + ); + if (data != null) + { + var keySize = (WinzipAesKeySize)data.DataBytes[4]; + + var salt = new byte[WinzipAesEncryptionData.KeyLengthInBytes(keySize) / 2]; + var passwordVerifyValue = new byte[2]; + await stream.ReadExactAsync(salt, 0, salt.Length).ConfigureAwait(false); + await stream.ReadExactAsync(passwordVerifyValue, 0, 2).ConfigureAwait(false); + + entryHeader.WinzipAesEncryptionData = new WinzipAesEncryptionData( + keySize, + salt, + passwordVerifyValue, + _password + ); + + entryHeader.CompressedSize -= (uint)(salt.Length + 2); + } + } + } + + if (entryHeader.IsDirectory) + { + return; + } + + switch (_mode) + { + case StreamingMode.Seekable: + { + entryHeader.DataStartPosition = stream.Position; + stream.Position += entryHeader.CompressedSize; + break; + } + + case StreamingMode.Streaming: + { + entryHeader.PackedStream = stream; + break; + } + + default: + { + throw new InvalidFormatException("Invalid StreamingMode"); + } + } + } +} diff --git a/src/SharpCompress/Common/Zip/ZipHeaderFactory.cs b/src/SharpCompress/Common/Zip/ZipHeaderFactory.cs index 8a58e220..e1ebb8c1 100644 --- a/src/SharpCompress/Common/Zip/ZipHeaderFactory.cs +++ b/src/SharpCompress/Common/Zip/ZipHeaderFactory.cs @@ -1,12 +1,14 @@ using System; using System.IO; using System.Linq; +using System.Threading.Tasks; +using SharpCompress; using SharpCompress.Common.Zip.Headers; using SharpCompress.IO; namespace SharpCompress.Common.Zip; -internal class ZipHeaderFactory +internal partial class ZipHeaderFactory { internal const uint ENTRY_HEADER_BYTES = 0x04034b50; internal const uint POST_DATA_DESCRIPTOR = 0x08074b50; @@ -21,12 +23,12 @@ internal class ZipHeaderFactory protected LocalEntryHeader? _lastEntryHeader; private readonly string? _password; private readonly StreamingMode _mode; - private readonly ArchiveEncoding _archiveEncoding; + private readonly IArchiveEncoding _archiveEncoding; protected ZipHeaderFactory( StreamingMode mode, string? password, - ArchiveEncoding archiveEncoding + IArchiveEncoding archiveEncoding ) { _mode = mode; @@ -55,9 +57,16 @@ internal class ZipHeaderFactory } case POST_DATA_DESCRIPTOR: { - if (FlagUtility.HasFlag(_lastEntryHeader!.Flags, HeaderFlags.UsePostDataDescriptor)) + if ( + _lastEntryHeader != null + && FlagUtility.HasFlag( + _lastEntryHeader.NotNull().Flags, + HeaderFlags.UsePostDataDescriptor + ) + ) { _lastEntryHeader.Crc = reader.ReadUInt32(); + _lastEntryHeader.IsCrcAvailable = true; _lastEntryHeader.CompressedSize = zip64 ? (long)reader.ReadUInt64() : reader.ReadUInt32(); @@ -142,8 +151,8 @@ internal class ZipHeaderFactory if (entryHeader.CompressionMethod == ZipCompressionMethod.WinzipAes) { - var data = entryHeader.Extra.SingleOrDefault( - x => x.Type == ExtraDataType.WinZipAes + var data = entryHeader.Extra.SingleOrDefault(x => + x.Type == ExtraDataType.WinZipAes ); if (data != null) { diff --git a/src/SharpCompress/Compressors/ADC/ADCBase.Async.cs b/src/SharpCompress/Compressors/ADC/ADCBase.Async.cs new file mode 100644 index 00000000..7bbbacde --- /dev/null +++ b/src/SharpCompress/Compressors/ADC/ADCBase.Async.cs @@ -0,0 +1,176 @@ +using System; +using System.Buffers; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Compressors.ADC; + +public static partial class ADCBase +{ + /// + /// Decompresses a byte buffer asynchronously that's compressed with ADC + /// + /// Compressed buffer + /// Max size for decompressed data + /// Cancellation token + /// Result containing bytes read and decompressed data + public static async ValueTask DecompressAsync( + byte[] input, + int bufferSize = 262144, + CancellationToken cancellationToken = default + ) => + await DecompressAsync(new MemoryStream(input), bufferSize, cancellationToken) + .ConfigureAwait(false); + + /// + /// Decompresses a stream asynchronously that's compressed with ADC + /// + /// Stream containing compressed data + /// Max size for decompressed data + /// Cancellation token + /// Result containing bytes read and decompressed data + public static async ValueTask DecompressAsync( + Stream input, + int bufferSize = 262144, + CancellationToken cancellationToken = default + ) + { + var result = new AdcDecompressResult(); + + if (input is null || input.Length == 0) + { + result.BytesRead = 0; + result.Output = null; + return result; + } + + var start = (int)input.Position; + var position = (int)input.Position; + int chunkSize; + int offset; + int chunkType; + var buffer = ArrayPool.Shared.Rent(bufferSize); + var outPosition = 0; + var full = false; + byte[] temp = ArrayPool.Shared.Rent(3); + + try + { + while (position < input.Length) + { + cancellationToken.ThrowIfCancellationRequested(); + var readByte = input.ReadByte(); + if (readByte == -1) + { + break; + } + + chunkType = GetChunkType((byte)readByte); + + switch (chunkType) + { + case PLAIN: + chunkSize = GetChunkSize((byte)readByte); + if (outPosition + chunkSize > bufferSize) + { + full = true; + break; + } + + var readCount = await input + .ReadAsync(buffer, outPosition, chunkSize, cancellationToken) + .ConfigureAwait(false); + outPosition += readCount; + position += readCount + 1; + break; + case TWO_BYTE: + chunkSize = GetChunkSize((byte)readByte); + temp[0] = (byte)readByte; + temp[1] = (byte)input.ReadByte(); + offset = GetOffset(temp.AsSpan(0, 2)); + if (outPosition + chunkSize > bufferSize) + { + full = true; + break; + } + + if (offset == 0) + { + var lastByte = buffer[outPosition - 1]; + for (var i = 0; i < chunkSize; i++) + { + buffer[outPosition] = lastByte; + outPosition++; + } + + position += 2; + } + else + { + for (var i = 0; i < chunkSize; i++) + { + buffer[outPosition] = buffer[outPosition - offset - 1]; + outPosition++; + } + + position += 2; + } + + break; + case THREE_BYTE: + chunkSize = GetChunkSize((byte)readByte); + temp[0] = (byte)readByte; + temp[1] = (byte)input.ReadByte(); + temp[2] = (byte)input.ReadByte(); + offset = GetOffset(temp.AsSpan(0, 3)); + if (outPosition + chunkSize > bufferSize) + { + full = true; + break; + } + + if (offset == 0) + { + var lastByte = buffer[outPosition - 1]; + for (var i = 0; i < chunkSize; i++) + { + buffer[outPosition] = lastByte; + outPosition++; + } + + position += 3; + } + else + { + for (var i = 0; i < chunkSize; i++) + { + buffer[outPosition] = buffer[outPosition - offset - 1]; + outPosition++; + } + + position += 3; + } + + break; + } + + if (full) + { + break; + } + } + + var output = new byte[outPosition]; + Array.Copy(buffer, output, outPosition); + result.BytesRead = position - start; + result.Output = output; + return result; + } + finally + { + ArrayPool.Shared.Return(buffer); + ArrayPool.Shared.Return(temp); + } + } +} diff --git a/src/SharpCompress/Compressors/ADC/ADCBase.cs b/src/SharpCompress/Compressors/ADC/ADCBase.cs index 6fc755a9..6a6f2c2e 100644 --- a/src/SharpCompress/Compressors/ADC/ADCBase.cs +++ b/src/SharpCompress/Compressors/ADC/ADCBase.cs @@ -24,14 +24,33 @@ // OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN // THE SOFTWARE. using System; +using System.Buffers; using System.IO; +using System.Threading; +using System.Threading.Tasks; namespace SharpCompress.Compressors.ADC; +/// +/// Result of an ADC decompression operation +/// +public class AdcDecompressResult +{ + /// + /// Number of bytes read from input + /// + public int BytesRead { get; set; } + + /// + /// Decompressed output buffer + /// + public byte[]? Output { get; set; } +} + /// /// Provides static methods for decompressing Apple Data Compression data /// -public static class ADCBase +public static partial class ADCBase { private const int PLAIN = 1; private const int TWO_BYTE = 2; @@ -78,6 +97,8 @@ public static class ADCBase public static int Decompress(byte[] input, out byte[]? output, int bufferSize = 262144) => Decompress(new MemoryStream(input), out output, bufferSize); + // Async methods moved to ADCBase.Async.cs + /// /// Decompresses a stream that's compressed with ADC /// diff --git a/src/SharpCompress/Compressors/ADC/ADCStream.Async.cs b/src/SharpCompress/Compressors/ADC/ADCStream.Async.cs new file mode 100644 index 00000000..d2e535cd --- /dev/null +++ b/src/SharpCompress/Compressors/ADC/ADCStream.Async.cs @@ -0,0 +1,97 @@ +// +// ADC.cs +// +// Author: +// Natalia Portillo +// +// Copyright (c) 2016 © Claunia.com +// +// Permission is hereby granted, free of charge, to any person obtaining a copy +// of this software and associated documentation files (the "Software"), to deal +// in the Software without restriction, including without limitation the rights +// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +// copies of the Software, and to permit persons to whom the Software is +// furnished to do so, subject to the following conditions: +// +// The above copyright notice and this permission notice shall be included in +// all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +// THE SOFTWARE. + +#nullable disable + +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.IO; + +namespace SharpCompress.Compressors.ADC; + +public sealed partial class ADCStream +{ + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken = default + ) + { + if (count == 0) + { + return 0; + } + ThrowHelper.ThrowIfNull(buffer); + ThrowHelper.ThrowIfNegative(count); + ThrowHelper.ThrowIfLessThan(offset, buffer.GetLowerBound(0)); + if ((offset + count) > buffer.GetLength(0)) + { + throw new ArgumentOutOfRangeException(nameof(count)); + } + + if (_outBuffer is null) + { + var result = await ADCBase + .DecompressAsync(_stream, cancellationToken: cancellationToken) + .ConfigureAwait(false); + _outBuffer = result.Output; + _outPosition = 0; + } + + var inPosition = offset; + var toCopy = count; + var copied = 0; + + while (_outPosition + toCopy >= _outBuffer.Length) + { + cancellationToken.ThrowIfCancellationRequested(); + var piece = _outBuffer.Length - _outPosition; + Array.Copy(_outBuffer, _outPosition, buffer, inPosition, piece); + inPosition += piece; + copied += piece; + _position += piece; + toCopy -= piece; + var result = await ADCBase + .DecompressAsync(_stream, cancellationToken: cancellationToken) + .ConfigureAwait(false); + _outBuffer = result.Output; + _outPosition = 0; + if (result.BytesRead == 0 || _outBuffer is null || _outBuffer.Length == 0) + { + return copied; + } + } + + Array.Copy(_outBuffer, _outPosition, buffer, inPosition, toCopy); + _outPosition += toCopy; + _position += toCopy; + copied += toCopy; + return copied; + } +} diff --git a/src/SharpCompress/Compressors/ADC/ADCStream.cs b/src/SharpCompress/Compressors/ADC/ADCStream.cs index 0660d11f..f2ef155e 100644 --- a/src/SharpCompress/Compressors/ADC/ADCStream.cs +++ b/src/SharpCompress/Compressors/ADC/ADCStream.cs @@ -1,4 +1,4 @@ -// +// // ADC.cs // // Author: @@ -28,13 +28,15 @@ using System; using System.IO; +using System.Threading; +using System.Threading.Tasks; namespace SharpCompress.Compressors.ADC; /// /// Provides a forward readable only stream that decompresses ADC data /// -public sealed class ADCStream : Stream +public sealed partial class ADCStream : Stream { /// /// This stream holds the compressed data @@ -108,18 +110,9 @@ public sealed class ADCStream : Stream { return 0; } - if (buffer is null) - { - throw new ArgumentNullException(nameof(buffer)); - } - if (count < 0) - { - throw new ArgumentOutOfRangeException(nameof(count)); - } - if (offset < buffer.GetLowerBound(0)) - { - throw new ArgumentOutOfRangeException(nameof(offset)); - } + ThrowHelper.ThrowIfNull(buffer); + ThrowHelper.ThrowIfNegative(count); + ThrowHelper.ThrowIfLessThan(offset, buffer.GetLowerBound(0)); if ((offset + count) > buffer.GetLength(0)) { throw new ArgumentOutOfRangeException(nameof(count)); diff --git a/src/SharpCompress/Compressors/ArcLzw/ArcLzwStream.Async.cs b/src/SharpCompress/Compressors/ArcLzw/ArcLzwStream.Async.cs new file mode 100644 index 00000000..56496c9a --- /dev/null +++ b/src/SharpCompress/Compressors/ArcLzw/ArcLzwStream.Async.cs @@ -0,0 +1,76 @@ +using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Compressors.RLE90; + +namespace SharpCompress.Compressors.ArcLzw; + +public partial class ArcLzwStream +{ + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + if (_processed) + { + return 0; + } + _processed = true; + var data = new byte[_compressedSize]; + int totalRead = 0; + while (totalRead < _compressedSize) + { + int read = await _stream + .ReadAsync(data, totalRead, _compressedSize - totalRead, cancellationToken) + .ConfigureAwait(false); + if (read == 0) + { + break; + } + totalRead += read; + } + var decoded = Decompress(data, _useCrunched); + var result = decoded.Count; + if (_useCrunched) + { + var unpacked = RLE.UnpackRLE(decoded.ToArray()); + unpacked.CopyTo(buffer, 0); + result = unpacked.Count; + } + else + { + decoded.CopyTo(buffer, 0); + } + return result; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + if (buffer.IsEmpty) + { + return 0; + } + + byte[] array = System.Buffers.ArrayPool.Shared.Rent(buffer.Length); + try + { + int read = await ReadAsync(array, 0, buffer.Length, cancellationToken) + .ConfigureAwait(false); + array.AsSpan(0, read).CopyTo(buffer.Span); + return read; + } + finally + { + System.Buffers.ArrayPool.Shared.Return(array); + } + } +#endif +} diff --git a/src/SharpCompress/Compressors/ArcLzw/ArcLzwStream.cs b/src/SharpCompress/Compressors/ArcLzw/ArcLzwStream.cs new file mode 100644 index 00000000..542d991a --- /dev/null +++ b/src/SharpCompress/Compressors/ArcLzw/ArcLzwStream.cs @@ -0,0 +1,214 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using SharpCompress.Common; +using SharpCompress.Compressors.RLE90; + +namespace SharpCompress.Compressors.ArcLzw; + +public partial class ArcLzwStream : Stream +{ + private Stream _stream; + private bool _processed; + private bool _useCrunched; + private int _compressedSize; + + private const int BITS = 12; + private const int CRUNCH_BITS = 12; + private const int SQUASH_BITS = 13; + private const int INIT_BITS = 9; + private const ushort FIRST = 257; + private const ushort CLEAR = 256; + + private ushort oldcode; + private byte finchar; + private int n_bits; + private ushort maxcode; + private ushort[] prefix = new ushort[8191]; + private byte[] suffix = new byte[8191]; + private bool clearFlag; + private Stack stack = new Stack(); + private ushort freeEnt; + private ushort maxcodemax; + + public ArcLzwStream(Stream stream, int compressedSize, bool useCrunched = true) + { + _stream = stream; + _useCrunched = useCrunched; + _compressedSize = compressedSize; + + oldcode = 0; + finchar = 0; + n_bits = 0; + maxcode = 0; + clearFlag = false; + freeEnt = FIRST; + maxcodemax = 0; + } + + private ushort? GetCode(BitReader reader) + { + if (clearFlag || freeEnt > maxcode) + { + if (freeEnt > maxcode) + { + n_bits++; + maxcode = (n_bits == BITS) ? maxcodemax : (ushort)((1 << n_bits) - 1); + } + if (clearFlag) + { + clearFlag = false; + n_bits = INIT_BITS; + maxcode = (ushort)((1 << n_bits) - 1); + } + } + return (ushort?)reader.ReadBits(n_bits); + } + + public List Decompress(byte[] input, bool useCrunched) + { + var result = new List(); + int bits = useCrunched ? CRUNCH_BITS : SQUASH_BITS; + + if (useCrunched) + { + if (input.Length == 0) + { + throw new InvalidFormatException("ArcLzwStream: compressed data is empty"); + } + if (input[0] != BITS) + { + throw new InvalidFormatException($"File packed with {input[0]}, expected {BITS}."); + } + + input = input.Skip(1).ToArray(); + } + + maxcodemax = (ushort)(1 << bits); + clearFlag = false; + n_bits = INIT_BITS; + maxcode = (ushort)((1 << n_bits) - 1); + + for (int i = 0; i < 256; i++) + { + suffix[i] = (byte)i; + } + + var reader = new BitReader(input); + freeEnt = FIRST; + + if (GetCode(reader) is ushort old) + { + oldcode = old; + finchar = (byte)oldcode; + result.Add(finchar); + } + + while (GetCode(reader) is ushort code) + { + if (code == CLEAR) + { + Array.Clear(prefix, 0, prefix.Length); + clearFlag = true; + freeEnt = (ushort)(FIRST - 1); + + if (GetCode(reader) is ushort c) + { + code = c; + } + else + { + break; + } + } + + ushort incode = code; + + if (code >= freeEnt) + { + stack.Push(finchar); + code = oldcode; + } + + while (code >= 256) + { + if (code >= suffix.Length) + { + throw new InvalidFormatException("ArcLzwStream: code out of range"); + } + stack.Push(suffix[code]); + code = prefix[code]; + } + + finchar = suffix[code]; + stack.Push(finchar); + + while (stack.Count > 0) + { + result.Add(stack.Pop()); + } + code = freeEnt; + if (code < maxcodemax) + { + prefix[code] = oldcode; + suffix[code] = finchar; + freeEnt = (ushort)(code + 1); + } + + oldcode = incode; + } + + return result; + } + + // Stream base class implementation + public override bool CanRead => true; + public override bool CanSeek => false; + public override bool CanWrite => false; + public override long Length => throw new NotImplementedException(); + public override long Position + { + get => _stream.Position; + set => throw new NotImplementedException(); + } + + public override void Flush() => throw new NotImplementedException(); + + public override int Read(byte[] buffer, int offset, int count) + { + if (_processed) + { + return 0; + } + _processed = true; + var data = new byte[_compressedSize]; + _stream.Read(data, 0, _compressedSize); + var decoded = Decompress(data, _useCrunched); + var result = decoded.Count; + if (_useCrunched) + { + var unpacked = RLE.UnpackRLE(decoded.ToArray()); + unpacked.CopyTo(buffer, 0); + result = unpacked.Count; + } + else + { + decoded.CopyTo(buffer, 0); + } + return result; + } + + public override long Seek(long offset, SeekOrigin origin) => + throw new NotImplementedException(); + + public override void SetLength(long value) => throw new NotImplementedException(); + + public override void Write(byte[] buffer, int offset, int count) => + throw new NotImplementedException(); + + protected override void Dispose(bool disposing) + { + base.Dispose(disposing); + } +} diff --git a/src/SharpCompress/Compressors/ArcLzw/BitReader.cs b/src/SharpCompress/Compressors/ArcLzw/BitReader.cs new file mode 100644 index 00000000..5c2a9cc8 --- /dev/null +++ b/src/SharpCompress/Compressors/ArcLzw/BitReader.cs @@ -0,0 +1,64 @@ +using System; + +namespace SharpCompress.Compressors.ArcLzw; + +public partial class ArcLzwStream +{ + public class BitReader + { + private readonly byte[] data; + private int bitPosition; + private int bytePosition; + + public BitReader(byte[] inputData) + { + data = inputData; + bitPosition = 0; + bytePosition = 0; + } + + public int? ReadBits(int bitCount) + { + if (bitCount <= 0 || bitCount > 16) + { + throw new ArgumentOutOfRangeException( + nameof(bitCount), + "Bit count must be between 1 and 16" + ); + } + + if (bytePosition >= data.Length) + { + return null; + } + + int result = 0; + int bitsRead = 0; + + while (bitsRead < bitCount) + { + if (bytePosition >= data.Length) + { + return null; + } + + int bitsAvailable = 8 - bitPosition; + int bitsToRead = Math.Min(bitCount - bitsRead, bitsAvailable); + + int mask = (1 << bitsToRead) - 1; + result |= ((data[bytePosition] >> bitPosition) & mask) << bitsRead; + + bitPosition += bitsToRead; + bitsRead += bitsToRead; + + if (bitPosition >= 8) + { + bitPosition = 0; + bytePosition++; + } + } + + return (ushort)result; + } + } +} diff --git a/src/SharpCompress/Compressors/Arj/BitReader.Async.cs b/src/SharpCompress/Compressors/Arj/BitReader.Async.cs new file mode 100644 index 00000000..b999327f --- /dev/null +++ b/src/SharpCompress/Compressors/Arj/BitReader.Async.cs @@ -0,0 +1,53 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Compressors.Arj; + +public partial class BitReader +{ + /// + /// Asynchronously reads a single bit from the stream. Returns 0 or 1. + /// + public async ValueTask ReadBitAsync(CancellationToken cancellationToken) + { + if (_bitCount == 0) + { + var buffer = new byte[1]; + int bytesRead = await _input + .ReadAsync(buffer, 0, 1, cancellationToken) + .ConfigureAwait(false); + if (bytesRead < 1) + { + throw new IncompleteArchiveException("No more data available in BitReader."); + } + + _bitBuffer = buffer[0]; + _bitCount = 8; + } + + int bit = (_bitBuffer >> (_bitCount - 1)) & 1; + _bitCount--; + return bit; + } + + /// + /// Asynchronously reads n bits (up to 32) from the stream. + /// + public async ValueTask ReadBitsAsync(int count, CancellationToken cancellationToken) + { + if (count < 0 || count > 32) + { + throw new ArgumentOutOfRangeException(nameof(count), "Count must be between 0 and 32."); + } + + int result = 0; + for (int i = 0; i < count; i++) + { + result = (result << 1) | await ReadBitAsync(cancellationToken).ConfigureAwait(false); + } + return result; + } +} diff --git a/src/SharpCompress/Compressors/Arj/BitReader.cs b/src/SharpCompress/Compressors/Arj/BitReader.cs new file mode 100644 index 00000000..640fc060 --- /dev/null +++ b/src/SharpCompress/Compressors/Arj/BitReader.cs @@ -0,0 +1,69 @@ +using System; +using System.IO; +using SharpCompress.Common; + +namespace SharpCompress.Compressors.Arj; + +[CLSCompliant(true)] +public partial class BitReader +{ + private readonly Stream _input; + private int _bitBuffer; // currently buffered bits + private int _bitCount; // number of bits in buffer + + public BitReader(Stream input) + { + _input = input ?? throw new ArgumentNullException(nameof(input)); + _bitBuffer = 0; + _bitCount = 0; + } + + /// + /// Reads a single bit from the stream. Returns 0 or 1. + /// + public int ReadBit() + { + if (_bitCount == 0) + { + int nextByte = _input.ReadByte(); + if (nextByte < 0) + { + throw new IncompleteArchiveException("No more data available in BitReader."); + } + + _bitBuffer = nextByte; + _bitCount = 8; + } + + int bit = (_bitBuffer >> (_bitCount - 1)) & 1; + _bitCount--; + return bit; + } + + /// + /// Reads n bits (up to 32) from the stream. + /// + public int ReadBits(int count) + { + if (count < 0 || count > 32) + { + throw new ArgumentOutOfRangeException(nameof(count), "Count must be between 0 and 32."); + } + + int result = 0; + for (int i = 0; i < count; i++) + { + result = (result << 1) | ReadBit(); + } + return result; + } + + /// + /// Resets any buffered bits. + /// + public void AlignToByte() + { + _bitCount = 0; + _bitBuffer = 0; + } +} diff --git a/src/SharpCompress/Compressors/Arj/HistoryIterator.cs b/src/SharpCompress/Compressors/Arj/HistoryIterator.cs new file mode 100644 index 00000000..2341be55 --- /dev/null +++ b/src/SharpCompress/Compressors/Arj/HistoryIterator.cs @@ -0,0 +1,42 @@ +using System; +using System.Collections; +using System.Collections.Generic; + +namespace SharpCompress.Compressors.Arj; + +/// +/// Iterator that reads & pushes values back into the ring buffer. +/// +public class HistoryIterator : IEnumerator +{ + private int _index; + private readonly IRingBuffer _ring; + + public HistoryIterator(IRingBuffer ring, int startIndex) + { + _ring = ring; + _index = startIndex; + } + + public bool MoveNext() + { + Current = _ring[_index]; + _index = unchecked(_index + 1); + + // Push value back into the ring buffer + _ring.Push(Current); + + return true; // iterator is infinite + } + + public void Reset() + { + throw new NotSupportedException(); + } + + public byte Current { get; private set; } + + object IEnumerator.Current => Current; + + public void Dispose() { } +} diff --git a/src/SharpCompress/Compressors/Arj/HuffmanTree.Async.cs b/src/SharpCompress/Compressors/Arj/HuffmanTree.Async.cs new file mode 100644 index 00000000..9f0c16e4 --- /dev/null +++ b/src/SharpCompress/Compressors/Arj/HuffmanTree.Async.cs @@ -0,0 +1,39 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Compressors.Arj; + +public sealed partial class HuffTree +{ + public async ValueTask ReadEntryAsync( + BitReader reader, + CancellationToken cancellationToken + ) + { + if (_tree.Count == 0) + { + throw new ArchiveOperationException("Tree not initialized"); + } + + TreeEntry node = _tree[0]; + while (true) + { + if (node.Type == NodeType.Leaf) + { + return node.LeafValue; + } + + int bit = await reader.ReadBitAsync(cancellationToken).ConfigureAwait(false); + int index = node.BranchIndex + bit; + + if (index >= _tree.Count) + { + throw new ArchiveOperationException("Invalid branch index during read"); + } + + node = _tree[index]; + } + } +} diff --git a/src/SharpCompress/Compressors/Arj/HuffmanTree.cs b/src/SharpCompress/Compressors/Arj/HuffmanTree.cs new file mode 100644 index 00000000..c9f5a505 --- /dev/null +++ b/src/SharpCompress/Compressors/Arj/HuffmanTree.cs @@ -0,0 +1,213 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Text; +using SharpCompress.Common; + +namespace SharpCompress.Compressors.Arj; + +[CLSCompliant(true)] +public enum NodeType +{ + Leaf, + Branch, +} + +[CLSCompliant(true)] +public sealed class TreeEntry +{ + public readonly NodeType Type; + public readonly int LeafValue; + public readonly int BranchIndex; + + public const int MAX_INDEX = 4096; + + private TreeEntry(NodeType type, int leafValue, int branchIndex) + { + Type = type; + LeafValue = leafValue; + BranchIndex = branchIndex; + } + + public static TreeEntry Leaf(int value) + { + return new TreeEntry(NodeType.Leaf, value, -1); + } + + public static TreeEntry Branch(int index) + { + if (index >= MAX_INDEX) + { + throw new ArgumentOutOfRangeException(nameof(index), "Branch index exceeds MAX_INDEX"); + } + return new TreeEntry(NodeType.Branch, 0, index); + } +} + +[CLSCompliant(true)] +public sealed partial class HuffTree +{ + private readonly List _tree; + + public HuffTree(int capacity = 0) + { + _tree = new List(capacity); + } + + public void SetSingle(int value) + { + _tree.Clear(); + _tree.Add(TreeEntry.Leaf(value)); + } + + public void BuildTree(byte[] lengths, int count) + { + ThrowHelper.ThrowIfNull(lengths); + + if (count < 0 || count > lengths.Length) + { + throw new ArgumentOutOfRangeException(nameof(count)); + } + + if (count > TreeEntry.MAX_INDEX / 2) + { + throw new ArgumentException( + $"Count exceeds maximum allowed: {TreeEntry.MAX_INDEX / 2}" + ); + } + byte[] slice = new byte[count]; + Array.Copy(lengths, slice, count); + + BuildTree(slice); + } + + public void BuildTree(byte[] valueLengths) + { + ThrowHelper.ThrowIfNull(valueLengths); + + if (valueLengths.Length > TreeEntry.MAX_INDEX / 2) + { + throw new ArchiveOperationException("Too many code lengths"); + } + + _tree.Clear(); + + int maxAllocated = 1; // start with a single (root) node + + for (byte currentLen = 1; ; currentLen++) + { + // add missing branches up to current limit + int maxLimit = maxAllocated; + + for (int i = _tree.Count; i < maxLimit; i++) + { + // TreeEntry.Branch may throw if index too large + try + { + _tree.Add(TreeEntry.Branch(maxAllocated)); + } + catch (ArgumentOutOfRangeException e) + { + _tree.Clear(); + throw new ArchiveOperationException("Branch index exceeds limit", e); + } + + // each branch node allocates two children + maxAllocated += 2; + } + + // fill tree with leaves found in the lengths table at the current length + bool moreLeaves = false; + + for (int value = 0; value < valueLengths.Length; value++) + { + byte len = valueLengths[value]; + if (len == currentLen) + { + _tree.Add(TreeEntry.Leaf(value)); + } + else if (len > currentLen) + { + moreLeaves = true; // there are more leaves to process + } + } + + // sanity check (too many leaves) + if (_tree.Count > maxAllocated) + { + throw new ArchiveOperationException("Too many leaves"); + } + + // stop when no longer finding longer codes + if (!moreLeaves) + { + break; + } + } + + // ensure tree is complete + if (_tree.Count != maxAllocated) + { + throw new ArchiveOperationException( + $"Missing some leaves: tree count = {_tree.Count}, expected = {maxAllocated}" + ); + } + } + + public int ReadEntry(BitReader reader) + { + if (_tree.Count == 0) + { + throw new ArchiveOperationException("Tree not initialized"); + } + + TreeEntry node = _tree[0]; + while (true) + { + if (node.Type == NodeType.Leaf) + { + return node.LeafValue; + } + + int bit = reader.ReadBit(); + int index = node.BranchIndex + bit; + + if (index >= _tree.Count) + { + throw new ArchiveOperationException("Invalid branch index during read"); + } + + node = _tree[index]; + } + } + + public override string ToString() + { + var result = new StringBuilder(); + + void FormatStep(int index, string prefix) + { + var node = _tree[index]; + if (node.Type == NodeType.Leaf) + { + result + .Append(prefix) + .Append(" -> ") + .Append(node.LeafValue.ToString(Constants.DefaultCultureInfo)) + .AppendLine(); + } + else + { + FormatStep(node.BranchIndex, prefix + "0"); + FormatStep(node.BranchIndex + 1, prefix + "1"); + } + } + + if (_tree.Count > 0) + { + FormatStep(0, ""); + } + + return result.ToString(); + } +} diff --git a/src/SharpCompress/Compressors/Arj/ILhaDecoderConfig.cs b/src/SharpCompress/Compressors/Arj/ILhaDecoderConfig.cs new file mode 100644 index 00000000..b546e14f --- /dev/null +++ b/src/SharpCompress/Compressors/Arj/ILhaDecoderConfig.cs @@ -0,0 +1,8 @@ +namespace SharpCompress.Compressors.Arj; + +public interface ILhaDecoderConfig +{ + int HistoryBits { get; } + int OffsetBits { get; } + RingBuffer RingBuffer { get; } +} diff --git a/src/SharpCompress/Compressors/Arj/IRingBuffer.cs b/src/SharpCompress/Compressors/Arj/IRingBuffer.cs new file mode 100644 index 00000000..dab0f3a6 --- /dev/null +++ b/src/SharpCompress/Compressors/Arj/IRingBuffer.cs @@ -0,0 +1,16 @@ +namespace SharpCompress.Compressors.Arj; + +public interface IRingBuffer +{ + int BufferSize { get; } + + int Cursor { get; } + void SetCursor(int pos); + + void Push(byte value); + + HistoryIterator IterFromOffset(int offset); + HistoryIterator IterFromPos(int pos); + + byte this[int index] { get; } +} diff --git a/src/SharpCompress/Compressors/Arj/LHDecoderStream.Async.cs b/src/SharpCompress/Compressors/Arj/LHDecoderStream.Async.cs new file mode 100644 index 00000000..b231a368 --- /dev/null +++ b/src/SharpCompress/Compressors/Arj/LHDecoderStream.Async.cs @@ -0,0 +1,197 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Compressors.Arj; + +public sealed partial class LHDecoderStream +{ + /// + /// Asynchronously decodes a single element (literal or back-reference) and appends it to _buffer. + /// Returns true if data was added, or false if all input has already been decoded. + /// + private async ValueTask DecodeNextAsync(CancellationToken cancellationToken) + { + if (_buffer.Count >= _originalSize) + { + _finishedDecoding = true; + return false; + } + + int len = await DecodeValAsync(0, 7, cancellationToken).ConfigureAwait(false); + if (len == 0) + { + byte nextChar = (byte) + await _bitReader.ReadBitsAsync(8, cancellationToken).ConfigureAwait(false); + _buffer.Add(nextChar); + } + else + { + int repCount = len + THRESHOLD - 1; + int backPtr = await DecodeValAsync(9, 13, cancellationToken).ConfigureAwait(false); + + if (backPtr >= _buffer.Count) + { + throw new InvalidFormatException("Invalid back_ptr in LH stream"); + } + + int srcIndex = _buffer.Count - 1 - backPtr; + for (int j = 0; j < repCount && _buffer.Count < _originalSize; j++) + { + byte b = _buffer[srcIndex]; + _buffer.Add(b); + srcIndex++; + // srcIndex may grow; it's allowed (source region can overlap destination) + } + } + + if (_buffer.Count >= _originalSize) + { + _finishedDecoding = true; + } + + return true; + } + + private async ValueTask DecodeValAsync( + int from, + int to, + CancellationToken cancellationToken + ) + { + int add = 0; + int bit = from; + + while ( + bit < to + && await _bitReader.ReadBitsAsync(1, cancellationToken).ConfigureAwait(false) == 1 + ) + { + add |= 1 << bit; + bit++; + } + + int res = + bit > 0 + ? await _bitReader.ReadBitsAsync(bit, cancellationToken).ConfigureAwait(false) + : 0; + return res + add; + } + + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + if (_disposed) + { + throw new ObjectDisposedException(nameof(LHDecoderStream)); + } + + ThrowHelper.ThrowIfNull(buffer); + + if (offset < 0 || count < 0 || offset + count > buffer.Length) + { + throw new ArgumentOutOfRangeException("offset/count"); + } + + if (_readPosition >= _originalSize) + { + return 0; // EOF + } + + int totalRead = 0; + + while (totalRead < count && _readPosition < _originalSize) + { + cancellationToken.ThrowIfCancellationRequested(); + + if (_readPosition >= _buffer.Count) + { + bool had = await DecodeNextAsync(cancellationToken).ConfigureAwait(false); + if (!had) + { + break; + } + } + + int available = _buffer.Count - (int)_readPosition; + if (available <= 0) + { + if (!_finishedDecoding) + { + continue; + } + break; + } + + int toCopy = Math.Min(available, count - totalRead); + _buffer.CopyTo((int)_readPosition, buffer, offset + totalRead, toCopy); + + _readPosition += toCopy; + totalRead += toCopy; + } + + return totalRead; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + if (_disposed) + { + throw new ObjectDisposedException(nameof(LHDecoderStream)); + } + + if (_readPosition >= _originalSize) + { + return 0; // EOF + } + + int totalRead = 0; + + while (totalRead < buffer.Length && _readPosition < _originalSize) + { + cancellationToken.ThrowIfCancellationRequested(); + + if (_readPosition >= _buffer.Count) + { + bool had = await DecodeNextAsync(cancellationToken).ConfigureAwait(false); + if (!had) + { + break; + } + } + + int available = _buffer.Count - (int)_readPosition; + if (available <= 0) + { + if (!_finishedDecoding) + { + continue; + } + break; + } + + int toCopy = Math.Min(available, buffer.Length - totalRead); + for (int i = 0; i < toCopy; i++) + { + buffer.Span[totalRead + i] = _buffer[(int)_readPosition + i]; + } + + _readPosition += toCopy; + totalRead += toCopy; + } + + return totalRead; + } +#endif +} diff --git a/src/SharpCompress/Compressors/Arj/LHDecoderStream.cs b/src/SharpCompress/Compressors/Arj/LHDecoderStream.cs new file mode 100644 index 00000000..210bf46e --- /dev/null +++ b/src/SharpCompress/Compressors/Arj/LHDecoderStream.cs @@ -0,0 +1,185 @@ +using System; +using System.Collections.Generic; +using System.IO; +using SharpCompress.Common; + +namespace SharpCompress.Compressors.Arj; + +[CLSCompliant(true)] +public sealed partial class LHDecoderStream : Stream +{ + private readonly BitReader _bitReader; + + // Buffer containing *all* bytes decoded so far. + private readonly List _buffer = new(); + + private long _readPosition; + private readonly int _originalSize; + private bool _finishedDecoding; + private bool _disposed; + + private const int THRESHOLD = 3; + + public LHDecoderStream(Stream compressedStream, int originalSize) + { + if (!compressedStream.CanRead) + { + throw new ArgumentException( + "compressedStream must be readable.", + nameof(compressedStream) + ); + } + + _bitReader = new BitReader(compressedStream); + _originalSize = originalSize; + _readPosition = 0; + _finishedDecoding = (originalSize == 0); + } + + public override bool CanRead => true; + public override bool CanSeek => false; + public override bool CanWrite => false; + + public override long Length => _originalSize; + + public override long Position + { + get => _readPosition; + set => throw new NotSupportedException(); + } + + /// + /// Decodes a single element (literal or back-reference) and appends it to _buffer. + /// Returns true if data was added, or false if all input has already been decoded. + /// + private bool DecodeNext() + { + if (_buffer.Count >= _originalSize) + { + _finishedDecoding = true; + return false; + } + + int len = DecodeVal(0, 7); + if (len == 0) + { + byte nextChar = (byte)_bitReader.ReadBits(8); + _buffer.Add(nextChar); + } + else + { + int repCount = len + THRESHOLD - 1; + int backPtr = DecodeVal(9, 13); + + if (backPtr >= _buffer.Count) + { + throw new InvalidFormatException("Invalid back_ptr in LH stream"); + } + + int srcIndex = _buffer.Count - 1 - backPtr; + for (int j = 0; j < repCount && _buffer.Count < _originalSize; j++) + { + byte b = _buffer[srcIndex]; + _buffer.Add(b); + srcIndex++; + // srcIndex may grow; it's allowed (source region can overlap destination) + } + } + + if (_buffer.Count >= _originalSize) + { + _finishedDecoding = true; + } + + return true; + } + + private int DecodeVal(int from, int to) + { + int add = 0; + int bit = from; + + while (bit < to && _bitReader.ReadBits(1) == 1) + { + add |= 1 << bit; + bit++; + } + + int res = bit > 0 ? _bitReader.ReadBits(bit) : 0; + return res + add; + } + + /// + /// Reads decompressed bytes into buffer[offset..offset+count]. + /// The method decodes additional data on demand when needed. + /// + public override int Read(byte[] buffer, int offset, int count) + { + if (_disposed) + { + throw new ObjectDisposedException(nameof(LHDecoderStream)); + } + + ThrowHelper.ThrowIfNull(buffer); + + if (offset < 0 || count < 0 || offset + count > buffer.Length) + { + throw new ArgumentOutOfRangeException("offset/count"); + } + + if (_readPosition >= _originalSize) + { + return 0; // EOF + } + + int totalRead = 0; + + while (totalRead < count && _readPosition < _originalSize) + { + if (_readPosition >= _buffer.Count) + { + bool had = DecodeNext(); + if (!had) + { + break; + } + } + + int available = _buffer.Count - (int)_readPosition; + if (available <= 0) + { + if (!_finishedDecoding) + { + continue; + } + break; + } + + int toCopy = Math.Min(available, count - totalRead); + _buffer.CopyTo((int)_readPosition, buffer, offset + totalRead, toCopy); + + _readPosition += toCopy; + totalRead += toCopy; + } + + return totalRead; + } + + public override void Flush() => throw new NotSupportedException(); + + public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); + + public override void SetLength(long value) => throw new NotSupportedException(); + + public override void Write(byte[] buffer, int offset, int count) => + throw new NotSupportedException(); + + protected override void Dispose(bool disposing) + { + if (disposing && !_disposed) + { + _disposed = true; + } + base.Dispose(disposing); + } +} diff --git a/src/SharpCompress/Compressors/Arj/Lh5DecoderCfg.cs b/src/SharpCompress/Compressors/Arj/Lh5DecoderCfg.cs new file mode 100644 index 00000000..81f025a9 --- /dev/null +++ b/src/SharpCompress/Compressors/Arj/Lh5DecoderCfg.cs @@ -0,0 +1,8 @@ +namespace SharpCompress.Compressors.Arj; + +public class Lh5DecoderCfg : ILhaDecoderConfig +{ + public int HistoryBits => 14; + public int OffsetBits => 4; + public RingBuffer RingBuffer { get; } = new RingBuffer(1 << 14); +} diff --git a/src/SharpCompress/Compressors/Arj/Lh7DecoderCfg.cs b/src/SharpCompress/Compressors/Arj/Lh7DecoderCfg.cs new file mode 100644 index 00000000..a7393454 --- /dev/null +++ b/src/SharpCompress/Compressors/Arj/Lh7DecoderCfg.cs @@ -0,0 +1,8 @@ +namespace SharpCompress.Compressors.Arj; + +public class Lh7DecoderCfg : ILhaDecoderConfig +{ + public int HistoryBits => 17; + public int OffsetBits => 5; + public RingBuffer RingBuffer { get; } = new RingBuffer(1 << 17); +} diff --git a/src/SharpCompress/Compressors/Arj/LhaStream.Async.cs b/src/SharpCompress/Compressors/Arj/LhaStream.Async.cs new file mode 100644 index 00000000..0d092e3b --- /dev/null +++ b/src/SharpCompress/Compressors/Arj/LhaStream.Async.cs @@ -0,0 +1,400 @@ +using System; +using System.IO; +using System.IO.Compression; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Compressors.Arj; + +public sealed partial class LhaStream +{ + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + ThrowHelper.ThrowIfNull(buffer); + if (offset < 0 || count < 0 || (offset + count) > buffer.Length) + { + throw new ArgumentOutOfRangeException(nameof(offset)); + } + + if (_producedBytes >= _originalSize) + { + return 0; // EOF + } + if (count == 0) + { + return 0; + } + + int bytesRead = await FillBufferAsync(buffer, cancellationToken).ConfigureAwait(false); + return bytesRead; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + if (_producedBytes >= _originalSize) + { + return 0; // EOF + } + if (buffer.Length == 0) + { + return 0; + } + + int bytesRead = await FillBufferAsync(buffer, cancellationToken).ConfigureAwait(false); + return bytesRead; + } +#endif + + private async ValueTask ReadCodeLengthAsync(CancellationToken cancellationToken) + { + byte len = (byte)await _bitReader.ReadBitsAsync(3, cancellationToken).ConfigureAwait(false); + if (len == 7) + { + while (await _bitReader.ReadBitAsync(cancellationToken).ConfigureAwait(false) != 0) + { + len++; + if (len > 255) + { + throw new ArchiveOperationException("Code length overflow"); + } + } + } + return len; + } + + private async ValueTask ReadCodeSkipAsync( + int skipRange, + CancellationToken cancellationToken + ) + { + int bits; + int increment; + + switch (skipRange) + { + case 0: + return 1; + case 1: + bits = 4; + increment = 3; // 3..=18 + break; + default: + bits = 9; + increment = 20; // 20..=531 + break; + } + + int skip = await _bitReader.ReadBitsAsync(bits, cancellationToken).ConfigureAwait(false); + return skip + increment; + } + + private async ValueTask ReadTempTreeAsync(CancellationToken cancellationToken) + { + byte[] codeLengths = new byte[NUM_TEMP_CODELEN]; + + // number of codes to read (5 bits) + int numCodes = await _bitReader.ReadBitsAsync(5, cancellationToken).ConfigureAwait(false); + + // single code only + if (numCodes == 0) + { + int code = await _bitReader.ReadBitsAsync(5, cancellationToken).ConfigureAwait(false); + _offsetTree.SetSingle((byte)code); + return; + } + + if (numCodes > NUM_TEMP_CODELEN) + { + throw new InvalidFormatException("temporary codelen table has invalid size"); + } + + // read actual lengths + int count = Math.Min(3, numCodes); + for (int i = 0; i < count; i++) + { + codeLengths[i] = (byte) + await ReadCodeLengthAsync(cancellationToken).ConfigureAwait(false); + } + + // 2-bit skip value follows + int skip = await _bitReader.ReadBitsAsync(2, cancellationToken).ConfigureAwait(false); + + if (3 + skip > numCodes) + { + throw new InvalidFormatException("temporary codelen table has invalid size"); + } + + for (int i = 3 + skip; i < numCodes; i++) + { + codeLengths[i] = (byte) + await ReadCodeLengthAsync(cancellationToken).ConfigureAwait(false); + } + + _offsetTree.BuildTree(codeLengths, numCodes); + } + + private async ValueTask ReadCommandTreeAsync(CancellationToken cancellationToken) + { + byte[] codeLengths = new byte[NUM_COMMANDS]; + + // number of codes to read (9 bits) + int numCodes = await _bitReader.ReadBitsAsync(9, cancellationToken).ConfigureAwait(false); + + // single code only + if (numCodes == 0) + { + int code = await _bitReader.ReadBitsAsync(9, cancellationToken).ConfigureAwait(false); + _commandTree.SetSingle((ushort)code); + return; + } + + if (numCodes > NUM_COMMANDS) + { + throw new InvalidFormatException("commands codelen table has invalid size"); + } + + int index = 0; + while (index < numCodes) + { + for (int n = 0; n < numCodes - index; n++) + { + int code = await _offsetTree + .ReadEntryAsync(_bitReader, cancellationToken) + .ConfigureAwait(false); + + if (code >= 0 && code <= 2) // skip range + { + int skipCount = await ReadCodeSkipAsync(code, cancellationToken) + .ConfigureAwait(false); + index += n + skipCount; + goto outerLoop; + } + else + { + codeLengths[index + n] = (byte)(code - 2); + } + } + break; + + outerLoop: + ; + } + + _commandTree.BuildTree(codeLengths, numCodes); + } + + private async ValueTask ReadOffsetTreeAsync(CancellationToken cancellationToken) + { + int numCodes = await _bitReader + .ReadBitsAsync(_config.OffsetBits, cancellationToken) + .ConfigureAwait(false); + if (numCodes == 0) + { + int code = await _bitReader + .ReadBitsAsync(_config.OffsetBits, cancellationToken) + .ConfigureAwait(false); + _offsetTree.SetSingle(code); + return; + } + + if (numCodes > _config.HistoryBits) + { + throw new InvalidFormatException("Offset code table too large"); + } + + byte[] codeLengths = new byte[NUM_TEMP_CODELEN]; + for (int i = 0; i < numCodes; i++) + { + codeLengths[i] = (byte) + await ReadCodeLengthAsync(cancellationToken).ConfigureAwait(false); + } + + _offsetTree.BuildTree(codeLengths, numCodes); + } + + private async ValueTask BeginNewBlockAsync(CancellationToken cancellationToken) + { + await ReadTempTreeAsync(cancellationToken).ConfigureAwait(false); + await ReadCommandTreeAsync(cancellationToken).ConfigureAwait(false); + await ReadOffsetTreeAsync(cancellationToken).ConfigureAwait(false); + } + + private ValueTask ReadCommandAsync(CancellationToken cancellationToken) => + _commandTree.ReadEntryAsync(_bitReader, cancellationToken); + + private async ValueTask ReadOffsetAsync(CancellationToken cancellationToken) + { + int bits = await _offsetTree + .ReadEntryAsync(_bitReader, cancellationToken) + .ConfigureAwait(false); + if (bits <= 1) + { + return bits; + } + + int res = await _bitReader.ReadBitsAsync(bits - 1, cancellationToken).ConfigureAwait(false); + return res | (1 << (bits - 1)); + } + + public async ValueTask FillBufferAsync(byte[] buffer, CancellationToken cancellationToken) + { + int bufLen = buffer.Length; + int bufIndex = 0; + + // stop when we reached original size + if (_producedBytes >= _originalSize) + { + return 0; + } + + // calculate limit, so that we don't go over the original size + int remaining = (int)Math.Min(bufLen, _originalSize - _producedBytes); + + while (bufIndex < remaining) + { + if (_copyProgress.HasValue) + { + var (offset, count) = _copyProgress.Value; + int copied = CopyFromHistory( + buffer, + bufIndex, + offset, + (int)Math.Min(count, remaining - bufIndex) + ); + bufIndex += copied; + _copyProgress = null; + } + + if (_remainingCommands == 0) + { + _remainingCommands = await _bitReader + .ReadBitsAsync(16, cancellationToken) + .ConfigureAwait(false); + if (bufIndex + _remainingCommands > remaining) + { + break; + } + await BeginNewBlockAsync(cancellationToken).ConfigureAwait(false); + } + + _remainingCommands--; + + int command = await ReadCommandAsync(cancellationToken).ConfigureAwait(false); + + if (command >= 0 && command <= 0xFF) + { + byte value = (byte)command; + buffer[bufIndex++] = value; + _ringBuffer.Push(value); + } + else + { + int count = command - 0x100 + 3; + int offset = await ReadOffsetAsync(cancellationToken).ConfigureAwait(false); + int copyCount = (int)Math.Min(count, remaining - bufIndex); + bufIndex += CopyFromHistory(buffer, bufIndex, offset, copyCount); + } + } + + _producedBytes += bufIndex; + return bufIndex; + } + +#if !LEGACY_DOTNET + public async ValueTask FillBufferAsync( + Memory buffer, + CancellationToken cancellationToken + ) + { + int bufLen = buffer.Length; + int bufIndex = 0; + + // stop when we reached original size + if (_producedBytes >= _originalSize) + { + return 0; + } + + // calculate limit, so that we don't go over the original size + int remaining = (int)Math.Min(bufLen, _originalSize - _producedBytes); + + while (bufIndex < remaining) + { + if (_copyProgress.HasValue) + { + var (offset, count) = _copyProgress.Value; + int copied = CopyFromHistory( + buffer.Span, + bufIndex, + offset, + (int)Math.Min(count, remaining - bufIndex) + ); + bufIndex += copied; + _copyProgress = null; + } + + if (_remainingCommands == 0) + { + _remainingCommands = await _bitReader + .ReadBitsAsync(16, cancellationToken) + .ConfigureAwait(false); + if (bufIndex + _remainingCommands > remaining) + { + break; + } + await BeginNewBlockAsync(cancellationToken).ConfigureAwait(false); + } + + _remainingCommands--; + + int command = await ReadCommandAsync(cancellationToken).ConfigureAwait(false); + + if (command >= 0 && command <= 0xFF) + { + byte value = (byte)command; + buffer.Span[bufIndex++] = value; + _ringBuffer.Push(value); + } + else + { + int count = command - 0x100 + 3; + int offset = await ReadOffsetAsync(cancellationToken).ConfigureAwait(false); + int copyCount = (int)Math.Min(count, remaining - bufIndex); + bufIndex += CopyFromHistory(buffer.Span, bufIndex, offset, copyCount); + } + } + + _producedBytes += bufIndex; + return bufIndex; + } + + private int CopyFromHistory(Span target, int targetIndex, int offset, int count) + { + var historyIter = _ringBuffer.IterFromOffset(offset); + int copied = 0; + + while (copied < count && historyIter.MoveNext() && (targetIndex + copied) < target.Length) + { + target[targetIndex + copied] = historyIter.Current; + copied++; + } + + if (copied < count) + { + _copyProgress = (offset, count - copied); + } + + return copied; + } +#endif +} diff --git a/src/SharpCompress/Compressors/Arj/LhaStream.cs b/src/SharpCompress/Compressors/Arj/LhaStream.cs new file mode 100644 index 00000000..f7c384ac --- /dev/null +++ b/src/SharpCompress/Compressors/Arj/LhaStream.cs @@ -0,0 +1,335 @@ +using System; +using System.Data; +using System.IO; +using System.IO.Compression; +using System.Linq; +using SharpCompress.Common; + +namespace SharpCompress.Compressors.Arj; + +[CLSCompliant(true)] +public sealed partial class LhaStream : Stream + where TDecoderConfig : ILhaDecoderConfig, new() +{ + private readonly BitReader _bitReader; + + private readonly HuffTree _commandTree; + private readonly HuffTree _offsetTree; + private int _remainingCommands; + private (int offset, int count)? _copyProgress; + private readonly RingBuffer _ringBuffer; + private readonly TDecoderConfig _config = new TDecoderConfig(); + + private const int NUM_COMMANDS = 510; + private const int NUM_TEMP_CODELEN = 20; + + private readonly int _originalSize; + private int _producedBytes = 0; + + public LhaStream(Stream compressedStream, int originalSize) + { + _bitReader = new BitReader(compressedStream); + _ringBuffer = _config.RingBuffer; + _commandTree = new HuffTree(NUM_COMMANDS * 2); + _offsetTree = new HuffTree(NUM_TEMP_CODELEN * 2); + _remainingCommands = 0; + _copyProgress = null; + _originalSize = originalSize; + } + + public override bool CanRead => true; + public override bool CanSeek => false; + public override bool CanWrite => false; + public override long Length => throw new NotSupportedException(); + public override long Position + { + get => throw new NotSupportedException(); + set => throw new NotSupportedException(); + } + + public override void Flush() { } + + public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); + + public override void SetLength(long value) => throw new NotSupportedException(); + + public override void Write(byte[] buffer, int offset, int count) => + throw new NotSupportedException(); + + public override int Read(byte[] buffer, int offset, int count) + { + ThrowHelper.ThrowIfNull(buffer); + if (offset < 0 || count < 0 || (offset + count) > buffer.Length) + { + throw new ArgumentOutOfRangeException(nameof(offset)); + } + + if (_producedBytes >= _originalSize) + { + return 0; // EOF + } + if (count == 0) + { + return 0; + } + + int bytesRead = FillBuffer(buffer); + return bytesRead; + } + + private byte ReadCodeLength() + { + byte len = (byte)_bitReader.ReadBits(3); + if (len == 7) + { + while (_bitReader.ReadBit() != 0) + { + len++; + if (len > 255) + { + throw new ArchiveOperationException("Code length overflow"); + } + } + } + return len; + } + + private int ReadCodeSkip(int skipRange) + { + int bits; + int increment; + + switch (skipRange) + { + case 0: + return 1; + case 1: + bits = 4; + increment = 3; // 3..=18 + break; + default: + bits = 9; + increment = 20; // 20..=531 + break; + } + + int skip = _bitReader.ReadBits(bits); + return skip + increment; + } + + private void ReadTempTree() + { + byte[] codeLengths = new byte[NUM_TEMP_CODELEN]; + + // number of codes to read (5 bits) + int numCodes = _bitReader.ReadBits(5); + + // single code only + if (numCodes == 0) + { + int code = _bitReader.ReadBits(5); + _offsetTree.SetSingle((byte)code); + return; + } + + if (numCodes > NUM_TEMP_CODELEN) + { + throw new InvalidFormatException("temporary codelen table has invalid size"); + } + + // read actual lengths + int count = Math.Min(3, numCodes); + for (int i = 0; i < count; i++) + { + codeLengths[i] = (byte)ReadCodeLength(); + } + + // 2-bit skip value follows + int skip = _bitReader.ReadBits(2); + + if (3 + skip > numCodes) + { + throw new InvalidFormatException("temporary codelen table has invalid size"); + } + + for (int i = 3 + skip; i < numCodes; i++) + { + codeLengths[i] = (byte)ReadCodeLength(); + } + + _offsetTree.BuildTree(codeLengths, numCodes); + } + + private void ReadCommandTree() + { + byte[] codeLengths = new byte[NUM_COMMANDS]; + + // number of codes to read (9 bits) + int numCodes = _bitReader.ReadBits(9); + + // single code only + if (numCodes == 0) + { + int code = _bitReader.ReadBits(9); + _commandTree.SetSingle((ushort)code); + return; + } + + if (numCodes > NUM_COMMANDS) + { + throw new InvalidFormatException("commands codelen table has invalid size"); + } + + int index = 0; + while (index < numCodes) + { + for (int n = 0; n < numCodes - index; n++) + { + int code = _offsetTree.ReadEntry(_bitReader); + + if (code >= 0 && code <= 2) // skip range + { + int skipCount = ReadCodeSkip(code); + index += n + skipCount; + goto outerLoop; + } + else + { + codeLengths[index + n] = (byte)(code - 2); + } + } + break; + + outerLoop: + ; + } + + _commandTree.BuildTree(codeLengths, numCodes); + } + + private void ReadOffsetTree() + { + int numCodes = _bitReader.ReadBits(_config.OffsetBits); + if (numCodes == 0) + { + int code = _bitReader.ReadBits(_config.OffsetBits); + _offsetTree.SetSingle(code); + return; + } + + if (numCodes > _config.HistoryBits) + { + throw new InvalidFormatException("Offset code table too large"); + } + + byte[] codeLengths = new byte[NUM_TEMP_CODELEN]; + for (int i = 0; i < numCodes; i++) + { + codeLengths[i] = (byte)ReadCodeLength(); + } + + _offsetTree.BuildTree(codeLengths, numCodes); + } + + private void BeginNewBlock() + { + ReadTempTree(); + ReadCommandTree(); + ReadOffsetTree(); + } + + private int ReadCommand() => _commandTree.ReadEntry(_bitReader); + + private int ReadOffset() + { + int bits = _offsetTree.ReadEntry(_bitReader); + if (bits <= 1) + { + return bits; + } + + int res = _bitReader.ReadBits(bits - 1); + return res | (1 << (bits - 1)); + } + + private int CopyFromHistory(byte[] target, int targetIndex, int offset, int count) + { + var historyIter = _ringBuffer.IterFromOffset(offset); + int copied = 0; + + while (copied < count && historyIter.MoveNext() && (targetIndex + copied) < target.Length) + { + target[targetIndex + copied] = historyIter.Current; + copied++; + } + + if (copied < count) + { + _copyProgress = (offset, count - copied); + } + + return copied; + } + + public int FillBuffer(byte[] buffer) + { + int bufLen = buffer.Length; + int bufIndex = 0; + + // stop when we reached original size + if (_producedBytes >= _originalSize) + { + return 0; + } + + // calculate limit, so that we don't go over the original size + int remaining = (int)Math.Min(bufLen, _originalSize - _producedBytes); + + while (bufIndex < remaining) + { + if (_copyProgress.HasValue) + { + var (offset, count) = _copyProgress.Value; + int copied = CopyFromHistory( + buffer, + bufIndex, + offset, + (int)Math.Min(count, remaining - bufIndex) + ); + bufIndex += copied; + _copyProgress = null; + } + + if (_remainingCommands == 0) + { + _remainingCommands = _bitReader.ReadBits(16); + if (bufIndex + _remainingCommands > remaining) + { + break; + } + BeginNewBlock(); + } + + _remainingCommands--; + + int command = ReadCommand(); + + if (command >= 0 && command <= 0xFF) + { + byte value = (byte)command; + buffer[bufIndex++] = value; + _ringBuffer.Push(value); + } + else + { + int count = command - 0x100 + 3; + int offset = ReadOffset(); + int copyCount = (int)Math.Min(count, remaining - bufIndex); + bufIndex += CopyFromHistory(buffer, bufIndex, offset, copyCount); + } + } + + _producedBytes += bufIndex; + return bufIndex; + } +} diff --git a/src/SharpCompress/Compressors/Arj/RingBuffer.cs b/src/SharpCompress/Compressors/Arj/RingBuffer.cs new file mode 100644 index 00000000..39285b97 --- /dev/null +++ b/src/SharpCompress/Compressors/Arj/RingBuffer.cs @@ -0,0 +1,66 @@ +using System; +using System.Collections; +using System.Collections.Generic; + +namespace SharpCompress.Compressors.Arj; + +/// +/// A fixed-size ring buffer where N must be a power of two. +/// +public class RingBuffer : IRingBuffer +{ + private readonly byte[] _buffer; + private int _cursor; + + public int BufferSize { get; } + + public int Cursor => _cursor; + + private readonly int _mask; + + public RingBuffer(int size) + { + if ((size & (size - 1)) != 0) + { + throw new ArgumentException("RingArrayBuffer size must be a power of two"); + } + + BufferSize = size; + _buffer = new byte[size]; + _cursor = 0; + _mask = size - 1; + + // Fill with spaces + for (int i = 0; i < size; i++) + { + _buffer[i] = (byte)' '; + } + } + + public void SetCursor(int pos) + { + _cursor = pos & _mask; + } + + public void Push(byte value) + { + int index = _cursor; + _buffer[index & _mask] = value; + _cursor = (index + 1) & _mask; + } + + public byte this[int index] => _buffer[index & _mask]; + + public HistoryIterator IterFromOffset(int offset) + { + int masked = (offset & _mask) + 1; + int startIndex = _cursor + BufferSize - masked; + return new HistoryIterator(this, startIndex); + } + + public HistoryIterator IterFromPos(int pos) + { + int startIndex = pos & _mask; + return new HistoryIterator(this, startIndex); + } +} diff --git a/src/SharpCompress/Compressors/BZip2/BZip2Constants.cs b/src/SharpCompress/Compressors/BZip2/BZip2Constants.cs index cfeb02e4..85c63b39 100644 --- a/src/SharpCompress/Compressors/BZip2/BZip2Constants.cs +++ b/src/SharpCompress/Compressors/BZip2/BZip2Constants.cs @@ -555,6 +555,6 @@ internal class BZip2Constants 858, 364, 936, - 638 + 638, }; } diff --git a/src/SharpCompress/Compressors/BZip2/BZip2Stream.Async.cs b/src/SharpCompress/Compressors/BZip2/BZip2Stream.Async.cs new file mode 100644 index 00000000..35cf9f1d --- /dev/null +++ b/src/SharpCompress/Compressors/BZip2/BZip2Stream.Async.cs @@ -0,0 +1,146 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Compressors.BZip2; + +public sealed partial class BZip2Stream : IAsyncDisposable +{ + /// + /// Asynchronously finalizes the BZip2 compressed stream, flushing all pending data. + /// Use this instead of when writing to an async-only stream. + /// + public async ValueTask FinishAsync(CancellationToken cancellationToken = default) + { + if (stream is CBZip2OutputStream output) + { + await output.FinishAsync(cancellationToken).ConfigureAwait(false); + } + } + + /// + /// Create a BZip2Stream asynchronously + /// + /// The stream to read from + /// Compression Mode + /// Decompress Concatenated + /// Leave the underlying stream open when this stream is disposed + /// + /// Decompression only. When true, an end-of-stream reached at a bzip2 block boundary is treated as a + /// normal end of stream rather than throwing. This allows decoding a truncated or partial stream - for + /// example a sub-range of blocks extracted for random access - that has no trailing stream footer. EOF + /// in the middle of a block is still reported as an error. Because a partial decode's running combined + /// CRC won't match the whole-stream value stored in the footer, that whole-stream CRC is not verified + /// in this mode (per-block CRCs are still checked). + /// + /// Cancellation Token + public static async ValueTask CreateAsync( + Stream stream, + CompressionMode compressionMode, + bool decompressConcatenated, + bool leaveOpen = false, + bool tolerateTruncatedStream = false, + CancellationToken cancellationToken = default + ) + { + var bZip2Stream = new BZip2Stream(); + bZip2Stream.Mode = compressionMode; + if (bZip2Stream.Mode == CompressionMode.Compress) + { + bZip2Stream.stream = new CBZip2OutputStream(stream, leaveOpen); + } + else + { + bZip2Stream.stream = await CBZip2InputStream + .CreateAsync( + stream, + decompressConcatenated, + leaveOpen, + tolerateTruncatedStream, + cancellationToken + ) + .ConfigureAwait(false); + } + + return bZip2Stream; + } + + /// + /// Asynchronously consumes two bytes to test if there is a BZip2 header + /// + /// + /// + /// + public static async ValueTask IsBZip2Async( + Stream stream, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + var buffer = new byte[2]; + var bytesRead = await stream + .ReadAsync(buffer, 0, 2, cancellationToken) + .ConfigureAwait(false); + if (bytesRead < 2 || buffer[0] != 'B' || buffer[1] != 'Z') + { + return false; + } + return true; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) => await stream.ReadAsync(buffer, cancellationToken).ConfigureAwait(false); + + public override async ValueTask WriteAsync( + ReadOnlyMemory buffer, + CancellationToken cancellationToken = default + ) => await stream.WriteAsync(buffer, cancellationToken).ConfigureAwait(false); +#endif + + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken = default + ) => await stream.ReadAsync(buffer, offset, count, cancellationToken).ConfigureAwait(false); + + public override async Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken = default + ) => await stream.WriteAsync(buffer, offset, count, cancellationToken).ConfigureAwait(false); + +#if !LEGACY_DOTNET || NETSTANDARD2_1 + public override async ValueTask DisposeAsync() +#else + public async ValueTask DisposeAsync() +#endif + { + if (isDisposed) + { + return; + } + + isDisposed = true; + if (stream is IAsyncDisposable asyncDisposable) + { + await asyncDisposable.DisposeAsync().ConfigureAwait(false); + } + else + { + stream.Dispose(); + } + +#if !LEGACY_DOTNET || NETSTANDARD2_1 + await base.DisposeAsync().ConfigureAwait(false); +#else + await Task.CompletedTask.ConfigureAwait(false); +#endif + GC.SuppressFinalize(this); + } +} diff --git a/src/SharpCompress/Compressors/BZip2/BZip2Stream.cs b/src/SharpCompress/Compressors/BZip2/BZip2Stream.cs index feea12d1..dc0c175b 100644 --- a/src/SharpCompress/Compressors/BZip2/BZip2Stream.cs +++ b/src/SharpCompress/Compressors/BZip2/BZip2Stream.cs @@ -1,38 +1,70 @@ -using System; +using System; using System.IO; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Providers; namespace SharpCompress.Compressors.BZip2; -public sealed class BZip2Stream : Stream +public sealed partial class BZip2Stream : Stream, IFinishable { - private readonly Stream stream; + private Stream stream = default!; private bool isDisposed; + private BZip2Stream() { } + /// /// Create a BZip2Stream /// /// The stream to read from /// Compression Mode /// Decompress Concatenated - public BZip2Stream(Stream stream, CompressionMode compressionMode, bool decompressConcatenated) + /// Leave the underlying stream open when this stream is disposed + /// + /// Decompression only. When true, an end-of-stream reached at a bzip2 block boundary is treated as a + /// normal end of stream rather than throwing. This allows decoding a truncated or partial stream - for + /// example a sub-range of blocks extracted for random access - that has no trailing stream footer. EOF + /// in the middle of a block is still reported as an error. Because a partial decode's running combined + /// CRC won't match the whole-stream value stored in the footer, that whole-stream CRC is not verified + /// in this mode (per-block CRCs are still checked). + /// + public static BZip2Stream Create( + Stream stream, + CompressionMode compressionMode, + bool decompressConcatenated, + bool leaveOpen = false, + bool tolerateTruncatedStream = false + ) { - Mode = compressionMode; - if (Mode == CompressionMode.Compress) + var bZip2Stream = new BZip2Stream(); + bZip2Stream.Mode = compressionMode; + if (bZip2Stream.Mode == CompressionMode.Compress) { - this.stream = new CBZip2OutputStream(stream); + bZip2Stream.stream = new CBZip2OutputStream(stream, leaveOpen); } else { - this.stream = new CBZip2InputStream(stream, decompressConcatenated); + bZip2Stream.stream = CBZip2InputStream.Create( + stream, + decompressConcatenated, + leaveOpen, + tolerateTruncatedStream + ); } + + return bZip2Stream; } + public ValueTask FinishAsync() => (stream as CBZip2OutputStream)?.FinishAsync() ?? default; + public void Finish() => (stream as CBZip2OutputStream)?.Finish(); protected override void Dispose(bool disposing) { if (isDisposed) { + base.Dispose(disposing); return; } isDisposed = true; @@ -40,9 +72,10 @@ public sealed class BZip2Stream : Stream { stream.Dispose(); } + base.Dispose(disposing); } - public CompressionMode Mode { get; } + public CompressionMode Mode { get; private set; } public override bool CanRead => stream.CanRead; @@ -69,8 +102,7 @@ public sealed class BZip2Stream : Stream public override void SetLength(long value) => stream.SetLength(value); -#if !NETFRAMEWORK && !NETSTANDARD2_0 - +#if !LEGACY_DOTNET public override int Read(Span buffer) => stream.Read(buffer); public override void Write(ReadOnlySpan buffer) => stream.Write(buffer); @@ -88,7 +120,7 @@ public sealed class BZip2Stream : Stream /// public static bool IsBZip2(Stream stream) { - var br = new BinaryReader(stream); + using var br = new BinaryReader(stream, Encoding.Default, leaveOpen: true); var chars = br.ReadBytes(2); if (chars.Length < 2 || chars[0] != 'B' || chars[1] != 'Z') { @@ -96,4 +128,6 @@ public sealed class BZip2Stream : Stream } return true; } + + // Async methods moved to BZip2Stream.Async.cs } diff --git a/src/SharpCompress/Compressors/BZip2/CBZip2InputStream.Async.cs b/src/SharpCompress/Compressors/BZip2/CBZip2InputStream.Async.cs new file mode 100644 index 00000000..c1053216 --- /dev/null +++ b/src/SharpCompress/Compressors/BZip2/CBZip2InputStream.Async.cs @@ -0,0 +1,974 @@ +#nullable disable + +using System; +using System.Buffers; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.IO; + +namespace SharpCompress.Compressors.BZip2; + +internal partial class CBZip2InputStream +{ + public async ValueTask ReadByteAsync(CancellationToken cancellationToken) + { + if (streamEnd) + { + return -1; + } + var retChar = currentChar; + switch (currentState) + { + case START_BLOCK_STATE: + break; + case RAND_PART_A_STATE: + break; + case RAND_PART_B_STATE: + await SetupRandPartBAsync(cancellationToken).ConfigureAwait(false); + break; + case RAND_PART_C_STATE: + await SetupRandPartCAsync(cancellationToken).ConfigureAwait(false); + break; + case NO_RAND_PART_A_STATE: + break; + case NO_RAND_PART_B_STATE: + await SetupNoRandPartBAsync(cancellationToken).ConfigureAwait(false); + break; + case NO_RAND_PART_C_STATE: + await SetupNoRandPartCAsync(cancellationToken).ConfigureAwait(false); + break; + default: + break; + } + return retChar; + } + + private async ValueTask InitializeAsync( + bool isFirstStream, + CancellationToken cancellationToken + ) + { + var singleByte = new byte[1]; + var read0 = await bsStream + .ReadAsync(singleByte, 0, 1, cancellationToken) + .ConfigureAwait(false); + var magic0 = read0 == 0 ? -1 : singleByte[0]; + var read1 = await bsStream + .ReadAsync(singleByte, 0, 1, cancellationToken) + .ConfigureAwait(false); + var magic1 = read1 == 0 ? -1 : singleByte[0]; + var read2 = await bsStream + .ReadAsync(singleByte, 0, 1, cancellationToken) + .ConfigureAwait(false); + var magic2 = read2 == 0 ? -1 : singleByte[0]; + if (magic0 == -1 && !isFirstStream) + { + return false; + } + if (magic0 != 'B' || magic1 != 'Z' || magic2 != 'h') + { + throw new InvalidFormatException("Not a BZIP2 marked stream"); + } + var read3 = await bsStream + .ReadAsync(singleByte, 0, 1, cancellationToken) + .ConfigureAwait(false); + var magic3 = read3 == 0 ? -1 : singleByte[0]; + if (magic3 < '1' || magic3 > '9') + { + BsFinishedWithStream(); + streamEnd = true; + return false; + } + + SetDecompressStructureSizes(magic3 - '0'); + bsLive = 0; + computedCombinedCRC = 0; + return true; + } + + private async ValueTask InitBlockAsync(CancellationToken cancellationToken) + { + char magic1, + magic2, + magic3, + magic4; + char magic5, + magic6; + + while (true) + { + // A clean EOF is only acceptable here, at the start of a block/footer header. + expectingBlockStart = tolerateTruncatedStream; + magic1 = await BsGetUCharAsync(cancellationToken).ConfigureAwait(false); + magic2 = await BsGetUCharAsync(cancellationToken).ConfigureAwait(false); + magic3 = await BsGetUCharAsync(cancellationToken).ConfigureAwait(false); + magic4 = await BsGetUCharAsync(cancellationToken).ConfigureAwait(false); + magic5 = await BsGetUCharAsync(cancellationToken).ConfigureAwait(false); + magic6 = await BsGetUCharAsync(cancellationToken).ConfigureAwait(false); + expectingBlockStart = false; + + if (hitEof) + { + // tolerateTruncatedStream: the input ended at a block boundary (no stream footer). Treat + // it as the end of the stream rather than throwing. + BsFinishedWithStream(); + streamEnd = true; + return; + } + + if ( + magic1 != 0x17 + || magic2 != 0x72 + || magic3 != 0x45 + || magic4 != 0x38 + || magic5 != 0x50 + || magic6 != 0x90 + ) + { + break; + } + + if (await CompleteAsync(cancellationToken).ConfigureAwait(false)) + { + return; + } + } + + if ( + magic1 != 0x31 + || magic2 != 0x41 + || magic3 != 0x59 + || magic4 != 0x26 + || magic5 != 0x53 + || magic6 != 0x59 + ) + { + BadBlockHeader(); + streamEnd = true; + return; + } + + storedBlockCRC = await BsGetInt32Async(cancellationToken).ConfigureAwait(false); + + if (await BsRAsync(1, cancellationToken).ConfigureAwait(false) == 1) + { + blockRandomised = true; + } + else + { + blockRandomised = false; + } + + // currBlockNo++; + await GetAndMoveToFrontDecodeAsync(cancellationToken).ConfigureAwait(false); + + mCrc.InitialiseCRC(); + currentState = START_BLOCK_STATE; + } + + private async ValueTask CompleteAsync(CancellationToken cancellationToken) + { + storedCombinedCRC = await BsGetInt32Async(cancellationToken).ConfigureAwait(false); + // See Complete() in CBZip2InputStream.cs: the whole-stream combined CRC is not verified in + // tolerateTruncatedStream mode (a partial decode won't match); per-block CRCs still are. + if (!tolerateTruncatedStream && storedCombinedCRC != computedCombinedCRC) + { + CrcError(); + } + + var complete = + !decompressConcatenated + || !(await InitializeAsync(false, cancellationToken).ConfigureAwait(false)); + if (complete) + { + BsFinishedWithStream(); + streamEnd = true; + } + + // Look for the next .bz2 stream if decompressing + // concatenated files. + return complete; + } + + private async ValueTask BsGetintAsync(CancellationToken cancellationToken) + { + var u = 0; + u = (u << 8) | (await BsRAsync(8, cancellationToken).ConfigureAwait(false)); + u = (u << 8) | (await BsRAsync(8, cancellationToken).ConfigureAwait(false)); + u = (u << 8) | (await BsRAsync(8, cancellationToken).ConfigureAwait(false)); + u = (u << 8) | (await BsRAsync(8, cancellationToken).ConfigureAwait(false)); + return u; + } + + private async ValueTask RecvDecodingTablesAsync(CancellationToken cancellationToken) + { + var len = InitCharArray(BZip2Constants.N_GROUPS, BZip2Constants.MAX_ALPHA_SIZE); + int i, + j, + t, + nGroups, + nSelectors, + alphaSize; + int minLen, + maxLen; + var inUse16 = new bool[16]; + + /* Receive the mapping table */ + for (i = 0; i < 16; i++) + { + if (await BsRAsync(1, cancellationToken).ConfigureAwait(false) == 1) + { + inUse16[i] = true; + } + else + { + inUse16[i] = false; + } + } + + for (i = 0; i < 256; i++) + { + inUse[i] = false; + } + + for (i = 0; i < 16; i++) + { + if (inUse16[i]) + { + for (j = 0; j < 16; j++) + { + if (await BsRAsync(1, cancellationToken).ConfigureAwait(false) == 1) + { + inUse[(i * 16) + j] = true; + } + } + } + } + + MakeMaps(); + alphaSize = nInUse + 2; + + /* Now the selectors */ + nGroups = await BsRAsync(3, cancellationToken).ConfigureAwait(false); + if (nGroups < 2 || nGroups > BZip2Constants.N_GROUPS) + { + throw new InvalidFormatException("BZip2: invalid number of Huffman trees"); + } + nSelectors = await BsRAsync(15, cancellationToken).ConfigureAwait(false); + for (i = 0; i < nSelectors; i++) + { + j = 0; + while (await BsRAsync(1, cancellationToken).ConfigureAwait(false) == 1) + { + j++; + if (j >= nGroups) + { + throw new InvalidFormatException("BZip2: invalid selector MTF value"); + } + } + if (i < BZip2Constants.MAX_SELECTORS) + { + selectorMtf[i] = (char)j; + } + } + + nSelectors = Math.Min(nSelectors, BZip2Constants.MAX_SELECTORS); + + /* Undo the MTF values for the selectors. */ + { + var pos = new char[BZip2Constants.N_GROUPS]; + char tmp, + v; + for (v = '\0'; v < nGroups; v++) + { + pos[v] = v; + } + + for (i = 0; i < nSelectors; i++) + { + v = selectorMtf[i]; + if (v >= nGroups) + { + throw new InvalidFormatException("BZip2: selector MTF value out of range"); + } + tmp = pos[v]; + while (v > 0) + { + pos[v] = pos[v - 1]; + v--; + } + pos[0] = tmp; + selector[i] = tmp; + } + } + + /* Now the coding tables */ + for (t = 0; t < nGroups; t++) + { + var curr = await BsRAsync(5, cancellationToken).ConfigureAwait(false); + for (i = 0; i < alphaSize; i++) + { + while (await BsRAsync(1, cancellationToken).ConfigureAwait(false) == 1) + { + if (await BsRAsync(1, cancellationToken).ConfigureAwait(false) == 0) + { + curr++; + } + else + { + curr--; + } + } + len[t][i] = (char)curr; + } + } + + /* Create the Huffman decoding tables */ + for (t = 0; t < nGroups; t++) + { + minLen = 32; + maxLen = 0; + for (i = 0; i < alphaSize; i++) + { + if (len[t][i] > maxLen) + { + maxLen = len[t][i]; + } + if (len[t][i] < minLen) + { + minLen = len[t][i]; + } + } + HbCreateDecodeTables(limit[t], basev[t], perm[t], len[t], minLen, maxLen, alphaSize); + minLens[t] = minLen; + } + } + + private async ValueTask GetAndMoveToFrontDecodeAsync(CancellationToken cancellationToken) + { + var yy = new char[256]; + int i, + j, + nextSym, + limitLast; + int EOB, + groupNo, + groupPos; + var singleByte = new byte[1]; + + limitLast = BZip2Constants.baseBlockSize * blockSize100k; + origPtr = await BsGetIntVSAsync(24, cancellationToken).ConfigureAwait(false); + + await RecvDecodingTablesAsync(cancellationToken).ConfigureAwait(false); + EOB = nInUse + 1; + groupNo = -1; + groupPos = 0; + + /* + Setting up the unzftab entries here is not strictly + necessary, but it does save having to do it later + in a separate pass, and so saves a block's worth of + cache misses. + */ + for (i = 0; i <= 255; i++) + { + unzftab[i] = 0; + } + + for (i = 0; i <= 255; i++) + { + yy[i] = (char)i; + } + + last = -1; + + { + int zt, + zn, + zvec, + zj; + if (groupPos == 0) + { + groupNo++; + groupPos = BZip2Constants.G_SIZE; + } + groupPos--; + if (groupNo < 0 || groupNo >= selector.Length) + { + throw new InvalidFormatException("BZip2: group selector out of range"); + } + zt = selector[groupNo]; + zn = minLens[zt]; + zvec = await BsRAsync(zn, cancellationToken).ConfigureAwait(false); + while (zvec > limit[zt][zn]) + { + zn++; + if (zn >= BZip2Constants.MAX_CODE_LEN) + { + throw new InvalidFormatException("BZip2: Huffman code too long"); + } + { + { + while (bsLive < 1) + { + int zzi; + int thech = '\0'; + try + { + var readCount = await bsStream + .ReadAsync(singleByte, 0, 1, cancellationToken) + .ConfigureAwait(false); + thech = readCount == 0 ? '\uffff' : singleByte[0]; + } + catch (IOException) + { + CompressedStreamEOF(); + } + if (thech == '\uffff') + { + CompressedStreamEOF(); + } + zzi = thech; + bsBuff = (bsBuff << 8) | (zzi & 0xff); + bsLive += 8; + } + } + zj = (bsBuff >> (bsLive - 1)) & 1; + bsLive--; + } + zvec = (zvec << 1) | zj; + } + { + int permIdx = zvec - basev[zt][zn]; + if (permIdx < 0 || permIdx >= perm[zt].Length) + { + throw new InvalidFormatException("BZip2: invalid Huffman symbol"); + } + nextSym = perm[zt][permIdx]; + } + } + + while (true) + { + if (nextSym == EOB) + { + break; + } + + if (nextSym == BZip2Constants.RUNA || nextSym == BZip2Constants.RUNB) + { + char ch; + var s = -1; + var N = 1; + do + { + if (nextSym == BZip2Constants.RUNA) + { + s += (0 + 1) * N; + } + else if (nextSym == BZip2Constants.RUNB) + { + s += (1 + 1) * N; + } + N *= 2; + { + int zt, + zn, + zvec, + zj; + if (groupPos == 0) + { + groupNo++; + groupPos = BZip2Constants.G_SIZE; + } + groupPos--; + if (groupNo < 0 || groupNo >= selector.Length) + { + throw new InvalidFormatException("BZip2: group selector out of range"); + } + zt = selector[groupNo]; + zn = minLens[zt]; + zvec = await BsRAsync(zn, cancellationToken).ConfigureAwait(false); + while (zvec > limit[zt][zn]) + { + zn++; + if (zn >= BZip2Constants.MAX_CODE_LEN) + { + throw new InvalidFormatException("BZip2: Huffman code too long"); + } + { + { + while (bsLive < 1) + { + int zzi; + int thech = '\0'; + try + { + var readCount = await bsStream + .ReadAsync(singleByte, 0, 1, cancellationToken) + .ConfigureAwait(false); + thech = readCount == 0 ? '\uffff' : singleByte[0]; + } + catch (IOException) + { + CompressedStreamEOF(); + } + if (thech == '\uffff') + { + CompressedStreamEOF(); + } + zzi = thech; + bsBuff = (bsBuff << 8) | (zzi & 0xff); + bsLive += 8; + } + } + zj = (bsBuff >> (bsLive - 1)) & 1; + bsLive--; + } + zvec = (zvec << 1) | zj; + } + { + int permIdx = zvec - basev[zt][zn]; + if (permIdx < 0 || permIdx >= perm[zt].Length) + { + throw new InvalidFormatException("BZip2: invalid Huffman symbol"); + } + nextSym = perm[zt][permIdx]; + } + } + } while (nextSym == BZip2Constants.RUNA || nextSym == BZip2Constants.RUNB); + + s++; + ch = seqToUnseq[yy[0]]; + unzftab[ch] += s; + + while (s > 0) + { + last++; + ll8[last] = ch; + s--; + } + + if (last >= limitLast) + { + BlockOverrun(); + } + } + else + { + char tmp; + last++; + if (last >= limitLast) + { + BlockOverrun(); + } + + if (nextSym - 1 < 0 || nextSym - 1 >= yy.Length) + { + throw new InvalidFormatException("BZip2: symbol out of range"); + } + tmp = yy[nextSym - 1]; + unzftab[seqToUnseq[tmp]]++; + ll8[last] = seqToUnseq[tmp]; + + /* + This loop is hammered during decompression, + hence the unrolling. + + for (j = nextSym-1; j > 0; j--) yy[j] = yy[j-1]; + */ + + j = nextSym - 1; + for (; j > 3; j -= 4) + { + yy[j] = yy[j - 1]; + yy[j - 1] = yy[j - 2]; + yy[j - 2] = yy[j - 3]; + yy[j - 3] = yy[j - 4]; + } + for (; j > 0; j--) + { + yy[j] = yy[j - 1]; + } + + yy[0] = tmp; + { + int zt, + zn, + zvec, + zj; + if (groupPos == 0) + { + groupNo++; + groupPos = BZip2Constants.G_SIZE; + } + groupPos--; + if (groupNo < 0 || groupNo >= selector.Length) + { + throw new InvalidFormatException("BZip2: group selector out of range"); + } + zt = selector[groupNo]; + zn = minLens[zt]; + zvec = await BsRAsync(zn, cancellationToken).ConfigureAwait(false); + while (zvec > limit[zt][zn]) + { + zn++; + if (zn >= BZip2Constants.MAX_CODE_LEN) + { + throw new InvalidFormatException("BZip2: Huffman code too long"); + } + { + { + while (bsLive < 1) + { + int zzi; + int thech = '\0'; + try + { + var readCount = await bsStream + .ReadAsync(singleByte, 0, 1, cancellationToken) + .ConfigureAwait(false); + thech = readCount == 0 ? '\uffff' : singleByte[0]; + } + catch (IOException) + { + CompressedStreamEOF(); + } + if (thech == '\uffff') + { + CompressedStreamEOF(); + } + zzi = thech; + bsBuff = (bsBuff << 8) | (zzi & 0xff); + bsLive += 8; + } + } + zj = (bsBuff >> (bsLive - 1)) & 1; + bsLive--; + } + zvec = (zvec << 1) | zj; + } + { + int permIdx = zvec - basev[zt][zn]; + if (permIdx < 0 || permIdx >= perm[zt].Length) + { + throw new InvalidFormatException("BZip2: invalid Huffman symbol"); + } + nextSym = perm[zt][permIdx]; + } + } + } + } + } + + private async ValueTask SetupBlockAsync(CancellationToken cancellationToken) + { + Span cftab = stackalloc int[257]; + char ch; + + cftab[0] = 0; + for (i = 1; i <= 256; i++) + { + cftab[i] = unzftab[i - 1]; + } + for (i = 1; i <= 256; i++) + { + cftab[i] += cftab[i - 1]; + } + + for (i = 0; i <= last; i++) + { + ch = ll8[i]; + if (cftab[ch] < 0 || cftab[ch] >= tt.Length) + { + throw new InvalidFormatException("BZip2: block data out of bounds"); + } + tt[cftab[ch]] = i; + cftab[ch]++; + } + + if (origPtr < 0 || origPtr >= tt.Length) + { + throw new InvalidFormatException("BZip2: origPtr out of bounds"); + } + tPos = tt[origPtr]; + + count = 0; + i2 = 0; + ch2 = 256; /* not a char and not EOF */ + + if (blockRandomised) + { + rNToGo = 0; + rTPos = 0; + await SetupRandPartAAsync(cancellationToken).ConfigureAwait(false); + } + else + { + SetupNoRandPartA(); + } + } + + private async ValueTask SetupRandPartAAsync(CancellationToken cancellationToken) + { + if (i2 <= last) + { + chPrev = ch2; + ch2 = ll8[tPos]; + tPos = tt[tPos]; + if (rNToGo == 0) + { + rNToGo = BZip2Constants.rNums[rTPos]; + rTPos++; + if (rTPos == 512) + { + rTPos = 0; + } + } + rNToGo--; + ch2 ^= (rNToGo == 1) ? (char)1 : (char)0; + i2++; + + currentChar = ch2; + currentState = RAND_PART_B_STATE; + mCrc.UpdateCRC(ch2); + } + else + { + EndBlock(); + await InitBlockAsync(cancellationToken).ConfigureAwait(false); + await SetupBlockAsync(cancellationToken).ConfigureAwait(false); + } + } + + private async ValueTask SetupNoRandPartAAsync(CancellationToken cancellationToken) + { + if (i2 <= last) + { + chPrev = ch2; + ch2 = ll8[tPos]; + tPos = tt[tPos]; + i2++; + + currentChar = ch2; + currentState = NO_RAND_PART_B_STATE; + mCrc.UpdateCRC(ch2); + } + else + { + EndBlock(); + await InitBlockAsync(cancellationToken).ConfigureAwait(false); + await SetupBlockAsync(cancellationToken).ConfigureAwait(false); + } + } + + private async ValueTask SetupRandPartBAsync(CancellationToken cancellationToken) + { + if (ch2 != chPrev) + { + currentState = RAND_PART_A_STATE; + count = 1; + await SetupRandPartAAsync(cancellationToken).ConfigureAwait(false); + } + else + { + count++; + if (count >= 4) + { + z = ll8[tPos]; + tPos = tt[tPos]; + if (rNToGo == 0) + { + rNToGo = BZip2Constants.rNums[rTPos]; + rTPos++; + if (rTPos == 512) + { + rTPos = 0; + } + } + rNToGo--; + z ^= (char)((rNToGo == 1) ? 1 : 0); + j2 = 0; + currentState = RAND_PART_C_STATE; + SetupRandPartC(); + } + else + { + currentState = RAND_PART_A_STATE; + await SetupRandPartAAsync(cancellationToken).ConfigureAwait(false); + } + } + } + + private async ValueTask SetupRandPartCAsync(CancellationToken cancellationToken) + { + if (j2 < z) + { + currentChar = ch2; + mCrc.UpdateCRC(ch2); + j2++; + } + else + { + currentState = RAND_PART_A_STATE; + i2++; + count = 0; + await SetupRandPartAAsync(cancellationToken).ConfigureAwait(false); + } + } + + private async ValueTask SetupNoRandPartBAsync(CancellationToken cancellationToken) + { + if (ch2 != chPrev) + { + currentState = NO_RAND_PART_A_STATE; + count = 1; + await SetupNoRandPartAAsync(cancellationToken).ConfigureAwait(false); + } + else + { + count++; + if (count >= 4) + { + z = ll8[tPos]; + tPos = tt[tPos]; + currentState = NO_RAND_PART_C_STATE; + j2 = 0; + await SetupNoRandPartCAsync(cancellationToken).ConfigureAwait(false); + } + else + { + currentState = NO_RAND_PART_A_STATE; + await SetupNoRandPartAAsync(cancellationToken).ConfigureAwait(false); + } + } + } + + private async ValueTask SetupNoRandPartCAsync(CancellationToken cancellationToken) + { + if (j2 < z) + { + currentChar = ch2; + mCrc.UpdateCRC(ch2); + j2++; + } + else + { + currentState = NO_RAND_PART_A_STATE; + i2++; + count = 0; + await SetupNoRandPartAAsync(cancellationToken).ConfigureAwait(false); + } + } + + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken = default + ) + { + var c = -1; + int k; + for (k = 0; k < count; ++k) + { + cancellationToken.ThrowIfCancellationRequested(); + c = await ReadByteAsync(cancellationToken).ConfigureAwait(false); + if (c == -1) + { + break; + } + buffer[k + offset] = (byte)c; + } + return k; + } + + private async ValueTask BsRAsync(int n, CancellationToken cancellationToken) + { + int v; + while (bsLive < n) + { + if (bsStream is null) + { + HandleCompressedStreamEof(); + if (hitEof) + { + return 0; + } + } + int zzi; + int thech = '\0'; + var b = ArrayPool.Shared.Rent(1); + try + { + await bsStream.ReadExactAsync(b, 0, 1, cancellationToken).ConfigureAwait(false); + thech = (char)b[0]; + } + catch (IOException) + { + HandleCompressedStreamEof(); + if (hitEof) + { + return 0; + } + } + finally + { + ArrayPool.Shared.Return(b); + } + if (thech == '\uffff') + { + HandleCompressedStreamEof(); + if (hitEof) + { + return 0; + } + } + zzi = thech; + bsBuff = (bsBuff << 8) | (zzi & 0xff); + bsLive += 8; + } + + v = (bsBuff >> (bsLive - n)) & ((1 << n) - 1); + bsLive -= n; + return v; + } + + private async ValueTask BsGetUCharAsync(CancellationToken cancellationToken) => + (char)await BsRAsync(8, cancellationToken).ConfigureAwait(false); + + private async ValueTask BsGetIntVSAsync( + int numBits, + CancellationToken cancellationToken + ) => await BsRAsync(numBits, cancellationToken).ConfigureAwait(false); + + private async ValueTask BsGetInt32Async(CancellationToken cancellationToken) => + await BsGetintAsync(cancellationToken).ConfigureAwait(false); + + public static async ValueTask CreateAsync( + Stream zStream, + bool decompressConcatenated, + bool leaveOpen = false, + bool tolerateTruncatedStream = false, + CancellationToken cancellationToken = default + ) + { + var cbZip2InputStream = new CBZip2InputStream( + decompressConcatenated, + leaveOpen, + tolerateTruncatedStream + ); + cbZip2InputStream.ll8 = null; + cbZip2InputStream.tt = null; + cbZip2InputStream.BsSetStream(zStream); + if (!await cbZip2InputStream.InitializeAsync(true, cancellationToken).ConfigureAwait(false)) + { + throw new InvalidFormatException("Not a valid BZip2 stream"); + } + await cbZip2InputStream.InitBlockAsync(cancellationToken).ConfigureAwait(false); + await cbZip2InputStream.SetupBlockAsync(cancellationToken).ConfigureAwait(false); + return cbZip2InputStream; + } +} diff --git a/src/SharpCompress/Compressors/BZip2/CBZip2InputStream.cs b/src/SharpCompress/Compressors/BZip2/CBZip2InputStream.cs index e03a5096..bcc0c73a 100644 --- a/src/SharpCompress/Compressors/BZip2/CBZip2InputStream.cs +++ b/src/SharpCompress/Compressors/BZip2/CBZip2InputStream.cs @@ -1,7 +1,11 @@ -#nullable disable +#nullable disable using System; +using System.Buffers; using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; /* * Copyright 2001,2004-2005 The Apache Software Foundation @@ -37,19 +41,35 @@ namespace SharpCompress.Compressors.BZip2; * start of the BZIP2 stream to make it compatible with other PGP programs. */ -internal class CBZip2InputStream : Stream +internal partial class CBZip2InputStream : Stream { private static void Cadvise() { //System.out.Println("CRC Error"); - //throw new CCoruptionError(); + throw new ArchiveOperationException("BZip2 error"); } private static void BadBGLengths() => Cadvise(); private static void BitStreamEOF() => Cadvise(); - private static void CompressedStreamEOF() => Cadvise(); + private static void CompressedStreamEOF() + { + throw new ArchiveOperationException("BZip2 compressed file ends unexpectedly"); + } + + // Handles the underlying stream running out while BsR needs more bits. In tolerateTruncatedStream + // mode an EOF that lands on a block boundary (expectingBlockStart) ends the stream cleanly; anywhere + // else - or when not tolerating truncation - it is an unexpected truncation and throws. + private void HandleCompressedStreamEof() + { + if (tolerateTruncatedStream && expectingBlockStart) + { + hitEof = true; + return; + } + CompressedStreamEOF(); + } private void MakeMaps() { @@ -87,7 +107,7 @@ internal class CBZip2InputStream : Stream private int bsBuff; private int bsLive; - private readonly CRC mCrc = new CRC(); + private readonly CRC mCrc = new(); private readonly bool[] inUse = new bool[256]; private int nInUse; @@ -142,6 +162,19 @@ internal class CBZip2InputStream : Stream private int computedBlockCRC, computedCombinedCRC; private readonly bool decompressConcatenated; + private readonly bool leaveOpen; + + // When true, an end-of-stream reached at a bzip2 block boundary (i.e. while reading a block header) + // is treated as a normal end of stream rather than throwing. Lets a caller decode a truncated or + // partial stream - e.g. a sub-range of blocks extracted for random access - that has no stream footer. + private readonly bool tolerateTruncatedStream; + + // true only while reading the 6 bytes of a block/stream-footer header, where a clean EOF is allowed + // (in tolerateTruncatedStream mode). EOF anywhere else is still an unexpected truncation. + private bool expectingBlockStart; + + // set by BsR when a tolerated end-of-stream is hit; checked by InitBlock to end the stream. + private bool hitEof; private int i2, count, @@ -155,20 +188,44 @@ internal class CBZip2InputStream : Stream private char z; private bool isDisposed; - public CBZip2InputStream(Stream zStream, bool decompressConcatenated) + private CBZip2InputStream( + bool decompressConcatenated, + bool leaveOpen, + bool tolerateTruncatedStream + ) { this.decompressConcatenated = decompressConcatenated; - ll8 = null; - tt = null; - BsSetStream(zStream); - Initialize(true); - InitBlock(); - SetupBlock(); + this.leaveOpen = leaveOpen; + this.tolerateTruncatedStream = tolerateTruncatedStream; + } + + public static CBZip2InputStream Create( + Stream zStream, + bool decompressConcatenated, + bool leaveOpen, + bool tolerateTruncatedStream = false + ) + { + var cbZip2InputStream = new CBZip2InputStream( + decompressConcatenated, + leaveOpen, + tolerateTruncatedStream + ); + cbZip2InputStream.ll8 = null; + cbZip2InputStream.tt = null; + cbZip2InputStream.BsSetStream(zStream); + if (!cbZip2InputStream.Initialize(true)) + { + throw new InvalidFormatException("Not a valid BZip2 stream"); + } + cbZip2InputStream.InitBlock(); + cbZip2InputStream.SetupBlock(); + return cbZip2InputStream; } protected override void Dispose(bool disposing) { - if (isDisposed) + if (isDisposed || leaveOpen) { return; } @@ -241,7 +298,7 @@ internal class CBZip2InputStream : Stream } if (magic0 != 'B' || magic1 != 'Z' || magic2 != 'h') { - throw new IOException("Not a BZIP2 marked stream"); + throw new InvalidFormatException("Not a BZIP2 marked stream"); } var magic3 = bsStream.ReadByte(); if (magic3 < '1' || magic3 > '9') @@ -268,12 +325,25 @@ internal class CBZip2InputStream : Stream while (true) { + // A clean EOF is only acceptable here, at the start of a block/footer header. + expectingBlockStart = tolerateTruncatedStream; magic1 = BsGetUChar(); magic2 = BsGetUChar(); magic3 = BsGetUChar(); magic4 = BsGetUChar(); magic5 = BsGetUChar(); magic6 = BsGetUChar(); + expectingBlockStart = false; + + if (hitEof) + { + // tolerateTruncatedStream: the input ended at a block boundary (no stream footer). Treat + // it as the end of the stream rather than throwing. + BsFinishedWithStream(); + streamEnd = true; + return; + } + if ( magic1 != 0x17 || magic2 != 0x72 @@ -340,7 +410,11 @@ internal class CBZip2InputStream : Stream private bool Complete() { storedCombinedCRC = BsGetInt32(); - if (storedCombinedCRC != computedCombinedCRC) + // In tolerateTruncatedStream mode the input may be only part of a stream (e.g. a sub-range of + // blocks decoded for random access), so the running combined CRC won't match the stored + // whole-stream value in the footer. Per-block CRCs are still validated; only this whole-stream + // check is skipped. + if (!tolerateTruncatedStream && storedCombinedCRC != computedCombinedCRC) { CrcError(); } @@ -365,7 +439,10 @@ internal class CBZip2InputStream : Stream private void BsFinishedWithStream() { - bsStream?.Dispose(); + if (!leaveOpen) + { + bsStream?.Dispose(); + } bsStream = null; } @@ -381,6 +458,14 @@ internal class CBZip2InputStream : Stream int v; while (bsLive < n) { + if (bsStream is null) + { + HandleCompressedStreamEof(); + if (hitEof) + { + return 0; + } + } int zzi; int thech = '\0'; try @@ -389,11 +474,19 @@ internal class CBZip2InputStream : Stream } catch (IOException) { - CompressedStreamEOF(); + HandleCompressedStreamEof(); + if (hitEof) + { + return 0; + } } if (thech == '\uffff') { - CompressedStreamEOF(); + HandleCompressedStreamEof(); + if (hitEof) + { + return 0; + } } zzi = thech; bsBuff = (bsBuff << 8) | (zzi & 0xff); @@ -455,6 +548,10 @@ internal class CBZip2InputStream : Stream } for (i = 0; i < alphaSize; i++) { + if (length[i] >= BZip2Constants.MAX_CODE_LEN) + { + throw new InvalidFormatException("BZip2: invalid Huffman code length"); + } basev[length[i] + 1]++; } @@ -531,6 +628,10 @@ internal class CBZip2InputStream : Stream /* Now the selectors */ nGroups = BsR(3); + if (nGroups < 2 || nGroups > BZip2Constants.N_GROUPS) + { + throw new InvalidFormatException("BZip2: invalid number of Huffman trees"); + } nSelectors = BsR(15); for (i = 0; i < nSelectors; i++) { @@ -538,10 +639,19 @@ internal class CBZip2InputStream : Stream while (BsR(1) == 1) { j++; + if (j >= nGroups) + { + throw new InvalidFormatException("BZip2: invalid selector MTF value"); + } + } + if (i < BZip2Constants.MAX_SELECTORS) + { + selectorMtf[i] = (char)j; } - selectorMtf[i] = (char)j; } + nSelectors = Math.Min(nSelectors, BZip2Constants.MAX_SELECTORS); + /* Undo the MTF values for the selectors. */ { var pos = new char[BZip2Constants.N_GROUPS]; @@ -555,6 +665,10 @@ internal class CBZip2InputStream : Stream for (i = 0; i < nSelectors; i++) { v = selectorMtf[i]; + if (v >= nGroups) + { + throw new InvalidFormatException("BZip2: selector MTF value out of range"); + } tmp = pos[v]; while (v > 0) { @@ -656,12 +770,20 @@ internal class CBZip2InputStream : Stream groupPos = BZip2Constants.G_SIZE; } groupPos--; + if (groupNo < 0 || groupNo >= selector.Length) + { + throw new InvalidFormatException("BZip2: group selector out of range"); + } zt = selector[groupNo]; zn = minLens[zt]; zvec = BsR(zn); while (zvec > limit[zt][zn]) { zn++; + if (zn >= BZip2Constants.MAX_CODE_LEN) + { + throw new InvalidFormatException("BZip2: Huffman code too long"); + } { { while (bsLive < 1) @@ -690,7 +812,14 @@ internal class CBZip2InputStream : Stream } zvec = (zvec << 1) | zj; } - nextSym = perm[zt][zvec - basev[zt][zn]]; + { + int permIdx = zvec - basev[zt][zn]; + if (permIdx < 0 || permIdx >= perm[zt].Length) + { + throw new InvalidFormatException("BZip2: invalid Huffman symbol"); + } + nextSym = perm[zt][permIdx]; + } } while (true) @@ -727,12 +856,20 @@ internal class CBZip2InputStream : Stream groupPos = BZip2Constants.G_SIZE; } groupPos--; + if (groupNo < 0 || groupNo >= selector.Length) + { + throw new InvalidFormatException("BZip2: group selector out of range"); + } zt = selector[groupNo]; zn = minLens[zt]; zvec = BsR(zn); while (zvec > limit[zt][zn]) { zn++; + if (zn >= BZip2Constants.MAX_CODE_LEN) + { + throw new InvalidFormatException("BZip2: Huffman code too long"); + } { { while (bsLive < 1) @@ -761,7 +898,14 @@ internal class CBZip2InputStream : Stream } zvec = (zvec << 1) | zj; } - nextSym = perm[zt][zvec - basev[zt][zn]]; + { + int permIdx = zvec - basev[zt][zn]; + if (permIdx < 0 || permIdx >= perm[zt].Length) + { + throw new InvalidFormatException("BZip2: invalid Huffman symbol"); + } + nextSym = perm[zt][permIdx]; + } } } while (nextSym == BZip2Constants.RUNA || nextSym == BZip2Constants.RUNB); @@ -790,6 +934,10 @@ internal class CBZip2InputStream : Stream BlockOverrun(); } + if (nextSym - 1 < 0 || nextSym - 1 >= yy.Length) + { + throw new InvalidFormatException("BZip2: symbol out of range"); + } tmp = yy[nextSym - 1]; unzftab[seqToUnseq[tmp]]++; ll8[last] = seqToUnseq[tmp]; @@ -826,12 +974,20 @@ internal class CBZip2InputStream : Stream groupPos = BZip2Constants.G_SIZE; } groupPos--; + if (groupNo < 0 || groupNo >= selector.Length) + { + throw new InvalidFormatException("BZip2: group selector out of range"); + } zt = selector[groupNo]; zn = minLens[zt]; zvec = BsR(zn); while (zvec > limit[zt][zn]) { zn++; + if (zn >= BZip2Constants.MAX_CODE_LEN) + { + throw new InvalidFormatException("BZip2: Huffman code too long"); + } { { while (bsLive < 1) @@ -856,7 +1012,14 @@ internal class CBZip2InputStream : Stream } zvec = (zvec << 1) | zj; } - nextSym = perm[zt][zvec - basev[zt][zn]]; + { + int permIdx = zvec - basev[zt][zn]; + if (permIdx < 0 || permIdx >= perm[zt].Length) + { + throw new InvalidFormatException("BZip2: invalid Huffman symbol"); + } + nextSym = perm[zt][permIdx]; + } } } } @@ -880,10 +1043,18 @@ internal class CBZip2InputStream : Stream for (i = 0; i <= last; i++) { ch = ll8[i]; + if (cftab[ch] < 0 || cftab[ch] >= tt.Length) + { + throw new InvalidFormatException("BZip2: block data out of bounds"); + } tt[cftab[ch]] = i; cftab[ch]++; } + if (origPtr < 0 || origPtr >= tt.Length) + { + throw new InvalidFormatException("BZip2: origPtr out of bounds"); + } tPos = tt[origPtr]; count = 0; @@ -1058,7 +1229,7 @@ internal class CBZip2InputStream : Stream { if (!(0 <= newSize100k && newSize100k <= 9 && 0 <= blockSize100k && blockSize100k <= 9)) { - // throw new IOException("Invalid block size"); + // throw new InvalidFormatException("Invalid block size"); } blockSize100k = newSize100k; diff --git a/src/SharpCompress/Compressors/BZip2/CBZip2OutputStream.Async.cs b/src/SharpCompress/Compressors/BZip2/CBZip2OutputStream.Async.cs new file mode 100644 index 00000000..d56e9244 --- /dev/null +++ b/src/SharpCompress/Compressors/BZip2/CBZip2OutputStream.Async.cs @@ -0,0 +1,706 @@ +using System; +using System.Threading; +using System.Threading.Tasks; + +#nullable disable + +namespace SharpCompress.Compressors.BZip2; + +internal sealed partial class CBZip2OutputStream : IAsyncDisposable +{ + private readonly byte[] bsAsyncWriteBuffer = new byte[1]; + + /// + /// Ensures the BZip2 stream header ('B', 'Z', 'h', blocksize) has been written + /// asynchronously before the first compressed byte is written. + /// + private async ValueTask EnsureStreamHeaderWrittenAsync(CancellationToken cancellationToken) + { + if (!_streamHeaderWritten) + { + _streamHeaderWritten = true; + // Write 'B', 'Z', 'h' async, then set up bit buffer as Initialize() would. + // Initialize() calls BsPutUChar('h') then BsPutUChar('0'+N): + // - First call buffers 'h' (bsLive=8) + // - Second call flushes 'h' to stream, then buffers '0'+N (bsLive=8) + // So after Initialize(), stream has 'h' and bit buffer has '0'+N with bsLive=8. + var header = new byte[] { (byte)'B', (byte)'Z', (byte)'h' }; + await bsStream + .WriteAsync(header, 0, header.Length, cancellationToken) + .ConfigureAwait(false); + // Replicate the bit buffer state that Initialize() leaves: + bytesOut = 1; // 'h' was written via BsW (Initialize increments bytesOut via BsW) + nBlocksRandomised = 0; + combinedCRC = 0; + bsBuff = (blockSize100k + '0') << 24; + bsLive = 8; + InitBlock(); + } + } + + public async ValueTask WriteByteAsync(byte bv, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + await EnsureStreamHeaderWrittenAsync(cancellationToken).ConfigureAwait(false); + var b = (256 + bv) % 256; + if (currentChar != -1) + { + if (currentChar == b) + { + runLength++; + if (runLength > 254) + { + await WriteRunAsync(cancellationToken).ConfigureAwait(false); + currentChar = -1; + runLength = 0; + } + } + else + { + await WriteRunAsync(cancellationToken).ConfigureAwait(false); + runLength = 1; + currentChar = b; + } + } + else + { + currentChar = b; + runLength++; + } + } + + private async ValueTask WriteRunAsync(CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + if (last < allowableBlockSize) + { + inUse[currentChar] = true; + for (var i = 0; i < runLength; i++) + { + mCrc.UpdateCRC((char)currentChar); + } + switch (runLength) + { + case 1: + last++; + block[last + 1] = (char)currentChar; + break; + case 2: + last++; + block[last + 1] = (char)currentChar; + last++; + block[last + 1] = (char)currentChar; + break; + case 3: + last++; + block[last + 1] = (char)currentChar; + last++; + block[last + 1] = (char)currentChar; + last++; + block[last + 1] = (char)currentChar; + break; + default: + inUse[runLength - 4] = true; + last++; + block[last + 1] = (char)currentChar; + last++; + block[last + 1] = (char)currentChar; + last++; + block[last + 1] = (char)currentChar; + last++; + block[last + 1] = (char)currentChar; + last++; + block[last + 1] = (char)(runLength - 4); + break; + } + } + else + { + await EndBlockAsync(cancellationToken).ConfigureAwait(false); + InitBlock(); + await WriteRunAsync(cancellationToken).ConfigureAwait(false); + } + } + + private async ValueTask EndBlockAsync(CancellationToken cancellationToken) + { + // Skip block processing for empty input (no data written) + if (last < 0) + { + return; + } + + blockCRC = mCrc.GetFinalCRC(); + combinedCRC = (combinedCRC << 1) | (int)(((uint)combinedCRC) >> 31); + combinedCRC ^= blockCRC; + + /* sort the block and establish posn of original string */ + DoReversibleTransformation(); + + await BsPutUCharAsync(0x31, cancellationToken).ConfigureAwait(false); + await BsPutUCharAsync(0x41, cancellationToken).ConfigureAwait(false); + await BsPutUCharAsync(0x59, cancellationToken).ConfigureAwait(false); + await BsPutUCharAsync(0x26, cancellationToken).ConfigureAwait(false); + await BsPutUCharAsync(0x53, cancellationToken).ConfigureAwait(false); + await BsPutUCharAsync(0x59, cancellationToken).ConfigureAwait(false); + + await BsPutintAsync(blockCRC, cancellationToken).ConfigureAwait(false); + + if (blockRandomised) + { + await BsWAsync(1, 1, cancellationToken).ConfigureAwait(false); + nBlocksRandomised++; + } + else + { + await BsWAsync(1, 0, cancellationToken).ConfigureAwait(false); + } + + await MoveToFrontCodeAndSendAsync(cancellationToken).ConfigureAwait(false); + } + + private async ValueTask EndCompressionAsync(CancellationToken cancellationToken) + { + await BsPutUCharAsync(0x17, cancellationToken).ConfigureAwait(false); + await BsPutUCharAsync(0x72, cancellationToken).ConfigureAwait(false); + await BsPutUCharAsync(0x45, cancellationToken).ConfigureAwait(false); + await BsPutUCharAsync(0x38, cancellationToken).ConfigureAwait(false); + await BsPutUCharAsync(0x50, cancellationToken).ConfigureAwait(false); + await BsPutUCharAsync(0x90, cancellationToken).ConfigureAwait(false); + + await BsPutintAsync(combinedCRC, cancellationToken).ConfigureAwait(false); + + await BsFinishedWithStreamAsync(cancellationToken).ConfigureAwait(false); + } + + private async ValueTask BsFinishedWithStreamAsync(CancellationToken cancellationToken) + { + while (bsLive > 0) + { + var ch = bsBuff >> 24; + bsAsyncWriteBuffer[0] = (byte)ch; + await bsStream + .WriteAsync(bsAsyncWriteBuffer, 0, 1, cancellationToken) + .ConfigureAwait(false); + bsBuff <<= 8; + bsLive -= 8; + bytesOut++; + } + } + + private async ValueTask BsWAsync(int n, int v, CancellationToken cancellationToken) + { + while (bsLive >= 8) + { + var ch = bsBuff >> 24; + bsAsyncWriteBuffer[0] = (byte)ch; + await bsStream + .WriteAsync(bsAsyncWriteBuffer, 0, 1, cancellationToken) + .ConfigureAwait(false); + bsBuff <<= 8; + bsLive -= 8; + bytesOut++; + } + bsBuff |= v << (32 - bsLive - n); + bsLive += n; + } + + private ValueTask BsPutUCharAsync(int c, CancellationToken cancellationToken) => + BsWAsync(8, c, cancellationToken); + + private async ValueTask BsPutintAsync(int u, CancellationToken cancellationToken) + { + await BsWAsync(8, (u >> 24) & 0xff, cancellationToken).ConfigureAwait(false); + await BsWAsync(8, (u >> 16) & 0xff, cancellationToken).ConfigureAwait(false); + await BsWAsync(8, (u >> 8) & 0xff, cancellationToken).ConfigureAwait(false); + await BsWAsync(8, u & 0xff, cancellationToken).ConfigureAwait(false); + } + + private ValueTask BsPutIntVSAsync(int numBits, int c, CancellationToken cancellationToken) => + BsWAsync(numBits, c, cancellationToken); + + private async ValueTask SendMTFValuesAsync(CancellationToken cancellationToken) + { + var len = CBZip2InputStream.InitCharArray( + BZip2Constants.N_GROUPS, + BZip2Constants.MAX_ALPHA_SIZE + ); + + int v, + t, + i, + j, + gs, + ge, + totc, + bt, + bc, + iter; + int nSelectors = 0, + alphaSize, + minLen, + maxLen, + selCtr; + int nGroups; //, nBytes; + + alphaSize = nInUse + 2; + for (t = 0; t < BZip2Constants.N_GROUPS; t++) + { + for (v = 0; v < alphaSize; v++) + { + len[t][v] = (char)GREATER_ICOST; + } + } + + if (nMTF <= 0) + { + Panic(); + } + + if (nMTF < 200) + { + nGroups = 2; + } + else if (nMTF < 600) + { + nGroups = 3; + } + else if (nMTF < 1200) + { + nGroups = 4; + } + else if (nMTF < 2400) + { + nGroups = 5; + } + else + { + nGroups = 6; + } + + { + int nPart, + remF, + tFreq, + aFreq; + + nPart = nGroups; + remF = nMTF; + gs = 0; + while (nPart > 0) + { + tFreq = remF / nPart; + ge = gs - 1; + aFreq = 0; + while (aFreq < tFreq && ge < alphaSize - 1) + { + ge++; + aFreq += mtfFreq[ge]; + } + + if (ge > gs && nPart != nGroups && nPart != 1 && ((nGroups - nPart) % 2 == 1)) + { + aFreq -= mtfFreq[ge]; + ge--; + } + + for (v = 0; v < alphaSize; v++) + { + if (v >= gs && v <= ge) + { + len[nPart - 1][v] = (char)LESSER_ICOST; + } + else + { + len[nPart - 1][v] = (char)GREATER_ICOST; + } + } + + nPart--; + gs = ge + 1; + remF -= aFreq; + } + } + + var rfreq = CBZip2InputStream.InitIntArray( + BZip2Constants.N_GROUPS, + BZip2Constants.MAX_ALPHA_SIZE + ); + var fave = new int[BZip2Constants.N_GROUPS]; + var cost = new short[BZip2Constants.N_GROUPS]; + for (iter = 0; iter < BZip2Constants.N_ITERS; iter++) + { + for (t = 0; t < nGroups; t++) + { + fave[t] = 0; + } + + for (t = 0; t < nGroups; t++) + { + for (v = 0; v < alphaSize; v++) + { + rfreq[t][v] = 0; + } + } + + nSelectors = 0; + totc = 0; + gs = 0; + while (true) + { + if (gs >= nMTF) + { + break; + } + ge = gs + BZip2Constants.G_SIZE - 1; + if (ge >= nMTF) + { + ge = nMTF - 1; + } + + for (t = 0; t < nGroups; t++) + { + cost[t] = 0; + } + + if (nGroups == 6) + { + short cost0, + cost1, + cost2, + cost3, + cost4, + cost5; + cost0 = cost1 = cost2 = cost3 = cost4 = cost5 = 0; + for (i = gs; i <= ge; i++) + { + var icv = szptr[i]; + cost0 += (short)len[0][icv]; + cost1 += (short)len[1][icv]; + cost2 += (short)len[2][icv]; + cost3 += (short)len[3][icv]; + cost4 += (short)len[4][icv]; + cost5 += (short)len[5][icv]; + } + cost[0] = cost0; + cost[1] = cost1; + cost[2] = cost2; + cost[3] = cost3; + cost[4] = cost4; + cost[5] = cost5; + } + else + { + for (i = gs; i <= ge; i++) + { + var icv = szptr[i]; + for (t = 0; t < nGroups; t++) + { + cost[t] += (short)len[t][icv]; + } + } + } + + bc = 999999999; + bt = -1; + for (t = 0; t < nGroups; t++) + { + if (cost[t] < bc) + { + bc = cost[t]; + bt = t; + } + } + ; + totc += bc; + fave[bt]++; + selector[nSelectors] = (char)bt; + nSelectors++; + + for (i = gs; i <= ge; i++) + { + rfreq[bt][szptr[i]]++; + } + + gs = ge + 1; + } + + for (t = 0; t < nGroups; t++) + { + HbMakeCodeLengths(len[t], rfreq[t], alphaSize, 20); + } + } + + rfreq = null; + fave = null; + cost = null; + + if (!(nGroups < 8)) + { + Panic(); + } + if (!(nSelectors < 32768 && nSelectors <= (2 + (900000 / BZip2Constants.G_SIZE)))) + { + Panic(); + } + + { + var pos = new char[BZip2Constants.N_GROUPS]; + char ll_i, + tmp2, + tmp; + for (i = 0; i < nGroups; i++) + { + pos[i] = (char)i; + } + for (i = 0; i < nSelectors; i++) + { + ll_i = selector[i]; + j = 0; + tmp = pos[j]; + while (ll_i != tmp) + { + j++; + tmp2 = tmp; + tmp = pos[j]; + pos[j] = tmp2; + } + pos[0] = tmp; + selectorMtf[i] = (char)j; + } + } + + var code = CBZip2InputStream.InitIntArray( + BZip2Constants.N_GROUPS, + BZip2Constants.MAX_ALPHA_SIZE + ); + + for (t = 0; t < nGroups; t++) + { + minLen = 32; + maxLen = 0; + for (i = 0; i < alphaSize; i++) + { + if (len[t][i] > maxLen) + { + maxLen = len[t][i]; + } + if (len[t][i] < minLen) + { + minLen = len[t][i]; + } + } + if (maxLen > 20) + { + Panic(); + } + if (minLen < 1) + { + Panic(); + } + HbAssignCodes(code[t], len[t], minLen, maxLen, alphaSize); + } + + { + var inUse16 = new bool[16]; + for (i = 0; i < 16; i++) + { + inUse16[i] = false; + for (j = 0; j < 16; j++) + { + if (inUse[(i * 16) + j]) + { + inUse16[i] = true; + } + } + } + + for (i = 0; i < 16; i++) + { + if (inUse16[i]) + { + await BsWAsync(1, 1, cancellationToken).ConfigureAwait(false); + } + else + { + await BsWAsync(1, 0, cancellationToken).ConfigureAwait(false); + } + } + + for (i = 0; i < 16; i++) + { + if (inUse16[i]) + { + for (j = 0; j < 16; j++) + { + if (inUse[(i * 16) + j]) + { + await BsWAsync(1, 1, cancellationToken).ConfigureAwait(false); + } + else + { + await BsWAsync(1, 0, cancellationToken).ConfigureAwait(false); + } + } + } + } + } + + await BsWAsync(3, nGroups, cancellationToken).ConfigureAwait(false); + await BsWAsync(15, nSelectors, cancellationToken).ConfigureAwait(false); + for (i = 0; i < nSelectors; i++) + { + for (j = 0; j < selectorMtf[i]; j++) + { + await BsWAsync(1, 1, cancellationToken).ConfigureAwait(false); + } + await BsWAsync(1, 0, cancellationToken).ConfigureAwait(false); + } + + for (t = 0; t < nGroups; t++) + { + int curr = len[t][0]; + await BsWAsync(5, curr, cancellationToken).ConfigureAwait(false); + for (i = 0; i < alphaSize; i++) + { + while (curr < len[t][i]) + { + await BsWAsync(2, 2, cancellationToken).ConfigureAwait(false); + curr++; + } + while (curr > len[t][i]) + { + await BsWAsync(2, 3, cancellationToken).ConfigureAwait(false); + curr--; + } + await BsWAsync(1, 0, cancellationToken).ConfigureAwait(false); + } + } + + selCtr = 0; + gs = 0; + while (true) + { + if (gs >= nMTF) + { + break; + } + ge = gs + BZip2Constants.G_SIZE - 1; + if (ge >= nMTF) + { + ge = nMTF - 1; + } + for (i = gs; i <= ge; i++) + { + await BsWAsync( + len[selector[selCtr]][szptr[i]], + code[selector[selCtr]][szptr[i]], + cancellationToken + ) + .ConfigureAwait(false); + } + + gs = ge + 1; + selCtr++; + } + if (!(selCtr == nSelectors)) + { + Panic(); + } + } + + private async ValueTask MoveToFrontCodeAndSendAsync(CancellationToken cancellationToken) + { + await BsPutIntVSAsync(24, origPtr, cancellationToken).ConfigureAwait(false); + GenerateMTFValues(); + await SendMTFValuesAsync(cancellationToken).ConfigureAwait(false); + } + + public override async Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken = default + ) + { + await EnsureStreamHeaderWrittenAsync(cancellationToken).ConfigureAwait(false); + for (var k = 0; k < count; ++k) + { + await WriteByteAsync(buffer[k + offset], cancellationToken).ConfigureAwait(false); + } + } + +#if !LEGACY_DOTNET + public override async ValueTask WriteAsync( + ReadOnlyMemory buffer, + CancellationToken cancellationToken = default + ) + { + await EnsureStreamHeaderWrittenAsync(cancellationToken).ConfigureAwait(false); + for (var k = 0; k < buffer.Length; ++k) + { + cancellationToken.ThrowIfCancellationRequested(); + var value = buffer.Span[k]; + await WriteByteAsync(value, cancellationToken).ConfigureAwait(false); + } + } +#endif + + /// + /// Asynchronously finalizes the BZip2 compressed stream, flushing all pending data. + /// Writes the remaining compressed data to the underlying stream using async I/O. + /// + public async ValueTask FinishAsync(CancellationToken cancellationToken = default) + { + if (finished) + { + return; + } + + await EnsureStreamHeaderWrittenAsync(cancellationToken).ConfigureAwait(false); + + if (runLength > 0) + { + await WriteRunAsync(cancellationToken).ConfigureAwait(false); + } + currentChar = -1; + await EndBlockAsync(cancellationToken).ConfigureAwait(false); + await EndCompressionAsync(cancellationToken).ConfigureAwait(false); + finished = true; + await bsStream.FlushAsync(cancellationToken).ConfigureAwait(false); + } + +#if !LEGACY_DOTNET || NETSTANDARD2_1 + public override async ValueTask DisposeAsync() +#else + public async ValueTask DisposeAsync() +#endif + { + if (disposed) + { + return; + } + + await FinishAsync().ConfigureAwait(false); + disposed = true; + if (!leaveOpen && bsStream is not null) + { + if (bsStream is IAsyncDisposable asyncDisposable) + { + await asyncDisposable.DisposeAsync().ConfigureAwait(false); + } + else + { + bsStream.Dispose(); + } + } + bsStream = null; + +#if !LEGACY_DOTNET || NETSTANDARD2_1 + await base.DisposeAsync().ConfigureAwait(false); +#else + await Task.CompletedTask.ConfigureAwait(false); +#endif + GC.SuppressFinalize(this); + } +} diff --git a/src/SharpCompress/Compressors/BZip2/CBZip2OutputStream.cs b/src/SharpCompress/Compressors/BZip2/CBZip2OutputStream.cs index a975ffbd..465909a4 100644 --- a/src/SharpCompress/Compressors/BZip2/CBZip2OutputStream.cs +++ b/src/SharpCompress/Compressors/BZip2/CBZip2OutputStream.cs @@ -1,6 +1,5 @@ using System; using System.IO; - /* * Copyright 2001,2004-2005 The Apache Software Foundation * @@ -38,7 +37,7 @@ namespace SharpCompress.Compressors.BZip2; * start of the BZIP2 stream to make it compatible with other PGP programs. */ -internal sealed class CBZip2OutputStream : Stream +internal sealed partial class CBZip2OutputStream : Stream { private const int SETMASK = (1 << 21); private const int CLEARMASK = (~SETMASK); @@ -284,7 +283,7 @@ internal sealed class CBZip2OutputStream : Stream private int bytesOut; private int bsBuff; private int bsLive; - private readonly CRC mCrc = new CRC(); + private readonly CRC mCrc = new(); private readonly bool[] inUse = new bool[256]; private int nInUse; @@ -318,20 +317,19 @@ internal sealed class CBZip2OutputStream : Stream private int currentChar = -1; private int runLength; + private readonly bool leaveOpen; - public CBZip2OutputStream(Stream inStream) - : this(inStream, 9) { } + public CBZip2OutputStream(Stream inStream, bool leaveOpen = false) + : this(inStream, 9, leaveOpen) { } - public CBZip2OutputStream(Stream inStream, int inBlockSize) + public CBZip2OutputStream(Stream inStream, int inBlockSize, bool leaveOpen = false) { + this.leaveOpen = leaveOpen; block = null; quadrant = null; zptr = null; ftab = null; - inStream.WriteByte((byte)'B'); - inStream.WriteByte((byte)'Z'); - BsSetStream(inStream); workFactor = 50; @@ -345,18 +343,37 @@ internal sealed class CBZip2OutputStream : Stream } blockSize100k = inBlockSize; AllocateCompressStructures(); - Initialize(); - InitBlock(); + // Defer Initialize() and InitBlock() to EnsureStreamHeaderWritten/Async: + // they write to the underlying stream which may be async-only. } + private bool _streamHeaderWritten; + /** * * modified by Oliver Merkel, 010128 * */ + /// + /// Ensures the BZip2 stream header ('B', 'Z', 'h', blocksize) has been written + /// synchronously before the first compressed byte is written. + /// + private void EnsureStreamHeaderWritten() + { + if (!_streamHeaderWritten) + { + _streamHeaderWritten = true; + bsStream.WriteByte((byte)'B'); + bsStream.WriteByte((byte)'Z'); + Initialize(); + InitBlock(); + } + } + public override void WriteByte(byte bv) { + EnsureStreamHeaderWritten(); var b = (256 + bv) % 256; if (currentChar != -1) { @@ -444,6 +461,7 @@ internal sealed class CBZip2OutputStream : Stream { if (disposed) { + base.Dispose(disposing); return; } @@ -451,9 +469,13 @@ internal sealed class CBZip2OutputStream : Stream disposed = true; Dispose(); - bsStream?.Dispose(); + if (!leaveOpen) + { + bsStream?.Dispose(); + } bsStream = null; } + base.Dispose(disposing); } public void Finish() @@ -463,6 +485,7 @@ internal sealed class CBZip2OutputStream : Stream return; } + EnsureStreamHeaderWritten(); if (runLength > 0) { WriteRun(); @@ -514,6 +537,12 @@ internal sealed class CBZip2OutputStream : Stream private void EndBlock() { + // Skip block processing for empty input (no data written) + if (last < 0) + { + return; + } + blockCRC = mCrc.GetFinalCRC(); combinedCRC = (combinedCRC << 1) | (int)(((uint)combinedCRC) >> 31); combinedCRC ^= blockCRC; @@ -1829,7 +1858,7 @@ internal sealed class CBZip2OutputStream : Stream 88573, 265720, 797161, - 2391484 + 2391484, }; private void AllocateCompressStructures() @@ -1988,6 +2017,7 @@ internal sealed class CBZip2OutputStream : Stream public override void Write(byte[] buffer, int offset, int count) { + EnsureStreamHeaderWritten(); for (var k = 0; k < count; ++k) { WriteByte(buffer[k + offset]); diff --git a/src/SharpCompress/Compressors/BZip2/CRC.cs b/src/SharpCompress/Compressors/BZip2/CRC.cs index 9f593c40..6e8e034a 100644 --- a/src/SharpCompress/Compressors/BZip2/CRC.cs +++ b/src/SharpCompress/Compressors/BZip2/CRC.cs @@ -288,7 +288,7 @@ internal class CRC unchecked((int)0xbcb4666d), unchecked((int)0xb8757bda), unchecked((int)0xb5365d03), - unchecked((int)0xb1f740b4) + unchecked((int)0xb1f740b4), }; public CRC() => InitialiseCRC(); diff --git a/src/SharpCompress/Compressors/CompressionMode.cs b/src/SharpCompress/Compressors/CompressionMode.cs index 4eba1121..0bf15e03 100644 --- a/src/SharpCompress/Compressors/CompressionMode.cs +++ b/src/SharpCompress/Compressors/CompressionMode.cs @@ -3,5 +3,5 @@ namespace SharpCompress.Compressors; public enum CompressionMode { Compress = 0, - Decompress = 1 + Decompress = 1, } diff --git a/src/SharpCompress/Compressors/Deflate/CRC32.cs b/src/SharpCompress/Compressors/Deflate/CRC32.cs index 892459fb..24eb65a7 100644 --- a/src/SharpCompress/Compressors/Deflate/CRC32.cs +++ b/src/SharpCompress/Compressors/Deflate/CRC32.cs @@ -33,6 +33,7 @@ // ------------------------------------------------------------------ using System; +using System.Buffers; using System.IO; namespace SharpCompress.Compressors.Deflate; @@ -120,22 +121,29 @@ public class CRC32 { //UInt32 crc32Result; //crc32Result = 0xFFFFFFFF; - var buffer = new byte[BUFFER_SIZE]; - var readSize = BUFFER_SIZE; - - TotalBytesRead = 0; - var count = input.Read(buffer, 0, readSize); - output?.Write(buffer, 0, count); - TotalBytesRead += count; - while (count > 0) + var buffer = ArrayPool.Shared.Rent(BUFFER_SIZE); + try { - SlurpBlock(buffer, 0, count); - count = input.Read(buffer, 0, readSize); + var readSize = BUFFER_SIZE; + + TotalBytesRead = 0; + var count = input.Read(buffer, 0, readSize); output?.Write(buffer, 0, count); TotalBytesRead += count; - } + while (count > 0) + { + SlurpBlock(buffer, 0, count); + count = input.Read(buffer, 0, readSize); + output?.Write(buffer, 0, count); + TotalBytesRead += count; + } - return ~runningCrc32Result; + return ~runningCrc32Result; + } + finally + { + ArrayPool.Shared.Return(buffer, clearArray: true); + } } } diff --git a/src/SharpCompress/Compressors/Deflate/DeflateManager.cs b/src/SharpCompress/Compressors/Deflate/DeflateManager.cs index 27ad4018..3b7a7ff9 100644 --- a/src/SharpCompress/Compressors/Deflate/DeflateManager.cs +++ b/src/SharpCompress/Compressors/Deflate/DeflateManager.cs @@ -69,8 +69,9 @@ // ----------------------------------------------------------------------- using System; - +using System.Buffers; using SharpCompress.Algorithms; +using SharpCompress.Common; namespace SharpCompress.Compressors.Deflate; @@ -107,7 +108,7 @@ internal sealed partial class DeflateManager 5, 5, 5, - 0 + 0, }; // extra bits for each distance code @@ -142,7 +143,7 @@ internal sealed partial class DeflateManager 12, 12, 13, - 13 + 13, }; internal enum BlockState @@ -150,14 +151,14 @@ internal sealed partial class DeflateManager NeedMore = 0, // block not completed, need more input or more output BlockDone, // block flush performed FinishStarted, // finish started, need only more output at next deflate - FinishDone // finish done, accept no more input or output + FinishDone, // finish done, accept no more input or output } internal enum DeflateFlavor { Store, Fast, - Slow + Slow, } private const int MEM_LEVEL_MAX = 9; @@ -215,7 +216,7 @@ internal sealed partial class DeflateManager new Config(8, 16, 128, 128, DeflateFlavor.Slow), new Config(8, 32, 128, 256, DeflateFlavor.Slow), new Config(32, 128, 258, 1024, DeflateFlavor.Slow), - new Config(32, 258, 258, 4096, DeflateFlavor.Slow) + new Config(32, 258, 258, 4096, DeflateFlavor.Slow), }; private static readonly Config[] Table; @@ -234,7 +235,7 @@ internal sealed partial class DeflateManager "insufficient memory", "buffer error", "incompatible version", - "" + "", }; // preset dictionary flag in zlib header @@ -343,9 +344,9 @@ internal sealed partial class DeflateManager private readonly short[] dyn_dtree; // distance tree private readonly short[] bl_tree; // Huffman tree for bit lengths - private readonly Tree treeLiterals = new Tree(); // desc for literal tree - private readonly Tree treeDistances = new Tree(); // desc for distance tree - private readonly Tree treeBitLengths = new Tree(); // desc for bit length tree + private readonly Tree treeLiterals = new(); // desc for literal tree + private readonly Tree treeDistances = new(); // desc for distance tree + private readonly Tree treeBitLengths = new(); // desc for bit length tree // number of codes at each bit length for an optimal tree private readonly short[] bl_count = new short[InternalConstants.MAX_BITS + 1]; @@ -1707,7 +1708,11 @@ internal sealed partial class DeflateManager if (memLevel < 1 || memLevel > MEM_LEVEL_MAX) { throw new ZlibException( - string.Format("memLevel must be in the range 1.. {0}", MEM_LEVEL_MAX) + string.Format( + Constants.DefaultCultureInfo, + "memLevel must be in the range 1.. {0}", + MEM_LEVEL_MAX + ) ); } @@ -1722,9 +1727,12 @@ internal sealed partial class DeflateManager hash_mask = hash_size - 1; hash_shift = ((hash_bits + MIN_MATCH - 1) / MIN_MATCH); - window = new byte[w_size * 2]; - prev = new short[w_size]; - head = new short[hash_size]; + window = ArrayPool.Shared.Rent(w_size * 2); + prev = ArrayPool.Shared.Rent(w_size); + head = ArrayPool.Shared.Rent(hash_size); + Array.Clear(window, 0, w_size * 2); + Array.Clear(prev, 0, w_size); + Array.Clear(head, 0, hash_size); // for memLevel==8, this will be 16384, 16k lit_bufsize = 1 << (memLevel + 6); @@ -1733,7 +1741,8 @@ internal sealed partial class DeflateManager // the output distance codes, and the output length codes (aka tree). // orig comment: This works just fine since the average // output size for (length,distance) codes is <= 24 bits. - pending = new byte[lit_bufsize * 4]; + pending = ArrayPool.Shared.Rent(lit_bufsize * 4); + Array.Clear(pending, 0, lit_bufsize * 4); _distanceOffset = lit_bufsize; _lengthOffset = (1 + 2) * lit_bufsize; @@ -1772,38 +1781,57 @@ internal sealed partial class DeflateManager internal int End() { + var result = ZlibConstants.Z_OK; if (status != INIT_STATE && status != BUSY_STATE && status != FINISH_STATE) { - return ZlibConstants.Z_STREAM_ERROR; + result = ZlibConstants.Z_STREAM_ERROR; + } + else if (status == BUSY_STATE) + { + result = ZlibConstants.Z_DATA_ERROR; } // Deallocate in reverse order of allocations: - pending = null; - head = null; - prev = null; - window = null; + ReturnBuffers(); // free // dstate=null; - return status == BUSY_STATE ? ZlibConstants.Z_DATA_ERROR : ZlibConstants.Z_OK; + return result; } - private void SetDeflater() + private void ReturnBuffers() { - switch (config.Flavor) + if (pending is not null) { - case DeflateFlavor.Store: - DeflateFunction = DeflateNone; - break; - case DeflateFlavor.Fast: - DeflateFunction = DeflateFast; - break; - case DeflateFlavor.Slow: - DeflateFunction = DeflateSlow; - break; + ArrayPool.Shared.Return(pending, clearArray: true); + pending = null; + } + if (head is not null) + { + ArrayPool.Shared.Return(head, clearArray: true); + head = null; + } + if (prev is not null) + { + ArrayPool.Shared.Return(prev, clearArray: true); + prev = null; + } + if (window is not null) + { + ArrayPool.Shared.Return(window, clearArray: true); + window = null; } } + private void SetDeflater() => + DeflateFunction = config.Flavor switch + { + DeflateFlavor.Store => DeflateNone, + DeflateFlavor.Fast => DeflateFast, + DeflateFlavor.Slow => DeflateSlow, + _ => DeflateFunction, + }; + internal int SetParams(CompressionLevel level, CompressionStrategy strategy) { var result = ZlibConstants.Z_OK; @@ -1884,7 +1912,13 @@ internal sealed partial class DeflateManager _codec.Message = _ErrorMessage[ ZlibConstants.Z_NEED_DICT - (ZlibConstants.Z_STREAM_ERROR) ]; - throw new ZlibException(string.Format("Something is fishy. [{0}]", _codec.Message)); + throw new ZlibException( + string.Format( + Constants.DefaultCultureInfo, + "Something is fishy. [{0}]", + _codec.Message + ) + ); //return ZlibConstants.Z_STREAM_ERROR; } @@ -1959,7 +1993,9 @@ internal sealed partial class DeflateManager // returning Z_STREAM_END instead of Z_BUFF_ERROR. } else if ( - _codec.AvailableBytesIn == 0 && (int)flush <= old_flush && flush != FlushType.Finish + _codec.AvailableBytesIn == 0 + && (int)flush <= old_flush + && flush != FlushType.Finish ) { // workitem 8557 diff --git a/src/SharpCompress/Compressors/Deflate/DeflateStream.Async.cs b/src/SharpCompress/Compressors/Deflate/DeflateStream.Async.cs new file mode 100644 index 00000000..454ba022 --- /dev/null +++ b/src/SharpCompress/Compressors/Deflate/DeflateStream.Async.cs @@ -0,0 +1,100 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.IO; + +namespace SharpCompress.Compressors.Deflate; + +public partial class DeflateStream +{ +#if !LEGACY_DOTNET || NETSTANDARD2_1 + public override async ValueTask DisposeAsync() +#else + public async ValueTask DisposeAsync() +#endif + { + if (!_disposed) + { + if (!_leaveOpen) + { + await _baseStream.DisposeAsync().ConfigureAwait(false); + } + _disposed = true; + } +#if !LEGACY_DOTNET || NETSTANDARD2_1 + await base.DisposeAsync().ConfigureAwait(false); +#else + await Task.CompletedTask.ConfigureAwait(false); +#endif + } + + public override async Task FlushAsync(CancellationToken cancellationToken) + { + if (_disposed) + { + throw new ObjectDisposedException("DeflateStream"); + } + await _baseStream.FlushAsync(cancellationToken).ConfigureAwait(false); + } + + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + if (_disposed) + { + throw new ObjectDisposedException("DeflateStream"); + } + return await _baseStream + .ReadAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + if (_disposed) + { + throw new ObjectDisposedException("DeflateStream"); + } + return await _baseStream.ReadAsync(buffer, cancellationToken).ConfigureAwait(false); + } +#endif + + public override async Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + if (_disposed) + { + throw new ObjectDisposedException("DeflateStream"); + } + await _baseStream + .WriteAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + } + +#if !LEGACY_DOTNET + public override async ValueTask WriteAsync( + ReadOnlyMemory buffer, + CancellationToken cancellationToken = default + ) + { + if (_disposed) + { + throw new ObjectDisposedException("DeflateStream"); + } + await _baseStream.WriteAsync(buffer, cancellationToken).ConfigureAwait(false); + } +#endif +} diff --git a/src/SharpCompress/Compressors/Deflate/DeflateStream.cs b/src/SharpCompress/Compressors/Deflate/DeflateStream.cs index 05003d7f..13f3b356 100644 --- a/src/SharpCompress/Compressors/Deflate/DeflateStream.cs +++ b/src/SharpCompress/Compressors/Deflate/DeflateStream.cs @@ -27,27 +27,48 @@ using System; using System.IO; using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.IO; namespace SharpCompress.Compressors.Deflate; -public class DeflateStream : Stream +public partial class DeflateStream : Stream, IStreamStack +#if LEGACY_DOTNET + , IAsyncDisposable +#endif { private readonly ZlibBaseStream _baseStream; private bool _disposed; + private readonly bool _leaveOpen; public DeflateStream( Stream stream, CompressionMode mode, CompressionLevel level = CompressionLevel.Default, Encoding? forceEncoding = null - ) => + ) + : this(stream, mode, level, leaveOpen: false, forceEncoding) { } + + public DeflateStream( + Stream stream, + CompressionMode mode, + CompressionLevel level, + bool leaveOpen, + Encoding? forceEncoding = null + ) + { + _leaveOpen = leaveOpen; _baseStream = new ZlibBaseStream( stream, mode, level, ZlibStreamFlavor.DEFLATE, + leaveOpen, forceEncoding ); + } #region Zlib properties @@ -103,6 +124,7 @@ public class DeflateStream : Stream { throw new ZlibException( string.Format( + Constants.DefaultCultureInfo, "Don't be silly. {0} bytes?? Use a bigger buffer, at least {1}.", value, ZlibConstants.WorkingBufferSizeMin @@ -233,7 +255,7 @@ public class DeflateStream : Stream { if (!_disposed) { - if (disposing) + if (disposing && !_leaveOpen) { _baseStream?.Dispose(); } @@ -246,6 +268,8 @@ public class DeflateStream : Stream } } + Stream IStreamStack.BaseStream() => _baseStream; + /// /// Flush the stream. /// @@ -290,6 +314,7 @@ public class DeflateStream : Stream { throw new ObjectDisposedException("DeflateStream"); } + return _baseStream.Read(buffer, offset, count); } @@ -366,9 +391,5 @@ public class DeflateStream : Stream #endregion public MemoryStream InputBuffer => - new MemoryStream( - _baseStream._z.InputBuffer, - _baseStream._z.NextIn, - _baseStream._z.AvailableBytesIn - ); + new(_baseStream._z.InputBuffer, _baseStream._z.NextIn, _baseStream._z.AvailableBytesIn); } diff --git a/src/SharpCompress/Compressors/Deflate/FlushType.cs b/src/SharpCompress/Compressors/Deflate/FlushType.cs index bcf7fe10..446962a3 100644 --- a/src/SharpCompress/Compressors/Deflate/FlushType.cs +++ b/src/SharpCompress/Compressors/Deflate/FlushType.cs @@ -39,5 +39,5 @@ public enum FlushType Full, /// Signals the end of the compression/decompression stream. - Finish + Finish, } diff --git a/src/SharpCompress/Compressors/Deflate/GZipStream.Async.cs b/src/SharpCompress/Compressors/Deflate/GZipStream.Async.cs new file mode 100644 index 00000000..dc2f823c --- /dev/null +++ b/src/SharpCompress/Compressors/Deflate/GZipStream.Async.cs @@ -0,0 +1,144 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.IO; + +namespace SharpCompress.Compressors.Deflate; + +public partial class GZipStream +{ + public override async Task FlushAsync(CancellationToken cancellationToken) + { + if (_disposed) + { + throw new ObjectDisposedException("GZipStream"); + } + await BaseStream.FlushAsync(cancellationToken).ConfigureAwait(false); + } + + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + if (_disposed) + { + throw new ObjectDisposedException("GZipStream"); + } + var n = await BaseStream + .ReadAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + + if (!_firstReadDone) + { + _firstReadDone = true; + FileName = BaseStream._GzipFileName; + Comment = BaseStream._GzipComment; + LastModified = BaseStream._GzipMtime; + } + return n; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + if (_disposed) + { + throw new ObjectDisposedException("GZipStream"); + } + var n = await BaseStream.ReadAsync(buffer, cancellationToken).ConfigureAwait(false); + + if (!_firstReadDone) + { + _firstReadDone = true; + FileName = BaseStream._GzipFileName; + Comment = BaseStream._GzipComment; + LastModified = BaseStream._GzipMtime; + } + return n; + } +#endif + + public override async Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + if (_disposed) + { + throw new ObjectDisposedException("GZipStream"); + } + if (BaseStream._streamMode == ZlibBaseStream.StreamMode.Undefined) + { + if (BaseStream._wantCompress) + { + // first write in compression, therefore, emit the GZIP header + _headerByteCount = await EmitHeaderAsync(cancellationToken).ConfigureAwait(false); + } + else + { + throw new ArchiveOperationException(); + } + } + + await BaseStream.WriteAsync(buffer, offset, count, cancellationToken).ConfigureAwait(false); + } + +#if !LEGACY_DOTNET + public override async ValueTask WriteAsync( + ReadOnlyMemory buffer, + CancellationToken cancellationToken = default + ) + { + if (_disposed) + { + throw new ObjectDisposedException("GZipStream"); + } + if (BaseStream._streamMode == ZlibBaseStream.StreamMode.Undefined) + { + if (BaseStream._wantCompress) + { + // first write in compression, therefore, emit the GZIP header + _headerByteCount = await EmitHeaderAsync(cancellationToken).ConfigureAwait(false); + } + else + { + throw new ArchiveOperationException(); + } + } + + await BaseStream.WriteAsync(buffer, cancellationToken).ConfigureAwait(false); + } +#endif + +#if !LEGACY_DOTNET || NETSTANDARD2_1 + public override async ValueTask DisposeAsync() +#else + public async ValueTask DisposeAsync() +#endif + { + if (_disposed) + { + return; + } + + _disposed = true; + if (BaseStream != null) + { + await BaseStream.DisposeAsync().ConfigureAwait(false); + } + +#if !LEGACY_DOTNET || NETSTANDARD2_1 + await base.DisposeAsync().ConfigureAwait(false); +#endif + } +} diff --git a/src/SharpCompress/Compressors/Deflate/GZipStream.cs b/src/SharpCompress/Compressors/Deflate/GZipStream.cs index c547df14..b0a690d8 100644 --- a/src/SharpCompress/Compressors/Deflate/GZipStream.cs +++ b/src/SharpCompress/Compressors/Deflate/GZipStream.cs @@ -30,20 +30,19 @@ using System; using System.Buffers.Binary; using System.IO; using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Options; namespace SharpCompress.Compressors.Deflate; -public class GZipStream : Stream +public partial class GZipStream : Stream +#if LEGACY_DOTNET + , IAsyncDisposable +#endif { - internal static readonly DateTime UNIX_EPOCH = new DateTime( - 1970, - 1, - 1, - 0, - 0, - 0, - DateTimeKind.Utc - ); + internal static readonly DateTime UNIX_EPOCH = new(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc); private string? _comment; private string? _fileName; @@ -59,17 +58,41 @@ public class GZipStream : Stream public GZipStream(Stream stream, CompressionMode mode) : this(stream, mode, CompressionLevel.Default, Encoding.UTF8) { } - public GZipStream(Stream stream, CompressionMode mode, CompressionLevel level) - : this(stream, mode, level, Encoding.UTF8) { } + public GZipStream(Stream stream, CompressionMode mode, IReaderOptions readerOptions) + : this(stream, mode, CompressionLevel.Default, readerOptions) { } public GZipStream( Stream stream, CompressionMode mode, CompressionLevel level, - Encoding encoding + IReaderOptions readerOptions + ) + : this( + stream, + mode, + level, + ( + readerOptions ?? throw new ArgumentNullException(nameof(readerOptions)) + ).ArchiveEncoding.GetEncoding(), + readerOptions.LeaveStreamOpen + ) { } + + public GZipStream( + Stream stream, + CompressionMode mode, + CompressionLevel level, + Encoding encoding, + bool leaveOpen = false ) { - BaseStream = new ZlibBaseStream(stream, mode, level, ZlibStreamFlavor.GZIP, encoding); + BaseStream = new ZlibBaseStream( + stream, + mode, + level, + ZlibStreamFlavor.GZIP, + leaveOpen, + encoding + ); _encoding = encoding; } @@ -105,6 +128,7 @@ public class GZipStream : Stream { throw new ZlibException( string.Format( + Constants.DefaultCultureInfo, "Don't be silly. {0} bytes?? Use a bigger buffer, at least {1}.", value, ZlibConstants.WorkingBufferSizeMin @@ -342,7 +366,7 @@ public class GZipStream : Stream } else { - throw new InvalidOperationException(); + throw new ArchiveOperationException(); } } @@ -391,16 +415,24 @@ public class GZipStream : Stream { return; } +#if LEGACY_DOTNET if (_fileName.Contains('/')) +#else + if (_fileName.Contains('/', StringComparison.Ordinal)) +#endif { _fileName = _fileName.Replace('/', '\\'); } if (_fileName.EndsWith('\\')) { - throw new InvalidOperationException("Illegal filename"); + throw new ArchiveOperationException("Illegal filename"); } +#if LEGACY_DOTNET if (_fileName.Contains('\\')) +#else + if (_fileName.Contains('\\', StringComparison.Ordinal)) +#endif { // trim any leading path _fileName = Path.GetFileName(_fileName); @@ -410,7 +442,7 @@ public class GZipStream : Stream public int Crc32 { get; private set; } - private int EmitHeader() + private byte[] BuildHeader() { var commentBytes = (Comment is null) ? null : _encoding.GetBytes(Comment); var filenameBytes = (FileName is null) ? null : _encoding.GetBytes(FileName); @@ -474,8 +506,22 @@ public class GZipStream : Stream header[i++] = 0; // terminate } - BaseStream._stream.Write(header, 0, header.Length); + return header; + } + private int EmitHeader() + { + var header = BuildHeader(); + BaseStream._stream.Write(header, 0, header.Length); + return header.Length; // bytes written + } + + private async ValueTask EmitHeaderAsync(CancellationToken cancellationToken) + { + var header = BuildHeader(); + await BaseStream + ._stream.WriteAsync(header, 0, header.Length, cancellationToken) + .ConfigureAwait(false); return header.Length; // bytes written } } diff --git a/src/SharpCompress/Compressors/Deflate/InfTree.cs b/src/SharpCompress/Compressors/Deflate/InfTree.cs index 74f4069d..2ea4cbc4 100644 --- a/src/SharpCompress/Compressors/Deflate/InfTree.cs +++ b/src/SharpCompress/Compressors/Deflate/InfTree.cs @@ -1615,7 +1615,7 @@ internal sealed class InfTree 79, 0, 9, - 255 + 255, }; //UPGRADE_NOTE: Final was removed from the declaration of 'fixed_td'. "ms-help://MS.VSCC.v80/dv_commoner/local/redirect.htm?index='!DefaultContextWindowIndex'&keyword='jlca1003'" @@ -1716,7 +1716,7 @@ internal sealed class InfTree 193, 192, 5, - 24577 + 24577, }; // Tables for deflate from PKZIP's appnote.txt. @@ -1753,7 +1753,7 @@ internal sealed class InfTree 227, 258, 0, - 0 + 0, }; // see note #13 above about 258 @@ -1790,7 +1790,7 @@ internal sealed class InfTree 5, 0, 112, - 112 + 112, }; //UPGRADE_NOTE: Final was removed from the declaration of 'cpdist'. "ms-help://MS.VSCC.v80/dv_commoner/local/redirect.htm?index='!DefaultContextWindowIndex'&keyword='jlca1003'" @@ -1825,7 +1825,7 @@ internal sealed class InfTree 8193, 12289, 16385, - 24577 + 24577, }; //UPGRADE_NOTE: Final was removed from the declaration of 'cpdext'. "ms-help://MS.VSCC.v80/dv_commoner/local/redirect.htm?index='!DefaultContextWindowIndex'&keyword='jlca1003'" @@ -1860,7 +1860,7 @@ internal sealed class InfTree 12, 12, 13, - 13 + 13, }; // If BMAX needs to be larger than 16, then h and x[] should be uLong. diff --git a/src/SharpCompress/Compressors/Deflate/Inflate.cs b/src/SharpCompress/Compressors/Deflate/Inflate.cs index ebea2805..7c972ef7 100644 --- a/src/SharpCompress/Compressors/Deflate/Inflate.cs +++ b/src/SharpCompress/Compressors/Deflate/Inflate.cs @@ -64,8 +64,9 @@ // ----------------------------------------------------------------------- using System; - +using System.Buffers; using SharpCompress.Algorithms; +using SharpCompress.Common; namespace SharpCompress.Compressors.Deflate; @@ -94,7 +95,7 @@ internal sealed class InflateBlocks 2, 14, 1, - 15 + 15, }; internal ZlibCodec _codec; // pointer back to this zlib stream @@ -106,25 +107,26 @@ internal sealed class InflateBlocks internal int[] blens; // bit lengths of codes internal uint check; // check on output internal object checkfn; // check function - internal InflateCodes codes = new InflateCodes(); // if CODES, current state + internal InflateCodes codes = new(); // if CODES, current state internal int end; // one byte after sliding window internal int[] hufts; // single malloc for tree space internal int index; // index into blens (or border) - internal InfTree inftree = new InfTree(); + internal InfTree inftree = new(); internal int last; // true if this block is the last block internal int left; // if STORED, bytes left to copy private InflateBlockMode mode; // current inflate_block mode internal int readAt; // window read pointer internal int table; // table lengths (14 bits) internal int[] tb = new int[1]; // bit length decoding tree - internal byte[] window; // sliding window + internal IMemoryOwner window; // sliding window internal int writeAt; // window write pointer internal InflateBlocks(ZlibCodec codec, object checkfn, int w) { _codec = codec; - hufts = new int[MANY * 3]; - window = new byte[w]; + hufts = ArrayPool.Shared.Rent(MANY * 3); + Array.Clear(hufts, 0, MANY * 3); + window = MemoryPool.Shared.Rent(w); end = w; this.checkfn = checkfn; mode = InflateBlockMode.TYPE; @@ -341,7 +343,7 @@ internal sealed class InflateBlocks { t = m; } - Array.Copy(_codec.InputBuffer, p, window, q, t); + _codec.InputBuffer.AsSpan(p, t).CopyTo(window.Memory.Span.Slice(q)); p += t; n -= t; q += t; @@ -716,13 +718,18 @@ internal sealed class InflateBlocks internal void Free() { Reset(); + window?.Dispose(); window = null; - hufts = null; + if (hufts is not null) + { + ArrayPool.Shared.Return(hufts, clearArray: true); + hufts = null; + } } internal void SetDictionary(byte[] d, int start, int n) { - Array.Copy(d, start, window, 0, n); + d.AsSpan(start, n).CopyTo(window.Memory.Span.Slice(0, n)); readAt = writeAt = n; } @@ -775,11 +782,16 @@ internal sealed class InflateBlocks // update check information if (checkfn != null) { - _codec._adler32 = check = Adler32.Calculate(check, window.AsSpan(readAt, nBytes)); + _codec._adler32 = check = Adler32.Calculate( + check, + window.Memory.Span.Slice(readAt, nBytes) + ); } // copy as far as end of window - Array.Copy(window, readAt, _codec.OutputBuffer, _codec.NextOut, nBytes); + window + .Memory.Span.Slice(readAt, nBytes) + .CopyTo(_codec.OutputBuffer.AsSpan(_codec.NextOut)); _codec.NextOut += nBytes; readAt += nBytes; @@ -816,7 +828,7 @@ internal sealed class InflateBlocks CODES = 6, // processing fixed or dynamic block DRY = 7, // output remaining window bytes DONE = 8, // finished last block, done - BAD = 9 // ot a data error--stuck here + BAD = 9, // ot a data error--stuck here } #endregion @@ -843,7 +855,7 @@ internal static class InternalInflateConstants 0x00001fff, 0x00003fff, 0x00007fff, - 0x0000ffff + 0x0000ffff, }; } @@ -1214,7 +1226,7 @@ internal sealed class InflateCodes } } - blocks.window[q++] = blocks.window[f++]; + blocks.window.Memory.Span[q++] = blocks.window.Memory.Span[f++]; m--; if (f == blocks.end) @@ -1260,7 +1272,7 @@ internal sealed class InflateCodes } r = ZlibConstants.Z_OK; - blocks.window[q++] = (byte)lit; + blocks.window.Memory.Span[q++] = (byte)lit; m--; mode = START; @@ -1397,7 +1409,7 @@ internal sealed class InflateCodes b >>= (tp[tp_index_t_3 + 1]); k -= (tp[tp_index_t_3 + 1]); - s.window[q++] = (byte)tp[tp_index_t_3 + 2]; + s.window.Memory.Span[q++] = (byte)tp[tp_index_t_3 + 2]; m--; continue; } @@ -1462,13 +1474,14 @@ internal sealed class InflateCodes r = q - d; if (q - r > 0 && 2 > (q - r)) { - s.window[q++] = s.window[r++]; // minimum count is three, - s.window[q++] = s.window[r++]; // so unroll loop a little + s.window.Memory.Span[q++] = s.window.Memory.Span[r++]; // minimum count is three, + s.window.Memory.Span[q++] = s.window.Memory.Span[r++]; // so unroll loop a little c -= 2; } else { - Array.Copy(s.window, r, s.window, q, 2); + s.window.Memory.Span.Slice(r, 2) + .CopyTo(s.window.Memory.Span.Slice(q)); q += 2; r += 2; c -= 2; @@ -1491,12 +1504,13 @@ internal sealed class InflateCodes { do { - s.window[q++] = s.window[r++]; + s.window.Memory.Span[q++] = s.window.Memory.Span[r++]; } while (--e != 0); } else { - Array.Copy(s.window, r, s.window, q, e); + s.window.Memory.Span.Slice(r, e) + .CopyTo(s.window.Memory.Span.Slice(q)); q += e; r += e; e = 0; @@ -1510,12 +1524,13 @@ internal sealed class InflateCodes { do { - s.window[q++] = s.window[r++]; + s.window.Memory.Span[q++] = s.window.Memory.Span[r++]; } while (--c != 0); } else { - Array.Copy(s.window, r, s.window, q, c); + s.window.Memory.Span.Slice(r, c) + .CopyTo(s.window.Memory.Span.Slice(q)); q += c; r += c; c = 0; @@ -1561,7 +1576,7 @@ internal sealed class InflateCodes { b >>= (tp[tp_index_t_3 + 1]); k -= (tp[tp_index_t_3 + 1]); - s.window[q++] = (byte)tp[tp_index_t_3 + 2]; + s.window.Memory.Span[q++] = (byte)tp[tp_index_t_3 + 2]; m--; break; } @@ -1737,6 +1752,7 @@ internal sealed class InflateManager { mode = InflateManagerMode.BAD; _codec.Message = string.Format( + Constants.DefaultCultureInfo, "unknown compression method (0x{0:X2})", method ); @@ -1747,6 +1763,7 @@ internal sealed class InflateManager { mode = InflateManagerMode.BAD; _codec.Message = string.Format( + Constants.DefaultCultureInfo, "invalid window size ({0})", (method >> 4) + 8 ); @@ -1936,7 +1953,13 @@ internal sealed class InflateManager return ZlibConstants.Z_STREAM_END; case InflateManagerMode.BAD: - throw new ZlibException(string.Format("Bad state ({0})", _codec.Message)); + throw new ZlibException( + string.Format( + Constants.DefaultCultureInfo, + "Bad state ({0})", + _codec.Message + ) + ); default: throw new ZlibException("Stream error."); @@ -2055,7 +2078,7 @@ internal sealed class InflateManager CHECK2 = 10, // two check bytes to go CHECK1 = 11, // one check byte to go DONE = 12, // finished check, done - BAD = 13 // got an error--stay here + BAD = 13, // got an error--stay here } #endregion diff --git a/src/SharpCompress/Compressors/Deflate/Tree.cs b/src/SharpCompress/Compressors/Deflate/Tree.cs index ec1eb51f..e5c9b52a 100644 --- a/src/SharpCompress/Compressors/Deflate/Tree.cs +++ b/src/SharpCompress/Compressors/Deflate/Tree.cs @@ -95,7 +95,7 @@ internal sealed partial class DeflateManager 2, 14, 1, - 15 + 15, }; // The lengths of the bit length codes are sent in order of decreasing @@ -618,7 +618,7 @@ internal sealed partial class DeflateManager 29, 29, 29, - 29 + 29, }; internal static readonly sbyte[] LengthCode = @@ -878,7 +878,7 @@ internal sealed partial class DeflateManager 27, 27, 27, - 28 + 28, }; internal static readonly int[] LengthBase = @@ -911,7 +911,7 @@ internal sealed partial class DeflateManager 160, 192, 224, - 0 + 0, }; internal static readonly int[] DistanceBase = @@ -945,7 +945,7 @@ internal sealed partial class DeflateManager 8192, 12288, 16384, - 24576 + 24576, }; internal short[] dyn_tree; // the dynamic tree diff --git a/src/SharpCompress/Compressors/Deflate/Zlib.cs b/src/SharpCompress/Compressors/Deflate/Zlib.cs index 6fbeb4d9..9ba751d8 100644 --- a/src/SharpCompress/Compressors/Deflate/Zlib.cs +++ b/src/SharpCompress/Compressors/Deflate/Zlib.cs @@ -62,8 +62,8 @@ // // ----------------------------------------------------------------------- -using System; using System.IO; +using SharpCompress.Common; namespace SharpCompress.Compressors.Deflate; @@ -143,7 +143,7 @@ public enum CompressionLevel /// /// A synonym for BestCompression. /// - Level9 = BestCompression + Level9 = BestCompression, } /// @@ -171,13 +171,13 @@ public enum CompressionStrategy /// Using HuffmanOnly will force the compressor to do Huffman encoding only, with no /// string matching. /// - HuffmanOnly = 2 + HuffmanOnly = 2, } /// /// A general purpose exception class for exceptions in the Zlib library. /// -public class ZlibException : Exception +public class ZlibException : SharpCompressException { /// /// The ZlibException class captures exception information generated @@ -859,7 +859,7 @@ internal sealed class StaticTree 99, 8, 227, - 8 + 8, }; internal static readonly short[] distTreeCodes = @@ -923,7 +923,7 @@ internal sealed class StaticTree 7, 5, 23, - 5 + 5, }; // extra bits for each bit length code @@ -947,7 +947,7 @@ internal sealed class StaticTree 0, 2, 3, - 7 + 7, }; internal static readonly StaticTree Literals; diff --git a/src/SharpCompress/Compressors/Deflate/ZlibBaseStream.cs b/src/SharpCompress/Compressors/Deflate/ZlibBaseStream.cs index 19a3c60a..c7ea2a6f 100644 --- a/src/SharpCompress/Compressors/Deflate/ZlibBaseStream.cs +++ b/src/SharpCompress/Compressors/Deflate/ZlibBaseStream.cs @@ -27,11 +27,16 @@ // ------------------------------------------------------------------ using System; +using System.Buffers; using System.Buffers.Binary; using System.Collections.Generic; using System.IO; -using SharpCompress.Common.Tar.Headers; using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Tar.Headers; +using SharpCompress.IO; namespace SharpCompress.Compressors.Deflate; @@ -39,11 +44,13 @@ internal enum ZlibStreamFlavor { ZLIB = 1950, DEFLATE = 1951, - GZIP = 1952 + GZIP = 1952, } -internal class ZlibBaseStream : Stream +internal class ZlibBaseStream : Stream, IStreamStack { + Stream IStreamStack.BaseStream() => _stream; + protected internal ZlibCodec _z; // deferred init... new ZlibCodec(); protected internal StreamMode _streamMode = StreamMode.Undefined; @@ -66,6 +73,7 @@ internal class ZlibBaseStream : Stream protected internal int _gzipHeaderByteCount; private readonly Encoding _encoding; + private readonly bool _leaveOpen; internal int Crc32 => crc?.Crc32Result ?? 0; @@ -75,9 +83,20 @@ internal class ZlibBaseStream : Stream CompressionLevel level, ZlibStreamFlavor flavor, Encoding encoding + ) + : this(stream, compressionMode, level, flavor, leaveOpen: false, encoding) { } + + public ZlibBaseStream( + Stream stream, + CompressionMode compressionMode, + CompressionLevel level, + ZlibStreamFlavor flavor, + bool leaveOpen, + Encoding encoding ) { _flushMode = FlushType.None; + _leaveOpen = leaveOpen; //this._workingBuffer = new byte[WORKING_BUFFER_SIZE_DEFAULT]; _stream = stream; @@ -102,7 +121,7 @@ internal class ZlibBaseStream : Stream { if (_z is null) { - bool wantRfc1950Header = (_flavor == ZlibStreamFlavor.ZLIB); + var wantRfc1950Header = (_flavor == ZlibStreamFlavor.ZLIB); _z = new ZlibCodec(); if (_compressionMode == CompressionMode.Decompress) { @@ -118,7 +137,7 @@ internal class ZlibBaseStream : Stream } } - private byte[] workingBuffer => _workingBuffer ??= new byte[_bufferSize]; + private byte[] workingBuffer => _workingBuffer ??= ArrayPool.Shared.Rent(_bufferSize); public override void Write(byte[] buffer, int offset, int count) { @@ -147,13 +166,13 @@ internal class ZlibBaseStream : Stream z.InputBuffer = buffer; _z.NextIn = offset; _z.AvailableBytesIn = count; - bool done = false; + var done = false; do { _z.OutputBuffer = workingBuffer; _z.NextOut = 0; _z.AvailableBytesOut = _workingBuffer.Length; - int rc = (_wantCompress) ? _z.Deflate(_flushMode) : _z.Inflate(_flushMode); + var rc = (_wantCompress) ? _z.Deflate(_flushMode) : _z.Inflate(_flushMode); if (rc != ZlibConstants.Z_OK && rc != ZlibConstants.Z_STREAM_END) { throw new ZlibException((_wantCompress ? "de" : "in") + "flating: " + _z.Message); @@ -172,6 +191,69 @@ internal class ZlibBaseStream : Stream } while (!done); } + public override async Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + // workitem 7159 + // calculate the CRC on the unccompressed data (before writing) + if (crc != null) + { + crc.SlurpBlock(buffer, offset, count); + } + + if (_streamMode == StreamMode.Undefined) + { + _streamMode = StreamMode.Writer; + } + else if (_streamMode != StreamMode.Writer) + { + throw new ZlibException("Cannot Write after Reading."); + } + + if (count == 0) + { + return; + } + + // first reference of z property will initialize the private var _z + z.InputBuffer = buffer; + _z.NextIn = offset; + _z.AvailableBytesIn = count; + var done = false; + do + { + _z.OutputBuffer = workingBuffer; + _z.NextOut = 0; + _z.AvailableBytesOut = _workingBuffer.Length; + var rc = (_wantCompress) ? _z.Deflate(_flushMode) : _z.Inflate(_flushMode); + if (rc != ZlibConstants.Z_OK && rc != ZlibConstants.Z_STREAM_END) + { + throw new ZlibException((_wantCompress ? "de" : "in") + "flating: " + _z.Message); + } + + await _stream + .WriteAsync( + _workingBuffer, + 0, + _workingBuffer.Length - _z.AvailableBytesOut, + cancellationToken + ) + .ConfigureAwait(false); + + done = _z.AvailableBytesIn == 0 && _z.AvailableBytesOut != 0; + + // If GZIP and de-compress, we're done when 8 bytes remain. + if (_flavor == ZlibStreamFlavor.GZIP && !_wantCompress) + { + done = (_z.AvailableBytesIn == 8 && _z.AvailableBytesOut != 0); + } + } while (!done); + } + private void finish() { if (_z is null) @@ -181,21 +263,23 @@ internal class ZlibBaseStream : Stream if (_streamMode == StreamMode.Writer) { - bool done = false; + var done = false; do { _z.OutputBuffer = workingBuffer; _z.NextOut = 0; _z.AvailableBytesOut = _workingBuffer.Length; - int rc = + var rc = (_wantCompress) ? _z.Deflate(FlushType.Finish) : _z.Inflate(FlushType.Finish); if (rc != ZlibConstants.Z_STREAM_END && rc != ZlibConstants.Z_OK) { - string verb = (_wantCompress ? "de" : "in") + "flating"; + var verb = (_wantCompress ? "de" : "in") + "flating"; if (_z.Message is null) { - throw new ZlibException(String.Format("{0}: (rc = {1})", verb, rc)); + throw new ZlibException( + String.Format(Constants.DefaultCultureInfo, "{0}: (rc = {1})", verb, rc) + ); } throw new ZlibException(verb + ": " + _z.Message); } @@ -225,7 +309,7 @@ internal class ZlibBaseStream : Stream Span intBuf = stackalloc byte[4]; BinaryPrimitives.WriteInt32LittleEndian(intBuf, crc.Crc32Result); _stream.Write(intBuf); - int c2 = (int)(crc.TotalBytesRead & 0x00000000FFFFFFFF); + var c2 = (int)(crc.TotalBytesRead & 0x00000000FFFFFFFF); BinaryPrimitives.WriteInt32LittleEndian(intBuf, c2); _stream.Write(intBuf); } @@ -256,14 +340,15 @@ internal class ZlibBaseStream : Stream { // Make sure we have read to the end of the stream _z.InputBuffer.AsSpan(_z.NextIn, _z.AvailableBytesIn).CopyTo(trailer); - int bytesNeeded = 8 - _z.AvailableBytesIn; - int bytesRead = _stream.Read( + var bytesNeeded = 8 - _z.AvailableBytesIn; + var bytesRead = _stream.Read( trailer.Slice(_z.AvailableBytesIn, bytesNeeded) ); if (bytesNeeded != bytesRead) { throw new ZlibException( String.Format( + Constants.DefaultCultureInfo, "Protocol error. AvailableBytesIn={0}, expected 8", _z.AvailableBytesIn + bytesRead ) @@ -275,15 +360,16 @@ internal class ZlibBaseStream : Stream _z.InputBuffer.AsSpan(_z.NextIn, trailer.Length).CopyTo(trailer); } - Int32 crc32_expected = BinaryPrimitives.ReadInt32LittleEndian(trailer); - Int32 crc32_actual = crc.Crc32Result; - Int32 isize_expected = BinaryPrimitives.ReadInt32LittleEndian(trailer.Slice(4)); - Int32 isize_actual = (Int32)(_z.TotalBytesOut & 0x00000000FFFFFFFF); + var crc32_expected = BinaryPrimitives.ReadInt32LittleEndian(trailer); + var crc32_actual = crc.Crc32Result; + var isize_expected = BinaryPrimitives.ReadInt32LittleEndian(trailer.Slice(4)); + var isize_actual = (Int32)(_z.TotalBytesOut & 0x00000000FFFFFFFF); if (crc32_actual != crc32_expected) { throw new ZlibException( String.Format( + Constants.DefaultCultureInfo, "Bad CRC32 in GZIP stream. (actual({0:X8})!=expected({1:X8}))", crc32_actual, crc32_expected @@ -295,6 +381,7 @@ internal class ZlibBaseStream : Stream { throw new ZlibException( String.Format( + Constants.DefaultCultureInfo, "Bad size in GZIP stream. (actual({0})!=expected({1}))", isize_actual, isize_expected @@ -310,6 +397,113 @@ internal class ZlibBaseStream : Stream } } + private async ValueTask finishAsync(CancellationToken cancellationToken = default) + { + if (_z is null) + { + return; + } + + if (_streamMode == StreamMode.Writer) + { + var done = false; + do + { + _z.OutputBuffer = workingBuffer; + _z.NextOut = 0; + _z.AvailableBytesOut = _workingBuffer.Length; + var rc = + (_wantCompress) ? _z.Deflate(FlushType.Finish) : _z.Inflate(FlushType.Finish); + + if (rc != ZlibConstants.Z_STREAM_END && rc != ZlibConstants.Z_OK) + { + var verb = (_wantCompress ? "de" : "in") + "flating"; + if (_z.Message is null) + { + throw new ZlibException( + String.Format(Constants.DefaultCultureInfo, "{0}: (rc = {1})", verb, rc) + ); + } + throw new ZlibException(verb + ": " + _z.Message); + } + + if (_workingBuffer.Length - _z.AvailableBytesOut > 0) + { + await _stream + .WriteAsync( + _workingBuffer, + 0, + _workingBuffer.Length - _z.AvailableBytesOut, + cancellationToken + ) + .ConfigureAwait(false); + } + + done = _z.AvailableBytesIn == 0 && _z.AvailableBytesOut != 0; + + // If GZIP and de-compress, we're done when 8 bytes remain. + if (_flavor == ZlibStreamFlavor.GZIP && !_wantCompress) + { + done = (_z.AvailableBytesIn == 8 && _z.AvailableBytesOut != 0); + } + } while (!done); + + await FlushAsync(cancellationToken).ConfigureAwait(false); + + // workitem 7159 + if (_flavor == ZlibStreamFlavor.GZIP) + { + if (_wantCompress) + { + // Emit the GZIP trailer: CRC32 and size mod 2^32 + byte[] intBuf = new byte[4]; + BinaryPrimitives.WriteInt32LittleEndian(intBuf, crc.Crc32Result); + await _stream.WriteAsync(intBuf, 0, 4, cancellationToken).ConfigureAwait(false); + var c2 = (int)(crc.TotalBytesRead & 0x00000000FFFFFFFF); + BinaryPrimitives.WriteInt32LittleEndian(intBuf, c2); + await _stream.WriteAsync(intBuf, 0, 4, cancellationToken).ConfigureAwait(false); + } + else + { + throw new ZlibException("Writing with decompression is not supported."); + } + } + } + // workitem 7159 + else if (_streamMode == StreamMode.Reader) + { + if (_flavor == ZlibStreamFlavor.GZIP) + { + if (!_wantCompress) + { + // workitem 8501: handle edge case (decompress empty stream) + if (_z.TotalBytesOut == 0L) + { + return; + } + + // Read and potentially verify the GZIP trailer: CRC32 and size mod 2^32 + byte[] trailer = new byte[8]; + + // workitem 8679 + if (_z.AvailableBytesIn != 8) + { + // Make sure we have read to the end of the stream + _z.InputBuffer.AsSpan(_z.NextIn, _z.AvailableBytesIn).CopyTo(trailer); + var bytesNeeded = 8 - _z.AvailableBytesIn; + var bytesRead = await _stream + .ReadAsync(trailer, _z.AvailableBytesIn, bytesNeeded, cancellationToken) + .ConfigureAwait(false); + } + } + else + { + throw new ZlibException("Reading with compression is not supported."); + } + } + } + } + private void end() { if (z is null) @@ -348,13 +542,112 @@ internal class ZlibBaseStream : Stream finally { end(); - _stream?.Dispose(); + ReturnWorkingBuffer(); + if (!_leaveOpen) + { + _stream?.Dispose(); + } _stream = null; } } } - public override void Flush() => _stream.Flush(); +#if !LEGACY_DOTNET || NETSTANDARD2_1 + public override async ValueTask DisposeAsync() +#else + public async ValueTask DisposeAsync() +#endif + { + if (isDisposed) + { + return; + } + isDisposed = true; +#if !LEGACY_DOTNET || NETSTANDARD2_1 + await base.DisposeAsync().ConfigureAwait(false); +#endif + if (_stream is null) + { + return; + } + try + { + await finishAsync().ConfigureAwait(false); + } + finally + { + end(); + ReturnWorkingBuffer(); + if (_stream != null) + { + if (!_leaveOpen) + { + if (_stream is IAsyncDisposable asyncDisposableStream) + { + await asyncDisposableStream.DisposeAsync().ConfigureAwait(false); + } + else + { + _stream.Dispose(); + } + } + _stream = null; + } + } + } + + private void ReturnWorkingBuffer() + { + if (_workingBuffer is null) + { + return; + } + + ArrayPool.Shared.Return(_workingBuffer, clearArray: true); + _workingBuffer = null; + } + + public override void Flush() + { + // Only flush the underlying stream when in write mode + // Flushing input streams during read operations is not meaningful + // and can cause issues with forward-only/non-seekable streams + if (_streamMode == StreamMode.Writer) + { + _stream.Flush(); + } + else if (z.AvailableBytesIn > 0) + { + // Rewind the underlying stream by the number of unconsumed bytes in the buffer + // This handles the case where the decompressor over-read past the end of the entry + if (_stream is IStreamStack stack) + { + stack.Rewind(z.AvailableBytesIn); + } + z.AvailableBytesIn = 0; + } + } + + public override async Task FlushAsync(CancellationToken cancellationToken) + { + // Only flush the underlying stream when in write mode + // Flushing input streams during read operations is not meaningful + // and can cause issues with forward-only/non-seekable streams + if (_streamMode == StreamMode.Writer) + { + await _stream.FlushAsync(cancellationToken).ConfigureAwait(false); + } + else if (z.AvailableBytesIn > 0) + { + // Rewind the underlying stream by the number of unconsumed bytes in the buffer + // This handles the case where the decompressor over-read past the end of the entry + if (_stream is IStreamStack stack) + { + stack.Rewind(z.AvailableBytesIn); + } + z.AvailableBytesIn = 0; + } + } public override Int64 Seek(Int64 offset, SeekOrigin origin) => throw new NotSupportedException(); @@ -380,11 +673,11 @@ internal class ZlibBaseStream : Stream private string ReadZeroTerminatedString() { var list = new List(); - bool done = false; + var done = false; do { // workitem 7740 - int n = _stream.Read(_buf1, 0, 1); + var n = _stream.Read(_buf1, 0, 1); if (n != 1) { throw new ZlibException("Unexpected EOF reading GZIP header."); @@ -398,17 +691,44 @@ internal class ZlibBaseStream : Stream list.Add(_buf1[0]); } } while (!done); - byte[] buffer = list.ToArray(); + var buffer = list.ToArray(); + return _encoding.GetString(buffer, 0, buffer.Length); + } + + private async ValueTask ReadZeroTerminatedStringAsync( + CancellationToken cancellationToken + ) + { + var list = new List(); + var done = false; + do + { + // workitem 7740 + var n = await _stream.ReadAsync(_buf1, 0, 1, cancellationToken).ConfigureAwait(false); + if (n != 1) + { + throw new ZlibException("Unexpected EOF reading GZIP header."); + } + if (_buf1[0] == 0) + { + done = true; + } + else + { + list.Add(_buf1[0]); + } + } while (!done); + var buffer = list.ToArray(); return _encoding.GetString(buffer, 0, buffer.Length); } private int _ReadAndValidateGzipHeader() { - int totalBytesRead = 0; + var totalBytesRead = 0; // read the header on the first read Span header = stackalloc byte[10]; - int n = _stream.Read(header); + var n = _stream.Read(header); // workitem 8501: handle edge case (decompress empty stream) if (n == 0) @@ -426,7 +746,7 @@ internal class ZlibBaseStream : Stream throw new ZlibException("Bad GZIP header."); } - int timet = BinaryPrimitives.ReadInt32LittleEndian(header.Slice(4)); + var timet = BinaryPrimitives.ReadInt32LittleEndian(header.Slice(4)); _GzipMtime = TarHeader.EPOCH.AddSeconds(timet); totalBytesRead += n; if ((header[3] & 0x04) == 0x04) @@ -435,8 +755,8 @@ internal class ZlibBaseStream : Stream n = _stream.Read(header.Slice(0, 2)); // 2-byte length field totalBytesRead += n; - short extraLength = (short)(header[0] + header[1] * 256); - byte[] extra = new byte[extraLength]; + var extraLength = (short)(header[0] + header[1] * 256); + var extra = new byte[extraLength]; n = _stream.Read(extra, 0, extra.Length); if (n != extraLength) { @@ -460,6 +780,70 @@ internal class ZlibBaseStream : Stream return totalBytesRead; } + private async ValueTask _ReadAndValidateGzipHeaderAsync( + CancellationToken cancellationToken + ) + { + var totalBytesRead = 0; + + // read the header on the first read + byte[] header = new byte[10]; + var n = await _stream.ReadAsync(header, 0, 10, cancellationToken).ConfigureAwait(false); + + // workitem 8501: handle edge case (decompress empty stream) + if (n == 0) + { + return 0; + } + + if (n != 10) + { + throw new ZlibException("Not a valid GZIP stream."); + } + + if (header[0] != 0x1F || header[1] != 0x8B || header[2] != 8) + { + throw new ZlibException("Bad GZIP header."); + } + + var timet = BinaryPrimitives.ReadInt32LittleEndian(header.AsSpan(4)); + _GzipMtime = TarHeader.EPOCH.AddSeconds(timet); + totalBytesRead += n; + if ((header[3] & 0x04) == 0x04) + { + // read and discard extra field + n = await _stream.ReadAsync(header, 0, 2, cancellationToken).ConfigureAwait(false); // 2-byte length field + totalBytesRead += n; + + var extraLength = (short)(header[0] + header[1] * 256); + var extra = new byte[extraLength]; + n = await _stream + .ReadAsync(extra, 0, extra.Length, cancellationToken) + .ConfigureAwait(false); + if (n != extraLength) + { + throw new ZlibException("Unexpected end-of-file reading GZIP header."); + } + totalBytesRead += n; + } + if ((header[3] & 0x08) == 0x08) + { + _GzipFileName = await ReadZeroTerminatedStringAsync(cancellationToken) + .ConfigureAwait(false); + } + if ((header[3] & 0x10) == 0x010) + { + _GzipComment = await ReadZeroTerminatedStringAsync(cancellationToken) + .ConfigureAwait(false); + } + if ((header[3] & 0x02) == 0x02) + { + await _stream.ReadAsync(_buf1, 0, 1, cancellationToken).ConfigureAwait(false); // CRC16, ignore + } + + return totalBytesRead; + } + public override Int32 Read(Byte[] buffer, Int32 offset, Int32 count) { // According to MS documentation, any implementation of the IO.Stream.Read function must: @@ -498,7 +882,7 @@ internal class ZlibBaseStream : Stream throw new ZlibException("Cannot Read after Writing."); } - int rc = 0; + var rc = 0; // set up the output of the deflate/inflate codec: _z.OutputBuffer = buffer; @@ -518,7 +902,12 @@ internal class ZlibBaseStream : Stream if (rc != ZlibConstants.Z_OK && rc != ZlibConstants.Z_STREAM_END) { throw new ZlibException( - String.Format("Deflating: rc={0} msg={1}", rc, _z.Message) + String.Format( + Constants.DefaultCultureInfo, + "Deflating: rc={0} msg={1}", + rc, + _z.Message + ) ); } @@ -532,18 +921,9 @@ internal class ZlibBaseStream : Stream return rc; } - if (buffer is null) - { - throw new ArgumentNullException(nameof(buffer)); - } - if (count < 0) - { - throw new ArgumentOutOfRangeException(nameof(count)); - } - if (offset < buffer.GetLowerBound(0)) - { - throw new ArgumentOutOfRangeException(nameof(offset)); - } + ThrowHelper.ThrowIfNull(buffer); + ThrowHelper.ThrowIfNegative(count); + ThrowHelper.ThrowIfLessThan(offset, buffer.GetLowerBound(0)); if ((offset + count) > buffer.GetLength(0)) { throw new ArgumentOutOfRangeException(nameof(count)); @@ -580,6 +960,7 @@ internal class ZlibBaseStream : Stream { throw new ZlibException( String.Format( + Constants.DefaultCultureInfo, "{0}flating: rc={1} msg={2}", (_wantCompress ? "de" : "in"), rc, @@ -619,7 +1000,12 @@ internal class ZlibBaseStream : Stream if (rc != ZlibConstants.Z_OK && rc != ZlibConstants.Z_STREAM_END) { throw new ZlibException( - String.Format("Deflating: rc={0} msg={1}", rc, _z.Message) + String.Format( + Constants.DefaultCultureInfo, + "Deflating: rc={0} msg={1}", + rc, + _z.Message + ) ); } } @@ -634,9 +1020,232 @@ internal class ZlibBaseStream : Stream crc.SlurpBlock(buffer, offset, rc); } + if (rc == ZlibConstants.Z_STREAM_END && z.AvailableBytesIn != 0 && !_wantCompress) + { + //rewind the buffer + this.Rewind(z.AvailableBytesIn); + z.AvailableBytesIn = 0; + } + return rc; } + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + // According to MS documentation, any implementation of the IO.Stream.Read function must: + // (a) throw an exception if offset & count reference an invalid part of the buffer, + // or if count < 0, or if buffer is null + // (b) return 0 only upon EOF, or if count = 0 + // (c) if not EOF, then return at least 1 byte, up to bytes + + if (_streamMode == StreamMode.Undefined) + { + if (!_stream.CanRead) + { + throw new ZlibException("The stream is not readable."); + } + + // for the first read, set up some controls. + _streamMode = StreamMode.Reader; + + // (The first reference to _z goes through the private accessor which + // may initialize it.) + z.AvailableBytesIn = 0; + if (_flavor == ZlibStreamFlavor.GZIP) + { + _gzipHeaderByteCount = await _ReadAndValidateGzipHeaderAsync(cancellationToken) + .ConfigureAwait(false); + + // workitem 8501: handle edge case (decompress empty stream) + if (_gzipHeaderByteCount == 0) + { + return 0; + } + } + } + + if (_streamMode != StreamMode.Reader) + { + throw new ZlibException("Cannot Read after Writing."); + } + + var rc = 0; + + // set up the output of the deflate/inflate codec: + _z.OutputBuffer = buffer; + _z.NextOut = offset; + _z.AvailableBytesOut = count; + + if (count == 0) + { + return 0; + } + if (nomoreinput && _wantCompress) + { + // no more input data available; therefore we flush to + // try to complete the read + rc = _z.Deflate(FlushType.Finish); + + if (rc != ZlibConstants.Z_OK && rc != ZlibConstants.Z_STREAM_END) + { + throw new ZlibException( + String.Format( + Constants.DefaultCultureInfo, + "Deflating: rc={0} msg={1}", + rc, + _z.Message + ) + ); + } + + rc = (count - _z.AvailableBytesOut); + + // calculate CRC after reading + if (crc != null) + { + crc.SlurpBlock(buffer, offset, rc); + } + + return rc; + } + ThrowHelper.ThrowIfNull(buffer); + ThrowHelper.ThrowIfNegative(count); + ThrowHelper.ThrowIfLessThan(offset, buffer.GetLowerBound(0)); + if ((offset + count) > buffer.GetLength(0)) + { + throw new ArgumentOutOfRangeException(nameof(count)); + } + + // This is necessary in case _workingBuffer has been resized. (new byte[]) + // (The first reference to _workingBuffer goes through the private accessor which + // may initialize it.) + _z.InputBuffer = workingBuffer; + + do + { + // need data in _workingBuffer in order to deflate/inflate. Here, we check if we have any. + if ((_z.AvailableBytesIn == 0) && (!nomoreinput)) + { + // No data available, so try to Read data from the captive stream. + _z.NextIn = 0; + _z.AvailableBytesIn = await _stream + .ReadAsync(_workingBuffer, 0, _workingBuffer.Length, cancellationToken) + .ConfigureAwait(false); + if (_z.AvailableBytesIn == 0) + { + nomoreinput = true; + } + } + + // we have data in InputBuffer; now compress or decompress as appropriate + rc = (_wantCompress) ? _z.Deflate(_flushMode) : _z.Inflate(_flushMode); + + if (nomoreinput && (rc == ZlibConstants.Z_BUF_ERROR)) + { + return 0; + } + + if (rc != ZlibConstants.Z_OK && rc != ZlibConstants.Z_STREAM_END) + { + throw new ZlibException( + String.Format( + Constants.DefaultCultureInfo, + "{0}flating: rc={1} msg={2}", + (_wantCompress ? "de" : "in"), + rc, + _z.Message + ) + ); + } + + if ( + (nomoreinput || rc == ZlibConstants.Z_STREAM_END) && (_z.AvailableBytesOut == count) + ) + { + break; // nothing more to read + } + } //while (_z.AvailableBytesOut == count && rc == ZlibConstants.Z_OK); + while (_z.AvailableBytesOut > 0 && !nomoreinput && rc == ZlibConstants.Z_OK); + + // workitem 8557 + // is there more room in output? + if (_z.AvailableBytesOut > 0) + { + if (rc == ZlibConstants.Z_OK && _z.AvailableBytesIn == 0) + { + // deferred + } + + // are we completely done reading? + if (nomoreinput) + { + // and in compression? + if (_wantCompress) + { + // no more input data available; therefore we flush to + // try to complete the read + rc = _z.Deflate(FlushType.Finish); + + if (rc != ZlibConstants.Z_OK && rc != ZlibConstants.Z_STREAM_END) + { + throw new ZlibException( + String.Format( + Constants.DefaultCultureInfo, + "Deflating: rc={0} msg={1}", + rc, + _z.Message + ) + ); + } + } + } + } + + rc = (count - _z.AvailableBytesOut); + + // calculate CRC after reading + if (crc != null) + { + crc.SlurpBlock(buffer, offset, rc); + } + + if (rc == ZlibConstants.Z_STREAM_END && z.AvailableBytesIn != 0 && !_wantCompress) + { + //rewind the buffer + this.Rewind(z.AvailableBytesIn); + z.AvailableBytesIn = 0; + } + + return rc; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + // Use ArrayPool to rent a buffer and delegate to byte[] ReadAsync + byte[] array = System.Buffers.ArrayPool.Shared.Rent(buffer.Length); + try + { + int read = await ReadAsync(array, 0, buffer.Length, cancellationToken) + .ConfigureAwait(false); + array.AsSpan(0, read).CopyTo(buffer.Span); + return read; + } + finally + { + System.Buffers.ArrayPool.Shared.Return(array); + } + } +#endif + public override Boolean CanRead => _stream.CanRead; public override Boolean CanSeek => _stream.CanSeek; @@ -655,6 +1264,6 @@ internal class ZlibBaseStream : Stream { Writer, Reader, - Undefined + Undefined, } } diff --git a/src/SharpCompress/Compressors/Deflate/ZlibCodec.cs b/src/SharpCompress/Compressors/Deflate/ZlibCodec.cs index 7c49c780..5146c28b 100644 --- a/src/SharpCompress/Compressors/Deflate/ZlibCodec.cs +++ b/src/SharpCompress/Compressors/Deflate/ZlibCodec.cs @@ -66,6 +66,7 @@ // ----------------------------------------------------------------------- using System; +using SharpCompress.Common; namespace SharpCompress.Compressors.Deflate; @@ -611,10 +612,9 @@ internal sealed class ZlibCodec throw new ZlibException("No Deflate State!"); } - // TODO: dinoch Tue, 03 Nov 2009 15:39 (test this) - //int ret = dstate.End(); + _ = dstate.End(); dstate = null; - return ZlibConstants.Z_OK; //ret; + return ZlibConstants.Z_OK; } /// @@ -696,6 +696,7 @@ internal sealed class ZlibCodec { throw new ZlibException( string.Format( + Constants.DefaultCultureInfo, "Invalid State. (pending.Length={0}, pendingCount={1})", dstate.pending.Length, dstate.pendingCount diff --git a/src/SharpCompress/Compressors/Deflate/ZlibConstants.cs b/src/SharpCompress/Compressors/Deflate/ZlibConstants.cs index 1a372a31..7bf8a3f6 100644 --- a/src/SharpCompress/Compressors/Deflate/ZlibConstants.cs +++ b/src/SharpCompress/Compressors/Deflate/ZlibConstants.cs @@ -60,7 +60,6 @@ // // ----------------------------------------------------------------------- - namespace SharpCompress.Compressors.Deflate; /// @@ -109,13 +108,9 @@ internal static class ZlibConstants public const int Z_BUF_ERROR = -5; /// - /// The size of the working buffer used in the ZlibCodec class. Defaults to 8192 bytes. + /// The size of the working buffer used in the ZlibCodec class. Defaults to 16384 bytes. /// -#if NETCF - public const int WorkingBufferSizeDefault = 8192; -#else public const int WorkingBufferSizeDefault = 16384; -#endif /// /// The minimum size of the working buffer used in the ZlibCodec class. Currently it is 128 bytes. diff --git a/src/SharpCompress/Compressors/Deflate/ZlibStream.Async.cs b/src/SharpCompress/Compressors/Deflate/ZlibStream.Async.cs new file mode 100644 index 00000000..0dee2197 --- /dev/null +++ b/src/SharpCompress/Compressors/Deflate/ZlibStream.Async.cs @@ -0,0 +1,95 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.IO; + +namespace SharpCompress.Compressors.Deflate; + +public partial class ZlibStream +{ + public override async Task FlushAsync(CancellationToken cancellationToken) + { + if (_disposed) + { + throw new ObjectDisposedException("ZlibStream"); + } + await _baseStream.FlushAsync(cancellationToken).ConfigureAwait(false); + } + +#if !LEGACY_DOTNET + public override async ValueTask DisposeAsync() + { + if (_disposed) + { + return; + } + _disposed = true; + if (_baseStream != null) + { + await _baseStream.DisposeAsync().ConfigureAwait(false); + } + await base.DisposeAsync().ConfigureAwait(false); + } +#endif + + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + if (_disposed) + { + throw new ObjectDisposedException("ZlibStream"); + } + return await _baseStream + .ReadAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + if (_disposed) + { + throw new ObjectDisposedException("ZlibStream"); + } + return await _baseStream.ReadAsync(buffer, cancellationToken).ConfigureAwait(false); + } +#endif + + public override async Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + if (_disposed) + { + throw new ObjectDisposedException("ZlibStream"); + } + await _baseStream + .WriteAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + } + +#if !LEGACY_DOTNET + public override async ValueTask WriteAsync( + ReadOnlyMemory buffer, + CancellationToken cancellationToken = default + ) + { + if (_disposed) + { + throw new ObjectDisposedException("ZlibStream"); + } + await _baseStream.WriteAsync(buffer, cancellationToken).ConfigureAwait(false); + } +#endif +} diff --git a/src/SharpCompress/Compressors/Deflate/ZlibStream.cs b/src/SharpCompress/Compressors/Deflate/ZlibStream.cs index e2c69b53..c4652d44 100644 --- a/src/SharpCompress/Compressors/Deflate/ZlibStream.cs +++ b/src/SharpCompress/Compressors/Deflate/ZlibStream.cs @@ -28,10 +28,13 @@ using System; using System.IO; using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; namespace SharpCompress.Compressors.Deflate; -public class ZlibStream : Stream +public partial class ZlibStream : Stream { private readonly ZlibBaseStream _baseStream; private bool _disposed; @@ -47,7 +50,10 @@ public class ZlibStream : Stream CompressionMode mode, CompressionLevel level, Encoding encoding - ) => _baseStream = new ZlibBaseStream(stream, mode, level, ZlibStreamFlavor.ZLIB, encoding); + ) + { + _baseStream = new ZlibBaseStream(stream, mode, level, ZlibStreamFlavor.ZLIB, encoding); + } #region Zlib properties @@ -102,6 +108,7 @@ public class ZlibStream : Stream { throw new ZlibException( string.Format( + Constants.DefaultCultureInfo, "Don't be silly. {0} bytes?? Use a bigger buffer, at least {1}.", value, ZlibConstants.WorkingBufferSizeMin diff --git a/src/SharpCompress/Compressors/Deflate64/BlockType.cs b/src/SharpCompress/Compressors/Deflate64/BlockType.cs index 5bd9eb83..b34ca903 100644 --- a/src/SharpCompress/Compressors/Deflate64/BlockType.cs +++ b/src/SharpCompress/Compressors/Deflate64/BlockType.cs @@ -8,5 +8,5 @@ internal enum BlockType { Uncompressed = 0, Static = 1, - Dynamic = 2 + Dynamic = 2, } diff --git a/src/SharpCompress/Compressors/Deflate64/Deflate64Stream.Async.cs b/src/SharpCompress/Compressors/Deflate64/Deflate64Stream.Async.cs new file mode 100644 index 00000000..64652872 --- /dev/null +++ b/src/SharpCompress/Compressors/Deflate64/Deflate64Stream.Async.cs @@ -0,0 +1,108 @@ +using System; +using System.IO; +using System.Runtime.InteropServices; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.IO; + +namespace SharpCompress.Compressors.Deflate64; + +public sealed partial class Deflate64Stream +{ + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + ValidateParameters(buffer, offset, count); + EnsureNotDisposed(); + + int bytesRead; + var currentOffset = offset; + var remainingCount = count; + + while (true) + { + bytesRead = _inflater.Inflate(buffer, currentOffset, remainingCount); + currentOffset += bytesRead; + remainingCount -= bytesRead; + + if (remainingCount == 0) + { + break; + } + + if (_inflater.Finished()) + { + // if we finished decompressing, we can't have anything left in the outputwindow. + break; + } + + var bytes = await _stream + .ReadAsync(_buffer, 0, _buffer.Length, cancellationToken) + .ConfigureAwait(false); + if (bytes <= 0) + { + break; + } + else if (bytes > _buffer.Length) + { + // The stream is either malicious or poorly implemented and returned a number of + // bytes larger than the buffer supplied to it. + throw new InvalidFormatException("Deflate64: invalid data"); + } + + _inflater.SetInput(_buffer, 0, bytes); + } + + return count - remainingCount; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + EnsureNotDisposed(); + + // InflaterManaged doesn't have a Span-based Inflate method, so we need to work with arrays + // For large buffers, we could rent from ArrayPool, but for simplicity we'll use the buffer's array if available + if ( + System.Runtime.InteropServices.MemoryMarshal.TryGetArray( + buffer, + out var arraySegment + ) + ) + { + // Fast path: the Memory is backed by an array + return await ReadAsync( + arraySegment.Array!, + arraySegment.Offset, + arraySegment.Count, + cancellationToken + ) + .ConfigureAwait(false); + } + else + { + // Slow path: rent a temporary array + var tempBuffer = System.Buffers.ArrayPool.Shared.Rent(buffer.Length); + try + { + var bytesRead = await ReadAsync(tempBuffer, 0, buffer.Length, cancellationToken) + .ConfigureAwait(false); + tempBuffer.AsMemory(0, bytesRead).CopyTo(buffer); + return bytesRead; + } + finally + { + System.Buffers.ArrayPool.Shared.Return(tempBuffer); + } + } + } +#endif +} diff --git a/src/SharpCompress/Compressors/Deflate64/Deflate64Stream.cs b/src/SharpCompress/Compressors/Deflate64/Deflate64Stream.cs index 149f7c92..9e9ff7db 100644 --- a/src/SharpCompress/Compressors/Deflate64/Deflate64Stream.cs +++ b/src/SharpCompress/Compressors/Deflate64/Deflate64Stream.cs @@ -2,31 +2,27 @@ // The .NET Foundation licenses this file to you under the MIT license. // See the LICENSE file in the project root for more information. -#nullable disable - -using SharpCompress.Common.Zip; using System; -using System.Diagnostics; using System.IO; using System.Runtime.CompilerServices; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Zip; namespace SharpCompress.Compressors.Deflate64; -public sealed class Deflate64Stream : Stream +public sealed partial class Deflate64Stream : Stream { private const int DEFAULT_BUFFER_SIZE = 8192; private Stream _stream; - private CompressionMode _mode; private InflaterManaged _inflater; private byte[] _buffer; public Deflate64Stream(Stream stream, CompressionMode mode) { - if (stream is null) - { - throw new ArgumentNullException(nameof(stream)); - } + ThrowHelper.ThrowIfNull(stream); if (mode != CompressionMode.Decompress) { @@ -40,58 +36,20 @@ public sealed class Deflate64Stream : Stream throw new ArgumentException("Deflate64: input stream is not readable", nameof(stream)); } - InitializeInflater(stream, ZipCompressionMethod.Deflate64); - } - - /// - /// Sets up this DeflateManagedStream to be used for Inflation/Decompression - /// - private void InitializeInflater( - Stream stream, - ZipCompressionMethod method = ZipCompressionMethod.Deflate - ) - { - Debug.Assert(stream != null); - Debug.Assert( - method == ZipCompressionMethod.Deflate || method == ZipCompressionMethod.Deflate64 - ); if (!stream.CanRead) { throw new ArgumentException("Deflate64: input stream is not readable", nameof(stream)); } - _inflater = new InflaterManaged(method == ZipCompressionMethod.Deflate64); + _inflater = new InflaterManaged(true); _stream = stream; - _mode = CompressionMode.Decompress; _buffer = new byte[DEFAULT_BUFFER_SIZE]; } - public override bool CanRead - { - get - { - if (_stream is null) - { - return false; - } + public override bool CanRead => _stream.CanRead; - return (_mode == CompressionMode.Decompress && _stream.CanRead); - } - } - - public override bool CanWrite - { - get - { - if (_stream is null) - { - return false; - } - - return (_mode == CompressionMode.Compress && _stream.CanWrite); - } - } + public override bool CanWrite => false; public override bool CanSeek => false; @@ -111,19 +69,18 @@ public sealed class Deflate64Stream : Stream public override void SetLength(long value) => throw new NotSupportedException("Deflate64: not supported"); - public override int Read(byte[] array, int offset, int count) + public override int Read(byte[] buffer, int offset, int count) { - EnsureDecompressionMode(); - ValidateParameters(array, offset, count); + ValidateParameters(buffer, offset, count); EnsureNotDisposed(); int bytesRead; - int currentOffset = offset; - int remainingCount = count; + var currentOffset = offset; + var remainingCount = count; while (true) { - bytesRead = _inflater.Inflate(array, currentOffset, remainingCount); + bytesRead = _inflater.Inflate(buffer, currentOffset, remainingCount); currentOffset += bytesRead; remainingCount -= bytesRead; @@ -135,14 +92,10 @@ public sealed class Deflate64Stream : Stream if (_inflater.Finished()) { // if we finished decompressing, we can't have anything left in the outputwindow. - Debug.Assert( - _inflater.AvailableOutput == 0, - "We should have copied all stuff out!" - ); break; } - int bytes = _stream.Read(_buffer, 0, _buffer.Length); + var bytes = _stream.Read(_buffer, 0, _buffer.Length); if (bytes <= 0) { break; @@ -151,7 +104,7 @@ public sealed class Deflate64Stream : Stream { // The stream is either malicious or poorly implemented and returned a number of // bytes larger than the buffer supplied to it. - throw new InvalidDataException("Deflate64: invalid data"); + throw new InvalidFormatException("Deflate64: invalid data"); } _inflater.SetInput(_buffer, 0, bytes); @@ -162,20 +115,11 @@ public sealed class Deflate64Stream : Stream private void ValidateParameters(byte[] array, int offset, int count) { - if (array is null) - { - throw new ArgumentNullException(nameof(array)); - } + ThrowHelper.ThrowIfNull(array); - if (offset < 0) - { - throw new ArgumentOutOfRangeException(nameof(offset)); - } + ThrowHelper.ThrowIfNegative(offset); - if (count < 0) - { - throw new ArgumentOutOfRangeException(nameof(count)); - } + ThrowHelper.ThrowIfNegative(count); if (array.Length - offset < count) { @@ -195,31 +139,11 @@ public sealed class Deflate64Stream : Stream private static void ThrowStreamClosedException() => throw new ObjectDisposedException(null, "Deflate64: stream has been disposed"); - private void EnsureDecompressionMode() - { - if (_mode != CompressionMode.Decompress) - { - ThrowCannotReadFromDeflateManagedStreamException(); - } - } - - [MethodImpl(MethodImplOptions.NoInlining)] - private static void ThrowCannotReadFromDeflateManagedStreamException() => - throw new InvalidOperationException("Deflate64: cannot read from this stream"); - - private void EnsureCompressionMode() - { - if (_mode != CompressionMode.Compress) - { - ThrowCannotWriteToDeflateManagedStreamException(); - } - } - [MethodImpl(MethodImplOptions.NoInlining)] private static void ThrowCannotWriteToDeflateManagedStreamException() => - throw new InvalidOperationException("Deflate64: cannot write to this stream"); + throw new ArchiveOperationException("Deflate64: cannot write to this stream"); - public override void Write(byte[] array, int offset, int count) => + public override void Write(byte[] buffer, int offset, int count) => ThrowCannotWriteToDeflateManagedStreamException(); // This is called by Dispose: @@ -253,20 +177,17 @@ public sealed class Deflate64Stream : Stream { if (disposing) { - _stream?.Dispose(); + _stream.Dispose(); } } finally { - _stream = null; - try { - _inflater?.Dispose(); + _inflater.Dispose(); } finally { - _inflater = null; base.Dispose(disposing); } } diff --git a/src/SharpCompress/Compressors/Deflate64/DeflateInput.cs b/src/SharpCompress/Compressors/Deflate64/DeflateInput.cs index 6a12df3b..6bac3c51 100644 --- a/src/SharpCompress/Compressors/Deflate64/DeflateInput.cs +++ b/src/SharpCompress/Compressors/Deflate64/DeflateInput.cs @@ -2,8 +2,6 @@ // The .NET Foundation licenses this file to you under the MIT license. // See the LICENSE file in the project root for more information. -using System.Diagnostics; - namespace SharpCompress.Compressors.Deflate64; internal sealed class DeflateInput @@ -16,13 +14,11 @@ internal sealed class DeflateInput internal void ConsumeBytes(int n) { - Debug.Assert(n <= Count, "Should use more bytes than what we have in the buffer"); StartIndex += n; Count -= n; - Debug.Assert(StartIndex + Count <= Buffer.Length, "Input buffer is in invalid state!"); } - internal InputState DumpState() => new InputState(Count, StartIndex); + internal InputState DumpState() => new(Count, StartIndex); internal void RestoreState(InputState state) { diff --git a/src/SharpCompress/Compressors/Deflate64/FastEncoderStatus.cs b/src/SharpCompress/Compressors/Deflate64/FastEncoderStatus.cs index 71e3f680..2cb29051 100644 --- a/src/SharpCompress/Compressors/Deflate64/FastEncoderStatus.cs +++ b/src/SharpCompress/Compressors/Deflate64/FastEncoderStatus.cs @@ -3,7 +3,6 @@ // See the LICENSE file in the project root for more information. using System; -using System.Diagnostics; namespace SharpCompress.Compressors.Deflate64; @@ -111,7 +110,7 @@ internal static class FastEncoderStatics 0x7e, 0x7c, 0x1f, - 0x3f + 0x3f, }; internal static ReadOnlySpan B_FINAL_FAST_ENCODER_TREE_STRUCTURE_DATA => @@ -214,7 +213,7 @@ internal static class FastEncoderStatics 0x7e, 0x7c, 0x1f, - 0x3f + 0x3f, }; // Output a currentMatch with length matchLen (>= MIN_MATCH) and displacement matchPos @@ -243,7 +242,6 @@ internal static class FastEncoderStatics // cache locality, fewer memory operations. // - // Encoding information for literal and Length. // The least 5 significant bits are the length // and the rest is the code bits. @@ -762,7 +760,7 @@ internal static class FastEncoderStatics 0x0039e7f1, 0x003be7f1, 0x003de7f1, - 0x000047eb + 0x000047eb, }; internal static readonly uint[] FAST_ENCODER_DISTANCE_CODE_INFO = @@ -798,7 +796,7 @@ internal static class FastEncoderStatics 0x000007d5, 0x000017d5, 0x00000000, - 0x00000100 + 0x00000100, }; internal static readonly uint[] BIT_MASK = @@ -818,7 +816,7 @@ internal static class FastEncoderStatics 4095, 8191, 16383, - 32767 + 32767, }; internal static readonly byte[] EXTRA_LENGTH_BITS = { @@ -850,7 +848,7 @@ internal static class FastEncoderStatics 5, 5, 5, - 0 + 0, }; internal static readonly byte[] EXTRA_DISTANCE_BITS = { @@ -885,7 +883,7 @@ internal static class FastEncoderStatics 13, 13, 0, - 0 + 0, }; internal const int NUM_CHARS = 256; internal const int NUM_LENGTH_BASE_CODES = 29; @@ -965,7 +963,6 @@ internal static class FastEncoderStatics { uint newCode = 0; - Debug.Assert(length > 0 && length <= 16, "Invalid len"); do { newCode |= (code & 1); diff --git a/src/SharpCompress/Compressors/Deflate64/HuffmanTree.cs b/src/SharpCompress/Compressors/Deflate64/HuffmanTree.cs index 051b613e..18eb27fb 100644 --- a/src/SharpCompress/Compressors/Deflate64/HuffmanTree.cs +++ b/src/SharpCompress/Compressors/Deflate64/HuffmanTree.cs @@ -3,8 +3,7 @@ // See the LICENSE file in the project root for more information. using System; -using System.Diagnostics; -using System.IO; +using SharpCompress.Common; namespace SharpCompress.Compressors.Deflate64; @@ -35,27 +34,16 @@ internal sealed class HuffmanTree private readonly short[] _left; private readonly short[] _right; private readonly byte[] _codeLengthArray; -#if DEBUG - private uint[]? _codeArrayDebug; -#endif private readonly int _tableMask; // huffman tree for static block - public static HuffmanTree StaticLiteralLengthTree { get; } = - new HuffmanTree(GetStaticLiteralTreeLength()); + public static HuffmanTree StaticLiteralLengthTree { get; } = new(GetStaticLiteralTreeLength()); - public static HuffmanTree StaticDistanceTree { get; } = - new HuffmanTree(GetStaticDistanceTreeLength()); + public static HuffmanTree StaticDistanceTree { get; } = new(GetStaticDistanceTreeLength()); public HuffmanTree(byte[] codeLengths) { - Debug.Assert( - codeLengths.Length == MAX_LITERAL_TREE_ELEMENTS - || codeLengths.Length == MAX_DIST_TREE_ELEMENTS - || codeLengths.Length == NUMBER_OF_CODE_LENGTH_TREE_ELEMENTS, - "we only expect three kinds of Length here" - ); _codeLengthArray = codeLengths; if (_codeLengthArray.Length == MAX_LITERAL_TREE_ELEMENTS) @@ -154,9 +142,6 @@ internal sealed class HuffmanTree private void CreateTable() { var codeArray = CalculateHuffmanCode(); -#if DEBUG - _codeArrayDebug = codeArray; -#endif var avail = (short)_codeLengthArray.Length; @@ -194,7 +179,7 @@ internal sealed class HuffmanTree var increment = 1 << len; if (start >= increment) { - throw new InvalidDataException("Deflate64: invalid Huffman data"); + throw new InvalidFormatException("Deflate64: invalid Huffman data"); } // Note the bits in the table are reverted. @@ -223,6 +208,10 @@ internal sealed class HuffmanTree do { + if (index < 0 || index >= array.Length) + { + throw new InvalidFormatException("Deflate64: invalid Huffman data"); + } var value = array[index]; if (value == 0) @@ -236,14 +225,9 @@ internal sealed class HuffmanTree if (value > 0) { // prevent an IndexOutOfRangeException from array[index] - throw new InvalidDataException("Deflate64: invalid Huffman data"); + throw new InvalidFormatException("Deflate64: invalid Huffman data"); } - Debug.Assert( - value < 0, - "CreateTable: Only negative numbers are used for tree pointers!" - ); - if ((start & codeBitMask) == 0) { // if current bit is 0, go change the left array @@ -260,6 +244,10 @@ internal sealed class HuffmanTree overflowBits--; } while (overflowBits != 0); + if (index < 0 || index >= array.Length) + { + throw new InvalidFormatException("Deflate64: invalid Huffman data"); + } array[index] = (short)ch; } } @@ -309,7 +297,7 @@ internal sealed class HuffmanTree // huffman code lengths must be at least 1 bit long if (codeLength <= 0) { - throw new InvalidDataException("Deflate64: invalid Huffman data"); + throw new InvalidFormatException("Deflate64: invalid Huffman data"); } // diff --git a/src/SharpCompress/Compressors/Deflate64/InflaterManaged.cs b/src/SharpCompress/Compressors/Deflate64/InflaterManaged.cs index 6a3c226b..4b3fdede 100644 --- a/src/SharpCompress/Compressors/Deflate64/InflaterManaged.cs +++ b/src/SharpCompress/Compressors/Deflate64/InflaterManaged.cs @@ -30,7 +30,7 @@ using System; using System.Diagnostics; -using System.IO; +using SharpCompress.Compressors.Deflate; namespace SharpCompress.Compressors.Deflate64; @@ -70,7 +70,7 @@ internal sealed class InflaterManaged 5, 5, 5, - 16 + 16, }; // The base length for length code 257 - 285. @@ -105,7 +105,7 @@ internal sealed class InflaterManaged 163, 195, 227, - 3 + 3, }; // The base distance for distance code 0 - 31 @@ -143,7 +143,7 @@ internal sealed class InflaterManaged 16385, 24577, 32769, - 49153 + 49153, }; // code lengths for code length alphabet is stored in following order @@ -184,7 +184,7 @@ internal sealed class InflaterManaged 0x07, 0x17, 0x0f, - 0x1f + 0x1f, }; private readonly OutputWindow _output; @@ -243,8 +243,8 @@ internal sealed class InflaterManaged private void Reset() => _state = //_hasFormatReader ? - //InflaterState.ReadingHeader : // start by reading Header info - InflaterState.ReadingBFinal; // start by reading BFinal bit + //InflaterState.ReadingHeader : // start by reading Header info + InflaterState.ReadingBFinal; // start by reading BFinal bit public void SetInput(byte[] inputBytes, int offset, int length) => _input.SetInput(inputBytes, offset, length); // append the bytes @@ -385,7 +385,7 @@ internal sealed class InflaterManaged } else { - throw new InvalidDataException("Deflate64: unknown block type"); + throw new ZlibException("Deflate64: unknown block type"); } } @@ -411,7 +411,7 @@ internal sealed class InflaterManaged } else { - throw new InvalidDataException("Deflate64: unknown block type"); + throw new ZlibException("Deflate64: unknown block type"); } // @@ -473,7 +473,7 @@ internal sealed class InflaterManaged // make sure complement matches if ((ushort)_blockLength != (ushort)(~blockLengthComplement)) { - throw new InvalidDataException("Deflate64: invalid block length"); + throw new ZlibException("Deflate64: invalid block length"); } } @@ -507,7 +507,7 @@ internal sealed class InflaterManaged default: Debug. /*Fail*/ Assert(false, "check why we are here!"); - throw new InvalidDataException("Deflate64: unknown state"); + throw new ZlibException("Deflate64: unknown state"); } } } @@ -569,10 +569,9 @@ internal sealed class InflaterManaged { if (symbol < 0 || symbol >= S_EXTRA_LENGTH_BITS.Length) { - throw new InvalidDataException("Deflate64: invalid data"); + throw new ZlibException("Deflate64: invalid data"); } _extraBits = S_EXTRA_LENGTH_BITS[symbol]; - Debug.Assert(_extraBits != 0, "We handle other cases separately!"); } _length = symbol; goto case InflaterState.HaveInitialLength; @@ -591,7 +590,7 @@ internal sealed class InflaterManaged if (_length < 0 || _length >= S_LENGTH_BASE.Length) { - throw new InvalidDataException("Deflate64: invalid data"); + throw new ZlibException("Deflate64: invalid data"); } _length = S_LENGTH_BASE[_length] + bits; } @@ -649,7 +648,7 @@ internal sealed class InflaterManaged default: Debug. /*Fail*/ Assert(false, "check why we are here!"); - throw new InvalidDataException("Deflate64: unknown state"); + throw new ZlibException("Deflate64: unknown state"); } } @@ -781,7 +780,7 @@ internal sealed class InflaterManaged if (_loopCounter == 0) { // can't have "prev code" on first code - throw new InvalidDataException(); + throw new ZlibException(); } var previousCode = _codeList[_loopCounter - 1]; @@ -789,7 +788,7 @@ internal sealed class InflaterManaged if (_loopCounter + repeatCount > _codeArraySize) { - throw new InvalidDataException(); + throw new ZlibException(); } for (var j = 0; j < repeatCount; j++) @@ -809,7 +808,7 @@ internal sealed class InflaterManaged if (_loopCounter + repeatCount > _codeArraySize) { - throw new InvalidDataException(); + throw new ZlibException(); } for (var j = 0; j < repeatCount; j++) @@ -830,7 +829,7 @@ internal sealed class InflaterManaged if (_loopCounter + repeatCount > _codeArraySize) { - throw new InvalidDataException(); + throw new ZlibException(); } for (var j = 0; j < repeatCount; j++) @@ -846,7 +845,7 @@ internal sealed class InflaterManaged default: Debug. /*Fail*/ Assert(false, "check why we are here!"); - throw new InvalidDataException("Deflate64: unknown state"); + throw new ZlibException("Deflate64: unknown state"); } var literalTreeCodeLength = new byte[HuffmanTree.MAX_LITERAL_TREE_ELEMENTS]; @@ -865,7 +864,7 @@ internal sealed class InflaterManaged // Make sure there is an end-of-block code, otherwise how could we ever end? if (literalTreeCodeLength[HuffmanTree.END_OF_BLOCK_CODE] == 0) { - throw new InvalidDataException(); + throw new ZlibException(); } _literalLengthTree = new HuffmanTree(literalTreeCodeLength); diff --git a/src/SharpCompress/Compressors/Deflate64/InflaterState.cs b/src/SharpCompress/Compressors/Deflate64/InflaterState.cs index 711c3245..16d627e0 100644 --- a/src/SharpCompress/Compressors/Deflate64/InflaterState.cs +++ b/src/SharpCompress/Compressors/Deflate64/InflaterState.cs @@ -37,5 +37,5 @@ internal enum InflaterState ReadingFooter = 22, VerifyingFooter = 23, - Done = 24 // Finished + Done = 24, // Finished } diff --git a/src/SharpCompress/Compressors/Deflate64/InputBuffer.cs b/src/SharpCompress/Compressors/Deflate64/InputBuffer.cs index 28d61e42..ec4033e1 100644 --- a/src/SharpCompress/Compressors/Deflate64/InputBuffer.cs +++ b/src/SharpCompress/Compressors/Deflate64/InputBuffer.cs @@ -5,7 +5,6 @@ #nullable disable using System; -using System.Diagnostics; namespace SharpCompress.Compressors.Deflate64; @@ -38,8 +37,6 @@ internal sealed class InputBuffer /// Returns false if input is not sufficient to make this true. public bool EnsureBitsAvailable(int count) { - Debug.Assert(0 < count && count <= 16, "count is invalid."); - // manual inlining to improve perf if (_bitsInBuffer < count) { @@ -106,8 +103,6 @@ internal sealed class InputBuffer /// Gets count bits from the input buffer. Returns -1 if not enough bits available. public int GetBits(int count) { - Debug.Assert(0 < count && count <= 16, "count is invalid."); - if (!EnsureBitsAvailable(count)) { return -1; @@ -127,12 +122,6 @@ internal sealed class InputBuffer /// Returns the number of bytes copied, 0 if no byte is available. public int CopyTo(byte[] output, int offset, int length) { - Debug.Assert(output != null); - Debug.Assert(offset >= 0); - Debug.Assert(length >= 0); - Debug.Assert(offset <= output.Length - length); - Debug.Assert((_bitsInBuffer % 8) == 0); - // Copy the bytes in bitBuffer first. var bytesFromBitBuffer = 0; while (_bitsInBuffer > 0 && length > 0) @@ -175,12 +164,6 @@ internal sealed class InputBuffer /// public void SetInput(byte[] buffer, int offset, int length) { - Debug.Assert(buffer != null); - Debug.Assert(offset >= 0); - Debug.Assert(length >= 0); - Debug.Assert(offset <= buffer.Length - length); - Debug.Assert(_start == _end); - _buffer = buffer; _start = offset; _end = offset + length; @@ -189,10 +172,6 @@ internal sealed class InputBuffer /// Skip n bits in the buffer. public void SkipBits(int n) { - Debug.Assert( - _bitsInBuffer >= n, - "No enough bits in the buffer, Did you call EnsureBitsAvailable?" - ); _bitBuffer >>= n; _bitsInBuffer -= n; } diff --git a/src/SharpCompress/Compressors/Deflate64/MatchState.cs b/src/SharpCompress/Compressors/Deflate64/MatchState.cs index 858e91e8..cabc0bad 100644 --- a/src/SharpCompress/Compressors/Deflate64/MatchState.cs +++ b/src/SharpCompress/Compressors/Deflate64/MatchState.cs @@ -8,5 +8,5 @@ internal enum MatchState { HasSymbol = 1, HasMatch = 2, - HasSymbolAndMatch = 3 + HasSymbolAndMatch = 3, } diff --git a/src/SharpCompress/Compressors/Deflate64/OutputWindow.cs b/src/SharpCompress/Compressors/Deflate64/OutputWindow.cs index 90b65cd3..e476e634 100644 --- a/src/SharpCompress/Compressors/Deflate64/OutputWindow.cs +++ b/src/SharpCompress/Compressors/Deflate64/OutputWindow.cs @@ -3,7 +3,6 @@ // See the LICENSE file in the project root for more information. using System; -using System.Diagnostics; namespace SharpCompress.Compressors.Deflate64; @@ -29,7 +28,6 @@ internal sealed class OutputWindow /// Add a byte to output window. public void Write(byte b) { - Debug.Assert(_bytesUsed < WINDOW_SIZE, "Can't add byte when window is full!"); _window[_end++] = b; _end &= WINDOW_MASK; ++_bytesUsed; @@ -37,8 +35,6 @@ internal sealed class OutputWindow public void WriteLengthDistance(int length, int distance) { - Debug.Assert((_bytesUsed + length) <= WINDOW_SIZE, "No Enough space"); - // move backwards distance bytes in the output stream, // and copy length bytes from this position to the output stream. _bytesUsed += length; @@ -143,10 +139,6 @@ internal sealed class OutputWindow } Array.Copy(_window, copyEnd - length, output, offset, length); _bytesUsed -= copied; - Debug.Assert( - _bytesUsed >= 0, - "check this function and find why we copied more bytes than we have" - ); return copied; } } diff --git a/src/SharpCompress/Compressors/Explode/ExplodeStream.Async.cs b/src/SharpCompress/Compressors/Explode/ExplodeStream.Async.cs new file mode 100644 index 00000000..0ed6952d --- /dev/null +++ b/src/SharpCompress/Compressors/Explode/ExplodeStream.Async.cs @@ -0,0 +1,407 @@ +using System; +using System.IO; +using System.IO.Compression; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Zip.Headers; +using SharpCompress.IO; + +namespace SharpCompress.Compressors.Explode; + +public partial class ExplodeStream +{ + internal static async ValueTask CreateAsync( + Stream inStr, + long compressedSize, + long uncompressedSize, + HeaderFlags generalPurposeBitFlag, + CancellationToken cancellationToken = default + ) + { + var ex = new ExplodeStream(inStr, compressedSize, uncompressedSize, generalPurposeBitFlag); + if (await ex.explode_SetTables_async(cancellationToken).ConfigureAwait(false) != 0) + { + throw new InvalidFormatException("ExplodeStream: invalid Huffman table data"); + } + ex.explode_var_init(); + return ex; + } + + private async ValueTask get_tree_async( + int[] arrBitLengths, + int numberExpected, + CancellationToken cancellationToken + ) + { + int inIndex = (await ReadSingleByteAsync(cancellationToken).ConfigureAwait(false)) + 1; + int outIndex = 0; + do + { + int nextByte = await ReadSingleByteAsync(cancellationToken).ConfigureAwait(false); + int bitLengthOfCodes = (nextByte & 0xf) + 1; + int numOfCodes = ((nextByte & 0xf0) >> 4) + 1; + if (outIndex + numOfCodes > numberExpected) + { + return 4; + } + + do + { + arrBitLengths[outIndex++] = bitLengthOfCodes; + } while ((--numOfCodes) != 0); + } while ((--inIndex) != 0); + + return outIndex != numberExpected ? 4 : 0; + } + + private async ValueTask ReadSingleByteAsync(CancellationToken cancellationToken) + { + var buffer = new byte[1]; + int bytesRead = await inStream + .ReadAsync(buffer, 0, 1, cancellationToken) + .ConfigureAwait(false); + if (bytesRead == 0) + { + return -1; + } + return buffer[0]; + } + + private async ValueTask explode_SetTables_async(CancellationToken cancellationToken) + { + int returnCode; + int[] arrBitLengthsForCodes = new int[256]; + + bitsForLiteralCodeTable = 0; + bitsForLengthCodeTable = 7; + bitsForDistanceCodeTable = (compressedSize) > 200000 ? 8 : 7; + + if ((generalPurposeBitFlag & HeaderFlags.Bit2) != 0) + { + bitsForLiteralCodeTable = 9; + if ( + ( + returnCode = await get_tree_async(arrBitLengthsForCodes, 256, cancellationToken) + .ConfigureAwait(false) + ) != 0 + ) + { + return returnCode; + } + + if ( + ( + returnCode = HuftTree.huftbuid( + arrBitLengthsForCodes, + 256, + 256, + [], + [], + out hufLiteralCodeTable, + ref bitsForLiteralCodeTable + ) + ) != 0 + ) + { + return returnCode; + } + + if ( + ( + returnCode = await get_tree_async(arrBitLengthsForCodes, 64, cancellationToken) + .ConfigureAwait(false) + ) != 0 + ) + { + return returnCode; + } + + if ( + ( + returnCode = HuftTree.huftbuid( + arrBitLengthsForCodes, + 64, + 0, + cplen3, + extra, + out hufLengthCodeTable, + ref bitsForLengthCodeTable + ) + ) != 0 + ) + { + return returnCode; + } + } + else + { + if ( + ( + returnCode = await get_tree_async(arrBitLengthsForCodes, 64, cancellationToken) + .ConfigureAwait(false) + ) != 0 + ) + { + return returnCode; + } + + hufLiteralCodeTable = null; + + if ( + ( + returnCode = HuftTree.huftbuid( + arrBitLengthsForCodes, + 64, + 0, + cplen2, + extra, + out hufLengthCodeTable, + ref bitsForLengthCodeTable + ) + ) != 0 + ) + { + return returnCode; + } + } + + if ( + ( + returnCode = await get_tree_async(arrBitLengthsForCodes, 64, cancellationToken) + .ConfigureAwait(false) + ) != 0 + ) + { + return (int)returnCode; + } + + if ((generalPurposeBitFlag & HeaderFlags.Bit1) != 0) + { + numOfUncodedLowerDistanceBits = 7; + returnCode = HuftTree.huftbuid( + arrBitLengthsForCodes, + 64, + 0, + cpdist8, + extra, + out hufDistanceCodeTable, + ref bitsForDistanceCodeTable + ); + } + else + { + numOfUncodedLowerDistanceBits = 6; + returnCode = HuftTree.huftbuid( + arrBitLengthsForCodes, + 64, + 0, + cpdist4, + extra, + out hufDistanceCodeTable, + ref bitsForDistanceCodeTable + ); + } + + return returnCode; + } + + private async ValueTask NeedBitsAsync(int numberOfBits, CancellationToken cancellationToken) + { + while (bitBufferCount < (numberOfBits)) + { + int byteRead = await ReadSingleByteAsync(cancellationToken).ConfigureAwait(false); + bitBuffer |= (uint)byteRead << bitBufferCount; + bitBufferCount += 8; + } + } + + private async ValueTask<(int returnCode, huftNode huftPointer, int e)> DecodeHuftAsync( + huftNode[] htab, + int bits, + uint mask, + CancellationToken cancellationToken + ) + { + await NeedBitsAsync(bits, cancellationToken).ConfigureAwait(false); + + int tabOffset = (int)(~bitBuffer & mask); + var huftPointer = htab[tabOffset]; + + while (true) + { + DumpBits(huftPointer.NumberOfBitsUsed); + int e = huftPointer.NumberOfExtraBits; + if (e <= 32) + { + return (0, huftPointer, e); + } + + if (e == INVALID_CODE) + { + return (1, huftPointer, e); + } + + e &= 31; + await NeedBitsAsync(e, cancellationToken).ConfigureAwait(false); + + tabOffset = (int)(~bitBuffer & mask_bits[e]); + huftPointer = huftPointer.ChildNodes[tabOffset]; + } + } + + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + int countIndex = 0; + while (countIndex < count && outBytesCount < unCompressedSize) + { + if (length == 0) + { + await NeedBitsAsync(1, cancellationToken).ConfigureAwait(false); + bool literal = (bitBuffer & 1) == 1; + DumpBits(1); + + huftNode huftPointer; + int extraBitLength; + if (literal) + { + byte nextByte; + if (hufLiteralCodeTable != null) + { + var literalResult = await DecodeHuftAsync( + hufLiteralCodeTable, + bitsForLiteralCodeTable, + maskForLiteralCodeTable, + cancellationToken + ) + .ConfigureAwait(false); + + if (literalResult.returnCode != 0) + { + throw new InvalidFormatException("Error decoding literal value"); + } + + huftPointer = literalResult.huftPointer; + nextByte = (byte)huftPointer.Value; + } + else + { + await NeedBitsAsync(8, cancellationToken).ConfigureAwait(false); + nextByte = (byte)bitBuffer; + DumpBits(8); + } + + buffer[offset + (countIndex++)] = nextByte; + windowsBuffer[windowIndex++] = nextByte; + outBytesCount++; + + if (windowIndex == WSIZE) + { + windowIndex = 0; + } + + continue; + } + + await NeedBitsAsync(numOfUncodedLowerDistanceBits, cancellationToken) + .ConfigureAwait(false); + distance = (int)(bitBuffer & maskForDistanceLowBits); + DumpBits(numOfUncodedLowerDistanceBits); + + var distanceResult = await DecodeHuftAsync( + hufDistanceCodeTable, + bitsForDistanceCodeTable, + maskForDistanceCodeTable, + cancellationToken + ) + .ConfigureAwait(false); + + if (distanceResult.returnCode != 0) + { + throw new InvalidFormatException("Error decoding distance high bits"); + } + + huftPointer = distanceResult.huftPointer; + distance = windowIndex - (distance + huftPointer.Value); + + var lengthResult = await DecodeHuftAsync( + hufLengthCodeTable, + bitsForLengthCodeTable, + maskForLengthCodeTable, + cancellationToken + ) + .ConfigureAwait(false); + + if (lengthResult.returnCode != 0) + { + throw new InvalidFormatException("Error decoding coded length"); + } + + huftPointer = lengthResult.huftPointer; + extraBitLength = lengthResult.e; + length = huftPointer.Value; + + if (extraBitLength != 0) + { + await NeedBitsAsync(8, cancellationToken).ConfigureAwait(false); + length += (int)(bitBuffer & 0xff); + DumpBits(8); + } + + if (length > (unCompressedSize - outBytesCount)) + { + length = (int)(unCompressedSize - outBytesCount); + } + + distance &= WSIZE - 1; + } + + while (length != 0 && countIndex < count) + { + byte nextByte = windowsBuffer[distance++]; + buffer[offset + (countIndex++)] = nextByte; + windowsBuffer[windowIndex++] = nextByte; + outBytesCount++; + + if (distance == WSIZE) + { + distance = 0; + } + + if (windowIndex == WSIZE) + { + windowIndex = 0; + } + + length--; + } + } + + return countIndex; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + if (buffer.IsEmpty || outBytesCount >= unCompressedSize) + { + return 0; + } + + byte[] arrayBuffer = new byte[buffer.Length]; + int result = await ReadAsync(arrayBuffer, 0, arrayBuffer.Length, cancellationToken) + .ConfigureAwait(false); + arrayBuffer.AsMemory(0, result).CopyTo(buffer); + return result; + } +#endif +} diff --git a/src/SharpCompress/Compressors/Explode/ExplodeStream.cs b/src/SharpCompress/Compressors/Explode/ExplodeStream.cs new file mode 100644 index 00000000..9ca53c8b --- /dev/null +++ b/src/SharpCompress/Compressors/Explode/ExplodeStream.cs @@ -0,0 +1,802 @@ +using System; +using System.IO; +using System.IO.Compression; +using SharpCompress.Common; +using SharpCompress.Common.Zip.Headers; + +namespace SharpCompress.Compressors.Explode; + +public partial class ExplodeStream : Stream +{ + private const int INVALID_CODE = 99; + private const int WSIZE = 64 * 1024; + + private readonly long unCompressedSize; + private readonly int compressedSize; + private readonly HeaderFlags generalPurposeBitFlag; + private readonly Stream inStream; + + private huftNode[]? hufLiteralCodeTable; /* literal code table */ + private huftNode[] hufLengthCodeTable = []; /* length code table */ + private huftNode[] hufDistanceCodeTable = []; /* distance code table */ + + private int bitsForLiteralCodeTable; + private int bitsForLengthCodeTable; + private int bitsForDistanceCodeTable; + private int numOfUncodedLowerDistanceBits; /* number of uncoded lower distance bits */ + + private ulong bitBuffer; + private int bitBufferCount; + + private readonly byte[] windowsBuffer; + private uint maskForLiteralCodeTable; + private uint maskForLengthCodeTable; + private uint maskForDistanceCodeTable; + private uint maskForDistanceLowBits; + private long outBytesCount; + + private int windowIndex; + private int distance; + private int length; + + private ExplodeStream( + Stream inStr, + long compressedSize, + long uncompressedSize, + HeaderFlags generalPurposeBitFlag + ) + { + inStream = inStr; + this.compressedSize = (int)compressedSize; + unCompressedSize = (long)uncompressedSize; + this.generalPurposeBitFlag = generalPurposeBitFlag; + windowsBuffer = new byte[WSIZE]; + } + + internal static ExplodeStream Create( + Stream inStr, + long compressedSize, + long uncompressedSize, + HeaderFlags generalPurposeBitFlag + ) + { + var ex = new ExplodeStream(inStr, compressedSize, uncompressedSize, generalPurposeBitFlag); + if (ex.explode_SetTables() != 0) + { + throw new InvalidFormatException("ExplodeStream: invalid Huffman table data"); + } + ex.explode_var_init(); + return ex; + } + + protected override void Dispose(bool disposing) + { + base.Dispose(disposing); + } + + public override void Flush() + { + throw new NotImplementedException(); + } + + public override long Seek(long offset, SeekOrigin origin) + { + throw new NotImplementedException(); + } + + public override void SetLength(long value) + { + throw new NotImplementedException(); + } + + public override void Write(byte[] buffer, int offset, int count) + { + throw new NotImplementedException(); + } + + public override bool CanRead => true; + public override bool CanSeek => false; + public override bool CanWrite => false; + public override long Length => unCompressedSize; + public override long Position + { + get => outBytesCount; + set { } + } + + static uint[] mask_bits = new uint[] + { + 0x0000, + 0x0001, + 0x0003, + 0x0007, + 0x000f, + 0x001f, + 0x003f, + 0x007f, + 0x00ff, + 0x01ff, + 0x03ff, + 0x07ff, + 0x0fff, + 0x1fff, + 0x3fff, + 0x7fff, + 0xffff, + }; + + /* Tables for length and distance */ + static int[] cplen2 = new int[] + { + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + }; + + static int[] cplen3 = new int[] + { + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + }; + + static int[] extra = new int[] + { + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 8, + }; + + static int[] cpdist4 = new int[] + { + 1, + 65, + 129, + 193, + 257, + 321, + 385, + 449, + 513, + 577, + 641, + 705, + 769, + 833, + 897, + 961, + 1025, + 1089, + 1153, + 1217, + 1281, + 1345, + 1409, + 1473, + 1537, + 1601, + 1665, + 1729, + 1793, + 1857, + 1921, + 1985, + 2049, + 2113, + 2177, + 2241, + 2305, + 2369, + 2433, + 2497, + 2561, + 2625, + 2689, + 2753, + 2817, + 2881, + 2945, + 3009, + 3073, + 3137, + 3201, + 3265, + 3329, + 3393, + 3457, + 3521, + 3585, + 3649, + 3713, + 3777, + 3841, + 3905, + 3969, + 4033, + }; + + static int[] cpdist8 = new int[] + { + 1, + 129, + 257, + 385, + 513, + 641, + 769, + 897, + 1025, + 1153, + 1281, + 1409, + 1537, + 1665, + 1793, + 1921, + 2049, + 2177, + 2305, + 2433, + 2561, + 2689, + 2817, + 2945, + 3073, + 3201, + 3329, + 3457, + 3585, + 3713, + 3841, + 3969, + 4097, + 4225, + 4353, + 4481, + 4609, + 4737, + 4865, + 4993, + 5121, + 5249, + 5377, + 5505, + 5633, + 5761, + 5889, + 6017, + 6145, + 6273, + 6401, + 6529, + 6657, + 6785, + 6913, + 7041, + 7169, + 7297, + 7425, + 7553, + 7681, + 7809, + 7937, + 8065, + }; + + private int get_tree(int[] arrBitLengths, int numberExpected) + /* Get the bit lengths for a code representation from the compressed + stream. If get_tree() returns 4, then there is an error in the data. + Otherwise zero is returned. */ + { + /* get bit lengths */ + int inIndex = inStream.ReadByte() + 1; /* length/count pairs to read */ + int outIndex = 0; /* next code */ + do + { + int nextByte = inStream.ReadByte(); + int bitLengthOfCodes = (nextByte & 0xf) + 1; /* bits in code (1..16) */ + int numOfCodes = ((nextByte & 0xf0) >> 4) + 1; /* codes with those bits (1..16) */ + if (outIndex + numOfCodes > numberExpected) + { + return 4; /* don't overflow arrBitLengths[] */ + } + + do + { + arrBitLengths[outIndex++] = bitLengthOfCodes; + } while ((--numOfCodes) != 0); + } while ((--inIndex) != 0); + + return outIndex != numberExpected ? 4 : 0; /* should have read numberExpected of them */ + } + + private int explode_SetTables() + { + int returnCode; /* return codes */ + int[] arrBitLengthsForCodes = new int[256]; /* bit lengths for codes */ + + bitsForLiteralCodeTable = 0; /* bits for tb */ + bitsForLengthCodeTable = 7; + bitsForDistanceCodeTable = (compressedSize) > 200000 ? 8 : 7; + + if ((generalPurposeBitFlag & HeaderFlags.Bit2) != 0) + /* With literal tree--minimum match length is 3 */ + { + bitsForLiteralCodeTable = 9; /* base table size for literals */ + if ((returnCode = get_tree(arrBitLengthsForCodes, 256)) != 0) + { + return returnCode; + } + + if ( + ( + returnCode = HuftTree.huftbuid( + arrBitLengthsForCodes, + 256, + 256, + [], + [], + out hufLiteralCodeTable, + ref bitsForLiteralCodeTable + ) + ) != 0 + ) + { + return returnCode; + } + + if ((returnCode = get_tree(arrBitLengthsForCodes, 64)) != 0) + { + return returnCode; + } + + if ( + ( + returnCode = HuftTree.huftbuid( + arrBitLengthsForCodes, + 64, + 0, + cplen3, + extra, + out hufLengthCodeTable, + ref bitsForLengthCodeTable + ) + ) != 0 + ) + { + return returnCode; + } + } + else + /* No literal tree--minimum match length is 2 */ + { + if ((returnCode = get_tree(arrBitLengthsForCodes, 64)) != 0) + { + return returnCode; + } + + hufLiteralCodeTable = null; + + if ( + ( + returnCode = HuftTree.huftbuid( + arrBitLengthsForCodes, + 64, + 0, + cplen2, + extra, + out hufLengthCodeTable, + ref bitsForLengthCodeTable + ) + ) != 0 + ) + { + return returnCode; + } + } + + if ((returnCode = get_tree(arrBitLengthsForCodes, 64)) != 0) + { + return (int)returnCode; + } + + if ((generalPurposeBitFlag & HeaderFlags.Bit1) != 0) /* true if 8K */ + { + numOfUncodedLowerDistanceBits = 7; + returnCode = HuftTree.huftbuid( + arrBitLengthsForCodes, + 64, + 0, + cpdist8, + extra, + out hufDistanceCodeTable, + ref bitsForDistanceCodeTable + ); + } + else /* else 4K */ + { + numOfUncodedLowerDistanceBits = 6; + returnCode = HuftTree.huftbuid( + arrBitLengthsForCodes, + 64, + 0, + cpdist4, + extra, + out hufDistanceCodeTable, + ref bitsForDistanceCodeTable + ); + } + + return returnCode; + } + + private void NeedBits(int numberOfBits) + { + while (bitBufferCount < (numberOfBits)) + { + bitBuffer |= (uint)inStream.ReadByte() << bitBufferCount; + bitBufferCount += 8; + } + } + + private void DumpBits(int numberOfBits) + { + bitBuffer >>= numberOfBits; + bitBufferCount -= numberOfBits; + } + + int DecodeHuft(huftNode[] htab, int bits, uint mask, out huftNode huftPointer, out int e) + { + NeedBits(bits); + + int tabOffset = (int)(~bitBuffer & mask); + huftPointer = htab[tabOffset]; + + while (true) + { + DumpBits(huftPointer.NumberOfBitsUsed); + e = huftPointer.NumberOfExtraBits; + if (e <= 32) + { + break; + } + + if (e == INVALID_CODE) + { + return 1; + } + + e &= 31; + NeedBits(e); + + tabOffset = (int)(~bitBuffer & mask_bits[e]); + huftPointer = huftPointer.ChildNodes[tabOffset]; + } + + return 0; + } + + private void explode_var_init() + { + /* explode the coded data */ + bitBuffer = 0; + bitBufferCount = 0; + maskForLiteralCodeTable = mask_bits[bitsForLiteralCodeTable]; //only used in explode_lit + maskForLengthCodeTable = mask_bits[bitsForLengthCodeTable]; + maskForDistanceCodeTable = mask_bits[bitsForDistanceCodeTable]; + maskForDistanceLowBits = mask_bits[numOfUncodedLowerDistanceBits]; + outBytesCount = 0; + + windowIndex = 0; /* initialize bit buffer, window */ + } + + public override int Read(byte[] buffer, int offset, int count) + { + int countIndex = 0; + while (countIndex < count && outBytesCount < unCompressedSize) /* do until unCompressedSize bytes uncompressed */ + { + if (length == 0) + { + NeedBits(1); + bool literal = (bitBuffer & 1) == 1; + DumpBits(1); + + huftNode huftPointer; + if (literal) /* then literal--decode it */ + { + byte nextByte; + if (hufLiteralCodeTable != null) + { + /* get coded literal */ + if ( + DecodeHuft( + hufLiteralCodeTable, + bitsForLiteralCodeTable, + maskForLiteralCodeTable, + out huftPointer, + out _ + ) != 0 + ) + { + throw new InvalidFormatException("Error decoding literal value"); + } + + nextByte = (byte)huftPointer.Value; + } + else + { + NeedBits(8); + nextByte = (byte)bitBuffer; + DumpBits(8); + } + + buffer[offset + (countIndex++)] = nextByte; + windowsBuffer[windowIndex++] = nextByte; + outBytesCount++; + + if (windowIndex == WSIZE) + { + windowIndex = 0; + } + + continue; + } + + NeedBits(numOfUncodedLowerDistanceBits); /* get distance low bits */ + distance = (int)(bitBuffer & maskForDistanceLowBits); + DumpBits(numOfUncodedLowerDistanceBits); + + /* get coded distance high bits */ + if ( + DecodeHuft( + hufDistanceCodeTable, + bitsForDistanceCodeTable, + maskForDistanceCodeTable, + out huftPointer, + out _ + ) != 0 + ) + { + throw new InvalidFormatException("Error decoding distance high bits"); + } + + distance = windowIndex - (distance + huftPointer.Value); /* construct offset */ + + /* get coded length */ + if ( + DecodeHuft( + hufLengthCodeTable, + bitsForLengthCodeTable, + maskForLengthCodeTable, + out huftPointer, + out int extraBitLength + ) != 0 + ) + { + throw new InvalidFormatException("Error decoding coded length"); + } + + length = huftPointer.Value; + + if (extraBitLength != 0) /* get length extra bits */ + { + NeedBits(8); + length += (int)(bitBuffer & 0xff); + DumpBits(8); + } + + if (length > (unCompressedSize - outBytesCount)) + { + length = (int)(unCompressedSize - outBytesCount); + } + + distance &= WSIZE - 1; + } + + while (length != 0 && countIndex < count) + { + byte nextByte = windowsBuffer[distance++]; + buffer[offset + (countIndex++)] = nextByte; + windowsBuffer[windowIndex++] = nextByte; + outBytesCount++; + + if (distance == WSIZE) + { + distance = 0; + } + + if (windowIndex == WSIZE) + { + windowIndex = 0; + } + + length--; + } + } + + return countIndex; + } +} diff --git a/src/SharpCompress/Compressors/Explode/HuftTree.cs b/src/SharpCompress/Compressors/Explode/HuftTree.cs new file mode 100644 index 00000000..05094757 --- /dev/null +++ b/src/SharpCompress/Compressors/Explode/HuftTree.cs @@ -0,0 +1,307 @@ +/* + * This code has been converted to C# based on the original huft_tree code found in + * inflate.c -- by Mark Adler version c17e, 30 Mar 2007 + */ + +namespace SharpCompress.Compressors.Explode; + +public class huftNode +{ + public int NumberOfExtraBits; /* number of extra bits or operation */ + public int NumberOfBitsUsed; /* number of bits in this code or subcode */ + public int Value; /* literal, length base, or distance base */ + public huftNode[] ChildNodes = []; /* next level of table */ +} + +public static class HuftTree +{ + private const int INVALID_CODE = 99; + + /* If BMAX needs to be larger than 16, then h and x[] should be ulg. */ + private const int BMAX = 16; /* maximum bit length of any code (16 for explode) */ + private const int N_MAX = 288; /* maximum number of codes in any set */ + + public static int huftbuid( + int[] arrBitLengthForCodes, + int numberOfCodes, + int numberOfSimpleValueCodes, + int[] arrBaseValuesForNonSimpleCodes, + int[] arrExtraBitsForNonSimpleCodes, + out huftNode[] outHufTable, + ref int outBitsForTable + ) + /* Given a list of code lengths and a maximum table size, make a set of + tables to decode that set of codes. Return zero on success, one if + the given code set is incomplete (the tables are still built in this + case), two if the input is invalid (all zero length codes or an + oversubscribed set of lengths), and three if not enough memory. + The code with value 256 is special, and the tables are constructed + so that no bits beyond that code are fetched when that code is + decoded. */ + { + outHufTable = []; + + /* Generate counts for each bit length */ + int lengthOfEOBcode = numberOfCodes > 256 ? arrBitLengthForCodes[256] : BMAX; /* set length of EOB code, if any */ + + int[] arrBitLengthCount = new int[BMAX + 1]; + for (int i = 0; i < BMAX + 1; i++) + { + arrBitLengthCount[i] = 0; + } + + int pIndex = 0; + int counterCurrentCode = numberOfCodes; + do + { + arrBitLengthCount[arrBitLengthForCodes[pIndex]]++; + pIndex++; /* assume all entries <= BMAX */ + } while ((--counterCurrentCode) != 0); + + if (arrBitLengthCount[0] == numberOfCodes) /* null input--all zero length codes */ + { + return 0; + } + + /* Find minimum and maximum length, bound *outBitsForTable by those */ + int counter; + for (counter = 1; counter <= BMAX; counter++) + { + if (arrBitLengthCount[counter] != 0) + { + break; + } + } + + int numberOfBitsInCurrentCode = counter; /* minimum code length */ + if (outBitsForTable < counter) + { + outBitsForTable = counter; + } + + for (counterCurrentCode = BMAX; counterCurrentCode != 0; counterCurrentCode--) + { + if (arrBitLengthCount[counterCurrentCode] != 0) + { + break; + } + } + + int maximumCodeLength = counterCurrentCode; /* maximum code length */ + if (outBitsForTable > counterCurrentCode) + { + outBitsForTable = counterCurrentCode; + } + + /* Adjust last length count to fill out codes, if needed */ + int numberOfDummyCodesAdded; + for ( + numberOfDummyCodesAdded = 1 << counter; + counter < counterCurrentCode; + counter++, numberOfDummyCodesAdded <<= 1 + ) + { + if ((numberOfDummyCodesAdded -= arrBitLengthCount[counter]) < 0) + { + return 2; /* bad input: more codes than bits */ + } + } + + if ((numberOfDummyCodesAdded -= arrBitLengthCount[counterCurrentCode]) < 0) + { + return 2; + } + + arrBitLengthCount[counterCurrentCode] += numberOfDummyCodesAdded; + + /* Generate starting offsets into the value table for each length */ + int[] bitOffset = new int[BMAX + 1]; + bitOffset[1] = 0; + counter = 0; + pIndex = 1; + int xIndex = 2; + while ((--counterCurrentCode) != 0) + { /* note that i == g from above */ + bitOffset[xIndex++] = (counter += arrBitLengthCount[pIndex++]); + } + + /* Make a table of values in order of bit lengths */ + int[] arrValuesInOrderOfBitLength = new int[N_MAX]; + for (int i = 0; i < N_MAX; i++) + { + arrValuesInOrderOfBitLength[i] = 0; + } + + pIndex = 0; + counterCurrentCode = 0; + do + { + if ((counter = arrBitLengthForCodes[pIndex++]) != 0) + { + arrValuesInOrderOfBitLength[bitOffset[counter]++] = counterCurrentCode; + } + } while (++counterCurrentCode < numberOfCodes); + + numberOfCodes = bitOffset[maximumCodeLength]; /* set numberOfCodes to length of v */ + + /* Generate the Huffman codes and for each, make the table entries */ + bitOffset[0] = counterCurrentCode = 0; /* first Huffman code is zero */ + pIndex = 0; /* grab values in bit order */ + int tableLevel = -1; /* no tables yet--level -1 */ + int bitsBeforeThisTable = 0; + int[] arrLX = new int[BMAX + 1]; + int stackOfBitsPerTable = 1; /* stack of bits per table */ + arrLX[stackOfBitsPerTable - 1] = 0; /* no bits decoded yet */ + + huftNode[][] arrHufTableStack = new huftNode[BMAX][]; + huftNode[] pointerToCurrentTable = []; + int numberOfEntriesInCurrentTable = 0; + + bool first = true; + + /* go through the bit lengths (k already is bits in shortest code) */ + for (; numberOfBitsInCurrentCode <= maximumCodeLength; numberOfBitsInCurrentCode++) + { + int counterForCodes = arrBitLengthCount[numberOfBitsInCurrentCode]; + while ((counterForCodes--) != 0) + { + /* here i is the Huffman code of length k bits for value *p */ + /* make tables up to required level */ + while ( + numberOfBitsInCurrentCode + > bitsBeforeThisTable + arrLX[stackOfBitsPerTable + tableLevel] + ) + { + bitsBeforeThisTable += arrLX[stackOfBitsPerTable + (tableLevel++)]; /* add bits already decoded */ + + /* compute minimum size table less than or equal to *outBitsForTable bits */ + numberOfEntriesInCurrentTable = + (numberOfEntriesInCurrentTable = maximumCodeLength - bitsBeforeThisTable) + > outBitsForTable + ? outBitsForTable + : numberOfEntriesInCurrentTable; /* upper limit */ + int fBitCounter1 = + 1 << (counter = numberOfBitsInCurrentCode - bitsBeforeThisTable); + if (fBitCounter1 > counterForCodes + 1) /* try a k-w bit table */ + { /* too few codes for k-w bit table */ + fBitCounter1 -= counterForCodes + 1; /* deduct codes from patterns left */ + xIndex = numberOfBitsInCurrentCode; + while (++counter < numberOfEntriesInCurrentTable) /* try smaller tables up to z bits */ + { + if ((fBitCounter1 <<= 1) <= arrBitLengthCount[++xIndex]) + { + break; /* enough codes to use up j bits */ + } + + fBitCounter1 -= arrBitLengthCount[xIndex]; /* else deduct codes from patterns */ + } + } + if ( + bitsBeforeThisTable + counter > lengthOfEOBcode + && bitsBeforeThisTable < lengthOfEOBcode + ) + { + counter = lengthOfEOBcode - bitsBeforeThisTable; /* make EOB code end at table */ + } + + numberOfEntriesInCurrentTable = 1 << counter; /* table entries for j-bit table */ + arrLX[stackOfBitsPerTable + tableLevel] = counter; /* set table size in stack */ + + /* allocate and link in new table */ + pointerToCurrentTable = new huftNode[numberOfEntriesInCurrentTable]; + + // set the pointer, pointed to by *outHufTable to the second huft in pointertoCurrentTable + if (first) + { + outHufTable = pointerToCurrentTable; /* link to list for huft_free() */ + first = false; + } + + arrHufTableStack[tableLevel] = pointerToCurrentTable; /* table starts after link */ + + /* connect to last table, if there is one */ + if (tableLevel != 0) + { + bitOffset[tableLevel] = counterCurrentCode; /* save pattern for backing up */ + + huftNode vHuft = new huftNode + { + NumberOfBitsUsed = arrLX[stackOfBitsPerTable + tableLevel - 1], /* bits to dump before this table */ + NumberOfExtraBits = 32 + counter, /* bits in this table */ + ChildNodes = pointerToCurrentTable, /* pointer to this table */ + }; + + counter = + (counterCurrentCode & ((1 << bitsBeforeThisTable) - 1)) + >> (bitsBeforeThisTable - arrLX[stackOfBitsPerTable + tableLevel - 1]); + arrHufTableStack[tableLevel - 1][counter] = vHuft; /* connect to last table */ + } + } + + /* set up table entry in r */ + huftNode vHuft1 = new huftNode + { + NumberOfBitsUsed = numberOfBitsInCurrentCode - bitsBeforeThisTable, + }; + + if (pIndex >= numberOfCodes) + { + vHuft1.NumberOfExtraBits = INVALID_CODE; /* out of values--invalid code */ + } + else if (arrValuesInOrderOfBitLength[pIndex] < numberOfSimpleValueCodes) + { + vHuft1.NumberOfExtraBits = ( + arrValuesInOrderOfBitLength[pIndex] < 256 ? 32 : 31 + ); /* 256 is end-of-block code */ + vHuft1.Value = arrValuesInOrderOfBitLength[pIndex++]; /* simple code is just the value */ + } + else + { + vHuft1.NumberOfExtraBits = arrExtraBitsForNonSimpleCodes[ + arrValuesInOrderOfBitLength[pIndex] - numberOfSimpleValueCodes + ]; /* non-simple--look up in lists */ + vHuft1.Value = arrBaseValuesForNonSimpleCodes[ + arrValuesInOrderOfBitLength[pIndex++] - numberOfSimpleValueCodes + ]; + } + + /* fill code-like entries with r */ + int fBitCounter2 = 1 << (numberOfBitsInCurrentCode - bitsBeforeThisTable); + for ( + counter = counterCurrentCode >> bitsBeforeThisTable; + counter < numberOfEntriesInCurrentTable; + counter += fBitCounter2 + ) + { + pointerToCurrentTable[counter] = vHuft1; + } + + /* backwards increment the k-bit code i */ + for ( + counter = 1 << (numberOfBitsInCurrentCode - 1); + (counterCurrentCode & counter) != 0; + counter >>= 1 + ) + { + counterCurrentCode ^= counter; + } + + counterCurrentCode ^= counter; + + /* backup over finished tables */ + while ( + (counterCurrentCode & ((1 << bitsBeforeThisTable) - 1)) != bitOffset[tableLevel] + ) + { + bitsBeforeThisTable -= arrLX[stackOfBitsPerTable + (--tableLevel)]; + } + } + } + + /* return actual size of base table */ + outBitsForTable = arrLX[stackOfBitsPerTable]; + + /* Return true (1) if we were given an incomplete table */ + return (numberOfDummyCodesAdded != 0 && maximumCodeLength != 1) ? 1 : 0; + } +} diff --git a/src/SharpCompress/Compressors/Filters/BCJ2Filter.cs b/src/SharpCompress/Compressors/Filters/BCJ2Filter.cs index f658b385..1ed531cc 100644 --- a/src/SharpCompress/Compressors/Filters/BCJ2Filter.cs +++ b/src/SharpCompress/Compressors/Filters/BCJ2Filter.cs @@ -1,4 +1,4 @@ -using System; +using System; using System.IO; namespace SharpCompress.Compressors.Filters; @@ -79,7 +79,7 @@ internal class BCJ2Filter : Stream public override bool CanWrite => false; - public override void Flush() => throw new NotSupportedException(); + public override void Flush() { } public override long Length => _baseStream.Length + _data1.Length + _data2.Length; diff --git a/src/SharpCompress/Compressors/Filters/BCJFilter.cs b/src/SharpCompress/Compressors/Filters/BCJFilter.cs index 83a93397..e7f00c98 100644 --- a/src/SharpCompress/Compressors/Filters/BCJFilter.cs +++ b/src/SharpCompress/Compressors/Filters/BCJFilter.cs @@ -13,7 +13,7 @@ internal class BCJFilter : Filter true, false, false, - false + false, }; private static readonly int[] MASK_TO_BIT_NUMBER = { 0, 1, 2, 2, 3, 3, 3, 3 }; diff --git a/src/SharpCompress/Compressors/Filters/BCJFilterARM.cs b/src/SharpCompress/Compressors/Filters/BCJFilterARM.cs index 86e90b10..5d861621 100644 --- a/src/SharpCompress/Compressors/Filters/BCJFilterARM.cs +++ b/src/SharpCompress/Compressors/Filters/BCJFilterARM.cs @@ -18,7 +18,7 @@ internal class BCJFilterARM : Filter { if ((buffer[i + 3] & 0xFF) == 0xEB) { - int src = + var src = ((buffer[i + 2] & 0xFF) << 16) | ((buffer[i + 1] & 0xFF) << 8) | (buffer[i] & 0xFF); diff --git a/src/SharpCompress/Compressors/Filters/BCJFilterARM64.cs b/src/SharpCompress/Compressors/Filters/BCJFilterARM64.cs new file mode 100644 index 00000000..c701a5b5 --- /dev/null +++ b/src/SharpCompress/Compressors/Filters/BCJFilterARM64.cs @@ -0,0 +1,69 @@ +using System; +using System.Buffers.Binary; +using System.IO; + +namespace SharpCompress.Compressors.Filters; + +internal class BCJFilterARM64 : Filter +{ + private int _pos; + + public BCJFilterARM64(bool isEncoder, Stream baseStream) + : base(isEncoder, baseStream, 8) => _pos = 0; + + protected override int Transform(byte[] buffer, int offset, int count) + { + var end = offset + count - 4; + int i; + + for (i = offset; i <= end; i += 4) + { + uint pc = (uint)(_pos + i - offset); + uint instr = BinaryPrimitives.ReadUInt32LittleEndian( + new ReadOnlySpan(buffer, i, 4) + ); + + if ((instr >> 26) == 0x25) + { + uint src = instr; + instr = 0x94000000; + + pc >>= 2; + if (!_isEncoder) + { + pc = 0U - pc; + } + + instr |= (src + pc) & 0x03FFFFFF; + BinaryPrimitives.WriteUInt32LittleEndian(new Span(buffer, i, 4), instr); + } + else if ((instr & 0x9F000000) == 0x90000000) + { + uint src = ((instr >> 29) & 3) | ((instr >> 3) & 0x001FFFFC); + + if (((src + 0x00020000) & 0x001C0000) != 0) + { + continue; + } + + instr &= 0x9000001F; + + pc >>= 12; + if (!_isEncoder) + { + pc = 0U - pc; + } + + uint dest = src + pc; + instr |= (dest & 3) << 29; + instr |= (dest & 0x0003FFFC) << 3; + instr |= (0U - (dest & 0x00020000)) & 0x00E00000; + BinaryPrimitives.WriteUInt32LittleEndian(new Span(buffer, i, 4), instr); + } + } + + i -= offset; + _pos += i; + return i; + } +} diff --git a/src/SharpCompress/Compressors/Filters/BCJFilterARMT.cs b/src/SharpCompress/Compressors/Filters/BCJFilterARMT.cs index d2ad1bbf..89954f56 100644 --- a/src/SharpCompress/Compressors/Filters/BCJFilterARMT.cs +++ b/src/SharpCompress/Compressors/Filters/BCJFilterARMT.cs @@ -18,7 +18,7 @@ internal class BCJFilterARMT : Filter { if ((buffer[i + 1] & 0xF8) == 0xF0 && (buffer[i + 3] & 0xF8) == 0xF8) { - int src = + var src = ((buffer[i + 1] & 0x07) << 19) | ((buffer[i] & 0xFF) << 11) | ((buffer[i + 3] & 0x07) << 8) @@ -27,9 +27,13 @@ internal class BCJFilterARMT : Filter int dest; if (_isEncoder) + { dest = src + (_pos + i - offset); + } else + { dest = src - (_pos + i - offset); + } dest >>>= 1; buffer[i + 1] = (byte)(0xF0 | ((dest >>> 19) & 0x07)); diff --git a/src/SharpCompress/Compressors/Filters/BCJFilterIA64.cs b/src/SharpCompress/Compressors/Filters/BCJFilterIA64.cs index 4d89ba39..80d308a3 100644 --- a/src/SharpCompress/Compressors/Filters/BCJFilterIA64.cs +++ b/src/SharpCompress/Compressors/Filters/BCJFilterIA64.cs @@ -39,7 +39,7 @@ internal class BCJFilterIA64 : Filter 4, 4, 0, - 0 + 0, }; public BCJFilterIA64(bool isEncoder, Stream baseStream) @@ -52,37 +52,45 @@ internal class BCJFilterIA64 : Filter for (i = offset; i <= end; i += 16) { - int instrTemplate = buffer[i] & 0x1F; - int mask = BRANCH_TABLE[instrTemplate]; + var instrTemplate = buffer[i] & 0x1F; + var mask = BRANCH_TABLE[instrTemplate]; for (int slot = 0, bitPos = 5; slot < 3; ++slot, bitPos += 41) { if (((mask >>> slot) & 1) == 0) + { continue; + } - int bytePos = bitPos >>> 3; - int bitRes = bitPos & 7; + var bytePos = bitPos >>> 3; + var bitRes = bitPos & 7; long instr = 0; - for (int j = 0; j < 6; ++j) + for (var j = 0; j < 6; ++j) { instr |= (buffer[i + bytePos + j] & 0xFFL) << (8 * j); } - long instrNorm = instr >>> bitRes; + var instrNorm = instr >>> bitRes; if (((instrNorm >>> 37) & 0x0F) != 0x05 || ((instrNorm >>> 9) & 0x07) != 0x00) + { continue; + } - int src = (int)((instrNorm >>> 13) & 0x0FFFFF); + var src = (int)((instrNorm >>> 13) & 0x0FFFFF); src |= ((int)(instrNorm >>> 36) & 1) << 20; src <<= 4; int dest; if (_isEncoder) + { dest = src + (_pos + i - offset); + } else + { dest = src - (_pos + i - offset); + } dest >>>= 4; @@ -93,7 +101,7 @@ internal class BCJFilterIA64 : Filter instr &= (1 << bitRes) - 1; instr |= instrNorm << bitRes; - for (int j = 0; j < 6; ++j) + for (var j = 0; j < 6; ++j) { buffer[i + bytePos + j] = (byte)(instr >>> (8 * j)); } diff --git a/src/SharpCompress/Compressors/Filters/BCJFilterPPC.cs b/src/SharpCompress/Compressors/Filters/BCJFilterPPC.cs index 11ed61a1..ccfa7480 100644 --- a/src/SharpCompress/Compressors/Filters/BCJFilterPPC.cs +++ b/src/SharpCompress/Compressors/Filters/BCJFilterPPC.cs @@ -18,7 +18,7 @@ internal class BCJFilterPPC : Filter { if ((buffer[i] & 0xFC) == 0x48 && (buffer[i + 3] & 0x03) == 0x01) { - int src = + var src = ((buffer[i] & 0x03) << 24) | ((buffer[i + 1] & 0xFF) << 16) | ((buffer[i + 2] & 0xFF) << 8) diff --git a/src/SharpCompress/Compressors/Filters/BCJFilterRISCV.cs b/src/SharpCompress/Compressors/Filters/BCJFilterRISCV.cs new file mode 100644 index 00000000..32ad6ea2 --- /dev/null +++ b/src/SharpCompress/Compressors/Filters/BCJFilterRISCV.cs @@ -0,0 +1,214 @@ +using System; +using System.Buffers.Binary; +using System.IO; + +namespace SharpCompress.Compressors.Filters; + +internal class BCJFilterRISCV : Filter +{ + private int _pos; + + public BCJFilterRISCV(bool isEncoder, Stream baseStream) + : base(isEncoder, baseStream, 8) => _pos = 0; + + private int Decode(byte[] buffer, int offset, int count) + { + if (count < 8) + { + return 0; + } + + var end = offset + count - 8; + int i; + for (i = offset; i <= end; i += 2) + { + uint inst = buffer[i]; + if (inst == 0xEF) + { + uint b1 = buffer[i + 1]; + if ((b1 & 0x0D) != 0) + { + continue; + } + + uint b2 = buffer[i + 2]; + uint b3 = buffer[i + 3]; + uint pc = (uint)(_pos + i); + + uint addr = ((b1 & 0xF0) << 13) | (b2 << 9) | (b3 << 1); + + addr -= pc; + + buffer[i + 1] = (byte)((b1 & 0x0F) | ((addr >> 8) & 0xF0)); + + buffer[i + 2] = (byte)( + ((addr >> 16) & 0x0F) | ((addr >> 7) & 0x10) | ((addr << 4) & 0xE0) + ); + + buffer[i + 3] = (byte)(((addr >> 4) & 0x7F) | ((addr >> 13) & 0x80)); + + i += 4 - 2; + } + else if ((inst & 0x7F) == 0x17) + { + uint inst2 = 0; + inst |= (uint)buffer[i + 1] << 8; + inst |= (uint)buffer[i + 2] << 16; + inst |= (uint)buffer[i + 3] << 24; + + if ((inst & 0xE80) != 0) + { + inst2 = BinaryPrimitives.ReadUInt32LittleEndian( + new ReadOnlySpan(buffer, i + 4, 4) + ); + if (((((inst) << 8) ^ (inst2)) & 0xF8003) != 3) + { + i += 6 - 2; + continue; + } + uint addr = inst & 0xFFFFF000; + addr += inst2 >> 20; + + inst = 0x17 | (2 << 7) | (inst2 << 12); + inst2 = addr; + } + else + { + uint inst2_rs1 = inst >> 27; + if ((uint)(((inst) - 0x3117) << 18) >= ((inst2_rs1) & 0x1D)) + { + i += 4 - 2; + continue; + } + + uint addr = BinaryPrimitives.ReadUInt32BigEndian( + new ReadOnlySpan(buffer, i + 4, 4) + ); + + addr -= (uint)(_pos + i); + + inst2 = (inst >> 12) | (addr << 20); + + inst = 0x17 | (inst2_rs1 << 7) | ((addr + 0x800) & 0xFFFFF000); + } + BinaryPrimitives.WriteUInt32LittleEndian(new Span(buffer, i, 4), inst); + BinaryPrimitives.WriteUInt32LittleEndian(new Span(buffer, i + 4, 4), inst2); + + i += 8 - 2; + } + } + i -= offset; + _pos += i; + return i; + } + + private int Encode(byte[] buffer, int offset, int count) + { + if (count < 8) + { + return 0; + } + + var end = offset + count - 8; + int i; + for (i = offset; i <= end; i += 2) + { + uint inst = buffer[i]; + if (inst == 0xEF) + { + uint b1 = buffer[i + 1]; + if ((b1 & 0x0D) != 0) + { + continue; + } + + uint b2 = buffer[i + 2]; + uint b3 = buffer[i + 3]; + uint pc = (uint)(_pos + i); + + uint addr = + ((b1 & 0xF0) << 8) + | ((b2 & 0x0F) << 16) + | ((b2 & 0x10) << 7) + | ((b2 & 0xE0) >> 4) + | ((b3 & 0x7F) << 4) + | ((b3 & 0x80) << 13); + + addr += pc; + + buffer[i + 1] = (byte)((b1 & 0x0F) | ((addr >> 13) & 0xF0)); + + buffer[i + 2] = (byte)(addr >> 9); + + buffer[i + 3] = (byte)(addr >> 1); + + i += 4 - 2; + } + else if ((inst & 0x7F) == 0x17) + { + inst |= (uint)buffer[i + 1] << 8; + inst |= (uint)buffer[i + 2] << 16; + inst |= (uint)buffer[i + 3] << 24; + + if ((inst & 0xE80) != 0) + { + uint inst2 = BinaryPrimitives.ReadUInt32LittleEndian( + new ReadOnlySpan(buffer, i + 4, 4) + ); + if (((((inst) << 8) ^ (inst2)) & 0xF8003) != 3) + { + i += 6 - 2; + continue; + } + uint addr = inst & 0xFFFFF000; + addr += (inst2 >> 20) - ((inst2 >> 19) & 0x1000); + + addr += (uint)(_pos + i); + inst = 0x17 | (2 << 7) | (inst2 << 12); + + BinaryPrimitives.WriteUInt32LittleEndian(new Span(buffer, i, 4), inst); + BinaryPrimitives.WriteUInt32BigEndian(new Span(buffer, i + 4, 4), addr); + } + else + { + uint fake_rs1 = inst >> 27; + if ((uint)(((inst) - 0x3117) << 18) >= ((fake_rs1) & 0x1D)) + { + i += 4 - 2; + continue; + } + + uint fake_addr = BinaryPrimitives.ReadUInt32LittleEndian( + new ReadOnlySpan(buffer, i + 4, 4) + ); + + uint fake_inst2 = (inst >> 12) | (fake_addr << 20); + + inst = 0x17 | (fake_rs1 << 7) | (fake_addr & 0xFFFFF000); + + BinaryPrimitives.WriteUInt32LittleEndian(new Span(buffer, i, 4), inst); + BinaryPrimitives.WriteUInt32LittleEndian( + new Span(buffer, i + 4, 4), + fake_inst2 + ); + } + i += 8 - 2; + } + } + i -= offset; + _pos += i; + return i; + } + + protected override int Transform(byte[] buffer, int offset, int count) + { + if (_isEncoder) + { + return Encode(buffer, offset, count); + } + else + { + return Decode(buffer, offset, count); + } + } +} diff --git a/src/SharpCompress/Compressors/Filters/BCJFilterSPARC.cs b/src/SharpCompress/Compressors/Filters/BCJFilterSPARC.cs index 67756d34..db7c75be 100644 --- a/src/SharpCompress/Compressors/Filters/BCJFilterSPARC.cs +++ b/src/SharpCompress/Compressors/Filters/BCJFilterSPARC.cs @@ -21,7 +21,7 @@ internal class BCJFilterSPARC : Filter || (buffer[i] == 0x7F && (buffer[i + 1] & 0xC0) == 0xC0) ) { - int src = + var src = ((buffer[i] & 0xFF) << 24) | ((buffer[i + 1] & 0xFF) << 16) | ((buffer[i + 2] & 0xFF) << 8) diff --git a/src/SharpCompress/Compressors/Filters/BranchExecFilter.cs b/src/SharpCompress/Compressors/Filters/BranchExecFilter.cs index 9fc439d4..d14b8564 100644 --- a/src/SharpCompress/Compressors/Filters/BranchExecFilter.cs +++ b/src/SharpCompress/Compressors/Filters/BranchExecFilter.cs @@ -5,8 +5,8 @@ */ using System; -using System.IO; using System.Runtime.CompilerServices; +using SharpCompress.Common; namespace SharpCompress.Compressors.Filters; @@ -18,55 +18,46 @@ public sealed class BranchExecFilter ARCH_x86_ALIGNMENT = 1, ARCH_PowerPC_ALIGNMENT = 4, ARCH_IA64_ALIGNMENT = 16, - ARCH_ARM_ALIGNMENT = 4, + ARCH_ARM_ALIGNMENT = ARCH_PowerPC_ALIGNMENT, ARCH_ARMTHUMB_ALIGNMENT = 2, - ARCH_SPARC_ALIGNMENT = 4, + ARCH_SPARC_ALIGNMENT = ARCH_PowerPC_ALIGNMENT, } [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static bool X86TestByte(byte b) - { - return b == 0x00 || b == 0xFF; - } + private static bool X86TestByte(byte b) => b == 0x00 || b == 0xFF; //Replaced X86Converter with bcj_x86() - https://github.com/torvalds/linux/blob/master/lib/xz/xz_dec_bcj.c //This was to fix an issue decoding a Test zip made with WinZip (that 7zip was also able to read). //The previous version of the code would corrupt 2 bytes in the Test.exe at 0x6CF9 (3D6D - should be 4000) - Test zip: WinZip27.Xz.zipx public static void X86Converter(byte[] buf, uint ip, ref uint state) { - bool[] mask_to_allowed_status = new[] - { - true, - true, - true, - false, - true, - false, - false, - false - }; + var mask_to_allowed_status = new[] { true, true, true, false, true, false, false, false }; - byte[] mask_to_bit_num = new byte[] { 0, 1, 2, 2, 3, 3, 3, 3 }; + var mask_to_bit_num = new byte[] { 0, 1, 2, 2, 3, 3, 3, 3 }; int i; - int prev_pos = -1; - uint prev_mask = state & 7; + var prev_pos = -1; + var prev_mask = state & 7; uint src; uint dest; uint j; byte b; - uint pos = ip; + var pos = ip; - uint size = (uint)buf.Length; + var size = (uint)buf.Length; if (size <= 4) + { return; + } size -= 4; for (i = 0; i < size; ++i) { if ((buf[i] & 0xFE) != 0xE8) + { continue; + } prev_pos = i - prev_pos; if (prev_pos > 3) @@ -102,12 +93,16 @@ public sealed class BranchExecFilter { dest = src - (pos + (uint)i + 5); if (prev_mask == 0) + { break; + } j = mask_to_bit_num[prev_mask] * 8u; b = (byte)(dest >> (24 - (int)j)); if (!X86TestByte(b)) + { break; + } src = dest ^ ((1u << (32 - (int)j)) - 1u); } @@ -257,7 +252,7 @@ public sealed class BranchExecFilter long size = data.Length; if (size < 16) { - throw new InvalidDataException("Unexpected data size"); + throw new InvalidFormatException("Unexpected data size"); } size -= 16; diff --git a/src/SharpCompress/Compressors/Filters/DeltaFilter.cs b/src/SharpCompress/Compressors/Filters/DeltaFilter.cs index c5fbeb41..82a76ae9 100644 --- a/src/SharpCompress/Compressors/Filters/DeltaFilter.cs +++ b/src/SharpCompress/Compressors/Filters/DeltaFilter.cs @@ -1,37 +1,34 @@ -using System; using System.IO; -namespace SharpCompress.Compressors.Filters +namespace SharpCompress.Compressors.Filters; + +internal class DeltaFilter : Filter { - internal class DeltaFilter : Filter + private const int DISTANCE_MAX = 256; + private const int DISTANCE_MASK = DISTANCE_MAX - 1; + + private int _distance; + private byte[] _history; + private int _position; + + public DeltaFilter(bool isEncoder, Stream baseStream, byte[] info) + : base(isEncoder, baseStream, 1) { - private const int DISTANCE_MIN = 1; - private const int DISTANCE_MAX = 256; - private const int DISTANCE_MASK = DISTANCE_MAX - 1; + _distance = info[0]; + _history = new byte[DISTANCE_MAX]; + _position = 0; + } - private int _distance; - private byte[] _history; - private int _position; + protected override int Transform(byte[] buffer, int offset, int count) + { + var end = offset + count; - public DeltaFilter(bool isEncoder, Stream baseStream, byte[] info) - : base(isEncoder, baseStream, 1) + for (var i = offset; i < end; i++) { - _distance = info[0]; - _history = new byte[DISTANCE_MAX]; - _position = 0; + buffer[i] += _history[(_distance + _position--) & DISTANCE_MASK]; + _history[_position & DISTANCE_MASK] = buffer[i]; } - protected override int Transform(byte[] buffer, int offset, int count) - { - int end = offset + count; - - for (int i = offset; i < end; i++) - { - buffer[i] += _history[(_distance + _position--) & DISTANCE_MASK]; - _history[_position & DISTANCE_MASK] = buffer[i]; - } - - return count; - } + return count; } } diff --git a/src/SharpCompress/Compressors/Filters/Filter.Async.cs b/src/SharpCompress/Compressors/Filters/Filter.Async.cs new file mode 100644 index 00000000..2fd03d37 --- /dev/null +++ b/src/SharpCompress/Compressors/Filters/Filter.Async.cs @@ -0,0 +1,228 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Compressors.Filters; + +internal abstract partial class Filter +{ + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + var size = 0; + + if (_transformed > 0) + { + var copySize = _transformed; + if (copySize > count) + { + copySize = count; + } + Buffer.BlockCopy(_tail, 0, buffer, offset, copySize); + _transformed -= copySize; + _read -= copySize; + offset += copySize; + count -= copySize; + size += copySize; + Buffer.BlockCopy(_tail, copySize, _tail, 0, _read); + } + if (count == 0) + { + return size; + } + + var inSize = _read; + if (inSize > count) + { + inSize = count; + } + Buffer.BlockCopy(_tail, 0, buffer, offset, inSize); + _read -= inSize; + Buffer.BlockCopy(_tail, inSize, _tail, 0, _read); + while (!_endReached && inSize < count) + { + var baseRead = await _baseStream + .ReadAsync(buffer, offset + inSize, count - inSize, cancellationToken) + .ConfigureAwait(false); + inSize += baseRead; + if (baseRead == 0) + { + _endReached = true; + } + } + while (!_endReached && _read < _tail.Length) + { + var baseRead = await _baseStream + .ReadAsync(_tail, _read, _tail.Length - _read, cancellationToken) + .ConfigureAwait(false); + _read += baseRead; + if (baseRead == 0) + { + _endReached = true; + } + } + + if (inSize > _tail.Length) + { + _transformed = Transform(buffer, offset, inSize); + offset += _transformed; + count -= _transformed; + size += _transformed; + inSize -= _transformed; + _transformed = 0; + } + + if (count == 0) + { + return size; + } + + Buffer.BlockCopy(buffer, offset, _window, 0, inSize); + Buffer.BlockCopy(_tail, 0, _window, inSize, _read); + if (inSize + _read > _tail.Length) + { + _transformed = Transform(_window, 0, inSize + _read); + } + else + { + _transformed = inSize + _read; + } + Buffer.BlockCopy(_window, 0, buffer, offset, inSize); + Buffer.BlockCopy(_window, inSize, _tail, 0, _read); + size += inSize; + _transformed -= inSize; + + return size; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + var size = 0; + var offset = 0; + var count = buffer.Length; + + if (_transformed > 0) + { + var copySize = _transformed; + if (copySize > count) + { + copySize = count; + } + _tail.AsSpan(0, copySize).CopyTo(buffer.Span.Slice(offset, copySize)); + _transformed -= copySize; + _read -= copySize; + offset += copySize; + count -= copySize; + size += copySize; + Buffer.BlockCopy(_tail, copySize, _tail, 0, _read); + } + if (count == 0) + { + return size; + } + + var inSize = _read; + if (inSize > count) + { + inSize = count; + } + _tail.AsSpan(0, inSize).CopyTo(buffer.Span.Slice(offset, inSize)); + _read -= inSize; + Buffer.BlockCopy(_tail, inSize, _tail, 0, _read); + while (!_endReached && inSize < count) + { + var baseRead = await _baseStream + .ReadAsync(buffer.Slice(offset + inSize, count - inSize), cancellationToken) + .ConfigureAwait(false); + inSize += baseRead; + if (baseRead == 0) + { + _endReached = true; + } + } + while (!_endReached && _read < _tail.Length) + { + var baseRead = await _baseStream + .ReadAsync(_tail.AsMemory(_read, _tail.Length - _read), cancellationToken) + .ConfigureAwait(false); + _read += baseRead; + if (baseRead == 0) + { + _endReached = true; + } + } + + if (inSize > _tail.Length) + { + // Transform operates in-place on a temporary array + var arrayBuffer = buffer.Slice(offset, inSize).ToArray(); + _transformed = Transform(arrayBuffer, 0, inSize); + // Copy transformed bytes back to the original buffer + arrayBuffer.AsSpan(0, inSize).CopyTo(buffer.Span.Slice(offset, inSize)); + offset += _transformed; + count -= _transformed; + size += _transformed; + inSize -= _transformed; + _transformed = 0; + } + + if (count == 0) + { + return size; + } + + var inputBytes = buffer.Slice(offset, inSize).ToArray(); + Buffer.BlockCopy(inputBytes, 0, _window, 0, inSize); + Buffer.BlockCopy(_tail, 0, _window, inSize, _read); + if (inSize + _read > _tail.Length) + { + _transformed = Transform(_window, 0, inSize + _read); + } + else + { + _transformed = inSize + _read; + } + _window.AsSpan(0, inSize).CopyTo(buffer.Span.Slice(offset, inSize)); + _window.AsSpan(inSize, _read).CopyTo(_tail.AsSpan()); + size += inSize; + _transformed -= inSize; + + return size; + } +#endif + + public override async Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + Transform(buffer, offset, count); + await _baseStream + .WriteAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + } + +#if !LEGACY_DOTNET + public override async ValueTask WriteAsync( + ReadOnlyMemory buffer, + CancellationToken cancellationToken = default + ) + { + // Transform is synchronous and requires byte[] + var array = buffer.ToArray(); + Transform(array, 0, array.Length); + await _baseStream.WriteAsync(array, cancellationToken).ConfigureAwait(false); + } +#endif +} diff --git a/src/SharpCompress/Compressors/Filters/Filter.cs b/src/SharpCompress/Compressors/Filters/Filter.cs index 93d95195..8841d618 100644 --- a/src/SharpCompress/Compressors/Filters/Filter.cs +++ b/src/SharpCompress/Compressors/Filters/Filter.cs @@ -1,9 +1,9 @@ -using System; +using System; using System.IO; namespace SharpCompress.Compressors.Filters; -internal abstract class Filter : Stream +internal abstract partial class Filter : Stream { protected bool _isEncoder; protected Stream _baseStream; @@ -40,7 +40,7 @@ internal abstract class Filter : Stream public override bool CanWrite => _isEncoder; - public override void Flush() => throw new NotSupportedException(); + public override void Flush() { } public override long Length => _baseStream.Length; diff --git a/src/SharpCompress/Compressors/LZMA/AesDecoderStream.Async.cs b/src/SharpCompress/Compressors/LZMA/AesDecoderStream.Async.cs new file mode 100644 index 00000000..c76f726f --- /dev/null +++ b/src/SharpCompress/Compressors/LZMA/AesDecoderStream.Async.cs @@ -0,0 +1,76 @@ +using System; +using System.IO; +using System.Security.Cryptography; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Compressors.LZMA; + +internal sealed partial class AesDecoderStream +{ + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken = default + ) + { + if (count == 0 || mWritten == mLimit) + { + return 0; + } + + if (mUnderflow > 0) + { + return HandleUnderflow(buffer, offset, count); + } + + // Need at least 16 bytes to proceed. + if (mEnding - mOffset < 16) + { + Buffer.BlockCopy(mBuffer, mOffset, mBuffer, 0, mEnding - mOffset); + mEnding -= mOffset; + mOffset = 0; + + do + { + cancellationToken.ThrowIfCancellationRequested(); + var read = await mStream + .ReadAsync(mBuffer, mEnding, mBuffer.Length - mEnding, cancellationToken) + .ConfigureAwait(false); + if (read == 0) + { + // We are not done decoding and have less than 16 bytes. + throw new IncompleteArchiveException("Unexpected end of stream."); + } + + mEnding += read; + } while (mEnding - mOffset < 16); + } + + // We shouldn't return more data than we are limited to. + if (count > mLimit - mWritten) + { + count = (int)(mLimit - mWritten); + } + + // We cannot transform less than 16 bytes into the target buffer, + // but we also cannot return zero, so we need to handle this. + if (count < 16) + { + return HandleUnderflow(buffer, offset, count); + } + + if (count > mEnding - mOffset) + { + count = mEnding - mOffset; + } + + // Otherwise we transform directly into the target buffer. + var processed = mDecoder.TransformBlock(mBuffer, mOffset, count & ~15, buffer, offset); + mOffset += processed; + mWritten += processed; + return processed; + } +} diff --git a/src/SharpCompress/Compressors/LZMA/AesDecoderStream.cs b/src/SharpCompress/Compressors/LZMA/AesDecoderStream.cs index 24823c73..52793292 100644 --- a/src/SharpCompress/Compressors/LZMA/AesDecoderStream.cs +++ b/src/SharpCompress/Compressors/LZMA/AesDecoderStream.cs @@ -2,11 +2,14 @@ using System; using System.IO; using System.Security.Cryptography; using System.Text; -using SharpCompress.Compressors.LZMA.Utilites; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Compressors.LZMA.Utilities; namespace SharpCompress.Compressors.LZMA; -internal sealed class AesDecoderStream : DecoderStream2 +internal sealed partial class AesDecoderStream : DecoderStream2 { private readonly Stream mStream; private readonly ICryptoTransform mDecoder; @@ -20,7 +23,8 @@ internal sealed class AesDecoderStream : DecoderStream2 public AesDecoderStream(Stream input, byte[] info, IPasswordProvider pass, long limit) { - if (pass.CryptoGetTextPassword() == null) + var password = pass.CryptoGetTextPassword(); + if (password == null) { throw new SharpCompress.Common.CryptographicException( "Encrypted 7Zip archive has no password specified." @@ -35,13 +39,13 @@ internal sealed class AesDecoderStream : DecoderStream2 throw new NotSupportedException("AES decoder does not support padding."); } - Init(info, out int numCyclesPower, out byte[] salt, out byte[] seed); + Init(info, out var numCyclesPower, out var salt, out var seed); - byte[] password = Encoding.Unicode.GetBytes(pass.CryptoGetTextPassword()); - byte[]? key = InitKey(numCyclesPower, salt, password); + var passwordBytes = Encoding.Unicode.GetBytes(password); + var key = InitKey(numCyclesPower, salt, passwordBytes); if (key == null) { - throw new InvalidOperationException("Initialized with null key"); + throw new ArchiveOperationException("Initialized with null key"); } using (var aes = Aes.Create()) @@ -100,11 +104,11 @@ internal sealed class AesDecoderStream : DecoderStream2 do { - int read = mStream.Read(mBuffer, mEnding, mBuffer.Length - mEnding); + var read = mStream.Read(mBuffer, mEnding, mBuffer.Length - mEnding); if (read == 0) { // We are not done decoding and have less than 16 bytes. - throw new EndOfStreamException(); + throw new IncompleteArchiveException("Unexpected end of stream."); } mEnding += read; @@ -133,7 +137,7 @@ internal sealed class AesDecoderStream : DecoderStream2 } // Otherwise we transform directly into the target buffer. - int processed = mDecoder.TransformBlock(mBuffer, mOffset, count & ~15, buffer, offset); + var processed = mDecoder.TransformBlock(mBuffer, mOffset, count & ~15, buffer, offset); mOffset += processed; mWritten += processed; return processed; @@ -143,7 +147,7 @@ internal sealed class AesDecoderStream : DecoderStream2 private void Init(byte[] info, out int numCyclesPower, out byte[] salt, out byte[] iv) { - byte bt = info[0]; + var bt = info[0]; numCyclesPower = bt & 0x3F; if ((bt & 0xC0) == 0) @@ -153,29 +157,29 @@ internal sealed class AesDecoderStream : DecoderStream2 return; } - int saltSize = (bt >> 7) & 1; - int ivSize = (bt >> 6) & 1; + var saltSize = (bt >> 7) & 1; + var ivSize = (bt >> 6) & 1; if (info.Length == 1) { - throw new InvalidOperationException(); + throw new ArchiveOperationException(); } - byte bt2 = info[1]; + var bt2 = info[1]; saltSize += (bt2 >> 4); ivSize += (bt2 & 15); if (info.Length < 2 + saltSize + ivSize) { - throw new InvalidOperationException(); + throw new ArchiveOperationException(); } salt = new byte[saltSize]; - for (int i = 0; i < saltSize; i++) + for (var i = 0; i < saltSize; i++) { salt[i] = info[i + 2]; } iv = new byte[16]; - for (int i = 0; i < ivSize; i++) + for (var i = 0; i < ivSize; i++) { iv[i] = info[i + saltSize + 2]; } @@ -198,7 +202,7 @@ internal sealed class AesDecoderStream : DecoderStream2 key[pos] = salt[pos]; } - for (int i = 0; i < pass.Length && pos < 32; i++) + for (var i = 0; i < pass.Length && pos < 32; i++) { key[pos++] = pass[i]; } @@ -207,31 +211,9 @@ internal sealed class AesDecoderStream : DecoderStream2 } else { -#if NETSTANDARD2_0 - using IncrementalHash sha = IncrementalHash.CreateHash(HashAlgorithmName.SHA256); - byte[] counter = new byte[8]; - long numRounds = 1L << mNumCyclesPower; - for (long round = 0; round < numRounds; round++) - { - sha.AppendData(salt, 0, salt.Length); - sha.AppendData(pass, 0, pass.Length); - sha.AppendData(counter, 0, 8); - - // This mirrors the counter so we don't have to convert long to byte[] each round. - // (It also ensures the counter is little endian, which BitConverter does not.) - for (int i = 0; i < 8; i++) - { - if (++counter[i] != 0) - { - break; - } - } - } - return sha.GetHashAndReset(); -#else using var sha = SHA256.Create(); - byte[] counter = new byte[8]; - long numRounds = 1L << mNumCyclesPower; + var counter = new byte[8]; + var numRounds = 1L << mNumCyclesPower; for (long round = 0; round < numRounds; round++) { sha.TransformBlock(salt, 0, salt.Length, null, 0); @@ -240,7 +222,7 @@ internal sealed class AesDecoderStream : DecoderStream2 // This mirrors the counter so we don't have to convert long to byte[] each round. // (It also ensures the counter is little endian, which BitConverter does not.) - for (int i = 0; i < 8; i++) + for (var i = 0; i < 8; i++) { if (++counter[i] != 0) { @@ -251,7 +233,6 @@ internal sealed class AesDecoderStream : DecoderStream2 sha.TransformFinalBlock(counter, 0, 0); return sha.Hash; -#endif } } @@ -261,7 +242,7 @@ internal sealed class AesDecoderStream : DecoderStream2 // Just transform as much as possible so we can feed from it as long as possible. if (mUnderflow == 0) { - int blockSize = (mEnding - mOffset) & ~15; + var blockSize = (mEnding - mOffset) & ~15; mUnderflow = mDecoder.TransformBlock(mBuffer, mOffset, blockSize, mBuffer, mOffset); } diff --git a/src/SharpCompress/Compressors/LZMA/Bcj2DecoderStream.cs b/src/SharpCompress/Compressors/LZMA/Bcj2DecoderStream.cs index fd18b16c..a73da73b 100644 --- a/src/SharpCompress/Compressors/LZMA/Bcj2DecoderStream.cs +++ b/src/SharpCompress/Compressors/LZMA/Bcj2DecoderStream.cs @@ -1,6 +1,9 @@ -using System; +using System; using System.Collections.Generic; using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; namespace SharpCompress.Compressors.LZMA; @@ -30,7 +33,7 @@ internal class Bcj2DecoderStream : DecoderStream2 var bt = _mStream.ReadByte(); if (bt < 0) { - throw new EndOfStreamException(); + throw new IncompleteArchiveException("Unexpected end of stream."); } return (byte)bt; @@ -86,13 +89,8 @@ internal class Bcj2DecoderStream : DecoderStream2 private bool _mFinished; private bool _isDisposed; - public Bcj2DecoderStream(Stream[] streams, byte[] info, long limit) + public Bcj2DecoderStream(Stream[] streams) { - if (info != null && info.Length > 0) - { - throw new NotSupportedException(); - } - if (streams.Length != 4) { throw new NotSupportedException(); @@ -163,6 +161,18 @@ internal class Bcj2DecoderStream : DecoderStream2 return count; } + public override Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + // Bcj2DecoderStream uses complex state machine with multiple streams + return Task.FromResult(Read(buffer, offset, count)); + } + public override int ReadByte() { if (_mFinished) @@ -224,7 +234,7 @@ internal class Bcj2DecoderStream : DecoderStream2 var b0 = s.ReadByte(); if (b0 < 0) { - throw new EndOfStreamException(); + throw new IncompleteArchiveException("Unexpected end of stream."); } src <<= 8; diff --git a/src/SharpCompress/Compressors/LZMA/CRC.cs b/src/SharpCompress/Compressors/LZMA/CRC.cs index ea2969bd..6a237df2 100644 --- a/src/SharpCompress/Compressors/LZMA/CRC.cs +++ b/src/SharpCompress/Compressors/LZMA/CRC.cs @@ -1,5 +1,6 @@ using System; using System.IO; +using SharpCompress.Common; namespace SharpCompress.Compressors.LZMA; @@ -39,7 +40,7 @@ internal static class Crc var delta = stream.Read(buffer, 0, (int)Math.Min(length, buffer.Length)); if (delta == 0) { - throw new EndOfStreamException(); + throw new IncompleteArchiveException("Unexpected end of stream."); } crc = Update(crc, buffer, 0, delta); length -= delta; diff --git a/src/SharpCompress/Compressors/LZMA/DecoderRegistry.Async.cs b/src/SharpCompress/Compressors/LZMA/DecoderRegistry.Async.cs new file mode 100644 index 00000000..84e03e7a --- /dev/null +++ b/src/SharpCompress/Compressors/LZMA/DecoderRegistry.Async.cs @@ -0,0 +1,96 @@ +using System; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.SevenZip; +using SharpCompress.Compressors.BZip2; +using SharpCompress.Compressors.Deflate; +using SharpCompress.Compressors.Filters; +using SharpCompress.Compressors.LZMA.Utilities; +using SharpCompress.Compressors.PPMd; +using SharpCompress.Compressors.ZStandard; + +namespace SharpCompress.Compressors.LZMA; + +internal static partial class DecoderRegistry +{ + internal static async ValueTask CreateDecoderStreamAsync( + CMethodId id, + Stream[] inStreams, + byte[]? info, + IPasswordProvider pass, + long limit, + CancellationToken cancellationToken + ) + { + switch (id._id) + { + case K_COPY: + if (info != null) + { + throw new NotSupportedException(); + } + return inStreams.Single(); + case K_DELTA: + return new DeltaFilter(false, inStreams.Single(), info.NotNull()); + case K_LZMA: + case K_LZMA2: + return await LzmaStream + .CreateAsync( + info.NotNull(), + inStreams.Single(), + -1, + limit, + null, + info.NotNull().Length < 5, + false + ) + .ConfigureAwait(false); + case CMethodId.K_AES_ID: + return new AesDecoderStream(inStreams.Single(), info.NotNull(), pass, limit); + case K_BCJ: + return new BCJFilter(false, inStreams.Single()); + case K_BCJ2: + return new Bcj2DecoderStream(inStreams); + case K_PPC: + return new BCJFilterPPC(false, inStreams.Single()); + case K_IA64: + return new BCJFilterIA64(false, inStreams.Single()); + case K_ARM: + return new BCJFilterARM(false, inStreams.Single()); + case K_ARMT: + return new BCJFilterARMT(false, inStreams.Single()); + case K_SPARC: + return new BCJFilterSPARC(false, inStreams.Single()); + case K_ARM64: + return new BCJFilterARM64(false, inStreams.Single()); + case K_RISCV: + return new BCJFilterRISCV(false, inStreams.Single()); + case K_B_ZIP2: + return await BZip2Stream + .CreateAsync( + inStreams.Single(), + CompressionMode.Decompress, + true, + cancellationToken: cancellationToken + ) + .ConfigureAwait(false); + case K_PPMD: + return await PpmdStream + .CreateAsync( + new PpmdProperties(info.NotNull()), + inStreams.Single(), + false, + cancellationToken + ) + .ConfigureAwait(false); + case K_DEFLATE: + return new DeflateStream(inStreams.Single(), CompressionMode.Decompress); + case K_ZSTD: + return new DecompressionStream(inStreams.Single()); + default: + throw new NotSupportedException(); + } + } +} diff --git a/src/SharpCompress/Compressors/LZMA/DecoderStream.cs b/src/SharpCompress/Compressors/LZMA/DecoderStream.cs index 16865351..ac50ac52 100644 --- a/src/SharpCompress/Compressors/LZMA/DecoderStream.cs +++ b/src/SharpCompress/Compressors/LZMA/DecoderStream.cs @@ -1,7 +1,10 @@ -using System; +using System; using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; using SharpCompress.Common.SevenZip; -using SharpCompress.Compressors.LZMA.Utilites; +using SharpCompress.Compressors.LZMA.Utilities; using SharpCompress.IO; namespace SharpCompress.Compressors.LZMA; @@ -14,7 +17,7 @@ internal abstract class DecoderStream2 : Stream public override bool CanWrite => false; - public override void Flush() => throw new NotSupportedException(); + public override void Flush() { } public override long Length => throw new NotSupportedException(); @@ -46,7 +49,7 @@ internal static class DecoderStreamHelper } } - throw new InvalidOperationException("Could not link output stream to coder."); + throw new InvalidFormatException("Could not link output stream to coder."); } private static void FindPrimaryOutStreamIndex( @@ -75,7 +78,7 @@ internal static class DecoderStreamHelper { if (foundPrimaryOutStream) { - throw new NotSupportedException("Multiple output streams."); + throw new InvalidFormatException("Multiple output streams."); } foundPrimaryOutStream = true; @@ -87,7 +90,7 @@ internal static class DecoderStreamHelper if (!foundPrimaryOutStream) { - throw new NotSupportedException("No output stream."); + throw new InvalidFormatException("No output stream."); } } @@ -179,6 +182,100 @@ internal static class DecoderStreamHelper ); } + private static async ValueTask CreateDecoderStreamAsync( + Stream[] packStreams, + long[] packSizes, + Stream[] outStreams, + CFolder folderInfo, + int coderIndex, + IPasswordProvider pass, + CancellationToken cancellationToken + ) + { + var coderInfo = folderInfo._coders[coderIndex]; + if (coderInfo._numOutStreams != 1) + { + throw new NotSupportedException("Multiple output streams are not supported."); + } + + var inStreamId = 0; + for (var i = 0; i < coderIndex; i++) + { + inStreamId += folderInfo._coders[i]._numInStreams; + } + + var outStreamId = 0; + for (var i = 0; i < coderIndex; i++) + { + outStreamId += folderInfo._coders[i]._numOutStreams; + } + + var inStreams = new Stream[coderInfo._numInStreams]; + + for (var i = 0; i < inStreams.Length; i++, inStreamId++) + { + var bindPairIndex = folderInfo.FindBindPairForInStream(inStreamId); + if (bindPairIndex >= 0) + { + var pairedOutIndex = folderInfo._bindPairs[bindPairIndex]._outIndex; + + if (outStreams[pairedOutIndex] != null) + { + throw new NotSupportedException( + "Overlapping stream bindings are not supported." + ); + } + + var otherCoderIndex = FindCoderIndexForOutStreamIndex(folderInfo, pairedOutIndex); + inStreams[i] = await CreateDecoderStreamAsync( + packStreams, + packSizes, + outStreams, + folderInfo, + otherCoderIndex, + pass, + cancellationToken + ) + .ConfigureAwait(false); + + //inStreamSizes[i] = folderInfo.UnpackSizes[pairedOutIndex]; + + if (outStreams[pairedOutIndex] != null) + { + throw new NotSupportedException( + "Overlapping stream bindings are not supported." + ); + } + + outStreams[pairedOutIndex] = inStreams[i]; + } + else + { + var index = folderInfo.FindPackStreamArrayIndex(inStreamId); + if (index < 0) + { + throw new NotSupportedException("Could not find input stream binding."); + } + + inStreams[i] = packStreams[index]; + + //inStreamSizes[i] = packSizes[index]; + } + } + + var unpackSize = folderInfo._unpackSizes[outStreamId]; + return await DecoderRegistry + .CreateDecoderStreamAsync( + coderInfo._methodId, + inStreams, + coderInfo._props, + pass, + unpackSize, + cancellationToken + ) + .ConfigureAwait(false); + } + internal static Stream CreateDecoderStream( Stream inStream, long startPos, @@ -215,4 +312,44 @@ internal static class DecoderStreamHelper pass ); } + + internal static async ValueTask CreateDecoderStreamAsync( + Stream inStream, + long startPos, + long[] packSizes, + CFolder folderInfo, + IPasswordProvider pass, + CancellationToken cancellationToken + ) + { + if (!folderInfo.CheckStructure()) + { + throw new NotSupportedException("Unsupported stream binding structure."); + } + + var inStreams = new Stream[folderInfo._packStreams.Count]; + for (var j = 0; j < folderInfo._packStreams.Count; j++) + { + inStreams[j] = new BufferedSubStream(inStream, startPos, packSizes[j]); + startPos += packSizes[j]; + } + + var outStreams = new Stream[folderInfo._unpackSizes.Count]; + + FindPrimaryOutStreamIndex( + folderInfo, + out var primaryCoderIndex, + out var primaryOutStreamIndex + ); + return await CreateDecoderStreamAsync( + inStreams, + packSizes, + outStreams, + folderInfo, + primaryCoderIndex, + pass, + cancellationToken + ) + .ConfigureAwait(false); + } } diff --git a/src/SharpCompress/Compressors/LZMA/ICoder.cs b/src/SharpCompress/Compressors/LZMA/ICoder.cs index dcddf929..22648533 100644 --- a/src/SharpCompress/Compressors/LZMA/ICoder.cs +++ b/src/SharpCompress/Compressors/LZMA/ICoder.cs @@ -1,12 +1,13 @@ using System; using System.IO; +using SharpCompress.Common; namespace SharpCompress.Compressors.LZMA; /// /// The exception that is thrown when an error in input stream occurs during decoding. /// -internal class DataErrorException : Exception +internal class DataErrorException : SharpCompressException { public DataErrorException() : base("Data Error") { } @@ -15,7 +16,7 @@ internal class DataErrorException : Exception /// /// The exception that is thrown when the value of an argument is outside the allowable range. /// -internal class InvalidParamException : Exception +internal class InvalidParamException : SharpCompressException { public InvalidParamException() : base("Invalid Parameter") { } @@ -147,7 +148,7 @@ internal enum CoderPropId /// /// Specifies mode with end marker. /// - EndMarker + EndMarker, } internal interface ISetCoderProperties diff --git a/src/SharpCompress/Compressors/LZMA/LZ/LzBinTree.cs b/src/SharpCompress/Compressors/LZMA/LZ/LzBinTree.cs index b123e5d5..85795e74 100644 --- a/src/SharpCompress/Compressors/LZMA/LZ/LzBinTree.cs +++ b/src/SharpCompress/Compressors/LZMA/LZ/LzBinTree.cs @@ -1,6 +1,7 @@ #nullable disable using System; +using System.Buffers; using System.IO; namespace SharpCompress.Compressors.LZMA.LZ; @@ -11,8 +12,8 @@ internal sealed class BinTree : InWindow private uint _cyclicBufferSize; private uint _matchMaxLen; - private uint[] _son; - private uint[] _hash; + private uint[] _son = []; + private uint[] _hash = []; private uint _cutValue = 0xFF; private uint _hashMask; @@ -91,10 +92,7 @@ internal sealed class BinTree : InWindow uint keepAddBufferAfter ) { - if (historySize > K_MAX_VAL_FOR_NORMALIZE - 256) - { - throw new ArgumentOutOfRangeException(nameof(historySize)); - } + ThrowHelper.ThrowIfGreaterThan(historySize, K_MAX_VAL_FOR_NORMALIZE - 256); _cutValue = 16 + (matchMaxLen >> 1); var windowReservSize = @@ -111,7 +109,12 @@ internal sealed class BinTree : InWindow var cyclicBufferSize = historySize + 1; if (_cyclicBufferSize != cyclicBufferSize) { - _son = new uint[(_cyclicBufferSize = cyclicBufferSize) * 2]; + if (_son.Length != 0) + { + ArrayPool.Shared.Return(_son); + } + _cyclicBufferSize = cyclicBufferSize; + _son = ArrayPool.Shared.Rent(checked((int)(_cyclicBufferSize * 2))); } var hs = K_BT2_HASH_SIZE; @@ -135,7 +138,27 @@ internal sealed class BinTree : InWindow } if (hs != _hashSizeSum) { - _hash = new uint[_hashSizeSum = hs]; + if (_hash.Length != 0) + { + ArrayPool.Shared.Return(_hash); + } + _hashSizeSum = hs; + _hash = ArrayPool.Shared.Rent(checked((int)_hashSizeSum)); + } + } + + public override void Dispose() + { + base.Dispose(); + if (_son.Length != 0) + { + ArrayPool.Shared.Return(_son); + _son = []; + } + if (_hash.Length != 0) + { + ArrayPool.Shared.Return(_hash); + _hash = []; } } diff --git a/src/SharpCompress/Compressors/LZMA/LZ/LzInWindow.cs b/src/SharpCompress/Compressors/LZMA/LZ/LzInWindow.cs index 9df3c27d..7ee8bc88 100644 --- a/src/SharpCompress/Compressors/LZMA/LZ/LzInWindow.cs +++ b/src/SharpCompress/Compressors/LZMA/LZ/LzInWindow.cs @@ -1,10 +1,12 @@ #nullable disable +using System; +using System.Buffers; using System.IO; namespace SharpCompress.Compressors.LZMA.LZ; -internal class InWindow +internal class InWindow : IDisposable { public byte[] _bufferBase; // pointer to buffer with data private Stream _stream; @@ -78,7 +80,22 @@ internal class InWindow } } - private void Free() => _bufferBase = null; + private void Free() + { + if (_bufferBase is null) + { + return; + } + + ArrayPool.Shared.Return(_bufferBase); + _bufferBase = null; + } + + public virtual void Dispose() + { + ReleaseStream(); + Free(); + } public void Create(uint keepSizeBefore, uint keepSizeAfter, uint keepSizeReserv) { @@ -89,7 +106,7 @@ internal class InWindow { Free(); _blockSize = blockSize; - _bufferBase = new byte[_blockSize]; + _bufferBase = ArrayPool.Shared.Rent(checked((int)_blockSize)); } _pointerToLastSafePosition = _blockSize - keepSizeAfter; _streamEndWasReached = false; diff --git a/src/SharpCompress/Compressors/LZMA/LZ/LzOutWindow.Async.cs b/src/SharpCompress/Compressors/LZMA/LZ/LzOutWindow.Async.cs new file mode 100644 index 00000000..011fdf2e --- /dev/null +++ b/src/SharpCompress/Compressors/LZMA/LZ/LzOutWindow.Async.cs @@ -0,0 +1,177 @@ +#nullable disable + +using System; +using System.Buffers; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Compressors.LZMA.LZ; + +internal partial class OutWindow : IAsyncDisposable +{ + public async ValueTask InitAsync(Stream stream) + { + await ReleaseStreamAsync().ConfigureAwait(false); + _stream = stream; + } + + public async ValueTask ReleaseStreamAsync(CancellationToken cancellationToken = default) + { + await FlushAsync(cancellationToken).ConfigureAwait(false); + _stream = null; + } + + public async ValueTask DisposeAsync() + { + await ReleaseStreamAsync().ConfigureAwait(false); + if (_buffer is null) + { + return; + } + ArrayPool.Shared.Return(_buffer); + _buffer = null; + } + + private async ValueTask FlushAsync(CancellationToken cancellationToken = default) + { + if (_stream is null) + { + return; + } + var size = _pos - _streamPos; + if (size == 0) + { + return; + } + await _stream + .WriteAsync(_buffer, _streamPos, size, cancellationToken) + .ConfigureAwait(false); + if (_pos >= _windowSize) + { + _pos = 0; + } + _streamPos = _pos; + } + + public async ValueTask CopyPendingAsync(CancellationToken cancellationToken = default) + { + if (_pendingLen < 1) + { + return; + } + var rem = _pendingLen; + var pos = (_pendingDist < _pos ? _pos : _pos + _windowSize) - _pendingDist - 1; + while (rem > 0 && HasSpace) + { + if (pos >= _windowSize) + { + pos = 0; + } + await PutByteAsync(_buffer[pos++], cancellationToken).ConfigureAwait(false); + rem--; + } + _pendingLen = rem; + } + + public async ValueTask CopyBlockAsync( + int distance, + int len, + CancellationToken cancellationToken = default + ) + { + var rem = len; + var pos = (distance < _pos ? _pos : _pos + _windowSize) - distance - 1; + var targetSize = HasSpace ? (int)Math.Min(rem, _limit - _total) : 0; + var sizeUntilWindowEnd = Math.Min(_windowSize - _pos, _windowSize - pos); + var sizeUntilOverlap = Math.Abs(pos - _pos); + var fastSize = Math.Min(Math.Min(sizeUntilWindowEnd, sizeUntilOverlap), targetSize); + if (fastSize >= 2) + { + _buffer.AsSpan(pos, fastSize).CopyTo(_buffer.AsSpan(_pos, fastSize)); + _pos += fastSize; + pos += fastSize; + _total += fastSize; + if (_pos >= _windowSize) + { + await FlushAsync(cancellationToken).ConfigureAwait(false); + } + rem -= fastSize; + } + while (rem > 0 && HasSpace) + { + if (pos >= _windowSize) + { + pos = 0; + } + await PutByteAsync(_buffer[pos++], cancellationToken).ConfigureAwait(false); + rem--; + } + _pendingLen = rem; + _pendingDist = distance; + } + + public async ValueTask PutByteAsync(byte b, CancellationToken cancellationToken = default) + { + _buffer[_pos++] = b; + _total++; + if (_pos >= _windowSize) + { + await FlushAsync(cancellationToken).ConfigureAwait(false); + } + } + + public async ValueTask CopyStreamAsync( + Stream stream, + int len, + CancellationToken cancellationToken = default + ) + { + var size = len; + while (size > 0 && _pos < _windowSize && _total < _limit) + { + cancellationToken.ThrowIfCancellationRequested(); + + var curSize = _windowSize - _pos; + if (curSize > _limit - _total) + { + curSize = (int)(_limit - _total); + } + if (curSize > size) + { + curSize = size; + } + var numReadBytes = await stream + .ReadAsync(_buffer, _pos, curSize, cancellationToken) + .ConfigureAwait(false); + if (numReadBytes == 0) + { + throw new DataErrorException(); + } + size -= numReadBytes; + _pos += numReadBytes; + _total += numReadBytes; + if (_pos >= _windowSize) + { + await FlushAsync(cancellationToken).ConfigureAwait(false); + } + } + return len - size; + } + + public async ValueTask TrainAsync(Stream stream) + { + var len = stream.Length; + var size = (len < _windowSize) ? (int)len : _windowSize; + stream.Position = len - size; + _total = 0; + _limit = size; + _pos = _windowSize - size; + await CopyStreamAsync(stream, size).ConfigureAwait(false); + if (_pos == _windowSize) + { + _pos = 0; + } + _streamPos = _pos; + } +} diff --git a/src/SharpCompress/Compressors/LZMA/LZ/LzOutWindow.cs b/src/SharpCompress/Compressors/LZMA/LZ/LzOutWindow.cs index 6efabba1..78300446 100644 --- a/src/SharpCompress/Compressors/LZMA/LZ/LzOutWindow.cs +++ b/src/SharpCompress/Compressors/LZMA/LZ/LzOutWindow.cs @@ -1,11 +1,15 @@ #nullable disable using System; +using System.Buffers; using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; namespace SharpCompress.Compressors.LZMA.LZ; -internal class OutWindow +internal partial class OutWindow : IDisposable { private byte[] _buffer; private int _windowSize; @@ -15,19 +19,26 @@ internal class OutWindow private int _pendingDist; private Stream _stream; - public long _total; - public long _limit; + private long _total; + private long _limit; + + public long Total => _total; public void Create(int windowSize) { + if (windowSize <= 0) + { + throw new InvalidFormatException($"LZMA: invalid dictionary size {windowSize}"); + } if (_windowSize != windowSize) { - _buffer = new byte[windowSize]; - } - else - { - _buffer[windowSize - 1] = 0; + if (_buffer is not null) + { + ArrayPool.Shared.Return(_buffer); + } + _buffer = ArrayPool.Shared.Rent(windowSize); } + _buffer[windowSize - 1] = 0; _windowSize = windowSize; _pos = 0; _streamPos = 0; @@ -36,7 +47,22 @@ internal class OutWindow _limit = 0; } - public void Reset() => Create(_windowSize); + public void Dispose() + { + ReleaseStream(); + if (_buffer is null) + { + return; + } + ArrayPool.Shared.Return(_buffer); + _buffer = null; + } + + public void Reset() + { + ReleaseStream(); + Create(_windowSize); + } public void Init(Stream stream) { @@ -66,7 +92,7 @@ internal class OutWindow _stream = null; } - public void Flush() + private void Flush() { if (_stream is null) { @@ -85,28 +111,56 @@ internal class OutWindow _streamPos = _pos; } - public void CopyBlock(int distance, int len) + public void CopyPending() { - var size = len; - var pos = _pos - distance - 1; - if (pos < 0) + if (_pendingLen < 1) { - pos += _windowSize; + return; } - for (; size > 0 && _pos < _windowSize && _total < _limit; size--) + var rem = _pendingLen; + var pos = (_pendingDist < _pos ? _pos : _pos + _windowSize) - _pendingDist - 1; + while (rem > 0 && HasSpace) { if (pos >= _windowSize) { pos = 0; } - _buffer[_pos++] = _buffer[pos++]; - _total++; + PutByte(_buffer[pos++]); + rem--; + } + _pendingLen = rem; + } + + public void CopyBlock(int distance, int len) + { + var rem = len; + var pos = (distance < _pos ? _pos : _pos + _windowSize) - distance - 1; + var targetSize = HasSpace ? (int)Math.Min(rem, _limit - _total) : 0; + var sizeUntilWindowEnd = Math.Min(_windowSize - _pos, _windowSize - pos); + var sizeUntilOverlap = Math.Abs(pos - _pos); + var fastSize = Math.Min(Math.Min(sizeUntilWindowEnd, sizeUntilOverlap), targetSize); + if (fastSize >= 2) + { + _buffer.AsSpan(pos, fastSize).CopyTo(_buffer.AsSpan(_pos, fastSize)); + _pos += fastSize; + pos += fastSize; + _total += fastSize; if (_pos >= _windowSize) { Flush(); } + rem -= fastSize; } - _pendingLen = size; + while (rem > 0 && HasSpace) + { + if (pos >= _windowSize) + { + pos = 0; + } + PutByte(_buffer[pos++]); + rem--; + } + _pendingLen = rem; _pendingDist = distance; } @@ -188,12 +242,45 @@ internal class OutWindow return size; } - public void CopyPending() + public int Read(Memory buffer, int offset, int count) { - if (_pendingLen > 0) + if (_streamPos >= _pos) { - CopyBlock(_pendingDist, _pendingLen); + return 0; } + + var size = _pos - _streamPos; + if (size > count) + { + size = count; + } + _buffer.AsMemory(_streamPos, size).CopyTo(buffer.Slice(offset, size)); + _streamPos += size; + if (_streamPos >= _windowSize) + { + _pos = 0; + _streamPos = 0; + } + return size; + } + + public int ReadByte() + { + if (_streamPos >= _pos) + { + return -1; + } + + int value = _buffer[_streamPos]; + + _streamPos++; + if (_streamPos >= _windowSize) + { + _pos = 0; + _streamPos = 0; + } + + return value; } public int AvailableBytes => _pos - _streamPos; diff --git a/src/SharpCompress/Compressors/LZMA/LZipStream.Async.cs b/src/SharpCompress/Compressors/LZMA/LZipStream.Async.cs new file mode 100644 index 00000000..949fbc97 --- /dev/null +++ b/src/SharpCompress/Compressors/LZMA/LZipStream.Async.cs @@ -0,0 +1,225 @@ +using System; +using System.Buffers; +using System.Buffers.Binary; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Crypto; + +namespace SharpCompress.Compressors.LZMA; + +public sealed partial class LZipStream +{ + public static LZipStream Create(Stream stream, CompressionMode mode, bool leaveOpen = false) + { + if (mode == CompressionMode.Compress) + { + WriteHeaderSize(stream); + } + return new LZipStream(stream, mode, leaveOpen); + } + + public static async ValueTask CreateAsync( + Stream stream, + CompressionMode mode, + bool leaveOpen = false, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + + if (mode == CompressionMode.Compress) + { + await WriteHeaderSizeAsync(stream).ConfigureAwait(false); + } + return new LZipStream(stream, mode, leaveOpen); + } + + public async ValueTask FinishAsync(CancellationToken cancellationToken) + { + if (_finished) + { + return; + } + + if (Mode == CompressionMode.Compress) + { + cancellationToken.ThrowIfCancellationRequested(); + var crc32Stream = (Crc32Stream)_stream; + await FinishWrappedStreamAsync(crc32Stream).ConfigureAwait(false); + var compressedCount = _countingWritableSubStream.NotNull().BytesWritten; + + var intBuf = ArrayPool.Shared.Rent(8); + try + { + BinaryPrimitives.WriteUInt32LittleEndian(intBuf, crc32Stream.Crc); + await _countingWritableSubStream + .NotNull() + .WriteAsync(intBuf, 0, 4, cancellationToken) + .ConfigureAwait(false); + + BinaryPrimitives.WriteInt64LittleEndian(intBuf, _writeCount); + await _countingWritableSubStream + .NotNull() + .WriteAsync(intBuf, 0, 8, cancellationToken) + .ConfigureAwait(false); + + // Total member size includes the 6-byte header and 20-byte trailer. + BinaryPrimitives.WriteUInt64LittleEndian( + intBuf, + (ulong)compressedCount + (ulong)(6 + 20) + ); + await _countingWritableSubStream + .NotNull() + .WriteAsync(intBuf, 0, 8, cancellationToken) + .ConfigureAwait(false); + } + finally + { + ArrayPool.Shared.Return(intBuf); + } + } + + _finished = true; + } + + private static async ValueTask WriteHeaderSizeAsync(Stream stream) => + // hard coding the dictionary size encoding + await stream.WriteAsync(headerBytes, 0, 6).ConfigureAwait(false); + + private static async ValueTask FinishWrappedStreamAsync(Crc32Stream crc32Stream) + { + if (crc32Stream.WrappedStream is IAsyncDisposable asyncDisposableWrappedStream) + { + await asyncDisposableWrappedStream.DisposeAsync().ConfigureAwait(false); + } + else + { +#pragma warning disable VSTHRD103 // Fallback for streams that do not support async disposal. + crc32Stream.WrappedStream.Dispose(); + crc32Stream.Dispose(); +#pragma warning restore VSTHRD103 + } + } + + /// + /// Asynchronously determines if the given stream is positioned at the start of a v1 LZip + /// file, as indicated by the ASCII characters "LZIP" and a version byte + /// of 1, followed by at least one byte. + /// + /// The stream to read from. Must not be null. + /// Cancellation token. + /// true if the given stream is an LZip file, false otherwise. + public static async ValueTask IsLZipFileAsync( + Stream stream, + CancellationToken cancellationToken = default + ) => await ValidateAndReadSizeAsync(stream, cancellationToken).ConfigureAwait(false) != 0; + + /// + /// Asynchronously reads the 6-byte header of the stream, and returns 0 if either the header + /// couldn't be read or it isn't a validate LZIP header, or the dictionary + /// size if it *is* a valid LZIP file. + /// + public static async ValueTask ValidateAndReadSizeAsync( + Stream stream, + CancellationToken cancellationToken + ) + { + // Read the header + var header = ArrayPool.Shared.Rent(6); + try + { + var n = await stream.ReadAsync(header, 0, 6, cancellationToken).ConfigureAwait(false); + + // TODO: Handle reading only part of the header? + + if (n != 6) + { + return 0; + } + + if ( + header[0] != 'L' + || header[1] != 'Z' + || header[2] != 'I' + || header[3] != 'P' + || header[4] != 1 /* version 1 */ + ) + { + return 0; + } + var basePower = header[5] & 0x1F; + var subtractionNumerator = (header[5] & 0xE0) >> 5; + if (basePower < 4 || basePower > 30) + { + return 0; + } + return (1 << basePower) - (subtractionNumerator * (1 << (basePower - 4))); + } + finally + { + ArrayPool.Shared.Return(header); + } + } + +#if !LEGACY_DOTNET + /// + /// Asynchronously reads bytes from the current stream into a buffer. + /// + public override ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) => ReadAndValidateAsync(buffer, cancellationToken); +#endif + + /// + /// Asynchronously reads bytes from the current stream into a buffer. + /// + public override Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken = default + ) => ReadAndValidateAsync(buffer, offset, count, cancellationToken); + + private async Task ReadAndValidateAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + var read = await _stream + .ReadAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + UpdateAndValidateAtEof(buffer.AsSpan(offset, read), read); + return read; + } + +#if !LEGACY_DOTNET + private async ValueTask ReadAndValidateAsync( + Memory buffer, + CancellationToken cancellationToken + ) + { + var read = await _stream.ReadAsync(buffer, cancellationToken).ConfigureAwait(false); + UpdateAndValidateAtEof(buffer.Span[..read], read); + return read; + } +#endif + + /// + /// Asynchronously writes bytes from a buffer to the current stream. + /// + public override async Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + cancellationToken.ThrowIfCancellationRequested(); + await _stream.WriteAsync(buffer, offset, count, cancellationToken).ConfigureAwait(false); + _writeCount += count; + } +} diff --git a/src/SharpCompress/Compressors/LZMA/LZipStream.cs b/src/SharpCompress/Compressors/LZMA/LZipStream.cs index 4b63a621..f2e1abea 100644 --- a/src/SharpCompress/Compressors/LZMA/LZipStream.cs +++ b/src/SharpCompress/Compressors/LZMA/LZipStream.cs @@ -1,8 +1,12 @@ using System; using System.Buffers.Binary; using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; using SharpCompress.Crypto; using SharpCompress.IO; +using SharpCompress.Providers; namespace SharpCompress.Compressors.LZMA; @@ -14,40 +18,85 @@ namespace SharpCompress.Compressors.LZMA; /// /// Stream supporting the LZIP format, as documented at http://www.nongnu.org/lzip/manual/lzip_manual.html /// -public sealed class LZipStream : Stream +public sealed partial class LZipStream : Stream, IFinishable { private readonly Stream _stream; - private readonly CountingWritableSubStream? _countingWritableSubStream; + private readonly CountingStream? _countingWritableSubStream; + private readonly CountingStream? _countingReadableSubStream; + private readonly uint[]? _crc32Table; + private readonly ulong? _expectedDataSize; + private readonly ulong? _expectedMemberSize; + private readonly bool _skipTrailerValidation; private bool _disposed; private bool _finished; + private bool _trailerValidated; + private uint _seed = Crc32Stream.DEFAULT_SEED; + private ulong _readCount; + private readonly long _memberStartPosition; + private readonly long _compressedDataStartPosition; private long _writeCount; + private readonly Stream? _originalStream; + private readonly bool _leaveOpen; - public LZipStream(Stream stream, CompressionMode mode) + private LZipStream(Stream stream, CompressionMode mode, bool leaveOpen = false) { Mode = mode; + _originalStream = stream; + _leaveOpen = leaveOpen; if (mode == CompressionMode.Decompress) { + _skipTrailerValidation = stream is SharpCompressStream; + _memberStartPosition = stream.CanSeek ? stream.Position : 0; var dSize = ValidateAndReadSize(stream); if (dSize == 0) { - throw new IOException("Not an LZip stream"); + throw new InvalidFormatException("Not an LZip stream"); } var properties = GetProperties(dSize); - _stream = new LzmaStream(properties, stream); + var trailerStream = GetSeekableTrailerStream(stream); + if (trailerStream is not null) + { + var position = trailerStream.Position; + trailerStream.Position = trailerStream.Length - 16; + Span sizeTrailer = stackalloc byte[16]; + trailerStream.ReadFully(sizeTrailer); + _expectedDataSize = BinaryPrimitives.ReadUInt64LittleEndian(sizeTrailer); + _expectedMemberSize = BinaryPrimitives.ReadUInt64LittleEndian(sizeTrailer[8..]); + if (_expectedDataSize > long.MaxValue) + { + throw new InvalidFormatException("LZip data size is too large."); + } + trailerStream.Position = position; + } + _compressedDataStartPosition = stream.CanSeek ? stream.Position : 0; + _countingReadableSubStream = new CountingStream( + SharpCompressStream.CreateNonDisposing(stream) + ); + _crc32Table = Crc32Stream.InitializeTable(Crc32Stream.DEFAULT_POLYNOMIAL); + _stream = LzmaStream.Create( + properties, + _countingReadableSubStream, + inputSize: -1, + outputSize: _expectedDataSize.HasValue + ? checked((long)_expectedDataSize.Value) + : -1, + leaveOpen: leaveOpen + ); } else { //default var dSize = 104 * 1024; - WriteHeaderSize(stream); - - _countingWritableSubStream = new CountingWritableSubStream(stream); + _countingWritableSubStream = new CountingStream( + SharpCompressStream.CreateNonDisposing(stream) + ); _stream = new Crc32Stream( - new LzmaStream( + LzmaStream.Create( new LzmaEncoderProperties(true, dSize), false, + null, _countingWritableSubStream ) ); @@ -63,18 +112,21 @@ public sealed class LZipStream : Stream var crc32Stream = (Crc32Stream)_stream; crc32Stream.WrappedStream.Dispose(); crc32Stream.Dispose(); - var compressedCount = _countingWritableSubStream!.Count; + var compressedCount = _countingWritableSubStream.NotNull().BytesWritten; Span intBuf = stackalloc byte[8]; BinaryPrimitives.WriteUInt32LittleEndian(intBuf, crc32Stream.Crc); - _countingWritableSubStream.Write(intBuf.Slice(0, 4)); + _countingWritableSubStream?.Write(intBuf.Slice(0, 4)); BinaryPrimitives.WriteInt64LittleEndian(intBuf, _writeCount); - _countingWritableSubStream.Write(intBuf); + _countingWritableSubStream?.Write(intBuf); //total with headers - BinaryPrimitives.WriteUInt64LittleEndian(intBuf, compressedCount + 6 + 20); - _countingWritableSubStream.Write(intBuf); + BinaryPrimitives.WriteUInt64LittleEndian( + intBuf, + (ulong)compressedCount + (ulong)(6 + 20) + ); + _countingWritableSubStream?.Write(intBuf); } _finished = true; } @@ -86,6 +138,7 @@ public sealed class LZipStream : Stream { if (_disposed) { + base.Dispose(disposing); return; } _disposed = true; @@ -93,7 +146,12 @@ public sealed class LZipStream : Stream { Finish(); _stream.Dispose(); + if (!_leaveOpen) + { + _originalStream?.Dispose(); + } } + base.Dispose(disposing); } public CompressionMode Mode { get; } @@ -116,18 +174,42 @@ public sealed class LZipStream : Stream set => throw new NotImplementedException(); } - public override int Read(byte[] buffer, int offset, int count) => - _stream.Read(buffer, offset, count); + public override int Read(byte[] buffer, int offset, int count) + { + var read = _stream.Read(buffer, offset, count); + UpdateAndValidateAtEof(buffer.AsSpan(offset, read), read); + return read; + } - public override int ReadByte() => _stream.ReadByte(); + public override int ReadByte() + { + var value = _stream.ReadByte(); + if (value == -1) + { + ValidateTrailer(); + } + else + { + Span buffer = stackalloc byte[1]; + buffer[0] = (byte)value; + UpdateChecksum(buffer); + } + + return value; + } public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); public override void SetLength(long value) => throw new NotImplementedException(); -#if !NETFRAMEWORK && !NETSTANDARD2_0 +#if !LEGACY_DOTNET - public override int Read(Span buffer) => _stream.Read(buffer); + public override int Read(Span buffer) + { + var read = _stream.Read(buffer); + UpdateAndValidateAtEof(buffer[..read], read); + return read; + } public override void Write(ReadOnlySpan buffer) { @@ -149,6 +231,8 @@ public sealed class LZipStream : Stream ++_writeCount; } + // Async methods moved to LZipStream.Async.cs + #endregion /// @@ -167,11 +251,6 @@ public sealed class LZipStream : Stream /// public static int ValidateAndReadSize(Stream stream) { - if (stream is null) - { - throw new ArgumentNullException(nameof(stream)); - } - // Read the header Span header = stackalloc byte[6]; var n = stream.Read(header); @@ -198,33 +277,27 @@ public sealed class LZipStream : Stream return (1 << basePower) - (subtractionNumerator * (1 << (basePower - 4))); } - private static readonly byte[] headerBytes = new byte[6] - { + // Async methods moved to LZipStream.Async.cs + + private static readonly byte[] headerBytes = + [ (byte)'L', (byte)'Z', (byte)'I', (byte)'P', 1, - 113 - }; - - public static void WriteHeaderSize(Stream stream) - { - if (stream is null) - { - throw new ArgumentNullException(nameof(stream)); - } + 113, + ]; + public static void WriteHeaderSize(Stream stream) => // hard coding the dictionary size encoding stream.Write(headerBytes, 0, 6); - } /// /// Creates a byte array to communicate the parameters and dictionary size to LzmaStream. /// private static byte[] GetProperties(int dictionarySize) => - new byte[] - { + [ // Parameters as per http://www.nongnu.org/lzip/manual/lzip_manual.html#Stream-format // but encoded as a single byte in the format LzmaStream expects. // literal_context_bits = 3 @@ -235,6 +308,132 @@ public sealed class LZipStream : Stream (byte)(dictionarySize & 0xff), (byte)((dictionarySize >> 8) & 0xff), (byte)((dictionarySize >> 16) & 0xff), - (byte)((dictionarySize >> 24) & 0xff) - }; + (byte)((dictionarySize >> 24) & 0xff), + ]; + + private static Stream? GetSeekableTrailerStream(Stream stream) + { + while (stream is SharpCompressStream { IsPassthrough: true } sharpCompressStream) + { + stream = sharpCompressStream.BaseStream(); + } + + if (stream is SeekableSharpCompressStream seekableSharpCompressStream) + { + stream = seekableSharpCompressStream.BaseStream(); + } + + return stream is SharpCompressStream ? null + : stream.CanSeek ? stream + : null; + } + + private static Stream? GetPhysicalSeekableStream(Stream stream) + { + while (stream is SharpCompressStream sharpCompressStream) + { + var baseStream = sharpCompressStream.BaseStream(); + if (ReferenceEquals(baseStream, stream) || !baseStream.CanSeek) + { + break; + } + + stream = baseStream; + } + + return stream.CanSeek ? stream : null; + } + + private static bool IsProbeWrapper(Stream stream) => + stream is SharpCompressStream { IsPassthrough: true } sharpCompressStream + && sharpCompressStream.BaseStream() is SharpCompressStream { IsPassthrough: false }; + + private void UpdateAndValidateAtEof(ReadOnlySpan buffer, int read) + { + if (Mode != CompressionMode.Decompress) + { + return; + } + + if (read > 0) + { + UpdateChecksum(buffer); + return; + } + + ValidateTrailer(); + } + + private void UpdateChecksum(ReadOnlySpan buffer) + { + _seed = Crc32Stream.CalculateCrc(_crc32Table.NotNull(), _seed, buffer); + _readCount += (ulong)buffer.Length; + } + + private void ValidateTrailer() + { + if (_trailerValidated || _skipTrailerValidation || Mode != CompressionMode.Decompress) + { + return; + } + + _trailerValidated = true; + + var countingStream = _countingReadableSubStream.NotNull(); + ulong? compressedDataSize = null; + Span trailer = stackalloc byte[20]; + if (_expectedMemberSize.HasValue && countingStream.CanSeek) + { + compressedDataSize = _expectedMemberSize.Value - 26; + countingStream.Position = _compressedDataStartPosition + (long)compressedDataSize.Value; + countingStream.ReadFully(trailer); + } + else if (GetPhysicalSeekableStream(countingStream.WrappedStream) is { } trailerStream) + { + var position = trailerStream.Position; + trailerStream.Position = trailerStream.Length - 20; + trailerStream.ReadFully(trailer); + trailerStream.Position = position; + } + else + { + compressedDataSize = _stream is LzmaStream lzmaStream + ? (ulong)lzmaStream.CompressedBytesRead + : (ulong)countingStream.BytesRead; + if (countingStream.CanSeek) + { + countingStream.Position = + _compressedDataStartPosition + (long)compressedDataSize.Value; + } + countingStream.ReadFully(trailer); + } + + var expectedCrc = BinaryPrimitives.ReadUInt32LittleEndian(trailer); + var expectedDataSize = BinaryPrimitives.ReadUInt64LittleEndian(trailer[4..]); + var expectedMemberSize = BinaryPrimitives.ReadUInt64LittleEndian(trailer[12..]); + + var actualCrc = ~_seed; + if (actualCrc != expectedCrc) + { + throw new InvalidFormatException( + $"LZip CRC mismatch. Expected 0x{expectedCrc:X8}, actual 0x{actualCrc:X8}." + ); + } + + if (_readCount != expectedDataSize) + { + throw new InvalidFormatException( + $"LZip data size mismatch. Expected {expectedDataSize}, actual {_readCount}." + ); + } + + var actualMemberSize = compressedDataSize ?? expectedMemberSize - 26; + actualMemberSize += 26; + if (actualMemberSize != expectedMemberSize) + { + throw new InvalidFormatException( + $"LZip member size mismatch. Expected {expectedMemberSize}, actual {actualMemberSize}." + ); + } + } } diff --git a/src/SharpCompress/Compressors/LZMA/Log.cs b/src/SharpCompress/Compressors/LZMA/Log.cs index e954e259..e0c9bcd9 100644 --- a/src/SharpCompress/Compressors/LZMA/Log.cs +++ b/src/SharpCompress/Compressors/LZMA/Log.cs @@ -1,12 +1,12 @@ -using System; +using System; using System.Collections.Generic; -using System.Diagnostics; +using SharpCompress.Common; namespace SharpCompress.Compressors.LZMA; internal static class Log { - private static readonly Stack INDENT = new Stack(); + private static readonly Stack INDENT = new(); private static bool NEEDS_INDENT = true; static Log() => INDENT.Push(""); @@ -17,7 +17,7 @@ internal static class Log { if (INDENT.Count == 1) { - throw new InvalidOperationException(); + throw new ArchiveOperationException(); } INDENT.Pop(); @@ -28,52 +28,44 @@ internal static class Log if (NEEDS_INDENT) { NEEDS_INDENT = false; - Debug.Write(INDENT.Peek()); } } public static void Write(object value) { EnsureIndent(); - Debug.Write(value); } public static void Write(string text) { EnsureIndent(); - Debug.Write(text); } public static void Write(string format, params object[] args) { EnsureIndent(); - Debug.Write(string.Format(format, args)); } public static void WriteLine() { - Debug.WriteLine(""); NEEDS_INDENT = true; } public static void WriteLine(object value) { EnsureIndent(); - Debug.WriteLine(value); NEEDS_INDENT = true; } public static void WriteLine(string text) { EnsureIndent(); - Debug.WriteLine(text); NEEDS_INDENT = true; } public static void WriteLine(string format, params object[] args) { EnsureIndent(); - Debug.WriteLine(string.Format(format, args)); NEEDS_INDENT = true; } } diff --git a/src/SharpCompress/Compressors/LZMA/Lzma2EncoderStream.cs b/src/SharpCompress/Compressors/LZMA/Lzma2EncoderStream.cs new file mode 100644 index 00000000..51e1b753 --- /dev/null +++ b/src/SharpCompress/Compressors/LZMA/Lzma2EncoderStream.cs @@ -0,0 +1,507 @@ +using System; +using System.Buffers; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.IO; + +namespace SharpCompress.Compressors.LZMA; + +/// +/// Write-only stream that compresses data using the LZMA2 framing format. +/// Buffers input, compresses in chunks, and writes LZMA2-framed output to the underlying stream. +/// Each chunk is independently compressed with a fresh LZMA encoder. +/// +internal sealed class Lzma2EncoderStream : Stream +{ + // Max uncompressed chunk size per LZMA2 spec: (0x1F << 16) + 0xFFFF + 1 = 2MB + private const int MAX_UNCOMPRESSED_CHUNK_SIZE = (0x1F << 16) + 0xFFFF + 1; + + // Max compressed payload per LZMA2 chunk header: 0xFFFF + 1 = 64KB + private const int MAX_COMPRESSED_CHUNK_SIZE = 0xFFFF + 1; + + // Max uncompressed sub-chunk for raw (uncompressed) chunks: 0xFFFF + 1 = 64KB + private const int MAX_UNCOMPRESSED_SUBCHUNK_SIZE = 0xFFFF + 1; + + private readonly Stream _output; + private readonly int _dictionarySize; + private readonly int _numFastBytes; + private readonly byte[] _buffer; + private readonly byte[] _properties; + private readonly Encoder _encoder; + private int _bufferPosition; + private bool _isFirstChunk = true; + private bool _isDisposed; + private byte _lzmaPropertiesByte; + + /// + /// Creates a new LZMA2 encoder stream. + /// + /// The stream to write LZMA2-framed compressed data to. + /// Dictionary size for LZMA compression. + /// Number of fast bytes for LZMA compression. + public Lzma2EncoderStream(Stream output, int dictionarySize, int numFastBytes) + { + _output = output; + _dictionarySize = dictionarySize; + _numFastBytes = numFastBytes; + _buffer = ArrayPool.Shared.Rent(MAX_UNCOMPRESSED_CHUNK_SIZE); + _bufferPosition = 0; + + var encoderProps = new LzmaEncoderProperties(eos: false, _dictionarySize, _numFastBytes); + _encoder = new Encoder(); + _encoder.SetCoderProperties(encoderProps.PropIDs, encoderProps.Properties); + + Span lzmaProperties = stackalloc byte[5]; + _encoder.WriteCoderProperties(lzmaProperties); + _lzmaPropertiesByte = lzmaProperties[0]; + _properties = [EncodeDictionarySize(_dictionarySize)]; + } + + /// + /// Gets the 1-byte LZMA2 properties (encoded dictionary size). + /// + public byte[] Properties => _properties; + + public override bool CanRead => false; + public override bool CanSeek => false; + public override bool CanWrite => true; + public override long Length => throw new NotSupportedException(); + + public override long Position + { + get => throw new NotSupportedException(); + set => throw new NotSupportedException(); + } + + public override void Write(byte[] buffer, int offset, int count) + { + while (count > 0) + { + var toCopy = Math.Min(count, MAX_UNCOMPRESSED_CHUNK_SIZE - _bufferPosition); + Buffer.BlockCopy(buffer, offset, _buffer, _bufferPosition, toCopy); + _bufferPosition += toCopy; + offset += toCopy; + count -= toCopy; + + if (_bufferPosition == MAX_UNCOMPRESSED_CHUNK_SIZE) + { + FlushChunk(); + } + } + } + + public override async Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + while (count > 0) + { + cancellationToken.ThrowIfCancellationRequested(); + + var toCopy = Math.Min(count, MAX_UNCOMPRESSED_CHUNK_SIZE - _bufferPosition); + Buffer.BlockCopy(buffer, offset, _buffer, _bufferPosition, toCopy); + _bufferPosition += toCopy; + offset += toCopy; + count -= toCopy; + + if (_bufferPosition == MAX_UNCOMPRESSED_CHUNK_SIZE) + { + await FlushChunkAsync(cancellationToken).ConfigureAwait(false); + } + } + } + +#if !LEGACY_DOTNET + public override async ValueTask WriteAsync( + ReadOnlyMemory buffer, + CancellationToken cancellationToken = default + ) + { + var offset = 0; + var count = buffer.Length; + while (count > 0) + { + cancellationToken.ThrowIfCancellationRequested(); + + var toCopy = Math.Min(count, MAX_UNCOMPRESSED_CHUNK_SIZE - _bufferPosition); + buffer.Slice(offset, toCopy).Span.CopyTo(_buffer.AsSpan(_bufferPosition, toCopy)); + _bufferPosition += toCopy; + offset += toCopy; + count -= toCopy; + + if (_bufferPosition == MAX_UNCOMPRESSED_CHUNK_SIZE) + { + await FlushChunkAsync(cancellationToken).ConfigureAwait(false); + } + } + } +#endif + + public override void Flush() { } + + public override int Read(byte[] buffer, int offset, int count) => + throw new NotSupportedException(); + + public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); + + public override void SetLength(long value) => throw new NotSupportedException(); + + protected override void Dispose(bool disposing) + { + if (disposing && !_isDisposed) + { + _isDisposed = true; + + // Flush remaining buffered data + if (_bufferPosition > 0) + { + FlushChunk(); + } + + // Write LZMA2 end marker + _output.WriteByte(0x00); + _encoder.Dispose(); + ArrayPool.Shared.Return(_buffer); + } + base.Dispose(disposing); + } + + private static readonly byte[] _endMarker = [0x00]; + +#if !LEGACY_DOTNET || NETSTANDARD2_1 + public override async ValueTask DisposeAsync() +#else + public async ValueTask DisposeAsync() +#endif + { + if (!_isDisposed) + { + _isDisposed = true; + + if (_bufferPosition > 0) + { + await FlushChunkAsync().ConfigureAwait(false); + } +#if !LEGACY_DOTNET || NETSTANDARD2_1 + + await _output.WriteAsync(new Memory(_endMarker)).ConfigureAwait(false); +#else + await _output.WriteAsync(_endMarker, 0, 1).ConfigureAwait(false); +#endif + + _encoder.Dispose(); + ArrayPool.Shared.Return(_buffer); + } + +#if !LEGACY_DOTNET || NETSTANDARD2_1 + await base.DisposeAsync().ConfigureAwait(false); +#endif + } + + private void FlushChunk() + { + if (_bufferPosition == 0) + { + return; + } + + var uncompressedSize = _bufferPosition; + var uncompressedData = _buffer.AsSpan(0, uncompressedSize); + _bufferPosition = 0; + + // Try compressing the data + (byte[] Buffer, int Length) compressed; + try + { + compressed = CompressBlock(_buffer, uncompressedSize); + } + catch (Exception ex) when (ex is not OutOfMemoryException and not StackOverflowException) + { + // If compression fails, write as uncompressed + WriteUncompressedChunks(uncompressedData); + return; + } + + // Check if compressed output fits in a single chunk and is actually smaller + if ( + compressed.Length <= MAX_COMPRESSED_CHUNK_SIZE + && compressed.Length < uncompressedData.Length + ) + { + WriteCompressedChunk(uncompressedData.Length, compressed.Buffer, compressed.Length); + } + else + { + WriteUncompressedChunks(uncompressedData); + } + } + + private async ValueTask FlushChunkAsync(CancellationToken cancellationToken = default) + { + if (_bufferPosition == 0) + { + return; + } + + var uncompressedSize = _bufferPosition; + var uncompressedData = _buffer.AsMemory(0, uncompressedSize); + _bufferPosition = 0; + + (byte[] Buffer, int Length) compressed; + try + { + compressed = CompressBlock(_buffer, uncompressedSize); + } + catch (Exception ex) when (ex is not OutOfMemoryException and not StackOverflowException) + { + await WriteUncompressedChunksAsync(uncompressedData, cancellationToken) + .ConfigureAwait(false); + return; + } + + if ( + compressed.Length <= MAX_COMPRESSED_CHUNK_SIZE + && compressed.Length < uncompressedData.Length + ) + { + await WriteCompressedChunkAsync( + uncompressedData.Length, + compressed.Buffer, + compressed.Length, + cancellationToken + ) + .ConfigureAwait(false); + } + else + { + await WriteUncompressedChunksAsync(uncompressedData, cancellationToken) + .ConfigureAwait(false); + } + } + + private (byte[] Buffer, int Length) CompressBlock(byte[] data, int length) + { + using var inputMs = new MemoryStream(data, 0, length, writable: false); + using var outputMs = new PooledMemoryStream(); + + _encoder.Code(inputMs, outputMs, length, -1, null); + + var fullCompressed = outputMs.ToArray(); + + // The LZMA range encoder flush writes trailing bytes the decoder doesn't consume. + // Trial-decode to find the exact byte count the decoder needs, so the LZMA2 + // chunk header reports a compressed size that matches what the decoder reads. + var consumed = FindConsumedBytes(fullCompressed, fullCompressed.Length, length); + return (fullCompressed, consumed); + } + + private int FindConsumedBytes(byte[] compressedData, int compressedSize, int uncompressedSize) + { + // Build 5-byte LZMA property header: [pb/lp/lc byte] [dictSize as LE int32] + Span props = stackalloc byte[5]; + props[0] = _lzmaPropertiesByte; + props[1] = (byte)_dictionarySize; + props[2] = (byte)(_dictionarySize >> 8); + props[3] = (byte)(_dictionarySize >> 16); + props[4] = (byte)(_dictionarySize >> 24); + + var decoder = new Decoder(); + decoder.SetDecoderProperties(props); + + using var input = new MemoryStream(compressedData, 0, compressedSize, writable: false); + decoder.Code(input, Stream.Null, compressedSize, uncompressedSize, null); + + return (int)input.Position; + } + + /// + /// Writes a compressed LZMA2 chunk. + /// Header: [control] [uncompSize_hi] [uncompSize_lo] [compSize_hi] [compSize_lo] [props?] + /// + private void WriteCompressedChunk( + int uncompressedSize, + byte[] compressedData, + int compressedSize + ) + { + var uncompSizeMinus1 = uncompressedSize - 1; + var compSizeMinus1 = compressedSize - 1; + + // Each chunk is compressed independently with a fresh LZMA encoder, + // so we must use 0xE0 (full reset: dictionary + state + properties) every time. + // The decoder uses outWindow.Total for literal context and posState; + // 0xE0 triggers outWindow.Reset() which zeros Total, matching the encoder's + // assumption that position starts at 0 for each chunk. + var control = (byte)(0xE0 | ((uncompSizeMinus1 >> 16) & 0x1F)); + _isFirstChunk = false; + + _output.WriteByte(control); + _output.WriteByte((byte)((uncompSizeMinus1 >> 8) & 0xFF)); + _output.WriteByte((byte)(uncompSizeMinus1 & 0xFF)); + _output.WriteByte((byte)((compSizeMinus1 >> 8) & 0xFF)); + _output.WriteByte((byte)(compSizeMinus1 & 0xFF)); + + // 0xE0 (>= 0xC0) requires properties byte + _output.WriteByte(_lzmaPropertiesByte); + + _output.Write(compressedData, 0, compressedSize); + } + + private async ValueTask WriteCompressedChunkAsync( + int uncompressedSize, + byte[] compressedData, + int compressedSize, + CancellationToken cancellationToken = default + ) + { + var uncompSizeMinus1 = uncompressedSize - 1; + var compSizeMinus1 = compressedSize - 1; + var control = (byte)(0xE0 | ((uncompSizeMinus1 >> 16) & 0x1F)); + _isFirstChunk = false; + + var header = ArrayPool.Shared.Rent(6); + try + { + header[0] = control; + header[1] = (byte)((uncompSizeMinus1 >> 8) & 0xFF); + header[2] = (byte)(uncompSizeMinus1 & 0xFF); + header[3] = (byte)((compSizeMinus1 >> 8) & 0xFF); + header[4] = (byte)(compSizeMinus1 & 0xFF); + header[5] = _lzmaPropertiesByte; + await _output.WriteAsync(header, 0, 6, cancellationToken).ConfigureAwait(false); + } + finally + { + ArrayPool.Shared.Return(header); + } + await _output + .WriteAsync(compressedData, 0, compressedSize, cancellationToken) + .ConfigureAwait(false); + } + + /// + /// Writes data as uncompressed LZMA2 sub-chunks (max 64KB each). + /// Header: [control] [size_hi] [size_lo] + /// + private void WriteUncompressedChunks(ReadOnlySpan data) + { + var offset = 0; + while (offset < data.Length) + { + var chunkSize = Math.Min(data.Length - offset, MAX_UNCOMPRESSED_SUBCHUNK_SIZE); + var sizeMinus1 = chunkSize - 1; + + byte control; + if (_isFirstChunk) + { + // 0x01: uncompressed with dictionary reset + control = 0x01; + _isFirstChunk = false; + } + else + { + // 0x02: uncompressed without dictionary reset + control = 0x02; + } + + _output.WriteByte(control); + _output.WriteByte((byte)((sizeMinus1 >> 8) & 0xFF)); + _output.WriteByte((byte)(sizeMinus1 & 0xFF)); + + _output.Write(data.Slice(offset, chunkSize)); + offset += chunkSize; + } + } + + private async ValueTask WriteUncompressedChunksAsync( + ReadOnlyMemory data, + CancellationToken cancellationToken = default + ) + { + var offset = 0; + while (offset < data.Length) + { + var chunkSize = Math.Min(data.Length - offset, MAX_UNCOMPRESSED_SUBCHUNK_SIZE); + var sizeMinus1 = chunkSize - 1; + + byte control; + if (_isFirstChunk) + { + control = 0x01; + _isFirstChunk = false; + } + else + { + control = 0x02; + } + + var header = ArrayPool.Shared.Rent(3); + try + { + header[0] = control; + header[1] = (byte)((sizeMinus1 >> 8) & 0xFF); + header[2] = (byte)(sizeMinus1 & 0xFF); + await _output.WriteAsync(header, 0, 3, cancellationToken).ConfigureAwait(false); + } + finally + { + ArrayPool.Shared.Return(header); + } + +#if !LEGACY_DOTNET || NETSTANDARD2_1 + await _output + .WriteAsync(data.Slice(offset, chunkSize), cancellationToken) + .ConfigureAwait(false); +#else + var chunk = ArrayPool.Shared.Rent(chunkSize); + try + { + data.Slice(offset, chunkSize).CopyTo(chunk); + await _output + .WriteAsync(chunk, 0, chunkSize, cancellationToken) + .ConfigureAwait(false); + } + finally + { + ArrayPool.Shared.Return(chunk); + } +#endif + offset += chunkSize; + } + } + + /// + /// Encodes a dictionary size into the 1-byte LZMA2 properties format. + /// Reverse of the decoder formula: dictSize = (2 | (p and 1)) shl ((p shr 1) + 11) + /// Finds the smallest p where the formula result >= target dictSize. + /// + internal static byte EncodeDictionarySize(int dictSize) + { + // Special case: very small dictionary sizes + if (dictSize <= (2 << 11)) + { + return 0; + } + + for (byte p = 0; p < 40; p++) + { + var shift = (p >> 1) + 11; + if (shift >= 31) + { + return p; + } + + var size = (2 | (p & 1)) << shift; + if (size >= dictSize) + { + return p; + } + } + + return 40; + } +} diff --git a/src/SharpCompress/Compressors/LZMA/LzmaDecoder.Async.cs b/src/SharpCompress/Compressors/LZMA/LzmaDecoder.Async.cs new file mode 100644 index 00000000..7b81b892 --- /dev/null +++ b/src/SharpCompress/Compressors/LZMA/LzmaDecoder.Async.cs @@ -0,0 +1,363 @@ +#nullable disable + +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Compressors.LZMA.LZ; +using SharpCompress.Compressors.LZMA.RangeCoder; + +namespace SharpCompress.Compressors.LZMA; + +public partial class Decoder : IAsyncDisposable +{ + public async ValueTask DisposeAsync() + { + if (_outWindow is not null) + { + await _outWindow.DisposeAsync().ConfigureAwait(false); + _outWindow = null; + } + } + + partial class LenDecoder + { + public async ValueTask DecodeAsync( + RangeCoder.Decoder rangeDecoder, + uint posState, + CancellationToken cancellationToken = default + ) + { + if ( + await _choice.DecodeAsync(rangeDecoder, cancellationToken).ConfigureAwait(false) + == 0 + ) + { + return await _lowCoder[posState] + .DecodeAsync(rangeDecoder, cancellationToken) + .ConfigureAwait(false); + } + var symbol = Base.K_NUM_LOW_LEN_SYMBOLS; + if ( + await _choice2.DecodeAsync(rangeDecoder, cancellationToken).ConfigureAwait(false) + == 0 + ) + { + symbol += await _midCoder[posState] + .DecodeAsync(rangeDecoder, cancellationToken) + .ConfigureAwait(false); + } + else + { + symbol += Base.K_NUM_MID_LEN_SYMBOLS; + symbol += await _highCoder + .DecodeAsync(rangeDecoder, cancellationToken) + .ConfigureAwait(false); + } + return symbol; + } + } + + partial class LiteralDecoder + { + partial struct Decoder2 + { + public async ValueTask DecodeNormalAsync( + RangeCoder.Decoder rangeDecoder, + CancellationToken cancellationToken = default + ) + { + uint symbol = 1; + do + { + symbol = + (symbol << 1) + | await _decoders[_baseIndex + symbol] + .DecodeAsync(rangeDecoder, cancellationToken) + .ConfigureAwait(false); + } while (symbol < 0x100); + return (byte)symbol; + } + + public async ValueTask DecodeWithMatchByteAsync( + RangeCoder.Decoder rangeDecoder, + byte matchByte, + CancellationToken cancellationToken = default + ) + { + uint symbol = 1; + do + { + var matchBit = (uint)(matchByte >> 7) & 1; + matchByte <<= 1; + var bit = await _decoders[_baseIndex + ((1 + matchBit) << 8) + symbol] + .DecodeAsync(rangeDecoder, cancellationToken) + .ConfigureAwait(false); + symbol = (symbol << 1) | bit; + if (matchBit != bit) + { + while (symbol < 0x100) + { + symbol = + (symbol << 1) + | await _decoders[_baseIndex + symbol] + .DecodeAsync(rangeDecoder, cancellationToken) + .ConfigureAwait(false); + } + break; + } + } while (symbol < 0x100); + return (byte)symbol; + } + } + + public async ValueTask DecodeNormalAsync( + RangeCoder.Decoder rangeDecoder, + uint pos, + byte prevByte, + CancellationToken cancellationToken = default + ) => + await _coders[GetState(pos, prevByte)] + .DecodeNormalAsync(rangeDecoder, cancellationToken) + .ConfigureAwait(false); + + public async ValueTask DecodeWithMatchByteAsync( + RangeCoder.Decoder rangeDecoder, + uint pos, + byte prevByte, + byte matchByte, + CancellationToken cancellationToken = default + ) => + await _coders[GetState(pos, prevByte)] + .DecodeWithMatchByteAsync(rangeDecoder, matchByte, cancellationToken) + .ConfigureAwait(false); + } + + public async ValueTask CodeAsync( + Stream inStream, + Stream outStream, + long inSize, + long outSize, + ICodeProgress progress, + CancellationToken cancellationToken = default + ) + { + if (_outWindow is null) + { + CreateDictionary(); + } + await _outWindow.InitAsync(outStream).ConfigureAwait(false); + if (outSize > 0) + { + _outWindow.SetLimit(outSize); + } + else + { + _outWindow.SetLimit(long.MaxValue - _outWindow.Total); + } + + var rangeDecoder = new RangeCoder.Decoder(); + await rangeDecoder.InitAsync(inStream, cancellationToken).ConfigureAwait(false); + + await CodeAsync(_dictionarySize, _outWindow, rangeDecoder, cancellationToken) + .ConfigureAwait(false); + + await _outWindow.ReleaseStreamAsync(cancellationToken).ConfigureAwait(false); + rangeDecoder.ReleaseStream(); + + await _outWindow.DisposeAsync().ConfigureAwait(false); + _outWindow = null; + } + + internal async ValueTask CodeAsync( + int dictionarySize, + OutWindow outWindow, + RangeCoder.Decoder rangeDecoder, + CancellationToken cancellationToken = default + ) + { + var dictionarySizeCheck = Math.Max(dictionarySize, 1); + + await outWindow.CopyPendingAsync(cancellationToken).ConfigureAwait(false); + + while (outWindow.HasSpace) + { + cancellationToken.ThrowIfCancellationRequested(); + + var posState = (uint)outWindow.Total & _posStateMask; + if ( + await _isMatchDecoders[(_state._index << Base.K_NUM_POS_STATES_BITS_MAX) + posState] + .DecodeAsync(rangeDecoder, cancellationToken) + .ConfigureAwait(false) == 0 + ) + { + byte b; + var prevByte = outWindow.GetByte(0); + if (!_state.IsCharState()) + { + b = await _literalDecoder + .DecodeWithMatchByteAsync( + rangeDecoder, + (uint)outWindow.Total, + prevByte, + outWindow.GetByte((int)_rep0), + cancellationToken + ) + .ConfigureAwait(false); + } + else + { + b = await _literalDecoder + .DecodeNormalAsync( + rangeDecoder, + (uint)outWindow.Total, + prevByte, + cancellationToken + ) + .ConfigureAwait(false); + } + await outWindow.PutByteAsync(b, cancellationToken).ConfigureAwait(false); + _state.UpdateChar(); + } + else + { + uint len; + if ( + await _isRepDecoders[_state._index] + .DecodeAsync(rangeDecoder, cancellationToken) + .ConfigureAwait(false) == 1 + ) + { + if ( + await _isRepG0Decoders[_state._index] + .DecodeAsync(rangeDecoder, cancellationToken) + .ConfigureAwait(false) == 0 + ) + { + if ( + await _isRep0LongDecoders[ + (_state._index << Base.K_NUM_POS_STATES_BITS_MAX) + posState + ] + .DecodeAsync(rangeDecoder, cancellationToken) + .ConfigureAwait(false) == 0 + ) + { + _state.UpdateShortRep(); + await outWindow + .PutByteAsync(outWindow.GetByte((int)_rep0), cancellationToken) + .ConfigureAwait(false); + continue; + } + } + else + { + uint distance; + if ( + await _isRepG1Decoders[_state._index] + .DecodeAsync(rangeDecoder, cancellationToken) + .ConfigureAwait(false) == 0 + ) + { + distance = _rep1; + } + else + { + if ( + await _isRepG2Decoders[_state._index] + .DecodeAsync(rangeDecoder, cancellationToken) + .ConfigureAwait(false) == 0 + ) + { + distance = _rep2; + } + else + { + distance = _rep3; + _rep3 = _rep2; + } + _rep2 = _rep1; + } + _rep1 = _rep0; + _rep0 = distance; + } + len = + await _repLenDecoder + .DecodeAsync(rangeDecoder, posState, cancellationToken) + .ConfigureAwait(false) + Base.K_MATCH_MIN_LEN; + _state.UpdateRep(); + } + else + { + _rep3 = _rep2; + _rep2 = _rep1; + _rep1 = _rep0; + len = + Base.K_MATCH_MIN_LEN + + await _lenDecoder + .DecodeAsync(rangeDecoder, posState, cancellationToken) + .ConfigureAwait(false); + _state.UpdateMatch(); + var posSlot = await _posSlotDecoder[Base.GetLenToPosState(len)] + .DecodeAsync(rangeDecoder, cancellationToken) + .ConfigureAwait(false); + if (posSlot >= Base.K_START_POS_MODEL_INDEX) + { + var numDirectBits = (int)((posSlot >> 1) - 1); + _rep0 = ((2 | (posSlot & 1)) << numDirectBits); + if (posSlot < Base.K_END_POS_MODEL_INDEX) + { + _rep0 += await BitTreeDecoder + .ReverseDecodeAsync( + _posDecoders, + _rep0 - posSlot - 1, + rangeDecoder, + numDirectBits, + cancellationToken + ) + .ConfigureAwait(false); + } + else + { + _rep0 += ( + await rangeDecoder + .DecodeDirectBitsAsync( + numDirectBits - Base.K_NUM_ALIGN_BITS, + cancellationToken + ) + .ConfigureAwait(false) << Base.K_NUM_ALIGN_BITS + ); + _rep0 += await _posAlignDecoder + .ReverseDecodeAsync(rangeDecoder, cancellationToken) + .ConfigureAwait(false); + } + } + else + { + _rep0 = posSlot; + } + } + if (_rep0 >= outWindow.Total || _rep0 >= dictionarySizeCheck) + { + if (_rep0 == 0xFFFFFFFF) + { + return true; + } + throw new DataErrorException(); + } + await outWindow + .CopyBlockAsync((int)_rep0, (int)len, cancellationToken) + .ConfigureAwait(false); + } + } + return false; + } + + public async ValueTask TrainAsync(Stream stream) + { + if (_outWindow is null) + { + CreateDictionary(); + } + await _outWindow.TrainAsync(stream).ConfigureAwait(false); + } +} diff --git a/src/SharpCompress/Compressors/LZMA/LzmaDecoder.cs b/src/SharpCompress/Compressors/LZMA/LzmaDecoder.cs index 31039761..02fd16d7 100644 --- a/src/SharpCompress/Compressors/LZMA/LzmaDecoder.cs +++ b/src/SharpCompress/Compressors/LZMA/LzmaDecoder.cs @@ -1,21 +1,31 @@ #nullable disable using System; +using System.Diagnostics.CodeAnalysis; using System.IO; +using System.Threading.Tasks; using SharpCompress.Compressors.LZMA.LZ; using SharpCompress.Compressors.LZMA.RangeCoder; namespace SharpCompress.Compressors.LZMA; -public class Decoder : ICoder, ISetDecoderProperties // ,System.IO.Stream +public partial class Decoder : ICoder, ISetDecoderProperties, IDisposable { - private class LenDecoder + internal bool HasEndMarker => _rep0 == uint.MaxValue; + + public void Dispose() { - private BitDecoder _choice = new BitDecoder(); - private BitDecoder _choice2 = new BitDecoder(); + _outWindow?.Dispose(); + _outWindow = null; + } + + private partial class LenDecoder + { + private BitDecoder _choice = new(); + private BitDecoder _choice2 = new(); private readonly BitTreeDecoder[] _lowCoder = new BitTreeDecoder[Base.K_NUM_POS_STATES_MAX]; private readonly BitTreeDecoder[] _midCoder = new BitTreeDecoder[Base.K_NUM_POS_STATES_MAX]; - private BitTreeDecoder _highCoder = new BitTreeDecoder(Base.K_NUM_HIGH_LEN_BITS); + private BitTreeDecoder _highCoder = new(Base.K_NUM_HIGH_LEN_BITS); private uint _numPosStates; public void Create(uint numPosStates) @@ -60,19 +70,24 @@ public class Decoder : ICoder, ISetDecoderProperties // ,System.IO.Stream } } - private class LiteralDecoder + private partial class LiteralDecoder { - private struct Decoder2 + private partial struct Decoder2 { private BitDecoder[] _decoders; + private int _baseIndex; - public void Create() => _decoders = new BitDecoder[0x300]; + public void Create(BitDecoder[] decoders, int baseIndex) + { + _decoders = decoders; + _baseIndex = baseIndex; + } public void Init() { for (var i = 0; i < 0x300; i++) { - _decoders[i].Init(); + _decoders[_baseIndex + i].Init(); } } @@ -81,7 +96,7 @@ public class Decoder : ICoder, ISetDecoderProperties // ,System.IO.Stream uint symbol = 1; do { - symbol = (symbol << 1) | _decoders[symbol].Decode(rangeDecoder); + symbol = (symbol << 1) | _decoders[_baseIndex + symbol].Decode(rangeDecoder); } while (symbol < 0x100); return (byte)symbol; } @@ -93,13 +108,15 @@ public class Decoder : ICoder, ISetDecoderProperties // ,System.IO.Stream { var matchBit = (uint)(matchByte >> 7) & 1; matchByte <<= 1; - var bit = _decoders[((1 + matchBit) << 8) + symbol].Decode(rangeDecoder); + var bit = _decoders[_baseIndex + ((1 + matchBit) << 8) + symbol] + .Decode(rangeDecoder); symbol = (symbol << 1) | bit; if (matchBit != bit) { while (symbol < 0x100) { - symbol = (symbol << 1) | _decoders[symbol].Decode(rangeDecoder); + symbol = + (symbol << 1) | _decoders[_baseIndex + symbol].Decode(rangeDecoder); } break; } @@ -109,6 +126,7 @@ public class Decoder : ICoder, ISetDecoderProperties // ,System.IO.Stream } private Decoder2[] _coders; + private BitDecoder[] _models; private int _numPrevBits; private int _numPosBits; private uint _posMask; @@ -123,10 +141,11 @@ public class Decoder : ICoder, ISetDecoderProperties // ,System.IO.Stream _posMask = ((uint)1 << numPosBits) - 1; _numPrevBits = numPrevBits; var numStates = (uint)1 << (_numPrevBits + _numPosBits); + _models = new BitDecoder[checked((int)(numStates * 0x300))]; _coders = new Decoder2[numStates]; for (uint i = 0; i < numStates; i++) { - _coders[i].Create(); + _coders[i].Create(_models, checked((int)(i * 0x300))); } } @@ -173,18 +192,18 @@ public class Decoder : ICoder, ISetDecoderProperties // ,System.IO.Stream Base.K_NUM_FULL_DISTANCES - Base.K_END_POS_MODEL_INDEX ]; - private BitTreeDecoder _posAlignDecoder = new BitTreeDecoder(Base.K_NUM_ALIGN_BITS); + private BitTreeDecoder _posAlignDecoder = new(Base.K_NUM_ALIGN_BITS); - private readonly LenDecoder _lenDecoder = new LenDecoder(); - private readonly LenDecoder _repLenDecoder = new LenDecoder(); + private readonly LenDecoder _lenDecoder = new(); + private readonly LenDecoder _repLenDecoder = new(); - private readonly LiteralDecoder _literalDecoder = new LiteralDecoder(); + private readonly LiteralDecoder _literalDecoder = new(); private int _dictionarySize; private uint _posStateMask; - private Base.State _state = new Base.State(); + private Base.State _state = new(); private uint _rep0, _rep1, _rep2, @@ -199,6 +218,9 @@ public class Decoder : ICoder, ISetDecoderProperties // ,System.IO.Stream } } +#if !LEGACY_DOTNET + [MemberNotNull(nameof(_outWindow))] +#endif private void CreateDictionary() { if (_dictionarySize < 0) @@ -294,7 +316,7 @@ public class Decoder : ICoder, ISetDecoderProperties // ,System.IO.Stream } else { - _outWindow.SetLimit(long.MaxValue - _outWindow._total); + _outWindow.SetLimit(long.MaxValue - _outWindow.Total); } var rangeDecoder = new RangeCoder.Decoder(); @@ -305,6 +327,7 @@ public class Decoder : ICoder, ISetDecoderProperties // ,System.IO.Stream _outWindow.ReleaseStream(); rangeDecoder.ReleaseStream(); + _outWindow.Dispose(); _outWindow = null; } @@ -316,11 +339,10 @@ public class Decoder : ICoder, ISetDecoderProperties // ,System.IO.Stream while (outWindow.HasSpace) { - var posState = (uint)outWindow._total & _posStateMask; + var posState = (uint)outWindow.Total & _posStateMask; if ( - _isMatchDecoders[ - (_state._index << Base.K_NUM_POS_STATES_BITS_MAX) + posState - ].Decode(rangeDecoder) == 0 + _isMatchDecoders[(_state._index << Base.K_NUM_POS_STATES_BITS_MAX) + posState] + .Decode(rangeDecoder) == 0 ) { byte b; @@ -329,18 +351,14 @@ public class Decoder : ICoder, ISetDecoderProperties // ,System.IO.Stream { b = _literalDecoder.DecodeWithMatchByte( rangeDecoder, - (uint)outWindow._total, + (uint)outWindow.Total, prevByte, outWindow.GetByte((int)_rep0) ); } else { - b = _literalDecoder.DecodeNormal( - rangeDecoder, - (uint)outWindow._total, - prevByte - ); + b = _literalDecoder.DecodeNormal(rangeDecoder, (uint)outWindow.Total, prevByte); } outWindow.PutByte(b); _state.UpdateChar(); @@ -355,7 +373,8 @@ public class Decoder : ICoder, ISetDecoderProperties // ,System.IO.Stream if ( _isRep0LongDecoders[ (_state._index << Base.K_NUM_POS_STATES_BITS_MAX) + posState - ].Decode(rangeDecoder) == 0 + ] + .Decode(rangeDecoder) == 0 ) { _state.UpdateShortRep(); @@ -424,7 +443,7 @@ public class Decoder : ICoder, ISetDecoderProperties // ,System.IO.Stream _rep0 = posSlot; } } - if (_rep0 >= outWindow._total || _rep0 >= dictionarySizeCheck) + if (_rep0 >= outWindow.Total || _rep0 >= dictionarySizeCheck) { if (_rep0 == 0xFFFFFFFF) { @@ -438,7 +457,10 @@ public class Decoder : ICoder, ISetDecoderProperties // ,System.IO.Stream return false; } - public void SetDecoderProperties(byte[] properties) + public void SetDecoderProperties(byte[] properties) => + SetDecoderProperties(properties.AsSpan()); + + internal void SetDecoderProperties(ReadOnlySpan properties) { if (properties.Length < 1) { @@ -473,29 +495,4 @@ public class Decoder : ICoder, ISetDecoderProperties // ,System.IO.Stream } _outWindow.Train(stream); } - - /* - public override bool CanRead { get { return true; }} - public override bool CanWrite { get { return true; }} - public override bool CanSeek { get { return true; }} - public override long Length { get { return 0; }} - public override long Position - { - get { return 0; } - set { } - } - public override void Flush() { } - public override int Read(byte[] buffer, int offset, int count) - { - return 0; - } - public override void Write(byte[] buffer, int offset, int count) - { - } - public override long Seek(long offset, System.IO.SeekOrigin origin) - { - return 0; - } - public override void SetLength(long value) {} - */ } diff --git a/src/SharpCompress/Compressors/LZMA/LzmaEncoder.cs b/src/SharpCompress/Compressors/LZMA/LzmaEncoder.cs index 558fe37b..c8d5279b 100644 --- a/src/SharpCompress/Compressors/LZMA/LzmaEncoder.cs +++ b/src/SharpCompress/Compressors/LZMA/LzmaEncoder.cs @@ -1,18 +1,22 @@ #nullable disable using System; +using System.Buffers; using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; using SharpCompress.Compressors.LZMA.LZ; using SharpCompress.Compressors.LZMA.RangeCoder; namespace SharpCompress.Compressors.LZMA; -internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties +internal partial class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties, IDisposable { private enum EMatchFinderType { Bt2, - Bt4 + Bt4, } private const uint K_IFINITY_PRICE = 0xFFFFFFF; @@ -61,7 +65,7 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties return (uint)(G_FAST_POS[pos >> 26] + 52); } - private Base.State _state = new Base.State(); + private Base.State _state = new(); private byte _previousByte; private readonly uint[] _repDistances = new uint[Base.K_NUM_REP_DISTANCES]; @@ -78,19 +82,24 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties private const int K_DEFAULT_DICTIONARY_LOG_SIZE = 22; private const uint K_NUM_FAST_BYTES_DEFAULT = 0x20; - private class LiteralEncoder + private partial class LiteralEncoder { - public struct Encoder2 + public partial struct Encoder2 { private BitEncoder[] _encoders; + private int _baseIndex; - public void Create() => _encoders = new BitEncoder[0x300]; + public void Create(BitEncoder[] encoders, int baseIndex) + { + _encoders = encoders; + _baseIndex = baseIndex; + } public void Init() { for (var i = 0; i < 0x300; i++) { - _encoders[i].Init(); + _encoders[_baseIndex + i].Init(); } } @@ -100,7 +109,24 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties for (var i = 7; i >= 0; i--) { var bit = (uint)((symbol >> i) & 1); - _encoders[context].Encode(rangeEncoder, bit); + _encoders[_baseIndex + context].Encode(rangeEncoder, bit); + context = (context << 1) | bit; + } + } + + public async ValueTask EncodeAsync( + RangeCoder.Encoder rangeEncoder, + byte symbol, + CancellationToken cancellationToken = default + ) + { + uint context = 1; + for (var i = 7; i >= 0; i--) + { + var bit = (uint)((symbol >> i) & 1); + await _encoders[_baseIndex + context] + .EncodeAsync(rangeEncoder, bit, cancellationToken) + .ConfigureAwait(false); context = (context << 1) | bit; } } @@ -119,7 +145,33 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties state += ((1 + matchBit) << 8); same = (matchBit == bit); } - _encoders[state].Encode(rangeEncoder, bit); + _encoders[_baseIndex + state].Encode(rangeEncoder, bit); + context = (context << 1) | bit; + } + } + + public async ValueTask EncodeMatchedAsync( + RangeCoder.Encoder rangeEncoder, + byte matchByte, + byte symbol, + CancellationToken cancellationToken = default + ) + { + uint context = 1; + var same = true; + for (var i = 7; i >= 0; i--) + { + var bit = (uint)((symbol >> i) & 1); + var state = context; + if (same) + { + var matchBit = (uint)((matchByte >> i) & 1); + state += ((1 + matchBit) << 8); + same = matchBit == bit; + } + await _encoders[_baseIndex + state] + .EncodeAsync(rangeEncoder, bit, cancellationToken) + .ConfigureAwait(false); context = (context << 1) | bit; } } @@ -135,7 +187,8 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties { var matchBit = (uint)(matchByte >> i) & 1; var bit = (uint)(symbol >> i) & 1; - price += _encoders[((1 + matchBit) << 8) + context].GetPrice(bit); + price += _encoders[_baseIndex + ((1 + matchBit) << 8) + context] + .GetPrice(bit); context = (context << 1) | bit; if (matchBit != bit) { @@ -147,7 +200,7 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties for (; i >= 0; i--) { var bit = (uint)(symbol >> i) & 1; - price += _encoders[context].GetPrice(bit); + price += _encoders[_baseIndex + context].GetPrice(bit); context = (context << 1) | bit; } return price; @@ -155,6 +208,7 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties } private Encoder2[] _coders; + private BitEncoder[] _models; private int _numPrevBits; private int _numPosBits; private uint _posMask; @@ -169,13 +223,37 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties _posMask = ((uint)1 << numPosBits) - 1; _numPrevBits = numPrevBits; var numStates = (uint)1 << (_numPrevBits + _numPosBits); - _coders = new Encoder2[numStates]; + var requiredModelLength = checked((int)(numStates * 0x300)); + if (_models is null || _models.Length < requiredModelLength) + { + if (_models is not null) + { + ArrayPool.Shared.Return(_models); + } + _models = ArrayPool.Shared.Rent(requiredModelLength); + } + if (_coders is null || _coders.Length != numStates) + { + _coders = new Encoder2[numStates]; + } for (uint i = 0; i < numStates; i++) { - _coders[i].Create(); + _coders[i].Create(_models, checked((int)(i * 0x300))); } } + public void Dispose() + { + if (_models is null) + { + return; + } + + ArrayPool.Shared.Return(_models); + _models = null; + _coders = null; + } + public void Init() { var numStates = (uint)1 << (_numPrevBits + _numPosBits); @@ -189,17 +267,17 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties _coders[((pos & _posMask) << _numPrevBits) + (uint)(prevByte >> (8 - _numPrevBits))]; } - private class LenEncoder + private partial class LenEncoder { - private BitEncoder _choice = new BitEncoder(); - private BitEncoder _choice2 = new BitEncoder(); + private BitEncoder _choice = new(); + private BitEncoder _choice2 = new(); private readonly BitTreeEncoder[] _lowCoder = new BitTreeEncoder[ Base.K_NUM_POS_STATES_ENCODING_MAX ]; private readonly BitTreeEncoder[] _midCoder = new BitTreeEncoder[ Base.K_NUM_POS_STATES_ENCODING_MAX ]; - private BitTreeEncoder _highCoder = new BitTreeEncoder(Base.K_NUM_HIGH_LEN_BITS); + private BitTreeEncoder _highCoder = new(Base.K_NUM_HIGH_LEN_BITS); public LenEncoder() { @@ -246,6 +324,49 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties } } + public async ValueTask EncodeAsync( + RangeCoder.Encoder rangeEncoder, + uint symbol, + uint posState, + CancellationToken cancellationToken = default + ) + { + if (symbol < Base.K_NUM_LOW_LEN_SYMBOLS) + { + await _choice.EncodeAsync(rangeEncoder, 0, cancellationToken).ConfigureAwait(false); + await _lowCoder[posState] + .EncodeAsync(rangeEncoder, symbol, cancellationToken) + .ConfigureAwait(false); + } + else + { + symbol -= Base.K_NUM_LOW_LEN_SYMBOLS; + await _choice.EncodeAsync(rangeEncoder, 1, cancellationToken).ConfigureAwait(false); + if (symbol < Base.K_NUM_MID_LEN_SYMBOLS) + { + await _choice2 + .EncodeAsync(rangeEncoder, 0, cancellationToken) + .ConfigureAwait(false); + await _midCoder[posState] + .EncodeAsync(rangeEncoder, symbol, cancellationToken) + .ConfigureAwait(false); + } + else + { + await _choice2 + .EncodeAsync(rangeEncoder, 1, cancellationToken) + .ConfigureAwait(false); + await _highCoder + .EncodeAsync( + rangeEncoder, + symbol - Base.K_NUM_MID_LEN_SYMBOLS, + cancellationToken + ) + .ConfigureAwait(false); + } + } + } + public void SetPrices(uint posState, uint numSymbols, uint[] prices, uint st) { var a0 = _choice.GetPrice0(); @@ -280,10 +401,7 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties } } - private const uint K_NUM_LEN_SPEC_SYMBOLS = - Base.K_NUM_LOW_LEN_SYMBOLS + Base.K_NUM_MID_LEN_SYMBOLS; - - private class LenPriceTableEncoder : LenEncoder + private partial class LenPriceTableEncoder : LenEncoder { private readonly uint[] _prices = new uint[ Base.K_NUM_LEN_SYMBOLS << Base.K_NUM_POS_STATES_BITS_ENCODING_MAX @@ -318,11 +436,26 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties UpdateTable(posState); } } + + public new async ValueTask EncodeAsync( + RangeCoder.Encoder rangeEncoder, + uint symbol, + uint posState, + CancellationToken cancellationToken = default + ) + { + await base.EncodeAsync(rangeEncoder, symbol, posState, cancellationToken) + .ConfigureAwait(false); + if (--_counters[posState] == 0) + { + UpdateTable(posState); + } + } } private const uint K_NUM_OPTS = 1 << 12; - private class Optimal + private struct Optimal { public Base.State _state; @@ -359,7 +492,7 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties private readonly Optimal[] _optimum = new Optimal[K_NUM_OPTS]; private BinTree _matchFinder; - private readonly RangeCoder.Encoder _rangeEncoder = new RangeCoder.Encoder(); + private readonly RangeCoder.Encoder _rangeEncoder = new(); private readonly BitEncoder[] _isMatch = new BitEncoder[ Base.K_NUM_STATES << Base.K_NUM_POS_STATES_BITS_MAX @@ -382,12 +515,12 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties Base.K_NUM_FULL_DISTANCES - Base.K_END_POS_MODEL_INDEX ]; - private BitTreeEncoder _posAlignEncoder = new BitTreeEncoder(Base.K_NUM_ALIGN_BITS); + private BitTreeEncoder _posAlignEncoder = new(Base.K_NUM_ALIGN_BITS); - private readonly LenPriceTableEncoder _lenEncoder = new LenPriceTableEncoder(); - private readonly LenPriceTableEncoder _repMatchLenEncoder = new LenPriceTableEncoder(); + private readonly LenPriceTableEncoder _lenEncoder = new(); + private readonly LenPriceTableEncoder _repMatchLenEncoder = new(); - private readonly LiteralEncoder _literalEncoder = new LiteralEncoder(); + private readonly LiteralEncoder _literalEncoder = new(); private readonly uint[] _matchDistances = new uint[(Base.K_MATCH_MAX_LEN * 2) + 2]; @@ -463,16 +596,19 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties public Encoder() { - for (var i = 0; i < K_NUM_OPTS; i++) - { - _optimum[i] = new Optimal(); - } for (var i = 0; i < Base.K_NUM_LEN_TO_POS_STATES; i++) { _posSlotEncoder[i] = new BitTreeEncoder(Base.K_NUM_POS_SLOT_BITS); } } + public void Dispose() + { + _literalEncoder.Dispose(); + _matchFinder?.Dispose(); + _matchFinder = null; + } + private void SetWriteEndMarkerMode(bool writeEndMarker) => _writeEndMark = writeEndMarker; private void Init() @@ -553,9 +689,8 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties if (repIndex == 0) { price = _isRepG0[state._index].GetPrice0(); - price += _isRep0Long[ - (state._index << Base.K_NUM_POS_STATES_BITS_MAX) + posState - ].GetPrice1(); + price += _isRep0Long[(state._index << Base.K_NUM_POS_STATES_BITS_MAX) + posState] + .GetPrice1(); } else { @@ -713,9 +848,8 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties .GetPrice(!_state.IsCharState(), matchByte, currentByte); _optimum[1].MakeAsChar(); - var matchPrice = _isMatch[ - (_state._index << Base.K_NUM_POS_STATES_BITS_MAX) + posState - ].GetPrice1(); + var matchPrice = _isMatch[(_state._index << Base.K_NUM_POS_STATES_BITS_MAX) + posState] + .GetPrice1(); var repMatchPrice = matchPrice + _isRep[_state._index].GetPrice1(); if (matchByte == currentByte) @@ -760,7 +894,7 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties do { var curAndLenPrice = price + _repMatchLenEncoder.GetPrice(repLen - 2, posState); - var optimum = _optimum[repLen]; + ref var optimum = ref _optimum[repLen]; if (curAndLenPrice < optimum._price) { optimum._price = curAndLenPrice; @@ -785,7 +919,7 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties { var distance = _matchDistances[offs + 1]; var curAndLenPrice = normalMatchPrice + GetPosLenPrice(distance, len, posState); - var optimum = _optimum[len]; + ref var optimum = ref _optimum[len]; if (curAndLenPrice < optimum._price) { optimum._price = curAndLenPrice; @@ -940,7 +1074,7 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties .GetSubCoder(position, _matchFinder.GetIndexByte(0 - 2)) .GetPrice(!state.IsCharState(), matchByte, currentByte); - var nextOptimum = _optimum[cur + 1]; + ref var nextOptimum = ref _optimum[cur + 1]; var nextIsChar = false; if (curAnd1Price < nextOptimum._price) @@ -995,9 +1129,8 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties var posStateNext = (position + 1) & _posStateMask; var nextRepMatchPrice = curAnd1Price - + _isMatch[ - (state2._index << Base.K_NUM_POS_STATES_BITS_MAX) + posStateNext - ].GetPrice1() + + _isMatch[(state2._index << Base.K_NUM_POS_STATES_BITS_MAX) + posStateNext] + .GetPrice1() + _isRep[state2._index].GetPrice1(); { var offset = cur + 1 + lenTest2; @@ -1007,7 +1140,7 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties } var curAndLenPrice = nextRepMatchPrice + GetRepPrice(0, lenTest2, state2, posStateNext); - var optimum = _optimum[offset]; + ref var optimum = ref _optimum[offset]; if (curAndLenPrice < optimum._price) { optimum._price = curAndLenPrice; @@ -1038,7 +1171,7 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties } var curAndLenPrice = repMatchPrice + GetRepPrice(repIndex, lenTest, state, posState); - var optimum = _optimum[cur + lenTest]; + ref var optimum = ref _optimum[cur + lenTest]; if (curAndLenPrice < optimum._price) { optimum._price = curAndLenPrice; @@ -1069,7 +1202,8 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties + GetRepPrice(repIndex, lenTest, state, posState) + _isMatch[ (state2._index << Base.K_NUM_POS_STATES_BITS_MAX) + posStateNext - ].GetPrice0() + ] + .GetPrice0() + _literalEncoder .GetSubCoder( position + lenTest, @@ -1088,7 +1222,8 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties curAndLenCharPrice + _isMatch[ (state2._index << Base.K_NUM_POS_STATES_BITS_MAX) + posStateNext - ].GetPrice1(); + ] + .GetPrice1(); var nextRepMatchPrice = nextMatchPrice + _isRep[state2._index].GetPrice1(); // for(; lenTest2 >= 2; lenTest2--) @@ -1100,7 +1235,7 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties } var curAndLenPrice = nextRepMatchPrice + GetRepPrice(0, lenTest2, state2, posStateNext); - var optimum = _optimum[cur + offset]; + ref var optimum = ref _optimum[cur + offset]; if (curAndLenPrice < optimum._price) { optimum._price = curAndLenPrice; @@ -1149,7 +1284,7 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties var curBack = _matchDistances[offs + 1]; var curAndLenPrice = normalMatchPrice + GetPosLenPrice(curBack, lenTest, posState); - var optimum = _optimum[cur + lenTest]; + ref var optimum = ref _optimum[cur + lenTest]; if (curAndLenPrice < optimum._price) { optimum._price = curAndLenPrice; @@ -1174,7 +1309,8 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties + _isMatch[ (state2._index << Base.K_NUM_POS_STATES_BITS_MAX) + posStateNext - ].GetPrice0() + ] + .GetPrice0() + _literalEncoder .GetSubCoder( position + lenTest, @@ -1194,7 +1330,8 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties + _isMatch[ (state2._index << Base.K_NUM_POS_STATES_BITS_MAX) + posStateNext - ].GetPrice1(); + ] + .GetPrice1(); var nextRepMatchPrice = nextMatchPrice + _isRep[state2._index].GetPrice1(); @@ -1206,7 +1343,7 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties curAndLenPrice = nextRepMatchPrice + GetRepPrice(0, lenTest2, state2, posStateNext); - optimum = _optimum[cur + offset]; + optimum = ref _optimum[cur + offset]; if (curAndLenPrice < optimum._price) { optimum._price = curAndLenPrice; @@ -1230,12 +1367,6 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties } } - private bool ChangePair(uint smallDist, uint bigDist) - { - const int kDif = 7; - return (smallDist < ((uint)(1) << (32 - kDif)) && bigDist >= (smallDist << kDif)); - } - private void WriteEndMarker(uint posState) { if (!_writeEndMark) @@ -1243,10 +1374,8 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties return; } - _isMatch[(_state._index << Base.K_NUM_POS_STATES_BITS_MAX) + posState].Encode( - _rangeEncoder, - 1 - ); + _isMatch[(_state._index << Base.K_NUM_POS_STATES_BITS_MAX) + posState] + .Encode(_rangeEncoder, 1); _isRep[_state._index].Encode(_rangeEncoder, 0); _state.UpdateMatch(); var len = Base.K_MATCH_MIN_LEN; @@ -1263,6 +1392,46 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties _posAlignEncoder.ReverseEncode(_rangeEncoder, posReduced & Base.K_ALIGN_MASK); } + private async ValueTask WriteEndMarkerAsync( + uint posState, + CancellationToken cancellationToken = default + ) + { + if (!_writeEndMark) + { + return; + } + + await _isMatch[(_state._index << Base.K_NUM_POS_STATES_BITS_MAX) + posState] + .EncodeAsync(_rangeEncoder, 1, cancellationToken) + .ConfigureAwait(false); + await _isRep[_state._index] + .EncodeAsync(_rangeEncoder, 0, cancellationToken) + .ConfigureAwait(false); + _state.UpdateMatch(); + var len = Base.K_MATCH_MIN_LEN; + await _lenEncoder + .EncodeAsync(_rangeEncoder, len - Base.K_MATCH_MIN_LEN, posState, cancellationToken) + .ConfigureAwait(false); + uint posSlot = (1 << Base.K_NUM_POS_SLOT_BITS) - 1; + var lenToPosState = Base.GetLenToPosState(len); + await _posSlotEncoder[lenToPosState] + .EncodeAsync(_rangeEncoder, posSlot, cancellationToken) + .ConfigureAwait(false); + var footerBits = 30; + var posReduced = (((uint)1) << footerBits) - 1; + await _rangeEncoder + .EncodeDirectBitsAsync( + posReduced >> Base.K_NUM_ALIGN_BITS, + footerBits - Base.K_NUM_ALIGN_BITS, + cancellationToken + ) + .ConfigureAwait(false); + await _posAlignEncoder + .ReverseEncodeAsync(_rangeEncoder, posReduced & Base.K_ALIGN_MASK, cancellationToken) + .ConfigureAwait(false); + } + private void Flush(uint nowPos) { ReleaseMfStream(); @@ -1271,6 +1440,17 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties _rangeEncoder.FlushStream(); } + private async ValueTask FlushAsync(uint nowPos, CancellationToken cancellationToken = default) + { + ReleaseMfStream(); + await WriteEndMarkerAsync(nowPos & _posStateMask, cancellationToken).ConfigureAwait(false); + for (var i = 0; i < 5; i++) + { + await _rangeEncoder.ShiftLowAsync(cancellationToken).ConfigureAwait(false); + } + await _rangeEncoder.FlushStreamAsync(cancellationToken).ConfigureAwait(false); + } + public void CodeOneBlock(out long inSize, out long outSize, out bool finished) { inSize = 0; @@ -1321,10 +1501,8 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties // it's not used ReadMatchDistances(out var len, out var numDistancePairs); var posState = (uint)(_nowPos64) & _posStateMask; - _isMatch[(_state._index << Base.K_NUM_POS_STATES_BITS_MAX) + posState].Encode( - _rangeEncoder, - 0 - ); + _isMatch[(_state._index << Base.K_NUM_POS_STATES_BITS_MAX) + posState] + .Encode(_rangeEncoder, 0); _state.UpdateChar(); var curByte = _matchFinder.GetIndexByte((int)(0 - _additionalOffset)); _literalEncoder @@ -1514,6 +1692,290 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties } } + public async ValueTask<(long inSize, long outSize, bool finished)> CodeOneBlockAsync( + CancellationToken cancellationToken = default + ) + { + long inSize = 0; + long outSize = 0; + var finished = true; + + if (_inStream != null) + { + _matchFinder.SetStream(_inStream); + _needReleaseMfStream = true; + _inStream = null; + } + + if (_finished) + { + return (inSize, outSize, finished); + } + _finished = true; + + var progressPosValuePrev = _nowPos64; + if (_nowPos64 == 0) + { + if (_trainSize > 0) + { + for ( + ; + _trainSize > 0 && (!_processingMode || !_matchFinder.IsDataStarved); + _trainSize-- + ) + { + _matchFinder.Skip(1); + } + if (_trainSize == 0) + { + _previousByte = _matchFinder.GetIndexByte(-1); + } + } + if (_processingMode && _matchFinder.IsDataStarved) + { + _finished = false; + return (inSize, outSize, finished); + } + if (_matchFinder.GetNumAvailableBytes() == 0) + { + await FlushAsync((uint)_nowPos64, cancellationToken).ConfigureAwait(false); + return (inSize, outSize, finished); + } + + ReadMatchDistances(out var len, out var numDistancePairs); + var posState = (uint)_nowPos64 & _posStateMask; + await _isMatch[(_state._index << Base.K_NUM_POS_STATES_BITS_MAX) + posState] + .EncodeAsync(_rangeEncoder, 0, cancellationToken) + .ConfigureAwait(false); + _state.UpdateChar(); + var curByte = _matchFinder.GetIndexByte((int)(0 - _additionalOffset)); + await _literalEncoder + .GetSubCoder((uint)_nowPos64, _previousByte) + .EncodeAsync(_rangeEncoder, curByte, cancellationToken) + .ConfigureAwait(false); + _previousByte = curByte; + _additionalOffset--; + _nowPos64++; + } + if (_processingMode && _matchFinder.IsDataStarved) + { + _finished = false; + return (inSize, outSize, finished); + } + if (_matchFinder.GetNumAvailableBytes() == 0) + { + await FlushAsync((uint)_nowPos64, cancellationToken).ConfigureAwait(false); + return (inSize, outSize, finished); + } + while (true) + { + cancellationToken.ThrowIfCancellationRequested(); + + if (_processingMode && _matchFinder.IsDataStarved) + { + _finished = false; + return (inSize, outSize, finished); + } + + var len = GetOptimum((uint)_nowPos64, out var pos); + + var posState = (uint)_nowPos64 & _posStateMask; + var complexState = (_state._index << Base.K_NUM_POS_STATES_BITS_MAX) + posState; + if (len == 1 && pos == 0xFFFFFFFF) + { + await _isMatch[complexState] + .EncodeAsync(_rangeEncoder, 0, cancellationToken) + .ConfigureAwait(false); + var curByte = _matchFinder.GetIndexByte((int)(0 - _additionalOffset)); + var subCoder = _literalEncoder.GetSubCoder((uint)_nowPos64, _previousByte); + if (!_state.IsCharState()) + { + var matchByte = _matchFinder.GetIndexByte( + (int)(0 - _repDistances[0] - 1 - _additionalOffset) + ); + await subCoder + .EncodeMatchedAsync(_rangeEncoder, matchByte, curByte, cancellationToken) + .ConfigureAwait(false); + } + else + { + await subCoder + .EncodeAsync(_rangeEncoder, curByte, cancellationToken) + .ConfigureAwait(false); + } + _previousByte = curByte; + _state.UpdateChar(); + } + else + { + await _isMatch[complexState] + .EncodeAsync(_rangeEncoder, 1, cancellationToken) + .ConfigureAwait(false); + if (pos < Base.K_NUM_REP_DISTANCES) + { + await _isRep[_state._index] + .EncodeAsync(_rangeEncoder, 1, cancellationToken) + .ConfigureAwait(false); + if (pos == 0) + { + await _isRepG0[_state._index] + .EncodeAsync(_rangeEncoder, 0, cancellationToken) + .ConfigureAwait(false); + await _isRep0Long[complexState] + .EncodeAsync(_rangeEncoder, len == 1 ? 0u : 1u, cancellationToken) + .ConfigureAwait(false); + } + else + { + await _isRepG0[_state._index] + .EncodeAsync(_rangeEncoder, 1, cancellationToken) + .ConfigureAwait(false); + if (pos == 1) + { + await _isRepG1[_state._index] + .EncodeAsync(_rangeEncoder, 0, cancellationToken) + .ConfigureAwait(false); + } + else + { + await _isRepG1[_state._index] + .EncodeAsync(_rangeEncoder, 1, cancellationToken) + .ConfigureAwait(false); + await _isRepG2[_state._index] + .EncodeAsync(_rangeEncoder, pos - 2, cancellationToken) + .ConfigureAwait(false); + } + } + if (len == 1) + { + _state.UpdateShortRep(); + } + else + { + await _repMatchLenEncoder + .EncodeAsync( + _rangeEncoder, + len - Base.K_MATCH_MIN_LEN, + posState, + cancellationToken + ) + .ConfigureAwait(false); + _state.UpdateRep(); + } + var distance = _repDistances[pos]; + if (pos != 0) + { + for (var i = pos; i >= 1; i--) + { + _repDistances[i] = _repDistances[i - 1]; + } + _repDistances[0] = distance; + } + } + else + { + await _isRep[_state._index] + .EncodeAsync(_rangeEncoder, 0, cancellationToken) + .ConfigureAwait(false); + _state.UpdateMatch(); + await _lenEncoder + .EncodeAsync( + _rangeEncoder, + len - Base.K_MATCH_MIN_LEN, + posState, + cancellationToken + ) + .ConfigureAwait(false); + pos -= Base.K_NUM_REP_DISTANCES; + var posSlot = GetPosSlot(pos); + var lenToPosState = Base.GetLenToPosState(len); + await _posSlotEncoder[lenToPosState] + .EncodeAsync(_rangeEncoder, posSlot, cancellationToken) + .ConfigureAwait(false); + + if (posSlot >= Base.K_START_POS_MODEL_INDEX) + { + var footerBits = (int)((posSlot >> 1) - 1); + var baseVal = ((2 | (posSlot & 1)) << footerBits); + var posReduced = pos - baseVal; + + if (posSlot < Base.K_END_POS_MODEL_INDEX) + { + await BitTreeEncoder + .ReverseEncodeAsync( + _posEncoders, + baseVal - posSlot - 1, + _rangeEncoder, + footerBits, + posReduced, + cancellationToken + ) + .ConfigureAwait(false); + } + else + { + await _rangeEncoder + .EncodeDirectBitsAsync( + posReduced >> Base.K_NUM_ALIGN_BITS, + footerBits - Base.K_NUM_ALIGN_BITS, + cancellationToken + ) + .ConfigureAwait(false); + await _posAlignEncoder + .ReverseEncodeAsync( + _rangeEncoder, + posReduced & Base.K_ALIGN_MASK, + cancellationToken + ) + .ConfigureAwait(false); + _alignPriceCount++; + } + } + var distance = pos; + for (var i = Base.K_NUM_REP_DISTANCES - 1; i >= 1; i--) + { + _repDistances[i] = _repDistances[i - 1]; + } + _repDistances[0] = distance; + _matchPriceCount++; + } + _previousByte = _matchFinder.GetIndexByte((int)(len - 1 - _additionalOffset)); + } + _additionalOffset -= len; + _nowPos64 += len; + if (_additionalOffset == 0) + { + if (_matchPriceCount >= (1 << 7)) + { + FillDistancesPrices(); + } + if (_alignPriceCount >= Base.K_ALIGN_TABLE_SIZE) + { + FillAlignPrices(); + } + inSize = _nowPos64; + outSize = _rangeEncoder.GetProcessedSizeAdd(); + if (_processingMode && _matchFinder.IsDataStarved) + { + _finished = false; + return (inSize, outSize, finished); + } + if (_matchFinder.GetNumAvailableBytes() == 0) + { + await FlushAsync((uint)_nowPos64, cancellationToken).ConfigureAwait(false); + return (inSize, outSize, finished); + } + + if (_nowPos64 - progressPosValuePrev >= (1 << 12)) + { + _finished = false; + finished = false; + return (inSize, outSize, finished); + } + } + } + } + private void ReleaseMfStream() { if (_matchFinder != null && _needReleaseMfStream) @@ -1610,11 +2072,41 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties } } + public async ValueTask CodeAsync( + Stream inStream, + bool final, + CancellationToken cancellationToken = default + ) + { + _matchFinder.SetStream(inStream); + _processingMode = !final; + try + { + while (true) + { + var (processedInSize, _, finished) = await CodeOneBlockAsync(cancellationToken) + .ConfigureAwait(false); + if (finished) + { + return processedInSize; + } + } + } + finally + { + _matchFinder.ReleaseStream(); + if (final) + { + ReleaseStreams(); + } + } + } + public void Train(Stream trainStream) { if (_nowPos64 > 0) { - throw new InvalidOperationException(); + throw new InvalidFormatException(); } _trainSize = (uint)trainStream.Length; if (_trainSize > 0) @@ -1730,7 +2222,7 @@ internal class Encoder : ICoder, ISetCoderProperties, IWriteCoderProperties ReadOnlySpan properties ) { - for (int i = 0; i < properties.Length; i++) + for (var i = 0; i < properties.Length; i++) { var prop = properties[i]; switch (propIDs[i]) diff --git a/src/SharpCompress/Compressors/LZMA/LzmaEncoderProperties.cs b/src/SharpCompress/Compressors/LZMA/LzmaEncoderProperties.cs index 274e757b..8076ac08 100644 --- a/src/SharpCompress/Compressors/LZMA/LzmaEncoderProperties.cs +++ b/src/SharpCompress/Compressors/LZMA/LzmaEncoderProperties.cs @@ -12,6 +12,16 @@ public class LzmaEncoderProperties internal ReadOnlySpan Properties => _properties; private readonly object[] _properties; + /// + /// The dictionary size configured for this encoder. + /// + internal int DictionarySize { get; } + + /// + /// The number of fast bytes configured for this encoder. + /// + internal int NumFastBytes { get; } + public LzmaEncoderProperties() : this(false) { } @@ -23,6 +33,8 @@ public class LzmaEncoderProperties public LzmaEncoderProperties(bool eos, int dictionary, int numFastBytes) { + DictionarySize = dictionary; + NumFastBytes = numFastBytes; var posStateBits = 2; var litContextBits = 3; var litPosBits = 0; @@ -38,7 +50,7 @@ public class LzmaEncoderProperties CoderPropId.Algorithm, CoderPropId.NumFastBytes, CoderPropId.MatchFinder, - CoderPropId.EndMarker + CoderPropId.EndMarker, }; _properties = new object[] { @@ -49,7 +61,7 @@ public class LzmaEncoderProperties algorithm, numFastBytes, mf, - eos + eos, }; } } diff --git a/src/SharpCompress/Compressors/LZMA/LzmaStream.Async.cs b/src/SharpCompress/Compressors/LZMA/LzmaStream.Async.cs new file mode 100644 index 00000000..c89fb923 --- /dev/null +++ b/src/SharpCompress/Compressors/LZMA/LzmaStream.Async.cs @@ -0,0 +1,472 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.IO; + +namespace SharpCompress.Compressors.LZMA; + +public partial class LzmaStream +{ + public static ValueTask CreateAsync( + byte[] properties, + Stream inputStream, + long inputSize, + long outputSize, + bool leaveOpen = false + ) => + CreateAsync( + properties, + inputStream, + inputSize, + outputSize, + null, + properties.Length < 5, + leaveOpen + ); + + public static async ValueTask CreateAsync( + byte[] properties, + Stream inputStream, + long inputSize, + long outputSize, + Stream? presetDictionary, + bool isLzma2, + bool leaveOpen = false + ) + { + var lzma = new LzmaStream( + properties, + inputStream, + inputSize, + outputSize, + isLzma2, + leaveOpen + ); + if (!isLzma2) + { + if (presetDictionary != null) + { + await lzma._outWindow.TrainAsync(presetDictionary).ConfigureAwait(false); + } + + await lzma._rangeDecoder.InitAsync(inputStream).ConfigureAwait(false); + } + else + { + if (presetDictionary != null) + { + await lzma._outWindow.TrainAsync(presetDictionary).ConfigureAwait(false); + lzma._needDictReset = false; + } + } + return lzma; + } + + /*public static async ValueTask CreateAsync( + LzmaEncoderProperties properties, + bool isLzma2, + Stream? presetDictionary, + Stream outputStream + ) + { + var lzma = new LzmaStream(properties, isLzma2, presetDictionary); + + lzma._encoder!.SetStreams(null, outputStream, -1, -1); + + if (presetDictionary != null) + { + lzma._encoder.Train(presetDictionary); + } + return lzma; + }*/ + + private async ValueTask DecodeChunkHeaderAsync(CancellationToken cancellationToken = default) + { + var headerBuffer = GetAsyncHeaderBuffer(); + await _inputStream! + .ReadExactAsync(headerBuffer, 0, 1, cancellationToken) + .ConfigureAwait(false); + var control = headerBuffer[0]; + _inputPosition++; + + if (control == 0x00) + { + if (_isLzma2 && _decoder is { HasEndMarker: true }) + { + throw new DataErrorException(); + } + + _endReached = true; + return; + } + + if (control >= 0xE0 || control == 0x01) + { + _needProps = true; + _needDictReset = false; + _outWindow.Reset(); + } + else if (_needDictReset) + { + throw new DataErrorException(); + } + + if (control >= 0x80) + { + _uncompressedChunk = false; + + _availableBytes = (control & 0x1F) << 16; + await _inputStream! + .ReadExactAsync(headerBuffer, 0, 2, cancellationToken) + .ConfigureAwait(false); + _availableBytes += (headerBuffer[0] << 8) + headerBuffer[1] + 1; + _inputPosition += 2; + + await _inputStream! + .ReadExactAsync(headerBuffer, 0, 2, cancellationToken) + .ConfigureAwait(false); + _rangeDecoderLimit = (headerBuffer[0] << 8) + headerBuffer[1] + 1; + _inputPosition += 2; + + if (control >= 0xC0) + { + _needProps = false; + await _inputStream! + .ReadExactAsync(headerBuffer, 0, 1, cancellationToken) + .ConfigureAwait(false); + Properties[0] = headerBuffer[0]; + _inputPosition++; + + _decoder = new Decoder(); + _decoder.SetDecoderProperties(Properties); + } + else if (_needProps) + { + throw new DataErrorException(); + } + else if (control >= 0xA0) + { + _decoder = new Decoder(); + _decoder.SetDecoderProperties(Properties); + } + + await _rangeDecoder.InitAsync(_inputStream, cancellationToken).ConfigureAwait(false); + } + else if (control > 0x02) + { + throw new DataErrorException(); + } + else + { + _uncompressedChunk = true; + await _inputStream! + .ReadExactAsync(headerBuffer, 0, 2, cancellationToken) + .ConfigureAwait(false); + _availableBytes = (headerBuffer[0] << 8) + headerBuffer[1] + 1; + _inputPosition += 2; + } + } + + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + if (_endReached) + { + return 0; + } + + var total = 0; + while (total < count) + { + cancellationToken.ThrowIfCancellationRequested(); + + if (_availableBytes == 0) + { + if (_isLzma2) + { + await DecodeChunkHeaderAsync(cancellationToken).ConfigureAwait(false); + } + else + { + _endReached = true; + } + if (_endReached) + { + break; + } + } + + var toProcess = count - total; + if (toProcess > _availableBytes) + { + toProcess = (int)_availableBytes; + } + + _outWindow.SetLimit(toProcess); + if (_uncompressedChunk) + { + _inputPosition += await _outWindow + .CopyStreamAsync(_inputStream, toProcess, cancellationToken) + .ConfigureAwait(false); + } + else if ( + await _decoder! + .CodeAsync(_dictionarySize, _outWindow, _rangeDecoder, cancellationToken) + .ConfigureAwait(false) + ) + { + HandleEndMarker(); + } + + var read = _outWindow.Read(buffer, offset, toProcess); + total += read; + offset += read; + _position += read; + _availableBytes -= read; + + if (_availableBytes == 0 && !_uncompressedChunk) + { + if (_isLzma2 && _decoder!.HasEndMarker) + { + throw new DataErrorException(); + } + + if ( + !_rangeDecoder.IsFinished + || (_rangeDecoderLimit >= 0 && _rangeDecoder._total != _rangeDecoderLimit) + ) + { + _outWindow.SetLimit(toProcess + 1); + if ( + !await _decoder! + .CodeAsync( + _dictionarySize, + _outWindow, + _rangeDecoder, + cancellationToken + ) + .ConfigureAwait(false) + ) + { + _rangeDecoder.ReleaseStream(); + throw new DataErrorException(); + } + } + + _rangeDecoder.ReleaseStream(); + + _inputPosition += _rangeDecoder._total; + if (_outWindow.HasPending) + { + throw new DataErrorException(); + } + } + } + + if (_endReached) + { + if (_inputSize >= 0 && _inputPosition != _inputSize) + { + throw new DataErrorException(); + } + if (_outputSize >= 0 && _position != _outputSize) + { + throw new DataErrorException(); + } + } + + return total; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + if (_endReached) + { + return 0; + } + + var total = 0; + var offset = 0; + var count = buffer.Length; + while (total < count) + { + cancellationToken.ThrowIfCancellationRequested(); + + if (_availableBytes == 0) + { + if (_isLzma2) + { + await DecodeChunkHeaderAsync(cancellationToken).ConfigureAwait(false); + } + else + { + _endReached = true; + } + if (_endReached) + { + break; + } + } + + var toProcess = count - total; + if (toProcess > _availableBytes) + { + toProcess = (int)_availableBytes; + } + + _outWindow.SetLimit(toProcess); + if (_uncompressedChunk) + { + _inputPosition += await _outWindow + .CopyStreamAsync(_inputStream, toProcess, cancellationToken) + .ConfigureAwait(false); + } + else if ( + await _decoder! + .CodeAsync(_dictionarySize, _outWindow, _rangeDecoder, cancellationToken) + .ConfigureAwait(false) + ) + { + HandleEndMarker(); + } + + var read = _outWindow.Read(buffer, offset, toProcess); + total += read; + offset += read; + _position += read; + _availableBytes -= read; + + if (_availableBytes == 0 && !_uncompressedChunk) + { + if (_isLzma2 && _decoder!.HasEndMarker) + { + throw new DataErrorException(); + } + + if ( + !_rangeDecoder.IsFinished + || (_rangeDecoderLimit >= 0 && _rangeDecoder._total != _rangeDecoderLimit) + ) + { + _outWindow.SetLimit(toProcess + 1); + if ( + !await _decoder! + .CodeAsync( + _dictionarySize, + _outWindow, + _rangeDecoder, + cancellationToken + ) + .ConfigureAwait(false) + ) + { + _rangeDecoder.ReleaseStream(); + throw new DataErrorException(); + } + } + + _rangeDecoder.ReleaseStream(); + + _inputPosition += _rangeDecoder._total; + if (_outWindow.HasPending) + { + throw new DataErrorException(); + } + } + } + + if (_endReached) + { + if (_inputSize >= 0 && _inputPosition != _inputSize) + { + throw new DataErrorException(); + } + if (_outputSize >= 0 && _position != _outputSize) + { + throw new DataErrorException(); + } + } + + return total; + } +#endif + + public override async Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + cancellationToken.ThrowIfCancellationRequested(); + if (_encoder != null) + { + _position = await _encoder + .CodeAsync(new MemoryStream(buffer, offset, count), false, cancellationToken) + .ConfigureAwait(false); + } + } + +#if !LEGACY_DOTNET + public override async ValueTask WriteAsync( + ReadOnlyMemory buffer, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + if (_encoder != null) + { + _position = await _encoder + .CodeAsync(new MemoryStream(buffer.ToArray()), false, cancellationToken) + .ConfigureAwait(false); + } + } +#endif + +#if !LEGACY_DOTNET || NETSTANDARD2_1 + public override async ValueTask DisposeAsync() +#else + public async ValueTask DisposeAsync() +#endif + { + if (_isDisposed) + { + return; + } + _isDisposed = true; + + if (_encoder != null) + { + _position = await _encoder.CodeAsync(null, true).ConfigureAwait(false); + _encoder.Dispose(); + } + + if (!_leaveOpen) + { + if (_inputStream is IAsyncDisposable asyncDisposableInputStream) + { + await asyncDisposableInputStream.DisposeAsync().ConfigureAwait(false); + } + else + { + _inputStream?.Dispose(); + } + } + await _outWindow.DisposeAsync().ConfigureAwait(false); + ReturnAsyncHeaderBuffer(); + +#if !LEGACY_DOTNET || NETSTANDARD2_1 + await base.DisposeAsync().ConfigureAwait(false); +#endif + GC.SuppressFinalize(this); + } +} diff --git a/src/SharpCompress/Compressors/LZMA/LzmaStream.cs b/src/SharpCompress/Compressors/LZMA/LzmaStream.cs index 701b8e64..f45b33a3 100644 --- a/src/SharpCompress/Compressors/LZMA/LzmaStream.cs +++ b/src/SharpCompress/Compressors/LZMA/LzmaStream.cs @@ -1,22 +1,25 @@ -#nullable disable - using System; +using System.Buffers; using System.Buffers.Binary; using System.IO; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Compressors.LZMA.LZ; +using SharpCompress.IO; namespace SharpCompress.Compressors.LZMA; -public class LzmaStream : Stream +public partial class LzmaStream : Stream, IStreamStack, IAsyncDisposable { - private readonly Stream _inputStream; + private readonly Stream? _inputStream; private readonly long _inputSize; private readonly long _outputSize; + private readonly bool _leaveOpen; private readonly int _dictionarySize; - private readonly OutWindow _outWindow = new OutWindow(); - private readonly RangeCoder.Decoder _rangeDecoder = new RangeCoder.Decoder(); - private Decoder _decoder; + private readonly OutWindow _outWindow = new(); + private readonly RangeCoder.Decoder _rangeDecoder = new(); + private Decoder? _decoder; private long _position; private bool _endReached; @@ -30,42 +33,28 @@ public class LzmaStream : Stream private bool _needDictReset = true; private bool _needProps = true; - private readonly Encoder _encoder; + private readonly Encoder? _encoder; + private byte[]? _asyncHeaderBuffer; private bool _isDisposed; - public LzmaStream(byte[] properties, Stream inputStream) - : this(properties, inputStream, -1, -1, null, properties.Length < 5) { } - - public LzmaStream(byte[] properties, Stream inputStream, long inputSize) - : this(properties, inputStream, inputSize, -1, null, properties.Length < 5) { } - - public LzmaStream(byte[] properties, Stream inputStream, long inputSize, long outputSize) - : this(properties, inputStream, inputSize, outputSize, null, properties.Length < 5) { } - - public LzmaStream( + private LzmaStream( byte[] properties, Stream inputStream, long inputSize, long outputSize, - Stream presetDictionary, - bool isLzma2 + bool isLzma2, + bool leaveOpen = false ) { _inputStream = inputStream; _inputSize = inputSize; _outputSize = outputSize; _isLzma2 = isLzma2; - + _leaveOpen = leaveOpen; if (!isLzma2) { _dictionarySize = BinaryPrimitives.ReadInt32LittleEndian(properties.AsSpan(1)); _outWindow.Create(_dictionarySize); - if (presetDictionary != null) - { - _outWindow.Train(presetDictionary); - } - - _rangeDecoder.Init(inputStream); _decoder = new Decoder(); _decoder.SetDecoderProperties(properties); @@ -80,26 +69,81 @@ public class LzmaStream : Stream _dictionarySize <<= (properties[0] >> 1) + 11; _outWindow.Create(_dictionarySize); - if (presetDictionary != null) - { - _outWindow.Train(presetDictionary); - _needDictReset = false; - } Properties = new byte[1]; _availableBytes = 0; } } - public LzmaStream(LzmaEncoderProperties properties, bool isLzma2, Stream outputStream) - : this(properties, isLzma2, null, outputStream) { } + public static LzmaStream Create( + byte[] properties, + Stream inputStream, + bool leaveOpen = false + ) => Create(properties, inputStream, -1, -1, null, properties.Length < 5, leaveOpen); - public LzmaStream( - LzmaEncoderProperties properties, + public static LzmaStream Create( + byte[] properties, + Stream inputStream, + long inputSize, + bool leaveOpen = false + ) => Create(properties, inputStream, inputSize, -1, null, properties.Length < 5, leaveOpen); + + public static LzmaStream Create( + byte[] properties, + Stream inputStream, + long inputSize, + long outputSize, + bool leaveOpen = false + ) => + Create( + properties, + inputStream, + inputSize, + outputSize, + null, + properties.Length < 5, + leaveOpen + ); + + public static LzmaStream Create( + byte[] properties, + Stream inputStream, + long inputSize, + long outputSize, + Stream? presetDictionary, bool isLzma2, - Stream presetDictionary, - Stream outputStream + bool leaveOpen = false ) + { + var lzma = new LzmaStream( + properties, + inputStream, + inputSize, + outputSize, + isLzma2, + leaveOpen + ); + if (!isLzma2) + { + if (presetDictionary != null) + { + lzma._outWindow.Train(presetDictionary); + } + + lzma._rangeDecoder.Init(inputStream); + } + else + { + if (presetDictionary != null) + { + lzma._outWindow.Train(presetDictionary); + lzma._needDictReset = false; + } + } + return lzma; + } + + private LzmaStream(LzmaEncoderProperties properties, bool isLzma2) { _isLzma2 = isLzma2; _availableBytes = 0; @@ -115,12 +159,30 @@ public class LzmaStream : Stream var prop = new byte[5]; _encoder.WriteCoderProperties(prop); Properties = prop; + } + + public static LzmaStream Create( + LzmaEncoderProperties properties, + bool isLzma2, + Stream outputStream + ) => Create(properties, isLzma2, null, outputStream); + + public static LzmaStream Create( + LzmaEncoderProperties properties, + bool isLzma2, + Stream? presetDictionary, + Stream outputStream + ) + { + var lzma = new LzmaStream(properties, isLzma2); + + lzma._encoder!.SetStreams(null, outputStream, -1, -1); - _encoder.SetStreams(null, outputStream, -1, -1); if (presetDictionary != null) { - _encoder.Train(presetDictionary); + lzma._encoder.Train(presetDictionary); } + return lzma; } public override bool CanRead => _encoder == null; @@ -131,6 +193,8 @@ public class LzmaStream : Stream public override void Flush() { } + Stream IStreamStack.BaseStream() => _inputStream!; + protected override void Dispose(bool disposing) { if (_isDisposed) @@ -143,12 +207,31 @@ public class LzmaStream : Stream if (_encoder != null) { _position = _encoder.Code(null, true); + _encoder.Dispose(); } - _inputStream?.Dispose(); + if (!_leaveOpen) + { + _inputStream?.Dispose(); + } + _outWindow.Dispose(); + ReturnAsyncHeaderBuffer(); } base.Dispose(disposing); } + private byte[] GetAsyncHeaderBuffer() => _asyncHeaderBuffer ??= ArrayPool.Shared.Rent(6); + + private void ReturnAsyncHeaderBuffer() + { + if (_asyncHeaderBuffer is null) + { + return; + } + + ArrayPool.Shared.Return(_asyncHeaderBuffer); + _asyncHeaderBuffer = null; + } + public override long Length => _position + _availableBytes; public override long Position @@ -194,9 +277,9 @@ public class LzmaStream : Stream { _inputPosition += _outWindow.CopyStream(_inputStream, toProcess); } - else if (_decoder.Code(_dictionarySize, _outWindow, _rangeDecoder) && _outputSize < 0) + else if (_decoder!.Code(_dictionarySize, _outWindow, _rangeDecoder)) { - _availableBytes = _outWindow.AvailableBytes; + HandleEndMarker(); } var read = _outWindow.Read(buffer, offset, toProcess); @@ -207,14 +290,28 @@ public class LzmaStream : Stream if (_availableBytes == 0 && !_uncompressedChunk) { - _rangeDecoder.ReleaseStream(); + if (_isLzma2 && _decoder!.HasEndMarker) + { + throw new DataErrorException(); + } + + // Check range corruption scenario if ( !_rangeDecoder.IsFinished || (_rangeDecoderLimit >= 0 && _rangeDecoder._total != _rangeDecoderLimit) ) { - throw new DataErrorException(); + // Stream might have End Of Stream marker + _outWindow.SetLimit(toProcess + 1); + if (!_decoder!.Code(_dictionarySize, _outWindow, _rangeDecoder)) + { + _rangeDecoder.ReleaseStream(); + throw new DataErrorException(); + } } + + _rangeDecoder.ReleaseStream(); + _inputPosition += _rangeDecoder._total; if (_outWindow.HasPending) { @@ -238,13 +335,99 @@ public class LzmaStream : Stream return total; } + public override int ReadByte() + { + if (_endReached) + { + return -1; + } + + if (_availableBytes == 0) + { + if (_isLzma2) + { + DecodeChunkHeader(); + } + else + { + _endReached = true; + } + } + + if (_endReached) + { + if (_inputSize >= 0 && _inputPosition != _inputSize) + { + throw new DataErrorException(); + } + if (_outputSize >= 0 && _position != _outputSize) + { + throw new DataErrorException(); + } + + return -1; + } + + _outWindow.SetLimit(1); + if (_uncompressedChunk) + { + _inputPosition += _outWindow.CopyStream(_inputStream, 1); + } + else if (_decoder!.Code(_dictionarySize, _outWindow, _rangeDecoder)) + { + HandleEndMarker(); + } + + var value = _outWindow.ReadByte(); + _position++; + _availableBytes--; + + if (_availableBytes == 0 && !_uncompressedChunk) + { + if (_isLzma2 && _decoder!.HasEndMarker) + { + throw new DataErrorException(); + } + + // Check range corruption scenario + if ( + !_rangeDecoder.IsFinished + || (_rangeDecoderLimit >= 0 && _rangeDecoder._total != _rangeDecoderLimit) + ) + { + // Stream might have End Of Stream marker + _outWindow.SetLimit(2); + if (!_decoder!.Code(_dictionarySize, _outWindow, _rangeDecoder)) + { + _rangeDecoder.ReleaseStream(); + throw new DataErrorException(); + } + } + + _rangeDecoder.ReleaseStream(); + + _inputPosition += _rangeDecoder._total; + if (_outWindow.HasPending) + { + throw new DataErrorException(); + } + } + + return value; + } + private void DecodeChunkHeader() { - var control = _inputStream.ReadByte(); + var control = _inputStream!.ReadByte(); _inputPosition++; if (control == 0x00) { + if (_isLzma2 && _decoder is { HasEndMarker: true }) + { + throw new DataErrorException(); + } + _endReached = true; return; } @@ -304,6 +487,19 @@ public class LzmaStream : Stream } } + private void HandleEndMarker() + { + if (_isLzma2) + { + throw new DataErrorException(); + } + + if (_outputSize < 0) + { + _availableBytes = _outWindow.AvailableBytes; + } + } + public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); public override void SetLength(long value) => throw new NotSupportedException(); @@ -317,4 +513,6 @@ public class LzmaStream : Stream } public byte[] Properties { get; } = new byte[5]; + + internal long CompressedBytesRead => _inputPosition; } diff --git a/src/SharpCompress/Compressors/LZMA/RangeCoder/RangeCoder.Async.cs b/src/SharpCompress/Compressors/LZMA/RangeCoder/RangeCoder.Async.cs new file mode 100644 index 00000000..61db6eae --- /dev/null +++ b/src/SharpCompress/Compressors/LZMA/RangeCoder/RangeCoder.Async.cs @@ -0,0 +1,211 @@ +#nullable disable + +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Compressors.LZMA.RangeCoder; + +internal partial class Encoder +{ + private byte[] SingleByteBuffer => _singleByteBuffer ??= new byte[1]; + + public async ValueTask ShiftLowAsync(CancellationToken cancellationToken = default) + { + if ((uint)_low < 0xFF000000 || (uint)(_low >> 32) == 1) + { + var temp = _cache; + do + { + var b = (byte)(temp + (_low >> 32)); + var buffer = SingleByteBuffer; + buffer[0] = b; + await _stream.WriteAsync(buffer, 0, 1, cancellationToken).ConfigureAwait(false); + temp = 0xFF; + } while (--_cacheSize != 0); + _cache = (byte)(((uint)_low) >> 24); + } + _cacheSize++; + _low = ((uint)_low) << 8; + } + + public async ValueTask EncodeBitAsync( + uint size0, + int numTotalBits, + uint symbol, + CancellationToken cancellationToken = default + ) + { + var newBound = (_range >> numTotalBits) * size0; + if (symbol == 0) + { + _range = newBound; + } + else + { + _low += newBound; + _range -= newBound; + } + while (_range < K_TOP_VALUE) + { + _range <<= 8; + await ShiftLowAsync(cancellationToken).ConfigureAwait(false); + } + } + + public async ValueTask EncodeDirectBitsAsync( + uint v, + int numTotalBits, + CancellationToken cancellationToken = default + ) + { + for (var i = numTotalBits - 1; i >= 0; i--) + { + _range >>= 1; + if (((v >> i) & 1) == 1) + { + _low += _range; + } + if (_range < K_TOP_VALUE) + { + _range <<= 8; + await ShiftLowAsync(cancellationToken).ConfigureAwait(false); + } + } + } + + public async ValueTask FlushStreamAsync(CancellationToken cancellationToken = default) => + await _stream.FlushAsync(cancellationToken).ConfigureAwait(false); +} + +internal partial class Decoder +{ + private byte[] SingleByteBuffer => _singleByteBuffer ??= new byte[1]; + + public async ValueTask InitAsync(Stream stream, CancellationToken cancellationToken = default) + { + _stream = stream; + + _code = 0; + _range = 0xFFFFFFFF; + var buffer = SingleByteBuffer; + for (var i = 0; i < 5; i++) + { + var read = await _stream + .ReadAsync(buffer, 0, 1, cancellationToken) + .ConfigureAwait(false); + if (read == 0) + { + throw new IncompleteArchiveException("Unexpected end of stream."); + } + _code = (_code << 8) | buffer[0]; + } + _total = 5; + } + + public async ValueTask NormalizeAsync(CancellationToken cancellationToken = default) + { + while (_range < K_TOP_VALUE) + { + var buffer = SingleByteBuffer; + var read = await _stream + .ReadAsync(buffer, 0, 1, cancellationToken) + .ConfigureAwait(false); + if (read == 0) + { + throw new IncompleteArchiveException("Unexpected end of stream."); + } + _code = (_code << 8) | buffer[0]; + _range <<= 8; + _total++; + } + } + + public async ValueTask Normalize2Async(CancellationToken cancellationToken = default) + { + if (_range < K_TOP_VALUE) + { + var buffer = SingleByteBuffer; + var read = await _stream + .ReadAsync(buffer, 0, 1, cancellationToken) + .ConfigureAwait(false); + if (read == 0) + { + throw new IncompleteArchiveException("Unexpected end of stream."); + } + _code = (_code << 8) | buffer[0]; + _range <<= 8; + _total++; + } + } + + public async ValueTask DecodeDirectBitsAsync( + int numTotalBits, + CancellationToken cancellationToken = default + ) + { + var range = _range; + var code = _code; + uint result = 0; + var buffer = SingleByteBuffer; + for (var i = numTotalBits; i > 0; i--) + { + range >>= 1; + var t = (code - range) >> 31; + code -= range & (t - 1); + result = (result << 1) | (1 - t); + + if (range < K_TOP_VALUE) + { + var read = await _stream + .ReadAsync(buffer, 0, 1, cancellationToken) + .ConfigureAwait(false); + if (read == 0) + { + throw new IncompleteArchiveException("Unexpected end of stream."); + } + code = (code << 8) | buffer[0]; + range <<= 8; + _total++; + } + } + _range = range; + _code = code; + return result; + } + + public async ValueTask DecodeBitAsync( + uint size0, + int numTotalBits, + CancellationToken cancellationToken = default + ) + { + var newBound = (_range >> numTotalBits) * size0; + uint symbol; + if (_code < newBound) + { + symbol = 0; + _range = newBound; + } + else + { + symbol = 1; + _code -= newBound; + _range -= newBound; + } + await NormalizeAsync(cancellationToken).ConfigureAwait(false); + return symbol; + } + + public async ValueTask DecodeAsync( + uint start, + uint size, + CancellationToken cancellationToken = default + ) + { + _code -= start * _range; + _range *= size; + await NormalizeAsync(cancellationToken).ConfigureAwait(false); + } +} diff --git a/src/SharpCompress/Compressors/LZMA/RangeCoder/RangeCoder.cs b/src/SharpCompress/Compressors/LZMA/RangeCoder/RangeCoder.cs index 87906809..147146c4 100644 --- a/src/SharpCompress/Compressors/LZMA/RangeCoder/RangeCoder.cs +++ b/src/SharpCompress/Compressors/LZMA/RangeCoder/RangeCoder.cs @@ -1,10 +1,11 @@ #nullable disable using System.IO; +using System.Runtime.CompilerServices; namespace SharpCompress.Compressors.LZMA.RangeCoder; -internal class Encoder +internal partial class Encoder { public const uint K_TOP_VALUE = (1 << 24); @@ -14,8 +15,7 @@ internal class Encoder public uint _range; private uint _cacheSize; private byte _cache; - - //long StartPosition; + private byte[] _singleByteBuffer = new byte[1]; public void SetStream(Stream stream) => _stream = stream; @@ -43,17 +43,6 @@ internal class Encoder public void CloseStream() => _stream.Dispose(); - public void Encode(uint start, uint size, uint total) - { - _low += start * (_range /= total); - _range *= size; - while (_range < K_TOP_VALUE) - { - _range <<= 8; - ShiftLow(); - } - } - public void ShiftLow() { if ((uint)_low < 0xFF000000 || (uint)(_low >> 32) == 1) @@ -87,44 +76,21 @@ internal class Encoder } } - public void EncodeBit(uint size0, int numTotalBits, uint symbol) - { - var newBound = (_range >> numTotalBits) * size0; - if (symbol == 0) - { - _range = newBound; - } - else - { - _low += newBound; - _range -= newBound; - } - while (_range < K_TOP_VALUE) - { - _range <<= 8; - ShiftLow(); - } - } - public long GetProcessedSizeAdd() => -1; - - //return _cacheSize + Stream.Position - StartPosition + 4; - // (long)Stream.GetProcessedSize(); } -internal class Decoder +internal partial class Decoder { public const uint K_TOP_VALUE = (1 << 24); public uint _range; public uint _code; - // public Buffer.InBuffer Stream = new Buffer.InBuffer(1 << 16); public Stream _stream; public long _total; + private byte[] _singleByteBuffer; public void Init(Stream stream) { - // Stream.Init(stream); _stream = stream; _code = 0; @@ -140,8 +106,6 @@ internal class Decoder // Stream.ReleaseStream(); _stream = null; - public void CloseStream() => _stream.Dispose(); - public void Normalize() { while (_range < K_TOP_VALUE) @@ -152,6 +116,7 @@ internal class Decoder } } + [MethodImpl(MethodImplOptions.AggressiveInlining)] public void Normalize2() { if (_range < K_TOP_VALUE) @@ -179,14 +144,6 @@ internal class Decoder for (var i = numTotalBits; i > 0; i--) { range >>= 1; - /* - result <<= 1; - if (code >= range) - { - code -= range; - result |= 1; - } - */ var t = (code - range) >> 31; code -= range & (t - 1); result = (result << 1) | (1 - t); @@ -223,6 +180,4 @@ internal class Decoder } public bool IsFinished => _code == 0; - - // ulong GetProcessedSize() {return Stream.GetProcessedSize(); } } diff --git a/src/SharpCompress/Compressors/LZMA/RangeCoder/RangeCoderBit.Async.cs b/src/SharpCompress/Compressors/LZMA/RangeCoder/RangeCoderBit.Async.cs new file mode 100644 index 00000000..8de17dd9 --- /dev/null +++ b/src/SharpCompress/Compressors/LZMA/RangeCoder/RangeCoderBit.Async.cs @@ -0,0 +1,62 @@ +#nullable disable + +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Compressors.LZMA.RangeCoder; + +internal partial struct BitEncoder +{ + public ValueTask EncodeAsync( + Encoder encoder, + uint symbol, + CancellationToken cancellationToken = default + ) + { + var newBound = (encoder._range >> K_NUM_BIT_MODEL_TOTAL_BITS) * _prob; + if (symbol == 0) + { + encoder._range = newBound; + _prob += (K_BIT_MODEL_TOTAL - _prob) >> K_NUM_MOVE_BITS; + } + else + { + encoder._low += newBound; + encoder._range -= newBound; + _prob -= (_prob) >> K_NUM_MOVE_BITS; + } + if (encoder._range < Encoder.K_TOP_VALUE) + { + encoder._range <<= 8; + return encoder.ShiftLowAsync(cancellationToken); + } + return default; + } +} + +internal partial struct BitDecoder +{ + public ValueTask DecodeAsync( + Decoder decoder, + CancellationToken cancellationToken = default + ) + { + var newBound = (decoder._range >> K_NUM_BIT_MODEL_TOTAL_BITS) * _prob; + if (decoder._code < newBound) + { + decoder._range = newBound; + _prob += (K_BIT_MODEL_TOTAL - _prob) >> K_NUM_MOVE_BITS; + return DecodeAsyncHelper(decoder.Normalize2Async(cancellationToken), 0); + } + decoder._range -= newBound; + decoder._code -= newBound; + _prob -= (_prob) >> K_NUM_MOVE_BITS; + return DecodeAsyncHelper(decoder.Normalize2Async(cancellationToken), 1); + } + + private static async ValueTask DecodeAsyncHelper(ValueTask normalizeTask, uint result) + { + await normalizeTask.ConfigureAwait(false); + return result; + } +} diff --git a/src/SharpCompress/Compressors/LZMA/RangeCoder/RangeCoderBit.cs b/src/SharpCompress/Compressors/LZMA/RangeCoder/RangeCoderBit.cs index dd93b526..cbdc0943 100644 --- a/src/SharpCompress/Compressors/LZMA/RangeCoder/RangeCoderBit.cs +++ b/src/SharpCompress/Compressors/LZMA/RangeCoder/RangeCoderBit.cs @@ -1,6 +1,6 @@ namespace SharpCompress.Compressors.LZMA.RangeCoder; -internal struct BitEncoder +internal partial struct BitEncoder { public const int K_NUM_BIT_MODEL_TOTAL_BITS = 11; public const uint K_BIT_MODEL_TOTAL = (1 << K_NUM_BIT_MODEL_TOTAL_BITS); @@ -26,8 +26,6 @@ internal struct BitEncoder public void Encode(Encoder encoder, uint symbol) { - // encoder.EncodeBit(Prob, kNumBitModelTotalBits, symbol); - // UpdateModel(symbol); var newBound = (encoder._range >> K_NUM_BIT_MODEL_TOTAL_BITS) * _prob; if (symbol == 0) { @@ -78,7 +76,7 @@ internal struct BitEncoder public uint GetPrice1() => PROB_PRICES[(K_BIT_MODEL_TOTAL - _prob) >> K_NUM_MOVE_REDUCING_BITS]; } -internal struct BitDecoder +internal partial struct BitDecoder { public const int K_NUM_BIT_MODEL_TOTAL_BITS = 11; public const uint K_BIT_MODEL_TOTAL = (1 << K_NUM_BIT_MODEL_TOTAL_BITS); @@ -86,18 +84,6 @@ internal struct BitDecoder private uint _prob; - public void UpdateModel(int numMoveBits, uint symbol) - { - if (symbol == 0) - { - _prob += (K_BIT_MODEL_TOTAL - _prob) >> numMoveBits; - } - else - { - _prob -= (_prob) >> numMoveBits; - } - } - public void Init() => _prob = K_BIT_MODEL_TOTAL >> 1; public uint Decode(Decoder rangeDecoder) @@ -107,24 +93,13 @@ internal struct BitDecoder { rangeDecoder._range = newBound; _prob += (K_BIT_MODEL_TOTAL - _prob) >> K_NUM_MOVE_BITS; - if (rangeDecoder._range < Decoder.K_TOP_VALUE) - { - rangeDecoder._code = - (rangeDecoder._code << 8) | (byte)rangeDecoder._stream.ReadByte(); - rangeDecoder._range <<= 8; - rangeDecoder._total++; - } + rangeDecoder.Normalize2(); return 0; } rangeDecoder._range -= newBound; rangeDecoder._code -= newBound; _prob -= (_prob) >> K_NUM_MOVE_BITS; - if (rangeDecoder._range < Decoder.K_TOP_VALUE) - { - rangeDecoder._code = (rangeDecoder._code << 8) | (byte)rangeDecoder._stream.ReadByte(); - rangeDecoder._range <<= 8; - rangeDecoder._total++; - } + rangeDecoder.Normalize2(); return 1; } } diff --git a/src/SharpCompress/Compressors/LZMA/RangeCoder/RangeCoderBitTree.Async.cs b/src/SharpCompress/Compressors/LZMA/RangeCoder/RangeCoderBitTree.Async.cs new file mode 100644 index 00000000..8766156a --- /dev/null +++ b/src/SharpCompress/Compressors/LZMA/RangeCoder/RangeCoderBitTree.Async.cs @@ -0,0 +1,127 @@ +#nullable disable + +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Compressors.LZMA.RangeCoder; + +internal readonly partial struct BitTreeEncoder +{ + public async ValueTask EncodeAsync( + Encoder rangeEncoder, + uint symbol, + CancellationToken cancellationToken = default + ) + { + uint m = 1; + for (var bitIndex = _numBitLevels; bitIndex > 0; ) + { + bitIndex--; + var bit = (symbol >> bitIndex) & 1; + await _models[m] + .EncodeAsync(rangeEncoder, bit, cancellationToken) + .ConfigureAwait(false); + m = (m << 1) | bit; + } + } + + public async ValueTask ReverseEncodeAsync( + Encoder rangeEncoder, + uint symbol, + CancellationToken cancellationToken = default + ) + { + uint m = 1; + for (uint i = 0; i < _numBitLevels; i++) + { + var bit = symbol & 1; + await _models[m] + .EncodeAsync(rangeEncoder, bit, cancellationToken) + .ConfigureAwait(false); + m = (m << 1) | bit; + symbol >>= 1; + } + } + + public static async ValueTask ReverseEncodeAsync( + BitEncoder[] models, + uint startIndex, + Encoder rangeEncoder, + int numBitLevels, + uint symbol, + CancellationToken cancellationToken = default + ) + { + uint m = 1; + for (var i = 0; i < numBitLevels; i++) + { + var bit = symbol & 1; + await models[startIndex + m] + .EncodeAsync(rangeEncoder, bit, cancellationToken) + .ConfigureAwait(false); + m = (m << 1) | bit; + symbol >>= 1; + } + } +} + +internal readonly partial struct BitTreeDecoder +{ + public async ValueTask DecodeAsync( + Decoder rangeDecoder, + CancellationToken cancellationToken = default + ) + { + uint m = 1; + for (var bitIndex = _numBitLevels; bitIndex > 0; bitIndex--) + { + m = + (m << 1) + + await _models[m] + .DecodeAsync(rangeDecoder, cancellationToken) + .ConfigureAwait(false); + } + return m - ((uint)1 << _numBitLevels); + } + + public async ValueTask ReverseDecodeAsync( + Decoder rangeDecoder, + CancellationToken cancellationToken = default + ) + { + uint m = 1; + uint symbol = 0; + for (var bitIndex = 0; bitIndex < _numBitLevels; bitIndex++) + { + var bit = await _models[m] + .DecodeAsync(rangeDecoder, cancellationToken) + .ConfigureAwait(false); + m <<= 1; + m += bit; + symbol |= (bit << bitIndex); + } + return symbol; + } + + public static async ValueTask ReverseDecodeAsync( + BitDecoder[] models, + uint startIndex, + Decoder rangeDecoder, + int numBitLevels, + CancellationToken cancellationToken = default + ) + { + uint m = 1; + uint symbol = 0; + for (var bitIndex = 0; bitIndex < numBitLevels; bitIndex++) + { + var bit = await models[startIndex + m] + .DecodeAsync(rangeDecoder, cancellationToken) + .ConfigureAwait(false); + m <<= 1; + m += bit; + symbol |= (bit << bitIndex); + } + return symbol; + } +} diff --git a/src/SharpCompress/Compressors/LZMA/RangeCoder/RangeCoderBitTree.cs b/src/SharpCompress/Compressors/LZMA/RangeCoder/RangeCoderBitTree.cs index b6a66e82..61ccafcd 100644 --- a/src/SharpCompress/Compressors/LZMA/RangeCoder/RangeCoderBitTree.cs +++ b/src/SharpCompress/Compressors/LZMA/RangeCoder/RangeCoderBitTree.cs @@ -1,6 +1,6 @@ namespace SharpCompress.Compressors.LZMA.RangeCoder; -internal readonly struct BitTreeEncoder +internal readonly partial struct BitTreeEncoder { private readonly BitEncoder[] _models; private readonly int _numBitLevels; @@ -109,7 +109,7 @@ internal readonly struct BitTreeEncoder } } -internal readonly struct BitTreeDecoder +internal readonly partial struct BitTreeDecoder { private readonly BitDecoder[] _models; private readonly int _numBitLevels; diff --git a/src/SharpCompress/Compressors/LZMA/Registry.cs b/src/SharpCompress/Compressors/LZMA/Registry.cs index f4cfde4b..ff734ef7 100644 --- a/src/SharpCompress/Compressors/LZMA/Registry.cs +++ b/src/SharpCompress/Compressors/LZMA/Registry.cs @@ -5,13 +5,13 @@ using SharpCompress.Common.SevenZip; using SharpCompress.Compressors.BZip2; using SharpCompress.Compressors.Deflate; using SharpCompress.Compressors.Filters; -using SharpCompress.Compressors.LZMA.Utilites; +using SharpCompress.Compressors.LZMA.Utilities; using SharpCompress.Compressors.PPMd; -using ZstdSharp; +using SharpCompress.Compressors.ZStandard; namespace SharpCompress.Compressors.LZMA; -internal static class DecoderRegistry +internal static partial class DecoderRegistry { private const uint K_COPY = 0x0; private const uint K_DELTA = 0x3; @@ -25,6 +25,8 @@ internal static class DecoderRegistry private const uint K_ARM = 0x03030501; private const uint K_ARMT = 0x03030701; private const uint K_SPARC = 0x03030805; + private const uint K_ARM64 = 0x0A; + private const uint K_RISCV = 0x0B; private const uint K_DEFLATE = 0x040108; private const uint K_B_ZIP2 = 0x040202; private const uint K_ZSTD = 0x4F71101; @@ -32,7 +34,7 @@ internal static class DecoderRegistry internal static Stream CreateDecoderStream( CMethodId id, Stream[] inStreams, - byte[] info, + byte[]? info, IPasswordProvider pass, long limit ) @@ -46,16 +48,16 @@ internal static class DecoderRegistry } return inStreams.Single(); case K_DELTA: - return new DeltaFilter(false, inStreams.Single(), info); + return new DeltaFilter(false, inStreams.Single(), info.NotNull()); case K_LZMA: case K_LZMA2: - return new LzmaStream(info, inStreams.Single(), -1, limit); + return LzmaStream.Create(info.NotNull(), inStreams.Single(), -1, limit); case CMethodId.K_AES_ID: - return new AesDecoderStream(inStreams.Single(), info, pass, limit); + return new AesDecoderStream(inStreams.Single(), info.NotNull(), pass, limit); case K_BCJ: return new BCJFilter(false, inStreams.Single()); case K_BCJ2: - return new Bcj2DecoderStream(inStreams, info, limit); + return new Bcj2DecoderStream(inStreams); case K_PPC: return new BCJFilterPPC(false, inStreams.Single()); case K_IA64: @@ -66,10 +68,18 @@ internal static class DecoderRegistry return new BCJFilterARMT(false, inStreams.Single()); case K_SPARC: return new BCJFilterSPARC(false, inStreams.Single()); + case K_ARM64: + return new BCJFilterARM64(false, inStreams.Single()); + case K_RISCV: + return new BCJFilterRISCV(false, inStreams.Single()); case K_B_ZIP2: - return new BZip2Stream(inStreams.Single(), CompressionMode.Decompress, true); + return BZip2Stream.Create(inStreams.Single(), CompressionMode.Decompress, true); case K_PPMD: - return new PpmdStream(new PpmdProperties(info), inStreams.Single(), false); + return PpmdStream.Create( + new PpmdProperties(info.NotNull()), + inStreams.Single(), + false + ); case K_DEFLATE: return new DeflateStream(inStreams.Single(), CompressionMode.Decompress); case K_ZSTD: diff --git a/src/SharpCompress/Compressors/LZMA/Utilites/CrcCheckStream.cs b/src/SharpCompress/Compressors/LZMA/Utilites/CrcCheckStream.cs deleted file mode 100644 index c10efda1..00000000 --- a/src/SharpCompress/Compressors/LZMA/Utilites/CrcCheckStream.cs +++ /dev/null @@ -1,104 +0,0 @@ -using System; -using System.Diagnostics; -using System.IO; - -namespace SharpCompress.Compressors.LZMA.Utilites; - -[CLSCompliant(false)] -public class CrcCheckStream : Stream -{ - private readonly uint _mExpectedCrc; - private uint _mCurrentCrc; - private bool _mClosed; - - private readonly long[] _mBytes = new long[256]; - private long _mLength; - - public CrcCheckStream(uint crc) - { - _mExpectedCrc = crc; - _mCurrentCrc = Crc.INIT_CRC; - } - - protected override void Dispose(bool disposing) - { - //Nanook - is not equal here - _mCurrentCrc is yet to be negated - //if (_mCurrentCrc != _mExpectedCrc) - //{ - // throw new InvalidOperationException(); - //} - try - { - if (disposing && !_mClosed) - { - _mClosed = true; - _mCurrentCrc = Crc.Finish(_mCurrentCrc); //now becomes equal -#if DEBUG - if (_mCurrentCrc == _mExpectedCrc) - { - Debug.WriteLine("CRC ok: " + _mExpectedCrc.ToString("x8")); - } - else - { - Debugger.Break(); - Debug.WriteLine("bad CRC"); - } - - var lengthInv = 1.0 / _mLength; - double entropy = 0; - for (var i = 0; i < 256; i++) - { - if (_mBytes[i] != 0) - { - var p = lengthInv * _mBytes[i]; - entropy -= p * Math.Log(p, 256); - } - } - Debug.WriteLine("entropy: " + (int)(entropy * 100) + "%"); -#endif - if (_mCurrentCrc != _mExpectedCrc) //moved test to here - { - throw new InvalidOperationException(); - } - } - } - finally - { - base.Dispose(disposing); - } - } - - public override bool CanRead => false; - - public override bool CanSeek => false; - - public override bool CanWrite => true; - - public override void Flush() { } - - public override long Length => throw new NotSupportedException(); - - public override long Position - { - get => throw new NotSupportedException(); - set => throw new NotSupportedException(); - } - - public override int Read(byte[] buffer, int offset, int count) => - throw new InvalidOperationException(); - - public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); - - public override void SetLength(long value) => throw new NotSupportedException(); - - public override void Write(byte[] buffer, int offset, int count) - { - _mLength += count; - for (var i = 0; i < count; i++) - { - _mBytes[buffer[offset + i]]++; - } - - _mCurrentCrc = Crc.Update(_mCurrentCrc, buffer, offset, count); - } -} diff --git a/src/SharpCompress/Compressors/LZMA/Utilites/IPasswordProvider.cs b/src/SharpCompress/Compressors/LZMA/Utilites/IPasswordProvider.cs deleted file mode 100644 index 2f0eb904..00000000 --- a/src/SharpCompress/Compressors/LZMA/Utilites/IPasswordProvider.cs +++ /dev/null @@ -1,6 +0,0 @@ -namespace SharpCompress.Compressors.LZMA.Utilites; - -internal interface IPasswordProvider -{ - string CryptoGetTextPassword(); -} diff --git a/src/SharpCompress/Compressors/LZMA/Utilites/Utils.cs b/src/SharpCompress/Compressors/LZMA/Utilites/Utils.cs deleted file mode 100644 index 19b0f374..00000000 --- a/src/SharpCompress/Compressors/LZMA/Utilites/Utils.cs +++ /dev/null @@ -1,91 +0,0 @@ -using System; -using System.Diagnostics; -using System.IO; - -namespace SharpCompress.Compressors.LZMA.Utilites; - -internal enum BlockType : byte -{ - #region Constants - - End = 0, - Header = 1, - ArchiveProperties = 2, - AdditionalStreamsInfo = 3, - MainStreamsInfo = 4, - FilesInfo = 5, - PackInfo = 6, - UnpackInfo = 7, - SubStreamsInfo = 8, - Size = 9, - Crc = 10, - Folder = 11, - CodersUnpackSize = 12, - NumUnpackStream = 13, - EmptyStream = 14, - EmptyFile = 15, - Anti = 16, - Name = 17, - CTime = 18, - ATime = 19, - MTime = 20, - WinAttributes = 21, - Comment = 22, - EncodedHeader = 23, - StartPos = 24, - Dummy = 25 - - #endregion -} - -internal static class Utils -{ - [Conditional("DEBUG")] - public static void Assert(bool expression) - { - if (!expression) - { - if (Debugger.IsAttached) - { - Debugger.Break(); - } - - throw new InvalidOperationException("Assertion failed."); - } - } - - public static void ReadExact(this Stream stream, byte[] buffer, int offset, int length) - { - if (stream is null) - { - throw new ArgumentNullException(nameof(stream)); - } - - if (buffer is null) - { - throw new ArgumentNullException(nameof(buffer)); - } - - if (offset < 0 || offset > buffer.Length) - { - throw new ArgumentOutOfRangeException(nameof(offset)); - } - - if (length < 0 || length > buffer.Length - offset) - { - throw new ArgumentOutOfRangeException(nameof(length)); - } - - while (length > 0) - { - var fetched = stream.Read(buffer, offset, length); - if (fetched <= 0) - { - throw new EndOfStreamException(); - } - - offset += fetched; - length -= fetched; - } - } -} diff --git a/src/SharpCompress/Compressors/LZMA/Utilities/BlockType.cs b/src/SharpCompress/Compressors/LZMA/Utilities/BlockType.cs new file mode 100644 index 00000000..8a4d7ddf --- /dev/null +++ b/src/SharpCompress/Compressors/LZMA/Utilities/BlockType.cs @@ -0,0 +1,35 @@ +namespace SharpCompress.Compressors.LZMA.Utilities; + +internal enum BlockType : byte +{ + #region Constants + + End = 0, + Header = 1, + ArchiveProperties = 2, + AdditionalStreamsInfo = 3, + MainStreamsInfo = 4, + FilesInfo = 5, + PackInfo = 6, + UnpackInfo = 7, + SubStreamsInfo = 8, + Size = 9, + Crc = 10, + Folder = 11, + CodersUnpackSize = 12, + NumUnpackStream = 13, + EmptyStream = 14, + EmptyFile = 15, + Anti = 16, + Name = 17, + CTime = 18, + ATime = 19, + MTime = 20, + WinAttributes = 21, + Comment = 22, + EncodedHeader = 23, + StartPos = 24, + Dummy = 25 + + #endregion +} diff --git a/src/SharpCompress/Compressors/LZMA/Utilities/CrcBuilderStream.Async.cs b/src/SharpCompress/Compressors/LZMA/Utilities/CrcBuilderStream.Async.cs new file mode 100644 index 00000000..e3fa5332 --- /dev/null +++ b/src/SharpCompress/Compressors/LZMA/Utilities/CrcBuilderStream.Async.cs @@ -0,0 +1,28 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Compressors.LZMA.Utilities; + +internal partial class CrcBuilderStream : Stream +{ + public override async Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + if (_mFinished) + { + throw new ArchiveOperationException("CRC calculation has been finished."); + } + + Processed += count; + _mCrc = Crc.Update(_mCrc, buffer, offset, count); + await _mTarget.WriteAsync(buffer, offset, count, cancellationToken).ConfigureAwait(false); + } +} diff --git a/src/SharpCompress/Compressors/LZMA/Utilites/CrcBuilderStream.cs b/src/SharpCompress/Compressors/LZMA/Utilities/CrcBuilderStream.cs similarity index 86% rename from src/SharpCompress/Compressors/LZMA/Utilites/CrcBuilderStream.cs rename to src/SharpCompress/Compressors/LZMA/Utilities/CrcBuilderStream.cs index 344924be..f694d749 100644 --- a/src/SharpCompress/Compressors/LZMA/Utilites/CrcBuilderStream.cs +++ b/src/SharpCompress/Compressors/LZMA/Utilities/CrcBuilderStream.cs @@ -1,9 +1,10 @@ -using System; +using System; using System.IO; +using SharpCompress.Common; -namespace SharpCompress.Compressors.LZMA.Utilites; +namespace SharpCompress.Compressors.LZMA.Utilities; -internal class CrcBuilderStream : Stream +internal partial class CrcBuilderStream : Stream { private readonly Stream _mTarget; private uint _mCrc; @@ -57,7 +58,7 @@ internal class CrcBuilderStream : Stream } public override int Read(byte[] buffer, int offset, int count) => - throw new InvalidOperationException(); + throw new ArchiveOperationException(); public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); @@ -67,7 +68,7 @@ internal class CrcBuilderStream : Stream { if (_mFinished) { - throw new InvalidOperationException("CRC calculation has been finished."); + throw new ArchiveOperationException("CRC calculation has been finished."); } Processed += count; diff --git a/src/SharpCompress/Compressors/LZMA/Utilities/CrcCheckStream.cs b/src/SharpCompress/Compressors/LZMA/Utilities/CrcCheckStream.cs new file mode 100644 index 00000000..47ef7342 --- /dev/null +++ b/src/SharpCompress/Compressors/LZMA/Utilities/CrcCheckStream.cs @@ -0,0 +1,84 @@ +using System; +using System.Buffers; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Compressors.LZMA.Utilities; + +[CLSCompliant(false)] +public class CrcCheckStream(uint crc) : Stream +{ + private uint _mCurrentCrc = Crc.INIT_CRC; + private bool _mClosed; + + private readonly long[] _mBytes = ArrayPool.Shared.Rent(256); + + protected override void Dispose(bool disposing) + { + try + { + if (disposing && !_mClosed) + { + _mClosed = true; + _mCurrentCrc = Crc.Finish(_mCurrentCrc); //now becomes equal + + if (_mCurrentCrc != crc) //moved test to here + { + throw new ArchiveOperationException(); + } + } + } + finally + { + base.Dispose(disposing); + ArrayPool.Shared.Return(_mBytes); + } + } + + public override bool CanRead => false; + + public override bool CanSeek => false; + + public override bool CanWrite => true; + + public override void Flush() { } + + public override long Length => throw new NotSupportedException(); + + public override long Position + { + get => throw new NotSupportedException(); + set => throw new NotSupportedException(); + } + + public override int Read(byte[] buffer, int offset, int count) => + throw new ArchiveOperationException(); + + public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); + + public override void SetLength(long value) => throw new NotSupportedException(); + + public override void Write(byte[] buffer, int offset, int count) + { + for (var i = 0; i < count; i++) + { + _mBytes[buffer[offset + i]]++; + } + + _mCurrentCrc = Crc.Update(_mCurrentCrc, buffer, offset, count); + } + + public override Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + cancellationToken.ThrowIfCancellationRequested(); + Write(buffer, offset, count); + return Task.CompletedTask; + } +} diff --git a/src/SharpCompress/Compressors/LZMA/Utilities/IPasswordProvider.cs b/src/SharpCompress/Compressors/LZMA/Utilities/IPasswordProvider.cs new file mode 100644 index 00000000..868b4a52 --- /dev/null +++ b/src/SharpCompress/Compressors/LZMA/Utilities/IPasswordProvider.cs @@ -0,0 +1,6 @@ +namespace SharpCompress.Compressors.LZMA.Utilities; + +internal interface IPasswordProvider +{ + string? CryptoGetTextPassword(); +} diff --git a/src/SharpCompress/Compressors/Lzw/LzwConstants.cs b/src/SharpCompress/Compressors/Lzw/LzwConstants.cs new file mode 100644 index 00000000..d36210e8 --- /dev/null +++ b/src/SharpCompress/Compressors/Lzw/LzwConstants.cs @@ -0,0 +1,64 @@ +namespace SharpCompress.Compressors.Lzw; + +/// +/// This class contains constants used for LZW +/// +[System.Diagnostics.CodeAnalysis.SuppressMessage( + "Naming", + "CA1707:Identifiers should not contain underscores", + Justification = "kept for backwards compatibility" +)] +public sealed class LzwConstants +{ + /// + /// Magic number found at start of LZW header: 0x1f 0x9d + /// + public const int MAGIC = 0x1f9d; + + /// + /// Maximum number of bits per code + /// + public const int MAX_BITS = 16; + + /* 3rd header byte: + * bit 0..4 Number of compression bits + * bit 5 Extended header + * bit 6 Free + * bit 7 Block mode + */ + + /// + /// Mask for 'number of compression bits' + /// + public const int BIT_MASK = 0x1f; + + /// + /// Indicates the presence of a fourth header byte + /// + public const int EXTENDED_MASK = 0x20; + + //public const int FREE_MASK = 0x40; + + /// + /// Reserved bits + /// + public const int RESERVED_MASK = 0x60; + + /// + /// Block compression: if table is full and compression rate is dropping, + /// clear the dictionary. + /// + public const int BLOCK_MODE_MASK = 0x80; + + /// + /// LZW file header size (in bytes) + /// + public const int HDR_SIZE = 3; + + /// + /// Initial number of bits per code + /// + public const int INIT_BITS = 9; + + private LzwConstants() { } +} diff --git a/src/SharpCompress/Compressors/Lzw/LzwStream.Async.cs b/src/SharpCompress/Compressors/Lzw/LzwStream.Async.cs new file mode 100644 index 00000000..4f6bbf40 --- /dev/null +++ b/src/SharpCompress/Compressors/Lzw/LzwStream.Async.cs @@ -0,0 +1,393 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.IO; + +namespace SharpCompress.Compressors.Lzw; + +public partial class LzwStream +{ + /// + /// Asynchronously checks if the stream is an LZW stream + /// + /// The stream to read from + /// Cancellation token + /// True if the stream is an LZW stream, false otherwise + public static async ValueTask IsLzwStreamAsync( + Stream stream, + CancellationToken cancellationToken = default + ) + { + try + { + byte[] hdr = new byte[LzwConstants.HDR_SIZE]; + + int result = await stream + .ReadAsync(hdr, 0, hdr.Length, cancellationToken) + .ConfigureAwait(false); + + // Check the magic marker + if (result < 0) + { + throw new IncompleteArchiveException("Failed to read LZW header"); + } + + if (hdr[0] != (LzwConstants.MAGIC >> 8) || hdr[1] != (LzwConstants.MAGIC & 0xff)) + { + throw new IncompleteArchiveException( + String.Format( + Constants.DefaultCultureInfo, + "Wrong LZW header. Magic bytes don't match. 0x{0:x2} 0x{1:x2}", + hdr[0], + hdr[1] + ) + ); + } + } + catch (Exception) + { + return false; + } + return true; + } + + /// + /// Reads decompressed data asynchronously into the provided buffer byte array + /// + /// The array to read and decompress data into + /// The offset indicating where the data should be placed + /// The number of bytes to decompress + /// Cancellation token + /// The number of bytes read. Zero signals the end of stream + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + if (!headerParsed) + { + try + { + await ParseHeaderAsync(cancellationToken).ConfigureAwait(false); + } + catch + { + eof = true; + throw; + } + } + + if (eof) + { + return 0; + } + + int start = offset; + + int[] lTabPrefix = tabPrefix; + byte[] lTabSuffix = tabSuffix; + byte[] lStack = stack; + int lNBits = nBits; + int lMaxCode = maxCode; + int lMaxMaxCode = maxMaxCode; + int lBitMask = bitMask; + int lOldCode = oldCode; + byte lFinChar = finChar; + int lStackP = stackP; + int lFreeEnt = freeEnt; + byte[] lData = data; + int lBitPos = bitPos; + + int sSize = lStack.Length - lStackP; + if (sSize > 0) + { + int num = (sSize >= count) ? count : sSize; + Array.Copy(lStack, lStackP, buffer, offset, num); + offset += num; + count -= num; + lStackP += num; + } + + if (count == 0) + { + stackP = lStackP; + return offset - start; + } + + MainLoop: + do + { + if (end < EXTRA) + { + await FillAsync(cancellationToken).ConfigureAwait(false); + } + + int bitIn = (got > 0) ? (end - end % lNBits) << 3 : (end << 3) - (lNBits - 1); + + while (lBitPos < bitIn) + { + if (count == 0) + { + nBits = lNBits; + maxCode = lMaxCode; + maxMaxCode = lMaxMaxCode; + bitMask = lBitMask; + oldCode = lOldCode; + finChar = lFinChar; + stackP = lStackP; + freeEnt = lFreeEnt; + bitPos = lBitPos; + + return offset - start; + } + + if (lFreeEnt > lMaxCode) + { + int nBytes = lNBits << 3; + lBitPos = (lBitPos - 1) + nBytes - (lBitPos - 1 + nBytes) % nBytes; + + lNBits++; + lMaxCode = (lNBits == maxBits) ? lMaxMaxCode : (1 << lNBits) - 1; + + lBitMask = (1 << lNBits) - 1; + lBitPos = ResetBuf(lBitPos); + goto MainLoop; + } + + int pos = lBitPos >> 3; + int code = + ( + ( + (lData[pos] & 0xFF) + | ((lData[pos + 1] & 0xFF) << 8) + | ((lData[pos + 2] & 0xFF) << 16) + ) >> (lBitPos & 0x7) + ) & lBitMask; + + lBitPos += lNBits; + + if (lOldCode == -1) + { + if (code >= 256) + { + throw new IncompleteArchiveException("corrupt input: " + code + " > 255"); + } + + lFinChar = (byte)(lOldCode = code); + buffer[offset++] = lFinChar; + count--; + continue; + } + + if (code == TBL_CLEAR && blockMode) + { + Array.Copy(zeros, 0, lTabPrefix, 0, zeros.Length); + lFreeEnt = TBL_FIRST - 1; + + int nBytes = lNBits << 3; + lBitPos = (lBitPos - 1) + nBytes - (lBitPos - 1 + nBytes) % nBytes; + lNBits = LzwConstants.INIT_BITS; + lMaxCode = (1 << lNBits) - 1; + lBitMask = lMaxCode; + + lBitPos = ResetBuf(lBitPos); + goto MainLoop; + } + + int inCode = code; + lStackP = lStack.Length; + + if (code >= lFreeEnt) + { + if (code > lFreeEnt) + { + throw new IncompleteArchiveException( + "corrupt input: code=" + code + ", freeEnt=" + lFreeEnt + ); + } + + lStack[--lStackP] = lFinChar; + code = lOldCode; + } + + while (code >= 256) + { + lStack[--lStackP] = lTabSuffix[code]; + code = lTabPrefix[code]; + } + + lFinChar = lTabSuffix[code]; + buffer[offset++] = lFinChar; + count--; + + sSize = lStack.Length - lStackP; + int num = (sSize >= count) ? count : sSize; + Array.Copy(lStack, lStackP, buffer, offset, num); + offset += num; + count -= num; + lStackP += num; + + if (lFreeEnt < lMaxMaxCode) + { + lTabPrefix[lFreeEnt] = lOldCode; + lTabSuffix[lFreeEnt] = lFinChar; + lFreeEnt++; + } + + lOldCode = inCode; + + if (count == 0) + { + nBits = lNBits; + maxCode = lMaxCode; + bitMask = lBitMask; + oldCode = lOldCode; + finChar = lFinChar; + stackP = lStackP; + freeEnt = lFreeEnt; + bitPos = lBitPos; + + return offset - start; + } + } + + lBitPos = ResetBuf(lBitPos); + } while (got > 0); + + nBits = lNBits; + maxCode = lMaxCode; + bitMask = lBitMask; + oldCode = lOldCode; + finChar = lFinChar; + stackP = lStackP; + freeEnt = lFreeEnt; + bitPos = lBitPos; + + eof = true; + return offset - start; + } + +#if !LEGACY_DOTNET + /// + /// Reads decompressed data asynchronously into the provided buffer + /// + /// The memory to read and decompress data into + /// Cancellation token + /// The number of bytes read. Zero signals the end of stream + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + if (buffer.IsEmpty) + { + return 0; + } + + byte[] array = System.Buffers.ArrayPool.Shared.Rent(buffer.Length); + try + { + int read = await ReadAsync(array, 0, buffer.Length, cancellationToken) + .ConfigureAwait(false); + array.AsSpan(0, read).CopyTo(buffer.Span); + return read; + } + finally + { + System.Buffers.ArrayPool.Shared.Return(array); + } + } +#endif + + private async ValueTask FillAsync(CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + got = await baseInputStream + .ReadAsync(data, end, data.Length - 1 - end, cancellationToken) + .ConfigureAwait(false); + if (got > 0) + { + end += got; + } + } + + private async ValueTask ParseHeaderAsync(CancellationToken cancellationToken) + { + headerParsed = true; + + byte[] hdr = new byte[LzwConstants.HDR_SIZE]; + + int result = await baseInputStream + .ReadAsync(hdr, 0, hdr.Length, cancellationToken) + .ConfigureAwait(false); + + if (result < 0) + { + throw new IncompleteArchiveException("Failed to read LZW header"); + } + + if (hdr[0] != (LzwConstants.MAGIC >> 8) || hdr[1] != (LzwConstants.MAGIC & 0xff)) + { + throw new IncompleteArchiveException( + String.Format( + Constants.DefaultCultureInfo, + "Wrong LZW header. Magic bytes don't match. 0x{0:x2} 0x{1:x2}", + hdr[0], + hdr[1] + ) + ); + } + + blockMode = (hdr[2] & LzwConstants.BLOCK_MODE_MASK) > 0; + maxBits = hdr[2] & LzwConstants.BIT_MASK; + + if (maxBits > LzwConstants.MAX_BITS) + { + throw new ArchiveException( + "Stream compressed with " + + maxBits + + " bits, but decompression can only handle " + + LzwConstants.MAX_BITS + + " bits." + ); + } + + if (maxBits < LzwConstants.INIT_BITS) + { + throw new InvalidFormatException( + "Stream compressed with " + + maxBits + + " bits, but minimum supported is " + + LzwConstants.INIT_BITS + + " bits." + ); + } + + if ((hdr[2] & LzwConstants.RESERVED_MASK) > 0) + { + throw new ArchiveException("Unsupported bits set in the header."); + } + + maxMaxCode = 1 << maxBits; + nBits = LzwConstants.INIT_BITS; + maxCode = (1 << nBits) - 1; + bitMask = maxCode; + oldCode = -1; + finChar = 0; + freeEnt = blockMode ? TBL_FIRST : 256; + + tabPrefix = new int[1 << maxBits]; + tabSuffix = new byte[1 << maxBits]; + stack = new byte[1 << maxBits]; + stackP = stack.Length; + + for (int idx = 255; idx >= 0; idx--) + { + tabSuffix[idx] = (byte)idx; + } + } +} diff --git a/src/SharpCompress/Compressors/Lzw/LzwStream.cs b/src/SharpCompress/Compressors/Lzw/LzwStream.cs new file mode 100644 index 00000000..0aa47222 --- /dev/null +++ b/src/SharpCompress/Compressors/Lzw/LzwStream.cs @@ -0,0 +1,633 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Compressors.Lzw; + +/// +/// This filter stream is used to decompress a LZW format stream. +/// Specifically, a stream that uses the LZC compression method. +/// This file format is usually associated with the .Z file extension. +/// +/// See http://en.wikipedia.org/wiki/Compress +/// See http://wiki.wxwidgets.org/Development:_Z_File_Format +/// +/// The file header consists of 3 (or optionally 4) bytes. The first two bytes +/// contain the magic marker "0x1f 0x9d", followed by a byte of flags. +/// +/// Based on Java code by Ronald Tschalar, which in turn was based on the unlzw.c +/// code in the gzip package. +/// +/// This sample shows how to unzip a compressed file +/// +/// using System; +/// using System.IO; +/// +/// using ICSharpCode.SharpZipLib.Core; +/// using ICSharpCode.SharpZipLib.LZW; +/// +/// class MainClass +/// { +/// public static void Main(string[] args) +/// { +/// using (Stream inStream = new LzwInputStream(File.OpenRead(args[0]))) +/// using (FileStream outStream = File.Create(Path.GetFileNameWithoutExtension(args[0]))) { +/// byte[] buffer = new byte[4096]; +/// StreamUtils.Copy(inStream, outStream, buffer); +/// // OR +/// inStream.Read(buffer, 0, buffer.Length); +/// // now do something with the buffer +/// } +/// } +/// } +/// +/// +public partial class LzwStream : Stream +{ + public static bool IsLzwStream(Stream stream) + { + try + { + byte[] hdr = new byte[LzwConstants.HDR_SIZE]; + + int result = stream.Read(hdr, 0, hdr.Length); + + // Check the magic marker + if (result < 0) + { + throw new IncompleteArchiveException("Failed to read LZW header"); + } + + if (hdr[0] != (LzwConstants.MAGIC >> 8) || hdr[1] != (LzwConstants.MAGIC & 0xff)) + { + throw new IncompleteArchiveException( + String.Format( + Constants.DefaultCultureInfo, + "Wrong LZW header. Magic bytes don't match. 0x{0:x2} 0x{1:x2}", + hdr[0], + hdr[1] + ) + ); + } + } + catch (Exception) + { + return false; + } + return true; + } + + /// + /// Gets or sets a flag indicating ownership of underlying stream. + /// When the flag is true will close the underlying stream also. + /// + /// The default value is true. + public bool IsStreamOwner { get; set; } = false; + + /// + /// Creates a LzwInputStream + /// + /// + /// The stream to read compressed data from (baseInputStream LZW format) + /// + public LzwStream(Stream baseInputStream) + { + this.baseInputStream = baseInputStream; + } + + /// + /// See + /// + /// + public override int ReadByte() + { + int b = Read(one, 0, 1); + if (b == 1) + { + return (one[0] & 0xff); + } + + return -1; + } + + /// + /// Reads decompressed data into the provided buffer byte array + /// + /// + /// The array to read and decompress data into + /// + /// + /// The offset indicating where the data should be placed + /// + /// + /// The number of bytes to decompress + /// + /// The number of bytes read. Zero signals the end of stream + public override int Read(byte[] buffer, int offset, int count) + { + if (!headerParsed) + { + try + { + ParseHeader(); + } + catch + { + eof = true; + throw; + } + } + + if (eof) + { + return 0; + } + + int start = offset; + + /* Using local copies of various variables speeds things up by as + * much as 30% in Java! Performance not tested in C#. + */ + int[] lTabPrefix = tabPrefix; + byte[] lTabSuffix = tabSuffix; + byte[] lStack = stack; + int lNBits = nBits; + int lMaxCode = maxCode; + int lMaxMaxCode = maxMaxCode; + int lBitMask = bitMask; + int lOldCode = oldCode; + byte lFinChar = finChar; + int lStackP = stackP; + int lFreeEnt = freeEnt; + byte[] lData = data; + int lBitPos = bitPos; + + // empty stack if stuff still left + int sSize = lStack.Length - lStackP; + if (sSize > 0) + { + int num = (sSize >= count) ? count : sSize; + Array.Copy(lStack, lStackP, buffer, offset, num); + offset += num; + count -= num; + lStackP += num; + } + + if (count == 0) + { + stackP = lStackP; + return offset - start; + } + + // loop, filling local buffer until enough data has been decompressed + MainLoop: + do + { + if (end < EXTRA) + { + Fill(); + } + + int bitIn = (got > 0) ? (end - end % lNBits) << 3 : (end << 3) - (lNBits - 1); + + while (lBitPos < bitIn) + { + #region A + + // handle 1-byte reads correctly + if (count == 0) + { + nBits = lNBits; + maxCode = lMaxCode; + maxMaxCode = lMaxMaxCode; + bitMask = lBitMask; + oldCode = lOldCode; + finChar = lFinChar; + stackP = lStackP; + freeEnt = lFreeEnt; + bitPos = lBitPos; + + return offset - start; + } + + // check for code-width expansion + if (lFreeEnt > lMaxCode) + { + int nBytes = lNBits << 3; + lBitPos = (lBitPos - 1) + nBytes - (lBitPos - 1 + nBytes) % nBytes; + + lNBits++; + lMaxCode = (lNBits == maxBits) ? lMaxMaxCode : (1 << lNBits) - 1; + + lBitMask = (1 << lNBits) - 1; + lBitPos = ResetBuf(lBitPos); + goto MainLoop; + } + + #endregion A + + #region B + + // read next code + int pos = lBitPos >> 3; + int code = + ( + ( + (lData[pos] & 0xFF) + | ((lData[pos + 1] & 0xFF) << 8) + | ((lData[pos + 2] & 0xFF) << 16) + ) >> (lBitPos & 0x7) + ) & lBitMask; + + lBitPos += lNBits; + + // handle first iteration + if (lOldCode == -1) + { + if (code >= 256) + { + throw new IncompleteArchiveException("corrupt input: " + code + " > 255"); + } + + lFinChar = (byte)(lOldCode = code); + buffer[offset++] = lFinChar; + count--; + continue; + } + + // handle CLEAR code + if (code == TBL_CLEAR && blockMode) + { + Array.Copy(zeros, 0, lTabPrefix, 0, zeros.Length); + lFreeEnt = TBL_FIRST - 1; + + int nBytes = lNBits << 3; + lBitPos = (lBitPos - 1) + nBytes - (lBitPos - 1 + nBytes) % nBytes; + lNBits = LzwConstants.INIT_BITS; + lMaxCode = (1 << lNBits) - 1; + lBitMask = lMaxCode; + + // Code tables reset + + lBitPos = ResetBuf(lBitPos); + goto MainLoop; + } + + #endregion B + + #region C + + // setup + int inCode = code; + lStackP = lStack.Length; + + // Handle KwK case + if (code >= lFreeEnt) + { + if (code > lFreeEnt) + { + throw new IncompleteArchiveException( + "corrupt input: code=" + code + ", freeEnt=" + lFreeEnt + ); + } + + lStack[--lStackP] = lFinChar; + code = lOldCode; + } + + // Generate output characters in reverse order + while (code >= 256) + { + lStack[--lStackP] = lTabSuffix[code]; + code = lTabPrefix[code]; + } + + lFinChar = lTabSuffix[code]; + buffer[offset++] = lFinChar; + count--; + + // And put them out in forward order + sSize = lStack.Length - lStackP; + int num = (sSize >= count) ? count : sSize; + Array.Copy(lStack, lStackP, buffer, offset, num); + offset += num; + count -= num; + lStackP += num; + + #endregion C + + #region D + + // generate new entry in table + if (lFreeEnt < lMaxMaxCode) + { + lTabPrefix[lFreeEnt] = lOldCode; + lTabSuffix[lFreeEnt] = lFinChar; + lFreeEnt++; + } + + // Remember previous code + lOldCode = inCode; + + // if output buffer full, then return + if (count == 0) + { + nBits = lNBits; + maxCode = lMaxCode; + bitMask = lBitMask; + oldCode = lOldCode; + finChar = lFinChar; + stackP = lStackP; + freeEnt = lFreeEnt; + bitPos = lBitPos; + + return offset - start; + } + + #endregion D + } // while + + lBitPos = ResetBuf(lBitPos); + } while (got > 0); // do..while + + nBits = lNBits; + maxCode = lMaxCode; + bitMask = lBitMask; + oldCode = lOldCode; + finChar = lFinChar; + stackP = lStackP; + freeEnt = lFreeEnt; + bitPos = lBitPos; + + eof = true; + return offset - start; + } + + /// + /// Moves the unread data in the buffer to the beginning and resets + /// the pointers. + /// + /// + /// + private int ResetBuf(int bitPosition) + { + int pos = bitPosition >> 3; + Array.Copy(data, pos, data, 0, end - pos); + end -= pos; + return 0; + } + + private void Fill() + { + got = baseInputStream.Read(data, end, data.Length - 1 - end); + if (got > 0) + { + end += got; + } + } + + private void ParseHeader() + { + headerParsed = true; + + byte[] hdr = new byte[LzwConstants.HDR_SIZE]; + + int result = baseInputStream.Read(hdr, 0, hdr.Length); + + // Check the magic marker + if (result < 0) + { + throw new IncompleteArchiveException("Failed to read LZW header"); + } + + if (hdr[0] != (LzwConstants.MAGIC >> 8) || hdr[1] != (LzwConstants.MAGIC & 0xff)) + { + throw new IncompleteArchiveException( + String.Format( + Constants.DefaultCultureInfo, + "Wrong LZW header. Magic bytes don't match. 0x{0:x2} 0x{1:x2}", + hdr[0], + hdr[1] + ) + ); + } + + // Check the 3rd header byte + blockMode = (hdr[2] & LzwConstants.BLOCK_MODE_MASK) > 0; + maxBits = hdr[2] & LzwConstants.BIT_MASK; + + if (maxBits > LzwConstants.MAX_BITS) + { + throw new ArchiveException( + "Stream compressed with " + + maxBits + + " bits, but decompression can only handle " + + LzwConstants.MAX_BITS + + " bits." + ); + } + + if (maxBits < LzwConstants.INIT_BITS) + { + throw new InvalidFormatException( + "Stream compressed with " + + maxBits + + " bits, but minimum supported is " + + LzwConstants.INIT_BITS + + " bits." + ); + } + + if ((hdr[2] & LzwConstants.RESERVED_MASK) > 0) + { + throw new ArchiveException("Unsupported bits set in the header."); + } + + // Initialize variables + maxMaxCode = 1 << maxBits; + nBits = LzwConstants.INIT_BITS; + maxCode = (1 << nBits) - 1; + bitMask = maxCode; + oldCode = -1; + finChar = 0; + freeEnt = blockMode ? TBL_FIRST : 256; + + tabPrefix = new int[1 << maxBits]; + tabSuffix = new byte[1 << maxBits]; + stack = new byte[1 << maxBits]; + stackP = stack.Length; + + for (int idx = 255; idx >= 0; idx--) + { + tabSuffix[idx] = (byte)idx; + } + } + + #region Stream Overrides + + /// + /// Gets a value indicating whether the current stream supports reading + /// + public override bool CanRead + { + get { return baseInputStream.CanRead; } + } + + /// + /// Gets a value of false indicating seeking is not supported for this stream. + /// + public override bool CanSeek + { + get { return false; } + } + + /// + /// Gets a value of false indicating that this stream is not writeable. + /// + public override bool CanWrite + { + get { return false; } + } + + /// + /// A value representing the length of the stream in bytes. + /// + public override long Length + { + get { return got; } + } + + /// + /// The current position within the stream. + /// Throws a NotSupportedException when attempting to set the position + /// + /// Attempting to set the position + public override long Position + { + get { return baseInputStream.Position; } + set { throw new NotSupportedException("InflaterInputStream Position not supported"); } + } + + /// + /// Flushes the baseInputStream + /// + public override void Flush() + { + baseInputStream.Flush(); + } + + /// + /// Sets the position within the current stream + /// Always throws a NotSupportedException + /// + /// The relative offset to seek to. + /// The defining where to seek from. + /// The new position in the stream. + /// Any access + public override long Seek(long offset, SeekOrigin origin) + { + throw new NotSupportedException("Seek not supported"); + } + + /// + /// Set the length of the current stream + /// Always throws a NotSupportedException + /// + /// The new length value for the stream. + /// Any access + public override void SetLength(long value) + { + throw new NotSupportedException("InflaterInputStream SetLength not supported"); + } + + /// + /// Writes a sequence of bytes to stream and advances the current position + /// This method always throws a NotSupportedException + /// + /// The buffer containing data to write. + /// The offset of the first byte to write. + /// The number of bytes to write. + /// Any access + public override void Write(byte[] buffer, int offset, int count) + { + throw new NotSupportedException("InflaterInputStream Write not supported"); + } + + /// + /// Writes one byte to the current stream and advances the current position + /// Always throws a NotSupportedException + /// + /// The byte to write. + /// Any access + public override void WriteByte(byte value) + { + throw new NotSupportedException("InflaterInputStream WriteByte not supported"); + } + + /// + /// Closes the input stream. When + /// is true the underlying stream is also closed. + /// + protected override void Dispose(bool disposing) + { + if (!isClosed) + { + isClosed = true; + if (IsStreamOwner) + { + baseInputStream.Dispose(); + } + } + base.Dispose(disposing); + } + + #endregion Stream Overrides + + #region Instance Fields + + private Stream baseInputStream; + + /// + /// Flag indicating wether this instance has been closed or not. + /// + private bool isClosed; + + private readonly byte[] one = new byte[1]; + private bool headerParsed; + + // string table stuff + private const int TBL_CLEAR = 0x100; + + private const int TBL_FIRST = TBL_CLEAR + 1; + + private int[] tabPrefix = []; // + private byte[] tabSuffix = []; // + private readonly int[] zeros = new int[256]; + private byte[] stack = []; // + + // various state + private bool blockMode; + + private int nBits; + private int maxBits; + private int maxMaxCode; + private int maxCode; + private int bitMask; + private int oldCode; + private byte finChar; + private int stackP; + private int freeEnt; + + // input buffer + private readonly byte[] data = new byte[1024 * 8]; + + private int bitPos; + private int end; + private int got; + private bool eof; + private const int EXTRA = 64; + + #endregion Instance Fields +} diff --git a/src/SharpCompress/Compressors/PPMd/H/ModelPPM.cs b/src/SharpCompress/Compressors/PPMd/H/ModelPPM.cs index 3b2d22ba..565f10ca 100644 --- a/src/SharpCompress/Compressors/PPMd/H/ModelPPM.cs +++ b/src/SharpCompress/Compressors/PPMd/H/ModelPPM.cs @@ -1,28 +1,36 @@ #nullable disable using System; +using System.Buffers; using System.IO; using System.Text; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Compressors.Rar; using Decoder = SharpCompress.Compressors.LZMA.RangeCoder.Decoder; namespace SharpCompress.Compressors.PPMd.H; -internal class ModelPpm +internal class ModelPpm : IDisposable { + private const int SEE2_CONTEXT_ROWS = 25; + private const int SEE2_CONTEXT_COLUMNS = 16; + private const int SEE2_CONTEXT_SIZE = SEE2_CONTEXT_ROWS * SEE2_CONTEXT_COLUMNS; + private const int BIN_SUMM_ROWS = 128; + private const int BIN_SUMM_COLUMNS = 64; + private const int BIN_SUMM_SIZE = BIN_SUMM_ROWS * BIN_SUMM_COLUMNS; + private void InitBlock() { - for (var i = 0; i < 25; i++) + for (var i = 0; i < SEE2_CONTEXT_SIZE; i++) { - _see2Cont[i] = new See2Context[16]; - } - for (var i2 = 0; i2 < 128; i2++) - { - _binSumm[i2] = new int[64]; + _see2Cont[i] = new See2Context(); } + + _binSumm = ArrayPool.Shared.Rent(BIN_SUMM_SIZE); } - public SubAllocator SubAlloc { get; } = new SubAllocator(); + public SubAllocator SubAlloc { get; } = new(); public virtual See2Context DummySee2Cont => _dummySee2Cont; @@ -66,8 +74,6 @@ internal class ModelPpm set => _hiBitsFlag = value & 0xff; } - public virtual int[][] BinSumm => _binSumm; - internal RangeCoder Coder { get; private set; } internal State FoundState { get; private set; } @@ -93,9 +99,9 @@ internal class ModelPpm public const int MAX_FREQ = 124; - private readonly See2Context[][] _see2Cont = new See2Context[25][]; + private readonly See2Context[] _see2Cont = new See2Context[SEE2_CONTEXT_SIZE]; - private See2Context _dummySee2Cont; + private readonly See2Context _dummySee2Cont = new(); private PpmContext _minContext; //medContext @@ -110,18 +116,12 @@ internal class ModelPpm private readonly int[] _charMask = new int[256]; - private readonly int[] _ns2Indx = new int[256]; - - private readonly int[] _ns2BsIndx = new int[256]; - - private readonly int[] _hb2Flag = new int[256]; - // byte EscCount, PrevSuccess, HiBitsFlag; private int _escCount, _prevSuccess, _hiBitsFlag; - private readonly int[][] _binSumm = new int[128][]; // binary SEE-contexts + private int[] _binSumm; // binary SEE-contexts private static readonly int[] INIT_BIN_ESC = { @@ -132,43 +132,100 @@ internal class ModelPpm 0x64A1, 0x5ABC, 0x6632, - 0x6051 + 0x6051, }; + private static readonly int[] NS2_INDX = CreateNs2Indx(); + + private static readonly int[] NS2_BS_INDX = CreateNs2BsIndx(); + + private static readonly int[] HB2_FLAG = CreateHb2Flag(); + + private static int[] CreateNs2Indx() + { + var result = new int[256]; + int i, + k, + m, + step; + for (i = 0; i < 3; i++) + { + result[i] = i; + } + for (m = i, k = 1, step = 1; i < 256; i++) + { + result[i] = m; + if (--k == 0) + { + k = ++step; + m++; + } + } + return result; + } + + private static int[] CreateNs2BsIndx() + { + var result = new int[256]; + result[0] = 0; + result[1] = 2; + for (var j = 0; j < 9; j++) + { + result[2 + j] = 4; + } + for (var j = 0; j < 256 - 11; j++) + { + result[11 + j] = 6; + } + return result; + } + + private static int[] CreateHb2Flag() + { + var result = new int[256]; + for (var j = 0; j < 0x100 - 0x40; j++) + { + result[0x40 + j] = 0x08; + } + return result; + } + // Temp fields //UPGRADE_NOTE: Final was removed from the declaration of 'tempState1 '. "ms-help://MS.VSCC.v80/dv_commoner/local/redirect.htm?index='!DefaultContextWindowIndex'&keyword='jlca1003'" - private readonly State _tempState1 = new State(null); + private readonly State _tempState1 = new(null); //UPGRADE_NOTE: Final was removed from the declaration of 'tempState2 '. "ms-help://MS.VSCC.v80/dv_commoner/local/redirect.htm?index='!DefaultContextWindowIndex'&keyword='jlca1003'" - private readonly State _tempState2 = new State(null); + private readonly State _tempState2 = new(null); //UPGRADE_NOTE: Final was removed from the declaration of 'tempState3 '. "ms-help://MS.VSCC.v80/dv_commoner/local/redirect.htm?index='!DefaultContextWindowIndex'&keyword='jlca1003'" - private readonly State _tempState3 = new State(null); + private readonly State _tempState3 = new(null); //UPGRADE_NOTE: Final was removed from the declaration of 'tempState4 '. "ms-help://MS.VSCC.v80/dv_commoner/local/redirect.htm?index='!DefaultContextWindowIndex'&keyword='jlca1003'" - private readonly State _tempState4 = new State(null); + private readonly State _tempState4 = new(null); //UPGRADE_NOTE: Final was removed from the declaration of 'tempStateRef1 '. "ms-help://MS.VSCC.v80/dv_commoner/local/redirect.htm?index='!DefaultContextWindowIndex'&keyword='jlca1003'" - private readonly StateRef _tempStateRef1 = new StateRef(); + private readonly StateRef _tempStateRef1 = new(); //UPGRADE_NOTE: Final was removed from the declaration of 'tempStateRef2 '. "ms-help://MS.VSCC.v80/dv_commoner/local/redirect.htm?index='!DefaultContextWindowIndex'&keyword='jlca1003'" - private readonly StateRef _tempStateRef2 = new StateRef(); + private readonly StateRef _tempStateRef2 = new(); //UPGRADE_NOTE: Final was removed from the declaration of 'tempPPMContext1 '. "ms-help://MS.VSCC.v80/dv_commoner/local/redirect.htm?index='!DefaultContextWindowIndex'&keyword='jlca1003'" - private readonly PpmContext _tempPpmContext1 = new PpmContext(null); + private readonly PpmContext _tempPpmContext1 = new(null); //UPGRADE_NOTE: Final was removed from the declaration of 'tempPPMContext2 '. "ms-help://MS.VSCC.v80/dv_commoner/local/redirect.htm?index='!DefaultContextWindowIndex'&keyword='jlca1003'" - private readonly PpmContext _tempPpmContext2 = new PpmContext(null); + private readonly PpmContext _tempPpmContext2 = new(null); //UPGRADE_NOTE: Final was removed from the declaration of 'tempPPMContext3 '. "ms-help://MS.VSCC.v80/dv_commoner/local/redirect.htm?index='!DefaultContextWindowIndex'&keyword='jlca1003'" - private readonly PpmContext _tempPpmContext3 = new PpmContext(null); + private readonly PpmContext _tempPpmContext3 = new(null); //UPGRADE_NOTE: Final was removed from the declaration of 'tempPPMContext4 '. "ms-help://MS.VSCC.v80/dv_commoner/local/redirect.htm?index='!DefaultContextWindowIndex'&keyword='jlca1003'" - private readonly PpmContext _tempPpmContext4 = new PpmContext(null); + private readonly PpmContext _tempPpmContext4 = new(null); //UPGRADE_NOTE: Final was removed from the declaration of 'ps '. "ms-help://MS.VSCC.v80/dv_commoner/local/redirect.htm?index='!DefaultContextWindowIndex'&keyword='jlca1003'" private readonly int[] _ps = new int[MAX_O]; + private bool _isDisposed; + public ModelPpm() { InitBlock(); @@ -178,6 +235,23 @@ internal class ModelPpm //medContext = null; } + public void Dispose() + { + if (_isDisposed) + { + return; + } + + _isDisposed = true; + SubAlloc.StopSubAllocator(); + var binSumm = _binSumm; + if (binSumm is not null) + { + _binSumm = null; + ArrayPool.Shared.Return(binSumm, clearArray: true); + } + } + private void RestartModelRare() { new Span(_charMask).Clear(); @@ -207,67 +281,30 @@ internal class ModelPpm state.SetSuccessor(0); } - for (var i = 0; i < 128; i++) + for (var i = 0; i < BIN_SUMM_ROWS; i++) { for (var k = 0; k < 8; k++) { - for (var m = 0; m < 64; m += 8) + for (var m = 0; m < BIN_SUMM_COLUMNS; m += 8) { - _binSumm[i][k + m] = BIN_SCALE - (INIT_BIN_ESC[k] / (i + 2)); + SetBinSumm(i, k + m, BIN_SCALE - (INIT_BIN_ESC[k] / (i + 2))); } } } - for (var i = 0; i < 25; i++) + for (var i = 0; i < SEE2_CONTEXT_ROWS; i++) { - for (var k = 0; k < 16; k++) + for (var k = 0; k < SEE2_CONTEXT_COLUMNS; k++) { - _see2Cont[i][k].Initialize((5 * i) + 10); + GetSee2Cont(i, k).Initialize((5 * i) + 10); } } } private void StartModelRare(int maxOrder) { - int i, - k, - m, - step; _escCount = 1; _maxOrder = maxOrder; RestartModelRare(); - - // Bug Fixed - _ns2BsIndx[0] = 0; - _ns2BsIndx[1] = 2; - for (var j = 0; j < 9; j++) - { - _ns2BsIndx[2 + j] = 4; - } - for (var j = 0; j < 256 - 11; j++) - { - _ns2BsIndx[11 + j] = 6; - } - for (i = 0; i < 3; i++) - { - _ns2Indx[i] = i; - } - for (m = i, k = 1, step = 1; i < 256; i++) - { - _ns2Indx[i] = m; - if ((--k) == 0) - { - k = ++step; - m++; - } - } - for (var j = 0; j < 0x40; j++) - { - _hb2Flag[j] = 0; - } - for (var j = 0; j < 0x100 - 0x40; j++) - { - _hb2Flag[0x40 + j] = 0x08; - } _dummySee2Cont.Shift = PERIOD_BITS; } @@ -279,13 +316,13 @@ internal class ModelPpm internal bool DecodeInit(IRarUnpack unpackRead, int escChar) { - var maxOrder = unpackRead.Char & 0xff; + var maxOrder = unpackRead.ReadChar() & 0xff; var reset = ((maxOrder & 0x20) != 0); var maxMb = 0; if (reset) { - maxMb = unpackRead.Char; + maxMb = unpackRead.ReadChar(); } else { @@ -296,7 +333,7 @@ internal class ModelPpm } if ((maxOrder & 0x40) != 0) { - escChar = unpackRead.Char; + escChar = unpackRead.ReadChar(); unpackRead.PpmEscChar = escChar; } Coder = new RangeCoder(unpackRead); @@ -318,19 +355,62 @@ internal class ModelPpm //medContext = new PPMContext(Heap); _maxContext = new PpmContext(Heap); FoundState = new State(Heap); - _dummySee2Cont = new See2Context(); - for (var i = 0; i < 25; i++) - { - for (var j = 0; j < 16; j++) - { - _see2Cont[i][j] = new See2Context(); - } - } StartModelRare(maxOrder); } return (_minContext.Address != 0); } + internal async ValueTask DecodeInitAsync( + IRarUnpack unpackRead, + int escChar, + CancellationToken cancellationToken = default + ) + { + var maxOrder = + await unpackRead.ReadCharAsync(cancellationToken).ConfigureAwait(false) & 0xff; + var reset = ((maxOrder & 0x20) != 0); + + var maxMb = 0; + if (reset) + { + maxMb = await unpackRead.ReadCharAsync(cancellationToken).ConfigureAwait(false); + } + else + { + if (SubAlloc.GetAllocatedMemory() == 0) + { + return false; + } + } + if ((maxOrder & 0x40) != 0) + { + escChar = await unpackRead.ReadCharAsync(cancellationToken).ConfigureAwait(false); + unpackRead.PpmEscChar = escChar; + } + Coder = new RangeCoder(); + await Coder.InitAsync(unpackRead, cancellationToken).ConfigureAwait(false); + if (reset) + { + maxOrder = (maxOrder & 0x1f) + 1; + if (maxOrder > 16) + { + maxOrder = 16 + ((maxOrder - 16) * 3); + } + if (maxOrder == 1) + { + SubAlloc.StopSubAllocator(); + return false; + } + SubAlloc.StartSubAllocator((maxMb + 1) << 20); + _minContext = new PpmContext(Heap); + + _maxContext = new PpmContext(Heap); + FoundState = new State(Heap); + StartModelRare(maxOrder); + } + return _minContext.Address != 0; + } + public virtual int DecodeChar() { // Debug @@ -400,17 +480,95 @@ internal class ModelPpm return (symbol); } - public virtual See2Context[][] GetSee2Cont() => _see2Cont; + public virtual async ValueTask DecodeCharAsync( + CancellationToken cancellationToken = default + ) + { + // Debug + //subAlloc.dumpHeap(); + + if (_minContext.Address <= SubAlloc.PText || _minContext.Address > SubAlloc.HeapEnd) + { + return (-1); + } + + if (_minContext.NumStats != 1) + { + if ( + _minContext.FreqData.GetStats() <= SubAlloc.PText + || _minContext.FreqData.GetStats() > SubAlloc.HeapEnd + ) + { + return (-1); + } + if (!_minContext.DecodeSymbol1(this)) + { + return (-1); + } + } + else + { + _minContext.DecodeBinSymbol(this); + } + Coder.Decode(); + while (FoundState.Address == 0) + { + await Coder.AriDecNormalizeAsync(cancellationToken).ConfigureAwait(false); + do + { + _orderFall++; + _minContext.Address = _minContext.GetSuffix(); // =MinContext->Suffix; + if (_minContext.Address <= SubAlloc.PText || _minContext.Address > SubAlloc.HeapEnd) + { + return (-1); + } + } while (_minContext.NumStats == _numMasked); + if (!_minContext.DecodeSymbol2(this)) + { + return (-1); + } + Coder.Decode(); + } + var symbol = FoundState.Symbol; + if ((_orderFall == 0) && FoundState.GetSuccessor() > SubAlloc.PText) + { + // MinContext=MaxContext=FoundState->Successor; + var addr = FoundState.GetSuccessor(); + _minContext.Address = addr; + _maxContext.Address = addr; + } + else + { + UpdateModel(); + + //this.foundState.Address=foundState.Address);//TODO just 4 debugging + if (_escCount == 0) + { + ClearMask(); + } + } + await Coder.AriDecNormalizeAsync(cancellationToken).ConfigureAwait(false); // ARI_DEC_NORMALIZE(Coder.code,Coder.low,Coder.range,Coder.UnpackRead); + return (symbol); + } + + public virtual See2Context GetSee2Cont(int row, int column) => + _see2Cont[(row * SEE2_CONTEXT_COLUMNS) + column]; + + public virtual int GetBinSumm(int row, int column) => + _binSumm[(row * BIN_SUMM_COLUMNS) + column]; + + public virtual void SetBinSumm(int row, int column, int value) => + _binSumm[(row * BIN_SUMM_COLUMNS) + column] = value; public virtual void IncEscCount(int dEscCount) => EscCount += dEscCount; public virtual void IncRunLength(int dRunLength) => RunLength += dRunLength; - public virtual int[] GetHb2Flag() => _hb2Flag; + public virtual int[] GetHb2Flag() => HB2_FLAG; - public virtual int[] GetNs2BsIndx() => _ns2BsIndx; + public virtual int[] GetNs2BsIndx() => NS2_BS_INDX; - public virtual int[] GetNs2Indx() => _ns2Indx; + public virtual int[] GetNs2Indx() => NS2_INDX; private int CreateSuccessors(bool skip, State p1) { @@ -774,14 +932,35 @@ internal class ModelPpm //medContext = new PPMContext(Heap); _maxContext = new PpmContext(Heap); FoundState = new State(Heap); - _dummySee2Cont = new See2Context(); - for (var i = 0; i < 25; i++) + StartModelRare(maxOrder); + + return (_minContext.Address != 0); + } + + internal async ValueTask DecodeInitAsync( + Stream stream, + int maxOrder, + int maxMemory, + CancellationToken cancellationToken = default + ) + { + if (stream != null) { - for (var j = 0; j < 16; j++) - { - _see2Cont[i][j] = new See2Context(); - } + Coder = new RangeCoder(); + await Coder.InitAsync(stream, cancellationToken).ConfigureAwait(false); } + + if (maxOrder == 1) + { + SubAlloc.StopSubAllocator(); + return (false); + } + SubAlloc.StartSubAllocator(maxMemory); + _minContext = new PpmContext(Heap); + + //medContext = new PPMContext(Heap); + _maxContext = new PpmContext(Heap); + FoundState = new State(Heap); StartModelRare(maxOrder); return (_minContext.Address != 0); @@ -841,7 +1020,7 @@ internal class ModelPpm { return -2; } - _hiBitsFlag = _hb2Flag[FoundState.Symbol]; + _hiBitsFlag = GetHb2Flag()[FoundState.Symbol]; decoder.Decode((uint)hiCnt, (uint)(_minContext.FreqData.SummFreq - hiCnt)); for (i = 0; i < 256; i++) { @@ -862,12 +1041,15 @@ internal class ModelPpm _hiBitsFlag = GetHb2Flag()[FoundState.Symbol]; var off1 = rs.Freq - 1; var off2 = _minContext.GetArrayIndex(this, rs); - var bs = _binSumm[off1][off2]; + var bs = GetBinSumm(off1, off2); if (decoder.DecodeBit((uint)bs, 14) == 0) { byte symbol; - _binSumm[off1][off2] = - (bs + INTERVAL - _minContext.GetMean(bs, PERIOD_BITS, 2)) & 0xFFFF; + SetBinSumm( + off1, + off2, + (bs + INTERVAL - _minContext.GetMean(bs, PERIOD_BITS, 2)) & 0xFFFF + ); FoundState.Address = rs.Address; symbol = (byte)rs.Symbol; rs.IncrementFreq((rs.Freq < 128) ? 1 : 0); @@ -877,7 +1059,7 @@ internal class ModelPpm return symbol; } bs = (bs - _minContext.GetMean(bs, PERIOD_BITS, 2)) & 0xFFFF; - _binSumm[off1][off2] = bs; + SetBinSumm(off1, off2, bs); _initEsc = PpmContext.EXP_ESCAPE[Utility.URShift(bs, 10)]; int i; for (i = 0; i < 256; i++) @@ -955,4 +1137,182 @@ internal class ModelPpm } while (i != 0); } } + + public async ValueTask DecodeCharAsync( + Decoder decoder, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + + if (_minContext.NumStats != 1) + { + var s = _tempState1.Initialize(Heap); + s.Address = _minContext.FreqData.GetStats(); + int i; + int count, + hiCnt; + if ( + (count = (int)decoder.GetThreshold((uint)_minContext.FreqData.SummFreq)) + < (hiCnt = s.Freq) + ) + { + byte symbol; + await decoder.DecodeAsync(0, (uint)s.Freq, cancellationToken).ConfigureAwait(false); + symbol = (byte)s.Symbol; + _minContext.update1_0(this, s.Address); + NextContext(); + return symbol; + } + _prevSuccess = 0; + i = _minContext.NumStats - 1; + do + { + s.IncrementAddress(); + if ((hiCnt += s.Freq) > count) + { + byte symbol; + await decoder + .DecodeAsync((uint)(hiCnt - s.Freq), (uint)s.Freq, cancellationToken) + .ConfigureAwait(false); + symbol = (byte)s.Symbol; + _minContext.Update1(this, s.Address); + NextContext(); + return symbol; + } + } while (--i > 0); + if (count >= _minContext.FreqData.SummFreq) + { + return -2; + } + _hiBitsFlag = GetHb2Flag()[FoundState.Symbol]; + await decoder + .DecodeAsync( + (uint)hiCnt, + (uint)(_minContext.FreqData.SummFreq - hiCnt), + cancellationToken + ) + .ConfigureAwait(false); + for (i = 0; i < 256; i++) + { + _charMask[i] = -1; + } + _charMask[s.Symbol] = 0; + i = _minContext.NumStats - 1; + do + { + s.DecrementAddress(); + _charMask[s.Symbol] = 0; + } while (--i > 0); + } + else + { + var rs = _tempState1.Initialize(Heap); + rs.Address = _minContext.GetOneState().Address; + _hiBitsFlag = GetHb2Flag()[FoundState.Symbol]; + var off1 = rs.Freq - 1; + var off2 = _minContext.GetArrayIndex(this, rs); + var bs = GetBinSumm(off1, off2); + if ( + await decoder.DecodeBitAsync((uint)bs, 14, cancellationToken).ConfigureAwait(false) + == 0 + ) + { + byte symbol; + SetBinSumm( + off1, + off2, + (bs + INTERVAL - _minContext.GetMean(bs, PERIOD_BITS, 2)) & 0xFFFF + ); + FoundState.Address = rs.Address; + symbol = (byte)rs.Symbol; + rs.IncrementFreq((rs.Freq < 128) ? 1 : 0); + _prevSuccess = 1; + IncRunLength(1); + NextContext(); + return symbol; + } + bs = (bs - _minContext.GetMean(bs, PERIOD_BITS, 2)) & 0xFFFF; + SetBinSumm(off1, off2, bs); + _initEsc = PpmContext.EXP_ESCAPE[Utility.URShift(bs, 10)]; + int i; + for (i = 0; i < 256; i++) + { + _charMask[i] = -1; + } + _charMask[rs.Symbol] = 0; + _prevSuccess = 0; + } + for (; ; ) + { + var s = _tempState1.Initialize(Heap); + int i; + int count, + hiCnt; + See2Context see; + int num, + numMasked = _minContext.NumStats; + do + { + _orderFall++; + _minContext.Address = _minContext.GetSuffix(); + if (_minContext.Address <= SubAlloc.PText || _minContext.Address > SubAlloc.HeapEnd) + { + return -1; + } + } while (_minContext.NumStats == numMasked); + hiCnt = 0; + s.Address = _minContext.FreqData.GetStats(); + i = 0; + num = _minContext.NumStats - numMasked; + do + { + var k = _charMask[s.Symbol]; + hiCnt += s.Freq & k; + _minContext._ps[i] = s.Address; + s.IncrementAddress(); + i -= k; + } while (i != num); + + see = _minContext.MakeEscFreq(this, numMasked, out var freqSum); + freqSum += hiCnt; + count = (int)decoder.GetThreshold((uint)freqSum); + + if (count < hiCnt) + { + byte symbol; + var ps = _tempState2.Initialize(Heap); + for ( + hiCnt = 0, i = 0, ps.Address = _minContext._ps[i]; + (hiCnt += ps.Freq) <= count; + i++, ps.Address = _minContext._ps[i] + ) + { + ; + } + s.Address = ps.Address; + await decoder + .DecodeAsync((uint)(hiCnt - s.Freq), (uint)s.Freq, cancellationToken) + .ConfigureAwait(false); + see.Update(); + symbol = (byte)s.Symbol; + _minContext.Update2(this, s.Address); + UpdateModel(); + return symbol; + } + if (count >= freqSum) + { + return -2; + } + await decoder + .DecodeAsync((uint)hiCnt, (uint)(freqSum - hiCnt), cancellationToken) + .ConfigureAwait(false); + see.Summ += freqSum; + do + { + s.Address = _minContext._ps[--i]; + _charMask[s.Symbol] = 0; + } while (i != 0); + } + } } diff --git a/src/SharpCompress/Compressors/PPMd/H/PPMContext.cs b/src/SharpCompress/Compressors/PPMd/H/PPMContext.cs index ab00615d..a2a073d2 100644 --- a/src/SharpCompress/Compressors/PPMd/H/PPMContext.cs +++ b/src/SharpCompress/Compressors/PPMd/H/PPMContext.cs @@ -64,19 +64,19 @@ internal class PpmContext : Pointer // Temp fields //UPGRADE_NOTE: Final was removed from the declaration of 'tempState1 '. "ms-help://MS.VSCC.v80/dv_commoner/local/redirect.htm?index='!DefaultContextWindowIndex'&keyword='jlca1003'" - private readonly State _tempState1 = new State(null); + private readonly State _tempState1 = new(null); //UPGRADE_NOTE: Final was removed from the declaration of 'tempState2 '. "ms-help://MS.VSCC.v80/dv_commoner/local/redirect.htm?index='!DefaultContextWindowIndex'&keyword='jlca1003'" - private readonly State _tempState2 = new State(null); + private readonly State _tempState2 = new(null); //UPGRADE_NOTE: Final was removed from the declaration of 'tempState3 '. "ms-help://MS.VSCC.v80/dv_commoner/local/redirect.htm?index='!DefaultContextWindowIndex'&keyword='jlca1003'" - private readonly State _tempState3 = new State(null); + private readonly State _tempState3 = new(null); //UPGRADE_NOTE: Final was removed from the declaration of 'tempState4 '. "ms-help://MS.VSCC.v80/dv_commoner/local/redirect.htm?index='!DefaultContextWindowIndex'&keyword='jlca1003'" - private readonly State _tempState4 = new State(null); + private readonly State _tempState4 = new(null); //UPGRADE_NOTE: Final was removed from the declaration of 'tempState5 '. "ms-help://MS.VSCC.v80/dv_commoner/local/redirect.htm?index='!DefaultContextWindowIndex'&keyword='jlca1003'" - private readonly State _tempState5 = new State(null); + private readonly State _tempState5 = new(null); private PpmContext _tempPpmContext; //UPGRADE_NOTE: Final was removed from the declaration of 'ps '. "ms-help://MS.VSCC.v80/dv_commoner/local/redirect.htm?index='!DefaultContextWindowIndex'&keyword='jlca1003'" @@ -265,7 +265,7 @@ internal class PpmContext : Pointer model.HiBitsFlag = model.GetHb2Flag()[model.FoundState.Symbol]; var off1 = rs.Freq - 1; var off2 = GetArrayIndex(model, rs); - var bs = model.BinSumm[off1][off2]; + var bs = model.GetBinSumm(off1, off2); if (model.Coder.GetCurrentShiftCount(ModelPpm.TOT_BITS) < bs) { model.FoundState.Address = rs.Address; @@ -273,7 +273,7 @@ internal class PpmContext : Pointer model.Coder.SubRange.LowCount = 0; model.Coder.SubRange.HighCount = bs; bs = ((bs + ModelPpm.INTERVAL - GetMean(bs, ModelPpm.PERIOD_BITS, 2)) & 0xffff); - model.BinSumm[off1][off2] = bs; + model.SetBinSumm(off1, off2, bs); model.PrevSuccess = 1; model.IncRunLength(1); } @@ -281,7 +281,7 @@ internal class PpmContext : Pointer { model.Coder.SubRange.LowCount = bs; bs = (bs - GetMean(bs, ModelPpm.PERIOD_BITS, 2)) & 0xFFFF; - model.BinSumm[off1][off2] = bs; + model.SetBinSumm(off1, off2, bs); model.Coder.SubRange.HighCount = ModelPpm.BIN_SCALE; model.InitEsc = EXP_ESCAPE[Utility.URShift(bs, 10)]; model.NumMasked = 1; @@ -431,7 +431,7 @@ internal class PpmContext : Pointer idx2 += 2 * ((_freqData.SummFreq < 11 * numStats) ? 1 : 0); idx2 += 4 * ((model.NumMasked > diff) ? 1 : 0); idx2 += model.HiBitsFlag; - psee2C = model.GetSee2Cont()[idx1][idx2]; + psee2C = model.GetSee2Cont(idx1, idx2); model.Coder.SubRange.Scale = psee2C.Mean; } else @@ -457,7 +457,7 @@ internal class PpmContext : Pointer idx2 += 2 * ((_freqData.SummFreq < 11 * numStats) ? 1 : 0); idx2 += 4 * ((numMasked > nonMasked) ? 1 : 0); idx2 += model.HiBitsFlag; - psee2C = model.GetSee2Cont()[idx1][idx2]; + psee2C = model.GetSee2Cont(idx1, idx2); escFreq = psee2C.Mean; } else diff --git a/src/SharpCompress/Compressors/PPMd/H/RangeCoder.cs b/src/SharpCompress/Compressors/PPMd/H/RangeCoder.cs index 6659cb83..7cda0fec 100644 --- a/src/SharpCompress/Compressors/PPMd/H/RangeCoder.cs +++ b/src/SharpCompress/Compressors/PPMd/H/RangeCoder.cs @@ -2,7 +2,10 @@ using System.IO; using System.Text; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Compressors.Rar; +using SharpCompress.IO; namespace SharpCompress.Compressors.PPMd.H; @@ -16,7 +19,7 @@ internal class RangeCoder private long _low, _code, _range; - private readonly IRarUnpack _unpackRead; + private IRarUnpack _unpackRead; private readonly Stream _stream; internal RangeCoder(IRarUnpack unpackRead) @@ -31,6 +34,26 @@ internal class RangeCoder Init(); } + internal RangeCoder() { } + + internal async ValueTask InitAsync( + IRarUnpack unpackRead, + CancellationToken cancellationToken = default + ) + { + _unpackRead = unpackRead; + SubRange = new SubRange(); + + _low = _code = 0L; + _range = 0xFFFFffffL; + for (var i = 0; i < 4; i++) + { + _code = + ((_code << 8) | await ReadCharAsync(cancellationToken).ConfigureAwait(false)) + & UINT_MASK; + } + } + private void Init() { SubRange = new SubRange(); @@ -39,7 +62,23 @@ internal class RangeCoder _range = 0xFFFFffffL; for (var i = 0; i < 4; i++) { - _code = ((_code << 8) | Char) & UINT_MASK; + _code = ((_code << 8) | ReadChar()) & UINT_MASK; + } + } + + internal async ValueTask InitAsync(Stream stream, CancellationToken cancellationToken = default) + { + SubRange = new SubRange(); + + _low = _code = 0L; + _range = 0xFFFFffffL; + + byte[] buffer = new byte[4]; + await stream.ReadFullyAsync(buffer, 0, 4, cancellationToken).ConfigureAwait(false); + + for (var i = 0; i < 4; i++) + { + _code = ((_code << 8) | buffer[i]) & UINT_MASK; } } @@ -52,20 +91,17 @@ internal class RangeCoder } } - private long Char + private long ReadChar() { - get + if (_unpackRead != null) { - if (_unpackRead != null) - { - return (_unpackRead.Char); - } - if (_stream != null) - { - return _stream.ReadByte(); - } - return -1; + return (_unpackRead.ReadChar()); } + if (_stream != null) + { + return _stream.ReadByte(); + } + return -1; } internal SubRange SubRange { get; private set; } @@ -100,7 +136,40 @@ internal class RangeCoder _range = (-_low & (BOT - 1)) & UINT_MASK; c2 = false; } - _code = ((_code << 8) | Char) & UINT_MASK; + _code = ((_code << 8) | ReadChar()) & UINT_MASK; + _range = (_range << 8) & UINT_MASK; + _low = (_low << 8) & UINT_MASK; + } + } + + private async ValueTask ReadCharAsync(CancellationToken cancellationToken = default) + { + if (_unpackRead != null) + { + return await _unpackRead.ReadCharAsync(cancellationToken).ConfigureAwait(false); + } + if (_stream != null) + { + byte[] buffer = new byte[1]; + await _stream.ReadFullyAsync(buffer, 0, 1, cancellationToken).ConfigureAwait(false); + return buffer[0]; + } + return -1; + } + + internal async ValueTask AriDecNormalizeAsync(CancellationToken cancellationToken = default) + { + var c2 = false; + while ((_low ^ (_low + _range)) < TOP || (c2 = _range < BOT)) + { + if (c2) + { + _range = (-_low & (BOT - 1)) & UINT_MASK; + c2 = false; + } + _code = + ((_code << 8) | await ReadCharAsync(cancellationToken).ConfigureAwait(false)) + & UINT_MASK; _range = (_range << 8) & UINT_MASK; _low = (_low << 8) & UINT_MASK; } diff --git a/src/SharpCompress/Compressors/PPMd/H/SubAllocator.cs b/src/SharpCompress/Compressors/PPMd/H/SubAllocator.cs index e507f3c3..df6dfa60 100644 --- a/src/SharpCompress/Compressors/PPMd/H/SubAllocator.cs +++ b/src/SharpCompress/Compressors/PPMd/H/SubAllocator.cs @@ -1,11 +1,12 @@ #nullable disable using System; +using System.Buffers; using System.Text; namespace SharpCompress.Compressors.PPMd.H; -internal class SubAllocator +internal class SubAllocator : IDisposable { public virtual int FakeUnitsStart { @@ -124,21 +125,32 @@ internal class SubAllocator { if (_subAllocatorSize != 0) { + var heap = _heap; _subAllocatorSize = 0; - - //ArrayFactory.BYTES_FACTORY.recycle(heap); _heap = null; _heapStart = 1; // rarfree(HeapStart); + for (var i = 0; i < _freeList.Length; i++) + { + _freeList[i] = null; + } + // Free temp fields _tempRarNode = null; _tempRarMemBlock1 = null; _tempRarMemBlock2 = null; _tempRarMemBlock3 = null; + + if (heap is not null) + { + ArrayPool.Shared.Return(heap, clearArray: true); + } } } + public void Dispose() => StopSubAllocator(); + public virtual int GetAllocatedMemory() => _subAllocatorSize; public virtual bool StartSubAllocator(int saSize) @@ -159,7 +171,8 @@ internal class SubAllocator _tempMemBlockPos = realAllocSize; realAllocSize += RarMemBlock.SIZE; - _heap = new byte[realAllocSize]; + _heap = ArrayPool.Shared.Rent(realAllocSize); + new Span(_heap, 0, realAllocSize).Clear(); _heapStart = 1; _heapEnd = _heapStart + allocSize - UNIT_SIZE; _subAllocatorSize = t; diff --git a/src/SharpCompress/Compressors/PPMd/I1/Allocator.cs b/src/SharpCompress/Compressors/PPMd/I1/Allocator.cs index 7b677a93..c4d8aa4b 100644 --- a/src/SharpCompress/Compressors/PPMd/I1/Allocator.cs +++ b/src/SharpCompress/Compressors/PPMd/I1/Allocator.cs @@ -395,10 +395,8 @@ internal class Allocator unitCountDifference -= unitCount; } - _memoryNodes[UNITS_TO_INDEX[unitCountDifference - 1]].Insert( - newPointer, - unitCountDifference - ); + _memoryNodes[UNITS_TO_INDEX[unitCountDifference - 1]] + .Insert(newPointer, unitCountDifference); } private void GlueFreeBlocks() @@ -457,10 +455,11 @@ internal class Allocator if (INDEX_TO_UNITS[index] != unitCount) { var unitCountDifference = unitCount - INDEX_TO_UNITS[--index]; - _memoryNodes[unitCountDifference - 1].Insert( - memoryNode0 + (unitCount - unitCountDifference), - unitCountDifference - ); + _memoryNodes[unitCountDifference - 1] + .Insert( + memoryNode0 + (unitCount - unitCountDifference), + unitCountDifference + ); } _memoryNodes[index].Insert(memoryNode0, INDEX_TO_UNITS[index]); diff --git a/src/SharpCompress/Compressors/PPMd/I1/Coder.cs b/src/SharpCompress/Compressors/PPMd/I1/Coder.cs index b033a6f8..5fdd3af0 100644 --- a/src/SharpCompress/Compressors/PPMd/I1/Coder.cs +++ b/src/SharpCompress/Compressors/PPMd/I1/Coder.cs @@ -1,6 +1,9 @@ #region Using using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.IO; #endregion @@ -44,6 +47,24 @@ internal class Coder } } + public async ValueTask RangeEncoderNormalizeAsync( + Stream stream, + CancellationToken cancellationToken = default + ) + { + while ( + (_low ^ (_low + _range)) < RANGE_TOP + || _range < RANGE_BOTTOM && ((_range = (uint)-_low & (RANGE_BOTTOM - 1)) != 0 || true) + ) + { + await stream + .WriteAsync(new[] { (byte)(_low >> 24) }, 0, 1, cancellationToken) + .ConfigureAwait(false); + _range <<= 8; + _low <<= 8; + } + } + public void RangeEncodeSymbol() { _low += _lowCount * (_range /= _scale); @@ -65,6 +86,21 @@ internal class Coder } } + public async ValueTask RangeEncoderFlushAsync( + Stream stream, + CancellationToken cancellationToken = default + ) + { + var buffer = new byte[4]; + for (var index = 0; index < buffer.Length; index++) + { + buffer[index] = (byte)(_low >> 24); + _low <<= 8; + } + + await stream.WriteAsync(buffer, 0, buffer.Length, cancellationToken).ConfigureAwait(false); + } + public void RangeDecoderInitialize(Stream stream) { _low = 0; @@ -76,6 +112,24 @@ internal class Coder } } + public async ValueTask RangeDecoderInitializeAsync( + Stream stream, + CancellationToken cancellationToken = default + ) + { + _low = 0; + _code = 0; + _range = uint.MaxValue; + + byte[] buffer = new byte[4]; + await stream.ReadFullyAsync(buffer, 0, 4, cancellationToken).ConfigureAwait(false); + + for (uint index = 0; index < 4; index++) + { + _code = (_code << 8) | buffer[index]; + } + } + public void RangeDecoderNormalize(Stream stream) { while ( @@ -89,6 +143,24 @@ internal class Coder } } + public async ValueTask RangeDecoderNormalizeAsync( + Stream stream, + CancellationToken cancellationToken = default + ) + { + while ( + (_low ^ (_low + _range)) < RANGE_TOP + || _range < RANGE_BOTTOM && ((_range = (uint)-_low & (RANGE_BOTTOM - 1)) != 0 || true) + ) + { + byte[] buffer = new byte[1]; + await stream.ReadFullyAsync(buffer, 0, 1, cancellationToken).ConfigureAwait(false); + _code = (_code << 8) | buffer[0]; + _range <<= 8; + _low <<= 8; + } + } + public uint RangeGetCurrentCount() => (_code - _low) / (_range /= _scale); public uint RangeGetCurrentShiftCount(int rangeShift) => diff --git a/src/SharpCompress/Compressors/PPMd/I1/MemoryNode.cs b/src/SharpCompress/Compressors/PPMd/I1/MemoryNode.cs index 5e978c5c..c8b411d8 100644 --- a/src/SharpCompress/Compressors/PPMd/I1/MemoryNode.cs +++ b/src/SharpCompress/Compressors/PPMd/I1/MemoryNode.cs @@ -27,7 +27,7 @@ internal struct MemoryNode { public uint _address; public byte[] _memory; - public static readonly MemoryNode ZERO = new MemoryNode(0, null); + public static readonly MemoryNode ZERO = new(0, null); public const int SIZE = 12; /// @@ -64,7 +64,7 @@ internal struct MemoryNode public MemoryNode Next { get => - new MemoryNode( + new( _memory[_address + 4] | (((uint)_memory[_address + 5]) << 8) | (((uint)_memory[_address + 6]) << 16) @@ -150,7 +150,7 @@ internal struct MemoryNode /// /// public static implicit operator MemoryNode(Pointer pointer) => - new MemoryNode(pointer._address, pointer._memory); + new(pointer._address, pointer._memory); /// /// Allow pointer-like addition on a memory node. diff --git a/src/SharpCompress/Compressors/PPMd/I1/Model.cs b/src/SharpCompress/Compressors/PPMd/I1/Model.cs index 786cb0e7..41c34f8f 100644 --- a/src/SharpCompress/Compressors/PPMd/I1/Model.cs +++ b/src/SharpCompress/Compressors/PPMd/I1/Model.cs @@ -2,6 +2,9 @@ using System; using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; // This is a port of Dmitry Shkarin's PPMd Variant I Revision 1. // Ported by Michael Bone (mjbone03@yahoo.com.au). @@ -59,7 +62,7 @@ internal partial class Model 0x64A1, 0x5ABC, 0x6632, - 0x6051 + 0x6051, }; private static ReadOnlySpan EXPONENTIAL_ESCAPES => @@ -149,15 +152,9 @@ internal partial class Model /// public void Encode(Stream target, Stream source, PpmdProperties properties) { - if (target is null) - { - throw new ArgumentNullException(nameof(target)); - } + ThrowHelper.ThrowIfNull(target); - if (source is null) - { - throw new ArgumentNullException(nameof(source)); - } + ThrowHelper.ThrowIfNull(source); EncodeStart(properties); EncodeBlock(target, source, true); @@ -232,20 +229,84 @@ internal partial class Model _coder.RangeEncoderFlush(target); } + internal async ValueTask EncodeBlockAsync( + Stream target, + Stream source, + bool final, + CancellationToken cancellationToken = default + ) + { + while (true) + { + cancellationToken.ThrowIfCancellationRequested(); + _minimumContext = _maximumContext; + _numberStatistics = _minimumContext.NumberStatistics; + + var c = source.ReadByte(); + if (c < 0 && !final) + { + return; + } + + if (_numberStatistics != 0) + { + EncodeSymbol1(c, _minimumContext); + _coder.RangeEncodeSymbol(); + } + else + { + EncodeBinarySymbol(c, _minimumContext); + _coder.RangeShiftEncodeSymbol(TOTAL_BIT_COUNT); + } + + while (_foundState == PpmState.ZERO) + { + await _coder + .RangeEncoderNormalizeAsync(target, cancellationToken) + .ConfigureAwait(false); + do + { + _orderFall++; + _minimumContext = _minimumContext.Suffix; + if (_minimumContext == PpmContext.ZERO) + { + goto StopEncoding; + } + } while (_minimumContext.NumberStatistics == _numberMasked); + EncodeSymbol2(c, _minimumContext); + _coder.RangeEncodeSymbol(); + } + + if (_orderFall == 0 && (Pointer)_foundState.Successor >= _allocator._baseUnit) + { + _maximumContext = _foundState.Successor; + } + else + { + UpdateModel(_minimumContext); + if (_escapeCount == 0) + { + ClearMask(); + } + } + + await _coder + .RangeEncoderNormalizeAsync(target, cancellationToken) + .ConfigureAwait(false); + } + + StopEncoding: + await _coder.RangeEncoderFlushAsync(target, cancellationToken).ConfigureAwait(false); + } + /// /// Dencode (ie. decompress) a given source stream, writing the decoded result to the target stream. /// public void Decode(Stream target, Stream source, PpmdProperties properties) { - if (target is null) - { - throw new ArgumentNullException(nameof(target)); - } + ThrowHelper.ThrowIfNull(target); - if (source is null) - { - throw new ArgumentNullException(nameof(source)); - } + ThrowHelper.ThrowIfNull(source); DecodeStart(source, properties); var buffer = new byte[65536]; @@ -263,6 +324,29 @@ internal partial class Model _coder.RangeDecoderInitialize(source); StartModel(properties.ModelOrder, properties.RestorationMethod); _minimumContext = _maximumContext; + if (_minimumContext == PpmContext.ZERO) + { + throw new InvalidFormatException("PPMd: model context not initialized"); + } + _numberStatistics = _minimumContext.NumberStatistics; + return _coder; + } + + internal async ValueTask DecodeStartAsync( + Stream source, + PpmdProperties properties, + CancellationToken cancellationToken = default + ) + { + _allocator = properties._allocator; + _coder = new Coder(); + await _coder.RangeDecoderInitializeAsync(source, cancellationToken).ConfigureAwait(false); + StartModel(properties.ModelOrder, properties.RestorationMethod); + _minimumContext = _maximumContext; + if (_minimumContext == PpmContext.ZERO) + { + throw new InvalidFormatException("PPMd: model context not initialized"); + } _numberStatistics = _minimumContext.NumberStatistics; return _coder; } @@ -330,6 +414,81 @@ internal partial class Model return total; } + internal async ValueTask DecodeBlockAsync( + Stream source, + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken = default + ) + { + if (_minimumContext == PpmContext.ZERO) + { + return 0; + } + + var total = 0; + while (total < count) + { + cancellationToken.ThrowIfCancellationRequested(); + + if (_numberStatistics != 0) + { + DecodeSymbol1(_minimumContext); + } + else + { + DecodeBinarySymbol(_minimumContext); + } + + _coder.RangeRemoveSubrange(); + + while (_foundState == PpmState.ZERO) + { + await _coder + .RangeDecoderNormalizeAsync(source, cancellationToken) + .ConfigureAwait(false); + do + { + _orderFall++; + _minimumContext = _minimumContext.Suffix; + if (_minimumContext == PpmContext.ZERO) + { + goto StopDecoding; + } + } while (_minimumContext.NumberStatistics == _numberMasked); + DecodeSymbol2(_minimumContext); + _coder.RangeRemoveSubrange(); + } + + buffer[offset] = _foundState.Symbol; + offset++; + total++; + + if (_orderFall == 0 && (Pointer)_foundState.Successor >= _allocator._baseUnit) + { + _maximumContext = _foundState.Successor; + } + else + { + UpdateModel(_minimumContext); + if (_escapeCount == 0) + { + ClearMask(); + } + } + + _minimumContext = _maximumContext; + _numberStatistics = _minimumContext.NumberStatistics; + await _coder + .RangeDecoderNormalizeAsync(source, cancellationToken) + .ConfigureAwait(false); + } + + StopDecoding: + return total; + } + #endregion #region Private Methods @@ -349,13 +508,16 @@ internal partial class Model if (modelOrder < 2) { _orderFall = _modelOrder; - for ( - var context = _maximumContext; - context.Suffix != PpmContext.ZERO; - context = context.Suffix - ) + if (_maximumContext != PpmContext.ZERO) { - _orderFall--; + for ( + var context = _maximumContext; + context.Suffix != PpmContext.ZERO; + context = context.Suffix + ) + { + _orderFall--; + } } return; } @@ -866,7 +1028,8 @@ internal partial class Model ); } else if ( - (currentContext.SummaryFrequency += 4) > 128 + (4 * currentContext.NumberStatistics) + (currentContext.SummaryFrequency += 4) + > 128 + (4 * currentContext.NumberStatistics) ) { Refresh((uint)((currentContext.NumberStatistics + 2) >> 1), true, currentContext); diff --git a/src/SharpCompress/Compressors/PPMd/I1/ModelRestorationMethod.cs b/src/SharpCompress/Compressors/PPMd/I1/ModelRestorationMethod.cs index 0a504c99..d85a1a32 100644 --- a/src/SharpCompress/Compressors/PPMd/I1/ModelRestorationMethod.cs +++ b/src/SharpCompress/Compressors/PPMd/I1/ModelRestorationMethod.cs @@ -1,7 +1,6 @@ #region Using - #endregion namespace SharpCompress.Compressors.PPMd.I1; @@ -24,5 +23,5 @@ internal enum ModelRestorationMethod /// /// Freeze the context tree (in some cases may result in poor compression). /// - Freeze = 2 + Freeze = 2, } diff --git a/src/SharpCompress/Compressors/PPMd/I1/Pointer.cs b/src/SharpCompress/Compressors/PPMd/I1/Pointer.cs index 72c557ba..cf798cd2 100644 --- a/src/SharpCompress/Compressors/PPMd/I1/Pointer.cs +++ b/src/SharpCompress/Compressors/PPMd/I1/Pointer.cs @@ -22,7 +22,7 @@ internal struct Pointer { public uint _address; public byte[] _memory; - public static readonly Pointer ZERO = new Pointer(0, null); + public static readonly Pointer ZERO = new(0, null); public const int SIZE = 1; /// @@ -41,26 +41,8 @@ internal struct Pointer /// public byte this[int offset] { - get - { -#if DEBUG - if (_address == 0) - { - throw new InvalidOperationException("The pointer being indexed is a null pointer."); - } -#endif - return _memory[_address + offset]; - } - set - { -#if DEBUG - if (_address == 0) - { - throw new InvalidOperationException("The pointer being indexed is a null pointer."); - } -#endif - _memory[_address + offset] = value; - } + get { return _memory[_address + offset]; } + set { _memory[_address + offset] = value; } } /// @@ -69,7 +51,7 @@ internal struct Pointer /// /// public static implicit operator Pointer(MemoryNode memoryNode) => - new Pointer(memoryNode._address, memoryNode._memory); + new(memoryNode._address, memoryNode._memory); /// /// Allow a to be implicitly converted to a . @@ -77,15 +59,14 @@ internal struct Pointer /// /// public static implicit operator Pointer(Model.PpmContext context) => - new Pointer(context._address, context._memory); + new(context._address, context._memory); /// /// Allow a to be implicitly converted to a . /// /// /// - public static implicit operator Pointer(PpmState state) => - new Pointer(state._address, state._memory); + public static implicit operator Pointer(PpmState state) => new(state._address, state._memory); /// /// Increase the address of a pointer by the given number of bytes. @@ -95,12 +76,6 @@ internal struct Pointer /// public static Pointer operator +(Pointer pointer, int offset) { -#if DEBUG - if (pointer._address == 0) - { - throw new InvalidOperationException("The pointer is a null pointer."); - } -#endif pointer._address = (uint)(pointer._address + offset); return pointer; } @@ -113,12 +88,6 @@ internal struct Pointer /// public static Pointer operator +(Pointer pointer, uint offset) { -#if DEBUG - if (pointer._address == 0) - { - throw new InvalidOperationException("The pointer is a null pointer."); - } -#endif pointer._address += offset; return pointer; } @@ -130,12 +99,6 @@ internal struct Pointer /// public static Pointer operator ++(Pointer pointer) { -#if DEBUG - if (pointer._address == 0) - { - throw new InvalidOperationException("The pointer being incremented is a null pointer."); - } -#endif pointer._address++; return pointer; } @@ -148,12 +111,6 @@ internal struct Pointer /// public static Pointer operator -(Pointer pointer, int offset) { -#if DEBUG - if (pointer._address == 0) - { - throw new InvalidOperationException("The pointer is a null pointer."); - } -#endif pointer._address = (uint)(pointer._address - offset); return pointer; } @@ -166,12 +123,6 @@ internal struct Pointer /// public static Pointer operator -(Pointer pointer, uint offset) { -#if DEBUG - if (pointer._address == 0) - { - throw new InvalidOperationException("The pointer is a null pointer."); - } -#endif pointer._address -= offset; return pointer; } @@ -183,12 +134,6 @@ internal struct Pointer /// public static Pointer operator --(Pointer pointer) { -#if DEBUG - if (pointer._address == 0) - { - throw new InvalidOperationException("The pointer being decremented is a null pointer."); - } -#endif pointer._address--; return pointer; } @@ -201,20 +146,6 @@ internal struct Pointer /// The number of bytes between the two pointers. public static uint operator -(Pointer pointer1, Pointer pointer2) { -#if DEBUG - if (pointer1._address == 0) - { - throw new InvalidOperationException( - "The pointer to the left of the subtraction operator is a null pointer." - ); - } - if (pointer2._address == 0) - { - throw new InvalidOperationException( - "The pointer to the right of the subtraction operator is a null pointer." - ); - } -#endif return pointer1._address - pointer2._address; } @@ -226,20 +157,6 @@ internal struct Pointer /// public static bool operator <(Pointer pointer1, Pointer pointer2) { -#if DEBUG - if (pointer1._address == 0) - { - throw new InvalidOperationException( - "The pointer to the left of the less than operator is a null pointer." - ); - } - if (pointer2._address == 0) - { - throw new InvalidOperationException( - "The pointer to the right of the less than operator is a null pointer." - ); - } -#endif return pointer1._address < pointer2._address; } @@ -251,20 +168,6 @@ internal struct Pointer /// public static bool operator <=(Pointer pointer1, Pointer pointer2) { -#if DEBUG - if (pointer1._address == 0) - { - throw new InvalidOperationException( - "The pointer to the left of the less than or equal to operator is a null pointer." - ); - } - if (pointer2._address == 0) - { - throw new InvalidOperationException( - "The pointer to the right of the less than or equal to operator is a null pointer." - ); - } -#endif return pointer1._address <= pointer2._address; } @@ -276,20 +179,6 @@ internal struct Pointer /// public static bool operator >(Pointer pointer1, Pointer pointer2) { -#if DEBUG - if (pointer1._address == 0) - { - throw new InvalidOperationException( - "The pointer to the left of the greater than operator is a null pointer." - ); - } - if (pointer2._address == 0) - { - throw new InvalidOperationException( - "The pointer to the right of the greater than operator is a null pointer." - ); - } -#endif return pointer1._address > pointer2._address; } @@ -301,20 +190,6 @@ internal struct Pointer /// public static bool operator >=(Pointer pointer1, Pointer pointer2) { -#if DEBUG - if (pointer1._address == 0) - { - throw new InvalidOperationException( - "The pointer to the left of the greater than or equal to operator is a null pointer." - ); - } - if (pointer2._address == 0) - { - throw new InvalidOperationException( - "The pointer to the right of the greater than or equal to operator is a null pointer." - ); - } -#endif return pointer1._address >= pointer2._address; } diff --git a/src/SharpCompress/Compressors/PPMd/I1/PpmContext.cs b/src/SharpCompress/Compressors/PPMd/I1/PpmContext.cs index a37d278d..701039c2 100644 --- a/src/SharpCompress/Compressors/PPMd/I1/PpmContext.cs +++ b/src/SharpCompress/Compressors/PPMd/I1/PpmContext.cs @@ -21,7 +21,7 @@ internal partial class Model { public uint _address; public byte[] _memory; - public static readonly PpmContext ZERO = new PpmContext(0, null); + public static readonly PpmContext ZERO = new(0, null); public const int SIZE = 12; /// @@ -70,7 +70,7 @@ internal partial class Model public PpmState Statistics { get => - new PpmState( + new( _memory[_address + 4] | (((uint)_memory[_address + 5]) << 8) | (((uint)_memory[_address + 6]) << 16) @@ -92,7 +92,7 @@ internal partial class Model public PpmContext Suffix { get => - new PpmContext( + new( _memory[_address + 8] | (((uint)_memory[_address + 9]) << 8) | (((uint)_memory[_address + 10]) << 16) @@ -133,7 +133,7 @@ internal partial class Model /// /// /// - public PpmState FirstState => new PpmState(_address + 2, _memory); + public PpmState FirstState => new(_address + 2, _memory); /// /// Gets or sets the symbol of the first PPM state. This is provided for convenience. The same @@ -164,7 +164,7 @@ internal partial class Model public PpmContext FirstStateSuccessor { get => - new PpmContext( + new( _memory[_address + 4] | (((uint)_memory[_address + 5]) << 8) | (((uint)_memory[_address + 6]) << 16) @@ -186,7 +186,7 @@ internal partial class Model /// /// public static implicit operator PpmContext(Pointer pointer) => - new PpmContext(pointer._address, pointer._memory); + new(pointer._address, pointer._memory); /// /// Allow pointer-like addition on a PPM context. diff --git a/src/SharpCompress/Compressors/PPMd/I1/PpmState.cs b/src/SharpCompress/Compressors/PPMd/I1/PpmState.cs index 757afd00..02e79a34 100644 --- a/src/SharpCompress/Compressors/PPMd/I1/PpmState.cs +++ b/src/SharpCompress/Compressors/PPMd/I1/PpmState.cs @@ -19,7 +19,7 @@ internal struct PpmState { public uint _address; public byte[] _memory; - public static readonly PpmState ZERO = new PpmState(0, null); + public static readonly PpmState ZERO = new(0, null); public const int SIZE = 6; /// @@ -55,7 +55,7 @@ internal struct PpmState public Model.PpmContext Successor { get => - new Model.PpmContext( + new( _memory[_address + 2] | (((uint)_memory[_address + 3]) << 8) | (((uint)_memory[_address + 4]) << 16) @@ -77,7 +77,7 @@ internal struct PpmState /// /// /// - public PpmState this[int offset] => new PpmState((uint)(_address + (offset * SIZE)), _memory); + public PpmState this[int offset] => new((uint)(_address + (offset * SIZE)), _memory); /// /// Allow a pointer to be implicitly converted to a PPM state. @@ -85,7 +85,7 @@ internal struct PpmState /// /// public static implicit operator PpmState(Pointer pointer) => - new PpmState(pointer._address, pointer._memory); + new(pointer._address, pointer._memory); /// /// Allow pointer-like addition on a PPM state. diff --git a/src/SharpCompress/Compressors/PPMd/I1/See2Context.cs b/src/SharpCompress/Compressors/PPMd/I1/See2Context.cs index bc52e432..647a2396 100644 --- a/src/SharpCompress/Compressors/PPMd/I1/See2Context.cs +++ b/src/SharpCompress/Compressors/PPMd/I1/See2Context.cs @@ -1,7 +1,6 @@ #region Using - #endregion namespace SharpCompress.Compressors.PPMd.I1; diff --git a/src/SharpCompress/Compressors/PPMd/PpmdStream.cs b/src/SharpCompress/Compressors/PPMd/PpmdStream.cs index bdc4d65a..b48cf4cf 100644 --- a/src/SharpCompress/Compressors/PPMd/PpmdStream.cs +++ b/src/SharpCompress/Compressors/PPMd/PpmdStream.cs @@ -1,64 +1,166 @@ -#nullable disable +#nullable disable using System; using System.IO; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Compressors.LZMA.RangeCoder; using SharpCompress.Compressors.PPMd.H; using SharpCompress.Compressors.PPMd.I1; +using SharpCompress.IO; namespace SharpCompress.Compressors.PPMd; -public class PpmdStream : Stream +public class PpmdStream : Stream, IAsyncDisposable { private readonly PpmdProperties _properties; private readonly Stream _stream; private readonly bool _compress; - private readonly Model _model; - private readonly ModelPpm _modelH; - private readonly Decoder _decoder; + private Model _model; + private ModelPpm _modelH; + private Decoder _decoder; private long _position; private bool _isDisposed; - public PpmdStream(PpmdProperties properties, Stream stream, bool compress) + private PpmdStream(PpmdProperties properties, Stream stream, bool compress) { _properties = properties; _stream = stream; _compress = compress; - if (properties.Version == PpmdVersion.I1) + InitializeSync(stream, compress); + } + + private PpmdStream( + PpmdProperties properties, + Stream stream, + bool compress, + bool skipInitialization + ) + { + _properties = properties; + _stream = stream; + _compress = compress; + + // Skip initialization - used by CreateAsync + } + + private void InitializeSync(Stream stream, bool compress) + { + if (_properties.Version == PpmdVersion.I1) { _model = new Model(); if (compress) { - _model.EncodeStart(properties); + _model.EncodeStart(_properties); } else { - _model.DecodeStart(stream, properties); + _model.DecodeStart(stream, _properties); } } - if (properties.Version == PpmdVersion.H) + if (_properties.Version == PpmdVersion.H) { _modelH = new ModelPpm(); if (compress) { throw new NotImplementedException(); } - _modelH.DecodeInit(stream, properties.ModelOrder, properties.AllocatorSize); + _modelH.DecodeInit(stream, _properties.ModelOrder, _properties.AllocatorSize); } - if (properties.Version == PpmdVersion.H7Z) + if (_properties.Version == PpmdVersion.H7Z) { _modelH = new ModelPpm(); if (compress) { throw new NotImplementedException(); } - _modelH.DecodeInit(null, properties.ModelOrder, properties.AllocatorSize); + _modelH.DecodeInit(null, _properties.ModelOrder, _properties.AllocatorSize); _decoder = new Decoder(); _decoder.Init(stream); } } + public static PpmdStream Create(PpmdProperties properties, Stream stream, bool compress) => + new PpmdStream(properties, stream, compress); + + public static async ValueTask CreateAsync( + PpmdProperties properties, + Stream stream, + bool compress, + CancellationToken cancellationToken = default + ) + { + ThrowHelper.ThrowIfNull(stream); + + if (properties.Version == PpmdVersion.H && compress) + { + throw new NotImplementedException("PPMd H version compression not supported"); + } + + if (properties.Version == PpmdVersion.H7Z && compress) + { + throw new NotImplementedException("PPMd H7Z version compression not supported"); + } + + var instance = new PpmdStream(properties, stream, compress, skipInitialization: true); + + try + { + if (properties.Version == PpmdVersion.I1) + { + instance._model = new Model(); + if (compress) + { + instance._model.EncodeStart(properties); + } + else + { + await instance + ._model.DecodeStartAsync(stream, properties, cancellationToken) + .ConfigureAwait(false); + } + } + else if (properties.Version == PpmdVersion.H) + { + instance._modelH = new ModelPpm(); + await instance + ._modelH.DecodeInitAsync( + stream, + properties.ModelOrder, + properties.AllocatorSize, + cancellationToken + ) + .ConfigureAwait(false); + } + else if (properties.Version == PpmdVersion.H7Z) + { + instance._modelH = new ModelPpm(); + await instance + ._modelH.DecodeInitAsync( + null, + properties.ModelOrder, + properties.AllocatorSize, + cancellationToken + ) + .ConfigureAwait(false); + instance._decoder = new Decoder(); + await instance._decoder.InitAsync(stream, cancellationToken).ConfigureAwait(false); + } + + return instance; + } + catch + { +#if LEGACY_DOTNET && !NETSTANDARD2_1 + instance.Dispose(); +#else + await instance.DisposeAsync().ConfigureAwait(false); +#endif + throw; + } + } + public override bool CanRead => !_compress; public override bool CanSeek => false; @@ -67,21 +169,47 @@ public class PpmdStream : Stream public override void Flush() { } - protected override void Dispose(bool isDisposing) + protected override void Dispose(bool disposing) { if (_isDisposed) { return; } _isDisposed = true; - if (isDisposing) + if (disposing) { if (_compress) { - _model.EncodeBlock(_stream, new MemoryStream(), true); + _model.EncodeBlock(_stream, Stream.Null, true); } + _modelH?.Dispose(); + _modelH = null; } - base.Dispose(isDisposing); + base.Dispose(disposing); + } + +#if !LEGACY_DOTNET || NETSTANDARD2_1 + public override async ValueTask DisposeAsync() +#else + public async ValueTask DisposeAsync() +#endif + { + if (_isDisposed) + { + return; + } + + _isDisposed = true; + if (_compress) + { + await _model.EncodeBlockAsync(_stream, new MemoryStream(), true).ConfigureAwait(false); + } + _modelH?.Dispose(); + _modelH = null; + +#if !LEGACY_DOTNET || NETSTANDARD2_1 + await base.DisposeAsync().ConfigureAwait(false); +#endif } public override long Length => throw new NotSupportedException(); @@ -129,6 +257,118 @@ public class PpmdStream : Stream public override void SetLength(long value) => throw new NotSupportedException(); + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + if (_compress) + { + return 0; + } + + cancellationToken.ThrowIfCancellationRequested(); + + var size = 0; + if (_properties.Version == PpmdVersion.I1) + { + size = await _model + .DecodeBlockAsync(_stream, buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + } + if (_properties.Version == PpmdVersion.H) + { + int c; + while ( + size < count + && (c = await _modelH.DecodeCharAsync(cancellationToken).ConfigureAwait(false)) >= 0 + ) + { + buffer[offset++] = (byte)c; + size++; + } + } + if (_properties.Version == PpmdVersion.H7Z) + { + int c; + while ( + size < count + && ( + c = await _modelH + .DecodeCharAsync(_decoder, cancellationToken) + .ConfigureAwait(false) + ) >= 0 + ) + { + buffer[offset++] = (byte)c; + size++; + } + } + _position += size; + return size; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + if (_compress) + { + return 0; + } + + cancellationToken.ThrowIfCancellationRequested(); + + var size = 0; + var offset = 0; + var count = buffer.Length; + + if (_properties.Version == PpmdVersion.I1) + { + // Need to use a temporary buffer since DecodeBlockAsync works with byte[] + var tempBuffer = new byte[count]; + size = await _model + .DecodeBlockAsync(_stream, tempBuffer, 0, count, cancellationToken) + .ConfigureAwait(false); + tempBuffer.AsMemory(0, size).CopyTo(buffer); + } + if (_properties.Version == PpmdVersion.H) + { + int c; + while ( + size < count + && (c = await _modelH.DecodeCharAsync(cancellationToken).ConfigureAwait(false)) >= 0 + ) + { + buffer.Span[offset++] = (byte)c; + size++; + } + } + if (_properties.Version == PpmdVersion.H7Z) + { + int c; + while ( + size < count + && ( + c = await _modelH + .DecodeCharAsync(_decoder, cancellationToken) + .ConfigureAwait(false) + ) >= 0 + ) + { + buffer.Span[offset++] = (byte)c; + size++; + } + } + _position += size; + return size; + } +#endif + public override void Write(byte[] buffer, int offset, int count) { if (_compress) @@ -136,4 +376,46 @@ public class PpmdStream : Stream _model.EncodeBlock(_stream, new MemoryStream(buffer, offset, count), false); } } + + public override async Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + cancellationToken.ThrowIfCancellationRequested(); + if (_compress) + { + await _model + .EncodeBlockAsync( + _stream, + new MemoryStream(buffer, offset, count), + false, + cancellationToken + ) + .ConfigureAwait(false); + } + } + +#if !LEGACY_DOTNET + public override async ValueTask WriteAsync( + ReadOnlyMemory buffer, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + if (_compress) + { + await _model + .EncodeBlockAsync( + _stream, + new MemoryStream(buffer.ToArray()), + false, + cancellationToken + ) + .ConfigureAwait(false); + } + } +#endif } diff --git a/src/SharpCompress/Compressors/PPMd/PpmdVersion.cs b/src/SharpCompress/Compressors/PPMd/PpmdVersion.cs index 791034ac..57626660 100644 --- a/src/SharpCompress/Compressors/PPMd/PpmdVersion.cs +++ b/src/SharpCompress/Compressors/PPMd/PpmdVersion.cs @@ -4,5 +4,5 @@ public enum PpmdVersion { H, H7Z, - I1 + I1, } diff --git a/src/SharpCompress/Compressors/RLE90/RLE.cs b/src/SharpCompress/Compressors/RLE90/RLE.cs new file mode 100644 index 00000000..68b52ea4 --- /dev/null +++ b/src/SharpCompress/Compressors/RLE90/RLE.cs @@ -0,0 +1,51 @@ +using System.Collections.Generic; +using System.Linq; + +namespace SharpCompress.Compressors.RLE90; + +public static class RLE +{ + private const byte DLE = 0x90; + + /// + /// Unpacks an RLE compressed buffer. + /// Format: DLE , where count == 0 -> DLE + /// + /// The compressed buffer to unpack. + /// A list of unpacked bytes. + public static List UnpackRLE(byte[] compressedBuffer) + { + var result = new List(compressedBuffer.Length * 2); // Optimized initial capacity + var countMode = false; + byte last = 0; + + foreach (var c in compressedBuffer) + { + if (!countMode) + { + if (c == DLE) + { + countMode = true; + } + else + { + result.Add(c); + last = c; + } + } + else + { + countMode = false; + if (c == 0) + { + result.Add(DLE); + } + else + { + result.AddRange(Enumerable.Repeat(last, c - 1)); + } + } + } + return result; + } +} diff --git a/src/SharpCompress/Compressors/RLE90/RunLength90Stream.Async.cs b/src/SharpCompress/Compressors/RLE90/RunLength90Stream.Async.cs new file mode 100644 index 00000000..95f82d2c --- /dev/null +++ b/src/SharpCompress/Compressors/RLE90/RunLength90Stream.Async.cs @@ -0,0 +1,114 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Compressors.RLE90; + +public partial class RunLength90Stream +{ + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + ThrowHelper.ThrowIfNull(buffer); + + if (offset < 0 || count < 0 || offset + count > buffer.Length) + { + throw new ArgumentOutOfRangeException(nameof(offset)); + } + + int bytesWritten = 0; + + while (bytesWritten < count && !_endOfCompressedData) + { + // Handle pending repeat bytes first + if (_repeatCount > 0) + { + int toWrite = Math.Min(_repeatCount, count - bytesWritten); + for (int i = 0; i < toWrite; i++) + { + buffer[offset + bytesWritten++] = _lastByte; + } + _repeatCount -= toWrite; + continue; + } + + // Try to read the next byte from compressed data + if (_bytesReadFromSource >= _compressedSize) + { + _endOfCompressedData = true; + break; + } + + byte[] singleByte = new byte[1]; + int bytesRead = await _stream + .ReadAsync(singleByte, 0, 1, cancellationToken) + .ConfigureAwait(false); + if (bytesRead == 0) + { + _endOfCompressedData = true; + break; + } + + _bytesReadFromSource++; + byte c = singleByte[0]; + + if (_inDleMode) + { + _inDleMode = false; + + if (c == 0) + { + buffer[offset + bytesWritten++] = DLE; + _lastByte = DLE; + } + else + { + _repeatCount = c - 1; + // We'll handle these repeats in next loop iteration. + } + } + else if (c == DLE) + { + _inDleMode = true; + } + else + { + buffer[offset + bytesWritten++] = c; + _lastByte = c; + } + } + + return bytesWritten; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + if (buffer.IsEmpty) + { + return 0; + } + + byte[] array = System.Buffers.ArrayPool.Shared.Rent(buffer.Length); + try + { + int read = await ReadAsync(array, 0, buffer.Length, cancellationToken) + .ConfigureAwait(false); + array.AsSpan(0, read).CopyTo(buffer.Span); + return read; + } + finally + { + System.Buffers.ArrayPool.Shared.Return(array); + } + } +#endif +} diff --git a/src/SharpCompress/Compressors/RLE90/RunLength90Stream.cs b/src/SharpCompress/Compressors/RLE90/RunLength90Stream.cs new file mode 100644 index 00000000..d7bffe85 --- /dev/null +++ b/src/SharpCompress/Compressors/RLE90/RunLength90Stream.cs @@ -0,0 +1,124 @@ +using System; +using System.IO; + +namespace SharpCompress.Compressors.RLE90; + +/// +/// Real-time streaming RLE90 decompression stream. +/// Decompresses bytes on demand without buffering the entire file in memory. +/// +public partial class RunLength90Stream : Stream +{ + private readonly Stream _stream; + private readonly int _compressedSize; + private int _bytesReadFromSource; + + private const byte DLE = 0x90; + private bool _inDleMode; + private byte _lastByte; + private int _repeatCount; + + private bool _endOfCompressedData; + + public RunLength90Stream(Stream stream, int compressedSize) + { + _stream = stream ?? throw new ArgumentNullException(nameof(stream)); + _compressedSize = compressedSize; + } + + protected override void Dispose(bool disposing) + { + base.Dispose(disposing); + } + + public override bool CanRead => true; + public override bool CanSeek => false; + public override bool CanWrite => false; + + public override long Length => throw new NotSupportedException(); + public override long Position + { + get => throw new NotSupportedException(); + set => throw new NotSupportedException(); + } + + public override void Flush() => throw new NotSupportedException(); + + public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); + + public override void SetLength(long value) => throw new NotSupportedException(); + + public override void Write(byte[] buffer, int offset, int count) => + throw new NotSupportedException(); + + public override int Read(byte[] buffer, int offset, int count) + { + ThrowHelper.ThrowIfNull(buffer); + + if (offset < 0 || count < 0 || offset + count > buffer.Length) + { + throw new ArgumentOutOfRangeException(nameof(offset)); + } + + int bytesWritten = 0; + + while (bytesWritten < count && !_endOfCompressedData) + { + // Handle pending repeat bytes first + if (_repeatCount > 0) + { + int toWrite = Math.Min(_repeatCount, count - bytesWritten); + for (int i = 0; i < toWrite; i++) + { + buffer[offset + bytesWritten++] = _lastByte; + } + _repeatCount -= toWrite; + continue; + } + + // Try to read the next byte from compressed data + if (_bytesReadFromSource >= _compressedSize) + { + _endOfCompressedData = true; + break; + } + + int next = _stream.ReadByte(); + if (next == -1) + { + _endOfCompressedData = true; + break; + } + + _bytesReadFromSource++; + byte c = (byte)next; + + if (_inDleMode) + { + _inDleMode = false; + + if (c == 0) + { + buffer[offset + bytesWritten++] = DLE; + _lastByte = DLE; + } + else + { + _repeatCount = c - 1; + // We’ll handle these repeats in next loop iteration. + } + } + else if (c == DLE) + { + _inDleMode = true; + } + else + { + buffer[offset + bytesWritten++] = c; + _lastByte = c; + } + } + + return bytesWritten; + } +} diff --git a/src/SharpCompress/Compressors/Rar/IRarUnpack.cs b/src/SharpCompress/Compressors/Rar/IRarUnpack.cs index 651e2c8a..657393ab 100644 --- a/src/SharpCompress/Compressors/Rar/IRarUnpack.cs +++ b/src/SharpCompress/Compressors/Rar/IRarUnpack.cs @@ -1,4 +1,6 @@ using System.IO; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common.Rar.Headers; namespace SharpCompress.Compressors.Rar; @@ -8,10 +10,19 @@ internal interface IRarUnpack void DoUnpack(FileHeader fileHeader, Stream readStream, Stream writeStream); void DoUnpack(); + ValueTask DoUnpackAsync( + FileHeader fileHeader, + Stream readStream, + Stream writeStream, + CancellationToken cancellationToken + ); + ValueTask DoUnpackAsync(CancellationToken cancellationToken); + // eg u/i pause/resume button bool Suspended { get; set; } long DestSize { get; } - int Char { get; } + int ReadChar(); + ValueTask ReadCharAsync(CancellationToken cancellationToken); int PpmEscChar { get; set; } } diff --git a/src/SharpCompress/Compressors/Rar/MultiVolumeReadOnlyAsyncStream.Async.cs b/src/SharpCompress/Compressors/Rar/MultiVolumeReadOnlyAsyncStream.Async.cs new file mode 100644 index 00000000..80a93308 --- /dev/null +++ b/src/SharpCompress/Compressors/Rar/MultiVolumeReadOnlyAsyncStream.Async.cs @@ -0,0 +1,168 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Rar; + +namespace SharpCompress.Compressors.Rar; + +internal sealed partial class MultiVolumeReadOnlyAsyncStream : MultiVolumeReadOnlyStreamBase +{ + internal static async ValueTask Create( + IAsyncEnumerable parts + ) + { + var stream = new MultiVolumeReadOnlyAsyncStream(parts); + await stream.filePartEnumerator.MoveNextAsync().ConfigureAwait(false); + stream.InitializeNextFilePart(); + return stream; + } + +#if NET8_0_OR_GREATER + public override async ValueTask DisposeAsync() + { + await base.DisposeAsync().ConfigureAwait(false); + if (filePartEnumerator != null) + { + await filePartEnumerator.DisposeAsync().ConfigureAwait(false); + } + currentStream = null; + } +#else + protected override void Dispose(bool disposing) + { + base.Dispose(disposing); + //acceptable for now? +#pragma warning disable VSTHRD002 // Avoid problematic synchronous waits + filePartEnumerator.DisposeAsync().AsTask().GetAwaiter().GetResult(); +#pragma warning restore VSTHRD002 // Avoid problematic synchronous waits + + currentStream = null; + } +#endif + + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + var totalRead = 0; + var currentOffset = offset; + var currentCount = count; + while (currentCount > 0) + { + var readSize = currentCount; + if (currentCount > maxPosition - currentPosition) + { + readSize = (int)(maxPosition - currentPosition); + } + + var read = await currentStream + .NotNull() + .ReadAsync(buffer, currentOffset, readSize, cancellationToken) + .ConfigureAwait(false); + if (read < 0) + { + throw new IncompleteArchiveException("Unexpected end of stream."); + } + + currentPosition += read; + currentOffset += read; + currentCount -= read; + totalRead += read; + if ( + ((maxPosition - currentPosition) == 0) + && filePartEnumerator.Current.FileHeader.IsSplitAfter + ) + { + if (filePartEnumerator.Current.FileHeader.R4Salt != null) + { + throw new InvalidFormatException( + "Sharpcompress currently does not support multi-volume decryption." + ); + } + + var fileName = filePartEnumerator.Current.FileHeader.FileName; + if (!await filePartEnumerator.MoveNextAsync().ConfigureAwait(false)) + { + throw new InvalidFormatException( + "Multi-part rar file is incomplete. Entry expects a new volume: " + + fileName + ); + } + + InitializeNextFilePart(); + } + else + { + break; + } + } + + return totalRead; + } + +#if NETCOREAPP2_1_OR_GREATER || NETSTANDARD2_1_OR_GREATER + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + var totalRead = 0; + var currentOffset = 0; + var currentCount = buffer.Length; + while (currentCount > 0) + { + var readSize = currentCount; + if (currentCount > maxPosition - currentPosition) + { + readSize = (int)(maxPosition - currentPosition); + } + + var read = await currentStream + .NotNull() + .ReadAsync(buffer.Slice(currentOffset, readSize), cancellationToken) + .ConfigureAwait(false); + if (read < 0) + { + throw new IncompleteArchiveException("Unexpected end of stream."); + } + + currentPosition += read; + currentOffset += read; + currentCount -= read; + totalRead += read; + if ( + ((maxPosition - currentPosition) == 0) + && filePartEnumerator.Current.FileHeader.IsSplitAfter + ) + { + if (filePartEnumerator.Current.FileHeader.R4Salt != null) + { + throw new InvalidFormatException( + "Sharpcompress currently does not support multi-volume decryption." + ); + } + var fileName = filePartEnumerator.Current.FileHeader.FileName; + if (!await filePartEnumerator.MoveNextAsync().ConfigureAwait(false)) + { + throw new InvalidFormatException( + "Multi-part rar file is incomplete. Entry expects a new volume: " + + fileName + ); + } + InitializeNextFilePart(); + } + else + { + break; + } + } + return totalRead; + } +#endif +} diff --git a/src/SharpCompress/Compressors/Rar/MultiVolumeReadOnlyAsyncStream.cs b/src/SharpCompress/Compressors/Rar/MultiVolumeReadOnlyAsyncStream.cs new file mode 100644 index 00000000..6f4c8b88 --- /dev/null +++ b/src/SharpCompress/Compressors/Rar/MultiVolumeReadOnlyAsyncStream.cs @@ -0,0 +1,61 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Rar; + +namespace SharpCompress.Compressors.Rar; + +internal sealed partial class MultiVolumeReadOnlyAsyncStream : MultiVolumeReadOnlyStreamBase +{ + private long currentPosition; + private long maxPosition; + + private IAsyncEnumerator filePartEnumerator; + private Stream? currentStream; + + private MultiVolumeReadOnlyAsyncStream(IAsyncEnumerable parts) + { + filePartEnumerator = parts.GetAsyncEnumerator(); + } + + // Async methods moved to MultiVolumeReadOnlyAsyncStream.Async.cs + + private void InitializeNextFilePart() + { + maxPosition = filePartEnumerator.Current.FileHeader.CompressedSize; + currentPosition = 0; + currentStream = filePartEnumerator.Current.GetCompressedStream(); + + CurrentCrc = filePartEnumerator.Current.FileHeader.FileCrc; + } + + public override int Read(byte[] buffer, int offset, int count) => + throw new NotSupportedException( + "Synchronous read is not supported in MultiVolumeReadOnlyAsyncStream." + ); + + public override bool CanRead => true; + + public override bool CanSeek => false; + + public override bool CanWrite => false; + + public override void Flush() { } + + public override long Length => throw new NotSupportedException(); + + public override long Position + { + get => throw new NotSupportedException(); + set => throw new NotSupportedException(); + } + + public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); + + public override void SetLength(long value) => throw new NotSupportedException(); + + public override void Write(byte[] buffer, int offset, int count) => + throw new NotSupportedException(); +} diff --git a/src/SharpCompress/Compressors/Rar/MultiVolumeReadOnlyStream.Async.cs b/src/SharpCompress/Compressors/Rar/MultiVolumeReadOnlyStream.Async.cs new file mode 100644 index 00000000..51bd2032 --- /dev/null +++ b/src/SharpCompress/Compressors/Rar/MultiVolumeReadOnlyStream.Async.cs @@ -0,0 +1,135 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Rar; + +namespace SharpCompress.Compressors.Rar; + +internal sealed partial class MultiVolumeReadOnlyStream : MultiVolumeReadOnlyStreamBase +{ + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + var totalRead = 0; + var currentOffset = offset; + var currentCount = count; + while (currentCount > 0) + { + var readSize = currentCount; + if (currentCount > maxPosition - currentPosition) + { + readSize = (int)(maxPosition - currentPosition); + } + + var read = await currentStream + .NotNull() + .ReadAsync(buffer, currentOffset, readSize, cancellationToken) + .ConfigureAwait(false); + if (read < 0) + { + throw new IncompleteArchiveException("Unexpected end of stream."); + } + + currentPosition += read; + currentOffset += read; + currentCount -= read; + totalRead += read; + if ( + ((maxPosition - currentPosition) == 0) + && filePartEnumerator.Current.FileHeader.IsSplitAfter + ) + { + if (filePartEnumerator.Current.FileHeader.R4Salt != null) + { + throw new InvalidFormatException( + "Sharpcompress currently does not support multi-volume decryption." + ); + } + + var fileName = filePartEnumerator.Current.FileHeader.FileName; + if (!filePartEnumerator.MoveNext()) + { + throw new InvalidFormatException( + "Multi-part rar file is incomplete. Entry expects a new volume: " + + fileName + ); + } + + InitializeNextFilePart(); + } + else + { + break; + } + } + + return totalRead; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + var totalRead = 0; + var currentOffset = 0; + var currentCount = buffer.Length; + while (currentCount > 0) + { + var readSize = currentCount; + if (currentCount > maxPosition - currentPosition) + { + readSize = (int)(maxPosition - currentPosition); + } + + var read = await currentStream + .NotNull() + .ReadAsync(buffer.Slice(currentOffset, readSize), cancellationToken) + .ConfigureAwait(false); + if (read < 0) + { + throw new IncompleteArchiveException("Unexpected end of stream."); + } + + currentPosition += read; + currentOffset += read; + currentCount -= read; + totalRead += read; + if ( + ((maxPosition - currentPosition) == 0) + && filePartEnumerator.Current.FileHeader.IsSplitAfter + ) + { + if (filePartEnumerator.Current.FileHeader.R4Salt != null) + { + throw new InvalidFormatException( + "Sharpcompress currently does not support multi-volume decryption." + ); + } + var fileName = filePartEnumerator.Current.FileHeader.FileName; + if (!filePartEnumerator.MoveNext()) + { + throw new InvalidFormatException( + "Multi-part rar file is incomplete. Entry expects a new volume: " + + fileName + ); + } + InitializeNextFilePart(); + } + else + { + break; + } + } + return totalRead; + } +#endif +} diff --git a/src/SharpCompress/Compressors/Rar/MultiVolumeReadOnlyStream.cs b/src/SharpCompress/Compressors/Rar/MultiVolumeReadOnlyStream.cs index 1e4adfde..6e8f732c 100644 --- a/src/SharpCompress/Compressors/Rar/MultiVolumeReadOnlyStream.cs +++ b/src/SharpCompress/Compressors/Rar/MultiVolumeReadOnlyStream.cs @@ -1,5 +1,3 @@ -#nullable disable - using System; using System.Collections.Generic; using System.IO; @@ -8,26 +6,16 @@ using SharpCompress.Common.Rar; namespace SharpCompress.Compressors.Rar; -internal sealed class MultiVolumeReadOnlyStream : Stream +internal sealed partial class MultiVolumeReadOnlyStream : MultiVolumeReadOnlyStreamBase { private long currentPosition; private long maxPosition; private IEnumerator filePartEnumerator; - private Stream currentStream; + private Stream? currentStream; - private readonly IExtractionListener streamListener; - - private long currentPartTotalReadBytes; - private long currentEntryTotalReadBytes; - - internal MultiVolumeReadOnlyStream( - IEnumerable parts, - IExtractionListener streamListener - ) + internal MultiVolumeReadOnlyStream(IEnumerable parts) { - this.streamListener = streamListener; - filePartEnumerator = parts.GetEnumerator(); filePartEnumerator.MoveNext(); InitializeNextFilePart(); @@ -38,11 +26,8 @@ internal sealed class MultiVolumeReadOnlyStream : Stream base.Dispose(disposing); if (disposing) { - if (filePartEnumerator != null) - { - filePartEnumerator.Dispose(); - filePartEnumerator = null; - } + filePartEnumerator.Dispose(); + currentStream = null; } } @@ -53,15 +38,7 @@ internal sealed class MultiVolumeReadOnlyStream : Stream currentPosition = 0; currentStream = filePartEnumerator.Current.GetCompressedStream(); - currentPartTotalReadBytes = 0; - CurrentCrc = filePartEnumerator.Current.FileHeader.FileCrc; - - streamListener.FireFilePartExtractionBegin( - filePartEnumerator.Current.FilePartName, - filePartEnumerator.Current.FileHeader.CompressedSize, - filePartEnumerator.Current.FileHeader.UncompressedSize - ); } public override int Read(byte[] buffer, int offset, int count) @@ -77,10 +54,10 @@ internal sealed class MultiVolumeReadOnlyStream : Stream readSize = (int)(maxPosition - currentPosition); } - var read = currentStream.Read(buffer, currentOffset, readSize); + var read = currentStream.NotNull().Read(buffer, currentOffset, readSize); if (read < 0) { - throw new EndOfStreamException(); + throw new IncompleteArchiveException("Unexpected end of stream."); } currentPosition += read; @@ -98,6 +75,7 @@ internal sealed class MultiVolumeReadOnlyStream : Stream "Sharpcompress currently does not support multi-volume decryption." ); } + var fileName = filePartEnumerator.Current.FileHeader.FileName; if (!filePartEnumerator.MoveNext()) { @@ -106,6 +84,7 @@ internal sealed class MultiVolumeReadOnlyStream : Stream + fileName ); } + InitializeNextFilePart(); } else @@ -113,12 +92,7 @@ internal sealed class MultiVolumeReadOnlyStream : Stream break; } } - currentPartTotalReadBytes += totalRead; - currentEntryTotalReadBytes += totalRead; - streamListener.FireCompressedBytesRead( - currentPartTotalReadBytes, - currentEntryTotalReadBytes - ); + return totalRead; } @@ -128,9 +102,7 @@ internal sealed class MultiVolumeReadOnlyStream : Stream public override bool CanWrite => false; - public uint CurrentCrc { get; private set; } - - public override void Flush() => throw new NotSupportedException(); + public override void Flush() { } public override long Length => throw new NotSupportedException(); diff --git a/src/SharpCompress/Compressors/Rar/MultiVolumeReadOnlyStreamBase.cs b/src/SharpCompress/Compressors/Rar/MultiVolumeReadOnlyStreamBase.cs new file mode 100644 index 00000000..d2dc9f38 --- /dev/null +++ b/src/SharpCompress/Compressors/Rar/MultiVolumeReadOnlyStreamBase.cs @@ -0,0 +1,8 @@ +using System.IO; + +namespace SharpCompress.Compressors.Rar; + +internal abstract class MultiVolumeReadOnlyStreamBase : Stream +{ + public byte[]? CurrentCrc { get; protected set; } +} diff --git a/src/SharpCompress/Compressors/Rar/RarBLAKE2spStream.Async.cs b/src/SharpCompress/Compressors/Rar/RarBLAKE2spStream.Async.cs new file mode 100644 index 00000000..53560283 --- /dev/null +++ b/src/SharpCompress/Compressors/Rar/RarBLAKE2spStream.Async.cs @@ -0,0 +1,76 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Rar.Headers; + +namespace SharpCompress.Compressors.Rar; + +internal partial class RarBLAKE2spStream : RarStream +{ + public static ValueTask CreateAsync( + IRarUnpack unpack, + FileHeader fileHeader, + MultiVolumeReadOnlyAsyncStream readStream, + CancellationToken cancellationToken = default + ) + { + var stream = new RarBLAKE2spStream(unpack, fileHeader, readStream); + return new ValueTask(stream); + } + + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + var result = await base.ReadAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + if (result != 0) + { + Update(_blake2sp!, new ReadOnlySpan(buffer, offset, result)); + } + else + { + EnsureHash(); + if (!disableCRCCheck && !GetCrc().SequenceEqual(readStream.CurrentCrc) && count != 0) + { + // NOTE: we use the last FileHeader in a multipart volume to check CRC + throw new InvalidFormatException("file crc mismatch"); + } + } + + return result; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + var result = await base.ReadAsync(buffer, cancellationToken).ConfigureAwait(false); + if (result != 0) + { + Update(_blake2sp!, buffer.Span.Slice(0, result)); + } + else + { + EnsureHash(); + if ( + !disableCRCCheck + && !GetCrc().SequenceEqual(readStream.CurrentCrc) + && buffer.Length != 0 + ) + { + // NOTE: we use the last FileHeader in a multipart volume to check CRC + throw new InvalidFormatException("file crc mismatch"); + } + } + + return result; + } +#endif +} diff --git a/src/SharpCompress/Compressors/Rar/RarBLAKE2spStream.cs b/src/SharpCompress/Compressors/Rar/RarBLAKE2spStream.cs new file mode 100644 index 00000000..54ecda29 --- /dev/null +++ b/src/SharpCompress/Compressors/Rar/RarBLAKE2spStream.cs @@ -0,0 +1,363 @@ +using System; +using System.Runtime.InteropServices; +using SharpCompress.Common; +using SharpCompress.Common.Rar.Headers; + +namespace SharpCompress.Compressors.Rar; + +internal partial class RarBLAKE2spStream : RarStream +{ + private readonly MultiVolumeReadOnlyStreamBase readStream; + private readonly bool disableCRCCheck; + + private const int BLAKE2S_NUM_ROUNDS = 10; + private const uint BLAKE2S_FINAL_FLAG = ~(uint)0; + private const int BLAKE2S_BLOCK_SIZE = 64; + private const int BLAKE2S_DIGEST_SIZE = 32; + private const int BLAKE2SP_PARALLEL_DEGREE = 8; + private const int BLAKE2S_INIT_IV_SIZE = 8; + + private static readonly uint[] k_BLAKE2S_IV = + { + 0x6A09E667U, + 0xBB67AE85U, + 0x3C6EF372U, + 0xA54FF53AU, + 0x510E527FU, + 0x9B05688CU, + 0x1F83D9ABU, + 0x5BE0CD19U, + }; + + private static readonly byte[][] k_BLAKE2S_Sigma = + { + new byte[] { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 }, + new byte[] { 14, 10, 4, 8, 9, 15, 13, 6, 1, 12, 0, 2, 11, 7, 5, 3 }, + new byte[] { 11, 8, 12, 0, 5, 2, 15, 13, 10, 14, 3, 6, 7, 1, 9, 4 }, + new byte[] { 7, 9, 3, 1, 13, 12, 11, 14, 2, 6, 5, 10, 4, 0, 15, 8 }, + new byte[] { 9, 0, 5, 7, 2, 4, 10, 15, 14, 1, 11, 12, 6, 8, 3, 13 }, + new byte[] { 2, 12, 6, 10, 0, 11, 8, 3, 4, 13, 7, 5, 15, 14, 1, 9 }, + new byte[] { 12, 5, 1, 15, 14, 13, 4, 10, 0, 7, 6, 3, 9, 2, 8, 11 }, + new byte[] { 13, 11, 7, 14, 12, 1, 3, 9, 5, 0, 15, 4, 8, 6, 2, 10 }, + new byte[] { 6, 15, 14, 9, 11, 3, 0, 8, 12, 2, 13, 7, 1, 4, 10, 5 }, + new byte[] { 10, 2, 8, 4, 7, 6, 1, 5, 15, 11, 9, 14, 3, 12, 13, 0 }, + }; + + private sealed class BLAKE2S + { + internal readonly uint[] h; + internal readonly uint[] t; + internal readonly uint[] f; + internal readonly byte[] b; + internal int bufferPosition; + internal uint lastNodeFlag; + + public BLAKE2S() + { + h = new uint[BLAKE2S_INIT_IV_SIZE]; + t = new uint[2]; + f = new uint[2]; + b = new byte[BLAKE2S_BLOCK_SIZE]; + } + }; + + private sealed class BLAKE2SP + { + internal readonly BLAKE2S S0 = new(); + internal readonly BLAKE2S S1 = new(); + internal readonly BLAKE2S S2 = new(); + internal readonly BLAKE2S S3 = new(); + internal readonly BLAKE2S S4 = new(); + internal readonly BLAKE2S S5 = new(); + internal readonly BLAKE2S S6 = new(); + internal readonly BLAKE2S S7 = new(); + internal readonly BLAKE2S Root; + internal int bufferPosition; + + public BLAKE2SP() => Root = new BLAKE2S(); + + internal BLAKE2S GetLeaf(int index) + { + switch (index) + { + case 0: + return S0; + case 1: + return S1; + case 2: + return S2; + case 3: + return S3; + case 4: + return S4; + case 5: + return S5; + case 6: + return S6; + default: + return S7; + } + } + }; + + private BLAKE2SP? _blake2sp; + private byte[]? _hash; + + private RarBLAKE2spStream( + IRarUnpack unpack, + FileHeader fileHeader, + MultiVolumeReadOnlyStreamBase readStream + ) + : base(unpack, fileHeader, readStream) + { + this.readStream = readStream; + + // TODO: rar uses a modified hash xor'ed with encryption key? + disableCRCCheck = fileHeader.IsEncrypted; + this._blake2sp = CreateBlake2sp(); + } + + public static RarBLAKE2spStream Create( + IRarUnpack unpack, + FileHeader fileHeader, + MultiVolumeReadOnlyStream readStream + ) + { + var stream = new RarBLAKE2spStream(unpack, fileHeader, readStream); + return stream; + } + + // Async methods moved to RarBLAKE2spStream.Async.cs + + public byte[] GetCrc() => + this._hash + ?? throw new InvalidOperationException( + "hash not computed, has the stream been fully drained?" + ); + + private static void ResetCrc(BLAKE2S hash) + { + k_BLAKE2S_IV.AsSpan().CopyTo(hash.h); + hash.t[0] = 0; + hash.t[1] = 0; + hash.f[0] = 0; + hash.f[1] = 0; + hash.bufferPosition = 0; + hash.lastNodeFlag = 0; + } + + private static void G( + Span m, + byte[] sigma, + int i, + ref uint a, + ref uint b, + ref uint c, + ref uint d + ) + { + a += b + m[sigma[2 * i]]; + d ^= a; + d = (d >> 16) | (d << 16); + c += d; + b ^= c; + b = (b >> 12) | (b << 20); + + a += b + m[sigma[2 * i + 1]]; + d ^= a; + d = (d >> 8) | (d << 24); + c += d; + b ^= c; + b = (b >> 7) | (b << 25); + } + + private static void Compress(BLAKE2S hash) + { + Span m = stackalloc uint[16]; + if (BitConverter.IsLittleEndian) + { + MemoryMarshal.Cast(hash.b).CopyTo(m); + } + else + { + for (var i = 0; i < 16; i++) + { + m[i] = BitConverter.ToUInt32(hash.b, i * 4); + } + } + + Span v = stackalloc uint[16]; + for (var i = 0; i < 8; i++) + { + v[i] = hash.h[i]; + } + + v[8] = k_BLAKE2S_IV[0]; + v[9] = k_BLAKE2S_IV[1]; + v[10] = k_BLAKE2S_IV[2]; + v[11] = k_BLAKE2S_IV[3]; + + v[12] = hash.t[0] ^ k_BLAKE2S_IV[4]; + v[13] = hash.t[1] ^ k_BLAKE2S_IV[5]; + v[14] = hash.f[0] ^ k_BLAKE2S_IV[6]; + v[15] = hash.f[1] ^ k_BLAKE2S_IV[7]; + + for (var r = 0; r < BLAKE2S_NUM_ROUNDS; r++) + { + var sigma = k_BLAKE2S_Sigma[r]; + G(m, sigma, 0, ref v[0], ref v[4], ref v[8], ref v[12]); + G(m, sigma, 1, ref v[1], ref v[5], ref v[9], ref v[13]); + G(m, sigma, 2, ref v[2], ref v[6], ref v[10], ref v[14]); + G(m, sigma, 3, ref v[3], ref v[7], ref v[11], ref v[15]); + G(m, sigma, 4, ref v[0], ref v[5], ref v[10], ref v[15]); + G(m, sigma, 5, ref v[1], ref v[6], ref v[11], ref v[12]); + G(m, sigma, 6, ref v[2], ref v[7], ref v[8], ref v[13]); + G(m, sigma, 7, ref v[3], ref v[4], ref v[9], ref v[14]); + } + + for (var i = 0; i < 8; i++) + { + hash.h[i] ^= v[i] ^ v[i + 8]; + } + } + + private static void Update(BLAKE2S hash, ReadOnlySpan data) + { + while (data.Length != 0) + { + var pos = hash.bufferPosition; + var chunkSize = BLAKE2S_BLOCK_SIZE - pos; + if (data.Length <= chunkSize) + { + data.CopyTo(hash.b.AsSpan(pos)); + hash.bufferPosition += data.Length; + return; + } + data.Slice(0, chunkSize).CopyTo(hash.b.AsSpan(pos)); + hash.t[0] += BLAKE2S_BLOCK_SIZE; + hash.t[1] += hash.t[0] < BLAKE2S_BLOCK_SIZE ? 1U : 0U; + Compress(hash); + hash.bufferPosition = 0; + data = data.Slice(chunkSize); + } + } + + private static void Final(BLAKE2S hash, Span output) + { + hash.t[0] += (uint)hash.bufferPosition; + hash.t[1] += hash.t[0] < hash.bufferPosition ? 1U : 0U; + hash.f[0] = BLAKE2S_FINAL_FLAG; + hash.f[1] = hash.lastNodeFlag; + hash.b.AsSpan(hash.bufferPosition).Clear(); + Compress(hash); + + if (BitConverter.IsLittleEndian) + { + MemoryMarshal.Cast(hash.h).CopyTo(output); + } + else + { + for (var i = 0; i < 8; i++) + { + var v = hash.h[i]; + output[i * 4] = (byte)v; + output[i * 4 + 1] = (byte)(v >> 8); + output[i * 4 + 2] = (byte)(v >> 16); + output[i * 4 + 3] = (byte)(v >> 24); + } + } + } + + private static BLAKE2SP CreateBlake2sp() + { + var blake2sp = new BLAKE2SP(); + + for (var i = 0; i < BLAKE2SP_PARALLEL_DEGREE; i++) + { + var blake2S = blake2sp.GetLeaf(i); + ResetCrc(blake2S); + + var h = blake2S.h; + // word[0]: digest_length | (fanout<<16) | (depth<<24) + h[0] ^= BLAKE2S_DIGEST_SIZE | (BLAKE2SP_PARALLEL_DEGREE << 16) | (2 << 24); + // word[2]: node_offset = leaf index + h[2] ^= (uint)i; + // word[3]: inner_length in bits 24-31 + h[3] ^= BLAKE2S_DIGEST_SIZE << 24; + } + + blake2sp.GetLeaf(BLAKE2SP_PARALLEL_DEGREE - 1).lastNodeFlag = BLAKE2S_FINAL_FLAG; + return blake2sp; + } + + private static void Update(BLAKE2SP hash, ReadOnlySpan data) + { + var pos = hash.bufferPosition; + while (data.Length != 0) + { + var index = pos / BLAKE2S_BLOCK_SIZE; + var chunkSize = BLAKE2S_BLOCK_SIZE - (pos & (BLAKE2S_BLOCK_SIZE - 1)); + if (chunkSize > data.Length) + { + chunkSize = data.Length; + } + Update(hash.GetLeaf(index), data.Slice(0, chunkSize)); + data = data.Slice(chunkSize); + pos = (pos + chunkSize) & (BLAKE2S_BLOCK_SIZE * BLAKE2SP_PARALLEL_DEGREE - 1); + } + hash.bufferPosition = pos; + } + + private static byte[] Final(BLAKE2SP blake2sp) + { + var blake2s = blake2sp.Root; + ResetCrc(blake2s); + + var h = blake2s.h; + // word[0]: digest_length | (fanout<<16) | (depth<<24) — same as leaves + h[0] ^= BLAKE2S_DIGEST_SIZE | (BLAKE2SP_PARALLEL_DEGREE << 16) | (2 << 24); + // word[3]: node_depth=1 (bits 16-23), inner_length=32 (bits 24-31) + h[3] ^= (1 << 16) | (BLAKE2S_DIGEST_SIZE << 24); + blake2s.lastNodeFlag = BLAKE2S_FINAL_FLAG; + + Span digest = stackalloc byte[BLAKE2S_DIGEST_SIZE]; + for (var i = 0; i < BLAKE2SP_PARALLEL_DEGREE; i++) + { + Final(blake2sp.GetLeaf(i), digest); + Update(blake2s, digest); + } + + Final(blake2s, digest); + return digest.ToArray(); + } + + private void EnsureHash() + { + if (this._hash == null) + { + this._hash = Final(this._blake2sp!); + // prevent incorrect usage past hash finality by failing fast + this._blake2sp = null; + } + } + + public override int Read(byte[] buffer, int offset, int count) + { + var result = base.Read(buffer, offset, count); + if (result != 0) + { + Update(this._blake2sp!, new ReadOnlySpan(buffer, offset, result)); + } + else + { + EnsureHash(); + if (!disableCRCCheck && !GetCrc().SequenceEqual(readStream.CurrentCrc) && count != 0) + { + // NOTE: we use the last FileHeader in a multipart volume to check CRC + throw new InvalidFormatException("file crc mismatch"); + } + } + + return result; + } +} diff --git a/src/SharpCompress/Compressors/Rar/RarCRC.cs b/src/SharpCompress/Compressors/Rar/RarCRC.cs index 8f4fb47e..7e6fc6cc 100644 --- a/src/SharpCompress/Compressors/Rar/RarCRC.cs +++ b/src/SharpCompress/Compressors/Rar/RarCRC.cs @@ -9,7 +9,7 @@ internal static class RarCRC public static uint CheckCrc(uint startCrc, byte b) => (crcTab[((int)startCrc ^ b) & 0xff] ^ (startCrc >> 8)); - public static uint CheckCrc(uint startCrc, byte[] data, int offset, int count) + public static uint CheckCrc(uint startCrc, ReadOnlySpan data, int offset, int count) { var size = Math.Min(data.Length - offset, count); diff --git a/src/SharpCompress/Compressors/Rar/RarCrcStream.Async.cs b/src/SharpCompress/Compressors/Rar/RarCrcStream.Async.cs new file mode 100644 index 00000000..2aedc9a6 --- /dev/null +++ b/src/SharpCompress/Compressors/Rar/RarCrcStream.Async.cs @@ -0,0 +1,74 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Rar.Headers; + +namespace SharpCompress.Compressors.Rar; + +internal partial class RarCrcStream : RarStream +{ + public static ValueTask CreateAsync( + IRarUnpack unpack, + FileHeader fileHeader, + MultiVolumeReadOnlyStreamBase readStream, + CancellationToken cancellationToken = default + ) + { + var stream = new RarCrcStream(unpack, fileHeader, readStream); + return new ValueTask(stream); + } + + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + var result = await base.ReadAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + if (result != 0) + { + currentCrc = RarCRC.CheckCrc(currentCrc, buffer, offset, result); + } + else if ( + !disableCRC + && GetCrc() != BitConverter.ToUInt32(readStream.NotNull().CurrentCrc.NotNull(), 0) + && count != 0 + ) + { + // NOTE: we use the last FileHeader in a multipart volume to check CRC + throw new InvalidFormatException("file crc mismatch"); + } + + return result; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + var result = await base.ReadAsync(buffer, cancellationToken).ConfigureAwait(false); + if (result != 0) + { + currentCrc = RarCRC.CheckCrc(currentCrc, buffer.Span, 0, result); + } + else if ( + !disableCRC + && GetCrc() != BitConverter.ToUInt32(readStream.NotNull().CurrentCrc.NotNull(), 0) + && buffer.Length != 0 + ) + { + // NOTE: we use the last FileHeader in a multipart volume to check CRC + throw new InvalidFormatException("file crc mismatch"); + } + + return result; + } +#endif +} diff --git a/src/SharpCompress/Compressors/Rar/RarCrcStream.cs b/src/SharpCompress/Compressors/Rar/RarCrcStream.cs index 98d50b00..a9025e19 100644 --- a/src/SharpCompress/Compressors/Rar/RarCrcStream.cs +++ b/src/SharpCompress/Compressors/Rar/RarCrcStream.cs @@ -1,24 +1,47 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common; using SharpCompress.Common.Rar.Headers; namespace SharpCompress.Compressors.Rar; -internal class RarCrcStream : RarStream +internal partial class RarCrcStream : RarStream { - private readonly MultiVolumeReadOnlyStream readStream; + private readonly MultiVolumeReadOnlyStreamBase readStream; private uint currentCrc; + private readonly bool disableCRC; - public RarCrcStream( + private RarCrcStream( IRarUnpack unpack, FileHeader fileHeader, - MultiVolumeReadOnlyStream readStream + MultiVolumeReadOnlyStreamBase readStream ) : base(unpack, fileHeader, readStream) { this.readStream = readStream; + disableCRC = fileHeader.IsEncrypted; ResetCrc(); } + public static RarCrcStream Create( + IRarUnpack unpack, + FileHeader fileHeader, + MultiVolumeReadOnlyStream readStream + ) + { + var stream = new RarCrcStream(unpack, fileHeader, readStream); + return stream; + } + + // Async methods moved to RarCrcStream.Async.cs + + protected override void Dispose(bool disposing) + { + base.Dispose(disposing); + } + public uint GetCrc() => ~currentCrc; public void ResetCrc() => currentCrc = 0xffffffff; @@ -30,7 +53,11 @@ internal class RarCrcStream : RarStream { currentCrc = RarCRC.CheckCrc(currentCrc, buffer, offset, result); } - else if (GetCrc() != readStream.CurrentCrc && count != 0) + else if ( + !disableCRC + && GetCrc() != BitConverter.ToUInt32(readStream.NotNull().CurrentCrc.NotNull(), 0) + && count != 0 + ) { // NOTE: we use the last FileHeader in a multipart volume to check CRC throw new InvalidFormatException("file crc mismatch"); diff --git a/src/SharpCompress/Compressors/Rar/RarStream.Async.cs b/src/SharpCompress/Compressors/Rar/RarStream.Async.cs new file mode 100644 index 00000000..432bd7c5 --- /dev/null +++ b/src/SharpCompress/Compressors/Rar/RarStream.Async.cs @@ -0,0 +1,143 @@ +#nullable disable + +using System; +using System.Buffers; +using System.IO; +using System.Runtime.InteropServices; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Rar.Headers; +using SharpCompress.IO; + +namespace SharpCompress.Compressors.Rar; + +internal partial class RarStream +{ + /// + /// Asynchronously initializes the RAR stream for reading. + /// + public async ValueTask InitializeAsync(CancellationToken cancellationToken = default) + { + if (initialized) + { + return; + } + + fetch = true; + await unpack + .DoUnpackAsync(fileHeader, readStream, this, cancellationToken) + .ConfigureAwait(false); + fetch = false; + initialized = true; + _position = 0; + } + + /// + /// Asynchronously reads bytes from the current stream into a buffer. + /// + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) => await ReadImplAsync(buffer, offset, count, cancellationToken).ConfigureAwait(false); + + /// + /// Internal async implementation of ReadAsync. + /// + private async ValueTask ReadImplAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + if (count == 0) + { + return 0; + } + + await InitializeAsync(cancellationToken).ConfigureAwait(false); + outTotal = 0; + if (tmpCount > 0) + { + var toCopy = tmpCount < count ? tmpCount : count; + Buffer.BlockCopy(tmpBuffer, tmpOffset, buffer, offset, toCopy); + tmpOffset += toCopy; + tmpCount -= toCopy; + offset += toCopy; + count -= toCopy; + outTotal += toCopy; + } + if (count > 0 && unpack.DestSize > 0) + { + outBuffer = buffer; + outOffset = offset; + outCount = count; + fetch = true; + await unpack.DoUnpackAsync(cancellationToken).ConfigureAwait(false); + fetch = false; + } + _position += outTotal; + if (count > 0 && outTotal == 0 && _position < Length) + { + // sanity check, eg if we try to decompress a redir entry + throw new ArchiveOperationException( + $"unpacked file size does not match header: expected {Length} found {_position}" + ); + } + return outTotal; + } + +#if !LEGACY_DOTNET + /// + /// Asynchronously reads bytes from the current stream into a memory buffer. + /// + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + if (MemoryMarshal.TryGetArray(buffer, out var segment)) + { + return await ReadImplAsync( + segment.Array!, + segment.Offset, + buffer.Length, + cancellationToken + ) + .ConfigureAwait(false); + } + + var array = ArrayPool.Shared.Rent(buffer.Length); + try + { + var bytesRead = await ReadImplAsync(array, 0, buffer.Length, cancellationToken) + .ConfigureAwait(false); + new ReadOnlySpan(array, 0, bytesRead).CopyTo(buffer.Span); + return bytesRead; + } + finally + { + ArrayPool.Shared.Return(array); + } + } +#endif + + /// + /// Asynchronously writes bytes to the current stream. + /// + public override Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + cancellationToken.ThrowIfCancellationRequested(); + Write(buffer, offset, count); + return Task.CompletedTask; + } +} diff --git a/src/SharpCompress/Compressors/Rar/RarStream.cs b/src/SharpCompress/Compressors/Rar/RarStream.cs index 0bc2e054..3604a37b 100644 --- a/src/SharpCompress/Compressors/Rar/RarStream.cs +++ b/src/SharpCompress/Compressors/Rar/RarStream.cs @@ -1,12 +1,14 @@ #nullable disable using System; +using System.Buffers; using System.IO; +using SharpCompress.Common; using SharpCompress.Common.Rar.Headers; namespace SharpCompress.Compressors.Rar; -internal class RarStream : Stream +internal partial class RarStream : Stream { private readonly IRarUnpack unpack; private readonly FileHeader fileHeader; @@ -14,7 +16,7 @@ internal class RarStream : Stream private bool fetch; - private byte[] tmpBuffer = new byte[65536]; + private byte[] tmpBuffer = ArrayPool.Shared.Rent(65536); private int tmpOffset; private int tmpCount; @@ -22,6 +24,7 @@ internal class RarStream : Stream private int outOffset; private int outCount; private int outTotal; + private bool initialized; private bool isDisposed; private long _position; @@ -30,9 +33,19 @@ internal class RarStream : Stream this.unpack = unpack; this.fileHeader = fileHeader; this.readStream = readStream; + } + + public void Initialize() + { + if (initialized) + { + return; + } + fetch = true; unpack.DoUnpack(fileHeader, readStream, this); fetch = false; + initialized = true; _position = 0; } @@ -40,9 +53,14 @@ internal class RarStream : Stream { if (!isDisposed) { + if (disposing) + { + ArrayPool.Shared.Return(this.tmpBuffer); + this.tmpBuffer = null; + readStream.Dispose(); + } isDisposed = true; base.Dispose(disposing); - readStream.Dispose(); } } @@ -65,6 +83,12 @@ internal class RarStream : Stream public override int Read(byte[] buffer, int offset, int count) { + if (count == 0) + { + return 0; + } + + Initialize(); outTotal = 0; if (tmpCount > 0) { @@ -86,6 +110,13 @@ internal class RarStream : Stream fetch = false; } _position += outTotal; + if (count > 0 && outTotal == 0 && _position < Length) + { + // sanity check, eg if we try to decompress a redir entry + throw new ArchiveOperationException( + $"unpacked file size does not match header: expected {Length} found {_position}" + ); + } return outTotal; } @@ -111,16 +142,7 @@ internal class RarStream : Stream } if (count > 0) { - if (tmpBuffer.Length < tmpCount + count) - { - var newBuffer = new byte[ - tmpBuffer.Length * 2 > tmpCount + count - ? tmpBuffer.Length * 2 - : tmpCount + count - ]; - Buffer.BlockCopy(tmpBuffer, 0, newBuffer, 0, tmpCount); - tmpBuffer = newBuffer; - } + EnsureBufferCapacity(count); Buffer.BlockCopy(buffer, offset, tmpBuffer, tmpCount, count); tmpCount += count; tmpOffset = 0; @@ -131,4 +153,20 @@ internal class RarStream : Stream unpack.Suspended = false; } } + + private void EnsureBufferCapacity(int count) + { + if (this.tmpBuffer.Length < this.tmpCount + count) + { + var newLength = + this.tmpBuffer.Length * 2 > this.tmpCount + count + ? this.tmpBuffer.Length * 2 + : this.tmpCount + count; + var newBuffer = ArrayPool.Shared.Rent(newLength); + Buffer.BlockCopy(this.tmpBuffer, 0, newBuffer, 0, this.tmpCount); + var oldBuffer = this.tmpBuffer; + this.tmpBuffer = newBuffer; + ArrayPool.Shared.Return(oldBuffer); + } + } } diff --git a/src/SharpCompress/Compressors/Rar/UnpackV1/Decode/CodeType.cs b/src/SharpCompress/Compressors/Rar/UnpackV1/Decode/CodeType.cs index 0ec7943d..67f24def 100644 --- a/src/SharpCompress/Compressors/Rar/UnpackV1/Decode/CodeType.cs +++ b/src/SharpCompress/Compressors/Rar/UnpackV1/Decode/CodeType.cs @@ -10,5 +10,5 @@ internal enum CodeType CODE_STARTFILE, CODE_ENDFILE, CODE_VM, - CODE_VMDATA + CODE_VMDATA, } diff --git a/src/SharpCompress/Compressors/Rar/UnpackV1/Decode/FilterType.cs b/src/SharpCompress/Compressors/Rar/UnpackV1/Decode/FilterType.cs index 67e38c45..d09fcc16 100644 --- a/src/SharpCompress/Compressors/Rar/UnpackV1/Decode/FilterType.cs +++ b/src/SharpCompress/Compressors/Rar/UnpackV1/Decode/FilterType.cs @@ -12,5 +12,5 @@ internal enum FilterType : byte FILTER_RGB, FILTER_ITANIUM, FILTER_PPM, - FILTER_NONE + FILTER_NONE, } diff --git a/src/SharpCompress/Compressors/Rar/UnpackV1/PPM/BlockTypes.cs b/src/SharpCompress/Compressors/Rar/UnpackV1/PPM/BlockTypes.cs index 3b4a9372..ded3c34b 100644 --- a/src/SharpCompress/Compressors/Rar/UnpackV1/PPM/BlockTypes.cs +++ b/src/SharpCompress/Compressors/Rar/UnpackV1/PPM/BlockTypes.cs @@ -3,5 +3,5 @@ namespace SharpCompress.Compressors.Rar.UnpackV1.PPM; internal enum BlockTypes { BLOCK_LZ = 0, - BLOCK_PPM = 1 + BLOCK_PPM = 1, } diff --git a/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack.Async.cs b/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack.Async.cs new file mode 100644 index 00000000..adc53734 --- /dev/null +++ b/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack.Async.cs @@ -0,0 +1,936 @@ +using System; +using System.Buffers; +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Rar.Headers; +using SharpCompress.Compressors.Rar.UnpackV1.Decode; +using SharpCompress.Compressors.Rar.UnpackV1.PPM; +using SharpCompress.Compressors.Rar.VM; + +namespace SharpCompress.Compressors.Rar.UnpackV1; + +internal sealed partial class Unpack +{ + public async ValueTask DoUnpackAsync( + FileHeader fileHeader, + Stream readStream, + Stream writeStream, + CancellationToken cancellationToken = default + ) + { + destUnpSize = fileHeader.UncompressedSize; + this.fileHeader = fileHeader; + this.readStream = readStream; + this.writeStream = writeStream; + if (!fileHeader.IsSolid) + { + if (fileHeader.IsStored) + { + ReleaseWindow(); + } + else + { + Init(); + } + } + + suspended = false; + await DoUnpackAsync(cancellationToken).ConfigureAwait(false); + } + + public async ValueTask DoUnpackAsync(CancellationToken cancellationToken = default) + { + if (fileHeader.CompressionMethod == 0) + { + await UnstoreFileAsync(cancellationToken).ConfigureAwait(false); + return; + } + + switch (fileHeader.CompressionAlgorithm) + { + case 15: + await unpack15Async(fileHeader.IsSolid, cancellationToken).ConfigureAwait(false); + break; + case 20: + case 26: + await unpack20Async(fileHeader.IsSolid, cancellationToken).ConfigureAwait(false); + break; + case 29: + case 36: + await Unpack29Async(fileHeader.IsSolid, cancellationToken).ConfigureAwait(false); + break; + case 50: + await Unpack5Async(fileHeader.IsSolid, cancellationToken).ConfigureAwait(false); + break; + default: + throw new InvalidFormatException( + "unknown rar compression version " + fileHeader.CompressionAlgorithm + ); + } + } + + private async ValueTask UnstoreFileAsync(CancellationToken cancellationToken = default) + { + var buffer = ArrayPool.Shared.Rent((int)Math.Min(0x10000, destUnpSize)); + try + { + do + { + var code = await readStream + .ReadAsync(buffer, 0, buffer.Length, cancellationToken) + .ConfigureAwait(false); + if (code == 0 || code == -1) + { + break; + } + + code = code < destUnpSize ? code : (int)destUnpSize; + await writeStream + .WriteAsync(buffer, 0, code, cancellationToken) + .ConfigureAwait(false); + destUnpSize -= code; + } while (!suspended && destUnpSize > 0); + } + finally + { + ArrayPool.Shared.Return(buffer); + } + } + + private async ValueTask Unpack29Async(bool solid, CancellationToken cancellationToken = default) + { + int[] DDecode = new int[PackDef.DC]; + byte[] DBits = new byte[PackDef.DC]; + + int Bits; + + if (DDecode[1] == 0) + { + int Dist = 0, + BitLength = 0, + Slot = 0; + for (var I = 0; I < DBitLengthCounts.Length; I++, BitLength++) + { + var count = DBitLengthCounts[I]; + for (var J = 0; J < count; J++, Slot++, Dist += (1 << BitLength)) + { + DDecode[Slot] = Dist; + DBits[Slot] = (byte)BitLength; + } + } + } + + FileExtracted = true; + + if (!suspended) + { + UnpInitData(solid); + if (!await unpReadBufAsync(cancellationToken).ConfigureAwait(false)) + { + return; + } + + if ( + (!solid || !tablesRead) + && !await ReadTablesAsync(cancellationToken).ConfigureAwait(false) + ) + { + return; + } + } + + if (ppmError) + { + return; + } + + while (true) + { + unpPtr &= PackDef.MAXWINMASK; + + if (inAddr > readBorder) + { + if (!await unpReadBufAsync(cancellationToken).ConfigureAwait(false)) + { + break; + } + } + + if (((wrPtr - unpPtr) & PackDef.MAXWINMASK) < 260 && wrPtr != unpPtr) + { + await UnpWriteBufAsync(cancellationToken).ConfigureAwait(false); + if (destUnpSize < 0) + { + return; + } + + if (suspended) + { + FileExtracted = false; + return; + } + } + + if (unpBlockType == BlockTypes.BLOCK_PPM) + { + var Ch = await ppm.DecodeCharAsync(cancellationToken).ConfigureAwait(false); + if (Ch == -1) + { + ppmError = true; + break; + } + + if (Ch == PpmEscChar) + { + var NextCh = await ppm.DecodeCharAsync(cancellationToken).ConfigureAwait(false); + if (NextCh == 0) + { + if (!await ReadTablesAsync(cancellationToken).ConfigureAwait(false)) + { + break; + } + + continue; + } + + if (NextCh == 2 || NextCh == -1) + { + break; + } + + if (NextCh == 3) + { + if (!await ReadVMCodePPMAsync(cancellationToken).ConfigureAwait(false)) + { + break; + } + + continue; + } + + if (NextCh == 4) + { + int Distance = 0, + Length = 0; + var failed = false; + for (var I = 0; I < 4 && !failed; I++) + { + var ch = await ppm.DecodeCharAsync(cancellationToken) + .ConfigureAwait(false); + if (ch == -1) + { + failed = true; + } + else + { + if (I == 3) + { + Length = ch & 0xff; + } + else + { + Distance = (Distance << 8) + (ch & 0xff); + } + } + } + + if (failed) + { + break; + } + + CopyString(Length + 32, Distance + 2); + continue; + } + + if (NextCh == 5) + { + var Length = await ppm.DecodeCharAsync(cancellationToken) + .ConfigureAwait(false); + if (Length == -1) + { + break; + } + + CopyString(Length + 4, 1); + continue; + } + } + + window[unpPtr++] = (byte)Ch; + continue; + } + + var Number = this.decodeNumber(LD); + if (Number < 256) + { + window[unpPtr++] = (byte)Number; + continue; + } + + if (Number >= 271) + { + var Length = LDecode[Number -= 271] + 3; + if ((Bits = LBits[Number]) > 0) + { + Length += Utility.URShift(GetBits(), (16 - Bits)); + AddBits(Bits); + } + + var DistNumber = this.decodeNumber(DD); + var Distance = DDecode[DistNumber] + 1; + if ((Bits = DBits[DistNumber]) > 0) + { + if (DistNumber > 9) + { + if (Bits > 4) + { + Distance += ((Utility.URShift(GetBits(), (20 - Bits))) << 4); + AddBits(Bits - 4); + } + + if (lowDistRepCount > 0) + { + lowDistRepCount--; + Distance += prevLowDist; + } + else + { + var LowDist = this.decodeNumber(LDD); + if (LowDist == 16) + { + lowDistRepCount = PackDef.LOW_DIST_REP_COUNT - 1; + Distance += prevLowDist; + } + else + { + Distance += LowDist; + prevLowDist = LowDist; + } + } + } + else + { + Distance += Utility.URShift(GetBits(), (16 - Bits)); + AddBits(Bits); + } + } + + if (Distance >= 0x2000) + { + Length++; + if (Distance >= 0x40000L) + { + Length++; + } + } + + InsertOldDist(Distance); + InsertLastMatch(Length, Distance); + CopyString(Length, Distance); + continue; + } + + if (Number == 256) + { + if (!await ReadEndOfBlockAsync(cancellationToken).ConfigureAwait(false)) + { + break; + } + + continue; + } + + if (Number == 257) + { + if (!await ReadVMCodeAsync(cancellationToken).ConfigureAwait(false)) + { + break; + } + + continue; + } + + if (Number == 258) + { + if (lastLength != 0) + { + CopyString(lastLength, lastDist); + } + + continue; + } + + if (Number < 263) + { + var DistNum = Number - 259; + var Distance = oldDist[DistNum]; + for (var I = DistNum; I > 0; I--) + { + oldDist[I] = oldDist[I - 1]; + } + + oldDist[0] = Distance; + + var LengthNumber = this.decodeNumber(RD); + var Length = LDecode[LengthNumber] + 2; + if ((Bits = LBits[LengthNumber]) > 0) + { + Length += Utility.URShift(GetBits(), (16 - Bits)); + AddBits(Bits); + } + + InsertLastMatch(Length, Distance); + CopyString(Length, Distance); + continue; + } + + if (Number < 272) + { + var Distance = SDDecode[Number -= 263] + 1; + if ((Bits = SDBits[Number]) > 0) + { + Distance += Utility.URShift(GetBits(), (16 - Bits)); + AddBits(Bits); + } + + InsertOldDist(Distance); + InsertLastMatch(2, Distance); + CopyString(2, Distance); + } + } + + await UnpWriteBufAsync(cancellationToken).ConfigureAwait(false); + } + + private async ValueTask UnpWriteBufAsync(CancellationToken cancellationToken = default) + { + var WrittenBorder = wrPtr; + var WriteSize = (unpPtr - WrittenBorder) & PackDef.MAXWINMASK; + for (var I = 0; I < prgStack.Count; I++) + { + var flt = prgStack[I]; + if (flt is null) + { + continue; + } + + if (flt.NextWindow) + { + flt.NextWindow = false; + continue; + } + + var BlockStart = flt.BlockStart; + var BlockLength = flt.BlockLength; + if (((BlockStart - WrittenBorder) & PackDef.MAXWINMASK) < WriteSize) + { + if (WrittenBorder != BlockStart) + { + await UnpWriteAreaAsync(WrittenBorder, BlockStart, cancellationToken) + .ConfigureAwait(false); + WrittenBorder = BlockStart; + WriteSize = (unpPtr - WrittenBorder) & PackDef.MAXWINMASK; + } + + if (BlockLength <= WriteSize) + { + var BlockEnd = (BlockStart + BlockLength) & PackDef.MAXWINMASK; + if (BlockStart < BlockEnd || BlockEnd == 0) + { + rarVM.setMemory(0, window, BlockStart, BlockLength); + } + else + { + var FirstPartLength = PackDef.MAXWINSIZE - BlockStart; + rarVM.setMemory(0, window, BlockStart, FirstPartLength); + rarVM.setMemory(FirstPartLength, window, 0, BlockEnd); + } + + var ParentPrg = filters[flt.ParentFilter].Program; + var Prg = flt.Program; + + if (ParentPrg.GlobalData.Count > RarVM.VM_FIXEDGLOBALSIZE) + { + Prg.GlobalData.Clear(); + for ( + var i = 0; + i < ParentPrg.GlobalData.Count - RarVM.VM_FIXEDGLOBALSIZE; + i++ + ) + { + Prg.GlobalData[RarVM.VM_FIXEDGLOBALSIZE + i] = ParentPrg.GlobalData[ + RarVM.VM_FIXEDGLOBALSIZE + i + ]; + } + } + + ExecuteCode(Prg); + + if (Prg.GlobalData.Count > RarVM.VM_FIXEDGLOBALSIZE) + { + if (ParentPrg.GlobalData.Count < Prg.GlobalData.Count) + { + ParentPrg.GlobalData.SetSize(Prg.GlobalData.Count); + } + + for (var i = 0; i < Prg.GlobalData.Count - RarVM.VM_FIXEDGLOBALSIZE; i++) + { + ParentPrg.GlobalData[RarVM.VM_FIXEDGLOBALSIZE + i] = Prg.GlobalData[ + RarVM.VM_FIXEDGLOBALSIZE + i + ]; + } + } + else + { + ParentPrg.GlobalData.Clear(); + } + + var FilteredDataOffset = Prg.FilteredDataOffset; + var FilteredDataSize = Prg.FilteredDataSize; + var FilteredData = ArrayPool.Shared.Rent(FilteredDataSize); + try + { + Array.Copy( + rarVM.Mem, + FilteredDataOffset, + FilteredData, + 0, + FilteredDataSize + ); + + prgStack[I] = null; + while (I + 1 < prgStack.Count) + { + var NextFilter = prgStack[I + 1]; + if ( + NextFilter is null + || NextFilter.BlockStart != BlockStart + || NextFilter.BlockLength != FilteredDataSize + || NextFilter.NextWindow + ) + { + break; + } + + rarVM.setMemory(0, FilteredData, 0, FilteredDataSize); + + var pPrg = filters[NextFilter.ParentFilter].Program; + var NextPrg = NextFilter.Program; + + if (pPrg.GlobalData.Count > RarVM.VM_FIXEDGLOBALSIZE) + { + NextPrg.GlobalData.SetSize(pPrg.GlobalData.Count); + + for ( + var i = 0; + i < pPrg.GlobalData.Count - RarVM.VM_FIXEDGLOBALSIZE; + i++ + ) + { + NextPrg.GlobalData[RarVM.VM_FIXEDGLOBALSIZE + i] = + pPrg.GlobalData[RarVM.VM_FIXEDGLOBALSIZE + i]; + } + } + + ExecuteCode(NextPrg); + + if (NextPrg.GlobalData.Count > RarVM.VM_FIXEDGLOBALSIZE) + { + if (pPrg.GlobalData.Count < NextPrg.GlobalData.Count) + { + pPrg.GlobalData.SetSize(NextPrg.GlobalData.Count); + } + + for ( + var i = 0; + i < NextPrg.GlobalData.Count - RarVM.VM_FIXEDGLOBALSIZE; + i++ + ) + { + pPrg.GlobalData[RarVM.VM_FIXEDGLOBALSIZE + i] = + NextPrg.GlobalData[RarVM.VM_FIXEDGLOBALSIZE + i]; + } + } + else + { + pPrg.GlobalData.Clear(); + } + + FilteredDataOffset = NextPrg.FilteredDataOffset; + FilteredDataSize = NextPrg.FilteredDataSize; + if (FilteredData.Length < FilteredDataSize) + { + ArrayPool.Shared.Return(FilteredData); + FilteredData = ArrayPool.Shared.Rent(FilteredDataSize); + } + + for (var i = 0; i < FilteredDataSize; i++) + { + FilteredData[i] = NextPrg.GlobalData[FilteredDataOffset + i]; + } + + I++; + prgStack[I] = null; + } + + await writeStream + .WriteAsync(FilteredData, 0, FilteredDataSize, cancellationToken) + .ConfigureAwait(false); + writtenFileSize += FilteredDataSize; + destUnpSize -= FilteredDataSize; + WrittenBorder = BlockEnd; + WriteSize = (unpPtr - WrittenBorder) & PackDef.MAXWINMASK; + } + finally + { + ArrayPool.Shared.Return(FilteredData); + } + } + else + { + for (var J = I; J < prgStack.Count; J++) + { + var filt = prgStack[J]; + if (filt != null && filt.NextWindow) + { + filt.NextWindow = false; + } + } + + wrPtr = WrittenBorder; + return; + } + } + } + + await UnpWriteAreaAsync(WrittenBorder, unpPtr, cancellationToken).ConfigureAwait(false); + wrPtr = unpPtr; + } + + private async ValueTask UnpWriteAreaAsync( + int startPtr, + int endPtr, + CancellationToken cancellationToken = default + ) + { + if (endPtr < startPtr) + { + await UnpWriteDataAsync( + window, + startPtr, + -startPtr & PackDef.MAXWINMASK, + cancellationToken + ) + .ConfigureAwait(false); + await UnpWriteDataAsync(window, 0, endPtr, cancellationToken).ConfigureAwait(false); + } + else + { + await UnpWriteDataAsync(window, startPtr, endPtr - startPtr, cancellationToken) + .ConfigureAwait(false); + } + } + + private async ValueTask UnpWriteDataAsync( + byte[] data, + int offset, + int size, + CancellationToken cancellationToken = default + ) + { + if (destUnpSize < 0) + { + return; + } + + var writeSize = size; + if (writeSize > destUnpSize) + { + writeSize = (int)destUnpSize; + } + + await writeStream + .WriteAsync(data, offset, writeSize, cancellationToken) + .ConfigureAwait(false); + + writtenFileSize += size; + destUnpSize -= size; + } + + private async ValueTask ReadTablesAsync(CancellationToken cancellationToken = default) + { + var bitLengthArray = ArrayPool.Shared.Rent(PackDef.BC); + var bitLength = new Memory(bitLengthArray, 0, PackDef.BC); + var tableArray = ArrayPool.Shared.Rent(PackDef.HUFF_TABLE_SIZE); + var table = new Memory(tableArray, 0, PackDef.HUFF_TABLE_SIZE); + + try + { + if (inAddr > readTop - 25) + { + if (!await unpReadBufAsync(cancellationToken).ConfigureAwait(false)) + { + return false; + } + } + + AddBits((8 - inBit) & 7); + long bitField = GetBits() & unchecked((int)0xffFFffFF); + if ((bitField & 0x8000) != 0) + { + unpBlockType = BlockTypes.BLOCK_PPM; + return await ppm.DecodeInitAsync(this, PpmEscChar, cancellationToken) + .ConfigureAwait(false); + } + + unpBlockType = BlockTypes.BLOCK_LZ; + + prevLowDist = 0; + lowDistRepCount = 0; + + if ((bitField & 0x4000) == 0) + { + new Span(unpOldTable).Clear(); + } + + AddBits(2); + + for (var i = 0; i < PackDef.BC; i++) + { + var length = (Utility.URShift(GetBits(), 12)) & 0xFF; + AddBits(4); + if (length == 15) + { + var zeroCount = (Utility.URShift(GetBits(), 12)) & 0xFF; + AddBits(4); + if (zeroCount == 0) + { + bitLength.Span[i] = 15; + } + else + { + zeroCount += 2; + while (zeroCount-- > 0 && i < bitLength.Length) + { + bitLength.Span[i++] = 0; + } + + i--; + } + } + else + { + bitLength.Span[i] = (byte)length; + } + } + + UnpackUtility.makeDecodeTables(bitLength.Span, 0, BD, PackDef.BC); + + var TableSize = PackDef.HUFF_TABLE_SIZE; + + for (var i = 0; i < TableSize; ) + { + if (inAddr > readTop - 5) + { + if (!await unpReadBufAsync(cancellationToken).ConfigureAwait(false)) + { + return false; + } + } + + var Number = this.decodeNumber(BD); + if (Number < 16) + { + table.Span[i] = (byte)((Number + unpOldTable[i]) & 0xf); + i++; + } + else if (Number < 18) + { + int N; + if (Number == 16) + { + N = (Utility.URShift(GetBits(), 13)) + 3; + AddBits(3); + } + else + { + N = (Utility.URShift(GetBits(), 9)) + 11; + AddBits(7); + } + + while (N-- > 0 && i < TableSize) + { + table.Span[i] = table.Span[i - 1]; + i++; + } + } + else + { + int N; + if (Number == 18) + { + N = (Utility.URShift(GetBits(), 13)) + 3; + AddBits(3); + } + else + { + N = (Utility.URShift(GetBits(), 9)) + 11; + AddBits(7); + } + + while (N-- > 0 && i < TableSize) + { + table.Span[i++] = 0; + } + } + } + + tablesRead = true; + if (inAddr > readTop) + { + return false; + } + + UnpackUtility.makeDecodeTables(table.Span, 0, LD, PackDef.NC); + UnpackUtility.makeDecodeTables(table.Span, PackDef.NC, DD, PackDef.DC); + UnpackUtility.makeDecodeTables(table.Span, PackDef.NC + PackDef.DC, LDD, PackDef.LDC); + UnpackUtility.makeDecodeTables( + table.Span, + PackDef.NC + PackDef.DC + PackDef.LDC, + RD, + PackDef.RC + ); + + table.Span.CopyTo(unpOldTable); + return true; + } + finally + { + ArrayPool.Shared.Return(bitLengthArray); + ArrayPool.Shared.Return(tableArray); + } + } + + private async ValueTask ReadEndOfBlockAsync(CancellationToken cancellationToken = default) + { + var BitField = GetBits(); + bool NewTable, + NewFile = false; + if ((BitField & 0x8000) != 0) + { + NewTable = true; + AddBits(1); + } + else + { + NewFile = true; + NewTable = (BitField & 0x4000) != 0; + AddBits(2); + } + + tablesRead = !NewTable; + return !( + NewFile || NewTable && !await ReadTablesAsync(cancellationToken).ConfigureAwait(false) + ); + } + + private async ValueTask ReadVMCodeAsync(CancellationToken cancellationToken = default) + { + var FirstByte = GetBits() >> 8; + AddBits(8); + var Length = (FirstByte & 7) + 1; + if (Length == 7) + { + Length = (GetBits() >> 8) + 7; + AddBits(8); + } + else if (Length == 8) + { + Length = GetBits(); + AddBits(16); + } + + var vmCode = new List(); + for (var I = 0; I < Length; I++) + { + if ( + inAddr >= readTop - 1 + && !await unpReadBufAsync(cancellationToken).ConfigureAwait(false) + && I < Length - 1 + ) + { + return false; + } + + vmCode.Add((byte)(GetBits() >> 8)); + AddBits(8); + } + + return AddVMCode(FirstByte, vmCode); + } + + public async ValueTask ReadCharAsync(CancellationToken cancellationToken = default) + { + if (inAddr > MAX_SIZE - 30) + { + await unpReadBufAsync(cancellationToken).ConfigureAwait(false); + } + + return InBuf[inAddr++] & 0xff; + } + + private async ValueTask ReadVMCodePPMAsync(CancellationToken cancellationToken = default) + { + var FirstByte = await ppm.DecodeCharAsync(cancellationToken).ConfigureAwait(false); + if (FirstByte == -1) + { + return false; + } + + var Length = (FirstByte & 7) + 1; + if (Length == 7) + { + var B1 = await ppm.DecodeCharAsync(cancellationToken).ConfigureAwait(false); + if (B1 == -1) + { + return false; + } + + Length = B1 + 7; + } + else if (Length == 8) + { + var B1 = await ppm.DecodeCharAsync(cancellationToken).ConfigureAwait(false); + if (B1 == -1) + { + return false; + } + + var B2 = await ppm.DecodeCharAsync(cancellationToken).ConfigureAwait(false); + if (B2 == -1) + { + return false; + } + + Length = (B1 * 256) + B2; + } + + var vmCode = new List(); + for (var I = 0; I < Length; I++) + { + var Ch = await ppm.DecodeCharAsync(cancellationToken).ConfigureAwait(false); + if (Ch == -1) + { + return false; + } + + vmCode.Add((byte)Ch); + } + + return AddVMCode(FirstByte, vmCode); + } +} diff --git a/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack.cs b/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack.cs index 2ca62d6b..dcde95d4 100644 --- a/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack.cs +++ b/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack.cs @@ -1,6 +1,7 @@ #nullable disable using System; +using System.Buffers; using System.Collections.Generic; using System.IO; using SharpCompress.Common; @@ -15,11 +16,33 @@ namespace SharpCompress.Compressors.Rar.UnpackV1; internal sealed partial class Unpack : BitInput, IRarUnpack { private readonly BitInput Inp; + private bool disposed; public Unpack() => // to ease in porting Unpack50.cs Inp = this; + public override void Dispose() + { + if (!disposed) + { + base.Dispose(); + ReleaseWindow(); + rarVM.Dispose(); + ppm.Dispose(); + disposed = true; + } + } + + private void ReleaseWindow() + { + if (window is not null) + { + ArrayPool.Shared.Return(window); + window = null; + } + } + public bool FileExtracted { get; private set; } public long DestSize @@ -38,33 +61,31 @@ internal sealed partial class Unpack : BitInput, IRarUnpack set => suspended = value; } - public int Char + public int ReadChar() { - get + if (inAddr > MAX_SIZE - 30) { - if (inAddr > MAX_SIZE - 30) - { - unpReadBuf(); - } - return (InBuf[inAddr++] & 0xff); + unpReadBuf(); } + + return (InBuf[inAddr++] & 0xff); } public int PpmEscChar { get; set; } - private readonly ModelPpm ppm = new ModelPpm(); + private readonly ModelPpm ppm = new(); - private readonly RarVM rarVM = new RarVM(); + private readonly RarVM rarVM = new(); // Filters code, one entry per filter - private readonly List filters = new List(); + private readonly List filters = new(); // Filters stack, several entrances of same filter are possible - private readonly List prgStack = new List(); + private readonly List prgStack = new(); // lengths of preceding blocks, one length per filter. Used to reduce size // required to write block length if lengths are repeating - private readonly List oldFilterLengths = new List(); + private readonly List oldFilterLengths = new(); private int lastFilter; @@ -74,8 +95,6 @@ internal sealed partial class Unpack : BitInput, IRarUnpack private BlockTypes unpBlockType; - //private bool externalWindow; - private long writtenFileSize; private bool ppmError; @@ -104,23 +123,18 @@ internal sealed partial class Unpack : BitInput, IRarUnpack 2, 14, 0, - 12 + 12, }; private FileHeader fileHeader; - private void Init(byte[] window) + private void Init() { - if (window is null) + if (this.window is null) { - this.window = new byte[PackDef.MAXWINSIZE]; + this.window = ArrayPool.Shared.Rent(PackDef.MAXWINSIZE); } - else - { - this.window = window; - //externalWindow = true; - } inAddr = 0; UnpInitData(false); } @@ -133,8 +147,16 @@ internal sealed partial class Unpack : BitInput, IRarUnpack this.writeStream = writeStream; if (!fileHeader.IsSolid) { - Init(null); + if (fileHeader.IsStored) + { + ReleaseWindow(); + } + else + { + Init(); + } } + suspended = false; DoUnpack(); } @@ -146,6 +168,7 @@ internal sealed partial class Unpack : BitInput, IRarUnpack UnstoreFile(); return; } + switch (fileHeader.CompressionAlgorithm) { case 15: // rar 1.5 compression @@ -175,31 +198,25 @@ internal sealed partial class Unpack : BitInput, IRarUnpack private void UnstoreFile() { - var buffer = new byte[0x10000]; - while (true) + Span buffer = stackalloc byte[(int)Math.Min(0x10000, destUnpSize)]; + do { - var code = readStream.Read(buffer, 0, (int)Math.Min(buffer.Length, destUnpSize)); + var code = readStream.Read(buffer); if (code == 0 || code == -1) { break; } + code = code < destUnpSize ? code : (int)destUnpSize; - writeStream.Write(buffer, 0, code); - if (destUnpSize >= 0) - { - destUnpSize -= code; - } - if (suspended) - { - return; - } - } + writeStream.Write(buffer.Slice(0, code)); + destUnpSize -= code; + } while (!suspended && destUnpSize > 0); } private void Unpack29(bool solid) { - var DDecode = new int[PackDef.DC]; - var DBits = new byte[PackDef.DC]; + Span DDecode = stackalloc int[PackDef.DC]; + Span DBits = stackalloc byte[PackDef.DC]; int Bits; @@ -228,6 +245,7 @@ internal sealed partial class Unpack : BitInput, IRarUnpack { return; } + if ((!solid || !tablesRead) && !ReadTables()) { return; @@ -256,16 +274,18 @@ internal sealed partial class Unpack : BitInput, IRarUnpack if (((wrPtr - unpPtr) & PackDef.MAXWINMASK) < 260 && wrPtr != unpPtr) { UnpWriteBuf(); - if (destUnpSize <= 0) + if (destUnpSize < 0) { return; } + if (suspended) { FileExtracted = false; return; } } + if (unpBlockType == BlockTypes.BLOCK_PPM) { var Ch = ppm.DecodeChar(); @@ -274,6 +294,7 @@ internal sealed partial class Unpack : BitInput, IRarUnpack ppmError = true; break; } + if (Ch == PpmEscChar) { var NextCh = ppm.DecodeChar(); @@ -283,20 +304,25 @@ internal sealed partial class Unpack : BitInput, IRarUnpack { break; } + continue; } + if (NextCh == 2 || NextCh == -1) { break; } + if (NextCh == 3) { if (!ReadVMCodePPM()) { break; } + continue; } + if (NextCh == 4) { int Distance = 0, @@ -323,13 +349,16 @@ internal sealed partial class Unpack : BitInput, IRarUnpack } } } + if (failed) { break; } + CopyString(Length + 32, Distance + 2); continue; } + if (NextCh == 5) { var Length = ppm.DecodeChar(); @@ -337,10 +366,12 @@ internal sealed partial class Unpack : BitInput, IRarUnpack { break; } + CopyString(Length + 4, 1); continue; } } + window[unpPtr++] = (byte)Ch; continue; } @@ -351,6 +382,7 @@ internal sealed partial class Unpack : BitInput, IRarUnpack window[unpPtr++] = (byte)Number; continue; } + if (Number >= 271) { var Length = LDecode[Number -= 271] + 3; @@ -371,6 +403,7 @@ internal sealed partial class Unpack : BitInput, IRarUnpack Distance += ((Utility.URShift(GetBits(), (20 - Bits))) << 4); AddBits(Bits - 4); } + if (lowDistRepCount > 0) { lowDistRepCount--; @@ -413,30 +446,37 @@ internal sealed partial class Unpack : BitInput, IRarUnpack CopyString(Length, Distance); continue; } + if (Number == 256) { if (!ReadEndOfBlock()) { break; } + continue; } + if (Number == 257) { if (!ReadVMCode()) { break; } + continue; } + if (Number == 258) { if (lastLength != 0) { CopyString(lastLength, lastDist); } + continue; } + if (Number < 263) { var DistNum = Number - 259; @@ -445,6 +485,7 @@ internal sealed partial class Unpack : BitInput, IRarUnpack { oldDist[I] = oldDist[I - 1]; } + oldDist[0] = Distance; var LengthNumber = this.decodeNumber(RD); @@ -454,10 +495,12 @@ internal sealed partial class Unpack : BitInput, IRarUnpack Length += Utility.URShift(GetBits(), (16 - Bits)); AddBits(Bits); } + InsertLastMatch(Length, Distance); CopyString(Length, Distance); continue; } + if (Number < 272) { var Distance = SDDecode[Number -= 263] + 1; @@ -466,11 +509,13 @@ internal sealed partial class Unpack : BitInput, IRarUnpack Distance += Utility.URShift(GetBits(), (16 - Bits)); AddBits(Bits); } + InsertOldDist(Distance); InsertLastMatch(2, Distance); CopyString(2, Distance); } } + UnpWriteBuf(); } @@ -485,11 +530,13 @@ internal sealed partial class Unpack : BitInput, IRarUnpack { continue; } + if (flt.NextWindow) { flt.NextWindow = false; // ->NextWindow=false; continue; } + var BlockStart = flt.BlockStart; // ->BlockStart; var BlockLength = flt.BlockLength; // ->BlockLength; if (((BlockStart - WrittenBorder) & PackDef.MAXWINMASK) < WriteSize) @@ -500,6 +547,7 @@ internal sealed partial class Unpack : BitInput, IRarUnpack WrittenBorder = BlockStart; WriteSize = (unpPtr - WrittenBorder) & PackDef.MAXWINMASK; } + if (BlockLength <= WriteSize) { var BlockEnd = (BlockStart + BlockLength) & PackDef.MAXWINMASK; @@ -567,104 +615,112 @@ internal sealed partial class Unpack : BitInput, IRarUnpack var FilteredDataOffset = Prg.FilteredDataOffset; var FilteredDataSize = Prg.FilteredDataSize; - var FilteredData = new byte[FilteredDataSize]; - - for (var i = 0; i < FilteredDataSize; i++) + var FilteredData = ArrayPool.Shared.Rent(FilteredDataSize); + try { - FilteredData[i] = rarVM.Mem[FilteredDataOffset + i]; + Array.Copy( + rarVM.Mem, + FilteredDataOffset, + FilteredData, + 0, + FilteredDataSize + ); - // Prg.GlobalData.get(FilteredDataOffset - // + - // i); - } - - prgStack[I] = null; - while (I + 1 < prgStack.Count) - { - var NextFilter = prgStack[I + 1]; - if ( - NextFilter is null - || NextFilter.BlockStart != BlockStart - || NextFilter.BlockLength != FilteredDataSize - || NextFilter.NextWindow - ) - { - break; - } - - // apply several filters to same data block - - rarVM.setMemory(0, FilteredData, 0, FilteredDataSize); - - // .SetMemory(0,FilteredData,FilteredDataSize); - - var pPrg = filters[NextFilter.ParentFilter].Program; - var NextPrg = NextFilter.Program; - - if (pPrg.GlobalData.Count > RarVM.VM_FIXEDGLOBALSIZE) - { - // copy global data from previous script execution - // if any - // NextPrg->GlobalData.Alloc(ParentPrg->GlobalData.Size()); - NextPrg.GlobalData.SetSize(pPrg.GlobalData.Count); - - // memcpy(&NextPrg->GlobalData[VM_FIXEDGLOBALSIZE],&ParentPrg->GlobalData[VM_FIXEDGLOBALSIZE],ParentPrg->GlobalData.Size()-VM_FIXEDGLOBALSIZE); - for ( - var i = 0; - i < pPrg.GlobalData.Count - RarVM.VM_FIXEDGLOBALSIZE; - i++ - ) - { - NextPrg.GlobalData[RarVM.VM_FIXEDGLOBALSIZE + i] = pPrg.GlobalData[ - RarVM.VM_FIXEDGLOBALSIZE + i - ]; - } - } - - ExecuteCode(NextPrg); - - if (NextPrg.GlobalData.Count > RarVM.VM_FIXEDGLOBALSIZE) - { - // save global data for next script execution - if (pPrg.GlobalData.Count < NextPrg.GlobalData.Count) - { - pPrg.GlobalData.SetSize(NextPrg.GlobalData.Count); - } - - // memcpy(&ParentPrg->GlobalData[VM_FIXEDGLOBALSIZE],&NextPrg->GlobalData[VM_FIXEDGLOBALSIZE],NextPrg->GlobalData.Size()-VM_FIXEDGLOBALSIZE); - for ( - var i = 0; - i < NextPrg.GlobalData.Count - RarVM.VM_FIXEDGLOBALSIZE; - i++ - ) - { - pPrg.GlobalData[RarVM.VM_FIXEDGLOBALSIZE + i] = NextPrg.GlobalData[ - RarVM.VM_FIXEDGLOBALSIZE + i - ]; - } - } - else - { - pPrg.GlobalData.Clear(); - } - FilteredDataOffset = NextPrg.FilteredDataOffset; - FilteredDataSize = NextPrg.FilteredDataSize; - - FilteredData = new byte[FilteredDataSize]; - for (var i = 0; i < FilteredDataSize; i++) - { - FilteredData[i] = NextPrg.GlobalData[FilteredDataOffset + i]; - } - - I++; prgStack[I] = null; + while (I + 1 < prgStack.Count) + { + var NextFilter = prgStack[I + 1]; + if ( + NextFilter is null + || NextFilter.BlockStart != BlockStart + || NextFilter.BlockLength != FilteredDataSize + || NextFilter.NextWindow + ) + { + break; + } + + // apply several filters to same data block + + rarVM.setMemory(0, FilteredData, 0, FilteredDataSize); + + // .SetMemory(0,FilteredData,FilteredDataSize); + + var pPrg = filters[NextFilter.ParentFilter].Program; + var NextPrg = NextFilter.Program; + + if (pPrg.GlobalData.Count > RarVM.VM_FIXEDGLOBALSIZE) + { + // copy global data from previous script execution + // if any + // NextPrg->GlobalData.Alloc(ParentPrg->GlobalData.Size()); + NextPrg.GlobalData.SetSize(pPrg.GlobalData.Count); + + // memcpy(&NextPrg->GlobalData[VM_FIXEDGLOBALSIZE],&ParentPrg->GlobalData[VM_FIXEDGLOBALSIZE],ParentPrg->GlobalData.Size()-VM_FIXEDGLOBALSIZE); + for ( + var i = 0; + i < pPrg.GlobalData.Count - RarVM.VM_FIXEDGLOBALSIZE; + i++ + ) + { + NextPrg.GlobalData[RarVM.VM_FIXEDGLOBALSIZE + i] = + pPrg.GlobalData[RarVM.VM_FIXEDGLOBALSIZE + i]; + } + } + + ExecuteCode(NextPrg); + + if (NextPrg.GlobalData.Count > RarVM.VM_FIXEDGLOBALSIZE) + { + // save global data for next script execution + if (pPrg.GlobalData.Count < NextPrg.GlobalData.Count) + { + pPrg.GlobalData.SetSize(NextPrg.GlobalData.Count); + } + + // memcpy(&ParentPrg->GlobalData[VM_FIXEDGLOBALSIZE],&NextPrg->GlobalData[VM_FIXEDGLOBALSIZE],NextPrg->GlobalData.Size()-VM_FIXEDGLOBALSIZE); + for ( + var i = 0; + i < NextPrg.GlobalData.Count - RarVM.VM_FIXEDGLOBALSIZE; + i++ + ) + { + pPrg.GlobalData[RarVM.VM_FIXEDGLOBALSIZE + i] = + NextPrg.GlobalData[RarVM.VM_FIXEDGLOBALSIZE + i]; + } + } + else + { + pPrg.GlobalData.Clear(); + } + + FilteredDataOffset = NextPrg.FilteredDataOffset; + FilteredDataSize = NextPrg.FilteredDataSize; + if (FilteredData.Length < FilteredDataSize) + { + ArrayPool.Shared.Return(FilteredData); + FilteredData = ArrayPool.Shared.Rent(FilteredDataSize); + } + + for (var i = 0; i < FilteredDataSize; i++) + { + FilteredData[i] = NextPrg.GlobalData[FilteredDataOffset + i]; + } + + I++; + prgStack[I] = null; + } + + writeStream.Write(FilteredData, 0, FilteredDataSize); + writtenFileSize += FilteredDataSize; + destUnpSize -= FilteredDataSize; + WrittenBorder = BlockEnd; + WriteSize = (unpPtr - WrittenBorder) & PackDef.MAXWINMASK; + } + finally + { + ArrayPool.Shared.Return(FilteredData); } - writeStream.Write(FilteredData, 0, FilteredDataSize); - unpSomeRead = true; - writtenFileSize += FilteredDataSize; - destUnpSize -= FilteredDataSize; - WrittenBorder = BlockEnd; - WriteSize = (unpPtr - WrittenBorder) & PackDef.MAXWINMASK; } else { @@ -676,6 +732,7 @@ internal sealed partial class Unpack : BitInput, IRarUnpack filt.NextWindow = false; } } + wrPtr = WrittenBorder; return; } @@ -688,15 +745,10 @@ internal sealed partial class Unpack : BitInput, IRarUnpack private void UnpWriteArea(int startPtr, int endPtr) { - if (endPtr != startPtr) - { - unpSomeRead = true; - } if (endPtr < startPtr) { UnpWriteData(window, startPtr, -startPtr & PackDef.MAXWINMASK); UnpWriteData(window, 0, endPtr); - unpAllBuf = true; } else { @@ -706,15 +758,19 @@ internal sealed partial class Unpack : BitInput, IRarUnpack private void UnpWriteData(byte[] data, int offset, int size) { - if (destUnpSize <= 0) + // allow destUnpSize == 0 here to ensure that 0 size writes + // go through RarStream's Write so that Suspended is set correctly + if (destUnpSize < 0) { return; } + var writeSize = size; if (writeSize > destUnpSize) { writeSize = (int)destUnpSize; } + writeStream.Write(data, offset, writeSize); writtenFileSize += size; @@ -748,19 +804,28 @@ internal sealed partial class Unpack : BitInput, IRarUnpack // System.out.println("copyString(" + length + ", " + distance + ")"); var destPtr = unpPtr - distance; + var safeZone = PackDef.MAXWINSIZE - 260; - // System.out.println(unpPtr+":"+distance); - if (destPtr >= 0 && destPtr < PackDef.MAXWINSIZE - 260 && unpPtr < PackDef.MAXWINSIZE - 260) + // Fast path: use Array.Copy for bulk operations when in safe zone + if (destPtr >= 0 && destPtr < safeZone && unpPtr < safeZone && distance >= length) { - window[unpPtr++] = window[destPtr++]; - - while (--length > 0) + // Non-overlapping copy: can use Array.Copy directly + Array.Copy(window, destPtr, window, unpPtr, length); + unpPtr += length; + } + else if (destPtr >= 0 && destPtr < safeZone && unpPtr < safeZone) + { + // Overlapping copy in safe zone: use byte-by-byte to handle self-referential copies + for (int i = 0; i < length; i++) { - window[unpPtr++] = window[destPtr++]; + window[unpPtr + i] = window[destPtr + i]; } + + unpPtr += length; } else { + // Slow path with wraparound mask while (length-- != 0) { window[unpPtr] = window[destPtr++ & PackDef.MAXWINMASK]; @@ -789,6 +854,7 @@ internal sealed partial class Unpack : BitInput, IRarUnpack InitFilters(); } + InitBitInput(); ppmError = false; writtenFileSize = 0; @@ -853,15 +919,16 @@ internal sealed partial class Unpack : BitInput, IRarUnpack NewTable = (BitField & 0x4000) != 0; AddBits(2); } + tablesRead = !NewTable; return !(NewFile || NewTable && !ReadTables()); } private bool ReadTables() { - var bitLength = new byte[PackDef.BC]; + Span bitLength = stackalloc byte[PackDef.BC]; + Span table = stackalloc byte[PackDef.HUFF_TABLE_SIZE]; - var table = new byte[PackDef.HUFF_TABLE_SIZE]; if (inAddr > readTop - 25) { if (!unpReadBuf()) @@ -869,6 +936,7 @@ internal sealed partial class Unpack : BitInput, IRarUnpack return (false); } } + AddBits((8 - inBit) & 7); long bitField = GetBits() & unchecked((int)0xffFFffFF); if ((bitField & 0x8000) != 0) @@ -876,6 +944,7 @@ internal sealed partial class Unpack : BitInput, IRarUnpack unpBlockType = BlockTypes.BLOCK_PPM; return (ppm.DecodeInit(this, PpmEscChar)); } + unpBlockType = BlockTypes.BLOCK_LZ; prevLowDist = 0; @@ -885,6 +954,7 @@ internal sealed partial class Unpack : BitInput, IRarUnpack { new Span(unpOldTable).Clear(); // memset(UnpOldTable,0,sizeof(UnpOldTable)); } + AddBits(2); for (var i = 0; i < PackDef.BC; i++) @@ -906,6 +976,7 @@ internal sealed partial class Unpack : BitInput, IRarUnpack { bitLength[i++] = 0; } + i--; } } @@ -928,6 +999,7 @@ internal sealed partial class Unpack : BitInput, IRarUnpack return (false); } } + var Number = this.decodeNumber(BD); if (Number < 16) { @@ -947,6 +1019,7 @@ internal sealed partial class Unpack : BitInput, IRarUnpack N = (Utility.URShift(GetBits(), 9)) + 11; AddBits(7); } + while (N-- > 0 && i < TableSize) { table[i] = table[i - 1]; @@ -966,17 +1039,20 @@ internal sealed partial class Unpack : BitInput, IRarUnpack N = (Utility.URShift(GetBits(), 9)) + 11; AddBits(7); } + while (N-- > 0 && i < TableSize) { table[i++] = 0; } } } + tablesRead = true; if (inAddr > readTop) { return (false); } + UnpackUtility.makeDecodeTables(table, 0, LD, PackDef.NC); UnpackUtility.makeDecodeTables(table, PackDef.NC, DD, PackDef.DC); UnpackUtility.makeDecodeTables(table, PackDef.NC + PackDef.DC, LDD, PackDef.LDC); @@ -989,7 +1065,7 @@ internal sealed partial class Unpack : BitInput, IRarUnpack // memcpy(unpOldTable,table,sizeof(unpOldTable)); - Buffer.BlockCopy(table, 0, unpOldTable, 0, unpOldTable.Length); + table.CopyTo(unpOldTable); return (true); } @@ -1016,10 +1092,12 @@ internal sealed partial class Unpack : BitInput, IRarUnpack { return (false); } + vmCode.Add((byte)(GetBits() >> 8)); AddBits(8); } - return (AddVMCode(FirstByte, vmCode, Length)); + + return AddVMCode(FirstByte, vmCode); } private bool ReadVMCodePPM() @@ -1029,6 +1107,7 @@ internal sealed partial class Unpack : BitInput, IRarUnpack { return (false); } + var Length = (FirstByte & 7) + 1; if (Length == 7) { @@ -1037,6 +1116,7 @@ internal sealed partial class Unpack : BitInput, IRarUnpack { return (false); } + Length = B1 + 7; } else if (Length == 8) @@ -1046,11 +1126,13 @@ internal sealed partial class Unpack : BitInput, IRarUnpack { return (false); } + var B2 = ppm.DecodeChar(); if (B2 == -1) { return (false); } + Length = (B1 * 256) + B2; } @@ -1062,14 +1144,16 @@ internal sealed partial class Unpack : BitInput, IRarUnpack { return (false); } + vmCode.Add((byte)Ch); // VMCode[I]=Ch; } - return (AddVMCode(FirstByte, vmCode, Length)); + + return AddVMCode(FirstByte, vmCode); } - private bool AddVMCode(int firstByte, List vmCode, int length) + private bool AddVMCode(int firstByte, List vmCode) { - var Inp = new BitInput(); + using var Inp = new BitInput(); Inp.InitBitInput(); // memcpy(Inp.InBuf,Code,Min(BitInput::MAX_SIZE,CodeSize)); @@ -1077,7 +1161,6 @@ internal sealed partial class Unpack : BitInput, IRarUnpack { Inp.InBuf[i] = vmCode[i]; } - rarVM.init(); int FiltPos; if ((firstByte & 0x80) != 0) @@ -1101,6 +1184,7 @@ internal sealed partial class Unpack : BitInput, IRarUnpack { return (false); } + lastFilter = FiltPos; var NewFilter = (FiltPos == filters.Count); @@ -1141,6 +1225,7 @@ internal sealed partial class Unpack : BitInput, IRarUnpack { BlockStart += 258; } + StackFilter.BlockStart = ((BlockStart + unpPtr) & PackDef.MAXWINMASK); if ((firstByte & 0x20) != 0) { @@ -1151,6 +1236,7 @@ internal sealed partial class Unpack : BitInput, IRarUnpack StackFilter.BlockLength = FiltPos < oldFilterLengths.Count ? oldFilterLengths[FiltPos] : 0; } + StackFilter.NextWindow = (wrPtr != unpPtr) && ((wrPtr - unpPtr) & PackDef.MAXWINMASK) <= BlockStart; @@ -1190,20 +1276,30 @@ internal sealed partial class Unpack : BitInput, IRarUnpack { return (false); } - var VMCode = new byte[VMCodeSize]; - for (var I = 0; I < VMCodeSize; I++) - { - if (Inp.Overflow(3)) - { - return (false); - } - VMCode[I] = (byte)(Inp.GetBits() >> 8); - Inp.AddBits(8); - } - // VM.Prepare(&VMCode[0],VMCodeSize,&Filter->Prg); - rarVM.prepare(VMCode, VMCodeSize, Filter.Program); + var VMCode = ArrayPool.Shared.Rent(VMCodeSize); + try + { + for (var I = 0; I < VMCodeSize; I++) + { + if (Inp.Overflow(3)) + { + return (false); + } + + VMCode[I] = (byte)(Inp.GetBits() >> 8); + Inp.AddBits(8); + } + + // VM.Prepare(&VMCode[0],VMCodeSize,&Filter->Prg); + rarVM.prepare(VMCode.AsSpan(0, VMCodeSize), Filter.Program); + } + finally + { + ArrayPool.Shared.Return(VMCode); + } } + StackFilter.Program.AltCommands = Filter.Program.Commands; // StackFilter->Prg.AltCmd=&Filter->Prg.Cmd[0]; StackFilter.Program.CommandCount = Filter.Program.CommandCount; @@ -1253,6 +1349,7 @@ internal sealed partial class Unpack : BitInput, IRarUnpack { globalData[0x30 + i] = 0x0; } + if ((firstByte & 8) != 0) // put data block passed as parameter if any { @@ -1260,11 +1357,13 @@ internal sealed partial class Unpack : BitInput, IRarUnpack { return (false); } + var DataSize = RarVM.ReadData(Inp); if (DataSize > RarVM.VM_GLOBALMEMSIZE - RarVM.VM_FIXEDGLOBALSIZE) { return (false); } + var CurSize = StackFilter.Program.GlobalData.Count; if (CurSize < DataSize + RarVM.VM_FIXEDGLOBALSIZE) { @@ -1273,6 +1372,7 @@ internal sealed partial class Unpack : BitInput, IRarUnpack DataSize + RarVM.VM_FIXEDGLOBALSIZE - CurSize ); } + var offset = RarVM.VM_FIXEDGLOBALSIZE; globalData = StackFilter.Program.GlobalData; for (var I = 0; I < DataSize; I++) @@ -1281,10 +1381,12 @@ internal sealed partial class Unpack : BitInput, IRarUnpack { return (false); } + globalData[offset + I] = (byte)(Utility.URShift(Inp.GetBits(), 8)); Inp.AddBits(8); } } + return (true); } @@ -1312,10 +1414,6 @@ internal sealed partial class Unpack : BitInput, IRarUnpack private void CleanUp() { - if (ppm != null) - { - var allocator = ppm.SubAlloc; - allocator?.StopSubAllocator(); - } + ppm.Dispose(); } } diff --git a/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack15.Async.cs b/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack15.Async.cs new file mode 100644 index 00000000..e94e7ba7 --- /dev/null +++ b/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack15.Async.cs @@ -0,0 +1,162 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Compressors.Rar.UnpackV1.Decode; + +namespace SharpCompress.Compressors.Rar.UnpackV1; + +internal partial class Unpack +{ + private async ValueTask unpack15Async(bool solid, CancellationToken cancellationToken = default) + { + if (suspended) + { + unpPtr = wrPtr; + } + else + { + UnpInitData(solid); + oldUnpInitData(solid); + await unpReadBufAsync(cancellationToken).ConfigureAwait(false); + if (!solid) + { + initHuff(); + unpPtr = 0; + } + else + { + unpPtr = wrPtr; + } + --destUnpSize; + } + if (destUnpSize >= 0) + { + getFlagsBuf(); + FlagsCnt = 8; + } + + while (destUnpSize >= 0) + { + unpPtr &= PackDef.MAXWINMASK; + + if ( + inAddr > readTop - 30 + && !await unpReadBufAsync(cancellationToken).ConfigureAwait(false) + ) + { + break; + } + if (((wrPtr - unpPtr) & PackDef.MAXWINMASK) < 270 && wrPtr != unpPtr) + { + await oldUnpWriteBufAsync(cancellationToken).ConfigureAwait(false); + if (suspended) + { + return; + } + } + if (StMode != 0) + { + huffDecode(); + continue; + } + + if (--FlagsCnt < 0) + { + getFlagsBuf(); + FlagsCnt = 7; + } + + if ((FlagBuf & 0x80) != 0) + { + FlagBuf <<= 1; + if (Nlzb > Nhfb) + { + longLZ(); + } + else + { + huffDecode(); + } + } + else + { + FlagBuf <<= 1; + if (--FlagsCnt < 0) + { + getFlagsBuf(); + FlagsCnt = 7; + } + if ((FlagBuf & 0x80) != 0) + { + FlagBuf <<= 1; + if (Nlzb > Nhfb) + { + huffDecode(); + } + else + { + longLZ(); + } + } + else + { + FlagBuf <<= 1; + shortLZ(); + } + } + } + await oldUnpWriteBufAsync(cancellationToken).ConfigureAwait(false); + } + + private async ValueTask unpReadBufAsync(CancellationToken cancellationToken = default) + { + var dataSize = readTop - inAddr; + if (dataSize < 0) + { + return false; + } + if (inAddr > MAX_SIZE / 2) + { + if (dataSize > 0) + { + Array.Copy(InBuf, inAddr, InBuf, 0, dataSize); + } + inAddr = 0; + readTop = dataSize; + } + else + { + dataSize = readTop; + } + + var readCode = await readStream + .ReadAsync(InBuf, dataSize, (MAX_SIZE - dataSize) & ~0xf, cancellationToken) + .ConfigureAwait(false); + if (readCode > 0) + { + readTop += readCode; + } + readBorder = readTop - 30; + return readCode != -1; + } + + private async ValueTask oldUnpWriteBufAsync(CancellationToken cancellationToken = default) + { + if (unpPtr < wrPtr) + { + await writeStream + .WriteAsync(window, wrPtr, -wrPtr & PackDef.MAXWINMASK, cancellationToken) + .ConfigureAwait(false); + await writeStream + .WriteAsync(window, 0, unpPtr, cancellationToken) + .ConfigureAwait(false); + } + else + { + await writeStream + .WriteAsync(window, wrPtr, unpPtr - wrPtr, cancellationToken) + .ConfigureAwait(false); + } + wrPtr = unpPtr; + } +} diff --git a/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack15.cs b/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack15.cs index 8513c24b..96c9ac6f 100644 --- a/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack15.cs +++ b/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack15.cs @@ -19,14 +19,9 @@ internal partial class Unpack private bool suspended; - internal bool unpAllBuf; - //private ComprDataIO unpIO; private Stream readStream; private Stream writeStream; - - internal bool unpSomeRead; - private int readTop; private long destUnpSize; @@ -88,7 +83,7 @@ internal partial class Unpack 0xf000, 0xf200, 0xf200, - 0xffff + 0xffff, }; private static readonly int[] PosL1 = { 0, 0, 0, 2, 3, 5, 7, 11, 16, 20, 24, 32, 32 }; @@ -106,7 +101,7 @@ internal partial class Unpack 0xf000, 0xf200, 0xf240, - 0xffff + 0xffff, }; private static readonly int[] PosL2 = { 0, 0, 0, 0, 5, 7, 9, 13, 18, 22, 26, 34, 36 }; @@ -123,7 +118,7 @@ internal partial class Unpack 0xf200, 0xf200, 0xf200, - 0xffff + 0xffff, }; private static readonly int[] PosHf0 = { 0, 0, 0, 0, 0, 8, 16, 24, 33, 33, 33, 33, 33 }; @@ -139,7 +134,7 @@ internal partial class Unpack 0xf200, 0xf200, 0xf7e0, - 0xffff + 0xffff, }; private static readonly int[] PosHf1 = { 0, 0, 0, 0, 0, 0, 4, 44, 60, 76, 80, 80, 127 }; @@ -155,7 +150,7 @@ internal partial class Unpack 0xfa00, 0xffff, 0xffff, - 0xffff + 0xffff, }; private static readonly int[] PosHf2 = { 0, 0, 0, 0, 0, 0, 2, 7, 53, 117, 233, 0, 0 }; @@ -170,7 +165,7 @@ internal partial class Unpack 0xfe80, 0xffff, 0xffff, - 0xffff + 0xffff, }; private static readonly int[] PosHf3 = { 0, 0, 0, 0, 0, 0, 0, 2, 16, 218, 251, 0, 0 }; @@ -199,7 +194,7 @@ internal partial class Unpack 0x90, 0x98, 0x9c, - 0xb0 + 0xb0, }; private static readonly int[] ShortLen2 = { 2, 3, 3, 3, 4, 4, 5, 6, 6, 4, 4, 5, 6, 6, 4, 0 }; @@ -220,7 +215,7 @@ internal partial class Unpack 0x90, 0x98, 0x9c, - 0xb0 + 0xb0, }; private void unpack15(bool solid) @@ -808,15 +803,10 @@ internal partial class Unpack private void oldUnpWriteBuf() { - if (unpPtr != wrPtr) - { - unpSomeRead = true; - } if (unpPtr < wrPtr) { writeStream.Write(window, wrPtr, -wrPtr & PackDef.MAXWINMASK); writeStream.Write(window, 0, unpPtr); - unpAllBuf = true; } else { diff --git a/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack20.Async.cs b/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack20.Async.cs new file mode 100644 index 00000000..f29f10e5 --- /dev/null +++ b/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack20.Async.cs @@ -0,0 +1,296 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Compressors.Rar.UnpackV1.Decode; + +namespace SharpCompress.Compressors.Rar.UnpackV1; + +internal partial class Unpack +{ + private async ValueTask unpack20Async(bool solid, CancellationToken cancellationToken = default) + { + int Bits; + + if (suspended) + { + unpPtr = wrPtr; + } + else + { + UnpInitData(solid); + if (!await unpReadBufAsync(cancellationToken).ConfigureAwait(false)) + { + return; + } + if (!solid) + { + if (!await ReadTables20Async(cancellationToken).ConfigureAwait(false)) + { + return; + } + } + --destUnpSize; + } + + while (destUnpSize >= 0) + { + unpPtr &= PackDef.MAXWINMASK; + + if (inAddr > readTop - 30) + { + if (!await unpReadBufAsync(cancellationToken).ConfigureAwait(false)) + { + break; + } + } + if (((wrPtr - unpPtr) & PackDef.MAXWINMASK) < 270 && wrPtr != unpPtr) + { + await oldUnpWriteBufAsync(cancellationToken).ConfigureAwait(false); + if (suspended) + { + return; + } + } + if (UnpAudioBlock != 0) + { + var AudioNumber = this.decodeNumber(MD[UnpCurChannel]); + + if (AudioNumber == 256) + { + if (!await ReadTables20Async(cancellationToken).ConfigureAwait(false)) + { + break; + } + continue; + } + window[unpPtr++] = DecodeAudio(AudioNumber); + if (++UnpCurChannel == UnpChannels) + { + UnpCurChannel = 0; + } + --destUnpSize; + continue; + } + + var Number = this.decodeNumber(LD); + if (Number < 256) + { + window[unpPtr++] = (byte)Number; + --destUnpSize; + continue; + } + if (Number > 269) + { + var Length = LDecode[Number -= 270] + 3; + if ((Bits = LBits[Number]) > 0) + { + Length += Utility.URShift(GetBits(), (16 - Bits)); + AddBits(Bits); + } + + var DistNumber = this.decodeNumber(DD); + var Distance = DDecode[DistNumber] + 1; + if ((Bits = DBits[DistNumber]) > 0) + { + Distance += Utility.URShift(GetBits(), (16 - Bits)); + AddBits(Bits); + } + + if (Distance >= 0x2000) + { + Length++; + if (Distance >= 0x40000L) + { + Length++; + } + } + + CopyString20(Length, Distance); + continue; + } + if (Number == 269) + { + if (!await ReadTables20Async(cancellationToken).ConfigureAwait(false)) + { + break; + } + continue; + } + if (Number == 256) + { + CopyString20(lastLength, lastDist); + continue; + } + if (Number < 261) + { + var Distance = oldDist[(oldDistPtr - (Number - 256)) & 3]; + var LengthNumber = this.decodeNumber(RD); + var Length = LDecode[LengthNumber] + 2; + if ((Bits = LBits[LengthNumber]) > 0) + { + Length += Utility.URShift(GetBits(), (16 - Bits)); + AddBits(Bits); + } + if (Distance >= 0x101) + { + Length++; + if (Distance >= 0x2000) + { + Length++; + if (Distance >= 0x40000) + { + Length++; + } + } + } + CopyString20(Length, Distance); + continue; + } + if (Number < 270) + { + var Distance = SDDecode[Number -= 261] + 1; + if ((Bits = SDBits[Number]) > 0) + { + Distance += Utility.URShift(GetBits(), (16 - Bits)); + AddBits(Bits); + } + CopyString20(2, Distance); + } + } + await ReadLastTablesAsync(cancellationToken).ConfigureAwait(false); + await oldUnpWriteBufAsync(cancellationToken).ConfigureAwait(false); + } + + private async ValueTask ReadTables20Async(CancellationToken cancellationToken = default) + { + byte[] BitLength = new byte[PackDef.BC20]; + byte[] Table = new byte[PackDef.MC20 * 4]; + int TableSize, + N, + I; + if (inAddr > readTop - 25) + { + if (!await unpReadBufAsync(cancellationToken).ConfigureAwait(false)) + { + return false; + } + } + var BitField = GetBits(); + UnpAudioBlock = (BitField & 0x8000); + + if (0 == (BitField & 0x4000)) + { + new Span(UnpOldTable20).Clear(); + } + AddBits(2); + + if (UnpAudioBlock != 0) + { + UnpChannels = ((Utility.URShift(BitField, 12)) & 3) + 1; + if (UnpCurChannel >= UnpChannels) + { + UnpCurChannel = 0; + } + AddBits(2); + TableSize = PackDef.MC20 * UnpChannels; + } + else + { + TableSize = PackDef.NC20 + PackDef.DC20 + PackDef.RC20; + } + for (I = 0; I < PackDef.BC20; I++) + { + BitLength[I] = (byte)(Utility.URShift(GetBits(), 12)); + AddBits(4); + } + UnpackUtility.makeDecodeTables(BitLength, 0, BD, PackDef.BC20); + I = 0; + while (I < TableSize) + { + if (inAddr > readTop - 5) + { + if (!await unpReadBufAsync(cancellationToken).ConfigureAwait(false)) + { + return false; + } + } + var Number = this.decodeNumber(BD); + if (Number < 16) + { + Table[I] = (byte)((Number + UnpOldTable20[I]) & 0xf); + I++; + } + else if (Number == 16) + { + N = (Utility.URShift(GetBits(), 14)) + 3; + AddBits(2); + while (N-- > 0 && I < TableSize) + { + Table[I] = Table[I - 1]; + I++; + } + } + else + { + if (Number == 17) + { + N = (Utility.URShift(GetBits(), 13)) + 3; + AddBits(3); + } + else + { + N = (Utility.URShift(GetBits(), 9)) + 11; + AddBits(7); + } + while (N-- > 0 && I < TableSize) + { + Table[I++] = 0; + } + } + } + if (inAddr > readTop) + { + return true; + } + if (UnpAudioBlock != 0) + { + for (I = 0; I < UnpChannels; I++) + { + UnpackUtility.makeDecodeTables(Table, I * PackDef.MC20, MD[I], PackDef.MC20); + } + } + else + { + UnpackUtility.makeDecodeTables(Table, 0, LD, PackDef.NC20); + UnpackUtility.makeDecodeTables(Table, PackDef.NC20, DD, PackDef.DC20); + UnpackUtility.makeDecodeTables(Table, PackDef.NC20 + PackDef.DC20, RD, PackDef.RC20); + } + + for (var i = 0; i < UnpOldTable20.Length; i++) + { + UnpOldTable20[i] = Table[i]; + } + return true; + } + + private async ValueTask ReadLastTablesAsync(CancellationToken cancellationToken = default) + { + if (readTop >= inAddr + 5) + { + if (UnpAudioBlock != 0) + { + if (this.decodeNumber(MD[UnpCurChannel]) == 256) + { + await ReadTables20Async(cancellationToken).ConfigureAwait(false); + } + } + else + { + if (this.decodeNumber(LD) == 269) + { + await ReadTables20Async(cancellationToken).ConfigureAwait(false); + } + } + } + } +} diff --git a/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack20.cs b/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack20.cs index 15b980da..69a266bb 100644 --- a/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack20.cs +++ b/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack20.cs @@ -14,7 +14,13 @@ namespace SharpCompress.Compressors.Rar.UnpackV1; internal partial class Unpack { - private readonly MultDecode[] MD = new MultDecode[4]; + private readonly MultDecode[] MD = new[] + { + new MultDecode(), + new MultDecode(), + new MultDecode(), + new MultDecode(), + }; private readonly byte[] UnpOldTable20 = new byte[PackDef.MC20 * 4]; @@ -25,15 +31,15 @@ internal partial class Unpack private readonly AudioVariables[] AudV = new AudioVariables[4]; - private readonly LitDecode LD = new LitDecode(); + private readonly LitDecode LD = new(); - private readonly DistDecode DD = new DistDecode(); + private readonly DistDecode DD = new(); - private readonly LowDistDecode LDD = new LowDistDecode(); + private readonly LowDistDecode LDD = new(); - private readonly RepDecode RD = new RepDecode(); + private readonly RepDecode RD = new(); - private readonly BitDecode BD = new BitDecode(); + private readonly BitDecode BD = new(); private static readonly int[] LDecode = { @@ -64,7 +70,7 @@ internal partial class Unpack 128, 160, 192, - 224 + 224, }; private static ReadOnlySpan LBits => @@ -97,7 +103,7 @@ internal partial class Unpack 5, 5, 5, - 5 + 5, }; private static readonly int[] DDecode = @@ -149,7 +155,7 @@ internal partial class Unpack 786432, 851968, 917504, - 983040 + 983040, }; private static readonly int[] DBits = @@ -201,7 +207,7 @@ internal partial class Unpack 16, 16, 16, - 16 + 16, }; private static readonly int[] SDDecode = { 0, 4, 8, 16, 32, 64, 128, 192 }; @@ -369,7 +375,7 @@ internal partial class Unpack destUnpSize -= Length; var DestPtr = unpPtr - Distance; - if (DestPtr < PackDef.MAXWINSIZE - 300 && unpPtr < PackDef.MAXWINSIZE - 300) + if (DestPtr >= 0 && DestPtr < PackDef.MAXWINSIZE - 300 && unpPtr < PackDef.MAXWINSIZE - 300) { window[unpPtr++] = window[DestPtr++]; window[unpPtr++] = window[DestPtr++]; @@ -391,8 +397,8 @@ internal partial class Unpack private bool ReadTables20() { - var BitLength = new byte[PackDef.BC20]; - var Table = new byte[PackDef.MC20 * 4]; + Span BitLength = stackalloc byte[PackDef.BC20]; + Span Table = stackalloc byte[PackDef.MC20 * 4]; int TableSize, N, I; @@ -475,6 +481,31 @@ internal partial class Unpack { Table[I++] = 0; } + // Nanook. Working port from Rar C code. Added when working on Audio Decode Fix. Seems equal to above, so commented it + //byte v; + //if (Number == 16) + //{ + // N = (Utility.URShift(GetBits(), 14)) + 3; + // AddBits(2); + // v = Table[I - 1]; + //} + //else + //{ + // N = (Number - 17) * 4; + // int bits = 3 + N; + // N += N + 3 + (Utility.URShift(GetBits(), 16 - bits)); + // AddBits(bits); + // v = 0; + //} + //N += I; + //if (N > TableSize) + //{ + // N = TableSize; // original unRAR + //} + //do + //{ + // Table[I++] = v; + //} while (I < N); } } if (inAddr > readTop) @@ -559,8 +590,7 @@ internal partial class Unpack PCh = (Utility.URShift(PCh, 3)) & 0xFF; var Ch = PCh - Delta; - - var D = ((byte)Delta) << 3; + var D = ((sbyte)Delta) << 3; v.Dif[0] += Math.Abs(D); // V->Dif[0]+=abs(D); v.Dif[1] += Math.Abs(D - v.D1); // V->Dif[1]+=abs(D-V->D1); @@ -574,7 +604,7 @@ internal partial class Unpack v.Dif[9] += Math.Abs(D - UnpChannelDelta); // V->Dif[9]+=abs(D-UnpChannelDelta); v.Dif[10] += Math.Abs(D + UnpChannelDelta); // V->Dif[10]+=abs(D+UnpChannelDelta); - v.LastDelta = (byte)(Ch - v.LastChar); + v.LastDelta = (sbyte)(Ch - v.LastChar); UnpChannelDelta = v.LastDelta; v.LastChar = Ch; // V->LastChar=Ch; diff --git a/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack50.Async.cs b/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack50.Async.cs new file mode 100644 index 00000000..43536225 --- /dev/null +++ b/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack50.Async.cs @@ -0,0 +1,343 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Compressors.Rar.UnpackV1.Decode; + +namespace SharpCompress.Compressors.Rar.UnpackV1; + +internal partial class Unpack +{ + private async ValueTask UnpReadBufAsync(CancellationToken cancellationToken = default) + { + var DataSize = ReadTop - Inp.InAddr; // Data left to process. + if (DataSize < 0) + { + return false; + } + + BlockHeader.BlockSize -= Inp.InAddr - BlockHeader.BlockStart; + if (Inp.InAddr > MAX_SIZE / 2) + { + if (DataSize > 0) + { + Array.Copy(InBuf, inAddr, InBuf, 0, DataSize); + } + + Inp.InAddr = 0; + ReadTop = DataSize; + } + else + { + DataSize = ReadTop; + } + + var ReadCode = 0; + if (MAX_SIZE != DataSize) + { + ReadCode = await readStream + .ReadAsync(InBuf, DataSize, MAX_SIZE - DataSize, cancellationToken) + .ConfigureAwait(false); + } + + if (ReadCode > 0) // Can be also -1. + { + ReadTop += ReadCode; + } + + ReadBorder = ReadTop - 30; + BlockHeader.BlockStart = Inp.InAddr; + if (BlockHeader.BlockSize != -1) // '-1' means not defined yet. + { + ReadBorder = Math.Min(ReadBorder, BlockHeader.BlockStart + BlockHeader.BlockSize - 1); + } + return ReadCode != -1; + } + + public async ValueTask Unpack5Async(bool Solid, CancellationToken cancellationToken = default) + { + FileExtracted = true; + + if (!Suspended) + { + UnpInitData(Solid); + if (!await UnpReadBufAsync(cancellationToken).ConfigureAwait(false)) + { + return; + } + + // Check TablesRead5 to be sure that we read tables at least once + // regardless of current block header TablePresent flag. + // So we can safefly use these tables below. + if ( + !await ReadBlockHeaderAsync(cancellationToken).ConfigureAwait(false) + || !await ReadTablesAsync(cancellationToken).ConfigureAwait(false) + || !TablesRead5 + ) + { + return; + } + } + + while (true) + { + UnpPtr &= MaxWinMask; + + if (Inp.InAddr >= ReadBorder) + { + var FileDone = false; + + // We use 'while', because for empty block containing only Huffman table, + // we'll be on the block border once again just after reading the table. + while ( + Inp.InAddr > BlockHeader.BlockStart + BlockHeader.BlockSize - 1 + || Inp.InAddr == BlockHeader.BlockStart + BlockHeader.BlockSize - 1 + && Inp.InBit >= BlockHeader.BlockBitSize + ) + { + if (BlockHeader.LastBlockInFile) + { + FileDone = true; + break; + } + if ( + !await ReadBlockHeaderAsync(cancellationToken).ConfigureAwait(false) + || !await ReadTablesAsync(cancellationToken).ConfigureAwait(false) + ) + { + return; + } + } + if (FileDone || !await UnpReadBufAsync(cancellationToken).ConfigureAwait(false)) + { + break; + } + } + + if ( + ((WriteBorder - UnpPtr) & MaxWinMask) < PackDef.MAX_LZ_MATCH + 3 + && WriteBorder != UnpPtr + ) + { + await UnpWriteBufAsync(cancellationToken).ConfigureAwait(false); + if (WrittenFileSize > DestUnpSize) + { + return; + } + + if (Suspended) + { + FileExtracted = false; + return; + } + } + + var MainSlot = this.DecodeNumber(LD); + if (MainSlot < 256) + { + Window[UnpPtr++] = (byte)MainSlot; + continue; + } + if (MainSlot >= 262) + { + var Length = SlotToLength(MainSlot - 262); + + int DBits; + uint Distance = 1, + DistSlot = this.DecodeNumber(DD); + if (DistSlot < 4) + { + DBits = 0; + Distance += DistSlot; + } + else + { + DBits = (int)((DistSlot / 2) - 1); + Distance += (2 | (DistSlot & 1)) << DBits; + } + + if (DBits > 0) + { + if (DBits >= 4) + { + if (DBits > 4) + { + Distance += ((Inp.getbits() >> (36 - DBits)) << 4); + Inp.AddBits(DBits - 4); + } + var LowDist = this.DecodeNumber(LDD); + Distance += LowDist; + } + else + { + Distance += Inp.getbits() >> (32 - DBits); + Inp.AddBits(DBits); + } + } + + if (Distance > 0x100) + { + Length++; + if (Distance > 0x2000) + { + Length++; + if (Distance > 0x40000) + { + Length++; + } + } + } + + InsertOldDist(Distance); + LastLength = Length; + CopyString(Length, Distance); + continue; + } + if (MainSlot == 256) + { + var Filter = new UnpackFilter(); + if ( + !await ReadFilterAsync(Filter, cancellationToken).ConfigureAwait(false) + || !await AddFilterAsync(Filter, cancellationToken).ConfigureAwait(false) + ) + { + break; + } + + continue; + } + if (MainSlot == 257) + { + if (LastLength != 0) + { + CopyString(LastLength, OldDistN(0)); + } + + continue; + } + if (MainSlot < 262) + { + var DistNum = (int)(MainSlot - 258); + var Distance = OldDistN(DistNum); + for (var I = DistNum; I > 0; I--) + { + SetOldDistN(I, OldDistN(I - 1)); + } + + SetOldDistN(0, Distance); + + var LengthSlot = this.DecodeNumber(RD); + var Length = SlotToLength(LengthSlot); + LastLength = Length; + CopyString(Length, Distance); + continue; + } + } + await UnpWriteBufAsync(cancellationToken).ConfigureAwait(false); + } + + private async ValueTask ReadBlockHeaderAsync( + CancellationToken cancellationToken = default + ) + { + Header.HeaderSize = 0; + + if (Inp.InAddr > ReadTop - 7) + { + if (!await UnpReadBufAsync(cancellationToken).ConfigureAwait(false)) + { + return false; + } + } + + Inp.faddbits((uint)((8 - Inp.InBit) & 7)); + + var BlockFlags = (byte)(Inp.fgetbits() >> 8); + Inp.faddbits(8); + var ByteCount = (uint)(((BlockFlags >> 3) & 3) + 1); + + if (ByteCount == 4) + { + return false; + } + + Header.HeaderSize = (int)(2 + ByteCount); + + Header.BlockBitSize = (BlockFlags & 7) + 1; + + var SavedCheckSum = (byte)(Inp.fgetbits() >> 8); + Inp.faddbits(8); + + var BlockSize = 0; + for (var I = 0; I < ByteCount; I++) + { + BlockSize += (int)(Inp.fgetbits() >> 8) << (I * 8); + Inp.AddBits(8); + } + + Header.BlockSize = BlockSize; + var CheckSum = (byte)(0x5a ^ BlockFlags ^ BlockSize ^ (BlockSize >> 8) ^ (BlockSize >> 16)); + if (CheckSum != SavedCheckSum) + { + return false; + } + + Header.BlockStart = Inp.InAddr; + ReadBorder = Math.Min(ReadBorder, Header.BlockStart + Header.BlockSize - 1); + + Header.LastBlockInFile = (BlockFlags & 0x40) != 0; + Header.TablePresent = (BlockFlags & 0x80) != 0; + return true; + } + + private async ValueTask ReadFilterAsync( + UnpackFilter Filter, + CancellationToken cancellationToken = default + ) + { + if (Inp.InAddr > ReadTop - 16) + { + if (!await UnpReadBufAsync(cancellationToken).ConfigureAwait(false)) + { + return false; + } + } + + Filter.uBlockStart = ReadFilterData(); + Filter.uBlockLength = ReadFilterData(); + if (Filter.BlockLength > MAX_FILTER_BLOCK_SIZE) + { + Filter.BlockLength = 0; + } + + Filter.Type = (byte)(Inp.fgetbits() >> 13); + Inp.faddbits(3); + + if (Filter.Type == (byte)FilterType.FILTER_DELTA) + { + Filter.Channels = (byte)((Inp.fgetbits() >> 11) + 1); + Inp.faddbits(5); + } + + return true; + } + + private async ValueTask AddFilterAsync( + UnpackFilter Filter, + CancellationToken cancellationToken = default + ) + { + if (Filters.Count >= MAX_UNPACK_FILTERS) + { + await UnpWriteBufAsync(cancellationToken).ConfigureAwait(false); + if (Filters.Count >= MAX_UNPACK_FILTERS) + { + InitFilters(); + } + } + + Filter.NextWindow = WrPtr != UnpPtr && ((WrPtr - UnpPtr) & MaxWinMask) <= Filter.BlockStart; + Filter.uBlockStart = (uint)((Filter.BlockStart + UnpPtr) & MaxWinMask); + Filters.Add(Filter); + return true; + } +} diff --git a/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack50.cs b/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack50.cs index bebaebee..496b6204 100644 --- a/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack50.cs +++ b/src/SharpCompress/Compressors/Rar/UnpackV1/Unpack50.cs @@ -372,7 +372,7 @@ internal partial class Unpack private bool ReadFilter(UnpackFilter Filter) { - if (!Inp.ExternalBuffer && Inp.InAddr > ReadTop - 16) + if (Inp.InAddr > ReadTop - 16) { if (!UnpReadBuf()) { @@ -622,7 +622,6 @@ internal partial class Unpack // WriteBorder=WrPtr; // } - // unused //x byte* ApplyFilter(byte *Data,uint DataSize,UnpackFilter *Flt) // byte[] ApplyFilter(byte []Data, uint DataSize, UnpackFilter Flt) @@ -763,7 +762,7 @@ internal partial class Unpack { Header.HeaderSize = 0; - if (!Inp.ExternalBuffer && Inp.InAddr > ReadTop - 7) + if (Inp.InAddr > ReadTop - 7) { if (!UnpReadBuf()) { @@ -815,116 +814,5 @@ internal partial class Unpack Header.TablePresent = (BlockFlags & 0x80) != 0; return true; } - - //? - // bool ReadTables(BitInput Inp, ref UnpackBlockHeader Header, ref UnpackBlockTables Tables) - // { - // if (!Header.TablePresent) - // return true; - // - // if (!Inp.ExternalBuffer && Inp.InAddr>ReadTop-25) - // if (!UnpReadBuf()) - // return false; - // - // byte BitLength[BC]; - // for (uint I=0;I> 12); - // Inp.faddbits(4); - // if (Length==15) - // { - // uint ZeroCount=(byte)(Inp.fgetbits() >> 12); - // Inp.faddbits(4); - // if (ZeroCount==0) - // BitLength[I]=15; - // else - // { - // ZeroCount+=2; - // while (ZeroCount-- > 0 && IReadTop-5) - // if (!UnpReadBuf()) - // return false; - // uint Number=DecodeNumber(Inp,&Tables.BD); - // if (Number<16) - // { - // Table[I]=Number; - // I++; - // } - // else - // if (Number<18) - // { - // uint N; - // if (Number==16) - // { - // N=(Inp.fgetbits() >> 13)+3; - // Inp.faddbits(3); - // } - // else - // { - // N=(Inp.fgetbits() >> 9)+11; - // Inp.faddbits(7); - // } - // if (I==0) - // { - // // We cannot have "repeat previous" code at the first position. - // // Multiple such codes would shift Inp position without changing I, - // // which can lead to reading beyond of Inp boundary in mutithreading - // // mode, where Inp.ExternalBuffer disables bounds check and we just - // // reserve a lot of buffer space to not need such check normally. - // return false; - // } - // else - // while (N-- > 0 && I> 13)+3; - // Inp.faddbits(3); - // } - // else - // { - // N=(Inp.fgetbits() >> 9)+11; - // Inp.faddbits(7); - // } - // while (N-- > 0 && IReadTop) - // return false; - // MakeDecodeTables(&Table[0],&Tables.LD,NC); - // MakeDecodeTables(&Table[NC],&Tables.DD,DC); - // MakeDecodeTables(&Table[NC+DC],&Tables.LDD,LDC); - // MakeDecodeTables(&Table[NC+DC+LDC],&Tables.RD,RC); - // return true; - // } - - //? - // void InitFilters() - // { - // Filters.SoftReset(); - // } } #endif diff --git a/src/SharpCompress/Compressors/Rar/UnpackV1/UnpackUtility.cs b/src/SharpCompress/Compressors/Rar/UnpackV1/UnpackUtility.cs index 09f5ee8f..c2e49dbc 100644 --- a/src/SharpCompress/Compressors/Rar/UnpackV1/UnpackUtility.cs +++ b/src/SharpCompress/Compressors/Rar/UnpackV1/UnpackUtility.cs @@ -1,5 +1,5 @@ using System; - +using System.Runtime.CompilerServices; using SharpCompress.Compressors.Rar.VM; namespace SharpCompress.Compressors.Rar.UnpackV1; @@ -10,167 +10,15 @@ internal static class UnpackUtility internal static uint DecodeNumber(this BitInput input, Decode.Decode dec) => (uint)input.decodeNumber(dec); + [MethodImpl(MethodImplOptions.AggressiveInlining)] internal static int decodeNumber(this BitInput input, Decode.Decode dec) { - int bits; long bitField = input.GetBits() & 0xfffe; - - // if (bitField < dec.getDecodeLen()[8]) { - // if (bitField < dec.getDecodeLen()[4]) { - // if (bitField < dec.getDecodeLen()[2]) { - // if (bitField < dec.getDecodeLen()[1]) { - // bits = 1; - // } else { - // bits = 2; - // } - // } else { - // if (bitField < dec.getDecodeLen()[3]) { - // bits = 3; - // } else { - // bits = 4; - // } - // } - // } else { - // if (bitField < dec.getDecodeLen()[6]) { - // if (bitField < dec.getDecodeLen()[5]) - // bits = 5; - // else - // bits = 6; - // } else { - // if (bitField < dec.getDecodeLen()[7]) { - // bits = 7; - // } else { - // bits = 8; - // } - // } - // } - // } else { - // if (bitField < dec.getDecodeLen()[12]) { - // if (bitField < dec.getDecodeLen()[10]) - // if (bitField < dec.getDecodeLen()[9]) - // bits = 9; - // else - // bits = 10; - // else if (bitField < dec.getDecodeLen()[11]) - // bits = 11; - // else - // bits = 12; - // } else { - // if (bitField < dec.getDecodeLen()[14]) { - // if (bitField < dec.getDecodeLen()[13]) { - // bits = 13; - // } else { - // bits = 14; - // } - // } else { - // bits = 15; - // } - // } - // } - // addbits(bits); - // int N = dec.getDecodePos()[bits] - // + (((int) bitField - dec.getDecodeLen()[bits - 1]) >>> (16 - bits)); - // if (N >= dec.getMaxNum()) { - // N = 0; - // } - // return (dec.getDecodeNum()[N]); var decodeLen = dec.DecodeLen; - if (bitField < decodeLen[8]) - { - if (bitField < decodeLen[4]) - { - if (bitField < decodeLen[2]) - { - if (bitField < decodeLen[1]) - { - bits = 1; - } - else - { - bits = 2; - } - } - else - { - if (bitField < decodeLen[3]) - { - bits = 3; - } - else - { - bits = 4; - } - } - } - else - { - if (bitField < decodeLen[6]) - { - if (bitField < decodeLen[5]) - { - bits = 5; - } - else - { - bits = 6; - } - } - else - { - if (bitField < decodeLen[7]) - { - bits = 7; - } - else - { - bits = 8; - } - } - } - } - else - { - if (bitField < decodeLen[12]) - { - if (bitField < decodeLen[10]) - { - if (bitField < decodeLen[9]) - { - bits = 9; - } - else - { - bits = 10; - } - } - else if (bitField < decodeLen[11]) - { - bits = 11; - } - else - { - bits = 12; - } - } - else - { - if (bitField < decodeLen[14]) - { - if (bitField < decodeLen[13]) - { - bits = 13; - } - else - { - bits = 14; - } - } - else - { - bits = 15; - } - } - } + + // Binary search to find the bit length - faster than nested ifs + int bits = FindDecodeBits(bitField, decodeLen); + input.AddBits(bits); var N = dec.DecodePos[bits] @@ -182,7 +30,58 @@ internal static class UnpackUtility return (dec.DecodeNum[N]); } - internal static void makeDecodeTables(byte[] lenTab, int offset, Decode.Decode dec, int size) + /// + /// Fast binary search to find which bit length matches the bitField. + /// Optimized with cached array access to minimize memory lookups. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static int FindDecodeBits(long bitField, int[] decodeLen) + { + // Cache critical values to reduce array access overhead + long len4 = decodeLen[4]; + long len8 = decodeLen[8]; + long len12 = decodeLen[12]; + + if (bitField < len8) + { + if (bitField < len4) + { + long len2 = decodeLen[2]; + if (bitField < len2) + { + return bitField < decodeLen[1] ? 1 : 2; + } + return bitField < decodeLen[3] ? 3 : 4; + } + + long len6 = decodeLen[6]; + if (bitField < len6) + { + return bitField < decodeLen[5] ? 5 : 6; + } + return bitField < decodeLen[7] ? 7 : 8; + } + + if (bitField < len12) + { + long len10 = decodeLen[10]; + if (bitField < len10) + { + return bitField < decodeLen[9] ? 9 : 10; + } + return bitField < decodeLen[11] ? 11 : 12; + } + + long len14 = decodeLen[14]; + return bitField < len14 ? (bitField < decodeLen[13] ? 13 : 14) : 15; + } + + internal static void makeDecodeTables( + Span lenTab, + int offset, + Decode.Decode dec, + int size + ) { Span lenCount = stackalloc int[16]; Span tmpPos = stackalloc int[16]; @@ -190,8 +89,7 @@ internal static class UnpackUtility long M, N; - new Span(dec.DecodeNum).Clear(); // memset(Dec->DecodeNum,0,Size*sizeof(*Dec->DecodeNum)); - + new Span(dec.DecodeNum).Clear(); for (i = 0; i < size; i++) { lenCount[lenTab[offset + i] & 0xF]++; diff --git a/src/SharpCompress/Compressors/Rar/UnpackV2017/BitInput.getbits_cpp.cs b/src/SharpCompress/Compressors/Rar/UnpackV2017/BitInput.getbits_cpp.cs index 17368531..8281ee01 100644 --- a/src/SharpCompress/Compressors/Rar/UnpackV2017/BitInput.getbits_cpp.cs +++ b/src/SharpCompress/Compressors/Rar/UnpackV2017/BitInput.getbits_cpp.cs @@ -1,10 +1,5 @@ -#if !Rar2017_64bit +using System.Buffers; using size_t = System.UInt32; -#else -using nint = System.Int64; -using nuint = System.UInt64; -using size_t = System.UInt64; -#endif #nullable disable @@ -22,7 +17,7 @@ internal partial class BitInput // read only 1 byte from the last position of buffer and avoid a crash // from access to next 3 bytes, which contents we do not need. size_t BufSize = MAX_SIZE + 3; - InBuf = new byte[BufSize]; + InBuf = ArrayPool.Shared.Rent(checked((int)BufSize)); // Ensure that we get predictable results when accessing bytes in area // not filled with read data. @@ -48,4 +43,13 @@ internal partial class BitInput public uint fgetbits() => // Function wrapped version of inline getbits to save code size. getbits(); + + public virtual void Dispose() + { + if (!ExternalBuffer && InBuf != null) + { + ArrayPool.Shared.Return(InBuf); + InBuf = null; + } + } } diff --git a/src/SharpCompress/Compressors/Rar/UnpackV2017/BitInput.getbits_hpp.cs b/src/SharpCompress/Compressors/Rar/UnpackV2017/BitInput.getbits_hpp.cs index f245ea19..b3fad1ad 100644 --- a/src/SharpCompress/Compressors/Rar/UnpackV2017/BitInput.getbits_hpp.cs +++ b/src/SharpCompress/Compressors/Rar/UnpackV2017/BitInput.getbits_hpp.cs @@ -1,6 +1,8 @@ -namespace SharpCompress.Compressors.Rar.UnpackV2017; +using System; -internal partial class BitInput +namespace SharpCompress.Compressors.Rar.UnpackV2017; + +internal partial class BitInput : IDisposable { public const int MAX_SIZE = 0x8000; // Size of input buffer. diff --git a/src/SharpCompress/Compressors/Rar/UnpackV2017/FragmentedWindow.unpack50frag_cpp.cs b/src/SharpCompress/Compressors/Rar/UnpackV2017/FragmentedWindow.unpack50frag_cpp.cs index bfa69829..ff1b4631 100644 --- a/src/SharpCompress/Compressors/Rar/UnpackV2017/FragmentedWindow.unpack50frag_cpp.cs +++ b/src/SharpCompress/Compressors/Rar/UnpackV2017/FragmentedWindow.unpack50frag_cpp.cs @@ -1,13 +1,9 @@ #nullable disable -#if !Rar2017_64bit -using size_t = System.UInt32; -#else -using nint = System.Int64; -using nuint = System.UInt64; -using size_t = System.UInt64; -#endif using System; +using System.Buffers; +using SharpCompress.Common; +using size_t = System.UInt32; namespace SharpCompress.Compressors.Rar.UnpackV2017; @@ -24,15 +20,16 @@ internal partial class FragmentedWindow // Reset(); //} - private void Reset() + public void Reset() { for (uint I = 0; I < Mem.Length; I++) { if (Mem[I] != null) { - //free(Mem[I]); + ArrayPool.Shared.Return(Mem[I]); Mem[I] = null; } + MemSize[I] = 0; } } @@ -55,7 +52,7 @@ internal partial class FragmentedWindow byte[] NewMem = null; while (Size >= MinSize) { - NewMem = new byte[Size]; + NewMem = ArrayPool.Shared.Rent(checked((int)Size)); if (NewMem != null) { break; @@ -69,7 +66,7 @@ internal partial class FragmentedWindow // sharpcompress: don't need this, freshly allocated above //Utility.Memset(NewMem,0,Size); - Mem[BlockNum] = NewMem ?? throw new InvalidOperationException(); + Mem[BlockNum] = NewMem ?? throw new ArchiveOperationException(); TotalSize += Size; MemSize[BlockNum] = TotalSize; BlockNum++; @@ -77,7 +74,7 @@ internal partial class FragmentedWindow if (TotalSize < WinSize) // Not found enough free blocks. //throw std::bad_alloc(); { - throw new InvalidOperationException(); + throw new ArchiveOperationException(); } } diff --git a/src/SharpCompress/Compressors/Rar/UnpackV2017/PackDef.compress_hpp.cs b/src/SharpCompress/Compressors/Rar/UnpackV2017/PackDef.compress_hpp.cs index ed4e74ea..22e53df0 100644 --- a/src/SharpCompress/Compressors/Rar/UnpackV2017/PackDef.compress_hpp.cs +++ b/src/SharpCompress/Compressors/Rar/UnpackV2017/PackDef.compress_hpp.cs @@ -36,7 +36,6 @@ internal static class PackDef // CODE_ENDFILE, CODE_FILTER, CODE_FILTERDATA // }; - //enum FilterType { // These values must not be changed, because we use them directly // in RAR5 compression and decompression code. diff --git a/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.cs b/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.cs index 53db46d2..920d7fa5 100644 --- a/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.cs +++ b/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.cs @@ -1,13 +1,10 @@ -#if !Rar2017_64bit -using size_t = System.UInt32; -#else -using nint = System.Int64; -using nuint = System.UInt64; -using size_t = System.UInt64; -#endif using System; +using System.Buffers; using System.IO; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common.Rar.Headers; +using size_t = System.UInt32; namespace SharpCompress.Compressors.Rar.UnpackV2017; @@ -29,9 +26,28 @@ internal partial class Unpack : IRarUnpack // NOTE: caller has logic to check for -1 for error we throw instead. readStream.Read(buf, offset, count); + private async ValueTask UnpIO_UnpReadAsync( + byte[] buf, + int offset, + int count, + CancellationToken cancellationToken = default + ) => + // NOTE: caller has logic to check for -1 for error we throw instead. + await readStream.ReadAsync(buf, offset, count, cancellationToken).ConfigureAwait(false); + private void UnpIO_UnpWrite(byte[] buf, size_t offset, uint count) => writeStream.Write(buf, checked((int)offset), checked((int)count)); + private async ValueTask UnpIO_UnpWriteAsync( + byte[] buf, + size_t offset, + uint count, + CancellationToken cancellationToken = default + ) => + await writeStream + .WriteAsync(buf, checked((int)offset), checked((int)count), cancellationToken) + .ConfigureAwait(false); + public void DoUnpack(FileHeader fileHeader, Stream readStream, Stream writeStream) { // as of 12/2017 .NET limits array indexing to using a signed integer @@ -53,6 +69,25 @@ internal partial class Unpack : IRarUnpack DoUnpack(); } + public async ValueTask DoUnpackAsync( + FileHeader fileHeader, + Stream readStream, + Stream writeStream, + CancellationToken cancellationToken = default + ) + { + DestUnpSize = fileHeader.UncompressedSize; + this.fileHeader = fileHeader; + this.readStream = readStream; + this.writeStream = writeStream; + if (!fileHeader.IsStored) + { + Init(fileHeader.WindowSize, fileHeader.IsSolid); + } + Suspended = false; + await DoUnpackAsync(cancellationToken).ConfigureAwait(false); + } + public void DoUnpack() { if (fileHeader.IsStored) @@ -65,36 +100,87 @@ internal partial class Unpack : IRarUnpack } } + public async ValueTask DoUnpackAsync(CancellationToken cancellationToken = default) + { + if (fileHeader.IsStored) + { + await UnstoreFileAsync(cancellationToken).ConfigureAwait(false); + } + else + { + // TODO: When compression methods are converted to async, call them here + // For now, fall back to synchronous version + await DoUnpackAsync( + fileHeader.CompressionAlgorithm, + fileHeader.IsSolid, + cancellationToken + ) + .ConfigureAwait(false); + } + } + private void UnstoreFile() { - var b = new byte[0x10000]; + Span b = stackalloc byte[(int)Math.Min(0x10000, DestUnpSize)]; do { - var n = readStream.Read(b, 0, (int)Math.Min(b.Length, DestUnpSize)); + var n = readStream.Read(b); if (n == 0) { break; } - writeStream.Write(b, 0, n); + writeStream.Write(b.Slice(0, n)); DestUnpSize -= n; } while (!Suspended); } + private async ValueTask UnstoreFileAsync(CancellationToken cancellationToken = default) + { + var buffer = ArrayPool.Shared.Rent((int)Math.Min(0x10000, DestUnpSize)); + try + { + do + { + var n = await readStream + .ReadAsync(buffer, 0, buffer.Length, cancellationToken) + .ConfigureAwait(false); + if (n == 0) + { + break; + } + await writeStream.WriteAsync(buffer, 0, n, cancellationToken).ConfigureAwait(false); + DestUnpSize -= n; + } while (!Suspended); + } + finally + { + ArrayPool.Shared.Return(buffer); + } + } + public bool Suspended { get; set; } public long DestSize => DestUnpSize; - public int Char + public int ReadChar() { - get + // TODO: coderb: not sure where the "MAXSIZE-30" comes from, ported from V1 code + if (InAddr > MAX_SIZE - 30) { - // TODO: coderb: not sure where the "MAXSIZE-30" comes from, ported from V1 code - if (InAddr > MAX_SIZE - 30) - { - UnpReadBuf(); - } - return InBuf[InAddr++]; + UnpReadBuf(); } + return InBuf[InAddr++]; + } + + public async ValueTask ReadCharAsync(CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + // TODO: coderb: not sure where the "MAXSIZE-30" comes from, ported from V1 code + if (InAddr > MAX_SIZE - 30) + { + await UnpReadBufAsync(cancellationToken).ConfigureAwait(false); + } + return InBuf[InAddr++]; } public int PpmEscChar @@ -103,6 +189,18 @@ internal partial class Unpack : IRarUnpack set => PPMEscChar = value; } - public static byte[] EnsureCapacity(byte[] array, int length) => - array.Length < length ? new byte[length] : array; + private byte[] EnsureCapacity(byte[] array, int length) + { + if (array.Length >= length) + { + return array; + } + + var newArray = ArrayPool.Shared.Rent(length); + if (array.Length != 0) + { + ArrayPool.Shared.Return(array); + } + return newArray; + } } diff --git a/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.rawint_hpp.cs b/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.rawint_hpp.cs index 03614dde..a650f21c 100644 --- a/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.rawint_hpp.cs +++ b/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.rawint_hpp.cs @@ -1,10 +1,4 @@ -#if !Rar2017_64bit -#else -using nint = System.Int64; -using nuint = System.UInt64; -using size_t = System.UInt64; -#endif -using uint32 = System.UInt32; +using uint32 = System.UInt32; namespace SharpCompress.Compressors.Rar.UnpackV2017; @@ -67,7 +61,6 @@ internal partial class Unpack //#endif //} - //#if defined(LITTLE_ENDIAN) && defined(ALLOW_MISALIGNED) //#define USE_MEM_BYTESWAP //#endif @@ -84,7 +77,6 @@ internal partial class Unpack //#endif //} - // Save integer to memory as big endian. //inline void RawPutBE4(uint32 i,byte *mem) //{ @@ -100,7 +92,6 @@ internal partial class Unpack //#endif //} - //inline uint32 ByteSwap32(uint32 i) //{ //#ifdef _MSC_VER diff --git a/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack15_async.cs b/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack15_async.cs new file mode 100644 index 00000000..e9114437 --- /dev/null +++ b/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack15_async.cs @@ -0,0 +1,100 @@ +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Compressors.Rar.UnpackV2017; + +internal partial class Unpack +{ + private async ValueTask Unpack15Async(bool Solid, CancellationToken cancellationToken = default) + { + UnpInitData(Solid); + UnpInitData15(Solid); + await UnpReadBufAsync(cancellationToken).ConfigureAwait(false); + if (!Solid) + { + InitHuff(); + UnpPtr = 0; + } + else + { + UnpPtr = WrPtr; + } + + --DestUnpSize; + if (DestUnpSize >= 0) + { + GetFlagsBuf(); + FlagsCnt = 8; + } + + while (DestUnpSize >= 0) + { + UnpPtr &= MaxWinMask; + + if ( + Inp.InAddr > ReadTop - 30 + && !await UnpReadBufAsync(cancellationToken).ConfigureAwait(false) + ) + { + break; + } + + if (((WrPtr - UnpPtr) & MaxWinMask) < 270 && WrPtr != UnpPtr) + { + await UnpWriteBuf20Async(cancellationToken).ConfigureAwait(false); + } + + if (StMode != 0) + { + HuffDecode(); + continue; + } + + if (--FlagsCnt < 0) + { + GetFlagsBuf(); + FlagsCnt = 7; + } + + if ((FlagBuf & 0x80) != 0) + { + FlagBuf <<= 1; + if (Nlzb > Nhfb) + { + LongLZ(); + } + else + { + HuffDecode(); + } + } + else + { + FlagBuf <<= 1; + if (--FlagsCnt < 0) + { + GetFlagsBuf(); + FlagsCnt = 7; + } + if ((FlagBuf & 0x80) != 0) + { + FlagBuf <<= 1; + if (Nlzb > Nhfb) + { + HuffDecode(); + } + else + { + LongLZ(); + } + } + else + { + FlagBuf <<= 1; + ShortLZ(); + } + } + } + await UnpWriteBuf20Async(cancellationToken).ConfigureAwait(false); + } +} diff --git a/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack15_cpp.cs b/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack15_cpp.cs index dabb0e08..c1886df8 100644 --- a/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack15_cpp.cs +++ b/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack15_cpp.cs @@ -19,7 +19,7 @@ internal partial class Unpack 0xf000, 0xf200, 0xf200, - 0xffff + 0xffff, }; private static readonly uint[] PosL1 = { 0, 0, 0, 2, 3, 5, 7, 11, 16, 20, 24, 32, 32 }; @@ -37,7 +37,7 @@ internal partial class Unpack 0xf000, 0xf200, 0xf240, - 0xffff + 0xffff, }; private static readonly uint[] PosL2 = { 0, 0, 0, 0, 5, 7, 9, 13, 18, 22, 26, 34, 36 }; @@ -54,7 +54,7 @@ internal partial class Unpack 0xf200, 0xf200, 0xf200, - 0xffff + 0xffff, }; private static readonly uint[] PosHf0 = { 0, 0, 0, 0, 0, 8, 16, 24, 33, 33, 33, 33, 33 }; @@ -70,7 +70,7 @@ internal partial class Unpack 0xf200, 0xf200, 0xf7e0, - 0xffff + 0xffff, }; private static readonly uint[] PosHf1 = { 0, 0, 0, 0, 0, 0, 4, 44, 60, 76, 80, 80, 127 }; @@ -86,7 +86,7 @@ internal partial class Unpack 0xfa00, 0xffff, 0xffff, - 0xffff + 0xffff, }; private static readonly uint[] PosHf2 = { 0, 0, 0, 0, 0, 0, 2, 7, 53, 117, 233, 0, 0 }; @@ -101,7 +101,7 @@ internal partial class Unpack 0xfe80, 0xffff, 0xffff, - 0xffff + 0xffff, }; private static readonly uint[] PosHf3 = { 0, 0, 0, 0, 0, 0, 0, 2, 16, 218, 251, 0, 0 }; @@ -225,7 +225,7 @@ internal partial class Unpack 6, 6, 4, - 0 + 0, }; public static readonly uint[] ShortXor1 = { @@ -243,7 +243,7 @@ internal partial class Unpack 0x90, 0x98, 0x9c, - 0xb0 + 0xb0, }; public static readonly uint[] ShortLen2 = { @@ -262,7 +262,7 @@ internal partial class Unpack 6, 6, 4, - 0 + 0, }; public static readonly uint[] ShortXor2 = { @@ -280,7 +280,7 @@ internal partial class Unpack 0x90, 0x98, 0x9c, - 0xb0 + 0xb0, }; } diff --git a/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack20_async.cs b/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack20_async.cs new file mode 100644 index 00000000..84d43060 --- /dev/null +++ b/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack20_async.cs @@ -0,0 +1,337 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using static SharpCompress.Compressors.Rar.UnpackV2017.PackDef; +using static SharpCompress.Compressors.Rar.UnpackV2017.Unpack.Unpack20Local; + +namespace SharpCompress.Compressors.Rar.UnpackV2017; + +internal partial class Unpack +{ + private async ValueTask Unpack20Async(bool Solid, CancellationToken cancellationToken = default) + { + uint Bits; + + if (Suspended) + { + UnpPtr = WrPtr; + } + else + { + UnpInitData(Solid); + if (!await UnpReadBufAsync(cancellationToken).ConfigureAwait(false)) + { + return; + } + + if ( + (!Solid || !TablesRead2) + && !await ReadTables20Async(cancellationToken).ConfigureAwait(false) + ) + { + return; + } + + --DestUnpSize; + } + + while (DestUnpSize >= 0) + { + UnpPtr &= MaxWinMask; + + if (Inp.InAddr > ReadTop - 30) + { + if (!await UnpReadBufAsync(cancellationToken).ConfigureAwait(false)) + { + break; + } + } + + if (((WrPtr - UnpPtr) & MaxWinMask) < 270 && WrPtr != UnpPtr) + { + await UnpWriteBuf20Async(cancellationToken).ConfigureAwait(false); + if (Suspended) + { + return; + } + } + if (UnpAudioBlock) + { + var AudioNumber = DecodeNumber(Inp, MD[UnpCurChannel]); + + if (AudioNumber == 256) + { + if (!await ReadTables20Async(cancellationToken).ConfigureAwait(false)) + { + break; + } + + continue; + } + Window[UnpPtr++] = DecodeAudio((int)AudioNumber); + if (++UnpCurChannel == UnpChannels) + { + UnpCurChannel = 0; + } + + --DestUnpSize; + continue; + } + + var Number = DecodeNumber(Inp, BlockTables.LD); + if (Number < 256) + { + Window[UnpPtr++] = (byte)Number; + --DestUnpSize; + continue; + } + if (Number > 269) + { + var Length = (uint)(LDecode[Number -= 270] + 3); + if ((Bits = LBits[Number]) > 0) + { + Length += Inp.getbits() >> (int)(16 - Bits); + Inp.addbits(Bits); + } + + var DistNumber = DecodeNumber(Inp, BlockTables.DD); + var Distance = DDecode[DistNumber] + 1; + if ((Bits = DBits[DistNumber]) > 0) + { + Distance += Inp.getbits() >> (int)(16 - Bits); + Inp.addbits(Bits); + } + + if (Distance >= 0x2000) + { + Length++; + if (Distance >= 0x40000L) + { + Length++; + } + } + + CopyString20(Length, Distance); + continue; + } + if (Number == 269) + { + if (!await ReadTables20Async(cancellationToken).ConfigureAwait(false)) + { + break; + } + + continue; + } + if (Number == 256) + { + CopyString20(LastLength, LastDist); + continue; + } + if (Number < 261) + { + var Distance = OldDist[(OldDistPtr - (Number - 256)) & 3]; + var LengthNumber = DecodeNumber(Inp, BlockTables.RD); + var Length = (uint)(LDecode[LengthNumber] + 2); + if ((Bits = LBits[LengthNumber]) > 0) + { + Length += Inp.getbits() >> (int)(16 - Bits); + Inp.addbits(Bits); + } + if (Distance >= 0x101) + { + Length++; + if (Distance >= 0x2000) + { + Length++; + if (Distance >= 0x40000) + { + Length++; + } + } + } + CopyString20(Length, Distance); + continue; + } + if (Number < 270) + { + var Distance = (uint)(SDDecode[Number -= 261] + 1); + if ((Bits = SDBits[Number]) > 0) + { + Distance += Inp.getbits() >> (int)(16 - Bits); + Inp.addbits(Bits); + } + CopyString20(2, Distance); + continue; + } + } + await ReadLastTables20Async(cancellationToken).ConfigureAwait(false); + await UnpWriteBuf20Async(cancellationToken).ConfigureAwait(false); + } + + private async ValueTask UnpWriteBuf20Async(CancellationToken cancellationToken = default) + { + if (UnpPtr != WrPtr) + { + UnpSomeRead = true; + } + + if (UnpPtr < WrPtr) + { + await UnpIO_UnpWriteAsync( + Window, + WrPtr, + (uint)(-(int)WrPtr & MaxWinMask), + cancellationToken + ) + .ConfigureAwait(false); + await UnpIO_UnpWriteAsync(Window, 0, UnpPtr, cancellationToken).ConfigureAwait(false); + UnpAllBuf = true; + } + else + { + await UnpIO_UnpWriteAsync(Window, WrPtr, UnpPtr - WrPtr, cancellationToken) + .ConfigureAwait(false); + } + + WrPtr = UnpPtr; + } + + private async ValueTask ReadTables20Async(CancellationToken cancellationToken = default) + { + byte[] BitLength = new byte[checked((int)BC20)]; + byte[] Table = new byte[checked((int)MC20 * 4)]; + if (Inp.InAddr > ReadTop - 25) + { + if (!await UnpReadBufAsync(cancellationToken).ConfigureAwait(false)) + { + return false; + } + } + + var BitField = Inp.getbits(); + UnpAudioBlock = (BitField & 0x8000) != 0; + + if ((BitField & 0x4000) != 0) + { + Array.Clear(UnpOldTable20, 0, UnpOldTable20.Length); + } + + Inp.addbits(2); + + uint TableSize; + if (UnpAudioBlock) + { + UnpChannels = ((BitField >> 12) & 3) + 1; + if (UnpCurChannel >= UnpChannels) + { + UnpCurChannel = 0; + } + + Inp.addbits(2); + TableSize = MC20 * UnpChannels; + } + else + { + TableSize = NC20 + DC20 + RC20; + } + + for (int I = 0; I < checked((int)BC20); I++) + { + BitLength[I] = (byte)(Inp.getbits() >> 12); + Inp.addbits(4); + } + MakeDecodeTables(BitLength, 0, BlockTables.BD, BC20); + for (int I = 0; I < checked((int)TableSize); ) + { + if (Inp.InAddr > ReadTop - 5) + { + if (!await UnpReadBufAsync(cancellationToken).ConfigureAwait(false)) + { + return false; + } + } + + var Number = DecodeNumber(Inp, BlockTables.BD); + if (Number < 16) + { + Table[I] = (byte)((Number + UnpOldTable20[I]) & 0xf); + I++; + } + else if (Number == 16) + { + var N = (Inp.getbits() >> 14) + 3; + Inp.addbits(2); + if (I == 0) + { + return false; // We cannot have "repeat previous" code at the first position. + } + else + { + while (N-- > 0 && I < TableSize) + { + Table[I] = Table[I - 1]; + I++; + } + } + } + else + { + uint N; + if (Number == 17) + { + N = (Inp.getbits() >> 13) + 3; + Inp.addbits(3); + } + else + { + N = (Inp.getbits() >> 9) + 11; + Inp.addbits(7); + } + while (N-- > 0 && I < TableSize) + { + Table[I++] = 0; + } + } + } + TablesRead2 = true; + if (Inp.InAddr > ReadTop) + { + return true; + } + + if (UnpAudioBlock) + { + for (int I = 0; I < UnpChannels; I++) + { + MakeDecodeTables(Table, (int)(I * MC20), MD[I], MC20); + } + } + else + { + MakeDecodeTables(Table, 0, BlockTables.LD, NC20); + MakeDecodeTables(Table, (int)NC20, BlockTables.DD, DC20); + MakeDecodeTables(Table, (int)(NC20 + DC20), BlockTables.RD, RC20); + } + Array.Copy(Table, 0, this.UnpOldTable20, 0, UnpOldTable20.Length); + return true; + } + + private async ValueTask ReadLastTables20Async(CancellationToken cancellationToken = default) + { + if (ReadTop >= Inp.InAddr + 5) + { + if (UnpAudioBlock) + { + if (DecodeNumber(Inp, MD[UnpCurChannel]) == 256) + { + await ReadTables20Async(cancellationToken).ConfigureAwait(false); + } + } + else if (DecodeNumber(Inp, BlockTables.LD) == 269) + { + await ReadTables20Async(cancellationToken).ConfigureAwait(false); + } + } + } +} diff --git a/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack20_cpp.cs b/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack20_cpp.cs index b92af290..55b4174c 100644 --- a/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack20_cpp.cs +++ b/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack20_cpp.cs @@ -1,9 +1,3 @@ -#if !Rar2017_64bit -#else -using nint = System.Int64; -using nuint = System.UInt64; -using size_t = System.UInt64; -#endif using System; using static SharpCompress.Compressors.Rar.UnpackV2017.PackDef; using static SharpCompress.Compressors.Rar.UnpackV2017.Unpack.Unpack20Local; @@ -51,7 +45,7 @@ internal partial class Unpack 128, 160, 192, - 224 + 224, }; public static readonly byte[] LBits = { @@ -82,7 +76,7 @@ internal partial class Unpack 5, 5, 5, - 5 + 5, }; public static readonly uint[] DDecode = { @@ -133,7 +127,7 @@ internal partial class Unpack 786432, 851968, 917504, - 983040 + 983040, }; public static readonly byte[] DBits = { @@ -184,7 +178,7 @@ internal partial class Unpack 16, 16, 16, - 16 + 16, }; public static readonly byte[] SDDecode = { 0, 4, 8, 16, 32, 64, 128, 192 }; public static readonly byte[] SDBits = { 2, 2, 3, 4, 5, 6, 6, 6 }; @@ -371,8 +365,8 @@ internal partial class Unpack private bool ReadTables20() { - var BitLength = new byte[BC20]; - var Table = new byte[MC20 * 4]; + Span BitLength = stackalloc byte[checked((int)BC20)]; + Span Table = stackalloc byte[checked((int)MC20 * 4)]; if (Inp.InAddr > ReadTop - 25) { if (!UnpReadBuf()) @@ -408,13 +402,13 @@ internal partial class Unpack TableSize = NC20 + DC20 + RC20; } - for (uint I = 0; I < BC20; I++) + for (int I = 0; I < checked((int)BC20); I++) { BitLength[I] = (byte)(Inp.getbits() >> 12); Inp.addbits(4); } MakeDecodeTables(BitLength, 0, BlockTables.BD, BC20); - for (uint I = 0; I < TableSize; ) + for (int I = 0; I < checked((int)TableSize); ) { if (Inp.InAddr > ReadTop - 5) { @@ -485,8 +479,7 @@ internal partial class Unpack MakeDecodeTables(Table, (int)NC20, BlockTables.DD, DC20); MakeDecodeTables(Table, (int)(NC20 + DC20), BlockTables.RD, RC20); } - //x memcpy(UnpOldTable20,Table,sizeof(UnpOldTable20)); - Array.Copy(Table, UnpOldTable20, UnpOldTable20.Length); + Table.CopyTo(this.UnpOldTable20); return true; } diff --git a/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack30_cpp.cs b/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack30_cpp.cs deleted file mode 100644 index 7a5caee0..00000000 --- a/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack30_cpp.cs +++ /dev/null @@ -1,793 +0,0 @@ -#if !Rar2017_64bit -#else -using nint = System.Int64; -using nuint = System.UInt64; -using size_t = System.UInt64; -#endif - -//using static SharpCompress.Compressors.Rar.UnpackV2017.Unpack.Unpack30Local; -/* -namespace SharpCompress.Compressors.Rar.UnpackV2017 -{ - internal partial class Unpack - { - -#if !RarV2017_RAR5ONLY -// We use it instead of direct PPM.DecodeChar call to be sure that -// we reset PPM structures in case of corrupt data. It is important, -// because these structures can be invalid after PPM.DecodeChar returned -1. -int SafePPMDecodeChar() -{ - int Ch=PPM.DecodeChar(); - if (Ch==-1) // Corrupt PPM data found. - { - PPM.CleanUp(); // Reset possibly corrupt PPM data structures. - UnpBlockType=BLOCK_LZ; // Set faster and more fail proof LZ mode. - } - return(Ch); -} - -internal static class Unpack30Local { - public static readonly byte[] LDecode={0,1,2,3,4,5,6,7,8,10,12,14,16,20,24,28,32,40,48,56,64,80,96,112,128,160,192,224}; - public static readonly byte[] LBits= {0,0,0,0,0,0,0,0,1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 4, 5, 5, 5, 5}; - public static readonly int[] DDecode = new int[DC]; - public static readonly byte[] DBits = new byte[DC]; - public static readonly int[] DBitLengthCounts= {4,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,14,0,12}; - public static readonly byte[] SDDecode={0,4,8,16,32,64,128,192}; - public static readonly byte[] SDBits= {2,2,3, 4, 5, 6, 6, 6}; -} -void Unpack29(bool Solid) -{ - uint Bits; - - if (DDecode[1]==0) - { - int Dist=0,BitLength=0,Slot=0; - for (int I=0;IReadBorder) - { - if (!UnpReadBuf30()) - break; - } - if (((WrPtr-UnpPtr) & MaxWinMask)<260 && WrPtr!=UnpPtr) - { - UnpWriteBuf30(); - if (WrittenFileSize>DestUnpSize) - return; - if (Suspended) - { - FileExtracted=false; - return; - } - } - if (UnpBlockType==BLOCK_PPM) - { - // Here speed is critical, so we do not use SafePPMDecodeChar, - // because sometimes even the inline function can introduce - // some additional penalty. - int Ch=PPM.DecodeChar(); - if (Ch==-1) // Corrupt PPM data found. - { - PPM.CleanUp(); // Reset possibly corrupt PPM data structures. - UnpBlockType=BLOCK_LZ; // Set faster and more fail proof LZ mode. - break; - } - if (Ch==PPMEscChar) - { - int NextCh=SafePPMDecodeChar(); - if (NextCh==0) // End of PPM encoding. - { - if (!ReadTables30()) - break; - continue; - } - if (NextCh==-1) // Corrupt PPM data found. - break; - if (NextCh==2) // End of file in PPM mode. - break; - if (NextCh==3) // Read VM code. - { - if (!ReadVMCodePPM()) - break; - continue; - } - if (NextCh==4) // LZ inside of PPM. - { - uint Distance=0,Length; - bool Failed=false; - for (int I=0;I<4 && !Failed;I++) - { - int _Ch=SafePPMDecodeChar(); - if (_Ch==-1) - Failed=true; - else - if (I==3) - Length=(byte)_Ch; - else - Distance=(Distance<<8)+(byte)_Ch; - } - if (Failed) - break; - - CopyString(Length+32,Distance+2); - continue; - } - if (NextCh==5) // One byte distance match (RLE) inside of PPM. - { - int Length=SafePPMDecodeChar(); - if (Length==-1) - break; - CopyString((uint)(Length+4),1); - continue; - } - // If we are here, NextCh must be 1, what means that current byte - // is equal to our 'escape' byte, so we just store it to Window. - } - Window[UnpPtr++]=(byte)Ch; - continue; - } - - uint Number=DecodeNumber(Inp,BlockTables.LD); - if (Number<256) - { - Window[UnpPtr++]=(byte)Number; - continue; - } - if (Number>=271) - { - uint Length=(uint)(LDecode[Number-=271]+3); - if ((Bits=LBits[Number])>0) - { - Length+=Inp.getbits()>>(int)(16-Bits); - Inp.addbits(Bits); - } - - uint DistNumber=DecodeNumber(Inp,BlockTables.DD); - uint Distance=(uint)(DDecode[DistNumber]+1); - if ((Bits=DBits[DistNumber])>0) - { - if (DistNumber>9) - { - if (Bits>4) - { - Distance+=((Inp.getbits()>>(int)(20-Bits))<<4); - Inp.addbits(Bits-4); - } - if (LowDistRepCount>0) - { - LowDistRepCount--; - Distance+=(uint)PrevLowDist; - } - else - { - uint LowDist=DecodeNumber(Inp,BlockTables.LDD); - if (LowDist==16) - { - LowDistRepCount=(int)(LOW_DIST_REP_COUNT-1); - Distance+=(uint)PrevLowDist; - } - else - { - Distance+=LowDist; - PrevLowDist=(int)LowDist; - } - } - } - else - { - Distance+=Inp.getbits()>>(int)(16-Bits); - Inp.addbits(Bits); - } - } - - if (Distance>=0x2000) - { - Length++; - if (Distance>=0x40000) - Length++; - } - - InsertOldDist(Distance); - LastLength=Length; - CopyString(Length,Distance); - continue; - } - if (Number==256) - { - if (!ReadEndOfBlock()) - break; - continue; - } - if (Number==257) - { - if (!ReadVMCode()) - break; - continue; - } - if (Number==258) - { - if (LastLength!=0) - CopyString(LastLength,OldDist[0]); - continue; - } - if (Number<263) - { - uint DistNum=Number-259; - uint Distance=OldDist[DistNum]; - for (uint I=DistNum;I>0;I--) - OldDist[I]=OldDist[I-1]; - OldDist[0]=Distance; - - uint LengthNumber=DecodeNumber(Inp,BlockTables.RD); - int Length=LDecode[LengthNumber]+2; - if ((Bits=LBits[LengthNumber])>0) - { - Length+=(int)(Inp.getbits()>>(int)(16-Bits)); - Inp.addbits(Bits); - } - LastLength=(uint)Length; - CopyString((uint)Length,Distance); - continue; - } - if (Number<272) - { - uint Distance=(uint)(SDDecode[Number-=263]+1); - if ((Bits=SDBits[Number])>0) - { - Distance+=Inp.getbits()>>(int)(16-Bits); - Inp.addbits(Bits); - } - InsertOldDist(Distance); - LastLength=2; - CopyString(2,Distance); - continue; - } - } - UnpWriteBuf30(); -} - - -// Return 'false' to quit unpacking the current file or 'true' to continue. -bool ReadEndOfBlock() -{ - uint BitField=Inp.getbits(); - bool NewTable,NewFile=false; - - // "1" - no new file, new table just here. - // "00" - new file, no new table. - // "01" - new file, new table (in beginning of next file). - - if ((BitField & 0x8000)!=0) - { - NewTable=true; - Inp.addbits(1); - } - else - { - NewFile=true; - NewTable=(BitField & 0x4000)!=0; - Inp.addbits(2); - } - TablesRead3=!NewTable; - - // Quit immediately if "new file" flag is set. If "new table" flag - // is present, we'll read the table in beginning of next file - // based on 'TablesRead3' 'false' value. - if (NewFile) - return false; - return ReadTables30(); // Quit only if we failed to read tables. -} - - -bool ReadVMCode() -{ - // Entire VM code is guaranteed to fully present in block defined - // by current Huffman table. Compressor checks that VM code does not cross - // Huffman block boundaries. - uint FirstByte=Inp.getbits()>>8; - Inp.addbits(8); - uint Length=(FirstByte & 7)+1; - if (Length==7) - { - Length=(Inp.getbits()>>8)+7; - Inp.addbits(8); - } - else - if (Length==8) - { - Length=Inp.getbits(); - Inp.addbits(16); - } - if (Length==0) - return false; - Array VMCode(Length); - for (uint I=0;I=ReadTop-1 && !UnpReadBuf30() && I>8; - Inp.addbits(8); - } - return AddVMCode(FirstByte,&VMCode[0],Length); -} - - -bool ReadVMCodePPM() -{ - uint FirstByte=(uint)SafePPMDecodeChar(); - if ((int)FirstByte==-1) - return false; - uint Length=(FirstByte & 7)+1; - if (Length==7) - { - int B1=SafePPMDecodeChar(); - if (B1==-1) - return false; - Length=B1+7; - } - else - if (Length==8) - { - int B1=SafePPMDecodeChar(); - if (B1==-1) - return false; - int B2=SafePPMDecodeChar(); - if (B2==-1) - return false; - Length=B1*256+B2; - } - if (Length==0) - return false; - Array VMCode(Length); - for (uint I=0;IFilters30.Count || FiltPos>OldFilterLengths.Count) - return false; - LastFilter=(int)FiltPos; - bool NewFilter=(FiltPos==Filters30.Count); - - UnpackFilter30 StackFilter=new UnpackFilter30(); // New filter for PrgStack. - - UnpackFilter30 Filter; - if (NewFilter) // New filter code, never used before since VM reset. - { - if (FiltPos>MAX3_UNPACK_FILTERS) - { - // Too many different filters, corrupt archive. - //delete StackFilter; - return false; - } - - Filters30.Add(1); - Filters30[Filters30.Count-1]=Filter=new UnpackFilter30(); - StackFilter.ParentFilter=(uint)(Filters30.Count-1); - - // Reserve one item to store the data block length of our new filter - // entry. We'll set it to real block length below, after reading it. - // But we need to initialize it now, because when processing corrupt - // data, we can access this item even before we set it to real value. - OldFilterLengths.Add(0); - } - else // Filter was used in the past. - { - Filter=Filters30[(int)FiltPos]; - StackFilter.ParentFilter=FiltPos; - } - - int EmptyCount=0; - for (int I=0;I0) - PrgStack[I]=null; - } - if (EmptyCount==0) - { - if (PrgStack.Count>MAX3_UNPACK_FILTERS) - { - //delete StackFilter; - return false; - } - PrgStack.Add(1); - EmptyCount=1; - } - size_t StackPos=(uint)(this.PrgStack.Count-EmptyCount); - PrgStack[(int)StackPos]=StackFilter; - - uint BlockStart=RarVM.ReadData(VMCodeInp); - if ((FirstByte & 0x40)!=0) - BlockStart+=258; - StackFilter.BlockStart=(uint)((BlockStart+UnpPtr)&MaxWinMask); - if ((FirstByte & 0x20)!=0) - { - StackFilter.BlockLength=RarVM.ReadData(VMCodeInp); - - // Store the last data block length for current filter. - OldFilterLengths[(int)FiltPos]=(int)StackFilter.BlockLength; - } - else - { - // Set the data block size to same value as the previous block size - // for same filter. It is possible for corrupt data to access a new - // and not filled yet item of OldFilterLengths array here. This is why - // we set new OldFilterLengths items to zero above. - StackFilter.BlockLength=FiltPos>9; - VMCodeInp.faddbits(7); - for (int I=0;I<7;I++) - if ((InitMask & (1<=0x10000 || VMCodeSize==0) - return false; - Array VMCode(VMCodeSize); - for (uint I=0;I>8; - VMCodeInp.faddbits(8); - } - VM.Prepare(&VMCode[0],VMCodeSize,&Filter->Prg); - } - StackFilter.Prg.Type=Filter.Prg.Type; - - return true; -} - - -bool UnpReadBuf30() -{ - int DataSize=ReadTop-Inp.InAddr; // Data left to process. - if (DataSize<0) - return false; - if (Inp.InAddr>BitInput.MAX_SIZE/2) - { - // If we already processed more than half of buffer, let's move - // remaining data into beginning to free more space for new data - // and ensure that calling function does not cross the buffer border - // even if we did not read anything here. Also it ensures that read size - // is not less than CRYPT_BLOCK_SIZE, so we can align it without risk - // to make it zero. - if (DataSize>0) - //x memmove(Inp.InBuf,Inp.InBuf+Inp.InAddr,DataSize); - Array.Copy(Inp.InBuf,Inp.InAddr,Inp.InBuf,0,DataSize); - Inp.InAddr=0; - ReadTop=DataSize; - } - else - DataSize=ReadTop; - int ReadCode=UnpIO_UnpRead(Inp.InBuf,DataSize,BitInput.MAX_SIZE-DataSize); - if (ReadCode>0) - ReadTop+=ReadCode; - ReadBorder=ReadTop-30; - return ReadCode!=-1; -} - - -void UnpWriteBuf30() -{ - uint WrittenBorder=(uint)WrPtr; - uint WriteSize=(uint)((UnpPtr-WrittenBorder)&MaxWinMask); - for (int I=0;IParentFilter]->Prg; - VM_PreparedProgram *Prg=&flt->Prg; - - ExecuteCode(Prg); - - byte[] FilteredData=Prg.FilteredData; - uint FilteredDataSize=Prg.FilteredDataSize; - - delete PrgStack[I]; - PrgStack[I]=null; - while (I+1Prg; - VM_PreparedProgram *NextPrg=&NextFilter->Prg; - - ExecuteCode(NextPrg); - - FilteredData=NextPrg.FilteredData; - FilteredDataSize=NextPrg.FilteredDataSize; - I++; - delete PrgStack[I]; - PrgStack[I]=null; - } - UnpIO_UnpWrite(FilteredData,0,FilteredDataSize); - UnpSomeRead=true; - WrittenFileSize+=FilteredDataSize; - WrittenBorder=BlockEnd; - WriteSize=(uint)((UnpPtr-WrittenBorder)&MaxWinMask); - } - else - { - // Current filter intersects the window write border, so we adjust - // the window border to process this filter next time, not now. - for (size_t J=I;JInitR[6]=(uint)WrittenFileSize; - VM.Execute(Prg); -} - - -bool ReadTables30() -{ - byte[] BitLength = new byte[BC]; - byte[] Table = new byte[HUFF_TABLE_SIZE30]; - if (Inp.InAddr>ReadTop-25) - if (!UnpReadBuf30()) - return(false); - Inp.faddbits((uint)((8-Inp.InBit)&7)); - uint BitField=Inp.fgetbits(); - if ((BitField & 0x8000) != 0) - { - UnpBlockType=BLOCK_PPM; - return(PPM.DecodeInit(this,PPMEscChar)); - } - UnpBlockType=BLOCK_LZ; - - PrevLowDist=0; - LowDistRepCount=0; - - if ((BitField & 0x4000) == 0) - Utility.Memset(UnpOldTable,0,UnpOldTable.Length); - Inp.faddbits(2); - - for (uint I=0;I> 12); - Inp.faddbits(4); - if (Length==15) - { - uint ZeroCount=(byte)(Inp.fgetbits() >> 12); - Inp.faddbits(4); - if (ZeroCount==0) - BitLength[I]=15; - else - { - ZeroCount+=2; - while (ZeroCount-- > 0 && IReadTop-5) - if (!UnpReadBuf30()) - return(false); - uint Number=DecodeNumber(Inp,BlockTables.BD); - if (Number<16) - { - Table[I]=(byte)((Number+this.UnpOldTable[I]) & 0xf); - I++; - } - else - if (Number<18) - { - uint N; - if (Number==16) - { - N=(Inp.fgetbits() >> 13)+3; - Inp.faddbits(3); - } - else - { - N=(Inp.fgetbits() >> 9)+11; - Inp.faddbits(7); - } - if (I==0) - return false; // We cannot have "repeat previous" code at the first position. - else - while (N-- > 0 && I> 13)+3; - Inp.faddbits(3); - } - else - { - N=(Inp.fgetbits() >> 9)+11; - Inp.faddbits(7); - } - while (N-- > 0 && IReadTop) - return false; - MakeDecodeTables(Table,0,BlockTables.LD,NC30); - MakeDecodeTables(Table,(int)NC30,BlockTables.DD,DC30); - MakeDecodeTables(Table,(int)(NC30+DC30),BlockTables.LDD,LDC30); - MakeDecodeTables(Table,(int)(NC30+DC30+LDC30),BlockTables.RD,RC30); - //x memcpy(UnpOldTable,Table,sizeof(UnpOldTable)); - Array.Copy(Table,0,UnpOldTable,0,UnpOldTable.Length); - return true; -} - -#endif - -void UnpInitData30(bool Solid) -{ - if (!Solid) - { - TablesRead3=false; - Utility.Memset(UnpOldTable, 0, UnpOldTable.Length); - PPMEscChar=2; - UnpBlockType=BLOCK_LZ; - } - InitFilters30(Solid); -} - - -void InitFilters30(bool Solid) -{ - if (!Solid) - { - //OldFilterLengths.SoftReset(); - OldFilterLengths.Clear(); - LastFilter=0; - - //for (size_t I=0;I= ReadBorder) + { + var FileDone = false; + + // We use 'while', because for empty block containing only Huffman table, + // we'll be on the block border once again just after reading the table. + while ( + Inp.InAddr > BlockHeader.BlockStart + BlockHeader.BlockSize - 1 + || Inp.InAddr == BlockHeader.BlockStart + BlockHeader.BlockSize - 1 + && Inp.InBit >= BlockHeader.BlockBitSize + ) + { + if (BlockHeader.LastBlockInFile) + { + FileDone = true; + break; + } + if ( + !await ReadBlockHeaderAsync(Inp, cancellationToken).ConfigureAwait(false) + || !await ReadTablesAsync(Inp, cancellationToken).ConfigureAwait(false) + ) + { + return; + } + } + if (FileDone || !await UnpReadBufAsync(cancellationToken).ConfigureAwait(false)) + { + break; + } + } + + if (((WriteBorder - UnpPtr) & MaxWinMask) < MAX_LZ_MATCH + 3 && WriteBorder != UnpPtr) + { + await UnpWriteBufAsync(cancellationToken).ConfigureAwait(false); + if (WrittenFileSize > DestUnpSize) + { + return; + } + + if (Suspended) + { + FileExtracted = false; + return; + } + } + + var MainSlot = DecodeNumber(Inp, BlockTables.LD); + if (MainSlot < 256) + { + if (Fragmented) + { + FragWindow[UnpPtr++] = (byte)MainSlot; + } + else + { + Window[UnpPtr++] = (byte)MainSlot; + } + continue; + } + if (MainSlot >= 262) + { + var Length = SlotToLength(Inp, MainSlot - 262); + + uint DBits, + Distance = 1, + DistSlot = DecodeNumber(Inp, BlockTables.DD); + if (DistSlot < 4) + { + DBits = 0; + Distance += DistSlot; + } + else + { + DBits = (DistSlot / 2) - 1; + Distance += (2 | (DistSlot & 1)) << (int)DBits; + } + + if (DBits > 0) + { + if (DBits >= 4) + { + if (DBits > 4) + { + Distance += ((Inp.getbits32() >> (int)(36 - DBits)) << 4); + Inp.addbits(DBits - 4); + } + + var LowDist = DecodeNumber(Inp, BlockTables.LDD); + Distance += LowDist; + } + else + { + Distance += Inp.getbits32() >> (int)(32 - DBits); + Inp.addbits(DBits); + } + } + + if (Distance > 0x100) + { + Length++; + if (Distance > 0x2000) + { + Length++; + if (Distance > 0x40000) + { + Length++; + } + } + } + + InsertOldDist(Distance); + LastLength = Length; + if (Fragmented) + { + FragWindow.CopyString(Length, Distance, ref UnpPtr, MaxWinMask); + } + else + { + CopyString(Length, Distance); + } + continue; + } + if (MainSlot == 256) + { + var Filter = RentFilter(); + if ( + !await ReadFilterAsync(Inp, Filter, cancellationToken).ConfigureAwait(false) + || !AddFilter(Filter) + ) + { + ReturnFilter(Filter); + break; + } + continue; + } + if (MainSlot == 257) + { + if (LastLength != 0) + { + if (Fragmented) + { + FragWindow.CopyString(LastLength, OldDist[0], ref UnpPtr, MaxWinMask); + } + else + { + CopyString(LastLength, OldDist[0]); + } + } + continue; + } + if (MainSlot < 262) + { + var DistNum = MainSlot - 258; + var Distance = OldDist[DistNum]; + for (var I = DistNum; I > 0; I--) + { + OldDist[I] = OldDist[I - 1]; + } + + OldDist[0] = Distance; + + var LengthSlot = DecodeNumber(Inp, BlockTables.RD); + var Length = SlotToLength(Inp, LengthSlot); + LastLength = Length; + if (Fragmented) + { + FragWindow.CopyString(Length, Distance, ref UnpPtr, MaxWinMask); + } + else + { + CopyString(Length, Distance); + } + + continue; + } + } + await UnpWriteBufAsync(cancellationToken).ConfigureAwait(false); + } + + private async ValueTask ReadFilterAsync( + BitInput Inp, + UnpackFilter Filter, + CancellationToken cancellationToken = default + ) + { + if (!Inp.ExternalBuffer && Inp.InAddr > ReadTop - 16) + { + if (!await UnpReadBufAsync(cancellationToken).ConfigureAwait(false)) + { + return false; + } + } + + Filter.BlockStart = ReadFilterData(Inp); + Filter.BlockLength = ReadFilterData(Inp); + if (Filter.BlockLength > MAX_FILTER_BLOCK_SIZE) + { + Filter.BlockLength = 0; + } + + Filter.Type = (byte)(Inp.fgetbits() >> 13); + Inp.faddbits(3); + + if (Filter.Type == FILTER_DELTA) + { + Filter.Channels = (byte)((Inp.fgetbits() >> 11) + 1); + Inp.faddbits(5); + } + + return true; + } + + private async ValueTask UnpReadBufAsync(CancellationToken cancellationToken = default) + { + var DataSize = ReadTop - Inp.InAddr; // Data left to process. + if (DataSize < 0) + { + return false; + } + + BlockHeader.BlockSize -= Inp.InAddr - BlockHeader.BlockStart; + if (Inp.InAddr > MAX_SIZE / 2) + { + if (DataSize > 0) + { + Buffer.BlockCopy(Inp.InBuf, Inp.InAddr, Inp.InBuf, 0, DataSize); + } + + Inp.InAddr = 0; + ReadTop = DataSize; + } + else + { + DataSize = ReadTop; + } + + var ReadCode = 0; + if (MAX_SIZE != DataSize) + { + ReadCode = await UnpIO_UnpReadAsync( + Inp.InBuf, + DataSize, + MAX_SIZE - DataSize, + cancellationToken + ) + .ConfigureAwait(false); + } + + if (ReadCode > 0) // Can be also -1. + { + ReadTop += ReadCode; + } + + ReadBorder = ReadTop - 30; + BlockHeader.BlockStart = Inp.InAddr; + if (BlockHeader.BlockSize != -1) // '-1' means not defined yet. + { + ReadBorder = Math.Min(ReadBorder, BlockHeader.BlockStart + BlockHeader.BlockSize - 1); + } + return ReadCode != -1; + } + + private async ValueTask UnpWriteBufAsync(CancellationToken cancellationToken = default) + { + var WrittenBorder = WrPtr; + var FullWriteSize = (UnpPtr - WrittenBorder) & MaxWinMask; + var WriteSizeLeft = FullWriteSize; + var NotAllFiltersProcessed = false; + + for (var I = 0; I < Filters.Count; I++) + { + var flt = Filters[I]; + if (flt.Type == FILTER_NONE) + { + continue; + } + + if (flt.NextWindow) + { + if (((flt.BlockStart - WrPtr) & MaxWinMask) <= FullWriteSize) + { + flt.NextWindow = false; + } + continue; + } + + var BlockStart = flt.BlockStart; + var BlockLength = flt.BlockLength; + if (((BlockStart - WrittenBorder) & MaxWinMask) < WriteSizeLeft) + { + if (WrittenBorder != BlockStart) + { + await UnpWriteAreaAsync(WrittenBorder, BlockStart, cancellationToken) + .ConfigureAwait(false); + WrittenBorder = BlockStart; + WriteSizeLeft = (UnpPtr - WrittenBorder) & MaxWinMask; + } + if (BlockLength <= WriteSizeLeft) + { + if (BlockLength > 0) + { + var BlockEnd = (BlockStart + BlockLength) & MaxWinMask; + + FilterSrcMemory = EnsureCapacity( + FilterSrcMemory, + checked((int)BlockLength) + ); + var Mem = FilterSrcMemory; + if (BlockStart < BlockEnd || BlockEnd == 0) + { + if (Fragmented) + { + FragWindow.CopyData(Mem, 0, BlockStart, BlockLength); + } + else + { + Buffer.BlockCopy(Window, (int)BlockStart, Mem, 0, (int)BlockLength); + } + } + else + { + var FirstPartLength = MaxWinSize - BlockStart; + if (Fragmented) + { + FragWindow.CopyData(Mem, 0, BlockStart, FirstPartLength); + FragWindow.CopyData(Mem, FirstPartLength, 0, BlockEnd); + } + else + { + Buffer.BlockCopy( + Window, + (int)BlockStart, + Mem, + 0, + (int)FirstPartLength + ); + Buffer.BlockCopy( + Window, + 0, + Mem, + (int)FirstPartLength, + (int)BlockEnd + ); + } + } + + var OutMem = ApplyFilter(Mem, BlockLength, flt); + + Filters[I].Type = FILTER_NONE; + + if (OutMem != null) + { + await UnpIO_UnpWriteAsync(OutMem, 0, BlockLength, cancellationToken) + .ConfigureAwait(false); + WrittenFileSize += BlockLength; + } + + WrittenBorder = BlockEnd; + WriteSizeLeft = (UnpPtr - WrittenBorder) & MaxWinMask; + } + } + else + { + NotAllFiltersProcessed = true; + for (var J = I; J < Filters.Count; J++) + { + var fltj = Filters[J]; + if ( + fltj.Type != FILTER_NONE + && fltj.NextWindow == false + && ((fltj.BlockStart - WrPtr) & MaxWinMask) < FullWriteSize + ) + { + fltj.NextWindow = true; + } + } + break; + } + } + } + + var EmptyCount = 0; + for (var I = 0; I < Filters.Count; I++) + { + if (EmptyCount > 0) + { + Filters[I - EmptyCount] = Filters[I]; + } + + if (Filters[I].Type == FILTER_NONE) + { + ReturnFilter(Filters[I]); + EmptyCount++; + } + } + if (EmptyCount > 0) + { + Filters.RemoveRange(Filters.Count - EmptyCount, EmptyCount); + } + + if (!NotAllFiltersProcessed) + { + await UnpWriteAreaAsync(WrittenBorder, UnpPtr, cancellationToken).ConfigureAwait(false); + WrPtr = UnpPtr; + } + + WriteBorder = (UnpPtr + Math.Min(MaxWinSize, UNPACK_MAX_WRITE)) & MaxWinMask; + + if ( + WriteBorder == UnpPtr + || WrPtr != UnpPtr + && ((WrPtr - UnpPtr) & MaxWinMask) < ((WriteBorder - UnpPtr) & MaxWinMask) + ) + { + WriteBorder = WrPtr; + } + } + + private async ValueTask UnpWriteAreaAsync( + size_t StartPtr, + size_t EndPtr, + CancellationToken cancellationToken = default + ) + { + if (EndPtr != StartPtr) + { + UnpSomeRead = true; + } + + if (EndPtr < StartPtr) + { + UnpAllBuf = true; + } + + if (Fragmented) + { + var SizeToWrite = (EndPtr - StartPtr) & MaxWinMask; + while (SizeToWrite > 0) + { + var BlockSize = FragWindow.GetBlockSize(StartPtr, SizeToWrite); + FragWindow.GetBuffer(StartPtr, out var __buffer, out var __offset); + await UnpWriteDataAsync(__buffer, __offset, BlockSize, cancellationToken) + .ConfigureAwait(false); + SizeToWrite -= BlockSize; + StartPtr = (StartPtr + BlockSize) & MaxWinMask; + } + } + else if (EndPtr < StartPtr) + { + await UnpWriteDataAsync(Window, StartPtr, MaxWinSize - StartPtr, cancellationToken) + .ConfigureAwait(false); + await UnpWriteDataAsync(Window, 0, EndPtr, cancellationToken).ConfigureAwait(false); + } + else + { + await UnpWriteDataAsync(Window, StartPtr, EndPtr - StartPtr, cancellationToken) + .ConfigureAwait(false); + } + } + + private async ValueTask UnpWriteDataAsync( + byte[] Data, + size_t offset, + size_t Size, + CancellationToken cancellationToken = default + ) + { + if (WrittenFileSize >= DestUnpSize) + { + return; + } + + var WriteSize = Size; + var LeftToWrite = DestUnpSize - WrittenFileSize; + if (WriteSize > LeftToWrite) + { + WriteSize = (size_t)LeftToWrite; + } + + await UnpIO_UnpWriteAsync(Data, offset, WriteSize, cancellationToken).ConfigureAwait(false); + WrittenFileSize += Size; + } + + private async ValueTask ReadBlockHeaderAsync( + BitInput Inp, + CancellationToken cancellationToken = default + ) + { + BlockHeader.HeaderSize = 0; + + if (!Inp.ExternalBuffer && Inp.InAddr > ReadTop - 7) + { + if (!await UnpReadBufAsync(cancellationToken).ConfigureAwait(false)) + { + return false; + } + } + + Inp.faddbits((uint)((8 - Inp.InBit) & 7)); + + var BlockFlags = (byte)(Inp.fgetbits() >> 8); + Inp.faddbits(8); + var ByteCount = (uint)(((BlockFlags >> 3) & 3) + 1); // Block size byte count. + + if (ByteCount == 4) + { + return false; + } + + BlockHeader.HeaderSize = (int)(2 + ByteCount); + + BlockHeader.BlockBitSize = (BlockFlags & 7) + 1; + + var SavedCheckSum = (byte)(Inp.fgetbits() >> 8); + Inp.faddbits(8); + + var BlockSize = 0; + for (uint I = 0; I < ByteCount; I++) + { + BlockSize += (int)((Inp.fgetbits() >> 8) << (int)(I * 8)); + Inp.addbits(8); + } + + BlockHeader.BlockSize = BlockSize; + var CheckSum = (byte)(0x5a ^ BlockFlags ^ BlockSize ^ (BlockSize >> 8) ^ (BlockSize >> 16)); + if (CheckSum != SavedCheckSum) + { + return false; + } + + BlockHeader.BlockStart = Inp.InAddr; + ReadBorder = Math.Min(ReadBorder, BlockHeader.BlockStart + BlockHeader.BlockSize - 1); + + BlockHeader.LastBlockInFile = (BlockFlags & 0x40) != 0; + BlockHeader.TablePresent = (BlockFlags & 0x80) != 0; + return true; + } + + private async ValueTask ReadTablesAsync( + BitInput Inp, + CancellationToken cancellationToken = default + ) + { + if (!BlockHeader.TablePresent) + { + return true; + } + + if (!Inp.ExternalBuffer && Inp.InAddr > ReadTop - 25) + { + if (!await UnpReadBufAsync(cancellationToken).ConfigureAwait(false)) + { + return false; + } + } + + var BitLength = new byte[checked((int)BC)]; + for (int I = 0; I < BC; I++) + { + uint Length = (byte)(Inp.fgetbits() >> 12); + Inp.faddbits(4); + if (Length == 15) + { + uint ZeroCount = (byte)(Inp.fgetbits() >> 12); + Inp.faddbits(4); + if (ZeroCount == 0) + { + BitLength[I] = 15; + } + else + { + ZeroCount += 2; + while (ZeroCount-- > 0 && I < BitLength.Length) + { + BitLength[I++] = 0; + } + + I--; + } + } + else + { + BitLength[I] = (byte)Length; + } + } + + MakeDecodeTables(BitLength, 0, BlockTables.BD, BC); + + var Table = new byte[checked((int)HUFF_TABLE_SIZE)]; + const int TableSize = checked((int)HUFF_TABLE_SIZE); + for (int I = 0; I < TableSize; ) + { + if (!Inp.ExternalBuffer && Inp.InAddr > ReadTop - 5) + { + if (!await UnpReadBufAsync(cancellationToken).ConfigureAwait(false)) + { + return false; + } + } + + var Number = DecodeNumber(Inp, BlockTables.BD); + if (Number < 16) + { + Table[I] = (byte)Number; + I++; + } + else if (Number < 18) + { + uint N; + if (Number == 16) + { + N = (Inp.fgetbits() >> 13) + 3; + Inp.faddbits(3); + } + else + { + N = (Inp.fgetbits() >> 9) + 11; + Inp.faddbits(7); + } + if (I == 0) + { + // We cannot have "repeat previous" code at the first position. + // Multiple such codes would shift Inp position without changing I, + // which can lead to reading beyond of Inp boundary in mutithreading + // mode, where Inp.ExternalBuffer disables bounds check and we just + // reserve a lot of buffer space to not need such check normally. + return false; + } + else + { + while (N-- > 0 && I < TableSize) + { + Table[I] = Table[I - 1]; + I++; + } + } + } + else + { + uint N; + if (Number == 18) + { + N = (Inp.fgetbits() >> 13) + 3; + Inp.faddbits(3); + } + else + { + N = (Inp.fgetbits() >> 9) + 11; + Inp.faddbits(7); + } + while (N-- > 0 && I < TableSize) + { + Table[I++] = 0; + } + } + } + TablesRead5 = true; + if (!Inp.ExternalBuffer && Inp.InAddr > ReadTop) + { + return false; + } + + MakeDecodeTables(Table, 0, BlockTables.LD, NC); + MakeDecodeTables(Table, (int)NC, BlockTables.DD, DC); + MakeDecodeTables(Table, (int)(NC + DC), BlockTables.LDD, LDC); + MakeDecodeTables(Table, (int)(NC + DC + LDC), BlockTables.RD, RC); + return true; + } +} diff --git a/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack50_cpp.cs b/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack50_cpp.cs index f85921bd..3fe250b5 100644 --- a/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack50_cpp.cs +++ b/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack50_cpp.cs @@ -1,17 +1,9 @@ -#nullable disable - -#if !Rar2017_64bit -using size_t = System.UInt32; -#else -using nint = System.Int64; -using nuint = System.UInt64; -using size_t = System.UInt64; -#endif -using int64 = System.Int64; +#nullable disable using System; using static SharpCompress.Compressors.Rar.UnpackV2017.PackDef; using static SharpCompress.Compressors.Rar.UnpackV2017.UnpackGlobal; +using size_t = System.UInt32; namespace SharpCompress.Compressors.Rar.UnpackV2017; @@ -32,11 +24,7 @@ internal partial class Unpack // Check TablesRead5 to be sure that we read tables at least once // regardless of current block header TablePresent flag. // So we can safefly use these tables below. - if ( - !ReadBlockHeader(Inp, ref BlockHeader) - || !ReadTables(Inp, ref BlockHeader, ref BlockTables) - || !TablesRead5 - ) + if (!ReadBlockHeader(Inp) || !ReadTables(Inp) || !TablesRead5) { return; } @@ -63,10 +51,7 @@ internal partial class Unpack FileDone = true; break; } - if ( - !ReadBlockHeader(Inp, ref BlockHeader) - || !ReadTables(Inp, ref BlockHeader, ref BlockTables) - ) + if (!ReadBlockHeader(Inp) || !ReadTables(Inp)) { return; } @@ -171,9 +156,10 @@ internal partial class Unpack } if (MainSlot == 256) { - var Filter = new UnpackFilter(); + var Filter = RentFilter(); if (!ReadFilter(Inp, Filter) || !AddFilter(Filter)) { + ReturnFilter(Filter); break; } @@ -415,7 +401,7 @@ internal partial class Unpack else //x memcpy(Mem,Window+BlockStart,BlockLength); { - Utility.Copy(Window, BlockStart, Mem, 0, BlockLength); + Buffer.BlockCopy(Window, (int)BlockStart, Mem, 0, (int)BlockLength); } } else @@ -429,9 +415,21 @@ internal partial class Unpack else { //x memcpy(Mem,Window+BlockStart,FirstPartLength); - Utility.Copy(Window, BlockStart, Mem, 0, FirstPartLength); + Buffer.BlockCopy( + Window, + (int)BlockStart, + Mem, + 0, + (int)FirstPartLength + ); //x memcpy(Mem+FirstPartLength,Window,BlockEnd); - Utility.Copy(Window, 0, Mem, FirstPartLength, BlockEnd); + Buffer.BlockCopy( + Window, + 0, + Mem, + (int)FirstPartLength, + (int)BlockEnd + ); } } @@ -490,6 +488,7 @@ internal partial class Unpack if (Filters[I].Type == FILTER_NONE) { + ReturnFilter(Filters[I]); EmptyCount++; } } @@ -531,7 +530,6 @@ internal partial class Unpack { case FILTER_E8: case FILTER_E8E9: - { var FileOffset = (uint)WrittenFileSize; @@ -570,7 +568,6 @@ internal partial class Unpack } return SrcData; case FILTER_ARM: - { var FileOffset = (uint)WrittenFileSize; // DataSize is unsigned, so we use "CurPos+3" and not "DataSize-3" @@ -682,9 +679,9 @@ internal partial class Unpack } } - private bool ReadBlockHeader(BitInput Inp, ref UnpackBlockHeader Header) + private bool ReadBlockHeader(BitInput Inp) { - Header.HeaderSize = 0; + BlockHeader.HeaderSize = 0; if (!Inp.ExternalBuffer && Inp.InAddr > ReadTop - 7) { @@ -705,9 +702,9 @@ internal partial class Unpack return false; } - Header.HeaderSize = (int)(2 + ByteCount); + BlockHeader.HeaderSize = (int)(2 + ByteCount); - Header.BlockBitSize = (BlockFlags & 7) + 1; + BlockHeader.BlockBitSize = (BlockFlags & 7) + 1; var SavedCheckSum = (byte)(Inp.fgetbits() >> 8); Inp.faddbits(8); @@ -719,28 +716,24 @@ internal partial class Unpack Inp.addbits(8); } - Header.BlockSize = BlockSize; + BlockHeader.BlockSize = BlockSize; var CheckSum = (byte)(0x5a ^ BlockFlags ^ BlockSize ^ (BlockSize >> 8) ^ (BlockSize >> 16)); if (CheckSum != SavedCheckSum) { return false; } - Header.BlockStart = Inp.InAddr; - ReadBorder = Math.Min(ReadBorder, Header.BlockStart + Header.BlockSize - 1); + BlockHeader.BlockStart = Inp.InAddr; + ReadBorder = Math.Min(ReadBorder, BlockHeader.BlockStart + BlockHeader.BlockSize - 1); - Header.LastBlockInFile = (BlockFlags & 0x40) != 0; - Header.TablePresent = (BlockFlags & 0x80) != 0; + BlockHeader.LastBlockInFile = (BlockFlags & 0x40) != 0; + BlockHeader.TablePresent = (BlockFlags & 0x80) != 0; return true; } - private bool ReadTables( - BitInput Inp, - ref UnpackBlockHeader Header, - ref UnpackBlockTables Tables - ) + private bool ReadTables(BitInput Inp) { - if (!Header.TablePresent) + if (!BlockHeader.TablePresent) { return true; } @@ -753,8 +746,8 @@ internal partial class Unpack } } - var BitLength = new byte[BC]; - for (uint I = 0; I < BC; I++) + Span BitLength = stackalloc byte[checked((int)BC)]; + for (int I = 0; I < BC; I++) { uint Length = (byte)(Inp.fgetbits() >> 12); Inp.faddbits(4); @@ -783,11 +776,11 @@ internal partial class Unpack } } - MakeDecodeTables(BitLength, 0, Tables.BD, BC); + MakeDecodeTables(BitLength, 0, BlockTables.BD, BC); - var Table = new byte[HUFF_TABLE_SIZE]; - const uint TableSize = HUFF_TABLE_SIZE; - for (uint I = 0; I < TableSize; ) + Span Table = stackalloc byte[checked((int)HUFF_TABLE_SIZE)]; + const int TableSize = checked((int)HUFF_TABLE_SIZE); + for (int I = 0; I < TableSize; ) { if (!Inp.ExternalBuffer && Inp.InAddr > ReadTop - 5) { @@ -797,7 +790,7 @@ internal partial class Unpack } } - var Number = DecodeNumber(Inp, Tables.BD); + var Number = DecodeNumber(Inp, BlockTables.BD); if (Number < 16) { Table[I] = (byte)Number; @@ -859,14 +852,42 @@ internal partial class Unpack return false; } - MakeDecodeTables(Table, 0, Tables.LD, NC); - MakeDecodeTables(Table, (int)NC, Tables.DD, DC); - MakeDecodeTables(Table, (int)(NC + DC), Tables.LDD, LDC); - MakeDecodeTables(Table, (int)(NC + DC + LDC), Tables.RD, RC); + MakeDecodeTables(Table, 0, BlockTables.LD, NC); + MakeDecodeTables(Table, (int)NC, BlockTables.DD, DC); + MakeDecodeTables(Table, (int)(NC + DC), BlockTables.LDD, LDC); + MakeDecodeTables(Table, (int)(NC + DC + LDC), BlockTables.RD, RC); return true; } - private void InitFilters() => - //Filters.SoftReset(); + private UnpackFilter RentFilter() + { + if (FilterPool.Count == 0) + { + return new UnpackFilter(); + } + + var last = FilterPool.Count - 1; + var filter = FilterPool[last]; + FilterPool.RemoveAt(last); + return filter; + } + + private void ReturnFilter(UnpackFilter filter) + { + filter.Type = FILTER_NONE; + filter.BlockStart = 0; + filter.BlockLength = 0; + filter.Channels = 0; + filter.NextWindow = false; + FilterPool.Add(filter); + } + + private void InitFilters() + { + for (var i = 0; i < Filters.Count; i++) + { + ReturnFilter(Filters[i]); + } Filters.Clear(); + } } diff --git a/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack_cpp.cs b/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack_cpp.cs index f9a74440..72c808f9 100644 --- a/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack_cpp.cs +++ b/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack_cpp.cs @@ -1,22 +1,20 @@ #nullable disable -#if !Rar2017_64bit -using size_t = System.UInt32; -#else -using nint = System.Int64; -using nuint = System.UInt64; -using size_t = System.UInt64; -#endif - using System; +using System.Buffers; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common; -using static SharpCompress.Compressors.Rar.UnpackV2017.UnpackGlobal; using static SharpCompress.Compressors.Rar.UnpackV2017.PackDef; +using static SharpCompress.Compressors.Rar.UnpackV2017.UnpackGlobal; +using size_t = System.UInt32; namespace SharpCompress.Compressors.Rar.UnpackV2017; internal sealed partial class Unpack : BitInput { + private bool disposed; + public Unpack( /* ComprDataIO *DataIO */ ) //:Inp(true),VMCodeInp(true) @@ -30,12 +28,12 @@ internal sealed partial class Unpack : BitInput Suspended = false; UnpAllBuf = false; UnpSomeRead = false; -#if RarV2017_RAR_SMP - MaxUserThreads = 1; - UnpThreadPool = CreateThreadPool(); - ReadBufMT = null; - UnpThreadData = null; -#endif + /*#if RarV2017_RAR_SMP + MaxUserThreads = 1; + UnpThreadPool = CreateThreadPool(); + ReadBufMT = null; + UnpThreadData = null; + #endif*/ MaxWinSize = 0; MaxWinMask = 0; @@ -43,26 +41,40 @@ internal sealed partial class Unpack : BitInput // It prevents crash if first DoUnpack call is later made with wrong // (true) 'Solid' value. UnpInitData(false); -#if !RarV2017_SFX_MODULE // RAR 1.5 decompression initialization UnpInitData15(false); InitHuff(); -#endif } - // later: may need Dispose() if we support thread pool - //Unpack::~Unpack() - //{ - // InitFilters30(false); - // - // if (Window!=null) - // free(Window); - //#if RarV2017_RAR_SMP - // DestroyThreadPool(UnpThreadPool); - // delete[] ReadBufMT; - // delete[] UnpThreadData; - //#endif - //} + public override void Dispose() + { + if (disposed) + { + return; + } + + base.Dispose(); + if (Window != null) + { + ArrayPool.Shared.Return(Window); + Window = null; + } + + FragWindow.Reset(); + if (FilterSrcMemory.Length != 0) + { + ArrayPool.Shared.Return(FilterSrcMemory); + FilterSrcMemory = Array.Empty(); + } + + if (FilterDstMemory.Length != 0) + { + ArrayPool.Shared.Return(FilterDstMemory); + FilterDstMemory = Array.Empty(); + } + + disposed = true; + } private void Init(size_t WinSize, bool Solid) { @@ -111,54 +123,70 @@ internal sealed partial class Unpack : BitInput throw new InvalidFormatException("Grow && Fragmented"); } - var NewWindow = Fragmented ? null : new byte[WinSize]; - - if (NewWindow == null) + byte[] NewWindow = null; + try { - if (Grow || WinSize < 0x1000000) + NewWindow = Fragmented ? null : ArrayPool.Shared.Rent((int)WinSize); + + if (NewWindow == null) { - // We do not support growth for new fragmented window. - // Also exclude RAR4 and small dictionaries. - //throw std::bad_alloc(); - throw new InvalidFormatException("Grow || WinSize<0x1000000"); - } - else - { - if (Window != null) // If allocated by preceding files. + if (Grow || WinSize < 0x1000000) { - //free(Window); - Window = null; + // We do not support growth for new fragmented window. + // Also exclude RAR4 and small dictionaries. + //throw std::bad_alloc(); + throw new InvalidFormatException("Grow || WinSize<0x1000000"); } - FragWindow.Init(WinSize); - Fragmented = true; + else + { + if (Window != null) // If allocated by preceding files. + { + //free(Window); + ArrayPool.Shared.Return(Window); + Window = null; + } + + FragWindow.Init(WinSize); + Fragmented = true; + } + } + + if (!Fragmented) + { + // Clean the window to generate the same output when unpacking corrupt + // RAR files, which may access unused areas of sliding dictionary. + // sharpcompress: don't need this, freshly allocated above + //memset(NewWindow,0,WinSize); + + // If Window is not NULL, it means that window size has grown. + // In solid streams we need to copy data to a new window in such case. + // RAR archiving code does not allow it in solid streams now, + // but let's implement it anyway just in case we'll change it sometimes. + if (Grow) + { + for (size_t I = 1; I <= MaxWinSize; I++) + { + NewWindow[(UnpPtr - I) & (WinSize - 1)] = Window[ + (UnpPtr - I) & (MaxWinSize - 1) + ]; + } + } + + if (Window != null) + { + ArrayPool.Shared.Return(Window); + } + + Window = NewWindow; + NewWindow = null; } } - - if (!Fragmented) + finally { - // Clean the window to generate the same output when unpacking corrupt - // RAR files, which may access unused areas of sliding dictionary. - // sharpcompress: don't need this, freshly allocated above - //memset(NewWindow,0,WinSize); - - - // If Window is not NULL, it means that window size has grown. - // In solid streams we need to copy data to a new window in such case. - // RAR archiving code does not allow it in solid streams now, - // but let's implement it anyway just in case we'll change it sometimes. - if (Grow) + if (NewWindow != null) { - for (size_t I = 1; I <= MaxWinSize; I++) - { - NewWindow[(UnpPtr - I) & (WinSize - 1)] = Window[ - (UnpPtr - I) & (MaxWinSize - 1) - ]; - } + ArrayPool.Shared.Return(NewWindow); } - - //if (Window!=null) - // free(Window); - Window = NewWindow; } MaxWinSize = WinSize; @@ -172,7 +200,6 @@ internal sealed partial class Unpack : BitInput // just for extra safety. switch (Method) { -#if !RarV2017_SFX_MODULE case 15: // rar 1.5 compression if (!Fragmented) { @@ -188,33 +215,64 @@ internal sealed partial class Unpack : BitInput } break; -#endif -#if !RarV2017_RAR5ONLY case 29: // rar 3.x compression if (!Fragmented) { throw new NotImplementedException(); } + break; + case 50: // RAR 5.0 compression algorithm. + Unpack5(Solid); + break; +#if !Rar2017_NOSTRICT + default: + throw new InvalidFormatException("unknown compression method " + Method); +#endif + } + } + + private async ValueTask DoUnpackAsync( + uint Method, + bool Solid, + CancellationToken cancellationToken = default + ) + { + // Methods <50 will crash in Fragmented mode when accessing NULL Window. + // They cannot be called in such mode now, but we check it below anyway + // just for extra safety. + switch (Method) + { +#if !RarV2017_SFX_MODULE + case 15: // rar 1.5 compression + if (!Fragmented) + { + await Unpack15Async(Solid, cancellationToken).ConfigureAwait(false); + } + + break; + case 20: // rar 2.x compression + case 26: // files larger than 2GB + if (!Fragmented) + { + await Unpack20Async(Solid, cancellationToken).ConfigureAwait(false); + } + + break; +#endif +#if !RarV2017_RAR5ONLY + case 29: // rar 3.x compression + if (!Fragmented) + { + // TODO: Create Unpack29Async when ready + throw new NotImplementedException(); + } + break; #endif case 50: // RAR 5.0 compression algorithm. -#if RarV2017_RAR_SMP - if (MaxUserThreads > 1) - { - // We do not use the multithreaded unpack routine to repack RAR archives - // in 'suspended' mode, because unlike the single threaded code it can - // write more than one dictionary for same loop pass. So we would need - // larger buffers of unknown size. Also we do not support multithreading - // in fragmented window mode. - if (!Fragmented) - { - Unpack5MT(Solid); - break; - } - } -#endif - Unpack5(Solid); + // RAR 5.0 has full async support via UnpReadBufAsync and UnpWriteBuf + await Unpack5Async(Solid, cancellationToken).ConfigureAwait(false); break; #if !Rar2017_NOSTRICT default: @@ -238,6 +296,7 @@ internal sealed partial class Unpack : BitInput UnpPtr = WrPtr = 0; WriteBorder = Math.Min(MaxWinSize, UNPACK_MAX_WRITE) & MaxWinMask; } + // Filters never share several solid files, so we can safely reset them // even in solid archive. InitFilters(); @@ -260,17 +319,17 @@ internal sealed partial class Unpack : BitInput // LengthTable contains the length in bits for every element of alphabet. // Dec is the structure to decode Huffman code/ // Size is size of length table and DecodeNum field in Dec structure, - private void MakeDecodeTables(byte[] LengthTable, int offset, DecodeTable Dec, uint Size) + private void MakeDecodeTables(Span LengthTable, int offset, DecodeTable Dec, uint Size) { // Size of alphabet and DecodePos array. Dec.MaxNum = Size; // Calculate how many entries for every bit length in LengthTable we have. - var LengthCount = new uint[16]; + Span LengthCount = stackalloc uint[16]; //memset(LengthCount,0,sizeof(LengthCount)); for (size_t I = 0; I < Size; I++) { - LengthCount[LengthTable[offset + I] & 0xf]++; + LengthCount[LengthTable[checked((int)(offset + I))] & 0xf]++; } // We must not calculate the number of zero length codes. @@ -310,16 +369,16 @@ internal sealed partial class Unpack : BitInput // Prepare the copy of DecodePos. We'll modify this copy below, // so we cannot use the original DecodePos. - var CopyDecodePos = new uint[Dec.DecodePos.Length]; + Span CopyDecodePos = stackalloc uint[16]; //memcpy(CopyDecodePos,Dec->DecodePos,sizeof(CopyDecodePos)); - Array.Copy(Dec.DecodePos, CopyDecodePos, CopyDecodePos.Length); + Dec.DecodePos.AsSpan().CopyTo(CopyDecodePos); // For every bit length in the bit length table and so for every item // of alphabet. for (uint I = 0; I < Size; I++) { // Get the current bit length. - var _CurBitLength = (byte)(LengthTable[offset + I] & 0xf); + var _CurBitLength = (byte)(LengthTable[checked((int)(offset + I))] & 0xf); if (_CurBitLength != 0) { diff --git a/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpackinline_cpp.cs b/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpackinline_cpp.cs index 584d0b51..1b4e49e2 100644 --- a/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpackinline_cpp.cs +++ b/src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpackinline_cpp.cs @@ -1,9 +1,3 @@ -#if !Rar2017_64bit -#else -using nint = System.Int64; -using nuint = System.UInt64; -using size_t = System.UInt64; -#endif using static SharpCompress.Compressors.Rar.UnpackV2017.PackDef; namespace SharpCompress.Compressors.Rar.UnpackV2017; diff --git a/src/SharpCompress/Compressors/Rar/UnpackV2017/unpack_hpp.cs b/src/SharpCompress/Compressors/Rar/UnpackV2017/unpack_hpp.cs index 18cd9697..dd78b27c 100644 --- a/src/SharpCompress/Compressors/Rar/UnpackV2017/unpack_hpp.cs +++ b/src/SharpCompress/Compressors/Rar/UnpackV2017/unpack_hpp.cs @@ -1,16 +1,9 @@ -#if !Rar2017_64bit -using size_t = System.UInt32; -#else -using nint = System.Int64; -using nuint = System.UInt64; -using size_t = System.UInt64; -#endif -using int64 = System.Int64; - +using System; using System.Collections.Generic; using static SharpCompress.Compressors.Rar.UnpackV2017.PackDef; using static SharpCompress.Compressors.Rar.UnpackV2017.UnpackGlobal; -using System; +using int64 = System.Int64; +using size_t = System.UInt32; // TODO: REMOVE THIS... WIP #pragma warning disable 169 @@ -20,8 +13,6 @@ namespace SharpCompress.Compressors.Rar.UnpackV2017; internal static class UnpackGlobal { - - // Maximum allowed number of compressed bits processed in quick mode. public const int MAX_QUICK_DECODE_BITS = 10; @@ -97,11 +88,11 @@ internal struct UnpackBlockHeader internal struct UnpackBlockTables { - public DecodeTable LD; // Decode literals. - public DecodeTable DD; // Decode distances. + public DecodeTable LD; // Decode literals. + public DecodeTable DD; // Decode distances. public DecodeTable LDD; // Decode lower bits of distances. - public DecodeTable RD; // Decode repeating distances. - public DecodeTable BD; // Decode bit lengths in Huffman table. + public DecodeTable RD; // Decode repeating distances. + public DecodeTable BD; // Decode bit lengths in Huffman table. public void Init() { @@ -113,8 +104,7 @@ internal struct UnpackBlockTables } }; - -#if RarV2017_RAR_SMP +/*#if RarV2017_RAR_SMP enum UNP_DEC_TYPE { UNPDT_LITERAL,UNPDT_MATCH,UNPDT_FULLREP,UNPDT_REP,UNPDT_FILTER }; @@ -161,49 +151,38 @@ if (Decoded!=NULL) free(Decoded); } }; -#endif - +#endif*/ //struct UnpackFilter internal class UnpackFilter { - public byte Type; - public uint BlockStart; - public uint BlockLength; - public byte Channels; + internal byte Type; + internal uint BlockStart; + internal uint BlockLength; + internal byte Channels; + // uint Width; // byte PosR; - public bool NextWindow; -}; - - -//struct UnpackFilter30 -internal class UnpackFilter30 -{ - public uint BlockStart; - public uint BlockLength; - public bool NextWindow; - - // Position of parent filter in Filters array used as prototype for filter - // in PrgStack array. Not defined for filters in Filters array. - public uint ParentFilter; - - /*#if !RarV2017_RAR5ONLY - public VM_PreparedProgram Prg; - #endif*/ + internal bool NextWindow; }; internal class AudioVariables // For RAR 2.0 archives only. { - public int K1, K2, K3, K4, K5; - public int D1, D2, D3, D4; + internal int K1, + K2, + K3, + K4, + K5; + internal int D1, + D2, + D3, + D4; public int LastDelta; public readonly uint[] Dif = new uint[11]; public uint ByteCount; public int LastChar; }; - // We can use the fragmented dictionary in case heap does not have the single // large enough memory block. It is slower than normal dictionary. internal partial class FragmentedWindow @@ -223,10 +202,8 @@ internal partial class FragmentedWindow //size_t GetBlockSize(size_t StartPos,size_t RequiredSize); }; - internal partial class Unpack { - //void Unpack5(bool Solid); //void Unpack5MT(bool Solid); //bool UnpReadBuf(); @@ -254,22 +231,23 @@ internal partial class Unpack //BitInput Inp; private BitInput Inp => this; // hopefully this gets inlined -#if RarV2017_RAR_SMP -void InitMT(); -bool UnpackLargeBlock(UnpackThreadData &D); -bool ProcessDecoded(UnpackThreadData &D); + /*#if RarV2017_RAR_SMP + void InitMT(); + bool UnpackLargeBlock(UnpackThreadData &D); + bool ProcessDecoded(UnpackThreadData &D); -ThreadPool *UnpThreadPool; -UnpackThreadData *UnpThreadData; -uint MaxUserThreads; -byte *ReadBufMT; -#endif + ThreadPool *UnpThreadPool; + UnpackThreadData *UnpThreadData; + uint MaxUserThreads; + byte *ReadBufMT; + #endif*/ private byte[] FilterSrcMemory = Array.Empty(); private byte[] FilterDstMemory = Array.Empty(); // Filters code, one entry per filter. - private readonly List Filters = new List(); + private readonly List Filters = new(); + private readonly List FilterPool = new(); private readonly uint[] OldDist = new uint[4]; private uint OldDistPtr; @@ -279,7 +257,8 @@ byte *ReadBufMT; // array. In RAR3 last distance is always stored in OldDist[0]. private uint LastDist; - private size_t UnpPtr, WrPtr; + private size_t UnpPtr, + WrPtr; // Top border of read packed data. private int ReadTop; @@ -296,7 +275,7 @@ byte *ReadBufMT; private byte[] Window; - private readonly FragmentedWindow FragWindow = new FragmentedWindow(); + private readonly FragmentedWindow FragWindow = new(); private bool Fragmented; private int64 DestUnpSize; @@ -307,7 +286,6 @@ byte *ReadBufMT; private int64 WrittenFileSize; private bool FileExtracted; - /***************************** Unpack v 1.5 *********************************/ //void Unpack15(bool Solid); //void ShortLZ(); @@ -320,12 +298,29 @@ byte *ReadBufMT; //void CopyString15(uint Distance,uint Length); //uint DecodeNum(uint Num,uint StartPos,uint *DecTab,uint *PosTab); - private readonly ushort[] ChSet = new ushort[256], ChSetA = new ushort[256], ChSetB = new ushort[256], ChSetC = new ushort[256]; - private readonly byte[] NToPl = new byte[256], NToPlB = new byte[256], NToPlC = new byte[256]; - private uint FlagBuf, AvrPlc, AvrPlcB, AvrLn1, AvrLn2, AvrLn3; - private int Buf60, NumHuf, StMode, LCount, FlagsCnt; + private readonly ushort[] ChSet = new ushort[256], + ChSetA = new ushort[256], + ChSetB = new ushort[256], + ChSetC = new ushort[256]; + private readonly byte[] NToPl = new byte[256], + NToPlB = new byte[256], + NToPlC = new byte[256]; + private uint FlagBuf, + AvrPlc, + AvrPlcB, + AvrLn1, + AvrLn2, + AvrLn3; + private int Buf60, + NumHuf, + StMode, + LCount, + FlagsCnt; + + private uint Nhfb, + Nlzb, + MaxDist3; - private uint Nhfb, Nlzb, MaxDist3; /***************************** Unpack v 1.5 *********************************/ /***************************** Unpack v 2.0 *********************************/ @@ -335,9 +330,11 @@ byte *ReadBufMT; private readonly byte[] UnpOldTable20 = new byte[MC20 * 4]; private bool UnpAudioBlock; - private uint UnpChannels, UnpCurChannel; + private uint UnpChannels, + UnpCurChannel; private int UnpChannelDelta; + //void CopyString20(uint Length,uint Distance); //bool ReadTables20(); //void UnpWriteBuf20(); @@ -345,6 +342,7 @@ byte *ReadBufMT; //void ReadLastTables(); //byte DecodeAudio(int Delta); private AudioVariables[] AudV = new AudioVariables[4]; + /***************************** Unpack v 2.0 *********************************/ /***************************** Unpack v 3.0 *********************************/ @@ -356,43 +354,22 @@ byte *ReadBufMT; #endif*/ private int PPMEscChar; - private readonly byte[] UnpOldTable = new byte[HUFF_TABLE_SIZE30]; - // If we already read decoding tables for Unpack v2,v3,v5. // We should not use a single variable for all algorithm versions, // because we can have a corrupt archive with one algorithm file // followed by another algorithm file with "solid" flag and we do not // want to reuse tables from one algorithm in another. - private bool TablesRead2, TablesRead5; + private bool TablesRead2, + TablesRead5; - // Virtual machine to execute filters code. - /*#if !RarV2017_RAR5ONLY - RarVM VM; + /*#if RarV2017_RAR_SMP + // More than 8 threads are unlikely to provide a noticeable gain + // for unpacking, but would use the additional memory. + void SetThreads(uint Threads) {MaxUserThreads=Min(Threads,8);} + + void UnpackDecode(UnpackThreadData &D); #endif*/ - // Buffer to read VM filters code. We moved it here from AddVMCode - // function to reduce time spent in BitInput constructor. - private readonly BitInput VMCodeInp = new BitInput(true); - - // Filters code, one entry per filter. - private readonly List Filters30 = new List(); - - // Filters stack, several entrances of same filter are possible. - private readonly List PrgStack = new List(); - - // Lengths of preceding data blocks, one length of one last block - // for every filter. Used to reduce the size required to write - // the data block length if lengths are repeating. - private readonly List OldFilterLengths = new List(); - -#if RarV2017_RAR_SMP -// More than 8 threads are unlikely to provide a noticeable gain -// for unpacking, but would use the additional memory. -void SetThreads(uint Threads) {MaxUserThreads=Min(Threads,8);} - -void UnpackDecode(UnpackThreadData &D); -#endif - private size_t MaxWinSize; private size_t MaxWinMask; } diff --git a/src/SharpCompress/Compressors/Rar/VM/BitInput.cs b/src/SharpCompress/Compressors/Rar/VM/BitInput.cs index fe349fe7..a35e05fb 100644 --- a/src/SharpCompress/Compressors/Rar/VM/BitInput.cs +++ b/src/SharpCompress/Compressors/Rar/VM/BitInput.cs @@ -1,6 +1,9 @@ +using System; +using System.Buffers; + namespace SharpCompress.Compressors.Rar.VM; -internal class BitInput +internal class BitInput : IDisposable { /// the max size of the input internal const int MAX_SIZE = 0x8000; @@ -19,10 +22,11 @@ internal class BitInput get => inBit; set => inBit = value; } - public bool ExternalBuffer; + private readonly byte[] _privateBuffer = ArrayPool.Shared.Rent(MAX_SIZE); + private bool _disposed; /// - internal BitInput() => InBuf = new byte[MAX_SIZE]; + internal BitInput() => InBuf = _privateBuffer; internal byte[] InBuf { get; } @@ -87,4 +91,14 @@ internal class BitInput /// true if an Oververflow would occur /// internal bool Overflow(int IncPtr) => (inAddr + IncPtr >= MAX_SIZE); + + public virtual void Dispose() + { + if (_disposed) + { + return; + } + ArrayPool.Shared.Return(_privateBuffer); + _disposed = true; + } } diff --git a/src/SharpCompress/Compressors/Rar/VM/RarVM.cs b/src/SharpCompress/Compressors/Rar/VM/RarVM.cs index 48b25c5c..207063a0 100644 --- a/src/SharpCompress/Compressors/Rar/VM/RarVM.cs +++ b/src/SharpCompress/Compressors/Rar/VM/RarVM.cs @@ -1,6 +1,5 @@ -#nullable disable - using System; +using System.Buffers; using System.Buffers.Binary; using System.Collections.Generic; @@ -16,7 +15,9 @@ internal sealed class RarVM : BitInput // Mem.set_Renamed(offset + 3, Byte.valueOf((sbyte) ((Utility.URShift(value_Renamed, 24)) & 0xff))); //} - internal byte[] Mem { get; private set; } + internal byte[] Mem => _memory.NotNull(); + + private byte[]? _memory = ArrayPool.Shared.Rent(VM_MEMSIZE + 4); public const int VM_MEMSIZE = 0x40000; @@ -30,6 +31,17 @@ internal sealed class RarVM : BitInput public const int VM_FIXEDGLOBALSIZE = 64; private const int regCount = 8; + private static readonly VMStandardFilterSignature[] StandardFilterSignatures = + { + new(53, 0xad576887, VMStandardFilters.VMSF_E8), + new(57, 0x3cd7e57e, VMStandardFilters.VMSF_E8E9), + new(120, 0x3769893f, VMStandardFilters.VMSF_ITANIUM), + new(29, 0x0e06077d, VMStandardFilters.VMSF_DELTA), + new(149, 0x1c2c5dc8, VMStandardFilters.VMSF_RGB), + new(216, 0xbc85e701, VMStandardFilters.VMSF_AUDIO), + new(40, 0x46b9c560, VMStandardFilters.VMSF_UPCASE), + }; + private readonly int[] R = new int[regCount]; private VMFlags flags; @@ -40,11 +52,18 @@ internal sealed class RarVM : BitInput private int IP; - internal RarVM() => - //InitBlock(); - Mem = null; + internal RarVM() { } - internal void init() => Mem ??= new byte[VM_MEMSIZE + 4]; + public override void Dispose() + { + base.Dispose(); + if (_memory is null) + { + return; + } + ArrayPool.Shared.Return(_memory); + _memory = null; + } private bool IsVMMem(byte[] mem) => Mem == mem; @@ -228,7 +247,6 @@ internal sealed class RarVM : BitInput break; case VMCommands.VM_CMP: - { var value1 = (VMFlags)GetValue(cmd.IsByteMode, Mem, op1); var result = value1 - GetValue(cmd.IsByteMode, Mem, op2); @@ -247,7 +265,6 @@ internal sealed class RarVM : BitInput break; case VMCommands.VM_CMPB: - { var value1 = (VMFlags)GetValue(true, Mem, op1); var result = value1 - GetValue(true, Mem, op2); @@ -265,7 +282,6 @@ internal sealed class RarVM : BitInput break; case VMCommands.VM_CMPD: - { var value1 = (VMFlags)GetValue(false, Mem, op1); var result = value1 - GetValue(false, Mem, op2); @@ -283,7 +299,6 @@ internal sealed class RarVM : BitInput break; case VMCommands.VM_ADD: - { var value1 = GetValue(cmd.IsByteMode, Mem, op1); var result = (int)( @@ -351,7 +366,6 @@ internal sealed class RarVM : BitInput break; case VMCommands.VM_SUB: - { var value1 = GetValue(cmd.IsByteMode, Mem, op1); var result = (int)( @@ -411,7 +425,6 @@ internal sealed class RarVM : BitInput break; case VMCommands.VM_INC: - { var result = (int)(GetValue(cmd.IsByteMode, Mem, op1) & (0xFFffFFffL + 1L)); if (cmd.IsByteMode) @@ -440,7 +453,6 @@ internal sealed class RarVM : BitInput break; case VMCommands.VM_DEC: - { var result = (int)(GetValue(cmd.IsByteMode, Mem, op1) & (0xFFffFFff - 1)); SetValue(cmd.IsByteMode, Mem, op1, result); @@ -463,7 +475,6 @@ internal sealed class RarVM : BitInput continue; case VMCommands.VM_XOR: - { var result = GetValue(cmd.IsByteMode, Mem, op1) ^ GetValue(cmd.IsByteMode, Mem, op2); @@ -475,7 +486,6 @@ internal sealed class RarVM : BitInput break; case VMCommands.VM_AND: - { var result = GetValue(cmd.IsByteMode, Mem, op1) & GetValue(cmd.IsByteMode, Mem, op2); @@ -487,7 +497,6 @@ internal sealed class RarVM : BitInput break; case VMCommands.VM_OR: - { var result = GetValue(cmd.IsByteMode, Mem, op1) | GetValue(cmd.IsByteMode, Mem, op2); @@ -499,7 +508,6 @@ internal sealed class RarVM : BitInput break; case VMCommands.VM_TEST: - { var result = GetValue(cmd.IsByteMode, Mem, op1) & GetValue(cmd.IsByteMode, Mem, op2); @@ -578,7 +586,6 @@ internal sealed class RarVM : BitInput break; case VMCommands.VM_SHL: - { var value1 = GetValue(cmd.IsByteMode, Mem, op1); var value2 = GetValue(cmd.IsByteMode, Mem, op2); @@ -596,7 +603,6 @@ internal sealed class RarVM : BitInput break; case VMCommands.VM_SHR: - { var value1 = GetValue(cmd.IsByteMode, Mem, op1); var value2 = GetValue(cmd.IsByteMode, Mem, op2); @@ -610,7 +616,6 @@ internal sealed class RarVM : BitInput break; case VMCommands.VM_SAR: - { var value1 = GetValue(cmd.IsByteMode, Mem, op1); var value2 = GetValue(cmd.IsByteMode, Mem, op2); @@ -624,7 +629,6 @@ internal sealed class RarVM : BitInput break; case VMCommands.VM_NEG: - { var result = -GetValue(cmd.IsByteMode, Mem, op1); flags = (VMFlags)( @@ -645,7 +649,6 @@ internal sealed class RarVM : BitInput break; case VMCommands.VM_PUSHA: - { for (int i = 0, SP = R[7] - 4; i < regCount; i++, SP -= 4) { @@ -656,7 +659,6 @@ internal sealed class RarVM : BitInput break; case VMCommands.VM_POPA: - { for (int i = 0, SP = R[7]; i < regCount; i++, SP += 4) { @@ -684,7 +686,6 @@ internal sealed class RarVM : BitInput break; case VMCommands.VM_XCHG: - { var value1 = GetValue(cmd.IsByteMode, Mem, op1); SetValue(cmd.IsByteMode, Mem, op1, GetValue(cmd.IsByteMode, Mem, op2)); @@ -693,7 +694,6 @@ internal sealed class RarVM : BitInput break; case VMCommands.VM_MUL: - { var result = (int)( ( @@ -707,7 +707,6 @@ internal sealed class RarVM : BitInput break; case VMCommands.VM_DIV: - { var divider = GetValue(cmd.IsByteMode, Mem, op2); if (divider != 0) @@ -719,7 +718,6 @@ internal sealed class RarVM : BitInput break; case VMCommands.VM_ADC: - { var value1 = GetValue(cmd.IsByteMode, Mem, op1); var FC = (int)(flags & VMFlags.VM_FC); @@ -749,7 +747,6 @@ internal sealed class RarVM : BitInput break; case VMCommands.VM_SBB: - { var value1 = GetValue(cmd.IsByteMode, Mem, op1); var FC = (int)(flags & VMFlags.VM_FC); @@ -798,14 +795,15 @@ internal sealed class RarVM : BitInput } } - public void prepare(byte[] code, int codeSize, VMPreparedProgram prg) + public void prepare(ReadOnlySpan code, VMPreparedProgram prg) { InitBitInput(); + var codeSize = code.Length; var cpLength = Math.Min(MAX_SIZE, codeSize); // memcpy(inBuf,Code,Min(CodeSize,BitInput::MAX_SIZE)); - Buffer.BlockCopy(code, 0, InBuf, 0, cpLength); + code.Slice(0, cpLength).CopyTo(InBuf); byte xorSum = 0; for (var i = 1; i < codeSize; i++) { @@ -817,7 +815,7 @@ internal sealed class RarVM : BitInput prg.CommandCount = 0; if (xorSum == code[0]) { - var filterType = IsStandardFilter(code, codeSize); + var filterType = IsStandardFilter(code); if (filterType != VMStandardFilters.VMSF_NONE) { var curCmd = new VMPreparedCommand(); @@ -1127,24 +1125,17 @@ internal sealed class RarVM : BitInput } } - private VMStandardFilters IsStandardFilter(byte[] code, int codeSize) + private VMStandardFilters IsStandardFilter(ReadOnlySpan code) { - VMStandardFilterSignature[] stdList = - { - new VMStandardFilterSignature(53, 0xad576887, VMStandardFilters.VMSF_E8), - new VMStandardFilterSignature(57, 0x3cd7e57e, VMStandardFilters.VMSF_E8E9), - new VMStandardFilterSignature(120, 0x3769893f, VMStandardFilters.VMSF_ITANIUM), - new VMStandardFilterSignature(29, 0x0e06077d, VMStandardFilters.VMSF_DELTA), - new VMStandardFilterSignature(149, 0x1c2c5dc8, VMStandardFilters.VMSF_RGB), - new VMStandardFilterSignature(216, 0xbc85e701, VMStandardFilters.VMSF_AUDIO), - new VMStandardFilterSignature(40, 0x46b9c560, VMStandardFilters.VMSF_UPCASE) - }; var CodeCRC = RarCRC.CheckCrc(0xffffffff, code, 0, code.Length) ^ 0xffffffff; - for (var i = 0; i < stdList.Length; i++) + for (var i = 0; i < StandardFilterSignatures.Length; i++) { - if (stdList[i].CRC == CodeCRC && stdList[i].Length == code.Length) + if ( + StandardFilterSignatures[i].CRC == CodeCRC + && StandardFilterSignatures[i].Length == code.Length + ) { - return (stdList[i].Type); + return (StandardFilterSignatures[i].Type); } } return (VMStandardFilters.VMSF_NONE); @@ -1152,11 +1143,11 @@ internal sealed class RarVM : BitInput private void ExecuteStandardFilter(VMStandardFilters filterType) { + var mem = Mem; switch (filterType) { case VMStandardFilters.VMSF_E8: case VMStandardFilters.VMSF_E8E9: - { var dataSize = R[4]; long fileOffset = R[6] & unchecked((int)0xFFffFFff); @@ -1171,7 +1162,7 @@ internal sealed class RarVM : BitInput ); for (var curPos = 0; curPos < dataSize - 4; ) { - var curByte = Mem[curPos++]; + var curByte = mem[curPos++]; if (curByte == 0xe8 || curByte == cmpByte2) { // #ifdef PRESENT_INT32 @@ -1187,19 +1178,19 @@ internal sealed class RarVM : BitInput // SET_VALUE(false,Data,Addr-Offset); // #else var offset = curPos + fileOffset; - long Addr = GetValue(false, Mem, curPos); + long Addr = GetValue(false, mem, curPos); if ((Addr & unchecked((int)0x80000000)) != 0) { if (((Addr + offset) & unchecked((int)0x80000000)) == 0) { - SetValue(false, Mem, curPos, (int)Addr + fileSize); + SetValue(false, mem, curPos, (int)Addr + fileSize); } } else { if (((Addr - fileSize) & unchecked((int)0x80000000)) != 0) { - SetValue(false, Mem, curPos, (int)(Addr - offset)); + SetValue(false, mem, curPos, (int)(Addr - offset)); } } @@ -1211,7 +1202,6 @@ internal sealed class RarVM : BitInput break; case VMStandardFilters.VMSF_ITANIUM: - { var dataSize = R[4]; long fileOffset = R[6] & unchecked((int)0xFFffFFff); @@ -1228,7 +1218,7 @@ internal sealed class RarVM : BitInput while (curPos < dataSize - 21) { - var Byte = (Mem[curPos] & 0x1f) - 0x10; + var Byte = (mem[curPos] & 0x1f) - 0x10; if (Byte >= 0) { var cmdMask = Masks[Byte]; @@ -1269,13 +1259,12 @@ internal sealed class RarVM : BitInput break; case VMStandardFilters.VMSF_DELTA: - { var dataSize = R[4] & unchecked((int)0xFFffFFff); var channels = R[0] & unchecked((int)0xFFffFFff); var srcPos = 0; var border = (dataSize * 2) & unchecked((int)0xFFffFFff); - SetValue(false, Mem, VM_GLOBALMEMADDR + 0x20, dataSize); + SetValue(false, mem, VM_GLOBALMEMADDR + 0x20, dataSize); if (dataSize >= VM_GLOBALMEMADDR / 2) { break; @@ -1293,14 +1282,13 @@ internal sealed class RarVM : BitInput destPos += channels ) { - Mem[destPos] = (PrevByte = (byte)(PrevByte - Mem[srcPos++])); + mem[destPos] = (PrevByte = (byte)(PrevByte - mem[srcPos++])); } } } break; case VMStandardFilters.VMSF_RGB: - { // byte *SrcData=Mem,*DestData=SrcData+DataSize; int dataSize = R[4], @@ -1309,7 +1297,7 @@ internal sealed class RarVM : BitInput var channels = 3; var srcPos = 0; var destDataPos = dataSize; - SetValue(false, Mem, VM_GLOBALMEMADDR + 0x20, dataSize); + SetValue(false, mem, VM_GLOBALMEMADDR + 0x20, dataSize); if (dataSize >= VM_GLOBALMEMADDR / 2 || posR < 0) { break; @@ -1325,8 +1313,8 @@ internal sealed class RarVM : BitInput if (upperPos >= 3) { var upperDataPos = destDataPos + upperPos; - var upperByte = Mem[upperDataPos] & 0xff; - var upperLeftByte = Mem[upperDataPos - 3] & 0xff; + var upperByte = mem[upperDataPos] & 0xff; + var upperLeftByte = mem[upperDataPos - 3] & 0xff; predicted = prevByte + upperByte - upperLeftByte; var pa = Math.Abs((int)(predicted - prevByte)); var pb = Math.Abs((int)(predicted - upperByte)); @@ -1352,21 +1340,20 @@ internal sealed class RarVM : BitInput predicted = prevByte; } - prevByte = ((predicted - Mem[srcPos++]) & 0xff) & 0xff; - Mem[destDataPos + i] = (byte)(prevByte & 0xff); + prevByte = ((predicted - mem[srcPos++]) & 0xff) & 0xff; + mem[destDataPos + i] = (byte)(prevByte & 0xff); } } for (int i = posR, border = dataSize - 2; i < border; i += 3) { - var G = Mem[destDataPos + i + 1]; - Mem[destDataPos + i] = (byte)(Mem[destDataPos + i] + G); - Mem[destDataPos + i + 2] = (byte)(Mem[destDataPos + i + 2] + G); + var G = mem[destDataPos + i + 1]; + mem[destDataPos + i] = (byte)(mem[destDataPos + i] + G); + mem[destDataPos + i + 2] = (byte)(mem[destDataPos + i + 2] + G); } } break; case VMStandardFilters.VMSF_AUDIO: - { int dataSize = R[4], channels = R[0]; @@ -1374,7 +1361,7 @@ internal sealed class RarVM : BitInput var destDataPos = dataSize; //byte *SrcData=Mem,*DestData=SrcData+DataSize; - SetValue(false, Mem, VM_GLOBALMEMADDR + 0x20, dataSize); + SetValue(false, mem, VM_GLOBALMEMADDR + 0x20, dataSize); if (dataSize >= VM_GLOBALMEMADDR / 2) { break; @@ -1404,10 +1391,10 @@ internal sealed class RarVM : BitInput var predicted = (8 * prevByte) + (K1 * D1) + (K2 * D2) + (K3 * D3); predicted = Utility.URShift(predicted, 3) & 0xff; - long curByte = Mem[srcPos++]; + long curByte = mem[srcPos++]; predicted -= curByte; - Mem[destDataPos + i] = (byte)predicted; + mem[destDataPos + i] = (byte)predicted; prevDelta = (byte)(predicted - prevByte); //fix java byte @@ -1497,7 +1484,6 @@ internal sealed class RarVM : BitInput break; case VMStandardFilters.VMSF_UPCASE: - { int dataSize = R[4], srcPos = 0, @@ -1508,15 +1494,15 @@ internal sealed class RarVM : BitInput } while (srcPos < dataSize) { - var curByte = Mem[srcPos++]; - if (curByte == 2 && (curByte = Mem[srcPos++]) != 2) + var curByte = mem[srcPos++]; + if (curByte == 2 && (curByte = mem[srcPos++]) != 2) { curByte = (byte)(curByte - 32); } - Mem[destPos++] = curByte; + mem[destPos++] = curByte; } - SetValue(false, Mem, VM_GLOBALMEMADDR + 0x1c, destPos - dataSize); - SetValue(false, Mem, VM_GLOBALMEMADDR + 0x20, dataSize); + SetValue(false, mem, VM_GLOBALMEMADDR + 0x1c, destPos - dataSize); + SetValue(false, mem, VM_GLOBALMEMADDR + 0x20, dataSize); } break; } @@ -1556,15 +1542,14 @@ internal sealed class RarVM : BitInput { if (pos < VM_MEMSIZE) { - //&& data!=Mem+Pos) - //memmove(Mem+Pos,Data,Min(DataSize,VM_MEMSIZE-Pos)); - for (var i = 0; i < Math.Min(data.Length - offset, dataSize); i++) + // Use Array.Copy for fast bulk memory operations instead of byte-by-byte loop + // Calculate how much data can actually fit in VM memory + int copyLength = Math.Min(dataSize, VM_MEMSIZE - pos); + copyLength = Math.Min(copyLength, data.Length - offset); + + if (copyLength > 0) { - if ((VM_MEMSIZE - pos) < i) - { - break; - } - Mem[pos + i] = data[offset + i]; + Array.Copy(data, offset, Mem, pos, copyLength); } } } diff --git a/src/SharpCompress/Compressors/Rar/VM/VMCmdFlags.cs b/src/SharpCompress/Compressors/Rar/VM/VMCmdFlags.cs index 00de0b6a..9dbafe1b 100644 --- a/src/SharpCompress/Compressors/Rar/VM/VMCmdFlags.cs +++ b/src/SharpCompress/Compressors/Rar/VM/VMCmdFlags.cs @@ -53,6 +53,6 @@ internal class VMCmdFlags VMCF_OP2 | VMCF_BYTEMODE, VMCF_OP2 | VMCF_BYTEMODE | VMCF_USEFLAGS | VMCF_CHFLAGS, VMCF_OP2 | VMCF_BYTEMODE | VMCF_USEFLAGS | VMCF_CHFLAGS, - VMCF_OP0 + VMCF_OP0, }; } diff --git a/src/SharpCompress/Compressors/Rar/VM/VMCommands.cs b/src/SharpCompress/Compressors/Rar/VM/VMCommands.cs index 7bb1a396..a10eb072 100644 --- a/src/SharpCompress/Compressors/Rar/VM/VMCommands.cs +++ b/src/SharpCompress/Compressors/Rar/VM/VMCommands.cs @@ -66,5 +66,5 @@ internal enum VMCommands VM_NEGB = 52, VM_NEGD = 53, - VM_STANDARD = 54 + VM_STANDARD = 54, } diff --git a/src/SharpCompress/Compressors/Rar/VM/VMFlags.cs b/src/SharpCompress/Compressors/Rar/VM/VMFlags.cs index fa9a72e7..12765775 100644 --- a/src/SharpCompress/Compressors/Rar/VM/VMFlags.cs +++ b/src/SharpCompress/Compressors/Rar/VM/VMFlags.cs @@ -5,5 +5,5 @@ internal enum VMFlags None = 0, VM_FC = 1, VM_FZ = 2, - VM_FS = 80000000 + VM_FS = 80000000, } diff --git a/src/SharpCompress/Compressors/Rar/VM/VMOpType.cs b/src/SharpCompress/Compressors/Rar/VM/VMOpType.cs index bcc4214e..77994746 100644 --- a/src/SharpCompress/Compressors/Rar/VM/VMOpType.cs +++ b/src/SharpCompress/Compressors/Rar/VM/VMOpType.cs @@ -5,5 +5,5 @@ internal enum VMOpType VM_OPREG = 0, VM_OPINT = 1, VM_OPREGMEM = 2, - VM_OPNONE = 3 + VM_OPNONE = 3, } diff --git a/src/SharpCompress/Compressors/Rar/VM/VMPreparedProgram.cs b/src/SharpCompress/Compressors/Rar/VM/VMPreparedProgram.cs index c0006e09..4869a21f 100644 --- a/src/SharpCompress/Compressors/Rar/VM/VMPreparedProgram.cs +++ b/src/SharpCompress/Compressors/Rar/VM/VMPreparedProgram.cs @@ -4,13 +4,13 @@ namespace SharpCompress.Compressors.Rar.VM; internal class VMPreparedProgram { - internal List Commands = new List(); - internal List AltCommands = new List(); + internal List Commands = new(16); + internal List AltCommands = new(16); public int CommandCount { get; set; } - internal List GlobalData = new List(); - internal List StaticData = new List(); + internal List GlobalData = new(RarVM.VM_FIXEDGLOBALSIZE); + internal List StaticData = new(); // static data contained in DB operators internal int[] InitR = new int[7]; diff --git a/src/SharpCompress/Compressors/Rar/VM/VMStandardFilters.cs b/src/SharpCompress/Compressors/Rar/VM/VMStandardFilters.cs index 4b90b019..2c435ebb 100644 --- a/src/SharpCompress/Compressors/Rar/VM/VMStandardFilters.cs +++ b/src/SharpCompress/Compressors/Rar/VM/VMStandardFilters.cs @@ -9,5 +9,5 @@ internal enum VMStandardFilters VMSF_RGB = 4, VMSF_AUDIO = 5, VMSF_DELTA = 6, - VMSF_UPCASE = 7 + VMSF_UPCASE = 7, } diff --git a/src/SharpCompress/Compressors/Reduce/ReduceStream.Async.cs b/src/SharpCompress/Compressors/Reduce/ReduceStream.Async.cs new file mode 100644 index 00000000..69ccc85d --- /dev/null +++ b/src/SharpCompress/Compressors/Reduce/ReduceStream.Async.cs @@ -0,0 +1,220 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.IO; + +namespace SharpCompress.Compressors.Reduce; + +public partial class ReduceStream +{ + public static async ValueTask CreateAsync( + Stream inStr, + long compsize, + long unCompSize, + int factor, + CancellationToken cancellationToken = default + ) + { + var stream = new ReduceStream(inStr, compsize, unCompSize, factor); + await stream.LoadNextByteTableAsync(cancellationToken).ConfigureAwait(false); + return stream; + } + + private async ValueTask NEXTBYTEAsync(CancellationToken cancellationToken) + { + if (inByteCount == compressedSize) + { + _inputExhausted = true; + return EOF; + } + + byte[] buffer = new byte[1]; + int bytesRead = await inStream + .ReadAsync(buffer, 0, 1, cancellationToken) + .ConfigureAwait(false); + if (bytesRead == 0) + { + _inputExhausted = true; + return EOF; + } + + inByteCount++; + return buffer[0]; + } + + private async ValueTask READBITSAsync(int nbits, CancellationToken cancellationToken) + { + if (nbits > bitBufferCount) + { + int temp; + while (bitBufferCount <= 8 * (int)(4 - 1)) + { + temp = await NEXTBYTEAsync(cancellationToken).ConfigureAwait(false); + if (temp == EOF) + { + break; + } + bitBuffer |= (ulong)temp << bitBufferCount; + bitBufferCount += 8; + } + } + byte zdest = (byte)(bitBuffer & (ulong)mask_bits[nbits]); + bitBuffer >>= nbits; + bitBufferCount -= nbits; + return zdest; + } + + private async ValueTask LoadNextByteTableAsync(CancellationToken cancellationToken) + { + nextByteTable = new byte[256][]; + for (int x = 255; x >= 0; x--) + { + byte Slen = await READBITSAsync(6, cancellationToken).ConfigureAwait(false); + nextByteTable[x] = new byte[Slen]; + for (int i = 0; i < Slen; i++) + { + nextByteTable[x][i] = await READBITSAsync(8, cancellationToken) + .ConfigureAwait(false); + } + } + } + + private async ValueTask GetNextByteAsync(CancellationToken cancellationToken) + { + if (nextByteTable[outByte].Length == 0) + { + outByte = await READBITSAsync(8, cancellationToken).ConfigureAwait(false); + return outByte; + } + byte nextBit = await READBITSAsync(1, cancellationToken).ConfigureAwait(false); + if (nextBit == 1) + { + outByte = await READBITSAsync(8, cancellationToken).ConfigureAwait(false); + return outByte; + } + byte nextByteIndex = await READBITSAsync( + bitCountTable[nextByteTable[outByte].Length], + cancellationToken + ) + .ConfigureAwait(false); + if (nextByteIndex >= nextByteTable[outByte].Length) + { + throw new InvalidFormatException("ReduceStream: next byte table index out of range"); + } + outByte = nextByteTable[outByte][nextByteIndex]; + return outByte; + } + + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + int countIndex = 0; + while (countIndex < count && outBytesCount < unCompressedSize) + { + if (length == 0) + { + if (_inputExhausted && bitBufferCount <= 0) + { + throw new InvalidFormatException( + "ReduceStream: compressed data exhausted before uncompressed size reached" + ); + } + + byte nextByte = await GetNextByteAsync(cancellationToken).ConfigureAwait(false); + if (nextByte != RunLengthCode) + { + buffer[offset + (countIndex++)] = nextByte; + windowsBuffer[windowIndex++] = nextByte; + outBytesCount++; + if (windowIndex == WSIZE) + { + windowIndex = 0; + } + + continue; + } + + nextByte = await GetNextByteAsync(cancellationToken).ConfigureAwait(false); + if (nextByte == 0) + { + buffer[offset + (countIndex++)] = RunLengthCode; + windowsBuffer[windowIndex++] = RunLengthCode; + outBytesCount++; + if (windowIndex == WSIZE) + { + windowIndex = 0; + } + + continue; + } + + int lengthDistanceByte = nextByte; + length = lengthDistanceByte & lengthMask; + if (length == lengthMask) + { + length += await GetNextByteAsync(cancellationToken).ConfigureAwait(false); + } + length += 3; + + int distanceHighByte = (lengthDistanceByte << factor) & distanceMask; + distance = + windowIndex + - ( + distanceHighByte + + await GetNextByteAsync(cancellationToken).ConfigureAwait(false) + + 1 + ); + + distance &= WSIZE - 1; + } + + while (length != 0 && countIndex < count) + { + byte nextByte = windowsBuffer[distance++]; + buffer[offset + (countIndex++)] = nextByte; + windowsBuffer[windowIndex++] = nextByte; + outBytesCount++; + + if (distance == WSIZE) + { + distance = 0; + } + + if (windowIndex == WSIZE) + { + windowIndex = 0; + } + + length--; + } + } + + return countIndex; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + if (buffer.IsEmpty || outBytesCount >= unCompressedSize) + { + return 0; + } + + byte[] arrayBuffer = new byte[buffer.Length]; + int result = await ReadAsync(arrayBuffer, 0, arrayBuffer.Length, cancellationToken) + .ConfigureAwait(false); + arrayBuffer.AsMemory(0, result).CopyTo(buffer); + return result; + } +#endif +} diff --git a/src/SharpCompress/Compressors/Reduce/ReduceStream.cs b/src/SharpCompress/Compressors/Reduce/ReduceStream.cs new file mode 100644 index 00000000..f935619a --- /dev/null +++ b/src/SharpCompress/Compressors/Reduce/ReduceStream.cs @@ -0,0 +1,291 @@ +using System; +using System.IO; +using SharpCompress.Common; + +namespace SharpCompress.Compressors.Reduce; + +public partial class ReduceStream : Stream +{ + private readonly long unCompressedSize; + private readonly long compressedSize; + private readonly Stream inStream; + + private long inByteCount; + private const int EOF = 1234; + + private readonly int factor; + private readonly int distanceMask; + private readonly int lengthMask; + + private long outBytesCount; + + private readonly byte[] windowsBuffer; + private int windowIndex; + private int length; + private int distance; + + private ReduceStream(Stream inStr, long compsize, long unCompSize, int factor) + { + inStream = inStr; + compressedSize = compsize; + unCompressedSize = unCompSize; + inByteCount = 0; + outBytesCount = 0; + + this.factor = factor; + distanceMask = (int)mask_bits[factor] << 8; + lengthMask = 0xff >> factor; + + windowIndex = 0; + length = 0; + distance = 0; + + windowsBuffer = new byte[WSIZE]; + + outByte = 0; + + LoadBitLengthTable(); + } + + public static ReduceStream Create(Stream inStr, long compsize, long unCompSize, int factor) + { + var stream = new ReduceStream(inStr, compsize, unCompSize, factor); + stream.LoadNextByteTable(); + return stream; + } + + protected override void Dispose(bool disposing) + { + base.Dispose(disposing); + } + + public override void Flush() + { + throw new NotImplementedException(); + } + + public override long Seek(long offset, SeekOrigin origin) + { + throw new NotImplementedException(); + } + + public override void SetLength(long value) + { + throw new NotImplementedException(); + } + + public override void Write(byte[] buffer, int offset, int count) + { + throw new NotImplementedException(); + } + + public override bool CanRead => true; + public override bool CanSeek => false; + public override bool CanWrite => false; + public override long Length => unCompressedSize; + public override long Position + { + get => outBytesCount; + set { } + } + + private const int RunLengthCode = 144; + private const int WSIZE = 0x4000; + + private readonly uint[] mask_bits = new uint[] + { + 0x0000, + 0x0001, + 0x0003, + 0x0007, + 0x000f, + 0x001f, + 0x003f, + 0x007f, + 0x00ff, + 0x01ff, + 0x03ff, + 0x07ff, + 0x0fff, + 0x1fff, + 0x3fff, + 0x7fff, + 0xffff, + }; + + private int bitBufferCount; + private ulong bitBuffer; + private bool _inputExhausted; + + private int NEXTBYTE() + { + if (inByteCount == compressedSize) + { + _inputExhausted = true; + return EOF; + } + + inByteCount++; + int b = inStream.ReadByte(); + if (b < 0) + { + _inputExhausted = true; + return EOF; + } + return b; + } + + private void READBITS(int nbits, out byte zdest) + { + if (nbits > bitBufferCount) + { + int temp; + while (bitBufferCount <= 8 * (int)(4 - 1) && (temp = NEXTBYTE()) != EOF) + { + bitBuffer |= (ulong)temp << bitBufferCount; + bitBufferCount += 8; + } + } + zdest = (byte)(bitBuffer & (ulong)mask_bits[nbits]); + bitBuffer >>= nbits; + bitBufferCount -= nbits; + } + + private byte[] bitCountTable = []; + + private void LoadBitLengthTable() + { + byte[] bitPos = { 0, 2, 4, 8, 16, 32, 64, 128, 255 }; + bitCountTable = new byte[256]; + + for (byte i = 1; i <= 8; i++) + { + int vMin = bitPos[i - 1] + 1; + int vMax = bitPos[i]; + for (int j = vMin; j <= vMax; j++) + { + bitCountTable[j] = i; + } + } + } + + private byte[][] nextByteTable = []; + + private void LoadNextByteTable() + { + nextByteTable = new byte[256][]; + for (int x = 255; x >= 0; x--) + { + READBITS(6, out byte Slen); + nextByteTable[x] = new byte[Slen]; + for (int i = 0; i < Slen; i++) + { + READBITS(8, out nextByteTable[x][i]); + } + } + } + + private byte outByte; + + private byte GetNextByte() + { + if (nextByteTable[outByte].Length == 0) + { + READBITS(8, out outByte); + return outByte; + } + READBITS(1, out byte nextBit); + if (nextBit == 1) + { + READBITS(8, out outByte); + return outByte; + } + READBITS(bitCountTable[nextByteTable[outByte].Length], out byte nextByteIndex); + if (nextByteIndex >= nextByteTable[outByte].Length) + { + throw new InvalidFormatException("ReduceStream: next byte table index out of range"); + } + outByte = nextByteTable[outByte][nextByteIndex]; + return outByte; + } + + public override int Read(byte[] buffer, int offset, int count) + { + int countIndex = 0; + while (countIndex < count && outBytesCount < unCompressedSize) + { + if (length == 0) + { + if (_inputExhausted && bitBufferCount <= 0) + { + throw new InvalidFormatException( + "ReduceStream: compressed data exhausted before uncompressed size reached" + ); + } + + byte nextByte = GetNextByte(); + if (nextByte != RunLengthCode) + { + buffer[offset + (countIndex++)] = nextByte; + windowsBuffer[windowIndex++] = nextByte; + outBytesCount++; + if (windowIndex == WSIZE) + { + windowIndex = 0; + } + + continue; + } + + nextByte = GetNextByte(); + if (nextByte == 0) + { + buffer[offset + (countIndex++)] = RunLengthCode; + windowsBuffer[windowIndex++] = RunLengthCode; + outBytesCount++; + if (windowIndex == WSIZE) + { + windowIndex = 0; + } + + continue; + } + + int lengthDistanceByte = nextByte; + length = lengthDistanceByte & lengthMask; + if (length == lengthMask) + { + length += GetNextByte(); + } + length += 3; + + int distanceHighByte = (lengthDistanceByte << factor) & distanceMask; + distance = windowIndex - (distanceHighByte + GetNextByte() + 1); + + distance &= WSIZE - 1; + } + + while (length != 0 && countIndex < count) + { + byte nextByte = windowsBuffer[distance++]; + buffer[offset + (countIndex++)] = nextByte; + windowsBuffer[windowIndex++] = nextByte; + outBytesCount++; + + if (distance == WSIZE) + { + distance = 0; + } + + if (windowIndex == WSIZE) + { + windowIndex = 0; + } + + length--; + } + } + + return countIndex; + } +} diff --git a/src/SharpCompress/Compressors/Shrink/BitStream.cs b/src/SharpCompress/Compressors/Shrink/BitStream.cs new file mode 100644 index 00000000..77df3fa3 --- /dev/null +++ b/src/SharpCompress/Compressors/Shrink/BitStream.cs @@ -0,0 +1,79 @@ +namespace SharpCompress.Compressors.Shrink; + +internal class BitStream +{ + private const int EOF = 1234; + private byte[] _src; + private int _srcLen; + private int _byteIdx; + private int _bitIdx; + private int _bitsLeft; + private ulong _bitBuffer; + private static uint[] _maskBits = new uint[17] + { + 0U, + 1U, + 3U, + 7U, + 15U, + 31U, + 63U, + (uint)sbyte.MaxValue, + byte.MaxValue, + 511U, + 1023U, + 2047U, + 4095U, + 8191U, + 16383U, + (uint)short.MaxValue, + ushort.MaxValue, + }; + + public BitStream(byte[] src, int srcLen) + { + _src = src; + _srcLen = srcLen; + _byteIdx = 0; + _bitIdx = 0; + } + + public int BytesRead => (_byteIdx << 3) + _bitIdx; + + private int NextByte() + { + if (_byteIdx >= _srcLen) + { + return EOF; + } + + return _src[_byteIdx++]; + } + + public int NextBits(int nbits) + { + var result = 0; + if (nbits > _bitsLeft) + { + int num; + while (_bitsLeft <= 24 && (num = NextByte()) != 1234) + { + _bitBuffer |= (ulong)num << _bitsLeft; + _bitsLeft += 8; + } + } + result = (int)(_bitBuffer & _maskBits[nbits]); + _bitBuffer >>= nbits; + _bitsLeft -= nbits; + return result; + } + + public bool Advance(int count) + { + if (_byteIdx > _srcLen) + { + return false; + } + return true; + } +} diff --git a/src/SharpCompress/Compressors/Shrink/HwUnshrink.cs b/src/SharpCompress/Compressors/Shrink/HwUnshrink.cs new file mode 100644 index 00000000..81e411c1 --- /dev/null +++ b/src/SharpCompress/Compressors/Shrink/HwUnshrink.cs @@ -0,0 +1,430 @@ +using System; + +namespace SharpCompress.Compressors.Shrink; + +public class HwUnshrink +{ + private const int MIN_CODE_SIZE = 9; + private const int MAX_CODE_SIZE = 13; + + private const ushort MAX_CODE = (ushort)((1U << MAX_CODE_SIZE) - 1); + private const ushort INVALID_CODE = ushort.MaxValue; + private const ushort CONTROL_CODE = 256; + private const ushort INC_CODE_SIZE = 1; + private const ushort PARTIAL_CLEAR = 2; + + private const int HASH_BITS = MAX_CODE_SIZE + 1; // For a load factor of 0.5. + private const int HASHTAB_SIZE = 1 << HASH_BITS; + private const ushort UNKNOWN_LEN = ushort.MaxValue; + + private struct CodeTabEntry + { + public int prefixCode; // INVALID_CODE means the entry is invalid. + public byte extByte; + public ushort len; + public int lastDstPos; + } + + private static void CodeTabInit(CodeTabEntry[] codeTab) + { + for (var i = 0; i <= byte.MaxValue; i++) + { + codeTab[i].prefixCode = (ushort)i; + codeTab[i].extByte = (byte)i; + codeTab[i].len = 1; + } + + for (var i = byte.MaxValue + 1; i <= MAX_CODE; i++) + { + codeTab[i].prefixCode = INVALID_CODE; + } + } + + private static void UnshrinkPartialClear(CodeTabEntry[] codeTab, ref CodeQueue queue) + { + var isPrefix = new bool[MAX_CODE + 1]; + int codeQueueSize; + + // Scan for codes that have been used as a prefix. + for (var i = CONTROL_CODE + 1; i <= MAX_CODE; i++) + { + if (codeTab[i].prefixCode != INVALID_CODE) + { + isPrefix[codeTab[i].prefixCode] = true; + } + } + + // Clear "non-prefix" codes in the table; populate the code queue. + codeQueueSize = 0; + for (var i = CONTROL_CODE + 1; i <= MAX_CODE; i++) + { + if (!isPrefix[i]) + { + codeTab[i].prefixCode = INVALID_CODE; + queue.codes[codeQueueSize++] = (ushort)i; + } + } + + queue.codes[codeQueueSize] = INVALID_CODE; // End-of-queue marker. + queue.nextIdx = 0; + } + + private static bool ReadCode( + BitStream stream, + ref int codeSize, + CodeTabEntry[] codeTab, + ref CodeQueue queue, + out int nextCode + ) + { + int code, + controlCode; + + code = (int)stream.NextBits(codeSize); + if (!stream.Advance(codeSize)) + { + nextCode = INVALID_CODE; + return false; + } + + // Handle regular codes (the common case). + if (code != CONTROL_CODE) + { + nextCode = code; + return true; + } + + // Handle control codes. + controlCode = (ushort)stream.NextBits(codeSize); + if (!stream.Advance(codeSize)) + { + nextCode = INVALID_CODE; + return true; + } + + if (controlCode == INC_CODE_SIZE && codeSize < MAX_CODE_SIZE) + { + codeSize++; + return ReadCode(stream, ref codeSize, codeTab, ref queue, out nextCode); + } + + if (controlCode == PARTIAL_CLEAR) + { + UnshrinkPartialClear(codeTab, ref queue); + return ReadCode(stream, ref codeSize, codeTab, ref queue, out nextCode); + } + + nextCode = INVALID_CODE; + return true; + } + + private static void CopyFromPrevPos(byte[] dst, int prevPos, int dstPos, int len) + { + if (dstPos + len > dst.Length) + { + // Not enough room in dst for the sloppy copy below. + Array.Copy(dst, prevPos, dst, dstPos, len); + return; + } + + if (prevPos + len > dstPos) + { + // Benign one-byte overlap possible in the KwKwK case. + //assert(prevPos + len == dstPos + 1); + //assert(dst[prevPos] == dst[prevPos + len - 1]); + } + + Buffer.BlockCopy(dst, prevPos, dst, dstPos, len); + } + + private static UnshrnkStatus OutputCode( + int code, + byte[] dst, + int dstPos, + int dstCap, + int prevCode, + CodeTabEntry[] codeTab, + ref CodeQueue queue, + out byte firstByte, + out int len + ) + { + int prefixCode; + + //assert(code <= MAX_CODE && code != CONTROL_CODE); + //assert(dstPos < dstCap); + firstByte = 0; + if (code <= byte.MaxValue) + { + // Output literal byte. + firstByte = (byte)code; + len = 1; + dst[dstPos] = (byte)code; + return UnshrnkStatus.Ok; + } + + if (codeTab[code].prefixCode == INVALID_CODE || codeTab[code].prefixCode == code) + { + // Reject invalid codes. Self-referential codes may exist in the table but cannot be used. + firstByte = 0; + len = 0; + return UnshrnkStatus.Error; + } + + if (codeTab[code].len != UNKNOWN_LEN) + { + // Output string with known length (the common case). + if (dstCap - dstPos < codeTab[code].len) + { + firstByte = 0; + len = 0; + return UnshrnkStatus.Full; + } + + CopyFromPrevPos(dst, codeTab[code].lastDstPos, dstPos, codeTab[code].len); + firstByte = dst[dstPos]; + len = codeTab[code].len; + return UnshrnkStatus.Ok; + } + + // Output a string of unknown length. + //assert(codeTab[code].len == UNKNOWN_LEN); + prefixCode = codeTab[code].prefixCode; + // assert(prefixCode > CONTROL_CODE); + + if (prefixCode == queue.codes[queue.nextIdx]) + { + // The prefix code hasn't been added yet, but we were just about to: the KwKwK case. + //assert(codeTab[prevCode].prefixCode != INVALID_CODE); + codeTab[prefixCode].prefixCode = prevCode; + codeTab[prefixCode].extByte = firstByte; + codeTab[prefixCode].len = (ushort)(codeTab[prevCode].len + 1); + codeTab[prefixCode].lastDstPos = codeTab[prevCode].lastDstPos; + dst[dstPos] = firstByte; + } + else if (codeTab[prefixCode].prefixCode == INVALID_CODE) + { + // The prefix code is still invalid. + firstByte = 0; + len = 0; + return UnshrnkStatus.Error; + } + + // Output the prefix string, then the extension byte. + len = codeTab[prefixCode].len + 1; + if (dstCap - dstPos < len) + { + firstByte = 0; + len = 0; + return UnshrnkStatus.Full; + } + + CopyFromPrevPos(dst, codeTab[prefixCode].lastDstPos, dstPos, codeTab[prefixCode].len); + dst[dstPos + len - 1] = codeTab[code].extByte; + firstByte = dst[dstPos]; + + // Update the code table now that the string has a length and pos. + //assert(prevCode != code); + codeTab[code].len = (ushort)len; + codeTab[code].lastDstPos = dstPos; + + return UnshrnkStatus.Ok; + } + + public static UnshrnkStatus Unshrink( + byte[] src, + int srcLen, + out int srcUsed, + byte[] dst, + int dstCap, + out int dstUsed + ) + { + var codeTab = new CodeTabEntry[HASHTAB_SIZE]; + var queue = new CodeQueue(); + var stream = new BitStream(src, srcLen); + int codeSize, + dstPos, + len; + int currCode, + prevCode, + newCode; + byte firstByte; + + CodeTabInit(codeTab); + CodeQueueInit(ref queue); + codeSize = MIN_CODE_SIZE; + dstPos = 0; + + // Handle the first code separately since there is no previous code. + if (!ReadCode(stream, ref codeSize, codeTab, ref queue, out currCode)) + { + srcUsed = stream.BytesRead; + dstUsed = 0; + return UnshrnkStatus.Ok; + } + + //assert(currCode != CONTROL_CODE); + if (currCode > byte.MaxValue) + { + srcUsed = stream.BytesRead; + dstUsed = 0; + return UnshrnkStatus.Error; // The first code must be a literal. + } + + if (dstPos == dstCap) + { + srcUsed = stream.BytesRead; + dstUsed = dstPos; + return UnshrnkStatus.Full; + } + + firstByte = (byte)currCode; + dst[dstPos] = (byte)currCode; + codeTab[currCode].lastDstPos = dstPos; + dstPos++; + + prevCode = currCode; + while (ReadCode(stream, ref codeSize, codeTab, ref queue, out currCode)) + { + if (currCode == INVALID_CODE) + { + srcUsed = stream.BytesRead; + dstUsed = 0; + return UnshrnkStatus.Error; + } + + if (dstPos == dstCap) + { + srcUsed = stream.BytesRead; + dstUsed = dstPos; + return UnshrnkStatus.Full; + } + + // Handle KwKwK: next code used before being added. + if (currCode == queue.codes[queue.nextIdx]) + { + if (codeTab[prevCode].prefixCode == INVALID_CODE) + { + // The previous code is no longer valid. + srcUsed = stream.BytesRead; + dstUsed = 0; + return UnshrnkStatus.Error; + } + + // Extend the previous code with its first byte. + //assert(currCode != prevCode); + codeTab[currCode].prefixCode = prevCode; + codeTab[currCode].extByte = firstByte; + codeTab[currCode].len = (ushort)(codeTab[prevCode].len + 1); + codeTab[currCode].lastDstPos = codeTab[prevCode].lastDstPos; + //assert(dstPos < dstCap); + dst[dstPos] = firstByte; + } + + // Output the string represented by the current code. + var status = OutputCode( + currCode, + dst, + dstPos, + dstCap, + prevCode, + codeTab, + ref queue, + out firstByte, + out len + ); + if (status != UnshrnkStatus.Ok) + { + srcUsed = stream.BytesRead; + dstUsed = 0; + return status; + } + + // Verify that the output matches walking the prefixes. + var c = currCode; + for (var i = 0; i < len; i++) + { + // assert(codeTab[c].len == len - i); + //assert(codeTab[c].extByte == dst[dstPos + len - i - 1]); + c = codeTab[c].prefixCode; + } + + // Add a new code to the string table if there's room. + // The string is the previous code's string extended with the first byte of the current code's string. + newCode = CodeQueueRemoveNext(ref queue); + if (newCode != INVALID_CODE) + { + //assert(codeTab[prevCode].lastDstPos < dstPos); + codeTab[newCode].prefixCode = prevCode; + codeTab[newCode].extByte = firstByte; + codeTab[newCode].len = (ushort)(codeTab[prevCode].len + 1); + codeTab[newCode].lastDstPos = codeTab[prevCode].lastDstPos; + + if (codeTab[prevCode].prefixCode == INVALID_CODE) + { + // prevCode was invalidated in a partial clearing. Until that code is re-used, the + // string represented by newCode is indeterminate. + codeTab[newCode].len = UNKNOWN_LEN; + } + // If prevCode was invalidated in a partial clearing, it's possible that newCode == prevCode, + // in which case it will never be used or cleared. + } + + codeTab[currCode].lastDstPos = dstPos; + dstPos += len; + + prevCode = currCode; + } + + srcUsed = stream.BytesRead; + dstUsed = dstPos; + + return UnshrnkStatus.Ok; + } + + public enum UnshrnkStatus + { + Ok, + Full, + Error, + } + + private struct CodeQueue + { + public int nextIdx; + public ushort[] codes; + } + + private static void CodeQueueInit(ref CodeQueue q) + { + int codeQueueSize; + ushort code; + + codeQueueSize = 0; + q.codes = new ushort[MAX_CODE - CONTROL_CODE + 2]; + + for (code = CONTROL_CODE + 1; code <= MAX_CODE; code++) + { + q.codes[codeQueueSize++] = code; + } + + //assert(codeQueueSize < q.codes.Length); + q.codes[codeQueueSize] = INVALID_CODE; // End-of-queue marker. + q.nextIdx = 0; + } + + private static ushort CodeQueueNext(ref CodeQueue q) => + //assert(q.nextIdx < q.codes.Length); + q.codes[q.nextIdx]; + + private static ushort CodeQueueRemoveNext(ref CodeQueue q) + { + var code = CodeQueueNext(ref q); + if (code != INVALID_CODE) + { + q.nextIdx++; + } + return code; + } +} diff --git a/src/SharpCompress/Compressors/Shrink/ShrinkStream.Async.cs b/src/SharpCompress/Compressors/Shrink/ShrinkStream.Async.cs new file mode 100644 index 00000000..c5f8c49a --- /dev/null +++ b/src/SharpCompress/Compressors/Shrink/ShrinkStream.Async.cs @@ -0,0 +1,99 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Compressors.Shrink; + +internal partial class ShrinkStream : Stream +{ + internal static async ValueTask CreateAsync( + Stream stream, + long uncompressedSize, + CancellationToken cancellationToken = default + ) + { + var shrinkStream = new ShrinkStream(stream, uncompressedSize); + await shrinkStream.DecompressAsync(cancellationToken).ConfigureAwait(false); + return shrinkStream; + } + + private async ValueTask DecompressAsync(CancellationToken cancellationToken) + { + if (_decompressed) + { + return; + } + + // Read actual compressed data from the stream rather than pre-allocating based on the + // declared compressed size, which may be crafted to cause an OutOfMemoryException. + // The stream is already bounded by ReadOnlySubStream in ZipFilePart. + using var srcMs = new MemoryStream(); + await _inStream.CopyToAsync(srcMs, 81920, cancellationToken).ConfigureAwait(false); + var src = srcMs.ToArray(); + var srcLen = src.Length; + + // Decompress synchronously (CPU-bound operation) + HwUnshrink.Unshrink(src, srcLen, out _, _byteOut, (int)_uncompressedSize, out var dstUsed); + _outBytesCount = dstUsed; + _decompressed = true; + } + + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + cancellationToken.ThrowIfCancellationRequested(); + + if (!_decompressed) + { + await DecompressAsync(cancellationToken).ConfigureAwait(false); + } + + // Copy from decompressed buffer + long remaining = _outBytesCount - _position; + if (remaining <= 0) + { + return 0; + } + + int toCopy = (int)Math.Min(count, remaining); + Buffer.BlockCopy(_byteOut, (int)_position, buffer, offset, toCopy); + _position += toCopy; + return toCopy; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + + if (!_decompressed) + { + await DecompressAsync(cancellationToken).ConfigureAwait(false); + } + + if (buffer.IsEmpty) + { + return 0; + } + + long remaining = _outBytesCount - _position; + if (remaining <= 0) + { + return 0; + } + + int toCopy = (int)Math.Min(buffer.Length, remaining); + _byteOut.AsMemory((int)_position, toCopy).CopyTo(buffer); + _position += toCopy; + return toCopy; + } +#endif +} diff --git a/src/SharpCompress/Compressors/Shrink/ShrinkStream.cs b/src/SharpCompress/Compressors/Shrink/ShrinkStream.cs new file mode 100644 index 00000000..a3e680b2 --- /dev/null +++ b/src/SharpCompress/Compressors/Shrink/ShrinkStream.cs @@ -0,0 +1,93 @@ +using System; +using System.IO; +using SharpCompress.Common; + +namespace SharpCompress.Compressors.Shrink; + +internal partial class ShrinkStream : Stream +{ + private readonly Stream _inStream; + + private readonly long _uncompressedSize; + private readonly byte[] _byteOut; + private long _outBytesCount; + private bool _decompressed; + private long _position; + + public ShrinkStream(Stream stream, long uncompressedSize) + { + if (uncompressedSize > int.MaxValue) + { + throw new InvalidFormatException( + $"Shrink: declared uncompressed size {uncompressedSize} exceeds maximum supported size." + ); + } + + _inStream = stream; + + _uncompressedSize = uncompressedSize; + _byteOut = new byte[(int)_uncompressedSize]; + _outBytesCount = 0L; + } + + public override bool CanRead => true; + + public override bool CanSeek => true; + + public override bool CanWrite => false; + + public override long Length => _uncompressedSize; + + public override long Position + { + get => _position; + set => throw new NotImplementedException(); + } + + public override void Flush() => throw new NotImplementedException(); + + public override int Read(byte[] buffer, int offset, int count) + { + if (!_decompressed) + { + // Read actual compressed data from the stream rather than pre-allocating based on the + // declared compressed size, which may be crafted to cause an OutOfMemoryException. + // The stream is already bounded by ReadOnlySubStream in ZipFilePart. + using var srcMs = new MemoryStream(); + _inStream.CopyTo(srcMs); + var src = srcMs.ToArray(); + var srcLen = src.Length; + + HwUnshrink.Unshrink( + src, + srcLen, + out _, + _byteOut, + (int)_uncompressedSize, + out var dstUsed + ); + _outBytesCount = dstUsed; + _decompressed = true; + _position = 0; + } + + long remaining = _outBytesCount - _position; + if (remaining <= 0) + { + return 0; + } + + int toCopy = (int)Math.Min(count, remaining); + Buffer.BlockCopy(_byteOut, (int)_position, buffer, offset, toCopy); + _position += toCopy; + return toCopy; + } + + public override long Seek(long offset, SeekOrigin origin) => + throw new NotImplementedException(); + + public override void SetLength(long value) => throw new NotImplementedException(); + + public override void Write(byte[] buffer, int offset, int count) => + throw new NotImplementedException(); +} diff --git a/src/SharpCompress/Compressors/Squeezed/BitReader.Async.cs b/src/SharpCompress/Compressors/Squeezed/BitReader.Async.cs new file mode 100644 index 00000000..3a4597bd --- /dev/null +++ b/src/SharpCompress/Compressors/Squeezed/BitReader.Async.cs @@ -0,0 +1,33 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Compressors.Squeezed; + +public partial class BitReader +{ + public async ValueTask ReadBitAsync(CancellationToken cancellationToken = default) + { + if (_bitCount == 0) + { + byte[] buffer = new byte[1]; + int bytesRead = await _stream + .ReadAsync(buffer, 0, 1, cancellationToken) + .ConfigureAwait(false); + if (bytesRead == 0) + { + throw new IncompleteArchiveException("Unexpected end of stream."); + } + + _bitBuffer = buffer[0]; + _bitCount = 8; + } + + bool bit = (_bitBuffer & 1) != 0; + _bitBuffer >>= 1; + _bitCount--; + return bit; + } +} diff --git a/src/SharpCompress/Compressors/Squeezed/BitReader.cs b/src/SharpCompress/Compressors/Squeezed/BitReader.cs new file mode 100644 index 00000000..2d8a9563 --- /dev/null +++ b/src/SharpCompress/Compressors/Squeezed/BitReader.cs @@ -0,0 +1,54 @@ +using System; +using System.IO; +using SharpCompress.Common; + +namespace SharpCompress.Compressors.Squeezed; + +public partial class BitReader +{ + private readonly Stream _stream; + private int _bitBuffer; + private int _bitCount; + + public BitReader(Stream stream) + { + _stream = stream; + _bitBuffer = 0; + _bitCount = 0; + } + + public bool ReadBit() + { + if (_bitCount == 0) + { + int nextByte = _stream.ReadByte(); + if (nextByte == -1) + { + throw new IncompleteArchiveException("Unexpected end of stream."); + } + + _bitBuffer = nextByte; + _bitCount = 8; + } + + bool bit = (_bitBuffer & 1) != 0; + _bitBuffer >>= 1; + _bitCount--; + return bit; + } + + public int ReadBits(int count) + { + if (count < 1 || count > 32) + { + throw new ArgumentOutOfRangeException(nameof(count), "Count must be between 1 and 32."); + } + + int value = 0; + for (int i = 0; i < count; i++) + { + value = (value << 1) | (ReadBit() ? 1 : 0); + } + return value; + } +} diff --git a/src/SharpCompress/Compressors/Squeezed/SqueezedStream.Async.cs b/src/SharpCompress/Compressors/Squeezed/SqueezedStream.Async.cs new file mode 100644 index 00000000..cfdd0e52 --- /dev/null +++ b/src/SharpCompress/Compressors/Squeezed/SqueezedStream.Async.cs @@ -0,0 +1,112 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Compressors.RLE90; +using SharpCompress.IO; + +namespace SharpCompress.Compressors.Squeezed; + +public partial class SqueezeStream +{ + public static async ValueTask CreateAsync( + Stream stream, + int compressedSize, + CancellationToken cancellationToken = default + ) + { + var squeezeStream = new SqueezeStream(stream, compressedSize); + squeezeStream._decodedStream = await squeezeStream + .BuildDecodedStreamAsync(cancellationToken) + .ConfigureAwait(false); + + return squeezeStream; + } + + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + return await _decodedStream + .ReadAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + return await _decodedStream.ReadAsync(buffer, cancellationToken).ConfigureAwait(false); + } +#endif + + private async ValueTask BuildDecodedStreamAsync(CancellationToken cancellationToken) + { + byte[] numNodesBytes = new byte[2]; + int bytesRead = await _stream + .ReadAsync(numNodesBytes, 0, 2, cancellationToken) + .ConfigureAwait(false); + + if (bytesRead != 2) + { + return new PooledMemoryStream(); + } + + int numnodes = numNodesBytes[0] | (numNodesBytes[1] << 8); + + if (numnodes >= NUMVALS || numnodes == 0) + { + return new PooledMemoryStream(); + } + + var dnode = new int[numnodes, 2]; + for (int j = 0; j < numnodes; j++) + { + byte[] nodeBytes = new byte[4]; + bytesRead = await _stream + .ReadAsync(nodeBytes, 0, 4, cancellationToken) + .ConfigureAwait(false); + + if (bytesRead != 4) + { + throw new IncompleteArchiveException("Unexpected end of stream."); + } + + dnode[j, 0] = (short)(nodeBytes[0] | (nodeBytes[1] << 8)); + dnode[j, 1] = (short)(nodeBytes[2] | (nodeBytes[3] << 8)); + } + + var bitReader = new BitReader(_stream); + var huffmanDecoded = new PooledMemoryStream(); + int i = 0; + + while (true) + { + bool bit = await bitReader.ReadBitAsync(cancellationToken).ConfigureAwait(false); + i = dnode[i, bit ? 1 : 0]; + if (i < 0) + { + i = -(i + 1); + if (i == SPEOF) + { + break; + } + huffmanDecoded.WriteByte((byte)i); + i = 0; + } + else if (i >= numnodes) + { + throw new InvalidFormatException("SqueezeStream: invalid Huffman tree node index"); + } + } + + huffmanDecoded.Position = 0; + return new RunLength90Stream(huffmanDecoded, (int)huffmanDecoded.Length); + } +} diff --git a/src/SharpCompress/Compressors/Squeezed/SqueezedStream.cs b/src/SharpCompress/Compressors/Squeezed/SqueezedStream.cs new file mode 100644 index 00000000..f6a40f1c --- /dev/null +++ b/src/SharpCompress/Compressors/Squeezed/SqueezedStream.cs @@ -0,0 +1,107 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Text; +using SharpCompress.Common; +using SharpCompress.Compressors.RLE90; +using SharpCompress.IO; + +namespace SharpCompress.Compressors.Squeezed; + +[CLSCompliant(true)] +public partial class SqueezeStream : Stream +{ + private readonly Stream _stream; + private const int NUMVALS = 257; + private const int SPEOF = 256; + + private Stream _decodedStream = null!; + + private SqueezeStream(Stream stream, int compressedSize) + { + _stream = stream ?? throw new ArgumentNullException(nameof(stream)); + } + + public static SqueezeStream Create(Stream stream, int compressedSize) + { + var squeezeStream = new SqueezeStream(stream, compressedSize); + squeezeStream._decodedStream = squeezeStream.BuildDecodedStream(); + + return squeezeStream; + } + + protected override void Dispose(bool disposing) + { + _decodedStream?.Dispose(); + base.Dispose(disposing); + } + + public override bool CanRead => true; + public override bool CanSeek => false; + public override bool CanWrite => false; + + public override long Length => throw new NotSupportedException(); + public override long Position + { + get => throw new NotSupportedException(); + set => throw new NotSupportedException(); + } + + public override void Flush() => throw new NotSupportedException(); + + public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); + + public override void SetLength(long value) => throw new NotSupportedException(); + + public override void Write(byte[] buffer, int offset, int count) => + throw new NotSupportedException(); + + public override int Read(byte[] buffer, int offset, int count) + { + return _decodedStream.Read(buffer, offset, count); + } + + private Stream BuildDecodedStream() + { + using var binaryReader = new BinaryReader(_stream, Encoding.Default, leaveOpen: true); + int numnodes = binaryReader.ReadUInt16(); + + if (numnodes >= NUMVALS || numnodes == 0) + { + return new PooledMemoryStream(); + } + + var dnode = new int[numnodes, 2]; + for (int j = 0; j < numnodes; j++) + { + dnode[j, 0] = binaryReader.ReadInt16(); + dnode[j, 1] = binaryReader.ReadInt16(); + } + + var bitReader = new BitReader(_stream); + var huffmanDecoded = new PooledMemoryStream(); + int i = 0; + + while (true) + { + i = dnode[i, bitReader.ReadBit() ? 1 : 0]; + if (i < 0) + { + i = -(i + 1); + if (i == SPEOF) + { + break; + } + huffmanDecoded.WriteByte((byte)i); + i = 0; + } + else if (i >= numnodes) + { + throw new InvalidFormatException("SqueezeStream: invalid Huffman tree node index"); + } + } + + huffmanDecoded.Position = 0; + return new RunLength90Stream(huffmanDecoded, (int)huffmanDecoded.Length); + } +} diff --git a/src/SharpCompress/Compressors/Xz/BinaryUtils.Async.cs b/src/SharpCompress/Compressors/Xz/BinaryUtils.Async.cs new file mode 100644 index 00000000..5b6796dc --- /dev/null +++ b/src/SharpCompress/Compressors/Xz/BinaryUtils.Async.cs @@ -0,0 +1,33 @@ +using System; +using System.Buffers.Binary; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Compressors.Xz; + +public static partial class BinaryUtils +{ + public static async ValueTask ReadLittleEndianInt32Async( + this Stream stream, + CancellationToken cancellationToken = default + ) + { + var bytes = new byte[4]; + var read = await stream.ReadFullyAsync(bytes, cancellationToken).ConfigureAwait(false); + if (!read) + { + throw new IncompleteArchiveException("Unexpected end of stream."); + } + return BinaryPrimitives.ReadInt32LittleEndian(bytes); + } + + internal static async ValueTask ReadLittleEndianUInt32Async( + this Stream stream, + CancellationToken cancellationToken = default + ) => + unchecked( + (uint)await ReadLittleEndianInt32Async(stream, cancellationToken).ConfigureAwait(false) + ); +} diff --git a/src/SharpCompress/Compressors/Xz/BinaryUtils.cs b/src/SharpCompress/Compressors/Xz/BinaryUtils.cs index 8e08ff98..bf20c915 100644 --- a/src/SharpCompress/Compressors/Xz/BinaryUtils.cs +++ b/src/SharpCompress/Compressors/Xz/BinaryUtils.cs @@ -1,11 +1,14 @@ using System; using System.Buffers.Binary; using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; namespace SharpCompress.Compressors.Xz; [CLSCompliant(false)] -public static class BinaryUtils +public static partial class BinaryUtils { public static int ReadLittleEndianInt32(this BinaryReader reader) { @@ -22,7 +25,7 @@ public static class BinaryUtils var read = stream.ReadFully(bytes); if (!read) { - throw new EndOfStreamException(); + throw new IncompleteArchiveException("Unexpected end of stream."); } return BinaryPrimitives.ReadInt32LittleEndian(bytes); } diff --git a/src/SharpCompress/Compressors/Xz/CheckType.cs b/src/SharpCompress/Compressors/Xz/CheckType.cs index a94f8261..4c4fafcd 100644 --- a/src/SharpCompress/Compressors/Xz/CheckType.cs +++ b/src/SharpCompress/Compressors/Xz/CheckType.cs @@ -5,5 +5,5 @@ public enum CheckType : byte NONE = 0x00, CRC32 = 0x01, CRC64 = 0x04, - SHA256 = 0x0A + SHA256 = 0x0A, } diff --git a/src/SharpCompress/Compressors/Xz/Crc32.cs b/src/SharpCompress/Compressors/Xz/Crc32.cs index 611238c5..acea8ecd 100644 --- a/src/SharpCompress/Compressors/Xz/Crc32.cs +++ b/src/SharpCompress/Compressors/Xz/Crc32.cs @@ -1,5 +1,3 @@ -#nullable disable - using System; namespace SharpCompress.Compressors.Xz; @@ -10,7 +8,7 @@ public static class Crc32 public const uint DefaultPolynomial = 0xedb88320u; public const uint DefaultSeed = 0xffffffffu; - private static uint[] defaultTable; + private static uint[]? defaultTable; public static uint Compute(byte[] buffer) => Compute(DefaultSeed, buffer); @@ -54,6 +52,9 @@ public static class Crc32 return createTable; } + public static uint Update(uint seed, ReadOnlySpan buffer) => + CalculateHash(InitializeTable(DefaultPolynomial), seed, buffer); + private static uint CalculateHash(uint[] table, uint seed, ReadOnlySpan buffer) { var crc = seed; diff --git a/src/SharpCompress/Compressors/Xz/Crc64.cs b/src/SharpCompress/Compressors/Xz/Crc64.cs index 2c7df9d5..3cb7fbe1 100644 --- a/src/SharpCompress/Compressors/Xz/Crc64.cs +++ b/src/SharpCompress/Compressors/Xz/Crc64.cs @@ -1,5 +1,3 @@ -#nullable disable - using System; namespace SharpCompress.Compressors.Xz; @@ -8,10 +6,13 @@ namespace SharpCompress.Compressors.Xz; public static class Crc64 { public const ulong DefaultSeed = 0x0; + internal const ulong XZ_SEED = 0xffffffffffffffff; - internal static ulong[] Table; + internal static ulong[]? Table; + private static ulong[]? _xzTable; public const ulong Iso3309Polynomial = 0xD800000000000000; + private const ulong XZ_POLYNOMIAL = 0xC96C5795D7870F42; public static ulong Compute(byte[] buffer) => Compute(DefaultSeed, buffer); @@ -22,6 +23,15 @@ public static class Crc64 return CalculateHash(seed, Table, buffer); } + public static ulong ComputeXz(byte[] buffer) => ~UpdateXz(XZ_SEED, buffer); + + public static ulong UpdateXz(ulong seed, ReadOnlySpan buffer) + { + _xzTable ??= CreateTable(XZ_POLYNOMIAL); + + return CalculateHash(seed, _xzTable, buffer); + } + public static ulong CalculateHash(ulong seed, ulong[] table, ReadOnlySpan buffer) { var crc = seed; diff --git a/src/SharpCompress/Compressors/Xz/Filters/ArmFilter.Async.cs b/src/SharpCompress/Compressors/Xz/Filters/ArmFilter.Async.cs new file mode 100644 index 00000000..321535aa --- /dev/null +++ b/src/SharpCompress/Compressors/Xz/Filters/ArmFilter.Async.cs @@ -0,0 +1,40 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Compressors.Filters; + +namespace SharpCompress.Compressors.Xz.Filters; + +public partial class ArmFilter +{ + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + var bytesRead = await BaseStream + .ReadAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + BranchExecFilter.ARMConverter(buffer, _ip); + _ip += (uint)bytesRead; + return bytesRead; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + var bytesRead = await BaseStream.ReadAsync(buffer, cancellationToken).ConfigureAwait(false); + var arrayBuffer = buffer.Slice(0, bytesRead).ToArray(); + BranchExecFilter.ARMConverter(arrayBuffer, _ip); + arrayBuffer.AsSpan(0, bytesRead).CopyTo(buffer.Span); + _ip += (uint)bytesRead; + return bytesRead; + } +#endif +} diff --git a/src/SharpCompress/Compressors/Xz/Filters/ArmFilter.cs b/src/SharpCompress/Compressors/Xz/Filters/ArmFilter.cs index af1e99d3..30590408 100644 --- a/src/SharpCompress/Compressors/Xz/Filters/ArmFilter.cs +++ b/src/SharpCompress/Compressors/Xz/Filters/ArmFilter.cs @@ -5,11 +5,12 @@ */ using System.IO; +using SharpCompress.Common; using SharpCompress.Compressors.Filters; namespace SharpCompress.Compressors.Xz.Filters; -public class ArmFilter : BlockFilter +public partial class ArmFilter : BlockFilter { public override bool AllowAsLast => false; @@ -25,19 +26,19 @@ public class ArmFilter : BlockFilter { if (properties.Length != 0 && properties.Length != 4) { - throw new InvalidDataException("ARM properties unexpected length"); + throw new InvalidFormatException("ARM properties unexpected length"); } if (properties.Length == 4) { // Even XZ doesn't support it. - throw new InvalidDataException("ARM properties offset is not supported"); + throw new InvalidFormatException("ARM properties offset is not supported"); //_offset = BitConverter.ToUInt32(properties, 0); // //if (_offset % (UInt32)BranchExec.Alignment.ARCH_ARM_ALIGNMENT != 0) //{ - // throw new InvalidDataException("Filter offset does not match alignment"); + // throw new InvalidFormatException("Filter offset does not match alignment"); //} } } diff --git a/src/SharpCompress/Compressors/Xz/Filters/ArmThumbFilter.Async.cs b/src/SharpCompress/Compressors/Xz/Filters/ArmThumbFilter.Async.cs new file mode 100644 index 00000000..3632716b --- /dev/null +++ b/src/SharpCompress/Compressors/Xz/Filters/ArmThumbFilter.Async.cs @@ -0,0 +1,40 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Compressors.Filters; + +namespace SharpCompress.Compressors.Xz.Filters; + +public partial class ArmThumbFilter +{ + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + var bytesRead = await BaseStream + .ReadAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + BranchExecFilter.ARMTConverter(buffer, _ip); + _ip += (uint)bytesRead; + return bytesRead; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + var bytesRead = await BaseStream.ReadAsync(buffer, cancellationToken).ConfigureAwait(false); + var arrayBuffer = buffer.Slice(0, bytesRead).ToArray(); + BranchExecFilter.ARMTConverter(arrayBuffer, _ip); + arrayBuffer.AsSpan(0, bytesRead).CopyTo(buffer.Span); + _ip += (uint)bytesRead; + return bytesRead; + } +#endif +} diff --git a/src/SharpCompress/Compressors/Xz/Filters/ArmThumbFilter.cs b/src/SharpCompress/Compressors/Xz/Filters/ArmThumbFilter.cs index f3ec7b1b..97d20867 100644 --- a/src/SharpCompress/Compressors/Xz/Filters/ArmThumbFilter.cs +++ b/src/SharpCompress/Compressors/Xz/Filters/ArmThumbFilter.cs @@ -5,11 +5,12 @@ */ using System.IO; +using SharpCompress.Common; using SharpCompress.Compressors.Filters; namespace SharpCompress.Compressors.Xz.Filters; -public class ArmThumbFilter : BlockFilter +public partial class ArmThumbFilter : BlockFilter { public override bool AllowAsLast => false; @@ -25,19 +26,19 @@ public class ArmThumbFilter : BlockFilter { if (properties.Length != 0 && properties.Length != 4) { - throw new InvalidDataException("ARM Thumb properties unexpected length"); + throw new InvalidFormatException("ARM Thumb properties unexpected length"); } if (properties.Length == 4) { // Even XZ doesn't support it. - throw new InvalidDataException("ARM Thumb properties offset is not supported"); + throw new InvalidFormatException("ARM Thumb properties offset is not supported"); //_offset = BitConverter.ToUInt32(properties, 0); // //if (_offset % (UInt32)BranchExec.Alignment.ARCH_ARMTHUMB_ALIGNMENT != 0) //{ - // throw new InvalidDataException("Filter offset does not match alignment"); + // throw new InvalidFormatException("Filter offset does not match alignment"); //} } } diff --git a/src/SharpCompress/Compressors/Xz/Filters/BlockFilter.cs b/src/SharpCompress/Compressors/Xz/Filters/BlockFilter.cs index 936a3a0d..c819af8c 100644 --- a/src/SharpCompress/Compressors/Xz/Filters/BlockFilter.cs +++ b/src/SharpCompress/Compressors/Xz/Filters/BlockFilter.cs @@ -1,6 +1,7 @@ using System; using System.Collections.Generic; using System.IO; +using SharpCompress.Common; namespace SharpCompress.Compressors.Xz.Filters; @@ -15,20 +16,19 @@ public abstract class BlockFilter : ReadOnlyStream ArchArmFilter = 0x07, ArchArmThumbFilter = 0x08, ArchSparcFilter = 0x09, - Lzma2 = 0x21 + Lzma2 = 0x21, } - private static readonly Dictionary> FILTER_MAP = - new() - { - { FilterTypes.ArchX86Filter, () => new X86Filter() }, - { FilterTypes.ArchPowerPcFilter, () => new PowerPCFilter() }, - { FilterTypes.ArchIa64Filter, () => new IA64Filter() }, - { FilterTypes.ArchArmFilter, () => new ArmFilter() }, - { FilterTypes.ArchArmThumbFilter, () => new ArmThumbFilter() }, - { FilterTypes.ArchSparcFilter, () => new SparcFilter() }, - { FilterTypes.Lzma2, () => new Lzma2Filter() } - }; + private static readonly Dictionary> FILTER_MAP = new() + { + { FilterTypes.ArchX86Filter, () => new X86Filter() }, + { FilterTypes.ArchPowerPcFilter, () => new PowerPCFilter() }, + { FilterTypes.ArchIa64Filter, () => new IA64Filter() }, + { FilterTypes.ArchArmFilter, () => new ArmFilter() }, + { FilterTypes.ArchArmThumbFilter, () => new ArmThumbFilter() }, + { FilterTypes.ArchSparcFilter, () => new SparcFilter() }, + { FilterTypes.Lzma2, () => new Lzma2Filter() }, + }; public abstract bool AllowAsLast { get; } public abstract bool AllowAsNonLast { get; } @@ -40,17 +40,17 @@ public abstract class BlockFilter : ReadOnlyStream public static BlockFilter Read(BinaryReader reader) { var filterType = (FilterTypes)reader.ReadXZInteger(); - if (!FILTER_MAP.ContainsKey(filterType)) + if (!FILTER_MAP.TryGetValue(filterType, out var createFilter)) { throw new NotImplementedException($"Filter {filterType} has not yet been implemented"); } - var filter = FILTER_MAP[filterType](); + var filter = createFilter(); var sizeOfProperties = reader.ReadXZInteger(); if (sizeOfProperties > int.MaxValue) { - throw new InvalidDataException("Block filter information too large"); + throw new InvalidFormatException("Block filter information too large"); } var properties = reader.ReadBytes((int)sizeOfProperties); diff --git a/src/SharpCompress/Compressors/Xz/Filters/IA64Filter.Async.cs b/src/SharpCompress/Compressors/Xz/Filters/IA64Filter.Async.cs new file mode 100644 index 00000000..cb1b4535 --- /dev/null +++ b/src/SharpCompress/Compressors/Xz/Filters/IA64Filter.Async.cs @@ -0,0 +1,40 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Compressors.Filters; + +namespace SharpCompress.Compressors.Xz.Filters; + +public partial class IA64Filter +{ + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + var bytesRead = await BaseStream + .ReadAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + BranchExecFilter.IA64Converter(buffer, _ip); + _ip += (uint)bytesRead; + return bytesRead; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + var bytesRead = await BaseStream.ReadAsync(buffer, cancellationToken).ConfigureAwait(false); + var arrayBuffer = buffer.Slice(0, bytesRead).ToArray(); + BranchExecFilter.IA64Converter(arrayBuffer, _ip); + arrayBuffer.AsSpan(0, bytesRead).CopyTo(buffer.Span); + _ip += (uint)bytesRead; + return bytesRead; + } +#endif +} diff --git a/src/SharpCompress/Compressors/Xz/Filters/IA64Filter.cs b/src/SharpCompress/Compressors/Xz/Filters/IA64Filter.cs index dc04c71b..b514f64e 100644 --- a/src/SharpCompress/Compressors/Xz/Filters/IA64Filter.cs +++ b/src/SharpCompress/Compressors/Xz/Filters/IA64Filter.cs @@ -5,11 +5,12 @@ */ using System.IO; +using SharpCompress.Common; using SharpCompress.Compressors.Filters; namespace SharpCompress.Compressors.Xz.Filters; -public class IA64Filter : BlockFilter +public partial class IA64Filter : BlockFilter { public override bool AllowAsLast => false; @@ -25,19 +26,19 @@ public class IA64Filter : BlockFilter { if (properties.Length != 0 && properties.Length != 4) { - throw new InvalidDataException("IA64 properties unexpected length"); + throw new InvalidFormatException("IA64 properties unexpected length"); } if (properties.Length == 4) { // Even XZ doesn't support it. - throw new InvalidDataException("IA64 properties offset is not supported"); + throw new InvalidFormatException("IA64 properties offset is not supported"); //_offset = BitConverter.ToUInt32(properties, 0); // //if (_offset % (UInt32)BranchExec.Alignment.ARCH_IA64_ALIGNMENT != 0) //{ - // throw new InvalidDataException("Filter offset does not match alignment"); + // throw new InvalidFormatException("Filter offset does not match alignment"); //} } } diff --git a/src/SharpCompress/Compressors/Xz/Filters/Lzma2Filter.Async.cs b/src/SharpCompress/Compressors/Xz/Filters/Lzma2Filter.Async.cs new file mode 100644 index 00000000..f8d04250 --- /dev/null +++ b/src/SharpCompress/Compressors/Xz/Filters/Lzma2Filter.Async.cs @@ -0,0 +1,23 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Compressors.Xz.Filters; + +public partial class Lzma2Filter +{ + public override Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) => BaseStream.ReadAsync(buffer, offset, count, cancellationToken); + +#if !LEGACY_DOTNET + public override ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) => BaseStream.ReadAsync(buffer, cancellationToken); +#endif +} diff --git a/src/SharpCompress/Compressors/Xz/Filters/Lzma2Filter.cs b/src/SharpCompress/Compressors/Xz/Filters/Lzma2Filter.cs index bed59b76..fef8017b 100644 --- a/src/SharpCompress/Compressors/Xz/Filters/Lzma2Filter.cs +++ b/src/SharpCompress/Compressors/Xz/Filters/Lzma2Filter.cs @@ -1,11 +1,12 @@ using System; using System.IO; +using SharpCompress.Common; using SharpCompress.Compressors.LZMA; namespace SharpCompress.Compressors.Xz.Filters; [CLSCompliant(false)] -public class Lzma2Filter : BlockFilter +public partial class Lzma2Filter : BlockFilter { public override bool AllowAsLast => true; public override bool AllowAsNonLast => false; @@ -18,7 +19,7 @@ public class Lzma2Filter : BlockFilter { if (_dictionarySize > 40) { - throw new OverflowException("Dictionary size greater than UInt32.Max"); + throw new InvalidFormatException("Dictionary size greater than UInt32.Max"); } if (_dictionarySize == 40) @@ -35,21 +36,21 @@ public class Lzma2Filter : BlockFilter { if (properties.Length != 1) { - throw new InvalidDataException("LZMA properties unexpected length"); + throw new InvalidFormatException("LZMA properties unexpected length"); } _dictionarySize = (byte)(properties[0] & 0x3F); var reserved = properties[0] & 0xC0; if (reserved != 0) { - throw new InvalidDataException("Reserved bits used in LZMA properties"); + throw new InvalidFormatException("Reserved bits used in LZMA properties"); } } public override void ValidateFilter() { } public override void SetBaseStream(Stream stream) => - BaseStream = new LzmaStream(new[] { _dictionarySize }, stream); + BaseStream = LzmaStream.Create(new[] { _dictionarySize }, stream); public override int Read(byte[] buffer, int offset, int count) => BaseStream.Read(buffer, offset, count); diff --git a/src/SharpCompress/Compressors/Xz/Filters/PowerPCFilter.Async.cs b/src/SharpCompress/Compressors/Xz/Filters/PowerPCFilter.Async.cs new file mode 100644 index 00000000..95c04b0b --- /dev/null +++ b/src/SharpCompress/Compressors/Xz/Filters/PowerPCFilter.Async.cs @@ -0,0 +1,40 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Compressors.Filters; + +namespace SharpCompress.Compressors.Xz.Filters; + +public partial class PowerPCFilter +{ + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + var bytesRead = await BaseStream + .ReadAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + BranchExecFilter.PowerPCConverter(buffer, _ip); + _ip += (uint)bytesRead; + return bytesRead; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + var bytesRead = await BaseStream.ReadAsync(buffer, cancellationToken).ConfigureAwait(false); + var arrayBuffer = buffer.Slice(0, bytesRead).ToArray(); + BranchExecFilter.PowerPCConverter(arrayBuffer, _ip); + arrayBuffer.AsSpan(0, bytesRead).CopyTo(buffer.Span); + _ip += (uint)bytesRead; + return bytesRead; + } +#endif +} diff --git a/src/SharpCompress/Compressors/Xz/Filters/PowerPCFilter.cs b/src/SharpCompress/Compressors/Xz/Filters/PowerPCFilter.cs index 7a03a3fe..d0c6e5f5 100644 --- a/src/SharpCompress/Compressors/Xz/Filters/PowerPCFilter.cs +++ b/src/SharpCompress/Compressors/Xz/Filters/PowerPCFilter.cs @@ -5,11 +5,12 @@ */ using System.IO; +using SharpCompress.Common; using SharpCompress.Compressors.Filters; namespace SharpCompress.Compressors.Xz.Filters; -public class PowerPCFilter : BlockFilter +public partial class PowerPCFilter : BlockFilter { public override bool AllowAsLast => false; @@ -25,19 +26,19 @@ public class PowerPCFilter : BlockFilter { if (properties.Length != 0 && properties.Length != 4) { - throw new InvalidDataException("PPC properties unexpected length"); + throw new InvalidFormatException("PPC properties unexpected length"); } if (properties.Length == 4) { // Even XZ doesn't support it. - throw new InvalidDataException("PPC properties offset is not supported"); + throw new InvalidFormatException("PPC properties offset is not supported"); //_offset = BitConverter.ToUInt32(properties, 0); // //if (_offset % (UInt32)BranchExec.Alignment.ARCH_PowerPC_ALIGNMENT != 0) //{ - // throw new InvalidDataException("Filter offset does not match alignment"); + // throw new InvalidFormatException("Filter offset does not match alignment"); //} } } diff --git a/src/SharpCompress/Compressors/Xz/Filters/SparcFilter.Async.cs b/src/SharpCompress/Compressors/Xz/Filters/SparcFilter.Async.cs new file mode 100644 index 00000000..d4b06191 --- /dev/null +++ b/src/SharpCompress/Compressors/Xz/Filters/SparcFilter.Async.cs @@ -0,0 +1,40 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Compressors.Filters; + +namespace SharpCompress.Compressors.Xz.Filters; + +public partial class SparcFilter +{ + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + var bytesRead = await BaseStream + .ReadAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + BranchExecFilter.SPARCConverter(buffer, _ip); + _ip += (uint)bytesRead; + return bytesRead; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + var bytesRead = await BaseStream.ReadAsync(buffer, cancellationToken).ConfigureAwait(false); + var arrayBuffer = buffer.Slice(0, bytesRead).ToArray(); + BranchExecFilter.SPARCConverter(arrayBuffer, _ip); + arrayBuffer.AsSpan(0, bytesRead).CopyTo(buffer.Span); + _ip += (uint)bytesRead; + return bytesRead; + } +#endif +} diff --git a/src/SharpCompress/Compressors/Xz/Filters/SparcFilter.cs b/src/SharpCompress/Compressors/Xz/Filters/SparcFilter.cs index 9b74d344..01c1cf36 100644 --- a/src/SharpCompress/Compressors/Xz/Filters/SparcFilter.cs +++ b/src/SharpCompress/Compressors/Xz/Filters/SparcFilter.cs @@ -5,11 +5,12 @@ */ using System.IO; +using SharpCompress.Common; using SharpCompress.Compressors.Filters; namespace SharpCompress.Compressors.Xz.Filters; -public class SparcFilter : BlockFilter +public partial class SparcFilter : BlockFilter { public override bool AllowAsLast => false; @@ -25,19 +26,19 @@ public class SparcFilter : BlockFilter { if (properties.Length != 0 && properties.Length != 4) { - throw new InvalidDataException("SPARC properties unexpected length"); + throw new InvalidFormatException("SPARC properties unexpected length"); } if (properties.Length == 4) { // Even XZ doesn't support it. - throw new InvalidDataException("SPARC properties offset is not supported"); + throw new InvalidFormatException("SPARC properties offset is not supported"); //_offset = BitConverter.ToUInt32(properties, 0); // //if (_offset % (UInt32)BranchExec.Alignment.ARCH_SPARC_ALIGNMENT != 0) //{ - // throw new InvalidDataException("Filter offset does not match alignment"); + // throw new InvalidFormatException("Filter offset does not match alignment"); //} } } diff --git a/src/SharpCompress/Compressors/Xz/Filters/X86Filter.Async.cs b/src/SharpCompress/Compressors/Xz/Filters/X86Filter.Async.cs new file mode 100644 index 00000000..6a8ff6f4 --- /dev/null +++ b/src/SharpCompress/Compressors/Xz/Filters/X86Filter.Async.cs @@ -0,0 +1,41 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Compressors.Filters; + +namespace SharpCompress.Compressors.Xz.Filters; + +public partial class X86Filter +{ + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + var bytesRead = await BaseStream + .ReadAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + BranchExecFilter.X86Converter(buffer, _ip, ref _state); + _ip += (uint)bytesRead; + return bytesRead; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + var bytesRead = await BaseStream.ReadAsync(buffer, cancellationToken).ConfigureAwait(false); + // X86Converter requires byte[], so we need to copy to an array + var arrayBuffer = buffer.Slice(0, bytesRead).ToArray(); + BranchExecFilter.X86Converter(arrayBuffer, _ip, ref _state); + arrayBuffer.AsSpan(0, bytesRead).CopyTo(buffer.Span); + _ip += (uint)bytesRead; + return bytesRead; + } +#endif +} diff --git a/src/SharpCompress/Compressors/Xz/Filters/X86Filter.cs b/src/SharpCompress/Compressors/Xz/Filters/X86Filter.cs index 74dbfb1d..37d959bc 100644 --- a/src/SharpCompress/Compressors/Xz/Filters/X86Filter.cs +++ b/src/SharpCompress/Compressors/Xz/Filters/X86Filter.cs @@ -5,11 +5,12 @@ */ using System.IO; +using SharpCompress.Common; using SharpCompress.Compressors.Filters; namespace SharpCompress.Compressors.Xz.Filters; -public class X86Filter : BlockFilter +public partial class X86Filter : BlockFilter { public override bool AllowAsLast => false; @@ -27,19 +28,19 @@ public class X86Filter : BlockFilter { if (properties.Length != 0 && properties.Length != 4) { - throw new InvalidDataException("X86 properties unexpected length"); + throw new InvalidFormatException("X86 properties unexpected length"); } if (properties.Length == 4) { // Even XZ doesn't support it. - throw new InvalidDataException("X86 properties offset is not supported"); + throw new InvalidFormatException("X86 properties offset is not supported"); //_offset = BitConverter.ToUInt32(properties, 0); // //if (_offset % (UInt32)BranchExec.Alignment.ARCH_x86_ALIGNMENT != 0) //{ - // throw new InvalidDataException("Filter offset does not match alignment"); + // throw new InvalidFormatException("Filter offset does not match alignment"); //} } } diff --git a/src/SharpCompress/Compressors/Xz/MultiByteIntegers.Async.cs b/src/SharpCompress/Compressors/Xz/MultiByteIntegers.Async.cs new file mode 100644 index 00000000..e348b68f --- /dev/null +++ b/src/SharpCompress/Compressors/Xz/MultiByteIntegers.Async.cs @@ -0,0 +1,45 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Compressors.Xz; + +internal static partial class MultiByteIntegers +{ + public static async ValueTask ReadXZIntegerAsync( + this BinaryReader reader, + int maxBytes = 9, + CancellationToken cancellationToken = default + ) + { + ThrowHelper.ThrowIfNegativeOrZero(maxBytes); + + if (maxBytes > 9) + { + maxBytes = 9; + } + + var LastByte = await reader.ReadByteAsync(cancellationToken).ConfigureAwait(false); + var Output = (ulong)LastByte & 0x7F; + + var i = 0; + while ((LastByte & 0x80) != 0) + { + if (++i >= maxBytes) + { + throw new InvalidFormatException(); + } + + LastByte = await reader.ReadByteAsync(cancellationToken).ConfigureAwait(false); + if (LastByte == 0) + { + throw new InvalidFormatException(); + } + + Output |= ((ulong)(LastByte & 0x7F)) << (i * 7); + } + return Output; + } +} diff --git a/src/SharpCompress/Compressors/Xz/MultiByteIntegers.cs b/src/SharpCompress/Compressors/Xz/MultiByteIntegers.cs index a38505da..bf9f29d2 100644 --- a/src/SharpCompress/Compressors/Xz/MultiByteIntegers.cs +++ b/src/SharpCompress/Compressors/Xz/MultiByteIntegers.cs @@ -1,16 +1,16 @@ using System; using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; namespace SharpCompress.Compressors.Xz; -internal static class MultiByteIntegers +internal static partial class MultiByteIntegers { public static ulong ReadXZInteger(this BinaryReader reader, int MaxBytes = 9) { - if (MaxBytes <= 0) - { - throw new ArgumentOutOfRangeException(nameof(MaxBytes)); - } + ThrowHelper.ThrowIfNegativeOrZero(MaxBytes); if (MaxBytes > 9) { @@ -25,13 +25,13 @@ internal static class MultiByteIntegers { if (++i >= MaxBytes) { - throw new InvalidDataException(); + throw new InvalidFormatException(); } LastByte = reader.ReadByte(); if (LastByte == 0) { - throw new InvalidDataException(); + throw new InvalidFormatException(); } Output |= ((ulong)(LastByte & 0x7F)) << (i * 7); diff --git a/src/SharpCompress/Compressors/Xz/ReadOnlyStream.cs b/src/SharpCompress/Compressors/Xz/ReadOnlyStream.cs index 71614df9..75188483 100644 --- a/src/SharpCompress/Compressors/Xz/ReadOnlyStream.cs +++ b/src/SharpCompress/Compressors/Xz/ReadOnlyStream.cs @@ -1,4 +1,4 @@ -#nullable disable +#nullable disable using System; using System.IO; @@ -23,7 +23,7 @@ public abstract class ReadOnlyStream : Stream set => throw new NotSupportedException(); } - public override void Flush() => throw new NotSupportedException(); + public override void Flush() { } public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); diff --git a/src/SharpCompress/Compressors/Xz/XZBlock.Async.cs b/src/SharpCompress/Compressors/Xz/XZBlock.Async.cs new file mode 100644 index 00000000..cac658a0 --- /dev/null +++ b/src/SharpCompress/Compressors/Xz/XZBlock.Async.cs @@ -0,0 +1,159 @@ +using System; +using System.Buffers; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Compressors.Xz; + +public sealed partial class XZBlock +{ + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken = default + ) + { + var bytesRead = 0; + if (!HeaderIsLoaded) + { + await LoadHeaderAsync(cancellationToken).ConfigureAwait(false); + } + + if (!_streamConnected) + { + ConnectStream(); + } + + if (!_endOfStream) + { + bytesRead = await _decomStream + .NotNull() + .ReadAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + UpdateCheck(buffer, offset, bytesRead); + } + + if (bytesRead != count) + { + _endOfStream = true; + } + + if (_endOfStream && !_paddingSkipped) + { + await SkipPaddingAsync(cancellationToken).ConfigureAwait(false); + } + + if (_endOfStream && !_crcChecked) + { + await CheckCrcAsync(cancellationToken).ConfigureAwait(false); + } + + return bytesRead; + } + + private async ValueTask SkipPaddingAsync(CancellationToken cancellationToken = default) + { + var bytes = (BaseStream.Position - _startPosition) % 4; + if (bytes > 0) + { + var size = 4 - (int)bytes; + var paddingBytes = ArrayPool.Shared.Rent(size); + try + { + await BaseStream + .ReadExactAsync(paddingBytes, 0, size, cancellationToken) + .ConfigureAwait(false); + for (var i = 0; i < size; i++) + { + if (paddingBytes[i] != 0) + { + throw new InvalidFormatException("Padding bytes were non-null"); + } + } + } + finally + { + ArrayPool.Shared.Return(paddingBytes); + } + } + _paddingSkipped = true; + } + + private async ValueTask CheckCrcAsync(CancellationToken cancellationToken = default) + { + var crc = ArrayPool.Shared.Rent(_checkSize); + try + { + await BaseStream + .ReadExactAsync(crc, 0, _checkSize, cancellationToken) + .ConfigureAwait(false); + VerifyCheck(crc.AsSpan().Slice(0, _checkSize)); + _crcChecked = true; + } + finally + { + ArrayPool.Shared.Return(crc); + } + } + + private async ValueTask LoadHeaderAsync(CancellationToken cancellationToken = default) + { + await ReadHeaderSizeAsync(cancellationToken).ConfigureAwait(false); + var headerCache = await CacheHeaderAsync(cancellationToken).ConfigureAwait(false); + + using (var cache = new MemoryStream(headerCache)) + using (var cachedReader = new BinaryReader(cache)) + { + cachedReader.BaseStream.Position = 1; // skip the header size byte + ReadBlockFlags(cachedReader); + ReadFilters(cachedReader); + } + HeaderIsLoaded = true; + } + + private async ValueTask ReadHeaderSizeAsync(CancellationToken cancellationToken = default) + { + var buffer = ArrayPool.Shared.Rent(1); + try + { + await BaseStream.ReadExactAsync(buffer, 0, 1, cancellationToken).ConfigureAwait(false); + _blockHeaderSizeByte = buffer[0]; + if (_blockHeaderSizeByte == 0) + { + throw new XZIndexMarkerReachedException(); + } + } + finally + { + ArrayPool.Shared.Return(buffer); + } + } + + private async ValueTask CacheHeaderAsync(CancellationToken cancellationToken = default) + { + var blockHeaderWithoutCrc = new byte[BlockHeaderSize - 4]; + blockHeaderWithoutCrc[0] = _blockHeaderSizeByte; + var read = await BaseStream + .ReadAsync(blockHeaderWithoutCrc, 1, BlockHeaderSize - 5, cancellationToken) + .ConfigureAwait(false); + if (read != BlockHeaderSize - 5) + { + throw new IncompleteArchiveException("Reached end of stream unexpectedly"); + } + + var crc = await BaseStream + .ReadLittleEndianUInt32Async(cancellationToken) + .ConfigureAwait(false); + var calcCrc = Crc32.Compute(blockHeaderWithoutCrc); + if (crc != calcCrc) + { + throw new InvalidFormatException("Block header corrupt"); + } + + return blockHeaderWithoutCrc; + } +} diff --git a/src/SharpCompress/Compressors/Xz/XZBlock.cs b/src/SharpCompress/Compressors/Xz/XZBlock.cs index ea907609..81db843b 100644 --- a/src/SharpCompress/Compressors/Xz/XZBlock.cs +++ b/src/SharpCompress/Compressors/Xz/XZBlock.cs @@ -1,37 +1,47 @@ -#nullable disable - using System; +using System.Buffers; +using System.Buffers.Binary; using System.Collections.Generic; using System.IO; using System.Linq; +using System.Security.Cryptography; +using SharpCompress.Common; using SharpCompress.Compressors.Xz.Filters; namespace SharpCompress.Compressors.Xz; [CLSCompliant(false)] -public sealed class XZBlock : XZReadOnlyStream +public sealed partial class XZBlock : XZReadOnlyStream { - public int BlockHeaderSize => (_blockHeaderSizeByte + 1) * 4; + private int BlockHeaderSize => (_blockHeaderSizeByte + 1) * 4; public ulong? CompressedSize { get; private set; } public ulong? UncompressedSize { get; private set; } - public Stack Filters { get; private set; } = new Stack(); - public bool HeaderIsLoaded { get; private set; } - private CheckType _checkType; + private readonly Stack _filters = new(); + private bool HeaderIsLoaded { get; set; } + private readonly CheckType _checkType; private readonly int _checkSize; + private uint _crc32 = Crc32.DefaultSeed; + private ulong _crc64 = Crc64.XZ_SEED; + private readonly SHA256? _sha256; private bool _streamConnected; private int _numFilters; private byte _blockHeaderSizeByte; - private Stream _decomStream; + private Stream? _decomStream; private bool _endOfStream; private bool _paddingSkipped; private bool _crcChecked; - private ulong _bytesRead; + private readonly long _startPosition; public XZBlock(Stream stream, CheckType checkType, int checkSize) : base(stream) { _checkType = checkType; _checkSize = checkSize; + if (checkType == CheckType.SHA256) + { + _sha256 = SHA256.Create(); + } + _startPosition = stream.Position; } public override int Read(byte[] buffer, int offset, int count) @@ -49,7 +59,8 @@ public sealed class XZBlock : XZReadOnlyStream if (!_endOfStream) { - bytesRead = _decomStream.Read(buffer, offset, count); + bytesRead = _decomStream.NotNull().Read(buffer, offset, count); + UpdateCheck(buffer, offset, bytesRead); } if (bytesRead != count) @@ -67,20 +78,19 @@ public sealed class XZBlock : XZReadOnlyStream CheckCrc(); } - _bytesRead += (ulong)bytesRead; return bytesRead; } private void SkipPadding() { - var bytes = (int)(BaseStream.Position % 4); + var bytes = (BaseStream.Position - _startPosition) % 4; if (bytes > 0) { var paddingBytes = new byte[4 - bytes]; BaseStream.Read(paddingBytes, 0, paddingBytes.Length); if (paddingBytes.Any(b => b != 0)) { - throw new InvalidDataException("Padding bytes were non-null"); + throw new InvalidFormatException("Padding bytes were non-null"); } } _paddingSkipped = true; @@ -88,19 +98,110 @@ public sealed class XZBlock : XZReadOnlyStream private void CheckCrc() { - var crc = new byte[_checkSize]; - BaseStream.Read(crc, 0, _checkSize); - // Actually do a check (and read in the bytes - // into the function throughout the stream read). - _crcChecked = true; + var crc = ArrayPool.Shared.Rent(_checkSize); + try + { + BaseStream.ReadExact(crc, 0, _checkSize); + VerifyCheck(crc.AsSpan().Slice(0, _checkSize)); + _crcChecked = true; + } + finally + { + ArrayPool.Shared.Return(crc); + } + } + + private void UpdateCheck(byte[] buffer, int offset, int count) + { + if (count == 0 || _checkType == CheckType.NONE) + { + return; + } + + var bytes = buffer.AsSpan(offset, count); + switch (_checkType) + { + case CheckType.CRC32: + _crc32 = Crc32.Update(_crc32, bytes); + break; + case CheckType.CRC64: + _crc64 = Crc64.UpdateXz(_crc64, bytes); + break; + case CheckType.SHA256: + _sha256.NotNull().TransformBlock(buffer, offset, count, null, 0); + break; + } + } + + private void VerifyCheck(ReadOnlySpan expected) + { + switch (_checkType) + { + case CheckType.NONE: + break; + case CheckType.CRC32: + GetLittleEndianBytes(~_crc32, expected); + break; + case CheckType.CRC64: + GetLittleEndianBytes(~_crc64, expected); + break; + case CheckType.SHA256: + FinalizeSha256Check(expected); + break; + default: + throw new InvalidFormatException("Unsupported XZ check type"); + } + } + + private static void GetLittleEndianBytes(uint value, ReadOnlySpan expected) + { + var bytes = ArrayPool.Shared.Rent(sizeof(uint)); + try + { + BinaryPrimitives.WriteUInt32LittleEndian(bytes, value); + if (!expected.SequenceEqual(bytes.AsSpan().Slice(0, sizeof(uint)))) + { + throw new InvalidFormatException("Block check corrupt"); + } + } + finally + { + ArrayPool.Shared.Return(bytes); + } + } + + private static void GetLittleEndianBytes(ulong value, ReadOnlySpan expected) + { + var bytes = ArrayPool.Shared.Rent(sizeof(ulong)); + try + { + BinaryPrimitives.WriteUInt64LittleEndian(bytes, value); + if (!expected.SequenceEqual(bytes.AsSpan().Slice(0, sizeof(ulong)))) + { + throw new InvalidFormatException("Block check corrupt"); + } + } + finally + { + ArrayPool.Shared.Return(bytes); + } + } + + private void FinalizeSha256Check(ReadOnlySpan expected) + { + _sha256.NotNull().TransformFinalBlock(Array.Empty(), 0, 0); + if (!expected.SequenceEqual(_sha256.NotNull().Hash)) + { + throw new InvalidFormatException("Block check corrupt"); + } } private void ConnectStream() { _decomStream = BaseStream; - while (Filters.Any()) + while (_filters.Any()) { - var filter = Filters.Pop(); + var filter = _filters.Pop(); filter.SetBaseStream(_decomStream); _decomStream = filter; } @@ -138,14 +239,14 @@ public sealed class XZBlock : XZReadOnlyStream var read = BaseStream.Read(blockHeaderWithoutCrc, 1, BlockHeaderSize - 5); if (read != BlockHeaderSize - 5) { - throw new EndOfStreamException("Reached end of stream unexectedly"); + throw new IncompleteArchiveException("Reached end of stream unexpectedly"); } var crc = BaseStream.ReadLittleEndianUInt32(); var calcCrc = Crc32.Compute(blockHeaderWithoutCrc); if (crc != calcCrc) { - throw new InvalidDataException("Block header corrupt"); + throw new InvalidFormatException("Block header corrupt"); } return blockHeaderWithoutCrc; @@ -159,7 +260,7 @@ public sealed class XZBlock : XZReadOnlyStream if (reserved != 0) { - throw new InvalidDataException( + throw new InvalidFormatException( "Reserved bytes used, perhaps an unknown XZ implementation" ); } @@ -189,7 +290,7 @@ public sealed class XZBlock : XZReadOnlyStream || (i + 1 < _numFilters && !filter.AllowAsNonLast) ) { - throw new InvalidDataException("Block Filters in bad order"); + throw new InvalidFormatException("Block Filters in bad order"); } if (filter.ChangesDataSize && i + 1 < _numFilters) @@ -198,11 +299,11 @@ public sealed class XZBlock : XZReadOnlyStream } filter.ValidateFilter(); - Filters.Push(filter); + _filters.Push(filter); } if (nonLastSizeChangers > 2) { - throw new InvalidDataException( + throw new InvalidFormatException( "More than two non-last block filters cannot change stream size" ); } @@ -212,7 +313,7 @@ public sealed class XZBlock : XZReadOnlyStream var blockHeaderPadding = reader.ReadBytes(blockHeaderPaddingSize); if (!blockHeaderPadding.All(b => b == 0)) { - throw new InvalidDataException("Block header contains unknown fields"); + throw new InvalidFormatException("Block header contains unknown fields"); } } } diff --git a/src/SharpCompress/Compressors/Xz/XZFooter.Async.cs b/src/SharpCompress/Compressors/Xz/XZFooter.Async.cs new file mode 100644 index 00000000..b136335d --- /dev/null +++ b/src/SharpCompress/Compressors/Xz/XZFooter.Async.cs @@ -0,0 +1,47 @@ +using System; +using System.IO; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.IO; + +namespace SharpCompress.Compressors.Xz; + +public partial class XZFooter +{ + public static async ValueTask FromStreamAsync( + Stream stream, + CancellationToken cancellationToken = default + ) + { + var footer = new XZFooter(new BinaryReader(stream, Encoding.UTF8, true)); + await footer.ProcessAsync(cancellationToken).ConfigureAwait(false); + return footer; + } + + public async ValueTask ProcessAsync(CancellationToken cancellationToken = default) + { + var crc = await _reader + .BaseStream.ReadLittleEndianUInt32Async(cancellationToken) + .ConfigureAwait(false); + var footerBytes = await _reader.ReadBytesAsync(6, cancellationToken).ConfigureAwait(false); + var myCrc = Crc32.Compute(footerBytes); + if (crc != myCrc) + { + throw new InvalidFormatException("Footer corrupt"); + } + + using (var stream = new MemoryStream(footerBytes)) + using (var reader = new BinaryReader(stream)) + { + BackwardSize = (reader.ReadLittleEndianUInt32() + 1) * 4; + StreamFlags = reader.ReadBytes(2); + } + var magBy = await _reader.ReadBytesAsync(2, cancellationToken).ConfigureAwait(false); + if (!magBy.AsSpan().SequenceEqual(_magicBytes)) + { + throw new InvalidFormatException("Magic footer missing"); + } + } +} diff --git a/src/SharpCompress/Compressors/Xz/XZFooter.cs b/src/SharpCompress/Compressors/Xz/XZFooter.cs index 9751fc2b..d6b80d5e 100644 --- a/src/SharpCompress/Compressors/Xz/XZFooter.cs +++ b/src/SharpCompress/Compressors/Xz/XZFooter.cs @@ -1,11 +1,14 @@ using System; using System.IO; using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; using SharpCompress.IO; namespace SharpCompress.Compressors.Xz; -public class XZFooter +public partial class XZFooter { private readonly BinaryReader _reader; private static ReadOnlySpan _magicBytes => "YZ"u8; @@ -21,9 +24,7 @@ public class XZFooter public static XZFooter FromStream(Stream stream) { - var footer = new XZFooter( - new BinaryReader(NonDisposingStream.Create(stream), Encoding.UTF8) - ); + var footer = new XZFooter(new BinaryReader(stream, Encoding.UTF8, true)); footer.Process(); return footer; } @@ -35,7 +36,7 @@ public class XZFooter var myCrc = Crc32.Compute(footerBytes); if (crc != myCrc) { - throw new InvalidDataException("Footer corrupt"); + throw new InvalidFormatException("Footer corrupt"); } using (var stream = new MemoryStream(footerBytes)) @@ -47,7 +48,7 @@ public class XZFooter var magBy = _reader.ReadBytes(2); if (!magBy.AsSpan().SequenceEqual(_magicBytes)) { - throw new InvalidDataException("Magic footer missing"); + throw new InvalidFormatException("Magic footer missing"); } } } diff --git a/src/SharpCompress/Compressors/Xz/XZHeader.Async.cs b/src/SharpCompress/Compressors/Xz/XZHeader.Async.cs new file mode 100644 index 00000000..aafe103b --- /dev/null +++ b/src/SharpCompress/Compressors/Xz/XZHeader.Async.cs @@ -0,0 +1,48 @@ +using System.IO; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.IO; + +namespace SharpCompress.Compressors.Xz; + +public partial class XZHeader +{ + public static async ValueTask FromStreamAsync( + Stream stream, + CancellationToken cancellationToken = default + ) + { + var header = new XZHeader(new BinaryReader(stream, Encoding.UTF8, true)); + await header.ProcessAsync(cancellationToken).ConfigureAwait(false); + return header; + } + + public async ValueTask ProcessAsync(CancellationToken cancellationToken = default) + { + CheckMagicBytes(await _reader.ReadBytesAsync(6, cancellationToken).ConfigureAwait(false)); + await ProcessStreamFlagsAsync(cancellationToken).ConfigureAwait(false); + } + + private async ValueTask ProcessStreamFlagsAsync(CancellationToken cancellationToken = default) + { + var streamFlags = await _reader.ReadBytesAsync(2, cancellationToken).ConfigureAwait(false); + var crc = await _reader + .BaseStream.ReadLittleEndianUInt32Async(cancellationToken) + .ConfigureAwait(false); + var calcCrc = Crc32.Compute(streamFlags); + if (crc != calcCrc) + { + throw new InvalidFormatException("Stream header corrupt"); + } + + BlockCheckType = (CheckType)(streamFlags[1] & 0x0F); + var futureUse = (byte)(streamFlags[1] & 0xF0); + if (futureUse != 0 || streamFlags[0] != 0) + { + throw new InvalidFormatException("Unknown XZ Stream Version"); + } + } +} diff --git a/src/SharpCompress/Compressors/Xz/XZHeader.cs b/src/SharpCompress/Compressors/Xz/XZHeader.cs index a5ed8c4a..0dda1583 100644 --- a/src/SharpCompress/Compressors/Xz/XZHeader.cs +++ b/src/SharpCompress/Compressors/Xz/XZHeader.cs @@ -1,25 +1,26 @@ using System.IO; using System.Linq; using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; using SharpCompress.IO; namespace SharpCompress.Compressors.Xz; -public class XZHeader +public partial class XZHeader { private readonly BinaryReader _reader; private readonly byte[] MagicHeader = { 0xFD, 0x37, 0x7A, 0x58, 0x5a, 0x00 }; public CheckType BlockCheckType { get; private set; } - public int BlockCheckSize => ((((int)BlockCheckType) + 2) / 3) * 4; + public int BlockCheckSize => 4 << ((((int)BlockCheckType + 2) / 3) - 1); public XZHeader(BinaryReader reader) => _reader = reader; public static XZHeader FromStream(Stream stream) { - var header = new XZHeader( - new BinaryReader(NonDisposingStream.Create(stream), Encoding.UTF8) - ); + var header = new XZHeader(new BinaryReader(stream, Encoding.UTF8, true)); header.Process(); return header; } @@ -37,14 +38,14 @@ public class XZHeader var calcCrc = Crc32.Compute(streamFlags); if (crc != calcCrc) { - throw new InvalidDataException("Stream header corrupt"); + throw new InvalidFormatException("Stream header corrupt"); } BlockCheckType = (CheckType)(streamFlags[1] & 0x0F); var futureUse = (byte)(streamFlags[1] & 0xF0); if (futureUse != 0 || streamFlags[0] != 0) { - throw new InvalidDataException("Unknown XZ Stream Version"); + throw new InvalidFormatException("Unknown XZ Stream Version"); } } @@ -52,7 +53,7 @@ public class XZHeader { if (!header.SequenceEqual(MagicHeader)) { - throw new InvalidDataException("Invalid XZ Stream"); + throw new InvalidFormatException("Invalid XZ Stream"); } } } diff --git a/src/SharpCompress/Compressors/Xz/XZIndex.Async.cs b/src/SharpCompress/Compressors/Xz/XZIndex.Async.cs new file mode 100644 index 00000000..4a2d9e50 --- /dev/null +++ b/src/SharpCompress/Compressors/Xz/XZIndex.Async.cs @@ -0,0 +1,82 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.IO; + +namespace SharpCompress.Compressors.Xz; + +public partial class XZIndex +{ + public static async ValueTask FromStreamAsync( + Stream stream, + bool indexMarkerAlreadyVerified, + CancellationToken cancellationToken = default + ) + { + var index = new XZIndex( + new BinaryReader(stream, Encoding.UTF8, true), + indexMarkerAlreadyVerified + ); + await index.ProcessAsync(cancellationToken).ConfigureAwait(false); + return index; + } + + public async ValueTask ProcessAsync(CancellationToken cancellationToken = default) + { + if (!_indexMarkerAlreadyVerified) + { + await VerifyIndexMarkerAsync(cancellationToken).ConfigureAwait(false); + } + + NumberOfRecords = await _reader + .ReadXZIntegerAsync(cancellationToken: cancellationToken) + .ConfigureAwait(false); + for (ulong i = 0; i < NumberOfRecords; i++) + { + Records.Add( + await XZIndexRecord + .FromBinaryReaderAsync(_reader, cancellationToken) + .ConfigureAwait(false) + ); + } + await SkipPaddingAsync(cancellationToken).ConfigureAwait(false); + await VerifyCrc32Async(cancellationToken).ConfigureAwait(false); + } + + private async ValueTask VerifyIndexMarkerAsync(CancellationToken cancellationToken = default) + { + var marker = await _reader.ReadByteAsync(cancellationToken).ConfigureAwait(false); + if (marker != 0) + { + throw new InvalidFormatException("Not an index block"); + } + } + + private async ValueTask SkipPaddingAsync(CancellationToken cancellationToken = default) + { + var bytes = (int)(_reader.BaseStream.Position - StreamStartPosition) % 4; + if (bytes > 0) + { + var paddingBytes = await _reader + .ReadBytesAsync(4 - bytes, cancellationToken) + .ConfigureAwait(false); + if (paddingBytes.Any(b => b != 0)) + { + throw new InvalidFormatException("Padding bytes were non-null"); + } + } + } + + private async ValueTask VerifyCrc32Async(CancellationToken cancellationToken = default) + { + var crc = await _reader + .BaseStream.ReadLittleEndianUInt32Async(cancellationToken) + .ConfigureAwait(false); + // TODO verify this matches + } +} diff --git a/src/SharpCompress/Compressors/Xz/XZIndex.cs b/src/SharpCompress/Compressors/Xz/XZIndex.cs index 631f27b2..3ca6b06e 100644 --- a/src/SharpCompress/Compressors/Xz/XZIndex.cs +++ b/src/SharpCompress/Compressors/Xz/XZIndex.cs @@ -3,17 +3,20 @@ using System.Collections.Generic; using System.IO; using System.Linq; using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; using SharpCompress.IO; namespace SharpCompress.Compressors.Xz; [CLSCompliant(false)] -public class XZIndex +public partial class XZIndex { private readonly BinaryReader _reader; public long StreamStartPosition { get; private set; } public ulong NumberOfRecords { get; private set; } - public List Records { get; } = new List(); + public List Records { get; } = new(); private readonly bool _indexMarkerAlreadyVerified; @@ -31,7 +34,7 @@ public class XZIndex public static XZIndex FromStream(Stream stream, bool indexMarkerAlreadyVerified) { var index = new XZIndex( - new BinaryReader(NonDisposingStream.Create(stream), Encoding.UTF8), + new BinaryReader(stream, Encoding.UTF8, true), indexMarkerAlreadyVerified ); index.Process(); @@ -59,7 +62,7 @@ public class XZIndex var marker = _reader.ReadByte(); if (marker != 0) { - throw new InvalidDataException("Not an index block"); + throw new InvalidFormatException("Not an index block"); } } @@ -71,7 +74,7 @@ public class XZIndex var paddingBytes = _reader.ReadBytes(4 - bytes); if (paddingBytes.Any(b => b != 0)) { - throw new InvalidDataException("Padding bytes were non-null"); + throw new InvalidFormatException("Padding bytes were non-null"); } } } diff --git a/src/SharpCompress/Compressors/Xz/XZIndexMarkerReachedException.cs b/src/SharpCompress/Compressors/Xz/XZIndexMarkerReachedException.cs index bb006a35..f7fe0428 100644 --- a/src/SharpCompress/Compressors/Xz/XZIndexMarkerReachedException.cs +++ b/src/SharpCompress/Compressors/Xz/XZIndexMarkerReachedException.cs @@ -1,5 +1,5 @@ -using System; +using SharpCompress.Common; namespace SharpCompress.Compressors.Xz; -public class XZIndexMarkerReachedException : Exception { } +public class XZIndexMarkerReachedException : SharpCompressException { } diff --git a/src/SharpCompress/Compressors/Xz/XZIndexRecord.Async.cs b/src/SharpCompress/Compressors/Xz/XZIndexRecord.Async.cs new file mode 100644 index 00000000..812b1853 --- /dev/null +++ b/src/SharpCompress/Compressors/Xz/XZIndexRecord.Async.cs @@ -0,0 +1,22 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Compressors.Xz; + +public partial class XZIndexRecord +{ + public static async ValueTask FromBinaryReaderAsync( + BinaryReader br, + CancellationToken cancellationToken = default + ) + { + var record = new XZIndexRecord(); + record.UnpaddedSize = await br.ReadXZIntegerAsync(cancellationToken: cancellationToken) + .ConfigureAwait(false); + record.UncompressedSize = await br.ReadXZIntegerAsync(cancellationToken: cancellationToken) + .ConfigureAwait(false); + return record; + } +} diff --git a/src/SharpCompress/Compressors/Xz/XZIndexRecord.cs b/src/SharpCompress/Compressors/Xz/XZIndexRecord.cs index e05b988b..d447c7b3 100644 --- a/src/SharpCompress/Compressors/Xz/XZIndexRecord.cs +++ b/src/SharpCompress/Compressors/Xz/XZIndexRecord.cs @@ -1,10 +1,12 @@ using System; using System.IO; +using System.Threading; +using System.Threading.Tasks; namespace SharpCompress.Compressors.Xz; [CLSCompliant(false)] -public class XZIndexRecord +public partial class XZIndexRecord { public ulong UnpaddedSize { get; private set; } public ulong UncompressedSize { get; private set; } diff --git a/src/SharpCompress/Compressors/Xz/XZReadOnlyStream.cs b/src/SharpCompress/Compressors/Xz/XZReadOnlyStream.cs index 4c1aa02b..0e3f8039 100644 --- a/src/SharpCompress/Compressors/Xz/XZReadOnlyStream.cs +++ b/src/SharpCompress/Compressors/Xz/XZReadOnlyStream.cs @@ -1,4 +1,5 @@ using System.IO; +using SharpCompress.Common; namespace SharpCompress.Compressors.Xz; @@ -9,7 +10,12 @@ public abstract class XZReadOnlyStream : ReadOnlyStream BaseStream = stream; if (!BaseStream.CanRead) { - throw new InvalidDataException("Must be able to read from stream"); + throw new InvalidFormatException("Must be able to read from stream"); } } + + protected override void Dispose(bool disposing) + { + base.Dispose(disposing); + } } diff --git a/src/SharpCompress/Compressors/Xz/XZStream.Async.cs b/src/SharpCompress/Compressors/Xz/XZStream.Async.cs new file mode 100644 index 00000000..25c6bcca --- /dev/null +++ b/src/SharpCompress/Compressors/Xz/XZStream.Async.cs @@ -0,0 +1,135 @@ +#nullable disable + +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.IO; + +namespace SharpCompress.Compressors.Xz; + +public sealed partial class XZStream +{ + public static async ValueTask IsXZStreamAsync( + Stream stream, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + try + { + return null + != await XZHeader.FromStreamAsync(stream, cancellationToken).ConfigureAwait(false); + } + catch (Exception) + { + return false; + } + } + + /// + /// Asynchronously reads bytes from the current stream into a buffer. + /// + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken = default + ) + { + var bytesRead = 0; + if (_endOfStream) + { + return bytesRead; + } + + if (!HeaderIsRead) + { + await ReadHeaderAsync(cancellationToken).ConfigureAwait(false); + } + + bytesRead = await ReadBlocksAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + if (bytesRead < count) + { + _endOfStream = true; + await ReadIndexAsync(cancellationToken).ConfigureAwait(false); + await ReadFooterAsync(cancellationToken).ConfigureAwait(false); + } + return bytesRead; + } + + /// + /// Asynchronously reads and validates the XZ header. + /// + private async ValueTask ReadHeaderAsync(CancellationToken cancellationToken = default) + { + Header = await XZHeader + .FromStreamAsync(BaseStream, cancellationToken) + .ConfigureAwait(false); + AssertBlockCheckTypeIsSupported(); + HeaderIsRead = true; + } + + /// + /// Asynchronously reads the XZ index. + /// + private async ValueTask ReadIndexAsync(CancellationToken cancellationToken = default) => + Index = await XZIndex + .FromStreamAsync(BaseStream, true, cancellationToken) + .ConfigureAwait(false); + + /// + /// Asynchronously reads the XZ footer. + /// + private async ValueTask ReadFooterAsync(CancellationToken cancellationToken = default) => + Footer = await XZFooter + .FromStreamAsync(BaseStream, cancellationToken) + .ConfigureAwait(false); + + /// + /// Asynchronously reads blocks of data from the stream. + /// + private async ValueTask ReadBlocksAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken = default + ) + { + var bytesRead = 0; + if (_currentBlock is null) + { + NextBlock(); + } + + for (; ; ) + { + try + { + if (bytesRead >= count) + { + break; + } + + var remaining = count - bytesRead; + var newOffset = offset + bytesRead; + var justRead = await _currentBlock + .ReadAsync(buffer, newOffset, remaining, cancellationToken) + .ConfigureAwait(false); + if (justRead < remaining) + { + NextBlock(); + } + + bytesRead += justRead; + } + catch (XZIndexMarkerReachedException) + { + break; + } + } + return bytesRead; + } +} diff --git a/src/SharpCompress/Compressors/Xz/XZStream.cs b/src/SharpCompress/Compressors/Xz/XZStream.cs index 26d3dcb2..ea193431 100644 --- a/src/SharpCompress/Compressors/Xz/XZStream.cs +++ b/src/SharpCompress/Compressors/Xz/XZStream.cs @@ -2,12 +2,23 @@ using System; using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; namespace SharpCompress.Compressors.Xz; [CLSCompliant(false)] -public sealed class XZStream : XZReadOnlyStream +public sealed partial class XZStream : XZReadOnlyStream { + public XZStream(Stream baseStream) + : base(baseStream) { } + + protected override void Dispose(bool disposing) + { + base.Dispose(disposing); + } + public static bool IsXZStream(Stream stream) { try @@ -25,15 +36,12 @@ public sealed class XZStream : XZReadOnlyStream switch (Header.BlockCheckType) { case CheckType.NONE: - break; case CheckType.CRC32: - break; case CheckType.CRC64: - break; case CheckType.SHA256: - throw new NotImplementedException(); + break; default: - throw new NotSupportedException("Check Type unknown to this version of decoder."); + throw new InvalidFormatException("Check Type unknown to this version of decoder."); } } @@ -45,9 +53,6 @@ public sealed class XZStream : XZReadOnlyStream private bool _endOfStream; - public XZStream(Stream stream) - : base(stream) { } - public override int Read(byte[] buffer, int offset, int count) { var bytesRead = 0; @@ -80,10 +85,11 @@ public sealed class XZStream : XZReadOnlyStream private void ReadIndex() => Index = XZIndex.FromStream(BaseStream, true); - // TODO veryfy Index + // TODO verify Index private void ReadFooter() => Footer = XZFooter.FromStream(BaseStream); // TODO verify footer + private int ReadBlocks(byte[] buffer, int offset, int count) { var bytesRead = 0; diff --git a/src/SharpCompress/Compressors/ZStandard/BitOperations.cs b/src/SharpCompress/Compressors/ZStandard/BitOperations.cs new file mode 100644 index 00000000..00da0c99 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/BitOperations.cs @@ -0,0 +1,310 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +#if !NETCOREAPP3_0_OR_GREATER + +using System.Runtime.CompilerServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +// Some routines inspired by the Stanford Bit Twiddling Hacks by Sean Eron Anderson: +// http://graphics.stanford.edu/~seander/bithacks.html + +namespace System.Numerics; + +/// +/// Utility methods for intrinsic bit-twiddling operations. +/// The methods use hardware intrinsics when available on the underlying platform, +/// otherwise they use optimized software fallbacks. +/// +public static unsafe class BitOperations +{ + // hack: should be public because of inline + public static readonly byte* TrailingZeroCountDeBruijn = GetArrayPointer( + new byte[] + { + 00, + 01, + 28, + 02, + 29, + 14, + 24, + 03, + 30, + 22, + 20, + 15, + 25, + 17, + 04, + 08, + 31, + 27, + 13, + 23, + 21, + 19, + 16, + 07, + 26, + 12, + 18, + 06, + 11, + 05, + 10, + 09, + } + ); + + // hack: should be public because of inline + public static readonly byte* Log2DeBruijn = GetArrayPointer( + new byte[] + { + 00, + 09, + 01, + 10, + 13, + 21, + 02, + 29, + 11, + 14, + 16, + 18, + 22, + 25, + 03, + 30, + 08, + 12, + 20, + 28, + 15, + 17, + 24, + 07, + 19, + 27, + 23, + 06, + 26, + 05, + 04, + 31, + } + ); + + /// + /// Returns the integer (floor) log of the specified value, base 2. + /// Note that by convention, input value 0 returns 0 since log(0) is undefined. + /// + /// The value. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int Log2(uint value) + { + // The 0->0 contract is fulfilled by setting the LSB to 1. + // Log(1) is 0, and setting the LSB for values > 1 does not change the log2 result. + value |= 1; + + // value lzcnt actual expected + // ..0001 31 31-31 0 + // ..0010 30 31-30 1 + // 0010.. 2 31-2 29 + // 0100.. 1 31-1 30 + // 1000.. 0 31-0 31 + + // Fallback contract is 0->0 + // No AggressiveInlining due to large method size + // Has conventional contract 0->0 (Log(0) is undefined) + + // Fill trailing zeros with ones, eg 00010010 becomes 00011111 + value |= value >> 01; + value |= value >> 02; + value |= value >> 04; + value |= value >> 08; + value |= value >> 16; + + // uint.MaxValue >> 27 is always in range [0 - 31] so we use Unsafe.AddByteOffset to avoid bounds check + return Log2DeBruijn[ + // Using deBruijn sequence, k=2, n=5 (2^5=32) : 0b_0000_0111_1100_0100_1010_1100_1101_1101u + (int)((value * 0x07C4ACDDu) >> 27) + ]; + } + + /// + /// Returns the integer (floor) log of the specified value, base 2. + /// Note that by convention, input value 0 returns 0 since log(0) is undefined. + /// + /// The value. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int Log2(ulong value) + { + value |= 1; + + uint hi = (uint)(value >> 32); + + if (hi == 0) + { + return Log2((uint)value); + } + + return 32 + Log2(hi); + } + + /// + /// Count the number of trailing zero bits in an integer value. + /// Similar in behavior to the x86 instruction TZCNT. + /// + /// The value. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int TrailingZeroCount(int value) => TrailingZeroCount((uint)value); + + /// + /// Count the number of trailing zero bits in an integer value. + /// Similar in behavior to the x86 instruction TZCNT. + /// + /// The value. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int TrailingZeroCount(uint value) + { + // Unguarded fallback contract is 0->0, BSF contract is 0->undefined + if (value == 0) + { + return 32; + } + + // uint.MaxValue >> 27 is always in range [0 - 31] so we use Unsafe.AddByteOffset to avoid bounds check + return TrailingZeroCountDeBruijn[ + // Using deBruijn sequence, k=2, n=5 (2^5=32) : 0b_0000_0111_0111_1100_1011_0101_0011_0001u + (int)(((value & (uint)-(int)value) * 0x077CB531u) >> 27) + ]; // Multi-cast mitigates redundant conv.u8 + } + + /// + /// Count the number of trailing zero bits in a mask. + /// Similar in behavior to the x86 instruction TZCNT. + /// + /// The value. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int TrailingZeroCount(long value) => TrailingZeroCount((ulong)value); + + /// + /// Count the number of trailing zero bits in a mask. + /// Similar in behavior to the x86 instruction TZCNT. + /// + /// The value. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int TrailingZeroCount(ulong value) + { + uint lo = (uint)value; + + if (lo == 0) + { + return 32 + TrailingZeroCount((uint)(value >> 32)); + } + + return TrailingZeroCount(lo); + } + + /// + /// Rotates the specified value left by the specified number of bits. + /// Similar in behavior to the x86 instruction ROL. + /// + /// The value to rotate. + /// The number of bits to rotate by. + /// Any value outside the range [0..31] is treated as congruent mod 32. + /// The rotated value. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static uint RotateLeft(uint value, int offset) => + (value << offset) | (value >> (32 - offset)); + + /// + /// Rotates the specified value left by the specified number of bits. + /// Similar in behavior to the x86 instruction ROL. + /// + /// The value to rotate. + /// The number of bits to rotate by. + /// Any value outside the range [0..63] is treated as congruent mod 64. + /// The rotated value. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static ulong RotateLeft(ulong value, int offset) => + (value << offset) | (value >> (64 - offset)); + + /// + /// Rotates the specified value right by the specified number of bits. + /// Similar in behavior to the x86 instruction ROR. + /// + /// The value to rotate. + /// The number of bits to rotate by. + /// Any value outside the range [0..31] is treated as congruent mod 32. + /// The rotated value. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static uint RotateRight(uint value, int offset) => + (value >> offset) | (value << (32 - offset)); + + /// + /// Rotates the specified value right by the specified number of bits. + /// Similar in behavior to the x86 instruction ROR. + /// + /// The value to rotate. + /// The number of bits to rotate by. + /// Any value outside the range [0..63] is treated as congruent mod 64. + /// The rotated value. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static ulong RotateRight(ulong value, int offset) => + (value >> offset) | (value << (64 - offset)); + + /// + /// Count the number of leading zero bits in a mask. + /// Similar in behavior to the x86 instruction LZCNT. + /// + /// The value. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int LeadingZeroCount(uint value) + { + // Unguarded fallback contract is 0->31, BSR contract is 0->undefined + if (value == 0) + { + return 32; + } + + // No AggressiveInlining due to large method size + // Has conventional contract 0->0 (Log(0) is undefined) + + // Fill trailing zeros with ones, eg 00010010 becomes 00011111 + value |= value >> 01; + value |= value >> 02; + value |= value >> 04; + value |= value >> 08; + value |= value >> 16; + + // uint.MaxValue >> 27 is always in range [0 - 31] so we use Unsafe.AddByteOffset to avoid bounds check + return 31 + ^ Log2DeBruijn[ + // uint|long -> IntPtr cast on 32-bit platforms does expensive overflow checks not needed here + (int)((value * 0x07C4ACDDu) >> 27) + ]; + } + + /// + /// Count the number of leading zero bits in a mask. + /// Similar in behavior to the x86 instruction LZCNT. + /// + /// The value. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int LeadingZeroCount(ulong value) + { + uint hi = (uint)(value >> 32); + + if (hi == 0) + { + return 32 + LeadingZeroCount((uint)value); + } + + return LeadingZeroCount(hi); + } +} + +#endif diff --git a/src/SharpCompress/Compressors/ZStandard/CompressionStream.Async.cs b/src/SharpCompress/Compressors/ZStandard/CompressionStream.Async.cs new file mode 100644 index 00000000..5f988e4c --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/CompressionStream.Async.cs @@ -0,0 +1,131 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Compressors.ZStandard.Unsafe; + +namespace SharpCompress.Compressors.ZStandard; + +public partial class CompressionStream +{ +#if !LEGACY_DOTNET || NETSTANDARD2_1 + public override async ValueTask DisposeAsync() +#else + public async ValueTask DisposeAsync() +#endif + { + if (compressor == null) + { +#if LEGACY_DOTNET + Dispose(true); + GC.SuppressFinalize(this); + await Task.CompletedTask.ConfigureAwait(false); +#else + await base.DisposeAsync().ConfigureAwait(false); +#endif + return; + } + + try + { + await FlushInternalAsync(ZSTD_EndDirective.ZSTD_e_end).ConfigureAwait(false); + } + finally + { + ReleaseUnmanagedResources(); + GC.SuppressFinalize(this); + } +#if LEGACY_DOTNET + Dispose(true); + await Task.CompletedTask.ConfigureAwait(false); +#else + await base.DisposeAsync().ConfigureAwait(false); +#endif + } + + public override async Task FlushAsync(CancellationToken cancellationToken) => + await FlushInternalAsync(ZSTD_EndDirective.ZSTD_e_flush, cancellationToken) + .ConfigureAwait(false); + + private async ValueTask FlushInternalAsync( + ZSTD_EndDirective directive, + CancellationToken cancellationToken = default + ) => await WriteInternalAsync(null, directive, cancellationToken).ConfigureAwait(false); + +#if !LEGACY_DOTNET + private async ValueTask WriteInternalAsync( + ReadOnlyMemory? buffer, + ZSTD_EndDirective directive, + CancellationToken cancellationToken = default + ) +#else + private async ValueTask WriteInternalAsync( + ReadOnlyMemory? buffer, + ZSTD_EndDirective directive, + CancellationToken cancellationToken = default + ) +#endif + { + EnsureNotDisposed(); + + var input = new ZSTD_inBuffer_s + { + pos = 0, + size = buffer.HasValue ? (nuint)buffer.Value.Length : 0, + }; + nuint remaining; + do + { + output.pos = 0; + remaining = CompressStream( + ref input, + buffer.HasValue ? buffer.Value.Span : null, + directive + ); + + var written = (int)output.pos; + if (written > 0) + { + await innerStream + .WriteAsync(outputBuffer, 0, written, cancellationToken) + .ConfigureAwait(false); + } + } while ( + directive == ZSTD_EndDirective.ZSTD_e_continue ? input.pos < input.size : remaining > 0 + ); + } + +#if !LEGACY_DOTNET || NETSTANDARD2_1 + + public override Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) => WriteAsync(new ReadOnlyMemory(buffer, offset, count), cancellationToken).AsTask(); + + public override async ValueTask WriteAsync( + ReadOnlyMemory buffer, + CancellationToken cancellationToken = default + ) => + await WriteInternalAsync(buffer, ZSTD_EndDirective.ZSTD_e_continue, cancellationToken) + .ConfigureAwait(false); +#else + + public override async Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) => + await WriteAsync(new ReadOnlyMemory(buffer, offset, count), cancellationToken) + .ConfigureAwait(false); + + public async ValueTask WriteAsync( + ReadOnlyMemory buffer, + CancellationToken cancellationToken = default + ) => + await WriteInternalAsync(buffer, ZSTD_EndDirective.ZSTD_e_continue, cancellationToken) + .ConfigureAwait(false); +#endif +} diff --git a/src/SharpCompress/Compressors/ZStandard/CompressionStream.cs b/src/SharpCompress/Compressors/ZStandard/CompressionStream.cs new file mode 100644 index 00000000..b9d2688e --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/CompressionStream.cs @@ -0,0 +1,208 @@ +using System; +using System.Buffers; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Compressors.ZStandard.Unsafe; + +namespace SharpCompress.Compressors.ZStandard; + +public partial class CompressionStream : Stream +#if LEGACY_DOTNET + , IAsyncDisposable +#endif +{ + private readonly Stream innerStream; + private readonly byte[] outputBuffer; + private readonly bool preserveCompressor; + private readonly bool leaveOpen; + private Compressor? compressor; + private ZSTD_outBuffer_s output; + + public CompressionStream( + Stream stream, + int level = Compressor.DefaultCompressionLevel, + int bufferSize = 0, + bool leaveOpen = true + ) + : this(stream, new Compressor(level), bufferSize, false, leaveOpen) { } + + public CompressionStream( + Stream stream, + Compressor compressor, + int bufferSize = 0, + bool preserveCompressor = true, + bool leaveOpen = true + ) + { + SharpCompress.ThrowHelper.ThrowIfNull(stream); + + if (!stream.CanWrite) + { + throw new ArgumentException("Stream is not writable", nameof(stream)); + } + + SharpCompress.ThrowHelper.ThrowIfNegative(bufferSize); + + innerStream = stream; + this.compressor = compressor; + this.preserveCompressor = preserveCompressor; + this.leaveOpen = leaveOpen; + + var outputBufferSize = + bufferSize > 0 + ? bufferSize + : (int)Unsafe.Methods.ZSTD_CStreamOutSize().EnsureZstdSuccess(); + outputBuffer = ArrayPool.Shared.Rent(outputBufferSize); + output = new ZSTD_outBuffer_s { pos = 0, size = (nuint)outputBufferSize }; + } + + public void SetParameter(ZSTD_cParameter parameter, int value) + { + EnsureNotDisposed(); + compressor.NotNull().SetParameter(parameter, value); + } + + public int GetParameter(ZSTD_cParameter parameter) + { + EnsureNotDisposed(); + return compressor.NotNull().GetParameter(parameter); + } + + public void LoadDictionary(byte[] dict) + { + EnsureNotDisposed(); + compressor.NotNull().LoadDictionary(dict); + } + + ~CompressionStream() => Dispose(false); + + protected override void Dispose(bool disposing) + { + if (compressor == null) + { + base.Dispose(disposing); + return; + } + + try + { + if (disposing) + { + FlushInternal(ZSTD_EndDirective.ZSTD_e_end); + } + } + finally + { + ReleaseUnmanagedResources(); + } + base.Dispose(disposing); + } + + private void ReleaseUnmanagedResources() + { + if (!preserveCompressor) + { + compressor.NotNull().Dispose(); + } + compressor = null; + + if (outputBuffer != null) + { + ArrayPool.Shared.Return(outputBuffer); + } + + if (!leaveOpen) + { + innerStream.Dispose(); + } + } + + public override void Flush() => FlushInternal(ZSTD_EndDirective.ZSTD_e_flush); + + private void FlushInternal(ZSTD_EndDirective directive) => + WriteInternal(ReadOnlySpan.Empty, directive); + + public override void Write(byte[] buffer, int offset, int count) => + Write(new ReadOnlySpan(buffer, offset, count)); + +#if !LEGACY_DOTNET || NETSTANDARD2_1 + public override void Write(ReadOnlySpan buffer) => + WriteInternal(buffer, ZSTD_EndDirective.ZSTD_e_continue); +#else + public void Write(ReadOnlySpan buffer) => + WriteInternal(buffer, ZSTD_EndDirective.ZSTD_e_continue); +#endif + + private void WriteInternal(ReadOnlySpan buffer, ZSTD_EndDirective directive) + { + EnsureNotDisposed(); + + var input = new ZSTD_inBuffer_s { pos = 0, size = (nuint)buffer.Length }; + nuint remaining; + do + { + output.pos = 0; + remaining = CompressStream(ref input, buffer, directive); + + var written = (int)output.pos; + if (written > 0) + { + innerStream.Write(outputBuffer, 0, written); + } + } while ( + directive == ZSTD_EndDirective.ZSTD_e_continue ? input.pos < input.size : remaining > 0 + ); + } + + internal unsafe nuint CompressStream( + ref ZSTD_inBuffer_s input, + ReadOnlySpan inputBuffer, + ZSTD_EndDirective directive + ) + { + fixed (byte* inputBufferPtr = inputBuffer) + fixed (byte* outputBufferPtr = outputBuffer) + { + input.src = inputBufferPtr; + output.dst = outputBufferPtr; + return compressor + .NotNull() + .CompressStream(ref input, ref output, directive) + .EnsureZstdSuccess(); + } + } + + public override bool CanRead => false; + public override bool CanSeek => false; + public override bool CanWrite => true; + + public override long Length => throw new NotSupportedException(); + + public override long Position + { + get => throw new NotSupportedException(); + set => throw new NotSupportedException(); + } + + public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); + + public override void SetLength(long value) => throw new NotSupportedException(); + + public override int Read(byte[] buffer, int offset, int count) => + throw new NotSupportedException(); + + private void EnsureNotDisposed() + { + if (compressor == null) + { + throw new ObjectDisposedException(nameof(CompressionStream)); + } + } + + public void SetPledgedSrcSize(ulong pledgedSrcSize) + { + EnsureNotDisposed(); + compressor.NotNull().SetPledgedSrcSize(pledgedSrcSize); + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Compressor.cs b/src/SharpCompress/Compressors/ZStandard/Compressor.cs new file mode 100644 index 00000000..c1a60ce0 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Compressor.cs @@ -0,0 +1,206 @@ +using System; +using SharpCompress.Compressors.ZStandard.Unsafe; + +namespace SharpCompress.Compressors.ZStandard; + +public unsafe class Compressor : IDisposable +{ + /// + /// Minimum negative compression level allowed + /// + public static int MinCompressionLevel => Unsafe.Methods.ZSTD_minCLevel(); + + /// + /// Maximum compression level available + /// + public static int MaxCompressionLevel => Unsafe.Methods.ZSTD_maxCLevel(); + + /// + /// Default compression level + /// + /// + public const int DefaultCompressionLevel = 3; + + private int level = DefaultCompressionLevel; + + private readonly SafeCctxHandle handle; + + public int Level + { + get => level; + set + { + if (level != value) + { + level = value; + SetParameter(ZSTD_cParameter.ZSTD_c_compressionLevel, value); + } + } + } + + public void SetParameter(ZSTD_cParameter parameter, int value) + { + using var cctx = handle.Acquire(); + Unsafe.Methods.ZSTD_CCtx_setParameter(cctx, parameter, value).EnsureZstdSuccess(); + } + + public int GetParameter(ZSTD_cParameter parameter) + { + using var cctx = handle.Acquire(); + int value; + Unsafe.Methods.ZSTD_CCtx_getParameter(cctx, parameter, &value).EnsureZstdSuccess(); + return value; + } + + public void LoadDictionary(byte[] dict) + { + var dictReadOnlySpan = new ReadOnlySpan(dict); + LoadDictionary(dictReadOnlySpan); + } + + public void LoadDictionary(ReadOnlySpan dict) + { + using var cctx = handle.Acquire(); + fixed (byte* dictPtr = dict) + { + Unsafe + .Methods.ZSTD_CCtx_loadDictionary(cctx, dictPtr, (nuint)dict.Length) + .EnsureZstdSuccess(); + } + } + + public Compressor(int level = DefaultCompressionLevel) + { + handle = SafeCctxHandle.Create(); + Level = level; + } + + public static int GetCompressBound(int length) => + (int)Unsafe.Methods.ZSTD_compressBound((nuint)length); + + public static ulong GetCompressBoundLong(ulong length) => + Unsafe.Methods.ZSTD_compressBound((nuint)length); + + public Span Wrap(ReadOnlySpan src) + { + var dest = new byte[GetCompressBound(src.Length)]; + var length = Wrap(src, dest); + return new Span(dest, 0, length); + } + + public int Wrap(byte[] src, byte[] dest, int offset) => + Wrap(src, new Span(dest, offset, dest.Length - offset)); + + public int Wrap(ReadOnlySpan src, Span dest) + { + fixed (byte* srcPtr = src) + fixed (byte* destPtr = dest) + { + using var cctx = handle.Acquire(); + return (int) + Unsafe + .Methods.ZSTD_compress2( + cctx, + destPtr, + (nuint)dest.Length, + srcPtr, + (nuint)src.Length + ) + .EnsureZstdSuccess(); + } + } + + public int Wrap(ArraySegment src, ArraySegment dest) => + Wrap((ReadOnlySpan)src, dest); + + public int Wrap( + byte[] src, + int srcOffset, + int srcLength, + byte[] dst, + int dstOffset, + int dstLength + ) => + Wrap( + new ReadOnlySpan(src, srcOffset, srcLength), + new Span(dst, dstOffset, dstLength) + ); + + public bool TryWrap(byte[] src, byte[] dest, int offset, out int written) => + TryWrap(src, new Span(dest, offset, dest.Length - offset), out written); + + public bool TryWrap(ReadOnlySpan src, Span dest, out int written) + { + fixed (byte* srcPtr = src) + fixed (byte* destPtr = dest) + { + nuint returnValue; + using (var cctx = handle.Acquire()) + { + returnValue = Unsafe.Methods.ZSTD_compress2( + cctx, + destPtr, + (nuint)dest.Length, + srcPtr, + (nuint)src.Length + ); + } + + if (returnValue == unchecked(0 - (nuint)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)) + { + written = default; + return false; + } + + returnValue.EnsureZstdSuccess(); + written = (int)returnValue; + return true; + } + } + + public bool TryWrap(ArraySegment src, ArraySegment dest, out int written) => + TryWrap((ReadOnlySpan)src, dest, out written); + + public bool TryWrap( + byte[] src, + int srcOffset, + int srcLength, + byte[] dst, + int dstOffset, + int dstLength, + out int written + ) => + TryWrap( + new ReadOnlySpan(src, srcOffset, srcLength), + new Span(dst, dstOffset, dstLength), + out written + ); + + public void Dispose() + { + handle.Dispose(); + GC.SuppressFinalize(this); + } + + internal nuint CompressStream( + ref ZSTD_inBuffer_s input, + ref ZSTD_outBuffer_s output, + ZSTD_EndDirective directive + ) + { + fixed (ZSTD_inBuffer_s* inputPtr = &input) + fixed (ZSTD_outBuffer_s* outputPtr = &output) + { + using var cctx = handle.Acquire(); + return Unsafe + .Methods.ZSTD_compressStream2(cctx, outputPtr, inputPtr, directive) + .EnsureZstdSuccess(); + } + } + + public void SetPledgedSrcSize(ulong pledgedSrcSize) + { + using var cctx = handle.Acquire(); + Unsafe.Methods.ZSTD_CCtx_setPledgedSrcSize(cctx, pledgedSrcSize).EnsureZstdSuccess(); + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Constants.cs b/src/SharpCompress/Compressors/ZStandard/Constants.cs new file mode 100644 index 00000000..cce84fc0 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Constants.cs @@ -0,0 +1,8 @@ +namespace SharpCompress.Compressors.ZStandard; + +internal class Constants +{ + //NOTE: https://docs.microsoft.com/en-us/dotnet/framework/configure-apps/file-schema/runtime/gcallowverylargeobjects-element#remarks + //NOTE: https://github.com/dotnet/runtime/blob/v5.0.0-rtm.20519.4/src/libraries/System.Private.CoreLib/src/System/Array.cs#L27 + public const ulong MaxByteArrayLength = 0x7FFFFFC7; +} diff --git a/src/SharpCompress/Compressors/ZStandard/DecompressionStream.Async.cs b/src/SharpCompress/Compressors/ZStandard/DecompressionStream.Async.cs new file mode 100644 index 00000000..264da100 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/DecompressionStream.Async.cs @@ -0,0 +1,92 @@ +using System; +using System.Buffers; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Compressors.ZStandard.Unsafe; + +namespace SharpCompress.Compressors.ZStandard; + +public partial class DecompressionStream +{ +#if !LEGACY_DOTNET || NETSTANDARD2_1 + public override Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) => ReadAsync(new Memory(buffer, offset, count), cancellationToken).AsTask(); + + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) +#else + + public override Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) => ReadAsync(new Memory(buffer, offset, count), cancellationToken).AsTask(); + + public async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) +#endif + { + EnsureNotDisposed(); + + // Guard against infinite loop (output.pos would never become non-zero) + if (buffer.Length == 0) + { + return 0; + } + + var output = new ZSTD_outBuffer_s { pos = 0, size = (nuint)buffer.Length }; + while (true) + { + // If there is still input available, or there might be data buffered in the decompressor context, flush that out + while (input.pos < input.size || !contextDrained) + { + nuint oldInputPos = input.pos; + nuint result = DecompressStream(ref output, buffer.Span); + if (output.pos > 0 || oldInputPos != input.pos) + { + // Keep result from last decompress call that made some progress, so we known if we're at end of frame + lastDecompressResult = result; + } + // If decompression filled the output buffer, there might still be data buffered in the decompressor context + contextDrained = output.pos < output.size; + // If we have data to return, return it immediately, so we won't stall on Read + if (output.pos > 0) + { + return (int)output.pos; + } + } + + // Otherwise, read some more input + int bytesRead; + if ( + ( + bytesRead = await innerStream + .ReadAsync(inputBuffer, 0, inputBufferSize, cancellationToken) + .ConfigureAwait(false) + ) == 0 + ) + { + if (checkEndOfStream && lastDecompressResult != 0) + { + throw new IncompleteArchiveException("Premature end of stream"); + } + + return 0; + } + + input.size = (nuint)bytesRead; + input.pos = 0; + } + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/DecompressionStream.cs b/src/SharpCompress/Compressors/ZStandard/DecompressionStream.cs new file mode 100644 index 00000000..20490d91 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/DecompressionStream.cs @@ -0,0 +1,220 @@ +using System; +using System.Buffers; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Compressors.ZStandard.Unsafe; + +namespace SharpCompress.Compressors.ZStandard; + +public partial class DecompressionStream : Stream +{ + private readonly Stream innerStream; + private readonly byte[] inputBuffer; + private readonly int inputBufferSize; + private readonly bool preserveDecompressor; + private readonly bool leaveOpen; + private readonly bool checkEndOfStream; + private Decompressor? decompressor; + private ZSTD_inBuffer_s input; + private nuint lastDecompressResult = 0; + private bool contextDrained = true; + + public DecompressionStream( + Stream stream, + int bufferSize = 0, + bool checkEndOfStream = true, + bool leaveOpen = true + ) + : this(stream, new Decompressor(), bufferSize, checkEndOfStream, false, leaveOpen) { } + + public DecompressionStream( + Stream stream, + Decompressor decompressor, + int bufferSize = 0, + bool checkEndOfStream = true, + bool preserveDecompressor = true, + bool leaveOpen = true + ) + { + SharpCompress.ThrowHelper.ThrowIfNull(stream); + + if (!stream.CanRead) + { + throw new ArgumentException("Stream is not readable", nameof(stream)); + } + + SharpCompress.ThrowHelper.ThrowIfNegative(bufferSize); + + innerStream = stream; + this.decompressor = decompressor; + this.preserveDecompressor = preserveDecompressor; + this.leaveOpen = leaveOpen; + this.checkEndOfStream = checkEndOfStream; + + inputBufferSize = + bufferSize > 0 + ? bufferSize + : (int)Unsafe.Methods.ZSTD_DStreamInSize().EnsureZstdSuccess(); + inputBuffer = ArrayPool.Shared.Rent(inputBufferSize); + input = new ZSTD_inBuffer_s { pos = (nuint)inputBufferSize, size = (nuint)inputBufferSize }; + } + + public void SetParameter(ZSTD_dParameter parameter, int value) + { + EnsureNotDisposed(); + decompressor.NotNull().SetParameter(parameter, value); + } + + public int GetParameter(ZSTD_dParameter parameter) + { + EnsureNotDisposed(); + return decompressor.NotNull().GetParameter(parameter); + } + + public void LoadDictionary(byte[] dict) + { + EnsureNotDisposed(); + decompressor.NotNull().LoadDictionary(dict); + } + + ~DecompressionStream() => Dispose(false); + + protected override void Dispose(bool disposing) + { + if (decompressor == null) + { + base.Dispose(disposing); + return; + } + + if (!preserveDecompressor) + { + decompressor.Dispose(); + } + decompressor = null; + + if (inputBuffer != null) + { + ArrayPool.Shared.Return(inputBuffer); + } + + if (!leaveOpen) + { + innerStream.Dispose(); + } + base.Dispose(disposing); + } + + public override int Read(byte[] buffer, int offset, int count) => + Read(new Span(buffer, offset, count)); + +#if !LEGACY_DOTNET || NETSTANDARD2_1 + public override int Read(Span buffer) +#else + public int Read(Span buffer) +#endif + { + EnsureNotDisposed(); + + // Guard against infinite loop (output.pos would never become non-zero) + if (buffer.Length == 0) + { + return 0; + } + + var output = new ZSTD_outBuffer_s { pos = 0, size = (nuint)buffer.Length }; + while (true) + { + // If there is still input available, or there might be data buffered in the decompressor context, flush that out + while (input.pos < input.size || !contextDrained) + { + nuint oldInputPos = input.pos; + nuint result = DecompressStream(ref output, buffer); + if (output.pos > 0 || oldInputPos != input.pos) + { + // Keep result from last decompress call that made some progress, so we known if we're at end of frame + lastDecompressResult = result; + } + // If decompression filled the output buffer, there might still be data buffered in the decompressor context + contextDrained = output.pos < output.size; + // If we have data to return, return it immediately, so we won't stall on Read + if (output.pos > 0) + { + return (int)output.pos; + } + } + + // Otherwise, read some more input + int bytesRead; + if ((bytesRead = innerStream.Read(inputBuffer, 0, inputBufferSize)) == 0) + { + if (checkEndOfStream && lastDecompressResult != 0) + { + throw new IncompleteArchiveException("Premature end of stream"); + } + + return 0; + } + + input.size = (nuint)bytesRead; + input.pos = 0; + } + } + + private unsafe nuint DecompressStream(ref ZSTD_outBuffer_s output, Span outputBuffer) + { + fixed (byte* inputBufferPtr = inputBuffer) + fixed (byte* outputBufferPtr = outputBuffer) + { + input.src = inputBufferPtr; + output.dst = outputBufferPtr; + return decompressor.NotNull().DecompressStream(ref input, ref output); + } + } + + public override bool CanRead => true; + public override bool CanSeek => false; + public override bool CanWrite => false; + + public override long Length => throw new NotSupportedException(); + + public override long Position + { + get => throw new NotSupportedException(); + set => throw new NotSupportedException(); + } + + public override void Flush() => throw new NotSupportedException(); + + public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); + + public override void SetLength(long value) => throw new NotSupportedException(); + + public override void Write(byte[] buffer, int offset, int count) => + throw new NotSupportedException(); + + private void EnsureNotDisposed() + { + if (decompressor == null) + { + throw new ObjectDisposedException(nameof(DecompressionStream)); + } + } + +#if LEGACY_DOTNET && !NETSTANDARD2_1 + public virtual ValueTask DisposeAsync() + { + try + { + Dispose(); + return default; + } + catch (Exception exc) + { + return new ValueTask(Task.FromException(exc)); + } + } +#endif +} diff --git a/src/SharpCompress/Compressors/ZStandard/Decompressor.cs b/src/SharpCompress/Compressors/ZStandard/Decompressor.cs new file mode 100644 index 00000000..06da3b8f --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Decompressor.cs @@ -0,0 +1,185 @@ +using System; +using SharpCompress.Compressors.ZStandard.Unsafe; + +namespace SharpCompress.Compressors.ZStandard; + +public unsafe class Decompressor : IDisposable +{ + private readonly SafeDctxHandle handle; + + public Decompressor() + { + handle = SafeDctxHandle.Create(); + } + + public void SetParameter(ZSTD_dParameter parameter, int value) + { + using var dctx = handle.Acquire(); + Unsafe.Methods.ZSTD_DCtx_setParameter(dctx, parameter, value).EnsureZstdSuccess(); + } + + public int GetParameter(ZSTD_dParameter parameter) + { + using var dctx = handle.Acquire(); + int value; + Unsafe.Methods.ZSTD_DCtx_getParameter(dctx, parameter, &value).EnsureZstdSuccess(); + return value; + } + + public void LoadDictionary(byte[] dict) + { + var dictReadOnlySpan = new ReadOnlySpan(dict); + this.LoadDictionary(dictReadOnlySpan); + } + + public void LoadDictionary(ReadOnlySpan dict) + { + using var dctx = handle.Acquire(); + fixed (byte* dictPtr = dict) + { + Unsafe + .Methods.ZSTD_DCtx_loadDictionary(dctx, dictPtr, (nuint)dict.Length) + .EnsureZstdSuccess(); + } + } + + public static ulong GetDecompressedSize(ReadOnlySpan src) + { + fixed (byte* srcPtr = src) + { + return Unsafe + .Methods.ZSTD_decompressBound(srcPtr, (nuint)src.Length) + .EnsureContentSizeOk(); + } + } + + public static ulong GetDecompressedSize(ArraySegment src) => + GetDecompressedSize((ReadOnlySpan)src); + + public static ulong GetDecompressedSize(byte[] src, int srcOffset, int srcLength) => + GetDecompressedSize(new ReadOnlySpan(src, srcOffset, srcLength)); + + public Span Unwrap(ReadOnlySpan src, int maxDecompressedSize = int.MaxValue) + { + var expectedDstSize = GetDecompressedSize(src); + if (expectedDstSize > (ulong)maxDecompressedSize) + { + throw new ZstdException( + ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall, + $"Decompressed content size {expectedDstSize} is greater than {nameof(maxDecompressedSize)} {maxDecompressedSize}" + ); + } + + if (expectedDstSize > Constants.MaxByteArrayLength) + { + throw new ZstdException( + ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall, + $"Decompressed content size {expectedDstSize} is greater than max possible byte array size {Constants.MaxByteArrayLength}" + ); + } + + var dest = new byte[expectedDstSize]; + var length = Unwrap(src, dest); + return new Span(dest, 0, length); + } + + public int Unwrap(byte[] src, byte[] dest, int offset) => + Unwrap(src, new Span(dest, offset, dest.Length - offset)); + + public int Unwrap(ReadOnlySpan src, Span dest) + { + fixed (byte* srcPtr = src) + fixed (byte* destPtr = dest) + { + using var dctx = handle.Acquire(); + return (int) + Unsafe + .Methods.ZSTD_decompressDCtx( + dctx, + destPtr, + (nuint)dest.Length, + srcPtr, + (nuint)src.Length + ) + .EnsureZstdSuccess(); + } + } + + public int Unwrap( + byte[] src, + int srcOffset, + int srcLength, + byte[] dst, + int dstOffset, + int dstLength + ) => + Unwrap( + new ReadOnlySpan(src, srcOffset, srcLength), + new Span(dst, dstOffset, dstLength) + ); + + public bool TryUnwrap(byte[] src, byte[] dest, int offset, out int written) => + TryUnwrap(src, new Span(dest, offset, dest.Length - offset), out written); + + public bool TryUnwrap(ReadOnlySpan src, Span dest, out int written) + { + fixed (byte* srcPtr = src) + fixed (byte* destPtr = dest) + { + nuint returnValue; + using (var dctx = handle.Acquire()) + { + returnValue = Unsafe.Methods.ZSTD_decompressDCtx( + dctx, + destPtr, + (nuint)dest.Length, + srcPtr, + (nuint)src.Length + ); + } + + if (returnValue == unchecked(0 - (nuint)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)) + { + written = default; + return false; + } + + returnValue.EnsureZstdSuccess(); + written = (int)returnValue; + return true; + } + } + + public bool TryUnwrap( + byte[] src, + int srcOffset, + int srcLength, + byte[] dst, + int dstOffset, + int dstLength, + out int written + ) => + TryUnwrap( + new ReadOnlySpan(src, srcOffset, srcLength), + new Span(dst, dstOffset, dstLength), + out written + ); + + public void Dispose() + { + handle.Dispose(); + GC.SuppressFinalize(this); + } + + internal nuint DecompressStream(ref ZSTD_inBuffer_s input, ref ZSTD_outBuffer_s output) + { + fixed (ZSTD_inBuffer_s* inputPtr = &input) + fixed (ZSTD_outBuffer_s* outputPtr = &output) + { + using var dctx = handle.Acquire(); + return Unsafe + .Methods.ZSTD_decompressStream(dctx, outputPtr, inputPtr) + .EnsureZstdSuccess(); + } + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/JobThreadPool.cs b/src/SharpCompress/Compressors/ZStandard/JobThreadPool.cs new file mode 100644 index 00000000..313b5923 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/JobThreadPool.cs @@ -0,0 +1,155 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Threading; + +namespace SharpCompress.Compressors.ZStandard; + +internal unsafe class JobThreadPool : IDisposable +{ + private int numThreads; + private readonly List threads; + private readonly BlockingCollection queue; + + private struct Job + { + public void* function; + public void* opaque; + } + + private class JobThread + { + private Thread Thread { get; } + public CancellationTokenSource CancellationTokenSource { get; } + + public JobThread(Thread thread) + { + CancellationTokenSource = new CancellationTokenSource(); + Thread = thread; + } + + public void Start() + { + Thread.Start(this); + } + + public void Cancel() + { + CancellationTokenSource.Cancel(); + } + + public void Join() + { + Thread.Join(); + } + } + + private void Worker(object? obj) + { + if (obj is not JobThread poolThread) + { + return; + } + + var cancellationToken = poolThread.CancellationTokenSource.Token; + while (!queue.IsCompleted && !cancellationToken.IsCancellationRequested) + { + try + { + if (queue.TryTake(out var job, -1, cancellationToken)) + { + ((delegate* managed)job.function)(job.opaque); + } + } + catch (InvalidOperationException) { } + catch (OperationCanceledException) { } + } + } + + public JobThreadPool(int num, int queueSize) + { + numThreads = num; + queue = new BlockingCollection(queueSize + 1); + threads = new List(num); + for (var i = 0; i < numThreads; i++) + { + CreateThread(); + } + } + + private void CreateThread() + { + var poolThread = new JobThread(new Thread(Worker)); + threads.Add(poolThread); + poolThread.Start(); + } + + public void Resize(int num) + { + lock (threads) + { + if (num < numThreads) + { + for (var i = numThreads - 1; i >= num; i--) + { + threads[i].Cancel(); + threads.RemoveAt(i); + } + } + else + { + for (var i = numThreads; i < num; i++) + { + CreateThread(); + } + } + } + + numThreads = num; + } + + public void Add(void* function, void* opaque) + { + queue.Add(new Job { function = function, opaque = opaque }); + } + + public bool TryAdd(void* function, void* opaque) + { + return queue.TryAdd(new Job { function = function, opaque = opaque }); + } + + public void Join(bool cancel = true) + { + queue.CompleteAdding(); + List jobThreads; + lock (threads) + { + jobThreads = new List(threads); + } + + if (cancel) + { + foreach (var thread in jobThreads) + { + thread.Cancel(); + } + } + + foreach (var thread in jobThreads) + { + thread.Join(); + } + } + + public void Dispose() + { + queue.Dispose(); + } + + public int Size() + { + // todo not implemented + // https://github.com/dotnet/runtime/issues/24200 + return 0; + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/SafeHandles.cs b/src/SharpCompress/Compressors/ZStandard/SafeHandles.cs new file mode 100644 index 00000000..896dd19e --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/SafeHandles.cs @@ -0,0 +1,169 @@ +using System; +using System.Runtime.InteropServices; +using SharpCompress.Compressors.ZStandard.Unsafe; + +namespace SharpCompress.Compressors.ZStandard; + +/// +/// Provides the base class for ZstdSharp implementations. +/// +/// +/// Even though ZstdSharp is a managed library, its internals are using unmanaged +/// memory and we are using safe handles in the library's high-level API to ensure +/// proper disposal of unmanaged resources and increase safety. +/// +/// +/// +internal abstract unsafe class SafeZstdHandle : SafeHandle +{ + /// + /// Parameterless constructor is hidden. Use the static Create factory + /// method to create a new safe handle instance. + /// + protected SafeZstdHandle() + : base(IntPtr.Zero, true) { } + + public sealed override bool IsInvalid => handle == IntPtr.Zero; +} + +/// +/// Safely wraps an unmanaged Zstd compression context. +/// +internal sealed unsafe class SafeCctxHandle : SafeZstdHandle +{ + /// + internal SafeCctxHandle() { } + + /// + /// Creates a new instance of . + /// + /// + /// Creation failed. + public static SafeCctxHandle Create() + { + var safeHandle = new SafeCctxHandle(); + bool success = false; + try + { + var cctx = Unsafe.Methods.ZSTD_createCCtx(); + if (cctx == null) + { + throw new ZstdException(ZSTD_ErrorCode.ZSTD_error_GENERIC, "Failed to create cctx"); + } + + safeHandle.SetHandle((IntPtr)cctx); + success = true; + } + finally + { + if (!success) + { + safeHandle.SetHandleAsInvalid(); + } + } + return safeHandle; + } + + /// + /// Acquires a reference to the safe handle. + /// + /// + /// A instance that can be implicitly converted to a pointer + /// to . + /// + public SafeHandleHolder Acquire() => new(this); + + protected override bool ReleaseHandle() + { + return Unsafe.Methods.ZSTD_freeCCtx((ZSTD_CCtx_s*)handle) == 0; + } +} + +/// +/// Safely wraps an unmanaged Zstd compression context. +/// +internal sealed unsafe class SafeDctxHandle : SafeZstdHandle +{ + /// + internal SafeDctxHandle() { } + + /// + /// Creates a new instance of . + /// + /// + /// Creation failed. + public static SafeDctxHandle Create() + { + var safeHandle = new SafeDctxHandle(); + bool success = false; + try + { + var dctx = Unsafe.Methods.ZSTD_createDCtx(); + if (dctx == null) + { + throw new ZstdException(ZSTD_ErrorCode.ZSTD_error_GENERIC, "Failed to create dctx"); + } + + safeHandle.SetHandle((IntPtr)dctx); + success = true; + } + finally + { + if (!success) + { + safeHandle.SetHandleAsInvalid(); + } + } + return safeHandle; + } + + /// + /// Acquires a reference to the safe handle. + /// + /// + /// A instance that can be implicitly converted to a pointer + /// to . + /// + public SafeHandleHolder Acquire() => new(this); + + protected override bool ReleaseHandle() + { + return Unsafe.Methods.ZSTD_freeDCtx((ZSTD_DCtx_s*)handle) == 0; + } +} + +/// +/// Provides a convenient interface to safely acquire pointers of a specific type +/// from a , by utilizing blocks. +/// +/// The type of pointers to return. +/// +/// Safe handle holders can be d to decrement the safe handle's +/// reference count, and can be implicitly converted to pointers to . +/// +internal unsafe ref struct SafeHandleHolder + where T : unmanaged +{ + private readonly SafeHandle _handle; + + private bool _refAdded; + + public SafeHandleHolder(SafeHandle safeHandle) + { + _handle = safeHandle; + _refAdded = false; + safeHandle.DangerousAddRef(ref _refAdded); + } + + public static implicit operator T*(SafeHandleHolder holder) => + (T*)holder._handle.DangerousGetHandle(); + + public void Dispose() + { + if (_refAdded) + { + _handle.DangerousRelease(); + _refAdded = false; + } + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/SynchronizationWrapper.cs b/src/SharpCompress/Compressors/ZStandard/SynchronizationWrapper.cs new file mode 100644 index 00000000..406cacd4 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/SynchronizationWrapper.cs @@ -0,0 +1,22 @@ +using System.Threading; + +namespace SharpCompress.Compressors.ZStandard; + +internal static unsafe class SynchronizationWrapper +{ + private static object UnwrapObject(void** obj) => UnmanagedObject.Unwrap(*obj); + + public static void Init(void** obj) => *obj = UnmanagedObject.Wrap(new object()); + + public static void Free(void** obj) => UnmanagedObject.Free(*obj); + + public static void Enter(void** obj) => Monitor.Enter(UnwrapObject(obj)); + + public static void Exit(void** obj) => Monitor.Exit(UnwrapObject(obj)); + + public static void Pulse(void** obj) => Monitor.Pulse(UnwrapObject(obj)); + + public static void PulseAll(void** obj) => Monitor.PulseAll(UnwrapObject(obj)); + + public static void Wait(void** mutex) => Monitor.Wait(UnwrapObject(mutex)); +} diff --git a/src/SharpCompress/Compressors/ZStandard/ThrowHelper.cs b/src/SharpCompress/Compressors/ZStandard/ThrowHelper.cs new file mode 100644 index 00000000..a0b53d99 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/ThrowHelper.cs @@ -0,0 +1,56 @@ +using SharpCompress.Compressors.ZStandard.Unsafe; + +namespace SharpCompress.Compressors.ZStandard; + +public static unsafe class ThrowHelper +{ + private const ulong ZSTD_CONTENTSIZE_UNKNOWN = unchecked(0UL - 1); + private const ulong ZSTD_CONTENTSIZE_ERROR = unchecked(0UL - 2); + + public static nuint EnsureZstdSuccess(this nuint returnValue) + { + if (Unsafe.Methods.ZSTD_isError(returnValue)) + { + ThrowException(returnValue, Unsafe.Methods.ZSTD_getErrorName(returnValue)); + } + + return returnValue; + } + + public static nuint EnsureZdictSuccess(this nuint returnValue) + { + if (Unsafe.Methods.ZDICT_isError(returnValue)) + { + ThrowException(returnValue, Unsafe.Methods.ZDICT_getErrorName(returnValue)); + } + + return returnValue; + } + + public static ulong EnsureContentSizeOk(this ulong returnValue) + { + if (returnValue == ZSTD_CONTENTSIZE_UNKNOWN) + { + throw new ZstdException( + ZSTD_ErrorCode.ZSTD_error_GENERIC, + "Decompressed content size is not specified" + ); + } + + if (returnValue == ZSTD_CONTENTSIZE_ERROR) + { + throw new ZstdException( + ZSTD_ErrorCode.ZSTD_error_GENERIC, + "Decompressed content size cannot be determined (e.g. invalid magic number, srcSize too small)" + ); + } + + return returnValue; + } + + private static void ThrowException(nuint returnValue, string message) + { + var code = 0 - returnValue; + throw new ZstdException((ZSTD_ErrorCode)code, message); + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/UnmanagedObject.cs b/src/SharpCompress/Compressors/ZStandard/UnmanagedObject.cs new file mode 100644 index 00000000..a5bb31be --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/UnmanagedObject.cs @@ -0,0 +1,18 @@ +using System; +using System.Runtime.InteropServices; + +namespace SharpCompress.Compressors.ZStandard; + +/* + * Wrap object to void* to make it unmanaged + */ +internal static unsafe class UnmanagedObject +{ + public static void* Wrap(object obj) => (void*)GCHandle.ToIntPtr(GCHandle.Alloc(obj)); + + private static GCHandle UnwrapGcHandle(void* value) => GCHandle.FromIntPtr((IntPtr)value); + + public static T Unwrap(void* value) => (T)UnwrapGcHandle(value).Target!; + + public static void Free(void* value) => UnwrapGcHandle(value).Free(); +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/Allocations.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/Allocations.cs new file mode 100644 index 00000000..1b557129 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/Allocations.cs @@ -0,0 +1,59 @@ +using System.Runtime.CompilerServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + /* custom memory allocation functions */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void* ZSTD_customMalloc(nuint size, ZSTD_customMem customMem) + { + if (customMem.customAlloc != null) + { + return ((delegate* managed)customMem.customAlloc)( + customMem.opaque, + size + ); + } + + return malloc(size); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void* ZSTD_customCalloc(nuint size, ZSTD_customMem customMem) + { + if (customMem.customAlloc != null) + { + /* calloc implemented as malloc+memset; + * not as efficient as calloc, but next best guess for custom malloc */ + void* ptr = ((delegate* managed)customMem.customAlloc)( + customMem.opaque, + size + ); + memset(ptr, 0, (uint)size); + return ptr; + } + + return calloc(1, size); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_customFree(void* ptr, ZSTD_customMem customMem) + { + if (ptr != null) + { + if (customMem.customFree != null) + { + ((delegate* managed)customMem.customFree)( + customMem.opaque, + ptr + ); + } + else + { + free(ptr); + } + } + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/BIT_CStream_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/BIT_CStream_t.cs new file mode 100644 index 00000000..ab23c39a --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/BIT_CStream_t.cs @@ -0,0 +1,14 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/* bitStream can mix input from multiple sources. + * A critical property of these streams is that they encode and decode in **reverse** direction. + * So the first bit sequence you add will be the last to be read, like a LIFO stack. + */ +public unsafe struct BIT_CStream_t +{ + public nuint bitContainer; + public uint bitPos; + public sbyte* startPtr; + public sbyte* ptr; + public sbyte* endPtr; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/BIT_DStream_status.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/BIT_DStream_status.cs new file mode 100644 index 00000000..60b468b1 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/BIT_DStream_status.cs @@ -0,0 +1,16 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum BIT_DStream_status +{ + /* fully refilled */ + BIT_DStream_unfinished = 0, + + /* still some bits left in bitstream */ + BIT_DStream_endOfBuffer = 1, + + /* bitstream entirely consumed, bit-exact */ + BIT_DStream_completed = 2, + + /* user requested more bits than present in bitstream */ + BIT_DStream_overflow = 3, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/BIT_DStream_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/BIT_DStream_t.cs new file mode 100644 index 00000000..c0e96134 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/BIT_DStream_t.cs @@ -0,0 +1,13 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/*-******************************************** + * bitStream decoding API (read backward) + **********************************************/ +public unsafe struct BIT_DStream_t +{ + public nuint bitContainer; + public uint bitsConsumed; + public sbyte* ptr; + public sbyte* start; + public sbyte* limitPtr; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/Bits.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/Bits.cs new file mode 100644 index 00000000..e5484ae1 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/Bits.cs @@ -0,0 +1,60 @@ +using System; +using System.Numerics; +using System.Runtime.CompilerServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_countTrailingZeros32(uint val) + { + assert(val != 0); + return (uint)BitOperations.TrailingZeroCount(val); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_countLeadingZeros32(uint val) + { + assert(val != 0); + return (uint)BitOperations.LeadingZeroCount(val); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_countTrailingZeros64(ulong val) + { + assert(val != 0); + return (uint)BitOperations.TrailingZeroCount(val); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_countLeadingZeros64(ulong val) + { + assert(val != 0); + return (uint)BitOperations.LeadingZeroCount(val); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_NbCommonBytes(nuint val) + { + assert(val != 0); + if (BitConverter.IsLittleEndian) + { + return MEM_64bits + ? (uint)BitOperations.TrailingZeroCount(val) >> 3 + : (uint)BitOperations.TrailingZeroCount((uint)val) >> 3; + } + + return MEM_64bits + ? (uint)BitOperations.LeadingZeroCount(val) >> 3 + : (uint)BitOperations.LeadingZeroCount((uint)val) >> 3; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_highbit32(uint val) + { + assert(val != 0); + return (uint)BitOperations.Log2(val); + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/Bitstream.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/Bitstream.cs new file mode 100644 index 00000000..513fe1b7 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/Bitstream.cs @@ -0,0 +1,768 @@ +using System; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; +#if NETCOREAPP3_0_OR_GREATER +using System.Runtime.Intrinsics.X86; +#endif + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_BIT_mask => + new uint[32] + { + 0, + 1, + 3, + 7, + 0xF, + 0x1F, + 0x3F, + 0x7F, + 0xFF, + 0x1FF, + 0x3FF, + 0x7FF, + 0xFFF, + 0x1FFF, + 0x3FFF, + 0x7FFF, + 0xFFFF, + 0x1FFFF, + 0x3FFFF, + 0x7FFFF, + 0xFFFFF, + 0x1FFFFF, + 0x3FFFFF, + 0x7FFFFF, + 0xFFFFFF, + 0x1FFFFFF, + 0x3FFFFFF, + 0x7FFFFFF, + 0xFFFFFFF, + 0x1FFFFFFF, + 0x3FFFFFFF, + 0x7FFFFFFF, + }; + private static uint* BIT_mask => + (uint*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_BIT_mask) + ); +#else + + private static readonly uint* BIT_mask = GetArrayPointer( + new uint[32] + { + 0, + 1, + 3, + 7, + 0xF, + 0x1F, + 0x3F, + 0x7F, + 0xFF, + 0x1FF, + 0x3FF, + 0x7FF, + 0xFFF, + 0x1FFF, + 0x3FFF, + 0x7FFF, + 0xFFFF, + 0x1FFFF, + 0x3FFFF, + 0x7FFFF, + 0xFFFFF, + 0x1FFFFF, + 0x3FFFFF, + 0x7FFFFF, + 0xFFFFFF, + 0x1FFFFFF, + 0x3FFFFFF, + 0x7FFFFFF, + 0xFFFFFFF, + 0x1FFFFFFF, + 0x3FFFFFFF, + 0x7FFFFFFF, + } + ); +#endif + /*-************************************************************** + * bitStream encoding + ****************************************************************/ + /*! BIT_initCStream() : + * `dstCapacity` must be > sizeof(size_t) + * @return : 0 if success, + * otherwise an error code (can be tested using ERR_isError()) */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint BIT_initCStream(ref BIT_CStream_t bitC, void* startPtr, nuint dstCapacity) + { + bitC.bitContainer = 0; + bitC.bitPos = 0; + bitC.startPtr = (sbyte*)startPtr; + bitC.ptr = bitC.startPtr; + bitC.endPtr = bitC.startPtr + dstCapacity - sizeof(nuint); + if (dstCapacity <= (nuint)sizeof(nuint)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + return 0; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint BIT_getLowerBits(nuint bitContainer, uint nbBits) + { + assert(nbBits < sizeof(uint) * 32 / sizeof(uint)); +#if NETCOREAPP3_1_OR_GREATER + if (Bmi2.X64.IsSupported) + { + return (nuint)Bmi2.X64.ZeroHighBits(bitContainer, nbBits); + } + + if (Bmi2.IsSupported) + { + return Bmi2.ZeroHighBits((uint)bitContainer, nbBits); + } +#endif + + return bitContainer & BIT_mask[nbBits]; + } + + /*! BIT_addBits() : + * can add up to 31 bits into `bitC`. + * Note : does not check for register overflow ! */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void BIT_addBits( + ref nuint bitC_bitContainer, + ref uint bitC_bitPos, + nuint value, + uint nbBits + ) + { + assert(nbBits < sizeof(uint) * 32 / sizeof(uint)); + assert(nbBits + bitC_bitPos < (uint)(sizeof(nuint) * 8)); + bitC_bitContainer |= BIT_getLowerBits(value, nbBits) << (int)bitC_bitPos; + bitC_bitPos += nbBits; + } + + /*! BIT_addBitsFast() : + * works only if `value` is _clean_, + * meaning all high bits above nbBits are 0 */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void BIT_addBitsFast( + ref nuint bitC_bitContainer, + ref uint bitC_bitPos, + nuint value, + uint nbBits + ) + { + assert(value >> (int)nbBits == 0); + assert(nbBits + bitC_bitPos < (uint)(sizeof(nuint) * 8)); + bitC_bitContainer |= value << (int)bitC_bitPos; + bitC_bitPos += nbBits; + } + + /*! BIT_flushBitsFast() : + * assumption : bitContainer has not overflowed + * unsafe version; does not check buffer overflow */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void BIT_flushBitsFast( + ref nuint bitC_bitContainer, + ref uint bitC_bitPos, + ref sbyte* bitC_ptr, + sbyte* bitC_endPtr + ) + { + nuint nbBytes = bitC_bitPos >> 3; + assert(bitC_bitPos < (uint)(sizeof(nuint) * 8)); + assert(bitC_ptr <= bitC_endPtr); + MEM_writeLEST(bitC_ptr, bitC_bitContainer); + bitC_ptr += nbBytes; + bitC_bitPos &= 7; + bitC_bitContainer >>= (int)(nbBytes * 8); + } + + /*! BIT_flushBits() : + * assumption : bitContainer has not overflowed + * safe version; check for buffer overflow, and prevents it. + * note : does not signal buffer overflow. + * overflow will be revealed later on using BIT_closeCStream() */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void BIT_flushBits( + ref nuint bitC_bitContainer, + ref uint bitC_bitPos, + ref sbyte* bitC_ptr, + sbyte* bitC_endPtr + ) + { + nuint nbBytes = bitC_bitPos >> 3; + assert(bitC_bitPos < (uint)(sizeof(nuint) * 8)); + assert(bitC_ptr <= bitC_endPtr); + MEM_writeLEST(bitC_ptr, bitC_bitContainer); + bitC_ptr += nbBytes; + if (bitC_ptr > bitC_endPtr) + { + bitC_ptr = bitC_endPtr; + } + + bitC_bitPos &= 7; + bitC_bitContainer >>= (int)(nbBytes * 8); + } + + /*! BIT_closeCStream() : + * @return : size of CStream, in bytes, + * or 0 if it could not fit into dstBuffer */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint BIT_closeCStream( + ref nuint bitC_bitContainer, + ref uint bitC_bitPos, + sbyte* bitC_ptr, + sbyte* bitC_endPtr, + sbyte* bitC_startPtr + ) + { + BIT_addBitsFast(ref bitC_bitContainer, ref bitC_bitPos, 1, 1); + BIT_flushBits(ref bitC_bitContainer, ref bitC_bitPos, ref bitC_ptr, bitC_endPtr); + if (bitC_ptr >= bitC_endPtr) + { + return 0; + } + + return (nuint)(bitC_ptr - bitC_startPtr) + (nuint)(bitC_bitPos > 0 ? 1 : 0); + } + + /*-******************************************************** + * bitStream decoding + **********************************************************/ + /*! BIT_initDStream() : + * Initialize a BIT_DStream_t. + * `bitD` : a pointer to an already allocated BIT_DStream_t structure. + * `srcSize` must be the *exact* size of the bitStream, in bytes. + * @return : size of stream (== srcSize), or an errorCode if a problem is detected + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint BIT_initDStream(BIT_DStream_t* bitD, void* srcBuffer, nuint srcSize) + { + if (srcSize < 1) + { + *bitD = new BIT_DStream_t(); + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + bitD->start = (sbyte*)srcBuffer; + bitD->limitPtr = bitD->start + sizeof(nuint); + if (srcSize >= (nuint)sizeof(nuint)) + { + bitD->ptr = (sbyte*)srcBuffer + srcSize - sizeof(nuint); + bitD->bitContainer = MEM_readLEST(bitD->ptr); + { + byte lastByte = ((byte*)srcBuffer)[srcSize - 1]; + bitD->bitsConsumed = lastByte != 0 ? 8 - ZSTD_highbit32(lastByte) : 0; + if (lastByte == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + } + } + else + { + bitD->ptr = bitD->start; + bitD->bitContainer = *(byte*)bitD->start; + switch (srcSize) + { + case 7: + bitD->bitContainer += (nuint)((byte*)srcBuffer)[6] << sizeof(nuint) * 8 - 16; + goto case 6; + case 6: + bitD->bitContainer += (nuint)((byte*)srcBuffer)[5] << sizeof(nuint) * 8 - 24; + goto case 5; + case 5: + bitD->bitContainer += (nuint)((byte*)srcBuffer)[4] << sizeof(nuint) * 8 - 32; + goto case 4; + case 4: + bitD->bitContainer += (nuint)((byte*)srcBuffer)[3] << 24; + goto case 3; + case 3: + bitD->bitContainer += (nuint)((byte*)srcBuffer)[2] << 16; + goto case 2; + case 2: + bitD->bitContainer += (nuint)((byte*)srcBuffer)[1] << 8; + goto default; + default: + break; + } + + { + byte lastByte = ((byte*)srcBuffer)[srcSize - 1]; + bitD->bitsConsumed = lastByte != 0 ? 8 - ZSTD_highbit32(lastByte) : 0; + if (lastByte == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + } + + bitD->bitsConsumed += (uint)((nuint)sizeof(nuint) - srcSize) * 8; + } + + return srcSize; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint BIT_getUpperBits(nuint bitContainer, uint start) + { + return bitContainer >> (int)start; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint BIT_getMiddleBits(nuint bitContainer, uint start, uint nbBits) + { + uint regMask = (uint)(sizeof(nuint) * 8 - 1); + assert(nbBits < sizeof(uint) * 32 / sizeof(uint)); +#if NETCOREAPP3_1_OR_GREATER + if (Bmi2.X64.IsSupported) + { + return (nuint)Bmi2.X64.ZeroHighBits(bitContainer >> (int)(start & regMask), nbBits); + } + + if (Bmi2.IsSupported) + { + return Bmi2.ZeroHighBits((uint)(bitContainer >> (int)(start & regMask)), nbBits); + } +#endif + + return (nuint)(bitContainer >> (int)(start & regMask) & ((ulong)1 << (int)nbBits) - 1); + } + + /*! BIT_lookBits() : + * Provides next n bits from local register. + * local register is not modified. + * On 32-bits, maxNbBits==24. + * On 64-bits, maxNbBits==56. + * @return : value extracted */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint BIT_lookBits(BIT_DStream_t* bitD, uint nbBits) + { + return BIT_getMiddleBits( + bitD->bitContainer, + (uint)(sizeof(nuint) * 8) - bitD->bitsConsumed - nbBits, + nbBits + ); + } + + /*! BIT_lookBitsFast() : + * unsafe version; only works if nbBits >= 1 */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint BIT_lookBitsFast(BIT_DStream_t* bitD, uint nbBits) + { + uint regMask = (uint)(sizeof(nuint) * 8 - 1); + assert(nbBits >= 1); + return bitD->bitContainer + << (int)(bitD->bitsConsumed & regMask) + >> (int)(regMask + 1 - nbBits & regMask); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void BIT_skipBits(BIT_DStream_t* bitD, uint nbBits) + { + bitD->bitsConsumed += nbBits; + } + + /*! BIT_readBits() : + * Read (consume) next n bits from local register and update. + * Pay attention to not read more than nbBits contained into local register. + * @return : extracted value. */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint BIT_readBits(BIT_DStream_t* bitD, uint nbBits) + { + nuint value = BIT_lookBits(bitD, nbBits); + BIT_skipBits(bitD, nbBits); + return value; + } + + /*! BIT_readBitsFast() : + * unsafe version; only works if nbBits >= 1 */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint BIT_readBitsFast(BIT_DStream_t* bitD, uint nbBits) + { + nuint value = BIT_lookBitsFast(bitD, nbBits); + assert(nbBits >= 1); + BIT_skipBits(bitD, nbBits); + return value; + } + + /*! BIT_reloadDStream_internal() : + * Simple variant of BIT_reloadDStream(), with two conditions: + * 1. bitstream is valid : bitsConsumed <= sizeof(bitD->bitContainer)*8 + * 2. look window is valid after shifted down : bitD->ptr >= bitD->start + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static BIT_DStream_status BIT_reloadDStream_internal(BIT_DStream_t* bitD) + { + assert(bitD->bitsConsumed <= (uint)(sizeof(nuint) * 8)); + bitD->ptr -= bitD->bitsConsumed >> 3; + assert(bitD->ptr >= bitD->start); + bitD->bitsConsumed &= 7; + bitD->bitContainer = MEM_readLEST(bitD->ptr); + return BIT_DStream_status.BIT_DStream_unfinished; + } + + /*! BIT_reloadDStreamFast() : + * Similar to BIT_reloadDStream(), but with two differences: + * 1. bitsConsumed <= sizeof(bitD->bitContainer)*8 must hold! + * 2. Returns BIT_DStream_overflow when bitD->ptr < bitD->limitPtr, at this + * point you must use BIT_reloadDStream() to reload. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static BIT_DStream_status BIT_reloadDStreamFast(BIT_DStream_t* bitD) + { + if (bitD->ptr < bitD->limitPtr) + { + return BIT_DStream_status.BIT_DStream_overflow; + } + + return BIT_reloadDStream_internal(bitD); + } + +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_static_zeroFilled => + new byte[] { 0, 0, 0, 0, 0, 0, 0, 0 }; + private static nuint* static_zeroFilled => + (nuint*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_static_zeroFilled) + ); +#else + + private static readonly nuint* static_zeroFilled = (nuint*)GetArrayPointer( + new byte[] { 0, 0, 0, 0, 0, 0, 0, 0 } + ); +#endif + /*! BIT_reloadDStream() : + * Refill `bitD` from buffer previously set in BIT_initDStream() . + * This function is safe, it guarantees it will not never beyond src buffer. + * @return : status of `BIT_DStream_t` internal register. + * when status == BIT_DStream_unfinished, internal register is filled with at least 25 or 57 bits */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static BIT_DStream_status BIT_reloadDStream(BIT_DStream_t* bitD) + { + if (bitD->bitsConsumed > (uint)(sizeof(nuint) * 8)) + { + bitD->ptr = (sbyte*)&static_zeroFilled[0]; + return BIT_DStream_status.BIT_DStream_overflow; + } + + assert(bitD->ptr >= bitD->start); + if (bitD->ptr >= bitD->limitPtr) + { + return BIT_reloadDStream_internal(bitD); + } + + if (bitD->ptr == bitD->start) + { + if (bitD->bitsConsumed < (uint)(sizeof(nuint) * 8)) + { + return BIT_DStream_status.BIT_DStream_endOfBuffer; + } + + return BIT_DStream_status.BIT_DStream_completed; + } + + { + uint nbBytes = bitD->bitsConsumed >> 3; + BIT_DStream_status result = BIT_DStream_status.BIT_DStream_unfinished; + if (bitD->ptr - nbBytes < bitD->start) + { + nbBytes = (uint)(bitD->ptr - bitD->start); + result = BIT_DStream_status.BIT_DStream_endOfBuffer; + } + + bitD->ptr -= nbBytes; + bitD->bitsConsumed -= nbBytes * 8; + bitD->bitContainer = MEM_readLEST(bitD->ptr); + return result; + } + } + + /*! BIT_endOfDStream() : + * @return : 1 if DStream has _exactly_ reached its end (all bits consumed). + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint BIT_endOfDStream(BIT_DStream_t* DStream) + { + return DStream->ptr == DStream->start && DStream->bitsConsumed == (uint)(sizeof(nuint) * 8) + ? 1U + : 0U; + } + + /*-******************************************************** + * bitStream decoding + **********************************************************/ + /*! BIT_initDStream() : + * Initialize a BIT_DStream_t. + * `bitD` : a pointer to an already allocated BIT_DStream_t structure. + * `srcSize` must be the *exact* size of the bitStream, in bytes. + * @return : size of stream (== srcSize), or an errorCode if a problem is detected + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint BIT_initDStream(ref BIT_DStream_t bitD, void* srcBuffer, nuint srcSize) + { + if (srcSize < 1) + { + bitD = new BIT_DStream_t(); + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + bitD.start = (sbyte*)srcBuffer; + bitD.limitPtr = bitD.start + sizeof(nuint); + if (srcSize >= (nuint)sizeof(nuint)) + { + bitD.ptr = (sbyte*)srcBuffer + srcSize - sizeof(nuint); + bitD.bitContainer = MEM_readLEST(bitD.ptr); + { + byte lastByte = ((byte*)srcBuffer)[srcSize - 1]; + bitD.bitsConsumed = lastByte != 0 ? 8 - ZSTD_highbit32(lastByte) : 0; + if (lastByte == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + } + } + else + { + bitD.ptr = bitD.start; + bitD.bitContainer = *(byte*)bitD.start; + switch (srcSize) + { + case 7: + bitD.bitContainer += (nuint)((byte*)srcBuffer)[6] << sizeof(nuint) * 8 - 16; + goto case 6; + case 6: + bitD.bitContainer += (nuint)((byte*)srcBuffer)[5] << sizeof(nuint) * 8 - 24; + goto case 5; + case 5: + bitD.bitContainer += (nuint)((byte*)srcBuffer)[4] << sizeof(nuint) * 8 - 32; + goto case 4; + case 4: + bitD.bitContainer += (nuint)((byte*)srcBuffer)[3] << 24; + goto case 3; + case 3: + bitD.bitContainer += (nuint)((byte*)srcBuffer)[2] << 16; + goto case 2; + case 2: + bitD.bitContainer += (nuint)((byte*)srcBuffer)[1] << 8; + goto default; + default: + break; + } + + { + byte lastByte = ((byte*)srcBuffer)[srcSize - 1]; + bitD.bitsConsumed = lastByte != 0 ? 8 - ZSTD_highbit32(lastByte) : 0; + if (lastByte == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + } + + bitD.bitsConsumed += (uint)((nuint)sizeof(nuint) - srcSize) * 8; + } + + return srcSize; + } + + /*! BIT_lookBits() : + * Provides next n bits from local register. + * local register is not modified. + * On 32-bits, maxNbBits==24. + * On 64-bits, maxNbBits==56. + * @return : value extracted */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint BIT_lookBits(nuint bitD_bitContainer, uint bitD_bitsConsumed, uint nbBits) + { + return BIT_getMiddleBits( + bitD_bitContainer, + (uint)(sizeof(nuint) * 8) - bitD_bitsConsumed - nbBits, + nbBits + ); + } + + /*! BIT_lookBitsFast() : + * unsafe version; only works if nbBits >= 1 */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint BIT_lookBitsFast( + nuint bitD_bitContainer, + uint bitD_bitsConsumed, + uint nbBits + ) + { + uint regMask = (uint)(sizeof(nuint) * 8 - 1); + assert(nbBits >= 1); + return bitD_bitContainer + << (int)(bitD_bitsConsumed & regMask) + >> (int)(regMask + 1 - nbBits & regMask); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void BIT_skipBits(ref uint bitD_bitsConsumed, uint nbBits) + { + bitD_bitsConsumed += nbBits; + } + + /*! BIT_readBits() : + * Read (consume) next n bits from local register and update. + * Pay attention to not read more than nbBits contained into local register. + * @return : extracted value. */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint BIT_readBits( + nuint bitD_bitContainer, + ref uint bitD_bitsConsumed, + uint nbBits + ) + { + nuint value = BIT_lookBits(bitD_bitContainer, bitD_bitsConsumed, nbBits); + BIT_skipBits(ref bitD_bitsConsumed, nbBits); + return value; + } + + /*! BIT_readBitsFast() : + * unsafe version; only works if nbBits >= 1 */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint BIT_readBitsFast( + nuint bitD_bitContainer, + ref uint bitD_bitsConsumed, + uint nbBits + ) + { + nuint value = BIT_lookBitsFast(bitD_bitContainer, bitD_bitsConsumed, nbBits); + assert(nbBits >= 1); + BIT_skipBits(ref bitD_bitsConsumed, nbBits); + return value; + } + + /*! BIT_reloadDStreamFast() : + * Similar to BIT_reloadDStream(), but with two differences: + * 1. bitsConsumed <= sizeof(bitD->bitContainer)*8 must hold! + * 2. Returns BIT_DStream_overflow when bitD->ptr < bitD->limitPtr, at this + * point you must use BIT_reloadDStream() to reload. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static BIT_DStream_status BIT_reloadDStreamFast( + ref nuint bitD_bitContainer, + ref uint bitD_bitsConsumed, + ref sbyte* bitD_ptr, + sbyte* bitD_start, + sbyte* bitD_limitPtr + ) + { + if (bitD_ptr < bitD_limitPtr) + { + return BIT_DStream_status.BIT_DStream_overflow; + } + + return BIT_reloadDStream_internal( + ref bitD_bitContainer, + ref bitD_bitsConsumed, + ref bitD_ptr, + bitD_start + ); + } + + /*! BIT_reloadDStream() : + * Refill `bitD` from buffer previously set in BIT_initDStream() . + * This function is safe, it guarantees it will not never beyond src buffer. + * @return : status of `BIT_DStream_t` internal register. + * when status == BIT_DStream_unfinished, internal register is filled with at least 25 or 57 bits */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static BIT_DStream_status BIT_reloadDStream( + ref nuint bitD_bitContainer, + ref uint bitD_bitsConsumed, + ref sbyte* bitD_ptr, + sbyte* bitD_start, + sbyte* bitD_limitPtr + ) + { + if (bitD_bitsConsumed > (uint)(sizeof(nuint) * 8)) + { + bitD_ptr = (sbyte*)&static_zeroFilled[0]; + return BIT_DStream_status.BIT_DStream_overflow; + } + + assert(bitD_ptr >= bitD_start); + if (bitD_ptr >= bitD_limitPtr) + { + return BIT_reloadDStream_internal( + ref bitD_bitContainer, + ref bitD_bitsConsumed, + ref bitD_ptr, + bitD_start + ); + } + + if (bitD_ptr == bitD_start) + { + if (bitD_bitsConsumed < (uint)(sizeof(nuint) * 8)) + { + return BIT_DStream_status.BIT_DStream_endOfBuffer; + } + + return BIT_DStream_status.BIT_DStream_completed; + } + + { + uint nbBytes = bitD_bitsConsumed >> 3; + BIT_DStream_status result = BIT_DStream_status.BIT_DStream_unfinished; + if (bitD_ptr - nbBytes < bitD_start) + { + nbBytes = (uint)(bitD_ptr - bitD_start); + result = BIT_DStream_status.BIT_DStream_endOfBuffer; + } + + bitD_ptr -= nbBytes; + bitD_bitsConsumed -= nbBytes * 8; + bitD_bitContainer = MEM_readLEST(bitD_ptr); + return result; + } + } + + /*! BIT_reloadDStream_internal() : + * Simple variant of BIT_reloadDStream(), with two conditions: + * 1. bitstream is valid : bitsConsumed <= sizeof(bitD->bitContainer)*8 + * 2. look window is valid after shifted down : bitD->ptr >= bitD->start + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static BIT_DStream_status BIT_reloadDStream_internal( + ref nuint bitD_bitContainer, + ref uint bitD_bitsConsumed, + ref sbyte* bitD_ptr, + sbyte* bitD_start + ) + { + assert(bitD_bitsConsumed <= (uint)(sizeof(nuint) * 8)); + bitD_ptr -= bitD_bitsConsumed >> 3; + assert(bitD_ptr >= bitD_start); + bitD_bitsConsumed &= 7; + bitD_bitContainer = MEM_readLEST(bitD_ptr); + return BIT_DStream_status.BIT_DStream_unfinished; + } + + /*! BIT_endOfDStream() : + * @return : 1 if DStream has _exactly_ reached its end (all bits consumed). + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint BIT_endOfDStream( + uint DStream_bitsConsumed, + sbyte* DStream_ptr, + sbyte* DStream_start + ) + { + return DStream_ptr == DStream_start && DStream_bitsConsumed == (uint)(sizeof(nuint) * 8) + ? 1U + : 0U; + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/BlockSummary.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/BlockSummary.cs new file mode 100644 index 00000000..9a53651d --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/BlockSummary.cs @@ -0,0 +1,8 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct BlockSummary +{ + public nuint nbSequences; + public nuint blockSize; + public nuint litSize; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/COVER_best_s.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/COVER_best_s.cs new file mode 100644 index 00000000..38083132 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/COVER_best_s.cs @@ -0,0 +1,20 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/** + * COVER_best_t is used for two purposes: + * 1. Synchronizing threads. + * 2. Saving the best parameters and dictionary. + * + * All of the methods except COVER_best_init() are thread safe if zstd is + * compiled with multithreaded support. + */ +public unsafe struct COVER_best_s +{ + public void* mutex; + public void* cond; + public nuint liveJobs; + public void* dict; + public nuint dictSize; + public ZDICT_cover_params_t parameters; + public nuint compressedSize; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/COVER_ctx_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/COVER_ctx_t.cs new file mode 100644 index 00000000..28712762 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/COVER_ctx_t.cs @@ -0,0 +1,19 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/*-************************************* + * Context + ***************************************/ +public unsafe struct COVER_ctx_t +{ + public byte* samples; + public nuint* offsets; + public nuint* samplesSizes; + public nuint nbSamples; + public nuint nbTrainSamples; + public nuint nbTestSamples; + public uint* suffix; + public nuint suffixSize; + public uint* freqs; + public uint* dmerAt; + public uint d; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/COVER_dictSelection.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/COVER_dictSelection.cs new file mode 100644 index 00000000..f88d1e67 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/COVER_dictSelection.cs @@ -0,0 +1,11 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/** + * Struct used for the dictionary selection function. + */ +public unsafe struct COVER_dictSelection +{ + public byte* dictContent; + public nuint dictSize; + public nuint totalCompressedSize; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/COVER_epoch_info_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/COVER_epoch_info_t.cs new file mode 100644 index 00000000..dbb29890 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/COVER_epoch_info_t.cs @@ -0,0 +1,10 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/** + *Number of epochs and size of each epoch. + */ +public struct COVER_epoch_info_t +{ + public uint num; + public uint size; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/COVER_map_pair_t_s.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/COVER_map_pair_t_s.cs new file mode 100644 index 00000000..502f33b0 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/COVER_map_pair_t_s.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct COVER_map_pair_t_s +{ + public uint key; + public uint value; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/COVER_map_s.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/COVER_map_s.cs new file mode 100644 index 00000000..dbfa337f --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/COVER_map_s.cs @@ -0,0 +1,9 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct COVER_map_s +{ + public COVER_map_pair_t_s* data; + public uint sizeLog; + public uint size; + public uint sizeMask; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/COVER_segment_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/COVER_segment_t.cs new file mode 100644 index 00000000..7e852ea9 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/COVER_segment_t.cs @@ -0,0 +1,11 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/** + * A segment is a range in the source as well as the score of the segment. + */ +public struct COVER_segment_t +{ + public uint begin; + public uint end; + public uint score; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/COVER_tryParameters_data_s.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/COVER_tryParameters_data_s.cs new file mode 100644 index 00000000..789ca9dc --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/COVER_tryParameters_data_s.cs @@ -0,0 +1,12 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/** + * Parameters for COVER_tryParameters(). + */ +public unsafe struct COVER_tryParameters_data_s +{ + public COVER_ctx_t* ctx; + public COVER_best_s* best; + public nuint dictBufferCapacity; + public ZDICT_cover_params_t parameters; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/Clevels.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/Clevels.cs new file mode 100644 index 00000000..9b3e7839 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/Clevels.cs @@ -0,0 +1,849 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + private static readonly ZSTD_compressionParameters[][] ZSTD_defaultCParameters = + new ZSTD_compressionParameters[4][] + { + new ZSTD_compressionParameters[23] + { + new ZSTD_compressionParameters( + windowLog: 19, + chainLog: 12, + hashLog: 13, + searchLog: 1, + minMatch: 6, + targetLength: 1, + strategy: ZSTD_strategy.ZSTD_fast + ), + new ZSTD_compressionParameters( + windowLog: 19, + chainLog: 13, + hashLog: 14, + searchLog: 1, + minMatch: 7, + targetLength: 0, + strategy: ZSTD_strategy.ZSTD_fast + ), + new ZSTD_compressionParameters( + windowLog: 20, + chainLog: 15, + hashLog: 16, + searchLog: 1, + minMatch: 6, + targetLength: 0, + strategy: ZSTD_strategy.ZSTD_fast + ), + new ZSTD_compressionParameters( + windowLog: 21, + chainLog: 16, + hashLog: 17, + searchLog: 1, + minMatch: 5, + targetLength: 0, + strategy: ZSTD_strategy.ZSTD_dfast + ), + new ZSTD_compressionParameters( + windowLog: 21, + chainLog: 18, + hashLog: 18, + searchLog: 1, + minMatch: 5, + targetLength: 0, + strategy: ZSTD_strategy.ZSTD_dfast + ), + new ZSTD_compressionParameters( + windowLog: 21, + chainLog: 18, + hashLog: 19, + searchLog: 3, + minMatch: 5, + targetLength: 2, + strategy: ZSTD_strategy.ZSTD_greedy + ), + new ZSTD_compressionParameters( + windowLog: 21, + chainLog: 18, + hashLog: 19, + searchLog: 3, + minMatch: 5, + targetLength: 4, + strategy: ZSTD_strategy.ZSTD_lazy + ), + new ZSTD_compressionParameters( + windowLog: 21, + chainLog: 19, + hashLog: 20, + searchLog: 4, + minMatch: 5, + targetLength: 8, + strategy: ZSTD_strategy.ZSTD_lazy + ), + new ZSTD_compressionParameters( + windowLog: 21, + chainLog: 19, + hashLog: 20, + searchLog: 4, + minMatch: 5, + targetLength: 16, + strategy: ZSTD_strategy.ZSTD_lazy2 + ), + new ZSTD_compressionParameters( + windowLog: 22, + chainLog: 20, + hashLog: 21, + searchLog: 4, + minMatch: 5, + targetLength: 16, + strategy: ZSTD_strategy.ZSTD_lazy2 + ), + new ZSTD_compressionParameters( + windowLog: 22, + chainLog: 21, + hashLog: 22, + searchLog: 5, + minMatch: 5, + targetLength: 16, + strategy: ZSTD_strategy.ZSTD_lazy2 + ), + new ZSTD_compressionParameters( + windowLog: 22, + chainLog: 21, + hashLog: 22, + searchLog: 6, + minMatch: 5, + targetLength: 16, + strategy: ZSTD_strategy.ZSTD_lazy2 + ), + new ZSTD_compressionParameters( + windowLog: 22, + chainLog: 22, + hashLog: 23, + searchLog: 6, + minMatch: 5, + targetLength: 32, + strategy: ZSTD_strategy.ZSTD_lazy2 + ), + new ZSTD_compressionParameters( + windowLog: 22, + chainLog: 22, + hashLog: 22, + searchLog: 4, + minMatch: 5, + targetLength: 32, + strategy: ZSTD_strategy.ZSTD_btlazy2 + ), + new ZSTD_compressionParameters( + windowLog: 22, + chainLog: 22, + hashLog: 23, + searchLog: 5, + minMatch: 5, + targetLength: 32, + strategy: ZSTD_strategy.ZSTD_btlazy2 + ), + new ZSTD_compressionParameters( + windowLog: 22, + chainLog: 23, + hashLog: 23, + searchLog: 6, + minMatch: 5, + targetLength: 32, + strategy: ZSTD_strategy.ZSTD_btlazy2 + ), + new ZSTD_compressionParameters( + windowLog: 22, + chainLog: 22, + hashLog: 22, + searchLog: 5, + minMatch: 5, + targetLength: 48, + strategy: ZSTD_strategy.ZSTD_btopt + ), + new ZSTD_compressionParameters( + windowLog: 23, + chainLog: 23, + hashLog: 22, + searchLog: 5, + minMatch: 4, + targetLength: 64, + strategy: ZSTD_strategy.ZSTD_btopt + ), + new ZSTD_compressionParameters( + windowLog: 23, + chainLog: 23, + hashLog: 22, + searchLog: 6, + minMatch: 3, + targetLength: 64, + strategy: ZSTD_strategy.ZSTD_btultra + ), + new ZSTD_compressionParameters( + windowLog: 23, + chainLog: 24, + hashLog: 22, + searchLog: 7, + minMatch: 3, + targetLength: 256, + strategy: ZSTD_strategy.ZSTD_btultra2 + ), + new ZSTD_compressionParameters( + windowLog: 25, + chainLog: 25, + hashLog: 23, + searchLog: 7, + minMatch: 3, + targetLength: 256, + strategy: ZSTD_strategy.ZSTD_btultra2 + ), + new ZSTD_compressionParameters( + windowLog: 26, + chainLog: 26, + hashLog: 24, + searchLog: 7, + minMatch: 3, + targetLength: 512, + strategy: ZSTD_strategy.ZSTD_btultra2 + ), + new ZSTD_compressionParameters( + windowLog: 27, + chainLog: 27, + hashLog: 25, + searchLog: 9, + minMatch: 3, + targetLength: 999, + strategy: ZSTD_strategy.ZSTD_btultra2 + ), + }, + new ZSTD_compressionParameters[23] + { + new ZSTD_compressionParameters( + windowLog: 18, + chainLog: 12, + hashLog: 13, + searchLog: 1, + minMatch: 5, + targetLength: 1, + strategy: ZSTD_strategy.ZSTD_fast + ), + new ZSTD_compressionParameters( + windowLog: 18, + chainLog: 13, + hashLog: 14, + searchLog: 1, + minMatch: 6, + targetLength: 0, + strategy: ZSTD_strategy.ZSTD_fast + ), + new ZSTD_compressionParameters( + windowLog: 18, + chainLog: 14, + hashLog: 14, + searchLog: 1, + minMatch: 5, + targetLength: 0, + strategy: ZSTD_strategy.ZSTD_dfast + ), + new ZSTD_compressionParameters( + windowLog: 18, + chainLog: 16, + hashLog: 16, + searchLog: 1, + minMatch: 4, + targetLength: 0, + strategy: ZSTD_strategy.ZSTD_dfast + ), + new ZSTD_compressionParameters( + windowLog: 18, + chainLog: 16, + hashLog: 17, + searchLog: 3, + minMatch: 5, + targetLength: 2, + strategy: ZSTD_strategy.ZSTD_greedy + ), + new ZSTD_compressionParameters( + windowLog: 18, + chainLog: 17, + hashLog: 18, + searchLog: 5, + minMatch: 5, + targetLength: 2, + strategy: ZSTD_strategy.ZSTD_greedy + ), + new ZSTD_compressionParameters( + windowLog: 18, + chainLog: 18, + hashLog: 19, + searchLog: 3, + minMatch: 5, + targetLength: 4, + strategy: ZSTD_strategy.ZSTD_lazy + ), + new ZSTD_compressionParameters( + windowLog: 18, + chainLog: 18, + hashLog: 19, + searchLog: 4, + minMatch: 4, + targetLength: 4, + strategy: ZSTD_strategy.ZSTD_lazy + ), + new ZSTD_compressionParameters( + windowLog: 18, + chainLog: 18, + hashLog: 19, + searchLog: 4, + minMatch: 4, + targetLength: 8, + strategy: ZSTD_strategy.ZSTD_lazy2 + ), + new ZSTD_compressionParameters( + windowLog: 18, + chainLog: 18, + hashLog: 19, + searchLog: 5, + minMatch: 4, + targetLength: 8, + strategy: ZSTD_strategy.ZSTD_lazy2 + ), + new ZSTD_compressionParameters( + windowLog: 18, + chainLog: 18, + hashLog: 19, + searchLog: 6, + minMatch: 4, + targetLength: 8, + strategy: ZSTD_strategy.ZSTD_lazy2 + ), + new ZSTD_compressionParameters( + windowLog: 18, + chainLog: 18, + hashLog: 19, + searchLog: 5, + minMatch: 4, + targetLength: 12, + strategy: ZSTD_strategy.ZSTD_btlazy2 + ), + new ZSTD_compressionParameters( + windowLog: 18, + chainLog: 19, + hashLog: 19, + searchLog: 7, + minMatch: 4, + targetLength: 12, + strategy: ZSTD_strategy.ZSTD_btlazy2 + ), + new ZSTD_compressionParameters( + windowLog: 18, + chainLog: 18, + hashLog: 19, + searchLog: 4, + minMatch: 4, + targetLength: 16, + strategy: ZSTD_strategy.ZSTD_btopt + ), + new ZSTD_compressionParameters( + windowLog: 18, + chainLog: 18, + hashLog: 19, + searchLog: 4, + minMatch: 3, + targetLength: 32, + strategy: ZSTD_strategy.ZSTD_btopt + ), + new ZSTD_compressionParameters( + windowLog: 18, + chainLog: 18, + hashLog: 19, + searchLog: 6, + minMatch: 3, + targetLength: 128, + strategy: ZSTD_strategy.ZSTD_btopt + ), + new ZSTD_compressionParameters( + windowLog: 18, + chainLog: 19, + hashLog: 19, + searchLog: 6, + minMatch: 3, + targetLength: 128, + strategy: ZSTD_strategy.ZSTD_btultra + ), + new ZSTD_compressionParameters( + windowLog: 18, + chainLog: 19, + hashLog: 19, + searchLog: 8, + minMatch: 3, + targetLength: 256, + strategy: ZSTD_strategy.ZSTD_btultra + ), + new ZSTD_compressionParameters( + windowLog: 18, + chainLog: 19, + hashLog: 19, + searchLog: 6, + minMatch: 3, + targetLength: 128, + strategy: ZSTD_strategy.ZSTD_btultra2 + ), + new ZSTD_compressionParameters( + windowLog: 18, + chainLog: 19, + hashLog: 19, + searchLog: 8, + minMatch: 3, + targetLength: 256, + strategy: ZSTD_strategy.ZSTD_btultra2 + ), + new ZSTD_compressionParameters( + windowLog: 18, + chainLog: 19, + hashLog: 19, + searchLog: 10, + minMatch: 3, + targetLength: 512, + strategy: ZSTD_strategy.ZSTD_btultra2 + ), + new ZSTD_compressionParameters( + windowLog: 18, + chainLog: 19, + hashLog: 19, + searchLog: 12, + minMatch: 3, + targetLength: 512, + strategy: ZSTD_strategy.ZSTD_btultra2 + ), + new ZSTD_compressionParameters( + windowLog: 18, + chainLog: 19, + hashLog: 19, + searchLog: 13, + minMatch: 3, + targetLength: 999, + strategy: ZSTD_strategy.ZSTD_btultra2 + ), + }, + new ZSTD_compressionParameters[23] + { + new ZSTD_compressionParameters( + windowLog: 17, + chainLog: 12, + hashLog: 12, + searchLog: 1, + minMatch: 5, + targetLength: 1, + strategy: ZSTD_strategy.ZSTD_fast + ), + new ZSTD_compressionParameters( + windowLog: 17, + chainLog: 12, + hashLog: 13, + searchLog: 1, + minMatch: 6, + targetLength: 0, + strategy: ZSTD_strategy.ZSTD_fast + ), + new ZSTD_compressionParameters( + windowLog: 17, + chainLog: 13, + hashLog: 15, + searchLog: 1, + minMatch: 5, + targetLength: 0, + strategy: ZSTD_strategy.ZSTD_fast + ), + new ZSTD_compressionParameters( + windowLog: 17, + chainLog: 15, + hashLog: 16, + searchLog: 2, + minMatch: 5, + targetLength: 0, + strategy: ZSTD_strategy.ZSTD_dfast + ), + new ZSTD_compressionParameters( + windowLog: 17, + chainLog: 17, + hashLog: 17, + searchLog: 2, + minMatch: 4, + targetLength: 0, + strategy: ZSTD_strategy.ZSTD_dfast + ), + new ZSTD_compressionParameters( + windowLog: 17, + chainLog: 16, + hashLog: 17, + searchLog: 3, + minMatch: 4, + targetLength: 2, + strategy: ZSTD_strategy.ZSTD_greedy + ), + new ZSTD_compressionParameters( + windowLog: 17, + chainLog: 16, + hashLog: 17, + searchLog: 3, + minMatch: 4, + targetLength: 4, + strategy: ZSTD_strategy.ZSTD_lazy + ), + new ZSTD_compressionParameters( + windowLog: 17, + chainLog: 16, + hashLog: 17, + searchLog: 3, + minMatch: 4, + targetLength: 8, + strategy: ZSTD_strategy.ZSTD_lazy2 + ), + new ZSTD_compressionParameters( + windowLog: 17, + chainLog: 16, + hashLog: 17, + searchLog: 4, + minMatch: 4, + targetLength: 8, + strategy: ZSTD_strategy.ZSTD_lazy2 + ), + new ZSTD_compressionParameters( + windowLog: 17, + chainLog: 16, + hashLog: 17, + searchLog: 5, + minMatch: 4, + targetLength: 8, + strategy: ZSTD_strategy.ZSTD_lazy2 + ), + new ZSTD_compressionParameters( + windowLog: 17, + chainLog: 16, + hashLog: 17, + searchLog: 6, + minMatch: 4, + targetLength: 8, + strategy: ZSTD_strategy.ZSTD_lazy2 + ), + new ZSTD_compressionParameters( + windowLog: 17, + chainLog: 17, + hashLog: 17, + searchLog: 5, + minMatch: 4, + targetLength: 8, + strategy: ZSTD_strategy.ZSTD_btlazy2 + ), + new ZSTD_compressionParameters( + windowLog: 17, + chainLog: 18, + hashLog: 17, + searchLog: 7, + minMatch: 4, + targetLength: 12, + strategy: ZSTD_strategy.ZSTD_btlazy2 + ), + new ZSTD_compressionParameters( + windowLog: 17, + chainLog: 18, + hashLog: 17, + searchLog: 3, + minMatch: 4, + targetLength: 12, + strategy: ZSTD_strategy.ZSTD_btopt + ), + new ZSTD_compressionParameters( + windowLog: 17, + chainLog: 18, + hashLog: 17, + searchLog: 4, + minMatch: 3, + targetLength: 32, + strategy: ZSTD_strategy.ZSTD_btopt + ), + new ZSTD_compressionParameters( + windowLog: 17, + chainLog: 18, + hashLog: 17, + searchLog: 6, + minMatch: 3, + targetLength: 256, + strategy: ZSTD_strategy.ZSTD_btopt + ), + new ZSTD_compressionParameters( + windowLog: 17, + chainLog: 18, + hashLog: 17, + searchLog: 6, + minMatch: 3, + targetLength: 128, + strategy: ZSTD_strategy.ZSTD_btultra + ), + new ZSTD_compressionParameters( + windowLog: 17, + chainLog: 18, + hashLog: 17, + searchLog: 8, + minMatch: 3, + targetLength: 256, + strategy: ZSTD_strategy.ZSTD_btultra + ), + new ZSTD_compressionParameters( + windowLog: 17, + chainLog: 18, + hashLog: 17, + searchLog: 10, + minMatch: 3, + targetLength: 512, + strategy: ZSTD_strategy.ZSTD_btultra + ), + new ZSTD_compressionParameters( + windowLog: 17, + chainLog: 18, + hashLog: 17, + searchLog: 5, + minMatch: 3, + targetLength: 256, + strategy: ZSTD_strategy.ZSTD_btultra2 + ), + new ZSTD_compressionParameters( + windowLog: 17, + chainLog: 18, + hashLog: 17, + searchLog: 7, + minMatch: 3, + targetLength: 512, + strategy: ZSTD_strategy.ZSTD_btultra2 + ), + new ZSTD_compressionParameters( + windowLog: 17, + chainLog: 18, + hashLog: 17, + searchLog: 9, + minMatch: 3, + targetLength: 512, + strategy: ZSTD_strategy.ZSTD_btultra2 + ), + new ZSTD_compressionParameters( + windowLog: 17, + chainLog: 18, + hashLog: 17, + searchLog: 11, + minMatch: 3, + targetLength: 999, + strategy: ZSTD_strategy.ZSTD_btultra2 + ), + }, + new ZSTD_compressionParameters[23] + { + new ZSTD_compressionParameters( + windowLog: 14, + chainLog: 12, + hashLog: 13, + searchLog: 1, + minMatch: 5, + targetLength: 1, + strategy: ZSTD_strategy.ZSTD_fast + ), + new ZSTD_compressionParameters( + windowLog: 14, + chainLog: 14, + hashLog: 15, + searchLog: 1, + minMatch: 5, + targetLength: 0, + strategy: ZSTD_strategy.ZSTD_fast + ), + new ZSTD_compressionParameters( + windowLog: 14, + chainLog: 14, + hashLog: 15, + searchLog: 1, + minMatch: 4, + targetLength: 0, + strategy: ZSTD_strategy.ZSTD_fast + ), + new ZSTD_compressionParameters( + windowLog: 14, + chainLog: 14, + hashLog: 15, + searchLog: 2, + minMatch: 4, + targetLength: 0, + strategy: ZSTD_strategy.ZSTD_dfast + ), + new ZSTD_compressionParameters( + windowLog: 14, + chainLog: 14, + hashLog: 14, + searchLog: 4, + minMatch: 4, + targetLength: 2, + strategy: ZSTD_strategy.ZSTD_greedy + ), + new ZSTD_compressionParameters( + windowLog: 14, + chainLog: 14, + hashLog: 14, + searchLog: 3, + minMatch: 4, + targetLength: 4, + strategy: ZSTD_strategy.ZSTD_lazy + ), + new ZSTD_compressionParameters( + windowLog: 14, + chainLog: 14, + hashLog: 14, + searchLog: 4, + minMatch: 4, + targetLength: 8, + strategy: ZSTD_strategy.ZSTD_lazy2 + ), + new ZSTD_compressionParameters( + windowLog: 14, + chainLog: 14, + hashLog: 14, + searchLog: 6, + minMatch: 4, + targetLength: 8, + strategy: ZSTD_strategy.ZSTD_lazy2 + ), + new ZSTD_compressionParameters( + windowLog: 14, + chainLog: 14, + hashLog: 14, + searchLog: 8, + minMatch: 4, + targetLength: 8, + strategy: ZSTD_strategy.ZSTD_lazy2 + ), + new ZSTD_compressionParameters( + windowLog: 14, + chainLog: 15, + hashLog: 14, + searchLog: 5, + minMatch: 4, + targetLength: 8, + strategy: ZSTD_strategy.ZSTD_btlazy2 + ), + new ZSTD_compressionParameters( + windowLog: 14, + chainLog: 15, + hashLog: 14, + searchLog: 9, + minMatch: 4, + targetLength: 8, + strategy: ZSTD_strategy.ZSTD_btlazy2 + ), + new ZSTD_compressionParameters( + windowLog: 14, + chainLog: 15, + hashLog: 14, + searchLog: 3, + minMatch: 4, + targetLength: 12, + strategy: ZSTD_strategy.ZSTD_btopt + ), + new ZSTD_compressionParameters( + windowLog: 14, + chainLog: 15, + hashLog: 14, + searchLog: 4, + minMatch: 3, + targetLength: 24, + strategy: ZSTD_strategy.ZSTD_btopt + ), + new ZSTD_compressionParameters( + windowLog: 14, + chainLog: 15, + hashLog: 14, + searchLog: 5, + minMatch: 3, + targetLength: 32, + strategy: ZSTD_strategy.ZSTD_btultra + ), + new ZSTD_compressionParameters( + windowLog: 14, + chainLog: 15, + hashLog: 15, + searchLog: 6, + minMatch: 3, + targetLength: 64, + strategy: ZSTD_strategy.ZSTD_btultra + ), + new ZSTD_compressionParameters( + windowLog: 14, + chainLog: 15, + hashLog: 15, + searchLog: 7, + minMatch: 3, + targetLength: 256, + strategy: ZSTD_strategy.ZSTD_btultra + ), + new ZSTD_compressionParameters( + windowLog: 14, + chainLog: 15, + hashLog: 15, + searchLog: 5, + minMatch: 3, + targetLength: 48, + strategy: ZSTD_strategy.ZSTD_btultra2 + ), + new ZSTD_compressionParameters( + windowLog: 14, + chainLog: 15, + hashLog: 15, + searchLog: 6, + minMatch: 3, + targetLength: 128, + strategy: ZSTD_strategy.ZSTD_btultra2 + ), + new ZSTD_compressionParameters( + windowLog: 14, + chainLog: 15, + hashLog: 15, + searchLog: 7, + minMatch: 3, + targetLength: 256, + strategy: ZSTD_strategy.ZSTD_btultra2 + ), + new ZSTD_compressionParameters( + windowLog: 14, + chainLog: 15, + hashLog: 15, + searchLog: 8, + minMatch: 3, + targetLength: 256, + strategy: ZSTD_strategy.ZSTD_btultra2 + ), + new ZSTD_compressionParameters( + windowLog: 14, + chainLog: 15, + hashLog: 15, + searchLog: 8, + minMatch: 3, + targetLength: 512, + strategy: ZSTD_strategy.ZSTD_btultra2 + ), + new ZSTD_compressionParameters( + windowLog: 14, + chainLog: 15, + hashLog: 15, + searchLog: 9, + minMatch: 3, + targetLength: 512, + strategy: ZSTD_strategy.ZSTD_btultra2 + ), + new ZSTD_compressionParameters( + windowLog: 14, + chainLog: 15, + hashLog: 15, + searchLog: 10, + minMatch: 3, + targetLength: 999, + strategy: ZSTD_strategy.ZSTD_btultra2 + ), + }, + }; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/Compiler.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/Compiler.cs new file mode 100644 index 00000000..97fead59 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/Compiler.cs @@ -0,0 +1,61 @@ +using System.Runtime.CompilerServices; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + /* @return 1 if @u is a 2^n value, 0 otherwise + * useful to check a value is valid for alignment restrictions */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static int ZSTD_isPower2(nuint u) + { + return (u & u - 1) == 0 ? 1 : 0; + } + + /** + * Helper function to perform a wrapped pointer difference without triggering + * UBSAN. + * + * @returns lhs - rhs with wrapping + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nint ZSTD_wrappedPtrDiff(byte* lhs, byte* rhs) + { + return (nint)(lhs - rhs); + } + + /** + * Helper function to perform a wrapped pointer add without triggering UBSAN. + * + * @return ptr + add with wrapping + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static byte* ZSTD_wrappedPtrAdd(byte* ptr, nint add) + { + return ptr + add; + } + + /** + * Helper function to perform a wrapped pointer subtraction without triggering + * UBSAN. + * + * @return ptr - sub with wrapping + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static byte* ZSTD_wrappedPtrSub(byte* ptr, nint sub) + { + return ptr - sub; + } + + /** + * Helper function to add to a pointer that works around C's undefined behavior + * of adding 0 to NULL. + * + * @returns `ptr + add` except it defines `NULL + 0 == NULL`. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static byte* ZSTD_maybeNullPtrAdd(byte* ptr, nint add) + { + return add > 0 ? ptr + add : ptr; + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/Cover.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/Cover.cs new file mode 100644 index 00000000..878fe30d --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/Cover.cs @@ -0,0 +1,447 @@ +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + private static int g_displayLevel = 0; + + /** + * Returns the sum of the sample sizes. + */ + private static nuint COVER_sum(nuint* samplesSizes, uint nbSamples) + { + nuint sum = 0; + uint i; + for (i = 0; i < nbSamples; ++i) + { + sum += samplesSizes[i]; + } + + return sum; + } + + /** + * Warns the user when their corpus is too small. + */ + private static void COVER_warnOnSmallCorpus(nuint maxDictSize, nuint nbDmers, int displayLevel) + { + double ratio = nbDmers / (double)maxDictSize; + if (ratio >= 10) + { + return; + } + } + + /** + * Computes the number of epochs and the size of each epoch. + * We will make sure that each epoch gets at least 10 * k bytes. + * + * The COVER algorithms divide the data up into epochs of equal size and + * select one segment from each epoch. + * + * @param maxDictSize The maximum allowed dictionary size. + * @param nbDmers The number of dmers we are training on. + * @param k The parameter k (segment size). + * @param passes The target number of passes over the dmer corpus. + * More passes means a better dictionary. + */ + private static COVER_epoch_info_t COVER_computeEpochs( + uint maxDictSize, + uint nbDmers, + uint k, + uint passes + ) + { + uint minEpochSize = k * 10; + COVER_epoch_info_t epochs; + epochs.num = 1 > maxDictSize / k / passes ? 1 : maxDictSize / k / passes; + epochs.size = nbDmers / epochs.num; + if (epochs.size >= minEpochSize) + { + assert(epochs.size * epochs.num <= nbDmers); + return epochs; + } + + epochs.size = minEpochSize < nbDmers ? minEpochSize : nbDmers; + epochs.num = nbDmers / epochs.size; + assert(epochs.size * epochs.num <= nbDmers); + return epochs; + } + + /** + * Checks total compressed size of a dictionary + */ + private static nuint COVER_checkTotalCompressedSize( + ZDICT_cover_params_t parameters, + nuint* samplesSizes, + byte* samples, + nuint* offsets, + nuint nbTrainSamples, + nuint nbSamples, + byte* dict, + nuint dictBufferCapacity + ) + { + nuint totalCompressedSize = unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + /* Pointers */ + ZSTD_CCtx_s* cctx; + ZSTD_CDict_s* cdict; + void* dst; + /* Local variables */ + nuint dstCapacity; + nuint i; + { + nuint maxSampleSize = 0; + i = parameters.splitPoint < 1 ? nbTrainSamples : 0; + for (; i < nbSamples; ++i) + { + maxSampleSize = samplesSizes[i] > maxSampleSize ? samplesSizes[i] : maxSampleSize; + } + + dstCapacity = ZSTD_compressBound(maxSampleSize); + dst = malloc(dstCapacity); + } + + cctx = ZSTD_createCCtx(); + cdict = ZSTD_createCDict(dict, dictBufferCapacity, parameters.zParams.compressionLevel); + if (dst == null || cctx == null || cdict == null) + { + goto _compressCleanup; + } + + totalCompressedSize = dictBufferCapacity; + i = parameters.splitPoint < 1 ? nbTrainSamples : 0; + for (; i < nbSamples; ++i) + { + nuint size = ZSTD_compress_usingCDict( + cctx, + dst, + dstCapacity, + samples + offsets[i], + samplesSizes[i], + cdict + ); + if (ERR_isError(size)) + { + totalCompressedSize = size; + goto _compressCleanup; + } + + totalCompressedSize += size; + } + + _compressCleanup: + ZSTD_freeCCtx(cctx); + ZSTD_freeCDict(cdict); + if (dst != null) + { + free(dst); + } + + return totalCompressedSize; + } + + /** + * Initialize the `COVER_best_t`. + */ + private static void COVER_best_init(COVER_best_s* best) + { + if (best == null) + { + return; + } + + SynchronizationWrapper.Init(&best->mutex); + best->liveJobs = 0; + best->dict = null; + best->dictSize = 0; + best->compressedSize = unchecked((nuint)(-1)); + best->parameters = new ZDICT_cover_params_t(); + } + + /** + * Wait until liveJobs == 0. + */ + private static void COVER_best_wait(COVER_best_s* best) + { + if (best == null) + { + return; + } + + SynchronizationWrapper.Enter(&best->mutex); + while (best->liveJobs != 0) + { + SynchronizationWrapper.Wait(&best->mutex); + } + + SynchronizationWrapper.Exit(&best->mutex); + } + + /** + * Call COVER_best_wait() and then destroy the COVER_best_t. + */ + private static void COVER_best_destroy(COVER_best_s* best) + { + if (best == null) + { + return; + } + + COVER_best_wait(best); + if (best->dict != null) + { + free(best->dict); + } + + SynchronizationWrapper.Free(&best->mutex); + } + + /** + * Called when a thread is about to be launched. + * Increments liveJobs. + */ + private static void COVER_best_start(COVER_best_s* best) + { + if (best == null) + { + return; + } + + SynchronizationWrapper.Enter(&best->mutex); + ++best->liveJobs; + SynchronizationWrapper.Exit(&best->mutex); + } + + /** + * Called when a thread finishes executing, both on error or success. + * Decrements liveJobs and signals any waiting threads if liveJobs == 0. + * If this dictionary is the best so far save it and its parameters. + */ + private static void COVER_best_finish( + COVER_best_s* best, + ZDICT_cover_params_t parameters, + COVER_dictSelection selection + ) + { + void* dict = selection.dictContent; + nuint compressedSize = selection.totalCompressedSize; + nuint dictSize = selection.dictSize; + if (best == null) + { + return; + } + + { + nuint liveJobs; + SynchronizationWrapper.Enter(&best->mutex); + --best->liveJobs; + liveJobs = best->liveJobs; + if (compressedSize < best->compressedSize) + { + if (best->dict == null || best->dictSize < dictSize) + { + if (best->dict != null) + { + free(best->dict); + } + + best->dict = malloc(dictSize); + if (best->dict == null) + { + best->compressedSize = unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC) + ); + best->dictSize = 0; + SynchronizationWrapper.Pulse(&best->mutex); + SynchronizationWrapper.Exit(&best->mutex); + return; + } + } + + if (dict != null) + { + memcpy(best->dict, dict, (uint)dictSize); + best->dictSize = dictSize; + best->parameters = parameters; + best->compressedSize = compressedSize; + } + } + + if (liveJobs == 0) + { + SynchronizationWrapper.PulseAll(&best->mutex); + } + + SynchronizationWrapper.Exit(&best->mutex); + } + } + + private static COVER_dictSelection setDictSelection(byte* buf, nuint s, nuint csz) + { + COVER_dictSelection ds; + ds.dictContent = buf; + ds.dictSize = s; + ds.totalCompressedSize = csz; + return ds; + } + + /** + * Error function for COVER_selectDict function. Returns a struct where + * return.totalCompressedSize is a ZSTD error. + */ + private static COVER_dictSelection COVER_dictSelectionError(nuint error) + { + return setDictSelection(null, 0, error); + } + + /** + * Error function for COVER_selectDict function. Checks if the return + * value is an error. + */ + private static uint COVER_dictSelectionIsError(COVER_dictSelection selection) + { + return ERR_isError(selection.totalCompressedSize) || selection.dictContent == null + ? 1U + : 0U; + } + + /** + * Always call after selectDict is called to free up used memory from + * newly created dictionary. + */ + private static void COVER_dictSelectionFree(COVER_dictSelection selection) + { + free(selection.dictContent); + } + + /** + * Called to finalize the dictionary and select one based on whether or not + * the shrink-dict flag was enabled. If enabled the dictionary used is the + * smallest dictionary within a specified regression of the compressed size + * from the largest dictionary. + */ + private static COVER_dictSelection COVER_selectDict( + byte* customDictContent, + nuint dictBufferCapacity, + nuint dictContentSize, + byte* samplesBuffer, + nuint* samplesSizes, + uint nbFinalizeSamples, + nuint nbCheckSamples, + nuint nbSamples, + ZDICT_cover_params_t @params, + nuint* offsets, + nuint totalCompressedSize + ) + { + nuint largestDict = 0; + nuint largestCompressed = 0; + byte* customDictContentEnd = customDictContent + dictContentSize; + byte* largestDictbuffer = (byte*)malloc(dictBufferCapacity); + byte* candidateDictBuffer = (byte*)malloc(dictBufferCapacity); + double regressionTolerance = (double)@params.shrinkDictMaxRegression / 100 + 1; + if (largestDictbuffer == null || candidateDictBuffer == null) + { + free(largestDictbuffer); + free(candidateDictBuffer); + return COVER_dictSelectionError(dictContentSize); + } + + memcpy(largestDictbuffer, customDictContent, (uint)dictContentSize); + dictContentSize = ZDICT_finalizeDictionary( + largestDictbuffer, + dictBufferCapacity, + customDictContent, + dictContentSize, + samplesBuffer, + samplesSizes, + nbFinalizeSamples, + @params.zParams + ); + if (ZDICT_isError(dictContentSize)) + { + free(largestDictbuffer); + free(candidateDictBuffer); + return COVER_dictSelectionError(dictContentSize); + } + + totalCompressedSize = COVER_checkTotalCompressedSize( + @params, + samplesSizes, + samplesBuffer, + offsets, + nbCheckSamples, + nbSamples, + largestDictbuffer, + dictContentSize + ); + if (ERR_isError(totalCompressedSize)) + { + free(largestDictbuffer); + free(candidateDictBuffer); + return COVER_dictSelectionError(totalCompressedSize); + } + + if (@params.shrinkDict == 0) + { + free(candidateDictBuffer); + return setDictSelection(largestDictbuffer, dictContentSize, totalCompressedSize); + } + + largestDict = dictContentSize; + largestCompressed = totalCompressedSize; + dictContentSize = 256; + while (dictContentSize < largestDict) + { + memcpy(candidateDictBuffer, largestDictbuffer, (uint)largestDict); + dictContentSize = ZDICT_finalizeDictionary( + candidateDictBuffer, + dictBufferCapacity, + customDictContentEnd - dictContentSize, + dictContentSize, + samplesBuffer, + samplesSizes, + nbFinalizeSamples, + @params.zParams + ); + if (ZDICT_isError(dictContentSize)) + { + free(largestDictbuffer); + free(candidateDictBuffer); + return COVER_dictSelectionError(dictContentSize); + } + + totalCompressedSize = COVER_checkTotalCompressedSize( + @params, + samplesSizes, + samplesBuffer, + offsets, + nbCheckSamples, + nbSamples, + candidateDictBuffer, + dictContentSize + ); + if (ERR_isError(totalCompressedSize)) + { + free(largestDictbuffer); + free(candidateDictBuffer); + return COVER_dictSelectionError(totalCompressedSize); + } + + if (totalCompressedSize <= largestCompressed * regressionTolerance) + { + free(largestDictbuffer); + return setDictSelection(candidateDictBuffer, dictContentSize, totalCompressedSize); + } + + dictContentSize *= 2; + } + + dictContentSize = largestDict; + totalCompressedSize = largestCompressed; + free(candidateDictBuffer); + return setDictSelection(largestDictbuffer, dictContentSize, totalCompressedSize); + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/DTableDesc.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/DTableDesc.cs new file mode 100644 index 00000000..6586ac46 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/DTableDesc.cs @@ -0,0 +1,12 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/*-***************************/ +/* generic DTableDesc */ +/*-***************************/ +public struct DTableDesc +{ + public byte maxTableLog; + public byte tableType; + public byte tableLog; + public byte reserved; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/EStats_ress_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/EStats_ress_t.cs new file mode 100644 index 00000000..9d3f1ba2 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/EStats_ress_t.cs @@ -0,0 +1,13 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct EStats_ress_t +{ + /* dictionary */ + public ZSTD_CDict_s* dict; + + /* working context */ + public ZSTD_CCtx_s* zc; + + /* must be ZSTD_BLOCKSIZE_MAX allocated */ + public void* workPlace; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/EntropyCommon.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/EntropyCommon.cs new file mode 100644 index 00000000..4ee17396 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/EntropyCommon.cs @@ -0,0 +1,506 @@ +using System.Runtime.CompilerServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + /*=== Version ===*/ + private static uint FSE_versionNumber() + { + return 0 * 100 * 100 + 9 * 100 + 0; + } + + /*=== Error Management ===*/ + private static bool FSE_isError(nuint code) + { + return ERR_isError(code); + } + + private static string FSE_getErrorName(nuint code) + { + return ERR_getErrorName(code); + } + + /* Error Management */ + private static bool HUF_isError(nuint code) + { + return ERR_isError(code); + } + + private static string HUF_getErrorName(nuint code) + { + return ERR_getErrorName(code); + } + + /*-************************************************************** + * FSE NCount encoding-decoding + ****************************************************************/ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint FSE_readNCount_body( + short* normalizedCounter, + uint* maxSVPtr, + uint* tableLogPtr, + void* headerBuffer, + nuint hbSize + ) + { + byte* istart = (byte*)headerBuffer; + byte* iend = istart + hbSize; + byte* ip = istart; + int nbBits; + int remaining; + int threshold; + uint bitStream; + int bitCount; + uint charnum = 0; + uint maxSV1 = *maxSVPtr + 1; + int previous0 = 0; + if (hbSize < 8) + { + sbyte* buffer = stackalloc sbyte[8]; + /* This function only works when hbSize >= 8 */ + memset(buffer, 0, sizeof(sbyte) * 8); + memcpy(buffer, headerBuffer, (uint)hbSize); + { + nuint countSize = FSE_readNCount( + normalizedCounter, + maxSVPtr, + tableLogPtr, + buffer, + sizeof(sbyte) * 8 + ); + if (FSE_isError(countSize)) + { + return countSize; + } + + if (countSize > hbSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + return countSize; + } + } + + assert(hbSize >= 8); + memset(normalizedCounter, 0, (*maxSVPtr + 1) * sizeof(short)); + bitStream = MEM_readLE32(ip); + nbBits = (int)((bitStream & 0xF) + 5); + if (nbBits > 15) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_tableLog_tooLarge)); + } + + bitStream >>= 4; + bitCount = 4; + *tableLogPtr = (uint)nbBits; + remaining = (1 << nbBits) + 1; + threshold = 1 << nbBits; + nbBits++; + for (; ; ) + { + if (previous0 != 0) + { + /* Count the number of repeats. Each time the + * 2-bit repeat code is 0b11 there is another + * repeat. + * Avoid UB by setting the high bit to 1. + */ + int repeats = (int)(ZSTD_countTrailingZeros32(~bitStream | 0x80000000) >> 1); + while (repeats >= 12) + { + charnum += 3 * 12; + if (ip <= iend - 7) + { + ip += 3; + } + else + { + bitCount -= (int)(8 * (iend - 7 - ip)); + bitCount &= 31; + ip = iend - 4; + } + + bitStream = MEM_readLE32(ip) >> bitCount; + repeats = (int)(ZSTD_countTrailingZeros32(~bitStream | 0x80000000) >> 1); + } + + charnum += (uint)(3 * repeats); + bitStream >>= 2 * repeats; + bitCount += 2 * repeats; + assert((bitStream & 3) < 3); + charnum += bitStream & 3; + bitCount += 2; + if (charnum >= maxSV1) + { + break; + } + + if (ip <= iend - 7 || ip + (bitCount >> 3) <= iend - 4) + { + assert(bitCount >> 3 <= 3); + ip += bitCount >> 3; + bitCount &= 7; + } + else + { + bitCount -= (int)(8 * (iend - 4 - ip)); + bitCount &= 31; + ip = iend - 4; + } + + bitStream = MEM_readLE32(ip) >> bitCount; + } + + { + int max = 2 * threshold - 1 - remaining; + int count; + if ((bitStream & (uint)(threshold - 1)) < (uint)max) + { + count = (int)(bitStream & (uint)(threshold - 1)); + bitCount += nbBits - 1; + } + else + { + count = (int)(bitStream & (uint)(2 * threshold - 1)); + if (count >= threshold) + { + count -= max; + } + + bitCount += nbBits; + } + + count--; + if (count >= 0) + { + remaining -= count; + } + else + { + assert(count == -1); + remaining += count; + } + + normalizedCounter[charnum++] = (short)count; + previous0 = count == 0 ? 1 : 0; + assert(threshold > 1); + if (remaining < threshold) + { + if (remaining <= 1) + { + break; + } + + nbBits = (int)(ZSTD_highbit32((uint)remaining) + 1); + threshold = 1 << nbBits - 1; + } + + if (charnum >= maxSV1) + { + break; + } + + if (ip <= iend - 7 || ip + (bitCount >> 3) <= iend - 4) + { + ip += bitCount >> 3; + bitCount &= 7; + } + else + { + bitCount -= (int)(8 * (iend - 4 - ip)); + bitCount &= 31; + ip = iend - 4; + } + + bitStream = MEM_readLE32(ip) >> bitCount; + } + } + + if (remaining != 1) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + if (charnum > maxSV1) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_maxSymbolValue_tooSmall)); + } + + if (bitCount > 32) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + *maxSVPtr = charnum - 1; + ip += bitCount + 7 >> 3; + return (nuint)(ip - istart); + } + + /* Avoids the FORCE_INLINE of the _body() function. */ + private static nuint FSE_readNCount_body_default( + short* normalizedCounter, + uint* maxSVPtr, + uint* tableLogPtr, + void* headerBuffer, + nuint hbSize + ) + { + return FSE_readNCount_body(normalizedCounter, maxSVPtr, tableLogPtr, headerBuffer, hbSize); + } + + /*! FSE_readNCount_bmi2(): + * Same as FSE_readNCount() but pass bmi2=1 when your CPU supports BMI2 and 0 otherwise. + */ + private static nuint FSE_readNCount_bmi2( + short* normalizedCounter, + uint* maxSVPtr, + uint* tableLogPtr, + void* headerBuffer, + nuint hbSize, + int bmi2 + ) + { + return FSE_readNCount_body_default( + normalizedCounter, + maxSVPtr, + tableLogPtr, + headerBuffer, + hbSize + ); + } + + /*! FSE_readNCount(): + Read compactly saved 'normalizedCounter' from 'rBuffer'. + @return : size read from 'rBuffer', + or an errorCode, which can be tested using FSE_isError(). + maxSymbolValuePtr[0] and tableLogPtr[0] will also be updated with their respective values */ + private static nuint FSE_readNCount( + short* normalizedCounter, + uint* maxSVPtr, + uint* tableLogPtr, + void* headerBuffer, + nuint hbSize + ) + { + return FSE_readNCount_bmi2( + normalizedCounter, + maxSVPtr, + tableLogPtr, + headerBuffer, + hbSize, + 0 + ); + } + + /*! HUF_readStats() : + Read compact Huffman tree, saved by HUF_writeCTable(). + `huffWeight` is destination buffer. + `rankStats` is assumed to be a table of at least HUF_TABLELOG_MAX U32. + @return : size read from `src` , or an error Code . + Note : Needed by HUF_readCTable() and HUF_readDTableX?() . + */ + private static nuint HUF_readStats( + byte* huffWeight, + nuint hwSize, + uint* rankStats, + uint* nbSymbolsPtr, + uint* tableLogPtr, + void* src, + nuint srcSize + ) + { + uint* wksp = stackalloc uint[219]; + return HUF_readStats_wksp( + huffWeight, + hwSize, + rankStats, + nbSymbolsPtr, + tableLogPtr, + src, + srcSize, + wksp, + sizeof(uint) * 219, + 0 + ); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint HUF_readStats_body( + byte* huffWeight, + nuint hwSize, + uint* rankStats, + uint* nbSymbolsPtr, + uint* tableLogPtr, + void* src, + nuint srcSize, + void* workSpace, + nuint wkspSize, + int bmi2 + ) + { + uint weightTotal; + byte* ip = (byte*)src; + nuint iSize; + nuint oSize; + if (srcSize == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + iSize = ip[0]; + if (iSize >= 128) + { + oSize = iSize - 127; + iSize = (oSize + 1) / 2; + if (iSize + 1 > srcSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + if (oSize >= hwSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + ip += 1; + { + uint n; + for (n = 0; n < oSize; n += 2) + { + huffWeight[n] = (byte)(ip[n / 2] >> 4); + huffWeight[n + 1] = (byte)(ip[n / 2] & 15); + } + } + } + else + { + if (iSize + 1 > srcSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + oSize = FSE_decompress_wksp_bmi2( + huffWeight, + hwSize - 1, + ip + 1, + iSize, + 6, + workSpace, + wkspSize, + bmi2 + ); + if (FSE_isError(oSize)) + { + return oSize; + } + } + + memset(rankStats, 0, (12 + 1) * sizeof(uint)); + weightTotal = 0; + { + uint n; + for (n = 0; n < oSize; n++) + { + if (huffWeight[n] > 12) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + rankStats[huffWeight[n]]++; + weightTotal += (uint)(1 << huffWeight[n] >> 1); + } + } + + if (weightTotal == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + { + uint tableLog = ZSTD_highbit32(weightTotal) + 1; + if (tableLog > 12) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + *tableLogPtr = tableLog; + { + uint total = (uint)(1 << (int)tableLog); + uint rest = total - weightTotal; + uint verif = (uint)(1 << (int)ZSTD_highbit32(rest)); + uint lastWeight = ZSTD_highbit32(rest) + 1; + if (verif != rest) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + huffWeight[oSize] = (byte)lastWeight; + rankStats[lastWeight]++; + } + } + + if (rankStats[1] < 2 || (rankStats[1] & 1) != 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + *nbSymbolsPtr = (uint)(oSize + 1); + return iSize + 1; + } + + /* Avoids the FORCE_INLINE of the _body() function. */ + private static nuint HUF_readStats_body_default( + byte* huffWeight, + nuint hwSize, + uint* rankStats, + uint* nbSymbolsPtr, + uint* tableLogPtr, + void* src, + nuint srcSize, + void* workSpace, + nuint wkspSize + ) + { + return HUF_readStats_body( + huffWeight, + hwSize, + rankStats, + nbSymbolsPtr, + tableLogPtr, + src, + srcSize, + workSpace, + wkspSize, + 0 + ); + } + + private static nuint HUF_readStats_wksp( + byte* huffWeight, + nuint hwSize, + uint* rankStats, + uint* nbSymbolsPtr, + uint* tableLogPtr, + void* src, + nuint srcSize, + void* workSpace, + nuint wkspSize, + int flags + ) + { + return HUF_readStats_body_default( + huffWeight, + hwSize, + rankStats, + nbSymbolsPtr, + tableLogPtr, + src, + srcSize, + workSpace, + wkspSize + ); + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ErrorPrivate.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ErrorPrivate.cs new file mode 100644 index 00000000..72b5f234 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ErrorPrivate.cs @@ -0,0 +1,113 @@ +using System.Runtime.CompilerServices; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static bool ERR_isError(nuint code) + { + return code > unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_maxCode)); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ZSTD_ErrorCode ERR_getErrorCode(nuint code) + { + if (!ERR_isError(code)) + { + return 0; + } + + return (ZSTD_ErrorCode)(0 - code); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static string ERR_getErrorName(nuint code) + { + return ERR_getErrorString(ERR_getErrorCode(code)); + } + + /*-**************************************** + * Error Strings + ******************************************/ + private static string ERR_getErrorString(ZSTD_ErrorCode code) + { + const string notErrorCode = "Unspecified error code"; + switch (code) + { + case ZSTD_ErrorCode.ZSTD_error_no_error: + return "No error detected"; + case ZSTD_ErrorCode.ZSTD_error_GENERIC: + return "Error (generic)"; + case ZSTD_ErrorCode.ZSTD_error_prefix_unknown: + return "Unknown frame descriptor"; + case ZSTD_ErrorCode.ZSTD_error_version_unsupported: + return "Version not supported"; + case ZSTD_ErrorCode.ZSTD_error_frameParameter_unsupported: + return "Unsupported frame parameter"; + case ZSTD_ErrorCode.ZSTD_error_frameParameter_windowTooLarge: + return "Frame requires too much memory for decoding"; + case ZSTD_ErrorCode.ZSTD_error_corruption_detected: + return "Data corruption detected"; + case ZSTD_ErrorCode.ZSTD_error_checksum_wrong: + return "Restored data doesn't match checksum"; + case ZSTD_ErrorCode.ZSTD_error_literals_headerWrong: + return "Header of Literals' block doesn't respect format specification"; + case ZSTD_ErrorCode.ZSTD_error_parameter_unsupported: + return "Unsupported parameter"; + case ZSTD_ErrorCode.ZSTD_error_parameter_combination_unsupported: + return "Unsupported combination of parameters"; + case ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound: + return "Parameter is out of bound"; + case ZSTD_ErrorCode.ZSTD_error_init_missing: + return "Context should be init first"; + case ZSTD_ErrorCode.ZSTD_error_memory_allocation: + return "Allocation error : not enough memory"; + case ZSTD_ErrorCode.ZSTD_error_workSpace_tooSmall: + return "workSpace buffer is not large enough"; + case ZSTD_ErrorCode.ZSTD_error_stage_wrong: + return "Operation not authorized at current processing stage"; + case ZSTD_ErrorCode.ZSTD_error_tableLog_tooLarge: + return "tableLog requires too much memory : unsupported"; + case ZSTD_ErrorCode.ZSTD_error_maxSymbolValue_tooLarge: + return "Unsupported max Symbol Value : too large"; + case ZSTD_ErrorCode.ZSTD_error_maxSymbolValue_tooSmall: + return "Specified maxSymbolValue is too small"; + case ZSTD_ErrorCode.ZSTD_error_cannotProduce_uncompressedBlock: + return "This mode cannot generate an uncompressed block"; + case ZSTD_ErrorCode.ZSTD_error_stabilityCondition_notRespected: + return "pledged buffer stability condition is not respected"; + case ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted: + return "Dictionary is corrupted"; + case ZSTD_ErrorCode.ZSTD_error_dictionary_wrong: + return "Dictionary mismatch"; + case ZSTD_ErrorCode.ZSTD_error_dictionaryCreation_failed: + return "Cannot create Dictionary from provided samples"; + case ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall: + return "Destination buffer is too small"; + case ZSTD_ErrorCode.ZSTD_error_srcSize_wrong: + return "Src size is incorrect"; + case ZSTD_ErrorCode.ZSTD_error_dstBuffer_null: + return "Operation on NULL destination buffer"; + case ZSTD_ErrorCode.ZSTD_error_noForwardProgress_destFull: + return "Operation made no progress over multiple calls, due to output buffer being full"; + case ZSTD_ErrorCode.ZSTD_error_noForwardProgress_inputEmpty: + return "Operation made no progress over multiple calls, due to input being empty"; + case ZSTD_ErrorCode.ZSTD_error_frameIndex_tooLarge: + return "Frame index is too large"; + case ZSTD_ErrorCode.ZSTD_error_seekableIO: + return "An I/O error occurred when reading/seeking"; + case ZSTD_ErrorCode.ZSTD_error_dstBuffer_wrong: + return "Destination buffer is wrong"; + case ZSTD_ErrorCode.ZSTD_error_srcBuffer_wrong: + return "Source buffer is wrong"; + case ZSTD_ErrorCode.ZSTD_error_sequenceProducer_failed: + return "Block-level external sequence producer returned an error code"; + case ZSTD_ErrorCode.ZSTD_error_externalSequences_invalid: + return "External sequences are not valid"; + case ZSTD_ErrorCode.ZSTD_error_maxCode: + default: + return notErrorCode; + } + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/EstimatedBlockSize.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/EstimatedBlockSize.cs new file mode 100644 index 00000000..a2f59adc --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/EstimatedBlockSize.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct EstimatedBlockSize +{ + public nuint estLitSize; + public nuint estBlockSize; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/FASTCOVER_accel_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/FASTCOVER_accel_t.cs new file mode 100644 index 00000000..c5d239fe --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/FASTCOVER_accel_t.cs @@ -0,0 +1,19 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/*-************************************* + * Acceleration + ***************************************/ +public struct FASTCOVER_accel_t +{ + /* Percentage of training samples used for ZDICT_finalizeDictionary */ + public uint finalize; + + /* Number of dmer skipped between each dmer counted in computeFrequency */ + public uint skip; + + public FASTCOVER_accel_t(uint finalize, uint skip) + { + this.finalize = finalize; + this.skip = skip; + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/FASTCOVER_ctx_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/FASTCOVER_ctx_t.cs new file mode 100644 index 00000000..8c79a30d --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/FASTCOVER_ctx_t.cs @@ -0,0 +1,19 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/*-************************************* + * Context + ***************************************/ +public unsafe struct FASTCOVER_ctx_t +{ + public byte* samples; + public nuint* offsets; + public nuint* samplesSizes; + public nuint nbSamples; + public nuint nbTrainSamples; + public nuint nbTestSamples; + public nuint nbDmers; + public uint* freqs; + public uint d; + public uint f; + public FASTCOVER_accel_t accelParams; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/FASTCOVER_tryParameters_data_s.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/FASTCOVER_tryParameters_data_s.cs new file mode 100644 index 00000000..3a6e14f0 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/FASTCOVER_tryParameters_data_s.cs @@ -0,0 +1,12 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/** + * Parameters for FASTCOVER_tryParameters(). + */ +public unsafe struct FASTCOVER_tryParameters_data_s +{ + public FASTCOVER_ctx_t* ctx; + public COVER_best_s* best; + public nuint dictBufferCapacity; + public ZDICT_cover_params_t parameters; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/FPStats.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/FPStats.cs new file mode 100644 index 00000000..7a97d8be --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/FPStats.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct FPStats +{ + public Fingerprint pastEvents; + public Fingerprint newEvents; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/FSE_CState_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/FSE_CState_t.cs new file mode 100644 index 00000000..dfdc049b --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/FSE_CState_t.cs @@ -0,0 +1,16 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/* ***************************************** + * FSE symbol compression API + *******************************************/ +/*! +This API consists of small unitary functions, which highly benefit from being inlined. +Hence their body are included in next section. + */ +public unsafe struct FSE_CState_t +{ + public nint value; + public void* stateTable; + public void* symbolTT; + public uint stateLog; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/FSE_DState_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/FSE_DState_t.cs new file mode 100644 index 00000000..61e0fb68 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/FSE_DState_t.cs @@ -0,0 +1,12 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/* ***************************************** + * FSE symbol decompression API + *******************************************/ +public unsafe struct FSE_DState_t +{ + public nuint state; + + /* precise table may vary, depending on U16 */ + public void* table; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/FSE_DTableHeader.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/FSE_DTableHeader.cs new file mode 100644 index 00000000..db091f0c --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/FSE_DTableHeader.cs @@ -0,0 +1,8 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/* ====== Decompression ====== */ +public struct FSE_DTableHeader +{ + public ushort tableLog; + public ushort fastMode; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/FSE_DecompressWksp.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/FSE_DecompressWksp.cs new file mode 100644 index 00000000..cd03c36f --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/FSE_DecompressWksp.cs @@ -0,0 +1,6 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct FSE_DecompressWksp +{ + public fixed short ncount[256]; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/FSE_decode_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/FSE_decode_t.cs new file mode 100644 index 00000000..66c49635 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/FSE_decode_t.cs @@ -0,0 +1,8 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct FSE_decode_t +{ + public ushort newState; + public byte symbol; + public byte nbBits; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/FSE_repeat.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/FSE_repeat.cs new file mode 100644 index 00000000..6f2ceb53 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/FSE_repeat.cs @@ -0,0 +1,13 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum FSE_repeat +{ + /**< Cannot use the previous table */ + FSE_repeat_none, + + /**< Can use the previous table but it must be checked */ + FSE_repeat_check, + + /**< Can use the previous table and it is assumed to be valid */ + FSE_repeat_valid, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/FSE_symbolCompressionTransform.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/FSE_symbolCompressionTransform.cs new file mode 100644 index 00000000..c4b43033 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/FSE_symbolCompressionTransform.cs @@ -0,0 +1,10 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/* ***************************************** + * Implementation of inlined functions + *******************************************/ +public struct FSE_symbolCompressionTransform +{ + public int deltaFindState; + public uint deltaNbBits; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/Fastcover.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/Fastcover.cs new file mode 100644 index 00000000..2aeccaab --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/Fastcover.cs @@ -0,0 +1,764 @@ +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + /*-************************************* + * Hash Functions + ***************************************/ + /** + * Hash the d-byte value pointed to by p and mod 2^f into the frequency vector + */ + private static nuint FASTCOVER_hashPtrToIndex(void* p, uint f, uint d) + { + if (d == 6) + { + return ZSTD_hash6Ptr(p, f); + } + + return ZSTD_hash8Ptr(p, f); + } + + private static readonly FASTCOVER_accel_t* FASTCOVER_defaultAccelParameters = GetArrayPointer( + new FASTCOVER_accel_t[11] + { + new FASTCOVER_accel_t(finalize: 100, skip: 0), + new FASTCOVER_accel_t(finalize: 100, skip: 0), + new FASTCOVER_accel_t(finalize: 50, skip: 1), + new FASTCOVER_accel_t(finalize: 34, skip: 2), + new FASTCOVER_accel_t(finalize: 25, skip: 3), + new FASTCOVER_accel_t(finalize: 20, skip: 4), + new FASTCOVER_accel_t(finalize: 17, skip: 5), + new FASTCOVER_accel_t(finalize: 14, skip: 6), + new FASTCOVER_accel_t(finalize: 13, skip: 7), + new FASTCOVER_accel_t(finalize: 11, skip: 8), + new FASTCOVER_accel_t(finalize: 10, skip: 9), + } + ); + + /*-************************************* + * Helper functions + ***************************************/ + /** + * Selects the best segment in an epoch. + * Segments of are scored according to the function: + * + * Let F(d) be the frequency of all dmers with hash value d. + * Let S_i be hash value of the dmer at position i of segment S which has length k. + * + * Score(S) = F(S_1) + F(S_2) + ... + F(S_{k-d+1}) + * + * Once the dmer with hash value d is in the dictionary we set F(d) = 0. + */ + private static COVER_segment_t FASTCOVER_selectSegment( + FASTCOVER_ctx_t* ctx, + uint* freqs, + uint begin, + uint end, + ZDICT_cover_params_t parameters, + ushort* segmentFreqs + ) + { + /* Constants */ + uint k = parameters.k; + uint d = parameters.d; + uint f = ctx->f; + uint dmersInK = k - d + 1; + /* Try each segment (activeSegment) and save the best (bestSegment) */ + COVER_segment_t bestSegment = new COVER_segment_t + { + begin = 0, + end = 0, + score = 0, + }; + COVER_segment_t activeSegment; + activeSegment.begin = begin; + activeSegment.end = begin; + activeSegment.score = 0; + while (activeSegment.end < end) + { + /* Get hash value of current dmer */ + nuint idx = FASTCOVER_hashPtrToIndex(ctx->samples + activeSegment.end, f, d); + if (segmentFreqs[idx] == 0) + { + activeSegment.score += freqs[idx]; + } + + activeSegment.end += 1; + segmentFreqs[idx] += 1; + if (activeSegment.end - activeSegment.begin == dmersInK + 1) + { + /* Get hash value of the dmer to be eliminated from active segment */ + nuint delIndex = FASTCOVER_hashPtrToIndex(ctx->samples + activeSegment.begin, f, d); + segmentFreqs[delIndex] -= 1; + if (segmentFreqs[delIndex] == 0) + { + activeSegment.score -= freqs[delIndex]; + } + + activeSegment.begin += 1; + } + + if (activeSegment.score > bestSegment.score) + { + bestSegment = activeSegment; + } + } + + while (activeSegment.begin < end) + { + nuint delIndex = FASTCOVER_hashPtrToIndex(ctx->samples + activeSegment.begin, f, d); + segmentFreqs[delIndex] -= 1; + activeSegment.begin += 1; + } + + { + /* Zero the frequency of hash value of each dmer covered by the chosen segment. */ + uint pos; + for (pos = bestSegment.begin; pos != bestSegment.end; ++pos) + { + nuint i = FASTCOVER_hashPtrToIndex(ctx->samples + pos, f, d); + freqs[i] = 0; + } + } + + return bestSegment; + } + + private static int FASTCOVER_checkParameters( + ZDICT_cover_params_t parameters, + nuint maxDictSize, + uint f, + uint accel + ) + { + if (parameters.d == 0 || parameters.k == 0) + { + return 0; + } + + if (parameters.d != 6 && parameters.d != 8) + { + return 0; + } + + if (parameters.k > maxDictSize) + { + return 0; + } + + if (parameters.d > parameters.k) + { + return 0; + } + + if (f > 31 || f == 0) + { + return 0; + } + + if (parameters.splitPoint <= 0 || parameters.splitPoint > 1) + { + return 0; + } + + if (accel > 10 || accel == 0) + { + return 0; + } + + return 1; + } + + /** + * Clean up a context initialized with `FASTCOVER_ctx_init()`. + */ + private static void FASTCOVER_ctx_destroy(FASTCOVER_ctx_t* ctx) + { + if (ctx == null) + { + return; + } + + free(ctx->freqs); + ctx->freqs = null; + free(ctx->offsets); + ctx->offsets = null; + } + + /** + * Calculate for frequency of hash value of each dmer in ctx->samples + */ + private static void FASTCOVER_computeFrequency(uint* freqs, FASTCOVER_ctx_t* ctx) + { + uint f = ctx->f; + uint d = ctx->d; + uint skip = ctx->accelParams.skip; + uint readLength = d > 8 ? d : 8; + nuint i; + assert(ctx->nbTrainSamples >= 5); + assert(ctx->nbTrainSamples <= ctx->nbSamples); + for (i = 0; i < ctx->nbTrainSamples; i++) + { + /* start of current dmer */ + nuint start = ctx->offsets[i]; + nuint currSampleEnd = ctx->offsets[i + 1]; + while (start + readLength <= currSampleEnd) + { + nuint dmerIndex = FASTCOVER_hashPtrToIndex(ctx->samples + start, f, d); + freqs[dmerIndex]++; + start = start + skip + 1; + } + } + } + + /** + * Prepare a context for dictionary building. + * The context is only dependent on the parameter `d` and can be used multiple + * times. + * Returns 0 on success or error code on error. + * The context must be destroyed with `FASTCOVER_ctx_destroy()`. + */ + private static nuint FASTCOVER_ctx_init( + FASTCOVER_ctx_t* ctx, + void* samplesBuffer, + nuint* samplesSizes, + uint nbSamples, + uint d, + double splitPoint, + uint f, + FASTCOVER_accel_t accelParams + ) + { + byte* samples = (byte*)samplesBuffer; + nuint totalSamplesSize = COVER_sum(samplesSizes, nbSamples); + /* Split samples into testing and training sets */ + uint nbTrainSamples = splitPoint < 1 ? (uint)(nbSamples * splitPoint) : nbSamples; + uint nbTestSamples = splitPoint < 1 ? nbSamples - nbTrainSamples : nbSamples; + nuint trainingSamplesSize = + splitPoint < 1 ? COVER_sum(samplesSizes, nbTrainSamples) : totalSamplesSize; + nuint testSamplesSize = + splitPoint < 1 + ? COVER_sum(samplesSizes + nbTrainSamples, nbTestSamples) + : totalSamplesSize; + if ( + totalSamplesSize < (d > sizeof(ulong) ? d : sizeof(ulong)) + || totalSamplesSize >= (sizeof(nuint) == 8 ? unchecked((uint)-1) : 1 * (1U << 30)) + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + if (nbTrainSamples < 5) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + if (nbTestSamples < 1) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + *ctx = new FASTCOVER_ctx_t + { + samples = samples, + samplesSizes = samplesSizes, + nbSamples = nbSamples, + nbTrainSamples = nbTrainSamples, + nbTestSamples = nbTestSamples, + nbDmers = trainingSamplesSize - (d > sizeof(ulong) ? d : sizeof(ulong)) + 1, + d = d, + f = f, + accelParams = accelParams, + offsets = (nuint*)calloc(nbSamples + 1, (ulong)sizeof(nuint)), + }; + if (ctx->offsets == null) + { + FASTCOVER_ctx_destroy(ctx); + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + + { + uint i; + ctx->offsets[0] = 0; + assert(nbSamples >= 5); + for (i = 1; i <= nbSamples; ++i) + { + ctx->offsets[i] = ctx->offsets[i - 1] + samplesSizes[i - 1]; + } + } + + ctx->freqs = (uint*)calloc((ulong)1 << (int)f, sizeof(uint)); + if (ctx->freqs == null) + { + FASTCOVER_ctx_destroy(ctx); + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + + FASTCOVER_computeFrequency(ctx->freqs, ctx); + return 0; + } + + /** + * Given the prepared context build the dictionary. + */ + private static nuint FASTCOVER_buildDictionary( + FASTCOVER_ctx_t* ctx, + uint* freqs, + void* dictBuffer, + nuint dictBufferCapacity, + ZDICT_cover_params_t parameters, + ushort* segmentFreqs + ) + { + byte* dict = (byte*)dictBuffer; + nuint tail = dictBufferCapacity; + /* Divide the data into epochs. We will select one segment from each epoch. */ + COVER_epoch_info_t epochs = COVER_computeEpochs( + (uint)dictBufferCapacity, + (uint)ctx->nbDmers, + parameters.k, + 1 + ); + const nuint maxZeroScoreRun = 10; + nuint zeroScoreRun = 0; + nuint epoch; + for (epoch = 0; tail > 0; epoch = (epoch + 1) % epochs.num) + { + uint epochBegin = (uint)(epoch * epochs.size); + uint epochEnd = epochBegin + epochs.size; + nuint segmentSize; + /* Select a segment */ + COVER_segment_t segment = FASTCOVER_selectSegment( + ctx, + freqs, + epochBegin, + epochEnd, + parameters, + segmentFreqs + ); + if (segment.score == 0) + { + if (++zeroScoreRun >= maxZeroScoreRun) + { + break; + } + + continue; + } + + zeroScoreRun = 0; + segmentSize = + segment.end - segment.begin + parameters.d - 1 < tail + ? segment.end - segment.begin + parameters.d - 1 + : tail; + if (segmentSize < parameters.d) + { + break; + } + + tail -= segmentSize; + memcpy(dict + tail, ctx->samples + segment.begin, (uint)segmentSize); + } + + return tail; + } + + /** + * Tries a set of parameters and updates the COVER_best_t with the results. + * This function is thread safe if zstd is compiled with multithreaded support. + * It takes its parameters as an *OWNING* opaque pointer to support threading. + */ + private static void FASTCOVER_tryParameters(void* opaque) + { + /* Save parameters as local variables */ + FASTCOVER_tryParameters_data_s* data = (FASTCOVER_tryParameters_data_s*)opaque; + FASTCOVER_ctx_t* ctx = data->ctx; + ZDICT_cover_params_t parameters = data->parameters; + nuint dictBufferCapacity = data->dictBufferCapacity; + nuint totalCompressedSize = unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + /* Initialize array to keep track of frequency of dmer within activeSegment */ + ushort* segmentFreqs = (ushort*)calloc((ulong)1 << (int)ctx->f, sizeof(ushort)); + /* Allocate space for hash table, dict, and freqs */ + byte* dict = (byte*)malloc(dictBufferCapacity); + COVER_dictSelection selection = COVER_dictSelectionError( + unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)) + ); + uint* freqs = (uint*)malloc(((ulong)1 << (int)ctx->f) * sizeof(uint)); + if (segmentFreqs == null || dict == null || freqs == null) + { + goto _cleanup; + } + + memcpy(freqs, ctx->freqs, (uint)(((ulong)1 << (int)ctx->f) * sizeof(uint))); + { + nuint tail = FASTCOVER_buildDictionary( + ctx, + freqs, + dict, + dictBufferCapacity, + parameters, + segmentFreqs + ); + uint nbFinalizeSamples = (uint)(ctx->nbTrainSamples * ctx->accelParams.finalize / 100); + selection = COVER_selectDict( + dict + tail, + dictBufferCapacity, + dictBufferCapacity - tail, + ctx->samples, + ctx->samplesSizes, + nbFinalizeSamples, + ctx->nbTrainSamples, + ctx->nbSamples, + parameters, + ctx->offsets, + totalCompressedSize + ); + if (COVER_dictSelectionIsError(selection) != 0) + { + goto _cleanup; + } + } + + _cleanup: + free(dict); + COVER_best_finish(data->best, parameters, selection); + free(data); + free(segmentFreqs); + COVER_dictSelectionFree(selection); + free(freqs); + } + + private static void FASTCOVER_convertToCoverParams( + ZDICT_fastCover_params_t fastCoverParams, + ZDICT_cover_params_t* coverParams + ) + { + coverParams->k = fastCoverParams.k; + coverParams->d = fastCoverParams.d; + coverParams->steps = fastCoverParams.steps; + coverParams->nbThreads = fastCoverParams.nbThreads; + coverParams->splitPoint = fastCoverParams.splitPoint; + coverParams->zParams = fastCoverParams.zParams; + coverParams->shrinkDict = fastCoverParams.shrinkDict; + } + + private static void FASTCOVER_convertToFastCoverParams( + ZDICT_cover_params_t coverParams, + ZDICT_fastCover_params_t* fastCoverParams, + uint f, + uint accel + ) + { + fastCoverParams->k = coverParams.k; + fastCoverParams->d = coverParams.d; + fastCoverParams->steps = coverParams.steps; + fastCoverParams->nbThreads = coverParams.nbThreads; + fastCoverParams->splitPoint = coverParams.splitPoint; + fastCoverParams->f = f; + fastCoverParams->accel = accel; + fastCoverParams->zParams = coverParams.zParams; + fastCoverParams->shrinkDict = coverParams.shrinkDict; + } + + /*! ZDICT_trainFromBuffer_fastCover(): + * Train a dictionary from an array of samples using a modified version of COVER algorithm. + * Samples must be stored concatenated in a single flat buffer `samplesBuffer`, + * supplied with an array of sizes `samplesSizes`, providing the size of each sample, in order. + * d and k are required. + * All other parameters are optional, will use default values if not provided + * The resulting dictionary will be saved into `dictBuffer`. + * @return: size of dictionary stored into `dictBuffer` (<= `dictBufferCapacity`) + * or an error code, which can be tested with ZDICT_isError(). + * See ZDICT_trainFromBuffer() for details on failure modes. + * Note: ZDICT_trainFromBuffer_fastCover() requires 6 * 2^f bytes of memory. + * Tips: In general, a reasonable dictionary has a size of ~ 100 KB. + * It's possible to select smaller or larger size, just by specifying `dictBufferCapacity`. + * In general, it's recommended to provide a few thousands samples, though this can vary a lot. + * It's recommended that total size of all samples be about ~x100 times the target size of dictionary. + */ + public static nuint ZDICT_trainFromBuffer_fastCover( + void* dictBuffer, + nuint dictBufferCapacity, + void* samplesBuffer, + nuint* samplesSizes, + uint nbSamples, + ZDICT_fastCover_params_t parameters + ) + { + byte* dict = (byte*)dictBuffer; + FASTCOVER_ctx_t ctx; + ZDICT_cover_params_t coverParams; + FASTCOVER_accel_t accelParams; + g_displayLevel = (int)parameters.zParams.notificationLevel; + parameters.splitPoint = 1; + parameters.f = parameters.f == 0 ? 20 : parameters.f; + parameters.accel = parameters.accel == 0 ? 1 : parameters.accel; + coverParams = new ZDICT_cover_params_t(); + FASTCOVER_convertToCoverParams(parameters, &coverParams); + if ( + FASTCOVER_checkParameters( + coverParams, + dictBufferCapacity, + parameters.f, + parameters.accel + ) == 0 + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + if (nbSamples == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + if (dictBufferCapacity < 256) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + accelParams = FASTCOVER_defaultAccelParameters[parameters.accel]; + { + nuint initVal = FASTCOVER_ctx_init( + &ctx, + samplesBuffer, + samplesSizes, + nbSamples, + coverParams.d, + parameters.splitPoint, + parameters.f, + accelParams + ); + if (ERR_isError(initVal)) + { + return initVal; + } + } + + COVER_warnOnSmallCorpus(dictBufferCapacity, ctx.nbDmers, g_displayLevel); + { + /* Initialize array to keep track of frequency of dmer within activeSegment */ + ushort* segmentFreqs = (ushort*)calloc((ulong)1 << (int)parameters.f, sizeof(ushort)); + nuint tail = FASTCOVER_buildDictionary( + &ctx, + ctx.freqs, + dictBuffer, + dictBufferCapacity, + coverParams, + segmentFreqs + ); + uint nbFinalizeSamples = (uint)(ctx.nbTrainSamples * ctx.accelParams.finalize / 100); + nuint dictionarySize = ZDICT_finalizeDictionary( + dict, + dictBufferCapacity, + dict + tail, + dictBufferCapacity - tail, + samplesBuffer, + samplesSizes, + nbFinalizeSamples, + coverParams.zParams + ); + if (!ERR_isError(dictionarySize)) { } + + FASTCOVER_ctx_destroy(&ctx); + free(segmentFreqs); + return dictionarySize; + } + } + + /*! ZDICT_optimizeTrainFromBuffer_fastCover(): + * The same requirements as above hold for all the parameters except `parameters`. + * This function tries many parameter combinations (specifically, k and d combinations) + * and picks the best parameters. `*parameters` is filled with the best parameters found, + * dictionary constructed with those parameters is stored in `dictBuffer`. + * All of the parameters d, k, steps, f, and accel are optional. + * If d is non-zero then we don't check multiple values of d, otherwise we check d = {6, 8}. + * if steps is zero it defaults to its default value. + * If k is non-zero then we don't check multiple values of k, otherwise we check steps values in [50, 2000]. + * If f is zero, default value of 20 is used. + * If accel is zero, default value of 1 is used. + * + * @return: size of dictionary stored into `dictBuffer` (<= `dictBufferCapacity`) + * or an error code, which can be tested with ZDICT_isError(). + * On success `*parameters` contains the parameters selected. + * See ZDICT_trainFromBuffer() for details on failure modes. + * Note: ZDICT_optimizeTrainFromBuffer_fastCover() requires about 6 * 2^f bytes of memory for each thread. + */ + public static nuint ZDICT_optimizeTrainFromBuffer_fastCover( + void* dictBuffer, + nuint dictBufferCapacity, + void* samplesBuffer, + nuint* samplesSizes, + uint nbSamples, + ZDICT_fastCover_params_t* parameters + ) + { + ZDICT_cover_params_t coverParams; + FASTCOVER_accel_t accelParams; + /* constants */ + uint nbThreads = parameters->nbThreads; + double splitPoint = parameters->splitPoint <= 0 ? 0.75 : parameters->splitPoint; + uint kMinD = parameters->d == 0 ? 6 : parameters->d; + uint kMaxD = parameters->d == 0 ? 8 : parameters->d; + uint kMinK = parameters->k == 0 ? 50 : parameters->k; + uint kMaxK = parameters->k == 0 ? 2000 : parameters->k; + uint kSteps = parameters->steps == 0 ? 40 : parameters->steps; + uint kStepSize = (kMaxK - kMinK) / kSteps > 1 ? (kMaxK - kMinK) / kSteps : 1; + uint kIterations = (1 + (kMaxD - kMinD) / 2) * (1 + (kMaxK - kMinK) / kStepSize); + uint f = parameters->f == 0 ? 20 : parameters->f; + uint accel = parameters->accel == 0 ? 1 : parameters->accel; + const uint shrinkDict = 0; + /* Local variables */ + int displayLevel = (int)parameters->zParams.notificationLevel; + uint iteration = 1; + uint d; + uint k; + COVER_best_s best; + void* pool = null; + int warned = 0; + if (splitPoint <= 0 || splitPoint > 1) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + if (accel == 0 || accel > 10) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + if (kMinK < kMaxD || kMaxK < kMinK) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + if (nbSamples == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + if (dictBufferCapacity < 256) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + if (nbThreads > 1) + { + pool = POOL_create(nbThreads, 1); + if (pool == null) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + } + + COVER_best_init(&best); + coverParams = new ZDICT_cover_params_t(); + FASTCOVER_convertToCoverParams(*parameters, &coverParams); + accelParams = FASTCOVER_defaultAccelParameters[accel]; + g_displayLevel = displayLevel == 0 ? 0 : displayLevel - 1; + for (d = kMinD; d <= kMaxD; d += 2) + { + /* Initialize the context for this value of d */ + FASTCOVER_ctx_t ctx; + { + nuint initVal = FASTCOVER_ctx_init( + &ctx, + samplesBuffer, + samplesSizes, + nbSamples, + d, + splitPoint, + f, + accelParams + ); + if (ERR_isError(initVal)) + { + COVER_best_destroy(&best); + POOL_free(pool); + return initVal; + } + } + + if (warned == 0) + { + COVER_warnOnSmallCorpus(dictBufferCapacity, ctx.nbDmers, displayLevel); + warned = 1; + } + + for (k = kMinK; k <= kMaxK; k += kStepSize) + { + /* Prepare the arguments */ + FASTCOVER_tryParameters_data_s* data = (FASTCOVER_tryParameters_data_s*)malloc( + (ulong)sizeof(FASTCOVER_tryParameters_data_s) + ); + if (data == null) + { + COVER_best_destroy(&best); + FASTCOVER_ctx_destroy(&ctx); + POOL_free(pool); + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + + data->ctx = &ctx; + data->best = &best; + data->dictBufferCapacity = dictBufferCapacity; + data->parameters = coverParams; + data->parameters.k = k; + data->parameters.d = d; + data->parameters.splitPoint = splitPoint; + data->parameters.steps = kSteps; + data->parameters.shrinkDict = shrinkDict; + data->parameters.zParams.notificationLevel = (uint)g_displayLevel; + if ( + FASTCOVER_checkParameters( + data->parameters, + dictBufferCapacity, + data->ctx->f, + accel + ) == 0 + ) + { + free(data); + continue; + } + + COVER_best_start(&best); + if (pool != null) + { + POOL_add( + pool, + (delegate* managed)(&FASTCOVER_tryParameters), + data + ); + } + else + { + FASTCOVER_tryParameters(data); + } + + ++iteration; + } + + COVER_best_wait(&best); + FASTCOVER_ctx_destroy(&ctx); + } + + { + nuint dictSize = best.dictSize; + if (ERR_isError(best.compressedSize)) + { + nuint compressedSize = best.compressedSize; + COVER_best_destroy(&best); + POOL_free(pool); + return compressedSize; + } + + FASTCOVER_convertToFastCoverParams(best.parameters, parameters, f, accel); + memcpy(dictBuffer, best.dict, (uint)dictSize); + COVER_best_destroy(&best); + POOL_free(pool); + return dictSize; + } + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/Fingerprint.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/Fingerprint.cs new file mode 100644 index 00000000..c4877235 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/Fingerprint.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct Fingerprint +{ + public fixed uint events[1024]; + public nuint nbEvents; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/Fse.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/Fse.cs new file mode 100644 index 00000000..32798dfe --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/Fse.cs @@ -0,0 +1,198 @@ +using System.Runtime.CompilerServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void FSE_initCState(FSE_CState_t* statePtr, uint* ct) + { + void* ptr = ct; + ushort* u16ptr = (ushort*)ptr; + uint tableLog = MEM_read16(ptr); + statePtr->value = (nint)1 << (int)tableLog; + statePtr->stateTable = u16ptr + 2; + statePtr->symbolTT = ct + 1 + (tableLog != 0 ? 1 << (int)(tableLog - 1) : 1); + statePtr->stateLog = tableLog; + } + + /*! FSE_initCState2() : + * Same as FSE_initCState(), but the first symbol to include (which will be the last to be read) + * uses the smallest state value possible, saving the cost of this symbol */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void FSE_initCState2(ref FSE_CState_t statePtr, uint* ct, uint symbol) + { + FSE_initCState(ref statePtr, ct); + { + FSE_symbolCompressionTransform symbolTT = ( + (FSE_symbolCompressionTransform*)statePtr.symbolTT + )[symbol]; + ushort* stateTable = (ushort*)statePtr.stateTable; + uint nbBitsOut = symbolTT.deltaNbBits + (1 << 15) >> 16; + statePtr.value = (nint)((nbBitsOut << 16) - symbolTT.deltaNbBits); + statePtr.value = stateTable[ + (statePtr.value >> (int)nbBitsOut) + symbolTT.deltaFindState + ]; + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void FSE_encodeSymbol( + ref nuint bitC_bitContainer, + ref uint bitC_bitPos, + ref FSE_CState_t statePtr, + uint symbol + ) + { + FSE_symbolCompressionTransform symbolTT = ( + (FSE_symbolCompressionTransform*)statePtr.symbolTT + )[symbol]; + ushort* stateTable = (ushort*)statePtr.stateTable; + uint nbBitsOut = (uint)statePtr.value + symbolTT.deltaNbBits >> 16; + BIT_addBits(ref bitC_bitContainer, ref bitC_bitPos, (nuint)statePtr.value, nbBitsOut); + statePtr.value = stateTable[(statePtr.value >> (int)nbBitsOut) + symbolTT.deltaFindState]; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void FSE_flushCState( + ref nuint bitC_bitContainer, + ref uint bitC_bitPos, + ref sbyte* bitC_ptr, + sbyte* bitC_endPtr, + ref FSE_CState_t statePtr + ) + { + BIT_addBits( + ref bitC_bitContainer, + ref bitC_bitPos, + (nuint)statePtr.value, + statePtr.stateLog + ); + BIT_flushBits(ref bitC_bitContainer, ref bitC_bitPos, ref bitC_ptr, bitC_endPtr); + } + + /* FSE_getMaxNbBits() : + * Approximate maximum cost of a symbol, in bits. + * Fractional get rounded up (i.e. a symbol with a normalized frequency of 3 gives the same result as a frequency of 2) + * note 1 : assume symbolValue is valid (<= maxSymbolValue) + * note 2 : if freq[symbolValue]==0, @return a fake cost of tableLog+1 bits */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint FSE_getMaxNbBits(void* symbolTTPtr, uint symbolValue) + { + FSE_symbolCompressionTransform* symbolTT = (FSE_symbolCompressionTransform*)symbolTTPtr; + return symbolTT[symbolValue].deltaNbBits + ((1 << 16) - 1) >> 16; + } + + /* FSE_bitCost() : + * Approximate symbol cost, as fractional value, using fixed-point format (accuracyLog fractional bits) + * note 1 : assume symbolValue is valid (<= maxSymbolValue) + * note 2 : if freq[symbolValue]==0, @return a fake cost of tableLog+1 bits */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint FSE_bitCost( + void* symbolTTPtr, + uint tableLog, + uint symbolValue, + uint accuracyLog + ) + { + FSE_symbolCompressionTransform* symbolTT = (FSE_symbolCompressionTransform*)symbolTTPtr; + uint minNbBits = symbolTT[symbolValue].deltaNbBits >> 16; + uint threshold = minNbBits + 1 << 16; + assert(tableLog < 16); + assert(accuracyLog < 31 - tableLog); + { + uint tableSize = (uint)(1 << (int)tableLog); + uint deltaFromThreshold = threshold - (symbolTT[symbolValue].deltaNbBits + tableSize); + /* linear interpolation (very approximate) */ + uint normalizedDeltaFromThreshold = + deltaFromThreshold << (int)accuracyLog >> (int)tableLog; + uint bitMultiplier = (uint)(1 << (int)accuracyLog); + assert(symbolTT[symbolValue].deltaNbBits + tableSize <= threshold); + assert(normalizedDeltaFromThreshold <= bitMultiplier); + return (minNbBits + 1) * bitMultiplier - normalizedDeltaFromThreshold; + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void FSE_initDState(ref FSE_DState_t DStatePtr, ref BIT_DStream_t bitD, uint* dt) + { + void* ptr = dt; + FSE_DTableHeader* DTableH = (FSE_DTableHeader*)ptr; + DStatePtr.state = BIT_readBits(bitD.bitContainer, ref bitD.bitsConsumed, DTableH->tableLog); + BIT_reloadDStream( + ref bitD.bitContainer, + ref bitD.bitsConsumed, + ref bitD.ptr, + bitD.start, + bitD.limitPtr + ); + DStatePtr.table = dt + 1; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static byte FSE_peekSymbol(FSE_DState_t* DStatePtr) + { + FSE_decode_t DInfo = ((FSE_decode_t*)DStatePtr->table)[DStatePtr->state]; + return DInfo.symbol; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void FSE_updateState(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD) + { + FSE_decode_t DInfo = ((FSE_decode_t*)DStatePtr->table)[DStatePtr->state]; + uint nbBits = DInfo.nbBits; + nuint lowBits = BIT_readBits(bitD, nbBits); + DStatePtr->state = DInfo.newState + lowBits; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static byte FSE_decodeSymbol( + ref FSE_DState_t DStatePtr, + nuint bitD_bitContainer, + ref uint bitD_bitsConsumed + ) + { + FSE_decode_t DInfo = ((FSE_decode_t*)DStatePtr.table)[DStatePtr.state]; + uint nbBits = DInfo.nbBits; + byte symbol = DInfo.symbol; + nuint lowBits = BIT_readBits(bitD_bitContainer, ref bitD_bitsConsumed, nbBits); + DStatePtr.state = DInfo.newState + lowBits; + return symbol; + } + + /*! FSE_decodeSymbolFast() : + unsafe, only works if no symbol has a probability > 50% */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static byte FSE_decodeSymbolFast( + ref FSE_DState_t DStatePtr, + nuint bitD_bitContainer, + ref uint bitD_bitsConsumed + ) + { + FSE_decode_t DInfo = ((FSE_decode_t*)DStatePtr.table)[DStatePtr.state]; + uint nbBits = DInfo.nbBits; + byte symbol = DInfo.symbol; + nuint lowBits = BIT_readBitsFast(bitD_bitContainer, ref bitD_bitsConsumed, nbBits); + DStatePtr.state = DInfo.newState + lowBits; + return symbol; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint FSE_endOfDState(FSE_DState_t* DStatePtr) + { + return DStatePtr->state == 0 ? 1U : 0U; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void FSE_initCState(ref FSE_CState_t statePtr, uint* ct) + { + void* ptr = ct; + ushort* u16ptr = (ushort*)ptr; + uint tableLog = MEM_read16(ptr); + statePtr.value = (nint)1 << (int)tableLog; + statePtr.stateTable = u16ptr + 2; + statePtr.symbolTT = ct + 1 + (tableLog != 0 ? 1 << (int)(tableLog - 1) : 1); + statePtr.stateLog = tableLog; + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/FseCompress.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/FseCompress.cs new file mode 100644 index 00000000..68fbba62 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/FseCompress.cs @@ -0,0 +1,895 @@ +using System; +using System.Runtime.InteropServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + /* FSE_buildCTable_wksp() : + * Same as FSE_buildCTable(), but using an externally allocated scratch buffer (`workSpace`). + * wkspSize should be sized to handle worst case situation, which is `1<> 1 : 1); + FSE_symbolCompressionTransform* symbolTT = (FSE_symbolCompressionTransform*)FSCT; + uint step = (tableSize >> 1) + (tableSize >> 3) + 3; + uint maxSV1 = maxSymbolValue + 1; + /* size = maxSV1 */ + ushort* cumul = (ushort*)workSpace; + /* size = tableSize */ + byte* tableSymbol = (byte*)(cumul + (maxSV1 + 1)); + uint highThreshold = tableSize - 1; + assert(((nuint)workSpace & 1) == 0); + if ( + sizeof(uint) + * ((maxSymbolValue + 2 + (1UL << (int)tableLog)) / 2 + sizeof(ulong) / sizeof(uint)) + > wkspSize + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_tableLog_tooLarge)); + } + + tableU16[-2] = (ushort)tableLog; + tableU16[-1] = (ushort)maxSymbolValue; + assert(tableLog < 16); + { + uint u; + cumul[0] = 0; + for (u = 1; u <= maxSV1; u++) + { + if (normalizedCounter[u - 1] == -1) + { + cumul[u] = (ushort)(cumul[u - 1] + 1); + tableSymbol[highThreshold--] = (byte)(u - 1); + } + else + { + assert(normalizedCounter[u - 1] >= 0); + cumul[u] = (ushort)(cumul[u - 1] + (ushort)normalizedCounter[u - 1]); + assert(cumul[u] >= cumul[u - 1]); + } + } + + cumul[maxSV1] = (ushort)(tableSize + 1); + } + + if (highThreshold == tableSize - 1) + { + /* size = tableSize + 8 (may write beyond tableSize) */ + byte* spread = tableSymbol + tableSize; + { + const ulong add = 0x0101010101010101UL; + nuint pos = 0; + ulong sv = 0; + uint s; + for (s = 0; s < maxSV1; ++s, sv += add) + { + int i; + int n = normalizedCounter[s]; + MEM_write64(spread + pos, sv); + for (i = 8; i < n; i += 8) + { + MEM_write64(spread + pos + i, sv); + } + + assert(n >= 0); + pos += (nuint)n; + } + } + + { + nuint position = 0; + nuint s; + /* Experimentally determined optimal unroll */ + const nuint unroll = 2; + assert(tableSize % unroll == 0); + for (s = 0; s < tableSize; s += unroll) + { + nuint u; + for (u = 0; u < unroll; ++u) + { + nuint uPosition = position + u * step & tableMask; + tableSymbol[uPosition] = spread[s + u]; + } + + position = position + unroll * step & tableMask; + } + + assert(position == 0); + } + } + else + { + uint position = 0; + uint symbol; + for (symbol = 0; symbol < maxSV1; symbol++) + { + int nbOccurrences; + int freq = normalizedCounter[symbol]; + for (nbOccurrences = 0; nbOccurrences < freq; nbOccurrences++) + { + tableSymbol[position] = (byte)symbol; + position = position + step & tableMask; + while (position > highThreshold) + { + position = position + step & tableMask; + } + } + } + + assert(position == 0); + } + + { + uint u; + for (u = 0; u < tableSize; u++) + { + /* note : static analyzer may not understand tableSymbol is properly initialized */ + byte s = tableSymbol[u]; + tableU16[cumul[s]++] = (ushort)(tableSize + u); + } + } + + { + uint total = 0; + uint s; + for (s = 0; s <= maxSymbolValue; s++) + { + switch (normalizedCounter[s]) + { + case 0: + symbolTT[s].deltaNbBits = (tableLog + 1 << 16) - (uint)(1 << (int)tableLog); + break; + case -1: + case 1: + symbolTT[s].deltaNbBits = (tableLog << 16) - (uint)(1 << (int)tableLog); + assert(total <= 2147483647); + symbolTT[s].deltaFindState = (int)(total - 1); + total++; + break; + default: + assert(normalizedCounter[s] > 1); + + { + uint maxBitsOut = + tableLog - ZSTD_highbit32((uint)normalizedCounter[s] - 1); + uint minStatePlus = (uint)normalizedCounter[s] << (int)maxBitsOut; + symbolTT[s].deltaNbBits = (maxBitsOut << 16) - minStatePlus; + symbolTT[s].deltaFindState = (int)(total - (uint)normalizedCounter[s]); + total += (uint)normalizedCounter[s]; + } + + break; + } + } + } + + return 0; + } + + /*-************************************************************** + * FSE NCount encoding + ****************************************************************/ + private static nuint FSE_NCountWriteBound(uint maxSymbolValue, uint tableLog) + { + nuint maxHeaderSize = ((maxSymbolValue + 1) * tableLog + 4 + 2) / 8 + 1 + 2; + return maxSymbolValue != 0 ? maxHeaderSize : 512; + } + + private static nuint FSE_writeNCount_generic( + void* header, + nuint headerBufferSize, + short* normalizedCounter, + uint maxSymbolValue, + uint tableLog, + uint writeIsSafe + ) + { + byte* ostart = (byte*)header; + byte* @out = ostart; + byte* oend = ostart + headerBufferSize; + int nbBits; + int tableSize = 1 << (int)tableLog; + int remaining; + int threshold; + uint bitStream = 0; + int bitCount = 0; + uint symbol = 0; + uint alphabetSize = maxSymbolValue + 1; + int previousIs0 = 0; + bitStream += tableLog - 5 << bitCount; + bitCount += 4; + remaining = tableSize + 1; + threshold = tableSize; + nbBits = (int)tableLog + 1; + while (symbol < alphabetSize && remaining > 1) + { + if (previousIs0 != 0) + { + uint start = symbol; + while (symbol < alphabetSize && normalizedCounter[symbol] == 0) + { + symbol++; + } + + if (symbol == alphabetSize) + { + break; + } + + while (symbol >= start + 24) + { + start += 24; + bitStream += 0xFFFFU << bitCount; + if (writeIsSafe == 0 && @out > oend - 2) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + @out[0] = (byte)bitStream; + @out[1] = (byte)(bitStream >> 8); + @out += 2; + bitStream >>= 16; + } + + while (symbol >= start + 3) + { + start += 3; + bitStream += 3U << bitCount; + bitCount += 2; + } + + bitStream += symbol - start << bitCount; + bitCount += 2; + if (bitCount > 16) + { + if (writeIsSafe == 0 && @out > oend - 2) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + @out[0] = (byte)bitStream; + @out[1] = (byte)(bitStream >> 8); + @out += 2; + bitStream >>= 16; + bitCount -= 16; + } + } + + { + int count = normalizedCounter[symbol++]; + int max = 2 * threshold - 1 - remaining; + remaining -= count < 0 ? -count : count; + count++; + if (count >= threshold) + { + count += max; + } + + bitStream += (uint)count << bitCount; + bitCount += nbBits; + bitCount -= count < max ? 1 : 0; + previousIs0 = count == 1 ? 1 : 0; + if (remaining < 1) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + + while (remaining < threshold) + { + nbBits--; + threshold >>= 1; + } + } + + if (bitCount > 16) + { + if (writeIsSafe == 0 && @out > oend - 2) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + @out[0] = (byte)bitStream; + @out[1] = (byte)(bitStream >> 8); + @out += 2; + bitStream >>= 16; + bitCount -= 16; + } + } + + if (remaining != 1) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + + assert(symbol <= alphabetSize); + if (writeIsSafe == 0 && @out > oend - 2) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + @out[0] = (byte)bitStream; + @out[1] = (byte)(bitStream >> 8); + @out += (bitCount + 7) / 8; + assert(@out >= ostart); + return (nuint)(@out - ostart); + } + + /*! FSE_writeNCount(): + Compactly save 'normalizedCounter' into 'buffer'. + @return : size of the compressed table, + or an errorCode, which can be tested using FSE_isError(). */ + private static nuint FSE_writeNCount( + void* buffer, + nuint bufferSize, + short* normalizedCounter, + uint maxSymbolValue, + uint tableLog + ) + { + if (tableLog > 14 - 2) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_tableLog_tooLarge)); + } + + if (tableLog < 5) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + + if (bufferSize < FSE_NCountWriteBound(maxSymbolValue, tableLog)) + { + return FSE_writeNCount_generic( + buffer, + bufferSize, + normalizedCounter, + maxSymbolValue, + tableLog, + 0 + ); + } + + return FSE_writeNCount_generic( + buffer, + bufferSize, + normalizedCounter, + maxSymbolValue, + tableLog, + 1 + ); + } + + /* provides the minimum logSize to safely represent a distribution */ + private static uint FSE_minTableLog(nuint srcSize, uint maxSymbolValue) + { + uint minBitsSrc = ZSTD_highbit32((uint)srcSize) + 1; + uint minBitsSymbols = ZSTD_highbit32(maxSymbolValue) + 2; + uint minBits = minBitsSrc < minBitsSymbols ? minBitsSrc : minBitsSymbols; + assert(srcSize > 1); + return minBits; + } + + /* ***************************************** + * FSE advanced API + ***************************************** */ + private static uint FSE_optimalTableLog_internal( + uint maxTableLog, + nuint srcSize, + uint maxSymbolValue, + uint minus + ) + { + uint maxBitsSrc = ZSTD_highbit32((uint)(srcSize - 1)) - minus; + uint tableLog = maxTableLog; + uint minBits = FSE_minTableLog(srcSize, maxSymbolValue); + assert(srcSize > 1); + if (tableLog == 0) + { + tableLog = 13 - 2; + } + + if (maxBitsSrc < tableLog) + { + tableLog = maxBitsSrc; + } + + if (minBits > tableLog) + { + tableLog = minBits; + } + + if (tableLog < 5) + { + tableLog = 5; + } + + if (tableLog > 14 - 2) + { + tableLog = 14 - 2; + } + + return tableLog; + } + + /*! FSE_optimalTableLog(): + dynamically downsize 'tableLog' when conditions are met. + It saves CPU time, by using smaller tables, while preserving or even improving compression ratio. + @return : recommended tableLog (necessarily <= 'maxTableLog') */ + private static uint FSE_optimalTableLog(uint maxTableLog, nuint srcSize, uint maxSymbolValue) + { + return FSE_optimalTableLog_internal(maxTableLog, srcSize, maxSymbolValue, 2); + } + + /* Secondary normalization method. + To be used when primary method fails. */ + private static nuint FSE_normalizeM2( + short* norm, + uint tableLog, + uint* count, + nuint total, + uint maxSymbolValue, + short lowProbCount + ) + { + const short NOT_YET_ASSIGNED = -2; + uint s; + uint distributed = 0; + uint ToDistribute; + /* Init */ + uint lowThreshold = (uint)(total >> (int)tableLog); + uint lowOne = (uint)(total * 3 >> (int)(tableLog + 1)); + for (s = 0; s <= maxSymbolValue; s++) + { + if (count[s] == 0) + { + norm[s] = 0; + continue; + } + + if (count[s] <= lowThreshold) + { + norm[s] = lowProbCount; + distributed++; + total -= count[s]; + continue; + } + + if (count[s] <= lowOne) + { + norm[s] = 1; + distributed++; + total -= count[s]; + continue; + } + + norm[s] = NOT_YET_ASSIGNED; + } + + ToDistribute = (uint)(1 << (int)tableLog) - distributed; + if (ToDistribute == 0) + { + return 0; + } + + if (total / ToDistribute > lowOne) + { + lowOne = (uint)(total * 3 / (ToDistribute * 2)); + for (s = 0; s <= maxSymbolValue; s++) + { + if (norm[s] == NOT_YET_ASSIGNED && count[s] <= lowOne) + { + norm[s] = 1; + distributed++; + total -= count[s]; + continue; + } + } + + ToDistribute = (uint)(1 << (int)tableLog) - distributed; + } + + if (distributed == maxSymbolValue + 1) + { + /* all values are pretty poor; + probably incompressible data (should have already been detected); + find max, then give all remaining points to max */ + uint maxV = 0, + maxC = 0; + for (s = 0; s <= maxSymbolValue; s++) + { + if (count[s] > maxC) + { + maxV = s; + maxC = count[s]; + } + } + + norm[maxV] += (short)ToDistribute; + return 0; + } + + if (total == 0) + { + for (s = 0; ToDistribute > 0; s = (s + 1) % (maxSymbolValue + 1)) + { + if (norm[s] > 0) + { + ToDistribute--; + norm[s]++; + } + } + + return 0; + } + + { + ulong vStepLog = 62 - tableLog; + ulong mid = (1UL << (int)(vStepLog - 1)) - 1; + /* scale on remaining */ + ulong rStep = (((ulong)1 << (int)vStepLog) * ToDistribute + mid) / (uint)total; + ulong tmpTotal = mid; + for (s = 0; s <= maxSymbolValue; s++) + { + if (norm[s] == NOT_YET_ASSIGNED) + { + ulong end = tmpTotal + count[s] * rStep; + uint sStart = (uint)(tmpTotal >> (int)vStepLog); + uint sEnd = (uint)(end >> (int)vStepLog); + uint weight = sEnd - sStart; + if (weight < 1) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + + norm[s] = (short)weight; + tmpTotal = end; + } + } + } + + return 0; + } + +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_rtbTable => + new uint[8] { 0, 473195, 504333, 520860, 550000, 700000, 750000, 830000 }; + private static uint* rtbTable => + (uint*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_rtbTable) + ); +#else + + private static readonly uint* rtbTable = GetArrayPointer( + new uint[8] { 0, 473195, 504333, 520860, 550000, 700000, 750000, 830000 } + ); +#endif + /*! FSE_normalizeCount(): + normalize counts so that sum(count[]) == Power_of_2 (2^tableLog) + 'normalizedCounter' is a table of short, of minimum size (maxSymbolValue+1). + useLowProbCount is a boolean parameter which trades off compressed size for + faster header decoding. When it is set to 1, the compressed data will be slightly + smaller. And when it is set to 0, FSE_readNCount() and FSE_buildDTable() will be + faster. If you are compressing a small amount of data (< 2 KB) then useLowProbCount=0 + is a good default, since header deserialization makes a big speed difference. + Otherwise, useLowProbCount=1 is a good default, since the speed difference is small. + @return : tableLog, + or an errorCode, which can be tested using FSE_isError() */ + private static nuint FSE_normalizeCount( + short* normalizedCounter, + uint tableLog, + uint* count, + nuint total, + uint maxSymbolValue, + uint useLowProbCount + ) + { + if (tableLog == 0) + { + tableLog = 13 - 2; + } + + if (tableLog < 5) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + + if (tableLog > 14 - 2) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_tableLog_tooLarge)); + } + + if (tableLog < FSE_minTableLog(total, maxSymbolValue)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + + { + short lowProbCount = (short)(useLowProbCount != 0 ? -1 : 1); + ulong scale = 62 - tableLog; + /* <== here, one division ! */ + ulong step = ((ulong)1 << 62) / (uint)total; + ulong vStep = 1UL << (int)(scale - 20); + int stillToDistribute = 1 << (int)tableLog; + uint s; + uint largest = 0; + short largestP = 0; + uint lowThreshold = (uint)(total >> (int)tableLog); + for (s = 0; s <= maxSymbolValue; s++) + { + if (count[s] == total) + { + return 0; + } + + if (count[s] == 0) + { + normalizedCounter[s] = 0; + continue; + } + + if (count[s] <= lowThreshold) + { + normalizedCounter[s] = lowProbCount; + stillToDistribute--; + } + else + { + short proba = (short)(count[s] * step >> (int)scale); + if (proba < 8) + { + ulong restToBeat = vStep * rtbTable[proba]; + proba += (short)( + count[s] * step - ((ulong)proba << (int)scale) > restToBeat ? 1 : 0 + ); + } + + if (proba > largestP) + { + largestP = proba; + largest = s; + } + + normalizedCounter[s] = proba; + stillToDistribute -= proba; + } + } + + if (-stillToDistribute >= normalizedCounter[largest] >> 1) + { + /* corner case, need another normalization method */ + nuint errorCode = FSE_normalizeM2( + normalizedCounter, + tableLog, + count, + total, + maxSymbolValue, + lowProbCount + ); + if (ERR_isError(errorCode)) + { + return errorCode; + } + } + else + { + normalizedCounter[largest] += (short)stillToDistribute; + } + } + + return tableLog; + } + + /* fake FSE_CTable, for rle input (always same symbol) */ + private static nuint FSE_buildCTable_rle(uint* ct, byte symbolValue) + { + void* ptr = ct; + ushort* tableU16 = (ushort*)ptr + 2; + void* FSCTptr = (uint*)ptr + 2; + FSE_symbolCompressionTransform* symbolTT = (FSE_symbolCompressionTransform*)FSCTptr; + tableU16[-2] = 0; + tableU16[-1] = symbolValue; + tableU16[0] = 0; + tableU16[1] = 0; + symbolTT[symbolValue].deltaNbBits = 0; + symbolTT[symbolValue].deltaFindState = 0; + return 0; + } + + private static nuint FSE_compress_usingCTable_generic( + void* dst, + nuint dstSize, + void* src, + nuint srcSize, + uint* ct, + uint fast + ) + { + byte* istart = (byte*)src; + byte* iend = istart + srcSize; + byte* ip = iend; + BIT_CStream_t bitC; + System.Runtime.CompilerServices.Unsafe.SkipInit(out bitC); + FSE_CState_t CState1, + CState2; + System.Runtime.CompilerServices.Unsafe.SkipInit(out CState1); + System.Runtime.CompilerServices.Unsafe.SkipInit(out CState2); + if (srcSize <= 2) + { + return 0; + } + + { + nuint initError = BIT_initCStream(ref bitC, dst, dstSize); + if (ERR_isError(initError)) + { + return 0; + } + } + + nuint bitC_bitContainer = bitC.bitContainer; + uint bitC_bitPos = bitC.bitPos; + sbyte* bitC_ptr = bitC.ptr; + sbyte* bitC_endPtr = bitC.endPtr; + if ((srcSize & 1) != 0) + { + FSE_initCState2(ref CState1, ct, *--ip); + FSE_initCState2(ref CState2, ct, *--ip); + FSE_encodeSymbol(ref bitC_bitContainer, ref bitC_bitPos, ref CState1, *--ip); + if (fast != 0) + { + BIT_flushBitsFast( + ref bitC_bitContainer, + ref bitC_bitPos, + ref bitC_ptr, + bitC_endPtr + ); + } + else + { + BIT_flushBits(ref bitC_bitContainer, ref bitC_bitPos, ref bitC_ptr, bitC_endPtr); + } + } + else + { + FSE_initCState2(ref CState2, ct, *--ip); + FSE_initCState2(ref CState1, ct, *--ip); + } + + srcSize -= 2; + if (sizeof(nuint) * 8 > (14 - 2) * 4 + 7 && (srcSize & 2) != 0) + { + FSE_encodeSymbol(ref bitC_bitContainer, ref bitC_bitPos, ref CState2, *--ip); + FSE_encodeSymbol(ref bitC_bitContainer, ref bitC_bitPos, ref CState1, *--ip); + if (fast != 0) + { + BIT_flushBitsFast( + ref bitC_bitContainer, + ref bitC_bitPos, + ref bitC_ptr, + bitC_endPtr + ); + } + else + { + BIT_flushBits(ref bitC_bitContainer, ref bitC_bitPos, ref bitC_ptr, bitC_endPtr); + } + } + + while (ip > istart) + { + FSE_encodeSymbol(ref bitC_bitContainer, ref bitC_bitPos, ref CState2, *--ip); + if (sizeof(nuint) * 8 < (14 - 2) * 2 + 7) + { + if (fast != 0) + { + BIT_flushBitsFast( + ref bitC_bitContainer, + ref bitC_bitPos, + ref bitC_ptr, + bitC_endPtr + ); + } + else + { + BIT_flushBits( + ref bitC_bitContainer, + ref bitC_bitPos, + ref bitC_ptr, + bitC_endPtr + ); + } + } + + FSE_encodeSymbol(ref bitC_bitContainer, ref bitC_bitPos, ref CState1, *--ip); + if (sizeof(nuint) * 8 > (14 - 2) * 4 + 7) + { + FSE_encodeSymbol(ref bitC_bitContainer, ref bitC_bitPos, ref CState2, *--ip); + FSE_encodeSymbol(ref bitC_bitContainer, ref bitC_bitPos, ref CState1, *--ip); + } + + if (fast != 0) + { + BIT_flushBitsFast( + ref bitC_bitContainer, + ref bitC_bitPos, + ref bitC_ptr, + bitC_endPtr + ); + } + else + { + BIT_flushBits(ref bitC_bitContainer, ref bitC_bitPos, ref bitC_ptr, bitC_endPtr); + } + } + + FSE_flushCState( + ref bitC_bitContainer, + ref bitC_bitPos, + ref bitC_ptr, + bitC_endPtr, + ref CState2 + ); + FSE_flushCState( + ref bitC_bitContainer, + ref bitC_bitPos, + ref bitC_ptr, + bitC_endPtr, + ref CState1 + ); + return BIT_closeCStream( + ref bitC_bitContainer, + ref bitC_bitPos, + bitC_ptr, + bitC_endPtr, + bitC.startPtr + ); + } + + /*! FSE_compress_usingCTable(): + Compress `src` using `ct` into `dst` which must be already allocated. + @return : size of compressed data (<= `dstCapacity`), + or 0 if compressed data could not fit into `dst`, + or an errorCode, which can be tested using FSE_isError() */ + private static nuint FSE_compress_usingCTable( + void* dst, + nuint dstSize, + void* src, + nuint srcSize, + uint* ct + ) + { + uint fast = dstSize >= srcSize + (srcSize >> 7) + 4 + (nuint)sizeof(nuint) ? 1U : 0U; + if (fast != 0) + { + return FSE_compress_usingCTable_generic(dst, dstSize, src, srcSize, ct, 1); + } + else + { + return FSE_compress_usingCTable_generic(dst, dstSize, src, srcSize, ct, 0); + } + } + + /*-***************************************** + * Tool functions + ******************************************/ + private static nuint FSE_compressBound(nuint size) + { + return 512 + (size + (size >> 7) + 4 + (nuint)sizeof(nuint)); + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/FseDecompress.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/FseDecompress.cs new file mode 100644 index 00000000..6fc7c376 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/FseDecompress.cs @@ -0,0 +1,509 @@ +using System.Runtime.CompilerServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + private static nuint FSE_buildDTable_internal( + uint* dt, + short* normalizedCounter, + uint maxSymbolValue, + uint tableLog, + void* workSpace, + nuint wkspSize + ) + { + /* because *dt is unsigned, 32-bits aligned on 32-bits */ + void* tdPtr = dt + 1; + FSE_decode_t* tableDecode = (FSE_decode_t*)tdPtr; + ushort* symbolNext = (ushort*)workSpace; + byte* spread = (byte*)(symbolNext + maxSymbolValue + 1); + uint maxSV1 = maxSymbolValue + 1; + uint tableSize = (uint)(1 << (int)tableLog); + uint highThreshold = tableSize - 1; + if (sizeof(short) * (maxSymbolValue + 1) + (1UL << (int)tableLog) + 8 > wkspSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_maxSymbolValue_tooLarge)); + } + + if (maxSymbolValue > 255) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_maxSymbolValue_tooLarge)); + } + + if (tableLog > 14 - 2) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_tableLog_tooLarge)); + } + + { + FSE_DTableHeader DTableH; + DTableH.tableLog = (ushort)tableLog; + DTableH.fastMode = 1; + { + short largeLimit = (short)(1 << (int)(tableLog - 1)); + uint s; + for (s = 0; s < maxSV1; s++) + { + if (normalizedCounter[s] == -1) + { + tableDecode[highThreshold--].symbol = (byte)s; + symbolNext[s] = 1; + } + else + { + if (normalizedCounter[s] >= largeLimit) + { + DTableH.fastMode = 0; + } + + symbolNext[s] = (ushort)normalizedCounter[s]; + } + } + } + + memcpy(dt, &DTableH, (uint)sizeof(FSE_DTableHeader)); + } + + if (highThreshold == tableSize - 1) + { + nuint tableMask = tableSize - 1; + nuint step = (tableSize >> 1) + (tableSize >> 3) + 3; + { + const ulong add = 0x0101010101010101UL; + nuint pos = 0; + ulong sv = 0; + uint s; + for (s = 0; s < maxSV1; ++s, sv += add) + { + int i; + int n = normalizedCounter[s]; + MEM_write64(spread + pos, sv); + for (i = 8; i < n; i += 8) + { + MEM_write64(spread + pos + i, sv); + } + + pos += (nuint)n; + } + } + + { + nuint position = 0; + nuint s; + const nuint unroll = 2; + assert(tableSize % unroll == 0); + for (s = 0; s < tableSize; s += unroll) + { + nuint u; + for (u = 0; u < unroll; ++u) + { + nuint uPosition = position + u * step & tableMask; + tableDecode[uPosition].symbol = spread[s + u]; + } + + position = position + unroll * step & tableMask; + } + + assert(position == 0); + } + } + else + { + uint tableMask = tableSize - 1; + uint step = (tableSize >> 1) + (tableSize >> 3) + 3; + uint s, + position = 0; + for (s = 0; s < maxSV1; s++) + { + int i; + for (i = 0; i < normalizedCounter[s]; i++) + { + tableDecode[position].symbol = (byte)s; + position = position + step & tableMask; + while (position > highThreshold) + { + position = position + step & tableMask; + } + } + } + + if (position != 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + } + + { + uint u; + for (u = 0; u < tableSize; u++) + { + byte symbol = tableDecode[u].symbol; + uint nextState = symbolNext[symbol]++; + tableDecode[u].nbBits = (byte)(tableLog - ZSTD_highbit32(nextState)); + tableDecode[u].newState = (ushort)( + (nextState << tableDecode[u].nbBits) - tableSize + ); + } + } + + return 0; + } + + private static nuint FSE_buildDTable_wksp( + uint* dt, + short* normalizedCounter, + uint maxSymbolValue, + uint tableLog, + void* workSpace, + nuint wkspSize + ) + { + return FSE_buildDTable_internal( + dt, + normalizedCounter, + maxSymbolValue, + tableLog, + workSpace, + wkspSize + ); + } + + /*-******************************************************* + * Decompression (Byte symbols) + *********************************************************/ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint FSE_decompress_usingDTable_generic( + void* dst, + nuint maxDstSize, + void* cSrc, + nuint cSrcSize, + uint* dt, + uint fast + ) + { + byte* ostart = (byte*)dst; + byte* op = ostart; + byte* omax = op + maxDstSize; + byte* olimit = omax - 3; + BIT_DStream_t bitD; + System.Runtime.CompilerServices.Unsafe.SkipInit(out bitD); + FSE_DState_t state1; + System.Runtime.CompilerServices.Unsafe.SkipInit(out state1); + FSE_DState_t state2; + System.Runtime.CompilerServices.Unsafe.SkipInit(out state2); + { + /* Init */ + nuint _var_err__ = BIT_initDStream(ref bitD, cSrc, cSrcSize); + if (ERR_isError(_var_err__)) + { + return _var_err__; + } + } + + FSE_initDState(ref state1, ref bitD, dt); + FSE_initDState(ref state2, ref bitD, dt); + nuint bitD_bitContainer = bitD.bitContainer; + uint bitD_bitsConsumed = bitD.bitsConsumed; + sbyte* bitD_ptr = bitD.ptr; + sbyte* bitD_start = bitD.start; + sbyte* bitD_limitPtr = bitD.limitPtr; + if ( + BIT_reloadDStream( + ref bitD_bitContainer, + ref bitD_bitsConsumed, + ref bitD_ptr, + bitD_start, + bitD_limitPtr + ) == BIT_DStream_status.BIT_DStream_overflow + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + for ( + ; + BIT_reloadDStream( + ref bitD_bitContainer, + ref bitD_bitsConsumed, + ref bitD_ptr, + bitD_start, + bitD_limitPtr + ) == BIT_DStream_status.BIT_DStream_unfinished + && op < olimit; + op += 4 + ) + { + op[0] = + fast != 0 + ? FSE_decodeSymbolFast(ref state1, bitD_bitContainer, ref bitD_bitsConsumed) + : FSE_decodeSymbol(ref state1, bitD_bitContainer, ref bitD_bitsConsumed); + if ((14 - 2) * 2 + 7 > sizeof(nuint) * 8) + { + BIT_reloadDStream( + ref bitD_bitContainer, + ref bitD_bitsConsumed, + ref bitD_ptr, + bitD_start, + bitD_limitPtr + ); + } + + op[1] = + fast != 0 + ? FSE_decodeSymbolFast(ref state2, bitD_bitContainer, ref bitD_bitsConsumed) + : FSE_decodeSymbol(ref state2, bitD_bitContainer, ref bitD_bitsConsumed); + if ((14 - 2) * 4 + 7 > sizeof(nuint) * 8) + { + if ( + BIT_reloadDStream( + ref bitD_bitContainer, + ref bitD_bitsConsumed, + ref bitD_ptr, + bitD_start, + bitD_limitPtr + ) > BIT_DStream_status.BIT_DStream_unfinished + ) + { + op += 2; + break; + } + } + + op[2] = + fast != 0 + ? FSE_decodeSymbolFast(ref state1, bitD_bitContainer, ref bitD_bitsConsumed) + : FSE_decodeSymbol(ref state1, bitD_bitContainer, ref bitD_bitsConsumed); + if ((14 - 2) * 2 + 7 > sizeof(nuint) * 8) + { + BIT_reloadDStream( + ref bitD_bitContainer, + ref bitD_bitsConsumed, + ref bitD_ptr, + bitD_start, + bitD_limitPtr + ); + } + + op[3] = + fast != 0 + ? FSE_decodeSymbolFast(ref state2, bitD_bitContainer, ref bitD_bitsConsumed) + : FSE_decodeSymbol(ref state2, bitD_bitContainer, ref bitD_bitsConsumed); + } + + while (true) + { + if (op > omax - 2) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + *op++ = + fast != 0 + ? FSE_decodeSymbolFast(ref state1, bitD_bitContainer, ref bitD_bitsConsumed) + : FSE_decodeSymbol(ref state1, bitD_bitContainer, ref bitD_bitsConsumed); + if ( + BIT_reloadDStream( + ref bitD_bitContainer, + ref bitD_bitsConsumed, + ref bitD_ptr, + bitD_start, + bitD_limitPtr + ) == BIT_DStream_status.BIT_DStream_overflow + ) + { + *op++ = + fast != 0 + ? FSE_decodeSymbolFast(ref state2, bitD_bitContainer, ref bitD_bitsConsumed) + : FSE_decodeSymbol(ref state2, bitD_bitContainer, ref bitD_bitsConsumed); + break; + } + + if (op > omax - 2) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + *op++ = + fast != 0 + ? FSE_decodeSymbolFast(ref state2, bitD_bitContainer, ref bitD_bitsConsumed) + : FSE_decodeSymbol(ref state2, bitD_bitContainer, ref bitD_bitsConsumed); + if ( + BIT_reloadDStream( + ref bitD_bitContainer, + ref bitD_bitsConsumed, + ref bitD_ptr, + bitD_start, + bitD_limitPtr + ) == BIT_DStream_status.BIT_DStream_overflow + ) + { + *op++ = + fast != 0 + ? FSE_decodeSymbolFast(ref state1, bitD_bitContainer, ref bitD_bitsConsumed) + : FSE_decodeSymbol(ref state1, bitD_bitContainer, ref bitD_bitsConsumed); + break; + } + } + + assert(op >= ostart); + return (nuint)(op - ostart); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint FSE_decompress_wksp_body( + void* dst, + nuint dstCapacity, + void* cSrc, + nuint cSrcSize, + uint maxLog, + void* workSpace, + nuint wkspSize, + int bmi2 + ) + { + byte* istart = (byte*)cSrc; + byte* ip = istart; + uint tableLog; + uint maxSymbolValue = 255; + FSE_DecompressWksp* wksp = (FSE_DecompressWksp*)workSpace; + nuint dtablePos = (nuint)(sizeof(FSE_DecompressWksp) / sizeof(uint)); + uint* dtable = (uint*)workSpace + dtablePos; + if (wkspSize < (nuint)sizeof(FSE_DecompressWksp)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + + { + nuint NCountLength = FSE_readNCount_bmi2( + wksp->ncount, + &maxSymbolValue, + &tableLog, + istart, + cSrcSize, + bmi2 + ); + if (ERR_isError(NCountLength)) + { + return NCountLength; + } + + if (tableLog > maxLog) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_tableLog_tooLarge)); + } + + assert(NCountLength <= cSrcSize); + ip += NCountLength; + cSrcSize -= NCountLength; + } + + if ( + ( + (ulong)(1 + (1 << (int)tableLog) + 1) + + ( + sizeof(short) * (maxSymbolValue + 1) + + (1UL << (int)tableLog) + + 8 + + sizeof(uint) + - 1 + ) / sizeof(uint) + + (255 + 1) / 2 + + 1 + ) * sizeof(uint) + > wkspSize + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_tableLog_tooLarge)); + } + + assert( + (nuint)(sizeof(FSE_DecompressWksp) + (1 + (1 << (int)tableLog)) * sizeof(uint)) + <= wkspSize + ); + workSpace = + (byte*)workSpace + + sizeof(FSE_DecompressWksp) + + (1 + (1 << (int)tableLog)) * sizeof(uint); + wkspSize -= (nuint)(sizeof(FSE_DecompressWksp) + (1 + (1 << (int)tableLog)) * sizeof(uint)); + { + nuint _var_err__ = FSE_buildDTable_internal( + dtable, + wksp->ncount, + maxSymbolValue, + tableLog, + workSpace, + wkspSize + ); + if (ERR_isError(_var_err__)) + { + return _var_err__; + } + } + + { + void* ptr = dtable; + FSE_DTableHeader* DTableH = (FSE_DTableHeader*)ptr; + uint fastMode = DTableH->fastMode; + if (fastMode != 0) + { + return FSE_decompress_usingDTable_generic( + dst, + dstCapacity, + ip, + cSrcSize, + dtable, + 1 + ); + } + + return FSE_decompress_usingDTable_generic(dst, dstCapacity, ip, cSrcSize, dtable, 0); + } + } + + /* Avoids the FORCE_INLINE of the _body() function. */ + private static nuint FSE_decompress_wksp_body_default( + void* dst, + nuint dstCapacity, + void* cSrc, + nuint cSrcSize, + uint maxLog, + void* workSpace, + nuint wkspSize + ) + { + return FSE_decompress_wksp_body( + dst, + dstCapacity, + cSrc, + cSrcSize, + maxLog, + workSpace, + wkspSize, + 0 + ); + } + + private static nuint FSE_decompress_wksp_bmi2( + void* dst, + nuint dstCapacity, + void* cSrc, + nuint cSrcSize, + uint maxLog, + void* workSpace, + nuint wkspSize, + int bmi2 + ) + { + return FSE_decompress_wksp_body_default( + dst, + dstCapacity, + cSrc, + cSrcSize, + maxLog, + workSpace, + wkspSize + ); + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/HIST_checkInput_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/HIST_checkInput_e.cs new file mode 100644 index 00000000..8aca6e8f --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/HIST_checkInput_e.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum HIST_checkInput_e +{ + trustInput, + checkMaxSymbolValue, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_CStream_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_CStream_t.cs new file mode 100644 index 00000000..b86d99f3 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_CStream_t.cs @@ -0,0 +1,22 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct HUF_CStream_t +{ + public _bitContainer_e__FixedBuffer bitContainer; + public _bitPos_e__FixedBuffer bitPos; + public byte* startPtr; + public byte* ptr; + public byte* endPtr; + + public unsafe struct _bitContainer_e__FixedBuffer + { + public nuint e0; + public nuint e1; + } + + public unsafe struct _bitPos_e__FixedBuffer + { + public nuint e0; + public nuint e1; + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_CTableHeader.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_CTableHeader.cs new file mode 100644 index 00000000..cc542493 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_CTableHeader.cs @@ -0,0 +1,8 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct HUF_CTableHeader +{ + public byte tableLog; + public byte maxSymbolValue; + public fixed byte unused[6]; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_CompressWeightsWksp.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_CompressWeightsWksp.cs new file mode 100644 index 00000000..789f4e7f --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_CompressWeightsWksp.cs @@ -0,0 +1,9 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct HUF_CompressWeightsWksp +{ + public fixed uint CTable[59]; + public fixed uint scratchBuffer[41]; + public fixed uint count[13]; + public fixed short norm[13]; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_DEltX1.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_DEltX1.cs new file mode 100644 index 00000000..94d25d9d --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_DEltX1.cs @@ -0,0 +1,11 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/*-***************************/ +/* single-symbol decoding */ +/*-***************************/ +public struct HUF_DEltX1 +{ + /* single-symbol decoding */ + public byte nbBits; + public byte @byte; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_DEltX2.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_DEltX2.cs new file mode 100644 index 00000000..0ab3f1d4 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_DEltX2.cs @@ -0,0 +1,12 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/* *************************/ +/* double-symbols decoding */ +/* *************************/ +public struct HUF_DEltX2 +{ + /* double-symbols decoding */ + public ushort sequence; + public byte nbBits; + public byte length; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_DecompressFastArgs.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_DecompressFastArgs.cs new file mode 100644 index 00000000..c9b8d2d5 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_DecompressFastArgs.cs @@ -0,0 +1,49 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/** + * The input/output arguments to the Huffman fast decoding loop: + * + * ip [in/out] - The input pointers, must be updated to reflect what is consumed. + * op [in/out] - The output pointers, must be updated to reflect what is written. + * bits [in/out] - The bitstream containers, must be updated to reflect the current state. + * dt [in] - The decoding table. + * ilowest [in] - The beginning of the valid range of the input. Decoders may read + * down to this pointer. It may be below iend[0]. + * oend [in] - The end of the output stream. op[3] must not cross oend. + * iend [in] - The end of each input stream. ip[i] may cross iend[i], + * as long as it is above ilowest, but that indicates corruption. + */ +public unsafe struct HUF_DecompressFastArgs +{ + public _ip_e__FixedBuffer ip; + public _op_e__FixedBuffer op; + public fixed ulong bits[4]; + public void* dt; + public byte* ilowest; + public byte* oend; + public _iend_e__FixedBuffer iend; + + public unsafe struct _ip_e__FixedBuffer + { + public byte* e0; + public byte* e1; + public byte* e2; + public byte* e3; + } + + public unsafe struct _op_e__FixedBuffer + { + public byte* e0; + public byte* e1; + public byte* e2; + public byte* e3; + } + + public unsafe struct _iend_e__FixedBuffer + { + public byte* e0; + public byte* e1; + public byte* e2; + public byte* e3; + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_ReadDTableX1_Workspace.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_ReadDTableX1_Workspace.cs new file mode 100644 index 00000000..529a3e5e --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_ReadDTableX1_Workspace.cs @@ -0,0 +1,10 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct HUF_ReadDTableX1_Workspace +{ + public fixed uint rankVal[13]; + public fixed uint rankStart[13]; + public fixed uint statsWksp[219]; + public fixed byte symbols[256]; + public fixed byte huffWeight[256]; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_ReadDTableX2_Workspace.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_ReadDTableX2_Workspace.cs new file mode 100644 index 00000000..a85bbb6c --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_ReadDTableX2_Workspace.cs @@ -0,0 +1,307 @@ +using System.Runtime.CompilerServices; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct HUF_ReadDTableX2_Workspace +{ + public _rankVal_e__FixedBuffer rankVal; + public fixed uint rankStats[13]; + public fixed uint rankStart0[15]; + public _sortedSymbol_e__FixedBuffer sortedSymbol; + public fixed byte weightList[256]; + public fixed uint calleeWksp[219]; + +#if NET8_0_OR_GREATER + [InlineArray(12)] + public unsafe struct _rankVal_e__FixedBuffer + { + public rankValCol_t e0; + } + +#else + public unsafe struct _rankVal_e__FixedBuffer + { + public rankValCol_t e0; + public rankValCol_t e1; + public rankValCol_t e2; + public rankValCol_t e3; + public rankValCol_t e4; + public rankValCol_t e5; + public rankValCol_t e6; + public rankValCol_t e7; + public rankValCol_t e8; + public rankValCol_t e9; + public rankValCol_t e10; + public rankValCol_t e11; + } +#endif + +#if NET8_0_OR_GREATER + [InlineArray(256)] + public unsafe struct _sortedSymbol_e__FixedBuffer + { + public sortedSymbol_t e0; + } + +#else + public unsafe struct _sortedSymbol_e__FixedBuffer + { + public sortedSymbol_t e0; + public sortedSymbol_t e1; + public sortedSymbol_t e2; + public sortedSymbol_t e3; + public sortedSymbol_t e4; + public sortedSymbol_t e5; + public sortedSymbol_t e6; + public sortedSymbol_t e7; + public sortedSymbol_t e8; + public sortedSymbol_t e9; + public sortedSymbol_t e10; + public sortedSymbol_t e11; + public sortedSymbol_t e12; + public sortedSymbol_t e13; + public sortedSymbol_t e14; + public sortedSymbol_t e15; + public sortedSymbol_t e16; + public sortedSymbol_t e17; + public sortedSymbol_t e18; + public sortedSymbol_t e19; + public sortedSymbol_t e20; + public sortedSymbol_t e21; + public sortedSymbol_t e22; + public sortedSymbol_t e23; + public sortedSymbol_t e24; + public sortedSymbol_t e25; + public sortedSymbol_t e26; + public sortedSymbol_t e27; + public sortedSymbol_t e28; + public sortedSymbol_t e29; + public sortedSymbol_t e30; + public sortedSymbol_t e31; + public sortedSymbol_t e32; + public sortedSymbol_t e33; + public sortedSymbol_t e34; + public sortedSymbol_t e35; + public sortedSymbol_t e36; + public sortedSymbol_t e37; + public sortedSymbol_t e38; + public sortedSymbol_t e39; + public sortedSymbol_t e40; + public sortedSymbol_t e41; + public sortedSymbol_t e42; + public sortedSymbol_t e43; + public sortedSymbol_t e44; + public sortedSymbol_t e45; + public sortedSymbol_t e46; + public sortedSymbol_t e47; + public sortedSymbol_t e48; + public sortedSymbol_t e49; + public sortedSymbol_t e50; + public sortedSymbol_t e51; + public sortedSymbol_t e52; + public sortedSymbol_t e53; + public sortedSymbol_t e54; + public sortedSymbol_t e55; + public sortedSymbol_t e56; + public sortedSymbol_t e57; + public sortedSymbol_t e58; + public sortedSymbol_t e59; + public sortedSymbol_t e60; + public sortedSymbol_t e61; + public sortedSymbol_t e62; + public sortedSymbol_t e63; + public sortedSymbol_t e64; + public sortedSymbol_t e65; + public sortedSymbol_t e66; + public sortedSymbol_t e67; + public sortedSymbol_t e68; + public sortedSymbol_t e69; + public sortedSymbol_t e70; + public sortedSymbol_t e71; + public sortedSymbol_t e72; + public sortedSymbol_t e73; + public sortedSymbol_t e74; + public sortedSymbol_t e75; + public sortedSymbol_t e76; + public sortedSymbol_t e77; + public sortedSymbol_t e78; + public sortedSymbol_t e79; + public sortedSymbol_t e80; + public sortedSymbol_t e81; + public sortedSymbol_t e82; + public sortedSymbol_t e83; + public sortedSymbol_t e84; + public sortedSymbol_t e85; + public sortedSymbol_t e86; + public sortedSymbol_t e87; + public sortedSymbol_t e88; + public sortedSymbol_t e89; + public sortedSymbol_t e90; + public sortedSymbol_t e91; + public sortedSymbol_t e92; + public sortedSymbol_t e93; + public sortedSymbol_t e94; + public sortedSymbol_t e95; + public sortedSymbol_t e96; + public sortedSymbol_t e97; + public sortedSymbol_t e98; + public sortedSymbol_t e99; + public sortedSymbol_t e100; + public sortedSymbol_t e101; + public sortedSymbol_t e102; + public sortedSymbol_t e103; + public sortedSymbol_t e104; + public sortedSymbol_t e105; + public sortedSymbol_t e106; + public sortedSymbol_t e107; + public sortedSymbol_t e108; + public sortedSymbol_t e109; + public sortedSymbol_t e110; + public sortedSymbol_t e111; + public sortedSymbol_t e112; + public sortedSymbol_t e113; + public sortedSymbol_t e114; + public sortedSymbol_t e115; + public sortedSymbol_t e116; + public sortedSymbol_t e117; + public sortedSymbol_t e118; + public sortedSymbol_t e119; + public sortedSymbol_t e120; + public sortedSymbol_t e121; + public sortedSymbol_t e122; + public sortedSymbol_t e123; + public sortedSymbol_t e124; + public sortedSymbol_t e125; + public sortedSymbol_t e126; + public sortedSymbol_t e127; + public sortedSymbol_t e128; + public sortedSymbol_t e129; + public sortedSymbol_t e130; + public sortedSymbol_t e131; + public sortedSymbol_t e132; + public sortedSymbol_t e133; + public sortedSymbol_t e134; + public sortedSymbol_t e135; + public sortedSymbol_t e136; + public sortedSymbol_t e137; + public sortedSymbol_t e138; + public sortedSymbol_t e139; + public sortedSymbol_t e140; + public sortedSymbol_t e141; + public sortedSymbol_t e142; + public sortedSymbol_t e143; + public sortedSymbol_t e144; + public sortedSymbol_t e145; + public sortedSymbol_t e146; + public sortedSymbol_t e147; + public sortedSymbol_t e148; + public sortedSymbol_t e149; + public sortedSymbol_t e150; + public sortedSymbol_t e151; + public sortedSymbol_t e152; + public sortedSymbol_t e153; + public sortedSymbol_t e154; + public sortedSymbol_t e155; + public sortedSymbol_t e156; + public sortedSymbol_t e157; + public sortedSymbol_t e158; + public sortedSymbol_t e159; + public sortedSymbol_t e160; + public sortedSymbol_t e161; + public sortedSymbol_t e162; + public sortedSymbol_t e163; + public sortedSymbol_t e164; + public sortedSymbol_t e165; + public sortedSymbol_t e166; + public sortedSymbol_t e167; + public sortedSymbol_t e168; + public sortedSymbol_t e169; + public sortedSymbol_t e170; + public sortedSymbol_t e171; + public sortedSymbol_t e172; + public sortedSymbol_t e173; + public sortedSymbol_t e174; + public sortedSymbol_t e175; + public sortedSymbol_t e176; + public sortedSymbol_t e177; + public sortedSymbol_t e178; + public sortedSymbol_t e179; + public sortedSymbol_t e180; + public sortedSymbol_t e181; + public sortedSymbol_t e182; + public sortedSymbol_t e183; + public sortedSymbol_t e184; + public sortedSymbol_t e185; + public sortedSymbol_t e186; + public sortedSymbol_t e187; + public sortedSymbol_t e188; + public sortedSymbol_t e189; + public sortedSymbol_t e190; + public sortedSymbol_t e191; + public sortedSymbol_t e192; + public sortedSymbol_t e193; + public sortedSymbol_t e194; + public sortedSymbol_t e195; + public sortedSymbol_t e196; + public sortedSymbol_t e197; + public sortedSymbol_t e198; + public sortedSymbol_t e199; + public sortedSymbol_t e200; + public sortedSymbol_t e201; + public sortedSymbol_t e202; + public sortedSymbol_t e203; + public sortedSymbol_t e204; + public sortedSymbol_t e205; + public sortedSymbol_t e206; + public sortedSymbol_t e207; + public sortedSymbol_t e208; + public sortedSymbol_t e209; + public sortedSymbol_t e210; + public sortedSymbol_t e211; + public sortedSymbol_t e212; + public sortedSymbol_t e213; + public sortedSymbol_t e214; + public sortedSymbol_t e215; + public sortedSymbol_t e216; + public sortedSymbol_t e217; + public sortedSymbol_t e218; + public sortedSymbol_t e219; + public sortedSymbol_t e220; + public sortedSymbol_t e221; + public sortedSymbol_t e222; + public sortedSymbol_t e223; + public sortedSymbol_t e224; + public sortedSymbol_t e225; + public sortedSymbol_t e226; + public sortedSymbol_t e227; + public sortedSymbol_t e228; + public sortedSymbol_t e229; + public sortedSymbol_t e230; + public sortedSymbol_t e231; + public sortedSymbol_t e232; + public sortedSymbol_t e233; + public sortedSymbol_t e234; + public sortedSymbol_t e235; + public sortedSymbol_t e236; + public sortedSymbol_t e237; + public sortedSymbol_t e238; + public sortedSymbol_t e239; + public sortedSymbol_t e240; + public sortedSymbol_t e241; + public sortedSymbol_t e242; + public sortedSymbol_t e243; + public sortedSymbol_t e244; + public sortedSymbol_t e245; + public sortedSymbol_t e246; + public sortedSymbol_t e247; + public sortedSymbol_t e248; + public sortedSymbol_t e249; + public sortedSymbol_t e250; + public sortedSymbol_t e251; + public sortedSymbol_t e252; + public sortedSymbol_t e253; + public sortedSymbol_t e254; + public sortedSymbol_t e255; + } +#endif +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_WriteCTableWksp.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_WriteCTableWksp.cs new file mode 100644 index 00000000..2400dba7 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_WriteCTableWksp.cs @@ -0,0 +1,10 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct HUF_WriteCTableWksp +{ + public HUF_CompressWeightsWksp wksp; + + /* precomputed conversion table */ + public fixed byte bitsToWeight[13]; + public fixed byte huffWeight[255]; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_buildCTable_wksp_tables.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_buildCTable_wksp_tables.cs new file mode 100644 index 00000000..4fe92db7 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_buildCTable_wksp_tables.cs @@ -0,0 +1,739 @@ +using System.Runtime.CompilerServices; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct HUF_buildCTable_wksp_tables +{ + public _huffNodeTbl_e__FixedBuffer huffNodeTbl; + public _rankPosition_e__FixedBuffer rankPosition; + +#if NET8_0_OR_GREATER + [InlineArray(512)] + public unsafe struct _huffNodeTbl_e__FixedBuffer + { + public nodeElt_s e0; + } + +#else + public unsafe struct _huffNodeTbl_e__FixedBuffer + { + public nodeElt_s e0; + public nodeElt_s e1; + public nodeElt_s e2; + public nodeElt_s e3; + public nodeElt_s e4; + public nodeElt_s e5; + public nodeElt_s e6; + public nodeElt_s e7; + public nodeElt_s e8; + public nodeElt_s e9; + public nodeElt_s e10; + public nodeElt_s e11; + public nodeElt_s e12; + public nodeElt_s e13; + public nodeElt_s e14; + public nodeElt_s e15; + public nodeElt_s e16; + public nodeElt_s e17; + public nodeElt_s e18; + public nodeElt_s e19; + public nodeElt_s e20; + public nodeElt_s e21; + public nodeElt_s e22; + public nodeElt_s e23; + public nodeElt_s e24; + public nodeElt_s e25; + public nodeElt_s e26; + public nodeElt_s e27; + public nodeElt_s e28; + public nodeElt_s e29; + public nodeElt_s e30; + public nodeElt_s e31; + public nodeElt_s e32; + public nodeElt_s e33; + public nodeElt_s e34; + public nodeElt_s e35; + public nodeElt_s e36; + public nodeElt_s e37; + public nodeElt_s e38; + public nodeElt_s e39; + public nodeElt_s e40; + public nodeElt_s e41; + public nodeElt_s e42; + public nodeElt_s e43; + public nodeElt_s e44; + public nodeElt_s e45; + public nodeElt_s e46; + public nodeElt_s e47; + public nodeElt_s e48; + public nodeElt_s e49; + public nodeElt_s e50; + public nodeElt_s e51; + public nodeElt_s e52; + public nodeElt_s e53; + public nodeElt_s e54; + public nodeElt_s e55; + public nodeElt_s e56; + public nodeElt_s e57; + public nodeElt_s e58; + public nodeElt_s e59; + public nodeElt_s e60; + public nodeElt_s e61; + public nodeElt_s e62; + public nodeElt_s e63; + public nodeElt_s e64; + public nodeElt_s e65; + public nodeElt_s e66; + public nodeElt_s e67; + public nodeElt_s e68; + public nodeElt_s e69; + public nodeElt_s e70; + public nodeElt_s e71; + public nodeElt_s e72; + public nodeElt_s e73; + public nodeElt_s e74; + public nodeElt_s e75; + public nodeElt_s e76; + public nodeElt_s e77; + public nodeElt_s e78; + public nodeElt_s e79; + public nodeElt_s e80; + public nodeElt_s e81; + public nodeElt_s e82; + public nodeElt_s e83; + public nodeElt_s e84; + public nodeElt_s e85; + public nodeElt_s e86; + public nodeElt_s e87; + public nodeElt_s e88; + public nodeElt_s e89; + public nodeElt_s e90; + public nodeElt_s e91; + public nodeElt_s e92; + public nodeElt_s e93; + public nodeElt_s e94; + public nodeElt_s e95; + public nodeElt_s e96; + public nodeElt_s e97; + public nodeElt_s e98; + public nodeElt_s e99; + public nodeElt_s e100; + public nodeElt_s e101; + public nodeElt_s e102; + public nodeElt_s e103; + public nodeElt_s e104; + public nodeElt_s e105; + public nodeElt_s e106; + public nodeElt_s e107; + public nodeElt_s e108; + public nodeElt_s e109; + public nodeElt_s e110; + public nodeElt_s e111; + public nodeElt_s e112; + public nodeElt_s e113; + public nodeElt_s e114; + public nodeElt_s e115; + public nodeElt_s e116; + public nodeElt_s e117; + public nodeElt_s e118; + public nodeElt_s e119; + public nodeElt_s e120; + public nodeElt_s e121; + public nodeElt_s e122; + public nodeElt_s e123; + public nodeElt_s e124; + public nodeElt_s e125; + public nodeElt_s e126; + public nodeElt_s e127; + public nodeElt_s e128; + public nodeElt_s e129; + public nodeElt_s e130; + public nodeElt_s e131; + public nodeElt_s e132; + public nodeElt_s e133; + public nodeElt_s e134; + public nodeElt_s e135; + public nodeElt_s e136; + public nodeElt_s e137; + public nodeElt_s e138; + public nodeElt_s e139; + public nodeElt_s e140; + public nodeElt_s e141; + public nodeElt_s e142; + public nodeElt_s e143; + public nodeElt_s e144; + public nodeElt_s e145; + public nodeElt_s e146; + public nodeElt_s e147; + public nodeElt_s e148; + public nodeElt_s e149; + public nodeElt_s e150; + public nodeElt_s e151; + public nodeElt_s e152; + public nodeElt_s e153; + public nodeElt_s e154; + public nodeElt_s e155; + public nodeElt_s e156; + public nodeElt_s e157; + public nodeElt_s e158; + public nodeElt_s e159; + public nodeElt_s e160; + public nodeElt_s e161; + public nodeElt_s e162; + public nodeElt_s e163; + public nodeElt_s e164; + public nodeElt_s e165; + public nodeElt_s e166; + public nodeElt_s e167; + public nodeElt_s e168; + public nodeElt_s e169; + public nodeElt_s e170; + public nodeElt_s e171; + public nodeElt_s e172; + public nodeElt_s e173; + public nodeElt_s e174; + public nodeElt_s e175; + public nodeElt_s e176; + public nodeElt_s e177; + public nodeElt_s e178; + public nodeElt_s e179; + public nodeElt_s e180; + public nodeElt_s e181; + public nodeElt_s e182; + public nodeElt_s e183; + public nodeElt_s e184; + public nodeElt_s e185; + public nodeElt_s e186; + public nodeElt_s e187; + public nodeElt_s e188; + public nodeElt_s e189; + public nodeElt_s e190; + public nodeElt_s e191; + public nodeElt_s e192; + public nodeElt_s e193; + public nodeElt_s e194; + public nodeElt_s e195; + public nodeElt_s e196; + public nodeElt_s e197; + public nodeElt_s e198; + public nodeElt_s e199; + public nodeElt_s e200; + public nodeElt_s e201; + public nodeElt_s e202; + public nodeElt_s e203; + public nodeElt_s e204; + public nodeElt_s e205; + public nodeElt_s e206; + public nodeElt_s e207; + public nodeElt_s e208; + public nodeElt_s e209; + public nodeElt_s e210; + public nodeElt_s e211; + public nodeElt_s e212; + public nodeElt_s e213; + public nodeElt_s e214; + public nodeElt_s e215; + public nodeElt_s e216; + public nodeElt_s e217; + public nodeElt_s e218; + public nodeElt_s e219; + public nodeElt_s e220; + public nodeElt_s e221; + public nodeElt_s e222; + public nodeElt_s e223; + public nodeElt_s e224; + public nodeElt_s e225; + public nodeElt_s e226; + public nodeElt_s e227; + public nodeElt_s e228; + public nodeElt_s e229; + public nodeElt_s e230; + public nodeElt_s e231; + public nodeElt_s e232; + public nodeElt_s e233; + public nodeElt_s e234; + public nodeElt_s e235; + public nodeElt_s e236; + public nodeElt_s e237; + public nodeElt_s e238; + public nodeElt_s e239; + public nodeElt_s e240; + public nodeElt_s e241; + public nodeElt_s e242; + public nodeElt_s e243; + public nodeElt_s e244; + public nodeElt_s e245; + public nodeElt_s e246; + public nodeElt_s e247; + public nodeElt_s e248; + public nodeElt_s e249; + public nodeElt_s e250; + public nodeElt_s e251; + public nodeElt_s e252; + public nodeElt_s e253; + public nodeElt_s e254; + public nodeElt_s e255; + public nodeElt_s e256; + public nodeElt_s e257; + public nodeElt_s e258; + public nodeElt_s e259; + public nodeElt_s e260; + public nodeElt_s e261; + public nodeElt_s e262; + public nodeElt_s e263; + public nodeElt_s e264; + public nodeElt_s e265; + public nodeElt_s e266; + public nodeElt_s e267; + public nodeElt_s e268; + public nodeElt_s e269; + public nodeElt_s e270; + public nodeElt_s e271; + public nodeElt_s e272; + public nodeElt_s e273; + public nodeElt_s e274; + public nodeElt_s e275; + public nodeElt_s e276; + public nodeElt_s e277; + public nodeElt_s e278; + public nodeElt_s e279; + public nodeElt_s e280; + public nodeElt_s e281; + public nodeElt_s e282; + public nodeElt_s e283; + public nodeElt_s e284; + public nodeElt_s e285; + public nodeElt_s e286; + public nodeElt_s e287; + public nodeElt_s e288; + public nodeElt_s e289; + public nodeElt_s e290; + public nodeElt_s e291; + public nodeElt_s e292; + public nodeElt_s e293; + public nodeElt_s e294; + public nodeElt_s e295; + public nodeElt_s e296; + public nodeElt_s e297; + public nodeElt_s e298; + public nodeElt_s e299; + public nodeElt_s e300; + public nodeElt_s e301; + public nodeElt_s e302; + public nodeElt_s e303; + public nodeElt_s e304; + public nodeElt_s e305; + public nodeElt_s e306; + public nodeElt_s e307; + public nodeElt_s e308; + public nodeElt_s e309; + public nodeElt_s e310; + public nodeElt_s e311; + public nodeElt_s e312; + public nodeElt_s e313; + public nodeElt_s e314; + public nodeElt_s e315; + public nodeElt_s e316; + public nodeElt_s e317; + public nodeElt_s e318; + public nodeElt_s e319; + public nodeElt_s e320; + public nodeElt_s e321; + public nodeElt_s e322; + public nodeElt_s e323; + public nodeElt_s e324; + public nodeElt_s e325; + public nodeElt_s e326; + public nodeElt_s e327; + public nodeElt_s e328; + public nodeElt_s e329; + public nodeElt_s e330; + public nodeElt_s e331; + public nodeElt_s e332; + public nodeElt_s e333; + public nodeElt_s e334; + public nodeElt_s e335; + public nodeElt_s e336; + public nodeElt_s e337; + public nodeElt_s e338; + public nodeElt_s e339; + public nodeElt_s e340; + public nodeElt_s e341; + public nodeElt_s e342; + public nodeElt_s e343; + public nodeElt_s e344; + public nodeElt_s e345; + public nodeElt_s e346; + public nodeElt_s e347; + public nodeElt_s e348; + public nodeElt_s e349; + public nodeElt_s e350; + public nodeElt_s e351; + public nodeElt_s e352; + public nodeElt_s e353; + public nodeElt_s e354; + public nodeElt_s e355; + public nodeElt_s e356; + public nodeElt_s e357; + public nodeElt_s e358; + public nodeElt_s e359; + public nodeElt_s e360; + public nodeElt_s e361; + public nodeElt_s e362; + public nodeElt_s e363; + public nodeElt_s e364; + public nodeElt_s e365; + public nodeElt_s e366; + public nodeElt_s e367; + public nodeElt_s e368; + public nodeElt_s e369; + public nodeElt_s e370; + public nodeElt_s e371; + public nodeElt_s e372; + public nodeElt_s e373; + public nodeElt_s e374; + public nodeElt_s e375; + public nodeElt_s e376; + public nodeElt_s e377; + public nodeElt_s e378; + public nodeElt_s e379; + public nodeElt_s e380; + public nodeElt_s e381; + public nodeElt_s e382; + public nodeElt_s e383; + public nodeElt_s e384; + public nodeElt_s e385; + public nodeElt_s e386; + public nodeElt_s e387; + public nodeElt_s e388; + public nodeElt_s e389; + public nodeElt_s e390; + public nodeElt_s e391; + public nodeElt_s e392; + public nodeElt_s e393; + public nodeElt_s e394; + public nodeElt_s e395; + public nodeElt_s e396; + public nodeElt_s e397; + public nodeElt_s e398; + public nodeElt_s e399; + public nodeElt_s e400; + public nodeElt_s e401; + public nodeElt_s e402; + public nodeElt_s e403; + public nodeElt_s e404; + public nodeElt_s e405; + public nodeElt_s e406; + public nodeElt_s e407; + public nodeElt_s e408; + public nodeElt_s e409; + public nodeElt_s e410; + public nodeElt_s e411; + public nodeElt_s e412; + public nodeElt_s e413; + public nodeElt_s e414; + public nodeElt_s e415; + public nodeElt_s e416; + public nodeElt_s e417; + public nodeElt_s e418; + public nodeElt_s e419; + public nodeElt_s e420; + public nodeElt_s e421; + public nodeElt_s e422; + public nodeElt_s e423; + public nodeElt_s e424; + public nodeElt_s e425; + public nodeElt_s e426; + public nodeElt_s e427; + public nodeElt_s e428; + public nodeElt_s e429; + public nodeElt_s e430; + public nodeElt_s e431; + public nodeElt_s e432; + public nodeElt_s e433; + public nodeElt_s e434; + public nodeElt_s e435; + public nodeElt_s e436; + public nodeElt_s e437; + public nodeElt_s e438; + public nodeElt_s e439; + public nodeElt_s e440; + public nodeElt_s e441; + public nodeElt_s e442; + public nodeElt_s e443; + public nodeElt_s e444; + public nodeElt_s e445; + public nodeElt_s e446; + public nodeElt_s e447; + public nodeElt_s e448; + public nodeElt_s e449; + public nodeElt_s e450; + public nodeElt_s e451; + public nodeElt_s e452; + public nodeElt_s e453; + public nodeElt_s e454; + public nodeElt_s e455; + public nodeElt_s e456; + public nodeElt_s e457; + public nodeElt_s e458; + public nodeElt_s e459; + public nodeElt_s e460; + public nodeElt_s e461; + public nodeElt_s e462; + public nodeElt_s e463; + public nodeElt_s e464; + public nodeElt_s e465; + public nodeElt_s e466; + public nodeElt_s e467; + public nodeElt_s e468; + public nodeElt_s e469; + public nodeElt_s e470; + public nodeElt_s e471; + public nodeElt_s e472; + public nodeElt_s e473; + public nodeElt_s e474; + public nodeElt_s e475; + public nodeElt_s e476; + public nodeElt_s e477; + public nodeElt_s e478; + public nodeElt_s e479; + public nodeElt_s e480; + public nodeElt_s e481; + public nodeElt_s e482; + public nodeElt_s e483; + public nodeElt_s e484; + public nodeElt_s e485; + public nodeElt_s e486; + public nodeElt_s e487; + public nodeElt_s e488; + public nodeElt_s e489; + public nodeElt_s e490; + public nodeElt_s e491; + public nodeElt_s e492; + public nodeElt_s e493; + public nodeElt_s e494; + public nodeElt_s e495; + public nodeElt_s e496; + public nodeElt_s e497; + public nodeElt_s e498; + public nodeElt_s e499; + public nodeElt_s e500; + public nodeElt_s e501; + public nodeElt_s e502; + public nodeElt_s e503; + public nodeElt_s e504; + public nodeElt_s e505; + public nodeElt_s e506; + public nodeElt_s e507; + public nodeElt_s e508; + public nodeElt_s e509; + public nodeElt_s e510; + public nodeElt_s e511; + } +#endif + +#if NET8_0_OR_GREATER + [InlineArray(192)] + public unsafe struct _rankPosition_e__FixedBuffer + { + public rankPos e0; + } + +#else + public unsafe struct _rankPosition_e__FixedBuffer + { + public rankPos e0; + public rankPos e1; + public rankPos e2; + public rankPos e3; + public rankPos e4; + public rankPos e5; + public rankPos e6; + public rankPos e7; + public rankPos e8; + public rankPos e9; + public rankPos e10; + public rankPos e11; + public rankPos e12; + public rankPos e13; + public rankPos e14; + public rankPos e15; + public rankPos e16; + public rankPos e17; + public rankPos e18; + public rankPos e19; + public rankPos e20; + public rankPos e21; + public rankPos e22; + public rankPos e23; + public rankPos e24; + public rankPos e25; + public rankPos e26; + public rankPos e27; + public rankPos e28; + public rankPos e29; + public rankPos e30; + public rankPos e31; + public rankPos e32; + public rankPos e33; + public rankPos e34; + public rankPos e35; + public rankPos e36; + public rankPos e37; + public rankPos e38; + public rankPos e39; + public rankPos e40; + public rankPos e41; + public rankPos e42; + public rankPos e43; + public rankPos e44; + public rankPos e45; + public rankPos e46; + public rankPos e47; + public rankPos e48; + public rankPos e49; + public rankPos e50; + public rankPos e51; + public rankPos e52; + public rankPos e53; + public rankPos e54; + public rankPos e55; + public rankPos e56; + public rankPos e57; + public rankPos e58; + public rankPos e59; + public rankPos e60; + public rankPos e61; + public rankPos e62; + public rankPos e63; + public rankPos e64; + public rankPos e65; + public rankPos e66; + public rankPos e67; + public rankPos e68; + public rankPos e69; + public rankPos e70; + public rankPos e71; + public rankPos e72; + public rankPos e73; + public rankPos e74; + public rankPos e75; + public rankPos e76; + public rankPos e77; + public rankPos e78; + public rankPos e79; + public rankPos e80; + public rankPos e81; + public rankPos e82; + public rankPos e83; + public rankPos e84; + public rankPos e85; + public rankPos e86; + public rankPos e87; + public rankPos e88; + public rankPos e89; + public rankPos e90; + public rankPos e91; + public rankPos e92; + public rankPos e93; + public rankPos e94; + public rankPos e95; + public rankPos e96; + public rankPos e97; + public rankPos e98; + public rankPos e99; + public rankPos e100; + public rankPos e101; + public rankPos e102; + public rankPos e103; + public rankPos e104; + public rankPos e105; + public rankPos e106; + public rankPos e107; + public rankPos e108; + public rankPos e109; + public rankPos e110; + public rankPos e111; + public rankPos e112; + public rankPos e113; + public rankPos e114; + public rankPos e115; + public rankPos e116; + public rankPos e117; + public rankPos e118; + public rankPos e119; + public rankPos e120; + public rankPos e121; + public rankPos e122; + public rankPos e123; + public rankPos e124; + public rankPos e125; + public rankPos e126; + public rankPos e127; + public rankPos e128; + public rankPos e129; + public rankPos e130; + public rankPos e131; + public rankPos e132; + public rankPos e133; + public rankPos e134; + public rankPos e135; + public rankPos e136; + public rankPos e137; + public rankPos e138; + public rankPos e139; + public rankPos e140; + public rankPos e141; + public rankPos e142; + public rankPos e143; + public rankPos e144; + public rankPos e145; + public rankPos e146; + public rankPos e147; + public rankPos e148; + public rankPos e149; + public rankPos e150; + public rankPos e151; + public rankPos e152; + public rankPos e153; + public rankPos e154; + public rankPos e155; + public rankPos e156; + public rankPos e157; + public rankPos e158; + public rankPos e159; + public rankPos e160; + public rankPos e161; + public rankPos e162; + public rankPos e163; + public rankPos e164; + public rankPos e165; + public rankPos e166; + public rankPos e167; + public rankPos e168; + public rankPos e169; + public rankPos e170; + public rankPos e171; + public rankPos e172; + public rankPos e173; + public rankPos e174; + public rankPos e175; + public rankPos e176; + public rankPos e177; + public rankPos e178; + public rankPos e179; + public rankPos e180; + public rankPos e181; + public rankPos e182; + public rankPos e183; + public rankPos e184; + public rankPos e185; + public rankPos e186; + public rankPos e187; + public rankPos e188; + public rankPos e189; + public rankPos e190; + public rankPos e191; + } +#endif +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_compress_tables_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_compress_tables_t.cs new file mode 100644 index 00000000..6e6e12c0 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_compress_tables_t.cs @@ -0,0 +1,280 @@ +using System.Runtime.CompilerServices; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct HUF_compress_tables_t +{ + public fixed uint count[256]; + public _CTable_e__FixedBuffer CTable; + public _wksps_e__Union wksps; + +#if NET8_0_OR_GREATER + [InlineArray(257)] + public unsafe struct _CTable_e__FixedBuffer + { + public nuint e0; + } + +#else + public unsafe struct _CTable_e__FixedBuffer + { + public nuint e0; + public nuint e1; + public nuint e2; + public nuint e3; + public nuint e4; + public nuint e5; + public nuint e6; + public nuint e7; + public nuint e8; + public nuint e9; + public nuint e10; + public nuint e11; + public nuint e12; + public nuint e13; + public nuint e14; + public nuint e15; + public nuint e16; + public nuint e17; + public nuint e18; + public nuint e19; + public nuint e20; + public nuint e21; + public nuint e22; + public nuint e23; + public nuint e24; + public nuint e25; + public nuint e26; + public nuint e27; + public nuint e28; + public nuint e29; + public nuint e30; + public nuint e31; + public nuint e32; + public nuint e33; + public nuint e34; + public nuint e35; + public nuint e36; + public nuint e37; + public nuint e38; + public nuint e39; + public nuint e40; + public nuint e41; + public nuint e42; + public nuint e43; + public nuint e44; + public nuint e45; + public nuint e46; + public nuint e47; + public nuint e48; + public nuint e49; + public nuint e50; + public nuint e51; + public nuint e52; + public nuint e53; + public nuint e54; + public nuint e55; + public nuint e56; + public nuint e57; + public nuint e58; + public nuint e59; + public nuint e60; + public nuint e61; + public nuint e62; + public nuint e63; + public nuint e64; + public nuint e65; + public nuint e66; + public nuint e67; + public nuint e68; + public nuint e69; + public nuint e70; + public nuint e71; + public nuint e72; + public nuint e73; + public nuint e74; + public nuint e75; + public nuint e76; + public nuint e77; + public nuint e78; + public nuint e79; + public nuint e80; + public nuint e81; + public nuint e82; + public nuint e83; + public nuint e84; + public nuint e85; + public nuint e86; + public nuint e87; + public nuint e88; + public nuint e89; + public nuint e90; + public nuint e91; + public nuint e92; + public nuint e93; + public nuint e94; + public nuint e95; + public nuint e96; + public nuint e97; + public nuint e98; + public nuint e99; + public nuint e100; + public nuint e101; + public nuint e102; + public nuint e103; + public nuint e104; + public nuint e105; + public nuint e106; + public nuint e107; + public nuint e108; + public nuint e109; + public nuint e110; + public nuint e111; + public nuint e112; + public nuint e113; + public nuint e114; + public nuint e115; + public nuint e116; + public nuint e117; + public nuint e118; + public nuint e119; + public nuint e120; + public nuint e121; + public nuint e122; + public nuint e123; + public nuint e124; + public nuint e125; + public nuint e126; + public nuint e127; + public nuint e128; + public nuint e129; + public nuint e130; + public nuint e131; + public nuint e132; + public nuint e133; + public nuint e134; + public nuint e135; + public nuint e136; + public nuint e137; + public nuint e138; + public nuint e139; + public nuint e140; + public nuint e141; + public nuint e142; + public nuint e143; + public nuint e144; + public nuint e145; + public nuint e146; + public nuint e147; + public nuint e148; + public nuint e149; + public nuint e150; + public nuint e151; + public nuint e152; + public nuint e153; + public nuint e154; + public nuint e155; + public nuint e156; + public nuint e157; + public nuint e158; + public nuint e159; + public nuint e160; + public nuint e161; + public nuint e162; + public nuint e163; + public nuint e164; + public nuint e165; + public nuint e166; + public nuint e167; + public nuint e168; + public nuint e169; + public nuint e170; + public nuint e171; + public nuint e172; + public nuint e173; + public nuint e174; + public nuint e175; + public nuint e176; + public nuint e177; + public nuint e178; + public nuint e179; + public nuint e180; + public nuint e181; + public nuint e182; + public nuint e183; + public nuint e184; + public nuint e185; + public nuint e186; + public nuint e187; + public nuint e188; + public nuint e189; + public nuint e190; + public nuint e191; + public nuint e192; + public nuint e193; + public nuint e194; + public nuint e195; + public nuint e196; + public nuint e197; + public nuint e198; + public nuint e199; + public nuint e200; + public nuint e201; + public nuint e202; + public nuint e203; + public nuint e204; + public nuint e205; + public nuint e206; + public nuint e207; + public nuint e208; + public nuint e209; + public nuint e210; + public nuint e211; + public nuint e212; + public nuint e213; + public nuint e214; + public nuint e215; + public nuint e216; + public nuint e217; + public nuint e218; + public nuint e219; + public nuint e220; + public nuint e221; + public nuint e222; + public nuint e223; + public nuint e224; + public nuint e225; + public nuint e226; + public nuint e227; + public nuint e228; + public nuint e229; + public nuint e230; + public nuint e231; + public nuint e232; + public nuint e233; + public nuint e234; + public nuint e235; + public nuint e236; + public nuint e237; + public nuint e238; + public nuint e239; + public nuint e240; + public nuint e241; + public nuint e242; + public nuint e243; + public nuint e244; + public nuint e245; + public nuint e246; + public nuint e247; + public nuint e248; + public nuint e249; + public nuint e250; + public nuint e251; + public nuint e252; + public nuint e253; + public nuint e254; + public nuint e255; + public nuint e256; + } +#endif +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_flags_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_flags_e.cs new file mode 100644 index 00000000..a38bc711 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_flags_e.cs @@ -0,0 +1,44 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/** + * Huffman flags bitset. + * For all flags, 0 is the default value. + */ +public enum HUF_flags_e +{ + /** + * If compiled with DYNAMIC_BMI2: Set flag only if the CPU supports BMI2 at runtime. + * Otherwise: Ignored. + */ + HUF_flags_bmi2 = 1 << 0, + + /** + * If set: Test possible table depths to find the one that produces the smallest header + encoded size. + * If unset: Use heuristic to find the table depth. + */ + HUF_flags_optimalDepth = 1 << 1, + + /** + * If set: If the previous table can encode the input, always reuse the previous table. + * If unset: If the previous table can encode the input, reuse the previous table if it results in a smaller output. + */ + HUF_flags_preferRepeat = 1 << 2, + + /** + * If set: Sample the input and check if the sample is uncompressible, if it is then don't attempt to compress. + * If unset: Always histogram the entire input. + */ + HUF_flags_suspectUncompressible = 1 << 3, + + /** + * If set: Don't use assembly implementations + * If unset: Allow using assembly implementations + */ + HUF_flags_disableAsm = 1 << 4, + + /** + * If set: Don't use the fast decoding loop, always use the fallback decoding loop. + * If unset: Use the fast decoding loop when possible. + */ + HUF_flags_disableFast = 1 << 5, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_nbStreams_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_nbStreams_e.cs new file mode 100644 index 00000000..ff7e26e7 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_nbStreams_e.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum HUF_nbStreams_e +{ + HUF_singleStream, + HUF_fourStreams, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_repeat.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_repeat.cs new file mode 100644 index 00000000..5ab8b1da --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/HUF_repeat.cs @@ -0,0 +1,13 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum HUF_repeat +{ + /**< Cannot use the previous table */ + HUF_repeat_none, + + /**< Can use the previous table but it must be checked. Note : The previous table must have been constructed by HUF_compress{1, 4}X_repeat */ + HUF_repeat_check, + + /**< Can use the previous table and it is assumed to be valid */ + HUF_repeat_valid, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/Hist.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/Hist.cs new file mode 100644 index 00000000..e9c5be35 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/Hist.cs @@ -0,0 +1,309 @@ +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + /* --- Error management --- */ + private static bool HIST_isError(nuint code) + { + return ERR_isError(code); + } + + /*-************************************************************** + * Histogram functions + ****************************************************************/ + private static void HIST_add(uint* count, void* src, nuint srcSize) + { + byte* ip = (byte*)src; + byte* end = ip + srcSize; + while (ip < end) + { + count[*ip++]++; + } + } + + /*! HIST_count_simple() : + * Same as HIST_countFast(), this function is unsafe, + * and will segfault if any value within `src` is `> *maxSymbolValuePtr`. + * It is also a bit slower for large inputs. + * However, it does not need any additional memory (not even on stack). + * @return : count of the most frequent symbol. + * Note this function doesn't produce any error (i.e. it must succeed). + */ + private static uint HIST_count_simple( + uint* count, + uint* maxSymbolValuePtr, + void* src, + nuint srcSize + ) + { + byte* ip = (byte*)src; + byte* end = ip + srcSize; + uint maxSymbolValue = *maxSymbolValuePtr; + uint largestCount = 0; + memset(count, 0, (maxSymbolValue + 1) * sizeof(uint)); + if (srcSize == 0) + { + *maxSymbolValuePtr = 0; + return 0; + } + + while (ip < end) + { + assert(*ip <= maxSymbolValue); + count[*ip++]++; + } + + while (count[maxSymbolValue] == 0) + { + maxSymbolValue--; + } + + *maxSymbolValuePtr = maxSymbolValue; + { + uint s; + for (s = 0; s <= maxSymbolValue; s++) + { + if (count[s] > largestCount) + { + largestCount = count[s]; + } + } + } + + return largestCount; + } + + /* HIST_count_parallel_wksp() : + * store histogram into 4 intermediate tables, recombined at the end. + * this design makes better use of OoO cpus, + * and is noticeably faster when some values are heavily repeated. + * But it needs some additional workspace for intermediate tables. + * `workSpace` must be a U32 table of size >= HIST_WKSP_SIZE_U32. + * @return : largest histogram frequency, + * or an error code (notably when histogram's alphabet is larger than *maxSymbolValuePtr) */ + private static nuint HIST_count_parallel_wksp( + uint* count, + uint* maxSymbolValuePtr, + void* source, + nuint sourceSize, + HIST_checkInput_e check, + uint* workSpace + ) + { + byte* ip = (byte*)source; + byte* iend = ip + sourceSize; + nuint countSize = (*maxSymbolValuePtr + 1) * sizeof(uint); + uint max = 0; + uint* Counting1 = workSpace; + uint* Counting2 = Counting1 + 256; + uint* Counting3 = Counting2 + 256; + uint* Counting4 = Counting3 + 256; + assert(*maxSymbolValuePtr <= 255); + if (sourceSize == 0) + { + memset(count, 0, (uint)countSize); + *maxSymbolValuePtr = 0; + return 0; + } + + memset(workSpace, 0, 4 * 256 * sizeof(uint)); + { + uint cached = MEM_read32(ip); + ip += 4; + while (ip < iend - 15) + { + uint c = cached; + cached = MEM_read32(ip); + ip += 4; + Counting1[(byte)c]++; + Counting2[(byte)(c >> 8)]++; + Counting3[(byte)(c >> 16)]++; + Counting4[c >> 24]++; + c = cached; + cached = MEM_read32(ip); + ip += 4; + Counting1[(byte)c]++; + Counting2[(byte)(c >> 8)]++; + Counting3[(byte)(c >> 16)]++; + Counting4[c >> 24]++; + c = cached; + cached = MEM_read32(ip); + ip += 4; + Counting1[(byte)c]++; + Counting2[(byte)(c >> 8)]++; + Counting3[(byte)(c >> 16)]++; + Counting4[c >> 24]++; + c = cached; + cached = MEM_read32(ip); + ip += 4; + Counting1[(byte)c]++; + Counting2[(byte)(c >> 8)]++; + Counting3[(byte)(c >> 16)]++; + Counting4[c >> 24]++; + } + + ip -= 4; + } + + while (ip < iend) + { + Counting1[*ip++]++; + } + + { + uint s; + for (s = 0; s < 256; s++) + { + Counting1[s] += Counting2[s] + Counting3[s] + Counting4[s]; + if (Counting1[s] > max) + { + max = Counting1[s]; + } + } + } + + { + uint maxSymbolValue = 255; + while (Counting1[maxSymbolValue] == 0) + { + maxSymbolValue--; + } + + if (check != default && maxSymbolValue > *maxSymbolValuePtr) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_maxSymbolValue_tooSmall)); + } + + *maxSymbolValuePtr = maxSymbolValue; + memmove(count, Counting1, countSize); + } + + return max; + } + + /* HIST_countFast_wksp() : + * Same as HIST_countFast(), but using an externally provided scratch buffer. + * `workSpace` is a writable buffer which must be 4-bytes aligned, + * `workSpaceSize` must be >= HIST_WKSP_SIZE + */ + private static nuint HIST_countFast_wksp( + uint* count, + uint* maxSymbolValuePtr, + void* source, + nuint sourceSize, + void* workSpace, + nuint workSpaceSize + ) + { + if (sourceSize < 1500) + { + return HIST_count_simple(count, maxSymbolValuePtr, source, sourceSize); + } + + if (((nuint)workSpace & 3) != 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + + if (workSpaceSize < 1024 * sizeof(uint)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_workSpace_tooSmall)); + } + + return HIST_count_parallel_wksp( + count, + maxSymbolValuePtr, + source, + sourceSize, + HIST_checkInput_e.trustInput, + (uint*)workSpace + ); + } + + /* HIST_count_wksp() : + * Same as HIST_count(), but using an externally provided scratch buffer. + * `workSpace` size must be table of >= HIST_WKSP_SIZE_U32 unsigned */ + private static nuint HIST_count_wksp( + uint* count, + uint* maxSymbolValuePtr, + void* source, + nuint sourceSize, + void* workSpace, + nuint workSpaceSize + ) + { + if (((nuint)workSpace & 3) != 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + + if (workSpaceSize < 1024 * sizeof(uint)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_workSpace_tooSmall)); + } + + if (*maxSymbolValuePtr < 255) + { + return HIST_count_parallel_wksp( + count, + maxSymbolValuePtr, + source, + sourceSize, + HIST_checkInput_e.checkMaxSymbolValue, + (uint*)workSpace + ); + } + + *maxSymbolValuePtr = 255; + return HIST_countFast_wksp( + count, + maxSymbolValuePtr, + source, + sourceSize, + workSpace, + workSpaceSize + ); + } + + /* fast variant (unsafe : won't check if src contains values beyond count[] limit) */ + private static nuint HIST_countFast( + uint* count, + uint* maxSymbolValuePtr, + void* source, + nuint sourceSize + ) + { + uint* tmpCounters = stackalloc uint[1024]; + return HIST_countFast_wksp( + count, + maxSymbolValuePtr, + source, + sourceSize, + tmpCounters, + sizeof(uint) * 1024 + ); + } + + /*! HIST_count(): + * Provides the precise count of each byte within a table 'count'. + * 'count' is a table of unsigned int, of minimum size (*maxSymbolValuePtr+1). + * Updates *maxSymbolValuePtr with actual largest symbol value detected. + * @return : count of the most frequent symbol (which isn't identified). + * or an error code, which can be tested using HIST_isError(). + * note : if return == srcSize, there is only one symbol. + */ + private static nuint HIST_count(uint* count, uint* maxSymbolValuePtr, void* src, nuint srcSize) + { + uint* tmpCounters = stackalloc uint[1024]; + return HIST_count_wksp( + count, + maxSymbolValuePtr, + src, + srcSize, + tmpCounters, + sizeof(uint) * 1024 + ); + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/HufCompress.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/HufCompress.cs new file mode 100644 index 00000000..1b8234da --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/HufCompress.cs @@ -0,0 +1,2058 @@ +using System.Runtime.CompilerServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + private static void* HUF_alignUpWorkspace(void* workspace, nuint* workspaceSizePtr, nuint align) + { + nuint mask = align - 1; + nuint rem = (nuint)workspace & mask; + nuint add = align - rem & mask; + byte* aligned = (byte*)workspace + add; + assert((align & align - 1) == 0); + assert(align <= 8); + if (*workspaceSizePtr >= add) + { + assert(add < align); + assert(((nuint)aligned & mask) == 0); + *workspaceSizePtr -= add; + return aligned; + } + else + { + *workspaceSizePtr = 0; + return null; + } + } + + private static nuint HUF_compressWeights( + void* dst, + nuint dstSize, + void* weightTable, + nuint wtSize, + void* workspace, + nuint workspaceSize + ) + { + byte* ostart = (byte*)dst; + byte* op = ostart; + byte* oend = ostart + dstSize; + uint maxSymbolValue = 12; + uint tableLog = 6; + HUF_CompressWeightsWksp* wksp = (HUF_CompressWeightsWksp*)HUF_alignUpWorkspace( + workspace, + &workspaceSize, + sizeof(uint) + ); + if (workspaceSize < (nuint)sizeof(HUF_CompressWeightsWksp)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + + if (wtSize <= 1) + { + return 0; + } + + { + /* never fails */ + uint maxCount = HIST_count_simple(wksp->count, &maxSymbolValue, weightTable, wtSize); + if (maxCount == wtSize) + { + return 1; + } + + if (maxCount == 1) + { + return 0; + } + } + + tableLog = FSE_optimalTableLog(tableLog, wtSize, maxSymbolValue); + { + /* useLowProbCount */ + nuint _var_err__ = FSE_normalizeCount( + wksp->norm, + tableLog, + wksp->count, + wtSize, + maxSymbolValue, + 0 + ); + if (ERR_isError(_var_err__)) + { + return _var_err__; + } + } + + { + nuint hSize = FSE_writeNCount( + op, + (nuint)(oend - op), + wksp->norm, + maxSymbolValue, + tableLog + ); + if (ERR_isError(hSize)) + { + return hSize; + } + + op += hSize; + } + + { + /* Compress */ + nuint _var_err__ = FSE_buildCTable_wksp( + wksp->CTable, + wksp->norm, + maxSymbolValue, + tableLog, + wksp->scratchBuffer, + sizeof(uint) * 41 + ); + if (ERR_isError(_var_err__)) + { + return _var_err__; + } + } + + { + nuint cSize = FSE_compress_usingCTable( + op, + (nuint)(oend - op), + weightTable, + wtSize, + wksp->CTable + ); + if (ERR_isError(cSize)) + { + return cSize; + } + + if (cSize == 0) + { + return 0; + } + + op += cSize; + } + + return (nuint)(op - ostart); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint HUF_getNbBits(nuint elt) + { + return elt & 0xFF; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint HUF_getNbBitsFast(nuint elt) + { + return elt; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint HUF_getValue(nuint elt) + { + return elt & ~(nuint)0xFF; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint HUF_getValueFast(nuint elt) + { + return elt; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void HUF_setNbBits(nuint* elt, nuint nbBits) + { + assert(nbBits <= 12); + *elt = nbBits; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void HUF_setValue(nuint* elt, nuint value) + { + nuint nbBits = HUF_getNbBits(*elt); + if (nbBits > 0) + { + assert(value >> (int)nbBits == 0); + *elt |= value << (int)((nuint)(sizeof(nuint) * 8) - nbBits); + } + } + + /** HUF_readCTableHeader() : + * @returns The header from the CTable specifying the tableLog and the maxSymbolValue. + */ + private static HUF_CTableHeader HUF_readCTableHeader(nuint* ctable) + { + HUF_CTableHeader header; + memcpy(&header, ctable, (uint)sizeof(nuint)); + return header; + } + + private static void HUF_writeCTableHeader(nuint* ctable, uint tableLog, uint maxSymbolValue) + { + HUF_CTableHeader header; + memset(&header, 0, (uint)sizeof(nuint)); + assert(tableLog < 256); + header.tableLog = (byte)tableLog; + assert(maxSymbolValue < 256); + header.maxSymbolValue = (byte)maxSymbolValue; + memcpy(ctable, &header, (uint)sizeof(nuint)); + } + + private static nuint HUF_writeCTable_wksp( + void* dst, + nuint maxDstSize, + nuint* CTable, + uint maxSymbolValue, + uint huffLog, + void* workspace, + nuint workspaceSize + ) + { + nuint* ct = CTable + 1; + byte* op = (byte*)dst; + uint n; + HUF_WriteCTableWksp* wksp = (HUF_WriteCTableWksp*)HUF_alignUpWorkspace( + workspace, + &workspaceSize, + sizeof(uint) + ); + assert(HUF_readCTableHeader(CTable).maxSymbolValue == maxSymbolValue); + assert(HUF_readCTableHeader(CTable).tableLog == huffLog); + if (workspaceSize < (nuint)sizeof(HUF_WriteCTableWksp)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + + if (maxSymbolValue > 255) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_maxSymbolValue_tooLarge)); + } + + wksp->bitsToWeight[0] = 0; + for (n = 1; n < huffLog + 1; n++) + { + wksp->bitsToWeight[n] = (byte)(huffLog + 1 - n); + } + + for (n = 0; n < maxSymbolValue; n++) + { + wksp->huffWeight[n] = wksp->bitsToWeight[HUF_getNbBits(ct[n])]; + } + + if (maxDstSize < 1) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + { + nuint hSize = HUF_compressWeights( + op + 1, + maxDstSize - 1, + wksp->huffWeight, + maxSymbolValue, + &wksp->wksp, + (nuint)sizeof(HUF_CompressWeightsWksp) + ); + if (ERR_isError(hSize)) + { + return hSize; + } + + if (hSize > 1 && hSize < maxSymbolValue / 2) + { + op[0] = (byte)hSize; + return hSize + 1; + } + } + + if (maxSymbolValue > 256 - 128) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + + if ((maxSymbolValue + 1) / 2 + 1 > maxDstSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + op[0] = (byte)(128 + (maxSymbolValue - 1)); + wksp->huffWeight[maxSymbolValue] = 0; + for (n = 0; n < maxSymbolValue; n += 2) + { + op[n / 2 + 1] = (byte)((wksp->huffWeight[n] << 4) + wksp->huffWeight[n + 1]); + } + + return (maxSymbolValue + 1) / 2 + 1; + } + + /** HUF_readCTable() : + * Loading a CTable saved with HUF_writeCTable() */ + private static nuint HUF_readCTable( + nuint* CTable, + uint* maxSymbolValuePtr, + void* src, + nuint srcSize, + uint* hasZeroWeights + ) + { + /* init not required, even though some static analyzer may complain */ + byte* huffWeight = stackalloc byte[256]; + /* large enough for values from 0 to 16 */ + uint* rankVal = stackalloc uint[13]; + uint tableLog = 0; + uint nbSymbols = 0; + nuint* ct = CTable + 1; + /* get symbol weights */ + nuint readSize = HUF_readStats( + huffWeight, + 255 + 1, + rankVal, + &nbSymbols, + &tableLog, + src, + srcSize + ); + if (ERR_isError(readSize)) + { + return readSize; + } + + *hasZeroWeights = rankVal[0] > 0 ? 1U : 0U; + if (tableLog > 12) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_tableLog_tooLarge)); + } + + if (nbSymbols > *maxSymbolValuePtr + 1) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_maxSymbolValue_tooSmall)); + } + + *maxSymbolValuePtr = nbSymbols - 1; + HUF_writeCTableHeader(CTable, tableLog, *maxSymbolValuePtr); + { + uint n, + nextRankStart = 0; + for (n = 1; n <= tableLog; n++) + { + uint curr = nextRankStart; + nextRankStart += rankVal[n] << (int)(n - 1); + rankVal[n] = curr; + } + } + + { + uint n; + for (n = 0; n < nbSymbols; n++) + { + uint w = huffWeight[n]; + HUF_setNbBits(ct + n, (nuint)((byte)(tableLog + 1 - w) & -(w != 0 ? 1 : 0))); + } + } + + { + ushort* nbPerRank = stackalloc ushort[14]; + /* support w=0=>n=tableLog+1 */ + memset(nbPerRank, 0, sizeof(ushort) * 14); + ushort* valPerRank = stackalloc ushort[14]; + memset(valPerRank, 0, sizeof(ushort) * 14); + { + uint n; + for (n = 0; n < nbSymbols; n++) + { + nbPerRank[HUF_getNbBits(ct[n])]++; + } + } + + valPerRank[tableLog + 1] = 0; + { + ushort min = 0; + /* start at n=tablelog <-> w=1 */ + uint n; + for (n = tableLog; n > 0; n--) + { + valPerRank[n] = min; + min += nbPerRank[n]; + min >>= 1; + } + } + + { + uint n; + for (n = 0; n < nbSymbols; n++) + { + HUF_setValue(ct + n, valPerRank[HUF_getNbBits(ct[n])]++); + } + } + } + + return readSize; + } + + /** HUF_getNbBitsFromCTable() : + * Read nbBits from CTable symbolTable, for symbol `symbolValue` presumed <= HUF_SYMBOLVALUE_MAX + * Note 1 : If symbolValue > HUF_readCTableHeader(symbolTable).maxSymbolValue, returns 0 + * Note 2 : is not inlined, as HUF_CElt definition is private + */ + private static uint HUF_getNbBitsFromCTable(nuint* CTable, uint symbolValue) + { + nuint* ct = CTable + 1; + assert(symbolValue <= 255); + if (symbolValue > HUF_readCTableHeader(CTable).maxSymbolValue) + { + return 0; + } + + return (uint)HUF_getNbBits(ct[symbolValue]); + } + + /** + * HUF_setMaxHeight(): + * Try to enforce @targetNbBits on the Huffman tree described in @huffNode. + * + * It attempts to convert all nodes with nbBits > @targetNbBits + * to employ @targetNbBits instead. Then it adjusts the tree + * so that it remains a valid canonical Huffman tree. + * + * @pre The sum of the ranks of each symbol == 2^largestBits, + * where largestBits == huffNode[lastNonNull].nbBits. + * @post The sum of the ranks of each symbol == 2^largestBits, + * where largestBits is the return value (expected <= targetNbBits). + * + * @param huffNode The Huffman tree modified in place to enforce targetNbBits. + * It's presumed sorted, from most frequent to rarest symbol. + * @param lastNonNull The symbol with the lowest count in the Huffman tree. + * @param targetNbBits The allowed number of bits, which the Huffman tree + * may not respect. After this function the Huffman tree will + * respect targetNbBits. + * @return The maximum number of bits of the Huffman tree after adjustment. + */ + private static uint HUF_setMaxHeight(nodeElt_s* huffNode, uint lastNonNull, uint targetNbBits) + { + uint largestBits = huffNode[lastNonNull].nbBits; + if (largestBits <= targetNbBits) + { + return largestBits; + } + + { + int totalCost = 0; + uint baseCost = (uint)(1 << (int)(largestBits - targetNbBits)); + int n = (int)lastNonNull; + while (huffNode[n].nbBits > targetNbBits) + { + totalCost += (int)(baseCost - (uint)(1 << (int)(largestBits - huffNode[n].nbBits))); + huffNode[n].nbBits = (byte)targetNbBits; + n--; + } + + assert(huffNode[n].nbBits <= targetNbBits); + while (huffNode[n].nbBits == targetNbBits) + { + --n; + } + + assert(((uint)totalCost & baseCost - 1) == 0); + totalCost >>= (int)(largestBits - targetNbBits); + assert(totalCost > 0); + { + const uint noSymbol = 0xF0F0F0F0; + uint* rankLast = stackalloc uint[14]; + memset(rankLast, 0xF0, sizeof(uint) * 14); + { + uint currentNbBits = targetNbBits; + int pos; + for (pos = n; pos >= 0; pos--) + { + if (huffNode[pos].nbBits >= currentNbBits) + { + continue; + } + + currentNbBits = huffNode[pos].nbBits; + rankLast[targetNbBits - currentNbBits] = (uint)pos; + } + } + + while (totalCost > 0) + { + /* Try to reduce the next power of 2 above totalCost because we + * gain back half the rank. + */ + uint nBitsToDecrease = ZSTD_highbit32((uint)totalCost) + 1; + for (; nBitsToDecrease > 1; nBitsToDecrease--) + { + uint highPos = rankLast[nBitsToDecrease]; + uint lowPos = rankLast[nBitsToDecrease - 1]; + if (highPos == noSymbol) + { + continue; + } + + if (lowPos == noSymbol) + { + break; + } + + { + uint highTotal = huffNode[highPos].count; + uint lowTotal = 2 * huffNode[lowPos].count; + if (highTotal <= lowTotal) + { + break; + } + } + } + + assert(rankLast[nBitsToDecrease] != noSymbol || nBitsToDecrease == 1); + while (nBitsToDecrease <= 12 && rankLast[nBitsToDecrease] == noSymbol) + { + nBitsToDecrease++; + } + + assert(rankLast[nBitsToDecrease] != noSymbol); + totalCost -= 1 << (int)(nBitsToDecrease - 1); + huffNode[rankLast[nBitsToDecrease]].nbBits++; + if (rankLast[nBitsToDecrease - 1] == noSymbol) + { + rankLast[nBitsToDecrease - 1] = rankLast[nBitsToDecrease]; + } + + if (rankLast[nBitsToDecrease] == 0) + { + rankLast[nBitsToDecrease] = noSymbol; + } + else + { + rankLast[nBitsToDecrease]--; + if ( + huffNode[rankLast[nBitsToDecrease]].nbBits + != targetNbBits - nBitsToDecrease + ) + { + rankLast[nBitsToDecrease] = noSymbol; + } + } + } + + while (totalCost < 0) + { + if (rankLast[1] == noSymbol) + { + while (huffNode[n].nbBits == targetNbBits) + { + n--; + } + + huffNode[n + 1].nbBits--; + assert(n >= 0); + rankLast[1] = (uint)(n + 1); + totalCost++; + continue; + } + + huffNode[rankLast[1] + 1].nbBits--; + rankLast[1]++; + totalCost++; + } + } + } + + return targetNbBits; + } + + /* Return the appropriate bucket index for a given count. See definition of + * RANK_POSITION_DISTINCT_COUNT_CUTOFF for explanation of bucketing strategy. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint HUF_getIndex(uint count) + { + return count < 192 - 1 - 32 - 1 + ZSTD_highbit32(192 - 1 - 32 - 1) + ? count + : ZSTD_highbit32(count) + (192 - 1 - 32 - 1); + } + + /* Helper swap function for HUF_quickSortPartition() */ + private static void HUF_swapNodes(nodeElt_s* a, nodeElt_s* b) + { + nodeElt_s tmp = *a; + *a = *b; + *b = tmp; + } + + /* Returns 0 if the huffNode array is not sorted by descending count */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static int HUF_isSorted(nodeElt_s* huffNode, uint maxSymbolValue1) + { + uint i; + for (i = 1; i < maxSymbolValue1; ++i) + { + if (huffNode[i].count > huffNode[i - 1].count) + { + return 0; + } + } + + return 1; + } + + /* Insertion sort by descending order */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void HUF_insertionSort(nodeElt_s* huffNode, int low, int high) + { + int i; + int size = high - low + 1; + huffNode += low; + for (i = 1; i < size; ++i) + { + nodeElt_s key = huffNode[i]; + int j = i - 1; + while (j >= 0 && huffNode[j].count < key.count) + { + huffNode[j + 1] = huffNode[j]; + j--; + } + + huffNode[j + 1] = key; + } + } + + /* Pivot helper function for quicksort. */ + private static int HUF_quickSortPartition(nodeElt_s* arr, int low, int high) + { + /* Simply select rightmost element as pivot. "Better" selectors like + * median-of-three don't experimentally appear to have any benefit. + */ + uint pivot = arr[high].count; + int i = low - 1; + int j = low; + for (; j < high; j++) + { + if (arr[j].count > pivot) + { + i++; + HUF_swapNodes(&arr[i], &arr[j]); + } + } + + HUF_swapNodes(&arr[i + 1], &arr[high]); + return i + 1; + } + + /* Classic quicksort by descending with partially iterative calls + * to reduce worst case callstack size. + */ + private static void HUF_simpleQuickSort(nodeElt_s* arr, int low, int high) + { + const int kInsertionSortThreshold = 8; + if (high - low < kInsertionSortThreshold) + { + HUF_insertionSort(arr, low, high); + return; + } + + while (low < high) + { + int idx = HUF_quickSortPartition(arr, low, high); + if (idx - low < high - idx) + { + HUF_simpleQuickSort(arr, low, idx - 1); + low = idx + 1; + } + else + { + HUF_simpleQuickSort(arr, idx + 1, high); + high = idx - 1; + } + } + } + + /** + * HUF_sort(): + * Sorts the symbols [0, maxSymbolValue] by count[symbol] in decreasing order. + * This is a typical bucket sorting strategy that uses either quicksort or insertion sort to sort each bucket. + * + * @param[out] huffNode Sorted symbols by decreasing count. Only members `.count` and `.byte` are filled. + * Must have (maxSymbolValue + 1) entries. + * @param[in] count Histogram of the symbols. + * @param[in] maxSymbolValue Maximum symbol value. + * @param rankPosition This is a scratch workspace. Must have RANK_POSITION_TABLE_SIZE entries. + */ + private static void HUF_sort( + nodeElt_s* huffNode, + uint* count, + uint maxSymbolValue, + rankPos* rankPosition + ) + { + uint n; + uint maxSymbolValue1 = maxSymbolValue + 1; + memset(rankPosition, 0, (uint)(sizeof(rankPos) * 192)); + for (n = 0; n < maxSymbolValue1; ++n) + { + uint lowerRank = HUF_getIndex(count[n]); + assert(lowerRank < 192 - 1); + rankPosition[lowerRank].@base++; + } + + assert(rankPosition[192 - 1].@base == 0); + for (n = 192 - 1; n > 0; --n) + { + rankPosition[n - 1].@base += rankPosition[n].@base; + rankPosition[n - 1].curr = rankPosition[n - 1].@base; + } + + for (n = 0; n < maxSymbolValue1; ++n) + { + uint c = count[n]; + uint r = HUF_getIndex(c) + 1; + uint pos = rankPosition[r].curr++; + assert(pos < maxSymbolValue1); + huffNode[pos].count = c; + huffNode[pos].@byte = (byte)n; + } + + for (n = 192 - 1 - 32 - 1 + ZSTD_highbit32(192 - 1 - 32 - 1); n < 192 - 1; ++n) + { + int bucketSize = rankPosition[n].curr - rankPosition[n].@base; + uint bucketStartIdx = rankPosition[n].@base; + if (bucketSize > 1) + { + assert(bucketStartIdx < maxSymbolValue1); + HUF_simpleQuickSort(huffNode + bucketStartIdx, 0, bucketSize - 1); + } + } + + assert(HUF_isSorted(huffNode, maxSymbolValue1) != 0); + } + + /* HUF_buildTree(): + * Takes the huffNode array sorted by HUF_sort() and builds an unlimited-depth Huffman tree. + * + * @param huffNode The array sorted by HUF_sort(). Builds the Huffman tree in this array. + * @param maxSymbolValue The maximum symbol value. + * @return The smallest node in the Huffman tree (by count). + */ + private static int HUF_buildTree(nodeElt_s* huffNode, uint maxSymbolValue) + { + nodeElt_s* huffNode0 = huffNode - 1; + int nonNullRank; + int lowS, + lowN; + int nodeNb = 255 + 1; + int n, + nodeRoot; + nonNullRank = (int)maxSymbolValue; + while (huffNode[nonNullRank].count == 0) + { + nonNullRank--; + } + + lowS = nonNullRank; + nodeRoot = nodeNb + lowS - 1; + lowN = nodeNb; + huffNode[nodeNb].count = huffNode[lowS].count + huffNode[lowS - 1].count; + huffNode[lowS].parent = huffNode[lowS - 1].parent = (ushort)nodeNb; + nodeNb++; + lowS -= 2; + for (n = nodeNb; n <= nodeRoot; n++) + { + huffNode[n].count = 1U << 30; + } + + huffNode0[0].count = 1U << 31; + while (nodeNb <= nodeRoot) + { + int n1 = huffNode[lowS].count < huffNode[lowN].count ? lowS-- : lowN++; + int n2 = huffNode[lowS].count < huffNode[lowN].count ? lowS-- : lowN++; + huffNode[nodeNb].count = huffNode[n1].count + huffNode[n2].count; + huffNode[n1].parent = huffNode[n2].parent = (ushort)nodeNb; + nodeNb++; + } + + huffNode[nodeRoot].nbBits = 0; + for (n = nodeRoot - 1; n >= 255 + 1; n--) + { + huffNode[n].nbBits = (byte)(huffNode[huffNode[n].parent].nbBits + 1); + } + + for (n = 0; n <= nonNullRank; n++) + { + huffNode[n].nbBits = (byte)(huffNode[huffNode[n].parent].nbBits + 1); + } + + return nonNullRank; + } + + /** + * HUF_buildCTableFromTree(): + * Build the CTable given the Huffman tree in huffNode. + * + * @param[out] CTable The output Huffman CTable. + * @param huffNode The Huffman tree. + * @param nonNullRank The last and smallest node in the Huffman tree. + * @param maxSymbolValue The maximum symbol value. + * @param maxNbBits The exact maximum number of bits used in the Huffman tree. + */ + private static void HUF_buildCTableFromTree( + nuint* CTable, + nodeElt_s* huffNode, + int nonNullRank, + uint maxSymbolValue, + uint maxNbBits + ) + { + nuint* ct = CTable + 1; + /* fill result into ctable (val, nbBits) */ + int n; + ushort* nbPerRank = stackalloc ushort[13]; + memset(nbPerRank, 0, sizeof(ushort) * 13); + ushort* valPerRank = stackalloc ushort[13]; + memset(valPerRank, 0, sizeof(ushort) * 13); + int alphabetSize = (int)(maxSymbolValue + 1); + for (n = 0; n <= nonNullRank; n++) + { + nbPerRank[huffNode[n].nbBits]++; + } + + { + ushort min = 0; + for (n = (int)maxNbBits; n > 0; n--) + { + valPerRank[n] = min; + min += nbPerRank[n]; + min >>= 1; + } + } + + for (n = 0; n < alphabetSize; n++) + { + HUF_setNbBits(ct + huffNode[n].@byte, huffNode[n].nbBits); + } + + for (n = 0; n < alphabetSize; n++) + { + HUF_setValue(ct + n, valPerRank[HUF_getNbBits(ct[n])]++); + } + + HUF_writeCTableHeader(CTable, maxNbBits, maxSymbolValue); + } + + private static nuint HUF_buildCTable_wksp( + nuint* CTable, + uint* count, + uint maxSymbolValue, + uint maxNbBits, + void* workSpace, + nuint wkspSize + ) + { + HUF_buildCTable_wksp_tables* wksp_tables = + (HUF_buildCTable_wksp_tables*)HUF_alignUpWorkspace(workSpace, &wkspSize, sizeof(uint)); + nodeElt_s* huffNode0 = &wksp_tables->huffNodeTbl.e0; + nodeElt_s* huffNode = huffNode0 + 1; + int nonNullRank; + if (wkspSize < (nuint)sizeof(HUF_buildCTable_wksp_tables)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_workSpace_tooSmall)); + } + + if (maxNbBits == 0) + { + maxNbBits = 11; + } + + if (maxSymbolValue > 255) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_maxSymbolValue_tooLarge)); + } + + memset(huffNode0, 0, (uint)(sizeof(nodeElt_s) * 512)); + HUF_sort(huffNode, count, maxSymbolValue, &wksp_tables->rankPosition.e0); + nonNullRank = HUF_buildTree(huffNode, maxSymbolValue); + maxNbBits = HUF_setMaxHeight(huffNode, (uint)nonNullRank, maxNbBits); + if (maxNbBits > 12) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + + HUF_buildCTableFromTree(CTable, huffNode, nonNullRank, maxSymbolValue, maxNbBits); + return maxNbBits; + } + + private static nuint HUF_estimateCompressedSize(nuint* CTable, uint* count, uint maxSymbolValue) + { + nuint* ct = CTable + 1; + nuint nbBits = 0; + int s; + for (s = 0; s <= (int)maxSymbolValue; ++s) + { + nbBits += HUF_getNbBits(ct[s]) * count[s]; + } + + return nbBits >> 3; + } + + private static int HUF_validateCTable(nuint* CTable, uint* count, uint maxSymbolValue) + { + HUF_CTableHeader header = HUF_readCTableHeader(CTable); + nuint* ct = CTable + 1; + int bad = 0; + int s; + assert(header.tableLog <= 12); + if (header.maxSymbolValue < maxSymbolValue) + { + return 0; + } + + for (s = 0; s <= (int)maxSymbolValue; ++s) + { + bad |= count[s] != 0 && HUF_getNbBits(ct[s]) == 0 ? 1 : 0; + } + + return bad == 0 ? 1 : 0; + } + + private static nuint HUF_compressBound(nuint size) + { + return 129 + (size + (size >> 8) + 8); + } + + /**! HUF_initCStream(): + * Initializes the bitstream. + * @returns 0 or an error code. + */ + private static nuint HUF_initCStream(ref HUF_CStream_t bitC, void* startPtr, nuint dstCapacity) + { + bitC = new HUF_CStream_t + { + startPtr = (byte*)startPtr, + ptr = (byte*)startPtr, + endPtr = (byte*)startPtr + dstCapacity - sizeof(nuint), + }; + if (dstCapacity <= (nuint)sizeof(nuint)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + return 0; + } + + /*! HUF_addBits(): + * Adds the symbol stored in HUF_CElt elt to the bitstream. + * + * @param elt The element we're adding. This is a (nbBits, value) pair. + * See the HUF_CStream_t docs for the format. + * @param idx Insert into the bitstream at this idx. + * @param kFast This is a template parameter. If the bitstream is guaranteed + * to have at least 4 unused bits after this call it may be 1, + * otherwise it must be 0. HUF_addBits() is faster when fast is set. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void HUF_addBits( + ref nuint bitC_bitContainer_e0, + ref nuint bitC_bitPos_e0, + nuint elt, + int kFast + ) + { + assert(HUF_getNbBits(elt) <= 12); + bitC_bitContainer_e0 >>= (int)HUF_getNbBits(elt); + bitC_bitContainer_e0 |= kFast != 0 ? HUF_getValueFast(elt) : HUF_getValue(elt); + bitC_bitPos_e0 += HUF_getNbBitsFast(elt); + assert((bitC_bitPos_e0 & 0xFF) <= (nuint)(sizeof(nuint) * 8)); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void HUF_zeroIndex1(ref nuint bitC_bitContainer_e1, ref nuint bitC_bitPos_e1) + { + bitC_bitContainer_e1 = 0; + bitC_bitPos_e1 = 0; + } + + /*! HUF_mergeIndex1() : + * Merges the bit container @ index 1 into the bit container @ index 0 + * and zeros the bit container @ index 1. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void HUF_mergeIndex1( + ref nuint bitC_bitContainer_e0, + ref nuint bitC_bitPos_e0, + ref nuint bitC_bitContainer_e1, + ref nuint bitC_bitPos_e1 + ) + { + assert((bitC_bitPos_e1 & 0xFF) < (nuint)(sizeof(nuint) * 8)); + bitC_bitContainer_e0 >>= (int)(bitC_bitPos_e1 & 0xFF); + bitC_bitContainer_e0 |= bitC_bitContainer_e1; + bitC_bitPos_e0 += bitC_bitPos_e1; + assert((bitC_bitPos_e0 & 0xFF) <= (nuint)(sizeof(nuint) * 8)); + } + + /*! HUF_flushBits() : + * Flushes the bits in the bit container @ index 0. + * + * @post bitPos will be < 8. + * @param kFast If kFast is set then we must know a-priori that + * the bit container will not overflow. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void HUF_flushBits( + ref nuint bitC_bitContainer_e0, + ref nuint bitC_bitPos_e0, + ref byte* bitC_ptr, + byte* bitC_endPtr, + int kFast + ) + { + /* The upper bits of bitPos are noisy, so we must mask by 0xFF. */ + nuint nbBits = bitC_bitPos_e0 & 0xFF; + nuint nbBytes = nbBits >> 3; + /* The top nbBits bits of bitContainer are the ones we need. */ + nuint bitContainer = bitC_bitContainer_e0 >> (int)((nuint)(sizeof(nuint) * 8) - nbBits); + bitC_bitPos_e0 &= 7; + assert(nbBits > 0); + assert(nbBits <= (nuint)(sizeof(nuint) * 8)); + assert(bitC_ptr <= bitC_endPtr); + MEM_writeLEST(bitC_ptr, bitContainer); + bitC_ptr += nbBytes; + assert(kFast == 0 || bitC_ptr <= bitC_endPtr); + if (kFast == 0 && bitC_ptr > bitC_endPtr) + { + bitC_ptr = bitC_endPtr; + } + } + + /*! HUF_endMark() + * @returns The Huffman stream end mark: A 1-bit value = 1. + */ + private static nuint HUF_endMark() + { + nuint endMark; + HUF_setNbBits(&endMark, 1); + HUF_setValue(&endMark, 1); + return endMark; + } + + /*! HUF_closeCStream() : + * @return Size of CStream, in bytes, + * or 0 if it could not fit into dstBuffer */ + private static nuint HUF_closeCStream(ref HUF_CStream_t bitC) + { + HUF_addBits(ref bitC.bitContainer.e0, ref bitC.bitPos.e0, HUF_endMark(), 0); + HUF_flushBits(ref bitC.bitContainer.e0, ref bitC.bitPos.e0, ref bitC.ptr, bitC.endPtr, 0); + { + nuint nbBits = bitC.bitPos.e0 & 0xFF; + if (bitC.ptr >= bitC.endPtr) + { + return 0; + } + + return (nuint)(bitC.ptr - bitC.startPtr) + (nuint)(nbBits > 0 ? 1 : 0); + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void HUF_encodeSymbol( + ref nuint bitCPtr_bitContainer_e0, + ref nuint bitCPtr_bitPos_e0, + uint symbol, + nuint* CTable, + int fast + ) + { + HUF_addBits(ref bitCPtr_bitContainer_e0, ref bitCPtr_bitPos_e0, CTable[symbol], fast); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void HUF_compress1X_usingCTable_internal_body_loop( + ref HUF_CStream_t bitC, + byte* ip, + nuint srcSize, + nuint* ct, + int kUnroll, + int kFastFlush, + int kLastFast + ) + { + byte* bitC_ptr = bitC.ptr; + byte* bitC_endPtr = bitC.endPtr; + nuint bitC_bitContainer_e0 = bitC.bitContainer.e0; + nuint bitC_bitPos_e0 = bitC.bitPos.e0; + nuint bitC_bitContainer_e1 = bitC.bitContainer.e1; + nuint bitC_bitPos_e1 = bitC.bitPos.e1; + /* Join to kUnroll */ + int n = (int)srcSize; + int rem = n % kUnroll; + if (rem > 0) + { + for (; rem > 0; --rem) + { + HUF_encodeSymbol(ref bitC_bitContainer_e0, ref bitC_bitPos_e0, ip[--n], ct, 0); + } + + HUF_flushBits( + ref bitC_bitContainer_e0, + ref bitC_bitPos_e0, + ref bitC_ptr, + bitC_endPtr, + kFastFlush + ); + } + + assert(n % kUnroll == 0); + if (n % (2 * kUnroll) != 0) + { + int u; + for (u = 1; u < kUnroll; ++u) + { + HUF_encodeSymbol(ref bitC_bitContainer_e0, ref bitC_bitPos_e0, ip[n - u], ct, 1); + } + + HUF_encodeSymbol( + ref bitC_bitContainer_e0, + ref bitC_bitPos_e0, + ip[n - kUnroll], + ct, + kLastFast + ); + HUF_flushBits( + ref bitC_bitContainer_e0, + ref bitC_bitPos_e0, + ref bitC_ptr, + bitC_endPtr, + kFastFlush + ); + n -= kUnroll; + } + + assert(n % (2 * kUnroll) == 0); + for (; n > 0; n -= 2 * kUnroll) + { + /* Encode kUnroll symbols into the bitstream @ index 0. */ + int u; + for (u = 1; u < kUnroll; ++u) + { + HUF_encodeSymbol(ref bitC_bitContainer_e0, ref bitC_bitPos_e0, ip[n - u], ct, 1); + } + + HUF_encodeSymbol( + ref bitC_bitContainer_e0, + ref bitC_bitPos_e0, + ip[n - kUnroll], + ct, + kLastFast + ); + HUF_flushBits( + ref bitC_bitContainer_e0, + ref bitC_bitPos_e0, + ref bitC_ptr, + bitC_endPtr, + kFastFlush + ); + HUF_zeroIndex1(ref bitC_bitContainer_e1, ref bitC_bitPos_e1); + for (u = 1; u < kUnroll; ++u) + { + HUF_encodeSymbol( + ref bitC_bitContainer_e1, + ref bitC_bitPos_e1, + ip[n - kUnroll - u], + ct, + 1 + ); + } + + HUF_encodeSymbol( + ref bitC_bitContainer_e1, + ref bitC_bitPos_e1, + ip[n - kUnroll - kUnroll], + ct, + kLastFast + ); + HUF_mergeIndex1( + ref bitC_bitContainer_e0, + ref bitC_bitPos_e0, + ref bitC_bitContainer_e1, + ref bitC_bitPos_e1 + ); + HUF_flushBits( + ref bitC_bitContainer_e0, + ref bitC_bitPos_e0, + ref bitC_ptr, + bitC_endPtr, + kFastFlush + ); + } + + assert(n == 0); + bitC.ptr = bitC_ptr; + bitC.endPtr = bitC_endPtr; + bitC.bitContainer.e0 = bitC_bitContainer_e0; + bitC.bitPos.e0 = bitC_bitPos_e0; + bitC.bitContainer.e1 = bitC_bitContainer_e1; + bitC.bitPos.e1 = bitC_bitPos_e1; + } + + /** + * Returns a tight upper bound on the output space needed by Huffman + * with 8 bytes buffer to handle over-writes. If the output is at least + * this large we don't need to do bounds checks during Huffman encoding. + */ + private static nuint HUF_tightCompressBound(nuint srcSize, nuint tableLog) + { + return (srcSize * tableLog >> 3) + 8; + } + + private static nuint HUF_compress1X_usingCTable_internal_body( + void* dst, + nuint dstSize, + void* src, + nuint srcSize, + nuint* CTable + ) + { + uint tableLog = HUF_readCTableHeader(CTable).tableLog; + nuint* ct = CTable + 1; + byte* ip = (byte*)src; + byte* ostart = (byte*)dst; + byte* oend = ostart + dstSize; + HUF_CStream_t bitC; + System.Runtime.CompilerServices.Unsafe.SkipInit(out bitC); + if (dstSize < 8) + { + return 0; + } + + { + byte* op = ostart; + nuint initErr = HUF_initCStream(ref bitC, op, (nuint)(oend - op)); + if (ERR_isError(initErr)) + { + return 0; + } + } + + if (dstSize < HUF_tightCompressBound(srcSize, tableLog) || tableLog > 11) + { + HUF_compress1X_usingCTable_internal_body_loop( + ref bitC, + ip, + srcSize, + ct, + MEM_32bits ? 2 : 4, + 0, + 0 + ); + } + else + { + if (MEM_32bits) + { + switch (tableLog) + { + case 11: + HUF_compress1X_usingCTable_internal_body_loop( + ref bitC, + ip, + srcSize, + ct, + 2, + 1, + 0 + ); + break; + case 10: + case 9: + case 8: + HUF_compress1X_usingCTable_internal_body_loop( + ref bitC, + ip, + srcSize, + ct, + 2, + 1, + 1 + ); + break; + case 7: + default: + HUF_compress1X_usingCTable_internal_body_loop( + ref bitC, + ip, + srcSize, + ct, + 3, + 1, + 1 + ); + break; + } + } + else + { + switch (tableLog) + { + case 11: + HUF_compress1X_usingCTable_internal_body_loop( + ref bitC, + ip, + srcSize, + ct, + 5, + 1, + 0 + ); + break; + case 10: + HUF_compress1X_usingCTable_internal_body_loop( + ref bitC, + ip, + srcSize, + ct, + 5, + 1, + 1 + ); + break; + case 9: + HUF_compress1X_usingCTable_internal_body_loop( + ref bitC, + ip, + srcSize, + ct, + 6, + 1, + 0 + ); + break; + case 8: + HUF_compress1X_usingCTable_internal_body_loop( + ref bitC, + ip, + srcSize, + ct, + 7, + 1, + 0 + ); + break; + case 7: + HUF_compress1X_usingCTable_internal_body_loop( + ref bitC, + ip, + srcSize, + ct, + 8, + 1, + 0 + ); + break; + case 6: + default: + HUF_compress1X_usingCTable_internal_body_loop( + ref bitC, + ip, + srcSize, + ct, + 9, + 1, + 1 + ); + break; + } + } + } + + assert(bitC.ptr <= bitC.endPtr); + return HUF_closeCStream(ref bitC); + } + + private static nuint HUF_compress1X_usingCTable_internal( + void* dst, + nuint dstSize, + void* src, + nuint srcSize, + nuint* CTable, + int flags + ) + { + return HUF_compress1X_usingCTable_internal_body(dst, dstSize, src, srcSize, CTable); + } + + /* ====================== */ + /* single stream variants */ + /* ====================== */ + private static nuint HUF_compress1X_usingCTable( + void* dst, + nuint dstSize, + void* src, + nuint srcSize, + nuint* CTable, + int flags + ) + { + return HUF_compress1X_usingCTable_internal(dst, dstSize, src, srcSize, CTable, flags); + } + + private static nuint HUF_compress4X_usingCTable_internal( + void* dst, + nuint dstSize, + void* src, + nuint srcSize, + nuint* CTable, + int flags + ) + { + /* first 3 segments */ + nuint segmentSize = (srcSize + 3) / 4; + byte* ip = (byte*)src; + byte* iend = ip + srcSize; + byte* ostart = (byte*)dst; + byte* oend = ostart + dstSize; + byte* op = ostart; + if (dstSize < 6 + 1 + 1 + 1 + 8) + { + return 0; + } + + if (srcSize < 12) + { + return 0; + } + + op += 6; + assert(op <= oend); + { + nuint cSize = HUF_compress1X_usingCTable_internal( + op, + (nuint)(oend - op), + ip, + segmentSize, + CTable, + flags + ); + if (ERR_isError(cSize)) + { + return cSize; + } + + if (cSize == 0 || cSize > 65535) + { + return 0; + } + + MEM_writeLE16(ostart, (ushort)cSize); + op += cSize; + } + + ip += segmentSize; + assert(op <= oend); + { + nuint cSize = HUF_compress1X_usingCTable_internal( + op, + (nuint)(oend - op), + ip, + segmentSize, + CTable, + flags + ); + if (ERR_isError(cSize)) + { + return cSize; + } + + if (cSize == 0 || cSize > 65535) + { + return 0; + } + + MEM_writeLE16(ostart + 2, (ushort)cSize); + op += cSize; + } + + ip += segmentSize; + assert(op <= oend); + { + nuint cSize = HUF_compress1X_usingCTable_internal( + op, + (nuint)(oend - op), + ip, + segmentSize, + CTable, + flags + ); + if (ERR_isError(cSize)) + { + return cSize; + } + + if (cSize == 0 || cSize > 65535) + { + return 0; + } + + MEM_writeLE16(ostart + 4, (ushort)cSize); + op += cSize; + } + + ip += segmentSize; + assert(op <= oend); + assert(ip <= iend); + { + nuint cSize = HUF_compress1X_usingCTable_internal( + op, + (nuint)(oend - op), + ip, + (nuint)(iend - ip), + CTable, + flags + ); + if (ERR_isError(cSize)) + { + return cSize; + } + + if (cSize == 0 || cSize > 65535) + { + return 0; + } + + op += cSize; + } + + return (nuint)(op - ostart); + } + + private static nuint HUF_compress4X_usingCTable( + void* dst, + nuint dstSize, + void* src, + nuint srcSize, + nuint* CTable, + int flags + ) + { + return HUF_compress4X_usingCTable_internal(dst, dstSize, src, srcSize, CTable, flags); + } + + private static nuint HUF_compressCTable_internal( + byte* ostart, + byte* op, + byte* oend, + void* src, + nuint srcSize, + HUF_nbStreams_e nbStreams, + nuint* CTable, + int flags + ) + { + nuint cSize = + nbStreams == HUF_nbStreams_e.HUF_singleStream + ? HUF_compress1X_usingCTable_internal( + op, + (nuint)(oend - op), + src, + srcSize, + CTable, + flags + ) + : HUF_compress4X_usingCTable_internal( + op, + (nuint)(oend - op), + src, + srcSize, + CTable, + flags + ); + if (ERR_isError(cSize)) + { + return cSize; + } + + if (cSize == 0) + { + return 0; + } + + op += cSize; + assert(op >= ostart); + if ((nuint)(op - ostart) >= srcSize - 1) + { + return 0; + } + + return (nuint)(op - ostart); + } + + private static uint HUF_cardinality(uint* count, uint maxSymbolValue) + { + uint cardinality = 0; + uint i; + for (i = 0; i < maxSymbolValue + 1; i++) + { + if (count[i] != 0) + { + cardinality += 1; + } + } + + return cardinality; + } + + /*! HUF_compress() does the following: + * 1. count symbol occurrence from source[] into table count[] using FSE_count() (exposed within "fse.h") + * 2. (optional) refine tableLog using HUF_optimalTableLog() + * 3. build Huffman table from count using HUF_buildCTable() + * 4. save Huffman table to memory buffer using HUF_writeCTable() + * 5. encode the data stream using HUF_compress4X_usingCTable() + * + * The following API allows targeting specific sub-functions for advanced tasks. + * For example, it's possible to compress several blocks using the same 'CTable', + * or to save and regenerate 'CTable' using external methods. + */ + private static uint HUF_minTableLog(uint symbolCardinality) + { + uint minBitsSymbols = ZSTD_highbit32(symbolCardinality) + 1; + return minBitsSymbols; + } + + private static uint HUF_optimalTableLog( + uint maxTableLog, + nuint srcSize, + uint maxSymbolValue, + void* workSpace, + nuint wkspSize, + nuint* table, + uint* count, + int flags + ) + { + assert(srcSize > 1); + assert(wkspSize >= (nuint)sizeof(HUF_buildCTable_wksp_tables)); + if ((flags & (int)HUF_flags_e.HUF_flags_optimalDepth) == 0) + { + return FSE_optimalTableLog_internal(maxTableLog, srcSize, maxSymbolValue, 1); + } + + { + byte* dst = (byte*)workSpace + sizeof(HUF_WriteCTableWksp); + nuint dstSize = wkspSize - (nuint)sizeof(HUF_WriteCTableWksp); + nuint hSize, + newSize; + uint symbolCardinality = HUF_cardinality(count, maxSymbolValue); + uint minTableLog = HUF_minTableLog(symbolCardinality); + nuint optSize = unchecked((nuint)~0) - 1; + uint optLog = maxTableLog, + optLogGuess; + for (optLogGuess = minTableLog; optLogGuess <= maxTableLog; optLogGuess++) + { + { + nuint maxBits = HUF_buildCTable_wksp( + table, + count, + maxSymbolValue, + optLogGuess, + workSpace, + wkspSize + ); + if (ERR_isError(maxBits)) + { + continue; + } + + if (maxBits < optLogGuess && optLogGuess > minTableLog) + { + break; + } + + hSize = HUF_writeCTable_wksp( + dst, + dstSize, + table, + maxSymbolValue, + (uint)maxBits, + workSpace, + wkspSize + ); + } + + if (ERR_isError(hSize)) + { + continue; + } + + newSize = HUF_estimateCompressedSize(table, count, maxSymbolValue) + hSize; + if (newSize > optSize + 1) + { + break; + } + + if (newSize < optSize) + { + optSize = newSize; + optLog = optLogGuess; + } + } + + assert(optLog <= 12); + return optLog; + } + } + + /* HUF_compress_internal() : + * `workSpace_align4` must be aligned on 4-bytes boundaries, + * and occupies the same space as a table of HUF_WORKSPACE_SIZE_U64 unsigned */ + private static nuint HUF_compress_internal( + void* dst, + nuint dstSize, + void* src, + nuint srcSize, + uint maxSymbolValue, + uint huffLog, + HUF_nbStreams_e nbStreams, + void* workSpace, + nuint wkspSize, + nuint* oldHufTable, + HUF_repeat* repeat, + int flags + ) + { + HUF_compress_tables_t* table = (HUF_compress_tables_t*)HUF_alignUpWorkspace( + workSpace, + &wkspSize, + sizeof(ulong) + ); + byte* ostart = (byte*)dst; + byte* oend = ostart + dstSize; + byte* op = ostart; + if (wkspSize < (nuint)sizeof(HUF_compress_tables_t)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_workSpace_tooSmall)); + } + + if (srcSize == 0) + { + return 0; + } + + if (dstSize == 0) + { + return 0; + } + + if (srcSize > 128 * 1024) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + if (huffLog > 12) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_tableLog_tooLarge)); + } + + if (maxSymbolValue > 255) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_maxSymbolValue_tooLarge)); + } + + if (maxSymbolValue == 0) + { + maxSymbolValue = 255; + } + + if (huffLog == 0) + { + huffLog = 11; + } + + if ( + (flags & (int)HUF_flags_e.HUF_flags_preferRepeat) != 0 + && repeat != null + && *repeat == HUF_repeat.HUF_repeat_valid + ) + { + return HUF_compressCTable_internal( + ostart, + op, + oend, + src, + srcSize, + nbStreams, + oldHufTable, + flags + ); + } + + if ((flags & (int)HUF_flags_e.HUF_flags_suspectUncompressible) != 0 && srcSize >= 4096 * 10) + { + nuint largestTotal = 0; + { + uint maxSymbolValueBegin = maxSymbolValue; + nuint largestBegin = HIST_count_simple( + table->count, + &maxSymbolValueBegin, + (byte*)src, + 4096 + ); + if (ERR_isError(largestBegin)) + { + return largestBegin; + } + + largestTotal += largestBegin; + } + + { + uint maxSymbolValueEnd = maxSymbolValue; + nuint largestEnd = HIST_count_simple( + table->count, + &maxSymbolValueEnd, + (byte*)src + srcSize - 4096, + 4096 + ); + if (ERR_isError(largestEnd)) + { + return largestEnd; + } + + largestTotal += largestEnd; + } + + if (largestTotal <= (2 * 4096 >> 7) + 4) + { + return 0; + } + } + + { + nuint largest = HIST_count_wksp( + table->count, + &maxSymbolValue, + (byte*)src, + srcSize, + table->wksps.hist_wksp, + sizeof(uint) * 1024 + ); + if (ERR_isError(largest)) + { + return largest; + } + + if (largest == srcSize) + { + *ostart = ((byte*)src)[0]; + return 1; + } + + if (largest <= (srcSize >> 7) + 4) + { + return 0; + } + } + + if ( + repeat != null + && *repeat == HUF_repeat.HUF_repeat_check + && HUF_validateCTable(oldHufTable, table->count, maxSymbolValue) == 0 + ) + { + *repeat = HUF_repeat.HUF_repeat_none; + } + + if ( + (flags & (int)HUF_flags_e.HUF_flags_preferRepeat) != 0 + && repeat != null + && *repeat != HUF_repeat.HUF_repeat_none + ) + { + return HUF_compressCTable_internal( + ostart, + op, + oend, + src, + srcSize, + nbStreams, + oldHufTable, + flags + ); + } + + huffLog = HUF_optimalTableLog( + huffLog, + srcSize, + maxSymbolValue, + &table->wksps, + (nuint)sizeof(_wksps_e__Union), + &table->CTable.e0, + table->count, + flags + ); + { + nuint maxBits = HUF_buildCTable_wksp( + &table->CTable.e0, + table->count, + maxSymbolValue, + huffLog, + &table->wksps.buildCTable_wksp, + (nuint)sizeof(HUF_buildCTable_wksp_tables) + ); + { + nuint _var_err__ = maxBits; + if (ERR_isError(_var_err__)) + { + return _var_err__; + } + } + + huffLog = (uint)maxBits; + } + + { + nuint hSize = HUF_writeCTable_wksp( + op, + dstSize, + &table->CTable.e0, + maxSymbolValue, + huffLog, + &table->wksps.writeCTable_wksp, + (nuint)sizeof(HUF_WriteCTableWksp) + ); + if (ERR_isError(hSize)) + { + return hSize; + } + + if (repeat != null && *repeat != HUF_repeat.HUF_repeat_none) + { + nuint oldSize = HUF_estimateCompressedSize( + oldHufTable, + table->count, + maxSymbolValue + ); + nuint newSize = HUF_estimateCompressedSize( + &table->CTable.e0, + table->count, + maxSymbolValue + ); + if (oldSize <= hSize + newSize || hSize + 12 >= srcSize) + { + return HUF_compressCTable_internal( + ostart, + op, + oend, + src, + srcSize, + nbStreams, + oldHufTable, + flags + ); + } + } + + if (hSize + 12U >= srcSize) + { + return 0; + } + + op += hSize; + if (repeat != null) + { + *repeat = HUF_repeat.HUF_repeat_none; + } + + if (oldHufTable != null) + { + memcpy(oldHufTable, &table->CTable.e0, sizeof(ulong) * 257); + } + } + + return HUF_compressCTable_internal( + ostart, + op, + oend, + src, + srcSize, + nbStreams, + &table->CTable.e0, + flags + ); + } + + /** HUF_compress1X_repeat() : + * Same as HUF_compress1X_wksp(), but considers using hufTable if *repeat != HUF_repeat_none. + * If it uses hufTable it does not modify hufTable or repeat. + * If it doesn't, it sets *repeat = HUF_repeat_none, and it sets hufTable to the table used. + * If preferRepeat then the old table will always be used if valid. + * If suspectUncompressible then some sampling checks will be run to potentially skip huffman coding */ + private static nuint HUF_compress1X_repeat( + void* dst, + nuint dstSize, + void* src, + nuint srcSize, + uint maxSymbolValue, + uint huffLog, + void* workSpace, + nuint wkspSize, + nuint* hufTable, + HUF_repeat* repeat, + int flags + ) + { + return HUF_compress_internal( + dst, + dstSize, + src, + srcSize, + maxSymbolValue, + huffLog, + HUF_nbStreams_e.HUF_singleStream, + workSpace, + wkspSize, + hufTable, + repeat, + flags + ); + } + + /* HUF_compress4X_repeat(): + * compress input using 4 streams. + * consider skipping quickly + * reuse an existing huffman compression table */ + private static nuint HUF_compress4X_repeat( + void* dst, + nuint dstSize, + void* src, + nuint srcSize, + uint maxSymbolValue, + uint huffLog, + void* workSpace, + nuint wkspSize, + nuint* hufTable, + HUF_repeat* repeat, + int flags + ) + { + return HUF_compress_internal( + dst, + dstSize, + src, + srcSize, + maxSymbolValue, + huffLog, + HUF_nbStreams_e.HUF_fourStreams, + workSpace, + wkspSize, + hufTable, + repeat, + flags + ); + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/HufDecompress.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/HufDecompress.cs new file mode 100644 index 00000000..fc3d6247 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/HufDecompress.cs @@ -0,0 +1,2787 @@ +using System; +using System.Runtime.CompilerServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + private static DTableDesc HUF_getDTableDesc(uint* table) + { + DTableDesc dtd; + memcpy(&dtd, table, (uint)sizeof(DTableDesc)); + return dtd; + } + + private static nuint HUF_initFastDStream(byte* ip) + { + byte lastByte = ip[7]; + nuint bitsConsumed = lastByte != 0 ? 8 - ZSTD_highbit32(lastByte) : 0; + nuint value = MEM_readLEST(ip) | 1; + assert(bitsConsumed <= 8); + assert(sizeof(nuint) == 8); + return value << (int)bitsConsumed; + } + + /** + * Initializes args for the fast decoding loop. + * @returns 1 on success + * 0 if the fallback implementation should be used. + * Or an error code on failure. + */ + private static nuint HUF_DecompressFastArgs_init( + HUF_DecompressFastArgs* args, + void* dst, + nuint dstSize, + void* src, + nuint srcSize, + uint* DTable + ) + { + void* dt = DTable + 1; + uint dtLog = HUF_getDTableDesc(DTable).tableLog; + byte* istart = (byte*)src; + byte* oend = ZSTD_maybeNullPtrAdd((byte*)dst, (nint)dstSize); + if (!BitConverter.IsLittleEndian || MEM_32bits) + { + return 0; + } + + if (dstSize == 0) + { + return 0; + } + + assert(dst != null); + if (srcSize < 10) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + if (dtLog != 11) + { + return 0; + } + + { + nuint length1 = MEM_readLE16(istart); + nuint length2 = MEM_readLE16(istart + 2); + nuint length3 = MEM_readLE16(istart + 4); + nuint length4 = srcSize - (length1 + length2 + length3 + 6); + args->iend.e0 = istart + 6; + args->iend.e1 = args->iend.e0 + length1; + args->iend.e2 = args->iend.e1 + length2; + args->iend.e3 = args->iend.e2 + length3; + if (length1 < 8 || length2 < 8 || length3 < 8 || length4 < 8) + { + return 0; + } + + if (length4 > srcSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + } + + args->ip.e0 = args->iend.e1 - sizeof(ulong); + args->ip.e1 = args->iend.e2 - sizeof(ulong); + args->ip.e2 = args->iend.e3 - sizeof(ulong); + args->ip.e3 = (byte*)src + srcSize - sizeof(ulong); + args->op.e0 = (byte*)dst; + args->op.e1 = args->op.e0 + (dstSize + 3) / 4; + args->op.e2 = args->op.e1 + (dstSize + 3) / 4; + args->op.e3 = args->op.e2 + (dstSize + 3) / 4; + if (args->op.e3 >= oend) + { + return 0; + } + + args->bits[0] = HUF_initFastDStream(args->ip.e0); + args->bits[1] = HUF_initFastDStream(args->ip.e1); + args->bits[2] = HUF_initFastDStream(args->ip.e2); + args->bits[3] = HUF_initFastDStream(args->ip.e3); + args->ilowest = istart; + args->oend = oend; + args->dt = dt; + return 1; + } + + private static nuint HUF_initRemainingDStream( + BIT_DStream_t* bit, + HUF_DecompressFastArgs* args, + int stream, + byte* segmentEnd + ) + { + if ((&args->op.e0)[stream] > segmentEnd) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + if ((&args->ip.e0)[stream] < (&args->iend.e0)[stream] - 8) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + assert(sizeof(nuint) == 8); + bit->bitContainer = MEM_readLEST((&args->ip.e0)[stream]); + bit->bitsConsumed = ZSTD_countTrailingZeros64(args->bits[stream]); + bit->start = (sbyte*)args->ilowest; + bit->limitPtr = bit->start + sizeof(nuint); + bit->ptr = (sbyte*)(&args->ip.e0)[stream]; + return 0; + } + + /** + * Packs 4 HUF_DEltX1 structs into a U64. This is used to lay down 4 entries at + * a time. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong HUF_DEltX1_set4(byte symbol, byte nbBits) + { + ulong D4; + if (BitConverter.IsLittleEndian) + { + D4 = (ulong)((symbol << 8) + nbBits); + } + else + { + D4 = (ulong)(symbol + (nbBits << 8)); + } + + assert(D4 < 1U << 16); + D4 *= 0x0001000100010001UL; + return D4; + } + + /** + * Increase the tableLog to targetTableLog and rescales the stats. + * If tableLog > targetTableLog this is a no-op. + * @returns New tableLog + */ + private static uint HUF_rescaleStats( + byte* huffWeight, + uint* rankVal, + uint nbSymbols, + uint tableLog, + uint targetTableLog + ) + { + if (tableLog > targetTableLog) + { + return tableLog; + } + + if (tableLog < targetTableLog) + { + uint scale = targetTableLog - tableLog; + uint s; + for (s = 0; s < nbSymbols; ++s) + { + huffWeight[s] += (byte)(huffWeight[s] == 0 ? 0 : scale); + } + + for (s = targetTableLog; s > scale; --s) + { + rankVal[s] = rankVal[s - scale]; + } + + for (s = scale; s > 0; --s) + { + rankVal[s] = 0; + } + } + + return targetTableLog; + } + + private static nuint HUF_readDTableX1_wksp( + uint* DTable, + void* src, + nuint srcSize, + void* workSpace, + nuint wkspSize, + int flags + ) + { + uint tableLog = 0; + uint nbSymbols = 0; + nuint iSize; + void* dtPtr = DTable + 1; + HUF_DEltX1* dt = (HUF_DEltX1*)dtPtr; + HUF_ReadDTableX1_Workspace* wksp = (HUF_ReadDTableX1_Workspace*)workSpace; + if ((nuint)sizeof(HUF_ReadDTableX1_Workspace) > wkspSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_tableLog_tooLarge)); + } + + iSize = HUF_readStats_wksp( + wksp->huffWeight, + 255 + 1, + wksp->rankVal, + &nbSymbols, + &tableLog, + src, + srcSize, + wksp->statsWksp, + sizeof(uint) * 219, + flags + ); + if (ERR_isError(iSize)) + { + return iSize; + } + + { + DTableDesc dtd = HUF_getDTableDesc(DTable); + uint maxTableLog = (uint)(dtd.maxTableLog + 1); + uint targetTableLog = maxTableLog < 11 ? maxTableLog : 11; + tableLog = HUF_rescaleStats( + wksp->huffWeight, + wksp->rankVal, + nbSymbols, + tableLog, + targetTableLog + ); + if (tableLog > (uint)(dtd.maxTableLog + 1)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_tableLog_tooLarge)); + } + + dtd.tableType = 0; + dtd.tableLog = (byte)tableLog; + memcpy(DTable, &dtd, (uint)sizeof(DTableDesc)); + } + + { + int n; + uint nextRankStart = 0; + const int unroll = 4; + int nLimit = (int)nbSymbols - unroll + 1; + for (n = 0; n < (int)tableLog + 1; n++) + { + uint curr = nextRankStart; + nextRankStart += wksp->rankVal[n]; + wksp->rankStart[n] = curr; + } + + for (n = 0; n < nLimit; n += unroll) + { + int u; + for (u = 0; u < unroll; ++u) + { + nuint w = wksp->huffWeight[n + u]; + wksp->symbols[wksp->rankStart[w]++] = (byte)(n + u); + } + } + + for (; n < (int)nbSymbols; ++n) + { + nuint w = wksp->huffWeight[n]; + wksp->symbols[wksp->rankStart[w]++] = (byte)n; + } + } + + { + uint w; + int symbol = (int)wksp->rankVal[0]; + int rankStart = 0; + for (w = 1; w < tableLog + 1; ++w) + { + int symbolCount = (int)wksp->rankVal[w]; + int length = 1 << (int)w >> 1; + int uStart = rankStart; + byte nbBits = (byte)(tableLog + 1 - w); + int s; + int u; + switch (length) + { + case 1: + for (s = 0; s < symbolCount; ++s) + { + HUF_DEltX1 D; + D.@byte = wksp->symbols[symbol + s]; + D.nbBits = nbBits; + dt[uStart] = D; + uStart += 1; + } + + break; + case 2: + for (s = 0; s < symbolCount; ++s) + { + HUF_DEltX1 D; + D.@byte = wksp->symbols[symbol + s]; + D.nbBits = nbBits; + dt[uStart + 0] = D; + dt[uStart + 1] = D; + uStart += 2; + } + + break; + case 4: + for (s = 0; s < symbolCount; ++s) + { + ulong D4 = HUF_DEltX1_set4(wksp->symbols[symbol + s], nbBits); + MEM_write64(dt + uStart, D4); + uStart += 4; + } + + break; + case 8: + for (s = 0; s < symbolCount; ++s) + { + ulong D4 = HUF_DEltX1_set4(wksp->symbols[symbol + s], nbBits); + MEM_write64(dt + uStart, D4); + MEM_write64(dt + uStart + 4, D4); + uStart += 8; + } + + break; + default: + for (s = 0; s < symbolCount; ++s) + { + ulong D4 = HUF_DEltX1_set4(wksp->symbols[symbol + s], nbBits); + for (u = 0; u < length; u += 16) + { + MEM_write64(dt + uStart + u + 0, D4); + MEM_write64(dt + uStart + u + 4, D4); + MEM_write64(dt + uStart + u + 8, D4); + MEM_write64(dt + uStart + u + 12, D4); + } + + assert(u == length); + uStart += length; + } + + break; + } + + symbol += symbolCount; + rankStart += symbolCount * length; + } + } + + return iSize; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static byte HUF_decodeSymbolX1(BIT_DStream_t* Dstream, HUF_DEltX1* dt, uint dtLog) + { + /* note : dtLog >= 1 */ + nuint val = BIT_lookBitsFast(Dstream, dtLog); + byte c = dt[val].@byte; + BIT_skipBits(Dstream, dt[val].nbBits); + return c; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint HUF_decodeStreamX1( + byte* p, + BIT_DStream_t* bitDPtr, + byte* pEnd, + HUF_DEltX1* dt, + uint dtLog + ) + { + byte* pStart = p; + if (pEnd - p > 3) + { + while ( + BIT_reloadDStream(bitDPtr) == BIT_DStream_status.BIT_DStream_unfinished + && p < pEnd - 3 + ) + { + if (MEM_64bits) + { + *p++ = HUF_decodeSymbolX1(bitDPtr, dt, dtLog); + } + + *p++ = HUF_decodeSymbolX1(bitDPtr, dt, dtLog); + if (MEM_64bits) + { + *p++ = HUF_decodeSymbolX1(bitDPtr, dt, dtLog); + } + + *p++ = HUF_decodeSymbolX1(bitDPtr, dt, dtLog); + } + } + else + { + BIT_reloadDStream(bitDPtr); + } + + if (MEM_32bits) + { + while ( + BIT_reloadDStream(bitDPtr) == BIT_DStream_status.BIT_DStream_unfinished && p < pEnd + ) + { + *p++ = HUF_decodeSymbolX1(bitDPtr, dt, dtLog); + } + } + + while (p < pEnd) + { + *p++ = HUF_decodeSymbolX1(bitDPtr, dt, dtLog); + } + + return (nuint)(pEnd - pStart); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint HUF_decompress1X1_usingDTable_internal_body( + void* dst, + nuint dstSize, + void* cSrc, + nuint cSrcSize, + uint* DTable + ) + { + byte* op = (byte*)dst; + byte* oend = ZSTD_maybeNullPtrAdd(op, (nint)dstSize); + void* dtPtr = DTable + 1; + HUF_DEltX1* dt = (HUF_DEltX1*)dtPtr; + BIT_DStream_t bitD; + DTableDesc dtd = HUF_getDTableDesc(DTable); + uint dtLog = dtd.tableLog; + { + nuint _var_err__ = BIT_initDStream(&bitD, cSrc, cSrcSize); + if (ERR_isError(_var_err__)) + { + return _var_err__; + } + } + + HUF_decodeStreamX1(op, &bitD, oend, dt, dtLog); + if (BIT_endOfDStream(&bitD) == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + return dstSize; + } + + /* HUF_decompress4X1_usingDTable_internal_body(): + * Conditions : + * @dstSize >= 6 + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint HUF_decompress4X1_usingDTable_internal_body( + void* dst, + nuint dstSize, + void* cSrc, + nuint cSrcSize, + uint* DTable + ) + { + if (cSrcSize < 10) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + if (dstSize < 6) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + { + byte* istart = (byte*)cSrc; + byte* ostart = (byte*)dst; + byte* oend = ostart + dstSize; + byte* olimit = oend - 3; + void* dtPtr = DTable + 1; + HUF_DEltX1* dt = (HUF_DEltX1*)dtPtr; + /* Init */ + BIT_DStream_t bitD1; + BIT_DStream_t bitD2; + BIT_DStream_t bitD3; + BIT_DStream_t bitD4; + nuint length1 = MEM_readLE16(istart); + nuint length2 = MEM_readLE16(istart + 2); + nuint length3 = MEM_readLE16(istart + 4); + nuint length4 = cSrcSize - (length1 + length2 + length3 + 6); + /* jumpTable */ + byte* istart1 = istart + 6; + byte* istart2 = istart1 + length1; + byte* istart3 = istart2 + length2; + byte* istart4 = istart3 + length3; + nuint segmentSize = (dstSize + 3) / 4; + byte* opStart2 = ostart + segmentSize; + byte* opStart3 = opStart2 + segmentSize; + byte* opStart4 = opStart3 + segmentSize; + byte* op1 = ostart; + byte* op2 = opStart2; + byte* op3 = opStart3; + byte* op4 = opStart4; + DTableDesc dtd = HUF_getDTableDesc(DTable); + uint dtLog = dtd.tableLog; + uint endSignal = 1; + if (length4 > cSrcSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + if (opStart4 > oend) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + assert(dstSize >= 6); + { + nuint _var_err__ = BIT_initDStream(&bitD1, istart1, length1); + if (ERR_isError(_var_err__)) + { + return _var_err__; + } + } + + { + nuint _var_err__ = BIT_initDStream(&bitD2, istart2, length2); + if (ERR_isError(_var_err__)) + { + return _var_err__; + } + } + + { + nuint _var_err__ = BIT_initDStream(&bitD3, istart3, length3); + if (ERR_isError(_var_err__)) + { + return _var_err__; + } + } + + { + nuint _var_err__ = BIT_initDStream(&bitD4, istart4, length4); + if (ERR_isError(_var_err__)) + { + return _var_err__; + } + } + + if ((nuint)(oend - op4) >= (nuint)sizeof(nuint)) + { + for (; (endSignal & (uint)(op4 < olimit ? 1 : 0)) != 0; ) + { + if (MEM_64bits) + { + *op1++ = HUF_decodeSymbolX1(&bitD1, dt, dtLog); + } + + if (MEM_64bits) + { + *op2++ = HUF_decodeSymbolX1(&bitD2, dt, dtLog); + } + + if (MEM_64bits) + { + *op3++ = HUF_decodeSymbolX1(&bitD3, dt, dtLog); + } + + if (MEM_64bits) + { + *op4++ = HUF_decodeSymbolX1(&bitD4, dt, dtLog); + } + + *op1++ = HUF_decodeSymbolX1(&bitD1, dt, dtLog); + *op2++ = HUF_decodeSymbolX1(&bitD2, dt, dtLog); + *op3++ = HUF_decodeSymbolX1(&bitD3, dt, dtLog); + *op4++ = HUF_decodeSymbolX1(&bitD4, dt, dtLog); + if (MEM_64bits) + { + *op1++ = HUF_decodeSymbolX1(&bitD1, dt, dtLog); + } + + if (MEM_64bits) + { + *op2++ = HUF_decodeSymbolX1(&bitD2, dt, dtLog); + } + + if (MEM_64bits) + { + *op3++ = HUF_decodeSymbolX1(&bitD3, dt, dtLog); + } + + if (MEM_64bits) + { + *op4++ = HUF_decodeSymbolX1(&bitD4, dt, dtLog); + } + + *op1++ = HUF_decodeSymbolX1(&bitD1, dt, dtLog); + *op2++ = HUF_decodeSymbolX1(&bitD2, dt, dtLog); + *op3++ = HUF_decodeSymbolX1(&bitD3, dt, dtLog); + *op4++ = HUF_decodeSymbolX1(&bitD4, dt, dtLog); + endSignal &= + BIT_reloadDStreamFast(&bitD1) == BIT_DStream_status.BIT_DStream_unfinished + ? 1U + : 0U; + endSignal &= + BIT_reloadDStreamFast(&bitD2) == BIT_DStream_status.BIT_DStream_unfinished + ? 1U + : 0U; + endSignal &= + BIT_reloadDStreamFast(&bitD3) == BIT_DStream_status.BIT_DStream_unfinished + ? 1U + : 0U; + endSignal &= + BIT_reloadDStreamFast(&bitD4) == BIT_DStream_status.BIT_DStream_unfinished + ? 1U + : 0U; + } + } + + if (op1 > opStart2) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + if (op2 > opStart3) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + if (op3 > opStart4) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + HUF_decodeStreamX1(op1, &bitD1, opStart2, dt, dtLog); + HUF_decodeStreamX1(op2, &bitD2, opStart3, dt, dtLog); + HUF_decodeStreamX1(op3, &bitD3, opStart4, dt, dtLog); + HUF_decodeStreamX1(op4, &bitD4, oend, dt, dtLog); + { + uint endCheck = + BIT_endOfDStream(&bitD1) + & BIT_endOfDStream(&bitD2) + & BIT_endOfDStream(&bitD3) + & BIT_endOfDStream(&bitD4); + if (endCheck == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + } + + return dstSize; + } + } + + private static nuint HUF_decompress4X1_usingDTable_internal_default( + void* dst, + nuint dstSize, + void* cSrc, + nuint cSrcSize, + uint* DTable + ) + { + return HUF_decompress4X1_usingDTable_internal_body(dst, dstSize, cSrc, cSrcSize, DTable); + } + + private static void HUF_decompress4X1_usingDTable_internal_fast_c_loop( + HUF_DecompressFastArgs* args + ) + { + ulong bits0, + bits1, + bits2, + bits3; + byte* ip0, + ip1, + ip2, + ip3; + byte* op0, + op1, + op2, + op3; + ushort* dtable = (ushort*)args->dt; + byte* oend = args->oend; + byte* ilowest = args->ilowest; + bits0 = args->bits[0]; + bits1 = args->bits[1]; + bits2 = args->bits[2]; + bits3 = args->bits[3]; + ip0 = args->ip.e0; + ip1 = args->ip.e1; + ip2 = args->ip.e2; + ip3 = args->ip.e3; + op0 = args->op.e0; + op1 = args->op.e1; + op2 = args->op.e2; + op3 = args->op.e3; + assert(BitConverter.IsLittleEndian); + assert(!MEM_32bits); + for (; ; ) + { + byte* olimit; + { + assert(op0 <= op1); + assert(ip0 >= ilowest); + } + + { + assert(op1 <= op2); + assert(ip1 >= ilowest); + } + + { + assert(op2 <= op3); + assert(ip2 >= ilowest); + } + + { + assert(op3 <= oend); + assert(ip3 >= ilowest); + } + + { + /* Each iteration produces 5 output symbols per stream */ + nuint oiters = (nuint)(oend - op3) / 5; + /* Each iteration consumes up to 11 bits * 5 = 55 bits < 7 bytes + * per stream. + */ + nuint iiters = (nuint)(ip0 - ilowest) / 7; + /* We can safely run iters iterations before running bounds checks */ + nuint iters = oiters < iiters ? oiters : iiters; + nuint symbols = iters * 5; + olimit = op3 + symbols; + if (op3 == olimit) + { + break; + } + + { + if (ip1 < ip0) + { + goto _out; + } + } + + { + if (ip2 < ip1) + { + goto _out; + } + } + + { + if (ip3 < ip2) + { + goto _out; + } + } + } + + { + assert(ip1 >= ip0); + } + + { + assert(ip2 >= ip1); + } + + { + assert(ip3 >= ip2); + } + + do + { + { + { + /* Decode 5 symbols in each of the 4 streams */ + int index = (int)(bits0 >> 53); + int entry = dtable[index]; + bits0 <<= entry & 0x3F; + op0[0] = (byte)(entry >> 8 & 0xFF); + } + + { + int index = (int)(bits1 >> 53); + int entry = dtable[index]; + bits1 <<= entry & 0x3F; + op1[0] = (byte)(entry >> 8 & 0xFF); + } + + { + int index = (int)(bits2 >> 53); + int entry = dtable[index]; + bits2 <<= entry & 0x3F; + op2[0] = (byte)(entry >> 8 & 0xFF); + } + + { + int index = (int)(bits3 >> 53); + int entry = dtable[index]; + bits3 <<= entry & 0x3F; + op3[0] = (byte)(entry >> 8 & 0xFF); + } + } + + { + { + int index = (int)(bits0 >> 53); + int entry = dtable[index]; + bits0 <<= entry & 0x3F; + op0[1] = (byte)(entry >> 8 & 0xFF); + } + + { + int index = (int)(bits1 >> 53); + int entry = dtable[index]; + bits1 <<= entry & 0x3F; + op1[1] = (byte)(entry >> 8 & 0xFF); + } + + { + int index = (int)(bits2 >> 53); + int entry = dtable[index]; + bits2 <<= entry & 0x3F; + op2[1] = (byte)(entry >> 8 & 0xFF); + } + + { + int index = (int)(bits3 >> 53); + int entry = dtable[index]; + bits3 <<= entry & 0x3F; + op3[1] = (byte)(entry >> 8 & 0xFF); + } + } + + { + { + int index = (int)(bits0 >> 53); + int entry = dtable[index]; + bits0 <<= entry & 0x3F; + op0[2] = (byte)(entry >> 8 & 0xFF); + } + + { + int index = (int)(bits1 >> 53); + int entry = dtable[index]; + bits1 <<= entry & 0x3F; + op1[2] = (byte)(entry >> 8 & 0xFF); + } + + { + int index = (int)(bits2 >> 53); + int entry = dtable[index]; + bits2 <<= entry & 0x3F; + op2[2] = (byte)(entry >> 8 & 0xFF); + } + + { + int index = (int)(bits3 >> 53); + int entry = dtable[index]; + bits3 <<= entry & 0x3F; + op3[2] = (byte)(entry >> 8 & 0xFF); + } + } + + { + { + int index = (int)(bits0 >> 53); + int entry = dtable[index]; + bits0 <<= entry & 0x3F; + op0[3] = (byte)(entry >> 8 & 0xFF); + } + + { + int index = (int)(bits1 >> 53); + int entry = dtable[index]; + bits1 <<= entry & 0x3F; + op1[3] = (byte)(entry >> 8 & 0xFF); + } + + { + int index = (int)(bits2 >> 53); + int entry = dtable[index]; + bits2 <<= entry & 0x3F; + op2[3] = (byte)(entry >> 8 & 0xFF); + } + + { + int index = (int)(bits3 >> 53); + int entry = dtable[index]; + bits3 <<= entry & 0x3F; + op3[3] = (byte)(entry >> 8 & 0xFF); + } + } + + { + { + int index = (int)(bits0 >> 53); + int entry = dtable[index]; + bits0 <<= entry & 0x3F; + op0[4] = (byte)(entry >> 8 & 0xFF); + } + + { + int index = (int)(bits1 >> 53); + int entry = dtable[index]; + bits1 <<= entry & 0x3F; + op1[4] = (byte)(entry >> 8 & 0xFF); + } + + { + int index = (int)(bits2 >> 53); + int entry = dtable[index]; + bits2 <<= entry & 0x3F; + op2[4] = (byte)(entry >> 8 & 0xFF); + } + + { + int index = (int)(bits3 >> 53); + int entry = dtable[index]; + bits3 <<= entry & 0x3F; + op3[4] = (byte)(entry >> 8 & 0xFF); + } + } + + { + { + /* Reload each of the 4 the bitstreams */ + int ctz = (int)ZSTD_countTrailingZeros64(bits0); + int nbBits = ctz & 7; + int nbBytes = ctz >> 3; + op0 += 5; + ip0 -= nbBytes; + bits0 = MEM_read64(ip0) | 1; + bits0 <<= nbBits; + } + + { + int ctz = (int)ZSTD_countTrailingZeros64(bits1); + int nbBits = ctz & 7; + int nbBytes = ctz >> 3; + op1 += 5; + ip1 -= nbBytes; + bits1 = MEM_read64(ip1) | 1; + bits1 <<= nbBits; + } + + { + int ctz = (int)ZSTD_countTrailingZeros64(bits2); + int nbBits = ctz & 7; + int nbBytes = ctz >> 3; + op2 += 5; + ip2 -= nbBytes; + bits2 = MEM_read64(ip2) | 1; + bits2 <<= nbBits; + } + + { + int ctz = (int)ZSTD_countTrailingZeros64(bits3); + int nbBits = ctz & 7; + int nbBytes = ctz >> 3; + op3 += 5; + ip3 -= nbBytes; + bits3 = MEM_read64(ip3) | 1; + bits3 <<= nbBits; + } + } + } while (op3 < olimit); + } + + _out: + args->bits[0] = bits0; + args->bits[1] = bits1; + args->bits[2] = bits2; + args->bits[3] = bits3; + args->ip.e0 = ip0; + args->ip.e1 = ip1; + args->ip.e2 = ip2; + args->ip.e3 = ip3; + args->op.e0 = op0; + args->op.e1 = op1; + args->op.e2 = op2; + args->op.e3 = op3; + } + + /** + * @returns @p dstSize on success (>= 6) + * 0 if the fallback implementation should be used + * An error if an error occurred + */ + private static nuint HUF_decompress4X1_usingDTable_internal_fast( + void* dst, + nuint dstSize, + void* cSrc, + nuint cSrcSize, + uint* DTable, + void* loopFn + ) + { + void* dt = DTable + 1; + byte* ilowest = (byte*)cSrc; + byte* oend = ZSTD_maybeNullPtrAdd((byte*)dst, (nint)dstSize); + HUF_DecompressFastArgs args; + { + nuint ret = HUF_DecompressFastArgs_init(&args, dst, dstSize, cSrc, cSrcSize, DTable); + { + nuint err_code = ret; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (ret == 0) + { + return 0; + } + } + + assert(args.ip.e0 >= args.ilowest); + ((delegate* managed)loopFn)(&args); + assert(args.ip.e0 >= ilowest); + assert(args.ip.e0 >= ilowest); + assert(args.ip.e1 >= ilowest); + assert(args.ip.e2 >= ilowest); + assert(args.ip.e3 >= ilowest); + assert(args.op.e3 <= oend); + assert(ilowest == args.ilowest); + assert(ilowest + 6 == args.iend.e0); + { + nuint segmentSize = (dstSize + 3) / 4; + byte* segmentEnd = (byte*)dst; + int i; + for (i = 0; i < 4; ++i) + { + BIT_DStream_t bit; + if (segmentSize <= (nuint)(oend - segmentEnd)) + { + segmentEnd += segmentSize; + } + else + { + segmentEnd = oend; + } + + { + nuint err_code = HUF_initRemainingDStream(&bit, &args, i, segmentEnd); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + (&args.op.e0)[i] += HUF_decodeStreamX1( + (&args.op.e0)[i], + &bit, + segmentEnd, + (HUF_DEltX1*)dt, + 11 + ); + if ((&args.op.e0)[i] != segmentEnd) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + } + } + + assert(dstSize != 0); + return dstSize; + } + + private static nuint HUF_decompress1X1_usingDTable_internal( + void* dst, + nuint dstSize, + void* cSrc, + nuint cSrcSize, + uint* DTable, + int flags + ) + { + return HUF_decompress1X1_usingDTable_internal_body(dst, dstSize, cSrc, cSrcSize, DTable); + } + + private static nuint HUF_decompress4X1_usingDTable_internal( + void* dst, + nuint dstSize, + void* cSrc, + nuint cSrcSize, + uint* DTable, + int flags + ) + { + void* fallbackFn = (delegate* managed)( + &HUF_decompress4X1_usingDTable_internal_default + ); + void* loopFn = (delegate* managed)( + &HUF_decompress4X1_usingDTable_internal_fast_c_loop + ); + if ((flags & (int)HUF_flags_e.HUF_flags_disableFast) == 0) + { + nuint ret = HUF_decompress4X1_usingDTable_internal_fast( + dst, + dstSize, + cSrc, + cSrcSize, + DTable, + loopFn + ); + if (ret != 0) + { + return ret; + } + } + + return ((delegate* managed)fallbackFn)( + dst, + dstSize, + cSrc, + cSrcSize, + DTable + ); + } + + private static nuint HUF_decompress4X1_DCtx_wksp( + uint* dctx, + void* dst, + nuint dstSize, + void* cSrc, + nuint cSrcSize, + void* workSpace, + nuint wkspSize, + int flags + ) + { + byte* ip = (byte*)cSrc; + nuint hSize = HUF_readDTableX1_wksp(dctx, cSrc, cSrcSize, workSpace, wkspSize, flags); + if (ERR_isError(hSize)) + { + return hSize; + } + + if (hSize >= cSrcSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + ip += hSize; + cSrcSize -= hSize; + return HUF_decompress4X1_usingDTable_internal(dst, dstSize, ip, cSrcSize, dctx, flags); + } + + /** + * Constructs a HUF_DEltX2 in a U32. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint HUF_buildDEltX2U32(uint symbol, uint nbBits, uint baseSeq, int level) + { + uint seq; + if (BitConverter.IsLittleEndian) + { + seq = level == 1 ? symbol : baseSeq + (symbol << 8); + return seq + (nbBits << 16) + ((uint)level << 24); + } + else + { + seq = level == 1 ? symbol << 8 : (baseSeq << 8) + symbol; + return (seq << 16) + (nbBits << 8) + (uint)level; + } + } + + /** + * Constructs a HUF_DEltX2. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static HUF_DEltX2 HUF_buildDEltX2(uint symbol, uint nbBits, uint baseSeq, int level) + { + HUF_DEltX2 DElt; + uint val = HUF_buildDEltX2U32(symbol, nbBits, baseSeq, level); + memcpy(&DElt, &val, sizeof(uint)); + return DElt; + } + + /** + * Constructs 2 HUF_DEltX2s and packs them into a U64. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong HUF_buildDEltX2U64(uint symbol, uint nbBits, ushort baseSeq, int level) + { + uint DElt = HUF_buildDEltX2U32(symbol, nbBits, baseSeq, level); + return DElt + ((ulong)DElt << 32); + } + + /** + * Fills the DTable rank with all the symbols from [begin, end) that are each + * nbBits long. + * + * @param DTableRank The start of the rank in the DTable. + * @param begin The first symbol to fill (inclusive). + * @param end The last symbol to fill (exclusive). + * @param nbBits Each symbol is nbBits long. + * @param tableLog The table log. + * @param baseSeq If level == 1 { 0 } else { the first level symbol } + * @param level The level in the table. Must be 1 or 2. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void HUF_fillDTableX2ForWeight( + HUF_DEltX2* DTableRank, + sortedSymbol_t* begin, + sortedSymbol_t* end, + uint nbBits, + uint tableLog, + ushort baseSeq, + int level + ) + { + /* quiet static-analyzer */ + uint length = 1U << (int)(tableLog - nbBits & 0x1F); + sortedSymbol_t* ptr; + assert(level >= 1 && level <= 2); + switch (length) + { + case 1: + for (ptr = begin; ptr != end; ++ptr) + { + HUF_DEltX2 DElt = HUF_buildDEltX2(ptr->symbol, nbBits, baseSeq, level); + *DTableRank++ = DElt; + } + + break; + case 2: + for (ptr = begin; ptr != end; ++ptr) + { + HUF_DEltX2 DElt = HUF_buildDEltX2(ptr->symbol, nbBits, baseSeq, level); + DTableRank[0] = DElt; + DTableRank[1] = DElt; + DTableRank += 2; + } + + break; + case 4: + for (ptr = begin; ptr != end; ++ptr) + { + ulong DEltX2 = HUF_buildDEltX2U64(ptr->symbol, nbBits, baseSeq, level); + memcpy(DTableRank + 0, &DEltX2, sizeof(ulong)); + memcpy(DTableRank + 2, &DEltX2, sizeof(ulong)); + DTableRank += 4; + } + + break; + case 8: + for (ptr = begin; ptr != end; ++ptr) + { + ulong DEltX2 = HUF_buildDEltX2U64(ptr->symbol, nbBits, baseSeq, level); + memcpy(DTableRank + 0, &DEltX2, sizeof(ulong)); + memcpy(DTableRank + 2, &DEltX2, sizeof(ulong)); + memcpy(DTableRank + 4, &DEltX2, sizeof(ulong)); + memcpy(DTableRank + 6, &DEltX2, sizeof(ulong)); + DTableRank += 8; + } + + break; + default: + for (ptr = begin; ptr != end; ++ptr) + { + ulong DEltX2 = HUF_buildDEltX2U64(ptr->symbol, nbBits, baseSeq, level); + HUF_DEltX2* DTableRankEnd = DTableRank + length; + for (; DTableRank != DTableRankEnd; DTableRank += 8) + { + memcpy(DTableRank + 0, &DEltX2, sizeof(ulong)); + memcpy(DTableRank + 2, &DEltX2, sizeof(ulong)); + memcpy(DTableRank + 4, &DEltX2, sizeof(ulong)); + memcpy(DTableRank + 6, &DEltX2, sizeof(ulong)); + } + } + + break; + } + } + + /* HUF_fillDTableX2Level2() : + * `rankValOrigin` must be a table of at least (HUF_TABLELOG_MAX + 1) U32 */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void HUF_fillDTableX2Level2( + HUF_DEltX2* DTable, + uint targetLog, + uint consumedBits, + uint* rankVal, + int minWeight, + int maxWeight1, + sortedSymbol_t* sortedSymbols, + uint* rankStart, + uint nbBitsBaseline, + ushort baseSeq + ) + { + if (minWeight > 1) + { + /* quiet static-analyzer */ + uint length = 1U << (int)(targetLog - consumedBits & 0x1F); + /* baseSeq */ + ulong DEltX2 = HUF_buildDEltX2U64(baseSeq, consumedBits, 0, 1); + int skipSize = (int)rankVal[minWeight]; + assert(length > 1); + assert((uint)skipSize < length); + switch (length) + { + case 2: + assert(skipSize == 1); + memcpy(DTable, &DEltX2, sizeof(ulong)); + break; + case 4: + assert(skipSize <= 4); + memcpy(DTable + 0, &DEltX2, sizeof(ulong)); + memcpy(DTable + 2, &DEltX2, sizeof(ulong)); + break; + default: + { + int i; + for (i = 0; i < skipSize; i += 8) + { + memcpy(DTable + i + 0, &DEltX2, sizeof(ulong)); + memcpy(DTable + i + 2, &DEltX2, sizeof(ulong)); + memcpy(DTable + i + 4, &DEltX2, sizeof(ulong)); + memcpy(DTable + i + 6, &DEltX2, sizeof(ulong)); + } + } + + break; + } + } + + { + int w; + for (w = minWeight; w < maxWeight1; ++w) + { + int begin = (int)rankStart[w]; + int end = (int)rankStart[w + 1]; + uint nbBits = nbBitsBaseline - (uint)w; + uint totalBits = nbBits + consumedBits; + HUF_fillDTableX2ForWeight( + DTable + rankVal[w], + sortedSymbols + begin, + sortedSymbols + end, + totalBits, + targetLog, + baseSeq, + 2 + ); + } + } + } + + private static void HUF_fillDTableX2( + HUF_DEltX2* DTable, + uint targetLog, + sortedSymbol_t* sortedList, + uint* rankStart, + rankValCol_t* rankValOrigin, + uint maxWeight, + uint nbBitsBaseline + ) + { + uint* rankVal = (uint*)&rankValOrigin[0]; + /* note : targetLog >= srcLog, hence scaleLog <= 1 */ + int scaleLog = (int)(nbBitsBaseline - targetLog); + uint minBits = nbBitsBaseline - maxWeight; + int w; + int wEnd = (int)maxWeight + 1; + for (w = 1; w < wEnd; ++w) + { + int begin = (int)rankStart[w]; + int end = (int)rankStart[w + 1]; + uint nbBits = nbBitsBaseline - (uint)w; + if (targetLog - nbBits >= minBits) + { + /* Enough room for a second symbol. */ + int start = (int)rankVal[w]; + /* quiet static-analyzer */ + uint length = 1U << (int)(targetLog - nbBits & 0x1F); + int minWeight = (int)(nbBits + (uint)scaleLog); + int s; + if (minWeight < 1) + { + minWeight = 1; + } + + for (s = begin; s != end; ++s) + { + HUF_fillDTableX2Level2( + DTable + start, + targetLog, + nbBits, + (uint*)&rankValOrigin[nbBits], + minWeight, + wEnd, + sortedList, + rankStart, + nbBitsBaseline, + sortedList[s].symbol + ); + start += (int)length; + } + } + else + { + HUF_fillDTableX2ForWeight( + DTable + rankVal[w], + sortedList + begin, + sortedList + end, + nbBits, + targetLog, + 0, + 1 + ); + } + } + } + + private static nuint HUF_readDTableX2_wksp( + uint* DTable, + void* src, + nuint srcSize, + void* workSpace, + nuint wkspSize, + int flags + ) + { + uint tableLog, + maxW, + nbSymbols; + DTableDesc dtd = HUF_getDTableDesc(DTable); + uint maxTableLog = dtd.maxTableLog; + nuint iSize; + /* force compiler to avoid strict-aliasing */ + void* dtPtr = DTable + 1; + HUF_DEltX2* dt = (HUF_DEltX2*)dtPtr; + uint* rankStart; + HUF_ReadDTableX2_Workspace* wksp = (HUF_ReadDTableX2_Workspace*)workSpace; + if ((nuint)sizeof(HUF_ReadDTableX2_Workspace) > wkspSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + + rankStart = wksp->rankStart0 + 1; + memset(wksp->rankStats, 0, sizeof(uint) * 13); + memset(wksp->rankStart0, 0, sizeof(uint) * 15); + if (maxTableLog > 12) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_tableLog_tooLarge)); + } + + iSize = HUF_readStats_wksp( + wksp->weightList, + 255 + 1, + wksp->rankStats, + &nbSymbols, + &tableLog, + src, + srcSize, + wksp->calleeWksp, + sizeof(uint) * 219, + flags + ); + if (ERR_isError(iSize)) + { + return iSize; + } + + if (tableLog > maxTableLog) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_tableLog_tooLarge)); + } + + if (tableLog <= 11 && maxTableLog > 11) + { + maxTableLog = 11; + } + + for (maxW = tableLog; wksp->rankStats[maxW] == 0; maxW--) { } + + { + uint w, + nextRankStart = 0; + for (w = 1; w < maxW + 1; w++) + { + uint curr = nextRankStart; + nextRankStart += wksp->rankStats[w]; + rankStart[w] = curr; + } + + rankStart[0] = nextRankStart; + rankStart[maxW + 1] = nextRankStart; + } + + { + uint s; + for (s = 0; s < nbSymbols; s++) + { + uint w = wksp->weightList[s]; + uint r = rankStart[w]++; + (&wksp->sortedSymbol.e0)[r].symbol = (byte)s; + } + + rankStart[0] = 0; + } + + { + uint* rankVal0 = (uint*)&wksp->rankVal.e0; + { + /* tableLog <= maxTableLog */ + int rescale = (int)(maxTableLog - tableLog - 1); + uint nextRankVal = 0; + uint w; + for (w = 1; w < maxW + 1; w++) + { + uint curr = nextRankVal; + nextRankVal += wksp->rankStats[w] << (int)(w + (uint)rescale); + rankVal0[w] = curr; + } + } + + { + uint minBits = tableLog + 1 - maxW; + uint consumed; + for (consumed = minBits; consumed < maxTableLog - minBits + 1; consumed++) + { + uint* rankValPtr = (uint*)&(&wksp->rankVal.e0)[consumed]; + uint w; + for (w = 1; w < maxW + 1; w++) + { + rankValPtr[w] = rankVal0[w] >> (int)consumed; + } + } + } + } + + HUF_fillDTableX2( + dt, + maxTableLog, + &wksp->sortedSymbol.e0, + wksp->rankStart0, + &wksp->rankVal.e0, + maxW, + tableLog + 1 + ); + dtd.tableLog = (byte)maxTableLog; + dtd.tableType = 1; + memcpy(DTable, &dtd, (uint)sizeof(DTableDesc)); + return iSize; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint HUF_decodeSymbolX2( + void* op, + BIT_DStream_t* DStream, + HUF_DEltX2* dt, + uint dtLog + ) + { + /* note : dtLog >= 1 */ + nuint val = BIT_lookBitsFast(DStream, dtLog); + memcpy(op, &dt[val].sequence, 2); + BIT_skipBits(DStream, dt[val].nbBits); + return dt[val].length; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint HUF_decodeLastSymbolX2( + void* op, + BIT_DStream_t* DStream, + HUF_DEltX2* dt, + uint dtLog + ) + { + /* note : dtLog >= 1 */ + nuint val = BIT_lookBitsFast(DStream, dtLog); + memcpy(op, &dt[val].sequence, 1); + if (dt[val].length == 1) + { + BIT_skipBits(DStream, dt[val].nbBits); + } + else + { + if (DStream->bitsConsumed < (uint)(sizeof(nuint) * 8)) + { + BIT_skipBits(DStream, dt[val].nbBits); + if (DStream->bitsConsumed > (uint)(sizeof(nuint) * 8)) + { + DStream->bitsConsumed = (uint)(sizeof(nuint) * 8); + } + } + } + + return 1; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint HUF_decodeStreamX2( + byte* p, + BIT_DStream_t* bitDPtr, + byte* pEnd, + HUF_DEltX2* dt, + uint dtLog + ) + { + byte* pStart = p; + if ((nuint)(pEnd - p) >= (nuint)sizeof(nuint)) + { + if (dtLog <= 11 && MEM_64bits) + { + while ( + BIT_reloadDStream(bitDPtr) == BIT_DStream_status.BIT_DStream_unfinished + && p < pEnd - 9 + ) + { + p += HUF_decodeSymbolX2(p, bitDPtr, dt, dtLog); + p += HUF_decodeSymbolX2(p, bitDPtr, dt, dtLog); + p += HUF_decodeSymbolX2(p, bitDPtr, dt, dtLog); + p += HUF_decodeSymbolX2(p, bitDPtr, dt, dtLog); + p += HUF_decodeSymbolX2(p, bitDPtr, dt, dtLog); + } + } + else + { + while ( + BIT_reloadDStream(bitDPtr) == BIT_DStream_status.BIT_DStream_unfinished + && p < pEnd - (sizeof(nuint) - 1) + ) + { + if (MEM_64bits) + { + p += HUF_decodeSymbolX2(p, bitDPtr, dt, dtLog); + } + + p += HUF_decodeSymbolX2(p, bitDPtr, dt, dtLog); + if (MEM_64bits) + { + p += HUF_decodeSymbolX2(p, bitDPtr, dt, dtLog); + } + + p += HUF_decodeSymbolX2(p, bitDPtr, dt, dtLog); + } + } + } + else + { + BIT_reloadDStream(bitDPtr); + } + + if ((nuint)(pEnd - p) >= 2) + { + while ( + BIT_reloadDStream(bitDPtr) == BIT_DStream_status.BIT_DStream_unfinished + && p <= pEnd - 2 + ) + { + p += HUF_decodeSymbolX2(p, bitDPtr, dt, dtLog); + } + + while (p <= pEnd - 2) + { + p += HUF_decodeSymbolX2(p, bitDPtr, dt, dtLog); + } + } + + if (p < pEnd) + { + p += HUF_decodeLastSymbolX2(p, bitDPtr, dt, dtLog); + } + + return (nuint)(p - pStart); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint HUF_decompress1X2_usingDTable_internal_body( + void* dst, + nuint dstSize, + void* cSrc, + nuint cSrcSize, + uint* DTable + ) + { + BIT_DStream_t bitD; + { + /* Init */ + nuint _var_err__ = BIT_initDStream(&bitD, cSrc, cSrcSize); + if (ERR_isError(_var_err__)) + { + return _var_err__; + } + } + + { + byte* ostart = (byte*)dst; + byte* oend = ZSTD_maybeNullPtrAdd(ostart, (nint)dstSize); + /* force compiler to not use strict-aliasing */ + void* dtPtr = DTable + 1; + HUF_DEltX2* dt = (HUF_DEltX2*)dtPtr; + DTableDesc dtd = HUF_getDTableDesc(DTable); + HUF_decodeStreamX2(ostart, &bitD, oend, dt, dtd.tableLog); + } + + if (BIT_endOfDStream(&bitD) == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + return dstSize; + } + + /* HUF_decompress4X2_usingDTable_internal_body(): + * Conditions: + * @dstSize >= 6 + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint HUF_decompress4X2_usingDTable_internal_body( + void* dst, + nuint dstSize, + void* cSrc, + nuint cSrcSize, + uint* DTable + ) + { + if (cSrcSize < 10) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + if (dstSize < 6) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + { + byte* istart = (byte*)cSrc; + byte* ostart = (byte*)dst; + byte* oend = ostart + dstSize; + byte* olimit = oend - (sizeof(nuint) - 1); + void* dtPtr = DTable + 1; + HUF_DEltX2* dt = (HUF_DEltX2*)dtPtr; + /* Init */ + BIT_DStream_t bitD1; + BIT_DStream_t bitD2; + BIT_DStream_t bitD3; + BIT_DStream_t bitD4; + nuint length1 = MEM_readLE16(istart); + nuint length2 = MEM_readLE16(istart + 2); + nuint length3 = MEM_readLE16(istart + 4); + nuint length4 = cSrcSize - (length1 + length2 + length3 + 6); + /* jumpTable */ + byte* istart1 = istart + 6; + byte* istart2 = istart1 + length1; + byte* istart3 = istart2 + length2; + byte* istart4 = istart3 + length3; + nuint segmentSize = (dstSize + 3) / 4; + byte* opStart2 = ostart + segmentSize; + byte* opStart3 = opStart2 + segmentSize; + byte* opStart4 = opStart3 + segmentSize; + byte* op1 = ostart; + byte* op2 = opStart2; + byte* op3 = opStart3; + byte* op4 = opStart4; + uint endSignal = 1; + DTableDesc dtd = HUF_getDTableDesc(DTable); + uint dtLog = dtd.tableLog; + if (length4 > cSrcSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + if (opStart4 > oend) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + assert(dstSize >= 6); + { + nuint _var_err__ = BIT_initDStream(&bitD1, istart1, length1); + if (ERR_isError(_var_err__)) + { + return _var_err__; + } + } + + { + nuint _var_err__ = BIT_initDStream(&bitD2, istart2, length2); + if (ERR_isError(_var_err__)) + { + return _var_err__; + } + } + + { + nuint _var_err__ = BIT_initDStream(&bitD3, istart3, length3); + if (ERR_isError(_var_err__)) + { + return _var_err__; + } + } + + { + nuint _var_err__ = BIT_initDStream(&bitD4, istart4, length4); + if (ERR_isError(_var_err__)) + { + return _var_err__; + } + } + + if ((nuint)(oend - op4) >= (nuint)sizeof(nuint)) + { + for (; (endSignal & (uint)(op4 < olimit ? 1 : 0)) != 0; ) + { + if (MEM_64bits) + { + op1 += HUF_decodeSymbolX2(op1, &bitD1, dt, dtLog); + } + + op1 += HUF_decodeSymbolX2(op1, &bitD1, dt, dtLog); + if (MEM_64bits) + { + op1 += HUF_decodeSymbolX2(op1, &bitD1, dt, dtLog); + } + + op1 += HUF_decodeSymbolX2(op1, &bitD1, dt, dtLog); + if (MEM_64bits) + { + op2 += HUF_decodeSymbolX2(op2, &bitD2, dt, dtLog); + } + + op2 += HUF_decodeSymbolX2(op2, &bitD2, dt, dtLog); + if (MEM_64bits) + { + op2 += HUF_decodeSymbolX2(op2, &bitD2, dt, dtLog); + } + + op2 += HUF_decodeSymbolX2(op2, &bitD2, dt, dtLog); + endSignal &= + BIT_reloadDStreamFast(&bitD1) == BIT_DStream_status.BIT_DStream_unfinished + ? 1U + : 0U; + endSignal &= + BIT_reloadDStreamFast(&bitD2) == BIT_DStream_status.BIT_DStream_unfinished + ? 1U + : 0U; + if (MEM_64bits) + { + op3 += HUF_decodeSymbolX2(op3, &bitD3, dt, dtLog); + } + + op3 += HUF_decodeSymbolX2(op3, &bitD3, dt, dtLog); + if (MEM_64bits) + { + op3 += HUF_decodeSymbolX2(op3, &bitD3, dt, dtLog); + } + + op3 += HUF_decodeSymbolX2(op3, &bitD3, dt, dtLog); + if (MEM_64bits) + { + op4 += HUF_decodeSymbolX2(op4, &bitD4, dt, dtLog); + } + + op4 += HUF_decodeSymbolX2(op4, &bitD4, dt, dtLog); + if (MEM_64bits) + { + op4 += HUF_decodeSymbolX2(op4, &bitD4, dt, dtLog); + } + + op4 += HUF_decodeSymbolX2(op4, &bitD4, dt, dtLog); + endSignal &= + BIT_reloadDStreamFast(&bitD3) == BIT_DStream_status.BIT_DStream_unfinished + ? 1U + : 0U; + endSignal &= + BIT_reloadDStreamFast(&bitD4) == BIT_DStream_status.BIT_DStream_unfinished + ? 1U + : 0U; + } + } + + if (op1 > opStart2) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + if (op2 > opStart3) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + if (op3 > opStart4) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + HUF_decodeStreamX2(op1, &bitD1, opStart2, dt, dtLog); + HUF_decodeStreamX2(op2, &bitD2, opStart3, dt, dtLog); + HUF_decodeStreamX2(op3, &bitD3, opStart4, dt, dtLog); + HUF_decodeStreamX2(op4, &bitD4, oend, dt, dtLog); + { + uint endCheck = + BIT_endOfDStream(&bitD1) + & BIT_endOfDStream(&bitD2) + & BIT_endOfDStream(&bitD3) + & BIT_endOfDStream(&bitD4); + if (endCheck == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + } + + return dstSize; + } + } + + private static nuint HUF_decompress4X2_usingDTable_internal_default( + void* dst, + nuint dstSize, + void* cSrc, + nuint cSrcSize, + uint* DTable + ) + { + return HUF_decompress4X2_usingDTable_internal_body(dst, dstSize, cSrc, cSrcSize, DTable); + } + + private static void HUF_decompress4X2_usingDTable_internal_fast_c_loop( + HUF_DecompressFastArgs* args + ) + { + ulong bits0, + bits1, + bits2, + bits3; + byte* ip0, + ip1, + ip2, + ip3; + byte* op0, + op1, + op2, + op3; + byte* oend0, + oend1, + oend2, + oend3; + HUF_DEltX2* dtable = (HUF_DEltX2*)args->dt; + byte* ilowest = args->ilowest; + bits0 = args->bits[0]; + bits1 = args->bits[1]; + bits2 = args->bits[2]; + bits3 = args->bits[3]; + ip0 = args->ip.e0; + ip1 = args->ip.e1; + ip2 = args->ip.e2; + ip3 = args->ip.e3; + op0 = args->op.e0; + op1 = args->op.e1; + op2 = args->op.e2; + op3 = args->op.e3; + oend0 = op1; + oend1 = op2; + oend2 = op3; + oend3 = args->oend; + assert(BitConverter.IsLittleEndian); + assert(!MEM_32bits); + for (; ; ) + { + byte* olimit; + { + assert(op0 <= oend0); + assert(ip0 >= ilowest); + } + + { + assert(op1 <= oend1); + assert(ip1 >= ilowest); + } + + { + assert(op2 <= oend2); + assert(ip2 >= ilowest); + } + + { + assert(op3 <= oend3); + assert(ip3 >= ilowest); + } + + { + /* Each loop does 5 table lookups for each of the 4 streams. + * Each table lookup consumes up to 11 bits of input, and produces + * up to 2 bytes of output. + */ + /* We can consume up to 7 bytes of input per iteration per stream. + * We also know that each input pointer is >= ip[0]. So we can run + * iters loops before running out of input. + */ + nuint iters = (nuint)(ip0 - ilowest) / 7; + { + nuint oiters = (nuint)(oend0 - op0) / 10; + iters = iters < oiters ? iters : oiters; + } + + { + nuint oiters = (nuint)(oend1 - op1) / 10; + iters = iters < oiters ? iters : oiters; + } + + { + nuint oiters = (nuint)(oend2 - op2) / 10; + iters = iters < oiters ? iters : oiters; + } + + { + nuint oiters = (nuint)(oend3 - op3) / 10; + iters = iters < oiters ? iters : oiters; + } + + olimit = op3 + iters * 5; + if (op3 == olimit) + { + break; + } + + { + if (ip1 < ip0) + { + goto _out; + } + } + + { + if (ip2 < ip1) + { + goto _out; + } + } + + { + if (ip3 < ip2) + { + goto _out; + } + } + } + + { + assert(ip1 >= ip0); + } + + { + assert(ip2 >= ip1); + } + + { + assert(ip3 >= ip2); + } + + do + { + { + { + /* Decode 5 symbols from each of the first 3 streams. + * The final stream will be decoded during the reload phase + * to reduce register pressure. + */ + int index = (int)(bits0 >> 53); + HUF_DEltX2 entry = dtable[index]; + MEM_write16(op0, entry.sequence); + bits0 <<= entry.nbBits & 0x3F; + op0 += entry.length; + } + + { + int index = (int)(bits1 >> 53); + HUF_DEltX2 entry = dtable[index]; + MEM_write16(op1, entry.sequence); + bits1 <<= entry.nbBits & 0x3F; + op1 += entry.length; + } + + { + int index = (int)(bits2 >> 53); + HUF_DEltX2 entry = dtable[index]; + MEM_write16(op2, entry.sequence); + bits2 <<= entry.nbBits & 0x3F; + op2 += entry.length; + } + } + + { + { + int index = (int)(bits0 >> 53); + HUF_DEltX2 entry = dtable[index]; + MEM_write16(op0, entry.sequence); + bits0 <<= entry.nbBits & 0x3F; + op0 += entry.length; + } + + { + int index = (int)(bits1 >> 53); + HUF_DEltX2 entry = dtable[index]; + MEM_write16(op1, entry.sequence); + bits1 <<= entry.nbBits & 0x3F; + op1 += entry.length; + } + + { + int index = (int)(bits2 >> 53); + HUF_DEltX2 entry = dtable[index]; + MEM_write16(op2, entry.sequence); + bits2 <<= entry.nbBits & 0x3F; + op2 += entry.length; + } + } + + { + { + int index = (int)(bits0 >> 53); + HUF_DEltX2 entry = dtable[index]; + MEM_write16(op0, entry.sequence); + bits0 <<= entry.nbBits & 0x3F; + op0 += entry.length; + } + + { + int index = (int)(bits1 >> 53); + HUF_DEltX2 entry = dtable[index]; + MEM_write16(op1, entry.sequence); + bits1 <<= entry.nbBits & 0x3F; + op1 += entry.length; + } + + { + int index = (int)(bits2 >> 53); + HUF_DEltX2 entry = dtable[index]; + MEM_write16(op2, entry.sequence); + bits2 <<= entry.nbBits & 0x3F; + op2 += entry.length; + } + } + + { + { + int index = (int)(bits0 >> 53); + HUF_DEltX2 entry = dtable[index]; + MEM_write16(op0, entry.sequence); + bits0 <<= entry.nbBits & 0x3F; + op0 += entry.length; + } + + { + int index = (int)(bits1 >> 53); + HUF_DEltX2 entry = dtable[index]; + MEM_write16(op1, entry.sequence); + bits1 <<= entry.nbBits & 0x3F; + op1 += entry.length; + } + + { + int index = (int)(bits2 >> 53); + HUF_DEltX2 entry = dtable[index]; + MEM_write16(op2, entry.sequence); + bits2 <<= entry.nbBits & 0x3F; + op2 += entry.length; + } + } + + { + { + int index = (int)(bits0 >> 53); + HUF_DEltX2 entry = dtable[index]; + MEM_write16(op0, entry.sequence); + bits0 <<= entry.nbBits & 0x3F; + op0 += entry.length; + } + + { + int index = (int)(bits1 >> 53); + HUF_DEltX2 entry = dtable[index]; + MEM_write16(op1, entry.sequence); + bits1 <<= entry.nbBits & 0x3F; + op1 += entry.length; + } + + { + int index = (int)(bits2 >> 53); + HUF_DEltX2 entry = dtable[index]; + MEM_write16(op2, entry.sequence); + bits2 <<= entry.nbBits & 0x3F; + op2 += entry.length; + } + } + + { + /* Decode one symbol from the final stream */ + int index = (int)(bits3 >> 53); + HUF_DEltX2 entry = dtable[index]; + MEM_write16(op3, entry.sequence); + bits3 <<= entry.nbBits & 0x3F; + op3 += entry.length; + } + + { + { + { + /* Decode 4 symbols from the final stream & reload bitstreams. + * The final stream is reloaded last, meaning that all 5 symbols + * are decoded from the final stream before it is reloaded. + */ + int index = (int)(bits3 >> 53); + HUF_DEltX2 entry = dtable[index]; + MEM_write16(op3, entry.sequence); + bits3 <<= entry.nbBits & 0x3F; + op3 += entry.length; + } + + { + int ctz = (int)ZSTD_countTrailingZeros64(bits0); + int nbBits = ctz & 7; + int nbBytes = ctz >> 3; + ip0 -= nbBytes; + bits0 = MEM_read64(ip0) | 1; + bits0 <<= nbBits; + } + } + + { + { + int index = (int)(bits3 >> 53); + HUF_DEltX2 entry = dtable[index]; + MEM_write16(op3, entry.sequence); + bits3 <<= entry.nbBits & 0x3F; + op3 += entry.length; + } + + { + int ctz = (int)ZSTD_countTrailingZeros64(bits1); + int nbBits = ctz & 7; + int nbBytes = ctz >> 3; + ip1 -= nbBytes; + bits1 = MEM_read64(ip1) | 1; + bits1 <<= nbBits; + } + } + + { + { + int index = (int)(bits3 >> 53); + HUF_DEltX2 entry = dtable[index]; + MEM_write16(op3, entry.sequence); + bits3 <<= entry.nbBits & 0x3F; + op3 += entry.length; + } + + { + int ctz = (int)ZSTD_countTrailingZeros64(bits2); + int nbBits = ctz & 7; + int nbBytes = ctz >> 3; + ip2 -= nbBytes; + bits2 = MEM_read64(ip2) | 1; + bits2 <<= nbBits; + } + } + + { + { + int index = (int)(bits3 >> 53); + HUF_DEltX2 entry = dtable[index]; + MEM_write16(op3, entry.sequence); + bits3 <<= entry.nbBits & 0x3F; + op3 += entry.length; + } + + { + int ctz = (int)ZSTD_countTrailingZeros64(bits3); + int nbBits = ctz & 7; + int nbBytes = ctz >> 3; + ip3 -= nbBytes; + bits3 = MEM_read64(ip3) | 1; + bits3 <<= nbBits; + } + } + } + } while (op3 < olimit); + } + + _out: + args->bits[0] = bits0; + args->bits[1] = bits1; + args->bits[2] = bits2; + args->bits[3] = bits3; + args->ip.e0 = ip0; + args->ip.e1 = ip1; + args->ip.e2 = ip2; + args->ip.e3 = ip3; + args->op.e0 = op0; + args->op.e1 = op1; + args->op.e2 = op2; + args->op.e3 = op3; + } + + private static nuint HUF_decompress4X2_usingDTable_internal_fast( + void* dst, + nuint dstSize, + void* cSrc, + nuint cSrcSize, + uint* DTable, + void* loopFn + ) + { + void* dt = DTable + 1; + byte* ilowest = (byte*)cSrc; + byte* oend = ZSTD_maybeNullPtrAdd((byte*)dst, (nint)dstSize); + HUF_DecompressFastArgs args; + { + nuint ret = HUF_DecompressFastArgs_init(&args, dst, dstSize, cSrc, cSrcSize, DTable); + { + nuint err_code = ret; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (ret == 0) + { + return 0; + } + } + + assert(args.ip.e0 >= args.ilowest); + ((delegate* managed)loopFn)(&args); + assert(args.ip.e0 >= ilowest); + assert(args.ip.e1 >= ilowest); + assert(args.ip.e2 >= ilowest); + assert(args.ip.e3 >= ilowest); + assert(args.op.e3 <= oend); + assert(ilowest == args.ilowest); + assert(ilowest + 6 == args.iend.e0); + { + nuint segmentSize = (dstSize + 3) / 4; + byte* segmentEnd = (byte*)dst; + int i; + for (i = 0; i < 4; ++i) + { + BIT_DStream_t bit; + if (segmentSize <= (nuint)(oend - segmentEnd)) + { + segmentEnd += segmentSize; + } + else + { + segmentEnd = oend; + } + + { + nuint err_code = HUF_initRemainingDStream(&bit, &args, i, segmentEnd); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + (&args.op.e0)[i] += HUF_decodeStreamX2( + (&args.op.e0)[i], + &bit, + segmentEnd, + (HUF_DEltX2*)dt, + 11 + ); + if ((&args.op.e0)[i] != segmentEnd) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + } + } + + return dstSize; + } + + private static nuint HUF_decompress4X2_usingDTable_internal( + void* dst, + nuint dstSize, + void* cSrc, + nuint cSrcSize, + uint* DTable, + int flags + ) + { + void* fallbackFn = (delegate* managed)( + &HUF_decompress4X2_usingDTable_internal_default + ); + void* loopFn = (delegate* managed)( + &HUF_decompress4X2_usingDTable_internal_fast_c_loop + ); + if ((flags & (int)HUF_flags_e.HUF_flags_disableFast) == 0) + { + nuint ret = HUF_decompress4X2_usingDTable_internal_fast( + dst, + dstSize, + cSrc, + cSrcSize, + DTable, + loopFn + ); + if (ret != 0) + { + return ret; + } + } + + return ((delegate* managed)fallbackFn)( + dst, + dstSize, + cSrc, + cSrcSize, + DTable + ); + } + + private static nuint HUF_decompress1X2_usingDTable_internal( + void* dst, + nuint dstSize, + void* cSrc, + nuint cSrcSize, + uint* DTable, + int flags + ) + { + return HUF_decompress1X2_usingDTable_internal_body(dst, dstSize, cSrc, cSrcSize, DTable); + } + + private static nuint HUF_decompress1X2_DCtx_wksp( + uint* DCtx, + void* dst, + nuint dstSize, + void* cSrc, + nuint cSrcSize, + void* workSpace, + nuint wkspSize, + int flags + ) + { + byte* ip = (byte*)cSrc; + nuint hSize = HUF_readDTableX2_wksp(DCtx, cSrc, cSrcSize, workSpace, wkspSize, flags); + if (ERR_isError(hSize)) + { + return hSize; + } + + if (hSize >= cSrcSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + ip += hSize; + cSrcSize -= hSize; + return HUF_decompress1X2_usingDTable_internal(dst, dstSize, ip, cSrcSize, DCtx, flags); + } + + private static nuint HUF_decompress4X2_DCtx_wksp( + uint* dctx, + void* dst, + nuint dstSize, + void* cSrc, + nuint cSrcSize, + void* workSpace, + nuint wkspSize, + int flags + ) + { + byte* ip = (byte*)cSrc; + nuint hSize = HUF_readDTableX2_wksp(dctx, cSrc, cSrcSize, workSpace, wkspSize, flags); + if (ERR_isError(hSize)) + { + return hSize; + } + + if (hSize >= cSrcSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + ip += hSize; + cSrcSize -= hSize; + return HUF_decompress4X2_usingDTable_internal(dst, dstSize, ip, cSrcSize, dctx, flags); + } + + private static readonly algo_time_t[][] algoTime = new algo_time_t[16][] + { + new algo_time_t[2] + { + new algo_time_t(tableTime: 0, decode256Time: 0), + new algo_time_t(tableTime: 1, decode256Time: 1), + }, + new algo_time_t[2] + { + new algo_time_t(tableTime: 0, decode256Time: 0), + new algo_time_t(tableTime: 1, decode256Time: 1), + }, + new algo_time_t[2] + { + new algo_time_t(tableTime: 150, decode256Time: 216), + new algo_time_t(tableTime: 381, decode256Time: 119), + }, + new algo_time_t[2] + { + new algo_time_t(tableTime: 170, decode256Time: 205), + new algo_time_t(tableTime: 514, decode256Time: 112), + }, + new algo_time_t[2] + { + new algo_time_t(tableTime: 177, decode256Time: 199), + new algo_time_t(tableTime: 539, decode256Time: 110), + }, + new algo_time_t[2] + { + new algo_time_t(tableTime: 197, decode256Time: 194), + new algo_time_t(tableTime: 644, decode256Time: 107), + }, + new algo_time_t[2] + { + new algo_time_t(tableTime: 221, decode256Time: 192), + new algo_time_t(tableTime: 735, decode256Time: 107), + }, + new algo_time_t[2] + { + new algo_time_t(tableTime: 256, decode256Time: 189), + new algo_time_t(tableTime: 881, decode256Time: 106), + }, + new algo_time_t[2] + { + new algo_time_t(tableTime: 359, decode256Time: 188), + new algo_time_t(tableTime: 1167, decode256Time: 109), + }, + new algo_time_t[2] + { + new algo_time_t(tableTime: 582, decode256Time: 187), + new algo_time_t(tableTime: 1570, decode256Time: 114), + }, + new algo_time_t[2] + { + new algo_time_t(tableTime: 688, decode256Time: 187), + new algo_time_t(tableTime: 1712, decode256Time: 122), + }, + new algo_time_t[2] + { + new algo_time_t(tableTime: 825, decode256Time: 186), + new algo_time_t(tableTime: 1965, decode256Time: 136), + }, + new algo_time_t[2] + { + new algo_time_t(tableTime: 976, decode256Time: 185), + new algo_time_t(tableTime: 2131, decode256Time: 150), + }, + new algo_time_t[2] + { + new algo_time_t(tableTime: 1180, decode256Time: 186), + new algo_time_t(tableTime: 2070, decode256Time: 175), + }, + new algo_time_t[2] + { + new algo_time_t(tableTime: 1377, decode256Time: 185), + new algo_time_t(tableTime: 1731, decode256Time: 202), + }, + new algo_time_t[2] + { + new algo_time_t(tableTime: 1412, decode256Time: 185), + new algo_time_t(tableTime: 1695, decode256Time: 202), + }, + }; + + /** HUF_selectDecoder() : + * Tells which decoder is likely to decode faster, + * based on a set of pre-computed metrics. + * @return : 0==HUF_decompress4X1, 1==HUF_decompress4X2 . + * Assumption : 0 < dstSize <= 128 KB */ + private static uint HUF_selectDecoder(nuint dstSize, nuint cSrcSize) + { + assert(dstSize > 0); + assert(dstSize <= 128 * 1024); + { + /* Q < 16 */ + uint Q = cSrcSize >= dstSize ? 15 : (uint)(cSrcSize * 16 / dstSize); + uint D256 = (uint)(dstSize >> 8); + uint DTime0 = algoTime[Q][0].tableTime + algoTime[Q][0].decode256Time * D256; + uint DTime1 = algoTime[Q][1].tableTime + algoTime[Q][1].decode256Time * D256; + DTime1 += DTime1 >> 5; + return DTime1 < DTime0 ? 1U : 0U; + } + } + + private static nuint HUF_decompress1X_DCtx_wksp( + uint* dctx, + void* dst, + nuint dstSize, + void* cSrc, + nuint cSrcSize, + void* workSpace, + nuint wkspSize, + int flags + ) + { + if (dstSize == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + if (cSrcSize > dstSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + if (cSrcSize == dstSize) + { + memcpy(dst, cSrc, (uint)dstSize); + return dstSize; + } + + if (cSrcSize == 1) + { + memset(dst, *(byte*)cSrc, (uint)dstSize); + return dstSize; + } + + { + uint algoNb = HUF_selectDecoder(dstSize, cSrcSize); + return algoNb != 0 + ? HUF_decompress1X2_DCtx_wksp( + dctx, + dst, + dstSize, + cSrc, + cSrcSize, + workSpace, + wkspSize, + flags + ) + : HUF_decompress1X1_DCtx_wksp( + dctx, + dst, + dstSize, + cSrc, + cSrcSize, + workSpace, + wkspSize, + flags + ); + } + } + + /* BMI2 variants. + * If the CPU has BMI2 support, pass bmi2=1, otherwise pass bmi2=0. + */ + private static nuint HUF_decompress1X_usingDTable( + void* dst, + nuint maxDstSize, + void* cSrc, + nuint cSrcSize, + uint* DTable, + int flags + ) + { + DTableDesc dtd = HUF_getDTableDesc(DTable); + return dtd.tableType != 0 + ? HUF_decompress1X2_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, flags) + : HUF_decompress1X1_usingDTable_internal( + dst, + maxDstSize, + cSrc, + cSrcSize, + DTable, + flags + ); + } + + private static nuint HUF_decompress1X1_DCtx_wksp( + uint* dctx, + void* dst, + nuint dstSize, + void* cSrc, + nuint cSrcSize, + void* workSpace, + nuint wkspSize, + int flags + ) + { + byte* ip = (byte*)cSrc; + nuint hSize = HUF_readDTableX1_wksp(dctx, cSrc, cSrcSize, workSpace, wkspSize, flags); + if (ERR_isError(hSize)) + { + return hSize; + } + + if (hSize >= cSrcSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + ip += hSize; + cSrcSize -= hSize; + return HUF_decompress1X1_usingDTable_internal(dst, dstSize, ip, cSrcSize, dctx, flags); + } + + private static nuint HUF_decompress4X_usingDTable( + void* dst, + nuint maxDstSize, + void* cSrc, + nuint cSrcSize, + uint* DTable, + int flags + ) + { + DTableDesc dtd = HUF_getDTableDesc(DTable); + return dtd.tableType != 0 + ? HUF_decompress4X2_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, flags) + : HUF_decompress4X1_usingDTable_internal( + dst, + maxDstSize, + cSrc, + cSrcSize, + DTable, + flags + ); + } + + private static nuint HUF_decompress4X_hufOnly_wksp( + uint* dctx, + void* dst, + nuint dstSize, + void* cSrc, + nuint cSrcSize, + void* workSpace, + nuint wkspSize, + int flags + ) + { + if (dstSize == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + if (cSrcSize == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + { + uint algoNb = HUF_selectDecoder(dstSize, cSrcSize); + return algoNb != 0 + ? HUF_decompress4X2_DCtx_wksp( + dctx, + dst, + dstSize, + cSrc, + cSrcSize, + workSpace, + wkspSize, + flags + ) + : HUF_decompress4X1_DCtx_wksp( + dctx, + dst, + dstSize, + cSrc, + cSrcSize, + workSpace, + wkspSize, + flags + ); + } + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/Mem.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/Mem.cs new file mode 100644 index 00000000..673c7c95 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/Mem.cs @@ -0,0 +1,162 @@ +using System; +using System.Buffers.Binary; +using System.Runtime.CompilerServices; +using BclUnsafe = System.Runtime.CompilerServices.Unsafe; + +// ReSharper disable InconsistentNaming +// ReSharper disable IdentifierTypo + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + /*-************************************************************** + * Memory I/O API + *****************************************************************/ + /*=== Static platform detection ===*/ + private static bool MEM_32bits + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get => sizeof(nint) == 4; + } + + private static bool MEM_64bits + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get => sizeof(nint) == 8; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + /* default method, safe and standard. + can sometimes prove slower */ + private static ushort MEM_read16(void* memPtr) => BclUnsafe.ReadUnaligned(memPtr); + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint MEM_read32(void* memPtr) => BclUnsafe.ReadUnaligned(memPtr); + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong MEM_read64(void* memPtr) => BclUnsafe.ReadUnaligned(memPtr); + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint MEM_readST(void* memPtr) => BclUnsafe.ReadUnaligned(memPtr); + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void MEM_write16(void* memPtr, ushort value) => + BclUnsafe.WriteUnaligned(memPtr, value); + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void MEM_write64(void* memPtr, ulong value) => + BclUnsafe.WriteUnaligned(memPtr, value); + + /*=== Little endian r/w ===*/ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ushort MEM_readLE16(void* memPtr) + { + var val = BclUnsafe.ReadUnaligned(memPtr); + if (!BitConverter.IsLittleEndian) + { + val = BinaryPrimitives.ReverseEndianness(val); + } + return val; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void MEM_writeLE16(void* memPtr, ushort val) + { + if (!BitConverter.IsLittleEndian) + { + val = BinaryPrimitives.ReverseEndianness(val); + } + BclUnsafe.WriteUnaligned(memPtr, val); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint MEM_readLE24(void* memPtr) => + (uint)(MEM_readLE16(memPtr) + (((byte*)memPtr)[2] << 16)); + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void MEM_writeLE24(void* memPtr, uint val) + { + MEM_writeLE16(memPtr, (ushort)val); + ((byte*)memPtr)[2] = (byte)(val >> 16); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint MEM_readLE32(void* memPtr) + { + var val = BclUnsafe.ReadUnaligned(memPtr); + if (!BitConverter.IsLittleEndian) + { + val = BinaryPrimitives.ReverseEndianness(val); + } + return val; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void MEM_writeLE32(void* memPtr, uint val32) + { + if (!BitConverter.IsLittleEndian) + { + val32 = BinaryPrimitives.ReverseEndianness(val32); + } + BclUnsafe.WriteUnaligned(memPtr, val32); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong MEM_readLE64(void* memPtr) + { + var val = BclUnsafe.ReadUnaligned(memPtr); + if (!BitConverter.IsLittleEndian) + { + val = BinaryPrimitives.ReverseEndianness(val); + } + return val; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void MEM_writeLE64(void* memPtr, ulong val64) + { + if (!BitConverter.IsLittleEndian) + { + val64 = BinaryPrimitives.ReverseEndianness(val64); + } + BclUnsafe.WriteUnaligned(memPtr, val64); + } + +#if !NET8_0_OR_GREATER + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ReverseEndiannessNative(nuint val) => + MEM_32bits + ? BinaryPrimitives.ReverseEndianness((uint)val) + : (nuint)BinaryPrimitives.ReverseEndianness(val); +#endif + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint MEM_readLEST(void* memPtr) + { + var val = BclUnsafe.ReadUnaligned(memPtr); + if (!BitConverter.IsLittleEndian) + { +#if NET8_0_OR_GREATER + val = BinaryPrimitives.ReverseEndianness(val); +#else + val = ReverseEndiannessNative(val); +#endif + } + return val; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void MEM_writeLEST(void* memPtr, nuint val) + { + if (!BitConverter.IsLittleEndian) + { +#if NET8_0_OR_GREATER + val = BinaryPrimitives.ReverseEndianness(val); +#else + val = ReverseEndiannessNative(val); +#endif + } + BclUnsafe.WriteUnaligned(memPtr, val); + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/Pool.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/Pool.cs new file mode 100644 index 00000000..495fe8e6 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/Pool.cs @@ -0,0 +1,128 @@ +using SharpCompress.Compressors.ZStandard.Unsafe; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + private static JobThreadPool GetThreadPool(void* ctx) => + UnmanagedObject.Unwrap(ctx); + + /* ZSTD_createThreadPool() : public access point */ + public static void* ZSTD_createThreadPool(nuint numThreads) + { + return POOL_create(numThreads, 0); + } + + /*! POOL_create() : + * Create a thread pool with at most `numThreads` threads. + * `numThreads` must be at least 1. + * The maximum number of queued jobs before blocking is `queueSize`. + * @return : POOL_ctx pointer on success, else NULL. + */ + private static void* POOL_create(nuint numThreads, nuint queueSize) + { + return POOL_create_advanced(numThreads, queueSize, Unsafe.Methods.ZSTD_defaultCMem); + } + + private static void* POOL_create_advanced( + nuint numThreads, + nuint queueSize, + ZSTD_customMem customMem + ) + { + var jobThreadPool = new JobThreadPool((int)numThreads, (int)queueSize); + return UnmanagedObject.Wrap(jobThreadPool); + } + + /*! POOL_join() : + Shutdown the queue, wake any sleeping threads, and join all of the threads. + */ + private static void POOL_join(void* ctx) + { + GetThreadPool(ctx).Join(); + } + + /*! POOL_free() : + * Free a thread pool returned by POOL_create(). + */ + private static void POOL_free(void* ctx) + { + if (ctx == null) + { + return; + } + + var jobThreadPool = GetThreadPool(ctx); + jobThreadPool.Join(); + jobThreadPool.Dispose(); + UnmanagedObject.Free(ctx); + } + + /*! POOL_joinJobs() : + * Waits for all queued jobs to finish executing. + */ + private static void POOL_joinJobs(void* ctx) + { + var jobThreadPool = GetThreadPool(ctx); + jobThreadPool.Join(false); + } + + public static void ZSTD_freeThreadPool(void* pool) + { + POOL_free(pool); + } + + /*! POOL_sizeof() : + * @return threadpool memory usage + * note : compatible with NULL (returns 0 in this case) + */ + private static nuint POOL_sizeof(void* ctx) + { + if (ctx == null) + { + return 0; + } + + var jobThreadPool = GetThreadPool(ctx); + return (nuint)jobThreadPool.Size(); + } + + /* @return : 0 on success, 1 on error */ + private static int POOL_resize(void* ctx, nuint numThreads) + { + if (ctx == null) + { + return 1; + } + + var jobThreadPool = GetThreadPool(ctx); + jobThreadPool.Resize((int)numThreads); + return 0; + } + + /*! POOL_add() : + * Add the job `function(opaque)` to the thread pool. `ctx` must be valid. + * Possibly blocks until there is room in the queue. + * Note : The function may be executed asynchronously, + * therefore, `opaque` must live until function has been completed. + */ + private static void POOL_add(void* ctx, void* function, void* opaque) + { + assert(ctx != null); + var jobThreadPool = GetThreadPool(ctx); + jobThreadPool.Add(function, opaque); + } + + /*! POOL_tryAdd() : + * Add the job `function(opaque)` to thread pool _if_ a queue slot is available. + * Returns immediately even if not (does not block). + * @return : 1 if successful, 0 if not. + */ + private static int POOL_tryAdd(void* ctx, void* function, void* opaque) + { + assert(ctx != null); + var jobThreadPool = GetThreadPool(ctx); + return jobThreadPool.TryAdd(function, opaque) ? 1 : 0; + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/RSyncState_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/RSyncState_t.cs new file mode 100644 index 00000000..7b5f08ad --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/RSyncState_t.cs @@ -0,0 +1,8 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct RSyncState_t +{ + public ulong hash; + public ulong hitMask; + public ulong primePower; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/Range.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/Range.cs new file mode 100644 index 00000000..fabf08b0 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/Range.cs @@ -0,0 +1,14 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/* ==== Serial State ==== */ +public unsafe struct Range +{ + public void* start; + public nuint size; + + public Range(void* start, nuint size) + { + this.start = start; + this.size = size; + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/RawSeqStore_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/RawSeqStore_t.cs new file mode 100644 index 00000000..07db05a1 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/RawSeqStore_t.cs @@ -0,0 +1,29 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct RawSeqStore_t +{ + /* The start of the sequences */ + public rawSeq* seq; + + /* The index in seq where reading stopped. pos <= size. */ + public nuint pos; + + /* The position within the sequence at seq[pos] where reading + stopped. posInSequence <= seq[pos].litLength + seq[pos].matchLength */ + public nuint posInSequence; + + /* The number of sequences. <= capacity. */ + public nuint size; + + /* The capacity starting from `seq` pointer */ + public nuint capacity; + + public RawSeqStore_t(rawSeq* seq, nuint pos, nuint posInSequence, nuint size, nuint capacity) + { + this.seq = seq; + this.pos = pos; + this.posInSequence = posInSequence; + this.size = size; + this.capacity = capacity; + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/RoundBuff_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/RoundBuff_t.cs new file mode 100644 index 00000000..84e42ca4 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/RoundBuff_t.cs @@ -0,0 +1,28 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct RoundBuff_t +{ + /* The round input buffer. All jobs get references + * to pieces of the buffer. ZSTDMT_tryGetInputRange() + * handles handing out job input buffers, and makes + * sure it doesn't overlap with any pieces still in use. + */ + public byte* buffer; + + /* The capacity of buffer. */ + public nuint capacity; + + /* The position of the current inBuff in the round + * buffer. Updated past the end if the inBuff once + * the inBuff is sent to the worker thread. + * pos <= capacity. + */ + public nuint pos; + + public RoundBuff_t(byte* buffer, nuint capacity, nuint pos) + { + this.buffer = buffer; + this.capacity = capacity; + this.pos = pos; + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/SeqCollector.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/SeqCollector.cs new file mode 100644 index 00000000..8026ed92 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/SeqCollector.cs @@ -0,0 +1,9 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct SeqCollector +{ + public int collectSequences; + public ZSTD_Sequence* seqStart; + public nuint seqIndex; + public nuint maxSequences; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/SeqDef_s.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/SeqDef_s.cs new file mode 100644 index 00000000..1e5aac57 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/SeqDef_s.cs @@ -0,0 +1,14 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/*********************************************** + * Sequences * + ***********************************************/ +public struct SeqDef_s +{ + /* offBase == Offset + ZSTD_REP_NUM, or repcode 1,2,3 */ + public uint offBase; + public ushort litLength; + + /* mlBase == matchLength - MINMATCH */ + public ushort mlBase; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/SeqStore_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/SeqStore_t.cs new file mode 100644 index 00000000..6aa75161 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/SeqStore_t.cs @@ -0,0 +1,27 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct SeqStore_t +{ + public SeqDef_s* sequencesStart; + + /* ptr to end of sequences */ + public SeqDef_s* sequences; + public byte* litStart; + + /* ptr to end of literals */ + public byte* lit; + public byte* llCode; + public byte* mlCode; + public byte* ofCode; + public nuint maxNbSeq; + public nuint maxNbLit; + + /* longLengthPos and longLengthType to allow us to represent either a single litLength or matchLength + * in the seqStore that has a value larger than U16 (if it exists). To do so, we increment + * the existing value of the litLength or matchLength by 0x10000. + */ + public ZSTD_longLengthType_e longLengthType; + + /* Index of the sequence to apply long length modification to */ + public uint longLengthPos; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/SerialState.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/SerialState.cs new file mode 100644 index 00000000..91d403e3 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/SerialState.cs @@ -0,0 +1,23 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct SerialState +{ + /* All variables in the struct are protected by mutex. */ + public void* mutex; + public void* cond; + public ZSTD_CCtx_params_s @params; + public ldmState_t ldmState; + public XXH64_state_s xxhState; + public uint nextJobID; + + /* Protects ldmWindow. + * Must be acquired after the main mutex when acquiring both. + */ + public void* ldmWindowMutex; + + /* Signaled when ldmWindow is updated */ + public void* ldmWindowCond; + + /* A thread-safe copy of ldmState.window */ + public ZSTD_window_t ldmWindow; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/SymbolEncodingType_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/SymbolEncodingType_e.cs new file mode 100644 index 00000000..06d78d45 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/SymbolEncodingType_e.cs @@ -0,0 +1,9 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum SymbolEncodingType_e +{ + set_basic, + set_rle, + set_compressed, + set_repeat, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/SyncPoint.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/SyncPoint.cs new file mode 100644 index 00000000..fda2ea51 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/SyncPoint.cs @@ -0,0 +1,10 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct SyncPoint +{ + /* The number of bytes to load from the input. */ + public nuint toLoad; + + /* Boolean declaring if we must flush because we found a synchronization point. */ + public int flush; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/XXH32_canonical_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/XXH32_canonical_t.cs new file mode 100644 index 00000000..7da74bdf --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/XXH32_canonical_t.cs @@ -0,0 +1,10 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/*! + * @brief Canonical (big endian) representation of @ref XXH32_hash_t. + */ +public unsafe struct XXH32_canonical_t +{ + /*!< Hash bytes, big endian */ + public fixed byte digest[4]; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/XXH32_state_s.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/XXH32_state_s.cs new file mode 100644 index 00000000..09559a84 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/XXH32_state_s.cs @@ -0,0 +1,34 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/*! + * @internal + * @brief Structure for XXH32 streaming API. + * + * @note This is only defined when @ref XXH_STATIC_LINKING_ONLY, + * @ref XXH_INLINE_ALL, or @ref XXH_IMPLEMENTATION is defined. Otherwise it is + * an opaque type. This allows fields to safely be changed. + * + * Typedef'd to @ref XXH32_state_t. + * Do not access the members of this struct directly. + * @see XXH64_state_s, XXH3_state_s + */ +public unsafe struct XXH32_state_s +{ + /*!< Total length hashed, modulo 2^32 */ + public uint total_len_32; + + /*!< Whether the hash is >= 16 (handles @ref total_len_32 overflow) */ + public uint large_len; + + /*!< Accumulator lanes */ + public fixed uint v[4]; + + /*!< Internal buffer for partial reads. Treated as unsigned char[16]. */ + public fixed uint mem32[4]; + + /*!< Amount of data in @ref mem32 */ + public uint memsize; + + /*!< Reserved field. Do not read nor write to it. */ + public uint reserved; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/XXH64_canonical_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/XXH64_canonical_t.cs new file mode 100644 index 00000000..a5040988 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/XXH64_canonical_t.cs @@ -0,0 +1,9 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/*! + * @brief Canonical (big endian) representation of @ref XXH64_hash_t. + */ +public unsafe struct XXH64_canonical_t +{ + public fixed byte digest[8]; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/XXH64_state_s.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/XXH64_state_s.cs new file mode 100644 index 00000000..d2503d08 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/XXH64_state_s.cs @@ -0,0 +1,34 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/*! + * @internal + * @brief Structure for XXH64 streaming API. + * + * @note This is only defined when @ref XXH_STATIC_LINKING_ONLY, + * @ref XXH_INLINE_ALL, or @ref XXH_IMPLEMENTATION is defined. Otherwise it is + * an opaque type. This allows fields to safely be changed. + * + * Typedef'd to @ref XXH64_state_t. + * Do not access the members of this struct directly. + * @see XXH32_state_s, XXH3_state_s + */ +public unsafe struct XXH64_state_s +{ + /*!< Total length hashed. This is always 64-bit. */ + public ulong total_len; + + /*!< Accumulator lanes */ + public fixed ulong v[4]; + + /*!< Internal buffer for partial reads. Treated as unsigned char[32]. */ + public fixed ulong mem64[4]; + + /*!< Amount of data in @ref mem64 */ + public uint memsize; + + /*!< Reserved field, needed for padding anyways*/ + public uint reserved32; + + /*!< Reserved field. Do not read or write to it. */ + public ulong reserved64; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/XXH_alignment.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/XXH_alignment.cs new file mode 100644 index 00000000..57b98809 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/XXH_alignment.cs @@ -0,0 +1,14 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/*! + * @internal + * @brief Enum to indicate whether a pointer is aligned. + */ +public enum XXH_alignment +{ + /*!< Aligned */ + XXH_aligned, + + /*!< Possibly unaligned */ + XXH_unaligned, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/XXH_errorcode.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/XXH_errorcode.cs new file mode 100644 index 00000000..eb65ceb1 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/XXH_errorcode.cs @@ -0,0 +1,13 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/*! + * @brief Exit code for the streaming API. + */ +public enum XXH_errorcode +{ + /*!< OK */ + XXH_OK = 0, + + /*!< Error */ + XXH_ERROR, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/Xxhash.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/Xxhash.cs new file mode 100644 index 00000000..347ac89c --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/Xxhash.cs @@ -0,0 +1,636 @@ +using System; +using System.Buffers.Binary; +using System.Numerics; +using System.Runtime.CompilerServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + /*! + * @internal + * @brief Modify this function to use a different routine than malloc(). + */ + private static void* XXH_malloc(nuint s) + { + return malloc(s); + } + + /*! + * @internal + * @brief Modify this function to use a different routine than free(). + */ + private static void XXH_free(void* p) + { + free(p); + } + + /*! + * @internal + * @brief Modify this function to use a different routine than memcpy(). + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void XXH_memcpy(void* dest, void* src, nuint size) + { + memcpy(dest, src, (uint)size); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint XXH_readLE32(void* ptr) + { + return BitConverter.IsLittleEndian + ? MEM_read32(ptr) + : BinaryPrimitives.ReverseEndianness(MEM_read32(ptr)); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint XXH_readBE32(void* ptr) + { + return BitConverter.IsLittleEndian + ? BinaryPrimitives.ReverseEndianness(MEM_read32(ptr)) + : MEM_read32(ptr); + } + + private static uint XXH_readLE32_align(void* ptr, XXH_alignment align) + { + if (align == XXH_alignment.XXH_unaligned) + { + return XXH_readLE32(ptr); + } + else + { + return BitConverter.IsLittleEndian + ? *(uint*)ptr + : BinaryPrimitives.ReverseEndianness(*(uint*)ptr); + } + } + + /* ************************************* + * Misc + ***************************************/ + /*! @ingroup public */ + private static uint ZSTD_XXH_versionNumber() + { + return 0 * 100 * 100 + 8 * 100 + 2; + } + + /*! + * @internal + * @brief Normal stripe processing routine. + * + * This shuffles the bits so that any bit from @p input impacts several bits in + * @p acc. + * + * @param acc The accumulator lane. + * @param input The stripe of input to mix. + * @return The mixed accumulator lane. + */ + private static uint XXH32_round(uint acc, uint input) + { + acc += input * 0x85EBCA77U; + acc = BitOperations.RotateLeft(acc, 13); + acc *= 0x9E3779B1U; + return acc; + } + + /*! + * @internal + * @brief Mixes all bits to finalize the hash. + * + * The final mix ensures that all input bits have a chance to impact any bit in + * the output digest, resulting in an unbiased distribution. + * + * @param hash The hash to avalanche. + * @return The avalanched hash. + */ + private static uint XXH32_avalanche(uint hash) + { + hash ^= hash >> 15; + hash *= 0x85EBCA77U; + hash ^= hash >> 13; + hash *= 0xC2B2AE3DU; + hash ^= hash >> 16; + return hash; + } + + /*! + * @internal + * @brief Processes the last 0-15 bytes of @p ptr. + * + * There may be up to 15 bytes remaining to consume from the input. + * This final stage will digest them to ensure that all input bytes are present + * in the final mix. + * + * @param hash The hash to finalize. + * @param ptr The pointer to the remaining input. + * @param len The remaining length, modulo 16. + * @param align Whether @p ptr is aligned. + * @return The finalized hash. + * @see XXH64_finalize(). + */ + private static uint XXH32_finalize(uint hash, byte* ptr, nuint len, XXH_alignment align) + { + len &= 15; + while (len >= 4) + { + { + hash += XXH_readLE32_align(ptr, align) * 0xC2B2AE3DU; + ptr += 4; + hash = BitOperations.RotateLeft(hash, 17) * 0x27D4EB2FU; + } + + len -= 4; + } + + while (len > 0) + { + { + hash += *ptr++ * 0x165667B1U; + hash = BitOperations.RotateLeft(hash, 11) * 0x9E3779B1U; + } + + --len; + } + + return XXH32_avalanche(hash); + } + + /*! + * @internal + * @brief The implementation for @ref XXH32(). + * + * @param input , len , seed Directly passed from @ref XXH32(). + * @param align Whether @p input is aligned. + * @return The calculated hash. + */ + private static uint XXH32_endian_align(byte* input, nuint len, uint seed, XXH_alignment align) + { + uint h32; + if (len >= 16) + { + byte* bEnd = input + len; + byte* limit = bEnd - 15; + uint v1 = seed + 0x9E3779B1U + 0x85EBCA77U; + uint v2 = seed + 0x85EBCA77U; + uint v3 = seed + 0; + uint v4 = seed - 0x9E3779B1U; + do + { + v1 = XXH32_round(v1, XXH_readLE32_align(input, align)); + input += 4; + v2 = XXH32_round(v2, XXH_readLE32_align(input, align)); + input += 4; + v3 = XXH32_round(v3, XXH_readLE32_align(input, align)); + input += 4; + v4 = XXH32_round(v4, XXH_readLE32_align(input, align)); + input += 4; + } while (input < limit); + h32 = + BitOperations.RotateLeft(v1, 1) + + BitOperations.RotateLeft(v2, 7) + + BitOperations.RotateLeft(v3, 12) + + BitOperations.RotateLeft(v4, 18); + } + else + { + h32 = seed + 0x165667B1U; + } + + h32 += (uint)len; + return XXH32_finalize(h32, input, len & 15, align); + } + + /*! @ingroup XXH32_family */ + private static uint ZSTD_XXH32(void* input, nuint len, uint seed) + { + return XXH32_endian_align((byte*)input, len, seed, XXH_alignment.XXH_unaligned); + } + + /*! @ingroup XXH32_family */ + private static XXH32_state_s* ZSTD_XXH32_createState() + { + return (XXH32_state_s*)XXH_malloc((nuint)sizeof(XXH32_state_s)); + } + + /*! @ingroup XXH32_family */ + private static XXH_errorcode ZSTD_XXH32_freeState(XXH32_state_s* statePtr) + { + XXH_free(statePtr); + return XXH_errorcode.XXH_OK; + } + + /*! @ingroup XXH32_family */ + private static void ZSTD_XXH32_copyState(XXH32_state_s* dstState, XXH32_state_s* srcState) + { + XXH_memcpy(dstState, srcState, (nuint)sizeof(XXH32_state_s)); + } + + /*! @ingroup XXH32_family */ + private static XXH_errorcode ZSTD_XXH32_reset(XXH32_state_s* statePtr, uint seed) + { + *statePtr = new XXH32_state_s(); + statePtr->v[0] = seed + 0x9E3779B1U + 0x85EBCA77U; + statePtr->v[1] = seed + 0x85EBCA77U; + statePtr->v[2] = seed + 0; + statePtr->v[3] = seed - 0x9E3779B1U; + return XXH_errorcode.XXH_OK; + } + + /*! @ingroup XXH32_family */ + private static XXH_errorcode ZSTD_XXH32_update(XXH32_state_s* state, void* input, nuint len) + { + if (input == null) + { + return XXH_errorcode.XXH_OK; + } + + { + byte* p = (byte*)input; + byte* bEnd = p + len; + state->total_len_32 += (uint)len; + state->large_len |= len >= 16 || state->total_len_32 >= 16 ? 1U : 0U; + if (state->memsize + len < 16) + { + XXH_memcpy((byte*)state->mem32 + state->memsize, input, len); + state->memsize += (uint)len; + return XXH_errorcode.XXH_OK; + } + + if (state->memsize != 0) + { + XXH_memcpy((byte*)state->mem32 + state->memsize, input, 16 - state->memsize); + { + uint* p32 = state->mem32; + state->v[0] = XXH32_round(state->v[0], XXH_readLE32(p32)); + p32++; + state->v[1] = XXH32_round(state->v[1], XXH_readLE32(p32)); + p32++; + state->v[2] = XXH32_round(state->v[2], XXH_readLE32(p32)); + p32++; + state->v[3] = XXH32_round(state->v[3], XXH_readLE32(p32)); + } + + p += 16 - state->memsize; + state->memsize = 0; + } + + if (p <= bEnd - 16) + { + byte* limit = bEnd - 16; + do + { + state->v[0] = XXH32_round(state->v[0], XXH_readLE32(p)); + p += 4; + state->v[1] = XXH32_round(state->v[1], XXH_readLE32(p)); + p += 4; + state->v[2] = XXH32_round(state->v[2], XXH_readLE32(p)); + p += 4; + state->v[3] = XXH32_round(state->v[3], XXH_readLE32(p)); + p += 4; + } while (p <= limit); + } + + if (p < bEnd) + { + XXH_memcpy(state->mem32, p, (nuint)(bEnd - p)); + state->memsize = (uint)(bEnd - p); + } + } + + return XXH_errorcode.XXH_OK; + } + + /*! @ingroup XXH32_family */ + private static uint ZSTD_XXH32_digest(XXH32_state_s* state) + { + uint h32; + if (state->large_len != 0) + { + h32 = + BitOperations.RotateLeft(state->v[0], 1) + + BitOperations.RotateLeft(state->v[1], 7) + + BitOperations.RotateLeft(state->v[2], 12) + + BitOperations.RotateLeft(state->v[3], 18); + } + else + { + h32 = state->v[2] + 0x165667B1U; + } + + h32 += state->total_len_32; + return XXH32_finalize(h32, (byte*)state->mem32, state->memsize, XXH_alignment.XXH_aligned); + } + + /*! @ingroup XXH32_family */ + private static void ZSTD_XXH32_canonicalFromHash(XXH32_canonical_t* dst, uint hash) + { + assert(sizeof(XXH32_canonical_t) == sizeof(uint)); + if (BitConverter.IsLittleEndian) + { + hash = BinaryPrimitives.ReverseEndianness(hash); + } + + XXH_memcpy(dst, &hash, (nuint)sizeof(XXH32_canonical_t)); + } + + /*! @ingroup XXH32_family */ + private static uint ZSTD_XXH32_hashFromCanonical(XXH32_canonical_t* src) + { + return XXH_readBE32(src); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong XXH_readLE64(void* ptr) + { + return BitConverter.IsLittleEndian + ? MEM_read64(ptr) + : BinaryPrimitives.ReverseEndianness(MEM_read64(ptr)); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong XXH_readBE64(void* ptr) + { + return BitConverter.IsLittleEndian + ? BinaryPrimitives.ReverseEndianness(MEM_read64(ptr)) + : MEM_read64(ptr); + } + + private static ulong XXH_readLE64_align(void* ptr, XXH_alignment align) + { + if (align == XXH_alignment.XXH_unaligned) + { + return XXH_readLE64(ptr); + } + else + { + return BitConverter.IsLittleEndian + ? *(ulong*)ptr + : BinaryPrimitives.ReverseEndianness(*(ulong*)ptr); + } + } + + /*! @copydoc XXH32_round */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong XXH64_round(ulong acc, ulong input) + { + acc += input * 0xC2B2AE3D27D4EB4FUL; + acc = BitOperations.RotateLeft(acc, 31); + acc *= 0x9E3779B185EBCA87UL; + return acc; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong XXH64_mergeRound(ulong acc, ulong val) + { + val = XXH64_round(0, val); + acc ^= val; + acc = acc * 0x9E3779B185EBCA87UL + 0x85EBCA77C2B2AE63UL; + return acc; + } + + /*! @copydoc XXH32_avalanche */ + private static ulong XXH64_avalanche(ulong hash) + { + hash ^= hash >> 33; + hash *= 0xC2B2AE3D27D4EB4FUL; + hash ^= hash >> 29; + hash *= 0x165667B19E3779F9UL; + hash ^= hash >> 32; + return hash; + } + + /*! + * @internal + * @brief Processes the last 0-31 bytes of @p ptr. + * + * There may be up to 31 bytes remaining to consume from the input. + * This final stage will digest them to ensure that all input bytes are present + * in the final mix. + * + * @param hash The hash to finalize. + * @param ptr The pointer to the remaining input. + * @param len The remaining length, modulo 32. + * @param align Whether @p ptr is aligned. + * @return The finalized hash + * @see XXH32_finalize(). + */ + private static ulong XXH64_finalize(ulong hash, byte* ptr, nuint len, XXH_alignment align) + { + len &= 31; + while (len >= 8) + { + ulong k1 = XXH64_round(0, XXH_readLE64_align(ptr, align)); + ptr += 8; + hash ^= k1; + hash = BitOperations.RotateLeft(hash, 27) * 0x9E3779B185EBCA87UL + 0x85EBCA77C2B2AE63UL; + len -= 8; + } + + if (len >= 4) + { + hash ^= XXH_readLE32_align(ptr, align) * 0x9E3779B185EBCA87UL; + ptr += 4; + hash = BitOperations.RotateLeft(hash, 23) * 0xC2B2AE3D27D4EB4FUL + 0x165667B19E3779F9UL; + len -= 4; + } + + while (len > 0) + { + hash ^= *ptr++ * 0x27D4EB2F165667C5UL; + hash = BitOperations.RotateLeft(hash, 11) * 0x9E3779B185EBCA87UL; + --len; + } + + return XXH64_avalanche(hash); + } + + /*! + * @internal + * @brief The implementation for @ref XXH64(). + * + * @param input , len , seed Directly passed from @ref XXH64(). + * @param align Whether @p input is aligned. + * @return The calculated hash. + */ + private static ulong XXH64_endian_align(byte* input, nuint len, ulong seed, XXH_alignment align) + { + ulong h64; + if (len >= 32) + { + byte* bEnd = input + len; + byte* limit = bEnd - 31; + ulong v1 = seed + 0x9E3779B185EBCA87UL + 0xC2B2AE3D27D4EB4FUL; + ulong v2 = seed + 0xC2B2AE3D27D4EB4FUL; + ulong v3 = seed + 0; + ulong v4 = seed - 0x9E3779B185EBCA87UL; + do + { + v1 = XXH64_round(v1, XXH_readLE64_align(input, align)); + input += 8; + v2 = XXH64_round(v2, XXH_readLE64_align(input, align)); + input += 8; + v3 = XXH64_round(v3, XXH_readLE64_align(input, align)); + input += 8; + v4 = XXH64_round(v4, XXH_readLE64_align(input, align)); + input += 8; + } while (input < limit); + h64 = + BitOperations.RotateLeft(v1, 1) + + BitOperations.RotateLeft(v2, 7) + + BitOperations.RotateLeft(v3, 12) + + BitOperations.RotateLeft(v4, 18); + h64 = XXH64_mergeRound(h64, v1); + h64 = XXH64_mergeRound(h64, v2); + h64 = XXH64_mergeRound(h64, v3); + h64 = XXH64_mergeRound(h64, v4); + } + else + { + h64 = seed + 0x27D4EB2F165667C5UL; + } + + h64 += len; + return XXH64_finalize(h64, input, len, align); + } + + /*! @ingroup XXH64_family */ + private static ulong ZSTD_XXH64(void* input, nuint len, ulong seed) + { + return XXH64_endian_align((byte*)input, len, seed, XXH_alignment.XXH_unaligned); + } + + /*! @ingroup XXH64_family*/ + private static XXH64_state_s* ZSTD_XXH64_createState() + { + return (XXH64_state_s*)XXH_malloc((nuint)sizeof(XXH64_state_s)); + } + + /*! @ingroup XXH64_family */ + private static XXH_errorcode ZSTD_XXH64_freeState(XXH64_state_s* statePtr) + { + XXH_free(statePtr); + return XXH_errorcode.XXH_OK; + } + + /*! @ingroup XXH64_family */ + private static void ZSTD_XXH64_copyState(XXH64_state_s* dstState, XXH64_state_s* srcState) + { + XXH_memcpy(dstState, srcState, (nuint)sizeof(XXH64_state_s)); + } + + /*! @ingroup XXH64_family */ + private static XXH_errorcode ZSTD_XXH64_reset(XXH64_state_s* statePtr, ulong seed) + { + *statePtr = new XXH64_state_s(); + statePtr->v[0] = seed + 0x9E3779B185EBCA87UL + 0xC2B2AE3D27D4EB4FUL; + statePtr->v[1] = seed + 0xC2B2AE3D27D4EB4FUL; + statePtr->v[2] = seed + 0; + statePtr->v[3] = seed - 0x9E3779B185EBCA87UL; + return XXH_errorcode.XXH_OK; + } + + /*! @ingroup XXH64_family */ + private static XXH_errorcode ZSTD_XXH64_update(XXH64_state_s* state, void* input, nuint len) + { + if (input == null) + { + return XXH_errorcode.XXH_OK; + } + + { + byte* p = (byte*)input; + byte* bEnd = p + len; + state->total_len += len; + if (state->memsize + len < 32) + { + XXH_memcpy((byte*)state->mem64 + state->memsize, input, len); + state->memsize += (uint)len; + return XXH_errorcode.XXH_OK; + } + + if (state->memsize != 0) + { + XXH_memcpy((byte*)state->mem64 + state->memsize, input, 32 - state->memsize); + state->v[0] = XXH64_round(state->v[0], XXH_readLE64(state->mem64 + 0)); + state->v[1] = XXH64_round(state->v[1], XXH_readLE64(state->mem64 + 1)); + state->v[2] = XXH64_round(state->v[2], XXH_readLE64(state->mem64 + 2)); + state->v[3] = XXH64_round(state->v[3], XXH_readLE64(state->mem64 + 3)); + p += 32 - state->memsize; + state->memsize = 0; + } + + if (p + 32 <= bEnd) + { + byte* limit = bEnd - 32; + do + { + state->v[0] = XXH64_round(state->v[0], XXH_readLE64(p)); + p += 8; + state->v[1] = XXH64_round(state->v[1], XXH_readLE64(p)); + p += 8; + state->v[2] = XXH64_round(state->v[2], XXH_readLE64(p)); + p += 8; + state->v[3] = XXH64_round(state->v[3], XXH_readLE64(p)); + p += 8; + } while (p <= limit); + } + + if (p < bEnd) + { + XXH_memcpy(state->mem64, p, (nuint)(bEnd - p)); + state->memsize = (uint)(bEnd - p); + } + } + + return XXH_errorcode.XXH_OK; + } + + /*! @ingroup XXH64_family */ + private static ulong ZSTD_XXH64_digest(XXH64_state_s* state) + { + ulong h64; + if (state->total_len >= 32) + { + h64 = + BitOperations.RotateLeft(state->v[0], 1) + + BitOperations.RotateLeft(state->v[1], 7) + + BitOperations.RotateLeft(state->v[2], 12) + + BitOperations.RotateLeft(state->v[3], 18); + h64 = XXH64_mergeRound(h64, state->v[0]); + h64 = XXH64_mergeRound(h64, state->v[1]); + h64 = XXH64_mergeRound(h64, state->v[2]); + h64 = XXH64_mergeRound(h64, state->v[3]); + } + else + { + h64 = state->v[2] + 0x27D4EB2F165667C5UL; + } + + h64 += state->total_len; + return XXH64_finalize( + h64, + (byte*)state->mem64, + (nuint)state->total_len, + XXH_alignment.XXH_aligned + ); + } + + /*! @ingroup XXH64_family */ + private static void ZSTD_XXH64_canonicalFromHash(XXH64_canonical_t* dst, ulong hash) + { + assert(sizeof(XXH64_canonical_t) == sizeof(ulong)); + if (BitConverter.IsLittleEndian) + { + hash = BinaryPrimitives.ReverseEndianness(hash); + } + + XXH_memcpy(dst, &hash, (nuint)sizeof(XXH64_canonical_t)); + } + + /*! @ingroup XXH64_family */ + private static ulong ZSTD_XXH64_hashFromCanonical(XXH64_canonical_t* src) + { + return XXH_readBE64(src); + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZDICT_cover_params_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZDICT_cover_params_t.cs new file mode 100644 index 00000000..dfc4fecf --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZDICT_cover_params_t.cs @@ -0,0 +1,30 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/*! ZDICT_cover_params_t: + * k and d are the only required parameters. + * For others, value 0 means default. + */ +public struct ZDICT_cover_params_t +{ + /* Segment size : constraint: 0 < k : Reasonable range [16, 2048+] */ + public uint k; + + /* dmer size : constraint: 0 < d <= k : Reasonable range [6, 16] */ + public uint d; + + /* Number of steps : Only used for optimization : 0 means default (40) : Higher means more parameters checked */ + public uint steps; + + /* Number of threads : constraint: 0 < nbThreads : 1 means single-threaded : Only used for optimization : Ignored if ZSTD_MULTITHREAD is not defined */ + public uint nbThreads; + + /* Percentage of samples used for training: Only used for optimization : the first nbSamples * splitPoint samples will be used to training, the last nbSamples * (1 - splitPoint) samples will be used for testing, 0 means default (1.0), 1.0 when all samples are used for both training and testing */ + public double splitPoint; + + /* Train dictionaries to shrink in size starting from the minimum size and selects the smallest dictionary that is shrinkDictMaxRegression% worse than the largest dictionary. 0 means no shrinking and 1 means shrinking */ + public uint shrinkDict; + + /* Sets shrinkDictMaxRegression so that a smaller dictionary can be at worse shrinkDictMaxRegression% worse than the max dict size dictionary. */ + public uint shrinkDictMaxRegression; + public ZDICT_params_t zParams; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZDICT_fastCover_params_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZDICT_fastCover_params_t.cs new file mode 100644 index 00000000..a5af135d --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZDICT_fastCover_params_t.cs @@ -0,0 +1,32 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct ZDICT_fastCover_params_t +{ + /* Segment size : constraint: 0 < k : Reasonable range [16, 2048+] */ + public uint k; + + /* dmer size : constraint: 0 < d <= k : Reasonable range [6, 16] */ + public uint d; + + /* log of size of frequency array : constraint: 0 < f <= 31 : 1 means default(20)*/ + public uint f; + + /* Number of steps : Only used for optimization : 0 means default (40) : Higher means more parameters checked */ + public uint steps; + + /* Number of threads : constraint: 0 < nbThreads : 1 means single-threaded : Only used for optimization : Ignored if ZSTD_MULTITHREAD is not defined */ + public uint nbThreads; + + /* Percentage of samples used for training: Only used for optimization : the first nbSamples * splitPoint samples will be used to training, the last nbSamples * (1 - splitPoint) samples will be used for testing, 0 means default (0.75), 1.0 when all samples are used for both training and testing */ + public double splitPoint; + + /* Acceleration level: constraint: 0 < accel <= 10, higher means faster and less accurate, 0 means default(1) */ + public uint accel; + + /* Train dictionaries to shrink in size starting from the minimum size and selects the smallest dictionary that is shrinkDictMaxRegression% worse than the largest dictionary. 0 means no shrinking and 1 means shrinking */ + public uint shrinkDict; + + /* Sets shrinkDictMaxRegression so that a smaller dictionary can be at worse shrinkDictMaxRegression% worse than the max dict size dictionary. */ + public uint shrinkDictMaxRegression; + public ZDICT_params_t zParams; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZDICT_legacy_params_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZDICT_legacy_params_t.cs new file mode 100644 index 00000000..fd061e59 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZDICT_legacy_params_t.cs @@ -0,0 +1,8 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct ZDICT_legacy_params_t +{ + /* 0 means default; larger => select more => larger dictionary */ + public uint selectivityLevel; + public ZDICT_params_t zParams; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZDICT_params_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZDICT_params_t.cs new file mode 100644 index 00000000..6c3fe758 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZDICT_params_t.cs @@ -0,0 +1,20 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct ZDICT_params_t +{ + /**< optimize for a specific zstd compression level; 0 means default */ + public int compressionLevel; + + /**< Write log to stderr; 0 = none (default); 1 = errors; 2 = progression; 3 = details; 4 = debug; */ + public uint notificationLevel; + + /**< force dictID value; 0 means auto mode (32-bits random value) + * NOTE: The zstd format reserves some dictionary IDs for future use. + * You may use them in private settings, but be warned that they + * may be used by zstd in a public dictionary registry in the future. + * These dictionary IDs are: + * - low range : <= 32767 + * - high range : >= (2^31) + */ + public uint dictID; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTDMT_CCtxPool.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTDMT_CCtxPool.cs new file mode 100644 index 00000000..a9c7fccc --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTDMT_CCtxPool.cs @@ -0,0 +1,12 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/* ===== CCtx Pool ===== */ +/* a single CCtx Pool can be invoked from multiple threads in parallel */ +public unsafe struct ZSTDMT_CCtxPool +{ + public void* poolMutex; + public int totalCCtx; + public int availCCtx; + public ZSTD_customMem cMem; + public ZSTD_CCtx_s** cctxs; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTDMT_CCtx_s.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTDMT_CCtx_s.cs new file mode 100644 index 00000000..3e4e2cfe --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTDMT_CCtx_s.cs @@ -0,0 +1,32 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct ZSTDMT_CCtx_s +{ + public void* factory; + public ZSTDMT_jobDescription* jobs; + public ZSTDMT_bufferPool_s* bufPool; + public ZSTDMT_CCtxPool* cctxPool; + public ZSTDMT_bufferPool_s* seqPool; + public ZSTD_CCtx_params_s @params; + public nuint targetSectionSize; + public nuint targetPrefixSize; + + /* 1 => one job is already prepared, but pool has shortage of workers. Don't create a new job. */ + public int jobReady; + public InBuff_t inBuff; + public RoundBuff_t roundBuff; + public SerialState serial; + public RSyncState_t rsync; + public uint jobIDMask; + public uint doneJobID; + public uint nextJobID; + public uint frameEnded; + public uint allJobsCompleted; + public ulong frameContentSize; + public ulong consumed; + public ulong produced; + public ZSTD_customMem cMem; + public ZSTD_CDict_s* cdictLocal; + public ZSTD_CDict_s* cdict; + public uint providedFactory; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTDMT_bufferPool_s.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTDMT_bufferPool_s.cs new file mode 100644 index 00000000..0c64573a --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTDMT_bufferPool_s.cs @@ -0,0 +1,11 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct ZSTDMT_bufferPool_s +{ + public void* poolMutex; + public nuint bufferSize; + public uint totalBuffers; + public uint nbBuffers; + public ZSTD_customMem cMem; + public buffer_s* buffers; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTDMT_jobDescription.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTDMT_jobDescription.cs new file mode 100644 index 00000000..cb2c18f6 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTDMT_jobDescription.cs @@ -0,0 +1,61 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct ZSTDMT_jobDescription +{ + /* SHARED - set0 by mtctx, then modified by worker AND read by mtctx */ + public nuint consumed; + + /* SHARED - set0 by mtctx, then modified by worker AND read by mtctx, then set0 by mtctx */ + public nuint cSize; + + /* Thread-safe - used by mtctx and worker */ + public void* job_mutex; + + /* Thread-safe - used by mtctx and worker */ + public void* job_cond; + + /* Thread-safe - used by mtctx and (all) workers */ + public ZSTDMT_CCtxPool* cctxPool; + + /* Thread-safe - used by mtctx and (all) workers */ + public ZSTDMT_bufferPool_s* bufPool; + + /* Thread-safe - used by mtctx and (all) workers */ + public ZSTDMT_bufferPool_s* seqPool; + + /* Thread-safe - used by mtctx and (all) workers */ + public SerialState* serial; + + /* set by worker (or mtctx), then read by worker & mtctx, then modified by mtctx => no barrier */ + public buffer_s dstBuff; + + /* set by mtctx, then read by worker & mtctx => no barrier */ + public Range prefix; + + /* set by mtctx, then read by worker & mtctx => no barrier */ + public Range src; + + /* set by mtctx, then read by worker => no barrier */ + public uint jobID; + + /* set by mtctx, then read by worker => no barrier */ + public uint firstJob; + + /* set by mtctx, then read by worker => no barrier */ + public uint lastJob; + + /* set by mtctx, then read by worker => no barrier */ + public ZSTD_CCtx_params_s @params; + + /* set by mtctx, then read by worker => no barrier */ + public ZSTD_CDict_s* cdict; + + /* set by mtctx, then read by worker => no barrier */ + public ulong fullFrameSize; + + /* used only by mtctx */ + public nuint dstFlushed; + + /* used only by mtctx */ + public uint frameChecksumNeeded; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_BlockCompressor_f.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_BlockCompressor_f.cs new file mode 100644 index 00000000..08a44808 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_BlockCompressor_f.cs @@ -0,0 +1,12 @@ +using System.Runtime.InteropServices; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +[UnmanagedFunctionPointer(CallingConvention.Cdecl)] +public unsafe delegate nuint ZSTD_BlockCompressor_f( + ZSTD_MatchState_t* bs, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize +); diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_BuildCTableWksp.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_BuildCTableWksp.cs new file mode 100644 index 00000000..afdeff1d --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_BuildCTableWksp.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct ZSTD_BuildCTableWksp +{ + public fixed short norm[53]; + public fixed uint wksp[285]; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_BuildSeqStore_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_BuildSeqStore_e.cs new file mode 100644 index 00000000..5378fc7e --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_BuildSeqStore_e.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum ZSTD_BuildSeqStore_e +{ + ZSTDbss_compress, + ZSTDbss_noCompress, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_CCtx_params_s.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_CCtx_params_s.cs new file mode 100644 index 00000000..e7a45dbe --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_CCtx_params_s.cs @@ -0,0 +1,86 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct ZSTD_CCtx_params_s +{ + public ZSTD_format_e format; + public ZSTD_compressionParameters cParams; + public ZSTD_frameParameters fParams; + public int compressionLevel; + + /* force back-references to respect limit of + * 1< 0: + * If litLength != 0: + * rep == 1 --> offset == repeat_offset_1 + * rep == 2 --> offset == repeat_offset_2 + * rep == 3 --> offset == repeat_offset_3 + * If litLength == 0: + * rep == 1 --> offset == repeat_offset_2 + * rep == 2 --> offset == repeat_offset_3 + * rep == 3 --> offset == repeat_offset_1 - 1 + * + * Note: This field is optional. ZSTD_generateSequences() will calculate the value of + * 'rep', but repeat offsets do not necessarily need to be calculated from an external + * sequence provider perspective. For example, ZSTD_compressSequences() does not + * use this 'rep' field at all (as of now). + */ + public uint rep; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_SequenceLength.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_SequenceLength.cs new file mode 100644 index 00000000..80e67fc9 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_SequenceLength.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct ZSTD_SequenceLength +{ + public uint litLength; + public uint matchLength; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_SequencePosition.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_SequencePosition.cs new file mode 100644 index 00000000..db0ec511 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_SequencePosition.cs @@ -0,0 +1,13 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct ZSTD_SequencePosition +{ + /* Index in array of ZSTD_Sequence */ + public uint idx; + + /* Position within sequence at idx */ + public uint posInSequence; + + /* Number of bytes given by sequences provided so far */ + public nuint posInSrc; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_blockSplitCtx.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_blockSplitCtx.cs new file mode 100644 index 00000000..b83e4e1f --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_blockSplitCtx.cs @@ -0,0 +1,12 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct ZSTD_blockSplitCtx +{ + public SeqStore_t fullSeqStoreChunk; + public SeqStore_t firstHalfSeqStore; + public SeqStore_t secondHalfSeqStore; + public SeqStore_t currSeqStore; + public SeqStore_t nextSeqStore; + public fixed uint partitions[196]; + public ZSTD_entropyCTablesMetadata_t entropyMetadata; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_blockState_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_blockState_t.cs new file mode 100644 index 00000000..5dd067a1 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_blockState_t.cs @@ -0,0 +1,8 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct ZSTD_blockState_t +{ + public ZSTD_compressedBlockState_t* prevCBlock; + public ZSTD_compressedBlockState_t* nextCBlock; + public ZSTD_MatchState_t matchState; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_bounds.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_bounds.cs new file mode 100644 index 00000000..d82a2932 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_bounds.cs @@ -0,0 +1,8 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct ZSTD_bounds +{ + public nuint error; + public int lowerBound; + public int upperBound; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_bufferMode_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_bufferMode_e.cs new file mode 100644 index 00000000..6ae23f31 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_bufferMode_e.cs @@ -0,0 +1,11 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/* Controls whether the input/output buffer is buffered or stable. */ +public enum ZSTD_bufferMode_e +{ + /* Buffer the input/output */ + ZSTD_bm_buffered = 0, + + /* ZSTD_inBuffer/ZSTD_outBuffer is stable */ + ZSTD_bm_stable = 1, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_buffered_policy_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_buffered_policy_e.cs new file mode 100644 index 00000000..8d5a1a91 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_buffered_policy_e.cs @@ -0,0 +1,12 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/** + * Indicates whether this compression proceeds directly from user-provided + * source buffer to user-provided destination buffer (ZSTDb_not_buffered), or + * whether the context needs to buffer the input/output (ZSTDb_buffered). + */ +public enum ZSTD_buffered_policy_e +{ + ZSTDb_not_buffered, + ZSTDb_buffered, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_cParameter.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_cParameter.cs new file mode 100644 index 00000000..cfe1f38a --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_cParameter.cs @@ -0,0 +1,218 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum ZSTD_cParameter +{ + /* Set compression parameters according to pre-defined cLevel table. + * Note that exact compression parameters are dynamically determined, + * depending on both compression level and srcSize (when known). + * Default level is ZSTD_CLEVEL_DEFAULT==3. + * Special: value 0 means default, which is controlled by ZSTD_CLEVEL_DEFAULT. + * Note 1 : it's possible to pass a negative compression level. + * Note 2 : setting a level does not automatically set all other compression parameters + * to default. Setting this will however eventually dynamically impact the compression + * parameters which have not been manually set. The manually set + * ones will 'stick'. */ + ZSTD_c_compressionLevel = 100, + + /* Maximum allowed back-reference distance, expressed as power of 2. + * This will set a memory budget for streaming decompression, + * with larger values requiring more memory + * and typically compressing more. + * Must be clamped between ZSTD_WINDOWLOG_MIN and ZSTD_WINDOWLOG_MAX. + * Special: value 0 means "use default windowLog". + * Note: Using a windowLog greater than ZSTD_WINDOWLOG_LIMIT_DEFAULT + * requires explicitly allowing such size at streaming decompression stage. */ + ZSTD_c_windowLog = 101, + + /* Size of the initial probe table, as a power of 2. + * Resulting memory usage is (1 << (hashLog+2)). + * Must be clamped between ZSTD_HASHLOG_MIN and ZSTD_HASHLOG_MAX. + * Larger tables improve compression ratio of strategies <= dFast, + * and improve speed of strategies > dFast. + * Special: value 0 means "use default hashLog". */ + ZSTD_c_hashLog = 102, + + /* Size of the multi-probe search table, as a power of 2. + * Resulting memory usage is (1 << (chainLog+2)). + * Must be clamped between ZSTD_CHAINLOG_MIN and ZSTD_CHAINLOG_MAX. + * Larger tables result in better and slower compression. + * This parameter is useless for "fast" strategy. + * It's still useful when using "dfast" strategy, + * in which case it defines a secondary probe table. + * Special: value 0 means "use default chainLog". */ + ZSTD_c_chainLog = 103, + + /* Number of search attempts, as a power of 2. + * More attempts result in better and slower compression. + * This parameter is useless for "fast" and "dFast" strategies. + * Special: value 0 means "use default searchLog". */ + ZSTD_c_searchLog = 104, + + /* Minimum size of searched matches. + * Note that Zstandard can still find matches of smaller size, + * it just tweaks its search algorithm to look for this size and larger. + * Larger values increase compression and decompression speed, but decrease ratio. + * Must be clamped between ZSTD_MINMATCH_MIN and ZSTD_MINMATCH_MAX. + * Note that currently, for all strategies < btopt, effective minimum is 4. + * , for all strategies > fast, effective maximum is 6. + * Special: value 0 means "use default minMatchLength". */ + ZSTD_c_minMatch = 105, + + /* Impact of this field depends on strategy. + * For strategies btopt, btultra & btultra2: + * Length of Match considered "good enough" to stop search. + * Larger values make compression stronger, and slower. + * For strategy fast: + * Distance between match sampling. + * Larger values make compression faster, and weaker. + * Special: value 0 means "use default targetLength". */ + ZSTD_c_targetLength = 106, + + /* See ZSTD_strategy enum definition. + * The higher the value of selected strategy, the more complex it is, + * resulting in stronger and slower compression. + * Special: value 0 means "use default strategy". */ + ZSTD_c_strategy = 107, + + /* v1.5.6+ + * Attempts to fit compressed block size into approximately targetCBlockSize. + * Bound by ZSTD_TARGETCBLOCKSIZE_MIN and ZSTD_TARGETCBLOCKSIZE_MAX. + * Note that it's not a guarantee, just a convergence target (default:0). + * No target when targetCBlockSize == 0. + * This is helpful in low bandwidth streaming environments to improve end-to-end latency, + * when a client can make use of partial documents (a prominent example being Chrome). + * Note: this parameter is stable since v1.5.6. + * It was present as an experimental parameter in earlier versions, + * but it's not recommended using it with earlier library versions + * due to massive performance regressions. + */ + ZSTD_c_targetCBlockSize = 130, + + /* Enable long distance matching. + * This parameter is designed to improve compression ratio + * for large inputs, by finding large matches at long distance. + * It increases memory usage and window size. + * Note: enabling this parameter increases default ZSTD_c_windowLog to 128 MB + * except when expressly set to a different value. + * Note: will be enabled by default if ZSTD_c_windowLog >= 128 MB and + * compression strategy >= ZSTD_btopt (== compression level 16+) */ + ZSTD_c_enableLongDistanceMatching = 160, + + /* Size of the table for long distance matching, as a power of 2. + * Larger values increase memory usage and compression ratio, + * but decrease compression speed. + * Must be clamped between ZSTD_HASHLOG_MIN and ZSTD_HASHLOG_MAX + * default: windowlog - 7. + * Special: value 0 means "automatically determine hashlog". */ + ZSTD_c_ldmHashLog = 161, + + /* Minimum match size for long distance matcher. + * Larger/too small values usually decrease compression ratio. + * Must be clamped between ZSTD_LDM_MINMATCH_MIN and ZSTD_LDM_MINMATCH_MAX. + * Special: value 0 means "use default value" (default: 64). */ + ZSTD_c_ldmMinMatch = 162, + + /* Log size of each bucket in the LDM hash table for collision resolution. + * Larger values improve collision resolution but decrease compression speed. + * The maximum value is ZSTD_LDM_BUCKETSIZELOG_MAX. + * Special: value 0 means "use default value" (default: 3). */ + ZSTD_c_ldmBucketSizeLog = 163, + + /* Frequency of inserting/looking up entries into the LDM hash table. + * Must be clamped between 0 and (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN). + * Default is MAX(0, (windowLog - ldmHashLog)), optimizing hash table usage. + * Larger values improve compression speed. + * Deviating far from default value will likely result in a compression ratio decrease. + * Special: value 0 means "automatically determine hashRateLog". */ + ZSTD_c_ldmHashRateLog = 164, + + /* Content size will be written into frame header _whenever known_ (default:1) + * Content size must be known at the beginning of compression. + * This is automatically the case when using ZSTD_compress2(), + * For streaming scenarios, content size must be provided with ZSTD_CCtx_setPledgedSrcSize() */ + ZSTD_c_contentSizeFlag = 200, + + /* A 32-bits checksum of content is written at end of frame (default:0) */ + ZSTD_c_checksumFlag = 201, + + /* When applicable, dictionary's ID is written into frame header (default:1) */ + ZSTD_c_dictIDFlag = 202, + + /* Select how many threads will be spawned to compress in parallel. + * When nbWorkers >= 1, triggers asynchronous mode when invoking ZSTD_compressStream*() : + * ZSTD_compressStream*() consumes input and flush output if possible, but immediately gives back control to caller, + * while compression is performed in parallel, within worker thread(s). + * (note : a strong exception to this rule is when first invocation of ZSTD_compressStream2() sets ZSTD_e_end : + * in which case, ZSTD_compressStream2() delegates to ZSTD_compress2(), which is always a blocking call). + * More workers improve speed, but also increase memory usage. + * Default value is `0`, aka "single-threaded mode" : no worker is spawned, + * compression is performed inside Caller's thread, and all invocations are blocking */ + ZSTD_c_nbWorkers = 400, + + /* Size of a compression job. This value is enforced only when nbWorkers >= 1. + * Each compression job is completed in parallel, so this value can indirectly impact the nb of active threads. + * 0 means default, which is dynamically determined based on compression parameters. + * Job size must be a minimum of overlap size, or ZSTDMT_JOBSIZE_MIN (= 512 KB), whichever is largest. + * The minimum size is automatically and transparently enforced. */ + ZSTD_c_jobSize = 401, + + /* Control the overlap size, as a fraction of window size. + * The overlap size is an amount of data reloaded from previous job at the beginning of a new job. + * It helps preserve compression ratio, while each job is compressed in parallel. + * This value is enforced only when nbWorkers >= 1. + * Larger values increase compression ratio, but decrease speed. + * Possible values range from 0 to 9 : + * - 0 means "default" : value will be determined by the library, depending on strategy + * - 1 means "no overlap" + * - 9 means "full overlap", using a full window size. + * Each intermediate rank increases/decreases load size by a factor 2 : + * 9: full window; 8: w/2; 7: w/4; 6: w/8; 5:w/16; 4: w/32; 3:w/64; 2:w/128; 1:no overlap; 0:default + * default value varies between 6 and 9, depending on strategy */ + ZSTD_c_overlapLog = 402, + + /* note : additional experimental parameters are also available + * within the experimental section of the API. + * At the time of this writing, they include : + * ZSTD_c_rsyncable + * ZSTD_c_format + * ZSTD_c_forceMaxWindow + * ZSTD_c_forceAttachDict + * ZSTD_c_literalCompressionMode + * ZSTD_c_srcSizeHint + * ZSTD_c_enableDedicatedDictSearch + * ZSTD_c_stableInBuffer + * ZSTD_c_stableOutBuffer + * ZSTD_c_blockDelimiters + * ZSTD_c_validateSequences + * ZSTD_c_blockSplitterLevel + * ZSTD_c_splitAfterSequences + * ZSTD_c_useRowMatchFinder + * ZSTD_c_prefetchCDictTables + * ZSTD_c_enableSeqProducerFallback + * ZSTD_c_maxBlockSize + * Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them. + * note : never ever use experimentalParam? names directly; + * also, the enums values themselves are unstable and can still change. + */ + ZSTD_c_experimentalParam1 = 500, + ZSTD_c_experimentalParam2 = 10, + ZSTD_c_experimentalParam3 = 1000, + ZSTD_c_experimentalParam4 = 1001, + ZSTD_c_experimentalParam5 = 1002, + + /* was ZSTD_c_experimentalParam6=1003; is now ZSTD_c_targetCBlockSize */ + ZSTD_c_experimentalParam7 = 1004, + ZSTD_c_experimentalParam8 = 1005, + ZSTD_c_experimentalParam9 = 1006, + ZSTD_c_experimentalParam10 = 1007, + ZSTD_c_experimentalParam11 = 1008, + ZSTD_c_experimentalParam12 = 1009, + ZSTD_c_experimentalParam13 = 1010, + ZSTD_c_experimentalParam14 = 1011, + ZSTD_c_experimentalParam15 = 1012, + ZSTD_c_experimentalParam16 = 1013, + ZSTD_c_experimentalParam17 = 1014, + ZSTD_c_experimentalParam18 = 1015, + ZSTD_c_experimentalParam19 = 1016, + ZSTD_c_experimentalParam20 = 1017, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_cStreamStage.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_cStreamStage.cs new file mode 100644 index 00000000..c929b7fa --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_cStreamStage.cs @@ -0,0 +1,8 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum ZSTD_cStreamStage +{ + zcss_init = 0, + zcss_load, + zcss_flush, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_compResetPolicy_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_compResetPolicy_e.cs new file mode 100644 index 00000000..d943004d --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_compResetPolicy_e.cs @@ -0,0 +1,14 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/** + * Controls, for this matchState reset, whether the tables need to be cleared / + * prepared for the coming compression (ZSTDcrp_makeClean), or whether the + * tables can be left unclean (ZSTDcrp_leaveDirty), because we know that a + * subsequent operation will overwrite the table space anyways (e.g., copying + * the matchState contents in from a CDict). + */ +public enum ZSTD_compResetPolicy_e +{ + ZSTDcrp_makeClean, + ZSTDcrp_leaveDirty, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_compressedBlockState_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_compressedBlockState_t.cs new file mode 100644 index 00000000..af281c96 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_compressedBlockState_t.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct ZSTD_compressedBlockState_t +{ + public ZSTD_entropyCTables_t entropy; + public fixed uint rep[3]; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_compressionParameters.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_compressionParameters.cs new file mode 100644 index 00000000..bacbdea5 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_compressionParameters.cs @@ -0,0 +1,44 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct ZSTD_compressionParameters +{ + /**< largest match distance : larger == more compression, more memory needed during decompression */ + public uint windowLog; + + /**< fully searched segment : larger == more compression, slower, more memory (useless for fast) */ + public uint chainLog; + + /**< dispatch table : larger == faster, more memory */ + public uint hashLog; + + /**< nb of searches : larger == more compression, slower */ + public uint searchLog; + + /**< match length searched : larger == faster decompression, sometimes less compression */ + public uint minMatch; + + /**< acceptable match size for optimal parser (only) : larger == more compression, slower */ + public uint targetLength; + + /**< see ZSTD_strategy definition above */ + public ZSTD_strategy strategy; + + public ZSTD_compressionParameters( + uint windowLog, + uint chainLog, + uint hashLog, + uint searchLog, + uint minMatch, + uint targetLength, + ZSTD_strategy strategy + ) + { + this.windowLog = windowLog; + this.chainLog = chainLog; + this.hashLog = hashLog; + this.searchLog = searchLog; + this.minMatch = minMatch; + this.targetLength = targetLength; + this.strategy = strategy; + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_compressionStage_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_compressionStage_e.cs new file mode 100644 index 00000000..9ffa4833 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_compressionStage_e.cs @@ -0,0 +1,12 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/*-************************************* + * Context memory management + ***************************************/ +public enum ZSTD_compressionStage_e +{ + ZSTDcs_created = 0, + ZSTDcs_init, + ZSTDcs_ongoing, + ZSTDcs_ending, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_customMem.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_customMem.cs new file mode 100644 index 00000000..16776a74 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_customMem.cs @@ -0,0 +1,15 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct ZSTD_customMem +{ + public void* customAlloc; + public void* customFree; + public void* opaque; + + public ZSTD_customMem(void* customAlloc, void* customFree, void* opaque) + { + this.customAlloc = customAlloc; + this.customFree = customFree; + this.opaque = opaque; + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_cwksp.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_cwksp.cs new file mode 100644 index 00000000..847b0866 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_cwksp.cs @@ -0,0 +1,110 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/** + * Zstd fits all its internal datastructures into a single continuous buffer, + * so that it only needs to perform a single OS allocation (or so that a buffer + * can be provided to it and it can perform no allocations at all). This buffer + * is called the workspace. + * + * Several optimizations complicate that process of allocating memory ranges + * from this workspace for each internal datastructure: + * + * - These different internal datastructures have different setup requirements: + * + * - The static objects need to be cleared once and can then be trivially + * reused for each compression. + * + * - Various buffers don't need to be initialized at all--they are always + * written into before they're read. + * + * - The matchstate tables have a unique requirement that they don't need + * their memory to be totally cleared, but they do need the memory to have + * some bound, i.e., a guarantee that all values in the memory they've been + * allocated is less than some maximum value (which is the starting value + * for the indices that they will then use for compression). When this + * guarantee is provided to them, they can use the memory without any setup + * work. When it can't, they have to clear the area. + * + * - These buffers also have different alignment requirements. + * + * - We would like to reuse the objects in the workspace for multiple + * compressions without having to perform any expensive reallocation or + * reinitialization work. + * + * - We would like to be able to efficiently reuse the workspace across + * multiple compressions **even when the compression parameters change** and + * we need to resize some of the objects (where possible). + * + * To attempt to manage this buffer, given these constraints, the ZSTD_cwksp + * abstraction was created. It works as follows: + * + * Workspace Layout: + * + * [ ... workspace ... ] + * [objects][tables ->] free space [<- buffers][<- aligned][<- init once] + * + * The various objects that live in the workspace are divided into the + * following categories, and are allocated separately: + * + * - Static objects: this is optionally the enclosing ZSTD_CCtx or ZSTD_CDict, + * so that literally everything fits in a single buffer. Note: if present, + * this must be the first object in the workspace, since ZSTD_customFree{CCtx, + * CDict}() rely on a pointer comparison to see whether one or two frees are + * required. + * + * - Fixed size objects: these are fixed-size, fixed-count objects that are + * nonetheless "dynamically" allocated in the workspace so that we can + * control how they're initialized separately from the broader ZSTD_CCtx. + * Examples: + * - Entropy Workspace + * - 2 x ZSTD_compressedBlockState_t + * - CDict dictionary contents + * + * - Tables: these are any of several different datastructures (hash tables, + * chain tables, binary trees) that all respect a common format: they are + * uint32_t arrays, all of whose values are between 0 and (nextSrc - base). + * Their sizes depend on the cparams. These tables are 64-byte aligned. + * + * - Init once: these buffers require to be initialized at least once before + * use. They should be used when we want to skip memory initialization + * while not triggering memory checkers (like Valgrind) when reading from + * from this memory without writing to it first. + * These buffers should be used carefully as they might contain data + * from previous compressions. + * Buffers are aligned to 64 bytes. + * + * - Aligned: these buffers don't require any initialization before they're + * used. The user of the buffer should make sure they write into a buffer + * location before reading from it. + * Buffers are aligned to 64 bytes. + * + * - Buffers: these buffers are used for various purposes that don't require + * any alignment or initialization before they're used. This means they can + * be moved around at no cost for a new compression. + * + * Allocating Memory: + * + * The various types of objects must be allocated in order, so they can be + * correctly packed into the workspace buffer. That order is: + * + * 1. Objects + * 2. Init once / Tables + * 3. Aligned / Tables + * 4. Buffers / Tables + * + * Attempts to reserve objects of different types out of order will fail. + */ +public unsafe struct ZSTD_cwksp +{ + public void* workspace; + public void* workspaceEnd; + public void* objectEnd; + public void* tableEnd; + public void* tableValidEnd; + public void* allocStart; + public void* initOnceStart; + public byte allocFailed; + public int workspaceOversizedDuration; + public ZSTD_cwksp_alloc_phase_e phase; + public ZSTD_cwksp_static_alloc_e isStatic; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_cwksp_alloc_phase_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_cwksp_alloc_phase_e.cs new file mode 100644 index 00000000..c9fd5aac --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_cwksp_alloc_phase_e.cs @@ -0,0 +1,12 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/*-************************************* + * Structures + ***************************************/ +public enum ZSTD_cwksp_alloc_phase_e +{ + ZSTD_cwksp_alloc_objects, + ZSTD_cwksp_alloc_aligned_init_once, + ZSTD_cwksp_alloc_aligned, + ZSTD_cwksp_alloc_buffers, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_cwksp_static_alloc_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_cwksp_static_alloc_e.cs new file mode 100644 index 00000000..8f290f3a --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_cwksp_static_alloc_e.cs @@ -0,0 +1,12 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/** + * Used to describe whether the workspace is statically allocated (and will not + * necessarily ever be freed), or if it's dynamically allocated and we can + * expect a well-formed caller to free this. + */ +public enum ZSTD_cwksp_static_alloc_e +{ + ZSTD_cwksp_dynamic_alloc, + ZSTD_cwksp_static_alloc, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dParameter.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dParameter.cs new file mode 100644 index 00000000..2dc4ecba --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dParameter.cs @@ -0,0 +1,38 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/* The advanced API pushes parameters one by one into an existing DCtx context. + * Parameters are sticky, and remain valid for all following frames + * using the same DCtx context. + * It's possible to reset parameters to default values using ZSTD_DCtx_reset(). + * Note : This API is compatible with existing ZSTD_decompressDCtx() and ZSTD_decompressStream(). + * Therefore, no new decompression function is necessary. + */ +public enum ZSTD_dParameter +{ + /* Select a size limit (in power of 2) beyond which + * the streaming API will refuse to allocate memory buffer + * in order to protect the host from unreasonable memory requirements. + * This parameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode. + * By default, a decompression context accepts window sizes <= (1 << ZSTD_WINDOWLOG_LIMIT_DEFAULT). + * Special: value 0 means "use default maximum windowLog". */ + ZSTD_d_windowLogMax = 100, + + /* note : additional experimental parameters are also available + * within the experimental section of the API. + * At the time of this writing, they include : + * ZSTD_d_format + * ZSTD_d_stableOutBuffer + * ZSTD_d_forceIgnoreChecksum + * ZSTD_d_refMultipleDDicts + * ZSTD_d_disableHuffmanAssembly + * ZSTD_d_maxBlockSize + * Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them. + * note : never ever use experimentalParam? names directly + */ + ZSTD_d_experimentalParam1 = 1000, + ZSTD_d_experimentalParam2 = 1001, + ZSTD_d_experimentalParam3 = 1002, + ZSTD_d_experimentalParam4 = 1003, + ZSTD_d_experimentalParam5 = 1004, + ZSTD_d_experimentalParam6 = 1005, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dStage.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dStage.cs new file mode 100644 index 00000000..69b992a2 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dStage.cs @@ -0,0 +1,13 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum ZSTD_dStage +{ + ZSTDds_getFrameHeaderSize, + ZSTDds_decodeFrameHeader, + ZSTDds_decodeBlockHeader, + ZSTDds_decompressBlock, + ZSTDds_decompressLastBlock, + ZSTDds_checkChecksum, + ZSTDds_decodeSkippableHeader, + ZSTDds_skipFrame, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dStreamStage.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dStreamStage.cs new file mode 100644 index 00000000..851e06f7 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dStreamStage.cs @@ -0,0 +1,10 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum ZSTD_dStreamStage +{ + zdss_init = 0, + zdss_loadHeader, + zdss_read, + zdss_load, + zdss_flush, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dictAttachPref_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dictAttachPref_e.cs new file mode 100644 index 00000000..8f989a42 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dictAttachPref_e.cs @@ -0,0 +1,16 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum ZSTD_dictAttachPref_e +{ + /* Use the default heuristic. */ + ZSTD_dictDefaultAttach = 0, + + /* Never copy the dictionary. */ + ZSTD_dictForceAttach = 1, + + /* Always copy the dictionary. */ + ZSTD_dictForceCopy = 2, + + /* Always reload the dictionary */ + ZSTD_dictForceLoad = 3, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dictContentType_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dictContentType_e.cs new file mode 100644 index 00000000..3b97b12d --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dictContentType_e.cs @@ -0,0 +1,13 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum ZSTD_dictContentType_e +{ + /* dictionary is "full" when starting with ZSTD_MAGIC_DICTIONARY, otherwise it is "rawContent" */ + ZSTD_dct_auto = 0, + + /* ensures dictionary is always loaded as rawContent, even if it starts with ZSTD_MAGIC_DICTIONARY */ + ZSTD_dct_rawContent = 1, + + /* refuses to load a dictionary if it does not respect Zstandard's specification, starting with ZSTD_MAGIC_DICTIONARY */ + ZSTD_dct_fullDict = 2, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dictLoadMethod_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dictLoadMethod_e.cs new file mode 100644 index 00000000..af7ed50b --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dictLoadMethod_e.cs @@ -0,0 +1,10 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum ZSTD_dictLoadMethod_e +{ + /**< Copy dictionary content internally */ + ZSTD_dlm_byCopy = 0, + + /**< Reference dictionary content -- the dictionary buffer must outlive its users. */ + ZSTD_dlm_byRef = 1, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dictMode_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dictMode_e.cs new file mode 100644 index 00000000..f234551d --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dictMode_e.cs @@ -0,0 +1,9 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum ZSTD_dictMode_e +{ + ZSTD_noDict = 0, + ZSTD_extDict = 1, + ZSTD_dictMatchState = 2, + ZSTD_dedicatedDictSearch = 3, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dictTableLoadMethod_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dictTableLoadMethod_e.cs new file mode 100644 index 00000000..7cf4f3e5 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dictTableLoadMethod_e.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum ZSTD_dictTableLoadMethod_e +{ + ZSTD_dtlm_fast, + ZSTD_dtlm_full, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dictUses_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dictUses_e.cs new file mode 100644 index 00000000..314507c5 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_dictUses_e.cs @@ -0,0 +1,13 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum ZSTD_dictUses_e +{ + /* Use the dictionary indefinitely */ + ZSTD_use_indefinitely = -1, + + /* Do not use the dictionary (if one exists free it) */ + ZSTD_dont_use = 0, + + /* Use the dictionary once and set to ZSTD_dont_use */ + ZSTD_use_once = 1, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_entropyCTablesMetadata_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_entropyCTablesMetadata_t.cs new file mode 100644 index 00000000..2455590b --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_entropyCTablesMetadata_t.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct ZSTD_entropyCTablesMetadata_t +{ + public ZSTD_hufCTablesMetadata_t hufMetadata; + public ZSTD_fseCTablesMetadata_t fseMetadata; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_entropyCTables_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_entropyCTables_t.cs new file mode 100644 index 00000000..f3d98358 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_entropyCTables_t.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct ZSTD_entropyCTables_t +{ + public ZSTD_hufCTables_t huf; + public ZSTD_fseCTables_t fse; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_entropyDTables_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_entropyDTables_t.cs new file mode 100644 index 00000000..91dd9d7d --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_entropyDTables_t.cs @@ -0,0 +1,1342 @@ +using System.Runtime.CompilerServices; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct ZSTD_entropyDTables_t +{ + /* Note : Space reserved for FSE Tables */ + public _LLTable_e__FixedBuffer LLTable; + + /* is also used as temporary workspace while building hufTable during DDict creation */ + public _OFTable_e__FixedBuffer OFTable; + + /* and therefore must be at least HUF_DECOMPRESS_WORKSPACE_SIZE large */ + public _MLTable_e__FixedBuffer MLTable; + + /* can accommodate HUF_decompress4X */ + public fixed uint hufTable[4097]; + public fixed uint rep[3]; + public fixed uint workspace[157]; + +#if NET8_0_OR_GREATER + [InlineArray(513)] + public unsafe struct _LLTable_e__FixedBuffer + { + public ZSTD_seqSymbol e0; + } + +#else + public unsafe struct _LLTable_e__FixedBuffer + { + public ZSTD_seqSymbol e0; + public ZSTD_seqSymbol e1; + public ZSTD_seqSymbol e2; + public ZSTD_seqSymbol e3; + public ZSTD_seqSymbol e4; + public ZSTD_seqSymbol e5; + public ZSTD_seqSymbol e6; + public ZSTD_seqSymbol e7; + public ZSTD_seqSymbol e8; + public ZSTD_seqSymbol e9; + public ZSTD_seqSymbol e10; + public ZSTD_seqSymbol e11; + public ZSTD_seqSymbol e12; + public ZSTD_seqSymbol e13; + public ZSTD_seqSymbol e14; + public ZSTD_seqSymbol e15; + public ZSTD_seqSymbol e16; + public ZSTD_seqSymbol e17; + public ZSTD_seqSymbol e18; + public ZSTD_seqSymbol e19; + public ZSTD_seqSymbol e20; + public ZSTD_seqSymbol e21; + public ZSTD_seqSymbol e22; + public ZSTD_seqSymbol e23; + public ZSTD_seqSymbol e24; + public ZSTD_seqSymbol e25; + public ZSTD_seqSymbol e26; + public ZSTD_seqSymbol e27; + public ZSTD_seqSymbol e28; + public ZSTD_seqSymbol e29; + public ZSTD_seqSymbol e30; + public ZSTD_seqSymbol e31; + public ZSTD_seqSymbol e32; + public ZSTD_seqSymbol e33; + public ZSTD_seqSymbol e34; + public ZSTD_seqSymbol e35; + public ZSTD_seqSymbol e36; + public ZSTD_seqSymbol e37; + public ZSTD_seqSymbol e38; + public ZSTD_seqSymbol e39; + public ZSTD_seqSymbol e40; + public ZSTD_seqSymbol e41; + public ZSTD_seqSymbol e42; + public ZSTD_seqSymbol e43; + public ZSTD_seqSymbol e44; + public ZSTD_seqSymbol e45; + public ZSTD_seqSymbol e46; + public ZSTD_seqSymbol e47; + public ZSTD_seqSymbol e48; + public ZSTD_seqSymbol e49; + public ZSTD_seqSymbol e50; + public ZSTD_seqSymbol e51; + public ZSTD_seqSymbol e52; + public ZSTD_seqSymbol e53; + public ZSTD_seqSymbol e54; + public ZSTD_seqSymbol e55; + public ZSTD_seqSymbol e56; + public ZSTD_seqSymbol e57; + public ZSTD_seqSymbol e58; + public ZSTD_seqSymbol e59; + public ZSTD_seqSymbol e60; + public ZSTD_seqSymbol e61; + public ZSTD_seqSymbol e62; + public ZSTD_seqSymbol e63; + public ZSTD_seqSymbol e64; + public ZSTD_seqSymbol e65; + public ZSTD_seqSymbol e66; + public ZSTD_seqSymbol e67; + public ZSTD_seqSymbol e68; + public ZSTD_seqSymbol e69; + public ZSTD_seqSymbol e70; + public ZSTD_seqSymbol e71; + public ZSTD_seqSymbol e72; + public ZSTD_seqSymbol e73; + public ZSTD_seqSymbol e74; + public ZSTD_seqSymbol e75; + public ZSTD_seqSymbol e76; + public ZSTD_seqSymbol e77; + public ZSTD_seqSymbol e78; + public ZSTD_seqSymbol e79; + public ZSTD_seqSymbol e80; + public ZSTD_seqSymbol e81; + public ZSTD_seqSymbol e82; + public ZSTD_seqSymbol e83; + public ZSTD_seqSymbol e84; + public ZSTD_seqSymbol e85; + public ZSTD_seqSymbol e86; + public ZSTD_seqSymbol e87; + public ZSTD_seqSymbol e88; + public ZSTD_seqSymbol e89; + public ZSTD_seqSymbol e90; + public ZSTD_seqSymbol e91; + public ZSTD_seqSymbol e92; + public ZSTD_seqSymbol e93; + public ZSTD_seqSymbol e94; + public ZSTD_seqSymbol e95; + public ZSTD_seqSymbol e96; + public ZSTD_seqSymbol e97; + public ZSTD_seqSymbol e98; + public ZSTD_seqSymbol e99; + public ZSTD_seqSymbol e100; + public ZSTD_seqSymbol e101; + public ZSTD_seqSymbol e102; + public ZSTD_seqSymbol e103; + public ZSTD_seqSymbol e104; + public ZSTD_seqSymbol e105; + public ZSTD_seqSymbol e106; + public ZSTD_seqSymbol e107; + public ZSTD_seqSymbol e108; + public ZSTD_seqSymbol e109; + public ZSTD_seqSymbol e110; + public ZSTD_seqSymbol e111; + public ZSTD_seqSymbol e112; + public ZSTD_seqSymbol e113; + public ZSTD_seqSymbol e114; + public ZSTD_seqSymbol e115; + public ZSTD_seqSymbol e116; + public ZSTD_seqSymbol e117; + public ZSTD_seqSymbol e118; + public ZSTD_seqSymbol e119; + public ZSTD_seqSymbol e120; + public ZSTD_seqSymbol e121; + public ZSTD_seqSymbol e122; + public ZSTD_seqSymbol e123; + public ZSTD_seqSymbol e124; + public ZSTD_seqSymbol e125; + public ZSTD_seqSymbol e126; + public ZSTD_seqSymbol e127; + public ZSTD_seqSymbol e128; + public ZSTD_seqSymbol e129; + public ZSTD_seqSymbol e130; + public ZSTD_seqSymbol e131; + public ZSTD_seqSymbol e132; + public ZSTD_seqSymbol e133; + public ZSTD_seqSymbol e134; + public ZSTD_seqSymbol e135; + public ZSTD_seqSymbol e136; + public ZSTD_seqSymbol e137; + public ZSTD_seqSymbol e138; + public ZSTD_seqSymbol e139; + public ZSTD_seqSymbol e140; + public ZSTD_seqSymbol e141; + public ZSTD_seqSymbol e142; + public ZSTD_seqSymbol e143; + public ZSTD_seqSymbol e144; + public ZSTD_seqSymbol e145; + public ZSTD_seqSymbol e146; + public ZSTD_seqSymbol e147; + public ZSTD_seqSymbol e148; + public ZSTD_seqSymbol e149; + public ZSTD_seqSymbol e150; + public ZSTD_seqSymbol e151; + public ZSTD_seqSymbol e152; + public ZSTD_seqSymbol e153; + public ZSTD_seqSymbol e154; + public ZSTD_seqSymbol e155; + public ZSTD_seqSymbol e156; + public ZSTD_seqSymbol e157; + public ZSTD_seqSymbol e158; + public ZSTD_seqSymbol e159; + public ZSTD_seqSymbol e160; + public ZSTD_seqSymbol e161; + public ZSTD_seqSymbol e162; + public ZSTD_seqSymbol e163; + public ZSTD_seqSymbol e164; + public ZSTD_seqSymbol e165; + public ZSTD_seqSymbol e166; + public ZSTD_seqSymbol e167; + public ZSTD_seqSymbol e168; + public ZSTD_seqSymbol e169; + public ZSTD_seqSymbol e170; + public ZSTD_seqSymbol e171; + public ZSTD_seqSymbol e172; + public ZSTD_seqSymbol e173; + public ZSTD_seqSymbol e174; + public ZSTD_seqSymbol e175; + public ZSTD_seqSymbol e176; + public ZSTD_seqSymbol e177; + public ZSTD_seqSymbol e178; + public ZSTD_seqSymbol e179; + public ZSTD_seqSymbol e180; + public ZSTD_seqSymbol e181; + public ZSTD_seqSymbol e182; + public ZSTD_seqSymbol e183; + public ZSTD_seqSymbol e184; + public ZSTD_seqSymbol e185; + public ZSTD_seqSymbol e186; + public ZSTD_seqSymbol e187; + public ZSTD_seqSymbol e188; + public ZSTD_seqSymbol e189; + public ZSTD_seqSymbol e190; + public ZSTD_seqSymbol e191; + public ZSTD_seqSymbol e192; + public ZSTD_seqSymbol e193; + public ZSTD_seqSymbol e194; + public ZSTD_seqSymbol e195; + public ZSTD_seqSymbol e196; + public ZSTD_seqSymbol e197; + public ZSTD_seqSymbol e198; + public ZSTD_seqSymbol e199; + public ZSTD_seqSymbol e200; + public ZSTD_seqSymbol e201; + public ZSTD_seqSymbol e202; + public ZSTD_seqSymbol e203; + public ZSTD_seqSymbol e204; + public ZSTD_seqSymbol e205; + public ZSTD_seqSymbol e206; + public ZSTD_seqSymbol e207; + public ZSTD_seqSymbol e208; + public ZSTD_seqSymbol e209; + public ZSTD_seqSymbol e210; + public ZSTD_seqSymbol e211; + public ZSTD_seqSymbol e212; + public ZSTD_seqSymbol e213; + public ZSTD_seqSymbol e214; + public ZSTD_seqSymbol e215; + public ZSTD_seqSymbol e216; + public ZSTD_seqSymbol e217; + public ZSTD_seqSymbol e218; + public ZSTD_seqSymbol e219; + public ZSTD_seqSymbol e220; + public ZSTD_seqSymbol e221; + public ZSTD_seqSymbol e222; + public ZSTD_seqSymbol e223; + public ZSTD_seqSymbol e224; + public ZSTD_seqSymbol e225; + public ZSTD_seqSymbol e226; + public ZSTD_seqSymbol e227; + public ZSTD_seqSymbol e228; + public ZSTD_seqSymbol e229; + public ZSTD_seqSymbol e230; + public ZSTD_seqSymbol e231; + public ZSTD_seqSymbol e232; + public ZSTD_seqSymbol e233; + public ZSTD_seqSymbol e234; + public ZSTD_seqSymbol e235; + public ZSTD_seqSymbol e236; + public ZSTD_seqSymbol e237; + public ZSTD_seqSymbol e238; + public ZSTD_seqSymbol e239; + public ZSTD_seqSymbol e240; + public ZSTD_seqSymbol e241; + public ZSTD_seqSymbol e242; + public ZSTD_seqSymbol e243; + public ZSTD_seqSymbol e244; + public ZSTD_seqSymbol e245; + public ZSTD_seqSymbol e246; + public ZSTD_seqSymbol e247; + public ZSTD_seqSymbol e248; + public ZSTD_seqSymbol e249; + public ZSTD_seqSymbol e250; + public ZSTD_seqSymbol e251; + public ZSTD_seqSymbol e252; + public ZSTD_seqSymbol e253; + public ZSTD_seqSymbol e254; + public ZSTD_seqSymbol e255; + public ZSTD_seqSymbol e256; + public ZSTD_seqSymbol e257; + public ZSTD_seqSymbol e258; + public ZSTD_seqSymbol e259; + public ZSTD_seqSymbol e260; + public ZSTD_seqSymbol e261; + public ZSTD_seqSymbol e262; + public ZSTD_seqSymbol e263; + public ZSTD_seqSymbol e264; + public ZSTD_seqSymbol e265; + public ZSTD_seqSymbol e266; + public ZSTD_seqSymbol e267; + public ZSTD_seqSymbol e268; + public ZSTD_seqSymbol e269; + public ZSTD_seqSymbol e270; + public ZSTD_seqSymbol e271; + public ZSTD_seqSymbol e272; + public ZSTD_seqSymbol e273; + public ZSTD_seqSymbol e274; + public ZSTD_seqSymbol e275; + public ZSTD_seqSymbol e276; + public ZSTD_seqSymbol e277; + public ZSTD_seqSymbol e278; + public ZSTD_seqSymbol e279; + public ZSTD_seqSymbol e280; + public ZSTD_seqSymbol e281; + public ZSTD_seqSymbol e282; + public ZSTD_seqSymbol e283; + public ZSTD_seqSymbol e284; + public ZSTD_seqSymbol e285; + public ZSTD_seqSymbol e286; + public ZSTD_seqSymbol e287; + public ZSTD_seqSymbol e288; + public ZSTD_seqSymbol e289; + public ZSTD_seqSymbol e290; + public ZSTD_seqSymbol e291; + public ZSTD_seqSymbol e292; + public ZSTD_seqSymbol e293; + public ZSTD_seqSymbol e294; + public ZSTD_seqSymbol e295; + public ZSTD_seqSymbol e296; + public ZSTD_seqSymbol e297; + public ZSTD_seqSymbol e298; + public ZSTD_seqSymbol e299; + public ZSTD_seqSymbol e300; + public ZSTD_seqSymbol e301; + public ZSTD_seqSymbol e302; + public ZSTD_seqSymbol e303; + public ZSTD_seqSymbol e304; + public ZSTD_seqSymbol e305; + public ZSTD_seqSymbol e306; + public ZSTD_seqSymbol e307; + public ZSTD_seqSymbol e308; + public ZSTD_seqSymbol e309; + public ZSTD_seqSymbol e310; + public ZSTD_seqSymbol e311; + public ZSTD_seqSymbol e312; + public ZSTD_seqSymbol e313; + public ZSTD_seqSymbol e314; + public ZSTD_seqSymbol e315; + public ZSTD_seqSymbol e316; + public ZSTD_seqSymbol e317; + public ZSTD_seqSymbol e318; + public ZSTD_seqSymbol e319; + public ZSTD_seqSymbol e320; + public ZSTD_seqSymbol e321; + public ZSTD_seqSymbol e322; + public ZSTD_seqSymbol e323; + public ZSTD_seqSymbol e324; + public ZSTD_seqSymbol e325; + public ZSTD_seqSymbol e326; + public ZSTD_seqSymbol e327; + public ZSTD_seqSymbol e328; + public ZSTD_seqSymbol e329; + public ZSTD_seqSymbol e330; + public ZSTD_seqSymbol e331; + public ZSTD_seqSymbol e332; + public ZSTD_seqSymbol e333; + public ZSTD_seqSymbol e334; + public ZSTD_seqSymbol e335; + public ZSTD_seqSymbol e336; + public ZSTD_seqSymbol e337; + public ZSTD_seqSymbol e338; + public ZSTD_seqSymbol e339; + public ZSTD_seqSymbol e340; + public ZSTD_seqSymbol e341; + public ZSTD_seqSymbol e342; + public ZSTD_seqSymbol e343; + public ZSTD_seqSymbol e344; + public ZSTD_seqSymbol e345; + public ZSTD_seqSymbol e346; + public ZSTD_seqSymbol e347; + public ZSTD_seqSymbol e348; + public ZSTD_seqSymbol e349; + public ZSTD_seqSymbol e350; + public ZSTD_seqSymbol e351; + public ZSTD_seqSymbol e352; + public ZSTD_seqSymbol e353; + public ZSTD_seqSymbol e354; + public ZSTD_seqSymbol e355; + public ZSTD_seqSymbol e356; + public ZSTD_seqSymbol e357; + public ZSTD_seqSymbol e358; + public ZSTD_seqSymbol e359; + public ZSTD_seqSymbol e360; + public ZSTD_seqSymbol e361; + public ZSTD_seqSymbol e362; + public ZSTD_seqSymbol e363; + public ZSTD_seqSymbol e364; + public ZSTD_seqSymbol e365; + public ZSTD_seqSymbol e366; + public ZSTD_seqSymbol e367; + public ZSTD_seqSymbol e368; + public ZSTD_seqSymbol e369; + public ZSTD_seqSymbol e370; + public ZSTD_seqSymbol e371; + public ZSTD_seqSymbol e372; + public ZSTD_seqSymbol e373; + public ZSTD_seqSymbol e374; + public ZSTD_seqSymbol e375; + public ZSTD_seqSymbol e376; + public ZSTD_seqSymbol e377; + public ZSTD_seqSymbol e378; + public ZSTD_seqSymbol e379; + public ZSTD_seqSymbol e380; + public ZSTD_seqSymbol e381; + public ZSTD_seqSymbol e382; + public ZSTD_seqSymbol e383; + public ZSTD_seqSymbol e384; + public ZSTD_seqSymbol e385; + public ZSTD_seqSymbol e386; + public ZSTD_seqSymbol e387; + public ZSTD_seqSymbol e388; + public ZSTD_seqSymbol e389; + public ZSTD_seqSymbol e390; + public ZSTD_seqSymbol e391; + public ZSTD_seqSymbol e392; + public ZSTD_seqSymbol e393; + public ZSTD_seqSymbol e394; + public ZSTD_seqSymbol e395; + public ZSTD_seqSymbol e396; + public ZSTD_seqSymbol e397; + public ZSTD_seqSymbol e398; + public ZSTD_seqSymbol e399; + public ZSTD_seqSymbol e400; + public ZSTD_seqSymbol e401; + public ZSTD_seqSymbol e402; + public ZSTD_seqSymbol e403; + public ZSTD_seqSymbol e404; + public ZSTD_seqSymbol e405; + public ZSTD_seqSymbol e406; + public ZSTD_seqSymbol e407; + public ZSTD_seqSymbol e408; + public ZSTD_seqSymbol e409; + public ZSTD_seqSymbol e410; + public ZSTD_seqSymbol e411; + public ZSTD_seqSymbol e412; + public ZSTD_seqSymbol e413; + public ZSTD_seqSymbol e414; + public ZSTD_seqSymbol e415; + public ZSTD_seqSymbol e416; + public ZSTD_seqSymbol e417; + public ZSTD_seqSymbol e418; + public ZSTD_seqSymbol e419; + public ZSTD_seqSymbol e420; + public ZSTD_seqSymbol e421; + public ZSTD_seqSymbol e422; + public ZSTD_seqSymbol e423; + public ZSTD_seqSymbol e424; + public ZSTD_seqSymbol e425; + public ZSTD_seqSymbol e426; + public ZSTD_seqSymbol e427; + public ZSTD_seqSymbol e428; + public ZSTD_seqSymbol e429; + public ZSTD_seqSymbol e430; + public ZSTD_seqSymbol e431; + public ZSTD_seqSymbol e432; + public ZSTD_seqSymbol e433; + public ZSTD_seqSymbol e434; + public ZSTD_seqSymbol e435; + public ZSTD_seqSymbol e436; + public ZSTD_seqSymbol e437; + public ZSTD_seqSymbol e438; + public ZSTD_seqSymbol e439; + public ZSTD_seqSymbol e440; + public ZSTD_seqSymbol e441; + public ZSTD_seqSymbol e442; + public ZSTD_seqSymbol e443; + public ZSTD_seqSymbol e444; + public ZSTD_seqSymbol e445; + public ZSTD_seqSymbol e446; + public ZSTD_seqSymbol e447; + public ZSTD_seqSymbol e448; + public ZSTD_seqSymbol e449; + public ZSTD_seqSymbol e450; + public ZSTD_seqSymbol e451; + public ZSTD_seqSymbol e452; + public ZSTD_seqSymbol e453; + public ZSTD_seqSymbol e454; + public ZSTD_seqSymbol e455; + public ZSTD_seqSymbol e456; + public ZSTD_seqSymbol e457; + public ZSTD_seqSymbol e458; + public ZSTD_seqSymbol e459; + public ZSTD_seqSymbol e460; + public ZSTD_seqSymbol e461; + public ZSTD_seqSymbol e462; + public ZSTD_seqSymbol e463; + public ZSTD_seqSymbol e464; + public ZSTD_seqSymbol e465; + public ZSTD_seqSymbol e466; + public ZSTD_seqSymbol e467; + public ZSTD_seqSymbol e468; + public ZSTD_seqSymbol e469; + public ZSTD_seqSymbol e470; + public ZSTD_seqSymbol e471; + public ZSTD_seqSymbol e472; + public ZSTD_seqSymbol e473; + public ZSTD_seqSymbol e474; + public ZSTD_seqSymbol e475; + public ZSTD_seqSymbol e476; + public ZSTD_seqSymbol e477; + public ZSTD_seqSymbol e478; + public ZSTD_seqSymbol e479; + public ZSTD_seqSymbol e480; + public ZSTD_seqSymbol e481; + public ZSTD_seqSymbol e482; + public ZSTD_seqSymbol e483; + public ZSTD_seqSymbol e484; + public ZSTD_seqSymbol e485; + public ZSTD_seqSymbol e486; + public ZSTD_seqSymbol e487; + public ZSTD_seqSymbol e488; + public ZSTD_seqSymbol e489; + public ZSTD_seqSymbol e490; + public ZSTD_seqSymbol e491; + public ZSTD_seqSymbol e492; + public ZSTD_seqSymbol e493; + public ZSTD_seqSymbol e494; + public ZSTD_seqSymbol e495; + public ZSTD_seqSymbol e496; + public ZSTD_seqSymbol e497; + public ZSTD_seqSymbol e498; + public ZSTD_seqSymbol e499; + public ZSTD_seqSymbol e500; + public ZSTD_seqSymbol e501; + public ZSTD_seqSymbol e502; + public ZSTD_seqSymbol e503; + public ZSTD_seqSymbol e504; + public ZSTD_seqSymbol e505; + public ZSTD_seqSymbol e506; + public ZSTD_seqSymbol e507; + public ZSTD_seqSymbol e508; + public ZSTD_seqSymbol e509; + public ZSTD_seqSymbol e510; + public ZSTD_seqSymbol e511; + public ZSTD_seqSymbol e512; + } +#endif + +#if NET8_0_OR_GREATER + [InlineArray(257)] + public unsafe struct _OFTable_e__FixedBuffer + { + public ZSTD_seqSymbol e0; + } + +#else + public unsafe struct _OFTable_e__FixedBuffer + { + public ZSTD_seqSymbol e0; + public ZSTD_seqSymbol e1; + public ZSTD_seqSymbol e2; + public ZSTD_seqSymbol e3; + public ZSTD_seqSymbol e4; + public ZSTD_seqSymbol e5; + public ZSTD_seqSymbol e6; + public ZSTD_seqSymbol e7; + public ZSTD_seqSymbol e8; + public ZSTD_seqSymbol e9; + public ZSTD_seqSymbol e10; + public ZSTD_seqSymbol e11; + public ZSTD_seqSymbol e12; + public ZSTD_seqSymbol e13; + public ZSTD_seqSymbol e14; + public ZSTD_seqSymbol e15; + public ZSTD_seqSymbol e16; + public ZSTD_seqSymbol e17; + public ZSTD_seqSymbol e18; + public ZSTD_seqSymbol e19; + public ZSTD_seqSymbol e20; + public ZSTD_seqSymbol e21; + public ZSTD_seqSymbol e22; + public ZSTD_seqSymbol e23; + public ZSTD_seqSymbol e24; + public ZSTD_seqSymbol e25; + public ZSTD_seqSymbol e26; + public ZSTD_seqSymbol e27; + public ZSTD_seqSymbol e28; + public ZSTD_seqSymbol e29; + public ZSTD_seqSymbol e30; + public ZSTD_seqSymbol e31; + public ZSTD_seqSymbol e32; + public ZSTD_seqSymbol e33; + public ZSTD_seqSymbol e34; + public ZSTD_seqSymbol e35; + public ZSTD_seqSymbol e36; + public ZSTD_seqSymbol e37; + public ZSTD_seqSymbol e38; + public ZSTD_seqSymbol e39; + public ZSTD_seqSymbol e40; + public ZSTD_seqSymbol e41; + public ZSTD_seqSymbol e42; + public ZSTD_seqSymbol e43; + public ZSTD_seqSymbol e44; + public ZSTD_seqSymbol e45; + public ZSTD_seqSymbol e46; + public ZSTD_seqSymbol e47; + public ZSTD_seqSymbol e48; + public ZSTD_seqSymbol e49; + public ZSTD_seqSymbol e50; + public ZSTD_seqSymbol e51; + public ZSTD_seqSymbol e52; + public ZSTD_seqSymbol e53; + public ZSTD_seqSymbol e54; + public ZSTD_seqSymbol e55; + public ZSTD_seqSymbol e56; + public ZSTD_seqSymbol e57; + public ZSTD_seqSymbol e58; + public ZSTD_seqSymbol e59; + public ZSTD_seqSymbol e60; + public ZSTD_seqSymbol e61; + public ZSTD_seqSymbol e62; + public ZSTD_seqSymbol e63; + public ZSTD_seqSymbol e64; + public ZSTD_seqSymbol e65; + public ZSTD_seqSymbol e66; + public ZSTD_seqSymbol e67; + public ZSTD_seqSymbol e68; + public ZSTD_seqSymbol e69; + public ZSTD_seqSymbol e70; + public ZSTD_seqSymbol e71; + public ZSTD_seqSymbol e72; + public ZSTD_seqSymbol e73; + public ZSTD_seqSymbol e74; + public ZSTD_seqSymbol e75; + public ZSTD_seqSymbol e76; + public ZSTD_seqSymbol e77; + public ZSTD_seqSymbol e78; + public ZSTD_seqSymbol e79; + public ZSTD_seqSymbol e80; + public ZSTD_seqSymbol e81; + public ZSTD_seqSymbol e82; + public ZSTD_seqSymbol e83; + public ZSTD_seqSymbol e84; + public ZSTD_seqSymbol e85; + public ZSTD_seqSymbol e86; + public ZSTD_seqSymbol e87; + public ZSTD_seqSymbol e88; + public ZSTD_seqSymbol e89; + public ZSTD_seqSymbol e90; + public ZSTD_seqSymbol e91; + public ZSTD_seqSymbol e92; + public ZSTD_seqSymbol e93; + public ZSTD_seqSymbol e94; + public ZSTD_seqSymbol e95; + public ZSTD_seqSymbol e96; + public ZSTD_seqSymbol e97; + public ZSTD_seqSymbol e98; + public ZSTD_seqSymbol e99; + public ZSTD_seqSymbol e100; + public ZSTD_seqSymbol e101; + public ZSTD_seqSymbol e102; + public ZSTD_seqSymbol e103; + public ZSTD_seqSymbol e104; + public ZSTD_seqSymbol e105; + public ZSTD_seqSymbol e106; + public ZSTD_seqSymbol e107; + public ZSTD_seqSymbol e108; + public ZSTD_seqSymbol e109; + public ZSTD_seqSymbol e110; + public ZSTD_seqSymbol e111; + public ZSTD_seqSymbol e112; + public ZSTD_seqSymbol e113; + public ZSTD_seqSymbol e114; + public ZSTD_seqSymbol e115; + public ZSTD_seqSymbol e116; + public ZSTD_seqSymbol e117; + public ZSTD_seqSymbol e118; + public ZSTD_seqSymbol e119; + public ZSTD_seqSymbol e120; + public ZSTD_seqSymbol e121; + public ZSTD_seqSymbol e122; + public ZSTD_seqSymbol e123; + public ZSTD_seqSymbol e124; + public ZSTD_seqSymbol e125; + public ZSTD_seqSymbol e126; + public ZSTD_seqSymbol e127; + public ZSTD_seqSymbol e128; + public ZSTD_seqSymbol e129; + public ZSTD_seqSymbol e130; + public ZSTD_seqSymbol e131; + public ZSTD_seqSymbol e132; + public ZSTD_seqSymbol e133; + public ZSTD_seqSymbol e134; + public ZSTD_seqSymbol e135; + public ZSTD_seqSymbol e136; + public ZSTD_seqSymbol e137; + public ZSTD_seqSymbol e138; + public ZSTD_seqSymbol e139; + public ZSTD_seqSymbol e140; + public ZSTD_seqSymbol e141; + public ZSTD_seqSymbol e142; + public ZSTD_seqSymbol e143; + public ZSTD_seqSymbol e144; + public ZSTD_seqSymbol e145; + public ZSTD_seqSymbol e146; + public ZSTD_seqSymbol e147; + public ZSTD_seqSymbol e148; + public ZSTD_seqSymbol e149; + public ZSTD_seqSymbol e150; + public ZSTD_seqSymbol e151; + public ZSTD_seqSymbol e152; + public ZSTD_seqSymbol e153; + public ZSTD_seqSymbol e154; + public ZSTD_seqSymbol e155; + public ZSTD_seqSymbol e156; + public ZSTD_seqSymbol e157; + public ZSTD_seqSymbol e158; + public ZSTD_seqSymbol e159; + public ZSTD_seqSymbol e160; + public ZSTD_seqSymbol e161; + public ZSTD_seqSymbol e162; + public ZSTD_seqSymbol e163; + public ZSTD_seqSymbol e164; + public ZSTD_seqSymbol e165; + public ZSTD_seqSymbol e166; + public ZSTD_seqSymbol e167; + public ZSTD_seqSymbol e168; + public ZSTD_seqSymbol e169; + public ZSTD_seqSymbol e170; + public ZSTD_seqSymbol e171; + public ZSTD_seqSymbol e172; + public ZSTD_seqSymbol e173; + public ZSTD_seqSymbol e174; + public ZSTD_seqSymbol e175; + public ZSTD_seqSymbol e176; + public ZSTD_seqSymbol e177; + public ZSTD_seqSymbol e178; + public ZSTD_seqSymbol e179; + public ZSTD_seqSymbol e180; + public ZSTD_seqSymbol e181; + public ZSTD_seqSymbol e182; + public ZSTD_seqSymbol e183; + public ZSTD_seqSymbol e184; + public ZSTD_seqSymbol e185; + public ZSTD_seqSymbol e186; + public ZSTD_seqSymbol e187; + public ZSTD_seqSymbol e188; + public ZSTD_seqSymbol e189; + public ZSTD_seqSymbol e190; + public ZSTD_seqSymbol e191; + public ZSTD_seqSymbol e192; + public ZSTD_seqSymbol e193; + public ZSTD_seqSymbol e194; + public ZSTD_seqSymbol e195; + public ZSTD_seqSymbol e196; + public ZSTD_seqSymbol e197; + public ZSTD_seqSymbol e198; + public ZSTD_seqSymbol e199; + public ZSTD_seqSymbol e200; + public ZSTD_seqSymbol e201; + public ZSTD_seqSymbol e202; + public ZSTD_seqSymbol e203; + public ZSTD_seqSymbol e204; + public ZSTD_seqSymbol e205; + public ZSTD_seqSymbol e206; + public ZSTD_seqSymbol e207; + public ZSTD_seqSymbol e208; + public ZSTD_seqSymbol e209; + public ZSTD_seqSymbol e210; + public ZSTD_seqSymbol e211; + public ZSTD_seqSymbol e212; + public ZSTD_seqSymbol e213; + public ZSTD_seqSymbol e214; + public ZSTD_seqSymbol e215; + public ZSTD_seqSymbol e216; + public ZSTD_seqSymbol e217; + public ZSTD_seqSymbol e218; + public ZSTD_seqSymbol e219; + public ZSTD_seqSymbol e220; + public ZSTD_seqSymbol e221; + public ZSTD_seqSymbol e222; + public ZSTD_seqSymbol e223; + public ZSTD_seqSymbol e224; + public ZSTD_seqSymbol e225; + public ZSTD_seqSymbol e226; + public ZSTD_seqSymbol e227; + public ZSTD_seqSymbol e228; + public ZSTD_seqSymbol e229; + public ZSTD_seqSymbol e230; + public ZSTD_seqSymbol e231; + public ZSTD_seqSymbol e232; + public ZSTD_seqSymbol e233; + public ZSTD_seqSymbol e234; + public ZSTD_seqSymbol e235; + public ZSTD_seqSymbol e236; + public ZSTD_seqSymbol e237; + public ZSTD_seqSymbol e238; + public ZSTD_seqSymbol e239; + public ZSTD_seqSymbol e240; + public ZSTD_seqSymbol e241; + public ZSTD_seqSymbol e242; + public ZSTD_seqSymbol e243; + public ZSTD_seqSymbol e244; + public ZSTD_seqSymbol e245; + public ZSTD_seqSymbol e246; + public ZSTD_seqSymbol e247; + public ZSTD_seqSymbol e248; + public ZSTD_seqSymbol e249; + public ZSTD_seqSymbol e250; + public ZSTD_seqSymbol e251; + public ZSTD_seqSymbol e252; + public ZSTD_seqSymbol e253; + public ZSTD_seqSymbol e254; + public ZSTD_seqSymbol e255; + public ZSTD_seqSymbol e256; + } +#endif + +#if NET8_0_OR_GREATER + [InlineArray(513)] + public unsafe struct _MLTable_e__FixedBuffer + { + public ZSTD_seqSymbol e0; + } + +#else + public unsafe struct _MLTable_e__FixedBuffer + { + public ZSTD_seqSymbol e0; + public ZSTD_seqSymbol e1; + public ZSTD_seqSymbol e2; + public ZSTD_seqSymbol e3; + public ZSTD_seqSymbol e4; + public ZSTD_seqSymbol e5; + public ZSTD_seqSymbol e6; + public ZSTD_seqSymbol e7; + public ZSTD_seqSymbol e8; + public ZSTD_seqSymbol e9; + public ZSTD_seqSymbol e10; + public ZSTD_seqSymbol e11; + public ZSTD_seqSymbol e12; + public ZSTD_seqSymbol e13; + public ZSTD_seqSymbol e14; + public ZSTD_seqSymbol e15; + public ZSTD_seqSymbol e16; + public ZSTD_seqSymbol e17; + public ZSTD_seqSymbol e18; + public ZSTD_seqSymbol e19; + public ZSTD_seqSymbol e20; + public ZSTD_seqSymbol e21; + public ZSTD_seqSymbol e22; + public ZSTD_seqSymbol e23; + public ZSTD_seqSymbol e24; + public ZSTD_seqSymbol e25; + public ZSTD_seqSymbol e26; + public ZSTD_seqSymbol e27; + public ZSTD_seqSymbol e28; + public ZSTD_seqSymbol e29; + public ZSTD_seqSymbol e30; + public ZSTD_seqSymbol e31; + public ZSTD_seqSymbol e32; + public ZSTD_seqSymbol e33; + public ZSTD_seqSymbol e34; + public ZSTD_seqSymbol e35; + public ZSTD_seqSymbol e36; + public ZSTD_seqSymbol e37; + public ZSTD_seqSymbol e38; + public ZSTD_seqSymbol e39; + public ZSTD_seqSymbol e40; + public ZSTD_seqSymbol e41; + public ZSTD_seqSymbol e42; + public ZSTD_seqSymbol e43; + public ZSTD_seqSymbol e44; + public ZSTD_seqSymbol e45; + public ZSTD_seqSymbol e46; + public ZSTD_seqSymbol e47; + public ZSTD_seqSymbol e48; + public ZSTD_seqSymbol e49; + public ZSTD_seqSymbol e50; + public ZSTD_seqSymbol e51; + public ZSTD_seqSymbol e52; + public ZSTD_seqSymbol e53; + public ZSTD_seqSymbol e54; + public ZSTD_seqSymbol e55; + public ZSTD_seqSymbol e56; + public ZSTD_seqSymbol e57; + public ZSTD_seqSymbol e58; + public ZSTD_seqSymbol e59; + public ZSTD_seqSymbol e60; + public ZSTD_seqSymbol e61; + public ZSTD_seqSymbol e62; + public ZSTD_seqSymbol e63; + public ZSTD_seqSymbol e64; + public ZSTD_seqSymbol e65; + public ZSTD_seqSymbol e66; + public ZSTD_seqSymbol e67; + public ZSTD_seqSymbol e68; + public ZSTD_seqSymbol e69; + public ZSTD_seqSymbol e70; + public ZSTD_seqSymbol e71; + public ZSTD_seqSymbol e72; + public ZSTD_seqSymbol e73; + public ZSTD_seqSymbol e74; + public ZSTD_seqSymbol e75; + public ZSTD_seqSymbol e76; + public ZSTD_seqSymbol e77; + public ZSTD_seqSymbol e78; + public ZSTD_seqSymbol e79; + public ZSTD_seqSymbol e80; + public ZSTD_seqSymbol e81; + public ZSTD_seqSymbol e82; + public ZSTD_seqSymbol e83; + public ZSTD_seqSymbol e84; + public ZSTD_seqSymbol e85; + public ZSTD_seqSymbol e86; + public ZSTD_seqSymbol e87; + public ZSTD_seqSymbol e88; + public ZSTD_seqSymbol e89; + public ZSTD_seqSymbol e90; + public ZSTD_seqSymbol e91; + public ZSTD_seqSymbol e92; + public ZSTD_seqSymbol e93; + public ZSTD_seqSymbol e94; + public ZSTD_seqSymbol e95; + public ZSTD_seqSymbol e96; + public ZSTD_seqSymbol e97; + public ZSTD_seqSymbol e98; + public ZSTD_seqSymbol e99; + public ZSTD_seqSymbol e100; + public ZSTD_seqSymbol e101; + public ZSTD_seqSymbol e102; + public ZSTD_seqSymbol e103; + public ZSTD_seqSymbol e104; + public ZSTD_seqSymbol e105; + public ZSTD_seqSymbol e106; + public ZSTD_seqSymbol e107; + public ZSTD_seqSymbol e108; + public ZSTD_seqSymbol e109; + public ZSTD_seqSymbol e110; + public ZSTD_seqSymbol e111; + public ZSTD_seqSymbol e112; + public ZSTD_seqSymbol e113; + public ZSTD_seqSymbol e114; + public ZSTD_seqSymbol e115; + public ZSTD_seqSymbol e116; + public ZSTD_seqSymbol e117; + public ZSTD_seqSymbol e118; + public ZSTD_seqSymbol e119; + public ZSTD_seqSymbol e120; + public ZSTD_seqSymbol e121; + public ZSTD_seqSymbol e122; + public ZSTD_seqSymbol e123; + public ZSTD_seqSymbol e124; + public ZSTD_seqSymbol e125; + public ZSTD_seqSymbol e126; + public ZSTD_seqSymbol e127; + public ZSTD_seqSymbol e128; + public ZSTD_seqSymbol e129; + public ZSTD_seqSymbol e130; + public ZSTD_seqSymbol e131; + public ZSTD_seqSymbol e132; + public ZSTD_seqSymbol e133; + public ZSTD_seqSymbol e134; + public ZSTD_seqSymbol e135; + public ZSTD_seqSymbol e136; + public ZSTD_seqSymbol e137; + public ZSTD_seqSymbol e138; + public ZSTD_seqSymbol e139; + public ZSTD_seqSymbol e140; + public ZSTD_seqSymbol e141; + public ZSTD_seqSymbol e142; + public ZSTD_seqSymbol e143; + public ZSTD_seqSymbol e144; + public ZSTD_seqSymbol e145; + public ZSTD_seqSymbol e146; + public ZSTD_seqSymbol e147; + public ZSTD_seqSymbol e148; + public ZSTD_seqSymbol e149; + public ZSTD_seqSymbol e150; + public ZSTD_seqSymbol e151; + public ZSTD_seqSymbol e152; + public ZSTD_seqSymbol e153; + public ZSTD_seqSymbol e154; + public ZSTD_seqSymbol e155; + public ZSTD_seqSymbol e156; + public ZSTD_seqSymbol e157; + public ZSTD_seqSymbol e158; + public ZSTD_seqSymbol e159; + public ZSTD_seqSymbol e160; + public ZSTD_seqSymbol e161; + public ZSTD_seqSymbol e162; + public ZSTD_seqSymbol e163; + public ZSTD_seqSymbol e164; + public ZSTD_seqSymbol e165; + public ZSTD_seqSymbol e166; + public ZSTD_seqSymbol e167; + public ZSTD_seqSymbol e168; + public ZSTD_seqSymbol e169; + public ZSTD_seqSymbol e170; + public ZSTD_seqSymbol e171; + public ZSTD_seqSymbol e172; + public ZSTD_seqSymbol e173; + public ZSTD_seqSymbol e174; + public ZSTD_seqSymbol e175; + public ZSTD_seqSymbol e176; + public ZSTD_seqSymbol e177; + public ZSTD_seqSymbol e178; + public ZSTD_seqSymbol e179; + public ZSTD_seqSymbol e180; + public ZSTD_seqSymbol e181; + public ZSTD_seqSymbol e182; + public ZSTD_seqSymbol e183; + public ZSTD_seqSymbol e184; + public ZSTD_seqSymbol e185; + public ZSTD_seqSymbol e186; + public ZSTD_seqSymbol e187; + public ZSTD_seqSymbol e188; + public ZSTD_seqSymbol e189; + public ZSTD_seqSymbol e190; + public ZSTD_seqSymbol e191; + public ZSTD_seqSymbol e192; + public ZSTD_seqSymbol e193; + public ZSTD_seqSymbol e194; + public ZSTD_seqSymbol e195; + public ZSTD_seqSymbol e196; + public ZSTD_seqSymbol e197; + public ZSTD_seqSymbol e198; + public ZSTD_seqSymbol e199; + public ZSTD_seqSymbol e200; + public ZSTD_seqSymbol e201; + public ZSTD_seqSymbol e202; + public ZSTD_seqSymbol e203; + public ZSTD_seqSymbol e204; + public ZSTD_seqSymbol e205; + public ZSTD_seqSymbol e206; + public ZSTD_seqSymbol e207; + public ZSTD_seqSymbol e208; + public ZSTD_seqSymbol e209; + public ZSTD_seqSymbol e210; + public ZSTD_seqSymbol e211; + public ZSTD_seqSymbol e212; + public ZSTD_seqSymbol e213; + public ZSTD_seqSymbol e214; + public ZSTD_seqSymbol e215; + public ZSTD_seqSymbol e216; + public ZSTD_seqSymbol e217; + public ZSTD_seqSymbol e218; + public ZSTD_seqSymbol e219; + public ZSTD_seqSymbol e220; + public ZSTD_seqSymbol e221; + public ZSTD_seqSymbol e222; + public ZSTD_seqSymbol e223; + public ZSTD_seqSymbol e224; + public ZSTD_seqSymbol e225; + public ZSTD_seqSymbol e226; + public ZSTD_seqSymbol e227; + public ZSTD_seqSymbol e228; + public ZSTD_seqSymbol e229; + public ZSTD_seqSymbol e230; + public ZSTD_seqSymbol e231; + public ZSTD_seqSymbol e232; + public ZSTD_seqSymbol e233; + public ZSTD_seqSymbol e234; + public ZSTD_seqSymbol e235; + public ZSTD_seqSymbol e236; + public ZSTD_seqSymbol e237; + public ZSTD_seqSymbol e238; + public ZSTD_seqSymbol e239; + public ZSTD_seqSymbol e240; + public ZSTD_seqSymbol e241; + public ZSTD_seqSymbol e242; + public ZSTD_seqSymbol e243; + public ZSTD_seqSymbol e244; + public ZSTD_seqSymbol e245; + public ZSTD_seqSymbol e246; + public ZSTD_seqSymbol e247; + public ZSTD_seqSymbol e248; + public ZSTD_seqSymbol e249; + public ZSTD_seqSymbol e250; + public ZSTD_seqSymbol e251; + public ZSTD_seqSymbol e252; + public ZSTD_seqSymbol e253; + public ZSTD_seqSymbol e254; + public ZSTD_seqSymbol e255; + public ZSTD_seqSymbol e256; + public ZSTD_seqSymbol e257; + public ZSTD_seqSymbol e258; + public ZSTD_seqSymbol e259; + public ZSTD_seqSymbol e260; + public ZSTD_seqSymbol e261; + public ZSTD_seqSymbol e262; + public ZSTD_seqSymbol e263; + public ZSTD_seqSymbol e264; + public ZSTD_seqSymbol e265; + public ZSTD_seqSymbol e266; + public ZSTD_seqSymbol e267; + public ZSTD_seqSymbol e268; + public ZSTD_seqSymbol e269; + public ZSTD_seqSymbol e270; + public ZSTD_seqSymbol e271; + public ZSTD_seqSymbol e272; + public ZSTD_seqSymbol e273; + public ZSTD_seqSymbol e274; + public ZSTD_seqSymbol e275; + public ZSTD_seqSymbol e276; + public ZSTD_seqSymbol e277; + public ZSTD_seqSymbol e278; + public ZSTD_seqSymbol e279; + public ZSTD_seqSymbol e280; + public ZSTD_seqSymbol e281; + public ZSTD_seqSymbol e282; + public ZSTD_seqSymbol e283; + public ZSTD_seqSymbol e284; + public ZSTD_seqSymbol e285; + public ZSTD_seqSymbol e286; + public ZSTD_seqSymbol e287; + public ZSTD_seqSymbol e288; + public ZSTD_seqSymbol e289; + public ZSTD_seqSymbol e290; + public ZSTD_seqSymbol e291; + public ZSTD_seqSymbol e292; + public ZSTD_seqSymbol e293; + public ZSTD_seqSymbol e294; + public ZSTD_seqSymbol e295; + public ZSTD_seqSymbol e296; + public ZSTD_seqSymbol e297; + public ZSTD_seqSymbol e298; + public ZSTD_seqSymbol e299; + public ZSTD_seqSymbol e300; + public ZSTD_seqSymbol e301; + public ZSTD_seqSymbol e302; + public ZSTD_seqSymbol e303; + public ZSTD_seqSymbol e304; + public ZSTD_seqSymbol e305; + public ZSTD_seqSymbol e306; + public ZSTD_seqSymbol e307; + public ZSTD_seqSymbol e308; + public ZSTD_seqSymbol e309; + public ZSTD_seqSymbol e310; + public ZSTD_seqSymbol e311; + public ZSTD_seqSymbol e312; + public ZSTD_seqSymbol e313; + public ZSTD_seqSymbol e314; + public ZSTD_seqSymbol e315; + public ZSTD_seqSymbol e316; + public ZSTD_seqSymbol e317; + public ZSTD_seqSymbol e318; + public ZSTD_seqSymbol e319; + public ZSTD_seqSymbol e320; + public ZSTD_seqSymbol e321; + public ZSTD_seqSymbol e322; + public ZSTD_seqSymbol e323; + public ZSTD_seqSymbol e324; + public ZSTD_seqSymbol e325; + public ZSTD_seqSymbol e326; + public ZSTD_seqSymbol e327; + public ZSTD_seqSymbol e328; + public ZSTD_seqSymbol e329; + public ZSTD_seqSymbol e330; + public ZSTD_seqSymbol e331; + public ZSTD_seqSymbol e332; + public ZSTD_seqSymbol e333; + public ZSTD_seqSymbol e334; + public ZSTD_seqSymbol e335; + public ZSTD_seqSymbol e336; + public ZSTD_seqSymbol e337; + public ZSTD_seqSymbol e338; + public ZSTD_seqSymbol e339; + public ZSTD_seqSymbol e340; + public ZSTD_seqSymbol e341; + public ZSTD_seqSymbol e342; + public ZSTD_seqSymbol e343; + public ZSTD_seqSymbol e344; + public ZSTD_seqSymbol e345; + public ZSTD_seqSymbol e346; + public ZSTD_seqSymbol e347; + public ZSTD_seqSymbol e348; + public ZSTD_seqSymbol e349; + public ZSTD_seqSymbol e350; + public ZSTD_seqSymbol e351; + public ZSTD_seqSymbol e352; + public ZSTD_seqSymbol e353; + public ZSTD_seqSymbol e354; + public ZSTD_seqSymbol e355; + public ZSTD_seqSymbol e356; + public ZSTD_seqSymbol e357; + public ZSTD_seqSymbol e358; + public ZSTD_seqSymbol e359; + public ZSTD_seqSymbol e360; + public ZSTD_seqSymbol e361; + public ZSTD_seqSymbol e362; + public ZSTD_seqSymbol e363; + public ZSTD_seqSymbol e364; + public ZSTD_seqSymbol e365; + public ZSTD_seqSymbol e366; + public ZSTD_seqSymbol e367; + public ZSTD_seqSymbol e368; + public ZSTD_seqSymbol e369; + public ZSTD_seqSymbol e370; + public ZSTD_seqSymbol e371; + public ZSTD_seqSymbol e372; + public ZSTD_seqSymbol e373; + public ZSTD_seqSymbol e374; + public ZSTD_seqSymbol e375; + public ZSTD_seqSymbol e376; + public ZSTD_seqSymbol e377; + public ZSTD_seqSymbol e378; + public ZSTD_seqSymbol e379; + public ZSTD_seqSymbol e380; + public ZSTD_seqSymbol e381; + public ZSTD_seqSymbol e382; + public ZSTD_seqSymbol e383; + public ZSTD_seqSymbol e384; + public ZSTD_seqSymbol e385; + public ZSTD_seqSymbol e386; + public ZSTD_seqSymbol e387; + public ZSTD_seqSymbol e388; + public ZSTD_seqSymbol e389; + public ZSTD_seqSymbol e390; + public ZSTD_seqSymbol e391; + public ZSTD_seqSymbol e392; + public ZSTD_seqSymbol e393; + public ZSTD_seqSymbol e394; + public ZSTD_seqSymbol e395; + public ZSTD_seqSymbol e396; + public ZSTD_seqSymbol e397; + public ZSTD_seqSymbol e398; + public ZSTD_seqSymbol e399; + public ZSTD_seqSymbol e400; + public ZSTD_seqSymbol e401; + public ZSTD_seqSymbol e402; + public ZSTD_seqSymbol e403; + public ZSTD_seqSymbol e404; + public ZSTD_seqSymbol e405; + public ZSTD_seqSymbol e406; + public ZSTD_seqSymbol e407; + public ZSTD_seqSymbol e408; + public ZSTD_seqSymbol e409; + public ZSTD_seqSymbol e410; + public ZSTD_seqSymbol e411; + public ZSTD_seqSymbol e412; + public ZSTD_seqSymbol e413; + public ZSTD_seqSymbol e414; + public ZSTD_seqSymbol e415; + public ZSTD_seqSymbol e416; + public ZSTD_seqSymbol e417; + public ZSTD_seqSymbol e418; + public ZSTD_seqSymbol e419; + public ZSTD_seqSymbol e420; + public ZSTD_seqSymbol e421; + public ZSTD_seqSymbol e422; + public ZSTD_seqSymbol e423; + public ZSTD_seqSymbol e424; + public ZSTD_seqSymbol e425; + public ZSTD_seqSymbol e426; + public ZSTD_seqSymbol e427; + public ZSTD_seqSymbol e428; + public ZSTD_seqSymbol e429; + public ZSTD_seqSymbol e430; + public ZSTD_seqSymbol e431; + public ZSTD_seqSymbol e432; + public ZSTD_seqSymbol e433; + public ZSTD_seqSymbol e434; + public ZSTD_seqSymbol e435; + public ZSTD_seqSymbol e436; + public ZSTD_seqSymbol e437; + public ZSTD_seqSymbol e438; + public ZSTD_seqSymbol e439; + public ZSTD_seqSymbol e440; + public ZSTD_seqSymbol e441; + public ZSTD_seqSymbol e442; + public ZSTD_seqSymbol e443; + public ZSTD_seqSymbol e444; + public ZSTD_seqSymbol e445; + public ZSTD_seqSymbol e446; + public ZSTD_seqSymbol e447; + public ZSTD_seqSymbol e448; + public ZSTD_seqSymbol e449; + public ZSTD_seqSymbol e450; + public ZSTD_seqSymbol e451; + public ZSTD_seqSymbol e452; + public ZSTD_seqSymbol e453; + public ZSTD_seqSymbol e454; + public ZSTD_seqSymbol e455; + public ZSTD_seqSymbol e456; + public ZSTD_seqSymbol e457; + public ZSTD_seqSymbol e458; + public ZSTD_seqSymbol e459; + public ZSTD_seqSymbol e460; + public ZSTD_seqSymbol e461; + public ZSTD_seqSymbol e462; + public ZSTD_seqSymbol e463; + public ZSTD_seqSymbol e464; + public ZSTD_seqSymbol e465; + public ZSTD_seqSymbol e466; + public ZSTD_seqSymbol e467; + public ZSTD_seqSymbol e468; + public ZSTD_seqSymbol e469; + public ZSTD_seqSymbol e470; + public ZSTD_seqSymbol e471; + public ZSTD_seqSymbol e472; + public ZSTD_seqSymbol e473; + public ZSTD_seqSymbol e474; + public ZSTD_seqSymbol e475; + public ZSTD_seqSymbol e476; + public ZSTD_seqSymbol e477; + public ZSTD_seqSymbol e478; + public ZSTD_seqSymbol e479; + public ZSTD_seqSymbol e480; + public ZSTD_seqSymbol e481; + public ZSTD_seqSymbol e482; + public ZSTD_seqSymbol e483; + public ZSTD_seqSymbol e484; + public ZSTD_seqSymbol e485; + public ZSTD_seqSymbol e486; + public ZSTD_seqSymbol e487; + public ZSTD_seqSymbol e488; + public ZSTD_seqSymbol e489; + public ZSTD_seqSymbol e490; + public ZSTD_seqSymbol e491; + public ZSTD_seqSymbol e492; + public ZSTD_seqSymbol e493; + public ZSTD_seqSymbol e494; + public ZSTD_seqSymbol e495; + public ZSTD_seqSymbol e496; + public ZSTD_seqSymbol e497; + public ZSTD_seqSymbol e498; + public ZSTD_seqSymbol e499; + public ZSTD_seqSymbol e500; + public ZSTD_seqSymbol e501; + public ZSTD_seqSymbol e502; + public ZSTD_seqSymbol e503; + public ZSTD_seqSymbol e504; + public ZSTD_seqSymbol e505; + public ZSTD_seqSymbol e506; + public ZSTD_seqSymbol e507; + public ZSTD_seqSymbol e508; + public ZSTD_seqSymbol e509; + public ZSTD_seqSymbol e510; + public ZSTD_seqSymbol e511; + public ZSTD_seqSymbol e512; + } +#endif +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_forceIgnoreChecksum_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_forceIgnoreChecksum_e.cs new file mode 100644 index 00000000..a8322621 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_forceIgnoreChecksum_e.cs @@ -0,0 +1,8 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum ZSTD_forceIgnoreChecksum_e +{ + /* Note: this enum controls ZSTD_d_forceIgnoreChecksum */ + ZSTD_d_validateChecksum = 0, + ZSTD_d_ignoreChecksum = 1, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_format_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_format_e.cs new file mode 100644 index 00000000..28f4de18 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_format_e.cs @@ -0,0 +1,12 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum ZSTD_format_e +{ + /* zstd frame format, specified in zstd_compression_format.md (default) */ + ZSTD_f_zstd1 = 0, + + /* Variant of zstd frame format, without initial 4-bytes magic number. + * Useful to save 4 bytes per generated frame. + * Decoder cannot recognise automatically this format, requiring this instruction. */ + ZSTD_f_zstd1_magicless = 1, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_frameHeader.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_frameHeader.cs new file mode 100644 index 00000000..8c80f759 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_frameHeader.cs @@ -0,0 +1,21 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct ZSTD_frameHeader +{ + /* if == ZSTD_CONTENTSIZE_UNKNOWN, it means this field is not available. 0 means "empty" */ + public ulong frameContentSize; + + /* can be very large, up to <= frameContentSize */ + public ulong windowSize; + public uint blockSizeMax; + + /* if == ZSTD_skippableFrame, frameContentSize is the size of skippable content */ + public ZSTD_frameType_e frameType; + public uint headerSize; + + /* for ZSTD_skippableFrame, contains the skippable magic variant [0-15] */ + public uint dictID; + public uint checksumFlag; + public uint _reserved1; + public uint _reserved2; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_frameParameters.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_frameParameters.cs new file mode 100644 index 00000000..137ccb37 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_frameParameters.cs @@ -0,0 +1,13 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct ZSTD_frameParameters +{ + /**< 1: content size will be in frame header (when known) */ + public int contentSizeFlag; + + /**< 1: generate a 32-bits checksum using XXH64 algorithm at end of frame, for error detection */ + public int checksumFlag; + + /**< 1: no dictID will be saved into frame header (dictID is only useful for dictionary compression) */ + public int noDictIDFlag; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_frameProgression.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_frameProgression.cs new file mode 100644 index 00000000..adf34b4d --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_frameProgression.cs @@ -0,0 +1,22 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct ZSTD_frameProgression +{ + /* nb input bytes read and buffered */ + public ulong ingested; + + /* nb input bytes actually compressed */ + public ulong consumed; + + /* nb of compressed bytes generated and buffered */ + public ulong produced; + + /* nb of compressed bytes flushed : not provided; can be tracked from caller side */ + public ulong flushed; + + /* MT only : latest started job nb */ + public uint currentJobID; + + /* MT only : nb of workers actively compressing at probe time */ + public uint nbActiveWorkers; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_frameSizeInfo.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_frameSizeInfo.cs new file mode 100644 index 00000000..7f9b6b08 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_frameSizeInfo.cs @@ -0,0 +1,14 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/** + * Contains the compressed frame size and an upper-bound for the decompressed frame size. + * Note: before using `compressedSize`, check for errors using ZSTD_isError(). + * similarly, before using `decompressedBound`, check for errors using: + * `decompressedBound != ZSTD_CONTENTSIZE_ERROR` + */ +public struct ZSTD_frameSizeInfo +{ + public nuint nbBlocks; + public nuint compressedSize; + public ulong decompressedBound; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_frameType_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_frameType_e.cs new file mode 100644 index 00000000..a399748f --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_frameType_e.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum ZSTD_frameType_e +{ + ZSTD_frame, + ZSTD_skippableFrame, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_fseCTablesMetadata_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_fseCTablesMetadata_t.cs new file mode 100644 index 00000000..c6aa0dc8 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_fseCTablesMetadata_t.cs @@ -0,0 +1,18 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/** ZSTD_fseCTablesMetadata_t : + * Stores symbol compression modes for a super-block in {ll, ol, ml}Type, and + * fse tables in fseTablesBuffer. + * fseTablesSize refers to the size of fse tables in bytes. + * This metadata is populated in ZSTD_buildBlockEntropyStats_sequences() */ +public unsafe struct ZSTD_fseCTablesMetadata_t +{ + public SymbolEncodingType_e llType; + public SymbolEncodingType_e ofType; + public SymbolEncodingType_e mlType; + public fixed byte fseTablesBuffer[133]; + public nuint fseTablesSize; + + /* This is to account for bug in 1.3.4. More detail in ZSTD_entropyCompressSeqStore_internal() */ + public nuint lastCountSize; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_fseCTables_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_fseCTables_t.cs new file mode 100644 index 00000000..3a9d621b --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_fseCTables_t.cs @@ -0,0 +1,11 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct ZSTD_fseCTables_t +{ + public fixed uint offcodeCTable[193]; + public fixed uint matchlengthCTable[363]; + public fixed uint litlengthCTable[329]; + public FSE_repeat offcode_repeatMode; + public FSE_repeat matchlength_repeatMode; + public FSE_repeat litlength_repeatMode; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_fseState.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_fseState.cs new file mode 100644 index 00000000..2e7b2a2c --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_fseState.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct ZSTD_fseState +{ + public nuint state; + public ZSTD_seqSymbol* table; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_getAllMatchesFn.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_getAllMatchesFn.cs new file mode 100644 index 00000000..4f7ee54f --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_getAllMatchesFn.cs @@ -0,0 +1,15 @@ +using System.Runtime.InteropServices; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +[UnmanagedFunctionPointer(CallingConvention.Cdecl)] +public unsafe delegate uint ZSTD_getAllMatchesFn( + ZSTD_match_t* param0, + ZSTD_MatchState_t* param1, + uint* param2, + byte* param3, + byte* param4, + uint* rep, + uint ll0, + uint lengthToBeat +); diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_hufCTablesMetadata_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_hufCTablesMetadata_t.cs new file mode 100644 index 00000000..481bd647 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_hufCTablesMetadata_t.cs @@ -0,0 +1,16 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/*********************************************** + * Entropy buffer statistics structs and funcs * + ***********************************************/ +/** ZSTD_hufCTablesMetadata_t : + * Stores Literals Block Type for a super-block in hType, and + * huffman tree description in hufDesBuffer. + * hufDesSize refers to the size of huffman tree description in bytes. + * This metadata is populated in ZSTD_buildBlockEntropyStats_literals() */ +public unsafe struct ZSTD_hufCTablesMetadata_t +{ + public SymbolEncodingType_e hType; + public fixed byte hufDesBuffer[128]; + public nuint hufDesSize; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_hufCTables_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_hufCTables_t.cs new file mode 100644 index 00000000..bedd8c8b --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_hufCTables_t.cs @@ -0,0 +1,279 @@ +using System.Runtime.CompilerServices; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct ZSTD_hufCTables_t +{ + public _CTable_e__FixedBuffer CTable; + public HUF_repeat repeatMode; + +#if NET8_0_OR_GREATER + [InlineArray(257)] + public unsafe struct _CTable_e__FixedBuffer + { + public nuint e0; + } + +#else + public unsafe struct _CTable_e__FixedBuffer + { + public nuint e0; + public nuint e1; + public nuint e2; + public nuint e3; + public nuint e4; + public nuint e5; + public nuint e6; + public nuint e7; + public nuint e8; + public nuint e9; + public nuint e10; + public nuint e11; + public nuint e12; + public nuint e13; + public nuint e14; + public nuint e15; + public nuint e16; + public nuint e17; + public nuint e18; + public nuint e19; + public nuint e20; + public nuint e21; + public nuint e22; + public nuint e23; + public nuint e24; + public nuint e25; + public nuint e26; + public nuint e27; + public nuint e28; + public nuint e29; + public nuint e30; + public nuint e31; + public nuint e32; + public nuint e33; + public nuint e34; + public nuint e35; + public nuint e36; + public nuint e37; + public nuint e38; + public nuint e39; + public nuint e40; + public nuint e41; + public nuint e42; + public nuint e43; + public nuint e44; + public nuint e45; + public nuint e46; + public nuint e47; + public nuint e48; + public nuint e49; + public nuint e50; + public nuint e51; + public nuint e52; + public nuint e53; + public nuint e54; + public nuint e55; + public nuint e56; + public nuint e57; + public nuint e58; + public nuint e59; + public nuint e60; + public nuint e61; + public nuint e62; + public nuint e63; + public nuint e64; + public nuint e65; + public nuint e66; + public nuint e67; + public nuint e68; + public nuint e69; + public nuint e70; + public nuint e71; + public nuint e72; + public nuint e73; + public nuint e74; + public nuint e75; + public nuint e76; + public nuint e77; + public nuint e78; + public nuint e79; + public nuint e80; + public nuint e81; + public nuint e82; + public nuint e83; + public nuint e84; + public nuint e85; + public nuint e86; + public nuint e87; + public nuint e88; + public nuint e89; + public nuint e90; + public nuint e91; + public nuint e92; + public nuint e93; + public nuint e94; + public nuint e95; + public nuint e96; + public nuint e97; + public nuint e98; + public nuint e99; + public nuint e100; + public nuint e101; + public nuint e102; + public nuint e103; + public nuint e104; + public nuint e105; + public nuint e106; + public nuint e107; + public nuint e108; + public nuint e109; + public nuint e110; + public nuint e111; + public nuint e112; + public nuint e113; + public nuint e114; + public nuint e115; + public nuint e116; + public nuint e117; + public nuint e118; + public nuint e119; + public nuint e120; + public nuint e121; + public nuint e122; + public nuint e123; + public nuint e124; + public nuint e125; + public nuint e126; + public nuint e127; + public nuint e128; + public nuint e129; + public nuint e130; + public nuint e131; + public nuint e132; + public nuint e133; + public nuint e134; + public nuint e135; + public nuint e136; + public nuint e137; + public nuint e138; + public nuint e139; + public nuint e140; + public nuint e141; + public nuint e142; + public nuint e143; + public nuint e144; + public nuint e145; + public nuint e146; + public nuint e147; + public nuint e148; + public nuint e149; + public nuint e150; + public nuint e151; + public nuint e152; + public nuint e153; + public nuint e154; + public nuint e155; + public nuint e156; + public nuint e157; + public nuint e158; + public nuint e159; + public nuint e160; + public nuint e161; + public nuint e162; + public nuint e163; + public nuint e164; + public nuint e165; + public nuint e166; + public nuint e167; + public nuint e168; + public nuint e169; + public nuint e170; + public nuint e171; + public nuint e172; + public nuint e173; + public nuint e174; + public nuint e175; + public nuint e176; + public nuint e177; + public nuint e178; + public nuint e179; + public nuint e180; + public nuint e181; + public nuint e182; + public nuint e183; + public nuint e184; + public nuint e185; + public nuint e186; + public nuint e187; + public nuint e188; + public nuint e189; + public nuint e190; + public nuint e191; + public nuint e192; + public nuint e193; + public nuint e194; + public nuint e195; + public nuint e196; + public nuint e197; + public nuint e198; + public nuint e199; + public nuint e200; + public nuint e201; + public nuint e202; + public nuint e203; + public nuint e204; + public nuint e205; + public nuint e206; + public nuint e207; + public nuint e208; + public nuint e209; + public nuint e210; + public nuint e211; + public nuint e212; + public nuint e213; + public nuint e214; + public nuint e215; + public nuint e216; + public nuint e217; + public nuint e218; + public nuint e219; + public nuint e220; + public nuint e221; + public nuint e222; + public nuint e223; + public nuint e224; + public nuint e225; + public nuint e226; + public nuint e227; + public nuint e228; + public nuint e229; + public nuint e230; + public nuint e231; + public nuint e232; + public nuint e233; + public nuint e234; + public nuint e235; + public nuint e236; + public nuint e237; + public nuint e238; + public nuint e239; + public nuint e240; + public nuint e241; + public nuint e242; + public nuint e243; + public nuint e244; + public nuint e245; + public nuint e246; + public nuint e247; + public nuint e248; + public nuint e249; + public nuint e250; + public nuint e251; + public nuint e252; + public nuint e253; + public nuint e254; + public nuint e255; + public nuint e256; + } +#endif +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_inBuffer_s.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_inBuffer_s.cs new file mode 100644 index 00000000..58242a2c --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_inBuffer_s.cs @@ -0,0 +1,16 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/**************************** + * Streaming + ****************************/ +public unsafe struct ZSTD_inBuffer_s +{ + /**< start of input buffer */ + public void* src; + + /**< size of input buffer */ + public nuint size; + + /**< position where reading stopped. Will be updated. Necessarily 0 <= pos <= size */ + public nuint pos; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_indexResetPolicy_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_indexResetPolicy_e.cs new file mode 100644 index 00000000..b3638aa0 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_indexResetPolicy_e.cs @@ -0,0 +1,12 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/** + * Controls, for this matchState reset, whether indexing can continue where it + * left off (ZSTDirp_continue), or whether it needs to be restarted from zero + * (ZSTDirp_reset). + */ +public enum ZSTD_indexResetPolicy_e +{ + ZSTDirp_continue, + ZSTDirp_reset, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_litLocation_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_litLocation_e.cs new file mode 100644 index 00000000..55d28562 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_litLocation_e.cs @@ -0,0 +1,13 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum ZSTD_litLocation_e +{ + /* Stored entirely within litExtraBuffer */ + ZSTD_not_in_dst = 0, + + /* Stored entirely within dst (in memory after current output write) */ + ZSTD_in_dst = 1, + + /* Split between litExtraBuffer and dst */ + ZSTD_split = 2, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_literalCompressionMode_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_literalCompressionMode_e.cs new file mode 100644 index 00000000..2c309b06 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_literalCompressionMode_e.cs @@ -0,0 +1,16 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum ZSTD_literalCompressionMode_e +{ + /**< Automatically determine the compression mode based on the compression level. + * Negative compression levels will be uncompressed, and positive compression + * levels will be compressed. */ + ZSTD_lcm_auto = 0, + + /**< Always attempt Huffman compression. Uncompressed literals will still be + * emitted if Huffman compression is not profitable. */ + ZSTD_lcm_huffman = 1, + + /**< Always emit uncompressed literals. */ + ZSTD_lcm_uncompressed = 2, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_localDict.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_localDict.cs new file mode 100644 index 00000000..4b970451 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_localDict.cs @@ -0,0 +1,10 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct ZSTD_localDict +{ + public void* dictBuffer; + public void* dict; + public nuint dictSize; + public ZSTD_dictContentType_e dictContentType; + public ZSTD_CDict_s* cdict; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_longLengthType_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_longLengthType_e.cs new file mode 100644 index 00000000..141643f2 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_longLengthType_e.cs @@ -0,0 +1,14 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/* Controls whether seqStore has a single "long" litLength or matchLength. See SeqStore_t. */ +public enum ZSTD_longLengthType_e +{ + /* no longLengthType */ + ZSTD_llt_none = 0, + + /* represents a long literal */ + ZSTD_llt_literalLength = 1, + + /* represents a long match */ + ZSTD_llt_matchLength = 2, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_longOffset_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_longOffset_e.cs new file mode 100644 index 00000000..86881bdd --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_longOffset_e.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum ZSTD_longOffset_e +{ + ZSTD_lo_isRegularOffset, + ZSTD_lo_isLongOffset = 1, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_match_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_match_t.cs new file mode 100644 index 00000000..fa44b9ec --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_match_t.cs @@ -0,0 +1,13 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/********************************* + * Compression internals structs * + *********************************/ +public struct ZSTD_match_t +{ + /* Offset sumtype code for the match, using ZSTD_storeSeq() format */ + public uint off; + + /* Raw length of match */ + public uint len; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_nextInputType_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_nextInputType_e.cs new file mode 100644 index 00000000..5ebea51e --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_nextInputType_e.cs @@ -0,0 +1,11 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum ZSTD_nextInputType_e +{ + ZSTDnit_frameHeader, + ZSTDnit_blockHeader, + ZSTDnit_block, + ZSTDnit_lastBlock, + ZSTDnit_checksum, + ZSTDnit_skippableFrame, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_optLdm_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_optLdm_t.cs new file mode 100644 index 00000000..8319db5b --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_optLdm_t.cs @@ -0,0 +1,17 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/* Struct containing info needed to make decision about ldm inclusion */ +public struct ZSTD_optLdm_t +{ + /* External match candidates store for this block */ + public RawSeqStore_t seqStore; + + /* Start position of the current match candidate */ + public uint startPosInBlock; + + /* End position of the current match candidate */ + public uint endPosInBlock; + + /* Offset of the match candidate */ + public uint offset; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_optimal_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_optimal_t.cs new file mode 100644 index 00000000..84845187 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_optimal_t.cs @@ -0,0 +1,19 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct ZSTD_optimal_t +{ + /* price from beginning of segment to this position */ + public int price; + + /* offset of previous match */ + public uint off; + + /* length of previous match */ + public uint mlen; + + /* nb of literals since previous match */ + public uint litlen; + + /* offset history after previous match */ + public fixed uint rep[3]; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_outBuffer_s.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_outBuffer_s.cs new file mode 100644 index 00000000..c2cc0089 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_outBuffer_s.cs @@ -0,0 +1,13 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct ZSTD_outBuffer_s +{ + /**< start of output buffer */ + public void* dst; + + /**< size of output buffer */ + public nuint size; + + /**< position where writing stopped. Will be updated. Necessarily 0 <= pos <= size */ + public nuint pos; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_overlap_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_overlap_e.cs new file mode 100644 index 00000000..bf5b85e1 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_overlap_e.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum ZSTD_overlap_e +{ + ZSTD_no_overlap, + ZSTD_overlap_src_before_dst, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_paramSwitch_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_paramSwitch_e.cs new file mode 100644 index 00000000..3ff21492 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_paramSwitch_e.cs @@ -0,0 +1,13 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum ZSTD_paramSwitch_e +{ + /* Let the library automatically determine whether the feature shall be enabled */ + ZSTD_ps_auto = 0, + + /* Force-enable the feature */ + ZSTD_ps_enable = 1, + + /* Do not use the feature */ + ZSTD_ps_disable = 2, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_parameters.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_parameters.cs new file mode 100644 index 00000000..ebdf1969 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_parameters.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct ZSTD_parameters +{ + public ZSTD_compressionParameters cParams; + public ZSTD_frameParameters fParams; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_prefixDict_s.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_prefixDict_s.cs new file mode 100644 index 00000000..2a0db472 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_prefixDict_s.cs @@ -0,0 +1,8 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct ZSTD_prefixDict_s +{ + public void* dict; + public nuint dictSize; + public ZSTD_dictContentType_e dictContentType; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_refMultipleDDicts_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_refMultipleDDicts_e.cs new file mode 100644 index 00000000..c01fcefc --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_refMultipleDDicts_e.cs @@ -0,0 +1,8 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum ZSTD_refMultipleDDicts_e +{ + /* Note: this enum controls ZSTD_d_refMultipleDDicts */ + ZSTD_rmd_refSingleDDict = 0, + ZSTD_rmd_refMultipleDDicts = 1, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_resetTarget_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_resetTarget_e.cs new file mode 100644 index 00000000..679e8b28 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_resetTarget_e.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum ZSTD_resetTarget_e +{ + ZSTD_resetTarget_CDict, + ZSTD_resetTarget_CCtx, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_seqSymbol.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_seqSymbol.cs new file mode 100644 index 00000000..622ff4fc --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_seqSymbol.cs @@ -0,0 +1,17 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct ZSTD_seqSymbol +{ + public ushort nextState; + public byte nbAdditionalBits; + public byte nbBits; + public uint baseValue; + + public ZSTD_seqSymbol(ushort nextState, byte nbAdditionalBits, byte nbBits, uint baseValue) + { + this.nextState = nextState; + this.nbAdditionalBits = nbAdditionalBits; + this.nbBits = nbBits; + this.baseValue = baseValue; + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_seqSymbol_header.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_seqSymbol_header.cs new file mode 100644 index 00000000..eb87911c --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_seqSymbol_header.cs @@ -0,0 +1,10 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/*-******************************************************* + * Decompression types + *********************************************************/ +public struct ZSTD_seqSymbol_header +{ + public uint fastMode; + public uint tableLog; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_sequenceFormat_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_sequenceFormat_e.cs new file mode 100644 index 00000000..3b6ca525 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_sequenceFormat_e.cs @@ -0,0 +1,10 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum ZSTD_sequenceFormat_e +{ + /* ZSTD_Sequence[] has no block delimiters, just sequences */ + ZSTD_sf_noBlockDelimiters = 0, + + /* ZSTD_Sequence[] contains explicit block delimiters */ + ZSTD_sf_explicitBlockDelimiters = 1, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_strategy.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_strategy.cs new file mode 100644 index 00000000..49d8e23e --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_strategy.cs @@ -0,0 +1,15 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/* Compression strategies, listed from fastest to strongest */ +public enum ZSTD_strategy +{ + ZSTD_fast = 1, + ZSTD_dfast = 2, + ZSTD_greedy = 3, + ZSTD_lazy = 4, + ZSTD_lazy2 = 5, + ZSTD_btlazy2 = 6, + ZSTD_btopt = 7, + ZSTD_btultra = 8, + ZSTD_btultra2 = 9, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_symbolEncodingTypeStats_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_symbolEncodingTypeStats_t.cs new file mode 100644 index 00000000..673f2b3d --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_symbolEncodingTypeStats_t.cs @@ -0,0 +1,16 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/* Type returned by ZSTD_buildSequencesStatistics containing finalized symbol encoding types + * and size of the sequences statistics + */ +public struct ZSTD_symbolEncodingTypeStats_t +{ + public uint LLtype; + public uint Offtype; + public uint MLtype; + public nuint size; + + /* Accounts for bug in 1.3.4. More detail in ZSTD_entropyCompressSeqStore_internal() */ + public nuint lastCountSize; + public int longOffsets; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_tableFillPurpose_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_tableFillPurpose_e.cs new file mode 100644 index 00000000..f218de33 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_tableFillPurpose_e.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum ZSTD_tableFillPurpose_e +{ + ZSTD_tfp_forCCtx, + ZSTD_tfp_forCDict, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_window_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_window_t.cs new file mode 100644 index 00000000..84895dab --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZSTD_window_t.cs @@ -0,0 +1,25 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct ZSTD_window_t +{ + /* next block here to continue on current prefix */ + public byte* nextSrc; + + /* All regular indexes relative to this position */ + public byte* @base; + + /* extDict indexes relative to this position */ + public byte* dictBase; + + /* below that point, need extDict */ + public uint dictLimit; + + /* below that point, no more valid data */ + public uint lowLimit; + + /* Number of times overflow correction has run since + * ZSTD_window_init(). Useful for debugging coredumps + * and for ZSTD_WINDOW_OVERFLOW_CORRECT_FREQUENTLY. + */ + public uint nbOverflowCorrections; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/Zdict.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/Zdict.cs new file mode 100644 index 00000000..80e29e1e --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/Zdict.cs @@ -0,0 +1,710 @@ +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + /*-******************************************************** + * Helper functions + **********************************************************/ + public static bool ZDICT_isError(nuint errorCode) + { + return ERR_isError(errorCode); + } + + public static string ZDICT_getErrorName(nuint errorCode) + { + return ERR_getErrorName(errorCode); + } + + private static void ZDICT_countEStats( + EStats_ress_t esr, + ZSTD_parameters* @params, + uint* countLit, + uint* offsetcodeCount, + uint* matchlengthCount, + uint* litlengthCount, + uint* repOffsets, + void* src, + nuint srcSize, + uint notificationLevel + ) + { + nuint blockSizeMax = (nuint)( + 1 << 17 < 1 << (int)@params->cParams.windowLog + ? 1 << 17 + : 1 << (int)@params->cParams.windowLog + ); + nuint cSize; + if (srcSize > blockSizeMax) + { + srcSize = blockSizeMax; + } + + { + nuint errorCode = ZSTD_compressBegin_usingCDict_deprecated(esr.zc, esr.dict); + if (ERR_isError(errorCode)) + { + return; + } + } + + cSize = ZSTD_compressBlock_deprecated(esr.zc, esr.workPlace, 1 << 17, src, srcSize); + if (ERR_isError(cSize)) + { + return; + } + + if (cSize != 0) + { + SeqStore_t* seqStorePtr = ZSTD_getSeqStore(esr.zc); + { + byte* bytePtr; + for (bytePtr = seqStorePtr->litStart; bytePtr < seqStorePtr->lit; bytePtr++) + { + countLit[*bytePtr]++; + } + } + + { + uint nbSeq = (uint)(seqStorePtr->sequences - seqStorePtr->sequencesStart); + ZSTD_seqToCodes(seqStorePtr); + { + byte* codePtr = seqStorePtr->ofCode; + uint u; + for (u = 0; u < nbSeq; u++) + { + offsetcodeCount[codePtr[u]]++; + } + } + + { + byte* codePtr = seqStorePtr->mlCode; + uint u; + for (u = 0; u < nbSeq; u++) + { + matchlengthCount[codePtr[u]]++; + } + } + + { + byte* codePtr = seqStorePtr->llCode; + uint u; + for (u = 0; u < nbSeq; u++) + { + litlengthCount[codePtr[u]]++; + } + } + + if (nbSeq >= 2) + { + SeqDef_s* seq = seqStorePtr->sequencesStart; + uint offset1 = seq[0].offBase - 3; + uint offset2 = seq[1].offBase - 3; + if (offset1 >= 1024) + { + offset1 = 0; + } + + if (offset2 >= 1024) + { + offset2 = 0; + } + + repOffsets[offset1] += 3; + repOffsets[offset2] += 1; + } + } + } + } + + private static nuint ZDICT_totalSampleSize(nuint* fileSizes, uint nbFiles) + { + nuint total = 0; + uint u; + for (u = 0; u < nbFiles; u++) + { + total += fileSizes[u]; + } + + return total; + } + + private static void ZDICT_insertSortCount(offsetCount_t* table, uint val, uint count) + { + uint u; + table[3].offset = val; + table[3].count = count; + for (u = 3; u > 0; u--) + { + offsetCount_t tmp; + if (table[u - 1].count >= table[u].count) + { + break; + } + + tmp = table[u - 1]; + table[u - 1] = table[u]; + table[u] = tmp; + } + } + + /* ZDICT_flatLit() : + * rewrite `countLit` to contain a mostly flat but still compressible distribution of literals. + * necessary to avoid generating a non-compressible distribution that HUF_writeCTable() cannot encode. + */ + private static void ZDICT_flatLit(uint* countLit) + { + int u; + for (u = 1; u < 256; u++) + { + countLit[u] = 2; + } + + countLit[0] = 4; + countLit[253] = 1; + countLit[254] = 1; + } + + private static nuint ZDICT_analyzeEntropy( + void* dstBuffer, + nuint maxDstSize, + int compressionLevel, + void* srcBuffer, + nuint* fileSizes, + uint nbFiles, + void* dictBuffer, + nuint dictBufferSize, + uint notificationLevel + ) + { + uint* countLit = stackalloc uint[256]; + /* no final ; */ + nuint* hufTable = stackalloc nuint[257]; + uint* offcodeCount = stackalloc uint[31]; + short* offcodeNCount = stackalloc short[31]; + uint offcodeMax = ZSTD_highbit32((uint)(dictBufferSize + 128 * (1 << 10))); + uint* matchLengthCount = stackalloc uint[53]; + short* matchLengthNCount = stackalloc short[53]; + uint* litLengthCount = stackalloc uint[36]; + short* litLengthNCount = stackalloc short[36]; + uint* repOffset = stackalloc uint[1024]; + offsetCount_t* bestRepOffset = stackalloc offsetCount_t[4]; + EStats_ress_t esr = new EStats_ress_t + { + dict = null, + zc = null, + workPlace = null, + }; + ZSTD_parameters @params; + uint u, + huffLog = 11, + Offlog = 8, + mlLog = 9, + llLog = 9, + total; + nuint pos = 0, + errorCode; + nuint eSize = 0; + nuint totalSrcSize = ZDICT_totalSampleSize(fileSizes, nbFiles); + nuint averageSampleSize = totalSrcSize / (nbFiles + (uint)(nbFiles == 0 ? 1 : 0)); + byte* dstPtr = (byte*)dstBuffer; + uint* wksp = stackalloc uint[1216]; + if (offcodeMax > 30) + { + eSize = unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionaryCreation_failed)); + goto _cleanup; + } + + for (u = 0; u < 256; u++) + { + countLit[u] = 1; + } + + for (u = 0; u <= offcodeMax; u++) + { + offcodeCount[u] = 1; + } + + for (u = 0; u <= 52; u++) + { + matchLengthCount[u] = 1; + } + + for (u = 0; u <= 35; u++) + { + litLengthCount[u] = 1; + } + + memset(repOffset, 0, sizeof(uint) * 1024); + repOffset[1] = repOffset[4] = repOffset[8] = 1; + memset(bestRepOffset, 0, (uint)(sizeof(offsetCount_t) * 4)); + if (compressionLevel == 0) + { + compressionLevel = 3; + } + + @params = ZSTD_getParams(compressionLevel, averageSampleSize, dictBufferSize); + esr.dict = ZSTD_createCDict_advanced( + dictBuffer, + dictBufferSize, + ZSTD_dictLoadMethod_e.ZSTD_dlm_byRef, + ZSTD_dictContentType_e.ZSTD_dct_rawContent, + @params.cParams, + ZSTD_defaultCMem + ); + esr.zc = ZSTD_createCCtx(); + esr.workPlace = malloc(1 << 17); + if (esr.dict == null || esr.zc == null || esr.workPlace == null) + { + eSize = unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + goto _cleanup; + } + + for (u = 0; u < nbFiles; u++) + { + ZDICT_countEStats( + esr, + &@params, + countLit, + offcodeCount, + matchLengthCount, + litLengthCount, + repOffset, + (sbyte*)srcBuffer + pos, + fileSizes[u], + notificationLevel + ); + pos += fileSizes[u]; + } + + if (notificationLevel >= 4) + { + for (u = 0; u <= offcodeMax; u++) { } + } + + { + nuint maxNbBits = HUF_buildCTable_wksp( + hufTable, + countLit, + 255, + huffLog, + wksp, + sizeof(uint) * 1216 + ); + if (ERR_isError(maxNbBits)) + { + eSize = maxNbBits; + goto _cleanup; + } + + if (maxNbBits == 8) + { + ZDICT_flatLit(countLit); + maxNbBits = HUF_buildCTable_wksp( + hufTable, + countLit, + 255, + huffLog, + wksp, + sizeof(uint) * 1216 + ); + assert(maxNbBits == 9); + } + + huffLog = (uint)maxNbBits; + } + + { + uint offset; + for (offset = 1; offset < 1024; offset++) + { + ZDICT_insertSortCount(bestRepOffset, offset, repOffset[offset]); + } + } + + total = 0; + for (u = 0; u <= offcodeMax; u++) + { + total += offcodeCount[u]; + } + + errorCode = FSE_normalizeCount(offcodeNCount, Offlog, offcodeCount, total, offcodeMax, 1); + if (ERR_isError(errorCode)) + { + eSize = errorCode; + goto _cleanup; + } + + Offlog = (uint)errorCode; + total = 0; + for (u = 0; u <= 52; u++) + { + total += matchLengthCount[u]; + } + + errorCode = FSE_normalizeCount(matchLengthNCount, mlLog, matchLengthCount, total, 52, 1); + if (ERR_isError(errorCode)) + { + eSize = errorCode; + goto _cleanup; + } + + mlLog = (uint)errorCode; + total = 0; + for (u = 0; u <= 35; u++) + { + total += litLengthCount[u]; + } + + errorCode = FSE_normalizeCount(litLengthNCount, llLog, litLengthCount, total, 35, 1); + if (ERR_isError(errorCode)) + { + eSize = errorCode; + goto _cleanup; + } + + llLog = (uint)errorCode; + { + nuint hhSize = HUF_writeCTable_wksp( + dstPtr, + maxDstSize, + hufTable, + 255, + huffLog, + wksp, + sizeof(uint) * 1216 + ); + if (ERR_isError(hhSize)) + { + eSize = hhSize; + goto _cleanup; + } + + dstPtr += hhSize; + maxDstSize -= hhSize; + eSize += hhSize; + } + + { + nuint ohSize = FSE_writeNCount(dstPtr, maxDstSize, offcodeNCount, 30, Offlog); + if (ERR_isError(ohSize)) + { + eSize = ohSize; + goto _cleanup; + } + + dstPtr += ohSize; + maxDstSize -= ohSize; + eSize += ohSize; + } + + { + nuint mhSize = FSE_writeNCount(dstPtr, maxDstSize, matchLengthNCount, 52, mlLog); + if (ERR_isError(mhSize)) + { + eSize = mhSize; + goto _cleanup; + } + + dstPtr += mhSize; + maxDstSize -= mhSize; + eSize += mhSize; + } + + { + nuint lhSize = FSE_writeNCount(dstPtr, maxDstSize, litLengthNCount, 35, llLog); + if (ERR_isError(lhSize)) + { + eSize = lhSize; + goto _cleanup; + } + + dstPtr += lhSize; + maxDstSize -= lhSize; + eSize += lhSize; + } + + if (maxDstSize < 12) + { + eSize = unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + goto _cleanup; + } + + MEM_writeLE32(dstPtr + 0, repStartValue[0]); + MEM_writeLE32(dstPtr + 4, repStartValue[1]); + MEM_writeLE32(dstPtr + 8, repStartValue[2]); + eSize += 12; + _cleanup: + ZSTD_freeCDict(esr.dict); + ZSTD_freeCCtx(esr.zc); + free(esr.workPlace); + return eSize; + } + + /** + * @returns the maximum repcode value + */ + private static uint ZDICT_maxRep(uint* reps) + { + uint maxRep = reps[0]; + int r; + for (r = 1; r < 3; ++r) + { + maxRep = maxRep > reps[r] ? maxRep : reps[r]; + } + + return maxRep; + } + + /*! ZDICT_finalizeDictionary(): + * Given a custom content as a basis for dictionary, and a set of samples, + * finalize dictionary by adding headers and statistics according to the zstd + * dictionary format. + * + * Samples must be stored concatenated in a flat buffer `samplesBuffer`, + * supplied with an array of sizes `samplesSizes`, providing the size of each + * sample in order. The samples are used to construct the statistics, so they + * should be representative of what you will compress with this dictionary. + * + * The compression level can be set in `parameters`. You should pass the + * compression level you expect to use in production. The statistics for each + * compression level differ, so tuning the dictionary for the compression level + * can help quite a bit. + * + * You can set an explicit dictionary ID in `parameters`, or allow us to pick + * a random dictionary ID for you, but we can't guarantee no collisions. + * + * The dstDictBuffer and the dictContent may overlap, and the content will be + * appended to the end of the header. If the header + the content doesn't fit in + * maxDictSize the beginning of the content is truncated to make room, since it + * is presumed that the most profitable content is at the end of the dictionary, + * since that is the cheapest to reference. + * + * `maxDictSize` must be >= max(dictContentSize, ZDICT_DICTSIZE_MIN). + * + * @return: size of dictionary stored into `dstDictBuffer` (<= `maxDictSize`), + * or an error code, which can be tested by ZDICT_isError(). + * Note: ZDICT_finalizeDictionary() will push notifications into stderr if + * instructed to, using notificationLevel>0. + * NOTE: This function currently may fail in several edge cases including: + * * Not enough samples + * * Samples are uncompressible + * * Samples are all exactly the same + */ + public static nuint ZDICT_finalizeDictionary( + void* dictBuffer, + nuint dictBufferCapacity, + void* customDictContent, + nuint dictContentSize, + void* samplesBuffer, + nuint* samplesSizes, + uint nbSamples, + ZDICT_params_t @params + ) + { + nuint hSize; + byte* header = stackalloc byte[256]; + int compressionLevel = @params.compressionLevel == 0 ? 3 : @params.compressionLevel; + uint notificationLevel = @params.notificationLevel; + /* The final dictionary content must be at least as large as the largest repcode */ + nuint minContentSize = ZDICT_maxRep(repStartValue); + nuint paddingSize; + if (dictBufferCapacity < dictContentSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + if (dictBufferCapacity < 256) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + MEM_writeLE32(header, 0xEC30A437); + { + ulong randomID = ZSTD_XXH64(customDictContent, dictContentSize, 0); + uint compliantID = (uint)(randomID % ((1U << 31) - 32768) + 32768); + uint dictID = @params.dictID != 0 ? @params.dictID : compliantID; + MEM_writeLE32(header + 4, dictID); + } + + hSize = 8; + { + nuint eSize = ZDICT_analyzeEntropy( + header + hSize, + 256 - hSize, + compressionLevel, + samplesBuffer, + samplesSizes, + nbSamples, + customDictContent, + dictContentSize, + notificationLevel + ); + if (ZDICT_isError(eSize)) + { + return eSize; + } + + hSize += eSize; + } + + if (hSize + dictContentSize > dictBufferCapacity) + { + dictContentSize = dictBufferCapacity - hSize; + } + + if (dictContentSize < minContentSize) + { + if (hSize + minContentSize > dictBufferCapacity) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + paddingSize = minContentSize - dictContentSize; + } + else + { + paddingSize = 0; + } + + { + nuint dictSize = hSize + paddingSize + dictContentSize; + /* The dictionary consists of the header, optional padding, and the content. + * The padding comes before the content because the "best" position in the + * dictionary is the last byte. + */ + byte* outDictHeader = (byte*)dictBuffer; + byte* outDictPadding = outDictHeader + hSize; + byte* outDictContent = outDictPadding + paddingSize; + assert(dictSize <= dictBufferCapacity); + assert(outDictContent + dictContentSize == (byte*)dictBuffer + dictSize); + memmove(outDictContent, customDictContent, dictContentSize); + memcpy(outDictHeader, header, (uint)hSize); + memset(outDictPadding, 0, (uint)paddingSize); + return dictSize; + } + } + + private static nuint ZDICT_addEntropyTablesFromBuffer_advanced( + void* dictBuffer, + nuint dictContentSize, + nuint dictBufferCapacity, + void* samplesBuffer, + nuint* samplesSizes, + uint nbSamples, + ZDICT_params_t @params + ) + { + int compressionLevel = @params.compressionLevel == 0 ? 3 : @params.compressionLevel; + uint notificationLevel = @params.notificationLevel; + nuint hSize = 8; + { + nuint eSize = ZDICT_analyzeEntropy( + (sbyte*)dictBuffer + hSize, + dictBufferCapacity - hSize, + compressionLevel, + samplesBuffer, + samplesSizes, + nbSamples, + (sbyte*)dictBuffer + dictBufferCapacity - dictContentSize, + dictContentSize, + notificationLevel + ); + if (ZDICT_isError(eSize)) + { + return eSize; + } + + hSize += eSize; + } + + MEM_writeLE32(dictBuffer, 0xEC30A437); + { + ulong randomID = ZSTD_XXH64( + (sbyte*)dictBuffer + dictBufferCapacity - dictContentSize, + dictContentSize, + 0 + ); + uint compliantID = (uint)(randomID % ((1U << 31) - 32768) + 32768); + uint dictID = @params.dictID != 0 ? @params.dictID : compliantID; + MEM_writeLE32((sbyte*)dictBuffer + 4, dictID); + } + + if (hSize + dictContentSize < dictBufferCapacity) + { + memmove( + (sbyte*)dictBuffer + hSize, + (sbyte*)dictBuffer + dictBufferCapacity - dictContentSize, + dictContentSize + ); + } + + return dictBufferCapacity < hSize + dictContentSize + ? dictBufferCapacity + : hSize + dictContentSize; + } + + /*! ZDICT_trainFromBuffer(): + * Train a dictionary from an array of samples. + * Redirect towards ZDICT_optimizeTrainFromBuffer_fastCover() single-threaded, with d=8, steps=4, + * f=20, and accel=1. + * Samples must be stored concatenated in a single flat buffer `samplesBuffer`, + * supplied with an array of sizes `samplesSizes`, providing the size of each sample, in order. + * The resulting dictionary will be saved into `dictBuffer`. + * @return: size of dictionary stored into `dictBuffer` (<= `dictBufferCapacity`) + * or an error code, which can be tested with ZDICT_isError(). + * Note: Dictionary training will fail if there are not enough samples to construct a + * dictionary, or if most of the samples are too small (< 8 bytes being the lower limit). + * If dictionary training fails, you should use zstd without a dictionary, as the dictionary + * would've been ineffective anyways. If you believe your samples would benefit from a dictionary + * please open an issue with details, and we can look into it. + * Note: ZDICT_trainFromBuffer()'s memory usage is about 6 MB. + * Tips: In general, a reasonable dictionary has a size of ~ 100 KB. + * It's possible to select smaller or larger size, just by specifying `dictBufferCapacity`. + * In general, it's recommended to provide a few thousands samples, though this can vary a lot. + * It's recommended that total size of all samples be about ~x100 times the target size of dictionary. + */ + public static nuint ZDICT_trainFromBuffer( + void* dictBuffer, + nuint dictBufferCapacity, + void* samplesBuffer, + nuint* samplesSizes, + uint nbSamples + ) + { + ZDICT_fastCover_params_t @params; + @params = new ZDICT_fastCover_params_t { d = 8, steps = 4 }; + @params.zParams.compressionLevel = 3; + return ZDICT_optimizeTrainFromBuffer_fastCover( + dictBuffer, + dictBufferCapacity, + samplesBuffer, + samplesSizes, + nbSamples, + &@params + ); + } + + public static nuint ZDICT_addEntropyTablesFromBuffer( + void* dictBuffer, + nuint dictContentSize, + nuint dictBufferCapacity, + void* samplesBuffer, + nuint* samplesSizes, + uint nbSamples + ) + { + ZDICT_params_t @params; + @params = new ZDICT_params_t(); + return ZDICT_addEntropyTablesFromBuffer_advanced( + dictBuffer, + dictContentSize, + dictBufferCapacity, + samplesBuffer, + samplesSizes, + nbSamples, + @params + ); + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/Zstd.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/Zstd.cs new file mode 100644 index 00000000..ce1018b0 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/Zstd.cs @@ -0,0 +1,10 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + private static readonly ZSTD_customMem ZSTD_defaultCMem = new ZSTD_customMem( + customAlloc: null, + customFree: null, + opaque: null + ); +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdCommon.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdCommon.cs new file mode 100644 index 00000000..629a435c --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdCommon.cs @@ -0,0 +1,48 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + /*-**************************************** + * Version + ******************************************/ + public static uint ZSTD_versionNumber() + { + return 1 * 100 * 100 + 5 * 100 + 7; + } + + /*! ZSTD_versionString() : + * Return runtime library version, like "1.4.5". Requires v1.3.0+. */ + public static string ZSTD_versionString() + { + return "1.5.7"; + } + + /*! ZSTD_isError() : + * tells if a return value is an error code + * symbol is required for external callers */ + public static bool ZSTD_isError(nuint code) + { + return ERR_isError(code); + } + + /*! ZSTD_getErrorName() : + * provides error code string from function result (useful for debugging) */ + public static string ZSTD_getErrorName(nuint code) + { + return ERR_getErrorName(code); + } + + /*! ZSTD_getError() : + * convert a `size_t` function result into a proper ZSTD_errorCode enum */ + public static ZSTD_ErrorCode ZSTD_getErrorCode(nuint code) + { + return ERR_getErrorCode(code); + } + + /*! ZSTD_getErrorString() : + * provides error code string from enum */ + public static string ZSTD_getErrorString(ZSTD_ErrorCode code) + { + return ERR_getErrorString(code); + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdCompress.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdCompress.cs new file mode 100644 index 00000000..0242d47b --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdCompress.cs @@ -0,0 +1,11583 @@ +using System; +using System.Numerics; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + /*-************************************* + * Helper functions + ***************************************/ + /* ZSTD_compressBound() + * Note that the result from this function is only valid for + * the one-pass compression functions. + * When employing the streaming mode, + * if flushes are frequently altering the size of blocks, + * the overhead from block headers can make the compressed data larger + * than the return value of ZSTD_compressBound(). + */ + public static nuint ZSTD_compressBound(nuint srcSize) + { + nuint r = + srcSize >= (sizeof(nuint) == 8 ? 0xFF00FF00FF00FF00UL : 0xFF00FF00U) + ? 0 + : srcSize + + (srcSize >> 8) + + (srcSize < 128 << 10 ? (128 << 10) - srcSize >> 11 : 0); + if (r == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + return r; + } + + public static ZSTD_CCtx_s* ZSTD_createCCtx() + { + return ZSTD_createCCtx_advanced(ZSTD_defaultCMem); + } + + private static void ZSTD_initCCtx(ZSTD_CCtx_s* cctx, ZSTD_customMem memManager) + { + assert(cctx != null); + *cctx = new ZSTD_CCtx_s { customMem = memManager, bmi2 = 0 }; + { + nuint err = ZSTD_CCtx_reset(cctx, ZSTD_ResetDirective.ZSTD_reset_parameters); + assert(!ERR_isError(err)); + } + } + + public static ZSTD_CCtx_s* ZSTD_createCCtx_advanced(ZSTD_customMem customMem) + { + if (((customMem.customAlloc == null ? 1 : 0) ^ (customMem.customFree == null ? 1 : 0)) != 0) + { + return null; + } + + { + ZSTD_CCtx_s* cctx = (ZSTD_CCtx_s*)ZSTD_customMalloc( + (nuint)sizeof(ZSTD_CCtx_s), + customMem + ); + if (cctx == null) + { + return null; + } + + ZSTD_initCCtx(cctx, customMem); + return cctx; + } + } + + /*! ZSTD_initStatic*() : + * Initialize an object using a pre-allocated fixed-size buffer. + * workspace: The memory area to emplace the object into. + * Provided pointer *must be 8-bytes aligned*. + * Buffer must outlive object. + * workspaceSize: Use ZSTD_estimate*Size() to determine + * how large workspace must be to support target scenario. + * @return : pointer to object (same address as workspace, just different type), + * or NULL if error (size too small, incorrect alignment, etc.) + * Note : zstd will never resize nor malloc() when using a static buffer. + * If the object requires more memory than available, + * zstd will just error out (typically ZSTD_error_memory_allocation). + * Note 2 : there is no corresponding "free" function. + * Since workspace is allocated externally, it must be freed externally too. + * Note 3 : cParams : use ZSTD_getCParams() to convert a compression level + * into its associated cParams. + * Limitation 1 : currently not compatible with internal dictionary creation, triggered by + * ZSTD_CCtx_loadDictionary(), ZSTD_initCStream_usingDict() or ZSTD_initDStream_usingDict(). + * Limitation 2 : static cctx currently not compatible with multi-threading. + * Limitation 3 : static dctx is incompatible with legacy support. + */ + public static ZSTD_CCtx_s* ZSTD_initStaticCCtx(void* workspace, nuint workspaceSize) + { + ZSTD_cwksp ws; + ZSTD_CCtx_s* cctx; + if (workspaceSize <= (nuint)sizeof(ZSTD_CCtx_s)) + { + return null; + } + + if (((nuint)workspace & 7) != 0) + { + return null; + } + + ZSTD_cwksp_init( + &ws, + workspace, + workspaceSize, + ZSTD_cwksp_static_alloc_e.ZSTD_cwksp_static_alloc + ); + cctx = (ZSTD_CCtx_s*)ZSTD_cwksp_reserve_object(&ws, (nuint)sizeof(ZSTD_CCtx_s)); + if (cctx == null) + { + return null; + } + + *cctx = new ZSTD_CCtx_s(); + ZSTD_cwksp_move(&cctx->workspace, &ws); + cctx->staticSize = workspaceSize; + if ( + ZSTD_cwksp_check_available( + &cctx->workspace, + (nuint)( + ( + (8 << 10) + 512 + sizeof(uint) * (52 + 2) > 8208 + ? (8 << 10) + 512 + sizeof(uint) * (52 + 2) + : 8208 + ) + + 2 * sizeof(ZSTD_compressedBlockState_t) + ) + ) == 0 + ) + { + return null; + } + + cctx->blockState.prevCBlock = (ZSTD_compressedBlockState_t*)ZSTD_cwksp_reserve_object( + &cctx->workspace, + (nuint)sizeof(ZSTD_compressedBlockState_t) + ); + cctx->blockState.nextCBlock = (ZSTD_compressedBlockState_t*)ZSTD_cwksp_reserve_object( + &cctx->workspace, + (nuint)sizeof(ZSTD_compressedBlockState_t) + ); + cctx->tmpWorkspace = ZSTD_cwksp_reserve_object( + &cctx->workspace, + (8 << 10) + 512 + sizeof(uint) * (52 + 2) > 8208 + ? (8 << 10) + 512 + sizeof(uint) * (52 + 2) + : 8208 + ); + cctx->tmpWkspSize = + (8 << 10) + 512 + sizeof(uint) * (52 + 2) > 8208 + ? (8 << 10) + 512 + sizeof(uint) * (52 + 2) + : 8208; + cctx->bmi2 = 0; + return cctx; + } + + /** + * Clears and frees all of the dictionaries in the CCtx. + */ + private static void ZSTD_clearAllDicts(ZSTD_CCtx_s* cctx) + { + ZSTD_customFree(cctx->localDict.dictBuffer, cctx->customMem); + ZSTD_freeCDict(cctx->localDict.cdict); + cctx->localDict = new ZSTD_localDict(); + cctx->prefixDict = new ZSTD_prefixDict_s(); + cctx->cdict = null; + } + + private static nuint ZSTD_sizeof_localDict(ZSTD_localDict dict) + { + nuint bufferSize = dict.dictBuffer != null ? dict.dictSize : 0; + nuint cdictSize = ZSTD_sizeof_CDict(dict.cdict); + return bufferSize + cdictSize; + } + + private static void ZSTD_freeCCtxContent(ZSTD_CCtx_s* cctx) + { + assert(cctx != null); + assert(cctx->staticSize == 0); + ZSTD_clearAllDicts(cctx); + ZSTDMT_freeCCtx(cctx->mtctx); + cctx->mtctx = null; + ZSTD_cwksp_free(&cctx->workspace, cctx->customMem); + } + + public static nuint ZSTD_freeCCtx(ZSTD_CCtx_s* cctx) + { + if (cctx == null) + { + return 0; + } + + if (cctx->staticSize != 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + + { + int cctxInWorkspace = ZSTD_cwksp_owns_buffer(&cctx->workspace, cctx); + ZSTD_freeCCtxContent(cctx); + if (cctxInWorkspace == 0) + { + ZSTD_customFree(cctx, cctx->customMem); + } + } + + return 0; + } + + private static nuint ZSTD_sizeof_mtctx(ZSTD_CCtx_s* cctx) + { + return ZSTDMT_sizeof_CCtx(cctx->mtctx); + } + + /*! ZSTD_sizeof_*() : Requires v1.4.0+ + * These functions give the _current_ memory usage of selected object. + * Note that object memory usage can evolve (increase or decrease) over time. */ + public static nuint ZSTD_sizeof_CCtx(ZSTD_CCtx_s* cctx) + { + if (cctx == null) + { + return 0; + } + + return (nuint)(cctx->workspace.workspace == cctx ? 0 : sizeof(ZSTD_CCtx_s)) + + ZSTD_cwksp_sizeof(&cctx->workspace) + + ZSTD_sizeof_localDict(cctx->localDict) + + ZSTD_sizeof_mtctx(cctx); + } + + public static nuint ZSTD_sizeof_CStream(ZSTD_CCtx_s* zcs) + { + return ZSTD_sizeof_CCtx(zcs); + } + + /* private API call, for dictBuilder only */ + private static SeqStore_t* ZSTD_getSeqStore(ZSTD_CCtx_s* ctx) + { + return &ctx->seqStore; + } + + /* Returns true if the strategy supports using a row based matchfinder */ + private static int ZSTD_rowMatchFinderSupported(ZSTD_strategy strategy) + { + return strategy >= ZSTD_strategy.ZSTD_greedy && strategy <= ZSTD_strategy.ZSTD_lazy2 + ? 1 + : 0; + } + + /* Returns true if the strategy and useRowMatchFinder mode indicate that we will use the row based matchfinder + * for this compression. + */ + private static int ZSTD_rowMatchFinderUsed(ZSTD_strategy strategy, ZSTD_paramSwitch_e mode) + { + assert(mode != ZSTD_paramSwitch_e.ZSTD_ps_auto); + return + ZSTD_rowMatchFinderSupported(strategy) != 0 && mode == ZSTD_paramSwitch_e.ZSTD_ps_enable + ? 1 + : 0; + } + + /* Returns row matchfinder usage given an initial mode and cParams */ + private static ZSTD_paramSwitch_e ZSTD_resolveRowMatchFinderMode( + ZSTD_paramSwitch_e mode, + ZSTD_compressionParameters* cParams + ) + { + if (mode != ZSTD_paramSwitch_e.ZSTD_ps_auto) + { + return mode; + } + + mode = ZSTD_paramSwitch_e.ZSTD_ps_disable; + if (ZSTD_rowMatchFinderSupported(cParams->strategy) == 0) + { + return mode; + } + + if (cParams->windowLog > 14) + { + mode = ZSTD_paramSwitch_e.ZSTD_ps_enable; + } + + return mode; + } + + /* Returns block splitter usage (generally speaking, when using slower/stronger compression modes) */ + private static ZSTD_paramSwitch_e ZSTD_resolveBlockSplitterMode( + ZSTD_paramSwitch_e mode, + ZSTD_compressionParameters* cParams + ) + { + if (mode != ZSTD_paramSwitch_e.ZSTD_ps_auto) + { + return mode; + } + + return cParams->strategy >= ZSTD_strategy.ZSTD_btopt && cParams->windowLog >= 17 + ? ZSTD_paramSwitch_e.ZSTD_ps_enable + : ZSTD_paramSwitch_e.ZSTD_ps_disable; + } + + /* Returns 1 if the arguments indicate that we should allocate a chainTable, 0 otherwise */ + private static int ZSTD_allocateChainTable( + ZSTD_strategy strategy, + ZSTD_paramSwitch_e useRowMatchFinder, + uint forDDSDict + ) + { + assert(useRowMatchFinder != ZSTD_paramSwitch_e.ZSTD_ps_auto); + return + forDDSDict != 0 + || strategy != ZSTD_strategy.ZSTD_fast + && ZSTD_rowMatchFinderUsed(strategy, useRowMatchFinder) == 0 + ? 1 + : 0; + } + + /* Returns ZSTD_ps_enable if compression parameters are such that we should + * enable long distance matching (wlog >= 27, strategy >= btopt). + * Returns ZSTD_ps_disable otherwise. + */ + private static ZSTD_paramSwitch_e ZSTD_resolveEnableLdm( + ZSTD_paramSwitch_e mode, + ZSTD_compressionParameters* cParams + ) + { + if (mode != ZSTD_paramSwitch_e.ZSTD_ps_auto) + { + return mode; + } + + return cParams->strategy >= ZSTD_strategy.ZSTD_btopt && cParams->windowLog >= 27 + ? ZSTD_paramSwitch_e.ZSTD_ps_enable + : ZSTD_paramSwitch_e.ZSTD_ps_disable; + } + + private static int ZSTD_resolveExternalSequenceValidation(int mode) + { + return mode; + } + + /* Resolves maxBlockSize to the default if no value is present. */ + private static nuint ZSTD_resolveMaxBlockSize(nuint maxBlockSize) + { + if (maxBlockSize == 0) + { + return 1 << 17; + } + else + { + return maxBlockSize; + } + } + + private static ZSTD_paramSwitch_e ZSTD_resolveExternalRepcodeSearch( + ZSTD_paramSwitch_e value, + int cLevel + ) + { + if (value != ZSTD_paramSwitch_e.ZSTD_ps_auto) + { + return value; + } + + if (cLevel < 10) + { + return ZSTD_paramSwitch_e.ZSTD_ps_disable; + } + else + { + return ZSTD_paramSwitch_e.ZSTD_ps_enable; + } + } + + /* Returns 1 if compression parameters are such that CDict hashtable and chaintable indices are tagged. + * If so, the tags need to be removed in ZSTD_resetCCtx_byCopyingCDict. */ + private static int ZSTD_CDictIndicesAreTagged(ZSTD_compressionParameters* cParams) + { + return + cParams->strategy == ZSTD_strategy.ZSTD_fast + || cParams->strategy == ZSTD_strategy.ZSTD_dfast + ? 1 + : 0; + } + + private static ZSTD_CCtx_params_s ZSTD_makeCCtxParamsFromCParams( + ZSTD_compressionParameters cParams + ) + { + ZSTD_CCtx_params_s cctxParams; + ZSTD_CCtxParams_init(&cctxParams, 3); + cctxParams.cParams = cParams; + cctxParams.ldmParams.enableLdm = ZSTD_resolveEnableLdm( + cctxParams.ldmParams.enableLdm, + &cParams + ); + if (cctxParams.ldmParams.enableLdm == ZSTD_paramSwitch_e.ZSTD_ps_enable) + { + ZSTD_ldm_adjustParameters(&cctxParams.ldmParams, &cParams); + assert(cctxParams.ldmParams.hashLog >= cctxParams.ldmParams.bucketSizeLog); + assert(cctxParams.ldmParams.hashRateLog < 32); + } + + cctxParams.postBlockSplitter = ZSTD_resolveBlockSplitterMode( + cctxParams.postBlockSplitter, + &cParams + ); + cctxParams.useRowMatchFinder = ZSTD_resolveRowMatchFinderMode( + cctxParams.useRowMatchFinder, + &cParams + ); + cctxParams.validateSequences = ZSTD_resolveExternalSequenceValidation( + cctxParams.validateSequences + ); + cctxParams.maxBlockSize = ZSTD_resolveMaxBlockSize(cctxParams.maxBlockSize); + cctxParams.searchForExternalRepcodes = ZSTD_resolveExternalRepcodeSearch( + cctxParams.searchForExternalRepcodes, + cctxParams.compressionLevel + ); + assert(ZSTD_checkCParams(cParams) == 0); + return cctxParams; + } + + private static ZSTD_CCtx_params_s* ZSTD_createCCtxParams_advanced(ZSTD_customMem customMem) + { + ZSTD_CCtx_params_s* @params; + if (((customMem.customAlloc == null ? 1 : 0) ^ (customMem.customFree == null ? 1 : 0)) != 0) + { + return null; + } + + @params = (ZSTD_CCtx_params_s*)ZSTD_customCalloc( + (nuint)sizeof(ZSTD_CCtx_params_s), + customMem + ); + if (@params == null) + { + return null; + } + + ZSTD_CCtxParams_init(@params, 3); + @params->customMem = customMem; + return @params; + } + + /*! ZSTD_CCtx_params : + * Quick howto : + * - ZSTD_createCCtxParams() : Create a ZSTD_CCtx_params structure + * - ZSTD_CCtxParams_setParameter() : Push parameters one by one into + * an existing ZSTD_CCtx_params structure. + * This is similar to + * ZSTD_CCtx_setParameter(). + * - ZSTD_CCtx_setParametersUsingCCtxParams() : Apply parameters to + * an existing CCtx. + * These parameters will be applied to + * all subsequent frames. + * - ZSTD_compressStream2() : Do compression using the CCtx. + * - ZSTD_freeCCtxParams() : Free the memory, accept NULL pointer. + * + * This can be used with ZSTD_estimateCCtxSize_advanced_usingCCtxParams() + * for static allocation of CCtx for single-threaded compression. + */ + public static ZSTD_CCtx_params_s* ZSTD_createCCtxParams() + { + return ZSTD_createCCtxParams_advanced(ZSTD_defaultCMem); + } + + public static nuint ZSTD_freeCCtxParams(ZSTD_CCtx_params_s* @params) + { + if (@params == null) + { + return 0; + } + + ZSTD_customFree(@params, @params->customMem); + return 0; + } + + /*! ZSTD_CCtxParams_reset() : + * Reset params to default values. + */ + public static nuint ZSTD_CCtxParams_reset(ZSTD_CCtx_params_s* @params) + { + return ZSTD_CCtxParams_init(@params, 3); + } + + /*! ZSTD_CCtxParams_init() : + * Initializes the compression parameters of cctxParams according to + * compression level. All other parameters are reset to their default values. + */ + public static nuint ZSTD_CCtxParams_init(ZSTD_CCtx_params_s* cctxParams, int compressionLevel) + { + if (cctxParams == null) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + + *cctxParams = new ZSTD_CCtx_params_s { compressionLevel = compressionLevel }; + cctxParams->fParams.contentSizeFlag = 1; + return 0; + } + + /** + * Initializes `cctxParams` from `params` and `compressionLevel`. + * @param compressionLevel If params are derived from a compression level then that compression level, otherwise ZSTD_NO_CLEVEL. + */ + private static void ZSTD_CCtxParams_init_internal( + ZSTD_CCtx_params_s* cctxParams, + ZSTD_parameters* @params, + int compressionLevel + ) + { + assert(ZSTD_checkCParams(@params->cParams) == 0); + *cctxParams = new ZSTD_CCtx_params_s + { + cParams = @params->cParams, + fParams = @params->fParams, + compressionLevel = compressionLevel, + useRowMatchFinder = ZSTD_resolveRowMatchFinderMode( + cctxParams->useRowMatchFinder, + &@params->cParams + ), + postBlockSplitter = ZSTD_resolveBlockSplitterMode( + cctxParams->postBlockSplitter, + &@params->cParams + ), + }; + cctxParams->ldmParams.enableLdm = ZSTD_resolveEnableLdm( + cctxParams->ldmParams.enableLdm, + &@params->cParams + ); + cctxParams->validateSequences = ZSTD_resolveExternalSequenceValidation( + cctxParams->validateSequences + ); + cctxParams->maxBlockSize = ZSTD_resolveMaxBlockSize(cctxParams->maxBlockSize); + cctxParams->searchForExternalRepcodes = ZSTD_resolveExternalRepcodeSearch( + cctxParams->searchForExternalRepcodes, + compressionLevel + ); + } + + /*! ZSTD_CCtxParams_init_advanced() : + * Initializes the compression and frame parameters of cctxParams according to + * params. All other parameters are reset to their default values. + */ + public static nuint ZSTD_CCtxParams_init_advanced( + ZSTD_CCtx_params_s* cctxParams, + ZSTD_parameters @params + ) + { + if (cctxParams == null) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + + { + nuint err_code = ZSTD_checkCParams(@params.cParams); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + ZSTD_CCtxParams_init_internal(cctxParams, &@params, 0); + return 0; + } + + /** + * Sets cctxParams' cParams and fParams from params, but otherwise leaves them alone. + * @param params Validated zstd parameters. + */ + private static void ZSTD_CCtxParams_setZstdParams( + ZSTD_CCtx_params_s* cctxParams, + ZSTD_parameters* @params + ) + { + assert(ZSTD_checkCParams(@params->cParams) == 0); + cctxParams->cParams = @params->cParams; + cctxParams->fParams = @params->fParams; + cctxParams->compressionLevel = 0; + } + + /*! ZSTD_cParam_getBounds() : + * All parameters must belong to an interval with lower and upper bounds, + * otherwise they will either trigger an error or be automatically clamped. + * @return : a structure, ZSTD_bounds, which contains + * - an error status field, which must be tested using ZSTD_isError() + * - lower and upper bounds, both inclusive + */ + public static ZSTD_bounds ZSTD_cParam_getBounds(ZSTD_cParameter param) + { + ZSTD_bounds bounds = new ZSTD_bounds + { + error = 0, + lowerBound = 0, + upperBound = 0, + }; + switch (param) + { + case ZSTD_cParameter.ZSTD_c_compressionLevel: + bounds.lowerBound = ZSTD_minCLevel(); + bounds.upperBound = ZSTD_maxCLevel(); + return bounds; + case ZSTD_cParameter.ZSTD_c_windowLog: + bounds.lowerBound = 10; + bounds.upperBound = sizeof(nuint) == 4 ? 30 : 31; + return bounds; + case ZSTD_cParameter.ZSTD_c_hashLog: + bounds.lowerBound = 6; + bounds.upperBound = + (sizeof(nuint) == 4 ? 30 : 31) < 30 + ? sizeof(nuint) == 4 + ? 30 + : 31 + : 30; + return bounds; + case ZSTD_cParameter.ZSTD_c_chainLog: + bounds.lowerBound = 6; + bounds.upperBound = sizeof(nuint) == 4 ? 29 : 30; + return bounds; + case ZSTD_cParameter.ZSTD_c_searchLog: + bounds.lowerBound = 1; + bounds.upperBound = (sizeof(nuint) == 4 ? 30 : 31) - 1; + return bounds; + case ZSTD_cParameter.ZSTD_c_minMatch: + bounds.lowerBound = 3; + bounds.upperBound = 7; + return bounds; + case ZSTD_cParameter.ZSTD_c_targetLength: + bounds.lowerBound = 0; + bounds.upperBound = 1 << 17; + return bounds; + case ZSTD_cParameter.ZSTD_c_strategy: + bounds.lowerBound = (int)ZSTD_strategy.ZSTD_fast; + bounds.upperBound = (int)ZSTD_strategy.ZSTD_btultra2; + return bounds; + case ZSTD_cParameter.ZSTD_c_contentSizeFlag: + bounds.lowerBound = 0; + bounds.upperBound = 1; + return bounds; + case ZSTD_cParameter.ZSTD_c_checksumFlag: + bounds.lowerBound = 0; + bounds.upperBound = 1; + return bounds; + case ZSTD_cParameter.ZSTD_c_dictIDFlag: + bounds.lowerBound = 0; + bounds.upperBound = 1; + return bounds; + case ZSTD_cParameter.ZSTD_c_nbWorkers: + bounds.lowerBound = 0; + bounds.upperBound = sizeof(void*) == 4 ? 64 : 256; + return bounds; + case ZSTD_cParameter.ZSTD_c_jobSize: + bounds.lowerBound = 0; + bounds.upperBound = MEM_32bits ? 512 * (1 << 20) : 1024 * (1 << 20); + return bounds; + case ZSTD_cParameter.ZSTD_c_overlapLog: + bounds.lowerBound = 0; + bounds.upperBound = 9; + return bounds; + case ZSTD_cParameter.ZSTD_c_experimentalParam8: + bounds.lowerBound = 0; + bounds.upperBound = 1; + return bounds; + case ZSTD_cParameter.ZSTD_c_enableLongDistanceMatching: + bounds.lowerBound = (int)ZSTD_paramSwitch_e.ZSTD_ps_auto; + bounds.upperBound = (int)ZSTD_paramSwitch_e.ZSTD_ps_disable; + return bounds; + case ZSTD_cParameter.ZSTD_c_ldmHashLog: + bounds.lowerBound = 6; + bounds.upperBound = + (sizeof(nuint) == 4 ? 30 : 31) < 30 + ? sizeof(nuint) == 4 + ? 30 + : 31 + : 30; + return bounds; + case ZSTD_cParameter.ZSTD_c_ldmMinMatch: + bounds.lowerBound = 4; + bounds.upperBound = 4096; + return bounds; + case ZSTD_cParameter.ZSTD_c_ldmBucketSizeLog: + bounds.lowerBound = 1; + bounds.upperBound = 8; + return bounds; + case ZSTD_cParameter.ZSTD_c_ldmHashRateLog: + bounds.lowerBound = 0; + bounds.upperBound = (sizeof(nuint) == 4 ? 30 : 31) - 6; + return bounds; + case ZSTD_cParameter.ZSTD_c_experimentalParam1: + bounds.lowerBound = 0; + bounds.upperBound = 1; + return bounds; + case ZSTD_cParameter.ZSTD_c_experimentalParam3: + bounds.lowerBound = 0; + bounds.upperBound = 1; + return bounds; + case ZSTD_cParameter.ZSTD_c_experimentalParam2: + bounds.lowerBound = (int)ZSTD_format_e.ZSTD_f_zstd1; + bounds.upperBound = (int)ZSTD_format_e.ZSTD_f_zstd1_magicless; + return bounds; + case ZSTD_cParameter.ZSTD_c_experimentalParam4: + bounds.lowerBound = (int)ZSTD_dictAttachPref_e.ZSTD_dictDefaultAttach; + bounds.upperBound = (int)ZSTD_dictAttachPref_e.ZSTD_dictForceLoad; + return bounds; + case ZSTD_cParameter.ZSTD_c_experimentalParam5: + bounds.lowerBound = (int)ZSTD_paramSwitch_e.ZSTD_ps_auto; + bounds.upperBound = (int)ZSTD_paramSwitch_e.ZSTD_ps_disable; + return bounds; + case ZSTD_cParameter.ZSTD_c_targetCBlockSize: + bounds.lowerBound = 1340; + bounds.upperBound = 1 << 17; + return bounds; + case ZSTD_cParameter.ZSTD_c_experimentalParam7: + bounds.lowerBound = 0; + bounds.upperBound = 2147483647; + return bounds; + case ZSTD_cParameter.ZSTD_c_experimentalParam9: + case ZSTD_cParameter.ZSTD_c_experimentalParam10: + bounds.lowerBound = (int)ZSTD_bufferMode_e.ZSTD_bm_buffered; + bounds.upperBound = (int)ZSTD_bufferMode_e.ZSTD_bm_stable; + return bounds; + case ZSTD_cParameter.ZSTD_c_experimentalParam11: + bounds.lowerBound = (int)ZSTD_sequenceFormat_e.ZSTD_sf_noBlockDelimiters; + bounds.upperBound = (int)ZSTD_sequenceFormat_e.ZSTD_sf_explicitBlockDelimiters; + return bounds; + case ZSTD_cParameter.ZSTD_c_experimentalParam12: + bounds.lowerBound = 0; + bounds.upperBound = 1; + return bounds; + case ZSTD_cParameter.ZSTD_c_experimentalParam13: + bounds.lowerBound = (int)ZSTD_paramSwitch_e.ZSTD_ps_auto; + bounds.upperBound = (int)ZSTD_paramSwitch_e.ZSTD_ps_disable; + return bounds; + case ZSTD_cParameter.ZSTD_c_experimentalParam20: + bounds.lowerBound = 0; + bounds.upperBound = 6; + return bounds; + case ZSTD_cParameter.ZSTD_c_experimentalParam14: + bounds.lowerBound = (int)ZSTD_paramSwitch_e.ZSTD_ps_auto; + bounds.upperBound = (int)ZSTD_paramSwitch_e.ZSTD_ps_disable; + return bounds; + case ZSTD_cParameter.ZSTD_c_experimentalParam15: + bounds.lowerBound = 0; + bounds.upperBound = 1; + return bounds; + case ZSTD_cParameter.ZSTD_c_experimentalParam16: + bounds.lowerBound = (int)ZSTD_paramSwitch_e.ZSTD_ps_auto; + bounds.upperBound = (int)ZSTD_paramSwitch_e.ZSTD_ps_disable; + return bounds; + case ZSTD_cParameter.ZSTD_c_experimentalParam17: + bounds.lowerBound = 0; + bounds.upperBound = 1; + return bounds; + case ZSTD_cParameter.ZSTD_c_experimentalParam18: + bounds.lowerBound = 1 << 10; + bounds.upperBound = 1 << 17; + return bounds; + case ZSTD_cParameter.ZSTD_c_experimentalParam19: + bounds.lowerBound = (int)ZSTD_paramSwitch_e.ZSTD_ps_auto; + bounds.upperBound = (int)ZSTD_paramSwitch_e.ZSTD_ps_disable; + return bounds; + default: + bounds.error = unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_unsupported) + ); + return bounds; + } + } + + /* ZSTD_cParam_clampBounds: + * Clamps the value into the bounded range. + */ + private static nuint ZSTD_cParam_clampBounds(ZSTD_cParameter cParam, int* value) + { + ZSTD_bounds bounds = ZSTD_cParam_getBounds(cParam); + if (ERR_isError(bounds.error)) + { + return bounds.error; + } + + if (*value < bounds.lowerBound) + { + *value = bounds.lowerBound; + } + + if (*value > bounds.upperBound) + { + *value = bounds.upperBound; + } + + return 0; + } + + private static int ZSTD_isUpdateAuthorized(ZSTD_cParameter param) + { + switch (param) + { + case ZSTD_cParameter.ZSTD_c_compressionLevel: + case ZSTD_cParameter.ZSTD_c_hashLog: + case ZSTD_cParameter.ZSTD_c_chainLog: + case ZSTD_cParameter.ZSTD_c_searchLog: + case ZSTD_cParameter.ZSTD_c_minMatch: + case ZSTD_cParameter.ZSTD_c_targetLength: + case ZSTD_cParameter.ZSTD_c_strategy: + case ZSTD_cParameter.ZSTD_c_experimentalParam20: + return 1; + case ZSTD_cParameter.ZSTD_c_experimentalParam2: + case ZSTD_cParameter.ZSTD_c_windowLog: + case ZSTD_cParameter.ZSTD_c_contentSizeFlag: + case ZSTD_cParameter.ZSTD_c_checksumFlag: + case ZSTD_cParameter.ZSTD_c_dictIDFlag: + case ZSTD_cParameter.ZSTD_c_experimentalParam3: + case ZSTD_cParameter.ZSTD_c_nbWorkers: + case ZSTD_cParameter.ZSTD_c_jobSize: + case ZSTD_cParameter.ZSTD_c_overlapLog: + case ZSTD_cParameter.ZSTD_c_experimentalParam1: + case ZSTD_cParameter.ZSTD_c_experimentalParam8: + case ZSTD_cParameter.ZSTD_c_enableLongDistanceMatching: + case ZSTD_cParameter.ZSTD_c_ldmHashLog: + case ZSTD_cParameter.ZSTD_c_ldmMinMatch: + case ZSTD_cParameter.ZSTD_c_ldmBucketSizeLog: + case ZSTD_cParameter.ZSTD_c_ldmHashRateLog: + case ZSTD_cParameter.ZSTD_c_experimentalParam4: + case ZSTD_cParameter.ZSTD_c_experimentalParam5: + case ZSTD_cParameter.ZSTD_c_targetCBlockSize: + case ZSTD_cParameter.ZSTD_c_experimentalParam7: + case ZSTD_cParameter.ZSTD_c_experimentalParam9: + case ZSTD_cParameter.ZSTD_c_experimentalParam10: + case ZSTD_cParameter.ZSTD_c_experimentalParam11: + case ZSTD_cParameter.ZSTD_c_experimentalParam12: + case ZSTD_cParameter.ZSTD_c_experimentalParam13: + case ZSTD_cParameter.ZSTD_c_experimentalParam14: + case ZSTD_cParameter.ZSTD_c_experimentalParam15: + case ZSTD_cParameter.ZSTD_c_experimentalParam16: + case ZSTD_cParameter.ZSTD_c_experimentalParam17: + case ZSTD_cParameter.ZSTD_c_experimentalParam18: + case ZSTD_cParameter.ZSTD_c_experimentalParam19: + default: + return 0; + } + } + + /*! ZSTD_CCtx_setParameter() : + * Set one compression parameter, selected by enum ZSTD_cParameter. + * All parameters have valid bounds. Bounds can be queried using ZSTD_cParam_getBounds(). + * Providing a value beyond bound will either clamp it, or trigger an error (depending on parameter). + * Setting a parameter is generally only possible during frame initialization (before starting compression). + * Exception : when using multi-threading mode (nbWorkers >= 1), + * the following parameters can be updated _during_ compression (within same frame): + * => compressionLevel, hashLog, chainLog, searchLog, minMatch, targetLength and strategy. + * new parameters will be active for next job only (after a flush()). + * @return : an error code (which can be tested using ZSTD_isError()). + */ + public static nuint ZSTD_CCtx_setParameter(ZSTD_CCtx_s* cctx, ZSTD_cParameter param, int value) + { + if (cctx->streamStage != ZSTD_cStreamStage.zcss_init) + { + if (ZSTD_isUpdateAuthorized(param) != 0) + { + cctx->cParamsChanged = 1; + } + else + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_stage_wrong)); + } + } + + switch (param) + { + case ZSTD_cParameter.ZSTD_c_nbWorkers: + if (value != 0 && cctx->staticSize != 0) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_unsupported) + ); + } + + break; + case ZSTD_cParameter.ZSTD_c_compressionLevel: + case ZSTD_cParameter.ZSTD_c_windowLog: + case ZSTD_cParameter.ZSTD_c_hashLog: + case ZSTD_cParameter.ZSTD_c_chainLog: + case ZSTD_cParameter.ZSTD_c_searchLog: + case ZSTD_cParameter.ZSTD_c_minMatch: + case ZSTD_cParameter.ZSTD_c_targetLength: + case ZSTD_cParameter.ZSTD_c_strategy: + case ZSTD_cParameter.ZSTD_c_ldmHashRateLog: + case ZSTD_cParameter.ZSTD_c_experimentalParam2: + case ZSTD_cParameter.ZSTD_c_contentSizeFlag: + case ZSTD_cParameter.ZSTD_c_checksumFlag: + case ZSTD_cParameter.ZSTD_c_dictIDFlag: + case ZSTD_cParameter.ZSTD_c_experimentalParam3: + case ZSTD_cParameter.ZSTD_c_experimentalParam4: + case ZSTD_cParameter.ZSTD_c_experimentalParam5: + case ZSTD_cParameter.ZSTD_c_jobSize: + case ZSTD_cParameter.ZSTD_c_overlapLog: + case ZSTD_cParameter.ZSTD_c_experimentalParam1: + case ZSTD_cParameter.ZSTD_c_experimentalParam8: + case ZSTD_cParameter.ZSTD_c_enableLongDistanceMatching: + case ZSTD_cParameter.ZSTD_c_ldmHashLog: + case ZSTD_cParameter.ZSTD_c_ldmMinMatch: + case ZSTD_cParameter.ZSTD_c_ldmBucketSizeLog: + case ZSTD_cParameter.ZSTD_c_targetCBlockSize: + case ZSTD_cParameter.ZSTD_c_experimentalParam7: + case ZSTD_cParameter.ZSTD_c_experimentalParam9: + case ZSTD_cParameter.ZSTD_c_experimentalParam10: + case ZSTD_cParameter.ZSTD_c_experimentalParam11: + case ZSTD_cParameter.ZSTD_c_experimentalParam12: + case ZSTD_cParameter.ZSTD_c_experimentalParam13: + case ZSTD_cParameter.ZSTD_c_experimentalParam20: + case ZSTD_cParameter.ZSTD_c_experimentalParam14: + case ZSTD_cParameter.ZSTD_c_experimentalParam15: + case ZSTD_cParameter.ZSTD_c_experimentalParam16: + case ZSTD_cParameter.ZSTD_c_experimentalParam17: + case ZSTD_cParameter.ZSTD_c_experimentalParam18: + case ZSTD_cParameter.ZSTD_c_experimentalParam19: + break; + default: + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_unsupported)); + } + + return ZSTD_CCtxParams_setParameter(&cctx->requestedParams, param, value); + } + + /*! ZSTD_CCtxParams_setParameter() : Requires v1.4.0+ + * Similar to ZSTD_CCtx_setParameter. + * Set one compression parameter, selected by enum ZSTD_cParameter. + * Parameters must be applied to a ZSTD_CCtx using + * ZSTD_CCtx_setParametersUsingCCtxParams(). + * @result : a code representing success or failure (which can be tested with + * ZSTD_isError()). + */ + public static nuint ZSTD_CCtxParams_setParameter( + ZSTD_CCtx_params_s* CCtxParams, + ZSTD_cParameter param, + int value + ) + { + switch (param) + { + case ZSTD_cParameter.ZSTD_c_experimentalParam2: + if (ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_experimentalParam2, value) == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + CCtxParams->format = (ZSTD_format_e)value; + return (nuint)CCtxParams->format; + case ZSTD_cParameter.ZSTD_c_compressionLevel: + { + { + nuint err_code = ZSTD_cParam_clampBounds(param, &value); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (value == 0) + { + CCtxParams->compressionLevel = 3; + } + else + { + CCtxParams->compressionLevel = value; + } + + if (CCtxParams->compressionLevel >= 0) + { + return (nuint)CCtxParams->compressionLevel; + } + + return 0; + } + + case ZSTD_cParameter.ZSTD_c_windowLog: + if (value != 0) + { + if (ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_windowLog, value) == 0) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound) + ); + } + } + + CCtxParams->cParams.windowLog = (uint)value; + return CCtxParams->cParams.windowLog; + case ZSTD_cParameter.ZSTD_c_hashLog: + if (value != 0) + { + if (ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_hashLog, value) == 0) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound) + ); + } + } + + CCtxParams->cParams.hashLog = (uint)value; + return CCtxParams->cParams.hashLog; + case ZSTD_cParameter.ZSTD_c_chainLog: + if (value != 0) + { + if (ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_chainLog, value) == 0) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound) + ); + } + } + + CCtxParams->cParams.chainLog = (uint)value; + return CCtxParams->cParams.chainLog; + case ZSTD_cParameter.ZSTD_c_searchLog: + if (value != 0) + { + if (ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_searchLog, value) == 0) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound) + ); + } + } + + CCtxParams->cParams.searchLog = (uint)value; + return (nuint)value; + case ZSTD_cParameter.ZSTD_c_minMatch: + if (value != 0) + { + if (ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_minMatch, value) == 0) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound) + ); + } + } + + CCtxParams->cParams.minMatch = (uint)value; + return CCtxParams->cParams.minMatch; + case ZSTD_cParameter.ZSTD_c_targetLength: + if (ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_targetLength, value) == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + CCtxParams->cParams.targetLength = (uint)value; + return CCtxParams->cParams.targetLength; + case ZSTD_cParameter.ZSTD_c_strategy: + if (value != 0) + { + if (ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_strategy, value) == 0) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound) + ); + } + } + + CCtxParams->cParams.strategy = (ZSTD_strategy)value; + return (nuint)CCtxParams->cParams.strategy; + case ZSTD_cParameter.ZSTD_c_contentSizeFlag: + CCtxParams->fParams.contentSizeFlag = value != 0 ? 1 : 0; + return (nuint)CCtxParams->fParams.contentSizeFlag; + case ZSTD_cParameter.ZSTD_c_checksumFlag: + CCtxParams->fParams.checksumFlag = value != 0 ? 1 : 0; + return (nuint)CCtxParams->fParams.checksumFlag; + case ZSTD_cParameter.ZSTD_c_dictIDFlag: + CCtxParams->fParams.noDictIDFlag = value == 0 ? 1 : 0; + return CCtxParams->fParams.noDictIDFlag == 0 ? 1U : 0U; + case ZSTD_cParameter.ZSTD_c_experimentalParam3: + CCtxParams->forceWindow = value != 0 ? 1 : 0; + return (nuint)CCtxParams->forceWindow; + case ZSTD_cParameter.ZSTD_c_experimentalParam4: + { + ZSTD_dictAttachPref_e pref = (ZSTD_dictAttachPref_e)value; + if ( + ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_experimentalParam4, (int)pref) + == 0 + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + CCtxParams->attachDictPref = pref; + return (nuint)CCtxParams->attachDictPref; + } + + case ZSTD_cParameter.ZSTD_c_experimentalParam5: + { + ZSTD_paramSwitch_e lcm = (ZSTD_paramSwitch_e)value; + if ( + ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_experimentalParam5, (int)lcm) + == 0 + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + CCtxParams->literalCompressionMode = lcm; + return (nuint)CCtxParams->literalCompressionMode; + } + + case ZSTD_cParameter.ZSTD_c_nbWorkers: + { + nuint err_code = ZSTD_cParam_clampBounds(param, &value); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + CCtxParams->nbWorkers = value; + return (nuint)CCtxParams->nbWorkers; + case ZSTD_cParameter.ZSTD_c_jobSize: + if (value != 0 && value < 512 * (1 << 10)) + { + value = 512 * (1 << 10); + } + + { + nuint err_code = ZSTD_cParam_clampBounds(param, &value); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + assert(value >= 0); + CCtxParams->jobSize = (nuint)value; + return CCtxParams->jobSize; + case ZSTD_cParameter.ZSTD_c_overlapLog: + { + nuint err_code = ZSTD_cParam_clampBounds( + ZSTD_cParameter.ZSTD_c_overlapLog, + &value + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + CCtxParams->overlapLog = value; + return (nuint)CCtxParams->overlapLog; + case ZSTD_cParameter.ZSTD_c_experimentalParam1: + { + nuint err_code = ZSTD_cParam_clampBounds( + ZSTD_cParameter.ZSTD_c_overlapLog, + &value + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + CCtxParams->rsyncable = value; + return (nuint)CCtxParams->rsyncable; + case ZSTD_cParameter.ZSTD_c_experimentalParam8: + CCtxParams->enableDedicatedDictSearch = value != 0 ? 1 : 0; + return (nuint)CCtxParams->enableDedicatedDictSearch; + case ZSTD_cParameter.ZSTD_c_enableLongDistanceMatching: + if ( + ZSTD_cParam_withinBounds( + ZSTD_cParameter.ZSTD_c_enableLongDistanceMatching, + value + ) == 0 + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + CCtxParams->ldmParams.enableLdm = (ZSTD_paramSwitch_e)value; + return (nuint)CCtxParams->ldmParams.enableLdm; + case ZSTD_cParameter.ZSTD_c_ldmHashLog: + if (value != 0) + { + if (ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_ldmHashLog, value) == 0) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound) + ); + } + } + + CCtxParams->ldmParams.hashLog = (uint)value; + return CCtxParams->ldmParams.hashLog; + case ZSTD_cParameter.ZSTD_c_ldmMinMatch: + if (value != 0) + { + if (ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_ldmMinMatch, value) == 0) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound) + ); + } + } + + CCtxParams->ldmParams.minMatchLength = (uint)value; + return CCtxParams->ldmParams.minMatchLength; + case ZSTD_cParameter.ZSTD_c_ldmBucketSizeLog: + if (value != 0) + { + if ( + ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_ldmBucketSizeLog, value) + == 0 + ) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound) + ); + } + } + + CCtxParams->ldmParams.bucketSizeLog = (uint)value; + return CCtxParams->ldmParams.bucketSizeLog; + case ZSTD_cParameter.ZSTD_c_ldmHashRateLog: + if (value != 0) + { + if (ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_ldmHashRateLog, value) == 0) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound) + ); + } + } + + CCtxParams->ldmParams.hashRateLog = (uint)value; + return CCtxParams->ldmParams.hashRateLog; + case ZSTD_cParameter.ZSTD_c_targetCBlockSize: + if (value != 0) + { + value = value > 1340 ? value : 1340; + if ( + ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_targetCBlockSize, value) + == 0 + ) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound) + ); + } + } + + CCtxParams->targetCBlockSize = (uint)value; + return CCtxParams->targetCBlockSize; + case ZSTD_cParameter.ZSTD_c_experimentalParam7: + if (value != 0) + { + if ( + ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_experimentalParam7, value) + == 0 + ) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound) + ); + } + } + + CCtxParams->srcSizeHint = value; + return (nuint)CCtxParams->srcSizeHint; + case ZSTD_cParameter.ZSTD_c_experimentalParam9: + if (ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_experimentalParam9, value) == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + CCtxParams->inBufferMode = (ZSTD_bufferMode_e)value; + return (nuint)CCtxParams->inBufferMode; + case ZSTD_cParameter.ZSTD_c_experimentalParam10: + if ( + ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_experimentalParam10, value) == 0 + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + CCtxParams->outBufferMode = (ZSTD_bufferMode_e)value; + return (nuint)CCtxParams->outBufferMode; + case ZSTD_cParameter.ZSTD_c_experimentalParam11: + if ( + ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_experimentalParam11, value) == 0 + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + CCtxParams->blockDelimiters = (ZSTD_sequenceFormat_e)value; + return (nuint)CCtxParams->blockDelimiters; + case ZSTD_cParameter.ZSTD_c_experimentalParam12: + if ( + ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_experimentalParam12, value) == 0 + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + CCtxParams->validateSequences = value; + return (nuint)CCtxParams->validateSequences; + case ZSTD_cParameter.ZSTD_c_experimentalParam13: + if ( + ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_experimentalParam13, value) == 0 + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + CCtxParams->postBlockSplitter = (ZSTD_paramSwitch_e)value; + return (nuint)CCtxParams->postBlockSplitter; + case ZSTD_cParameter.ZSTD_c_experimentalParam20: + if ( + ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_experimentalParam20, value) == 0 + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + CCtxParams->preBlockSplitter_level = value; + return (nuint)CCtxParams->preBlockSplitter_level; + case ZSTD_cParameter.ZSTD_c_experimentalParam14: + if ( + ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_experimentalParam14, value) == 0 + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + CCtxParams->useRowMatchFinder = (ZSTD_paramSwitch_e)value; + return (nuint)CCtxParams->useRowMatchFinder; + case ZSTD_cParameter.ZSTD_c_experimentalParam15: + if ( + ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_experimentalParam15, value) == 0 + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + CCtxParams->deterministicRefPrefix = !(value == 0) ? 1 : 0; + return (nuint)CCtxParams->deterministicRefPrefix; + case ZSTD_cParameter.ZSTD_c_experimentalParam16: + if ( + ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_experimentalParam16, value) == 0 + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + CCtxParams->prefetchCDictTables = (ZSTD_paramSwitch_e)value; + return (nuint)CCtxParams->prefetchCDictTables; + case ZSTD_cParameter.ZSTD_c_experimentalParam17: + if ( + ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_experimentalParam17, value) == 0 + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + CCtxParams->enableMatchFinderFallback = value; + return (nuint)CCtxParams->enableMatchFinderFallback; + case ZSTD_cParameter.ZSTD_c_experimentalParam18: + if (value != 0) + { + if ( + ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_experimentalParam18, value) + == 0 + ) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound) + ); + } + } + + assert(value >= 0); + CCtxParams->maxBlockSize = (nuint)value; + return CCtxParams->maxBlockSize; + case ZSTD_cParameter.ZSTD_c_experimentalParam19: + if ( + ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_experimentalParam19, value) == 0 + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + CCtxParams->searchForExternalRepcodes = (ZSTD_paramSwitch_e)value; + return (nuint)CCtxParams->searchForExternalRepcodes; + default: + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_unsupported)); + } + } + + /*! ZSTD_CCtx_getParameter() : + * Get the requested compression parameter value, selected by enum ZSTD_cParameter, + * and store it into int* value. + * @return : 0, or an error code (which can be tested with ZSTD_isError()). + */ + public static nuint ZSTD_CCtx_getParameter(ZSTD_CCtx_s* cctx, ZSTD_cParameter param, int* value) + { + return ZSTD_CCtxParams_getParameter(&cctx->requestedParams, param, value); + } + + /*! ZSTD_CCtxParams_getParameter() : + * Similar to ZSTD_CCtx_getParameter. + * Get the requested value of one compression parameter, selected by enum ZSTD_cParameter. + * @result : 0, or an error code (which can be tested with ZSTD_isError()). + */ + public static nuint ZSTD_CCtxParams_getParameter( + ZSTD_CCtx_params_s* CCtxParams, + ZSTD_cParameter param, + int* value + ) + { + switch (param) + { + case ZSTD_cParameter.ZSTD_c_experimentalParam2: + *value = (int)CCtxParams->format; + break; + case ZSTD_cParameter.ZSTD_c_compressionLevel: + *value = CCtxParams->compressionLevel; + break; + case ZSTD_cParameter.ZSTD_c_windowLog: + *value = (int)CCtxParams->cParams.windowLog; + break; + case ZSTD_cParameter.ZSTD_c_hashLog: + *value = (int)CCtxParams->cParams.hashLog; + break; + case ZSTD_cParameter.ZSTD_c_chainLog: + *value = (int)CCtxParams->cParams.chainLog; + break; + case ZSTD_cParameter.ZSTD_c_searchLog: + *value = (int)CCtxParams->cParams.searchLog; + break; + case ZSTD_cParameter.ZSTD_c_minMatch: + *value = (int)CCtxParams->cParams.minMatch; + break; + case ZSTD_cParameter.ZSTD_c_targetLength: + *value = (int)CCtxParams->cParams.targetLength; + break; + case ZSTD_cParameter.ZSTD_c_strategy: + *value = (int)CCtxParams->cParams.strategy; + break; + case ZSTD_cParameter.ZSTD_c_contentSizeFlag: + *value = CCtxParams->fParams.contentSizeFlag; + break; + case ZSTD_cParameter.ZSTD_c_checksumFlag: + *value = CCtxParams->fParams.checksumFlag; + break; + case ZSTD_cParameter.ZSTD_c_dictIDFlag: + *value = CCtxParams->fParams.noDictIDFlag == 0 ? 1 : 0; + break; + case ZSTD_cParameter.ZSTD_c_experimentalParam3: + *value = CCtxParams->forceWindow; + break; + case ZSTD_cParameter.ZSTD_c_experimentalParam4: + *value = (int)CCtxParams->attachDictPref; + break; + case ZSTD_cParameter.ZSTD_c_experimentalParam5: + *value = (int)CCtxParams->literalCompressionMode; + break; + case ZSTD_cParameter.ZSTD_c_nbWorkers: + *value = CCtxParams->nbWorkers; + break; + case ZSTD_cParameter.ZSTD_c_jobSize: + assert(CCtxParams->jobSize <= 2147483647); + *value = (int)CCtxParams->jobSize; + break; + case ZSTD_cParameter.ZSTD_c_overlapLog: + *value = CCtxParams->overlapLog; + break; + case ZSTD_cParameter.ZSTD_c_experimentalParam1: + *value = CCtxParams->rsyncable; + break; + case ZSTD_cParameter.ZSTD_c_experimentalParam8: + *value = CCtxParams->enableDedicatedDictSearch; + break; + case ZSTD_cParameter.ZSTD_c_enableLongDistanceMatching: + *value = (int)CCtxParams->ldmParams.enableLdm; + break; + case ZSTD_cParameter.ZSTD_c_ldmHashLog: + *value = (int)CCtxParams->ldmParams.hashLog; + break; + case ZSTD_cParameter.ZSTD_c_ldmMinMatch: + *value = (int)CCtxParams->ldmParams.minMatchLength; + break; + case ZSTD_cParameter.ZSTD_c_ldmBucketSizeLog: + *value = (int)CCtxParams->ldmParams.bucketSizeLog; + break; + case ZSTD_cParameter.ZSTD_c_ldmHashRateLog: + *value = (int)CCtxParams->ldmParams.hashRateLog; + break; + case ZSTD_cParameter.ZSTD_c_targetCBlockSize: + *value = (int)CCtxParams->targetCBlockSize; + break; + case ZSTD_cParameter.ZSTD_c_experimentalParam7: + *value = CCtxParams->srcSizeHint; + break; + case ZSTD_cParameter.ZSTD_c_experimentalParam9: + *value = (int)CCtxParams->inBufferMode; + break; + case ZSTD_cParameter.ZSTD_c_experimentalParam10: + *value = (int)CCtxParams->outBufferMode; + break; + case ZSTD_cParameter.ZSTD_c_experimentalParam11: + *value = (int)CCtxParams->blockDelimiters; + break; + case ZSTD_cParameter.ZSTD_c_experimentalParam12: + *value = CCtxParams->validateSequences; + break; + case ZSTD_cParameter.ZSTD_c_experimentalParam13: + *value = (int)CCtxParams->postBlockSplitter; + break; + case ZSTD_cParameter.ZSTD_c_experimentalParam20: + *value = CCtxParams->preBlockSplitter_level; + break; + case ZSTD_cParameter.ZSTD_c_experimentalParam14: + *value = (int)CCtxParams->useRowMatchFinder; + break; + case ZSTD_cParameter.ZSTD_c_experimentalParam15: + *value = CCtxParams->deterministicRefPrefix; + break; + case ZSTD_cParameter.ZSTD_c_experimentalParam16: + *value = (int)CCtxParams->prefetchCDictTables; + break; + case ZSTD_cParameter.ZSTD_c_experimentalParam17: + *value = CCtxParams->enableMatchFinderFallback; + break; + case ZSTD_cParameter.ZSTD_c_experimentalParam18: + *value = (int)CCtxParams->maxBlockSize; + break; + case ZSTD_cParameter.ZSTD_c_experimentalParam19: + *value = (int)CCtxParams->searchForExternalRepcodes; + break; + default: + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_unsupported)); + } + + return 0; + } + + /** ZSTD_CCtx_setParametersUsingCCtxParams() : + * just applies `params` into `cctx` + * no action is performed, parameters are merely stored. + * If ZSTDMT is enabled, parameters are pushed to cctx->mtctx. + * This is possible even if a compression is ongoing. + * In which case, new parameters will be applied on the fly, starting with next compression job. + */ + public static nuint ZSTD_CCtx_setParametersUsingCCtxParams( + ZSTD_CCtx_s* cctx, + ZSTD_CCtx_params_s* @params + ) + { + if (cctx->streamStage != ZSTD_cStreamStage.zcss_init) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_stage_wrong)); + } + + if (cctx->cdict != null) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_stage_wrong)); + } + + cctx->requestedParams = *@params; + return 0; + } + + /*! ZSTD_CCtx_setCParams() : + * Set all parameters provided within @p cparams into the working @p cctx. + * Note : if modifying parameters during compression (MT mode only), + * note that changes to the .windowLog parameter will be ignored. + * @return 0 on success, or an error code (can be checked with ZSTD_isError()). + * On failure, no parameters are updated. + */ + public static nuint ZSTD_CCtx_setCParams(ZSTD_CCtx_s* cctx, ZSTD_compressionParameters cparams) + { + { + /* only update if all parameters are valid */ + nuint err_code = ZSTD_checkCParams(cparams); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_CCtx_setParameter( + cctx, + ZSTD_cParameter.ZSTD_c_windowLog, + (int)cparams.windowLog + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_CCtx_setParameter( + cctx, + ZSTD_cParameter.ZSTD_c_chainLog, + (int)cparams.chainLog + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_CCtx_setParameter( + cctx, + ZSTD_cParameter.ZSTD_c_hashLog, + (int)cparams.hashLog + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_CCtx_setParameter( + cctx, + ZSTD_cParameter.ZSTD_c_searchLog, + (int)cparams.searchLog + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_CCtx_setParameter( + cctx, + ZSTD_cParameter.ZSTD_c_minMatch, + (int)cparams.minMatch + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_CCtx_setParameter( + cctx, + ZSTD_cParameter.ZSTD_c_targetLength, + (int)cparams.targetLength + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_CCtx_setParameter( + cctx, + ZSTD_cParameter.ZSTD_c_strategy, + (int)cparams.strategy + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return 0; + } + + /*! ZSTD_CCtx_setFParams() : + * Set all parameters provided within @p fparams into the working @p cctx. + * @return 0 on success, or an error code (can be checked with ZSTD_isError()). + */ + public static nuint ZSTD_CCtx_setFParams(ZSTD_CCtx_s* cctx, ZSTD_frameParameters fparams) + { + { + nuint err_code = ZSTD_CCtx_setParameter( + cctx, + ZSTD_cParameter.ZSTD_c_contentSizeFlag, + fparams.contentSizeFlag != 0 ? 1 : 0 + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_CCtx_setParameter( + cctx, + ZSTD_cParameter.ZSTD_c_checksumFlag, + fparams.checksumFlag != 0 ? 1 : 0 + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_CCtx_setParameter( + cctx, + ZSTD_cParameter.ZSTD_c_dictIDFlag, + fparams.noDictIDFlag == 0 ? 1 : 0 + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return 0; + } + + /*! ZSTD_CCtx_setParams() : + * Set all parameters provided within @p params into the working @p cctx. + * @return 0 on success, or an error code (can be checked with ZSTD_isError()). + */ + public static nuint ZSTD_CCtx_setParams(ZSTD_CCtx_s* cctx, ZSTD_parameters @params) + { + { + /* First check cParams, because we want to update all or none. */ + nuint err_code = ZSTD_checkCParams(@params.cParams); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + /* Next set fParams, because this could fail if the cctx isn't in init stage. */ + nuint err_code = ZSTD_CCtx_setFParams(cctx, @params.fParams); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + /* Finally set cParams, which should succeed. */ + nuint err_code = ZSTD_CCtx_setCParams(cctx, @params.cParams); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return 0; + } + + /*! ZSTD_CCtx_setPledgedSrcSize() : + * Total input data size to be compressed as a single frame. + * Value will be written in frame header, unless if explicitly forbidden using ZSTD_c_contentSizeFlag. + * This value will also be controlled at end of frame, and trigger an error if not respected. + * @result : 0, or an error code (which can be tested with ZSTD_isError()). + * Note 1 : pledgedSrcSize==0 actually means zero, aka an empty frame. + * In order to mean "unknown content size", pass constant ZSTD_CONTENTSIZE_UNKNOWN. + * ZSTD_CONTENTSIZE_UNKNOWN is default value for any new frame. + * Note 2 : pledgedSrcSize is only valid once, for the next frame. + * It's discarded at the end of the frame, and replaced by ZSTD_CONTENTSIZE_UNKNOWN. + * Note 3 : Whenever all input data is provided and consumed in a single round, + * for example with ZSTD_compress2(), + * or invoking immediately ZSTD_compressStream2(,,,ZSTD_e_end), + * this value is automatically overridden by srcSize instead. + */ + public static nuint ZSTD_CCtx_setPledgedSrcSize(ZSTD_CCtx_s* cctx, ulong pledgedSrcSize) + { + if (cctx->streamStage != ZSTD_cStreamStage.zcss_init) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_stage_wrong)); + } + + cctx->pledgedSrcSizePlusOne = pledgedSrcSize + 1; + return 0; + } + + /** + * Initializes the local dictionary using requested parameters. + * NOTE: Initialization does not employ the pledged src size, + * because the dictionary may be used for multiple compressions. + */ + private static nuint ZSTD_initLocalDict(ZSTD_CCtx_s* cctx) + { + ZSTD_localDict* dl = &cctx->localDict; + if (dl->dict == null) + { + assert(dl->dictBuffer == null); + assert(dl->cdict == null); + assert(dl->dictSize == 0); + return 0; + } + + if (dl->cdict != null) + { + assert(cctx->cdict == dl->cdict); + return 0; + } + + assert(dl->dictSize > 0); + assert(cctx->cdict == null); + assert(cctx->prefixDict.dict == null); + dl->cdict = ZSTD_createCDict_advanced2( + dl->dict, + dl->dictSize, + ZSTD_dictLoadMethod_e.ZSTD_dlm_byRef, + dl->dictContentType, + &cctx->requestedParams, + cctx->customMem + ); + if (dl->cdict == null) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + + cctx->cdict = dl->cdict; + return 0; + } + + /*! ZSTD_CCtx_loadDictionary_advanced() : + * Same as ZSTD_CCtx_loadDictionary(), but gives finer control over + * how to load the dictionary (by copy ? by reference ?) + * and how to interpret it (automatic ? force raw mode ? full mode only ?) */ + public static nuint ZSTD_CCtx_loadDictionary_advanced( + ZSTD_CCtx_s* cctx, + void* dict, + nuint dictSize, + ZSTD_dictLoadMethod_e dictLoadMethod, + ZSTD_dictContentType_e dictContentType + ) + { + if (cctx->streamStage != ZSTD_cStreamStage.zcss_init) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_stage_wrong)); + } + + ZSTD_clearAllDicts(cctx); + if (dict == null || dictSize == 0) + { + return 0; + } + + if (dictLoadMethod == ZSTD_dictLoadMethod_e.ZSTD_dlm_byRef) + { + cctx->localDict.dict = dict; + } + else + { + /* copy dictionary content inside CCtx to own its lifetime */ + void* dictBuffer; + if (cctx->staticSize != 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + + dictBuffer = ZSTD_customMalloc(dictSize, cctx->customMem); + if (dictBuffer == null) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + + memcpy(dictBuffer, dict, (uint)dictSize); + cctx->localDict.dictBuffer = dictBuffer; + cctx->localDict.dict = dictBuffer; + } + + cctx->localDict.dictSize = dictSize; + cctx->localDict.dictContentType = dictContentType; + return 0; + } + + /*! ZSTD_CCtx_loadDictionary_byReference() : + * Same as ZSTD_CCtx_loadDictionary(), but dictionary content is referenced, instead of being copied into CCtx. + * It saves some memory, but also requires that `dict` outlives its usage within `cctx` */ + public static nuint ZSTD_CCtx_loadDictionary_byReference( + ZSTD_CCtx_s* cctx, + void* dict, + nuint dictSize + ) + { + return ZSTD_CCtx_loadDictionary_advanced( + cctx, + dict, + dictSize, + ZSTD_dictLoadMethod_e.ZSTD_dlm_byRef, + ZSTD_dictContentType_e.ZSTD_dct_auto + ); + } + + /*! ZSTD_CCtx_loadDictionary() : Requires v1.4.0+ + * Create an internal CDict from `dict` buffer. + * Decompression will have to use same dictionary. + * @result : 0, or an error code (which can be tested with ZSTD_isError()). + * Special: Loading a NULL (or 0-size) dictionary invalidates previous dictionary, + * meaning "return to no-dictionary mode". + * Note 1 : Dictionary is sticky, it will be used for all future compressed frames, + * until parameters are reset, a new dictionary is loaded, or the dictionary + * is explicitly invalidated by loading a NULL dictionary. + * Note 2 : Loading a dictionary involves building tables. + * It's also a CPU consuming operation, with non-negligible impact on latency. + * Tables are dependent on compression parameters, and for this reason, + * compression parameters can no longer be changed after loading a dictionary. + * Note 3 :`dict` content will be copied internally. + * Use experimental ZSTD_CCtx_loadDictionary_byReference() to reference content instead. + * In such a case, dictionary buffer must outlive its users. + * Note 4 : Use ZSTD_CCtx_loadDictionary_advanced() + * to precisely select how dictionary content must be interpreted. + * Note 5 : This method does not benefit from LDM (long distance mode). + * If you want to employ LDM on some large dictionary content, + * prefer employing ZSTD_CCtx_refPrefix() described below. + */ + public static nuint ZSTD_CCtx_loadDictionary(ZSTD_CCtx_s* cctx, void* dict, nuint dictSize) + { + return ZSTD_CCtx_loadDictionary_advanced( + cctx, + dict, + dictSize, + ZSTD_dictLoadMethod_e.ZSTD_dlm_byCopy, + ZSTD_dictContentType_e.ZSTD_dct_auto + ); + } + + /*! ZSTD_CCtx_refCDict() : Requires v1.4.0+ + * Reference a prepared dictionary, to be used for all future compressed frames. + * Note that compression parameters are enforced from within CDict, + * and supersede any compression parameter previously set within CCtx. + * The parameters ignored are labelled as "superseded-by-cdict" in the ZSTD_cParameter enum docs. + * The ignored parameters will be used again if the CCtx is returned to no-dictionary mode. + * The dictionary will remain valid for future compressed frames using same CCtx. + * @result : 0, or an error code (which can be tested with ZSTD_isError()). + * Special : Referencing a NULL CDict means "return to no-dictionary mode". + * Note 1 : Currently, only one dictionary can be managed. + * Referencing a new dictionary effectively "discards" any previous one. + * Note 2 : CDict is just referenced, its lifetime must outlive its usage within CCtx. */ + public static nuint ZSTD_CCtx_refCDict(ZSTD_CCtx_s* cctx, ZSTD_CDict_s* cdict) + { + if (cctx->streamStage != ZSTD_cStreamStage.zcss_init) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_stage_wrong)); + } + + ZSTD_clearAllDicts(cctx); + cctx->cdict = cdict; + return 0; + } + + public static nuint ZSTD_CCtx_refThreadPool(ZSTD_CCtx_s* cctx, void* pool) + { + if (cctx->streamStage != ZSTD_cStreamStage.zcss_init) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_stage_wrong)); + } + + cctx->pool = pool; + return 0; + } + + /*! ZSTD_CCtx_refPrefix() : Requires v1.4.0+ + * Reference a prefix (single-usage dictionary) for next compressed frame. + * A prefix is **only used once**. Tables are discarded at end of frame (ZSTD_e_end). + * Decompression will need same prefix to properly regenerate data. + * Compressing with a prefix is similar in outcome as performing a diff and compressing it, + * but performs much faster, especially during decompression (compression speed is tunable with compression level). + * This method is compatible with LDM (long distance mode). + * @result : 0, or an error code (which can be tested with ZSTD_isError()). + * Special: Adding any prefix (including NULL) invalidates any previous prefix or dictionary + * Note 1 : Prefix buffer is referenced. It **must** outlive compression. + * Its content must remain unmodified during compression. + * Note 2 : If the intention is to diff some large src data blob with some prior version of itself, + * ensure that the window size is large enough to contain the entire source. + * See ZSTD_c_windowLog. + * Note 3 : Referencing a prefix involves building tables, which are dependent on compression parameters. + * It's a CPU consuming operation, with non-negligible impact on latency. + * If there is a need to use the same prefix multiple times, consider loadDictionary instead. + * Note 4 : By default, the prefix is interpreted as raw content (ZSTD_dct_rawContent). + * Use experimental ZSTD_CCtx_refPrefix_advanced() to alter dictionary interpretation. */ + public static nuint ZSTD_CCtx_refPrefix(ZSTD_CCtx_s* cctx, void* prefix, nuint prefixSize) + { + return ZSTD_CCtx_refPrefix_advanced( + cctx, + prefix, + prefixSize, + ZSTD_dictContentType_e.ZSTD_dct_rawContent + ); + } + + /*! ZSTD_CCtx_refPrefix_advanced() : + * Same as ZSTD_CCtx_refPrefix(), but gives finer control over + * how to interpret prefix content (automatic ? force raw mode (default) ? full mode only ?) */ + public static nuint ZSTD_CCtx_refPrefix_advanced( + ZSTD_CCtx_s* cctx, + void* prefix, + nuint prefixSize, + ZSTD_dictContentType_e dictContentType + ) + { + if (cctx->streamStage != ZSTD_cStreamStage.zcss_init) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_stage_wrong)); + } + + ZSTD_clearAllDicts(cctx); + if (prefix != null && prefixSize > 0) + { + cctx->prefixDict.dict = prefix; + cctx->prefixDict.dictSize = prefixSize; + cctx->prefixDict.dictContentType = dictContentType; + } + + return 0; + } + + /*! ZSTD_CCtx_reset() : + * Also dumps dictionary */ + public static nuint ZSTD_CCtx_reset(ZSTD_CCtx_s* cctx, ZSTD_ResetDirective reset) + { + if ( + reset == ZSTD_ResetDirective.ZSTD_reset_session_only + || reset == ZSTD_ResetDirective.ZSTD_reset_session_and_parameters + ) + { + cctx->streamStage = ZSTD_cStreamStage.zcss_init; + cctx->pledgedSrcSizePlusOne = 0; + } + + if ( + reset == ZSTD_ResetDirective.ZSTD_reset_parameters + || reset == ZSTD_ResetDirective.ZSTD_reset_session_and_parameters + ) + { + if (cctx->streamStage != ZSTD_cStreamStage.zcss_init) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_stage_wrong)); + } + + ZSTD_clearAllDicts(cctx); + return ZSTD_CCtxParams_reset(&cctx->requestedParams); + } + + return 0; + } + + /** ZSTD_checkCParams() : + control CParam values remain within authorized range. + @return : 0, or an error code if one value is beyond authorized range */ + public static nuint ZSTD_checkCParams(ZSTD_compressionParameters cParams) + { + if (ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_windowLog, (int)cParams.windowLog) == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + if (ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_chainLog, (int)cParams.chainLog) == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + if (ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_hashLog, (int)cParams.hashLog) == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + if (ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_searchLog, (int)cParams.searchLog) == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + if (ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_minMatch, (int)cParams.minMatch) == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + if ( + ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_targetLength, (int)cParams.targetLength) + == 0 + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + if (ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_strategy, (int)cParams.strategy) == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + return 0; + } + + /** ZSTD_clampCParams() : + * make CParam values within valid range. + * @return : valid CParams */ + private static ZSTD_compressionParameters ZSTD_clampCParams(ZSTD_compressionParameters cParams) + { + { + ZSTD_bounds bounds = ZSTD_cParam_getBounds(ZSTD_cParameter.ZSTD_c_windowLog); + if ((int)cParams.windowLog < bounds.lowerBound) + { + cParams.windowLog = (uint)bounds.lowerBound; + } + else if ((int)cParams.windowLog > bounds.upperBound) + { + cParams.windowLog = (uint)bounds.upperBound; + } + } + + { + ZSTD_bounds bounds = ZSTD_cParam_getBounds(ZSTD_cParameter.ZSTD_c_chainLog); + if ((int)cParams.chainLog < bounds.lowerBound) + { + cParams.chainLog = (uint)bounds.lowerBound; + } + else if ((int)cParams.chainLog > bounds.upperBound) + { + cParams.chainLog = (uint)bounds.upperBound; + } + } + + { + ZSTD_bounds bounds = ZSTD_cParam_getBounds(ZSTD_cParameter.ZSTD_c_hashLog); + if ((int)cParams.hashLog < bounds.lowerBound) + { + cParams.hashLog = (uint)bounds.lowerBound; + } + else if ((int)cParams.hashLog > bounds.upperBound) + { + cParams.hashLog = (uint)bounds.upperBound; + } + } + + { + ZSTD_bounds bounds = ZSTD_cParam_getBounds(ZSTD_cParameter.ZSTD_c_searchLog); + if ((int)cParams.searchLog < bounds.lowerBound) + { + cParams.searchLog = (uint)bounds.lowerBound; + } + else if ((int)cParams.searchLog > bounds.upperBound) + { + cParams.searchLog = (uint)bounds.upperBound; + } + } + + { + ZSTD_bounds bounds = ZSTD_cParam_getBounds(ZSTD_cParameter.ZSTD_c_minMatch); + if ((int)cParams.minMatch < bounds.lowerBound) + { + cParams.minMatch = (uint)bounds.lowerBound; + } + else if ((int)cParams.minMatch > bounds.upperBound) + { + cParams.minMatch = (uint)bounds.upperBound; + } + } + + { + ZSTD_bounds bounds = ZSTD_cParam_getBounds(ZSTD_cParameter.ZSTD_c_targetLength); + if ((int)cParams.targetLength < bounds.lowerBound) + { + cParams.targetLength = (uint)bounds.lowerBound; + } + else if ((int)cParams.targetLength > bounds.upperBound) + { + cParams.targetLength = (uint)bounds.upperBound; + } + } + + { + ZSTD_bounds bounds = ZSTD_cParam_getBounds(ZSTD_cParameter.ZSTD_c_strategy); + if ((int)cParams.strategy < bounds.lowerBound) + { + cParams.strategy = (ZSTD_strategy)bounds.lowerBound; + } + else if ((int)cParams.strategy > bounds.upperBound) + { + cParams.strategy = (ZSTD_strategy)bounds.upperBound; + } + } + + return cParams; + } + + /** ZSTD_cycleLog() : + * condition for correct operation : hashLog > 1 */ + private static uint ZSTD_cycleLog(uint hashLog, ZSTD_strategy strat) + { + uint btScale = (uint)strat >= (uint)ZSTD_strategy.ZSTD_btlazy2 ? 1U : 0U; + return hashLog - btScale; + } + + /** ZSTD_dictAndWindowLog() : + * Returns an adjusted window log that is large enough to fit the source and the dictionary. + * The zstd format says that the entire dictionary is valid if one byte of the dictionary + * is within the window. So the hashLog and chainLog should be large enough to reference both + * the dictionary and the window. So we must use this adjusted dictAndWindowLog when downsizing + * the hashLog and windowLog. + * NOTE: srcSize must not be ZSTD_CONTENTSIZE_UNKNOWN. + */ + private static uint ZSTD_dictAndWindowLog(uint windowLog, ulong srcSize, ulong dictSize) + { + ulong maxWindowSize = 1UL << (sizeof(nuint) == 4 ? 30 : 31); + if (dictSize == 0) + { + return windowLog; + } + + assert(windowLog <= (uint)(sizeof(nuint) == 4 ? 30 : 31)); + assert(srcSize != unchecked(0UL - 1)); + { + ulong windowSize = 1UL << (int)windowLog; + ulong dictAndWindowSize = dictSize + windowSize; + if (windowSize >= dictSize + srcSize) + { + return windowLog; + } + else if (dictAndWindowSize >= maxWindowSize) + { + return (uint)(sizeof(nuint) == 4 ? 30 : 31); + } + else + { + return ZSTD_highbit32((uint)dictAndWindowSize - 1) + 1; + } + } + } + + /** ZSTD_adjustCParams_internal() : + * optimize `cPar` for a specified input (`srcSize` and `dictSize`). + * mostly downsize to reduce memory consumption and initialization latency. + * `srcSize` can be ZSTD_CONTENTSIZE_UNKNOWN when not known. + * `mode` is the mode for parameter adjustment. See docs for `ZSTD_CParamMode_e`. + * note : `srcSize==0` means 0! + * condition : cPar is presumed validated (can be checked using ZSTD_checkCParams()). */ + private static ZSTD_compressionParameters ZSTD_adjustCParams_internal( + ZSTD_compressionParameters cPar, + ulong srcSize, + nuint dictSize, + ZSTD_CParamMode_e mode, + ZSTD_paramSwitch_e useRowMatchFinder + ) + { + /* (1<<9) + 1 */ + const ulong minSrcSize = 513; + ulong maxWindowResize = 1UL << (sizeof(nuint) == 4 ? 30 : 31) - 1; + assert(ZSTD_checkCParams(cPar) == 0); + switch (mode) + { + case ZSTD_CParamMode_e.ZSTD_cpm_unknown: + case ZSTD_CParamMode_e.ZSTD_cpm_noAttachDict: + break; + case ZSTD_CParamMode_e.ZSTD_cpm_createCDict: + if (dictSize != 0 && srcSize == unchecked(0UL - 1)) + { + srcSize = minSrcSize; + } + + break; + case ZSTD_CParamMode_e.ZSTD_cpm_attachDict: + dictSize = 0; + break; + default: + assert(0 != 0); + break; + } + + if (srcSize <= maxWindowResize && dictSize <= maxWindowResize) + { + uint tSize = (uint)(srcSize + dictSize); + const uint hashSizeMin = 1 << 6; + uint srcLog = tSize < hashSizeMin ? 6 : ZSTD_highbit32(tSize - 1) + 1; + if (cPar.windowLog > srcLog) + { + cPar.windowLog = srcLog; + } + } + + if (srcSize != unchecked(0UL - 1)) + { + uint dictAndWindowLog = ZSTD_dictAndWindowLog(cPar.windowLog, srcSize, dictSize); + uint cycleLog = ZSTD_cycleLog(cPar.chainLog, cPar.strategy); + if (cPar.hashLog > dictAndWindowLog + 1) + { + cPar.hashLog = dictAndWindowLog + 1; + } + + if (cycleLog > dictAndWindowLog) + { + cPar.chainLog -= cycleLog - dictAndWindowLog; + } + } + + if (cPar.windowLog < 10) + { + cPar.windowLog = 10; + } + + if ( + mode == ZSTD_CParamMode_e.ZSTD_cpm_createCDict + && ZSTD_CDictIndicesAreTagged(&cPar) != 0 + ) + { + const uint maxShortCacheHashLog = 32 - 8; + if (cPar.hashLog > maxShortCacheHashLog) + { + cPar.hashLog = maxShortCacheHashLog; + } + + if (cPar.chainLog > maxShortCacheHashLog) + { + cPar.chainLog = maxShortCacheHashLog; + } + } + + if (useRowMatchFinder == ZSTD_paramSwitch_e.ZSTD_ps_auto) + { + useRowMatchFinder = ZSTD_paramSwitch_e.ZSTD_ps_enable; + } + + if (ZSTD_rowMatchFinderUsed(cPar.strategy, useRowMatchFinder) != 0) + { + /* Switch to 32-entry rows if searchLog is 5 (or more) */ + uint rowLog = + cPar.searchLog <= 4 ? 4 + : cPar.searchLog <= 6 ? cPar.searchLog + : 6; + const uint maxRowHashLog = 32 - 8; + uint maxHashLog = maxRowHashLog + rowLog; + assert(cPar.hashLog >= rowLog); + if (cPar.hashLog > maxHashLog) + { + cPar.hashLog = maxHashLog; + } + } + + return cPar; + } + + /*! ZSTD_adjustCParams() : + * optimize params for a given `srcSize` and `dictSize`. + * `srcSize` can be unknown, in which case use ZSTD_CONTENTSIZE_UNKNOWN. + * `dictSize` must be `0` when there is no dictionary. + * cPar can be invalid : all parameters will be clamped within valid range in the @return struct. + * This function never fails (wide contract) */ + public static ZSTD_compressionParameters ZSTD_adjustCParams( + ZSTD_compressionParameters cPar, + ulong srcSize, + nuint dictSize + ) + { + cPar = ZSTD_clampCParams(cPar); + if (srcSize == 0) + { + srcSize = unchecked(0UL - 1); + } + + return ZSTD_adjustCParams_internal( + cPar, + srcSize, + dictSize, + ZSTD_CParamMode_e.ZSTD_cpm_unknown, + ZSTD_paramSwitch_e.ZSTD_ps_auto + ); + } + + private static void ZSTD_overrideCParams( + ZSTD_compressionParameters* cParams, + ZSTD_compressionParameters* overrides + ) + { + if (overrides->windowLog != 0) + { + cParams->windowLog = overrides->windowLog; + } + + if (overrides->hashLog != 0) + { + cParams->hashLog = overrides->hashLog; + } + + if (overrides->chainLog != 0) + { + cParams->chainLog = overrides->chainLog; + } + + if (overrides->searchLog != 0) + { + cParams->searchLog = overrides->searchLog; + } + + if (overrides->minMatch != 0) + { + cParams->minMatch = overrides->minMatch; + } + + if (overrides->targetLength != 0) + { + cParams->targetLength = overrides->targetLength; + } + + if (overrides->strategy != default) + { + cParams->strategy = overrides->strategy; + } + } + + /* ZSTD_getCParamsFromCCtxParams() : + * cParams are built depending on compressionLevel, src size hints, + * LDM and manually set compression parameters. + * Note: srcSizeHint == 0 means 0! + */ + private static ZSTD_compressionParameters ZSTD_getCParamsFromCCtxParams( + ZSTD_CCtx_params_s* CCtxParams, + ulong srcSizeHint, + nuint dictSize, + ZSTD_CParamMode_e mode + ) + { + ZSTD_compressionParameters cParams; + if (srcSizeHint == unchecked(0UL - 1) && CCtxParams->srcSizeHint > 0) + { + assert(CCtxParams->srcSizeHint >= 0); + srcSizeHint = (ulong)CCtxParams->srcSizeHint; + } + + cParams = ZSTD_getCParams_internal( + CCtxParams->compressionLevel, + srcSizeHint, + dictSize, + mode + ); + if (CCtxParams->ldmParams.enableLdm == ZSTD_paramSwitch_e.ZSTD_ps_enable) + { + cParams.windowLog = 27; + } + + ZSTD_overrideCParams(&cParams, &CCtxParams->cParams); + assert(ZSTD_checkCParams(cParams) == 0); + return ZSTD_adjustCParams_internal( + cParams, + srcSizeHint, + dictSize, + mode, + CCtxParams->useRowMatchFinder + ); + } + + private static nuint ZSTD_sizeof_matchState( + ZSTD_compressionParameters* cParams, + ZSTD_paramSwitch_e useRowMatchFinder, + int enableDedicatedDictSearch, + uint forCCtx + ) + { + /* chain table size should be 0 for fast or row-hash strategies */ + nuint chainSize = + ZSTD_allocateChainTable( + cParams->strategy, + useRowMatchFinder, + enableDedicatedDictSearch != 0 && forCCtx == 0 ? 1U : 0U + ) != 0 + ? (nuint)1 << (int)cParams->chainLog + : 0; + nuint hSize = (nuint)1 << (int)cParams->hashLog; + uint hashLog3 = + forCCtx != 0 && cParams->minMatch == 3 + ? 17 < cParams->windowLog + ? 17 + : cParams->windowLog + : 0; + nuint h3Size = hashLog3 != 0 ? (nuint)1 << (int)hashLog3 : 0; + /* We don't use ZSTD_cwksp_alloc_size() here because the tables aren't + * surrounded by redzones in ASAN. */ + nuint tableSpace = chainSize * sizeof(uint) + hSize * sizeof(uint) + h3Size * sizeof(uint); + nuint optPotentialSpace = + ZSTD_cwksp_aligned64_alloc_size((52 + 1) * sizeof(uint)) + + ZSTD_cwksp_aligned64_alloc_size((35 + 1) * sizeof(uint)) + + ZSTD_cwksp_aligned64_alloc_size((31 + 1) * sizeof(uint)) + + ZSTD_cwksp_aligned64_alloc_size((1 << 8) * sizeof(uint)) + + ZSTD_cwksp_aligned64_alloc_size((nuint)(((1 << 12) + 3) * sizeof(ZSTD_match_t))) + + ZSTD_cwksp_aligned64_alloc_size((nuint)(((1 << 12) + 3) * sizeof(ZSTD_optimal_t))); + nuint lazyAdditionalSpace = + ZSTD_rowMatchFinderUsed(cParams->strategy, useRowMatchFinder) != 0 + ? ZSTD_cwksp_aligned64_alloc_size(hSize) + : 0; + nuint optSpace = + forCCtx != 0 && cParams->strategy >= ZSTD_strategy.ZSTD_btopt ? optPotentialSpace : 0; + nuint slackSpace = ZSTD_cwksp_slack_space_required(); + assert(useRowMatchFinder != ZSTD_paramSwitch_e.ZSTD_ps_auto); + return tableSpace + optSpace + slackSpace + lazyAdditionalSpace; + } + + /* Helper function for calculating memory requirements. + * Gives a tighter bound than ZSTD_sequenceBound() by taking minMatch into account. */ + private static nuint ZSTD_maxNbSeq(nuint blockSize, uint minMatch, int useSequenceProducer) + { + uint divider = (uint)(minMatch == 3 || useSequenceProducer != 0 ? 3 : 4); + return blockSize / divider; + } + + private static nuint ZSTD_estimateCCtxSize_usingCCtxParams_internal( + ZSTD_compressionParameters* cParams, + ldmParams_t* ldmParams, + int isStatic, + ZSTD_paramSwitch_e useRowMatchFinder, + nuint buffInSize, + nuint buffOutSize, + ulong pledgedSrcSize, + int useSequenceProducer, + nuint maxBlockSize + ) + { + nuint windowSize = (nuint)( + 1UL << (int)cParams->windowLog <= 1UL ? 1UL + : 1UL << (int)cParams->windowLog <= pledgedSrcSize ? 1UL << (int)cParams->windowLog + : pledgedSrcSize + ); + nuint blockSize = + ZSTD_resolveMaxBlockSize(maxBlockSize) < windowSize + ? ZSTD_resolveMaxBlockSize(maxBlockSize) + : windowSize; + nuint maxNbSeq = ZSTD_maxNbSeq(blockSize, cParams->minMatch, useSequenceProducer); + nuint tokenSpace = + ZSTD_cwksp_alloc_size(32 + blockSize) + + ZSTD_cwksp_aligned64_alloc_size(maxNbSeq * (nuint)sizeof(SeqDef_s)) + + 3 * ZSTD_cwksp_alloc_size(maxNbSeq * sizeof(byte)); + nuint tmpWorkSpace = ZSTD_cwksp_alloc_size( + (8 << 10) + 512 + sizeof(uint) * (52 + 2) > 8208 + ? (8 << 10) + 512 + sizeof(uint) * (52 + 2) + : 8208 + ); + nuint blockStateSpace = + 2 * ZSTD_cwksp_alloc_size((nuint)sizeof(ZSTD_compressedBlockState_t)); + /* enableDedicatedDictSearch */ + nuint matchStateSize = ZSTD_sizeof_matchState(cParams, useRowMatchFinder, 0, 1); + nuint ldmSpace = ZSTD_ldm_getTableSize(*ldmParams); + nuint maxNbLdmSeq = ZSTD_ldm_getMaxNbSeq(*ldmParams, blockSize); + nuint ldmSeqSpace = + ldmParams->enableLdm == ZSTD_paramSwitch_e.ZSTD_ps_enable + ? ZSTD_cwksp_aligned64_alloc_size(maxNbLdmSeq * (nuint)sizeof(rawSeq)) + : 0; + nuint bufferSpace = ZSTD_cwksp_alloc_size(buffInSize) + ZSTD_cwksp_alloc_size(buffOutSize); + nuint cctxSpace = isStatic != 0 ? ZSTD_cwksp_alloc_size((nuint)sizeof(ZSTD_CCtx_s)) : 0; + nuint maxNbExternalSeq = ZSTD_sequenceBound(blockSize); + nuint externalSeqSpace = + useSequenceProducer != 0 + ? ZSTD_cwksp_aligned64_alloc_size(maxNbExternalSeq * (nuint)sizeof(ZSTD_Sequence)) + : 0; + nuint neededSpace = + cctxSpace + + tmpWorkSpace + + blockStateSpace + + ldmSpace + + ldmSeqSpace + + matchStateSize + + tokenSpace + + bufferSpace + + externalSeqSpace; + return neededSpace; + } + + public static nuint ZSTD_estimateCCtxSize_usingCCtxParams(ZSTD_CCtx_params_s* @params) + { + ZSTD_compressionParameters cParams = ZSTD_getCParamsFromCCtxParams( + @params, + unchecked(0UL - 1), + 0, + ZSTD_CParamMode_e.ZSTD_cpm_noAttachDict + ); + ZSTD_paramSwitch_e useRowMatchFinder = ZSTD_resolveRowMatchFinderMode( + @params->useRowMatchFinder, + &cParams + ); + if (@params->nbWorkers > 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + + return ZSTD_estimateCCtxSize_usingCCtxParams_internal( + &cParams, + &@params->ldmParams, + 1, + useRowMatchFinder, + 0, + 0, + unchecked(0UL - 1), + ZSTD_hasExtSeqProd(@params), + @params->maxBlockSize + ); + } + + public static nuint ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams) + { + ZSTD_CCtx_params_s initialParams = ZSTD_makeCCtxParamsFromCParams(cParams); + if (ZSTD_rowMatchFinderSupported(cParams.strategy) != 0) + { + /* Pick bigger of not using and using row-based matchfinder for greedy and lazy strategies */ + nuint noRowCCtxSize; + nuint rowCCtxSize; + initialParams.useRowMatchFinder = ZSTD_paramSwitch_e.ZSTD_ps_disable; + noRowCCtxSize = ZSTD_estimateCCtxSize_usingCCtxParams(&initialParams); + initialParams.useRowMatchFinder = ZSTD_paramSwitch_e.ZSTD_ps_enable; + rowCCtxSize = ZSTD_estimateCCtxSize_usingCCtxParams(&initialParams); + return noRowCCtxSize > rowCCtxSize ? noRowCCtxSize : rowCCtxSize; + } + else + { + return ZSTD_estimateCCtxSize_usingCCtxParams(&initialParams); + } + } + +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_srcSizeTiers => + new ulong[4] { 16 * (1 << 10), 128 * (1 << 10), 256 * (1 << 10), unchecked(0UL - 1) }; + private static ulong* srcSizeTiers => + (ulong*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_srcSizeTiers) + ); +#else + + private static readonly ulong* srcSizeTiers = GetArrayPointer( + new ulong[4] + { + (ulong)(16 * (1 << 10)), + (ulong)(128 * (1 << 10)), + (ulong)(256 * (1 << 10)), + (unchecked(0UL - 1)), + } + ); +#endif + + private static nuint ZSTD_estimateCCtxSize_internal(int compressionLevel) + { + int tier = 0; + nuint largestSize = 0; + for (; tier < 4; ++tier) + { + /* Choose the set of cParams for a given level across all srcSizes that give the largest cctxSize */ + ZSTD_compressionParameters cParams = ZSTD_getCParams_internal( + compressionLevel, + srcSizeTiers[tier], + 0, + ZSTD_CParamMode_e.ZSTD_cpm_noAttachDict + ); + largestSize = + ZSTD_estimateCCtxSize_usingCParams(cParams) > largestSize + ? ZSTD_estimateCCtxSize_usingCParams(cParams) + : largestSize; + } + + return largestSize; + } + + /*! ZSTD_estimate*() : + * These functions make it possible to estimate memory usage + * of a future {D,C}Ctx, before its creation. + * This is useful in combination with ZSTD_initStatic(), + * which makes it possible to employ a static buffer for ZSTD_CCtx* state. + * + * ZSTD_estimateCCtxSize() will provide a memory budget large enough + * to compress data of any size using one-shot compression ZSTD_compressCCtx() or ZSTD_compress2() + * associated with any compression level up to max specified one. + * The estimate will assume the input may be arbitrarily large, + * which is the worst case. + * + * Note that the size estimation is specific for one-shot compression, + * it is not valid for streaming (see ZSTD_estimateCStreamSize*()) + * nor other potential ways of using a ZSTD_CCtx* state. + * + * When srcSize can be bound by a known and rather "small" value, + * this knowledge can be used to provide a tighter budget estimation + * because the ZSTD_CCtx* state will need less memory for small inputs. + * This tighter estimation can be provided by employing more advanced functions + * ZSTD_estimateCCtxSize_usingCParams(), which can be used in tandem with ZSTD_getCParams(), + * and ZSTD_estimateCCtxSize_usingCCtxParams(), which can be used in tandem with ZSTD_CCtxParams_setParameter(). + * Both can be used to estimate memory using custom compression parameters and arbitrary srcSize limits. + * + * Note : only single-threaded compression is supported. + * ZSTD_estimateCCtxSize_usingCCtxParams() will return an error code if ZSTD_c_nbWorkers is >= 1. + */ + public static nuint ZSTD_estimateCCtxSize(int compressionLevel) + { + int level; + nuint memBudget = 0; + for ( + level = compressionLevel < 1 ? compressionLevel : 1; + level <= compressionLevel; + level++ + ) + { + /* Ensure monotonically increasing memory usage as compression level increases */ + nuint newMB = ZSTD_estimateCCtxSize_internal(level); + if (newMB > memBudget) + { + memBudget = newMB; + } + } + + return memBudget; + } + + public static nuint ZSTD_estimateCStreamSize_usingCCtxParams(ZSTD_CCtx_params_s* @params) + { + if (@params->nbWorkers > 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + + { + ZSTD_compressionParameters cParams = ZSTD_getCParamsFromCCtxParams( + @params, + unchecked(0UL - 1), + 0, + ZSTD_CParamMode_e.ZSTD_cpm_noAttachDict + ); + nuint blockSize = + ZSTD_resolveMaxBlockSize(@params->maxBlockSize) < (nuint)1 << (int)cParams.windowLog + ? ZSTD_resolveMaxBlockSize(@params->maxBlockSize) + : (nuint)1 << (int)cParams.windowLog; + nuint inBuffSize = + @params->inBufferMode == ZSTD_bufferMode_e.ZSTD_bm_buffered + ? ((nuint)1 << (int)cParams.windowLog) + blockSize + : 0; + nuint outBuffSize = + @params->outBufferMode == ZSTD_bufferMode_e.ZSTD_bm_buffered + ? ZSTD_compressBound(blockSize) + 1 + : 0; + ZSTD_paramSwitch_e useRowMatchFinder = ZSTD_resolveRowMatchFinderMode( + @params->useRowMatchFinder, + &@params->cParams + ); + return ZSTD_estimateCCtxSize_usingCCtxParams_internal( + &cParams, + &@params->ldmParams, + 1, + useRowMatchFinder, + inBuffSize, + outBuffSize, + unchecked(0UL - 1), + ZSTD_hasExtSeqProd(@params), + @params->maxBlockSize + ); + } + } + + public static nuint ZSTD_estimateCStreamSize_usingCParams(ZSTD_compressionParameters cParams) + { + ZSTD_CCtx_params_s initialParams = ZSTD_makeCCtxParamsFromCParams(cParams); + if (ZSTD_rowMatchFinderSupported(cParams.strategy) != 0) + { + /* Pick bigger of not using and using row-based matchfinder for greedy and lazy strategies */ + nuint noRowCCtxSize; + nuint rowCCtxSize; + initialParams.useRowMatchFinder = ZSTD_paramSwitch_e.ZSTD_ps_disable; + noRowCCtxSize = ZSTD_estimateCStreamSize_usingCCtxParams(&initialParams); + initialParams.useRowMatchFinder = ZSTD_paramSwitch_e.ZSTD_ps_enable; + rowCCtxSize = ZSTD_estimateCStreamSize_usingCCtxParams(&initialParams); + return noRowCCtxSize > rowCCtxSize ? noRowCCtxSize : rowCCtxSize; + } + else + { + return ZSTD_estimateCStreamSize_usingCCtxParams(&initialParams); + } + } + + private static nuint ZSTD_estimateCStreamSize_internal(int compressionLevel) + { + ZSTD_compressionParameters cParams = ZSTD_getCParams_internal( + compressionLevel, + unchecked(0UL - 1), + 0, + ZSTD_CParamMode_e.ZSTD_cpm_noAttachDict + ); + return ZSTD_estimateCStreamSize_usingCParams(cParams); + } + + /*! ZSTD_estimateCStreamSize() : + * ZSTD_estimateCStreamSize() will provide a memory budget large enough for streaming compression + * using any compression level up to the max specified one. + * It will also consider src size to be arbitrarily "large", which is a worst case scenario. + * If srcSize is known to always be small, ZSTD_estimateCStreamSize_usingCParams() can provide a tighter estimation. + * ZSTD_estimateCStreamSize_usingCParams() can be used in tandem with ZSTD_getCParams() to create cParams from compressionLevel. + * ZSTD_estimateCStreamSize_usingCCtxParams() can be used in tandem with ZSTD_CCtxParams_setParameter(). Only single-threaded compression is supported. This function will return an error code if ZSTD_c_nbWorkers is >= 1. + * Note : CStream size estimation is only correct for single-threaded compression. + * ZSTD_estimateCStreamSize_usingCCtxParams() will return an error code if ZSTD_c_nbWorkers is >= 1. + * Note 2 : ZSTD_estimateCStreamSize* functions are not compatible with the Block-Level Sequence Producer API at this time. + * Size estimates assume that no external sequence producer is registered. + * + * ZSTD_DStream memory budget depends on frame's window Size. + * This information can be passed manually, using ZSTD_estimateDStreamSize, + * or deducted from a valid frame Header, using ZSTD_estimateDStreamSize_fromFrame(); + * Any frame requesting a window size larger than max specified one will be rejected. + * Note : if streaming is init with function ZSTD_init?Stream_usingDict(), + * an internal ?Dict will be created, which additional size is not estimated here. + * In this case, get total size by adding ZSTD_estimate?DictSize + */ + public static nuint ZSTD_estimateCStreamSize(int compressionLevel) + { + int level; + nuint memBudget = 0; + for ( + level = compressionLevel < 1 ? compressionLevel : 1; + level <= compressionLevel; + level++ + ) + { + nuint newMB = ZSTD_estimateCStreamSize_internal(level); + if (newMB > memBudget) + { + memBudget = newMB; + } + } + + return memBudget; + } + + /* ZSTD_getFrameProgression(): + * tells how much data has been consumed (input) and produced (output) for current frame. + * able to count progression inside worker threads (non-blocking mode). + */ + public static ZSTD_frameProgression ZSTD_getFrameProgression(ZSTD_CCtx_s* cctx) + { + if (cctx->appliedParams.nbWorkers > 0) + { + return ZSTDMT_getFrameProgression(cctx->mtctx); + } + + { + ZSTD_frameProgression fp; + nuint buffered = cctx->inBuff == null ? 0 : cctx->inBuffPos - cctx->inToCompress; + assert(buffered <= 1 << 17); + fp.ingested = cctx->consumedSrcSize + buffered; + fp.consumed = cctx->consumedSrcSize; + fp.produced = cctx->producedCSize; + fp.flushed = cctx->producedCSize; + fp.currentJobID = 0; + fp.nbActiveWorkers = 0; + return fp; + } + } + + /*! ZSTD_toFlushNow() + * Only useful for multithreading scenarios currently (nbWorkers >= 1). + */ + public static nuint ZSTD_toFlushNow(ZSTD_CCtx_s* cctx) + { + if (cctx->appliedParams.nbWorkers > 0) + { + return ZSTDMT_toFlushNow(cctx->mtctx); + } + + return 0; + } + + private static void ZSTD_assertEqualCParams( + ZSTD_compressionParameters cParams1, + ZSTD_compressionParameters cParams2 + ) + { + assert(cParams1.windowLog == cParams2.windowLog); + assert(cParams1.chainLog == cParams2.chainLog); + assert(cParams1.hashLog == cParams2.hashLog); + assert(cParams1.searchLog == cParams2.searchLog); + assert(cParams1.minMatch == cParams2.minMatch); + assert(cParams1.targetLength == cParams2.targetLength); + assert(cParams1.strategy == cParams2.strategy); + } + + private static void ZSTD_reset_compressedBlockState(ZSTD_compressedBlockState_t* bs) + { + int i; + for (i = 0; i < 3; ++i) + { + bs->rep[i] = repStartValue[i]; + } + + bs->entropy.huf.repeatMode = HUF_repeat.HUF_repeat_none; + bs->entropy.fse.offcode_repeatMode = FSE_repeat.FSE_repeat_none; + bs->entropy.fse.matchlength_repeatMode = FSE_repeat.FSE_repeat_none; + bs->entropy.fse.litlength_repeatMode = FSE_repeat.FSE_repeat_none; + } + + /*! ZSTD_invalidateMatchState() + * Invalidate all the matches in the match finder tables. + * Requires nextSrc and base to be set (can be NULL). + */ + private static void ZSTD_invalidateMatchState(ZSTD_MatchState_t* ms) + { + ZSTD_window_clear(&ms->window); + ms->nextToUpdate = ms->window.dictLimit; + ms->loadedDictEnd = 0; + ms->opt.litLengthSum = 0; + ms->dictMatchState = null; + } + + /* Mixes bits in a 64 bits in a value, based on XXH3_rrmxmx */ + private static ulong ZSTD_bitmix(ulong val, ulong len) + { + val ^= BitOperations.RotateRight(val, 49) ^ BitOperations.RotateRight(val, 24); + val *= 0x9FB21C651E98DF25UL; + val ^= (val >> 35) + len; + val *= 0x9FB21C651E98DF25UL; + return val ^ val >> 28; + } + + /* Mixes in the hashSalt and hashSaltEntropy to create a new hashSalt */ + private static void ZSTD_advanceHashSalt(ZSTD_MatchState_t* ms) + { + ms->hashSalt = ZSTD_bitmix(ms->hashSalt, 8) ^ ZSTD_bitmix(ms->hashSaltEntropy, 4); + } + + private static nuint ZSTD_reset_matchState( + ZSTD_MatchState_t* ms, + ZSTD_cwksp* ws, + ZSTD_compressionParameters* cParams, + ZSTD_paramSwitch_e useRowMatchFinder, + ZSTD_compResetPolicy_e crp, + ZSTD_indexResetPolicy_e forceResetIndex, + ZSTD_resetTarget_e forWho + ) + { + /* disable chain table allocation for fast or row-based strategies */ + nuint chainSize = + ZSTD_allocateChainTable( + cParams->strategy, + useRowMatchFinder, + ms->dedicatedDictSearch != 0 && forWho == ZSTD_resetTarget_e.ZSTD_resetTarget_CDict + ? 1U + : 0U + ) != 0 + ? (nuint)1 << (int)cParams->chainLog + : 0; + nuint hSize = (nuint)1 << (int)cParams->hashLog; + uint hashLog3 = + forWho == ZSTD_resetTarget_e.ZSTD_resetTarget_CCtx && cParams->minMatch == 3 + ? 17 < cParams->windowLog + ? 17 + : cParams->windowLog + : 0; + nuint h3Size = hashLog3 != 0 ? (nuint)1 << (int)hashLog3 : 0; + assert(useRowMatchFinder != ZSTD_paramSwitch_e.ZSTD_ps_auto); + if (forceResetIndex == ZSTD_indexResetPolicy_e.ZSTDirp_reset) + { + ZSTD_window_init(&ms->window); + ZSTD_cwksp_mark_tables_dirty(ws); + } + + ms->hashLog3 = hashLog3; + ms->lazySkipping = 0; + ZSTD_invalidateMatchState(ms); + assert(ZSTD_cwksp_reserve_failed(ws) == 0); + ZSTD_cwksp_clear_tables(ws); + ms->hashTable = (uint*)ZSTD_cwksp_reserve_table(ws, hSize * sizeof(uint)); + ms->chainTable = (uint*)ZSTD_cwksp_reserve_table(ws, chainSize * sizeof(uint)); + ms->hashTable3 = (uint*)ZSTD_cwksp_reserve_table(ws, h3Size * sizeof(uint)); + if (ZSTD_cwksp_reserve_failed(ws) != 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + + if (crp != ZSTD_compResetPolicy_e.ZSTDcrp_leaveDirty) + { + ZSTD_cwksp_clean_tables(ws); + } + + if (ZSTD_rowMatchFinderUsed(cParams->strategy, useRowMatchFinder) != 0) + { + /* Row match finder needs an additional table of hashes ("tags") */ + nuint tagTableSize = hSize; + if (forWho == ZSTD_resetTarget_e.ZSTD_resetTarget_CCtx) + { + ms->tagTable = (byte*)ZSTD_cwksp_reserve_aligned_init_once(ws, tagTableSize); + ZSTD_advanceHashSalt(ms); + } + else + { + ms->tagTable = (byte*)ZSTD_cwksp_reserve_aligned64(ws, tagTableSize); + memset(ms->tagTable, 0, (uint)tagTableSize); + ms->hashSalt = 0; + } + + { + uint rowLog = + cParams->searchLog <= 4 ? 4 + : cParams->searchLog <= 6 ? cParams->searchLog + : 6; + assert(cParams->hashLog >= rowLog); + ms->rowHashLog = cParams->hashLog - rowLog; + } + } + + if ( + forWho == ZSTD_resetTarget_e.ZSTD_resetTarget_CCtx + && cParams->strategy >= ZSTD_strategy.ZSTD_btopt + ) + { + ms->opt.litFreq = (uint*)ZSTD_cwksp_reserve_aligned64(ws, (1 << 8) * sizeof(uint)); + ms->opt.litLengthFreq = (uint*)ZSTD_cwksp_reserve_aligned64( + ws, + (35 + 1) * sizeof(uint) + ); + ms->opt.matchLengthFreq = (uint*)ZSTD_cwksp_reserve_aligned64( + ws, + (52 + 1) * sizeof(uint) + ); + ms->opt.offCodeFreq = (uint*)ZSTD_cwksp_reserve_aligned64(ws, (31 + 1) * sizeof(uint)); + ms->opt.matchTable = (ZSTD_match_t*)ZSTD_cwksp_reserve_aligned64( + ws, + (nuint)(((1 << 12) + 3) * sizeof(ZSTD_match_t)) + ); + ms->opt.priceTable = (ZSTD_optimal_t*)ZSTD_cwksp_reserve_aligned64( + ws, + (nuint)(((1 << 12) + 3) * sizeof(ZSTD_optimal_t)) + ); + } + + ms->cParams = *cParams; + if (ZSTD_cwksp_reserve_failed(ws) != 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + + return 0; + } + + private static int ZSTD_indexTooCloseToMax(ZSTD_window_t w) + { + return + (nuint)(w.nextSrc - w.@base) + > (MEM_64bits ? 3500U * (1 << 20) : 2000U * (1 << 20)) - 16 * (1 << 20) + ? 1 + : 0; + } + + /** ZSTD_dictTooBig(): + * When dictionaries are larger than ZSTD_CHUNKSIZE_MAX they can't be loaded in + * one go generically. So we ensure that in that case we reset the tables to zero, + * so that we can load as much of the dictionary as possible. + */ + private static int ZSTD_dictTooBig(nuint loadedDictSize) + { + return + loadedDictSize + > unchecked((uint)-1) - (MEM_64bits ? 3500U * (1 << 20) : 2000U * (1 << 20)) + ? 1 + : 0; + } + + /*! ZSTD_resetCCtx_internal() : + * @param loadedDictSize The size of the dictionary to be loaded + * into the context, if any. If no dictionary is used, or the + * dictionary is being attached / copied, then pass 0. + * note : `params` are assumed fully validated at this stage. + */ + private static nuint ZSTD_resetCCtx_internal( + ZSTD_CCtx_s* zc, + ZSTD_CCtx_params_s* @params, + ulong pledgedSrcSize, + nuint loadedDictSize, + ZSTD_compResetPolicy_e crp, + ZSTD_buffered_policy_e zbuff + ) + { + ZSTD_cwksp* ws = &zc->workspace; + assert(!ERR_isError(ZSTD_checkCParams(@params->cParams))); + zc->isFirstBlock = 1; + zc->appliedParams = *@params; + @params = &zc->appliedParams; + assert(@params->useRowMatchFinder != ZSTD_paramSwitch_e.ZSTD_ps_auto); + assert(@params->postBlockSplitter != ZSTD_paramSwitch_e.ZSTD_ps_auto); + assert(@params->ldmParams.enableLdm != ZSTD_paramSwitch_e.ZSTD_ps_auto); + assert(@params->maxBlockSize != 0); + if (@params->ldmParams.enableLdm == ZSTD_paramSwitch_e.ZSTD_ps_enable) + { + ZSTD_ldm_adjustParameters(&zc->appliedParams.ldmParams, &@params->cParams); + assert(@params->ldmParams.hashLog >= @params->ldmParams.bucketSizeLog); + assert(@params->ldmParams.hashRateLog < 32); + } + + { + nuint windowSize = + 1 + > (nuint)( + (ulong)1 << (int)@params->cParams.windowLog < pledgedSrcSize + ? (ulong)1 << (int)@params->cParams.windowLog + : pledgedSrcSize + ) + ? 1 + : (nuint)( + (ulong)1 << (int)@params->cParams.windowLog < pledgedSrcSize + ? (ulong)1 << (int)@params->cParams.windowLog + : pledgedSrcSize + ); + nuint blockSize = + @params->maxBlockSize < windowSize ? @params->maxBlockSize : windowSize; + nuint maxNbSeq = ZSTD_maxNbSeq( + blockSize, + @params->cParams.minMatch, + ZSTD_hasExtSeqProd(@params) + ); + nuint buffOutSize = + zbuff == ZSTD_buffered_policy_e.ZSTDb_buffered + && @params->outBufferMode == ZSTD_bufferMode_e.ZSTD_bm_buffered + ? ZSTD_compressBound(blockSize) + 1 + : 0; + nuint buffInSize = + zbuff == ZSTD_buffered_policy_e.ZSTDb_buffered + && @params->inBufferMode == ZSTD_bufferMode_e.ZSTD_bm_buffered + ? windowSize + blockSize + : 0; + nuint maxNbLdmSeq = ZSTD_ldm_getMaxNbSeq(@params->ldmParams, blockSize); + int indexTooClose = ZSTD_indexTooCloseToMax(zc->blockState.matchState.window); + int dictTooBig = ZSTD_dictTooBig(loadedDictSize); + ZSTD_indexResetPolicy_e needsIndexReset = + indexTooClose != 0 || dictTooBig != 0 || zc->initialized == 0 + ? ZSTD_indexResetPolicy_e.ZSTDirp_reset + : ZSTD_indexResetPolicy_e.ZSTDirp_continue; + nuint neededSpace = ZSTD_estimateCCtxSize_usingCCtxParams_internal( + &@params->cParams, + &@params->ldmParams, + zc->staticSize != 0 ? 1 : 0, + @params->useRowMatchFinder, + buffInSize, + buffOutSize, + pledgedSrcSize, + ZSTD_hasExtSeqProd(@params), + @params->maxBlockSize + ); + { + nuint err_code = neededSpace; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (zc->staticSize == 0) + { + ZSTD_cwksp_bump_oversized_duration(ws, 0); + } + + { + int workspaceTooSmall = ZSTD_cwksp_sizeof(ws) < neededSpace ? 1 : 0; + int workspaceWasteful = ZSTD_cwksp_check_wasteful(ws, neededSpace); + int resizeWorkspace = workspaceTooSmall != 0 || workspaceWasteful != 0 ? 1 : 0; + if (resizeWorkspace != 0) + { + if (zc->staticSize != 0) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation) + ); + } + + needsIndexReset = ZSTD_indexResetPolicy_e.ZSTDirp_reset; + ZSTD_cwksp_free(ws, zc->customMem); + { + nuint err_code = ZSTD_cwksp_create(ws, neededSpace, zc->customMem); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + assert( + ZSTD_cwksp_check_available( + ws, + (nuint)(2 * sizeof(ZSTD_compressedBlockState_t)) + ) != 0 + ); + zc->blockState.prevCBlock = + (ZSTD_compressedBlockState_t*)ZSTD_cwksp_reserve_object( + ws, + (nuint)sizeof(ZSTD_compressedBlockState_t) + ); + if (zc->blockState.prevCBlock == null) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation) + ); + } + + zc->blockState.nextCBlock = + (ZSTD_compressedBlockState_t*)ZSTD_cwksp_reserve_object( + ws, + (nuint)sizeof(ZSTD_compressedBlockState_t) + ); + if (zc->blockState.nextCBlock == null) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation) + ); + } + + zc->tmpWorkspace = ZSTD_cwksp_reserve_object( + ws, + (8 << 10) + 512 + sizeof(uint) * (52 + 2) > 8208 + ? (8 << 10) + 512 + sizeof(uint) * (52 + 2) + : 8208 + ); + if (zc->tmpWorkspace == null) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation) + ); + } + + zc->tmpWkspSize = + (8 << 10) + 512 + sizeof(uint) * (52 + 2) > 8208 + ? (8 << 10) + 512 + sizeof(uint) * (52 + 2) + : 8208; + } + } + + ZSTD_cwksp_clear(ws); + zc->blockState.matchState.cParams = @params->cParams; + zc->blockState.matchState.prefetchCDictTables = + @params->prefetchCDictTables == ZSTD_paramSwitch_e.ZSTD_ps_enable ? 1 : 0; + zc->pledgedSrcSizePlusOne = pledgedSrcSize + 1; + zc->consumedSrcSize = 0; + zc->producedCSize = 0; + if (pledgedSrcSize == unchecked(0UL - 1)) + { + zc->appliedParams.fParams.contentSizeFlag = 0; + } + + zc->blockSizeMax = blockSize; + ZSTD_XXH64_reset(&zc->xxhState, 0); + zc->stage = ZSTD_compressionStage_e.ZSTDcs_init; + zc->dictID = 0; + zc->dictContentSize = 0; + ZSTD_reset_compressedBlockState(zc->blockState.prevCBlock); + { + nuint err_code = ZSTD_reset_matchState( + &zc->blockState.matchState, + ws, + &@params->cParams, + @params->useRowMatchFinder, + crp, + needsIndexReset, + ZSTD_resetTarget_e.ZSTD_resetTarget_CCtx + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + zc->seqStore.sequencesStart = (SeqDef_s*)ZSTD_cwksp_reserve_aligned64( + ws, + maxNbSeq * (nuint)sizeof(SeqDef_s) + ); + if (@params->ldmParams.enableLdm == ZSTD_paramSwitch_e.ZSTD_ps_enable) + { + /* TODO: avoid memset? */ + nuint ldmHSize = (nuint)1 << (int)@params->ldmParams.hashLog; + zc->ldmState.hashTable = (ldmEntry_t*)ZSTD_cwksp_reserve_aligned64( + ws, + ldmHSize * (nuint)sizeof(ldmEntry_t) + ); + memset(zc->ldmState.hashTable, 0, (uint)(ldmHSize * (nuint)sizeof(ldmEntry_t))); + zc->ldmSequences = (rawSeq*)ZSTD_cwksp_reserve_aligned64( + ws, + maxNbLdmSeq * (nuint)sizeof(rawSeq) + ); + zc->maxNbLdmSequences = maxNbLdmSeq; + ZSTD_window_init(&zc->ldmState.window); + zc->ldmState.loadedDictEnd = 0; + } + + if (ZSTD_hasExtSeqProd(@params) != 0) + { + nuint maxNbExternalSeq = ZSTD_sequenceBound(blockSize); + zc->extSeqBufCapacity = maxNbExternalSeq; + zc->extSeqBuf = (ZSTD_Sequence*)ZSTD_cwksp_reserve_aligned64( + ws, + maxNbExternalSeq * (nuint)sizeof(ZSTD_Sequence) + ); + } + + zc->seqStore.litStart = ZSTD_cwksp_reserve_buffer(ws, blockSize + 32); + zc->seqStore.maxNbLit = blockSize; + zc->bufferedPolicy = zbuff; + zc->inBuffSize = buffInSize; + zc->inBuff = (sbyte*)ZSTD_cwksp_reserve_buffer(ws, buffInSize); + zc->outBuffSize = buffOutSize; + zc->outBuff = (sbyte*)ZSTD_cwksp_reserve_buffer(ws, buffOutSize); + if (@params->ldmParams.enableLdm == ZSTD_paramSwitch_e.ZSTD_ps_enable) + { + /* TODO: avoid memset? */ + nuint numBuckets = + (nuint)1 + << (int)(@params->ldmParams.hashLog - @params->ldmParams.bucketSizeLog); + zc->ldmState.bucketOffsets = ZSTD_cwksp_reserve_buffer(ws, numBuckets); + memset(zc->ldmState.bucketOffsets, 0, (uint)numBuckets); + } + + ZSTD_referenceExternalSequences(zc, null, 0); + zc->seqStore.maxNbSeq = maxNbSeq; + zc->seqStore.llCode = ZSTD_cwksp_reserve_buffer(ws, maxNbSeq * sizeof(byte)); + zc->seqStore.mlCode = ZSTD_cwksp_reserve_buffer(ws, maxNbSeq * sizeof(byte)); + zc->seqStore.ofCode = ZSTD_cwksp_reserve_buffer(ws, maxNbSeq * sizeof(byte)); + assert(ZSTD_cwksp_estimated_space_within_bounds(ws, neededSpace) != 0); + zc->initialized = 1; + return 0; + } + } + + /* ZSTD_invalidateRepCodes() : + * ensures next compression will not use repcodes from previous block. + * Note : only works with regular variant; + * do not use with extDict variant ! */ + private static void ZSTD_invalidateRepCodes(ZSTD_CCtx_s* cctx) + { + int i; + for (i = 0; i < 3; i++) + { + cctx->blockState.prevCBlock->rep[i] = 0; + } + + assert(ZSTD_window_hasExtDict(cctx->blockState.matchState.window) == 0); + } + + private static readonly nuint* attachDictSizeCutoffs = GetArrayPointer( + new nuint[10] + { + 8 * (1 << 10), + 8 * (1 << 10), + 16 * (1 << 10), + 32 * (1 << 10), + 32 * (1 << 10), + 32 * (1 << 10), + 32 * (1 << 10), + 32 * (1 << 10), + 8 * (1 << 10), + 8 * (1 << 10), + } + ); + + private static int ZSTD_shouldAttachDict( + ZSTD_CDict_s* cdict, + ZSTD_CCtx_params_s* @params, + ulong pledgedSrcSize + ) + { + nuint cutoff = attachDictSizeCutoffs[(int)cdict->matchState.cParams.strategy]; + int dedicatedDictSearch = cdict->matchState.dedicatedDictSearch; + return + dedicatedDictSearch != 0 + || ( + pledgedSrcSize <= cutoff + || pledgedSrcSize == unchecked(0UL - 1) + || @params->attachDictPref == ZSTD_dictAttachPref_e.ZSTD_dictForceAttach + ) + && @params->attachDictPref != ZSTD_dictAttachPref_e.ZSTD_dictForceCopy + && @params->forceWindow == 0 + ? 1 + : 0; + } + + private static nuint ZSTD_resetCCtx_byAttachingCDict( + ZSTD_CCtx_s* cctx, + ZSTD_CDict_s* cdict, + ZSTD_CCtx_params_s @params, + ulong pledgedSrcSize, + ZSTD_buffered_policy_e zbuff + ) + { + { + ZSTD_compressionParameters adjusted_cdict_cParams = cdict->matchState.cParams; + uint windowLog = @params.cParams.windowLog; + assert(windowLog != 0); + if (cdict->matchState.dedicatedDictSearch != 0) + { + ZSTD_dedicatedDictSearch_revertCParams(&adjusted_cdict_cParams); + } + + @params.cParams = ZSTD_adjustCParams_internal( + adjusted_cdict_cParams, + pledgedSrcSize, + cdict->dictContentSize, + ZSTD_CParamMode_e.ZSTD_cpm_attachDict, + @params.useRowMatchFinder + ); + @params.cParams.windowLog = windowLog; + @params.useRowMatchFinder = cdict->useRowMatchFinder; + { + nuint err_code = ZSTD_resetCCtx_internal( + cctx, + &@params, + pledgedSrcSize, + 0, + ZSTD_compResetPolicy_e.ZSTDcrp_makeClean, + zbuff + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + assert(cctx->appliedParams.cParams.strategy == adjusted_cdict_cParams.strategy); + } + + { + uint cdictEnd = (uint)( + cdict->matchState.window.nextSrc - cdict->matchState.window.@base + ); + uint cdictLen = cdictEnd - cdict->matchState.window.dictLimit; + if (cdictLen != 0) + { + cctx->blockState.matchState.dictMatchState = &cdict->matchState; + if (cctx->blockState.matchState.window.dictLimit < cdictEnd) + { + cctx->blockState.matchState.window.nextSrc = + cctx->blockState.matchState.window.@base + cdictEnd; + ZSTD_window_clear(&cctx->blockState.matchState.window); + } + + cctx->blockState.matchState.loadedDictEnd = cctx->blockState + .matchState + .window + .dictLimit; + } + } + + cctx->dictID = cdict->dictID; + cctx->dictContentSize = cdict->dictContentSize; + memcpy( + cctx->blockState.prevCBlock, + &cdict->cBlockState, + (uint)sizeof(ZSTD_compressedBlockState_t) + ); + return 0; + } + + private static void ZSTD_copyCDictTableIntoCCtx( + uint* dst, + uint* src, + nuint tableSize, + ZSTD_compressionParameters* cParams + ) + { + if (ZSTD_CDictIndicesAreTagged(cParams) != 0) + { + /* Remove tags from the CDict table if they are present. + * See docs on "short cache" in zstd_compress_internal.h for context. */ + nuint i; + for (i = 0; i < tableSize; i++) + { + uint taggedIndex = src[i]; + uint index = taggedIndex >> 8; + dst[i] = index; + } + } + else + { + memcpy(dst, src, (uint)(tableSize * sizeof(uint))); + } + } + + private static nuint ZSTD_resetCCtx_byCopyingCDict( + ZSTD_CCtx_s* cctx, + ZSTD_CDict_s* cdict, + ZSTD_CCtx_params_s @params, + ulong pledgedSrcSize, + ZSTD_buffered_policy_e zbuff + ) + { + ZSTD_compressionParameters* cdict_cParams = &cdict->matchState.cParams; + assert(cdict->matchState.dedicatedDictSearch == 0); + { + uint windowLog = @params.cParams.windowLog; + assert(windowLog != 0); + @params.cParams = *cdict_cParams; + @params.cParams.windowLog = windowLog; + @params.useRowMatchFinder = cdict->useRowMatchFinder; + { + nuint err_code = ZSTD_resetCCtx_internal( + cctx, + &@params, + pledgedSrcSize, + 0, + ZSTD_compResetPolicy_e.ZSTDcrp_leaveDirty, + zbuff + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + assert(cctx->appliedParams.cParams.strategy == cdict_cParams->strategy); + assert(cctx->appliedParams.cParams.hashLog == cdict_cParams->hashLog); + assert(cctx->appliedParams.cParams.chainLog == cdict_cParams->chainLog); + } + + ZSTD_cwksp_mark_tables_dirty(&cctx->workspace); + assert(@params.useRowMatchFinder != ZSTD_paramSwitch_e.ZSTD_ps_auto); + { + /* DDS guaranteed disabled */ + nuint chainSize = + ZSTD_allocateChainTable(cdict_cParams->strategy, cdict->useRowMatchFinder, 0) != 0 + ? (nuint)1 << (int)cdict_cParams->chainLog + : 0; + nuint hSize = (nuint)1 << (int)cdict_cParams->hashLog; + ZSTD_copyCDictTableIntoCCtx( + cctx->blockState.matchState.hashTable, + cdict->matchState.hashTable, + hSize, + cdict_cParams + ); + if ( + ZSTD_allocateChainTable( + cctx->appliedParams.cParams.strategy, + cctx->appliedParams.useRowMatchFinder, + 0 + ) != 0 + ) + { + ZSTD_copyCDictTableIntoCCtx( + cctx->blockState.matchState.chainTable, + cdict->matchState.chainTable, + chainSize, + cdict_cParams + ); + } + + if (ZSTD_rowMatchFinderUsed(cdict_cParams->strategy, cdict->useRowMatchFinder) != 0) + { + nuint tagTableSize = hSize; + memcpy( + cctx->blockState.matchState.tagTable, + cdict->matchState.tagTable, + (uint)tagTableSize + ); + cctx->blockState.matchState.hashSalt = cdict->matchState.hashSalt; + } + } + + assert(cctx->blockState.matchState.hashLog3 <= 31); + { + uint h3log = cctx->blockState.matchState.hashLog3; + nuint h3Size = h3log != 0 ? (nuint)1 << (int)h3log : 0; + assert(cdict->matchState.hashLog3 == 0); + memset(cctx->blockState.matchState.hashTable3, 0, (uint)(h3Size * sizeof(uint))); + } + + ZSTD_cwksp_mark_tables_clean(&cctx->workspace); + { + ZSTD_MatchState_t* srcMatchState = &cdict->matchState; + ZSTD_MatchState_t* dstMatchState = &cctx->blockState.matchState; + dstMatchState->window = srcMatchState->window; + dstMatchState->nextToUpdate = srcMatchState->nextToUpdate; + dstMatchState->loadedDictEnd = srcMatchState->loadedDictEnd; + } + + cctx->dictID = cdict->dictID; + cctx->dictContentSize = cdict->dictContentSize; + memcpy( + cctx->blockState.prevCBlock, + &cdict->cBlockState, + (uint)sizeof(ZSTD_compressedBlockState_t) + ); + return 0; + } + + /* We have a choice between copying the dictionary context into the working + * context, or referencing the dictionary context from the working context + * in-place. We decide here which strategy to use. */ + private static nuint ZSTD_resetCCtx_usingCDict( + ZSTD_CCtx_s* cctx, + ZSTD_CDict_s* cdict, + ZSTD_CCtx_params_s* @params, + ulong pledgedSrcSize, + ZSTD_buffered_policy_e zbuff + ) + { + if (ZSTD_shouldAttachDict(cdict, @params, pledgedSrcSize) != 0) + { + return ZSTD_resetCCtx_byAttachingCDict(cctx, cdict, *@params, pledgedSrcSize, zbuff); + } + else + { + return ZSTD_resetCCtx_byCopyingCDict(cctx, cdict, *@params, pledgedSrcSize, zbuff); + } + } + + /*! ZSTD_copyCCtx_internal() : + * Duplicate an existing context `srcCCtx` into another one `dstCCtx`. + * Only works during stage ZSTDcs_init (i.e. after creation, but before first call to ZSTD_compressContinue()). + * The "context", in this case, refers to the hash and chain tables, + * entropy tables, and dictionary references. + * `windowLog` value is enforced if != 0, otherwise value is copied from srcCCtx. + * @return : 0, or an error code */ + private static nuint ZSTD_copyCCtx_internal( + ZSTD_CCtx_s* dstCCtx, + ZSTD_CCtx_s* srcCCtx, + ZSTD_frameParameters fParams, + ulong pledgedSrcSize, + ZSTD_buffered_policy_e zbuff + ) + { + if (srcCCtx->stage != ZSTD_compressionStage_e.ZSTDcs_init) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_stage_wrong)); + } + + memcpy(&dstCCtx->customMem, &srcCCtx->customMem, (uint)sizeof(ZSTD_customMem)); + { + ZSTD_CCtx_params_s @params = dstCCtx->requestedParams; + @params.cParams = srcCCtx->appliedParams.cParams; + assert(srcCCtx->appliedParams.useRowMatchFinder != ZSTD_paramSwitch_e.ZSTD_ps_auto); + assert(srcCCtx->appliedParams.postBlockSplitter != ZSTD_paramSwitch_e.ZSTD_ps_auto); + assert(srcCCtx->appliedParams.ldmParams.enableLdm != ZSTD_paramSwitch_e.ZSTD_ps_auto); + @params.useRowMatchFinder = srcCCtx->appliedParams.useRowMatchFinder; + @params.postBlockSplitter = srcCCtx->appliedParams.postBlockSplitter; + @params.ldmParams = srcCCtx->appliedParams.ldmParams; + @params.fParams = fParams; + @params.maxBlockSize = srcCCtx->appliedParams.maxBlockSize; + ZSTD_resetCCtx_internal( + dstCCtx, + &@params, + pledgedSrcSize, + 0, + ZSTD_compResetPolicy_e.ZSTDcrp_leaveDirty, + zbuff + ); + assert( + dstCCtx->appliedParams.cParams.windowLog == srcCCtx->appliedParams.cParams.windowLog + ); + assert( + dstCCtx->appliedParams.cParams.strategy == srcCCtx->appliedParams.cParams.strategy + ); + assert( + dstCCtx->appliedParams.cParams.hashLog == srcCCtx->appliedParams.cParams.hashLog + ); + assert( + dstCCtx->appliedParams.cParams.chainLog == srcCCtx->appliedParams.cParams.chainLog + ); + assert( + dstCCtx->blockState.matchState.hashLog3 == srcCCtx->blockState.matchState.hashLog3 + ); + } + + ZSTD_cwksp_mark_tables_dirty(&dstCCtx->workspace); + { + nuint chainSize = + ZSTD_allocateChainTable( + srcCCtx->appliedParams.cParams.strategy, + srcCCtx->appliedParams.useRowMatchFinder, + 0 + ) != 0 + ? (nuint)1 << (int)srcCCtx->appliedParams.cParams.chainLog + : 0; + nuint hSize = (nuint)1 << (int)srcCCtx->appliedParams.cParams.hashLog; + uint h3log = srcCCtx->blockState.matchState.hashLog3; + nuint h3Size = h3log != 0 ? (nuint)1 << (int)h3log : 0; + memcpy( + dstCCtx->blockState.matchState.hashTable, + srcCCtx->blockState.matchState.hashTable, + (uint)(hSize * sizeof(uint)) + ); + memcpy( + dstCCtx->blockState.matchState.chainTable, + srcCCtx->blockState.matchState.chainTable, + (uint)(chainSize * sizeof(uint)) + ); + memcpy( + dstCCtx->blockState.matchState.hashTable3, + srcCCtx->blockState.matchState.hashTable3, + (uint)(h3Size * sizeof(uint)) + ); + } + + ZSTD_cwksp_mark_tables_clean(&dstCCtx->workspace); + { + ZSTD_MatchState_t* srcMatchState = &srcCCtx->blockState.matchState; + ZSTD_MatchState_t* dstMatchState = &dstCCtx->blockState.matchState; + dstMatchState->window = srcMatchState->window; + dstMatchState->nextToUpdate = srcMatchState->nextToUpdate; + dstMatchState->loadedDictEnd = srcMatchState->loadedDictEnd; + } + + dstCCtx->dictID = srcCCtx->dictID; + dstCCtx->dictContentSize = srcCCtx->dictContentSize; + memcpy( + dstCCtx->blockState.prevCBlock, + srcCCtx->blockState.prevCBlock, + (uint)sizeof(ZSTD_compressedBlockState_t) + ); + return 0; + } + + /*! ZSTD_copyCCtx() : + * Duplicate an existing context `srcCCtx` into another one `dstCCtx`. + * Only works during stage ZSTDcs_init (i.e. after creation, but before first call to ZSTD_compressContinue()). + * pledgedSrcSize==0 means "unknown". + * @return : 0, or an error code */ + public static nuint ZSTD_copyCCtx( + ZSTD_CCtx_s* dstCCtx, + ZSTD_CCtx_s* srcCCtx, + ulong pledgedSrcSize + ) + { + /*content*/ + ZSTD_frameParameters fParams = new ZSTD_frameParameters + { + contentSizeFlag = 1, + checksumFlag = 0, + noDictIDFlag = 0, + }; + ZSTD_buffered_policy_e zbuff = srcCCtx->bufferedPolicy; + if (pledgedSrcSize == 0) + { + pledgedSrcSize = unchecked(0UL - 1); + } + + fParams.contentSizeFlag = pledgedSrcSize != unchecked(0UL - 1) ? 1 : 0; + return ZSTD_copyCCtx_internal(dstCCtx, srcCCtx, fParams, pledgedSrcSize, zbuff); + } + + /*! ZSTD_reduceTable() : + * reduce table indexes by `reducerValue`, or squash to zero. + * PreserveMark preserves "unsorted mark" for btlazy2 strategy. + * It must be set to a clear 0/1 value, to remove branch during inlining. + * Presume table size is a multiple of ZSTD_ROWSIZE + * to help auto-vectorization */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_reduceTable_internal( + uint* table, + uint size, + uint reducerValue, + int preserveMark + ) + { + int nbRows = (int)size / 16; + int cellNb = 0; + int rowNb; + /* Protect special index values < ZSTD_WINDOW_START_INDEX. */ + uint reducerThreshold = reducerValue + 2; + assert((size & 16 - 1) == 0); + assert(size < 1U << 31); + for (rowNb = 0; rowNb < nbRows; rowNb++) + { + int column; + for (column = 0; column < 16; column++) + { + uint newVal; + if (preserveMark != 0 && table[cellNb] == 1) + { + newVal = 1; + } + else if (table[cellNb] < reducerThreshold) + { + newVal = 0; + } + else + { + newVal = table[cellNb] - reducerValue; + } + + table[cellNb] = newVal; + cellNb++; + } + } + } + + private static void ZSTD_reduceTable(uint* table, uint size, uint reducerValue) + { + ZSTD_reduceTable_internal(table, size, reducerValue, 0); + } + + private static void ZSTD_reduceTable_btlazy2(uint* table, uint size, uint reducerValue) + { + ZSTD_reduceTable_internal(table, size, reducerValue, 1); + } + + /*! ZSTD_reduceIndex() : + * rescale all indexes to avoid future overflow (indexes are U32) */ + private static void ZSTD_reduceIndex( + ZSTD_MatchState_t* ms, + ZSTD_CCtx_params_s* @params, + uint reducerValue + ) + { + { + uint hSize = (uint)1 << (int)@params->cParams.hashLog; + ZSTD_reduceTable(ms->hashTable, hSize, reducerValue); + } + + if ( + ZSTD_allocateChainTable( + @params->cParams.strategy, + @params->useRowMatchFinder, + (uint)ms->dedicatedDictSearch + ) != 0 + ) + { + uint chainSize = (uint)1 << (int)@params->cParams.chainLog; + if (@params->cParams.strategy == ZSTD_strategy.ZSTD_btlazy2) + { + ZSTD_reduceTable_btlazy2(ms->chainTable, chainSize, reducerValue); + } + else + { + ZSTD_reduceTable(ms->chainTable, chainSize, reducerValue); + } + } + + if (ms->hashLog3 != 0) + { + uint h3Size = (uint)1 << (int)ms->hashLog3; + ZSTD_reduceTable(ms->hashTable3, h3Size, reducerValue); + } + } + + /* See doc/zstd_compression_format.md for detailed format description */ + private static int ZSTD_seqToCodes(SeqStore_t* seqStorePtr) + { + SeqDef_s* sequences = seqStorePtr->sequencesStart; + byte* llCodeTable = seqStorePtr->llCode; + byte* ofCodeTable = seqStorePtr->ofCode; + byte* mlCodeTable = seqStorePtr->mlCode; + uint nbSeq = (uint)(seqStorePtr->sequences - seqStorePtr->sequencesStart); + uint u; + int longOffsets = 0; + assert(nbSeq <= seqStorePtr->maxNbSeq); + for (u = 0; u < nbSeq; u++) + { + uint llv = sequences[u].litLength; + uint ofCode = ZSTD_highbit32(sequences[u].offBase); + uint mlv = sequences[u].mlBase; + llCodeTable[u] = (byte)ZSTD_LLcode(llv); + ofCodeTable[u] = (byte)ofCode; + mlCodeTable[u] = (byte)ZSTD_MLcode(mlv); + assert(!(MEM_64bits && ofCode >= (uint)(MEM_32bits ? 25 : 57))); + if (MEM_32bits && ofCode >= (uint)(MEM_32bits ? 25 : 57)) + { + longOffsets = 1; + } + } + + if (seqStorePtr->longLengthType == ZSTD_longLengthType_e.ZSTD_llt_literalLength) + { + llCodeTable[seqStorePtr->longLengthPos] = 35; + } + + if (seqStorePtr->longLengthType == ZSTD_longLengthType_e.ZSTD_llt_matchLength) + { + mlCodeTable[seqStorePtr->longLengthPos] = 52; + } + + return longOffsets; + } + + /* ZSTD_useTargetCBlockSize(): + * Returns if target compressed block size param is being used. + * If used, compression will do best effort to make a compressed block size to be around targetCBlockSize. + * Returns 1 if true, 0 otherwise. */ + private static int ZSTD_useTargetCBlockSize(ZSTD_CCtx_params_s* cctxParams) + { + return cctxParams->targetCBlockSize != 0 ? 1 : 0; + } + + /* ZSTD_blockSplitterEnabled(): + * Returns if block splitting param is being used + * If used, compression will do best effort to split a block in order to improve compression ratio. + * At the time this function is called, the parameter must be finalized. + * Returns 1 if true, 0 otherwise. */ + private static int ZSTD_blockSplitterEnabled(ZSTD_CCtx_params_s* cctxParams) + { + assert(cctxParams->postBlockSplitter != ZSTD_paramSwitch_e.ZSTD_ps_auto); + return cctxParams->postBlockSplitter == ZSTD_paramSwitch_e.ZSTD_ps_enable ? 1 : 0; + } + + /* ZSTD_buildSequencesStatistics(): + * Returns a ZSTD_symbolEncodingTypeStats_t, or a zstd error code in the `size` field. + * Modifies `nextEntropy` to have the appropriate values as a side effect. + * nbSeq must be greater than 0. + * + * entropyWkspSize must be of size at least ENTROPY_WORKSPACE_SIZE - (MaxSeq + 1)*sizeof(U32) + */ + private static ZSTD_symbolEncodingTypeStats_t ZSTD_buildSequencesStatistics( + SeqStore_t* seqStorePtr, + nuint nbSeq, + ZSTD_fseCTables_t* prevEntropy, + ZSTD_fseCTables_t* nextEntropy, + byte* dst, + byte* dstEnd, + ZSTD_strategy strategy, + uint* countWorkspace, + void* entropyWorkspace, + nuint entropyWkspSize + ) + { + byte* ostart = dst; + byte* oend = dstEnd; + byte* op = ostart; + uint* CTable_LitLength = nextEntropy->litlengthCTable; + uint* CTable_OffsetBits = nextEntropy->offcodeCTable; + uint* CTable_MatchLength = nextEntropy->matchlengthCTable; + byte* ofCodeTable = seqStorePtr->ofCode; + byte* llCodeTable = seqStorePtr->llCode; + byte* mlCodeTable = seqStorePtr->mlCode; + ZSTD_symbolEncodingTypeStats_t stats; + System.Runtime.CompilerServices.Unsafe.SkipInit(out stats); + stats.lastCountSize = 0; + stats.longOffsets = ZSTD_seqToCodes(seqStorePtr); + assert(op <= oend); + assert(nbSeq != 0); + { + uint max = 35; + /* can't fail */ + nuint mostFrequent = HIST_countFast_wksp( + countWorkspace, + &max, + llCodeTable, + nbSeq, + entropyWorkspace, + entropyWkspSize + ); + nextEntropy->litlength_repeatMode = prevEntropy->litlength_repeatMode; + stats.LLtype = (uint)ZSTD_selectEncodingType( + &nextEntropy->litlength_repeatMode, + countWorkspace, + max, + mostFrequent, + nbSeq, + 9, + prevEntropy->litlengthCTable, + LL_defaultNorm, + LL_defaultNormLog, + ZSTD_DefaultPolicy_e.ZSTD_defaultAllowed, + strategy + ); + assert( + SymbolEncodingType_e.set_basic < SymbolEncodingType_e.set_compressed + && SymbolEncodingType_e.set_rle < SymbolEncodingType_e.set_compressed + ); + assert( + !( + stats.LLtype < (uint)SymbolEncodingType_e.set_compressed + && nextEntropy->litlength_repeatMode != FSE_repeat.FSE_repeat_none + ) + ); + { + nuint countSize = ZSTD_buildCTable( + op, + (nuint)(oend - op), + CTable_LitLength, + 9, + (SymbolEncodingType_e)stats.LLtype, + countWorkspace, + max, + llCodeTable, + nbSeq, + LL_defaultNorm, + LL_defaultNormLog, + 35, + prevEntropy->litlengthCTable, + sizeof(uint) * 329, + entropyWorkspace, + entropyWkspSize + ); + if (ERR_isError(countSize)) + { + stats.size = countSize; + return stats; + } + + if (stats.LLtype == (uint)SymbolEncodingType_e.set_compressed) + { + stats.lastCountSize = countSize; + } + + op += countSize; + assert(op <= oend); + } + } + + { + uint max = 31; + nuint mostFrequent = HIST_countFast_wksp( + countWorkspace, + &max, + ofCodeTable, + nbSeq, + entropyWorkspace, + entropyWkspSize + ); + /* We can only use the basic table if max <= DefaultMaxOff, otherwise the offsets are too large */ + ZSTD_DefaultPolicy_e defaultPolicy = + max <= 28 + ? ZSTD_DefaultPolicy_e.ZSTD_defaultAllowed + : ZSTD_DefaultPolicy_e.ZSTD_defaultDisallowed; + nextEntropy->offcode_repeatMode = prevEntropy->offcode_repeatMode; + stats.Offtype = (uint)ZSTD_selectEncodingType( + &nextEntropy->offcode_repeatMode, + countWorkspace, + max, + mostFrequent, + nbSeq, + 8, + prevEntropy->offcodeCTable, + OF_defaultNorm, + OF_defaultNormLog, + defaultPolicy, + strategy + ); + assert( + !( + stats.Offtype < (uint)SymbolEncodingType_e.set_compressed + && nextEntropy->offcode_repeatMode != FSE_repeat.FSE_repeat_none + ) + ); + { + nuint countSize = ZSTD_buildCTable( + op, + (nuint)(oend - op), + CTable_OffsetBits, + 8, + (SymbolEncodingType_e)stats.Offtype, + countWorkspace, + max, + ofCodeTable, + nbSeq, + OF_defaultNorm, + OF_defaultNormLog, + 28, + prevEntropy->offcodeCTable, + sizeof(uint) * 193, + entropyWorkspace, + entropyWkspSize + ); + if (ERR_isError(countSize)) + { + stats.size = countSize; + return stats; + } + + if (stats.Offtype == (uint)SymbolEncodingType_e.set_compressed) + { + stats.lastCountSize = countSize; + } + + op += countSize; + assert(op <= oend); + } + } + + { + uint max = 52; + nuint mostFrequent = HIST_countFast_wksp( + countWorkspace, + &max, + mlCodeTable, + nbSeq, + entropyWorkspace, + entropyWkspSize + ); + nextEntropy->matchlength_repeatMode = prevEntropy->matchlength_repeatMode; + stats.MLtype = (uint)ZSTD_selectEncodingType( + &nextEntropy->matchlength_repeatMode, + countWorkspace, + max, + mostFrequent, + nbSeq, + 9, + prevEntropy->matchlengthCTable, + ML_defaultNorm, + ML_defaultNormLog, + ZSTD_DefaultPolicy_e.ZSTD_defaultAllowed, + strategy + ); + assert( + !( + stats.MLtype < (uint)SymbolEncodingType_e.set_compressed + && nextEntropy->matchlength_repeatMode != FSE_repeat.FSE_repeat_none + ) + ); + { + nuint countSize = ZSTD_buildCTable( + op, + (nuint)(oend - op), + CTable_MatchLength, + 9, + (SymbolEncodingType_e)stats.MLtype, + countWorkspace, + max, + mlCodeTable, + nbSeq, + ML_defaultNorm, + ML_defaultNormLog, + 52, + prevEntropy->matchlengthCTable, + sizeof(uint) * 363, + entropyWorkspace, + entropyWkspSize + ); + if (ERR_isError(countSize)) + { + stats.size = countSize; + return stats; + } + + if (stats.MLtype == (uint)SymbolEncodingType_e.set_compressed) + { + stats.lastCountSize = countSize; + } + + op += countSize; + assert(op <= oend); + } + } + + stats.size = (nuint)(op - ostart); + return stats; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_entropyCompressSeqStore_internal( + void* dst, + nuint dstCapacity, + void* literals, + nuint litSize, + SeqStore_t* seqStorePtr, + ZSTD_entropyCTables_t* prevEntropy, + ZSTD_entropyCTables_t* nextEntropy, + ZSTD_CCtx_params_s* cctxParams, + void* entropyWorkspace, + nuint entropyWkspSize, + int bmi2 + ) + { + ZSTD_strategy strategy = cctxParams->cParams.strategy; + uint* count = (uint*)entropyWorkspace; + uint* CTable_LitLength = nextEntropy->fse.litlengthCTable; + uint* CTable_OffsetBits = nextEntropy->fse.offcodeCTable; + uint* CTable_MatchLength = nextEntropy->fse.matchlengthCTable; + SeqDef_s* sequences = seqStorePtr->sequencesStart; + nuint nbSeq = (nuint)(seqStorePtr->sequences - seqStorePtr->sequencesStart); + byte* ofCodeTable = seqStorePtr->ofCode; + byte* llCodeTable = seqStorePtr->llCode; + byte* mlCodeTable = seqStorePtr->mlCode; + byte* ostart = (byte*)dst; + byte* oend = ostart + dstCapacity; + byte* op = ostart; + nuint lastCountSize; + int longOffsets = 0; + entropyWorkspace = count + (52 + 1); + entropyWkspSize -= (52 + 1) * sizeof(uint); + assert(entropyWkspSize >= (8 << 10) + 512); + { + nuint numSequences = (nuint)(seqStorePtr->sequences - seqStorePtr->sequencesStart); + /* Base suspicion of uncompressibility on ratio of literals to sequences */ + int suspectUncompressible = numSequences == 0 || litSize / numSequences >= 20 ? 1 : 0; + nuint cSize = ZSTD_compressLiterals( + op, + dstCapacity, + literals, + litSize, + entropyWorkspace, + entropyWkspSize, + &prevEntropy->huf, + &nextEntropy->huf, + cctxParams->cParams.strategy, + ZSTD_literalsCompressionIsDisabled(cctxParams), + suspectUncompressible, + bmi2 + ); + { + nuint err_code = cSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + assert(cSize <= dstCapacity); + op += cSize; + } + + if (oend - op < 3 + 1) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + if (nbSeq < 128) + { + *op++ = (byte)nbSeq; + } + else if (nbSeq < 0x7F00) + { + op[0] = (byte)((nbSeq >> 8) + 0x80); + op[1] = (byte)nbSeq; + op += 2; + } + else + { + op[0] = 0xFF; + MEM_writeLE16(op + 1, (ushort)(nbSeq - 0x7F00)); + op += 3; + } + + assert(op <= oend); + if (nbSeq == 0) + { + memcpy(&nextEntropy->fse, &prevEntropy->fse, (uint)sizeof(ZSTD_fseCTables_t)); + return (nuint)(op - ostart); + } + + { + byte* seqHead = op++; + /* build stats for sequences */ + ZSTD_symbolEncodingTypeStats_t stats = ZSTD_buildSequencesStatistics( + seqStorePtr, + nbSeq, + &prevEntropy->fse, + &nextEntropy->fse, + op, + oend, + strategy, + count, + entropyWorkspace, + entropyWkspSize + ); + { + nuint err_code = stats.size; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + *seqHead = (byte)((stats.LLtype << 6) + (stats.Offtype << 4) + (stats.MLtype << 2)); + lastCountSize = stats.lastCountSize; + op += stats.size; + longOffsets = stats.longOffsets; + } + + { + nuint bitstreamSize = ZSTD_encodeSequences( + op, + (nuint)(oend - op), + CTable_MatchLength, + mlCodeTable, + CTable_OffsetBits, + ofCodeTable, + CTable_LitLength, + llCodeTable, + sequences, + nbSeq, + longOffsets, + bmi2 + ); + { + nuint err_code = bitstreamSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + op += bitstreamSize; + assert(op <= oend); + if (lastCountSize != 0 && lastCountSize + bitstreamSize < 4) + { + assert(lastCountSize + bitstreamSize == 3); + return 0; + } + } + + return (nuint)(op - ostart); + } + + private static nuint ZSTD_entropyCompressSeqStore_wExtLitBuffer( + void* dst, + nuint dstCapacity, + void* literals, + nuint litSize, + nuint blockSize, + SeqStore_t* seqStorePtr, + ZSTD_entropyCTables_t* prevEntropy, + ZSTD_entropyCTables_t* nextEntropy, + ZSTD_CCtx_params_s* cctxParams, + void* entropyWorkspace, + nuint entropyWkspSize, + int bmi2 + ) + { + nuint cSize = ZSTD_entropyCompressSeqStore_internal( + dst, + dstCapacity, + literals, + litSize, + seqStorePtr, + prevEntropy, + nextEntropy, + cctxParams, + entropyWorkspace, + entropyWkspSize, + bmi2 + ); + if (cSize == 0) + { + return 0; + } + + if ( + cSize == unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)) + && blockSize <= dstCapacity + ) + { + return 0; + } + + { + nuint err_code = cSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint maxCSize = blockSize - ZSTD_minGain(blockSize, cctxParams->cParams.strategy); + if (cSize >= maxCSize) + { + return 0; + } + } + + assert(cSize < 1 << 17); + return cSize; + } + + private static nuint ZSTD_entropyCompressSeqStore( + SeqStore_t* seqStorePtr, + ZSTD_entropyCTables_t* prevEntropy, + ZSTD_entropyCTables_t* nextEntropy, + ZSTD_CCtx_params_s* cctxParams, + void* dst, + nuint dstCapacity, + nuint srcSize, + void* entropyWorkspace, + nuint entropyWkspSize, + int bmi2 + ) + { + return ZSTD_entropyCompressSeqStore_wExtLitBuffer( + dst, + dstCapacity, + seqStorePtr->litStart, + (nuint)(seqStorePtr->lit - seqStorePtr->litStart), + srcSize, + seqStorePtr, + prevEntropy, + nextEntropy, + cctxParams, + entropyWorkspace, + entropyWkspSize, + bmi2 + ); + } + + private static readonly ZSTD_BlockCompressor_f?[][] blockCompressor = + new ZSTD_BlockCompressor_f?[4][] + { + new ZSTD_BlockCompressor_f[10] + { + ZSTD_compressBlock_fast, + ZSTD_compressBlock_fast, + ZSTD_compressBlock_doubleFast, + ZSTD_compressBlock_greedy, + ZSTD_compressBlock_lazy, + ZSTD_compressBlock_lazy2, + ZSTD_compressBlock_btlazy2, + ZSTD_compressBlock_btopt, + ZSTD_compressBlock_btultra, + ZSTD_compressBlock_btultra2, + }, + new ZSTD_BlockCompressor_f[10] + { + ZSTD_compressBlock_fast_extDict, + ZSTD_compressBlock_fast_extDict, + ZSTD_compressBlock_doubleFast_extDict, + ZSTD_compressBlock_greedy_extDict, + ZSTD_compressBlock_lazy_extDict, + ZSTD_compressBlock_lazy2_extDict, + ZSTD_compressBlock_btlazy2_extDict, + ZSTD_compressBlock_btopt_extDict, + ZSTD_compressBlock_btultra_extDict, + ZSTD_compressBlock_btultra_extDict, + }, + new ZSTD_BlockCompressor_f[10] + { + ZSTD_compressBlock_fast_dictMatchState, + ZSTD_compressBlock_fast_dictMatchState, + ZSTD_compressBlock_doubleFast_dictMatchState, + ZSTD_compressBlock_greedy_dictMatchState, + ZSTD_compressBlock_lazy_dictMatchState, + ZSTD_compressBlock_lazy2_dictMatchState, + ZSTD_compressBlock_btlazy2_dictMatchState, + ZSTD_compressBlock_btopt_dictMatchState, + ZSTD_compressBlock_btultra_dictMatchState, + ZSTD_compressBlock_btultra_dictMatchState, + }, + new ZSTD_BlockCompressor_f?[10] + { + null, + null, + null, + ZSTD_compressBlock_greedy_dedicatedDictSearch, + ZSTD_compressBlock_lazy_dedicatedDictSearch, + ZSTD_compressBlock_lazy2_dedicatedDictSearch, + null, + null, + null, + null, + }, + }; + private static readonly ZSTD_BlockCompressor_f[][] rowBasedBlockCompressors = + new ZSTD_BlockCompressor_f[4][] + { + new ZSTD_BlockCompressor_f[3] + { + ZSTD_compressBlock_greedy_row, + ZSTD_compressBlock_lazy_row, + ZSTD_compressBlock_lazy2_row, + }, + new ZSTD_BlockCompressor_f[3] + { + ZSTD_compressBlock_greedy_extDict_row, + ZSTD_compressBlock_lazy_extDict_row, + ZSTD_compressBlock_lazy2_extDict_row, + }, + new ZSTD_BlockCompressor_f[3] + { + ZSTD_compressBlock_greedy_dictMatchState_row, + ZSTD_compressBlock_lazy_dictMatchState_row, + ZSTD_compressBlock_lazy2_dictMatchState_row, + }, + new ZSTD_BlockCompressor_f[3] + { + ZSTD_compressBlock_greedy_dedicatedDictSearch_row, + ZSTD_compressBlock_lazy_dedicatedDictSearch_row, + ZSTD_compressBlock_lazy2_dedicatedDictSearch_row, + }, + }; + + /* ZSTD_selectBlockCompressor() : + * Not static, but internal use only (used by long distance matcher) + * assumption : strat is a valid strategy */ + private static ZSTD_BlockCompressor_f ZSTD_selectBlockCompressor( + ZSTD_strategy strat, + ZSTD_paramSwitch_e useRowMatchFinder, + ZSTD_dictMode_e dictMode + ) + { + ZSTD_BlockCompressor_f? selectedCompressor; + assert(ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_strategy, (int)strat) != 0); + if (ZSTD_rowMatchFinderUsed(strat, useRowMatchFinder) != 0) + { + assert(useRowMatchFinder != ZSTD_paramSwitch_e.ZSTD_ps_auto); + selectedCompressor = rowBasedBlockCompressors[(int)dictMode][ + (int)strat - (int)ZSTD_strategy.ZSTD_greedy + ]; + } + else + { + selectedCompressor = blockCompressor[(int)dictMode][(int)strat]; + } + + assert(selectedCompressor != null); + return selectedCompressor.NotNull(); + } + + private static void ZSTD_storeLastLiterals( + SeqStore_t* seqStorePtr, + byte* anchor, + nuint lastLLSize + ) + { + memcpy(seqStorePtr->lit, anchor, (uint)lastLLSize); + seqStorePtr->lit += lastLLSize; + } + + private static void ZSTD_resetSeqStore(SeqStore_t* ssPtr) + { + ssPtr->lit = ssPtr->litStart; + ssPtr->sequences = ssPtr->sequencesStart; + ssPtr->longLengthType = ZSTD_longLengthType_e.ZSTD_llt_none; + } + + /* ZSTD_postProcessSequenceProducerResult() : + * Validates and post-processes sequences obtained through the external matchfinder API: + * - Checks whether nbExternalSeqs represents an error condition. + * - Appends a block delimiter to outSeqs if one is not already present. + * See zstd.h for context regarding block delimiters. + * Returns the number of sequences after post-processing, or an error code. */ + private static nuint ZSTD_postProcessSequenceProducerResult( + ZSTD_Sequence* outSeqs, + nuint nbExternalSeqs, + nuint outSeqsCapacity, + nuint srcSize + ) + { + if (nbExternalSeqs > outSeqsCapacity) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_sequenceProducer_failed)); + } + + if (nbExternalSeqs == 0 && srcSize > 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_sequenceProducer_failed)); + } + + if (srcSize == 0) + { + outSeqs[0] = new ZSTD_Sequence(); + return 1; + } + + { + ZSTD_Sequence lastSeq = outSeqs[nbExternalSeqs - 1]; + if (lastSeq.offset == 0 && lastSeq.matchLength == 0) + { + return nbExternalSeqs; + } + + if (nbExternalSeqs == outSeqsCapacity) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_sequenceProducer_failed)); + } + + outSeqs[nbExternalSeqs] = new ZSTD_Sequence(); + return nbExternalSeqs + 1; + } + } + + /* ZSTD_fastSequenceLengthSum() : + * Returns sum(litLen) + sum(matchLen) + lastLits for *seqBuf*. + * Similar to another function in zstd_compress.c (determine_blockSize), + * except it doesn't check for a block delimiter to end summation. + * Removing the early exit allows the compiler to auto-vectorize (https://godbolt.org/z/cY1cajz9P). + * This function can be deleted and replaced by determine_blockSize after we resolve issue #3456. */ + private static nuint ZSTD_fastSequenceLengthSum(ZSTD_Sequence* seqBuf, nuint seqBufSize) + { + nuint matchLenSum, + litLenSum, + i; + matchLenSum = 0; + litLenSum = 0; + for (i = 0; i < seqBufSize; i++) + { + litLenSum += seqBuf[i].litLength; + matchLenSum += seqBuf[i].matchLength; + } + + return litLenSum + matchLenSum; + } + + /** + * Function to validate sequences produced by a block compressor. + */ + private static void ZSTD_validateSeqStore( + SeqStore_t* seqStore, + ZSTD_compressionParameters* cParams + ) { } + + private static nuint ZSTD_buildSeqStore(ZSTD_CCtx_s* zc, void* src, nuint srcSize) + { + ZSTD_MatchState_t* ms = &zc->blockState.matchState; + assert(srcSize <= 1 << 17); + ZSTD_assertEqualCParams(zc->appliedParams.cParams, ms->cParams); + if (srcSize < (nuint)(1 + 1) + ZSTD_blockHeaderSize + 1 + 1) + { + if (zc->appliedParams.cParams.strategy >= ZSTD_strategy.ZSTD_btopt) + { + ZSTD_ldm_skipRawSeqStoreBytes(&zc->externSeqStore, srcSize); + } + else + { + ZSTD_ldm_skipSequences( + &zc->externSeqStore, + srcSize, + zc->appliedParams.cParams.minMatch + ); + } + + return (nuint)ZSTD_BuildSeqStore_e.ZSTDbss_noCompress; + } + + ZSTD_resetSeqStore(&zc->seqStore); + ms->opt.symbolCosts = &zc->blockState.prevCBlock->entropy; + ms->opt.literalCompressionMode = zc->appliedParams.literalCompressionMode; + assert(ms->dictMatchState == null || ms->loadedDictEnd == ms->window.dictLimit); + { + byte* @base = ms->window.@base; + byte* istart = (byte*)src; + uint curr = (uint)(istart - @base); + if (curr > ms->nextToUpdate + 384) + { + ms->nextToUpdate = + curr + - (192 < curr - ms->nextToUpdate - 384 ? 192 : curr - ms->nextToUpdate - 384); + } + } + + { + ZSTD_dictMode_e dictMode = ZSTD_matchState_dictMode(ms); + nuint lastLLSize; + { + int i; + for (i = 0; i < 3; ++i) + { + zc->blockState.nextCBlock->rep[i] = zc->blockState.prevCBlock->rep[i]; + } + } + + if (zc->externSeqStore.pos < zc->externSeqStore.size) + { + assert(zc->appliedParams.ldmParams.enableLdm == ZSTD_paramSwitch_e.ZSTD_ps_disable); + if (ZSTD_hasExtSeqProd(&zc->appliedParams) != 0) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_combination_unsupported) + ); + } + + lastLLSize = ZSTD_ldm_blockCompress( + &zc->externSeqStore, + ms, + &zc->seqStore, + zc->blockState.nextCBlock->rep, + zc->appliedParams.useRowMatchFinder, + src, + srcSize + ); + assert(zc->externSeqStore.pos <= zc->externSeqStore.size); + } + else if (zc->appliedParams.ldmParams.enableLdm == ZSTD_paramSwitch_e.ZSTD_ps_enable) + { + RawSeqStore_t ldmSeqStore = kNullRawSeqStore; + if (ZSTD_hasExtSeqProd(&zc->appliedParams) != 0) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_combination_unsupported) + ); + } + + ldmSeqStore.seq = zc->ldmSequences; + ldmSeqStore.capacity = zc->maxNbLdmSequences; + { + /* Updates ldmSeqStore.size */ + nuint err_code = ZSTD_ldm_generateSequences( + &zc->ldmState, + &ldmSeqStore, + &zc->appliedParams.ldmParams, + src, + srcSize + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + lastLLSize = ZSTD_ldm_blockCompress( + &ldmSeqStore, + ms, + &zc->seqStore, + zc->blockState.nextCBlock->rep, + zc->appliedParams.useRowMatchFinder, + src, + srcSize + ); + assert(ldmSeqStore.pos == ldmSeqStore.size); + } + else if (ZSTD_hasExtSeqProd(&zc->appliedParams) != 0) + { + assert(zc->extSeqBufCapacity >= ZSTD_sequenceBound(srcSize)); + assert(zc->appliedParams.extSeqProdFunc != null); + { + uint windowSize = (uint)1 << (int)zc->appliedParams.cParams.windowLog; + nuint nbExternalSeqs = ( + (delegate* managed< + void*, + ZSTD_Sequence*, + nuint, + void*, + nuint, + void*, + nuint, + int, + nuint, + nuint>) + zc->appliedParams.extSeqProdFunc + )( + zc->appliedParams.extSeqProdState, + zc->extSeqBuf, + zc->extSeqBufCapacity, + src, + srcSize, + null, + 0, + zc->appliedParams.compressionLevel, + windowSize + ); + nuint nbPostProcessedSeqs = ZSTD_postProcessSequenceProducerResult( + zc->extSeqBuf, + nbExternalSeqs, + zc->extSeqBufCapacity, + srcSize + ); + if (!ERR_isError(nbPostProcessedSeqs)) + { + ZSTD_SequencePosition seqPos = new ZSTD_SequencePosition + { + idx = 0, + posInSequence = 0, + posInSrc = 0, + }; + nuint seqLenSum = ZSTD_fastSequenceLengthSum( + zc->extSeqBuf, + nbPostProcessedSeqs + ); + if (seqLenSum > srcSize) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_externalSequences_invalid) + ); + } + + { + nuint err_code = ZSTD_transferSequences_wBlockDelim( + zc, + &seqPos, + zc->extSeqBuf, + nbPostProcessedSeqs, + src, + srcSize, + zc->appliedParams.searchForExternalRepcodes + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + ms->ldmSeqStore = null; + return (nuint)ZSTD_BuildSeqStore_e.ZSTDbss_compress; + } + + if (zc->appliedParams.enableMatchFinderFallback == 0) + { + return nbPostProcessedSeqs; + } + + { + ZSTD_BlockCompressor_f blockCompressor = ZSTD_selectBlockCompressor( + zc->appliedParams.cParams.strategy, + zc->appliedParams.useRowMatchFinder, + dictMode + ); + ms->ldmSeqStore = null; + lastLLSize = blockCompressor( + ms, + &zc->seqStore, + zc->blockState.nextCBlock->rep, + src, + srcSize + ); + } + } + } + else + { + ZSTD_BlockCompressor_f blockCompressor = ZSTD_selectBlockCompressor( + zc->appliedParams.cParams.strategy, + zc->appliedParams.useRowMatchFinder, + dictMode + ); + ms->ldmSeqStore = null; + lastLLSize = blockCompressor( + ms, + &zc->seqStore, + zc->blockState.nextCBlock->rep, + src, + srcSize + ); + } + + { + byte* lastLiterals = (byte*)src + srcSize - lastLLSize; + ZSTD_storeLastLiterals(&zc->seqStore, lastLiterals, lastLLSize); + } + } + + ZSTD_validateSeqStore(&zc->seqStore, &zc->appliedParams.cParams); + return (nuint)ZSTD_BuildSeqStore_e.ZSTDbss_compress; + } + + private static nuint ZSTD_copyBlockSequences( + SeqCollector* seqCollector, + SeqStore_t* seqStore, + uint* prevRepcodes + ) + { + SeqDef_s* inSeqs = seqStore->sequencesStart; + nuint nbInSequences = (nuint)(seqStore->sequences - inSeqs); + nuint nbInLiterals = (nuint)(seqStore->lit - seqStore->litStart); + ZSTD_Sequence* outSeqs = + seqCollector->seqIndex == 0 + ? seqCollector->seqStart + : seqCollector->seqStart + seqCollector->seqIndex; + nuint nbOutSequences = nbInSequences + 1; + nuint nbOutLiterals = 0; + repcodes_s repcodes; + nuint i; + assert(seqCollector->seqIndex <= seqCollector->maxSequences); + if (nbOutSequences > seqCollector->maxSequences - seqCollector->seqIndex) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + memcpy(&repcodes, prevRepcodes, (uint)sizeof(repcodes_s)); + for (i = 0; i < nbInSequences; ++i) + { + uint rawOffset; + outSeqs[i].litLength = inSeqs[i].litLength; + outSeqs[i].matchLength = (uint)(inSeqs[i].mlBase + 3); + outSeqs[i].rep = 0; + if (i == seqStore->longLengthPos) + { + if (seqStore->longLengthType == ZSTD_longLengthType_e.ZSTD_llt_literalLength) + { + outSeqs[i].litLength += 0x10000; + } + else if (seqStore->longLengthType == ZSTD_longLengthType_e.ZSTD_llt_matchLength) + { + outSeqs[i].matchLength += 0x10000; + } + } + + if (1 <= inSeqs[i].offBase && inSeqs[i].offBase <= 3) + { + assert(1 <= inSeqs[i].offBase && inSeqs[i].offBase <= 3); + uint repcode = inSeqs[i].offBase; + assert(repcode > 0); + outSeqs[i].rep = repcode; + if (outSeqs[i].litLength != 0) + { + rawOffset = repcodes.rep[repcode - 1]; + } + else + { + if (repcode == 3) + { + assert(repcodes.rep[0] > 1); + rawOffset = repcodes.rep[0] - 1; + } + else + { + rawOffset = repcodes.rep[repcode]; + } + } + } + else + { + assert(inSeqs[i].offBase > 3); + rawOffset = inSeqs[i].offBase - 3; + } + + outSeqs[i].offset = rawOffset; + ZSTD_updateRep(repcodes.rep, inSeqs[i].offBase, inSeqs[i].litLength == 0 ? 1U : 0U); + nbOutLiterals += outSeqs[i].litLength; + } + + assert(nbInLiterals >= nbOutLiterals); + { + nuint lastLLSize = nbInLiterals - nbOutLiterals; + outSeqs[nbInSequences].litLength = (uint)lastLLSize; + outSeqs[nbInSequences].matchLength = 0; + outSeqs[nbInSequences].offset = 0; + assert(nbOutSequences == nbInSequences + 1); + } + + seqCollector->seqIndex += nbOutSequences; + assert(seqCollector->seqIndex <= seqCollector->maxSequences); + return 0; + } + + /*! ZSTD_sequenceBound() : + * `srcSize` : size of the input buffer + * @return : upper-bound for the number of sequences that can be generated + * from a buffer of srcSize bytes + * + * note : returns number of sequences - to get bytes, multiply by sizeof(ZSTD_Sequence). + */ + public static nuint ZSTD_sequenceBound(nuint srcSize) + { + nuint maxNbSeq = srcSize / 3 + 1; + nuint maxNbDelims = srcSize / (1 << 10) + 1; + return maxNbSeq + maxNbDelims; + } + + /*! ZSTD_generateSequences() : + * WARNING: This function is meant for debugging and informational purposes ONLY! + * Its implementation is flawed, and it will be deleted in a future version. + * It is not guaranteed to succeed, as there are several cases where it will give + * up and fail. You should NOT use this function in production code. + * + * This function is deprecated, and will be removed in a future version. + * + * Generate sequences using ZSTD_compress2(), given a source buffer. + * + * @param zc The compression context to be used for ZSTD_compress2(). Set any + * compression parameters you need on this context. + * @param outSeqs The output sequences buffer of size @p outSeqsSize + * @param outSeqsCapacity The size of the output sequences buffer. + * ZSTD_sequenceBound(srcSize) is an upper bound on the number + * of sequences that can be generated. + * @param src The source buffer to generate sequences from of size @p srcSize. + * @param srcSize The size of the source buffer. + * + * Each block will end with a dummy sequence + * with offset == 0, matchLength == 0, and litLength == length of last literals. + * litLength may be == 0, and if so, then the sequence of (of: 0 ml: 0 ll: 0) + * simply acts as a block delimiter. + * + * @returns The number of sequences generated, necessarily less than + * ZSTD_sequenceBound(srcSize), or an error code that can be checked + * with ZSTD_isError(). + */ + public static nuint ZSTD_generateSequences( + ZSTD_CCtx_s* zc, + ZSTD_Sequence* outSeqs, + nuint outSeqsSize, + void* src, + nuint srcSize + ) + { + nuint dstCapacity = ZSTD_compressBound(srcSize); + /* Make C90 happy. */ + void* dst; + SeqCollector seqCollector; + { + int targetCBlockSize; + { + nuint err_code = ZSTD_CCtx_getParameter( + zc, + ZSTD_cParameter.ZSTD_c_targetCBlockSize, + &targetCBlockSize + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (targetCBlockSize != 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_unsupported)); + } + } + + { + int nbWorkers; + { + nuint err_code = ZSTD_CCtx_getParameter( + zc, + ZSTD_cParameter.ZSTD_c_nbWorkers, + &nbWorkers + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (nbWorkers != 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_unsupported)); + } + } + + dst = ZSTD_customMalloc(dstCapacity, ZSTD_defaultCMem); + if (dst == null) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + + seqCollector.collectSequences = 1; + seqCollector.seqStart = outSeqs; + seqCollector.seqIndex = 0; + seqCollector.maxSequences = outSeqsSize; + zc->seqCollector = seqCollector; + { + nuint ret = ZSTD_compress2(zc, dst, dstCapacity, src, srcSize); + ZSTD_customFree(dst, ZSTD_defaultCMem); + { + nuint err_code = ret; + if (ERR_isError(err_code)) + { + return err_code; + } + } + } + + assert(zc->seqCollector.seqIndex <= ZSTD_sequenceBound(srcSize)); + return zc->seqCollector.seqIndex; + } + + /*! ZSTD_mergeBlockDelimiters() : + * Given an array of ZSTD_Sequence, remove all sequences that represent block delimiters/last literals + * by merging them into the literals of the next sequence. + * + * As such, the final generated result has no explicit representation of block boundaries, + * and the final last literals segment is not represented in the sequences. + * + * The output of this function can be fed into ZSTD_compressSequences() with CCtx + * setting of ZSTD_c_blockDelimiters as ZSTD_sf_noBlockDelimiters + * @return : number of sequences left after merging + */ + public static nuint ZSTD_mergeBlockDelimiters(ZSTD_Sequence* sequences, nuint seqsSize) + { + nuint @in = 0; + nuint @out = 0; + for (; @in < seqsSize; ++@in) + { + if (sequences[@in].offset == 0 && sequences[@in].matchLength == 0) + { + if (@in != seqsSize - 1) + { + sequences[@in + 1].litLength += sequences[@in].litLength; + } + } + else + { + sequences[@out] = sequences[@in]; + ++@out; + } + } + + return @out; + } + + /* Unrolled loop to read four size_ts of input at a time. Returns 1 if is RLE, 0 if not. */ + private static int ZSTD_isRLE(byte* src, nuint length) + { + byte* ip = src; + byte value = ip[0]; + nuint valueST = (nuint)(value * 0x0101010101010101UL); + nuint unrollSize = (nuint)(sizeof(nuint) * 4); + nuint unrollMask = unrollSize - 1; + nuint prefixLength = length & unrollMask; + nuint i; + if (length == 1) + { + return 1; + } + + if (prefixLength != 0 && ZSTD_count(ip + 1, ip, ip + prefixLength) != prefixLength - 1) + { + return 0; + } + + for (i = prefixLength; i != length; i += unrollSize) + { + nuint u; + for (u = 0; u < unrollSize; u += (nuint)sizeof(nuint)) + { + if (MEM_readST(ip + i + u) != valueST) + { + return 0; + } + } + } + + return 1; + } + + /* Returns true if the given block may be RLE. + * This is just a heuristic based on the compressibility. + * It may return both false positives and false negatives. + */ + private static int ZSTD_maybeRLE(SeqStore_t* seqStore) + { + nuint nbSeqs = (nuint)(seqStore->sequences - seqStore->sequencesStart); + nuint nbLits = (nuint)(seqStore->lit - seqStore->litStart); + return nbSeqs < 4 && nbLits < 10 ? 1 : 0; + } + + private static void ZSTD_blockState_confirmRepcodesAndEntropyTables(ZSTD_blockState_t* bs) + { + ZSTD_compressedBlockState_t* tmp = bs->prevCBlock; + bs->prevCBlock = bs->nextCBlock; + bs->nextCBlock = tmp; + } + + /* Writes the block header */ + private static void writeBlockHeader(void* op, nuint cSize, nuint blockSize, uint lastBlock) + { + uint cBlockHeader = + cSize == 1 + ? lastBlock + ((uint)blockType_e.bt_rle << 1) + (uint)(blockSize << 3) + : lastBlock + ((uint)blockType_e.bt_compressed << 1) + (uint)(cSize << 3); + MEM_writeLE24(op, cBlockHeader); + } + + /** ZSTD_buildBlockEntropyStats_literals() : + * Builds entropy for the literals. + * Stores literals block type (raw, rle, compressed, repeat) and + * huffman description table to hufMetadata. + * Requires ENTROPY_WORKSPACE_SIZE workspace + * @return : size of huffman description table, or an error code + */ + private static nuint ZSTD_buildBlockEntropyStats_literals( + void* src, + nuint srcSize, + ZSTD_hufCTables_t* prevHuf, + ZSTD_hufCTables_t* nextHuf, + ZSTD_hufCTablesMetadata_t* hufMetadata, + int literalsCompressionIsDisabled, + void* workspace, + nuint wkspSize, + int hufFlags + ) + { + byte* wkspStart = (byte*)workspace; + byte* wkspEnd = wkspStart + wkspSize; + byte* countWkspStart = wkspStart; + uint* countWksp = (uint*)workspace; + const nuint countWkspSize = (255 + 1) * sizeof(uint); + byte* nodeWksp = countWkspStart + countWkspSize; + nuint nodeWkspSize = (nuint)(wkspEnd - nodeWksp); + uint maxSymbolValue = 255; + uint huffLog = 11; + HUF_repeat repeat = prevHuf->repeatMode; + memcpy(nextHuf, prevHuf, (uint)sizeof(ZSTD_hufCTables_t)); + if (literalsCompressionIsDisabled != 0) + { + hufMetadata->hType = SymbolEncodingType_e.set_basic; + return 0; + } + + { + nuint minLitSize = (nuint)(prevHuf->repeatMode == HUF_repeat.HUF_repeat_valid ? 6 : 63); + if (srcSize <= minLitSize) + { + hufMetadata->hType = SymbolEncodingType_e.set_basic; + return 0; + } + } + + { + nuint largest = HIST_count_wksp( + countWksp, + &maxSymbolValue, + (byte*)src, + srcSize, + workspace, + wkspSize + ); + { + nuint err_code = largest; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (largest == srcSize) + { + hufMetadata->hType = SymbolEncodingType_e.set_rle; + return 0; + } + + if (largest <= (srcSize >> 7) + 4) + { + hufMetadata->hType = SymbolEncodingType_e.set_basic; + return 0; + } + } + + if ( + repeat == HUF_repeat.HUF_repeat_check + && HUF_validateCTable(&prevHuf->CTable.e0, countWksp, maxSymbolValue) == 0 + ) + { + repeat = HUF_repeat.HUF_repeat_none; + } + + memset(&nextHuf->CTable.e0, 0, sizeof(ulong) * 257); + huffLog = HUF_optimalTableLog( + huffLog, + srcSize, + maxSymbolValue, + nodeWksp, + nodeWkspSize, + &nextHuf->CTable.e0, + countWksp, + hufFlags + ); + assert(huffLog <= 11); + { + nuint maxBits = HUF_buildCTable_wksp( + &nextHuf->CTable.e0, + countWksp, + maxSymbolValue, + huffLog, + nodeWksp, + nodeWkspSize + ); + { + nuint err_code = maxBits; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + huffLog = (uint)maxBits; + } + + { + nuint newCSize = HUF_estimateCompressedSize( + &nextHuf->CTable.e0, + countWksp, + maxSymbolValue + ); + nuint hSize = HUF_writeCTable_wksp( + hufMetadata->hufDesBuffer, + sizeof(byte) * 128, + &nextHuf->CTable.e0, + maxSymbolValue, + huffLog, + nodeWksp, + nodeWkspSize + ); + if (repeat != HUF_repeat.HUF_repeat_none) + { + nuint oldCSize = HUF_estimateCompressedSize( + &prevHuf->CTable.e0, + countWksp, + maxSymbolValue + ); + if (oldCSize < srcSize && (oldCSize <= hSize + newCSize || hSize + 12 >= srcSize)) + { + memcpy(nextHuf, prevHuf, (uint)sizeof(ZSTD_hufCTables_t)); + hufMetadata->hType = SymbolEncodingType_e.set_repeat; + return 0; + } + } + + if (newCSize + hSize >= srcSize) + { + memcpy(nextHuf, prevHuf, (uint)sizeof(ZSTD_hufCTables_t)); + hufMetadata->hType = SymbolEncodingType_e.set_basic; + return 0; + } + + hufMetadata->hType = SymbolEncodingType_e.set_compressed; + nextHuf->repeatMode = HUF_repeat.HUF_repeat_check; + return hSize; + } + } + + /* ZSTD_buildDummySequencesStatistics(): + * Returns a ZSTD_symbolEncodingTypeStats_t with all encoding types as set_basic, + * and updates nextEntropy to the appropriate repeatMode. + */ + private static ZSTD_symbolEncodingTypeStats_t ZSTD_buildDummySequencesStatistics( + ZSTD_fseCTables_t* nextEntropy + ) + { + ZSTD_symbolEncodingTypeStats_t stats = new ZSTD_symbolEncodingTypeStats_t + { + LLtype = (uint)SymbolEncodingType_e.set_basic, + Offtype = (uint)SymbolEncodingType_e.set_basic, + MLtype = (uint)SymbolEncodingType_e.set_basic, + size = 0, + lastCountSize = 0, + longOffsets = 0, + }; + nextEntropy->litlength_repeatMode = FSE_repeat.FSE_repeat_none; + nextEntropy->offcode_repeatMode = FSE_repeat.FSE_repeat_none; + nextEntropy->matchlength_repeatMode = FSE_repeat.FSE_repeat_none; + return stats; + } + + /** ZSTD_buildBlockEntropyStats_sequences() : + * Builds entropy for the sequences. + * Stores symbol compression modes and fse table to fseMetadata. + * Requires ENTROPY_WORKSPACE_SIZE wksp. + * @return : size of fse tables or error code */ + private static nuint ZSTD_buildBlockEntropyStats_sequences( + SeqStore_t* seqStorePtr, + ZSTD_fseCTables_t* prevEntropy, + ZSTD_fseCTables_t* nextEntropy, + ZSTD_CCtx_params_s* cctxParams, + ZSTD_fseCTablesMetadata_t* fseMetadata, + void* workspace, + nuint wkspSize + ) + { + ZSTD_strategy strategy = cctxParams->cParams.strategy; + nuint nbSeq = (nuint)(seqStorePtr->sequences - seqStorePtr->sequencesStart); + byte* ostart = fseMetadata->fseTablesBuffer; + byte* oend = ostart + sizeof(byte) * 133; + byte* op = ostart; + uint* countWorkspace = (uint*)workspace; + uint* entropyWorkspace = countWorkspace + (52 + 1); + nuint entropyWorkspaceSize = wkspSize - (52 + 1) * sizeof(uint); + ZSTD_symbolEncodingTypeStats_t stats; + stats = + nbSeq != 0 + ? ZSTD_buildSequencesStatistics( + seqStorePtr, + nbSeq, + prevEntropy, + nextEntropy, + op, + oend, + strategy, + countWorkspace, + entropyWorkspace, + entropyWorkspaceSize + ) + : ZSTD_buildDummySequencesStatistics(nextEntropy); + { + nuint err_code = stats.size; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + fseMetadata->llType = (SymbolEncodingType_e)stats.LLtype; + fseMetadata->ofType = (SymbolEncodingType_e)stats.Offtype; + fseMetadata->mlType = (SymbolEncodingType_e)stats.MLtype; + fseMetadata->lastCountSize = stats.lastCountSize; + return stats.size; + } + + /** ZSTD_buildBlockEntropyStats() : + * Builds entropy for the block. + * Requires workspace size ENTROPY_WORKSPACE_SIZE + * @return : 0 on success, or an error code + * Note : also employed in superblock + */ + private static nuint ZSTD_buildBlockEntropyStats( + SeqStore_t* seqStorePtr, + ZSTD_entropyCTables_t* prevEntropy, + ZSTD_entropyCTables_t* nextEntropy, + ZSTD_CCtx_params_s* cctxParams, + ZSTD_entropyCTablesMetadata_t* entropyMetadata, + void* workspace, + nuint wkspSize + ) + { + nuint litSize = (nuint)(seqStorePtr->lit - seqStorePtr->litStart); + int huf_useOptDepth = cctxParams->cParams.strategy >= ZSTD_strategy.ZSTD_btultra ? 1 : 0; + int hufFlags = huf_useOptDepth != 0 ? (int)HUF_flags_e.HUF_flags_optimalDepth : 0; + entropyMetadata->hufMetadata.hufDesSize = ZSTD_buildBlockEntropyStats_literals( + seqStorePtr->litStart, + litSize, + &prevEntropy->huf, + &nextEntropy->huf, + &entropyMetadata->hufMetadata, + ZSTD_literalsCompressionIsDisabled(cctxParams), + workspace, + wkspSize, + hufFlags + ); + { + nuint err_code = entropyMetadata->hufMetadata.hufDesSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + entropyMetadata->fseMetadata.fseTablesSize = ZSTD_buildBlockEntropyStats_sequences( + seqStorePtr, + &prevEntropy->fse, + &nextEntropy->fse, + cctxParams, + &entropyMetadata->fseMetadata, + workspace, + wkspSize + ); + { + nuint err_code = entropyMetadata->fseMetadata.fseTablesSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return 0; + } + + /* Returns the size estimate for the literals section (header + content) of a block */ + private static nuint ZSTD_estimateBlockSize_literal( + byte* literals, + nuint litSize, + ZSTD_hufCTables_t* huf, + ZSTD_hufCTablesMetadata_t* hufMetadata, + void* workspace, + nuint wkspSize, + int writeEntropy + ) + { + uint* countWksp = (uint*)workspace; + uint maxSymbolValue = 255; + nuint literalSectionHeaderSize = (nuint)( + 3 + (litSize >= 1 * (1 << 10) ? 1 : 0) + (litSize >= 16 * (1 << 10) ? 1 : 0) + ); + uint singleStream = litSize < 256 ? 1U : 0U; + if (hufMetadata->hType == SymbolEncodingType_e.set_basic) + { + return litSize; + } + else if (hufMetadata->hType == SymbolEncodingType_e.set_rle) + { + return 1; + } + else if ( + hufMetadata->hType == SymbolEncodingType_e.set_compressed + || hufMetadata->hType == SymbolEncodingType_e.set_repeat + ) + { + nuint largest = HIST_count_wksp( + countWksp, + &maxSymbolValue, + literals, + litSize, + workspace, + wkspSize + ); + if (ERR_isError(largest)) + { + return litSize; + } + + { + nuint cLitSizeEstimate = HUF_estimateCompressedSize( + &huf->CTable.e0, + countWksp, + maxSymbolValue + ); + if (writeEntropy != 0) + { + cLitSizeEstimate += hufMetadata->hufDesSize; + } + + if (singleStream == 0) + { + cLitSizeEstimate += 6; + } + + return cLitSizeEstimate + literalSectionHeaderSize; + } + } + + assert(0 != 0); + return 0; + } + + /* Returns the size estimate for the FSE-compressed symbols (of, ml, ll) of a block */ + private static nuint ZSTD_estimateBlockSize_symbolType( + SymbolEncodingType_e type, + byte* codeTable, + nuint nbSeq, + uint maxCode, + uint* fseCTable, + byte* additionalBits, + short* defaultNorm, + uint defaultNormLog, + uint defaultMax, + void* workspace, + nuint wkspSize + ) + { + uint* countWksp = (uint*)workspace; + byte* ctp = codeTable; + byte* ctStart = ctp; + byte* ctEnd = ctStart + nbSeq; + nuint cSymbolTypeSizeEstimateInBits = 0; + uint max = maxCode; + HIST_countFast_wksp(countWksp, &max, codeTable, nbSeq, workspace, wkspSize); + if (type == SymbolEncodingType_e.set_basic) + { + assert(max <= defaultMax); + cSymbolTypeSizeEstimateInBits = ZSTD_crossEntropyCost( + defaultNorm, + defaultNormLog, + countWksp, + max + ); + } + else if (type == SymbolEncodingType_e.set_rle) + { + cSymbolTypeSizeEstimateInBits = 0; + } + else if ( + type == SymbolEncodingType_e.set_compressed + || type == SymbolEncodingType_e.set_repeat + ) + { + cSymbolTypeSizeEstimateInBits = ZSTD_fseBitCost(fseCTable, countWksp, max); + } + + if (ERR_isError(cSymbolTypeSizeEstimateInBits)) + { + return nbSeq * 10; + } + + while (ctp < ctEnd) + { + if (additionalBits != null) + { + cSymbolTypeSizeEstimateInBits += additionalBits[*ctp]; + } + else + { + cSymbolTypeSizeEstimateInBits += *ctp; + } + + ctp++; + } + + return cSymbolTypeSizeEstimateInBits >> 3; + } + + /* Returns the size estimate for the sequences section (header + content) of a block */ + private static nuint ZSTD_estimateBlockSize_sequences( + byte* ofCodeTable, + byte* llCodeTable, + byte* mlCodeTable, + nuint nbSeq, + ZSTD_fseCTables_t* fseTables, + ZSTD_fseCTablesMetadata_t* fseMetadata, + void* workspace, + nuint wkspSize, + int writeEntropy + ) + { + /* seqHead */ + nuint sequencesSectionHeaderSize = (nuint)( + 1 + 1 + (nbSeq >= 128 ? 1 : 0) + (nbSeq >= 0x7F00 ? 1 : 0) + ); + nuint cSeqSizeEstimate = 0; + cSeqSizeEstimate += ZSTD_estimateBlockSize_symbolType( + fseMetadata->ofType, + ofCodeTable, + nbSeq, + 31, + fseTables->offcodeCTable, + null, + OF_defaultNorm, + OF_defaultNormLog, + 28, + workspace, + wkspSize + ); + cSeqSizeEstimate += ZSTD_estimateBlockSize_symbolType( + fseMetadata->llType, + llCodeTable, + nbSeq, + 35, + fseTables->litlengthCTable, + LL_bits, + LL_defaultNorm, + LL_defaultNormLog, + 35, + workspace, + wkspSize + ); + cSeqSizeEstimate += ZSTD_estimateBlockSize_symbolType( + fseMetadata->mlType, + mlCodeTable, + nbSeq, + 52, + fseTables->matchlengthCTable, + ML_bits, + ML_defaultNorm, + ML_defaultNormLog, + 52, + workspace, + wkspSize + ); + if (writeEntropy != 0) + { + cSeqSizeEstimate += fseMetadata->fseTablesSize; + } + + return cSeqSizeEstimate + sequencesSectionHeaderSize; + } + + /* Returns the size estimate for a given stream of literals, of, ll, ml */ + private static nuint ZSTD_estimateBlockSize( + byte* literals, + nuint litSize, + byte* ofCodeTable, + byte* llCodeTable, + byte* mlCodeTable, + nuint nbSeq, + ZSTD_entropyCTables_t* entropy, + ZSTD_entropyCTablesMetadata_t* entropyMetadata, + void* workspace, + nuint wkspSize, + int writeLitEntropy, + int writeSeqEntropy + ) + { + nuint literalsSize = ZSTD_estimateBlockSize_literal( + literals, + litSize, + &entropy->huf, + &entropyMetadata->hufMetadata, + workspace, + wkspSize, + writeLitEntropy + ); + nuint seqSize = ZSTD_estimateBlockSize_sequences( + ofCodeTable, + llCodeTable, + mlCodeTable, + nbSeq, + &entropy->fse, + &entropyMetadata->fseMetadata, + workspace, + wkspSize, + writeSeqEntropy + ); + return seqSize + literalsSize + ZSTD_blockHeaderSize; + } + + /* Builds entropy statistics and uses them for blocksize estimation. + * + * @return: estimated compressed size of the seqStore, or a zstd error. + */ + private static nuint ZSTD_buildEntropyStatisticsAndEstimateSubBlockSize( + SeqStore_t* seqStore, + ZSTD_CCtx_s* zc + ) + { + ZSTD_entropyCTablesMetadata_t* entropyMetadata = &zc->blockSplitCtx.entropyMetadata; + { + nuint err_code = ZSTD_buildBlockEntropyStats( + seqStore, + &zc->blockState.prevCBlock->entropy, + &zc->blockState.nextCBlock->entropy, + &zc->appliedParams, + entropyMetadata, + zc->tmpWorkspace, + zc->tmpWkspSize + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return ZSTD_estimateBlockSize( + seqStore->litStart, + (nuint)(seqStore->lit - seqStore->litStart), + seqStore->ofCode, + seqStore->llCode, + seqStore->mlCode, + (nuint)(seqStore->sequences - seqStore->sequencesStart), + &zc->blockState.nextCBlock->entropy, + entropyMetadata, + zc->tmpWorkspace, + zc->tmpWkspSize, + entropyMetadata->hufMetadata.hType == SymbolEncodingType_e.set_compressed ? 1 : 0, + 1 + ); + } + + /* Returns literals bytes represented in a seqStore */ + private static nuint ZSTD_countSeqStoreLiteralsBytes(SeqStore_t* seqStore) + { + nuint literalsBytes = 0; + nuint nbSeqs = (nuint)(seqStore->sequences - seqStore->sequencesStart); + nuint i; + for (i = 0; i < nbSeqs; ++i) + { + SeqDef_s seq = seqStore->sequencesStart[i]; + literalsBytes += seq.litLength; + if ( + i == seqStore->longLengthPos + && seqStore->longLengthType == ZSTD_longLengthType_e.ZSTD_llt_literalLength + ) + { + literalsBytes += 0x10000; + } + } + + return literalsBytes; + } + + /* Returns match bytes represented in a seqStore */ + private static nuint ZSTD_countSeqStoreMatchBytes(SeqStore_t* seqStore) + { + nuint matchBytes = 0; + nuint nbSeqs = (nuint)(seqStore->sequences - seqStore->sequencesStart); + nuint i; + for (i = 0; i < nbSeqs; ++i) + { + SeqDef_s seq = seqStore->sequencesStart[i]; + matchBytes += (nuint)(seq.mlBase + 3); + if ( + i == seqStore->longLengthPos + && seqStore->longLengthType == ZSTD_longLengthType_e.ZSTD_llt_matchLength + ) + { + matchBytes += 0x10000; + } + } + + return matchBytes; + } + + /* Derives the seqStore that is a chunk of the originalSeqStore from [startIdx, endIdx). + * Stores the result in resultSeqStore. + */ + private static void ZSTD_deriveSeqStoreChunk( + SeqStore_t* resultSeqStore, + SeqStore_t* originalSeqStore, + nuint startIdx, + nuint endIdx + ) + { + *resultSeqStore = *originalSeqStore; + if (startIdx > 0) + { + resultSeqStore->sequences = originalSeqStore->sequencesStart + startIdx; + resultSeqStore->litStart += ZSTD_countSeqStoreLiteralsBytes(resultSeqStore); + } + + if (originalSeqStore->longLengthType != ZSTD_longLengthType_e.ZSTD_llt_none) + { + if ( + originalSeqStore->longLengthPos < startIdx + || originalSeqStore->longLengthPos > endIdx + ) + { + resultSeqStore->longLengthType = ZSTD_longLengthType_e.ZSTD_llt_none; + } + else + { + resultSeqStore->longLengthPos -= (uint)startIdx; + } + } + + resultSeqStore->sequencesStart = originalSeqStore->sequencesStart + startIdx; + resultSeqStore->sequences = originalSeqStore->sequencesStart + endIdx; + if (endIdx == (nuint)(originalSeqStore->sequences - originalSeqStore->sequencesStart)) + { + assert(resultSeqStore->lit == originalSeqStore->lit); + } + else + { + nuint literalsBytes = ZSTD_countSeqStoreLiteralsBytes(resultSeqStore); + resultSeqStore->lit = resultSeqStore->litStart + literalsBytes; + } + + resultSeqStore->llCode += startIdx; + resultSeqStore->mlCode += startIdx; + resultSeqStore->ofCode += startIdx; + } + + /** + * Returns the raw offset represented by the combination of offBase, ll0, and repcode history. + * offBase must represent a repcode in the numeric representation of ZSTD_storeSeq(). + */ + private static uint ZSTD_resolveRepcodeToRawOffset(uint* rep, uint offBase, uint ll0) + { + assert(1 <= offBase && offBase <= 3); + /* [ 0 - 3 ] */ + uint adjustedRepCode = offBase - 1 + ll0; + assert(1 <= offBase && offBase <= 3); + if (adjustedRepCode == 3) + { + assert(ll0 != 0); + return rep[0] - 1; + } + + return rep[adjustedRepCode]; + } + + /** + * ZSTD_seqStore_resolveOffCodes() reconciles any possible divergences in offset history that may arise + * due to emission of RLE/raw blocks that disturb the offset history, + * and replaces any repcodes within the seqStore that may be invalid. + * + * dRepcodes are updated as would be on the decompression side. + * cRepcodes are updated exactly in accordance with the seqStore. + * + * Note : this function assumes seq->offBase respects the following numbering scheme : + * 0 : invalid + * 1-3 : repcode 1-3 + * 4+ : real_offset+3 + */ + private static void ZSTD_seqStore_resolveOffCodes( + repcodes_s* dRepcodes, + repcodes_s* cRepcodes, + SeqStore_t* seqStore, + uint nbSeq + ) + { + uint idx = 0; + uint longLitLenIdx = + seqStore->longLengthType == ZSTD_longLengthType_e.ZSTD_llt_literalLength + ? seqStore->longLengthPos + : nbSeq; + for (; idx < nbSeq; ++idx) + { + SeqDef_s* seq = seqStore->sequencesStart + idx; + uint ll0 = seq->litLength == 0 && idx != longLitLenIdx ? 1U : 0U; + uint offBase = seq->offBase; + assert(offBase > 0); + if (1 <= offBase && offBase <= 3) + { + uint dRawOffset = ZSTD_resolveRepcodeToRawOffset(dRepcodes->rep, offBase, ll0); + uint cRawOffset = ZSTD_resolveRepcodeToRawOffset(cRepcodes->rep, offBase, ll0); + if (dRawOffset != cRawOffset) + { + assert(cRawOffset > 0); + seq->offBase = cRawOffset + 3; + } + } + + ZSTD_updateRep(dRepcodes->rep, seq->offBase, ll0); + ZSTD_updateRep(cRepcodes->rep, offBase, ll0); + } + } + + /* ZSTD_compressSeqStore_singleBlock(): + * Compresses a seqStore into a block with a block header, into the buffer dst. + * + * Returns the total size of that block (including header) or a ZSTD error code. + */ + private static nuint ZSTD_compressSeqStore_singleBlock( + ZSTD_CCtx_s* zc, + SeqStore_t* seqStore, + repcodes_s* dRep, + repcodes_s* cRep, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize, + uint lastBlock, + uint isPartition + ) + { + const uint rleMaxLength = 25; + byte* op = (byte*)dst; + byte* ip = (byte*)src; + nuint cSize; + nuint cSeqsSize; + /* In case of an RLE or raw block, the simulated decompression repcode history must be reset */ + repcodes_s dRepOriginal = *dRep; + if (isPartition != 0) + { + ZSTD_seqStore_resolveOffCodes( + dRep, + cRep, + seqStore, + (uint)(seqStore->sequences - seqStore->sequencesStart) + ); + } + + if (dstCapacity < ZSTD_blockHeaderSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + cSeqsSize = ZSTD_entropyCompressSeqStore( + seqStore, + &zc->blockState.prevCBlock->entropy, + &zc->blockState.nextCBlock->entropy, + &zc->appliedParams, + op + ZSTD_blockHeaderSize, + dstCapacity - ZSTD_blockHeaderSize, + srcSize, + zc->tmpWorkspace, + zc->tmpWkspSize, + zc->bmi2 + ); + { + nuint err_code = cSeqsSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if ( + zc->isFirstBlock == 0 + && cSeqsSize < rleMaxLength + && ZSTD_isRLE((byte*)src, srcSize) != 0 + ) + { + cSeqsSize = 1; + } + + if (zc->seqCollector.collectSequences != 0) + { + { + nuint err_code = ZSTD_copyBlockSequences( + &zc->seqCollector, + seqStore, + dRepOriginal.rep + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + ZSTD_blockState_confirmRepcodesAndEntropyTables(&zc->blockState); + return 0; + } + + if (cSeqsSize == 0) + { + cSize = ZSTD_noCompressBlock(op, dstCapacity, ip, srcSize, lastBlock); + { + nuint err_code = cSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + *dRep = dRepOriginal; + } + else if (cSeqsSize == 1) + { + cSize = ZSTD_rleCompressBlock(op, dstCapacity, *ip, srcSize, lastBlock); + { + nuint err_code = cSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + *dRep = dRepOriginal; + } + else + { + ZSTD_blockState_confirmRepcodesAndEntropyTables(&zc->blockState); + writeBlockHeader(op, cSeqsSize, srcSize, lastBlock); + cSize = ZSTD_blockHeaderSize + cSeqsSize; + } + + if ( + zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode == FSE_repeat.FSE_repeat_valid + ) + { + zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode = FSE_repeat.FSE_repeat_check; + } + + return cSize; + } + + /* Helper function to perform the recursive search for block splits. + * Estimates the cost of seqStore prior to split, and estimates the cost of splitting the sequences in half. + * If advantageous to split, then we recurse down the two sub-blocks. + * If not, or if an error occurred in estimation, then we do not recurse. + * + * Note: The recursion depth is capped by a heuristic minimum number of sequences, + * defined by MIN_SEQUENCES_BLOCK_SPLITTING. + * In theory, this means the absolute largest recursion depth is 10 == log2(maxNbSeqInBlock/MIN_SEQUENCES_BLOCK_SPLITTING). + * In practice, recursion depth usually doesn't go beyond 4. + * + * Furthermore, the number of splits is capped by ZSTD_MAX_NB_BLOCK_SPLITS. + * At ZSTD_MAX_NB_BLOCK_SPLITS == 196 with the current existing blockSize + * maximum of 128 KB, this value is actually impossible to reach. + */ + private static void ZSTD_deriveBlockSplitsHelper( + seqStoreSplits* splits, + nuint startIdx, + nuint endIdx, + ZSTD_CCtx_s* zc, + SeqStore_t* origSeqStore + ) + { + SeqStore_t* fullSeqStoreChunk = &zc->blockSplitCtx.fullSeqStoreChunk; + SeqStore_t* firstHalfSeqStore = &zc->blockSplitCtx.firstHalfSeqStore; + SeqStore_t* secondHalfSeqStore = &zc->blockSplitCtx.secondHalfSeqStore; + nuint estimatedOriginalSize; + nuint estimatedFirstHalfSize; + nuint estimatedSecondHalfSize; + nuint midIdx = (startIdx + endIdx) / 2; + assert(endIdx >= startIdx); + if (endIdx - startIdx < 300 || splits->idx >= 196) + { + return; + } + + ZSTD_deriveSeqStoreChunk(fullSeqStoreChunk, origSeqStore, startIdx, endIdx); + ZSTD_deriveSeqStoreChunk(firstHalfSeqStore, origSeqStore, startIdx, midIdx); + ZSTD_deriveSeqStoreChunk(secondHalfSeqStore, origSeqStore, midIdx, endIdx); + estimatedOriginalSize = ZSTD_buildEntropyStatisticsAndEstimateSubBlockSize( + fullSeqStoreChunk, + zc + ); + estimatedFirstHalfSize = ZSTD_buildEntropyStatisticsAndEstimateSubBlockSize( + firstHalfSeqStore, + zc + ); + estimatedSecondHalfSize = ZSTD_buildEntropyStatisticsAndEstimateSubBlockSize( + secondHalfSeqStore, + zc + ); + if ( + ERR_isError(estimatedOriginalSize) + || ERR_isError(estimatedFirstHalfSize) + || ERR_isError(estimatedSecondHalfSize) + ) + { + return; + } + + if (estimatedFirstHalfSize + estimatedSecondHalfSize < estimatedOriginalSize) + { + ZSTD_deriveBlockSplitsHelper(splits, startIdx, midIdx, zc, origSeqStore); + splits->splitLocations[splits->idx] = (uint)midIdx; + splits->idx++; + ZSTD_deriveBlockSplitsHelper(splits, midIdx, endIdx, zc, origSeqStore); + } + } + + /* Base recursive function. + * Populates a table with intra-block partition indices that can improve compression ratio. + * + * @return: number of splits made (which equals the size of the partition table - 1). + */ + private static nuint ZSTD_deriveBlockSplits(ZSTD_CCtx_s* zc, uint* partitions, uint nbSeq) + { + seqStoreSplits splits; + splits.splitLocations = partitions; + splits.idx = 0; + if (nbSeq <= 4) + { + return 0; + } + + ZSTD_deriveBlockSplitsHelper(&splits, 0, nbSeq, zc, &zc->seqStore); + splits.splitLocations[splits.idx] = nbSeq; + return splits.idx; + } + + /* ZSTD_compressBlock_splitBlock(): + * Attempts to split a given block into multiple blocks to improve compression ratio. + * + * Returns combined size of all blocks (which includes headers), or a ZSTD error code. + */ + private static nuint ZSTD_compressBlock_splitBlock_internal( + ZSTD_CCtx_s* zc, + void* dst, + nuint dstCapacity, + void* src, + nuint blockSize, + uint lastBlock, + uint nbSeq + ) + { + nuint cSize = 0; + byte* ip = (byte*)src; + byte* op = (byte*)dst; + nuint i = 0; + nuint srcBytesTotal = 0; + /* size == ZSTD_MAX_NB_BLOCK_SPLITS */ + uint* partitions = zc->blockSplitCtx.partitions; + SeqStore_t* nextSeqStore = &zc->blockSplitCtx.nextSeqStore; + SeqStore_t* currSeqStore = &zc->blockSplitCtx.currSeqStore; + nuint numSplits = ZSTD_deriveBlockSplits(zc, partitions, nbSeq); + /* If a block is split and some partitions are emitted as RLE/uncompressed, then repcode history + * may become invalid. In order to reconcile potentially invalid repcodes, we keep track of two + * separate repcode histories that simulate repcode history on compression and decompression side, + * and use the histories to determine whether we must replace a particular repcode with its raw offset. + * + * 1) cRep gets updated for each partition, regardless of whether the block was emitted as uncompressed + * or RLE. This allows us to retrieve the offset value that an invalid repcode references within + * a nocompress/RLE block. + * 2) dRep gets updated only for compressed partitions, and when a repcode gets replaced, will use + * the replacement offset value rather than the original repcode to update the repcode history. + * dRep also will be the final repcode history sent to the next block. + * + * See ZSTD_seqStore_resolveOffCodes() for more details. + */ + repcodes_s dRep; + repcodes_s cRep; + memcpy(dRep.rep, zc->blockState.prevCBlock->rep, (uint)sizeof(repcodes_s)); + memcpy(cRep.rep, zc->blockState.prevCBlock->rep, (uint)sizeof(repcodes_s)); + *nextSeqStore = new SeqStore_t(); + if (numSplits == 0) + { + nuint cSizeSingleBlock = ZSTD_compressSeqStore_singleBlock( + zc, + &zc->seqStore, + &dRep, + &cRep, + op, + dstCapacity, + ip, + blockSize, + lastBlock, + 0 + ); + { + nuint err_code = cSizeSingleBlock; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + assert(zc->blockSizeMax <= 1 << 17); + assert(cSizeSingleBlock <= zc->blockSizeMax + ZSTD_blockHeaderSize); + return cSizeSingleBlock; + } + + ZSTD_deriveSeqStoreChunk(currSeqStore, &zc->seqStore, 0, partitions[0]); + for (i = 0; i <= numSplits; ++i) + { + nuint cSizeChunk; + uint lastPartition = i == numSplits ? 1U : 0U; + uint lastBlockEntireSrc = 0; + nuint srcBytes = + ZSTD_countSeqStoreLiteralsBytes(currSeqStore) + + ZSTD_countSeqStoreMatchBytes(currSeqStore); + srcBytesTotal += srcBytes; + if (lastPartition != 0) + { + srcBytes += blockSize - srcBytesTotal; + lastBlockEntireSrc = lastBlock; + } + else + { + ZSTD_deriveSeqStoreChunk( + nextSeqStore, + &zc->seqStore, + partitions[i], + partitions[i + 1] + ); + } + + cSizeChunk = ZSTD_compressSeqStore_singleBlock( + zc, + currSeqStore, + &dRep, + &cRep, + op, + dstCapacity, + ip, + srcBytes, + lastBlockEntireSrc, + 1 + ); + { + nuint err_code = cSizeChunk; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + ip += srcBytes; + op += cSizeChunk; + dstCapacity -= cSizeChunk; + cSize += cSizeChunk; + *currSeqStore = *nextSeqStore; + assert(cSizeChunk <= zc->blockSizeMax + ZSTD_blockHeaderSize); + } + + memcpy(zc->blockState.prevCBlock->rep, dRep.rep, (uint)sizeof(repcodes_s)); + return cSize; + } + + private static nuint ZSTD_compressBlock_splitBlock( + ZSTD_CCtx_s* zc, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize, + uint lastBlock + ) + { + uint nbSeq; + nuint cSize; + assert(zc->appliedParams.postBlockSplitter == ZSTD_paramSwitch_e.ZSTD_ps_enable); + { + nuint bss = ZSTD_buildSeqStore(zc, src, srcSize); + { + nuint err_code = bss; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (bss == (nuint)ZSTD_BuildSeqStore_e.ZSTDbss_noCompress) + { + if ( + zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode + == FSE_repeat.FSE_repeat_valid + ) + { + zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode = + FSE_repeat.FSE_repeat_check; + } + + if (zc->seqCollector.collectSequences != 0) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_sequenceProducer_failed) + ); + } + + cSize = ZSTD_noCompressBlock(dst, dstCapacity, src, srcSize, lastBlock); + { + nuint err_code = cSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return cSize; + } + + nbSeq = (uint)(zc->seqStore.sequences - zc->seqStore.sequencesStart); + } + + cSize = ZSTD_compressBlock_splitBlock_internal( + zc, + dst, + dstCapacity, + src, + srcSize, + lastBlock, + nbSeq + ); + { + nuint err_code = cSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return cSize; + } + + private static nuint ZSTD_compressBlock_internal( + ZSTD_CCtx_s* zc, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize, + uint frame + ) + { + /* This is an estimated upper bound for the length of an rle block. + * This isn't the actual upper bound. + * Finding the real threshold needs further investigation. + */ + const uint rleMaxLength = 25; + nuint cSize; + byte* ip = (byte*)src; + byte* op = (byte*)dst; + { + nuint bss = ZSTD_buildSeqStore(zc, src, srcSize); + { + nuint err_code = bss; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (bss == (nuint)ZSTD_BuildSeqStore_e.ZSTDbss_noCompress) + { + if (zc->seqCollector.collectSequences != 0) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_sequenceProducer_failed) + ); + } + + cSize = 0; + goto @out; + } + } + + if (zc->seqCollector.collectSequences != 0) + { + { + nuint err_code = ZSTD_copyBlockSequences( + &zc->seqCollector, + ZSTD_getSeqStore(zc), + zc->blockState.prevCBlock->rep + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + ZSTD_blockState_confirmRepcodesAndEntropyTables(&zc->blockState); + return 0; + } + + cSize = ZSTD_entropyCompressSeqStore( + &zc->seqStore, + &zc->blockState.prevCBlock->entropy, + &zc->blockState.nextCBlock->entropy, + &zc->appliedParams, + dst, + dstCapacity, + srcSize, + zc->tmpWorkspace, + zc->tmpWkspSize, + zc->bmi2 + ); + if ( + frame != 0 + && zc->isFirstBlock == 0 + && cSize < rleMaxLength + && ZSTD_isRLE(ip, srcSize) != 0 + ) + { + cSize = 1; + op[0] = ip[0]; + } + + @out: + if (!ERR_isError(cSize) && cSize > 1) + { + ZSTD_blockState_confirmRepcodesAndEntropyTables(&zc->blockState); + } + + if ( + zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode == FSE_repeat.FSE_repeat_valid + ) + { + zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode = FSE_repeat.FSE_repeat_check; + } + + return cSize; + } + + private static nuint ZSTD_compressBlock_targetCBlockSize_body( + ZSTD_CCtx_s* zc, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize, + nuint bss, + uint lastBlock + ) + { + if (bss == (nuint)ZSTD_BuildSeqStore_e.ZSTDbss_compress) + { + if ( + zc->isFirstBlock == 0 + && ZSTD_maybeRLE(&zc->seqStore) != 0 + && ZSTD_isRLE((byte*)src, srcSize) != 0 + ) + { + return ZSTD_rleCompressBlock(dst, dstCapacity, *(byte*)src, srcSize, lastBlock); + } + + { + nuint cSize = ZSTD_compressSuperBlock( + zc, + dst, + dstCapacity, + src, + srcSize, + lastBlock + ); + if (cSize != unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall))) + { + nuint maxCSize = + srcSize - ZSTD_minGain(srcSize, zc->appliedParams.cParams.strategy); + { + nuint err_code = cSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (cSize != 0 && cSize < maxCSize + ZSTD_blockHeaderSize) + { + ZSTD_blockState_confirmRepcodesAndEntropyTables(&zc->blockState); + return cSize; + } + } + } + } + + return ZSTD_noCompressBlock(dst, dstCapacity, src, srcSize, lastBlock); + } + + private static nuint ZSTD_compressBlock_targetCBlockSize( + ZSTD_CCtx_s* zc, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize, + uint lastBlock + ) + { + nuint cSize = 0; + nuint bss = ZSTD_buildSeqStore(zc, src, srcSize); + { + nuint err_code = bss; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + cSize = ZSTD_compressBlock_targetCBlockSize_body( + zc, + dst, + dstCapacity, + src, + srcSize, + bss, + lastBlock + ); + { + nuint err_code = cSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if ( + zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode == FSE_repeat.FSE_repeat_valid + ) + { + zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode = FSE_repeat.FSE_repeat_check; + } + + return cSize; + } + + private static void ZSTD_overflowCorrectIfNeeded( + ZSTD_MatchState_t* ms, + ZSTD_cwksp* ws, + ZSTD_CCtx_params_s* @params, + void* ip, + void* iend + ) + { + uint cycleLog = ZSTD_cycleLog(@params->cParams.chainLog, @params->cParams.strategy); + uint maxDist = (uint)1 << (int)@params->cParams.windowLog; + if ( + ZSTD_window_needOverflowCorrection( + ms->window, + cycleLog, + maxDist, + ms->loadedDictEnd, + ip, + iend + ) != 0 + ) + { + uint correction = ZSTD_window_correctOverflow(&ms->window, cycleLog, maxDist, ip); + ZSTD_cwksp_mark_tables_dirty(ws); + ZSTD_reduceIndex(ms, @params, correction); + ZSTD_cwksp_mark_tables_clean(ws); + if (ms->nextToUpdate < correction) + { + ms->nextToUpdate = 0; + } + else + { + ms->nextToUpdate -= correction; + } + + ms->loadedDictEnd = 0; + ms->dictMatchState = null; + } + } + +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_splitLevels => + new int[10] { 0, 0, 1, 2, 2, 3, 3, 4, 4, 4 }; + private static int* splitLevels => + (int*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_splitLevels) + ); +#else + + private static readonly int* splitLevels = GetArrayPointer( + new int[10] { 0, 0, 1, 2, 2, 3, 3, 4, 4, 4 } + ); +#endif + + private static nuint ZSTD_optimalBlockSize( + ZSTD_CCtx_s* cctx, + void* src, + nuint srcSize, + nuint blockSizeMax, + int splitLevel, + ZSTD_strategy strat, + long savings + ) + { + if (srcSize < 128 * (1 << 10) || blockSizeMax < 128 * (1 << 10)) + { + return srcSize < blockSizeMax ? srcSize : blockSizeMax; + } + + if (savings < 3) + { + return 128 * (1 << 10); + } + + if (splitLevel == 1) + { + return 128 * (1 << 10); + } + + if (splitLevel == 0) + { + assert(ZSTD_strategy.ZSTD_fast <= strat && strat <= ZSTD_strategy.ZSTD_btultra2); + splitLevel = splitLevels[(int)strat]; + } + else + { + assert(2 <= splitLevel && splitLevel <= 6); + splitLevel -= 2; + } + + return ZSTD_splitBlock( + src, + blockSizeMax, + splitLevel, + cctx->tmpWorkspace, + cctx->tmpWkspSize + ); + } + + /*! ZSTD_compress_frameChunk() : + * Compress a chunk of data into one or multiple blocks. + * All blocks will be terminated, all input will be consumed. + * Function will issue an error if there is not enough `dstCapacity` to hold the compressed content. + * Frame is supposed already started (header already produced) + * @return : compressed size, or an error code + */ + private static nuint ZSTD_compress_frameChunk( + ZSTD_CCtx_s* cctx, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize, + uint lastFrameChunk + ) + { + nuint blockSizeMax = cctx->blockSizeMax; + nuint remaining = srcSize; + byte* ip = (byte*)src; + byte* ostart = (byte*)dst; + byte* op = ostart; + uint maxDist = (uint)1 << (int)cctx->appliedParams.cParams.windowLog; + long savings = (long)cctx->consumedSrcSize - (long)cctx->producedCSize; + assert(cctx->appliedParams.cParams.windowLog <= (uint)(sizeof(nuint) == 4 ? 30 : 31)); + if (cctx->appliedParams.fParams.checksumFlag != 0 && srcSize != 0) + { + ZSTD_XXH64_update(&cctx->xxhState, src, srcSize); + } + + while (remaining != 0) + { + ZSTD_MatchState_t* ms = &cctx->blockState.matchState; + nuint blockSize = ZSTD_optimalBlockSize( + cctx, + ip, + remaining, + blockSizeMax, + cctx->appliedParams.preBlockSplitter_level, + cctx->appliedParams.cParams.strategy, + savings + ); + uint lastBlock = lastFrameChunk & (uint)(blockSize == remaining ? 1 : 0); + assert(blockSize <= remaining); + if (dstCapacity < ZSTD_blockHeaderSize + (nuint)(1 + 1) + 1) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + ZSTD_overflowCorrectIfNeeded( + ms, + &cctx->workspace, + &cctx->appliedParams, + ip, + ip + blockSize + ); + ZSTD_checkDictValidity( + &ms->window, + ip + blockSize, + maxDist, + &ms->loadedDictEnd, + &ms->dictMatchState + ); + ZSTD_window_enforceMaxDist( + &ms->window, + ip, + maxDist, + &ms->loadedDictEnd, + &ms->dictMatchState + ); + if (ms->nextToUpdate < ms->window.lowLimit) + { + ms->nextToUpdate = ms->window.lowLimit; + } + + { + nuint cSize; + if (ZSTD_useTargetCBlockSize(&cctx->appliedParams) != 0) + { + cSize = ZSTD_compressBlock_targetCBlockSize( + cctx, + op, + dstCapacity, + ip, + blockSize, + lastBlock + ); + { + nuint err_code = cSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + assert(cSize > 0); + assert(cSize <= blockSize + ZSTD_blockHeaderSize); + } + else if (ZSTD_blockSplitterEnabled(&cctx->appliedParams) != 0) + { + cSize = ZSTD_compressBlock_splitBlock( + cctx, + op, + dstCapacity, + ip, + blockSize, + lastBlock + ); + { + nuint err_code = cSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + assert(cSize > 0 || cctx->seqCollector.collectSequences == 1); + } + else + { + cSize = ZSTD_compressBlock_internal( + cctx, + op + ZSTD_blockHeaderSize, + dstCapacity - ZSTD_blockHeaderSize, + ip, + blockSize, + 1 + ); + { + nuint err_code = cSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (cSize == 0) + { + cSize = ZSTD_noCompressBlock(op, dstCapacity, ip, blockSize, lastBlock); + { + nuint err_code = cSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + } + else + { + uint cBlockHeader = + cSize == 1 + ? lastBlock + + ((uint)blockType_e.bt_rle << 1) + + (uint)(blockSize << 3) + : lastBlock + + ((uint)blockType_e.bt_compressed << 1) + + (uint)(cSize << 3); + MEM_writeLE24(op, cBlockHeader); + cSize += ZSTD_blockHeaderSize; + } + } + + savings += (long)blockSize - (long)cSize; + ip += blockSize; + assert(remaining >= blockSize); + remaining -= blockSize; + op += cSize; + assert(dstCapacity >= cSize); + dstCapacity -= cSize; + cctx->isFirstBlock = 0; + } + } + + if (lastFrameChunk != 0 && op > ostart) + { + cctx->stage = ZSTD_compressionStage_e.ZSTDcs_ending; + } + + return (nuint)(op - ostart); + } + + private static nuint ZSTD_writeFrameHeader( + void* dst, + nuint dstCapacity, + ZSTD_CCtx_params_s* @params, + ulong pledgedSrcSize, + uint dictID + ) + { + byte* op = (byte*)dst; + /* 0-3 */ + uint dictIDSizeCodeLength = (uint)( + (dictID > 0 ? 1 : 0) + (dictID >= 256 ? 1 : 0) + (dictID >= 65536 ? 1 : 0) + ); + /* 0-3 */ + uint dictIDSizeCode = @params->fParams.noDictIDFlag != 0 ? 0 : dictIDSizeCodeLength; + uint checksumFlag = @params->fParams.checksumFlag > 0 ? 1U : 0U; + uint windowSize = (uint)1 << (int)@params->cParams.windowLog; + uint singleSegment = + @params->fParams.contentSizeFlag != 0 && windowSize >= pledgedSrcSize ? 1U : 0U; + byte windowLogByte = (byte)(@params->cParams.windowLog - 10 << 3); + uint fcsCode = (uint)( + @params->fParams.contentSizeFlag != 0 + ? (pledgedSrcSize >= 256 ? 1 : 0) + + (pledgedSrcSize >= 65536 + 256 ? 1 : 0) + + (pledgedSrcSize >= 0xFFFFFFFFU ? 1 : 0) + : 0 + ); + byte frameHeaderDescriptionByte = (byte)( + dictIDSizeCode + (checksumFlag << 2) + (singleSegment << 5) + (fcsCode << 6) + ); + nuint pos = 0; + assert(!(@params->fParams.contentSizeFlag != 0 && pledgedSrcSize == unchecked(0UL - 1))); + if (dstCapacity < 18) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + if (@params->format == ZSTD_format_e.ZSTD_f_zstd1) + { + MEM_writeLE32(dst, 0xFD2FB528); + pos = 4; + } + + op[pos++] = frameHeaderDescriptionByte; + if (singleSegment == 0) + { + op[pos++] = windowLogByte; + } + + switch (dictIDSizeCode) + { + default: + assert(0 != 0); + goto case 0; + case 0: + break; + case 1: + op[pos] = (byte)dictID; + pos++; + break; + case 2: + MEM_writeLE16(op + pos, (ushort)dictID); + pos += 2; + break; + case 3: + MEM_writeLE32(op + pos, dictID); + pos += 4; + break; + } + + switch (fcsCode) + { + default: + assert(0 != 0); + goto case 0; + case 0: + if (singleSegment != 0) + { + op[pos++] = (byte)pledgedSrcSize; + } + + break; + case 1: + MEM_writeLE16(op + pos, (ushort)(pledgedSrcSize - 256)); + pos += 2; + break; + case 2: + MEM_writeLE32(op + pos, (uint)pledgedSrcSize); + pos += 4; + break; + case 3: + MEM_writeLE64(op + pos, pledgedSrcSize); + pos += 8; + break; + } + + return pos; + } + + /* ZSTD_writeSkippableFrame_advanced() : + * Writes out a skippable frame with the specified magic number variant (16 are supported), + * from ZSTD_MAGIC_SKIPPABLE_START to ZSTD_MAGIC_SKIPPABLE_START+15, and the desired source data. + * + * Returns the total number of bytes written, or a ZSTD error code. + */ + public static nuint ZSTD_writeSkippableFrame( + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize, + uint magicVariant + ) + { + byte* op = (byte*)dst; + if (dstCapacity < srcSize + 8) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + if (srcSize > 0xFFFFFFFF) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + if (magicVariant > 15) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + MEM_writeLE32(op, 0x184D2A50 + magicVariant); + MEM_writeLE32(op + 4, (uint)srcSize); + memcpy(op + 8, src, (uint)srcSize); + return srcSize + 8; + } + + /* ZSTD_writeLastEmptyBlock() : + * output an empty Block with end-of-frame mark to complete a frame + * @return : size of data written into `dst` (== ZSTD_blockHeaderSize (defined in zstd_internal.h)) + * or an error code if `dstCapacity` is too small (stage == ZSTD_compressionStage_e.ZSTDcs_init); + assert( + nbSeq == 0 + || cctx->appliedParams.ldmParams.enableLdm != ZSTD_paramSwitch_e.ZSTD_ps_enable + ); + cctx->externSeqStore.seq = seq; + cctx->externSeqStore.size = nbSeq; + cctx->externSeqStore.capacity = nbSeq; + cctx->externSeqStore.pos = 0; + cctx->externSeqStore.posInSequence = 0; + } + + private static nuint ZSTD_compressContinue_internal( + ZSTD_CCtx_s* cctx, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize, + uint frame, + uint lastFrameChunk + ) + { + ZSTD_MatchState_t* ms = &cctx->blockState.matchState; + nuint fhSize = 0; + if (cctx->stage == ZSTD_compressionStage_e.ZSTDcs_created) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_stage_wrong)); + } + + if (frame != 0 && cctx->stage == ZSTD_compressionStage_e.ZSTDcs_init) + { + fhSize = ZSTD_writeFrameHeader( + dst, + dstCapacity, + &cctx->appliedParams, + cctx->pledgedSrcSizePlusOne - 1, + cctx->dictID + ); + { + nuint err_code = fhSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + assert(fhSize <= dstCapacity); + dstCapacity -= fhSize; + dst = (sbyte*)dst + fhSize; + cctx->stage = ZSTD_compressionStage_e.ZSTDcs_ongoing; + } + + if (srcSize == 0) + { + return fhSize; + } + + if (ZSTD_window_update(&ms->window, src, srcSize, ms->forceNonContiguous) == 0) + { + ms->forceNonContiguous = 0; + ms->nextToUpdate = ms->window.dictLimit; + } + + if (cctx->appliedParams.ldmParams.enableLdm == ZSTD_paramSwitch_e.ZSTD_ps_enable) + { + ZSTD_window_update(&cctx->ldmState.window, src, srcSize, 0); + } + + if (frame == 0) + { + ZSTD_overflowCorrectIfNeeded( + ms, + &cctx->workspace, + &cctx->appliedParams, + src, + (byte*)src + srcSize + ); + } + + { + nuint cSize = + frame != 0 + ? ZSTD_compress_frameChunk(cctx, dst, dstCapacity, src, srcSize, lastFrameChunk) + : ZSTD_compressBlock_internal(cctx, dst, dstCapacity, src, srcSize, 0); + { + nuint err_code = cSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + cctx->consumedSrcSize += srcSize; + cctx->producedCSize += cSize + fhSize; + assert( + !( + cctx->appliedParams.fParams.contentSizeFlag != 0 + && cctx->pledgedSrcSizePlusOne == 0 + ) + ); + if (cctx->pledgedSrcSizePlusOne != 0) + { + if (cctx->consumedSrcSize + 1 > cctx->pledgedSrcSizePlusOne) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + } + + return cSize + fhSize; + } + } + + private static nuint ZSTD_compressContinue_public( + ZSTD_CCtx_s* cctx, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize + ) + { + return ZSTD_compressContinue_internal(cctx, dst, dstCapacity, src, srcSize, 1, 0); + } + + /* NOTE: Must just wrap ZSTD_compressContinue_public() */ + public static nuint ZSTD_compressContinue( + ZSTD_CCtx_s* cctx, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize + ) + { + return ZSTD_compressContinue_public(cctx, dst, dstCapacity, src, srcSize); + } + + private static nuint ZSTD_getBlockSize_deprecated(ZSTD_CCtx_s* cctx) + { + ZSTD_compressionParameters cParams = cctx->appliedParams.cParams; + assert(ZSTD_checkCParams(cParams) == 0); + return cctx->appliedParams.maxBlockSize < (nuint)1 << (int)cParams.windowLog + ? cctx->appliedParams.maxBlockSize + : (nuint)1 << (int)cParams.windowLog; + } + + /* NOTE: Must just wrap ZSTD_getBlockSize_deprecated() */ + public static nuint ZSTD_getBlockSize(ZSTD_CCtx_s* cctx) + { + return ZSTD_getBlockSize_deprecated(cctx); + } + + /* NOTE: Must just wrap ZSTD_compressBlock_deprecated() */ + private static nuint ZSTD_compressBlock_deprecated( + ZSTD_CCtx_s* cctx, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize + ) + { + { + nuint blockSizeMax = ZSTD_getBlockSize_deprecated(cctx); + if (srcSize > blockSizeMax) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + } + + return ZSTD_compressContinue_internal(cctx, dst, dstCapacity, src, srcSize, 0, 0); + } + + /* NOTE: Must just wrap ZSTD_compressBlock_deprecated() */ + public static nuint ZSTD_compressBlock( + ZSTD_CCtx_s* cctx, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_deprecated(cctx, dst, dstCapacity, src, srcSize); + } + + /*! ZSTD_loadDictionaryContent() : + * @return : 0, or an error code + */ + private static nuint ZSTD_loadDictionaryContent( + ZSTD_MatchState_t* ms, + ldmState_t* ls, + ZSTD_cwksp* ws, + ZSTD_CCtx_params_s* @params, + void* src, + nuint srcSize, + ZSTD_dictTableLoadMethod_e dtlm, + ZSTD_tableFillPurpose_e tfp + ) + { + byte* ip = (byte*)src; + byte* iend = ip + srcSize; + int loadLdmDict = + @params->ldmParams.enableLdm == ZSTD_paramSwitch_e.ZSTD_ps_enable && ls != null ? 1 : 0; + ZSTD_assertEqualCParams(@params->cParams, ms->cParams); + { + /* Allow the dictionary to set indices up to exactly ZSTD_CURRENT_MAX. + * Dictionaries right at the edge will immediately trigger overflow + * correction, but I don't want to insert extra constraints here. + */ + uint maxDictSize = (MEM_64bits ? 3500U * (1 << 20) : 2000U * (1 << 20)) - 2; + int CDictTaggedIndices = ZSTD_CDictIndicesAreTagged(&@params->cParams); + if (CDictTaggedIndices != 0 && tfp == ZSTD_tableFillPurpose_e.ZSTD_tfp_forCDict) + { + /* Some dictionary matchfinders in zstd use "short cache", + * which treats the lower ZSTD_SHORT_CACHE_TAG_BITS of each + * CDict hashtable entry as a tag rather than as part of an index. + * When short cache is used, we need to truncate the dictionary + * so that its indices don't overlap with the tag. */ + const uint shortCacheMaxDictSize = (1U << 32 - 8) - 2; + maxDictSize = + maxDictSize < shortCacheMaxDictSize ? maxDictSize : shortCacheMaxDictSize; + assert(loadLdmDict == 0); + } + + if (srcSize > maxDictSize) + { + ip = iend - maxDictSize; + src = ip; + srcSize = maxDictSize; + } + } + + if (srcSize > unchecked((uint)-1) - (MEM_64bits ? 3500U * (1 << 20) : 2000U * (1 << 20))) + { + assert(ZSTD_window_isEmpty(ms->window) != 0); + } + + ZSTD_window_update(&ms->window, src, srcSize, 0); + if (loadLdmDict != 0) + { + ZSTD_window_update(&ls->window, src, srcSize, 0); + ls->loadedDictEnd = @params->forceWindow != 0 ? 0 : (uint)(iend - ls->window.@base); + ZSTD_ldm_fillHashTable(ls, ip, iend, &@params->ldmParams); + } + + { + uint maxDictSize = + 1U + << (int)( + ( + @params->cParams.hashLog + 3 > @params->cParams.chainLog + 1 + ? @params->cParams.hashLog + 3 + : @params->cParams.chainLog + 1 + ) < 31 + ? @params->cParams.hashLog + 3 > @params->cParams.chainLog + 1 + ? @params->cParams.hashLog + 3 + : @params->cParams.chainLog + 1 + : 31 + ); + if (srcSize > maxDictSize) + { + ip = iend - maxDictSize; + src = ip; + srcSize = maxDictSize; + } + } + + ms->nextToUpdate = (uint)(ip - ms->window.@base); + ms->loadedDictEnd = @params->forceWindow != 0 ? 0 : (uint)(iend - ms->window.@base); + ms->forceNonContiguous = @params->deterministicRefPrefix; + if (srcSize <= 8) + { + return 0; + } + + ZSTD_overflowCorrectIfNeeded(ms, ws, @params, ip, iend); + switch (@params->cParams.strategy) + { + case ZSTD_strategy.ZSTD_fast: + ZSTD_fillHashTable(ms, iend, dtlm, tfp); + break; + case ZSTD_strategy.ZSTD_dfast: + ZSTD_fillDoubleHashTable(ms, iend, dtlm, tfp); + break; + case ZSTD_strategy.ZSTD_greedy: + case ZSTD_strategy.ZSTD_lazy: + case ZSTD_strategy.ZSTD_lazy2: + assert(srcSize >= 8); + if (ms->dedicatedDictSearch != 0) + { + assert(ms->chainTable != null); + ZSTD_dedicatedDictSearch_lazy_loadDictionary(ms, iend - 8); + } + else + { + assert(@params->useRowMatchFinder != ZSTD_paramSwitch_e.ZSTD_ps_auto); + if (@params->useRowMatchFinder == ZSTD_paramSwitch_e.ZSTD_ps_enable) + { + nuint tagTableSize = (nuint)1 << (int)@params->cParams.hashLog; + memset(ms->tagTable, 0, (uint)tagTableSize); + ZSTD_row_update(ms, iend - 8); + } + else + { + ZSTD_insertAndFindFirstIndex(ms, iend - 8); + } + } + + break; + case ZSTD_strategy.ZSTD_btlazy2: + case ZSTD_strategy.ZSTD_btopt: + case ZSTD_strategy.ZSTD_btultra: + case ZSTD_strategy.ZSTD_btultra2: + assert(srcSize >= 8); + ZSTD_updateTree(ms, iend - 8, iend); + break; + default: + assert(0 != 0); + break; + } + + ms->nextToUpdate = (uint)(iend - ms->window.@base); + return 0; + } + + /* Dictionaries that assign zero probability to symbols that show up causes problems + * when FSE encoding. Mark dictionaries with zero probability symbols as FSE_repeat_check + * and only dictionaries with 100% valid symbols can be assumed valid. + */ + private static FSE_repeat ZSTD_dictNCountRepeat( + short* normalizedCounter, + uint dictMaxSymbolValue, + uint maxSymbolValue + ) + { + uint s; + if (dictMaxSymbolValue < maxSymbolValue) + { + return FSE_repeat.FSE_repeat_check; + } + + for (s = 0; s <= maxSymbolValue; ++s) + { + if (normalizedCounter[s] == 0) + { + return FSE_repeat.FSE_repeat_check; + } + } + + return FSE_repeat.FSE_repeat_valid; + } + + /* ZSTD_loadCEntropy() : + * dict : must point at beginning of a valid zstd dictionary. + * return : size of dictionary header (size of magic number + dict ID + entropy tables) + * assumptions : magic number supposed already checked + * and dictSize >= 8 */ + private static nuint ZSTD_loadCEntropy( + ZSTD_compressedBlockState_t* bs, + void* workspace, + void* dict, + nuint dictSize + ) + { + short* offcodeNCount = stackalloc short[32]; + uint offcodeMaxValue = 31; + /* skip magic num and dict ID */ + byte* dictPtr = (byte*)dict; + byte* dictEnd = dictPtr + dictSize; + dictPtr += 8; + bs->entropy.huf.repeatMode = HUF_repeat.HUF_repeat_check; + { + uint maxSymbolValue = 255; + uint hasZeroWeights = 1; + nuint hufHeaderSize = HUF_readCTable( + &bs->entropy.huf.CTable.e0, + &maxSymbolValue, + dictPtr, + (nuint)(dictEnd - dictPtr), + &hasZeroWeights + ); + if (hasZeroWeights == 0 && maxSymbolValue == 255) + { + bs->entropy.huf.repeatMode = HUF_repeat.HUF_repeat_valid; + } + + if (ERR_isError(hufHeaderSize)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + dictPtr += hufHeaderSize; + } + + { + uint offcodeLog; + nuint offcodeHeaderSize = FSE_readNCount( + offcodeNCount, + &offcodeMaxValue, + &offcodeLog, + dictPtr, + (nuint)(dictEnd - dictPtr) + ); + if (ERR_isError(offcodeHeaderSize)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + if (offcodeLog > 8) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + if ( + ERR_isError( + FSE_buildCTable_wksp( + bs->entropy.fse.offcodeCTable, + offcodeNCount, + 31, + offcodeLog, + workspace, + (8 << 10) + 512 + ) + ) + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + dictPtr += offcodeHeaderSize; + } + + { + short* matchlengthNCount = stackalloc short[53]; + uint matchlengthMaxValue = 52, + matchlengthLog; + nuint matchlengthHeaderSize = FSE_readNCount( + matchlengthNCount, + &matchlengthMaxValue, + &matchlengthLog, + dictPtr, + (nuint)(dictEnd - dictPtr) + ); + if (ERR_isError(matchlengthHeaderSize)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + if (matchlengthLog > 9) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + if ( + ERR_isError( + FSE_buildCTable_wksp( + bs->entropy.fse.matchlengthCTable, + matchlengthNCount, + matchlengthMaxValue, + matchlengthLog, + workspace, + (8 << 10) + 512 + ) + ) + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + bs->entropy.fse.matchlength_repeatMode = ZSTD_dictNCountRepeat( + matchlengthNCount, + matchlengthMaxValue, + 52 + ); + dictPtr += matchlengthHeaderSize; + } + + { + short* litlengthNCount = stackalloc short[36]; + uint litlengthMaxValue = 35, + litlengthLog; + nuint litlengthHeaderSize = FSE_readNCount( + litlengthNCount, + &litlengthMaxValue, + &litlengthLog, + dictPtr, + (nuint)(dictEnd - dictPtr) + ); + if (ERR_isError(litlengthHeaderSize)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + if (litlengthLog > 9) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + if ( + ERR_isError( + FSE_buildCTable_wksp( + bs->entropy.fse.litlengthCTable, + litlengthNCount, + litlengthMaxValue, + litlengthLog, + workspace, + (8 << 10) + 512 + ) + ) + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + bs->entropy.fse.litlength_repeatMode = ZSTD_dictNCountRepeat( + litlengthNCount, + litlengthMaxValue, + 35 + ); + dictPtr += litlengthHeaderSize; + } + + if (dictPtr + 12 > dictEnd) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + bs->rep[0] = MEM_readLE32(dictPtr + 0); + bs->rep[1] = MEM_readLE32(dictPtr + 4); + bs->rep[2] = MEM_readLE32(dictPtr + 8); + dictPtr += 12; + { + nuint dictContentSize = (nuint)(dictEnd - dictPtr); + uint offcodeMax = 31; + if (dictContentSize <= unchecked((uint)-1) - 128 * (1 << 10)) + { + /* The maximum offset that must be supported */ + uint maxOffset = (uint)dictContentSize + 128 * (1 << 10); + offcodeMax = ZSTD_highbit32(maxOffset); + } + + bs->entropy.fse.offcode_repeatMode = ZSTD_dictNCountRepeat( + offcodeNCount, + offcodeMaxValue, + offcodeMax < 31 ? offcodeMax : 31 + ); + { + uint u; + for (u = 0; u < 3; u++) + { + if (bs->rep[u] == 0) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted) + ); + } + + if (bs->rep[u] > dictContentSize) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted) + ); + } + } + } + } + + return (nuint)(dictPtr - (byte*)dict); + } + + /* Dictionary format : + * See : + * https://github.com/facebook/zstd/blob/release/doc/zstd_compression_format.md#dictionary-format + */ + /*! ZSTD_loadZstdDictionary() : + * @return : dictID, or an error code + * assumptions : magic number supposed already checked + * dictSize supposed >= 8 + */ + private static nuint ZSTD_loadZstdDictionary( + ZSTD_compressedBlockState_t* bs, + ZSTD_MatchState_t* ms, + ZSTD_cwksp* ws, + ZSTD_CCtx_params_s* @params, + void* dict, + nuint dictSize, + ZSTD_dictTableLoadMethod_e dtlm, + ZSTD_tableFillPurpose_e tfp, + void* workspace + ) + { + byte* dictPtr = (byte*)dict; + byte* dictEnd = dictPtr + dictSize; + nuint dictID; + nuint eSize; + assert(dictSize >= 8); + assert(MEM_readLE32(dictPtr) == 0xEC30A437); + dictID = @params->fParams.noDictIDFlag != 0 ? 0 : MEM_readLE32(dictPtr + 4); + eSize = ZSTD_loadCEntropy(bs, workspace, dict, dictSize); + { + nuint err_code = eSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + dictPtr += eSize; + { + nuint dictContentSize = (nuint)(dictEnd - dictPtr); + { + nuint err_code = ZSTD_loadDictionaryContent( + ms, + null, + ws, + @params, + dictPtr, + dictContentSize, + dtlm, + tfp + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + } + + return dictID; + } + + /** ZSTD_compress_insertDictionary() : + * @return : dictID, or an error code */ + private static nuint ZSTD_compress_insertDictionary( + ZSTD_compressedBlockState_t* bs, + ZSTD_MatchState_t* ms, + ldmState_t* ls, + ZSTD_cwksp* ws, + ZSTD_CCtx_params_s* @params, + void* dict, + nuint dictSize, + ZSTD_dictContentType_e dictContentType, + ZSTD_dictTableLoadMethod_e dtlm, + ZSTD_tableFillPurpose_e tfp, + void* workspace + ) + { + if (dict == null || dictSize < 8) + { + if (dictContentType == ZSTD_dictContentType_e.ZSTD_dct_fullDict) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_wrong)); + } + + return 0; + } + + ZSTD_reset_compressedBlockState(bs); + if (dictContentType == ZSTD_dictContentType_e.ZSTD_dct_rawContent) + { + return ZSTD_loadDictionaryContent(ms, ls, ws, @params, dict, dictSize, dtlm, tfp); + } + + if (MEM_readLE32(dict) != 0xEC30A437) + { + if (dictContentType == ZSTD_dictContentType_e.ZSTD_dct_auto) + { + return ZSTD_loadDictionaryContent(ms, ls, ws, @params, dict, dictSize, dtlm, tfp); + } + + if (dictContentType == ZSTD_dictContentType_e.ZSTD_dct_fullDict) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_wrong)); + } + + assert(0 != 0); + } + + return ZSTD_loadZstdDictionary(bs, ms, ws, @params, dict, dictSize, dtlm, tfp, workspace); + } + + /*! ZSTD_compressBegin_internal() : + * Assumption : either @dict OR @cdict (or none) is non-NULL, never both + * @return : 0, or an error code */ + private static nuint ZSTD_compressBegin_internal( + ZSTD_CCtx_s* cctx, + void* dict, + nuint dictSize, + ZSTD_dictContentType_e dictContentType, + ZSTD_dictTableLoadMethod_e dtlm, + ZSTD_CDict_s* cdict, + ZSTD_CCtx_params_s* @params, + ulong pledgedSrcSize, + ZSTD_buffered_policy_e zbuff + ) + { + nuint dictContentSize = cdict != null ? cdict->dictContentSize : dictSize; + assert(!ERR_isError(ZSTD_checkCParams(@params->cParams))); + assert(!(dict != null && cdict != null)); + if ( + cdict != null + && cdict->dictContentSize > 0 + && ( + pledgedSrcSize < 128 * (1 << 10) + || pledgedSrcSize < cdict->dictContentSize * 6UL + || pledgedSrcSize == unchecked(0UL - 1) + || cdict->compressionLevel == 0 + ) + && @params->attachDictPref != ZSTD_dictAttachPref_e.ZSTD_dictForceLoad + ) + { + return ZSTD_resetCCtx_usingCDict(cctx, cdict, @params, pledgedSrcSize, zbuff); + } + + { + nuint err_code = ZSTD_resetCCtx_internal( + cctx, + @params, + pledgedSrcSize, + dictContentSize, + ZSTD_compResetPolicy_e.ZSTDcrp_makeClean, + zbuff + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint dictID = + cdict != null + ? ZSTD_compress_insertDictionary( + cctx->blockState.prevCBlock, + &cctx->blockState.matchState, + &cctx->ldmState, + &cctx->workspace, + &cctx->appliedParams, + cdict->dictContent, + cdict->dictContentSize, + cdict->dictContentType, + dtlm, + ZSTD_tableFillPurpose_e.ZSTD_tfp_forCCtx, + cctx->tmpWorkspace + ) + : ZSTD_compress_insertDictionary( + cctx->blockState.prevCBlock, + &cctx->blockState.matchState, + &cctx->ldmState, + &cctx->workspace, + &cctx->appliedParams, + dict, + dictSize, + dictContentType, + dtlm, + ZSTD_tableFillPurpose_e.ZSTD_tfp_forCCtx, + cctx->tmpWorkspace + ); + { + nuint err_code = dictID; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + assert(dictID <= 0xffffffff); + cctx->dictID = (uint)dictID; + cctx->dictContentSize = dictContentSize; + } + + return 0; + } + + /* ZSTD_compressBegin_advanced_internal() : + * Private use only. To be called from zstdmt_compress.c. */ + private static nuint ZSTD_compressBegin_advanced_internal( + ZSTD_CCtx_s* cctx, + void* dict, + nuint dictSize, + ZSTD_dictContentType_e dictContentType, + ZSTD_dictTableLoadMethod_e dtlm, + ZSTD_CDict_s* cdict, + ZSTD_CCtx_params_s* @params, + ulong pledgedSrcSize + ) + { + { + /* compression parameters verification and optimization */ + nuint err_code = ZSTD_checkCParams(@params->cParams); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return ZSTD_compressBegin_internal( + cctx, + dict, + dictSize, + dictContentType, + dtlm, + cdict, + @params, + pledgedSrcSize, + ZSTD_buffered_policy_e.ZSTDb_not_buffered + ); + } + + /*! ZSTD_compressBegin_advanced() : + * @return : 0, or an error code */ + public static nuint ZSTD_compressBegin_advanced( + ZSTD_CCtx_s* cctx, + void* dict, + nuint dictSize, + ZSTD_parameters @params, + ulong pledgedSrcSize + ) + { + ZSTD_CCtx_params_s cctxParams; + ZSTD_CCtxParams_init_internal(&cctxParams, &@params, 0); + return ZSTD_compressBegin_advanced_internal( + cctx, + dict, + dictSize, + ZSTD_dictContentType_e.ZSTD_dct_auto, + ZSTD_dictTableLoadMethod_e.ZSTD_dtlm_fast, + null, + &cctxParams, + pledgedSrcSize + ); + } + + private static nuint ZSTD_compressBegin_usingDict_deprecated( + ZSTD_CCtx_s* cctx, + void* dict, + nuint dictSize, + int compressionLevel + ) + { + ZSTD_CCtx_params_s cctxParams; + { + ZSTD_parameters @params = ZSTD_getParams_internal( + compressionLevel, + unchecked(0UL - 1), + dictSize, + ZSTD_CParamMode_e.ZSTD_cpm_noAttachDict + ); + ZSTD_CCtxParams_init_internal( + &cctxParams, + &@params, + compressionLevel == 0 ? 3 : compressionLevel + ); + } + + return ZSTD_compressBegin_internal( + cctx, + dict, + dictSize, + ZSTD_dictContentType_e.ZSTD_dct_auto, + ZSTD_dictTableLoadMethod_e.ZSTD_dtlm_fast, + null, + &cctxParams, + unchecked(0UL - 1), + ZSTD_buffered_policy_e.ZSTDb_not_buffered + ); + } + + public static nuint ZSTD_compressBegin_usingDict( + ZSTD_CCtx_s* cctx, + void* dict, + nuint dictSize, + int compressionLevel + ) + { + return ZSTD_compressBegin_usingDict_deprecated(cctx, dict, dictSize, compressionLevel); + } + + /*===== Buffer-less streaming compression functions =====*/ + public static nuint ZSTD_compressBegin(ZSTD_CCtx_s* cctx, int compressionLevel) + { + return ZSTD_compressBegin_usingDict_deprecated(cctx, null, 0, compressionLevel); + } + + /*! ZSTD_writeEpilogue() : + * Ends a frame. + * @return : nb of bytes written into dst (or an error code) */ + private static nuint ZSTD_writeEpilogue(ZSTD_CCtx_s* cctx, void* dst, nuint dstCapacity) + { + byte* ostart = (byte*)dst; + byte* op = ostart; + if (cctx->stage == ZSTD_compressionStage_e.ZSTDcs_created) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_stage_wrong)); + } + + if (cctx->stage == ZSTD_compressionStage_e.ZSTDcs_init) + { + nuint fhSize = ZSTD_writeFrameHeader(dst, dstCapacity, &cctx->appliedParams, 0, 0); + { + nuint err_code = fhSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + dstCapacity -= fhSize; + op += fhSize; + cctx->stage = ZSTD_compressionStage_e.ZSTDcs_ongoing; + } + + if (cctx->stage != ZSTD_compressionStage_e.ZSTDcs_ending) + { + /* last block */ + uint cBlockHeader24 = 1 + ((uint)blockType_e.bt_raw << 1) + 0; + if (dstCapacity < 3) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + MEM_writeLE24(op, cBlockHeader24); + op += ZSTD_blockHeaderSize; + dstCapacity -= ZSTD_blockHeaderSize; + } + + if (cctx->appliedParams.fParams.checksumFlag != 0) + { + uint checksum = (uint)ZSTD_XXH64_digest(&cctx->xxhState); + if (dstCapacity < 4) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + MEM_writeLE32(op, checksum); + op += 4; + } + + cctx->stage = ZSTD_compressionStage_e.ZSTDcs_created; + return (nuint)(op - ostart); + } + + /** ZSTD_CCtx_trace() : + * Trace the end of a compression call. + */ + private static void ZSTD_CCtx_trace(ZSTD_CCtx_s* cctx, nuint extraCSize) { } + + private static nuint ZSTD_compressEnd_public( + ZSTD_CCtx_s* cctx, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize + ) + { + nuint endResult; + nuint cSize = ZSTD_compressContinue_internal(cctx, dst, dstCapacity, src, srcSize, 1, 1); + { + nuint err_code = cSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + endResult = ZSTD_writeEpilogue(cctx, (sbyte*)dst + cSize, dstCapacity - cSize); + { + nuint err_code = endResult; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + assert( + !(cctx->appliedParams.fParams.contentSizeFlag != 0 && cctx->pledgedSrcSizePlusOne == 0) + ); + if (cctx->pledgedSrcSizePlusOne != 0) + { + if (cctx->pledgedSrcSizePlusOne != cctx->consumedSrcSize + 1) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + } + + ZSTD_CCtx_trace(cctx, endResult); + return cSize + endResult; + } + + /* NOTE: Must just wrap ZSTD_compressEnd_public() */ + public static nuint ZSTD_compressEnd( + ZSTD_CCtx_s* cctx, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize + ) + { + return ZSTD_compressEnd_public(cctx, dst, dstCapacity, src, srcSize); + } + + /*! ZSTD_compress_advanced() : + * Note : this function is now DEPRECATED. + * It can be replaced by ZSTD_compress2(), in combination with ZSTD_CCtx_setParameter() and other parameter setters. + * This prototype will generate compilation warnings. */ + public static nuint ZSTD_compress_advanced( + ZSTD_CCtx_s* cctx, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize, + void* dict, + nuint dictSize, + ZSTD_parameters @params + ) + { + { + nuint err_code = ZSTD_checkCParams(@params.cParams); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + ZSTD_CCtxParams_init_internal(&cctx->simpleApiParams, &@params, 0); + return ZSTD_compress_advanced_internal( + cctx, + dst, + dstCapacity, + src, + srcSize, + dict, + dictSize, + &cctx->simpleApiParams + ); + } + + /* Internal */ + private static nuint ZSTD_compress_advanced_internal( + ZSTD_CCtx_s* cctx, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize, + void* dict, + nuint dictSize, + ZSTD_CCtx_params_s* @params + ) + { + { + nuint err_code = ZSTD_compressBegin_internal( + cctx, + dict, + dictSize, + ZSTD_dictContentType_e.ZSTD_dct_auto, + ZSTD_dictTableLoadMethod_e.ZSTD_dtlm_fast, + null, + @params, + srcSize, + ZSTD_buffered_policy_e.ZSTDb_not_buffered + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return ZSTD_compressEnd_public(cctx, dst, dstCapacity, src, srcSize); + } + + /************************** + * Simple dictionary API + ***************************/ + /*! ZSTD_compress_usingDict() : + * Compression at an explicit compression level using a Dictionary. + * A dictionary can be any arbitrary data segment (also called a prefix), + * or a buffer with specified information (see zdict.h). + * Note : This function loads the dictionary, resulting in significant startup delay. + * It's intended for a dictionary used only once. + * Note 2 : When `dict == NULL || dictSize < 8` no dictionary is used. */ + public static nuint ZSTD_compress_usingDict( + ZSTD_CCtx_s* cctx, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize, + void* dict, + nuint dictSize, + int compressionLevel + ) + { + { + ZSTD_parameters @params = ZSTD_getParams_internal( + compressionLevel, + srcSize, + dict != null ? dictSize : 0, + ZSTD_CParamMode_e.ZSTD_cpm_noAttachDict + ); + assert(@params.fParams.contentSizeFlag == 1); + ZSTD_CCtxParams_init_internal( + &cctx->simpleApiParams, + &@params, + compressionLevel == 0 ? 3 : compressionLevel + ); + } + + return ZSTD_compress_advanced_internal( + cctx, + dst, + dstCapacity, + src, + srcSize, + dict, + dictSize, + &cctx->simpleApiParams + ); + } + + /*! ZSTD_compressCCtx() : + * Same as ZSTD_compress(), using an explicit ZSTD_CCtx. + * Important : in order to mirror `ZSTD_compress()` behavior, + * this function compresses at the requested compression level, + * __ignoring any other advanced parameter__ . + * If any advanced parameter was set using the advanced API, + * they will all be reset. Only @compressionLevel remains. + */ + public static nuint ZSTD_compressCCtx( + ZSTD_CCtx_s* cctx, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize, + int compressionLevel + ) + { + assert(cctx != null); + return ZSTD_compress_usingDict( + cctx, + dst, + dstCapacity, + src, + srcSize, + null, + 0, + compressionLevel + ); + } + + /*************************************** + * Simple Core API + ***************************************/ + /*! ZSTD_compress() : + * Compresses `src` content as a single zstd compressed frame into already allocated `dst`. + * NOTE: Providing `dstCapacity >= ZSTD_compressBound(srcSize)` guarantees that zstd will have + * enough space to successfully compress the data. + * @return : compressed size written into `dst` (<= `dstCapacity), + * or an error code if it fails (which can be tested using ZSTD_isError()). */ + public static nuint ZSTD_compress( + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize, + int compressionLevel + ) + { + nuint result; + ZSTD_CCtx_s ctxBody; + ZSTD_initCCtx(&ctxBody, ZSTD_defaultCMem); + result = ZSTD_compressCCtx(&ctxBody, dst, dstCapacity, src, srcSize, compressionLevel); + ZSTD_freeCCtxContent(&ctxBody); + return result; + } + + /*! ZSTD_estimateCDictSize_advanced() : + * Estimate amount of memory that will be needed to create a dictionary with following arguments */ + public static nuint ZSTD_estimateCDictSize_advanced( + nuint dictSize, + ZSTD_compressionParameters cParams, + ZSTD_dictLoadMethod_e dictLoadMethod + ) + { + return ZSTD_cwksp_alloc_size((nuint)sizeof(ZSTD_CDict_s)) + + ZSTD_cwksp_alloc_size((8 << 10) + 512) + + ZSTD_sizeof_matchState( + &cParams, + ZSTD_resolveRowMatchFinderMode(ZSTD_paramSwitch_e.ZSTD_ps_auto, &cParams), + 1, + 0 + ) + + ( + dictLoadMethod == ZSTD_dictLoadMethod_e.ZSTD_dlm_byRef + ? 0 + : ZSTD_cwksp_alloc_size(ZSTD_cwksp_align(dictSize, (nuint)sizeof(void*))) + ); + } + + /*! ZSTD_estimate?DictSize() : + * ZSTD_estimateCDictSize() will bet that src size is relatively "small", and content is copied, like ZSTD_createCDict(). + * ZSTD_estimateCDictSize_advanced() makes it possible to control compression parameters precisely, like ZSTD_createCDict_advanced(). + * Note : dictionaries created by reference (`ZSTD_dlm_byRef`) are logically smaller. + */ + public static nuint ZSTD_estimateCDictSize(nuint dictSize, int compressionLevel) + { + ZSTD_compressionParameters cParams = ZSTD_getCParams_internal( + compressionLevel, + unchecked(0UL - 1), + dictSize, + ZSTD_CParamMode_e.ZSTD_cpm_createCDict + ); + return ZSTD_estimateCDictSize_advanced( + dictSize, + cParams, + ZSTD_dictLoadMethod_e.ZSTD_dlm_byCopy + ); + } + + public static nuint ZSTD_sizeof_CDict(ZSTD_CDict_s* cdict) + { + if (cdict == null) + { + return 0; + } + + return (nuint)(cdict->workspace.workspace == cdict ? 0 : sizeof(ZSTD_CDict_s)) + + ZSTD_cwksp_sizeof(&cdict->workspace); + } + + private static nuint ZSTD_initCDict_internal( + ZSTD_CDict_s* cdict, + void* dictBuffer, + nuint dictSize, + ZSTD_dictLoadMethod_e dictLoadMethod, + ZSTD_dictContentType_e dictContentType, + ZSTD_CCtx_params_s @params + ) + { + assert(ZSTD_checkCParams(@params.cParams) == 0); + cdict->matchState.cParams = @params.cParams; + cdict->matchState.dedicatedDictSearch = @params.enableDedicatedDictSearch; + if ( + dictLoadMethod == ZSTD_dictLoadMethod_e.ZSTD_dlm_byRef + || dictBuffer == null + || dictSize == 0 + ) + { + cdict->dictContent = dictBuffer; + } + else + { + void* internalBuffer = ZSTD_cwksp_reserve_object( + &cdict->workspace, + ZSTD_cwksp_align(dictSize, (nuint)sizeof(void*)) + ); + if (internalBuffer == null) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + + cdict->dictContent = internalBuffer; + memcpy(internalBuffer, dictBuffer, (uint)dictSize); + } + + cdict->dictContentSize = dictSize; + cdict->dictContentType = dictContentType; + cdict->entropyWorkspace = (uint*)ZSTD_cwksp_reserve_object( + &cdict->workspace, + (8 << 10) + 512 + ); + ZSTD_reset_compressedBlockState(&cdict->cBlockState); + { + nuint err_code = ZSTD_reset_matchState( + &cdict->matchState, + &cdict->workspace, + &@params.cParams, + @params.useRowMatchFinder, + ZSTD_compResetPolicy_e.ZSTDcrp_makeClean, + ZSTD_indexResetPolicy_e.ZSTDirp_reset, + ZSTD_resetTarget_e.ZSTD_resetTarget_CDict + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + @params.compressionLevel = 3; + @params.fParams.contentSizeFlag = 1; + { + nuint dictID = ZSTD_compress_insertDictionary( + &cdict->cBlockState, + &cdict->matchState, + null, + &cdict->workspace, + &@params, + cdict->dictContent, + cdict->dictContentSize, + dictContentType, + ZSTD_dictTableLoadMethod_e.ZSTD_dtlm_full, + ZSTD_tableFillPurpose_e.ZSTD_tfp_forCDict, + cdict->entropyWorkspace + ); + { + nuint err_code = dictID; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + assert(dictID <= unchecked((uint)-1)); + cdict->dictID = (uint)dictID; + } + } + + return 0; + } + + private static ZSTD_CDict_s* ZSTD_createCDict_advanced_internal( + nuint dictSize, + ZSTD_dictLoadMethod_e dictLoadMethod, + ZSTD_compressionParameters cParams, + ZSTD_paramSwitch_e useRowMatchFinder, + int enableDedicatedDictSearch, + ZSTD_customMem customMem + ) + { + if (((customMem.customAlloc == null ? 1 : 0) ^ (customMem.customFree == null ? 1 : 0)) != 0) + { + return null; + } + + { + nuint workspaceSize = + ZSTD_cwksp_alloc_size((nuint)sizeof(ZSTD_CDict_s)) + + ZSTD_cwksp_alloc_size((8 << 10) + 512) + + ZSTD_sizeof_matchState(&cParams, useRowMatchFinder, enableDedicatedDictSearch, 0) + + ( + dictLoadMethod == ZSTD_dictLoadMethod_e.ZSTD_dlm_byRef + ? 0 + : ZSTD_cwksp_alloc_size(ZSTD_cwksp_align(dictSize, (nuint)sizeof(void*))) + ); + void* workspace = ZSTD_customMalloc(workspaceSize, customMem); + ZSTD_cwksp ws; + ZSTD_CDict_s* cdict; + if (workspace == null) + { + ZSTD_customFree(workspace, customMem); + return null; + } + + ZSTD_cwksp_init( + &ws, + workspace, + workspaceSize, + ZSTD_cwksp_static_alloc_e.ZSTD_cwksp_dynamic_alloc + ); + cdict = (ZSTD_CDict_s*)ZSTD_cwksp_reserve_object(&ws, (nuint)sizeof(ZSTD_CDict_s)); + assert(cdict != null); + ZSTD_cwksp_move(&cdict->workspace, &ws); + cdict->customMem = customMem; + cdict->compressionLevel = 0; + cdict->useRowMatchFinder = useRowMatchFinder; + return cdict; + } + } + + public static ZSTD_CDict_s* ZSTD_createCDict_advanced( + void* dictBuffer, + nuint dictSize, + ZSTD_dictLoadMethod_e dictLoadMethod, + ZSTD_dictContentType_e dictContentType, + ZSTD_compressionParameters cParams, + ZSTD_customMem customMem + ) + { + ZSTD_CCtx_params_s cctxParams; + cctxParams = new ZSTD_CCtx_params_s(); + ZSTD_CCtxParams_init(&cctxParams, 0); + cctxParams.cParams = cParams; + cctxParams.customMem = customMem; + return ZSTD_createCDict_advanced2( + dictBuffer, + dictSize, + dictLoadMethod, + dictContentType, + &cctxParams, + customMem + ); + } + + /* + * This API is temporary and is expected to change or disappear in the future! + */ + public static ZSTD_CDict_s* ZSTD_createCDict_advanced2( + void* dict, + nuint dictSize, + ZSTD_dictLoadMethod_e dictLoadMethod, + ZSTD_dictContentType_e dictContentType, + ZSTD_CCtx_params_s* originalCctxParams, + ZSTD_customMem customMem + ) + { + ZSTD_CCtx_params_s cctxParams = *originalCctxParams; + ZSTD_compressionParameters cParams; + ZSTD_CDict_s* cdict; + if (((customMem.customAlloc == null ? 1 : 0) ^ (customMem.customFree == null ? 1 : 0)) != 0) + { + return null; + } + + if (cctxParams.enableDedicatedDictSearch != 0) + { + cParams = ZSTD_dedicatedDictSearch_getCParams(cctxParams.compressionLevel, dictSize); + ZSTD_overrideCParams(&cParams, &cctxParams.cParams); + } + else + { + cParams = ZSTD_getCParamsFromCCtxParams( + &cctxParams, + unchecked(0UL - 1), + dictSize, + ZSTD_CParamMode_e.ZSTD_cpm_createCDict + ); + } + + if (ZSTD_dedicatedDictSearch_isSupported(&cParams) == 0) + { + cctxParams.enableDedicatedDictSearch = 0; + cParams = ZSTD_getCParamsFromCCtxParams( + &cctxParams, + unchecked(0UL - 1), + dictSize, + ZSTD_CParamMode_e.ZSTD_cpm_createCDict + ); + } + + cctxParams.cParams = cParams; + cctxParams.useRowMatchFinder = ZSTD_resolveRowMatchFinderMode( + cctxParams.useRowMatchFinder, + &cParams + ); + cdict = ZSTD_createCDict_advanced_internal( + dictSize, + dictLoadMethod, + cctxParams.cParams, + cctxParams.useRowMatchFinder, + cctxParams.enableDedicatedDictSearch, + customMem + ); + if ( + cdict == null + || ERR_isError( + ZSTD_initCDict_internal( + cdict, + dict, + dictSize, + dictLoadMethod, + dictContentType, + cctxParams + ) + ) + ) + { + ZSTD_freeCDict(cdict); + return null; + } + + return cdict; + } + + /*! ZSTD_createCDict() : + * When compressing multiple messages or blocks using the same dictionary, + * it's recommended to digest the dictionary only once, since it's a costly operation. + * ZSTD_createCDict() will create a state from digesting a dictionary. + * The resulting state can be used for future compression operations with very limited startup cost. + * ZSTD_CDict can be created once and shared by multiple threads concurrently, since its usage is read-only. + * @dictBuffer can be released after ZSTD_CDict creation, because its content is copied within CDict. + * Note 1 : Consider experimental function `ZSTD_createCDict_byReference()` if you prefer to not duplicate @dictBuffer content. + * Note 2 : A ZSTD_CDict can be created from an empty @dictBuffer, + * in which case the only thing that it transports is the @compressionLevel. + * This can be useful in a pipeline featuring ZSTD_compress_usingCDict() exclusively, + * expecting a ZSTD_CDict parameter with any data, including those without a known dictionary. */ + public static ZSTD_CDict_s* ZSTD_createCDict(void* dict, nuint dictSize, int compressionLevel) + { + ZSTD_compressionParameters cParams = ZSTD_getCParams_internal( + compressionLevel, + unchecked(0UL - 1), + dictSize, + ZSTD_CParamMode_e.ZSTD_cpm_createCDict + ); + ZSTD_CDict_s* cdict = ZSTD_createCDict_advanced( + dict, + dictSize, + ZSTD_dictLoadMethod_e.ZSTD_dlm_byCopy, + ZSTD_dictContentType_e.ZSTD_dct_auto, + cParams, + ZSTD_defaultCMem + ); + if (cdict != null) + { + cdict->compressionLevel = compressionLevel == 0 ? 3 : compressionLevel; + } + + return cdict; + } + + /*! ZSTD_createCDict_byReference() : + * Create a digested dictionary for compression + * Dictionary content is just referenced, not duplicated. + * As a consequence, `dictBuffer` **must** outlive CDict, + * and its content must remain unmodified throughout the lifetime of CDict. + * note: equivalent to ZSTD_createCDict_advanced(), with dictLoadMethod==ZSTD_dlm_byRef */ + public static ZSTD_CDict_s* ZSTD_createCDict_byReference( + void* dict, + nuint dictSize, + int compressionLevel + ) + { + ZSTD_compressionParameters cParams = ZSTD_getCParams_internal( + compressionLevel, + unchecked(0UL - 1), + dictSize, + ZSTD_CParamMode_e.ZSTD_cpm_createCDict + ); + ZSTD_CDict_s* cdict = ZSTD_createCDict_advanced( + dict, + dictSize, + ZSTD_dictLoadMethod_e.ZSTD_dlm_byRef, + ZSTD_dictContentType_e.ZSTD_dct_auto, + cParams, + ZSTD_defaultCMem + ); + if (cdict != null) + { + cdict->compressionLevel = compressionLevel == 0 ? 3 : compressionLevel; + } + + return cdict; + } + + /*! ZSTD_freeCDict() : + * Function frees memory allocated by ZSTD_createCDict(). + * If a NULL pointer is passed, no operation is performed. */ + public static nuint ZSTD_freeCDict(ZSTD_CDict_s* cdict) + { + if (cdict == null) + { + return 0; + } + + { + ZSTD_customMem cMem = cdict->customMem; + int cdictInWorkspace = ZSTD_cwksp_owns_buffer(&cdict->workspace, cdict); + ZSTD_cwksp_free(&cdict->workspace, cMem); + if (cdictInWorkspace == 0) + { + ZSTD_customFree(cdict, cMem); + } + + return 0; + } + } + + /*! ZSTD_initStaticCDict_advanced() : + * Generate a digested dictionary in provided memory area. + * workspace: The memory area to emplace the dictionary into. + * Provided pointer must 8-bytes aligned. + * It must outlive dictionary usage. + * workspaceSize: Use ZSTD_estimateCDictSize() + * to determine how large workspace must be. + * cParams : use ZSTD_getCParams() to transform a compression level + * into its relevant cParams. + * @return : pointer to ZSTD_CDict*, or NULL if error (size too small) + * Note : there is no corresponding "free" function. + * Since workspace was allocated externally, it must be freed externally. + */ + public static ZSTD_CDict_s* ZSTD_initStaticCDict( + void* workspace, + nuint workspaceSize, + void* dict, + nuint dictSize, + ZSTD_dictLoadMethod_e dictLoadMethod, + ZSTD_dictContentType_e dictContentType, + ZSTD_compressionParameters cParams + ) + { + ZSTD_paramSwitch_e useRowMatchFinder = ZSTD_resolveRowMatchFinderMode( + ZSTD_paramSwitch_e.ZSTD_ps_auto, + &cParams + ); + /* enableDedicatedDictSearch */ + nuint matchStateSize = ZSTD_sizeof_matchState(&cParams, useRowMatchFinder, 1, 0); + nuint neededSize = + ZSTD_cwksp_alloc_size((nuint)sizeof(ZSTD_CDict_s)) + + ( + dictLoadMethod == ZSTD_dictLoadMethod_e.ZSTD_dlm_byRef + ? 0 + : ZSTD_cwksp_alloc_size(ZSTD_cwksp_align(dictSize, (nuint)sizeof(void*))) + ) + + ZSTD_cwksp_alloc_size((8 << 10) + 512) + + matchStateSize; + ZSTD_CDict_s* cdict; + ZSTD_CCtx_params_s @params; + if (((nuint)workspace & 7) != 0) + { + return null; + } + + { + ZSTD_cwksp ws; + ZSTD_cwksp_init( + &ws, + workspace, + workspaceSize, + ZSTD_cwksp_static_alloc_e.ZSTD_cwksp_static_alloc + ); + cdict = (ZSTD_CDict_s*)ZSTD_cwksp_reserve_object(&ws, (nuint)sizeof(ZSTD_CDict_s)); + if (cdict == null) + { + return null; + } + + ZSTD_cwksp_move(&cdict->workspace, &ws); + } + + if (workspaceSize < neededSize) + { + return null; + } + + ZSTD_CCtxParams_init(&@params, 0); + @params.cParams = cParams; + @params.useRowMatchFinder = useRowMatchFinder; + cdict->useRowMatchFinder = useRowMatchFinder; + cdict->compressionLevel = 0; + if ( + ERR_isError( + ZSTD_initCDict_internal( + cdict, + dict, + dictSize, + dictLoadMethod, + dictContentType, + @params + ) + ) + ) + { + return null; + } + + return cdict; + } + + /*! ZSTD_getCParamsFromCDict() : + * as the name implies */ + private static ZSTD_compressionParameters ZSTD_getCParamsFromCDict(ZSTD_CDict_s* cdict) + { + assert(cdict != null); + return cdict->matchState.cParams; + } + + /*! ZSTD_getDictID_fromCDict() : + * Provides the dictID of the dictionary loaded into `cdict`. + * If @return == 0, the dictionary is not conformant to Zstandard specification, or empty. + * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */ + public static uint ZSTD_getDictID_fromCDict(ZSTD_CDict_s* cdict) + { + if (cdict == null) + { + return 0; + } + + return cdict->dictID; + } + + /* ZSTD_compressBegin_usingCDict_internal() : + * Implementation of various ZSTD_compressBegin_usingCDict* functions. + */ + private static nuint ZSTD_compressBegin_usingCDict_internal( + ZSTD_CCtx_s* cctx, + ZSTD_CDict_s* cdict, + ZSTD_frameParameters fParams, + ulong pledgedSrcSize + ) + { + ZSTD_CCtx_params_s cctxParams; + if (cdict == null) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_wrong)); + } + + { + ZSTD_parameters @params; + @params.fParams = fParams; + @params.cParams = + pledgedSrcSize < 128 * (1 << 10) + || pledgedSrcSize < cdict->dictContentSize * 6UL + || pledgedSrcSize == unchecked(0UL - 1) + || cdict->compressionLevel == 0 + ? ZSTD_getCParamsFromCDict(cdict) + : ZSTD_getCParams( + cdict->compressionLevel, + pledgedSrcSize, + cdict->dictContentSize + ); + ZSTD_CCtxParams_init_internal(&cctxParams, &@params, cdict->compressionLevel); + } + + if (pledgedSrcSize != unchecked(0UL - 1)) + { + uint limitedSrcSize = (uint)(pledgedSrcSize < 1U << 19 ? pledgedSrcSize : 1U << 19); + uint limitedSrcLog = limitedSrcSize > 1 ? ZSTD_highbit32(limitedSrcSize - 1) + 1 : 1; + cctxParams.cParams.windowLog = + cctxParams.cParams.windowLog > limitedSrcLog + ? cctxParams.cParams.windowLog + : limitedSrcLog; + } + + return ZSTD_compressBegin_internal( + cctx, + null, + 0, + ZSTD_dictContentType_e.ZSTD_dct_auto, + ZSTD_dictTableLoadMethod_e.ZSTD_dtlm_fast, + cdict, + &cctxParams, + pledgedSrcSize, + ZSTD_buffered_policy_e.ZSTDb_not_buffered + ); + } + + /* ZSTD_compressBegin_usingCDict_advanced() : + * This function is DEPRECATED. + * cdict must be != NULL */ + public static nuint ZSTD_compressBegin_usingCDict_advanced( + ZSTD_CCtx_s* cctx, + ZSTD_CDict_s* cdict, + ZSTD_frameParameters fParams, + ulong pledgedSrcSize + ) + { + return ZSTD_compressBegin_usingCDict_internal(cctx, cdict, fParams, pledgedSrcSize); + } + + /* ZSTD_compressBegin_usingCDict() : + * cdict must be != NULL */ + private static nuint ZSTD_compressBegin_usingCDict_deprecated( + ZSTD_CCtx_s* cctx, + ZSTD_CDict_s* cdict + ) + { + /*content*/ + ZSTD_frameParameters fParams = new ZSTD_frameParameters + { + contentSizeFlag = 0, + checksumFlag = 0, + noDictIDFlag = 0, + }; + return ZSTD_compressBegin_usingCDict_internal(cctx, cdict, fParams, unchecked(0UL - 1)); + } + + public static nuint ZSTD_compressBegin_usingCDict(ZSTD_CCtx_s* cctx, ZSTD_CDict_s* cdict) + { + return ZSTD_compressBegin_usingCDict_deprecated(cctx, cdict); + } + + /*! ZSTD_compress_usingCDict_internal(): + * Implementation of various ZSTD_compress_usingCDict* functions. + */ + private static nuint ZSTD_compress_usingCDict_internal( + ZSTD_CCtx_s* cctx, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize, + ZSTD_CDict_s* cdict, + ZSTD_frameParameters fParams + ) + { + { + /* will check if cdict != NULL */ + nuint err_code = ZSTD_compressBegin_usingCDict_internal(cctx, cdict, fParams, srcSize); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return ZSTD_compressEnd_public(cctx, dst, dstCapacity, src, srcSize); + } + + /*! ZSTD_compress_usingCDict_advanced(): + * This function is DEPRECATED. + */ + public static nuint ZSTD_compress_usingCDict_advanced( + ZSTD_CCtx_s* cctx, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize, + ZSTD_CDict_s* cdict, + ZSTD_frameParameters fParams + ) + { + return ZSTD_compress_usingCDict_internal( + cctx, + dst, + dstCapacity, + src, + srcSize, + cdict, + fParams + ); + } + + /*! ZSTD_compress_usingCDict() : + * Compression using a digested Dictionary. + * Faster startup than ZSTD_compress_usingDict(), recommended when same dictionary is used multiple times. + * Note that compression parameters are decided at CDict creation time + * while frame parameters are hardcoded */ + public static nuint ZSTD_compress_usingCDict( + ZSTD_CCtx_s* cctx, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize, + ZSTD_CDict_s* cdict + ) + { + /*content*/ + ZSTD_frameParameters fParams = new ZSTD_frameParameters + { + contentSizeFlag = 1, + checksumFlag = 0, + noDictIDFlag = 0, + }; + return ZSTD_compress_usingCDict_internal( + cctx, + dst, + dstCapacity, + src, + srcSize, + cdict, + fParams + ); + } + + /* ****************************************************************** + * Streaming + ********************************************************************/ + public static ZSTD_CCtx_s* ZSTD_createCStream() + { + return ZSTD_createCStream_advanced(ZSTD_defaultCMem); + } + + public static ZSTD_CCtx_s* ZSTD_initStaticCStream(void* workspace, nuint workspaceSize) + { + return ZSTD_initStaticCCtx(workspace, workspaceSize); + } + + public static ZSTD_CCtx_s* ZSTD_createCStream_advanced(ZSTD_customMem customMem) + { + return ZSTD_createCCtx_advanced(customMem); + } + + public static nuint ZSTD_freeCStream(ZSTD_CCtx_s* zcs) + { + return ZSTD_freeCCtx(zcs); + } + + /*====== Initialization ======*/ + public static nuint ZSTD_CStreamInSize() + { + return 1 << 17; + } + + public static nuint ZSTD_CStreamOutSize() + { + return ZSTD_compressBound(1 << 17) + ZSTD_blockHeaderSize + 4; + } + + private static ZSTD_CParamMode_e ZSTD_getCParamMode( + ZSTD_CDict_s* cdict, + ZSTD_CCtx_params_s* @params, + ulong pledgedSrcSize + ) + { + if (cdict != null && ZSTD_shouldAttachDict(cdict, @params, pledgedSrcSize) != 0) + { + return ZSTD_CParamMode_e.ZSTD_cpm_attachDict; + } + else + { + return ZSTD_CParamMode_e.ZSTD_cpm_noAttachDict; + } + } + + /* ZSTD_resetCStream(): + * pledgedSrcSize == 0 means "unknown" */ + public static nuint ZSTD_resetCStream(ZSTD_CCtx_s* zcs, ulong pss) + { + /* temporary : 0 interpreted as "unknown" during transition period. + * Users willing to specify "unknown" **must** use ZSTD_CONTENTSIZE_UNKNOWN. + * 0 will be interpreted as "empty" in the future. + */ + ulong pledgedSrcSize = pss == 0 ? unchecked(0UL - 1) : pss; + { + nuint err_code = ZSTD_CCtx_reset(zcs, ZSTD_ResetDirective.ZSTD_reset_session_only); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return 0; + } + + /*! ZSTD_initCStream_internal() : + * Note : for lib/compress only. Used by zstdmt_compress.c. + * Assumption 1 : params are valid + * Assumption 2 : either dict, or cdict, is defined, not both */ + private static nuint ZSTD_initCStream_internal( + ZSTD_CCtx_s* zcs, + void* dict, + nuint dictSize, + ZSTD_CDict_s* cdict, + ZSTD_CCtx_params_s* @params, + ulong pledgedSrcSize + ) + { + { + nuint err_code = ZSTD_CCtx_reset(zcs, ZSTD_ResetDirective.ZSTD_reset_session_only); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + assert(!ERR_isError(ZSTD_checkCParams(@params->cParams))); + zcs->requestedParams = *@params; + assert(!(dict != null && cdict != null)); + if (dict != null) + { + nuint err_code = ZSTD_CCtx_loadDictionary(zcs, dict, dictSize); + if (ERR_isError(err_code)) + { + return err_code; + } + } + else + { + /* Dictionary is cleared if !cdict */ + nuint err_code = ZSTD_CCtx_refCDict(zcs, cdict); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return 0; + } + + /* ZSTD_initCStream_usingCDict_advanced() : + * same as ZSTD_initCStream_usingCDict(), with control over frame parameters */ + public static nuint ZSTD_initCStream_usingCDict_advanced( + ZSTD_CCtx_s* zcs, + ZSTD_CDict_s* cdict, + ZSTD_frameParameters fParams, + ulong pledgedSrcSize + ) + { + { + nuint err_code = ZSTD_CCtx_reset(zcs, ZSTD_ResetDirective.ZSTD_reset_session_only); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + zcs->requestedParams.fParams = fParams; + { + nuint err_code = ZSTD_CCtx_refCDict(zcs, cdict); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return 0; + } + + /* note : cdict must outlive compression session */ + public static nuint ZSTD_initCStream_usingCDict(ZSTD_CCtx_s* zcs, ZSTD_CDict_s* cdict) + { + { + nuint err_code = ZSTD_CCtx_reset(zcs, ZSTD_ResetDirective.ZSTD_reset_session_only); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_CCtx_refCDict(zcs, cdict); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return 0; + } + + /* ZSTD_initCStream_advanced() : + * pledgedSrcSize must be exact. + * if srcSize is not known at init time, use value ZSTD_CONTENTSIZE_UNKNOWN. + * dict is loaded with default parameters ZSTD_dct_auto and ZSTD_dlm_byCopy. */ + public static nuint ZSTD_initCStream_advanced( + ZSTD_CCtx_s* zcs, + void* dict, + nuint dictSize, + ZSTD_parameters @params, + ulong pss + ) + { + /* for compatibility with older programs relying on this behavior. + * Users should now specify ZSTD_CONTENTSIZE_UNKNOWN. + * This line will be removed in the future. + */ + ulong pledgedSrcSize = + pss == 0 && @params.fParams.contentSizeFlag == 0 ? unchecked(0UL - 1) : pss; + { + nuint err_code = ZSTD_CCtx_reset(zcs, ZSTD_ResetDirective.ZSTD_reset_session_only); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_checkCParams(@params.cParams); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + ZSTD_CCtxParams_setZstdParams(&zcs->requestedParams, &@params); + { + nuint err_code = ZSTD_CCtx_loadDictionary(zcs, dict, dictSize); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return 0; + } + + /*! ZSTD_initCStream_usingDict() : + * This function is DEPRECATED, and is equivalent to: + * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); + * ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel); + * ZSTD_CCtx_loadDictionary(zcs, dict, dictSize); + * + * Creates of an internal CDict (incompatible with static CCtx), except if + * dict == NULL or dictSize < 8, in which case no dict is used. + * Note: dict is loaded with ZSTD_dct_auto (treated as a full zstd dictionary if + * it begins with ZSTD_MAGIC_DICTIONARY, else as raw content) and ZSTD_dlm_byCopy. + * This prototype will generate compilation warnings. + */ + public static nuint ZSTD_initCStream_usingDict( + ZSTD_CCtx_s* zcs, + void* dict, + nuint dictSize, + int compressionLevel + ) + { + { + nuint err_code = ZSTD_CCtx_reset(zcs, ZSTD_ResetDirective.ZSTD_reset_session_only); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_CCtx_setParameter( + zcs, + ZSTD_cParameter.ZSTD_c_compressionLevel, + compressionLevel + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_CCtx_loadDictionary(zcs, dict, dictSize); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return 0; + } + + /*! ZSTD_initCStream_srcSize() : + * This function is DEPRECATED, and equivalent to: + * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); + * ZSTD_CCtx_refCDict(zcs, NULL); // clear the dictionary (if any) + * ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel); + * ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize); + * + * pledgedSrcSize must be correct. If it is not known at init time, use + * ZSTD_CONTENTSIZE_UNKNOWN. Note that, for compatibility with older programs, + * "0" also disables frame content size field. It may be enabled in the future. + * This prototype will generate compilation warnings. + */ + public static nuint ZSTD_initCStream_srcSize(ZSTD_CCtx_s* zcs, int compressionLevel, ulong pss) + { + /* temporary : 0 interpreted as "unknown" during transition period. + * Users willing to specify "unknown" **must** use ZSTD_CONTENTSIZE_UNKNOWN. + * 0 will be interpreted as "empty" in the future. + */ + ulong pledgedSrcSize = pss == 0 ? unchecked(0UL - 1) : pss; + { + nuint err_code = ZSTD_CCtx_reset(zcs, ZSTD_ResetDirective.ZSTD_reset_session_only); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_CCtx_refCDict(zcs, null); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_CCtx_setParameter( + zcs, + ZSTD_cParameter.ZSTD_c_compressionLevel, + compressionLevel + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return 0; + } + + /*! + * Equivalent to: + * + * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); + * ZSTD_CCtx_refCDict(zcs, NULL); // clear the dictionary (if any) + * ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel); + * + * Note that ZSTD_initCStream() clears any previously set dictionary. Use the new API + * to compress with a dictionary. + */ + public static nuint ZSTD_initCStream(ZSTD_CCtx_s* zcs, int compressionLevel) + { + { + nuint err_code = ZSTD_CCtx_reset(zcs, ZSTD_ResetDirective.ZSTD_reset_session_only); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_CCtx_refCDict(zcs, null); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_CCtx_setParameter( + zcs, + ZSTD_cParameter.ZSTD_c_compressionLevel, + compressionLevel + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return 0; + } + + /*====== Compression ======*/ + private static nuint ZSTD_nextInputSizeHint(ZSTD_CCtx_s* cctx) + { + if (cctx->appliedParams.inBufferMode == ZSTD_bufferMode_e.ZSTD_bm_stable) + { + return cctx->blockSizeMax - cctx->stableIn_notConsumed; + } + + assert(cctx->appliedParams.inBufferMode == ZSTD_bufferMode_e.ZSTD_bm_buffered); + { + nuint hintInSize = cctx->inBuffTarget - cctx->inBuffPos; + if (hintInSize == 0) + { + hintInSize = cctx->blockSizeMax; + } + + return hintInSize; + } + } + + /** ZSTD_compressStream_generic(): + * internal function for all *compressStream*() variants + * @return : hint size for next input to complete ongoing block */ + private static nuint ZSTD_compressStream_generic( + ZSTD_CCtx_s* zcs, + ZSTD_outBuffer_s* output, + ZSTD_inBuffer_s* input, + ZSTD_EndDirective flushMode + ) + { + assert(input != null); + sbyte* istart = (sbyte*)input->src; + sbyte* iend = istart != null ? istart + input->size : istart; + sbyte* ip = istart != null ? istart + input->pos : istart; + assert(output != null); + sbyte* ostart = (sbyte*)output->dst; + sbyte* oend = ostart != null ? ostart + output->size : ostart; + sbyte* op = ostart != null ? ostart + output->pos : ostart; + uint someMoreWork = 1; + assert(zcs != null); + if (zcs->appliedParams.inBufferMode == ZSTD_bufferMode_e.ZSTD_bm_stable) + { + assert(input->pos >= zcs->stableIn_notConsumed); + input->pos -= zcs->stableIn_notConsumed; + if (ip != null) + { + ip -= zcs->stableIn_notConsumed; + } + + zcs->stableIn_notConsumed = 0; + } + + assert(input->pos <= input->size); + assert(output->pos <= output->size); + assert((uint)flushMode <= (uint)ZSTD_EndDirective.ZSTD_e_end); + while (someMoreWork != 0) + { + switch (zcs->streamStage) + { + case ZSTD_cStreamStage.zcss_init: + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_init_missing)); + case ZSTD_cStreamStage.zcss_load: + if ( + flushMode == ZSTD_EndDirective.ZSTD_e_end + && ( + (nuint)(oend - op) >= ZSTD_compressBound((nuint)(iend - ip)) + || zcs->appliedParams.outBufferMode == ZSTD_bufferMode_e.ZSTD_bm_stable + ) + && zcs->inBuffPos == 0 + ) + { + /* shortcut to compression pass directly into output buffer */ + nuint cSize = ZSTD_compressEnd_public( + zcs, + op, + (nuint)(oend - op), + ip, + (nuint)(iend - ip) + ); + { + nuint err_code = cSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + ip = iend; + op += cSize; + zcs->frameEnded = 1; + ZSTD_CCtx_reset(zcs, ZSTD_ResetDirective.ZSTD_reset_session_only); + someMoreWork = 0; + break; + } + + if (zcs->appliedParams.inBufferMode == ZSTD_bufferMode_e.ZSTD_bm_buffered) + { + nuint toLoad = zcs->inBuffTarget - zcs->inBuffPos; + nuint loaded = ZSTD_limitCopy( + zcs->inBuff + zcs->inBuffPos, + toLoad, + ip, + (nuint)(iend - ip) + ); + zcs->inBuffPos += loaded; + if (ip != null) + { + ip += loaded; + } + + if ( + flushMode == ZSTD_EndDirective.ZSTD_e_continue + && zcs->inBuffPos < zcs->inBuffTarget + ) + { + someMoreWork = 0; + break; + } + + if ( + flushMode == ZSTD_EndDirective.ZSTD_e_flush + && zcs->inBuffPos == zcs->inToCompress + ) + { + someMoreWork = 0; + break; + } + } + else + { + assert(zcs->appliedParams.inBufferMode == ZSTD_bufferMode_e.ZSTD_bm_stable); + if ( + flushMode == ZSTD_EndDirective.ZSTD_e_continue + && (nuint)(iend - ip) < zcs->blockSizeMax + ) + { + zcs->stableIn_notConsumed = (nuint)(iend - ip); + ip = iend; + someMoreWork = 0; + break; + } + + if (flushMode == ZSTD_EndDirective.ZSTD_e_flush && ip == iend) + { + someMoreWork = 0; + break; + } + } + + { + int inputBuffered = + zcs->appliedParams.inBufferMode == ZSTD_bufferMode_e.ZSTD_bm_buffered + ? 1 + : 0; + void* cDst; + nuint cSize; + nuint oSize = (nuint)(oend - op); + nuint iSize = + inputBuffered != 0 ? zcs->inBuffPos - zcs->inToCompress + : (nuint)(iend - ip) < zcs->blockSizeMax ? (nuint)(iend - ip) + : zcs->blockSizeMax; + if ( + oSize >= ZSTD_compressBound(iSize) + || zcs->appliedParams.outBufferMode == ZSTD_bufferMode_e.ZSTD_bm_stable + ) + { + cDst = op; + } + else + { + cDst = zcs->outBuff; + oSize = zcs->outBuffSize; + } + + if (inputBuffered != 0) + { + uint lastBlock = + flushMode == ZSTD_EndDirective.ZSTD_e_end && ip == iend ? 1U : 0U; + cSize = + lastBlock != 0 + ? ZSTD_compressEnd_public( + zcs, + cDst, + oSize, + zcs->inBuff + zcs->inToCompress, + iSize + ) + : ZSTD_compressContinue_public( + zcs, + cDst, + oSize, + zcs->inBuff + zcs->inToCompress, + iSize + ); + { + nuint err_code = cSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + zcs->frameEnded = lastBlock; + zcs->inBuffTarget = zcs->inBuffPos + zcs->blockSizeMax; + if (zcs->inBuffTarget > zcs->inBuffSize) + { + zcs->inBuffPos = 0; + zcs->inBuffTarget = zcs->blockSizeMax; + } + + zcs->inToCompress = zcs->inBuffPos; + } + else + { + uint lastBlock = + flushMode == ZSTD_EndDirective.ZSTD_e_end && ip + iSize == iend + ? 1U + : 0U; + cSize = + lastBlock != 0 + ? ZSTD_compressEnd_public(zcs, cDst, oSize, ip, iSize) + : ZSTD_compressContinue_public(zcs, cDst, oSize, ip, iSize); + if (ip != null) + { + ip += iSize; + } + + { + nuint err_code = cSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + zcs->frameEnded = lastBlock; + } + + if (cDst == op) + { + op += cSize; + if (zcs->frameEnded != 0) + { + someMoreWork = 0; + ZSTD_CCtx_reset(zcs, ZSTD_ResetDirective.ZSTD_reset_session_only); + } + + break; + } + + zcs->outBuffContentSize = cSize; + zcs->outBuffFlushedSize = 0; + zcs->streamStage = ZSTD_cStreamStage.zcss_flush; + } + + goto case ZSTD_cStreamStage.zcss_flush; + case ZSTD_cStreamStage.zcss_flush: + assert(zcs->appliedParams.outBufferMode == ZSTD_bufferMode_e.ZSTD_bm_buffered); + + { + nuint toFlush = zcs->outBuffContentSize - zcs->outBuffFlushedSize; + nuint flushed = ZSTD_limitCopy( + op, + (nuint)(oend - op), + zcs->outBuff + zcs->outBuffFlushedSize, + toFlush + ); + if (flushed != 0) + { + op += flushed; + } + + zcs->outBuffFlushedSize += flushed; + if (toFlush != flushed) + { + assert(op == oend); + someMoreWork = 0; + break; + } + + zcs->outBuffContentSize = zcs->outBuffFlushedSize = 0; + if (zcs->frameEnded != 0) + { + someMoreWork = 0; + ZSTD_CCtx_reset(zcs, ZSTD_ResetDirective.ZSTD_reset_session_only); + break; + } + + zcs->streamStage = ZSTD_cStreamStage.zcss_load; + break; + } + + default: + assert(0 != 0); + break; + } + } + + input->pos = (nuint)(ip - istart); + output->pos = (nuint)(op - ostart); + if (zcs->frameEnded != 0) + { + return 0; + } + + return ZSTD_nextInputSizeHint(zcs); + } + + private static nuint ZSTD_nextInputSizeHint_MTorST(ZSTD_CCtx_s* cctx) + { + if (cctx->appliedParams.nbWorkers >= 1) + { + assert(cctx->mtctx != null); + return ZSTDMT_nextInputSizeHint(cctx->mtctx); + } + + return ZSTD_nextInputSizeHint(cctx); + } + + /*! + * Alternative for ZSTD_compressStream2(zcs, output, input, ZSTD_e_continue). + * NOTE: The return value is different. ZSTD_compressStream() returns a hint for + * the next read size (if non-zero and not an error). ZSTD_compressStream2() + * returns the minimum nb of bytes left to flush (if non-zero and not an error). + */ + public static nuint ZSTD_compressStream( + ZSTD_CCtx_s* zcs, + ZSTD_outBuffer_s* output, + ZSTD_inBuffer_s* input + ) + { + { + nuint err_code = ZSTD_compressStream2( + zcs, + output, + input, + ZSTD_EndDirective.ZSTD_e_continue + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return ZSTD_nextInputSizeHint_MTorST(zcs); + } + + /* After a compression call set the expected input/output buffer. + * This is validated at the start of the next compression call. + */ + private static void ZSTD_setBufferExpectations( + ZSTD_CCtx_s* cctx, + ZSTD_outBuffer_s* output, + ZSTD_inBuffer_s* input + ) + { + if (cctx->appliedParams.inBufferMode == ZSTD_bufferMode_e.ZSTD_bm_stable) + { + cctx->expectedInBuffer = *input; + } + + if (cctx->appliedParams.outBufferMode == ZSTD_bufferMode_e.ZSTD_bm_stable) + { + cctx->expectedOutBufferSize = output->size - output->pos; + } + } + + /* Validate that the input/output buffers match the expectations set by + * ZSTD_setBufferExpectations. + */ + private static nuint ZSTD_checkBufferStability( + ZSTD_CCtx_s* cctx, + ZSTD_outBuffer_s* output, + ZSTD_inBuffer_s* input, + ZSTD_EndDirective endOp + ) + { + if (cctx->appliedParams.inBufferMode == ZSTD_bufferMode_e.ZSTD_bm_stable) + { + ZSTD_inBuffer_s expect = cctx->expectedInBuffer; + if (expect.src != input->src || expect.pos != input->pos) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_stabilityCondition_notRespected) + ); + } + } + + if (cctx->appliedParams.outBufferMode == ZSTD_bufferMode_e.ZSTD_bm_stable) + { + nuint outBufferSize = output->size - output->pos; + if (cctx->expectedOutBufferSize != outBufferSize) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_stabilityCondition_notRespected) + ); + } + } + + return 0; + } + + /* + * If @endOp == ZSTD_e_end, @inSize becomes pledgedSrcSize. + * Otherwise, it's ignored. + * @return: 0 on success, or a ZSTD_error code otherwise. + */ + private static nuint ZSTD_CCtx_init_compressStream2( + ZSTD_CCtx_s* cctx, + ZSTD_EndDirective endOp, + nuint inSize + ) + { + ZSTD_CCtx_params_s @params = cctx->requestedParams; + ZSTD_prefixDict_s prefixDict = cctx->prefixDict; + { + /* Init the local dict if present. */ + nuint err_code = ZSTD_initLocalDict(cctx); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + cctx->prefixDict = new ZSTD_prefixDict_s(); + assert(prefixDict.dict == null || cctx->cdict == null); + if (cctx->cdict != null && cctx->localDict.cdict == null) + { + @params.compressionLevel = cctx->cdict->compressionLevel; + } + + if (endOp == ZSTD_EndDirective.ZSTD_e_end) + { + cctx->pledgedSrcSizePlusOne = inSize + 1; + } + + { + nuint dictSize = + prefixDict.dict != null ? prefixDict.dictSize + : cctx->cdict != null ? cctx->cdict->dictContentSize + : 0; + ZSTD_CParamMode_e mode = ZSTD_getCParamMode( + cctx->cdict, + &@params, + cctx->pledgedSrcSizePlusOne - 1 + ); + @params.cParams = ZSTD_getCParamsFromCCtxParams( + &@params, + cctx->pledgedSrcSizePlusOne - 1, + dictSize, + mode + ); + } + + @params.postBlockSplitter = ZSTD_resolveBlockSplitterMode( + @params.postBlockSplitter, + &@params.cParams + ); + @params.ldmParams.enableLdm = ZSTD_resolveEnableLdm( + @params.ldmParams.enableLdm, + &@params.cParams + ); + @params.useRowMatchFinder = ZSTD_resolveRowMatchFinderMode( + @params.useRowMatchFinder, + &@params.cParams + ); + @params.validateSequences = ZSTD_resolveExternalSequenceValidation( + @params.validateSequences + ); + @params.maxBlockSize = ZSTD_resolveMaxBlockSize(@params.maxBlockSize); + @params.searchForExternalRepcodes = ZSTD_resolveExternalRepcodeSearch( + @params.searchForExternalRepcodes, + @params.compressionLevel + ); + if (ZSTD_hasExtSeqProd(&@params) != 0 && @params.nbWorkers >= 1) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_combination_unsupported) + ); + } + + if (cctx->pledgedSrcSizePlusOne - 1 <= 512 * (1 << 10)) + { + @params.nbWorkers = 0; + } + + if (@params.nbWorkers > 0) + { + if (cctx->mtctx == null) + { + cctx->mtctx = ZSTDMT_createCCtx_advanced( + (uint)@params.nbWorkers, + cctx->customMem, + cctx->pool + ); + if (cctx->mtctx == null) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + } + + { + nuint err_code = ZSTDMT_initCStream_internal( + cctx->mtctx, + prefixDict.dict, + prefixDict.dictSize, + prefixDict.dictContentType, + cctx->cdict, + @params, + cctx->pledgedSrcSizePlusOne - 1 + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + cctx->dictID = cctx->cdict != null ? cctx->cdict->dictID : 0; + cctx->dictContentSize = + cctx->cdict != null ? cctx->cdict->dictContentSize : prefixDict.dictSize; + cctx->consumedSrcSize = 0; + cctx->producedCSize = 0; + cctx->streamStage = ZSTD_cStreamStage.zcss_load; + cctx->appliedParams = @params; + } + else + { + ulong pledgedSrcSize = cctx->pledgedSrcSizePlusOne - 1; + assert(!ERR_isError(ZSTD_checkCParams(@params.cParams))); + { + nuint err_code = ZSTD_compressBegin_internal( + cctx, + prefixDict.dict, + prefixDict.dictSize, + prefixDict.dictContentType, + ZSTD_dictTableLoadMethod_e.ZSTD_dtlm_fast, + cctx->cdict, + &@params, + pledgedSrcSize, + ZSTD_buffered_policy_e.ZSTDb_buffered + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + assert(cctx->appliedParams.nbWorkers == 0); + cctx->inToCompress = 0; + cctx->inBuffPos = 0; + if (cctx->appliedParams.inBufferMode == ZSTD_bufferMode_e.ZSTD_bm_buffered) + { + cctx->inBuffTarget = + cctx->blockSizeMax + (nuint)(cctx->blockSizeMax == pledgedSrcSize ? 1 : 0); + } + else + { + cctx->inBuffTarget = 0; + } + + cctx->outBuffContentSize = cctx->outBuffFlushedSize = 0; + cctx->streamStage = ZSTD_cStreamStage.zcss_load; + cctx->frameEnded = 0; + } + + return 0; + } + + /* @return provides a minimum amount of data remaining to be flushed from internal buffers + */ + public static nuint ZSTD_compressStream2( + ZSTD_CCtx_s* cctx, + ZSTD_outBuffer_s* output, + ZSTD_inBuffer_s* input, + ZSTD_EndDirective endOp + ) + { + if (output->pos > output->size) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + if (input->pos > input->size) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + if ((uint)endOp > (uint)ZSTD_EndDirective.ZSTD_e_end) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + assert(cctx != null); + if (cctx->streamStage == ZSTD_cStreamStage.zcss_init) + { + /* no obligation to start from pos==0 */ + nuint inputSize = input->size - input->pos; + nuint totalInputSize = inputSize + cctx->stableIn_notConsumed; + if ( + cctx->requestedParams.inBufferMode == ZSTD_bufferMode_e.ZSTD_bm_stable + && endOp == ZSTD_EndDirective.ZSTD_e_continue + && totalInputSize < 1 << 17 + ) + { + if (cctx->stableIn_notConsumed != 0) + { + if (input->src != cctx->expectedInBuffer.src) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_stabilityCondition_notRespected) + ); + } + + if (input->pos != cctx->expectedInBuffer.size) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_stabilityCondition_notRespected) + ); + } + } + + input->pos = input->size; + cctx->expectedInBuffer = *input; + cctx->stableIn_notConsumed += inputSize; + return (nuint)(cctx->requestedParams.format == ZSTD_format_e.ZSTD_f_zstd1 ? 6 : 2); + } + + { + nuint err_code = ZSTD_CCtx_init_compressStream2(cctx, endOp, totalInputSize); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + ZSTD_setBufferExpectations(cctx, output, input); + } + + { + /* end of transparent initialization stage */ + nuint err_code = ZSTD_checkBufferStability(cctx, output, input, endOp); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (cctx->appliedParams.nbWorkers > 0) + { + nuint flushMin; + if (cctx->cParamsChanged != 0) + { + ZSTDMT_updateCParams_whileCompressing(cctx->mtctx, &cctx->requestedParams); + cctx->cParamsChanged = 0; + } + + if (cctx->stableIn_notConsumed != 0) + { + assert(cctx->appliedParams.inBufferMode == ZSTD_bufferMode_e.ZSTD_bm_stable); + assert(input->pos >= cctx->stableIn_notConsumed); + input->pos -= cctx->stableIn_notConsumed; + cctx->stableIn_notConsumed = 0; + } + + for (; ; ) + { + nuint ipos = input->pos; + nuint opos = output->pos; + flushMin = ZSTDMT_compressStream_generic(cctx->mtctx, output, input, endOp); + cctx->consumedSrcSize += input->pos - ipos; + cctx->producedCSize += output->pos - opos; + if (ERR_isError(flushMin) || endOp == ZSTD_EndDirective.ZSTD_e_end && flushMin == 0) + { + if (flushMin == 0) + { + ZSTD_CCtx_trace(cctx, 0); + } + + ZSTD_CCtx_reset(cctx, ZSTD_ResetDirective.ZSTD_reset_session_only); + } + + { + nuint err_code = flushMin; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (endOp == ZSTD_EndDirective.ZSTD_e_continue) + { + if ( + input->pos != ipos + || output->pos != opos + || input->pos == input->size + || output->pos == output->size + ) + { + break; + } + } + else + { + assert( + endOp == ZSTD_EndDirective.ZSTD_e_flush + || endOp == ZSTD_EndDirective.ZSTD_e_end + ); + if (flushMin == 0 || output->pos == output->size) + { + break; + } + } + } + + assert( + endOp == ZSTD_EndDirective.ZSTD_e_continue + || flushMin == 0 + || output->pos == output->size + ); + ZSTD_setBufferExpectations(cctx, output, input); + return flushMin; + } + + { + nuint err_code = ZSTD_compressStream_generic(cctx, output, input, endOp); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + ZSTD_setBufferExpectations(cctx, output, input); + return cctx->outBuffContentSize - cctx->outBuffFlushedSize; + } + + /*! ZSTD_compressStream2_simpleArgs() : + * Same as ZSTD_compressStream2(), + * but using only integral types as arguments. + * This variant might be helpful for binders from dynamic languages + * which have troubles handling structures containing memory pointers. + */ + public static nuint ZSTD_compressStream2_simpleArgs( + ZSTD_CCtx_s* cctx, + void* dst, + nuint dstCapacity, + nuint* dstPos, + void* src, + nuint srcSize, + nuint* srcPos, + ZSTD_EndDirective endOp + ) + { + ZSTD_outBuffer_s output; + ZSTD_inBuffer_s input; + output.dst = dst; + output.size = dstCapacity; + output.pos = *dstPos; + input.src = src; + input.size = srcSize; + input.pos = *srcPos; + { + nuint cErr = ZSTD_compressStream2(cctx, &output, &input, endOp); + *dstPos = output.pos; + *srcPos = input.pos; + return cErr; + } + } + + /*! ZSTD_compress2() : + * Behave the same as ZSTD_compressCCtx(), but compression parameters are set using the advanced API. + * (note that this entry point doesn't even expose a compression level parameter). + * ZSTD_compress2() always starts a new frame. + * Should cctx hold data from a previously unfinished frame, everything about it is forgotten. + * - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*() + * - The function is always blocking, returns when compression is completed. + * NOTE: Providing `dstCapacity >= ZSTD_compressBound(srcSize)` guarantees that zstd will have + * enough space to successfully compress the data, though it is possible it fails for other reasons. + * @return : compressed size written into `dst` (<= `dstCapacity), + * or an error code if it fails (which can be tested using ZSTD_isError()). + */ + public static nuint ZSTD_compress2( + ZSTD_CCtx_s* cctx, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize + ) + { + ZSTD_bufferMode_e originalInBufferMode = cctx->requestedParams.inBufferMode; + ZSTD_bufferMode_e originalOutBufferMode = cctx->requestedParams.outBufferMode; + ZSTD_CCtx_reset(cctx, ZSTD_ResetDirective.ZSTD_reset_session_only); + cctx->requestedParams.inBufferMode = ZSTD_bufferMode_e.ZSTD_bm_stable; + cctx->requestedParams.outBufferMode = ZSTD_bufferMode_e.ZSTD_bm_stable; + { + nuint oPos = 0; + nuint iPos = 0; + nuint result = ZSTD_compressStream2_simpleArgs( + cctx, + dst, + dstCapacity, + &oPos, + src, + srcSize, + &iPos, + ZSTD_EndDirective.ZSTD_e_end + ); + cctx->requestedParams.inBufferMode = originalInBufferMode; + cctx->requestedParams.outBufferMode = originalOutBufferMode; + { + nuint err_code = result; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (result != 0) + { + assert(oPos == dstCapacity); + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + assert(iPos == srcSize); + return oPos; + } + } + + /* ZSTD_validateSequence() : + * @offBase : must use the format required by ZSTD_storeSeq() + * @returns a ZSTD error code if sequence is not valid + */ + private static nuint ZSTD_validateSequence( + uint offBase, + uint matchLength, + uint minMatch, + nuint posInSrc, + uint windowLog, + nuint dictSize, + int useSequenceProducer + ) + { + uint windowSize = 1U << (int)windowLog; + /* posInSrc represents the amount of data the decoder would decode up to this point. + * As long as the amount of data decoded is less than or equal to window size, offsets may be + * larger than the total length of output decoded in order to reference the dict, even larger than + * window size. After output surpasses windowSize, we're limited to windowSize offsets again. + */ + nuint offsetBound = posInSrc > windowSize ? windowSize : posInSrc + dictSize; + nuint matchLenLowerBound = (nuint)(minMatch == 3 || useSequenceProducer != 0 ? 3 : 4); + { + assert(offsetBound > 0); + if (offBase > offsetBound + 3) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_externalSequences_invalid) + ); + } + } + + if (matchLength < matchLenLowerBound) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_externalSequences_invalid)); + } + + return 0; + } + + /* Returns an offset code, given a sequence's raw offset, the ongoing repcode array, and whether litLength == 0 */ + private static uint ZSTD_finalizeOffBase(uint rawOffset, uint* rep, uint ll0) + { + assert(rawOffset > 0); + uint offBase = rawOffset + 3; + if (ll0 == 0 && rawOffset == rep[0]) + { + assert(1 >= 1); + assert(1 <= 3); + offBase = 1; + } + else if (rawOffset == rep[1]) + { + assert(2 - ll0 >= 1); + assert(2 - ll0 <= 3); + offBase = 2 - ll0; + } + else if (rawOffset == rep[2]) + { + assert(3 - ll0 >= 1); + assert(3 - ll0 <= 3); + offBase = 3 - ll0; + } + else if (ll0 != 0 && rawOffset == rep[0] - 1) + { + assert(3 >= 1); + assert(3 <= 3); + offBase = 3; + } + + return offBase; + } + + /* This function scans through an array of ZSTD_Sequence, + * storing the sequences it reads, until it reaches a block delimiter. + * Note that the block delimiter includes the last literals of the block. + * @blockSize must be == sum(sequence_lengths). + * @returns @blockSize on success, and a ZSTD_error otherwise. + */ + private static nuint ZSTD_transferSequences_wBlockDelim( + ZSTD_CCtx_s* cctx, + ZSTD_SequencePosition* seqPos, + ZSTD_Sequence* inSeqs, + nuint inSeqsSize, + void* src, + nuint blockSize, + ZSTD_paramSwitch_e externalRepSearch + ) + { + uint idx = seqPos->idx; + uint startIdx = idx; + byte* ip = (byte*)src; + byte* iend = ip + blockSize; + repcodes_s updatedRepcodes; + uint dictSize; + if (cctx->cdict != null) + { + dictSize = (uint)cctx->cdict->dictContentSize; + } + else if (cctx->prefixDict.dict != null) + { + dictSize = (uint)cctx->prefixDict.dictSize; + } + else + { + dictSize = 0; + } + + memcpy(updatedRepcodes.rep, cctx->blockState.prevCBlock->rep, (uint)sizeof(repcodes_s)); + for (; idx < inSeqsSize && (inSeqs[idx].matchLength != 0 || inSeqs[idx].offset != 0); ++idx) + { + uint litLength = inSeqs[idx].litLength; + uint matchLength = inSeqs[idx].matchLength; + uint offBase; + if (externalRepSearch == ZSTD_paramSwitch_e.ZSTD_ps_disable) + { + assert(inSeqs[idx].offset > 0); + offBase = inSeqs[idx].offset + 3; + } + else + { + uint ll0 = litLength == 0 ? 1U : 0U; + offBase = ZSTD_finalizeOffBase(inSeqs[idx].offset, updatedRepcodes.rep, ll0); + ZSTD_updateRep(updatedRepcodes.rep, offBase, ll0); + } + + if (cctx->appliedParams.validateSequences != 0) + { + seqPos->posInSrc += litLength + matchLength; + { + nuint err_code = ZSTD_validateSequence( + offBase, + matchLength, + cctx->appliedParams.cParams.minMatch, + seqPos->posInSrc, + cctx->appliedParams.cParams.windowLog, + dictSize, + ZSTD_hasExtSeqProd(&cctx->appliedParams) + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + } + + if (idx - seqPos->idx >= cctx->seqStore.maxNbSeq) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_externalSequences_invalid) + ); + } + + ZSTD_storeSeq(&cctx->seqStore, litLength, ip, iend, offBase, matchLength); + ip += matchLength + litLength; + } + + if (idx == inSeqsSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_externalSequences_invalid)); + } + + assert(externalRepSearch != ZSTD_paramSwitch_e.ZSTD_ps_auto); + assert(idx >= startIdx); + if (externalRepSearch == ZSTD_paramSwitch_e.ZSTD_ps_disable && idx != startIdx) + { + uint* rep = updatedRepcodes.rep; + /* index of last non-block-delimiter sequence */ + uint lastSeqIdx = idx - 1; + if (lastSeqIdx >= startIdx + 2) + { + rep[2] = inSeqs[lastSeqIdx - 2].offset; + rep[1] = inSeqs[lastSeqIdx - 1].offset; + rep[0] = inSeqs[lastSeqIdx].offset; + } + else if (lastSeqIdx == startIdx + 1) + { + rep[2] = rep[0]; + rep[1] = inSeqs[lastSeqIdx - 1].offset; + rep[0] = inSeqs[lastSeqIdx].offset; + } + else + { + assert(lastSeqIdx == startIdx); + rep[2] = rep[1]; + rep[1] = rep[0]; + rep[0] = inSeqs[lastSeqIdx].offset; + } + } + + memcpy(cctx->blockState.nextCBlock->rep, updatedRepcodes.rep, (uint)sizeof(repcodes_s)); + if (inSeqs[idx].litLength != 0) + { + ZSTD_storeLastLiterals(&cctx->seqStore, ip, inSeqs[idx].litLength); + ip += inSeqs[idx].litLength; + seqPos->posInSrc += inSeqs[idx].litLength; + } + + if (ip != iend) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_externalSequences_invalid)); + } + + seqPos->idx = idx + 1; + return blockSize; + } + + /* + * This function attempts to scan through @blockSize bytes in @src + * represented by the sequences in @inSeqs, + * storing any (partial) sequences. + * + * Occasionally, we may want to reduce the actual number of bytes consumed from @src + * to avoid splitting a match, notably if it would produce a match smaller than MINMATCH. + * + * @returns the number of bytes consumed from @src, necessarily <= @blockSize. + * Otherwise, it may return a ZSTD error if something went wrong. + */ + private static nuint ZSTD_transferSequences_noDelim( + ZSTD_CCtx_s* cctx, + ZSTD_SequencePosition* seqPos, + ZSTD_Sequence* inSeqs, + nuint inSeqsSize, + void* src, + nuint blockSize, + ZSTD_paramSwitch_e externalRepSearch + ) + { + uint idx = seqPos->idx; + uint startPosInSequence = seqPos->posInSequence; + uint endPosInSequence = seqPos->posInSequence + (uint)blockSize; + nuint dictSize; + byte* istart = (byte*)src; + byte* ip = istart; + /* May be adjusted if we decide to process fewer than blockSize bytes */ + byte* iend = istart + blockSize; + repcodes_s updatedRepcodes; + uint bytesAdjustment = 0; + uint finalMatchSplit = 0; + if (cctx->cdict != null) + { + dictSize = cctx->cdict->dictContentSize; + } + else if (cctx->prefixDict.dict != null) + { + dictSize = cctx->prefixDict.dictSize; + } + else + { + dictSize = 0; + } + + memcpy(updatedRepcodes.rep, cctx->blockState.prevCBlock->rep, (uint)sizeof(repcodes_s)); + while (endPosInSequence != 0 && idx < inSeqsSize && finalMatchSplit == 0) + { + ZSTD_Sequence currSeq = inSeqs[idx]; + uint litLength = currSeq.litLength; + uint matchLength = currSeq.matchLength; + uint rawOffset = currSeq.offset; + uint offBase; + if (endPosInSequence >= currSeq.litLength + currSeq.matchLength) + { + if (startPosInSequence >= litLength) + { + startPosInSequence -= litLength; + litLength = 0; + matchLength -= startPosInSequence; + } + else + { + litLength -= startPosInSequence; + } + + endPosInSequence -= currSeq.litLength + currSeq.matchLength; + startPosInSequence = 0; + } + else + { + if (endPosInSequence > litLength) + { + uint firstHalfMatchLength; + litLength = + startPosInSequence >= litLength ? 0 : litLength - startPosInSequence; + firstHalfMatchLength = endPosInSequence - startPosInSequence - litLength; + if ( + matchLength > blockSize + && firstHalfMatchLength >= cctx->appliedParams.cParams.minMatch + ) + { + /* Only ever split the match if it is larger than the block size */ + uint secondHalfMatchLength = + currSeq.matchLength + currSeq.litLength - endPosInSequence; + if (secondHalfMatchLength < cctx->appliedParams.cParams.minMatch) + { + endPosInSequence -= + cctx->appliedParams.cParams.minMatch - secondHalfMatchLength; + bytesAdjustment = + cctx->appliedParams.cParams.minMatch - secondHalfMatchLength; + firstHalfMatchLength -= bytesAdjustment; + } + + matchLength = firstHalfMatchLength; + finalMatchSplit = 1; + } + else + { + bytesAdjustment = endPosInSequence - currSeq.litLength; + endPosInSequence = currSeq.litLength; + break; + } + } + else + { + break; + } + } + + { + uint ll0 = litLength == 0 ? 1U : 0U; + offBase = ZSTD_finalizeOffBase(rawOffset, updatedRepcodes.rep, ll0); + ZSTD_updateRep(updatedRepcodes.rep, offBase, ll0); + } + + if (cctx->appliedParams.validateSequences != 0) + { + seqPos->posInSrc += litLength + matchLength; + { + nuint err_code = ZSTD_validateSequence( + offBase, + matchLength, + cctx->appliedParams.cParams.minMatch, + seqPos->posInSrc, + cctx->appliedParams.cParams.windowLog, + dictSize, + ZSTD_hasExtSeqProd(&cctx->appliedParams) + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + } + + if (idx - seqPos->idx >= cctx->seqStore.maxNbSeq) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_externalSequences_invalid) + ); + } + + ZSTD_storeSeq(&cctx->seqStore, litLength, ip, iend, offBase, matchLength); + ip += matchLength + litLength; + if (finalMatchSplit == 0) + { + idx++; + } + } + + assert( + idx == inSeqsSize || endPosInSequence <= inSeqs[idx].litLength + inSeqs[idx].matchLength + ); + seqPos->idx = idx; + seqPos->posInSequence = endPosInSequence; + memcpy(cctx->blockState.nextCBlock->rep, updatedRepcodes.rep, (uint)sizeof(repcodes_s)); + iend -= bytesAdjustment; + if (ip != iend) + { + /* Store any last literals */ + uint lastLLSize = (uint)(iend - ip); + assert(ip <= iend); + ZSTD_storeLastLiterals(&cctx->seqStore, ip, lastLLSize); + seqPos->posInSrc += lastLLSize; + } + + return (nuint)(iend - istart); + } + + private static void* ZSTD_selectSequenceCopier(ZSTD_sequenceFormat_e mode) + { + assert( + ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_experimentalParam11, (int)mode) != 0 + ); + if (mode == ZSTD_sequenceFormat_e.ZSTD_sf_explicitBlockDelimiters) + { + return (delegate* managed< + ZSTD_CCtx_s*, + ZSTD_SequencePosition*, + ZSTD_Sequence*, + nuint, + void*, + nuint, + ZSTD_paramSwitch_e, + nuint>)(&ZSTD_transferSequences_wBlockDelim); + } + + assert(mode == ZSTD_sequenceFormat_e.ZSTD_sf_noBlockDelimiters); + return (delegate* managed< + ZSTD_CCtx_s*, + ZSTD_SequencePosition*, + ZSTD_Sequence*, + nuint, + void*, + nuint, + ZSTD_paramSwitch_e, + nuint>)(&ZSTD_transferSequences_noDelim); + } + + /* Discover the size of next block by searching for the delimiter. + * Note that a block delimiter **must** exist in this mode, + * otherwise it's an input error. + * The block size retrieved will be later compared to ensure it remains within bounds */ + private static nuint blockSize_explicitDelimiter( + ZSTD_Sequence* inSeqs, + nuint inSeqsSize, + ZSTD_SequencePosition seqPos + ) + { + int end = 0; + nuint blockSize = 0; + nuint spos = seqPos.idx; + assert(spos <= inSeqsSize); + while (spos < inSeqsSize) + { + end = inSeqs[spos].offset == 0 ? 1 : 0; + blockSize += inSeqs[spos].litLength + inSeqs[spos].matchLength; + if (end != 0) + { + if (inSeqs[spos].matchLength != 0) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_externalSequences_invalid) + ); + } + + break; + } + + spos++; + } + + if (end == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_externalSequences_invalid)); + } + + return blockSize; + } + + private static nuint determine_blockSize( + ZSTD_sequenceFormat_e mode, + nuint blockSize, + nuint remaining, + ZSTD_Sequence* inSeqs, + nuint inSeqsSize, + ZSTD_SequencePosition seqPos + ) + { + if (mode == ZSTD_sequenceFormat_e.ZSTD_sf_noBlockDelimiters) + { + return remaining < blockSize ? remaining : blockSize; + } + + assert(mode == ZSTD_sequenceFormat_e.ZSTD_sf_explicitBlockDelimiters); + { + nuint explicitBlockSize = blockSize_explicitDelimiter(inSeqs, inSeqsSize, seqPos); + { + nuint err_code = explicitBlockSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (explicitBlockSize > blockSize) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_externalSequences_invalid) + ); + } + + if (explicitBlockSize > remaining) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_externalSequences_invalid) + ); + } + + return explicitBlockSize; + } + } + + /* Compress all provided sequences, block-by-block. + * + * Returns the cumulative size of all compressed blocks (including their headers), + * otherwise a ZSTD error. + */ + private static nuint ZSTD_compressSequences_internal( + ZSTD_CCtx_s* cctx, + void* dst, + nuint dstCapacity, + ZSTD_Sequence* inSeqs, + nuint inSeqsSize, + void* src, + nuint srcSize + ) + { + nuint cSize = 0; + nuint remaining = srcSize; + ZSTD_SequencePosition seqPos = new ZSTD_SequencePosition + { + idx = 0, + posInSequence = 0, + posInSrc = 0, + }; + byte* ip = (byte*)src; + byte* op = (byte*)dst; + void* sequenceCopier = ZSTD_selectSequenceCopier(cctx->appliedParams.blockDelimiters); + if (remaining == 0) + { + /* last block */ + uint cBlockHeader24 = 1 + ((uint)blockType_e.bt_raw << 1); + if (dstCapacity < 4) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + MEM_writeLE32(op, cBlockHeader24); + op += ZSTD_blockHeaderSize; + dstCapacity -= ZSTD_blockHeaderSize; + cSize += ZSTD_blockHeaderSize; + } + + while (remaining != 0) + { + nuint compressedSeqsSize; + nuint cBlockSize; + nuint blockSize = determine_blockSize( + cctx->appliedParams.blockDelimiters, + cctx->blockSizeMax, + remaining, + inSeqs, + inSeqsSize, + seqPos + ); + uint lastBlock = blockSize == remaining ? 1U : 0U; + { + nuint err_code = blockSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + assert(blockSize <= remaining); + ZSTD_resetSeqStore(&cctx->seqStore); + blockSize = ( + (delegate* managed< + ZSTD_CCtx_s*, + ZSTD_SequencePosition*, + ZSTD_Sequence*, + nuint, + void*, + nuint, + ZSTD_paramSwitch_e, + nuint>)sequenceCopier + )( + cctx, + &seqPos, + inSeqs, + inSeqsSize, + ip, + blockSize, + cctx->appliedParams.searchForExternalRepcodes + ); + { + nuint err_code = blockSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (blockSize < (nuint)(1 + 1) + ZSTD_blockHeaderSize + 1 + 1) + { + cBlockSize = ZSTD_noCompressBlock(op, dstCapacity, ip, blockSize, lastBlock); + { + nuint err_code = cBlockSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + cSize += cBlockSize; + ip += blockSize; + op += cBlockSize; + remaining -= blockSize; + dstCapacity -= cBlockSize; + continue; + } + + if (dstCapacity < ZSTD_blockHeaderSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + compressedSeqsSize = ZSTD_entropyCompressSeqStore( + &cctx->seqStore, + &cctx->blockState.prevCBlock->entropy, + &cctx->blockState.nextCBlock->entropy, + &cctx->appliedParams, + op + ZSTD_blockHeaderSize, + dstCapacity - ZSTD_blockHeaderSize, + blockSize, + cctx->tmpWorkspace, + cctx->tmpWkspSize, + cctx->bmi2 + ); + { + nuint err_code = compressedSeqsSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if ( + cctx->isFirstBlock == 0 + && ZSTD_maybeRLE(&cctx->seqStore) != 0 + && ZSTD_isRLE(ip, blockSize) != 0 + ) + { + compressedSeqsSize = 1; + } + + if (compressedSeqsSize == 0) + { + cBlockSize = ZSTD_noCompressBlock(op, dstCapacity, ip, blockSize, lastBlock); + { + nuint err_code = cBlockSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + } + else if (compressedSeqsSize == 1) + { + cBlockSize = ZSTD_rleCompressBlock(op, dstCapacity, *ip, blockSize, lastBlock); + { + nuint err_code = cBlockSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + } + else + { + uint cBlockHeader; + ZSTD_blockState_confirmRepcodesAndEntropyTables(&cctx->blockState); + if ( + cctx->blockState.prevCBlock->entropy.fse.offcode_repeatMode + == FSE_repeat.FSE_repeat_valid + ) + { + cctx->blockState.prevCBlock->entropy.fse.offcode_repeatMode = + FSE_repeat.FSE_repeat_check; + } + + cBlockHeader = + lastBlock + + ((uint)blockType_e.bt_compressed << 1) + + (uint)(compressedSeqsSize << 3); + MEM_writeLE24(op, cBlockHeader); + cBlockSize = ZSTD_blockHeaderSize + compressedSeqsSize; + } + + cSize += cBlockSize; + if (lastBlock != 0) + { + break; + } + else + { + ip += blockSize; + op += cBlockSize; + remaining -= blockSize; + dstCapacity -= cBlockSize; + cctx->isFirstBlock = 0; + } + } + + return cSize; + } + + /*! ZSTD_compressSequences() : + * Compress an array of ZSTD_Sequence, associated with @src buffer, into dst. + * @src contains the entire input (not just the literals). + * If @srcSize > sum(sequence.length), the remaining bytes are considered all literals + * If a dictionary is included, then the cctx should reference the dict (see: ZSTD_CCtx_refCDict(), ZSTD_CCtx_loadDictionary(), etc.). + * The entire source is compressed into a single frame. + * + * The compression behavior changes based on cctx params. In particular: + * If ZSTD_c_blockDelimiters == ZSTD_sf_noBlockDelimiters, the array of ZSTD_Sequence is expected to contain + * no block delimiters (defined in ZSTD_Sequence). Block boundaries are roughly determined based on + * the block size derived from the cctx, and sequences may be split. This is the default setting. + * + * If ZSTD_c_blockDelimiters == ZSTD_sf_explicitBlockDelimiters, the array of ZSTD_Sequence is expected to contain + * valid block delimiters (defined in ZSTD_Sequence). Behavior is undefined if no block delimiters are provided. + * + * When ZSTD_c_blockDelimiters == ZSTD_sf_explicitBlockDelimiters, it's possible to decide generating repcodes + * using the advanced parameter ZSTD_c_repcodeResolution. Repcodes will improve compression ratio, though the benefit + * can vary greatly depending on Sequences. On the other hand, repcode resolution is an expensive operation. + * By default, it's disabled at low (<10) compression levels, and enabled above the threshold (>=10). + * ZSTD_c_repcodeResolution makes it possible to directly manage this processing in either direction. + * + * If ZSTD_c_validateSequences == 0, this function blindly accepts the Sequences provided. Invalid Sequences cause undefined + * behavior. If ZSTD_c_validateSequences == 1, then the function will detect invalid Sequences (see doc/zstd_compression_format.md for + * specifics regarding offset/matchlength requirements) and then bail out and return an error. + * + * In addition to the two adjustable experimental params, there are other important cctx params. + * - ZSTD_c_minMatch MUST be set as less than or equal to the smallest match generated by the match finder. It has a minimum value of ZSTD_MINMATCH_MIN. + * - ZSTD_c_compressionLevel accordingly adjusts the strength of the entropy coder, as it would in typical compression. + * - ZSTD_c_windowLog affects offset validation: this function will return an error at higher debug levels if a provided offset + * is larger than what the spec allows for a given window log and dictionary (if present). See: doc/zstd_compression_format.md + * + * Note: Repcodes are, as of now, always re-calculated within this function, ZSTD_Sequence.rep is effectively unused. + * Dev Note: Once ability to ingest repcodes become available, the explicit block delims mode must respect those repcodes exactly, + * and cannot emit an RLE block that disagrees with the repcode history. + * @return : final compressed size, or a ZSTD error code. + */ + public static nuint ZSTD_compressSequences( + ZSTD_CCtx_s* cctx, + void* dst, + nuint dstCapacity, + ZSTD_Sequence* inSeqs, + nuint inSeqsSize, + void* src, + nuint srcSize + ) + { + byte* op = (byte*)dst; + nuint cSize = 0; + assert(cctx != null); + { + nuint err_code = ZSTD_CCtx_init_compressStream2( + cctx, + ZSTD_EndDirective.ZSTD_e_end, + srcSize + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint frameHeaderSize = ZSTD_writeFrameHeader( + op, + dstCapacity, + &cctx->appliedParams, + srcSize, + cctx->dictID + ); + op += frameHeaderSize; + assert(frameHeaderSize <= dstCapacity); + dstCapacity -= frameHeaderSize; + cSize += frameHeaderSize; + } + + if (cctx->appliedParams.fParams.checksumFlag != 0 && srcSize != 0) + { + ZSTD_XXH64_update(&cctx->xxhState, src, srcSize); + } + + { + nuint cBlocksSize = ZSTD_compressSequences_internal( + cctx, + op, + dstCapacity, + inSeqs, + inSeqsSize, + src, + srcSize + ); + { + nuint err_code = cBlocksSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + cSize += cBlocksSize; + assert(cBlocksSize <= dstCapacity); + dstCapacity -= cBlocksSize; + } + + if (cctx->appliedParams.fParams.checksumFlag != 0) + { + uint checksum = (uint)ZSTD_XXH64_digest(&cctx->xxhState); + if (dstCapacity < 4) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + MEM_writeLE32((sbyte*)dst + cSize, checksum); + cSize += 4; + } + + return cSize; + } + + private static nuint convertSequences_noRepcodes( + SeqDef_s* dstSeqs, + ZSTD_Sequence* inSeqs, + nuint nbSequences + ) + { + nuint longLen = 0; + nuint n; + for (n = 0; n < nbSequences; n++) + { + assert(inSeqs[n].offset > 0); + dstSeqs[n].offBase = inSeqs[n].offset + 3; + dstSeqs[n].litLength = (ushort)inSeqs[n].litLength; + dstSeqs[n].mlBase = (ushort)(inSeqs[n].matchLength - 3); + if (inSeqs[n].matchLength > 65535 + 3) + { + assert(longLen == 0); + longLen = n + 1; + } + + if (inSeqs[n].litLength > 65535) + { + assert(longLen == 0); + longLen = n + nbSequences + 1; + } + } + + return longLen; + } + + /* + * Precondition: Sequences must end on an explicit Block Delimiter + * @return: 0 on success, or an error code. + * Note: Sequence validation functionality has been disabled (removed). + * This is helpful to generate a lean main pipeline, improving performance. + * It may be re-inserted later. + */ + private static nuint ZSTD_convertBlockSequences( + ZSTD_CCtx_s* cctx, + ZSTD_Sequence* inSeqs, + nuint nbSequences, + int repcodeResolution + ) + { + repcodes_s updatedRepcodes; + nuint seqNb = 0; + if (nbSequences >= cctx->seqStore.maxNbSeq) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_externalSequences_invalid)); + } + + memcpy(updatedRepcodes.rep, cctx->blockState.prevCBlock->rep, (uint)sizeof(repcodes_s)); + assert(nbSequences >= 1); + assert(inSeqs[nbSequences - 1].matchLength == 0); + assert(inSeqs[nbSequences - 1].offset == 0); + if (repcodeResolution == 0) + { + nuint longl = convertSequences_noRepcodes( + cctx->seqStore.sequencesStart, + inSeqs, + nbSequences - 1 + ); + cctx->seqStore.sequences = cctx->seqStore.sequencesStart + nbSequences - 1; + if (longl != 0) + { + assert(cctx->seqStore.longLengthType == ZSTD_longLengthType_e.ZSTD_llt_none); + if (longl <= nbSequences - 1) + { + cctx->seqStore.longLengthType = ZSTD_longLengthType_e.ZSTD_llt_matchLength; + cctx->seqStore.longLengthPos = (uint)(longl - 1); + } + else + { + assert(longl <= 2 * (nbSequences - 1)); + cctx->seqStore.longLengthType = ZSTD_longLengthType_e.ZSTD_llt_literalLength; + cctx->seqStore.longLengthPos = (uint)(longl - (nbSequences - 1) - 1); + } + } + } + else + { + for (seqNb = 0; seqNb < nbSequences - 1; seqNb++) + { + uint litLength = inSeqs[seqNb].litLength; + uint matchLength = inSeqs[seqNb].matchLength; + uint ll0 = litLength == 0 ? 1U : 0U; + uint offBase = ZSTD_finalizeOffBase(inSeqs[seqNb].offset, updatedRepcodes.rep, ll0); + ZSTD_storeSeqOnly(&cctx->seqStore, litLength, offBase, matchLength); + ZSTD_updateRep(updatedRepcodes.rep, offBase, ll0); + } + } + + if (repcodeResolution == 0 && nbSequences > 1) + { + uint* rep = updatedRepcodes.rep; + if (nbSequences >= 4) + { + /* index of last full sequence */ + uint lastSeqIdx = (uint)nbSequences - 2; + rep[2] = inSeqs[lastSeqIdx - 2].offset; + rep[1] = inSeqs[lastSeqIdx - 1].offset; + rep[0] = inSeqs[lastSeqIdx].offset; + } + else if (nbSequences == 3) + { + rep[2] = rep[0]; + rep[1] = inSeqs[0].offset; + rep[0] = inSeqs[1].offset; + } + else + { + assert(nbSequences == 2); + rep[2] = rep[1]; + rep[1] = rep[0]; + rep[0] = inSeqs[0].offset; + } + } + + memcpy(cctx->blockState.nextCBlock->rep, updatedRepcodes.rep, (uint)sizeof(repcodes_s)); + return 0; + } + + private static BlockSummary ZSTD_get1BlockSummary(ZSTD_Sequence* seqs, nuint nbSeqs) + { + nuint totalMatchSize = 0; + nuint litSize = 0; + nuint n; + assert(seqs != null); + for (n = 0; n < nbSeqs; n++) + { + totalMatchSize += seqs[n].matchLength; + litSize += seqs[n].litLength; + if (seqs[n].matchLength == 0) + { + assert(seqs[n].offset == 0); + break; + } + } + + if (n == nbSeqs) + { + BlockSummary bs; + System.Runtime.CompilerServices.Unsafe.SkipInit(out bs); + bs.nbSequences = unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_externalSequences_invalid) + ); + return bs; + } + + { + BlockSummary bs; + bs.nbSequences = n + 1; + bs.blockSize = litSize + totalMatchSize; + bs.litSize = litSize; + return bs; + } + } + + private static nuint ZSTD_compressSequencesAndLiterals_internal( + ZSTD_CCtx_s* cctx, + void* dst, + nuint dstCapacity, + ZSTD_Sequence* inSeqs, + nuint nbSequences, + void* literals, + nuint litSize, + nuint srcSize + ) + { + nuint remaining = srcSize; + nuint cSize = 0; + byte* op = (byte*)dst; + int repcodeResolution = + cctx->appliedParams.searchForExternalRepcodes == ZSTD_paramSwitch_e.ZSTD_ps_enable + ? 1 + : 0; + assert(cctx->appliedParams.searchForExternalRepcodes != ZSTD_paramSwitch_e.ZSTD_ps_auto); + if (nbSequences == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_externalSequences_invalid)); + } + + if (nbSequences == 1 && inSeqs[0].litLength == 0) + { + /* last block */ + uint cBlockHeader24 = 1 + ((uint)blockType_e.bt_raw << 1); + if (dstCapacity < 3) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + MEM_writeLE24(op, cBlockHeader24); + op += ZSTD_blockHeaderSize; + dstCapacity -= ZSTD_blockHeaderSize; + cSize += ZSTD_blockHeaderSize; + } + + while (nbSequences != 0) + { + nuint compressedSeqsSize, + cBlockSize, + conversionStatus; + BlockSummary block = ZSTD_get1BlockSummary(inSeqs, nbSequences); + uint lastBlock = block.nbSequences == nbSequences ? 1U : 0U; + { + nuint err_code = block.nbSequences; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + assert(block.nbSequences <= nbSequences); + if (block.litSize > litSize) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_externalSequences_invalid) + ); + } + + ZSTD_resetSeqStore(&cctx->seqStore); + conversionStatus = ZSTD_convertBlockSequences( + cctx, + inSeqs, + block.nbSequences, + repcodeResolution + ); + { + nuint err_code = conversionStatus; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + inSeqs += block.nbSequences; + nbSequences -= block.nbSequences; + remaining -= block.blockSize; + if (dstCapacity < ZSTD_blockHeaderSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + compressedSeqsSize = ZSTD_entropyCompressSeqStore_internal( + op + ZSTD_blockHeaderSize, + dstCapacity - ZSTD_blockHeaderSize, + literals, + block.litSize, + &cctx->seqStore, + &cctx->blockState.prevCBlock->entropy, + &cctx->blockState.nextCBlock->entropy, + &cctx->appliedParams, + cctx->tmpWorkspace, + cctx->tmpWkspSize, + cctx->bmi2 + ); + { + nuint err_code = compressedSeqsSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (compressedSeqsSize > cctx->blockSizeMax) + { + compressedSeqsSize = 0; + } + + litSize -= block.litSize; + literals = (sbyte*)literals + block.litSize; + if (compressedSeqsSize == 0) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_cannotProduce_uncompressedBlock) + ); + } + else + { + uint cBlockHeader; + assert(compressedSeqsSize > 1); + ZSTD_blockState_confirmRepcodesAndEntropyTables(&cctx->blockState); + if ( + cctx->blockState.prevCBlock->entropy.fse.offcode_repeatMode + == FSE_repeat.FSE_repeat_valid + ) + { + cctx->blockState.prevCBlock->entropy.fse.offcode_repeatMode = + FSE_repeat.FSE_repeat_check; + } + + cBlockHeader = + lastBlock + + ((uint)blockType_e.bt_compressed << 1) + + (uint)(compressedSeqsSize << 3); + MEM_writeLE24(op, cBlockHeader); + cBlockSize = ZSTD_blockHeaderSize + compressedSeqsSize; + } + + cSize += cBlockSize; + op += cBlockSize; + dstCapacity -= cBlockSize; + cctx->isFirstBlock = 0; + if (lastBlock != 0) + { + assert(nbSequences == 0); + break; + } + } + + if (litSize != 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_externalSequences_invalid)); + } + + if (remaining != 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_externalSequences_invalid)); + } + + return cSize; + } + + /*! ZSTD_compressSequencesAndLiterals() : + * This is a variant of ZSTD_compressSequences() which, + * instead of receiving (src,srcSize) as input parameter, receives (literals,litSize), + * aka all the literals, already extracted and laid out into a single continuous buffer. + * This can be useful if the process generating the sequences also happens to generate the buffer of literals, + * thus skipping an extraction + caching stage. + * It's a speed optimization, useful when the right conditions are met, + * but it also features the following limitations: + * - Only supports explicit delimiter mode + * - Currently does not support Sequences validation (so input Sequences are trusted) + * - Not compatible with frame checksum, which must be disabled + * - If any block is incompressible, will fail and return an error + * - @litSize must be == sum of all @.litLength fields in @inSeqs. Any discrepancy will generate an error. + * - @litBufCapacity is the size of the underlying buffer into which literals are written, starting at address @literals. + * @litBufCapacity must be at least 8 bytes larger than @litSize. + * - @decompressedSize must be correct, and correspond to the sum of all Sequences. Any discrepancy will generate an error. + * @return : final compressed size, or a ZSTD error code. + */ + public static nuint ZSTD_compressSequencesAndLiterals( + ZSTD_CCtx_s* cctx, + void* dst, + nuint dstCapacity, + ZSTD_Sequence* inSeqs, + nuint inSeqsSize, + void* literals, + nuint litSize, + nuint litCapacity, + nuint decompressedSize + ) + { + byte* op = (byte*)dst; + nuint cSize = 0; + assert(cctx != null); + if (litCapacity < litSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_workSpace_tooSmall)); + } + + { + nuint err_code = ZSTD_CCtx_init_compressStream2( + cctx, + ZSTD_EndDirective.ZSTD_e_end, + decompressedSize + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (cctx->appliedParams.blockDelimiters == ZSTD_sequenceFormat_e.ZSTD_sf_noBlockDelimiters) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_frameParameter_unsupported)); + } + + if (cctx->appliedParams.validateSequences != 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_unsupported)); + } + + if (cctx->appliedParams.fParams.checksumFlag != 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_frameParameter_unsupported)); + } + + { + nuint frameHeaderSize = ZSTD_writeFrameHeader( + op, + dstCapacity, + &cctx->appliedParams, + decompressedSize, + cctx->dictID + ); + op += frameHeaderSize; + assert(frameHeaderSize <= dstCapacity); + dstCapacity -= frameHeaderSize; + cSize += frameHeaderSize; + } + + { + nuint cBlocksSize = ZSTD_compressSequencesAndLiterals_internal( + cctx, + op, + dstCapacity, + inSeqs, + inSeqsSize, + literals, + litSize, + decompressedSize + ); + { + nuint err_code = cBlocksSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + cSize += cBlocksSize; + assert(cBlocksSize <= dstCapacity); + dstCapacity -= cBlocksSize; + } + + return cSize; + } + + /*====== Finalize ======*/ + private static ZSTD_inBuffer_s inBuffer_forEndFlush(ZSTD_CCtx_s* zcs) + { + ZSTD_inBuffer_s nullInput = new ZSTD_inBuffer_s + { + src = null, + size = 0, + pos = 0, + }; + int stableInput = + zcs->appliedParams.inBufferMode == ZSTD_bufferMode_e.ZSTD_bm_stable ? 1 : 0; + return stableInput != 0 ? zcs->expectedInBuffer : nullInput; + } + + /*! ZSTD_flushStream() : + * @return : amount of data remaining to flush */ + public static nuint ZSTD_flushStream(ZSTD_CCtx_s* zcs, ZSTD_outBuffer_s* output) + { + ZSTD_inBuffer_s input = inBuffer_forEndFlush(zcs); + input.size = input.pos; + return ZSTD_compressStream2(zcs, output, &input, ZSTD_EndDirective.ZSTD_e_flush); + } + + /*! Equivalent to ZSTD_compressStream2(zcs, output, &emptyInput, ZSTD_e_end). */ + public static nuint ZSTD_endStream(ZSTD_CCtx_s* zcs, ZSTD_outBuffer_s* output) + { + ZSTD_inBuffer_s input = inBuffer_forEndFlush(zcs); + nuint remainingToFlush = ZSTD_compressStream2( + zcs, + output, + &input, + ZSTD_EndDirective.ZSTD_e_end + ); + { + nuint err_code = remainingToFlush; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (zcs->appliedParams.nbWorkers > 0) + { + return remainingToFlush; + } + + { + nuint lastBlockSize = (nuint)(zcs->frameEnded != 0 ? 0 : 3); + nuint checksumSize = (nuint)( + zcs->frameEnded != 0 ? 0 : zcs->appliedParams.fParams.checksumFlag * 4 + ); + nuint toFlush = remainingToFlush + lastBlockSize + checksumSize; + return toFlush; + } + } + + public static int ZSTD_maxCLevel() + { + return 22; + } + + public static int ZSTD_minCLevel() + { + return -(1 << 17); + } + + public static int ZSTD_defaultCLevel() + { + return 3; + } + + private static ZSTD_compressionParameters ZSTD_dedicatedDictSearch_getCParams( + int compressionLevel, + nuint dictSize + ) + { + ZSTD_compressionParameters cParams = ZSTD_getCParams_internal( + compressionLevel, + 0, + dictSize, + ZSTD_CParamMode_e.ZSTD_cpm_createCDict + ); + switch (cParams.strategy) + { + case ZSTD_strategy.ZSTD_fast: + case ZSTD_strategy.ZSTD_dfast: + break; + case ZSTD_strategy.ZSTD_greedy: + case ZSTD_strategy.ZSTD_lazy: + case ZSTD_strategy.ZSTD_lazy2: + cParams.hashLog += 2; + break; + case ZSTD_strategy.ZSTD_btlazy2: + case ZSTD_strategy.ZSTD_btopt: + case ZSTD_strategy.ZSTD_btultra: + case ZSTD_strategy.ZSTD_btultra2: + break; + } + + return cParams; + } + + private static int ZSTD_dedicatedDictSearch_isSupported(ZSTD_compressionParameters* cParams) + { + return + cParams->strategy >= ZSTD_strategy.ZSTD_greedy + && cParams->strategy <= ZSTD_strategy.ZSTD_lazy2 + && cParams->hashLog > cParams->chainLog + && cParams->chainLog <= 24 + ? 1 + : 0; + } + + /** + * Reverses the adjustment applied to cparams when enabling dedicated dict + * search. This is used to recover the params set to be used in the working + * context. (Otherwise, those tables would also grow.) + */ + private static void ZSTD_dedicatedDictSearch_revertCParams(ZSTD_compressionParameters* cParams) + { + switch (cParams->strategy) + { + case ZSTD_strategy.ZSTD_fast: + case ZSTD_strategy.ZSTD_dfast: + break; + case ZSTD_strategy.ZSTD_greedy: + case ZSTD_strategy.ZSTD_lazy: + case ZSTD_strategy.ZSTD_lazy2: + cParams->hashLog -= 2; + if (cParams->hashLog < 6) + { + cParams->hashLog = 6; + } + + break; + case ZSTD_strategy.ZSTD_btlazy2: + case ZSTD_strategy.ZSTD_btopt: + case ZSTD_strategy.ZSTD_btultra: + case ZSTD_strategy.ZSTD_btultra2: + break; + } + } + + private static ulong ZSTD_getCParamRowSize( + ulong srcSizeHint, + nuint dictSize, + ZSTD_CParamMode_e mode + ) + { + switch (mode) + { + case ZSTD_CParamMode_e.ZSTD_cpm_unknown: + case ZSTD_CParamMode_e.ZSTD_cpm_noAttachDict: + case ZSTD_CParamMode_e.ZSTD_cpm_createCDict: + break; + case ZSTD_CParamMode_e.ZSTD_cpm_attachDict: + dictSize = 0; + break; + default: + assert(0 != 0); + break; + } + + { + int unknown = srcSizeHint == unchecked(0UL - 1) ? 1 : 0; + nuint addedSize = (nuint)(unknown != 0 && dictSize > 0 ? 500 : 0); + return unknown != 0 && dictSize == 0 + ? unchecked(0UL - 1) + : srcSizeHint + dictSize + addedSize; + } + } + + /*! ZSTD_getCParams_internal() : + * @return ZSTD_compressionParameters structure for a selected compression level, srcSize and dictSize. + * Note: srcSizeHint 0 means 0, use ZSTD_CONTENTSIZE_UNKNOWN for unknown. + * Use dictSize == 0 for unknown or unused. + * Note: `mode` controls how we treat the `dictSize`. See docs for `ZSTD_CParamMode_e`. */ + private static ZSTD_compressionParameters ZSTD_getCParams_internal( + int compressionLevel, + ulong srcSizeHint, + nuint dictSize, + ZSTD_CParamMode_e mode + ) + { + ulong rSize = ZSTD_getCParamRowSize(srcSizeHint, dictSize, mode); + uint tableID = (uint)( + (rSize <= 256 * (1 << 10) ? 1 : 0) + + (rSize <= 128 * (1 << 10) ? 1 : 0) + + (rSize <= 16 * (1 << 10) ? 1 : 0) + ); + int row; + if (compressionLevel == 0) + { + row = 3; + } + else if (compressionLevel < 0) + { + row = 0; + } + else if (compressionLevel > 22) + { + row = 22; + } + else + { + row = compressionLevel; + } + + { + ZSTD_compressionParameters cp = ZSTD_defaultCParameters[tableID][row]; + if (compressionLevel < 0) + { + int clampedCompressionLevel = + ZSTD_minCLevel() > compressionLevel ? ZSTD_minCLevel() : compressionLevel; + cp.targetLength = (uint)-clampedCompressionLevel; + } + + return ZSTD_adjustCParams_internal( + cp, + srcSizeHint, + dictSize, + mode, + ZSTD_paramSwitch_e.ZSTD_ps_auto + ); + } + } + + /*! ZSTD_getCParams() : + * @return ZSTD_compressionParameters structure for a selected compression level, srcSize and dictSize. + * Size values are optional, provide 0 if not known or unused */ + public static ZSTD_compressionParameters ZSTD_getCParams( + int compressionLevel, + ulong srcSizeHint, + nuint dictSize + ) + { + if (srcSizeHint == 0) + { + srcSizeHint = unchecked(0UL - 1); + } + + return ZSTD_getCParams_internal( + compressionLevel, + srcSizeHint, + dictSize, + ZSTD_CParamMode_e.ZSTD_cpm_unknown + ); + } + + /*! ZSTD_getParams() : + * same idea as ZSTD_getCParams() + * @return a `ZSTD_parameters` structure (instead of `ZSTD_compressionParameters`). + * Fields of `ZSTD_frameParameters` are set to default values */ + private static ZSTD_parameters ZSTD_getParams_internal( + int compressionLevel, + ulong srcSizeHint, + nuint dictSize, + ZSTD_CParamMode_e mode + ) + { + ZSTD_parameters @params; + ZSTD_compressionParameters cParams = ZSTD_getCParams_internal( + compressionLevel, + srcSizeHint, + dictSize, + mode + ); + @params = new ZSTD_parameters { cParams = cParams }; + @params.fParams.contentSizeFlag = 1; + return @params; + } + + /*! ZSTD_getParams() : + * same idea as ZSTD_getCParams() + * @return a `ZSTD_parameters` structure (instead of `ZSTD_compressionParameters`). + * Fields of `ZSTD_frameParameters` are set to default values */ + public static ZSTD_parameters ZSTD_getParams( + int compressionLevel, + ulong srcSizeHint, + nuint dictSize + ) + { + if (srcSizeHint == 0) + { + srcSizeHint = unchecked(0UL - 1); + } + + return ZSTD_getParams_internal( + compressionLevel, + srcSizeHint, + dictSize, + ZSTD_CParamMode_e.ZSTD_cpm_unknown + ); + } + + /*! ZSTD_registerSequenceProducer() : + * Instruct zstd to use a block-level external sequence producer function. + * + * The sequenceProducerState must be initialized by the caller, and the caller is + * responsible for managing its lifetime. This parameter is sticky across + * compressions. It will remain set until the user explicitly resets compression + * parameters. + * + * Sequence producer registration is considered to be an "advanced parameter", + * part of the "advanced API". This means it will only have an effect on compression + * APIs which respect advanced parameters, such as compress2() and compressStream2(). + * Older compression APIs such as compressCCtx(), which predate the introduction of + * "advanced parameters", will ignore any external sequence producer setting. + * + * The sequence producer can be "cleared" by registering a NULL function pointer. This + * removes all limitations described above in the "LIMITATIONS" section of the API docs. + * + * The user is strongly encouraged to read the full API documentation (above) before + * calling this function. */ + public static void ZSTD_registerSequenceProducer( + ZSTD_CCtx_s* zc, + void* extSeqProdState, + void* extSeqProdFunc + ) + { + assert(zc != null); + ZSTD_CCtxParams_registerSequenceProducer( + &zc->requestedParams, + extSeqProdState, + extSeqProdFunc + ); + } + + /*! ZSTD_CCtxParams_registerSequenceProducer() : + * Same as ZSTD_registerSequenceProducer(), but operates on ZSTD_CCtx_params. + * This is used for accurate size estimation with ZSTD_estimateCCtxSize_usingCCtxParams(), + * which is needed when creating a ZSTD_CCtx with ZSTD_initStaticCCtx(). + * + * If you are using the external sequence producer API in a scenario where ZSTD_initStaticCCtx() + * is required, then this function is for you. Otherwise, you probably don't need it. + * + * See tests/zstreamtest.c for example usage. */ + public static void ZSTD_CCtxParams_registerSequenceProducer( + ZSTD_CCtx_params_s* @params, + void* extSeqProdState, + void* extSeqProdFunc + ) + { + assert(@params != null); + if (extSeqProdFunc != null) + { + @params->extSeqProdFunc = extSeqProdFunc; + @params->extSeqProdState = extSeqProdState; + } + else + { + @params->extSeqProdFunc = null; + @params->extSeqProdState = null; + } + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdCompressInternal.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdCompressInternal.cs new file mode 100644 index 00000000..dac7bb52 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdCompressInternal.cs @@ -0,0 +1,1504 @@ +using System; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + /** + * Returns the ZSTD_SequenceLength for the given sequences. It handles the decoding of long sequences + * indicated by longLengthPos and longLengthType, and adds MINMATCH back to matchLength. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ZSTD_SequenceLength ZSTD_getSequenceLength(SeqStore_t* seqStore, SeqDef_s* seq) + { + ZSTD_SequenceLength seqLen; + seqLen.litLength = seq->litLength; + seqLen.matchLength = (uint)(seq->mlBase + 3); + if (seqStore->longLengthPos == (uint)(seq - seqStore->sequencesStart)) + { + if (seqStore->longLengthType == ZSTD_longLengthType_e.ZSTD_llt_literalLength) + { + seqLen.litLength += 0x10000; + } + + if (seqStore->longLengthType == ZSTD_longLengthType_e.ZSTD_llt_matchLength) + { + seqLen.matchLength += 0x10000; + } + } + + return seqLen; + } + + private static readonly RawSeqStore_t kNullRawSeqStore = new RawSeqStore_t( + seq: null, + pos: 0, + posInSequence: 0, + size: 0, + capacity: 0 + ); +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_LL_Code => + new byte[64] + { + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 16, + 17, + 17, + 18, + 18, + 19, + 19, + 20, + 20, + 20, + 20, + 21, + 21, + 21, + 21, + 22, + 22, + 22, + 22, + 22, + 22, + 22, + 22, + 23, + 23, + 23, + 23, + 23, + 23, + 23, + 23, + 24, + 24, + 24, + 24, + 24, + 24, + 24, + 24, + 24, + 24, + 24, + 24, + 24, + 24, + 24, + 24, + }; + private static byte* LL_Code => + (byte*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_LL_Code) + ); +#else + + private static readonly byte* LL_Code = GetArrayPointer( + new byte[64] + { + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 16, + 17, + 17, + 18, + 18, + 19, + 19, + 20, + 20, + 20, + 20, + 21, + 21, + 21, + 21, + 22, + 22, + 22, + 22, + 22, + 22, + 22, + 22, + 23, + 23, + 23, + 23, + 23, + 23, + 23, + 23, + 24, + 24, + 24, + 24, + 24, + 24, + 24, + 24, + 24, + 24, + 24, + 24, + 24, + 24, + 24, + 24, + } + ); +#endif + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_LLcode(uint litLength) + { + const uint LL_deltaCode = 19; + return litLength > 63 ? ZSTD_highbit32(litLength) + LL_deltaCode : LL_Code[litLength]; + } + +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_ML_Code => + new byte[128] + { + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 32, + 33, + 33, + 34, + 34, + 35, + 35, + 36, + 36, + 36, + 36, + 37, + 37, + 37, + 37, + 38, + 38, + 38, + 38, + 38, + 38, + 38, + 38, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 40, + 40, + 40, + 40, + 40, + 40, + 40, + 40, + 40, + 40, + 40, + 40, + 40, + 40, + 40, + 40, + 41, + 41, + 41, + 41, + 41, + 41, + 41, + 41, + 41, + 41, + 41, + 41, + 41, + 41, + 41, + 41, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + }; + private static byte* ML_Code => + (byte*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_ML_Code) + ); +#else + + private static readonly byte* ML_Code = GetArrayPointer( + new byte[128] + { + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 32, + 33, + 33, + 34, + 34, + 35, + 35, + 36, + 36, + 36, + 36, + 37, + 37, + 37, + 37, + 38, + 38, + 38, + 38, + 38, + 38, + 38, + 38, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 40, + 40, + 40, + 40, + 40, + 40, + 40, + 40, + 40, + 40, + 40, + 40, + 40, + 40, + 40, + 40, + 41, + 41, + 41, + 41, + 41, + 41, + 41, + 41, + 41, + 41, + 41, + 41, + 41, + 41, + 41, + 41, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + 42, + } + ); +#endif + /* ZSTD_MLcode() : + * note : mlBase = matchLength - MINMATCH; + * because it's the format it's stored in seqStore->sequences */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_MLcode(uint mlBase) + { + const uint ML_deltaCode = 36; + return mlBase > 127 ? ZSTD_highbit32(mlBase) + ML_deltaCode : ML_Code[mlBase]; + } + + /* ZSTD_cParam_withinBounds: + * @return 1 if value is within cParam bounds, + * 0 otherwise */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static int ZSTD_cParam_withinBounds(ZSTD_cParameter cParam, int value) + { + ZSTD_bounds bounds = ZSTD_cParam_getBounds(cParam); + if (ERR_isError(bounds.error)) + { + return 0; + } + + if (value < bounds.lowerBound) + { + return 0; + } + + if (value > bounds.upperBound) + { + return 0; + } + + return 1; + } + + /* ZSTD_selectAddr: + * @return index >= lowLimit ? candidate : backup, + * tries to force branchless codegen. */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static byte* ZSTD_selectAddr(uint index, uint lowLimit, byte* candidate, byte* backup) + { + return index >= lowLimit ? candidate : backup; + } + + /* ZSTD_noCompressBlock() : + * Writes uncompressed block to dst buffer from given src. + * Returns the size of the block */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_noCompressBlock( + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize, + uint lastBlock + ) + { + uint cBlockHeader24 = lastBlock + ((uint)blockType_e.bt_raw << 1) + (uint)(srcSize << 3); + if (srcSize + ZSTD_blockHeaderSize > dstCapacity) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + MEM_writeLE24(dst, cBlockHeader24); + memcpy((byte*)dst + ZSTD_blockHeaderSize, src, (uint)srcSize); + return ZSTD_blockHeaderSize + srcSize; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_rleCompressBlock( + void* dst, + nuint dstCapacity, + byte src, + nuint srcSize, + uint lastBlock + ) + { + byte* op = (byte*)dst; + uint cBlockHeader = lastBlock + ((uint)blockType_e.bt_rle << 1) + (uint)(srcSize << 3); + if (dstCapacity < 4) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + MEM_writeLE24(op, cBlockHeader); + op[3] = src; + return 4; + } + + /* ZSTD_minGain() : + * minimum compression required + * to generate a compress block or a compressed literals section. + * note : use same formula for both situations */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_minGain(nuint srcSize, ZSTD_strategy strat) + { + uint minlog = strat >= ZSTD_strategy.ZSTD_btultra ? (uint)strat - 1 : 6; + assert(ZSTD_cParam_withinBounds(ZSTD_cParameter.ZSTD_c_strategy, (int)strat) != 0); + return (srcSize >> (int)minlog) + 2; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static int ZSTD_literalsCompressionIsDisabled(ZSTD_CCtx_params_s* cctxParams) + { + switch (cctxParams->literalCompressionMode) + { + case ZSTD_paramSwitch_e.ZSTD_ps_enable: + return 0; + case ZSTD_paramSwitch_e.ZSTD_ps_disable: + return 1; + default: + assert(0 != 0); + goto case ZSTD_paramSwitch_e.ZSTD_ps_auto; + case ZSTD_paramSwitch_e.ZSTD_ps_auto: + return + cctxParams->cParams.strategy == ZSTD_strategy.ZSTD_fast + && cctxParams->cParams.targetLength > 0 + ? 1 + : 0; + } + } + + /*! ZSTD_safecopyLiterals() : + * memcpy() function that won't read beyond more than WILDCOPY_OVERLENGTH bytes past ilimit_w. + * Only called when the sequence ends past ilimit_w, so it only needs to be optimized for single + * large copies. + */ + private static void ZSTD_safecopyLiterals(byte* op, byte* ip, byte* iend, byte* ilimit_w) + { + assert(iend > ilimit_w); + if (ip <= ilimit_w) + { + ZSTD_wildcopy(op, ip, (nint)(ilimit_w - ip), ZSTD_overlap_e.ZSTD_no_overlap); + op += ilimit_w - ip; + ip = ilimit_w; + } + + while (ip < iend) + { + *op++ = *ip++; + } + } + + /*! ZSTD_storeSeqOnly() : + * Store a sequence (litlen, litPtr, offBase and matchLength) into SeqStore_t. + * Literals themselves are not copied, but @litPtr is updated. + * @offBase : Users should employ macros REPCODE_TO_OFFBASE() and OFFSET_TO_OFFBASE(). + * @matchLength : must be >= MINMATCH + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_storeSeqOnly( + SeqStore_t* seqStorePtr, + nuint litLength, + uint offBase, + nuint matchLength + ) + { + assert( + (nuint)(seqStorePtr->sequences - seqStorePtr->sequencesStart) < seqStorePtr->maxNbSeq + ); + assert(litLength <= 1 << 17); + if (litLength > 0xFFFF) + { + assert(seqStorePtr->longLengthType == ZSTD_longLengthType_e.ZSTD_llt_none); + seqStorePtr->longLengthType = ZSTD_longLengthType_e.ZSTD_llt_literalLength; + seqStorePtr->longLengthPos = (uint)( + seqStorePtr->sequences - seqStorePtr->sequencesStart + ); + } + + seqStorePtr->sequences[0].litLength = (ushort)litLength; + seqStorePtr->sequences[0].offBase = offBase; + assert(matchLength <= 1 << 17); + assert(matchLength >= 3); + { + nuint mlBase = matchLength - 3; + if (mlBase > 0xFFFF) + { + assert(seqStorePtr->longLengthType == ZSTD_longLengthType_e.ZSTD_llt_none); + seqStorePtr->longLengthType = ZSTD_longLengthType_e.ZSTD_llt_matchLength; + seqStorePtr->longLengthPos = (uint)( + seqStorePtr->sequences - seqStorePtr->sequencesStart + ); + } + + seqStorePtr->sequences[0].mlBase = (ushort)mlBase; + } + + seqStorePtr->sequences++; + } + + /*! ZSTD_storeSeq() : + * Store a sequence (litlen, litPtr, offBase and matchLength) into SeqStore_t. + * @offBase : Users should employ macros REPCODE_TO_OFFBASE() and OFFSET_TO_OFFBASE(). + * @matchLength : must be >= MINMATCH + * Allowed to over-read literals up to litLimit. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_storeSeq( + SeqStore_t* seqStorePtr, + nuint litLength, + byte* literals, + byte* litLimit, + uint offBase, + nuint matchLength + ) + { + byte* litLimit_w = litLimit - 32; + byte* litEnd = literals + litLength; + assert( + (nuint)(seqStorePtr->sequences - seqStorePtr->sequencesStart) < seqStorePtr->maxNbSeq + ); + assert(seqStorePtr->maxNbLit <= 128 * (1 << 10)); + assert(seqStorePtr->lit + litLength <= seqStorePtr->litStart + seqStorePtr->maxNbLit); + assert(literals + litLength <= litLimit); + if (litEnd <= litLimit_w) + { + ZSTD_copy16(seqStorePtr->lit, literals); + if (litLength > 16) + { + ZSTD_wildcopy( + seqStorePtr->lit + 16, + literals + 16, + (nint)litLength - 16, + ZSTD_overlap_e.ZSTD_no_overlap + ); + } + } + else + { + ZSTD_safecopyLiterals(seqStorePtr->lit, literals, litEnd, litLimit_w); + } + + seqStorePtr->lit += litLength; + ZSTD_storeSeqOnly(seqStorePtr, litLength, offBase, matchLength); + } + + /* ZSTD_updateRep() : + * updates in-place @rep (array of repeat offsets) + * @offBase : sum-type, using numeric representation of ZSTD_storeSeq() + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_updateRep(uint* rep, uint offBase, uint ll0) + { + if (offBase > 3) + { + rep[2] = rep[1]; + rep[1] = rep[0]; + assert(offBase > 3); + rep[0] = offBase - 3; + } + else + { + assert(1 <= offBase && offBase <= 3); + uint repCode = offBase - 1 + ll0; + if (repCode > 0) + { + uint currentOffset = repCode == 3 ? rep[0] - 1 : rep[repCode]; + rep[2] = repCode >= 2 ? rep[1] : rep[2]; + rep[1] = rep[0]; + rep[0] = currentOffset; + } + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static repcodes_s ZSTD_newRep(uint* rep, uint offBase, uint ll0) + { + repcodes_s newReps; + memcpy(&newReps, rep, (uint)sizeof(repcodes_s)); + ZSTD_updateRep(newReps.rep, offBase, ll0); + return newReps; + } + + /*-************************************* + * Match length counter + ***************************************/ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_count(byte* pIn, byte* pMatch, byte* pInLimit) + { + byte* pStart = pIn; + byte* pInLoopLimit = pInLimit - (sizeof(nuint) - 1); + if (pIn < pInLoopLimit) + { + { + nuint diff = MEM_readST(pMatch) ^ MEM_readST(pIn); + if (diff != 0) + { + return ZSTD_NbCommonBytes(diff); + } + } + + pIn += sizeof(nuint); + pMatch += sizeof(nuint); + while (pIn < pInLoopLimit) + { + nuint diff = MEM_readST(pMatch) ^ MEM_readST(pIn); + if (diff == 0) + { + pIn += sizeof(nuint); + pMatch += sizeof(nuint); + continue; + } + + pIn += ZSTD_NbCommonBytes(diff); + return (nuint)(pIn - pStart); + } + } + + if (MEM_64bits && pIn < pInLimit - 3 && MEM_read32(pMatch) == MEM_read32(pIn)) + { + pIn += 4; + pMatch += 4; + } + + if (pIn < pInLimit - 1 && MEM_read16(pMatch) == MEM_read16(pIn)) + { + pIn += 2; + pMatch += 2; + } + + if (pIn < pInLimit && *pMatch == *pIn) + { + pIn++; + } + + return (nuint)(pIn - pStart); + } + + /** ZSTD_count_2segments() : + * can count match length with `ip` & `match` in 2 different segments. + * convention : on reaching mEnd, match count continue starting from iStart + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_count_2segments( + byte* ip, + byte* match, + byte* iEnd, + byte* mEnd, + byte* iStart + ) + { + byte* vEnd = ip + (mEnd - match) < iEnd ? ip + (mEnd - match) : iEnd; + nuint matchLength = ZSTD_count(ip, match, vEnd); + if (match + matchLength != mEnd) + { + return matchLength; + } + + return matchLength + ZSTD_count(ip + matchLength, iStart, iEnd); + } + + private const uint prime3bytes = 506832829U; + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_hash3(uint u, uint h, uint s) + { + assert(h <= 32); + return ((u << 32 - 24) * prime3bytes ^ s) >> (int)(32 - h); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_hash3Ptr(void* ptr, uint h) + { + return ZSTD_hash3(MEM_readLE32(ptr), h, 0); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_hash3PtrS(void* ptr, uint h, uint s) + { + return ZSTD_hash3(MEM_readLE32(ptr), h, s); + } + + private const uint prime4bytes = 2654435761U; + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_hash4(uint u, uint h, uint s) + { + assert(h <= 32); + return (u * prime4bytes ^ s) >> (int)(32 - h); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_hash4Ptr(void* ptr, uint h) + { + return ZSTD_hash4(MEM_readLE32(ptr), h, 0); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_hash4PtrS(void* ptr, uint h, uint s) + { + return ZSTD_hash4(MEM_readLE32(ptr), h, s); + } + + private const ulong prime5bytes = 889523592379UL; + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_hash5(ulong u, uint h, ulong s) + { + assert(h <= 64); + return (nuint)(((u << 64 - 40) * prime5bytes ^ s) >> (int)(64 - h)); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_hash5Ptr(void* p, uint h) + { + return ZSTD_hash5(MEM_readLE64(p), h, 0); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_hash5PtrS(void* p, uint h, ulong s) + { + return ZSTD_hash5(MEM_readLE64(p), h, s); + } + + private const ulong prime6bytes = 227718039650203UL; + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_hash6(ulong u, uint h, ulong s) + { + assert(h <= 64); + return (nuint)(((u << 64 - 48) * prime6bytes ^ s) >> (int)(64 - h)); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_hash6Ptr(void* p, uint h) + { + return ZSTD_hash6(MEM_readLE64(p), h, 0); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_hash6PtrS(void* p, uint h, ulong s) + { + return ZSTD_hash6(MEM_readLE64(p), h, s); + } + + private const ulong prime7bytes = 58295818150454627UL; + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_hash7(ulong u, uint h, ulong s) + { + assert(h <= 64); + return (nuint)(((u << 64 - 56) * prime7bytes ^ s) >> (int)(64 - h)); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_hash7Ptr(void* p, uint h) + { + return ZSTD_hash7(MEM_readLE64(p), h, 0); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_hash7PtrS(void* p, uint h, ulong s) + { + return ZSTD_hash7(MEM_readLE64(p), h, s); + } + + private const ulong prime8bytes = 0xCF1BBCDCB7A56463UL; + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_hash8(ulong u, uint h, ulong s) + { + assert(h <= 64); + return (nuint)((u * prime8bytes ^ s) >> (int)(64 - h)); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_hash8Ptr(void* p, uint h) + { + return ZSTD_hash8(MEM_readLE64(p), h, 0); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_hash8PtrS(void* p, uint h, ulong s) + { + return ZSTD_hash8(MEM_readLE64(p), h, s); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_hashPtr(void* p, uint hBits, uint mls) + { + assert(hBits <= 32); + if (mls == 5) + { + return ZSTD_hash5Ptr(p, hBits); + } + + if (mls == 6) + { + return ZSTD_hash6Ptr(p, hBits); + } + + if (mls == 7) + { + return ZSTD_hash7Ptr(p, hBits); + } + + if (mls == 8) + { + return ZSTD_hash8Ptr(p, hBits); + } + + return ZSTD_hash4Ptr(p, hBits); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_hashPtrSalted(void* p, uint hBits, uint mls, ulong hashSalt) + { + assert(hBits <= 32); + if (mls == 5) + { + return ZSTD_hash5PtrS(p, hBits, hashSalt); + } + + if (mls == 6) + { + return ZSTD_hash6PtrS(p, hBits, hashSalt); + } + + if (mls == 7) + { + return ZSTD_hash7PtrS(p, hBits, hashSalt); + } + + if (mls == 8) + { + return ZSTD_hash8PtrS(p, hBits, hashSalt); + } + + return ZSTD_hash4PtrS(p, hBits, (uint)hashSalt); + } + + /** ZSTD_ipow() : + * Return base^exponent. + */ + private static ulong ZSTD_ipow(ulong @base, ulong exponent) + { + ulong power = 1; + while (exponent != 0) + { + if ((exponent & 1) != 0) + { + power *= @base; + } + + exponent >>= 1; + @base *= @base; + } + + return power; + } + + /** ZSTD_rollingHash_append() : + * Add the buffer to the hash value. + */ + private static ulong ZSTD_rollingHash_append(ulong hash, void* buf, nuint size) + { + byte* istart = (byte*)buf; + nuint pos; + for (pos = 0; pos < size; ++pos) + { + hash *= prime8bytes; + hash += (ulong)(istart[pos] + 10); + } + + return hash; + } + + /** ZSTD_rollingHash_compute() : + * Compute the rolling hash value of the buffer. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong ZSTD_rollingHash_compute(void* buf, nuint size) + { + return ZSTD_rollingHash_append(0, buf, size); + } + + /** ZSTD_rollingHash_primePower() : + * Compute the primePower to be passed to ZSTD_rollingHash_rotate() for a hash + * over a window of length bytes. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong ZSTD_rollingHash_primePower(uint length) + { + return ZSTD_ipow(prime8bytes, length - 1); + } + + /** ZSTD_rollingHash_rotate() : + * Rotate the rolling hash by one byte. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong ZSTD_rollingHash_rotate( + ulong hash, + byte toRemove, + byte toAdd, + ulong primePower + ) + { + hash -= (ulong)(toRemove + 10) * primePower; + hash *= prime8bytes; + hash += (ulong)(toAdd + 10); + return hash; + } + + /** + * ZSTD_window_clear(): + * Clears the window containing the history by simply setting it to empty. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_window_clear(ZSTD_window_t* window) + { + nuint endT = (nuint)(window->nextSrc - window->@base); + uint end = (uint)endT; + window->lowLimit = end; + window->dictLimit = end; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_window_isEmpty(ZSTD_window_t window) + { + return window.dictLimit == 2 && window.lowLimit == 2 && window.nextSrc - window.@base == 2 + ? 1U + : 0U; + } + + /** + * ZSTD_window_hasExtDict(): + * Returns non-zero if the window has a non-empty extDict. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_window_hasExtDict(ZSTD_window_t window) + { + return window.lowLimit < window.dictLimit ? 1U : 0U; + } + + /** + * ZSTD_matchState_dictMode(): + * Inspects the provided matchState and figures out what dictMode should be + * passed to the compressor. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ZSTD_dictMode_e ZSTD_matchState_dictMode(ZSTD_MatchState_t* ms) + { + return ZSTD_window_hasExtDict(ms->window) != 0 ? ZSTD_dictMode_e.ZSTD_extDict + : ms->dictMatchState != null + ? ms->dictMatchState->dedicatedDictSearch != 0 + ? ZSTD_dictMode_e.ZSTD_dedicatedDictSearch + : ZSTD_dictMode_e.ZSTD_dictMatchState + : ZSTD_dictMode_e.ZSTD_noDict; + } + + /** + * ZSTD_window_canOverflowCorrect(): + * Returns non-zero if the indices are large enough for overflow correction + * to work correctly without impacting compression ratio. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_window_canOverflowCorrect( + ZSTD_window_t window, + uint cycleLog, + uint maxDist, + uint loadedDictEnd, + void* src + ) + { + uint cycleSize = 1U << (int)cycleLog; + uint curr = (uint)((byte*)src - window.@base); + uint minIndexToOverflowCorrect = + cycleSize + (maxDist > cycleSize ? maxDist : cycleSize) + 2; + /* Adjust the min index to backoff the overflow correction frequency, + * so we don't waste too much CPU in overflow correction. If this + * computation overflows we don't really care, we just need to make + * sure it is at least minIndexToOverflowCorrect. + */ + uint adjustment = window.nbOverflowCorrections + 1; + uint adjustedIndex = + minIndexToOverflowCorrect * adjustment > minIndexToOverflowCorrect + ? minIndexToOverflowCorrect * adjustment + : minIndexToOverflowCorrect; + uint indexLargeEnough = curr > adjustedIndex ? 1U : 0U; + /* Only overflow correct early if the dictionary is invalidated already, + * so we don't hurt compression ratio. + */ + uint dictionaryInvalidated = curr > maxDist + loadedDictEnd ? 1U : 0U; + return indexLargeEnough != 0 && dictionaryInvalidated != 0 ? 1U : 0U; + } + + /** + * ZSTD_window_needOverflowCorrection(): + * Returns non-zero if the indices are getting too large and need overflow + * protection. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_window_needOverflowCorrection( + ZSTD_window_t window, + uint cycleLog, + uint maxDist, + uint loadedDictEnd, + void* src, + void* srcEnd + ) + { + uint curr = (uint)((byte*)srcEnd - window.@base); + return curr > (MEM_64bits ? 3500U * (1 << 20) : 2000U * (1 << 20)) ? 1U : 0U; + } + + /** + * ZSTD_window_correctOverflow(): + * Reduces the indices to protect from index overflow. + * Returns the correction made to the indices, which must be applied to every + * stored index. + * + * The least significant cycleLog bits of the indices must remain the same, + * which may be 0. Every index up to maxDist in the past must be valid. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_window_correctOverflow( + ZSTD_window_t* window, + uint cycleLog, + uint maxDist, + void* src + ) + { + /* preemptive overflow correction: + * 1. correction is large enough: + * lowLimit > (3<<29) ==> current > 3<<29 + 1< (3<<29 + 1< (3<<29) - (1< (3<<29) - (1<<30) (NOTE: chainLog <= 30) + * > 1<<29 + * + * 2. (ip+ZSTD_CHUNKSIZE_MAX - cctx->base) doesn't overflow: + * After correction, current is less than (1<base < 1<<32. + * 3. (cctx->lowLimit + 1< 3<<29 + 1<@base); + uint currentCycle = curr & cycleMask; + /* Ensure newCurrent - maxDist >= ZSTD_WINDOW_START_INDEX. */ + uint currentCycleCorrection = + currentCycle < 2 + ? cycleSize > 2 + ? cycleSize + : 2 + : 0; + uint newCurrent = + currentCycle + currentCycleCorrection + (maxDist > cycleSize ? maxDist : cycleSize); + uint correction = curr - newCurrent; + assert((maxDist & maxDist - 1) == 0); + assert((curr & cycleMask) == (newCurrent & cycleMask)); + assert(curr > newCurrent); + { + assert(correction > 1 << 28); + } + + window->@base += correction; + window->dictBase += correction; + if (window->lowLimit < correction + 2) + { + window->lowLimit = 2; + } + else + { + window->lowLimit -= correction; + } + + if (window->dictLimit < correction + 2) + { + window->dictLimit = 2; + } + else + { + window->dictLimit -= correction; + } + + assert(newCurrent >= maxDist); + assert(newCurrent - maxDist >= 2); + assert(window->lowLimit <= newCurrent); + assert(window->dictLimit <= newCurrent); + ++window->nbOverflowCorrections; + return correction; + } + + /** + * ZSTD_window_enforceMaxDist(): + * Updates lowLimit so that: + * (srcEnd - base) - lowLimit == maxDist + loadedDictEnd + * + * It ensures index is valid as long as index >= lowLimit. + * This must be called before a block compression call. + * + * loadedDictEnd is only defined if a dictionary is in use for current compression. + * As the name implies, loadedDictEnd represents the index at end of dictionary. + * The value lies within context's referential, it can be directly compared to blockEndIdx. + * + * If loadedDictEndPtr is NULL, no dictionary is in use, and we use loadedDictEnd == 0. + * If loadedDictEndPtr is not NULL, we set it to zero after updating lowLimit. + * This is because dictionaries are allowed to be referenced fully + * as long as the last byte of the dictionary is in the window. + * Once input has progressed beyond window size, dictionary cannot be referenced anymore. + * + * In normal dict mode, the dictionary lies between lowLimit and dictLimit. + * In dictMatchState mode, lowLimit and dictLimit are the same, + * and the dictionary is below them. + * forceWindow and dictMatchState are therefore incompatible. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_window_enforceMaxDist( + ZSTD_window_t* window, + void* blockEnd, + uint maxDist, + uint* loadedDictEndPtr, + ZSTD_MatchState_t** dictMatchStatePtr + ) + { + uint blockEndIdx = (uint)((byte*)blockEnd - window->@base); + uint loadedDictEnd = loadedDictEndPtr != null ? *loadedDictEndPtr : 0; + if (blockEndIdx > maxDist + loadedDictEnd) + { + uint newLowLimit = blockEndIdx - maxDist; + if (window->lowLimit < newLowLimit) + { + window->lowLimit = newLowLimit; + } + + if (window->dictLimit < window->lowLimit) + { + window->dictLimit = window->lowLimit; + } + + if (loadedDictEndPtr != null) + { + *loadedDictEndPtr = 0; + } + + if (dictMatchStatePtr != null) + { + *dictMatchStatePtr = null; + } + } + } + + /* Similar to ZSTD_window_enforceMaxDist(), + * but only invalidates dictionary + * when input progresses beyond window size. + * assumption : loadedDictEndPtr and dictMatchStatePtr are valid (non NULL) + * loadedDictEnd uses same referential as window->base + * maxDist is the window size */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_checkDictValidity( + ZSTD_window_t* window, + void* blockEnd, + uint maxDist, + uint* loadedDictEndPtr, + ZSTD_MatchState_t** dictMatchStatePtr + ) + { + assert(loadedDictEndPtr != null); + assert(dictMatchStatePtr != null); + { + uint blockEndIdx = (uint)((byte*)blockEnd - window->@base); + uint loadedDictEnd = *loadedDictEndPtr; + assert(blockEndIdx >= loadedDictEnd); + if (blockEndIdx > loadedDictEnd + maxDist || loadedDictEnd != window->dictLimit) + { + *loadedDictEndPtr = 0; + *dictMatchStatePtr = null; + } + } + } + +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_stringToByte_20_00 => new byte[] { 32, 0 }; + private static byte* stringToByte_20_00 => + (byte*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_stringToByte_20_00) + ); +#else + + private static readonly byte* stringToByte_20_00 = GetArrayPointer(new byte[] { 32, 0 }); +#endif + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_window_init(ZSTD_window_t* window) + { + *window = new ZSTD_window_t + { + @base = stringToByte_20_00, + dictBase = stringToByte_20_00, + dictLimit = 2, + lowLimit = 2, + nextSrc = stringToByte_20_00 + 2, + nbOverflowCorrections = 0, + }; + } + + /** + * ZSTD_window_update(): + * Updates the window by appending [src, src + srcSize) to the window. + * If it is not contiguous, the current prefix becomes the extDict, and we + * forget about the extDict. Handles overlap of the prefix and extDict. + * Returns non-zero if the segment is contiguous. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_window_update( + ZSTD_window_t* window, + void* src, + nuint srcSize, + int forceNonContiguous + ) + { + byte* ip = (byte*)src; + uint contiguous = 1; + if (srcSize == 0) + { + return contiguous; + } + + assert(window->@base != null); + assert(window->dictBase != null); + if (src != window->nextSrc || forceNonContiguous != 0) + { + /* not contiguous */ + nuint distanceFromBase = (nuint)(window->nextSrc - window->@base); + window->lowLimit = window->dictLimit; + assert(distanceFromBase == (uint)distanceFromBase); + window->dictLimit = (uint)distanceFromBase; + window->dictBase = window->@base; + window->@base = ip - distanceFromBase; + if (window->dictLimit - window->lowLimit < 8) + { + window->lowLimit = window->dictLimit; + } + + contiguous = 0; + } + + window->nextSrc = ip + srcSize; + if ( + ip + srcSize > window->dictBase + window->lowLimit + && ip < window->dictBase + window->dictLimit + ) + { + nuint highInputIdx = (nuint)(ip + srcSize - window->dictBase); + uint lowLimitMax = + highInputIdx > window->dictLimit ? window->dictLimit : (uint)highInputIdx; + assert(highInputIdx < 0xffffffff); + window->lowLimit = lowLimitMax; + } + + return contiguous; + } + + /** + * Returns the lowest allowed match index. It may either be in the ext-dict or the prefix. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_getLowestMatchIndex(ZSTD_MatchState_t* ms, uint curr, uint windowLog) + { + uint maxDistance = 1U << (int)windowLog; + uint lowestValid = ms->window.lowLimit; + uint withinWindow = curr - lowestValid > maxDistance ? curr - maxDistance : lowestValid; + uint isDictionary = ms->loadedDictEnd != 0 ? 1U : 0U; + /* When using a dictionary the entire dictionary is valid if a single byte of the dictionary + * is within the window. We invalidate the dictionary (and set loadedDictEnd to 0) when it isn't + * valid for the entire block. So this check is sufficient to find the lowest valid match index. + */ + uint matchLowest = isDictionary != 0 ? lowestValid : withinWindow; + return matchLowest; + } + + /** + * Returns the lowest allowed match index in the prefix. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_getLowestPrefixIndex(ZSTD_MatchState_t* ms, uint curr, uint windowLog) + { + uint maxDistance = 1U << (int)windowLog; + uint lowestValid = ms->window.dictLimit; + uint withinWindow = curr - lowestValid > maxDistance ? curr - maxDistance : lowestValid; + uint isDictionary = ms->loadedDictEnd != 0 ? 1U : 0U; + /* When computing the lowest prefix index we need to take the dictionary into account to handle + * the edge case where the dictionary and the source are contiguous in memory. + */ + uint matchLowest = isDictionary != 0 ? lowestValid : withinWindow; + return matchLowest; + } + + /* index_safety_check: + * intentional underflow : ensure repIndex isn't overlapping dict + prefix + * @return 1 if values are not overlapping, + * 0 otherwise */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static int ZSTD_index_overlap_check(uint prefixLowestIndex, uint repIndex) + { + return prefixLowestIndex - 1 - repIndex >= 3 ? 1 : 0; + } + + /* Helper function for ZSTD_fillHashTable and ZSTD_fillDoubleHashTable. + * Unpacks hashAndTag into (hash, tag), then packs (index, tag) into hashTable[hash]. */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_writeTaggedIndex(uint* hashTable, nuint hashAndTag, uint index) + { + nuint hash = hashAndTag >> 8; + uint tag = (uint)(hashAndTag & (1U << 8) - 1); + assert(index >> 32 - 8 == 0); + hashTable[hash] = index << 8 | tag; + } + + /* Helper function for short cache matchfinders. + * Unpacks tag1 and tag2 from lower bits of packedTag1 and packedTag2, then checks if the tags match. */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static int ZSTD_comparePackedTags(nuint packedTag1, nuint packedTag2) + { + uint tag1 = (uint)(packedTag1 & (1U << 8) - 1); + uint tag2 = (uint)(packedTag2 & (1U << 8) - 1); + return tag1 == tag2 ? 1 : 0; + } + + /* Returns 1 if an external sequence producer is registered, otherwise returns 0. */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static int ZSTD_hasExtSeqProd(ZSTD_CCtx_params_s* @params) + { + return @params->extSeqProdFunc != null ? 1 : 0; + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdCompressLiterals.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdCompressLiterals.cs new file mode 100644 index 00000000..c4af37eb --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdCompressLiterals.cs @@ -0,0 +1,320 @@ +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + /* ************************************************************** + * Literals compression - special cases + ****************************************************************/ + private static nuint ZSTD_noCompressLiterals( + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize + ) + { + byte* ostart = (byte*)dst; + uint flSize = (uint)(1 + (srcSize > 31 ? 1 : 0) + (srcSize > 4095 ? 1 : 0)); + if (srcSize + flSize > dstCapacity) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + switch (flSize) + { + case 1: + ostart[0] = (byte)((uint)SymbolEncodingType_e.set_basic + (srcSize << 3)); + break; + case 2: + MEM_writeLE16( + ostart, + (ushort)((uint)SymbolEncodingType_e.set_basic + (1 << 2) + (srcSize << 4)) + ); + break; + case 3: + MEM_writeLE32( + ostart, + (uint)((uint)SymbolEncodingType_e.set_basic + (3 << 2) + (srcSize << 4)) + ); + break; + default: + assert(0 != 0); + break; + } + + memcpy(ostart + flSize, src, (uint)srcSize); + return srcSize + flSize; + } + + private static int allBytesIdentical(void* src, nuint srcSize) + { + assert(srcSize >= 1); + assert(src != null); + { + byte b = ((byte*)src)[0]; + nuint p; + for (p = 1; p < srcSize; p++) + { + if (((byte*)src)[p] != b) + { + return 0; + } + } + + return 1; + } + } + + /* ZSTD_compressRleLiteralsBlock() : + * Conditions : + * - All bytes in @src are identical + * - dstCapacity >= 4 */ + private static nuint ZSTD_compressRleLiteralsBlock( + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize + ) + { + byte* ostart = (byte*)dst; + uint flSize = (uint)(1 + (srcSize > 31 ? 1 : 0) + (srcSize > 4095 ? 1 : 0)); + assert(dstCapacity >= 4); + assert(allBytesIdentical(src, srcSize) != 0); + switch (flSize) + { + case 1: + ostart[0] = (byte)((uint)SymbolEncodingType_e.set_rle + (srcSize << 3)); + break; + case 2: + MEM_writeLE16( + ostart, + (ushort)((uint)SymbolEncodingType_e.set_rle + (1 << 2) + (srcSize << 4)) + ); + break; + case 3: + MEM_writeLE32( + ostart, + (uint)((uint)SymbolEncodingType_e.set_rle + (3 << 2) + (srcSize << 4)) + ); + break; + default: + assert(0 != 0); + break; + } + + ostart[flSize] = *(byte*)src; + return flSize + 1; + } + + /* ZSTD_minLiteralsToCompress() : + * returns minimal amount of literals + * for literal compression to even be attempted. + * Minimum is made tighter as compression strategy increases. + */ + private static nuint ZSTD_minLiteralsToCompress(ZSTD_strategy strategy, HUF_repeat huf_repeat) + { + assert((int)strategy >= 0); + assert((int)strategy <= 9); + { + int shift = 9 - (int)strategy < 3 ? 9 - (int)strategy : 3; + nuint mintc = huf_repeat == HUF_repeat.HUF_repeat_valid ? 6 : (nuint)8 << shift; + return mintc; + } + } + + /* ZSTD_compressLiterals(): + * @entropyWorkspace: must be aligned on 4-bytes boundaries + * @entropyWorkspaceSize : must be >= HUF_WORKSPACE_SIZE + * @suspectUncompressible: sampling checks, to potentially skip huffman coding + */ + private static nuint ZSTD_compressLiterals( + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize, + void* entropyWorkspace, + nuint entropyWorkspaceSize, + ZSTD_hufCTables_t* prevHuf, + ZSTD_hufCTables_t* nextHuf, + ZSTD_strategy strategy, + int disableLiteralCompression, + int suspectUncompressible, + int bmi2 + ) + { + nuint lhSize = (nuint)( + 3 + (srcSize >= 1 * (1 << 10) ? 1 : 0) + (srcSize >= 16 * (1 << 10) ? 1 : 0) + ); + byte* ostart = (byte*)dst; + uint singleStream = srcSize < 256 ? 1U : 0U; + SymbolEncodingType_e hType = SymbolEncodingType_e.set_compressed; + nuint cLitSize; + memcpy(nextHuf, prevHuf, (uint)sizeof(ZSTD_hufCTables_t)); + if (disableLiteralCompression != 0) + { + return ZSTD_noCompressLiterals(dst, dstCapacity, src, srcSize); + } + + if (srcSize < ZSTD_minLiteralsToCompress(strategy, prevHuf->repeatMode)) + { + return ZSTD_noCompressLiterals(dst, dstCapacity, src, srcSize); + } + + if (dstCapacity < lhSize + 1) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + { + HUF_repeat repeat = prevHuf->repeatMode; + int flags = + 0 + | (bmi2 != 0 ? (int)HUF_flags_e.HUF_flags_bmi2 : 0) + | ( + strategy < ZSTD_strategy.ZSTD_lazy && srcSize <= 1024 + ? (int)HUF_flags_e.HUF_flags_preferRepeat + : 0 + ) + | ( + strategy >= ZSTD_strategy.ZSTD_btultra + ? (int)HUF_flags_e.HUF_flags_optimalDepth + : 0 + ) + | ( + suspectUncompressible != 0 + ? (int)HUF_flags_e.HUF_flags_suspectUncompressible + : 0 + ); + void* huf_compress; + if (repeat == HUF_repeat.HUF_repeat_valid && lhSize == 3) + { + singleStream = 1; + } + + huf_compress = + singleStream != 0 + ? (delegate* managed< + void*, + nuint, + void*, + nuint, + uint, + uint, + void*, + nuint, + nuint*, + HUF_repeat*, + int, + nuint>)(&HUF_compress1X_repeat) + : (delegate* managed< + void*, + nuint, + void*, + nuint, + uint, + uint, + void*, + nuint, + nuint*, + HUF_repeat*, + int, + nuint>)(&HUF_compress4X_repeat); + cLitSize = ( + (delegate* managed< + void*, + nuint, + void*, + nuint, + uint, + uint, + void*, + nuint, + nuint*, + HUF_repeat*, + int, + nuint>)huf_compress + )( + ostart + lhSize, + dstCapacity - lhSize, + src, + srcSize, + 255, + 11, + entropyWorkspace, + entropyWorkspaceSize, + &nextHuf->CTable.e0, + &repeat, + flags + ); + if (repeat != HUF_repeat.HUF_repeat_none) + { + hType = SymbolEncodingType_e.set_repeat; + } + } + + { + nuint minGain = ZSTD_minGain(srcSize, strategy); + if (cLitSize == 0 || cLitSize >= srcSize - minGain || ERR_isError(cLitSize)) + { + memcpy(nextHuf, prevHuf, (uint)sizeof(ZSTD_hufCTables_t)); + return ZSTD_noCompressLiterals(dst, dstCapacity, src, srcSize); + } + } + + if (cLitSize == 1) + { + if (srcSize >= 8 || allBytesIdentical(src, srcSize) != 0) + { + memcpy(nextHuf, prevHuf, (uint)sizeof(ZSTD_hufCTables_t)); + return ZSTD_compressRleLiteralsBlock(dst, dstCapacity, src, srcSize); + } + } + + if (hType == SymbolEncodingType_e.set_compressed) + { + nextHuf->repeatMode = HUF_repeat.HUF_repeat_check; + } + + switch (lhSize) + { + case 3: + { + uint lhc = + (uint)hType + + ((singleStream == 0 ? 1U : 0U) << 2) + + ((uint)srcSize << 4) + + ((uint)cLitSize << 14); + MEM_writeLE24(ostart, lhc); + break; + } + + case 4: + assert(srcSize >= 6); + + { + uint lhc = + (uint)(hType + (2 << 2)) + ((uint)srcSize << 4) + ((uint)cLitSize << 18); + MEM_writeLE32(ostart, lhc); + break; + } + + case 5: + assert(srcSize >= 6); + + { + uint lhc = + (uint)(hType + (3 << 2)) + ((uint)srcSize << 4) + ((uint)cLitSize << 22); + MEM_writeLE32(ostart, lhc); + ostart[4] = (byte)(cLitSize >> 10); + break; + } + + default: + assert(0 != 0); + break; + } + + return lhSize + cLitSize; + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdCompressSequences.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdCompressSequences.cs new file mode 100644 index 00000000..10dca3cb --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdCompressSequences.cs @@ -0,0 +1,1228 @@ +using System; +using System.Runtime.InteropServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_kInverseProbabilityLog256 => + new uint[256] + { + 0, + 2048, + 1792, + 1642, + 1536, + 1453, + 1386, + 1329, + 1280, + 1236, + 1197, + 1162, + 1130, + 1100, + 1073, + 1047, + 1024, + 1001, + 980, + 960, + 941, + 923, + 906, + 889, + 874, + 859, + 844, + 830, + 817, + 804, + 791, + 779, + 768, + 756, + 745, + 734, + 724, + 714, + 704, + 694, + 685, + 676, + 667, + 658, + 650, + 642, + 633, + 626, + 618, + 610, + 603, + 595, + 588, + 581, + 574, + 567, + 561, + 554, + 548, + 542, + 535, + 529, + 523, + 517, + 512, + 506, + 500, + 495, + 489, + 484, + 478, + 473, + 468, + 463, + 458, + 453, + 448, + 443, + 438, + 434, + 429, + 424, + 420, + 415, + 411, + 407, + 402, + 398, + 394, + 390, + 386, + 382, + 377, + 373, + 370, + 366, + 362, + 358, + 354, + 350, + 347, + 343, + 339, + 336, + 332, + 329, + 325, + 322, + 318, + 315, + 311, + 308, + 305, + 302, + 298, + 295, + 292, + 289, + 286, + 282, + 279, + 276, + 273, + 270, + 267, + 264, + 261, + 258, + 256, + 253, + 250, + 247, + 244, + 241, + 239, + 236, + 233, + 230, + 228, + 225, + 222, + 220, + 217, + 215, + 212, + 209, + 207, + 204, + 202, + 199, + 197, + 194, + 192, + 190, + 187, + 185, + 182, + 180, + 178, + 175, + 173, + 171, + 168, + 166, + 164, + 162, + 159, + 157, + 155, + 153, + 151, + 149, + 146, + 144, + 142, + 140, + 138, + 136, + 134, + 132, + 130, + 128, + 126, + 123, + 121, + 119, + 117, + 115, + 114, + 112, + 110, + 108, + 106, + 104, + 102, + 100, + 98, + 96, + 94, + 93, + 91, + 89, + 87, + 85, + 83, + 82, + 80, + 78, + 76, + 74, + 73, + 71, + 69, + 67, + 66, + 64, + 62, + 61, + 59, + 57, + 55, + 54, + 52, + 50, + 49, + 47, + 46, + 44, + 42, + 41, + 39, + 37, + 36, + 34, + 33, + 31, + 30, + 28, + 26, + 25, + 23, + 22, + 20, + 19, + 17, + 16, + 14, + 13, + 11, + 10, + 8, + 7, + 5, + 4, + 2, + 1, + }; + private static uint* kInverseProbabilityLog256 => + (uint*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_kInverseProbabilityLog256) + ); +#else + + private static readonly uint* kInverseProbabilityLog256 = GetArrayPointer( + new uint[256] + { + 0, + 2048, + 1792, + 1642, + 1536, + 1453, + 1386, + 1329, + 1280, + 1236, + 1197, + 1162, + 1130, + 1100, + 1073, + 1047, + 1024, + 1001, + 980, + 960, + 941, + 923, + 906, + 889, + 874, + 859, + 844, + 830, + 817, + 804, + 791, + 779, + 768, + 756, + 745, + 734, + 724, + 714, + 704, + 694, + 685, + 676, + 667, + 658, + 650, + 642, + 633, + 626, + 618, + 610, + 603, + 595, + 588, + 581, + 574, + 567, + 561, + 554, + 548, + 542, + 535, + 529, + 523, + 517, + 512, + 506, + 500, + 495, + 489, + 484, + 478, + 473, + 468, + 463, + 458, + 453, + 448, + 443, + 438, + 434, + 429, + 424, + 420, + 415, + 411, + 407, + 402, + 398, + 394, + 390, + 386, + 382, + 377, + 373, + 370, + 366, + 362, + 358, + 354, + 350, + 347, + 343, + 339, + 336, + 332, + 329, + 325, + 322, + 318, + 315, + 311, + 308, + 305, + 302, + 298, + 295, + 292, + 289, + 286, + 282, + 279, + 276, + 273, + 270, + 267, + 264, + 261, + 258, + 256, + 253, + 250, + 247, + 244, + 241, + 239, + 236, + 233, + 230, + 228, + 225, + 222, + 220, + 217, + 215, + 212, + 209, + 207, + 204, + 202, + 199, + 197, + 194, + 192, + 190, + 187, + 185, + 182, + 180, + 178, + 175, + 173, + 171, + 168, + 166, + 164, + 162, + 159, + 157, + 155, + 153, + 151, + 149, + 146, + 144, + 142, + 140, + 138, + 136, + 134, + 132, + 130, + 128, + 126, + 123, + 121, + 119, + 117, + 115, + 114, + 112, + 110, + 108, + 106, + 104, + 102, + 100, + 98, + 96, + 94, + 93, + 91, + 89, + 87, + 85, + 83, + 82, + 80, + 78, + 76, + 74, + 73, + 71, + 69, + 67, + 66, + 64, + 62, + 61, + 59, + 57, + 55, + 54, + 52, + 50, + 49, + 47, + 46, + 44, + 42, + 41, + 39, + 37, + 36, + 34, + 33, + 31, + 30, + 28, + 26, + 25, + 23, + 22, + 20, + 19, + 17, + 16, + 14, + 13, + 11, + 10, + 8, + 7, + 5, + 4, + 2, + 1, + } + ); +#endif + + private static uint ZSTD_getFSEMaxSymbolValue(uint* ctable) + { + void* ptr = ctable; + ushort* u16ptr = (ushort*)ptr; + uint maxSymbolValue = MEM_read16(u16ptr + 1); + return maxSymbolValue; + } + + /** + * Returns true if we should use ncount=-1 else we should + * use ncount=1 for low probability symbols instead. + */ + private static uint ZSTD_useLowProbCount(nuint nbSeq) + { + return nbSeq >= 2048 ? 1U : 0U; + } + + /** + * Returns the cost in bytes of encoding the normalized count header. + * Returns an error if any of the helper functions return an error. + */ + private static nuint ZSTD_NCountCost(uint* count, uint max, nuint nbSeq, uint FSELog) + { + byte* wksp = stackalloc byte[512]; + short* norm = stackalloc short[53]; + uint tableLog = FSE_optimalTableLog(FSELog, nbSeq, max); + { + nuint err_code = FSE_normalizeCount( + norm, + tableLog, + count, + nbSeq, + max, + ZSTD_useLowProbCount(nbSeq) + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return FSE_writeNCount(wksp, sizeof(byte) * 512, norm, max, tableLog); + } + + /** + * Returns the cost in bits of encoding the distribution described by count + * using the entropy bound. + */ + private static nuint ZSTD_entropyCost(uint* count, uint max, nuint total) + { + uint cost = 0; + uint s; + assert(total > 0); + for (s = 0; s <= max; ++s) + { + uint norm = (uint)(256 * count[s] / total); + if (count[s] != 0 && norm == 0) + { + norm = 1; + } + + assert(count[s] < total); + cost += count[s] * kInverseProbabilityLog256[norm]; + } + + return cost >> 8; + } + + /** + * Returns the cost in bits of encoding the distribution in count using ctable. + * Returns an error if ctable cannot represent all the symbols in count. + */ + private static nuint ZSTD_fseBitCost(uint* ctable, uint* count, uint max) + { + const uint kAccuracyLog = 8; + nuint cost = 0; + uint s; + FSE_CState_t cstate; + FSE_initCState(&cstate, ctable); + if (ZSTD_getFSEMaxSymbolValue(ctable) < max) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + + for (s = 0; s <= max; ++s) + { + uint tableLog = cstate.stateLog; + uint badCost = tableLog + 1 << (int)kAccuracyLog; + uint bitCost = FSE_bitCost(cstate.symbolTT, tableLog, s, kAccuracyLog); + if (count[s] == 0) + { + continue; + } + + if (bitCost >= badCost) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + + cost += (nuint)count[s] * bitCost; + } + + return cost >> (int)kAccuracyLog; + } + + /** + * Returns the cost in bits of encoding the distribution in count using the + * table described by norm. The max symbol support by norm is assumed >= max. + * norm must be valid for every symbol with non-zero probability in count. + */ + private static nuint ZSTD_crossEntropyCost(short* norm, uint accuracyLog, uint* count, uint max) + { + uint shift = 8 - accuracyLog; + nuint cost = 0; + uint s; + assert(accuracyLog <= 8); + for (s = 0; s <= max; ++s) + { + uint normAcc = norm[s] != -1 ? (uint)norm[s] : 1; + uint norm256 = normAcc << (int)shift; + assert(norm256 > 0); + assert(norm256 < 256); + cost += count[s] * kInverseProbabilityLog256[norm256]; + } + + return cost >> 8; + } + + private static SymbolEncodingType_e ZSTD_selectEncodingType( + FSE_repeat* repeatMode, + uint* count, + uint max, + nuint mostFrequent, + nuint nbSeq, + uint FSELog, + uint* prevCTable, + short* defaultNorm, + uint defaultNormLog, + ZSTD_DefaultPolicy_e isDefaultAllowed, + ZSTD_strategy strategy + ) + { + if (mostFrequent == nbSeq) + { + *repeatMode = FSE_repeat.FSE_repeat_none; + if (isDefaultAllowed != default && nbSeq <= 2) + { + return SymbolEncodingType_e.set_basic; + } + + return SymbolEncodingType_e.set_rle; + } + + if (strategy < ZSTD_strategy.ZSTD_lazy) + { + if (isDefaultAllowed != default) + { + const nuint staticFse_nbSeq_max = 1000; + nuint mult = (nuint)(10 - strategy); + const nuint baseLog = 3; + /* 28-36 for offset, 56-72 for lengths */ + nuint dynamicFse_nbSeq_min = + ((nuint)1 << (int)defaultNormLog) * mult >> (int)baseLog; + assert(defaultNormLog >= 5 && defaultNormLog <= 6); + assert(mult <= 9 && mult >= 7); + if (*repeatMode == FSE_repeat.FSE_repeat_valid && nbSeq < staticFse_nbSeq_max) + { + return SymbolEncodingType_e.set_repeat; + } + + if ( + nbSeq < dynamicFse_nbSeq_min + || mostFrequent < nbSeq >> (int)(defaultNormLog - 1) + ) + { + *repeatMode = FSE_repeat.FSE_repeat_none; + return SymbolEncodingType_e.set_basic; + } + } + } + else + { + nuint basicCost = + isDefaultAllowed != default + ? ZSTD_crossEntropyCost(defaultNorm, defaultNormLog, count, max) + : unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + nuint repeatCost = + *repeatMode != FSE_repeat.FSE_repeat_none + ? ZSTD_fseBitCost(prevCTable, count, max) + : unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + nuint NCountCost = ZSTD_NCountCost(count, max, nbSeq, FSELog); + nuint compressedCost = (NCountCost << 3) + ZSTD_entropyCost(count, max, nbSeq); + + assert(!ERR_isError(NCountCost)); + assert(compressedCost < unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_maxCode))); + if (basicCost <= repeatCost && basicCost <= compressedCost) + { + assert(isDefaultAllowed != default); + *repeatMode = FSE_repeat.FSE_repeat_none; + return SymbolEncodingType_e.set_basic; + } + + if (repeatCost <= compressedCost) + { + assert(!ERR_isError(repeatCost)); + return SymbolEncodingType_e.set_repeat; + } + + assert(compressedCost < basicCost && compressedCost < repeatCost); + } + + *repeatMode = FSE_repeat.FSE_repeat_check; + return SymbolEncodingType_e.set_compressed; + } + + private static nuint ZSTD_buildCTable( + void* dst, + nuint dstCapacity, + uint* nextCTable, + uint FSELog, + SymbolEncodingType_e type, + uint* count, + uint max, + byte* codeTable, + nuint nbSeq, + short* defaultNorm, + uint defaultNormLog, + uint defaultMax, + uint* prevCTable, + nuint prevCTableSize, + void* entropyWorkspace, + nuint entropyWorkspaceSize + ) + { + byte* op = (byte*)dst; + byte* oend = op + dstCapacity; + switch (type) + { + case SymbolEncodingType_e.set_rle: + { + nuint err_code = FSE_buildCTable_rle(nextCTable, (byte)max); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (dstCapacity == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + *op = codeTable[0]; + return 1; + case SymbolEncodingType_e.set_repeat: + memcpy(nextCTable, prevCTable, (uint)prevCTableSize); + return 0; + case SymbolEncodingType_e.set_basic: + { + /* note : could be pre-calculated */ + nuint err_code = FSE_buildCTable_wksp( + nextCTable, + defaultNorm, + defaultMax, + defaultNormLog, + entropyWorkspace, + entropyWorkspaceSize + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return 0; + case SymbolEncodingType_e.set_compressed: + { + ZSTD_BuildCTableWksp* wksp = (ZSTD_BuildCTableWksp*)entropyWorkspace; + nuint nbSeq_1 = nbSeq; + uint tableLog = FSE_optimalTableLog(FSELog, nbSeq, max); + if (count[codeTable[nbSeq - 1]] > 1) + { + count[codeTable[nbSeq - 1]]--; + nbSeq_1--; + } + + assert(nbSeq_1 > 1); + assert(entropyWorkspaceSize >= (nuint)sizeof(ZSTD_BuildCTableWksp)); + { + nuint err_code = FSE_normalizeCount( + wksp->norm, + tableLog, + count, + nbSeq_1, + max, + ZSTD_useLowProbCount(nbSeq_1) + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + assert(oend >= op); + { + /* overflow protected */ + nuint NCountSize = FSE_writeNCount( + op, + (nuint)(oend - op), + wksp->norm, + max, + tableLog + ); + { + nuint err_code = NCountSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = FSE_buildCTable_wksp( + nextCTable, + wksp->norm, + max, + tableLog, + wksp->wksp, + sizeof(uint) * 285 + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return NCountSize; + } + } + + default: + assert(0 != 0); + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + } + + private static nuint ZSTD_encodeSequences_body( + void* dst, + nuint dstCapacity, + uint* CTable_MatchLength, + byte* mlCodeTable, + uint* CTable_OffsetBits, + byte* ofCodeTable, + uint* CTable_LitLength, + byte* llCodeTable, + SeqDef_s* sequences, + nuint nbSeq, + int longOffsets + ) + { + BIT_CStream_t blockStream; + System.Runtime.CompilerServices.Unsafe.SkipInit(out blockStream); + FSE_CState_t stateMatchLength; + System.Runtime.CompilerServices.Unsafe.SkipInit(out stateMatchLength); + FSE_CState_t stateOffsetBits; + System.Runtime.CompilerServices.Unsafe.SkipInit(out stateOffsetBits); + FSE_CState_t stateLitLength; + System.Runtime.CompilerServices.Unsafe.SkipInit(out stateLitLength); + if (ERR_isError(BIT_initCStream(ref blockStream, dst, dstCapacity))) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + nuint blockStream_bitContainer = blockStream.bitContainer; + uint blockStream_bitPos = blockStream.bitPos; + sbyte* blockStream_ptr = blockStream.ptr; + sbyte* blockStream_endPtr = blockStream.endPtr; + FSE_initCState2(ref stateMatchLength, CTable_MatchLength, mlCodeTable[nbSeq - 1]); + FSE_initCState2(ref stateOffsetBits, CTable_OffsetBits, ofCodeTable[nbSeq - 1]); + FSE_initCState2(ref stateLitLength, CTable_LitLength, llCodeTable[nbSeq - 1]); + BIT_addBits( + ref blockStream_bitContainer, + ref blockStream_bitPos, + sequences[nbSeq - 1].litLength, + LL_bits[llCodeTable[nbSeq - 1]] + ); + if (MEM_32bits) + { + BIT_flushBits( + ref blockStream_bitContainer, + ref blockStream_bitPos, + ref blockStream_ptr, + blockStream_endPtr + ); + } + + BIT_addBits( + ref blockStream_bitContainer, + ref blockStream_bitPos, + sequences[nbSeq - 1].mlBase, + ML_bits[mlCodeTable[nbSeq - 1]] + ); + if (MEM_32bits) + { + BIT_flushBits( + ref blockStream_bitContainer, + ref blockStream_bitPos, + ref blockStream_ptr, + blockStream_endPtr + ); + } + + if (longOffsets != 0) + { + uint ofBits = ofCodeTable[nbSeq - 1]; + uint extraBits = + ofBits + - ( + ofBits < (uint)(MEM_32bits ? 25 : 57) - 1 + ? ofBits + : (uint)(MEM_32bits ? 25 : 57) - 1 + ); + if (extraBits != 0) + { + BIT_addBits( + ref blockStream_bitContainer, + ref blockStream_bitPos, + sequences[nbSeq - 1].offBase, + extraBits + ); + BIT_flushBits( + ref blockStream_bitContainer, + ref blockStream_bitPos, + ref blockStream_ptr, + blockStream_endPtr + ); + } + + BIT_addBits( + ref blockStream_bitContainer, + ref blockStream_bitPos, + sequences[nbSeq - 1].offBase >> (int)extraBits, + ofBits - extraBits + ); + } + else + { + BIT_addBits( + ref blockStream_bitContainer, + ref blockStream_bitPos, + sequences[nbSeq - 1].offBase, + ofCodeTable[nbSeq - 1] + ); + } + + BIT_flushBits( + ref blockStream_bitContainer, + ref blockStream_bitPos, + ref blockStream_ptr, + blockStream_endPtr + ); + { + nuint n; + for (n = nbSeq - 2; n < nbSeq; n--) + { + byte llCode = llCodeTable[n]; + byte ofCode = ofCodeTable[n]; + byte mlCode = mlCodeTable[n]; + uint llBits = LL_bits[llCode]; + uint ofBits = ofCode; + uint mlBits = ML_bits[mlCode]; + FSE_encodeSymbol( + ref blockStream_bitContainer, + ref blockStream_bitPos, + ref stateOffsetBits, + ofCode + ); + FSE_encodeSymbol( + ref blockStream_bitContainer, + ref blockStream_bitPos, + ref stateMatchLength, + mlCode + ); + if (MEM_32bits) + { + BIT_flushBits( + ref blockStream_bitContainer, + ref blockStream_bitPos, + ref blockStream_ptr, + blockStream_endPtr + ); + } + + FSE_encodeSymbol( + ref blockStream_bitContainer, + ref blockStream_bitPos, + ref stateLitLength, + llCode + ); + if (MEM_32bits || ofBits + mlBits + llBits >= 64 - 7 - (9 + 9 + 8)) + { + BIT_flushBits( + ref blockStream_bitContainer, + ref blockStream_bitPos, + ref blockStream_ptr, + blockStream_endPtr + ); + } + + BIT_addBits( + ref blockStream_bitContainer, + ref blockStream_bitPos, + sequences[n].litLength, + llBits + ); + if (MEM_32bits && llBits + mlBits > 24) + { + BIT_flushBits( + ref blockStream_bitContainer, + ref blockStream_bitPos, + ref blockStream_ptr, + blockStream_endPtr + ); + } + + BIT_addBits( + ref blockStream_bitContainer, + ref blockStream_bitPos, + sequences[n].mlBase, + mlBits + ); + if (MEM_32bits || ofBits + mlBits + llBits > 56) + { + BIT_flushBits( + ref blockStream_bitContainer, + ref blockStream_bitPos, + ref blockStream_ptr, + blockStream_endPtr + ); + } + + if (longOffsets != 0) + { + uint extraBits = + ofBits + - ( + ofBits < (uint)(MEM_32bits ? 25 : 57) - 1 + ? ofBits + : (uint)(MEM_32bits ? 25 : 57) - 1 + ); + if (extraBits != 0) + { + BIT_addBits( + ref blockStream_bitContainer, + ref blockStream_bitPos, + sequences[n].offBase, + extraBits + ); + BIT_flushBits( + ref blockStream_bitContainer, + ref blockStream_bitPos, + ref blockStream_ptr, + blockStream_endPtr + ); + } + + BIT_addBits( + ref blockStream_bitContainer, + ref blockStream_bitPos, + sequences[n].offBase >> (int)extraBits, + ofBits - extraBits + ); + } + else + { + BIT_addBits( + ref blockStream_bitContainer, + ref blockStream_bitPos, + sequences[n].offBase, + ofBits + ); + } + + BIT_flushBits( + ref blockStream_bitContainer, + ref blockStream_bitPos, + ref blockStream_ptr, + blockStream_endPtr + ); + } + } + + FSE_flushCState( + ref blockStream_bitContainer, + ref blockStream_bitPos, + ref blockStream_ptr, + blockStream_endPtr, + ref stateMatchLength + ); + FSE_flushCState( + ref blockStream_bitContainer, + ref blockStream_bitPos, + ref blockStream_ptr, + blockStream_endPtr, + ref stateOffsetBits + ); + FSE_flushCState( + ref blockStream_bitContainer, + ref blockStream_bitPos, + ref blockStream_ptr, + blockStream_endPtr, + ref stateLitLength + ); + { + nuint streamSize = BIT_closeCStream( + ref blockStream_bitContainer, + ref blockStream_bitPos, + blockStream_ptr, + blockStream_endPtr, + blockStream.startPtr + ); + if (streamSize == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + return streamSize; + } + } + + private static nuint ZSTD_encodeSequences_default( + void* dst, + nuint dstCapacity, + uint* CTable_MatchLength, + byte* mlCodeTable, + uint* CTable_OffsetBits, + byte* ofCodeTable, + uint* CTable_LitLength, + byte* llCodeTable, + SeqDef_s* sequences, + nuint nbSeq, + int longOffsets + ) + { + return ZSTD_encodeSequences_body( + dst, + dstCapacity, + CTable_MatchLength, + mlCodeTable, + CTable_OffsetBits, + ofCodeTable, + CTable_LitLength, + llCodeTable, + sequences, + nbSeq, + longOffsets + ); + } + + private static nuint ZSTD_encodeSequences( + void* dst, + nuint dstCapacity, + uint* CTable_MatchLength, + byte* mlCodeTable, + uint* CTable_OffsetBits, + byte* ofCodeTable, + uint* CTable_LitLength, + byte* llCodeTable, + SeqDef_s* sequences, + nuint nbSeq, + int longOffsets, + int bmi2 + ) + { + return ZSTD_encodeSequences_default( + dst, + dstCapacity, + CTable_MatchLength, + mlCodeTable, + CTable_OffsetBits, + ofCodeTable, + CTable_LitLength, + llCodeTable, + sequences, + nbSeq, + longOffsets + ); + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdCompressSuperblock.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdCompressSuperblock.cs new file mode 100644 index 00000000..99364ad7 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdCompressSuperblock.cs @@ -0,0 +1,1033 @@ +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + /** ZSTD_compressSubBlock_literal() : + * Compresses literals section for a sub-block. + * When we have to write the Huffman table we will sometimes choose a header + * size larger than necessary. This is because we have to pick the header size + * before we know the table size + compressed size, so we have a bound on the + * table size. If we guessed incorrectly, we fall back to uncompressed literals. + * + * We write the header when writeEntropy=1 and set entropyWritten=1 when we succeeded + * in writing the header, otherwise it is set to 0. + * + * hufMetadata->hType has literals block type info. + * If it is set_basic, all sub-blocks literals section will be Raw_Literals_Block. + * If it is set_rle, all sub-blocks literals section will be RLE_Literals_Block. + * If it is set_compressed, first sub-block's literals section will be Compressed_Literals_Block + * If it is set_compressed, first sub-block's literals section will be Treeless_Literals_Block + * and the following sub-blocks' literals sections will be Treeless_Literals_Block. + * @return : compressed size of literals section of a sub-block + * Or 0 if unable to compress. + * Or error code */ + private static nuint ZSTD_compressSubBlock_literal( + nuint* hufTable, + ZSTD_hufCTablesMetadata_t* hufMetadata, + byte* literals, + nuint litSize, + void* dst, + nuint dstSize, + int bmi2, + int writeEntropy, + int* entropyWritten + ) + { + nuint header = (nuint)(writeEntropy != 0 ? 200 : 0); + nuint lhSize = (nuint)( + 3 + + (litSize >= 1 * (1 << 10) - header ? 1 : 0) + + (litSize >= 16 * (1 << 10) - header ? 1 : 0) + ); + byte* ostart = (byte*)dst; + byte* oend = ostart + dstSize; + byte* op = ostart + lhSize; + uint singleStream = lhSize == 3 ? 1U : 0U; + SymbolEncodingType_e hType = + writeEntropy != 0 ? hufMetadata->hType : SymbolEncodingType_e.set_repeat; + nuint cLitSize = 0; + *entropyWritten = 0; + if (litSize == 0 || hufMetadata->hType == SymbolEncodingType_e.set_basic) + { + return ZSTD_noCompressLiterals(dst, dstSize, literals, litSize); + } + else if (hufMetadata->hType == SymbolEncodingType_e.set_rle) + { + return ZSTD_compressRleLiteralsBlock(dst, dstSize, literals, litSize); + } + + assert(litSize > 0); + assert( + hufMetadata->hType == SymbolEncodingType_e.set_compressed + || hufMetadata->hType == SymbolEncodingType_e.set_repeat + ); + if (writeEntropy != 0 && hufMetadata->hType == SymbolEncodingType_e.set_compressed) + { + memcpy(op, hufMetadata->hufDesBuffer, (uint)hufMetadata->hufDesSize); + op += hufMetadata->hufDesSize; + cLitSize += hufMetadata->hufDesSize; + } + + { + int flags = bmi2 != 0 ? (int)HUF_flags_e.HUF_flags_bmi2 : 0; + nuint cSize = + singleStream != 0 + ? HUF_compress1X_usingCTable( + op, + (nuint)(oend - op), + literals, + litSize, + hufTable, + flags + ) + : HUF_compress4X_usingCTable( + op, + (nuint)(oend - op), + literals, + litSize, + hufTable, + flags + ); + op += cSize; + cLitSize += cSize; + if (cSize == 0 || ERR_isError(cSize)) + { + return 0; + } + + if (writeEntropy == 0 && cLitSize >= litSize) + { + return ZSTD_noCompressLiterals(dst, dstSize, literals, litSize); + } + + if ( + lhSize + < (nuint)( + 3 + (cLitSize >= 1 * (1 << 10) ? 1 : 0) + (cLitSize >= 16 * (1 << 10) ? 1 : 0) + ) + ) + { + assert(cLitSize > litSize); + return ZSTD_noCompressLiterals(dst, dstSize, literals, litSize); + } + } + + switch (lhSize) + { + case 3: + { + uint lhc = + (uint)hType + + ((singleStream == 0 ? 1U : 0U) << 2) + + ((uint)litSize << 4) + + ((uint)cLitSize << 14); + MEM_writeLE24(ostart, lhc); + break; + } + + case 4: + { + uint lhc = (uint)(hType + (2 << 2)) + ((uint)litSize << 4) + ((uint)cLitSize << 18); + MEM_writeLE32(ostart, lhc); + break; + } + + case 5: + { + uint lhc = (uint)(hType + (3 << 2)) + ((uint)litSize << 4) + ((uint)cLitSize << 22); + MEM_writeLE32(ostart, lhc); + ostart[4] = (byte)(cLitSize >> 10); + break; + } + + default: + assert(0 != 0); + break; + } + + *entropyWritten = 1; + return (nuint)(op - ostart); + } + + private static nuint ZSTD_seqDecompressedSize( + SeqStore_t* seqStore, + SeqDef_s* sequences, + nuint nbSeqs, + nuint litSize, + int lastSubBlock + ) + { + nuint matchLengthSum = 0; + nuint litLengthSum = 0; + nuint n; + for (n = 0; n < nbSeqs; n++) + { + ZSTD_SequenceLength seqLen = ZSTD_getSequenceLength(seqStore, sequences + n); + litLengthSum += seqLen.litLength; + matchLengthSum += seqLen.matchLength; + } + + if (lastSubBlock == 0) + { + assert(litLengthSum == litSize); + } + else + { + assert(litLengthSum <= litSize); + } + + return matchLengthSum + litSize; + } + + /** ZSTD_compressSubBlock_sequences() : + * Compresses sequences section for a sub-block. + * fseMetadata->llType, fseMetadata->ofType, and fseMetadata->mlType have + * symbol compression modes for the super-block. + * The first successfully compressed block will have these in its header. + * We set entropyWritten=1 when we succeed in compressing the sequences. + * The following sub-blocks will always have repeat mode. + * @return : compressed size of sequences section of a sub-block + * Or 0 if it is unable to compress + * Or error code. */ + private static nuint ZSTD_compressSubBlock_sequences( + ZSTD_fseCTables_t* fseTables, + ZSTD_fseCTablesMetadata_t* fseMetadata, + SeqDef_s* sequences, + nuint nbSeq, + byte* llCode, + byte* mlCode, + byte* ofCode, + ZSTD_CCtx_params_s* cctxParams, + void* dst, + nuint dstCapacity, + int bmi2, + int writeEntropy, + int* entropyWritten + ) + { + int longOffsets = cctxParams->cParams.windowLog > (uint)(MEM_32bits ? 25 : 57) ? 1 : 0; + byte* ostart = (byte*)dst; + byte* oend = ostart + dstCapacity; + byte* op = ostart; + byte* seqHead; + *entropyWritten = 0; + if (oend - op < 3 + 1) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + if (nbSeq < 128) + { + *op++ = (byte)nbSeq; + } + else if (nbSeq < 0x7F00) + { + op[0] = (byte)((nbSeq >> 8) + 0x80); + op[1] = (byte)nbSeq; + op += 2; + } + else + { + op[0] = 0xFF; + MEM_writeLE16(op + 1, (ushort)(nbSeq - 0x7F00)); + op += 3; + } + + if (nbSeq == 0) + { + return (nuint)(op - ostart); + } + + seqHead = op++; + if (writeEntropy != 0) + { + uint LLtype = (uint)fseMetadata->llType; + uint Offtype = (uint)fseMetadata->ofType; + uint MLtype = (uint)fseMetadata->mlType; + *seqHead = (byte)((LLtype << 6) + (Offtype << 4) + (MLtype << 2)); + memcpy(op, fseMetadata->fseTablesBuffer, (uint)fseMetadata->fseTablesSize); + op += fseMetadata->fseTablesSize; + } + else + { + uint repeat = (uint)SymbolEncodingType_e.set_repeat; + *seqHead = (byte)((repeat << 6) + (repeat << 4) + (repeat << 2)); + } + + { + nuint bitstreamSize = ZSTD_encodeSequences( + op, + (nuint)(oend - op), + fseTables->matchlengthCTable, + mlCode, + fseTables->offcodeCTable, + ofCode, + fseTables->litlengthCTable, + llCode, + sequences, + nbSeq, + longOffsets, + bmi2 + ); + { + nuint err_code = bitstreamSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + op += bitstreamSize; + if ( + writeEntropy != 0 + && fseMetadata->lastCountSize != 0 + && fseMetadata->lastCountSize + bitstreamSize < 4 + ) + { + assert(fseMetadata->lastCountSize + bitstreamSize == 3); + return 0; + } + } + + if (op - seqHead < 4) + { + return 0; + } + + *entropyWritten = 1; + return (nuint)(op - ostart); + } + + /** ZSTD_compressSubBlock() : + * Compresses a single sub-block. + * @return : compressed size of the sub-block + * Or 0 if it failed to compress. */ + private static nuint ZSTD_compressSubBlock( + ZSTD_entropyCTables_t* entropy, + ZSTD_entropyCTablesMetadata_t* entropyMetadata, + SeqDef_s* sequences, + nuint nbSeq, + byte* literals, + nuint litSize, + byte* llCode, + byte* mlCode, + byte* ofCode, + ZSTD_CCtx_params_s* cctxParams, + void* dst, + nuint dstCapacity, + int bmi2, + int writeLitEntropy, + int writeSeqEntropy, + int* litEntropyWritten, + int* seqEntropyWritten, + uint lastBlock + ) + { + byte* ostart = (byte*)dst; + byte* oend = ostart + dstCapacity; + byte* op = ostart + ZSTD_blockHeaderSize; + { + nuint cLitSize = ZSTD_compressSubBlock_literal( + &entropy->huf.CTable.e0, + &entropyMetadata->hufMetadata, + literals, + litSize, + op, + (nuint)(oend - op), + bmi2, + writeLitEntropy, + litEntropyWritten + ); + { + nuint err_code = cLitSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (cLitSize == 0) + { + return 0; + } + + op += cLitSize; + } + + { + nuint cSeqSize = ZSTD_compressSubBlock_sequences( + &entropy->fse, + &entropyMetadata->fseMetadata, + sequences, + nbSeq, + llCode, + mlCode, + ofCode, + cctxParams, + op, + (nuint)(oend - op), + bmi2, + writeSeqEntropy, + seqEntropyWritten + ); + { + nuint err_code = cSeqSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (cSeqSize == 0) + { + return 0; + } + + op += cSeqSize; + } + + { + nuint cSize = (nuint)(op - ostart) - ZSTD_blockHeaderSize; + uint cBlockHeader24 = + lastBlock + ((uint)blockType_e.bt_compressed << 1) + (uint)(cSize << 3); + MEM_writeLE24(ostart, cBlockHeader24); + } + + return (nuint)(op - ostart); + } + + private static nuint ZSTD_estimateSubBlockSize_literal( + byte* literals, + nuint litSize, + ZSTD_hufCTables_t* huf, + ZSTD_hufCTablesMetadata_t* hufMetadata, + void* workspace, + nuint wkspSize, + int writeEntropy + ) + { + uint* countWksp = (uint*)workspace; + uint maxSymbolValue = 255; + /* Use hard coded size of 3 bytes */ + nuint literalSectionHeaderSize = 3; + if (hufMetadata->hType == SymbolEncodingType_e.set_basic) + { + return litSize; + } + else if (hufMetadata->hType == SymbolEncodingType_e.set_rle) + { + return 1; + } + else if ( + hufMetadata->hType == SymbolEncodingType_e.set_compressed + || hufMetadata->hType == SymbolEncodingType_e.set_repeat + ) + { + nuint largest = HIST_count_wksp( + countWksp, + &maxSymbolValue, + literals, + litSize, + workspace, + wkspSize + ); + if (ERR_isError(largest)) + { + return litSize; + } + + { + nuint cLitSizeEstimate = HUF_estimateCompressedSize( + &huf->CTable.e0, + countWksp, + maxSymbolValue + ); + if (writeEntropy != 0) + { + cLitSizeEstimate += hufMetadata->hufDesSize; + } + + return cLitSizeEstimate + literalSectionHeaderSize; + } + } + + assert(0 != 0); + return 0; + } + + private static nuint ZSTD_estimateSubBlockSize_symbolType( + SymbolEncodingType_e type, + byte* codeTable, + uint maxCode, + nuint nbSeq, + uint* fseCTable, + byte* additionalBits, + short* defaultNorm, + uint defaultNormLog, + uint defaultMax, + void* workspace, + nuint wkspSize + ) + { + uint* countWksp = (uint*)workspace; + byte* ctp = codeTable; + byte* ctStart = ctp; + byte* ctEnd = ctStart + nbSeq; + nuint cSymbolTypeSizeEstimateInBits = 0; + uint max = maxCode; + HIST_countFast_wksp(countWksp, &max, codeTable, nbSeq, workspace, wkspSize); + if (type == SymbolEncodingType_e.set_basic) + { + assert(max <= defaultMax); + cSymbolTypeSizeEstimateInBits = + max <= defaultMax + ? ZSTD_crossEntropyCost(defaultNorm, defaultNormLog, countWksp, max) + : unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + else if (type == SymbolEncodingType_e.set_rle) + { + cSymbolTypeSizeEstimateInBits = 0; + } + else if ( + type == SymbolEncodingType_e.set_compressed + || type == SymbolEncodingType_e.set_repeat + ) + { + cSymbolTypeSizeEstimateInBits = ZSTD_fseBitCost(fseCTable, countWksp, max); + } + + if (ERR_isError(cSymbolTypeSizeEstimateInBits)) + { + return nbSeq * 10; + } + + while (ctp < ctEnd) + { + if (additionalBits != null) + { + cSymbolTypeSizeEstimateInBits += additionalBits[*ctp]; + } + else + { + cSymbolTypeSizeEstimateInBits += *ctp; + } + + ctp++; + } + + return cSymbolTypeSizeEstimateInBits / 8; + } + + private static nuint ZSTD_estimateSubBlockSize_sequences( + byte* ofCodeTable, + byte* llCodeTable, + byte* mlCodeTable, + nuint nbSeq, + ZSTD_fseCTables_t* fseTables, + ZSTD_fseCTablesMetadata_t* fseMetadata, + void* workspace, + nuint wkspSize, + int writeEntropy + ) + { + /* Use hard coded size of 3 bytes */ + const nuint sequencesSectionHeaderSize = 3; + nuint cSeqSizeEstimate = 0; + if (nbSeq == 0) + { + return sequencesSectionHeaderSize; + } + + cSeqSizeEstimate += ZSTD_estimateSubBlockSize_symbolType( + fseMetadata->ofType, + ofCodeTable, + 31, + nbSeq, + fseTables->offcodeCTable, + null, + OF_defaultNorm, + OF_defaultNormLog, + 28, + workspace, + wkspSize + ); + cSeqSizeEstimate += ZSTD_estimateSubBlockSize_symbolType( + fseMetadata->llType, + llCodeTable, + 35, + nbSeq, + fseTables->litlengthCTable, + LL_bits, + LL_defaultNorm, + LL_defaultNormLog, + 35, + workspace, + wkspSize + ); + cSeqSizeEstimate += ZSTD_estimateSubBlockSize_symbolType( + fseMetadata->mlType, + mlCodeTable, + 52, + nbSeq, + fseTables->matchlengthCTable, + ML_bits, + ML_defaultNorm, + ML_defaultNormLog, + 52, + workspace, + wkspSize + ); + if (writeEntropy != 0) + { + cSeqSizeEstimate += fseMetadata->fseTablesSize; + } + + return cSeqSizeEstimate + sequencesSectionHeaderSize; + } + + private static EstimatedBlockSize ZSTD_estimateSubBlockSize( + byte* literals, + nuint litSize, + byte* ofCodeTable, + byte* llCodeTable, + byte* mlCodeTable, + nuint nbSeq, + ZSTD_entropyCTables_t* entropy, + ZSTD_entropyCTablesMetadata_t* entropyMetadata, + void* workspace, + nuint wkspSize, + int writeLitEntropy, + int writeSeqEntropy + ) + { + EstimatedBlockSize ebs; + ebs.estLitSize = ZSTD_estimateSubBlockSize_literal( + literals, + litSize, + &entropy->huf, + &entropyMetadata->hufMetadata, + workspace, + wkspSize, + writeLitEntropy + ); + ebs.estBlockSize = ZSTD_estimateSubBlockSize_sequences( + ofCodeTable, + llCodeTable, + mlCodeTable, + nbSeq, + &entropy->fse, + &entropyMetadata->fseMetadata, + workspace, + wkspSize, + writeSeqEntropy + ); + ebs.estBlockSize += ebs.estLitSize + ZSTD_blockHeaderSize; + return ebs; + } + + private static int ZSTD_needSequenceEntropyTables(ZSTD_fseCTablesMetadata_t* fseMetadata) + { + if ( + fseMetadata->llType == SymbolEncodingType_e.set_compressed + || fseMetadata->llType == SymbolEncodingType_e.set_rle + ) + { + return 1; + } + + if ( + fseMetadata->mlType == SymbolEncodingType_e.set_compressed + || fseMetadata->mlType == SymbolEncodingType_e.set_rle + ) + { + return 1; + } + + if ( + fseMetadata->ofType == SymbolEncodingType_e.set_compressed + || fseMetadata->ofType == SymbolEncodingType_e.set_rle + ) + { + return 1; + } + + return 0; + } + + private static nuint countLiterals(SeqStore_t* seqStore, SeqDef_s* sp, nuint seqCount) + { + nuint n, + total = 0; + assert(sp != null); + for (n = 0; n < seqCount; n++) + { + total += ZSTD_getSequenceLength(seqStore, sp + n).litLength; + } + + return total; + } + + private static nuint sizeBlockSequences( + SeqDef_s* sp, + nuint nbSeqs, + nuint targetBudget, + nuint avgLitCost, + nuint avgSeqCost, + int firstSubBlock + ) + { + nuint n, + budget = 0, + inSize = 0; + /* generous estimate */ + nuint headerSize = (nuint)firstSubBlock * 120 * 256; + assert(firstSubBlock == 0 || firstSubBlock == 1); + budget += headerSize; + budget += sp[0].litLength * avgLitCost + avgSeqCost; + if (budget > targetBudget) + { + return 1; + } + + inSize = (nuint)(sp[0].litLength + (sp[0].mlBase + 3)); + for (n = 1; n < nbSeqs; n++) + { + nuint currentCost = sp[n].litLength * avgLitCost + avgSeqCost; + budget += currentCost; + inSize += (nuint)(sp[n].litLength + (sp[n].mlBase + 3)); + if (budget > targetBudget && budget < inSize * 256) + { + break; + } + } + + return n; + } + + /** ZSTD_compressSubBlock_multi() : + * Breaks super-block into multiple sub-blocks and compresses them. + * Entropy will be written into the first block. + * The following blocks use repeat_mode to compress. + * Sub-blocks are all compressed, except the last one when beneficial. + * @return : compressed size of the super block (which features multiple ZSTD blocks) + * or 0 if it failed to compress. */ + private static nuint ZSTD_compressSubBlock_multi( + SeqStore_t* seqStorePtr, + ZSTD_compressedBlockState_t* prevCBlock, + ZSTD_compressedBlockState_t* nextCBlock, + ZSTD_entropyCTablesMetadata_t* entropyMetadata, + ZSTD_CCtx_params_s* cctxParams, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize, + int bmi2, + uint lastBlock, + void* workspace, + nuint wkspSize + ) + { + SeqDef_s* sstart = seqStorePtr->sequencesStart; + SeqDef_s* send = seqStorePtr->sequences; + /* tracks progresses within seqStorePtr->sequences */ + SeqDef_s* sp = sstart; + nuint nbSeqs = (nuint)(send - sstart); + byte* lstart = seqStorePtr->litStart; + byte* lend = seqStorePtr->lit; + byte* lp = lstart; + nuint nbLiterals = (nuint)(lend - lstart); + byte* ip = (byte*)src; + byte* iend = ip + srcSize; + byte* ostart = (byte*)dst; + byte* oend = ostart + dstCapacity; + byte* op = ostart; + byte* llCodePtr = seqStorePtr->llCode; + byte* mlCodePtr = seqStorePtr->mlCode; + byte* ofCodePtr = seqStorePtr->ofCode; + /* enforce minimum size, to reduce undesirable side effects */ + const nuint minTarget = 1340; + nuint targetCBlockSize = + minTarget > cctxParams->targetCBlockSize ? minTarget : cctxParams->targetCBlockSize; + int writeLitEntropy = + entropyMetadata->hufMetadata.hType == SymbolEncodingType_e.set_compressed ? 1 : 0; + int writeSeqEntropy = 1; + if (nbSeqs > 0) + { + EstimatedBlockSize ebs = ZSTD_estimateSubBlockSize( + lp, + nbLiterals, + ofCodePtr, + llCodePtr, + mlCodePtr, + nbSeqs, + &nextCBlock->entropy, + entropyMetadata, + workspace, + wkspSize, + writeLitEntropy, + writeSeqEntropy + ); + /* quick estimation */ + nuint avgLitCost = nbLiterals != 0 ? ebs.estLitSize * 256 / nbLiterals : 256; + nuint avgSeqCost = (ebs.estBlockSize - ebs.estLitSize) * 256 / nbSeqs; + nuint nbSubBlocks = + (ebs.estBlockSize + targetCBlockSize / 2) / targetCBlockSize > 1 + ? (ebs.estBlockSize + targetCBlockSize / 2) / targetCBlockSize + : 1; + nuint n, + avgBlockBudget, + blockBudgetSupp = 0; + avgBlockBudget = ebs.estBlockSize * 256 / nbSubBlocks; + if (ebs.estBlockSize > srcSize) + { + return 0; + } + + assert(nbSubBlocks > 0); + for (n = 0; n < nbSubBlocks - 1; n++) + { + /* determine nb of sequences for current sub-block + nbLiterals from next sequence */ + nuint seqCount = sizeBlockSequences( + sp, + (nuint)(send - sp), + avgBlockBudget + blockBudgetSupp, + avgLitCost, + avgSeqCost, + n == 0 ? 1 : 0 + ); + assert(seqCount <= (nuint)(send - sp)); + if (sp + seqCount == send) + { + break; + } + + assert(seqCount > 0); + { + int litEntropyWritten = 0; + int seqEntropyWritten = 0; + nuint litSize = countLiterals(seqStorePtr, sp, seqCount); + nuint decompressedSize = ZSTD_seqDecompressedSize( + seqStorePtr, + sp, + seqCount, + litSize, + 0 + ); + nuint cSize = ZSTD_compressSubBlock( + &nextCBlock->entropy, + entropyMetadata, + sp, + seqCount, + lp, + litSize, + llCodePtr, + mlCodePtr, + ofCodePtr, + cctxParams, + op, + (nuint)(oend - op), + bmi2, + writeLitEntropy, + writeSeqEntropy, + &litEntropyWritten, + &seqEntropyWritten, + 0 + ); + { + nuint err_code = cSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (cSize > 0 && cSize < decompressedSize) + { + assert(ip + decompressedSize <= iend); + ip += decompressedSize; + lp += litSize; + op += cSize; + llCodePtr += seqCount; + mlCodePtr += seqCount; + ofCodePtr += seqCount; + if (litEntropyWritten != 0) + { + writeLitEntropy = 0; + } + + if (seqEntropyWritten != 0) + { + writeSeqEntropy = 0; + } + + sp += seqCount; + blockBudgetSupp = 0; + } + } + } + } + + { + int litEntropyWritten = 0; + int seqEntropyWritten = 0; + nuint litSize = (nuint)(lend - lp); + nuint seqCount = (nuint)(send - sp); + nuint decompressedSize = ZSTD_seqDecompressedSize( + seqStorePtr, + sp, + seqCount, + litSize, + 1 + ); + nuint cSize = ZSTD_compressSubBlock( + &nextCBlock->entropy, + entropyMetadata, + sp, + seqCount, + lp, + litSize, + llCodePtr, + mlCodePtr, + ofCodePtr, + cctxParams, + op, + (nuint)(oend - op), + bmi2, + writeLitEntropy, + writeSeqEntropy, + &litEntropyWritten, + &seqEntropyWritten, + lastBlock + ); + { + nuint err_code = cSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (cSize > 0 && cSize < decompressedSize) + { + assert(ip + decompressedSize <= iend); + ip += decompressedSize; + lp += litSize; + op += cSize; + llCodePtr += seqCount; + mlCodePtr += seqCount; + ofCodePtr += seqCount; + if (litEntropyWritten != 0) + { + writeLitEntropy = 0; + } + + if (seqEntropyWritten != 0) + { + writeSeqEntropy = 0; + } + + sp += seqCount; + } + } + + if (writeLitEntropy != 0) + { + memcpy( + &nextCBlock->entropy.huf, + &prevCBlock->entropy.huf, + (uint)sizeof(ZSTD_hufCTables_t) + ); + } + + if ( + writeSeqEntropy != 0 + && ZSTD_needSequenceEntropyTables(&entropyMetadata->fseMetadata) != 0 + ) + { + return 0; + } + + if (ip < iend) + { + /* some data left : last part of the block sent uncompressed */ + nuint rSize = (nuint)(iend - ip); + nuint cSize = ZSTD_noCompressBlock(op, (nuint)(oend - op), ip, rSize, lastBlock); + { + nuint err_code = cSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + assert(cSize != 0); + op += cSize; + if (sp < send) + { + SeqDef_s* seq; + repcodes_s rep; + memcpy(&rep, prevCBlock->rep, (uint)sizeof(repcodes_s)); + for (seq = sstart; seq < sp; ++seq) + { + ZSTD_updateRep( + rep.rep, + seq->offBase, + ZSTD_getSequenceLength(seqStorePtr, seq).litLength == 0 ? 1U : 0U + ); + } + + memcpy(nextCBlock->rep, &rep, (uint)sizeof(repcodes_s)); + } + } + + return (nuint)(op - ostart); + } + + /* ZSTD_compressSuperBlock() : + * Used to compress a super block when targetCBlockSize is being used. + * The given block will be compressed into multiple sub blocks that are around targetCBlockSize. */ + private static nuint ZSTD_compressSuperBlock( + ZSTD_CCtx_s* zc, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize, + uint lastBlock + ) + { + ZSTD_entropyCTablesMetadata_t entropyMetadata; + { + nuint err_code = ZSTD_buildBlockEntropyStats( + &zc->seqStore, + &zc->blockState.prevCBlock->entropy, + &zc->blockState.nextCBlock->entropy, + &zc->appliedParams, + &entropyMetadata, + zc->tmpWorkspace, + zc->tmpWkspSize + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return ZSTD_compressSubBlock_multi( + &zc->seqStore, + zc->blockState.prevCBlock, + zc->blockState.nextCBlock, + &entropyMetadata, + &zc->appliedParams, + dst, + dstCapacity, + src, + srcSize, + zc->bmi2, + lastBlock, + zc->tmpWorkspace, + zc->tmpWkspSize + ); + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdCwksp.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdCwksp.cs new file mode 100644 index 00000000..455ab861 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdCwksp.cs @@ -0,0 +1,588 @@ +using System.Runtime.CompilerServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_cwksp_assert_internal_consistency(ZSTD_cwksp* ws) + { + assert(ws->workspace <= ws->objectEnd); + assert(ws->objectEnd <= ws->tableEnd); + assert(ws->objectEnd <= ws->tableValidEnd); + assert(ws->tableEnd <= ws->allocStart); + assert(ws->tableValidEnd <= ws->allocStart); + assert(ws->allocStart <= ws->workspaceEnd); + assert(ws->initOnceStart <= ZSTD_cwksp_initialAllocStart(ws)); + assert(ws->workspace <= ws->initOnceStart); + } + + /** + * Align must be a power of 2. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_cwksp_align(nuint size, nuint align) + { + nuint mask = align - 1; + assert(ZSTD_isPower2(align) != 0); + return size + mask & ~mask; + } + + /** + * Use this to determine how much space in the workspace we will consume to + * allocate this object. (Normally it should be exactly the size of the object, + * but under special conditions, like ASAN, where we pad each object, it might + * be larger.) + * + * Since tables aren't currently redzoned, you don't need to call through this + * to figure out how much space you need for the matchState tables. Everything + * else is though. + * + * Do not use for sizing aligned buffers. Instead, use ZSTD_cwksp_aligned64_alloc_size(). + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_cwksp_alloc_size(nuint size) + { + if (size == 0) + { + return 0; + } + + return size; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_cwksp_aligned_alloc_size(nuint size, nuint alignment) + { + return ZSTD_cwksp_alloc_size(ZSTD_cwksp_align(size, alignment)); + } + + /** + * Returns an adjusted alloc size that is the nearest larger multiple of 64 bytes. + * Used to determine the number of bytes required for a given "aligned". + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_cwksp_aligned64_alloc_size(nuint size) + { + return ZSTD_cwksp_aligned_alloc_size(size, 64); + } + + /** + * Returns the amount of additional space the cwksp must allocate + * for internal purposes (currently only alignment). + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_cwksp_slack_space_required() + { + /* For alignment, the wksp will always allocate an additional 2*ZSTD_CWKSP_ALIGNMENT_BYTES + * bytes to align the beginning of tables section and end of buffers; + */ + const nuint slackSpace = 64 * 2; + return slackSpace; + } + + /** + * Return the number of additional bytes required to align a pointer to the given number of bytes. + * alignBytes must be a power of two. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_cwksp_bytes_to_align_ptr(void* ptr, nuint alignBytes) + { + nuint alignBytesMask = alignBytes - 1; + nuint bytes = alignBytes - ((nuint)ptr & alignBytesMask) & alignBytesMask; + assert(ZSTD_isPower2(alignBytes) != 0); + assert(bytes < alignBytes); + return bytes; + } + + /** + * Returns the initial value for allocStart which is used to determine the position from + * which we can allocate from the end of the workspace. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void* ZSTD_cwksp_initialAllocStart(ZSTD_cwksp* ws) + { + sbyte* endPtr = (sbyte*)ws->workspaceEnd; + assert(ZSTD_isPower2(64) != 0); + endPtr = endPtr - (nuint)endPtr % 64; + return endPtr; + } + + /** + * Internal function. Do not use directly. + * Reserves the given number of bytes within the aligned/buffer segment of the wksp, + * which counts from the end of the wksp (as opposed to the object/table segment). + * + * Returns a pointer to the beginning of that space. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void* ZSTD_cwksp_reserve_internal_buffer_space(ZSTD_cwksp* ws, nuint bytes) + { + void* alloc = (byte*)ws->allocStart - bytes; + void* bottom = ws->tableEnd; + ZSTD_cwksp_assert_internal_consistency(ws); + assert(alloc >= bottom); + if (alloc < bottom) + { + ws->allocFailed = 1; + return null; + } + + if (alloc < ws->tableValidEnd) + { + ws->tableValidEnd = alloc; + } + + ws->allocStart = alloc; + return alloc; + } + + /** + * Moves the cwksp to the next phase, and does any necessary allocations. + * cwksp initialization must necessarily go through each phase in order. + * Returns a 0 on success, or zstd error + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_cwksp_internal_advance_phase( + ZSTD_cwksp* ws, + ZSTD_cwksp_alloc_phase_e phase + ) + { + assert(phase >= ws->phase); + if (phase > ws->phase) + { + if ( + ws->phase < ZSTD_cwksp_alloc_phase_e.ZSTD_cwksp_alloc_aligned_init_once + && phase >= ZSTD_cwksp_alloc_phase_e.ZSTD_cwksp_alloc_aligned_init_once + ) + { + ws->tableValidEnd = ws->objectEnd; + ws->initOnceStart = ZSTD_cwksp_initialAllocStart(ws); + { + void* alloc = ws->objectEnd; + nuint bytesToAlign = ZSTD_cwksp_bytes_to_align_ptr(alloc, 64); + void* objectEnd = (byte*)alloc + bytesToAlign; + if (objectEnd > ws->workspaceEnd) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation) + ); + } + + ws->objectEnd = objectEnd; + ws->tableEnd = objectEnd; + if (ws->tableValidEnd < ws->tableEnd) + { + ws->tableValidEnd = ws->tableEnd; + } + } + } + + ws->phase = phase; + ZSTD_cwksp_assert_internal_consistency(ws); + } + + return 0; + } + + /** + * Returns whether this object/buffer/etc was allocated in this workspace. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static int ZSTD_cwksp_owns_buffer(ZSTD_cwksp* ws, void* ptr) + { + return ptr != null && ws->workspace <= ptr && ptr < ws->workspaceEnd ? 1 : 0; + } + + /** + * Internal function. Do not use directly. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void* ZSTD_cwksp_reserve_internal( + ZSTD_cwksp* ws, + nuint bytes, + ZSTD_cwksp_alloc_phase_e phase + ) + { + void* alloc; + if (ERR_isError(ZSTD_cwksp_internal_advance_phase(ws, phase)) || bytes == 0) + { + return null; + } + + alloc = ZSTD_cwksp_reserve_internal_buffer_space(ws, bytes); + return alloc; + } + + /** + * Reserves and returns unaligned memory. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static byte* ZSTD_cwksp_reserve_buffer(ZSTD_cwksp* ws, nuint bytes) + { + return (byte*)ZSTD_cwksp_reserve_internal( + ws, + bytes, + ZSTD_cwksp_alloc_phase_e.ZSTD_cwksp_alloc_buffers + ); + } + + /** + * Reserves and returns memory sized on and aligned on ZSTD_CWKSP_ALIGNMENT_BYTES (64 bytes). + * This memory has been initialized at least once in the past. + * This doesn't mean it has been initialized this time, and it might contain data from previous + * operations. + * The main usage is for algorithms that might need read access into uninitialized memory. + * The algorithm must maintain safety under these conditions and must make sure it doesn't + * leak any of the past data (directly or in side channels). + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void* ZSTD_cwksp_reserve_aligned_init_once(ZSTD_cwksp* ws, nuint bytes) + { + nuint alignedBytes = ZSTD_cwksp_align(bytes, 64); + void* ptr = ZSTD_cwksp_reserve_internal( + ws, + alignedBytes, + ZSTD_cwksp_alloc_phase_e.ZSTD_cwksp_alloc_aligned_init_once + ); + assert(((nuint)ptr & 64 - 1) == 0); + if (ptr != null && ptr < ws->initOnceStart) + { + memset( + ptr, + 0, + (uint)( + (nuint)((byte*)ws->initOnceStart - (byte*)ptr) < alignedBytes + ? (nuint)((byte*)ws->initOnceStart - (byte*)ptr) + : alignedBytes + ) + ); + ws->initOnceStart = ptr; + } + + return ptr; + } + + /** + * Reserves and returns memory sized on and aligned on ZSTD_CWKSP_ALIGNMENT_BYTES (64 bytes). + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void* ZSTD_cwksp_reserve_aligned64(ZSTD_cwksp* ws, nuint bytes) + { + void* ptr = ZSTD_cwksp_reserve_internal( + ws, + ZSTD_cwksp_align(bytes, 64), + ZSTD_cwksp_alloc_phase_e.ZSTD_cwksp_alloc_aligned + ); + assert(((nuint)ptr & 64 - 1) == 0); + return ptr; + } + + /** + * Aligned on 64 bytes. These buffers have the special property that + * their values remain constrained, allowing us to reuse them without + * memset()-ing them. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void* ZSTD_cwksp_reserve_table(ZSTD_cwksp* ws, nuint bytes) + { + ZSTD_cwksp_alloc_phase_e phase = + ZSTD_cwksp_alloc_phase_e.ZSTD_cwksp_alloc_aligned_init_once; + void* alloc; + void* end; + void* top; + if (ws->phase < phase) + { + if (ERR_isError(ZSTD_cwksp_internal_advance_phase(ws, phase))) + { + return null; + } + } + + alloc = ws->tableEnd; + end = (byte*)alloc + bytes; + top = ws->allocStart; + assert((bytes & sizeof(uint) - 1) == 0); + ZSTD_cwksp_assert_internal_consistency(ws); + assert(end <= top); + if (end > top) + { + ws->allocFailed = 1; + return null; + } + + ws->tableEnd = end; + assert((bytes & 64 - 1) == 0); + assert(((nuint)alloc & 64 - 1) == 0); + return alloc; + } + + /** + * Aligned on sizeof(void*). + * Note : should happen only once, at workspace first initialization + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void* ZSTD_cwksp_reserve_object(ZSTD_cwksp* ws, nuint bytes) + { + nuint roundedBytes = ZSTD_cwksp_align(bytes, (nuint)sizeof(void*)); + void* alloc = ws->objectEnd; + void* end = (byte*)alloc + roundedBytes; + assert((nuint)alloc % (nuint)sizeof(void*) == 0); + assert(bytes % (nuint)sizeof(void*) == 0); + ZSTD_cwksp_assert_internal_consistency(ws); + if ( + ws->phase != ZSTD_cwksp_alloc_phase_e.ZSTD_cwksp_alloc_objects + || end > ws->workspaceEnd + ) + { + ws->allocFailed = 1; + return null; + } + + ws->objectEnd = end; + ws->tableEnd = end; + ws->tableValidEnd = end; + return alloc; + } + + /** + * with alignment control + * Note : should happen only once, at workspace first initialization + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void* ZSTD_cwksp_reserve_object_aligned( + ZSTD_cwksp* ws, + nuint byteSize, + nuint alignment + ) + { + nuint mask = alignment - 1; + nuint surplus = alignment > (nuint)sizeof(void*) ? alignment - (nuint)sizeof(void*) : 0; + void* start = ZSTD_cwksp_reserve_object(ws, byteSize + surplus); + if (start == null) + { + return null; + } + + if (surplus == 0) + { + return start; + } + + assert(ZSTD_isPower2(alignment) != 0); + return (void*)((nuint)start + surplus & ~mask); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_cwksp_mark_tables_dirty(ZSTD_cwksp* ws) + { + assert(ws->tableValidEnd >= ws->objectEnd); + assert(ws->tableValidEnd <= ws->allocStart); + ws->tableValidEnd = ws->objectEnd; + ZSTD_cwksp_assert_internal_consistency(ws); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_cwksp_mark_tables_clean(ZSTD_cwksp* ws) + { + assert(ws->tableValidEnd >= ws->objectEnd); + assert(ws->tableValidEnd <= ws->allocStart); + if (ws->tableValidEnd < ws->tableEnd) + { + ws->tableValidEnd = ws->tableEnd; + } + + ZSTD_cwksp_assert_internal_consistency(ws); + } + + /** + * Zero the part of the allocated tables not already marked clean. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_cwksp_clean_tables(ZSTD_cwksp* ws) + { + assert(ws->tableValidEnd >= ws->objectEnd); + assert(ws->tableValidEnd <= ws->allocStart); + if (ws->tableValidEnd < ws->tableEnd) + { + memset( + ws->tableValidEnd, + 0, + (uint)(nuint)((byte*)ws->tableEnd - (byte*)ws->tableValidEnd) + ); + } + + ZSTD_cwksp_mark_tables_clean(ws); + } + + /** + * Invalidates table allocations. + * All other allocations remain valid. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_cwksp_clear_tables(ZSTD_cwksp* ws) + { + ws->tableEnd = ws->objectEnd; + ZSTD_cwksp_assert_internal_consistency(ws); + } + + /** + * Invalidates all buffer, aligned, and table allocations. + * Object allocations remain valid. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_cwksp_clear(ZSTD_cwksp* ws) + { + ws->tableEnd = ws->objectEnd; + ws->allocStart = ZSTD_cwksp_initialAllocStart(ws); + ws->allocFailed = 0; + if (ws->phase > ZSTD_cwksp_alloc_phase_e.ZSTD_cwksp_alloc_aligned_init_once) + { + ws->phase = ZSTD_cwksp_alloc_phase_e.ZSTD_cwksp_alloc_aligned_init_once; + } + + ZSTD_cwksp_assert_internal_consistency(ws); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_cwksp_sizeof(ZSTD_cwksp* ws) + { + return (nuint)((byte*)ws->workspaceEnd - (byte*)ws->workspace); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_cwksp_used(ZSTD_cwksp* ws) + { + return (nuint)((byte*)ws->tableEnd - (byte*)ws->workspace) + + (nuint)((byte*)ws->workspaceEnd - (byte*)ws->allocStart); + } + + /** + * The provided workspace takes ownership of the buffer [start, start+size). + * Any existing values in the workspace are ignored (the previously managed + * buffer, if present, must be separately freed). + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_cwksp_init( + ZSTD_cwksp* ws, + void* start, + nuint size, + ZSTD_cwksp_static_alloc_e isStatic + ) + { + assert(((nuint)start & (nuint)(sizeof(void*) - 1)) == 0); + ws->workspace = start; + ws->workspaceEnd = (byte*)start + size; + ws->objectEnd = ws->workspace; + ws->tableValidEnd = ws->objectEnd; + ws->initOnceStart = ZSTD_cwksp_initialAllocStart(ws); + ws->phase = ZSTD_cwksp_alloc_phase_e.ZSTD_cwksp_alloc_objects; + ws->isStatic = isStatic; + ZSTD_cwksp_clear(ws); + ws->workspaceOversizedDuration = 0; + ZSTD_cwksp_assert_internal_consistency(ws); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_cwksp_create(ZSTD_cwksp* ws, nuint size, ZSTD_customMem customMem) + { + void* workspace = ZSTD_customMalloc(size, customMem); + if (workspace == null) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + + ZSTD_cwksp_init(ws, workspace, size, ZSTD_cwksp_static_alloc_e.ZSTD_cwksp_dynamic_alloc); + return 0; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_cwksp_free(ZSTD_cwksp* ws, ZSTD_customMem customMem) + { + void* ptr = ws->workspace; + *ws = new ZSTD_cwksp(); + ZSTD_customFree(ptr, customMem); + } + + /** + * Moves the management of a workspace from one cwksp to another. The src cwksp + * is left in an invalid state (src must be re-init()'ed before it's used again). + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_cwksp_move(ZSTD_cwksp* dst, ZSTD_cwksp* src) + { + *dst = *src; + *src = new ZSTD_cwksp(); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static int ZSTD_cwksp_reserve_failed(ZSTD_cwksp* ws) + { + return ws->allocFailed; + } + + /* ZSTD_alignmentSpaceWithinBounds() : + * Returns if the estimated space needed for a wksp is within an acceptable limit of the + * actual amount of space used. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static int ZSTD_cwksp_estimated_space_within_bounds( + ZSTD_cwksp* ws, + nuint estimatedSpace + ) + { + return + estimatedSpace - ZSTD_cwksp_slack_space_required() <= ZSTD_cwksp_used(ws) + && ZSTD_cwksp_used(ws) <= estimatedSpace + ? 1 + : 0; + } + + /*-************************************* + * Functions + ***************************************/ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_cwksp_available_space(ZSTD_cwksp* ws) + { + return (nuint)((byte*)ws->allocStart - (byte*)ws->tableEnd); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static int ZSTD_cwksp_check_available(ZSTD_cwksp* ws, nuint additionalNeededSpace) + { + return ZSTD_cwksp_available_space(ws) >= additionalNeededSpace ? 1 : 0; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static int ZSTD_cwksp_check_too_large(ZSTD_cwksp* ws, nuint additionalNeededSpace) + { + return ZSTD_cwksp_check_available(ws, additionalNeededSpace * 3); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static int ZSTD_cwksp_check_wasteful(ZSTD_cwksp* ws, nuint additionalNeededSpace) + { + return + ZSTD_cwksp_check_too_large(ws, additionalNeededSpace) != 0 + && ws->workspaceOversizedDuration > 128 + ? 1 + : 0; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_cwksp_bump_oversized_duration( + ZSTD_cwksp* ws, + nuint additionalNeededSpace + ) + { + if (ZSTD_cwksp_check_too_large(ws, additionalNeededSpace) != 0) + { + ws->workspaceOversizedDuration++; + } + else + { + ws->workspaceOversizedDuration = 0; + } + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdDdict.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdDdict.cs new file mode 100644 index 00000000..5166e40c --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdDdict.cs @@ -0,0 +1,330 @@ +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + /* note: several prototypes are already published in `zstd.h` : + * ZSTD_createDDict() + * ZSTD_createDDict_byReference() + * ZSTD_createDDict_advanced() + * ZSTD_freeDDict() + * ZSTD_initStaticDDict() + * ZSTD_sizeof_DDict() + * ZSTD_estimateDDictSize() + * ZSTD_getDictID_fromDict() + */ + private static void* ZSTD_DDict_dictContent(ZSTD_DDict_s* ddict) + { + assert(ddict != null); + return ddict->dictContent; + } + + private static nuint ZSTD_DDict_dictSize(ZSTD_DDict_s* ddict) + { + assert(ddict != null); + return ddict->dictSize; + } + + private static void ZSTD_copyDDictParameters(ZSTD_DCtx_s* dctx, ZSTD_DDict_s* ddict) + { + assert(dctx != null); + assert(ddict != null); + dctx->dictID = ddict->dictID; + dctx->prefixStart = ddict->dictContent; + dctx->virtualStart = ddict->dictContent; + dctx->dictEnd = (byte*)ddict->dictContent + ddict->dictSize; + dctx->previousDstEnd = dctx->dictEnd; + if (ddict->entropyPresent != 0) + { + dctx->litEntropy = 1; + dctx->fseEntropy = 1; + dctx->LLTptr = &ddict->entropy.LLTable.e0; + dctx->MLTptr = &ddict->entropy.MLTable.e0; + dctx->OFTptr = &ddict->entropy.OFTable.e0; + dctx->HUFptr = ddict->entropy.hufTable; + dctx->entropy.rep[0] = ddict->entropy.rep[0]; + dctx->entropy.rep[1] = ddict->entropy.rep[1]; + dctx->entropy.rep[2] = ddict->entropy.rep[2]; + } + else + { + dctx->litEntropy = 0; + dctx->fseEntropy = 0; + } + } + + private static nuint ZSTD_loadEntropy_intoDDict( + ZSTD_DDict_s* ddict, + ZSTD_dictContentType_e dictContentType + ) + { + ddict->dictID = 0; + ddict->entropyPresent = 0; + if (dictContentType == ZSTD_dictContentType_e.ZSTD_dct_rawContent) + { + return 0; + } + + if (ddict->dictSize < 8) + { + if (dictContentType == ZSTD_dictContentType_e.ZSTD_dct_fullDict) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + return 0; + } + + { + uint magic = MEM_readLE32(ddict->dictContent); + if (magic != 0xEC30A437) + { + if (dictContentType == ZSTD_dictContentType_e.ZSTD_dct_fullDict) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + return 0; + } + } + + ddict->dictID = MEM_readLE32((sbyte*)ddict->dictContent + 4); + if (ERR_isError(ZSTD_loadDEntropy(&ddict->entropy, ddict->dictContent, ddict->dictSize))) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + ddict->entropyPresent = 1; + return 0; + } + + private static nuint ZSTD_initDDict_internal( + ZSTD_DDict_s* ddict, + void* dict, + nuint dictSize, + ZSTD_dictLoadMethod_e dictLoadMethod, + ZSTD_dictContentType_e dictContentType + ) + { + if (dictLoadMethod == ZSTD_dictLoadMethod_e.ZSTD_dlm_byRef || dict == null || dictSize == 0) + { + ddict->dictBuffer = null; + ddict->dictContent = dict; + if (dict == null) + { + dictSize = 0; + } + } + else + { + void* internalBuffer = ZSTD_customMalloc(dictSize, ddict->cMem); + ddict->dictBuffer = internalBuffer; + ddict->dictContent = internalBuffer; + if (internalBuffer == null) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + + memcpy(internalBuffer, dict, (uint)dictSize); + } + + ddict->dictSize = dictSize; + ddict->entropy.hufTable[0] = 12 * 0x1000001; + { + /* parse dictionary content */ + nuint err_code = ZSTD_loadEntropy_intoDDict(ddict, dictContentType); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return 0; + } + + public static ZSTD_DDict_s* ZSTD_createDDict_advanced( + void* dict, + nuint dictSize, + ZSTD_dictLoadMethod_e dictLoadMethod, + ZSTD_dictContentType_e dictContentType, + ZSTD_customMem customMem + ) + { + if (((customMem.customAlloc == null ? 1 : 0) ^ (customMem.customFree == null ? 1 : 0)) != 0) + { + return null; + } + + { + ZSTD_DDict_s* ddict = (ZSTD_DDict_s*)ZSTD_customMalloc( + (nuint)sizeof(ZSTD_DDict_s), + customMem + ); + if (ddict == null) + { + return null; + } + + ddict->cMem = customMem; + { + nuint initResult = ZSTD_initDDict_internal( + ddict, + dict, + dictSize, + dictLoadMethod, + dictContentType + ); + if (ERR_isError(initResult)) + { + ZSTD_freeDDict(ddict); + return null; + } + } + + return ddict; + } + } + + /*! ZSTD_createDDict() : + * Create a digested dictionary, to start decompression without startup delay. + * `dict` content is copied inside DDict. + * Consequently, `dict` can be released after `ZSTD_DDict` creation */ + public static ZSTD_DDict_s* ZSTD_createDDict(void* dict, nuint dictSize) + { + ZSTD_customMem allocator = new ZSTD_customMem + { + customAlloc = null, + customFree = null, + opaque = null, + }; + return ZSTD_createDDict_advanced( + dict, + dictSize, + ZSTD_dictLoadMethod_e.ZSTD_dlm_byCopy, + ZSTD_dictContentType_e.ZSTD_dct_auto, + allocator + ); + } + + /*! ZSTD_createDDict_byReference() : + * Create a digested dictionary, to start decompression without startup delay. + * Dictionary content is simply referenced, it will be accessed during decompression. + * Warning : dictBuffer must outlive DDict (DDict must be freed before dictBuffer) */ + public static ZSTD_DDict_s* ZSTD_createDDict_byReference(void* dictBuffer, nuint dictSize) + { + ZSTD_customMem allocator = new ZSTD_customMem + { + customAlloc = null, + customFree = null, + opaque = null, + }; + return ZSTD_createDDict_advanced( + dictBuffer, + dictSize, + ZSTD_dictLoadMethod_e.ZSTD_dlm_byRef, + ZSTD_dictContentType_e.ZSTD_dct_auto, + allocator + ); + } + + public static ZSTD_DDict_s* ZSTD_initStaticDDict( + void* sBuffer, + nuint sBufferSize, + void* dict, + nuint dictSize, + ZSTD_dictLoadMethod_e dictLoadMethod, + ZSTD_dictContentType_e dictContentType + ) + { + nuint neededSpace = + (nuint)sizeof(ZSTD_DDict_s) + + (dictLoadMethod == ZSTD_dictLoadMethod_e.ZSTD_dlm_byRef ? 0 : dictSize); + ZSTD_DDict_s* ddict = (ZSTD_DDict_s*)sBuffer; + assert(sBuffer != null); + assert(dict != null); + if (((nuint)sBuffer & 7) != 0) + { + return null; + } + + if (sBufferSize < neededSpace) + { + return null; + } + + if (dictLoadMethod == ZSTD_dictLoadMethod_e.ZSTD_dlm_byCopy) + { + memcpy(ddict + 1, dict, (uint)dictSize); + dict = ddict + 1; + } + + if ( + ERR_isError( + ZSTD_initDDict_internal( + ddict, + dict, + dictSize, + ZSTD_dictLoadMethod_e.ZSTD_dlm_byRef, + dictContentType + ) + ) + ) + { + return null; + } + + return ddict; + } + + /*! ZSTD_freeDDict() : + * Function frees memory allocated with ZSTD_createDDict() + * If a NULL pointer is passed, no operation is performed. */ + public static nuint ZSTD_freeDDict(ZSTD_DDict_s* ddict) + { + if (ddict == null) + { + return 0; + } + + { + ZSTD_customMem cMem = ddict->cMem; + ZSTD_customFree(ddict->dictBuffer, cMem); + ZSTD_customFree(ddict, cMem); + return 0; + } + } + + /*! ZSTD_estimateDDictSize() : + * Estimate amount of memory that will be needed to create a dictionary for decompression. + * Note : dictionary created by reference using ZSTD_dlm_byRef are smaller */ + public static nuint ZSTD_estimateDDictSize(nuint dictSize, ZSTD_dictLoadMethod_e dictLoadMethod) + { + return (nuint)sizeof(ZSTD_DDict_s) + + (dictLoadMethod == ZSTD_dictLoadMethod_e.ZSTD_dlm_byRef ? 0 : dictSize); + } + + public static nuint ZSTD_sizeof_DDict(ZSTD_DDict_s* ddict) + { + if (ddict == null) + { + return 0; + } + + return (nuint)sizeof(ZSTD_DDict_s) + (ddict->dictBuffer != null ? ddict->dictSize : 0); + } + + /*! ZSTD_getDictID_fromDDict() : + * Provides the dictID of the dictionary loaded into `ddict`. + * If @return == 0, the dictionary is not conformant to Zstandard specification, or empty. + * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */ + public static uint ZSTD_getDictID_fromDDict(ZSTD_DDict_s* ddict) + { + if (ddict == null) + { + return 0; + } + + return ddict->dictID; + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdDecompress.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdDecompress.cs new file mode 100644 index 00000000..ab954fb6 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdDecompress.cs @@ -0,0 +1,3602 @@ +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + /* Hash function to determine starting position of dict insertion within the table + * Returns an index between [0, hashSet->ddictPtrTableSize] + */ + private static nuint ZSTD_DDictHashSet_getIndex(ZSTD_DDictHashSet* hashSet, uint dictID) + { + ulong hash = ZSTD_XXH64(&dictID, sizeof(uint), 0); + return (nuint)(hash & hashSet->ddictPtrTableSize - 1); + } + + /* Adds DDict to a hashset without resizing it. + * If inserting a DDict with a dictID that already exists in the set, replaces the one in the set. + * Returns 0 if successful, or a zstd error code if something went wrong. + */ + private static nuint ZSTD_DDictHashSet_emplaceDDict( + ZSTD_DDictHashSet* hashSet, + ZSTD_DDict_s* ddict + ) + { + uint dictID = ZSTD_getDictID_fromDDict(ddict); + nuint idx = ZSTD_DDictHashSet_getIndex(hashSet, dictID); + nuint idxRangeMask = hashSet->ddictPtrTableSize - 1; + if (hashSet->ddictPtrCount == hashSet->ddictPtrTableSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + + while (hashSet->ddictPtrTable[idx] != null) + { + if (ZSTD_getDictID_fromDDict(hashSet->ddictPtrTable[idx]) == dictID) + { + hashSet->ddictPtrTable[idx] = ddict; + return 0; + } + + idx &= idxRangeMask; + idx++; + } + + hashSet->ddictPtrTable[idx] = ddict; + hashSet->ddictPtrCount++; + return 0; + } + + /* Expands hash table by factor of DDICT_HASHSET_RESIZE_FACTOR and + * rehashes all values, allocates new table, frees old table. + * Returns 0 on success, otherwise a zstd error code. + */ + private static nuint ZSTD_DDictHashSet_expand( + ZSTD_DDictHashSet* hashSet, + ZSTD_customMem customMem + ) + { + nuint newTableSize = hashSet->ddictPtrTableSize * 2; + ZSTD_DDict_s** newTable = (ZSTD_DDict_s**)ZSTD_customCalloc( + (nuint)sizeof(ZSTD_DDict_s*) * newTableSize, + customMem + ); + ZSTD_DDict_s** oldTable = hashSet->ddictPtrTable; + nuint oldTableSize = hashSet->ddictPtrTableSize; + nuint i; + if (newTable == null) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + + hashSet->ddictPtrTable = newTable; + hashSet->ddictPtrTableSize = newTableSize; + hashSet->ddictPtrCount = 0; + for (i = 0; i < oldTableSize; ++i) + { + if (oldTable[i] != null) + { + nuint err_code = ZSTD_DDictHashSet_emplaceDDict(hashSet, oldTable[i]); + if (ERR_isError(err_code)) + { + return err_code; + } + } + } + + ZSTD_customFree(oldTable, customMem); + return 0; + } + + /* Fetches a DDict with the given dictID + * Returns the ZSTD_DDict* with the requested dictID. If it doesn't exist, then returns NULL. + */ + private static ZSTD_DDict_s* ZSTD_DDictHashSet_getDDict(ZSTD_DDictHashSet* hashSet, uint dictID) + { + nuint idx = ZSTD_DDictHashSet_getIndex(hashSet, dictID); + nuint idxRangeMask = hashSet->ddictPtrTableSize - 1; + for (; ; ) + { + nuint currDictID = ZSTD_getDictID_fromDDict(hashSet->ddictPtrTable[idx]); + if (currDictID == dictID || currDictID == 0) + { + break; + } + else + { + idx &= idxRangeMask; + idx++; + } + } + + return hashSet->ddictPtrTable[idx]; + } + + /* Allocates space for and returns a ddict hash set + * The hash set's ZSTD_DDict* table has all values automatically set to NULL to begin with. + * Returns NULL if allocation failed. + */ + private static ZSTD_DDictHashSet* ZSTD_createDDictHashSet(ZSTD_customMem customMem) + { + ZSTD_DDictHashSet* ret = (ZSTD_DDictHashSet*)ZSTD_customMalloc( + (nuint)sizeof(ZSTD_DDictHashSet), + customMem + ); + if (ret == null) + { + return null; + } + + ret->ddictPtrTable = (ZSTD_DDict_s**)ZSTD_customCalloc( + (nuint)(64 * sizeof(ZSTD_DDict_s*)), + customMem + ); + if (ret->ddictPtrTable == null) + { + ZSTD_customFree(ret, customMem); + return null; + } + + ret->ddictPtrTableSize = 64; + ret->ddictPtrCount = 0; + return ret; + } + + /* Frees the table of ZSTD_DDict* within a hashset, then frees the hashset itself. + * Note: The ZSTD_DDict* within the table are NOT freed. + */ + private static void ZSTD_freeDDictHashSet(ZSTD_DDictHashSet* hashSet, ZSTD_customMem customMem) + { + if (hashSet != null && hashSet->ddictPtrTable != null) + { + ZSTD_customFree(hashSet->ddictPtrTable, customMem); + } + + if (hashSet != null) + { + ZSTD_customFree(hashSet, customMem); + } + } + + /* Public function: Adds a DDict into the ZSTD_DDictHashSet, possibly triggering a resize of the hash set. + * Returns 0 on success, or a ZSTD error. + */ + private static nuint ZSTD_DDictHashSet_addDDict( + ZSTD_DDictHashSet* hashSet, + ZSTD_DDict_s* ddict, + ZSTD_customMem customMem + ) + { + if (hashSet->ddictPtrCount * 4 / hashSet->ddictPtrTableSize * 3 != 0) + { + nuint err_code = ZSTD_DDictHashSet_expand(hashSet, customMem); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_DDictHashSet_emplaceDDict(hashSet, ddict); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return 0; + } + + /*-************************************************************* + * Context management + ***************************************************************/ + public static nuint ZSTD_sizeof_DCtx(ZSTD_DCtx_s* dctx) + { + if (dctx == null) + { + return 0; + } + + return (nuint)sizeof(ZSTD_DCtx_s) + + ZSTD_sizeof_DDict(dctx->ddictLocal) + + dctx->inBuffSize + + dctx->outBuffSize; + } + + public static nuint ZSTD_estimateDCtxSize() + { + return (nuint)sizeof(ZSTD_DCtx_s); + } + + private static nuint ZSTD_startingInputLength(ZSTD_format_e format) + { + nuint startingInputLength = (nuint)(format == ZSTD_format_e.ZSTD_f_zstd1 ? 5 : 1); + assert( + format == ZSTD_format_e.ZSTD_f_zstd1 || format == ZSTD_format_e.ZSTD_f_zstd1_magicless + ); + return startingInputLength; + } + + private static void ZSTD_DCtx_resetParameters(ZSTD_DCtx_s* dctx) + { + assert(dctx->streamStage == ZSTD_dStreamStage.zdss_init); + dctx->format = ZSTD_format_e.ZSTD_f_zstd1; + dctx->maxWindowSize = ((uint)1 << 27) + 1; + dctx->outBufferMode = ZSTD_bufferMode_e.ZSTD_bm_buffered; + dctx->forceIgnoreChecksum = ZSTD_forceIgnoreChecksum_e.ZSTD_d_validateChecksum; + dctx->refMultipleDDicts = ZSTD_refMultipleDDicts_e.ZSTD_rmd_refSingleDDict; + dctx->disableHufAsm = 0; + dctx->maxBlockSizeParam = 0; + } + + private static void ZSTD_initDCtx_internal(ZSTD_DCtx_s* dctx) + { + dctx->staticSize = 0; + dctx->ddict = null; + dctx->ddictLocal = null; + dctx->dictEnd = null; + dctx->ddictIsCold = 0; + dctx->dictUses = ZSTD_dictUses_e.ZSTD_dont_use; + dctx->inBuff = null; + dctx->inBuffSize = 0; + dctx->outBuffSize = 0; + dctx->streamStage = ZSTD_dStreamStage.zdss_init; + dctx->noForwardProgress = 0; + dctx->oversizedDuration = 0; + dctx->isFrameDecompression = 1; + dctx->ddictSet = null; + ZSTD_DCtx_resetParameters(dctx); + } + + public static ZSTD_DCtx_s* ZSTD_initStaticDCtx(void* workspace, nuint workspaceSize) + { + ZSTD_DCtx_s* dctx = (ZSTD_DCtx_s*)workspace; + if (((nuint)workspace & 7) != 0) + { + return null; + } + + if (workspaceSize < (nuint)sizeof(ZSTD_DCtx_s)) + { + return null; + } + + ZSTD_initDCtx_internal(dctx); + dctx->staticSize = workspaceSize; + dctx->inBuff = (sbyte*)(dctx + 1); + return dctx; + } + + private static ZSTD_DCtx_s* ZSTD_createDCtx_internal(ZSTD_customMem customMem) + { + if (((customMem.customAlloc == null ? 1 : 0) ^ (customMem.customFree == null ? 1 : 0)) != 0) + { + return null; + } + + { + ZSTD_DCtx_s* dctx = (ZSTD_DCtx_s*)ZSTD_customMalloc( + (nuint)sizeof(ZSTD_DCtx_s), + customMem + ); + if (dctx == null) + { + return null; + } + + dctx->customMem = customMem; + ZSTD_initDCtx_internal(dctx); + return dctx; + } + } + + public static ZSTD_DCtx_s* ZSTD_createDCtx_advanced(ZSTD_customMem customMem) + { + return ZSTD_createDCtx_internal(customMem); + } + + public static ZSTD_DCtx_s* ZSTD_createDCtx() + { + return ZSTD_createDCtx_internal(ZSTD_defaultCMem); + } + + private static void ZSTD_clearDict(ZSTD_DCtx_s* dctx) + { + ZSTD_freeDDict(dctx->ddictLocal); + dctx->ddictLocal = null; + dctx->ddict = null; + dctx->dictUses = ZSTD_dictUses_e.ZSTD_dont_use; + } + + public static nuint ZSTD_freeDCtx(ZSTD_DCtx_s* dctx) + { + if (dctx == null) + { + return 0; + } + + if (dctx->staticSize != 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + + { + ZSTD_customMem cMem = dctx->customMem; + ZSTD_clearDict(dctx); + ZSTD_customFree(dctx->inBuff, cMem); + dctx->inBuff = null; + if (dctx->ddictSet != null) + { + ZSTD_freeDDictHashSet(dctx->ddictSet, cMem); + dctx->ddictSet = null; + } + + ZSTD_customFree(dctx, cMem); + return 0; + } + } + + /* no longer useful */ + public static void ZSTD_copyDCtx(ZSTD_DCtx_s* dstDCtx, ZSTD_DCtx_s* srcDCtx) + { + nuint toCopy = (nuint)((sbyte*)&dstDCtx->inBuff - (sbyte*)dstDCtx); + memcpy(dstDCtx, srcDCtx, (uint)toCopy); + } + + /* Given a dctx with a digested frame params, re-selects the correct ZSTD_DDict based on + * the requested dict ID from the frame. If there exists a reference to the correct ZSTD_DDict, then + * accordingly sets the ddict to be used to decompress the frame. + * + * If no DDict is found, then no action is taken, and the ZSTD_DCtx::ddict remains as-is. + * + * ZSTD_d_refMultipleDDicts must be enabled for this function to be called. + */ + private static void ZSTD_DCtx_selectFrameDDict(ZSTD_DCtx_s* dctx) + { + assert(dctx->refMultipleDDicts != default && dctx->ddictSet != null); + if (dctx->ddict != null) + { + ZSTD_DDict_s* frameDDict = ZSTD_DDictHashSet_getDDict( + dctx->ddictSet, + dctx->fParams.dictID + ); + if (frameDDict != null) + { + ZSTD_clearDict(dctx); + dctx->dictID = dctx->fParams.dictID; + dctx->ddict = frameDDict; + dctx->dictUses = ZSTD_dictUses_e.ZSTD_use_indefinitely; + } + } + } + + /*! ZSTD_isFrame() : + * Tells if the content of `buffer` starts with a valid Frame Identifier. + * Note : Frame Identifier is 4 bytes. If `size < 4`, @return will always be 0. + * Note 2 : Legacy Frame Identifiers are considered valid only if Legacy Support is enabled. + * Note 3 : Skippable Frame Identifiers are considered valid. */ + public static uint ZSTD_isFrame(void* buffer, nuint size) + { + if (size < 4) + { + return 0; + } + + { + uint magic = MEM_readLE32(buffer); + if (magic == 0xFD2FB528) + { + return 1; + } + + if ((magic & 0xFFFFFFF0) == 0x184D2A50) + { + return 1; + } + } + + return 0; + } + + /*! ZSTD_isSkippableFrame() : + * Tells if the content of `buffer` starts with a valid Frame Identifier for a skippable frame. + * Note : Frame Identifier is 4 bytes. If `size < 4`, @return will always be 0. + */ + public static uint ZSTD_isSkippableFrame(void* buffer, nuint size) + { + if (size < 4) + { + return 0; + } + + { + uint magic = MEM_readLE32(buffer); + if ((magic & 0xFFFFFFF0) == 0x184D2A50) + { + return 1; + } + } + + return 0; + } + + /** ZSTD_frameHeaderSize_internal() : + * srcSize must be large enough to reach header size fields. + * note : only works for formats ZSTD_f_zstd1 and ZSTD_f_zstd1_magicless. + * @return : size of the Frame Header + * or an error code, which can be tested with ZSTD_isError() */ + private static nuint ZSTD_frameHeaderSize_internal( + void* src, + nuint srcSize, + ZSTD_format_e format + ) + { + nuint minInputSize = ZSTD_startingInputLength(format); + if (srcSize < minInputSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + { + byte fhd = ((byte*)src)[minInputSize - 1]; + uint dictID = (uint)(fhd & 3); + uint singleSegment = (uint)(fhd >> 5 & 1); + uint fcsId = (uint)(fhd >> 6); + return minInputSize + + (nuint)(singleSegment == 0 ? 1 : 0) + + ZSTD_did_fieldSize[dictID] + + ZSTD_fcs_fieldSize[fcsId] + + (nuint)(singleSegment != 0 && fcsId == 0 ? 1 : 0); + } + } + + /** ZSTD_frameHeaderSize() : + * srcSize must be >= ZSTD_frameHeaderSize_prefix. + * @return : size of the Frame Header, + * or an error code (if srcSize is too small) */ + public static nuint ZSTD_frameHeaderSize(void* src, nuint srcSize) + { + return ZSTD_frameHeaderSize_internal(src, srcSize, ZSTD_format_e.ZSTD_f_zstd1); + } + + /** ZSTD_getFrameHeader_advanced() : + * decode Frame Header, or require larger `srcSize`. + * note : only works for formats ZSTD_f_zstd1 and ZSTD_f_zstd1_magicless + * @return : 0, `zfhPtr` is correctly filled, + * >0, `srcSize` is too small, value is wanted `srcSize` amount, + ** or an error code, which can be tested using ZSTD_isError() */ + public static nuint ZSTD_getFrameHeader_advanced( + ZSTD_frameHeader* zfhPtr, + void* src, + nuint srcSize, + ZSTD_format_e format + ) + { + byte* ip = (byte*)src; + nuint minInputSize = ZSTD_startingInputLength(format); + if (srcSize > 0) + { + if (src == null) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + } + + if (srcSize < minInputSize) + { + if (srcSize > 0 && format != ZSTD_format_e.ZSTD_f_zstd1_magicless) + { + /* when receiving less than @minInputSize bytes, + * control these bytes at least correspond to a supported magic number + * in order to error out early if they don't. + **/ + nuint toCopy = 4 < srcSize ? 4 : srcSize; + byte* hbuf = stackalloc byte[4]; + MEM_writeLE32(hbuf, 0xFD2FB528); + assert(src != null); + memcpy(hbuf, src, (uint)toCopy); + if (MEM_readLE32(hbuf) != 0xFD2FB528) + { + MEM_writeLE32(hbuf, 0x184D2A50); + memcpy(hbuf, src, (uint)toCopy); + if ((MEM_readLE32(hbuf) & 0xFFFFFFF0) != 0x184D2A50) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_prefix_unknown)); + } + } + } + + return minInputSize; + } + + *zfhPtr = new ZSTD_frameHeader(); + if (format != ZSTD_format_e.ZSTD_f_zstd1_magicless && MEM_readLE32(src) != 0xFD2FB528) + { + if ((MEM_readLE32(src) & 0xFFFFFFF0) == 0x184D2A50) + { + if (srcSize < 8) + { + return 8; + } + + *zfhPtr = new ZSTD_frameHeader + { + frameType = ZSTD_frameType_e.ZSTD_skippableFrame, + dictID = MEM_readLE32(src) - 0x184D2A50, + headerSize = 8, + frameContentSize = MEM_readLE32((sbyte*)src + 4), + }; + return 0; + } + + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_prefix_unknown)); + } + + { + nuint fhsize = ZSTD_frameHeaderSize_internal(src, srcSize, format); + if (srcSize < fhsize) + { + return fhsize; + } + + zfhPtr->headerSize = (uint)fhsize; + } + + { + byte fhdByte = ip[minInputSize - 1]; + nuint pos = minInputSize; + uint dictIDSizeCode = (uint)(fhdByte & 3); + uint checksumFlag = (uint)(fhdByte >> 2 & 1); + uint singleSegment = (uint)(fhdByte >> 5 & 1); + uint fcsID = (uint)(fhdByte >> 6); + ulong windowSize = 0; + uint dictID = 0; + ulong frameContentSize = unchecked(0UL - 1); + if ((fhdByte & 0x08) != 0) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_frameParameter_unsupported) + ); + } + + if (singleSegment == 0) + { + byte wlByte = ip[pos++]; + uint windowLog = (uint)((wlByte >> 3) + 10); + if (windowLog > (uint)(sizeof(nuint) == 4 ? 30 : 31)) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_frameParameter_windowTooLarge) + ); + } + + windowSize = 1UL << (int)windowLog; + windowSize += (windowSize >> 3) * (ulong)(wlByte & 7); + } + + switch (dictIDSizeCode) + { + default: + assert(0 != 0); + goto case 0; + case 0: + break; + case 1: + dictID = ip[pos]; + pos++; + break; + case 2: + dictID = MEM_readLE16(ip + pos); + pos += 2; + break; + case 3: + dictID = MEM_readLE32(ip + pos); + pos += 4; + break; + } + + switch (fcsID) + { + default: + assert(0 != 0); + goto case 0; + case 0: + if (singleSegment != 0) + { + frameContentSize = ip[pos]; + } + + break; + case 1: + frameContentSize = (ulong)(MEM_readLE16(ip + pos) + 256); + break; + case 2: + frameContentSize = MEM_readLE32(ip + pos); + break; + case 3: + frameContentSize = MEM_readLE64(ip + pos); + break; + } + + if (singleSegment != 0) + { + windowSize = frameContentSize; + } + + zfhPtr->frameType = ZSTD_frameType_e.ZSTD_frame; + zfhPtr->frameContentSize = frameContentSize; + zfhPtr->windowSize = windowSize; + zfhPtr->blockSizeMax = (uint)(windowSize < 1 << 17 ? windowSize : 1 << 17); + zfhPtr->dictID = dictID; + zfhPtr->checksumFlag = checksumFlag; + } + + return 0; + } + + /** ZSTD_getFrameHeader() : + * decode Frame Header, or require larger `srcSize`. + * note : this function does not consume input, it only reads it. + * @return : 0, `zfhPtr` is correctly filled, + * >0, `srcSize` is too small, value is wanted `srcSize` amount, + * or an error code, which can be tested using ZSTD_isError() */ + public static nuint ZSTD_getFrameHeader(ZSTD_frameHeader* zfhPtr, void* src, nuint srcSize) + { + return ZSTD_getFrameHeader_advanced(zfhPtr, src, srcSize, ZSTD_format_e.ZSTD_f_zstd1); + } + + /** ZSTD_getFrameContentSize() : + * compatible with legacy mode + * @return : decompressed size of the single frame pointed to be `src` if known, otherwise + * - ZSTD_CONTENTSIZE_UNKNOWN if the size cannot be determined + * - ZSTD_CONTENTSIZE_ERROR if an error occurred (e.g. invalid magic number, srcSize too small) */ + public static ulong ZSTD_getFrameContentSize(void* src, nuint srcSize) + { + ZSTD_frameHeader zfh; + if (ZSTD_getFrameHeader(&zfh, src, srcSize) != 0) + { + return unchecked(0UL - 2); + } + + if (zfh.frameType == ZSTD_frameType_e.ZSTD_skippableFrame) + { + return 0; + } + else + { + return zfh.frameContentSize; + } + } + + private static nuint readSkippableFrameSize(void* src, nuint srcSize) + { + const nuint skippableHeaderSize = 8; + uint sizeU32; + if (srcSize < 8) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + sizeU32 = MEM_readLE32((byte*)src + 4); + if (sizeU32 + 8 < sizeU32) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_frameParameter_unsupported)); + } + + { + nuint skippableSize = skippableHeaderSize + sizeU32; + if (skippableSize > srcSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + return skippableSize; + } + } + + /*! ZSTD_readSkippableFrame() : + * Retrieves content of a skippable frame, and writes it to dst buffer. + * + * The parameter magicVariant will receive the magicVariant that was supplied when the frame was written, + * i.e. magicNumber - ZSTD_MAGIC_SKIPPABLE_START. This can be NULL if the caller is not interested + * in the magicVariant. + * + * Returns an error if destination buffer is not large enough, or if this is not a valid skippable frame. + * + * @return : number of bytes written or a ZSTD error. + */ + public static nuint ZSTD_readSkippableFrame( + void* dst, + nuint dstCapacity, + uint* magicVariant, + void* src, + nuint srcSize + ) + { + if (srcSize < 8) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + { + uint magicNumber = MEM_readLE32(src); + nuint skippableFrameSize = readSkippableFrameSize(src, srcSize); + nuint skippableContentSize = skippableFrameSize - 8; + if (ZSTD_isSkippableFrame(src, srcSize) == 0) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_frameParameter_unsupported) + ); + } + + if (skippableFrameSize < 8 || skippableFrameSize > srcSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + if (skippableContentSize > dstCapacity) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + if (skippableContentSize > 0 && dst != null) + { + memcpy(dst, (byte*)src + 8, (uint)skippableContentSize); + } + + if (magicVariant != null) + { + *magicVariant = magicNumber - 0x184D2A50; + } + + return skippableContentSize; + } + } + + /** ZSTD_findDecompressedSize() : + * `srcSize` must be the exact length of some number of ZSTD compressed and/or + * skippable frames + * note: compatible with legacy mode + * @return : decompressed size of the frames contained */ + public static ulong ZSTD_findDecompressedSize(void* src, nuint srcSize) + { + ulong totalDstSize = 0; + while (srcSize >= ZSTD_startingInputLength(ZSTD_format_e.ZSTD_f_zstd1)) + { + uint magicNumber = MEM_readLE32(src); + if ((magicNumber & 0xFFFFFFF0) == 0x184D2A50) + { + nuint skippableSize = readSkippableFrameSize(src, srcSize); + if (ERR_isError(skippableSize)) + { + return unchecked(0UL - 2); + } + + assert(skippableSize <= srcSize); + src = (byte*)src + skippableSize; + srcSize -= skippableSize; + continue; + } + + { + ulong fcs = ZSTD_getFrameContentSize(src, srcSize); + if (fcs >= unchecked(0UL - 2)) + { + return fcs; + } + + if (totalDstSize + fcs < totalDstSize) + { + return unchecked(0UL - 2); + } + + totalDstSize += fcs; + } + + { + nuint frameSrcSize = ZSTD_findFrameCompressedSize(src, srcSize); + if (ERR_isError(frameSrcSize)) + { + return unchecked(0UL - 2); + } + + assert(frameSrcSize <= srcSize); + src = (byte*)src + frameSrcSize; + srcSize -= frameSrcSize; + } + } + + if (srcSize != 0) + { + return unchecked(0UL - 2); + } + + return totalDstSize; + } + + /** ZSTD_getDecompressedSize() : + * compatible with legacy mode + * @return : decompressed size if known, 0 otherwise + note : 0 can mean any of the following : + - frame content is empty + - decompressed size field is not present in frame header + - frame header unknown / not supported + - frame header not complete (`srcSize` too small) */ + public static ulong ZSTD_getDecompressedSize(void* src, nuint srcSize) + { + ulong ret = ZSTD_getFrameContentSize(src, srcSize); + return ret >= unchecked(0UL - 2) ? 0 : ret; + } + + /** ZSTD_decodeFrameHeader() : + * `headerSize` must be the size provided by ZSTD_frameHeaderSize(). + * If multiple DDict references are enabled, also will choose the correct DDict to use. + * @return : 0 if success, or an error code, which can be tested using ZSTD_isError() */ + private static nuint ZSTD_decodeFrameHeader(ZSTD_DCtx_s* dctx, void* src, nuint headerSize) + { + nuint result = ZSTD_getFrameHeader_advanced(&dctx->fParams, src, headerSize, dctx->format); + if (ERR_isError(result)) + { + return result; + } + + if (result > 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + if ( + dctx->refMultipleDDicts == ZSTD_refMultipleDDicts_e.ZSTD_rmd_refMultipleDDicts + && dctx->ddictSet != null + ) + { + ZSTD_DCtx_selectFrameDDict(dctx); + } + + if (dctx->fParams.dictID != 0 && dctx->dictID != dctx->fParams.dictID) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_wrong)); + } + + dctx->validateChecksum = (uint)( + dctx->fParams.checksumFlag != 0 && dctx->forceIgnoreChecksum == default ? 1 : 0 + ); + if (dctx->validateChecksum != 0) + { + ZSTD_XXH64_reset(&dctx->xxhState, 0); + } + + dctx->processedCSize += headerSize; + return 0; + } + + private static ZSTD_frameSizeInfo ZSTD_errorFrameSizeInfo(nuint ret) + { + ZSTD_frameSizeInfo frameSizeInfo; + System.Runtime.CompilerServices.Unsafe.SkipInit(out frameSizeInfo); + frameSizeInfo.compressedSize = ret; + frameSizeInfo.decompressedBound = unchecked(0UL - 2); + return frameSizeInfo; + } + + private static ZSTD_frameSizeInfo ZSTD_findFrameSizeInfo( + void* src, + nuint srcSize, + ZSTD_format_e format + ) + { + ZSTD_frameSizeInfo frameSizeInfo; + frameSizeInfo = new ZSTD_frameSizeInfo(); + if ( + format == ZSTD_format_e.ZSTD_f_zstd1 + && srcSize >= 8 + && (MEM_readLE32(src) & 0xFFFFFFF0) == 0x184D2A50 + ) + { + frameSizeInfo.compressedSize = readSkippableFrameSize(src, srcSize); + assert( + ERR_isError(frameSizeInfo.compressedSize) || frameSizeInfo.compressedSize <= srcSize + ); + return frameSizeInfo; + } + else + { + byte* ip = (byte*)src; + byte* ipstart = ip; + nuint remainingSize = srcSize; + nuint nbBlocks = 0; + ZSTD_frameHeader zfh; + { + nuint ret = ZSTD_getFrameHeader_advanced(&zfh, src, srcSize, format); + if (ERR_isError(ret)) + { + return ZSTD_errorFrameSizeInfo(ret); + } + + if (ret > 0) + { + return ZSTD_errorFrameSizeInfo( + unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)) + ); + } + } + + ip += zfh.headerSize; + remainingSize -= zfh.headerSize; + while (true) + { + blockProperties_t blockProperties; + nuint cBlockSize = ZSTD_getcBlockSize(ip, remainingSize, &blockProperties); + if (ERR_isError(cBlockSize)) + { + return ZSTD_errorFrameSizeInfo(cBlockSize); + } + + if (ZSTD_blockHeaderSize + cBlockSize > remainingSize) + { + return ZSTD_errorFrameSizeInfo( + unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)) + ); + } + + ip += ZSTD_blockHeaderSize + cBlockSize; + remainingSize -= ZSTD_blockHeaderSize + cBlockSize; + nbBlocks++; + if (blockProperties.lastBlock != 0) + { + break; + } + } + + if (zfh.checksumFlag != 0) + { + if (remainingSize < 4) + { + return ZSTD_errorFrameSizeInfo( + unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)) + ); + } + + ip += 4; + } + + frameSizeInfo.nbBlocks = nbBlocks; + frameSizeInfo.compressedSize = (nuint)(ip - ipstart); + frameSizeInfo.decompressedBound = + zfh.frameContentSize != unchecked(0UL - 1) + ? zfh.frameContentSize + : (ulong)nbBlocks * zfh.blockSizeMax; + return frameSizeInfo; + } + } + + private static nuint ZSTD_findFrameCompressedSize_advanced( + void* src, + nuint srcSize, + ZSTD_format_e format + ) + { + ZSTD_frameSizeInfo frameSizeInfo = ZSTD_findFrameSizeInfo(src, srcSize, format); + return frameSizeInfo.compressedSize; + } + + /** ZSTD_findFrameCompressedSize() : + * See docs in zstd.h + * Note: compatible with legacy mode */ + public static nuint ZSTD_findFrameCompressedSize(void* src, nuint srcSize) + { + return ZSTD_findFrameCompressedSize_advanced(src, srcSize, ZSTD_format_e.ZSTD_f_zstd1); + } + + /** ZSTD_decompressBound() : + * compatible with legacy mode + * `src` must point to the start of a ZSTD frame or a skippable frame + * `srcSize` must be at least as large as the frame contained + * @return : the maximum decompressed size of the compressed source + */ + public static ulong ZSTD_decompressBound(void* src, nuint srcSize) + { + ulong bound = 0; + while (srcSize > 0) + { + ZSTD_frameSizeInfo frameSizeInfo = ZSTD_findFrameSizeInfo( + src, + srcSize, + ZSTD_format_e.ZSTD_f_zstd1 + ); + nuint compressedSize = frameSizeInfo.compressedSize; + ulong decompressedBound = frameSizeInfo.decompressedBound; + if (ERR_isError(compressedSize) || decompressedBound == unchecked(0UL - 2)) + { + return unchecked(0UL - 2); + } + + assert(srcSize >= compressedSize); + src = (byte*)src + compressedSize; + srcSize -= compressedSize; + bound += decompressedBound; + } + + return bound; + } + + /*! ZSTD_decompressionMargin() : + * Zstd supports in-place decompression, where the input and output buffers overlap. + * In this case, the output buffer must be at least (Margin + Output_Size) bytes large, + * and the input buffer must be at the end of the output buffer. + * + * _______________________ Output Buffer ________________________ + * | | + * | ____ Input Buffer ____| + * | | | + * v v v + * |---------------------------------------|-----------|----------| + * ^ ^ ^ + * |___________________ Output_Size ___________________|_ Margin _| + * + * NOTE: See also ZSTD_DECOMPRESSION_MARGIN(). + * NOTE: This applies only to single-pass decompression through ZSTD_decompress() or + * ZSTD_decompressDCtx(). + * NOTE: This function supports multi-frame input. + * + * @param src The compressed frame(s) + * @param srcSize The size of the compressed frame(s) + * @returns The decompression margin or an error that can be checked with ZSTD_isError(). + */ + public static nuint ZSTD_decompressionMargin(void* src, nuint srcSize) + { + nuint margin = 0; + uint maxBlockSize = 0; + while (srcSize > 0) + { + ZSTD_frameSizeInfo frameSizeInfo = ZSTD_findFrameSizeInfo( + src, + srcSize, + ZSTD_format_e.ZSTD_f_zstd1 + ); + nuint compressedSize = frameSizeInfo.compressedSize; + ulong decompressedBound = frameSizeInfo.decompressedBound; + ZSTD_frameHeader zfh; + { + nuint err_code = ZSTD_getFrameHeader(&zfh, src, srcSize); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (ERR_isError(compressedSize) || decompressedBound == unchecked(0UL - 2)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + if (zfh.frameType == ZSTD_frameType_e.ZSTD_frame) + { + margin += zfh.headerSize; + margin += (nuint)(zfh.checksumFlag != 0 ? 4 : 0); + margin += 3 * frameSizeInfo.nbBlocks; + maxBlockSize = maxBlockSize > zfh.blockSizeMax ? maxBlockSize : zfh.blockSizeMax; + } + else + { + assert(zfh.frameType == ZSTD_frameType_e.ZSTD_skippableFrame); + margin += compressedSize; + } + + assert(srcSize >= compressedSize); + src = (byte*)src + compressedSize; + srcSize -= compressedSize; + } + + margin += maxBlockSize; + return margin; + } + + /** ZSTD_insertBlock() : + * insert `src` block into `dctx` history. Useful to track uncompressed blocks. */ + public static nuint ZSTD_insertBlock(ZSTD_DCtx_s* dctx, void* blockStart, nuint blockSize) + { + ZSTD_checkContinuity(dctx, blockStart, blockSize); + dctx->previousDstEnd = (sbyte*)blockStart + blockSize; + return blockSize; + } + + private static nuint ZSTD_copyRawBlock(void* dst, nuint dstCapacity, void* src, nuint srcSize) + { + if (srcSize > dstCapacity) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + if (dst == null) + { + if (srcSize == 0) + { + return 0; + } + + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstBuffer_null)); + } + + memmove(dst, src, srcSize); + return srcSize; + } + + private static nuint ZSTD_setRleBlock(void* dst, nuint dstCapacity, byte b, nuint regenSize) + { + if (regenSize > dstCapacity) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + if (dst == null) + { + if (regenSize == 0) + { + return 0; + } + + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstBuffer_null)); + } + + memset(dst, b, (uint)regenSize); + return regenSize; + } + + private static void ZSTD_DCtx_trace_end( + ZSTD_DCtx_s* dctx, + ulong uncompressedSize, + ulong compressedSize, + int streaming + ) { } + + /*! ZSTD_decompressFrame() : + * @dctx must be properly initialized + * will update *srcPtr and *srcSizePtr, + * to make *srcPtr progress by one frame. */ + private static nuint ZSTD_decompressFrame( + ZSTD_DCtx_s* dctx, + void* dst, + nuint dstCapacity, + void** srcPtr, + nuint* srcSizePtr + ) + { + byte* istart = (byte*)*srcPtr; + byte* ip = istart; + byte* ostart = (byte*)dst; + byte* oend = dstCapacity != 0 ? ostart + dstCapacity : ostart; + byte* op = ostart; + nuint remainingSrcSize = *srcSizePtr; + if ( + remainingSrcSize + < (nuint)(dctx->format == ZSTD_format_e.ZSTD_f_zstd1 ? 6 : 2) + ZSTD_blockHeaderSize + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + { + nuint frameHeaderSize = ZSTD_frameHeaderSize_internal( + ip, + (nuint)(dctx->format == ZSTD_format_e.ZSTD_f_zstd1 ? 5 : 1), + dctx->format + ); + if (ERR_isError(frameHeaderSize)) + { + return frameHeaderSize; + } + + if (remainingSrcSize < frameHeaderSize + ZSTD_blockHeaderSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + { + nuint err_code = ZSTD_decodeFrameHeader(dctx, ip, frameHeaderSize); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + ip += frameHeaderSize; + remainingSrcSize -= frameHeaderSize; + } + + if (dctx->maxBlockSizeParam != 0) + { + dctx->fParams.blockSizeMax = + dctx->fParams.blockSizeMax < (uint)dctx->maxBlockSizeParam + ? dctx->fParams.blockSizeMax + : (uint)dctx->maxBlockSizeParam; + } + + while (true) + { + byte* oBlockEnd = oend; + nuint decodedSize; + blockProperties_t blockProperties; + nuint cBlockSize = ZSTD_getcBlockSize(ip, remainingSrcSize, &blockProperties); + if (ERR_isError(cBlockSize)) + { + return cBlockSize; + } + + ip += ZSTD_blockHeaderSize; + remainingSrcSize -= ZSTD_blockHeaderSize; + if (cBlockSize > remainingSrcSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + if (ip >= op && ip < oBlockEnd) + { + oBlockEnd = op + (ip - op); + } + + switch (blockProperties.blockType) + { + case blockType_e.bt_compressed: + assert(dctx->isFrameDecompression == 1); + decodedSize = ZSTD_decompressBlock_internal( + dctx, + op, + (nuint)(oBlockEnd - op), + ip, + cBlockSize, + streaming_operation.not_streaming + ); + break; + case blockType_e.bt_raw: + decodedSize = ZSTD_copyRawBlock(op, (nuint)(oend - op), ip, cBlockSize); + break; + case blockType_e.bt_rle: + decodedSize = ZSTD_setRleBlock( + op, + (nuint)(oBlockEnd - op), + *ip, + blockProperties.origSize + ); + break; + case blockType_e.bt_reserved: + default: + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + { + nuint err_code = decodedSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (dctx->validateChecksum != 0) + { + ZSTD_XXH64_update(&dctx->xxhState, op, decodedSize); + } + + if (decodedSize != 0) + { + op += decodedSize; + } + + assert(ip != null); + ip += cBlockSize; + remainingSrcSize -= cBlockSize; + if (blockProperties.lastBlock != 0) + { + break; + } + } + + if (dctx->fParams.frameContentSize != unchecked(0UL - 1)) + { + if ((ulong)(op - ostart) != dctx->fParams.frameContentSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + } + + if (dctx->fParams.checksumFlag != 0) + { + if (remainingSrcSize < 4) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_checksum_wrong)); + } + + if (dctx->forceIgnoreChecksum == default) + { + uint checkCalc = (uint)ZSTD_XXH64_digest(&dctx->xxhState); + uint checkRead; + checkRead = MEM_readLE32(ip); + if (checkRead != checkCalc) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_checksum_wrong)); + } + } + + ip += 4; + remainingSrcSize -= 4; + } + + ZSTD_DCtx_trace_end(dctx, (ulong)(op - ostart), (ulong)(ip - istart), 0); + *srcPtr = ip; + *srcSizePtr = remainingSrcSize; + return (nuint)(op - ostart); + } + + private static nuint ZSTD_decompressMultiFrame( + ZSTD_DCtx_s* dctx, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize, + void* dict, + nuint dictSize, + ZSTD_DDict_s* ddict + ) + { + void* dststart = dst; + int moreThan1Frame = 0; + assert(dict == null || ddict == null); + if (ddict != null) + { + dict = ZSTD_DDict_dictContent(ddict); + dictSize = ZSTD_DDict_dictSize(ddict); + } + + while (srcSize >= ZSTD_startingInputLength(dctx->format)) + { + if (dctx->format == ZSTD_format_e.ZSTD_f_zstd1 && srcSize >= 4) + { + uint magicNumber = MEM_readLE32(src); + if ((magicNumber & 0xFFFFFFF0) == 0x184D2A50) + { + /* skippable frame detected : skip it */ + nuint skippableSize = readSkippableFrameSize(src, srcSize); + { + nuint err_code = skippableSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + assert(skippableSize <= srcSize); + src = (byte*)src + skippableSize; + srcSize -= skippableSize; + continue; + } + } + + if (ddict != null) + { + /* we were called from ZSTD_decompress_usingDDict */ + nuint err_code = ZSTD_decompressBegin_usingDDict(dctx, ddict); + if (ERR_isError(err_code)) + { + return err_code; + } + } + else + { + /* this will initialize correctly with no dict if dict == NULL, so + * use this in all cases but ddict */ + nuint err_code = ZSTD_decompressBegin_usingDict(dctx, dict, dictSize); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + ZSTD_checkContinuity(dctx, dst, dstCapacity); + { + nuint res = ZSTD_decompressFrame(dctx, dst, dstCapacity, &src, &srcSize); + if ( + ZSTD_getErrorCode(res) == ZSTD_ErrorCode.ZSTD_error_prefix_unknown + && moreThan1Frame == 1 + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + if (ERR_isError(res)) + { + return res; + } + + assert(res <= dstCapacity); + if (res != 0) + { + dst = (byte*)dst + res; + } + + dstCapacity -= res; + } + + moreThan1Frame = 1; + } + + if (srcSize != 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + return (nuint)((byte*)dst - (byte*)dststart); + } + + /*! ZSTD_decompress_usingDict() : + * Decompression using a known Dictionary. + * Dictionary must be identical to the one used during compression. + * Note : This function loads the dictionary, resulting in significant startup delay. + * It's intended for a dictionary used only once. + * Note : When `dict == NULL || dictSize < 8` no dictionary is used. */ + public static nuint ZSTD_decompress_usingDict( + ZSTD_DCtx_s* dctx, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize, + void* dict, + nuint dictSize + ) + { + return ZSTD_decompressMultiFrame( + dctx, + dst, + dstCapacity, + src, + srcSize, + dict, + dictSize, + null + ); + } + + private static ZSTD_DDict_s* ZSTD_getDDict(ZSTD_DCtx_s* dctx) + { + switch (dctx->dictUses) + { + default: + assert(0 != 0); + goto case ZSTD_dictUses_e.ZSTD_dont_use; + case ZSTD_dictUses_e.ZSTD_dont_use: + ZSTD_clearDict(dctx); + return null; + case ZSTD_dictUses_e.ZSTD_use_indefinitely: + return dctx->ddict; + case ZSTD_dictUses_e.ZSTD_use_once: + dctx->dictUses = ZSTD_dictUses_e.ZSTD_dont_use; + return dctx->ddict; + } + } + + /*! ZSTD_decompressDCtx() : + * Same as ZSTD_decompress(), + * requires an allocated ZSTD_DCtx. + * Compatible with sticky parameters (see below). + */ + public static nuint ZSTD_decompressDCtx( + ZSTD_DCtx_s* dctx, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize + ) + { + return ZSTD_decompress_usingDDict( + dctx, + dst, + dstCapacity, + src, + srcSize, + ZSTD_getDDict(dctx) + ); + } + + /*! ZSTD_decompress() : + * `compressedSize` : must be the _exact_ size of some number of compressed and/or skippable frames. + * Multiple compressed frames can be decompressed at once with this method. + * The result will be the concatenation of all decompressed frames, back to back. + * `dstCapacity` is an upper bound of originalSize to regenerate. + * First frame's decompressed size can be extracted using ZSTD_getFrameContentSize(). + * If maximum upper bound isn't known, prefer using streaming mode to decompress data. + * @return : the number of bytes decompressed into `dst` (<= `dstCapacity`), + * or an errorCode if it fails (which can be tested using ZSTD_isError()). */ + public static nuint ZSTD_decompress(void* dst, nuint dstCapacity, void* src, nuint srcSize) + { + nuint regenSize; + ZSTD_DCtx_s* dctx = ZSTD_createDCtx_internal(ZSTD_defaultCMem); + if (dctx == null) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + + regenSize = ZSTD_decompressDCtx(dctx, dst, dstCapacity, src, srcSize); + ZSTD_freeDCtx(dctx); + return regenSize; + } + + /*-************************************** + * Advanced Streaming Decompression API + * Bufferless and synchronous + ****************************************/ + public static nuint ZSTD_nextSrcSizeToDecompress(ZSTD_DCtx_s* dctx) + { + return dctx->expected; + } + + /** + * Similar to ZSTD_nextSrcSizeToDecompress(), but when a block input can be streamed, we + * allow taking a partial block as the input. Currently only raw uncompressed blocks can + * be streamed. + * + * For blocks that can be streamed, this allows us to reduce the latency until we produce + * output, and avoid copying the input. + * + * @param inputSize - The total amount of input that the caller currently has. + */ + private static nuint ZSTD_nextSrcSizeToDecompressWithInputSize( + ZSTD_DCtx_s* dctx, + nuint inputSize + ) + { + if ( + !( + dctx->stage == ZSTD_dStage.ZSTDds_decompressBlock + || dctx->stage == ZSTD_dStage.ZSTDds_decompressLastBlock + ) + ) + { + return dctx->expected; + } + + if (dctx->bType != blockType_e.bt_raw) + { + return dctx->expected; + } + + return inputSize <= 1 ? 1 + : inputSize <= dctx->expected ? inputSize + : dctx->expected; + } + + public static ZSTD_nextInputType_e ZSTD_nextInputType(ZSTD_DCtx_s* dctx) + { + switch (dctx->stage) + { + default: + assert(0 != 0); + goto case ZSTD_dStage.ZSTDds_getFrameHeaderSize; + case ZSTD_dStage.ZSTDds_getFrameHeaderSize: + case ZSTD_dStage.ZSTDds_decodeFrameHeader: + return ZSTD_nextInputType_e.ZSTDnit_frameHeader; + case ZSTD_dStage.ZSTDds_decodeBlockHeader: + return ZSTD_nextInputType_e.ZSTDnit_blockHeader; + case ZSTD_dStage.ZSTDds_decompressBlock: + return ZSTD_nextInputType_e.ZSTDnit_block; + case ZSTD_dStage.ZSTDds_decompressLastBlock: + return ZSTD_nextInputType_e.ZSTDnit_lastBlock; + case ZSTD_dStage.ZSTDds_checkChecksum: + return ZSTD_nextInputType_e.ZSTDnit_checksum; + case ZSTD_dStage.ZSTDds_decodeSkippableHeader: + case ZSTD_dStage.ZSTDds_skipFrame: + return ZSTD_nextInputType_e.ZSTDnit_skippableFrame; + } + } + + private static int ZSTD_isSkipFrame(ZSTD_DCtx_s* dctx) + { + return dctx->stage == ZSTD_dStage.ZSTDds_skipFrame ? 1 : 0; + } + + /** ZSTD_decompressContinue() : + * srcSize : must be the exact nb of bytes expected (see ZSTD_nextSrcSizeToDecompress()) + * @return : nb of bytes generated into `dst` (necessarily <= `dstCapacity) + * or an error code, which can be tested using ZSTD_isError() */ + public static nuint ZSTD_decompressContinue( + ZSTD_DCtx_s* dctx, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize + ) + { + if (srcSize != ZSTD_nextSrcSizeToDecompressWithInputSize(dctx, srcSize)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + ZSTD_checkContinuity(dctx, dst, dstCapacity); + dctx->processedCSize += srcSize; + switch (dctx->stage) + { + case ZSTD_dStage.ZSTDds_getFrameHeaderSize: + assert(src != null); + if (dctx->format == ZSTD_format_e.ZSTD_f_zstd1) + { + assert(srcSize >= 4); + if ((MEM_readLE32(src) & 0xFFFFFFF0) == 0x184D2A50) + { + memcpy(dctx->headerBuffer, src, (uint)srcSize); + dctx->expected = 8 - srcSize; + dctx->stage = ZSTD_dStage.ZSTDds_decodeSkippableHeader; + return 0; + } + } + + dctx->headerSize = ZSTD_frameHeaderSize_internal(src, srcSize, dctx->format); + if (ERR_isError(dctx->headerSize)) + { + return dctx->headerSize; + } + + memcpy(dctx->headerBuffer, src, (uint)srcSize); + dctx->expected = dctx->headerSize - srcSize; + dctx->stage = ZSTD_dStage.ZSTDds_decodeFrameHeader; + return 0; + case ZSTD_dStage.ZSTDds_decodeFrameHeader: + assert(src != null); + memcpy(dctx->headerBuffer + (dctx->headerSize - srcSize), src, (uint)srcSize); + + { + nuint err_code = ZSTD_decodeFrameHeader( + dctx, + dctx->headerBuffer, + dctx->headerSize + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + dctx->expected = ZSTD_blockHeaderSize; + dctx->stage = ZSTD_dStage.ZSTDds_decodeBlockHeader; + return 0; + case ZSTD_dStage.ZSTDds_decodeBlockHeader: + { + blockProperties_t bp; + nuint cBlockSize = ZSTD_getcBlockSize(src, ZSTD_blockHeaderSize, &bp); + if (ERR_isError(cBlockSize)) + { + return cBlockSize; + } + + if (cBlockSize > dctx->fParams.blockSizeMax) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + dctx->expected = cBlockSize; + dctx->bType = bp.blockType; + dctx->rleSize = bp.origSize; + if (cBlockSize != 0) + { + dctx->stage = + bp.lastBlock != 0 + ? ZSTD_dStage.ZSTDds_decompressLastBlock + : ZSTD_dStage.ZSTDds_decompressBlock; + return 0; + } + + if (bp.lastBlock != 0) + { + if (dctx->fParams.checksumFlag != 0) + { + dctx->expected = 4; + dctx->stage = ZSTD_dStage.ZSTDds_checkChecksum; + } + else + { + dctx->expected = 0; + dctx->stage = ZSTD_dStage.ZSTDds_getFrameHeaderSize; + } + } + else + { + dctx->expected = ZSTD_blockHeaderSize; + dctx->stage = ZSTD_dStage.ZSTDds_decodeBlockHeader; + } + + return 0; + } + + case ZSTD_dStage.ZSTDds_decompressLastBlock: + case ZSTD_dStage.ZSTDds_decompressBlock: + { + nuint rSize; + switch (dctx->bType) + { + case blockType_e.bt_compressed: + assert(dctx->isFrameDecompression == 1); + rSize = ZSTD_decompressBlock_internal( + dctx, + dst, + dstCapacity, + src, + srcSize, + streaming_operation.is_streaming + ); + dctx->expected = 0; + break; + case blockType_e.bt_raw: + assert(srcSize <= dctx->expected); + rSize = ZSTD_copyRawBlock(dst, dstCapacity, src, srcSize); + + { + nuint err_code = rSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + assert(rSize == srcSize); + dctx->expected -= rSize; + break; + case blockType_e.bt_rle: + rSize = ZSTD_setRleBlock(dst, dstCapacity, *(byte*)src, dctx->rleSize); + dctx->expected = 0; + break; + case blockType_e.bt_reserved: + default: + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected) + ); + } + + { + nuint err_code = rSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (rSize > dctx->fParams.blockSizeMax) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + dctx->decodedSize += rSize; + if (dctx->validateChecksum != 0) + { + ZSTD_XXH64_update(&dctx->xxhState, dst, rSize); + } + + dctx->previousDstEnd = (sbyte*)dst + rSize; + if (dctx->expected > 0) + { + return rSize; + } + + if (dctx->stage == ZSTD_dStage.ZSTDds_decompressLastBlock) + { + if ( + dctx->fParams.frameContentSize != unchecked(0UL - 1) + && dctx->decodedSize != dctx->fParams.frameContentSize + ) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected) + ); + } + + if (dctx->fParams.checksumFlag != 0) + { + dctx->expected = 4; + dctx->stage = ZSTD_dStage.ZSTDds_checkChecksum; + } + else + { + ZSTD_DCtx_trace_end(dctx, dctx->decodedSize, dctx->processedCSize, 1); + dctx->expected = 0; + dctx->stage = ZSTD_dStage.ZSTDds_getFrameHeaderSize; + } + } + else + { + dctx->stage = ZSTD_dStage.ZSTDds_decodeBlockHeader; + dctx->expected = ZSTD_blockHeaderSize; + } + + return rSize; + } + + case ZSTD_dStage.ZSTDds_checkChecksum: + assert(srcSize == 4); + + { + if (dctx->validateChecksum != 0) + { + uint h32 = (uint)ZSTD_XXH64_digest(&dctx->xxhState); + uint check32 = MEM_readLE32(src); + if (check32 != h32) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_checksum_wrong) + ); + } + } + + ZSTD_DCtx_trace_end(dctx, dctx->decodedSize, dctx->processedCSize, 1); + dctx->expected = 0; + dctx->stage = ZSTD_dStage.ZSTDds_getFrameHeaderSize; + return 0; + } + + case ZSTD_dStage.ZSTDds_decodeSkippableHeader: + assert(src != null); + assert(srcSize <= 8); + assert(dctx->format != ZSTD_format_e.ZSTD_f_zstd1_magicless); + memcpy(dctx->headerBuffer + (8 - srcSize), src, (uint)srcSize); + dctx->expected = MEM_readLE32(dctx->headerBuffer + 4); + dctx->stage = ZSTD_dStage.ZSTDds_skipFrame; + return 0; + case ZSTD_dStage.ZSTDds_skipFrame: + dctx->expected = 0; + dctx->stage = ZSTD_dStage.ZSTDds_getFrameHeaderSize; + return 0; + default: + assert(0 != 0); + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + } + + private static nuint ZSTD_refDictContent(ZSTD_DCtx_s* dctx, void* dict, nuint dictSize) + { + dctx->dictEnd = dctx->previousDstEnd; + dctx->virtualStart = + (sbyte*)dict - ((sbyte*)dctx->previousDstEnd - (sbyte*)dctx->prefixStart); + dctx->prefixStart = dict; + dctx->previousDstEnd = (sbyte*)dict + dictSize; + return 0; + } + + /*! ZSTD_loadDEntropy() : + * dict : must point at beginning of a valid zstd dictionary. + * @return : size of entropy tables read */ + private static nuint ZSTD_loadDEntropy( + ZSTD_entropyDTables_t* entropy, + void* dict, + nuint dictSize + ) + { + byte* dictPtr = (byte*)dict; + byte* dictEnd = dictPtr + dictSize; + if (dictSize <= 8) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + assert(MEM_readLE32(dict) == 0xEC30A437); + dictPtr += 8; + { + /* use fse tables as temporary workspace; implies fse tables are grouped together */ + void* workspace = &entropy->LLTable; + nuint workspaceSize = (nuint)( + sizeof(ZSTD_seqSymbol) * 513 + + sizeof(ZSTD_seqSymbol) * 257 + + sizeof(ZSTD_seqSymbol) * 513 + ); + nuint hSize = HUF_readDTableX2_wksp( + entropy->hufTable, + dictPtr, + (nuint)(dictEnd - dictPtr), + workspace, + workspaceSize, + 0 + ); + if (ERR_isError(hSize)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + dictPtr += hSize; + } + + { + short* offcodeNCount = stackalloc short[32]; + uint offcodeMaxValue = 31, + offcodeLog; + nuint offcodeHeaderSize = FSE_readNCount( + offcodeNCount, + &offcodeMaxValue, + &offcodeLog, + dictPtr, + (nuint)(dictEnd - dictPtr) + ); + if (ERR_isError(offcodeHeaderSize)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + if (offcodeMaxValue > 31) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + if (offcodeLog > 8) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + ZSTD_buildFSETable( + &entropy->OFTable.e0, + offcodeNCount, + offcodeMaxValue, + OF_base, + OF_bits, + offcodeLog, + entropy->workspace, + sizeof(uint) * 157, + 0 + ); + dictPtr += offcodeHeaderSize; + } + + { + short* matchlengthNCount = stackalloc short[53]; + uint matchlengthMaxValue = 52, + matchlengthLog; + nuint matchlengthHeaderSize = FSE_readNCount( + matchlengthNCount, + &matchlengthMaxValue, + &matchlengthLog, + dictPtr, + (nuint)(dictEnd - dictPtr) + ); + if (ERR_isError(matchlengthHeaderSize)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + if (matchlengthMaxValue > 52) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + if (matchlengthLog > 9) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + ZSTD_buildFSETable( + &entropy->MLTable.e0, + matchlengthNCount, + matchlengthMaxValue, + ML_base, + ML_bits, + matchlengthLog, + entropy->workspace, + sizeof(uint) * 157, + 0 + ); + dictPtr += matchlengthHeaderSize; + } + + { + short* litlengthNCount = stackalloc short[36]; + uint litlengthMaxValue = 35, + litlengthLog; + nuint litlengthHeaderSize = FSE_readNCount( + litlengthNCount, + &litlengthMaxValue, + &litlengthLog, + dictPtr, + (nuint)(dictEnd - dictPtr) + ); + if (ERR_isError(litlengthHeaderSize)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + if (litlengthMaxValue > 35) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + if (litlengthLog > 9) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + ZSTD_buildFSETable( + &entropy->LLTable.e0, + litlengthNCount, + litlengthMaxValue, + LL_base, + LL_bits, + litlengthLog, + entropy->workspace, + sizeof(uint) * 157, + 0 + ); + dictPtr += litlengthHeaderSize; + } + + if (dictPtr + 12 > dictEnd) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + { + int i; + nuint dictContentSize = (nuint)(dictEnd - (dictPtr + 12)); + for (i = 0; i < 3; i++) + { + uint rep = MEM_readLE32(dictPtr); + dictPtr += 4; + if (rep == 0 || rep > dictContentSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + entropy->rep[i] = rep; + } + } + + return (nuint)(dictPtr - (byte*)dict); + } + + private static nuint ZSTD_decompress_insertDictionary( + ZSTD_DCtx_s* dctx, + void* dict, + nuint dictSize + ) + { + if (dictSize < 8) + { + return ZSTD_refDictContent(dctx, dict, dictSize); + } + + { + uint magic = MEM_readLE32(dict); + if (magic != 0xEC30A437) + { + return ZSTD_refDictContent(dctx, dict, dictSize); + } + } + + dctx->dictID = MEM_readLE32((sbyte*)dict + 4); + { + nuint eSize = ZSTD_loadDEntropy(&dctx->entropy, dict, dictSize); + if (ERR_isError(eSize)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + + dict = (sbyte*)dict + eSize; + dictSize -= eSize; + } + + dctx->litEntropy = dctx->fseEntropy = 1; + return ZSTD_refDictContent(dctx, dict, dictSize); + } + + public static nuint ZSTD_decompressBegin(ZSTD_DCtx_s* dctx) + { + assert(dctx != null); + dctx->expected = ZSTD_startingInputLength(dctx->format); + dctx->stage = ZSTD_dStage.ZSTDds_getFrameHeaderSize; + dctx->processedCSize = 0; + dctx->decodedSize = 0; + dctx->previousDstEnd = null; + dctx->prefixStart = null; + dctx->virtualStart = null; + dctx->dictEnd = null; + dctx->entropy.hufTable[0] = 12 * 0x1000001; + dctx->litEntropy = dctx->fseEntropy = 0; + dctx->dictID = 0; + dctx->bType = blockType_e.bt_reserved; + dctx->isFrameDecompression = 1; + memcpy(dctx->entropy.rep, repStartValue, sizeof(uint) * 3); + dctx->LLTptr = &dctx->entropy.LLTable.e0; + dctx->MLTptr = &dctx->entropy.MLTable.e0; + dctx->OFTptr = &dctx->entropy.OFTable.e0; + dctx->HUFptr = dctx->entropy.hufTable; + return 0; + } + + public static nuint ZSTD_decompressBegin_usingDict( + ZSTD_DCtx_s* dctx, + void* dict, + nuint dictSize + ) + { + { + nuint err_code = ZSTD_decompressBegin(dctx); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (dict != null && dictSize != 0) + { + if (ERR_isError(ZSTD_decompress_insertDictionary(dctx, dict, dictSize))) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted)); + } + } + + return 0; + } + + /* ====== ZSTD_DDict ====== */ + public static nuint ZSTD_decompressBegin_usingDDict(ZSTD_DCtx_s* dctx, ZSTD_DDict_s* ddict) + { + assert(dctx != null); + if (ddict != null) + { + sbyte* dictStart = (sbyte*)ZSTD_DDict_dictContent(ddict); + nuint dictSize = ZSTD_DDict_dictSize(ddict); + void* dictEnd = dictStart + dictSize; + dctx->ddictIsCold = dctx->dictEnd != dictEnd ? 1 : 0; + } + + { + nuint err_code = ZSTD_decompressBegin(dctx); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (ddict != null) + { + ZSTD_copyDDictParameters(dctx, ddict); + } + + return 0; + } + + /*! ZSTD_getDictID_fromDict() : + * Provides the dictID stored within dictionary. + * if @return == 0, the dictionary is not conformant with Zstandard specification. + * It can still be loaded, but as a content-only dictionary. */ + public static uint ZSTD_getDictID_fromDict(void* dict, nuint dictSize) + { + if (dictSize < 8) + { + return 0; + } + + if (MEM_readLE32(dict) != 0xEC30A437) + { + return 0; + } + + return MEM_readLE32((sbyte*)dict + 4); + } + + /*! ZSTD_getDictID_fromFrame() : + * Provides the dictID required to decompress frame stored within `src`. + * If @return == 0, the dictID could not be decoded. + * This could for one of the following reasons : + * - The frame does not require a dictionary (most common case). + * - The frame was built with dictID intentionally removed. + * Needed dictionary is a hidden piece of information. + * Note : this use case also happens when using a non-conformant dictionary. + * - `srcSize` is too small, and as a result, frame header could not be decoded. + * Note : possible if `srcSize < ZSTD_FRAMEHEADERSIZE_MAX`. + * - This is not a Zstandard frame. + * When identifying the exact failure cause, it's possible to use + * ZSTD_getFrameHeader(), which will provide a more precise error code. */ + public static uint ZSTD_getDictID_fromFrame(void* src, nuint srcSize) + { + ZSTD_frameHeader zfp = new ZSTD_frameHeader + { + frameContentSize = 0, + windowSize = 0, + blockSizeMax = 0, + frameType = ZSTD_frameType_e.ZSTD_frame, + headerSize = 0, + dictID = 0, + checksumFlag = 0, + _reserved1 = 0, + _reserved2 = 0, + }; + nuint hError = ZSTD_getFrameHeader(&zfp, src, srcSize); + if (ERR_isError(hError)) + { + return 0; + } + + return zfp.dictID; + } + + /*! ZSTD_decompress_usingDDict() : + * Decompression using a pre-digested Dictionary + * Use dictionary without significant overhead. */ + public static nuint ZSTD_decompress_usingDDict( + ZSTD_DCtx_s* dctx, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize, + ZSTD_DDict_s* ddict + ) + { + return ZSTD_decompressMultiFrame(dctx, dst, dstCapacity, src, srcSize, null, 0, ddict); + } + + /*===================================== + * Streaming decompression + *====================================*/ + public static ZSTD_DCtx_s* ZSTD_createDStream() + { + return ZSTD_createDCtx_internal(ZSTD_defaultCMem); + } + + public static ZSTD_DCtx_s* ZSTD_initStaticDStream(void* workspace, nuint workspaceSize) + { + return ZSTD_initStaticDCtx(workspace, workspaceSize); + } + + public static ZSTD_DCtx_s* ZSTD_createDStream_advanced(ZSTD_customMem customMem) + { + return ZSTD_createDCtx_internal(customMem); + } + + public static nuint ZSTD_freeDStream(ZSTD_DCtx_s* zds) + { + return ZSTD_freeDCtx(zds); + } + + /* *** Initialization *** */ + public static nuint ZSTD_DStreamInSize() + { + return (nuint)(1 << 17) + ZSTD_blockHeaderSize; + } + + public static nuint ZSTD_DStreamOutSize() + { + return 1 << 17; + } + + /*! ZSTD_DCtx_loadDictionary_advanced() : + * Same as ZSTD_DCtx_loadDictionary(), + * but gives direct control over + * how to load the dictionary (by copy ? by reference ?) + * and how to interpret it (automatic ? force raw mode ? full mode only ?). */ + public static nuint ZSTD_DCtx_loadDictionary_advanced( + ZSTD_DCtx_s* dctx, + void* dict, + nuint dictSize, + ZSTD_dictLoadMethod_e dictLoadMethod, + ZSTD_dictContentType_e dictContentType + ) + { + if (dctx->streamStage != ZSTD_dStreamStage.zdss_init) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_stage_wrong)); + } + + ZSTD_clearDict(dctx); + if (dict != null && dictSize != 0) + { + dctx->ddictLocal = ZSTD_createDDict_advanced( + dict, + dictSize, + dictLoadMethod, + dictContentType, + dctx->customMem + ); + if (dctx->ddictLocal == null) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + + dctx->ddict = dctx->ddictLocal; + dctx->dictUses = ZSTD_dictUses_e.ZSTD_use_indefinitely; + } + + return 0; + } + + /*! ZSTD_DCtx_loadDictionary_byReference() : + * Same as ZSTD_DCtx_loadDictionary(), + * but references `dict` content instead of copying it into `dctx`. + * This saves memory if `dict` remains around., + * However, it's imperative that `dict` remains accessible (and unmodified) while being used, so it must outlive decompression. */ + public static nuint ZSTD_DCtx_loadDictionary_byReference( + ZSTD_DCtx_s* dctx, + void* dict, + nuint dictSize + ) + { + return ZSTD_DCtx_loadDictionary_advanced( + dctx, + dict, + dictSize, + ZSTD_dictLoadMethod_e.ZSTD_dlm_byRef, + ZSTD_dictContentType_e.ZSTD_dct_auto + ); + } + + /*! ZSTD_DCtx_loadDictionary() : Requires v1.4.0+ + * Create an internal DDict from dict buffer, to be used to decompress all future frames. + * The dictionary remains valid for all future frames, until explicitly invalidated, or + * a new dictionary is loaded. + * @result : 0, or an error code (which can be tested with ZSTD_isError()). + * Special : Adding a NULL (or 0-size) dictionary invalidates any previous dictionary, + * meaning "return to no-dictionary mode". + * Note 1 : Loading a dictionary involves building tables, + * which has a non-negligible impact on CPU usage and latency. + * It's recommended to "load once, use many times", to amortize the cost + * Note 2 :`dict` content will be copied internally, so `dict` can be released after loading. + * Use ZSTD_DCtx_loadDictionary_byReference() to reference dictionary content instead. + * Note 3 : Use ZSTD_DCtx_loadDictionary_advanced() to take control of + * how dictionary content is loaded and interpreted. + */ + public static nuint ZSTD_DCtx_loadDictionary(ZSTD_DCtx_s* dctx, void* dict, nuint dictSize) + { + return ZSTD_DCtx_loadDictionary_advanced( + dctx, + dict, + dictSize, + ZSTD_dictLoadMethod_e.ZSTD_dlm_byCopy, + ZSTD_dictContentType_e.ZSTD_dct_auto + ); + } + + /*! ZSTD_DCtx_refPrefix_advanced() : + * Same as ZSTD_DCtx_refPrefix(), but gives finer control over + * how to interpret prefix content (automatic ? force raw mode (default) ? full mode only ?) */ + public static nuint ZSTD_DCtx_refPrefix_advanced( + ZSTD_DCtx_s* dctx, + void* prefix, + nuint prefixSize, + ZSTD_dictContentType_e dictContentType + ) + { + { + nuint err_code = ZSTD_DCtx_loadDictionary_advanced( + dctx, + prefix, + prefixSize, + ZSTD_dictLoadMethod_e.ZSTD_dlm_byRef, + dictContentType + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + dctx->dictUses = ZSTD_dictUses_e.ZSTD_use_once; + return 0; + } + + /*! ZSTD_DCtx_refPrefix() : Requires v1.4.0+ + * Reference a prefix (single-usage dictionary) to decompress next frame. + * This is the reverse operation of ZSTD_CCtx_refPrefix(), + * and must use the same prefix as the one used during compression. + * Prefix is **only used once**. Reference is discarded at end of frame. + * End of frame is reached when ZSTD_decompressStream() returns 0. + * @result : 0, or an error code (which can be tested with ZSTD_isError()). + * Note 1 : Adding any prefix (including NULL) invalidates any previously set prefix or dictionary + * Note 2 : Prefix buffer is referenced. It **must** outlive decompression. + * Prefix buffer must remain unmodified up to the end of frame, + * reached when ZSTD_decompressStream() returns 0. + * Note 3 : By default, the prefix is treated as raw content (ZSTD_dct_rawContent). + * Use ZSTD_CCtx_refPrefix_advanced() to alter dictMode (Experimental section) + * Note 4 : Referencing a raw content prefix has almost no cpu nor memory cost. + * A full dictionary is more costly, as it requires building tables. + */ + public static nuint ZSTD_DCtx_refPrefix(ZSTD_DCtx_s* dctx, void* prefix, nuint prefixSize) + { + return ZSTD_DCtx_refPrefix_advanced( + dctx, + prefix, + prefixSize, + ZSTD_dictContentType_e.ZSTD_dct_rawContent + ); + } + + /* ZSTD_initDStream_usingDict() : + * return : expected size, aka ZSTD_startingInputLength(). + * this function cannot fail */ + public static nuint ZSTD_initDStream_usingDict(ZSTD_DCtx_s* zds, void* dict, nuint dictSize) + { + { + nuint err_code = ZSTD_DCtx_reset(zds, ZSTD_ResetDirective.ZSTD_reset_session_only); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_DCtx_loadDictionary(zds, dict, dictSize); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return ZSTD_startingInputLength(zds->format); + } + + /* note : this variant can't fail */ + public static nuint ZSTD_initDStream(ZSTD_DCtx_s* zds) + { + { + nuint err_code = ZSTD_DCtx_reset(zds, ZSTD_ResetDirective.ZSTD_reset_session_only); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_DCtx_refDDict(zds, null); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return ZSTD_startingInputLength(zds->format); + } + + /* ZSTD_initDStream_usingDDict() : + * ddict will just be referenced, and must outlive decompression session + * this function cannot fail */ + public static nuint ZSTD_initDStream_usingDDict(ZSTD_DCtx_s* dctx, ZSTD_DDict_s* ddict) + { + { + nuint err_code = ZSTD_DCtx_reset(dctx, ZSTD_ResetDirective.ZSTD_reset_session_only); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + { + nuint err_code = ZSTD_DCtx_refDDict(dctx, ddict); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return ZSTD_startingInputLength(dctx->format); + } + + /* ZSTD_resetDStream() : + * return : expected size, aka ZSTD_startingInputLength(). + * this function cannot fail */ + public static nuint ZSTD_resetDStream(ZSTD_DCtx_s* dctx) + { + { + nuint err_code = ZSTD_DCtx_reset(dctx, ZSTD_ResetDirective.ZSTD_reset_session_only); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return ZSTD_startingInputLength(dctx->format); + } + + /*! ZSTD_DCtx_refDDict() : Requires v1.4.0+ + * Reference a prepared dictionary, to be used to decompress next frames. + * The dictionary remains active for decompression of future frames using same DCtx. + * + * If called with ZSTD_d_refMultipleDDicts enabled, repeated calls of this function + * will store the DDict references in a table, and the DDict used for decompression + * will be determined at decompression time, as per the dict ID in the frame. + * The memory for the table is allocated on the first call to refDDict, and can be + * freed with ZSTD_freeDCtx(). + * + * If called with ZSTD_d_refMultipleDDicts disabled (the default), only one dictionary + * will be managed, and referencing a dictionary effectively "discards" any previous one. + * + * @result : 0, or an error code (which can be tested with ZSTD_isError()). + * Special: referencing a NULL DDict means "return to no-dictionary mode". + * Note 2 : DDict is just referenced, its lifetime must outlive its usage from DCtx. + */ + public static nuint ZSTD_DCtx_refDDict(ZSTD_DCtx_s* dctx, ZSTD_DDict_s* ddict) + { + if (dctx->streamStage != ZSTD_dStreamStage.zdss_init) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_stage_wrong)); + } + + ZSTD_clearDict(dctx); + if (ddict != null) + { + dctx->ddict = ddict; + dctx->dictUses = ZSTD_dictUses_e.ZSTD_use_indefinitely; + if (dctx->refMultipleDDicts == ZSTD_refMultipleDDicts_e.ZSTD_rmd_refMultipleDDicts) + { + if (dctx->ddictSet == null) + { + dctx->ddictSet = ZSTD_createDDictHashSet(dctx->customMem); + if (dctx->ddictSet == null) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation) + ); + } + } + + assert(dctx->staticSize == 0); + { + nuint err_code = ZSTD_DDictHashSet_addDDict( + dctx->ddictSet, + ddict, + dctx->customMem + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + } + } + + return 0; + } + + /* ZSTD_DCtx_setMaxWindowSize() : + * note : no direct equivalence in ZSTD_DCtx_setParameter, + * since this version sets windowSize, and the other sets windowLog */ + public static nuint ZSTD_DCtx_setMaxWindowSize(ZSTD_DCtx_s* dctx, nuint maxWindowSize) + { + ZSTD_bounds bounds = ZSTD_dParam_getBounds(ZSTD_dParameter.ZSTD_d_windowLogMax); + nuint min = (nuint)1 << bounds.lowerBound; + nuint max = (nuint)1 << bounds.upperBound; + if (dctx->streamStage != ZSTD_dStreamStage.zdss_init) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_stage_wrong)); + } + + if (maxWindowSize < min) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + if (maxWindowSize > max) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound)); + } + + dctx->maxWindowSize = maxWindowSize; + return 0; + } + + /*! ZSTD_DCtx_setFormat() : + * This function is REDUNDANT. Prefer ZSTD_DCtx_setParameter(). + * Instruct the decoder context about what kind of data to decode next. + * This instruction is mandatory to decode data without a fully-formed header, + * such ZSTD_f_zstd1_magicless for example. + * @return : 0, or an error code (which can be tested using ZSTD_isError()). */ + public static nuint ZSTD_DCtx_setFormat(ZSTD_DCtx_s* dctx, ZSTD_format_e format) + { + return ZSTD_DCtx_setParameter(dctx, ZSTD_dParameter.ZSTD_d_experimentalParam1, (int)format); + } + + /*! ZSTD_dParam_getBounds() : + * All parameters must belong to an interval with lower and upper bounds, + * otherwise they will either trigger an error or be automatically clamped. + * @return : a structure, ZSTD_bounds, which contains + * - an error status field, which must be tested using ZSTD_isError() + * - both lower and upper bounds, inclusive + */ + public static ZSTD_bounds ZSTD_dParam_getBounds(ZSTD_dParameter dParam) + { + ZSTD_bounds bounds = new ZSTD_bounds + { + error = 0, + lowerBound = 0, + upperBound = 0, + }; + switch (dParam) + { + case ZSTD_dParameter.ZSTD_d_windowLogMax: + bounds.lowerBound = 10; + bounds.upperBound = sizeof(nuint) == 4 ? 30 : 31; + return bounds; + case ZSTD_dParameter.ZSTD_d_experimentalParam1: + bounds.lowerBound = (int)ZSTD_format_e.ZSTD_f_zstd1; + bounds.upperBound = (int)ZSTD_format_e.ZSTD_f_zstd1_magicless; + return bounds; + case ZSTD_dParameter.ZSTD_d_experimentalParam2: + bounds.lowerBound = (int)ZSTD_bufferMode_e.ZSTD_bm_buffered; + bounds.upperBound = (int)ZSTD_bufferMode_e.ZSTD_bm_stable; + return bounds; + case ZSTD_dParameter.ZSTD_d_experimentalParam3: + bounds.lowerBound = (int)ZSTD_forceIgnoreChecksum_e.ZSTD_d_validateChecksum; + bounds.upperBound = (int)ZSTD_forceIgnoreChecksum_e.ZSTD_d_ignoreChecksum; + return bounds; + case ZSTD_dParameter.ZSTD_d_experimentalParam4: + bounds.lowerBound = (int)ZSTD_refMultipleDDicts_e.ZSTD_rmd_refSingleDDict; + bounds.upperBound = (int)ZSTD_refMultipleDDicts_e.ZSTD_rmd_refMultipleDDicts; + return bounds; + case ZSTD_dParameter.ZSTD_d_experimentalParam5: + bounds.lowerBound = 0; + bounds.upperBound = 1; + return bounds; + case ZSTD_dParameter.ZSTD_d_experimentalParam6: + bounds.lowerBound = 1 << 10; + bounds.upperBound = 1 << 17; + return bounds; + default: + break; + } + + bounds.error = unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_unsupported)); + return bounds; + } + + /* ZSTD_dParam_withinBounds: + * @return 1 if value is within dParam bounds, + * 0 otherwise */ + private static int ZSTD_dParam_withinBounds(ZSTD_dParameter dParam, int value) + { + ZSTD_bounds bounds = ZSTD_dParam_getBounds(dParam); + if (ERR_isError(bounds.error)) + { + return 0; + } + + if (value < bounds.lowerBound) + { + return 0; + } + + if (value > bounds.upperBound) + { + return 0; + } + + return 1; + } + + /*! ZSTD_DCtx_getParameter() : + * Get the requested decompression parameter value, selected by enum ZSTD_dParameter, + * and store it into int* value. + * @return : 0, or an error code (which can be tested with ZSTD_isError()). + */ + public static nuint ZSTD_DCtx_getParameter(ZSTD_DCtx_s* dctx, ZSTD_dParameter param, int* value) + { + switch (param) + { + case ZSTD_dParameter.ZSTD_d_windowLogMax: + *value = (int)ZSTD_highbit32((uint)dctx->maxWindowSize); + return 0; + case ZSTD_dParameter.ZSTD_d_experimentalParam1: + *value = (int)dctx->format; + return 0; + case ZSTD_dParameter.ZSTD_d_experimentalParam2: + *value = (int)dctx->outBufferMode; + return 0; + case ZSTD_dParameter.ZSTD_d_experimentalParam3: + *value = (int)dctx->forceIgnoreChecksum; + return 0; + case ZSTD_dParameter.ZSTD_d_experimentalParam4: + *value = (int)dctx->refMultipleDDicts; + return 0; + case ZSTD_dParameter.ZSTD_d_experimentalParam5: + *value = dctx->disableHufAsm; + return 0; + case ZSTD_dParameter.ZSTD_d_experimentalParam6: + *value = dctx->maxBlockSizeParam; + return 0; + default: + break; + } + + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_unsupported)); + } + + /*! ZSTD_DCtx_setParameter() : + * Set one compression parameter, selected by enum ZSTD_dParameter. + * All parameters have valid bounds. Bounds can be queried using ZSTD_dParam_getBounds(). + * Providing a value beyond bound will either clamp it, or trigger an error (depending on parameter). + * Setting a parameter is only possible during frame initialization (before starting decompression). + * @return : 0, or an error code (which can be tested using ZSTD_isError()). + */ + public static nuint ZSTD_DCtx_setParameter(ZSTD_DCtx_s* dctx, ZSTD_dParameter dParam, int value) + { + if (dctx->streamStage != ZSTD_dStreamStage.zdss_init) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_stage_wrong)); + } + + switch (dParam) + { + case ZSTD_dParameter.ZSTD_d_windowLogMax: + if (value == 0) + { + value = 27; + } + + { + if (ZSTD_dParam_withinBounds(ZSTD_dParameter.ZSTD_d_windowLogMax, value) == 0) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound) + ); + } + } + + dctx->maxWindowSize = (nuint)1 << value; + return 0; + case ZSTD_dParameter.ZSTD_d_experimentalParam1: + { + if ( + ZSTD_dParam_withinBounds(ZSTD_dParameter.ZSTD_d_experimentalParam1, value) + == 0 + ) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound) + ); + } + } + + dctx->format = (ZSTD_format_e)value; + return 0; + case ZSTD_dParameter.ZSTD_d_experimentalParam2: + { + if ( + ZSTD_dParam_withinBounds(ZSTD_dParameter.ZSTD_d_experimentalParam2, value) + == 0 + ) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound) + ); + } + } + + dctx->outBufferMode = (ZSTD_bufferMode_e)value; + return 0; + case ZSTD_dParameter.ZSTD_d_experimentalParam3: + { + if ( + ZSTD_dParam_withinBounds(ZSTD_dParameter.ZSTD_d_experimentalParam3, value) + == 0 + ) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound) + ); + } + } + + dctx->forceIgnoreChecksum = (ZSTD_forceIgnoreChecksum_e)value; + return 0; + case ZSTD_dParameter.ZSTD_d_experimentalParam4: + { + if ( + ZSTD_dParam_withinBounds(ZSTD_dParameter.ZSTD_d_experimentalParam4, value) + == 0 + ) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound) + ); + } + } + + if (dctx->staticSize != 0) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_unsupported) + ); + } + + dctx->refMultipleDDicts = (ZSTD_refMultipleDDicts_e)value; + return 0; + case ZSTD_dParameter.ZSTD_d_experimentalParam5: + { + if ( + ZSTD_dParam_withinBounds(ZSTD_dParameter.ZSTD_d_experimentalParam5, value) + == 0 + ) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound) + ); + } + } + + dctx->disableHufAsm = value != 0 ? 1 : 0; + return 0; + case ZSTD_dParameter.ZSTD_d_experimentalParam6: + if (value != 0) + { + if ( + ZSTD_dParam_withinBounds(ZSTD_dParameter.ZSTD_d_experimentalParam6, value) + == 0 + ) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_outOfBound) + ); + } + } + + dctx->maxBlockSizeParam = value; + return 0; + default: + break; + } + + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_parameter_unsupported)); + } + + /*! ZSTD_DCtx_reset() : + * Return a DCtx to clean state. + * Session and parameters can be reset jointly or separately. + * Parameters can only be reset when no active frame is being decompressed. + * @return : 0, or an error code, which can be tested with ZSTD_isError() + */ + public static nuint ZSTD_DCtx_reset(ZSTD_DCtx_s* dctx, ZSTD_ResetDirective reset) + { + if ( + reset == ZSTD_ResetDirective.ZSTD_reset_session_only + || reset == ZSTD_ResetDirective.ZSTD_reset_session_and_parameters + ) + { + dctx->streamStage = ZSTD_dStreamStage.zdss_init; + dctx->noForwardProgress = 0; + dctx->isFrameDecompression = 1; + } + + if ( + reset == ZSTD_ResetDirective.ZSTD_reset_parameters + || reset == ZSTD_ResetDirective.ZSTD_reset_session_and_parameters + ) + { + if (dctx->streamStage != ZSTD_dStreamStage.zdss_init) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_stage_wrong)); + } + + ZSTD_clearDict(dctx); + ZSTD_DCtx_resetParameters(dctx); + } + + return 0; + } + + public static nuint ZSTD_sizeof_DStream(ZSTD_DCtx_s* dctx) + { + return ZSTD_sizeof_DCtx(dctx); + } + + private static nuint ZSTD_decodingBufferSize_internal( + ulong windowSize, + ulong frameContentSize, + nuint blockSizeMax + ) + { + nuint blockSize = + (nuint)(windowSize < 1 << 17 ? windowSize : 1 << 17) < blockSizeMax + ? (nuint)(windowSize < 1 << 17 ? windowSize : 1 << 17) + : blockSizeMax; + /* We need blockSize + WILDCOPY_OVERLENGTH worth of buffer so that if a block + * ends at windowSize + WILDCOPY_OVERLENGTH + 1 bytes, we can start writing + * the block at the beginning of the output buffer, and maintain a full window. + * + * We need another blockSize worth of buffer so that we can store split + * literals at the end of the block without overwriting the extDict window. + */ + ulong neededRBSize = windowSize + blockSize * 2 + 32 * 2; + ulong neededSize = frameContentSize < neededRBSize ? frameContentSize : neededRBSize; + nuint minRBSize = (nuint)neededSize; + if (minRBSize != neededSize) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_frameParameter_windowTooLarge) + ); + } + + return minRBSize; + } + + /*===== Buffer-less streaming decompression functions =====*/ + public static nuint ZSTD_decodingBufferSize_min(ulong windowSize, ulong frameContentSize) + { + return ZSTD_decodingBufferSize_internal(windowSize, frameContentSize, 1 << 17); + } + + public static nuint ZSTD_estimateDStreamSize(nuint windowSize) + { + nuint blockSize = windowSize < 1 << 17 ? windowSize : 1 << 17; + /* no block can be larger */ + nuint inBuffSize = blockSize; + nuint outBuffSize = ZSTD_decodingBufferSize_min(windowSize, unchecked(0UL - 1)); + return ZSTD_estimateDCtxSize() + inBuffSize + outBuffSize; + } + + public static nuint ZSTD_estimateDStreamSize_fromFrame(void* src, nuint srcSize) + { + /* note : should be user-selectable, but requires an additional parameter (or a dctx) */ + uint windowSizeMax = 1U << (sizeof(nuint) == 4 ? 30 : 31); + ZSTD_frameHeader zfh; + nuint err = ZSTD_getFrameHeader(&zfh, src, srcSize); + if (ERR_isError(err)) + { + return err; + } + + if (err > 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + if (zfh.windowSize > windowSizeMax) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_frameParameter_windowTooLarge) + ); + } + + return ZSTD_estimateDStreamSize((nuint)zfh.windowSize); + } + + /* ***** Decompression ***** */ + private static int ZSTD_DCtx_isOverflow( + ZSTD_DCtx_s* zds, + nuint neededInBuffSize, + nuint neededOutBuffSize + ) + { + return zds->inBuffSize + zds->outBuffSize >= (neededInBuffSize + neededOutBuffSize) * 3 + ? 1 + : 0; + } + + private static void ZSTD_DCtx_updateOversizedDuration( + ZSTD_DCtx_s* zds, + nuint neededInBuffSize, + nuint neededOutBuffSize + ) + { + if (ZSTD_DCtx_isOverflow(zds, neededInBuffSize, neededOutBuffSize) != 0) + { + zds->oversizedDuration++; + } + else + { + zds->oversizedDuration = 0; + } + } + + private static int ZSTD_DCtx_isOversizedTooLong(ZSTD_DCtx_s* zds) + { + return zds->oversizedDuration >= 128 ? 1 : 0; + } + + /* Checks that the output buffer hasn't changed if ZSTD_obm_stable is used. */ + private static nuint ZSTD_checkOutBuffer(ZSTD_DCtx_s* zds, ZSTD_outBuffer_s* output) + { + ZSTD_outBuffer_s expect = zds->expectedOutBuffer; + if (zds->outBufferMode != ZSTD_bufferMode_e.ZSTD_bm_stable) + { + return 0; + } + + if (zds->streamStage == ZSTD_dStreamStage.zdss_init) + { + return 0; + } + + if (expect.dst == output->dst && expect.pos == output->pos && expect.size == output->size) + { + return 0; + } + + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstBuffer_wrong)); + } + + /* Calls ZSTD_decompressContinue() with the right parameters for ZSTD_decompressStream() + * and updates the stage and the output buffer state. This call is extracted so it can be + * used both when reading directly from the ZSTD_inBuffer, and in buffered input mode. + * NOTE: You must break after calling this function since the streamStage is modified. + */ + private static nuint ZSTD_decompressContinueStream( + ZSTD_DCtx_s* zds, + sbyte** op, + sbyte* oend, + void* src, + nuint srcSize + ) + { + int isSkipFrame = ZSTD_isSkipFrame(zds); + if (zds->outBufferMode == ZSTD_bufferMode_e.ZSTD_bm_buffered) + { + nuint dstSize = isSkipFrame != 0 ? 0 : zds->outBuffSize - zds->outStart; + nuint decodedSize = ZSTD_decompressContinue( + zds, + zds->outBuff + zds->outStart, + dstSize, + src, + srcSize + ); + { + nuint err_code = decodedSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (decodedSize == 0 && isSkipFrame == 0) + { + zds->streamStage = ZSTD_dStreamStage.zdss_read; + } + else + { + zds->outEnd = zds->outStart + decodedSize; + zds->streamStage = ZSTD_dStreamStage.zdss_flush; + } + } + else + { + /* Write directly into the output buffer */ + nuint dstSize = isSkipFrame != 0 ? 0 : (nuint)(oend - *op); + nuint decodedSize = ZSTD_decompressContinue(zds, *op, dstSize, src, srcSize); + { + nuint err_code = decodedSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + *op += decodedSize; + zds->streamStage = ZSTD_dStreamStage.zdss_read; + assert(*op <= oend); + assert(zds->outBufferMode == ZSTD_bufferMode_e.ZSTD_bm_stable); + } + + return 0; + } + + /*! ZSTD_decompressStream() : + * Streaming decompression function. + * Call repetitively to consume full input updating it as necessary. + * Function will update both input and output `pos` fields exposing current state via these fields: + * - `input.pos < input.size`, some input remaining and caller should provide remaining input + * on the next call. + * - `output.pos < output.size`, decoder flushed internal output buffer. + * - `output.pos == output.size`, unflushed data potentially present in the internal buffers, + * check ZSTD_decompressStream() @return value, + * if > 0, invoke it again to flush remaining data to output. + * Note : with no additional input, amount of data flushed <= ZSTD_BLOCKSIZE_MAX. + * + * @return : 0 when a frame is completely decoded and fully flushed, + * or an error code, which can be tested using ZSTD_isError(), + * or any other value > 0, which means there is some decoding or flushing to do to complete current frame. + * + * Note: when an operation returns with an error code, the @zds state may be left in undefined state. + * It's UB to invoke `ZSTD_decompressStream()` on such a state. + * In order to re-use such a state, it must be first reset, + * which can be done explicitly (`ZSTD_DCtx_reset()`), + * or is implied for operations starting some new decompression job (`ZSTD_initDStream`, `ZSTD_decompressDCtx()`, `ZSTD_decompress_usingDict()`) + */ + public static nuint ZSTD_decompressStream( + ZSTD_DCtx_s* zds, + ZSTD_outBuffer_s* output, + ZSTD_inBuffer_s* input + ) + { + sbyte* src = (sbyte*)input->src; + sbyte* istart = input->pos != 0 ? src + input->pos : src; + sbyte* iend = input->size != 0 ? src + input->size : src; + sbyte* ip = istart; + sbyte* dst = (sbyte*)output->dst; + sbyte* ostart = output->pos != 0 ? dst + output->pos : dst; + sbyte* oend = output->size != 0 ? dst + output->size : dst; + sbyte* op = ostart; + uint someMoreWork = 1; + assert(zds != null); + if (input->pos > input->size) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + if (output->pos > output->size) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + { + nuint err_code = ZSTD_checkOutBuffer(zds, output); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + while (someMoreWork != 0) + { + switch (zds->streamStage) + { + case ZSTD_dStreamStage.zdss_init: + zds->streamStage = ZSTD_dStreamStage.zdss_loadHeader; + zds->lhSize = zds->inPos = zds->outStart = zds->outEnd = 0; + zds->hostageByte = 0; + zds->expectedOutBuffer = *output; + goto case ZSTD_dStreamStage.zdss_loadHeader; + case ZSTD_dStreamStage.zdss_loadHeader: + { + nuint hSize = ZSTD_getFrameHeader_advanced( + &zds->fParams, + zds->headerBuffer, + zds->lhSize, + zds->format + ); + if (zds->refMultipleDDicts != default && zds->ddictSet != null) + { + ZSTD_DCtx_selectFrameDDict(zds); + } + + if (ERR_isError(hSize)) + { + return hSize; + } + + if (hSize != 0) + { + /* if hSize!=0, hSize > zds->lhSize */ + nuint toLoad = hSize - zds->lhSize; + nuint remainingInput = (nuint)(iend - ip); + assert(iend >= ip); + if (toLoad > remainingInput) + { + if (remainingInput > 0) + { + memcpy( + zds->headerBuffer + zds->lhSize, + ip, + (uint)remainingInput + ); + zds->lhSize += remainingInput; + } + + input->pos = input->size; + { + /* check first few bytes */ + nuint err_code = ZSTD_getFrameHeader_advanced( + &zds->fParams, + zds->headerBuffer, + zds->lhSize, + zds->format + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + return ( + (nuint)(zds->format == ZSTD_format_e.ZSTD_f_zstd1 ? 6 : 2) + > hSize + ? (nuint)( + zds->format == ZSTD_format_e.ZSTD_f_zstd1 ? 6 : 2 + ) + : hSize + ) + - zds->lhSize + + ZSTD_blockHeaderSize; + } + + assert(ip != null); + memcpy(zds->headerBuffer + zds->lhSize, ip, (uint)toLoad); + zds->lhSize = hSize; + ip += toLoad; + break; + } + } + + if ( + zds->fParams.frameContentSize != unchecked(0UL - 1) + && zds->fParams.frameType != ZSTD_frameType_e.ZSTD_skippableFrame + && (nuint)(oend - op) >= zds->fParams.frameContentSize + ) + { + nuint cSize = ZSTD_findFrameCompressedSize_advanced( + istart, + (nuint)(iend - istart), + zds->format + ); + if (cSize <= (nuint)(iend - istart)) + { + /* shortcut : using single-pass mode */ + nuint decompressedSize = ZSTD_decompress_usingDDict( + zds, + op, + (nuint)(oend - op), + istart, + cSize, + ZSTD_getDDict(zds) + ); + if (ERR_isError(decompressedSize)) + { + return decompressedSize; + } + + assert(istart != null); + ip = istart + cSize; + op = op != null ? op + decompressedSize : op; + zds->expected = 0; + zds->streamStage = ZSTD_dStreamStage.zdss_init; + someMoreWork = 0; + break; + } + } + + if ( + zds->outBufferMode == ZSTD_bufferMode_e.ZSTD_bm_stable + && zds->fParams.frameType != ZSTD_frameType_e.ZSTD_skippableFrame + && zds->fParams.frameContentSize != unchecked(0UL - 1) + && (nuint)(oend - op) < zds->fParams.frameContentSize + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + { + nuint err_code = ZSTD_decompressBegin_usingDDict(zds, ZSTD_getDDict(zds)); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if ( + zds->format == ZSTD_format_e.ZSTD_f_zstd1 + && (MEM_readLE32(zds->headerBuffer) & 0xFFFFFFF0) == 0x184D2A50 + ) + { + zds->expected = MEM_readLE32(zds->headerBuffer + 4); + zds->stage = ZSTD_dStage.ZSTDds_skipFrame; + } + else + { + { + nuint err_code = ZSTD_decodeFrameHeader( + zds, + zds->headerBuffer, + zds->lhSize + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + zds->expected = ZSTD_blockHeaderSize; + zds->stage = ZSTD_dStage.ZSTDds_decodeBlockHeader; + } + + zds->fParams.windowSize = + zds->fParams.windowSize > 1U << 10 ? zds->fParams.windowSize : 1U << 10; + if (zds->fParams.windowSize > zds->maxWindowSize) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_frameParameter_windowTooLarge) + ); + } + + if (zds->maxBlockSizeParam != 0) + { + zds->fParams.blockSizeMax = + zds->fParams.blockSizeMax < (uint)zds->maxBlockSizeParam + ? zds->fParams.blockSizeMax + : (uint)zds->maxBlockSizeParam; + } + + { + /* frame checksum */ + nuint neededInBuffSize = + zds->fParams.blockSizeMax > 4 ? zds->fParams.blockSizeMax : 4; + nuint neededOutBuffSize = + zds->outBufferMode == ZSTD_bufferMode_e.ZSTD_bm_buffered + ? ZSTD_decodingBufferSize_internal( + zds->fParams.windowSize, + zds->fParams.frameContentSize, + zds->fParams.blockSizeMax + ) + : 0; + ZSTD_DCtx_updateOversizedDuration(zds, neededInBuffSize, neededOutBuffSize); + { + int tooSmall = + zds->inBuffSize < neededInBuffSize + || zds->outBuffSize < neededOutBuffSize + ? 1 + : 0; + int tooLarge = ZSTD_DCtx_isOversizedTooLong(zds); + if (tooSmall != 0 || tooLarge != 0) + { + nuint bufferSize = neededInBuffSize + neededOutBuffSize; + if (zds->staticSize != 0) + { + assert(zds->staticSize >= (nuint)sizeof(ZSTD_DCtx_s)); + if (bufferSize > zds->staticSize - (nuint)sizeof(ZSTD_DCtx_s)) + { + return unchecked( + (nuint)( + -(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation + ) + ); + } + } + else + { + ZSTD_customFree(zds->inBuff, zds->customMem); + zds->inBuffSize = 0; + zds->outBuffSize = 0; + zds->inBuff = (sbyte*)ZSTD_customMalloc( + bufferSize, + zds->customMem + ); + if (zds->inBuff == null) + { + return unchecked( + (nuint)( + -(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation + ) + ); + } + } + + zds->inBuffSize = neededInBuffSize; + zds->outBuff = zds->inBuff + zds->inBuffSize; + zds->outBuffSize = neededOutBuffSize; + } + } + } + + zds->streamStage = ZSTD_dStreamStage.zdss_read; + goto case ZSTD_dStreamStage.zdss_read; + case ZSTD_dStreamStage.zdss_read: + { + nuint neededInSize = ZSTD_nextSrcSizeToDecompressWithInputSize( + zds, + (nuint)(iend - ip) + ); + if (neededInSize == 0) + { + zds->streamStage = ZSTD_dStreamStage.zdss_init; + someMoreWork = 0; + break; + } + + if ((nuint)(iend - ip) >= neededInSize) + { + { + nuint err_code = ZSTD_decompressContinueStream( + zds, + &op, + oend, + ip, + neededInSize + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + assert(ip != null); + ip += neededInSize; + break; + } + } + + if (ip == iend) + { + someMoreWork = 0; + break; + } + + zds->streamStage = ZSTD_dStreamStage.zdss_load; + goto case ZSTD_dStreamStage.zdss_load; + case ZSTD_dStreamStage.zdss_load: + { + nuint neededInSize = ZSTD_nextSrcSizeToDecompress(zds); + nuint toLoad = neededInSize - zds->inPos; + int isSkipFrame = ZSTD_isSkipFrame(zds); + nuint loadedSize; + assert( + neededInSize + == ZSTD_nextSrcSizeToDecompressWithInputSize(zds, (nuint)(iend - ip)) + ); + if (isSkipFrame != 0) + { + loadedSize = toLoad < (nuint)(iend - ip) ? toLoad : (nuint)(iend - ip); + } + else + { + if (toLoad > zds->inBuffSize - zds->inPos) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected) + ); + } + + loadedSize = ZSTD_limitCopy( + zds->inBuff + zds->inPos, + toLoad, + ip, + (nuint)(iend - ip) + ); + } + + if (loadedSize != 0) + { + ip += loadedSize; + zds->inPos += loadedSize; + } + + if (loadedSize < toLoad) + { + someMoreWork = 0; + break; + } + + zds->inPos = 0; + { + nuint err_code = ZSTD_decompressContinueStream( + zds, + &op, + oend, + zds->inBuff, + neededInSize + ); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + break; + } + + case ZSTD_dStreamStage.zdss_flush: + { + nuint toFlushSize = zds->outEnd - zds->outStart; + nuint flushedSize = ZSTD_limitCopy( + op, + (nuint)(oend - op), + zds->outBuff + zds->outStart, + toFlushSize + ); + op = op != null ? op + flushedSize : op; + zds->outStart += flushedSize; + if (flushedSize == toFlushSize) + { + zds->streamStage = ZSTD_dStreamStage.zdss_read; + if ( + zds->outBuffSize < zds->fParams.frameContentSize + && zds->outStart + zds->fParams.blockSizeMax > zds->outBuffSize + ) + { + zds->outStart = zds->outEnd = 0; + } + + break; + } + } + + someMoreWork = 0; + break; + default: + assert(0 != 0); + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + } + + input->pos = (nuint)(ip - (sbyte*)input->src); + output->pos = (nuint)(op - (sbyte*)output->dst); + zds->expectedOutBuffer = *output; + if (ip == istart && op == ostart) + { + zds->noForwardProgress++; + if (zds->noForwardProgress >= 16) + { + if (op == oend) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_noForwardProgress_destFull) + ); + } + + if (ip == iend) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_noForwardProgress_inputEmpty) + ); + } + + assert(0 != 0); + } + } + else + { + zds->noForwardProgress = 0; + } + + { + nuint nextSrcSizeHint = ZSTD_nextSrcSizeToDecompress(zds); + if (nextSrcSizeHint == 0) + { + if (zds->outEnd == zds->outStart) + { + if (zds->hostageByte != 0) + { + if (input->pos >= input->size) + { + zds->streamStage = ZSTD_dStreamStage.zdss_read; + return 1; + } + + input->pos++; + } + + return 0; + } + + if (zds->hostageByte == 0) + { + input->pos--; + zds->hostageByte = 1; + } + + return 1; + } + + nextSrcSizeHint += + ZSTD_blockHeaderSize + * (nuint)(ZSTD_nextInputType(zds) == ZSTD_nextInputType_e.ZSTDnit_block ? 1 : 0); + assert(zds->inPos <= nextSrcSizeHint); + nextSrcSizeHint -= zds->inPos; + return nextSrcSizeHint; + } + } + + /*! ZSTD_decompressStream_simpleArgs() : + * Same as ZSTD_decompressStream(), + * but using only integral types as arguments. + * This can be helpful for binders from dynamic languages + * which have troubles handling structures containing memory pointers. + */ + public static nuint ZSTD_decompressStream_simpleArgs( + ZSTD_DCtx_s* dctx, + void* dst, + nuint dstCapacity, + nuint* dstPos, + void* src, + nuint srcSize, + nuint* srcPos + ) + { + ZSTD_outBuffer_s output; + ZSTD_inBuffer_s input; + output.dst = dst; + output.size = dstCapacity; + output.pos = *dstPos; + input.src = src; + input.size = srcSize; + input.pos = *srcPos; + { + nuint cErr = ZSTD_decompressStream(dctx, &output, &input); + *dstPos = output.pos; + *srcPos = input.pos; + return cErr; + } + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdDecompressBlock.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdDecompressBlock.cs new file mode 100644 index 00000000..297b2332 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdDecompressBlock.cs @@ -0,0 +1,3344 @@ +using System; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + /*_******************************************************* + * Memory operations + **********************************************************/ + private static void ZSTD_copy4(void* dst, void* src) + { + memcpy(dst, src, 4); + } + + /*-************************************************************* + * Block decoding + ***************************************************************/ + private static nuint ZSTD_blockSizeMax(ZSTD_DCtx_s* dctx) + { + nuint blockSizeMax = dctx->isFrameDecompression != 0 ? dctx->fParams.blockSizeMax : 1 << 17; + assert(blockSizeMax <= 1 << 17); + return blockSizeMax; + } + + /*! ZSTD_getcBlockSize() : + * Provides the size of compressed block from block header `src` */ + private static nuint ZSTD_getcBlockSize(void* src, nuint srcSize, blockProperties_t* bpPtr) + { + if (srcSize < ZSTD_blockHeaderSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + { + uint cBlockHeader = MEM_readLE24(src); + uint cSize = cBlockHeader >> 3; + bpPtr->lastBlock = cBlockHeader & 1; + bpPtr->blockType = (blockType_e)(cBlockHeader >> 1 & 3); + bpPtr->origSize = cSize; + if (bpPtr->blockType == blockType_e.bt_rle) + { + return 1; + } + + if (bpPtr->blockType == blockType_e.bt_reserved) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + return cSize; + } + } + + /* Allocate buffer for literals, either overlapping current dst, or split between dst and litExtraBuffer, or stored entirely within litExtraBuffer */ + private static void ZSTD_allocateLiteralsBuffer( + ZSTD_DCtx_s* dctx, + void* dst, + nuint dstCapacity, + nuint litSize, + streaming_operation streaming, + nuint expectedWriteSize, + uint splitImmediately + ) + { + nuint blockSizeMax = ZSTD_blockSizeMax(dctx); + assert(litSize <= blockSizeMax); + assert(dctx->isFrameDecompression != 0 || streaming == streaming_operation.not_streaming); + assert(expectedWriteSize <= blockSizeMax); + if ( + streaming == streaming_operation.not_streaming + && dstCapacity > blockSizeMax + 32 + litSize + 32 + ) + { + dctx->litBuffer = (byte*)dst + blockSizeMax + 32; + dctx->litBufferEnd = dctx->litBuffer + litSize; + dctx->litBufferLocation = ZSTD_litLocation_e.ZSTD_in_dst; + } + else if (litSize <= 1 << 16) + { + dctx->litBuffer = dctx->litExtraBuffer; + dctx->litBufferEnd = dctx->litBuffer + litSize; + dctx->litBufferLocation = ZSTD_litLocation_e.ZSTD_not_in_dst; + } + else + { + assert(blockSizeMax > 1 << 16); + if (splitImmediately != 0) + { + dctx->litBuffer = (byte*)dst + expectedWriteSize - litSize + (1 << 16) - 32; + dctx->litBufferEnd = dctx->litBuffer + litSize - (1 << 16); + } + else + { + dctx->litBuffer = (byte*)dst + expectedWriteSize - litSize; + dctx->litBufferEnd = (byte*)dst + expectedWriteSize; + } + + dctx->litBufferLocation = ZSTD_litLocation_e.ZSTD_split; + assert(dctx->litBufferEnd <= (byte*)dst + expectedWriteSize); + } + } + + /*! ZSTD_decodeLiteralsBlock() : + * Where it is possible to do so without being stomped by the output during decompression, the literals block will be stored + * in the dstBuffer. If there is room to do so, it will be stored in full in the excess dst space after where the current + * block will be output. Otherwise it will be stored at the end of the current dst blockspace, with a small portion being + * stored in dctx->litExtraBuffer to help keep it "ahead" of the current output write. + * + * @return : nb of bytes read from src (< srcSize ) + * note : symbol not declared but exposed for fullbench */ + private static nuint ZSTD_decodeLiteralsBlock( + ZSTD_DCtx_s* dctx, + void* src, + nuint srcSize, + void* dst, + nuint dstCapacity, + streaming_operation streaming + ) + { + if (srcSize < 1 + 1) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + { + byte* istart = (byte*)src; + SymbolEncodingType_e litEncType = (SymbolEncodingType_e)(istart[0] & 3); + nuint blockSizeMax = ZSTD_blockSizeMax(dctx); + switch (litEncType) + { + case SymbolEncodingType_e.set_repeat: + if (dctx->litEntropy == 0) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dictionary_corrupted) + ); + } + + goto case SymbolEncodingType_e.set_compressed; + case SymbolEncodingType_e.set_compressed: + if (srcSize < 5) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected) + ); + } + + { + nuint lhSize, + litSize, + litCSize; + uint singleStream = 0; + uint lhlCode = (uint)(istart[0] >> 2 & 3); + uint lhc = MEM_readLE32(istart); + nuint hufSuccess; + nuint expectedWriteSize = + blockSizeMax < dstCapacity ? blockSizeMax : dstCapacity; + int flags = + 0 + | (ZSTD_DCtx_get_bmi2(dctx) != 0 ? (int)HUF_flags_e.HUF_flags_bmi2 : 0) + | ( + dctx->disableHufAsm != 0 ? (int)HUF_flags_e.HUF_flags_disableAsm : 0 + ); + switch (lhlCode) + { + case 0: + case 1: + default: + singleStream = lhlCode == 0 ? 1U : 0U; + lhSize = 3; + litSize = lhc >> 4 & 0x3FF; + litCSize = lhc >> 14 & 0x3FF; + break; + case 2: + lhSize = 4; + litSize = lhc >> 4 & 0x3FFF; + litCSize = lhc >> 18; + break; + case 3: + lhSize = 5; + litSize = lhc >> 4 & 0x3FFFF; + litCSize = (lhc >> 22) + ((nuint)istart[4] << 10); + break; + } + + if (litSize > 0 && dst == null) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall) + ); + } + + if (litSize > blockSizeMax) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected) + ); + } + + if (singleStream == 0) + { + if (litSize < 6) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_literals_headerWrong) + ); + } + } + + if (litCSize + lhSize > srcSize) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected) + ); + } + + if (expectedWriteSize < litSize) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall) + ); + } + + ZSTD_allocateLiteralsBuffer( + dctx, + dst, + dstCapacity, + litSize, + streaming, + expectedWriteSize, + 0 + ); + if (dctx->ddictIsCold != 0 && litSize > 768) + { + sbyte* _ptr = (sbyte*)dctx->HUFptr; + const nuint _size = sizeof(uint) * 4097; + nuint _pos; + for (_pos = 0; _pos < _size; _pos += 64) + { +#if NETCOREAPP3_0_OR_GREATER + if (System.Runtime.Intrinsics.X86.Sse.IsSupported) + { + System.Runtime.Intrinsics.X86.Sse.Prefetch1(_ptr + _pos); + } +#endif + } + } + + if (litEncType == SymbolEncodingType_e.set_repeat) + { + if (singleStream != 0) + { + hufSuccess = HUF_decompress1X_usingDTable( + dctx->litBuffer, + litSize, + istart + lhSize, + litCSize, + dctx->HUFptr, + flags + ); + } + else + { + assert(litSize >= 6); + hufSuccess = HUF_decompress4X_usingDTable( + dctx->litBuffer, + litSize, + istart + lhSize, + litCSize, + dctx->HUFptr, + flags + ); + } + } + else + { + if (singleStream != 0) + { + hufSuccess = HUF_decompress1X1_DCtx_wksp( + dctx->entropy.hufTable, + dctx->litBuffer, + litSize, + istart + lhSize, + litCSize, + dctx->workspace, + sizeof(uint) * 640, + flags + ); + } + else + { + hufSuccess = HUF_decompress4X_hufOnly_wksp( + dctx->entropy.hufTable, + dctx->litBuffer, + litSize, + istart + lhSize, + litCSize, + dctx->workspace, + sizeof(uint) * 640, + flags + ); + } + } + + if (dctx->litBufferLocation == ZSTD_litLocation_e.ZSTD_split) + { + assert(litSize > 1 << 16); + memcpy(dctx->litExtraBuffer, dctx->litBufferEnd - (1 << 16), 1 << 16); + memmove( + dctx->litBuffer + (1 << 16) - 32, + dctx->litBuffer, + litSize - (1 << 16) + ); + dctx->litBuffer += (1 << 16) - 32; + dctx->litBufferEnd -= 32; + assert(dctx->litBufferEnd <= (byte*)dst + blockSizeMax); + } + + if (ERR_isError(hufSuccess)) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected) + ); + } + + dctx->litPtr = dctx->litBuffer; + dctx->litSize = litSize; + dctx->litEntropy = 1; + if (litEncType == SymbolEncodingType_e.set_compressed) + { + dctx->HUFptr = dctx->entropy.hufTable; + } + + return litCSize + lhSize; + } + + case SymbolEncodingType_e.set_basic: + { + nuint litSize, + lhSize; + uint lhlCode = (uint)(istart[0] >> 2 & 3); + nuint expectedWriteSize = + blockSizeMax < dstCapacity ? blockSizeMax : dstCapacity; + switch (lhlCode) + { + case 0: + case 2: + default: + lhSize = 1; + litSize = (nuint)(istart[0] >> 3); + break; + case 1: + lhSize = 2; + litSize = (nuint)(MEM_readLE16(istart) >> 4); + break; + case 3: + lhSize = 3; + if (srcSize < 3) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected) + ); + } + + litSize = MEM_readLE24(istart) >> 4; + break; + } + + if (litSize > 0 && dst == null) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + if (litSize > blockSizeMax) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected) + ); + } + + if (expectedWriteSize < litSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + ZSTD_allocateLiteralsBuffer( + dctx, + dst, + dstCapacity, + litSize, + streaming, + expectedWriteSize, + 1 + ); + if (lhSize + litSize + 32 > srcSize) + { + if (litSize + lhSize > srcSize) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected) + ); + } + + if (dctx->litBufferLocation == ZSTD_litLocation_e.ZSTD_split) + { + memcpy(dctx->litBuffer, istart + lhSize, (uint)(litSize - (1 << 16))); + memcpy( + dctx->litExtraBuffer, + istart + lhSize + litSize - (1 << 16), + 1 << 16 + ); + } + else + { + memcpy(dctx->litBuffer, istart + lhSize, (uint)litSize); + } + + dctx->litPtr = dctx->litBuffer; + dctx->litSize = litSize; + return lhSize + litSize; + } + + dctx->litPtr = istart + lhSize; + dctx->litSize = litSize; + dctx->litBufferEnd = dctx->litPtr + litSize; + dctx->litBufferLocation = ZSTD_litLocation_e.ZSTD_not_in_dst; + return lhSize + litSize; + } + + case SymbolEncodingType_e.set_rle: + { + uint lhlCode = (uint)(istart[0] >> 2 & 3); + nuint litSize, + lhSize; + nuint expectedWriteSize = + blockSizeMax < dstCapacity ? blockSizeMax : dstCapacity; + switch (lhlCode) + { + case 0: + case 2: + default: + lhSize = 1; + litSize = (nuint)(istart[0] >> 3); + break; + case 1: + lhSize = 2; + if (srcSize < 3) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected) + ); + } + + litSize = (nuint)(MEM_readLE16(istart) >> 4); + break; + case 3: + lhSize = 3; + if (srcSize < 4) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected) + ); + } + + litSize = MEM_readLE24(istart) >> 4; + break; + } + + if (litSize > 0 && dst == null) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + if (litSize > blockSizeMax) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected) + ); + } + + if (expectedWriteSize < litSize) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + ZSTD_allocateLiteralsBuffer( + dctx, + dst, + dstCapacity, + litSize, + streaming, + expectedWriteSize, + 1 + ); + if (dctx->litBufferLocation == ZSTD_litLocation_e.ZSTD_split) + { + memset(dctx->litBuffer, istart[lhSize], (uint)(litSize - (1 << 16))); + memset(dctx->litExtraBuffer, istart[lhSize], 1 << 16); + } + else + { + memset(dctx->litBuffer, istart[lhSize], (uint)litSize); + } + + dctx->litPtr = dctx->litBuffer; + dctx->litSize = litSize; + return lhSize + 1; + } + + default: + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + } + } + + /* Hidden declaration for fullbench */ + private static nuint ZSTD_decodeLiteralsBlock_wrapper( + ZSTD_DCtx_s* dctx, + void* src, + nuint srcSize, + void* dst, + nuint dstCapacity + ) + { + dctx->isFrameDecompression = 0; + return ZSTD_decodeLiteralsBlock( + dctx, + src, + srcSize, + dst, + dstCapacity, + streaming_operation.not_streaming + ); + } + + private static readonly ZSTD_seqSymbol* LL_defaultDTable = GetArrayPointer( + new ZSTD_seqSymbol[65] + { + new ZSTD_seqSymbol(nextState: 1, nbAdditionalBits: 1, nbBits: 1, baseValue: 6), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 4, baseValue: 0), + new ZSTD_seqSymbol(nextState: 16, nbAdditionalBits: 0, nbBits: 4, baseValue: 0), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 0, nbBits: 5, baseValue: 1), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 5, baseValue: 3), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 5, baseValue: 4), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 5, baseValue: 6), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 5, baseValue: 7), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 5, baseValue: 9), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 5, baseValue: 10), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 5, baseValue: 12), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 14), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 1, nbBits: 5, baseValue: 16), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 1, nbBits: 5, baseValue: 20), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 1, nbBits: 5, baseValue: 22), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 2, nbBits: 5, baseValue: 28), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 3, nbBits: 5, baseValue: 32), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 4, nbBits: 5, baseValue: 48), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 6, nbBits: 5, baseValue: 64), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 7, nbBits: 5, baseValue: 128), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 8, nbBits: 6, baseValue: 256), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 10, nbBits: 6, baseValue: 1024), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 12, nbBits: 6, baseValue: 4096), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 0, nbBits: 4, baseValue: 0), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 4, baseValue: 1), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 5, baseValue: 2), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 0, nbBits: 5, baseValue: 4), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 5, baseValue: 5), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 0, nbBits: 5, baseValue: 7), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 5, baseValue: 8), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 0, nbBits: 5, baseValue: 10), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 5, baseValue: 11), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 13), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 1, nbBits: 5, baseValue: 16), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 1, nbBits: 5, baseValue: 18), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 1, nbBits: 5, baseValue: 22), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 2, nbBits: 5, baseValue: 24), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 3, nbBits: 5, baseValue: 32), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 3, nbBits: 5, baseValue: 40), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 6, nbBits: 4, baseValue: 64), + new ZSTD_seqSymbol(nextState: 16, nbAdditionalBits: 6, nbBits: 4, baseValue: 64), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 7, nbBits: 5, baseValue: 128), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 9, nbBits: 6, baseValue: 512), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 11, nbBits: 6, baseValue: 2048), + new ZSTD_seqSymbol(nextState: 48, nbAdditionalBits: 0, nbBits: 4, baseValue: 0), + new ZSTD_seqSymbol(nextState: 16, nbAdditionalBits: 0, nbBits: 4, baseValue: 1), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 0, nbBits: 5, baseValue: 2), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 0, nbBits: 5, baseValue: 3), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 0, nbBits: 5, baseValue: 5), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 0, nbBits: 5, baseValue: 6), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 0, nbBits: 5, baseValue: 8), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 0, nbBits: 5, baseValue: 9), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 0, nbBits: 5, baseValue: 11), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 0, nbBits: 5, baseValue: 12), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 15), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 1, nbBits: 5, baseValue: 18), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 1, nbBits: 5, baseValue: 20), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 2, nbBits: 5, baseValue: 24), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 2, nbBits: 5, baseValue: 28), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 3, nbBits: 5, baseValue: 40), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 4, nbBits: 5, baseValue: 48), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 16, nbBits: 6, baseValue: 65536), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 15, nbBits: 6, baseValue: 32768), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 14, nbBits: 6, baseValue: 16384), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 13, nbBits: 6, baseValue: 8192), + } + ); + private static readonly ZSTD_seqSymbol* OF_defaultDTable = GetArrayPointer( + new ZSTD_seqSymbol[33] + { + new ZSTD_seqSymbol(nextState: 1, nbAdditionalBits: 1, nbBits: 1, baseValue: 5), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 5, baseValue: 0), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 6, nbBits: 4, baseValue: 61), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 9, nbBits: 5, baseValue: 509), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 15, nbBits: 5, baseValue: 32765), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 21, nbBits: 5, baseValue: 2097149), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 3, nbBits: 5, baseValue: 5), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 7, nbBits: 4, baseValue: 125), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 12, nbBits: 5, baseValue: 4093), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 18, nbBits: 5, baseValue: 262141), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 23, nbBits: 5, baseValue: 8388605), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 5, nbBits: 5, baseValue: 29), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 8, nbBits: 4, baseValue: 253), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 14, nbBits: 5, baseValue: 16381), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 20, nbBits: 5, baseValue: 1048573), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 2, nbBits: 5, baseValue: 1), + new ZSTD_seqSymbol(nextState: 16, nbAdditionalBits: 7, nbBits: 4, baseValue: 125), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 11, nbBits: 5, baseValue: 2045), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 17, nbBits: 5, baseValue: 131069), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 22, nbBits: 5, baseValue: 4194301), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 4, nbBits: 5, baseValue: 13), + new ZSTD_seqSymbol(nextState: 16, nbAdditionalBits: 8, nbBits: 4, baseValue: 253), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 13, nbBits: 5, baseValue: 8189), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 19, nbBits: 5, baseValue: 524285), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 1, nbBits: 5, baseValue: 1), + new ZSTD_seqSymbol(nextState: 16, nbAdditionalBits: 6, nbBits: 4, baseValue: 61), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 10, nbBits: 5, baseValue: 1021), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 16, nbBits: 5, baseValue: 65533), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 28, nbBits: 5, baseValue: 268435453), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 27, nbBits: 5, baseValue: 134217725), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 26, nbBits: 5, baseValue: 67108861), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 25, nbBits: 5, baseValue: 33554429), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 24, nbBits: 5, baseValue: 16777213), + } + ); + private static readonly ZSTD_seqSymbol* ML_defaultDTable = GetArrayPointer( + new ZSTD_seqSymbol[65] + { + new ZSTD_seqSymbol(nextState: 1, nbAdditionalBits: 1, nbBits: 1, baseValue: 6), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 3), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 4, baseValue: 4), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 0, nbBits: 5, baseValue: 5), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 5, baseValue: 6), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 5, baseValue: 8), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 5, baseValue: 9), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 5, baseValue: 11), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 13), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 16), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 19), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 22), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 25), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 28), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 31), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 34), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 1, nbBits: 6, baseValue: 37), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 1, nbBits: 6, baseValue: 41), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 2, nbBits: 6, baseValue: 47), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 3, nbBits: 6, baseValue: 59), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 4, nbBits: 6, baseValue: 83), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 7, nbBits: 6, baseValue: 131), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 9, nbBits: 6, baseValue: 515), + new ZSTD_seqSymbol(nextState: 16, nbAdditionalBits: 0, nbBits: 4, baseValue: 4), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 4, baseValue: 5), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 0, nbBits: 5, baseValue: 6), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 5, baseValue: 7), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 0, nbBits: 5, baseValue: 9), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 5, baseValue: 10), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 12), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 15), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 18), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 21), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 24), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 27), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 30), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 33), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 1, nbBits: 6, baseValue: 35), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 1, nbBits: 6, baseValue: 39), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 2, nbBits: 6, baseValue: 43), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 3, nbBits: 6, baseValue: 51), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 4, nbBits: 6, baseValue: 67), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 5, nbBits: 6, baseValue: 99), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 8, nbBits: 6, baseValue: 259), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 0, nbBits: 4, baseValue: 4), + new ZSTD_seqSymbol(nextState: 48, nbAdditionalBits: 0, nbBits: 4, baseValue: 4), + new ZSTD_seqSymbol(nextState: 16, nbAdditionalBits: 0, nbBits: 4, baseValue: 5), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 0, nbBits: 5, baseValue: 7), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 0, nbBits: 5, baseValue: 8), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 0, nbBits: 5, baseValue: 10), + new ZSTD_seqSymbol(nextState: 32, nbAdditionalBits: 0, nbBits: 5, baseValue: 11), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 14), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 17), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 20), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 23), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 26), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 29), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 0, nbBits: 6, baseValue: 32), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 16, nbBits: 6, baseValue: 65539), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 15, nbBits: 6, baseValue: 32771), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 14, nbBits: 6, baseValue: 16387), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 13, nbBits: 6, baseValue: 8195), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 12, nbBits: 6, baseValue: 4099), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 11, nbBits: 6, baseValue: 2051), + new ZSTD_seqSymbol(nextState: 0, nbAdditionalBits: 10, nbBits: 6, baseValue: 1027), + } + ); + + private static void ZSTD_buildSeqTable_rle(ZSTD_seqSymbol* dt, uint baseValue, byte nbAddBits) + { + void* ptr = dt; + ZSTD_seqSymbol_header* DTableH = (ZSTD_seqSymbol_header*)ptr; + ZSTD_seqSymbol* cell = dt + 1; + DTableH->tableLog = 0; + DTableH->fastMode = 0; + cell->nbBits = 0; + cell->nextState = 0; + assert(nbAddBits < 255); + cell->nbAdditionalBits = nbAddBits; + cell->baseValue = baseValue; + } + + /* ZSTD_buildFSETable() : + * generate FSE decoding table for one symbol (ll, ml or off) + * cannot fail if input is valid => + * all inputs are presumed validated at this stage */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_buildFSETable_body( + ZSTD_seqSymbol* dt, + short* normalizedCounter, + uint maxSymbolValue, + uint* baseValue, + byte* nbAdditionalBits, + uint tableLog, + void* wksp, + nuint wkspSize + ) + { + ZSTD_seqSymbol* tableDecode = dt + 1; + uint maxSV1 = maxSymbolValue + 1; + uint tableSize = (uint)(1 << (int)tableLog); + ushort* symbolNext = (ushort*)wksp; + byte* spread = (byte*)(symbolNext + 52 + 1); + uint highThreshold = tableSize - 1; + assert(maxSymbolValue <= 52); + assert(tableLog <= 9); + assert(wkspSize >= sizeof(short) * (52 + 1) + (1U << 9) + sizeof(ulong)); + { + ZSTD_seqSymbol_header DTableH; + DTableH.tableLog = tableLog; + DTableH.fastMode = 1; + { + short largeLimit = (short)(1 << (int)(tableLog - 1)); + uint s; + for (s = 0; s < maxSV1; s++) + { + if (normalizedCounter[s] == -1) + { + tableDecode[highThreshold--].baseValue = s; + symbolNext[s] = 1; + } + else + { + if (normalizedCounter[s] >= largeLimit) + { + DTableH.fastMode = 0; + } + + assert(normalizedCounter[s] >= 0); + symbolNext[s] = (ushort)normalizedCounter[s]; + } + } + } + + memcpy(dt, &DTableH, (uint)sizeof(ZSTD_seqSymbol_header)); + } + + assert(tableSize <= 512); + if (highThreshold == tableSize - 1) + { + nuint tableMask = tableSize - 1; + nuint step = (tableSize >> 1) + (tableSize >> 3) + 3; + { + const ulong add = 0x0101010101010101UL; + nuint pos = 0; + ulong sv = 0; + uint s; + for (s = 0; s < maxSV1; ++s, sv += add) + { + int i; + int n = normalizedCounter[s]; + MEM_write64(spread + pos, sv); + for (i = 8; i < n; i += 8) + { + MEM_write64(spread + pos + i, sv); + } + + assert(n >= 0); + pos += (nuint)n; + } + } + + { + nuint position = 0; + nuint s; + const nuint unroll = 2; + assert(tableSize % unroll == 0); + for (s = 0; s < tableSize; s += unroll) + { + nuint u; + for (u = 0; u < unroll; ++u) + { + nuint uPosition = position + u * step & tableMask; + tableDecode[uPosition].baseValue = spread[s + u]; + } + + position = position + unroll * step & tableMask; + } + + assert(position == 0); + } + } + else + { + uint tableMask = tableSize - 1; + uint step = (tableSize >> 1) + (tableSize >> 3) + 3; + uint s, + position = 0; + for (s = 0; s < maxSV1; s++) + { + int i; + int n = normalizedCounter[s]; + for (i = 0; i < n; i++) + { + tableDecode[position].baseValue = s; + position = position + step & tableMask; + while (position > highThreshold) + { + position = position + step & tableMask; + } + } + } + + assert(position == 0); + } + + { + uint u; + for (u = 0; u < tableSize; u++) + { + uint symbol = tableDecode[u].baseValue; + uint nextState = symbolNext[symbol]++; + tableDecode[u].nbBits = (byte)(tableLog - ZSTD_highbit32(nextState)); + tableDecode[u].nextState = (ushort)( + (nextState << tableDecode[u].nbBits) - tableSize + ); + assert(nbAdditionalBits[symbol] < 255); + tableDecode[u].nbAdditionalBits = nbAdditionalBits[symbol]; + tableDecode[u].baseValue = baseValue[symbol]; + } + } + } + + /* Avoids the FORCE_INLINE of the _body() function. */ + private static void ZSTD_buildFSETable_body_default( + ZSTD_seqSymbol* dt, + short* normalizedCounter, + uint maxSymbolValue, + uint* baseValue, + byte* nbAdditionalBits, + uint tableLog, + void* wksp, + nuint wkspSize + ) + { + ZSTD_buildFSETable_body( + dt, + normalizedCounter, + maxSymbolValue, + baseValue, + nbAdditionalBits, + tableLog, + wksp, + wkspSize + ); + } + + /* ZSTD_buildFSETable() : + * generate FSE decoding table for one symbol (ll, ml or off) + * this function must be called with valid parameters only + * (dt is large enough, normalizedCounter distribution total is a power of 2, max is within range, etc.) + * in which case it cannot fail. + * The workspace must be 4-byte aligned and at least ZSTD_BUILD_FSE_TABLE_WKSP_SIZE bytes, which is + * defined in zstd_decompress_internal.h. + * Internal use only. + */ + private static void ZSTD_buildFSETable( + ZSTD_seqSymbol* dt, + short* normalizedCounter, + uint maxSymbolValue, + uint* baseValue, + byte* nbAdditionalBits, + uint tableLog, + void* wksp, + nuint wkspSize, + int bmi2 + ) + { + ZSTD_buildFSETable_body_default( + dt, + normalizedCounter, + maxSymbolValue, + baseValue, + nbAdditionalBits, + tableLog, + wksp, + wkspSize + ); + } + + /*! ZSTD_buildSeqTable() : + * @return : nb bytes read from src, + * or an error code if it fails */ + private static nuint ZSTD_buildSeqTable( + ZSTD_seqSymbol* DTableSpace, + ZSTD_seqSymbol** DTablePtr, + SymbolEncodingType_e type, + uint max, + uint maxLog, + void* src, + nuint srcSize, + uint* baseValue, + byte* nbAdditionalBits, + ZSTD_seqSymbol* defaultTable, + uint flagRepeatTable, + int ddictIsCold, + int nbSeq, + uint* wksp, + nuint wkspSize, + int bmi2 + ) + { + switch (type) + { + case SymbolEncodingType_e.set_rle: + if (srcSize == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + if (*(byte*)src > max) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + { + uint symbol = *(byte*)src; + uint baseline = baseValue[symbol]; + byte nbBits = nbAdditionalBits[symbol]; + ZSTD_buildSeqTable_rle(DTableSpace, baseline, nbBits); + } + + *DTablePtr = DTableSpace; + return 1; + case SymbolEncodingType_e.set_basic: + *DTablePtr = defaultTable; + return 0; + case SymbolEncodingType_e.set_repeat: + if (flagRepeatTable == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + if (ddictIsCold != 0 && nbSeq > 24) + { + void* pStart = *DTablePtr; + nuint pSize = (nuint)(sizeof(ZSTD_seqSymbol) * (1 + (1 << (int)maxLog))); + { + sbyte* _ptr = (sbyte*)pStart; + nuint _size = pSize; + nuint _pos; + for (_pos = 0; _pos < _size; _pos += 64) + { +#if NETCOREAPP3_0_OR_GREATER + if (System.Runtime.Intrinsics.X86.Sse.IsSupported) + { + System.Runtime.Intrinsics.X86.Sse.Prefetch1(_ptr + _pos); + } +#endif + } + } + } + + return 0; + case SymbolEncodingType_e.set_compressed: + { + uint tableLog; + short* norm = stackalloc short[53]; + nuint headerSize = FSE_readNCount(norm, &max, &tableLog, src, srcSize); + if (ERR_isError(headerSize)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + if (tableLog > maxLog) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + ZSTD_buildFSETable( + DTableSpace, + norm, + max, + baseValue, + nbAdditionalBits, + tableLog, + wksp, + wkspSize, + bmi2 + ); + *DTablePtr = DTableSpace; + return headerSize; + } + + default: + assert(0 != 0); + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_GENERIC)); + } + } + + /*! ZSTD_decodeSeqHeaders() : + * decode sequence header from src */ + /* Used by: zstd_decompress_block, fullbench */ + private static nuint ZSTD_decodeSeqHeaders( + ZSTD_DCtx_s* dctx, + int* nbSeqPtr, + void* src, + nuint srcSize + ) + { + byte* istart = (byte*)src; + byte* iend = istart + srcSize; + byte* ip = istart; + int nbSeq; + if (srcSize < 1) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + nbSeq = *ip++; + if (nbSeq > 0x7F) + { + if (nbSeq == 0xFF) + { + if (ip + 2 > iend) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + nbSeq = MEM_readLE16(ip) + 0x7F00; + ip += 2; + } + else + { + if (ip >= iend) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + nbSeq = (nbSeq - 0x80 << 8) + *ip++; + } + } + + *nbSeqPtr = nbSeq; + if (nbSeq == 0) + { + if (ip != iend) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + return (nuint)(ip - istart); + } + + if (ip + 1 > iend) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + if ((*ip & 3) != 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + { + SymbolEncodingType_e LLtype = (SymbolEncodingType_e)(*ip >> 6); + SymbolEncodingType_e OFtype = (SymbolEncodingType_e)(*ip >> 4 & 3); + SymbolEncodingType_e MLtype = (SymbolEncodingType_e)(*ip >> 2 & 3); + ip++; + { + nuint llhSize = ZSTD_buildSeqTable( + &dctx->entropy.LLTable.e0, + &dctx->LLTptr, + LLtype, + 35, + 9, + ip, + (nuint)(iend - ip), + LL_base, + LL_bits, + LL_defaultDTable, + dctx->fseEntropy, + dctx->ddictIsCold, + nbSeq, + dctx->workspace, + sizeof(uint) * 640, + ZSTD_DCtx_get_bmi2(dctx) + ); + if (ERR_isError(llhSize)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + ip += llhSize; + } + + { + nuint ofhSize = ZSTD_buildSeqTable( + &dctx->entropy.OFTable.e0, + &dctx->OFTptr, + OFtype, + 31, + 8, + ip, + (nuint)(iend - ip), + OF_base, + OF_bits, + OF_defaultDTable, + dctx->fseEntropy, + dctx->ddictIsCold, + nbSeq, + dctx->workspace, + sizeof(uint) * 640, + ZSTD_DCtx_get_bmi2(dctx) + ); + if (ERR_isError(ofhSize)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + ip += ofhSize; + } + + { + nuint mlhSize = ZSTD_buildSeqTable( + &dctx->entropy.MLTable.e0, + &dctx->MLTptr, + MLtype, + 52, + 9, + ip, + (nuint)(iend - ip), + ML_base, + ML_bits, + ML_defaultDTable, + dctx->fseEntropy, + dctx->ddictIsCold, + nbSeq, + dctx->workspace, + sizeof(uint) * 640, + ZSTD_DCtx_get_bmi2(dctx) + ); + if (ERR_isError(mlhSize)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + ip += mlhSize; + } + } + + return (nuint)(ip - istart); + } + +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_dec32table => new uint[8] { 0, 1, 2, 1, 4, 4, 4, 4 }; + private static uint* dec32table => + (uint*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_dec32table) + ); +#else + + private static readonly uint* dec32table = GetArrayPointer( + new uint[8] { 0, 1, 2, 1, 4, 4, 4, 4 } + ); +#endif +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_dec64table => new int[8] { 8, 8, 8, 7, 8, 9, 10, 11 }; + private static int* dec64table => + (int*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_dec64table) + ); +#else + + private static readonly int* dec64table = GetArrayPointer( + new int[8] { 8, 8, 8, 7, 8, 9, 10, 11 } + ); +#endif + /*! ZSTD_overlapCopy8() : + * Copies 8 bytes from ip to op and updates op and ip where ip <= op. + * If the offset is < 8 then the offset is spread to at least 8 bytes. + * + * Precondition: *ip <= *op + * Postcondition: *op - *op >= 8 + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_overlapCopy8(byte** op, byte** ip, nuint offset) + { + assert(*ip <= *op); + if (offset < 8) + { + int sub2 = dec64table[offset]; + (*op)[0] = (*ip)[0]; + (*op)[1] = (*ip)[1]; + (*op)[2] = (*ip)[2]; + (*op)[3] = (*ip)[3]; + *ip += dec32table[offset]; + ZSTD_copy4(*op + 4, *ip); + *ip -= sub2; + } + else + { + ZSTD_copy8(*op, *ip); + } + + *ip += 8; + *op += 8; + assert(*op - *ip >= 8); + } + + /*! ZSTD_safecopy() : + * Specialized version of memcpy() that is allowed to READ up to WILDCOPY_OVERLENGTH past the input buffer + * and write up to 16 bytes past oend_w (op >= oend_w is allowed). + * This function is only called in the uncommon case where the sequence is near the end of the block. It + * should be fast for a single long sequence, but can be slow for several short sequences. + * + * @param ovtype controls the overlap detection + * - ZSTD_no_overlap: The source and destination are guaranteed to be at least WILDCOPY_VECLEN bytes apart. + * - ZSTD_overlap_src_before_dst: The src and dst may overlap and may be any distance apart. + * The src buffer must be before the dst buffer. + */ + private static void ZSTD_safecopy( + byte* op, + byte* oend_w, + byte* ip, + nint length, + ZSTD_overlap_e ovtype + ) + { + nint diff = (nint)(op - ip); + byte* oend = op + length; + assert( + ovtype == ZSTD_overlap_e.ZSTD_no_overlap && (diff <= -8 || diff >= 8 || op >= oend_w) + || ovtype == ZSTD_overlap_e.ZSTD_overlap_src_before_dst && diff >= 0 + ); + if (length < 8) + { + while (op < oend) + { + *op++ = *ip++; + } + + return; + } + + if (ovtype == ZSTD_overlap_e.ZSTD_overlap_src_before_dst) + { + assert(length >= 8); + ZSTD_overlapCopy8(&op, &ip, (nuint)diff); + length -= 8; + assert(op - ip >= 8); + assert(op <= oend); + } + + if (oend <= oend_w) + { + ZSTD_wildcopy(op, ip, length, ovtype); + return; + } + + if (op <= oend_w) + { + assert(oend > oend_w); + ZSTD_wildcopy(op, ip, (nint)(oend_w - op), ovtype); + ip += oend_w - op; + op += oend_w - op; + } + + while (op < oend) + { + *op++ = *ip++; + } + } + + /* ZSTD_safecopyDstBeforeSrc(): + * This version allows overlap with dst before src, or handles the non-overlap case with dst after src + * Kept separate from more common ZSTD_safecopy case to avoid performance impact to the safecopy common case */ + private static void ZSTD_safecopyDstBeforeSrc(byte* op, byte* ip, nint length) + { + nint diff = (nint)(op - ip); + byte* oend = op + length; + if (length < 8 || diff > -8) + { + while (op < oend) + { + *op++ = *ip++; + } + + return; + } + + if (op <= oend - 32 && diff < -16) + { + ZSTD_wildcopy(op, ip, (nint)(oend - 32 - op), ZSTD_overlap_e.ZSTD_no_overlap); + ip += oend - 32 - op; + op += oend - 32 - op; + } + + while (op < oend) + { + *op++ = *ip++; + } + } + + /* ZSTD_execSequenceEnd(): + * This version handles cases that are near the end of the output buffer. It requires + * more careful checks to make sure there is no overflow. By separating out these hard + * and unlikely cases, we can speed up the common cases. + * + * NOTE: This function needs to be fast for a single long sequence, but doesn't need + * to be optimized for many small sequences, since those fall into ZSTD_execSequence(). + */ + private static nuint ZSTD_execSequenceEnd( + byte* op, + byte* oend, + seq_t sequence, + byte** litPtr, + byte* litLimit, + byte* prefixStart, + byte* virtualStart, + byte* dictEnd + ) + { + byte* oLitEnd = op + sequence.litLength; + nuint sequenceLength = sequence.litLength + sequence.matchLength; + byte* iLitEnd = *litPtr + sequence.litLength; + byte* match = oLitEnd - sequence.offset; + byte* oend_w = oend - 32; + if (sequenceLength > (nuint)(oend - op)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + if (sequence.litLength > (nuint)(litLimit - *litPtr)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + assert(op < op + sequenceLength); + assert(oLitEnd < op + sequenceLength); + ZSTD_safecopy( + op, + oend_w, + *litPtr, + (nint)sequence.litLength, + ZSTD_overlap_e.ZSTD_no_overlap + ); + op = oLitEnd; + *litPtr = iLitEnd; + if (sequence.offset > (nuint)(oLitEnd - prefixStart)) + { + if (sequence.offset > (nuint)(oLitEnd - virtualStart)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + match = dictEnd - (prefixStart - match); + if (match + sequence.matchLength <= dictEnd) + { + memmove(oLitEnd, match, sequence.matchLength); + return sequenceLength; + } + + { + nuint length1 = (nuint)(dictEnd - match); + memmove(oLitEnd, match, length1); + op = oLitEnd + length1; + sequence.matchLength -= length1; + match = prefixStart; + } + } + + ZSTD_safecopy( + op, + oend_w, + match, + (nint)sequence.matchLength, + ZSTD_overlap_e.ZSTD_overlap_src_before_dst + ); + return sequenceLength; + } + + /* ZSTD_execSequenceEndSplitLitBuffer(): + * This version is intended to be used during instances where the litBuffer is still split. It is kept separate to avoid performance impact for the good case. + */ + private static nuint ZSTD_execSequenceEndSplitLitBuffer( + byte* op, + byte* oend, + byte* oend_w, + seq_t sequence, + byte** litPtr, + byte* litLimit, + byte* prefixStart, + byte* virtualStart, + byte* dictEnd + ) + { + byte* oLitEnd = op + sequence.litLength; + nuint sequenceLength = sequence.litLength + sequence.matchLength; + byte* iLitEnd = *litPtr + sequence.litLength; + byte* match = oLitEnd - sequence.offset; + if (sequenceLength > (nuint)(oend - op)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + if (sequence.litLength > (nuint)(litLimit - *litPtr)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + assert(op < op + sequenceLength); + assert(oLitEnd < op + sequenceLength); + if (op > *litPtr && op < *litPtr + sequence.litLength) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + ZSTD_safecopyDstBeforeSrc(op, *litPtr, (nint)sequence.litLength); + op = oLitEnd; + *litPtr = iLitEnd; + if (sequence.offset > (nuint)(oLitEnd - prefixStart)) + { + if (sequence.offset > (nuint)(oLitEnd - virtualStart)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + match = dictEnd - (prefixStart - match); + if (match + sequence.matchLength <= dictEnd) + { + memmove(oLitEnd, match, sequence.matchLength); + return sequenceLength; + } + + { + nuint length1 = (nuint)(dictEnd - match); + memmove(oLitEnd, match, length1); + op = oLitEnd + length1; + sequence.matchLength -= length1; + match = prefixStart; + } + } + + ZSTD_safecopy( + op, + oend_w, + match, + (nint)sequence.matchLength, + ZSTD_overlap_e.ZSTD_overlap_src_before_dst + ); + return sequenceLength; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_execSequence( + byte* op, + byte* oend, + seq_t sequence, + byte** litPtr, + byte* litLimit, + byte* prefixStart, + byte* virtualStart, + byte* dictEnd + ) + { + var sequence_litLength = sequence.litLength; + var sequence_matchLength = sequence.matchLength; + var sequence_offset = sequence.offset; + byte* oLitEnd = op + sequence_litLength; + nuint sequenceLength = sequence_litLength + sequence_matchLength; + /* risk : address space overflow (32-bits) */ + byte* oMatchEnd = op + sequenceLength; + /* risk : address space underflow on oend=NULL */ + byte* oend_w = oend - 32; + byte* iLitEnd = *litPtr + sequence_litLength; + byte* match = oLitEnd - sequence_offset; + assert(op != null); + assert(oend_w < oend); + if ( + iLitEnd > litLimit + || oMatchEnd > oend_w + || MEM_32bits && (nuint)(oend - op) < sequenceLength + 32 + ) + { + return ZSTD_execSequenceEnd( + op, + oend, + new seq_t + { + litLength = sequence_litLength, + matchLength = sequence_matchLength, + offset = sequence_offset, + }, + litPtr, + litLimit, + prefixStart, + virtualStart, + dictEnd + ); + } + + assert(op <= oLitEnd); + assert(oLitEnd < oMatchEnd); + assert(oMatchEnd <= oend); + assert(iLitEnd <= litLimit); + assert(oLitEnd <= oend_w); + assert(oMatchEnd <= oend_w); + assert(32 >= 16); + ZSTD_copy16(op, *litPtr); + if (sequence_litLength > 16) + { + ZSTD_wildcopy( + op + 16, + *litPtr + 16, + (nint)(sequence_litLength - 16), + ZSTD_overlap_e.ZSTD_no_overlap + ); + } + + op = oLitEnd; + *litPtr = iLitEnd; + if (sequence_offset > (nuint)(oLitEnd - prefixStart)) + { + if (sequence_offset > (nuint)(oLitEnd - virtualStart)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + match = dictEnd + (match - prefixStart); + if (match + sequence_matchLength <= dictEnd) + { + memmove(oLitEnd, match, sequence_matchLength); + return sequenceLength; + } + + { + nuint length1 = (nuint)(dictEnd - match); + memmove(oLitEnd, match, length1); + op = oLitEnd + length1; + sequence_matchLength -= length1; + match = prefixStart; + } + } + + assert(op <= oMatchEnd); + assert(oMatchEnd <= oend_w); + assert(match >= prefixStart); + assert(sequence_matchLength >= 1); + if (sequence_offset >= 16) + { + ZSTD_wildcopy(op, match, (nint)sequence_matchLength, ZSTD_overlap_e.ZSTD_no_overlap); + return sequenceLength; + } + + assert(sequence_offset < 16); + ZSTD_overlapCopy8(ref op, ref match, sequence_offset); + if (sequence_matchLength > 8) + { + assert(op < oMatchEnd); + ZSTD_wildcopy( + op, + match, + (nint)sequence_matchLength - 8, + ZSTD_overlap_e.ZSTD_overlap_src_before_dst + ); + } + + return sequenceLength; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_execSequenceSplitLitBuffer( + byte* op, + byte* oend, + byte* oend_w, + seq_t sequence, + byte** litPtr, + byte* litLimit, + byte* prefixStart, + byte* virtualStart, + byte* dictEnd + ) + { + byte* oLitEnd = op + sequence.litLength; + nuint sequenceLength = sequence.litLength + sequence.matchLength; + /* risk : address space overflow (32-bits) */ + byte* oMatchEnd = op + sequenceLength; + byte* iLitEnd = *litPtr + sequence.litLength; + byte* match = oLitEnd - sequence.offset; + assert(op != null); + assert(oend_w < oend); + if ( + iLitEnd > litLimit + || oMatchEnd > oend_w + || MEM_32bits && (nuint)(oend - op) < sequenceLength + 32 + ) + { + return ZSTD_execSequenceEndSplitLitBuffer( + op, + oend, + oend_w, + sequence, + litPtr, + litLimit, + prefixStart, + virtualStart, + dictEnd + ); + } + + assert(op <= oLitEnd); + assert(oLitEnd < oMatchEnd); + assert(oMatchEnd <= oend); + assert(iLitEnd <= litLimit); + assert(oLitEnd <= oend_w); + assert(oMatchEnd <= oend_w); + assert(32 >= 16); + ZSTD_copy16(op, *litPtr); + if (sequence.litLength > 16) + { + ZSTD_wildcopy( + op + 16, + *litPtr + 16, + (nint)(sequence.litLength - 16), + ZSTD_overlap_e.ZSTD_no_overlap + ); + } + + op = oLitEnd; + *litPtr = iLitEnd; + if (sequence.offset > (nuint)(oLitEnd - prefixStart)) + { + if (sequence.offset > (nuint)(oLitEnd - virtualStart)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + match = dictEnd + (match - prefixStart); + if (match + sequence.matchLength <= dictEnd) + { + memmove(oLitEnd, match, sequence.matchLength); + return sequenceLength; + } + + { + nuint length1 = (nuint)(dictEnd - match); + memmove(oLitEnd, match, length1); + op = oLitEnd + length1; + sequence.matchLength -= length1; + match = prefixStart; + } + } + + assert(op <= oMatchEnd); + assert(oMatchEnd <= oend_w); + assert(match >= prefixStart); + assert(sequence.matchLength >= 1); + if (sequence.offset >= 16) + { + ZSTD_wildcopy(op, match, (nint)sequence.matchLength, ZSTD_overlap_e.ZSTD_no_overlap); + return sequenceLength; + } + + assert(sequence.offset < 16); + ZSTD_overlapCopy8(&op, &match, sequence.offset); + if (sequence.matchLength > 8) + { + assert(op < oMatchEnd); + ZSTD_wildcopy( + op, + match, + (nint)sequence.matchLength - 8, + ZSTD_overlap_e.ZSTD_overlap_src_before_dst + ); + } + + return sequenceLength; + } + + private static void ZSTD_initFseState( + ZSTD_fseState* DStatePtr, + BIT_DStream_t* bitD, + ZSTD_seqSymbol* dt + ) + { + void* ptr = dt; + ZSTD_seqSymbol_header* DTableH = (ZSTD_seqSymbol_header*)ptr; + DStatePtr->state = BIT_readBits(bitD, DTableH->tableLog); + BIT_reloadDStream(bitD); + DStatePtr->table = dt + 1; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_updateFseStateWithDInfo( + ZSTD_fseState* DStatePtr, + BIT_DStream_t* bitD, + ushort nextState, + uint nbBits + ) + { + nuint lowBits = BIT_readBits(bitD, nbBits); + DStatePtr->state = nextState + lowBits; + } + + /** + * ZSTD_decodeSequence(): + * @p longOffsets : tells the decoder to reload more bit while decoding large offsets + * only used in 32-bit mode + * @return : Sequence (litL + matchL + offset) + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static seq_t ZSTD_decodeSequence( + seqState_t* seqState, + ZSTD_longOffset_e longOffsets, + int isLastSeq + ) + { + seq_t seq; + ZSTD_seqSymbol* llDInfo = seqState->stateLL.table + seqState->stateLL.state; + ZSTD_seqSymbol* mlDInfo = seqState->stateML.table + seqState->stateML.state; + ZSTD_seqSymbol* ofDInfo = seqState->stateOffb.table + seqState->stateOffb.state; + seq.matchLength = mlDInfo->baseValue; + seq.litLength = llDInfo->baseValue; + { + uint ofBase = ofDInfo->baseValue; + byte llBits = llDInfo->nbAdditionalBits; + byte mlBits = mlDInfo->nbAdditionalBits; + byte ofBits = ofDInfo->nbAdditionalBits; + byte totalBits = (byte)(llBits + mlBits + ofBits); + ushort llNext = llDInfo->nextState; + ushort mlNext = mlDInfo->nextState; + ushort ofNext = ofDInfo->nextState; + uint llnbBits = llDInfo->nbBits; + uint mlnbBits = mlDInfo->nbBits; + uint ofnbBits = ofDInfo->nbBits; + assert(llBits <= 16); + assert(mlBits <= 16); + assert(ofBits <= 31); + { + nuint offset; + if (ofBits > 1) + { + if (MEM_32bits && longOffsets != default && ofBits >= 25) + { + /* Always read extra bits, this keeps the logic simple, + * avoids branches, and avoids accidentally reading 0 bits. + */ + const uint extraBits = 30 - 25; + offset = + ofBase + + ( + BIT_readBitsFast(&seqState->DStream, ofBits - extraBits) + << (int)extraBits + ); + BIT_reloadDStream(&seqState->DStream); + offset += BIT_readBitsFast(&seqState->DStream, extraBits); + } + else + { + offset = ofBase + BIT_readBitsFast(&seqState->DStream, ofBits); + if (MEM_32bits) + { + BIT_reloadDStream(&seqState->DStream); + } + } + + seqState->prevOffset.e2 = seqState->prevOffset.e1; + seqState->prevOffset.e1 = seqState->prevOffset.e0; + seqState->prevOffset.e0 = offset; + } + else + { + uint ll0 = llDInfo->baseValue == 0 ? 1U : 0U; + if (ofBits == 0) + { + offset = (&seqState->prevOffset.e0)[ll0]; + seqState->prevOffset.e1 = (&seqState->prevOffset.e0)[ll0 == 0 ? 1 : 0]; + seqState->prevOffset.e0 = offset; + } + else + { + offset = ofBase + ll0 + BIT_readBitsFast(&seqState->DStream, 1); + { + nuint temp = + offset == 3 + ? seqState->prevOffset.e0 - 1 + : (&seqState->prevOffset.e0)[offset]; + temp -= temp == 0 ? 1U : 0U; + if (offset != 1) + { + seqState->prevOffset.e2 = seqState->prevOffset.e1; + } + + seqState->prevOffset.e1 = seqState->prevOffset.e0; + seqState->prevOffset.e0 = offset = temp; + } + } + } + + seq.offset = offset; + } + + if (mlBits > 0) + { + seq.matchLength += BIT_readBitsFast(&seqState->DStream, mlBits); + } + + if (MEM_32bits && mlBits + llBits >= 25 - (30 - 25)) + { + BIT_reloadDStream(&seqState->DStream); + } + + if (MEM_64bits && totalBits >= 57 - (9 + 9 + 8)) + { + BIT_reloadDStream(&seqState->DStream); + } + + if (llBits > 0) + { + seq.litLength += BIT_readBitsFast(&seqState->DStream, llBits); + } + + if (MEM_32bits) + { + BIT_reloadDStream(&seqState->DStream); + } + + if (isLastSeq == 0) + { + ZSTD_updateFseStateWithDInfo( + &seqState->stateLL, + &seqState->DStream, + llNext, + llnbBits + ); + ZSTD_updateFseStateWithDInfo( + &seqState->stateML, + &seqState->DStream, + mlNext, + mlnbBits + ); + if (MEM_32bits) + { + BIT_reloadDStream(&seqState->DStream); + } + + ZSTD_updateFseStateWithDInfo( + &seqState->stateOffb, + &seqState->DStream, + ofNext, + ofnbBits + ); + BIT_reloadDStream(&seqState->DStream); + } + } + + return seq; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_decompressSequences_bodySplitLitBuffer( + ZSTD_DCtx_s* dctx, + void* dst, + nuint maxDstSize, + void* seqStart, + nuint seqSize, + int nbSeq, + ZSTD_longOffset_e isLongOffset + ) + { + byte* ip = (byte*)seqStart; + byte* iend = ip + seqSize; + byte* ostart = (byte*)dst; + byte* oend = ZSTD_maybeNullPtrAdd(ostart, (nint)maxDstSize); + byte* op = ostart; + byte* litPtr = dctx->litPtr; + byte* litBufferEnd = dctx->litBufferEnd; + byte* prefixStart = (byte*)dctx->prefixStart; + byte* vBase = (byte*)dctx->virtualStart; + byte* dictEnd = (byte*)dctx->dictEnd; + if (nbSeq != 0) + { + seqState_t seqState; + dctx->fseEntropy = 1; + { + uint i; + for (i = 0; i < 3; i++) + { + (&seqState.prevOffset.e0)[i] = dctx->entropy.rep[i]; + } + } + + if (ERR_isError(BIT_initDStream(&seqState.DStream, ip, (nuint)(iend - ip)))) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + ZSTD_initFseState(&seqState.stateLL, &seqState.DStream, dctx->LLTptr); + ZSTD_initFseState(&seqState.stateOffb, &seqState.DStream, dctx->OFTptr); + ZSTD_initFseState(&seqState.stateML, &seqState.DStream, dctx->MLTptr); + assert(dst != null); + { + /* some static analyzer believe that @sequence is not initialized (it necessarily is, since for(;;) loop as at least one iteration) */ + seq_t sequence = new seq_t + { + litLength = 0, + matchLength = 0, + offset = 0, + }; + for (; nbSeq != 0; nbSeq--) + { + sequence = ZSTD_decodeSequence(&seqState, isLongOffset, nbSeq == 1 ? 1 : 0); + if (litPtr + sequence.litLength > dctx->litBufferEnd) + { + break; + } + + { + nuint oneSeqSize = ZSTD_execSequenceSplitLitBuffer( + op, + oend, + litPtr + sequence.litLength - 32, + sequence, + &litPtr, + litBufferEnd, + prefixStart, + vBase, + dictEnd + ); + if (ERR_isError(oneSeqSize)) + { + return oneSeqSize; + } + + op += oneSeqSize; + } + } + + if (nbSeq > 0) + { + nuint leftoverLit = (nuint)(dctx->litBufferEnd - litPtr); + if (leftoverLit != 0) + { + if (leftoverLit > (nuint)(oend - op)) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall) + ); + } + + ZSTD_safecopyDstBeforeSrc(op, litPtr, (nint)leftoverLit); + sequence.litLength -= leftoverLit; + op += leftoverLit; + } + + litPtr = dctx->litExtraBuffer; + litBufferEnd = dctx->litExtraBuffer + (1 << 16); + dctx->litBufferLocation = ZSTD_litLocation_e.ZSTD_not_in_dst; + { + nuint oneSeqSize = ZSTD_execSequence( + op, + oend, + sequence, + &litPtr, + litBufferEnd, + prefixStart, + vBase, + dictEnd + ); + if (ERR_isError(oneSeqSize)) + { + return oneSeqSize; + } + + op += oneSeqSize; + } + + nbSeq--; + } + } + + if (nbSeq > 0) + { + for (; nbSeq != 0; nbSeq--) + { + seq_t sequence = ZSTD_decodeSequence( + &seqState, + isLongOffset, + nbSeq == 1 ? 1 : 0 + ); + nuint oneSeqSize = ZSTD_execSequence( + op, + oend, + sequence, + &litPtr, + litBufferEnd, + prefixStart, + vBase, + dictEnd + ); + if (ERR_isError(oneSeqSize)) + { + return oneSeqSize; + } + + op += oneSeqSize; + } + } + + if (nbSeq != 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + if (BIT_endOfDStream(&seqState.DStream) == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + { + uint i; + for (i = 0; i < 3; i++) + { + dctx->entropy.rep[i] = (uint)(&seqState.prevOffset.e0)[i]; + } + } + } + + if (dctx->litBufferLocation == ZSTD_litLocation_e.ZSTD_split) + { + /* split hasn't been reached yet, first get dst then copy litExtraBuffer */ + nuint lastLLSize = (nuint)(litBufferEnd - litPtr); + if (lastLLSize > (nuint)(oend - op)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + if (op != null) + { + memmove(op, litPtr, lastLLSize); + op += lastLLSize; + } + + litPtr = dctx->litExtraBuffer; + litBufferEnd = dctx->litExtraBuffer + (1 << 16); + dctx->litBufferLocation = ZSTD_litLocation_e.ZSTD_not_in_dst; + } + + { + nuint lastLLSize = (nuint)(litBufferEnd - litPtr); + if (lastLLSize > (nuint)(oend - op)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + if (op != null) + { + memcpy(op, litPtr, (uint)lastLLSize); + op += lastLLSize; + } + } + + return (nuint)(op - ostart); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_decompressSequences_body( + ZSTD_DCtx_s* dctx, + void* dst, + nuint maxDstSize, + void* seqStart, + nuint seqSize, + int nbSeq, + ZSTD_longOffset_e isLongOffset + ) + { + // HACK, force nbSeq to stack (better register usage) + System.Threading.Volatile.Read(ref nbSeq); + byte* ip = (byte*)seqStart; + byte* iend = ip + seqSize; + byte* ostart = (byte*)dst; + byte* oend = + dctx->litBufferLocation == ZSTD_litLocation_e.ZSTD_not_in_dst + ? ZSTD_maybeNullPtrAdd(ostart, (nint)maxDstSize) + : dctx->litBuffer; + byte* op = ostart; + byte* litPtr = dctx->litPtr; + byte* litEnd = litPtr + dctx->litSize; + byte* prefixStart = (byte*)dctx->prefixStart; + byte* vBase = (byte*)dctx->virtualStart; + byte* dictEnd = (byte*)dctx->dictEnd; + if (nbSeq != 0) + { + seqState_t seqState; + System.Runtime.CompilerServices.Unsafe.SkipInit(out seqState); + dctx->fseEntropy = 1; + { + uint i; + for (i = 0; i < 3; i++) + { + System.Runtime.CompilerServices.Unsafe.Add(ref seqState.prevOffset.e0, (int)i) = + dctx->entropy.rep[i]; + } + } + + if (ERR_isError(BIT_initDStream(ref seqState.DStream, ip, (nuint)(iend - ip)))) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + ZSTD_initFseState(ref seqState.stateLL, ref seqState.DStream, dctx->LLTptr); + ZSTD_initFseState(ref seqState.stateOffb, ref seqState.DStream, dctx->OFTptr); + ZSTD_initFseState(ref seqState.stateML, ref seqState.DStream, dctx->MLTptr); + assert(dst != null); + nuint seqState_DStream_bitContainer = seqState.DStream.bitContainer; + uint seqState_DStream_bitsConsumed = seqState.DStream.bitsConsumed; + sbyte* seqState_DStream_ptr = seqState.DStream.ptr; + sbyte* seqState_DStream_start = seqState.DStream.start; + sbyte* seqState_DStream_limitPtr = seqState.DStream.limitPtr; + for (; nbSeq != 0; nbSeq--) + { + nuint sequence_litLength; + nuint sequence_matchLength; + nuint sequence_offset; + ZSTD_seqSymbol* llDInfo = seqState.stateLL.table + seqState.stateLL.state; + ZSTD_seqSymbol* mlDInfo = seqState.stateML.table + seqState.stateML.state; + ZSTD_seqSymbol* ofDInfo = seqState.stateOffb.table + seqState.stateOffb.state; + sequence_matchLength = mlDInfo->baseValue; + sequence_litLength = llDInfo->baseValue; + { + uint ofBase = ofDInfo->baseValue; + byte llBits = llDInfo->nbAdditionalBits; + byte mlBits = mlDInfo->nbAdditionalBits; + byte ofBits = ofDInfo->nbAdditionalBits; + byte totalBits = (byte)(llBits + mlBits + ofBits); + ushort llNext = llDInfo->nextState; + ushort mlNext = mlDInfo->nextState; + ushort ofNext = ofDInfo->nextState; + uint llnbBits = llDInfo->nbBits; + uint mlnbBits = mlDInfo->nbBits; + uint ofnbBits = ofDInfo->nbBits; + assert(llBits <= 16); + assert(mlBits <= 16); + assert(ofBits <= 31); + { + nuint offset; + if (ofBits > 1) + { + if (MEM_32bits && isLongOffset != default && ofBits >= 25) + { + /* Always read extra bits, this keeps the logic simple, + * avoids branches, and avoids accidentally reading 0 bits. + */ + const uint extraBits = 30 - 25; + offset = + ofBase + + ( + BIT_readBitsFast( + seqState_DStream_bitContainer, + ref seqState_DStream_bitsConsumed, + ofBits - extraBits + ) << (int)extraBits + ); + BIT_reloadDStream( + ref seqState_DStream_bitContainer, + ref seqState_DStream_bitsConsumed, + ref seqState_DStream_ptr, + seqState_DStream_start, + seqState_DStream_limitPtr + ); + offset += BIT_readBitsFast( + seqState_DStream_bitContainer, + ref seqState_DStream_bitsConsumed, + extraBits + ); + } + else + { + offset = + ofBase + + BIT_readBitsFast( + seqState_DStream_bitContainer, + ref seqState_DStream_bitsConsumed, + ofBits + ); + if (MEM_32bits) + { + BIT_reloadDStream( + ref seqState_DStream_bitContainer, + ref seqState_DStream_bitsConsumed, + ref seqState_DStream_ptr, + seqState_DStream_start, + seqState_DStream_limitPtr + ); + } + } + + seqState.prevOffset.e2 = seqState.prevOffset.e1; + seqState.prevOffset.e1 = seqState.prevOffset.e0; + seqState.prevOffset.e0 = offset; + } + else + { + uint ll0 = llDInfo->baseValue == 0 ? 1U : 0U; + if (ofBits == 0) + { + offset = System.Runtime.CompilerServices.Unsafe.Add( + ref seqState.prevOffset.e0, + (int)ll0 + ); + seqState.prevOffset.e1 = System.Runtime.CompilerServices.Unsafe.Add( + ref seqState.prevOffset.e0, + ll0 == 0 ? 1 : 0 + ); + seqState.prevOffset.e0 = offset; + } + else + { + offset = + ofBase + + ll0 + + BIT_readBitsFast( + seqState_DStream_bitContainer, + ref seqState_DStream_bitsConsumed, + 1 + ); + { + nuint temp = + offset == 3 + ? seqState.prevOffset.e0 - 1 + : System.Runtime.CompilerServices.Unsafe.Add( + ref seqState.prevOffset.e0, + (int)offset + ); + temp -= temp == 0 ? 1U : 0U; + if (offset != 1) + { + seqState.prevOffset.e2 = seqState.prevOffset.e1; + } + + seqState.prevOffset.e1 = seqState.prevOffset.e0; + seqState.prevOffset.e0 = offset = temp; + } + } + } + + sequence_offset = offset; + } + + if (mlBits > 0) + { + sequence_matchLength += BIT_readBitsFast( + seqState_DStream_bitContainer, + ref seqState_DStream_bitsConsumed, + mlBits + ); + } + + if (MEM_32bits && mlBits + llBits >= 25 - (30 - 25)) + { + BIT_reloadDStream( + ref seqState_DStream_bitContainer, + ref seqState_DStream_bitsConsumed, + ref seqState_DStream_ptr, + seqState_DStream_start, + seqState_DStream_limitPtr + ); + } + + if (MEM_64bits && totalBits >= 57 - (9 + 9 + 8)) + { + BIT_reloadDStream( + ref seqState_DStream_bitContainer, + ref seqState_DStream_bitsConsumed, + ref seqState_DStream_ptr, + seqState_DStream_start, + seqState_DStream_limitPtr + ); + } + + if (llBits > 0) + { + sequence_litLength += BIT_readBitsFast( + seqState_DStream_bitContainer, + ref seqState_DStream_bitsConsumed, + llBits + ); + } + + if (MEM_32bits) + { + BIT_reloadDStream( + ref seqState_DStream_bitContainer, + ref seqState_DStream_bitsConsumed, + ref seqState_DStream_ptr, + seqState_DStream_start, + seqState_DStream_limitPtr + ); + } + + if ((nbSeq == 1 ? 1 : 0) == 0) + { + ZSTD_updateFseStateWithDInfo( + ref seqState.stateLL, + seqState_DStream_bitContainer, + ref seqState_DStream_bitsConsumed, + llNext, + llnbBits + ); + ZSTD_updateFseStateWithDInfo( + ref seqState.stateML, + seqState_DStream_bitContainer, + ref seqState_DStream_bitsConsumed, + mlNext, + mlnbBits + ); + if (MEM_32bits) + { + BIT_reloadDStream( + ref seqState_DStream_bitContainer, + ref seqState_DStream_bitsConsumed, + ref seqState_DStream_ptr, + seqState_DStream_start, + seqState_DStream_limitPtr + ); + } + + ZSTD_updateFseStateWithDInfo( + ref seqState.stateOffb, + seqState_DStream_bitContainer, + ref seqState_DStream_bitsConsumed, + ofNext, + ofnbBits + ); + BIT_reloadDStream( + ref seqState_DStream_bitContainer, + ref seqState_DStream_bitsConsumed, + ref seqState_DStream_ptr, + seqState_DStream_start, + seqState_DStream_limitPtr + ); + } + } + + nuint oneSeqSize; + { + byte* oLitEnd = op + sequence_litLength; + oneSeqSize = sequence_litLength + sequence_matchLength; + /* risk : address space overflow (32-bits) */ + byte* oMatchEnd = op + oneSeqSize; + /* risk : address space underflow on oend=NULL */ + byte* oend_w = oend - 32; + byte* iLitEnd = litPtr + sequence_litLength; + byte* match = oLitEnd - sequence_offset; + assert(op != null); + assert(oend_w < oend); + if ( + iLitEnd > litEnd + || oMatchEnd > oend_w + || MEM_32bits && (nuint)(oend - op) < oneSeqSize + 32 + ) + { + oneSeqSize = ZSTD_execSequenceEnd( + op, + oend, + new seq_t + { + litLength = sequence_litLength, + matchLength = sequence_matchLength, + offset = sequence_offset, + }, + &litPtr, + litEnd, + prefixStart, + vBase, + dictEnd + ); + goto returnOneSeqSize; + } + + assert(op <= oLitEnd); + assert(oLitEnd < oMatchEnd); + assert(oMatchEnd <= oend); + assert(iLitEnd <= litEnd); + assert(oLitEnd <= oend_w); + assert(oMatchEnd <= oend_w); + assert(32 >= 16); + ZSTD_copy16(op, litPtr); + if (sequence_litLength > 16) + { + ZSTD_wildcopy( + op + 16, + litPtr + 16, + (nint)(sequence_litLength - 16), + ZSTD_overlap_e.ZSTD_no_overlap + ); + } + + byte* opInner = oLitEnd; + litPtr = iLitEnd; + if (sequence_offset > (nuint)(oLitEnd - prefixStart)) + { + if (sequence_offset > (nuint)(oLitEnd - vBase)) + { + oneSeqSize = unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected) + ); + goto returnOneSeqSize; + } + + match = dictEnd + (match - prefixStart); + if (match + sequence_matchLength <= dictEnd) + { + memmove(oLitEnd, match, sequence_matchLength); + goto returnOneSeqSize; + } + + { + nuint length1 = (nuint)(dictEnd - match); + memmove(oLitEnd, match, length1); + opInner = oLitEnd + length1; + sequence_matchLength -= length1; + match = prefixStart; + } + } + + assert(opInner <= oMatchEnd); + assert(oMatchEnd <= oend_w); + assert(match >= prefixStart); + assert(sequence_matchLength >= 1); + if (sequence_offset >= 16) + { + ZSTD_wildcopy( + opInner, + match, + (nint)sequence_matchLength, + ZSTD_overlap_e.ZSTD_no_overlap + ); + goto returnOneSeqSize; + } + + assert(sequence_offset < 16); + ZSTD_overlapCopy8(ref opInner, ref match, sequence_offset); + if (sequence_matchLength > 8) + { + assert(opInner < oMatchEnd); + ZSTD_wildcopy( + opInner, + match, + (nint)sequence_matchLength - 8, + ZSTD_overlap_e.ZSTD_overlap_src_before_dst + ); + } + + returnOneSeqSize: + ; + } + + if (ERR_isError(oneSeqSize)) + { + return oneSeqSize; + } + + op += oneSeqSize; + } + + assert(nbSeq == 0); + if ( + BIT_endOfDStream( + seqState_DStream_bitsConsumed, + seqState_DStream_ptr, + seqState_DStream_start + ) == 0 + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + { + uint i; + for (i = 0; i < 3; i++) + { + dctx->entropy.rep[i] = (uint) + System.Runtime.CompilerServices.Unsafe.Add( + ref seqState.prevOffset.e0, + (int)i + ); + } + } + } + + { + nuint lastLLSize = (nuint)(litEnd - litPtr); + if (lastLLSize > (nuint)(oend - op)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + if (op != null) + { + memcpy(op, litPtr, (uint)lastLLSize); + op += lastLLSize; + } + } + + return (nuint)(op - ostart); + } + + private static nuint ZSTD_decompressSequences_default( + ZSTD_DCtx_s* dctx, + void* dst, + nuint maxDstSize, + void* seqStart, + nuint seqSize, + int nbSeq, + ZSTD_longOffset_e isLongOffset + ) + { + return ZSTD_decompressSequences_body( + dctx, + dst, + maxDstSize, + seqStart, + seqSize, + nbSeq, + isLongOffset + ); + } + + private static nuint ZSTD_decompressSequencesSplitLitBuffer_default( + ZSTD_DCtx_s* dctx, + void* dst, + nuint maxDstSize, + void* seqStart, + nuint seqSize, + int nbSeq, + ZSTD_longOffset_e isLongOffset + ) + { + return ZSTD_decompressSequences_bodySplitLitBuffer( + dctx, + dst, + maxDstSize, + seqStart, + seqSize, + nbSeq, + isLongOffset + ); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_prefetchMatch( + nuint prefetchPos, + seq_t sequence, + byte* prefixStart, + byte* dictEnd + ) + { + prefetchPos += sequence.litLength; + { + byte* matchBase = sequence.offset > prefetchPos ? dictEnd : prefixStart; + /* note : this operation can overflow when seq.offset is really too large, which can only happen when input is corrupted. + * No consequence though : memory address is only used for prefetching, not for dereferencing */ + byte* match = ZSTD_wrappedPtrSub( + ZSTD_wrappedPtrAdd(matchBase, (nint)prefetchPos), + (nint)sequence.offset + ); +#if NETCOREAPP3_0_OR_GREATER + if (System.Runtime.Intrinsics.X86.Sse.IsSupported) + { + System.Runtime.Intrinsics.X86.Sse.Prefetch0(match); + System.Runtime.Intrinsics.X86.Sse.Prefetch0(match + 64); + } +#endif + } + + return prefetchPos + sequence.matchLength; + } + + /* This decoding function employs prefetching + * to reduce latency impact of cache misses. + * It's generally employed when block contains a significant portion of long-distance matches + * or when coupled with a "cold" dictionary */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_decompressSequencesLong_body( + ZSTD_DCtx_s* dctx, + void* dst, + nuint maxDstSize, + void* seqStart, + nuint seqSize, + int nbSeq, + ZSTD_longOffset_e isLongOffset + ) + { + byte* ip = (byte*)seqStart; + byte* iend = ip + seqSize; + byte* ostart = (byte*)dst; + byte* oend = + dctx->litBufferLocation == ZSTD_litLocation_e.ZSTD_in_dst + ? dctx->litBuffer + : ZSTD_maybeNullPtrAdd(ostart, (nint)maxDstSize); + byte* op = ostart; + byte* litPtr = dctx->litPtr; + byte* litBufferEnd = dctx->litBufferEnd; + byte* prefixStart = (byte*)dctx->prefixStart; + byte* dictStart = (byte*)dctx->virtualStart; + byte* dictEnd = (byte*)dctx->dictEnd; + if (nbSeq != 0) + { + seq_t* sequences = stackalloc seq_t[8]; + int seqAdvance = nbSeq < 8 ? nbSeq : 8; + seqState_t seqState; + int seqNb; + /* track position relative to prefixStart */ + nuint prefetchPos = (nuint)(op - prefixStart); + dctx->fseEntropy = 1; + { + int i; + for (i = 0; i < 3; i++) + { + (&seqState.prevOffset.e0)[i] = dctx->entropy.rep[i]; + } + } + + assert(dst != null); + assert(iend >= ip); + if (ERR_isError(BIT_initDStream(&seqState.DStream, ip, (nuint)(iend - ip)))) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + ZSTD_initFseState(&seqState.stateLL, &seqState.DStream, dctx->LLTptr); + ZSTD_initFseState(&seqState.stateOffb, &seqState.DStream, dctx->OFTptr); + ZSTD_initFseState(&seqState.stateML, &seqState.DStream, dctx->MLTptr); + for (seqNb = 0; seqNb < seqAdvance; seqNb++) + { + seq_t sequence = ZSTD_decodeSequence( + &seqState, + isLongOffset, + seqNb == nbSeq - 1 ? 1 : 0 + ); + prefetchPos = ZSTD_prefetchMatch(prefetchPos, sequence, prefixStart, dictEnd); + sequences[seqNb] = sequence; + } + + for (; seqNb < nbSeq; seqNb++) + { + seq_t sequence = ZSTD_decodeSequence( + &seqState, + isLongOffset, + seqNb == nbSeq - 1 ? 1 : 0 + ); + if ( + dctx->litBufferLocation == ZSTD_litLocation_e.ZSTD_split + && litPtr + sequences[seqNb - 8 & 8 - 1].litLength > dctx->litBufferEnd + ) + { + /* lit buffer is reaching split point, empty out the first buffer and transition to litExtraBuffer */ + nuint leftoverLit = (nuint)(dctx->litBufferEnd - litPtr); + if (leftoverLit != 0) + { + if (leftoverLit > (nuint)(oend - op)) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall) + ); + } + + ZSTD_safecopyDstBeforeSrc(op, litPtr, (nint)leftoverLit); + sequences[seqNb - 8 & 8 - 1].litLength -= leftoverLit; + op += leftoverLit; + } + + litPtr = dctx->litExtraBuffer; + litBufferEnd = dctx->litExtraBuffer + (1 << 16); + dctx->litBufferLocation = ZSTD_litLocation_e.ZSTD_not_in_dst; + { + nuint oneSeqSize = ZSTD_execSequence( + op, + oend, + sequences[seqNb - 8 & 8 - 1], + &litPtr, + litBufferEnd, + prefixStart, + dictStart, + dictEnd + ); + if (ERR_isError(oneSeqSize)) + { + return oneSeqSize; + } + + prefetchPos = ZSTD_prefetchMatch( + prefetchPos, + sequence, + prefixStart, + dictEnd + ); + sequences[seqNb & 8 - 1] = sequence; + op += oneSeqSize; + } + } + else + { + /* lit buffer is either wholly contained in first or second split, or not split at all*/ + nuint oneSeqSize = + dctx->litBufferLocation == ZSTD_litLocation_e.ZSTD_split + ? ZSTD_execSequenceSplitLitBuffer( + op, + oend, + litPtr + sequences[seqNb - 8 & 8 - 1].litLength - 32, + sequences[seqNb - 8 & 8 - 1], + &litPtr, + litBufferEnd, + prefixStart, + dictStart, + dictEnd + ) + : ZSTD_execSequence( + op, + oend, + sequences[seqNb - 8 & 8 - 1], + &litPtr, + litBufferEnd, + prefixStart, + dictStart, + dictEnd + ); + if (ERR_isError(oneSeqSize)) + { + return oneSeqSize; + } + + prefetchPos = ZSTD_prefetchMatch(prefetchPos, sequence, prefixStart, dictEnd); + sequences[seqNb & 8 - 1] = sequence; + op += oneSeqSize; + } + } + + if (BIT_endOfDStream(&seqState.DStream) == 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_corruption_detected)); + } + + seqNb -= seqAdvance; + for (; seqNb < nbSeq; seqNb++) + { + seq_t* sequence = &sequences[seqNb & 8 - 1]; + if ( + dctx->litBufferLocation == ZSTD_litLocation_e.ZSTD_split + && litPtr + sequence->litLength > dctx->litBufferEnd + ) + { + nuint leftoverLit = (nuint)(dctx->litBufferEnd - litPtr); + if (leftoverLit != 0) + { + if (leftoverLit > (nuint)(oend - op)) + { + return unchecked( + (nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall) + ); + } + + ZSTD_safecopyDstBeforeSrc(op, litPtr, (nint)leftoverLit); + sequence->litLength -= leftoverLit; + op += leftoverLit; + } + + litPtr = dctx->litExtraBuffer; + litBufferEnd = dctx->litExtraBuffer + (1 << 16); + dctx->litBufferLocation = ZSTD_litLocation_e.ZSTD_not_in_dst; + { + nuint oneSeqSize = ZSTD_execSequence( + op, + oend, + *sequence, + &litPtr, + litBufferEnd, + prefixStart, + dictStart, + dictEnd + ); + if (ERR_isError(oneSeqSize)) + { + return oneSeqSize; + } + + op += oneSeqSize; + } + } + else + { + nuint oneSeqSize = + dctx->litBufferLocation == ZSTD_litLocation_e.ZSTD_split + ? ZSTD_execSequenceSplitLitBuffer( + op, + oend, + litPtr + sequence->litLength - 32, + *sequence, + &litPtr, + litBufferEnd, + prefixStart, + dictStart, + dictEnd + ) + : ZSTD_execSequence( + op, + oend, + *sequence, + &litPtr, + litBufferEnd, + prefixStart, + dictStart, + dictEnd + ); + if (ERR_isError(oneSeqSize)) + { + return oneSeqSize; + } + + op += oneSeqSize; + } + } + + { + uint i; + for (i = 0; i < 3; i++) + { + dctx->entropy.rep[i] = (uint)(&seqState.prevOffset.e0)[i]; + } + } + } + + if (dctx->litBufferLocation == ZSTD_litLocation_e.ZSTD_split) + { + nuint lastLLSize = (nuint)(litBufferEnd - litPtr); + if (lastLLSize > (nuint)(oend - op)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + if (op != null) + { + memmove(op, litPtr, lastLLSize); + op += lastLLSize; + } + + litPtr = dctx->litExtraBuffer; + litBufferEnd = dctx->litExtraBuffer + (1 << 16); + } + + { + nuint lastLLSize = (nuint)(litBufferEnd - litPtr); + if (lastLLSize > (nuint)(oend - op)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + if (op != null) + { + memmove(op, litPtr, lastLLSize); + op += lastLLSize; + } + } + + return (nuint)(op - ostart); + } + + private static nuint ZSTD_decompressSequencesLong_default( + ZSTD_DCtx_s* dctx, + void* dst, + nuint maxDstSize, + void* seqStart, + nuint seqSize, + int nbSeq, + ZSTD_longOffset_e isLongOffset + ) + { + return ZSTD_decompressSequencesLong_body( + dctx, + dst, + maxDstSize, + seqStart, + seqSize, + nbSeq, + isLongOffset + ); + } + + private static nuint ZSTD_decompressSequences( + ZSTD_DCtx_s* dctx, + void* dst, + nuint maxDstSize, + void* seqStart, + nuint seqSize, + int nbSeq, + ZSTD_longOffset_e isLongOffset + ) + { + return ZSTD_decompressSequences_default( + dctx, + dst, + maxDstSize, + seqStart, + seqSize, + nbSeq, + isLongOffset + ); + } + + private static nuint ZSTD_decompressSequencesSplitLitBuffer( + ZSTD_DCtx_s* dctx, + void* dst, + nuint maxDstSize, + void* seqStart, + nuint seqSize, + int nbSeq, + ZSTD_longOffset_e isLongOffset + ) + { + return ZSTD_decompressSequencesSplitLitBuffer_default( + dctx, + dst, + maxDstSize, + seqStart, + seqSize, + nbSeq, + isLongOffset + ); + } + + /* ZSTD_decompressSequencesLong() : + * decompression function triggered when a minimum share of offsets is considered "long", + * aka out of cache. + * note : "long" definition seems overloaded here, sometimes meaning "wider than bitstream register", and sometimes meaning "farther than memory cache distance". + * This function will try to mitigate main memory latency through the use of prefetching */ + private static nuint ZSTD_decompressSequencesLong( + ZSTD_DCtx_s* dctx, + void* dst, + nuint maxDstSize, + void* seqStart, + nuint seqSize, + int nbSeq, + ZSTD_longOffset_e isLongOffset + ) + { + return ZSTD_decompressSequencesLong_default( + dctx, + dst, + maxDstSize, + seqStart, + seqSize, + nbSeq, + isLongOffset + ); + } + + /** + * @returns The total size of the history referenceable by zstd, including + * both the prefix and the extDict. At @p op any offset larger than this + * is invalid. + */ + private static nuint ZSTD_totalHistorySize(byte* op, byte* virtualStart) + { + return (nuint)(op - virtualStart); + } + + /* ZSTD_getOffsetInfo() : + * condition : offTable must be valid + * @return : "share" of long offsets (arbitrarily defined as > (1<<23)) + * compared to maximum possible of (1< table[u].nbAdditionalBits + ? info.maxNbAdditionalBits + : table[u].nbAdditionalBits; + if (table[u].nbAdditionalBits > 22) + { + info.longOffsetShare += 1; + } + } + + assert(tableLog <= 8); + info.longOffsetShare <<= (int)(8 - tableLog); + } + + return info; + } + + /** + * @returns The maximum offset we can decode in one read of our bitstream, without + * reloading more bits in the middle of the offset bits read. Any offsets larger + * than this must use the long offset decoder. + */ + private static nuint ZSTD_maxShortOffset() + { + if (MEM_64bits) + { + return unchecked((nuint)(-1)); + } + else + { + /* The maximum offBase is (1 << (STREAM_ACCUMULATOR_MIN + 1)) - 1. + * This offBase would require STREAM_ACCUMULATOR_MIN extra bits. + * Then we have to subtract ZSTD_REP_NUM to get the maximum possible offset. + */ + nuint maxOffbase = ((nuint)1 << (int)((uint)(MEM_32bits ? 25 : 57) + 1)) - 1; + nuint maxOffset = maxOffbase - 3; + assert(ZSTD_highbit32((uint)maxOffbase) == (uint)(MEM_32bits ? 25 : 57)); + return maxOffset; + } + } + + /* ZSTD_decompressBlock_internal() : + * decompress block, starting at `src`, + * into destination buffer `dst`. + * @return : decompressed block size, + * or an error code (which can be tested using ZSTD_isError()) + */ + private static nuint ZSTD_decompressBlock_internal( + ZSTD_DCtx_s* dctx, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize, + streaming_operation streaming + ) + { + byte* ip = (byte*)src; + if (srcSize > ZSTD_blockSizeMax(dctx)) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_srcSize_wrong)); + } + + { + nuint litCSize = ZSTD_decodeLiteralsBlock( + dctx, + src, + srcSize, + dst, + dstCapacity, + streaming + ); + if (ERR_isError(litCSize)) + { + return litCSize; + } + + ip += litCSize; + srcSize -= litCSize; + } + + { + /* Compute the maximum block size, which must also work when !frame and fParams are unset. + * Additionally, take the min with dstCapacity to ensure that the totalHistorySize fits in a size_t. + */ + nuint blockSizeMax = + dstCapacity < ZSTD_blockSizeMax(dctx) ? dstCapacity : ZSTD_blockSizeMax(dctx); + nuint totalHistorySize = ZSTD_totalHistorySize( + ZSTD_maybeNullPtrAdd((byte*)dst, (nint)blockSizeMax), + (byte*)dctx->virtualStart + ); + /* isLongOffset must be true if there are long offsets. + * Offsets are long if they are larger than ZSTD_maxShortOffset(). + * We don't expect that to be the case in 64-bit mode. + * + * We check here to see if our history is large enough to allow long offsets. + * If it isn't, then we can't possible have (valid) long offsets. If the offset + * is invalid, then it is okay to read it incorrectly. + * + * If isLongOffsets is true, then we will later check our decoding table to see + * if it is even possible to generate long offsets. + */ + ZSTD_longOffset_e isLongOffset = (ZSTD_longOffset_e)( + MEM_32bits && totalHistorySize > ZSTD_maxShortOffset() ? 1 : 0 + ); + int usePrefetchDecoder = dctx->ddictIsCold; + int nbSeq; + nuint seqHSize = ZSTD_decodeSeqHeaders(dctx, &nbSeq, ip, srcSize); + if (ERR_isError(seqHSize)) + { + return seqHSize; + } + + ip += seqHSize; + srcSize -= seqHSize; + if ((dst == null || dstCapacity == 0) && nbSeq > 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + if ( + MEM_64bits + && sizeof(nuint) == sizeof(void*) + && unchecked((nuint)(-1)) - (nuint)dst < 1 << 20 + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + if ( + isLongOffset != default + || usePrefetchDecoder == 0 && totalHistorySize > 1U << 24 && nbSeq > 8 + ) + { + ZSTD_OffsetInfo info = ZSTD_getOffsetInfo(dctx->OFTptr, nbSeq); + if ( + isLongOffset != default + && info.maxNbAdditionalBits <= (uint)(MEM_32bits ? 25 : 57) + ) + { + isLongOffset = ZSTD_longOffset_e.ZSTD_lo_isRegularOffset; + } + + if (usePrefetchDecoder == 0) + { + /* heuristic values, correspond to 2.73% and 7.81% */ + uint minShare = (uint)(MEM_64bits ? 7 : 20); + usePrefetchDecoder = info.longOffsetShare >= minShare ? 1 : 0; + } + } + + dctx->ddictIsCold = 0; + if (usePrefetchDecoder != 0) + { + return ZSTD_decompressSequencesLong( + dctx, + dst, + dstCapacity, + ip, + srcSize, + nbSeq, + isLongOffset + ); + } + + if (dctx->litBufferLocation == ZSTD_litLocation_e.ZSTD_split) + { + return ZSTD_decompressSequencesSplitLitBuffer( + dctx, + dst, + dstCapacity, + ip, + srcSize, + nbSeq, + isLongOffset + ); + } + else + { + return ZSTD_decompressSequences( + dctx, + dst, + dstCapacity, + ip, + srcSize, + nbSeq, + isLongOffset + ); + } + } + } + + /*! ZSTD_checkContinuity() : + * check if next `dst` follows previous position, where decompression ended. + * If yes, do nothing (continue on current segment). + * If not, classify previous segment as "external dictionary", and start a new segment. + * This function cannot fail. */ + private static void ZSTD_checkContinuity(ZSTD_DCtx_s* dctx, void* dst, nuint dstSize) + { + if (dst != dctx->previousDstEnd && dstSize > 0) + { + dctx->dictEnd = dctx->previousDstEnd; + dctx->virtualStart = + (sbyte*)dst - ((sbyte*)dctx->previousDstEnd - (sbyte*)dctx->prefixStart); + dctx->prefixStart = dst; + dctx->previousDstEnd = dst; + } + } + + /* Internal definition of ZSTD_decompressBlock() to avoid deprecation warnings. */ + private static nuint ZSTD_decompressBlock_deprecated( + ZSTD_DCtx_s* dctx, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize + ) + { + nuint dSize; + dctx->isFrameDecompression = 0; + ZSTD_checkContinuity(dctx, dst, dstCapacity); + dSize = ZSTD_decompressBlock_internal( + dctx, + dst, + dstCapacity, + src, + srcSize, + streaming_operation.not_streaming + ); + { + nuint err_code = dSize; + if (ERR_isError(err_code)) + { + return err_code; + } + } + + dctx->previousDstEnd = (sbyte*)dst + dSize; + return dSize; + } + + /* NOTE: Must just wrap ZSTD_decompressBlock_deprecated() */ + public static nuint ZSTD_decompressBlock( + ZSTD_DCtx_s* dctx, + void* dst, + nuint dstCapacity, + void* src, + nuint srcSize + ) + { + return ZSTD_decompressBlock_deprecated(dctx, dst, dstCapacity, src, srcSize); + } + + private static void ZSTD_initFseState( + ref ZSTD_fseState DStatePtr, + ref BIT_DStream_t bitD, + ZSTD_seqSymbol* dt + ) + { + void* ptr = dt; + ZSTD_seqSymbol_header* DTableH = (ZSTD_seqSymbol_header*)ptr; + DStatePtr.state = BIT_readBits(bitD.bitContainer, ref bitD.bitsConsumed, DTableH->tableLog); + BIT_reloadDStream( + ref bitD.bitContainer, + ref bitD.bitsConsumed, + ref bitD.ptr, + bitD.start, + bitD.limitPtr + ); + DStatePtr.table = dt + 1; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_updateFseStateWithDInfo( + ref ZSTD_fseState DStatePtr, + nuint bitD_bitContainer, + ref uint bitD_bitsConsumed, + ushort nextState, + uint nbBits + ) + { + nuint lowBits = BIT_readBits(bitD_bitContainer, ref bitD_bitsConsumed, nbBits); + DStatePtr.state = nextState + lowBits; + } + + /*! ZSTD_overlapCopy8() : + * Copies 8 bytes from ip to op and updates op and ip where ip <= op. + * If the offset is < 8 then the offset is spread to at least 8 bytes. + * + * Precondition: *ip <= *op + * Postcondition: *op - *op >= 8 + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_overlapCopy8(ref byte* op, ref byte* ip, nuint offset) + { + assert(ip <= op); + if (offset < 8) + { + int sub2 = dec64table[offset]; + op[0] = ip[0]; + op[1] = ip[1]; + op[2] = ip[2]; + op[3] = ip[3]; + ip += dec32table[offset]; + ZSTD_copy4(op + 4, ip); + ip -= sub2; + } + else + { + ZSTD_copy8(op, ip); + } + + ip += 8; + op += 8; + assert(op - ip >= 8); + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdDecompressInternal.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdDecompressInternal.cs new file mode 100644 index 00000000..670f5846 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdDecompressInternal.cs @@ -0,0 +1,394 @@ +using System; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_LL_base => + new uint[36] + { + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 18, + 20, + 22, + 24, + 28, + 32, + 40, + 48, + 64, + 0x80, + 0x100, + 0x200, + 0x400, + 0x800, + 0x1000, + 0x2000, + 0x4000, + 0x8000, + 0x10000, + }; + private static uint* LL_base => + (uint*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_LL_base) + ); +#else + + private static readonly uint* LL_base = GetArrayPointer( + new uint[36] + { + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 18, + 20, + 22, + 24, + 28, + 32, + 40, + 48, + 64, + 0x80, + 0x100, + 0x200, + 0x400, + 0x800, + 0x1000, + 0x2000, + 0x4000, + 0x8000, + 0x10000, + } + ); +#endif +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_OF_base => + new uint[32] + { + 0, + 1, + 1, + 5, + 0xD, + 0x1D, + 0x3D, + 0x7D, + 0xFD, + 0x1FD, + 0x3FD, + 0x7FD, + 0xFFD, + 0x1FFD, + 0x3FFD, + 0x7FFD, + 0xFFFD, + 0x1FFFD, + 0x3FFFD, + 0x7FFFD, + 0xFFFFD, + 0x1FFFFD, + 0x3FFFFD, + 0x7FFFFD, + 0xFFFFFD, + 0x1FFFFFD, + 0x3FFFFFD, + 0x7FFFFFD, + 0xFFFFFFD, + 0x1FFFFFFD, + 0x3FFFFFFD, + 0x7FFFFFFD, + }; + private static uint* OF_base => + (uint*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_OF_base) + ); +#else + + private static readonly uint* OF_base = GetArrayPointer( + new uint[32] + { + 0, + 1, + 1, + 5, + 0xD, + 0x1D, + 0x3D, + 0x7D, + 0xFD, + 0x1FD, + 0x3FD, + 0x7FD, + 0xFFD, + 0x1FFD, + 0x3FFD, + 0x7FFD, + 0xFFFD, + 0x1FFFD, + 0x3FFFD, + 0x7FFFD, + 0xFFFFD, + 0x1FFFFD, + 0x3FFFFD, + 0x7FFFFD, + 0xFFFFFD, + 0x1FFFFFD, + 0x3FFFFFD, + 0x7FFFFFD, + 0xFFFFFFD, + 0x1FFFFFFD, + 0x3FFFFFFD, + 0x7FFFFFFD, + } + ); +#endif +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_OF_bits => + new byte[32] + { + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + }; + private static byte* OF_bits => + (byte*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_OF_bits) + ); +#else + + private static readonly byte* OF_bits = GetArrayPointer( + new byte[32] + { + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + } + ); +#endif +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_ML_base => + new uint[53] + { + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 37, + 39, + 41, + 43, + 47, + 51, + 59, + 67, + 83, + 99, + 0x83, + 0x103, + 0x203, + 0x403, + 0x803, + 0x1003, + 0x2003, + 0x4003, + 0x8003, + 0x10003, + }; + private static uint* ML_base => + (uint*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_ML_base) + ); +#else + + private static readonly uint* ML_base = GetArrayPointer( + new uint[53] + { + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 37, + 39, + 41, + 43, + 47, + 51, + 59, + 67, + 83, + 99, + 0x83, + 0x103, + 0x203, + 0x403, + 0x803, + 0x1003, + 0x2003, + 0x4003, + 0x8003, + 0x10003, + } + ); +#endif + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static int ZSTD_DCtx_get_bmi2(ZSTD_DCtx_s* dctx) + { + return 0; + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdDoubleFast.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdDoubleFast.cs new file mode 100644 index 00000000..c7d37df6 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdDoubleFast.cs @@ -0,0 +1,1131 @@ +using System; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + private static void ZSTD_fillDoubleHashTableForCDict( + ZSTD_MatchState_t* ms, + void* end, + ZSTD_dictTableLoadMethod_e dtlm + ) + { + ZSTD_compressionParameters* cParams = &ms->cParams; + uint* hashLarge = ms->hashTable; + uint hBitsL = cParams->hashLog + 8; + uint mls = cParams->minMatch; + uint* hashSmall = ms->chainTable; + uint hBitsS = cParams->chainLog + 8; + byte* @base = ms->window.@base; + byte* ip = @base + ms->nextToUpdate; + byte* iend = (byte*)end - 8; + const uint fastHashFillStep = 3; + for (; ip + fastHashFillStep - 1 <= iend; ip += fastHashFillStep) + { + uint curr = (uint)(ip - @base); + uint i; + for (i = 0; i < fastHashFillStep; ++i) + { + nuint smHashAndTag = ZSTD_hashPtr(ip + i, hBitsS, mls); + nuint lgHashAndTag = ZSTD_hashPtr(ip + i, hBitsL, 8); + if (i == 0) + { + ZSTD_writeTaggedIndex(hashSmall, smHashAndTag, curr + i); + } + + if (i == 0 || hashLarge[lgHashAndTag >> 8] == 0) + { + ZSTD_writeTaggedIndex(hashLarge, lgHashAndTag, curr + i); + } + + if (dtlm == ZSTD_dictTableLoadMethod_e.ZSTD_dtlm_fast) + { + break; + } + } + } + } + + private static void ZSTD_fillDoubleHashTableForCCtx( + ZSTD_MatchState_t* ms, + void* end, + ZSTD_dictTableLoadMethod_e dtlm + ) + { + ZSTD_compressionParameters* cParams = &ms->cParams; + uint* hashLarge = ms->hashTable; + uint hBitsL = cParams->hashLog; + uint mls = cParams->minMatch; + uint* hashSmall = ms->chainTable; + uint hBitsS = cParams->chainLog; + byte* @base = ms->window.@base; + byte* ip = @base + ms->nextToUpdate; + byte* iend = (byte*)end - 8; + const uint fastHashFillStep = 3; + for (; ip + fastHashFillStep - 1 <= iend; ip += fastHashFillStep) + { + uint curr = (uint)(ip - @base); + uint i; + for (i = 0; i < fastHashFillStep; ++i) + { + nuint smHash = ZSTD_hashPtr(ip + i, hBitsS, mls); + nuint lgHash = ZSTD_hashPtr(ip + i, hBitsL, 8); + if (i == 0) + { + hashSmall[smHash] = curr + i; + } + + if (i == 0 || hashLarge[lgHash] == 0) + { + hashLarge[lgHash] = curr + i; + } + + if (dtlm == ZSTD_dictTableLoadMethod_e.ZSTD_dtlm_fast) + { + break; + } + } + } + } + + private static void ZSTD_fillDoubleHashTable( + ZSTD_MatchState_t* ms, + void* end, + ZSTD_dictTableLoadMethod_e dtlm, + ZSTD_tableFillPurpose_e tfp + ) + { + if (tfp == ZSTD_tableFillPurpose_e.ZSTD_tfp_forCDict) + { + ZSTD_fillDoubleHashTableForCDict(ms, end, dtlm); + } + else + { + ZSTD_fillDoubleHashTableForCCtx(ms, end, dtlm); + } + } + +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_dummy => + new byte[10] { 0x12, 0x34, 0x56, 0x78, 0x9a, 0xbc, 0xde, 0xf0, 0xe2, 0xb4 }; + private static byte* dummy => + (byte*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_dummy) + ); +#else + + private static readonly byte* dummy = GetArrayPointer( + new byte[10] { 0x12, 0x34, 0x56, 0x78, 0x9a, 0xbc, 0xde, 0xf0, 0xe2, 0xb4 } + ); +#endif + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_compressBlock_doubleFast_noDict_generic( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize, + uint mls + ) + { + ZSTD_compressionParameters* cParams = &ms->cParams; + uint* hashLong = ms->hashTable; + uint hBitsL = cParams->hashLog; + uint* hashSmall = ms->chainTable; + uint hBitsS = cParams->chainLog; + byte* @base = ms->window.@base; + byte* istart = (byte*)src; + byte* anchor = istart; + uint endIndex = (uint)((nuint)(istart - @base) + srcSize); + /* presumes that, if there is a dictionary, it must be using Attach mode */ + uint prefixLowestIndex = ZSTD_getLowestPrefixIndex(ms, endIndex, cParams->windowLog); + byte* prefixLowest = @base + prefixLowestIndex; + byte* iend = istart + srcSize; + byte* ilimit = iend - 8; + uint offset_1 = rep[0], + offset_2 = rep[1]; + uint offsetSaved1 = 0, + offsetSaved2 = 0; + nuint mLength; + uint offset; + uint curr; + /* how many positions to search before increasing step size */ + const nuint kStepIncr = 1 << 8; + /* the position at which to increment the step size if no match is found */ + byte* nextStep; + /* the current step size */ + nuint step; + /* the long hash at ip */ + nuint hl0; + /* the long hash at ip1 */ + nuint hl1; + /* the long match index for ip */ + uint idxl0; + /* the long match index for ip1 */ + uint idxl1; + /* the long match for ip */ + byte* matchl0; + /* the short match for ip */ + byte* matchs0; + /* the long match for ip1 */ + byte* matchl1; + /* matchs0 or safe address */ + byte* matchs0_safe; + /* the current position */ + byte* ip = istart; + /* the next position */ + byte* ip1; + ip += ip - prefixLowest == 0 ? 1 : 0; + { + uint current = (uint)(ip - @base); + uint windowLow = ZSTD_getLowestPrefixIndex(ms, current, cParams->windowLog); + uint maxRep = current - windowLow; + if (offset_2 > maxRep) + { + offsetSaved2 = offset_2; + offset_2 = 0; + } + + if (offset_1 > maxRep) + { + offsetSaved1 = offset_1; + offset_1 = 0; + } + } + + while (true) + { + step = 1; + nextStep = ip + kStepIncr; + ip1 = ip + step; + if (ip1 > ilimit) + { + goto _cleanup; + } + + hl0 = ZSTD_hashPtr(ip, hBitsL, 8); + idxl0 = hashLong[hl0]; + matchl0 = @base + idxl0; + do + { + nuint hs0 = ZSTD_hashPtr(ip, hBitsS, mls); + uint idxs0 = hashSmall[hs0]; + curr = (uint)(ip - @base); + matchs0 = @base + idxs0; + hashLong[hl0] = hashSmall[hs0] = curr; + if (offset_1 > 0 && MEM_read32(ip + 1 - offset_1) == MEM_read32(ip + 1)) + { + mLength = ZSTD_count(ip + 1 + 4, ip + 1 + 4 - offset_1, iend) + 4; + ip++; + assert(1 >= 1); + assert(1 <= 3); + ZSTD_storeSeq(seqStore, (nuint)(ip - anchor), anchor, iend, 1, mLength); + goto _match_stored; + } + + hl1 = ZSTD_hashPtr(ip1, hBitsL, 8); + { + byte* matchl0_safe = ZSTD_selectAddr( + idxl0, + prefixLowestIndex, + matchl0, + &dummy[0] + ); + if (MEM_read64(matchl0_safe) == MEM_read64(ip) && matchl0_safe == matchl0) + { + mLength = ZSTD_count(ip + 8, matchl0 + 8, iend) + 8; + offset = (uint)(ip - matchl0); + while (ip > anchor && matchl0 > prefixLowest && ip[-1] == matchl0[-1]) + { + ip--; + matchl0--; + mLength++; + } + + goto _match_found; + } + } + + idxl1 = hashLong[hl1]; + matchl1 = @base + idxl1; + matchs0_safe = ZSTD_selectAddr(idxs0, prefixLowestIndex, matchs0, &dummy[0]); + if (MEM_read32(matchs0_safe) == MEM_read32(ip) && matchs0_safe == matchs0) + { + goto _search_next_long; + } + + if (ip1 >= nextStep) + { +#if NETCOREAPP3_0_OR_GREATER + if (System.Runtime.Intrinsics.X86.Sse.IsSupported) + { + System.Runtime.Intrinsics.X86.Sse.Prefetch0(ip1 + 64); + System.Runtime.Intrinsics.X86.Sse.Prefetch0(ip1 + 128); + } +#endif + + step++; + nextStep += kStepIncr; + } + + ip = ip1; + ip1 += step; + hl0 = hl1; + idxl0 = idxl1; + matchl0 = matchl1; + } while (ip1 <= ilimit); + _cleanup: + offsetSaved2 = offsetSaved1 != 0 && offset_1 != 0 ? offsetSaved1 : offsetSaved2; + rep[0] = offset_1 != 0 ? offset_1 : offsetSaved1; + rep[1] = offset_2 != 0 ? offset_2 : offsetSaved2; + return (nuint)(iend - anchor); + _search_next_long: + mLength = ZSTD_count(ip + 4, matchs0 + 4, iend) + 4; + offset = (uint)(ip - matchs0); + if (idxl1 > prefixLowestIndex && MEM_read64(matchl1) == MEM_read64(ip1)) + { + nuint l1len = ZSTD_count(ip1 + 8, matchl1 + 8, iend) + 8; + if (l1len > mLength) + { + ip = ip1; + mLength = l1len; + offset = (uint)(ip - matchl1); + matchs0 = matchl1; + } + } + + while (ip > anchor && matchs0 > prefixLowest && ip[-1] == matchs0[-1]) + { + ip--; + matchs0--; + mLength++; + } + + _match_found: + offset_2 = offset_1; + offset_1 = offset; + if (step < 4) + { + hashLong[hl1] = (uint)(ip1 - @base); + } + + assert(offset > 0); + ZSTD_storeSeq(seqStore, (nuint)(ip - anchor), anchor, iend, offset + 3, mLength); + _match_stored: + ip += mLength; + anchor = ip; + if (ip <= ilimit) + { + { + uint indexToInsert = curr + 2; + hashLong[ZSTD_hashPtr(@base + indexToInsert, hBitsL, 8)] = indexToInsert; + hashLong[ZSTD_hashPtr(ip - 2, hBitsL, 8)] = (uint)(ip - 2 - @base); + hashSmall[ZSTD_hashPtr(@base + indexToInsert, hBitsS, mls)] = indexToInsert; + hashSmall[ZSTD_hashPtr(ip - 1, hBitsS, mls)] = (uint)(ip - 1 - @base); + } + + while (ip <= ilimit && offset_2 > 0 && MEM_read32(ip) == MEM_read32(ip - offset_2)) + { + /* store sequence */ + nuint rLength = ZSTD_count(ip + 4, ip + 4 - offset_2, iend) + 4; + /* swap offset_2 <=> offset_1 */ + uint tmpOff = offset_2; + offset_2 = offset_1; + offset_1 = tmpOff; + hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = (uint)(ip - @base); + hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = (uint)(ip - @base); + assert(1 >= 1); + assert(1 <= 3); + ZSTD_storeSeq(seqStore, 0, anchor, iend, 1, rLength); + ip += rLength; + anchor = ip; + continue; + } + } + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_compressBlock_doubleFast_dictMatchState_generic( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize, + uint mls + ) + { + ZSTD_compressionParameters* cParams = &ms->cParams; + uint* hashLong = ms->hashTable; + uint hBitsL = cParams->hashLog; + uint* hashSmall = ms->chainTable; + uint hBitsS = cParams->chainLog; + byte* @base = ms->window.@base; + byte* istart = (byte*)src; + byte* ip = istart; + byte* anchor = istart; + uint endIndex = (uint)((nuint)(istart - @base) + srcSize); + /* presumes that, if there is a dictionary, it must be using Attach mode */ + uint prefixLowestIndex = ZSTD_getLowestPrefixIndex(ms, endIndex, cParams->windowLog); + byte* prefixLowest = @base + prefixLowestIndex; + byte* iend = istart + srcSize; + byte* ilimit = iend - 8; + uint offset_1 = rep[0], + offset_2 = rep[1]; + ZSTD_MatchState_t* dms = ms->dictMatchState; + ZSTD_compressionParameters* dictCParams = &dms->cParams; + uint* dictHashLong = dms->hashTable; + uint* dictHashSmall = dms->chainTable; + uint dictStartIndex = dms->window.dictLimit; + byte* dictBase = dms->window.@base; + byte* dictStart = dictBase + dictStartIndex; + byte* dictEnd = dms->window.nextSrc; + uint dictIndexDelta = prefixLowestIndex - (uint)(dictEnd - dictBase); + uint dictHBitsL = dictCParams->hashLog + 8; + uint dictHBitsS = dictCParams->chainLog + 8; + uint dictAndPrefixLength = (uint)(ip - prefixLowest + (dictEnd - dictStart)); + assert(ms->window.dictLimit + (1U << (int)cParams->windowLog) >= endIndex); + if (ms->prefetchCDictTables != 0) + { + nuint hashTableBytes = ((nuint)1 << (int)dictCParams->hashLog) * sizeof(uint); + nuint chainTableBytes = ((nuint)1 << (int)dictCParams->chainLog) * sizeof(uint); + { + sbyte* _ptr = (sbyte*)dictHashLong; + nuint _size = hashTableBytes; + nuint _pos; + for (_pos = 0; _pos < _size; _pos += 64) + { +#if NETCOREAPP3_0_OR_GREATER + if (System.Runtime.Intrinsics.X86.Sse.IsSupported) + { + System.Runtime.Intrinsics.X86.Sse.Prefetch1(_ptr + _pos); + } +#endif + } + } + + { + sbyte* _ptr = (sbyte*)dictHashSmall; + nuint _size = chainTableBytes; + nuint _pos; + for (_pos = 0; _pos < _size; _pos += 64) + { +#if NETCOREAPP3_0_OR_GREATER + if (System.Runtime.Intrinsics.X86.Sse.IsSupported) + { + System.Runtime.Intrinsics.X86.Sse.Prefetch1(_ptr + _pos); + } +#endif + } + } + } + + ip += dictAndPrefixLength == 0 ? 1 : 0; + assert(offset_1 <= dictAndPrefixLength); + assert(offset_2 <= dictAndPrefixLength); + while (ip < ilimit) + { + nuint mLength; + uint offset; + nuint h2 = ZSTD_hashPtr(ip, hBitsL, 8); + nuint h = ZSTD_hashPtr(ip, hBitsS, mls); + nuint dictHashAndTagL = ZSTD_hashPtr(ip, dictHBitsL, 8); + nuint dictHashAndTagS = ZSTD_hashPtr(ip, dictHBitsS, mls); + uint dictMatchIndexAndTagL = dictHashLong[dictHashAndTagL >> 8]; + uint dictMatchIndexAndTagS = dictHashSmall[dictHashAndTagS >> 8]; + int dictTagsMatchL = ZSTD_comparePackedTags(dictMatchIndexAndTagL, dictHashAndTagL); + int dictTagsMatchS = ZSTD_comparePackedTags(dictMatchIndexAndTagS, dictHashAndTagS); + uint curr = (uint)(ip - @base); + uint matchIndexL = hashLong[h2]; + uint matchIndexS = hashSmall[h]; + byte* matchLong = @base + matchIndexL; + byte* match = @base + matchIndexS; + uint repIndex = curr + 1 - offset_1; + byte* repMatch = + repIndex < prefixLowestIndex + ? dictBase + (repIndex - dictIndexDelta) + : @base + repIndex; + hashLong[h2] = hashSmall[h] = curr; + if ( + ZSTD_index_overlap_check(prefixLowestIndex, repIndex) != 0 + && MEM_read32(repMatch) == MEM_read32(ip + 1) + ) + { + byte* repMatchEnd = repIndex < prefixLowestIndex ? dictEnd : iend; + mLength = + ZSTD_count_2segments(ip + 1 + 4, repMatch + 4, iend, repMatchEnd, prefixLowest) + + 4; + ip++; + assert(1 >= 1); + assert(1 <= 3); + ZSTD_storeSeq(seqStore, (nuint)(ip - anchor), anchor, iend, 1, mLength); + goto _match_stored; + } + + if (matchIndexL >= prefixLowestIndex && MEM_read64(matchLong) == MEM_read64(ip)) + { + mLength = ZSTD_count(ip + 8, matchLong + 8, iend) + 8; + offset = (uint)(ip - matchLong); + while (ip > anchor && matchLong > prefixLowest && ip[-1] == matchLong[-1]) + { + ip--; + matchLong--; + mLength++; + } + + goto _match_found; + } + else if (dictTagsMatchL != 0) + { + /* check dictMatchState long match */ + uint dictMatchIndexL = dictMatchIndexAndTagL >> 8; + byte* dictMatchL = dictBase + dictMatchIndexL; + assert(dictMatchL < dictEnd); + if (dictMatchL > dictStart && MEM_read64(dictMatchL) == MEM_read64(ip)) + { + mLength = + ZSTD_count_2segments(ip + 8, dictMatchL + 8, iend, dictEnd, prefixLowest) + + 8; + offset = curr - dictMatchIndexL - dictIndexDelta; + while (ip > anchor && dictMatchL > dictStart && ip[-1] == dictMatchL[-1]) + { + ip--; + dictMatchL--; + mLength++; + } + + goto _match_found; + } + } + + if (matchIndexS > prefixLowestIndex) + { + if (MEM_read32(match) == MEM_read32(ip)) + { + goto _search_next_long; + } + } + else if (dictTagsMatchS != 0) + { + /* check dictMatchState short match */ + uint dictMatchIndexS = dictMatchIndexAndTagS >> 8; + match = dictBase + dictMatchIndexS; + matchIndexS = dictMatchIndexS + dictIndexDelta; + if (match > dictStart && MEM_read32(match) == MEM_read32(ip)) + { + goto _search_next_long; + } + } + + ip += (ip - anchor >> 8) + 1; + continue; + _search_next_long: + { + nuint hl3 = ZSTD_hashPtr(ip + 1, hBitsL, 8); + nuint dictHashAndTagL3 = ZSTD_hashPtr(ip + 1, dictHBitsL, 8); + uint matchIndexL3 = hashLong[hl3]; + uint dictMatchIndexAndTagL3 = dictHashLong[dictHashAndTagL3 >> 8]; + int dictTagsMatchL3 = ZSTD_comparePackedTags( + dictMatchIndexAndTagL3, + dictHashAndTagL3 + ); + byte* matchL3 = @base + matchIndexL3; + hashLong[hl3] = curr + 1; + if (matchIndexL3 >= prefixLowestIndex && MEM_read64(matchL3) == MEM_read64(ip + 1)) + { + mLength = ZSTD_count(ip + 9, matchL3 + 8, iend) + 8; + ip++; + offset = (uint)(ip - matchL3); + while (ip > anchor && matchL3 > prefixLowest && ip[-1] == matchL3[-1]) + { + ip--; + matchL3--; + mLength++; + } + + goto _match_found; + } + else if (dictTagsMatchL3 != 0) + { + /* check dict long +1 match */ + uint dictMatchIndexL3 = dictMatchIndexAndTagL3 >> 8; + byte* dictMatchL3 = dictBase + dictMatchIndexL3; + assert(dictMatchL3 < dictEnd); + if (dictMatchL3 > dictStart && MEM_read64(dictMatchL3) == MEM_read64(ip + 1)) + { + mLength = + ZSTD_count_2segments( + ip + 1 + 8, + dictMatchL3 + 8, + iend, + dictEnd, + prefixLowest + ) + 8; + ip++; + offset = curr + 1 - dictMatchIndexL3 - dictIndexDelta; + while (ip > anchor && dictMatchL3 > dictStart && ip[-1] == dictMatchL3[-1]) + { + ip--; + dictMatchL3--; + mLength++; + } + + goto _match_found; + } + } + } + + if (matchIndexS < prefixLowestIndex) + { + mLength = ZSTD_count_2segments(ip + 4, match + 4, iend, dictEnd, prefixLowest) + 4; + offset = curr - matchIndexS; + while (ip > anchor && match > dictStart && ip[-1] == match[-1]) + { + ip--; + match--; + mLength++; + } + } + else + { + mLength = ZSTD_count(ip + 4, match + 4, iend) + 4; + offset = (uint)(ip - match); + while (ip > anchor && match > prefixLowest && ip[-1] == match[-1]) + { + ip--; + match--; + mLength++; + } + } + + _match_found: + offset_2 = offset_1; + offset_1 = offset; + assert(offset > 0); + ZSTD_storeSeq(seqStore, (nuint)(ip - anchor), anchor, iend, offset + 3, mLength); + _match_stored: + ip += mLength; + anchor = ip; + if (ip <= ilimit) + { + { + uint indexToInsert = curr + 2; + hashLong[ZSTD_hashPtr(@base + indexToInsert, hBitsL, 8)] = indexToInsert; + hashLong[ZSTD_hashPtr(ip - 2, hBitsL, 8)] = (uint)(ip - 2 - @base); + hashSmall[ZSTD_hashPtr(@base + indexToInsert, hBitsS, mls)] = indexToInsert; + hashSmall[ZSTD_hashPtr(ip - 1, hBitsS, mls)] = (uint)(ip - 1 - @base); + } + + while (ip <= ilimit) + { + uint current2 = (uint)(ip - @base); + uint repIndex2 = current2 - offset_2; + byte* repMatch2 = + repIndex2 < prefixLowestIndex + ? dictBase + repIndex2 - dictIndexDelta + : @base + repIndex2; + if ( + ZSTD_index_overlap_check(prefixLowestIndex, repIndex2) != 0 + && MEM_read32(repMatch2) == MEM_read32(ip) + ) + { + byte* repEnd2 = repIndex2 < prefixLowestIndex ? dictEnd : iend; + nuint repLength2 = + ZSTD_count_2segments(ip + 4, repMatch2 + 4, iend, repEnd2, prefixLowest) + + 4; + /* swap offset_2 <=> offset_1 */ + uint tmpOffset = offset_2; + offset_2 = offset_1; + offset_1 = tmpOffset; + assert(1 >= 1); + assert(1 <= 3); + ZSTD_storeSeq(seqStore, 0, anchor, iend, 1, repLength2); + hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = current2; + hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = current2; + ip += repLength2; + anchor = ip; + continue; + } + + break; + } + } + } + + rep[0] = offset_1; + rep[1] = offset_2; + return (nuint)(iend - anchor); + } + + private static nuint ZSTD_compressBlock_doubleFast_noDict_4( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_doubleFast_noDict_generic(ms, seqStore, rep, src, srcSize, 4); + } + + private static nuint ZSTD_compressBlock_doubleFast_noDict_5( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_doubleFast_noDict_generic(ms, seqStore, rep, src, srcSize, 5); + } + + private static nuint ZSTD_compressBlock_doubleFast_noDict_6( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_doubleFast_noDict_generic(ms, seqStore, rep, src, srcSize, 6); + } + + private static nuint ZSTD_compressBlock_doubleFast_noDict_7( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_doubleFast_noDict_generic(ms, seqStore, rep, src, srcSize, 7); + } + + private static nuint ZSTD_compressBlock_doubleFast_dictMatchState_4( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_doubleFast_dictMatchState_generic( + ms, + seqStore, + rep, + src, + srcSize, + 4 + ); + } + + private static nuint ZSTD_compressBlock_doubleFast_dictMatchState_5( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_doubleFast_dictMatchState_generic( + ms, + seqStore, + rep, + src, + srcSize, + 5 + ); + } + + private static nuint ZSTD_compressBlock_doubleFast_dictMatchState_6( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_doubleFast_dictMatchState_generic( + ms, + seqStore, + rep, + src, + srcSize, + 6 + ); + } + + private static nuint ZSTD_compressBlock_doubleFast_dictMatchState_7( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_doubleFast_dictMatchState_generic( + ms, + seqStore, + rep, + src, + srcSize, + 7 + ); + } + + private static nuint ZSTD_compressBlock_doubleFast( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + uint mls = ms->cParams.minMatch; + switch (mls) + { + default: + case 4: + return ZSTD_compressBlock_doubleFast_noDict_4(ms, seqStore, rep, src, srcSize); + case 5: + return ZSTD_compressBlock_doubleFast_noDict_5(ms, seqStore, rep, src, srcSize); + case 6: + return ZSTD_compressBlock_doubleFast_noDict_6(ms, seqStore, rep, src, srcSize); + case 7: + return ZSTD_compressBlock_doubleFast_noDict_7(ms, seqStore, rep, src, srcSize); + } + } + + private static nuint ZSTD_compressBlock_doubleFast_dictMatchState( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + uint mls = ms->cParams.minMatch; + switch (mls) + { + default: + case 4: + return ZSTD_compressBlock_doubleFast_dictMatchState_4( + ms, + seqStore, + rep, + src, + srcSize + ); + case 5: + return ZSTD_compressBlock_doubleFast_dictMatchState_5( + ms, + seqStore, + rep, + src, + srcSize + ); + case 6: + return ZSTD_compressBlock_doubleFast_dictMatchState_6( + ms, + seqStore, + rep, + src, + srcSize + ); + case 7: + return ZSTD_compressBlock_doubleFast_dictMatchState_7( + ms, + seqStore, + rep, + src, + srcSize + ); + } + } + + private static nuint ZSTD_compressBlock_doubleFast_extDict_generic( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize, + uint mls + ) + { + ZSTD_compressionParameters* cParams = &ms->cParams; + uint* hashLong = ms->hashTable; + uint hBitsL = cParams->hashLog; + uint* hashSmall = ms->chainTable; + uint hBitsS = cParams->chainLog; + byte* istart = (byte*)src; + byte* ip = istart; + byte* anchor = istart; + byte* iend = istart + srcSize; + byte* ilimit = iend - 8; + byte* @base = ms->window.@base; + uint endIndex = (uint)((nuint)(istart - @base) + srcSize); + uint lowLimit = ZSTD_getLowestMatchIndex(ms, endIndex, cParams->windowLog); + uint dictStartIndex = lowLimit; + uint dictLimit = ms->window.dictLimit; + uint prefixStartIndex = dictLimit > lowLimit ? dictLimit : lowLimit; + byte* prefixStart = @base + prefixStartIndex; + byte* dictBase = ms->window.dictBase; + byte* dictStart = dictBase + dictStartIndex; + byte* dictEnd = dictBase + prefixStartIndex; + uint offset_1 = rep[0], + offset_2 = rep[1]; + if (prefixStartIndex == dictStartIndex) + { + return ZSTD_compressBlock_doubleFast(ms, seqStore, rep, src, srcSize); + } + + while (ip < ilimit) + { + nuint hSmall = ZSTD_hashPtr(ip, hBitsS, mls); + uint matchIndex = hashSmall[hSmall]; + byte* matchBase = matchIndex < prefixStartIndex ? dictBase : @base; + byte* match = matchBase + matchIndex; + nuint hLong = ZSTD_hashPtr(ip, hBitsL, 8); + uint matchLongIndex = hashLong[hLong]; + byte* matchLongBase = matchLongIndex < prefixStartIndex ? dictBase : @base; + byte* matchLong = matchLongBase + matchLongIndex; + uint curr = (uint)(ip - @base); + /* offset_1 expected <= curr +1 */ + uint repIndex = curr + 1 - offset_1; + byte* repBase = repIndex < prefixStartIndex ? dictBase : @base; + byte* repMatch = repBase + repIndex; + nuint mLength; + hashSmall[hSmall] = hashLong[hLong] = curr; + if ( + ( + ZSTD_index_overlap_check(prefixStartIndex, repIndex) + & (offset_1 <= curr + 1 - dictStartIndex ? 1 : 0) + ) != 0 + && MEM_read32(repMatch) == MEM_read32(ip + 1) + ) + { + byte* repMatchEnd = repIndex < prefixStartIndex ? dictEnd : iend; + mLength = + ZSTD_count_2segments(ip + 1 + 4, repMatch + 4, iend, repMatchEnd, prefixStart) + + 4; + ip++; + assert(1 >= 1); + assert(1 <= 3); + ZSTD_storeSeq(seqStore, (nuint)(ip - anchor), anchor, iend, 1, mLength); + } + else + { + if (matchLongIndex > dictStartIndex && MEM_read64(matchLong) == MEM_read64(ip)) + { + byte* matchEnd = matchLongIndex < prefixStartIndex ? dictEnd : iend; + byte* lowMatchPtr = matchLongIndex < prefixStartIndex ? dictStart : prefixStart; + uint offset; + mLength = + ZSTD_count_2segments(ip + 8, matchLong + 8, iend, matchEnd, prefixStart) + + 8; + offset = curr - matchLongIndex; + while (ip > anchor && matchLong > lowMatchPtr && ip[-1] == matchLong[-1]) + { + ip--; + matchLong--; + mLength++; + } + + offset_2 = offset_1; + offset_1 = offset; + assert(offset > 0); + ZSTD_storeSeq( + seqStore, + (nuint)(ip - anchor), + anchor, + iend, + offset + 3, + mLength + ); + } + else if (matchIndex > dictStartIndex && MEM_read32(match) == MEM_read32(ip)) + { + nuint h3 = ZSTD_hashPtr(ip + 1, hBitsL, 8); + uint matchIndex3 = hashLong[h3]; + byte* match3Base = matchIndex3 < prefixStartIndex ? dictBase : @base; + byte* match3 = match3Base + matchIndex3; + uint offset; + hashLong[h3] = curr + 1; + if (matchIndex3 > dictStartIndex && MEM_read64(match3) == MEM_read64(ip + 1)) + { + byte* matchEnd = matchIndex3 < prefixStartIndex ? dictEnd : iend; + byte* lowMatchPtr = + matchIndex3 < prefixStartIndex ? dictStart : prefixStart; + mLength = + ZSTD_count_2segments(ip + 9, match3 + 8, iend, matchEnd, prefixStart) + + 8; + ip++; + offset = curr + 1 - matchIndex3; + while (ip > anchor && match3 > lowMatchPtr && ip[-1] == match3[-1]) + { + ip--; + match3--; + mLength++; + } + } + else + { + byte* matchEnd = matchIndex < prefixStartIndex ? dictEnd : iend; + byte* lowMatchPtr = matchIndex < prefixStartIndex ? dictStart : prefixStart; + mLength = + ZSTD_count_2segments(ip + 4, match + 4, iend, matchEnd, prefixStart) + + 4; + offset = curr - matchIndex; + while (ip > anchor && match > lowMatchPtr && ip[-1] == match[-1]) + { + ip--; + match--; + mLength++; + } + } + + offset_2 = offset_1; + offset_1 = offset; + assert(offset > 0); + ZSTD_storeSeq( + seqStore, + (nuint)(ip - anchor), + anchor, + iend, + offset + 3, + mLength + ); + } + else + { + ip += (ip - anchor >> 8) + 1; + continue; + } + } + + ip += mLength; + anchor = ip; + if (ip <= ilimit) + { + { + uint indexToInsert = curr + 2; + hashLong[ZSTD_hashPtr(@base + indexToInsert, hBitsL, 8)] = indexToInsert; + hashLong[ZSTD_hashPtr(ip - 2, hBitsL, 8)] = (uint)(ip - 2 - @base); + hashSmall[ZSTD_hashPtr(@base + indexToInsert, hBitsS, mls)] = indexToInsert; + hashSmall[ZSTD_hashPtr(ip - 1, hBitsS, mls)] = (uint)(ip - 1 - @base); + } + + while (ip <= ilimit) + { + uint current2 = (uint)(ip - @base); + uint repIndex2 = current2 - offset_2; + byte* repMatch2 = + repIndex2 < prefixStartIndex ? dictBase + repIndex2 : @base + repIndex2; + if ( + ( + ZSTD_index_overlap_check(prefixStartIndex, repIndex2) + & (offset_2 <= current2 - dictStartIndex ? 1 : 0) + ) != 0 + && MEM_read32(repMatch2) == MEM_read32(ip) + ) + { + byte* repEnd2 = repIndex2 < prefixStartIndex ? dictEnd : iend; + nuint repLength2 = + ZSTD_count_2segments(ip + 4, repMatch2 + 4, iend, repEnd2, prefixStart) + + 4; + /* swap offset_2 <=> offset_1 */ + uint tmpOffset = offset_2; + offset_2 = offset_1; + offset_1 = tmpOffset; + assert(1 >= 1); + assert(1 <= 3); + ZSTD_storeSeq(seqStore, 0, anchor, iend, 1, repLength2); + hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = current2; + hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = current2; + ip += repLength2; + anchor = ip; + continue; + } + + break; + } + } + } + + rep[0] = offset_1; + rep[1] = offset_2; + return (nuint)(iend - anchor); + } + + private static nuint ZSTD_compressBlock_doubleFast_extDict_4( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_doubleFast_extDict_generic(ms, seqStore, rep, src, srcSize, 4); + } + + private static nuint ZSTD_compressBlock_doubleFast_extDict_5( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_doubleFast_extDict_generic(ms, seqStore, rep, src, srcSize, 5); + } + + private static nuint ZSTD_compressBlock_doubleFast_extDict_6( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_doubleFast_extDict_generic(ms, seqStore, rep, src, srcSize, 6); + } + + private static nuint ZSTD_compressBlock_doubleFast_extDict_7( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_doubleFast_extDict_generic(ms, seqStore, rep, src, srcSize, 7); + } + + private static nuint ZSTD_compressBlock_doubleFast_extDict( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + uint mls = ms->cParams.minMatch; + switch (mls) + { + default: + case 4: + return ZSTD_compressBlock_doubleFast_extDict_4(ms, seqStore, rep, src, srcSize); + case 5: + return ZSTD_compressBlock_doubleFast_extDict_5(ms, seqStore, rep, src, srcSize); + case 6: + return ZSTD_compressBlock_doubleFast_extDict_6(ms, seqStore, rep, src, srcSize); + case 7: + return ZSTD_compressBlock_doubleFast_extDict_7(ms, seqStore, rep, src, srcSize); + } + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdFast.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdFast.cs new file mode 100644 index 00000000..c2dd99c6 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdFast.cs @@ -0,0 +1,1224 @@ +using System.Runtime.CompilerServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + private static void ZSTD_fillHashTableForCDict( + ZSTD_MatchState_t* ms, + void* end, + ZSTD_dictTableLoadMethod_e dtlm + ) + { + ZSTD_compressionParameters* cParams = &ms->cParams; + uint* hashTable = ms->hashTable; + uint hBits = cParams->hashLog + 8; + uint mls = cParams->minMatch; + byte* @base = ms->window.@base; + byte* ip = @base + ms->nextToUpdate; + byte* iend = (byte*)end - 8; + const uint fastHashFillStep = 3; + assert(dtlm == ZSTD_dictTableLoadMethod_e.ZSTD_dtlm_full); + for (; ip + fastHashFillStep < iend + 2; ip += fastHashFillStep) + { + uint curr = (uint)(ip - @base); + { + nuint hashAndTag = ZSTD_hashPtr(ip, hBits, mls); + ZSTD_writeTaggedIndex(hashTable, hashAndTag, curr); + } + + if (dtlm == ZSTD_dictTableLoadMethod_e.ZSTD_dtlm_fast) + { + continue; + } + + { + uint p; + for (p = 1; p < fastHashFillStep; ++p) + { + nuint hashAndTag = ZSTD_hashPtr(ip + p, hBits, mls); + if (hashTable[hashAndTag >> 8] == 0) + { + ZSTD_writeTaggedIndex(hashTable, hashAndTag, curr + p); + } + } + } + } + } + + private static void ZSTD_fillHashTableForCCtx( + ZSTD_MatchState_t* ms, + void* end, + ZSTD_dictTableLoadMethod_e dtlm + ) + { + ZSTD_compressionParameters* cParams = &ms->cParams; + uint* hashTable = ms->hashTable; + uint hBits = cParams->hashLog; + uint mls = cParams->minMatch; + byte* @base = ms->window.@base; + byte* ip = @base + ms->nextToUpdate; + byte* iend = (byte*)end - 8; + const uint fastHashFillStep = 3; + assert(dtlm == ZSTD_dictTableLoadMethod_e.ZSTD_dtlm_fast); + for (; ip + fastHashFillStep < iend + 2; ip += fastHashFillStep) + { + uint curr = (uint)(ip - @base); + nuint hash0 = ZSTD_hashPtr(ip, hBits, mls); + hashTable[hash0] = curr; + if (dtlm == ZSTD_dictTableLoadMethod_e.ZSTD_dtlm_fast) + { + continue; + } + + { + uint p; + for (p = 1; p < fastHashFillStep; ++p) + { + nuint hash = ZSTD_hashPtr(ip + p, hBits, mls); + if (hashTable[hash] == 0) + { + hashTable[hash] = curr + p; + } + } + } + } + } + + private static void ZSTD_fillHashTable( + ZSTD_MatchState_t* ms, + void* end, + ZSTD_dictTableLoadMethod_e dtlm, + ZSTD_tableFillPurpose_e tfp + ) + { + if (tfp == ZSTD_tableFillPurpose_e.ZSTD_tfp_forCDict) + { + ZSTD_fillHashTableForCDict(ms, end, dtlm); + } + else + { + ZSTD_fillHashTableForCCtx(ms, end, dtlm); + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static int ZSTD_match4Found_cmov( + byte* currentPtr, + byte* matchAddress, + uint matchIdx, + uint idxLowLimit + ) + { + /* currentIdx >= lowLimit is a (somewhat) unpredictable branch. + * However expression below compiles into conditional move. + */ + byte* mvalAddr = ZSTD_selectAddr(matchIdx, idxLowLimit, matchAddress, dummy); + if (MEM_read32(currentPtr) != MEM_read32(mvalAddr)) + { + return 0; + } + + return matchIdx >= idxLowLimit ? 1 : 0; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static int ZSTD_match4Found_branch( + byte* currentPtr, + byte* matchAddress, + uint matchIdx, + uint idxLowLimit + ) + { + /* using a branch instead of a cmov, + * because it's faster in scenarios where matchIdx >= idxLowLimit is generally true, + * aka almost all candidates are within range */ + uint mval; + if (matchIdx >= idxLowLimit) + { + mval = MEM_read32(matchAddress); + } + else + { + mval = MEM_read32(currentPtr) ^ 1; + } + + return MEM_read32(currentPtr) == mval ? 1 : 0; + } + + /** + * If you squint hard enough (and ignore repcodes), the search operation at any + * given position is broken into 4 stages: + * + * 1. Hash (map position to hash value via input read) + * 2. Lookup (map hash val to index via hashtable read) + * 3. Load (map index to value at that position via input read) + * 4. Compare + * + * Each of these steps involves a memory read at an address which is computed + * from the previous step. This means these steps must be sequenced and their + * latencies are cumulative. + * + * Rather than do 1->2->3->4 sequentially for a single position before moving + * onto the next, this implementation interleaves these operations across the + * next few positions: + * + * R = Repcode Read & Compare + * H = Hash + * T = Table Lookup + * M = Match Read & Compare + * + * Pos | Time --> + * ----+------------------- + * N | ... M + * N+1 | ... TM + * N+2 | R H T M + * N+3 | H TM + * N+4 | R H T M + * N+5 | H ... + * N+6 | R ... + * + * This is very much analogous to the pipelining of execution in a CPU. And just + * like a CPU, we have to dump the pipeline when we find a match (i.e., take a + * branch). + * + * When this happens, we throw away our current state, and do the following prep + * to re-enter the loop: + * + * Pos | Time --> + * ----+------------------- + * N | H T + * N+1 | H + * + * This is also the work we do at the beginning to enter the loop initially. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_compressBlock_fast_noDict_generic( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize, + uint mls, + int useCmov + ) + { + ZSTD_compressionParameters* cParams = &ms->cParams; + uint* hashTable = ms->hashTable; + uint hlog = cParams->hashLog; + /* min 2 */ + nuint stepSize = cParams->targetLength + (uint)(cParams->targetLength == 0 ? 1 : 0) + 1; + byte* @base = ms->window.@base; + byte* istart = (byte*)src; + uint endIndex = (uint)((nuint)(istart - @base) + srcSize); + uint prefixStartIndex = ZSTD_getLowestPrefixIndex(ms, endIndex, cParams->windowLog); + byte* prefixStart = @base + prefixStartIndex; + byte* iend = istart + srcSize; + byte* ilimit = iend - 8; + byte* anchor = istart; + byte* ip0 = istart; + byte* ip1; + byte* ip2; + byte* ip3; + uint current0; + uint rep_offset1 = rep[0]; + uint rep_offset2 = rep[1]; + uint offsetSaved1 = 0, + offsetSaved2 = 0; + /* hash for ip0 */ + nuint hash0; + /* hash for ip1 */ + nuint hash1; + /* match idx for ip0 */ + uint matchIdx; + uint offcode; + byte* match0; + nuint mLength; + /* ip0 and ip1 are always adjacent. The targetLength skipping and + * uncompressibility acceleration is applied to every other position, + * matching the behavior of #1562. step therefore represents the gap + * between pairs of positions, from ip0 to ip2 or ip1 to ip3. */ + nuint step; + byte* nextStep; + const nuint kStepIncr = 1 << 8 - 1; + void* matchFound = + useCmov != 0 + ? (delegate* managed)(&ZSTD_match4Found_cmov) + : (delegate* managed)(&ZSTD_match4Found_branch); + ip0 += ip0 == prefixStart ? 1 : 0; + { + uint curr = (uint)(ip0 - @base); + uint windowLow = ZSTD_getLowestPrefixIndex(ms, curr, cParams->windowLog); + uint maxRep = curr - windowLow; + if (rep_offset2 > maxRep) + { + offsetSaved2 = rep_offset2; + rep_offset2 = 0; + } + + if (rep_offset1 > maxRep) + { + offsetSaved1 = rep_offset1; + rep_offset1 = 0; + } + } + + _start: + step = stepSize; + nextStep = ip0 + kStepIncr; + ip1 = ip0 + 1; + ip2 = ip0 + step; + ip3 = ip2 + 1; + if (ip3 >= ilimit) + { + goto _cleanup; + } + + hash0 = ZSTD_hashPtr(ip0, hlog, mls); + hash1 = ZSTD_hashPtr(ip1, hlog, mls); + matchIdx = hashTable[hash0]; + do + { + /* load repcode match for ip[2]*/ + uint rval = MEM_read32(ip2 - rep_offset1); + current0 = (uint)(ip0 - @base); + hashTable[hash0] = current0; + if (MEM_read32(ip2) == rval && rep_offset1 > 0) + { + ip0 = ip2; + match0 = ip0 - rep_offset1; + mLength = ip0[-1] == match0[-1] ? 1U : 0U; + ip0 -= mLength; + match0 -= mLength; + assert(1 >= 1); + assert(1 <= 3); + offcode = 1; + mLength += 4; + hashTable[hash1] = (uint)(ip1 - @base); + goto _match; + } + + if ( + ((delegate* managed)matchFound)( + ip0, + @base + matchIdx, + matchIdx, + prefixStartIndex + ) != 0 + ) + { + hashTable[hash1] = (uint)(ip1 - @base); + goto _offset; + } + + matchIdx = hashTable[hash1]; + hash0 = hash1; + hash1 = ZSTD_hashPtr(ip2, hlog, mls); + ip0 = ip1; + ip1 = ip2; + ip2 = ip3; + current0 = (uint)(ip0 - @base); + hashTable[hash0] = current0; + if ( + ((delegate* managed)matchFound)( + ip0, + @base + matchIdx, + matchIdx, + prefixStartIndex + ) != 0 + ) + { + if (step <= 4) + { + hashTable[hash1] = (uint)(ip1 - @base); + } + + goto _offset; + } + + matchIdx = hashTable[hash1]; + hash0 = hash1; + hash1 = ZSTD_hashPtr(ip2, hlog, mls); + ip0 = ip1; + ip1 = ip2; + ip2 = ip0 + step; + ip3 = ip1 + step; + if (ip2 >= nextStep) + { + step++; +#if NETCOREAPP3_0_OR_GREATER + if (System.Runtime.Intrinsics.X86.Sse.IsSupported) + { + System.Runtime.Intrinsics.X86.Sse.Prefetch0(ip1 + 64); + System.Runtime.Intrinsics.X86.Sse.Prefetch0(ip1 + 128); + } +#endif + + nextStep += kStepIncr; + } + } while (ip3 < ilimit); + _cleanup: + offsetSaved2 = offsetSaved1 != 0 && rep_offset1 != 0 ? offsetSaved1 : offsetSaved2; + rep[0] = rep_offset1 != 0 ? rep_offset1 : offsetSaved1; + rep[1] = rep_offset2 != 0 ? rep_offset2 : offsetSaved2; + return (nuint)(iend - anchor); + _offset: + match0 = @base + matchIdx; + rep_offset2 = rep_offset1; + rep_offset1 = (uint)(ip0 - match0); + assert(rep_offset1 > 0); + offcode = rep_offset1 + 3; + mLength = 4; + while (ip0 > anchor && match0 > prefixStart && ip0[-1] == match0[-1]) + { + ip0--; + match0--; + mLength++; + } + + _match: + mLength += ZSTD_count(ip0 + mLength, match0 + mLength, iend); + ZSTD_storeSeq(seqStore, (nuint)(ip0 - anchor), anchor, iend, offcode, mLength); + ip0 += mLength; + anchor = ip0; + if (ip0 <= ilimit) + { + assert(@base + current0 + 2 > istart); + hashTable[ZSTD_hashPtr(@base + current0 + 2, hlog, mls)] = current0 + 2; + hashTable[ZSTD_hashPtr(ip0 - 2, hlog, mls)] = (uint)(ip0 - 2 - @base); + if (rep_offset2 > 0) + { + while (ip0 <= ilimit && MEM_read32(ip0) == MEM_read32(ip0 - rep_offset2)) + { + /* store sequence */ + nuint rLength = ZSTD_count(ip0 + 4, ip0 + 4 - rep_offset2, iend) + 4; + { + /* swap rep_offset2 <=> rep_offset1 */ + uint tmpOff = rep_offset2; + rep_offset2 = rep_offset1; + rep_offset1 = tmpOff; + } + + hashTable[ZSTD_hashPtr(ip0, hlog, mls)] = (uint)(ip0 - @base); + ip0 += rLength; + assert(1 >= 1); + assert(1 <= 3); + ZSTD_storeSeq(seqStore, 0, anchor, iend, 1, rLength); + anchor = ip0; + continue; + } + } + } + + goto _start; + } + + private static nuint ZSTD_compressBlock_fast_noDict_4_1( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_fast_noDict_generic(ms, seqStore, rep, src, srcSize, 4, 1); + } + + private static nuint ZSTD_compressBlock_fast_noDict_5_1( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_fast_noDict_generic(ms, seqStore, rep, src, srcSize, 5, 1); + } + + private static nuint ZSTD_compressBlock_fast_noDict_6_1( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_fast_noDict_generic(ms, seqStore, rep, src, srcSize, 6, 1); + } + + private static nuint ZSTD_compressBlock_fast_noDict_7_1( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_fast_noDict_generic(ms, seqStore, rep, src, srcSize, 7, 1); + } + + private static nuint ZSTD_compressBlock_fast_noDict_4_0( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_fast_noDict_generic(ms, seqStore, rep, src, srcSize, 4, 0); + } + + private static nuint ZSTD_compressBlock_fast_noDict_5_0( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_fast_noDict_generic(ms, seqStore, rep, src, srcSize, 5, 0); + } + + private static nuint ZSTD_compressBlock_fast_noDict_6_0( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_fast_noDict_generic(ms, seqStore, rep, src, srcSize, 6, 0); + } + + private static nuint ZSTD_compressBlock_fast_noDict_7_0( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_fast_noDict_generic(ms, seqStore, rep, src, srcSize, 7, 0); + } + + private static nuint ZSTD_compressBlock_fast( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + uint mml = ms->cParams.minMatch; + /* use cmov when "candidate in range" branch is likely unpredictable */ + int useCmov = ms->cParams.windowLog < 19 ? 1 : 0; + assert(ms->dictMatchState == null); + if (useCmov != 0) + { + switch (mml) + { + default: + case 4: + return ZSTD_compressBlock_fast_noDict_4_1(ms, seqStore, rep, src, srcSize); + case 5: + return ZSTD_compressBlock_fast_noDict_5_1(ms, seqStore, rep, src, srcSize); + case 6: + return ZSTD_compressBlock_fast_noDict_6_1(ms, seqStore, rep, src, srcSize); + case 7: + return ZSTD_compressBlock_fast_noDict_7_1(ms, seqStore, rep, src, srcSize); + } + } + else + { + switch (mml) + { + default: + case 4: + return ZSTD_compressBlock_fast_noDict_4_0(ms, seqStore, rep, src, srcSize); + case 5: + return ZSTD_compressBlock_fast_noDict_5_0(ms, seqStore, rep, src, srcSize); + case 6: + return ZSTD_compressBlock_fast_noDict_6_0(ms, seqStore, rep, src, srcSize); + case 7: + return ZSTD_compressBlock_fast_noDict_7_0(ms, seqStore, rep, src, srcSize); + } + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_compressBlock_fast_dictMatchState_generic( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize, + uint mls, + uint hasStep + ) + { + ZSTD_compressionParameters* cParams = &ms->cParams; + uint* hashTable = ms->hashTable; + uint hlog = cParams->hashLog; + /* support stepSize of 0 */ + uint stepSize = cParams->targetLength + (uint)(cParams->targetLength == 0 ? 1 : 0); + byte* @base = ms->window.@base; + byte* istart = (byte*)src; + byte* ip0 = istart; + /* we assert below that stepSize >= 1 */ + byte* ip1 = ip0 + stepSize; + byte* anchor = istart; + uint prefixStartIndex = ms->window.dictLimit; + byte* prefixStart = @base + prefixStartIndex; + byte* iend = istart + srcSize; + byte* ilimit = iend - 8; + uint offset_1 = rep[0], + offset_2 = rep[1]; + ZSTD_MatchState_t* dms = ms->dictMatchState; + ZSTD_compressionParameters* dictCParams = &dms->cParams; + uint* dictHashTable = dms->hashTable; + uint dictStartIndex = dms->window.dictLimit; + byte* dictBase = dms->window.@base; + byte* dictStart = dictBase + dictStartIndex; + byte* dictEnd = dms->window.nextSrc; + uint dictIndexDelta = prefixStartIndex - (uint)(dictEnd - dictBase); + uint dictAndPrefixLength = (uint)(istart - prefixStart + dictEnd - dictStart); + uint dictHBits = dictCParams->hashLog + 8; + /* if a dictionary is still attached, it necessarily means that + * it is within window size. So we just check it. */ + uint maxDistance = 1U << (int)cParams->windowLog; + uint endIndex = (uint)((nuint)(istart - @base) + srcSize); + assert(endIndex - prefixStartIndex <= maxDistance); + assert(prefixStartIndex >= (uint)(dictEnd - dictBase)); + if (ms->prefetchCDictTables != 0) + { + nuint hashTableBytes = ((nuint)1 << (int)dictCParams->hashLog) * sizeof(uint); + { + sbyte* _ptr = (sbyte*)dictHashTable; + nuint _size = hashTableBytes; + nuint _pos; + for (_pos = 0; _pos < _size; _pos += 64) + { +#if NETCOREAPP3_0_OR_GREATER + if (System.Runtime.Intrinsics.X86.Sse.IsSupported) + { + System.Runtime.Intrinsics.X86.Sse.Prefetch1(_ptr + _pos); + } +#endif + } + } + } + + ip0 += dictAndPrefixLength == 0 ? 1 : 0; + assert(offset_1 <= dictAndPrefixLength); + assert(offset_2 <= dictAndPrefixLength); + assert(stepSize >= 1); + while (ip1 <= ilimit) + { + nuint mLength; + nuint hash0 = ZSTD_hashPtr(ip0, hlog, mls); + nuint dictHashAndTag0 = ZSTD_hashPtr(ip0, dictHBits, mls); + uint dictMatchIndexAndTag = dictHashTable[dictHashAndTag0 >> 8]; + int dictTagsMatch = ZSTD_comparePackedTags(dictMatchIndexAndTag, dictHashAndTag0); + uint matchIndex = hashTable[hash0]; + uint curr = (uint)(ip0 - @base); + nuint step = stepSize; + const nuint kStepIncr = 1 << 8; + byte* nextStep = ip0 + kStepIncr; + while (true) + { + byte* match = @base + matchIndex; + uint repIndex = curr + 1 - offset_1; + byte* repMatch = + repIndex < prefixStartIndex + ? dictBase + (repIndex - dictIndexDelta) + : @base + repIndex; + nuint hash1 = ZSTD_hashPtr(ip1, hlog, mls); + nuint dictHashAndTag1 = ZSTD_hashPtr(ip1, dictHBits, mls); + hashTable[hash0] = curr; + if ( + ZSTD_index_overlap_check(prefixStartIndex, repIndex) != 0 + && MEM_read32(repMatch) == MEM_read32(ip0 + 1) + ) + { + byte* repMatchEnd = repIndex < prefixStartIndex ? dictEnd : iend; + mLength = + ZSTD_count_2segments( + ip0 + 1 + 4, + repMatch + 4, + iend, + repMatchEnd, + prefixStart + ) + 4; + ip0++; + assert(1 >= 1); + assert(1 <= 3); + ZSTD_storeSeq(seqStore, (nuint)(ip0 - anchor), anchor, iend, 1, mLength); + break; + } + + if (dictTagsMatch != 0) + { + /* Found a possible dict match */ + uint dictMatchIndex = dictMatchIndexAndTag >> 8; + byte* dictMatch = dictBase + dictMatchIndex; + if (dictMatchIndex > dictStartIndex && MEM_read32(dictMatch) == MEM_read32(ip0)) + { + if (matchIndex <= prefixStartIndex) + { + uint offset = curr - dictMatchIndex - dictIndexDelta; + mLength = + ZSTD_count_2segments( + ip0 + 4, + dictMatch + 4, + iend, + dictEnd, + prefixStart + ) + 4; + while ( + ip0 > anchor && dictMatch > dictStart && ip0[-1] == dictMatch[-1] + ) + { + ip0--; + dictMatch--; + mLength++; + } + + offset_2 = offset_1; + offset_1 = offset; + assert(offset > 0); + ZSTD_storeSeq( + seqStore, + (nuint)(ip0 - anchor), + anchor, + iend, + offset + 3, + mLength + ); + break; + } + } + } + + if (ZSTD_match4Found_cmov(ip0, match, matchIndex, prefixStartIndex) != 0) + { + /* found a regular match of size >= 4 */ + uint offset = (uint)(ip0 - match); + mLength = ZSTD_count(ip0 + 4, match + 4, iend) + 4; + while (ip0 > anchor && match > prefixStart && ip0[-1] == match[-1]) + { + ip0--; + match--; + mLength++; + } + + offset_2 = offset_1; + offset_1 = offset; + assert(offset > 0); + ZSTD_storeSeq( + seqStore, + (nuint)(ip0 - anchor), + anchor, + iend, + offset + 3, + mLength + ); + break; + } + + dictMatchIndexAndTag = dictHashTable[dictHashAndTag1 >> 8]; + dictTagsMatch = ZSTD_comparePackedTags(dictMatchIndexAndTag, dictHashAndTag1); + matchIndex = hashTable[hash1]; + if (ip1 >= nextStep) + { + step++; + nextStep += kStepIncr; + } + + ip0 = ip1; + ip1 = ip1 + step; + if (ip1 > ilimit) + { + goto _cleanup; + } + + curr = (uint)(ip0 - @base); + hash0 = hash1; + } + + assert(mLength != 0); + ip0 += mLength; + anchor = ip0; + if (ip0 <= ilimit) + { + assert(@base + curr + 2 > istart); + hashTable[ZSTD_hashPtr(@base + curr + 2, hlog, mls)] = curr + 2; + hashTable[ZSTD_hashPtr(ip0 - 2, hlog, mls)] = (uint)(ip0 - 2 - @base); + while (ip0 <= ilimit) + { + uint current2 = (uint)(ip0 - @base); + uint repIndex2 = current2 - offset_2; + byte* repMatch2 = + repIndex2 < prefixStartIndex + ? dictBase - dictIndexDelta + repIndex2 + : @base + repIndex2; + if ( + ZSTD_index_overlap_check(prefixStartIndex, repIndex2) != 0 + && MEM_read32(repMatch2) == MEM_read32(ip0) + ) + { + byte* repEnd2 = repIndex2 < prefixStartIndex ? dictEnd : iend; + nuint repLength2 = + ZSTD_count_2segments(ip0 + 4, repMatch2 + 4, iend, repEnd2, prefixStart) + + 4; + /* swap offset_2 <=> offset_1 */ + uint tmpOffset = offset_2; + offset_2 = offset_1; + offset_1 = tmpOffset; + assert(1 >= 1); + assert(1 <= 3); + ZSTD_storeSeq(seqStore, 0, anchor, iend, 1, repLength2); + hashTable[ZSTD_hashPtr(ip0, hlog, mls)] = current2; + ip0 += repLength2; + anchor = ip0; + continue; + } + + break; + } + } + + assert(ip0 == anchor); + ip1 = ip0 + stepSize; + } + + _cleanup: + rep[0] = offset_1; + rep[1] = offset_2; + return (nuint)(iend - anchor); + } + + private static nuint ZSTD_compressBlock_fast_dictMatchState_4_0( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_fast_dictMatchState_generic( + ms, + seqStore, + rep, + src, + srcSize, + 4, + 0 + ); + } + + private static nuint ZSTD_compressBlock_fast_dictMatchState_5_0( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_fast_dictMatchState_generic( + ms, + seqStore, + rep, + src, + srcSize, + 5, + 0 + ); + } + + private static nuint ZSTD_compressBlock_fast_dictMatchState_6_0( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_fast_dictMatchState_generic( + ms, + seqStore, + rep, + src, + srcSize, + 6, + 0 + ); + } + + private static nuint ZSTD_compressBlock_fast_dictMatchState_7_0( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_fast_dictMatchState_generic( + ms, + seqStore, + rep, + src, + srcSize, + 7, + 0 + ); + } + + private static nuint ZSTD_compressBlock_fast_dictMatchState( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + uint mls = ms->cParams.minMatch; + assert(ms->dictMatchState != null); + switch (mls) + { + default: + case 4: + return ZSTD_compressBlock_fast_dictMatchState_4_0(ms, seqStore, rep, src, srcSize); + case 5: + return ZSTD_compressBlock_fast_dictMatchState_5_0(ms, seqStore, rep, src, srcSize); + case 6: + return ZSTD_compressBlock_fast_dictMatchState_6_0(ms, seqStore, rep, src, srcSize); + case 7: + return ZSTD_compressBlock_fast_dictMatchState_7_0(ms, seqStore, rep, src, srcSize); + } + } + + private static nuint ZSTD_compressBlock_fast_extDict_generic( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize, + uint mls, + uint hasStep + ) + { + ZSTD_compressionParameters* cParams = &ms->cParams; + uint* hashTable = ms->hashTable; + uint hlog = cParams->hashLog; + /* support stepSize of 0 */ + nuint stepSize = cParams->targetLength + (uint)(cParams->targetLength == 0 ? 1 : 0) + 1; + byte* @base = ms->window.@base; + byte* dictBase = ms->window.dictBase; + byte* istart = (byte*)src; + byte* anchor = istart; + uint endIndex = (uint)((nuint)(istart - @base) + srcSize); + uint lowLimit = ZSTD_getLowestMatchIndex(ms, endIndex, cParams->windowLog); + uint dictStartIndex = lowLimit; + byte* dictStart = dictBase + dictStartIndex; + uint dictLimit = ms->window.dictLimit; + uint prefixStartIndex = dictLimit < lowLimit ? lowLimit : dictLimit; + byte* prefixStart = @base + prefixStartIndex; + byte* dictEnd = dictBase + prefixStartIndex; + byte* iend = istart + srcSize; + byte* ilimit = iend - 8; + uint offset_1 = rep[0], + offset_2 = rep[1]; + uint offsetSaved1 = 0, + offsetSaved2 = 0; + byte* ip0 = istart; + byte* ip1; + byte* ip2; + byte* ip3; + uint current0; + /* hash for ip0 */ + nuint hash0; + /* hash for ip1 */ + nuint hash1; + /* match idx for ip0 */ + uint idx; + /* base pointer for idx */ + byte* idxBase; + uint offcode; + byte* match0; + nuint mLength; + /* initialize to avoid warning, assert != 0 later */ + byte* matchEnd = null; + nuint step; + byte* nextStep; + const nuint kStepIncr = 1 << 8 - 1; + if (prefixStartIndex == dictStartIndex) + { + return ZSTD_compressBlock_fast(ms, seqStore, rep, src, srcSize); + } + + { + uint curr = (uint)(ip0 - @base); + uint maxRep = curr - dictStartIndex; + if (offset_2 >= maxRep) + { + offsetSaved2 = offset_2; + offset_2 = 0; + } + + if (offset_1 >= maxRep) + { + offsetSaved1 = offset_1; + offset_1 = 0; + } + } + + _start: + step = stepSize; + nextStep = ip0 + kStepIncr; + ip1 = ip0 + 1; + ip2 = ip0 + step; + ip3 = ip2 + 1; + if (ip3 >= ilimit) + { + goto _cleanup; + } + + hash0 = ZSTD_hashPtr(ip0, hlog, mls); + hash1 = ZSTD_hashPtr(ip1, hlog, mls); + idx = hashTable[hash0]; + idxBase = idx < prefixStartIndex ? dictBase : @base; + do + { + { + uint current2 = (uint)(ip2 - @base); + uint repIndex = current2 - offset_1; + byte* repBase = repIndex < prefixStartIndex ? dictBase : @base; + uint rval; + if (prefixStartIndex - repIndex >= 4 && offset_1 > 0) + { + rval = MEM_read32(repBase + repIndex); + } + else + { + rval = MEM_read32(ip2) ^ 1; + } + + current0 = (uint)(ip0 - @base); + hashTable[hash0] = current0; + if (MEM_read32(ip2) == rval) + { + ip0 = ip2; + match0 = repBase + repIndex; + matchEnd = repIndex < prefixStartIndex ? dictEnd : iend; + assert(match0 != prefixStart && match0 != dictStart); + mLength = ip0[-1] == match0[-1] ? 1U : 0U; + ip0 -= mLength; + match0 -= mLength; + assert(1 >= 1); + assert(1 <= 3); + offcode = 1; + mLength += 4; + goto _match; + } + } + + { + uint mval = idx >= dictStartIndex ? MEM_read32(idxBase + idx) : MEM_read32(ip0) ^ 1; + if (MEM_read32(ip0) == mval) + { + goto _offset; + } + } + + idx = hashTable[hash1]; + idxBase = idx < prefixStartIndex ? dictBase : @base; + hash0 = hash1; + hash1 = ZSTD_hashPtr(ip2, hlog, mls); + ip0 = ip1; + ip1 = ip2; + ip2 = ip3; + current0 = (uint)(ip0 - @base); + hashTable[hash0] = current0; + { + uint mval = idx >= dictStartIndex ? MEM_read32(idxBase + idx) : MEM_read32(ip0) ^ 1; + if (MEM_read32(ip0) == mval) + { + goto _offset; + } + } + + idx = hashTable[hash1]; + idxBase = idx < prefixStartIndex ? dictBase : @base; + hash0 = hash1; + hash1 = ZSTD_hashPtr(ip2, hlog, mls); + ip0 = ip1; + ip1 = ip2; + ip2 = ip0 + step; + ip3 = ip1 + step; + if (ip2 >= nextStep) + { + step++; +#if NETCOREAPP3_0_OR_GREATER + if (System.Runtime.Intrinsics.X86.Sse.IsSupported) + { + System.Runtime.Intrinsics.X86.Sse.Prefetch0(ip1 + 64); + System.Runtime.Intrinsics.X86.Sse.Prefetch0(ip1 + 128); + } +#endif + + nextStep += kStepIncr; + } + } while (ip3 < ilimit); + _cleanup: + offsetSaved2 = offsetSaved1 != 0 && offset_1 != 0 ? offsetSaved1 : offsetSaved2; + rep[0] = offset_1 != 0 ? offset_1 : offsetSaved1; + rep[1] = offset_2 != 0 ? offset_2 : offsetSaved2; + return (nuint)(iend - anchor); + _offset: + { + uint offset = current0 - idx; + byte* lowMatchPtr = idx < prefixStartIndex ? dictStart : prefixStart; + matchEnd = idx < prefixStartIndex ? dictEnd : iend; + match0 = idxBase + idx; + offset_2 = offset_1; + offset_1 = offset; + assert(offset > 0); + offcode = offset + 3; + mLength = 4; + while (ip0 > anchor && match0 > lowMatchPtr && ip0[-1] == match0[-1]) + { + ip0--; + match0--; + mLength++; + } + } + + _match: + assert(matchEnd != null); + mLength += ZSTD_count_2segments( + ip0 + mLength, + match0 + mLength, + iend, + matchEnd, + prefixStart + ); + ZSTD_storeSeq(seqStore, (nuint)(ip0 - anchor), anchor, iend, offcode, mLength); + ip0 += mLength; + anchor = ip0; + if (ip1 < ip0) + { + hashTable[hash1] = (uint)(ip1 - @base); + } + + if (ip0 <= ilimit) + { + assert(@base + current0 + 2 > istart); + hashTable[ZSTD_hashPtr(@base + current0 + 2, hlog, mls)] = current0 + 2; + hashTable[ZSTD_hashPtr(ip0 - 2, hlog, mls)] = (uint)(ip0 - 2 - @base); + while (ip0 <= ilimit) + { + uint repIndex2 = (uint)(ip0 - @base) - offset_2; + byte* repMatch2 = + repIndex2 < prefixStartIndex ? dictBase + repIndex2 : @base + repIndex2; + if ( + (ZSTD_index_overlap_check(prefixStartIndex, repIndex2) & (offset_2 > 0 ? 1 : 0)) + != 0 + && MEM_read32(repMatch2) == MEM_read32(ip0) + ) + { + byte* repEnd2 = repIndex2 < prefixStartIndex ? dictEnd : iend; + nuint repLength2 = + ZSTD_count_2segments(ip0 + 4, repMatch2 + 4, iend, repEnd2, prefixStart) + + 4; + { + /* swap offset_2 <=> offset_1 */ + uint tmpOffset = offset_2; + offset_2 = offset_1; + offset_1 = tmpOffset; + } + + assert(1 >= 1); + assert(1 <= 3); + ZSTD_storeSeq(seqStore, 0, anchor, iend, 1, repLength2); + hashTable[ZSTD_hashPtr(ip0, hlog, mls)] = (uint)(ip0 - @base); + ip0 += repLength2; + anchor = ip0; + continue; + } + + break; + } + } + + goto _start; + } + + private static nuint ZSTD_compressBlock_fast_extDict_4_0( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_fast_extDict_generic(ms, seqStore, rep, src, srcSize, 4, 0); + } + + private static nuint ZSTD_compressBlock_fast_extDict_5_0( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_fast_extDict_generic(ms, seqStore, rep, src, srcSize, 5, 0); + } + + private static nuint ZSTD_compressBlock_fast_extDict_6_0( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_fast_extDict_generic(ms, seqStore, rep, src, srcSize, 6, 0); + } + + private static nuint ZSTD_compressBlock_fast_extDict_7_0( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_fast_extDict_generic(ms, seqStore, rep, src, srcSize, 7, 0); + } + + private static nuint ZSTD_compressBlock_fast_extDict( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + uint mls = ms->cParams.minMatch; + assert(ms->dictMatchState == null); + switch (mls) + { + default: + case 4: + return ZSTD_compressBlock_fast_extDict_4_0(ms, seqStore, rep, src, srcSize); + case 5: + return ZSTD_compressBlock_fast_extDict_5_0(ms, seqStore, rep, src, srcSize); + case 6: + return ZSTD_compressBlock_fast_extDict_6_0(ms, seqStore, rep, src, srcSize); + case 7: + return ZSTD_compressBlock_fast_extDict_7_0(ms, seqStore, rep, src, srcSize); + } + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdInternal.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdInternal.cs new file mode 100644 index 00000000..4180d6b1 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdInternal.cs @@ -0,0 +1,641 @@ +using System; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; +#if NETCOREAPP3_0_OR_GREATER +using System.Runtime.Intrinsics.X86; +#endif +#if NET6_0_OR_GREATER +using System.Runtime.Intrinsics.Arm; +#endif + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_repStartValue => new uint[3] { 1, 4, 8 }; + private static uint* repStartValue => + (uint*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_repStartValue) + ); +#else + + private static readonly uint* repStartValue = GetArrayPointer(new uint[3] { 1, 4, 8 }); +#endif + private static readonly nuint* ZSTD_fcs_fieldSize = GetArrayPointer( + new nuint[4] { 0, 2, 4, 8 } + ); + private static readonly nuint* ZSTD_did_fieldSize = GetArrayPointer( + new nuint[4] { 0, 1, 2, 4 } + ); + private const uint ZSTD_blockHeaderSize = 3; +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_LL_bits => + new byte[36] + { + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 1, + 1, + 1, + 1, + 2, + 2, + 3, + 3, + 4, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + }; + private static byte* LL_bits => + (byte*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_LL_bits) + ); +#else + + private static readonly byte* LL_bits = GetArrayPointer( + new byte[36] + { + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 1, + 1, + 1, + 1, + 2, + 2, + 3, + 3, + 4, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + } + ); +#endif +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_LL_defaultNorm => + new short[36] + { + 4, + 3, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 1, + 1, + 1, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 3, + 2, + 1, + 1, + 1, + 1, + 1, + -1, + -1, + -1, + -1, + }; + private static short* LL_defaultNorm => + (short*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_LL_defaultNorm) + ); +#else + + private static readonly short* LL_defaultNorm = GetArrayPointer( + new short[36] + { + 4, + 3, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 1, + 1, + 1, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 3, + 2, + 1, + 1, + 1, + 1, + 1, + (short)(-1), + (short)(-1), + (short)(-1), + (short)(-1), + } + ); +#endif + private const uint LL_defaultNormLog = 6; +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_ML_bits => + new byte[53] + { + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 1, + 1, + 1, + 1, + 2, + 2, + 3, + 3, + 4, + 4, + 5, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + }; + private static byte* ML_bits => + (byte*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_ML_bits) + ); +#else + + private static readonly byte* ML_bits = GetArrayPointer( + new byte[53] + { + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 1, + 1, + 1, + 1, + 2, + 2, + 3, + 3, + 4, + 4, + 5, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + } + ); +#endif +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_ML_defaultNorm => + new short[53] + { + 1, + 4, + 3, + 2, + 2, + 2, + 2, + 2, + 2, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + -1, + -1, + -1, + -1, + -1, + -1, + -1, + }; + private static short* ML_defaultNorm => + (short*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_ML_defaultNorm) + ); +#else + + private static readonly short* ML_defaultNorm = GetArrayPointer( + new short[53] + { + 1, + 4, + 3, + 2, + 2, + 2, + 2, + 2, + 2, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + (short)(-1), + (short)(-1), + (short)(-1), + (short)(-1), + (short)(-1), + (short)(-1), + (short)(-1), + } + ); +#endif + private const uint ML_defaultNormLog = 6; +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_OF_defaultNorm => + new short[29] + { + 1, + 1, + 1, + 1, + 1, + 1, + 2, + 2, + 2, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + -1, + -1, + -1, + -1, + -1, + }; + private static short* OF_defaultNorm => + (short*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_OF_defaultNorm) + ); +#else + + private static readonly short* OF_defaultNorm = GetArrayPointer( + new short[29] + { + 1, + 1, + 1, + 1, + 1, + 1, + 2, + 2, + 2, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + (short)(-1), + (short)(-1), + (short)(-1), + (short)(-1), + (short)(-1), + } + ); +#endif + private const uint OF_defaultNormLog = 5; + + /*-******************************************* + * Shared functions to include for inlining + *********************************************/ + private static void ZSTD_copy8(void* dst, void* src) + { + memcpy(dst, src, 8); + } + + /* Need to use memmove here since the literal buffer can now be located within + the dst buffer. In circumstances where the op "catches up" to where the + literal buffer is, there can be partial overlaps in this call on the final + copy if the literal is being shifted by less than 16 bytes. */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_copy16(void* dst, void* src) + { +#if NET6_0_OR_GREATER + if (AdvSimd.IsSupported) + { + AdvSimd.Store((byte*)dst, AdvSimd.LoadVector128((byte*)src)); + } + else +#endif +#if NETCOREAPP3_0_OR_GREATER + if (Sse2.IsSupported) + { + Sse2.Store((byte*)dst, Sse2.LoadVector128((byte*)src)); + } + else +#endif + { + var v1 = System.Runtime.CompilerServices.Unsafe.ReadUnaligned((ulong*)src); + var v2 = System.Runtime.CompilerServices.Unsafe.ReadUnaligned((ulong*)src + 1); + System.Runtime.CompilerServices.Unsafe.WriteUnaligned((ulong*)dst, v1); + System.Runtime.CompilerServices.Unsafe.WriteUnaligned((ulong*)dst + 1, v2); + } + } + + /*! ZSTD_wildcopy() : + * Custom version of ZSTD_memcpy(), can over read/write up to WILDCOPY_OVERLENGTH bytes (if length==0) + * @param ovtype controls the overlap detection + * - ZSTD_no_overlap: The source and destination are guaranteed to be at least WILDCOPY_VECLEN bytes apart. + * - ZSTD_overlap_src_before_dst: The src and dst may overlap, but they MUST be at least 8 bytes apart. + * The src buffer must be before the dst buffer. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_wildcopy(void* dst, void* src, nint length, ZSTD_overlap_e ovtype) + { + nint diff = (nint)((byte*)dst - (byte*)src); + byte* ip = (byte*)src; + byte* op = (byte*)dst; + byte* oend = op + length; + if (ovtype == ZSTD_overlap_e.ZSTD_overlap_src_before_dst && diff < 16) + { + do + { + ZSTD_copy8(op, ip); + op += 8; + ip += 8; + } while (op < oend); + } + else + { + assert(diff >= 16 || diff <= -16); + ZSTD_copy16(op, ip); + if (16 >= length) + { + return; + } + + op += 16; + ip += 16; + do + { + { + ZSTD_copy16(op, ip); + op += 16; + ip += 16; + } + + { + ZSTD_copy16(op, ip); + op += 16; + ip += 16; + } + } while (op < oend); + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_limitCopy(void* dst, nuint dstCapacity, void* src, nuint srcSize) + { + nuint length = dstCapacity < srcSize ? dstCapacity : srcSize; + if (length > 0) + { + memcpy(dst, src, (uint)length); + } + + return length; + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdLazy.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdLazy.cs new file mode 100644 index 00000000..fdf26282 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdLazy.cs @@ -0,0 +1,4841 @@ +using System; +using System.Numerics; +using System.Runtime.CompilerServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; +#if NETCOREAPP3_0_OR_GREATER +using System.Runtime.Intrinsics; +using System.Runtime.Intrinsics.X86; +#endif +#if NET6_0_OR_GREATER +using System.Runtime.Intrinsics.Arm; +#endif + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + /*-************************************* + * Binary Tree search + ***************************************/ + private static void ZSTD_updateDUBT(ZSTD_MatchState_t* ms, byte* ip, byte* iend, uint mls) + { + ZSTD_compressionParameters* cParams = &ms->cParams; + uint* hashTable = ms->hashTable; + uint hashLog = cParams->hashLog; + uint* bt = ms->chainTable; + uint btLog = cParams->chainLog - 1; + uint btMask = (uint)((1 << (int)btLog) - 1); + byte* @base = ms->window.@base; + uint target = (uint)(ip - @base); + uint idx = ms->nextToUpdate; + assert(ip + 8 <= iend); + assert(idx >= ms->window.dictLimit); + for (; idx < target; idx++) + { + /* assumption : ip + 8 <= iend */ + nuint h = ZSTD_hashPtr(@base + idx, hashLog, mls); + uint matchIndex = hashTable[h]; + uint* nextCandidatePtr = bt + 2 * (idx & btMask); + uint* sortMarkPtr = nextCandidatePtr + 1; + hashTable[h] = idx; + *nextCandidatePtr = matchIndex; + *sortMarkPtr = 1; + } + + ms->nextToUpdate = target; + } + + /** ZSTD_insertDUBT1() : + * sort one already inserted but unsorted position + * assumption : curr >= btlow == (curr - btmask) + * doesn't fail */ + private static void ZSTD_insertDUBT1( + ZSTD_MatchState_t* ms, + uint curr, + byte* inputEnd, + uint nbCompares, + uint btLow, + ZSTD_dictMode_e dictMode + ) + { + ZSTD_compressionParameters* cParams = &ms->cParams; + uint* bt = ms->chainTable; + uint btLog = cParams->chainLog - 1; + uint btMask = (uint)((1 << (int)btLog) - 1); + nuint commonLengthSmaller = 0, + commonLengthLarger = 0; + byte* @base = ms->window.@base; + byte* dictBase = ms->window.dictBase; + uint dictLimit = ms->window.dictLimit; + byte* ip = curr >= dictLimit ? @base + curr : dictBase + curr; + byte* iend = curr >= dictLimit ? inputEnd : dictBase + dictLimit; + byte* dictEnd = dictBase + dictLimit; + byte* prefixStart = @base + dictLimit; + byte* match; + uint* smallerPtr = bt + 2 * (curr & btMask); + uint* largerPtr = smallerPtr + 1; + /* this candidate is unsorted : next sorted candidate is reached through *smallerPtr, while *largerPtr contains previous unsorted candidate (which is already saved and can be overwritten) */ + uint matchIndex = *smallerPtr; + /* to be nullified at the end */ + uint dummy32; + uint windowValid = ms->window.lowLimit; + uint maxDistance = 1U << (int)cParams->windowLog; + uint windowLow = curr - windowValid > maxDistance ? curr - maxDistance : windowValid; + assert(curr >= btLow); + assert(ip < iend); + for (; nbCompares != 0 && matchIndex > windowLow; --nbCompares) + { + uint* nextPtr = bt + 2 * (matchIndex & btMask); + /* guaranteed minimum nb of common bytes */ + nuint matchLength = + commonLengthSmaller < commonLengthLarger ? commonLengthSmaller : commonLengthLarger; + assert(matchIndex < curr); + if ( + dictMode != ZSTD_dictMode_e.ZSTD_extDict + || matchIndex + matchLength >= dictLimit + || curr < dictLimit + ) + { + byte* mBase = + dictMode != ZSTD_dictMode_e.ZSTD_extDict + || matchIndex + matchLength >= dictLimit + ? @base + : dictBase; + assert(matchIndex + matchLength >= dictLimit || curr < dictLimit); + match = mBase + matchIndex; + matchLength += ZSTD_count(ip + matchLength, match + matchLength, iend); + } + else + { + match = dictBase + matchIndex; + matchLength += ZSTD_count_2segments( + ip + matchLength, + match + matchLength, + iend, + dictEnd, + prefixStart + ); + if (matchIndex + matchLength >= dictLimit) + { + match = @base + matchIndex; + } + } + + if (ip + matchLength == iend) + { + break; + } + + if (match[matchLength] < ip[matchLength]) + { + *smallerPtr = matchIndex; + commonLengthSmaller = matchLength; + if (matchIndex <= btLow) + { + smallerPtr = &dummy32; + break; + } + + smallerPtr = nextPtr + 1; + matchIndex = nextPtr[1]; + } + else + { + *largerPtr = matchIndex; + commonLengthLarger = matchLength; + if (matchIndex <= btLow) + { + largerPtr = &dummy32; + break; + } + + largerPtr = nextPtr; + matchIndex = nextPtr[0]; + } + } + + *smallerPtr = *largerPtr = 0; + } + + private static nuint ZSTD_DUBT_findBetterDictMatch( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iend, + nuint* offsetPtr, + nuint bestLength, + uint nbCompares, + uint mls, + ZSTD_dictMode_e dictMode + ) + { + ZSTD_MatchState_t* dms = ms->dictMatchState; + ZSTD_compressionParameters* dmsCParams = &dms->cParams; + uint* dictHashTable = dms->hashTable; + uint hashLog = dmsCParams->hashLog; + nuint h = ZSTD_hashPtr(ip, hashLog, mls); + uint dictMatchIndex = dictHashTable[h]; + byte* @base = ms->window.@base; + byte* prefixStart = @base + ms->window.dictLimit; + uint curr = (uint)(ip - @base); + byte* dictBase = dms->window.@base; + byte* dictEnd = dms->window.nextSrc; + uint dictHighLimit = (uint)(dms->window.nextSrc - dms->window.@base); + uint dictLowLimit = dms->window.lowLimit; + uint dictIndexDelta = ms->window.lowLimit - dictHighLimit; + uint* dictBt = dms->chainTable; + uint btLog = dmsCParams->chainLog - 1; + uint btMask = (uint)((1 << (int)btLog) - 1); + uint btLow = btMask >= dictHighLimit - dictLowLimit ? dictLowLimit : dictHighLimit - btMask; + nuint commonLengthSmaller = 0, + commonLengthLarger = 0; + assert(dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState); + for (; nbCompares != 0 && dictMatchIndex > dictLowLimit; --nbCompares) + { + uint* nextPtr = dictBt + 2 * (dictMatchIndex & btMask); + /* guaranteed minimum nb of common bytes */ + nuint matchLength = + commonLengthSmaller < commonLengthLarger ? commonLengthSmaller : commonLengthLarger; + byte* match = dictBase + dictMatchIndex; + matchLength += ZSTD_count_2segments( + ip + matchLength, + match + matchLength, + iend, + dictEnd, + prefixStart + ); + if (dictMatchIndex + matchLength >= dictHighLimit) + { + match = @base + dictMatchIndex + dictIndexDelta; + } + + if (matchLength > bestLength) + { + uint matchIndex = dictMatchIndex + dictIndexDelta; + if ( + 4 * (int)(matchLength - bestLength) + > (int)( + ZSTD_highbit32(curr - matchIndex + 1) + - ZSTD_highbit32((uint)offsetPtr[0] + 1) + ) + ) + { + bestLength = matchLength; + assert(curr - matchIndex > 0); + *offsetPtr = curr - matchIndex + 3; + } + + if (ip + matchLength == iend) + { + break; + } + } + + if (match[matchLength] < ip[matchLength]) + { + if (dictMatchIndex <= btLow) + { + break; + } + + commonLengthSmaller = matchLength; + dictMatchIndex = nextPtr[1]; + } + else + { + if (dictMatchIndex <= btLow) + { + break; + } + + commonLengthLarger = matchLength; + dictMatchIndex = nextPtr[0]; + } + } + + if (bestLength >= 3) + { + assert(*offsetPtr > 3); + uint mIndex = curr - (uint)(*offsetPtr - 3); + } + + return bestLength; + } + + private static nuint ZSTD_DUBT_findBestMatch( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iend, + nuint* offBasePtr, + uint mls, + ZSTD_dictMode_e dictMode + ) + { + ZSTD_compressionParameters* cParams = &ms->cParams; + uint* hashTable = ms->hashTable; + uint hashLog = cParams->hashLog; + nuint h = ZSTD_hashPtr(ip, hashLog, mls); + uint matchIndex = hashTable[h]; + byte* @base = ms->window.@base; + uint curr = (uint)(ip - @base); + uint windowLow = ZSTD_getLowestMatchIndex(ms, curr, cParams->windowLog); + uint* bt = ms->chainTable; + uint btLog = cParams->chainLog - 1; + uint btMask = (uint)((1 << (int)btLog) - 1); + uint btLow = btMask >= curr ? 0 : curr - btMask; + uint unsortLimit = btLow > windowLow ? btLow : windowLow; + uint* nextCandidate = bt + 2 * (matchIndex & btMask); + uint* unsortedMark = bt + 2 * (matchIndex & btMask) + 1; + uint nbCompares = 1U << (int)cParams->searchLog; + uint nbCandidates = nbCompares; + uint previousCandidate = 0; + assert(ip <= iend - 8); + assert(dictMode != ZSTD_dictMode_e.ZSTD_dedicatedDictSearch); + while (matchIndex > unsortLimit && *unsortedMark == 1 && nbCandidates > 1) + { + *unsortedMark = previousCandidate; + previousCandidate = matchIndex; + matchIndex = *nextCandidate; + nextCandidate = bt + 2 * (matchIndex & btMask); + unsortedMark = bt + 2 * (matchIndex & btMask) + 1; + nbCandidates--; + } + + if (matchIndex > unsortLimit && *unsortedMark == 1) + { + *nextCandidate = *unsortedMark = 0; + } + + matchIndex = previousCandidate; + while (matchIndex != 0) + { + uint* nextCandidateIdxPtr = bt + 2 * (matchIndex & btMask) + 1; + uint nextCandidateIdx = *nextCandidateIdxPtr; + ZSTD_insertDUBT1(ms, matchIndex, iend, nbCandidates, unsortLimit, dictMode); + matchIndex = nextCandidateIdx; + nbCandidates++; + } + + { + nuint commonLengthSmaller = 0, + commonLengthLarger = 0; + byte* dictBase = ms->window.dictBase; + uint dictLimit = ms->window.dictLimit; + byte* dictEnd = dictBase + dictLimit; + byte* prefixStart = @base + dictLimit; + uint* smallerPtr = bt + 2 * (curr & btMask); + uint* largerPtr = bt + 2 * (curr & btMask) + 1; + uint matchEndIdx = curr + 8 + 1; + /* to be nullified at the end */ + uint dummy32; + nuint bestLength = 0; + matchIndex = hashTable[h]; + hashTable[h] = curr; + for (; nbCompares != 0 && matchIndex > windowLow; --nbCompares) + { + uint* nextPtr = bt + 2 * (matchIndex & btMask); + /* guaranteed minimum nb of common bytes */ + nuint matchLength = + commonLengthSmaller < commonLengthLarger + ? commonLengthSmaller + : commonLengthLarger; + byte* match; + if ( + dictMode != ZSTD_dictMode_e.ZSTD_extDict + || matchIndex + matchLength >= dictLimit + ) + { + match = @base + matchIndex; + matchLength += ZSTD_count(ip + matchLength, match + matchLength, iend); + } + else + { + match = dictBase + matchIndex; + matchLength += ZSTD_count_2segments( + ip + matchLength, + match + matchLength, + iend, + dictEnd, + prefixStart + ); + if (matchIndex + matchLength >= dictLimit) + { + match = @base + matchIndex; + } + } + + if (matchLength > bestLength) + { + if (matchLength > matchEndIdx - matchIndex) + { + matchEndIdx = matchIndex + (uint)matchLength; + } + + if ( + 4 * (int)(matchLength - bestLength) + > (int)( + ZSTD_highbit32(curr - matchIndex + 1) + - ZSTD_highbit32((uint)*offBasePtr) + ) + ) + { + bestLength = matchLength; + assert(curr - matchIndex > 0); + *offBasePtr = curr - matchIndex + 3; + } + + if (ip + matchLength == iend) + { + if (dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState) + { + nbCompares = 0; + } + + break; + } + } + + if (match[matchLength] < ip[matchLength]) + { + *smallerPtr = matchIndex; + commonLengthSmaller = matchLength; + if (matchIndex <= btLow) + { + smallerPtr = &dummy32; + break; + } + + smallerPtr = nextPtr + 1; + matchIndex = nextPtr[1]; + } + else + { + *largerPtr = matchIndex; + commonLengthLarger = matchLength; + if (matchIndex <= btLow) + { + largerPtr = &dummy32; + break; + } + + largerPtr = nextPtr; + matchIndex = nextPtr[0]; + } + } + + *smallerPtr = *largerPtr = 0; + assert(nbCompares <= 1U << (sizeof(nuint) == 4 ? 30 : 31) - 1); + if (dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState && nbCompares != 0) + { + bestLength = ZSTD_DUBT_findBetterDictMatch( + ms, + ip, + iend, + offBasePtr, + bestLength, + nbCompares, + mls, + dictMode + ); + } + + assert(matchEndIdx > curr + 8); + ms->nextToUpdate = matchEndIdx - 8; + if (bestLength >= 3) + { + assert(*offBasePtr > 3); + uint mIndex = curr - (uint)(*offBasePtr - 3); + } + + return bestLength; + } + } + + /** ZSTD_BtFindBestMatch() : Tree updater, providing best match */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_BtFindBestMatch( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offBasePtr, + uint mls, + ZSTD_dictMode_e dictMode + ) + { + if (ip < ms->window.@base + ms->nextToUpdate) + { + return 0; + } + + ZSTD_updateDUBT(ms, ip, iLimit, mls); + return ZSTD_DUBT_findBestMatch(ms, ip, iLimit, offBasePtr, mls, dictMode); + } + + /*********************************** + * Dedicated dict search + ***********************************/ + private static void ZSTD_dedicatedDictSearch_lazy_loadDictionary( + ZSTD_MatchState_t* ms, + byte* ip + ) + { + byte* @base = ms->window.@base; + uint target = (uint)(ip - @base); + uint* hashTable = ms->hashTable; + uint* chainTable = ms->chainTable; + uint chainSize = (uint)(1 << (int)ms->cParams.chainLog); + uint idx = ms->nextToUpdate; + uint minChain = chainSize < target - idx ? target - chainSize : idx; + const uint bucketSize = 1 << 2; + uint cacheSize = bucketSize - 1; + uint chainAttempts = (uint)(1 << (int)ms->cParams.searchLog) - cacheSize; + uint chainLimit = chainAttempts > 255 ? 255 : chainAttempts; + /* We know the hashtable is oversized by a factor of `bucketSize`. + * We are going to temporarily pretend `bucketSize == 1`, keeping only a + * single entry. We will use the rest of the space to construct a temporary + * chaintable. + */ + uint hashLog = ms->cParams.hashLog - 2; + uint* tmpHashTable = hashTable; + uint* tmpChainTable = hashTable + ((nuint)1 << (int)hashLog); + uint tmpChainSize = (uint)((1 << 2) - 1) << (int)hashLog; + uint tmpMinChain = tmpChainSize < target ? target - tmpChainSize : idx; + uint hashIdx; + assert(ms->cParams.chainLog <= 24); + assert(ms->cParams.hashLog > ms->cParams.chainLog); + assert(idx != 0); + assert(tmpMinChain <= minChain); + for (; idx < target; idx++) + { + uint h = (uint)ZSTD_hashPtr(@base + idx, hashLog, ms->cParams.minMatch); + if (idx >= tmpMinChain) + { + tmpChainTable[idx - tmpMinChain] = hashTable[h]; + } + + tmpHashTable[h] = idx; + } + + { + uint chainPos = 0; + for (hashIdx = 0; hashIdx < 1U << (int)hashLog; hashIdx++) + { + uint count; + uint countBeyondMinChain = 0; + uint i = tmpHashTable[hashIdx]; + for (count = 0; i >= tmpMinChain && count < cacheSize; count++) + { + if (i < minChain) + { + countBeyondMinChain++; + } + + i = tmpChainTable[i - tmpMinChain]; + } + + if (count == cacheSize) + { + for (count = 0; count < chainLimit; ) + { + if (i < minChain) + { + if (i == 0 || ++countBeyondMinChain > cacheSize) + { + break; + } + } + + chainTable[chainPos++] = i; + count++; + if (i < tmpMinChain) + { + break; + } + + i = tmpChainTable[i - tmpMinChain]; + } + } + else + { + count = 0; + } + + if (count != 0) + { + tmpHashTable[hashIdx] = (chainPos - count << 8) + count; + } + else + { + tmpHashTable[hashIdx] = 0; + } + } + + assert(chainPos <= chainSize); + } + + for (hashIdx = (uint)(1 << (int)hashLog); hashIdx != 0; ) + { + uint bucketIdx = --hashIdx << 2; + uint chainPackedPointer = tmpHashTable[hashIdx]; + uint i; + for (i = 0; i < cacheSize; i++) + { + hashTable[bucketIdx + i] = 0; + } + + hashTable[bucketIdx + bucketSize - 1] = chainPackedPointer; + } + + for (idx = ms->nextToUpdate; idx < target; idx++) + { + uint h = (uint)ZSTD_hashPtr(@base + idx, hashLog, ms->cParams.minMatch) << 2; + uint i; + for (i = cacheSize - 1; i != 0; i--) + { + hashTable[h + i] = hashTable[h + i - 1]; + } + + hashTable[h] = idx; + } + + ms->nextToUpdate = target; + } + + /* Returns the longest match length found in the dedicated dict search structure. + * If none are longer than the argument ml, then ml will be returned. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_dedicatedDictSearch_lazy_search( + nuint* offsetPtr, + nuint ml, + uint nbAttempts, + ZSTD_MatchState_t* dms, + byte* ip, + byte* iLimit, + byte* prefixStart, + uint curr, + uint dictLimit, + nuint ddsIdx + ) + { + uint ddsLowestIndex = dms->window.dictLimit; + byte* ddsBase = dms->window.@base; + byte* ddsEnd = dms->window.nextSrc; + uint ddsSize = (uint)(ddsEnd - ddsBase); + uint ddsIndexDelta = dictLimit - ddsSize; + const uint bucketSize = 1 << 2; + uint bucketLimit = nbAttempts < bucketSize - 1 ? nbAttempts : bucketSize - 1; + uint ddsAttempt; + uint matchIndex; + for (ddsAttempt = 0; ddsAttempt < bucketSize - 1; ddsAttempt++) + { +#if NETCOREAPP3_0_OR_GREATER + if (Sse.IsSupported) + { + Sse.Prefetch0(ddsBase + dms->hashTable[ddsIdx + ddsAttempt]); + } +#endif + } + + { + uint chainPackedPointer = dms->hashTable[ddsIdx + bucketSize - 1]; + uint chainIndex = chainPackedPointer >> 8; +#if NETCOREAPP3_0_OR_GREATER + if (Sse.IsSupported) + { + Sse.Prefetch0(&dms->chainTable[chainIndex]); + } +#endif + } + + for (ddsAttempt = 0; ddsAttempt < bucketLimit; ddsAttempt++) + { + nuint currentMl = 0; + byte* match; + matchIndex = dms->hashTable[ddsIdx + ddsAttempt]; + match = ddsBase + matchIndex; + if (matchIndex == 0) + { + return ml; + } + + assert(matchIndex >= ddsLowestIndex); + assert(match + 4 <= ddsEnd); + if (MEM_read32(match) == MEM_read32(ip)) + { + currentMl = + ZSTD_count_2segments(ip + 4, match + 4, iLimit, ddsEnd, prefixStart) + 4; + } + + if (currentMl > ml) + { + ml = currentMl; + assert(curr - (matchIndex + ddsIndexDelta) > 0); + *offsetPtr = curr - (matchIndex + ddsIndexDelta) + 3; + if (ip + currentMl == iLimit) + { + return ml; + } + } + } + + { + uint chainPackedPointer = dms->hashTable[ddsIdx + bucketSize - 1]; + uint chainIndex = chainPackedPointer >> 8; + uint chainLength = chainPackedPointer & 0xFF; + uint chainAttempts = nbAttempts - ddsAttempt; + uint chainLimit = chainAttempts > chainLength ? chainLength : chainAttempts; + uint chainAttempt; + for (chainAttempt = 0; chainAttempt < chainLimit; chainAttempt++) + { +#if NETCOREAPP3_0_OR_GREATER + if (Sse.IsSupported) + { + Sse.Prefetch0(ddsBase + dms->chainTable[chainIndex + chainAttempt]); + } +#endif + } + + for (chainAttempt = 0; chainAttempt < chainLimit; chainAttempt++, chainIndex++) + { + nuint currentMl = 0; + byte* match; + matchIndex = dms->chainTable[chainIndex]; + match = ddsBase + matchIndex; + assert(matchIndex >= ddsLowestIndex); + assert(match + 4 <= ddsEnd); + if (MEM_read32(match) == MEM_read32(ip)) + { + currentMl = + ZSTD_count_2segments(ip + 4, match + 4, iLimit, ddsEnd, prefixStart) + 4; + } + + if (currentMl > ml) + { + ml = currentMl; + assert(curr - (matchIndex + ddsIndexDelta) > 0); + *offsetPtr = curr - (matchIndex + ddsIndexDelta) + 3; + if (ip + currentMl == iLimit) + { + break; + } + } + } + } + + return ml; + } + + /* Update chains up to ip (excluded) + Assumption : always within prefix (i.e. not within extDict) */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_insertAndFindFirstIndex_internal( + ZSTD_MatchState_t* ms, + ZSTD_compressionParameters* cParams, + byte* ip, + uint mls, + uint lazySkipping + ) + { + uint* hashTable = ms->hashTable; + uint hashLog = cParams->hashLog; + uint* chainTable = ms->chainTable; + uint chainMask = (uint)((1 << (int)cParams->chainLog) - 1); + byte* @base = ms->window.@base; + uint target = (uint)(ip - @base); + uint idx = ms->nextToUpdate; + while (idx < target) + { + nuint h = ZSTD_hashPtr(@base + idx, hashLog, mls); + chainTable[idx & chainMask] = hashTable[h]; + hashTable[h] = idx; + idx++; + if (lazySkipping != 0) + { + break; + } + } + + ms->nextToUpdate = target; + return hashTable[ZSTD_hashPtr(ip, hashLog, mls)]; + } + + private static uint ZSTD_insertAndFindFirstIndex(ZSTD_MatchState_t* ms, byte* ip) + { + ZSTD_compressionParameters* cParams = &ms->cParams; + return ZSTD_insertAndFindFirstIndex_internal(ms, cParams, ip, ms->cParams.minMatch, 0); + } + + /* inlining is important to hardwire a hot branch (template emulation) */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_HcFindBestMatch( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr, + uint mls, + ZSTD_dictMode_e dictMode + ) + { + ZSTD_compressionParameters* cParams = &ms->cParams; + uint* chainTable = ms->chainTable; + uint chainSize = (uint)(1 << (int)cParams->chainLog); + uint chainMask = chainSize - 1; + byte* @base = ms->window.@base; + byte* dictBase = ms->window.dictBase; + uint dictLimit = ms->window.dictLimit; + byte* prefixStart = @base + dictLimit; + byte* dictEnd = dictBase + dictLimit; + uint curr = (uint)(ip - @base); + uint maxDistance = 1U << (int)cParams->windowLog; + uint lowestValid = ms->window.lowLimit; + uint withinMaxDistance = + curr - lowestValid > maxDistance ? curr - maxDistance : lowestValid; + uint isDictionary = ms->loadedDictEnd != 0 ? 1U : 0U; + uint lowLimit = isDictionary != 0 ? lowestValid : withinMaxDistance; + uint minChain = curr > chainSize ? curr - chainSize : 0; + uint nbAttempts = 1U << (int)cParams->searchLog; + nuint ml = 4 - 1; + ZSTD_MatchState_t* dms = ms->dictMatchState; + uint ddsHashLog = + dictMode == ZSTD_dictMode_e.ZSTD_dedicatedDictSearch ? dms->cParams.hashLog - 2 : 0; + nuint ddsIdx = + dictMode == ZSTD_dictMode_e.ZSTD_dedicatedDictSearch + ? ZSTD_hashPtr(ip, ddsHashLog, mls) << 2 + : 0; + uint matchIndex; + if (dictMode == ZSTD_dictMode_e.ZSTD_dedicatedDictSearch) + { + uint* entry = &dms->hashTable[ddsIdx]; +#if NETCOREAPP3_0_OR_GREATER + if (Sse.IsSupported) + { + Sse.Prefetch0(entry); + } +#endif + } + + matchIndex = ZSTD_insertAndFindFirstIndex_internal( + ms, + cParams, + ip, + mls, + (uint)ms->lazySkipping + ); + for (; matchIndex >= lowLimit && nbAttempts > 0; nbAttempts--) + { + nuint currentMl = 0; + if (dictMode != ZSTD_dictMode_e.ZSTD_extDict || matchIndex >= dictLimit) + { + byte* match = @base + matchIndex; + assert(matchIndex >= dictLimit); + if (MEM_read32(match + ml - 3) == MEM_read32(ip + ml - 3)) + { + currentMl = ZSTD_count(ip, match, iLimit); + } + } + else + { + byte* match = dictBase + matchIndex; + assert(match + 4 <= dictEnd); + if (MEM_read32(match) == MEM_read32(ip)) + { + currentMl = + ZSTD_count_2segments(ip + 4, match + 4, iLimit, dictEnd, prefixStart) + 4; + } + } + + if (currentMl > ml) + { + ml = currentMl; + assert(curr - matchIndex > 0); + *offsetPtr = curr - matchIndex + 3; + if (ip + currentMl == iLimit) + { + break; + } + } + + if (matchIndex <= minChain) + { + break; + } + + matchIndex = chainTable[matchIndex & chainMask]; + } + + assert(nbAttempts <= 1U << (sizeof(nuint) == 4 ? 30 : 31) - 1); + if (dictMode == ZSTD_dictMode_e.ZSTD_dedicatedDictSearch) + { + ml = ZSTD_dedicatedDictSearch_lazy_search( + offsetPtr, + ml, + nbAttempts, + dms, + ip, + iLimit, + prefixStart, + curr, + dictLimit, + ddsIdx + ); + } + else if (dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState) + { + uint* dmsChainTable = dms->chainTable; + uint dmsChainSize = (uint)(1 << (int)dms->cParams.chainLog); + uint dmsChainMask = dmsChainSize - 1; + uint dmsLowestIndex = dms->window.dictLimit; + byte* dmsBase = dms->window.@base; + byte* dmsEnd = dms->window.nextSrc; + uint dmsSize = (uint)(dmsEnd - dmsBase); + uint dmsIndexDelta = dictLimit - dmsSize; + uint dmsMinChain = dmsSize > dmsChainSize ? dmsSize - dmsChainSize : 0; + matchIndex = dms->hashTable[ZSTD_hashPtr(ip, dms->cParams.hashLog, mls)]; + for (; matchIndex >= dmsLowestIndex && nbAttempts > 0; nbAttempts--) + { + nuint currentMl = 0; + byte* match = dmsBase + matchIndex; + assert(match + 4 <= dmsEnd); + if (MEM_read32(match) == MEM_read32(ip)) + { + currentMl = + ZSTD_count_2segments(ip + 4, match + 4, iLimit, dmsEnd, prefixStart) + 4; + } + + if (currentMl > ml) + { + ml = currentMl; + assert(curr > matchIndex + dmsIndexDelta); + assert(curr - (matchIndex + dmsIndexDelta) > 0); + *offsetPtr = curr - (matchIndex + dmsIndexDelta) + 3; + if (ip + currentMl == iLimit) + { + break; + } + } + + if (matchIndex <= dmsMinChain) + { + break; + } + + matchIndex = dmsChainTable[matchIndex & dmsChainMask]; + } + } + + return ml; + } + + /* ZSTD_VecMask_next(): + * Starting from the LSB, returns the idx of the next non-zero bit. + * Basically counting the nb of trailing zeroes. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_VecMask_next(ulong val) + { + assert(val != 0); + return (uint)BitOperations.TrailingZeroCount(val); + } + + /* ZSTD_row_nextIndex(): + * Returns the next index to insert at within a tagTable row, and updates the "head" + * value to reflect the update. Essentially cycles backwards from [1, {entries per row}) + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_row_nextIndex(byte* tagRow, uint rowMask) + { + uint next = (uint)(*tagRow - 1) & rowMask; + next += next == 0 ? rowMask : 0; + *tagRow = (byte)next; + return next; + } + + /* ZSTD_isAligned(): + * Checks that a pointer is aligned to "align" bytes which must be a power of 2. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static int ZSTD_isAligned(void* ptr, nuint align) + { + assert((align & align - 1) == 0); + return ((nuint)ptr & align - 1) == 0 ? 1 : 0; + } + + /* ZSTD_row_prefetch(): + * Performs prefetching for the hashTable and tagTable at a given row. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_row_prefetch(uint* hashTable, byte* tagTable, uint relRow, uint rowLog) + { +#if NETCOREAPP3_0_OR_GREATER + if (Sse.IsSupported) + { + Sse.Prefetch0(hashTable + relRow); + } +#endif + + if (rowLog >= 5) + { +#if NETCOREAPP3_0_OR_GREATER + if (Sse.IsSupported) + { + Sse.Prefetch0(hashTable + relRow + 16); + } +#endif + } + +#if NETCOREAPP3_0_OR_GREATER + if (Sse.IsSupported) + { + Sse.Prefetch0(tagTable + relRow); + } +#endif + + if (rowLog == 6) + { +#if NETCOREAPP3_0_OR_GREATER + if (Sse.IsSupported) + { + Sse.Prefetch0(tagTable + relRow + 32); + } +#endif + } + + assert(rowLog == 4 || rowLog == 5 || rowLog == 6); + assert(ZSTD_isAligned(hashTable + relRow, 64) != 0); + assert(ZSTD_isAligned(tagTable + relRow, (nuint)1 << (int)rowLog) != 0); + } + + /* ZSTD_row_fillHashCache(): + * Fill up the hash cache starting at idx, prefetching up to ZSTD_ROW_HASH_CACHE_SIZE entries, + * but not beyond iLimit. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_row_fillHashCache( + ZSTD_MatchState_t* ms, + byte* @base, + uint rowLog, + uint mls, + uint idx, + byte* iLimit + ) + { + uint* hashTable = ms->hashTable; + byte* tagTable = ms->tagTable; + uint hashLog = ms->rowHashLog; + uint maxElemsToPrefetch = @base + idx > iLimit ? 0 : (uint)(iLimit - (@base + idx) + 1); + uint lim = idx + (8 < maxElemsToPrefetch ? 8 : maxElemsToPrefetch); + for (; idx < lim; ++idx) + { + uint hash = (uint)ZSTD_hashPtrSalted(@base + idx, hashLog + 8, mls, ms->hashSalt); + uint row = hash >> 8 << (int)rowLog; + ZSTD_row_prefetch(hashTable, tagTable, row, rowLog); + ms->hashCache[idx & 8 - 1] = hash; + } + } + + /* ZSTD_row_nextCachedHash(): + * Returns the hash of base + idx, and replaces the hash in the hash cache with the byte at + * base + idx + ZSTD_ROW_HASH_CACHE_SIZE. Also prefetches the appropriate rows from hashTable and tagTable. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_row_nextCachedHash( + uint* cache, + uint* hashTable, + byte* tagTable, + byte* @base, + uint idx, + uint hashLog, + uint rowLog, + uint mls, + ulong hashSalt + ) + { + uint newHash = (uint)ZSTD_hashPtrSalted(@base + idx + 8, hashLog + 8, mls, hashSalt); + uint row = newHash >> 8 << (int)rowLog; + ZSTD_row_prefetch(hashTable, tagTable, row, rowLog); + { + uint hash = cache[idx & 8 - 1]; + cache[idx & 8 - 1] = newHash; + return hash; + } + } + + /* ZSTD_row_update_internalImpl(): + * Updates the hash table with positions starting from updateStartIdx until updateEndIdx. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_row_update_internalImpl( + ZSTD_MatchState_t* ms, + uint updateStartIdx, + uint updateEndIdx, + uint mls, + uint rowLog, + uint rowMask, + uint useCache + ) + { + uint* hashTable = ms->hashTable; + byte* tagTable = ms->tagTable; + uint hashLog = ms->rowHashLog; + byte* @base = ms->window.@base; + for (; updateStartIdx < updateEndIdx; ++updateStartIdx) + { + uint hash = + useCache != 0 + ? ZSTD_row_nextCachedHash( + ms->hashCache, + hashTable, + tagTable, + @base, + updateStartIdx, + hashLog, + rowLog, + mls, + ms->hashSalt + ) + : (uint)ZSTD_hashPtrSalted( + @base + updateStartIdx, + hashLog + 8, + mls, + ms->hashSalt + ); + uint relRow = hash >> 8 << (int)rowLog; + uint* row = hashTable + relRow; + byte* tagRow = tagTable + relRow; + uint pos = ZSTD_row_nextIndex(tagRow, rowMask); + assert( + hash == ZSTD_hashPtrSalted(@base + updateStartIdx, hashLog + 8, mls, ms->hashSalt) + ); + tagRow[pos] = (byte)(hash & (1U << 8) - 1); + row[pos] = updateStartIdx; + } + } + + /* ZSTD_row_update_internal(): + * Inserts the byte at ip into the appropriate position in the hash table, and updates ms->nextToUpdate. + * Skips sections of long matches as is necessary. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_row_update_internal( + ZSTD_MatchState_t* ms, + byte* ip, + uint mls, + uint rowLog, + uint rowMask, + uint useCache + ) + { + uint idx = ms->nextToUpdate; + byte* @base = ms->window.@base; + uint target = (uint)(ip - @base); + const uint kSkipThreshold = 384; + const uint kMaxMatchStartPositionsToUpdate = 96; + const uint kMaxMatchEndPositionsToUpdate = 32; + if (useCache != 0) + { + if (target - idx > kSkipThreshold) + { + uint bound = idx + kMaxMatchStartPositionsToUpdate; + ZSTD_row_update_internalImpl(ms, idx, bound, mls, rowLog, rowMask, useCache); + idx = target - kMaxMatchEndPositionsToUpdate; + ZSTD_row_fillHashCache(ms, @base, rowLog, mls, idx, ip + 1); + } + } + + assert(target >= idx); + ZSTD_row_update_internalImpl(ms, idx, target, mls, rowLog, rowMask, useCache); + ms->nextToUpdate = target; + } + + /* ZSTD_row_update(): + * External wrapper for ZSTD_row_update_internal(). Used for filling the hashtable during dictionary + * processing. + */ + private static void ZSTD_row_update(ZSTD_MatchState_t* ms, byte* ip) + { + uint rowLog = + ms->cParams.searchLog <= 4 ? 4 + : ms->cParams.searchLog <= 6 ? ms->cParams.searchLog + : 6; + uint rowMask = (1U << (int)rowLog) - 1; + /* mls caps out at 6 */ + uint mls = ms->cParams.minMatch < 6 ? ms->cParams.minMatch : 6; + ZSTD_row_update_internal(ms, ip, mls, rowLog, rowMask, 0); + } + + /* Returns the mask width of bits group of which will be set to 1. Given not all + * architectures have easy movemask instruction, this helps to iterate over + * groups of bits easier and faster. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_row_matchMaskGroupWidth(uint rowEntries) + { + assert(rowEntries == 16 || rowEntries == 32 || rowEntries == 64); + assert(rowEntries <= 64); +#if NET6_0_OR_GREATER + if (AdvSimd.IsSupported && BitConverter.IsLittleEndian) + { + if (rowEntries == 16) + { + return 4; + } +#if NET9_0_OR_GREATER + if (AdvSimd.Arm64.IsSupported) + { + if (rowEntries == 32) + { + return 2; + } + + if (rowEntries == 64) + { + return 1; + } + } +#endif + } +#endif + return 1; + } + +#if NETCOREAPP3_0_OR_GREATER + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong ZSTD_row_getSSEMask(int nbChunks, byte* src, byte tag, uint head) + { + Vector128 comparisonMask = Vector128.Create(tag); + assert(nbChunks is 1 or 2 or 4); + if (nbChunks == 1) + { + Vector128 chunk0 = Sse2.LoadVector128(src); + Vector128 equalMask0 = Sse2.CompareEqual(chunk0, comparisonMask); + int matches0 = Sse2.MoveMask(equalMask0); + return BitOperations.RotateRight((ushort)matches0, (int)head); + } + + if (nbChunks == 2) + { + Vector128 chunk0 = Sse2.LoadVector128(src); + Vector128 equalMask0 = Sse2.CompareEqual(chunk0, comparisonMask); + int matches0 = Sse2.MoveMask(equalMask0); + Vector128 chunk1 = Sse2.LoadVector128(src + 16); + Vector128 equalMask1 = Sse2.CompareEqual(chunk1, comparisonMask); + int matches1 = Sse2.MoveMask(equalMask1); + return BitOperations.RotateRight((uint)matches1 << 16 | (uint)matches0, (int)head); + } + + { + Vector128 chunk0 = Sse2.LoadVector128(src); + Vector128 equalMask0 = Sse2.CompareEqual(chunk0, comparisonMask); + int matches0 = Sse2.MoveMask(equalMask0); + Vector128 chunk1 = Sse2.LoadVector128(src + 16 * 1); + Vector128 equalMask1 = Sse2.CompareEqual(chunk1, comparisonMask); + int matches1 = Sse2.MoveMask(equalMask1); + Vector128 chunk2 = Sse2.LoadVector128(src + 16 * 2); + Vector128 equalMask2 = Sse2.CompareEqual(chunk2, comparisonMask); + int matches2 = Sse2.MoveMask(equalMask2); + Vector128 chunk3 = Sse2.LoadVector128(src + 16 * 3); + Vector128 equalMask3 = Sse2.CompareEqual(chunk3, comparisonMask); + int matches3 = Sse2.MoveMask(equalMask3); + return BitOperations.RotateRight( + (ulong)matches3 << 48 + | (ulong)matches2 << 32 + | (ulong)matches1 << 16 + | (uint)matches0, + (int)head + ); + } + } +#endif + + /* Returns a ZSTD_VecMask (U64) that has the nth group (determined by + * ZSTD_row_matchMaskGroupWidth) of bits set to 1 if the newly-computed "tag" + * matches the hash at the nth position in a row of the tagTable. + * Each row is a circular buffer beginning at the value of "headGrouped". So we + * must rotate the "matches" bitfield to match up with the actual layout of the + * entries within the hashTable */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong ZSTD_row_getMatchMask( + byte* tagRow, + byte tag, + uint headGrouped, + uint rowEntries + ) + { + byte* src = tagRow; + assert(rowEntries == 16 || rowEntries == 32 || rowEntries == 64); + assert(rowEntries <= 64); + assert(ZSTD_row_matchMaskGroupWidth(rowEntries) * rowEntries <= sizeof(ulong) * 8); +#if NETCOREAPP3_0_OR_GREATER + if (Sse2.IsSupported) + { + return ZSTD_row_getSSEMask((int)(rowEntries / 16), src, tag, headGrouped); + } +#endif + +#if NET6_0_OR_GREATER + if (AdvSimd.IsSupported && BitConverter.IsLittleEndian) + { + if (rowEntries == 16) + { + /* vshrn_n_u16 shifts by 4 every u16 and narrows to 8 lower bits. + * After that groups of 4 bits represent the equalMask. We lower + * all bits except the highest in these groups by doing AND with + * 0x88 = 0b10001000. + */ + Vector128 chunk = AdvSimd.LoadVector128(src); + Vector128 equalMask = AdvSimd + .CompareEqual(chunk, AdvSimd.DuplicateToVector128(tag)) + .As(); + Vector64 res = AdvSimd.ShiftRightLogicalNarrowingLower(equalMask, 4); + ulong matches = res.As().GetElement(0); + return BitOperations.RotateRight(matches, (int)headGrouped) & 0x8888888888888888; + } + else if (rowEntries == 32) + { +#if NET9_0_OR_GREATER + if (AdvSimd.Arm64.IsSupported) + { + /* Same idea as with rowEntries == 16 but doing AND with + * 0x55 = 0b01010101. + */ + (Vector128 chunk0, Vector128 chunk1) = + AdvSimd.Arm64.Load2xVector128AndUnzip((ushort*)src); + Vector128 dup = AdvSimd.DuplicateToVector128(tag); + Vector64 t0 = AdvSimd.ShiftRightLogicalNarrowingLower( + AdvSimd.CompareEqual(chunk0.As(), dup).As(), + 6 + ); + Vector64 t1 = AdvSimd.ShiftRightLogicalNarrowingLower( + AdvSimd.CompareEqual(chunk1.As(), dup).As(), + 6 + ); + Vector64 res = AdvSimd.ShiftLeftAndInsert(t0, t1, 4); + ulong matches = res.As().GetElement(0); + return BitOperations.RotateRight(matches, (int)headGrouped) + & 0x5555555555555555; + } +#endif + } + else + { /* rowEntries == 64 */ +#if NET9_0_OR_GREATER + if (AdvSimd.Arm64.IsSupported) + { + ( + Vector128 chunk0, + Vector128 chunk1, + Vector128 chunk2, + Vector128 chunk3 + ) = AdvSimd.Arm64.Load4xVector128AndUnzip(src); + Vector128 dup = AdvSimd.DuplicateToVector128(tag); + Vector128 cmp0 = AdvSimd.CompareEqual(chunk0, dup); + Vector128 cmp1 = AdvSimd.CompareEqual(chunk1, dup); + Vector128 cmp2 = AdvSimd.CompareEqual(chunk2, dup); + Vector128 cmp3 = AdvSimd.CompareEqual(chunk3, dup); + + Vector128 t0 = AdvSimd.ShiftRightAndInsert(cmp1, cmp0, 1); + Vector128 t1 = AdvSimd.ShiftRightAndInsert(cmp3, cmp2, 1); + Vector128 t2 = AdvSimd.ShiftRightAndInsert(t1, t0, 2); + Vector128 t3 = AdvSimd.ShiftRightAndInsert(t2, t2, 4); + Vector64 t4 = AdvSimd.ShiftRightLogicalNarrowingLower( + t3.As(), + 4 + ); + ulong matches = t4.As().GetElement(0); + return BitOperations.RotateRight(matches, (int)headGrouped); + } +#endif + } + } +#endif + + { + nuint chunkSize = (nuint)sizeof(nuint); + nuint shiftAmount = chunkSize * 8 - chunkSize; + nuint xFF = ~(nuint)0; + nuint x01 = xFF / 0xFF; + nuint x80 = x01 << 7; + nuint splatChar = tag * x01; + ulong matches = 0; + int i = (int)(rowEntries - chunkSize); + assert(sizeof(nuint) == 4 || sizeof(nuint) == 8); + if (BitConverter.IsLittleEndian) + { + nuint extractMagic = xFF / 0x7F >> (int)chunkSize; + do + { + nuint chunk = MEM_readST(&src[i]); + chunk ^= splatChar; + chunk = ((chunk | x80) - x01 | chunk) & x80; + matches <<= (int)chunkSize; + matches |= chunk * extractMagic >> (int)shiftAmount; + i -= (int)chunkSize; + } while (i >= 0); + } + else + { + nuint msb = xFF ^ xFF >> 1; + nuint extractMagic = msb / 0x1FF | msb; + do + { + nuint chunk = MEM_readST(&src[i]); + chunk ^= splatChar; + chunk = ((chunk | x80) - x01 | chunk) & x80; + matches <<= (int)chunkSize; + matches |= (chunk >> 7) * extractMagic >> (int)shiftAmount; + i -= (int)chunkSize; + } while (i >= 0); + } + + matches = ~matches; + if (rowEntries == 16) + { + return BitOperations.RotateRight((ushort)matches, (int)headGrouped); + } + else if (rowEntries == 32) + { + return BitOperations.RotateRight((uint)matches, (int)headGrouped); + } + else + { + return BitOperations.RotateRight(matches, (int)headGrouped); + } + } + } + + /* The high-level approach of the SIMD row based match finder is as follows: + * - Figure out where to insert the new entry: + * - Generate a hash for current input position and split it into a one byte of tag and `rowHashLog` bits of index. + * - The hash is salted by a value that changes on every context reset, so when the same table is used + * we will avoid collisions that would otherwise slow us down by introducing phantom matches. + * - The hashTable is effectively split into groups or "rows" of 15 or 31 entries of U32, and the index determines + * which row to insert into. + * - Determine the correct position within the row to insert the entry into. Each row of 15 or 31 can + * be considered as a circular buffer with a "head" index that resides in the tagTable (overall 16 or 32 bytes + * per row). + * - Use SIMD to efficiently compare the tags in the tagTable to the 1-byte tag calculated for the position and + * generate a bitfield that we can cycle through to check the collisions in the hash table. + * - Pick the longest match. + * - Insert the tag into the equivalent row and position in the tagTable. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_RowFindBestMatch( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr, + uint mls, + ZSTD_dictMode_e dictMode, + uint rowLog + ) + { + uint* hashTable = ms->hashTable; + byte* tagTable = ms->tagTable; + uint* hashCache = ms->hashCache; + uint hashLog = ms->rowHashLog; + ZSTD_compressionParameters* cParams = &ms->cParams; + byte* @base = ms->window.@base; + byte* dictBase = ms->window.dictBase; + uint dictLimit = ms->window.dictLimit; + byte* prefixStart = @base + dictLimit; + byte* dictEnd = dictBase + dictLimit; + uint curr = (uint)(ip - @base); + uint maxDistance = 1U << (int)cParams->windowLog; + uint lowestValid = ms->window.lowLimit; + uint withinMaxDistance = + curr - lowestValid > maxDistance ? curr - maxDistance : lowestValid; + uint isDictionary = ms->loadedDictEnd != 0 ? 1U : 0U; + uint lowLimit = isDictionary != 0 ? lowestValid : withinMaxDistance; + uint rowEntries = 1U << (int)rowLog; + uint rowMask = rowEntries - 1; + /* nb of searches is capped at nb entries per row */ + uint cappedSearchLog = cParams->searchLog < rowLog ? cParams->searchLog : rowLog; + uint groupWidth = ZSTD_row_matchMaskGroupWidth(rowEntries); + ulong hashSalt = ms->hashSalt; + uint nbAttempts = 1U << (int)cappedSearchLog; + nuint ml = 4 - 1; + uint hash; + /* DMS/DDS variables that may be referenced laster */ + ZSTD_MatchState_t* dms = ms->dictMatchState; + /* Initialize the following variables to satisfy static analyzer */ + nuint ddsIdx = 0; + /* cctx hash tables are limited in searches, but allow extra searches into DDS */ + uint ddsExtraAttempts = 0; + uint dmsTag = 0; + uint* dmsRow = null; + byte* dmsTagRow = null; + if (dictMode == ZSTD_dictMode_e.ZSTD_dedicatedDictSearch) + { + uint ddsHashLog = dms->cParams.hashLog - 2; + { + ddsIdx = ZSTD_hashPtr(ip, ddsHashLog, mls) << 2; +#if NETCOREAPP3_0_OR_GREATER + if (Sse.IsSupported) + { + Sse.Prefetch0(&dms->hashTable[ddsIdx]); + } +#endif + } + + ddsExtraAttempts = + cParams->searchLog > rowLog ? 1U << (int)(cParams->searchLog - rowLog) : 0; + } + + if (dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState) + { + /* Prefetch DMS rows */ + uint* dmsHashTable = dms->hashTable; + byte* dmsTagTable = dms->tagTable; + uint dmsHash = (uint)ZSTD_hashPtr(ip, dms->rowHashLog + 8, mls); + uint dmsRelRow = dmsHash >> 8 << (int)rowLog; + dmsTag = dmsHash & (1U << 8) - 1; + dmsTagRow = dmsTagTable + dmsRelRow; + dmsRow = dmsHashTable + dmsRelRow; + ZSTD_row_prefetch(dmsHashTable, dmsTagTable, dmsRelRow, rowLog); + } + + if (ms->lazySkipping == 0) + { + ZSTD_row_update_internal(ms, ip, mls, rowLog, rowMask, 1); + hash = ZSTD_row_nextCachedHash( + hashCache, + hashTable, + tagTable, + @base, + curr, + hashLog, + rowLog, + mls, + hashSalt + ); + } + else + { + hash = (uint)ZSTD_hashPtrSalted(ip, hashLog + 8, mls, hashSalt); + ms->nextToUpdate = curr; + } + + ms->hashSaltEntropy += hash; + { + uint relRow = hash >> 8 << (int)rowLog; + uint tag = hash & (1U << 8) - 1; + uint* row = hashTable + relRow; + byte* tagRow = tagTable + relRow; + uint headGrouped = (*tagRow & rowMask) * groupWidth; + uint* matchBuffer = stackalloc uint[64]; + nuint numMatches = 0; + nuint currMatch = 0; + ulong matches = ZSTD_row_getMatchMask(tagRow, (byte)tag, headGrouped, rowEntries); + for (; matches > 0 && nbAttempts > 0; matches &= matches - 1) + { + uint matchPos = (headGrouped + ZSTD_VecMask_next(matches)) / groupWidth & rowMask; + uint matchIndex = row[matchPos]; + if (matchPos == 0) + { + continue; + } + + assert(numMatches < rowEntries); + if (matchIndex < lowLimit) + { + break; + } + + if (dictMode != ZSTD_dictMode_e.ZSTD_extDict || matchIndex >= dictLimit) + { +#if NETCOREAPP3_0_OR_GREATER + if (Sse.IsSupported) + { + Sse.Prefetch0(@base + matchIndex); + } +#endif + } + else + { +#if NETCOREAPP3_0_OR_GREATER + if (Sse.IsSupported) + { + Sse.Prefetch0(dictBase + matchIndex); + } +#endif + } + + matchBuffer[numMatches++] = matchIndex; + --nbAttempts; + } + + { + uint pos = ZSTD_row_nextIndex(tagRow, rowMask); + tagRow[pos] = (byte)tag; + row[pos] = ms->nextToUpdate++; + } + + for (; currMatch < numMatches; ++currMatch) + { + uint matchIndex = matchBuffer[currMatch]; + nuint currentMl = 0; + assert(matchIndex < curr); + assert(matchIndex >= lowLimit); + if (dictMode != ZSTD_dictMode_e.ZSTD_extDict || matchIndex >= dictLimit) + { + byte* match = @base + matchIndex; + assert(matchIndex >= dictLimit); + if (MEM_read32(match + ml - 3) == MEM_read32(ip + ml - 3)) + { + currentMl = ZSTD_count(ip, match, iLimit); + } + } + else + { + byte* match = dictBase + matchIndex; + assert(match + 4 <= dictEnd); + if (MEM_read32(match) == MEM_read32(ip)) + { + currentMl = + ZSTD_count_2segments(ip + 4, match + 4, iLimit, dictEnd, prefixStart) + + 4; + } + } + + if (currentMl > ml) + { + ml = currentMl; + assert(curr - matchIndex > 0); + *offsetPtr = curr - matchIndex + 3; + if (ip + currentMl == iLimit) + { + break; + } + } + } + } + + assert(nbAttempts <= 1U << (sizeof(nuint) == 4 ? 30 : 31) - 1); + if (dictMode == ZSTD_dictMode_e.ZSTD_dedicatedDictSearch) + { + ml = ZSTD_dedicatedDictSearch_lazy_search( + offsetPtr, + ml, + nbAttempts + ddsExtraAttempts, + dms, + ip, + iLimit, + prefixStart, + curr, + dictLimit, + ddsIdx + ); + } + else if (dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState) + { + /* TODO: Measure and potentially add prefetching to DMS */ + uint dmsLowestIndex = dms->window.dictLimit; + byte* dmsBase = dms->window.@base; + byte* dmsEnd = dms->window.nextSrc; + uint dmsSize = (uint)(dmsEnd - dmsBase); + uint dmsIndexDelta = dictLimit - dmsSize; + { + uint headGrouped = (*dmsTagRow & rowMask) * groupWidth; + uint* matchBuffer = stackalloc uint[64]; + nuint numMatches = 0; + nuint currMatch = 0; + ulong matches = ZSTD_row_getMatchMask( + dmsTagRow, + (byte)dmsTag, + headGrouped, + rowEntries + ); + for (; matches > 0 && nbAttempts > 0; matches &= matches - 1) + { + uint matchPos = + (headGrouped + ZSTD_VecMask_next(matches)) / groupWidth & rowMask; + uint matchIndex = dmsRow[matchPos]; + if (matchPos == 0) + { + continue; + } + + if (matchIndex < dmsLowestIndex) + { + break; + } +#if NETCOREAPP3_0_OR_GREATER + if (Sse.IsSupported) + { + Sse.Prefetch0(dmsBase + matchIndex); + } +#endif + + matchBuffer[numMatches++] = matchIndex; + --nbAttempts; + } + + for (; currMatch < numMatches; ++currMatch) + { + uint matchIndex = matchBuffer[currMatch]; + nuint currentMl = 0; + assert(matchIndex >= dmsLowestIndex); + assert(matchIndex < curr); + { + byte* match = dmsBase + matchIndex; + assert(match + 4 <= dmsEnd); + if (MEM_read32(match) == MEM_read32(ip)) + { + currentMl = + ZSTD_count_2segments(ip + 4, match + 4, iLimit, dmsEnd, prefixStart) + + 4; + } + } + + if (currentMl > ml) + { + ml = currentMl; + assert(curr > matchIndex + dmsIndexDelta); + assert(curr - (matchIndex + dmsIndexDelta) > 0); + *offsetPtr = curr - (matchIndex + dmsIndexDelta) + 3; + if (ip + currentMl == iLimit) + { + break; + } + } + } + } + } + + return ml; + } + + /* Generate row search fns for each combination of (dictMode, mls, rowLog) */ + private static nuint ZSTD_RowFindBestMatch_noDict_4_4( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 4 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 4 + ); + return ZSTD_RowFindBestMatch(ms, ip, iLimit, offsetPtr, 4, ZSTD_dictMode_e.ZSTD_noDict, 4); + } + + private static nuint ZSTD_RowFindBestMatch_noDict_4_5( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 4 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 5 + ); + return ZSTD_RowFindBestMatch(ms, ip, iLimit, offsetPtr, 4, ZSTD_dictMode_e.ZSTD_noDict, 5); + } + + private static nuint ZSTD_RowFindBestMatch_noDict_4_6( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 4 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 6 + ); + return ZSTD_RowFindBestMatch(ms, ip, iLimit, offsetPtr, 4, ZSTD_dictMode_e.ZSTD_noDict, 6); + } + + private static nuint ZSTD_RowFindBestMatch_noDict_5_4( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 5 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 4 + ); + return ZSTD_RowFindBestMatch(ms, ip, iLimit, offsetPtr, 5, ZSTD_dictMode_e.ZSTD_noDict, 4); + } + + private static nuint ZSTD_RowFindBestMatch_noDict_5_5( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 5 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 5 + ); + return ZSTD_RowFindBestMatch(ms, ip, iLimit, offsetPtr, 5, ZSTD_dictMode_e.ZSTD_noDict, 5); + } + + private static nuint ZSTD_RowFindBestMatch_noDict_5_6( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 5 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 6 + ); + return ZSTD_RowFindBestMatch(ms, ip, iLimit, offsetPtr, 5, ZSTD_dictMode_e.ZSTD_noDict, 6); + } + + private static nuint ZSTD_RowFindBestMatch_noDict_6_4( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 6 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 4 + ); + return ZSTD_RowFindBestMatch(ms, ip, iLimit, offsetPtr, 6, ZSTD_dictMode_e.ZSTD_noDict, 4); + } + + private static nuint ZSTD_RowFindBestMatch_noDict_6_5( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 6 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 5 + ); + return ZSTD_RowFindBestMatch(ms, ip, iLimit, offsetPtr, 6, ZSTD_dictMode_e.ZSTD_noDict, 5); + } + + private static nuint ZSTD_RowFindBestMatch_noDict_6_6( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 6 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 6 + ); + return ZSTD_RowFindBestMatch(ms, ip, iLimit, offsetPtr, 6, ZSTD_dictMode_e.ZSTD_noDict, 6); + } + + private static nuint ZSTD_RowFindBestMatch_extDict_4_4( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 4 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 4 + ); + return ZSTD_RowFindBestMatch(ms, ip, iLimit, offsetPtr, 4, ZSTD_dictMode_e.ZSTD_extDict, 4); + } + + private static nuint ZSTD_RowFindBestMatch_extDict_4_5( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 4 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 5 + ); + return ZSTD_RowFindBestMatch(ms, ip, iLimit, offsetPtr, 4, ZSTD_dictMode_e.ZSTD_extDict, 5); + } + + private static nuint ZSTD_RowFindBestMatch_extDict_4_6( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 4 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 6 + ); + return ZSTD_RowFindBestMatch(ms, ip, iLimit, offsetPtr, 4, ZSTD_dictMode_e.ZSTD_extDict, 6); + } + + private static nuint ZSTD_RowFindBestMatch_extDict_5_4( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 5 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 4 + ); + return ZSTD_RowFindBestMatch(ms, ip, iLimit, offsetPtr, 5, ZSTD_dictMode_e.ZSTD_extDict, 4); + } + + private static nuint ZSTD_RowFindBestMatch_extDict_5_5( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 5 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 5 + ); + return ZSTD_RowFindBestMatch(ms, ip, iLimit, offsetPtr, 5, ZSTD_dictMode_e.ZSTD_extDict, 5); + } + + private static nuint ZSTD_RowFindBestMatch_extDict_5_6( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 5 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 6 + ); + return ZSTD_RowFindBestMatch(ms, ip, iLimit, offsetPtr, 5, ZSTD_dictMode_e.ZSTD_extDict, 6); + } + + private static nuint ZSTD_RowFindBestMatch_extDict_6_4( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 6 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 4 + ); + return ZSTD_RowFindBestMatch(ms, ip, iLimit, offsetPtr, 6, ZSTD_dictMode_e.ZSTD_extDict, 4); + } + + private static nuint ZSTD_RowFindBestMatch_extDict_6_5( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 6 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 5 + ); + return ZSTD_RowFindBestMatch(ms, ip, iLimit, offsetPtr, 6, ZSTD_dictMode_e.ZSTD_extDict, 5); + } + + private static nuint ZSTD_RowFindBestMatch_extDict_6_6( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 6 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 6 + ); + return ZSTD_RowFindBestMatch(ms, ip, iLimit, offsetPtr, 6, ZSTD_dictMode_e.ZSTD_extDict, 6); + } + + private static nuint ZSTD_RowFindBestMatch_dictMatchState_4_4( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 4 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 4 + ); + return ZSTD_RowFindBestMatch( + ms, + ip, + iLimit, + offsetPtr, + 4, + ZSTD_dictMode_e.ZSTD_dictMatchState, + 4 + ); + } + + private static nuint ZSTD_RowFindBestMatch_dictMatchState_4_5( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 4 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 5 + ); + return ZSTD_RowFindBestMatch( + ms, + ip, + iLimit, + offsetPtr, + 4, + ZSTD_dictMode_e.ZSTD_dictMatchState, + 5 + ); + } + + private static nuint ZSTD_RowFindBestMatch_dictMatchState_4_6( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 4 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 6 + ); + return ZSTD_RowFindBestMatch( + ms, + ip, + iLimit, + offsetPtr, + 4, + ZSTD_dictMode_e.ZSTD_dictMatchState, + 6 + ); + } + + private static nuint ZSTD_RowFindBestMatch_dictMatchState_5_4( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 5 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 4 + ); + return ZSTD_RowFindBestMatch( + ms, + ip, + iLimit, + offsetPtr, + 5, + ZSTD_dictMode_e.ZSTD_dictMatchState, + 4 + ); + } + + private static nuint ZSTD_RowFindBestMatch_dictMatchState_5_5( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 5 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 5 + ); + return ZSTD_RowFindBestMatch( + ms, + ip, + iLimit, + offsetPtr, + 5, + ZSTD_dictMode_e.ZSTD_dictMatchState, + 5 + ); + } + + private static nuint ZSTD_RowFindBestMatch_dictMatchState_5_6( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 5 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 6 + ); + return ZSTD_RowFindBestMatch( + ms, + ip, + iLimit, + offsetPtr, + 5, + ZSTD_dictMode_e.ZSTD_dictMatchState, + 6 + ); + } + + private static nuint ZSTD_RowFindBestMatch_dictMatchState_6_4( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 6 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 4 + ); + return ZSTD_RowFindBestMatch( + ms, + ip, + iLimit, + offsetPtr, + 6, + ZSTD_dictMode_e.ZSTD_dictMatchState, + 4 + ); + } + + private static nuint ZSTD_RowFindBestMatch_dictMatchState_6_5( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 6 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 5 + ); + return ZSTD_RowFindBestMatch( + ms, + ip, + iLimit, + offsetPtr, + 6, + ZSTD_dictMode_e.ZSTD_dictMatchState, + 5 + ); + } + + private static nuint ZSTD_RowFindBestMatch_dictMatchState_6_6( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 6 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 6 + ); + return ZSTD_RowFindBestMatch( + ms, + ip, + iLimit, + offsetPtr, + 6, + ZSTD_dictMode_e.ZSTD_dictMatchState, + 6 + ); + } + + private static nuint ZSTD_RowFindBestMatch_dedicatedDictSearch_4_4( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 4 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 4 + ); + return ZSTD_RowFindBestMatch( + ms, + ip, + iLimit, + offsetPtr, + 4, + ZSTD_dictMode_e.ZSTD_dedicatedDictSearch, + 4 + ); + } + + private static nuint ZSTD_RowFindBestMatch_dedicatedDictSearch_4_5( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 4 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 5 + ); + return ZSTD_RowFindBestMatch( + ms, + ip, + iLimit, + offsetPtr, + 4, + ZSTD_dictMode_e.ZSTD_dedicatedDictSearch, + 5 + ); + } + + private static nuint ZSTD_RowFindBestMatch_dedicatedDictSearch_4_6( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 4 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 6 + ); + return ZSTD_RowFindBestMatch( + ms, + ip, + iLimit, + offsetPtr, + 4, + ZSTD_dictMode_e.ZSTD_dedicatedDictSearch, + 6 + ); + } + + private static nuint ZSTD_RowFindBestMatch_dedicatedDictSearch_5_4( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 5 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 4 + ); + return ZSTD_RowFindBestMatch( + ms, + ip, + iLimit, + offsetPtr, + 5, + ZSTD_dictMode_e.ZSTD_dedicatedDictSearch, + 4 + ); + } + + private static nuint ZSTD_RowFindBestMatch_dedicatedDictSearch_5_5( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 5 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 5 + ); + return ZSTD_RowFindBestMatch( + ms, + ip, + iLimit, + offsetPtr, + 5, + ZSTD_dictMode_e.ZSTD_dedicatedDictSearch, + 5 + ); + } + + private static nuint ZSTD_RowFindBestMatch_dedicatedDictSearch_5_6( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 5 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 6 + ); + return ZSTD_RowFindBestMatch( + ms, + ip, + iLimit, + offsetPtr, + 5, + ZSTD_dictMode_e.ZSTD_dedicatedDictSearch, + 6 + ); + } + + private static nuint ZSTD_RowFindBestMatch_dedicatedDictSearch_6_4( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 6 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 4 + ); + return ZSTD_RowFindBestMatch( + ms, + ip, + iLimit, + offsetPtr, + 6, + ZSTD_dictMode_e.ZSTD_dedicatedDictSearch, + 4 + ); + } + + private static nuint ZSTD_RowFindBestMatch_dedicatedDictSearch_6_5( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 6 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 5 + ); + return ZSTD_RowFindBestMatch( + ms, + ip, + iLimit, + offsetPtr, + 6, + ZSTD_dictMode_e.ZSTD_dedicatedDictSearch, + 5 + ); + } + + private static nuint ZSTD_RowFindBestMatch_dedicatedDictSearch_6_6( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 6 + ); + assert( + ( + 4 > (6 < ms->cParams.searchLog ? 6 : ms->cParams.searchLog) ? 4 + : 6 < ms->cParams.searchLog ? 6 + : ms->cParams.searchLog + ) == 6 + ); + return ZSTD_RowFindBestMatch( + ms, + ip, + iLimit, + offsetPtr, + 6, + ZSTD_dictMode_e.ZSTD_dedicatedDictSearch, + 6 + ); + } + + /* Generate binary Tree search fns for each combination of (dictMode, mls) */ + private static nuint ZSTD_BtFindBestMatch_noDict_4( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offBasePtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 4 + ); + return ZSTD_BtFindBestMatch(ms, ip, iLimit, offBasePtr, 4, ZSTD_dictMode_e.ZSTD_noDict); + } + + private static nuint ZSTD_BtFindBestMatch_noDict_5( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offBasePtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 5 + ); + return ZSTD_BtFindBestMatch(ms, ip, iLimit, offBasePtr, 5, ZSTD_dictMode_e.ZSTD_noDict); + } + + private static nuint ZSTD_BtFindBestMatch_noDict_6( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offBasePtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 6 + ); + return ZSTD_BtFindBestMatch(ms, ip, iLimit, offBasePtr, 6, ZSTD_dictMode_e.ZSTD_noDict); + } + + private static nuint ZSTD_BtFindBestMatch_extDict_4( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offBasePtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 4 + ); + return ZSTD_BtFindBestMatch(ms, ip, iLimit, offBasePtr, 4, ZSTD_dictMode_e.ZSTD_extDict); + } + + private static nuint ZSTD_BtFindBestMatch_extDict_5( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offBasePtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 5 + ); + return ZSTD_BtFindBestMatch(ms, ip, iLimit, offBasePtr, 5, ZSTD_dictMode_e.ZSTD_extDict); + } + + private static nuint ZSTD_BtFindBestMatch_extDict_6( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offBasePtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 6 + ); + return ZSTD_BtFindBestMatch(ms, ip, iLimit, offBasePtr, 6, ZSTD_dictMode_e.ZSTD_extDict); + } + + private static nuint ZSTD_BtFindBestMatch_dictMatchState_4( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offBasePtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 4 + ); + return ZSTD_BtFindBestMatch( + ms, + ip, + iLimit, + offBasePtr, + 4, + ZSTD_dictMode_e.ZSTD_dictMatchState + ); + } + + private static nuint ZSTD_BtFindBestMatch_dictMatchState_5( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offBasePtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 5 + ); + return ZSTD_BtFindBestMatch( + ms, + ip, + iLimit, + offBasePtr, + 5, + ZSTD_dictMode_e.ZSTD_dictMatchState + ); + } + + private static nuint ZSTD_BtFindBestMatch_dictMatchState_6( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offBasePtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 6 + ); + return ZSTD_BtFindBestMatch( + ms, + ip, + iLimit, + offBasePtr, + 6, + ZSTD_dictMode_e.ZSTD_dictMatchState + ); + } + + private static nuint ZSTD_BtFindBestMatch_dedicatedDictSearch_4( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offBasePtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 4 + ); + return ZSTD_BtFindBestMatch( + ms, + ip, + iLimit, + offBasePtr, + 4, + ZSTD_dictMode_e.ZSTD_dedicatedDictSearch + ); + } + + private static nuint ZSTD_BtFindBestMatch_dedicatedDictSearch_5( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offBasePtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 5 + ); + return ZSTD_BtFindBestMatch( + ms, + ip, + iLimit, + offBasePtr, + 5, + ZSTD_dictMode_e.ZSTD_dedicatedDictSearch + ); + } + + private static nuint ZSTD_BtFindBestMatch_dedicatedDictSearch_6( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offBasePtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 6 + ); + return ZSTD_BtFindBestMatch( + ms, + ip, + iLimit, + offBasePtr, + 6, + ZSTD_dictMode_e.ZSTD_dedicatedDictSearch + ); + } + + /* Generate hash chain search fns for each combination of (dictMode, mls) */ + private static nuint ZSTD_HcFindBestMatch_noDict_4( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 4 + ); + return ZSTD_HcFindBestMatch(ms, ip, iLimit, offsetPtr, 4, ZSTD_dictMode_e.ZSTD_noDict); + } + + private static nuint ZSTD_HcFindBestMatch_noDict_5( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 5 + ); + return ZSTD_HcFindBestMatch(ms, ip, iLimit, offsetPtr, 5, ZSTD_dictMode_e.ZSTD_noDict); + } + + private static nuint ZSTD_HcFindBestMatch_noDict_6( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 6 + ); + return ZSTD_HcFindBestMatch(ms, ip, iLimit, offsetPtr, 6, ZSTD_dictMode_e.ZSTD_noDict); + } + + private static nuint ZSTD_HcFindBestMatch_extDict_4( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 4 + ); + return ZSTD_HcFindBestMatch(ms, ip, iLimit, offsetPtr, 4, ZSTD_dictMode_e.ZSTD_extDict); + } + + private static nuint ZSTD_HcFindBestMatch_extDict_5( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 5 + ); + return ZSTD_HcFindBestMatch(ms, ip, iLimit, offsetPtr, 5, ZSTD_dictMode_e.ZSTD_extDict); + } + + private static nuint ZSTD_HcFindBestMatch_extDict_6( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 6 + ); + return ZSTD_HcFindBestMatch(ms, ip, iLimit, offsetPtr, 6, ZSTD_dictMode_e.ZSTD_extDict); + } + + private static nuint ZSTD_HcFindBestMatch_dictMatchState_4( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 4 + ); + return ZSTD_HcFindBestMatch( + ms, + ip, + iLimit, + offsetPtr, + 4, + ZSTD_dictMode_e.ZSTD_dictMatchState + ); + } + + private static nuint ZSTD_HcFindBestMatch_dictMatchState_5( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 5 + ); + return ZSTD_HcFindBestMatch( + ms, + ip, + iLimit, + offsetPtr, + 5, + ZSTD_dictMode_e.ZSTD_dictMatchState + ); + } + + private static nuint ZSTD_HcFindBestMatch_dictMatchState_6( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 6 + ); + return ZSTD_HcFindBestMatch( + ms, + ip, + iLimit, + offsetPtr, + 6, + ZSTD_dictMode_e.ZSTD_dictMatchState + ); + } + + private static nuint ZSTD_HcFindBestMatch_dedicatedDictSearch_4( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 4 + ); + return ZSTD_HcFindBestMatch( + ms, + ip, + iLimit, + offsetPtr, + 4, + ZSTD_dictMode_e.ZSTD_dedicatedDictSearch + ); + } + + private static nuint ZSTD_HcFindBestMatch_dedicatedDictSearch_5( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 5 + ); + return ZSTD_HcFindBestMatch( + ms, + ip, + iLimit, + offsetPtr, + 5, + ZSTD_dictMode_e.ZSTD_dedicatedDictSearch + ); + } + + private static nuint ZSTD_HcFindBestMatch_dedicatedDictSearch_6( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iLimit, + nuint* offsetPtr + ) + { + assert( + ( + 4 > (6 < ms->cParams.minMatch ? 6 : ms->cParams.minMatch) ? 4 + : 6 < ms->cParams.minMatch ? 6 + : ms->cParams.minMatch + ) == 6 + ); + return ZSTD_HcFindBestMatch( + ms, + ip, + iLimit, + offsetPtr, + 6, + ZSTD_dictMode_e.ZSTD_dedicatedDictSearch + ); + } + + /** + * Searches for the longest match at @p ip. + * Dispatches to the correct implementation function based on the + * (searchMethod, dictMode, mls, rowLog). We use switch statements + * here instead of using an indirect function call through a function + * pointer because after Spectre and Meltdown mitigations, indirect + * function calls can be very costly, especially in the kernel. + * + * NOTE: dictMode and searchMethod should be templated, so those switch + * statements should be optimized out. Only the mls & rowLog switches + * should be left. + * + * @param ms The match state. + * @param ip The position to search at. + * @param iend The end of the input data. + * @param[out] offsetPtr Stores the match offset into this pointer. + * @param mls The minimum search length, in the range [4, 6]. + * @param rowLog The row log (if applicable), in the range [4, 6]. + * @param searchMethod The search method to use (templated). + * @param dictMode The dictMode (templated). + * + * @returns The length of the longest match found, or < mls if no match is found. + * If a match is found its offset is stored in @p offsetPtr. + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_searchMax( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iend, + nuint* offsetPtr, + uint mls, + uint rowLog, + searchMethod_e searchMethod, + ZSTD_dictMode_e dictMode + ) + { + if (dictMode == ZSTD_dictMode_e.ZSTD_noDict) + { + if (searchMethod == searchMethod_e.search_rowHash) + { + if (mls == 4) + { + if (rowLog == 4) + { + return ZSTD_RowFindBestMatch_noDict_4_4(ms, ip, iend, offsetPtr); + } + + return rowLog == 5 + ? ZSTD_RowFindBestMatch_noDict_4_5(ms, ip, iend, offsetPtr) + : ZSTD_RowFindBestMatch_noDict_4_6(ms, ip, iend, offsetPtr); + } + + if (mls == 5) + { + if (rowLog == 4) + { + return ZSTD_RowFindBestMatch_noDict_5_4(ms, ip, iend, offsetPtr); + } + + return rowLog == 5 + ? ZSTD_RowFindBestMatch_noDict_5_5(ms, ip, iend, offsetPtr) + : ZSTD_RowFindBestMatch_noDict_5_6(ms, ip, iend, offsetPtr); + } + + if (rowLog == 4) + { + return ZSTD_RowFindBestMatch_noDict_6_4(ms, ip, iend, offsetPtr); + } + + return rowLog == 5 + ? ZSTD_RowFindBestMatch_noDict_6_5(ms, ip, iend, offsetPtr) + : ZSTD_RowFindBestMatch_noDict_6_6(ms, ip, iend, offsetPtr); + } + + if (searchMethod == searchMethod_e.search_hashChain) + { + if (mls == 4) + { + return ZSTD_HcFindBestMatch_noDict_4(ms, ip, iend, offsetPtr); + } + + return mls == 5 + ? ZSTD_HcFindBestMatch_noDict_5(ms, ip, iend, offsetPtr) + : ZSTD_HcFindBestMatch_noDict_6(ms, ip, iend, offsetPtr); + } + + // searchMethod_e.search_binaryTree + if (mls == 4) + { + return ZSTD_BtFindBestMatch_noDict_4(ms, ip, iend, offsetPtr); + } + + return mls == 5 + ? ZSTD_BtFindBestMatch_noDict_5(ms, ip, iend, offsetPtr) + : ZSTD_BtFindBestMatch_noDict_6(ms, ip, iend, offsetPtr); + } + + if (dictMode == ZSTD_dictMode_e.ZSTD_extDict) + { + if (searchMethod == searchMethod_e.search_rowHash) + { + if (mls == 4) + { + if (rowLog == 4) + { + return ZSTD_RowFindBestMatch_extDict_4_4(ms, ip, iend, offsetPtr); + } + + if (rowLog == 5) + { + return ZSTD_RowFindBestMatch_extDict_4_5(ms, ip, iend, offsetPtr); + } + + return ZSTD_RowFindBestMatch_extDict_4_6(ms, ip, iend, offsetPtr); + } + + if (mls == 5) + { + if (rowLog == 4) + { + return ZSTD_RowFindBestMatch_extDict_5_4(ms, ip, iend, offsetPtr); + } + + if (rowLog == 5) + { + return ZSTD_RowFindBestMatch_extDict_5_5(ms, ip, iend, offsetPtr); + } + + return ZSTD_RowFindBestMatch_extDict_5_6(ms, ip, iend, offsetPtr); + } + + if (mls == 6) + { + if (rowLog == 4) + { + return ZSTD_RowFindBestMatch_extDict_6_4(ms, ip, iend, offsetPtr); + } + + if (rowLog == 5) + { + return ZSTD_RowFindBestMatch_extDict_6_5(ms, ip, iend, offsetPtr); + } + + return ZSTD_RowFindBestMatch_extDict_6_6(ms, ip, iend, offsetPtr); + } + } + + if (searchMethod == searchMethod_e.search_hashChain) + { + if (mls == 4) + { + return ZSTD_HcFindBestMatch_extDict_4(ms, ip, iend, offsetPtr); + } + + if (mls == 5) + { + return ZSTD_HcFindBestMatch_extDict_5(ms, ip, iend, offsetPtr); + } + + return ZSTD_HcFindBestMatch_extDict_6(ms, ip, iend, offsetPtr); + } + + // searchMethod_e.search_binaryTree + if (mls == 4) + { + return ZSTD_BtFindBestMatch_extDict_4(ms, ip, iend, offsetPtr); + } + + if (mls == 5) + { + return ZSTD_BtFindBestMatch_extDict_5(ms, ip, iend, offsetPtr); + } + + return ZSTD_BtFindBestMatch_extDict_6(ms, ip, iend, offsetPtr); + } + + if (dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState) + { + if (searchMethod == searchMethod_e.search_rowHash) + { + if (mls == 4) + { + if (rowLog == 4) + { + return ZSTD_RowFindBestMatch_dictMatchState_4_4(ms, ip, iend, offsetPtr); + } + + if (rowLog == 5) + { + return ZSTD_RowFindBestMatch_dictMatchState_4_5(ms, ip, iend, offsetPtr); + } + + return ZSTD_RowFindBestMatch_dictMatchState_4_6(ms, ip, iend, offsetPtr); + } + + if (mls == 5) + { + if (rowLog == 4) + { + return ZSTD_RowFindBestMatch_dictMatchState_5_4(ms, ip, iend, offsetPtr); + } + + if (rowLog == 5) + { + return ZSTD_RowFindBestMatch_dictMatchState_5_5(ms, ip, iend, offsetPtr); + } + + return ZSTD_RowFindBestMatch_dictMatchState_5_6(ms, ip, iend, offsetPtr); + } + + if (mls == 6) + { + if (rowLog == 4) + { + return ZSTD_RowFindBestMatch_dictMatchState_6_4(ms, ip, iend, offsetPtr); + } + + if (rowLog == 5) + { + return ZSTD_RowFindBestMatch_dictMatchState_6_5(ms, ip, iend, offsetPtr); + } + + return ZSTD_RowFindBestMatch_dictMatchState_6_6(ms, ip, iend, offsetPtr); + } + } + + if (searchMethod == searchMethod_e.search_hashChain) + { + if (mls == 4) + { + return ZSTD_HcFindBestMatch_dictMatchState_4(ms, ip, iend, offsetPtr); + } + + if (mls == 5) + { + return ZSTD_HcFindBestMatch_dictMatchState_5(ms, ip, iend, offsetPtr); + } + + return ZSTD_HcFindBestMatch_dictMatchState_6(ms, ip, iend, offsetPtr); + } + + // search_binaryTree + if (mls == 4) + { + return ZSTD_BtFindBestMatch_dictMatchState_4(ms, ip, iend, offsetPtr); + } + + if (mls == 5) + { + return ZSTD_BtFindBestMatch_dictMatchState_5(ms, ip, iend, offsetPtr); + } + + return ZSTD_BtFindBestMatch_dictMatchState_6(ms, ip, iend, offsetPtr); + } + + if (searchMethod == searchMethod_e.search_rowHash) + { + if (mls == 4) + { + if (rowLog == 4) + { + return ZSTD_RowFindBestMatch_dedicatedDictSearch_4_4(ms, ip, iend, offsetPtr); + } + + if (rowLog == 5) + { + return ZSTD_RowFindBestMatch_dedicatedDictSearch_4_5(ms, ip, iend, offsetPtr); + } + + return ZSTD_RowFindBestMatch_dedicatedDictSearch_4_6(ms, ip, iend, offsetPtr); + } + + if (mls == 5) + { + if (rowLog == 4) + { + return ZSTD_RowFindBestMatch_dedicatedDictSearch_5_4(ms, ip, iend, offsetPtr); + } + + if (rowLog == 5) + { + return ZSTD_RowFindBestMatch_dedicatedDictSearch_5_5(ms, ip, iend, offsetPtr); + } + + return ZSTD_RowFindBestMatch_dedicatedDictSearch_5_6(ms, ip, iend, offsetPtr); + } + + if (mls == 6) + { + if (rowLog == 4) + { + return ZSTD_RowFindBestMatch_dedicatedDictSearch_6_4(ms, ip, iend, offsetPtr); + } + + if (rowLog == 5) + { + return ZSTD_RowFindBestMatch_dedicatedDictSearch_6_5(ms, ip, iend, offsetPtr); + } + + return ZSTD_RowFindBestMatch_dedicatedDictSearch_6_6(ms, ip, iend, offsetPtr); + } + } + + if (searchMethod == searchMethod_e.search_hashChain) + { + if (mls == 4) + { + return ZSTD_HcFindBestMatch_dedicatedDictSearch_4(ms, ip, iend, offsetPtr); + } + + if (mls == 5) + { + return ZSTD_HcFindBestMatch_dedicatedDictSearch_5(ms, ip, iend, offsetPtr); + } + + return ZSTD_HcFindBestMatch_dedicatedDictSearch_6(ms, ip, iend, offsetPtr); + } + + // searchMethod_e.search_binaryTree + if (mls == 4) + { + return ZSTD_BtFindBestMatch_dedicatedDictSearch_4(ms, ip, iend, offsetPtr); + } + + if (mls == 5) + { + return ZSTD_BtFindBestMatch_dedicatedDictSearch_5(ms, ip, iend, offsetPtr); + } + + return ZSTD_BtFindBestMatch_dedicatedDictSearch_6(ms, ip, iend, offsetPtr); + } + + /* ******************************* + * Common parser - lazy strategy + *********************************/ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_compressBlock_lazy_generic( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize, + searchMethod_e searchMethod, + uint depth, + ZSTD_dictMode_e dictMode + ) + { + byte* istart = (byte*)src; + byte* ip = istart; + byte* anchor = istart; + byte* iend = istart + srcSize; + byte* ilimit = searchMethod == searchMethod_e.search_rowHash ? iend - 8 - 8 : iend - 8; + byte* @base = ms->window.@base; + uint prefixLowestIndex = ms->window.dictLimit; + byte* prefixLowest = @base + prefixLowestIndex; + uint mls = + ms->cParams.minMatch <= 4 ? 4 + : ms->cParams.minMatch <= 6 ? ms->cParams.minMatch + : 6; + uint rowLog = + ms->cParams.searchLog <= 4 ? 4 + : ms->cParams.searchLog <= 6 ? ms->cParams.searchLog + : 6; + uint offset_1 = rep[0], + offset_2 = rep[1]; + uint offsetSaved1 = 0, + offsetSaved2 = 0; + int isDMS = dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState ? 1 : 0; + int isDDS = dictMode == ZSTD_dictMode_e.ZSTD_dedicatedDictSearch ? 1 : 0; + int isDxS = isDMS != 0 || isDDS != 0 ? 1 : 0; + ZSTD_MatchState_t* dms = ms->dictMatchState; + uint dictLowestIndex = isDxS != 0 ? dms->window.dictLimit : 0; + byte* dictBase = isDxS != 0 ? dms->window.@base : null; + byte* dictLowest = isDxS != 0 ? dictBase + dictLowestIndex : null; + byte* dictEnd = isDxS != 0 ? dms->window.nextSrc : null; + uint dictIndexDelta = isDxS != 0 ? prefixLowestIndex - (uint)(dictEnd - dictBase) : 0; + uint dictAndPrefixLength = (uint)(ip - prefixLowest + (dictEnd - dictLowest)); + ip += dictAndPrefixLength == 0 ? 1 : 0; + if (dictMode == ZSTD_dictMode_e.ZSTD_noDict) + { + uint curr = (uint)(ip - @base); + uint windowLow = ZSTD_getLowestPrefixIndex(ms, curr, ms->cParams.windowLog); + uint maxRep = curr - windowLow; + if (offset_2 > maxRep) + { + offsetSaved2 = offset_2; + offset_2 = 0; + } + + if (offset_1 > maxRep) + { + offsetSaved1 = offset_1; + offset_1 = 0; + } + } + + ms->lazySkipping = 0; + if (searchMethod == searchMethod_e.search_rowHash) + { + ZSTD_row_fillHashCache(ms, @base, rowLog, mls, ms->nextToUpdate, ilimit); + } + + while (ip < ilimit) + { + nuint matchLength = 0; + assert(1 >= 1); + assert(1 <= 3); + nuint offBase = 1; + byte* start = ip + 1; + if (isDxS != 0) + { + uint repIndex = (uint)(ip - @base) + 1 - offset_1; + byte* repMatch = + ( + dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState + || dictMode == ZSTD_dictMode_e.ZSTD_dedicatedDictSearch + ) + && repIndex < prefixLowestIndex + ? dictBase + (repIndex - dictIndexDelta) + : @base + repIndex; + if ( + ZSTD_index_overlap_check(prefixLowestIndex, repIndex) != 0 + && MEM_read32(repMatch) == MEM_read32(ip + 1) + ) + { + byte* repMatchEnd = repIndex < prefixLowestIndex ? dictEnd : iend; + matchLength = + ZSTD_count_2segments( + ip + 1 + 4, + repMatch + 4, + iend, + repMatchEnd, + prefixLowest + ) + 4; + if (depth == 0) + { + goto _storeSequence; + } + } + } + + if ( + dictMode == ZSTD_dictMode_e.ZSTD_noDict + && offset_1 > 0 + && MEM_read32(ip + 1 - offset_1) == MEM_read32(ip + 1) + ) + { + matchLength = ZSTD_count(ip + 1 + 4, ip + 1 + 4 - offset_1, iend) + 4; + if (depth == 0) + { + goto _storeSequence; + } + } + + { + nuint offbaseFound = 999999999; + nuint ml2 = ZSTD_searchMax( + ms, + ip, + iend, + &offbaseFound, + mls, + rowLog, + searchMethod, + dictMode + ); + if (ml2 > matchLength) + { + matchLength = ml2; + start = ip; + offBase = offbaseFound; + } + } + + if (matchLength < 4) + { + /* jump faster over incompressible sections */ + nuint step = ((nuint)(ip - anchor) >> 8) + 1; + ip += step; + ms->lazySkipping = step > 8 ? 1 : 0; + continue; + } + + if (depth >= 1) + { + while (ip < ilimit) + { + ip++; + if ( + dictMode == ZSTD_dictMode_e.ZSTD_noDict + && offBase != 0 + && offset_1 > 0 + && MEM_read32(ip) == MEM_read32(ip - offset_1) + ) + { + nuint mlRep = ZSTD_count(ip + 4, ip + 4 - offset_1, iend) + 4; + int gain2 = (int)(mlRep * 3); + int gain1 = (int)(matchLength * 3 - ZSTD_highbit32((uint)offBase) + 1); + if (mlRep >= 4 && gain2 > gain1) + { + matchLength = mlRep; + assert(1 >= 1); + assert(1 <= 3); + offBase = 1; + start = ip; + } + } + + if (isDxS != 0) + { + uint repIndex = (uint)(ip - @base) - offset_1; + byte* repMatch = + repIndex < prefixLowestIndex + ? dictBase + (repIndex - dictIndexDelta) + : @base + repIndex; + if ( + ZSTD_index_overlap_check(prefixLowestIndex, repIndex) != 0 + && MEM_read32(repMatch) == MEM_read32(ip) + ) + { + byte* repMatchEnd = repIndex < prefixLowestIndex ? dictEnd : iend; + nuint mlRep = + ZSTD_count_2segments( + ip + 4, + repMatch + 4, + iend, + repMatchEnd, + prefixLowest + ) + 4; + int gain2 = (int)(mlRep * 3); + int gain1 = (int)(matchLength * 3 - ZSTD_highbit32((uint)offBase) + 1); + if (mlRep >= 4 && gain2 > gain1) + { + matchLength = mlRep; + assert(1 >= 1); + assert(1 <= 3); + offBase = 1; + start = ip; + } + } + } + + { + nuint ofbCandidate = 999999999; + nuint ml2 = ZSTD_searchMax( + ms, + ip, + iend, + &ofbCandidate, + mls, + rowLog, + searchMethod, + dictMode + ); + /* raw approx */ + int gain2 = (int)(ml2 * 4 - ZSTD_highbit32((uint)ofbCandidate)); + int gain1 = (int)(matchLength * 4 - ZSTD_highbit32((uint)offBase) + 4); + if (ml2 >= 4 && gain2 > gain1) + { + matchLength = ml2; + offBase = ofbCandidate; + start = ip; + continue; + } + } + + if (depth == 2 && ip < ilimit) + { + ip++; + if ( + dictMode == ZSTD_dictMode_e.ZSTD_noDict + && offBase != 0 + && offset_1 > 0 + && MEM_read32(ip) == MEM_read32(ip - offset_1) + ) + { + nuint mlRep = ZSTD_count(ip + 4, ip + 4 - offset_1, iend) + 4; + int gain2 = (int)(mlRep * 4); + int gain1 = (int)(matchLength * 4 - ZSTD_highbit32((uint)offBase) + 1); + if (mlRep >= 4 && gain2 > gain1) + { + matchLength = mlRep; + assert(1 >= 1); + assert(1 <= 3); + offBase = 1; + start = ip; + } + } + + if (isDxS != 0) + { + uint repIndex = (uint)(ip - @base) - offset_1; + byte* repMatch = + repIndex < prefixLowestIndex + ? dictBase + (repIndex - dictIndexDelta) + : @base + repIndex; + if ( + ZSTD_index_overlap_check(prefixLowestIndex, repIndex) != 0 + && MEM_read32(repMatch) == MEM_read32(ip) + ) + { + byte* repMatchEnd = repIndex < prefixLowestIndex ? dictEnd : iend; + nuint mlRep = + ZSTD_count_2segments( + ip + 4, + repMatch + 4, + iend, + repMatchEnd, + prefixLowest + ) + 4; + int gain2 = (int)(mlRep * 4); + int gain1 = (int)( + matchLength * 4 - ZSTD_highbit32((uint)offBase) + 1 + ); + if (mlRep >= 4 && gain2 > gain1) + { + matchLength = mlRep; + assert(1 >= 1); + assert(1 <= 3); + offBase = 1; + start = ip; + } + } + } + + { + nuint ofbCandidate = 999999999; + nuint ml2 = ZSTD_searchMax( + ms, + ip, + iend, + &ofbCandidate, + mls, + rowLog, + searchMethod, + dictMode + ); + /* raw approx */ + int gain2 = (int)(ml2 * 4 - ZSTD_highbit32((uint)ofbCandidate)); + int gain1 = (int)(matchLength * 4 - ZSTD_highbit32((uint)offBase) + 7); + if (ml2 >= 4 && gain2 > gain1) + { + matchLength = ml2; + offBase = ofbCandidate; + start = ip; + continue; + } + } + } + + break; + } + } + + if (offBase > 3) + { + if (dictMode == ZSTD_dictMode_e.ZSTD_noDict) + { + assert(offBase > 3); + assert(offBase > 3); + while ( + start > anchor + && start - (offBase - 3) > prefixLowest + && start[-1] == (start - (offBase - 3))[-1] + ) + { + start--; + matchLength++; + } + } + + if (isDxS != 0) + { + assert(offBase > 3); + uint matchIndex = (uint)((nuint)(start - @base) - (offBase - 3)); + byte* match = + matchIndex < prefixLowestIndex + ? dictBase + matchIndex - dictIndexDelta + : @base + matchIndex; + byte* mStart = matchIndex < prefixLowestIndex ? dictLowest : prefixLowest; + while (start > anchor && match > mStart && start[-1] == match[-1]) + { + start--; + match--; + matchLength++; + } + } + + offset_2 = offset_1; + assert(offBase > 3); + offset_1 = (uint)(offBase - 3); + } + + _storeSequence: + { + nuint litLength = (nuint)(start - anchor); + ZSTD_storeSeq(seqStore, litLength, anchor, iend, (uint)offBase, matchLength); + anchor = ip = start + matchLength; + } + + if (ms->lazySkipping != 0) + { + if (searchMethod == searchMethod_e.search_rowHash) + { + ZSTD_row_fillHashCache(ms, @base, rowLog, mls, ms->nextToUpdate, ilimit); + } + + ms->lazySkipping = 0; + } + + if (isDxS != 0) + { + while (ip <= ilimit) + { + uint current2 = (uint)(ip - @base); + uint repIndex = current2 - offset_2; + byte* repMatch = + repIndex < prefixLowestIndex + ? dictBase - dictIndexDelta + repIndex + : @base + repIndex; + if ( + ZSTD_index_overlap_check(prefixLowestIndex, repIndex) != 0 + && MEM_read32(repMatch) == MEM_read32(ip) + ) + { + byte* repEnd2 = repIndex < prefixLowestIndex ? dictEnd : iend; + matchLength = + ZSTD_count_2segments(ip + 4, repMatch + 4, iend, repEnd2, prefixLowest) + + 4; + offBase = offset_2; + offset_2 = offset_1; + offset_1 = (uint)offBase; + assert(1 >= 1); + assert(1 <= 3); + ZSTD_storeSeq(seqStore, 0, anchor, iend, 1, matchLength); + ip += matchLength; + anchor = ip; + continue; + } + + break; + } + } + + if (dictMode == ZSTD_dictMode_e.ZSTD_noDict) + { + while (ip <= ilimit && offset_2 > 0 && MEM_read32(ip) == MEM_read32(ip - offset_2)) + { + matchLength = ZSTD_count(ip + 4, ip + 4 - offset_2, iend) + 4; + offBase = offset_2; + offset_2 = offset_1; + offset_1 = (uint)offBase; + assert(1 >= 1); + assert(1 <= 3); + ZSTD_storeSeq(seqStore, 0, anchor, iend, 1, matchLength); + ip += matchLength; + anchor = ip; + continue; + } + } + } + + offsetSaved2 = offsetSaved1 != 0 && offset_1 != 0 ? offsetSaved1 : offsetSaved2; + rep[0] = offset_1 != 0 ? offset_1 : offsetSaved1; + rep[1] = offset_2 != 0 ? offset_2 : offsetSaved2; + return (nuint)(iend - anchor); + } + + private static nuint ZSTD_compressBlock_greedy( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_hashChain, + 0, + ZSTD_dictMode_e.ZSTD_noDict + ); + } + + private static nuint ZSTD_compressBlock_greedy_dictMatchState( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_hashChain, + 0, + ZSTD_dictMode_e.ZSTD_dictMatchState + ); + } + + private static nuint ZSTD_compressBlock_greedy_dedicatedDictSearch( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_hashChain, + 0, + ZSTD_dictMode_e.ZSTD_dedicatedDictSearch + ); + } + + private static nuint ZSTD_compressBlock_greedy_row( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_rowHash, + 0, + ZSTD_dictMode_e.ZSTD_noDict + ); + } + + private static nuint ZSTD_compressBlock_greedy_dictMatchState_row( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_rowHash, + 0, + ZSTD_dictMode_e.ZSTD_dictMatchState + ); + } + + private static nuint ZSTD_compressBlock_greedy_dedicatedDictSearch_row( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_rowHash, + 0, + ZSTD_dictMode_e.ZSTD_dedicatedDictSearch + ); + } + + private static nuint ZSTD_compressBlock_lazy( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_hashChain, + 1, + ZSTD_dictMode_e.ZSTD_noDict + ); + } + + private static nuint ZSTD_compressBlock_lazy_dictMatchState( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_hashChain, + 1, + ZSTD_dictMode_e.ZSTD_dictMatchState + ); + } + + private static nuint ZSTD_compressBlock_lazy_dedicatedDictSearch( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_hashChain, + 1, + ZSTD_dictMode_e.ZSTD_dedicatedDictSearch + ); + } + + private static nuint ZSTD_compressBlock_lazy_row( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_rowHash, + 1, + ZSTD_dictMode_e.ZSTD_noDict + ); + } + + private static nuint ZSTD_compressBlock_lazy_dictMatchState_row( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_rowHash, + 1, + ZSTD_dictMode_e.ZSTD_dictMatchState + ); + } + + private static nuint ZSTD_compressBlock_lazy_dedicatedDictSearch_row( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_rowHash, + 1, + ZSTD_dictMode_e.ZSTD_dedicatedDictSearch + ); + } + + private static nuint ZSTD_compressBlock_lazy2( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_hashChain, + 2, + ZSTD_dictMode_e.ZSTD_noDict + ); + } + + private static nuint ZSTD_compressBlock_lazy2_dictMatchState( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_hashChain, + 2, + ZSTD_dictMode_e.ZSTD_dictMatchState + ); + } + + private static nuint ZSTD_compressBlock_lazy2_dedicatedDictSearch( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_hashChain, + 2, + ZSTD_dictMode_e.ZSTD_dedicatedDictSearch + ); + } + + private static nuint ZSTD_compressBlock_lazy2_row( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_rowHash, + 2, + ZSTD_dictMode_e.ZSTD_noDict + ); + } + + private static nuint ZSTD_compressBlock_lazy2_dictMatchState_row( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_rowHash, + 2, + ZSTD_dictMode_e.ZSTD_dictMatchState + ); + } + + private static nuint ZSTD_compressBlock_lazy2_dedicatedDictSearch_row( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_rowHash, + 2, + ZSTD_dictMode_e.ZSTD_dedicatedDictSearch + ); + } + + private static nuint ZSTD_compressBlock_btlazy2( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_binaryTree, + 2, + ZSTD_dictMode_e.ZSTD_noDict + ); + } + + private static nuint ZSTD_compressBlock_btlazy2_dictMatchState( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_binaryTree, + 2, + ZSTD_dictMode_e.ZSTD_dictMatchState + ); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_compressBlock_lazy_extDict_generic( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize, + searchMethod_e searchMethod, + uint depth + ) + { + byte* istart = (byte*)src; + byte* ip = istart; + byte* anchor = istart; + byte* iend = istart + srcSize; + byte* ilimit = searchMethod == searchMethod_e.search_rowHash ? iend - 8 - 8 : iend - 8; + byte* @base = ms->window.@base; + uint dictLimit = ms->window.dictLimit; + byte* prefixStart = @base + dictLimit; + byte* dictBase = ms->window.dictBase; + byte* dictEnd = dictBase + dictLimit; + byte* dictStart = dictBase + ms->window.lowLimit; + uint windowLog = ms->cParams.windowLog; + uint mls = + ms->cParams.minMatch <= 4 ? 4 + : ms->cParams.minMatch <= 6 ? ms->cParams.minMatch + : 6; + uint rowLog = + ms->cParams.searchLog <= 4 ? 4 + : ms->cParams.searchLog <= 6 ? ms->cParams.searchLog + : 6; + uint offset_1 = rep[0], + offset_2 = rep[1]; + ms->lazySkipping = 0; + ip += ip == prefixStart ? 1 : 0; + if (searchMethod == searchMethod_e.search_rowHash) + { + ZSTD_row_fillHashCache(ms, @base, rowLog, mls, ms->nextToUpdate, ilimit); + } + + while (ip < ilimit) + { + nuint matchLength = 0; + assert(1 >= 1); + assert(1 <= 3); + nuint offBase = 1; + byte* start = ip + 1; + uint curr = (uint)(ip - @base); + { + uint windowLow = ZSTD_getLowestMatchIndex(ms, curr + 1, windowLog); + uint repIndex = curr + 1 - offset_1; + byte* repBase = repIndex < dictLimit ? dictBase : @base; + byte* repMatch = repBase + repIndex; + if ( + ( + ZSTD_index_overlap_check(dictLimit, repIndex) + & (offset_1 <= curr + 1 - windowLow ? 1 : 0) + ) != 0 + ) + { + if (MEM_read32(ip + 1) == MEM_read32(repMatch)) + { + /* repcode detected we should take it */ + byte* repEnd = repIndex < dictLimit ? dictEnd : iend; + matchLength = + ZSTD_count_2segments( + ip + 1 + 4, + repMatch + 4, + iend, + repEnd, + prefixStart + ) + 4; + if (depth == 0) + { + goto _storeSequence; + } + } + } + } + + { + nuint ofbCandidate = 999999999; + nuint ml2 = ZSTD_searchMax( + ms, + ip, + iend, + &ofbCandidate, + mls, + rowLog, + searchMethod, + ZSTD_dictMode_e.ZSTD_extDict + ); + if (ml2 > matchLength) + { + matchLength = ml2; + start = ip; + offBase = ofbCandidate; + } + } + + if (matchLength < 4) + { + nuint step = (nuint)(ip - anchor) >> 8; + ip += step + 1; + ms->lazySkipping = step > 8 ? 1 : 0; + continue; + } + + if (depth >= 1) + { + while (ip < ilimit) + { + ip++; + curr++; + if (offBase != 0) + { + uint windowLow = ZSTD_getLowestMatchIndex(ms, curr, windowLog); + uint repIndex = curr - offset_1; + byte* repBase = repIndex < dictLimit ? dictBase : @base; + byte* repMatch = repBase + repIndex; + if ( + ( + ZSTD_index_overlap_check(dictLimit, repIndex) + & (offset_1 <= curr - windowLow ? 1 : 0) + ) != 0 + ) + { + if (MEM_read32(ip) == MEM_read32(repMatch)) + { + /* repcode detected */ + byte* repEnd = repIndex < dictLimit ? dictEnd : iend; + nuint repLength = + ZSTD_count_2segments( + ip + 4, + repMatch + 4, + iend, + repEnd, + prefixStart + ) + 4; + int gain2 = (int)(repLength * 3); + int gain1 = (int)( + matchLength * 3 - ZSTD_highbit32((uint)offBase) + 1 + ); + if (repLength >= 4 && gain2 > gain1) + { + matchLength = repLength; + assert(1 >= 1); + assert(1 <= 3); + offBase = 1; + start = ip; + } + } + } + } + + { + nuint ofbCandidate = 999999999; + nuint ml2 = ZSTD_searchMax( + ms, + ip, + iend, + &ofbCandidate, + mls, + rowLog, + searchMethod, + ZSTD_dictMode_e.ZSTD_extDict + ); + /* raw approx */ + int gain2 = (int)(ml2 * 4 - ZSTD_highbit32((uint)ofbCandidate)); + int gain1 = (int)(matchLength * 4 - ZSTD_highbit32((uint)offBase) + 4); + if (ml2 >= 4 && gain2 > gain1) + { + matchLength = ml2; + offBase = ofbCandidate; + start = ip; + continue; + } + } + + if (depth == 2 && ip < ilimit) + { + ip++; + curr++; + if (offBase != 0) + { + uint windowLow = ZSTD_getLowestMatchIndex(ms, curr, windowLog); + uint repIndex = curr - offset_1; + byte* repBase = repIndex < dictLimit ? dictBase : @base; + byte* repMatch = repBase + repIndex; + if ( + ( + ZSTD_index_overlap_check(dictLimit, repIndex) + & (offset_1 <= curr - windowLow ? 1 : 0) + ) != 0 + ) + { + if (MEM_read32(ip) == MEM_read32(repMatch)) + { + /* repcode detected */ + byte* repEnd = repIndex < dictLimit ? dictEnd : iend; + nuint repLength = + ZSTD_count_2segments( + ip + 4, + repMatch + 4, + iend, + repEnd, + prefixStart + ) + 4; + int gain2 = (int)(repLength * 4); + int gain1 = (int)( + matchLength * 4 - ZSTD_highbit32((uint)offBase) + 1 + ); + if (repLength >= 4 && gain2 > gain1) + { + matchLength = repLength; + assert(1 >= 1); + assert(1 <= 3); + offBase = 1; + start = ip; + } + } + } + } + + { + nuint ofbCandidate = 999999999; + nuint ml2 = ZSTD_searchMax( + ms, + ip, + iend, + &ofbCandidate, + mls, + rowLog, + searchMethod, + ZSTD_dictMode_e.ZSTD_extDict + ); + /* raw approx */ + int gain2 = (int)(ml2 * 4 - ZSTD_highbit32((uint)ofbCandidate)); + int gain1 = (int)(matchLength * 4 - ZSTD_highbit32((uint)offBase) + 7); + if (ml2 >= 4 && gain2 > gain1) + { + matchLength = ml2; + offBase = ofbCandidate; + start = ip; + continue; + } + } + } + + break; + } + } + + if (offBase > 3) + { + assert(offBase > 3); + uint matchIndex = (uint)((nuint)(start - @base) - (offBase - 3)); + byte* match = matchIndex < dictLimit ? dictBase + matchIndex : @base + matchIndex; + byte* mStart = matchIndex < dictLimit ? dictStart : prefixStart; + while (start > anchor && match > mStart && start[-1] == match[-1]) + { + start--; + match--; + matchLength++; + } + + offset_2 = offset_1; + assert(offBase > 3); + offset_1 = (uint)(offBase - 3); + } + + _storeSequence: + { + nuint litLength = (nuint)(start - anchor); + ZSTD_storeSeq(seqStore, litLength, anchor, iend, (uint)offBase, matchLength); + anchor = ip = start + matchLength; + } + + if (ms->lazySkipping != 0) + { + if (searchMethod == searchMethod_e.search_rowHash) + { + ZSTD_row_fillHashCache(ms, @base, rowLog, mls, ms->nextToUpdate, ilimit); + } + + ms->lazySkipping = 0; + } + + while (ip <= ilimit) + { + uint repCurrent = (uint)(ip - @base); + uint windowLow = ZSTD_getLowestMatchIndex(ms, repCurrent, windowLog); + uint repIndex = repCurrent - offset_2; + byte* repBase = repIndex < dictLimit ? dictBase : @base; + byte* repMatch = repBase + repIndex; + if ( + ( + ZSTD_index_overlap_check(dictLimit, repIndex) + & (offset_2 <= repCurrent - windowLow ? 1 : 0) + ) != 0 + ) + { + if (MEM_read32(ip) == MEM_read32(repMatch)) + { + /* repcode detected we should take it */ + byte* repEnd = repIndex < dictLimit ? dictEnd : iend; + matchLength = + ZSTD_count_2segments(ip + 4, repMatch + 4, iend, repEnd, prefixStart) + + 4; + offBase = offset_2; + offset_2 = offset_1; + offset_1 = (uint)offBase; + assert(1 >= 1); + assert(1 <= 3); + ZSTD_storeSeq(seqStore, 0, anchor, iend, 1, matchLength); + ip += matchLength; + anchor = ip; + continue; + } + } + + break; + } + } + + rep[0] = offset_1; + rep[1] = offset_2; + return (nuint)(iend - anchor); + } + + private static nuint ZSTD_compressBlock_greedy_extDict( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_extDict_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_hashChain, + 0 + ); + } + + private static nuint ZSTD_compressBlock_greedy_extDict_row( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_extDict_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_rowHash, + 0 + ); + } + + private static nuint ZSTD_compressBlock_lazy_extDict( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_extDict_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_hashChain, + 1 + ); + } + + private static nuint ZSTD_compressBlock_lazy_extDict_row( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_extDict_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_rowHash, + 1 + ); + } + + private static nuint ZSTD_compressBlock_lazy2_extDict( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_extDict_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_hashChain, + 2 + ); + } + + private static nuint ZSTD_compressBlock_lazy2_extDict_row( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_extDict_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_rowHash, + 2 + ); + } + + private static nuint ZSTD_compressBlock_btlazy2_extDict( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_lazy_extDict_generic( + ms, + seqStore, + rep, + src, + srcSize, + searchMethod_e.search_binaryTree, + 2 + ); + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdLdm.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdLdm.cs new file mode 100644 index 00000000..db905095 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdLdm.cs @@ -0,0 +1,971 @@ +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + /** ZSTD_ldm_gear_init(): + * + * Initializes the rolling hash state such that it will honor the + * settings in params. */ + private static void ZSTD_ldm_gear_init(ldmRollingHashState_t* state, ldmParams_t* @params) + { + uint maxBitsInMask = @params->minMatchLength < 64 ? @params->minMatchLength : 64; + uint hashRateLog = @params->hashRateLog; + state->rolling = ~(uint)0; + if (hashRateLog > 0 && hashRateLog <= maxBitsInMask) + { + state->stopMask = + ((ulong)1 << (int)hashRateLog) - 1 << (int)(maxBitsInMask - hashRateLog); + } + else + { + state->stopMask = ((ulong)1 << (int)hashRateLog) - 1; + } + } + + /** ZSTD_ldm_gear_reset() + * Feeds [data, data + minMatchLength) into the hash without registering any + * splits. This effectively resets the hash state. This is used when skipping + * over data, either at the beginning of a block, or skipping sections. + */ + private static void ZSTD_ldm_gear_reset( + ldmRollingHashState_t* state, + byte* data, + nuint minMatchLength + ) + { + ulong hash = state->rolling; + nuint n = 0; + while (n + 3 < minMatchLength) + { + { + hash = (hash << 1) + ZSTD_ldm_gearTab[data[n] & 0xff]; + n += 1; + } + + { + hash = (hash << 1) + ZSTD_ldm_gearTab[data[n] & 0xff]; + n += 1; + } + + { + hash = (hash << 1) + ZSTD_ldm_gearTab[data[n] & 0xff]; + n += 1; + } + + { + hash = (hash << 1) + ZSTD_ldm_gearTab[data[n] & 0xff]; + n += 1; + } + } + + while (n < minMatchLength) + { + hash = (hash << 1) + ZSTD_ldm_gearTab[data[n] & 0xff]; + n += 1; + } + } + + /** ZSTD_ldm_gear_feed(): + * + * Registers in the splits array all the split points found in the first + * size bytes following the data pointer. This function terminates when + * either all the data has been processed or LDM_BATCH_SIZE splits are + * present in the splits array. + * + * Precondition: The splits array must not be full. + * Returns: The number of bytes processed. */ + private static nuint ZSTD_ldm_gear_feed( + ldmRollingHashState_t* state, + byte* data, + nuint size, + nuint* splits, + uint* numSplits + ) + { + nuint n; + ulong hash, + mask; + hash = state->rolling; + mask = state->stopMask; + n = 0; + while (n + 3 < size) + { + { + hash = (hash << 1) + ZSTD_ldm_gearTab[data[n] & 0xff]; + n += 1; + if ((hash & mask) == 0) + { + splits[*numSplits] = n; + *numSplits += 1; + if (*numSplits == 64) + { + goto done; + } + } + } + + { + hash = (hash << 1) + ZSTD_ldm_gearTab[data[n] & 0xff]; + n += 1; + if ((hash & mask) == 0) + { + splits[*numSplits] = n; + *numSplits += 1; + if (*numSplits == 64) + { + goto done; + } + } + } + + { + hash = (hash << 1) + ZSTD_ldm_gearTab[data[n] & 0xff]; + n += 1; + if ((hash & mask) == 0) + { + splits[*numSplits] = n; + *numSplits += 1; + if (*numSplits == 64) + { + goto done; + } + } + } + + { + hash = (hash << 1) + ZSTD_ldm_gearTab[data[n] & 0xff]; + n += 1; + if ((hash & mask) == 0) + { + splits[*numSplits] = n; + *numSplits += 1; + if (*numSplits == 64) + { + goto done; + } + } + } + } + + while (n < size) + { + hash = (hash << 1) + ZSTD_ldm_gearTab[data[n] & 0xff]; + n += 1; + if ((hash & mask) == 0) + { + splits[*numSplits] = n; + *numSplits += 1; + if (*numSplits == 64) + { + goto done; + } + } + } + + done: + state->rolling = hash; + return n; + } + + /** ZSTD_ldm_adjustParameters() : + * If the params->hashRateLog is not set, set it to its default value based on + * windowLog and params->hashLog. + * + * Ensures that params->bucketSizeLog is <= params->hashLog (setting it to + * params->hashLog if it is not). + * + * Ensures that the minMatchLength >= targetLength during optimal parsing. + */ + private static void ZSTD_ldm_adjustParameters( + ldmParams_t* @params, + ZSTD_compressionParameters* cParams + ) + { + @params->windowLog = cParams->windowLog; + if (@params->hashRateLog == 0) + { + if (@params->hashLog > 0) + { + assert( + @params->hashLog + <= (uint)( + (sizeof(nuint) == 4 ? 30 : 31) < 30 + ? sizeof(nuint) == 4 + ? 30 + : 31 + : 30 + ) + ); + if (@params->windowLog > @params->hashLog) + { + @params->hashRateLog = @params->windowLog - @params->hashLog; + } + } + else + { + assert(1 <= (int)cParams->strategy && (int)cParams->strategy <= 9); + @params->hashRateLog = (uint)(7 - (int)cParams->strategy / 3); + } + } + + if (@params->hashLog == 0) + { + @params->hashLog = + @params->windowLog - @params->hashRateLog <= 6 ? 6 + : @params->windowLog - @params->hashRateLog + <= (uint)( + (sizeof(nuint) == 4 ? 30 : 31) < 30 + ? sizeof(nuint) == 4 + ? 30 + : 31 + : 30 + ) + ? @params->windowLog - @params->hashRateLog + : (uint)( + (sizeof(nuint) == 4 ? 30 : 31) < 30 + ? sizeof(nuint) == 4 + ? 30 + : 31 + : 30 + ); + } + + if (@params->minMatchLength == 0) + { + @params->minMatchLength = 64; + if (cParams->strategy >= ZSTD_strategy.ZSTD_btultra) + { + @params->minMatchLength /= 2; + } + } + + if (@params->bucketSizeLog == 0) + { + assert(1 <= (int)cParams->strategy && (int)cParams->strategy <= 9); + @params->bucketSizeLog = + (uint)cParams->strategy <= 4 ? 4 + : (uint)cParams->strategy <= 8 ? (uint)cParams->strategy + : 8; + } + + @params->bucketSizeLog = + @params->bucketSizeLog < @params->hashLog ? @params->bucketSizeLog : @params->hashLog; + } + + /** ZSTD_ldm_getTableSize() : + * Estimate the space needed for long distance matching tables or 0 if LDM is + * disabled. + */ + private static nuint ZSTD_ldm_getTableSize(ldmParams_t @params) + { + nuint ldmHSize = (nuint)1 << (int)@params.hashLog; + nuint ldmBucketSizeLog = + @params.bucketSizeLog < @params.hashLog ? @params.bucketSizeLog : @params.hashLog; + nuint ldmBucketSize = (nuint)1 << (int)(@params.hashLog - ldmBucketSizeLog); + nuint totalSize = + ZSTD_cwksp_alloc_size(ldmBucketSize) + + ZSTD_cwksp_alloc_size(ldmHSize * (nuint)sizeof(ldmEntry_t)); + return @params.enableLdm == ZSTD_paramSwitch_e.ZSTD_ps_enable ? totalSize : 0; + } + + /** ZSTD_ldm_getSeqSpace() : + * Return an upper bound on the number of sequences that can be produced by + * the long distance matcher, or 0 if LDM is disabled. + */ + private static nuint ZSTD_ldm_getMaxNbSeq(ldmParams_t @params, nuint maxChunkSize) + { + return @params.enableLdm == ZSTD_paramSwitch_e.ZSTD_ps_enable + ? maxChunkSize / @params.minMatchLength + : 0; + } + + /** ZSTD_ldm_getBucket() : + * Returns a pointer to the start of the bucket associated with hash. */ + private static ldmEntry_t* ZSTD_ldm_getBucket( + ldmState_t* ldmState, + nuint hash, + uint bucketSizeLog + ) + { + return ldmState->hashTable + (hash << (int)bucketSizeLog); + } + + /** ZSTD_ldm_insertEntry() : + * Insert the entry with corresponding hash into the hash table */ + private static void ZSTD_ldm_insertEntry( + ldmState_t* ldmState, + nuint hash, + ldmEntry_t entry, + uint bucketSizeLog + ) + { + byte* pOffset = ldmState->bucketOffsets + hash; + uint offset = *pOffset; + *(ZSTD_ldm_getBucket(ldmState, hash, bucketSizeLog) + offset) = entry; + *pOffset = (byte)(offset + 1 & (1U << (int)bucketSizeLog) - 1); + } + + /** ZSTD_ldm_countBackwardsMatch() : + * Returns the number of bytes that match backwards before pIn and pMatch. + * + * We count only bytes where pMatch >= pBase and pIn >= pAnchor. */ + private static nuint ZSTD_ldm_countBackwardsMatch( + byte* pIn, + byte* pAnchor, + byte* pMatch, + byte* pMatchBase + ) + { + nuint matchLength = 0; + while (pIn > pAnchor && pMatch > pMatchBase && pIn[-1] == pMatch[-1]) + { + pIn--; + pMatch--; + matchLength++; + } + + return matchLength; + } + + /** ZSTD_ldm_countBackwardsMatch_2segments() : + * Returns the number of bytes that match backwards from pMatch, + * even with the backwards match spanning 2 different segments. + * + * On reaching `pMatchBase`, start counting from mEnd */ + private static nuint ZSTD_ldm_countBackwardsMatch_2segments( + byte* pIn, + byte* pAnchor, + byte* pMatch, + byte* pMatchBase, + byte* pExtDictStart, + byte* pExtDictEnd + ) + { + nuint matchLength = ZSTD_ldm_countBackwardsMatch(pIn, pAnchor, pMatch, pMatchBase); + if (pMatch - matchLength != pMatchBase || pMatchBase == pExtDictStart) + { + return matchLength; + } + + matchLength += ZSTD_ldm_countBackwardsMatch( + pIn - matchLength, + pAnchor, + pExtDictEnd, + pExtDictStart + ); + return matchLength; + } + + /** ZSTD_ldm_fillFastTables() : + * + * Fills the relevant tables for the ZSTD_fast and ZSTD_dfast strategies. + * This is similar to ZSTD_loadDictionaryContent. + * + * The tables for the other strategies are filled within their + * block compressors. */ + private static nuint ZSTD_ldm_fillFastTables(ZSTD_MatchState_t* ms, void* end) + { + byte* iend = (byte*)end; + switch (ms->cParams.strategy) + { + case ZSTD_strategy.ZSTD_fast: + ZSTD_fillHashTable( + ms, + iend, + ZSTD_dictTableLoadMethod_e.ZSTD_dtlm_fast, + ZSTD_tableFillPurpose_e.ZSTD_tfp_forCCtx + ); + break; + case ZSTD_strategy.ZSTD_dfast: + ZSTD_fillDoubleHashTable( + ms, + iend, + ZSTD_dictTableLoadMethod_e.ZSTD_dtlm_fast, + ZSTD_tableFillPurpose_e.ZSTD_tfp_forCCtx + ); + break; + case ZSTD_strategy.ZSTD_greedy: + case ZSTD_strategy.ZSTD_lazy: + case ZSTD_strategy.ZSTD_lazy2: + case ZSTD_strategy.ZSTD_btlazy2: + case ZSTD_strategy.ZSTD_btopt: + case ZSTD_strategy.ZSTD_btultra: + case ZSTD_strategy.ZSTD_btultra2: + break; + default: + assert(0 != 0); + break; + } + + return 0; + } + + private static void ZSTD_ldm_fillHashTable( + ldmState_t* ldmState, + byte* ip, + byte* iend, + ldmParams_t* @params + ) + { + uint minMatchLength = @params->minMatchLength; + uint bucketSizeLog = @params->bucketSizeLog; + uint hBits = @params->hashLog - bucketSizeLog; + byte* @base = ldmState->window.@base; + byte* istart = ip; + ldmRollingHashState_t hashState; + nuint* splits = &ldmState->splitIndices.e0; + uint numSplits; + ZSTD_ldm_gear_init(&hashState, @params); + while (ip < iend) + { + nuint hashed; + uint n; + numSplits = 0; + hashed = ZSTD_ldm_gear_feed(&hashState, ip, (nuint)(iend - ip), splits, &numSplits); + for (n = 0; n < numSplits; n++) + { + if (ip + splits[n] >= istart + minMatchLength) + { + byte* split = ip + splits[n] - minMatchLength; + ulong xxhash = ZSTD_XXH64(split, minMatchLength, 0); + uint hash = (uint)(xxhash & ((uint)1 << (int)hBits) - 1); + ldmEntry_t entry; + entry.offset = (uint)(split - @base); + entry.checksum = (uint)(xxhash >> 32); + ZSTD_ldm_insertEntry(ldmState, hash, entry, @params->bucketSizeLog); + } + } + + ip += hashed; + } + } + + /** ZSTD_ldm_limitTableUpdate() : + * + * Sets cctx->nextToUpdate to a position corresponding closer to anchor + * if it is far way + * (after a long match, only update tables a limited amount). */ + private static void ZSTD_ldm_limitTableUpdate(ZSTD_MatchState_t* ms, byte* anchor) + { + uint curr = (uint)(anchor - ms->window.@base); + if (curr > ms->nextToUpdate + 1024) + { + ms->nextToUpdate = + curr + - (512 < curr - ms->nextToUpdate - 1024 ? 512 : curr - ms->nextToUpdate - 1024); + } + } + + private static nuint ZSTD_ldm_generateSequences_internal( + ldmState_t* ldmState, + RawSeqStore_t* rawSeqStore, + ldmParams_t* @params, + void* src, + nuint srcSize + ) + { + /* LDM parameters */ + int extDict = (int)ZSTD_window_hasExtDict(ldmState->window); + uint minMatchLength = @params->minMatchLength; + uint entsPerBucket = 1U << (int)@params->bucketSizeLog; + uint hBits = @params->hashLog - @params->bucketSizeLog; + /* Prefix and extDict parameters */ + uint dictLimit = ldmState->window.dictLimit; + uint lowestIndex = extDict != 0 ? ldmState->window.lowLimit : dictLimit; + byte* @base = ldmState->window.@base; + byte* dictBase = extDict != 0 ? ldmState->window.dictBase : null; + byte* dictStart = extDict != 0 ? dictBase + lowestIndex : null; + byte* dictEnd = extDict != 0 ? dictBase + dictLimit : null; + byte* lowPrefixPtr = @base + dictLimit; + /* Input bounds */ + byte* istart = (byte*)src; + byte* iend = istart + srcSize; + byte* ilimit = iend - 8; + /* Input positions */ + byte* anchor = istart; + byte* ip = istart; + /* Rolling hash state */ + ldmRollingHashState_t hashState; + /* Arrays for staged-processing */ + nuint* splits = &ldmState->splitIndices.e0; + ldmMatchCandidate_t* candidates = &ldmState->matchCandidates.e0; + uint numSplits; + if (srcSize < minMatchLength) + { + return (nuint)(iend - anchor); + } + + ZSTD_ldm_gear_init(&hashState, @params); + ZSTD_ldm_gear_reset(&hashState, ip, minMatchLength); + ip += minMatchLength; + while (ip < ilimit) + { + nuint hashed; + uint n; + numSplits = 0; + hashed = ZSTD_ldm_gear_feed(&hashState, ip, (nuint)(ilimit - ip), splits, &numSplits); + for (n = 0; n < numSplits; n++) + { + byte* split = ip + splits[n] - minMatchLength; + ulong xxhash = ZSTD_XXH64(split, minMatchLength, 0); + uint hash = (uint)(xxhash & ((uint)1 << (int)hBits) - 1); + candidates[n].split = split; + candidates[n].hash = hash; + candidates[n].checksum = (uint)(xxhash >> 32); + candidates[n].bucket = ZSTD_ldm_getBucket(ldmState, hash, @params->bucketSizeLog); +#if NETCOREAPP3_0_OR_GREATER + if (System.Runtime.Intrinsics.X86.Sse.IsSupported) + { + System.Runtime.Intrinsics.X86.Sse.Prefetch0(candidates[n].bucket); + } +#endif + } + + for (n = 0; n < numSplits; n++) + { + nuint forwardMatchLength = 0, + backwardMatchLength = 0, + bestMatchLength = 0, + mLength; + uint offset; + byte* split = candidates[n].split; + uint checksum = candidates[n].checksum; + uint hash = candidates[n].hash; + ldmEntry_t* bucket = candidates[n].bucket; + ldmEntry_t* cur; + ldmEntry_t* bestEntry = null; + ldmEntry_t newEntry; + newEntry.offset = (uint)(split - @base); + newEntry.checksum = checksum; + if (split < anchor) + { + ZSTD_ldm_insertEntry(ldmState, hash, newEntry, @params->bucketSizeLog); + continue; + } + + for (cur = bucket; cur < bucket + entsPerBucket; cur++) + { + nuint curForwardMatchLength, + curBackwardMatchLength, + curTotalMatchLength; + if (cur->checksum != checksum || cur->offset <= lowestIndex) + { + continue; + } + + if (extDict != 0) + { + byte* curMatchBase = cur->offset < dictLimit ? dictBase : @base; + byte* pMatch = curMatchBase + cur->offset; + byte* matchEnd = cur->offset < dictLimit ? dictEnd : iend; + byte* lowMatchPtr = cur->offset < dictLimit ? dictStart : lowPrefixPtr; + curForwardMatchLength = ZSTD_count_2segments( + split, + pMatch, + iend, + matchEnd, + lowPrefixPtr + ); + if (curForwardMatchLength < minMatchLength) + { + continue; + } + + curBackwardMatchLength = ZSTD_ldm_countBackwardsMatch_2segments( + split, + anchor, + pMatch, + lowMatchPtr, + dictStart, + dictEnd + ); + } + else + { + byte* pMatch = @base + cur->offset; + curForwardMatchLength = ZSTD_count(split, pMatch, iend); + if (curForwardMatchLength < minMatchLength) + { + continue; + } + + curBackwardMatchLength = ZSTD_ldm_countBackwardsMatch( + split, + anchor, + pMatch, + lowPrefixPtr + ); + } + + curTotalMatchLength = curForwardMatchLength + curBackwardMatchLength; + if (curTotalMatchLength > bestMatchLength) + { + bestMatchLength = curTotalMatchLength; + forwardMatchLength = curForwardMatchLength; + backwardMatchLength = curBackwardMatchLength; + bestEntry = cur; + } + } + + if (bestEntry == null) + { + ZSTD_ldm_insertEntry(ldmState, hash, newEntry, @params->bucketSizeLog); + continue; + } + + offset = (uint)(split - @base) - bestEntry->offset; + mLength = forwardMatchLength + backwardMatchLength; + { + rawSeq* seq = rawSeqStore->seq + rawSeqStore->size; + if (rawSeqStore->size == rawSeqStore->capacity) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_dstSize_tooSmall)); + } + + seq->litLength = (uint)(split - backwardMatchLength - anchor); + seq->matchLength = (uint)mLength; + seq->offset = offset; + rawSeqStore->size++; + } + + ZSTD_ldm_insertEntry(ldmState, hash, newEntry, @params->bucketSizeLog); + anchor = split + forwardMatchLength; + if (anchor > ip + hashed) + { + ZSTD_ldm_gear_reset(&hashState, anchor - minMatchLength, minMatchLength); + ip = anchor - hashed; + break; + } + } + + ip += hashed; + } + + return (nuint)(iend - anchor); + } + + /*! ZSTD_ldm_reduceTable() : + * reduce table indexes by `reducerValue` */ + private static void ZSTD_ldm_reduceTable(ldmEntry_t* table, uint size, uint reducerValue) + { + uint u; + for (u = 0; u < size; u++) + { + if (table[u].offset < reducerValue) + { + table[u].offset = 0; + } + else + { + table[u].offset -= reducerValue; + } + } + } + + /** + * ZSTD_ldm_generateSequences(): + * + * Generates the sequences using the long distance match finder. + * Generates long range matching sequences in `sequences`, which parse a prefix + * of the source. `sequences` must be large enough to store every sequence, + * which can be checked with `ZSTD_ldm_getMaxNbSeq()`. + * @returns 0 or an error code. + * + * NOTE: The user must have called ZSTD_window_update() for all of the input + * they have, even if they pass it to ZSTD_ldm_generateSequences() in chunks. + * NOTE: This function returns an error if it runs out of space to store + * sequences. + */ + private static nuint ZSTD_ldm_generateSequences( + ldmState_t* ldmState, + RawSeqStore_t* sequences, + ldmParams_t* @params, + void* src, + nuint srcSize + ) + { + uint maxDist = 1U << (int)@params->windowLog; + byte* istart = (byte*)src; + byte* iend = istart + srcSize; + const nuint kMaxChunkSize = 1 << 20; + nuint nbChunks = srcSize / kMaxChunkSize + (nuint)(srcSize % kMaxChunkSize != 0 ? 1 : 0); + nuint chunk; + nuint leftoverSize = 0; + assert( + unchecked((uint)-1) - (MEM_64bits ? 3500U * (1 << 20) : 2000U * (1 << 20)) + >= kMaxChunkSize + ); + assert(ldmState->window.nextSrc >= (byte*)src + srcSize); + assert(sequences->pos <= sequences->size); + assert(sequences->size <= sequences->capacity); + for (chunk = 0; chunk < nbChunks && sequences->size < sequences->capacity; ++chunk) + { + byte* chunkStart = istart + chunk * kMaxChunkSize; + nuint remaining = (nuint)(iend - chunkStart); + byte* chunkEnd = remaining < kMaxChunkSize ? iend : chunkStart + kMaxChunkSize; + nuint chunkSize = (nuint)(chunkEnd - chunkStart); + nuint newLeftoverSize; + nuint prevSize = sequences->size; + assert(chunkStart < iend); + if ( + ZSTD_window_needOverflowCorrection( + ldmState->window, + 0, + maxDist, + ldmState->loadedDictEnd, + chunkStart, + chunkEnd + ) != 0 + ) + { + uint ldmHSize = 1U << (int)@params->hashLog; + uint correction = ZSTD_window_correctOverflow( + &ldmState->window, + 0, + maxDist, + chunkStart + ); + ZSTD_ldm_reduceTable(ldmState->hashTable, ldmHSize, correction); + ldmState->loadedDictEnd = 0; + } + + ZSTD_window_enforceMaxDist( + &ldmState->window, + chunkEnd, + maxDist, + &ldmState->loadedDictEnd, + null + ); + newLeftoverSize = ZSTD_ldm_generateSequences_internal( + ldmState, + sequences, + @params, + chunkStart, + chunkSize + ); + if (ERR_isError(newLeftoverSize)) + { + return newLeftoverSize; + } + + if (prevSize < sequences->size) + { + sequences->seq[prevSize].litLength += (uint)leftoverSize; + leftoverSize = newLeftoverSize; + } + else + { + assert(newLeftoverSize == chunkSize); + leftoverSize += chunkSize; + } + } + + return 0; + } + + /** + * ZSTD_ldm_skipSequences(): + * + * Skip past `srcSize` bytes worth of sequences in `rawSeqStore`. + * Avoids emitting matches less than `minMatch` bytes. + * Must be called for data that is not passed to ZSTD_ldm_blockCompress(). + */ + private static void ZSTD_ldm_skipSequences( + RawSeqStore_t* rawSeqStore, + nuint srcSize, + uint minMatch + ) + { + while (srcSize > 0 && rawSeqStore->pos < rawSeqStore->size) + { + rawSeq* seq = rawSeqStore->seq + rawSeqStore->pos; + if (srcSize <= seq->litLength) + { + seq->litLength -= (uint)srcSize; + return; + } + + srcSize -= seq->litLength; + seq->litLength = 0; + if (srcSize < seq->matchLength) + { + seq->matchLength -= (uint)srcSize; + if (seq->matchLength < minMatch) + { + if (rawSeqStore->pos + 1 < rawSeqStore->size) + { + seq[1].litLength += seq[0].matchLength; + } + + rawSeqStore->pos++; + } + + return; + } + + srcSize -= seq->matchLength; + seq->matchLength = 0; + rawSeqStore->pos++; + } + } + + /** + * If the sequence length is longer than remaining then the sequence is split + * between this block and the next. + * + * Returns the current sequence to handle, or if the rest of the block should + * be literals, it returns a sequence with offset == 0. + */ + private static rawSeq maybeSplitSequence( + RawSeqStore_t* rawSeqStore, + uint remaining, + uint minMatch + ) + { + rawSeq sequence = rawSeqStore->seq[rawSeqStore->pos]; + assert(sequence.offset > 0); + if (remaining >= sequence.litLength + sequence.matchLength) + { + rawSeqStore->pos++; + return sequence; + } + + if (remaining <= sequence.litLength) + { + sequence.offset = 0; + } + else if (remaining < sequence.litLength + sequence.matchLength) + { + sequence.matchLength = remaining - sequence.litLength; + if (sequence.matchLength < minMatch) + { + sequence.offset = 0; + } + } + + ZSTD_ldm_skipSequences(rawSeqStore, remaining, minMatch); + return sequence; + } + + /* ZSTD_ldm_skipRawSeqStoreBytes(): + * Moves forward in rawSeqStore by nbBytes, updating fields 'pos' and 'posInSequence'. + * Not to be used in conjunction with ZSTD_ldm_skipSequences(). + * Must be called for data with is not passed to ZSTD_ldm_blockCompress(). + */ + private static void ZSTD_ldm_skipRawSeqStoreBytes(RawSeqStore_t* rawSeqStore, nuint nbBytes) + { + uint currPos = (uint)(rawSeqStore->posInSequence + nbBytes); + while (currPos != 0 && rawSeqStore->pos < rawSeqStore->size) + { + rawSeq currSeq = rawSeqStore->seq[rawSeqStore->pos]; + if (currPos >= currSeq.litLength + currSeq.matchLength) + { + currPos -= currSeq.litLength + currSeq.matchLength; + rawSeqStore->pos++; + } + else + { + rawSeqStore->posInSequence = currPos; + break; + } + } + + if (currPos == 0 || rawSeqStore->pos == rawSeqStore->size) + { + rawSeqStore->posInSequence = 0; + } + } + + /** + * ZSTD_ldm_blockCompress(): + * + * Compresses a block using the predefined sequences, along with a secondary + * block compressor. The literals section of every sequence is passed to the + * secondary block compressor, and those sequences are interspersed with the + * predefined sequences. Returns the length of the last literals. + * Updates `rawSeqStore.pos` to indicate how many sequences have been consumed. + * `rawSeqStore.seq` may also be updated to split the last sequence between two + * blocks. + * @return The length of the last literals. + * + * NOTE: The source must be at most the maximum block size, but the predefined + * sequences can be any size, and may be longer than the block. In the case that + * they are longer than the block, the last sequences may need to be split into + * two. We handle that case correctly, and update `rawSeqStore` appropriately. + * NOTE: This function does not return any errors. + */ + private static nuint ZSTD_ldm_blockCompress( + RawSeqStore_t* rawSeqStore, + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + ZSTD_paramSwitch_e useRowMatchFinder, + void* src, + nuint srcSize + ) + { + ZSTD_compressionParameters* cParams = &ms->cParams; + uint minMatch = cParams->minMatch; + ZSTD_BlockCompressor_f blockCompressor = ZSTD_selectBlockCompressor( + cParams->strategy, + useRowMatchFinder, + ZSTD_matchState_dictMode(ms) + ); + /* Input bounds */ + byte* istart = (byte*)src; + byte* iend = istart + srcSize; + /* Input positions */ + byte* ip = istart; + if (cParams->strategy >= ZSTD_strategy.ZSTD_btopt) + { + nuint lastLLSize; + ms->ldmSeqStore = rawSeqStore; + lastLLSize = blockCompressor(ms, seqStore, rep, src, srcSize); + ZSTD_ldm_skipRawSeqStoreBytes(rawSeqStore, srcSize); + return lastLLSize; + } + + assert(rawSeqStore->pos <= rawSeqStore->size); + assert(rawSeqStore->size <= rawSeqStore->capacity); + while (rawSeqStore->pos < rawSeqStore->size && ip < iend) + { + /* maybeSplitSequence updates rawSeqStore->pos */ + rawSeq sequence = maybeSplitSequence(rawSeqStore, (uint)(iend - ip), minMatch); + if (sequence.offset == 0) + { + break; + } + + assert(ip + sequence.litLength + sequence.matchLength <= iend); + ZSTD_ldm_limitTableUpdate(ms, ip); + ZSTD_ldm_fillFastTables(ms, ip); + { + int i; + nuint newLitLength = blockCompressor(ms, seqStore, rep, ip, sequence.litLength); + ip += sequence.litLength; + for (i = 3 - 1; i > 0; i--) + { + rep[i] = rep[i - 1]; + } + + rep[0] = sequence.offset; + assert(sequence.offset > 0); + ZSTD_storeSeq( + seqStore, + newLitLength, + ip - newLitLength, + iend, + sequence.offset + 3, + sequence.matchLength + ); + ip += sequence.matchLength; + } + } + + ZSTD_ldm_limitTableUpdate(ms, ip); + ZSTD_ldm_fillFastTables(ms, ip); + return blockCompressor(ms, seqStore, rep, ip, (nuint)(iend - ip)); + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdLdmGeartab.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdLdmGeartab.cs new file mode 100644 index 00000000..5edf1197 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdLdmGeartab.cs @@ -0,0 +1,539 @@ +using System; +using System.Runtime.InteropServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_ZSTD_ldm_gearTab => + new ulong[256] + { + 0xf5b8f72c5f77775c, + 0x84935f266b7ac412, + 0xb647ada9ca730ccc, + 0xb065bb4b114fb1de, + 0x34584e7e8c3a9fd0, + 0x4e97e17c6ae26b05, + 0x3a03d743bc99a604, + 0xcecd042422c4044f, + 0x76de76c58524259e, + 0x9c8528f65badeaca, + 0x86563706e2097529, + 0x2902475fa375d889, + 0xafb32a9739a5ebe6, + 0xce2714da3883e639, + 0x21eaf821722e69e, + 0x37b628620b628, + 0x49a8d455d88caf5, + 0x8556d711e6958140, + 0x4f7ae74fc605c1f, + 0x829f0c3468bd3a20, + 0x4ffdc885c625179e, + 0x8473de048a3daf1b, + 0x51008822b05646b2, + 0x69d75d12b2d1cc5f, + 0x8c9d4a19159154bc, + 0xc3cc10f4abbd4003, + 0xd06ddc1cecb97391, + 0xbe48e6e7ed80302e, + 0x3481db31cee03547, + 0xacc3f67cdaa1d210, + 0x65cb771d8c7f96cc, + 0x8eb27177055723dd, + 0xc789950d44cd94be, + 0x934feadc3700b12b, + 0x5e485f11edbdf182, + 0x1e2e2a46fd64767a, + 0x2969ca71d82efa7c, + 0x9d46e9935ebbba2e, + 0xe056b67e05e6822b, + 0x94d73f55739d03a0, + 0xcd7010bdb69b5a03, + 0x455ef9fcd79b82f4, + 0x869cb54a8749c161, + 0x38d1a4fa6185d225, + 0xb475166f94bbe9bb, + 0xa4143548720959f1, + 0x7aed4780ba6b26ba, + 0xd0ce264439e02312, + 0x84366d746078d508, + 0xa8ce973c72ed17be, + 0x21c323a29a430b01, + 0x9962d617e3af80ee, + 0xab0ce91d9c8cf75b, + 0x530e8ee6d19a4dbc, + 0x2ef68c0cf53f5d72, + 0xc03a681640a85506, + 0x496e4e9f9c310967, + 0x78580472b59b14a0, + 0x273824c23b388577, + 0x66bf923ad45cb553, + 0x47ae1a5a2492ba86, + 0x35e304569e229659, + 0x4765182a46870b6f, + 0x6cbab625e9099412, + 0xddac9a2e598522c1, + 0x7172086e666624f2, + 0xdf5003ca503b7837, + 0x88c0c1db78563d09, + 0x58d51865acfc289d, + 0x177671aec65224f1, + 0xfb79d8a241e967d7, + 0x2be1e101cad9a49a, + 0x6625682f6e29186b, + 0x399553457ac06e50, + 0x35dffb4c23abb74, + 0x429db2591f54aade, + 0xc52802a8037d1009, + 0x6acb27381f0b25f3, + 0xf45e2551ee4f823b, + 0x8b0ea2d99580c2f7, + 0x3bed519cbcb4e1e1, + 0xff452823dbb010a, + 0x9d42ed614f3dd267, + 0x5b9313c06257c57b, + 0xa114b8008b5e1442, + 0xc1fe311c11c13d4b, + 0x66e8763ea34c5568, + 0x8b982af1c262f05d, + 0xee8876faaa75fbb7, + 0x8a62a4d0d172bb2a, + 0xc13d94a3b7449a97, + 0x6dbbba9dc15d037c, + 0xc786101f1d92e0f1, + 0xd78681a907a0b79b, + 0xf61aaf2962c9abb9, + 0x2cfd16fcd3cb7ad9, + 0x868c5b6744624d21, + 0x25e650899c74ddd7, + 0xba042af4a7c37463, + 0x4eb1a539465a3eca, + 0xbe09dbf03b05d5ca, + 0x774e5a362b5472ba, + 0x47a1221229d183cd, + 0x504b0ca18ef5a2df, + 0xdffbdfbde2456eb9, + 0x46cd2b2fbee34634, + 0xf2aef8fe819d98c3, + 0x357f5276d4599d61, + 0x24a5483879c453e3, + 0x88026889192b4b9, + 0x28da96671782dbec, + 0x4ef37c40588e9aaa, + 0x8837b90651bc9fb3, + 0xc164f741d3f0e5d6, + 0xbc135a0a704b70ba, + 0x69cd868f7622ada, + 0xbc37ba89e0b9c0ab, + 0x47c14a01323552f6, + 0x4f00794bacee98bb, + 0x7107de7d637a69d5, + 0x88af793bb6f2255e, + 0xf3c6466b8799b598, + 0xc288c616aa7f3b59, + 0x81ca63cf42fca3fd, + 0x88d85ace36a2674b, + 0xd056bd3792389e7, + 0xe55c396c4e9dd32d, + 0xbefb504571e6c0a6, + 0x96ab32115e91e8cc, + 0xbf8acb18de8f38d1, + 0x66dae58801672606, + 0x833b6017872317fb, + 0xb87c16f2d1c92864, + 0xdb766a74e58b669c, + 0x89659f85c61417be, + 0xc8daad856011ea0c, + 0x76a4b565b6fe7eae, + 0xa469d085f6237312, + 0xaaf0365683a3e96c, + 0x4dbb746f8424f7b8, + 0x638755af4e4acc1, + 0x3d7807f5bde64486, + 0x17be6d8f5bbb7639, + 0x903f0cd44dc35dc, + 0x67b672eafdf1196c, + 0xa676ff93ed4c82f1, + 0x521d1004c5053d9d, + 0x37ba9ad09ccc9202, + 0x84e54d297aacfb51, + 0xa0b4b776a143445, + 0x820d471e20b348e, + 0x1874383cb83d46dc, + 0x97edeec7a1efe11c, + 0xb330e50b1bdc42aa, + 0x1dd91955ce70e032, + 0xa514cdb88f2939d5, + 0x2791233fd90db9d3, + 0x7b670a4cc50f7a9b, + 0x77c07d2a05c6dfa5, + 0xe3778b6646d0a6fa, + 0xb39c8eda47b56749, + 0x933ed448addbef28, + 0xaf846af6ab7d0bf4, + 0xe5af208eb666e49, + 0x5e6622f73534cd6a, + 0x297daeca42ef5b6e, + 0x862daef3d35539a6, + 0xe68722498f8e1ea9, + 0x981c53093dc0d572, + 0xfa09b0bfbf86fbf5, + 0x30b1e96166219f15, + 0x70e7d466bdc4fb83, + 0x5a66736e35f2a8e9, + 0xcddb59d2b7c1baef, + 0xd6c7d247d26d8996, + 0xea4e39eac8de1ba3, + 0x539c8bb19fa3aff2, + 0x9f90e4c5fd508d8, + 0xa34e5956fbaf3385, + 0x2e2f8e151d3ef375, + 0x173691e9b83faec1, + 0xb85a8d56bf016379, + 0x8382381267408ae3, + 0xb90f901bbdc0096d, + 0x7c6ad32933bcec65, + 0x76bb5e2f2c8ad595, + 0x390f851a6cf46d28, + 0xc3e6064da1c2da72, + 0xc52a0c101cfa5389, + 0xd78eaf84a3fbc530, + 0x3781b9e2288b997e, + 0x73c2f6dea83d05c4, + 0x4228e364c5b5ed7, + 0x9d7a3edf0da43911, + 0x8edcfeda24686756, + 0x5e7667a7b7a9b3a1, + 0x4c4f389fa143791d, + 0xb08bc1023da7cddc, + 0x7ab4be3ae529b1cc, + 0x754e6132dbe74ff9, + 0x71635442a839df45, + 0x2f6fb1643fbe52de, + 0x961e0a42cf7a8177, + 0xf3b45d83d89ef2ea, + 0xee3de4cf4a6e3e9b, + 0xcd6848542c3295e7, + 0xe4cee1664c78662f, + 0x9947548b474c68c4, + 0x25d73777a5ed8b0b, + 0xc915b1d636b7fc, + 0x21c2ba75d9b0d2da, + 0x5f6b5dcf608a64a1, + 0xdcf333255ff9570c, + 0x633b922418ced4ee, + 0xc136dde0b004b34a, + 0x58cc83b05d4b2f5a, + 0x5eb424dda28e42d2, + 0x62df47369739cd98, + 0xb4e0b42485e4ce17, + 0x16e1f0c1f9a8d1e7, + 0x8ec3916707560ebf, + 0x62ba6e2df2cc9db3, + 0xcbf9f4ff77d83a16, + 0x78d9d7d07d2bbcc4, + 0xef554ce1e02c41f4, + 0x8d7581127eccf94d, + 0xa9b53336cb3c8a05, + 0x38c42c0bf45c4f91, + 0x640893cdf4488863, + 0x80ec34bc575ea568, + 0x39f324f5b48eaa40, + 0xe9d9ed1f8eff527f, + 0x9224fc058cc5a214, + 0xbaba00b04cfe7741, + 0x309a9f120fcf52af, + 0xa558f3ec65626212, + 0x424bec8b7adabe2f, + 0x41622513a6aea433, + 0xb88da2d5324ca798, + 0xd287733b245528a4, + 0x9a44697e6d68aec3, + 0x7b1093be2f49bb28, + 0x50bbec632e3d8aad, + 0x6cd90723e1ea8283, + 0x897b9e7431b02bf3, + 0x219efdcb338a7047, + 0x3b0311f0a27c0656, + 0xdb17bf91c0db96e7, + 0x8cd4fd6b4e85a5b2, + 0xfab071054ba6409d, + 0x40d6fe831fa9dfd9, + 0xaf358debad7d791e, + 0xeb8d0e25a65e3e58, + 0xbbcbd3df14e08580, + 0xcf751f27ecdab2b, + 0x2b4da14f2613d8f4, + }; + private static ulong* ZSTD_ldm_gearTab => + (ulong*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_ZSTD_ldm_gearTab) + ); +#else + + private static readonly ulong* ZSTD_ldm_gearTab = GetArrayPointer( + new ulong[256] + { + 0xf5b8f72c5f77775c, + 0x84935f266b7ac412, + 0xb647ada9ca730ccc, + 0xb065bb4b114fb1de, + 0x34584e7e8c3a9fd0, + 0x4e97e17c6ae26b05, + 0x3a03d743bc99a604, + 0xcecd042422c4044f, + 0x76de76c58524259e, + 0x9c8528f65badeaca, + 0x86563706e2097529, + 0x2902475fa375d889, + 0xafb32a9739a5ebe6, + 0xce2714da3883e639, + 0x21eaf821722e69e, + 0x37b628620b628, + 0x49a8d455d88caf5, + 0x8556d711e6958140, + 0x4f7ae74fc605c1f, + 0x829f0c3468bd3a20, + 0x4ffdc885c625179e, + 0x8473de048a3daf1b, + 0x51008822b05646b2, + 0x69d75d12b2d1cc5f, + 0x8c9d4a19159154bc, + 0xc3cc10f4abbd4003, + 0xd06ddc1cecb97391, + 0xbe48e6e7ed80302e, + 0x3481db31cee03547, + 0xacc3f67cdaa1d210, + 0x65cb771d8c7f96cc, + 0x8eb27177055723dd, + 0xc789950d44cd94be, + 0x934feadc3700b12b, + 0x5e485f11edbdf182, + 0x1e2e2a46fd64767a, + 0x2969ca71d82efa7c, + 0x9d46e9935ebbba2e, + 0xe056b67e05e6822b, + 0x94d73f55739d03a0, + 0xcd7010bdb69b5a03, + 0x455ef9fcd79b82f4, + 0x869cb54a8749c161, + 0x38d1a4fa6185d225, + 0xb475166f94bbe9bb, + 0xa4143548720959f1, + 0x7aed4780ba6b26ba, + 0xd0ce264439e02312, + 0x84366d746078d508, + 0xa8ce973c72ed17be, + 0x21c323a29a430b01, + 0x9962d617e3af80ee, + 0xab0ce91d9c8cf75b, + 0x530e8ee6d19a4dbc, + 0x2ef68c0cf53f5d72, + 0xc03a681640a85506, + 0x496e4e9f9c310967, + 0x78580472b59b14a0, + 0x273824c23b388577, + 0x66bf923ad45cb553, + 0x47ae1a5a2492ba86, + 0x35e304569e229659, + 0x4765182a46870b6f, + 0x6cbab625e9099412, + 0xddac9a2e598522c1, + 0x7172086e666624f2, + 0xdf5003ca503b7837, + 0x88c0c1db78563d09, + 0x58d51865acfc289d, + 0x177671aec65224f1, + 0xfb79d8a241e967d7, + 0x2be1e101cad9a49a, + 0x6625682f6e29186b, + 0x399553457ac06e50, + 0x35dffb4c23abb74, + 0x429db2591f54aade, + 0xc52802a8037d1009, + 0x6acb27381f0b25f3, + 0xf45e2551ee4f823b, + 0x8b0ea2d99580c2f7, + 0x3bed519cbcb4e1e1, + 0xff452823dbb010a, + 0x9d42ed614f3dd267, + 0x5b9313c06257c57b, + 0xa114b8008b5e1442, + 0xc1fe311c11c13d4b, + 0x66e8763ea34c5568, + 0x8b982af1c262f05d, + 0xee8876faaa75fbb7, + 0x8a62a4d0d172bb2a, + 0xc13d94a3b7449a97, + 0x6dbbba9dc15d037c, + 0xc786101f1d92e0f1, + 0xd78681a907a0b79b, + 0xf61aaf2962c9abb9, + 0x2cfd16fcd3cb7ad9, + 0x868c5b6744624d21, + 0x25e650899c74ddd7, + 0xba042af4a7c37463, + 0x4eb1a539465a3eca, + 0xbe09dbf03b05d5ca, + 0x774e5a362b5472ba, + 0x47a1221229d183cd, + 0x504b0ca18ef5a2df, + 0xdffbdfbde2456eb9, + 0x46cd2b2fbee34634, + 0xf2aef8fe819d98c3, + 0x357f5276d4599d61, + 0x24a5483879c453e3, + 0x88026889192b4b9, + 0x28da96671782dbec, + 0x4ef37c40588e9aaa, + 0x8837b90651bc9fb3, + 0xc164f741d3f0e5d6, + 0xbc135a0a704b70ba, + 0x69cd868f7622ada, + 0xbc37ba89e0b9c0ab, + 0x47c14a01323552f6, + 0x4f00794bacee98bb, + 0x7107de7d637a69d5, + 0x88af793bb6f2255e, + 0xf3c6466b8799b598, + 0xc288c616aa7f3b59, + 0x81ca63cf42fca3fd, + 0x88d85ace36a2674b, + 0xd056bd3792389e7, + 0xe55c396c4e9dd32d, + 0xbefb504571e6c0a6, + 0x96ab32115e91e8cc, + 0xbf8acb18de8f38d1, + 0x66dae58801672606, + 0x833b6017872317fb, + 0xb87c16f2d1c92864, + 0xdb766a74e58b669c, + 0x89659f85c61417be, + 0xc8daad856011ea0c, + 0x76a4b565b6fe7eae, + 0xa469d085f6237312, + 0xaaf0365683a3e96c, + 0x4dbb746f8424f7b8, + 0x638755af4e4acc1, + 0x3d7807f5bde64486, + 0x17be6d8f5bbb7639, + 0x903f0cd44dc35dc, + 0x67b672eafdf1196c, + 0xa676ff93ed4c82f1, + 0x521d1004c5053d9d, + 0x37ba9ad09ccc9202, + 0x84e54d297aacfb51, + 0xa0b4b776a143445, + 0x820d471e20b348e, + 0x1874383cb83d46dc, + 0x97edeec7a1efe11c, + 0xb330e50b1bdc42aa, + 0x1dd91955ce70e032, + 0xa514cdb88f2939d5, + 0x2791233fd90db9d3, + 0x7b670a4cc50f7a9b, + 0x77c07d2a05c6dfa5, + 0xe3778b6646d0a6fa, + 0xb39c8eda47b56749, + 0x933ed448addbef28, + 0xaf846af6ab7d0bf4, + 0xe5af208eb666e49, + 0x5e6622f73534cd6a, + 0x297daeca42ef5b6e, + 0x862daef3d35539a6, + 0xe68722498f8e1ea9, + 0x981c53093dc0d572, + 0xfa09b0bfbf86fbf5, + 0x30b1e96166219f15, + 0x70e7d466bdc4fb83, + 0x5a66736e35f2a8e9, + 0xcddb59d2b7c1baef, + 0xd6c7d247d26d8996, + 0xea4e39eac8de1ba3, + 0x539c8bb19fa3aff2, + 0x9f90e4c5fd508d8, + 0xa34e5956fbaf3385, + 0x2e2f8e151d3ef375, + 0x173691e9b83faec1, + 0xb85a8d56bf016379, + 0x8382381267408ae3, + 0xb90f901bbdc0096d, + 0x7c6ad32933bcec65, + 0x76bb5e2f2c8ad595, + 0x390f851a6cf46d28, + 0xc3e6064da1c2da72, + 0xc52a0c101cfa5389, + 0xd78eaf84a3fbc530, + 0x3781b9e2288b997e, + 0x73c2f6dea83d05c4, + 0x4228e364c5b5ed7, + 0x9d7a3edf0da43911, + 0x8edcfeda24686756, + 0x5e7667a7b7a9b3a1, + 0x4c4f389fa143791d, + 0xb08bc1023da7cddc, + 0x7ab4be3ae529b1cc, + 0x754e6132dbe74ff9, + 0x71635442a839df45, + 0x2f6fb1643fbe52de, + 0x961e0a42cf7a8177, + 0xf3b45d83d89ef2ea, + 0xee3de4cf4a6e3e9b, + 0xcd6848542c3295e7, + 0xe4cee1664c78662f, + 0x9947548b474c68c4, + 0x25d73777a5ed8b0b, + 0xc915b1d636b7fc, + 0x21c2ba75d9b0d2da, + 0x5f6b5dcf608a64a1, + 0xdcf333255ff9570c, + 0x633b922418ced4ee, + 0xc136dde0b004b34a, + 0x58cc83b05d4b2f5a, + 0x5eb424dda28e42d2, + 0x62df47369739cd98, + 0xb4e0b42485e4ce17, + 0x16e1f0c1f9a8d1e7, + 0x8ec3916707560ebf, + 0x62ba6e2df2cc9db3, + 0xcbf9f4ff77d83a16, + 0x78d9d7d07d2bbcc4, + 0xef554ce1e02c41f4, + 0x8d7581127eccf94d, + 0xa9b53336cb3c8a05, + 0x38c42c0bf45c4f91, + 0x640893cdf4488863, + 0x80ec34bc575ea568, + 0x39f324f5b48eaa40, + 0xe9d9ed1f8eff527f, + 0x9224fc058cc5a214, + 0xbaba00b04cfe7741, + 0x309a9f120fcf52af, + 0xa558f3ec65626212, + 0x424bec8b7adabe2f, + 0x41622513a6aea433, + 0xb88da2d5324ca798, + 0xd287733b245528a4, + 0x9a44697e6d68aec3, + 0x7b1093be2f49bb28, + 0x50bbec632e3d8aad, + 0x6cd90723e1ea8283, + 0x897b9e7431b02bf3, + 0x219efdcb338a7047, + 0x3b0311f0a27c0656, + 0xdb17bf91c0db96e7, + 0x8cd4fd6b4e85a5b2, + 0xfab071054ba6409d, + 0x40d6fe831fa9dfd9, + 0xaf358debad7d791e, + 0xeb8d0e25a65e3e58, + 0xbbcbd3df14e08580, + 0xcf751f27ecdab2b, + 0x2b4da14f2613d8f4, + } + ); +#endif +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdOpt.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdOpt.cs new file mode 100644 index 00000000..00066de5 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdOpt.cs @@ -0,0 +1,2326 @@ +using System; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + /* ZSTD_bitWeight() : + * provide estimated "cost" of a stat in full bits only */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_bitWeight(uint stat) + { + return ZSTD_highbit32(stat + 1) * (1 << 8); + } + + /* ZSTD_fracWeight() : + * provide fractional-bit "cost" of a stat, + * using linear interpolation approximation */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_fracWeight(uint rawStat) + { + uint stat = rawStat + 1; + uint hb = ZSTD_highbit32(stat); + uint BWeight = hb * (1 << 8); + /* Fweight was meant for "Fractional weight" + * but it's effectively a value between 1 and 2 + * using fixed point arithmetic */ + uint FWeight = stat << 8 >> (int)hb; + uint weight = BWeight + FWeight; + assert(hb + 8 < 31); + return weight; + } + + private static int ZSTD_compressedLiterals(optState_t* optPtr) + { + return optPtr->literalCompressionMode != ZSTD_paramSwitch_e.ZSTD_ps_disable ? 1 : 0; + } + + private static void ZSTD_setBasePrices(optState_t* optPtr, int optLevel) + { + if (ZSTD_compressedLiterals(optPtr) != 0) + { + optPtr->litSumBasePrice = + optLevel != 0 ? ZSTD_fracWeight(optPtr->litSum) : ZSTD_bitWeight(optPtr->litSum); + } + + optPtr->litLengthSumBasePrice = + optLevel != 0 + ? ZSTD_fracWeight(optPtr->litLengthSum) + : ZSTD_bitWeight(optPtr->litLengthSum); + optPtr->matchLengthSumBasePrice = + optLevel != 0 + ? ZSTD_fracWeight(optPtr->matchLengthSum) + : ZSTD_bitWeight(optPtr->matchLengthSum); + optPtr->offCodeSumBasePrice = + optLevel != 0 + ? ZSTD_fracWeight(optPtr->offCodeSum) + : ZSTD_bitWeight(optPtr->offCodeSum); + } + + private static uint sum_u32(uint* table, nuint nbElts) + { + nuint n; + uint total = 0; + for (n = 0; n < nbElts; n++) + { + total += table[n]; + } + + return total; + } + + private static uint ZSTD_downscaleStats( + uint* table, + uint lastEltIndex, + uint shift, + base_directive_e base1 + ) + { + uint s, + sum = 0; + assert(shift < 30); + for (s = 0; s < lastEltIndex + 1; s++) + { + uint @base = (uint)( + base1 != default ? 1 + : table[s] > 0 ? 1 + : 0 + ); + uint newStat = @base + (table[s] >> (int)shift); + sum += newStat; + table[s] = newStat; + } + + return sum; + } + + /* ZSTD_scaleStats() : + * reduce all elt frequencies in table if sum too large + * return the resulting sum of elements */ + private static uint ZSTD_scaleStats(uint* table, uint lastEltIndex, uint logTarget) + { + uint prevsum = sum_u32(table, lastEltIndex + 1); + uint factor = prevsum >> (int)logTarget; + assert(logTarget < 30); + if (factor <= 1) + { + return prevsum; + } + + return ZSTD_downscaleStats( + table, + lastEltIndex, + ZSTD_highbit32(factor), + base_directive_e.base_1guaranteed + ); + } + +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_baseLLfreqs => + new uint[36] + { + 4, + 2, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + }; + private static uint* baseLLfreqs => + (uint*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_baseLLfreqs) + ); +#else + + private static readonly uint* baseLLfreqs = GetArrayPointer( + new uint[36] + { + 4, + 2, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + } + ); +#endif +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_baseOFCfreqs => + new uint[32] + { + 6, + 2, + 1, + 1, + 2, + 3, + 4, + 4, + 4, + 3, + 2, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + }; + private static uint* baseOFCfreqs => + (uint*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_baseOFCfreqs) + ); +#else + + private static readonly uint* baseOFCfreqs = GetArrayPointer( + new uint[32] + { + 6, + 2, + 1, + 1, + 2, + 3, + 4, + 4, + 4, + 3, + 2, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, + } + ); +#endif + /* ZSTD_rescaleFreqs() : + * if first block (detected by optPtr->litLengthSum == 0) : init statistics + * take hints from dictionary if there is one + * and init from zero if there is none, + * using src for literals stats, and baseline stats for sequence symbols + * otherwise downscale existing stats, to be used as seed for next block. + */ + private static void ZSTD_rescaleFreqs( + optState_t* optPtr, + byte* src, + nuint srcSize, + int optLevel + ) + { + int compressedLiterals = ZSTD_compressedLiterals(optPtr); + optPtr->priceType = ZSTD_OptPrice_e.zop_dynamic; + if (optPtr->litLengthSum == 0) + { + if (srcSize <= 8) + { + optPtr->priceType = ZSTD_OptPrice_e.zop_predef; + } + + assert(optPtr->symbolCosts != null); + if (optPtr->symbolCosts->huf.repeatMode == HUF_repeat.HUF_repeat_valid) + { + optPtr->priceType = ZSTD_OptPrice_e.zop_dynamic; + if (compressedLiterals != 0) + { + /* generate literals statistics from huffman table */ + uint lit; + assert(optPtr->litFreq != null); + optPtr->litSum = 0; + for (lit = 0; lit <= (1 << 8) - 1; lit++) + { + /* scale to 2K */ + const uint scaleLog = 11; + uint bitCost = HUF_getNbBitsFromCTable( + &optPtr->symbolCosts->huf.CTable.e0, + lit + ); + assert(bitCost <= scaleLog); + optPtr->litFreq[lit] = (uint)( + bitCost != 0 ? 1 << (int)(scaleLog - bitCost) : 1 + ); + optPtr->litSum += optPtr->litFreq[lit]; + } + } + + { + uint ll; + FSE_CState_t llstate; + FSE_initCState(&llstate, optPtr->symbolCosts->fse.litlengthCTable); + optPtr->litLengthSum = 0; + for (ll = 0; ll <= 35; ll++) + { + /* scale to 1K */ + const uint scaleLog = 10; + uint bitCost = FSE_getMaxNbBits(llstate.symbolTT, ll); + assert(bitCost < scaleLog); + optPtr->litLengthFreq[ll] = (uint)( + bitCost != 0 ? 1 << (int)(scaleLog - bitCost) : 1 + ); + optPtr->litLengthSum += optPtr->litLengthFreq[ll]; + } + } + + { + uint ml; + FSE_CState_t mlstate; + FSE_initCState(&mlstate, optPtr->symbolCosts->fse.matchlengthCTable); + optPtr->matchLengthSum = 0; + for (ml = 0; ml <= 52; ml++) + { + const uint scaleLog = 10; + uint bitCost = FSE_getMaxNbBits(mlstate.symbolTT, ml); + assert(bitCost < scaleLog); + optPtr->matchLengthFreq[ml] = (uint)( + bitCost != 0 ? 1 << (int)(scaleLog - bitCost) : 1 + ); + optPtr->matchLengthSum += optPtr->matchLengthFreq[ml]; + } + } + + { + uint of; + FSE_CState_t ofstate; + FSE_initCState(&ofstate, optPtr->symbolCosts->fse.offcodeCTable); + optPtr->offCodeSum = 0; + for (of = 0; of <= 31; of++) + { + const uint scaleLog = 10; + uint bitCost = FSE_getMaxNbBits(ofstate.symbolTT, of); + assert(bitCost < scaleLog); + optPtr->offCodeFreq[of] = (uint)( + bitCost != 0 ? 1 << (int)(scaleLog - bitCost) : 1 + ); + optPtr->offCodeSum += optPtr->offCodeFreq[of]; + } + } + } + else + { + assert(optPtr->litFreq != null); + if (compressedLiterals != 0) + { + /* base initial cost of literals on direct frequency within src */ + uint lit = (1 << 8) - 1; + HIST_count_simple(optPtr->litFreq, &lit, src, srcSize); + optPtr->litSum = ZSTD_downscaleStats( + optPtr->litFreq, + (1 << 8) - 1, + 8, + base_directive_e.base_0possible + ); + } + + { + memcpy(optPtr->litLengthFreq, baseLLfreqs, sizeof(uint) * 36); + optPtr->litLengthSum = sum_u32(baseLLfreqs, 35 + 1); + } + + { + uint ml; + for (ml = 0; ml <= 52; ml++) + { + optPtr->matchLengthFreq[ml] = 1; + } + } + + optPtr->matchLengthSum = 52 + 1; + { + memcpy(optPtr->offCodeFreq, baseOFCfreqs, sizeof(uint) * 32); + optPtr->offCodeSum = sum_u32(baseOFCfreqs, 31 + 1); + } + } + } + else + { + if (compressedLiterals != 0) + { + optPtr->litSum = ZSTD_scaleStats(optPtr->litFreq, (1 << 8) - 1, 12); + } + + optPtr->litLengthSum = ZSTD_scaleStats(optPtr->litLengthFreq, 35, 11); + optPtr->matchLengthSum = ZSTD_scaleStats(optPtr->matchLengthFreq, 52, 11); + optPtr->offCodeSum = ZSTD_scaleStats(optPtr->offCodeFreq, 31, 11); + } + + ZSTD_setBasePrices(optPtr, optLevel); + } + + /* ZSTD_rawLiteralsCost() : + * price of literals (only) in specified segment (which length can be 0). + * does not include price of literalLength symbol */ + private static uint ZSTD_rawLiteralsCost( + byte* literals, + uint litLength, + optState_t* optPtr, + int optLevel + ) + { + if (litLength == 0) + { + return 0; + } + + if (ZSTD_compressedLiterals(optPtr) == 0) + { + return (litLength << 3) * (1 << 8); + } + + if (optPtr->priceType == ZSTD_OptPrice_e.zop_predef) + { + return litLength * 6 * (1 << 8); + } + + { + uint price = optPtr->litSumBasePrice * litLength; + uint litPriceMax = optPtr->litSumBasePrice - (1 << 8); + uint u; + assert(optPtr->litSumBasePrice >= 1 << 8); + for (u = 0; u < litLength; u++) + { + uint litPrice = + optLevel != 0 + ? ZSTD_fracWeight(optPtr->litFreq[literals[u]]) + : ZSTD_bitWeight(optPtr->litFreq[literals[u]]); + if (litPrice > litPriceMax) + { + litPrice = litPriceMax; + } + + price -= litPrice; + } + + return price; + } + } + + /* ZSTD_litLengthPrice() : + * cost of literalLength symbol */ + private static uint ZSTD_litLengthPrice(uint litLength, optState_t* optPtr, int optLevel) + { + assert(litLength <= 1 << 17); + if (optPtr->priceType == ZSTD_OptPrice_e.zop_predef) + { + return optLevel != 0 ? ZSTD_fracWeight(litLength) : ZSTD_bitWeight(litLength); + } + + if (litLength == 1 << 17) + { + return (1 << 8) + ZSTD_litLengthPrice((1 << 17) - 1, optPtr, optLevel); + } + + { + uint llCode = ZSTD_LLcode(litLength); + return (uint)(LL_bits[llCode] * (1 << 8)) + + optPtr->litLengthSumBasePrice + - ( + optLevel != 0 + ? ZSTD_fracWeight(optPtr->litLengthFreq[llCode]) + : ZSTD_bitWeight(optPtr->litLengthFreq[llCode]) + ); + } + } + + /* ZSTD_getMatchPrice() : + * Provides the cost of the match part (offset + matchLength) of a sequence. + * Must be combined with ZSTD_fullLiteralsCost() to get the full cost of a sequence. + * @offBase : sumtype, representing an offset or a repcode, and using numeric representation of ZSTD_storeSeq() + * @optLevel: when <2, favors small offset for decompression speed (improved cache efficiency) + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_getMatchPrice( + uint offBase, + uint matchLength, + optState_t* optPtr, + int optLevel + ) + { + uint price; + uint offCode = ZSTD_highbit32(offBase); + uint mlBase = matchLength - 3; + assert(matchLength >= 3); + if (optPtr->priceType == ZSTD_OptPrice_e.zop_predef) + { + return (optLevel != 0 ? ZSTD_fracWeight(mlBase) : ZSTD_bitWeight(mlBase)) + + (16 + offCode) * (1 << 8); + } + + price = + offCode * (1 << 8) + + ( + optPtr->offCodeSumBasePrice + - ( + optLevel != 0 + ? ZSTD_fracWeight(optPtr->offCodeFreq[offCode]) + : ZSTD_bitWeight(optPtr->offCodeFreq[offCode]) + ) + ); + if (optLevel < 2 && offCode >= 20) + { + price += (offCode - 19) * 2 * (1 << 8); + } + + { + uint mlCode = ZSTD_MLcode(mlBase); + price += + (uint)(ML_bits[mlCode] * (1 << 8)) + + ( + optPtr->matchLengthSumBasePrice + - ( + optLevel != 0 + ? ZSTD_fracWeight(optPtr->matchLengthFreq[mlCode]) + : ZSTD_bitWeight(optPtr->matchLengthFreq[mlCode]) + ) + ); + } + + price += (1 << 8) / 5; + return price; + } + + /* ZSTD_updateStats() : + * assumption : literals + litLength <= iend */ + private static void ZSTD_updateStats( + optState_t* optPtr, + uint litLength, + byte* literals, + uint offBase, + uint matchLength + ) + { + if (ZSTD_compressedLiterals(optPtr) != 0) + { + uint u; + for (u = 0; u < litLength; u++) + { + optPtr->litFreq[literals[u]] += 2; + } + + optPtr->litSum += litLength * 2; + } + + { + uint llCode = ZSTD_LLcode(litLength); + optPtr->litLengthFreq[llCode]++; + optPtr->litLengthSum++; + } + + { + uint offCode = ZSTD_highbit32(offBase); + assert(offCode <= 31); + optPtr->offCodeFreq[offCode]++; + optPtr->offCodeSum++; + } + + { + uint mlBase = matchLength - 3; + uint mlCode = ZSTD_MLcode(mlBase); + optPtr->matchLengthFreq[mlCode]++; + optPtr->matchLengthSum++; + } + } + + /* ZSTD_readMINMATCH() : + * function safe only for comparisons + * assumption : memPtr must be at least 4 bytes before end of buffer */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_readMINMATCH(void* memPtr, uint length) + { + switch (length) + { + default: + case 4: + return MEM_read32(memPtr); + case 3: + if (BitConverter.IsLittleEndian) + { + return MEM_read32(memPtr) << 8; + } + else + { + return MEM_read32(memPtr) >> 8; + } + } + } + + /* Update hashTable3 up to ip (excluded) + Assumption : always within prefix (i.e. not within extDict) */ + private static uint ZSTD_insertAndFindFirstIndexHash3( + ZSTD_MatchState_t* ms, + uint* nextToUpdate3, + byte* ip + ) + { + uint* hashTable3 = ms->hashTable3; + uint hashLog3 = ms->hashLog3; + byte* @base = ms->window.@base; + uint idx = *nextToUpdate3; + uint target = (uint)(ip - @base); + nuint hash3 = ZSTD_hash3Ptr(ip, hashLog3); + assert(hashLog3 > 0); + while (idx < target) + { + hashTable3[ZSTD_hash3Ptr(@base + idx, hashLog3)] = idx; + idx++; + } + + *nextToUpdate3 = target; + return hashTable3[hash3]; + } + + /*-************************************* + * Binary Tree search + ***************************************/ + /** ZSTD_insertBt1() : add one or multiple positions to tree. + * @param ip assumed <= iend-8 . + * @param target The target of ZSTD_updateTree_internal() - we are filling to this position + * @return : nb of positions added */ + private static uint ZSTD_insertBt1( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iend, + uint target, + uint mls, + int extDict + ) + { + ZSTD_compressionParameters* cParams = &ms->cParams; + uint* hashTable = ms->hashTable; + uint hashLog = cParams->hashLog; + nuint h = ZSTD_hashPtr(ip, hashLog, mls); + uint* bt = ms->chainTable; + uint btLog = cParams->chainLog - 1; + uint btMask = (uint)((1 << (int)btLog) - 1); + uint matchIndex = hashTable[h]; + nuint commonLengthSmaller = 0, + commonLengthLarger = 0; + byte* @base = ms->window.@base; + byte* dictBase = ms->window.dictBase; + uint dictLimit = ms->window.dictLimit; + byte* dictEnd = dictBase + dictLimit; + byte* prefixStart = @base + dictLimit; + byte* match; + uint curr = (uint)(ip - @base); + uint btLow = btMask >= curr ? 0 : curr - btMask; + uint* smallerPtr = bt + 2 * (curr & btMask); + uint* largerPtr = smallerPtr + 1; + /* to be nullified at the end */ + uint dummy32; + /* windowLow is based on target because + * we only need positions that will be in the window at the end of the tree update. + */ + uint windowLow = ZSTD_getLowestMatchIndex(ms, target, cParams->windowLog); + uint matchEndIdx = curr + 8 + 1; + nuint bestLength = 8; + uint nbCompares = 1U << (int)cParams->searchLog; + assert(curr <= target); + assert(ip <= iend - 8); + hashTable[h] = curr; + assert(windowLow > 0); + for (; nbCompares != 0 && matchIndex >= windowLow; --nbCompares) + { + uint* nextPtr = bt + 2 * (matchIndex & btMask); + /* guaranteed minimum nb of common bytes */ + nuint matchLength = + commonLengthSmaller < commonLengthLarger ? commonLengthSmaller : commonLengthLarger; + assert(matchIndex < curr); + if (extDict == 0 || matchIndex + matchLength >= dictLimit) + { + assert(matchIndex + matchLength >= dictLimit); + match = @base + matchIndex; + matchLength += ZSTD_count(ip + matchLength, match + matchLength, iend); + } + else + { + match = dictBase + matchIndex; + matchLength += ZSTD_count_2segments( + ip + matchLength, + match + matchLength, + iend, + dictEnd, + prefixStart + ); + if (matchIndex + matchLength >= dictLimit) + { + match = @base + matchIndex; + } + } + + if (matchLength > bestLength) + { + bestLength = matchLength; + if (matchLength > matchEndIdx - matchIndex) + { + matchEndIdx = matchIndex + (uint)matchLength; + } + } + + if (ip + matchLength == iend) + { + break; + } + + if (match[matchLength] < ip[matchLength]) + { + *smallerPtr = matchIndex; + commonLengthSmaller = matchLength; + if (matchIndex <= btLow) + { + smallerPtr = &dummy32; + break; + } + + smallerPtr = nextPtr + 1; + matchIndex = nextPtr[1]; + } + else + { + *largerPtr = matchIndex; + commonLengthLarger = matchLength; + if (matchIndex <= btLow) + { + largerPtr = &dummy32; + break; + } + + largerPtr = nextPtr; + matchIndex = nextPtr[0]; + } + } + + *smallerPtr = *largerPtr = 0; + { + uint positions = 0; + if (bestLength > 384) + { + positions = 192 < (uint)(bestLength - 384) ? 192 : (uint)(bestLength - 384); + } + + assert(matchEndIdx > curr + 8); + return positions > matchEndIdx - (curr + 8) ? positions : matchEndIdx - (curr + 8); + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void ZSTD_updateTree_internal( + ZSTD_MatchState_t* ms, + byte* ip, + byte* iend, + uint mls, + ZSTD_dictMode_e dictMode + ) + { + byte* @base = ms->window.@base; + uint target = (uint)(ip - @base); + uint idx = ms->nextToUpdate; + while (idx < target) + { + uint forward = ZSTD_insertBt1( + ms, + @base + idx, + iend, + target, + mls, + dictMode == ZSTD_dictMode_e.ZSTD_extDict ? 1 : 0 + ); + assert(idx < idx + forward); + idx += forward; + } + + assert((nuint)(ip - @base) <= unchecked((uint)-1)); + assert((nuint)(iend - @base) <= unchecked((uint)-1)); + ms->nextToUpdate = target; + } + + /* used in ZSTD_loadDictionaryContent() */ + private static void ZSTD_updateTree(ZSTD_MatchState_t* ms, byte* ip, byte* iend) + { + ZSTD_updateTree_internal(ms, ip, iend, ms->cParams.minMatch, ZSTD_dictMode_e.ZSTD_noDict); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_insertBtAndGetAllMatches( + ZSTD_match_t* matches, + ZSTD_MatchState_t* ms, + uint* nextToUpdate3, + byte* ip, + byte* iLimit, + ZSTD_dictMode_e dictMode, + uint* rep, + uint ll0, + uint lengthToBeat, + uint mls + ) + { + ZSTD_compressionParameters* cParams = &ms->cParams; + uint sufficient_len = + cParams->targetLength < (1 << 12) - 1 ? cParams->targetLength : (1 << 12) - 1; + byte* @base = ms->window.@base; + uint curr = (uint)(ip - @base); + uint hashLog = cParams->hashLog; + uint minMatch = (uint)(mls == 3 ? 3 : 4); + uint* hashTable = ms->hashTable; + nuint h = ZSTD_hashPtr(ip, hashLog, mls); + uint matchIndex = hashTable[h]; + uint* bt = ms->chainTable; + uint btLog = cParams->chainLog - 1; + uint btMask = (1U << (int)btLog) - 1; + nuint commonLengthSmaller = 0, + commonLengthLarger = 0; + byte* dictBase = ms->window.dictBase; + uint dictLimit = ms->window.dictLimit; + byte* dictEnd = dictBase + dictLimit; + byte* prefixStart = @base + dictLimit; + uint btLow = btMask >= curr ? 0 : curr - btMask; + uint windowLow = ZSTD_getLowestMatchIndex(ms, curr, cParams->windowLog); + uint matchLow = windowLow != 0 ? windowLow : 1; + uint* smallerPtr = bt + 2 * (curr & btMask); + uint* largerPtr = bt + 2 * (curr & btMask) + 1; + /* farthest referenced position of any match => detects repetitive patterns */ + uint matchEndIdx = curr + 8 + 1; + /* to be nullified at the end */ + uint dummy32; + uint mnum = 0; + uint nbCompares = 1U << (int)cParams->searchLog; + ZSTD_MatchState_t* dms = + dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState ? ms->dictMatchState : null; + ZSTD_compressionParameters* dmsCParams = + dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState ? &dms->cParams : null; + byte* dmsBase = dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState ? dms->window.@base : null; + byte* dmsEnd = dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState ? dms->window.nextSrc : null; + uint dmsHighLimit = + dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState ? (uint)(dmsEnd - dmsBase) : 0; + uint dmsLowLimit = + dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState ? dms->window.lowLimit : 0; + uint dmsIndexDelta = + dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState ? windowLow - dmsHighLimit : 0; + uint dmsHashLog = + dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState ? dmsCParams->hashLog : hashLog; + uint dmsBtLog = + dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState ? dmsCParams->chainLog - 1 : btLog; + uint dmsBtMask = + dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState ? (1U << (int)dmsBtLog) - 1 : 0; + uint dmsBtLow = + dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState + && dmsBtMask < dmsHighLimit - dmsLowLimit + ? dmsHighLimit - dmsBtMask + : dmsLowLimit; + nuint bestLength = lengthToBeat - 1; + assert(ll0 <= 1); + { + uint lastR = 3 + ll0; + uint repCode; + for (repCode = ll0; repCode < lastR; repCode++) + { + uint repOffset = repCode == 3 ? rep[0] - 1 : rep[repCode]; + uint repIndex = curr - repOffset; + uint repLen = 0; + assert(curr >= dictLimit); + if (repOffset - 1 < curr - dictLimit) + { + if ( + repIndex >= windowLow + && ZSTD_readMINMATCH(ip, minMatch) + == ZSTD_readMINMATCH(ip - repOffset, minMatch) + ) + { + repLen = + (uint)ZSTD_count(ip + minMatch, ip + minMatch - repOffset, iLimit) + + minMatch; + } + } + else + { + byte* repMatch = + dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState + ? dmsBase + repIndex - dmsIndexDelta + : dictBase + repIndex; + assert(curr >= windowLow); + if ( + dictMode == ZSTD_dictMode_e.ZSTD_extDict + && ( + (repOffset - 1 < curr - windowLow ? 1 : 0) + & ZSTD_index_overlap_check(dictLimit, repIndex) + ) != 0 + && ZSTD_readMINMATCH(ip, minMatch) == ZSTD_readMINMATCH(repMatch, minMatch) + ) + { + repLen = + (uint)ZSTD_count_2segments( + ip + minMatch, + repMatch + minMatch, + iLimit, + dictEnd, + prefixStart + ) + minMatch; + } + + if ( + dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState + && ( + (repOffset - 1 < curr - (dmsLowLimit + dmsIndexDelta) ? 1 : 0) + & ZSTD_index_overlap_check(dictLimit, repIndex) + ) != 0 + && ZSTD_readMINMATCH(ip, minMatch) == ZSTD_readMINMATCH(repMatch, minMatch) + ) + { + repLen = + (uint)ZSTD_count_2segments( + ip + minMatch, + repMatch + minMatch, + iLimit, + dmsEnd, + prefixStart + ) + minMatch; + } + } + + if (repLen > bestLength) + { + bestLength = repLen; + assert(repCode - ll0 + 1 >= 1); + assert(repCode - ll0 + 1 <= 3); + matches[mnum].off = repCode - ll0 + 1; + matches[mnum].len = repLen; + mnum++; + if (repLen > sufficient_len || ip + repLen == iLimit) + { + return mnum; + } + } + } + } + + if (mls == 3 && bestLength < mls) + { + uint matchIndex3 = ZSTD_insertAndFindFirstIndexHash3(ms, nextToUpdate3, ip); + if (matchIndex3 >= matchLow && curr - matchIndex3 < 1 << 18) + { + nuint mlen; + if ( + dictMode == ZSTD_dictMode_e.ZSTD_noDict + || dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState + || matchIndex3 >= dictLimit + ) + { + byte* match = @base + matchIndex3; + mlen = ZSTD_count(ip, match, iLimit); + } + else + { + byte* match = dictBase + matchIndex3; + mlen = ZSTD_count_2segments(ip, match, iLimit, dictEnd, prefixStart); + } + + if (mlen >= mls) + { + bestLength = mlen; + assert(curr > matchIndex3); + assert(mnum == 0); + assert(curr - matchIndex3 > 0); + matches[0].off = curr - matchIndex3 + 3; + matches[0].len = (uint)mlen; + mnum = 1; + if (mlen > sufficient_len || ip + mlen == iLimit) + { + ms->nextToUpdate = curr + 1; + return 1; + } + } + } + } + + hashTable[h] = curr; + for (; nbCompares != 0 && matchIndex >= matchLow; --nbCompares) + { + uint* nextPtr = bt + 2 * (matchIndex & btMask); + byte* match; + /* guaranteed minimum nb of common bytes */ + nuint matchLength = + commonLengthSmaller < commonLengthLarger ? commonLengthSmaller : commonLengthLarger; + assert(curr > matchIndex); + if ( + dictMode == ZSTD_dictMode_e.ZSTD_noDict + || dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState + || matchIndex + matchLength >= dictLimit + ) + { + assert(matchIndex + matchLength >= dictLimit); + match = @base + matchIndex; + matchLength += ZSTD_count(ip + matchLength, match + matchLength, iLimit); + } + else + { + match = dictBase + matchIndex; + assert(memcmp(match, ip, matchLength) == 0); + matchLength += ZSTD_count_2segments( + ip + matchLength, + match + matchLength, + iLimit, + dictEnd, + prefixStart + ); + if (matchIndex + matchLength >= dictLimit) + { + match = @base + matchIndex; + } + } + + if (matchLength > bestLength) + { + assert(matchEndIdx > matchIndex); + if (matchLength > matchEndIdx - matchIndex) + { + matchEndIdx = matchIndex + (uint)matchLength; + } + + bestLength = matchLength; + assert(curr - matchIndex > 0); + matches[mnum].off = curr - matchIndex + 3; + matches[mnum].len = (uint)matchLength; + mnum++; + if (matchLength > 1 << 12 || ip + matchLength == iLimit) + { + if (dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState) + { + nbCompares = 0; + } + + break; + } + } + + if (match[matchLength] < ip[matchLength]) + { + *smallerPtr = matchIndex; + commonLengthSmaller = matchLength; + if (matchIndex <= btLow) + { + smallerPtr = &dummy32; + break; + } + + smallerPtr = nextPtr + 1; + matchIndex = nextPtr[1]; + } + else + { + *largerPtr = matchIndex; + commonLengthLarger = matchLength; + if (matchIndex <= btLow) + { + largerPtr = &dummy32; + break; + } + + largerPtr = nextPtr; + matchIndex = nextPtr[0]; + } + } + + *smallerPtr = *largerPtr = 0; + assert(nbCompares <= 1U << (sizeof(nuint) == 4 ? 30 : 31) - 1); + if (dictMode == ZSTD_dictMode_e.ZSTD_dictMatchState && nbCompares != 0) + { + nuint dmsH = ZSTD_hashPtr(ip, dmsHashLog, mls); + uint dictMatchIndex = dms->hashTable[dmsH]; + uint* dmsBt = dms->chainTable; + commonLengthSmaller = commonLengthLarger = 0; + for (; nbCompares != 0 && dictMatchIndex > dmsLowLimit; --nbCompares) + { + uint* nextPtr = dmsBt + 2 * (dictMatchIndex & dmsBtMask); + /* guaranteed minimum nb of common bytes */ + nuint matchLength = + commonLengthSmaller < commonLengthLarger + ? commonLengthSmaller + : commonLengthLarger; + byte* match = dmsBase + dictMatchIndex; + matchLength += ZSTD_count_2segments( + ip + matchLength, + match + matchLength, + iLimit, + dmsEnd, + prefixStart + ); + if (dictMatchIndex + matchLength >= dmsHighLimit) + { + match = @base + dictMatchIndex + dmsIndexDelta; + } + + if (matchLength > bestLength) + { + matchIndex = dictMatchIndex + dmsIndexDelta; + if (matchLength > matchEndIdx - matchIndex) + { + matchEndIdx = matchIndex + (uint)matchLength; + } + + bestLength = matchLength; + assert(curr - matchIndex > 0); + matches[mnum].off = curr - matchIndex + 3; + matches[mnum].len = (uint)matchLength; + mnum++; + if (matchLength > 1 << 12 || ip + matchLength == iLimit) + { + break; + } + } + + if (dictMatchIndex <= dmsBtLow) + { + break; + } + + if (match[matchLength] < ip[matchLength]) + { + commonLengthSmaller = matchLength; + dictMatchIndex = nextPtr[1]; + } + else + { + commonLengthLarger = matchLength; + dictMatchIndex = nextPtr[0]; + } + } + } + + assert(matchEndIdx > curr + 8); + ms->nextToUpdate = matchEndIdx - 8; + return mnum; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint ZSTD_btGetAllMatches_internal( + ZSTD_match_t* matches, + ZSTD_MatchState_t* ms, + uint* nextToUpdate3, + byte* ip, + byte* iHighLimit, + uint* rep, + uint ll0, + uint lengthToBeat, + ZSTD_dictMode_e dictMode, + uint mls + ) + { + assert( + ( + ms->cParams.minMatch <= 3 ? 3 + : ms->cParams.minMatch <= 6 ? ms->cParams.minMatch + : 6 + ) == mls + ); + if (ip < ms->window.@base + ms->nextToUpdate) + { + return 0; + } + + ZSTD_updateTree_internal(ms, ip, iHighLimit, mls, dictMode); + return ZSTD_insertBtAndGetAllMatches( + matches, + ms, + nextToUpdate3, + ip, + iHighLimit, + dictMode, + rep, + ll0, + lengthToBeat, + mls + ); + } + + private static uint ZSTD_btGetAllMatches_noDict_3( + ZSTD_match_t* matches, + ZSTD_MatchState_t* ms, + uint* nextToUpdate3, + byte* ip, + byte* iHighLimit, + uint* rep, + uint ll0, + uint lengthToBeat + ) + { + return ZSTD_btGetAllMatches_internal( + matches, + ms, + nextToUpdate3, + ip, + iHighLimit, + rep, + ll0, + lengthToBeat, + ZSTD_dictMode_e.ZSTD_noDict, + 3 + ); + } + + private static uint ZSTD_btGetAllMatches_noDict_4( + ZSTD_match_t* matches, + ZSTD_MatchState_t* ms, + uint* nextToUpdate3, + byte* ip, + byte* iHighLimit, + uint* rep, + uint ll0, + uint lengthToBeat + ) + { + return ZSTD_btGetAllMatches_internal( + matches, + ms, + nextToUpdate3, + ip, + iHighLimit, + rep, + ll0, + lengthToBeat, + ZSTD_dictMode_e.ZSTD_noDict, + 4 + ); + } + + private static uint ZSTD_btGetAllMatches_noDict_5( + ZSTD_match_t* matches, + ZSTD_MatchState_t* ms, + uint* nextToUpdate3, + byte* ip, + byte* iHighLimit, + uint* rep, + uint ll0, + uint lengthToBeat + ) + { + return ZSTD_btGetAllMatches_internal( + matches, + ms, + nextToUpdate3, + ip, + iHighLimit, + rep, + ll0, + lengthToBeat, + ZSTD_dictMode_e.ZSTD_noDict, + 5 + ); + } + + private static uint ZSTD_btGetAllMatches_noDict_6( + ZSTD_match_t* matches, + ZSTD_MatchState_t* ms, + uint* nextToUpdate3, + byte* ip, + byte* iHighLimit, + uint* rep, + uint ll0, + uint lengthToBeat + ) + { + return ZSTD_btGetAllMatches_internal( + matches, + ms, + nextToUpdate3, + ip, + iHighLimit, + rep, + ll0, + lengthToBeat, + ZSTD_dictMode_e.ZSTD_noDict, + 6 + ); + } + + private static uint ZSTD_btGetAllMatches_extDict_3( + ZSTD_match_t* matches, + ZSTD_MatchState_t* ms, + uint* nextToUpdate3, + byte* ip, + byte* iHighLimit, + uint* rep, + uint ll0, + uint lengthToBeat + ) + { + return ZSTD_btGetAllMatches_internal( + matches, + ms, + nextToUpdate3, + ip, + iHighLimit, + rep, + ll0, + lengthToBeat, + ZSTD_dictMode_e.ZSTD_extDict, + 3 + ); + } + + private static uint ZSTD_btGetAllMatches_extDict_4( + ZSTD_match_t* matches, + ZSTD_MatchState_t* ms, + uint* nextToUpdate3, + byte* ip, + byte* iHighLimit, + uint* rep, + uint ll0, + uint lengthToBeat + ) + { + return ZSTD_btGetAllMatches_internal( + matches, + ms, + nextToUpdate3, + ip, + iHighLimit, + rep, + ll0, + lengthToBeat, + ZSTD_dictMode_e.ZSTD_extDict, + 4 + ); + } + + private static uint ZSTD_btGetAllMatches_extDict_5( + ZSTD_match_t* matches, + ZSTD_MatchState_t* ms, + uint* nextToUpdate3, + byte* ip, + byte* iHighLimit, + uint* rep, + uint ll0, + uint lengthToBeat + ) + { + return ZSTD_btGetAllMatches_internal( + matches, + ms, + nextToUpdate3, + ip, + iHighLimit, + rep, + ll0, + lengthToBeat, + ZSTD_dictMode_e.ZSTD_extDict, + 5 + ); + } + + private static uint ZSTD_btGetAllMatches_extDict_6( + ZSTD_match_t* matches, + ZSTD_MatchState_t* ms, + uint* nextToUpdate3, + byte* ip, + byte* iHighLimit, + uint* rep, + uint ll0, + uint lengthToBeat + ) + { + return ZSTD_btGetAllMatches_internal( + matches, + ms, + nextToUpdate3, + ip, + iHighLimit, + rep, + ll0, + lengthToBeat, + ZSTD_dictMode_e.ZSTD_extDict, + 6 + ); + } + + private static uint ZSTD_btGetAllMatches_dictMatchState_3( + ZSTD_match_t* matches, + ZSTD_MatchState_t* ms, + uint* nextToUpdate3, + byte* ip, + byte* iHighLimit, + uint* rep, + uint ll0, + uint lengthToBeat + ) + { + return ZSTD_btGetAllMatches_internal( + matches, + ms, + nextToUpdate3, + ip, + iHighLimit, + rep, + ll0, + lengthToBeat, + ZSTD_dictMode_e.ZSTD_dictMatchState, + 3 + ); + } + + private static uint ZSTD_btGetAllMatches_dictMatchState_4( + ZSTD_match_t* matches, + ZSTD_MatchState_t* ms, + uint* nextToUpdate3, + byte* ip, + byte* iHighLimit, + uint* rep, + uint ll0, + uint lengthToBeat + ) + { + return ZSTD_btGetAllMatches_internal( + matches, + ms, + nextToUpdate3, + ip, + iHighLimit, + rep, + ll0, + lengthToBeat, + ZSTD_dictMode_e.ZSTD_dictMatchState, + 4 + ); + } + + private static uint ZSTD_btGetAllMatches_dictMatchState_5( + ZSTD_match_t* matches, + ZSTD_MatchState_t* ms, + uint* nextToUpdate3, + byte* ip, + byte* iHighLimit, + uint* rep, + uint ll0, + uint lengthToBeat + ) + { + return ZSTD_btGetAllMatches_internal( + matches, + ms, + nextToUpdate3, + ip, + iHighLimit, + rep, + ll0, + lengthToBeat, + ZSTD_dictMode_e.ZSTD_dictMatchState, + 5 + ); + } + + private static uint ZSTD_btGetAllMatches_dictMatchState_6( + ZSTD_match_t* matches, + ZSTD_MatchState_t* ms, + uint* nextToUpdate3, + byte* ip, + byte* iHighLimit, + uint* rep, + uint ll0, + uint lengthToBeat + ) + { + return ZSTD_btGetAllMatches_internal( + matches, + ms, + nextToUpdate3, + ip, + iHighLimit, + rep, + ll0, + lengthToBeat, + ZSTD_dictMode_e.ZSTD_dictMatchState, + 6 + ); + } + + private static readonly ZSTD_getAllMatchesFn[][] getAllMatchesFns = new ZSTD_getAllMatchesFn[ + 3 + ][] + { + new ZSTD_getAllMatchesFn[4] + { + ZSTD_btGetAllMatches_noDict_3, + ZSTD_btGetAllMatches_noDict_4, + ZSTD_btGetAllMatches_noDict_5, + ZSTD_btGetAllMatches_noDict_6, + }, + new ZSTD_getAllMatchesFn[4] + { + ZSTD_btGetAllMatches_extDict_3, + ZSTD_btGetAllMatches_extDict_4, + ZSTD_btGetAllMatches_extDict_5, + ZSTD_btGetAllMatches_extDict_6, + }, + new ZSTD_getAllMatchesFn[4] + { + ZSTD_btGetAllMatches_dictMatchState_3, + ZSTD_btGetAllMatches_dictMatchState_4, + ZSTD_btGetAllMatches_dictMatchState_5, + ZSTD_btGetAllMatches_dictMatchState_6, + }, + }; + + private static ZSTD_getAllMatchesFn ZSTD_selectBtGetAllMatches( + ZSTD_MatchState_t* ms, + ZSTD_dictMode_e dictMode + ) + { + uint mls = + ms->cParams.minMatch <= 3 ? 3 + : ms->cParams.minMatch <= 6 ? ms->cParams.minMatch + : 6; + assert((uint)dictMode < 3); + assert(mls - 3 < 4); + return getAllMatchesFns[(int)dictMode][mls - 3]; + } + + /* ZSTD_optLdm_skipRawSeqStoreBytes(): + * Moves forward in @rawSeqStore by @nbBytes, + * which will update the fields 'pos' and 'posInSequence'. + */ + private static void ZSTD_optLdm_skipRawSeqStoreBytes(RawSeqStore_t* rawSeqStore, nuint nbBytes) + { + uint currPos = (uint)(rawSeqStore->posInSequence + nbBytes); + while (currPos != 0 && rawSeqStore->pos < rawSeqStore->size) + { + rawSeq currSeq = rawSeqStore->seq[rawSeqStore->pos]; + if (currPos >= currSeq.litLength + currSeq.matchLength) + { + currPos -= currSeq.litLength + currSeq.matchLength; + rawSeqStore->pos++; + } + else + { + rawSeqStore->posInSequence = currPos; + break; + } + } + + if (currPos == 0 || rawSeqStore->pos == rawSeqStore->size) + { + rawSeqStore->posInSequence = 0; + } + } + + /* ZSTD_opt_getNextMatchAndUpdateSeqStore(): + * Calculates the beginning and end of the next match in the current block. + * Updates 'pos' and 'posInSequence' of the ldmSeqStore. + */ + private static void ZSTD_opt_getNextMatchAndUpdateSeqStore( + ZSTD_optLdm_t* optLdm, + uint currPosInBlock, + uint blockBytesRemaining + ) + { + rawSeq currSeq; + uint currBlockEndPos; + uint literalsBytesRemaining; + uint matchBytesRemaining; + if (optLdm->seqStore.size == 0 || optLdm->seqStore.pos >= optLdm->seqStore.size) + { + optLdm->startPosInBlock = 0xffffffff; + optLdm->endPosInBlock = 0xffffffff; + return; + } + + currSeq = optLdm->seqStore.seq[optLdm->seqStore.pos]; + assert(optLdm->seqStore.posInSequence <= currSeq.litLength + currSeq.matchLength); + currBlockEndPos = currPosInBlock + blockBytesRemaining; + literalsBytesRemaining = + optLdm->seqStore.posInSequence < currSeq.litLength + ? currSeq.litLength - (uint)optLdm->seqStore.posInSequence + : 0; + matchBytesRemaining = + literalsBytesRemaining == 0 + ? currSeq.matchLength - ((uint)optLdm->seqStore.posInSequence - currSeq.litLength) + : currSeq.matchLength; + if (literalsBytesRemaining >= blockBytesRemaining) + { + optLdm->startPosInBlock = 0xffffffff; + optLdm->endPosInBlock = 0xffffffff; + ZSTD_optLdm_skipRawSeqStoreBytes(&optLdm->seqStore, blockBytesRemaining); + return; + } + + optLdm->startPosInBlock = currPosInBlock + literalsBytesRemaining; + optLdm->endPosInBlock = optLdm->startPosInBlock + matchBytesRemaining; + optLdm->offset = currSeq.offset; + if (optLdm->endPosInBlock > currBlockEndPos) + { + optLdm->endPosInBlock = currBlockEndPos; + ZSTD_optLdm_skipRawSeqStoreBytes(&optLdm->seqStore, currBlockEndPos - currPosInBlock); + } + else + { + ZSTD_optLdm_skipRawSeqStoreBytes( + &optLdm->seqStore, + literalsBytesRemaining + matchBytesRemaining + ); + } + } + + /* ZSTD_optLdm_maybeAddMatch(): + * Adds a match if it's long enough, + * based on it's 'matchStartPosInBlock' and 'matchEndPosInBlock', + * into 'matches'. Maintains the correct ordering of 'matches'. + */ + private static void ZSTD_optLdm_maybeAddMatch( + ZSTD_match_t* matches, + uint* nbMatches, + ZSTD_optLdm_t* optLdm, + uint currPosInBlock, + uint minMatch + ) + { + uint posDiff = currPosInBlock - optLdm->startPosInBlock; + /* Note: ZSTD_match_t actually contains offBase and matchLength (before subtracting MINMATCH) */ + uint candidateMatchLength = optLdm->endPosInBlock - optLdm->startPosInBlock - posDiff; + if ( + currPosInBlock < optLdm->startPosInBlock + || currPosInBlock >= optLdm->endPosInBlock + || candidateMatchLength < minMatch + ) + { + return; + } + + if ( + *nbMatches == 0 + || candidateMatchLength > matches[*nbMatches - 1].len && *nbMatches < 1 << 12 + ) + { + assert(optLdm->offset > 0); + uint candidateOffBase = optLdm->offset + 3; + matches[*nbMatches].len = candidateMatchLength; + matches[*nbMatches].off = candidateOffBase; + (*nbMatches)++; + } + } + + /* ZSTD_optLdm_processMatchCandidate(): + * Wrapper function to update ldm seq store and call ldm functions as necessary. + */ + private static void ZSTD_optLdm_processMatchCandidate( + ZSTD_optLdm_t* optLdm, + ZSTD_match_t* matches, + uint* nbMatches, + uint currPosInBlock, + uint remainingBytes, + uint minMatch + ) + { + if (optLdm->seqStore.size == 0 || optLdm->seqStore.pos >= optLdm->seqStore.size) + { + return; + } + + if (currPosInBlock >= optLdm->endPosInBlock) + { + if (currPosInBlock > optLdm->endPosInBlock) + { + /* The position at which ZSTD_optLdm_processMatchCandidate() is called is not necessarily + * at the end of a match from the ldm seq store, and will often be some bytes + * over beyond matchEndPosInBlock. As such, we need to correct for these "overshoots" + */ + uint posOvershoot = currPosInBlock - optLdm->endPosInBlock; + ZSTD_optLdm_skipRawSeqStoreBytes(&optLdm->seqStore, posOvershoot); + } + + ZSTD_opt_getNextMatchAndUpdateSeqStore(optLdm, currPosInBlock, remainingBytes); + } + + ZSTD_optLdm_maybeAddMatch(matches, nbMatches, optLdm, currPosInBlock, minMatch); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static nuint ZSTD_compressBlock_opt_generic( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize, + int optLevel, + ZSTD_dictMode_e dictMode + ) + { + optState_t* optStatePtr = &ms->opt; + byte* istart = (byte*)src; + byte* ip = istart; + byte* anchor = istart; + byte* iend = istart + srcSize; + byte* ilimit = iend - 8; + byte* @base = ms->window.@base; + byte* prefixStart = @base + ms->window.dictLimit; + ZSTD_compressionParameters* cParams = &ms->cParams; + ZSTD_getAllMatchesFn getAllMatches = ZSTD_selectBtGetAllMatches(ms, dictMode); + uint sufficient_len = + cParams->targetLength < (1 << 12) - 1 ? cParams->targetLength : (1 << 12) - 1; + uint minMatch = (uint)(cParams->minMatch == 3 ? 3 : 4); + uint nextToUpdate3 = ms->nextToUpdate; + ZSTD_optimal_t* opt = optStatePtr->priceTable; + ZSTD_match_t* matches = optStatePtr->matchTable; + ZSTD_optimal_t lastStretch; + ZSTD_optLdm_t optLdm; + lastStretch = new ZSTD_optimal_t(); + optLdm.seqStore = ms->ldmSeqStore != null ? *ms->ldmSeqStore : kNullRawSeqStore; + optLdm.endPosInBlock = optLdm.startPosInBlock = optLdm.offset = 0; + ZSTD_opt_getNextMatchAndUpdateSeqStore(&optLdm, (uint)(ip - istart), (uint)(iend - ip)); + assert(optLevel <= 2); + ZSTD_rescaleFreqs(optStatePtr, (byte*)src, srcSize, optLevel); + ip += ip == prefixStart ? 1 : 0; + while (ip < ilimit) + { + uint cur, + last_pos = 0; + { + uint litlen = (uint)(ip - anchor); + uint ll0 = litlen == 0 ? 1U : 0U; + uint nbMatches = getAllMatches( + matches, + ms, + &nextToUpdate3, + ip, + iend, + rep, + ll0, + minMatch + ); + ZSTD_optLdm_processMatchCandidate( + &optLdm, + matches, + &nbMatches, + (uint)(ip - istart), + (uint)(iend - ip), + minMatch + ); + if (nbMatches == 0) + { + ip++; + continue; + } + + opt[0].mlen = 0; + opt[0].litlen = litlen; + opt[0].price = (int)ZSTD_litLengthPrice(litlen, optStatePtr, optLevel); + memcpy(&opt[0].rep[0], rep, sizeof(uint) * 3); + { + uint maxML = matches[nbMatches - 1].len; + uint maxOffBase = matches[nbMatches - 1].off; + if (maxML > sufficient_len) + { + lastStretch.litlen = 0; + lastStretch.mlen = maxML; + lastStretch.off = maxOffBase; + cur = 0; + last_pos = maxML; + goto _shortestPath; + } + } + + assert(opt[0].price >= 0); + { + uint pos; + uint matchNb; + for (pos = 1; pos < minMatch; pos++) + { + opt[pos].price = 1 << 30; + opt[pos].mlen = 0; + opt[pos].litlen = litlen + pos; + } + + for (matchNb = 0; matchNb < nbMatches; matchNb++) + { + uint offBase = matches[matchNb].off; + uint end = matches[matchNb].len; + for (; pos <= end; pos++) + { + int matchPrice = (int)ZSTD_getMatchPrice( + offBase, + pos, + optStatePtr, + optLevel + ); + int sequencePrice = opt[0].price + matchPrice; + opt[pos].mlen = pos; + opt[pos].off = offBase; + opt[pos].litlen = 0; + opt[pos].price = + sequencePrice + (int)ZSTD_litLengthPrice(0, optStatePtr, optLevel); + } + } + + last_pos = pos - 1; + opt[pos].price = 1 << 30; + } + } + + for (cur = 1; cur <= last_pos; cur++) + { + byte* inr = ip + cur; + assert(cur <= 1 << 12); + { + uint litlen = opt[cur - 1].litlen + 1; + int price = + opt[cur - 1].price + + (int)ZSTD_rawLiteralsCost(ip + cur - 1, 1, optStatePtr, optLevel) + + ( + (int)ZSTD_litLengthPrice(litlen, optStatePtr, optLevel) + - (int)ZSTD_litLengthPrice(litlen - 1, optStatePtr, optLevel) + ); + assert(price < 1000000000); + if (price <= opt[cur].price) + { + ZSTD_optimal_t prevMatch = opt[cur]; + opt[cur] = opt[cur - 1]; + opt[cur].litlen = litlen; + opt[cur].price = price; + if ( + optLevel >= 1 + && prevMatch.litlen == 0 + && (int)ZSTD_litLengthPrice(1, optStatePtr, optLevel) + - (int)ZSTD_litLengthPrice(1 - 1, optStatePtr, optLevel) + < 0 + && ip + cur < iend + ) + { + /* check next position, in case it would be cheaper */ + int with1literal = + prevMatch.price + + (int)ZSTD_rawLiteralsCost(ip + cur, 1, optStatePtr, optLevel) + + ( + (int)ZSTD_litLengthPrice(1, optStatePtr, optLevel) + - (int)ZSTD_litLengthPrice(1 - 1, optStatePtr, optLevel) + ); + int withMoreLiterals = + price + + (int)ZSTD_rawLiteralsCost(ip + cur, 1, optStatePtr, optLevel) + + ( + (int)ZSTD_litLengthPrice(litlen + 1, optStatePtr, optLevel) + - (int)ZSTD_litLengthPrice( + litlen + 1 - 1, + optStatePtr, + optLevel + ) + ); + if ( + with1literal < withMoreLiterals + && with1literal < opt[cur + 1].price + ) + { + /* update offset history - before it disappears */ + uint prev = cur - prevMatch.mlen; + repcodes_s newReps = ZSTD_newRep( + opt[prev].rep, + prevMatch.off, + opt[prev].litlen == 0 ? 1U : 0U + ); + assert(cur >= prevMatch.mlen); + opt[cur + 1] = prevMatch; + memcpy(opt[cur + 1].rep, &newReps, (uint)sizeof(repcodes_s)); + opt[cur + 1].litlen = 1; + opt[cur + 1].price = with1literal; + if (last_pos < cur + 1) + { + last_pos = cur + 1; + } + } + } + } + } + + assert(cur >= opt[cur].mlen); + if (opt[cur].litlen == 0) + { + /* just finished a match => alter offset history */ + uint prev = cur - opt[cur].mlen; + repcodes_s newReps = ZSTD_newRep( + opt[prev].rep, + opt[cur].off, + opt[prev].litlen == 0 ? 1U : 0U + ); + memcpy(opt[cur].rep, &newReps, (uint)sizeof(repcodes_s)); + } + + if (inr > ilimit) + { + continue; + } + + if (cur == last_pos) + { + break; + } + + if (optLevel == 0 && opt[cur + 1].price <= opt[cur].price + (1 << 8) / 2) + { + continue; + } + + assert(opt[cur].price >= 0); + { + uint ll0 = opt[cur].litlen == 0 ? 1U : 0U; + int previousPrice = opt[cur].price; + int basePrice = + previousPrice + (int)ZSTD_litLengthPrice(0, optStatePtr, optLevel); + uint nbMatches = getAllMatches( + matches, + ms, + &nextToUpdate3, + inr, + iend, + opt[cur].rep, + ll0, + minMatch + ); + uint matchNb; + ZSTD_optLdm_processMatchCandidate( + &optLdm, + matches, + &nbMatches, + (uint)(inr - istart), + (uint)(iend - inr), + minMatch + ); + if (nbMatches == 0) + { + continue; + } + + { + uint longestML = matches[nbMatches - 1].len; + if ( + longestML > sufficient_len + || cur + longestML >= 1 << 12 + || ip + cur + longestML >= iend + ) + { + lastStretch.mlen = longestML; + lastStretch.off = matches[nbMatches - 1].off; + lastStretch.litlen = 0; + last_pos = cur + longestML; + goto _shortestPath; + } + } + + for (matchNb = 0; matchNb < nbMatches; matchNb++) + { + uint offset = matches[matchNb].off; + uint lastML = matches[matchNb].len; + uint startML = matchNb > 0 ? matches[matchNb - 1].len + 1 : minMatch; + uint mlen; + for (mlen = lastML; mlen >= startML; mlen--) + { + uint pos = cur + mlen; + int price = + basePrice + + (int)ZSTD_getMatchPrice(offset, mlen, optStatePtr, optLevel); + if (pos > last_pos || price < opt[pos].price) + { + while (last_pos < pos) + { + last_pos++; + opt[last_pos].price = 1 << 30; + opt[last_pos].litlen = 0 == 0 ? 1U : 0U; + } + + opt[pos].mlen = mlen; + opt[pos].off = offset; + opt[pos].litlen = 0; + opt[pos].price = price; + } + else + { + if (optLevel == 0) + { + break; + } + } + } + } + } + + opt[last_pos + 1].price = 1 << 30; + } + + lastStretch = opt[last_pos]; + assert(cur >= lastStretch.mlen); + cur = last_pos - lastStretch.mlen; + _shortestPath: + assert(opt[0].mlen == 0); + assert(last_pos >= lastStretch.mlen); + assert(cur == last_pos - lastStretch.mlen); + if (lastStretch.mlen == 0) + { + assert(lastStretch.litlen == (uint)(ip - anchor) + last_pos); + ip += last_pos; + continue; + } + + assert(lastStretch.off > 0); + if (lastStretch.litlen == 0) + { + /* finishing on a match : update offset history */ + repcodes_s reps = ZSTD_newRep( + opt[cur].rep, + lastStretch.off, + opt[cur].litlen == 0 ? 1U : 0U + ); + memcpy(rep, &reps, (uint)sizeof(repcodes_s)); + } + else + { + memcpy(rep, lastStretch.rep, (uint)sizeof(repcodes_s)); + assert(cur >= lastStretch.litlen); + cur -= lastStretch.litlen; + } + + { + uint storeEnd = cur + 2; + uint storeStart = storeEnd; + uint stretchPos = cur; + assert(storeEnd < (1 << 12) + 3); + if (lastStretch.litlen > 0) + { + opt[storeEnd].litlen = lastStretch.litlen; + opt[storeEnd].mlen = 0; + storeStart = storeEnd - 1; + opt[storeStart] = lastStretch; + } + + { + opt[storeEnd] = lastStretch; + storeStart = storeEnd; + } + + while (true) + { + ZSTD_optimal_t nextStretch = opt[stretchPos]; + opt[storeStart].litlen = nextStretch.litlen; + if (nextStretch.mlen == 0) + { + break; + } + + storeStart--; + opt[storeStart] = nextStretch; + assert(nextStretch.litlen + nextStretch.mlen <= stretchPos); + stretchPos -= nextStretch.litlen + nextStretch.mlen; + } + + { + uint storePos; + for (storePos = storeStart; storePos <= storeEnd; storePos++) + { + uint llen = opt[storePos].litlen; + uint mlen = opt[storePos].mlen; + uint offBase = opt[storePos].off; + uint advance = llen + mlen; + if (mlen == 0) + { + assert(storePos == storeEnd); + ip = anchor + llen; + continue; + } + + assert(anchor + llen <= iend); + ZSTD_updateStats(optStatePtr, llen, anchor, offBase, mlen); + ZSTD_storeSeq(seqStore, llen, anchor, iend, offBase, mlen); + anchor += advance; + ip = anchor; + } + } + + ZSTD_setBasePrices(optStatePtr, optLevel); + } + } + + return (nuint)(iend - anchor); + } + + private static nuint ZSTD_compressBlock_opt0( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize, + ZSTD_dictMode_e dictMode + ) + { + return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 0, dictMode); + } + + private static nuint ZSTD_compressBlock_opt2( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize, + ZSTD_dictMode_e dictMode + ) + { + return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 2, dictMode); + } + + private static nuint ZSTD_compressBlock_btopt( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_opt0( + ms, + seqStore, + rep, + src, + srcSize, + ZSTD_dictMode_e.ZSTD_noDict + ); + } + + /* ZSTD_initStats_ultra(): + * make a first compression pass, just to seed stats with more accurate starting values. + * only works on first block, with no dictionary and no ldm. + * this function cannot error out, its narrow contract must be respected. + */ + private static void ZSTD_initStats_ultra( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + /* updated rep codes will sink here */ + uint* tmpRep = stackalloc uint[3]; + memcpy(tmpRep, rep, sizeof(uint) * 3); + assert(ms->opt.litLengthSum == 0); + assert(seqStore->sequences == seqStore->sequencesStart); + assert(ms->window.dictLimit == ms->window.lowLimit); + assert(ms->window.dictLimit - ms->nextToUpdate <= 1); + ZSTD_compressBlock_opt2(ms, seqStore, tmpRep, src, srcSize, ZSTD_dictMode_e.ZSTD_noDict); + ZSTD_resetSeqStore(seqStore); + ms->window.@base -= srcSize; + ms->window.dictLimit += (uint)srcSize; + ms->window.lowLimit = ms->window.dictLimit; + ms->nextToUpdate = ms->window.dictLimit; + } + + private static nuint ZSTD_compressBlock_btultra( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_opt2( + ms, + seqStore, + rep, + src, + srcSize, + ZSTD_dictMode_e.ZSTD_noDict + ); + } + + /* note : no btultra2 variant for extDict nor dictMatchState, + * because btultra2 is not meant to work with dictionaries + * and is only specific for the first block (no prefix) */ + private static nuint ZSTD_compressBlock_btultra2( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + uint curr = (uint)((byte*)src - ms->window.@base); + assert(srcSize <= 1 << 17); + if ( + ms->opt.litLengthSum == 0 + && seqStore->sequences == seqStore->sequencesStart + && ms->window.dictLimit == ms->window.lowLimit + && curr == ms->window.dictLimit + && srcSize > 8 + ) + { + ZSTD_initStats_ultra(ms, seqStore, rep, src, srcSize); + } + + return ZSTD_compressBlock_opt2( + ms, + seqStore, + rep, + src, + srcSize, + ZSTD_dictMode_e.ZSTD_noDict + ); + } + + private static nuint ZSTD_compressBlock_btopt_dictMatchState( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_opt0( + ms, + seqStore, + rep, + src, + srcSize, + ZSTD_dictMode_e.ZSTD_dictMatchState + ); + } + + private static nuint ZSTD_compressBlock_btopt_extDict( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_opt0( + ms, + seqStore, + rep, + src, + srcSize, + ZSTD_dictMode_e.ZSTD_extDict + ); + } + + private static nuint ZSTD_compressBlock_btultra_dictMatchState( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_opt2( + ms, + seqStore, + rep, + src, + srcSize, + ZSTD_dictMode_e.ZSTD_dictMatchState + ); + } + + private static nuint ZSTD_compressBlock_btultra_extDict( + ZSTD_MatchState_t* ms, + SeqStore_t* seqStore, + uint* rep, + void* src, + nuint srcSize + ) + { + return ZSTD_compressBlock_opt2( + ms, + seqStore, + rep, + src, + srcSize, + ZSTD_dictMode_e.ZSTD_extDict + ); + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdPresplit.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdPresplit.cs new file mode 100644 index 00000000..0adf6c72 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdPresplit.cs @@ -0,0 +1,309 @@ +using System; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + /* for hashLog > 8, hash 2 bytes. + * for hashLog == 8, just take the byte, no hashing. + * The speed of this method relies on compile-time constant propagation */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static uint hash2(void* p, uint hashLog) + { + assert(hashLog >= 8); + if (hashLog == 8) + { + return ((byte*)p)[0]; + } + + assert(hashLog <= 10); + return MEM_read16(p) * 0x9e3779b9 >> (int)(32 - hashLog); + } + + private static void initStats(FPStats* fpstats) + { + *fpstats = new FPStats(); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void addEvents_generic( + Fingerprint* fp, + void* src, + nuint srcSize, + nuint samplingRate, + uint hashLog + ) + { + sbyte* p = (sbyte*)src; + nuint limit = srcSize - 2 + 1; + nuint n; + assert(srcSize >= 2); + for (n = 0; n < limit; n += samplingRate) + { + fp->events[hash2(p + n, hashLog)]++; + } + + fp->nbEvents += limit / samplingRate; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void recordFingerprint_generic( + Fingerprint* fp, + void* src, + nuint srcSize, + nuint samplingRate, + uint hashLog + ) + { + memset(fp, 0, (uint)(sizeof(uint) * ((nuint)1 << (int)hashLog))); + fp->nbEvents = 0; + addEvents_generic(fp, src, srcSize, samplingRate, hashLog); + } + + private static void ZSTD_recordFingerprint_1(Fingerprint* fp, void* src, nuint srcSize) + { + recordFingerprint_generic(fp, src, srcSize, 1, 10); + } + + private static void ZSTD_recordFingerprint_5(Fingerprint* fp, void* src, nuint srcSize) + { + recordFingerprint_generic(fp, src, srcSize, 5, 10); + } + + private static void ZSTD_recordFingerprint_11(Fingerprint* fp, void* src, nuint srcSize) + { + recordFingerprint_generic(fp, src, srcSize, 11, 9); + } + + private static void ZSTD_recordFingerprint_43(Fingerprint* fp, void* src, nuint srcSize) + { + recordFingerprint_generic(fp, src, srcSize, 43, 8); + } + + private static ulong abs64(long s64) + { + return (ulong)(s64 < 0 ? -s64 : s64); + } + + private static ulong fpDistance(Fingerprint* fp1, Fingerprint* fp2, uint hashLog) + { + ulong distance = 0; + nuint n; + assert(hashLog <= 10); + for (n = 0; n < (nuint)1 << (int)hashLog; n++) + { + distance += abs64( + fp1->events[n] * (long)fp2->nbEvents - fp2->events[n] * (long)fp1->nbEvents + ); + } + + return distance; + } + + /* Compare newEvents with pastEvents + * return 1 when considered "too different" + */ + private static int compareFingerprints( + Fingerprint* @ref, + Fingerprint* newfp, + int penalty, + uint hashLog + ) + { + assert(@ref->nbEvents > 0); + assert(newfp->nbEvents > 0); + { + ulong p50 = @ref->nbEvents * (ulong)newfp->nbEvents; + ulong deviation = fpDistance(@ref, newfp, hashLog); + ulong threshold = p50 * (ulong)(16 - 2 + penalty) / 16; + return deviation >= threshold ? 1 : 0; + } + } + + private static void mergeEvents(Fingerprint* acc, Fingerprint* newfp) + { + nuint n; + for (n = 0; n < 1 << 10; n++) + { + acc->events[n] += newfp->events[n]; + } + + acc->nbEvents += newfp->nbEvents; + } + + private static void flushEvents(FPStats* fpstats) + { + nuint n; + for (n = 0; n < 1 << 10; n++) + { + fpstats->pastEvents.events[n] = fpstats->newEvents.events[n]; + } + + fpstats->pastEvents.nbEvents = fpstats->newEvents.nbEvents; + fpstats->newEvents = new Fingerprint(); + } + + private static void removeEvents(Fingerprint* acc, Fingerprint* slice) + { + nuint n; + for (n = 0; n < 1 << 10; n++) + { + assert(acc->events[n] >= slice->events[n]); + acc->events[n] -= slice->events[n]; + } + + acc->nbEvents -= slice->nbEvents; + } + + private static readonly void*[] records_fs = new void*[4] + { + (delegate* managed)(&ZSTD_recordFingerprint_43), + (delegate* managed)(&ZSTD_recordFingerprint_11), + (delegate* managed)(&ZSTD_recordFingerprint_5), + (delegate* managed)(&ZSTD_recordFingerprint_1), + }; +#if NET7_0_OR_GREATER + private static ReadOnlySpan Span_hashParams => new uint[4] { 8, 9, 10, 10 }; + private static uint* hashParams => + (uint*) + System.Runtime.CompilerServices.Unsafe.AsPointer( + ref MemoryMarshal.GetReference(Span_hashParams) + ); +#else + + private static readonly uint* hashParams = GetArrayPointer(new uint[4] { 8, 9, 10, 10 }); +#endif + + private static nuint ZSTD_splitBlock_byChunks( + void* blockStart, + nuint blockSize, + int level, + void* workspace, + nuint wkspSize + ) + { + assert(0 <= level && level <= 3); + void* record_f = records_fs[level]; + FPStats* fpstats = (FPStats*)workspace; + sbyte* p = (sbyte*)blockStart; + int penalty = 3; + nuint pos = 0; + assert(blockSize == 128 << 10); + assert(workspace != null); + assert((nuint)workspace % (nuint)Math.Max(sizeof(uint), sizeof(ulong)) == 0); + assert(wkspSize >= (nuint)sizeof(FPStats)); + initStats(fpstats); + ((delegate* managed)record_f)( + &fpstats->pastEvents, + p, + 8 << 10 + ); + for (pos = 8 << 10; pos <= blockSize - (8 << 10); pos += 8 << 10) + { + ((delegate* managed)record_f)( + &fpstats->newEvents, + p + pos, + 8 << 10 + ); + if ( + compareFingerprints( + &fpstats->pastEvents, + &fpstats->newEvents, + penalty, + hashParams[level] + ) != 0 + ) + { + return pos; + } + else + { + mergeEvents(&fpstats->pastEvents, &fpstats->newEvents); + if (penalty > 0) + { + penalty--; + } + } + } + + assert(pos == blockSize); + return blockSize; + } + + /* ZSTD_splitBlock_fromBorders(): very fast strategy : + * compare fingerprint from beginning and end of the block, + * derive from their difference if it's preferable to split in the middle, + * repeat the process a second time, for finer grained decision. + * 3 times did not brought improvements, so I stopped at 2. + * Benefits are good enough for a cheap heuristic. + * More accurate splitting saves more, but speed impact is also more perceptible. + * For better accuracy, use more elaborate variant *_byChunks. + */ + private static nuint ZSTD_splitBlock_fromBorders( + void* blockStart, + nuint blockSize, + void* workspace, + nuint wkspSize + ) + { + FPStats* fpstats = (FPStats*)workspace; + Fingerprint* middleEvents = (Fingerprint*)(void*)((sbyte*)workspace + 512 * sizeof(uint)); + assert(blockSize == 128 << 10); + assert(workspace != null); + assert((nuint)workspace % (nuint)Math.Max(sizeof(uint), sizeof(ulong)) == 0); + assert(wkspSize >= (nuint)sizeof(FPStats)); + initStats(fpstats); + HIST_add(fpstats->pastEvents.events, blockStart, 512); + HIST_add(fpstats->newEvents.events, (sbyte*)blockStart + blockSize - 512, 512); + fpstats->pastEvents.nbEvents = fpstats->newEvents.nbEvents = 512; + if (compareFingerprints(&fpstats->pastEvents, &fpstats->newEvents, 0, 8) == 0) + { + return blockSize; + } + + HIST_add(middleEvents->events, (sbyte*)blockStart + blockSize / 2 - 512 / 2, 512); + middleEvents->nbEvents = 512; + { + ulong distFromBegin = fpDistance(&fpstats->pastEvents, middleEvents, 8); + ulong distFromEnd = fpDistance(&fpstats->newEvents, middleEvents, 8); + const ulong minDistance = 512 * 512 / 3; + if (abs64((long)distFromBegin - (long)distFromEnd) < minDistance) + { + return 64 * (1 << 10); + } + + return (nuint)(distFromBegin > distFromEnd ? 32 * (1 << 10) : 96 * (1 << 10)); + } + } + + /* ZSTD_splitBlock(): + * @level must be a value between 0 and 4. + * higher levels spend more energy to detect block boundaries. + * @workspace must be aligned for size_t. + * @wkspSize must be at least >= ZSTD_SLIPBLOCK_WORKSPACESIZE + * note: + * For the time being, this function only accepts full 128 KB blocks. + * Therefore, @blockSize must be == 128 KB. + * While this could be extended to smaller sizes in the future, + * it is not yet clear if this would be useful. TBD. + */ + private static nuint ZSTD_splitBlock( + void* blockStart, + nuint blockSize, + int level, + void* workspace, + nuint wkspSize + ) + { + assert(0 <= level && level <= 4); + if (level == 0) + { + return ZSTD_splitBlock_fromBorders(blockStart, blockSize, workspace, wkspSize); + } + + return ZSTD_splitBlock_byChunks(blockStart, blockSize, level - 1, workspace, wkspSize); + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdmtCompress.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdmtCompress.cs new file mode 100644 index 00000000..04ca1cd4 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ZstdmtCompress.cs @@ -0,0 +1,2043 @@ +using System.Runtime.CompilerServices; +using static SharpCompress.Compressors.ZStandard.UnsafeHelper; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public static unsafe partial class Methods +{ + private static readonly buffer_s g_nullBuffer = new buffer_s(start: null, capacity: 0); + + private static void ZSTDMT_freeBufferPool(ZSTDMT_bufferPool_s* bufPool) + { + if (bufPool == null) + { + return; + } + + if (bufPool->buffers != null) + { + uint u; + for (u = 0; u < bufPool->totalBuffers; u++) + { + ZSTD_customFree(bufPool->buffers[u].start, bufPool->cMem); + } + + ZSTD_customFree(bufPool->buffers, bufPool->cMem); + } + + SynchronizationWrapper.Free(&bufPool->poolMutex); + ZSTD_customFree(bufPool, bufPool->cMem); + } + + private static ZSTDMT_bufferPool_s* ZSTDMT_createBufferPool( + uint maxNbBuffers, + ZSTD_customMem cMem + ) + { + ZSTDMT_bufferPool_s* bufPool = (ZSTDMT_bufferPool_s*)ZSTD_customCalloc( + (nuint)sizeof(ZSTDMT_bufferPool_s), + cMem + ); + if (bufPool == null) + { + return null; + } + + SynchronizationWrapper.Init(&bufPool->poolMutex); + bufPool->buffers = (buffer_s*)ZSTD_customCalloc( + maxNbBuffers * (uint)sizeof(buffer_s), + cMem + ); + if (bufPool->buffers == null) + { + ZSTDMT_freeBufferPool(bufPool); + return null; + } + + bufPool->bufferSize = 64 * (1 << 10); + bufPool->totalBuffers = maxNbBuffers; + bufPool->nbBuffers = 0; + bufPool->cMem = cMem; + return bufPool; + } + + /* only works at initialization, not during compression */ + private static nuint ZSTDMT_sizeof_bufferPool(ZSTDMT_bufferPool_s* bufPool) + { + nuint poolSize = (nuint)sizeof(ZSTDMT_bufferPool_s); + nuint arraySize = bufPool->totalBuffers * (uint)sizeof(buffer_s); + uint u; + nuint totalBufferSize = 0; + SynchronizationWrapper.Enter(&bufPool->poolMutex); + for (u = 0; u < bufPool->totalBuffers; u++) + { + totalBufferSize += bufPool->buffers[u].capacity; + } + + SynchronizationWrapper.Exit(&bufPool->poolMutex); + return poolSize + arraySize + totalBufferSize; + } + + /* ZSTDMT_setBufferSize() : + * all future buffers provided by this buffer pool will have _at least_ this size + * note : it's better for all buffers to have same size, + * as they become freely interchangeable, reducing malloc/free usages and memory fragmentation */ + private static void ZSTDMT_setBufferSize(ZSTDMT_bufferPool_s* bufPool, nuint bSize) + { + SynchronizationWrapper.Enter(&bufPool->poolMutex); + bufPool->bufferSize = bSize; + SynchronizationWrapper.Exit(&bufPool->poolMutex); + } + + private static ZSTDMT_bufferPool_s* ZSTDMT_expandBufferPool( + ZSTDMT_bufferPool_s* srcBufPool, + uint maxNbBuffers + ) + { + if (srcBufPool == null) + { + return null; + } + + if (srcBufPool->totalBuffers >= maxNbBuffers) + { + return srcBufPool; + } + + { + ZSTD_customMem cMem = srcBufPool->cMem; + /* forward parameters */ + nuint bSize = srcBufPool->bufferSize; + ZSTDMT_bufferPool_s* newBufPool; + ZSTDMT_freeBufferPool(srcBufPool); + newBufPool = ZSTDMT_createBufferPool(maxNbBuffers, cMem); + if (newBufPool == null) + { + return newBufPool; + } + + ZSTDMT_setBufferSize(newBufPool, bSize); + return newBufPool; + } + } + + /** ZSTDMT_getBuffer() : + * assumption : bufPool must be valid + * @return : a buffer, with start pointer and size + * note: allocation may fail, in this case, start==NULL and size==0 */ + private static buffer_s ZSTDMT_getBuffer(ZSTDMT_bufferPool_s* bufPool) + { + nuint bSize = bufPool->bufferSize; + SynchronizationWrapper.Enter(&bufPool->poolMutex); + if (bufPool->nbBuffers != 0) + { + buffer_s buf = bufPool->buffers[--bufPool->nbBuffers]; + nuint availBufferSize = buf.capacity; + bufPool->buffers[bufPool->nbBuffers] = g_nullBuffer; + if (availBufferSize >= bSize && availBufferSize >> 3 <= bSize) + { + SynchronizationWrapper.Exit(&bufPool->poolMutex); + return buf; + } + + ZSTD_customFree(buf.start, bufPool->cMem); + } + + SynchronizationWrapper.Exit(&bufPool->poolMutex); + { + buffer_s buffer; + void* start = ZSTD_customMalloc(bSize, bufPool->cMem); + buffer.start = start; + buffer.capacity = start == null ? 0 : bSize; + return buffer; + } + } + + /* store buffer for later re-use, up to pool capacity */ + private static void ZSTDMT_releaseBuffer(ZSTDMT_bufferPool_s* bufPool, buffer_s buf) + { + if (buf.start == null) + { + return; + } + + SynchronizationWrapper.Enter(&bufPool->poolMutex); + if (bufPool->nbBuffers < bufPool->totalBuffers) + { + bufPool->buffers[bufPool->nbBuffers++] = buf; + SynchronizationWrapper.Exit(&bufPool->poolMutex); + return; + } + + SynchronizationWrapper.Exit(&bufPool->poolMutex); + ZSTD_customFree(buf.start, bufPool->cMem); + } + + private static nuint ZSTDMT_sizeof_seqPool(ZSTDMT_bufferPool_s* seqPool) + { + return ZSTDMT_sizeof_bufferPool(seqPool); + } + + private static RawSeqStore_t bufferToSeq(buffer_s buffer) + { + RawSeqStore_t seq = kNullRawSeqStore; + seq.seq = (rawSeq*)buffer.start; + seq.capacity = buffer.capacity / (nuint)sizeof(rawSeq); + return seq; + } + + private static buffer_s seqToBuffer(RawSeqStore_t seq) + { + buffer_s buffer; + buffer.start = seq.seq; + buffer.capacity = seq.capacity * (nuint)sizeof(rawSeq); + return buffer; + } + + private static RawSeqStore_t ZSTDMT_getSeq(ZSTDMT_bufferPool_s* seqPool) + { + if (seqPool->bufferSize == 0) + { + return kNullRawSeqStore; + } + + return bufferToSeq(ZSTDMT_getBuffer(seqPool)); + } + + private static void ZSTDMT_releaseSeq(ZSTDMT_bufferPool_s* seqPool, RawSeqStore_t seq) + { + ZSTDMT_releaseBuffer(seqPool, seqToBuffer(seq)); + } + + private static void ZSTDMT_setNbSeq(ZSTDMT_bufferPool_s* seqPool, nuint nbSeq) + { + ZSTDMT_setBufferSize(seqPool, nbSeq * (nuint)sizeof(rawSeq)); + } + + private static ZSTDMT_bufferPool_s* ZSTDMT_createSeqPool(uint nbWorkers, ZSTD_customMem cMem) + { + ZSTDMT_bufferPool_s* seqPool = ZSTDMT_createBufferPool(nbWorkers, cMem); + if (seqPool == null) + { + return null; + } + + ZSTDMT_setNbSeq(seqPool, 0); + return seqPool; + } + + private static void ZSTDMT_freeSeqPool(ZSTDMT_bufferPool_s* seqPool) + { + ZSTDMT_freeBufferPool(seqPool); + } + + private static ZSTDMT_bufferPool_s* ZSTDMT_expandSeqPool( + ZSTDMT_bufferPool_s* pool, + uint nbWorkers + ) + { + return ZSTDMT_expandBufferPool(pool, nbWorkers); + } + + /* note : all CCtx borrowed from the pool must be reverted back to the pool _before_ freeing the pool */ + private static void ZSTDMT_freeCCtxPool(ZSTDMT_CCtxPool* pool) + { + if (pool == null) + { + return; + } + + SynchronizationWrapper.Free(&pool->poolMutex); + if (pool->cctxs != null) + { + int cid; + for (cid = 0; cid < pool->totalCCtx; cid++) + { + ZSTD_freeCCtx(pool->cctxs[cid]); + } + + ZSTD_customFree(pool->cctxs, pool->cMem); + } + + ZSTD_customFree(pool, pool->cMem); + } + + /* ZSTDMT_createCCtxPool() : + * implies nbWorkers >= 1 , checked by caller ZSTDMT_createCCtx() */ + private static ZSTDMT_CCtxPool* ZSTDMT_createCCtxPool(int nbWorkers, ZSTD_customMem cMem) + { + ZSTDMT_CCtxPool* cctxPool = (ZSTDMT_CCtxPool*)ZSTD_customCalloc( + (nuint)sizeof(ZSTDMT_CCtxPool), + cMem + ); + assert(nbWorkers > 0); + if (cctxPool == null) + { + return null; + } + + SynchronizationWrapper.Init(&cctxPool->poolMutex); + cctxPool->totalCCtx = nbWorkers; + cctxPool->cctxs = (ZSTD_CCtx_s**)ZSTD_customCalloc( + (nuint)(nbWorkers * sizeof(ZSTD_CCtx_s*)), + cMem + ); + if (cctxPool->cctxs == null) + { + ZSTDMT_freeCCtxPool(cctxPool); + return null; + } + + cctxPool->cMem = cMem; + cctxPool->cctxs[0] = ZSTD_createCCtx_advanced(cMem); + if (cctxPool->cctxs[0] == null) + { + ZSTDMT_freeCCtxPool(cctxPool); + return null; + } + + cctxPool->availCCtx = 1; + return cctxPool; + } + + private static ZSTDMT_CCtxPool* ZSTDMT_expandCCtxPool(ZSTDMT_CCtxPool* srcPool, int nbWorkers) + { + if (srcPool == null) + { + return null; + } + + if (nbWorkers <= srcPool->totalCCtx) + { + return srcPool; + } + + { + ZSTD_customMem cMem = srcPool->cMem; + ZSTDMT_freeCCtxPool(srcPool); + return ZSTDMT_createCCtxPool(nbWorkers, cMem); + } + } + + /* only works during initialization phase, not during compression */ + private static nuint ZSTDMT_sizeof_CCtxPool(ZSTDMT_CCtxPool* cctxPool) + { + SynchronizationWrapper.Enter(&cctxPool->poolMutex); + { + uint nbWorkers = (uint)cctxPool->totalCCtx; + nuint poolSize = (nuint)sizeof(ZSTDMT_CCtxPool); + nuint arraySize = (nuint)(cctxPool->totalCCtx * sizeof(ZSTD_CCtx_s*)); + nuint totalCCtxSize = 0; + uint u; + for (u = 0; u < nbWorkers; u++) + { + totalCCtxSize += ZSTD_sizeof_CCtx(cctxPool->cctxs[u]); + } + + SynchronizationWrapper.Exit(&cctxPool->poolMutex); + assert(nbWorkers > 0); + return poolSize + arraySize + totalCCtxSize; + } + } + + private static ZSTD_CCtx_s* ZSTDMT_getCCtx(ZSTDMT_CCtxPool* cctxPool) + { + SynchronizationWrapper.Enter(&cctxPool->poolMutex); + if (cctxPool->availCCtx != 0) + { + cctxPool->availCCtx--; + { + ZSTD_CCtx_s* cctx = cctxPool->cctxs[cctxPool->availCCtx]; + SynchronizationWrapper.Exit(&cctxPool->poolMutex); + return cctx; + } + } + + SynchronizationWrapper.Exit(&cctxPool->poolMutex); + return ZSTD_createCCtx_advanced(cctxPool->cMem); + } + + private static void ZSTDMT_releaseCCtx(ZSTDMT_CCtxPool* pool, ZSTD_CCtx_s* cctx) + { + if (cctx == null) + { + return; + } + + SynchronizationWrapper.Enter(&pool->poolMutex); + if (pool->availCCtx < pool->totalCCtx) + { + pool->cctxs[pool->availCCtx++] = cctx; + } + else + { + ZSTD_freeCCtx(cctx); + } + + SynchronizationWrapper.Exit(&pool->poolMutex); + } + + private static int ZSTDMT_serialState_reset( + SerialState* serialState, + ZSTDMT_bufferPool_s* seqPool, + ZSTD_CCtx_params_s @params, + nuint jobSize, + void* dict, + nuint dictSize, + ZSTD_dictContentType_e dictContentType + ) + { + if (@params.ldmParams.enableLdm == ZSTD_paramSwitch_e.ZSTD_ps_enable) + { + ZSTD_ldm_adjustParameters(&@params.ldmParams, &@params.cParams); + assert(@params.ldmParams.hashLog >= @params.ldmParams.bucketSizeLog); + assert(@params.ldmParams.hashRateLog < 32); + } + else + { + @params.ldmParams = new ldmParams_t(); + } + + serialState->nextJobID = 0; + if (@params.fParams.checksumFlag != 0) + { + ZSTD_XXH64_reset(&serialState->xxhState, 0); + } + + if (@params.ldmParams.enableLdm == ZSTD_paramSwitch_e.ZSTD_ps_enable) + { + ZSTD_customMem cMem = @params.customMem; + uint hashLog = @params.ldmParams.hashLog; + nuint hashSize = ((nuint)1 << (int)hashLog) * (nuint)sizeof(ldmEntry_t); + uint bucketLog = @params.ldmParams.hashLog - @params.ldmParams.bucketSizeLog; + uint prevBucketLog = + serialState->@params.ldmParams.hashLog + - serialState->@params.ldmParams.bucketSizeLog; + nuint numBuckets = (nuint)1 << (int)bucketLog; + ZSTDMT_setNbSeq(seqPool, ZSTD_ldm_getMaxNbSeq(@params.ldmParams, jobSize)); + ZSTD_window_init(&serialState->ldmState.window); + if ( + serialState->ldmState.hashTable == null + || serialState->@params.ldmParams.hashLog < hashLog + ) + { + ZSTD_customFree(serialState->ldmState.hashTable, cMem); + serialState->ldmState.hashTable = (ldmEntry_t*)ZSTD_customMalloc(hashSize, cMem); + } + + if (serialState->ldmState.bucketOffsets == null || prevBucketLog < bucketLog) + { + ZSTD_customFree(serialState->ldmState.bucketOffsets, cMem); + serialState->ldmState.bucketOffsets = (byte*)ZSTD_customMalloc(numBuckets, cMem); + } + + if ( + serialState->ldmState.hashTable == null + || serialState->ldmState.bucketOffsets == null + ) + { + return 1; + } + + memset(serialState->ldmState.hashTable, 0, (uint)hashSize); + memset(serialState->ldmState.bucketOffsets, 0, (uint)numBuckets); + serialState->ldmState.loadedDictEnd = 0; + if (dictSize > 0) + { + if (dictContentType == ZSTD_dictContentType_e.ZSTD_dct_rawContent) + { + byte* dictEnd = (byte*)dict + dictSize; + ZSTD_window_update(&serialState->ldmState.window, dict, dictSize, 0); + ZSTD_ldm_fillHashTable( + &serialState->ldmState, + (byte*)dict, + dictEnd, + &@params.ldmParams + ); + serialState->ldmState.loadedDictEnd = + @params.forceWindow != 0 + ? 0 + : (uint)(dictEnd - serialState->ldmState.window.@base); + } + } + + serialState->ldmWindow = serialState->ldmState.window; + } + + serialState->@params = @params; + serialState->@params.jobSize = (uint)jobSize; + return 0; + } + + private static int ZSTDMT_serialState_init(SerialState* serialState) + { + int initError = 0; + *serialState = new SerialState(); + SynchronizationWrapper.Init(&serialState->mutex); + initError |= 0; + initError |= 0; + SynchronizationWrapper.Init(&serialState->ldmWindowMutex); + initError |= 0; + initError |= 0; + return initError; + } + + private static void ZSTDMT_serialState_free(SerialState* serialState) + { + ZSTD_customMem cMem = serialState->@params.customMem; + SynchronizationWrapper.Free(&serialState->mutex); + SynchronizationWrapper.Free(&serialState->ldmWindowMutex); + ZSTD_customFree(serialState->ldmState.hashTable, cMem); + ZSTD_customFree(serialState->ldmState.bucketOffsets, cMem); + } + + private static void ZSTDMT_serialState_genSequences( + SerialState* serialState, + RawSeqStore_t* seqStore, + Range src, + uint jobID + ) + { + SynchronizationWrapper.Enter(&serialState->mutex); + while (serialState->nextJobID < jobID) + { + SynchronizationWrapper.Wait(&serialState->mutex); + } + + if (serialState->nextJobID == jobID) + { + if (serialState->@params.ldmParams.enableLdm == ZSTD_paramSwitch_e.ZSTD_ps_enable) + { + nuint error; + assert( + seqStore->seq != null + && seqStore->pos == 0 + && seqStore->size == 0 + && seqStore->capacity > 0 + ); + assert(src.size <= serialState->@params.jobSize); + ZSTD_window_update(&serialState->ldmState.window, src.start, src.size, 0); + error = ZSTD_ldm_generateSequences( + &serialState->ldmState, + seqStore, + &serialState->@params.ldmParams, + src.start, + src.size + ); + assert(!ERR_isError(error)); + SynchronizationWrapper.Enter(&serialState->ldmWindowMutex); + serialState->ldmWindow = serialState->ldmState.window; + SynchronizationWrapper.Pulse(&serialState->ldmWindowMutex); + SynchronizationWrapper.Exit(&serialState->ldmWindowMutex); + } + + if (serialState->@params.fParams.checksumFlag != 0 && src.size > 0) + { + ZSTD_XXH64_update(&serialState->xxhState, src.start, src.size); + } + } + + serialState->nextJobID++; + SynchronizationWrapper.PulseAll(&serialState->mutex); + SynchronizationWrapper.Exit(&serialState->mutex); + } + + private static void ZSTDMT_serialState_applySequences( + SerialState* serialState, + ZSTD_CCtx_s* jobCCtx, + RawSeqStore_t* seqStore + ) + { + if (seqStore->size > 0) + { + assert(serialState->@params.ldmParams.enableLdm == ZSTD_paramSwitch_e.ZSTD_ps_enable); + assert(jobCCtx != null); + ZSTD_referenceExternalSequences(jobCCtx, seqStore->seq, seqStore->size); + } + } + + private static void ZSTDMT_serialState_ensureFinished( + SerialState* serialState, + uint jobID, + nuint cSize + ) + { + SynchronizationWrapper.Enter(&serialState->mutex); + if (serialState->nextJobID <= jobID) + { + assert(ERR_isError(cSize)); + serialState->nextJobID = jobID + 1; + SynchronizationWrapper.PulseAll(&serialState->mutex); + SynchronizationWrapper.Enter(&serialState->ldmWindowMutex); + ZSTD_window_clear(&serialState->ldmWindow); + SynchronizationWrapper.Pulse(&serialState->ldmWindowMutex); + SynchronizationWrapper.Exit(&serialState->ldmWindowMutex); + } + + SynchronizationWrapper.Exit(&serialState->mutex); + } + + private static readonly Range kNullRange = new Range(start: null, size: 0); + + /* ZSTDMT_compressionJob() is a POOL_function type */ + private static void ZSTDMT_compressionJob(void* jobDescription) + { + ZSTDMT_jobDescription* job = (ZSTDMT_jobDescription*)jobDescription; + /* do not modify job->params ! copy it, modify the copy */ + ZSTD_CCtx_params_s jobParams = job->@params; + ZSTD_CCtx_s* cctx = ZSTDMT_getCCtx(job->cctxPool); + RawSeqStore_t rawSeqStore = ZSTDMT_getSeq(job->seqPool); + buffer_s dstBuff = job->dstBuff; + nuint lastCBlockSize = 0; + if (cctx == null) + { + SynchronizationWrapper.Enter(&job->job_mutex); + job->cSize = unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + SynchronizationWrapper.Exit(&job->job_mutex); + goto _endJob; + } + + if (dstBuff.start == null) + { + dstBuff = ZSTDMT_getBuffer(job->bufPool); + if (dstBuff.start == null) + { + SynchronizationWrapper.Enter(&job->job_mutex); + job->cSize = unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + SynchronizationWrapper.Exit(&job->job_mutex); + goto _endJob; + } + + job->dstBuff = dstBuff; + } + + if ( + jobParams.ldmParams.enableLdm == ZSTD_paramSwitch_e.ZSTD_ps_enable + && rawSeqStore.seq == null + ) + { + SynchronizationWrapper.Enter(&job->job_mutex); + job->cSize = unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + SynchronizationWrapper.Exit(&job->job_mutex); + goto _endJob; + } + + if (job->jobID != 0) + { + jobParams.fParams.checksumFlag = 0; + } + + jobParams.ldmParams.enableLdm = ZSTD_paramSwitch_e.ZSTD_ps_disable; + jobParams.nbWorkers = 0; + ZSTDMT_serialState_genSequences(job->serial, &rawSeqStore, job->src, job->jobID); + if (job->cdict != null) + { + nuint initError = ZSTD_compressBegin_advanced_internal( + cctx, + null, + 0, + ZSTD_dictContentType_e.ZSTD_dct_auto, + ZSTD_dictTableLoadMethod_e.ZSTD_dtlm_fast, + job->cdict, + &jobParams, + job->fullFrameSize + ); + assert(job->firstJob != 0); + if (ERR_isError(initError)) + { + SynchronizationWrapper.Enter(&job->job_mutex); + job->cSize = initError; + SynchronizationWrapper.Exit(&job->job_mutex); + goto _endJob; + } + } + else + { + ulong pledgedSrcSize = job->firstJob != 0 ? job->fullFrameSize : job->src.size; + { + nuint forceWindowError = ZSTD_CCtxParams_setParameter( + &jobParams, + ZSTD_cParameter.ZSTD_c_experimentalParam3, + job->firstJob == 0 ? 1 : 0 + ); + if (ERR_isError(forceWindowError)) + { + SynchronizationWrapper.Enter(&job->job_mutex); + job->cSize = forceWindowError; + SynchronizationWrapper.Exit(&job->job_mutex); + goto _endJob; + } + } + + if (job->firstJob == 0) + { + nuint err = ZSTD_CCtxParams_setParameter( + &jobParams, + ZSTD_cParameter.ZSTD_c_experimentalParam15, + 0 + ); + if (ERR_isError(err)) + { + SynchronizationWrapper.Enter(&job->job_mutex); + job->cSize = err; + SynchronizationWrapper.Exit(&job->job_mutex); + goto _endJob; + } + } + + { + nuint initError = ZSTD_compressBegin_advanced_internal( + cctx, + job->prefix.start, + job->prefix.size, + ZSTD_dictContentType_e.ZSTD_dct_rawContent, + ZSTD_dictTableLoadMethod_e.ZSTD_dtlm_fast, + null, + &jobParams, + pledgedSrcSize + ); + if (ERR_isError(initError)) + { + SynchronizationWrapper.Enter(&job->job_mutex); + job->cSize = initError; + SynchronizationWrapper.Exit(&job->job_mutex); + goto _endJob; + } + } + } + + ZSTDMT_serialState_applySequences(job->serial, cctx, &rawSeqStore); + if (job->firstJob == 0) + { + nuint hSize = ZSTD_compressContinue_public( + cctx, + dstBuff.start, + dstBuff.capacity, + job->src.start, + 0 + ); + if (ERR_isError(hSize)) + { + SynchronizationWrapper.Enter(&job->job_mutex); + job->cSize = hSize; + SynchronizationWrapper.Exit(&job->job_mutex); + goto _endJob; + } + + ZSTD_invalidateRepCodes(cctx); + } + + { + const nuint chunkSize = 4 * (1 << 17); + int nbChunks = (int)((job->src.size + (chunkSize - 1)) / chunkSize); + byte* ip = (byte*)job->src.start; + byte* ostart = (byte*)dstBuff.start; + byte* op = ostart; + byte* oend = op + dstBuff.capacity; + int chunkNb; + assert(job->cSize == 0); + for (chunkNb = 1; chunkNb < nbChunks; chunkNb++) + { + nuint cSize = ZSTD_compressContinue_public( + cctx, + op, + (nuint)(oend - op), + ip, + chunkSize + ); + if (ERR_isError(cSize)) + { + SynchronizationWrapper.Enter(&job->job_mutex); + job->cSize = cSize; + SynchronizationWrapper.Exit(&job->job_mutex); + goto _endJob; + } + + ip += chunkSize; + op += cSize; + assert(op < oend); + SynchronizationWrapper.Enter(&job->job_mutex); + job->cSize += cSize; + job->consumed = chunkSize * (nuint)chunkNb; + SynchronizationWrapper.Pulse(&job->job_mutex); + SynchronizationWrapper.Exit(&job->job_mutex); + } + + assert(chunkSize > 0); + assert((chunkSize & chunkSize - 1) == 0); + if (((uint)(nbChunks > 0 ? 1 : 0) | job->lastJob) != 0) + { + nuint lastBlockSize1 = job->src.size & chunkSize - 1; + nuint lastBlockSize = + lastBlockSize1 == 0 && job->src.size >= chunkSize ? chunkSize : lastBlockSize1; + nuint cSize = + job->lastJob != 0 + ? ZSTD_compressEnd_public(cctx, op, (nuint)(oend - op), ip, lastBlockSize) + : ZSTD_compressContinue_public( + cctx, + op, + (nuint)(oend - op), + ip, + lastBlockSize + ); + if (ERR_isError(cSize)) + { + SynchronizationWrapper.Enter(&job->job_mutex); + job->cSize = cSize; + SynchronizationWrapper.Exit(&job->job_mutex); + goto _endJob; + } + + lastCBlockSize = cSize; + } + } + + ZSTD_CCtx_trace(cctx, 0); + _endJob: + ZSTDMT_serialState_ensureFinished(job->serial, job->jobID, job->cSize); + ZSTDMT_releaseSeq(job->seqPool, rawSeqStore); + ZSTDMT_releaseCCtx(job->cctxPool, cctx); + SynchronizationWrapper.Enter(&job->job_mutex); + if (ERR_isError(job->cSize)) + { + assert(lastCBlockSize == 0); + } + + job->cSize += lastCBlockSize; + job->consumed = job->src.size; + SynchronizationWrapper.Pulse(&job->job_mutex); + SynchronizationWrapper.Exit(&job->job_mutex); + } + + private static readonly RoundBuff_t kNullRoundBuff = new RoundBuff_t( + buffer: null, + capacity: 0, + pos: 0 + ); + + private static void ZSTDMT_freeJobsTable( + ZSTDMT_jobDescription* jobTable, + uint nbJobs, + ZSTD_customMem cMem + ) + { + uint jobNb; + if (jobTable == null) + { + return; + } + + for (jobNb = 0; jobNb < nbJobs; jobNb++) + { + SynchronizationWrapper.Free(&jobTable[jobNb].job_mutex); + } + + ZSTD_customFree(jobTable, cMem); + } + + /* ZSTDMT_allocJobsTable() + * allocate and init a job table. + * update *nbJobsPtr to next power of 2 value, as size of table */ + private static ZSTDMT_jobDescription* ZSTDMT_createJobsTable( + uint* nbJobsPtr, + ZSTD_customMem cMem + ) + { + uint nbJobsLog2 = ZSTD_highbit32(*nbJobsPtr) + 1; + uint nbJobs = (uint)(1 << (int)nbJobsLog2); + uint jobNb; + ZSTDMT_jobDescription* jobTable = (ZSTDMT_jobDescription*)ZSTD_customCalloc( + nbJobs * (uint)sizeof(ZSTDMT_jobDescription), + cMem + ); + int initError = 0; + if (jobTable == null) + { + return null; + } + + *nbJobsPtr = nbJobs; + for (jobNb = 0; jobNb < nbJobs; jobNb++) + { + SynchronizationWrapper.Init(&jobTable[jobNb].job_mutex); + initError |= 0; + initError |= 0; + } + + if (initError != 0) + { + ZSTDMT_freeJobsTable(jobTable, nbJobs, cMem); + return null; + } + + return jobTable; + } + + private static nuint ZSTDMT_expandJobsTable(ZSTDMT_CCtx_s* mtctx, uint nbWorkers) + { + uint nbJobs = nbWorkers + 2; + if (nbJobs > mtctx->jobIDMask + 1) + { + ZSTDMT_freeJobsTable(mtctx->jobs, mtctx->jobIDMask + 1, mtctx->cMem); + mtctx->jobIDMask = 0; + mtctx->jobs = ZSTDMT_createJobsTable(&nbJobs, mtctx->cMem); + if (mtctx->jobs == null) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + + assert(nbJobs != 0 && (nbJobs & nbJobs - 1) == 0); + mtctx->jobIDMask = nbJobs - 1; + } + + return 0; + } + + /* ZSTDMT_CCtxParam_setNbWorkers(): + * Internal use only */ + private static nuint ZSTDMT_CCtxParam_setNbWorkers(ZSTD_CCtx_params_s* @params, uint nbWorkers) + { + return ZSTD_CCtxParams_setParameter( + @params, + ZSTD_cParameter.ZSTD_c_nbWorkers, + (int)nbWorkers + ); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ZSTDMT_CCtx_s* ZSTDMT_createCCtx_advanced_internal( + uint nbWorkers, + ZSTD_customMem cMem, + void* pool + ) + { + ZSTDMT_CCtx_s* mtctx; + uint nbJobs = nbWorkers + 2; + int initError; + if (nbWorkers < 1) + { + return null; + } + + nbWorkers = + nbWorkers < (uint)(sizeof(void*) == 4 ? 64 : 256) + ? nbWorkers + : (uint)(sizeof(void*) == 4 ? 64 : 256); + if (((cMem.customAlloc != null ? 1 : 0) ^ (cMem.customFree != null ? 1 : 0)) != 0) + { + return null; + } + + mtctx = (ZSTDMT_CCtx_s*)ZSTD_customCalloc((nuint)sizeof(ZSTDMT_CCtx_s), cMem); + if (mtctx == null) + { + return null; + } + + ZSTDMT_CCtxParam_setNbWorkers(&mtctx->@params, nbWorkers); + mtctx->cMem = cMem; + mtctx->allJobsCompleted = 1; + if (pool != null) + { + mtctx->factory = pool; + mtctx->providedFactory = 1; + } + else + { + mtctx->factory = POOL_create_advanced(nbWorkers, 0, cMem); + mtctx->providedFactory = 0; + } + + mtctx->jobs = ZSTDMT_createJobsTable(&nbJobs, cMem); + assert(nbJobs > 0); + assert((nbJobs & nbJobs - 1) == 0); + mtctx->jobIDMask = nbJobs - 1; + mtctx->bufPool = ZSTDMT_createBufferPool(2 * nbWorkers + 3, cMem); + mtctx->cctxPool = ZSTDMT_createCCtxPool((int)nbWorkers, cMem); + mtctx->seqPool = ZSTDMT_createSeqPool(nbWorkers, cMem); + initError = ZSTDMT_serialState_init(&mtctx->serial); + mtctx->roundBuff = kNullRoundBuff; + if ( + ( + ( + mtctx->factory == null + || mtctx->jobs == null + || mtctx->bufPool == null + || mtctx->cctxPool == null + || mtctx->seqPool == null + ? 1 + : 0 + ) | initError + ) != 0 + ) + { + ZSTDMT_freeCCtx(mtctx); + return null; + } + + return mtctx; + } + + /* Requires ZSTD_MULTITHREAD to be defined during compilation, otherwise it will return NULL. */ + private static ZSTDMT_CCtx_s* ZSTDMT_createCCtx_advanced( + uint nbWorkers, + ZSTD_customMem cMem, + void* pool + ) + { + return ZSTDMT_createCCtx_advanced_internal(nbWorkers, cMem, pool); + } + + /* ZSTDMT_releaseAllJobResources() : + * note : ensure all workers are killed first ! */ + private static void ZSTDMT_releaseAllJobResources(ZSTDMT_CCtx_s* mtctx) + { + uint jobID; + for (jobID = 0; jobID <= mtctx->jobIDMask; jobID++) + { + /* Copy the mutex/cond out */ + void* mutex = mtctx->jobs[jobID].job_mutex; + void* cond = mtctx->jobs[jobID].job_cond; + ZSTDMT_releaseBuffer(mtctx->bufPool, mtctx->jobs[jobID].dstBuff); + mtctx->jobs[jobID] = new ZSTDMT_jobDescription { job_mutex = mutex, job_cond = cond }; + } + + mtctx->inBuff.buffer = g_nullBuffer; + mtctx->inBuff.filled = 0; + mtctx->allJobsCompleted = 1; + } + + private static void ZSTDMT_waitForAllJobsCompleted(ZSTDMT_CCtx_s* mtctx) + { + while (mtctx->doneJobID < mtctx->nextJobID) + { + uint jobID = mtctx->doneJobID & mtctx->jobIDMask; + SynchronizationWrapper.Enter(&mtctx->jobs[jobID].job_mutex); + while (mtctx->jobs[jobID].consumed < mtctx->jobs[jobID].src.size) + { + SynchronizationWrapper.Wait(&mtctx->jobs[jobID].job_mutex); + } + + SynchronizationWrapper.Exit(&mtctx->jobs[jobID].job_mutex); + mtctx->doneJobID++; + } + } + + private static nuint ZSTDMT_freeCCtx(ZSTDMT_CCtx_s* mtctx) + { + if (mtctx == null) + { + return 0; + } + + if (mtctx->providedFactory == 0) + { + POOL_free(mtctx->factory); + } + + ZSTDMT_releaseAllJobResources(mtctx); + ZSTDMT_freeJobsTable(mtctx->jobs, mtctx->jobIDMask + 1, mtctx->cMem); + ZSTDMT_freeBufferPool(mtctx->bufPool); + ZSTDMT_freeCCtxPool(mtctx->cctxPool); + ZSTDMT_freeSeqPool(mtctx->seqPool); + ZSTDMT_serialState_free(&mtctx->serial); + ZSTD_freeCDict(mtctx->cdictLocal); + if (mtctx->roundBuff.buffer != null) + { + ZSTD_customFree(mtctx->roundBuff.buffer, mtctx->cMem); + } + + ZSTD_customFree(mtctx, mtctx->cMem); + return 0; + } + + private static nuint ZSTDMT_sizeof_CCtx(ZSTDMT_CCtx_s* mtctx) + { + if (mtctx == null) + { + return 0; + } + + return (nuint)sizeof(ZSTDMT_CCtx_s) + + POOL_sizeof(mtctx->factory) + + ZSTDMT_sizeof_bufferPool(mtctx->bufPool) + + (mtctx->jobIDMask + 1) * (uint)sizeof(ZSTDMT_jobDescription) + + ZSTDMT_sizeof_CCtxPool(mtctx->cctxPool) + + ZSTDMT_sizeof_seqPool(mtctx->seqPool) + + ZSTD_sizeof_CDict(mtctx->cdictLocal) + + mtctx->roundBuff.capacity; + } + + /* ZSTDMT_resize() : + * @return : error code if fails, 0 on success */ + private static nuint ZSTDMT_resize(ZSTDMT_CCtx_s* mtctx, uint nbWorkers) + { + if (POOL_resize(mtctx->factory, nbWorkers) != 0) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + + { + nuint err_code = ZSTDMT_expandJobsTable(mtctx, nbWorkers); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + mtctx->bufPool = ZSTDMT_expandBufferPool(mtctx->bufPool, 2 * nbWorkers + 3); + if (mtctx->bufPool == null) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + + mtctx->cctxPool = ZSTDMT_expandCCtxPool(mtctx->cctxPool, (int)nbWorkers); + if (mtctx->cctxPool == null) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + + mtctx->seqPool = ZSTDMT_expandSeqPool(mtctx->seqPool, nbWorkers); + if (mtctx->seqPool == null) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + + ZSTDMT_CCtxParam_setNbWorkers(&mtctx->@params, nbWorkers); + return 0; + } + + /*! ZSTDMT_updateCParams_whileCompressing() : + * Updates a selected set of compression parameters, remaining compatible with currently active frame. + * New parameters will be applied to next compression job. */ + private static void ZSTDMT_updateCParams_whileCompressing( + ZSTDMT_CCtx_s* mtctx, + ZSTD_CCtx_params_s* cctxParams + ) + { + /* Do not modify windowLog while compressing */ + uint saved_wlog = mtctx->@params.cParams.windowLog; + int compressionLevel = cctxParams->compressionLevel; + mtctx->@params.compressionLevel = compressionLevel; + { + ZSTD_compressionParameters cParams = ZSTD_getCParamsFromCCtxParams( + cctxParams, + unchecked(0UL - 1), + 0, + ZSTD_CParamMode_e.ZSTD_cpm_noAttachDict + ); + cParams.windowLog = saved_wlog; + mtctx->@params.cParams = cParams; + } + } + + /* ZSTDMT_getFrameProgression(): + * tells how much data has been consumed (input) and produced (output) for current frame. + * able to count progression inside worker threads. + * Note : mutex will be acquired during statistics collection inside workers. */ + private static ZSTD_frameProgression ZSTDMT_getFrameProgression(ZSTDMT_CCtx_s* mtctx) + { + ZSTD_frameProgression fps; + fps.ingested = mtctx->consumed + mtctx->inBuff.filled; + fps.consumed = mtctx->consumed; + fps.produced = fps.flushed = mtctx->produced; + fps.currentJobID = mtctx->nextJobID; + fps.nbActiveWorkers = 0; + { + uint jobNb; + uint lastJobNb = mtctx->nextJobID + (uint)mtctx->jobReady; + assert(mtctx->jobReady <= 1); + for (jobNb = mtctx->doneJobID; jobNb < lastJobNb; jobNb++) + { + uint wJobID = jobNb & mtctx->jobIDMask; + ZSTDMT_jobDescription* jobPtr = &mtctx->jobs[wJobID]; + SynchronizationWrapper.Enter(&jobPtr->job_mutex); + { + nuint cResult = jobPtr->cSize; + nuint produced = ERR_isError(cResult) ? 0 : cResult; + nuint flushed = ERR_isError(cResult) ? 0 : jobPtr->dstFlushed; + assert(flushed <= produced); + fps.ingested += jobPtr->src.size; + fps.consumed += jobPtr->consumed; + fps.produced += produced; + fps.flushed += flushed; + fps.nbActiveWorkers += jobPtr->consumed < jobPtr->src.size ? 1U : 0U; + } + + SynchronizationWrapper.Exit(&mtctx->jobs[wJobID].job_mutex); + } + } + + return fps; + } + + /*! ZSTDMT_toFlushNow() + * Tell how many bytes are ready to be flushed immediately. + * Probe the oldest active job (not yet entirely flushed) and check its output buffer. + * If return 0, it means there is no active job, + * or, it means oldest job is still active, but everything produced has been flushed so far, + * therefore flushing is limited by speed of oldest job. */ + private static nuint ZSTDMT_toFlushNow(ZSTDMT_CCtx_s* mtctx) + { + nuint toFlush; + uint jobID = mtctx->doneJobID; + assert(jobID <= mtctx->nextJobID); + if (jobID == mtctx->nextJobID) + { + return 0; + } + + { + uint wJobID = jobID & mtctx->jobIDMask; + ZSTDMT_jobDescription* jobPtr = &mtctx->jobs[wJobID]; + SynchronizationWrapper.Enter(&jobPtr->job_mutex); + { + nuint cResult = jobPtr->cSize; + nuint produced = ERR_isError(cResult) ? 0 : cResult; + nuint flushed = ERR_isError(cResult) ? 0 : jobPtr->dstFlushed; + assert(flushed <= produced); + assert(jobPtr->consumed <= jobPtr->src.size); + toFlush = produced - flushed; + } + + SynchronizationWrapper.Exit(&mtctx->jobs[wJobID].job_mutex); + } + + return toFlush; + } + + /* ------------------------------------------ */ + /* ===== Multi-threaded compression ===== */ + /* ------------------------------------------ */ + private static uint ZSTDMT_computeTargetJobLog(ZSTD_CCtx_params_s* @params) + { + uint jobLog; + if (@params->ldmParams.enableLdm == ZSTD_paramSwitch_e.ZSTD_ps_enable) + { + jobLog = + 21 > ZSTD_cycleLog(@params->cParams.chainLog, @params->cParams.strategy) + 3 + ? 21 + : ZSTD_cycleLog(@params->cParams.chainLog, @params->cParams.strategy) + 3; + } + else + { + jobLog = 20 > @params->cParams.windowLog + 2 ? 20 : @params->cParams.windowLog + 2; + } + + return jobLog < (uint)(MEM_32bits ? 29 : 30) ? jobLog : (uint)(MEM_32bits ? 29 : 30); + } + + private static int ZSTDMT_overlapLog_default(ZSTD_strategy strat) + { + switch (strat) + { + case ZSTD_strategy.ZSTD_btultra2: + return 9; + case ZSTD_strategy.ZSTD_btultra: + case ZSTD_strategy.ZSTD_btopt: + return 8; + case ZSTD_strategy.ZSTD_btlazy2: + case ZSTD_strategy.ZSTD_lazy2: + return 7; + case ZSTD_strategy.ZSTD_lazy: + case ZSTD_strategy.ZSTD_greedy: + case ZSTD_strategy.ZSTD_dfast: + case ZSTD_strategy.ZSTD_fast: + default: + break; + } + + return 6; + } + + private static int ZSTDMT_overlapLog(int ovlog, ZSTD_strategy strat) + { + assert(0 <= ovlog && ovlog <= 9); + if (ovlog == 0) + { + return ZSTDMT_overlapLog_default(strat); + } + + return ovlog; + } + + private static nuint ZSTDMT_computeOverlapSize(ZSTD_CCtx_params_s* @params) + { + int overlapRLog = 9 - ZSTDMT_overlapLog(@params->overlapLog, @params->cParams.strategy); + int ovLog = (int)(overlapRLog >= 8 ? 0 : @params->cParams.windowLog - (uint)overlapRLog); + assert(0 <= overlapRLog && overlapRLog <= 8); + if (@params->ldmParams.enableLdm == ZSTD_paramSwitch_e.ZSTD_ps_enable) + { + ovLog = (int)( + ( + @params->cParams.windowLog < ZSTDMT_computeTargetJobLog(@params) - 2 + ? @params->cParams.windowLog + : ZSTDMT_computeTargetJobLog(@params) - 2 + ) - (uint)overlapRLog + ); + } + + assert(0 <= ovLog && ovLog <= (sizeof(nuint) == 4 ? 30 : 31)); + return ovLog == 0 ? 0 : (nuint)1 << ovLog; + } + + /* ====================================== */ + /* ======= Streaming API ======= */ + /* ====================================== */ + private static nuint ZSTDMT_initCStream_internal( + ZSTDMT_CCtx_s* mtctx, + void* dict, + nuint dictSize, + ZSTD_dictContentType_e dictContentType, + ZSTD_CDict_s* cdict, + ZSTD_CCtx_params_s @params, + ulong pledgedSrcSize + ) + { + assert(!ERR_isError(ZSTD_checkCParams(@params.cParams))); + assert(!(dict != null && cdict != null)); + if (@params.nbWorkers != mtctx->@params.nbWorkers) + { + /* init */ + nuint err_code = ZSTDMT_resize(mtctx, (uint)@params.nbWorkers); + if (ERR_isError(err_code)) + { + return err_code; + } + } + + if (@params.jobSize != 0 && @params.jobSize < 512 * (1 << 10)) + { + @params.jobSize = 512 * (1 << 10); + } + + if (@params.jobSize > (nuint)(MEM_32bits ? 512 * (1 << 20) : 1024 * (1 << 20))) + { + @params.jobSize = (nuint)(MEM_32bits ? 512 * (1 << 20) : 1024 * (1 << 20)); + } + + if (mtctx->allJobsCompleted == 0) + { + ZSTDMT_waitForAllJobsCompleted(mtctx); + ZSTDMT_releaseAllJobResources(mtctx); + mtctx->allJobsCompleted = 1; + } + + mtctx->@params = @params; + mtctx->frameContentSize = pledgedSrcSize; + ZSTD_freeCDict(mtctx->cdictLocal); + if (dict != null) + { + mtctx->cdictLocal = ZSTD_createCDict_advanced( + dict, + dictSize, + ZSTD_dictLoadMethod_e.ZSTD_dlm_byCopy, + dictContentType, + @params.cParams, + mtctx->cMem + ); + mtctx->cdict = mtctx->cdictLocal; + if (mtctx->cdictLocal == null) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + } + else + { + mtctx->cdictLocal = null; + mtctx->cdict = cdict; + } + + mtctx->targetPrefixSize = ZSTDMT_computeOverlapSize(&@params); + mtctx->targetSectionSize = @params.jobSize; + if (mtctx->targetSectionSize == 0) + { + mtctx->targetSectionSize = (nuint)(1UL << (int)ZSTDMT_computeTargetJobLog(&@params)); + } + + assert( + mtctx->targetSectionSize <= (nuint)(MEM_32bits ? 512 * (1 << 20) : 1024 * (1 << 20)) + ); + if (@params.rsyncable != 0) + { + /* Aim for the targetsectionSize as the average job size. */ + uint jobSizeKB = (uint)(mtctx->targetSectionSize >> 10); + assert(jobSizeKB >= 1); + uint rsyncBits = ZSTD_highbit32(jobSizeKB) + 10; + assert(rsyncBits >= 17 + 2); + mtctx->rsync.hash = 0; + mtctx->rsync.hitMask = (1UL << (int)rsyncBits) - 1; + mtctx->rsync.primePower = ZSTD_rollingHash_primePower(32); + } + + if (mtctx->targetSectionSize < mtctx->targetPrefixSize) + { + mtctx->targetSectionSize = mtctx->targetPrefixSize; + } + + ZSTDMT_setBufferSize(mtctx->bufPool, ZSTD_compressBound(mtctx->targetSectionSize)); + { + /* If ldm is enabled we need windowSize space. */ + nuint windowSize = + mtctx->@params.ldmParams.enableLdm == ZSTD_paramSwitch_e.ZSTD_ps_enable + ? 1U << (int)mtctx->@params.cParams.windowLog + : 0; + /* Two buffers of slack, plus extra space for the overlap + * This is the minimum slack that LDM works with. One extra because + * flush might waste up to targetSectionSize-1 bytes. Another extra + * for the overlap (if > 0), then one to fill which doesn't overlap + * with the LDM window. + */ + nuint nbSlackBuffers = (nuint)(2 + (mtctx->targetPrefixSize > 0 ? 1 : 0)); + nuint slackSize = mtctx->targetSectionSize * nbSlackBuffers; + /* Compute the total size, and always have enough slack */ + nuint nbWorkers = (nuint)(mtctx->@params.nbWorkers > 1 ? mtctx->@params.nbWorkers : 1); + nuint sectionsSize = mtctx->targetSectionSize * nbWorkers; + nuint capacity = (windowSize > sectionsSize ? windowSize : sectionsSize) + slackSize; + if (mtctx->roundBuff.capacity < capacity) + { + if (mtctx->roundBuff.buffer != null) + { + ZSTD_customFree(mtctx->roundBuff.buffer, mtctx->cMem); + } + + mtctx->roundBuff.buffer = (byte*)ZSTD_customMalloc(capacity, mtctx->cMem); + if (mtctx->roundBuff.buffer == null) + { + mtctx->roundBuff.capacity = 0; + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + + mtctx->roundBuff.capacity = capacity; + } + } + + mtctx->roundBuff.pos = 0; + mtctx->inBuff.buffer = g_nullBuffer; + mtctx->inBuff.filled = 0; + mtctx->inBuff.prefix = kNullRange; + mtctx->doneJobID = 0; + mtctx->nextJobID = 0; + mtctx->frameEnded = 0; + mtctx->allJobsCompleted = 0; + mtctx->consumed = 0; + mtctx->produced = 0; + ZSTD_freeCDict(mtctx->cdictLocal); + mtctx->cdictLocal = null; + mtctx->cdict = null; + if (dict != null) + { + if (dictContentType == ZSTD_dictContentType_e.ZSTD_dct_rawContent) + { + mtctx->inBuff.prefix.start = (byte*)dict; + mtctx->inBuff.prefix.size = dictSize; + } + else + { + mtctx->cdictLocal = ZSTD_createCDict_advanced( + dict, + dictSize, + ZSTD_dictLoadMethod_e.ZSTD_dlm_byRef, + dictContentType, + @params.cParams, + mtctx->cMem + ); + mtctx->cdict = mtctx->cdictLocal; + if (mtctx->cdictLocal == null) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + } + } + else + { + mtctx->cdict = cdict; + } + + if ( + ZSTDMT_serialState_reset( + &mtctx->serial, + mtctx->seqPool, + @params, + mtctx->targetSectionSize, + dict, + dictSize, + dictContentType + ) != 0 + ) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + } + + return 0; + } + + /* ZSTDMT_writeLastEmptyBlock() + * Write a single empty block with an end-of-frame to finish a frame. + * Job must be created from streaming variant. + * This function is always successful if expected conditions are fulfilled. + */ + private static void ZSTDMT_writeLastEmptyBlock(ZSTDMT_jobDescription* job) + { + assert(job->lastJob == 1); + assert(job->src.size == 0); + assert(job->firstJob == 0); + assert(job->dstBuff.start == null); + job->dstBuff = ZSTDMT_getBuffer(job->bufPool); + if (job->dstBuff.start == null) + { + job->cSize = unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_memory_allocation)); + return; + } + + assert(job->dstBuff.capacity >= ZSTD_blockHeaderSize); + job->src = kNullRange; + job->cSize = ZSTD_writeLastEmptyBlock(job->dstBuff.start, job->dstBuff.capacity); + assert(!ERR_isError(job->cSize)); + assert(job->consumed == 0); + } + + private static nuint ZSTDMT_createCompressionJob( + ZSTDMT_CCtx_s* mtctx, + nuint srcSize, + ZSTD_EndDirective endOp + ) + { + uint jobID = mtctx->nextJobID & mtctx->jobIDMask; + int endFrame = endOp == ZSTD_EndDirective.ZSTD_e_end ? 1 : 0; + if (mtctx->nextJobID > mtctx->doneJobID + mtctx->jobIDMask) + { + assert((mtctx->nextJobID & mtctx->jobIDMask) == (mtctx->doneJobID & mtctx->jobIDMask)); + return 0; + } + + if (mtctx->jobReady == 0) + { + byte* src = (byte*)mtctx->inBuff.buffer.start; + mtctx->jobs[jobID].src.start = src; + mtctx->jobs[jobID].src.size = srcSize; + assert(mtctx->inBuff.filled >= srcSize); + mtctx->jobs[jobID].prefix = mtctx->inBuff.prefix; + mtctx->jobs[jobID].consumed = 0; + mtctx->jobs[jobID].cSize = 0; + mtctx->jobs[jobID].@params = mtctx->@params; + mtctx->jobs[jobID].cdict = mtctx->nextJobID == 0 ? mtctx->cdict : null; + mtctx->jobs[jobID].fullFrameSize = mtctx->frameContentSize; + mtctx->jobs[jobID].dstBuff = g_nullBuffer; + mtctx->jobs[jobID].cctxPool = mtctx->cctxPool; + mtctx->jobs[jobID].bufPool = mtctx->bufPool; + mtctx->jobs[jobID].seqPool = mtctx->seqPool; + mtctx->jobs[jobID].serial = &mtctx->serial; + mtctx->jobs[jobID].jobID = mtctx->nextJobID; + mtctx->jobs[jobID].firstJob = mtctx->nextJobID == 0 ? 1U : 0U; + mtctx->jobs[jobID].lastJob = (uint)endFrame; + mtctx->jobs[jobID].frameChecksumNeeded = + mtctx->@params.fParams.checksumFlag != 0 && endFrame != 0 && mtctx->nextJobID > 0 + ? 1U + : 0U; + mtctx->jobs[jobID].dstFlushed = 0; + mtctx->roundBuff.pos += srcSize; + mtctx->inBuff.buffer = g_nullBuffer; + mtctx->inBuff.filled = 0; + if (endFrame == 0) + { + nuint newPrefixSize = + srcSize < mtctx->targetPrefixSize ? srcSize : mtctx->targetPrefixSize; + mtctx->inBuff.prefix.start = src + srcSize - newPrefixSize; + mtctx->inBuff.prefix.size = newPrefixSize; + } + else + { + mtctx->inBuff.prefix = kNullRange; + mtctx->frameEnded = (uint)endFrame; + if (mtctx->nextJobID == 0) + { + mtctx->@params.fParams.checksumFlag = 0; + } + } + + if (srcSize == 0 && mtctx->nextJobID > 0) + { + assert(endOp == ZSTD_EndDirective.ZSTD_e_end); + ZSTDMT_writeLastEmptyBlock(mtctx->jobs + jobID); + mtctx->nextJobID++; + return 0; + } + } + + if ( + POOL_tryAdd( + mtctx->factory, + (delegate* managed)(&ZSTDMT_compressionJob), + &mtctx->jobs[jobID] + ) != 0 + ) + { + mtctx->nextJobID++; + mtctx->jobReady = 0; + } + else + { + mtctx->jobReady = 1; + } + + return 0; + } + + /*! ZSTDMT_flushProduced() : + * flush whatever data has been produced but not yet flushed in current job. + * move to next job if current one is fully flushed. + * `output` : `pos` will be updated with amount of data flushed . + * `blockToFlush` : if >0, the function will block and wait if there is no data available to flush . + * @return : amount of data remaining within internal buffer, 0 if no more, 1 if unknown but > 0, or an error code */ + private static nuint ZSTDMT_flushProduced( + ZSTDMT_CCtx_s* mtctx, + ZSTD_outBuffer_s* output, + uint blockToFlush, + ZSTD_EndDirective end + ) + { + uint wJobID = mtctx->doneJobID & mtctx->jobIDMask; + assert(output->size >= output->pos); + SynchronizationWrapper.Enter(&mtctx->jobs[wJobID].job_mutex); + if (blockToFlush != 0 && mtctx->doneJobID < mtctx->nextJobID) + { + assert(mtctx->jobs[wJobID].dstFlushed <= mtctx->jobs[wJobID].cSize); + while (mtctx->jobs[wJobID].dstFlushed == mtctx->jobs[wJobID].cSize) + { + if (mtctx->jobs[wJobID].consumed == mtctx->jobs[wJobID].src.size) + { + break; + } + + SynchronizationWrapper.Wait(&mtctx->jobs[wJobID].job_mutex); + } + } + + { + /* shared */ + nuint cSize = mtctx->jobs[wJobID].cSize; + /* shared */ + nuint srcConsumed = mtctx->jobs[wJobID].consumed; + /* read-only, could be done after mutex lock, but no-declaration-after-statement */ + nuint srcSize = mtctx->jobs[wJobID].src.size; + SynchronizationWrapper.Exit(&mtctx->jobs[wJobID].job_mutex); + if (ERR_isError(cSize)) + { + ZSTDMT_waitForAllJobsCompleted(mtctx); + ZSTDMT_releaseAllJobResources(mtctx); + return cSize; + } + + assert(srcConsumed <= srcSize); + if (srcConsumed == srcSize && mtctx->jobs[wJobID].frameChecksumNeeded != 0) + { + uint checksum = (uint)ZSTD_XXH64_digest(&mtctx->serial.xxhState); + MEM_writeLE32( + (sbyte*)mtctx->jobs[wJobID].dstBuff.start + mtctx->jobs[wJobID].cSize, + checksum + ); + cSize += 4; + mtctx->jobs[wJobID].cSize += 4; + mtctx->jobs[wJobID].frameChecksumNeeded = 0; + } + + if (cSize > 0) + { + nuint toFlush = + cSize - mtctx->jobs[wJobID].dstFlushed < output->size - output->pos + ? cSize - mtctx->jobs[wJobID].dstFlushed + : output->size - output->pos; + assert(mtctx->doneJobID < mtctx->nextJobID); + assert(cSize >= mtctx->jobs[wJobID].dstFlushed); + assert(mtctx->jobs[wJobID].dstBuff.start != null); + if (toFlush > 0) + { + memcpy( + (sbyte*)output->dst + output->pos, + (sbyte*)mtctx->jobs[wJobID].dstBuff.start + mtctx->jobs[wJobID].dstFlushed, + (uint)toFlush + ); + } + + output->pos += toFlush; + mtctx->jobs[wJobID].dstFlushed += toFlush; + if (srcConsumed == srcSize && mtctx->jobs[wJobID].dstFlushed == cSize) + { + ZSTDMT_releaseBuffer(mtctx->bufPool, mtctx->jobs[wJobID].dstBuff); + mtctx->jobs[wJobID].dstBuff = g_nullBuffer; + mtctx->jobs[wJobID].cSize = 0; + mtctx->consumed += srcSize; + mtctx->produced += cSize; + mtctx->doneJobID++; + } + } + + if (cSize > mtctx->jobs[wJobID].dstFlushed) + { + return cSize - mtctx->jobs[wJobID].dstFlushed; + } + + if (srcSize > srcConsumed) + { + return 1; + } + } + + if (mtctx->doneJobID < mtctx->nextJobID) + { + return 1; + } + + if (mtctx->jobReady != 0) + { + return 1; + } + + if (mtctx->inBuff.filled > 0) + { + return 1; + } + + mtctx->allJobsCompleted = mtctx->frameEnded; + if (end == ZSTD_EndDirective.ZSTD_e_end) + { + return mtctx->frameEnded == 0 ? 1U : 0U; + } + + return 0; + } + + /** + * Returns the range of data used by the earliest job that is not yet complete. + * If the data of the first job is broken up into two segments, we cover both + * sections. + */ + private static Range ZSTDMT_getInputDataInUse(ZSTDMT_CCtx_s* mtctx) + { + uint firstJobID = mtctx->doneJobID; + uint lastJobID = mtctx->nextJobID; + uint jobID; + /* no need to check during first round */ + nuint roundBuffCapacity = mtctx->roundBuff.capacity; + nuint nbJobs1stRoundMin = roundBuffCapacity / mtctx->targetSectionSize; + if (lastJobID < nbJobs1stRoundMin) + { + return kNullRange; + } + + for (jobID = firstJobID; jobID < lastJobID; ++jobID) + { + uint wJobID = jobID & mtctx->jobIDMask; + nuint consumed; + SynchronizationWrapper.Enter(&mtctx->jobs[wJobID].job_mutex); + consumed = mtctx->jobs[wJobID].consumed; + SynchronizationWrapper.Exit(&mtctx->jobs[wJobID].job_mutex); + if (consumed < mtctx->jobs[wJobID].src.size) + { + Range range = mtctx->jobs[wJobID].prefix; + if (range.size == 0) + { + range = mtctx->jobs[wJobID].src; + } + + assert(range.start <= mtctx->jobs[wJobID].src.start); + return range; + } + } + + return kNullRange; + } + + /** + * Returns non-zero iff buffer and range overlap. + */ + private static int ZSTDMT_isOverlapped(buffer_s buffer, Range range) + { + byte* bufferStart = (byte*)buffer.start; + byte* rangeStart = (byte*)range.start; + if (rangeStart == null || bufferStart == null) + { + return 0; + } + + { + byte* bufferEnd = bufferStart + buffer.capacity; + byte* rangeEnd = rangeStart + range.size; + if (bufferStart == bufferEnd || rangeStart == rangeEnd) + { + return 0; + } + + return bufferStart < rangeEnd && rangeStart < bufferEnd ? 1 : 0; + } + } + + private static int ZSTDMT_doesOverlapWindow(buffer_s buffer, ZSTD_window_t window) + { + Range extDict; + Range prefix; + extDict.start = window.dictBase + window.lowLimit; + extDict.size = window.dictLimit - window.lowLimit; + prefix.start = window.@base + window.dictLimit; + prefix.size = (nuint)(window.nextSrc - (window.@base + window.dictLimit)); + return ZSTDMT_isOverlapped(buffer, extDict) != 0 || ZSTDMT_isOverlapped(buffer, prefix) != 0 + ? 1 + : 0; + } + + private static void ZSTDMT_waitForLdmComplete(ZSTDMT_CCtx_s* mtctx, buffer_s buffer) + { + if (mtctx->@params.ldmParams.enableLdm == ZSTD_paramSwitch_e.ZSTD_ps_enable) + { + void** mutex = &mtctx->serial.ldmWindowMutex; + SynchronizationWrapper.Enter(mutex); + while (ZSTDMT_doesOverlapWindow(buffer, mtctx->serial.ldmWindow) != 0) + { + SynchronizationWrapper.Wait(mutex); + } + + SynchronizationWrapper.Exit(mutex); + } + } + + /** + * Attempts to set the inBuff to the next section to fill. + * If any part of the new section is still in use we give up. + * Returns non-zero if the buffer is filled. + */ + private static int ZSTDMT_tryGetInputRange(ZSTDMT_CCtx_s* mtctx) + { + Range inUse = ZSTDMT_getInputDataInUse(mtctx); + nuint spaceLeft = mtctx->roundBuff.capacity - mtctx->roundBuff.pos; + nuint spaceNeeded = mtctx->targetSectionSize; + buffer_s buffer; + assert(mtctx->inBuff.buffer.start == null); + assert(mtctx->roundBuff.capacity >= spaceNeeded); + if (spaceLeft < spaceNeeded) + { + /* ZSTD_invalidateRepCodes() doesn't work for extDict variants. + * Simply copy the prefix to the beginning in that case. + */ + byte* start = mtctx->roundBuff.buffer; + nuint prefixSize = mtctx->inBuff.prefix.size; + buffer.start = start; + buffer.capacity = prefixSize; + if (ZSTDMT_isOverlapped(buffer, inUse) != 0) + { + return 0; + } + + ZSTDMT_waitForLdmComplete(mtctx, buffer); + memmove(start, mtctx->inBuff.prefix.start, prefixSize); + mtctx->inBuff.prefix.start = start; + mtctx->roundBuff.pos = prefixSize; + } + + buffer.start = mtctx->roundBuff.buffer + mtctx->roundBuff.pos; + buffer.capacity = spaceNeeded; + if (ZSTDMT_isOverlapped(buffer, inUse) != 0) + { + return 0; + } + + assert(ZSTDMT_isOverlapped(buffer, mtctx->inBuff.prefix) == 0); + ZSTDMT_waitForLdmComplete(mtctx, buffer); + mtctx->inBuff.buffer = buffer; + mtctx->inBuff.filled = 0; + assert(mtctx->roundBuff.pos + buffer.capacity <= mtctx->roundBuff.capacity); + return 1; + } + + /** + * Searches through the input for a synchronization point. If one is found, we + * will instruct the caller to flush, and return the number of bytes to load. + * Otherwise, we will load as many bytes as possible and instruct the caller + * to continue as normal. + */ + private static SyncPoint findSynchronizationPoint(ZSTDMT_CCtx_s* mtctx, ZSTD_inBuffer_s input) + { + byte* istart = (byte*)input.src + input.pos; + ulong primePower = mtctx->rsync.primePower; + ulong hitMask = mtctx->rsync.hitMask; + SyncPoint syncPoint; + ulong hash; + byte* prev; + nuint pos; + syncPoint.toLoad = + input.size - input.pos < mtctx->targetSectionSize - mtctx->inBuff.filled + ? input.size - input.pos + : mtctx->targetSectionSize - mtctx->inBuff.filled; + syncPoint.flush = 0; + if (mtctx->@params.rsyncable == 0) + { + return syncPoint; + } + + if (mtctx->inBuff.filled + input.size - input.pos < 1 << 17) + { + return syncPoint; + } + + if (mtctx->inBuff.filled + syncPoint.toLoad < 32) + { + return syncPoint; + } + + if (mtctx->inBuff.filled < 1 << 17) + { + pos = (1 << 17) - mtctx->inBuff.filled; + if (pos >= 32) + { + prev = istart + pos - 32; + hash = ZSTD_rollingHash_compute(prev, 32); + } + else + { + assert(mtctx->inBuff.filled >= 32); + prev = (byte*)mtctx->inBuff.buffer.start + mtctx->inBuff.filled - 32; + hash = ZSTD_rollingHash_compute(prev + pos, 32 - pos); + hash = ZSTD_rollingHash_append(hash, istart, pos); + } + } + else + { + assert(mtctx->inBuff.filled >= 1 << 17); + assert(1 << 17 >= 32); + pos = 0; + prev = (byte*)mtctx->inBuff.buffer.start + mtctx->inBuff.filled - 32; + hash = ZSTD_rollingHash_compute(prev, 32); + if ((hash & hitMask) == hitMask) + { + syncPoint.toLoad = 0; + syncPoint.flush = 1; + return syncPoint; + } + } + + assert(pos < 32 || ZSTD_rollingHash_compute(istart + pos - 32, 32) == hash); + for (; pos < syncPoint.toLoad; ++pos) + { + byte toRemove = pos < 32 ? prev[pos] : istart[pos - 32]; + hash = ZSTD_rollingHash_rotate(hash, toRemove, istart[pos], primePower); + assert(mtctx->inBuff.filled + pos >= 1 << 17); + if ((hash & hitMask) == hitMask) + { + syncPoint.toLoad = pos + 1; + syncPoint.flush = 1; + ++pos; + break; + } + } + + assert(pos < 32 || ZSTD_rollingHash_compute(istart + pos - 32, 32) == hash); + return syncPoint; + } + + /* === Streaming functions === */ + private static nuint ZSTDMT_nextInputSizeHint(ZSTDMT_CCtx_s* mtctx) + { + nuint hintInSize = mtctx->targetSectionSize - mtctx->inBuff.filled; + if (hintInSize == 0) + { + hintInSize = mtctx->targetSectionSize; + } + + return hintInSize; + } + + /** ZSTDMT_compressStream_generic() : + * internal use only - exposed to be invoked from zstd_compress.c + * assumption : output and input are valid (pos <= size) + * @return : minimum amount of data remaining to flush, 0 if none */ + private static nuint ZSTDMT_compressStream_generic( + ZSTDMT_CCtx_s* mtctx, + ZSTD_outBuffer_s* output, + ZSTD_inBuffer_s* input, + ZSTD_EndDirective endOp + ) + { + uint forwardInputProgress = 0; + assert(output->pos <= output->size); + assert(input->pos <= input->size); + if (mtctx->frameEnded != 0 && endOp == ZSTD_EndDirective.ZSTD_e_continue) + { + return unchecked((nuint)(-(int)ZSTD_ErrorCode.ZSTD_error_stage_wrong)); + } + + if (mtctx->jobReady == 0 && input->size > input->pos) + { + if (mtctx->inBuff.buffer.start == null) + { + assert(mtctx->inBuff.filled == 0); + if (ZSTDMT_tryGetInputRange(mtctx) == 0) + { + assert(mtctx->doneJobID != mtctx->nextJobID); + } + } + + if (mtctx->inBuff.buffer.start != null) + { + SyncPoint syncPoint = findSynchronizationPoint(mtctx, *input); + if (syncPoint.flush != 0 && endOp == ZSTD_EndDirective.ZSTD_e_continue) + { + endOp = ZSTD_EndDirective.ZSTD_e_flush; + } + + assert(mtctx->inBuff.buffer.capacity >= mtctx->targetSectionSize); + memcpy( + (sbyte*)mtctx->inBuff.buffer.start + mtctx->inBuff.filled, + (sbyte*)input->src + input->pos, + (uint)syncPoint.toLoad + ); + input->pos += syncPoint.toLoad; + mtctx->inBuff.filled += syncPoint.toLoad; + forwardInputProgress = syncPoint.toLoad > 0 ? 1U : 0U; + } + } + + if (input->pos < input->size && endOp == ZSTD_EndDirective.ZSTD_e_end) + { + assert( + mtctx->inBuff.filled == 0 + || mtctx->inBuff.filled == mtctx->targetSectionSize + || mtctx->@params.rsyncable != 0 + ); + endOp = ZSTD_EndDirective.ZSTD_e_flush; + } + + if ( + mtctx->jobReady != 0 + || mtctx->inBuff.filled >= mtctx->targetSectionSize + || endOp != ZSTD_EndDirective.ZSTD_e_continue && mtctx->inBuff.filled > 0 + || endOp == ZSTD_EndDirective.ZSTD_e_end && mtctx->frameEnded == 0 + ) + { + nuint jobSize = mtctx->inBuff.filled; + assert(mtctx->inBuff.filled <= mtctx->targetSectionSize); + { + nuint err_code = ZSTDMT_createCompressionJob(mtctx, jobSize, endOp); + if (ERR_isError(err_code)) + { + return err_code; + } + } + } + + { + /* block if there was no forward input progress */ + nuint remainingToFlush = ZSTDMT_flushProduced( + mtctx, + output, + forwardInputProgress == 0 ? 1U : 0U, + endOp + ); + if (input->pos < input->size) + { + return remainingToFlush > 1 ? remainingToFlush : 1; + } + + return remainingToFlush; + } + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/_wksps_e__Union.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/_wksps_e__Union.cs new file mode 100644 index 00000000..af51bc19 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/_wksps_e__Union.cs @@ -0,0 +1,16 @@ +using System.Runtime.InteropServices; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +[StructLayout(LayoutKind.Explicit)] +public unsafe struct _wksps_e__Union +{ + [FieldOffset(0)] + public HUF_buildCTable_wksp_tables buildCTable_wksp; + + [FieldOffset(0)] + public HUF_WriteCTableWksp writeCTable_wksp; + + [FieldOffset(0)] + public fixed uint hist_wksp[1024]; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/algo_time_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/algo_time_t.cs new file mode 100644 index 00000000..869a6880 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/algo_time_t.cs @@ -0,0 +1,13 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct algo_time_t +{ + public uint tableTime; + public uint decode256Time; + + public algo_time_t(uint tableTime, uint decode256Time) + { + this.tableTime = tableTime; + this.decode256Time = decode256Time; + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/base_directive_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/base_directive_e.cs new file mode 100644 index 00000000..b461bbfd --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/base_directive_e.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum base_directive_e +{ + base_0possible = 0, + base_1guaranteed = 1, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/blockProperties_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/blockProperties_t.cs new file mode 100644 index 00000000..e1f68e38 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/blockProperties_t.cs @@ -0,0 +1,8 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct blockProperties_t +{ + public blockType_e blockType; + public uint lastBlock; + public uint origSize; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/blockType_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/blockType_e.cs new file mode 100644 index 00000000..1c0df0a2 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/blockType_e.cs @@ -0,0 +1,9 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum blockType_e +{ + bt_raw, + bt_rle, + bt_compressed, + bt_reserved, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/buffer_s.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/buffer_s.cs new file mode 100644 index 00000000..6ddf1b22 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/buffer_s.cs @@ -0,0 +1,15 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/* ===== Buffer Pool ===== */ +/* a single Buffer Pool can be invoked from multiple threads in parallel */ +public unsafe struct buffer_s +{ + public void* start; + public nuint capacity; + + public buffer_s(void* start, nuint capacity) + { + this.start = start; + this.capacity = capacity; + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/dictItem.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/dictItem.cs new file mode 100644 index 00000000..929fea69 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/dictItem.cs @@ -0,0 +1,8 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct dictItem +{ + public uint pos; + public uint length; + public uint savings; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/inBuff_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/inBuff_t.cs new file mode 100644 index 00000000..1ea5fe48 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/inBuff_t.cs @@ -0,0 +1,12 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/* ------------------------------------------ */ +/* ===== Multi-threaded compression ===== */ +/* ------------------------------------------ */ +public struct InBuff_t +{ + /* read-only non-owned prefix buffer */ + public Range prefix; + public buffer_s buffer; + public nuint filled; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ldmEntry_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ldmEntry_t.cs new file mode 100644 index 00000000..81304cea --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ldmEntry_t.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct ldmEntry_t +{ + public uint offset; + public uint checksum; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ldmMatchCandidate_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ldmMatchCandidate_t.cs new file mode 100644 index 00000000..e6f6d465 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ldmMatchCandidate_t.cs @@ -0,0 +1,9 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct ldmMatchCandidate_t +{ + public byte* split; + public uint hash; + public uint checksum; + public ldmEntry_t* bucket; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ldmParams_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ldmParams_t.cs new file mode 100644 index 00000000..71c108dd --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ldmParams_t.cs @@ -0,0 +1,22 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct ldmParams_t +{ + /* ZSTD_ps_enable to enable LDM. ZSTD_ps_auto by default */ + public ZSTD_paramSwitch_e enableLdm; + + /* Log size of hashTable */ + public uint hashLog; + + /* Log bucket size for collision resolution, at most 8 */ + public uint bucketSizeLog; + + /* Minimum match length */ + public uint minMatchLength; + + /* Log number of entries to skip */ + public uint hashRateLog; + + /* Window log for the LDM */ + public uint windowLog; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ldmRollingHashState_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ldmRollingHashState_t.cs new file mode 100644 index 00000000..73d11fd4 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ldmRollingHashState_t.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct ldmRollingHashState_t +{ + public ulong rolling; + public ulong stopMask; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/ldmState_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/ldmState_t.cs new file mode 100644 index 00000000..5596880d --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/ldmState_t.cs @@ -0,0 +1,170 @@ +using System.Runtime.CompilerServices; + +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct ldmState_t +{ + /* State for the window round buffer management */ + public ZSTD_window_t window; + public ldmEntry_t* hashTable; + public uint loadedDictEnd; + + /* Next position in bucket to insert entry */ + public byte* bucketOffsets; + public _splitIndices_e__FixedBuffer splitIndices; + public _matchCandidates_e__FixedBuffer matchCandidates; + +#if NET8_0_OR_GREATER + [InlineArray(64)] + public unsafe struct _splitIndices_e__FixedBuffer + { + public nuint e0; + } + +#else + public unsafe struct _splitIndices_e__FixedBuffer + { + public nuint e0; + public nuint e1; + public nuint e2; + public nuint e3; + public nuint e4; + public nuint e5; + public nuint e6; + public nuint e7; + public nuint e8; + public nuint e9; + public nuint e10; + public nuint e11; + public nuint e12; + public nuint e13; + public nuint e14; + public nuint e15; + public nuint e16; + public nuint e17; + public nuint e18; + public nuint e19; + public nuint e20; + public nuint e21; + public nuint e22; + public nuint e23; + public nuint e24; + public nuint e25; + public nuint e26; + public nuint e27; + public nuint e28; + public nuint e29; + public nuint e30; + public nuint e31; + public nuint e32; + public nuint e33; + public nuint e34; + public nuint e35; + public nuint e36; + public nuint e37; + public nuint e38; + public nuint e39; + public nuint e40; + public nuint e41; + public nuint e42; + public nuint e43; + public nuint e44; + public nuint e45; + public nuint e46; + public nuint e47; + public nuint e48; + public nuint e49; + public nuint e50; + public nuint e51; + public nuint e52; + public nuint e53; + public nuint e54; + public nuint e55; + public nuint e56; + public nuint e57; + public nuint e58; + public nuint e59; + public nuint e60; + public nuint e61; + public nuint e62; + public nuint e63; + } +#endif + +#if NET8_0_OR_GREATER + [InlineArray(64)] + public unsafe struct _matchCandidates_e__FixedBuffer + { + public ldmMatchCandidate_t e0; + } + +#else + public unsafe struct _matchCandidates_e__FixedBuffer + { + public ldmMatchCandidate_t e0; + public ldmMatchCandidate_t e1; + public ldmMatchCandidate_t e2; + public ldmMatchCandidate_t e3; + public ldmMatchCandidate_t e4; + public ldmMatchCandidate_t e5; + public ldmMatchCandidate_t e6; + public ldmMatchCandidate_t e7; + public ldmMatchCandidate_t e8; + public ldmMatchCandidate_t e9; + public ldmMatchCandidate_t e10; + public ldmMatchCandidate_t e11; + public ldmMatchCandidate_t e12; + public ldmMatchCandidate_t e13; + public ldmMatchCandidate_t e14; + public ldmMatchCandidate_t e15; + public ldmMatchCandidate_t e16; + public ldmMatchCandidate_t e17; + public ldmMatchCandidate_t e18; + public ldmMatchCandidate_t e19; + public ldmMatchCandidate_t e20; + public ldmMatchCandidate_t e21; + public ldmMatchCandidate_t e22; + public ldmMatchCandidate_t e23; + public ldmMatchCandidate_t e24; + public ldmMatchCandidate_t e25; + public ldmMatchCandidate_t e26; + public ldmMatchCandidate_t e27; + public ldmMatchCandidate_t e28; + public ldmMatchCandidate_t e29; + public ldmMatchCandidate_t e30; + public ldmMatchCandidate_t e31; + public ldmMatchCandidate_t e32; + public ldmMatchCandidate_t e33; + public ldmMatchCandidate_t e34; + public ldmMatchCandidate_t e35; + public ldmMatchCandidate_t e36; + public ldmMatchCandidate_t e37; + public ldmMatchCandidate_t e38; + public ldmMatchCandidate_t e39; + public ldmMatchCandidate_t e40; + public ldmMatchCandidate_t e41; + public ldmMatchCandidate_t e42; + public ldmMatchCandidate_t e43; + public ldmMatchCandidate_t e44; + public ldmMatchCandidate_t e45; + public ldmMatchCandidate_t e46; + public ldmMatchCandidate_t e47; + public ldmMatchCandidate_t e48; + public ldmMatchCandidate_t e49; + public ldmMatchCandidate_t e50; + public ldmMatchCandidate_t e51; + public ldmMatchCandidate_t e52; + public ldmMatchCandidate_t e53; + public ldmMatchCandidate_t e54; + public ldmMatchCandidate_t e55; + public ldmMatchCandidate_t e56; + public ldmMatchCandidate_t e57; + public ldmMatchCandidate_t e58; + public ldmMatchCandidate_t e59; + public ldmMatchCandidate_t e60; + public ldmMatchCandidate_t e61; + public ldmMatchCandidate_t e62; + public ldmMatchCandidate_t e63; + } +#endif +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/nodeElt_s.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/nodeElt_s.cs new file mode 100644 index 00000000..de28886c --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/nodeElt_s.cs @@ -0,0 +1,12 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/* ************************************************************** + * Required declarations + ****************************************************************/ +public struct nodeElt_s +{ + public uint count; + public ushort parent; + public byte @byte; + public byte nbBits; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/offsetCount_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/offsetCount_t.cs new file mode 100644 index 00000000..9a5c8098 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/offsetCount_t.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct offsetCount_t +{ + public uint offset; + public uint count; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/optState_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/optState_t.cs new file mode 100644 index 00000000..d15e8684 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/optState_t.cs @@ -0,0 +1,53 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct optState_t +{ + /* table of literals statistics, of size 256 */ + public uint* litFreq; + + /* table of litLength statistics, of size (MaxLL+1) */ + public uint* litLengthFreq; + + /* table of matchLength statistics, of size (MaxML+1) */ + public uint* matchLengthFreq; + + /* table of offCode statistics, of size (MaxOff+1) */ + public uint* offCodeFreq; + + /* list of found matches, of size ZSTD_OPT_SIZE */ + public ZSTD_match_t* matchTable; + + /* All positions tracked by optimal parser, of size ZSTD_OPT_SIZE */ + public ZSTD_optimal_t* priceTable; + + /* nb of literals */ + public uint litSum; + + /* nb of litLength codes */ + public uint litLengthSum; + + /* nb of matchLength codes */ + public uint matchLengthSum; + + /* nb of offset codes */ + public uint offCodeSum; + + /* to compare to log2(litfreq) */ + public uint litSumBasePrice; + + /* to compare to log2(llfreq) */ + public uint litLengthSumBasePrice; + + /* to compare to log2(mlfreq) */ + public uint matchLengthSumBasePrice; + + /* to compare to log2(offreq) */ + public uint offCodeSumBasePrice; + + /* prices can be determined dynamically, or follow a pre-defined cost structure */ + public ZSTD_OptPrice_e priceType; + + /* pre-calculated dictionary statistics */ + public ZSTD_entropyCTables_t* symbolCosts; + public ZSTD_paramSwitch_e literalCompressionMode; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/rankPos.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/rankPos.cs new file mode 100644 index 00000000..c93cb1f6 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/rankPos.cs @@ -0,0 +1,7 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct rankPos +{ + public ushort @base; + public ushort curr; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/rankValCol_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/rankValCol_t.cs new file mode 100644 index 00000000..94eea418 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/rankValCol_t.cs @@ -0,0 +1,6 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct rankValCol_t +{ + public fixed uint Body[13]; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/rawSeq.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/rawSeq.cs new file mode 100644 index 00000000..e2c315d7 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/rawSeq.cs @@ -0,0 +1,13 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct rawSeq +{ + /* Offset of sequence */ + public uint offset; + + /* Length of literals prior to match */ + public uint litLength; + + /* Raw length of match */ + public uint matchLength; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/repcodes_s.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/repcodes_s.cs new file mode 100644 index 00000000..f9b97bda --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/repcodes_s.cs @@ -0,0 +1,6 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct repcodes_s +{ + public fixed uint rep[3]; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/searchMethod_e.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/searchMethod_e.cs new file mode 100644 index 00000000..10395b04 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/searchMethod_e.cs @@ -0,0 +1,8 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public enum searchMethod_e +{ + search_hashChain = 0, + search_binaryTree = 1, + search_rowHash = 2, +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/seqState_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/seqState_t.cs new file mode 100644 index 00000000..a6aec9c3 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/seqState_t.cs @@ -0,0 +1,17 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public unsafe struct seqState_t +{ + public BIT_DStream_t DStream; + public ZSTD_fseState stateLL; + public ZSTD_fseState stateOffb; + public ZSTD_fseState stateML; + public _prevOffset_e__FixedBuffer prevOffset; + + public unsafe struct _prevOffset_e__FixedBuffer + { + public nuint e0; + public nuint e1; + public nuint e2; + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/seqStoreSplits.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/seqStoreSplits.cs new file mode 100644 index 00000000..83679d5f --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/seqStoreSplits.cs @@ -0,0 +1,11 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/* Struct to keep track of where we are in our recursive calls. */ +public unsafe struct seqStoreSplits +{ + /* Array of split indices */ + public uint* splitLocations; + + /* The current index within splitLocations being worked on */ + public nuint idx; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/seq_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/seq_t.cs new file mode 100644 index 00000000..88395a25 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/seq_t.cs @@ -0,0 +1,8 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct seq_t +{ + public nuint litLength; + public nuint matchLength; + public nuint offset; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/sortedSymbol_t.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/sortedSymbol_t.cs new file mode 100644 index 00000000..acfe3dd9 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/sortedSymbol_t.cs @@ -0,0 +1,6 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +public struct sortedSymbol_t +{ + public byte symbol; +} diff --git a/src/SharpCompress/Compressors/ZStandard/Unsafe/streaming_operation.cs b/src/SharpCompress/Compressors/ZStandard/Unsafe/streaming_operation.cs new file mode 100644 index 00000000..77f189ec --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/Unsafe/streaming_operation.cs @@ -0,0 +1,8 @@ +namespace SharpCompress.Compressors.ZStandard.Unsafe; + +/* Streaming state is used to inform allocation of the literal buffer */ +public enum streaming_operation +{ + not_streaming = 0, + is_streaming = 1, +} diff --git a/src/SharpCompress/Compressors/ZStandard/UnsafeHelper.cs b/src/SharpCompress/Compressors/ZStandard/UnsafeHelper.cs new file mode 100644 index 00000000..a5597644 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/UnsafeHelper.cs @@ -0,0 +1,105 @@ +using System; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; + +namespace SharpCompress.Compressors.ZStandard; + +public static unsafe class UnsafeHelper +{ + public static void* PoisonMemory(void* destination, ulong size) + { + memset(destination, 0xCC, (uint)size); + return destination; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void* malloc(ulong size) + { +#if NET8_0_OR_GREATER + var ptr = NativeMemory.Alloc((nuint)size); +#else + var ptr = (void*)Marshal.AllocHGlobal((nint)size); +#endif + return ptr; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void* calloc(ulong num, ulong size) + { +#if NET8_0_OR_GREATER + return NativeMemory.AllocZeroed((nuint)num, (nuint)size); +#else + var total = num * size; + assert(total <= uint.MaxValue); + var destination = (void*)Marshal.AllocHGlobal((nint)total); + memset(destination, 0, (uint)total); + return destination; +#endif + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void memcpy(void* destination, void* source, uint size) => + System.Runtime.CompilerServices.Unsafe.CopyBlockUnaligned(destination, source, size); + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void memset(void* memPtr, byte val, uint size) => + System.Runtime.CompilerServices.Unsafe.InitBlockUnaligned(memPtr, val, size); + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void free(void* ptr) + { +#if NET8_0_OR_GREATER + NativeMemory.Free(ptr); +#else + Marshal.FreeHGlobal((IntPtr)ptr); +#endif + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static T* GetArrayPointer(T[] array) + where T : unmanaged + { + var size = (uint)(sizeof(T) * array.Length); +#if NET9_0_OR_GREATER + // This function is used to allocate memory for static data blocks. + // We have to use AllocateTypeAssociatedMemory and link the memory's + // lifetime to this assembly, in order to prevent memory leaks when + // loading the assembly in an unloadable AssemblyLoadContext. + // While introduced in .NET 5, we call this only in .NET 9+, because + // it's not implemented in the Mono runtime until then. + var destination = (T*) + RuntimeHelpers.AllocateTypeAssociatedMemory(typeof(UnsafeHelper), (int)size); +#else + var destination = (T*)malloc(size); +#endif + fixed (void* source = &array[0]) + { + System.Runtime.CompilerServices.Unsafe.CopyBlockUnaligned(destination, source, size); + } + + return destination; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void assert(bool condition, string? message = null) + { + if (!condition) + { + throw new ArgumentException(message ?? "assert failed"); + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void memmove(void* destination, void* source, ulong size) => + Buffer.MemoryCopy(source, destination, size, size); + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int memcmp(void* buf1, void* buf2, ulong size) + { + assert(size <= int.MaxValue); + var intSize = (int)size; + return new ReadOnlySpan(buf1, intSize).SequenceCompareTo( + new ReadOnlySpan(buf2, intSize) + ); + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/ZStandardStream.Async.cs b/src/SharpCompress/Compressors/ZStandard/ZStandardStream.Async.cs new file mode 100644 index 00000000..f5932619 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/ZStandardStream.Async.cs @@ -0,0 +1,33 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.IO; + +namespace SharpCompress.Compressors.ZStandard; + +internal partial class ZStandardStream +{ + internal static async ValueTask IsZStandardAsync( + Stream stream, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + var buffer = new byte[4]; + var bytesRead = await stream + .ReadAsync(buffer, 0, 4, cancellationToken) + .ConfigureAwait(false); + if (bytesRead < 4) + { + return false; + } + + var magic = BitConverter.ToUInt32(buffer, 0); + if (ZstandardConstants.MAGIC != magic) + { + return false; + } + return true; + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/ZStandardStream.cs b/src/SharpCompress/Compressors/ZStandard/ZStandardStream.cs new file mode 100644 index 00000000..7b65f1ce --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/ZStandardStream.cs @@ -0,0 +1,48 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Compressors.ZStandard; + +internal partial class ZStandardStream : DecompressionStream +{ + private readonly Stream stream; + + internal static bool IsZStandard(Stream stream) + { + var buffer = new byte[4]; + var bytesRead = stream.Read(buffer, 0, 4); + if (bytesRead < 4) + { + return false; + } + + var magic = BitConverter.ToUInt32(buffer, 0); + if (ZstandardConstants.MAGIC != magic) + { + return false; + } + return true; + } + + public ZStandardStream(Stream baseInputStream) + : base(baseInputStream) + { + this.stream = baseInputStream; + } + + /// + /// The current position within the stream. + /// Throws a NotSupportedException when attempting to set the position + /// + /// Attempting to set the position + public override long Position + { + get { return stream.Position; } + set { throw new NotSupportedException("InflaterInputStream Position not supported"); } + } +} diff --git a/src/SharpCompress/Compressors/ZStandard/ZstandardConstants.cs b/src/SharpCompress/Compressors/ZStandard/ZstandardConstants.cs new file mode 100644 index 00000000..a43b2ef1 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/ZstandardConstants.cs @@ -0,0 +1,22 @@ +namespace SharpCompress.Compressors.ZStandard; + +internal class ZstandardConstants +{ + /// + /// Magic number found at start of ZStandard frame: 0xFD 0x2F 0xB5 0x28 + /// + public const uint MAGIC = 0xFD2FB528; + + /// + /// Maximum uncompressed size of a single ZStandard block: ZSTD_BLOCKSIZE_MAX = 128 KB. + /// + public const int BlockSizeMax = 1 << 17; // 131072 bytes + + /// + /// Recommended input (compressed) buffer size for streaming decompression: + /// ZSTD_DStreamInSize = ZSTD_BLOCKSIZE_MAX + ZSTD_blockHeaderSize (3 bytes). + /// The ring buffer must be at least this large to hold the compressed bytes read + /// during format detection before the first rewind. + /// + public const int DStreamInSize = BlockSizeMax + 3; +} diff --git a/src/SharpCompress/Compressors/ZStandard/ZstdException.cs b/src/SharpCompress/Compressors/ZStandard/ZstdException.cs new file mode 100644 index 00000000..55d79579 --- /dev/null +++ b/src/SharpCompress/Compressors/ZStandard/ZstdException.cs @@ -0,0 +1,12 @@ +using SharpCompress.Common; +using SharpCompress.Compressors.ZStandard.Unsafe; + +namespace SharpCompress.Compressors.ZStandard; + +public class ZstdException : SharpCompressException +{ + public ZstdException(ZSTD_ErrorCode code, string message) + : base(message) => Code = code; + + public ZSTD_ErrorCode Code { get; } +} diff --git a/src/SharpCompress/Crypto/BlockTransformer.cs b/src/SharpCompress/Crypto/BlockTransformer.cs new file mode 100644 index 00000000..d3196aaa --- /dev/null +++ b/src/SharpCompress/Crypto/BlockTransformer.cs @@ -0,0 +1,17 @@ +using System; +using System.Security.Cryptography; + +namespace SharpCompress.Crypto; + +internal class BlockTransformer(ICryptoTransform transformer) : IDisposable +{ + public byte[] ProcessBlock(ReadOnlySpan cipherText) + { + var decryptedBytes = new byte[cipherText.Length]; + transformer.TransformBlock(cipherText.ToArray(), 0, cipherText.Length, decryptedBytes, 0); + + return decryptedBytes; + } + + public void Dispose() { } +} diff --git a/src/SharpCompress/Crypto/Crc32Stream.cs b/src/SharpCompress/Crypto/Crc32Stream.cs index 7f0af6ca..67dcb913 100644 --- a/src/SharpCompress/Crypto/Crc32Stream.cs +++ b/src/SharpCompress/Crypto/Crc32Stream.cs @@ -1,64 +1,102 @@ -#nullable disable - using System; using System.IO; +using System.Threading; +using System.Threading.Tasks; namespace SharpCompress.Crypto; [CLSCompliant(false)] public sealed class Crc32Stream : Stream { - public const uint DefaultPolynomial = 0xedb88320u; - public const uint DefaultSeed = 0xffffffffu; - - private static uint[] defaultTable; - - private readonly uint[] table; - private uint hash; - private readonly Stream stream; + private readonly uint[] _table; + private uint seed; + + public const uint DEFAULT_POLYNOMIAL = 0xedb88320u; + public const uint DEFAULT_SEED = 0xffffffffu; + + private static uint[]? _defaultTable; public Crc32Stream(Stream stream) - : this(stream, DefaultPolynomial, DefaultSeed) { } + : this(stream, DEFAULT_POLYNOMIAL, DEFAULT_SEED) { } public Crc32Stream(Stream stream, uint polynomial, uint seed) { this.stream = stream; - table = InitializeTable(polynomial); - hash = seed; + _table = InitializeTable(polynomial); + this.seed = seed; } public Stream WrappedStream => stream; public override void Flush() => stream.Flush(); + public override async Task FlushAsync(CancellationToken cancellationToken) => + await stream.FlushAsync(cancellationToken).ConfigureAwait(false); + public override int Read(byte[] buffer, int offset, int count) => throw new NotSupportedException(); + public override Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) => throw new NotSupportedException(); + +#if !LEGACY_DOTNET + public override ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) => throw new NotSupportedException(); +#endif + public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); public override void SetLength(long value) => throw new NotSupportedException(); -#if !NETFRAMEWORK && !NETSTANDARD2_0 +#if !LEGACY_DOTNET public override void Write(ReadOnlySpan buffer) { stream.Write(buffer); - hash = CalculateCrc(table, hash, buffer); + seed = CalculateCrc(_table, seed, buffer); } #endif public override void Write(byte[] buffer, int offset, int count) { stream.Write(buffer, offset, count); - hash = CalculateCrc(table, hash, buffer.AsSpan(offset, count)); + seed = CalculateCrc(_table, seed, buffer.AsSpan(offset, count)); } + public override async Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + await stream.WriteAsync(buffer, offset, count, cancellationToken).ConfigureAwait(false); + seed = CalculateCrc(_table, seed, buffer.AsSpan(offset, count)); + } + +#if !LEGACY_DOTNET + public override async ValueTask WriteAsync( + ReadOnlyMemory buffer, + CancellationToken cancellationToken = default + ) + { + await stream.WriteAsync(buffer, cancellationToken).ConfigureAwait(false); + seed = CalculateCrc(_table, seed, buffer.Span); + } +#endif + public override void WriteByte(byte value) { stream.WriteByte(value); - hash = CalculateCrc(table, hash, value); + seed = CalculateCrc(_table, seed, value); } public override bool CanRead => stream.CanRead; @@ -71,21 +109,21 @@ public sealed class Crc32Stream : Stream set => throw new NotSupportedException(); } - public uint Crc => ~hash; + public uint Crc => ~seed; - public static uint Compute(byte[] buffer) => Compute(DefaultSeed, buffer); + public static uint Compute(byte[] buffer) => Compute(DEFAULT_SEED, buffer); public static uint Compute(uint seed, byte[] buffer) => - Compute(DefaultPolynomial, seed, buffer); + Compute(DEFAULT_POLYNOMIAL, seed, buffer); public static uint Compute(uint polynomial, uint seed, ReadOnlySpan buffer) => ~CalculateCrc(InitializeTable(polynomial), seed, buffer); - private static uint[] InitializeTable(uint polynomial) + internal static uint[] InitializeTable(uint polynomial) { - if (polynomial == DefaultPolynomial && defaultTable != null) + if (polynomial == DEFAULT_POLYNOMIAL && _defaultTable != null) { - return defaultTable; + return _defaultTable; } var createTable = new uint[256]; @@ -107,15 +145,15 @@ public sealed class Crc32Stream : Stream createTable[i] = entry; } - if (polynomial == DefaultPolynomial) + if (polynomial == DEFAULT_POLYNOMIAL) { - defaultTable = createTable; + _defaultTable = createTable; } return createTable; } - private static uint CalculateCrc(uint[] table, uint crc, ReadOnlySpan buffer) + internal static uint CalculateCrc(uint[] table, uint crc, ReadOnlySpan buffer) { unchecked { @@ -127,6 +165,6 @@ public sealed class Crc32Stream : Stream return crc; } - private static uint CalculateCrc(uint[] table, uint crc, byte b) => + internal static uint CalculateCrc(uint[] table, uint crc, byte b) => (crc >> 8) ^ table[(crc ^ b) & 0xFF]; } diff --git a/src/SharpCompress/Crypto/CryptoException.cs b/src/SharpCompress/Crypto/CryptoException.cs deleted file mode 100644 index 234fd5e5..00000000 --- a/src/SharpCompress/Crypto/CryptoException.cs +++ /dev/null @@ -1,14 +0,0 @@ -using System; - -namespace SharpCompress.Crypto; - -public class CryptoException : Exception -{ - public CryptoException() { } - - public CryptoException(string message) - : base(message) { } - - public CryptoException(string message, Exception exception) - : base(message, exception) { } -} diff --git a/src/SharpCompress/Crypto/DataLengthException.cs b/src/SharpCompress/Crypto/DataLengthException.cs deleted file mode 100644 index 828c1a8f..00000000 --- a/src/SharpCompress/Crypto/DataLengthException.cs +++ /dev/null @@ -1,24 +0,0 @@ -using System; - -namespace SharpCompress.Crypto; - -public class DataLengthException : CryptoException -{ - /** - * base constructor. - */ - - public DataLengthException() { } - - /** - * create a DataLengthException with the given message. - * - * @param message the message to be carried with the exception. - */ - - public DataLengthException(string message) - : base(message) { } - - public DataLengthException(string message, Exception exception) - : base(message, exception) { } -} diff --git a/src/SharpCompress/Crypto/IBlockCipher.cs b/src/SharpCompress/Crypto/IBlockCipher.cs deleted file mode 100644 index 50eb5fb3..00000000 --- a/src/SharpCompress/Crypto/IBlockCipher.cs +++ /dev/null @@ -1,33 +0,0 @@ -using System; - -namespace SharpCompress.Crypto; - -/// Base interface for a symmetric key block cipher. -public interface IBlockCipher -{ - /// The name of the algorithm this cipher implements. - string AlgorithmName { get; } - - /// Initialise the cipher. - /// Initialise for encryption if true, for decryption if false. - /// The key or other data required by the cipher. - void Init(bool forEncryption, ICipherParameters parameters); - - /// The block size for this cipher, in bytes. - int GetBlockSize(); - - /// Indicates whether this cipher can handle partial blocks. - bool IsPartialBlockOkay { get; } - - /// Process a block. - /// The input buffer. - /// The output buffer. - /// If input block is wrong size, or outBuf too small. - /// The number of bytes processed and produced. - int ProcessBlock(ReadOnlySpan inBuf, Span outBuf); - - /// - /// Reset the cipher to the same state as it was after the last init (if there was one). - /// - void Reset(); -} diff --git a/src/SharpCompress/Crypto/ICipherParameters.cs b/src/SharpCompress/Crypto/ICipherParameters.cs deleted file mode 100644 index 9934637a..00000000 --- a/src/SharpCompress/Crypto/ICipherParameters.cs +++ /dev/null @@ -1,3 +0,0 @@ -namespace SharpCompress.Crypto; - -public interface ICipherParameters { } diff --git a/src/SharpCompress/Crypto/KeyParameter.cs b/src/SharpCompress/Crypto/KeyParameter.cs deleted file mode 100644 index fce334f9..00000000 --- a/src/SharpCompress/Crypto/KeyParameter.cs +++ /dev/null @@ -1,39 +0,0 @@ -using System; - -namespace SharpCompress.Crypto; - -public class KeyParameter : ICipherParameters -{ - private readonly byte[] key; - - public KeyParameter(byte[] key) - { - if (key is null) - { - throw new ArgumentNullException(nameof(key)); - } - - this.key = (byte[])key.Clone(); - } - - public KeyParameter(byte[] key, int keyOff, int keyLen) - { - if (key is null) - { - throw new ArgumentNullException(nameof(key)); - } - if (keyOff < 0 || keyOff > key.Length) - { - throw new ArgumentOutOfRangeException(nameof(keyOff)); - } - if (keyLen < 0 || (keyOff + keyLen) > key.Length) - { - throw new ArgumentOutOfRangeException(nameof(keyLen)); - } - - this.key = new byte[keyLen]; - Array.Copy(key, keyOff, this.key, 0, keyLen); - } - - public byte[] GetKey() => (byte[])key.Clone(); -} diff --git a/src/SharpCompress/Crypto/RijndaelEngine.cs b/src/SharpCompress/Crypto/RijndaelEngine.cs deleted file mode 100644 index 98dd5bc1..00000000 --- a/src/SharpCompress/Crypto/RijndaelEngine.cs +++ /dev/null @@ -1,1909 +0,0 @@ -using System; - -namespace SharpCompress.Crypto; - -public sealed class RijndaelEngine : IBlockCipher -{ - private const int MAXROUNDS = 14; - - private const int MAXKC = (256 / 4); - - private static ReadOnlySpan Logtable => - new byte[] - { - 0, - 0, - 25, - 1, - 50, - 2, - 26, - 198, - 75, - 199, - 27, - 104, - 51, - 238, - 223, - 3, - 100, - 4, - 224, - 14, - 52, - 141, - 129, - 239, - 76, - 113, - 8, - 200, - 248, - 105, - 28, - 193, - 125, - 194, - 29, - 181, - 249, - 185, - 39, - 106, - 77, - 228, - 166, - 114, - 154, - 201, - 9, - 120, - 101, - 47, - 138, - 5, - 33, - 15, - 225, - 36, - 18, - 240, - 130, - 69, - 53, - 147, - 218, - 142, - 150, - 143, - 219, - 189, - 54, - 208, - 206, - 148, - 19, - 92, - 210, - 241, - 64, - 70, - 131, - 56, - 102, - 221, - 253, - 48, - 191, - 6, - 139, - 98, - 179, - 37, - 226, - 152, - 34, - 136, - 145, - 16, - 126, - 110, - 72, - 195, - 163, - 182, - 30, - 66, - 58, - 107, - 40, - 84, - 250, - 133, - 61, - 186, - 43, - 121, - 10, - 21, - 155, - 159, - 94, - 202, - 78, - 212, - 172, - 229, - 243, - 115, - 167, - 87, - 175, - 88, - 168, - 80, - 244, - 234, - 214, - 116, - 79, - 174, - 233, - 213, - 231, - 230, - 173, - 232, - 44, - 215, - 117, - 122, - 235, - 22, - 11, - 245, - 89, - 203, - 95, - 176, - 156, - 169, - 81, - 160, - 127, - 12, - 246, - 111, - 23, - 196, - 73, - 236, - 216, - 67, - 31, - 45, - 164, - 118, - 123, - 183, - 204, - 187, - 62, - 90, - 251, - 96, - 177, - 134, - 59, - 82, - 161, - 108, - 170, - 85, - 41, - 157, - 151, - 178, - 135, - 144, - 97, - 190, - 220, - 252, - 188, - 149, - 207, - 205, - 55, - 63, - 91, - 209, - 83, - 57, - 132, - 60, - 65, - 162, - 109, - 71, - 20, - 42, - 158, - 93, - 86, - 242, - 211, - 171, - 68, - 17, - 146, - 217, - 35, - 32, - 46, - 137, - 180, - 124, - 184, - 38, - 119, - 153, - 227, - 165, - 103, - 74, - 237, - 222, - 197, - 49, - 254, - 24, - 13, - 99, - 140, - 128, - 192, - 247, - 112, - 7 - }; - - private static ReadOnlySpan Alogtable => - new byte[] - { - 0, - 3, - 5, - 15, - 17, - 51, - 85, - 255, - 26, - 46, - 114, - 150, - 161, - 248, - 19, - 53, - 95, - 225, - 56, - 72, - 216, - 115, - 149, - 164, - 247, - 2, - 6, - 10, - 30, - 34, - 102, - 170, - 229, - 52, - 92, - 228, - 55, - 89, - 235, - 38, - 106, - 190, - 217, - 112, - 144, - 171, - 230, - 49, - 83, - 245, - 4, - 12, - 20, - 60, - 68, - 204, - 79, - 209, - 104, - 184, - 211, - 110, - 178, - 205, - 76, - 212, - 103, - 169, - 224, - 59, - 77, - 215, - 98, - 166, - 241, - 8, - 24, - 40, - 120, - 136, - 131, - 158, - 185, - 208, - 107, - 189, - 220, - 127, - 129, - 152, - 179, - 206, - 73, - 219, - 118, - 154, - 181, - 196, - 87, - 249, - 16, - 48, - 80, - 240, - 11, - 29, - 39, - 105, - 187, - 214, - 97, - 163, - 254, - 25, - 43, - 125, - 135, - 146, - 173, - 236, - 47, - 113, - 147, - 174, - 233, - 32, - 96, - 160, - 251, - 22, - 58, - 78, - 210, - 109, - 183, - 194, - 93, - 231, - 50, - 86, - 250, - 21, - 63, - 65, - 195, - 94, - 226, - 61, - 71, - 201, - 64, - 192, - 91, - 237, - 44, - 116, - 156, - 191, - 218, - 117, - 159, - 186, - 213, - 100, - 172, - 239, - 42, - 126, - 130, - 157, - 188, - 223, - 122, - 142, - 137, - 128, - 155, - 182, - 193, - 88, - 232, - 35, - 101, - 175, - 234, - 37, - 111, - 177, - 200, - 67, - 197, - 84, - 252, - 31, - 33, - 99, - 165, - 244, - 7, - 9, - 27, - 45, - 119, - 153, - 176, - 203, - 70, - 202, - 69, - 207, - 74, - 222, - 121, - 139, - 134, - 145, - 168, - 227, - 62, - 66, - 198, - 81, - 243, - 14, - 18, - 54, - 90, - 238, - 41, - 123, - 141, - 140, - 143, - 138, - 133, - 148, - 167, - 242, - 13, - 23, - 57, - 75, - 221, - 124, - 132, - 151, - 162, - 253, - 28, - 36, - 108, - 180, - 199, - 82, - 246, - 1, - 3, - 5, - 15, - 17, - 51, - 85, - 255, - 26, - 46, - 114, - 150, - 161, - 248, - 19, - 53, - 95, - 225, - 56, - 72, - 216, - 115, - 149, - 164, - 247, - 2, - 6, - 10, - 30, - 34, - 102, - 170, - 229, - 52, - 92, - 228, - 55, - 89, - 235, - 38, - 106, - 190, - 217, - 112, - 144, - 171, - 230, - 49, - 83, - 245, - 4, - 12, - 20, - 60, - 68, - 204, - 79, - 209, - 104, - 184, - 211, - 110, - 178, - 205, - 76, - 212, - 103, - 169, - 224, - 59, - 77, - 215, - 98, - 166, - 241, - 8, - 24, - 40, - 120, - 136, - 131, - 158, - 185, - 208, - 107, - 189, - 220, - 127, - 129, - 152, - 179, - 206, - 73, - 219, - 118, - 154, - 181, - 196, - 87, - 249, - 16, - 48, - 80, - 240, - 11, - 29, - 39, - 105, - 187, - 214, - 97, - 163, - 254, - 25, - 43, - 125, - 135, - 146, - 173, - 236, - 47, - 113, - 147, - 174, - 233, - 32, - 96, - 160, - 251, - 22, - 58, - 78, - 210, - 109, - 183, - 194, - 93, - 231, - 50, - 86, - 250, - 21, - 63, - 65, - 195, - 94, - 226, - 61, - 71, - 201, - 64, - 192, - 91, - 237, - 44, - 116, - 156, - 191, - 218, - 117, - 159, - 186, - 213, - 100, - 172, - 239, - 42, - 126, - 130, - 157, - 188, - 223, - 122, - 142, - 137, - 128, - 155, - 182, - 193, - 88, - 232, - 35, - 101, - 175, - 234, - 37, - 111, - 177, - 200, - 67, - 197, - 84, - 252, - 31, - 33, - 99, - 165, - 244, - 7, - 9, - 27, - 45, - 119, - 153, - 176, - 203, - 70, - 202, - 69, - 207, - 74, - 222, - 121, - 139, - 134, - 145, - 168, - 227, - 62, - 66, - 198, - 81, - 243, - 14, - 18, - 54, - 90, - 238, - 41, - 123, - 141, - 140, - 143, - 138, - 133, - 148, - 167, - 242, - 13, - 23, - 57, - 75, - 221, - 124, - 132, - 151, - 162, - 253, - 28, - 36, - 108, - 180, - 199, - 82, - 246, - 1 - }; - - private static ReadOnlySpan S => - new byte[] - { - 99, - 124, - 119, - 123, - 242, - 107, - 111, - 197, - 48, - 1, - 103, - 43, - 254, - 215, - 171, - 118, - 202, - 130, - 201, - 125, - 250, - 89, - 71, - 240, - 173, - 212, - 162, - 175, - 156, - 164, - 114, - 192, - 183, - 253, - 147, - 38, - 54, - 63, - 247, - 204, - 52, - 165, - 229, - 241, - 113, - 216, - 49, - 21, - 4, - 199, - 35, - 195, - 24, - 150, - 5, - 154, - 7, - 18, - 128, - 226, - 235, - 39, - 178, - 117, - 9, - 131, - 44, - 26, - 27, - 110, - 90, - 160, - 82, - 59, - 214, - 179, - 41, - 227, - 47, - 132, - 83, - 209, - 0, - 237, - 32, - 252, - 177, - 91, - 106, - 203, - 190, - 57, - 74, - 76, - 88, - 207, - 208, - 239, - 170, - 251, - 67, - 77, - 51, - 133, - 69, - 249, - 2, - 127, - 80, - 60, - 159, - 168, - 81, - 163, - 64, - 143, - 146, - 157, - 56, - 245, - 188, - 182, - 218, - 33, - 16, - 255, - 243, - 210, - 205, - 12, - 19, - 236, - 95, - 151, - 68, - 23, - 196, - 167, - 126, - 61, - 100, - 93, - 25, - 115, - 96, - 129, - 79, - 220, - 34, - 42, - 144, - 136, - 70, - 238, - 184, - 20, - 222, - 94, - 11, - 219, - 224, - 50, - 58, - 10, - 73, - 6, - 36, - 92, - 194, - 211, - 172, - 98, - 145, - 149, - 228, - 121, - 231, - 200, - 55, - 109, - 141, - 213, - 78, - 169, - 108, - 86, - 244, - 234, - 101, - 122, - 174, - 8, - 186, - 120, - 37, - 46, - 28, - 166, - 180, - 198, - 232, - 221, - 116, - 31, - 75, - 189, - 139, - 138, - 112, - 62, - 181, - 102, - 72, - 3, - 246, - 14, - 97, - 53, - 87, - 185, - 134, - 193, - 29, - 158, - 225, - 248, - 152, - 17, - 105, - 217, - 142, - 148, - 155, - 30, - 135, - 233, - 206, - 85, - 40, - 223, - 140, - 161, - 137, - 13, - 191, - 230, - 66, - 104, - 65, - 153, - 45, - 15, - 176, - 84, - 187, - 22 - }; - - private static ReadOnlySpan Si => - new byte[] - { - 82, - 9, - 106, - 213, - 48, - 54, - 165, - 56, - 191, - 64, - 163, - 158, - 129, - 243, - 215, - 251, - 124, - 227, - 57, - 130, - 155, - 47, - 255, - 135, - 52, - 142, - 67, - 68, - 196, - 222, - 233, - 203, - 84, - 123, - 148, - 50, - 166, - 194, - 35, - 61, - 238, - 76, - 149, - 11, - 66, - 250, - 195, - 78, - 8, - 46, - 161, - 102, - 40, - 217, - 36, - 178, - 118, - 91, - 162, - 73, - 109, - 139, - 209, - 37, - 114, - 248, - 246, - 100, - 134, - 104, - 152, - 22, - 212, - 164, - 92, - 204, - 93, - 101, - 182, - 146, - 108, - 112, - 72, - 80, - 253, - 237, - 185, - 218, - 94, - 21, - 70, - 87, - 167, - 141, - 157, - 132, - 144, - 216, - 171, - 0, - 140, - 188, - 211, - 10, - 247, - 228, - 88, - 5, - 184, - 179, - 69, - 6, - 208, - 44, - 30, - 143, - 202, - 63, - 15, - 2, - 193, - 175, - 189, - 3, - 1, - 19, - 138, - 107, - 58, - 145, - 17, - 65, - 79, - 103, - 220, - 234, - 151, - 242, - 207, - 206, - 240, - 180, - 230, - 115, - 150, - 172, - 116, - 34, - 231, - 173, - 53, - 133, - 226, - 249, - 55, - 232, - 28, - 117, - 223, - 110, - 71, - 241, - 26, - 113, - 29, - 41, - 197, - 137, - 111, - 183, - 98, - 14, - 170, - 24, - 190, - 27, - 252, - 86, - 62, - 75, - 198, - 210, - 121, - 32, - 154, - 219, - 192, - 254, - 120, - 205, - 90, - 244, - 31, - 221, - 168, - 51, - 136, - 7, - 199, - 49, - 177, - 18, - 16, - 89, - 39, - 128, - 236, - 95, - 96, - 81, - 127, - 169, - 25, - 181, - 74, - 13, - 45, - 229, - 122, - 159, - 147, - 201, - 156, - 239, - 160, - 224, - 59, - 77, - 174, - 42, - 245, - 176, - 200, - 235, - 187, - 60, - 131, - 83, - 153, - 97, - 23, - 43, - 4, - 126, - 186, - 119, - 214, - 38, - 225, - 105, - 20, - 99, - 85, - 33, - 12, - 125 - }; - - private static ReadOnlySpan rcon => - new byte[] - { - 0x01, - 0x02, - 0x04, - 0x08, - 0x10, - 0x20, - 0x40, - 0x80, - 0x1b, - 0x36, - 0x6c, - 0xd8, - 0xab, - 0x4d, - 0x9a, - 0x2f, - 0x5e, - 0xbc, - 0x63, - 0xc6, - 0x97, - 0x35, - 0x6a, - 0xd4, - 0xb3, - 0x7d, - 0xfa, - 0xef, - 0xc5, - 0x91 - }; - - private static readonly byte[][] shifts0 = - { - new byte[] { 0, 8, 16, 24 }, - new byte[] { 0, 8, 16, 24 }, - new byte[] { 0, 8, 16, 24 }, - new byte[] { 0, 8, 16, 32 }, - new byte[] { 0, 8, 24, 32 } - }; - - private static readonly byte[][] shifts1 = - { - new byte[] { 0, 24, 16, 8 }, - new byte[] { 0, 32, 24, 16 }, - new byte[] { 0, 40, 32, 24 }, - new byte[] { 0, 48, 40, 24 }, - new byte[] { 0, 56, 40, 32 } - }; - - /** - * multiply two elements of GF(2^m) - * needed for MixColumn and InvMixColumn - */ - - private byte Mul0x2(int b) - { - if (b != 0) - { - return Alogtable[25 + (Logtable[b] & 0xff)]; - } - return 0; - } - - private byte Mul0x3(int b) - { - if (b != 0) - { - return Alogtable[1 + (Logtable[b] & 0xff)]; - } - return 0; - } - - private byte Mul0x9(int b) - { - if (b >= 0) - { - return Alogtable[199 + b]; - } - return 0; - } - - private byte Mul0xb(int b) - { - if (b >= 0) - { - return Alogtable[104 + b]; - } - return 0; - } - - private byte Mul0xd(int b) - { - if (b >= 0) - { - return Alogtable[238 + b]; - } - return 0; - } - - private byte Mul0xe(int b) - { - if (b >= 0) - { - return Alogtable[223 + b]; - } - return 0; - } - - /** - * xor corresponding text input and round key input bytes - */ - - private void KeyAddition(long[] rk) - { - A0 ^= rk[0]; - A1 ^= rk[1]; - A2 ^= rk[2]; - A3 ^= rk[3]; - } - - private long Shift(long r, int shift) - { - //return (((long)((ulong) r >> shift) | (r << (BC - shift)))) & BC_MASK; - - var temp = (ulong)r >> shift; - - // NB: This corrects for Mono Bug #79087 (fixed in 1.1.17) - if (shift > 31) - { - temp &= 0xFFFFFFFFUL; - } - - return ((long)temp | (r << (BC - shift))) & BC_MASK; - } - - /** - * Row 0 remains unchanged - * The other three rows are shifted a variable amount - */ - - private void ShiftRow(byte[] shiftsSC) - { - A1 = Shift(A1, shiftsSC[1]); - A2 = Shift(A2, shiftsSC[2]); - A3 = Shift(A3, shiftsSC[3]); - } - - private long ApplyS(long r, ReadOnlySpan box) - { - long res = 0; - - for (var j = 0; j < BC; j += 8) - { - res |= (long)(box[(int)((r >> j) & 0xff)] & 0xff) << j; - } - - return res; - } - - /** - * Replace every byte of the input by the byte at that place - * in the nonlinear S-box - */ - - private void Substitution(ReadOnlySpan box) - { - A0 = ApplyS(A0, box); - A1 = ApplyS(A1, box); - A2 = ApplyS(A2, box); - A3 = ApplyS(A3, box); - } - - /** - * Mix the bytes of every column in a linear way - */ - - private void MixColumn() - { - long r0, - r1, - r2, - r3; - - r0 = r1 = r2 = r3 = 0; - - for (var j = 0; j < BC; j += 8) - { - var a0 = (int)((A0 >> j) & 0xff); - var a1 = (int)((A1 >> j) & 0xff); - var a2 = (int)((A2 >> j) & 0xff); - var a3 = (int)((A3 >> j) & 0xff); - - r0 |= (long)((Mul0x2(a0) ^ Mul0x3(a1) ^ a2 ^ a3) & 0xff) << j; - - r1 |= (long)((Mul0x2(a1) ^ Mul0x3(a2) ^ a3 ^ a0) & 0xff) << j; - - r2 |= (long)((Mul0x2(a2) ^ Mul0x3(a3) ^ a0 ^ a1) & 0xff) << j; - - r3 |= (long)((Mul0x2(a3) ^ Mul0x3(a0) ^ a1 ^ a2) & 0xff) << j; - } - - A0 = r0; - A1 = r1; - A2 = r2; - A3 = r3; - } - - /** - * Mix the bytes of every column in a linear way - * This is the opposite operation of Mixcolumn - */ - - private void InvMixColumn() - { - long r0, - r1, - r2, - r3; - - r0 = r1 = r2 = r3 = 0; - for (var j = 0; j < BC; j += 8) - { - var a0 = (int)((A0 >> j) & 0xff); - var a1 = (int)((A1 >> j) & 0xff); - var a2 = (int)((A2 >> j) & 0xff); - var a3 = (int)((A3 >> j) & 0xff); - - // - // pre-lookup the log table - // - a0 = (a0 != 0) ? (Logtable[a0 & 0xff] & 0xff) : -1; - a1 = (a1 != 0) ? (Logtable[a1 & 0xff] & 0xff) : -1; - a2 = (a2 != 0) ? (Logtable[a2 & 0xff] & 0xff) : -1; - a3 = (a3 != 0) ? (Logtable[a3 & 0xff] & 0xff) : -1; - - r0 |= (long)((Mul0xe(a0) ^ Mul0xb(a1) ^ Mul0xd(a2) ^ Mul0x9(a3)) & 0xff) << j; - - r1 |= (long)((Mul0xe(a1) ^ Mul0xb(a2) ^ Mul0xd(a3) ^ Mul0x9(a0)) & 0xff) << j; - - r2 |= (long)((Mul0xe(a2) ^ Mul0xb(a3) ^ Mul0xd(a0) ^ Mul0x9(a1)) & 0xff) << j; - - r3 |= (long)((Mul0xe(a3) ^ Mul0xb(a0) ^ Mul0xd(a1) ^ Mul0x9(a2)) & 0xff) << j; - } - - A0 = r0; - A1 = r1; - A2 = r2; - A3 = r3; - } - - /** - * Calculate the necessary round keys - * The number of calculations depends on keyBits and blockBits - */ - - private long[][] GenerateWorkingKey(byte[] key) - { - int t, - rconpointer = 0; - var keyBits = key.Length * 8; - var tk = new byte[4, MAXKC]; - - //long[,] W = new long[MAXROUNDS+1,4]; - var W = new long[MAXROUNDS + 1][]; - - for (var i = 0; i < MAXROUNDS + 1; i++) - { - W[i] = new long[4]; - } - - var KC = keyBits switch - { - 128 => 4, - 160 => 5, - 192 => 6, - 224 => 7, - 256 => 8, - _ => throw new ArgumentException("Key length not 128/160/192/224/256 bits."), - }; - if (keyBits >= blockBits) - { - ROUNDS = KC + 6; - } - else - { - ROUNDS = (BC / 8) + 6; - } - - // - // copy the key into the processing area - // - var index = 0; - - for (var i = 0; i < key.Length; i++) - { - tk[i % 4, i / 4] = key[index++]; - } - - t = 0; - - // - // copy values into round key array - // - for (var j = 0; (j < KC) && (t < (ROUNDS + 1) * (BC / 8)); j++, t++) - { - for (var i = 0; i < 4; i++) - { - W[t / (BC / 8)][i] |= (long)(tk[i, j] & 0xff) << ((t * 8) % BC); - } - } - - // - // while not enough round key material calculated - // calculate new values - // - while (t < (ROUNDS + 1) * (BC / 8)) - { - for (var i = 0; i < 4; i++) - { - tk[i, 0] ^= S[tk[(i + 1) % 4, KC - 1] & 0xff]; - } - tk[0, 0] ^= rcon[rconpointer++]; - - if (KC <= 6) - { - for (var j = 1; j < KC; j++) - { - for (var i = 0; i < 4; i++) - { - tk[i, j] ^= tk[i, j - 1]; - } - } - } - else - { - for (var j = 1; j < 4; j++) - { - for (var i = 0; i < 4; i++) - { - tk[i, j] ^= tk[i, j - 1]; - } - } - for (var i = 0; i < 4; i++) - { - tk[i, 4] ^= S[tk[i, 3] & 0xff]; - } - for (var j = 5; j < KC; j++) - { - for (var i = 0; i < 4; i++) - { - tk[i, j] ^= tk[i, j - 1]; - } - } - } - - // - // copy values into round key array - // - for (var j = 0; (j < KC) && (t < (ROUNDS + 1) * (BC / 8)); j++, t++) - { - for (var i = 0; i < 4; i++) - { - W[t / (BC / 8)][i] |= (long)(tk[i, j] & 0xff) << ((t * 8) % (BC)); - } - } - } - return W; - } - - private readonly int BC; - private readonly long BC_MASK; - private int ROUNDS; - private readonly int blockBits; - private long[][]? workingKey; - private long A0, - A1, - A2, - A3; - private bool forEncryption; - private readonly byte[] shifts0SC; - private readonly byte[] shifts1SC; - - /** - * default constructor - 128 bit block size. - */ - - public RijndaelEngine() - : this(128) { } - - /** - * basic constructor - set the cipher up for a given blocksize - * - * @param blocksize the blocksize in bits, must be 128, 192, or 256. - */ - - public RijndaelEngine(int blockBits) - { - switch (blockBits) - { - case 128: - BC = 32; - BC_MASK = 0xffffffffL; - shifts0SC = shifts0[0]; - shifts1SC = shifts1[0]; - break; - case 160: - BC = 40; - BC_MASK = 0xffffffffffL; - shifts0SC = shifts0[1]; - shifts1SC = shifts1[1]; - break; - case 192: - BC = 48; - BC_MASK = 0xffffffffffffL; - shifts0SC = shifts0[2]; - shifts1SC = shifts1[2]; - break; - case 224: - BC = 56; - BC_MASK = 0xffffffffffffffL; - shifts0SC = shifts0[3]; - shifts1SC = shifts1[3]; - break; - case 256: - BC = 64; - BC_MASK = unchecked((long)0xffffffffffffffffL); - shifts0SC = shifts0[4]; - shifts1SC = shifts1[4]; - break; - default: - throw new ArgumentException("unknown blocksize to Rijndael"); - } - - this.blockBits = blockBits; - } - - /** - * initialise a Rijndael cipher. - * - * @param forEncryption whether or not we are for encryption. - * @param parameters the parameters required to set up the cipher. - * @exception ArgumentException if the parameters argument is - * inappropriate. - */ - - public void Init(bool forEncryption, ICipherParameters parameters) - { - if (parameters is KeyParameter parameter) - { - workingKey = GenerateWorkingKey(parameter.GetKey()); - this.forEncryption = forEncryption; - return; - } - - throw new ArgumentException( - "invalid parameter passed to Rijndael init - " + parameters.GetType() - ); - } - - public string AlgorithmName => "Rijndael"; - - public bool IsPartialBlockOkay => false; - - public int GetBlockSize() => BC / 2; - - public int ProcessBlock(ReadOnlySpan input, Span output) - { - if (workingKey is null) - { - throw new InvalidOperationException("Rijndael engine not initialised"); - } - - if (BC / 2 > input.Length) - { - throw new DataLengthException("input buffer too short"); - } - - if (BC / 2 > output.Length) - { - throw new DataLengthException("output buffer too short"); - } - - UnPackBlock(input); - - if (forEncryption) - { - EncryptBlock(workingKey); - } - else - { - DecryptBlock(workingKey); - } - - PackBlock(output); - - return BC / 2; - } - - public void Reset() { } - - private void UnPackBlock(ReadOnlySpan bytes) - { - var index = 0; - - A0 = bytes[index++] & 0xff; - A1 = bytes[index++] & 0xff; - A2 = bytes[index++] & 0xff; - A3 = bytes[index++] & 0xff; - - for (var j = 8; j != BC; j += 8) - { - A0 |= (long)(bytes[index++] & 0xff) << j; - A1 |= (long)(bytes[index++] & 0xff) << j; - A2 |= (long)(bytes[index++] & 0xff) << j; - A3 |= (long)(bytes[index++] & 0xff) << j; - } - } - - private void PackBlock(Span bytes) - { - var index = 0; - - for (var j = 0; j != BC; j += 8) - { - bytes[index++] = (byte)(A0 >> j); - bytes[index++] = (byte)(A1 >> j); - bytes[index++] = (byte)(A2 >> j); - bytes[index++] = (byte)(A3 >> j); - } - } - - private void EncryptBlock(long[][] rk) - { - int r; - - // - // begin with a key addition - // - KeyAddition(rk[0]); - - // - // ROUNDS-1 ordinary rounds - // - for (r = 1; r < ROUNDS; r++) - { - Substitution(S); - ShiftRow(shifts0SC); - MixColumn(); - KeyAddition(rk[r]); - } - - // - // Last round is special: there is no MixColumn - // - Substitution(S); - ShiftRow(shifts0SC); - KeyAddition(rk[ROUNDS]); - } - - private void DecryptBlock(long[][] rk) - { - int r; - - // To decrypt: apply the inverse operations of the encrypt routine, - // in opposite order - // - // (KeyAddition is an involution: it 's equal to its inverse) - // (the inverse of Substitution with table S is Substitution with the inverse table of S) - // (the inverse of Shiftrow is Shiftrow over a suitable distance) - // - - // First the special round: - // without InvMixColumn - // with extra KeyAddition - // - KeyAddition(rk[ROUNDS]); - Substitution(Si); - ShiftRow(shifts1SC); - - // - // ROUNDS-1 ordinary rounds - // - for (r = ROUNDS - 1; r > 0; r--) - { - KeyAddition(rk[r]); - InvMixColumn(); - Substitution(Si); - ShiftRow(shifts1SC); - } - - // - // End with the extra key addition - // - KeyAddition(rk[0]); - } -} diff --git a/src/SharpCompress/Factories/AceFactory.cs b/src/SharpCompress/Factories/AceFactory.cs new file mode 100644 index 00000000..5117045e --- /dev/null +++ b/src/SharpCompress/Factories/AceFactory.cs @@ -0,0 +1,47 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Ace.Headers; +using SharpCompress.Readers; +using SharpCompress.Readers.Ace; + +namespace SharpCompress.Factories; + +public class AceFactory : Factory, IReaderFactory +{ + public override string Name => "Ace"; + + public override ArchiveType? KnownArchiveType => ArchiveType.Ace; + + public override IEnumerable GetSupportedExtensions() + { + yield return "ace"; + } + + public override bool IsArchive(Stream stream, ReaderOptions readerOptions) => + AceHeader.IsArchive(stream); + + public override ValueTask IsArchiveAsync( + Stream stream, + ReaderOptions readerOptions, + CancellationToken cancellationToken = default + ) => AceHeader.IsArchiveAsync(stream, cancellationToken); + + public IReader OpenReader(Stream stream, ReaderOptions? options) => + AceReader.OpenReader(stream, options); + + public ValueTask OpenAsyncReader( + Stream stream, + ReaderOptions? options, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncReader)AceReader.OpenReader(stream, options)); + } +} diff --git a/src/SharpCompress/Factories/ArcFactory.cs b/src/SharpCompress/Factories/ArcFactory.cs new file mode 100644 index 00000000..cbc95e01 --- /dev/null +++ b/src/SharpCompress/Factories/ArcFactory.cs @@ -0,0 +1,87 @@ +using System; +using System.Buffers; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Security.Cryptography; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Readers; +using SharpCompress.Readers.Arc; +using static System.Net.Mime.MediaTypeNames; + +namespace SharpCompress.Factories; + +public class ArcFactory : Factory, IReaderFactory +{ + public override string Name => "Arc"; + + public override ArchiveType? KnownArchiveType => ArchiveType.Arc; + + public override IEnumerable GetSupportedExtensions() + { + yield return "arc"; + } + + public override bool IsArchive(Stream stream, ReaderOptions readerOptions) + { + //You may have to use some(paranoid) checks to ensure that you actually are + //processing an ARC file, since other archivers also adopted the idea of putting + //a 01Ah byte at offset 0, namely the Hyper archiver. To check if you have a + //Hyper - archive, check the next two bytes for "HP" or "ST"(or look below for + //"HYP").Also the ZOO archiver also does put a 01Ah at the start of the file, + //see the ZOO entry below. + var buffer = ArrayPool.Shared.Rent(2); + try + { + if (stream.ReadFully(buffer.AsSpan(0, 2))) + { + return buffer[0] == 0x1A && buffer[1] < 10; //rather thin, but this is all we have + } + return false; + } + finally + { + ArrayPool.Shared.Return(buffer); + } + } + + public IReader OpenReader(Stream stream, ReaderOptions? options) => + ArcReader.OpenReader(stream, options); + + public ValueTask OpenAsyncReader( + Stream stream, + ReaderOptions? options, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncReader)ArcReader.OpenReader(stream, options)); + } + + public override async ValueTask IsArchiveAsync( + Stream stream, + ReaderOptions readerOptions, + CancellationToken cancellationToken = default + ) + { + //You may have to use some(paranoid) checks to ensure that you actually are + //processing an ARC file, since other archivers also adopted the idea of putting + //a 01Ah byte at offset 0, namely the Hyper archiver. To check if you have a + //Hyper - archive, check the next two bytes for "HP" or "ST"(or look below for + //"HYP").Also the ZOO archiver also does put a 01Ah at the start of the file, + //see the ZOO entry below. + var buffer = ArrayPool.Shared.Rent(2); + try + { + await stream.ReadExactAsync(buffer, 0, 2, cancellationToken).ConfigureAwait(false); + return buffer[0] == 0x1A && buffer[1] < 10; //rather thin, but this is all we have + } + finally + { + ArrayPool.Shared.Return(buffer); + } + } +} diff --git a/src/SharpCompress/Factories/ArjFactory.cs b/src/SharpCompress/Factories/ArjFactory.cs new file mode 100644 index 00000000..de1a8382 --- /dev/null +++ b/src/SharpCompress/Factories/ArjFactory.cs @@ -0,0 +1,47 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Arj.Headers; +using SharpCompress.Readers; +using SharpCompress.Readers.Arj; + +namespace SharpCompress.Factories; + +public class ArjFactory : Factory, IReaderFactory +{ + public override string Name => "Arj"; + + public override ArchiveType? KnownArchiveType => ArchiveType.Arj; + + public override IEnumerable GetSupportedExtensions() + { + yield return "arj"; + } + + public override bool IsArchive(Stream stream, ReaderOptions readerOptions) => + ArjHeader.IsArchive(stream); + + public override ValueTask IsArchiveAsync( + Stream stream, + ReaderOptions readerOptions, + CancellationToken cancellationToken = default + ) => ArjHeader.IsArchiveAsync(stream, cancellationToken); + + public IReader OpenReader(Stream stream, ReaderOptions? options) => + ArjReader.OpenReader(stream, options); + + public ValueTask OpenAsyncReader( + Stream stream, + ReaderOptions? options, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncReader)ArjReader.OpenReader(stream, options)); + } +} diff --git a/src/SharpCompress/Factories/Factory.cs b/src/SharpCompress/Factories/Factory.cs index 7ec68508..115c6e49 100644 --- a/src/SharpCompress/Factories/Factory.cs +++ b/src/SharpCompress/Factories/Factory.cs @@ -1,6 +1,8 @@ using System; using System.Collections.Generic; using System.IO; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common; using SharpCompress.IO; using SharpCompress.Readers; @@ -14,12 +16,16 @@ public abstract class Factory : IFactory { RegisterFactory(new ZipFactory()); RegisterFactory(new RarFactory()); - RegisterFactory(new SevenZipFactory()); + RegisterFactory(new TarFactory()); //put tar before most RegisterFactory(new GZipFactory()); - RegisterFactory(new TarFactory()); + RegisterFactory(new LzwFactory()); + RegisterFactory(new ArcFactory()); + RegisterFactory(new ArjFactory()); + RegisterFactory(new AceFactory()); + RegisterFactory(new SevenZipFactory()); } - private static readonly HashSet _factories = new HashSet(); + private static readonly HashSet _factories = new(); /// /// Gets the collection of registered . @@ -33,7 +39,7 @@ public abstract class Factory : IFactory /// must not be null. public static void RegisterFactory(Factory factory) { - factory.CheckNotNull(nameof(factory)); + factory.NotNull(nameof(factory)); _factories.Add(factory); } @@ -48,23 +54,28 @@ public abstract class Factory : IFactory public abstract IEnumerable GetSupportedExtensions(); /// - public abstract bool IsArchive(Stream stream, string? password = null); + public abstract bool IsArchive(Stream stream, ReaderOptions readerOptions); + public abstract ValueTask IsArchiveAsync( + Stream stream, + ReaderOptions readerOptions, + CancellationToken cancellationToken = default + ); /// public virtual FileInfo? GetFilePart(int index, FileInfo part1) => null; /// - /// Tries to open an from a . + /// Tries to open an from a . /// /// /// This method provides extra insight to support loading compressed TAR files. /// - /// + /// /// /// /// internal virtual bool TryOpenReader( - RewindableStream rewindableStream, + SharpCompressStream stream, ReaderOptions options, out IReader? reader ) @@ -73,15 +84,36 @@ public abstract class Factory : IFactory if (this is IReaderFactory readerFactory) { - rewindableStream.Rewind(false); - if (IsArchive(rewindableStream, options.Password)) + stream.Rewind(); + if (IsArchive(stream, options)) { - rewindableStream.Rewind(true); - reader = readerFactory.OpenReader(rewindableStream, options); + stream.Rewind(true); + reader = readerFactory.OpenReader(stream, options); return true; } } - + stream.Rewind(); return false; } + + internal virtual async ValueTask TryOpenReaderAsync( + SharpCompressStream stream, + ReaderOptions options, + CancellationToken cancellationToken = default + ) + { + if (this is IReaderFactory readerFactory) + { + stream.Rewind(); + if (await IsArchiveAsync(stream, options, cancellationToken).ConfigureAwait(false)) + { + stream.Rewind(true); + return await readerFactory + .OpenAsyncReader(stream, options, cancellationToken) + .ConfigureAwait(false); + } + } + stream.Rewind(); + return null; + } } diff --git a/src/SharpCompress/Factories/GZipFactory.cs b/src/SharpCompress/Factories/GZipFactory.cs index 5e522f5b..dd870d51 100644 --- a/src/SharpCompress/Factories/GZipFactory.cs +++ b/src/SharpCompress/Factories/GZipFactory.cs @@ -1,12 +1,15 @@ +using System; using System.Collections.Generic; using System.IO; -using System.IO.Compression; - +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Archives; using SharpCompress.Archives.GZip; using SharpCompress.Archives.Tar; using SharpCompress.Common; +using SharpCompress.Common.Options; using SharpCompress.IO; +using SharpCompress.Providers; using SharpCompress.Readers; using SharpCompress.Readers.GZip; using SharpCompress.Readers.Tar; @@ -24,7 +27,7 @@ public class GZipFactory IMultiArchiveFactory, IReaderFactory, IWriterFactory, - IWriteableArchiveFactory + IWritableArchiveFactory { #region IFactory @@ -41,32 +44,87 @@ public class GZipFactory } /// - public override bool IsArchive(Stream stream, string? password = null) => + public override bool IsArchive(Stream stream, ReaderOptions readerOptions) => GZipArchive.IsGZipFile(stream); + /// + public override ValueTask IsArchiveAsync( + Stream stream, + ReaderOptions readerOptions, + CancellationToken cancellationToken = default + ) => GZipArchive.IsGZipFileAsync(stream, cancellationToken); + #endregion #region IArchiveFactory /// - public IArchive Open(Stream stream, ReaderOptions? readerOptions = null) => - GZipArchive.Open(stream, readerOptions); + public IArchive OpenArchive(Stream stream, ReaderOptions? readerOptions = null) => + GZipArchive.OpenArchive(stream, readerOptions); /// - public IArchive Open(FileInfo fileInfo, ReaderOptions? readerOptions = null) => - GZipArchive.Open(fileInfo, readerOptions); + public ValueTask OpenAsyncArchive( + Stream stream, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncArchive)OpenArchive(stream, readerOptions)); + } + + /// + public ValueTask OpenAsyncArchive( + FileInfo fileInfo, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncArchive)OpenArchive(fileInfo, readerOptions)); + } #endregion #region IMultiArchiveFactory /// - public IArchive Open(IReadOnlyList streams, ReaderOptions? readerOptions = null) => - GZipArchive.Open(streams, readerOptions); + public IArchive OpenArchive( + IReadOnlyList streams, + ReaderOptions? readerOptions = null + ) => GZipArchive.OpenArchive(streams, readerOptions); /// - public IArchive Open(IReadOnlyList fileInfos, ReaderOptions? readerOptions = null) => - GZipArchive.Open(fileInfos, readerOptions); + public async ValueTask OpenAsyncArchive( + IReadOnlyList streams, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return await GZipArchive + .OpenAsyncArchive(streams, readerOptions, cancellationToken) + .ConfigureAwait(false); + } + + /// + public IArchive OpenArchive( + IReadOnlyList fileInfos, + ReaderOptions? readerOptions = null + ) => GZipArchive.OpenArchive(fileInfos, readerOptions); + + /// + public async ValueTask OpenAsyncArchive( + IReadOnlyList fileInfos, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return await GZipArchive + .OpenAsyncArchive(fileInfos, readerOptions, cancellationToken) + .ConfigureAwait(false); + } #endregion @@ -74,57 +132,96 @@ public class GZipFactory /// internal override bool TryOpenReader( - RewindableStream rewindableStream, + SharpCompressStream sharpCompressStream, ReaderOptions options, out IReader? reader ) { reader = null; - rewindableStream.Rewind(false); - if (GZipArchive.IsGZipFile(rewindableStream)) + if (GZipArchive.IsGZipFile(sharpCompressStream)) { - rewindableStream.Rewind(false); - var testStream = new GZipStream(rewindableStream, CompressionMode.Decompress); + sharpCompressStream.Rewind(); + using var testStream = options.Providers.CreateDecompressStream( + CompressionType.GZip, + SharpCompressStream.CreateNonDisposing(sharpCompressStream), + CompressionContext.FromStream(sharpCompressStream).WithReaderOptions(options) + ); if (TarArchive.IsTarFile(testStream)) { - rewindableStream.Rewind(true); - reader = new TarReader(rewindableStream, options, CompressionType.GZip); + sharpCompressStream.StopRecording(); + reader = new TarReader(sharpCompressStream, options, CompressionType.GZip); return true; } - - rewindableStream.Rewind(true); - reader = OpenReader(rewindableStream, options); + sharpCompressStream.StopRecording(); + reader = OpenReader(sharpCompressStream, options); return true; } - + sharpCompressStream.Rewind(); return false; } /// public IReader OpenReader(Stream stream, ReaderOptions? options) => - GZipReader.Open(stream, options); + GZipReader.OpenReader(stream, options); + + /// + public ValueTask OpenAsyncReader( + Stream stream, + ReaderOptions? options, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncReader)GZipReader.OpenReader(stream, options)); + } + + /// + public IArchive OpenArchive(FileInfo fileInfo, ReaderOptions? readerOptions = null) => + GZipArchive.OpenArchive(fileInfo, readerOptions); #endregion #region IWriterFactory /// - public IWriter Open(Stream stream, WriterOptions writerOptions) + public IWriter OpenWriter(Stream stream, IWriterOptions writerOptions) { if (writerOptions.CompressionType != CompressionType.GZip) { throw new InvalidFormatException("GZip archives only support GZip compression type."); } - return new GZipWriter(stream, new GZipWriterOptions(writerOptions)); + + GZipWriterOptions gzipOptions = writerOptions switch + { + GZipWriterOptions gwo => gwo, + WriterOptions wo => new GZipWriterOptions(wo), + _ => throw new ArgumentException( + $"Expected WriterOptions or GZipWriterOptions, got {writerOptions.GetType().Name}", + nameof(writerOptions) + ), + }; + return new GZipWriter(stream, gzipOptions); + } + + /// + public ValueTask OpenAsyncWriter( + Stream stream, + IWriterOptions writerOptions, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + var writer = OpenWriter(stream, writerOptions); + return new((IAsyncWriter)writer); } #endregion - #region IWriteableArchiveFactory + #region IWritableArchiveFactory /// - public IWritableArchive CreateWriteableArchive() => GZipArchive.Create(); + public IWritableArchive CreateArchive() => GZipArchive.CreateArchive(); #endregion } diff --git a/src/SharpCompress/Factories/IFactory.cs b/src/SharpCompress/Factories/IFactory.cs index 08bedd50..76bddb64 100644 --- a/src/SharpCompress/Factories/IFactory.cs +++ b/src/SharpCompress/Factories/IFactory.cs @@ -1,5 +1,8 @@ using System.Collections.Generic; using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Readers; namespace SharpCompress.Factories; @@ -34,8 +37,20 @@ public interface IFactory /// Returns true if the stream represents an archive of the format defined by this type. /// /// A stream, pointing to the beginning of the archive. - /// optional password - bool IsArchive(Stream stream, string? password = null); + /// Options controlling archive detection. + bool IsArchive(Stream stream, ReaderOptions readerOptions); + + /// + /// Returns true if the stream represents an archive of the format defined by this type asynchronously. + /// + /// A stream, pointing to the beginning of the archive. + /// Options controlling archive detection. + /// cancellation token + ValueTask IsArchiveAsync( + Stream stream, + ReaderOptions readerOptions, + CancellationToken cancellationToken = default + ); /// /// From a passed in archive (zip, rar, 7z, 001), return all parts. diff --git a/src/SharpCompress/Factories/LzwFactory.cs b/src/SharpCompress/Factories/LzwFactory.cs new file mode 100644 index 00000000..5bc333c6 --- /dev/null +++ b/src/SharpCompress/Factories/LzwFactory.cs @@ -0,0 +1,99 @@ +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Archives.Tar; +using SharpCompress.Common; +using SharpCompress.Compressors.Lzw; +using SharpCompress.IO; +using SharpCompress.Readers; +using SharpCompress.Readers.Lzw; +using SharpCompress.Readers.Tar; + +namespace SharpCompress.Factories; + +/// +/// Represents the foundation factory of LZW archive. +/// +public class LzwFactory : Factory, IReaderFactory +{ + #region IFactory + + /// + public override string Name => "Lzw"; + + /// + public override ArchiveType? KnownArchiveType => ArchiveType.Lzw; + + /// + public override IEnumerable GetSupportedExtensions() + { + yield return "z"; + } + + /// + public override bool IsArchive(Stream stream, ReaderOptions readerOptions) => + LzwStream.IsLzwStream(stream); + + /// + public override ValueTask IsArchiveAsync( + Stream stream, + ReaderOptions readerOptions, + CancellationToken cancellationToken = default + ) => LzwStream.IsLzwStreamAsync(stream, cancellationToken); + + #endregion + + #region IReaderFactory + + /// + internal override bool TryOpenReader( + SharpCompressStream sharpCompressStream, + ReaderOptions options, + out IReader? reader + ) + { + reader = null; + + if (LzwStream.IsLzwStream(sharpCompressStream)) + { + sharpCompressStream.Rewind(); + using ( + var testStream = options.Providers.CreateDecompressStream( + CompressionType.Lzw, + SharpCompressStream.CreateNonDisposing(sharpCompressStream) + ) + ) + { + if (TarArchive.IsTarFile(testStream)) + { + sharpCompressStream.StopRecording(); + reader = new TarReader(sharpCompressStream, options, CompressionType.Lzw); + return true; + } + } + sharpCompressStream.StopRecording(); + reader = OpenReader(sharpCompressStream, options); + return true; + } + sharpCompressStream.Rewind(); + return false; + } + + /// + public IReader OpenReader(Stream stream, ReaderOptions? options) => + LzwReader.OpenReader(stream, options); + + /// + public ValueTask OpenAsyncReader( + Stream stream, + ReaderOptions? options, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return LzwReader.OpenAsyncReader(stream, options, cancellationToken); + } + + #endregion +} diff --git a/src/SharpCompress/Factories/RarFactory.cs b/src/SharpCompress/Factories/RarFactory.cs index cfdb7ff2..2efc920d 100644 --- a/src/SharpCompress/Factories/RarFactory.cs +++ b/src/SharpCompress/Factories/RarFactory.cs @@ -1,5 +1,7 @@ using System.Collections.Generic; using System.IO; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Archives; using SharpCompress.Archives.Rar; using SharpCompress.Common; @@ -29,8 +31,15 @@ public class RarFactory : Factory, IArchiveFactory, IMultiArchiveFactory, IReade } /// - public override bool IsArchive(Stream stream, string? password = null) => - RarArchive.IsRarFile(stream); + public override bool IsArchive(Stream stream, ReaderOptions readerOptions) => + RarArchive.IsRarFile(stream, readerOptions); + + /// + public override ValueTask IsArchiveAsync( + Stream stream, + ReaderOptions readerOptions, + CancellationToken cancellationToken = default + ) => RarArchive.IsRarFileAsync(stream, readerOptions, cancellationToken); /// public override FileInfo? GetFilePart(int index, FileInfo part1) => @@ -41,24 +50,76 @@ public class RarFactory : Factory, IArchiveFactory, IMultiArchiveFactory, IReade #region IArchiveFactory /// - public IArchive Open(Stream stream, ReaderOptions? readerOptions = null) => - RarArchive.Open(stream, readerOptions); + public IArchive OpenArchive(Stream stream, ReaderOptions? readerOptions = null) => + RarArchive.OpenArchive(stream, readerOptions); /// - public IArchive Open(FileInfo fileInfo, ReaderOptions? readerOptions = null) => - RarArchive.Open(fileInfo, readerOptions); + public ValueTask OpenAsyncArchive( + Stream stream, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncArchive)OpenArchive(stream, readerOptions)); + } + + /// + public IArchive OpenArchive(FileInfo fileInfo, ReaderOptions? readerOptions = null) => + RarArchive.OpenArchive(fileInfo, readerOptions); + + /// + public ValueTask OpenAsyncArchive( + FileInfo fileInfo, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncArchive)OpenArchive(fileInfo, readerOptions)); + } #endregion #region IMultiArchiveFactory /// - public IArchive Open(IReadOnlyList streams, ReaderOptions? readerOptions = null) => - RarArchive.Open(streams, readerOptions); + public IArchive OpenArchive( + IReadOnlyList streams, + ReaderOptions? readerOptions = null + ) => RarArchive.OpenArchive(streams, readerOptions); /// - public IArchive Open(IReadOnlyList fileInfos, ReaderOptions? readerOptions = null) => - RarArchive.Open(fileInfos, readerOptions); + public async ValueTask OpenAsyncArchive( + IReadOnlyList streams, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return await RarArchive + .OpenAsyncArchive(streams, readerOptions, cancellationToken) + .ConfigureAwait(false); + } + + /// + public IArchive OpenArchive( + IReadOnlyList fileInfos, + ReaderOptions? readerOptions = null + ) => RarArchive.OpenArchive(fileInfos, readerOptions); + + /// + public async ValueTask OpenAsyncArchive( + IReadOnlyList fileInfos, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return await RarArchive + .OpenAsyncArchive(fileInfos, readerOptions, cancellationToken) + .ConfigureAwait(false); + } #endregion @@ -66,7 +127,18 @@ public class RarFactory : Factory, IArchiveFactory, IMultiArchiveFactory, IReade /// public IReader OpenReader(Stream stream, ReaderOptions? options) => - RarReader.Open(stream, options); + RarReader.OpenReader(stream, options); + + /// + public ValueTask OpenAsyncReader( + Stream stream, + ReaderOptions? options, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncReader)RarReader.OpenReader(stream, options)); + } #endregion } diff --git a/src/SharpCompress/Factories/SevenZipFactory.cs b/src/SharpCompress/Factories/SevenZipFactory.cs index cfc99d57..bac7b533 100644 --- a/src/SharpCompress/Factories/SevenZipFactory.cs +++ b/src/SharpCompress/Factories/SevenZipFactory.cs @@ -1,17 +1,23 @@ +using System; using System.Collections.Generic; using System.IO; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Archives; using SharpCompress.Archives.SevenZip; using SharpCompress.Common; +using SharpCompress.Common.Options; using SharpCompress.IO; using SharpCompress.Readers; +using SharpCompress.Writers; +using SharpCompress.Writers.SevenZip; namespace SharpCompress.Factories; /// /// Represents the foundation factory of 7Zip archive. /// -public class SevenZipFactory : Factory, IArchiveFactory, IMultiArchiveFactory +public class SevenZipFactory : Factory, IArchiveFactory, IMultiArchiveFactory, IWriterFactory { #region IFactory @@ -28,39 +34,98 @@ public class SevenZipFactory : Factory, IArchiveFactory, IMultiArchiveFactory } /// - public override bool IsArchive(Stream stream, string? password = null) => - SevenZipArchive.IsSevenZipFile(stream); + public override bool IsArchive(Stream stream, ReaderOptions readerOptions) => + SevenZipArchive.IsSevenZipFile(stream, readerOptions); + + /// + public override ValueTask IsArchiveAsync( + Stream stream, + ReaderOptions readerOptions, + CancellationToken cancellationToken = default + ) => SevenZipArchive.IsSevenZipFileAsync(stream, readerOptions, cancellationToken); #endregion #region IArchiveFactory /// - public IArchive Open(Stream stream, ReaderOptions? readerOptions = null) => - SevenZipArchive.Open(stream, readerOptions); + public IArchive OpenArchive(Stream stream, ReaderOptions? readerOptions = null) => + SevenZipArchive.OpenArchive(stream, readerOptions); /// - public IArchive Open(FileInfo fileInfo, ReaderOptions? readerOptions = null) => - SevenZipArchive.Open(fileInfo, readerOptions); + public ValueTask OpenAsyncArchive( + Stream stream, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncArchive)OpenArchive(stream, readerOptions)); + } + + /// + public IArchive OpenArchive(FileInfo fileInfo, ReaderOptions? readerOptions = null) => + SevenZipArchive.OpenArchive(fileInfo, readerOptions); + + /// + public ValueTask OpenAsyncArchive( + FileInfo fileInfo, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncArchive)OpenArchive(fileInfo, readerOptions)); + } #endregion #region IMultiArchiveFactory /// - public IArchive Open(IReadOnlyList streams, ReaderOptions? readerOptions = null) => - SevenZipArchive.Open(streams, readerOptions); + public IArchive OpenArchive( + IReadOnlyList streams, + ReaderOptions? readerOptions = null + ) => SevenZipArchive.OpenArchive(streams, readerOptions); /// - public IArchive Open(IReadOnlyList fileInfos, ReaderOptions? readerOptions = null) => - SevenZipArchive.Open(fileInfos, readerOptions); + public async ValueTask OpenAsyncArchive( + IReadOnlyList streams, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return await SevenZipArchive + .OpenAsyncArchive(streams, readerOptions, cancellationToken) + .ConfigureAwait(false); + } + + /// + public IArchive OpenArchive( + IReadOnlyList fileInfos, + ReaderOptions? readerOptions = null + ) => SevenZipArchive.OpenArchive(fileInfos, readerOptions); + + /// + public async ValueTask OpenAsyncArchive( + IReadOnlyList fileInfos, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return await SevenZipArchive + .OpenAsyncArchive(fileInfos, readerOptions, cancellationToken) + .ConfigureAwait(false); + } #endregion #region reader internal override bool TryOpenReader( - RewindableStream rewindableStream, + SharpCompressStream sharpCompressStream, ReaderOptions options, out IReader? reader ) @@ -70,4 +135,35 @@ public class SevenZipFactory : Factory, IArchiveFactory, IMultiArchiveFactory } #endregion + + #region IWriterFactory + + /// + public IWriter OpenWriter(Stream stream, IWriterOptions writerOptions) + { + SevenZipWriterOptions sevenZipOptions = writerOptions switch + { + SevenZipWriterOptions szo => szo, + WriterOptions wo => new SevenZipWriterOptions(wo), + _ => throw new ArgumentException( + $"Expected WriterOptions or SevenZipWriterOptions, got {writerOptions.GetType().Name}", + nameof(writerOptions) + ), + }; + return new SevenZipWriter(stream, sevenZipOptions); + } + + /// + public ValueTask OpenAsyncWriter( + Stream stream, + IWriterOptions writerOptions, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + var writer = OpenWriter(stream, writerOptions); + return new((IAsyncWriter)writer); + } + + #endregion } diff --git a/src/SharpCompress/Factories/TarFactory.cs b/src/SharpCompress/Factories/TarFactory.cs index 61965b86..182a33e4 100644 --- a/src/SharpCompress/Factories/TarFactory.cs +++ b/src/SharpCompress/Factories/TarFactory.cs @@ -1,14 +1,14 @@ +using System; using System.Collections.Generic; using System.IO; - +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Archives; using SharpCompress.Archives.Tar; using SharpCompress.Common; -using SharpCompress.Compressors; -using SharpCompress.Compressors.BZip2; -using SharpCompress.Compressors.LZMA; -using SharpCompress.Compressors.Xz; +using SharpCompress.Common.Options; using SharpCompress.IO; +using SharpCompress.Providers; using SharpCompress.Readers; using SharpCompress.Readers.Tar; using SharpCompress.Writers; @@ -25,7 +25,7 @@ public class TarFactory IMultiArchiveFactory, IReaderFactory, IWriterFactory, - IWriteableArchiveFactory + IWritableArchiveFactory { #region IFactory @@ -38,126 +38,36 @@ public class TarFactory /// public override IEnumerable GetSupportedExtensions() { - // from https://en.wikipedia.org/wiki/Tar_(computing)#Suffixes_for_compressed_files - - yield return "tar"; - - // gzip - yield return "taz"; - yield return "tgz"; - - // bzip2 - yield return "tb2"; - yield return "tbz"; - yield return "tbz2"; - yield return "tz2"; - - // lzma - // yield return "tlz"; // unsupported - - // xz - // yield return "txz"; // unsupported - - // compress - yield return "tZ"; - yield return "taZ"; - - // zstd - // yield return "tzst"; // unsupported + foreach (var testOption in TarWrapper.Wrappers) + { + foreach (var ext in testOption.KnownExtensions) + { + yield return ext; + } + } } /// - public override bool IsArchive(Stream stream, string? password = null) => - TarArchive.IsTarFile(stream); - - #endregion - - #region IArchiveFactory - - /// - public IArchive Open(Stream stream, ReaderOptions? readerOptions = null) => - TarArchive.Open(stream, readerOptions); - - /// - public IArchive Open(FileInfo fileInfo, ReaderOptions? readerOptions = null) => - TarArchive.Open(fileInfo, readerOptions); - - #endregion - - #region IMultiArchiveFactory - - /// - public IArchive Open(IReadOnlyList streams, ReaderOptions? readerOptions = null) => - TarArchive.Open(streams, readerOptions); - - /// - public IArchive Open(IReadOnlyList fileInfos, ReaderOptions? readerOptions = null) => - TarArchive.Open(fileInfos, readerOptions); - - #endregion - - #region IReaderFactory - - /// - internal override bool TryOpenReader( - RewindableStream rewindableStream, - ReaderOptions options, - out IReader? reader - ) + public override bool IsArchive(Stream stream, ReaderOptions readerOptions) { - reader = null; - - rewindableStream.Rewind(false); - if (TarArchive.IsTarFile(rewindableStream)) + var providers = readerOptions.Providers; + var sharpCompressStream = new SharpCompressStream(stream); + sharpCompressStream.StartRecording(TarWrapper.MaximumRewindBufferSize); + foreach (var wrapper in TarWrapper.Wrappers) { - rewindableStream.Rewind(true); - reader = OpenReader(rewindableStream, options); - return true; - } - - rewindableStream.Rewind(false); - if (BZip2Stream.IsBZip2(rewindableStream)) - { - rewindableStream.Rewind(false); - var testStream = new BZip2Stream( - NonDisposingStream.Create(rewindableStream), - CompressionMode.Decompress, - false - ); - if (TarArchive.IsTarFile(testStream)) + sharpCompressStream.Rewind(); + if (wrapper.IsMatch(sharpCompressStream)) { - rewindableStream.Rewind(true); - reader = new TarReader(rewindableStream, options, CompressionType.BZip2); - return true; - } - } - - rewindableStream.Rewind(false); - if (LZipStream.IsLZipFile(rewindableStream)) - { - rewindableStream.Rewind(false); - var testStream = new LZipStream( - NonDisposingStream.Create(rewindableStream), - CompressionMode.Decompress - ); - if (TarArchive.IsTarFile(testStream)) - { - rewindableStream.Rewind(true); - reader = new TarReader(rewindableStream, options, CompressionType.LZip); - return true; - } - } - - rewindableStream.Rewind(false); - if (XZStream.IsXZStream(rewindableStream)) - { - rewindableStream.Rewind(true); - var testStream = new XZStream(rewindableStream); - if (TarArchive.IsTarFile(testStream)) - { - rewindableStream.Rewind(true); - reader = new TarReader(rewindableStream, options, CompressionType.Xz); - return true; + sharpCompressStream.Rewind(); + var decompressedStream = CreateProbeDecompressionStream( + sharpCompressStream, + wrapper.CompressionType + ); + if (TarArchive.IsTarFile(decompressedStream)) + { + sharpCompressStream.Rewind(); + return true; + } } } @@ -165,23 +75,406 @@ public class TarFactory } /// - public IReader OpenReader(Stream stream, ReaderOptions? options) => - TarReader.Open(stream, options); + public override async ValueTask IsArchiveAsync( + Stream stream, + ReaderOptions readerOptions, + CancellationToken cancellationToken = default + ) + { + var providers = readerOptions.Providers; + var sharpCompressStream = new SharpCompressStream(stream); + sharpCompressStream.StartRecording(TarWrapper.MaximumRewindBufferSize); + foreach (var wrapper in TarWrapper.Wrappers) + { + sharpCompressStream.Rewind(); + if ( + await wrapper + .IsMatchAsync(sharpCompressStream, cancellationToken) + .ConfigureAwait(false) + ) + { + sharpCompressStream.Rewind(); + var decompressedStream = await CreateProbeDecompressionStreamAsync( + sharpCompressStream, + wrapper.CompressionType, + cancellationToken: cancellationToken + ) + .ConfigureAwait(false); + if ( + await TarArchive + .IsTarFileAsync(decompressedStream, cancellationToken) + .ConfigureAwait(false) + ) + { + sharpCompressStream.Rewind(); + return true; + } + } + } + + return false; + } + + #endregion + + private static Stream CreateProbeDecompressionStream( + Stream stream, + CompressionType compressionType, + IReaderOptions? readerOptions = null + ) + { + var providers = readerOptions?.Providers ?? CompressionProviderRegistry.Default; + var nonDisposingStream = SharpCompressStream.CreateNonDisposing(stream); + if (compressionType == CompressionType.None) + { + return nonDisposingStream; + } + + if (compressionType == CompressionType.GZip && readerOptions is not null) + { + return providers.CreateDecompressStream( + compressionType, + nonDisposingStream, + CompressionContext.FromStream(nonDisposingStream).WithReaderOptions(readerOptions) + ); + } + + return providers.CreateDecompressStream(compressionType, nonDisposingStream); + } + + private static async ValueTask CreateProbeDecompressionStreamAsync( + Stream stream, + CompressionType compressionType, + IReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + var nonDisposingStream = SharpCompressStream.CreateNonDisposing(stream); + if (compressionType == CompressionType.None) + { + return nonDisposingStream; + } + var providers = readerOptions?.Providers ?? CompressionProviderRegistry.Default; + + if (compressionType == CompressionType.GZip && readerOptions is not null) + { + return await providers + .CreateDecompressStreamAsync( + compressionType, + nonDisposingStream, + CompressionContext + .FromStream(nonDisposingStream) + .WithReaderOptions(readerOptions), + cancellationToken + ) + .ConfigureAwait(false); + } + + return await providers + .CreateDecompressStreamAsync(compressionType, nonDisposingStream, cancellationToken) + .ConfigureAwait(false); + } + + public static CompressionType GetCompressionType( + Stream stream, + IReaderOptions? readerOptions = null + ) + { + stream.Seek(0, SeekOrigin.Begin); + foreach (var wrapper in TarWrapper.Wrappers) + { + stream.Seek(0, SeekOrigin.Begin); + if (wrapper.IsMatch(stream)) + { + stream.Seek(0, SeekOrigin.Begin); + var decompressedStream = CreateProbeDecompressionStream( + stream, + wrapper.CompressionType, + readerOptions + ); + if (TarArchive.IsTarFile(decompressedStream)) + { + return wrapper.CompressionType; + } + } + } + throw new InvalidFormatException("Not a tar file."); + } + + public static async ValueTask GetCompressionTypeAsync( + Stream stream, + IReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + stream.Seek(0, SeekOrigin.Begin); + foreach (var wrapper in TarWrapper.Wrappers) + { + stream.Seek(0, SeekOrigin.Begin); + if (await wrapper.IsMatchAsync(stream, cancellationToken).ConfigureAwait(false)) + { + stream.Seek(0, SeekOrigin.Begin); + var decompressedStream = await CreateProbeDecompressionStreamAsync( + stream, + wrapper.CompressionType, + readerOptions, + cancellationToken: cancellationToken + ) + .ConfigureAwait(false); + if ( + await TarArchive + .IsTarFileAsync(decompressedStream, cancellationToken) + .ConfigureAwait(false) + ) + { + return wrapper.CompressionType; + } + } + } + throw new InvalidFormatException("Not a tar file."); + } + + #region IArchiveFactory + + /// + public IArchive OpenArchive(Stream stream, ReaderOptions? readerOptions = null) => + TarArchive.OpenArchive(stream, readerOptions); + + /// + public async ValueTask OpenAsyncArchive( + Stream stream, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) => + await TarArchive + .OpenAsyncArchive(stream, readerOptions, cancellationToken) + .ConfigureAwait(false); + + /// + public IArchive OpenArchive(FileInfo fileInfo, ReaderOptions? readerOptions = null) => + TarArchive.OpenArchive(fileInfo, readerOptions); + + /// + public async ValueTask OpenAsyncArchive( + FileInfo fileInfo, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) => + await TarArchive + .OpenAsyncArchive(fileInfo, readerOptions, cancellationToken) + .ConfigureAwait(false); + + #endregion + + #region IMultiArchiveFactory + + /// + public IArchive OpenArchive( + IReadOnlyList streams, + ReaderOptions? readerOptions = null + ) => TarArchive.OpenArchive(streams, readerOptions); + + /// + public async ValueTask OpenAsyncArchive( + IReadOnlyList streams, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return await TarArchive + .OpenAsyncArchive(streams, readerOptions, cancellationToken) + .ConfigureAwait(false); + } + + /// + public IArchive OpenArchive( + IReadOnlyList fileInfos, + ReaderOptions? readerOptions = null + ) => TarArchive.OpenArchive(fileInfos, readerOptions); + + /// + public async ValueTask OpenAsyncArchive( + IReadOnlyList fileInfos, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return await TarArchive + .OpenAsyncArchive(fileInfos, readerOptions, cancellationToken) + .ConfigureAwait(false); + } + + #endregion + + #region IReaderFactory + + /// + public IReader OpenReader(Stream stream, ReaderOptions? options) + { + options ??= ReaderOptions.ForExternalStream; + var sharpCompressStream = new SharpCompressStream(stream); + sharpCompressStream.StartRecording(TarWrapper.MaximumRewindBufferSize); + foreach (var wrapper in TarWrapper.Wrappers) + { + sharpCompressStream.Rewind(); + if (wrapper.IsMatch(sharpCompressStream)) + { + sharpCompressStream.Rewind(); + var decompressedStream = CreateProbeDecompressionStream( + sharpCompressStream, + wrapper.CompressionType, + options + ); + if (TarArchive.IsTarFile(decompressedStream)) + { + sharpCompressStream.StopRecording(); + return new TarReader(sharpCompressStream, options, wrapper.CompressionType); + } + } + } + throw new InvalidFormatException("Not a tar file."); + } + + /// + public async ValueTask OpenAsyncReader( + Stream stream, + ReaderOptions? options, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + options ??= ReaderOptions.ForExternalStream; + var sharpCompressStream = new SharpCompressStream(stream); + sharpCompressStream.StartRecording(TarWrapper.MaximumRewindBufferSize); + foreach (var wrapper in TarWrapper.Wrappers) + { + sharpCompressStream.Rewind(); + if ( + await wrapper + .IsMatchAsync(sharpCompressStream, cancellationToken) + .ConfigureAwait(false) + ) + { + sharpCompressStream.Rewind(); + var decompressedStream = await CreateProbeDecompressionStreamAsync( + sharpCompressStream, + wrapper.CompressionType, + options, + cancellationToken + ) + .ConfigureAwait(false); + if ( + await TarArchive + .IsTarFileAsync(decompressedStream, cancellationToken) + .ConfigureAwait(false) + ) + { + sharpCompressStream.Rewind(); + sharpCompressStream.StopRecording(); + return new TarReader(sharpCompressStream, options, wrapper.CompressionType); + } + } + } + + sharpCompressStream.Rewind(); + return (IAsyncReader)TarReader.OpenReader(sharpCompressStream, options); + } #endregion #region IWriterFactory /// - public IWriter Open(Stream stream, WriterOptions writerOptions) => - new TarWriter(stream, new TarWriterOptions(writerOptions)); + public IWriter OpenWriter(Stream stream, IWriterOptions writerOptions) + { + TarWriterOptions tarOptions = writerOptions switch + { + TarWriterOptions two => two, + WriterOptions wo => new TarWriterOptions(wo), + _ => throw new ArgumentException( + $"Expected WriterOptions or TarWriterOptions, got {writerOptions.GetType().Name}", + nameof(writerOptions) + ), + }; + + if (!stream.CanWrite) + { + throw new ArgumentException("Tars require writable streams."); + } + return new TarWriter(stream, tarOptions); + } + + /// + public async ValueTask OpenAsyncWriter( + Stream stream, + IWriterOptions writerOptions, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + TarWriterOptions tarOptions = writerOptions switch + { + TarWriterOptions two => two, + WriterOptions wo => new TarWriterOptions(wo), + _ => throw new ArgumentException( + $"Expected WriterOptions or TarWriterOptions, got {writerOptions.GetType().Name}", + nameof(writerOptions) + ), + }; + + if (!stream.CanWrite) + { + throw new ArgumentException("Tars require writable streams."); + } + if (writerOptions.LeaveStreamOpen) + { + stream = SharpCompressStream.CreateNonDisposing(stream); + } + + var providers = writerOptions.Providers; + + stream = writerOptions.CompressionType switch + { + CompressionType.None => stream, + CompressionType.BZip2 => await providers + .CreateCompressStreamAsync( + CompressionType.BZip2, + stream, + writerOptions.CompressionLevel, + cancellationToken + ) + .ConfigureAwait(false), + CompressionType.GZip => await providers + .CreateCompressStreamAsync( + CompressionType.GZip, + stream, + writerOptions.CompressionLevel, + cancellationToken + ) + .ConfigureAwait(false), + CompressionType.LZip => await providers + .CreateCompressStreamAsync( + CompressionType.LZip, + stream, + writerOptions.CompressionLevel, + cancellationToken + ) + .ConfigureAwait(false), + _ => throw new InvalidFormatException( + "Tar does not support compression: " + writerOptions.CompressionType + ), + }; + return new TarWriter(stream, tarOptions, streamIsPrepared: true); + } #endregion - #region IWriteableArchiveFactory + #region IWritableArchiveFactory /// - public IWritableArchive CreateWriteableArchive() => TarArchive.Create(); + public IWritableArchive CreateArchive() => TarArchive.CreateArchive(); #endregion } diff --git a/src/SharpCompress/Factories/TarWrapper.cs b/src/SharpCompress/Factories/TarWrapper.cs new file mode 100644 index 00000000..ba0a5fbd --- /dev/null +++ b/src/SharpCompress/Factories/TarWrapper.cs @@ -0,0 +1,148 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Archives.GZip; +using SharpCompress.Common; +using SharpCompress.Compressors; +using SharpCompress.Compressors.BZip2; +using SharpCompress.Compressors.Deflate; +using SharpCompress.Compressors.LZMA; +using SharpCompress.Compressors.Lzw; +using SharpCompress.Compressors.Xz; +using SharpCompress.Compressors.ZStandard; + +namespace SharpCompress.Factories; + +public class TarWrapper( + CompressionType type, + Func canHandle, + Func> canHandleAsync, + Func createStream, + Func> createStreamAsync, + IEnumerable knownExtensions, + bool wrapInSharpCompressStream = true, + int? minimumRewindBufferSize = null +) +{ + public CompressionType CompressionType { get; } = type; + public Func IsMatch { get; } = canHandle; + public Func> IsMatchAsync { get; } = canHandleAsync; + public bool WrapInSharpCompressStream { get; } = wrapInSharpCompressStream; + + /// + /// The minimum ring buffer size required to detect and probe this format. + /// Format detection reads a decompressed block to check the tar header, so + /// the ring buffer must be large enough to hold the compressed bytes consumed + /// during that probe. Defaults to . + /// + public int MinimumRewindBufferSize { get; } = + minimumRewindBufferSize ?? Common.Constants.RewindableBufferSize; + + public Func CreateStream { get; } = createStream; + public Func> CreateStreamAsync { get; } = + createStreamAsync; + + public IEnumerable KnownExtensions { get; } = knownExtensions; + + // https://en.wikipedia.org/wiki/Tar_(computing)#Suffixes_for_compressed_files + public static TarWrapper[] Wrappers { get; } = + [ + new( + CompressionType.None, + (_) => true, + (_, _) => new ValueTask(true), + (stream) => stream, + (stream, _) => new ValueTask(stream), + ["tar"], + false + ), // We always do a test for IsTarFile later + new( + CompressionType.BZip2, + BZip2Stream.IsBZip2, + BZip2Stream.IsBZip2Async, + (stream) => BZip2Stream.Create(stream, CompressionMode.Decompress, false), + async (stream, _) => + await BZip2Stream + .CreateAsync(stream, CompressionMode.Decompress, false) + .ConfigureAwait(false), + ["tar.bz2", "tb2", "tbz", "tbz2", "tz2"], + // BZip2 decompresses in whole blocks; the compressed size of the first block + // can be close to the uncompressed maximum (9 × 100 000 = 900 000 bytes). + // The ring buffer must hold all compressed bytes read during format detection. + minimumRewindBufferSize: BZip2Constants.baseBlockSize * 9 + ), + new( + CompressionType.GZip, + GZipArchive.IsGZipFile, + GZipArchive.IsGZipFileAsync, + (stream) => new GZipStream(stream, CompressionMode.Decompress), + (stream, _) => + new ValueTask(new GZipStream(stream, CompressionMode.Decompress)), + ["tar.gz", "taz", "tgz"] + ), + new( + CompressionType.ZStandard, + ZStandardStream.IsZStandard, + ZStandardStream.IsZStandardAsync, + (stream) => new ZStandardStream(stream), + (stream, _) => new ValueTask(new ZStandardStream(stream)), + ["tar.zst", "tar.zstd", "tzst", "tzstd"], + // ZStandard decompresses in blocks; the compressed size of the first block + // can be up to ZSTD_BLOCKSIZE_MAX + ZSTD_blockHeaderSize = 131075 bytes. + // The ring buffer must hold all compressed bytes read during format detection. + minimumRewindBufferSize: ZstandardConstants.DStreamInSize + ), + new( + CompressionType.LZip, + LZipStream.IsLZipFile, + LZipStream.IsLZipFileAsync, + (stream) => LZipStream.Create(stream, CompressionMode.Decompress), + async (stream, _) => + await LZipStream + .CreateAsync(stream, CompressionMode.Decompress) + .ConfigureAwait(false), + ["tar.lz"] + ), + new( + CompressionType.Xz, + XZStream.IsXZStream, + XZStream.IsXZStreamAsync, + (stream) => new XZStream(stream), + (stream, _) => new ValueTask(new XZStream(stream)), + ["tar.xz", "txz"], + false + ), + new( + CompressionType.Lzw, + LzwStream.IsLzwStream, + LzwStream.IsLzwStreamAsync, + (stream) => new LzwStream(stream), + (stream, _) => new ValueTask(new LzwStream(stream)), + ["tar.Z", "tZ", "taZ"], + false + ), + ]; + + /// + /// The largest across all registered wrappers. + /// Use this as the ring buffer size when creating a stream for Tar format detection so + /// that the buffer is sized correctly at construction and never needs to be reallocated. + /// + public static int MaximumRewindBufferSize { get; } = GetMaximumRewindBufferSize(); + + // Computed after Wrappers is initialised so the static initialisation order is safe. + private static int GetMaximumRewindBufferSize() + { + var max = 0; + foreach (var w in Wrappers) + { + if (w.MinimumRewindBufferSize > max) + { + max = w.MinimumRewindBufferSize; + } + } + return max; + } +} diff --git a/src/SharpCompress/Factories/ZStandardFactory.cs b/src/SharpCompress/Factories/ZStandardFactory.cs new file mode 100644 index 00000000..6ee920dc --- /dev/null +++ b/src/SharpCompress/Factories/ZStandardFactory.cs @@ -0,0 +1,32 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Archives; +using SharpCompress.Compressors.ZStandard; +using SharpCompress.Readers; + +namespace SharpCompress.Factories; + +internal class ZStandardFactory : Factory +{ + public override string Name => "ZStandard"; + + public override IEnumerable GetSupportedExtensions() + { + yield return "zst"; + yield return "zstd"; + } + + public override bool IsArchive(Stream stream, ReaderOptions readerOptions) => + ZStandardStream.IsZStandard(stream); + + public override ValueTask IsArchiveAsync( + Stream stream, + ReaderOptions readerOptions, + CancellationToken cancellationToken = default + ) => ZStandardStream.IsZStandardAsync(stream, cancellationToken); +} diff --git a/src/SharpCompress/Factories/ZipFactory.cs b/src/SharpCompress/Factories/ZipFactory.cs index dca697dc..a9f97052 100644 --- a/src/SharpCompress/Factories/ZipFactory.cs +++ b/src/SharpCompress/Factories/ZipFactory.cs @@ -1,9 +1,13 @@ +using System; using System.Collections.Generic; using System.IO; - +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Archives; using SharpCompress.Archives.Zip; using SharpCompress.Common; +using SharpCompress.Common.Options; +using SharpCompress.IO; using SharpCompress.Readers; using SharpCompress.Readers.Zip; using SharpCompress.Writers; @@ -20,7 +24,7 @@ public class ZipFactory IMultiArchiveFactory, IReaderFactory, IWriterFactory, - IWriteableArchiveFactory + IWritableArchiveFactory { #region IFactory @@ -39,13 +43,10 @@ public class ZipFactory } /// - public override bool IsArchive(Stream stream, string? password = null) + public override bool IsArchive(Stream stream, ReaderOptions readerOptions) { var startPosition = stream.CanSeek ? stream.Position : -1; - - // probe for single volume zip - - if (ZipArchive.IsZipFile(stream, password)) + if (ZipArchive.IsZipFile(stream, readerOptions.Password)) { return true; } @@ -60,11 +61,56 @@ public class ZipFactory stream.Position = startPosition; //test the zip (last) file of a multipart zip - if (ZipArchive.IsZipMulti(stream)) + if (ZipArchive.IsZipMulti(stream, readerOptions.Password)) { return true; } + stream.Position = startPosition; + + return false; + } + + /// + public override async ValueTask IsArchiveAsync( + Stream stream, + ReaderOptions readerOptions, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + var startPosition = stream.CanSeek ? stream.Position : -1; + + // probe for single volume zip + if ( + await ZipArchive + .IsZipFileAsync(stream, readerOptions.Password, cancellationToken) + .ConfigureAwait(false) + ) + { + return true; + } + + // probe for a multipart zip + if (!stream.CanSeek) + { + return false; + } + + stream.Position = startPosition; + + //test the zip (last) file of a multipart zip + if ( + await ZipArchive + .IsZipMultiAsync(stream, readerOptions.Password, cancellationToken) + .ConfigureAwait(false) + ) + { + return true; + } + + stream.Position = startPosition; + return false; } @@ -77,24 +123,76 @@ public class ZipFactory #region IArchiveFactory /// - public IArchive Open(Stream stream, ReaderOptions? readerOptions = null) => - ZipArchive.Open(stream, readerOptions); + public IArchive OpenArchive(Stream stream, ReaderOptions? readerOptions = null) => + ZipArchive.OpenArchive(stream, readerOptions); /// - public IArchive Open(FileInfo fileInfo, ReaderOptions? readerOptions = null) => - ZipArchive.Open(fileInfo, readerOptions); + public ValueTask OpenAsyncArchive( + Stream stream, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncArchive)OpenArchive(stream, readerOptions)); + } + + /// + public IArchive OpenArchive(FileInfo fileInfo, ReaderOptions? readerOptions = null) => + ZipArchive.OpenArchive(fileInfo, readerOptions); + + /// + public ValueTask OpenAsyncArchive( + FileInfo fileInfo, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncArchive)OpenArchive(fileInfo, readerOptions)); + } #endregion #region IMultiArchiveFactory /// - public IArchive Open(IReadOnlyList streams, ReaderOptions? readerOptions = null) => - ZipArchive.Open(streams, readerOptions); + public IArchive OpenArchive( + IReadOnlyList streams, + ReaderOptions? readerOptions = null + ) => ZipArchive.OpenArchive(streams, readerOptions); /// - public IArchive Open(IReadOnlyList fileInfos, ReaderOptions? readerOptions = null) => - ZipArchive.Open(fileInfos, readerOptions); + public async ValueTask OpenAsyncArchive( + IReadOnlyList streams, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return await ZipArchive + .OpenAsyncArchive(streams, readerOptions, cancellationToken) + .ConfigureAwait(false); + } + + /// + public IArchive OpenArchive( + IReadOnlyList fileInfos, + ReaderOptions? readerOptions = null + ) => ZipArchive.OpenArchive(fileInfos, readerOptions); + + /// + public async ValueTask OpenAsyncArchive( + IReadOnlyList fileInfos, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return await ZipArchive + .OpenAsyncArchive(fileInfos, readerOptions, cancellationToken) + .ConfigureAwait(false); + } #endregion @@ -102,22 +200,56 @@ public class ZipFactory /// public IReader OpenReader(Stream stream, ReaderOptions? options) => - ZipReader.Open(stream, options); + ZipReader.OpenReader(stream, options); + + /// + public ValueTask OpenAsyncReader( + Stream stream, + ReaderOptions? options, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncReader)ZipReader.OpenReader(stream, options)); + } #endregion #region IWriterFactory /// - public IWriter Open(Stream stream, WriterOptions writerOptions) => - new ZipWriter(stream, new ZipWriterOptions(writerOptions)); + public IWriter OpenWriter(Stream stream, IWriterOptions writerOptions) + { + ZipWriterOptions zipOptions = writerOptions switch + { + ZipWriterOptions zwo => zwo, + WriterOptions wo => new ZipWriterOptions(wo), + _ => throw new ArgumentException( + $"Expected WriterOptions or ZipWriterOptions, got {writerOptions.GetType().Name}", + nameof(writerOptions) + ), + }; + return new ZipWriter(stream, zipOptions); + } + + /// + public ValueTask OpenAsyncWriter( + Stream stream, + IWriterOptions writerOptions, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + var writer = OpenWriter(stream, writerOptions); + return new((IAsyncWriter)writer); + } #endregion - #region IWriteableArchiveFactory + #region IWritableArchiveFactory /// - public IWritableArchive CreateWriteableArchive() => ZipArchive.Create(); + public IWritableArchive CreateArchive() => ZipArchive.CreateArchive(); #endregion } diff --git a/src/SharpCompress/IO/AsyncBinaryReader.cs b/src/SharpCompress/IO/AsyncBinaryReader.cs new file mode 100644 index 00000000..f3e8c91b --- /dev/null +++ b/src/SharpCompress/IO/AsyncBinaryReader.cs @@ -0,0 +1,101 @@ +using System; +using System.Buffers.Binary; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.IO; + +public sealed class AsyncBinaryReader : IDisposable +#if NET8_0_OR_GREATER + , IAsyncDisposable +#endif +{ + private readonly Stream _stream; + private readonly Stream _originalStream; + private readonly bool _leaveOpen; + private readonly byte[] _buffer = new byte[8]; + private bool _disposed; + + public AsyncBinaryReader(Stream stream, bool leaveOpen = false) + { + if (!stream.CanRead) + { + throw new ArgumentException("Stream must be readable."); + } + + _originalStream = stream ?? throw new ArgumentNullException(nameof(stream)); + _leaveOpen = leaveOpen; + _stream = stream; + } + + public Stream BaseStream => _stream; + + public async ValueTask ReadByteAsync(CancellationToken ct = default) + { + await _stream.ReadExactAsync(_buffer, 0, 1, ct).ConfigureAwait(false); + return _buffer[0]; + } + + public async ValueTask ReadUInt16Async(CancellationToken ct = default) + { + await _stream.ReadExactAsync(_buffer, 0, 2, ct).ConfigureAwait(false); + return BinaryPrimitives.ReadUInt16LittleEndian(_buffer); + } + + public async ValueTask ReadUInt32Async(CancellationToken ct = default) + { + await _stream.ReadExactAsync(_buffer, 0, 4, ct).ConfigureAwait(false); + return BinaryPrimitives.ReadUInt32LittleEndian(_buffer); + } + + public async ValueTask ReadUInt64Async(CancellationToken ct = default) + { + await _stream.ReadExactAsync(_buffer, 0, 8, ct).ConfigureAwait(false); + return BinaryPrimitives.ReadUInt64LittleEndian(_buffer); + } + + public async ValueTask ReadBytesAsync( + byte[] bytes, + int offset, + int count, + CancellationToken ct = default + ) => await _stream.ReadExactAsync(bytes, offset, count, ct).ConfigureAwait(false); + + public async ValueTask SkipAsync(int count, CancellationToken ct = default) => + await _stream.SkipAsync(count, ct).ConfigureAwait(false); + + public void Dispose() + { + if (_disposed) + { + return; + } + + _disposed = true; + + // Dispose the original stream if we own it + if (!_leaveOpen) + { + _originalStream.Dispose(); + } + } + +#if NET8_0_OR_GREATER + public async ValueTask DisposeAsync() + { + if (_disposed) + { + return; + } + + _disposed = true; + + // Dispose the original stream if we own it + if (!_leaveOpen) + { + await _originalStream.DisposeAsync().ConfigureAwait(false); + } + } +#endif +} diff --git a/src/SharpCompress/IO/BufferedSubStream.Async.cs b/src/SharpCompress/IO/BufferedSubStream.Async.cs new file mode 100644 index 00000000..ef4ebe30 --- /dev/null +++ b/src/SharpCompress/IO/BufferedSubStream.Async.cs @@ -0,0 +1,92 @@ +using System; +using System.Buffers; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.IO; + +internal partial class BufferedSubStream +{ + private async ValueTask RefillCacheAsync(CancellationToken cancellationToken) + { + if (_isDisposed) + { + throw new ObjectDisposedException(nameof(BufferedSubStream)); + } + + var count = (int)Math.Min(BytesLeftToRead, _cache!.Length); + _cacheOffset = 0; + if (count == 0) + { + _cacheLength = 0; + return; + } + // Only seek if we're not already at the correct position + // This avoids expensive seek operations when reading sequentially + if (_stream.CanSeek && _stream.Position != origin) + { + _stream.Position = origin; + } + _cacheLength = await _stream + .ReadAsync(_cache, 0, count, cancellationToken) + .ConfigureAwait(false); + origin += _cacheLength; + BytesLeftToRead -= _cacheLength; + } + + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + if (count > Length) + { + count = (int)Length; + } + + if (count > 0) + { + if (_cacheOffset == _cacheLength) + { + await RefillCacheAsync(cancellationToken).ConfigureAwait(false); + } + + count = Math.Min(count, _cacheLength - _cacheOffset); + Buffer.BlockCopy(_cache!, _cacheOffset, buffer, offset, count); + _cacheOffset += count; + } + + return count; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + var count = buffer.Length; + if (count > Length) + { + count = (int)Length; + } + + if (count > 0) + { + if (_cacheOffset == _cacheLength) + { + await RefillCacheAsync(cancellationToken).ConfigureAwait(false); + } + + count = Math.Min(count, _cacheLength - _cacheOffset); + _cache!.AsSpan(_cacheOffset, count).CopyTo(buffer.Span); + _cacheOffset += count; + } + + return count; + } +#endif +} diff --git a/src/SharpCompress/IO/BufferedSubStream.cs b/src/SharpCompress/IO/BufferedSubStream.cs old mode 100644 new mode 100755 index 99e2bfef..e177fab5 --- a/src/SharpCompress/IO/BufferedSubStream.cs +++ b/src/SharpCompress/IO/BufferedSubStream.cs @@ -1,23 +1,46 @@ -using System; +using System; +using System.Buffers; using System.IO; +using System.Threading; +using System.Threading.Tasks; namespace SharpCompress.IO; -internal class BufferedSubStream : NonDisposingStream +internal partial class BufferedSubStream : Stream, IStreamStack { - private long position; - private int cacheOffset; - private int cacheLength; - private readonly byte[] cache; + Stream IStreamStack.BaseStream() => _stream; + + private readonly Stream _stream; public BufferedSubStream(Stream stream, long origin, long bytesToRead) - : base(stream, throwOnDispose: false) { - position = origin; - BytesLeftToRead = bytesToRead; - cache = new byte[32 << 10]; + _stream = stream ?? throw new ArgumentNullException(nameof(stream)); + this.origin = origin; + this.BytesLeftToRead = bytesToRead; } + protected override void Dispose(bool disposing) + { + if (_isDisposed) + { + return; + } + _isDisposed = true; + + if (disposing && _cache is not null) + { + ArrayPool.Shared.Return(_cache); + _cache = null; + } + base.Dispose(disposing); + } + + private int _cacheOffset; + private int _cacheLength; + private byte[]? _cache = ArrayPool.Shared.Rent(81920); + private long origin; + private bool _isDisposed; + private long BytesLeftToRead { get; set; } public override bool CanRead => true; @@ -26,9 +49,9 @@ internal class BufferedSubStream : NonDisposingStream public override bool CanWrite => false; - public override void Flush() => throw new NotSupportedException(); + public override void Flush() { } - public override long Length => BytesLeftToRead; + public override long Length => BytesLeftToRead + _cacheLength - _cacheOffset; public override long Position { @@ -36,37 +59,69 @@ internal class BufferedSubStream : NonDisposingStream set => throw new NotSupportedException(); } + private void RefillCache() + { + if (_isDisposed) + { + throw new ObjectDisposedException(nameof(BufferedSubStream)); + } + + var count = (int)Math.Min(BytesLeftToRead, _cache!.Length); + _cacheOffset = 0; + if (count == 0) + { + _cacheLength = 0; + return; + } + + // Only seek if we're not already at the correct position + // This avoids expensive seek operations when reading sequentially + if (_stream.CanSeek && _stream.Position != origin) + { + _stream.Position = origin; + } + + _cacheLength = _stream.Read(_cache, 0, count); + origin += _cacheLength; + BytesLeftToRead -= _cacheLength; + } + public override int Read(byte[] buffer, int offset, int count) { - if (count > BytesLeftToRead) + if (count > Length) { - count = (int)BytesLeftToRead; + count = (int)Length; } if (count > 0) { - if (cacheLength == 0) + if (_cacheOffset == _cacheLength) { - cacheOffset = 0; - Stream.Position = position; - cacheLength = Stream.Read(cache, 0, cache.Length); - position += cacheLength; + RefillCache(); } - if (count > cacheLength) - { - count = cacheLength; - } - - Buffer.BlockCopy(cache, cacheOffset, buffer, offset, count); - cacheOffset += count; - cacheLength -= count; - BytesLeftToRead -= count; + count = Math.Min(count, _cacheLength - _cacheOffset); + Buffer.BlockCopy(_cache!, _cacheOffset, buffer, offset, count); + _cacheOffset += count; } return count; } + public override int ReadByte() + { + if (_cacheOffset == _cacheLength) + { + RefillCache(); + if (_cacheLength == 0) + { + return -1; + } + } + + return _cache![_cacheOffset++]; + } + public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); public override void SetLength(long value) => throw new NotSupportedException(); diff --git a/src/SharpCompress/IO/CountingStream.cs b/src/SharpCompress/IO/CountingStream.cs new file mode 100644 index 00000000..ef545c9c --- /dev/null +++ b/src/SharpCompress/IO/CountingStream.cs @@ -0,0 +1,150 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.IO; + +/// +/// A simple stream wrapper that counts bytes read and written without buffering. +/// +internal class CountingStream : Stream +{ + private readonly Stream _stream; + private long _bytesRead; + private long _bytesWritten; + + public CountingStream(Stream stream) + { + _stream = stream ?? throw new ArgumentNullException(nameof(stream)); + } + + internal Stream WrappedStream => _stream; + + /// + /// Gets the total number of bytes read from this stream. + /// + public long BytesRead => _bytesRead; + + /// + /// Gets the total number of bytes written to this stream. + /// + public long BytesWritten => _bytesWritten; + + public override bool CanRead => _stream.CanRead; + + public override bool CanSeek => _stream.CanSeek; + + public override bool CanWrite => _stream.CanWrite; + + public override long Length => _stream.Length; + + public override long Position + { + get => _stream.Position; + set => _stream.Position = value; + } + + public override void Flush() => _stream.Flush(); + + public override async Task FlushAsync(CancellationToken cancellationToken) => + await _stream.FlushAsync(cancellationToken).ConfigureAwait(false); + + public override int Read(byte[] buffer, int offset, int count) + { + var read = _stream.Read(buffer, offset, count); + _bytesRead += read; + return read; + } + + public override int ReadByte() + { + var value = _stream.ReadByte(); + if (value != -1) + { + _bytesRead++; + } + + return value; + } + +#if !LEGACY_DOTNET + public override int Read(Span buffer) + { + var read = _stream.Read(buffer); + _bytesRead += read; + return read; + } +#endif + + public override long Seek(long offset, SeekOrigin origin) => _stream.Seek(offset, origin); + + public override void SetLength(long value) => _stream.SetLength(value); + + public override void Write(byte[] buffer, int offset, int count) + { + _stream.Write(buffer, offset, count); + _bytesWritten += count; + } + + public override void WriteByte(byte value) + { + _stream.WriteByte(value); + _bytesWritten++; + } + + public override async Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + await _stream.WriteAsync(buffer, offset, count, cancellationToken).ConfigureAwait(false); + _bytesWritten += count; + } + + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + var read = await _stream + .ReadAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + _bytesRead += read; + return read; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + var read = await _stream.ReadAsync(buffer, cancellationToken).ConfigureAwait(false); + _bytesRead += read; + return read; + } + + public override async ValueTask WriteAsync( + ReadOnlyMemory buffer, + CancellationToken cancellationToken = default + ) + { + await _stream.WriteAsync(buffer, cancellationToken).ConfigureAwait(false); + _bytesWritten += buffer.Length; + } +#endif + + protected override void Dispose(bool disposing) + { + if (disposing) + { + _stream.Dispose(); + } + base.Dispose(disposing); + } +} diff --git a/src/SharpCompress/IO/CountingWritableSubStream.cs b/src/SharpCompress/IO/CountingWritableSubStream.cs deleted file mode 100644 index ce9ace71..00000000 --- a/src/SharpCompress/IO/CountingWritableSubStream.cs +++ /dev/null @@ -1,47 +0,0 @@ -using System; -using System.IO; - -namespace SharpCompress.IO; - -internal class CountingWritableSubStream : NonDisposingStream -{ - internal CountingWritableSubStream(Stream stream) - : base(stream, throwOnDispose: false) { } - - public ulong Count { get; private set; } - - public override bool CanRead => false; - - public override bool CanSeek => false; - - public override bool CanWrite => true; - - public override void Flush() => Stream.Flush(); - - public override long Length => throw new NotSupportedException(); - - public override long Position - { - get => throw new NotSupportedException(); - set => throw new NotSupportedException(); - } - - public override int Read(byte[] buffer, int offset, int count) => - throw new NotSupportedException(); - - public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); - - public override void SetLength(long value) => throw new NotSupportedException(); - - public override void Write(byte[] buffer, int offset, int count) - { - Stream.Write(buffer, offset, count); - Count += (uint)count; - } - - public override void WriteByte(byte value) - { - Stream.WriteByte(value); - ++Count; - } -} diff --git a/src/SharpCompress/IO/DataDescriptorStream.cs b/src/SharpCompress/IO/DataDescriptorStream.cs index 8ba9eb0f..fb93153c 100644 --- a/src/SharpCompress/IO/DataDescriptorStream.cs +++ b/src/SharpCompress/IO/DataDescriptorStream.cs @@ -1,30 +1,29 @@ using System; using System.IO; -using System.Runtime.CompilerServices; namespace SharpCompress.IO; -public class DataDescriptorStream : Stream +public class DataDescriptorStream : Stream, IStreamStack { + Stream IStreamStack.BaseStream() => _stream; + private readonly Stream _stream; private long _start; - private int _search_position; + private int _searchPosition; private bool _isDisposed; private bool _done; - private static byte[] DataDescriptorMarker = new byte[] { 0x50, 0x4b, 0x07, 0x08 }; - private static long DataDescriptorSize = 24; + private static byte[] _dataDescriptorMarker = new byte[] { 0x50, 0x4b, 0x07, 0x08 }; + private static long _dataDescriptorSize = 24; public DataDescriptorStream(Stream stream) { _stream = stream; _start = _stream.Position; - _search_position = 0; + _searchPosition = 0; _done = false; } - internal bool IsRecording { get; private set; } - protected override void Dispose(bool disposing) { if (_isDisposed) @@ -45,13 +44,13 @@ public class DataDescriptorStream : Stream public override bool CanWrite => false; - public override void Flush() => throw new NotSupportedException(); + public override void Flush() { } public override long Length => _stream.Length; public override long Position { - get => _stream.Position; + get => _stream.Position - _start; set => _stream.Position = value; } @@ -60,20 +59,20 @@ public class DataDescriptorStream : Stream var br = new BinaryReader(stream); br.ReadUInt32(); br.ReadUInt32(); // CRC32 can be checked if we calculate it - var compressed_size = br.ReadUInt32(); - var uncompressed_size = br.ReadUInt32(); - var uncompressed_64bit = br.ReadInt64(); + var compressedSize = br.ReadUInt32(); + var uncompressedSize = br.ReadUInt32(); + var uncompressed64Bit = br.ReadInt64(); - stream.Position -= DataDescriptorSize; + stream.Position -= _dataDescriptorSize; - var test_64bit = ((long)uncompressed_size << 32) | compressed_size; + var test64Bit = ((long)uncompressedSize << 32) | compressedSize; - if (test_64bit == size && test_64bit == uncompressed_64bit) + if (test64Bit == size && test64Bit == uncompressed64Bit) { return true; } - if (compressed_size == size && compressed_size == uncompressed_size) + if (compressedSize == size && compressedSize == uncompressedSize) { return true; } @@ -88,24 +87,24 @@ public class DataDescriptorStream : Stream return 0; } - int read = _stream.Read(buffer, offset, count); + var read = _stream.Read(buffer, offset, count); - for (int i = 0; i < read; i++) + for (var i = 0; i < read; i++) { - if (buffer[offset + i] == DataDescriptorMarker[_search_position]) + if (buffer[offset + i] == _dataDescriptorMarker[_searchPosition]) { - _search_position++; + _searchPosition++; - if (_search_position == 4) + if (_searchPosition == 4) { - _search_position = 0; + _searchPosition = 0; - if (read - i > DataDescriptorSize) + if (read - i > _dataDescriptorSize) { var check = new MemoryStream( buffer, offset + i - 3, - (int)DataDescriptorSize + (int)_dataDescriptorSize ); _done = validate_data_descriptor( check, @@ -131,15 +130,15 @@ public class DataDescriptorStream : Stream } else { - _search_position = 0; + _searchPosition = 0; } } - if (_search_position > 0) + if (_searchPosition > 0) { - read -= _search_position; - _stream.Position -= _search_position; - _search_position = 0; + read -= _searchPosition; + _stream.Position -= _searchPosition; + _searchPosition = 0; } return read; diff --git a/src/SharpCompress/IO/IStreamStack.cs b/src/SharpCompress/IO/IStreamStack.cs new file mode 100644 index 00000000..c24a9c90 --- /dev/null +++ b/src/SharpCompress/IO/IStreamStack.cs @@ -0,0 +1,80 @@ +using System; +using System.Collections; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; + +namespace SharpCompress.IO; + +public interface IStreamStack +{ + /// + /// Returns the immediate underlying stream in the stack. + /// + Stream BaseStream(); +} + +public static class StreamStackExtensions +{ + public static T? GetStream(this IStreamStack stack) + where T : Stream + { + var baseStream = stack.BaseStream(); + if (baseStream is T tStream) + { + return tStream; + } + else if (baseStream is IStreamStack innerStack) + { + return innerStack.GetStream(); + } + else + { + return null; + } + } + + /// + /// Gets the root underlying stream at the bottom of the stack. + /// This is useful for seeking when the intermediate streams don't support it. + /// + public static Stream GetRootStream(this IStreamStack stack) + { + var current = stack.BaseStream(); + while (current is IStreamStack streamStack) + { + current = streamStack.BaseStream(); + } + return current; + } + + internal static void Rewind(this IStreamStack stream, int count) + { + IStreamStack? current = stream; + + while (current != null) + { + if (current is SharpCompressStream sharpCompressStream) + { + // Try to rewind within the buffer. If the position is outside the buffered + // region, silently ignore (matching release behavior where streams without + // buffering simply didn't rewind). + var targetPosition = sharpCompressStream.Position - count; + if (targetPosition >= 0) + { + try + { + sharpCompressStream.Position = targetPosition; + } + catch (NotSupportedException) + { + // Cannot seek outside buffered region - silently ignore + } + } + return; + } + current = current.BaseStream() as IStreamStack; + } + } +} diff --git a/src/SharpCompress/IO/ListeningStream.cs b/src/SharpCompress/IO/ListeningStream.cs deleted file mode 100644 index a1bf715b..00000000 --- a/src/SharpCompress/IO/ListeningStream.cs +++ /dev/null @@ -1,71 +0,0 @@ -using System.IO; -using SharpCompress.Common; - -namespace SharpCompress.IO; - -internal class ListeningStream : Stream -{ - private long currentEntryTotalReadBytes; - private readonly IExtractionListener listener; - - public ListeningStream(IExtractionListener listener, Stream stream) - { - Stream = stream; - this.listener = listener; - } - - protected override void Dispose(bool disposing) - { - if (disposing) - { - Stream.Dispose(); - } - base.Dispose(disposing); - } - - public Stream Stream { get; } - - public override bool CanRead => Stream.CanRead; - - public override bool CanSeek => Stream.CanSeek; - - public override bool CanWrite => Stream.CanWrite; - - public override void Flush() => Stream.Flush(); - - public override long Length => Stream.Length; - - public override long Position - { - get => Stream.Position; - set => Stream.Position = value; - } - - public override int Read(byte[] buffer, int offset, int count) - { - var read = Stream.Read(buffer, offset, count); - currentEntryTotalReadBytes += read; - listener.FireCompressedBytesRead(currentEntryTotalReadBytes, currentEntryTotalReadBytes); - return read; - } - - public override int ReadByte() - { - var value = Stream.ReadByte(); - if (value == -1) - { - return -1; - } - - ++currentEntryTotalReadBytes; - listener.FireCompressedBytesRead(currentEntryTotalReadBytes, currentEntryTotalReadBytes); - return value; - } - - public override long Seek(long offset, SeekOrigin origin) => Stream.Seek(offset, origin); - - public override void SetLength(long value) => Stream.SetLength(value); - - public override void Write(byte[] buffer, int offset, int count) => - Stream.Write(buffer, offset, count); -} diff --git a/src/SharpCompress/IO/MarkingBinaryReader.cs b/src/SharpCompress/IO/MarkingBinaryReader.cs index 424b9e08..bbe17f04 100644 --- a/src/SharpCompress/IO/MarkingBinaryReader.cs +++ b/src/SharpCompress/IO/MarkingBinaryReader.cs @@ -1,14 +1,14 @@ using System; using System.Buffers.Binary; using System.IO; +using System.Text; +using SharpCompress.Common; namespace SharpCompress.IO; -internal class MarkingBinaryReader : BinaryReader +internal class MarkingBinaryReader(Stream stream) + : BinaryReader(stream, Encoding.UTF8, leaveOpen: true) //always leave the stream open { - public MarkingBinaryReader(Stream stream) - : base(stream) { } - public virtual long CurrentReadByteCount { get; protected set; } public virtual void Mark() => CurrentReadByteCount = 0; @@ -44,8 +44,9 @@ internal class MarkingBinaryReader : BinaryReader var bytes = base.ReadBytes(count); if (bytes.Length != count) { - throw new EndOfStreamException( + throw new InvalidFormatException( string.Format( + Constants.DefaultCultureInfo, "Could not read the requested amount of bytes. End of stream reached. Requested: {0} Read: {1}", count, bytes.Length @@ -114,7 +115,7 @@ internal class MarkingBinaryReader : BinaryReader shift += 7; } while (shift <= maxShift); - throw new FormatException("malformed vint"); + throw new InvalidFormatException("malformed vint"); } public uint ReadRarVIntUInt32(int maxBytes = 5) => @@ -152,6 +153,6 @@ internal class MarkingBinaryReader : BinaryReader shift += 7; } while (shift <= maxShift); - throw new FormatException("malformed vint"); + throw new InvalidFormatException("malformed vint"); } } diff --git a/src/SharpCompress/IO/NonDisposingStream.cs b/src/SharpCompress/IO/NonDisposingStream.cs deleted file mode 100644 index 334296d7..00000000 --- a/src/SharpCompress/IO/NonDisposingStream.cs +++ /dev/null @@ -1,73 +0,0 @@ -using System; -using System.IO; - -namespace SharpCompress.IO; - -public class NonDisposingStream : Stream -{ - public static NonDisposingStream Create(Stream stream, bool throwOnDispose = false) - { - if ( - stream is NonDisposingStream nonDisposingStream - && nonDisposingStream.ThrowOnDispose == throwOnDispose - ) - { - return nonDisposingStream; - } - return new NonDisposingStream(stream, throwOnDispose); - } - - protected NonDisposingStream(Stream stream, bool throwOnDispose = false) - { - Stream = stream; - ThrowOnDispose = throwOnDispose; - } - - public bool ThrowOnDispose { get; set; } - - protected override void Dispose(bool disposing) - { - if (ThrowOnDispose) - { - throw new InvalidOperationException( - $"Attempt to dispose of a {nameof(NonDisposingStream)} when {nameof(ThrowOnDispose)} is {ThrowOnDispose}" - ); - } - } - - protected Stream Stream { get; } - - public override bool CanRead => Stream.CanRead; - - public override bool CanSeek => Stream.CanSeek; - - public override bool CanWrite => Stream.CanWrite; - - public override void Flush() => Stream.Flush(); - - public override long Length => Stream.Length; - - public override long Position - { - get => Stream.Position; - set => Stream.Position = value; - } - - public override int Read(byte[] buffer, int offset, int count) => - Stream.Read(buffer, offset, count); - - public override long Seek(long offset, SeekOrigin origin) => Stream.Seek(offset, origin); - - public override void SetLength(long value) => Stream.SetLength(value); - - public override void Write(byte[] buffer, int offset, int count) => - Stream.Write(buffer, offset, count); - -#if !NETFRAMEWORK && !NETSTANDARD2_0 - - public override int Read(Span buffer) => Stream.Read(buffer); - - public override void Write(ReadOnlySpan buffer) => Stream.Write(buffer); - -#endif -} diff --git a/src/SharpCompress/IO/PooledMemoryStream.cs b/src/SharpCompress/IO/PooledMemoryStream.cs new file mode 100644 index 00000000..3116386f --- /dev/null +++ b/src/SharpCompress/IO/PooledMemoryStream.cs @@ -0,0 +1,701 @@ +using System; +using System.Buffers; +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.IO; + +/// +/// MemoryStream implementation backed by pooled byte arrays. +/// Uses to reduce GC pressure for temporary buffers. +/// +/// +/// This implementation is not thread-safe. Use appropriate synchronization for concurrent access. +/// Buffers exposed via or are allocated as +/// fresh non-pooled arrays to avoid exposing pooled memory. +/// +public sealed class PooledMemoryStream : MemoryStream +{ + private const int MaxStreamLength = int.MaxValue; + + private readonly ArrayPool _arrayPool; + private readonly int _blockSize; + + private List? _blocks; + private bool _isOpen; + private int _position; + private int _length; + private int _capacity; + + public PooledMemoryStream() + : this(0) { } + + public PooledMemoryStream(int capacity) + : this(capacity, Constants.BufferSize, ArrayPool.Shared) { } + + public PooledMemoryStream(int capacity, int blockSize) + : this(capacity, blockSize, ArrayPool.Shared) { } + + public PooledMemoryStream(int capacity, int blockSize, ArrayPool arrayPool) + { + ThrowHelper.ThrowIfNull(arrayPool, nameof(arrayPool)); + ThrowHelper.ThrowIfNegative(capacity, nameof(capacity)); + ThrowHelper.ThrowIfNegativeOrZero(blockSize, nameof(blockSize)); + + _arrayPool = arrayPool; + _blockSize = blockSize; + + _blocks = new List(); + _isOpen = true; + _position = 0; + _length = 0; + _capacity = capacity; + + EnsureSegmentedAllocated(capacity); + } + + public override bool CanRead => _isOpen; + + public override bool CanSeek => _isOpen; + + public override bool CanWrite => _isOpen; + + public override long Length + { + get + { + EnsureNotClosed(); + return _length; + } + } + + public override long Position + { + get + { + EnsureNotClosed(); + return _position; + } + set + { + EnsureNotClosed(); + ThrowHelper.ThrowIfNegative(value, nameof(value)); + ThrowHelper.ThrowIfGreaterThan(value, MaxStreamLength, nameof(value)); + + _position = (int)value; + } + } + + public override int Capacity + { + get + { + EnsureNotClosed(); + return _capacity; + } + set + { + ThrowHelper.ThrowIfLessThan(value, _length, nameof(value)); + + EnsureNotClosed(); + + var target = value; + if (target == _capacity) + { + return; + } + + SetCapacityAbsolute(target); + } + } + + public override void Flush() + { + EnsureNotClosed(); + } + + public override Task FlushAsync(CancellationToken cancellationToken) + { + if (cancellationToken.IsCancellationRequested) + { + return Task.FromCanceled(cancellationToken); + } + + EnsureNotClosed(); + return Task.CompletedTask; + } + + public override long Seek(long offset, SeekOrigin loc) + { + EnsureNotClosed(); + + var anchor = loc switch + { + SeekOrigin.Begin => 0, + SeekOrigin.Current => _position, + SeekOrigin.End => _length, + _ => throw new ArgumentException("Invalid seek origin.", nameof(loc)), + }; + + var target = anchor + offset; + if (target < 0) + { + throw new IOException("Attempted to seek before the beginning of the stream."); + } + + if (target > MaxStreamLength) + { + throw new ArgumentOutOfRangeException(nameof(offset)); + } + + _position = (int)target; + return _position; + } + + public override void SetLength(long value) + { + EnsureWritable(); + + ThrowHelper.ThrowIfNegative(value, nameof(value)); + ThrowHelper.ThrowIfGreaterThan(value, MaxStreamLength, nameof(value)); + var newLength = (int)value; + if (newLength > _capacity) + { + EnsureCapacityForAppend(newLength); + } + + if (newLength > _length) + { + ClearRange(_length, newLength - _length); + } + + _length = newLength; + if (_position > newLength) + { + _position = newLength; + } + } + + public override int Read(byte[] buffer, int offset, int count) + { + ValidateReadWriteBufferArguments(buffer, offset, count); + EnsureNotClosed(); + + var available = _length - _position; + if (available <= 0) + { + return 0; + } + + if (count > available) + { + count = available; + } + + CopyFromSegmented(_position, buffer, offset, count); + + _position += count; + return count; + } + + public override int ReadByte() + { + EnsureNotClosed(); + if (_position >= _length) + { + return -1; + } + + var blockIndex = _position / _blockSize; + var blockOffset = _position % _blockSize; + var value = _blocks![blockIndex][blockOffset]; + + _position++; + return value; + } + + public override void Write(byte[] buffer, int offset, int count) + { + ValidateReadWriteBufferArguments(buffer, offset, count); + EnsureWritable(); + + if (count == 0) + { + return; + } + + var endPosition = _position + count; + if (endPosition < 0) + { + throw new IOException("Stream is too long."); + } + + if (endPosition > _capacity) + { + EnsureCapacityForAppend(endPosition); + } + + if (_position > _length) + { + ClearRange(_length, _position - _length); + } + + CopyToSegmented(_position, buffer, offset, count); + + _position = endPosition; + if (_position > _length) + { + _length = _position; + } + } + + public override void WriteByte(byte value) + { + EnsureWritable(); + + var endPosition = _position + 1; + if (endPosition < 0) + { + throw new IOException("Stream is too long."); + } + + if (endPosition > _capacity) + { + EnsureCapacityForAppend(endPosition); + } + + if (_position > _length) + { + ClearRange(_length, _position - _length); + } + + var blockIndex = _position / _blockSize; + var blockOffset = _position % _blockSize; + _blocks![blockIndex][blockOffset] = value; + + _position = endPosition; + if (_position > _length) + { + _length = _position; + } + } + + private byte[] CreateExposableBuffer() + { + var exposable = new byte[_capacity]; + if (_length == 0) + { + return exposable; + } + + CopyFromSegmented(0, exposable, 0, _length); + + return exposable; + } + + public override byte[] GetBuffer() + { + EnsureNotClosed(); + return CreateExposableBuffer(); + } + + public override bool TryGetBuffer(out ArraySegment buffer) + { + EnsureNotClosed(); + + var exposableBuffer = CreateExposableBuffer(); + buffer = new ArraySegment(exposableBuffer, 0, _length); + return true; + } + + public override byte[] ToArray() + { + EnsureNotClosed(); + + var count = _length; + if (count == 0) + { + return Array.Empty(); + } + + var copy = new byte[count]; + CopyFromSegmented(0, copy, 0, count); + + return copy; + } + + public override void WriteTo(Stream stream) + { + ThrowHelper.ThrowIfNull(stream, nameof(stream)); + EnsureNotClosed(); + + var count = _length; + if (count == 0) + { + return; + } + + var position = 0; + var remaining = count; + while (remaining > 0) + { + var blockIndex = position / _blockSize; + var blockOffset = position % _blockSize; + var toWrite = Math.Min(remaining, _blockSize - blockOffset); + stream.Write(_blocks![blockIndex], blockOffset, toWrite); + position += toWrite; + remaining -= toWrite; + } + } + + public override Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + if (cancellationToken.IsCancellationRequested) + { + return Task.FromCanceled(cancellationToken); + } + + return Task.FromResult(Read(buffer, offset, count)); + } + + public override Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + if (cancellationToken.IsCancellationRequested) + { + return Task.FromCanceled(cancellationToken); + } + + Write(buffer, offset, count); + return Task.CompletedTask; + } + +#if !LEGACY_DOTNET + public override int Read(Span buffer) + { + EnsureNotClosed(); + + var available = _length - _position; + if (available <= 0) + { + return 0; + } + + var count = Math.Min(available, buffer.Length); + var sourcePosition = _position; + var destinationOffset = 0; + var remaining = count; + + while (remaining > 0) + { + var blockIndex = sourcePosition / _blockSize; + var blockOffset = sourcePosition % _blockSize; + var toCopy = Math.Min(remaining, _blockSize - blockOffset); + _blocks! + [blockIndex] + .AsSpan(blockOffset, toCopy) + .CopyTo(buffer.Slice(destinationOffset, toCopy)); + + sourcePosition += toCopy; + destinationOffset += toCopy; + remaining -= toCopy; + } + + _position += count; + return count; + } + + public override void Write(ReadOnlySpan buffer) + { + EnsureWritable(); + if (buffer.Length == 0) + { + return; + } + + var endPosition = _position + buffer.Length; + if (endPosition < 0) + { + throw new IOException("Stream is too long."); + } + + if (endPosition > _capacity) + { + EnsureCapacityForAppend(endPosition); + } + + if (_position > _length) + { + ClearRange(_length, _position - _length); + } + + var sourceOffset = 0; + var destinationPosition = _position; + var remaining = buffer.Length; + + while (remaining > 0) + { + var blockIndex = destinationPosition / _blockSize; + var blockOffset = destinationPosition % _blockSize; + var toCopy = Math.Min(remaining, _blockSize - blockOffset); + + buffer + .Slice(sourceOffset, toCopy) + .CopyTo(_blocks![blockIndex].AsSpan(blockOffset, toCopy)); + + sourceOffset += toCopy; + destinationPosition += toCopy; + remaining -= toCopy; + } + + _position = endPosition; + if (_position > _length) + { + _length = _position; + } + } + + public override ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + if (cancellationToken.IsCancellationRequested) + { + return ValueTask.FromCanceled(cancellationToken); + } + + return ValueTask.FromResult(Read(buffer.Span)); + } + + public override ValueTask WriteAsync( + ReadOnlyMemory buffer, + CancellationToken cancellationToken = default + ) + { + if (cancellationToken.IsCancellationRequested) + { + return ValueTask.FromCanceled(cancellationToken); + } + + Write(buffer.Span); + return ValueTask.CompletedTask; + } +#endif + + protected override void Dispose(bool disposing) + { + if (_isOpen) + { + _isOpen = false; + + if (disposing) + { + ReturnPooledBuffers(); + } + } + + base.Dispose(disposing); + } + + private void EnsureNotClosed() + { + if (!_isOpen) + { + throw new ObjectDisposedException(nameof(PooledMemoryStream)); + } + } + + private void EnsureWritable() + { + EnsureNotClosed(); + } + + private void EnsureCapacityForAppend(int requiredLength) + { + if (requiredLength < 0) + { + throw new IOException("Stream is too long."); + } + + if (requiredLength <= _capacity) + { + return; + } + + var nextCapacity = RoundUpToBlockBoundary(requiredLength); + SetCapacityAbsolute(nextCapacity); + } + + private void SetCapacityAbsolute(int newCapacity) + { + ThrowHelper.ThrowIfLessThan(newCapacity, _length, nameof(newCapacity)); + + EnsureSegmentedAllocated(newCapacity); + + _capacity = newCapacity; + if (_length > _capacity) + { + _length = _capacity; + } + if (_position > _capacity) + { + _position = _capacity; + } + } + + private void EnsureSegmentedAllocated(int capacity) + { + var requiredAllocated = RoundUpToBlockBoundary(capacity); + var requiredBlocks = requiredAllocated == 0 ? 0 : requiredAllocated / _blockSize; + + _blocks ??= new List(); + + while (_blocks.Count < requiredBlocks) + { + _blocks.Add(_arrayPool.Rent(_blockSize)); + } + + while (_blocks.Count > requiredBlocks) + { + var index = _blocks.Count - 1; + var block = _blocks[index]; + _blocks.RemoveAt(index); + _arrayPool.Return(block); + } + } + + private int RoundUpToBlockBoundary(int value) + { + if (value <= 0) + { + return 0; + } + + var rounded = ((long)value + _blockSize - 1) / _blockSize * _blockSize; + if (rounded > MaxStreamLength) + { + throw new IOException("Stream is too long."); + } + + return (int)rounded; + } + + private void ClearRange(int absoluteStart, int count) + { + if (count <= 0) + { + return; + } + + var position = absoluteStart; + var remaining = count; + while (remaining > 0) + { + var blockIndex = position / _blockSize; + var blockOffset = position % _blockSize; + var toClear = Math.Min(remaining, _blockSize - blockOffset); + Array.Clear(_blocks![blockIndex], blockOffset, toClear); + position += toClear; + remaining -= toClear; + } + } + + private void CopyFromSegmented( + int absoluteSourcePosition, + byte[] destination, + int offset, + int count + ) + { + var sourcePosition = absoluteSourcePosition; + var destinationOffset = offset; + var remaining = count; + + while (remaining > 0) + { + var blockIndex = sourcePosition / _blockSize; + var blockOffset = sourcePosition % _blockSize; + var toCopy = Math.Min(remaining, _blockSize - blockOffset); + Buffer.BlockCopy( + _blocks![blockIndex], + blockOffset, + destination, + destinationOffset, + toCopy + ); + + sourcePosition += toCopy; + destinationOffset += toCopy; + remaining -= toCopy; + } + } + + private void CopyToSegmented( + int absoluteDestinationPosition, + byte[] source, + int offset, + int count + ) + { + var sourceOffset = offset; + var destinationPosition = absoluteDestinationPosition; + var remaining = count; + + while (remaining > 0) + { + var blockIndex = destinationPosition / _blockSize; + var blockOffset = destinationPosition % _blockSize; + var toCopy = Math.Min(remaining, _blockSize - blockOffset); + Buffer.BlockCopy(source, sourceOffset, _blocks![blockIndex], blockOffset, toCopy); + + sourceOffset += toCopy; + destinationPosition += toCopy; + remaining -= toCopy; + } + } + + private void ReturnSegmentedBlocks() + { + if (_blocks is null) + { + return; + } + + for (var i = 0; i < _blocks.Count; i++) + { + _arrayPool.Return(_blocks[i]); + } + + _blocks.Clear(); + } + + private void ReturnPooledBuffers() + { + ReturnSegmentedBlocks(); + _blocks = null; + } + + private static void ValidateReadWriteBufferArguments(byte[] buffer, int offset, int count) + { + ThrowHelper.ThrowIfNull(buffer, nameof(buffer)); + ThrowHelper.ThrowIfNegative(offset, nameof(offset)); + ThrowHelper.ThrowIfNegative(count, nameof(count)); + if (buffer.Length - offset < count) + { + throw new ArgumentException("Offset and length are out of bounds."); + } + } +} diff --git a/src/SharpCompress/IO/ProgressReportingStream.Async.cs b/src/SharpCompress/IO/ProgressReportingStream.Async.cs new file mode 100644 index 00000000..5fed0c61 --- /dev/null +++ b/src/SharpCompress/IO/ProgressReportingStream.Async.cs @@ -0,0 +1,56 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.IO; + +internal sealed partial class ProgressReportingStream +{ + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + var bytesRead = await _baseStream + .ReadAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + if (bytesRead > 0) + { + _bytesTransferred += bytesRead; + ReportProgress(); + } + return bytesRead; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + var bytesRead = await _baseStream + .ReadAsync(buffer, cancellationToken) + .ConfigureAwait(false); + if (bytesRead > 0) + { + _bytesTransferred += bytesRead; + ReportProgress(); + } + return bytesRead; + } +#endif + +#if !LEGACY_DOTNET + public override async ValueTask DisposeAsync() + { + if (!_leaveOpen) + { + await _baseStream.DisposeAsync().ConfigureAwait(false); + } + await base.DisposeAsync().ConfigureAwait(false); + } +#endif +} diff --git a/src/SharpCompress/IO/ProgressReportingStream.cs b/src/SharpCompress/IO/ProgressReportingStream.cs new file mode 100644 index 00000000..ec2f4fad --- /dev/null +++ b/src/SharpCompress/IO/ProgressReportingStream.cs @@ -0,0 +1,111 @@ +using System; +using System.IO; +using SharpCompress.Common; + +namespace SharpCompress.IO; + +/// +/// A stream wrapper that reports progress as data is read from the source. +/// Used to track compression or extraction progress by wrapping the source stream. +/// +internal sealed partial class ProgressReportingStream : Stream +{ + private readonly Stream _baseStream; + private readonly IProgress _progress; + private readonly string _entryPath; + private readonly long? _totalBytes; + private long _bytesTransferred; + private readonly bool _leaveOpen; + + public ProgressReportingStream( + Stream baseStream, + IProgress progress, + string entryPath, + long? totalBytes, + bool leaveOpen = false + ) + { + _baseStream = baseStream; + _progress = progress; + _entryPath = entryPath; + _totalBytes = totalBytes; + _leaveOpen = leaveOpen; + } + + public override bool CanRead => _baseStream.CanRead; + + public override bool CanSeek => _baseStream.CanSeek; + + public override bool CanWrite => false; + + public override long Length => _baseStream.Length; + + public override long Position + { + get => _baseStream.Position; + set => + throw new NotSupportedException( + "Directly setting Position is not supported in ProgressReportingStream to maintain progress tracking integrity." + ); + } + + public override void Flush() => _baseStream.Flush(); + + public override int Read(byte[] buffer, int offset, int count) + { + var bytesRead = _baseStream.Read(buffer, offset, count); + if (bytesRead > 0) + { + _bytesTransferred += bytesRead; + ReportProgress(); + } + return bytesRead; + } + +#if !LEGACY_DOTNET + public override int Read(Span buffer) + { + var bytesRead = _baseStream.Read(buffer); + if (bytesRead > 0) + { + _bytesTransferred += bytesRead; + ReportProgress(); + } + return bytesRead; + } +#endif + + public override int ReadByte() + { + var value = _baseStream.ReadByte(); + if (value != -1) + { + _bytesTransferred++; + ReportProgress(); + } + return value; + } + + public override long Seek(long offset, SeekOrigin origin) => _baseStream.Seek(offset, origin); + + public override void SetLength(long value) => _baseStream.SetLength(value); + + public override void Write(byte[] buffer, int offset, int count) => + throw new NotSupportedException( + "ProgressReportingStream is designed for read operations to track progress." + ); + + private void ReportProgress() + { + _progress.Report(new ProgressReport(_entryPath, _bytesTransferred, _totalBytes)); + } + + protected override void Dispose(bool disposing) + { + if (disposing && !_leaveOpen) + { + _baseStream.Dispose(); + } + base.Dispose(disposing); + } +} diff --git a/src/SharpCompress/IO/ReadOnlySubStream.Async.cs b/src/SharpCompress/IO/ReadOnlySubStream.Async.cs new file mode 100644 index 00000000..a51039df --- /dev/null +++ b/src/SharpCompress/IO/ReadOnlySubStream.Async.cs @@ -0,0 +1,50 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.IO; + +internal partial class ReadOnlySubStream +{ + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + if (BytesLeftToRead < count) + { + count = (int)BytesLeftToRead; + } + var read = await _stream + .ReadAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + if (read > 0) + { + BytesLeftToRead -= read; + _position += read; + } + return read; + } + +#if !LEGACY_DOTNET + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + var sliceLen = BytesLeftToRead < buffer.Length ? BytesLeftToRead : buffer.Length; + var read = await _stream + .ReadAsync(buffer.Slice(0, (int)sliceLen), cancellationToken) + .ConfigureAwait(false); + if (read > 0) + { + BytesLeftToRead -= read; + _position += read; + } + return read; + } +#endif +} diff --git a/src/SharpCompress/IO/ReadOnlySubStream.cs b/src/SharpCompress/IO/ReadOnlySubStream.cs index c0b39d54..33317c0e 100644 --- a/src/SharpCompress/IO/ReadOnlySubStream.cs +++ b/src/SharpCompress/IO/ReadOnlySubStream.cs @@ -3,17 +3,23 @@ using System.IO; namespace SharpCompress.IO; -internal class ReadOnlySubStream : NonDisposingStream +internal partial class ReadOnlySubStream : Stream, IStreamStack { + Stream IStreamStack.BaseStream() => _stream; + + private readonly Stream _stream; + private readonly bool _leaveOpen; private long _position; - public ReadOnlySubStream(Stream stream, long bytesToRead) - : this(stream, null, bytesToRead) { } + public ReadOnlySubStream(Stream stream, long bytesToRead, bool leaveOpen = true) + : this(stream, null, bytesToRead, leaveOpen) { } - public ReadOnlySubStream(Stream stream, long? origin, long bytesToRead) - : base(stream, throwOnDispose: false) + public ReadOnlySubStream(Stream stream, long? origin, long bytesToRead, bool leaveOpen = true) { - if (origin != null) + _stream = stream ?? throw new ArgumentNullException(nameof(stream)); + _leaveOpen = leaveOpen; + + if (origin != null && stream.Position != origin.Value) { stream.Position = origin.Value; } @@ -29,7 +35,7 @@ internal class ReadOnlySubStream : NonDisposingStream public override bool CanWrite => false; - public override void Flush() => throw new NotSupportedException(); + public override void Flush() { } public override long Length => throw new NotSupportedException(); @@ -45,7 +51,7 @@ internal class ReadOnlySubStream : NonDisposingStream { count = (int)BytesLeftToRead; } - var read = Stream.Read(buffer, offset, count); + var read = _stream.Read(buffer, offset, count); if (read > 0) { BytesLeftToRead -= read; @@ -60,7 +66,7 @@ internal class ReadOnlySubStream : NonDisposingStream { return -1; } - var value = Stream.ReadByte(); + var value = _stream.ReadByte(); if (value != -1) { --BytesLeftToRead; @@ -69,11 +75,11 @@ internal class ReadOnlySubStream : NonDisposingStream return value; } -#if !NETFRAMEWORK && !NETSTANDARD2_0 +#if !LEGACY_DOTNET public override int Read(Span buffer) { - var slice_len = BytesLeftToRead < buffer.Length ? BytesLeftToRead : buffer.Length; - var read = Stream.Read(buffer.Slice(0, (int)slice_len)); + var sliceLen = BytesLeftToRead < buffer.Length ? BytesLeftToRead : buffer.Length; + var read = _stream.Read(buffer.Slice(0, (int)sliceLen)); if (read > 0) { BytesLeftToRead -= read; @@ -89,4 +95,13 @@ internal class ReadOnlySubStream : NonDisposingStream public override void Write(byte[] buffer, int offset, int count) => throw new NotSupportedException(); + + protected override void Dispose(bool disposing) + { + if (disposing && !_leaveOpen) + { + _stream.Dispose(); + } + base.Dispose(disposing); + } } diff --git a/src/SharpCompress/IO/RewindableStream.cs b/src/SharpCompress/IO/RewindableStream.cs deleted file mode 100644 index ccdc8ee0..00000000 --- a/src/SharpCompress/IO/RewindableStream.cs +++ /dev/null @@ -1,149 +0,0 @@ -using System; -using System.IO; - -namespace SharpCompress.IO; - -public class RewindableStream : Stream -{ - private readonly Stream stream; - private MemoryStream bufferStream = new MemoryStream(); - private bool isRewound; - private bool isDisposed; - - public RewindableStream(Stream stream) => this.stream = stream; - - internal bool IsRecording { get; private set; } - - protected override void Dispose(bool disposing) - { - if (isDisposed) - { - return; - } - isDisposed = true; - base.Dispose(disposing); - if (disposing) - { - stream.Dispose(); - } - } - - public void Rewind(bool stopRecording) - { - isRewound = true; - IsRecording = !stopRecording; - bufferStream.Position = 0; - } - - public void Rewind(MemoryStream buffer) - { - if (bufferStream.Position >= buffer.Length) - { - bufferStream.Position -= buffer.Length; - } - else - { - bufferStream.TransferTo(buffer); - //create new memorystream to allow proper resizing as memorystream could be a user provided buffer - //https://github.com/adamhathcock/sharpcompress/issues/306 - bufferStream = new MemoryStream(); - buffer.Position = 0; - buffer.TransferTo(bufferStream); - bufferStream.Position = 0; - } - isRewound = true; - } - - public void StartRecording() - { - //if (isRewound && bufferStream.Position != 0) - // throw new System.NotImplementedException(); - if (bufferStream.Position != 0) - { - var data = bufferStream.ToArray(); - var position = bufferStream.Position; - bufferStream.SetLength(0); - bufferStream.Write(data, (int)position, data.Length - (int)position); - bufferStream.Position = 0; - } - IsRecording = true; - } - - public override bool CanRead => true; - - public override bool CanSeek => stream.CanSeek; - - public override bool CanWrite => false; - - public override void Flush() => throw new NotSupportedException(); - - public override long Length => stream.Length; - - public override long Position - { - get => stream.Position + bufferStream.Position - bufferStream.Length; - set - { - if (!isRewound) - { - stream.Position = value; - } - else if (value < stream.Position - bufferStream.Length || value >= stream.Position) - { - stream.Position = value; - isRewound = false; - bufferStream.SetLength(0); - } - else - { - bufferStream.Position = value - stream.Position + bufferStream.Length; - } - } - } - - public override int Read(byte[] buffer, int offset, int count) - { - //don't actually read if we don't really want to read anything - //currently a network stream bug on Windows for .NET Core - if (count == 0) - { - return 0; - } - int read; - if (isRewound && bufferStream.Position != bufferStream.Length) - { - // don't read more than left - var readCount = Math.Min(count, (int)(bufferStream.Length - bufferStream.Position)); - read = bufferStream.Read(buffer, offset, readCount); - if (read < readCount) - { - var tempRead = stream.Read(buffer, offset + read, count - read); - if (IsRecording) - { - bufferStream.Write(buffer, offset + read, tempRead); - } - read += tempRead; - } - if (bufferStream.Position == bufferStream.Length && !IsRecording) - { - isRewound = false; - bufferStream.SetLength(0); - } - return read; - } - - read = stream.Read(buffer, offset, count); - if (IsRecording) - { - bufferStream.Write(buffer, offset, read); - } - return read; - } - - public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); - - public override void SetLength(long value) => throw new NotSupportedException(); - - public override void Write(byte[] buffer, int offset, int count) => - throw new NotSupportedException(); -} diff --git a/src/SharpCompress/IO/RingBuffer.cs b/src/SharpCompress/IO/RingBuffer.cs new file mode 100644 index 00000000..7d9a888f --- /dev/null +++ b/src/SharpCompress/IO/RingBuffer.cs @@ -0,0 +1,152 @@ +using System; +using System.Buffers; + +namespace SharpCompress.IO; + +/// +/// A circular buffer that keeps the last N bytes written to it. +/// Used for limited backward seeking on forward-only streams. +/// +internal sealed class RingBuffer : IDisposable +{ + private byte[]? _buffer; + private readonly int _capacity; + private int _writePos; // Next write position in circular buffer + private int _length; // Number of valid bytes (0 to _capacity) + private bool _isDisposed; + + /// + /// Creates a new RingBuffer with the specified capacity. + /// + /// Maximum number of bytes to keep in the buffer. + public RingBuffer(int capacity) + { + if (capacity <= 0) + { + throw new ArgumentOutOfRangeException(nameof(capacity), "Capacity must be positive."); + } + _capacity = capacity; + _buffer = ArrayPool.Shared.Rent(capacity); + _writePos = 0; + _length = 0; + } + + /// + /// Gets the maximum capacity of the buffer. + /// + public int Capacity => _capacity; + + /// + /// Gets the number of valid bytes currently in the buffer. + /// + public int Length => _length; + + /// + /// Writes data to the buffer. If the data exceeds capacity, + /// only the last bytes are kept. + /// + /// Source data array. + /// Offset in source array. + /// Number of bytes to write. + public void Write(byte[] data, int offset, int count) + { + ThrowIfDisposed(); + + if (count == 0) + { + return; + } + + // If data is larger than buffer, only keep the last _capacity bytes + if (count >= _capacity) + { + Array.Copy(data, offset + count - _capacity, _buffer!, 0, _capacity); + _writePos = 0; + _length = _capacity; + return; + } + + // Write data to circular buffer (may wrap around) + int firstPart = Math.Min(count, _capacity - _writePos); + Array.Copy(data, offset, _buffer!, _writePos, firstPart); + if (firstPart < count) + { + // Wrap around + Array.Copy(data, offset + firstPart, _buffer!, 0, count - firstPart); + } + + _writePos = (_writePos + count) % _capacity; + _length = Math.Min(_length + count, _capacity); + } + + /// + /// Reads data from the buffer at a logical position relative to the end. + /// + /// How many bytes from the end (most recent write) to start reading. + /// Destination buffer. + /// Offset in destination buffer. + /// Maximum bytes to read. + /// Number of bytes actually read. + /// If bytesFromEnd exceeds available data. + public int ReadFromEnd(long bytesFromEnd, byte[] buffer, int offset, int count) + { + ThrowIfDisposed(); + + if (bytesFromEnd > _length) + { + throw new ArgumentOutOfRangeException( + nameof(bytesFromEnd), + $"Requested position ({bytesFromEnd} bytes from end) is outside buffer range (length={_length})." + ); + } + + if (bytesFromEnd <= 0 || count <= 0) + { + return 0; + } + + // Calculate starting index in circular buffer + // _writePos is where next byte would be written (one past last valid byte) + int bufferIndex = (int)((_writePos - bytesFromEnd + _capacity) % _capacity); + int availableFromBuffer = (int)Math.Min(bytesFromEnd, count); + + // Read from rolling buffer (may wrap around) + int firstPart = Math.Min(availableFromBuffer, _capacity - bufferIndex); + Array.Copy(_buffer!, bufferIndex, buffer, offset, firstPart); + if (firstPart < availableFromBuffer) + { + // Wrap around + Array.Copy(_buffer!, 0, buffer, offset + firstPart, availableFromBuffer - firstPart); + } + + return availableFromBuffer; + } + + /// + /// Checks if a position (as bytes from the end) is within the buffered range. + /// + /// Position as bytes from end. + /// True if the position is available in the buffer. + public bool CanReadFromEnd(long bytesFromEnd) => bytesFromEnd >= 0 && bytesFromEnd <= _length; + + public void Dispose() + { + if (!_isDisposed) + { + _isDisposed = true; + if (_buffer is not null) + { + ArrayPool.Shared.Return(_buffer); + _buffer = null; + } + } + } + + private void ThrowIfDisposed() + { + if (_isDisposed) + { + throw new ObjectDisposedException(nameof(RingBuffer)); + } + } +} diff --git a/src/SharpCompress/IO/SeekableSharpCompressStream.Async.cs b/src/SharpCompress/IO/SeekableSharpCompressStream.Async.cs new file mode 100644 index 00000000..d9b727a9 --- /dev/null +++ b/src/SharpCompress/IO/SeekableSharpCompressStream.Async.cs @@ -0,0 +1,65 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.IO; + +internal sealed partial class SeekableSharpCompressStream +{ + public override Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) => _stream.ReadAsync(buffer, offset, count, cancellationToken); + +#if !LEGACY_DOTNET + public override ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) => _stream.ReadAsync(buffer, cancellationToken); + + public override ValueTask WriteAsync( + ReadOnlyMemory buffer, + CancellationToken cancellationToken = default + ) => _stream.WriteAsync(buffer, cancellationToken); + + public override ValueTask DisposeAsync() + { + if (_isDisposed) + { + return base.DisposeAsync(); + } + if (ThrowOnDispose) + { + throw new ArchiveOperationException( + $"Attempt to dispose of a {nameof(SeekableSharpCompressStream)} when {nameof(ThrowOnDispose)} is true" + ); + } + _isDisposed = true; + if (!LeaveStreamOpen) + { + _stream.Dispose(); + } + return base.DisposeAsync(); + } +#endif + + public override Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) => _stream.WriteAsync(buffer, offset, count, cancellationToken); + + public override Task FlushAsync(CancellationToken cancellationToken) => + _stream.FlushAsync(cancellationToken); + + public override Task CopyToAsync( + Stream destination, + int bufferSize, + CancellationToken cancellationToken + ) => _stream.CopyToAsync(destination, bufferSize, cancellationToken); +} diff --git a/src/SharpCompress/IO/SeekableSharpCompressStream.cs b/src/SharpCompress/IO/SeekableSharpCompressStream.cs new file mode 100644 index 00000000..bad123bb --- /dev/null +++ b/src/SharpCompress/IO/SeekableSharpCompressStream.cs @@ -0,0 +1,107 @@ +using System; +using System.IO; +using SharpCompress.Common; + +namespace SharpCompress.IO; + +internal sealed partial class SeekableSharpCompressStream : SharpCompressStream +{ + public override Stream BaseStream() => _stream; + + private readonly Stream _stream; + private long? _recordedPosition; + private bool _isDisposed; + + /// + /// Gets or sets whether to leave the underlying stream open when disposed. + /// + public override bool LeaveStreamOpen { get; } + + public SeekableSharpCompressStream(Stream stream, bool leaveStreamOpen = false) + : base(Null, true, false, null) + { + ThrowHelper.ThrowIfNull(stream); + if (!stream.CanSeek) + { + throw new ArgumentException("Stream must be seekable", nameof(stream)); + } + + LeaveStreamOpen = leaveStreamOpen; + _stream = stream; + } + + public override bool CanRead => _stream.CanRead; + + public override bool CanSeek => _stream.CanSeek; + + public override bool CanWrite => _stream.CanWrite; + + public override long Length => _stream.Length; + + public override long Position + { + get => _stream.Position; + set => _stream.Position = value; + } + + internal override bool IsRecording => _recordedPosition.HasValue; + + public override void Flush() => _stream.Flush(); + + public override int Read(byte[] buffer, int offset, int count) => + _stream.Read(buffer, offset, count); + +#if !LEGACY_DOTNET + public override int Read(Span buffer) => _stream.Read(buffer); +#endif + + public override long Seek(long offset, SeekOrigin origin) => _stream.Seek(offset, origin); + + public override void SetLength(long value) => _stream.SetLength(value); + + public override void Write(byte[] buffer, int offset, int count) => + _stream.Write(buffer, offset, count); + +#if !LEGACY_DOTNET + public override void Write(ReadOnlySpan buffer) => _stream.Write(buffer); +#endif + + public override void Rewind(bool stopRecording = false) + { + if (!_recordedPosition.HasValue) + { + return; + } + + _stream.Seek(_recordedPosition.Value, SeekOrigin.Begin); + if (stopRecording) + { + _recordedPosition = null; + } + } + + public override void StartRecording(int? minBufferSize = null) => + _recordedPosition = _stream.Position; + + public override void StopRecording() => _recordedPosition = null; + + protected override void Dispose(bool disposing) + { + if (_isDisposed) + { + return; + } + if (ThrowOnDispose) + { + throw new ArchiveOperationException( + $"Attempt to dispose of a {nameof(SeekableSharpCompressStream)} when {nameof(ThrowOnDispose)} is true" + ); + } + _isDisposed = true; + if (disposing && !LeaveStreamOpen) + { + _stream.Dispose(); + } + base.Dispose(disposing); + } +} diff --git a/src/SharpCompress/IO/SharpCompressStream.Async.cs b/src/SharpCompress/IO/SharpCompressStream.Async.cs new file mode 100644 index 00000000..9b22148a --- /dev/null +++ b/src/SharpCompress/IO/SharpCompressStream.Async.cs @@ -0,0 +1,274 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.IO; + +public partial class SharpCompressStream +{ + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + if (count == 0) + { + return 0; + } + + // In passthrough mode, delegate directly to underlying stream + if (_isPassthrough) + { + return await stream + .ReadAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + } + + // If ring buffer is enabled, use ring buffer logic + if (_ringBuffer is not null) + { + return await ReadWithRingBufferAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + } + + // No buffering - read directly from stream + int read = await stream + .ReadAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + streamPosition += read; + _logicalPosition = streamPosition; + return read; + } + + /// + /// Async version of ReadWithRingBuffer. + /// + private async ValueTask ReadWithRingBufferAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + int totalRead = 0; + + // If logical position is behind stream position, read from ring buffer first + while (count > 0 && _logicalPosition < streamPosition) + { + long bytesFromEnd = streamPosition - _logicalPosition; + + // Verify data is available in ring buffer + if (!_ringBuffer!.CanReadFromEnd(bytesFromEnd)) + { + throw new ArchiveOperationException( + $"Ring buffer underflow: trying to read {bytesFromEnd} bytes back, " + + $"but buffer only holds {_ringBuffer.Length} bytes." + ); + } + + int available = _ringBuffer.ReadFromEnd(bytesFromEnd, buffer, offset, count); + totalRead += available; + offset += available; + count -= available; + _logicalPosition += available; + } + + // If more data needed and we're caught up, read from underlying stream + if (count > 0 && _logicalPosition == streamPosition) + { + int read = await stream + .ReadAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + if (read > 0) + { + _ringBuffer!.Write(buffer, offset, read); + streamPosition += read; + _logicalPosition += read; + totalRead += read; + } + } + + return totalRead; + } + +#if !LEGACY_DOTNET + public override ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + if (buffer.Length == 0) + { + return ValueTask.FromResult(0); + } + + // In passthrough mode, delegate directly to underlying stream + if (_isPassthrough) + { + return stream.ReadAsync(buffer, cancellationToken); + } + + return ReadAsyncCore(buffer, cancellationToken); + } + + private async ValueTask ReadAsyncCore( + Memory buffer, + CancellationToken cancellationToken + ) + { + // If ring buffer is enabled, use ring buffer logic + if (_ringBuffer is not null) + { + return await ReadWithRingBufferAsync(buffer, cancellationToken).ConfigureAwait(false); + } + + // No buffering - read directly from stream + int read = await stream.ReadAsync(buffer, cancellationToken).ConfigureAwait(false); + streamPosition += read; + _logicalPosition = streamPosition; + return read; + } + + /// + /// Async version of ReadWithRingBuffer for Memory<byte>. + /// + private async ValueTask ReadWithRingBufferAsync( + Memory buffer, + CancellationToken cancellationToken + ) + { + int totalRead = 0; + int count = buffer.Length; + int offset = 0; + + // If logical position is behind stream position, read from ring buffer first + // Note: We need to use a temporary byte array because RingBuffer.ReadFromEnd expects byte[] + while (count > 0 && _logicalPosition < streamPosition) + { + long bytesFromEnd = streamPosition - _logicalPosition; + + // Verify data is available in ring buffer + if (!_ringBuffer!.CanReadFromEnd(bytesFromEnd)) + { + throw new ArchiveOperationException( + $"Ring buffer underflow: trying to read {bytesFromEnd} bytes back, " + + $"but buffer only holds {_ringBuffer.Length} bytes." + ); + } + + var tempBuffer = new byte[Math.Min(count, (int)bytesFromEnd)]; + int available = _ringBuffer.ReadFromEnd(bytesFromEnd, tempBuffer, 0, tempBuffer.Length); + tempBuffer.AsSpan(0, available).CopyTo(buffer.Span.Slice(offset)); + + totalRead += available; + offset += available; + count -= available; + _logicalPosition += available; + } + + // If more data needed and we're caught up, read from underlying stream + if (count > 0 && _logicalPosition == streamPosition) + { + int read = await stream + .ReadAsync(buffer.Slice(offset, count), cancellationToken) + .ConfigureAwait(false); + if (read > 0) + { + // RingBuffer.Write expects byte[], so we need to copy + var tempBuffer = buffer.Slice(offset, read).ToArray(); + _ringBuffer!.Write(tempBuffer, 0, read); + streamPosition += read; + _logicalPosition += read; + totalRead += read; + } + } + + return totalRead; + } +#endif + + public override Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + if (_isPassthrough) + { + return stream.WriteAsync(buffer, offset, count, cancellationToken); + } + throw new NotSupportedException(); + } + +#if !LEGACY_DOTNET + public override ValueTask WriteAsync( + ReadOnlyMemory buffer, + CancellationToken cancellationToken = default + ) + { + if (_isPassthrough) + { + return stream.WriteAsync(buffer, cancellationToken); + } + throw new NotSupportedException(); + } +#endif + + public override Task FlushAsync(CancellationToken cancellationToken) + { + if (_isPassthrough) + { + return stream.FlushAsync(cancellationToken); + } + throw new NotSupportedException(); + } + + public override async Task CopyToAsync( + Stream destination, + int bufferSize, + CancellationToken cancellationToken + ) + { + byte[] buffer = new byte[bufferSize]; + int bytesRead; + while ( + ( + bytesRead = await ReadAsync(buffer, 0, buffer.Length, cancellationToken) + .ConfigureAwait(false) + ) != 0 + ) + { + await destination + .WriteAsync(buffer, 0, bytesRead, cancellationToken) + .ConfigureAwait(false); + } + } + +#if !LEGACY_DOTNET + public override async ValueTask DisposeAsync() + { + if (!isDisposed) + { + if (ThrowOnDispose) + { + throw new ArchiveOperationException( + $"Attempt to dispose of a {nameof(SharpCompressStream)} when {nameof(ThrowOnDispose)} is true" + ); + } + isDisposed = true; + if (!LeaveStreamOpen) + { + await stream.DisposeAsync().ConfigureAwait(false); + } + _ringBuffer?.Dispose(); + _ringBuffer = null; + } + await base.DisposeAsync().ConfigureAwait(false); + } +#endif +} diff --git a/src/SharpCompress/IO/SharpCompressStream.Create.cs b/src/SharpCompress/IO/SharpCompressStream.Create.cs new file mode 100644 index 00000000..37d91ebf --- /dev/null +++ b/src/SharpCompress/IO/SharpCompressStream.Create.cs @@ -0,0 +1,112 @@ +using System; +using System.IO; +using SharpCompress.Common; + +namespace SharpCompress.IO; + +public partial class SharpCompressStream +{ + /// + /// Creates a that acts as a zero-overhead passthrough wrapper + /// around without taking ownership of it. + /// + /// + /// + /// This is a thin wrapper: all reads, writes, and seeks are forwarded directly to the underlying + /// stream with no ring-buffer overhead. delegates to the underlying + /// stream's own value. + /// + /// + /// The resulting stream does not support , , + /// or . Call on the passthrough stream to obtain + /// a recording-capable wrapper when needed. + /// + /// + /// Because the stream does not take ownership, the underlying stream is never disposed when + /// this wrapper is disposed. Use this when you need to satisfy an API that expects a + /// without transferring lifetime responsibility. + /// + /// + /// The underlying stream to wrap. Must not be . + /// + /// A passthrough that does not dispose . + /// + public static SharpCompressStream CreateNonDisposing(Stream stream) => + new(stream, leaveStreamOpen: true, passthrough: true, bufferSize: null); + + /// + /// Creates a that supports recording and rewinding over + /// , choosing the most efficient strategy based on the stream's + /// capabilities. + /// + /// + /// Seekable streams — wraps in a thin delegate that calls the underlying + /// stream's native directly. No ring buffer is allocated. + /// stores the current position; seeks + /// back to it. + /// Non-seekable streams (network streams, compressed streams, pipes) — allocates + /// a ring buffer of bytes. All bytes read from the underlying + /// stream are kept in the ring buffer so that can replay them without + /// re-reading the underlying stream. If more bytes have been read than the ring buffer can hold, + /// a subsequent rewind will throw ; increase + /// or to + /// avoid this. + /// Already-wrapped streams — if is already a + /// (or a stack that contains one), it is returned as-is to + /// prevent double-wrapping and double-buffering. + /// + /// The underlying stream to wrap. Must not be . + /// + /// Size in bytes of the ring buffer allocated for non-seekable streams. + /// Defaults to (81 920 bytes) when + /// . Has no effect when is seekable, because + /// no ring buffer is needed in that case. + /// + /// + /// A wrapping . The returned instance + /// owns the stream and will dispose it unless the original source was a non-disposing passthrough + /// wrapper. + /// + public static SharpCompressStream Create(Stream stream, int? bufferSize = null) + { + var rewindableBufferSize = bufferSize ?? Constants.RewindableBufferSize; + + // If it's a passthrough SharpCompressStream, unwrap it and create proper seekable wrapper + if (stream is SharpCompressStream sharpCompressStream) + { + if (sharpCompressStream._isPassthrough) + { + // Unwrap the passthrough and create appropriate wrapper + var underlying = sharpCompressStream.stream; + if (underlying.CanSeek) + { + // Create SeekableSharpCompressStream that preserves LeaveStreamOpen + return new SeekableSharpCompressStream(underlying, true); + } + // Non-seekable underlying stream - wrap with rolling buffer + return new SharpCompressStream(underlying, true, false, rewindableBufferSize); + } + // Not passthrough - return as-is + return sharpCompressStream; + } + + // Check if stream is wrapping a SharpCompressStream (e.g., via IStreamStack) + if (stream is IStreamStack streamStack) + { + var underlying = streamStack.GetStream(); + if (underlying is not null) + { + return underlying; + } + } + + if (stream.CanSeek) + { + return new SeekableSharpCompressStream(stream); + } + + // For non-seekable streams, create a SharpCompressStream with rolling buffer + // to allow limited backward seeking (required by decompressors that over-read) + return new SharpCompressStream(stream, false, false, rewindableBufferSize); + } +} diff --git a/src/SharpCompress/IO/SharpCompressStream.cs b/src/SharpCompress/IO/SharpCompressStream.cs new file mode 100644 index 00000000..f2bda765 --- /dev/null +++ b/src/SharpCompress/IO/SharpCompressStream.cs @@ -0,0 +1,451 @@ +using System; +using System.IO; +using SharpCompress.Common; + +namespace SharpCompress.IO; + +/// +/// Stream wrapper that provides optional ring-buffered reading for non-seekable +/// or forward-only streams, enabling limited backward seeking required by some +/// decompressors and archive formats. +/// +/// +/// In most cases, callers should obtain an instance via the static +/// SharpCompressStream.Create(...) methods rather than constructing this +/// class directly. The Create methods select an appropriate configuration +/// (such as passthrough vs buffered mode and buffer size) for the underlying +/// stream and usage scenario. +/// +public partial class SharpCompressStream : Stream, IStreamStack +{ + public virtual Stream BaseStream() => stream; + + private readonly Stream stream; + private bool isDisposed; + private long streamPosition; + + // Ring buffer for recording mode and over-read protection. + // Single unified buffering mechanism for both use cases. + private RingBuffer? _ringBuffer; + private long _logicalPosition; // The current logical read position (can be behind streamPosition) + + // Recording state: anchor position when StartRecording was called + private long? _recordingStartPosition; + private bool _isRecording; + + // Passthrough mode - no buffering, delegates CanSeek to underlying stream + private readonly bool _isPassthrough; + + /// + /// Gets whether this stream is in passthrough mode (no buffering, delegates to underlying stream). + /// + internal bool IsPassthrough => _isPassthrough; + + /// + /// Gets whether to leave the underlying stream open when disposed. + /// + public virtual bool LeaveStreamOpen { get; } + + /// + /// Gets or sets whether to throw an exception when Dispose is called. + /// Useful for testing to ensure streams are not disposed prematurely. + /// + internal bool ThrowOnDispose { get; set; } + + public SharpCompressStream(Stream stream) + { + this.stream = stream; + _logicalPosition = 0; + } + + /// + /// Private constructor for passthrough mode. + /// + protected SharpCompressStream( + Stream stream, + bool leaveStreamOpen, + bool passthrough, + int? bufferSize + ) + { + this.stream = stream; + LeaveStreamOpen = leaveStreamOpen; + _isPassthrough = passthrough; + _logicalPosition = 0; + + if (bufferSize.HasValue && bufferSize.Value > 0) + { + _ringBuffer = new RingBuffer(bufferSize.Value); + } + } + + /// + /// Gets whether the stream is actively recording reads to the ring buffer. + /// + internal virtual bool IsRecording => _isRecording; + + protected override void Dispose(bool disposing) + { + if (isDisposed) + { + return; + } + if (ThrowOnDispose) + { + throw new ArchiveOperationException( + $"Attempt to dispose of a {nameof(SharpCompressStream)} when {nameof(ThrowOnDispose)} is true" + ); + } + isDisposed = true; + base.Dispose(disposing); + if (disposing) + { + if (!LeaveStreamOpen) + { + stream.Dispose(); + } + _ringBuffer?.Dispose(); + _ringBuffer = null; + } + } + + public void Rewind() => Rewind(false); + + public virtual void Rewind(bool stopRecording) + { + if (_isPassthrough) + { + throw new ArchiveOperationException( + "Rewind cannot be called on a passthrough stream. Use Create() first." + ); + } + + if (_recordingStartPosition is null) + { + throw new ArchiveOperationException( + "Rewind can only be called after StartRecording() has been called." + ); + } + + // Verify recording anchor is within ring buffer range + long anchorAge = streamPosition - _recordingStartPosition.Value; + if (anchorAge > _ringBuffer!.Length) + { + throw new ArchiveOperationException( + $"Cannot rewind: recording anchor is {anchorAge} bytes behind current position, " + + $"but ring buffer only holds {_ringBuffer.Length} bytes. " + + $"Recording buffer overflow - increase DefaultRollingBufferSize or reduce format detection reads." + ); + } + + // Rewind logical position to recording anchor + _logicalPosition = _recordingStartPosition.Value; + + if (stopRecording) + { + _isRecording = false; + // Note: We keep _recordingStartPosition so Rewind() can be called again + // (frozen recording mode). The anchor is only cleared when a new recording + // starts or the stream is disposed. + } + } + + public virtual void StopRecording() + { + if (_isPassthrough) + { + throw new ArchiveOperationException( + "StopRecording cannot be called on a passthrough stream. Use Create() first." + ); + } + if (!IsRecording) + { + throw new ArchiveOperationException( + "StopRecording can only be called when recording is active." + ); + } + + // Mark that we're no longer actively recording + _isRecording = false; + + // Rewind to recording anchor position + _logicalPosition = _recordingStartPosition!.Value; + + // Note: We keep _recordingStartPosition so future Rewind() calls still work + // (frozen recording mode) until Rewind(stopRecording: true) is called + } + + /// + /// Begins recording reads so that can replay them. + /// + /// + /// Minimum ring buffer capacity in bytes. When provided and larger than + /// , the ring buffer is allocated + /// with this size. Pass the largest amount of compressed data that may be consumed + /// during format detection before the first rewind. Defaults to + /// when null or not supplied. + /// + public virtual void StartRecording(int? minBufferSize = null) + { + if (_isPassthrough) + { + throw new ArchiveOperationException( + "StartRecording cannot be called on a passthrough stream. Use Create() first." + ); + } + if (IsRecording) + { + throw new ArchiveOperationException( + "StartRecording can only be called when not already recording." + ); + } + + // Allocate ring buffer with the requested minimum size (at least the global default). + if (_ringBuffer is null) + { + var requiredSize = + minBufferSize.GetValueOrDefault() > Constants.RewindableBufferSize + ? minBufferSize.GetValueOrDefault() + : Constants.RewindableBufferSize; + _ringBuffer = new RingBuffer(requiredSize); + } + else if (minBufferSize.HasValue && minBufferSize.Value > _ringBuffer.Capacity) + { + throw new ArchiveOperationException( + $"StartRecording requires a ring buffer of at least {minBufferSize.Value} bytes, but the stream was created with capacity {_ringBuffer.Capacity}." + ); + } + + // Mark current position as recording anchor + _recordingStartPosition = streamPosition; + _logicalPosition = streamPosition; + _isRecording = true; + } + + public override bool CanRead => true; + + public override bool CanSeek => !_isPassthrough || stream.CanSeek; + + public override bool CanWrite => _isPassthrough && stream.CanWrite; + + public override void Flush() + { + if (_isPassthrough) + { + stream.Flush(); + return; + } + throw new NotSupportedException(); + } + + public override long Length + { + get + { + if (_isPassthrough) + { + return stream.Length; + } + + if (_ringBuffer is not null) + { + return _ringBuffer.Length; + } + throw new NotSupportedException(); + } + } + + public override long Position + { + get + { + // In passthrough mode, delegate to underlying stream + if (_isPassthrough) + { + return stream.Position; + } + // Use logical position (same for both recording and ring buffer modes) + return _logicalPosition; + } + set + { + // In passthrough mode, delegate to underlying stream + if (_isPassthrough) + { + stream.Position = value; + return; + } + SeekToPosition(value); + } + } + + private void SeekToPosition(long targetPosition) + { + // If we have a recording anchor, allow seeking within the recorded range + if (_recordingStartPosition is not null) + { + if (targetPosition >= _recordingStartPosition.Value && targetPosition <= streamPosition) + { + _logicalPosition = targetPosition; + return; + } + throw new NotSupportedException( + $"Cannot seek to position {targetPosition}. Valid recorded range: " + + $"[{_recordingStartPosition.Value}, {streamPosition}]" + ); + } + + // If ring buffer is enabled (and not recording), check if we can seek within it + if (_ringBuffer is not null) + { + long ringBufferStart = streamPosition - _ringBuffer.Length; + if (targetPosition >= ringBufferStart && targetPosition <= streamPosition) + { + _logicalPosition = targetPosition; + return; + } + throw new NotSupportedException( + $"Cannot seek to position {targetPosition}. Valid ring buffer range: " + + $"[{ringBufferStart}, {streamPosition}]" + ); + } + + // No buffering available + throw new NotSupportedException("Cannot seek on non-buffered stream."); + } + + public override int Read(byte[] buffer, int offset, int count) + { + if (count == 0) + { + return 0; + } + + // In passthrough mode, delegate directly to underlying stream + if (_isPassthrough) + { + return stream.Read(buffer, offset, count); + } + + // If ring buffer exists, use unified buffered read logic + if (_ringBuffer is not null) + { + return ReadWithRingBuffer(buffer, offset, count); + } + + // No buffering - read directly from stream + int read = stream.Read(buffer, offset, count); + streamPosition += read; + _logicalPosition = streamPosition; + return read; + } + + /// + /// Reads data using the ring buffer. If logical position is behind stream position, + /// serves data from the ring buffer first. Handles both recording mode and + /// over-read protection uniformly. + /// + private int ReadWithRingBuffer(byte[] buffer, int offset, int count) + { + int totalRead = 0; + + // If logical position is behind stream position, read from ring buffer first + while (count > 0 && _logicalPosition < streamPosition) + { + long bytesFromEnd = streamPosition - _logicalPosition; + + // Verify data is available in ring buffer + if (!_ringBuffer!.CanReadFromEnd(bytesFromEnd)) + { + throw new ArchiveOperationException( + $"Ring buffer underflow: trying to read {bytesFromEnd} bytes back, " + + $"but buffer only holds {_ringBuffer.Length} bytes." + ); + } + + int available = _ringBuffer.ReadFromEnd(bytesFromEnd, buffer, offset, count); + totalRead += available; + offset += available; + count -= available; + _logicalPosition += available; + } + + // If more data needed and we're caught up, read from underlying stream + if (count > 0 && _logicalPosition == streamPosition) + { + // Use async read if stream doesn't support sync reads (e.g., AsyncOnlyStream) + int read = stream.Read(buffer, offset, count); + if (read > 0) + { + _ringBuffer!.Write(buffer, offset, read); + streamPosition += read; + _logicalPosition += read; + totalRead += read; + } + } + + return totalRead; + } + + public override long Seek(long offset, SeekOrigin origin) + { + // In passthrough mode, delegate to underlying stream + if (_isPassthrough) + { + return stream.Seek(offset, origin); + } + + long targetPosition = origin switch + { + SeekOrigin.Begin => offset, + SeekOrigin.Current => Position + offset, + SeekOrigin.End => throw new NotSupportedException("Seeking from end is not supported."), + _ => throw new ArgumentOutOfRangeException(nameof(origin)), + }; + + SeekToPosition(targetPosition); + return targetPosition; + } + + public override void SetLength(long value) + { + if (_isPassthrough) + { + stream.SetLength(value); + return; + } + throw new NotSupportedException(); + } + + public override void Write(byte[] buffer, int offset, int count) + { + if (_isPassthrough) + { + stream.Write(buffer, offset, count); + return; + } + throw new NotSupportedException(); + } + +#if !LEGACY_DOTNET + public override int Read(Span buffer) + { + if (_isPassthrough) + { + return stream.Read(buffer); + } + // Fall back to base implementation for buffered modes + return base.Read(buffer); + } + + public override void Write(ReadOnlySpan buffer) + { + if (_isPassthrough) + { + stream.Write(buffer); + return; + } + throw new NotSupportedException(); + } +#endif +} diff --git a/src/SharpCompress/IO/SourceStream.Async.cs b/src/SharpCompress/IO/SourceStream.Async.cs new file mode 100644 index 00000000..73576fc2 --- /dev/null +++ b/src/SharpCompress/IO/SourceStream.Async.cs @@ -0,0 +1,110 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.IO; + +public partial class SourceStream +{ + public override async Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + if (count <= 0) + { + return 0; + } + + var total = count; + var r = -1; + + while (count != 0 && r != 0) + { + r = await Current + .ReadAsync( + buffer, + offset, + (int)Math.Min(count, Current.Length - Current.Position), + cancellationToken + ) + .ConfigureAwait(false); + count -= r; + offset += r; + + if (!IsVolumes && count != 0 && Current.Position == Current.Length) + { + var length = Current.Length; + + // Load next file if present + if (!SetStream(_stream + 1)) + { + break; + } + + // Current stream switched + // Add length of previous stream + _prevSize += length; + Current.Seek(0, SeekOrigin.Begin); + r = -1; //BugFix: reset to allow loop if count is still not 0 - was breaking split zipx (lzma xz etc) + } + } + + return total - count; + } + +#if !LEGACY_DOTNET + + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + if (buffer.Length <= 0) + { + return 0; + } + + var total = buffer.Length; + var count = buffer.Length; + var offset = 0; + var r = -1; + + while (count != 0 && r != 0) + { + r = await Current + .ReadAsync( + buffer.Slice(offset, (int)Math.Min(count, Current.Length - Current.Position)), + cancellationToken + ) + .ConfigureAwait(false); + count -= r; + offset += r; + + if (!IsVolumes && count != 0 && Current.Position == Current.Length) + { + var length = Current.Length; + + // Load next file if present + if (!SetStream(_stream + 1)) + { + break; + } + + // Current stream switched + // Add length of previous stream + _prevSize += length; + Current.Seek(0, SeekOrigin.Begin); + r = -1; + } + } + + return total - count; + } +#endif +} diff --git a/src/SharpCompress/IO/SourceStream.cs b/src/SharpCompress/IO/SourceStream.cs index 26ce0af1..4a837713 100644 --- a/src/SharpCompress/IO/SourceStream.cs +++ b/src/SharpCompress/IO/SourceStream.cs @@ -2,17 +2,22 @@ using System; using System.Collections.Generic; using System.IO; using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; using SharpCompress.Readers; namespace SharpCompress.IO; -public class SourceStream : Stream +public partial class SourceStream : Stream, IStreamStack { + Stream IStreamStack.BaseStream() => _streams[_stream]; + private long _prevSize; private readonly List _files; private readonly List _streams; - private readonly Func _getFilePart; - private readonly Func _getStreamPart; + private readonly Func? _getFilePart; + private readonly Func? _getStreamPart; private int _stream; public SourceStream(FileInfo file, Func getPart, ReaderOptions options) @@ -38,8 +43,8 @@ public class SourceStream : Stream if (!IsFileMode) { _streams.Add(stream!); - _getStreamPart = getStreamPart!; - _getFilePart = _ => null!; + _getStreamPart = getStreamPart; + _getFilePart = _ => null; if (stream is FileStream fileStream) { _files.Add(new FileInfo(fileStream.Name)); @@ -49,8 +54,8 @@ public class SourceStream : Stream { _files.Add(file!); _streams.Add(_files[0].OpenRead()); - _getFilePart = getFilePart!; - _getStreamPart = _ => null!; + _getFilePart = getFilePart; + _getStreamPart = _ => null; } _stream = 0; _prevSize = 0; @@ -78,7 +83,7 @@ public class SourceStream : Stream { if (IsFileMode) { - var f = _getFilePart(_streams.Count); + var f = _getFilePart.NotNull("GetFilePart is null")(_streams.Count); if (f == null) { _stream = _streams.Count - 1; @@ -90,7 +95,7 @@ public class SourceStream : Stream } else { - var s = _getStreamPart(_streams.Count); + var s = _getStreamPart.NotNull("GetStreamPart is null")(_streams.Count); if (s == null) { _stream = _streams.Count - 1; @@ -196,8 +201,26 @@ public class SourceStream : Stream SetStream(0); while (_prevSize + Current.Length < pos) { - _prevSize += Current.Length; - SetStream(_stream + 1); + var currentLength = Current.Length; + _prevSize += currentLength; + + if (!SetStream(_stream + 1)) + { + // No more streams available, cannot seek to requested position + throw new ArchiveOperationException( + $"Cannot seek to position {pos}. End of stream reached at position {_prevSize}." + ); + } + + // Safety check: if we have a zero-length stream and we're still not + // making progress toward the target position, we're in an invalid state + if (currentLength <= 0 && Current.Length <= 0) + { + // Both old and new stream have zero length - cannot make progress + throw new ArchiveOperationException( + $"Cannot seek to position {pos}. Encountered zero-length streams at position {_prevSize}." + ); + } } } @@ -216,7 +239,7 @@ public class SourceStream : Stream public override void Close() { - if (IsFileMode || !ReaderOptions.LeaveStreamOpen) //close if file mode or options specify it + if (!ReaderOptions.LeaveStreamOpen) //close if file mode or options specify it { foreach (var stream in _streams) { diff --git a/src/SharpCompress/IO/StreamingMode.cs b/src/SharpCompress/IO/StreamingMode.cs index 0fdb674d..33f7d236 100644 --- a/src/SharpCompress/IO/StreamingMode.cs +++ b/src/SharpCompress/IO/StreamingMode.cs @@ -3,5 +3,5 @@ namespace SharpCompress.IO; public enum StreamingMode { Streaming, - Seekable + Seekable, } diff --git a/src/SharpCompress/Lazy.cs b/src/SharpCompress/Lazy.cs deleted file mode 100644 index 7a6abd55..00000000 --- a/src/SharpCompress/Lazy.cs +++ /dev/null @@ -1,27 +0,0 @@ -#nullable disable - -using System; - -namespace SharpCompress; - -public class Lazy -{ - private readonly Func _lazyFunc; - private bool _evaluated; - private T _value; - - public Lazy(Func lazyFunc) => _lazyFunc = lazyFunc; - - public T Value - { - get - { - if (!_evaluated) - { - _value = _lazyFunc(); - _evaluated = true; - } - return _value; - } - } -} diff --git a/src/SharpCompress/LazyAsyncReadOnlyCollection.cs b/src/SharpCompress/LazyAsyncReadOnlyCollection.cs new file mode 100644 index 00000000..88d08fdf --- /dev/null +++ b/src/SharpCompress/LazyAsyncReadOnlyCollection.cs @@ -0,0 +1,102 @@ +using System; +using System.Collections; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress; + +internal sealed class LazyAsyncReadOnlyCollection(IAsyncEnumerable source) + : IAsyncEnumerable +{ + private readonly List _backing = new(); + private readonly IAsyncEnumerator _source = source.GetAsyncEnumerator(); + private bool _fullyLoaded; + + private class LazyLoader( + LazyAsyncReadOnlyCollection lazyReadOnlyCollection, + CancellationToken cancellationToken + ) : IAsyncEnumerator + { + private bool _disposed; + private int _index = -1; + + public ValueTask DisposeAsync() + { + if (!_disposed) + { + _disposed = true; + } + return default; + } + + public async ValueTask MoveNextAsync() + { + cancellationToken.ThrowIfCancellationRequested(); + if (_index + 1 < lazyReadOnlyCollection._backing.Count) + { + _index++; + return true; + } + if ( + !lazyReadOnlyCollection._fullyLoaded + && await lazyReadOnlyCollection._source.MoveNextAsync().ConfigureAwait(false) + ) + { + lazyReadOnlyCollection._backing.Add(lazyReadOnlyCollection._source.Current); + _index++; + return true; + } + lazyReadOnlyCollection._fullyLoaded = true; + return false; + } + + #region IEnumerator Members + + public T Current => lazyReadOnlyCollection._backing[_index]; + + #endregion + + #region IDisposable Members + + public void Dispose() + { + if (!_disposed) + { + _disposed = true; + } + } + + #endregion + } + + internal async ValueTask EnsureFullyLoaded() + { + if (!_fullyLoaded) + { + var loader = new LazyLoader(this, CancellationToken.None); + while (await loader.MoveNextAsync().ConfigureAwait(false)) + { + // Intentionally empty + } + _fullyLoaded = true; + } + } + + internal IEnumerable GetLoaded() => _backing; + + #region ICollection Members + + public void Add(T item) => throw new NotSupportedException(); + + public void Clear() => throw new NotSupportedException(); + + public bool IsReadOnly => true; + + public bool Remove(T item) => throw new NotSupportedException(); + + #endregion + + public IAsyncEnumerator GetAsyncEnumerator(CancellationToken cancellationToken = default) => + new LazyLoader(this, cancellationToken); +} diff --git a/src/SharpCompress/LazyReadOnlyCollection.cs b/src/SharpCompress/LazyReadOnlyCollection.cs index 8f5ee834..2a7e95ff 100644 --- a/src/SharpCompress/LazyReadOnlyCollection.cs +++ b/src/SharpCompress/LazyReadOnlyCollection.cs @@ -1,5 +1,3 @@ -#nullable disable - using System; using System.Collections; using System.Collections.Generic; @@ -8,24 +6,24 @@ namespace SharpCompress; internal sealed class LazyReadOnlyCollection : ICollection { - private readonly List backing = new List(); - private readonly IEnumerator source; - private bool fullyLoaded; + private readonly List _backing = new(); + private readonly IEnumerator _source; + private bool _fullyLoaded; - public LazyReadOnlyCollection(IEnumerable source) => this.source = source.GetEnumerator(); + public LazyReadOnlyCollection(IEnumerable source) => _source = source.GetEnumerator(); private class LazyLoader : IEnumerator { - private readonly LazyReadOnlyCollection lazyReadOnlyCollection; - private bool disposed; - private int index = -1; + private readonly LazyReadOnlyCollection _lazyReadOnlyCollection; + private bool _disposed; + private int _index = -1; internal LazyLoader(LazyReadOnlyCollection lazyReadOnlyCollection) => - this.lazyReadOnlyCollection = lazyReadOnlyCollection; + _lazyReadOnlyCollection = lazyReadOnlyCollection; #region IEnumerator Members - public T Current => lazyReadOnlyCollection.backing[index]; + public T Current => _lazyReadOnlyCollection._backing[_index]; #endregion @@ -33,9 +31,9 @@ internal sealed class LazyReadOnlyCollection : ICollection public void Dispose() { - if (!disposed) + if (!_disposed) { - disposed = true; + _disposed = true; } } @@ -43,22 +41,22 @@ internal sealed class LazyReadOnlyCollection : ICollection #region IEnumerator Members - object IEnumerator.Current => Current; + object IEnumerator.Current => Current!; public bool MoveNext() { - if (index + 1 < lazyReadOnlyCollection.backing.Count) + if (_index + 1 < _lazyReadOnlyCollection._backing.Count) { - index++; + _index++; return true; } - if (!lazyReadOnlyCollection.fullyLoaded && lazyReadOnlyCollection.source.MoveNext()) + if (!_lazyReadOnlyCollection._fullyLoaded && _lazyReadOnlyCollection._source.MoveNext()) { - lazyReadOnlyCollection.backing.Add(lazyReadOnlyCollection.source.Current); - index++; + _lazyReadOnlyCollection._backing.Add(_lazyReadOnlyCollection._source.Current); + _index++; return true; } - lazyReadOnlyCollection.fullyLoaded = true; + _lazyReadOnlyCollection._fullyLoaded = true; return false; } @@ -69,14 +67,14 @@ internal sealed class LazyReadOnlyCollection : ICollection internal void EnsureFullyLoaded() { - if (!fullyLoaded) + if (!_fullyLoaded) { this.ForEach(x => { }); - fullyLoaded = true; + _fullyLoaded = true; } } - internal IEnumerable GetLoaded() => backing; + internal IEnumerable GetLoaded() => _backing; #region ICollection Members @@ -87,13 +85,13 @@ internal sealed class LazyReadOnlyCollection : ICollection public bool Contains(T item) { EnsureFullyLoaded(); - return backing.Contains(item); + return _backing.Contains(item); } public void CopyTo(T[] array, int arrayIndex) { EnsureFullyLoaded(); - backing.CopyTo(array, arrayIndex); + _backing.CopyTo(array, arrayIndex); } public int Count @@ -101,7 +99,7 @@ internal sealed class LazyReadOnlyCollection : ICollection get { EnsureFullyLoaded(); - return backing.Count; + return _backing.Count; } } diff --git a/src/SharpCompress/NotNullExtensions.cs b/src/SharpCompress/NotNullExtensions.cs new file mode 100644 index 00000000..4c25e008 --- /dev/null +++ b/src/SharpCompress/NotNullExtensions.cs @@ -0,0 +1,85 @@ +using System; +using System.Collections.Generic; +using System.Diagnostics.CodeAnalysis; +using System.Linq; +using System.Runtime.CompilerServices; + +namespace SharpCompress; + +internal static class NotNullExtensions +{ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static IEnumerable Empty(this IEnumerable? source) => source ?? []; + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static IEnumerable Empty(this T? source) + { + if (source is null) + { + return []; + } + return source.AsEnumerable(); + } + +#if LEGACY_DOTNET + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static T NotNull(this T? obj, string? message = null) + where T : class + { + if (obj is null) + { + throw new ArgumentNullException(message ?? "Value is null"); + } + return obj; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static T NotNull(this T? obj, string? message = null) + where T : struct + { + if (obj is null) + { + throw new ArgumentNullException(message ?? "Value is null"); + } + return obj.Value; + } +#else + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static T NotNull( + [NotNull] this T? obj, + [CallerArgumentExpression(nameof(obj))] string? paramName = null + ) + where T : class + { + ThrowHelper.ThrowIfNull(obj, paramName); + return obj; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static T NotNull( + [NotNull] this T? obj, + [CallerArgumentExpression(nameof(obj))] string? paramName = null + ) + where T : struct + { + if (!obj.HasValue) + { + throw new ArgumentNullException(paramName); + } + + return obj.Value; + } +#endif + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static string NotNullOrEmpty(this string obj, string name) + { + obj.NotNull(name); + if (obj.Length == 0) + { + throw new ArgumentException("String is empty.", name); + } + return obj; + } +} diff --git a/src/SharpCompress/Polyfills/AsyncEnumerableExtensions.cs b/src/SharpCompress/Polyfills/AsyncEnumerableExtensions.cs new file mode 100644 index 00000000..df716ecd --- /dev/null +++ b/src/SharpCompress/Polyfills/AsyncEnumerableExtensions.cs @@ -0,0 +1,181 @@ +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress; + +public static class AsyncEnumerableEx +{ + public static async IAsyncEnumerable Empty() + where T : notnull + { + await Task.Yield(); + yield break; + } +} + +public static class EnumerableExtensions +{ + public static async IAsyncEnumerable ToAsyncEnumerable(this IEnumerable source) + { + foreach (var item in source) + { + yield return item; + } + } +} + +public static class AsyncEnumerableExtensions +{ +#if !NET10_0_OR_GREATER + extension(IAsyncEnumerable source) + { + public async IAsyncEnumerable Select(Func selector) + { + await foreach (var element in source.ConfigureAwait(false)) + { + yield return selector(element); + } + } + + public async ValueTask CountAsync(CancellationToken cancellationToken = default) + { + await using var e = source.GetAsyncEnumerator(cancellationToken); + + var count = 0; + while (await e.MoveNextAsync().ConfigureAwait(false)) + { + checked + { + count++; + } + } + + return count; + } + + public async IAsyncEnumerable Take(int count) + { + await foreach (var element in source.ConfigureAwait(false)) + { + yield return element; + + if (--count == 0) + { + break; + } + } + } + + public async ValueTask> ToListAsync() + { + var list = new List(); + await foreach (var item in source.ConfigureAwait(false)) + { + list.Add(item); + } + return list; + } + + public async ValueTask AllAsync(Func predicate) + { + await foreach (var item in source.ConfigureAwait(false)) + { + if (!predicate(item)) + { + return false; + } + } + + return true; + } + + public async IAsyncEnumerable Where(Func predicate) + { + await foreach (var item in source.ConfigureAwait(false)) + { + if (predicate(item)) + { + yield return item; + } + } + } + + public async ValueTask SingleAsync(Func? predicate = null) + { + IAsyncEnumerator enumerator; + if (predicate is null) + { + enumerator = source.GetAsyncEnumerator(); + } + else + { + enumerator = source.Where(predicate).GetAsyncEnumerator(); + } + + if (!await enumerator.MoveNextAsync().ConfigureAwait(false)) + { + throw new ArchiveOperationException("The source sequence is empty."); + } + var value = enumerator.Current; + if (await enumerator.MoveNextAsync().ConfigureAwait(false)) + { + throw new ArchiveOperationException( + "The source sequence contains more than one element." + ); + } + return value; + } + + public async ValueTask FirstAsync() + { + await foreach (var item in source.ConfigureAwait(false)) + { + return item; + } + throw new ArchiveOperationException("The source sequence is empty."); + } + + public async ValueTask FirstOrDefaultAsync( + CancellationToken cancellationToken = default + ) + { + await foreach ( + var item in source.WithCancellation(cancellationToken).ConfigureAwait(false) + ) + { + return item; + } + + return default; + } + } +#endif + + public static async IAsyncEnumerable CastAsync( + this IAsyncEnumerable source + ) + where TResult : class + { + await foreach (var item in source.ConfigureAwait(false)) + { + yield return (item as TResult).NotNull(); + } + } + + public static async ValueTask AggregateAsync( + this IAsyncEnumerable source, + TAccumulate seed, + Func func + ) + { + var result = seed; + await foreach (var element in source.ConfigureAwait(false)) + { + result = func(result, element); + } + return result; + } +} diff --git a/src/SharpCompress/Polyfills/BinaryReaderExtensions.cs b/src/SharpCompress/Polyfills/BinaryReaderExtensions.cs new file mode 100644 index 00000000..24a0eaf6 --- /dev/null +++ b/src/SharpCompress/Polyfills/BinaryReaderExtensions.cs @@ -0,0 +1,65 @@ +using System; +using System.Buffers; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress; + +public static class BinaryReaderExtensions +{ + extension(BinaryReader reader) + { + public async ValueTask ReadByteAsync(CancellationToken cancellationToken = default) + { + var buffer = new byte[1]; + await reader + .BaseStream.ReadExactAsync(buffer, 0, 1, cancellationToken) + .ConfigureAwait(false); + return buffer[0]; + } + + public async ValueTask ReadBytesAsync( + int count, + CancellationToken cancellationToken = default + ) + { + if (count < 0) + { + throw new ArgumentOutOfRangeException(nameof(count), "Count must be non-negative."); + } + + if (count == 0) + { + return Array.Empty(); + } + + // For small allocations, direct allocation is more efficient than pooling + // due to ArrayPool overhead and the need to copy data to return array + if (count <= 256) + { + var bytes = new byte[count]; + await reader + .BaseStream.ReadExactAsync(bytes, 0, count, cancellationToken) + .ConfigureAwait(false); + return bytes; + } + + // For larger allocations, use ArrayPool to reduce GC pressure + var buffer = ArrayPool.Shared.Rent(count); + try + { + await reader + .BaseStream.ReadExactAsync(buffer, 0, count, cancellationToken) + .ConfigureAwait(false); + var bytes = new byte[count]; + Array.Copy(buffer, 0, bytes, 0, count); + return bytes; + } + finally + { + ArrayPool.Shared.Return(buffer); + } + } + } +} diff --git a/src/SharpCompress/Polyfills/StreamExtensions.cs b/src/SharpCompress/Polyfills/StreamExtensions.cs index ab118a4f..c6e66b82 100644 --- a/src/SharpCompress/Polyfills/StreamExtensions.cs +++ b/src/SharpCompress/Polyfills/StreamExtensions.cs @@ -1,46 +1,72 @@ -#if NETFRAMEWORK || NETSTANDARD2_0 - using System; using System.Buffers; using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.IO; namespace SharpCompress; -internal static class StreamExtensions +public static class StreamExtensions { - internal static int Read(this Stream stream, Span buffer) + extension(Stream stream) { - var temp = ArrayPool.Shared.Rent(buffer.Length); - - try + public void Skip(long advanceAmount) { - var read = stream.Read(temp, 0, buffer.Length); + if (stream.CanSeek && stream is not SharpCompressStream) + { + stream.Position += advanceAmount; + return; + } - temp.AsSpan(0, read).CopyTo(buffer); - - return read; + using var readOnlySubStream = new ReadOnlySubStream(stream, advanceAmount); + readOnlySubStream.CopyTo(Stream.Null); } - finally + + public void Skip() => stream.CopyTo(Stream.Null); + + public async ValueTask SkipAsync(CancellationToken cancellationToken = default) { - ArrayPool.Shared.Return(temp); + cancellationToken.ThrowIfCancellationRequested(); +#if NET6_0_OR_GREATER + await stream.CopyToAsync(Stream.Null, cancellationToken).ConfigureAwait(false); +#else + await stream.CopyToAsync(Stream.Null).ConfigureAwait(false); +#endif } - } - internal static void Write(this Stream stream, ReadOnlySpan buffer) - { - var temp = ArrayPool.Shared.Rent(buffer.Length); - - buffer.CopyTo(temp); - - try + internal int Read(Span buffer) { - stream.Write(temp, 0, buffer.Length); + var temp = ArrayPool.Shared.Rent(buffer.Length); + + try + { + var read = stream.Read(temp, 0, buffer.Length); + + temp.AsSpan(0, read).CopyTo(buffer); + + return read; + } + finally + { + ArrayPool.Shared.Return(temp); + } } - finally + + internal void Write(ReadOnlySpan buffer) { - ArrayPool.Shared.Return(temp); + var temp = ArrayPool.Shared.Rent(buffer.Length); + + buffer.CopyTo(temp); + + try + { + stream.Write(temp, 0, buffer.Length); + } + finally + { + ArrayPool.Shared.Return(temp); + } } } } - -#endif diff --git a/src/SharpCompress/Polyfills/StringExtensions.cs b/src/SharpCompress/Polyfills/StringExtensions.cs index 57d1e1bc..84e1cbbe 100644 --- a/src/SharpCompress/Polyfills/StringExtensions.cs +++ b/src/SharpCompress/Polyfills/StringExtensions.cs @@ -1,4 +1,6 @@ -#if NETFRAMEWORK || NETSTANDARD2_0 +using System; + +#if LEGACY_DOTNET namespace SharpCompress; @@ -10,4 +12,9 @@ internal static class StringExtensions internal static bool Contains(this string text, char value) => text.IndexOf(value) > -1; } +[AttributeUsage( + AttributeTargets.Parameter | AttributeTargets.Property | AttributeTargets.ReturnValue +)] +public class NotNullAttribute : Attribute; + #endif diff --git a/src/SharpCompress/Providers/CompressionContext.cs b/src/SharpCompress/Providers/CompressionContext.cs new file mode 100644 index 00000000..e74a7e9c --- /dev/null +++ b/src/SharpCompress/Providers/CompressionContext.cs @@ -0,0 +1,66 @@ +using System.IO; +using SharpCompress.Common.Options; + +namespace SharpCompress.Providers; + +/// +/// Provides context information for compression operations. +/// Carries format-specific parameters that some compression types require. +/// +public sealed record CompressionContext +{ + /// + /// The size of the input data, or -1 if unknown. + /// + public long InputSize { get; set; } = -1; + + /// + /// The expected output size, or -1 if unknown. + /// + public long OutputSize { get; set; } = -1; + + /// + /// Properties bytes for the compression format (e.g., LZMA properties). + /// + public byte[]? Properties { get; set; } + + /// + /// Whether the underlying stream supports seeking. + /// + public bool CanSeek { get; set; } + + /// + /// Additional format-specific options. + /// + /// + /// This value is consumed by provider implementations that need caller-supplied metadata + /// that is not tied to ReaderOptions. For archive header encoding, use instead. + /// Examples of valid FormatOptions values include compression properties (e.g., LZMA properties), + /// format flags, or algorithm-specific configuration. + /// + public object? FormatOptions { get; set; } + + /// + /// Creates a CompressionContext from a stream. + /// + /// The stream to extract context from. + /// A CompressionContext populated from the stream. + public static CompressionContext FromStream(Stream stream) => + new() { CanSeek = stream.CanSeek, InputSize = stream.CanSeek ? stream.Length : -1 }; + + /// + /// Reader options for accessing archive metadata such as header encoding. + /// + public IReaderOptions? ReaderOptions { get; set; } + + /// + /// Returns a new with the specified reader options. + /// + /// The reader options to set. + /// A new instance. + public CompressionContext WithReaderOptions(IReaderOptions? readerOptions) => + this with + { + ReaderOptions = readerOptions, + }; +} diff --git a/src/SharpCompress/Providers/CompressionContextExtensions.cs b/src/SharpCompress/Providers/CompressionContextExtensions.cs new file mode 100644 index 00000000..fdfa5dc4 --- /dev/null +++ b/src/SharpCompress/Providers/CompressionContextExtensions.cs @@ -0,0 +1,18 @@ +using System.Text; +using SharpCompress.Common; +using SharpCompress.Common.Options; + +namespace SharpCompress.Providers; + +public static class CompressionContextExtensions +{ + /// + /// Resolves the archive header encoding from . + /// + /// + /// Returns when ReaderOptions is set, + /// otherwise falls back to UTF-8. + /// + public static Encoding ResolveArchiveEncoding(this CompressionContext context) => + context.ReaderOptions?.ArchiveEncoding.GetEncoding() ?? Encoding.UTF8; +} diff --git a/src/SharpCompress/Providers/CompressionProviderBase.cs b/src/SharpCompress/Providers/CompressionProviderBase.cs new file mode 100644 index 00000000..e98ae72d --- /dev/null +++ b/src/SharpCompress/Providers/CompressionProviderBase.cs @@ -0,0 +1,130 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Providers; + +/// +/// Base class for compression providers that provides default async implementations +/// delegating to synchronous methods. Providers can inherit from this class for +/// simpler implementations or implement ICompressionProvider directly for full control. +/// +/// +/// +/// This base class implements the async methods by calling the synchronous versions. +/// Providers that need true async implementations should override these methods. +/// +/// +public abstract class CompressionProviderBase : ICompressionProvider +{ + /// + public abstract CompressionType CompressionType { get; } + + /// + public abstract bool SupportsCompression { get; } + + /// + public abstract bool SupportsDecompression { get; } + + /// + public abstract Stream CreateCompressStream(Stream destination, int compressionLevel); + + /// + public virtual Stream CreateCompressStream( + Stream destination, + int compressionLevel, + CompressionContext context + ) => CreateCompressStream(destination, compressionLevel); + + /// + public abstract Stream CreateDecompressStream(Stream source); + + /// + public virtual Stream CreateDecompressStream(Stream source, CompressionContext context) => + CreateDecompressStream(source); + + /// + /// Asynchronously creates a compression stream. + /// Default implementation delegates to the synchronous CreateCompressStream. + /// + public virtual ValueTask CreateCompressStreamAsync( + Stream destination, + int compressionLevel, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new ValueTask(CreateCompressStream(destination, compressionLevel)); + } + + /// + /// Asynchronously creates a compression stream with context. + /// Default implementation delegates to the synchronous CreateCompressStream with context. + /// + public virtual ValueTask CreateCompressStreamAsync( + Stream destination, + int compressionLevel, + CompressionContext context, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new ValueTask(CreateCompressStream(destination, compressionLevel, context)); + } + + /// + /// Asynchronously creates a decompression stream. + /// Default implementation delegates to the synchronous CreateDecompressStream. + /// + public virtual ValueTask CreateDecompressStreamAsync( + Stream source, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new ValueTask(CreateDecompressStream(source)); + } + + /// + /// Asynchronously creates a decompression stream with context. + /// Default implementation delegates to the synchronous CreateDecompressStream with context. + /// + public virtual ValueTask CreateDecompressStreamAsync( + Stream source, + CompressionContext context, + CancellationToken cancellationToken = default + ) + { + return CreateDecompressStreamAsync(source, cancellationToken); + } + + protected static void ValidateRequiredSizes(CompressionContext context, string algorithmName) + { + if (context.InputSize < 0 || context.OutputSize < 0) + { + throw new ArgumentException( + $"{algorithmName} decompression requires InputSize and OutputSize in CompressionContext.", + nameof(context) + ); + } + } + + protected static T RequireFormatOption( + CompressionContext context, + string algorithmName, + string optionName + ) + { + if (context.FormatOptions is not T options) + { + throw new ArgumentException( + $"{algorithmName} decompression requires {optionName} in CompressionContext.FormatOptions.", + nameof(context) + ); + } + + return options; + } +} diff --git a/src/SharpCompress/Providers/CompressionProviderRegistry.cs b/src/SharpCompress/Providers/CompressionProviderRegistry.cs new file mode 100644 index 00000000..ea0e217a --- /dev/null +++ b/src/SharpCompress/Providers/CompressionProviderRegistry.cs @@ -0,0 +1,318 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Providers.Default; + +namespace SharpCompress.Providers; + +/// +/// A registry of compression providers, keyed by CompressionType. +/// Used to resolve which implementation to use for a given compression type. +/// +/// +/// +/// This class is immutable. Use the With method to create modified copies +/// that add or replace providers: +/// +/// +/// var customRegistry = CompressionProviderRegistry.Default +/// .With(new MyCustomGZipProvider()); +/// var options = new WriterOptions(CompressionType.GZip) +/// { +/// Providers = customRegistry +/// }; +/// +/// +public sealed class CompressionProviderRegistry +{ + /// + /// The default registry using SharpCompress internal implementations. + /// + public static CompressionProviderRegistry Default { get; } = CreateDefault(); + + /// + /// The empty registry for tests + /// + public static CompressionProviderRegistry Empty { get; } = CreateEmpty(); + + private readonly Dictionary _providers; + + private CompressionProviderRegistry( + Dictionary providers + ) => _providers = providers; + + /// + /// Gets the provider for a given compression type, or null if none is registered. + /// + /// The compression type to look up. + /// The provider for the type, or null if not found. + public ICompressionProvider? GetProvider(CompressionType type) + { + _providers.TryGetValue(type, out var provider); + return provider; + } + + /// + /// Creates a compression stream for the specified type. + /// + /// The compression type. + /// The destination stream. + /// The compression level. + /// A compression stream. + /// If no provider is registered for the type. + /// If the provider does not support compression. + public Stream CreateCompressStream(CompressionType type, Stream destination, int level) + { + var provider = GetProvider(type); + if (provider is null) + { + throw new ArchiveOperationException( + $"No compression provider registered for type: {type}" + ); + } + return provider.CreateCompressStream(destination, level); + } + + /// + /// Creates a decompression stream for the specified type. + /// + /// The compression type. + /// The source stream. + /// A decompression stream. + /// If no provider is registered for the type. + /// If the provider does not support decompression. + public Stream CreateDecompressStream(CompressionType type, Stream source) + { + var provider = GetProvider(type); + if (provider is null) + { + throw new ArchiveOperationException( + $"No compression provider registered for type: {type}" + ); + } + return provider.CreateDecompressStream(source); + } + + /// + /// Creates a compression stream for the specified type with context. + /// + /// The compression type. + /// The destination stream. + /// The compression level. + /// Context information for the compression. + /// A compression stream. + /// If no provider is registered for the type. + /// If the provider does not support compression. + public Stream CreateCompressStream( + CompressionType type, + Stream destination, + int level, + CompressionContext context + ) + { + var provider = GetProvider(type); + if (provider is null) + { + throw new ArchiveOperationException( + $"No compression provider registered for type: {type}" + ); + } + return provider.CreateCompressStream(destination, level, context); + } + + /// + /// Creates a decompression stream for the specified type with context. + /// + /// The compression type. + /// The source stream. + /// Context information for the decompression. + /// A decompression stream. + /// If no provider is registered for the type. + /// If the provider does not support decompression. + public Stream CreateDecompressStream( + CompressionType type, + Stream source, + CompressionContext context + ) + { + var provider = GetProvider(type); + if (provider is null) + { + throw new ArchiveOperationException( + $"No compression provider registered for type: {type}" + ); + } + return provider.CreateDecompressStream(source, context); + } + + /// + /// Asynchronously creates a compression stream for the specified type. + /// + /// The compression type. + /// The destination stream. + /// The compression level. + /// Cancellation token. + /// A task containing the compression stream. + /// If no provider is registered for the type. + /// If the provider does not support compression. + public ValueTask CreateCompressStreamAsync( + CompressionType type, + Stream destination, + int level, + CancellationToken cancellationToken = default + ) + { + var provider = GetProvider(type); + if (provider is null) + { + throw new ArchiveOperationException( + $"No compression provider registered for type: {type}" + ); + } + return provider.CreateCompressStreamAsync(destination, level, cancellationToken); + } + + /// + /// Asynchronously creates a decompression stream for the specified type. + /// + /// The compression type. + /// The source stream. + /// Cancellation token. + /// A task containing the decompression stream. + /// If no provider is registered for the type. + /// If the provider does not support decompression. + public ValueTask CreateDecompressStreamAsync( + CompressionType type, + Stream source, + CancellationToken cancellationToken = default + ) + { + var provider = GetProvider(type); + if (provider is null) + { + throw new ArchiveOperationException( + $"No compression provider registered for type: {type}" + ); + } + return provider.CreateDecompressStreamAsync(source, cancellationToken); + } + + /// + /// Asynchronously creates a compression stream for the specified type with context. + /// + /// The compression type. + /// The destination stream. + /// The compression level. + /// Context information for the compression. + /// Cancellation token. + /// A task containing the compression stream. + /// If no provider is registered for the type. + /// If the provider does not support compression. + public ValueTask CreateCompressStreamAsync( + CompressionType type, + Stream destination, + int level, + CompressionContext context, + CancellationToken cancellationToken = default + ) + { + var provider = GetProvider(type); + if (provider is null) + { + throw new ArchiveOperationException( + $"No compression provider registered for type: {type}" + ); + } + return provider.CreateCompressStreamAsync(destination, level, context, cancellationToken); + } + + /// + /// Asynchronously creates a decompression stream for the specified type with context. + /// + /// The compression type. + /// The source stream. + /// Context information for the decompression. + /// Cancellation token. + /// A task containing the decompression stream. + /// If no provider is registered for the type. + /// If the provider does not support decompression. + public ValueTask CreateDecompressStreamAsync( + CompressionType type, + Stream source, + CompressionContext context, + CancellationToken cancellationToken = default + ) + { + var provider = GetProvider(type); + if (provider is null) + { + throw new ArchiveOperationException( + $"No compression provider registered for type: {type}" + ); + } + return provider.CreateDecompressStreamAsync(source, context, cancellationToken); + } + + /// + /// Gets the provider as an ICompressionProviderHooks if it supports complex initialization. + /// + /// The compression type. + /// The compressing provider, or null if the provider doesn't support complex initialization. + public ICompressionProviderHooks? GetCompressingProvider(CompressionType type) + { + var provider = GetProvider(type); + return provider as ICompressionProviderHooks; + } + + /// + /// Creates a new registry with the specified provider added or replaced. + /// + /// The provider to add or replace. + /// A new registry instance with the provider included. + /// If provider is null. + public CompressionProviderRegistry With(ICompressionProvider provider) + { + ThrowHelper.ThrowIfNull(provider); + + var newProviders = new Dictionary(_providers) + { + [provider.CompressionType] = provider, + }; + + return new CompressionProviderRegistry(newProviders); + } + + private static CompressionProviderRegistry CreateDefault() + { + var providers = new Dictionary + { + [CompressionType.Deflate] = new DeflateCompressionProvider(), + [CompressionType.GZip] = new GZipCompressionProvider(), + [CompressionType.BZip2] = new BZip2CompressionProvider(), + [CompressionType.ZStandard] = new ZStandardCompressionProvider(), + [CompressionType.LZip] = new LZipCompressionProvider(), + [CompressionType.Xz] = new XzCompressionProvider(), + [CompressionType.Lzw] = new LzwCompressionProvider(), + [CompressionType.Deflate64] = new Deflate64CompressionProvider(), + [CompressionType.Shrink] = new ShrinkCompressionProvider(), + [CompressionType.Reduce1] = new Reduce1CompressionProvider(), + [CompressionType.Reduce2] = new Reduce2CompressionProvider(), + [CompressionType.Reduce3] = new Reduce3CompressionProvider(), + [CompressionType.Reduce4] = new Reduce4CompressionProvider(), + [CompressionType.Explode] = new ExplodeCompressionProvider(), + [CompressionType.LZMA] = new LzmaCompressingProvider(), + [CompressionType.PPMd] = new PpmdCompressingProvider(), + }; + + return new CompressionProviderRegistry(providers); + } + + private static CompressionProviderRegistry CreateEmpty() + { + var providers = new Dictionary(); + return new CompressionProviderRegistry(providers); + } +} diff --git a/src/SharpCompress/Providers/ContextRequiredDecompressionProviderBase.cs b/src/SharpCompress/Providers/ContextRequiredDecompressionProviderBase.cs new file mode 100644 index 00000000..1e7ac74f --- /dev/null +++ b/src/SharpCompress/Providers/ContextRequiredDecompressionProviderBase.cs @@ -0,0 +1,30 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Providers; + +public abstract class ContextRequiredDecompressionProviderBase : DecompressionOnlyProviderBase +{ + protected abstract string DecompressionContextRequirementDescription { get; } + + protected virtual string DecompressionContextRequirementSuffix => string.Empty; + + public sealed override Stream CreateDecompressStream(Stream source) => + throw new ArchiveOperationException( + $"{DecompressionContextRequirementDescription}. " + + $"Use CreateDecompressStream(Stream, CompressionContext) overload{DecompressionContextRequirementSuffix}." + ); + + public sealed override ValueTask CreateDecompressStreamAsync( + Stream source, + CancellationToken cancellationToken = default + ) => + throw new ArchiveOperationException( + $"{DecompressionContextRequirementDescription}. " + + "Use CreateDecompressStreamAsync(Stream, CompressionContext, CancellationToken) " + + $"overload{DecompressionContextRequirementSuffix}." + ); +} diff --git a/src/SharpCompress/Providers/DecompressionOnlyProviderBase.cs b/src/SharpCompress/Providers/DecompressionOnlyProviderBase.cs new file mode 100644 index 00000000..bdd8e561 --- /dev/null +++ b/src/SharpCompress/Providers/DecompressionOnlyProviderBase.cs @@ -0,0 +1,22 @@ +using System; +using System.IO; +using SharpCompress.Common; + +namespace SharpCompress.Providers; + +public abstract class DecompressionOnlyProviderBase : CompressionProviderBase +{ + public override bool SupportsCompression => false; + public override bool SupportsDecompression => true; + + protected abstract string CompressionNotSupportedMessage { get; } + + public sealed override Stream CreateCompressStream(Stream destination, int compressionLevel) => + throw new NotSupportedException(CompressionNotSupportedMessage); + + public sealed override Stream CreateCompressStream( + Stream destination, + int compressionLevel, + CompressionContext context + ) => throw new NotSupportedException(CompressionNotSupportedMessage); +} diff --git a/src/SharpCompress/Providers/Default/BZip2CompressionProvider.cs b/src/SharpCompress/Providers/Default/BZip2CompressionProvider.cs new file mode 100644 index 00000000..29062c03 --- /dev/null +++ b/src/SharpCompress/Providers/Default/BZip2CompressionProvider.cs @@ -0,0 +1,63 @@ +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Compressors; +using SharpCompress.Compressors.BZip2; + +namespace SharpCompress.Providers.Default; + +/// +/// Provides BZip2 compression using SharpCompress's internal implementation. +/// +public sealed class BZip2CompressionProvider : CompressionProviderBase +{ + public override CompressionType CompressionType => CompressionType.BZip2; + public override bool SupportsCompression => true; + public override bool SupportsDecompression => true; + + public override Stream CreateCompressStream(Stream destination, int compressionLevel) + { + // BZip2 doesn't use compressionLevel parameter in this implementation + return BZip2Stream.Create(destination, CompressionMode.Compress, false); + } + + public override async ValueTask CreateCompressStreamAsync( + Stream destination, + int compressionLevel, + CancellationToken cancellationToken = default + ) + { + // BZip2 doesn't use compressionLevel parameter in this implementation + return await BZip2Stream + .CreateAsync( + destination, + CompressionMode.Compress, + false, + false, + cancellationToken: cancellationToken + ) + .ConfigureAwait(false); + } + + public override Stream CreateDecompressStream(Stream source) + { + return BZip2Stream.Create(source, CompressionMode.Decompress, false); + } + + public override async ValueTask CreateDecompressStreamAsync( + Stream source, + CancellationToken cancellationToken = default + ) + { + return await BZip2Stream + .CreateAsync( + source, + CompressionMode.Decompress, + false, + false, + cancellationToken: cancellationToken + ) + .ConfigureAwait(false); + } +} diff --git a/src/SharpCompress/Providers/Default/Deflate64CompressionProvider.cs b/src/SharpCompress/Providers/Default/Deflate64CompressionProvider.cs new file mode 100644 index 00000000..482cdfe0 --- /dev/null +++ b/src/SharpCompress/Providers/Default/Deflate64CompressionProvider.cs @@ -0,0 +1,22 @@ +using System.IO; +using SharpCompress.Common; +using SharpCompress.Compressors; +using SharpCompress.Compressors.Deflate64; + +namespace SharpCompress.Providers.Default; + +/// +/// Provides Deflate64 decompression using SharpCompress's internal implementation. +/// Note: Deflate64 compression is not supported; this provider is decompression-only. +/// +public sealed class Deflate64CompressionProvider : DecompressionOnlyProviderBase +{ + public override CompressionType CompressionType => CompressionType.Deflate64; + protected override string CompressionNotSupportedMessage => + "Deflate64 compression is not supported by SharpCompress's internal implementation."; + + public override Stream CreateDecompressStream(Stream source) + { + return new Deflate64Stream(source, CompressionMode.Decompress); + } +} diff --git a/src/SharpCompress/Providers/Default/DeflateCompressionProvider.cs b/src/SharpCompress/Providers/Default/DeflateCompressionProvider.cs new file mode 100644 index 00000000..3a49899a --- /dev/null +++ b/src/SharpCompress/Providers/Default/DeflateCompressionProvider.cs @@ -0,0 +1,27 @@ +using System.IO; +using SharpCompress.Common; +using SharpCompress.Compressors; +using SharpCompress.Compressors.Deflate; + +namespace SharpCompress.Providers.Default; + +/// +/// Provides Deflate compression using SharpCompress's internal implementation. +/// +public sealed class DeflateCompressionProvider : CompressionProviderBase +{ + public override CompressionType CompressionType => CompressionType.Deflate; + public override bool SupportsCompression => true; + public override bool SupportsDecompression => true; + + public override Stream CreateCompressStream(Stream destination, int compressionLevel) + { + var level = (CompressionLevel)compressionLevel; + return new DeflateStream(destination, CompressionMode.Compress, level); + } + + public override Stream CreateDecompressStream(Stream source) + { + return new DeflateStream(source, CompressionMode.Decompress); + } +} diff --git a/src/SharpCompress/Providers/Default/ExplodeCompressionProvider.cs b/src/SharpCompress/Providers/Default/ExplodeCompressionProvider.cs new file mode 100644 index 00000000..fb00ba89 --- /dev/null +++ b/src/SharpCompress/Providers/Default/ExplodeCompressionProvider.cs @@ -0,0 +1,49 @@ +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Zip.Headers; +using SharpCompress.Compressors.Explode; + +namespace SharpCompress.Providers.Default; + +/// +/// Provides Explode decompression using SharpCompress's internal implementation. +/// Note: Explode compression is not supported; this provider is decompression-only. +/// +/// +/// Explode requires compressed size, uncompressed size, and flags which must be provided via CompressionContext. +/// +public sealed class ExplodeCompressionProvider : ContextRequiredDecompressionProviderBase +{ + public override CompressionType CompressionType => CompressionType.Explode; + protected override string CompressionNotSupportedMessage => + "Explode compression is not supported by SharpCompress's internal implementation."; + + protected override string DecompressionContextRequirementDescription => + "Explode decompression requires compressed size, uncompressed size, and flags"; + + protected override string DecompressionContextRequirementSuffix => " with FormatOptions"; + + public override Stream CreateDecompressStream(Stream source, CompressionContext context) + { + ValidateRequiredSizes(context, "Explode"); + var flags = RequireFormatOption(context, "Explode", "HeaderFlags"); + + return ExplodeStream.Create(source, context.InputSize, context.OutputSize, flags); + } + + public override async ValueTask CreateDecompressStreamAsync( + Stream source, + CompressionContext context, + CancellationToken cancellationToken = default + ) + { + ValidateRequiredSizes(context, "Explode"); + var flags = RequireFormatOption(context, "Explode", "HeaderFlags"); + + return await ExplodeStream + .CreateAsync(source, context.InputSize, context.OutputSize, flags, cancellationToken) + .ConfigureAwait(false); + } +} diff --git a/src/SharpCompress/Providers/Default/GZipCompressionProvider.cs b/src/SharpCompress/Providers/Default/GZipCompressionProvider.cs new file mode 100644 index 00000000..1365e0a7 --- /dev/null +++ b/src/SharpCompress/Providers/Default/GZipCompressionProvider.cs @@ -0,0 +1,50 @@ +using System.IO; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Compressors; +using SharpCompress.Compressors.Deflate; + +namespace SharpCompress.Providers.Default; + +/// +/// Provides GZip compression using SharpCompress's internal implementation. +/// +public sealed class GZipCompressionProvider : CompressionProviderBase +{ + public override CompressionType CompressionType => CompressionType.GZip; + public override bool SupportsCompression => true; + public override bool SupportsDecompression => true; + + public override Stream CreateCompressStream(Stream destination, int compressionLevel) + { + var level = (CompressionLevel)compressionLevel; + return new GZipStream(destination, CompressionMode.Compress, level, Encoding.UTF8); + } + + public override Stream CreateDecompressStream(Stream source) + { + return new GZipStream(source, CompressionMode.Decompress); + } + + public override Stream CreateDecompressStream(Stream source, CompressionContext context) + { + return new GZipStream( + source, + CompressionMode.Decompress, + CompressionLevel.Default, + context.ResolveArchiveEncoding() + ); + } + + public override ValueTask CreateDecompressStreamAsync( + Stream source, + CompressionContext context, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new ValueTask(CreateDecompressStream(source, context)); + } +} diff --git a/src/SharpCompress/Providers/Default/LZipCompressionProvider.cs b/src/SharpCompress/Providers/Default/LZipCompressionProvider.cs new file mode 100644 index 00000000..74022dd5 --- /dev/null +++ b/src/SharpCompress/Providers/Default/LZipCompressionProvider.cs @@ -0,0 +1,49 @@ +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Compressors; +using SharpCompress.Compressors.LZMA; + +namespace SharpCompress.Providers.Default; + +/// +/// Provides LZip compression using SharpCompress's internal implementation. +/// +public sealed class LZipCompressionProvider : CompressionProviderBase +{ + public override CompressionType CompressionType => CompressionType.LZip; + public override bool SupportsCompression => true; + public override bool SupportsDecompression => true; + + public override Stream CreateCompressStream(Stream destination, int compressionLevel) + { + return LZipStream.Create(destination, CompressionMode.Compress); + } + + public override async ValueTask CreateCompressStreamAsync( + Stream destination, + int compressionLevel, + CancellationToken cancellationToken = default + ) => + await LZipStream + .CreateAsync( + destination, + CompressionMode.Compress, + cancellationToken: cancellationToken + ) + .ConfigureAwait(false); + + public override Stream CreateDecompressStream(Stream source) + { + return LZipStream.Create(source, CompressionMode.Decompress); + } + + public override async ValueTask CreateDecompressStreamAsync( + Stream source, + CancellationToken cancellationToken = default + ) => + await LZipStream + .CreateAsync(source, CompressionMode.Decompress, cancellationToken: cancellationToken) + .ConfigureAwait(false); +} diff --git a/src/SharpCompress/Providers/Default/LzmaCompressingProvider.cs b/src/SharpCompress/Providers/Default/LzmaCompressingProvider.cs new file mode 100644 index 00000000..ce2e6c53 --- /dev/null +++ b/src/SharpCompress/Providers/Default/LzmaCompressingProvider.cs @@ -0,0 +1,116 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Compressors.LZMA; + +namespace SharpCompress.Providers.Default; + +/// +/// Provides LZMA compression and decompression using SharpCompress's internal implementation. +/// This is a complex provider that requires initialization data for compression. +/// +public sealed class LzmaCompressingProvider : CompressionProviderBase, ICompressionProviderHooks +{ + public override CompressionType CompressionType => CompressionType.LZMA; + public override bool SupportsCompression => true; + public override bool SupportsDecompression => true; + + public override Stream CreateCompressStream(Stream destination, int compressionLevel) + { + throw new ArchiveOperationException( + "LZMA compression requires context with CanSeek information. " + + "Use CreateCompressStream(Stream, int, CompressionContext) overload." + ); + } + + public override Stream CreateCompressStream( + Stream destination, + int compressionLevel, + CompressionContext context + ) + { + // LZMA stream creation returns the encoder stream + // Note: Pre-compression data and properties are handled via ICompressionProviderHooks methods + var props = new LzmaEncoderProperties(!context.CanSeek); + return LzmaStream.Create(props, false, destination); + } + + public override Stream CreateDecompressStream(Stream source) + { + throw new ArchiveOperationException( + "LZMA decompression requires properties. " + + "Use CreateDecompressStream(Stream, CompressionContext) overload with Properties." + ); + } + + public override Stream CreateDecompressStream(Stream source, CompressionContext context) + { + if (context.Properties is null || context.Properties.Length < 5) + { + throw new ArgumentException( + "LZMA decompression requires Properties (at least 5 bytes) in CompressionContext.", + nameof(context) + ); + } + + return LzmaStream.Create(context.Properties, source, context.InputSize, context.OutputSize); + } + + public override ValueTask CreateDecompressStreamAsync( + Stream source, + CancellationToken cancellationToken = default + ) => + throw new ArchiveOperationException( + "LZMA decompression requires properties. " + + "Use CreateDecompressStreamAsync(Stream, CompressionContext, CancellationToken) overload with Properties." + ); + + public override async ValueTask CreateDecompressStreamAsync( + Stream source, + CompressionContext context, + CancellationToken cancellationToken = default + ) + { + if (context.Properties is null || context.Properties.Length < 5) + { + throw new ArgumentException( + "LZMA decompression requires Properties (at least 5 bytes) in CompressionContext.", + nameof(context) + ); + } + + return await LzmaStream + .CreateAsync( + context.Properties, + source, + context.InputSize, + context.OutputSize, + leaveOpen: false + ) + .ConfigureAwait(false); + } + + public byte[]? GetPreCompressionData(CompressionContext context) + { + // Zip format writes these magic bytes before the LZMA stream + return new byte[] { 9, 20, 5, 0 }; + } + + public byte[]? GetCompressionProperties(Stream stream, CompressionContext context) + { + // The LZMA stream exposes its properties after creation + if (stream is LzmaStream lzmaStream) + { + return lzmaStream.Properties; + } + return null; + } + + public byte[]? GetPostCompressionData(Stream stream, CompressionContext context) + { + // No post-compression data needed for LZMA in Zip + return null; + } +} diff --git a/src/SharpCompress/Providers/Default/LzwCompressionProvider.cs b/src/SharpCompress/Providers/Default/LzwCompressionProvider.cs new file mode 100644 index 00000000..73821cbe --- /dev/null +++ b/src/SharpCompress/Providers/Default/LzwCompressionProvider.cs @@ -0,0 +1,21 @@ +using System.IO; +using SharpCompress.Common; +using SharpCompress.Compressors.Lzw; + +namespace SharpCompress.Providers.Default; + +/// +/// Provides LZW compression decompression using SharpCompress's internal implementation. +/// Note: Compression is not supported by this provider. +/// +public sealed class LzwCompressionProvider : DecompressionOnlyProviderBase +{ + public override CompressionType CompressionType => CompressionType.Lzw; + protected override string CompressionNotSupportedMessage => + "LZW compression is not supported by SharpCompress's internal implementation."; + + public override Stream CreateDecompressStream(Stream source) + { + return new LzwStream(source); + } +} diff --git a/src/SharpCompress/Providers/Default/PpmdCompressingProvider.cs b/src/SharpCompress/Providers/Default/PpmdCompressingProvider.cs new file mode 100644 index 00000000..818976bd --- /dev/null +++ b/src/SharpCompress/Providers/Default/PpmdCompressingProvider.cs @@ -0,0 +1,116 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Compressors.PPMd; + +namespace SharpCompress.Providers.Default; + +/// +/// Provides PPMd compression and decompression using SharpCompress's internal implementation. +/// This is a complex provider that requires initialization data for compression. +/// +public sealed class PpmdCompressingProvider : CompressionProviderBase, ICompressionProviderHooks +{ + public override CompressionType CompressionType => CompressionType.PPMd; + public override bool SupportsCompression => true; + public override bool SupportsDecompression => true; + + public override Stream CreateCompressStream(Stream destination, int compressionLevel) + { + // Ppmd doesn't use compressionLevel, uses PpmdProperties instead + var props = new PpmdProperties(); + return PpmdStream.Create(props, destination, true); + } + + public override Stream CreateCompressStream( + Stream destination, + int compressionLevel, + CompressionContext context + ) + { + // Context not used for Ppmd compression, but we could use FormatOptions for custom properties + if (context.FormatOptions is PpmdProperties customProps) + { + return PpmdStream.Create(customProps, destination, true); + } + + return CreateCompressStream(destination, compressionLevel); + } + + public override Stream CreateDecompressStream(Stream source) + { + throw new ArchiveOperationException( + "PPMd decompression requires properties. " + + "Use CreateDecompressStream(Stream, CompressionContext) overload with Properties." + ); + } + + public override Stream CreateDecompressStream(Stream source, CompressionContext context) + { + if (context.Properties is null || context.Properties.Length < 2) + { + throw new ArgumentException( + "PPMd decompression requires Properties (at least 2 bytes) in CompressionContext.", + nameof(context) + ); + } + + var props = new PpmdProperties(context.Properties); + return PpmdStream.Create(props, source, false); + } + + public override ValueTask CreateDecompressStreamAsync( + Stream source, + CancellationToken cancellationToken = default + ) => + throw new ArchiveOperationException( + "PPMd decompression requires properties. " + + "Use CreateDecompressStreamAsync(Stream, CompressionContext, CancellationToken) overload with Properties." + ); + + public override async ValueTask CreateDecompressStreamAsync( + Stream source, + CompressionContext context, + CancellationToken cancellationToken = default + ) + { + if (context.Properties is null || context.Properties.Length < 2) + { + throw new ArgumentException( + "PPMd decompression requires Properties (at least 2 bytes) in CompressionContext.", + nameof(context) + ); + } + + var props = new PpmdProperties(context.Properties); + return await PpmdStream + .CreateAsync(props, source, false, cancellationToken) + .ConfigureAwait(false); + } + + public byte[]? GetPreCompressionData(CompressionContext context) + { + // Ppmd writes its properties before the compressed data + if (context.FormatOptions is PpmdProperties customProps) + { + return customProps.Properties; + } + + var defaultProps = new PpmdProperties(); + return defaultProps.Properties; + } + + public byte[]? GetCompressionProperties(Stream stream, CompressionContext context) + { + // Properties are already written in GetPreCompressionData + return null; + } + + public byte[]? GetPostCompressionData(Stream stream, CompressionContext context) + { + // No post-compression data needed for Ppmd + return null; + } +} diff --git a/src/SharpCompress/Providers/Default/Reduce1CompressionProvider.cs b/src/SharpCompress/Providers/Default/Reduce1CompressionProvider.cs new file mode 100644 index 00000000..c984b4ad --- /dev/null +++ b/src/SharpCompress/Providers/Default/Reduce1CompressionProvider.cs @@ -0,0 +1,13 @@ +using SharpCompress.Common; + +namespace SharpCompress.Providers.Default; + +/// +/// Provides Reduce1 decompression using SharpCompress's internal implementation. +/// Note: Reduce compression is not supported; this provider is decompression-only. +/// +public sealed class Reduce1CompressionProvider : ReduceCompressionProviderBase +{ + public override CompressionType CompressionType => CompressionType.Reduce1; + protected override int Factor => 1; +} diff --git a/src/SharpCompress/Providers/Default/Reduce2CompressionProvider.cs b/src/SharpCompress/Providers/Default/Reduce2CompressionProvider.cs new file mode 100644 index 00000000..9ca67bf2 --- /dev/null +++ b/src/SharpCompress/Providers/Default/Reduce2CompressionProvider.cs @@ -0,0 +1,13 @@ +using SharpCompress.Common; + +namespace SharpCompress.Providers.Default; + +/// +/// Provides Reduce2 decompression using SharpCompress's internal implementation. +/// Note: Reduce compression is not supported; this provider is decompression-only. +/// +public sealed class Reduce2CompressionProvider : ReduceCompressionProviderBase +{ + public override CompressionType CompressionType => CompressionType.Reduce2; + protected override int Factor => 2; +} diff --git a/src/SharpCompress/Providers/Default/Reduce3CompressionProvider.cs b/src/SharpCompress/Providers/Default/Reduce3CompressionProvider.cs new file mode 100644 index 00000000..01fa708c --- /dev/null +++ b/src/SharpCompress/Providers/Default/Reduce3CompressionProvider.cs @@ -0,0 +1,13 @@ +using SharpCompress.Common; + +namespace SharpCompress.Providers.Default; + +/// +/// Provides Reduce3 decompression using SharpCompress's internal implementation. +/// Note: Reduce compression is not supported; this provider is decompression-only. +/// +public sealed class Reduce3CompressionProvider : ReduceCompressionProviderBase +{ + public override CompressionType CompressionType => CompressionType.Reduce3; + protected override int Factor => 3; +} diff --git a/src/SharpCompress/Providers/Default/Reduce4CompressionProvider.cs b/src/SharpCompress/Providers/Default/Reduce4CompressionProvider.cs new file mode 100644 index 00000000..14b8e543 --- /dev/null +++ b/src/SharpCompress/Providers/Default/Reduce4CompressionProvider.cs @@ -0,0 +1,13 @@ +using SharpCompress.Common; + +namespace SharpCompress.Providers.Default; + +/// +/// Provides Reduce4 decompression using SharpCompress's internal implementation. +/// Note: Reduce compression is not supported; this provider is decompression-only. +/// +public sealed class Reduce4CompressionProvider : ReduceCompressionProviderBase +{ + public override CompressionType CompressionType => CompressionType.Reduce4; + protected override int Factor => 4; +} diff --git a/src/SharpCompress/Providers/Default/ReduceCompressionProviderBase.cs b/src/SharpCompress/Providers/Default/ReduceCompressionProviderBase.cs new file mode 100644 index 00000000..17b01a1a --- /dev/null +++ b/src/SharpCompress/Providers/Default/ReduceCompressionProviderBase.cs @@ -0,0 +1,36 @@ +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Compressors.Reduce; + +namespace SharpCompress.Providers.Default; + +public abstract class ReduceCompressionProviderBase : ContextRequiredDecompressionProviderBase +{ + protected abstract int Factor { get; } + + protected override string DecompressionContextRequirementDescription => + "Reduce decompression requires compressed and uncompressed sizes"; + + protected override string CompressionNotSupportedMessage => + "Reduce compression is not supported by SharpCompress's internal implementation."; + + public sealed override Stream CreateDecompressStream(Stream source, CompressionContext context) + { + ValidateRequiredSizes(context, "Reduce"); + return ReduceStream.Create(source, context.InputSize, context.OutputSize, Factor); + } + + public sealed override async ValueTask CreateDecompressStreamAsync( + Stream source, + CompressionContext context, + CancellationToken cancellationToken = default + ) + { + ValidateRequiredSizes(context, "Reduce"); + return await ReduceStream + .CreateAsync(source, context.InputSize, context.OutputSize, Factor, cancellationToken) + .ConfigureAwait(false); + } +} diff --git a/src/SharpCompress/Providers/Default/ShrinkCompressionProvider.cs b/src/SharpCompress/Providers/Default/ShrinkCompressionProvider.cs new file mode 100644 index 00000000..72c374ea --- /dev/null +++ b/src/SharpCompress/Providers/Default/ShrinkCompressionProvider.cs @@ -0,0 +1,44 @@ +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Compressors.Shrink; + +namespace SharpCompress.Providers.Default; + +/// +/// Provides Shrink decompression using SharpCompress's internal implementation. +/// Note: Shrink compression is not supported; this provider is decompression-only. +/// +/// +/// Shrink requires compressed and uncompressed sizes which must be provided via CompressionContext. +/// +public sealed class ShrinkCompressionProvider : ContextRequiredDecompressionProviderBase +{ + public override CompressionType CompressionType => CompressionType.Shrink; + protected override string CompressionNotSupportedMessage => + "Shrink compression is not supported by SharpCompress's internal implementation."; + + protected override string DecompressionContextRequirementDescription => + "Shrink decompression requires compressed and uncompressed sizes"; + + public override Stream CreateDecompressStream(Stream source, CompressionContext context) + { + ValidateRequiredSizes(context, "Shrink"); + + return new ShrinkStream(source, context.OutputSize); + } + + public override async ValueTask CreateDecompressStreamAsync( + Stream source, + CompressionContext context, + CancellationToken cancellationToken = default + ) + { + ValidateRequiredSizes(context, "Shrink"); + + return await ShrinkStream + .CreateAsync(source, context.OutputSize, cancellationToken) + .ConfigureAwait(false); + } +} diff --git a/src/SharpCompress/Providers/Default/XzCompressionProvider.cs b/src/SharpCompress/Providers/Default/XzCompressionProvider.cs new file mode 100644 index 00000000..db005ab0 --- /dev/null +++ b/src/SharpCompress/Providers/Default/XzCompressionProvider.cs @@ -0,0 +1,21 @@ +using System.IO; +using SharpCompress.Common; +using SharpCompress.Compressors.Xz; + +namespace SharpCompress.Providers.Default; + +/// +/// Provides XZ compression decompression using SharpCompress's internal implementation. +/// Note: Compression is not supported by this provider. +/// +public sealed class XzCompressionProvider : DecompressionOnlyProviderBase +{ + public override CompressionType CompressionType => CompressionType.Xz; + protected override string CompressionNotSupportedMessage => + "XZ compression is not supported by SharpCompress's internal implementation."; + + public override Stream CreateDecompressStream(Stream source) + { + return new XZStream(source); + } +} diff --git a/src/SharpCompress/Providers/Default/ZStandardCompressionProvider.cs b/src/SharpCompress/Providers/Default/ZStandardCompressionProvider.cs new file mode 100644 index 00000000..973e0be8 --- /dev/null +++ b/src/SharpCompress/Providers/Default/ZStandardCompressionProvider.cs @@ -0,0 +1,25 @@ +using System.IO; +using SharpCompress.Common; +using ZStd = SharpCompress.Compressors.ZStandard; + +namespace SharpCompress.Providers.Default; + +/// +/// Provides ZStandard compression using SharpCompress's internal implementation. +/// +public sealed class ZStandardCompressionProvider : CompressionProviderBase +{ + public override CompressionType CompressionType => CompressionType.ZStandard; + public override bool SupportsCompression => true; + public override bool SupportsDecompression => true; + + public override Stream CreateCompressStream(Stream destination, int compressionLevel) + { + return new ZStd.CompressionStream(destination, compressionLevel); + } + + public override Stream CreateDecompressStream(Stream source) + { + return new ZStd.DecompressionStream(source); + } +} diff --git a/src/SharpCompress/Providers/ICompressionProvider.cs b/src/SharpCompress/Providers/ICompressionProvider.cs new file mode 100644 index 00000000..57d4ba3a --- /dev/null +++ b/src/SharpCompress/Providers/ICompressionProvider.cs @@ -0,0 +1,151 @@ +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Providers; + +/// +/// Provides compression and decompression stream creation for a specific compression type. +/// Implement this interface to supply alternative compression implementations. +/// +/// +/// +/// This interface abstracts the creation of compression and decompression streams, +/// allowing SharpCompress to use different implementations of the same compression type. +/// For example, you can provide an implementation that uses System.IO.Compression +/// for Deflate/GZip instead of the internal DotNetZip-derived implementation. +/// +/// +/// Implementations should be thread-safe for concurrent decompression operations, +/// but CreateCompressStream/CreateDecompressStream themselves return new stream instances +/// that are not shared. +/// +/// +/// For simpler implementations, derive from which provides +/// default async implementations that delegate to the synchronous methods. +/// +/// +public interface ICompressionProvider +{ + /// + /// The compression type this provider handles. + /// + CompressionType CompressionType { get; } + + /// + /// Whether this provider supports compression (writing). + /// + bool SupportsCompression { get; } + + /// + /// Whether this provider supports decompression (reading). + /// + bool SupportsDecompression { get; } + + /// + /// Creates a compression stream that compresses data written to it. + /// + /// The destination stream to write compressed data to. + /// The compression level (0-9, algorithm-specific). + /// A stream that compresses data written to it. + /// Thrown if SupportsCompression is false. + Stream CreateCompressStream(Stream destination, int compressionLevel); + + /// + /// Creates a compression stream with context information. + /// + /// The destination stream. + /// The compression level. + /// Context information about the compression. + /// A compression stream. + /// Thrown if SupportsCompression is false. + Stream CreateCompressStream( + Stream destination, + int compressionLevel, + CompressionContext context + ); + + /// + /// Creates a decompression stream that decompresses data read from it. + /// + /// The source stream to read compressed data from. + /// A stream that decompresses data read from it. + /// Thrown if SupportsDecompression is false. + Stream CreateDecompressStream(Stream source); + + /// + /// Creates a decompression stream with context information. + /// + /// The source stream. + /// + /// Context information about the decompression. Providers may use + /// for archive header encoding + /// (via ) and + /// for format-specific metadata + /// such as compression properties or algorithm-specific configuration. + /// + /// A decompression stream. + /// Thrown if SupportsDecompression is false. + Stream CreateDecompressStream(Stream source, CompressionContext context); + + /// + /// Asynchronously creates a compression stream that compresses data written to it. + /// + /// The destination stream to write compressed data to. + /// The compression level (0-9, algorithm-specific). + /// Cancellation token. + /// A task containing the compression stream. + /// Thrown if SupportsCompression is false. + ValueTask CreateCompressStreamAsync( + Stream destination, + int compressionLevel, + CancellationToken cancellationToken = default + ); + + /// + /// Asynchronously creates a compression stream with context information. + /// + /// The destination stream. + /// The compression level. + /// Context information about the compression. + /// Cancellation token. + /// A task containing the compression stream. + /// Thrown if SupportsCompression is false. + ValueTask CreateCompressStreamAsync( + Stream destination, + int compressionLevel, + CompressionContext context, + CancellationToken cancellationToken = default + ); + + /// + /// Asynchronously creates a decompression stream that decompresses data read from it. + /// + /// The source stream to read compressed data from. + /// Cancellation token. + /// A task containing the decompression stream. + /// Thrown if SupportsDecompression is false. + ValueTask CreateDecompressStreamAsync( + Stream source, + CancellationToken cancellationToken = default + ); + + /// + /// Asynchronously creates a decompression stream with context information. + /// + /// The source stream. + /// + /// Context information about the decompression. Providers may use + /// for format-specific metadata + /// (for example, archive header encoding). + /// + /// Cancellation token. + /// A task containing the decompression stream. + /// Thrown if SupportsDecompression is false. + ValueTask CreateDecompressStreamAsync( + Stream source, + CompressionContext context, + CancellationToken cancellationToken = default + ); +} diff --git a/src/SharpCompress/Providers/ICompressionProviderHooks.cs b/src/SharpCompress/Providers/ICompressionProviderHooks.cs new file mode 100644 index 00000000..796f013d --- /dev/null +++ b/src/SharpCompress/Providers/ICompressionProviderHooks.cs @@ -0,0 +1,42 @@ +using System.IO; + +namespace SharpCompress.Providers; + +/// +/// Extended compression provider interface for formats that require initialization/finalization data. +/// +/// +/// Some compression formats (like LZMA and PPMd in Zip) require special handling: +/// - Data written before compression starts (magic bytes, properties headers) +/// - Data written after compression completes (properties, footers) +/// This interface extends ICompressionProvider to support these complex initialization patterns +/// while keeping the simple ICompressionProvider interface for formats that don't need it. +/// +public interface ICompressionProviderHooks : ICompressionProvider +{ + /// + /// Gets initialization data to write before compression starts. + /// Returns null if no pre-compression data is needed. + /// + /// Context information. + /// Bytes to write before compression, or null. + byte[]? GetPreCompressionData(CompressionContext context); + + /// + /// Gets properties/data to write after creating the compression stream but before writing data. + /// Returns null if no properties are needed. + /// + /// The compression stream that was created. + /// Context information. + /// Bytes to write after stream creation, or null. + byte[]? GetCompressionProperties(Stream stream, CompressionContext context); + + /// + /// Gets data to write after compression is complete. + /// Returns null if no post-compression data is needed. + /// + /// The compression stream. + /// Context information. + /// Bytes to write after compression, or null. + byte[]? GetPostCompressionData(Stream stream, CompressionContext context); +} diff --git a/src/SharpCompress/Providers/IFinishable.cs b/src/SharpCompress/Providers/IFinishable.cs new file mode 100644 index 00000000..da532674 --- /dev/null +++ b/src/SharpCompress/Providers/IFinishable.cs @@ -0,0 +1,24 @@ +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Providers; + +/// +/// Interface for compression streams that require explicit finalization +/// before disposal to ensure all compressed data is flushed properly. +/// +/// +/// Some compression formats (notably BZip2 and LZip) require explicit +/// finalization to write trailer/footer data. Implementing this interface +/// allows generic code to handle finalization without knowing the specific stream type. +/// +public interface IFinishable +{ + /// + /// Finalizes the compression, flushing any remaining buffered data + /// and writing format-specific trailer/footer bytes. + /// + void Finish(); + + ValueTask FinishAsync(CancellationToken cancellationToken); +} diff --git a/src/SharpCompress/Providers/System/SystemDeflateCompressionProvider.cs b/src/SharpCompress/Providers/System/SystemDeflateCompressionProvider.cs new file mode 100644 index 00000000..8e5faa01 --- /dev/null +++ b/src/SharpCompress/Providers/System/SystemDeflateCompressionProvider.cs @@ -0,0 +1,51 @@ +using System.IO; +using System.IO.Compression; +using SharpCompress.Common; + +namespace SharpCompress.Providers.System; + +/// +/// Provides Deflate compression using System.IO.Compression.DeflateStream. +/// +/// +/// On modern .NET (5+), System.IO.Compression uses hardware-accelerated zlib +/// and is significantly faster than SharpCompress's pure C# implementation. +/// +public sealed class SystemDeflateCompressionProvider : CompressionProviderBase +{ + public override CompressionType CompressionType => CompressionType.Deflate; + public override bool SupportsCompression => true; + public override bool SupportsDecompression => true; + + public override Stream CreateCompressStream(Stream destination, int compressionLevel) + { + var bclLevel = MapCompressionLevel(compressionLevel); + return new DeflateStream(destination, bclLevel, leaveOpen: false); + } + + public override Stream CreateDecompressStream(Stream source) + { + return new DeflateStream( + source, + global::System.IO.Compression.CompressionMode.Decompress, + leaveOpen: false + ); + } + + /// + /// Maps SharpCompress compression level (0-9) to BCL CompressionLevel. + /// + private static global::System.IO.Compression.CompressionLevel MapCompressionLevel(int level) + { + // Map 0-9 to appropriate BCL levels + return level switch + { + 0 => global::System.IO.Compression.CompressionLevel.NoCompression, + <= 2 => global::System.IO.Compression.CompressionLevel.Fastest, +#if NET7_0_OR_GREATER + >= 8 => global::System.IO.Compression.CompressionLevel.SmallestSize, +#endif + _ => global::System.IO.Compression.CompressionLevel.Optimal, + }; + } +} diff --git a/src/SharpCompress/Providers/System/SystemGZipCompressionProvider.cs b/src/SharpCompress/Providers/System/SystemGZipCompressionProvider.cs new file mode 100644 index 00000000..d0335ef6 --- /dev/null +++ b/src/SharpCompress/Providers/System/SystemGZipCompressionProvider.cs @@ -0,0 +1,51 @@ +using System.IO; +using System.IO.Compression; +using SharpCompress.Common; + +namespace SharpCompress.Providers.System; + +/// +/// Provides GZip compression using System.IO.Compression.GZipStream. +/// +/// +/// On modern .NET (5+), System.IO.Compression uses hardware-accelerated zlib +/// and is significantly faster than SharpCompress's pure C# implementation. +/// +public sealed class SystemGZipCompressionProvider : CompressionProviderBase +{ + public override CompressionType CompressionType => CompressionType.GZip; + public override bool SupportsCompression => true; + public override bool SupportsDecompression => true; + + public override Stream CreateCompressStream(Stream destination, int compressionLevel) + { + var bclLevel = MapCompressionLevel(compressionLevel); + return new GZipStream(destination, bclLevel, leaveOpen: false); + } + + public override Stream CreateDecompressStream(Stream source) + { + return new GZipStream( + source, + global::System.IO.Compression.CompressionMode.Decompress, + leaveOpen: false + ); + } + + /// + /// Maps SharpCompress compression level (0-9) to BCL CompressionLevel. + /// + private static global::System.IO.Compression.CompressionLevel MapCompressionLevel(int level) + { + // Map 0-9 to appropriate BCL levels + return level switch + { + 0 => global::System.IO.Compression.CompressionLevel.NoCompression, + <= 2 => global::System.IO.Compression.CompressionLevel.Fastest, +#if NET7_0_OR_GREATER + >= 8 => global::System.IO.Compression.CompressionLevel.SmallestSize, +#endif + _ => global::System.IO.Compression.CompressionLevel.Optimal, + }; + } +} diff --git a/src/SharpCompress/ReadOnlyCollection.cs b/src/SharpCompress/ReadOnlyCollection.cs deleted file mode 100644 index 745a59aa..00000000 --- a/src/SharpCompress/ReadOnlyCollection.cs +++ /dev/null @@ -1,30 +0,0 @@ -using System; -using System.Collections; -using System.Collections.Generic; - -namespace SharpCompress; - -public class ReadOnlyCollection : ICollection -{ - private readonly ICollection collection; - - public ReadOnlyCollection(ICollection collection) => this.collection = collection; - - public void Add(T item) => throw new NotSupportedException(); - - public void Clear() => throw new NotSupportedException(); - - public bool Contains(T item) => collection.Contains(item); - - public void CopyTo(T[] array, int arrayIndex) => collection.CopyTo(array, arrayIndex); - - public int Count => collection.Count; - - public bool IsReadOnly => true; - - public bool Remove(T item) => throw new NotSupportedException(); - - public IEnumerator GetEnumerator() => collection.GetEnumerator(); - - IEnumerator IEnumerable.GetEnumerator() => throw new NotSupportedException(); -} diff --git a/src/SharpCompress/Readers/AbstractReader.Async.cs b/src/SharpCompress/Readers/AbstractReader.Async.cs new file mode 100644 index 00000000..8628c0c2 --- /dev/null +++ b/src/SharpCompress/Readers/AbstractReader.Async.cs @@ -0,0 +1,210 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Readers; + +public abstract partial class AbstractReader + where TEntry : Entry + where TVolume : Volume +{ + public virtual async ValueTask DisposeAsync() + { + if (_entriesForCurrentReadStreamAsync is not null) + { + await _entriesForCurrentReadStreamAsync.DisposeAsync().ConfigureAwait(false); + } + + // If Volume implements IAsyncDisposable, use async disposal + if (Volume is IAsyncDisposable asyncDisposable) + { + await asyncDisposable.DisposeAsync().ConfigureAwait(false); + } + else + { + Volume?.Dispose(); + } + } + + public async ValueTask MoveToNextEntryAsync(CancellationToken cancellationToken = default) + { + if (_completed) + { + return false; + } + if (Cancelled) + { + throw new ReaderCancelledException("Reader has been cancelled."); + } + if (_entriesForCurrentReadStreamAsync is null) + { + return await LoadStreamForReadingAsync( + await RequestInitialStreamAsync(cancellationToken).ConfigureAwait(false) + ) + .ConfigureAwait(false); + } + if (!_wroteCurrentEntry) + { + await SkipEntryAsync(cancellationToken).ConfigureAwait(false); + } + _wroteCurrentEntry = false; + if (await NextEntryForCurrentStreamAsync(cancellationToken).ConfigureAwait(false)) + { + return true; + } + _completed = true; + return false; + } + + protected async ValueTask LoadStreamForReadingAsync(Stream stream) + { + if (_entriesForCurrentReadStreamAsync is not null) + { + await _entriesForCurrentReadStreamAsync.DisposeAsync().ConfigureAwait(false); + } + if (stream is null || !stream.CanRead) + { + throw new MultipartStreamRequiredException( + "File is split into multiple archives: '" + + Entry.Key + + "'. A new readable stream is required. Use Cancel if it was intended." + ); + } + _entriesForCurrentReadStreamAsync = GetEntriesAsync(stream).GetAsyncEnumerator(); + return await _entriesForCurrentReadStreamAsync.MoveNextAsync().ConfigureAwait(false); + } + + private async ValueTask SkipEntryAsync(CancellationToken cancellationToken) + { + if (!Entry.IsDirectory) + { + await SkipAsync(cancellationToken).ConfigureAwait(false); + } + } + + private async ValueTask SkipAsync(CancellationToken cancellationToken) + { + var part = Entry.Parts.First(); + + if (!Entry.IsSplitAfter && !Entry.IsSolid && Entry.CompressedSize > 0) + { + //not solid and has a known compressed size then we can skip raw bytes. + var rawStream = part.GetRawStream(); + + if (rawStream != null) + { + var bytesToAdvance = Entry.CompressedSize; + await rawStream.SkipAsync(bytesToAdvance, cancellationToken).ConfigureAwait(false); + part.Skipped = true; + return; + } + } + //don't know the size so we have to try to decompress to skip +#if LEGACY_DOTNET + using var s = await OpenEntryStreamAsync(cancellationToken).ConfigureAwait(false); + await s.SkipEntryAsync(cancellationToken).ConfigureAwait(false); +#else + await using var s = await OpenEntryStreamAsync(cancellationToken).ConfigureAwait(false); + await s.SkipEntryAsync(cancellationToken).ConfigureAwait(false); +#endif + } + + public async ValueTask WriteEntryToAsync( + Stream writableStream, + CancellationToken cancellationToken = default + ) + { + if (_wroteCurrentEntry) + { + throw new ArgumentException( + "WriteEntryToAsync or OpenEntryStreamAsync can only be called once." + ); + } + + ThrowHelper.ThrowIfNull(writableStream); + if (!writableStream.CanWrite) + { + throw new ArgumentException( + "A writable Stream was required. Use Cancel if that was intended." + ); + } + + await WriteAsync(writableStream, cancellationToken).ConfigureAwait(false); + _wroteCurrentEntry = true; + } + + private async ValueTask WriteAsync(Stream writeStream, CancellationToken cancellationToken) + { +#if LEGACY_DOTNET + using Stream s = await OpenEntryStreamAsync(cancellationToken).ConfigureAwait(false); + var sourceStream = WrapWithProgress(s, Entry); + await sourceStream + .CopyToAsync(writeStream, Options.BufferSize, cancellationToken) + .ConfigureAwait(false); +#else + await using Stream s = await OpenEntryStreamAsync(cancellationToken).ConfigureAwait(false); + var sourceStream = WrapWithProgress(s, Entry); + await sourceStream + .CopyToAsync(writeStream, Options.BufferSize, cancellationToken) + .ConfigureAwait(false); +#endif + } + + public async ValueTask OpenEntryStreamAsync( + CancellationToken cancellationToken = default + ) + { + if (_wroteCurrentEntry) + { + throw new ArgumentException( + "WriteEntryToAsync or OpenEntryStreamAsync can only be called once." + ); + } + var stream = await GetEntryStreamAsync(cancellationToken).ConfigureAwait(false); + _wroteCurrentEntry = true; + return stream; + } + + protected virtual async ValueTask GetEntryStreamAsync( + CancellationToken cancellationToken = default + ) + { + var stream = await Entry + .Parts.First() + .GetCompressedStreamAsync(cancellationToken) + .ConfigureAwait(false); + return CreateEntryStream(stream); + } + + internal virtual ValueTask NextEntryForCurrentStreamAsync() => + _entriesForCurrentReadStreamAsync.NotNull().MoveNextAsync(); + + /// + /// Moves the current async enumerator to the next entry. + /// + private ValueTask NextEntryForCurrentStreamAsync(CancellationToken cancellationToken) + { + if (_entriesForCurrentReadStreamAsync is not null) + { + return _entriesForCurrentReadStreamAsync.MoveNextAsync(); + } + + return new ValueTask(NextEntryForCurrentStream()); + } + + // Async iterator method + protected virtual async IAsyncEnumerable GetEntriesAsync(Stream stream) + { +#pragma warning disable VSTHRD111 + await Task.CompletedTask; +#pragma warning restore VSTHRD111 + foreach (var entry in GetEntries(stream)) + { + yield return entry; + } + } +} diff --git a/src/SharpCompress/Readers/AbstractReader.cs b/src/SharpCompress/Readers/AbstractReader.cs index 7927937f..b6a5d32c 100644 --- a/src/SharpCompress/Readers/AbstractReader.cs +++ b/src/SharpCompress/Readers/AbstractReader.cs @@ -2,52 +2,66 @@ using System; using System.Collections.Generic; using System.IO; using System.Linq; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common; +using SharpCompress.IO; namespace SharpCompress.Readers; /// /// A generic push reader that reads unseekable comrpessed streams. /// -public abstract class AbstractReader : IReader, IReaderExtractionListener +public abstract partial class AbstractReader : IReader, IAsyncReader where TEntry : Entry where TVolume : Volume { - private bool completed; - private IEnumerator? entriesForCurrentReadStream; - private bool wroteCurrentEntry; + private bool _completed; + private IEnumerator? _entriesForCurrentReadStream; + private IAsyncEnumerator? _entriesForCurrentReadStreamAsync; + private bool _wroteCurrentEntry; + private readonly bool _disposeVolume; - public event EventHandler>? EntryExtractionProgress; - - public event EventHandler? CompressedBytesRead; - public event EventHandler? FilePartExtractionBegin; - - internal AbstractReader(ReaderOptions options, ArchiveType archiveType) + internal AbstractReader(ReaderOptions options, ArchiveType type, bool disposeVolume = true) { - ArchiveType = archiveType; + Type = type; + _disposeVolume = disposeVolume; Options = options; } internal ReaderOptions Options { get; } - public ArchiveType ArchiveType { get; } + public ArchiveType Type { get; } /// /// Current volume that the current entry resides in /// - public abstract TVolume Volume { get; } + public abstract TVolume? Volume { get; } /// - /// Current file entry + /// Current file entry (from either sync or async enumeration). /// - public TEntry Entry => entriesForCurrentReadStream!.Current; + public TEntry Entry + { + get + { + if (_entriesForCurrentReadStreamAsync is not null) + { + return _entriesForCurrentReadStreamAsync.Current; + } + return _entriesForCurrentReadStream.NotNull().Current; + } + } #region IDisposable Members - public void Dispose() + public virtual void Dispose() { - entriesForCurrentReadStream?.Dispose(); - Volume?.Dispose(); + _entriesForCurrentReadStream?.Dispose(); + if (_disposeVolume) + { + Volume?.Dispose(); + } } #endregion @@ -61,7 +75,7 @@ public abstract class AbstractReader : IReader, IReaderExtracti /// public void Cancel() { - if (!completed) + if (!_completed) { Cancelled = true; } @@ -69,35 +83,47 @@ public abstract class AbstractReader : IReader, IReaderExtracti public bool MoveToNextEntry() { - if (completed) + if (_entriesForCurrentReadStreamAsync is not null) + { + throw new ArchiveOperationException( + $"{nameof(MoveToNextEntry)} cannot be used after {nameof(MoveToNextEntryAsync)} has been used." + ); + } + if (_completed) { return false; } if (Cancelled) { - throw new InvalidOperationException("Reader has been cancelled."); + throw new ReaderCancelledException("Reader has been cancelled."); } - if (entriesForCurrentReadStream is null) + if (_entriesForCurrentReadStream is null) { return LoadStreamForReading(RequestInitialStream()); } - if (!wroteCurrentEntry) + if (!_wroteCurrentEntry) { SkipEntry(); } - wroteCurrentEntry = false; + _wroteCurrentEntry = false; if (NextEntryForCurrentStream()) { return true; } - completed = true; + _completed = true; return false; } protected bool LoadStreamForReading(Stream stream) { - entriesForCurrentReadStream?.Dispose(); - if ((stream is null) || (!stream.CanRead)) + if (_entriesForCurrentReadStreamAsync is not null) + { + throw new ArchiveOperationException( + $"{nameof(LoadStreamForReading)} cannot be used after {nameof(LoadStreamForReadingAsync)} has been used." + ); + } + _entriesForCurrentReadStream?.Dispose(); + if (stream is null || !stream.CanRead) { throw new MultipartStreamRequiredException( "File is split into multiple archives: '" @@ -105,13 +131,19 @@ public abstract class AbstractReader : IReader, IReaderExtracti + "'. A new readable stream is required. Use Cancel if it was intended." ); } - entriesForCurrentReadStream = GetEntries(stream).GetEnumerator(); - return entriesForCurrentReadStream.MoveNext(); + _entriesForCurrentReadStream = GetEntries(stream).GetEnumerator(); + return _entriesForCurrentReadStream.MoveNext(); } - protected virtual Stream RequestInitialStream() => Volume.Stream; + protected virtual Stream RequestInitialStream() => + Volume.NotNull("Volume isn't loaded.").Stream; - internal virtual bool NextEntryForCurrentStream() => entriesForCurrentReadStream!.MoveNext(); + protected virtual ValueTask RequestInitialStreamAsync( + CancellationToken cancellationToken = default + ) => new(RequestInitialStream()); + + internal virtual bool NextEntryForCurrentStream() => + _entriesForCurrentReadStream.NotNull().MoveNext(); protected abstract IEnumerable GetEntries(Stream stream); @@ -129,7 +161,7 @@ public abstract class AbstractReader : IReader, IReaderExtracti { var part = Entry.Parts.First(); - if (!Entry.IsSolid && Entry.CompressedSize > 0) + if (!Entry.IsSplitAfter && !Entry.IsSolid && Entry.CompressedSize > 0) { //not solid and has a known compressed size then we can skip raw bytes. var rawStream = part.GetRawStream(); @@ -147,17 +179,17 @@ public abstract class AbstractReader : IReader, IReaderExtracti s.SkipEntry(); } - public void WriteEntryTo(Stream writableStream) + public void WriteEntryTo(Stream writableStream) => + WriteEntryTo(writableStream, Options.BufferSize); + + private void WriteEntryTo(Stream writableStream, int bufferSize) { - if (wroteCurrentEntry) + if (_wroteCurrentEntry) { throw new ArgumentException("WriteEntryTo or OpenEntryStream can only be called once."); } - if (writableStream is null) - { - throw new ArgumentNullException(nameof(writableStream)); - } + ThrowHelper.ThrowIfNull(writableStream); if (!writableStream.CanWrite) { throw new ArgumentException( @@ -165,33 +197,69 @@ public abstract class AbstractReader : IReader, IReaderExtracti ); } - Write(writableStream); - wroteCurrentEntry = true; + Write(writableStream, bufferSize); + _wroteCurrentEntry = true; } - internal void Write(Stream writeStream) + internal void Write(Stream writeStream) => Write(writeStream, Options.BufferSize); + + internal void Write(Stream writeStream, int bufferSize) { - var streamListener = this as IReaderExtractionListener; using Stream s = OpenEntryStream(); - s.TransferTo(writeStream, Entry, streamListener); + var sourceStream = WrapWithProgress(s, Entry); + sourceStream.CopyTo(writeStream, bufferSize); + } + + private Stream WrapWithProgress(Stream source, Entry entry) + { + var progress = Options.Progress; + if (progress is null) + { + return source; + } + + var entryPath = entry.Key ?? string.Empty; + long? totalBytes = GetEntrySizeSafe(entry); + return new ProgressReportingStream( + source, + progress, + entryPath, + totalBytes, + leaveOpen: true + ); + } + + private static long? GetEntrySizeSafe(Entry entry) + { + try + { + var size = entry.Size; + // Return the actual size (including 0 for empty entries) + // Negative values indicate unknown size + return size >= 0 ? size : null; + } + catch (NotImplementedException) + { + return null; + } } public EntryStream OpenEntryStream() { - if (wroteCurrentEntry) + if (_wroteCurrentEntry) { throw new ArgumentException("WriteEntryTo or OpenEntryStream can only be called once."); } var stream = GetEntryStream(); - wroteCurrentEntry = true; + _wroteCurrentEntry = true; return stream; } /// /// Retains a reference to the entry stream, so we can check whether it completed later. /// - protected EntryStream CreateEntryStream(Stream decompressed) => - new EntryStream(this, decompressed); + protected EntryStream CreateEntryStream(Stream? decompressed) => + new(this, decompressed.NotNull()); protected virtual EntryStream GetEntryStream() => CreateEntryStream(Entry.Parts.First().GetCompressedStream()); @@ -199,43 +267,5 @@ public abstract class AbstractReader : IReader, IReaderExtracti #endregion IEntry IReader.Entry => Entry; - - void IExtractionListener.FireCompressedBytesRead( - long currentPartCompressedBytes, - long compressedReadBytes - ) => - CompressedBytesRead?.Invoke( - this, - new CompressedBytesReadEventArgs( - currentFilePartCompressedBytesRead: currentPartCompressedBytes, - compressedBytesRead: compressedReadBytes - ) - ); - - void IExtractionListener.FireFilePartExtractionBegin( - string name, - long size, - long compressedSize - ) => - FilePartExtractionBegin?.Invoke( - this, - new FilePartExtractionBeginEventArgs( - compressedSize: compressedSize, - size: size, - name: name - ) - ); - - void IReaderExtractionListener.FireEntryExtractionProgress( - Entry entry, - long bytesTransferred, - int iterations - ) => - EntryExtractionProgress?.Invoke( - this, - new ReaderExtractionEventArgs( - entry, - new ReaderProgress(entry, bytesTransferred, iterations) - ) - ); + IEntry IAsyncReader.Entry => Entry; } diff --git a/src/SharpCompress/Readers/Ace/AceReader.Factory.cs b/src/SharpCompress/Readers/Ace/AceReader.Factory.cs new file mode 100644 index 00000000..98c3124a --- /dev/null +++ b/src/SharpCompress/Readers/Ace/AceReader.Factory.cs @@ -0,0 +1,90 @@ +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Readers.Ace; + +public partial class AceReader +#if NET8_0_OR_GREATER + : IReaderOpenable +#endif +{ + /// + /// Opens an AceReader for non-seeking usage with a single volume. + /// + /// The stream containing the ACE archive. + /// Reader options. + /// An AceReader instance. + public static IReader OpenReader(Stream stream, ReaderOptions? readerOptions = null) + { + stream.RequireReadable(); + return new SingleVolumeAceReader(stream, readerOptions ?? ReaderOptions.ForExternalStream); + } + + /// + /// Opens an AceReader for Non-seeking usage with multiple volumes + /// + /// + /// + /// + public static IReader OpenReader(IEnumerable streams, ReaderOptions? options = null) + { + var streamArray = streams.RequireReadable(); + return new MultiVolumeAceReader(streamArray, options ?? ReaderOptions.ForExternalStream); + } + + public static ValueTask OpenAsyncReader( + string filePath, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + filePath.NotNullOrEmpty(nameof(filePath)); + return new((IAsyncReader)OpenReader(new FileInfo(filePath), readerOptions)); + } + + public static ValueTask OpenAsyncReader( + Stream stream, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncReader)OpenReader(stream, readerOptions)); + } + + public static IAsyncReader OpenAsyncReader( + IEnumerable streams, + ReaderOptions? options = null + ) + { + var streamArray = streams.RequireReadable(); + return new MultiVolumeAceReader(streamArray, options ?? ReaderOptions.ForExternalStream); + } + + public static ValueTask OpenAsyncReader( + FileInfo fileInfo, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncReader)OpenReader(fileInfo, readerOptions)); + } + + public static IReader OpenReader(string filePath, ReaderOptions? readerOptions = null) + { + filePath.NotNullOrEmpty(nameof(filePath)); + return OpenReader(new FileInfo(filePath), readerOptions); + } + + public static IReader OpenReader(FileInfo fileInfo, ReaderOptions? readerOptions = null) + { + fileInfo.NotNull(nameof(fileInfo)); + readerOptions ??= ReaderOptions.ForFilePath; + return OpenReader(fileInfo.OpenRead(), readerOptions); + } +} diff --git a/src/SharpCompress/Readers/Ace/AceReader.cs b/src/SharpCompress/Readers/Ace/AceReader.cs new file mode 100644 index 00000000..351e8566 --- /dev/null +++ b/src/SharpCompress/Readers/Ace/AceReader.cs @@ -0,0 +1,123 @@ +using System.Collections.Generic; +using System.IO; +using SharpCompress.Common; +using SharpCompress.Common.Ace; +using SharpCompress.Common.Ace.Headers; + +namespace SharpCompress.Readers.Ace; + +/// +/// Reader for ACE archives. +/// ACE is a proprietary archive format. This implementation supports both ACE 1.0 and ACE 2.0 formats +/// and can read archive metadata and extract uncompressed (stored) entries. +/// Compressed entries require proprietary decompression algorithms that are not publicly documented. +/// +/// +/// ACE 2.0 additions over ACE 1.0: +/// - Improved LZ77 compression (compression type 2) +/// - Recovery record support +/// - Additional header flags +/// +public abstract partial class AceReader : AbstractReader +{ + private readonly IArchiveEncoding _archiveEncoding; + + internal AceReader(ReaderOptions options) + : base(options, ArchiveType.Ace) + { + _archiveEncoding = Options.ArchiveEncoding; + } + + /// + /// Derived class must create or manage the Volume itself. + /// AbstractReader.Volume is get-only, so it cannot be set here. + /// + public override AceVolume? Volume => _volume; + + private AceVolume? _volume; + + protected abstract void ValidateArchive(AceVolume archive); + + protected override IEnumerable GetEntries(Stream stream) + { + if (_volume == null) + { + //this resets the stream + _volume = new AceVolume(stream, Options, 0); + ValidateArchive(_volume); + } + + var mainHeaderReader = new AceMainHeader(_archiveEncoding); + var mainHeader = mainHeaderReader.Read(stream); + if (mainHeader == null) + { + yield break; + } + + if (mainHeader.IsMultiVolume) + { + throw new MultiVolumeExtractionException("Multi volumes are currently not supported"); + } + + var localHeaderReader = new AceFileHeader(_archiveEncoding); + while (true) + { + var localHeader = localHeaderReader.Read(stream); + if (localHeader?.IsFileEncrypted == true) + { + throw new CryptographicException( + "Password protected archives are currently not supported" + ); + } + if (localHeader == null) + { + break; + } + + yield return new AceEntry(new AceFilePart((AceFileHeader)localHeader, stream), Options); + } + } + + protected override async IAsyncEnumerable GetEntriesAsync(Stream stream) + { + if (_volume == null) + { + //this resets the stream + _volume = new AceVolume(stream, Options, 0); + ValidateArchive(_volume); + } + + var mainHeaderReader = new AceMainHeader(_archiveEncoding); + var mainHeader = await mainHeaderReader.ReadAsync(stream).ConfigureAwait(false); + if (mainHeader == null) + { + yield break; + } + + if (mainHeader?.IsMultiVolume == true) + { + throw new MultiVolumeExtractionException("Multi volumes are currently not supported"); + } + + var localHeaderReader = new AceFileHeader(_archiveEncoding); + while (true) + { + var localHeader = await localHeaderReader.ReadAsync(stream).ConfigureAwait(false); + if (localHeader?.IsFileEncrypted == true) + { + throw new CryptographicException( + "Password protected archives are currently not supported" + ); + } + if (localHeader == null) + { + break; + } + + yield return new AceEntry(new AceFilePart((AceFileHeader)localHeader, stream), Options); + } + } + + protected virtual IEnumerable CreateFilePartEnumerableForCurrentEntry() => + Entry.Parts; +} diff --git a/src/SharpCompress/Readers/Ace/MultiVolumeAceReader.cs b/src/SharpCompress/Readers/Ace/MultiVolumeAceReader.cs new file mode 100644 index 00000000..c8abfae9 --- /dev/null +++ b/src/SharpCompress/Readers/Ace/MultiVolumeAceReader.cs @@ -0,0 +1,114 @@ +using System; +using System.Collections; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Ace; + +namespace SharpCompress.Readers.Ace; + +internal class MultiVolumeAceReader : AceReader +{ + private readonly IEnumerator streams; + private Stream? tempStream; + + internal MultiVolumeAceReader(IEnumerable streams, ReaderOptions options) + : base(options) => this.streams = streams.GetEnumerator(); + + protected override void ValidateArchive(AceVolume archive) { } + + protected override Stream RequestInitialStream() + { + if (streams.MoveNext()) + { + return streams.Current; + } + throw new MultiVolumeExtractionException( + "No stream provided when requested by MultiVolumeAceReader" + ); + } + + internal override bool NextEntryForCurrentStream() + { + if (!base.NextEntryForCurrentStream()) + { + // if we're got another stream to try to process then do so + return streams.MoveNext() && LoadStreamForReading(streams.Current); + } + return true; + } + + protected override IEnumerable CreateFilePartEnumerableForCurrentEntry() + { + var enumerator = new MultiVolumeStreamEnumerator(this, streams, tempStream); + tempStream = null; + return enumerator; + } + + private class MultiVolumeStreamEnumerator : IEnumerable, IEnumerator + { + private readonly MultiVolumeAceReader reader; + private readonly IEnumerator nextReadableStreams; + private Stream? tempStream; + private bool isFirst = true; + + internal MultiVolumeStreamEnumerator( + MultiVolumeAceReader r, + IEnumerator nextReadableStreams, + Stream? tempStream + ) + { + reader = r; + this.nextReadableStreams = nextReadableStreams; + this.tempStream = tempStream; + } + + public IEnumerator GetEnumerator() => this; + + IEnumerator IEnumerable.GetEnumerator() => this; + + public FilePart Current { get; private set; } = null!; + + public void Dispose() { } + + object IEnumerator.Current => Current; + + public bool MoveNext() + { + if (isFirst) + { + Current = reader.Entry.Parts.First(); + isFirst = false; //first stream already to go + return true; + } + + if (!reader.Entry.IsSplitAfter) + { + return false; + } + if (tempStream != null) + { + reader.LoadStreamForReading(tempStream); + tempStream = null; + } + else if (!nextReadableStreams.MoveNext()) + { + throw new MultiVolumeExtractionException( + "No stream provided when requested by MultiVolumeAceReader" + ); + } + else + { + reader.LoadStreamForReading(nextReadableStreams.Current); + } + + Current = reader.Entry.Parts.First(); + return true; + } + + public void Reset() { } + } +} diff --git a/src/SharpCompress/Readers/Ace/SingleVolumeAceReader.cs b/src/SharpCompress/Readers/Ace/SingleVolumeAceReader.cs new file mode 100644 index 00000000..ba782d23 --- /dev/null +++ b/src/SharpCompress/Readers/Ace/SingleVolumeAceReader.cs @@ -0,0 +1,30 @@ +using System; +using System.IO; +using SharpCompress.Common; +using SharpCompress.Common.Ace; + +namespace SharpCompress.Readers.Ace; + +internal class SingleVolumeAceReader : AceReader +{ + private readonly Stream _stream; + + internal SingleVolumeAceReader(Stream stream, ReaderOptions options) + : base(options) + { + stream.RequireReadable(); + _stream = stream; + } + + protected override Stream RequestInitialStream() => _stream; + + protected override void ValidateArchive(AceVolume archive) + { + if (archive.IsMultiVolume) + { + throw new MultiVolumeExtractionException( + "Streamed archive is a Multi-volume archive. Use a different AceReader method to extract." + ); + } + } +} diff --git a/src/SharpCompress/Readers/Arc/ArcReader.Async.cs b/src/SharpCompress/Readers/Arc/ArcReader.Async.cs new file mode 100644 index 00000000..004135cd --- /dev/null +++ b/src/SharpCompress/Readers/Arc/ArcReader.Async.cs @@ -0,0 +1,25 @@ +using System.Collections.Generic; +using System.IO; +using System.Threading; +using SharpCompress.Common.Arc; + +namespace SharpCompress.Readers.Arc; + +public partial class ArcReader +{ + protected override async IAsyncEnumerable GetEntriesAsync(Stream stream) + { + ArcEntryHeader headerReader = new ArcEntryHeader(Options.ArchiveEncoding); + ArcEntryHeader? header; + while ( + ( + header = await headerReader + .ReadHeaderAsync(stream, CancellationToken.None) + .ConfigureAwait(false) + ) != null + ) + { + yield return new ArcEntry(new ArcFilePart(header, stream), Options); + } + } +} diff --git a/src/SharpCompress/Readers/Arc/ArcReader.Factory.cs b/src/SharpCompress/Readers/Arc/ArcReader.Factory.cs new file mode 100644 index 00000000..e6bbfe9e --- /dev/null +++ b/src/SharpCompress/Readers/Arc/ArcReader.Factory.cs @@ -0,0 +1,55 @@ +#if NET8_0_OR_GREATER +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Readers.Arc; + +public partial class ArcReader : IReaderOpenable +{ + public static ValueTask OpenAsyncReader( + string filePath, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + filePath.NotNullOrEmpty(nameof(filePath)); + return new((IAsyncReader)OpenReader(new FileInfo(filePath), readerOptions)); + } + + public static ValueTask OpenAsyncReader( + Stream stream, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncReader)OpenReader(stream, readerOptions)); + } + + public static ValueTask OpenAsyncReader( + FileInfo fileInfo, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncReader)OpenReader(fileInfo, readerOptions)); + } + + public static IReader OpenReader(string filePath, ReaderOptions? readerOptions = null) + { + filePath.NotNullOrEmpty(nameof(filePath)); + return OpenReader(new FileInfo(filePath), readerOptions); + } + + public static IReader OpenReader(FileInfo fileInfo, ReaderOptions? readerOptions = null) + { + fileInfo.NotNull(nameof(fileInfo)); + readerOptions ??= ReaderOptions.ForFilePath; + return OpenReader(fileInfo.OpenRead(), readerOptions); + } +} +#endif diff --git a/src/SharpCompress/Readers/Arc/ArcReader.cs b/src/SharpCompress/Readers/Arc/ArcReader.cs new file mode 100644 index 00000000..e2a8303d --- /dev/null +++ b/src/SharpCompress/Readers/Arc/ArcReader.cs @@ -0,0 +1,40 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Arc; + +namespace SharpCompress.Readers.Arc; + +public partial class ArcReader : AbstractReader +{ + private ArcReader(Stream stream, ReaderOptions options) + : base(options, ArchiveType.Arc) => Volume = new ArcVolume(stream, options, 0); + + public override ArcVolume Volume { get; } + + /// + /// Opens an ArcReader for Non-seeking usage with a single volume + /// + /// + /// + /// + public static IReader OpenReader(Stream stream, ReaderOptions? readerOptions = null) + { + stream.RequireReadable(); + return new ArcReader(stream, readerOptions ?? ReaderOptions.ForExternalStream); + } + + protected override IEnumerable GetEntries(Stream stream) + { + ArcEntryHeader headerReader = new ArcEntryHeader(Options.ArchiveEncoding); + ArcEntryHeader? header; + while ((header = headerReader.ReadHeader(stream)) != null) + { + yield return new ArcEntry(new ArcFilePart(header, stream), Options); + } + } +} diff --git a/src/SharpCompress/Readers/Arj/ArjReader.Factory.cs b/src/SharpCompress/Readers/Arj/ArjReader.Factory.cs new file mode 100644 index 00000000..1e5689d3 --- /dev/null +++ b/src/SharpCompress/Readers/Arj/ArjReader.Factory.cs @@ -0,0 +1,55 @@ +#if NET8_0_OR_GREATER +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Readers.Arj; + +public partial class ArjReader : IReaderOpenable +{ + public static ValueTask OpenAsyncReader( + string filePath, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + filePath.NotNullOrEmpty(nameof(filePath)); + return new((IAsyncReader)OpenReader(new FileInfo(filePath), readerOptions)); + } + + public static ValueTask OpenAsyncReader( + Stream stream, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncReader)OpenReader(stream, readerOptions)); + } + + public static ValueTask OpenAsyncReader( + FileInfo fileInfo, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncReader)OpenReader(fileInfo, readerOptions)); + } + + public static IReader OpenReader(string filePath, ReaderOptions? readerOptions = null) + { + filePath.NotNullOrEmpty(nameof(filePath)); + return OpenReader(new FileInfo(filePath), readerOptions); + } + + public static IReader OpenReader(FileInfo fileInfo, ReaderOptions? readerOptions = null) + { + fileInfo.NotNull(nameof(fileInfo)); + readerOptions ??= ReaderOptions.ForFilePath; + return OpenReader(fileInfo.OpenRead(), readerOptions); + } +} +#endif diff --git a/src/SharpCompress/Readers/Arj/ArjReader.cs b/src/SharpCompress/Readers/Arj/ArjReader.cs new file mode 100644 index 00000000..79389b87 --- /dev/null +++ b/src/SharpCompress/Readers/Arj/ArjReader.cs @@ -0,0 +1,150 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Arj; +using SharpCompress.Common.Arj.Headers; +using SharpCompress.Readers.Rar; + +namespace SharpCompress.Readers.Arj; + +public abstract partial class ArjReader : AbstractReader +{ + internal ArjReader(ReaderOptions options) + : base(options, ArchiveType.Arj) { } + + /// + /// Derived class must create or manage the Volume itself. + /// AbstractReader.Volume is get-only, so it cannot be set here. + /// + public override ArjVolume? Volume => _volume; + + private ArjVolume? _volume; + + /// + /// Opens an ArjReader for Non-seeking usage with a single volume + /// + /// + /// + /// + public static IReader OpenReader(Stream stream, ReaderOptions? readerOptions = null) + { + stream.RequireReadable(); + return new SingleVolumeArjReader(stream, readerOptions ?? ReaderOptions.ForExternalStream); + } + + /// + /// Opens an ArjReader for Non-seeking usage with multiple volumes + /// + /// + /// + /// + public static IReader OpenReader(IEnumerable streams, ReaderOptions? options = null) + { + var streamArray = streams.RequireReadable(); + return new MultiVolumeArjReader(streamArray, options ?? ReaderOptions.ForExternalStream); + } + + protected abstract void ValidateArchive(ArjVolume archive); + + protected override IEnumerable GetEntries(Stream stream) + { + var encoding = new ArchiveEncoding(); + var mainHeaderReader = new ArjMainHeader(encoding); + var localHeaderReader = new ArjLocalHeader(encoding); + + var mainHeader = mainHeaderReader.Read(stream); + if (mainHeader?.IsVolume == true) + { + throw new MultiVolumeExtractionException("Multi volumes are currently not supported"); + } + if (mainHeader?.IsGabled == true) + { + throw new CryptographicException( + "Password protected archives are currently not supported" + ); + } + + if (_volume == null) + { + _volume = new ArjVolume(stream, Options, 0); + ValidateArchive(_volume); + } + + while (true) + { + var localHeader = localHeaderReader.Read(stream); + if (localHeader == null) + { + break; + } + + // Skip non-file headers (like CommentHeader) + if ( + localHeader.FileType != FileType.Binary + && localHeader.FileType != FileType.Text7Bit + ) + { + continue; + } + + yield return new ArjEntry( + new ArjFilePart((ArjLocalHeader)localHeader, stream), + Options + ); + } + } + + protected override async IAsyncEnumerable GetEntriesAsync(Stream stream) + { + var encoding = new ArchiveEncoding(); + var mainHeaderReader = new ArjMainHeader(encoding); + var localHeaderReader = new ArjLocalHeader(encoding); + + var mainHeader = await mainHeaderReader.ReadAsync(stream).ConfigureAwait(false); + if (mainHeader?.IsVolume == true) + { + throw new MultiVolumeExtractionException("Multi volumes are currently not supported"); + } + if (mainHeader?.IsGabled == true) + { + throw new CryptographicException( + "Password protected archives are currently not supported" + ); + } + + if (_volume == null) + { + _volume = new ArjVolume(stream, Options, 0); + ValidateArchive(_volume); + } + + while (true) + { + var localHeader = await localHeaderReader.ReadAsync(stream).ConfigureAwait(false); + if (localHeader == null) + { + break; + } + + // Skip non-file headers (like CommentHeader) + if ( + localHeader.FileType != FileType.Binary + && localHeader.FileType != FileType.Text7Bit + ) + { + continue; + } + + yield return new ArjEntry( + new ArjFilePart((ArjLocalHeader)localHeader, stream), + Options + ); + } + } + + protected virtual IEnumerable CreateFilePartEnumerableForCurrentEntry() => + Entry.Parts; +} diff --git a/src/SharpCompress/Readers/Arj/MultiVolumeArjReader.cs b/src/SharpCompress/Readers/Arj/MultiVolumeArjReader.cs new file mode 100644 index 00000000..1eb4a9ea --- /dev/null +++ b/src/SharpCompress/Readers/Arj/MultiVolumeArjReader.cs @@ -0,0 +1,115 @@ +using System; +using System.Collections; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Arj; +using SharpCompress.Readers.Rar; + +namespace SharpCompress.Readers.Arj; + +internal class MultiVolumeArjReader : ArjReader +{ + private readonly IEnumerator streams; + private Stream? tempStream; + + internal MultiVolumeArjReader(IEnumerable streams, ReaderOptions options) + : base(options) => this.streams = streams.GetEnumerator(); + + protected override void ValidateArchive(ArjVolume archive) { } + + protected override Stream RequestInitialStream() + { + if (streams.MoveNext()) + { + return streams.Current; + } + throw new MultiVolumeExtractionException( + "No stream provided when requested by MultiVolumeArjReader" + ); + } + + internal override bool NextEntryForCurrentStream() + { + if (!base.NextEntryForCurrentStream()) + { + // if we're got another stream to try to process then do so + return streams.MoveNext() && LoadStreamForReading(streams.Current); + } + return true; + } + + protected override IEnumerable CreateFilePartEnumerableForCurrentEntry() + { + var enumerator = new MultiVolumeStreamEnumerator(this, streams, tempStream); + tempStream = null; + return enumerator; + } + + private class MultiVolumeStreamEnumerator : IEnumerable, IEnumerator + { + private readonly MultiVolumeArjReader reader; + private readonly IEnumerator nextReadableStreams; + private Stream? tempStream; + private bool isFirst = true; + + internal MultiVolumeStreamEnumerator( + MultiVolumeArjReader r, + IEnumerator nextReadableStreams, + Stream? tempStream + ) + { + reader = r; + this.nextReadableStreams = nextReadableStreams; + this.tempStream = tempStream; + } + + public IEnumerator GetEnumerator() => this; + + IEnumerator IEnumerable.GetEnumerator() => this; + + public FilePart Current { get; private set; } = null!; + + public void Dispose() { } + + object IEnumerator.Current => Current; + + public bool MoveNext() + { + if (isFirst) + { + Current = reader.Entry.Parts.First(); + isFirst = false; //first stream already to go + return true; + } + + if (!reader.Entry.IsSplitAfter) + { + return false; + } + if (tempStream != null) + { + reader.LoadStreamForReading(tempStream); + tempStream = null; + } + else if (!nextReadableStreams.MoveNext()) + { + throw new MultiVolumeExtractionException( + "No stream provided when requested by MultiVolumeArjReader" + ); + } + else + { + reader.LoadStreamForReading(nextReadableStreams.Current); + } + + Current = reader.Entry.Parts.First(); + return true; + } + + public void Reset() { } + } +} diff --git a/src/SharpCompress/Readers/Arj/SingleVolumeArjReader.cs b/src/SharpCompress/Readers/Arj/SingleVolumeArjReader.cs new file mode 100644 index 00000000..0128f60e --- /dev/null +++ b/src/SharpCompress/Readers/Arj/SingleVolumeArjReader.cs @@ -0,0 +1,30 @@ +using System; +using System.IO; +using SharpCompress.Common; +using SharpCompress.Common.Arj; + +namespace SharpCompress.Readers.Arj; + +internal class SingleVolumeArjReader : ArjReader +{ + private readonly Stream _stream; + + internal SingleVolumeArjReader(Stream stream, ReaderOptions options) + : base(options) + { + stream.RequireReadable(); + _stream = stream; + } + + protected override Stream RequestInitialStream() => _stream; + + protected override void ValidateArchive(ArjVolume archive) + { + if (archive.IsMultiVolume) + { + throw new MultiVolumeExtractionException( + "Streamed archive is a Multi-volume archive. Use a different ArjReader method to extract." + ); + } + } +} diff --git a/src/SharpCompress/Readers/GZip/GZipReader.Async.cs b/src/SharpCompress/Readers/GZip/GZipReader.Async.cs new file mode 100644 index 00000000..720cb792 --- /dev/null +++ b/src/SharpCompress/Readers/GZip/GZipReader.Async.cs @@ -0,0 +1,15 @@ +using System.Collections.Generic; +using System.IO; +using SharpCompress.Common; +using SharpCompress.Common.GZip; + +namespace SharpCompress.Readers.GZip; + +public partial class GZipReader +{ + /// + /// Returns entries asynchronously for streams that only support async reads. + /// + protected override IAsyncEnumerable GetEntriesAsync(Stream stream) => + GZipEntry.GetEntriesAsync(stream, Options); +} diff --git a/src/SharpCompress/Readers/GZip/GZipReader.Factory.cs b/src/SharpCompress/Readers/GZip/GZipReader.Factory.cs new file mode 100644 index 00000000..a6bca9f6 --- /dev/null +++ b/src/SharpCompress/Readers/GZip/GZipReader.Factory.cs @@ -0,0 +1,61 @@ +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Readers.GZip; + +public partial class GZipReader +#if NET8_0_OR_GREATER + : IReaderOpenable +#endif +{ + public static ValueTask OpenAsyncReader( + string filePath, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + filePath.NotNullOrEmpty(nameof(filePath)); + return new((IAsyncReader)OpenReader(new FileInfo(filePath), readerOptions)); + } + + public static ValueTask OpenAsyncReader( + Stream stream, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncReader)OpenReader(stream, readerOptions)); + } + + public static ValueTask OpenAsyncReader( + FileInfo fileInfo, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncReader)OpenReader(fileInfo, readerOptions)); + } + + public static IReader OpenReader(string filePath, ReaderOptions? readerOptions = null) + { + filePath.NotNullOrEmpty(nameof(filePath)); + return OpenReader(new FileInfo(filePath), readerOptions); + } + + public static IReader OpenReader(FileInfo fileInfo, ReaderOptions? readerOptions = null) + { + fileInfo.NotNull(nameof(fileInfo)); + readerOptions ??= ReaderOptions.ForFilePath; + return OpenReader(fileInfo.OpenRead(), readerOptions); + } + + public static IReader OpenReader(Stream stream, ReaderOptions? readerOptions = null) + { + stream.RequireReadable(); + return new GZipReader(stream, readerOptions ?? ReaderOptions.ForExternalStream); + } +} diff --git a/src/SharpCompress/Readers/GZip/GZipReader.cs b/src/SharpCompress/Readers/GZip/GZipReader.cs index 27394cc7..eb09ab6c 100644 --- a/src/SharpCompress/Readers/GZip/GZipReader.cs +++ b/src/SharpCompress/Readers/GZip/GZipReader.cs @@ -1,33 +1,19 @@ -using System.Collections.Generic; +using System.Collections.Generic; using System.IO; using SharpCompress.Common; using SharpCompress.Common.GZip; namespace SharpCompress.Readers.GZip; -public class GZipReader : AbstractReader +public partial class GZipReader : AbstractReader { - internal GZipReader(Stream stream, ReaderOptions options) - : base(options, ArchiveType.GZip) => Volume = new GZipVolume(stream, options); + private GZipReader(Stream stream, ReaderOptions options) + : base(options, ArchiveType.GZip) => Volume = new GZipVolume(stream, options, 0); public override GZipVolume Volume { get; } - #region Open - - /// - /// Opens a GZipReader for Non-seeking usage with a single volume - /// - /// - /// - /// - public static GZipReader Open(Stream stream, ReaderOptions? options = null) - { - stream.CheckNotNull(nameof(stream)); - return new GZipReader(stream, options ?? new ReaderOptions()); - } - - #endregion Open - protected override IEnumerable GetEntries(Stream stream) => GZipEntry.GetEntries(stream, Options); + + // GetEntriesAsync moved to GZipReader.Async.cs } diff --git a/src/SharpCompress/Readers/IAsyncReader.cs b/src/SharpCompress/Readers/IAsyncReader.cs new file mode 100644 index 00000000..c3cc8de0 --- /dev/null +++ b/src/SharpCompress/Readers/IAsyncReader.cs @@ -0,0 +1,41 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Readers; + +public interface IAsyncReader : IAsyncDisposable +{ + ArchiveType Type { get; } + + IEntry Entry { get; } + + /// + /// Decompresses the current entry to the stream asynchronously. This cannot be called twice for the current entry. + /// + /// + /// + ValueTask WriteEntryToAsync( + Stream writableStream, + CancellationToken cancellationToken = default + ); + + bool Cancelled { get; } + void Cancel(); + + /// + /// Moves to the next entry asynchronously by reading more data from the underlying stream. This skips if data has not been read. + /// + /// + /// + ValueTask MoveToNextEntryAsync(CancellationToken cancellationToken = default); + + /// + /// Opens the current entry asynchronously as a stream that will decompress as it is read. + /// Read the entire stream or use SkipEntry on EntryStream. + /// + /// + ValueTask OpenEntryStreamAsync(CancellationToken cancellationToken = default); +} diff --git a/src/SharpCompress/Readers/IAsyncReaderExtensions.cs b/src/SharpCompress/Readers/IAsyncReaderExtensions.cs new file mode 100644 index 00000000..7515b257 --- /dev/null +++ b/src/SharpCompress/Readers/IAsyncReaderExtensions.cs @@ -0,0 +1,162 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.IO; + +namespace SharpCompress.Readers; + +public static class IAsyncReaderExtensions +{ + extension(IAsyncReader reader) + { + /// + /// Extract to specific directory asynchronously, retaining filename + /// + public async ValueTask WriteEntryToDirectoryAsync( + string destinationDirectory, + ExtractionOptions? options = null, + CancellationToken cancellationToken = default + ) => + await reader + .Entry.WriteEntryToDirectoryAsync( + destinationDirectory, + options, + async (path, ct) => + await reader.WriteEntryToFileAsync(path, options, ct).ConfigureAwait(false), + cancellationToken + ) + .ConfigureAwait(false); + + /// + /// Extract to specific file asynchronously + /// + public async ValueTask WriteEntryToFileAsync( + string destinationFileName, + ExtractionOptions? options = null, + CancellationToken cancellationToken = default + ) + { + options ??= new ExtractionOptions(); + await reader + .Entry.WriteEntryToFileAsync( + destinationFileName, + options, + async (x, fm, ct) => + { + using var fs = File.Open(x, fm); + await CopyEntryToAsync(reader, fs, options, ct).ConfigureAwait(false); + }, + cancellationToken + ) + .ConfigureAwait(false); + } + + /// + /// Extract all remaining unread entries to specific directory asynchronously, retaining filename + /// + public async ValueTask WriteAllToDirectoryAsync( + string destinationDirectory, + ExtractionOptions? options = null, + CancellationToken cancellationToken = default + ) + { + while (await reader.MoveToNextEntryAsync(cancellationToken).ConfigureAwait(false)) + { + await reader + .WriteEntryToDirectoryAsync(destinationDirectory, options, cancellationToken) + .ConfigureAwait(false); + } + } + + public async ValueTask WriteEntryToAsync( + string destinationFileName, + ExtractionOptions? options = null, + CancellationToken cancellationToken = default + ) + { + options ??= new ExtractionOptions(); + await reader + .Entry.WriteEntryToFileAsync( + destinationFileName, + options, + async (x, fm, ct) => + { + using var fs = File.Open(x, fm); + await CopyEntryToAsync(reader, fs, options, ct).ConfigureAwait(false); + }, + cancellationToken + ) + .ConfigureAwait(false); + } + + public async ValueTask WriteEntryToAsync( + FileInfo destinationFileInfo, + ExtractionOptions? options = null, + CancellationToken cancellationToken = default + ) => + await reader + .WriteEntryToAsync(destinationFileInfo.FullName, options, cancellationToken) + .ConfigureAwait(false); + } + + private static async ValueTask CopyEntryToAsync( + IAsyncReader reader, + Stream writableStream, + ExtractionOptions options, + CancellationToken cancellationToken + ) + { +#if LEGACY_DOTNET + using var entryStream = await reader + .OpenEntryStreamAsync(cancellationToken) + .ConfigureAwait(false); +#else + await using var entryStream = await reader + .OpenEntryStreamAsync(cancellationToken) + .ConfigureAwait(false); +#endif + var checkedStream = IEntryExtensions.WrapWithChecksumValidation( + reader.Entry, + entryStream, + options + ); + var sourceStream = WrapWithProgress(checkedStream, reader.Entry); + await sourceStream + .CopyToAsync(writableStream, options.BufferSize, cancellationToken) + .ConfigureAwait(false); + } + + private static Stream WrapWithProgress(Stream source, IEntry entry) + { + var progress = entry.Options.Progress; + if (progress is null) + { + return source; + } + + var entryPath = entry.Key ?? string.Empty; + var totalBytes = GetEntrySizeSafe(entry); + return new ProgressReportingStream( + source, + progress, + entryPath, + totalBytes, + leaveOpen: true + ); + } + + private static long? GetEntrySizeSafe(IEntry entry) + { + try + { + var size = entry.Size; + return size >= 0 ? size : null; + } + catch (NotImplementedException) + { + return null; + } + } +} diff --git a/src/SharpCompress/Readers/IReader.cs b/src/SharpCompress/Readers/IReader.cs index 50fc7f4d..f7642ffc 100644 --- a/src/SharpCompress/Readers/IReader.cs +++ b/src/SharpCompress/Readers/IReader.cs @@ -6,12 +6,7 @@ namespace SharpCompress.Readers; public interface IReader : IDisposable { - event EventHandler> EntryExtractionProgress; - - event EventHandler CompressedBytesRead; - event EventHandler FilePartExtractionBegin; - - ArchiveType ArchiveType { get; } + ArchiveType Type { get; } IEntry Entry { get; } diff --git a/src/SharpCompress/Readers/IReaderExtensions.cs b/src/SharpCompress/Readers/IReaderExtensions.cs index 0e51d72a..8eb97193 100644 --- a/src/SharpCompress/Readers/IReaderExtensions.cs +++ b/src/SharpCompress/Readers/IReaderExtensions.cs @@ -1,68 +1,116 @@ -using System.IO; +using System; +using System.IO; using SharpCompress.Common; +using SharpCompress.IO; namespace SharpCompress.Readers; public static class IReaderExtensions { - public static void WriteEntryTo(this IReader reader, string filePath) + extension(IReader reader) { - using Stream stream = File.Open(filePath, FileMode.Create, FileAccess.Write); - reader.WriteEntryTo(stream); - } - - public static void WriteEntryTo(this IReader reader, FileInfo filePath) - { - using Stream stream = filePath.Open(FileMode.Create); - reader.WriteEntryTo(stream); - } - - /// - /// Extract all remaining unread entries to specific directory, retaining filename - /// - public static void WriteAllToDirectory( - this IReader reader, - string destinationDirectory, - ExtractionOptions? options = null - ) - { - while (reader.MoveToNextEntry()) + public void WriteEntryTo(string filePath) { - reader.WriteEntryToDirectory(destinationDirectory, options); + using Stream stream = File.Open(filePath, FileMode.Create, FileAccess.Write); + reader.WriteEntryTo(stream); + } + + public void WriteEntryTo(FileInfo filePath) + { + using Stream stream = filePath.Open(FileMode.Create); + reader.WriteEntryTo(stream); + } + + /// + /// Extract all remaining unread entries to specific directory, retaining filename + /// + public void WriteAllToDirectory( + string destinationDirectory, + ExtractionOptions? options = null + ) + { + while (reader.MoveToNextEntry()) + { + reader.WriteEntryToDirectory(destinationDirectory, options); + } + } + + /// + /// Extract to specific directory, retaining filename + /// + public void WriteEntryToDirectory( + string destinationDirectory, + ExtractionOptions? options = null + ) => + reader.Entry.WriteEntryToDirectory( + destinationDirectory, + options, + (path) => reader.WriteEntryToFile(path, options) + ); + + /// + /// Extract to specific file + /// + public void WriteEntryToFile(string destinationFileName, ExtractionOptions? options = null) + { + options ??= new ExtractionOptions(); + reader.Entry.WriteEntryToFile( + destinationFileName, + options, + (x, fm) => + { + using var fs = File.Open(x, fm); + CopyEntryTo(reader, fs, options ?? new ExtractionOptions()); + } + ); } } - /// - /// Extract to specific directory, retaining filename - /// - public static void WriteEntryToDirectory( - this IReader reader, - string destinationDirectory, - ExtractionOptions? options = null - ) => - ExtractionMethods.WriteEntryToDirectory( + private static void CopyEntryTo( + IReader reader, + Stream writableStream, + ExtractionOptions options + ) + { + using var entryStream = reader.OpenEntryStream(); + var checkedStream = IEntryExtensions.WrapWithChecksumValidation( reader.Entry, - destinationDirectory, - options, - reader.WriteEntryToFile + entryStream, + options ); + var sourceStream = WrapWithProgress(checkedStream, reader.Entry); + sourceStream.CopyTo(writableStream, options.BufferSize); + } - /// - /// Extract to specific file - /// - public static void WriteEntryToFile( - this IReader reader, - string destinationFileName, - ExtractionOptions? options = null - ) => - ExtractionMethods.WriteEntryToFile( - reader.Entry, - destinationFileName, - options, - (x, fm) => - { - using var fs = File.Open(destinationFileName, fm); - reader.WriteEntryTo(fs); - } + private static Stream WrapWithProgress(Stream source, IEntry entry) + { + var progress = entry.Options.Progress; + if (progress is null) + { + return source; + } + + var entryPath = entry.Key ?? string.Empty; + var totalBytes = GetEntrySizeSafe(entry); + return new ProgressReportingStream( + source, + progress, + entryPath, + totalBytes, + leaveOpen: true ); + } + + private static long? GetEntrySizeSafe(IEntry entry) + { + try + { + var size = entry.Size; + return size >= 0 ? size : null; + } + catch (NotImplementedException) + { + return null; + } + } } diff --git a/src/SharpCompress/Readers/IReaderExtractionListener.cs b/src/SharpCompress/Readers/IReaderExtractionListener.cs deleted file mode 100644 index 49e5dae4..00000000 --- a/src/SharpCompress/Readers/IReaderExtractionListener.cs +++ /dev/null @@ -1,8 +0,0 @@ -using SharpCompress.Common; - -namespace SharpCompress.Readers; - -public interface IReaderExtractionListener : IExtractionListener -{ - void FireEntryExtractionProgress(Entry entry, long sizeTransferred, int iterations); -} diff --git a/src/SharpCompress/Readers/IReaderFactory.cs b/src/SharpCompress/Readers/IReaderFactory.cs index 08c190a3..9b3c533c 100644 --- a/src/SharpCompress/Readers/IReaderFactory.cs +++ b/src/SharpCompress/Readers/IReaderFactory.cs @@ -1,14 +1,29 @@ using System.IO; +using System.Threading; +using System.Threading.Tasks; namespace SharpCompress.Readers; public interface IReaderFactory : Factories.IFactory { /// - /// Opens a Reader for Non-seeking usage + /// Opens a Reader for Non-seeking usage. /// - /// - /// - /// + /// An open, readable stream. + /// Reader options. + /// The opened reader. IReader OpenReader(Stream stream, ReaderOptions? options); + + /// + /// Opens a Reader for Non-seeking usage asynchronously. + /// + /// An open, readable stream. + /// Reader options. + /// Cancellation token. + /// A containing the opened async reader. + ValueTask OpenAsyncReader( + Stream stream, + ReaderOptions? options, + CancellationToken cancellationToken = default + ); } diff --git a/src/SharpCompress/Readers/IReaderOpenable.cs b/src/SharpCompress/Readers/IReaderOpenable.cs new file mode 100644 index 00000000..58604211 --- /dev/null +++ b/src/SharpCompress/Readers/IReaderOpenable.cs @@ -0,0 +1,37 @@ +#if NET8_0_OR_GREATER +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Readers; + +public interface IReaderOpenable +{ + public static abstract IReader OpenReader(string filePath, ReaderOptions? readerOptions = null); + + public static abstract IReader OpenReader( + FileInfo fileInfo, + ReaderOptions? readerOptions = null + ); + + public static abstract IReader OpenReader(Stream stream, ReaderOptions? readerOptions = null); + + public static abstract ValueTask OpenAsyncReader( + string filePath, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ); + + public static abstract ValueTask OpenAsyncReader( + Stream stream, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ); + + public static abstract ValueTask OpenAsyncReader( + FileInfo fileInfo, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ); +} +#endif diff --git a/src/SharpCompress/Readers/Lzw/LzwReader.Async.cs b/src/SharpCompress/Readers/Lzw/LzwReader.Async.cs new file mode 100644 index 00000000..6479c9b5 --- /dev/null +++ b/src/SharpCompress/Readers/Lzw/LzwReader.Async.cs @@ -0,0 +1,15 @@ +using System.Collections.Generic; +using System.IO; +using SharpCompress.Common; +using SharpCompress.Common.Lzw; + +namespace SharpCompress.Readers.Lzw; + +public partial class LzwReader +{ + /// + /// Returns entries asynchronously for streams that only support async reads. + /// + protected override IAsyncEnumerable GetEntriesAsync(Stream stream) => + LzwEntry.GetEntriesAsync(stream, Options); +} diff --git a/src/SharpCompress/Readers/Lzw/LzwReader.Factory.cs b/src/SharpCompress/Readers/Lzw/LzwReader.Factory.cs new file mode 100644 index 00000000..a1535c89 --- /dev/null +++ b/src/SharpCompress/Readers/Lzw/LzwReader.Factory.cs @@ -0,0 +1,61 @@ +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Readers.Lzw; + +public partial class LzwReader +#if NET8_0_OR_GREATER + : IReaderOpenable +#endif +{ + public static ValueTask OpenAsyncReader( + string filePath, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + filePath.NotNullOrEmpty(nameof(filePath)); + return new((IAsyncReader)OpenReader(new FileInfo(filePath), readerOptions)); + } + + public static ValueTask OpenAsyncReader( + Stream stream, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncReader)OpenReader(stream, readerOptions)); + } + + public static ValueTask OpenAsyncReader( + FileInfo fileInfo, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncReader)OpenReader(fileInfo, readerOptions)); + } + + public static IReader OpenReader(string filePath, ReaderOptions? readerOptions = null) + { + filePath.NotNullOrEmpty(nameof(filePath)); + return OpenReader(new FileInfo(filePath), readerOptions); + } + + public static IReader OpenReader(FileInfo fileInfo, ReaderOptions? readerOptions = null) + { + fileInfo.NotNull(nameof(fileInfo)); + readerOptions ??= ReaderOptions.ForFilePath; + return OpenReader(fileInfo.OpenRead(), readerOptions); + } + + public static IReader OpenReader(Stream stream, ReaderOptions? readerOptions = null) + { + stream.RequireReadable(); + return new LzwReader(stream, readerOptions ?? ReaderOptions.ForExternalStream); + } +} diff --git a/src/SharpCompress/Readers/Lzw/LzwReader.cs b/src/SharpCompress/Readers/Lzw/LzwReader.cs new file mode 100644 index 00000000..875faf7a --- /dev/null +++ b/src/SharpCompress/Readers/Lzw/LzwReader.cs @@ -0,0 +1,19 @@ +using System.Collections.Generic; +using System.IO; +using SharpCompress.Common; +using SharpCompress.Common.Lzw; + +namespace SharpCompress.Readers.Lzw; + +public partial class LzwReader : AbstractReader +{ + private LzwReader(Stream stream, ReaderOptions options) + : base(options, ArchiveType.Lzw) => Volume = new LzwVolume(stream, options, 0); + + public override LzwVolume Volume { get; } + + protected override IEnumerable GetEntries(Stream stream) => + LzwEntry.GetEntries(stream, Options); + + // GetEntriesAsync moved to LzwReader.Async.cs +} diff --git a/src/SharpCompress/Readers/Rar/MultiVolumeRarReader.Async.cs b/src/SharpCompress/Readers/Rar/MultiVolumeRarReader.Async.cs new file mode 100644 index 00000000..620b28c6 --- /dev/null +++ b/src/SharpCompress/Readers/Rar/MultiVolumeRarReader.Async.cs @@ -0,0 +1,84 @@ +using System.Collections; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Rar; + +namespace SharpCompress.Readers.Rar; + +internal partial class MultiVolumeRarReader : RarReader +{ + protected override IAsyncEnumerable CreateFilePartEnumerableForCurrentEntryAsync() + { + var enumerator = new MultiVolumeStreamAsyncEnumerator(this, streams, tempStream); + tempStream = null; + return enumerator; + } + + private class MultiVolumeStreamAsyncEnumerator + : IAsyncEnumerable, + IAsyncEnumerator + { + private readonly MultiVolumeRarReader reader; + private readonly IEnumerator nextReadableStreams; + private Stream? tempStream; + private bool isFirst = true; + + internal MultiVolumeStreamAsyncEnumerator( + MultiVolumeRarReader r, + IEnumerator nextReadableStreams, + Stream? tempStream + ) + { + reader = r; + this.nextReadableStreams = nextReadableStreams; + this.tempStream = tempStream; + } + + public FilePart Current { get; private set; } = null!; + + public async ValueTask MoveNextAsync() + { + if (isFirst) + { + Current = reader.Entry.Parts.First(); + isFirst = false; //first stream already to go + return true; + } + + if (!reader.Entry.IsSplitAfter) + { + return false; + } + if (tempStream != null) + { + await reader.LoadStreamForReadingAsync(tempStream).ConfigureAwait(false); + tempStream = null; + } + else if (!nextReadableStreams.MoveNext()) + { + throw new MultiVolumeExtractionException( + "No stream provided when requested by MultiVolumeRarReader" + ); + } + else + { + await reader + .LoadStreamForReadingAsync(nextReadableStreams.Current) + .ConfigureAwait(false); + } + + Current = reader.Entry.Parts.First(); + return true; + } + + public IAsyncEnumerator GetAsyncEnumerator( + CancellationToken cancellationToken = new() + ) => this; + + public ValueTask DisposeAsync() => new(); + } +} diff --git a/src/SharpCompress/Readers/Rar/MultiVolumeRarReader.cs b/src/SharpCompress/Readers/Rar/MultiVolumeRarReader.cs index 76899e0c..5456ff6d 100644 --- a/src/SharpCompress/Readers/Rar/MultiVolumeRarReader.cs +++ b/src/SharpCompress/Readers/Rar/MultiVolumeRarReader.cs @@ -1,23 +1,23 @@ -#nullable disable - using System.Collections; using System.Collections.Generic; using System.IO; using System.Linq; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common; using SharpCompress.Common.Rar; namespace SharpCompress.Readers.Rar; -internal class MultiVolumeRarReader : RarReader +internal partial class MultiVolumeRarReader : RarReader { private readonly IEnumerator streams; - private Stream tempStream; + private Stream? tempStream; internal MultiVolumeRarReader(IEnumerable streams, ReaderOptions options) : base(options) => this.streams = streams.GetEnumerator(); - internal override void ValidateArchive(RarVolume archive) { } + protected override void ValidateArchive(RarVolume archive) { } protected override Stream RequestInitialStream() { @@ -47,17 +47,19 @@ internal class MultiVolumeRarReader : RarReader return enumerator; } + // Async method and MultiVolumeStreamAsyncEnumerator moved to MultiVolumeRarReader.Async.cs + private class MultiVolumeStreamEnumerator : IEnumerable, IEnumerator { private readonly MultiVolumeRarReader reader; private readonly IEnumerator nextReadableStreams; - private Stream tempStream; + private Stream? tempStream; private bool isFirst = true; internal MultiVolumeStreamEnumerator( MultiVolumeRarReader r, IEnumerator nextReadableStreams, - Stream tempStream + Stream? tempStream ) { reader = r; @@ -69,7 +71,7 @@ internal class MultiVolumeRarReader : RarReader IEnumerator IEnumerable.GetEnumerator() => this; - public FilePart Current { get; private set; } + public FilePart Current { get; private set; } = null!; public void Dispose() { } diff --git a/src/SharpCompress/Readers/Rar/NonSeekableStreamFilePart.cs b/src/SharpCompress/Readers/Rar/NonSeekableStreamFilePart.cs index 3a43e60c..94ee6bf2 100644 --- a/src/SharpCompress/Readers/Rar/NonSeekableStreamFilePart.cs +++ b/src/SharpCompress/Readers/Rar/NonSeekableStreamFilePart.cs @@ -9,7 +9,7 @@ internal class NonSeekableStreamFilePart : RarFilePart internal NonSeekableStreamFilePart(MarkHeader mh, FileHeader fh, int index = 0) : base(mh, fh, index) { } - internal override Stream GetCompressedStream() => FileHeader.PackedStream; + internal override Stream? GetCompressedStream() => FileHeader.PackedStream; internal override Stream? GetRawStream() => FileHeader.PackedStream; diff --git a/src/SharpCompress/Readers/Rar/RarReader.Async.cs b/src/SharpCompress/Readers/Rar/RarReader.Async.cs new file mode 100644 index 00000000..d8cd74e4 --- /dev/null +++ b/src/SharpCompress/Readers/Rar/RarReader.Async.cs @@ -0,0 +1,62 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Rar; +using SharpCompress.Compressors.Rar; + +namespace SharpCompress.Readers.Rar; + +public abstract partial class RarReader +{ + /// + /// Returns file parts asynchronously for the current entry. + /// Used for async stream operations in solid RAR archives. + /// + protected virtual IAsyncEnumerable CreateFilePartEnumerableForCurrentEntryAsync() => + Entry.Parts.ToAsyncEnumerable(); + + /// + /// Asynchronously creates an entry stream for the current entry. + /// Supports both RAR v3 and v5 archives with proper CRC verification. + /// + protected override async ValueTask GetEntryStreamAsync( + CancellationToken cancellationToken = default + ) + { + if (Entry.IsRedir) + { + throw new ArchiveOperationException("no stream for redirect entry"); + } + + var stream = await MultiVolumeReadOnlyAsyncStream + .Create(CreateFilePartEnumerableForCurrentEntryAsync().CastAsync()) + .ConfigureAwait(false); + if (Entry.IsRarV3) + { + return CreateEntryStream( + await RarCrcStream + .CreateAsync(UnpackV1.Value, Entry.FileHeader, stream, cancellationToken) + .ConfigureAwait(false) + ); + } + + if (Entry.FileHeader.FileCrc?.Length > 5) + { + return CreateEntryStream( + await RarBLAKE2spStream + .CreateAsync(UnpackV2017.Value, Entry.FileHeader, stream, cancellationToken) + .ConfigureAwait(false) + ); + } + + return CreateEntryStream( + await RarCrcStream + .CreateAsync(UnpackV2017.Value, Entry.FileHeader, stream, cancellationToken) + .ConfigureAwait(false) + ); + } +} diff --git a/src/SharpCompress/Readers/Rar/RarReader.Factory.cs b/src/SharpCompress/Readers/Rar/RarReader.Factory.cs new file mode 100644 index 00000000..20ec829a --- /dev/null +++ b/src/SharpCompress/Readers/Rar/RarReader.Factory.cs @@ -0,0 +1,42 @@ +#if NET8_0_OR_GREATER +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Readers.Rar; + +public partial class RarReader : IReaderOpenable +{ + public static ValueTask OpenAsyncReader( + string filePath, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + filePath.NotNullOrEmpty(nameof(filePath)); + return new((IAsyncReader)OpenReader(new FileInfo(filePath), readerOptions)); + } + + public static ValueTask OpenAsyncReader( + Stream stream, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncReader)OpenReader(stream, readerOptions)); + } + + public static ValueTask OpenAsyncReader( + FileInfo fileInfo, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncReader)OpenReader(fileInfo, readerOptions)); + } +} +#endif diff --git a/src/SharpCompress/Readers/Rar/RarReader.cs b/src/SharpCompress/Readers/Rar/RarReader.cs index da8c132b..829cf97a 100644 --- a/src/SharpCompress/Readers/Rar/RarReader.cs +++ b/src/SharpCompress/Readers/Rar/RarReader.cs @@ -1,6 +1,8 @@ +using System; using System.Collections.Generic; using System.IO; using System.Linq; +using System.Threading.Tasks; using SharpCompress.Common; using SharpCompress.Common.Rar; using SharpCompress.Compressors.Rar; @@ -10,31 +12,72 @@ namespace SharpCompress.Readers.Rar; /// /// This class faciliates Reading a Rar Archive in a non-seekable forward-only manner /// -public abstract class RarReader : AbstractReader +public abstract partial class RarReader : AbstractReader { + private bool _disposed; private RarVolume? volume; - internal Lazy UnpackV2017 { get; } = - new Lazy(() => new Compressors.Rar.UnpackV2017.Unpack()); - internal Lazy UnpackV1 { get; } = - new Lazy(() => new Compressors.Rar.UnpackV1.Unpack()); + private Lazy UnpackV2017 { get; } = + new(() => new Compressors.Rar.UnpackV2017.Unpack()); + private Lazy UnpackV1 { get; } = new(() => new Compressors.Rar.UnpackV1.Unpack()); internal RarReader(ReaderOptions options) : base(options, ArchiveType.Rar) { } - internal abstract void ValidateArchive(RarVolume archive); + public override void Dispose() + { + if (!_disposed) + { + if (UnpackV1.IsValueCreated && UnpackV1.Value is IDisposable unpackV1) + { + unpackV1.Dispose(); + } + if (UnpackV2017.IsValueCreated && UnpackV2017.Value is IDisposable unpackV2017) + { + unpackV2017.Dispose(); + } - public override RarVolume Volume => volume!; + _disposed = true; + base.Dispose(); + } + } + + protected abstract void ValidateArchive(RarVolume archive); + + public override RarVolume? Volume => volume; + + public static IReader OpenReader(string filePath, ReaderOptions? readerOptions = null) + { + filePath.NotNullOrEmpty(nameof(filePath)); + return OpenReader(new FileInfo(filePath), readerOptions); + } + + public static IReader OpenReader(FileInfo fileInfo, ReaderOptions? readerOptions = null) + { + readerOptions ??= ReaderOptions.ForFilePath; + return OpenReader(fileInfo.OpenRead(), readerOptions); + } + + public static IReader OpenReader(IEnumerable filePaths, ReaderOptions? options = null) + { + return OpenReader(filePaths.Select(x => new FileInfo(x)), options); + } + + public static IReader OpenReader(IEnumerable fileInfos, ReaderOptions? options = null) + { + options ??= ReaderOptions.ForFilePath; + return OpenReader(fileInfos.Select(x => x.OpenRead()), options); + } /// /// Opens a RarReader for Non-seeking usage with a single volume /// /// - /// + /// /// - public static RarReader Open(Stream stream, ReaderOptions? options = null) + public static IReader OpenReader(Stream stream, ReaderOptions? readerOptions = null) { - stream.CheckNotNull(nameof(stream)); - return new SingleVolumeRarReader(stream, options ?? new ReaderOptions()); + stream.RequireReadable(); + return new SingleVolumeRarReader(stream, readerOptions ?? ReaderOptions.ForExternalStream); } /// @@ -43,19 +86,29 @@ public abstract class RarReader : AbstractReader /// /// /// - public static RarReader Open(IEnumerable streams, ReaderOptions? options = null) + public static IReader OpenReader(IEnumerable streams, ReaderOptions? options = null) { - streams.CheckNotNull(nameof(streams)); - return new MultiVolumeRarReader(streams, options ?? new ReaderOptions()); + var streamArray = streams.RequireReadable(); + return new MultiVolumeRarReader(streamArray, options ?? ReaderOptions.ForExternalStream); } protected override IEnumerable GetEntries(Stream stream) { - volume = new RarReaderVolume(stream, Options); + volume = new RarReaderVolume(stream, Options, 0); foreach (var fp in volume.ReadFileParts()) { ValidateArchive(volume); - yield return new RarReaderEntry(volume.IsSolidArchive, fp); + yield return new RarReaderEntry(volume.IsSolidArchive, fp, Options); + } + } + + protected override async IAsyncEnumerable GetEntriesAsync(Stream stream) + { + volume = new RarReaderVolume(stream, Options, 0); + await foreach (var fp in volume.ReadFilePartsAsync().ConfigureAwait(false)) + { + ValidateArchive(volume); + yield return new RarReaderEntry(volume.IsSolidArchive, fp, Options); } } @@ -64,14 +117,28 @@ public abstract class RarReader : AbstractReader protected override EntryStream GetEntryStream() { + if (Entry.IsRedir) + { + throw new ArchiveOperationException("no stream for redirect entry"); + } + var stream = new MultiVolumeReadOnlyStream( - CreateFilePartEnumerableForCurrentEntry().Cast(), - this + CreateFilePartEnumerableForCurrentEntry().Cast() ); if (Entry.IsRarV3) { - return CreateEntryStream(new RarCrcStream(UnpackV1.Value, Entry.FileHeader, stream)); + return CreateEntryStream(RarCrcStream.Create(UnpackV1.Value, Entry.FileHeader, stream)); } - return CreateEntryStream(new RarCrcStream(UnpackV2017.Value, Entry.FileHeader, stream)); + + if (Entry.FileHeader.FileCrc?.Length > 5) + { + return CreateEntryStream( + RarBLAKE2spStream.Create(UnpackV2017.Value, Entry.FileHeader, stream) + ); + } + + return CreateEntryStream(RarCrcStream.Create(UnpackV2017.Value, Entry.FileHeader, stream)); } + + // GetEntryStreamAsync moved to RarReader.Async.cs } diff --git a/src/SharpCompress/Readers/Rar/RarReaderEntry.cs b/src/SharpCompress/Readers/Rar/RarReaderEntry.cs index d6f7222f..de55de91 100644 --- a/src/SharpCompress/Readers/Rar/RarReaderEntry.cs +++ b/src/SharpCompress/Readers/Rar/RarReaderEntry.cs @@ -1,5 +1,6 @@ using System.Collections.Generic; using SharpCompress.Common; +using SharpCompress.Common.Options; using SharpCompress.Common.Rar; using SharpCompress.Common.Rar.Headers; @@ -7,7 +8,8 @@ namespace SharpCompress.Readers.Rar; public class RarReaderEntry : RarEntry { - internal RarReaderEntry(bool solid, RarFilePart part) + internal RarReaderEntry(bool solid, RarFilePart part, IReaderOptions readerOptions) + : base(readerOptions) { Part = part; IsSolid = solid; diff --git a/src/SharpCompress/Readers/Rar/RarReaderVolume.cs b/src/SharpCompress/Readers/Rar/RarReaderVolume.cs index 85beb897..d38ea524 100644 --- a/src/SharpCompress/Readers/Rar/RarReaderVolume.cs +++ b/src/SharpCompress/Readers/Rar/RarReaderVolume.cs @@ -1,5 +1,7 @@ using System.Collections.Generic; using System.IO; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common.Rar; using SharpCompress.Common.Rar.Headers; using SharpCompress.IO; @@ -8,11 +10,14 @@ namespace SharpCompress.Readers.Rar; public class RarReaderVolume : RarVolume { - internal RarReaderVolume(Stream stream, ReaderOptions options, int index = 0) + internal RarReaderVolume(Stream stream, ReaderOptions options, int index) : base(StreamingMode.Streaming, stream, options, index) { } internal override RarFilePart CreateFilePart(MarkHeader markHeader, FileHeader fileHeader) => new NonSeekableStreamFilePart(markHeader, fileHeader, Index); internal override IEnumerable ReadFileParts() => GetVolumeFileParts(); + + internal override IAsyncEnumerable ReadFilePartsAsync() => + GetVolumeFilePartsAsync(); } diff --git a/src/SharpCompress/Readers/Rar/SingleVolumeRarReader.cs b/src/SharpCompress/Readers/Rar/SingleVolumeRarReader.cs index 768aa115..165c92f7 100644 --- a/src/SharpCompress/Readers/Rar/SingleVolumeRarReader.cs +++ b/src/SharpCompress/Readers/Rar/SingleVolumeRarReader.cs @@ -11,7 +11,7 @@ internal class SingleVolumeRarReader : RarReader internal SingleVolumeRarReader(Stream stream, ReaderOptions options) : base(options) => this.stream = stream; - internal override void ValidateArchive(RarVolume archive) + protected override void ValidateArchive(RarVolume archive) { if (archive.IsMultiVolume) { diff --git a/src/SharpCompress/Readers/ReaderFactory.Async.cs b/src/SharpCompress/Readers/ReaderFactory.Async.cs new file mode 100644 index 00000000..72bd7b09 --- /dev/null +++ b/src/SharpCompress/Readers/ReaderFactory.Async.cs @@ -0,0 +1,109 @@ +using System; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Factories; +using SharpCompress.IO; + +namespace SharpCompress.Readers; + +public static partial class ReaderFactory +{ + /// + /// Opens a Reader from a filepath asynchronously + /// + /// + /// + /// + /// + public static ValueTask OpenAsyncReader( + string filePath, + ReaderOptions? options = null, + CancellationToken cancellationToken = default + ) + { + filePath.NotNullOrEmpty(nameof(filePath)); + return OpenAsyncReader( + new FileInfo(filePath), + options ?? ReaderOptions.ForFilePath, + cancellationToken + ); + } + + /// + /// Opens a Reader from a FileInfo asynchronously + /// + /// + /// + /// + /// + public static async ValueTask OpenAsyncReader( + FileInfo fileInfo, + ReaderOptions? options = null, + CancellationToken cancellationToken = default + ) + { + options ??= ReaderOptions.ForFilePath; + var stream = fileInfo.OpenAsyncReadStream(cancellationToken); + return await OpenAsyncReader(stream, options, cancellationToken).ConfigureAwait(false); + } + + public static async ValueTask OpenAsyncReader( + Stream stream, + ReaderOptions? options = null, + CancellationToken cancellationToken = default + ) + { + stream.RequireReadable(); + options ??= ReaderOptions.ForExternalStream; + + var sharpCompressStream = SharpCompressStream.Create( + stream, + bufferSize: options.RewindableBufferSize + ); + sharpCompressStream.StartRecording(); + + var factories = Factory.Factories.OfType(); + + Factory? testedFactory = null; + if (!string.IsNullOrWhiteSpace(options.ExtensionHint)) + { + testedFactory = factories.FirstOrDefault(a => + a.GetSupportedExtensions() + .Contains(options.ExtensionHint, StringComparer.CurrentCultureIgnoreCase) + ); + if (testedFactory is not null) + { + var reader = await testedFactory + .TryOpenReaderAsync(sharpCompressStream, options, cancellationToken) + .ConfigureAwait(false); + if (reader is not null) + { + return reader; + } + } + sharpCompressStream.Rewind(); + } + + foreach (var factory in factories) + { + if (testedFactory == factory) + { + continue; // Already tested above + } + var reader = await factory + .TryOpenReaderAsync(sharpCompressStream, options, cancellationToken) + .ConfigureAwait(false); + if (reader is not null) + { + return reader; + } + } + + throw new InvalidFormatException( + "Cannot determine compressed stream type. Supported Reader Formats: Arc, Arj, Zip, GZip, BZip2, Tar, Rar, LZip, XZ, ZStandard" + ); + } +} diff --git a/src/SharpCompress/Readers/ReaderFactory.cs b/src/SharpCompress/Readers/ReaderFactory.cs index a079c235..72c4dd71 100644 --- a/src/SharpCompress/Readers/ReaderFactory.cs +++ b/src/SharpCompress/Readers/ReaderFactory.cs @@ -1,36 +1,81 @@ using System; using System.IO; using System.Linq; +using SharpCompress.Common; +using SharpCompress.Factories; using SharpCompress.IO; namespace SharpCompress.Readers; -public static class ReaderFactory +public static partial class ReaderFactory { + public static IReader OpenReader(string filePath, ReaderOptions? options = null) + { + filePath.NotNullOrEmpty(nameof(filePath)); + return OpenReader(new FileInfo(filePath), options ?? ReaderOptions.ForFilePath); + } + + public static IReader OpenReader(FileInfo fileInfo, ReaderOptions? options = null) + { + options ??= ReaderOptions.ForFilePath; + return OpenReader(fileInfo.OpenRead(), options); + } + /// /// Opens a Reader for Non-seeking usage /// /// /// /// - public static IReader Open(Stream stream, ReaderOptions? options = null) + public static IReader OpenReader(Stream stream, ReaderOptions? options = null) { - stream.CheckNotNull(nameof(stream)); - options ??= new ReaderOptions() { LeaveStreamOpen = false }; + stream.RequireReadable(); + options ??= ReaderOptions.ForExternalStream; - var rewindableStream = new RewindableStream(stream); - rewindableStream.StartRecording(); + var sharpCompressStream = SharpCompressStream.Create( + stream, + bufferSize: options.RewindableBufferSize + ); + sharpCompressStream.StartRecording(); - foreach (var factory in Factories.Factory.Factories.OfType()) + var factories = Factories.Factory.Factories.OfType(); + + Factory? testedFactory = null; + + if (!string.IsNullOrWhiteSpace(options.ExtensionHint)) { - if (factory.TryOpenReader(rewindableStream, options, out var reader) && reader != null) + testedFactory = factories.FirstOrDefault(a => + a.GetSupportedExtensions() + .Contains(options.ExtensionHint, StringComparer.CurrentCultureIgnoreCase) + ); + if ( + testedFactory?.TryOpenReader(sharpCompressStream, options, out var reader) == true + && reader != null + ) + { + sharpCompressStream.Rewind(true); + return reader; + } + } + + foreach (var factory in factories) + { + if (testedFactory == factory) + { + continue; // Already tested above + } + sharpCompressStream.Rewind(); + if ( + factory.TryOpenReader(sharpCompressStream, options, out var reader) + && reader != null + ) { return reader; } } - throw new InvalidOperationException( - "Cannot determine compressed stream type. Supported Reader Formats: Zip, GZip, BZip2, Tar, Rar, LZip, XZ" + throw new InvalidFormatException( + "Cannot determine compressed stream type. Supported Reader Formats: Ace, Arc, Arj, Zip, GZip, BZip2, Tar, Rar, LZip, Lzw, XZ, ZStandard" ); } } diff --git a/src/SharpCompress/Readers/ReaderOptions.cs b/src/SharpCompress/Readers/ReaderOptions.cs index 706829cb..aced7169 100644 --- a/src/SharpCompress/Readers/ReaderOptions.cs +++ b/src/SharpCompress/Readers/ReaderOptions.cs @@ -1,15 +1,188 @@ +using System; using SharpCompress.Common; +using SharpCompress.Common.Options; +using SharpCompress.Compressors; +using SharpCompress.Providers; namespace SharpCompress.Readers; -public class ReaderOptions : OptionsBase +/// +/// Options for configuring reader behavior when opening archives. +/// +/// +/// Use preset properties, setters, and fluent helpers for common configurations: +/// +/// var options = ReaderOptions.ForExternalStream +/// .WithPassword("secret") +/// .WithLookForHeader(true); +/// +/// Or use object initializers for simple cases: +/// +/// var options = new ReaderOptions { Password = "secret", LeaveStreamOpen = false }; +/// +/// +public sealed record ReaderOptions : IReaderOptions { + /// + /// Whether SharpCompress leaves the supplied streams open when the reader/archive is disposed. + /// As of v0.21, the library is documented to close streams by default; this option now defaults to false. + /// Set to true when passing caller-owned streams that should not be disposed. + /// + /// + /// + /// Default behavior (LeaveStreamOpen = false): + /// When you open an archive from a file path (e.g., GZipArchive.OpenArchive(filePath)), + /// SharpCompress manages the stream lifetime and closes it on Dispose. + /// + /// + /// Caller-provided streams (LeaveStreamOpen = true): + /// When you pass a stream you created (FileStream, MemoryStream, NetworkStream, etc.), + /// set LeaveStreamOpen = true to prevent SharpCompress from disposing it. + /// Use preset for convenience. + /// + /// + /// Example: + /// + /// // File-based: stream managed by library + /// using var archive = GZipArchive.OpenArchive(filePath); // LeaveStreamOpen = false + /// + /// // Caller-provided stream: caller manages lifetime + /// using var stream = File.OpenRead(filePath); + /// var options = new ReaderOptions { LeaveStreamOpen = true }; + /// using var archive = GZipArchive.OpenArchive(stream, options); + /// + /// + /// + public bool LeaveStreamOpen { get; set; } = false; + + /// + /// Encoding to use for archive entry names. + /// + public IArchiveEncoding ArchiveEncoding { get; set; } = new ArchiveEncoding(); + /// /// Look for RarArchive (Check for self-extracting archives or cases where RarArchive isn't at the start of the file) /// public bool LookForHeader { get; set; } + /// + /// Password for encrypted archives. + /// public string? Password { get; set; } + /// + /// Disable checking for incomplete archives. + /// public bool DisableCheckIncomplete { get; set; } + + /// + /// Buffer size for stream operations. + /// + public int BufferSize { get; set; } = Constants.BufferSize; + + /// + /// Provide a hint for the extension of the archive being read, can speed up finding the correct decoder. Should be without the leading period in the form like: tar.gz or zip + /// + public string? ExtensionHint { get; set; } + + /// + /// An optional progress reporter for tracking extraction operations. + /// When set, progress updates will be reported as entries are extracted. + /// + public IProgress? Progress { get; set; } + + /// + /// Size of the rewindable buffer for non-seekable streams. + /// Used during format detection to enable multiple rewinds. + /// + /// + /// + /// When opening archives from non-seekable streams (network streams, pipes, + /// compressed streams), SharpCompress uses a ring buffer to enable format + /// auto-detection. This buffer allows the library to try multiple decoders + /// by rewinding and re-reading the same data. + /// + /// + /// Default: Constants.RewindableBufferSize (163840 bytes / 160KB) - sized to cover + /// ZStandard's worst-case first block on a tar archive (~131KB including header overhead). + /// + /// + /// Typical usage: 500-1000 bytes for most archives + /// + /// + /// Increase if: + /// + /// Opening self-extracting RAR archives (may need 512KB+) + /// Format detection fails with "recording anchor" errors + /// Using custom formats with large headers + /// + /// + /// + /// Memory impact: Buffer is allocated for non-seekable streams only. + /// Seekable streams (FileStream, MemoryStream) use zero-copy seeking instead. + /// + /// + /// + /// + /// // For self-extracting archives, use larger buffer + /// var options = new ReaderOptions + /// { + /// RewindableBufferSize = 1_048_576, // 1MB + /// LookForHeader = true + /// }; + /// using var reader = ReaderFactory.OpenReader(networkStream, options); + /// + /// + public int? RewindableBufferSize { get; set; } + + /// + /// Registry of compression providers. + /// Defaults to but can be replaced with custom implementations, such as + /// System.IO.Compression for Deflate/GZip on modern .NET. + /// + public CompressionProviderRegistry Providers { get; set; } = + CompressionProviderRegistry.Default; + + /// + /// Creates a new ReaderOptions instance with default values. + /// + public ReaderOptions() { } + + /// + /// Gets ReaderOptions configured for caller-provided streams. + /// + internal static ReaderOptions Default => new(); + + public static ReaderOptions ForExternalStream => Default.WithLeaveStreamOpen(true); + + /// + /// Gets ReaderOptions configured for file-based overloads that open their own stream. + /// + public static ReaderOptions ForFilePath => Default; + + /// + /// Creates ReaderOptions for reading encrypted archives. + /// + /// The password for encrypted archives. + public static ReaderOptions ForEncryptedArchive(string? password = null) => + Default.WithPassword(password); + + /// + /// Creates ReaderOptions for archives with custom character encoding. + /// + /// The encoding for archive entry names. + public static ReaderOptions ForEncoding(IArchiveEncoding encoding) => + Default.WithArchiveEncoding(encoding); + + /// + /// Creates ReaderOptions for self-extracting archives that require header search. + /// + public static ReaderOptions ForSelfExtractingArchive(string? password = null) => + Default.WithLookForHeader(true).WithPassword(password).WithRewindableBufferSize(1_048_576); // 1MB for SFX archives + + // Note: Parameterized constructors have been removed. + // Use fluent With*() helpers or object initializers instead: + // new ReaderOptions().WithPassword("secret").WithLookForHeader(true) + // or + // new ReaderOptions { Password = "secret", LookForHeader = true } } diff --git a/src/SharpCompress/Readers/ReaderOptionsExtensions.cs b/src/SharpCompress/Readers/ReaderOptionsExtensions.cs new file mode 100644 index 00000000..2415b544 --- /dev/null +++ b/src/SharpCompress/Readers/ReaderOptionsExtensions.cs @@ -0,0 +1,101 @@ +using System; +using SharpCompress.Common; +using SharpCompress.Common.Options; +using SharpCompress.Compressors; +using SharpCompress.Providers; + +namespace SharpCompress.Readers; + +/// +/// Extension methods for fluent configuration of reader options. +/// +public static class ReaderOptionsExtensions +{ + /// + /// Creates a copy with the specified LeaveStreamOpen value. + /// + public static ReaderOptions WithLeaveStreamOpen( + this ReaderOptions options, + bool leaveStreamOpen + ) => options with { LeaveStreamOpen = leaveStreamOpen }; + + /// + /// Creates a copy with the specified password. + /// + public static ReaderOptions WithPassword(this ReaderOptions options, string? password) => + options with + { + Password = password, + }; + + /// + /// Creates a copy with the specified archive encoding. + /// + public static ReaderOptions WithArchiveEncoding( + this ReaderOptions options, + IArchiveEncoding encoding + ) => options with { ArchiveEncoding = encoding }; + + /// + /// Creates a copy with the specified LookForHeader value. + /// + public static ReaderOptions WithLookForHeader(this ReaderOptions options, bool lookForHeader) => + options with + { + LookForHeader = lookForHeader, + }; + + /// + /// Creates a copy with the specified DisableCheckIncomplete value. + /// + public static ReaderOptions WithDisableCheckIncomplete( + this ReaderOptions options, + bool disableCheckIncomplete + ) => options with { DisableCheckIncomplete = disableCheckIncomplete }; + + /// + /// Creates a copy with the specified buffer size. + /// + public static ReaderOptions WithBufferSize(this ReaderOptions options, int bufferSize) => + options with + { + BufferSize = bufferSize, + }; + + /// + /// Creates a copy with the specified extension hint. + /// + public static ReaderOptions WithExtensionHint( + this ReaderOptions options, + string? extensionHint + ) => options with { ExtensionHint = extensionHint }; + + /// + /// Creates a copy with the specified progress reporter. + /// + public static ReaderOptions WithProgress( + this ReaderOptions options, + IProgress? progress + ) => options with { Progress = progress }; + + /// + /// Creates a copy with the specified rewindable buffer size. + /// + public static ReaderOptions WithRewindableBufferSize( + this ReaderOptions options, + int? rewindableBufferSize + ) => options with { RewindableBufferSize = rewindableBufferSize }; + + /// + /// Creates a copy with the specified compression provider registry. + /// + /// Thrown if is null. + public static ReaderOptions WithProviders( + this ReaderOptions options, + CompressionProviderRegistry providers + ) + { + _ = providers ?? throw new ArgumentNullException(nameof(providers)); + return options with { Providers = providers }; + } +} diff --git a/src/SharpCompress/Readers/ReaderProgress.cs b/src/SharpCompress/Readers/ReaderProgress.cs deleted file mode 100644 index 2cffab9a..00000000 --- a/src/SharpCompress/Readers/ReaderProgress.cs +++ /dev/null @@ -1,21 +0,0 @@ -using System; -using SharpCompress.Common; - -namespace SharpCompress.Readers; - -public class ReaderProgress -{ - private readonly IEntry _entry; - public long BytesTransferred { get; } - public int Iterations { get; } - - public int PercentageRead => (int)Math.Round(PercentageReadExact); - public double PercentageReadExact => (float)BytesTransferred / _entry.Size * 100; - - public ReaderProgress(IEntry entry, long bytesTransferred, int iterations) - { - _entry = entry; - BytesTransferred = bytesTransferred; - Iterations = iterations; - } -} diff --git a/src/SharpCompress/Readers/Tar/TarReader.Async.cs b/src/SharpCompress/Readers/Tar/TarReader.Async.cs new file mode 100644 index 00000000..45a9c85b --- /dev/null +++ b/src/SharpCompress/Readers/Tar/TarReader.Async.cs @@ -0,0 +1,22 @@ +using System.Collections.Generic; +using System.IO; +using SharpCompress.Common.Tar; +using SharpCompress.IO; + +namespace SharpCompress.Readers.Tar; + +public partial class TarReader +{ + /// + /// Returns entries asynchronously for streams that only support async reads. + /// Uses async decompression for compressed tar archives (gzip, bzip2, zstandard, etc.). + /// + protected override IAsyncEnumerable GetEntriesAsync(Stream stream) => + TarEntry.GetEntriesAsync( + StreamingMode.Streaming, + stream, + compressionType, + Options.ArchiveEncoding, + Options + ); +} diff --git a/src/SharpCompress/Readers/Tar/TarReader.Factory.cs b/src/SharpCompress/Readers/Tar/TarReader.Factory.cs new file mode 100644 index 00000000..aa13c779 --- /dev/null +++ b/src/SharpCompress/Readers/Tar/TarReader.Factory.cs @@ -0,0 +1,213 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Archives.Tar; +using SharpCompress.Common; +using SharpCompress.Factories; +using SharpCompress.IO; +using SharpCompress.Providers; + +namespace SharpCompress.Readers.Tar; + +public partial class TarReader +#if NET8_0_OR_GREATER + : IReaderOpenable +#endif +{ + private static Stream CreateProbeDecompressionStream( + Stream stream, + CompressionType compressionType, + CompressionProviderRegistry providers, + ReaderOptions options + ) + { + var nonDisposingStream = SharpCompressStream.CreateNonDisposing(stream); + if (compressionType == CompressionType.None) + { + return nonDisposingStream; + } + + if (compressionType == CompressionType.GZip) + { + return providers.CreateDecompressStream( + compressionType, + nonDisposingStream, + CompressionContext.FromStream(nonDisposingStream).WithReaderOptions(options) + ); + } + + return providers.CreateDecompressStream(compressionType, nonDisposingStream); + } + + private static async ValueTask CreateProbeDecompressionStreamAsync( + Stream stream, + CompressionType compressionType, + CompressionProviderRegistry providers, + ReaderOptions options, + CancellationToken cancellationToken = default + ) + { + var nonDisposingStream = SharpCompressStream.CreateNonDisposing(stream); + if (compressionType == CompressionType.None) + { + return nonDisposingStream; + } + + if (compressionType == CompressionType.GZip) + { + return await providers + .CreateDecompressStreamAsync( + compressionType, + nonDisposingStream, + CompressionContext.FromStream(nonDisposingStream).WithReaderOptions(options), + cancellationToken + ) + .ConfigureAwait(false); + } + + return await providers + .CreateDecompressStreamAsync(compressionType, nonDisposingStream, cancellationToken) + .ConfigureAwait(false); + } + + public static ValueTask OpenAsyncReader( + string filePath, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + filePath.NotNullOrEmpty(nameof(filePath)); + return OpenAsyncReader(new FileInfo(filePath), readerOptions, cancellationToken); + } + + public static async ValueTask OpenAsyncReader( + Stream stream, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + stream.NotNull(nameof(stream)); + readerOptions ??= ReaderOptions.ForExternalStream; + var sharpCompressStream = SharpCompressStream.Create( + stream, + bufferSize: Math.Max( + readerOptions.RewindableBufferSize ?? 0, + TarWrapper.MaximumRewindBufferSize + ) + ); + long pos = sharpCompressStream.Position; + foreach (var wrapper in TarWrapper.Wrappers) + { + sharpCompressStream.Position = pos; + if ( + !await wrapper + .IsMatchAsync(sharpCompressStream, cancellationToken) + .ConfigureAwait(false) + ) + { + continue; + } + + sharpCompressStream.Position = pos; + var testStream = await CreateProbeDecompressionStreamAsync( + sharpCompressStream, + wrapper.CompressionType, + readerOptions.Providers, + readerOptions, + cancellationToken + ) + .ConfigureAwait(false); + if ( + await TarArchive.IsTarFileAsync(testStream, cancellationToken).ConfigureAwait(false) + ) + { + sharpCompressStream.Position = pos; + return new TarReader(sharpCompressStream, readerOptions, wrapper.CompressionType); + } + + if (wrapper.CompressionType != CompressionType.None) + { + throw new InvalidFormatException("Not a tar file."); + } + } + + sharpCompressStream.Position = pos; + return new TarReader(sharpCompressStream, readerOptions, CompressionType.None); + } + + public static async ValueTask OpenAsyncReader( + FileInfo fileInfo, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + readerOptions ??= ReaderOptions.ForFilePath; + var stream = fileInfo.OpenAsyncReadStream(cancellationToken); + return await OpenAsyncReader(stream, readerOptions, cancellationToken) + .ConfigureAwait(false); + } + + public static IReader OpenReader(string filePath, ReaderOptions? readerOptions = null) + { + filePath.NotNullOrEmpty(nameof(filePath)); + return OpenReader(new FileInfo(filePath), readerOptions); + } + + public static IReader OpenReader(FileInfo fileInfo, ReaderOptions? readerOptions = null) + { + fileInfo.NotNull(nameof(fileInfo)); + readerOptions ??= ReaderOptions.ForFilePath; + return OpenReader(fileInfo.OpenRead(), readerOptions); + } + + /// + /// Opens a TarReader for Non-seeking usage with a single volume + /// + /// + /// + /// + public static IReader OpenReader(Stream stream, ReaderOptions? readerOptions = null) + { + stream.RequireReadable(); + readerOptions ??= ReaderOptions.ForExternalStream; + var sharpCompressStream = SharpCompressStream.Create( + stream, + bufferSize: Math.Max( + readerOptions.RewindableBufferSize ?? 0, + TarWrapper.MaximumRewindBufferSize + ) + ); + long pos = sharpCompressStream.Position; + foreach (var wrapper in TarWrapper.Wrappers) + { + sharpCompressStream.Position = pos; + if (!wrapper.IsMatch(sharpCompressStream)) + { + continue; + } + + sharpCompressStream.Position = pos; + var testStream = CreateProbeDecompressionStream( + sharpCompressStream, + wrapper.CompressionType, + readerOptions.Providers, + readerOptions + ); + if (TarArchive.IsTarFile(testStream)) + { + sharpCompressStream.Position = pos; + return new TarReader(sharpCompressStream, readerOptions, wrapper.CompressionType); + } + + if (wrapper.CompressionType != CompressionType.None) + { + throw new InvalidFormatException("Not a tar file."); + } + } + + sharpCompressStream.Position = pos; + return new TarReader(sharpCompressStream, readerOptions, CompressionType.None); + } +} diff --git a/src/SharpCompress/Readers/Tar/TarReader.cs b/src/SharpCompress/Readers/Tar/TarReader.cs index c9802e46..f7c5abba 100644 --- a/src/SharpCompress/Readers/Tar/TarReader.cs +++ b/src/SharpCompress/Readers/Tar/TarReader.cs @@ -1,20 +1,23 @@ -using System; +using System; using System.Collections.Generic; using System.IO; -using SharpCompress.Archives.GZip; -using SharpCompress.Archives.Tar; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common; using SharpCompress.Common.Tar; using SharpCompress.Compressors; using SharpCompress.Compressors.BZip2; using SharpCompress.Compressors.Deflate; -using SharpCompress.IO; using SharpCompress.Compressors.LZMA; +using SharpCompress.Compressors.Lzw; using SharpCompress.Compressors.Xz; +using SharpCompress.Compressors.ZStandard; +using SharpCompress.IO; +using SharpCompress.Providers; namespace SharpCompress.Readers.Tar; -public class TarReader : AbstractReader +public partial class TarReader : AbstractReader { private readonly CompressionType compressionType; @@ -30,101 +33,83 @@ public class TarReader : AbstractReader protected override Stream RequestInitialStream() { var stream = base.RequestInitialStream(); - switch (compressionType) + + var providers = Options.Providers; + + return compressionType switch { - case CompressionType.BZip2: - { - return new BZip2Stream(stream, CompressionMode.Decompress, false); - } - case CompressionType.GZip: - { - return new GZipStream(stream, CompressionMode.Decompress); - } - case CompressionType.LZip: - { - return new LZipStream(stream, CompressionMode.Decompress); - } - case CompressionType.Xz: - { - return new XZStream(stream); - } - case CompressionType.None: - { - return stream; - } - default: - { - throw new NotSupportedException("Invalid compression type: " + compressionType); - } - } + CompressionType.BZip2 => providers.CreateDecompressStream( + CompressionType.BZip2, + stream + ), + CompressionType.GZip => providers.CreateDecompressStream( + CompressionType.GZip, + stream, + CompressionContext.FromStream(stream).WithReaderOptions(Options) + ), + CompressionType.ZStandard => providers.CreateDecompressStream( + CompressionType.ZStandard, + stream + ), + CompressionType.LZip => providers.CreateDecompressStream(CompressionType.LZip, stream), + CompressionType.Xz => providers.CreateDecompressStream(CompressionType.Xz, stream), + CompressionType.Lzw => providers.CreateDecompressStream(CompressionType.Lzw, stream), + CompressionType.None => stream, + _ => throw new NotSupportedException("Invalid compression type: " + compressionType), + }; } - #region Open - - /// - /// Opens a TarReader for Non-seeking usage with a single volume - /// - /// - /// - /// - public static TarReader Open(Stream stream, ReaderOptions? options = null) + protected override ValueTask RequestInitialStreamAsync( + CancellationToken cancellationToken = default + ) { - stream.CheckNotNull(nameof(stream)); - options = options ?? new ReaderOptions(); - RewindableStream rewindableStream = new RewindableStream(stream); - rewindableStream.StartRecording(); - if (GZipArchive.IsGZipFile(rewindableStream)) - { - rewindableStream.Rewind(false); - GZipStream testStream = new GZipStream(rewindableStream, CompressionMode.Decompress); - if (TarArchive.IsTarFile(testStream)) - { - rewindableStream.Rewind(true); - return new TarReader(rewindableStream, options, CompressionType.GZip); - } - throw new InvalidFormatException("Not a tar file."); - } + var stream = base.RequestInitialStream(); + var providers = Options.Providers; - rewindableStream.Rewind(false); - if (BZip2Stream.IsBZip2(rewindableStream)) + return compressionType switch { - rewindableStream.Rewind(false); - BZip2Stream testStream = new BZip2Stream( - rewindableStream, - CompressionMode.Decompress, - false - ); - if (TarArchive.IsTarFile(testStream)) - { - rewindableStream.Rewind(true); - return new TarReader(rewindableStream, options, CompressionType.BZip2); - } - throw new InvalidFormatException("Not a tar file."); - } - - rewindableStream.Rewind(false); - if (LZipStream.IsLZipFile(rewindableStream)) - { - rewindableStream.Rewind(false); - LZipStream testStream = new LZipStream(rewindableStream, CompressionMode.Decompress); - if (TarArchive.IsTarFile(testStream)) - { - rewindableStream.Rewind(true); - return new TarReader(rewindableStream, options, CompressionType.LZip); - } - throw new InvalidFormatException("Not a tar file."); - } - rewindableStream.Rewind(true); - return new TarReader(rewindableStream, options, CompressionType.None); + CompressionType.BZip2 => providers.CreateDecompressStreamAsync( + CompressionType.BZip2, + stream, + cancellationToken + ), + CompressionType.GZip => providers.CreateDecompressStreamAsync( + CompressionType.GZip, + stream, + CompressionContext.FromStream(stream).WithReaderOptions(Options), + cancellationToken + ), + CompressionType.ZStandard => providers.CreateDecompressStreamAsync( + CompressionType.ZStandard, + stream, + cancellationToken + ), + CompressionType.LZip => providers.CreateDecompressStreamAsync( + CompressionType.LZip, + stream, + cancellationToken + ), + CompressionType.Xz => providers.CreateDecompressStreamAsync( + CompressionType.Xz, + stream, + cancellationToken + ), + CompressionType.Lzw => providers.CreateDecompressStreamAsync( + CompressionType.Lzw, + stream, + cancellationToken + ), + CompressionType.None => new ValueTask(stream), + _ => throw new NotSupportedException("Invalid compression type: " + compressionType), + }; } - #endregion Open - protected override IEnumerable GetEntries(Stream stream) => TarEntry.GetEntries( StreamingMode.Streaming, stream, compressionType, - Options.ArchiveEncoding + Options.ArchiveEncoding, + Options ); } diff --git a/src/SharpCompress/Readers/Zip/ZipReader.Async.cs b/src/SharpCompress/Readers/Zip/ZipReader.Async.cs new file mode 100644 index 00000000..26ab7ac2 --- /dev/null +++ b/src/SharpCompress/Readers/Zip/ZipReader.Async.cs @@ -0,0 +1,122 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Options; +using SharpCompress.Common.Zip; +using SharpCompress.Common.Zip.Headers; +using SharpCompress.Compressors; + +namespace SharpCompress.Readers.Zip; + +public partial class ZipReader +{ + /// + /// Adapts an async header sequence into an async entry sequence. + /// + private sealed class ZipEntryAsyncEnumerable : IAsyncEnumerable + { + private readonly StreamingZipHeaderFactory _headerFactory; + private readonly Stream _stream; + private readonly IReaderOptions _options; + + public ZipEntryAsyncEnumerable( + StreamingZipHeaderFactory headerFactory, + Stream stream, + IReaderOptions options + ) + { + _headerFactory = headerFactory; + _stream = stream; + _options = options; + } + + public IAsyncEnumerator GetAsyncEnumerator( + CancellationToken cancellationToken = default + ) => new ZipEntryAsyncEnumerator(_headerFactory, _stream, _options, cancellationToken); + } + + /// + /// Yields entries from streaming ZIP headers without requiring synchronous stream reads. + /// + private sealed class ZipEntryAsyncEnumerator : IAsyncEnumerator, IDisposable + { + private readonly Stream _stream; + private readonly IAsyncEnumerator _headerEnumerator; + private readonly IReaderOptions _options; + private ZipEntry? _current; + + public ZipEntryAsyncEnumerator( + StreamingZipHeaderFactory headerFactory, + Stream stream, + IReaderOptions options, + CancellationToken cancellationToken + ) + { + _stream = stream; + _options = options; + _headerEnumerator = headerFactory + .ReadStreamHeaderAsync(stream) + .GetAsyncEnumerator(cancellationToken); + } + + public ZipEntry Current => + _current ?? throw new ArchiveOperationException("No current entry is available."); + + /// + /// Advances to the next non-directory entry-relevant header and materializes a , + /// using async I/O for improved performance on non-seekable streams. + /// + public async ValueTask MoveNextAsync() + { + while (await _headerEnumerator.MoveNextAsync().ConfigureAwait(false)) + { + var header = _headerEnumerator.Current; + switch (header.ZipHeaderType) + { + case ZipHeaderType.LocalEntry: + _current = new ZipEntry( + new StreamingZipFilePart( + (LocalEntryHeader)header, + _stream, + _options.Providers + ), + _options + ); + return true; + case ZipHeaderType.DirectoryEntry: + // DirectoryEntry headers are intentionally skipped in streaming mode. + break; + case ZipHeaderType.DirectoryEnd: + _current = null; + return false; + } + } + + _current = null; + return false; + } + + /// + /// Disposes the underlying header enumerator asynchronously. + /// + public ValueTask DisposeAsync() + { + Dispose(); + return default; + } + + /// + /// Synchronously disposes the underlying header enumerator. + /// + public void Dispose() + { + if (_headerEnumerator is IDisposable disposable) + { + disposable.Dispose(); + } + } + } +} diff --git a/src/SharpCompress/Readers/Zip/ZipReader.Factory.cs b/src/SharpCompress/Readers/Zip/ZipReader.Factory.cs new file mode 100644 index 00000000..7058937d --- /dev/null +++ b/src/SharpCompress/Readers/Zip/ZipReader.Factory.cs @@ -0,0 +1,55 @@ +#if NET8_0_OR_GREATER +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Readers.Zip; + +public partial class ZipReader : IReaderOpenable +{ + public static ValueTask OpenAsyncReader( + string filePath, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + filePath.NotNullOrEmpty(nameof(filePath)); + return new((IAsyncReader)OpenReader(new FileInfo(filePath), readerOptions)); + } + + public static ValueTask OpenAsyncReader( + Stream stream, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncReader)OpenReader(stream, readerOptions)); + } + + public static ValueTask OpenAsyncReader( + FileInfo fileInfo, + ReaderOptions? readerOptions = null, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncReader)OpenReader(fileInfo, readerOptions)); + } + + public static IReader OpenReader(string filePath, ReaderOptions? readerOptions = null) + { + filePath.NotNullOrEmpty(nameof(filePath)); + return OpenReader(new FileInfo(filePath), readerOptions); + } + + public static IReader OpenReader(FileInfo fileInfo, ReaderOptions? readerOptions = null) + { + fileInfo.NotNull(nameof(fileInfo)); + readerOptions ??= ReaderOptions.ForFilePath; + return OpenReader(fileInfo.OpenRead(), readerOptions); + } +} +#endif diff --git a/src/SharpCompress/Readers/Zip/ZipReader.cs b/src/SharpCompress/Readers/Zip/ZipReader.cs index b00690f0..a9ca5c50 100644 --- a/src/SharpCompress/Readers/Zip/ZipReader.cs +++ b/src/SharpCompress/Readers/Zip/ZipReader.cs @@ -1,12 +1,15 @@ +using System; using System.Collections.Generic; using System.IO; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common; using SharpCompress.Common.Zip; using SharpCompress.Common.Zip.Headers; namespace SharpCompress.Readers.Zip; -public class ZipReader : AbstractReader +public partial class ZipReader : AbstractReader { private readonly StreamingZipHeaderFactory _headerFactory; @@ -40,22 +43,22 @@ public class ZipReader : AbstractReader /// Opens a ZipReader for Non-seeking usage with a single volume /// /// - /// + /// /// - public static ZipReader Open(Stream stream, ReaderOptions? options = null) + public static IReader OpenReader(Stream stream, ReaderOptions? readerOptions = null) { - stream.CheckNotNull(nameof(stream)); - return new ZipReader(stream, options ?? new ReaderOptions()); + stream.RequireReadable(); + return new ZipReader(stream, readerOptions ?? ReaderOptions.ForExternalStream); } - public static ZipReader Open( + public static IReader OpenReader( Stream stream, ReaderOptions? options, IEnumerable entries ) { - stream.CheckNotNull(nameof(stream)); - return new ZipReader(stream, options ?? new ReaderOptions(), entries); + stream.RequireReadable(); + return new ZipReader(stream, options ?? ReaderOptions.ForExternalStream, entries); } #endregion Open @@ -69,13 +72,25 @@ public class ZipReader : AbstractReader switch (h.ZipHeaderType) { case ZipHeaderType.LocalEntry: - { yield return new ZipEntry( - new StreamingZipFilePart((LocalEntryHeader)h, stream) + new StreamingZipFilePart( + (LocalEntryHeader)h, + stream, + Options.Providers + ), + Options ); } break; + case ZipHeaderType.DirectoryEntry: + // DirectoryEntry headers in the central directory are intentionally skipped. + // In streaming mode, we can only read forward, and DirectoryEntry headers + // reference LocalEntry headers that have already been processed. The file + // data comes from LocalEntry headers, not DirectoryEntry headers. + // For multi-volume ZIPs where file data spans multiple files, use ZipArchive + // instead, which requires seekable streams. + break; case ZipHeaderType.DirectoryEnd: { yield break; @@ -84,4 +99,12 @@ public class ZipReader : AbstractReader } } } + + /// + /// Returns entries asynchronously for streams that only support async reads. + /// + protected override IAsyncEnumerable GetEntriesAsync(Stream stream) => + new ZipEntryAsyncEnumerable(_headerFactory, stream, Options); + + // Async nested classes moved to ZipReader.Async.cs } diff --git a/src/SharpCompress/SharpCompress.csproj b/src/SharpCompress/SharpCompress.csproj index efdd0a47..991eccb9 100644 --- a/src/SharpCompress/SharpCompress.csproj +++ b/src/SharpCompress/SharpCompress.csproj @@ -1,44 +1,45 @@ - - - SharpCompress - Pure C# Decompression/Compression - en-US - 0.34.1 - 0.34.1 - 0.34.1 - Adam Hathcock - net462;netstandard2.0;netstandard2.1;net6.0;net7.0 - SharpCompress - ../../SharpCompress.snk - False - SharpCompress - rar;unrar;zip;unzip;bzip2;gzip;tar;7zip;lzip;xz - https://github.com/adamhathcock/sharpcompress - MIT - Copyright (c) 2014 Adam Hathcock - false - false - SharpCompress is a compression library for NET Standard 2.0/2.1/NET 6.0/NET 7.0 that can unrar, decompress 7zip, decompress xz, zip/unzip, tar/untar lzip/unlzip, bzip2/unbzip2 and gzip/ungzip with forward-only reading and file random access APIs. Write support for zip/tar/bzip2/gzip is implemented. - true - true - snupkg - true - latest - True - - - - - - - - - - - - - - - - - + + + SharpCompress - Pure C# Decompression/Compression + en-US + 0.0.0 + 0.0.0.0 + 0.0.0.0 + Adam Hathcock + net48;netstandard2.0;netstandard2.1;net6.0;net8.0;net10.0 + SharpCompress + ../../SharpCompress.snk + true + SharpCompress + rar;unrar;zip;unzip;bzip2;gzip;tar;7zip;lzip;xz + https://github.com/adamhathcock/sharpcompress + MIT + Copyright (c) 2025 Adam Hathcock + false + false + SharpCompress is a compression library for NET 4.8/NET Standard 2.0/NET Standard 2.1/NET 5.0/NET 6.0/NET 7.0/NET 8.0/NET 9.0/NET 10.0 that can unrar, decompress 7zip, decompress xz, zip/unzip, tar/untar lzip/unlzip, bzip2/unbzip2 and gzip/ungzip with forward-only reading and file random access APIs. Write support for zip/tar/bzip2/gzip/7zip is implemented. + true + true + embedded + latest + true + README.md + true + true + $(AllowedOutputExtensionsInPackageBuildOutputFolder);.pdb + + + $(DefineConstants);LEGACY_DOTNET + + + true + true + + + + + + + + diff --git a/src/SharpCompress/StreamValidationExtensions.cs b/src/SharpCompress/StreamValidationExtensions.cs new file mode 100644 index 00000000..28bd9116 --- /dev/null +++ b/src/SharpCompress/StreamValidationExtensions.cs @@ -0,0 +1,56 @@ +using System; +using System.Collections.Generic; +using System.IO; + +namespace SharpCompress; + +internal static class StreamValidationExtensions +{ + internal static void RequireReadable(this Stream stream) + { + stream.NotNull(nameof(stream)); + + if (!stream.CanRead) + { + throw new ArgumentException("Stream must be readable", nameof(stream)); + } + } + + internal static void RequireSeekable(this Stream stream) + { + stream.NotNull(nameof(stream)); + + if (!stream.CanSeek) + { + throw new ArgumentException("Stream must be seekable", nameof(stream)); + } + } + + internal static void RequireWritable(this Stream stream) + { + stream.NotNull(nameof(stream)); + + if (!stream.CanWrite) + { + throw new ArgumentException("Stream must be writable", nameof(stream)); + } + } + + internal static IEnumerable RequireSeekable(this IEnumerable streams) + { + foreach (var stream in streams) + { + stream.RequireSeekable(); + yield return stream; + } + } + + internal static IEnumerable RequireReadable(this IEnumerable streams) + { + foreach (var stream in streams) + { + stream.RequireReadable(); + yield return stream; + } + } +} diff --git a/src/SharpCompress/ThrowHelper.cs b/src/SharpCompress/ThrowHelper.cs new file mode 100644 index 00000000..c202d87c --- /dev/null +++ b/src/SharpCompress/ThrowHelper.cs @@ -0,0 +1,80 @@ +using System; +using System.Diagnostics.CodeAnalysis; +using System.Runtime.CompilerServices; + +namespace SharpCompress; + +internal static class ThrowHelper +{ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void ThrowIfNull([NotNull] object? argument, string? paramName = null) + { + if (argument is null) + { + throw new ArgumentNullException(paramName); + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void ThrowIfNegative(int value, string? paramName = null) + { + if (value < 0) + { + throw new ArgumentOutOfRangeException(paramName); + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void ThrowIfNegative(long value, string? paramName = null) + { + if (value < 0) + { + throw new ArgumentOutOfRangeException(paramName); + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void ThrowIfNegativeOrZero(int value, string? paramName = null) + { + if (value <= 0) + { + throw new ArgumentOutOfRangeException(paramName); + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void ThrowIfLessThan(int value, int other, string? paramName = null) + { + if (value < other) + { + throw new ArgumentOutOfRangeException(paramName); + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void ThrowIfGreaterThan(int value, int other, string? paramName = null) + { + if (value > other) + { + throw new ArgumentOutOfRangeException(paramName); + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void ThrowIfGreaterThan(long value, long other, string? paramName = null) + { + if (value > other) + { + throw new ArgumentOutOfRangeException(paramName); + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void ThrowIfGreaterThan(uint value, uint other, string? paramName = null) + { + if (value > other) + { + throw new ArgumentOutOfRangeException(paramName); + } + } +} diff --git a/src/SharpCompress/Utility.Async.cs b/src/SharpCompress/Utility.Async.cs new file mode 100644 index 00000000..df323d69 --- /dev/null +++ b/src/SharpCompress/Utility.Async.cs @@ -0,0 +1,249 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress; + +internal static partial class Utility +{ + extension(Stream source) + { + /// + /// Read exactly the requested number of bytes from a stream asynchronously. Throws EndOfStreamException if not enough data is available. + /// + public async ValueTask ReadExactAsync( + byte[] buffer, + int offset, + int length, + CancellationToken cancellationToken = default + ) + { +#if LEGACY_DOTNET + if (source is null) + { + throw new ArgumentNullException(); + } +#else + ThrowHelper.ThrowIfNull(source); +#endif + + ThrowHelper.ThrowIfNull(buffer); + + if (offset < 0 || offset > buffer.Length) + { + throw new ArgumentOutOfRangeException(nameof(offset)); + } + + if (length < 0 || length > buffer.Length - offset) + { + throw new ArgumentOutOfRangeException(nameof(length)); + } + + while (length > 0) + { +#if NETCOREAPP2_1_OR_GREATER || NETSTANDARD2_1_OR_GREATER + var fetched = await source + .ReadAsync(buffer.AsMemory(offset, length), cancellationToken) + .ConfigureAwait(false); +#else + var fetched = await source + .ReadAsync(buffer, offset, length, cancellationToken) + .ConfigureAwait(false); +#endif + if (fetched <= 0) + { + throw new IncompleteArchiveException("Unexpected end of stream."); + } + + offset += fetched; + length -= fetched; + } + } + + public async ValueTask TransferToAsync( + Stream destination, + long maxLength, + int? bufferSize = null, + CancellationToken cancellationToken = default + ) + { + // Use ReadOnlySubStream to limit reading and leverage framework's CopyToAsync + using var limitedStream = new IO.ReadOnlySubStream(source, maxLength); + await limitedStream + .CopyToAsync(destination, bufferSize ?? Constants.BufferSize, cancellationToken) + .ConfigureAwait(false); + return limitedStream.Position; + } + + public async ValueTask ReadFullyAsync( + byte[] buffer, + CancellationToken cancellationToken = default + ) + { + var total = 0; + int read; + while ( + ( +#if NETCOREAPP2_1_OR_GREATER || NETSTANDARD2_1_OR_GREATER + read = await source + .ReadAsync(buffer.AsMemory(total, buffer.Length - total), cancellationToken) + .ConfigureAwait(false) +#else + read = await source + .ReadAsync(buffer, total, buffer.Length - total, cancellationToken) + .ConfigureAwait(false) +#endif + ) > 0 + ) + { + total += read; + if (total >= buffer.Length) + { + return true; + } + } + return (total >= buffer.Length); + } + + public async ValueTask ReadFullyAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken = default + ) + { + var total = 0; + int read; + while ( + ( +#if NETCOREAPP2_1_OR_GREATER || NETSTANDARD2_1_OR_GREATER + read = await source + .ReadAsync( + buffer.AsMemory(offset + total, count - total), + cancellationToken + ) + .ConfigureAwait(false) +#else + read = await source + .ReadAsync(buffer, offset + total, count - total, cancellationToken) + .ConfigureAwait(false) +#endif + ) > 0 + ) + { + total += read; + if (total >= count) + { + return true; + } + } + return (total >= count); + } + } + + /// + /// Opens a file stream for asynchronous writing. + /// Uses File.OpenHandle with FileOptions.Asynchronous on .NET 8.0+ for optimal performance. + /// Falls back to FileStream constructor with async options on legacy frameworks. + /// + /// The file path to open. + /// Cancellation token. + /// A FileStream configured for asynchronous operations. + public static Stream OpenAsyncWriteStream(string path, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + +#if NET8_0_OR_GREATER + // Use File.OpenHandle with async options for .NET 8.0+ + var handle = File.OpenHandle( + path, + FileMode.Create, + FileAccess.Write, + FileShare.None, + FileOptions.Asynchronous + ); + return new FileStream(handle, FileAccess.Write); +#else + // For older target frameworks, use FileStream constructor with async options + return new FileStream( + path, + FileMode.Create, + FileAccess.Write, + FileShare.None, + bufferSize: 4096, //default + FileOptions.Asynchronous + ); +#endif + } + + /// + /// Opens a file stream for asynchronous writing from a FileInfo. + /// Uses File.OpenHandle with FileOptions.Asynchronous on .NET 8.0+ for optimal performance. + /// Falls back to FileStream constructor with async options on legacy frameworks. + /// + /// The FileInfo to open. + /// Cancellation token. + /// A FileStream configured for asynchronous operations. + public static Stream OpenAsyncWriteStream( + this FileInfo fileInfo, + CancellationToken cancellationToken + ) + { + fileInfo.NotNull(nameof(fileInfo)); + return OpenAsyncWriteStream(fileInfo.FullName, cancellationToken); + } + + /// + /// Opens a file stream for asynchronous reading. + /// Uses File.OpenHandle with FileOptions.Asynchronous on .NET 8.0+ for optimal performance. + /// Falls back to FileStream constructor with async options on legacy frameworks. + /// + /// The file path to open. + /// Cancellation token. + /// A FileStream configured for asynchronous operations. + public static Stream OpenAsyncReadStream(string path, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + +#if NET8_0_OR_GREATER + // Use File.OpenHandle with async options for .NET 8.0+ + var handle = File.OpenHandle( + path, + FileMode.Open, + FileAccess.Read, + FileShare.Read, + FileOptions.Asynchronous + ); + return new FileStream(handle, FileAccess.Read); +#else + // For older target frameworks, use FileStream constructor with async options + return new FileStream( + path, + FileMode.Open, + FileAccess.Read, + FileShare.Read, + bufferSize: 4096, + FileOptions.Asynchronous + ); +#endif + } + + /// + /// Opens a file stream for asynchronous reading from a FileInfo. + /// Uses File.OpenHandle with FileOptions.Asynchronous on .NET 8.0+ for optimal performance. + /// Falls back to FileStream constructor with async options on legacy frameworks. + /// + /// The FileInfo to open. + /// Cancellation token. + /// A FileStream configured for asynchronous operations. + public static Stream OpenAsyncReadStream( + this FileInfo fileInfo, + CancellationToken cancellationToken + ) + { + fileInfo.NotNull(nameof(fileInfo)); + return OpenAsyncReadStream(fileInfo.FullName, cancellationToken); + } +} diff --git a/src/SharpCompress/Utility.cs b/src/SharpCompress/Utility.cs index 9471c3e1..c03c3643 100644 --- a/src/SharpCompress/Utility.cs +++ b/src/SharpCompress/Utility.cs @@ -1,16 +1,40 @@ using System; using System.Buffers; using System.Collections.Generic; +using System.Collections.ObjectModel; using System.IO; -using SharpCompress.Readers; +using System.Runtime.InteropServices; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.IO; namespace SharpCompress; -[CLSCompliant(false)] -public static class Utility +internal static partial class Utility { - public static ReadOnlyCollection ToReadOnly(this ICollection items) => - new ReadOnlyCollection(items); + /// + /// Gets the appropriate StringComparison for path checks based on the file system. + /// Windows uses case-insensitive file systems, while Unix-like systems use case-sensitive file systems. + /// + internal static StringComparison PathComparison => + RuntimeInformation.IsOSPlatform(OSPlatform.Windows) + ? StringComparison.OrdinalIgnoreCase + : StringComparison.Ordinal; + + public static bool UseSyncOverAsyncDispose() + { + var useSyncOverAsync = false; +#if LEGACY_DOTNET + useSyncOverAsync = true; +#endif + return useSyncOverAsync; + } + + private static readonly HashSet invalidChars = new(Path.GetInvalidFileNameChars()); + + public static ReadOnlyCollection ToReadOnly(this IList items) => new(items); /// /// Performs an unsigned bitwise right shift with the specified number @@ -18,14 +42,7 @@ public static class Utility /// Number to operate on /// Amount of bits to shift /// The resulting number from the shift operation - public static int URShift(int number, int bits) - { - if (number >= 0) - { - return number >> bits; - } - return (number >> bits) + (2 << ~bits); - } + public static int URShift(int number, int bits) => (int)((uint)number >> bits); /// /// Performs an unsigned bitwise right shift with the specified number @@ -33,14 +50,7 @@ public static class Utility /// Number to operate on /// Amount of bits to shift /// The resulting number from the shift operation - public static long URShift(long number, int bits) - { - if (number >= 0) - { - return number >> bits; - } - return (number >> bits) + (2L << ~bits); - } + public static long URShift(long number, int bits) => (long)((ulong)number >> bits); public static void SetSize(this List list, int count) { @@ -67,143 +77,11 @@ public static class Utility } } - public static void Copy( - Array sourceArray, - long sourceIndex, - Array destinationArray, - long destinationIndex, - long length - ) - { - if (sourceIndex > int.MaxValue || sourceIndex < int.MinValue) - { - throw new ArgumentOutOfRangeException(nameof(sourceIndex)); - } - - if (destinationIndex > int.MaxValue || destinationIndex < int.MinValue) - { - throw new ArgumentOutOfRangeException(nameof(destinationIndex)); - } - - if (length > int.MaxValue || length < int.MinValue) - { - throw new ArgumentOutOfRangeException(nameof(length)); - } - - Array.Copy( - sourceArray, - (int)sourceIndex, - destinationArray, - (int)destinationIndex, - (int)length - ); - } - public static IEnumerable AsEnumerable(this T item) { yield return item; } - public static void CheckNotNull(this object obj, string name) - { - if (obj is null) - { - throw new ArgumentNullException(name); - } - } - - public static void CheckNotNullOrEmpty(this string obj, string name) - { - obj.CheckNotNull(name); - if (obj.Length == 0) - { - throw new ArgumentException("String is empty.", name); - } - } - - public static void Skip(this Stream source, long advanceAmount) - { - if (source.CanSeek) - { - source.Position += advanceAmount; - return; - } - - var buffer = GetTransferByteArray(); - try - { - var read = 0; - var readCount = 0; - do - { - readCount = buffer.Length; - if (readCount > advanceAmount) - { - readCount = (int)advanceAmount; - } - read = source.Read(buffer, 0, readCount); - if (read <= 0) - { - break; - } - advanceAmount -= read; - if (advanceAmount == 0) - { - break; - } - } while (true); - } - finally - { - ArrayPool.Shared.Return(buffer); - } - } - - public static void Skip(this Stream source) - { - var buffer = GetTransferByteArray(); - try - { - do { } while (source.Read(buffer, 0, buffer.Length) == buffer.Length); - } - finally - { - ArrayPool.Shared.Return(buffer); - } - } - - public static bool Find(this Stream source, byte[] array) - { - var buffer = GetTransferByteArray(); - try - { - var count = 0; - var len = source.Read(buffer, 0, buffer.Length); - - do - { - for (var i = 0; i < len; i++) - { - if (array[count] == buffer[i]) - { - count++; - if (count == array.Length) - { - source.Position = source.Position - len + i - array.Length + 1; - return true; - } - } - } - } while ((len = source.Read(buffer, 0, buffer.Length)) > 0); - } - finally - { - ArrayPool.Shared.Return(buffer); - } - - return false; - } - public static DateTime DosDateToDateTime(ushort iDate, ushort iTime) { var year = (iDate / 512) + 1980; @@ -270,85 +148,148 @@ public static class Utility return sTime.AddSeconds(unixtime); } - public static long TransferTo(this Stream source, Stream destination) + extension(Stream source) { - var array = GetTransferByteArray(); - try + public long TransferTo(Stream destination, long maxLength, int? bufferSize) { - long total = 0; - while (ReadTransferBlock(source, array, out var count)) + // Use ReadOnlySubStream to limit reading and leverage framework's CopyTo + using var limitedStream = new IO.ReadOnlySubStream(source, maxLength); + limitedStream.CopyTo(destination, bufferSize ?? Constants.BufferSize); + return limitedStream.Position; + } + + public async ValueTask SkipAsync( + long advanceAmount, + CancellationToken cancellationToken = default + ) + { + if (source.CanSeek && source is not SharpCompressStream) { - total += count; - destination.Write(array, 0, count); + source.Position += advanceAmount; + return; } - return total; - } - finally - { - ArrayPool.Shared.Return(array); - } - } - public static long TransferTo( - this Stream source, - Stream destination, - Common.Entry entry, - IReaderExtractionListener readerExtractionListener - ) - { - var array = GetTransferByteArray(); - try - { - var iterations = 0; - long total = 0; - while (ReadTransferBlock(source, array, out var count)) + var array = ArrayPool.Shared.Rent(Constants.BufferSize); + try { - total += count; - destination.Write(array, 0, count); - iterations++; - readerExtractionListener.FireEntryExtractionProgress(entry, total, iterations); + while (advanceAmount > 0) + { + var toRead = (int)Math.Min(array.Length, advanceAmount); + var read = await source + .ReadAsync(array, 0, toRead, cancellationToken) + .ConfigureAwait(false); + if (read <= 0) + { + break; + } + + advanceAmount -= read; + } } - return total; - } - finally - { - ArrayPool.Shared.Return(array); - } - } - - private static bool ReadTransferBlock(Stream source, byte[] array, out int count) => - (count = source.Read(array, 0, array.Length)) != 0; - - private static byte[] GetTransferByteArray() => ArrayPool.Shared.Rent(81920); - - public static bool ReadFully(this Stream stream, byte[] buffer) - { - var total = 0; - int read; - while ((read = stream.Read(buffer, total, buffer.Length - total)) > 0) - { - total += read; - if (total >= buffer.Length) + finally { + ArrayPool.Shared.Return(array); + } + } + +#if NET8_0_OR_GREATER + public bool ReadFully(byte[] buffer) + { + try + { + source.ReadExactly(buffer); return true; } - } - return (total >= buffer.Length); - } - - public static bool ReadFully(this Stream stream, Span buffer) - { - var total = 0; - int read; - while ((read = stream.Read(buffer.Slice(total, buffer.Length - total))) > 0) - { - total += read; - if (total >= buffer.Length) + catch (EndOfStreamException) { - return true; + return false; + } + } + + public bool ReadFully(Span buffer) + { + try + { + source.ReadExactly(buffer); + return true; + } + catch (EndOfStreamException) + { + return false; + } + } +#else + public bool ReadFully(byte[] buffer) + { + var total = 0; + int read; + while ((read = source.Read(buffer, total, buffer.Length - total)) > 0) + { + total += read; + if (total >= buffer.Length) + { + return true; + } + } + + return (total >= buffer.Length); + } + + public bool ReadFully(Span buffer) + { + var total = 0; + int read; + while ((read = source.Read(buffer.Slice(total, buffer.Length - total))) > 0) + { + total += read; + if (total >= buffer.Length) + { + return true; + } + } + + return (total >= buffer.Length); + } +#endif + + /// + /// Read exactly the requested number of bytes from a stream. Throws EndOfStreamException if not enough data is available. + /// + public void ReadExact(byte[] buffer, int offset, int length) + { +#if LEGACY_DOTNET + if (source is null) + { + throw new ArgumentNullException(); + } +#else + ThrowHelper.ThrowIfNull(source); +#endif + + ThrowHelper.ThrowIfNull(buffer); + + if (offset < 0 || offset > buffer.Length) + { + throw new ArgumentOutOfRangeException(nameof(offset)); + } + + if (length < 0 || length > buffer.Length - offset) + { + throw new ArgumentOutOfRangeException(nameof(length)); + } + + while (length > 0) + { + var fetched = source.Read(buffer, offset, length); + if (fetched <= 0) + { + throw new IncompleteArchiveException("Unexpected end of stream."); + } + + offset += fetched; + length -= fetched; } } - return (total >= buffer.Length); } public static string TrimNulls(this string source) => source.Replace('\0', ' ').Trim(); @@ -391,4 +332,16 @@ public static class Utility buffer[offset + 2] = (byte)(number >> 8); buffer[offset + 3] = (byte)number; } + + public static string ReplaceInvalidFileNameChars(string fileName) + { + var sb = new StringBuilder(fileName.Length); + foreach (var c in fileName) + { + var newChar = invalidChars.Contains(c) ? '_' : c; + sb.Append(newChar); + } + + return sb.ToString(); + } } diff --git a/src/SharpCompress/Writers/AbstractWriter.Async.cs b/src/SharpCompress/Writers/AbstractWriter.Async.cs new file mode 100644 index 00000000..fa1f3ae1 --- /dev/null +++ b/src/SharpCompress/Writers/AbstractWriter.Async.cs @@ -0,0 +1,35 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Writers; + +public abstract partial class AbstractWriter +{ + public abstract ValueTask WriteAsync( + string filename, + Stream source, + DateTime? modificationTime, + CancellationToken cancellationToken = default + ); + + public abstract ValueTask WriteDirectoryAsync( + string directoryName, + DateTime? modificationTime, + CancellationToken cancellationToken = default + ); + + public virtual ValueTask DisposeAsync() + { + if (!_isDisposed) + { + GC.SuppressFinalize(this); + Dispose(true); + _isDisposed = true; + } + + return new(); + } +} diff --git a/src/SharpCompress/Writers/AbstractWriter.cs b/src/SharpCompress/Writers/AbstractWriter.cs index 209c49f7..b561e954 100644 --- a/src/SharpCompress/Writers/AbstractWriter.cs +++ b/src/SharpCompress/Writers/AbstractWriter.cs @@ -1,36 +1,61 @@ -#nullable disable - using System; using System.IO; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common; +using SharpCompress.Common.Options; +using SharpCompress.IO; namespace SharpCompress.Writers; -public abstract class AbstractWriter : IWriter +public abstract partial class AbstractWriter(ArchiveType type, IWriterOptions writerOptions) + : IWriter, + IAsyncWriter { - private bool _isDisposed; + protected bool _isDisposed; - protected AbstractWriter(ArchiveType type, WriterOptions writerOptions) + //always initializes the stream + + protected void InitializeStream(Stream stream) => OutputStream = stream; + + protected Stream? OutputStream { get; private set; } + + public ArchiveType Type { get; } = type; + + protected IWriterOptions WriterOptions { get; } = writerOptions; + + /// + /// Wraps the source stream with a progress-reporting stream if progress reporting is enabled. + /// + /// The source stream to wrap. + /// The path of the entry being written. + /// A stream that reports progress, or the original stream if progress is not enabled. + protected Stream WrapWithProgress(Stream source, string entryPath) { - WriterType = type; - WriterOptions = writerOptions; + if (WriterOptions.Progress is null) + { + return source; + } + + long? totalBytes = source.CanSeek ? source.Length : null; + return new ProgressReportingStream( + source, + WriterOptions.Progress, + entryPath, + totalBytes, + leaveOpen: true + ); } - protected void InitalizeStream(Stream stream) => OutputStream = stream; - - protected Stream OutputStream { get; private set; } - - public ArchiveType WriterType { get; } - - protected WriterOptions WriterOptions { get; } - public abstract void Write(string filename, Stream source, DateTime? modificationTime); + public abstract void WriteDirectory(string directoryName, DateTime? modificationTime); + protected virtual void Dispose(bool isDisposing) { if (isDisposing) { - OutputStream.Dispose(); + OutputStream?.Dispose(); } } diff --git a/src/SharpCompress/Writers/CompressionLevelValidation.cs b/src/SharpCompress/Writers/CompressionLevelValidation.cs new file mode 100644 index 00000000..ab964191 --- /dev/null +++ b/src/SharpCompress/Writers/CompressionLevelValidation.cs @@ -0,0 +1,49 @@ +using System; +using SharpCompress.Common; + +namespace SharpCompress.Writers; + +internal static class CompressionLevelValidation +{ + public static void Validate(CompressionType compressionType, int compressionLevel) + { + switch (compressionType) + { + case CompressionType.Deflate: + case CompressionType.Deflate64: + case CompressionType.GZip: + EnsureRange(compressionLevel, 0, 9, compressionType); + break; + case CompressionType.ZStandard: + EnsureRange(compressionLevel, 1, 22, compressionType); + break; + default: + if (compressionLevel != 0) + { + throw new ArgumentOutOfRangeException( + nameof(compressionLevel), + compressionLevel, + $"Compression type {compressionType} does not support configurable compression levels. Use 0." + ); + } + break; + } + } + + private static void EnsureRange( + int compressionLevel, + int minInclusive, + int maxInclusive, + CompressionType compressionType + ) + { + if (compressionLevel < minInclusive || compressionLevel > maxInclusive) + { + throw new ArgumentOutOfRangeException( + nameof(compressionLevel), + compressionLevel, + $"Compression level for {compressionType} must be between {minInclusive} and {maxInclusive}." + ); + } + } +} diff --git a/src/SharpCompress/Writers/GZip/GZipWriter.Async.cs b/src/SharpCompress/Writers/GZip/GZipWriter.Async.cs new file mode 100644 index 00000000..a1aa613a --- /dev/null +++ b/src/SharpCompress/Writers/GZip/GZipWriter.Async.cs @@ -0,0 +1,39 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Compressors.Deflate; + +namespace SharpCompress.Writers.GZip; + +public partial class GZipWriter +{ + public override async ValueTask WriteAsync( + string filename, + Stream source, + DateTime? modificationTime, + CancellationToken cancellationToken = default + ) + { + if (_wroteToStream) + { + throw new ArgumentException("Can only write a single stream to a GZip file."); + } + var stream = (GZipStream)OutputStream.NotNull(); + stream.FileName = filename; + stream.LastModified = modificationTime; + var progressStream = WrapWithProgress(source, filename); +#if LEGACY_DOTNET + await progressStream.CopyToAsync(stream).ConfigureAwait(false); +#else + await progressStream.CopyToAsync(stream, cancellationToken).ConfigureAwait(false); +#endif + _wroteToStream = true; + } + + public override ValueTask WriteDirectoryAsync( + string directoryName, + DateTime? modificationTime, + CancellationToken cancellationToken = default + ) => throw new NotSupportedException("GZip archives do not support directory entries."); +} diff --git a/src/SharpCompress/Writers/GZip/GZipWriter.Factory.cs b/src/SharpCompress/Writers/GZip/GZipWriter.Factory.cs new file mode 100644 index 00000000..cae0b91d --- /dev/null +++ b/src/SharpCompress/Writers/GZip/GZipWriter.Factory.cs @@ -0,0 +1,59 @@ +#if NET8_0_OR_GREATER +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Writers.GZip; + +public partial class GZipWriter : IWriterOpenable +{ + public static IWriter OpenWriter(string filePath, GZipWriterOptions writerOptions) + { + filePath.NotNullOrEmpty(nameof(filePath)); + return OpenWriter(new FileInfo(filePath), writerOptions); + } + + public static IWriter OpenWriter(FileInfo fileInfo, GZipWriterOptions writerOptions) + { + fileInfo.NotNull(nameof(fileInfo)); + return new GZipWriter(fileInfo.OpenWrite(), writerOptions with { LeaveStreamOpen = false }); + } + + public static IWriter OpenWriter(Stream stream, GZipWriterOptions writerOptions) + { + stream.RequireWritable(); + return new GZipWriter(stream, writerOptions); + } + + public static ValueTask OpenAsyncWriter( + string filePath, + GZipWriterOptions writerOptions, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncWriter)OpenWriter(filePath, writerOptions)); + } + + public static ValueTask OpenAsyncWriter( + Stream stream, + GZipWriterOptions writerOptions, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncWriter)OpenWriter(stream, writerOptions)); + } + + public static ValueTask OpenAsyncWriter( + FileInfo fileInfo, + GZipWriterOptions writerOptions, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncWriter)OpenWriter(fileInfo, writerOptions)); + } +} +#endif diff --git a/src/SharpCompress/Writers/GZip/GZipWriter.cs b/src/SharpCompress/Writers/GZip/GZipWriter.cs index 5c990a7d..3c62ff2d 100644 --- a/src/SharpCompress/Writers/GZip/GZipWriter.cs +++ b/src/SharpCompress/Writers/GZip/GZipWriter.cs @@ -1,13 +1,15 @@ using System; using System.IO; +using System.Threading.Tasks; using SharpCompress.Common; using SharpCompress.Compressors; using SharpCompress.Compressors.Deflate; using SharpCompress.IO; +using SharpCompress.Providers; namespace SharpCompress.Writers.GZip; -public sealed class GZipWriter : AbstractWriter +public sealed partial class GZipWriter : AbstractWriter { private bool _wroteToStream; @@ -16,16 +18,26 @@ public sealed class GZipWriter : AbstractWriter { if (WriterOptions.LeaveStreamOpen) { - destination = NonDisposingStream.Create(destination); + destination = SharpCompressStream.CreateNonDisposing(destination); } - InitalizeStream( - new GZipStream( - destination, - CompressionMode.Compress, - options?.CompressionLevel ?? CompressionLevel.Default, - WriterOptions.ArchiveEncoding.GetEncoding() - ) + + // Use the configured compression providers + var providers = WriterOptions.Providers; + + // Create the GZip stream using the provider + var compressionStream = providers.CreateCompressStream( + CompressionType.GZip, + destination, + WriterOptions.CompressionLevel ); + + // If using internal GZipStream, set the encoding for header filename + if (compressionStream is GZipStream gzipStream) + { + // Note: FileName and LastModified will be set in Write() + } + + InitializeStream(compressionStream); } protected override void Dispose(bool isDisposing) @@ -33,21 +45,51 @@ public sealed class GZipWriter : AbstractWriter if (isDisposing) { //dispose here to finish the GZip, GZip won't close the underlying stream - OutputStream.Dispose(); + OutputStream.NotNull().Dispose(); } base.Dispose(isDisposing); } +#pragma warning disable CA2215 // base.DisposeAsync() calls the sync Dispose path for writers. + public override async ValueTask DisposeAsync() + { + if (_isDisposed) + { + return; + } + + GC.SuppressFinalize(this); + _isDisposed = true; + if (OutputStream is IAsyncDisposable asyncDisposable) + { + await asyncDisposable.DisposeAsync().ConfigureAwait(false); + } + else + { + OutputStream.NotNull().Dispose(); + } + } +#pragma warning restore CA2215 + public override void Write(string filename, Stream source, DateTime? modificationTime) { if (_wroteToStream) { throw new ArgumentException("Can only write a single stream to a GZip file."); } - var stream = (GZipStream)OutputStream; - stream.FileName = filename; - stream.LastModified = modificationTime; - source.TransferTo(stream); + + // Set metadata on the stream if it's the internal GZipStream + if (OutputStream is GZipStream gzipStream) + { + gzipStream.FileName = filename; + gzipStream.LastModified = modificationTime; + } + + var progressStream = WrapWithProgress(source, filename); + progressStream.CopyTo(OutputStream.NotNull(), WriterOptions.BufferSize); _wroteToStream = true; } + + public override void WriteDirectory(string directoryName, DateTime? modificationTime) => + throw new NotSupportedException("GZip archives do not support directory entries."); } diff --git a/src/SharpCompress/Writers/GZip/GZipWriterOptions.cs b/src/SharpCompress/Writers/GZip/GZipWriterOptions.cs index 35e19faf..ca1b6a9d 100644 --- a/src/SharpCompress/Writers/GZip/GZipWriterOptions.cs +++ b/src/SharpCompress/Writers/GZip/GZipWriterOptions.cs @@ -1,24 +1,140 @@ +using System; using SharpCompress.Common; -using SharpCompress.Compressors.Deflate; +using SharpCompress.Common.Options; +using SharpCompress.Compressors; +using SharpCompress.Providers; +using SharpCompress.Writers; +using D = SharpCompress.Compressors.Deflate; namespace SharpCompress.Writers.GZip; -public class GZipWriterOptions : WriterOptions +/// +/// Options for configuring GZip writer behavior. +/// +/// +/// Use factory methods, property setters, or fluent helpers for creation: +/// +/// var options = WriterOptions.ForGZip().WithLeaveStreamOpen(false).WithCompressionLevel(9); +/// +/// +public sealed record GZipWriterOptions : IWriterOptions { - public GZipWriterOptions() - : base(CompressionType.GZip) { } + private int _compressionLevel = (int)D.CompressionLevel.Default; - internal GZipWriterOptions(WriterOptions options) - : base(options.CompressionType) + /// + /// The compression type (always GZip for this writer). + /// + public CompressionType CompressionType { - LeaveStreamOpen = options.LeaveStreamOpen; - ArchiveEncoding = options.ArchiveEncoding; - - if (options is GZipWriterOptions writerOptions) + get => CompressionType.GZip; + set { - CompressionLevel = writerOptions.CompressionLevel; + if (value != CompressionType.GZip) + { + throw new ArgumentOutOfRangeException( + nameof(CompressionType), + value, + "GZipWriterOptions only supports CompressionType.GZip." + ); + } } } - public CompressionLevel CompressionLevel { get; set; } = CompressionLevel.Default; + /// + /// The compression level to be used (0-9 for Deflate). + /// + public int CompressionLevel + { + get => _compressionLevel; + set + { + CompressionLevelValidation.Validate(CompressionType.GZip, value); + _compressionLevel = value; + } + } + + /// + /// SharpCompress will keep the supplied streams open. Default is true. + /// + public bool LeaveStreamOpen { get; set; } = true; + + /// + /// Encoding to use for archive entry names. + /// + public IArchiveEncoding ArchiveEncoding { get; set; } = new ArchiveEncoding(); + + /// + /// An optional progress reporter for tracking compression operations. + /// + public IProgress? Progress { get; set; } + + /// + /// Buffer size for writer stream copy operations. + /// + public int BufferSize { get; set; } = Constants.BufferSize; + + /// + /// Registry of compression providers. + /// Defaults to but can be replaced with custom implementations, such as + /// System.IO.Compression for GZip on modern .NET. + /// + public CompressionProviderRegistry Providers { get; set; } = + CompressionProviderRegistry.Default; + + /// + /// Creates a new GZipWriterOptions instance with default values. + /// + public GZipWriterOptions() { } + + /// + /// Creates a new GZipWriterOptions instance with the specified compression level. + /// + /// The compression level (0-9). + public GZipWriterOptions(int compressionLevel) + { + CompressionLevel = compressionLevel; + } + + /// + /// Creates a new GZipWriterOptions instance with the specified Deflate compression level. + /// + /// The Deflate compression level. + public GZipWriterOptions(D.CompressionLevel compressionLevel) + { + CompressionLevel = (int)compressionLevel; + } + + // Note: Constructor with boolean leaveStreamOpen parameter removed. + // Use the fluent WithLeaveStreamOpen() helper or object initializer instead: + // new GZipWriterOptions() { LeaveStreamOpen = false } + // or + // WriterOptions.ForGZip().WithLeaveStreamOpen(false) + + /// + /// Creates a new GZipWriterOptions instance from an existing WriterOptions instance. + /// + /// The WriterOptions to copy values from. + public GZipWriterOptions(WriterOptions options) + { + CompressionLevel = options.CompressionLevel; + LeaveStreamOpen = options.LeaveStreamOpen; + ArchiveEncoding = options.ArchiveEncoding; + Progress = options.Progress; + BufferSize = options.BufferSize; + Providers = options.Providers; + } + + /// + /// Creates a new GZipWriterOptions instance from an existing IWriterOptions instance. + /// + /// The IWriterOptions to copy values from. + public GZipWriterOptions(IWriterOptions options) + { + CompressionLevel = options.CompressionLevel; + LeaveStreamOpen = options.LeaveStreamOpen; + ArchiveEncoding = options.ArchiveEncoding; + Progress = options.Progress; + BufferSize = options.BufferSize; + Providers = options.Providers; + } } diff --git a/src/SharpCompress/Writers/IAsyncWriter.cs b/src/SharpCompress/Writers/IAsyncWriter.cs new file mode 100644 index 00000000..2cf1c8e4 --- /dev/null +++ b/src/SharpCompress/Writers/IAsyncWriter.cs @@ -0,0 +1,23 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Writers; + +public interface IAsyncWriter : IAsyncDisposable +{ + ArchiveType Type { get; } + ValueTask WriteAsync( + string filename, + Stream source, + DateTime? modificationTime, + CancellationToken cancellationToken = default + ); + ValueTask WriteDirectoryAsync( + string directoryName, + DateTime? modificationTime, + CancellationToken cancellationToken = default + ); +} diff --git a/src/SharpCompress/Writers/IWriter.cs b/src/SharpCompress/Writers/IWriter.cs index bde2fcd9..e1fee248 100644 --- a/src/SharpCompress/Writers/IWriter.cs +++ b/src/SharpCompress/Writers/IWriter.cs @@ -6,6 +6,7 @@ namespace SharpCompress.Writers; public interface IWriter : IDisposable { - ArchiveType WriterType { get; } + ArchiveType Type { get; } void Write(string filename, Stream source, DateTime? modificationTime); + void WriteDirectory(string directoryName, DateTime? modificationTime); } diff --git a/src/SharpCompress/Writers/IWriterExtensions.cs b/src/SharpCompress/Writers/IWriterExtensions.cs index 56f3966a..a178dd5d 100644 --- a/src/SharpCompress/Writers/IWriterExtensions.cs +++ b/src/SharpCompress/Writers/IWriterExtensions.cs @@ -1,55 +1,130 @@ -using System; +using System; using System.IO; using System.Linq; +using System.Threading; +using System.Threading.Tasks; namespace SharpCompress.Writers; public static class IWriterExtensions { - public static void Write(this IWriter writer, string entryPath, Stream source) => - writer.Write(entryPath, source, null); - - public static void Write(this IWriter writer, string entryPath, FileInfo source) + extension(IWriter writer) { - if (!source.Exists) + public void Write(string entryPath, Stream source) => writer.Write(entryPath, source, null); + + public void Write(string entryPath, FileInfo source) { - throw new ArgumentException("Source does not exist: " + source.FullName); - } - using var stream = source.OpenRead(); - writer.Write(entryPath, stream, source.LastWriteTime); - } + if (!source.Exists) + { + throw new ArgumentException("Source does not exist: " + source.FullName); + } - public static void Write(this IWriter writer, string entryPath, string source) => - writer.Write(entryPath, new FileInfo(source)); - - public static void WriteAll( - this IWriter writer, - string directory, - string searchPattern = "*", - SearchOption option = SearchOption.TopDirectoryOnly - ) => writer.WriteAll(directory, searchPattern, null, option); - - public static void WriteAll( - this IWriter writer, - string directory, - string searchPattern = "*", - Func? fileSearchFunc = null, - SearchOption option = SearchOption.TopDirectoryOnly - ) - { - if (!Directory.Exists(directory)) - { - throw new ArgumentException("Directory does not exist: " + directory); + using var stream = source.OpenRead(); + writer.Write(entryPath, stream, source.LastWriteTime); } - fileSearchFunc ??= n => true; - foreach ( - var file in Directory - .EnumerateFiles(directory, searchPattern, option) - .Where(fileSearchFunc) + public void Write(string entryPath, string source) => + writer.Write(entryPath, new FileInfo(source)); + + public void WriteAll( + string directory, + string searchPattern = "*", + SearchOption option = SearchOption.TopDirectoryOnly + ) => writer.WriteAll(directory, searchPattern, null, option); + + public void WriteAll( + string directory, + string searchPattern = "*", + Func? fileSearchFunc = null, + SearchOption option = SearchOption.TopDirectoryOnly ) { - writer.Write(file.Substring(directory.Length), file); + if (!Directory.Exists(directory)) + { + throw new ArgumentException("Directory does not exist: " + directory); + } + + fileSearchFunc ??= n => true; + foreach ( + var file in Directory + .EnumerateFiles(directory, searchPattern, option) + .Where(fileSearchFunc) + ) + { + writer.Write(file.Substring(directory.Length), file); + } } + + public void WriteDirectory(string directoryName) => + writer.WriteDirectory(directoryName, null); + } + + extension(IAsyncWriter writer) + { + public ValueTask WriteAsync( + string entryPath, + Stream source, + CancellationToken cancellationToken = default + ) => writer.WriteAsync(entryPath, source, null, cancellationToken); + + public async ValueTask WriteAsync( + string entryPath, + FileInfo source, + CancellationToken cancellationToken = default + ) + { + if (!source.Exists) + { + throw new ArgumentException("Source does not exist: " + source.FullName); + } + using var stream = source.OpenRead(); + await writer + .WriteAsync(entryPath, stream, source.LastWriteTime, cancellationToken) + .ConfigureAwait(false); + } + + public ValueTask WriteAsync( + string entryPath, + string source, + CancellationToken cancellationToken = default + ) => writer.WriteAsync(entryPath, new FileInfo(source), cancellationToken); + + public ValueTask WriteAllAsync( + string directory, + string searchPattern = "*", + SearchOption option = SearchOption.TopDirectoryOnly, + CancellationToken cancellationToken = default + ) => writer.WriteAllAsync(directory, searchPattern, null, option, cancellationToken); + + public async ValueTask WriteAllAsync( + string directory, + string searchPattern = "*", + Func? fileSearchFunc = null, + SearchOption option = SearchOption.TopDirectoryOnly, + CancellationToken cancellationToken = default + ) + { + if (!Directory.Exists(directory)) + { + throw new ArgumentException("Directory does not exist: " + directory); + } + + fileSearchFunc ??= n => true; + foreach ( + var file in Directory + .EnumerateFiles(directory, searchPattern, option) + .Where(fileSearchFunc) + ) + { + await writer + .WriteAsync(file.Substring(directory.Length), file, cancellationToken) + .ConfigureAwait(false); + } + } + + public ValueTask WriteDirectoryAsync( + string directoryName, + CancellationToken cancellationToken = default + ) => writer.WriteDirectoryAsync(directoryName, null, cancellationToken); } } diff --git a/src/SharpCompress/Writers/IWriterFactory.cs b/src/SharpCompress/Writers/IWriterFactory.cs index 8e7bc701..c9eef005 100644 --- a/src/SharpCompress/Writers/IWriterFactory.cs +++ b/src/SharpCompress/Writers/IWriterFactory.cs @@ -1,10 +1,18 @@ using System.IO; - +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Options; using SharpCompress.Factories; namespace SharpCompress.Writers; public interface IWriterFactory : IFactory { - IWriter Open(Stream stream, WriterOptions writerOptions); + IWriter OpenWriter(Stream stream, IWriterOptions writerOptions); + + ValueTask OpenAsyncWriter( + Stream stream, + IWriterOptions writerOptions, + CancellationToken cancellationToken = default + ); } diff --git a/src/SharpCompress/Writers/IWriterOpenable.cs b/src/SharpCompress/Writers/IWriterOpenable.cs new file mode 100644 index 00000000..1c584f2f --- /dev/null +++ b/src/SharpCompress/Writers/IWriterOpenable.cs @@ -0,0 +1,42 @@ +#if NET8_0_OR_GREATER +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common.Options; + +namespace SharpCompress.Writers; + +public interface IWriterOpenable + where TWriterOptions : IWriterOptions +{ + public static abstract IWriter OpenWriter(string filePath, TWriterOptions writerOptions); + + public static abstract IWriter OpenWriter(FileInfo fileInfo, TWriterOptions writerOptions); + public static abstract IWriter OpenWriter(Stream stream, TWriterOptions writerOptions); + + /// + /// Opens a Writer asynchronously. + /// + /// The stream to write to. + /// Writer options. + /// Cancellation token. + /// A task that returns an async writer. + public static abstract ValueTask OpenAsyncWriter( + Stream stream, + TWriterOptions writerOptions, + CancellationToken cancellationToken = default + ); + + public static abstract ValueTask OpenAsyncWriter( + string filePath, + TWriterOptions writerOptions, + CancellationToken cancellationToken = default + ); + + public static abstract ValueTask OpenAsyncWriter( + FileInfo fileInfo, + TWriterOptions writerOptions, + CancellationToken cancellationToken = default + ); +} +#endif diff --git a/src/SharpCompress/Writers/SevenZip/SevenZipWriter.Async.cs b/src/SharpCompress/Writers/SevenZip/SevenZipWriter.Async.cs new file mode 100644 index 00000000..9b24371d --- /dev/null +++ b/src/SharpCompress/Writers/SevenZip/SevenZipWriter.Async.cs @@ -0,0 +1,243 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.SevenZip; +using SharpCompress.Compressors.LZMA; +using SharpCompress.Crypto; +using SharpCompress.IO; + +namespace SharpCompress.Writers.SevenZip; + +public partial class SevenZipWriter +{ + /// + /// Asynchronously disposes the writer, finalizing the 7z archive. + /// + public override async ValueTask DisposeAsync() + { + if (_isDisposed) + { + return; + } + GC.SuppressFinalize(this); + _isDisposed = true; + + if (!finalized) + { + finalized = true; + await FinalizeArchiveAsync().ConfigureAwait(false); + } + if (OutputStream is IAsyncDisposable asyncDisposable) + { + await asyncDisposable.DisposeAsync().ConfigureAwait(false); + } + else + { + OutputStream?.Dispose(); + } + // base.DisposeAsync() is a no-op since _isDisposed is already set + await base.DisposeAsync().ConfigureAwait(false); + } + + private async ValueTask FinalizeArchiveAsync() + { + var output = OutputStream.NotNull(); + + // Current position = end of packed data streams + var endOfPackedData = output.Position; + + // Build the header structures + var mainStreamsInfo = BuildStreamsInfo(); + var filesInfo = new SevenZipFilesInfoWriter { Entries = entries.ToArray() }; + + // Write header to a temporary stream first + using var headerStream = new MemoryStream(); + ArchiveHeaderWriter.WriteRawHeader(headerStream, mainStreamsInfo, filesInfo); + + // Optionally compress the header + if (sevenZipOptions.CompressHeader && headerStream.Length > 0) + { + await WriteCompressedHeaderAsync(headerStream, endOfPackedData).ConfigureAwait(false); + } + else + { + await WriteRawHeaderToOutputAsync(headerStream, endOfPackedData).ConfigureAwait(false); + } + } + + private async ValueTask WriteCompressedHeaderAsync( + MemoryStream rawHeaderStream, + long endOfPackedData + ) + { + var output = OutputStream.NotNull(); + + // Compress header using LZMA (always LZMA, not LZMA2, matching 7-Zip standard behavior) + rawHeaderStream.Position = 0; + var headerCompressor = new SevenZipStreamsCompressor(output); + var headerPacked = await headerCompressor + .CompressAsync(rawHeaderStream, CompressionType.LZMA, sevenZipOptions.LzmaProperties) + .ConfigureAwait(false); + + // Build EncodedHeader StreamsInfo (describes how to decompress the header) + var headerPackPos = (ulong)(endOfPackedData - SevenZipSignatureHeaderWriter.HeaderSize); + var headerStreamsInfo = new SevenZipStreamsInfoWriter + { + PackInfo = new SevenZipPackInfoWriter + { + PackPos = headerPackPos, + Sizes = headerPacked.Sizes, + CRCs = headerPacked.CRCs, + }, + UnPackInfo = new SevenZipUnPackInfoWriter { Folders = [headerPacked.Folder] }, + }; + + // Write encoded header to a second temporary stream + using var encodedHeaderStream = new MemoryStream(); + ArchiveHeaderWriter.WriteEncodedHeader(encodedHeaderStream, headerStreamsInfo); + + // Write the encoded header to the output + var headerStartPos = output.Position; + encodedHeaderStream.Position = 0; + await encodedHeaderStream.CopyToAsync(output).ConfigureAwait(false); + + // Compute CRC of the encoded header + var headerCrc = Crc32Stream.Compute( + Crc32Stream.DEFAULT_POLYNOMIAL, + Crc32Stream.DEFAULT_SEED, + encodedHeaderStream.GetBuffer().AsSpan(0, (int)encodedHeaderStream.Length) + ); + + // Back-patch signature header + var nextHeaderOffset = (ulong)(headerStartPos - SevenZipSignatureHeaderWriter.HeaderSize); + var nextHeaderSize = (ulong)encodedHeaderStream.Length; + + await SevenZipSignatureHeaderWriter + .WriteFinalAsync(output, nextHeaderOffset, nextHeaderSize, headerCrc) + .ConfigureAwait(false); + + // Seek to end + output.Seek(0, SeekOrigin.End); + } + + private async ValueTask WriteRawHeaderToOutputAsync( + MemoryStream rawHeaderStream, + long endOfPackedData + ) + { + var output = OutputStream.NotNull(); + + // Write raw header directly + var headerStartPos = output.Position; + rawHeaderStream.Position = 0; + await rawHeaderStream.CopyToAsync(output).ConfigureAwait(false); + + // Compute CRC of the raw header + var headerCrc = Crc32Stream.Compute( + Crc32Stream.DEFAULT_POLYNOMIAL, + Crc32Stream.DEFAULT_SEED, + rawHeaderStream.GetBuffer().AsSpan(0, (int)rawHeaderStream.Length) + ); + + // Back-patch signature header + var nextHeaderOffset = (ulong)(headerStartPos - SevenZipSignatureHeaderWriter.HeaderSize); + var nextHeaderSize = (ulong)rawHeaderStream.Length; + + await SevenZipSignatureHeaderWriter + .WriteFinalAsync(output, nextHeaderOffset, nextHeaderSize, headerCrc) + .ConfigureAwait(false); + + // Seek to end + output.Seek(0, SeekOrigin.End); + } + + /// + /// Asynchronously writes a file entry to the 7z archive. + /// + public override async ValueTask WriteAsync( + string filename, + Stream source, + DateTime? modificationTime, + CancellationToken cancellationToken = default + ) + { + if (finalized) + { + throw new ObjectDisposedException( + nameof(SevenZipWriter), + "Cannot write to a finalized archive." + ); + } + + cancellationToken.ThrowIfCancellationRequested(); + await EnsurePlaceholderWrittenAsync(cancellationToken).ConfigureAwait(false); + + filename = NormalizeFilename(filename); + var progressStream = WrapWithProgress(source, filename); + + var isEmpty = source.CanSeek && source.Length == 0; + + if (isEmpty) + { + entries.Add( + new SevenZipWriteEntry + { + Name = filename, + ModificationTime = modificationTime, + IsDirectory = false, + IsEmpty = true, + } + ); + return; + } + + var output = OutputStream.NotNull(); + var outputPosBefore = output.Position; + var compressor = new SevenZipStreamsCompressor(output); + var packed = await compressor + .CompressAsync( + progressStream, + sevenZipOptions.CompressionType, + sevenZipOptions.LzmaProperties, + cancellationToken + ) + .ConfigureAwait(false); + + var actuallyEmpty = packed.Folder.GetUnpackSize() == 0; + if (!actuallyEmpty) + { + packedStreams.Add(packed); + } + else + { + output.Position = outputPosBefore; + output.SetLength(outputPosBefore); + } + + entries.Add( + new SevenZipWriteEntry + { + Name = filename, + ModificationTime = modificationTime, + IsDirectory = false, + IsEmpty = isEmpty || actuallyEmpty, + } + ); + } + + /// + /// Asynchronously writes a directory entry to the 7z archive. + /// + public override ValueTask WriteDirectoryAsync( + string directoryName, + DateTime? modificationTime, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + WriteDirectory(directoryName, modificationTime); + return new ValueTask(); + } +} diff --git a/src/SharpCompress/Writers/SevenZip/SevenZipWriter.Factory.cs b/src/SharpCompress/Writers/SevenZip/SevenZipWriter.Factory.cs new file mode 100644 index 00000000..1191a83a --- /dev/null +++ b/src/SharpCompress/Writers/SevenZip/SevenZipWriter.Factory.cs @@ -0,0 +1,82 @@ +#if NET8_0_OR_GREATER +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Writers.SevenZip; + +public partial class SevenZipWriter : IWriterOpenable +{ + /// + /// Opens a new SevenZipWriter for the specified file path. + /// + public static IWriter OpenWriter(string filePath, SevenZipWriterOptions writerOptions) + { + filePath.NotNullOrEmpty(nameof(filePath)); + return OpenWriter(new FileInfo(filePath), writerOptions); + } + + /// + /// Opens a new SevenZipWriter for the specified file. + /// + public static IWriter OpenWriter(FileInfo fileInfo, SevenZipWriterOptions writerOptions) + { + fileInfo.NotNull(nameof(fileInfo)); + return new SevenZipWriter( + fileInfo.OpenWrite(), + writerOptions with + { + LeaveStreamOpen = false, + } + ); + } + + /// + /// Opens a new SevenZipWriter for the specified stream. + /// + public static IWriter OpenWriter(Stream stream, SevenZipWriterOptions writerOptions) + { + stream.RequireWritable(); + return new SevenZipWriter(stream, writerOptions); + } + + /// + /// Opens a new async SevenZipWriter for the specified file path. + /// + public static ValueTask OpenAsyncWriter( + string filePath, + SevenZipWriterOptions writerOptions, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncWriter)OpenWriter(filePath, writerOptions)); + } + + /// + /// Opens a new async SevenZipWriter for the specified stream. + /// + public static ValueTask OpenAsyncWriter( + Stream stream, + SevenZipWriterOptions writerOptions, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncWriter)OpenWriter(stream, writerOptions)); + } + + /// + /// Opens a new async SevenZipWriter for the specified file. + /// + public static ValueTask OpenAsyncWriter( + FileInfo fileInfo, + SevenZipWriterOptions writerOptions, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncWriter)OpenWriter(fileInfo, writerOptions)); + } +} +#endif diff --git a/src/SharpCompress/Writers/SevenZip/SevenZipWriter.cs b/src/SharpCompress/Writers/SevenZip/SevenZipWriter.cs new file mode 100644 index 00000000..f29bc20d --- /dev/null +++ b/src/SharpCompress/Writers/SevenZip/SevenZipWriter.cs @@ -0,0 +1,389 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.SevenZip; +using SharpCompress.Compressors.LZMA; +using SharpCompress.Crypto; +using SharpCompress.IO; + +namespace SharpCompress.Writers.SevenZip; + +/// +/// Writes 7z archives in non-solid mode (each file compressed independently). +/// Requires a seekable output stream for back-patching the signature header. +/// TODO: solid mode support in a future iteration. +/// TODO: IWritableArchive support in a future iteration. +/// +public partial class SevenZipWriter : AbstractWriter +{ + private readonly SevenZipWriterOptions sevenZipOptions; + private readonly List entries = []; + private readonly List packedStreams = []; + private bool finalized; + private bool _placeholderWritten; + + /// + /// Creates a new SevenZipWriter writing to the specified stream. + /// + /// Seekable output stream. + /// Writer options. + public SevenZipWriter(Stream destination, SevenZipWriterOptions options) + : base(ArchiveType.SevenZip, options) + { + if (!destination.CanSeek) + { + throw new ArchiveOperationException( + "7z writing requires a seekable stream for header back-patching." + ); + } + + sevenZipOptions = options; + + if (options.LeaveStreamOpen) + { + destination = SharpCompressStream.CreateNonDisposing(destination); + } + + InitializeStream(destination); + } + + /// + /// Ensures the placeholder signature header has been written synchronously. + /// Called before the first sync write. + /// + private void EnsurePlaceholderWritten() + { + if (!_placeholderWritten) + { + _placeholderWritten = true; + // Write placeholder signature header (32 bytes) - will be back-patched on finalize + SevenZipSignatureHeaderWriter.WritePlaceholder(OutputStream.NotNull()); + } + } + + /// + /// Ensures the placeholder signature header has been written asynchronously. + /// Called before the first async write. + /// + private async ValueTask EnsurePlaceholderWrittenAsync(CancellationToken cancellationToken) + { + if (!_placeholderWritten) + { + _placeholderWritten = true; + // Write placeholder signature header (32 bytes) - will be back-patched on finalize + await SevenZipSignatureHeaderWriter + .WritePlaceholderAsync(OutputStream.NotNull(), cancellationToken) + .ConfigureAwait(false); + } + } + + /// + /// Writes a file entry to the archive. + /// + public override void Write(string filename, Stream source, DateTime? modificationTime) + { + if (finalized) + { + throw new ObjectDisposedException( + nameof(SevenZipWriter), + "Cannot write to a finalized archive." + ); + } + + EnsurePlaceholderWritten(); + + filename = NormalizeFilename(filename); + var progressStream = WrapWithProgress(source, filename); + + var isEmpty = source.CanSeek && source.Length == 0; + + if (isEmpty) + { + // Empty file - no compression, just record metadata + entries.Add( + new SevenZipWriteEntry + { + Name = filename, + ModificationTime = modificationTime, + IsDirectory = false, + IsEmpty = true, + } + ); + return; + } + + // Compress file data to output stream + var output = OutputStream.NotNull(); + var outputPosBefore = output.Position; + var compressor = new SevenZipStreamsCompressor(output); + var packed = compressor.Compress( + progressStream, + sevenZipOptions.CompressionType, + sevenZipOptions.LzmaProperties + ); + + // Check if the stream was actually empty (handles non-seekable streams with no data) + var actuallyEmpty = packed.Folder.GetUnpackSize() == 0; + if (!actuallyEmpty) + { + packedStreams.Add(packed); + } + else + { + // Rewind output to erase orphaned encoder header/end-marker bytes + // so they don't shift subsequent pack stream offsets + output.Position = outputPosBefore; + output.SetLength(outputPosBefore); + } + + entries.Add( + new SevenZipWriteEntry + { + Name = filename, + ModificationTime = modificationTime, + IsDirectory = false, + IsEmpty = isEmpty || actuallyEmpty, + } + ); + } + + /// + /// Writes a directory entry to the archive. + /// + public override void WriteDirectory(string directoryName, DateTime? modificationTime) + { + if (finalized) + { + throw new ObjectDisposedException( + nameof(SevenZipWriter), + "Cannot write to a finalized archive." + ); + } + + directoryName = NormalizeFilename(directoryName); + directoryName = directoryName.TrimEnd('/'); + + entries.Add( + new SevenZipWriteEntry + { + Name = directoryName, + ModificationTime = modificationTime, + IsDirectory = true, + IsEmpty = true, + Attributes = 0x10, // FILE_ATTRIBUTE_DIRECTORY + } + ); + } + + /// + /// Finalizes the archive - writes metadata headers and back-patches the signature header. + /// + protected override void Dispose(bool isDisposing) + { + if (isDisposing && !finalized && !_isDisposed) + { + finalized = true; + FinalizeArchive(); + } + base.Dispose(isDisposing); + } + + private void FinalizeArchive() + { + var output = OutputStream.NotNull(); + + // Current position = end of packed data streams + var endOfPackedData = output.Position; + + // Build the header structures + var mainStreamsInfo = BuildStreamsInfo(); + var filesInfo = new SevenZipFilesInfoWriter { Entries = entries.ToArray() }; + + // Write header to a temporary stream first + using var headerStream = new PooledMemoryStream(); + ArchiveHeaderWriter.WriteRawHeader(headerStream, mainStreamsInfo, filesInfo); + + // Optionally compress the header + if (sevenZipOptions.CompressHeader && headerStream.Length > 0) + { + WriteCompressedHeader(headerStream, endOfPackedData); + } + else + { + WriteRawHeaderToOutput(headerStream, endOfPackedData); + } + } + + private void WriteCompressedHeader(MemoryStream rawHeaderStream, long endOfPackedData) + { + var output = OutputStream.NotNull(); + + // Compress header using LZMA (always LZMA, not LZMA2, matching 7-Zip standard behavior) + rawHeaderStream.Position = 0; + var headerCompressor = new SevenZipStreamsCompressor(output); + var headerPacked = headerCompressor.Compress( + rawHeaderStream, + CompressionType.LZMA, + sevenZipOptions.LzmaProperties + ); + + // Build EncodedHeader StreamsInfo (describes how to decompress the header) + var headerPackPos = (ulong)(endOfPackedData - SevenZipSignatureHeaderWriter.HeaderSize); + var headerStreamsInfo = new SevenZipStreamsInfoWriter + { + PackInfo = new SevenZipPackInfoWriter + { + PackPos = headerPackPos, + Sizes = headerPacked.Sizes, + CRCs = headerPacked.CRCs, + }, + UnPackInfo = new SevenZipUnPackInfoWriter { Folders = [headerPacked.Folder] }, + }; + + // Write encoded header to a second temporary stream + using var encodedHeaderStream = new PooledMemoryStream(); + ArchiveHeaderWriter.WriteEncodedHeader(encodedHeaderStream, headerStreamsInfo); + + // Write the encoded header to the output + var headerStartPos = output.Position; + encodedHeaderStream.Position = 0; + encodedHeaderStream.CopyTo(output); + + // Compute CRC of the encoded header without allocating a contiguous buffer + var encodedHeaderCrcSink = new Crc32Stream(Stream.Null); + encodedHeaderStream.WriteTo(encodedHeaderCrcSink); + var headerCrc = encodedHeaderCrcSink.Crc; + + // Back-patch signature header + var nextHeaderOffset = (ulong)(headerStartPos - SevenZipSignatureHeaderWriter.HeaderSize); + var nextHeaderSize = (ulong)encodedHeaderStream.Length; + + SevenZipSignatureHeaderWriter.WriteFinal( + output, + nextHeaderOffset, + nextHeaderSize, + headerCrc + ); + + // Seek to end + output.Seek(0, SeekOrigin.End); + } + + private void WriteRawHeaderToOutput(MemoryStream rawHeaderStream, long endOfPackedData) + { + var output = OutputStream.NotNull(); + + // Write raw header directly + var headerStartPos = output.Position; + rawHeaderStream.Position = 0; + rawHeaderStream.CopyTo(output); + + // Compute CRC of the raw header without allocating a contiguous buffer + var rawHeaderCrcSink = new Crc32Stream(Stream.Null); + rawHeaderStream.WriteTo(rawHeaderCrcSink); + var headerCrc = rawHeaderCrcSink.Crc; + + // Back-patch signature header + var nextHeaderOffset = (ulong)(headerStartPos - SevenZipSignatureHeaderWriter.HeaderSize); + var nextHeaderSize = (ulong)rawHeaderStream.Length; + + SevenZipSignatureHeaderWriter.WriteFinal( + output, + nextHeaderOffset, + nextHeaderSize, + headerCrc + ); + + // Seek to end + output.Seek(0, SeekOrigin.End); + } + + private SevenZipStreamsInfoWriter? BuildStreamsInfo() + { + if (packedStreams.Count == 0) + { + return null; + } + + // Collect all packed sizes and CRCs across all folders + var totalPackStreams = 0; + for (var i = 0; i < packedStreams.Count; i++) + { + totalPackStreams += packedStreams[i].Sizes.Length; + } + + var allSizes = new ulong[totalPackStreams]; + var allCRCs = new uint?[totalPackStreams]; + var folders = new CFolder[packedStreams.Count]; + + var sizeIndex = 0; + for (var i = 0; i < packedStreams.Count; i++) + { + var ps = packedStreams[i]; + for (var j = 0; j < ps.Sizes.Length; j++) + { + allSizes[sizeIndex] = ps.Sizes[j]; + allCRCs[sizeIndex] = ps.CRCs[j]; + sizeIndex++; + } + folders[i] = ps.Folder; + } + + // Build per-file unpack sizes and CRCs for SubStreamsInfo + // In non-solid mode, each folder has exactly 1 file + var numUnPackStreamsPerFolder = new ulong[packedStreams.Count]; + var unpackSizes = new ulong[packedStreams.Count]; + var fileCRCs = new uint?[packedStreams.Count]; + + for (var i = 0; i < packedStreams.Count; i++) + { + numUnPackStreamsPerFolder[i] = 1; + unpackSizes[i] = (ulong)packedStreams[i].Folder.GetUnpackSize(); + fileCRCs[i] = packedStreams[i].Folder._unpackCrc; + + // Clear folder-level CRC (it's moved to SubStreamsInfo) + packedStreams[i].Folder._unpackCrc = null; + } + + return new SevenZipStreamsInfoWriter + { + PackInfo = new SevenZipPackInfoWriter + { + PackPos = 0, + Sizes = allSizes, + CRCs = allCRCs, + }, + UnPackInfo = new SevenZipUnPackInfoWriter { Folders = folders }, + SubStreamsInfo = new SevenZipSubStreamsInfoWriter + { + Folders = folders, + NumUnPackStreamsInFolders = numUnPackStreamsPerFolder, + UnPackSizes = unpackSizes, + CRCs = fileCRCs, + }, + }; + } + + /// + /// Normalizes a filename for 7z archive storage. + /// Converts backslashes to forward slashes and removes leading slashes. + /// + private static string NormalizeFilename(string filename) + { + filename = filename.Replace('\\', '/'); + + // Remove drive letter prefix (e.g., "C:/") + if (filename.Length >= 3 && filename[1] == ':' && filename[2] == '/') + { + filename = filename.Substring(3); + } + + // Remove leading slashes + filename = filename.TrimStart('/'); + + return filename; + } +} diff --git a/src/SharpCompress/Writers/SevenZip/SevenZipWriterOptions.cs b/src/SharpCompress/Writers/SevenZip/SevenZipWriterOptions.cs new file mode 100644 index 00000000..6ed26a42 --- /dev/null +++ b/src/SharpCompress/Writers/SevenZip/SevenZipWriterOptions.cs @@ -0,0 +1,135 @@ +using System; +using SharpCompress.Common; +using SharpCompress.Common.Options; +using SharpCompress.Compressors.LZMA; +using SharpCompress.Providers; + +namespace SharpCompress.Writers.SevenZip; + +/// +/// Options for configuring 7z writer behavior. +/// +public sealed record SevenZipWriterOptions : IWriterOptions +{ + private CompressionType _compressionType; + private int _compressionLevel; + + /// + /// The compression type to use. Supported: LZMA and LZMA2 (default). + /// + public CompressionType CompressionType + { + get => _compressionType; + set + { + if (value != CompressionType.LZMA && value != CompressionType.LZMA2) + { + throw new ArgumentException( + $"SevenZipWriter only supports CompressionType.LZMA and CompressionType.LZMA2. Got: {value}", + nameof(value) + ); + } + _compressionType = value; + } + } + + /// + /// Compression level (not used for LZMA in this implementation; reserved for future use). + /// + public int CompressionLevel + { + get => _compressionLevel; + set => _compressionLevel = value; + } + + /// + /// SharpCompress will keep the supplied streams open. Default is true. + /// + public bool LeaveStreamOpen { get; set; } = true; + + /// + /// Encoding to use for archive entry names. + /// + public IArchiveEncoding ArchiveEncoding { get; set; } = new ArchiveEncoding(); + + /// + /// An optional progress reporter for tracking compression operations. + /// + public IProgress? Progress { get; set; } + + /// + /// Buffer size for writer stream copy operations. + /// + public int BufferSize { get; set; } = Constants.BufferSize; + + /// + /// Registry of compression providers. + /// Defaults to but can be replaced with custom implementations. + /// + public CompressionProviderRegistry Providers { get; set; } = + CompressionProviderRegistry.Default; + + /// + /// Whether to compress the archive header itself using LZMA. + /// Default is true, matching standard 7-Zip behavior. + /// + public bool CompressHeader { get; set; } = true; + + /// + /// Custom LZMA encoder properties. Null uses defaults (1MB dictionary, 32 fast bytes). + /// + public LzmaEncoderProperties? LzmaProperties { get; set; } + + /// + /// Creates a new SevenZipWriterOptions instance with LZMA2 compression (default). + /// + public SevenZipWriterOptions() + { + CompressionType = CompressionType.LZMA2; + } + + /// + /// Creates a new SevenZipWriterOptions instance with the specified compression type. + /// + /// The compression type for the archive. + public SevenZipWriterOptions(CompressionType compressionType) + { + CompressionType = compressionType; + } + + /// + /// Creates a new SevenZipWriterOptions instance from an existing WriterOptions instance. + /// + /// The WriterOptions to copy values from. + public SevenZipWriterOptions(WriterOptions options) + { + CompressionType = options.CompressionType; + CompressionLevel = options.CompressionLevel; + LeaveStreamOpen = options.LeaveStreamOpen; + ArchiveEncoding = options.ArchiveEncoding; + Progress = options.Progress; + BufferSize = options.BufferSize; + Providers = options.Providers; + } + + /// + /// Creates a new SevenZipWriterOptions from an existing IWriterOptions instance. + /// + /// The IWriterOptions to copy values from. + public SevenZipWriterOptions(IWriterOptions options) + { + CompressionType = options.CompressionType; + CompressionLevel = options.CompressionLevel; + LeaveStreamOpen = options.LeaveStreamOpen; + ArchiveEncoding = options.ArchiveEncoding; + Progress = options.Progress; + BufferSize = options.BufferSize; + Providers = options.Providers; + } + + /// + /// Implicit conversion from CompressionType to SevenZipWriterOptions. + /// + public static implicit operator SevenZipWriterOptions(CompressionType compressionType) => + new(compressionType); +} diff --git a/src/SharpCompress/Writers/Tar/TarWriter.Async.cs b/src/SharpCompress/Writers/Tar/TarWriter.Async.cs new file mode 100644 index 00000000..9244608f --- /dev/null +++ b/src/SharpCompress/Writers/Tar/TarWriter.Async.cs @@ -0,0 +1,128 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Tar.Headers; +using SharpCompress.IO; +using SharpCompress.Providers; + +namespace SharpCompress.Writers.Tar; + +public partial class TarWriter +{ + /// + /// Asynchronously disposes the writer, writing the archive finalization record if required. + /// + public override async ValueTask DisposeAsync() + { + if (_isDisposed) + { + return; + } + GC.SuppressFinalize(this); + _isDisposed = true; + + if (_finalizeArchiveOnClose) + { + await OutputStream.NotNull().WriteAsync(new byte[1024], 0, 1024).ConfigureAwait(false); + } + if (OutputStream is IFinishable finishable) + { + await finishable.FinishAsync(CancellationToken.None).ConfigureAwait(false); + } + if (OutputStream is IAsyncDisposable asyncDisposableOutputStream) + { + await asyncDisposableOutputStream.DisposeAsync().ConfigureAwait(false); + } + else + { + OutputStream?.Dispose(); + } + // base.DisposeAsync() is a no-op since _isDisposed is already set + await base.DisposeAsync().ConfigureAwait(false); + } + + /// + /// Asynchronously writes a directory entry to the TAR archive. + /// + public override async ValueTask WriteDirectoryAsync( + string directoryName, + DateTime? modificationTime, + CancellationToken cancellationToken = default + ) + { + var normalizedName = NormalizeDirectoryName(directoryName); + if (string.IsNullOrEmpty(normalizedName)) + { + return; + } + + var header = new TarHeader(WriterOptions.ArchiveEncoding, _headerFormat); + header.LastModifiedTime = modificationTime ?? TarHeader.EPOCH; + header.Name = normalizedName; + header.Size = 0; + header.EntryType = EntryType.Directory; + await header.WriteAsync(OutputStream.NotNull(), cancellationToken).ConfigureAwait(false); + } + + /// + /// Asynchronously writes a file entry to the TAR archive. + /// + public override async ValueTask WriteAsync( + string filename, + Stream source, + DateTime? modificationTime, + CancellationToken cancellationToken = default + ) => + await WriteAsync(filename, source, modificationTime, null, cancellationToken) + .ConfigureAwait(false); + + /// + /// Asynchronously writes a file entry with optional size specification. + /// + public async ValueTask WriteAsync( + string filename, + Stream source, + DateTime? modificationTime, + long? size, + CancellationToken cancellationToken = default + ) + { + if (!source.CanSeek && size is null) + { + throw new ArgumentException("Seekable stream is required if no size is given."); + } + + var realSize = size ?? source.Length; + + var header = new TarHeader(WriterOptions.ArchiveEncoding, _headerFormat); + + header.LastModifiedTime = modificationTime ?? TarHeader.EPOCH; + header.Name = NormalizeFilename(filename); + header.Size = realSize; + await header.WriteAsync(OutputStream.NotNull(), cancellationToken).ConfigureAwait(false); + var progressStream = WrapWithProgress(source, filename); + var written = await progressStream + .TransferToAsync( + OutputStream.NotNull(), + realSize, + WriterOptions.BufferSize, + cancellationToken + ) + .ConfigureAwait(false); + await PadTo512Async(written, cancellationToken).ConfigureAwait(false); + } + + private async ValueTask PadTo512Async(long size, CancellationToken cancellationToken = default) + { + var zeros = unchecked((int)(((size + 511L) & ~511L) - size)); + if (zeros > 0) + { + await OutputStream + .NotNull() + .WriteAsync(new byte[zeros], 0, zeros, cancellationToken) + .ConfigureAwait(false); + } + } +} diff --git a/src/SharpCompress/Writers/Tar/TarWriter.Factory.cs b/src/SharpCompress/Writers/Tar/TarWriter.Factory.cs new file mode 100644 index 00000000..7613374b --- /dev/null +++ b/src/SharpCompress/Writers/Tar/TarWriter.Factory.cs @@ -0,0 +1,59 @@ +#if NET8_0_OR_GREATER +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Writers.Tar; + +public partial class TarWriter : IWriterOpenable +{ + public static IWriter OpenWriter(string filePath, TarWriterOptions writerOptions) + { + filePath.NotNullOrEmpty(nameof(filePath)); + return OpenWriter(new FileInfo(filePath), writerOptions); + } + + public static IWriter OpenWriter(FileInfo fileInfo, TarWriterOptions writerOptions) + { + fileInfo.NotNull(nameof(fileInfo)); + return new TarWriter(fileInfo.OpenWrite(), writerOptions with { LeaveStreamOpen = false }); + } + + public static IWriter OpenWriter(Stream stream, TarWriterOptions writerOptions) + { + stream.RequireWritable(); + return new TarWriter(stream, writerOptions); + } + + public static ValueTask OpenAsyncWriter( + string filePath, + TarWriterOptions writerOptions, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncWriter)OpenWriter(filePath, writerOptions)); + } + + public static ValueTask OpenAsyncWriter( + Stream stream, + TarWriterOptions writerOptions, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncWriter)OpenWriter(stream, writerOptions)); + } + + public static ValueTask OpenAsyncWriter( + FileInfo fileInfo, + TarWriterOptions writerOptions, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncWriter)OpenWriter(fileInfo, writerOptions)); + } +} +#endif diff --git a/src/SharpCompress/Writers/Tar/TarWriter.cs b/src/SharpCompress/Writers/Tar/TarWriter.cs index 2427db37..8136cc58 100644 --- a/src/SharpCompress/Writers/Tar/TarWriter.cs +++ b/src/SharpCompress/Writers/Tar/TarWriter.cs @@ -3,60 +3,71 @@ using System.IO; using SharpCompress.Common; using SharpCompress.Common.Tar.Headers; using SharpCompress.Compressors; -using SharpCompress.Compressors.BZip2; -using SharpCompress.Compressors.Deflate; -using SharpCompress.Compressors.LZMA; using SharpCompress.IO; +using SharpCompress.Providers; namespace SharpCompress.Writers.Tar; -public class TarWriter : AbstractWriter +public partial class TarWriter : AbstractWriter { - private readonly bool finalizeArchiveOnClose; + private readonly bool _finalizeArchiveOnClose; + private readonly TarHeaderWriteFormat _headerFormat; public TarWriter(Stream destination, TarWriterOptions options) - : base(ArchiveType.Tar, options) + : base(ArchiveType.Tar, GetEffectiveOptions(options)) { - finalizeArchiveOnClose = options.FinalizeArchiveOnClose; + _finalizeArchiveOnClose = options.FinalizeArchiveOnClose; + _headerFormat = options.HeaderFormat; - if (!destination.CanWrite) + InitializeStream(CreateOutputStream(destination, options)); + } + + internal TarWriter(Stream destination, TarWriterOptions options, bool streamIsPrepared) + : base(ArchiveType.Tar, GetEffectiveOptions(options)) + { + _finalizeArchiveOnClose = options.FinalizeArchiveOnClose; + _headerFormat = options.HeaderFormat; + + InitializeStream(streamIsPrepared ? destination : CreateOutputStream(destination, options)); + } + + private static TarWriterOptions GetEffectiveOptions(TarWriterOptions options) => + options with { - throw new ArgumentException("Tars require writable streams."); - } - if (WriterOptions.LeaveStreamOpen) + CompressionType = CompressionType.None, + LeaveStreamOpen = false, + }; + + private static Stream CreateOutputStream(Stream destination, TarWriterOptions options) + { + if (options.LeaveStreamOpen) { - destination = NonDisposingStream.Create(destination); + destination = SharpCompressStream.CreateNonDisposing(destination); } - switch (options.CompressionType) + + var providers = options.Providers; + return options.CompressionType switch { - case CompressionType.None: - break; - case CompressionType.BZip2: - - { - destination = new BZip2Stream(destination, CompressionMode.Compress, false); - } - break; - case CompressionType.GZip: - - { - destination = new GZipStream(destination, CompressionMode.Compress); - } - break; - case CompressionType.LZip: - - { - destination = new LZipStream(destination, CompressionMode.Compress); - } - break; - default: - { - throw new InvalidFormatException( - "Tar does not support compression: " + options.CompressionType - ); - } - } - InitalizeStream(destination); + CompressionType.None => destination, + CompressionType.BZip2 => providers.CreateCompressStream( + CompressionType.BZip2, + destination, + options.CompressionLevel + ), + CompressionType.GZip => providers.CreateCompressStream( + CompressionType.GZip, + destination, + options.CompressionLevel + ), + CompressionType.LZip => providers.CreateCompressStream( + CompressionType.LZip, + destination, + options.CompressionLevel + ), + _ => throw new InvalidFormatException( + "Tar does not support compression: " + options.CompressionType + ), + }; } public override void Write(string filename, Stream source, DateTime? modificationTime) => @@ -66,7 +77,11 @@ public class TarWriter : AbstractWriter { filename = filename.Replace('\\', '/'); +#if LEGACY_DOTNET var pos = filename.IndexOf(':'); +#else + var pos = filename.IndexOf(':', StringComparison.Ordinal); +#endif if (pos >= 0) { filename = filename.Remove(0, pos + 1); @@ -75,6 +90,33 @@ public class TarWriter : AbstractWriter return filename.Trim('/'); } + private string NormalizeDirectoryName(string directoryName) + { + directoryName = NormalizeFilename(directoryName); + // Ensure directory name ends with '/' for tar format + if (!string.IsNullOrEmpty(directoryName) && !directoryName.EndsWith('/')) + { + directoryName += '/'; + } + return directoryName; + } + + public override void WriteDirectory(string directoryName, DateTime? modificationTime) + { + var normalizedName = NormalizeDirectoryName(directoryName); + if (string.IsNullOrEmpty(normalizedName)) + { + return; // Skip empty or root directory + } + + var header = new TarHeader(WriterOptions.ArchiveEncoding, _headerFormat); + header.LastModifiedTime = modificationTime ?? TarHeader.EPOCH; + header.Name = normalizedName; + header.Size = 0; + header.EntryType = EntryType.Directory; + header.Write(OutputStream.NotNull()); + } + public void Write(string filename, Stream source, DateTime? modificationTime, long? size) { if (!source.CanSeek && size is null) @@ -84,14 +126,18 @@ public class TarWriter : AbstractWriter var realSize = size ?? source.Length; - var header = new TarHeader(WriterOptions.ArchiveEncoding); + var header = new TarHeader(WriterOptions.ArchiveEncoding, _headerFormat); header.LastModifiedTime = modificationTime ?? TarHeader.EPOCH; header.Name = NormalizeFilename(filename); header.Size = realSize; - header.Write(OutputStream); - - size = source.TransferTo(OutputStream); + header.Write(OutputStream.NotNull()); + var progressStream = WrapWithProgress(source, filename); + size = progressStream.TransferTo( + OutputStream.NotNull(), + realSize, + WriterOptions.BufferSize + ); PadTo512(size.Value); } @@ -99,30 +145,23 @@ public class TarWriter : AbstractWriter { var zeros = unchecked((int)(((size + 511L) & ~511L) - size)); - OutputStream.Write(stackalloc byte[zeros]); + OutputStream.NotNull().Write(stackalloc byte[zeros]); } protected override void Dispose(bool isDisposing) { - if (isDisposing) + if (isDisposing && !_isDisposed) { - if (finalizeArchiveOnClose) + if (_finalizeArchiveOnClose) { - OutputStream.Write(stackalloc byte[1024]); + OutputStream.NotNull().Write(stackalloc byte[1024]); } - switch (OutputStream) + // Use IFinishable interface for generic finalization + if (OutputStream is IFinishable finishable) { - case BZip2Stream b: - { - b.Finish(); - break; - } - case LZipStream l: - { - l.Finish(); - break; - } + finishable.Finish(); } + _isDisposed = true; } base.Dispose(isDisposing); } diff --git a/src/SharpCompress/Writers/Tar/TarWriterOptions.cs b/src/SharpCompress/Writers/Tar/TarWriterOptions.cs index 8f2e866d..d7aaf22b 100755 --- a/src/SharpCompress/Writers/Tar/TarWriterOptions.cs +++ b/src/SharpCompress/Writers/Tar/TarWriterOptions.cs @@ -1,17 +1,137 @@ +using System; using SharpCompress.Common; +using SharpCompress.Common.Options; +using SharpCompress.Common.Tar.Headers; +using SharpCompress.Compressors; +using SharpCompress.Providers; namespace SharpCompress.Writers.Tar; -public class TarWriterOptions : WriterOptions +/// +/// Options for configuring Tar writer behavior. +/// +/// +/// Configure tar writing with constructors, property setters, or the with expression: +/// +/// var options = new TarWriterOptions(CompressionType.GZip, true); +/// options = options with { HeaderFormat = TarHeaderWriteFormat.V7 }; +/// +/// +public sealed record TarWriterOptions : IWriterOptions { + /// + /// The compression type to use for the archive. + /// + public CompressionType CompressionType { get; set; } + + /// + /// The compression level to be used when the compression type supports variable levels. + /// + public int CompressionLevel { get; set; } + + /// + /// SharpCompress will keep the supplied streams open. Default is true. + /// + public bool LeaveStreamOpen { get; set; } = true; + + /// + /// Encoding to use for archive entry names. + /// + public IArchiveEncoding ArchiveEncoding { get; set; } = new ArchiveEncoding(); + + /// + /// An optional progress reporter for tracking compression operations. + /// + public IProgress? Progress { get; set; } + + /// + /// Buffer size for writer stream copy operations. + /// + public int BufferSize { get; set; } = Constants.BufferSize; + + /// + /// Registry of compression providers. + /// Defaults to but can be replaced with custom implementations. + /// + public CompressionProviderRegistry Providers { get; set; } = + CompressionProviderRegistry.Default; + /// /// Indicates if archive should be finalized (by 2 empty blocks) on close. /// - public bool FinalizeArchiveOnClose { get; } + public bool FinalizeArchiveOnClose { get; set; } = true; + /// + /// The format to use when writing tar headers. + /// + public TarHeaderWriteFormat HeaderFormat { get; set; } = TarHeaderWriteFormat.GNU_TAR_LONG_LINK; + + /// + /// Creates a new TarWriterOptions instance with the specified compression type and finalization option. + /// + /// The compression type for the archive. + /// Whether to finalize the archive on close. public TarWriterOptions(CompressionType compressionType, bool finalizeArchiveOnClose) - : base(compressionType) => FinalizeArchiveOnClose = finalizeArchiveOnClose; + { + CompressionType = compressionType; + FinalizeArchiveOnClose = finalizeArchiveOnClose; + CompressionLevel = compressionType switch + { + CompressionType.ZStandard => 3, + _ => 0, + }; + } - internal TarWriterOptions(WriterOptions options) - : this(options.CompressionType, true) => ArchiveEncoding = options.ArchiveEncoding; + /// + /// Creates a new TarWriterOptions instance with the specified compression type, finalization option, and header format. + /// + /// The compression type for the archive. + /// Whether to finalize the archive on close. + /// The tar header format. + public TarWriterOptions( + CompressionType compressionType, + bool finalizeArchiveOnClose, + TarHeaderWriteFormat headerFormat + ) + : this(compressionType, finalizeArchiveOnClose) + { + HeaderFormat = headerFormat; + } + + /// + /// Creates a new TarWriterOptions instance from an existing WriterOptions instance. + /// + /// The WriterOptions to copy values from. + public TarWriterOptions(WriterOptions options) + { + CompressionType = options.CompressionType; + CompressionLevel = options.CompressionLevel; + LeaveStreamOpen = options.LeaveStreamOpen; + ArchiveEncoding = options.ArchiveEncoding; + Progress = options.Progress; + BufferSize = options.BufferSize; + Providers = options.Providers; + } + + /// + /// Creates a new TarWriterOptions instance from an existing IWriterOptions instance. + /// + /// The IWriterOptions to copy values from. + public TarWriterOptions(IWriterOptions options) + { + CompressionType = options.CompressionType; + CompressionLevel = options.CompressionLevel; + LeaveStreamOpen = options.LeaveStreamOpen; + ArchiveEncoding = options.ArchiveEncoding; + Progress = options.Progress; + BufferSize = options.BufferSize; + Providers = options.Providers; + } + + /// + /// Implicit conversion from CompressionType to TarWriterOptions with finalize enabled. + /// + /// The compression type. + public static implicit operator TarWriterOptions(CompressionType compressionType) => + new(compressionType, true); } diff --git a/src/SharpCompress/Writers/WriterFactory.cs b/src/SharpCompress/Writers/WriterFactory.cs index fcbf2d68..847766f9 100644 --- a/src/SharpCompress/Writers/WriterFactory.cs +++ b/src/SharpCompress/Writers/WriterFactory.cs @@ -1,22 +1,120 @@ using System; using System.IO; using System.Linq; - +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common; +using SharpCompress.Common.Options; namespace SharpCompress.Writers; public static class WriterFactory { - public static IWriter Open(Stream stream, ArchiveType archiveType, WriterOptions writerOptions) + public static IWriter OpenWriter( + string filePath, + ArchiveType archiveType, + IWriterOptions writerOptions + ) { - var factory = Factories.Factory.Factories - .OfType() + filePath.NotNullOrEmpty(nameof(filePath)); + return OpenWriter(new FileInfo(filePath), archiveType, writerOptions); + } + + public static IWriter OpenWriter( + FileInfo fileInfo, + ArchiveType archiveType, + IWriterOptions writerOptions + ) + { + fileInfo.NotNull(nameof(fileInfo)); + return OpenWriter( + fileInfo.OpenWrite(), + archiveType, + writerOptions.WithLeaveStreamOpen(false) + ); + } + + public static async ValueTask OpenAsyncWriter( + string filePath, + ArchiveType archiveType, + IWriterOptions writerOptions, + CancellationToken cancellationToken = default + ) + { + filePath.NotNullOrEmpty(nameof(filePath)); + return await OpenAsyncWriter( + new FileInfo(filePath), + archiveType, + writerOptions, + cancellationToken + ) + .ConfigureAwait(false); + } + + public static async ValueTask OpenAsyncWriter( + FileInfo fileInfo, + ArchiveType archiveType, + IWriterOptions writerOptions, + CancellationToken cancellationToken = default + ) + { + fileInfo.NotNull(nameof(fileInfo)); + var stream = fileInfo.OpenAsyncWriteStream(cancellationToken); + return await OpenAsyncWriter( + stream, + archiveType, + writerOptions.WithLeaveStreamOpen(false), + cancellationToken + ) + .ConfigureAwait(false); + } + + public static IWriter OpenWriter( + Stream stream, + ArchiveType archiveType, + IWriterOptions writerOptions + ) + { + stream.RequireWritable(); + + var factory = Factories + .Factory.Factories.OfType() .FirstOrDefault(item => item.KnownArchiveType == archiveType); if (factory != null) { - return factory.Open(stream, writerOptions); + return factory.OpenWriter(stream, writerOptions); + } + + throw new NotSupportedException("Archive Type does not have a Writer: " + archiveType); + } + + /// + /// Opens a Writer asynchronously. + /// + /// The stream to write to. + /// The archive type. + /// Writer options. + /// Cancellation token. + /// A containing the async writer. + public static async ValueTask OpenAsyncWriter( + Stream stream, + ArchiveType archiveType, + IWriterOptions writerOptions, + CancellationToken cancellationToken = default + ) + { + stream.RequireWritable(); + + var factory = Factories + .Factory.Factories.OfType() + .FirstOrDefault(item => item.KnownArchiveType == archiveType); + + if (factory != null) + { + return await factory + .OpenAsyncWriter(stream, writerOptions, cancellationToken) + .ConfigureAwait(false); } throw new NotSupportedException("Archive Type does not have a Writer: " + archiveType); diff --git a/src/SharpCompress/Writers/WriterOptions.cs b/src/SharpCompress/Writers/WriterOptions.cs index 9d323c2b..d70ea6aa 100644 --- a/src/SharpCompress/Writers/WriterOptions.cs +++ b/src/SharpCompress/Writers/WriterOptions.cs @@ -1,13 +1,132 @@ -using SharpCompress.Common; +using System; +using SharpCompress.Common; +using SharpCompress.Common.Options; +using SharpCompress.Providers; +using D = SharpCompress.Compressors.Deflate; namespace SharpCompress.Writers; -public class WriterOptions : OptionsBase +/// +/// Options for configuring writer behavior when creating archives. +/// +/// +/// Use factory methods, property setters, or fluent helpers for creation: +/// +/// var options = WriterOptions.ForZip().WithLeaveStreamOpen(false).WithCompressionLevel(9); +/// +/// +public sealed record WriterOptions : IWriterOptions { - public WriterOptions(CompressionType compressionType) => CompressionType = compressionType; - + /// + /// The compression type to use for the archive. + /// public CompressionType CompressionType { get; set; } + /// + /// The compression level to be used when the compression type supports variable levels. + /// Valid ranges depend on the compression algorithm: + /// - Deflate/GZip: 0-9 (0=no compression, 6=default, 9=best compression) + /// - ZStandard: 1-22 (1=fastest, 3=default, 22=best compression) + /// Note: BZip2 and LZMA do not support compression levels in this implementation. + /// Defaults are set automatically based on compression type in the constructor. + /// + public int CompressionLevel + { + get; + set + { + CompressionLevelValidation.Validate(CompressionType, value); + field = value; + } + } + + /// + /// SharpCompress will keep the supplied streams open. Default is true. + /// + public bool LeaveStreamOpen { get; set; } = true; + + /// + /// Encoding to use for archive entry names. + /// + public IArchiveEncoding ArchiveEncoding { get; set; } = new ArchiveEncoding(); + + /// + /// An optional progress reporter for tracking compression operations. + /// When set, progress updates will be reported as entries are written. + /// + public IProgress? Progress { get; set; } + + /// + /// Buffer size for writer stream copy operations. + /// + public int BufferSize { get; set; } = Constants.BufferSize; + + /// + /// Registry of compression providers. + /// Defaults to but can be replaced with custom implementations, such as + /// System.IO.Compression for Deflate/GZip on modern .NET. + /// + public CompressionProviderRegistry Providers { get; set; } = + CompressionProviderRegistry.Default; + + /// + /// Creates a new WriterOptions instance with the specified compression type. + /// Compression level is automatically set based on the compression type. + /// + /// The compression type for the archive. + public WriterOptions(CompressionType compressionType) + { + CompressionType = compressionType; + CompressionLevel = compressionType switch + { + CompressionType.ZStandard => 3, + CompressionType.Deflate => (int)D.CompressionLevel.Default, + CompressionType.Deflate64 => (int)D.CompressionLevel.Default, + CompressionType.GZip => (int)D.CompressionLevel.Default, + _ => 0, + }; + } + + /// + /// Creates a new WriterOptions instance with the specified compression type and level. + /// + /// The compression type for the archive. + /// The compression level (algorithm-specific). + public WriterOptions(CompressionType compressionType, int compressionLevel) + { + CompressionType = compressionType; + CompressionLevel = compressionLevel; + } + + // Note: Constructors with boolean leaveStreamOpen parameter removed. + // Use the fluent WithLeaveStreamOpen() helper or object initializer instead: + // new WriterOptions(type) { LeaveStreamOpen = false } + // or + // WriterOptions.ForZip().WithLeaveStreamOpen(false) + + /// + /// Implicit conversion from CompressionType to WriterOptions. + /// + /// The compression type. public static implicit operator WriterOptions(CompressionType compressionType) => - new WriterOptions(compressionType); + new(compressionType); + + /// + /// Creates a new ZipWriterOptions for writing ZIP archives. + /// + /// The compression type for the archive. Defaults to Deflate. + public static WriterOptions ForZip(CompressionType compressionType = CompressionType.Deflate) => + new(compressionType); + + /// + /// Creates a new WriterOptions for writing TAR archives. + /// + /// The compression type for the archive. Defaults to None. + public static WriterOptions ForTar(CompressionType compressionType = CompressionType.None) => + new(compressionType); + + /// + /// Creates a new WriterOptions for writing GZip compressed files. + /// + public static WriterOptions ForGZip() => new(CompressionType.GZip); } diff --git a/src/SharpCompress/Writers/WriterOptionsExtensions.cs b/src/SharpCompress/Writers/WriterOptionsExtensions.cs new file mode 100644 index 00000000..2aca567a --- /dev/null +++ b/src/SharpCompress/Writers/WriterOptionsExtensions.cs @@ -0,0 +1,110 @@ +using System; +using SharpCompress.Common; +using SharpCompress.Common.Options; +using SharpCompress.Compressors; +using SharpCompress.Providers; +using SharpCompress.Writers.GZip; +using SharpCompress.Writers.SevenZip; +using SharpCompress.Writers.Tar; +using SharpCompress.Writers.Zip; + +namespace SharpCompress.Writers; + +/// +/// Extension methods for fluent configuration of writer options. +/// +public static class WriterOptionsExtensions +{ + /// + /// Creates a copy with the specified LeaveStreamOpen value. + /// + /// The source options. + /// Whether to leave the stream open. + /// A new options instance with the specified LeaveStreamOpen value. + public static WriterOptions WithLeaveStreamOpen( + this WriterOptions options, + bool leaveStreamOpen + ) => options with { LeaveStreamOpen = leaveStreamOpen }; + + /// + /// Creates a copy with the specified LeaveStreamOpen value. + /// Works with any IWriterOptions implementation. + /// + /// The source options. + /// Whether to leave the stream open. + /// A new options instance with the specified LeaveStreamOpen value. + public static IWriterOptions WithLeaveStreamOpen( + this IWriterOptions options, + bool leaveStreamOpen + ) => + options switch + { + WriterOptions writerOptions => writerOptions with { LeaveStreamOpen = leaveStreamOpen }, + ZipWriterOptions zipOptions => zipOptions with { LeaveStreamOpen = leaveStreamOpen }, + TarWriterOptions tarOptions => tarOptions with { LeaveStreamOpen = leaveStreamOpen }, + GZipWriterOptions gzipOptions => gzipOptions with { LeaveStreamOpen = leaveStreamOpen }, + SevenZipWriterOptions sevenZipOptions => sevenZipOptions with + { + LeaveStreamOpen = leaveStreamOpen, + }, + _ => throw new NotSupportedException( + $"Cannot set LeaveStreamOpen on options of type {options.GetType().Name}. " + + "Options must be a record type implementing IWriterOptions." + ), + }; + + /// + /// Creates a copy with the specified buffer size. + /// + public static WriterOptions WithBufferSize(this WriterOptions options, int bufferSize) => + options with + { + BufferSize = bufferSize, + }; + + /// + /// Creates a copy with the specified compression level. + /// + /// The source options. + /// The compression level (algorithm-specific). + /// A new options instance with the specified compression level. + public static WriterOptions WithCompressionLevel( + this WriterOptions options, + int compressionLevel + ) => options with { CompressionLevel = compressionLevel }; + + /// + /// Creates a copy with the specified archive encoding. + /// + /// The source options. + /// The archive encoding to use. + /// A new options instance with the specified archive encoding. + public static WriterOptions WithArchiveEncoding( + this WriterOptions options, + IArchiveEncoding archiveEncoding + ) => options with { ArchiveEncoding = archiveEncoding }; + + /// + /// Creates a copy with the specified progress reporter. + /// + /// The source options. + /// The progress reporter. + /// A new options instance with the specified progress reporter. + public static WriterOptions WithProgress( + this WriterOptions options, + IProgress progress + ) => options with { Progress = progress }; + + /// + /// Creates a copy with the specified compression provider registry. + /// + /// Thrown if is null. + public static WriterOptions WithProviders( + this WriterOptions options, + CompressionProviderRegistry providers + ) + { + _ = providers ?? throw new ArgumentNullException(nameof(providers)); + return options with { Providers = providers }; + } +} diff --git a/src/SharpCompress/Writers/Zip/ZipCentralDirectoryEntry.cs b/src/SharpCompress/Writers/Zip/ZipCentralDirectoryEntry.cs index 111c7239..b866b535 100644 --- a/src/SharpCompress/Writers/Zip/ZipCentralDirectoryEntry.cs +++ b/src/SharpCompress/Writers/Zip/ZipCentralDirectoryEntry.cs @@ -12,13 +12,13 @@ internal class ZipCentralDirectoryEntry { private readonly ZipCompressionMethod compression; private readonly string fileName; - private readonly ArchiveEncoding archiveEncoding; + private readonly IArchiveEncoding archiveEncoding; public ZipCentralDirectoryEntry( ZipCompressionMethod compression, string fileName, ulong headerOffset, - ArchiveEncoding archiveEncoding + IArchiveEncoding archiveEncoding ) { this.compression = compression; @@ -48,7 +48,29 @@ internal class ZipCentralDirectoryEntry var decompressedvalue = zip64 ? uint.MaxValue : (uint)Decompressed; var headeroffsetvalue = zip64 ? uint.MaxValue : (uint)HeaderOffset; var extralength = zip64 ? (2 + 2 + 8 + 8 + 8 + 4) : 0; - var version = (byte)(zip64 ? 45 : 20); // Version 20 required for deflate/encryption + + // Determine version needed to extract: + // - Version 63 for LZMA, PPMd, BZip2, ZStandard (advanced compression methods) + // - Version 45 for Zip64 extensions (when Zip64HeaderOffset != 0 or actual sizes require it) + // - Version 20 for standard Deflate/None compression + byte version; + if ( + compression == ZipCompressionMethod.LZMA + || compression == ZipCompressionMethod.PPMd + || compression == ZipCompressionMethod.BZip2 + || compression == ZipCompressionMethod.ZStandard + ) + { + version = 63; + } + else if (zip64 || Zip64HeaderOffset != 0) + { + version = 45; + } + else + { + version = 20; + } var flags = Equals(archiveEncoding.GetEncoding(), Encoding.UTF8) ? HeaderFlags.Efs diff --git a/src/SharpCompress/Writers/Zip/ZipWriter.Async.cs b/src/SharpCompress/Writers/Zip/ZipWriter.Async.cs new file mode 100644 index 00000000..014a4806 --- /dev/null +++ b/src/SharpCompress/Writers/Zip/ZipWriter.Async.cs @@ -0,0 +1,214 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Zip; +using SharpCompress.Common.Zip.Headers; + +namespace SharpCompress.Writers.Zip; + +public partial class ZipWriter +{ + /// + /// Asynchronously disposes the writer, writing the ZIP central directory and end record. + /// + public override async ValueTask DisposeAsync() + { + if (_isDisposed) + { + return; + } + GC.SuppressFinalize(this); + _isDisposed = true; + + // Buffer the entire central directory + end record into memory, then write async. + // This avoids synchronous writes to the underlying stream during finalization. + using var ms = new MemoryStream(); + ulong size = 0; + foreach (var entry in entries) + { + size += entry.Write(ms); + } + WriteEndRecord(ms, size); + ms.Position = 0; + await ms.CopyToAsync(OutputStream.NotNull()).ConfigureAwait(false); + + if (OutputStream is IAsyncDisposable asyncDisposable) + { + await asyncDisposable.DisposeAsync().ConfigureAwait(false); + } + else + { + OutputStream?.Dispose(); + } + // base.DisposeAsync() is a no-op since _isDisposed is already set + await base.DisposeAsync().ConfigureAwait(false); + } + + /// + /// Asynchronously writes an entry to the ZIP archive. + /// + public override async ValueTask WriteAsync( + string filename, + Stream source, + DateTime? modificationTime, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + await WriteAsync( + filename, + source, + new ZipWriterEntryOptions { ModificationDateTime = modificationTime }, + cancellationToken + ) + .ConfigureAwait(false); + } + + /// + /// Asynchronously writes an entry to the ZIP archive with specified options. + /// + public async ValueTask WriteAsync( + string entryPath, + Stream source, + ZipWriterEntryOptions zipWriterEntryOptions, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + await using var output = await WriteToStreamAsync( + entryPath, + zipWriterEntryOptions, + cancellationToken + ) + .ConfigureAwait(false); + var progressStream = WrapWithProgress(source, entryPath); + await progressStream.CopyToAsync(output, 81920, cancellationToken).ConfigureAwait(false); + } + + private async ValueTask WriteToStreamAsync( + string entryPath, + ZipWriterEntryOptions options, + CancellationToken cancellationToken + ) + { + cancellationToken.ThrowIfCancellationRequested(); + options.ValidateWithFallback(compressionType, compressionLevel); + var compression = ToZipCompressionMethod(options.CompressionType ?? compressionType); + + entryPath = NormalizeFilename(entryPath); + options.ModificationDateTime ??= DateTime.Now; + options.EntryComment ??= string.Empty; + var entry = new ZipCentralDirectoryEntry( + compression, + entryPath, + (ulong)streamPosition, + WriterOptions.ArchiveEncoding + ) + { + Comment = options.EntryComment, + ModificationTime = options.ModificationDateTime, + }; + + var useZip64 = isZip64; + if (options.EnableZip64.HasValue) + { + useZip64 = options.EnableZip64.Value; + } + + var headersize = (uint) + await WriteHeaderAsync(entryPath, options, entry, useZip64, cancellationToken) + .ConfigureAwait(false); + streamPosition += headersize; + return await ZipWritingStream + .CreateAsync( + this, + OutputStream.NotNull(), + entry, + compression, + options.CompressionLevel ?? compressionLevel, + cancellationToken + ) + .ConfigureAwait(false); + } + + private async ValueTask WriteHeaderAsync( + string filename, + ZipWriterEntryOptions zipWriterEntryOptions, + ZipCentralDirectoryEntry entry, + bool useZip64, + CancellationToken cancellationToken + ) + { + // Build the header synchronously into a MemoryStream, then async-copy to OutputStream. + // This avoids any synchronous writes to the potentially async-only output stream. + using var ms = new MemoryStream(); + var result = WriteHeader(ms, filename, zipWriterEntryOptions, entry, useZip64); + ms.Position = 0; + await ms.CopyToAsync(OutputStream.NotNull(), 81920, cancellationToken) + .ConfigureAwait(false); + return result; + } + + /// + /// Asynchronously writes a directory entry to the ZIP archive. + /// Uses synchronous implementation for directory entries as they are lightweight. + /// + public override async ValueTask WriteDirectoryAsync( + string directoryName, + DateTime? modificationTime, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + + var normalizedName = NormalizeDirectoryName(directoryName); + if (string.IsNullOrEmpty(normalizedName)) + { + return; + } + + var options = new ZipWriterEntryOptions { ModificationDateTime = modificationTime }; + await WriteDirectoryEntryAsync(normalizedName, options, cancellationToken) + .ConfigureAwait(false); + } + + private async ValueTask WriteDirectoryEntryAsync( + string directoryPath, + ZipWriterEntryOptions options, + CancellationToken cancellationToken + ) + { + var compression = ZipCompressionMethod.None; + + options.ModificationDateTime ??= DateTime.Now; + options.EntryComment ??= string.Empty; + + var entry = new ZipCentralDirectoryEntry( + compression, + directoryPath, + (ulong)streamPosition, + WriterOptions.ArchiveEncoding + ) + { + Comment = options.EntryComment, + ModificationTime = options.ModificationDateTime, + Crc = 0, + Compressed = 0, + Decompressed = 0, + }; + + var useZip64 = isZip64; + if (options.EnableZip64.HasValue) + { + useZip64 = options.EnableZip64.Value; + } + + var headersize = (uint) + await WriteHeaderAsync(directoryPath, options, entry, useZip64, cancellationToken) + .ConfigureAwait(false); + streamPosition += headersize; + entries.Add(entry); + } +} diff --git a/src/SharpCompress/Writers/Zip/ZipWriter.Factory.cs b/src/SharpCompress/Writers/Zip/ZipWriter.Factory.cs new file mode 100644 index 00000000..825de870 --- /dev/null +++ b/src/SharpCompress/Writers/Zip/ZipWriter.Factory.cs @@ -0,0 +1,59 @@ +#if NET8_0_OR_GREATER +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; + +namespace SharpCompress.Writers.Zip; + +public partial class ZipWriter : IWriterOpenable +{ + public static IWriter OpenWriter(string filePath, ZipWriterOptions writerOptions) + { + filePath.NotNullOrEmpty(nameof(filePath)); + return OpenWriter(new FileInfo(filePath), writerOptions); + } + + public static IWriter OpenWriter(FileInfo fileInfo, ZipWriterOptions writerOptions) + { + fileInfo.NotNull(nameof(fileInfo)); + return new ZipWriter(fileInfo.OpenWrite(), writerOptions with { LeaveStreamOpen = false }); + } + + public static IWriter OpenWriter(Stream stream, ZipWriterOptions writerOptions) + { + stream.RequireWritable(); + return new ZipWriter(stream, writerOptions); + } + + public static ValueTask OpenAsyncWriter( + string filePath, + ZipWriterOptions writerOptions, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncWriter)OpenWriter(filePath, writerOptions)); + } + + public static ValueTask OpenAsyncWriter( + Stream stream, + ZipWriterOptions writerOptions, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncWriter)OpenWriter(stream, writerOptions)); + } + + public static ValueTask OpenAsyncWriter( + FileInfo fileInfo, + ZipWriterOptions writerOptions, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new((IAsyncWriter)OpenWriter(fileInfo, writerOptions)); + } +} +#endif diff --git a/src/SharpCompress/Writers/Zip/ZipWriter.cs b/src/SharpCompress/Writers/Zip/ZipWriter.cs index e1fd222c..128cb9d7 100644 --- a/src/SharpCompress/Writers/Zip/ZipWriter.cs +++ b/src/SharpCompress/Writers/Zip/ZipWriter.cs @@ -3,6 +3,8 @@ using System.Buffers.Binary; using System.Collections.Generic; using System.IO; using System.Text; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common; using SharpCompress.Common.Zip; using SharpCompress.Common.Zip.Headers; @@ -11,15 +13,17 @@ using SharpCompress.Compressors.BZip2; using SharpCompress.Compressors.Deflate; using SharpCompress.Compressors.LZMA; using SharpCompress.Compressors.PPMd; +using SharpCompress.Compressors.ZStandard; using SharpCompress.IO; +using SharpCompress.Providers; namespace SharpCompress.Writers.Zip; -public class ZipWriter : AbstractWriter +public partial class ZipWriter : AbstractWriter { private readonly CompressionType compressionType; - private readonly CompressionLevel compressionLevel; - private readonly List entries = new List(); + private readonly int compressionLevel; + private readonly List entries = new(); private readonly string zipComment; private long streamPosition; private PpmdProperties? ppmdProps; @@ -36,63 +40,46 @@ public class ZipWriter : AbstractWriter } compressionType = zipWriterOptions.CompressionType; - compressionLevel = zipWriterOptions.DeflateCompressionLevel; + compressionLevel = zipWriterOptions.CompressionLevel; if (WriterOptions.LeaveStreamOpen) { - destination = NonDisposingStream.Create(destination); + destination = SharpCompressStream.CreateNonDisposing(destination); } - InitalizeStream(destination); + InitializeStream(destination); } private PpmdProperties PpmdProperties => ppmdProps ??= new PpmdProperties(); protected override void Dispose(bool isDisposing) { - if (isDisposing) + if (isDisposing && !_isDisposed) { ulong size = 0; foreach (var entry in entries) { - size += entry.Write(OutputStream); + size += entry.Write(OutputStream.NotNull()); } WriteEndRecord(size); } base.Dispose(isDisposing); } - private static ZipCompressionMethod ToZipCompressionMethod(CompressionType compressionType) - { - switch (compressionType) + private static ZipCompressionMethod ToZipCompressionMethod(CompressionType compressionType) => + compressionType switch { - case CompressionType.None: - { - return ZipCompressionMethod.None; - } - case CompressionType.Deflate: - { - return ZipCompressionMethod.Deflate; - } - case CompressionType.BZip2: - { - return ZipCompressionMethod.BZip2; - } - case CompressionType.LZMA: - { - return ZipCompressionMethod.LZMA; - } - case CompressionType.PPMd: - { - return ZipCompressionMethod.PPMd; - } - default: - throw new InvalidFormatException("Invalid compression method: " + compressionType); - } - } + CompressionType.None => ZipCompressionMethod.None, + CompressionType.Deflate => ZipCompressionMethod.Deflate, + CompressionType.BZip2 => ZipCompressionMethod.BZip2, + CompressionType.LZMA => ZipCompressionMethod.LZMA, + CompressionType.PPMd => ZipCompressionMethod.PPMd, + CompressionType.ZStandard => ZipCompressionMethod.ZStandard, + _ => throw new InvalidFormatException("Invalid compression method: " + compressionType), + }; - public override void Write(string entryPath, Stream source, DateTime? modificationTime) => + public override void Write(string filename, Stream source, DateTime? modificationTime) => Write( - entryPath, + filename, source, new ZipWriterEntryOptions() { ModificationDateTime = modificationTime } ); @@ -100,11 +87,13 @@ public class ZipWriter : AbstractWriter public void Write(string entryPath, Stream source, ZipWriterEntryOptions zipWriterEntryOptions) { using var output = WriteToStream(entryPath, zipWriterEntryOptions); - source.TransferTo(output); + var progressStream = WrapWithProgress(source, entryPath); + progressStream.CopyTo(output, WriterOptions.BufferSize); } public Stream WriteToStream(string entryPath, ZipWriterEntryOptions options) { + options.ValidateWithFallback(compressionType, compressionLevel); var compression = ToZipCompressionMethod(options.CompressionType ?? compressionType); entryPath = NormalizeFilename(entryPath); @@ -118,7 +107,7 @@ public class ZipWriter : AbstractWriter ) { Comment = options.EntryComment, - ModificationTime = options.ModificationDateTime + ModificationTime = options.ModificationDateTime, }; // Use the archive default setting for zip64 and allow overrides @@ -132,10 +121,10 @@ public class ZipWriter : AbstractWriter streamPosition += headersize; return new ZipWritingStream( this, - OutputStream, + OutputStream.NotNull(), entry, compression, - options.DeflateCompressionLevel ?? compressionLevel + options.CompressionLevel ?? compressionLevel ); } @@ -143,7 +132,11 @@ public class ZipWriter : AbstractWriter { filename = filename.Replace('\\', '/'); +#if LEGACY_DOTNET var pos = filename.IndexOf(':'); +#else + var pos = filename.IndexOf(':', StringComparison.Ordinal); +#endif if (pos >= 0) { filename = filename.Remove(0, pos + 1); @@ -152,15 +145,81 @@ public class ZipWriter : AbstractWriter return filename.Trim('/'); } + private string NormalizeDirectoryName(string directoryName) + { + directoryName = NormalizeFilename(directoryName); + // Ensure directory name ends with '/' for zip format + if (!string.IsNullOrEmpty(directoryName) && !directoryName.EndsWith('/')) + { + directoryName += '/'; + } + return directoryName; + } + + public override void WriteDirectory(string directoryName, DateTime? modificationTime) + { + var normalizedName = NormalizeDirectoryName(directoryName); + if (string.IsNullOrEmpty(normalizedName)) + { + return; // Skip empty or root directory + } + + var options = new ZipWriterEntryOptions { ModificationDateTime = modificationTime }; + WriteDirectoryEntry(normalizedName, options); + } + + // WriteDirectoryAsync moved to ZipWriter.Async.cs + + private void WriteDirectoryEntry(string directoryPath, ZipWriterEntryOptions options) + { + var compression = ZipCompressionMethod.None; + + options.ModificationDateTime ??= DateTime.Now; + options.EntryComment ??= string.Empty; + + var entry = new ZipCentralDirectoryEntry( + compression, + directoryPath, + (ulong)streamPosition, + WriterOptions.ArchiveEncoding + ) + { + Comment = options.EntryComment, + ModificationTime = options.ModificationDateTime, + Crc = 0, + Compressed = 0, + Decompressed = 0, + }; + + // Use the archive default setting for zip64 and allow overrides + var useZip64 = isZip64; + if (options.EnableZip64.HasValue) + { + useZip64 = options.EnableZip64.Value; + } + + var headersize = (uint)WriteHeader(directoryPath, options, entry, useZip64); + streamPosition += headersize; + entries.Add(entry); + } + private int WriteHeader( string filename, ZipWriterEntryOptions zipWriterEntryOptions, ZipCentralDirectoryEntry entry, bool useZip64 + ) => WriteHeader(OutputStream.NotNull(), filename, zipWriterEntryOptions, entry, useZip64); + + private int WriteHeader( + Stream stream, + string filename, + ZipWriterEntryOptions zipWriterEntryOptions, + ZipCentralDirectoryEntry entry, + bool useZip64 ) { // We err on the side of caution until the zip specification clarifies how to support this - if (!OutputStream.CanSeek && useZip64) + if (!stream.CanSeek && useZip64) { throw new NotSupportedException( "Zip64 extensions are not supported on non-seekable streams" @@ -174,26 +233,26 @@ public class ZipWriter : AbstractWriter Span intBuf = stackalloc byte[4]; BinaryPrimitives.WriteUInt32LittleEndian(intBuf, ZipHeaderFactory.ENTRY_HEADER_BYTES); - OutputStream.Write(intBuf); + stream.Write(intBuf); if (explicitZipCompressionInfo == ZipCompressionMethod.Deflate) { - if (OutputStream.CanSeek && useZip64) + if (stream.CanSeek && useZip64) { - OutputStream.Write(stackalloc byte[] { 45, 0 }); //smallest allowed version for zip64 + stream.Write(stackalloc byte[] { 45, 0 }); //smallest allowed version for zip64 } else { - OutputStream.Write(stackalloc byte[] { 20, 0 }); //older version which is more compatible + stream.Write(stackalloc byte[] { 20, 0 }); //older version which is more compatible } } else { - OutputStream.Write(stackalloc byte[] { 63, 0 }); //version says we used PPMd or LZMA + stream.Write(stackalloc byte[] { 63, 0 }); //version says we used PPMd or LZMA } var flags = Equals(WriterOptions.ArchiveEncoding.GetEncoding(), Encoding.UTF8) ? HeaderFlags.Efs : 0; - if (!OutputStream.CanSeek) + if (!stream.CanSeek) { flags |= HeaderFlags.UsePostDataDescriptor; @@ -204,53 +263,58 @@ public class ZipWriter : AbstractWriter } BinaryPrimitives.WriteUInt16LittleEndian(intBuf, (ushort)flags); - OutputStream.Write(intBuf.Slice(0, 2)); + stream.Write(intBuf.Slice(0, 2)); BinaryPrimitives.WriteUInt16LittleEndian(intBuf, (ushort)explicitZipCompressionInfo); - OutputStream.Write(intBuf.Slice(0, 2)); // zipping method + stream.Write(intBuf.Slice(0, 2)); // zipping method BinaryPrimitives.WriteUInt32LittleEndian( intBuf, zipWriterEntryOptions.ModificationDateTime.DateTimeToDosTime() ); - OutputStream.Write(intBuf); + stream.Write(intBuf); // zipping date and time - OutputStream.Write(stackalloc byte[] { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }); + stream.Write(stackalloc byte[] { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }); // unused CRC, un/compressed size, updated later BinaryPrimitives.WriteUInt16LittleEndian(intBuf, (ushort)encodedFilename.Length); - OutputStream.Write(intBuf.Slice(0, 2)); // filename length + stream.Write(intBuf.Slice(0, 2)); // filename length var extralength = 0; - if (OutputStream.CanSeek && useZip64) + if (stream.CanSeek && useZip64) { extralength = 2 + 2 + 8 + 8; } BinaryPrimitives.WriteUInt16LittleEndian(intBuf, (ushort)extralength); - OutputStream.Write(intBuf.Slice(0, 2)); // extra length - OutputStream.Write(encodedFilename, 0, encodedFilename.Length); + stream.Write(intBuf.Slice(0, 2)); // extra length + stream.Write(encodedFilename, 0, encodedFilename.Length); if (extralength != 0) { - OutputStream.Write(new byte[extralength], 0, extralength); // reserve space for zip64 data + stream.Write(new byte[extralength], 0, extralength); // reserve space for zip64 data entry.Zip64HeaderOffset = (ushort)(6 + 2 + 2 + 4 + 12 + 2 + 2 + encodedFilename.Length); } return 6 + 2 + 2 + 4 + 12 + 2 + 2 + encodedFilename.Length + extralength; } - private void WriteFooter(uint crc, uint compressed, uint uncompressed) + private void WriteFooter(uint crc, uint compressed, uint uncompressed) => + WriteFooter(OutputStream.NotNull(), crc, compressed, uncompressed); + + private static void WriteFooter(Stream stream, uint crc, uint compressed, uint uncompressed) { Span intBuf = stackalloc byte[4]; BinaryPrimitives.WriteUInt32LittleEndian(intBuf, crc); - OutputStream.Write(intBuf); + stream.Write(intBuf); BinaryPrimitives.WriteUInt32LittleEndian(intBuf, compressed); - OutputStream.Write(intBuf); + stream.Write(intBuf); BinaryPrimitives.WriteUInt32LittleEndian(intBuf, uncompressed); - OutputStream.Write(intBuf); + stream.Write(intBuf); } - private void WriteEndRecord(ulong size) + private void WriteEndRecord(ulong size) => WriteEndRecord(OutputStream.NotNull(), size); + + private void WriteEndRecord(Stream stream, ulong size) { var zip64EndOfCentralDirectoryNeeded = entries.Count > ushort.MaxValue @@ -267,308 +331,56 @@ public class ZipWriter : AbstractWriter var recordlen = 2 + 2 + 4 + 4 + 8 + 8 + 8 + 8; // Write zip64 end of central directory record - OutputStream.Write(stackalloc byte[] { 80, 75, 6, 6 }); + stream.Write(stackalloc byte[] { 80, 75, 6, 6 }); BinaryPrimitives.WriteUInt64LittleEndian(intBuf, (ulong)recordlen); - OutputStream.Write(intBuf); // Size of zip64 end of central directory record + stream.Write(intBuf); // Size of zip64 end of central directory record BinaryPrimitives.WriteUInt16LittleEndian(intBuf, 45); - OutputStream.Write(intBuf.Slice(0, 2)); // Made by + stream.Write(intBuf.Slice(0, 2)); // Made by BinaryPrimitives.WriteUInt16LittleEndian(intBuf, 45); - OutputStream.Write(intBuf.Slice(0, 2)); // Version needed + stream.Write(intBuf.Slice(0, 2)); // Version needed BinaryPrimitives.WriteUInt32LittleEndian(intBuf, 0); - OutputStream.Write(intBuf.Slice(0, 4)); // Disk number - OutputStream.Write(intBuf.Slice(0, 4)); // Central dir disk + stream.Write(intBuf.Slice(0, 4)); // Disk number + stream.Write(intBuf.Slice(0, 4)); // Central dir disk // TODO: entries.Count is int, so max 2^31 files BinaryPrimitives.WriteUInt64LittleEndian(intBuf, (ulong)entries.Count); - OutputStream.Write(intBuf); // Entries in this disk - OutputStream.Write(intBuf); // Total entries + stream.Write(intBuf); // Entries in this disk + stream.Write(intBuf); // Total entries BinaryPrimitives.WriteUInt64LittleEndian(intBuf, size); - OutputStream.Write(intBuf); // Central Directory size + stream.Write(intBuf); // Central Directory size BinaryPrimitives.WriteUInt64LittleEndian(intBuf, (ulong)streamPosition); - OutputStream.Write(intBuf); // Disk offset + stream.Write(intBuf); // Disk offset // Write zip64 end of central directory locator - OutputStream.Write(stackalloc byte[] { 80, 75, 6, 7 }); + stream.Write(stackalloc byte[] { 80, 75, 6, 7 }); BinaryPrimitives.WriteUInt32LittleEndian(intBuf, 0); - OutputStream.Write(intBuf.Slice(0, 4)); // Entry disk + stream.Write(intBuf.Slice(0, 4)); // Entry disk BinaryPrimitives.WriteUInt64LittleEndian(intBuf, (ulong)streamPosition + size); - OutputStream.Write(intBuf); // Offset to the zip64 central directory + stream.Write(intBuf); // Offset to the zip64 central directory BinaryPrimitives.WriteUInt32LittleEndian(intBuf, 1); - OutputStream.Write(intBuf.Slice(0, 4)); // Number of disks + stream.Write(intBuf.Slice(0, 4)); // Number of disks streamPosition += 4 + 8 + recordlen + (4 + 4 + 8 + 4); } // Write normal end of central directory record - OutputStream.Write(stackalloc byte[] { 80, 75, 5, 6, 0, 0, 0, 0 }); - BinaryPrimitives.WriteUInt16LittleEndian(intBuf, (ushort)entries.Count); - OutputStream.Write(intBuf.Slice(0, 2)); - OutputStream.Write(intBuf.Slice(0, 2)); + stream.Write(stackalloc byte[] { 80, 75, 5, 6, 0, 0, 0, 0 }); + BinaryPrimitives.WriteUInt16LittleEndian( + intBuf, + (ushort)(entries.Count < 0xFFFF ? entries.Count : 0xFFFF) + ); + stream.Write(intBuf.Slice(0, 2)); + stream.Write(intBuf.Slice(0, 2)); BinaryPrimitives.WriteUInt32LittleEndian(intBuf, sizevalue); - OutputStream.Write(intBuf.Slice(0, 4)); + stream.Write(intBuf.Slice(0, 4)); BinaryPrimitives.WriteUInt32LittleEndian(intBuf, streampositionvalue); - OutputStream.Write(intBuf.Slice(0, 4)); + stream.Write(intBuf.Slice(0, 4)); var encodedComment = WriterOptions.ArchiveEncoding.Encode(zipComment); BinaryPrimitives.WriteUInt16LittleEndian(intBuf, (ushort)encodedComment.Length); - OutputStream.Write(intBuf.Slice(0, 2)); - OutputStream.Write(encodedComment, 0, encodedComment.Length); + stream.Write(intBuf.Slice(0, 2)); + stream.Write(encodedComment, 0, encodedComment.Length); } - - #region Nested type: ZipWritingStream - - internal class ZipWritingStream : Stream - { - private readonly CRC32 crc = new CRC32(); - private readonly ZipCentralDirectoryEntry entry; - private readonly Stream originalStream; - private readonly Stream writeStream; - private readonly ZipWriter writer; - private readonly ZipCompressionMethod zipCompressionMethod; - private readonly CompressionLevel compressionLevel; - private CountingWritableSubStream? counting; - private ulong decompressed; - - // Flag to prevent throwing exceptions on Dispose - private bool limitsExceeded; - private bool isDisposed; - - internal ZipWritingStream( - ZipWriter writer, - Stream originalStream, - ZipCentralDirectoryEntry entry, - ZipCompressionMethod zipCompressionMethod, - CompressionLevel compressionLevel - ) - { - this.writer = writer; - this.originalStream = originalStream; - this.writer = writer; - this.entry = entry; - this.zipCompressionMethod = zipCompressionMethod; - this.compressionLevel = compressionLevel; - writeStream = GetWriteStream(originalStream); - } - - public override bool CanRead => false; - - public override bool CanSeek => false; - - public override bool CanWrite => true; - - public override long Length => throw new NotSupportedException(); - - public override long Position - { - get => throw new NotSupportedException(); - set => throw new NotSupportedException(); - } - - private Stream GetWriteStream(Stream writeStream) - { - counting = new CountingWritableSubStream(writeStream); - Stream output = counting; - switch (zipCompressionMethod) - { - case ZipCompressionMethod.None: - { - return output; - } - case ZipCompressionMethod.Deflate: - { - return new DeflateStream(counting, CompressionMode.Compress, compressionLevel); - } - case ZipCompressionMethod.BZip2: - { - return new BZip2Stream(counting, CompressionMode.Compress, false); - } - case ZipCompressionMethod.LZMA: - { - counting.WriteByte(9); - counting.WriteByte(20); - counting.WriteByte(5); - counting.WriteByte(0); - - var lzmaStream = new LzmaStream( - new LzmaEncoderProperties(!originalStream.CanSeek), - false, - counting - ); - counting.Write(lzmaStream.Properties, 0, lzmaStream.Properties.Length); - return lzmaStream; - } - case ZipCompressionMethod.PPMd: - { - counting.Write(writer.PpmdProperties.Properties, 0, 2); - return new PpmdStream(writer.PpmdProperties, counting, true); - } - default: - { - throw new NotSupportedException("CompressionMethod: " + zipCompressionMethod); - } - } - } - - protected override void Dispose(bool disposing) - { - if (isDisposed) - { - return; - } - - isDisposed = true; - - base.Dispose(disposing); - if (disposing) - { - writeStream.Dispose(); - - if (limitsExceeded) - { - // We have written invalid data into the archive, - // so we destroy it now, instead of allowing the user to continue - // with a defunct archive - originalStream.Dispose(); - return; - } - - entry.Crc = (uint)crc.Crc32Result; - entry.Compressed = counting!.Count; - entry.Decompressed = decompressed; - - var zip64 = - entry.Compressed >= uint.MaxValue || entry.Decompressed >= uint.MaxValue; - var compressedvalue = zip64 ? uint.MaxValue : (uint)counting.Count; - var decompressedvalue = zip64 ? uint.MaxValue : (uint)entry.Decompressed; - - if (originalStream.CanSeek) - { - originalStream.Position = (long)(entry.HeaderOffset + 6); - originalStream.WriteByte(0); - - if (counting.Count == 0 && entry.Decompressed == 0) - { - // set compression to STORED for zero byte files (no compression data) - originalStream.Position = (long)(entry.HeaderOffset + 8); - originalStream.WriteByte(0); - originalStream.WriteByte(0); - } - - originalStream.Position = (long)(entry.HeaderOffset + 14); - - writer.WriteFooter(entry.Crc, compressedvalue, decompressedvalue); - - // Ideally, we should not throw from Dispose() - // We should not get here as the Write call checks the limits - if (zip64 && entry.Zip64HeaderOffset == 0) - { - throw new NotSupportedException( - "Attempted to write a stream that is larger than 4GiB without setting the zip64 option" - ); - } - - // If we have pre-allocated space for zip64 data, - // fill it out, even if it is not required - if (entry.Zip64HeaderOffset != 0) - { - originalStream.Position = (long)( - entry.HeaderOffset + entry.Zip64HeaderOffset - ); - Span intBuf = stackalloc byte[8]; - BinaryPrimitives.WriteUInt16LittleEndian(intBuf, 0x0001); - originalStream.Write(intBuf.Slice(0, 2)); - BinaryPrimitives.WriteUInt16LittleEndian(intBuf, 8 + 8); - originalStream.Write(intBuf.Slice(0, 2)); - - BinaryPrimitives.WriteUInt64LittleEndian(intBuf, entry.Decompressed); - originalStream.Write(intBuf); - BinaryPrimitives.WriteUInt64LittleEndian(intBuf, entry.Compressed); - originalStream.Write(intBuf); - } - - originalStream.Position = writer.streamPosition + (long)entry.Compressed; - writer.streamPosition += (long)entry.Compressed; - } - else - { - // We have a streaming archive, so we should add a post-data-descriptor, - // but we cannot as it does not hold the zip64 values - // Throwing an exception until the zip specification is clarified - - // Ideally, we should not throw from Dispose() - // We should not get here as the Write call checks the limits - if (zip64) - { - throw new NotSupportedException( - "Streams larger than 4GiB are not supported for non-seekable streams" - ); - } - - Span intBuf = stackalloc byte[4]; - BinaryPrimitives.WriteUInt32LittleEndian( - intBuf, - ZipHeaderFactory.POST_DATA_DESCRIPTOR - ); - originalStream.Write(intBuf); - writer.WriteFooter(entry.Crc, compressedvalue, decompressedvalue); - writer.streamPosition += (long)entry.Compressed + 16; - } - writer.entries.Add(entry); - } - } - - public override void Flush() => writeStream.Flush(); - - public override int Read(byte[] buffer, int offset, int count) => - throw new NotSupportedException(); - - public override long Seek(long offset, SeekOrigin origin) => - throw new NotSupportedException(); - - public override void SetLength(long value) => throw new NotSupportedException(); - - public override void Write(byte[] buffer, int offset, int count) - { - // We check the limits first, because we can keep the archive consistent - // if we can prevent the writes from happening - if (entry.Zip64HeaderOffset == 0) - { - // Pre-check, the counting.Count is not exact, as we do not know the size before having actually compressed it - if ( - limitsExceeded - || ((decompressed + (uint)count) > uint.MaxValue) - || (counting!.Count + (uint)count) > uint.MaxValue - ) - { - throw new NotSupportedException( - "Attempted to write a stream that is larger than 4GiB without setting the zip64 option" - ); - } - } - - decompressed += (uint)count; - crc.SlurpBlock(buffer, offset, count); - writeStream.Write(buffer, offset, count); - - if (entry.Zip64HeaderOffset == 0) - { - // Post-check, this is accurate - if ((decompressed > uint.MaxValue) || counting!.Count > uint.MaxValue) - { - // We have written the data, so the archive is now broken - // Throwing the exception here, allows us to avoid - // throwing an exception in Dispose() which is discouraged - // as it can mask other errors - limitsExceeded = true; - throw new NotSupportedException( - "Attempted to write a stream that is larger than 4GiB without setting the zip64 option" - ); - } - } - } - } - - #endregion Nested type: ZipWritingStream } diff --git a/src/SharpCompress/Writers/Zip/ZipWriterEntryOptions.cs b/src/SharpCompress/Writers/Zip/ZipWriterEntryOptions.cs index a718f3b5..ef601c21 100644 --- a/src/SharpCompress/Writers/Zip/ZipWriterEntryOptions.cs +++ b/src/SharpCompress/Writers/Zip/ZipWriterEntryOptions.cs @@ -1,17 +1,48 @@ using System; using SharpCompress.Common; using SharpCompress.Compressors.Deflate; +using SharpCompress.Writers; namespace SharpCompress.Writers.Zip; public class ZipWriterEntryOptions { - public CompressionType? CompressionType { get; set; } + private CompressionType? compressionType; + private int? compressionLevel; + + public CompressionType? CompressionType + { + get => compressionType; + set + { + if (value.HasValue && compressionLevel.HasValue) + { + CompressionLevelValidation.Validate(value.Value, compressionLevel.Value); + } + compressionType = value; + } + } /// - /// When CompressionType.Deflate is used, this property is referenced. Defaults to CompressionLevel.Default. + /// The compression level to be used when the compression type supports variable levels. + /// Valid ranges depend on the compression algorithm: + /// - Deflate/GZip: 0-9 (0=no compression, 6=default, 9=best compression) + /// - ZStandard: 1-22 (1=fastest, 3=default, 22=best compression) + /// When null, uses the archive's default compression level for the specified compression type. + /// Note: BZip2 and LZMA do not support compression levels in this implementation. /// - public CompressionLevel? DeflateCompressionLevel { get; set; } + public int? CompressionLevel + { + get => compressionLevel; + set + { + if (value.HasValue && compressionType.HasValue) + { + CompressionLevelValidation.Validate(compressionType.Value, value.Value); + } + compressionLevel = value; + } + } public string? EntryComment { get; set; } @@ -24,4 +55,12 @@ public class ZipWriterEntryOptions /// This option is not supported with non-seekable streams. /// public bool? EnableZip64 { get; set; } + + internal void ValidateWithFallback(CompressionType fallbackCompressionType, int fallbackLevel) + { + CompressionLevelValidation.Validate( + CompressionType ?? fallbackCompressionType, + CompressionLevel ?? fallbackLevel + ); + } } diff --git a/src/SharpCompress/Writers/Zip/ZipWriterOptions.cs b/src/SharpCompress/Writers/Zip/ZipWriterOptions.cs index fcbc9db3..73c92ee6 100644 --- a/src/SharpCompress/Writers/Zip/ZipWriterOptions.cs +++ b/src/SharpCompress/Writers/Zip/ZipWriterOptions.cs @@ -1,32 +1,81 @@ +using System; using SharpCompress.Common; +using SharpCompress.Common.Options; +using SharpCompress.Compressors; using SharpCompress.Compressors.Deflate; +using SharpCompress.Providers; +using SharpCompress.Writers; +using D = SharpCompress.Compressors.Deflate; namespace SharpCompress.Writers.Zip; -public class ZipWriterOptions : WriterOptions +/// +/// Options for configuring Zip writer behavior. +/// +/// +/// Configure zip writing with constructors, property setters, or the with expression: +/// +/// var options = new ZipWriterOptions(CompressionType.Zip); +/// options = options with { UseZip64 = true }; +/// +/// +public sealed record ZipWriterOptions : IWriterOptions { - public ZipWriterOptions(CompressionType compressionType) - : base(compressionType) { } + private CompressionType _compressionType; + private int _compressionLevel; - internal ZipWriterOptions(WriterOptions options) - : base(options.CompressionType) + /// + /// The compression type to use for the archive. + /// + public CompressionType CompressionType { - LeaveStreamOpen = options.LeaveStreamOpen; - ArchiveEncoding = options.ArchiveEncoding; + get => _compressionType; + set => _compressionType = value; + } - if (options is ZipWriterOptions writerOptions) + /// + /// The compression level to be used when the compression type supports variable levels. + /// + public int CompressionLevel + { + get => _compressionLevel; + set { - UseZip64 = writerOptions.UseZip64; - DeflateCompressionLevel = writerOptions.DeflateCompressionLevel; - ArchiveComment = writerOptions.ArchiveComment; + CompressionLevelValidation.Validate(CompressionType, value); + _compressionLevel = value; } } /// - /// When CompressionType.Deflate is used, this property is referenced. Defaults to CompressionLevel.Default. + /// SharpCompress will keep the supplied streams open. Default is true. /// - public CompressionLevel DeflateCompressionLevel { get; set; } = CompressionLevel.Default; + public bool LeaveStreamOpen { get; set; } = true; + /// + /// Encoding to use for archive entry names. + /// + public IArchiveEncoding ArchiveEncoding { get; set; } = new ArchiveEncoding(); + + /// + /// An optional progress reporter for tracking compression operations. + /// + public IProgress? Progress { get; set; } + + /// + /// Buffer size for writer stream copy operations. + /// + public int BufferSize { get; set; } = Constants.BufferSize; + + /// + /// Registry of compression providers. + /// Defaults to but can be replaced with custom implementations. + /// + public CompressionProviderRegistry Providers { get; set; } = + CompressionProviderRegistry.Default; + + /// + /// Optional comment for the archive. + /// public string? ArchiveComment { get; set; } /// @@ -37,4 +86,75 @@ public class ZipWriterOptions : WriterOptions /// are less than 4GiB in length. /// public bool UseZip64 { get; set; } + + /// + /// Creates a new ZipWriterOptions instance with the specified compression type. + /// + /// The compression type for the archive. + public ZipWriterOptions(CompressionType compressionType) + { + CompressionType = compressionType; + CompressionLevel = compressionType switch + { + CompressionType.ZStandard => 3, + CompressionType.Deflate => (int)D.CompressionLevel.Default, + CompressionType.Deflate64 => (int)D.CompressionLevel.Default, + CompressionType.GZip => (int)D.CompressionLevel.Default, + _ => 0, + }; + } + + /// + /// Creates a new ZipWriterOptions instance with the specified compression type and level. + /// + /// The compression type for the archive. + /// The compression level (algorithm-specific). + public ZipWriterOptions(CompressionType compressionType, int compressionLevel) + { + CompressionType = compressionType; + CompressionLevel = compressionLevel; + } + + /// + /// Creates a new ZipWriterOptions instance with the specified compression type and Deflate compression level. + /// + /// The compression type for the archive. + /// The Deflate compression level. + public ZipWriterOptions(CompressionType compressionType, D.CompressionLevel compressionLevel) + : this(compressionType, (int)compressionLevel) { } + + /// + /// Creates a new ZipWriterOptions instance from an existing WriterOptions instance. + /// + /// The WriterOptions to copy values from. + public ZipWriterOptions(WriterOptions options) + : this(options.CompressionType, options.CompressionLevel) + { + LeaveStreamOpen = options.LeaveStreamOpen; + ArchiveEncoding = options.ArchiveEncoding; + Progress = options.Progress; + BufferSize = options.BufferSize; + Providers = options.Providers; + } + + /// + /// Creates a new ZipWriterOptions instance from an existing IWriterOptions instance. + /// + /// The IWriterOptions to copy values from. + public ZipWriterOptions(IWriterOptions options) + : this(options.CompressionType, options.CompressionLevel) + { + LeaveStreamOpen = options.LeaveStreamOpen; + ArchiveEncoding = options.ArchiveEncoding; + Progress = options.Progress; + BufferSize = options.BufferSize; + Providers = options.Providers; + } + + /// + /// Implicit conversion from CompressionType to ZipWriterOptions. + /// + /// The compression type. + public static implicit operator ZipWriterOptions(CompressionType compressionType) => + new(compressionType); } diff --git a/src/SharpCompress/Writers/Zip/ZipWritingStream.cs b/src/SharpCompress/Writers/Zip/ZipWritingStream.cs new file mode 100644 index 00000000..55231c97 --- /dev/null +++ b/src/SharpCompress/Writers/Zip/ZipWritingStream.cs @@ -0,0 +1,750 @@ +using System; +using System.Buffers.Binary; +using System.IO; +using System.Runtime.InteropServices; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Zip; +using SharpCompress.Common.Zip.Headers; +using SharpCompress.Compressors.Deflate; +using SharpCompress.IO; +using SharpCompress.Providers; + +namespace SharpCompress.Writers.Zip; + +public partial class ZipWriter +{ + internal class ZipWritingStream : Stream + { + private readonly CRC32 crc = new(); + private readonly ZipCentralDirectoryEntry entry; + private readonly Stream originalStream; + private Stream writeStream; + private readonly ZipWriter writer; + private readonly ZipCompressionMethod zipCompressionMethod; + private readonly int compressionLevel; + private ICompressionProviderHooks? compressionProviderHooks; + private CompressionContext? compressionContext; + private CountingStream? counting; + private ulong decompressed; + + // Flag to prevent throwing exceptions on Dispose + private bool limitsExceeded; + private bool isDisposed; + + internal ZipWritingStream( + ZipWriter writer, + Stream originalStream, + ZipCentralDirectoryEntry entry, + ZipCompressionMethod zipCompressionMethod, + int compressionLevel, + Stream? compressionStream = null + ) + : this(writer, originalStream, entry, zipCompressionMethod, compressionLevel) + { + writeStream = GetWriteStream(compressionStream ?? originalStream); + } + + private ZipWritingStream( + ZipWriter writer, + Stream originalStream, + ZipCentralDirectoryEntry entry, + ZipCompressionMethod zipCompressionMethod, + int compressionLevel + ) + { + this.writer = writer; + this.originalStream = originalStream; + this.entry = entry; + this.zipCompressionMethod = zipCompressionMethod; + this.compressionLevel = compressionLevel; + writeStream = Stream.Null; + } + + internal static async ValueTask CreateAsync( + ZipWriter writer, + Stream originalStream, + ZipCentralDirectoryEntry entry, + ZipCompressionMethod zipCompressionMethod, + int compressionLevel, + CancellationToken cancellationToken + ) + { + var stream = new ZipWritingStream( + writer, + originalStream, + entry, + zipCompressionMethod, + compressionLevel + ); + stream.writeStream = await stream + .GetWriteStreamAsync(originalStream, cancellationToken) + .ConfigureAwait(false); + return stream; + } + + public override bool CanRead => false; + + public override bool CanSeek => false; + + public override bool CanWrite => true; + + public override long Length => throw new NotSupportedException(); + + public override long Position + { + get => throw new NotSupportedException(); + set => throw new NotSupportedException(); + } + + private Stream GetWriteStream(Stream writeStream) + { + counting = new CountingStream(SharpCompressStream.CreateNonDisposing(writeStream)); + Stream output = counting; + + var providers = writer.WriterOptions.Providers; + + switch (zipCompressionMethod) + { + case ZipCompressionMethod.None: + { + return output; + } + case ZipCompressionMethod.Deflate: + { + return providers.CreateCompressStream( + CompressionType.Deflate, + counting, + compressionLevel + ); + } + case ZipCompressionMethod.BZip2: + { + return providers.CreateCompressStream( + CompressionType.BZip2, + counting, + compressionLevel + ); + } + case ZipCompressionMethod.LZMA: + { + var compressingProvider = providers.GetCompressingProvider( + CompressionType.LZMA + ); + if (compressingProvider is null) + { + throw new ArchiveOperationException("LZMA compression provider not found."); + } + + var context = new CompressionContext { CanSeek = originalStream.CanSeek }; + compressionProviderHooks = compressingProvider; + compressionContext = context; + + var preData = compressingProvider.GetPreCompressionData(context); + if (preData is not null) + { + counting.Write(preData, 0, preData.Length); + } + + var lzmaStream = compressingProvider.CreateCompressStream( + counting, + compressionLevel, + context + ); + + var props = compressingProvider.GetCompressionProperties(lzmaStream, context); + if (props is not null) + { + counting.Write(props, 0, props.Length); + } + + return lzmaStream; + } + case ZipCompressionMethod.PPMd: + { + var compressingProvider = providers.GetCompressingProvider( + CompressionType.PPMd + ); + if (compressingProvider is null) + { + throw new ArchiveOperationException("PPMd compression provider not found."); + } + + var context = new CompressionContext + { + CanSeek = originalStream.CanSeek, + FormatOptions = writer.PpmdProperties, + }; + compressionProviderHooks = compressingProvider; + compressionContext = context; + + var preData = compressingProvider.GetPreCompressionData(context); + if (preData is not null) + { + counting.Write(preData, 0, preData.Length); + } + + return compressingProvider.CreateCompressStream( + counting, + compressionLevel, + context + ); + } + case ZipCompressionMethod.ZStandard: + { + return providers.CreateCompressStream( + CompressionType.ZStandard, + counting, + compressionLevel + ); + } + default: + { + throw new NotSupportedException("CompressionMethod: " + zipCompressionMethod); + } + } + } + + private async ValueTask GetWriteStreamAsync( + Stream writeStream, + CancellationToken cancellationToken + ) + { + counting = new CountingStream(SharpCompressStream.CreateNonDisposing(writeStream)); + Stream output = counting; + + var providers = writer.WriterOptions.Providers; + + switch (zipCompressionMethod) + { + case ZipCompressionMethod.None: + { + return output; + } + case ZipCompressionMethod.Deflate: + { + return await providers + .CreateCompressStreamAsync( + CompressionType.Deflate, + counting, + compressionLevel, + cancellationToken + ) + .ConfigureAwait(false); + } + case ZipCompressionMethod.BZip2: + { + return await providers + .CreateCompressStreamAsync( + CompressionType.BZip2, + counting, + compressionLevel, + cancellationToken + ) + .ConfigureAwait(false); + } + case ZipCompressionMethod.LZMA: + { + var compressingProvider = providers.GetCompressingProvider( + CompressionType.LZMA + ); + if (compressingProvider is null) + { + throw new ArchiveOperationException("LZMA compression provider not found."); + } + + var context = new CompressionContext { CanSeek = originalStream.CanSeek }; + compressionProviderHooks = compressingProvider; + compressionContext = context; + + var preData = compressingProvider.GetPreCompressionData(context); + if (preData is not null) + { + await counting + .WriteAsync(preData, 0, preData.Length, cancellationToken) + .ConfigureAwait(false); + } + + var lzmaStream = await compressingProvider + .CreateCompressStreamAsync( + counting, + compressionLevel, + context, + cancellationToken + ) + .ConfigureAwait(false); + + var props = compressingProvider.GetCompressionProperties(lzmaStream, context); + if (props is not null) + { + await counting + .WriteAsync(props, 0, props.Length, cancellationToken) + .ConfigureAwait(false); + } + + return lzmaStream; + } + case ZipCompressionMethod.PPMd: + { + var compressingProvider = providers.GetCompressingProvider( + CompressionType.PPMd + ); + if (compressingProvider is null) + { + throw new ArchiveOperationException("PPMd compression provider not found."); + } + + var context = new CompressionContext + { + CanSeek = originalStream.CanSeek, + FormatOptions = writer.PpmdProperties, + }; + compressionProviderHooks = compressingProvider; + compressionContext = context; + + var preData = compressingProvider.GetPreCompressionData(context); + if (preData is not null) + { + await counting + .WriteAsync(preData, 0, preData.Length, cancellationToken) + .ConfigureAwait(false); + } + + return await compressingProvider + .CreateCompressStreamAsync( + counting, + compressionLevel, + context, + cancellationToken + ) + .ConfigureAwait(false); + } + case ZipCompressionMethod.ZStandard: + { + return await providers + .CreateCompressStreamAsync( + CompressionType.ZStandard, + counting, + compressionLevel, + cancellationToken + ) + .ConfigureAwait(false); + } + default: + { + throw new NotSupportedException("CompressionMethod: " + zipCompressionMethod); + } + } + } + + protected override void Dispose(bool disposing) + { + if (isDisposed) + { + return; + } + + isDisposed = true; + + base.Dispose(disposing); + if (disposing) + { + writeStream.Dispose(); + + if (limitsExceeded) + { + // We have written invalid data into the archive, + // so we destroy it now, instead of allowing the user to continue + // with a defunct archive + originalStream.Dispose(); + return; + } + + WritePostCompressionData(); + + var countingCount = counting?.BytesWritten ?? 0; + entry.Crc = (uint)crc.Crc32Result; + entry.Compressed = (ulong)countingCount; + entry.Decompressed = decompressed; + + var zip64 = + entry.Compressed >= uint.MaxValue || entry.Decompressed >= uint.MaxValue; + var compressedvalue = zip64 ? uint.MaxValue : (uint)countingCount; + var decompressedvalue = zip64 ? uint.MaxValue : (uint)entry.Decompressed; + + if (originalStream.CanSeek) + { + originalStream.Position = (long)(entry.HeaderOffset + 6); + originalStream.WriteByte(0); + + if (countingCount == 0 && entry.Decompressed == 0) + { + // set compression to STORED for zero byte files (no compression data) + originalStream.Position = (long)(entry.HeaderOffset + 8); + originalStream.WriteByte(0); + originalStream.WriteByte(0); + } + + originalStream.Position = (long)(entry.HeaderOffset + 14); + + writer.WriteFooter(entry.Crc, compressedvalue, decompressedvalue); + + // Ideally, we should not throw from Dispose() + // We should not get here as the Write call checks the limits + if (zip64 && entry.Zip64HeaderOffset == 0) + { + throw new NotSupportedException( + "Attempted to write a stream that is larger than 4GiB without setting the zip64 option" + ); + } + + // If we have pre-allocated space for zip64 data, + // fill it out, even if it is not required + if (entry.Zip64HeaderOffset != 0) + { + originalStream.Position = (long)( + entry.HeaderOffset + entry.Zip64HeaderOffset + ); + Span intBuf = stackalloc byte[8]; + BinaryPrimitives.WriteUInt16LittleEndian(intBuf, 0x0001); + originalStream.Write(intBuf.Slice(0, 2)); + BinaryPrimitives.WriteUInt16LittleEndian(intBuf, 8 + 8); + originalStream.Write(intBuf.Slice(0, 2)); + + BinaryPrimitives.WriteUInt64LittleEndian(intBuf, entry.Decompressed); + originalStream.Write(intBuf); + BinaryPrimitives.WriteUInt64LittleEndian(intBuf, entry.Compressed); + originalStream.Write(intBuf); + } + + originalStream.Position = writer.streamPosition + (long)entry.Compressed; + writer.streamPosition += (long)entry.Compressed; + } + else + { + // We have a streaming archive, so we should add a post-data-descriptor, + // but we cannot as it does not hold the zip64 values + // Throwing an exception until the zip specification is clarified + + // Ideally, we should not throw from Dispose() + // We should not get here as the Write call checks the limits + if (zip64) + { + throw new NotSupportedException( + "Streams larger than 4GiB are not supported for non-seekable streams" + ); + } + + Span intBuf = stackalloc byte[4]; + BinaryPrimitives.WriteUInt32LittleEndian( + intBuf, + ZipHeaderFactory.POST_DATA_DESCRIPTOR + ); + originalStream.Write(intBuf); + writer.WriteFooter(entry.Crc, compressedvalue, decompressedvalue); + writer.streamPosition += (long)entry.Compressed + 16; + } + writer.entries.Add(entry); + } + } + + public override void Flush() => writeStream.Flush(); + + public override int Read(byte[] buffer, int offset, int count) => + throw new NotSupportedException(); + + public override long Seek(long offset, SeekOrigin origin) => + throw new NotSupportedException(); + + public override void SetLength(long value) => throw new NotSupportedException(); + + public override void Write(byte[] buffer, int offset, int count) + { + CheckWriteLimits(count); + + decompressed += (uint)count; + crc.SlurpBlock(buffer, offset, count); + writeStream.Write(buffer, offset, count); + + CheckPostWriteLimits(); + } + + public override async Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + cancellationToken.ThrowIfCancellationRequested(); + CheckWriteLimits(count); + + decompressed += (uint)count; + crc.SlurpBlock(buffer, offset, count); + await writeStream + .WriteAsync(buffer, offset, count, cancellationToken) + .ConfigureAwait(false); + + CheckPostWriteLimits(); + } + +#if !LEGACY_DOTNET + public override async ValueTask WriteAsync( + ReadOnlyMemory buffer, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + CheckWriteLimits(buffer.Length); + + decompressed += (uint)buffer.Length; + if (MemoryMarshal.TryGetArray(buffer, out var segment)) + { + crc.SlurpBlock(segment.Array!, segment.Offset, segment.Count); + } + else + { + var array = buffer.ToArray(); + crc.SlurpBlock(array, 0, array.Length); + } + await writeStream.WriteAsync(buffer, cancellationToken).ConfigureAwait(false); + + CheckPostWriteLimits(); + } +#endif + + private void CheckWriteLimits(int count) + { + // We check the limits first, because we can keep the archive consistent + // if we can prevent the writes from happening. The compressed byte count + // is only an estimate until compression has actually happened. + if (entry.Zip64HeaderOffset != 0) + { + return; + } + + var countingCount = counting?.BytesWritten ?? 0; + if ( + limitsExceeded + || ((decompressed + (uint)count) > uint.MaxValue) + || (countingCount + (uint)count) > uint.MaxValue + ) + { + throw new NotSupportedException( + "Attempted to write a stream that is larger than 4GiB without setting the zip64 option" + ); + } + } + + private void CheckPostWriteLimits() + { + if (entry.Zip64HeaderOffset != 0) + { + return; + } + + var countingCount = counting?.BytesWritten ?? 0; + if ((decompressed > uint.MaxValue) || countingCount > uint.MaxValue) + { + // We have written the data, so the archive is now broken. Throwing + // here avoids throwing from Dispose(), which can mask other errors. + limitsExceeded = true; + throw new NotSupportedException( + "Attempted to write a stream that is larger than 4GiB without setting the zip64 option" + ); + } + } + + private void WritePostCompressionData() + { + if ( + compressionProviderHooks is null + || compressionContext is null + || counting is null + || zipCompressionMethod == ZipCompressionMethod.None + ) + { + return; + } + + var postData = compressionProviderHooks.GetPostCompressionData( + writeStream, + compressionContext + ); + if (postData is null || postData.Length == 0) + { + return; + } + + counting.Write(postData, 0, postData.Length); + } + + private async ValueTask WritePostCompressionDataAsync(CancellationToken cancellationToken) + { + if ( + compressionProviderHooks is null + || compressionContext is null + || counting is null + || zipCompressionMethod == ZipCompressionMethod.None + ) + { + return; + } + + var postData = compressionProviderHooks.GetPostCompressionData( + writeStream, + compressionContext + ); + if (postData is null || postData.Length == 0) + { + return; + } + + await counting + .WriteAsync(postData, 0, postData.Length, cancellationToken) + .ConfigureAwait(false); + } + +#if NET48 || NETSTANDARD2_0 + public async ValueTask DisposeAsync() +#else + public override async ValueTask DisposeAsync() +#endif + { + if (isDisposed) + { + return; + } + + isDisposed = true; + + if (writeStream is IAsyncDisposable asyncDisposableWriteStream) + { + await asyncDisposableWriteStream.DisposeAsync().ConfigureAwait(false); + } + else + { + writeStream.Dispose(); + } + + if (limitsExceeded) + { + // We have written invalid data into the archive, so destroy it + if (originalStream is IAsyncDisposable asyncDisposableOriginalStream) + { + await asyncDisposableOriginalStream.DisposeAsync().ConfigureAwait(false); + } + else + { + originalStream.Dispose(); + } + return; + } + + await WritePostCompressionDataAsync(CancellationToken.None).ConfigureAwait(false); + + var countingCount = counting?.BytesWritten ?? 0; + entry.Crc = (uint)crc.Crc32Result; + entry.Compressed = (ulong)countingCount; + entry.Decompressed = decompressed; + + var zip64 = entry.Compressed >= uint.MaxValue || entry.Decompressed >= uint.MaxValue; + var compressedvalue = zip64 ? uint.MaxValue : (uint)countingCount; + var decompressedvalue = zip64 ? uint.MaxValue : (uint)entry.Decompressed; + + if (originalStream.CanSeek) + { + originalStream.Position = (long)(entry.HeaderOffset + 6); + await originalStream.WriteAsync(new byte[] { 0 }, 0, 1).ConfigureAwait(false); + + if (countingCount == 0 && entry.Decompressed == 0) + { + // set compression to STORED for zero byte files + originalStream.Position = (long)(entry.HeaderOffset + 8); + await originalStream + .WriteAsync(new byte[] { 0, 0 }, 0, 2) + .ConfigureAwait(false); + } + + originalStream.Position = (long)(entry.HeaderOffset + 14); + + await WriteFooterAsync( + originalStream, + entry.Crc, + compressedvalue, + decompressedvalue + ) + .ConfigureAwait(false); + + if (zip64 && entry.Zip64HeaderOffset == 0) + { + throw new NotSupportedException( + "Attempted to write a stream that is larger than 4GiB without setting the zip64 option" + ); + } + + if (entry.Zip64HeaderOffset != 0) + { + originalStream.Position = (long)(entry.HeaderOffset + entry.Zip64HeaderOffset); + var intBuf = new byte[8]; + BinaryPrimitives.WriteUInt16LittleEndian(intBuf, 0x0001); + await originalStream.WriteAsync(intBuf, 0, 2).ConfigureAwait(false); + BinaryPrimitives.WriteUInt16LittleEndian(intBuf, 8 + 8); + await originalStream.WriteAsync(intBuf, 0, 2).ConfigureAwait(false); + + BinaryPrimitives.WriteUInt64LittleEndian(intBuf, entry.Decompressed); + await originalStream.WriteAsync(intBuf, 0, 8).ConfigureAwait(false); + BinaryPrimitives.WriteUInt64LittleEndian(intBuf, entry.Compressed); + await originalStream.WriteAsync(intBuf, 0, 8).ConfigureAwait(false); + } + + originalStream.Position = writer.streamPosition + (long)entry.Compressed; + writer.streamPosition += (long)entry.Compressed; + } + else + { + if (zip64) + { + throw new NotSupportedException( + "Streams larger than 4GiB are not supported for non-seekable streams" + ); + } + + var intBuf = new byte[4]; + BinaryPrimitives.WriteUInt32LittleEndian( + intBuf, + ZipHeaderFactory.POST_DATA_DESCRIPTOR + ); + await originalStream.WriteAsync(intBuf, 0, 4).ConfigureAwait(false); + await WriteFooterAsync( + originalStream, + entry.Crc, + compressedvalue, + decompressedvalue + ) + .ConfigureAwait(false); + writer.streamPosition += (long)entry.Compressed + 16; + } + writer.entries.Add(entry); +#if !NET48 && !NETSTANDARD2_0 + // base.DisposeAsync() is a no-op since isDisposed is already set + await base.DisposeAsync().ConfigureAwait(false); +#endif + } + + private static async ValueTask WriteFooterAsync( + Stream stream, + uint crc, + uint compressed, + uint uncompressed + ) + { + var buf = new byte[12]; + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(0), crc); + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(4), compressed); + BinaryPrimitives.WriteUInt32LittleEndian(buf.AsSpan(8), uncompressed); + await stream.WriteAsync(buf, 0, buf.Length).ConfigureAwait(false); + } + } +} diff --git a/src/SharpCompress/packages.lock.json b/src/SharpCompress/packages.lock.json new file mode 100644 index 00000000..13d0ce05 --- /dev/null +++ b/src/SharpCompress/packages.lock.json @@ -0,0 +1,505 @@ +{ + "version": 2, + "dependencies": { + ".NETFramework,Version=v4.8": { + "Microsoft.Bcl.AsyncInterfaces": { + "type": "Direct", + "requested": "[8.0.0, )", + "resolved": "8.0.0", + "contentHash": "3WA9q9yVqJp222P3x1wYIGDAkpjAku0TMUaaQV22g6L67AI0LdOIrVS7Ht2vJfLHGSPVuqN94vIr15qn+HEkHw==", + "dependencies": { + "System.Threading.Tasks.Extensions": "4.5.4" + } + }, + "Microsoft.NETFramework.ReferenceAssemblies": { + "type": "Direct", + "requested": "[1.0.3, )", + "resolved": "1.0.3", + "contentHash": "vUc9Npcs14QsyOD01tnv/m8sQUnGTGOw1BCmKcv77LBJY7OxhJ+zJF7UD/sCL3lYNFuqmQEVlkfS4Quif6FyYg==", + "dependencies": { + "Microsoft.NETFramework.ReferenceAssemblies.net48": "1.0.3" + } + }, + "Microsoft.SourceLink.GitHub": { + "type": "Direct", + "requested": "[10.0.300, )", + "resolved": "10.0.300", + "contentHash": "QzCtLkXVb3l4IxcpvJCbzUwMLihAmLN6vVLjQGSzYSF8d2dvXxqJAZk83RV3gYnp2egz8jRMgSR2woY3vOahTA==", + "dependencies": { + "Microsoft.Build.Tasks.Git": "10.0.300", + "Microsoft.SourceLink.Common": "10.0.300", + "System.IO.Hashing": "10.0.8" + } + }, + "Microsoft.VisualStudio.Threading.Analyzers": { + "type": "Direct", + "requested": "[17.14.15, )", + "resolved": "17.14.15", + "contentHash": "mXQPJsbuUD2ydq4/ffd8h8tSOFCXec+2xJOVNCvXjuMOq/+5EKHq3D2m2MC2+nUaXeFMSt66VS/J4HdKBixgcw==" + }, + "PolySharp": { + "type": "Direct", + "requested": "[1.16.0, )", + "resolved": "1.16.0", + "contentHash": "3kdIIceBPumwjw279FuiVMfVENT2cGASXJgcigdySsbX2dJB8ofUgG6i47yqF/k1qu6fvNR3csrSekZPviR6kQ==" + }, + "System.Text.Encoding.CodePages": { + "type": "Direct", + "requested": "[8.0.0, )", + "resolved": "8.0.0", + "contentHash": "OZIsVplFGaVY90G2SbpgU7EnCoOO5pw1t4ic21dBF3/1omrJFpAGoNAVpPyMVOC90/hvgkGG3VFqR13YgZMQfg==", + "dependencies": { + "System.Memory": "4.5.5", + "System.Runtime.CompilerServices.Unsafe": "6.0.0" + } + }, + "Microsoft.Build.Tasks.Git": { + "type": "Transitive", + "resolved": "10.0.300", + "contentHash": "P0kaQwVZx4xIUe2FtrLyBadYNXuAljttJUPvjBYRuHhPE8L77L42KakLDkaADRiUrGspoLcMwayjrbQhYTr0zA==", + "dependencies": { + "System.IO.Hashing": "10.0.8" + } + }, + "Microsoft.NETFramework.ReferenceAssemblies.net48": { + "type": "Transitive", + "resolved": "1.0.3", + "contentHash": "zMk4D+9zyiEWByyQ7oPImPN/Jhpj166Ky0Nlla4eXlNL8hI/BtSJsgR8Inldd4NNpIAH3oh8yym0W2DrhXdSLQ==" + }, + "Microsoft.SourceLink.Common": { + "type": "Transitive", + "resolved": "10.0.300", + "contentHash": "0jlkXaUGjYlWTIVPve5MftjKHnT3SlAtq9BCLV4J9IjdPrxV/+4rMlBSjfr1khG8/GC6KGojjola8E1VvWF0qQ==" + }, + "System.Buffers": { + "type": "Transitive", + "resolved": "4.6.1", + "contentHash": "N8GXpmiLMtljq7gwvyS+1QvKT/W2J8sNAvx+HVg4NGmsG/H+2k/y9QI23auLJRterrzCiDH+IWAw4V/GPwsMlw==" + }, + "System.IO.Hashing": { + "type": "Transitive", + "resolved": "10.0.8", + "contentHash": "+dJsbPJ3FyUbTZNplFj0RCKePFizmv6ewDV46JE9q/IVH4c3xTCftHfHelLsAKf0jryIPqgMb5GpS0x7TAY3mg==", + "dependencies": { + "System.Buffers": "4.6.1", + "System.Memory": "4.6.3" + } + }, + "System.Memory": { + "type": "Transitive", + "resolved": "4.6.3", + "contentHash": "qdcDOgnFZY40+Q9876JUHnlHu7bosOHX8XISRoH94fwk6hgaeQGSgfZd8srWRZNt5bV9ZW2TljcegDNxsf+96A==", + "dependencies": { + "System.Buffers": "4.6.1", + "System.Numerics.Vectors": "4.6.1", + "System.Runtime.CompilerServices.Unsafe": "6.1.2" + } + }, + "System.Numerics.Vectors": { + "type": "Transitive", + "resolved": "4.6.1", + "contentHash": "sQxefTnhagrhoq2ReR0D/6K0zJcr9Hrd6kikeXsA1I8kOCboTavcUC4r7TSfpKFeE163uMuxZcyfO1mGO3EN8Q==" + }, + "System.Runtime.CompilerServices.Unsafe": { + "type": "Transitive", + "resolved": "6.1.2", + "contentHash": "2hBr6zdbIBTDE3EhK7NSVNdX58uTK6iHW/P/Axmm9sl1xoGSLqDvMtpecn226TNwHByFokYwJmt/aQQNlO5CRw==" + }, + "System.Threading.Tasks.Extensions": { + "type": "Transitive", + "resolved": "4.5.4", + "contentHash": "zteT+G8xuGu6mS+mzDzYXbzS7rd3K6Fjb9RiZlYlJPam2/hU7JCBZBVEcywNuR+oZ1ncTvc/cq0faRr3P01OVg==", + "dependencies": { + "System.Runtime.CompilerServices.Unsafe": "4.5.3" + } + } + }, + ".NETStandard,Version=v2.0": { + "Microsoft.Bcl.AsyncInterfaces": { + "type": "Direct", + "requested": "[8.0.0, )", + "resolved": "8.0.0", + "contentHash": "3WA9q9yVqJp222P3x1wYIGDAkpjAku0TMUaaQV22g6L67AI0LdOIrVS7Ht2vJfLHGSPVuqN94vIr15qn+HEkHw==", + "dependencies": { + "System.Threading.Tasks.Extensions": "4.5.4" + } + }, + "Microsoft.NETFramework.ReferenceAssemblies": { + "type": "Direct", + "requested": "[1.0.3, )", + "resolved": "1.0.3", + "contentHash": "vUc9Npcs14QsyOD01tnv/m8sQUnGTGOw1BCmKcv77LBJY7OxhJ+zJF7UD/sCL3lYNFuqmQEVlkfS4Quif6FyYg==", + "dependencies": { + "Microsoft.NETFramework.ReferenceAssemblies.net461": "1.0.3" + } + }, + "Microsoft.SourceLink.GitHub": { + "type": "Direct", + "requested": "[10.0.300, )", + "resolved": "10.0.300", + "contentHash": "QzCtLkXVb3l4IxcpvJCbzUwMLihAmLN6vVLjQGSzYSF8d2dvXxqJAZk83RV3gYnp2egz8jRMgSR2woY3vOahTA==", + "dependencies": { + "Microsoft.Build.Tasks.Git": "10.0.300", + "Microsoft.SourceLink.Common": "10.0.300", + "System.IO.Hashing": "10.0.8" + } + }, + "Microsoft.VisualStudio.Threading.Analyzers": { + "type": "Direct", + "requested": "[17.14.15, )", + "resolved": "17.14.15", + "contentHash": "mXQPJsbuUD2ydq4/ffd8h8tSOFCXec+2xJOVNCvXjuMOq/+5EKHq3D2m2MC2+nUaXeFMSt66VS/J4HdKBixgcw==" + }, + "NETStandard.Library": { + "type": "Direct", + "requested": "[2.0.3, )", + "resolved": "2.0.3", + "contentHash": "st47PosZSHrjECdjeIzZQbzivYBJFv6P2nv4cj2ypdI204DO+vZ7l5raGMiX4eXMJ53RfOIg+/s4DHVZ54Nu2A==", + "dependencies": { + "Microsoft.NETCore.Platforms": "1.1.0" + } + }, + "PolySharp": { + "type": "Direct", + "requested": "[1.16.0, )", + "resolved": "1.16.0", + "contentHash": "3kdIIceBPumwjw279FuiVMfVENT2cGASXJgcigdySsbX2dJB8ofUgG6i47yqF/k1qu6fvNR3csrSekZPviR6kQ==" + }, + "System.Text.Encoding.CodePages": { + "type": "Direct", + "requested": "[8.0.0, )", + "resolved": "8.0.0", + "contentHash": "OZIsVplFGaVY90G2SbpgU7EnCoOO5pw1t4ic21dBF3/1omrJFpAGoNAVpPyMVOC90/hvgkGG3VFqR13YgZMQfg==", + "dependencies": { + "System.Memory": "4.5.5", + "System.Runtime.CompilerServices.Unsafe": "6.0.0" + } + }, + "Microsoft.Build.Tasks.Git": { + "type": "Transitive", + "resolved": "10.0.300", + "contentHash": "P0kaQwVZx4xIUe2FtrLyBadYNXuAljttJUPvjBYRuHhPE8L77L42KakLDkaADRiUrGspoLcMwayjrbQhYTr0zA==", + "dependencies": { + "System.IO.Hashing": "10.0.8" + } + }, + "Microsoft.NETCore.Platforms": { + "type": "Transitive", + "resolved": "1.1.0", + "contentHash": "kz0PEW2lhqygehI/d6XsPCQzD7ff7gUJaVGPVETX611eadGsA3A877GdSlU0LRVMCTH/+P3o2iDTak+S08V2+A==" + }, + "Microsoft.NETFramework.ReferenceAssemblies.net461": { + "type": "Transitive", + "resolved": "1.0.3", + "contentHash": "AmOJZwCqnOCNp6PPcf9joyogScWLtwy0M1WkqfEQ0M9nYwyDD7EX9ZjscKS5iYnyvteX7kzSKFCKt9I9dXA6mA==" + }, + "Microsoft.SourceLink.Common": { + "type": "Transitive", + "resolved": "10.0.300", + "contentHash": "0jlkXaUGjYlWTIVPve5MftjKHnT3SlAtq9BCLV4J9IjdPrxV/+4rMlBSjfr1khG8/GC6KGojjola8E1VvWF0qQ==" + }, + "System.Buffers": { + "type": "Transitive", + "resolved": "4.6.1", + "contentHash": "N8GXpmiLMtljq7gwvyS+1QvKT/W2J8sNAvx+HVg4NGmsG/H+2k/y9QI23auLJRterrzCiDH+IWAw4V/GPwsMlw==" + }, + "System.IO.Hashing": { + "type": "Transitive", + "resolved": "10.0.8", + "contentHash": "+dJsbPJ3FyUbTZNplFj0RCKePFizmv6ewDV46JE9q/IVH4c3xTCftHfHelLsAKf0jryIPqgMb5GpS0x7TAY3mg==", + "dependencies": { + "System.Buffers": "4.6.1", + "System.Memory": "4.6.3" + } + }, + "System.Memory": { + "type": "Transitive", + "resolved": "4.6.3", + "contentHash": "qdcDOgnFZY40+Q9876JUHnlHu7bosOHX8XISRoH94fwk6hgaeQGSgfZd8srWRZNt5bV9ZW2TljcegDNxsf+96A==", + "dependencies": { + "System.Buffers": "4.6.1", + "System.Numerics.Vectors": "4.6.1", + "System.Runtime.CompilerServices.Unsafe": "6.1.2" + } + }, + "System.Numerics.Vectors": { + "type": "Transitive", + "resolved": "4.6.1", + "contentHash": "sQxefTnhagrhoq2ReR0D/6K0zJcr9Hrd6kikeXsA1I8kOCboTavcUC4r7TSfpKFeE163uMuxZcyfO1mGO3EN8Q==" + }, + "System.Runtime.CompilerServices.Unsafe": { + "type": "Transitive", + "resolved": "6.1.2", + "contentHash": "2hBr6zdbIBTDE3EhK7NSVNdX58uTK6iHW/P/Axmm9sl1xoGSLqDvMtpecn226TNwHByFokYwJmt/aQQNlO5CRw==" + }, + "System.Threading.Tasks.Extensions": { + "type": "Transitive", + "resolved": "4.5.4", + "contentHash": "zteT+G8xuGu6mS+mzDzYXbzS7rd3K6Fjb9RiZlYlJPam2/hU7JCBZBVEcywNuR+oZ1ncTvc/cq0faRr3P01OVg==", + "dependencies": { + "System.Runtime.CompilerServices.Unsafe": "4.5.3" + } + } + }, + ".NETStandard,Version=v2.1": { + "Microsoft.Bcl.AsyncInterfaces": { + "type": "Direct", + "requested": "[8.0.0, )", + "resolved": "8.0.0", + "contentHash": "3WA9q9yVqJp222P3x1wYIGDAkpjAku0TMUaaQV22g6L67AI0LdOIrVS7Ht2vJfLHGSPVuqN94vIr15qn+HEkHw==" + }, + "Microsoft.NETFramework.ReferenceAssemblies": { + "type": "Direct", + "requested": "[1.0.3, )", + "resolved": "1.0.3", + "contentHash": "vUc9Npcs14QsyOD01tnv/m8sQUnGTGOw1BCmKcv77LBJY7OxhJ+zJF7UD/sCL3lYNFuqmQEVlkfS4Quif6FyYg==", + "dependencies": { + "Microsoft.NETFramework.ReferenceAssemblies.net461": "1.0.3" + } + }, + "Microsoft.SourceLink.GitHub": { + "type": "Direct", + "requested": "[10.0.300, )", + "resolved": "10.0.300", + "contentHash": "QzCtLkXVb3l4IxcpvJCbzUwMLihAmLN6vVLjQGSzYSF8d2dvXxqJAZk83RV3gYnp2egz8jRMgSR2woY3vOahTA==", + "dependencies": { + "Microsoft.Build.Tasks.Git": "10.0.300", + "Microsoft.SourceLink.Common": "10.0.300", + "System.IO.Hashing": "10.0.8" + } + }, + "Microsoft.VisualStudio.Threading.Analyzers": { + "type": "Direct", + "requested": "[17.14.15, )", + "resolved": "17.14.15", + "contentHash": "mXQPJsbuUD2ydq4/ffd8h8tSOFCXec+2xJOVNCvXjuMOq/+5EKHq3D2m2MC2+nUaXeFMSt66VS/J4HdKBixgcw==" + }, + "PolySharp": { + "type": "Direct", + "requested": "[1.16.0, )", + "resolved": "1.16.0", + "contentHash": "3kdIIceBPumwjw279FuiVMfVENT2cGASXJgcigdySsbX2dJB8ofUgG6i47yqF/k1qu6fvNR3csrSekZPviR6kQ==" + }, + "System.Text.Encoding.CodePages": { + "type": "Direct", + "requested": "[8.0.0, )", + "resolved": "8.0.0", + "contentHash": "OZIsVplFGaVY90G2SbpgU7EnCoOO5pw1t4ic21dBF3/1omrJFpAGoNAVpPyMVOC90/hvgkGG3VFqR13YgZMQfg==", + "dependencies": { + "System.Runtime.CompilerServices.Unsafe": "6.0.0" + } + }, + "Microsoft.Build.Tasks.Git": { + "type": "Transitive", + "resolved": "10.0.300", + "contentHash": "P0kaQwVZx4xIUe2FtrLyBadYNXuAljttJUPvjBYRuHhPE8L77L42KakLDkaADRiUrGspoLcMwayjrbQhYTr0zA==", + "dependencies": { + "System.IO.Hashing": "10.0.8" + } + }, + "Microsoft.NETFramework.ReferenceAssemblies.net461": { + "type": "Transitive", + "resolved": "1.0.3", + "contentHash": "AmOJZwCqnOCNp6PPcf9joyogScWLtwy0M1WkqfEQ0M9nYwyDD7EX9ZjscKS5iYnyvteX7kzSKFCKt9I9dXA6mA==" + }, + "Microsoft.SourceLink.Common": { + "type": "Transitive", + "resolved": "10.0.300", + "contentHash": "0jlkXaUGjYlWTIVPve5MftjKHnT3SlAtq9BCLV4J9IjdPrxV/+4rMlBSjfr1khG8/GC6KGojjola8E1VvWF0qQ==" + }, + "System.IO.Hashing": { + "type": "Transitive", + "resolved": "10.0.8", + "contentHash": "+dJsbPJ3FyUbTZNplFj0RCKePFizmv6ewDV46JE9q/IVH4c3xTCftHfHelLsAKf0jryIPqgMb5GpS0x7TAY3mg==" + }, + "System.Runtime.CompilerServices.Unsafe": { + "type": "Transitive", + "resolved": "6.0.0", + "contentHash": "/iUeP3tq1S0XdNNoMz5C9twLSrM/TH+qElHkXWaPvuNOt+99G75NrV0OS2EqHx5wMN7popYjpc8oTjC1y16DLg==" + } + }, + "net10.0": { + "Microsoft.NET.ILLink.Tasks": { + "type": "Direct", + "requested": "[10.0.8, )", + "resolved": "10.0.8", + "contentHash": "dVbSXGIFNR5nZcv2tOLoWI+a9T4jtFd77IYjuND+QVe360qWgAF7H0WtoopYhRw/+SgpGUTyrkrh+65+ClNnfw==" + }, + "Microsoft.NETFramework.ReferenceAssemblies": { + "type": "Direct", + "requested": "[1.0.3, )", + "resolved": "1.0.3", + "contentHash": "vUc9Npcs14QsyOD01tnv/m8sQUnGTGOw1BCmKcv77LBJY7OxhJ+zJF7UD/sCL3lYNFuqmQEVlkfS4Quif6FyYg==", + "dependencies": { + "Microsoft.NETFramework.ReferenceAssemblies.net461": "1.0.3" + } + }, + "Microsoft.SourceLink.GitHub": { + "type": "Direct", + "requested": "[10.0.300, )", + "resolved": "10.0.300", + "contentHash": "QzCtLkXVb3l4IxcpvJCbzUwMLihAmLN6vVLjQGSzYSF8d2dvXxqJAZk83RV3gYnp2egz8jRMgSR2woY3vOahTA==", + "dependencies": { + "Microsoft.Build.Tasks.Git": "10.0.300", + "Microsoft.SourceLink.Common": "10.0.300", + "System.IO.Hashing": "10.0.8" + } + }, + "Microsoft.VisualStudio.Threading.Analyzers": { + "type": "Direct", + "requested": "[17.14.15, )", + "resolved": "17.14.15", + "contentHash": "mXQPJsbuUD2ydq4/ffd8h8tSOFCXec+2xJOVNCvXjuMOq/+5EKHq3D2m2MC2+nUaXeFMSt66VS/J4HdKBixgcw==" + }, + "PolySharp": { + "type": "Direct", + "requested": "[1.16.0, )", + "resolved": "1.16.0", + "contentHash": "3kdIIceBPumwjw279FuiVMfVENT2cGASXJgcigdySsbX2dJB8ofUgG6i47yqF/k1qu6fvNR3csrSekZPviR6kQ==" + }, + "Microsoft.Build.Tasks.Git": { + "type": "Transitive", + "resolved": "10.0.300", + "contentHash": "P0kaQwVZx4xIUe2FtrLyBadYNXuAljttJUPvjBYRuHhPE8L77L42KakLDkaADRiUrGspoLcMwayjrbQhYTr0zA==", + "dependencies": { + "System.IO.Hashing": "10.0.8" + } + }, + "Microsoft.NETFramework.ReferenceAssemblies.net461": { + "type": "Transitive", + "resolved": "1.0.3", + "contentHash": "AmOJZwCqnOCNp6PPcf9joyogScWLtwy0M1WkqfEQ0M9nYwyDD7EX9ZjscKS5iYnyvteX7kzSKFCKt9I9dXA6mA==" + }, + "Microsoft.SourceLink.Common": { + "type": "Transitive", + "resolved": "10.0.300", + "contentHash": "0jlkXaUGjYlWTIVPve5MftjKHnT3SlAtq9BCLV4J9IjdPrxV/+4rMlBSjfr1khG8/GC6KGojjola8E1VvWF0qQ==" + }, + "System.IO.Hashing": { + "type": "Transitive", + "resolved": "10.0.8", + "contentHash": "+dJsbPJ3FyUbTZNplFj0RCKePFizmv6ewDV46JE9q/IVH4c3xTCftHfHelLsAKf0jryIPqgMb5GpS0x7TAY3mg==" + } + }, + "net6.0": { + "Microsoft.NETFramework.ReferenceAssemblies": { + "type": "Direct", + "requested": "[1.0.3, )", + "resolved": "1.0.3", + "contentHash": "vUc9Npcs14QsyOD01tnv/m8sQUnGTGOw1BCmKcv77LBJY7OxhJ+zJF7UD/sCL3lYNFuqmQEVlkfS4Quif6FyYg==", + "dependencies": { + "Microsoft.NETFramework.ReferenceAssemblies.net461": "1.0.3" + } + }, + "Microsoft.SourceLink.GitHub": { + "type": "Direct", + "requested": "[10.0.300, )", + "resolved": "10.0.300", + "contentHash": "QzCtLkXVb3l4IxcpvJCbzUwMLihAmLN6vVLjQGSzYSF8d2dvXxqJAZk83RV3gYnp2egz8jRMgSR2woY3vOahTA==", + "dependencies": { + "Microsoft.Build.Tasks.Git": "10.0.300", + "Microsoft.SourceLink.Common": "10.0.300", + "System.IO.Hashing": "10.0.8" + } + }, + "Microsoft.VisualStudio.Threading.Analyzers": { + "type": "Direct", + "requested": "[17.14.15, )", + "resolved": "17.14.15", + "contentHash": "mXQPJsbuUD2ydq4/ffd8h8tSOFCXec+2xJOVNCvXjuMOq/+5EKHq3D2m2MC2+nUaXeFMSt66VS/J4HdKBixgcw==" + }, + "PolySharp": { + "type": "Direct", + "requested": "[1.16.0, )", + "resolved": "1.16.0", + "contentHash": "3kdIIceBPumwjw279FuiVMfVENT2cGASXJgcigdySsbX2dJB8ofUgG6i47yqF/k1qu6fvNR3csrSekZPviR6kQ==" + }, + "Microsoft.Build.Tasks.Git": { + "type": "Transitive", + "resolved": "10.0.300", + "contentHash": "P0kaQwVZx4xIUe2FtrLyBadYNXuAljttJUPvjBYRuHhPE8L77L42KakLDkaADRiUrGspoLcMwayjrbQhYTr0zA==", + "dependencies": { + "System.IO.Hashing": "10.0.8" + } + }, + "Microsoft.NETFramework.ReferenceAssemblies.net461": { + "type": "Transitive", + "resolved": "1.0.3", + "contentHash": "AmOJZwCqnOCNp6PPcf9joyogScWLtwy0M1WkqfEQ0M9nYwyDD7EX9ZjscKS5iYnyvteX7kzSKFCKt9I9dXA6mA==" + }, + "Microsoft.SourceLink.Common": { + "type": "Transitive", + "resolved": "10.0.300", + "contentHash": "0jlkXaUGjYlWTIVPve5MftjKHnT3SlAtq9BCLV4J9IjdPrxV/+4rMlBSjfr1khG8/GC6KGojjola8E1VvWF0qQ==" + }, + "System.IO.Hashing": { + "type": "Transitive", + "resolved": "10.0.8", + "contentHash": "+dJsbPJ3FyUbTZNplFj0RCKePFizmv6ewDV46JE9q/IVH4c3xTCftHfHelLsAKf0jryIPqgMb5GpS0x7TAY3mg==" + } + }, + "net8.0": { + "Microsoft.NET.ILLink.Tasks": { + "type": "Direct", + "requested": "[8.0.27, )", + "resolved": "8.0.27", + "contentHash": "rQi9TxifHRnXP7lVRZH05DxD2/XGbJp12q0ozcbrlBlBnyyzssFTH/2vLhtKWUp2CT1qVscTrcYTFiwTyKPKRg==" + }, + "Microsoft.NETFramework.ReferenceAssemblies": { + "type": "Direct", + "requested": "[1.0.3, )", + "resolved": "1.0.3", + "contentHash": "vUc9Npcs14QsyOD01tnv/m8sQUnGTGOw1BCmKcv77LBJY7OxhJ+zJF7UD/sCL3lYNFuqmQEVlkfS4Quif6FyYg==", + "dependencies": { + "Microsoft.NETFramework.ReferenceAssemblies.net461": "1.0.3" + } + }, + "Microsoft.SourceLink.GitHub": { + "type": "Direct", + "requested": "[10.0.300, )", + "resolved": "10.0.300", + "contentHash": "QzCtLkXVb3l4IxcpvJCbzUwMLihAmLN6vVLjQGSzYSF8d2dvXxqJAZk83RV3gYnp2egz8jRMgSR2woY3vOahTA==", + "dependencies": { + "Microsoft.Build.Tasks.Git": "10.0.300", + "Microsoft.SourceLink.Common": "10.0.300", + "System.IO.Hashing": "10.0.8" + } + }, + "Microsoft.VisualStudio.Threading.Analyzers": { + "type": "Direct", + "requested": "[17.14.15, )", + "resolved": "17.14.15", + "contentHash": "mXQPJsbuUD2ydq4/ffd8h8tSOFCXec+2xJOVNCvXjuMOq/+5EKHq3D2m2MC2+nUaXeFMSt66VS/J4HdKBixgcw==" + }, + "PolySharp": { + "type": "Direct", + "requested": "[1.16.0, )", + "resolved": "1.16.0", + "contentHash": "3kdIIceBPumwjw279FuiVMfVENT2cGASXJgcigdySsbX2dJB8ofUgG6i47yqF/k1qu6fvNR3csrSekZPviR6kQ==" + }, + "Microsoft.Build.Tasks.Git": { + "type": "Transitive", + "resolved": "10.0.300", + "contentHash": "P0kaQwVZx4xIUe2FtrLyBadYNXuAljttJUPvjBYRuHhPE8L77L42KakLDkaADRiUrGspoLcMwayjrbQhYTr0zA==", + "dependencies": { + "System.IO.Hashing": "10.0.8" + } + }, + "Microsoft.NETFramework.ReferenceAssemblies.net461": { + "type": "Transitive", + "resolved": "1.0.3", + "contentHash": "AmOJZwCqnOCNp6PPcf9joyogScWLtwy0M1WkqfEQ0M9nYwyDD7EX9ZjscKS5iYnyvteX7kzSKFCKt9I9dXA6mA==" + }, + "Microsoft.SourceLink.Common": { + "type": "Transitive", + "resolved": "10.0.300", + "contentHash": "0jlkXaUGjYlWTIVPve5MftjKHnT3SlAtq9BCLV4J9IjdPrxV/+4rMlBSjfr1khG8/GC6KGojjola8E1VvWF0qQ==" + }, + "System.IO.Hashing": { + "type": "Transitive", + "resolved": "10.0.8", + "contentHash": "+dJsbPJ3FyUbTZNplFj0RCKePFizmv6ewDV46JE9q/IVH4c3xTCftHfHelLsAKf0jryIPqgMb5GpS0x7TAY3mg==" + } + } + } +} \ No newline at end of file diff --git a/tests/SharpCompress.AotSmoke/Program.cs b/tests/SharpCompress.AotSmoke/Program.cs new file mode 100644 index 00000000..2bdbabe6 --- /dev/null +++ b/tests/SharpCompress.AotSmoke/Program.cs @@ -0,0 +1,59 @@ +using System; +using System.IO; +using System.Text; +using SharpCompress.Archives; +using SharpCompress.Common; +using SharpCompress.Readers; +using SharpCompress.Writers; + +var original = "SharpCompress AOT smoke test"; +using var archiveStream = new MemoryStream(); + +using ( + var writer = WriterFactory.OpenWriter( + archiveStream, + ArchiveType.Zip, + new WriterOptions(CompressionType.Deflate) { LeaveStreamOpen = true } + ) +) +{ + using var entryStream = new MemoryStream(Encoding.UTF8.GetBytes(original)); + writer.Write("payload.txt", entryStream, DateTime.UtcNow); +} + +archiveStream.Position = 0; +using (var reader = ReaderFactory.OpenReader(archiveStream, ReaderOptions.ForExternalStream)) +{ + if (!reader.MoveToNextEntry() || reader.Entry.IsDirectory) + { + throw new InvalidOperationException("Expected a file entry."); + } + + using var extracted = new MemoryStream(); + reader.WriteEntryTo(extracted); + var actual = Encoding.UTF8.GetString(extracted.ToArray()); + if (!string.Equals(original, actual, StringComparison.Ordinal)) + { + throw new InvalidOperationException("ReaderFactory round-trip content mismatch."); + } +} + +archiveStream.Position = 0; +using (var archive = ArchiveFactory.OpenArchive(archiveStream, ReaderOptions.ForExternalStream)) +{ + var entryCount = 0; + foreach (var entry in archive.Entries) + { + if (!entry.IsDirectory) + { + entryCount++; + } + } + + if (entryCount != 1) + { + throw new InvalidOperationException("ArchiveFactory did not see the expected entry."); + } +} + +Console.WriteLine("SharpCompress AOT smoke test passed."); diff --git a/tests/SharpCompress.AotSmoke/SharpCompress.AotSmoke.csproj b/tests/SharpCompress.AotSmoke/SharpCompress.AotSmoke.csproj new file mode 100644 index 00000000..aed0b97d --- /dev/null +++ b/tests/SharpCompress.AotSmoke/SharpCompress.AotSmoke.csproj @@ -0,0 +1,13 @@ + + + Exe + net10.0 + linux-x64 + true + true + full + + + + + diff --git a/tests/SharpCompress.AotSmoke/packages.lock.json b/tests/SharpCompress.AotSmoke/packages.lock.json new file mode 100644 index 00000000..bfbb6d92 --- /dev/null +++ b/tests/SharpCompress.AotSmoke/packages.lock.json @@ -0,0 +1,93 @@ +{ + "version": 2, + "dependencies": { + "net10.0": { + "Microsoft.DotNet.ILCompiler": { + "type": "Direct", + "requested": "[10.0.8, )", + "resolved": "10.0.8", + "contentHash": "RJxitcN5CCyZDcPNXKLsecwKvACzmy8C1z8hGM9+hFcnPhv1jDysJFFIeUHIPWaZ6wDAfYtZcgKEtegvL2Nz8A==" + }, + "Microsoft.NET.ILLink.Tasks": { + "type": "Direct", + "requested": "[10.0.8, )", + "resolved": "10.0.8", + "contentHash": "dVbSXGIFNR5nZcv2tOLoWI+a9T4jtFd77IYjuND+QVe360qWgAF7H0WtoopYhRw/+SgpGUTyrkrh+65+ClNnfw==" + }, + "Microsoft.NETFramework.ReferenceAssemblies": { + "type": "Direct", + "requested": "[1.0.3, )", + "resolved": "1.0.3", + "contentHash": "vUc9Npcs14QsyOD01tnv/m8sQUnGTGOw1BCmKcv77LBJY7OxhJ+zJF7UD/sCL3lYNFuqmQEVlkfS4Quif6FyYg==", + "dependencies": { + "Microsoft.NETFramework.ReferenceAssemblies.net461": "1.0.3" + } + }, + "Microsoft.SourceLink.GitHub": { + "type": "Direct", + "requested": "[10.0.300, )", + "resolved": "10.0.300", + "contentHash": "QzCtLkXVb3l4IxcpvJCbzUwMLihAmLN6vVLjQGSzYSF8d2dvXxqJAZk83RV3gYnp2egz8jRMgSR2woY3vOahTA==", + "dependencies": { + "Microsoft.Build.Tasks.Git": "10.0.300", + "Microsoft.SourceLink.Common": "10.0.300", + "System.IO.Hashing": "10.0.8" + } + }, + "Microsoft.VisualStudio.Threading.Analyzers": { + "type": "Direct", + "requested": "[17.14.15, )", + "resolved": "17.14.15", + "contentHash": "mXQPJsbuUD2ydq4/ffd8h8tSOFCXec+2xJOVNCvXjuMOq/+5EKHq3D2m2MC2+nUaXeFMSt66VS/J4HdKBixgcw==" + }, + "PolySharp": { + "type": "Direct", + "requested": "[1.16.0, )", + "resolved": "1.16.0", + "contentHash": "3kdIIceBPumwjw279FuiVMfVENT2cGASXJgcigdySsbX2dJB8ofUgG6i47yqF/k1qu6fvNR3csrSekZPviR6kQ==" + }, + "Microsoft.Build.Tasks.Git": { + "type": "Transitive", + "resolved": "10.0.300", + "contentHash": "P0kaQwVZx4xIUe2FtrLyBadYNXuAljttJUPvjBYRuHhPE8L77L42KakLDkaADRiUrGspoLcMwayjrbQhYTr0zA==", + "dependencies": { + "System.IO.Hashing": "10.0.8" + } + }, + "Microsoft.NETFramework.ReferenceAssemblies.net461": { + "type": "Transitive", + "resolved": "1.0.3", + "contentHash": "AmOJZwCqnOCNp6PPcf9joyogScWLtwy0M1WkqfEQ0M9nYwyDD7EX9ZjscKS5iYnyvteX7kzSKFCKt9I9dXA6mA==" + }, + "Microsoft.SourceLink.Common": { + "type": "Transitive", + "resolved": "10.0.300", + "contentHash": "0jlkXaUGjYlWTIVPve5MftjKHnT3SlAtq9BCLV4J9IjdPrxV/+4rMlBSjfr1khG8/GC6KGojjola8E1VvWF0qQ==" + }, + "System.IO.Hashing": { + "type": "Transitive", + "resolved": "10.0.8", + "contentHash": "+dJsbPJ3FyUbTZNplFj0RCKePFizmv6ewDV46JE9q/IVH4c3xTCftHfHelLsAKf0jryIPqgMb5GpS0x7TAY3mg==" + }, + "sharpcompress": { + "type": "Project" + } + }, + "net10.0/linux-x64": { + "Microsoft.DotNet.ILCompiler": { + "type": "Direct", + "requested": "[10.0.8, )", + "resolved": "10.0.8", + "contentHash": "RJxitcN5CCyZDcPNXKLsecwKvACzmy8C1z8hGM9+hFcnPhv1jDysJFFIeUHIPWaZ6wDAfYtZcgKEtegvL2Nz8A==", + "dependencies": { + "runtime.linux-x64.Microsoft.DotNet.ILCompiler": "10.0.8" + } + }, + "runtime.linux-x64.Microsoft.DotNet.ILCompiler": { + "type": "Transitive", + "resolved": "10.0.8", + "contentHash": "0jxyi69frgaqADCnEpHE+f65NoiRTAjfjvNDMOxWV77BumQ56eMDL4ECw29DcJTqwaYJQ92PqDS6y6CiLf7kgw==" + } + } + } +} \ No newline at end of file diff --git a/tests/SharpCompress.Performance/Benchmarks/ArchiveBenchmarkBase.cs b/tests/SharpCompress.Performance/Benchmarks/ArchiveBenchmarkBase.cs new file mode 100644 index 00000000..21013236 --- /dev/null +++ b/tests/SharpCompress.Performance/Benchmarks/ArchiveBenchmarkBase.cs @@ -0,0 +1,39 @@ +using System; +using System.IO; + +namespace SharpCompress.Performance.Benchmarks; + +public abstract class ArchiveBenchmarkBase +{ + protected static readonly string TEST_ARCHIVES_PATH; + + static ArchiveBenchmarkBase() + { + var baseDirectory = AppDomain.CurrentDomain.BaseDirectory; + var index = baseDirectory.IndexOf( + "SharpCompress.Performance", + StringComparison.OrdinalIgnoreCase + ); + + if (index == -1) + { + throw new InvalidOperationException( + "Could not find SharpCompress.Performance in the base directory path" + ); + } + + var path = baseDirectory.Substring(0, index); + var solutionBasePath = Path.GetDirectoryName(path) ?? throw new InvalidOperationException(); + TEST_ARCHIVES_PATH = Path.Combine(solutionBasePath, "TestArchives", "Archives"); + + if (!Directory.Exists(TEST_ARCHIVES_PATH)) + { + throw new InvalidOperationException( + $"Test archives directory not found: {TEST_ARCHIVES_PATH}" + ); + } + } + + protected static string GetArchivePath(string fileName) => + Path.Combine(TEST_ARCHIVES_PATH, fileName); +} diff --git a/tests/SharpCompress.Performance/Benchmarks/GZipBenchmarks.cs b/tests/SharpCompress.Performance/Benchmarks/GZipBenchmarks.cs new file mode 100644 index 00000000..e9804009 --- /dev/null +++ b/tests/SharpCompress.Performance/Benchmarks/GZipBenchmarks.cs @@ -0,0 +1,63 @@ +using System; +using System.IO; +using System.Threading.Tasks; +using BenchmarkDotNet.Attributes; +using SharpCompress.Compressors; +using SharpCompress.Compressors.Deflate; + +namespace SharpCompress.Performance.Benchmarks; + +[MemoryDiagnoser] +public class GZipBenchmarks +{ + private byte[] _sourceData = null!; + private byte[] _compressedData = null!; + + [GlobalSetup] + public void Setup() + { + // Create 100KB of test data + _sourceData = new byte[100 * 1024]; + new Random(42).NextBytes(_sourceData); + + // Pre-compress for decompression benchmark + using var compressStream = new MemoryStream(); + using (var gzipStream = new GZipStream(compressStream, CompressionMode.Compress)) + { + gzipStream.Write(_sourceData, 0, _sourceData.Length); + } + _compressedData = compressStream.ToArray(); + } + + [Benchmark(Description = "GZip: Compress 100KB")] + public void GZipCompress() + { + using var outputStream = new MemoryStream(); + using var gzipStream = new GZipStream(outputStream, CompressionMode.Compress); + gzipStream.Write(_sourceData, 0, _sourceData.Length); + } + + [Benchmark(Description = "GZip: Compress 100KB (Async)")] + public async Task GZipCompressAsync() + { + using var outputStream = new MemoryStream(); + using var gzipStream = new GZipStream(outputStream, CompressionMode.Compress); + await gzipStream.WriteAsync(_sourceData, 0, _sourceData.Length).ConfigureAwait(false); + } + + [Benchmark(Description = "GZip: Decompress 100KB")] + public void GZipDecompress() + { + using var inputStream = new MemoryStream(_compressedData); + using var gzipStream = new GZipStream(inputStream, CompressionMode.Decompress); + gzipStream.CopyTo(Stream.Null); + } + + [Benchmark(Description = "GZip: Decompress 100KB (Async)")] + public async Task GZipDecompressAsync() + { + using var inputStream = new MemoryStream(_compressedData); + using var gzipStream = new GZipStream(inputStream, CompressionMode.Decompress); + await gzipStream.CopyToAsync(Stream.Null).ConfigureAwait(false); + } +} diff --git a/tests/SharpCompress.Performance/Benchmarks/RarBenchmarks.cs b/tests/SharpCompress.Performance/Benchmarks/RarBenchmarks.cs new file mode 100644 index 00000000..ffe207cd --- /dev/null +++ b/tests/SharpCompress.Performance/Benchmarks/RarBenchmarks.cs @@ -0,0 +1,73 @@ +using System; +using System.IO; +using System.Linq; +using System.Threading.Tasks; +using BenchmarkDotNet.Attributes; +using SharpCompress.Archives.Rar; +using SharpCompress.Readers; + +namespace SharpCompress.Performance.Benchmarks; + +[MemoryDiagnoser] +public class RarBenchmarks : ArchiveBenchmarkBase +{ + private byte[] _rarBytes = null!; + + [GlobalSetup] + public void Setup() + { + _rarBytes = File.ReadAllBytes(GetArchivePath("Rar.rar")); + } + + [Benchmark(Description = "Rar: Extract all entries (Archive API)")] + public void RarExtractArchiveApi() + { + using var stream = new MemoryStream(_rarBytes); + using var archive = RarArchive.OpenArchive(stream); + foreach (var entry in archive.Entries.Where(e => !e.IsDirectory)) + { + using var entryStream = entry.OpenEntryStream(); + entryStream.CopyTo(Stream.Null); + } + } + + [Benchmark(Description = "Rar: Extract all entries (Archive API, Async)")] + public async Task RarExtractArchiveApiAsync() + { + using var stream = new MemoryStream(_rarBytes); + await using var archive = await RarArchive.OpenAsyncArchive(stream).ConfigureAwait(false); + await foreach (var entry in archive.EntriesAsync.Where(e => !e.IsDirectory)) + { + await using var entryStream = await entry.OpenEntryStreamAsync().ConfigureAwait(false); + await entryStream.CopyToAsync(Stream.Null).ConfigureAwait(false); + } + } + + [Benchmark(Description = "Rar: Extract all entries (Reader API)")] + public void RarExtractReaderApi() + { + using var stream = new MemoryStream(_rarBytes); + using var reader = ReaderFactory.OpenReader(stream); + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + reader.WriteEntryTo(Stream.Null); + } + } + } + + [Benchmark(Description = "Rar: Extract all entries (Reader API, Async)")] + public async Task RarExtractReaderApiAsync() + { + using var stream = new MemoryStream(_rarBytes); + await using var reader = await ReaderFactory.OpenAsyncReader(stream).ConfigureAwait(false); + while (await reader.MoveToNextEntryAsync().ConfigureAwait(false)) + { + if (!reader.Entry.IsDirectory) + { + await reader.WriteEntryToAsync(Stream.Null).ConfigureAwait(false); + } + } + } +} diff --git a/tests/SharpCompress.Performance/Benchmarks/SevenZipBenchmarks.cs b/tests/SharpCompress.Performance/Benchmarks/SevenZipBenchmarks.cs new file mode 100644 index 00000000..38435ef2 --- /dev/null +++ b/tests/SharpCompress.Performance/Benchmarks/SevenZipBenchmarks.cs @@ -0,0 +1,102 @@ +using System; +using System.IO; +using System.Linq; +using System.Threading.Tasks; +using BenchmarkDotNet.Attributes; +using SharpCompress.Archives.SevenZip; + +namespace SharpCompress.Performance.Benchmarks; + +[MemoryDiagnoser] +public class SevenZipBenchmarks : ArchiveBenchmarkBase +{ + private byte[] _lzmaBytes = null!; + private byte[] _lzma2Bytes = null!; + + [GlobalSetup] + public void Setup() + { + _lzmaBytes = File.ReadAllBytes(GetArchivePath("7Zip.LZMA.7z")); + _lzma2Bytes = File.ReadAllBytes(GetArchivePath("7Zip.LZMA2.7z")); + } + + [Benchmark(Description = "7Zip LZMA: Extract all entries")] + public void SevenZipLzmaExtract() + { + using var stream = new MemoryStream(_lzmaBytes); + using var archive = SevenZipArchive.OpenArchive(stream); + foreach (var entry in archive.Entries.Where(e => !e.IsDirectory)) + { + using var entryStream = entry.OpenEntryStream(); + entryStream.CopyTo(Stream.Null); + } + } + + [Benchmark(Description = "7Zip LZMA: Extract all entries (Async)")] + public async Task SevenZipLzmaExtractAsync() + { + using var stream = new MemoryStream(_lzmaBytes); + await using var archive = await SevenZipArchive + .OpenAsyncArchive(stream) + .ConfigureAwait(false); + await foreach (var entry in archive.EntriesAsync.Where(e => !e.IsDirectory)) + { + await using var entryStream = await entry.OpenEntryStreamAsync().ConfigureAwait(false); + await entryStream.CopyToAsync(Stream.Null).ConfigureAwait(false); + } + } + + [Benchmark(Description = "7Zip LZMA2: Extract all entries")] + public void SevenZipLzma2Extract() + { + using var stream = new MemoryStream(_lzma2Bytes); + using var archive = SevenZipArchive.OpenArchive(stream); + foreach (var entry in archive.Entries.Where(e => !e.IsDirectory)) + { + using var entryStream = entry.OpenEntryStream(); + entryStream.CopyTo(Stream.Null); + } + } + + [Benchmark(Description = "7Zip LZMA2: Extract all entries (Async)")] + public async Task SevenZipLzma2ExtractAsync() + { + using var stream = new MemoryStream(_lzma2Bytes); + await using var archive = await SevenZipArchive + .OpenAsyncArchive(stream) + .ConfigureAwait(false); + await foreach (var entry in archive.EntriesAsync.Where(e => !e.IsDirectory)) + { + await using var entryStream = await entry.OpenEntryStreamAsync().ConfigureAwait(false); + await entryStream.CopyToAsync(Stream.Null).ConfigureAwait(false); + } + } + + [Benchmark(Description = "7Zip LZMA2 Reader: Extract all entries")] + public void SevenZipLzma2Extract_Reader() + { + using var stream = new MemoryStream(_lzma2Bytes); + using var archive = SevenZipArchive.OpenArchive(stream); + using var reader = archive.ExtractAllEntries(); + foreach (var entry in archive.Entries.Where(e => !e.IsDirectory)) + { + using var entryStream = entry.OpenEntryStream(); + entryStream.CopyTo(Stream.Null); + } + } + + [Benchmark(Description = "7Zip LZMA2 Reader: Extract all entries (Async)")] + public async Task SevenZipLzma2ExtractAsync_Reader() + { + using var stream = new MemoryStream(_lzma2Bytes); + await using var archive = await SevenZipArchive + .OpenAsyncArchive(stream) + .ConfigureAwait(false); + await using var reader = await archive.ExtractAllEntriesAsync(); + while (await reader.MoveToNextEntryAsync().ConfigureAwait(false)) + { + await using var entryStream = await reader.OpenEntryStreamAsync().ConfigureAwait(false); + await entryStream.CopyToAsync(Stream.Null).ConfigureAwait(false); + } + } +} diff --git a/tests/SharpCompress.Performance/Benchmarks/TarBenchmarks.cs b/tests/SharpCompress.Performance/Benchmarks/TarBenchmarks.cs new file mode 100644 index 00000000..7a7cbadd --- /dev/null +++ b/tests/SharpCompress.Performance/Benchmarks/TarBenchmarks.cs @@ -0,0 +1,166 @@ +using System; +using System.IO; +using System.Linq; +using System.Threading.Tasks; +using BenchmarkDotNet.Attributes; +using SharpCompress.Archives; +using SharpCompress.Archives.Tar; +using SharpCompress.Common; +using SharpCompress.Compressors; +using SharpCompress.Providers; +using SharpCompress.Providers.System; +using SharpCompress.Readers; +using SharpCompress.Writers; + +namespace SharpCompress.Performance.Benchmarks; + +[MemoryDiagnoser] +public class TarBenchmarks : ArchiveBenchmarkBase +{ + private byte[] _tarBytes = null!; + private byte[] _tarGzBytes = null!; + + [GlobalSetup] + public void Setup() + { + _tarBytes = File.ReadAllBytes(GetArchivePath("Tar.tar")); + _tarGzBytes = File.ReadAllBytes(GetArchivePath("Tar.tar.gz")); + } + + [Benchmark(Description = "Tar: Extract all entries (Archive API)")] + public void TarExtractArchiveApi() + { + using var stream = new MemoryStream(_tarBytes); + using var archive = TarArchive.OpenArchive(stream); + foreach (var entry in archive.Entries.Where(e => !e.IsDirectory)) + { + using var entryStream = entry.OpenEntryStream(); + entryStream.CopyTo(Stream.Null); + } + } + + [Benchmark(Description = "Tar: Extract all entries (Archive API, Async)")] + public async Task TarExtractArchiveApiAsync() + { + using var stream = new MemoryStream(_tarBytes); + await using var archive = await TarArchive.OpenAsyncArchive(stream).ConfigureAwait(false); + await foreach (var entry in archive.EntriesAsync.Where(e => !e.IsDirectory)) + { + await using var entryStream = await entry.OpenEntryStreamAsync().ConfigureAwait(false); + await entryStream.CopyToAsync(Stream.Null).ConfigureAwait(false); + } + } + + [Benchmark(Description = "Tar: Extract all entries (Reader API)")] + public void TarExtractReaderApi() + { + using var stream = new MemoryStream(_tarBytes); + using var reader = ReaderFactory.OpenReader(stream); + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + reader.WriteEntryTo(Stream.Null); + } + } + } + + [Benchmark(Description = "Tar: Extract all entries (Archive API) - SystemGzip")] + public void SystemTarExtractArchiveApi() + { + using var stream = new MemoryStream(_tarGzBytes); + using var archive = ArchiveFactory.OpenArchive( + stream, + ReaderOptions.ForExternalStream.WithProviders( + CompressionProviderRegistry.Default.With(new SystemGZipCompressionProvider()) + ) + ); + foreach (var entry in archive.Entries.Where(e => !e.IsDirectory)) + { + using var entryStream = entry.OpenEntryStream(); + entryStream.CopyTo(Stream.Null); + } + } + + [Benchmark(Description = "Tar: Extract all entries (Reader API) - SystemGzip")] + public void SystemTarExtractReaderApi() + { + using var stream = new MemoryStream(_tarGzBytes); + using var reader = ReaderFactory.OpenReader( + stream, + ReaderOptions.ForExternalStream.WithProviders( + CompressionProviderRegistry.Default.With(new SystemGZipCompressionProvider()) + ) + ); + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + reader.WriteEntryTo(Stream.Null); + } + } + } + + [Benchmark(Description = "Tar.GZip: Extract all entries")] + public void TarGzipExtract() + { + using var stream = new MemoryStream(_tarGzBytes); + using var archive = ArchiveFactory.OpenArchive(stream); + foreach (var entry in archive.Entries.Where(e => !e.IsDirectory)) + { + using var entryStream = entry.OpenEntryStream(); + entryStream.CopyTo(Stream.Null); + } + } + + [Benchmark(Description = "Tar.GZip: Extract all entries (Async)")] + public async Task TarGzipExtractAsync() + { + using var stream = new MemoryStream(_tarGzBytes); + await using var archive = await ArchiveFactory + .OpenAsyncArchive(stream) + .ConfigureAwait(false); + await foreach (var entry in archive.EntriesAsync.Where(e => !e.IsDirectory)) + { + await using var entryStream = await entry.OpenEntryStreamAsync().ConfigureAwait(false); + await entryStream.CopyToAsync(Stream.Null).ConfigureAwait(false); + } + } + + [Benchmark(Description = "Tar: Create archive with small files")] + public void TarCreateSmallFiles() + { + using var outputStream = new MemoryStream(); + using var writer = WriterFactory.OpenWriter( + outputStream, + ArchiveType.Tar, + new WriterOptions(CompressionType.None) { LeaveStreamOpen = true } + ); + + // Create 10 small files + for (int i = 0; i < 10; i++) + { + var data = new byte[1024]; // 1KB each + using var entryStream = new MemoryStream(data); + writer.Write($"file{i}.txt", entryStream); + } + } + + [Benchmark(Description = "Tar: Create archive with small files (Async)")] + public async Task TarCreateSmallFilesAsync() + { + using var outputStream = new MemoryStream(); + await using var writer = await WriterFactory.OpenAsyncWriter( + outputStream, + ArchiveType.Tar, + new WriterOptions(CompressionType.None) { LeaveStreamOpen = true } + ); + + for (int i = 0; i < 10; i++) + { + var data = new byte[1024]; + using var entryStream = new MemoryStream(data); + await writer.WriteAsync($"file{i}.txt", entryStream).ConfigureAwait(false); + } + } +} diff --git a/tests/SharpCompress.Performance/Benchmarks/ZipBenchmarks.cs b/tests/SharpCompress.Performance/Benchmarks/ZipBenchmarks.cs new file mode 100644 index 00000000..c5b3e090 --- /dev/null +++ b/tests/SharpCompress.Performance/Benchmarks/ZipBenchmarks.cs @@ -0,0 +1,153 @@ +using System; +using System.IO; +using System.Linq; +using System.Threading.Tasks; +using BenchmarkDotNet.Attributes; +using SharpCompress.Archives.Zip; +using SharpCompress.Common; +using SharpCompress.Compressors; +using SharpCompress.Providers; +using SharpCompress.Providers.System; +using SharpCompress.Readers; +using SharpCompress.Writers; + +namespace SharpCompress.Performance.Benchmarks; + +[MemoryDiagnoser] +public class ZipBenchmarks : ArchiveBenchmarkBase +{ + private string _archivePath = null!; + private byte[] _archiveBytes = null!; + + [GlobalSetup] + public void Setup() + { + _archivePath = GetArchivePath("Zip.deflate.zip"); + _archiveBytes = File.ReadAllBytes(_archivePath); + } + + [Benchmark(Description = "Zip: Extract all entries (Archive API) - SystemDeflate")] + public void SystemZipExtractArchiveApi() + { + using var stream = new MemoryStream(_archiveBytes); + using var archive = ZipArchive.OpenArchive( + stream, + ReaderOptions.ForExternalStream.WithProviders( + CompressionProviderRegistry.Empty.With(new SystemDeflateCompressionProvider()) + ) + ); + foreach (var entry in archive.Entries.Where(e => !e.IsDirectory)) + { + using var entryStream = entry.OpenEntryStream(); + entryStream.CopyTo(Stream.Null); + } + } + + [Benchmark(Description = "Zip: Extract all entries (Archive API)")] + public void ZipExtractArchiveApi() + { + using var stream = new MemoryStream(_archiveBytes); + using var archive = ZipArchive.OpenArchive(stream); + foreach (var entry in archive.Entries.Where(e => !e.IsDirectory)) + { + using var entryStream = entry.OpenEntryStream(); + entryStream.CopyTo(Stream.Null); + } + } + + [Benchmark(Description = "Zip: Extract all entries (Archive API, Async)")] + public async Task ZipExtractArchiveApiAsync() + { + using var stream = new MemoryStream(_archiveBytes); + await using var archive = await ZipArchive.OpenAsyncArchive(stream).ConfigureAwait(false); + await foreach (var entry in archive.EntriesAsync.Where(e => !e.IsDirectory)) + { + await using var entryStream = await entry.OpenEntryStreamAsync().ConfigureAwait(false); + await entryStream.CopyToAsync(Stream.Null).ConfigureAwait(false); + } + } + + [Benchmark(Description = "Zip: Extract all entries (Reader API) - SystemDeflate")] + public void SystemZipExtractReaderApi() + { + using var stream = new MemoryStream(_archiveBytes); + using var reader = ReaderFactory.OpenReader( + stream, + ReaderOptions.ForExternalStream.WithProviders( + CompressionProviderRegistry.Empty.With(new SystemDeflateCompressionProvider()) + ) + ); + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + reader.WriteEntryTo(Stream.Null); + } + } + } + + [Benchmark(Description = "Zip: Extract all entries (Reader API)")] + public void ZipExtractReaderApi() + { + using var stream = new MemoryStream(_archiveBytes); + using var reader = ReaderFactory.OpenReader(stream); + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + reader.WriteEntryTo(Stream.Null); + } + } + } + + [Benchmark(Description = "Zip: Extract all entries (Reader API, Async)")] + public async Task ZipExtractReaderApiAsync() + { + using var stream = new MemoryStream(_archiveBytes); + await using var reader = await ReaderFactory.OpenAsyncReader(stream).ConfigureAwait(false); + while (await reader.MoveToNextEntryAsync().ConfigureAwait(false)) + { + if (!reader.Entry.IsDirectory) + { + await reader.WriteEntryToAsync(Stream.Null).ConfigureAwait(false); + } + } + } + + [Benchmark(Description = "Zip: Create archive with small files")] + public void ZipCreateSmallFiles() + { + using var outputStream = new MemoryStream(); + using var writer = WriterFactory.OpenWriter( + outputStream, + ArchiveType.Zip, + new WriterOptions(CompressionType.Deflate) { LeaveStreamOpen = true } + ); + + // Create 10 small files + for (int i = 0; i < 10; i++) + { + var data = new byte[1024]; // 1KB each + using var entryStream = new MemoryStream(data); + writer.Write($"file{i}.txt", entryStream); + } + } + + [Benchmark(Description = "Zip: Create archive with small files (Async)")] + public async Task ZipCreateSmallFilesAsync() + { + using var outputStream = new MemoryStream(); + await using var writer = await WriterFactory.OpenAsyncWriter( + outputStream, + ArchiveType.Zip, + new WriterOptions(CompressionType.Deflate) { LeaveStreamOpen = true } + ); + + for (int i = 0; i < 10; i++) + { + var data = new byte[1024]; + using var entryStream = new MemoryStream(data); + await writer.WriteAsync($"file{i}.txt", entryStream).ConfigureAwait(false); + } + } +} diff --git a/tests/SharpCompress.Performance/JetbrainsProfiler.cs b/tests/SharpCompress.Performance/JetbrainsProfiler.cs new file mode 100644 index 00000000..9404fd81 --- /dev/null +++ b/tests/SharpCompress.Performance/JetbrainsProfiler.cs @@ -0,0 +1,49 @@ +using System; +using JetBrains.Profiler.SelfApi; + +namespace SharpCompress.Test; + +public static class JetbrainsProfiler +{ + private sealed class CpuClass : IDisposable + { + public CpuClass(string snapshotPath) + { + DotTrace.Init(); + var config2 = new DotTrace.Config(); + config2.SaveToDir(snapshotPath); + DotTrace.Attach(config2); + DotTrace.StartCollectingData(); + } + + public void Dispose() + { + DotTrace.StopCollectingData(); + DotTrace.SaveData(); + DotTrace.Detach(); + } + } + + private sealed class MemoryClass : IDisposable + { + public MemoryClass(string snapshotPath) + { + DotMemory.Init(); + var config = new DotMemory.Config(); + config.UseLogLevelVerbose(); + config.SaveToDir(snapshotPath); + DotMemory.Attach(config); + DotMemory.GetSnapshot("Before"); + } + + public void Dispose() + { + DotMemory.GetSnapshot("After"); + DotMemory.Detach(); + } + } + + public static IDisposable Cpu(string snapshotPath) => new CpuClass(snapshotPath); + + public static IDisposable Memory(string snapshotPath) => new MemoryClass(snapshotPath); +} diff --git a/tests/SharpCompress.Performance/LargeMemoryStream.cs b/tests/SharpCompress.Performance/LargeMemoryStream.cs new file mode 100644 index 00000000..f8922f9f --- /dev/null +++ b/tests/SharpCompress.Performance/LargeMemoryStream.cs @@ -0,0 +1,305 @@ +using System; +using System.Collections.Generic; +using System.IO; + +namespace SharpCompress.Performance; + +/// +/// A Stream implementation backed by a List of byte arrays that supports large position values. +/// This allows handling streams larger than typical 32-bit or even standard 64-bit constraints +/// by chunking data into multiple byte array segments. +/// +public class LargeMemoryStream : Stream +{ + private readonly List _chunks; + private readonly int _chunkSize; + private long _position; + private bool _isDisposed; + + /// + /// Initializes a new instance of the LargeMemoryStream class. + /// + /// The size of each chunk in the backing byte array list. Defaults to 1MB. + public LargeMemoryStream(int chunkSize = 1024 * 1024) + { + if (chunkSize <= 0) + { + throw new ArgumentException("Chunk size must be greater than zero.", nameof(chunkSize)); + } + + _chunks = new List(); + _chunkSize = chunkSize; + _position = 0; + } + + public override bool CanRead => true; + + public override bool CanSeek => true; + + public override bool CanWrite => true; + + public override long Length + { + get + { + ThrowIfDisposed(); + if (_chunks.Count == 0) + { + return 0; + } + + long length = (long)(_chunks.Count - 1) * _chunkSize; + length += _chunks[_chunks.Count - 1].Length; + return length; + } + } + + public override long Position + { + get + { + ThrowIfDisposed(); + return _position; + } + set + { + ThrowIfDisposed(); + if (value < 0) + { + throw new ArgumentOutOfRangeException( + nameof(value), + "Position cannot be negative." + ); + } + + _position = value; + } + } + + public override void Flush() + { + ThrowIfDisposed(); + // No-op for in-memory stream + } + + public override int Read(byte[] buffer, int offset, int count) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(buffer); + + if (offset < 0 || count < 0 || offset + count > buffer.Length) + { + throw new ArgumentOutOfRangeException(nameof(offset)); + } + + long length = Length; + if (_position >= length) + { + return 0; + } + + int bytesToRead = (int)Math.Min(count, length - _position); + int bytesRead = 0; + + while (bytesRead < bytesToRead) + { + long chunkIndex = _position / _chunkSize; + int chunkOffset = (int)(_position % _chunkSize); + + if (chunkIndex >= _chunks.Count) + { + break; + } + + byte[] chunk = _chunks[(int)chunkIndex]; + int availableInChunk = chunk.Length - chunkOffset; + int bytesToCopyFromChunk = Math.Min(availableInChunk, bytesToRead - bytesRead); + + Array.Copy(chunk, chunkOffset, buffer, offset + bytesRead, bytesToCopyFromChunk); + + _position += bytesToCopyFromChunk; + bytesRead += bytesToCopyFromChunk; + } + + return bytesRead; + } + + public override void Write(byte[] buffer, int offset, int count) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(buffer); + + if (offset < 0 || count < 0 || offset + count > buffer.Length) + { + throw new ArgumentOutOfRangeException(nameof(offset)); + } + + int bytesWritten = 0; + + while (bytesWritten < count) + { + long chunkIndex = _position / _chunkSize; + int chunkOffset = (int)(_position % _chunkSize); + + // Ensure we have enough chunks + while (_chunks.Count <= chunkIndex) + { + _chunks.Add(new byte[_chunkSize]); + } + + byte[] chunk = _chunks[(int)chunkIndex]; + int availableInChunk = chunk.Length - chunkOffset; + int bytesToCopyToChunk = Math.Min(availableInChunk, count - bytesWritten); + + Array.Copy(buffer, offset + bytesWritten, chunk, chunkOffset, bytesToCopyToChunk); + + _position += bytesToCopyToChunk; + bytesWritten += bytesToCopyToChunk; + } + } + + public override long Seek(long offset, SeekOrigin origin) + { + ThrowIfDisposed(); + + long newPosition = origin switch + { + SeekOrigin.Begin => offset, + SeekOrigin.Current => _position + offset, + SeekOrigin.End => Length + offset, + _ => throw new ArgumentOutOfRangeException(nameof(origin)), + }; + + if (newPosition < 0) + { + throw new ArgumentOutOfRangeException( + nameof(offset), + "Cannot seek before the beginning of the stream." + ); + } + + _position = newPosition; + return _position; + } + + public override void SetLength(long value) + { + ThrowIfDisposed(); + if (value < 0) + { + throw new ArgumentOutOfRangeException(nameof(value), "Length cannot be negative."); + } + + long currentLength = Length; + + if (value < currentLength) + { + // Truncate + long chunkIndex = (value + _chunkSize - 1) / _chunkSize; + if (chunkIndex > 0) + { + chunkIndex--; + } + + _chunks.RemoveRange((int)(chunkIndex + 1), _chunks.Count - (int)(chunkIndex + 1)); + + if (chunkIndex < _chunks.Count) + { + int lastChunkSize = (int)(value - chunkIndex * _chunkSize); + var x = _chunks[(int)chunkIndex]; + Array.Resize(ref x, lastChunkSize); + } + + if (_position > value) + { + _position = value; + } + } + else if (value > currentLength) + { + // Extend with zeros + long chunkIndex = currentLength / _chunkSize; + int chunkOffset = (int)(currentLength % _chunkSize); + + while ((long)_chunks.Count * _chunkSize < value) + { + _chunks.Add(new byte[_chunkSize]); + } + + // Resize the last chunk if needed + if (_chunks.Count > 0) + { + long lastChunkNeededSize = value - (long)(_chunks.Count - 1) * _chunkSize; + if (lastChunkNeededSize < _chunkSize) + { + var x = _chunks[^1]; + Array.Resize(ref x, (int)lastChunkNeededSize); + } + } + } + } + + /// + /// Gets the number of chunks in the backing list. + /// + public int ChunkCount => _chunks.Count; + + /// + /// Gets the size of each chunk in bytes. + /// + public int ChunkSize => _chunkSize; + + /// + /// Converts the stream contents to a single byte array. + /// This may consume significant memory for large streams. + /// + public byte[] ToArray() + { + ThrowIfDisposed(); + long length = Length; + byte[] result = new byte[length]; + long currentPosition = _position; + + try + { + _position = 0; + int totalRead = 0; + while (totalRead < length) + { + int bytesToRead = (int)Math.Min(length - totalRead, int.MaxValue); + int bytesRead = Read(result, totalRead, bytesToRead); + if (bytesRead == 0) + { + break; + } + + totalRead += bytesRead; + } + } + finally + { + _position = currentPosition; + } + + return result; + } + + private void ThrowIfDisposed() + { + ObjectDisposedException.ThrowIf(_isDisposed, this); + } + + protected override void Dispose(bool disposing) + { + if (!_isDisposed) + { + if (disposing) + { + _chunks.Clear(); + } + _isDisposed = true; + } + + base.Dispose(disposing); + } +} diff --git a/tests/SharpCompress.Performance/Program.cs b/tests/SharpCompress.Performance/Program.cs new file mode 100644 index 00000000..81d9d7f3 --- /dev/null +++ b/tests/SharpCompress.Performance/Program.cs @@ -0,0 +1,108 @@ +using System; +using System.Threading.Tasks; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; +using BenchmarkDotNet.Running; +using BenchmarkDotNet.Toolchains.InProcess.Emit; + +namespace SharpCompress.Performance; + +public class Program +{ + public static async Task Main(string[] args) + { + // Check if profiling mode is requested + if (args.Length > 0 && args[0].Equals("--profile", StringComparison.OrdinalIgnoreCase)) + { + await RunWithProfiler(args); + return; + } + + // Default: Run BenchmarkDotNet + var config = DefaultConfig.Instance.AddJob( + Job.Default.WithToolchain(InProcessEmitToolchain.Instance) + .WithWarmupCount(5) // Minimal warmup iterations for CI + .WithIterationCount(30) // Minimal measurement iterations for CI + .WithInvocationCount(30) + .WithUnrollFactor(2) + ); + + BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args, config); + } + + private static async Task RunWithProfiler(string[] args) + { + var profileType = "cpu"; // Default to CPU profiling + var outputPath = "./profiler-snapshots"; + + // Parse arguments + for (int i = 1; i < args.Length; i++) + { + if (args[i].Equals("--type", StringComparison.OrdinalIgnoreCase) && i + 1 < args.Length) + { + profileType = args[++i].ToLowerInvariant(); + } + else if ( + args[i].Equals("--output", StringComparison.OrdinalIgnoreCase) + && i + 1 < args.Length + ) + { + outputPath = args[++i]; + } + } + + Console.WriteLine($"Running with JetBrains Profiler ({profileType} mode)"); + Console.WriteLine($"Output path: {outputPath}"); + Console.WriteLine(); + Console.WriteLine( + "Usage: dotnet run --project SharpCompress.Performance.csproj -c Release -- --profile [--type cpu|memory] [--output ]" + ); + Console.WriteLine(); + + // Run a sample benchmark with profiling + await RunSampleBenchmarkWithProfiler(profileType, outputPath); + } + + private static async Task RunSampleBenchmarkWithProfiler(string profileType, string outputPath) + { + try + { + IDisposable? profiler = null; + + if (profileType == "cpu") + { + profiler = Test.JetbrainsProfiler.Cpu(outputPath); + } + else if (profileType == "memory") + { + profiler = Test.JetbrainsProfiler.Memory(outputPath); + } + + using (profiler) + { + // Run a simple benchmark iteration + var zipBenchmark = new Benchmarks.RarBenchmarks(); + zipBenchmark.Setup(); + + Console.WriteLine("Running benchmark iterations..."); + for (int i = 0; i < 100; i++) + { + await zipBenchmark.RarExtractArchiveApiAsync(); + if (i % 3 == 0) + { + Console.Write("."); + } + } + Console.WriteLine(); + Console.WriteLine("Benchmark iterations completed."); + } + + Console.WriteLine($"Profiler snapshot saved to: {outputPath}"); + } + catch (Exception ex) + { + Console.WriteLine($"Error running profiler: {ex.Message}"); + Console.WriteLine("Make sure JetBrains profiler tools are installed and accessible."); + } + } +} diff --git a/tests/SharpCompress.Performance/README.md b/tests/SharpCompress.Performance/README.md new file mode 100644 index 00000000..43e9df9e --- /dev/null +++ b/tests/SharpCompress.Performance/README.md @@ -0,0 +1,143 @@ +# SharpCompress Performance Benchmarks + +This project contains performance benchmarks for SharpCompress using [BenchmarkDotNet](https://benchmarkdotnet.org/). + +## Overview + +The benchmarks test all major archive formats supported by SharpCompress: +- **Zip**: Read (Archive & Reader API) and Write operations, each with sync and async variants +- **Tar**: Read (Archive & Reader API) and Write operations, including Tar.GZip, each with sync and async variants +- **Rar**: Read operations (Archive & Reader API), each with sync and async variants +- **7Zip**: Read operations for LZMA and LZMA2 compression, each with sync and async variants +- **GZip**: Compression and decompression, each with sync and async variants + +## Running Benchmarks + +### Run all benchmarks +```bash +dotnet run --project tests/SharpCompress.Performance/SharpCompress.Performance.csproj --configuration Release +``` + +### Run specific benchmark class +```bash +dotnet run --project tests/SharpCompress.Performance/SharpCompress.Performance.csproj --configuration Release -- --filter "*ZipBenchmarks*" +``` + +### Run with specific job configuration +```bash +# Quick run for testing (1 warmup, 1 iteration) +dotnet run --project tests/SharpCompress.Performance/SharpCompress.Performance.csproj --configuration Release -- --job Dry + +# Short run (3 warmup, 3 iterations) +dotnet run --project tests/SharpCompress.Performance/SharpCompress.Performance.csproj --configuration Release -- --job Short + +# Medium run (default) +dotnet run --project tests/SharpCompress.Performance/SharpCompress.Performance.csproj --configuration Release -- --job Medium +``` + +### Export results +```bash +# Export to JSON +dotnet run --project tests/SharpCompress.Performance/SharpCompress.Performance.csproj --configuration Release -- --exporters json + +# Export to multiple formats +dotnet run --project tests/SharpCompress.Performance/SharpCompress.Performance.csproj --configuration Release -- --exporters json markdown html +``` + +### List available benchmarks +```bash +dotnet run --project tests/SharpCompress.Performance/SharpCompress.Performance.csproj --configuration Release -- --list flat +``` + +## Baseline Results + +The baseline results are stored in `baseline-results.md` and represent the expected performance characteristics of the library. These results are used in CI to detect significant performance regressions. + +### Generate Baseline (Automated) + +Use the build target to generate baseline results: +```bash +dotnet run --project build/build.csproj -- generate-baseline +``` + +This will: +1. Build the performance project +2. Run all benchmarks +3. Combine the markdown reports into `baseline-results.md` +4. Clean up temporary artifacts + +### Generate Baseline (Manual) + +To manually update the baseline: +1. Run the benchmarks: `dotnet run --project tests/SharpCompress.Performance/SharpCompress.Performance.csproj --configuration Release -- --exporters markdown --artifacts baseline-output` +2. Combine the results: `cat baseline-output/results/*-report-github.md > baseline-results.md` +3. Review the changes and commit if appropriate + +## JetBrains Profiler Integration + +The performance project supports JetBrains profiler for detailed CPU and memory profiling during local development. + +### Prerequisites + +Install JetBrains profiler tools from: https://www.jetbrains.com/profiler/ + +### Run with CPU Profiling +```bash +dotnet run --project tests/SharpCompress.Performance/SharpCompress.Performance.csproj --configuration Release -- --profile --type cpu --output ./my-cpu-snapshots +``` + +### Run with Memory Profiling +```bash +dotnet run --project tests/SharpCompress.Performance/SharpCompress.Performance.csproj --configuration Release -- --profile --type memory --output ./my-memory-snapshots +``` + +### Profiler Options +- `--profile`: Enable profiler mode +- `--type cpu|memory`: Choose profiling type (default: cpu) +- `--output `: Specify snapshot output directory (default: ./profiler-snapshots) + +The profiler will run a sample benchmark and save snapshots that can be opened in JetBrains profiler tools for detailed analysis. + +## CI Integration + +The performance benchmarks run automatically in GitHub Actions on: +- Push to `master` or `release` branches +- Pull requests to `master` or `release` branches +- Manual workflow dispatch + +Results are displayed in the GitHub Actions summary and uploaded as artifacts. + +## Benchmark Configuration + +The benchmarks are configured with minimal iterations for CI efficiency: +- **Warmup Count**: 1 iteration +- **Iteration Count**: 3 iterations +- **Invocation Count**: 1 +- **Unroll Factor**: 1 +- **Toolchain**: InProcessEmitToolchain (for fast execution) + +These settings provide a good balance between speed and accuracy for CI purposes. For more accurate results, use the `Short`, `Medium`, or `Long` job configurations. + +## Memory Diagnostics + +All benchmarks include memory diagnostics using `[MemoryDiagnoser]`, which provides: +- Total allocated memory per operation +- Gen 0/1/2 collection counts + +## Understanding Results + +Key metrics in the benchmark results: +- **Mean**: Average execution time +- **Error**: Half of 99.9% confidence interval +- **StdDev**: Standard deviation +- **Allocated**: Total managed memory allocated per operation + +## Contributing + +When adding new benchmarks: +1. Create a new class in the `Benchmarks/` directory +2. Inherit from `ArchiveBenchmarkBase` for archive-related benchmarks +3. Add `[MemoryDiagnoser]` attribute to the class +4. Use `[Benchmark(Description = "...")]` for each benchmark method +5. Add `[GlobalSetup]` for one-time initialization +6. Update this README if needed diff --git a/tests/SharpCompress.Performance/SharpCompress.Performance.csproj b/tests/SharpCompress.Performance/SharpCompress.Performance.csproj new file mode 100644 index 00000000..d4ad46d2 --- /dev/null +++ b/tests/SharpCompress.Performance/SharpCompress.Performance.csproj @@ -0,0 +1,11 @@ + + + Exe + net10.0 + + + + + + + diff --git a/tests/SharpCompress.Performance/baseline-results.md b/tests/SharpCompress.Performance/baseline-results.md new file mode 100644 index 00000000..fa34fd66 --- /dev/null +++ b/tests/SharpCompress.Performance/baseline-results.md @@ -0,0 +1,49 @@ +| Method | Mean | Error | StdDev | Allocated | +|---------------------------- |---------:|---------:|---------:|----------:| +| SharpCompress_0_44_Original | 581.8 ms | 11.56 ms | 17.65 ms | 48.77 MB | +| Method | Mean | Error | StdDev | Median | Gen0 | Gen1 | Gen2 | Allocated | +|-------------------- |-----------:|----------:|----------:|-----------:|---------:|---------:|---------:|----------:| +| ZipArchiveRead | 959.2 μs | 52.16 μs | 153.78 μs | 928.7 μs | 27.3438 | 5.8594 | - | 345.75 KB | +| TarArchiveRead | 252.1 μs | 20.97 μs | 61.82 μs | 251.9 μs | 12.2070 | 5.8594 | - | 154.78 KB | +| TarGzArchiveRead | 600.9 μs | 19.25 μs | 53.98 μs | 607.8 μs | 16.6016 | 6.8359 | - | 204.95 KB | +| TarBz2ArchiveRead | NA | NA | NA | NA | NA | NA | NA | NA | +| SevenZipArchiveRead | 8,354.4 μs | 273.01 μs | 747.35 μs | 8,093.2 μs | 109.3750 | 109.3750 | 109.3750 | 787.99 KB | +| RarArchiveRead | 1,648.6 μs | 131.91 μs | 388.94 μs | 1,617.6 μs | 17.5781 | 5.8594 | - | 222.62 KB | +| Method | Mean | Error | StdDev | Gen0 | Gen1 | Gen2 | Allocated | +|--------------------------------- |-----------:|--------:|---------:|--------:|--------:|--------:|----------:| +| 'GZip: Compress 100KB' | 3,317.1 μs | 7.15 μs | 10.02 μs | 33.3333 | 33.3333 | 33.3333 | 519.31 KB | +| 'GZip: Compress 100KB (Async)' | 3,280.3 μs | 8.30 μs | 11.63 μs | 33.3333 | 33.3333 | 33.3333 | 519.46 KB | +| 'GZip: Decompress 100KB' | 432.5 μs | 2.43 μs | 3.56 μs | - | - | - | 33.92 KB | +| 'GZip: Decompress 100KB (Async)' | 442.8 μs | 1.20 μs | 1.76 μs | - | - | - | 34.24 KB | +| Method | Mean | Error | StdDev | Gen0 | Gen1 | Gen2 | Allocated | +|------------------------------------------------ |-----------:|----------:|----------:|---------:|---------:|---------:|-----------:| +| 'Rar: Extract all entries (Archive API)' | 908.2 μs | 12.42 μs | 17.01 μs | - | - | - | 90.68 KB | +| 'Rar: Extract all entries (Archive API, Async)' | 1,175.4 μs | 118.74 μs | 177.72 μs | - | - | - | 96.09 KB | +| 'Rar: Extract all entries (Reader API)' | 1,215.1 μs | 2.26 μs | 3.09 μs | - | - | - | 148.85 KB | +| 'Rar: Extract all entries (Reader API, Async)' | 1,592.0 μs | 22.58 μs | 33.10 μs | 500.0000 | 500.0000 | 500.0000 | 4776.76 KB | +| Method | Mean | Error | StdDev | Gen0 | Gen1 | Gen2 | Allocated | +|------------------------------------------------- |----------:|----------:|----------:|---------:|---------:|---------:|-----------:| +| '7Zip LZMA: Extract all entries' | 7.723 ms | 0.0111 ms | 0.0152 ms | 33.3333 | 33.3333 | 33.3333 | 272.68 KB | +| '7Zip LZMA: Extract all entries (Async)' | 35.827 ms | 0.0381 ms | 0.0546 ms | 200.0000 | 33.3333 | 33.3333 | 3402.82 KB | +| '7Zip LZMA2: Extract all entries' | 7.758 ms | 0.0074 ms | 0.0104 ms | 33.3333 | 33.3333 | 33.3333 | 272.46 KB | +| '7Zip LZMA2: Extract all entries (Async)' | 36.317 ms | 0.0345 ms | 0.0506 ms | 200.0000 | 33.3333 | 33.3333 | 3409.72 KB | +| '7Zip LZMA2 Reader: Extract all entries' | 7.706 ms | 0.0114 ms | 0.0163 ms | 33.3333 | 33.3333 | 33.3333 | 273.03 KB | +| '7Zip LZMA2 Reader: Extract all entries (Async)' | 22.951 ms | 0.0973 ms | 0.1426 ms | 100.0000 | 100.0000 | 100.0000 | 2420.81 KB | +| Method | Mean | Error | StdDev | Gen0 | Gen1 | Gen2 | Allocated | +|------------------------------------------------ |----------:|---------:|---------:|--------:|--------:|--------:|----------:| +| 'Tar: Extract all entries (Archive API)' | 40.82 μs | 0.292 μs | 0.427 μs | - | - | - | 16.36 KB | +| 'Tar: Extract all entries (Archive API, Async)' | 105.12 μs | 6.183 μs | 9.254 μs | - | - | - | 14.57 KB | +| 'Tar: Extract all entries (Reader API)' | 187.89 μs | 1.571 μs | 2.254 μs | 66.6667 | 66.6667 | 66.6667 | 341.24 KB | +| 'Tar: Extract all entries (Reader API, Async)' | 229.78 μs | 4.852 μs | 6.802 μs | 66.6667 | 66.6667 | 66.6667 | 376.64 KB | +| 'Tar.GZip: Extract all entries' | NA | NA | NA | NA | NA | NA | NA | +| 'Tar.GZip: Extract all entries (Async)' | NA | NA | NA | NA | NA | NA | NA | +| 'Tar: Create archive with small files' | 46.98 μs | 0.287 μs | 0.394 μs | - | - | - | 68.11 KB | +| 'Tar: Create archive with small files (Async)' | 53.14 μs | 0.352 μs | 0.493 μs | - | - | - | 68.11 KB | +| Method | Mean | Error | StdDev | Gen0 | Gen1 | Allocated | +|------------------------------------------------ |---------:|---------:|---------:|---------:|--------:|-----------:| +| 'Zip: Extract all entries (Archive API)' | 556.7 μs | 3.38 μs | 4.74 μs | - | - | 180.22 KB | +| 'Zip: Extract all entries (Archive API, Async)' | 615.7 μs | 15.98 μs | 22.92 μs | - | - | 125.52 KB | +| 'Zip: Extract all entries (Reader API)' | 542.2 μs | 1.10 μs | 1.46 μs | - | - | 121.04 KB | +| 'Zip: Extract all entries (Reader API, Async)' | 562.8 μs | 2.42 μs | 3.55 μs | - | - | 123.34 KB | +| 'Zip: Create archive with small files' | 271.1 μs | 12.93 μs | 18.95 μs | 166.6667 | 33.3333 | 2806.28 KB | +| 'Zip: Create archive with small files (Async)' | 394.3 μs | 25.59 μs | 36.71 μs | 166.6667 | 33.3333 | 2811.42 KB | diff --git a/tests/SharpCompress.Performance/packages.lock.json b/tests/SharpCompress.Performance/packages.lock.json new file mode 100644 index 00000000..739ca4c7 --- /dev/null +++ b/tests/SharpCompress.Performance/packages.lock.json @@ -0,0 +1,260 @@ +{ + "version": 2, + "dependencies": { + "net10.0": { + "BenchmarkDotNet": { + "type": "Direct", + "requested": "[0.15.8, )", + "resolved": "0.15.8", + "contentHash": "paCfrWxSeHqn3rUZc0spYXVFnHCF0nzRhG0nOLnyTjZYs8spsimBaaNmb3vwqvALKIplbYq/TF393vYiYSnh/Q==", + "dependencies": { + "BenchmarkDotNet.Annotations": "0.15.8", + "CommandLineParser": "2.9.1", + "Gee.External.Capstone": "2.3.0", + "Iced": "1.21.0", + "Microsoft.CodeAnalysis.CSharp": "4.14.0", + "Microsoft.Diagnostics.Runtime": "3.1.512801", + "Microsoft.Diagnostics.Tracing.TraceEvent": "3.1.21", + "Microsoft.DotNet.PlatformAbstractions": "3.1.6", + "Perfolizer": "[0.6.1]", + "System.Management": "9.0.5" + } + }, + "JetBrains.Profiler.SelfApi": { + "type": "Direct", + "requested": "[2.5.18, )", + "resolved": "2.5.18", + "contentHash": "zOHtZGrLzYey6d57XLvLUWbB7tK1WbXz2z3wSwDR6HwV5AEROmtu1di3OZE1kxGXMilPbyzXyKJT5eBwC+j32Q==", + "dependencies": { + "JetBrains.HabitatDetector": "1.5.0", + "JetBrains.Profiler.Api": "1.4.13" + } + }, + "Microsoft.NETFramework.ReferenceAssemblies": { + "type": "Direct", + "requested": "[1.0.3, )", + "resolved": "1.0.3", + "contentHash": "vUc9Npcs14QsyOD01tnv/m8sQUnGTGOw1BCmKcv77LBJY7OxhJ+zJF7UD/sCL3lYNFuqmQEVlkfS4Quif6FyYg==", + "dependencies": { + "Microsoft.NETFramework.ReferenceAssemblies.net461": "1.0.3" + } + }, + "Microsoft.SourceLink.GitHub": { + "type": "Direct", + "requested": "[10.0.300, )", + "resolved": "10.0.300", + "contentHash": "QzCtLkXVb3l4IxcpvJCbzUwMLihAmLN6vVLjQGSzYSF8d2dvXxqJAZk83RV3gYnp2egz8jRMgSR2woY3vOahTA==", + "dependencies": { + "Microsoft.Build.Tasks.Git": "10.0.300", + "Microsoft.SourceLink.Common": "10.0.300", + "System.IO.Hashing": "10.0.8" + } + }, + "Microsoft.VisualStudio.Threading.Analyzers": { + "type": "Direct", + "requested": "[17.14.15, )", + "resolved": "17.14.15", + "contentHash": "mXQPJsbuUD2ydq4/ffd8h8tSOFCXec+2xJOVNCvXjuMOq/+5EKHq3D2m2MC2+nUaXeFMSt66VS/J4HdKBixgcw==" + }, + "PolySharp": { + "type": "Direct", + "requested": "[1.16.0, )", + "resolved": "1.16.0", + "contentHash": "3kdIIceBPumwjw279FuiVMfVENT2cGASXJgcigdySsbX2dJB8ofUgG6i47yqF/k1qu6fvNR3csrSekZPviR6kQ==" + }, + "BenchmarkDotNet.Annotations": { + "type": "Transitive", + "resolved": "0.15.8", + "contentHash": "hfucY0ycAsB0SsoaZcaAp9oq5wlWBJcylvEJb9pmvdYUx6PD6S4mDiYnZWjdjAlLhIpe/xtGCwzORfzAzPqvzA==" + }, + "CommandLineParser": { + "type": "Transitive", + "resolved": "2.9.1", + "contentHash": "OE0sl1/sQ37bjVsPKKtwQlWDgqaxWgtme3xZz7JssWUzg5JpMIyHgCTY9MVMxOg48fJ1AgGT3tgdH5m/kQ5xhA==" + }, + "Gee.External.Capstone": { + "type": "Transitive", + "resolved": "2.3.0", + "contentHash": "2ap/rYmjtzCOT8hxrnEW/QeiOt+paD8iRrIcdKX0cxVwWLFa1e+JDBNeECakmccXrSFeBQuu5AV8SNkipFMMMw==" + }, + "Iced": { + "type": "Transitive", + "resolved": "1.21.0", + "contentHash": "dv5+81Q1TBQvVMSOOOmRcjJmvWcX3BZPZsIq31+RLc5cNft0IHAyNlkdb7ZarOWG913PyBoFDsDXoCIlKmLclg==" + }, + "JetBrains.FormatRipper": { + "type": "Transitive", + "resolved": "2.4.0", + "contentHash": "k5eGab1DArJH0k94ZO9oxDxg8go1KvR1oPGPzyVvfplEHetgrc2hGZ6Cken8fVsdS/Xp3hMnHd9L5MXb7JJM4A==" + }, + "JetBrains.HabitatDetector": { + "type": "Transitive", + "resolved": "1.5.0", + "contentHash": "GazZUoCunH1vrruUvy147lYfgcm2Ns6MCh76XmKZBViosvvYMLyt+JD2PWL8Lm6Ctdo3iQoPYgkStrxhrHbjEQ==", + "dependencies": { + "JetBrains.FormatRipper": "2.4.0" + } + }, + "JetBrains.Profiler.Api": { + "type": "Transitive", + "resolved": "1.4.13", + "contentHash": "abkSB+SOgLMPfOGJtl2XQ+ioSUYBtZxiKDOGCpNZ4WU7ZbAzhldrpRwDMsQnIJJcNhEawGtbhcI+Gk3ktx2RXQ==", + "dependencies": { + "JetBrains.HabitatDetector": "1.5.0" + } + }, + "Microsoft.Build.Tasks.Git": { + "type": "Transitive", + "resolved": "10.0.300", + "contentHash": "P0kaQwVZx4xIUe2FtrLyBadYNXuAljttJUPvjBYRuHhPE8L77L42KakLDkaADRiUrGspoLcMwayjrbQhYTr0zA==", + "dependencies": { + "System.IO.Hashing": "10.0.8" + } + }, + "Microsoft.CodeAnalysis.Analyzers": { + "type": "Transitive", + "resolved": "3.11.0", + "contentHash": "v/EW3UE8/lbEYHoC2Qq7AR/DnmvpgdtAMndfQNmpuIMx/Mto8L5JnuCfdBYtgvalQOtfNCnxFejxuRrryvUTsg==" + }, + "Microsoft.CodeAnalysis.Common": { + "type": "Transitive", + "resolved": "4.14.0", + "contentHash": "PC3tuwZYnC+idaPuoC/AZpEdwrtX7qFpmnrfQkgobGIWiYmGi5MCRtl5mx6QrfMGQpK78X2lfIEoZDLg/qnuHg==", + "dependencies": { + "Microsoft.CodeAnalysis.Analyzers": "3.11.0" + } + }, + "Microsoft.CodeAnalysis.CSharp": { + "type": "Transitive", + "resolved": "4.14.0", + "contentHash": "568a6wcTivauIhbeWcCwfWwIn7UV7MeHEBvFB2uzGIpM2OhJ4eM/FZ8KS0yhPoNxnSpjGzz7x7CIjTxhslojQA==", + "dependencies": { + "Microsoft.CodeAnalysis.Analyzers": "3.11.0", + "Microsoft.CodeAnalysis.Common": "[4.14.0]" + } + }, + "Microsoft.Diagnostics.NETCore.Client": { + "type": "Transitive", + "resolved": "0.2.510501", + "contentHash": "juoqJYMDs+lRrrZyOkXXMImJHneCF23cuvO4waFRd2Ds7j+ZuGIPbJm0Y/zz34BdeaGiiwGWraMUlln05W1PCQ==", + "dependencies": { + "Microsoft.Extensions.Logging": "6.0.0" + } + }, + "Microsoft.Diagnostics.Runtime": { + "type": "Transitive", + "resolved": "3.1.512801", + "contentHash": "0lMUDr2oxNZa28D6NH5BuSQEe5T9tZziIkvkD44YkkCGQXPJqvFjLq5ZQq1hYLl3RjQJrY+hR0jFgap+EWPDTw==", + "dependencies": { + "Microsoft.Diagnostics.NETCore.Client": "0.2.410101" + } + }, + "Microsoft.Diagnostics.Tracing.TraceEvent": { + "type": "Transitive", + "resolved": "3.1.21", + "contentHash": "/OrJFKaojSR6TkUKtwh8/qA9XWNtxLrXMqvEb89dBSKCWjaGVTbKMYodIUgF5deCEtmd6GXuRerciXGl5bhZ7Q==", + "dependencies": { + "Microsoft.Diagnostics.NETCore.Client": "0.2.510501", + "System.Reflection.TypeExtensions": "4.7.0" + } + }, + "Microsoft.DotNet.PlatformAbstractions": { + "type": "Transitive", + "resolved": "3.1.6", + "contentHash": "jek4XYaQ/PGUwDKKhwR8K47Uh1189PFzMeLqO83mXrXQVIpARZCcfuDedH50YDTepBkfijCZN5U/vZi++erxtg==" + }, + "Microsoft.Extensions.DependencyInjection": { + "type": "Transitive", + "resolved": "6.0.0", + "contentHash": "k6PWQMuoBDGGHOQTtyois2u4AwyVcIwL2LaSLlTZQm2CYcJ1pxbt6jfAnpWmzENA/wfrYRI/X9DTLoUkE4AsLw==", + "dependencies": { + "Microsoft.Extensions.DependencyInjection.Abstractions": "6.0.0" + } + }, + "Microsoft.Extensions.DependencyInjection.Abstractions": { + "type": "Transitive", + "resolved": "6.0.0", + "contentHash": "xlzi2IYREJH3/m6+lUrQlujzX8wDitm4QGnUu6kUXTQAWPuZY8i+ticFJbzfqaetLA6KR/rO6Ew/HuYD+bxifg==" + }, + "Microsoft.Extensions.Logging": { + "type": "Transitive", + "resolved": "6.0.0", + "contentHash": "eIbyj40QDg1NDz0HBW0S5f3wrLVnKWnDJ/JtZ+yJDFnDj90VoPuoPmFkeaXrtu+0cKm5GRAwoDf+dBWXK0TUdg==", + "dependencies": { + "Microsoft.Extensions.DependencyInjection": "6.0.0", + "Microsoft.Extensions.DependencyInjection.Abstractions": "6.0.0", + "Microsoft.Extensions.Logging.Abstractions": "6.0.0", + "Microsoft.Extensions.Options": "6.0.0" + } + }, + "Microsoft.Extensions.Logging.Abstractions": { + "type": "Transitive", + "resolved": "6.0.0", + "contentHash": "/HggWBbTwy8TgebGSX5DBZ24ndhzi93sHUBDvP1IxbZD7FDokYzdAr6+vbWGjw2XAfR2EJ1sfKUotpjHnFWPxA==" + }, + "Microsoft.Extensions.Options": { + "type": "Transitive", + "resolved": "6.0.0", + "contentHash": "dzXN0+V1AyjOe2xcJ86Qbo233KHuLEY0njf/P2Kw8SfJU+d45HNS2ctJdnEnrWbM9Ye2eFgaC5Mj9otRMU6IsQ==", + "dependencies": { + "Microsoft.Extensions.DependencyInjection.Abstractions": "6.0.0", + "Microsoft.Extensions.Primitives": "6.0.0" + } + }, + "Microsoft.Extensions.Primitives": { + "type": "Transitive", + "resolved": "6.0.0", + "contentHash": "9+PnzmQFfEFNR9J2aDTfJGGupShHjOuGw4VUv+JB044biSHrnmCIMD+mJHmb2H7YryrfBEXDurxQ47gJZdCKNQ==" + }, + "Microsoft.NETFramework.ReferenceAssemblies.net461": { + "type": "Transitive", + "resolved": "1.0.3", + "contentHash": "AmOJZwCqnOCNp6PPcf9joyogScWLtwy0M1WkqfEQ0M9nYwyDD7EX9ZjscKS5iYnyvteX7kzSKFCKt9I9dXA6mA==" + }, + "Microsoft.SourceLink.Common": { + "type": "Transitive", + "resolved": "10.0.300", + "contentHash": "0jlkXaUGjYlWTIVPve5MftjKHnT3SlAtq9BCLV4J9IjdPrxV/+4rMlBSjfr1khG8/GC6KGojjola8E1VvWF0qQ==" + }, + "Perfolizer": { + "type": "Transitive", + "resolved": "0.6.1", + "contentHash": "CR1QmWg4XYBd1Pb7WseP+sDmV8nGPwvmowKynExTqr3OuckIGVMhvmN4LC5PGzfXqDlR295+hz/T7syA1CxEqA==", + "dependencies": { + "Pragmastat": "3.2.4" + } + }, + "Pragmastat": { + "type": "Transitive", + "resolved": "3.2.4", + "contentHash": "I5qFifWw/gaTQT52MhzjZpkm/JPlfjSeO/DTZJjO7+hTKI+0aGRgOgZ3NN6D96dDuuqbIAZSeA5RimtHjqrA2A==" + }, + "System.CodeDom": { + "type": "Transitive", + "resolved": "9.0.5", + "contentHash": "cuzLM2MWutf9ZBEMPYYfd0DXwYdvntp7VCT6a/wvbKCa2ZuvGmW74xi+YBa2mrfEieAXqM4TNKlMmSnfAfpUoQ==" + }, + "System.IO.Hashing": { + "type": "Transitive", + "resolved": "10.0.8", + "contentHash": "+dJsbPJ3FyUbTZNplFj0RCKePFizmv6ewDV46JE9q/IVH4c3xTCftHfHelLsAKf0jryIPqgMb5GpS0x7TAY3mg==" + }, + "System.Management": { + "type": "Transitive", + "resolved": "9.0.5", + "contentHash": "n6o9PZm9p25+zAzC3/48K0oHnaPKTInRrxqFq1fi/5TPbMLjuoCm/h//mS3cUmSy+9AO1Z+qsC/Ilt/ZFatv5Q==", + "dependencies": { + "System.CodeDom": "9.0.5" + } + }, + "System.Reflection.TypeExtensions": { + "type": "Transitive", + "resolved": "4.7.0", + "contentHash": "VybpaOQQhqE6siHppMktjfGBw1GCwvCqiufqmP8F1nj7fTUNtW35LOEt3UZTEsECfo+ELAl/9o9nJx3U91i7vA==" + }, + "sharpcompress": { + "type": "Project" + } + } + } +} \ No newline at end of file diff --git a/tests/SharpCompress.Test/ADCTest.cs b/tests/SharpCompress.Test/ADCTest.cs index 11bb2fb0..cb2b00a9 100644 --- a/tests/SharpCompress.Test/ADCTest.cs +++ b/tests/SharpCompress.Test/ADCTest.cs @@ -23,8 +23,10 @@ // LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, // OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN // THE SOFTWARE. +using System; using System.IO; -using SharpCompress.Compressors; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Compressors.ADC; using SharpCompress.Compressors.Deflate; using SharpCompress.Crypto; @@ -32,7 +34,7 @@ using Xunit; namespace SharpCompress.Test; -public class ADCTest : TestBase +public class AdcTest : TestBase { [Fact] public void TestBuffer() @@ -65,14 +67,14 @@ public class ADCTest : TestBase } [Fact] - public void TestADCStreamWholeChunk() + public void TestAdcStreamWholeChunk() { using var decFs = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "adc_decompressed.bin")); var decompressed = new byte[decFs.Length]; decFs.Read(decompressed, 0, decompressed.Length); using var cmpFs = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "adc_compressed.bin")); - using var decStream = new ADCStream(cmpFs, CompressionMode.Decompress); + using var decStream = new ADCStream(cmpFs); var test = new byte[262144]; decStream.Read(test, 0, test.Length); @@ -81,14 +83,14 @@ public class ADCTest : TestBase } [Fact] - public void TestADCStream() + public void TestAdcStream() { using var decFs = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "adc_decompressed.bin")); var decompressed = new byte[decFs.Length]; decFs.Read(decompressed, 0, decompressed.Length); using var cmpFs = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "adc_compressed.bin")); - using var decStream = new ADCStream(cmpFs, CompressionMode.Decompress); + using var decStream = new ADCStream(cmpFs); using var decMs = new MemoryStream(); var test = new byte[512]; var count = 0; @@ -115,11 +117,43 @@ public class ADCTest : TestBase decFs.Seek(0, SeekOrigin.Begin); - var crc32a = crcStream.Crc; + var crc32A = crcStream.Crc; - var crc32b = Crc32Stream.Compute(memory.ToArray()); + var crc32B = Crc32Stream.Compute(memory.ToArray()); - Assert.Equal(crc32, crc32a); - Assert.Equal(crc32, crc32b); + Assert.Equal(crc32, crc32A); + Assert.Equal(crc32, crc32B); } + + [Fact] + public async Task TestCrc32StreamWriteAsync() + { + var buffer = File.ReadAllBytes(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar")); + var crc32 = Crc32Stream.Compute(buffer); + + using var memory = new MemoryStream(); + using var crcStream = new Crc32Stream(memory, 0xEDB88320, 0xFFFFFFFF); + + await crcStream.WriteAsync(buffer, 0, buffer.Length, CancellationToken.None); + + Assert.Equal(buffer, memory.ToArray()); + Assert.Equal(crc32, crcStream.Crc); + } + +#if !LEGACY_DOTNET + [Fact] + public async Task TestCrc32StreamWriteMemoryAsync() + { + var buffer = File.ReadAllBytes(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar")); + var crc32 = Crc32Stream.Compute(buffer); + + using var memory = new MemoryStream(); + using var crcStream = new Crc32Stream(memory, 0xEDB88320, 0xFFFFFFFF); + + await crcStream.WriteAsync(buffer.AsMemory(), CancellationToken.None); + + Assert.Equal(buffer, memory.ToArray()); + Assert.Equal(crc32, crcStream.Crc); + } +#endif } diff --git a/tests/SharpCompress.Test/Ace/AceReaderAsyncTests.cs b/tests/SharpCompress.Test/Ace/AceReaderAsyncTests.cs new file mode 100644 index 00000000..332ca652 --- /dev/null +++ b/tests/SharpCompress.Test/Ace/AceReaderAsyncTests.cs @@ -0,0 +1,134 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Readers; +using SharpCompress.Readers.Ace; +using SharpCompress.Test.Mocks; +using Xunit; + +namespace SharpCompress.Test.Ace; + +public class AceReaderAsyncTests : ReaderTests +{ + public AceReaderAsyncTests() + { + UseExtensionInsteadOfNameToVerify = true; + UseCaseInsensitiveToVerify = true; + } + + [Fact] + public async ValueTask Ace_Uncompressed_Read_Async() => + await ReadAsync("Ace.store.ace", CompressionType.None); + + [Fact] + public async ValueTask Ace_Encrypted_Read_Async() + { + var exception = await Assert.ThrowsAsync(() => + ReadAsync("Ace.encrypted.ace") + ); + } + + [Theory] + [InlineData("Ace.method1.ace", CompressionType.AceLZ77)] + [InlineData("Ace.method1-solid.ace", CompressionType.AceLZ77)] + [InlineData("Ace.method2.ace", CompressionType.AceLZ77)] + [InlineData("Ace.method2-solid.ace", CompressionType.AceLZ77)] + public async ValueTask Ace_Unsupported_ShouldThrow_Async( + string fileName, + CompressionType compressionType + ) + { + var exception = await Assert.ThrowsAsync(() => + ReadAsync(fileName, compressionType) + ); + } + + [Theory] + [InlineData("Ace.store.largefile.ace", CompressionType.None)] + public async ValueTask Ace_LargeFileTest_Read_Async( + string fileName, + CompressionType compressionType + ) + { + await ReadForBufferBoundaryCheckAsync(fileName, compressionType); + } + + [Fact] + public async ValueTask Ace_Multi_Reader_Async() + { + var exception = await Assert.ThrowsAsync(() => + DoMultiReaderAsync( + new[] { "Ace.store.split.ace", "Ace.store.split.c01" }, + streams => AceReader.OpenAsyncReader(streams, null) + ) + ); + } + + private async Task ReadAsync(string testArchive, CompressionType expectedCompression) + { + testArchive = Path.Combine(TEST_ARCHIVES_PATH, testArchive); + using Stream stream = File.OpenRead(testArchive); + await using var reader = await ReaderFactory.OpenAsyncReader( + new AsyncOnlyStream(stream), + ReaderOptions.ForExternalStream + ); + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + Assert.Equal(expectedCompression, reader.Entry.CompressionType); + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + VerifyFiles(); + } + + private async Task ReadForBufferBoundaryCheckAsync( + string testArchive, + CompressionType expectedCompression + ) + { + testArchive = Path.Combine(TEST_ARCHIVES_PATH, testArchive); + using Stream stream = File.OpenRead(testArchive); + await using var reader = await ReaderFactory.OpenAsyncReader( + new AsyncOnlyStream(stream), + ReaderOptions.ForExternalStream with + { + LookForHeader = true, + } + ); + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + Assert.Equal(expectedCompression, reader.Entry.CompressionType); + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + CompareFilesByPath( + Path.Combine(SCRATCH_FILES_PATH, "alice29.txt"), + Path.Combine(MISC_TEST_FILES_PATH, "alice29.txt") + ); + } + + private async Task DoMultiReaderAsync( + string[] archives, + Func, IAsyncReader> readerFactory + ) + { + await using var reader = readerFactory( + archives.Select(s => Path.Combine(TEST_ARCHIVES_PATH, s)).Select(File.OpenRead) + ); + + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + } +} diff --git a/tests/SharpCompress.Test/Ace/AceReaderTests.cs b/tests/SharpCompress.Test/Ace/AceReaderTests.cs new file mode 100644 index 00000000..d6c609e9 --- /dev/null +++ b/tests/SharpCompress.Test/Ace/AceReaderTests.cs @@ -0,0 +1,58 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Readers; +using SharpCompress.Readers.Ace; +using Xunit; + +namespace SharpCompress.Test.Ace; + +public class AceReaderTests : ReaderTests +{ + public AceReaderTests() + { + UseExtensionInsteadOfNameToVerify = true; + UseCaseInsensitiveToVerify = true; + } + + [Fact] + public void Ace_Uncompressed_Read() => Read("Ace.store.ace", CompressionType.None); + + [Fact] + public void Ace_Encrypted_Read() + { + var exception = Assert.Throws(() => Read("Ace.encrypted.ace")); + } + + [Theory] + [InlineData("Ace.method1.ace", CompressionType.AceLZ77)] + [InlineData("Ace.method1-solid.ace", CompressionType.AceLZ77)] + [InlineData("Ace.method2.ace", CompressionType.AceLZ77)] + [InlineData("Ace.method2-solid.ace", CompressionType.AceLZ77)] + public void Ace_Unsupported_ShouldThrow(string fileName, CompressionType compressionType) + { + var exception = Assert.Throws(() => Read(fileName, compressionType)); + } + + [Theory] + [InlineData("Ace.store.largefile.ace", CompressionType.None)] + public void Ace_LargeFileTest_Read(string fileName, CompressionType compressionType) + { + ReadForBufferBoundaryCheck(fileName, compressionType); + } + + [Fact] + public void Ace_Multi_Reader() + { + var exception = Assert.Throws(() => + DoMultiReader( + ["Ace.store.split.ace", "Ace.store.split.c01"], + streams => AceReader.OpenReader(streams) + ) + ); + } +} diff --git a/tests/SharpCompress.Test/AdcAsyncTest.cs b/tests/SharpCompress.Test/AdcAsyncTest.cs new file mode 100644 index 00000000..611c10e0 --- /dev/null +++ b/tests/SharpCompress.Test/AdcAsyncTest.cs @@ -0,0 +1,61 @@ +using System.IO; +using System.Threading.Tasks; +using SharpCompress.Compressors.ADC; +using Xunit; + +namespace SharpCompress.Test; + +public class AdcAsyncTest : TestBase +{ + [Fact] + public async ValueTask TestAdcStreamAsyncWholeChunk() + { + using var decFs = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "adc_decompressed.bin")); + var decompressed = new byte[decFs.Length]; + decFs.Read(decompressed, 0, decompressed.Length); + + using var cmpFs = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "adc_compressed.bin")); + using var decStream = new ADCStream(cmpFs); + var test = new byte[262144]; + + await decStream.ReadAsync(test, 0, test.Length); + + Assert.Equal(decompressed, test); + } + + [Fact] + public async ValueTask TestAdcStreamAsync() + { + using var decFs = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "adc_decompressed.bin")); + var decompressed = new byte[decFs.Length]; + decFs.Read(decompressed, 0, decompressed.Length); + + using var cmpFs = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "adc_compressed.bin")); + using var decStream = new ADCStream(cmpFs); + using var decMs = new MemoryStream(); + var test = new byte[512]; + var count = 0; + + do + { + count = await decStream.ReadAsync(test, 0, test.Length); + decMs.Write(test, 0, count); + } while (count > 0); + + Assert.Equal(decompressed, decMs.ToArray()); + } + + [Fact] + public async ValueTask TestAdcStreamAsyncWithCancellation() + { + using var cmpFs = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "adc_compressed.bin")); + using var decStream = new ADCStream(cmpFs); + var test = new byte[512]; + using var cts = new System.Threading.CancellationTokenSource(); + + // Read should complete without cancellation + var bytesRead = await decStream.ReadAsync(test, 0, test.Length, cts.Token); + + Assert.True(bytesRead > 0); + } +} diff --git a/tests/SharpCompress.Test/Arc/ArcReaderAsyncTests.cs b/tests/SharpCompress.Test/Arc/ArcReaderAsyncTests.cs new file mode 100644 index 00000000..40ab08e1 --- /dev/null +++ b/tests/SharpCompress.Test/Arc/ArcReaderAsyncTests.cs @@ -0,0 +1,24 @@ +using System.Threading.Tasks; +using SharpCompress.Common; +using Xunit; + +namespace SharpCompress.Test.Arc; + +public class ArcReaderAsyncTests : ReaderTests +{ + public ArcReaderAsyncTests() + { + UseExtensionInsteadOfNameToVerify = true; + UseCaseInsensitiveToVerify = true; + } + + [Fact] + public async ValueTask Arc_Uncompressed_Read_Async() => + await ReadAsync("Arc.uncompressed.arc", CompressionType.None); + + [Fact] + public async ValueTask Arc_Squeezed_Read_Async() => await ReadAsync("Arc.squeezed.arc"); + + [Fact] + public async ValueTask Arc_Crunched_Read_Async() => await ReadAsync("Arc.crunched.arc"); +} diff --git a/tests/SharpCompress.Test/Arc/ArcReaderTests.cs b/tests/SharpCompress.Test/Arc/ArcReaderTests.cs new file mode 100644 index 00000000..c5de5ebd --- /dev/null +++ b/tests/SharpCompress.Test/Arc/ArcReaderTests.cs @@ -0,0 +1,40 @@ +using System; +using SharpCompress.Common; +using Xunit; + +namespace SharpCompress.Test.Arc; + +public class ArcReaderTests : ReaderTests +{ + public ArcReaderTests() + { + UseExtensionInsteadOfNameToVerify = true; + UseCaseInsensitiveToVerify = true; + } + + [Fact] + public void Arc_Uncompressed_Read() => Read("Arc.uncompressed.arc", CompressionType.None); + + [Fact] + public void Arc_Squeezed_Read() => Read("Arc.squeezed.arc"); + + [Fact] + public void Arc_Crunched_Read() => Read("Arc.crunched.arc"); + + [Theory] + [InlineData("Arc.crunched.largefile.arc", CompressionType.Crunched)] + public void Arc_LargeFile_ShouldThrow(string fileName, CompressionType compressionType) + { + var exception = Assert.Throws(() => + ReadForBufferBoundaryCheck(fileName, compressionType) + ); + } + + [Theory] + [InlineData("Arc.uncompressed.largefile.arc", CompressionType.None)] + [InlineData("Arc.squeezed.largefile.arc", CompressionType.Squeezed)] + public void Arc_LargeFileTest_Read(string fileName, CompressionType compressionType) + { + ReadForBufferBoundaryCheck(fileName, compressionType); + } +} diff --git a/tests/SharpCompress.Test/ArchiveFactoryTests.cs b/tests/SharpCompress.Test/ArchiveFactoryTests.cs new file mode 100644 index 00000000..15d53cc5 --- /dev/null +++ b/tests/SharpCompress.Test/ArchiveFactoryTests.cs @@ -0,0 +1,737 @@ +using System; +using System.IO; +using System.Text; +using System.Threading.Tasks; +using SharpCompress.Archives; +using SharpCompress.Common; +using SharpCompress.Factories; +using SharpCompress.Readers; +using SharpCompress.Test.Mocks; +using Xunit; + +namespace SharpCompress.Test; + +public class ArchiveFactoryTests : TestBase +{ + [Theory] + [InlineData("Zip.deflate.zip", typeof(ZipFactory))] + [InlineData("Tar.noEmptyDirs.tar", typeof(TarFactory))] + [InlineData("Rar.rar", typeof(RarFactory))] + [InlineData("7Zip.nonsolid.7z", typeof(SevenZipFactory))] + public async ValueTask FindFactoryAsync_String_ReturnsExpectedFactory( + string archiveName, + System.Type expectedFactoryType + ) + { + var factory = await ArchiveFactory.FindFactoryAsync( + Path.Combine(TEST_ARCHIVES_PATH, archiveName) + ); + + Assert.IsType(expectedFactoryType, factory); + } + + [Theory] + [InlineData("Zip.deflate.zip", typeof(ZipFactory))] + [InlineData("Tar.noEmptyDirs.tar", typeof(TarFactory))] + [InlineData("Rar.rar", typeof(RarFactory))] + [InlineData("7Zip.nonsolid.7z", typeof(SevenZipFactory))] + public async ValueTask FindFactoryAsync_FileInfo_ReturnsExpectedFactory( + string archiveName, + System.Type expectedFactoryType + ) + { + var factory = await ArchiveFactory.FindFactoryAsync( + new FileInfo(Path.Combine(TEST_ARCHIVES_PATH, archiveName)) + ); + + Assert.IsType(expectedFactoryType, factory); + } + + [Theory] + [InlineData("Zip.deflate.zip", typeof(ZipFactory))] + [InlineData("Tar.noEmptyDirs.tar", typeof(TarFactory))] + public async ValueTask FindFactoryAsync_Stream_PreservesPosition( + string archiveName, + System.Type expectedFactoryType + ) + { + using var stream = CreatePrefixedArchiveStream(archiveName, 7); + var startPosition = stream.Position; + + var factory = await ArchiveFactory.FindFactoryAsync(stream); + + Assert.IsType(expectedFactoryType, factory); + Assert.Equal(startPosition, stream.Position); + } + + [Fact] + public void OpenArchive_StreamCollection_Throws_On_NonSeekable_Stream() + { + using var nonSeekable = new ForwardOnlyStream(new MemoryStream()); + using var seekable = new MemoryStream(); + + Assert.Throws(() => ArchiveFactory.OpenArchive([nonSeekable, seekable])); + } + + [Fact] + public async ValueTask OpenAsyncArchive_StreamCollection_Throws_On_NonSeekable_Stream() + { + using var nonSeekable = new ForwardOnlyStream(new MemoryStream()); + using var seekable = new MemoryStream(); + + await Assert.ThrowsAsync(() => + ArchiveFactory.OpenAsyncArchive([nonSeekable, seekable]).AsTask() + ); + } + + [Fact] + public async ValueTask FindFactoryAsync_InvalidData_ThrowsArchiveOperationException() + { + using var stream = new MemoryStream(Encoding.ASCII.GetBytes("not an archive")); + + await Assert.ThrowsAsync(async () => + await ArchiveFactory.FindFactoryAsync(stream) + ); + } + + [Fact] + public void OpenArchive_Stream_Throws_On_Unreadable_Stream() + { + using var unreadable = new TestStream(new MemoryStream(), false, true, true); + + Assert.Throws(() => ArchiveFactory.OpenArchive(unreadable)); + } + + [Fact] + public async ValueTask OpenAsyncArchive_Stream_Throws_On_Unreadable_Stream() + { + using var unreadable = new TestStream(new MemoryStream(), false, true, true); + + await Assert.ThrowsAsync(() => + ArchiveFactory.OpenAsyncArchive(unreadable).AsTask() + ); + } + + [Theory] + [InlineData("Zip.deflate.zip")] + [InlineData("Tar.noEmptyDirs.tar")] + [InlineData("Rar.rar")] + [InlineData("7Zip.nonsolid.7z")] + public void OpenArchive_SingleVolume_VolumeFileName_MatchesPath(string archiveName) + { + var archivePath = GetTestArchivePath(archiveName); + using var archive = ArchiveFactory.OpenArchive(archivePath); + + var volume = Assert.Single(archive.Volumes); + Assert.Equal(archivePath, volume.FileName); + } + + [Theory] + [InlineData("Zip.deflate.zip", ArchiveType.Zip)] + [InlineData("Tar.noEmptyDirs.tar", ArchiveType.Tar)] + [InlineData("Rar.rar", ArchiveType.Rar)] + [InlineData("7Zip.nonsolid.7z", ArchiveType.SevenZip)] + public void IsArchive_String_ReturnsExpectedType(string archiveName, ArchiveType expectedType) + { + var result = ArchiveFactory.IsArchive( + Path.Combine(TEST_ARCHIVES_PATH, archiveName), + out var type + ); + + Assert.True(result); + Assert.Equal(expectedType, type); + } + + [Theory] + [InlineData("Zip.deflate.zip", ArchiveType.Zip)] + [InlineData("Tar.noEmptyDirs.tar", ArchiveType.Tar)] + public void IsArchive_Stream_PreservesPosition(string archiveName, ArchiveType expectedType) + { + using var stream = CreatePrefixedArchiveStream(archiveName, 11); + var startPosition = stream.Position; + + var result = ArchiveFactory.IsArchive(stream, out var type); + + Assert.True(result); + Assert.Equal(expectedType, type); + Assert.Equal(startPosition, stream.Position); + } + + [Theory] + [InlineData("7Zip.LZMA2.exe", ArchiveType.SevenZip)] + [InlineData("Rar.jpeg.jpg", ArchiveType.Rar)] + public void IsArchive_WithReaderOptions_ReturnsExpectedType( + string archiveName, + ArchiveType expectedType + ) + { + var result = ArchiveFactory.IsArchive( + GetTestArchivePath(archiveName), + ReaderOptions.ForFilePath.WithLookForHeader(true), + out var type + ); + + Assert.True(result); + Assert.Equal(expectedType, type); + } + + [Theory] + [InlineData("7Zip.LZMA2.exe", ArchiveType.SevenZip)] + [InlineData("Rar.jpeg.jpg", ArchiveType.Rar)] + public async ValueTask IsArchiveAsync_WithReaderOptions_ReturnsExpectedType( + string archiveName, + ArchiveType expectedType + ) + { + var result = await ArchiveFactory.IsArchiveAsync( + GetTestArchivePath(archiveName), + ReaderOptions.ForFilePath.WithLookForHeader(true) + ); + + Assert.True(result.IsArchive); + Assert.Equal(expectedType, result.Type); + } + + [Theory] + [InlineData("Zip.deflate.zip", ArchiveType.Zip)] + [InlineData("Tar.noEmptyDirs.tar", ArchiveType.Tar)] + [InlineData("Rar.rar", ArchiveType.Rar)] + [InlineData("7Zip.nonsolid.7z", ArchiveType.SevenZip)] + public async ValueTask IsArchiveAsync_String_ReturnsExpectedType( + string archiveName, + ArchiveType expectedType + ) + { + var result = await ArchiveFactory.IsArchiveAsync( + Path.Combine(TEST_ARCHIVES_PATH, archiveName) + ); + + Assert.True(result.IsArchive); + Assert.Equal(expectedType, result.Type); + } + + [Theory] + [InlineData("Zip.deflate.zip", ArchiveType.Zip)] + [InlineData("Tar.noEmptyDirs.tar", ArchiveType.Tar)] + public async ValueTask IsArchiveAsync_Stream_PreservesPosition( + string archiveName, + ArchiveType expectedType + ) + { + using var stream = CreatePrefixedArchiveStream(archiveName, 11); + var startPosition = stream.Position; + + var result = await ArchiveFactory.IsArchiveAsync(stream); + + Assert.True(result.IsArchive); + Assert.Equal(expectedType, result.Type); + Assert.Equal(startPosition, stream.Position); + } + + [Fact] + public async ValueTask IsArchiveAsync_InvalidData_ReturnsFalseAndNullType() + { + using var stream = new MemoryStream(Encoding.ASCII.GetBytes("not an archive")); + + var result = await ArchiveFactory.IsArchiveAsync(stream); + + Assert.False(result.IsArchive); + Assert.Null(result.Type); + Assert.Equal(0, stream.Position); + } + + [Theory] + [InlineData("Zip.deflate.zip", ArchiveType.Zip, true)] + [InlineData("Tar.noEmptyDirs.tar", ArchiveType.Tar, true)] + [InlineData("Rar.rar", ArchiveType.Rar, true)] + [InlineData("7Zip.nonsolid.7z", ArchiveType.SevenZip, true)] + [InlineData("Ace.store.ace", ArchiveType.Ace, false)] + [InlineData("Arc.uncompressed.arc", ArchiveType.Arc, false)] + public void GetArchiveInformation_ReturnsExpectedInfo( + string archiveName, + ArchiveType expectedType, + bool expectedRandomAccess + ) + { + var info = ArchiveFactory.GetArchiveInformation( + Path.Combine(TEST_ARCHIVES_PATH, archiveName) + ); + + Assert.NotNull(info); + Assert.Equal(expectedType, info.Type); + Assert.Equal(expectedRandomAccess, info.SupportsRandomAccess); + } + + [Theory] + [InlineData("7Zip.LZMA2.exe", ArchiveType.SevenZip, true)] + [InlineData("Rar.jpeg.jpg", ArchiveType.Rar, true)] + public void GetArchiveInformation_WithReaderOptions_ReturnsExpectedInfo( + string archiveName, + ArchiveType expectedType, + bool expectedRandomAccess + ) + { + var info = ArchiveFactory.GetArchiveInformation( + GetTestArchivePath(archiveName), + ReaderOptions.ForFilePath.WithLookForHeader(true) + ); + + Assert.NotNull(info); + Assert.Equal(expectedType, info.Type); + Assert.Equal(expectedRandomAccess, info.SupportsRandomAccess); + } + + [Theory] + [InlineData("Zip.deflate.zip", ArchiveType.Zip, true)] + [InlineData("Tar.noEmptyDirs.tar", ArchiveType.Tar, true)] + [InlineData("Rar.rar", ArchiveType.Rar, true)] + [InlineData("7Zip.nonsolid.7z", ArchiveType.SevenZip, true)] + [InlineData("Ace.store.ace", ArchiveType.Ace, false)] + [InlineData("Arc.uncompressed.arc", ArchiveType.Arc, false)] + public async ValueTask GetArchiveInformationAsync_ReturnsExpectedInfo( + string archiveName, + ArchiveType expectedType, + bool expectedRandomAccess + ) + { + var info = await ArchiveFactory.GetArchiveInformationAsync( + Path.Combine(TEST_ARCHIVES_PATH, archiveName) + ); + + Assert.NotNull(info); + Assert.Equal(expectedType, info.Type); + Assert.Equal(expectedRandomAccess, info.SupportsRandomAccess); + } + + [Theory] + [InlineData("7Zip.LZMA2.exe", ArchiveType.SevenZip, true)] + [InlineData("Rar.jpeg.jpg", ArchiveType.Rar, true)] + public async ValueTask GetArchiveInformationAsync_WithReaderOptions_ReturnsExpectedInfo( + string archiveName, + ArchiveType expectedType, + bool expectedRandomAccess + ) + { + var info = await ArchiveFactory.GetArchiveInformationAsync( + GetTestArchivePath(archiveName), + ReaderOptions.ForFilePath.WithLookForHeader(true) + ); + + Assert.NotNull(info); + Assert.Equal(expectedType, info.Type); + Assert.Equal(expectedRandomAccess, info.SupportsRandomAccess); + } + + [Theory] + [InlineData("64bitstream.zip.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.ARM.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.ARM64.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.ARMT.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.BCJ.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.BCJ2.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.BZip2.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.Copy.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.EmptyStream.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.Filters.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.IA64.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.LZMA.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.LZMA.Aes.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.LZMA2.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.LZMA2.Aes.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.PPC.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.PPMd.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.RISCV.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.SPARC.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.Tar.tar", ArchiveType.Tar, true)] + [InlineData("7Zip.Tar.tar.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.ZSTD.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.delta.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.delta.distance.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.encryptedFiles.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.eos.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.nonsolid.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.solid.1block.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.solid.7z", ArchiveType.SevenZip, true)] + [InlineData("Ace.encrypted.ace", ArchiveType.Ace, false)] + [InlineData("Ace.method1-solid.ace", ArchiveType.Ace, false)] + [InlineData("Ace.method1.ace", ArchiveType.Ace, false)] + [InlineData("Ace.method2-solid.ace", ArchiveType.Ace, false)] + [InlineData("Ace.method2.ace", ArchiveType.Ace, false)] + [InlineData("Ace.store.ace", ArchiveType.Ace, false)] + [InlineData("Ace.store.largefile.ace", ArchiveType.Ace, false)] + [InlineData("Arc.crunched.arc", ArchiveType.Arc, false)] + [InlineData("Arc.crunched.largefile.arc", ArchiveType.Arc, false)] + [InlineData("Arc.squashed.arc", ArchiveType.Arc, false)] + [InlineData("Arc.squashed.largefile.arc", ArchiveType.Arc, false)] + [InlineData("Arc.squeezed.arc", ArchiveType.Arc, false)] + [InlineData("Arc.squeezed.largefile.arc", ArchiveType.Arc, false)] + [InlineData("Arc.uncompressed.arc", ArchiveType.Arc, false)] + [InlineData("Arc.uncompressed.largefile.arc", ArchiveType.Arc, false)] + [InlineData("Arj.encrypted.arj", ArchiveType.Arj, false)] + [InlineData("Arj.method1.arj", ArchiveType.Arj, false)] + [InlineData("Arj.method1.largefile.arj", ArchiveType.Arj, false)] + [InlineData("Arj.method2.arj", ArchiveType.Arj, false)] + [InlineData("Arj.method2.largefile.arj", ArchiveType.Arj, false)] + [InlineData("Arj.method3.arj", ArchiveType.Arj, false)] + [InlineData("Arj.method3.largefile.arj", ArchiveType.Arj, false)] + [InlineData("Arj.method4.arj", ArchiveType.Arj, false)] + [InlineData("Arj.method4.largefile.arj", ArchiveType.Arj, false)] + [InlineData("Arj.store.arj", ArchiveType.Arj, false)] + [InlineData("Arj.store.largefile.arj", ArchiveType.Arj, false)] + [InlineData("Issue_685.zip", ArchiveType.Zip, true)] + [InlineData("PrePostHeaders.zip", ArchiveType.Zip, true)] + [InlineData("Rar.Audio_program.rar", ArchiveType.Rar, true)] + [InlineData("Rar.Encrypted.rar", ArchiveType.Rar, true)] + [InlineData("Rar.comment.rar", ArchiveType.Rar, true)] + [InlineData("Rar.encrypted_filesAndHeader.rar", ArchiveType.Rar, true)] + [InlineData("Rar.encrypted_filesOnly.rar", ArchiveType.Rar, true)] + [InlineData("Rar.issue1050.rar", ArchiveType.Rar, true)] + [InlineData("Rar.malformed_512byte.rar", ArchiveType.Rar, true)] + [InlineData("Rar.none.rar", ArchiveType.Rar, true)] + [InlineData("Rar.rar", ArchiveType.Rar, true)] + [InlineData("Rar.solid.rar", ArchiveType.Rar, true)] + [InlineData("Rar.test_invalid_exttime.rar", ArchiveType.Rar, true)] + [InlineData("Rar15.rar", ArchiveType.Rar, true)] + [InlineData("Rar2.rar", ArchiveType.Rar, true)] + [InlineData("Rar4.rar", ArchiveType.Rar, true)] + [InlineData("Rar5.comment.rar", ArchiveType.Rar, true)] + [InlineData("Rar5.crc_blake2.rar", ArchiveType.Rar, true)] + [InlineData("Rar5.encrypted_filesAndHeader.rar", ArchiveType.Rar, true)] + [InlineData("Rar5.encrypted_filesOnly.rar", ArchiveType.Rar, true)] + [InlineData("Rar5.none.rar", ArchiveType.Rar, true)] + [InlineData("Rar5.rar", ArchiveType.Rar, true)] + [InlineData("Rar5.solid.rar", ArchiveType.Rar, true)] + [InlineData("Tar.ContainsRar.tar", ArchiveType.Tar, true)] + [InlineData("Tar.ContainsTarGz.tar", ArchiveType.Tar, true)] + [InlineData("Tar.Empty.tar", ArchiveType.Tar, true)] + [InlineData("Tar.LongPathsWithLongNameExtension.tar", ArchiveType.Tar, true)] + [InlineData("Tar.mod.tar", ArchiveType.Tar, true)] + [InlineData("Tar.noEmptyDirs.tar", ArchiveType.Tar, true)] + [InlineData("Tar.noEmptyDirs.tar.bz2", ArchiveType.Tar, true)] + [InlineData("Tar.noEmptyDirs.tar.lz", ArchiveType.Tar, true)] + [InlineData("Tar.oldgnu.tar.gz", ArchiveType.Tar, true)] + [InlineData("Tar.tar", ArchiveType.Tar, true)] + [InlineData("Tar.tar.Z", ArchiveType.Tar, true)] + [InlineData("Tar.tar.bz2", ArchiveType.Tar, true)] + [InlineData("Tar.tar.gz", ArchiveType.Tar, true)] + [InlineData("Tar.tar.lz", ArchiveType.Tar, true)] + [InlineData("Tar.tar.xz", ArchiveType.Tar, true)] + [InlineData("Tar.tar.zst", ArchiveType.Tar, true)] + [InlineData("TarCorrupted.tar", ArchiveType.Tar, true)] + [InlineData("TarWithSymlink.tar.gz", ArchiveType.Tar, true)] + [InlineData("WinZip26.zip", ArchiveType.Zip, true)] + [InlineData("WinZip26_BZip2.zipx", ArchiveType.Zip, true)] + [InlineData("WinZip26_LZMA.zipx", ArchiveType.Zip, true)] + [InlineData("WinZip27_XZ.zipx", ArchiveType.Zip, true)] + [InlineData("WinZip27_ZSTD.zipx", ArchiveType.Zip, true)] + [InlineData("Zip.644.zip", ArchiveType.Zip, true)] + [InlineData("Zip.EntryComment.zip", ArchiveType.Zip, true)] + [InlineData("Zip.Evil.zip", ArchiveType.Zip, true)] + [InlineData("Zip.LongComment.zip", ArchiveType.Zip, true)] + [InlineData("Zip.UnicodePathExtra.zip", ArchiveType.Zip, true)] + [InlineData("Zip.badlocalextra.zip", ArchiveType.Zip, true)] + [InlineData("Zip.bzip2.dd.zip", ArchiveType.Zip, true)] + [InlineData("Zip.bzip2.noEmptyDirs.zip", ArchiveType.Zip, true)] + [InlineData("Zip.bzip2.pkware.zip", ArchiveType.Zip, true)] + [InlineData("Zip.bzip2.zip", ArchiveType.Zip, true)] + [InlineData("Zip.deflate.WinzipAES.zip", ArchiveType.Zip, true)] + [InlineData("Zip.deflate.WinzipAES2.zip", ArchiveType.Zip, true)] + [InlineData("Zip.deflate.dd-.zip", ArchiveType.Zip, true)] + [InlineData("Zip.deflate.dd.zip", ArchiveType.Zip, true)] + [InlineData("Zip.deflate.mod.zip", ArchiveType.Zip, true)] + [InlineData("Zip.deflate.mod2.zip", ArchiveType.Zip, true)] + [InlineData("Zip.deflate.noEmptyDirs.zip", ArchiveType.Zip, true)] + [InlineData("Zip.deflate.pkware.zip", ArchiveType.Zip, true)] + [InlineData("Zip.deflate.zip", ArchiveType.Zip, true)] + [InlineData("Zip.deflate64.zip", ArchiveType.Zip, true)] + [InlineData("Zip.implode.zip", ArchiveType.Zip, true)] + [InlineData("Zip.lzma.WinzipAES.zip", ArchiveType.Zip, true)] + [InlineData("Zip.lzma.dd.zip", ArchiveType.Zip, true)] + [InlineData("Zip.lzma.empty.zip", ArchiveType.Zip, true)] + [InlineData("Zip.lzma.noEmptyDirs.zip", ArchiveType.Zip, true)] + [InlineData("Zip.lzma.zip", ArchiveType.Zip, true)] + [InlineData("Zip.none.datadescriptors.zip", ArchiveType.Zip, true)] + [InlineData("Zip.none.encrypted.zip", ArchiveType.Zip, true)] + [InlineData("Zip.none.issue86.zip", ArchiveType.Zip, true)] + [InlineData("Zip.none.noEmptyDirs.zip", ArchiveType.Zip, true)] + [InlineData("Zip.none.zip", ArchiveType.Zip, true)] + [InlineData("Zip.ppmd.dd.zip", ArchiveType.Zip, true)] + [InlineData("Zip.ppmd.noEmptyDirs.zip", ArchiveType.Zip, true)] + [InlineData("Zip.ppmd.zip", ArchiveType.Zip, true)] + [InlineData("Zip.reduce1.zip", ArchiveType.Zip, true)] + [InlineData("Zip.reduce2.zip", ArchiveType.Zip, true)] + [InlineData("Zip.reduce3.zip", ArchiveType.Zip, true)] + [InlineData("Zip.reduce4.zip", ArchiveType.Zip, true)] + [InlineData("Zip.shrink.zip", ArchiveType.Zip, true)] + [InlineData("Zip.uncompressed.zip", ArchiveType.Zip, true)] + [InlineData("Zip.zip64.compressedonly.zip", ArchiveType.Zip, true)] + [InlineData("Zip.zip64.zip", ArchiveType.Zip, true)] + [InlineData("Zip.zipx", ArchiveType.Zip, true)] + [InlineData("Zip.zstd.WinzipAES.mixed.zip", ArchiveType.Zip, true)] + [InlineData("large_test.txt.Z", ArchiveType.Lzw, false)] + [InlineData("test_477.zip", ArchiveType.Zip, true)] + [InlineData("ustar with long names.tar", ArchiveType.Tar, true)] + [InlineData("very long filename.tar", ArchiveType.Tar, true)] + [InlineData("zipcrypto.zip", ArchiveType.Zip, true)] + [InlineData("SharpCompress.AES.zip", ArchiveType.Zip, true)] + [InlineData("SharpCompress.Encrypted.zip", ArchiveType.Zip, true)] + [InlineData("SharpCompress.Encrypted2.zip", ArchiveType.Zip, true)] + public void GetArchiveInformation_DetectsSingleFileTestArchives( + string archiveName, + ArchiveType expectedType, + bool expectedSeekable + ) + { + var info = ArchiveFactory.GetArchiveInformation(GetTestArchivePath(archiveName)); + + Assert.NotNull(info); + Assert.Equal(expectedType, info.Type); + Assert.Equal(expectedSeekable, info.SupportsRandomAccess); + } + + [Theory] + [InlineData("64bitstream.zip.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.ARM.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.ARM64.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.ARMT.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.BCJ.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.BCJ2.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.BZip2.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.Copy.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.EmptyStream.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.Filters.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.IA64.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.LZMA.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.LZMA.Aes.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.LZMA2.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.LZMA2.Aes.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.PPC.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.PPMd.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.RISCV.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.SPARC.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.Tar.tar", ArchiveType.Tar, true)] + [InlineData("7Zip.Tar.tar.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.ZSTD.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.delta.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.delta.distance.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.encryptedFiles.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.eos.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.nonsolid.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.solid.1block.7z", ArchiveType.SevenZip, true)] + [InlineData("7Zip.solid.7z", ArchiveType.SevenZip, true)] + [InlineData("Ace.encrypted.ace", ArchiveType.Ace, false)] + [InlineData("Ace.method1-solid.ace", ArchiveType.Ace, false)] + [InlineData("Ace.method1.ace", ArchiveType.Ace, false)] + [InlineData("Ace.method2-solid.ace", ArchiveType.Ace, false)] + [InlineData("Ace.method2.ace", ArchiveType.Ace, false)] + [InlineData("Ace.store.ace", ArchiveType.Ace, false)] + [InlineData("Ace.store.largefile.ace", ArchiveType.Ace, false)] + [InlineData("Arc.crunched.arc", ArchiveType.Arc, false)] + [InlineData("Arc.crunched.largefile.arc", ArchiveType.Arc, false)] + [InlineData("Arc.squashed.arc", ArchiveType.Arc, false)] + [InlineData("Arc.squashed.largefile.arc", ArchiveType.Arc, false)] + [InlineData("Arc.squeezed.arc", ArchiveType.Arc, false)] + [InlineData("Arc.squeezed.largefile.arc", ArchiveType.Arc, false)] + [InlineData("Arc.uncompressed.arc", ArchiveType.Arc, false)] + [InlineData("Arc.uncompressed.largefile.arc", ArchiveType.Arc, false)] + [InlineData("Arj.encrypted.arj", ArchiveType.Arj, false)] + [InlineData("Arj.method1.arj", ArchiveType.Arj, false)] + [InlineData("Arj.method1.largefile.arj", ArchiveType.Arj, false)] + [InlineData("Arj.method2.arj", ArchiveType.Arj, false)] + [InlineData("Arj.method2.largefile.arj", ArchiveType.Arj, false)] + [InlineData("Arj.method3.arj", ArchiveType.Arj, false)] + [InlineData("Arj.method3.largefile.arj", ArchiveType.Arj, false)] + [InlineData("Arj.method4.arj", ArchiveType.Arj, false)] + [InlineData("Arj.method4.largefile.arj", ArchiveType.Arj, false)] + [InlineData("Arj.store.arj", ArchiveType.Arj, false)] + [InlineData("Arj.store.largefile.arj", ArchiveType.Arj, false)] + [InlineData("Issue_685.zip", ArchiveType.Zip, true)] + [InlineData("PrePostHeaders.zip", ArchiveType.Zip, true)] + [InlineData("Rar.Audio_program.rar", ArchiveType.Rar, true)] + [InlineData("Rar.Encrypted.rar", ArchiveType.Rar, true)] + [InlineData("Rar.comment.rar", ArchiveType.Rar, true)] + [InlineData("Rar.encrypted_filesAndHeader.rar", ArchiveType.Rar, true)] + [InlineData("Rar.encrypted_filesOnly.rar", ArchiveType.Rar, true)] + [InlineData("Rar.issue1050.rar", ArchiveType.Rar, true)] + [InlineData("Rar.malformed_512byte.rar", ArchiveType.Rar, true)] + [InlineData("Rar.none.rar", ArchiveType.Rar, true)] + [InlineData("Rar.rar", ArchiveType.Rar, true)] + [InlineData("Rar.solid.rar", ArchiveType.Rar, true)] + [InlineData("Rar.test_invalid_exttime.rar", ArchiveType.Rar, true)] + [InlineData("Rar15.rar", ArchiveType.Rar, true)] + [InlineData("Rar2.rar", ArchiveType.Rar, true)] + [InlineData("Rar4.rar", ArchiveType.Rar, true)] + [InlineData("Rar5.comment.rar", ArchiveType.Rar, true)] + [InlineData("Rar5.crc_blake2.rar", ArchiveType.Rar, true)] + [InlineData("Rar5.encrypted_filesAndHeader.rar", ArchiveType.Rar, true)] + [InlineData("Rar5.encrypted_filesOnly.rar", ArchiveType.Rar, true)] + [InlineData("Rar5.none.rar", ArchiveType.Rar, true)] + [InlineData("Rar5.rar", ArchiveType.Rar, true)] + [InlineData("Rar5.solid.rar", ArchiveType.Rar, true)] + [InlineData("Tar.ContainsRar.tar", ArchiveType.Tar, true)] + [InlineData("Tar.ContainsTarGz.tar", ArchiveType.Tar, true)] + [InlineData("Tar.Empty.tar", ArchiveType.Tar, true)] + [InlineData("Tar.LongPathsWithLongNameExtension.tar", ArchiveType.Tar, true)] + [InlineData("Tar.mod.tar", ArchiveType.Tar, true)] + [InlineData("Tar.noEmptyDirs.tar", ArchiveType.Tar, true)] + [InlineData("Tar.noEmptyDirs.tar.bz2", ArchiveType.Tar, true)] + [InlineData("Tar.noEmptyDirs.tar.lz", ArchiveType.Tar, true)] + [InlineData("Tar.oldgnu.tar.gz", ArchiveType.Tar, true)] + [InlineData("Tar.tar", ArchiveType.Tar, true)] + [InlineData("Tar.tar.Z", ArchiveType.Tar, true)] + [InlineData("Tar.tar.bz2", ArchiveType.Tar, true)] + [InlineData("Tar.tar.gz", ArchiveType.Tar, true)] + [InlineData("Tar.tar.lz", ArchiveType.Tar, true)] + [InlineData("Tar.tar.xz", ArchiveType.Tar, true)] + [InlineData("Tar.tar.zst", ArchiveType.Tar, true)] + [InlineData("TarCorrupted.tar", ArchiveType.Tar, true)] + [InlineData("TarWithSymlink.tar.gz", ArchiveType.Tar, true)] + [InlineData("WinZip26.zip", ArchiveType.Zip, true)] + [InlineData("WinZip26_BZip2.zipx", ArchiveType.Zip, true)] + [InlineData("WinZip26_LZMA.zipx", ArchiveType.Zip, true)] + [InlineData("WinZip27_XZ.zipx", ArchiveType.Zip, true)] + [InlineData("WinZip27_ZSTD.zipx", ArchiveType.Zip, true)] + [InlineData("Zip.644.zip", ArchiveType.Zip, true)] + [InlineData("Zip.EntryComment.zip", ArchiveType.Zip, true)] + [InlineData("Zip.Evil.zip", ArchiveType.Zip, true)] + [InlineData("Zip.LongComment.zip", ArchiveType.Zip, true)] + [InlineData("Zip.UnicodePathExtra.zip", ArchiveType.Zip, true)] + [InlineData("Zip.badlocalextra.zip", ArchiveType.Zip, true)] + [InlineData("Zip.bzip2.dd.zip", ArchiveType.Zip, true)] + [InlineData("Zip.bzip2.noEmptyDirs.zip", ArchiveType.Zip, true)] + [InlineData("Zip.bzip2.pkware.zip", ArchiveType.Zip, true)] + [InlineData("Zip.bzip2.zip", ArchiveType.Zip, true)] + [InlineData("Zip.deflate.WinzipAES.zip", ArchiveType.Zip, true)] + [InlineData("Zip.deflate.WinzipAES2.zip", ArchiveType.Zip, true)] + [InlineData("Zip.deflate.dd-.zip", ArchiveType.Zip, true)] + [InlineData("Zip.deflate.dd.zip", ArchiveType.Zip, true)] + [InlineData("Zip.deflate.mod.zip", ArchiveType.Zip, true)] + [InlineData("Zip.deflate.mod2.zip", ArchiveType.Zip, true)] + [InlineData("Zip.deflate.noEmptyDirs.zip", ArchiveType.Zip, true)] + [InlineData("Zip.deflate.pkware.zip", ArchiveType.Zip, true)] + [InlineData("Zip.deflate.zip", ArchiveType.Zip, true)] + [InlineData("Zip.deflate64.zip", ArchiveType.Zip, true)] + [InlineData("Zip.implode.zip", ArchiveType.Zip, true)] + [InlineData("Zip.lzma.WinzipAES.zip", ArchiveType.Zip, true)] + [InlineData("Zip.lzma.dd.zip", ArchiveType.Zip, true)] + [InlineData("Zip.lzma.empty.zip", ArchiveType.Zip, true)] + [InlineData("Zip.lzma.noEmptyDirs.zip", ArchiveType.Zip, true)] + [InlineData("Zip.lzma.zip", ArchiveType.Zip, true)] + [InlineData("Zip.none.datadescriptors.zip", ArchiveType.Zip, true)] + [InlineData("Zip.none.encrypted.zip", ArchiveType.Zip, true)] + [InlineData("Zip.none.issue86.zip", ArchiveType.Zip, true)] + [InlineData("Zip.none.noEmptyDirs.zip", ArchiveType.Zip, true)] + [InlineData("Zip.none.zip", ArchiveType.Zip, true)] + [InlineData("Zip.ppmd.dd.zip", ArchiveType.Zip, true)] + [InlineData("Zip.ppmd.noEmptyDirs.zip", ArchiveType.Zip, true)] + [InlineData("Zip.ppmd.zip", ArchiveType.Zip, true)] + [InlineData("Zip.reduce1.zip", ArchiveType.Zip, true)] + [InlineData("Zip.reduce2.zip", ArchiveType.Zip, true)] + [InlineData("Zip.reduce3.zip", ArchiveType.Zip, true)] + [InlineData("Zip.reduce4.zip", ArchiveType.Zip, true)] + [InlineData("Zip.shrink.zip", ArchiveType.Zip, true)] + [InlineData("Zip.uncompressed.zip", ArchiveType.Zip, true)] + [InlineData("Zip.zip64.compressedonly.zip", ArchiveType.Zip, true)] + [InlineData("Zip.zip64.zip", ArchiveType.Zip, true)] + [InlineData("Zip.zipx", ArchiveType.Zip, true)] + [InlineData("Zip.zstd.WinzipAES.mixed.zip", ArchiveType.Zip, true)] + [InlineData("large_test.txt.Z", ArchiveType.Lzw, false)] + [InlineData("test_477.zip", ArchiveType.Zip, true)] + [InlineData("ustar with long names.tar", ArchiveType.Tar, true)] + [InlineData("very long filename.tar", ArchiveType.Tar, true)] + [InlineData("zipcrypto.zip", ArchiveType.Zip, true)] + [InlineData("SharpCompress.AES.zip", ArchiveType.Zip, true)] + [InlineData("SharpCompress.Encrypted.zip", ArchiveType.Zip, true)] + [InlineData("SharpCompress.Encrypted2.zip", ArchiveType.Zip, true)] + public async ValueTask GetArchiveInformationAsync_DetectsSingleFileTestArchives( + string archiveName, + ArchiveType expectedType, + bool expectedSeekable + ) + { + var info = await ArchiveFactory.GetArchiveInformationAsync(GetTestArchivePath(archiveName)); + + Assert.NotNull(info); + Assert.Equal(expectedType, info.Type); + Assert.Equal(expectedSeekable, info.SupportsRandomAccess); + } + + [Fact] + public void GetArchiveInformation_ReturnsNull_ForNonArchive() + { + using var stream = new MemoryStream(Encoding.ASCII.GetBytes("not an archive")); + + var info = ArchiveFactory.GetArchiveInformation(stream); + + Assert.Null(info); + } + + [Fact] + public async ValueTask GetArchiveInformationAsync_ReturnsNull_ForNonArchive() + { + using var stream = new MemoryStream(Encoding.ASCII.GetBytes("not an archive")); + + var info = await ArchiveFactory.GetArchiveInformationAsync(stream); + + Assert.Null(info); + } + + [Theory] + [InlineData("Zip.deflate.zip", ArchiveType.Zip)] + [InlineData("Tar.noEmptyDirs.tar", ArchiveType.Tar)] + public void GetArchiveInformation_Stream_PreservesPosition( + string archiveName, + ArchiveType expectedType + ) + { + using var stream = CreatePrefixedArchiveStream(archiveName, 13); + var startPosition = stream.Position; + + var info = ArchiveFactory.GetArchiveInformation(stream); + + Assert.NotNull(info); + Assert.Equal(expectedType, info.Type); + Assert.Equal(startPosition, stream.Position); + } + + [Theory] + [InlineData("Zip.deflate.zip", ArchiveType.Zip)] + [InlineData("Tar.noEmptyDirs.tar", ArchiveType.Tar)] + public async ValueTask GetArchiveInformationAsync_Stream_PreservesPosition( + string archiveName, + ArchiveType expectedType + ) + { + using var stream = CreatePrefixedArchiveStream(archiveName, 13); + var startPosition = stream.Position; + + var info = await ArchiveFactory.GetArchiveInformationAsync(stream); + + Assert.NotNull(info); + Assert.Equal(expectedType, info.Type); + Assert.Equal(startPosition, stream.Position); + } + + private MemoryStream CreatePrefixedArchiveStream(string archiveName, int prefixLength) + { + var archiveBytes = File.ReadAllBytes(GetTestArchivePath(archiveName)); + var buffer = new byte[prefixLength + archiveBytes.Length]; + + archiveBytes.CopyTo(buffer, prefixLength); + + var stream = new MemoryStream(buffer); + stream.Position = prefixLength; + return stream; + } + + private static string GetTestArchivePath(string archiveName) + { + var archivesPath = Path.Combine(TEST_ARCHIVES_PATH, archiveName); + if (File.Exists(archivesPath)) + { + return archivesPath; + } + + return Path.GetFullPath(Path.Combine(TEST_ARCHIVES_PATH, "..", archiveName)); + } +} diff --git a/tests/SharpCompress.Test/ArchiveTests.cs b/tests/SharpCompress.Test/ArchiveTests.cs index 08377a0e..492cc85e 100644 --- a/tests/SharpCompress.Test/ArchiveTests.cs +++ b/tests/SharpCompress.Test/ArchiveTests.cs @@ -2,10 +2,17 @@ using System; using System.Collections.Generic; using System.IO; using System.Linq; +using System.Threading.Tasks; +using AwesomeAssertions; using SharpCompress.Archives; using SharpCompress.Common; +using SharpCompress.Compressors.Xz; +using SharpCompress.Crypto; using SharpCompress.IO; using SharpCompress.Readers; +using SharpCompress.Test.Mocks; +using SharpCompress.Writers; +using SharpCompress.Writers.Zip; using Xunit; namespace SharpCompress.Test; @@ -36,11 +43,11 @@ public class ArchiveTests : ReaderTests { foreach (var path in testArchives) { - using (var stream = NonDisposingStream.Create(File.OpenRead(path), true)) + using (var stream = SharpCompressStream.CreateNonDisposing(File.OpenRead(path))) { try { - using var archive = ArchiveFactory.Open(stream); + using var archive = ArchiveFactory.OpenArchive(stream); Assert.True(archive.IsSolid); using (var reader = archive.ExtractAllEntries()) { @@ -55,10 +62,7 @@ public class ArchiveTests : ReaderTests } foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) { - entry.WriteToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions { ExtractFullPath = true, Overwrite = true } - ); + entry.WriteToDirectory(SCRATCH_FILES_PATH); } stream.ThrowOnDispose = false; } @@ -76,33 +80,75 @@ public class ArchiveTests : ReaderTests protected void ArchiveStreamRead(string testArchive, ReaderOptions? readerOptions = null) { testArchive = Path.Combine(TEST_ARCHIVES_PATH, testArchive); - ArchiveStreamRead(readerOptions, testArchive); + ArchiveStreamRead( + ArchiveFactory.FindFactory(testArchive), + Path.GetExtension(testArchive), + readerOptions, + testArchive + ); } protected void ArchiveStreamRead( + IArchiveFactory archiveFactory, + string testArchive, + ReaderOptions? readerOptions = null + ) + { + testArchive = Path.Combine(TEST_ARCHIVES_PATH, testArchive); + ArchiveStreamRead( + archiveFactory, + Path.GetExtension(testArchive), + readerOptions, + testArchive + ); + } + + protected void ArchiveStreamRead( + string extension, + ReaderOptions? readerOptions = null, + params string[] testArchives + ) + { + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, testArchives[0]); + ArchiveStreamRead( + ArchiveFactory.FindFactory(testArchive), + extension, + readerOptions, + testArchives + ); + } + + protected void ArchiveStreamRead( + IArchiveFactory archiveFactory, + string extension, ReaderOptions? readerOptions = null, params string[] testArchives ) => ArchiveStreamRead( + archiveFactory, readerOptions, - testArchives.Select(x => Path.Combine(TEST_ARCHIVES_PATH, x)) + testArchives.Select(x => Path.Combine(TEST_ARCHIVES_PATH, x)), + extension ); - protected void ArchiveStreamRead(ReaderOptions? readerOptions, IEnumerable testArchives) + protected void ArchiveStreamRead( + IArchiveFactory archiveFactory, + ReaderOptions? readerOptions, + IEnumerable testArchives, + string extension + ) { + ExtensionTest(extension, archiveFactory); foreach (var path in testArchives) { - using (var stream = NonDisposingStream.Create(File.OpenRead(path), true)) - using (var archive = ArchiveFactory.Open(stream, readerOptions)) + using (var stream = SharpCompressStream.CreateNonDisposing(File.OpenRead(path))) + using (var archive = archiveFactory.OpenArchive(stream, readerOptions)) { try { foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) { - entry.WriteToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + entry.WriteToDirectory(SCRATCH_FILES_PATH); } } catch (IndexOutOfRangeException) @@ -132,25 +178,15 @@ public class ArchiveTests : ReaderTests ) { using ( - var archive = ArchiveFactory.Open( - testArchives.Select(a => new FileInfo(a)), + var archive = ArchiveFactory.OpenArchive( + testArchives.Select(a => new FileInfo(a)).ToArray(), readerOptions ) ) { - try + foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) { - foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) - { - entry.WriteToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); - } - } - catch (IndexOutOfRangeException) - { - throw; + entry.WriteToDirectory(SCRATCH_FILES_PATH); } } VerifyFiles(); @@ -171,25 +207,15 @@ public class ArchiveTests : ReaderTests ) { using ( - var archive = ArchiveFactory.Open( - testArchives.Select(f => new FileInfo(f)), + var archive = ArchiveFactory.OpenArchive( + testArchives.Select(f => new FileInfo(f)).ToArray(), readerOptions ) ) { - try + foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) { - foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) - { - entry.WriteToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); - } - } - catch (IndexOutOfRangeException) - { - throw; + entry.WriteToDirectory(SCRATCH_FILES_PATH); } } VerifyFiles(); @@ -206,78 +232,106 @@ public class ArchiveTests : ReaderTests testArchives.Select(x => Path.Combine(TEST_ARCHIVES_PATH, x)) ); - protected void ArchiveOpenEntryVolumeIndexTest( + private void ArchiveOpenEntryVolumeIndexTest( int[][] results, ReaderOptions? readerOptions, IEnumerable testArchives ) { var src = testArchives.ToArray(); - using var archive = ArchiveFactory.Open( - testArchives.Select(f => new FileInfo(f)), + using var archive = ArchiveFactory.OpenArchive( + src.Select(f => new FileInfo(f)).ToArray(), readerOptions ); - try + var idx = 0; + foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) { - var idx = 0; - foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) - { - Assert.Equal(entry.VolumeIndexFirst, results[idx][0]); - Assert.Equal(entry.VolumeIndexLast, results[idx][1]); - Assert.Equal( - src[entry.VolumeIndexFirst], - archive.Volumes.First(a => a.Index == entry.VolumeIndexFirst).FileName - ); - Assert.Equal( - src[entry.VolumeIndexLast], - archive.Volumes.First(a => a.Index == entry.VolumeIndexLast).FileName - ); + Assert.Equal(entry.VolumeIndexFirst, results[idx][0]); + Assert.Equal(entry.VolumeIndexLast, results[idx][1]); + Assert.Equal( + src[entry.VolumeIndexFirst], + archive.Volumes.First(a => a.Index == entry.VolumeIndexFirst).FileName + ); + Assert.Equal( + src[entry.VolumeIndexLast], + archive.Volumes.First(a => a.Index == entry.VolumeIndexLast).FileName + ); - idx++; - } - } - catch (IndexOutOfRangeException) - { - throw; + idx++; } } - protected void ArchiveFileRead(string testArchive, ReaderOptions? readerOptions = null) + protected void ArchiveExtractToDirectory( + string testArchive, + ReaderOptions? readerOptions = null + ) { testArchive = Path.Combine(TEST_ARCHIVES_PATH, testArchive); - using (var archive = ArchiveFactory.Open(testArchive, readerOptions)) + using (var archive = ArchiveFactory.OpenArchive(new FileInfo(testArchive), readerOptions)) + { + archive.WriteToDirectory(SCRATCH_FILES_PATH); + } + VerifyFiles(); + } + + protected void ArchiveFileRead( + string testArchive, + ReaderOptions? readerOptions = null, + IArchiveFactory? archiveFactory = null + ) + { + testArchive = Path.Combine(TEST_ARCHIVES_PATH, testArchive); + archiveFactory ??= ArchiveFactory.FindFactory(testArchive); + ExtensionTest(testArchive, archiveFactory); + using (var archive = archiveFactory.OpenArchive(new FileInfo(testArchive), readerOptions)) { foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) { - entry.WriteToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + entry.WriteToDirectory(SCRATCH_FILES_PATH); } } VerifyFiles(); } + private void ExtensionTest(string fullPath, IArchiveFactory archiveFactory) + { + var extension = Path.GetExtension(fullPath).Substring(1); + if (!int.TryParse(extension, out _) && "exe" != extension) //exclude parts + { + extension.Should().BeOneOf(archiveFactory.GetSupportedExtensions()); + } + } + + protected void ArchiveFileSkip( + string testArchive, + string fileOrder, + ReaderOptions? readerOptions = null + ) + { + if (!Environment.OSVersion.IsWindows()) + { + fileOrder = fileOrder.Replace('\\', '/'); + } + var expected = new Stack(fileOrder.Split(' ')); + testArchive = Path.Combine(TEST_ARCHIVES_PATH, testArchive); + using var archive = ArchiveFactory.OpenArchive(testArchive, readerOptions); + foreach (var entry in archive.Entries) + { + Assert.Equal(expected.Pop(), entry.Key); + } + } + /// /// Demonstrate the ExtractionOptions.PreserveFileTime and ExtractionOptions.PreserveAttributes extract options /// protected void ArchiveFileReadEx(string testArchive) { testArchive = Path.Combine(TEST_ARCHIVES_PATH, testArchive); - using (var archive = ArchiveFactory.Open(testArchive)) + using (var archive = ArchiveFactory.OpenArchive(testArchive)) { foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) { - entry.WriteToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() - { - ExtractFullPath = true, - Overwrite = true, - PreserveAttributes = true, - PreserveFileTime = true - } - ); + entry.WriteToDirectory(SCRATCH_FILES_PATH); } } VerifyFilesEx(); @@ -286,25 +340,304 @@ public class ArchiveTests : ReaderTests protected void ArchiveDeltaDistanceRead(string testArchive) { testArchive = Path.Combine(TEST_ARCHIVES_PATH, testArchive); - using (var archive = Archives.ArchiveFactory.Open(testArchive, null)) - using (var reader = archive.ExtractAllEntries()) - while (reader.MoveToNextEntry()) + using var archive = ArchiveFactory.OpenArchive(testArchive); + foreach (var entry in archive.Entries) + { + if (!entry.IsDirectory) { - if (!reader.Entry.IsDirectory) + var memory = new MemoryStream(); + entry.WriteTo(memory); + + memory.Position = 0; + + for (var y = 0; y < 9; y++) { - var memory = new MemoryStream(); - reader.WriteEntryTo(memory); - - memory.Position = 0; - - for (int y = 0; y < 9; y++) - for (int x = 0; x < 256; x++) - { - Assert.Equal(x, memory.ReadByte()); - } - - Assert.Equal((int)-1, memory.ReadByte()); + for (var x = 0; x < 256; x++) + { + Assert.Equal(x, memory.ReadByte()); + } } + + Assert.Equal(-1, memory.ReadByte()); } + } + } + + /// + /// Calculates CRC32 for the given data using SharpCompress implementation + /// + protected static uint CalculateCrc32(byte[] data) => Crc32.Compute(data); + + /// + /// Creates a writer with the specified compression type and level + /// + protected static IWriter CreateWriterWithLevel( + Stream stream, + CompressionType compressionType, + int? compressionLevel = null + ) + { + var writerOptions = compressionLevel.HasValue + ? new WriterOptions(compressionType, compressionLevel.Value) + : new WriterOptions(compressionType); + return WriterFactory.OpenWriter(stream, ArchiveType.Zip, writerOptions); + } + + protected static async ValueTask CreateWriterWithLevelAsync( + Stream stream, + CompressionType compressionType, + int? compressionLevel = null + ) + { + var writerOptions = compressionLevel.HasValue + ? new WriterOptions(compressionType, compressionLevel.Value) { LeaveStreamOpen = true } + : new WriterOptions(compressionType) { LeaveStreamOpen = true }; + return await WriterFactory.OpenAsyncWriter( + new AsyncOnlyStream(stream), + ArchiveType.Zip, + writerOptions + ); + } + + /// + /// Verifies archive content against expected files with CRC32 validation + /// + protected void VerifyArchiveContent( + MemoryStream zipStream, + Dictionary expectedFiles + ) + { + zipStream.Position = 0; + using var archive = ArchiveFactory.OpenArchive(zipStream); + Assert.Equal(expectedFiles.Count, archive.Entries.Count()); + + foreach (var entry in archive.Entries.Where(e => !e.IsDirectory)) + { + using var entryStream = entry.OpenEntryStream(); + using var extractedStream = new MemoryStream(); + entryStream.CopyTo(extractedStream); + var extractedData = extractedStream.ToArray(); + + Assert.True( + expectedFiles.ContainsKey(entry.Key.NotNull()), + $"Unexpected entry: {entry.Key}" + ); + + var (expectedData, expectedCrc) = expectedFiles[entry.Key.NotNull()]; + var actualCrc = CalculateCrc32(extractedData); + + Assert.Equal(expectedCrc, actualCrc); + Assert.Equal(expectedData.Length, extractedData.Length); + + // For large files, spot check rather than full comparison for performance + if (expectedData.Length > 1024 * 1024) + { + VerifyDataSpotCheck(expectedData, extractedData); + } + else + { + Assert.Equal(expectedData, extractedData); + } + } + } + + /// + /// Performs efficient spot checks on large data arrays + /// + protected static void VerifyDataSpotCheck(byte[] expected, byte[] actual) + { + // Check first, middle, and last 1KB + Assert.Equal(expected.Take(1024), actual.Take(1024)); + var mid = expected.Length / 2; + Assert.Equal(expected.Skip(mid).Take(1024), actual.Skip(mid).Take(1024)); + Assert.Equal( + expected.Skip(Math.Max(0, expected.Length - 1024)), + actual.Skip(Math.Max(0, actual.Length - 1024)) + ); + } + + /// + /// Verifies compression ratio meets expectations + /// + protected void VerifyCompressionRatio( + long originalSize, + long compressedSize, + double maxRatio, + string context + ) + { + var compressionRatio = (double)compressedSize / originalSize; + Assert.True( + compressionRatio < maxRatio, + $"Expected better compression for {context}. Original: {originalSize}, Compressed: {compressedSize}, Ratio: {compressionRatio:P}" + ); + } + + /// + /// Creates a memory-based archive with specified files and compression + /// + protected MemoryStream CreateMemoryArchive( + Dictionary files, + CompressionType compressionType, + int? compressionLevel = null + ) + { + var zipStream = new MemoryStream(); + using (var writer = CreateWriterWithLevel(zipStream, compressionType, compressionLevel)) + { + foreach (var kvp in files) + { + writer.Write(kvp.Key, new MemoryStream(kvp.Value)); + } + } + return zipStream; + } + + /// + /// Verifies streaming CRC calculation for large data + /// + protected void VerifyStreamingCrc(Stream entryStream, uint expectedCrc, long expectedLength) + { + using var crcStream = new Crc32Stream(Stream.Null); + const int bufferSize = 64 * 1024; + var buffer = new byte[bufferSize]; + int totalBytesRead = 0; + int bytesRead; + + while ((bytesRead = entryStream.Read(buffer, 0, bufferSize)) > 0) + { + crcStream.Write(buffer, 0, bytesRead); + totalBytesRead += bytesRead; + } + + var actualCrc = crcStream.Crc; + Assert.Equal(expectedCrc, actualCrc); + Assert.Equal(expectedLength, totalBytesRead); + } + + /// + /// Creates and verifies a basic archive with compression testing + /// + protected void CreateAndVerifyBasicArchive( + Dictionary testFiles, + CompressionType compressionType, + int? compressionLevel = null, + double maxCompressionRatio = 0.8 + ) + { + // Calculate expected CRCs + var expectedFiles = testFiles.ToDictionary( + kvp => kvp.Key, + kvp => (data: kvp.Value, crc: CalculateCrc32(kvp.Value)) + ); + + // Create archive + using var zipStream = CreateMemoryArchive(testFiles, compressionType, compressionLevel); + + // Verify compression occurred if expected + if (compressionType != CompressionType.None) + { + var originalSize = testFiles.Values.Sum(data => (long)data.Length); + VerifyCompressionRatio( + originalSize, + zipStream.Length, + maxCompressionRatio, + compressionType.ToString() + ); + } + + // Verify content + VerifyArchiveContent(zipStream, expectedFiles); + } + + /// + /// Verifies archive entries have correct compression type + /// + protected void VerifyCompressionType( + MemoryStream zipStream, + CompressionType expectedCompressionType + ) + { + zipStream.Position = 0; + using var archive = ArchiveFactory.OpenArchive(zipStream); + + foreach (var entry in archive.Entries.Where(e => !e.IsDirectory)) + { + Assert.Equal(expectedCompressionType, entry.CompressionType); + } + } + + /// + /// Extracts and verifies a single entry from archive + /// + protected (byte[] data, uint crc) ExtractAndVerifyEntry( + MemoryStream zipStream, + string entryName + ) + { + zipStream.Position = 0; + using var archive = ArchiveFactory.OpenArchive(zipStream); + + var entry = archive.Entries.FirstOrDefault(e => e.Key == entryName && !e.IsDirectory); + Assert.NotNull(entry); + + using var entryStream = entry.OpenEntryStream(); + using var extractedStream = new MemoryStream(); + entryStream.CopyTo(extractedStream); + + var extractedData = extractedStream.ToArray(); + var crc = CalculateCrc32(extractedData); + + return (extractedData, crc); + } + + protected async Task ArchiveStreamReadAsync( + string testArchive, + ReaderOptions? readerOptions = null + ) + { + testArchive = Path.Combine(TEST_ARCHIVES_PATH, testArchive); + await ArchiveStreamReadAsync( + ArchiveFactory.FindFactory(testArchive), + readerOptions, + new[] { testArchive } + ); + } + + protected async Task ArchiveStreamReadAsync( + IArchiveFactory archiveFactory, + ReaderOptions? readerOptions, + IEnumerable testArchives + ) + { + foreach (var path in testArchives) + { + using (var stream = SharpCompressStream.CreateNonDisposing(File.OpenRead(path))) + await using ( + var archive = await archiveFactory.OpenAsyncArchive( + new AsyncOnlyStream(stream), + readerOptions + ) + ) + { + try + { + await foreach ( + var entry in archive.EntriesAsync.Where(entry => !entry.IsDirectory) + ) + { + await entry.WriteToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + catch (IndexOutOfRangeException) + { + //SevenZipArchive_BZip2_Split test needs this + stream.ThrowOnDispose = false; + throw; + } + stream.ThrowOnDispose = false; + } + VerifyFiles(); + } } } diff --git a/tests/SharpCompress.Test/Arj/ArjReaderAsyncTests.cs b/tests/SharpCompress.Test/Arj/ArjReaderAsyncTests.cs new file mode 100644 index 00000000..c2418e4b --- /dev/null +++ b/tests/SharpCompress.Test/Arj/ArjReaderAsyncTests.cs @@ -0,0 +1,167 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Readers; +using SharpCompress.Readers.Arj; +using SharpCompress.Test.Mocks; +using Xunit; +using Xunit.Sdk; + +namespace SharpCompress.Test.Arj; + +public class ArjReaderAsyncTests : ReaderTests +{ + public ArjReaderAsyncTests() + { + UseExtensionInsteadOfNameToVerify = true; + UseCaseInsensitiveToVerify = true; + } + + [Fact] + public async ValueTask Arj_Uncompressed_Read_Async() => + await ReadAsync("Arj.store.arj", CompressionType.None); + + [Fact] + public async ValueTask Arj_Method1_Read_Async() => await ReadAsync("Arj.method1.arj"); + + [Fact] + public async ValueTask Arj_Method2_Read_Async() => await ReadAsync("Arj.method2.arj"); + + [Fact] + public async ValueTask Arj_Method3_Read_Async() => await ReadAsync("Arj.method3.arj"); + + [Fact] + public async ValueTask Arj_Method4_Read_Async() => await ReadAsync("Arj.method4.arj"); + + [Fact] + public async ValueTask Arj_Encrypted_Read_Async() + { + var exception = await Assert.ThrowsAsync(() => + ReadAsync("Arj.encrypted.arj") + ); + } + + [Fact] + public async ValueTask Arj_Multi_Reader_Async() + { + var exception = await Assert.ThrowsAsync(() => + DoMultiReaderAsync( + [ + "Arj.store.split.arj", + "Arj.store.split.a01", + "Arj.store.split.a02", + "Arj.store.split.a03", + "Arj.store.split.a04", + "Arj.store.split.a05", + ], + streams => ReaderFactory.OpenAsyncReader(new AsyncOnlyStream(streams.First())) + ) + ); + } + + [Theory] + [InlineData("Arj.method1.largefile.arj", CompressionType.ArjLZ77)] + [InlineData("Arj.method2.largefile.arj", CompressionType.ArjLZ77)] + [InlineData("Arj.method3.largefile.arj", CompressionType.ArjLZ77)] + public async ValueTask Arj_LargeFile_ShouldThrow_Async( + string fileName, + CompressionType compressionType + ) + { + var exception = await Assert.ThrowsAsync(() => + ReadForBufferBoundaryCheckAsync(fileName, compressionType) + ); + } + + [Theory] + [InlineData("Arj.store.largefile.arj", CompressionType.None)] + [InlineData("Arj.method4.largefile.arj", CompressionType.ArjLZ77)] + public async ValueTask Arj_LargeFileTest_Read_Async( + string fileName, + CompressionType compressionType + ) + { + await ReadForBufferBoundaryCheckAsync(fileName, compressionType); + } + + private async Task ReadAsync(string testArchive, CompressionType? expectedCompression = null) + { + testArchive = Path.Combine(TEST_ARCHIVES_PATH, testArchive); + using Stream stream = File.OpenRead(testArchive); + await using var reader = await ReaderFactory.OpenAsyncReader( + new AsyncOnlyStream(stream), + ReaderOptions.ForExternalStream + ); + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + if (expectedCompression.HasValue) + { + Assert.Equal(expectedCompression.Value, reader.Entry.CompressionType); + } + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + VerifyFiles(); + } + + private async Task ReadForBufferBoundaryCheckAsync( + string testArchive, + CompressionType expectedCompression + ) + { + testArchive = Path.Combine(TEST_ARCHIVES_PATH, testArchive); + using Stream stream = File.OpenRead(testArchive); + await using var reader = await ReaderFactory.OpenAsyncReader( + new AsyncOnlyStream(stream), + ReaderOptions.ForExternalStream with + { + LookForHeader = true, + } + ); + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + Assert.Equal(expectedCompression, reader.Entry.CompressionType); + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + CompareFilesByPath( + Path.Combine(SCRATCH_FILES_PATH, "alice29.txt"), + Path.Combine(MISC_TEST_FILES_PATH, "alice29.txt") + ); + } + + private async Task DoMultiReaderAsync( + string[] archiveNames, + Func, ValueTask> openReader + ) + { + var testArchives = archiveNames.Select(s => Path.Combine(TEST_ARCHIVES_PATH, s)).ToList(); + var streams = testArchives.Select(File.OpenRead).ToList(); + try + { + await using var reader = await openReader(streams); + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + } + finally + { + foreach (var stream in streams) + { + stream.Dispose(); + } + } + } +} diff --git a/tests/SharpCompress.Test/Arj/ArjReaderTests.cs b/tests/SharpCompress.Test/Arj/ArjReaderTests.cs new file mode 100644 index 00000000..f4b5b4d5 --- /dev/null +++ b/tests/SharpCompress.Test/Arj/ArjReaderTests.cs @@ -0,0 +1,80 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Readers; +using SharpCompress.Readers.Arj; +using Xunit; +using Xunit.Sdk; + +namespace SharpCompress.Test.Arj; + +public class ArjReaderTests : ReaderTests +{ + public ArjReaderTests() + { + UseExtensionInsteadOfNameToVerify = true; + UseCaseInsensitiveToVerify = true; + } + + [Fact] + public void Arj_Uncompressed_Read() => Read("Arj.store.arj", CompressionType.None); + + [Fact] + public void Arj_Method1_Read() => Read("Arj.method1.arj"); + + [Fact] + public void Arj_Method2_Read() => Read("Arj.method2.arj"); + + [Fact] + public void Arj_Method3_Read() => Read("Arj.method3.arj"); + + [Fact] + public void Arj_Method4_Read() => Read("Arj.method4.arj"); + + [Fact] + public void Arj_Encrypted_Read() + { + var exception = Assert.Throws(() => Read("Arj.encrypted.arj")); + } + + [Fact] + public void Arj_Multi_Reader() + { + var exception = Assert.Throws(() => + DoMultiReader( + [ + "Arj.store.split.arj", + "Arj.store.split.a01", + "Arj.store.split.a02", + "Arj.store.split.a03", + "Arj.store.split.a04", + "Arj.store.split.a05", + ], + streams => ArjReader.OpenReader(streams) + ) + ); + } + + [Theory] + [InlineData("Arj.method1.largefile.arj", CompressionType.ArjLZ77)] + [InlineData("Arj.method2.largefile.arj", CompressionType.ArjLZ77)] + [InlineData("Arj.method3.largefile.arj", CompressionType.ArjLZ77)] + public void Arj_LargeFile_ShouldThrow(string fileName, CompressionType compressionType) + { + var exception = Assert.Throws(() => + ReadForBufferBoundaryCheck(fileName, compressionType) + ); + } + + [Theory] + [InlineData("Arj.store.largefile.arj", CompressionType.None)] + [InlineData("Arj.method4.largefile.arj", CompressionType.ArjLZ77)] + public void Arj_LargeFileTest_Read(string fileName, CompressionType compressionType) + { + ReadForBufferBoundaryCheck(fileName, compressionType); + } +} diff --git a/tests/SharpCompress.Test/AsyncParityAndCancellationTests.cs b/tests/SharpCompress.Test/AsyncParityAndCancellationTests.cs new file mode 100644 index 00000000..49beacfa --- /dev/null +++ b/tests/SharpCompress.Test/AsyncParityAndCancellationTests.cs @@ -0,0 +1,341 @@ +#if NET8_0_OR_GREATER +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Archives; +using SharpCompress.Common; +using SharpCompress.Readers; +using SharpCompress.Test.Mocks; +using SharpCompress.Writers; +using Xunit; + +namespace SharpCompress.Test; + +public class AsyncParityAndCancellationTests : TestBase +{ + [Theory] + [InlineData("Zip.deflate.zip")] + [InlineData("Tar.tar")] + [InlineData("Tar.tar.gz")] + [InlineData("Rar.rar")] + [InlineData("7Zip.nonsolid.7z")] + public async Task ArchiveAsyncEntries_ShouldMatchSyncEntries(string archiveName) + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, archiveName); + + var syncEntries = ReadArchiveEntries(archivePath); + var asyncEntries = await ReadArchiveEntriesAsync(archivePath); + + Assert.Equal(syncEntries, asyncEntries); + } + + [Theory] + [InlineData("Zip.deflate.zip")] + [InlineData("Tar.tar")] + [InlineData("Tar.tar.gz")] + [InlineData("Rar.rar")] + public async Task ReaderAsyncEntries_ShouldMatchSyncEntries(string archiveName) + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, archiveName); + + var syncEntries = ReadReaderEntries(archivePath); + var asyncEntries = await ReadReaderEntriesAsync(archivePath); + + Assert.Equal(syncEntries, asyncEntries); + } + + [Fact] + public async Task AsyncReaderExtraction_ShouldRespectCancellationBeforeStart() + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.zip"); + await using var stream = File.OpenRead(archivePath); + await using var reader = await ReaderFactory.OpenAsyncReader(stream); + using var cts = new CancellationTokenSource(); + cts.Cancel(); + + await Assert.ThrowsAnyAsync(async () => + await reader.WriteAllToDirectoryAsync(SCRATCH_FILES_PATH, cancellationToken: cts.Token) + ); + } + + [Fact] + public async Task AsyncArchiveExtraction_ShouldRespectCancellationBeforeStart() + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.zip"); + await using var archive = await ArchiveFactory.OpenAsyncArchive(archivePath); + using var cts = new CancellationTokenSource(); + cts.Cancel(); + + await Assert.ThrowsAnyAsync(async () => + await archive.WriteToDirectoryAsync(SCRATCH_FILES_PATH, cancellationToken: cts.Token) + ); + } + + [Fact] + public async Task AsyncReaderExtraction_ShouldRespectCancellationDuringRead() + { + var archiveBytes = CreateLargeTarArchive(); + using var cts = new CancellationTokenSource(); + await using var stream = new CancelAfterBytesReadStream( + new MemoryStream(archiveBytes), + cts, + cancelAfterBytes: 2048 + ); + await Assert.ThrowsAnyAsync(async () => + { + await using var reader = await ReaderFactory.OpenAsyncReader( + stream, + cancellationToken: cts.Token + ); + await reader.WriteAllToDirectoryAsync(SCRATCH_FILES_PATH, cancellationToken: cts.Token); + }); + } + + [Fact] + public async Task OpenAsyncReader_CallerProvidedStream_ShouldRemainOpenByDefault() + { + var archiveBytes = await File.ReadAllBytesAsync( + Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.zip") + ); + var stream = new TestStream(new MemoryStream(archiveBytes)); + + try + { + await using (var reader = await ReaderFactory.OpenAsyncReader(stream)) + { + Assert.True(await reader.MoveToNextEntryAsync()); + } + + Assert.False(stream.IsDisposed); + } + finally + { + stream.Dispose(); + } + } + + [Fact] + public async Task OpenAsyncArchive_CallerProvidedStream_ShouldRemainOpenByDefault() + { + var archiveBytes = await File.ReadAllBytesAsync( + Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.zip") + ); + var stream = new TestStream(new MemoryStream(archiveBytes)); + + try + { + await using (var archive = await ArchiveFactory.OpenAsyncArchive(stream)) + { + await foreach (var _ in archive.EntriesAsync) + { + break; + } + } + + Assert.False(stream.IsDisposed); + } + finally + { + await stream.DisposeAsync(); + } + } + + private static List ReadArchiveEntries(string archivePath) + { + using var archive = ArchiveFactory.OpenArchive(archivePath); + return archive + .Entries.Where(entry => !entry.IsDirectory) + .Select(entry => + { + using var stream = entry.OpenEntryStream(); + using var memory = new MemoryStream(); + stream.CopyTo(memory); + return new EntrySnapshot( + entry.Key ?? string.Empty, + entry.Size, + entry.CompressionType, + Convert.ToBase64String(memory.ToArray()) + ); + }) + .OrderBy(entry => entry.Key, StringComparer.Ordinal) + .ToList(); + } + + private static async Task> ReadArchiveEntriesAsync(string archivePath) + { + await using var archive = await ArchiveFactory.OpenAsyncArchive(archivePath); + var entries = new List(); + await foreach (var entry in archive.EntriesAsync) + { + if (entry.IsDirectory) + { + continue; + } + + await using var stream = await entry.OpenEntryStreamAsync(); + using var memory = new MemoryStream(); + await stream.CopyToAsync(memory); + entries.Add( + new EntrySnapshot( + entry.Key ?? string.Empty, + entry.Size, + entry.CompressionType, + Convert.ToBase64String(memory.ToArray()) + ) + ); + } + + return entries.OrderBy(entry => entry.Key, StringComparer.Ordinal).ToList(); + } + + private static List ReadReaderEntries(string archivePath) + { + using var stream = File.OpenRead(archivePath); + using var reader = ReaderFactory.OpenReader(stream); + var entries = new List(); + while (reader.MoveToNextEntry()) + { + if (reader.Entry.IsDirectory) + { + continue; + } + + using var memory = new MemoryStream(); + reader.WriteEntryTo(memory); + entries.Add( + new EntrySnapshot( + reader.Entry.Key ?? string.Empty, + reader.Entry.Size, + reader.Entry.CompressionType, + Convert.ToBase64String(memory.ToArray()) + ) + ); + } + + return entries.OrderBy(entry => entry.Key, StringComparer.Ordinal).ToList(); + } + + private static async Task> ReadReaderEntriesAsync(string archivePath) + { + await using var stream = File.OpenRead(archivePath); + await using var reader = await ReaderFactory.OpenAsyncReader(stream); + var entries = new List(); + while (await reader.MoveToNextEntryAsync()) + { + if (reader.Entry.IsDirectory) + { + continue; + } + + using var memory = new MemoryStream(); + await reader.WriteEntryToAsync(memory); + entries.Add( + new EntrySnapshot( + reader.Entry.Key ?? string.Empty, + reader.Entry.Size, + reader.Entry.CompressionType, + Convert.ToBase64String(memory.ToArray()) + ) + ); + } + + return entries.OrderBy(entry => entry.Key, StringComparer.Ordinal).ToList(); + } + + private static byte[] CreateLargeTarArchive() + { + using var stream = new MemoryStream(); + using ( + var writer = WriterFactory.OpenWriter( + stream, + ArchiveType.Tar, + new WriterOptions(CompressionType.None) + ) + ) + { + writer.Write("large.bin", new MemoryStream(new byte[64 * 1024])); + } + return stream.ToArray(); + } + + private sealed record EntrySnapshot( + string Key, + long Size, + CompressionType CompressionType, + string Content + ); + + private sealed class CancelAfterBytesReadStream( + Stream stream, + CancellationTokenSource cancellationTokenSource, + long cancelAfterBytes + ) : Stream + { + private long _bytesRead; + + public override bool CanRead => stream.CanRead; + public override bool CanSeek => stream.CanSeek; + public override bool CanWrite => false; + public override long Length => stream.Length; + public override long Position + { + get => stream.Position; + set => stream.Position = value; + } + + public override void Flush() => stream.Flush(); + + public override int Read(byte[] buffer, int offset, int count) => + throw new NotSupportedException("Use async reads for this test stream."); + + public override async ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + var read = await stream.ReadAsync(buffer, cancellationToken).ConfigureAwait(false); + _bytesRead = read; + if (_bytesRead > cancelAfterBytes) + { + cancellationTokenSource.Cancel(); + cancellationToken.ThrowIfCancellationRequested(); + } + + return read; + } + + public override Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) => ReadAsync(buffer.AsMemory(offset, count), cancellationToken).AsTask(); + + protected override void Dispose(bool disposing) + { + if (disposing) + { + stream.Dispose(); + } + base.Dispose(disposing); + } + + public override async ValueTask DisposeAsync() + { + await stream.DisposeAsync().ConfigureAwait(false); + await base.DisposeAsync().ConfigureAwait(false); + } + + public override long Seek(long offset, SeekOrigin origin) => stream.Seek(offset, origin); + + public override void SetLength(long value) => throw new NotSupportedException(); + + public override void Write(byte[] buffer, int offset, int count) => + throw new NotSupportedException(); + } +} +#endif diff --git a/tests/SharpCompress.Test/BZip2/BZip2ReaderTests.cs b/tests/SharpCompress.Test/BZip2/BZip2ReaderTests.cs new file mode 100644 index 00000000..70cea8c6 --- /dev/null +++ b/tests/SharpCompress.Test/BZip2/BZip2ReaderTests.cs @@ -0,0 +1,21 @@ +using System; +using System.IO; +using SharpCompress.Common; +using SharpCompress.IO; +using SharpCompress.Readers; +using SharpCompress.Readers.GZip; +using Xunit; + +namespace SharpCompress.Test.BZip2; + +public class BZip2ReaderTests : ReaderTests +{ + [Fact] + public void BZip2_Reader_Factory() + { + Stream stream = new MemoryStream( + new byte[] { 0x42, 0x5a, 0x68, 0x34, 0x31, 0x41, 0x59, 0x26, 0x53, 0x59, 0x35 } + ); + Assert.Throws(() => ReaderFactory.OpenReader(stream)); + } +} diff --git a/tests/SharpCompress.Test/BZip2/BZip2StreamAsyncTests.cs b/tests/SharpCompress.Test/BZip2/BZip2StreamAsyncTests.cs new file mode 100644 index 00000000..d5b2ef78 --- /dev/null +++ b/tests/SharpCompress.Test/BZip2/BZip2StreamAsyncTests.cs @@ -0,0 +1,240 @@ +using System; +using System.Buffers; +using System.IO; +using System.Text; +using System.Threading.Tasks; +using SharpCompress.Compressors.BZip2; +using SharpCompress.Test.Mocks; +using Xunit; + +namespace SharpCompress.Test.BZip2; + +public class BZip2StreamAsyncTests +{ + private byte[] CreateTestData(int size) + { + var data = new byte[size]; + // Create compressible data with repetitive pattern + for (int i = 0; i < size; i++) + { + data[i] = (byte)('A' + (i % 26)); + } + return data; + } + + [Fact] + public async ValueTask BZip2CompressDecompressAsyncTest() + { + var testData = CreateTestData(10000); + byte[] compressed; + + // Compress + using (var memoryStream = new MemoryStream()) + { + await using ( + var bzip2Stream = await BZip2Stream.CreateAsync( + new AsyncOnlyStream(memoryStream), + SharpCompress.Compressors.CompressionMode.Compress, + false + ) + ) + { + await bzip2Stream.WriteAsync(testData, 0, testData.Length); + } + compressed = memoryStream.ToArray(); + } + + // Verify compression occurred + Assert.True(compressed.Length > 0); + Assert.True(compressed.Length < testData.Length); + + // Decompress + byte[] decompressed; + using (var memoryStream = new MemoryStream(compressed)) + { + using ( + var bzip2Stream = await BZip2Stream.CreateAsync( + new AsyncOnlyStream(memoryStream), + SharpCompress.Compressors.CompressionMode.Decompress, + false + ) + ) + { + decompressed = new byte[testData.Length]; + var totalRead = 0; + int bytesRead; + while ( + ( + bytesRead = await bzip2Stream.ReadAsync( + decompressed, + totalRead, + testData.Length - totalRead + ) + ) > 0 + ) + { + totalRead += bytesRead; + } + } + } + + // Verify decompression + Assert.Equal(testData, decompressed); + } + + [Fact] + public async ValueTask BZip2ReadAsyncWithCancellationTest() + { + var testData = Encoding.ASCII.GetBytes(new string('A', 5000)); // Repetitive data compresses well + byte[] compressed; + + // Compress + using (var memoryStream = new MemoryStream()) + { + await using ( + var bzip2Stream = await BZip2Stream.CreateAsync( + new AsyncOnlyStream(memoryStream), + SharpCompress.Compressors.CompressionMode.Compress, + false + ) + ) + { + await bzip2Stream.WriteAsync(testData, 0, testData.Length); + } + compressed = memoryStream.ToArray(); + } + + // Decompress with cancellation support + using (var memoryStream = new MemoryStream(compressed)) + { + using ( + var bzip2Stream = await BZip2Stream.CreateAsync( + new AsyncOnlyStream(memoryStream), + SharpCompress.Compressors.CompressionMode.Decompress, + false + ) + ) + { + var buffer = new byte[1024]; + using var cts = new System.Threading.CancellationTokenSource(); + + // Read should complete without cancellation + var bytesRead = await bzip2Stream.ReadAsync(buffer, 0, buffer.Length, cts.Token); + Assert.True(bytesRead > 0); + } + } + } + + [Fact] + public async ValueTask BZip2MultipleAsyncWritesTest() + { + using (var memoryStream = new MemoryStream()) + { + await using ( + var bzip2Stream = await BZip2Stream.CreateAsync( + new AsyncOnlyStream(memoryStream), + SharpCompress.Compressors.CompressionMode.Compress, + false + ) + ) + { + var data1 = Encoding.ASCII.GetBytes("Hello "); + var data2 = Encoding.ASCII.GetBytes("World"); + var data3 = Encoding.ASCII.GetBytes("!"); + + await bzip2Stream.WriteAsync(data1, 0, data1.Length); + await bzip2Stream.WriteAsync(data2, 0, data2.Length); + await bzip2Stream.WriteAsync(data3, 0, data3.Length); + } + + var compressed = memoryStream.ToArray(); + Assert.True(compressed.Length > 0); + + // Decompress and verify +#if LEGACY_DOTNET + using (var readStream = new MemoryStream(compressed)) + { + using ( + var bzip2Stream = await BZip2Stream.CreateAsync( + new AsyncOnlyStream(readStream), + SharpCompress.Compressors.CompressionMode.Decompress, + false + ) + ) + { +#else + await using (var readStream = new MemoryStream(compressed)) + { + await using ( + var bzip2Stream = await BZip2Stream.CreateAsync( + new AsyncOnlyStream(readStream), + SharpCompress.Compressors.CompressionMode.Decompress, + false + ) + ) + { +#endif + var result = new StringBuilder(); + var buffer = new byte[256]; + int bytesRead; + while ((bytesRead = await bzip2Stream.ReadAsync(buffer, 0, buffer.Length)) > 0) + { + result.Append(Encoding.ASCII.GetString(buffer, 0, bytesRead)); + } + + Assert.Equal("Hello World!", result.ToString()); + } + } + } + } + + [Fact] + public async ValueTask BZip2LargeDataAsyncTest() + { + var largeData = CreateTestData(100000); + + // Compress + byte[] compressed; + using (var memoryStream = new MemoryStream()) + { + await using ( + var bzip2Stream = await BZip2Stream.CreateAsync( + new AsyncOnlyStream(memoryStream), + SharpCompress.Compressors.CompressionMode.Compress, + false + ) + ) + { + await bzip2Stream.WriteAsync(largeData, 0, largeData.Length); + } + compressed = memoryStream.ToArray(); + } + + // Decompress + byte[] decompressed; + using (var memoryStream = new MemoryStream(compressed)) + { + using ( + var bzip2Stream = await BZip2Stream.CreateAsync( + new AsyncOnlyStream(memoryStream), + SharpCompress.Compressors.CompressionMode.Decompress, + false + ) + ) + { + decompressed = new byte[largeData.Length]; + var totalRead = 0; + int bytesRead; + var buffer = new byte[4096]; + while ((bytesRead = await bzip2Stream.ReadAsync(buffer, 0, buffer.Length)) > 0) + { + Array.Copy(buffer, 0, decompressed, totalRead, bytesRead); + totalRead += bytesRead; + } + } + } + + // Verify + Assert.Equal(largeData, decompressed); + } +} diff --git a/tests/SharpCompress.Test/BZip2/BZip2StreamTests.cs b/tests/SharpCompress.Test/BZip2/BZip2StreamTests.cs new file mode 100644 index 00000000..dc603878 --- /dev/null +++ b/tests/SharpCompress.Test/BZip2/BZip2StreamTests.cs @@ -0,0 +1,139 @@ +using System; +using System.IO; +using System.Text; +using SharpCompress.Common; +using SharpCompress.Compressors; +using SharpCompress.Compressors.BZip2; +using Xunit; + +namespace SharpCompress.Test.BZip2; + +public class BZip2StreamTests +{ + [Fact] + public void BZip2Stream_Throws_On_Corrupt_Checksum() + { + var compressed = Compress("BZip2 checksum validation test data."); + compressed[^5] ^= 1; + + using var stream = BZip2Stream.Create( + new MemoryStream(compressed), + SharpCompress.Compressors.CompressionMode.Decompress, + false + ); + using var output = new MemoryStream(); + + Assert.Throws(() => stream.CopyTo(output)); + } + + // A stream that ends exactly where a block header is expected (no stream footer) is the shape a caller + // sees when decoding a truncated stream or a sub-range of blocks extracted for random access. "BZh9" is + // a valid header with no blocks and no footer, so the very first block-header read hits end-of-input. + [Fact] + public void BZip2Stream_TolerateTruncatedStream_DecodesFooterlessStreamAsEmpty() + { + var headerOnly = Encoding.ASCII.GetBytes("BZh9"); + + Assert.Throws(() => + Decompress(headerOnly, tolerateTruncatedStream: false) + ); + + Assert.Empty(Decompress(headerOnly, tolerateTruncatedStream: true)); + } + + // A real block followed by end-of-input at the next block boundary: decode a complete stream, then + // append another header that stops before its first block. With tolerance the first stream's data comes + // back and the truncated continuation ends cleanly; without it, the end-of-input throws. + [Fact] + public void BZip2Stream_TolerateTruncatedStream_DecodesStreamTruncatedAtBlockBoundary() + { + const string text = "Some data that bzip2 will put into a single block."; + var truncated = Concat(Compress(text), Encoding.ASCII.GetBytes("BZh9")); + + Assert.Throws(() => + Decompress(truncated, tolerateTruncatedStream: false, decompressConcatenated: true) + ); + + var result = Decompress( + truncated, + tolerateTruncatedStream: true, + decompressConcatenated: true + ); + Assert.Equal(text, Encoding.ASCII.GetString(result)); + } + + // A partial decode's running combined CRC won't match the whole-stream value in the footer, so the + // flag skips that whole-stream check (per-block CRCs are still enforced). Corrupting the stored + // combined CRC is fatal by default but tolerated with the flag. + [Fact] + public void BZip2Stream_TolerateTruncatedStream_SkipsWholeStreamCrc() + { + const string text = "BZip2 combined-CRC validation test data."; + var compressed = Compress(text); + compressed[^5] ^= 1; + + Assert.Throws(() => + Decompress(compressed, tolerateTruncatedStream: false) + ); + + var result = Decompress(compressed, tolerateTruncatedStream: true); + Assert.Equal(text, Encoding.ASCII.GetString(result)); + } + + // The flag must not change decoding of a normal, well-formed stream. + [Fact] + public void BZip2Stream_TolerateTruncatedStream_StillDecodesCompleteStream() + { + const string text = + "Round trip with tolerateTruncatedStream set on a complete, valid stream."; + + var result = Decompress(Compress(text), tolerateTruncatedStream: true); + + Assert.Equal(text, Encoding.ASCII.GetString(result)); + } + + private static byte[] Decompress( + byte[] compressed, + bool tolerateTruncatedStream, + bool decompressConcatenated = false + ) + { + using var stream = BZip2Stream.Create( + new MemoryStream(compressed), + CompressionMode.Decompress, + decompressConcatenated, + leaveOpen: false, + tolerateTruncatedStream: tolerateTruncatedStream + ); + using var output = new MemoryStream(); + stream.CopyTo(output); + return output.ToArray(); + } + + private static byte[] Concat(byte[] a, byte[] b) + { + var result = new byte[a.Length + b.Length]; + Buffer.BlockCopy(a, 0, result, 0, a.Length); + Buffer.BlockCopy(b, 0, result, a.Length, b.Length); + return result; + } + + private static byte[] Compress(string value) + { + using var memoryStream = new MemoryStream(); + using ( + var bzip2Stream = BZip2Stream.Create( + memoryStream, + SharpCompress.Compressors.CompressionMode.Compress, + false, + leaveOpen: true + ) + ) + { + var bytes = Encoding.ASCII.GetBytes(value); + bzip2Stream.Write(bytes, 0, bytes.Length); + } + + return memoryStream.ToArray(); + } +} diff --git a/tests/SharpCompress.Test/BinaryReaderParityTests.cs b/tests/SharpCompress.Test/BinaryReaderParityTests.cs new file mode 100644 index 00000000..b6b5c910 --- /dev/null +++ b/tests/SharpCompress.Test/BinaryReaderParityTests.cs @@ -0,0 +1,188 @@ +using System; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.IO; +using Xunit; + +namespace SharpCompress.Test; + +public class BinaryReaderParityTests : TestBase +{ + private readonly byte[] _testData; + + public BinaryReaderParityTests() + { + // Create test data with various patterns + _testData = new byte[256]; + for (int i = 0; i < 256; i++) + { + _testData[i] = (byte)i; + } + } + + [Fact] + public async Task ReadByte_Parity() + { + using var syncStream = new MemoryStream(_testData); + using var asyncStream = new MemoryStream(_testData); + using var syncReader = new BinaryReader(syncStream); + using var asyncReader = new AsyncBinaryReader(asyncStream); + + for (int i = 0; i < 10; i++) + { + byte syncByte = syncReader.ReadByte(); + byte asyncByte = await asyncReader.ReadByteAsync(); + Assert.Equal(syncByte, asyncByte); + } + } + + [Fact] + public async Task ReadBytes_Parity() + { + using var syncStream = new MemoryStream(_testData); + using var asyncStream = new MemoryStream(_testData); + using var syncReader = new BinaryReader(syncStream); + using var asyncReader = new AsyncBinaryReader(asyncStream); + + var syncBytes = syncReader.ReadBytes(32); + var asyncBuffer = new byte[32]; + await asyncReader.ReadBytesAsync(asyncBuffer, 0, 32); + + Assert.Equal(syncBytes, asyncBuffer); + } + + [Fact] + public async Task ReadUInt16_Parity() + { + using var syncStream = new MemoryStream(_testData); + using var asyncStream = new MemoryStream(_testData); + using var syncReader = new BinaryReader(syncStream); + using var asyncReader = new AsyncBinaryReader(asyncStream); + + ushort syncValue = syncReader.ReadUInt16(); + ushort asyncValue = await asyncReader.ReadUInt16Async(); + + Assert.Equal(syncValue, asyncValue); + } + + [Fact] + public async Task ReadUInt32_Parity() + { + using var syncStream = new MemoryStream(_testData); + using var asyncStream = new MemoryStream(_testData); + using var syncReader = new BinaryReader(syncStream); + using var asyncReader = new AsyncBinaryReader(asyncStream); + + uint syncValue = syncReader.ReadUInt32(); + uint asyncValue = await asyncReader.ReadUInt32Async(); + + Assert.Equal(syncValue, asyncValue); + } + + [Fact] + public async Task ReadUInt64_Parity() + { + using var syncStream = new MemoryStream(_testData); + using var asyncStream = new MemoryStream(_testData); + using var syncReader = new BinaryReader(syncStream); + using var asyncReader = new AsyncBinaryReader(asyncStream); + + ulong syncValue = syncReader.ReadUInt64(); + ulong asyncValue = await asyncReader.ReadUInt64Async(); + + Assert.Equal(syncValue, asyncValue); + } + + [Fact] + public async Task Position_Parity() + { + using var syncStream = new MemoryStream(_testData); + using var asyncStream = new MemoryStream(_testData); + using var syncReader = new BinaryReader(syncStream); + using var asyncReader = new AsyncBinaryReader(asyncStream); + + // Read some bytes + syncReader.ReadBytes(10); + var asyncBuffer = new byte[10]; + await asyncReader.ReadBytesAsync(asyncBuffer, 0, 10); + + Assert.Equal(syncStream.Position, asyncStream.Position); + Assert.Equal(syncReader.BaseStream.Position, asyncReader.BaseStream.Position); + } + + [Fact] + public async Task MultipleReads_Parity() + { + using var syncStream = new MemoryStream(_testData); + using var asyncStream = new MemoryStream(_testData); + using var syncReader = new BinaryReader(syncStream); + using var asyncReader = new AsyncBinaryReader(asyncStream); + + // Mix of different read operations + Assert.Equal(syncReader.ReadByte(), await asyncReader.ReadByteAsync()); + + var syncBytes = new byte[16]; + var asyncBytes = new byte[16]; + syncStream.Read(syncBytes, 0, 16); + await asyncReader.ReadBytesAsync(asyncBytes, 0, 16); + Assert.Equal(syncBytes, asyncBytes); + + Assert.Equal(syncReader.ReadUInt16(), await asyncReader.ReadUInt16Async()); + Assert.Equal(syncReader.ReadUInt32(), await asyncReader.ReadUInt32Async()); + Assert.Equal(syncReader.ReadUInt64(), await asyncReader.ReadUInt64Async()); + } + + [Fact] + public async Task ReadByte_Async_Properly_Async() + { + using var stream = new MemoryStream(_testData); + using var reader = new AsyncBinaryReader(stream); + + var bytes = new byte[10]; + for (int i = 0; i < 10; i++) + { + bytes[i] = await reader.ReadByteAsync(); + } + + Assert.Equal(_testData.Take(10).ToArray(), bytes); + } + + [Fact] + public async Task ReadBytes_Async_Properly_Async() + { + using var stream = new MemoryStream(_testData); + using var reader = new AsyncBinaryReader(stream); + + var buffer = new byte[64]; + await reader.ReadBytesAsync(buffer, 0, 64); + + Assert.Equal(_testData.Take(64).ToArray(), buffer); + } + + [Fact] + public async Task SkipAsync_Moves_Position_Correctly() + { + using var stream = new MemoryStream(_testData); + using var reader = new AsyncBinaryReader(stream); + + await reader.SkipAsync(10); + + Assert.Equal(10, stream.Position); + Assert.Equal(10, reader.BaseStream.Position); + } + + [Fact] + public async Task SkipAsync_Then_Read_Works_Correctly() + { + using var stream = new MemoryStream(_testData); + using var reader = new AsyncBinaryReader(stream); + + await reader.SkipAsync(10); + byte b = await reader.ReadByteAsync(); + + Assert.Equal(_testData[10], b); + } +} diff --git a/tests/SharpCompress.Test/CompressionProviderTests.cs b/tests/SharpCompress.Test/CompressionProviderTests.cs new file mode 100644 index 00000000..1b043efd --- /dev/null +++ b/tests/SharpCompress.Test/CompressionProviderTests.cs @@ -0,0 +1,893 @@ +using System; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using AwesomeAssertions; +using SharpCompress.Archives.Tar; +using SharpCompress.Common; +using SharpCompress.Common.Options; +using SharpCompress.Compressors; +using SharpCompress.IO; +using SharpCompress.Providers; +using SharpCompress.Providers.Default; +using SharpCompress.Providers.System; +using SharpCompress.Readers; +using SharpCompress.Readers.Tar; +using SharpCompress.Writers; +using SharpCompress.Writers.Tar; +using Xunit; + +namespace SharpCompress.Test; + +public class CompressionProviderTests +{ + private sealed class TrackingCompressionProvider : ICompressionProvider + { + private readonly ICompressionProvider _inner; + + public TrackingCompressionProvider(ICompressionProvider inner) + { + _inner = inner; + } + + public int CompressionCalls { get; private set; } + + public int DecompressionCalls { get; private set; } + + public int AsyncCompressionCalls { get; private set; } + + public int AsyncDecompressionCalls { get; private set; } + + public CompressionType CompressionType => _inner.CompressionType; + + public bool SupportsCompression => _inner.SupportsCompression; + + public bool SupportsDecompression => _inner.SupportsDecompression; + + public Stream CreateCompressStream(Stream destination, int compressionLevel) + { + CompressionCalls++; + return _inner.CreateCompressStream(destination, compressionLevel); + } + + public Stream CreateCompressStream( + Stream destination, + int compressionLevel, + CompressionContext context + ) + { + CompressionCalls++; + return _inner.CreateCompressStream(destination, compressionLevel, context); + } + + public Stream CreateDecompressStream(Stream source) + { + DecompressionCalls++; + return _inner.CreateDecompressStream(source); + } + + public Stream CreateDecompressStream(Stream source, CompressionContext context) + { + DecompressionCalls++; + return _inner.CreateDecompressStream(source, context); + } + + public ValueTask CreateCompressStreamAsync( + Stream destination, + int compressionLevel, + CancellationToken cancellationToken = default + ) + { + AsyncCompressionCalls++; + return _inner.CreateCompressStreamAsync( + destination, + compressionLevel, + cancellationToken + ); + } + + public ValueTask CreateCompressStreamAsync( + Stream destination, + int compressionLevel, + CompressionContext context, + CancellationToken cancellationToken = default + ) + { + AsyncCompressionCalls++; + return _inner.CreateCompressStreamAsync( + destination, + compressionLevel, + context, + cancellationToken + ); + } + + public ValueTask CreateDecompressStreamAsync( + Stream source, + CancellationToken cancellationToken = default + ) + { + AsyncDecompressionCalls++; + return _inner.CreateDecompressStreamAsync(source, cancellationToken); + } + + public ValueTask CreateDecompressStreamAsync( + Stream source, + CompressionContext context, + CancellationToken cancellationToken = default + ) + { + AsyncDecompressionCalls++; + return _inner.CreateDecompressStreamAsync(source, context, cancellationToken); + } + } + + private sealed class TrackingLzmaHooksProvider : ICompressionProviderHooks + { + public int PreCalls { get; private set; } + public int PropertiesCalls { get; private set; } + public int PostCalls { get; private set; } + + public CompressionType CompressionType => CompressionType.LZMA; + + public bool SupportsCompression => true; + + public bool SupportsDecompression => false; + + public Stream CreateCompressStream(Stream destination, int compressionLevel) + { + CompressionContext context = new() { CanSeek = destination.CanSeek }; + return CreateCompressStream(destination, compressionLevel, context); + } + + public Stream CreateCompressStream( + Stream destination, + int compressionLevel, + CompressionContext context + ) => SharpCompressStream.CreateNonDisposing(destination); + + public Stream CreateDecompressStream(Stream source) => throw new NotSupportedException(); + + public Stream CreateDecompressStream(Stream source, CompressionContext context) => + throw new NotSupportedException(); + + public ValueTask CreateCompressStreamAsync( + Stream destination, + int compressionLevel, + CancellationToken cancellationToken = default + ) + { + CompressionContext context = new() { CanSeek = destination.CanSeek }; + return CreateCompressStreamAsync( + destination, + compressionLevel, + context, + cancellationToken + ); + } + + public ValueTask CreateCompressStreamAsync( + Stream destination, + int compressionLevel, + CompressionContext context, + CancellationToken cancellationToken = default + ) + { + cancellationToken.ThrowIfCancellationRequested(); + return new ValueTask(SharpCompressStream.CreateNonDisposing(destination)); + } + + public ValueTask CreateDecompressStreamAsync( + Stream source, + CancellationToken cancellationToken = default + ) => throw new NotSupportedException(); + + public ValueTask CreateDecompressStreamAsync( + Stream source, + CompressionContext context, + CancellationToken cancellationToken = default + ) => throw new NotSupportedException(); + + public byte[]? GetPreCompressionData(CompressionContext context) + { + PreCalls++; + return []; + } + + public byte[]? GetCompressionProperties(Stream stream, CompressionContext context) + { + PropertiesCalls++; + return []; + } + + public byte[]? GetPostCompressionData(Stream stream, CompressionContext context) + { + PostCalls++; + return [1, 2, 3]; + } + } + + private sealed class ContextRequiredGZipProvider : CompressionProviderBase + { + private readonly GZipCompressionProvider _inner = new(); + + public override CompressionType CompressionType => CompressionType.GZip; + + public override bool SupportsCompression => true; + + public override bool SupportsDecompression => true; + + public override Stream CreateCompressStream(Stream destination, int compressionLevel) => + _inner.CreateCompressStream(destination, compressionLevel); + + public override Stream CreateDecompressStream(Stream source) => + throw new InvalidOperationException("Context is required for GZip decompression."); + + public override Stream CreateDecompressStream(Stream source, CompressionContext context) + { + context.ReaderOptions.Should().NotBeNull(); + return _inner.CreateDecompressStream(source, context); + } + } + + [Fact] + public void CompressionProviderRegistry_Default_ReturnsInternalProviders() + { + var registry = CompressionProviderRegistry.Default; + + registry.GetProvider(CompressionType.Deflate).Should().NotBeNull(); + registry.GetProvider(CompressionType.GZip).Should().NotBeNull(); + registry.GetProvider(CompressionType.BZip2).Should().NotBeNull(); + registry.GetProvider(CompressionType.ZStandard).Should().NotBeNull(); + registry.GetProvider(CompressionType.LZip).Should().NotBeNull(); + registry.GetProvider(CompressionType.Xz).Should().NotBeNull(); + registry.GetProvider(CompressionType.Lzw).Should().NotBeNull(); + } + + [Fact] + public void CompressionProviderRegistry_With_ReplacesProvider() + { + var customProvider = new DeflateCompressionProvider(); + var registry = CompressionProviderRegistry.Default.With(customProvider); + + // Should return the new provider + var retrieved = registry.GetProvider(CompressionType.Deflate); + retrieved.Should().BeSameAs(customProvider); + } + + [Fact] + public void CompressionProviderRegistry_With_DoesNotModifyOriginal() + { + var original = CompressionProviderRegistry.Default; + var customProvider = new DeflateCompressionProvider(); + var modified = original.With(customProvider); + + // Original should still have the default provider + var originalProvider = original.GetProvider(CompressionType.Deflate); + var modifiedProvider = modified.GetProvider(CompressionType.Deflate); + originalProvider.Should().NotBeSameAs(modifiedProvider); + originalProvider.Should().NotBeSameAs(customProvider); + } + + [Fact] + public void DeflateProvider_RoundTrip_Works() + { + var provider = new DeflateCompressionProvider(); + var original = Encoding.UTF8.GetBytes("Hello, World! This is a test of compression."); + + using var compressedStream = new MemoryStream(); + // Wrap in NonDisposingStream so the compression stream doesn't close it + var nonDisposingStream = SharpCompressStream.CreateNonDisposing(compressedStream); + using (var compressStream = provider.CreateCompressStream(nonDisposingStream, 6)) + { + compressStream.Write(original, 0, original.Length); + } + + compressedStream.Position = 0; + using var decompressStream = provider.CreateDecompressStream(compressedStream); + using var resultStream = new MemoryStream(); + decompressStream.CopyTo(resultStream); + + var result = resultStream.ToArray(); + result.Should().Equal(original); + } + + [Fact] + public void GZipProvider_RoundTrip_Works() + { + var provider = new GZipCompressionProvider(); + var original = Encoding.UTF8.GetBytes("Hello, World! This is a test of compression."); + + using var compressedStream = new MemoryStream(); + // Wrap in NonDisposingStream so the compression stream doesn't close it + var nonDisposingStream = SharpCompressStream.CreateNonDisposing(compressedStream); + using (var compressStream = provider.CreateCompressStream(nonDisposingStream, 6)) + { + compressStream.Write(original, 0, original.Length); + } + + compressedStream.Position = 0; + using var decompressStream = provider.CreateDecompressStream(compressedStream); + using var resultStream = new MemoryStream(); + decompressStream.CopyTo(resultStream); + + var result = resultStream.ToArray(); + result.Should().Equal(original); + } + + [Fact] + public void GZipProvider_Decompress_WithReaderOptionsContext_UsesArchiveEncoding() + { + var provider = new GZipCompressionProvider(); + var data = Encoding.UTF8.GetBytes("gzip filename encoding"); + var expectedFileName = "café.txt"; + var archiveEncoding = new ArchiveEncoding { Default = Encoding.GetEncoding("iso-8859-1") }; + + using var compressedStream = CreateGZipWithFileName( + data, + expectedFileName, + archiveEncoding.Default + ); + + compressedStream.Position = 0; + var readerOptions = ReaderOptions.ForExternalStream with + { + ArchiveEncoding = archiveEncoding, + }; + var context = CompressionContext.FromStream(compressedStream) with + { + ReaderOptions = readerOptions, + }; + + using var decompressStream = provider.CreateDecompressStream(compressedStream, context); + using var resultStream = new MemoryStream(); + decompressStream.CopyTo(resultStream); + + resultStream.ToArray().Should().Equal(data); + decompressStream.Should().BeOfType(); + var gzipStream = (SharpCompress.Compressors.Deflate.GZipStream)decompressStream; + gzipStream.FileName.Should().Be(expectedFileName); + } + + [Fact] + public void GZipProvider_Decompress_WithNullReaderOptions_FallsBackToUtf8() + { + var provider = new GZipCompressionProvider(); + var data = Encoding.UTF8.GetBytes("gzip filename encoding"); + var expectedFileName = "café.txt"; + + using var compressedStream = CreateGZipWithFileName(data, expectedFileName, Encoding.UTF8); + + compressedStream.Position = 0; + var context = CompressionContext.FromStream(compressedStream); + // ReaderOptions is null by default + + using var decompressStream = provider.CreateDecompressStream(compressedStream, context); + using var resultStream = new MemoryStream(); + decompressStream.CopyTo(resultStream); + + resultStream.ToArray().Should().Equal(data); + var gzipStream = (SharpCompress.Compressors.Deflate.GZipStream)decompressStream; + gzipStream.FileName.Should().Be(expectedFileName); + } + + [Fact] + public void BZip2Provider_SupportsCompressionAndDecompression() + { + var provider = new BZip2CompressionProvider(); + + // Verify the provider reports correct capabilities + provider.CompressionType.Should().Be(CompressionType.BZip2); + provider.SupportsCompression.Should().BeTrue(); + provider.SupportsDecompression.Should().BeTrue(); + } + + [Fact] + public void TarWriter_WithCustomProvider_UsesProvider() + { + var customProvider = new GZipCompressionProvider(); + var registry = CompressionProviderRegistry.Default.With(customProvider); + + using var stream = new MemoryStream(); + var options = new TarWriterOptions(CompressionType.GZip, true) { Providers = registry }; + + using (var writer = new TarWriter(stream, options)) + { + var data = Encoding.UTF8.GetBytes("Test content"); + writer.Write("test.txt", new MemoryStream(data), DateTime.Now); + } + + // Should have written compressed data + stream.Position = 0; + stream.Length.Should().BeGreaterThan(0); + } + + [Fact] + public void TarWriter_WithoutCustomProvider_UsesDefault() + { + using var stream = new MemoryStream(); + var options = new TarWriterOptions(CompressionType.GZip, true); + + using (var writer = new TarWriter(stream, options)) + { + var data = Encoding.UTF8.GetBytes("Test content"); + writer.Write("test.txt", new MemoryStream(data), DateTime.Now); + } + + stream.Position = 0; + stream.Length.Should().BeGreaterThan(0); + } + + [Fact] + public void TarReader_WithCustomProvider_UsesProvider() + { + // First, create a tar.gz file + using var archiveStream = new MemoryStream(); + var writeOptions = new TarWriterOptions(CompressionType.GZip, true); + using (var writer = new TarWriter(archiveStream, writeOptions)) + { + var data = Encoding.UTF8.GetBytes("Test content for reading"); + writer.Write("test.txt", new MemoryStream(data), DateTime.Now); + } + + // Now read it back with a custom provider + archiveStream.Position = 0; + var customProvider = new GZipCompressionProvider(); + var registry = CompressionProviderRegistry.Default.With(customProvider); + var readOptions = ReaderOptions.ForExternalStream.WithProviders(registry); + + using var reader = TarReader.OpenReader(archiveStream, readOptions); + reader.MoveToNextEntry().Should().BeTrue(); + using var entryStream = reader.OpenEntryStream(); + using var resultStream = new MemoryStream(); + entryStream.CopyTo(resultStream); + + var result = Encoding.UTF8.GetString(resultStream.ToArray()); + result.Should().Be("Test content for reading"); + } + + [Fact] + public void TarReader_OpenReader_WithContextRequiredGZipProvider_Succeeds() + { + using var archiveStream = new MemoryStream(); + using ( + var writer = new TarWriter( + archiveStream, + new TarWriterOptions(CompressionType.GZip, true) + ) + ) + { + var data = Encoding.UTF8.GetBytes("Test content for context-required provider"); + writer.Write("test.txt", new MemoryStream(data), DateTime.Now); + } + + archiveStream.Position = 0; + var registry = CompressionProviderRegistry.Default.With(new ContextRequiredGZipProvider()); + var readOptions = ReaderOptions.ForExternalStream.WithProviders(registry); + + using var reader = TarReader.OpenReader(archiveStream, readOptions); + reader.MoveToNextEntry().Should().BeTrue(); + using var entryStream = reader.OpenEntryStream(); + using var resultStream = new MemoryStream(); + entryStream.CopyTo(resultStream); + + var result = Encoding.UTF8.GetString(resultStream.ToArray()); + result.Should().Be("Test content for context-required provider"); + } + + [Fact] + public void WriterOptions_WithProviders_CanBeCloned() + { + var customProvider = new DeflateCompressionProvider(); + var registry = CompressionProviderRegistry.Default.With(customProvider); + + var original = new WriterOptions(CompressionType.GZip) + { + Providers = registry, + LeaveStreamOpen = false, + }; + + // Clone using 'with' expression + var clone = original with + { + LeaveStreamOpen = true, + }; + + clone.CompressionType.Should().Be(original.CompressionType); + clone.CompressionLevel.Should().Be(original.CompressionLevel); + clone.Providers.Should().BeSameAs(original.Providers); + clone.LeaveStreamOpen.Should().BeTrue(); + } + + [Fact] + public void ReaderOptions_WithProviders_CanBeCloned() + { + var customProvider = new DeflateCompressionProvider(); + var registry = CompressionProviderRegistry.Default.With(customProvider); + + var original = ReaderOptions.ForExternalStream with + { + Providers = registry, + LeaveStreamOpen = false, + }; + + // Clone using 'with' expression + var clone = original with + { + LeaveStreamOpen = true, + }; + + clone.Providers.Should().BeSameAs(original.Providers); + clone.LeaveStreamOpen.Should().BeTrue(); + } + + [Fact] + public void TarArchive_OpenArchive_UsesCustomGZipProvider() + { + using var archiveStream = new MemoryStream(); + using ( + var writer = new TarWriter( + archiveStream, + new TarWriterOptions(CompressionType.GZip, true) + ) + ) + { + var data = Encoding.UTF8.GetBytes("tar archive provider usage"); + writer.Write("test.txt", new MemoryStream(data), DateTime.Now); + } + + var trackingProvider = new TrackingCompressionProvider(new GZipCompressionProvider()); + var registry = CompressionProviderRegistry.Default.With(trackingProvider); + var readOptions = ReaderOptions.ForExternalStream.WithProviders(registry); + + archiveStream.Position = 0; + using var archive = TarArchive.OpenArchive(archiveStream, readOptions); + var entry = archive.Entries.First(x => !x.IsDirectory); + using var entryStream = entry.OpenEntryStream(); + using var resultStream = new MemoryStream(); + entryStream.CopyTo(resultStream); + + trackingProvider.DecompressionCalls.Should().BeGreaterThan(0); + } + + [Fact] + public async Task TarArchive_OpenAsyncArchive_UsesCustomGZipProvider() + { + using var archiveStream = new MemoryStream(); + using ( + var writer = new TarWriter( + archiveStream, + new TarWriterOptions(CompressionType.GZip, true) + ) + ) + { + var data = Encoding.UTF8.GetBytes("tar async archive provider usage"); + writer.Write("test.txt", new MemoryStream(data), DateTime.Now); + } + + var trackingProvider = new TrackingCompressionProvider(new GZipCompressionProvider()); + var registry = CompressionProviderRegistry.Default.With(trackingProvider); + var readOptions = ReaderOptions.ForExternalStream.WithProviders(registry); + + archiveStream.Position = 0; + await using var archive = await TarArchive.OpenAsyncArchive(archiveStream, readOptions); + await foreach (var entry in archive.EntriesAsync) + { + if (entry.IsDirectory) + { + continue; + } + + using var entryStream = await entry.OpenEntryStreamAsync(); + using var resultStream = new MemoryStream(); + await entryStream.CopyToAsync(resultStream); + break; + } + + trackingProvider.AsyncDecompressionCalls.Should().BeGreaterThan(0); + } + + [Fact] + public async Task ZipReader_OpenEntryStreamAsync_UsesCustomDeflateProvider() + { + using var zipStream = new MemoryStream(); + using ( + var writer = WriterFactory.OpenWriter( + zipStream, + ArchiveType.Zip, + new WriterOptions(CompressionType.Deflate) { LeaveStreamOpen = true } + ) + ) + { + var data = Encoding.UTF8.GetBytes("zip async provider usage"); + writer.Write("test.txt", new MemoryStream(data)); + } + + var trackingProvider = new TrackingCompressionProvider(new DeflateCompressionProvider()); + var registry = CompressionProviderRegistry.Default.With(trackingProvider); + var options = ReaderOptions.ForExternalStream.WithProviders(registry); + + zipStream.Position = 0; + await using var reader = await ReaderFactory.OpenAsyncReader(zipStream, options); + (await reader.MoveToNextEntryAsync()).Should().BeTrue(); + using var entryStream = await reader.OpenEntryStreamAsync(); + using var resultStream = new MemoryStream(); + await entryStream.CopyToAsync(resultStream); + + trackingProvider.AsyncDecompressionCalls.Should().BeGreaterThan(0); + } + + [Fact] + public void LzwReader_OpenReader_UsesCustomLzwProvider() + { + var archivePath = Path.Combine(TestBase.TEST_ARCHIVES_PATH, "Tar.tar.Z"); + var trackingProvider = new TrackingCompressionProvider(new LzwCompressionProvider()); + var registry = CompressionProviderRegistry.Default.With(trackingProvider); + var options = ReaderOptions.ForExternalStream.WithProviders(registry); + + using var stream = File.OpenRead(archivePath); + using var reader = ReaderFactory.OpenReader(stream, options); + reader.MoveToNextEntry().Should().BeTrue(); + reader.WriteEntryTo(Stream.Null); + + trackingProvider.DecompressionCalls.Should().BeGreaterThan(0); + } + + [Fact] + public void ZipWriter_LzmaProviderHook_WritesPostCompressionData() + { + var trackingProvider = new TrackingLzmaHooksProvider(); + var registry = CompressionProviderRegistry.Default.With(trackingProvider); + using var zipStream = new MemoryStream(); + + using ( + var writer = WriterFactory.OpenWriter( + zipStream, + ArchiveType.Zip, + new WriterOptions(CompressionType.LZMA) + { + LeaveStreamOpen = true, + Providers = registry, + } + ) + ) + { + var data = Encoding.UTF8.GetBytes("hook provider"); + writer.Write("test.txt", new MemoryStream(data)); + } + + trackingProvider.PreCalls.Should().BeGreaterThan(0); + trackingProvider.PropertiesCalls.Should().BeGreaterThan(0); + trackingProvider.PostCalls.Should().BeGreaterThan(0); + } + + #region System.IO.Compression Tests + + private static MemoryStream CreateGZipWithFileName( + byte[] data, + string fileName, + Encoding encoding + ) + { + var compressedStream = new MemoryStream(); + using ( + var compressStream = new SharpCompress.Compressors.Deflate.GZipStream( + SharpCompressStream.CreateNonDisposing(compressedStream), + CompressionMode.Compress, + SharpCompress.Compressors.Deflate.CompressionLevel.Default, + encoding + ) + ) + { + compressStream.FileName = fileName; + compressStream.Write(data, 0, data.Length); + } + + compressedStream.Position = 0; + return compressedStream; + } + + [Fact] + public void SystemGZipProvider_RoundTrip_Works() + { + var provider = new SystemGZipCompressionProvider(); + var original = Encoding.UTF8.GetBytes( + "Hello, World! This is a test of System.IO.Compression.GZipStream." + ); + + using var compressedStream = new MemoryStream(); + var nonDisposingStream = SharpCompressStream.CreateNonDisposing(compressedStream); + using (var compressStream = provider.CreateCompressStream(nonDisposingStream, 6)) + { + compressStream.Write(original, 0, original.Length); + } + + compressedStream.Position = 0; + using var decompressStream = provider.CreateDecompressStream(compressedStream); + using var resultStream = new MemoryStream(); + decompressStream.CopyTo(resultStream); + + var result = resultStream.ToArray(); + result.Should().Equal(original); + } + + [Theory] + [InlineData(0)] // No compression + [InlineData(3)] // Fast + [InlineData(6)] // Default + [InlineData(9)] // Best compression + public void SystemGZipProvider_DifferentCompressionLevels_Work(int level) + { + var provider = new SystemGZipCompressionProvider(); + var original = Encoding.UTF8.GetBytes( + "Test data for compression level testing with System.IO.Compression." + ); + + using var compressedStream = new MemoryStream(); + var nonDisposingStream = SharpCompressStream.CreateNonDisposing(compressedStream); + using (var compressStream = provider.CreateCompressStream(nonDisposingStream, level)) + { + compressStream.Write(original, 0, original.Length); + } + + compressedStream.Position = 0; + using var decompressStream = provider.CreateDecompressStream(compressedStream); + using var resultStream = new MemoryStream(); + decompressStream.CopyTo(resultStream); + + var result = resultStream.ToArray(); + result.Should().Equal(original); + } + + [Fact] + public void SystemGZipProvider_Compress_InternalProvider_Decompress_CrossCompatibility() + { + // Compress with System.IO.Compression + var systemProvider = new SystemGZipCompressionProvider(); + var original = Encoding.UTF8.GetBytes( + "Cross-compatibility test between System.IO.Compression and internal GZip." + ); + + using var compressedStream = new MemoryStream(); + var nonDisposingStream = SharpCompressStream.CreateNonDisposing(compressedStream); + using (var compressStream = systemProvider.CreateCompressStream(nonDisposingStream, 6)) + { + compressStream.Write(original, 0, original.Length); + } + + // Decompress with internal provider + compressedStream.Position = 0; + var internalProvider = new GZipCompressionProvider(); + using var decompressStream = internalProvider.CreateDecompressStream(compressedStream); + using var resultStream = new MemoryStream(); + decompressStream.CopyTo(resultStream); + + var result = resultStream.ToArray(); + result.Should().Equal(original); + } + + [Fact] + public void InternalProvider_Compress_SystemGZipProvider_Decompress_CrossCompatibility() + { + // Compress with internal provider + var internalProvider = new GZipCompressionProvider(); + var original = Encoding.UTF8.GetBytes( + "Cross-compatibility test between internal GZip and System.IO.Compression." + ); + + using var compressedStream = new MemoryStream(); + var nonDisposingStream = SharpCompressStream.CreateNonDisposing(compressedStream); + using (var compressStream = internalProvider.CreateCompressStream(nonDisposingStream, 6)) + { + compressStream.Write(original, 0, original.Length); + } + + // Decompress with System.IO.Compression + compressedStream.Position = 0; + var systemProvider = new SystemGZipCompressionProvider(); + using var decompressStream = systemProvider.CreateDecompressStream(compressedStream); + using var resultStream = new MemoryStream(); + decompressStream.CopyTo(resultStream); + + var result = resultStream.ToArray(); + result.Should().Equal(original); + } + + [Fact] + public void TarWriter_WithSystemGZipProvider_CreatesReadableArchive() + { + // Create tar.gz using System.IO.Compression provider + var systemProvider = new SystemGZipCompressionProvider(); + var registry = CompressionProviderRegistry.Default.With(systemProvider); + + using var archiveStream = new MemoryStream(); + var writeOptions = new TarWriterOptions(CompressionType.GZip, true) + { + Providers = registry, + }; + + using (var writer = new TarWriter(archiveStream, writeOptions)) + { + var data = Encoding.UTF8.GetBytes("Content written with System.IO.Compression"); + writer.Write("test.txt", new MemoryStream(data), DateTime.Now); + } + + // Read back using internal provider (should be compatible) + archiveStream.Position = 0; + var readOptions = ReaderOptions.ForExternalStream; + using var reader = TarReader.OpenReader(archiveStream, readOptions); + reader.MoveToNextEntry().Should().BeTrue(); + using var entryStream = reader.OpenEntryStream(); + using var resultStream = new MemoryStream(); + entryStream.CopyTo(resultStream); + + var result = Encoding.UTF8.GetString(resultStream.ToArray()); + result.Should().Be("Content written with System.IO.Compression"); + } + + [Fact] + public void SystemGZipProvider_SupportsCompressionAndDecompression() + { + var provider = new SystemGZipCompressionProvider(); + + // Verify the provider reports correct capabilities + provider.CompressionType.Should().Be(CompressionType.GZip); + provider.SupportsCompression.Should().BeTrue(); + provider.SupportsDecompression.Should().BeTrue(); + } + + [Fact] + public void SystemDeflateProvider_RoundTrip_Works() + { + var provider = new SystemDeflateCompressionProvider(); + var original = Encoding.UTF8.GetBytes( + "Hello, World! This is a test of System.IO.Compression.DeflateStream." + ); + + using var compressedStream = new MemoryStream(); + var nonDisposingStream = SharpCompressStream.CreateNonDisposing(compressedStream); + using (var compressStream = provider.CreateCompressStream(nonDisposingStream, 6)) + { + compressStream.Write(original, 0, original.Length); + } + + compressedStream.Position = 0; + using var decompressStream = provider.CreateDecompressStream(compressedStream); + using var resultStream = new MemoryStream(); + decompressStream.CopyTo(resultStream); + + var result = resultStream.ToArray(); + result.Should().Equal(original); + } + + [Fact] + public void SystemDeflateProvider_Compress_InternalProvider_Decompress_CrossCompatibility() + { + // Compress with System.IO.Compression Deflate + var systemProvider = new SystemDeflateCompressionProvider(); + var original = Encoding.UTF8.GetBytes( + "Cross-compatibility test between System.IO.Compression and internal Deflate." + ); + + using var compressedStream = new MemoryStream(); + var nonDisposingStream = SharpCompressStream.CreateNonDisposing(compressedStream); + using (var compressStream = systemProvider.CreateCompressStream(nonDisposingStream, 6)) + { + compressStream.Write(original, 0, original.Length); + } + + // Decompress with internal provider + compressedStream.Position = 0; + var internalProvider = new DeflateCompressionProvider(); + using var decompressStream = internalProvider.CreateDecompressStream(compressedStream); + using var resultStream = new MemoryStream(); + decompressStream.CopyTo(resultStream); + + var result = resultStream.ToArray(); + result.Should().Equal(original); + } + + #endregion +} diff --git a/tests/SharpCompress.Test/ExceptionHierarchyTests.cs b/tests/SharpCompress.Test/ExceptionHierarchyTests.cs new file mode 100644 index 00000000..4fd7f25b --- /dev/null +++ b/tests/SharpCompress.Test/ExceptionHierarchyTests.cs @@ -0,0 +1,121 @@ +using System; +using SharpCompress.Common; +using SharpCompress.Compressors.Deflate; +using SharpCompress.Compressors.LZMA; +using SharpCompress.Compressors.Xz; +using Xunit; + +namespace SharpCompress.Test; + +public class ExceptionHierarchyTests +{ + [Fact] + public void AllSharpCompressExceptions_InheritFromSharpCompressException() + { + // Verify that ArchiveException inherits from SharpCompressException + Assert.True(typeof(SharpCompressException).IsAssignableFrom(typeof(ArchiveException))); + + // Verify that ArchiveOperationException inherits from SharpCompressException + Assert.True( + typeof(SharpCompressException).IsAssignableFrom(typeof(ArchiveOperationException)) + ); + + // Verify that ExtractionException inherits from SharpCompressException + Assert.True(typeof(SharpCompressException).IsAssignableFrom(typeof(ExtractionException))); + + // Verify that InvalidFormatException inherits from SharpCompressException (through ExtractionException) + Assert.True( + typeof(SharpCompressException).IsAssignableFrom(typeof(InvalidFormatException)) + ); + + // Verify that CryptographicException inherits from SharpCompressException + Assert.True( + typeof(SharpCompressException).IsAssignableFrom(typeof(CryptographicException)) + ); + + // Verify that IncompleteArchiveException inherits from SharpCompressException (through ArchiveException) + Assert.True( + typeof(SharpCompressException).IsAssignableFrom(typeof(IncompleteArchiveException)) + ); + + // Verify that ReaderCancelledException inherits from SharpCompressException + Assert.True( + typeof(SharpCompressException).IsAssignableFrom(typeof(ReaderCancelledException)) + ); + + // Verify that MultipartStreamRequiredException inherits from SharpCompressException (through ExtractionException) + Assert.True( + typeof(SharpCompressException).IsAssignableFrom( + typeof(MultipartStreamRequiredException) + ) + ); + + // Verify that MultiVolumeExtractionException inherits from SharpCompressException (through ExtractionException) + Assert.True( + typeof(SharpCompressException).IsAssignableFrom(typeof(MultiVolumeExtractionException)) + ); + + // Verify that ZlibException inherits from SharpCompressException + Assert.True(typeof(SharpCompressException).IsAssignableFrom(typeof(ZlibException))); + + // Verify that XZIndexMarkerReachedException inherits from SharpCompressException + Assert.True( + typeof(SharpCompressException).IsAssignableFrom(typeof(XZIndexMarkerReachedException)) + ); + } + + [Fact] + public void SharpCompressException_CanBeCaughtByBaseType() + { + // Test that a derived exception can be caught as SharpCompressException + var exception = new InvalidFormatException("Test message"); + var caughtException = false; + + try + { + throw exception; + } + catch (SharpCompressException ex) + { + caughtException = true; + Assert.Same(exception, ex); + } + + Assert.True(caughtException, "Exception should have been caught as SharpCompressException"); + } + + [Fact] + public void InternalLzmaExceptions_InheritFromSharpCompressException() + { + // Use reflection to verify internal exception types + var dataErrorExceptionType = Type.GetType( + "SharpCompress.Compressors.LZMA.DataErrorException, SharpCompress" + ); + Assert.NotNull(dataErrorExceptionType); + Assert.True(typeof(SharpCompressException).IsAssignableFrom(dataErrorExceptionType)); + + var invalidParamExceptionType = Type.GetType( + "SharpCompress.Compressors.LZMA.InvalidParamException, SharpCompress" + ); + Assert.NotNull(invalidParamExceptionType); + Assert.True(typeof(SharpCompressException).IsAssignableFrom(invalidParamExceptionType)); + } + + [Fact] + public void ExceptionConstructors_WorkCorrectly() + { + // Test parameterless constructor + var ex1 = new SharpCompressException(); + Assert.NotNull(ex1); + + // Test message constructor + var ex2 = new SharpCompressException("Test message"); + Assert.Equal("Test message", ex2.Message); + + // Test message and inner exception constructor + var inner = new InvalidOperationException("Inner"); + var ex3 = new SharpCompressException("Test message", inner); + Assert.Equal("Test message", ex3.Message); + Assert.Same(inner, ex3.InnerException); + } +} diff --git a/tests/SharpCompress.Test/ExtractAll.cs b/tests/SharpCompress.Test/ExtractAll.cs new file mode 100644 index 00000000..8b71edf3 --- /dev/null +++ b/tests/SharpCompress.Test/ExtractAll.cs @@ -0,0 +1,44 @@ +using System.IO; +using System.Linq; +using System.Threading.Tasks; +using SharpCompress.Archives; +using SharpCompress.Common; +using SharpCompress.Readers; +using Xunit; + +namespace SharpCompress.Test; + +public class ExtractAllTests : TestBase +{ + [Theory] + [InlineData("Zip.deflate.zip")] + [InlineData("Rar5.rar")] + [InlineData("Rar.rar")] + [InlineData("Rar.solid.rar")] + [InlineData("7Zip.solid.7z")] + [InlineData("7Zip.nonsolid.7z")] + [InlineData("7Zip.LZMA.7z")] + public async ValueTask ExtractAllEntriesAsync(string archivePath) + { + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, archivePath); + + await using var archive = await ArchiveFactory.OpenAsyncArchive(testArchive); + await archive.WriteToDirectoryAsync(SCRATCH_FILES_PATH); + } + + [Theory] + [InlineData("Zip.deflate.zip")] + [InlineData("Rar5.rar")] + [InlineData("Rar.rar")] + [InlineData("Rar.solid.rar")] + [InlineData("7Zip.solid.7z")] + [InlineData("7Zip.nonsolid.7z")] + [InlineData("7Zip.LZMA.7z")] + public void ExtractAllEntriesSync(string archivePath) + { + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, archivePath); + + using var archive = ArchiveFactory.OpenArchive(testArchive); + archive.WriteToDirectory(SCRATCH_FILES_PATH); + } +} diff --git a/tests/SharpCompress.Test/ExtractAllEntriesTests.cs b/tests/SharpCompress.Test/ExtractAllEntriesTests.cs new file mode 100644 index 00000000..cc8a32bd --- /dev/null +++ b/tests/SharpCompress.Test/ExtractAllEntriesTests.cs @@ -0,0 +1,58 @@ +using System.IO; +using System.Linq; +using SharpCompress.Archives; +using SharpCompress.Common; +using SharpCompress.Readers; +using Xunit; + +namespace SharpCompress.Test; + +/// +/// Tests for the ExtractAllEntries method behavior on both solid and non-solid +/// archives, including progress reporting and current usage restrictions. +/// +public class ExtractAllEntriesTests : TestBase +{ + [Fact] + public void ExtractAllEntries_WithProgressReporting_NonSolidArchive() + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.zip"); + + using var archive = ArchiveFactory.OpenArchive(archivePath); + Assert.Throws(() => + { + using var reader = archive.ExtractAllEntries(); + }); + } + + [Fact] + public void ExtractAllEntries_WithProgressReporting_SolidArchive() + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, "Rar.solid.rar"); + + using var archive = ArchiveFactory.OpenArchive(archivePath); + Assert.True(archive.IsSolid); + + // Calculate total size like user code does + double totalSize = archive.Entries.Where(e => !e.IsDirectory).Sum(e => e.Size); + long completed = 0; + var progressReports = 0; + + using var reader = archive.ExtractAllEntries(); + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + reader.WriteEntryToDirectory(SCRATCH_FILES_PATH); + + completed += reader.Entry.Size; + var progress = completed / totalSize; + progressReports++; + + Assert.True(progress >= 0 && progress <= 1.0); + } + } + + Assert.True(progressReports > 0); + } +} diff --git a/tests/SharpCompress.Test/ExtractionTests.cs b/tests/SharpCompress.Test/ExtractionTests.cs new file mode 100644 index 00000000..24ed6bf4 --- /dev/null +++ b/tests/SharpCompress.Test/ExtractionTests.cs @@ -0,0 +1,107 @@ +using System; +using System.IO; +using SharpCompress.Common; +using SharpCompress.Readers; +using SharpCompress.Writers; +using SharpCompress.Writers.Zip; +using Xunit; + +namespace SharpCompress.Test; + +public class ExtractionTests : TestBase +{ + [Fact] + public void Extraction_ShouldHandleCaseInsensitivePathsOnWindows() + { + // This test validates that extraction succeeds when Path.GetFullPath returns paths + // with casing that matches the platform's file system behavior. On Windows, + // Path.GetFullPath can return different casing than the actual directory on disk + // (e.g., "system32" vs "System32"), and the extraction should succeed because + // Windows file systems are case-insensitive. On Unix-like systems, this test + // verifies that the case-sensitive comparison is used correctly. + + var testArchive = Path.Combine(SCRATCH2_FILES_PATH, "test-extraction.zip"); + var extractPath = SCRATCH_FILES_PATH; + + // Create a simple test archive with a single file + using (var stream = File.Create(testArchive)) + { + using var writer = (ZipWriter) + WriterFactory.OpenWriter( + stream, + ArchiveType.Zip, + new WriterOptions(CompressionType.Deflate) + ); + + // Create a test file to add to the archive + var testFilePath = Path.Combine(SCRATCH2_FILES_PATH, "testfile.txt"); + File.WriteAllText(testFilePath, "Test content"); + + writer.Write("testfile.txt", testFilePath); + } + + // Extract the archive - this should succeed regardless of path casing + using (var stream = File.OpenRead(testArchive)) + { + using var reader = ReaderFactory.OpenReader(stream); + + // This should not throw an exception even if Path.GetFullPath returns + // a path with different casing than the actual directory + var exception = Record.Exception(() => + reader.WriteAllToDirectory( + extractPath, + new ExtractionOptions { ExtractFullPath = false, Overwrite = true } + ) + ); + + Assert.Null(exception); + } + + // Verify the file was extracted successfully + var extractedFile = Path.Combine(extractPath, "testfile.txt"); + Assert.True(File.Exists(extractedFile)); + Assert.Equal("Test content", File.ReadAllText(extractedFile)); + } + + [Fact] + public void Extraction_ShouldPreventPathTraversalAttacks() + { + // This test ensures that the security check still works to prevent + // path traversal attacks (e.g., using "../" to escape the destination directory) + + var testArchive = Path.Combine(SCRATCH2_FILES_PATH, "test-traversal.zip"); + var extractPath = SCRATCH_FILES_PATH; + + // Create a test archive with a path traversal attempt + using (var stream = File.Create(testArchive)) + { + using var writer = (ZipWriter) + WriterFactory.OpenWriter( + stream, + ArchiveType.Zip, + new WriterOptions(CompressionType.Deflate) + ); + + var testFilePath = Path.Combine(SCRATCH2_FILES_PATH, "testfile2.txt"); + File.WriteAllText(testFilePath, "Test content"); + + // Try to write with a path that attempts to escape the destination directory + writer.Write("../../evil.txt", testFilePath); + } + + // Extract the archive - this should throw an exception for path traversal + using (var stream = File.OpenRead(testArchive)) + { + using var reader = ReaderFactory.OpenReader(stream); + + var exception = Assert.Throws(() => + reader.WriteAllToDirectory( + extractPath, + new ExtractionOptions { ExtractFullPath = true, Overwrite = true } + ) + ); + + Assert.Contains("outside of the destination", exception.Message); + } + } +} diff --git a/tests/SharpCompress.Test/Filters/BranchExecTests.cs b/tests/SharpCompress.Test/Filters/BranchExecTests.cs index 5c576dcc..99aad4c7 100644 --- a/tests/SharpCompress.Test/Filters/BranchExecTests.cs +++ b/tests/SharpCompress.Test/Filters/BranchExecTests.cs @@ -12,1177 +12,1165 @@ namespace SharpCompress.Test.Filters; public class BranchExecTests { - private static byte[] x86resultData { get; } = - new byte[] - { - 0x12, - 0x00, - 0x00, - 0x00, - 0x02, - 0x0B, - 0x00, - 0x00, - 0xE8, - 0xBD, - 0x00, - 0x00, - 0x00, - 0x07, - 0x00, - 0x00, - 0x12, - 0x00, - 0x00, - 0x00, - 0x6D, - 0x01, - 0x00, - 0x00, - 0xF0, - 0xCA, - 0x00, - 0x00, - 0x1C, - 0x00, - 0x00, - 0x00, - 0x12, - 0x00, - 0x00, - 0x00, - 0xBC, - 0x09, - 0x00, - 0x00, - 0x14, - 0xC2, - 0x00, - 0x00, - 0xE0, - 0x01, - 0x00, - 0x00, - 0x12, - 0x00, - 0x00, - 0x00, - 0x98, - 0x0B, - 0x00, - 0x00, - 0x60, - 0x75, - 0x0A, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x10, - 0x00, - 0xF1, - 0xFF, - 0x42, - 0x01, - 0x00, - 0x00, - 0x08, - 0xC8, - 0x00, - 0x00, - 0x1C, - 0x00, - }; + private static byte[] X86ResultData { get; } = + [ + 0x12, + 0x00, + 0x00, + 0x00, + 0x02, + 0x0B, + 0x00, + 0x00, + 0xE8, + 0xBD, + 0x00, + 0x00, + 0x00, + 0x07, + 0x00, + 0x00, + 0x12, + 0x00, + 0x00, + 0x00, + 0x6D, + 0x01, + 0x00, + 0x00, + 0xF0, + 0xCA, + 0x00, + 0x00, + 0x1C, + 0x00, + 0x00, + 0x00, + 0x12, + 0x00, + 0x00, + 0x00, + 0xBC, + 0x09, + 0x00, + 0x00, + 0x14, + 0xC2, + 0x00, + 0x00, + 0xE0, + 0x01, + 0x00, + 0x00, + 0x12, + 0x00, + 0x00, + 0x00, + 0x98, + 0x0B, + 0x00, + 0x00, + 0x60, + 0x75, + 0x0A, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x10, + 0x00, + 0xF1, + 0xFF, + 0x42, + 0x01, + 0x00, + 0x00, + 0x08, + 0xC8, + 0x00, + 0x00, + 0x1C, + 0x00, + ]; - private static byte[] x86Data { get; } = - new byte[] - { - 0x12, - 0x00, - 0x00, - 0x00, - 0x02, - 0x0B, - 0x00, - 0x00, - 0xE8, - 0xCA, - 0x20, - 0x00, - 0x00, - 0x07, - 0x00, - 0x00, - 0x12, - 0x00, - 0x00, - 0x00, - 0x6D, - 0x01, - 0x00, - 0x00, - 0xF0, - 0xCA, - 0x00, - 0x00, - 0x1C, - 0x00, - 0x00, - 0x00, - 0x12, - 0x00, - 0x00, - 0x00, - 0xBC, - 0x09, - 0x00, - 0x00, - 0x14, - 0xC2, - 0x00, - 0x00, - 0xE0, - 0x01, - 0x00, - 0x00, - 0x12, - 0x00, - 0x00, - 0x00, - 0x98, - 0x0B, - 0x00, - 0x00, - 0x60, - 0x75, - 0x0A, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x10, - 0x00, - 0xF1, - 0xFF, - 0x42, - 0x01, - 0x00, - 0x00, - 0x08, - 0xC8, - 0x00, - 0x00, - 0x1C, - 0x00, - }; + private static byte[] X86Data { get; } = + [ + 0x12, + 0x00, + 0x00, + 0x00, + 0x02, + 0x0B, + 0x00, + 0x00, + 0xE8, + 0xCA, + 0x20, + 0x00, + 0x00, + 0x07, + 0x00, + 0x00, + 0x12, + 0x00, + 0x00, + 0x00, + 0x6D, + 0x01, + 0x00, + 0x00, + 0xF0, + 0xCA, + 0x00, + 0x00, + 0x1C, + 0x00, + 0x00, + 0x00, + 0x12, + 0x00, + 0x00, + 0x00, + 0xBC, + 0x09, + 0x00, + 0x00, + 0x14, + 0xC2, + 0x00, + 0x00, + 0xE0, + 0x01, + 0x00, + 0x00, + 0x12, + 0x00, + 0x00, + 0x00, + 0x98, + 0x0B, + 0x00, + 0x00, + 0x60, + 0x75, + 0x0A, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x10, + 0x00, + 0xF1, + 0xFF, + 0x42, + 0x01, + 0x00, + 0x00, + 0x08, + 0xC8, + 0x00, + 0x00, + 0x1C, + 0x00, + ]; - private static byte[] ppcResultData { get; } = - new byte[] - { - 0xF8, - 0x6B, - 0x2E, - 0x8C, - 0x95, - 0xC5, - 0x4B, - 0x1B, - 0x94, - 0x78, - 0x9E, - 0x7C, - 0xBD, - 0x8B, - 0xA8, - 0xAF, - 0x31, - 0x20, - 0xFE, - 0x0F, - 0xB3, - 0x15, - 0x9A, - 0x7C, - 0xD5, - 0x5C, - 0xC2, - 0xC0, - 0xEC, - 0xE9, - 0x43, - 0x2B, - 0xD0, - 0x9F, - 0x2C, - 0xFC, - 0xB8, - 0x2B, - 0x6B, - 0x15, - 0xCD, - 0x3F, - 0x0C, - 0xAF, - 0x8F, - 0x68, - 0xB0, - 0x6E, - 0x6B, - 0x30, - 0x2E, - 0x8C, - 0x3F, - 0x7E, - 0x96, - 0x7C, - 0x93, - 0xB2, - 0xA4, - 0x0E, - 0x43, - 0xEA, - 0x20, - 0x10, - 0x38, - 0x6D, - 0x37, - 0xF8, - 0x87, - 0xFE, - 0xA9, - 0x63, - 0x75, - 0xF5, - 0x56, - 0x34, - 0x4A, - 0xE3, - 0xCF, - 0x89, - 0x18, - 0x08, - 0xC2, - 0x76, - 0x74, - 0x12, - 0xEC, - 0xA7, - 0x6D, - 0xC2, - 0xB7, - 0x1B, - 0x7A, - 0xB2, - 0xD4, - 0xED - }; + private static byte[] PpcResultData { get; } = + [ + 0xF8, + 0x6B, + 0x2E, + 0x8C, + 0x95, + 0xC5, + 0x4B, + 0x1B, + 0x94, + 0x78, + 0x9E, + 0x7C, + 0xBD, + 0x8B, + 0xA8, + 0xAF, + 0x31, + 0x20, + 0xFE, + 0x0F, + 0xB3, + 0x15, + 0x9A, + 0x7C, + 0xD5, + 0x5C, + 0xC2, + 0xC0, + 0xEC, + 0xE9, + 0x43, + 0x2B, + 0xD0, + 0x9F, + 0x2C, + 0xFC, + 0xB8, + 0x2B, + 0x6B, + 0x15, + 0xCD, + 0x3F, + 0x0C, + 0xAF, + 0x8F, + 0x68, + 0xB0, + 0x6E, + 0x6B, + 0x30, + 0x2E, + 0x8C, + 0x3F, + 0x7E, + 0x96, + 0x7C, + 0x93, + 0xB2, + 0xA4, + 0x0E, + 0x43, + 0xEA, + 0x20, + 0x10, + 0x38, + 0x6D, + 0x37, + 0xF8, + 0x87, + 0xFE, + 0xA9, + 0x63, + 0x75, + 0xF5, + 0x56, + 0x34, + 0x4A, + 0xE3, + 0xCF, + 0x89, + 0x18, + 0x08, + 0xC2, + 0x76, + 0x74, + 0x12, + 0xEC, + 0xA7, + 0x6D, + 0xC2, + 0xB7, + 0x1B, + 0x7A, + 0xB2, + 0xD4, + 0xED, + ]; - private static byte[] ppcData { get; } = - new byte[] - { - 0xF8, - 0x6B, - 0x2E, - 0x8C, - 0x95, - 0xC5, - 0x4B, - 0x1B, - 0x94, - 0x78, - 0x9E, - 0x7C, - 0xBD, - 0x8B, - 0xA8, - 0xAF, - 0x31, - 0x20, - 0xFE, - 0x0F, - 0xB3, - 0x15, - 0x9A, - 0x7C, - 0xD5, - 0x5C, - 0xC2, - 0xC0, - 0xEC, - 0xE9, - 0x43, - 0x2B, - 0xD0, - 0x9F, - 0x2C, - 0xFC, - 0xB8, - 0x2B, - 0x6B, - 0x15, - 0xCD, - 0x3F, - 0x0C, - 0xAF, - 0x8F, - 0x68, - 0xB0, - 0x6E, - 0x6B, - 0x30, - 0x2E, - 0x8C, - 0x3F, - 0x7E, - 0x96, - 0x7C, - 0x93, - 0xB2, - 0xA4, - 0x0E, - 0x43, - 0xEA, - 0x20, - 0x10, - 0x38, - 0x6D, - 0x37, - 0xF8, - 0x87, - 0xFE, - 0xA9, - 0x63, - 0x75, - 0xF5, - 0x56, - 0x34, - 0x4A, - 0xE3, - 0xD6, - 0x75, - 0x18, - 0x08, - 0xC2, - 0x76, - 0x74, - 0x12, - 0xEC, - 0xA7, - 0x6D, - 0xC2, - 0xB7, - 0x1B, - 0x7A, - 0xB2, - 0xD4, - 0xED - }; + private static byte[] PpcData { get; } = + [ + 0xF8, + 0x6B, + 0x2E, + 0x8C, + 0x95, + 0xC5, + 0x4B, + 0x1B, + 0x94, + 0x78, + 0x9E, + 0x7C, + 0xBD, + 0x8B, + 0xA8, + 0xAF, + 0x31, + 0x20, + 0xFE, + 0x0F, + 0xB3, + 0x15, + 0x9A, + 0x7C, + 0xD5, + 0x5C, + 0xC2, + 0xC0, + 0xEC, + 0xE9, + 0x43, + 0x2B, + 0xD0, + 0x9F, + 0x2C, + 0xFC, + 0xB8, + 0x2B, + 0x6B, + 0x15, + 0xCD, + 0x3F, + 0x0C, + 0xAF, + 0x8F, + 0x68, + 0xB0, + 0x6E, + 0x6B, + 0x30, + 0x2E, + 0x8C, + 0x3F, + 0x7E, + 0x96, + 0x7C, + 0x93, + 0xB2, + 0xA4, + 0x0E, + 0x43, + 0xEA, + 0x20, + 0x10, + 0x38, + 0x6D, + 0x37, + 0xF8, + 0x87, + 0xFE, + 0xA9, + 0x63, + 0x75, + 0xF5, + 0x56, + 0x34, + 0x4A, + 0xE3, + 0xD6, + 0x75, + 0x18, + 0x08, + 0xC2, + 0x76, + 0x74, + 0x12, + 0xEC, + 0xA7, + 0x6D, + 0xC2, + 0xB7, + 0x1B, + 0x7A, + 0xB2, + 0xD4, + 0xED, + ]; - private static byte[] armResultData { get; } = - new byte[] - { - 0x7C, - 0xFC, - 0x0A, - 0x00, - 0x16, - 0x42, - 0x01, - 0x00, - 0x80, - 0xFC, - 0x0A, - 0x00, - 0x16, - 0xB8, - 0x00, - 0x00, - 0x84, - 0xFC, - 0x0A, - 0x00, - 0x16, - 0x3A, - 0x00, - 0x00, - 0x04, - 0xE0, - 0x2D, - 0xE5, - 0x04, - 0xD0, - 0x4D, - 0xE2, - 0x0E, - 0x04, - 0x00, - 0xEB, - 0x04, - 0xD0, - 0x8D, - 0xE2, - 0x04, - 0xE0, - 0x9D, - 0xE4, - 0x1E, - 0xFF, - 0x2F, - 0xE1, - 0x04, - 0xE0, - 0x2D, - 0xE5, - 0x04, - 0xE0, - 0x9F, - 0xE5, - 0x0E, - 0xE0, - 0x8F, - 0xE0, - 0x08, - 0xF0, - 0xBE, - 0xE5, - 0xF0, - 0x3A, - 0x0A, - 0x00, - 0x00, - 0xC6, - 0x8F, - 0xE2, - 0xA3, - 0xCA, - 0x8C, - 0xE2, - 0xF0, - 0xFA, - 0xBC, - 0xE5, - 0x00, - 0xC6, - 0x8F, - 0xE2, - 0xA3, - 0xCA, - 0x8C, - 0xE2, - 0xE8, - 0xFA, - 0xBC, - 0xE5, - 0x00, - 0xC6, - 0x8F, - 0xE2 - }; + private static byte[] ArmResultData { get; } = + [ + 0x7C, + 0xFC, + 0x0A, + 0x00, + 0x16, + 0x42, + 0x01, + 0x00, + 0x80, + 0xFC, + 0x0A, + 0x00, + 0x16, + 0xB8, + 0x00, + 0x00, + 0x84, + 0xFC, + 0x0A, + 0x00, + 0x16, + 0x3A, + 0x00, + 0x00, + 0x04, + 0xE0, + 0x2D, + 0xE5, + 0x04, + 0xD0, + 0x4D, + 0xE2, + 0x0E, + 0x04, + 0x00, + 0xEB, + 0x04, + 0xD0, + 0x8D, + 0xE2, + 0x04, + 0xE0, + 0x9D, + 0xE4, + 0x1E, + 0xFF, + 0x2F, + 0xE1, + 0x04, + 0xE0, + 0x2D, + 0xE5, + 0x04, + 0xE0, + 0x9F, + 0xE5, + 0x0E, + 0xE0, + 0x8F, + 0xE0, + 0x08, + 0xF0, + 0xBE, + 0xE5, + 0xF0, + 0x3A, + 0x0A, + 0x00, + 0x00, + 0xC6, + 0x8F, + 0xE2, + 0xA3, + 0xCA, + 0x8C, + 0xE2, + 0xF0, + 0xFA, + 0xBC, + 0xE5, + 0x00, + 0xC6, + 0x8F, + 0xE2, + 0xA3, + 0xCA, + 0x8C, + 0xE2, + 0xE8, + 0xFA, + 0xBC, + 0xE5, + 0x00, + 0xC6, + 0x8F, + 0xE2, + ]; - private static byte[] armData { get; } = - new byte[] - { - 0x7C, - 0xFC, - 0x0A, - 0x00, - 0x16, - 0x42, - 0x01, - 0x00, - 0x80, - 0xFC, - 0x0A, - 0x00, - 0x16, - 0xB8, - 0x00, - 0x00, - 0x84, - 0xFC, - 0x0A, - 0x00, - 0x16, - 0x3A, - 0x00, - 0x00, - 0x04, - 0xE0, - 0x2D, - 0xE5, - 0x04, - 0xD0, - 0x4D, - 0xE2, - 0x18, - 0x13, - 0x00, - 0xEB, - 0x04, - 0xD0, - 0x8D, - 0xE2, - 0x04, - 0xE0, - 0x9D, - 0xE4, - 0x1E, - 0xFF, - 0x2F, - 0xE1, - 0x04, - 0xE0, - 0x2D, - 0xE5, - 0x04, - 0xE0, - 0x9F, - 0xE5, - 0x0E, - 0xE0, - 0x8F, - 0xE0, - 0x08, - 0xF0, - 0xBE, - 0xE5, - 0xF0, - 0x3A, - 0x0A, - 0x00, - 0x00, - 0xC6, - 0x8F, - 0xE2, - 0xA3, - 0xCA, - 0x8C, - 0xE2, - 0xF0, - 0xFA, - 0xBC, - 0xE5, - 0x00, - 0xC6, - 0x8F, - 0xE2, - 0xA3, - 0xCA, - 0x8C, - 0xE2, - 0xE8, - 0xFA, - 0xBC, - 0xE5, - 0x00, - 0xC6, - 0x8F, - 0xE2 - }; + private static byte[] ArmData { get; } = + [ + 0x7C, + 0xFC, + 0x0A, + 0x00, + 0x16, + 0x42, + 0x01, + 0x00, + 0x80, + 0xFC, + 0x0A, + 0x00, + 0x16, + 0xB8, + 0x00, + 0x00, + 0x84, + 0xFC, + 0x0A, + 0x00, + 0x16, + 0x3A, + 0x00, + 0x00, + 0x04, + 0xE0, + 0x2D, + 0xE5, + 0x04, + 0xD0, + 0x4D, + 0xE2, + 0x18, + 0x13, + 0x00, + 0xEB, + 0x04, + 0xD0, + 0x8D, + 0xE2, + 0x04, + 0xE0, + 0x9D, + 0xE4, + 0x1E, + 0xFF, + 0x2F, + 0xE1, + 0x04, + 0xE0, + 0x2D, + 0xE5, + 0x04, + 0xE0, + 0x9F, + 0xE5, + 0x0E, + 0xE0, + 0x8F, + 0xE0, + 0x08, + 0xF0, + 0xBE, + 0xE5, + 0xF0, + 0x3A, + 0x0A, + 0x00, + 0x00, + 0xC6, + 0x8F, + 0xE2, + 0xA3, + 0xCA, + 0x8C, + 0xE2, + 0xF0, + 0xFA, + 0xBC, + 0xE5, + 0x00, + 0xC6, + 0x8F, + 0xE2, + 0xA3, + 0xCA, + 0x8C, + 0xE2, + 0xE8, + 0xFA, + 0xBC, + 0xE5, + 0x00, + 0xC6, + 0x8F, + 0xE2, + ]; - private static byte[] armtResultData { get; } = - new byte[] - { - 0x95, - 0x23, - 0xB6, - 0xB1, - 0xBE, - 0x60, - 0x79, - 0xF0, - 0xF6, - 0x01, - 0xD9, - 0x7F, - 0x2E, - 0x03, - 0x31, - 0x1C, - 0xFD, - 0xD3, - 0x40, - 0x0F, - 0x21, - 0x3C, - 0x06, - 0x97, - 0xE5, - 0xC3, - 0x57, - 0x11, - 0x76, - 0x6F, - 0xE3, - 0x70, - 0xED, - 0x49, - 0xCB, - 0xB5, - 0xC9, - 0x42, - 0x59, - 0x10, - 0x2F, - 0xBD, - 0xAE, - 0xB1, - 0x40, - 0x4D, - 0x9D, - 0x7C, - 0xE9, - 0xFC, - 0x48, - 0x3E, - 0xBC, - 0x7F, - 0x0B, - 0x23, - 0xB0, - 0x8A, - 0x4D, - 0x02, - 0x39, - 0xC4, - 0xFB, - 0x66, - 0x83, - 0x7F, - 0xA7, - 0xBD, - 0x12, - 0xAC, - 0xED, - 0x31, - 0x34, - 0x93, - 0x4D, - 0x8D, - 0xD7, - 0x94, - 0x93, - 0x1C, - 0x0A, - 0x50, - 0x54, - 0x4B, - 0x03, - 0x55, - 0x27, - 0xFE, - 0xCE, - 0x29, - 0x66, - 0x52, - 0x81, - 0xAE, - 0x69, - 0xA0, - 0x69, - 0xF2, - 0x3D, - 0xFF, - 0xA1, - 0x8A, - 0x5D, - 0x61, - 0x7D, - 0xC5, - 0x94, - 0x0A, - 0x7D, - 0xED, - 0x11, - 0x0F - }; + private static byte[] ArmtResultData { get; } = + [ + 0x95, + 0x23, + 0xB6, + 0xB1, + 0xBE, + 0x60, + 0x79, + 0xF0, + 0xF6, + 0x01, + 0xD9, + 0x7F, + 0x2E, + 0x03, + 0x31, + 0x1C, + 0xFD, + 0xD3, + 0x40, + 0x0F, + 0x21, + 0x3C, + 0x06, + 0x97, + 0xE5, + 0xC3, + 0x57, + 0x11, + 0x76, + 0x6F, + 0xE3, + 0x70, + 0xED, + 0x49, + 0xCB, + 0xB5, + 0xC9, + 0x42, + 0x59, + 0x10, + 0x2F, + 0xBD, + 0xAE, + 0xB1, + 0x40, + 0x4D, + 0x9D, + 0x7C, + 0xE9, + 0xFC, + 0x48, + 0x3E, + 0xBC, + 0x7F, + 0x0B, + 0x23, + 0xB0, + 0x8A, + 0x4D, + 0x02, + 0x39, + 0xC4, + 0xFB, + 0x66, + 0x83, + 0x7F, + 0xA7, + 0xBD, + 0x12, + 0xAC, + 0xED, + 0x31, + 0x34, + 0x93, + 0x4D, + 0x8D, + 0xD7, + 0x94, + 0x93, + 0x1C, + 0x0A, + 0x50, + 0x54, + 0x4B, + 0x03, + 0x55, + 0x27, + 0xFE, + 0xCE, + 0x29, + 0x66, + 0x52, + 0x81, + 0xAE, + 0x69, + 0xA0, + 0x69, + 0xF2, + 0x3D, + 0xFF, + 0xA1, + 0x8A, + 0x5D, + 0x61, + 0x7D, + 0xC5, + 0x94, + 0x0A, + 0x7D, + 0xED, + 0x11, + 0x0F, + ]; - private static byte[] armtData { get; } = - new byte[] - { - 0x95, - 0x23, - 0xB6, - 0xB1, - 0xBE, - 0x60, - 0x79, - 0xF0, - 0xF6, - 0x01, - 0xD9, - 0x7F, - 0x2E, - 0x03, - 0x31, - 0x1C, - 0xFD, - 0xD3, - 0x40, - 0x0F, - 0x21, - 0x3C, - 0x06, - 0x97, - 0xE5, - 0xC3, - 0x57, - 0x11, - 0x76, - 0x6F, - 0xE3, - 0x70, - 0xED, - 0x49, - 0xCB, - 0xB5, - 0xC9, - 0x42, - 0x59, - 0x10, - 0x2F, - 0xBD, - 0xAE, - 0xB1, - 0x40, - 0x4D, - 0x9D, - 0x7C, - 0xE9, - 0xFC, - 0x48, - 0x3E, - 0xBC, - 0x7F, - 0x0B, - 0x23, - 0xB0, - 0x8A, - 0x4D, - 0x02, - 0x39, - 0xC4, - 0xFB, - 0x66, - 0x83, - 0x7F, - 0xA7, - 0xBD, - 0x12, - 0xAC, - 0xED, - 0x31, - 0x34, - 0x93, - 0x4D, - 0x8D, - 0xD7, - 0x94, - 0x93, - 0x1C, - 0x0A, - 0x50, - 0x54, - 0x4B, - 0x03, - 0x55, - 0x27, - 0xFE, - 0xCE, - 0x29, - 0x66, - 0x52, - 0x81, - 0xAE, - 0x69, - 0xA0, - 0x6A, - 0xF2, - 0x6F, - 0xFC, - 0xA1, - 0x8A, - 0x5D, - 0x61, - 0x7D, - 0xC5, - 0x94, - 0x0A, - 0x7D, - 0xED, - 0x11, - 0x0F - }; + private static byte[] ArmtData { get; } = + [ + 0x95, + 0x23, + 0xB6, + 0xB1, + 0xBE, + 0x60, + 0x79, + 0xF0, + 0xF6, + 0x01, + 0xD9, + 0x7F, + 0x2E, + 0x03, + 0x31, + 0x1C, + 0xFD, + 0xD3, + 0x40, + 0x0F, + 0x21, + 0x3C, + 0x06, + 0x97, + 0xE5, + 0xC3, + 0x57, + 0x11, + 0x76, + 0x6F, + 0xE3, + 0x70, + 0xED, + 0x49, + 0xCB, + 0xB5, + 0xC9, + 0x42, + 0x59, + 0x10, + 0x2F, + 0xBD, + 0xAE, + 0xB1, + 0x40, + 0x4D, + 0x9D, + 0x7C, + 0xE9, + 0xFC, + 0x48, + 0x3E, + 0xBC, + 0x7F, + 0x0B, + 0x23, + 0xB0, + 0x8A, + 0x4D, + 0x02, + 0x39, + 0xC4, + 0xFB, + 0x66, + 0x83, + 0x7F, + 0xA7, + 0xBD, + 0x12, + 0xAC, + 0xED, + 0x31, + 0x34, + 0x93, + 0x4D, + 0x8D, + 0xD7, + 0x94, + 0x93, + 0x1C, + 0x0A, + 0x50, + 0x54, + 0x4B, + 0x03, + 0x55, + 0x27, + 0xFE, + 0xCE, + 0x29, + 0x66, + 0x52, + 0x81, + 0xAE, + 0x69, + 0xA0, + 0x6A, + 0xF2, + 0x6F, + 0xFC, + 0xA1, + 0x8A, + 0x5D, + 0x61, + 0x7D, + 0xC5, + 0x94, + 0x0A, + 0x7D, + 0xED, + 0x11, + 0x0F, + ]; - private static byte[] ia64ResultData { get; } = - new byte[] - { - 0x4D, - 0xF8, - 0xF2, - 0x0D, - 0x06, - 0x2F, - 0x74, - 0x0F, - 0xF0, - 0x91, - 0x06, - 0x0B, - 0x19, - 0x22, - 0x91, - 0x5A, - 0x66, - 0x56, - 0xA7, - 0x15, - 0x77, - 0x1E, - 0x2F, - 0xA3, - 0xE4, - 0xDE, - 0x93, - 0x1C, - 0xD5, - 0xCE, - 0x6E, - 0x45, - 0x36, - 0x15, - 0x15, - 0x65, - 0x4E, - 0xC5, - 0xA3, - 0x8C, - 0x5A, - 0x8B, - 0x8A, - 0x1C, - 0x12, - 0x5B, - 0x39, - 0x1F, - 0xA0, - 0xF2, - 0x93, - 0x7C, - 0x7F, - 0x5D, - 0xD9, - 0x30, - 0x1F, - 0xF6, - 0x5C, - 0x10, - 0x62, - 0x3E, - 0xB4, - 0x64, - 0x56, - 0x48, - 0xB2, - 0x20, - 0x39, - 0xE8, - 0x44, - 0x10, - 0x87, - 0x9E, - 0x2C, - 0xFC, - 0x29, - 0x0E, - 0x20, - 0x76, - 0xCE, - 0xDA, - 0x93, - 0x1C, - 0xED, - 0x54, - 0x0D, - 0xAF, - 0xEC, - 0xDE, - 0x93, - 0x1C, - 0x2B, - 0x72, - 0xD5, - 0x0D - }; + private static byte[] Ia64ResultData { get; } = + [ + 0x4D, + 0xF8, + 0xF2, + 0x0D, + 0x06, + 0x2F, + 0x74, + 0x0F, + 0xF0, + 0x91, + 0x06, + 0x0B, + 0x19, + 0x22, + 0x91, + 0x5A, + 0x66, + 0x56, + 0xA7, + 0x15, + 0x77, + 0x1E, + 0x2F, + 0xA3, + 0xE4, + 0xDE, + 0x93, + 0x1C, + 0xD5, + 0xCE, + 0x6E, + 0x45, + 0x36, + 0x15, + 0x15, + 0x65, + 0x4E, + 0xC5, + 0xA3, + 0x8C, + 0x5A, + 0x8B, + 0x8A, + 0x1C, + 0x12, + 0x5B, + 0x39, + 0x1F, + 0xA0, + 0xF2, + 0x93, + 0x7C, + 0x7F, + 0x5D, + 0xD9, + 0x30, + 0x1F, + 0xF6, + 0x5C, + 0x10, + 0x62, + 0x3E, + 0xB4, + 0x64, + 0x56, + 0x48, + 0xB2, + 0x20, + 0x39, + 0xE8, + 0x44, + 0x10, + 0x87, + 0x9E, + 0x2C, + 0xFC, + 0x29, + 0x0E, + 0x20, + 0x76, + 0xCE, + 0xDA, + 0x93, + 0x1C, + 0xED, + 0x54, + 0x0D, + 0xAF, + 0xEC, + 0xDE, + 0x93, + 0x1C, + 0x2B, + 0x72, + 0xD5, + 0x0D, + ]; - private static byte[] ia64Data { get; } = - new byte[] - { - 0x4D, - 0xF8, - 0xF2, - 0x0D, - 0x06, - 0x2F, - 0x74, - 0x0F, - 0xF0, - 0x91, - 0x06, - 0x0B, - 0x19, - 0x22, - 0x91, - 0x5A, - 0x66, - 0x56, - 0xA7, - 0x15, - 0x77, - 0x1E, - 0x2F, - 0xA3, - 0xE4, - 0xDE, - 0x93, - 0x1C, - 0xD5, - 0xCE, - 0x6E, - 0x45, - 0x36, - 0x15, - 0x15, - 0x65, - 0x4E, - 0xC5, - 0xA3, - 0x8C, - 0x5A, - 0x8B, - 0x8A, - 0x1C, - 0x12, - 0x5B, - 0x39, - 0x1F, - 0xA0, - 0xF2, - 0x93, - 0x7C, - 0x7F, - 0x5D, - 0xD9, - 0x30, - 0x1F, - 0xF6, - 0x5C, - 0x10, - 0x62, - 0x3E, - 0xB4, - 0x64, - 0x56, - 0x48, - 0xB2, - 0x20, - 0x39, - 0xE8, - 0x44, - 0x80, - 0x8C, - 0x9E, - 0x2C, - 0xFC, - 0x29, - 0x0E, - 0x20, - 0x76, - 0xCE, - 0xDA, - 0x93, - 0x1C, - 0xED, - 0x54, - 0x0D, - 0xAF, - 0xEC, - 0xDE, - 0x93, - 0x1C, - 0x2B, - 0x72, - 0xD5, - 0x0D - }; + private static byte[] Ia64Data { get; } = + [ + 0x4D, + 0xF8, + 0xF2, + 0x0D, + 0x06, + 0x2F, + 0x74, + 0x0F, + 0xF0, + 0x91, + 0x06, + 0x0B, + 0x19, + 0x22, + 0x91, + 0x5A, + 0x66, + 0x56, + 0xA7, + 0x15, + 0x77, + 0x1E, + 0x2F, + 0xA3, + 0xE4, + 0xDE, + 0x93, + 0x1C, + 0xD5, + 0xCE, + 0x6E, + 0x45, + 0x36, + 0x15, + 0x15, + 0x65, + 0x4E, + 0xC5, + 0xA3, + 0x8C, + 0x5A, + 0x8B, + 0x8A, + 0x1C, + 0x12, + 0x5B, + 0x39, + 0x1F, + 0xA0, + 0xF2, + 0x93, + 0x7C, + 0x7F, + 0x5D, + 0xD9, + 0x30, + 0x1F, + 0xF6, + 0x5C, + 0x10, + 0x62, + 0x3E, + 0xB4, + 0x64, + 0x56, + 0x48, + 0xB2, + 0x20, + 0x39, + 0xE8, + 0x44, + 0x80, + 0x8C, + 0x9E, + 0x2C, + 0xFC, + 0x29, + 0x0E, + 0x20, + 0x76, + 0xCE, + 0xDA, + 0x93, + 0x1C, + 0xED, + 0x54, + 0x0D, + 0xAF, + 0xEC, + 0xDE, + 0x93, + 0x1C, + 0x2B, + 0x72, + 0xD5, + 0x0D, + ]; - private static byte[] sparcResultData { get; } = - new byte[] - { - 0x78, - 0x2E, - 0x73, - 0x6F, - 0x2E, - 0x33, - 0x00, - 0x00, - 0x07, - 0x01, - 0x00, - 0x00, - 0x09, - 0x00, - 0x00, - 0x00, - 0x40, - 0x00, - 0x00, - 0x00, - 0x0B, - 0x00, - 0x00, - 0x00, - 0xA2, - 0x0D, - 0x10, - 0x12, - 0x30, - 0x03, - 0xC9, - 0x37, - 0x40, - 0x1A, - 0x85, - 0xD9, - 0x44, - 0x34, - 0x40, - 0x32, - 0x85, - 0xA0, - 0x30, - 0x40, - 0x00, - 0x70, - 0x00, - 0x40, - 0x84, - 0x80, - 0x04, - 0x00, - 0xE4, - 0xAC, - 0x07, - 0x04, - 0x21, - 0x44, - 0x02, - 0x20, - 0x10, - 0x00, - 0x40, - 0xC2, - 0x89, - 0x98, - 0x85, - 0x00, - 0x58, - 0x6A, - 0x41, - 0x8E, - 0x18, - 0xA1, - 0x91, - 0x00, - 0x10, - 0x00, - }; + private static byte[] SparcResultData { get; } = + [ + 0x78, + 0x2E, + 0x73, + 0x6F, + 0x2E, + 0x33, + 0x00, + 0x00, + 0x07, + 0x01, + 0x00, + 0x00, + 0x09, + 0x00, + 0x00, + 0x00, + 0x40, + 0x00, + 0x00, + 0x00, + 0x0B, + 0x00, + 0x00, + 0x00, + 0xA2, + 0x0D, + 0x10, + 0x12, + 0x30, + 0x03, + 0xC9, + 0x37, + 0x40, + 0x1A, + 0x85, + 0xD9, + 0x44, + 0x34, + 0x40, + 0x32, + 0x85, + 0xA0, + 0x30, + 0x40, + 0x00, + 0x70, + 0x00, + 0x40, + 0x84, + 0x80, + 0x04, + 0x00, + 0xE4, + 0xAC, + 0x07, + 0x04, + 0x21, + 0x44, + 0x02, + 0x20, + 0x10, + 0x00, + 0x40, + 0xC2, + 0x89, + 0x98, + 0x85, + 0x00, + 0x58, + 0x6A, + 0x41, + 0x8E, + 0x18, + 0xA1, + 0x91, + 0x00, + 0x10, + 0x00, + ]; - private static byte[] sparcData { get; } = - new byte[] - { - 0x78, - 0x2E, - 0x73, - 0x6F, - 0x2E, - 0x33, - 0x00, - 0x00, - 0x07, - 0x01, - 0x00, - 0x00, - 0x09, - 0x00, - 0x00, - 0x00, - 0x40, - 0x00, - 0x00, - 0x44, - 0x0B, - 0x00, - 0x00, - 0x00, - 0xA2, - 0x0D, - 0x10, - 0x12, - 0x30, - 0x03, - 0xC9, - 0x37, - 0x40, - 0x1A, - 0x86, - 0x21, - 0x44, - 0x34, - 0x40, - 0x32, - 0x85, - 0xA0, - 0x30, - 0x40, - 0x00, - 0x70, - 0x00, - 0x40, - 0x84, - 0x80, - 0x04, - 0x00, - 0xE4, - 0xAC, - 0x07, - 0x04, - 0x21, - 0x44, - 0x02, - 0x20, - 0x10, - 0x00, - 0x40, - 0xC2, - 0x89, - 0x98, - 0x85, - 0x00, - 0x58, - 0x6A, - 0x41, - 0x8E, - 0x18, - 0xA1, - 0x91, - 0x00, - 0x10, - 0x00, - }; + private static byte[] SparcData { get; } = + [ + 0x78, + 0x2E, + 0x73, + 0x6F, + 0x2E, + 0x33, + 0x00, + 0x00, + 0x07, + 0x01, + 0x00, + 0x00, + 0x09, + 0x00, + 0x00, + 0x00, + 0x40, + 0x00, + 0x00, + 0x44, + 0x0B, + 0x00, + 0x00, + 0x00, + 0xA2, + 0x0D, + 0x10, + 0x12, + 0x30, + 0x03, + 0xC9, + 0x37, + 0x40, + 0x1A, + 0x86, + 0x21, + 0x44, + 0x34, + 0x40, + 0x32, + 0x85, + 0xA0, + 0x30, + 0x40, + 0x00, + 0x70, + 0x00, + 0x40, + 0x84, + 0x80, + 0x04, + 0x00, + 0xE4, + 0xAC, + 0x07, + 0x04, + 0x21, + 0x44, + 0x02, + 0x20, + 0x10, + 0x00, + 0x40, + 0xC2, + 0x89, + 0x98, + 0x85, + 0x00, + 0x58, + 0x6A, + 0x41, + 0x8E, + 0x18, + 0xA1, + 0x91, + 0x00, + 0x10, + 0x00, + ]; private void CompareBuffer(byte[] testBuffer, byte[] targetBuffer) => Assert.Equal(testBuffer, targetBuffer); @@ -1192,53 +1180,53 @@ public class BranchExecTests { uint state = 0; uint ip = 0x2000; - var testData = x86Data; + var testData = X86Data; BranchExecFilter.X86Converter(testData, ip, ref state); - CompareBuffer(testData, x86resultData); + CompareBuffer(testData, X86ResultData); } [Fact] - public void PowerPCConverterDecodeTest() + public void PowerPcConverterDecodeTest() { uint ip = 0x6A0; - var testData = ppcData; + var testData = PpcData; BranchExecFilter.PowerPCConverter(testData, ip); - CompareBuffer(testData, ppcResultData); + CompareBuffer(testData, PpcResultData); } [Fact] - public void ARMConverteDecoderTest() + public void ArmConverteDecoderTest() { uint ip = 0x3C00; - var testData = armData; + var testData = ArmData; BranchExecFilter.ARMConverter(testData, ip); - CompareBuffer(testData, armResultData); + CompareBuffer(testData, ArmResultData); } [Fact] - public void ARMTConverterDecodeTest() + public void ArmtConverterDecodeTest() { uint ip = 0xA00; - var testData = armtData; + var testData = ArmtData; BranchExecFilter.ARMTConverter(testData, ip); - CompareBuffer(testData, armtResultData); + CompareBuffer(testData, ArmtResultData); } [Fact] - public void IA64ConverterDecodeTest() + public void Ia64ConverterDecodeTest() { uint ip = 0xAA0; - var testData = ia64Data; + var testData = Ia64Data; BranchExecFilter.IA64Converter(testData, ip); - CompareBuffer(testData, ia64ResultData); + CompareBuffer(testData, Ia64ResultData); } [Fact] - public void SPARCConverterDecodeTest() + public void SparcConverterDecodeTest() { uint ip = 0x100; - var testData = sparcData; + var testData = SparcData; BranchExecFilter.SPARCConverter(testData, ip); - CompareBuffer(testData, sparcResultData); + CompareBuffer(testData, SparcResultData); } } diff --git a/tests/SharpCompress.Test/GZip/AsyncTests.cs b/tests/SharpCompress.Test/GZip/AsyncTests.cs new file mode 100644 index 00000000..9a0a8590 --- /dev/null +++ b/tests/SharpCompress.Test/GZip/AsyncTests.cs @@ -0,0 +1,272 @@ +using System; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Archives; +using SharpCompress.Archives.GZip; +using SharpCompress.Archives.Zip; +using SharpCompress.Common; +using SharpCompress.Compressors; +using SharpCompress.Compressors.Deflate; +using SharpCompress.Readers; +using SharpCompress.Test.Mocks; +using SharpCompress.Writers; +using Xunit; + +namespace SharpCompress.Test.GZip; + +public class AsyncTests : TestBase +{ + [Fact] + public async ValueTask Reader_Async_Extract_All() + { + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz"); +#if NETFRAMEWORK + using var stream = File.OpenRead(testArchive); +#else + await using var stream = File.OpenRead(testArchive); +#endif + await using var reader = await ReaderFactory.OpenAsyncReader(new AsyncOnlyStream(stream)); + + await reader.WriteAllToDirectoryAsync(SCRATCH_FILES_PATH); + + // Just verify some files were extracted + var extractedFiles = Directory.GetFiles( + SCRATCH_FILES_PATH, + "*", + SearchOption.AllDirectories + ); + Assert.True(extractedFiles.Length > 0, "No files were extracted"); + } + + [Fact] + public async ValueTask Reader_Async_Extract_Single_Entry() + { + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz"); +#if NETFRAMEWORK + using var stream = File.OpenRead(testArchive); +#else + await using var stream = File.OpenRead(testArchive); +#endif + await using var reader = await ReaderFactory.OpenAsyncReader(new AsyncOnlyStream(stream)); + + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + var outputPath = Path.Combine(SCRATCH_FILES_PATH, reader.Entry.Key!); + Directory.CreateDirectory(Path.GetDirectoryName(outputPath)!); +#if NETFRAMEWORK + using var outputStream = File.Create(outputPath); +#else + await using var outputStream = File.Create(outputPath); +#endif + await reader.WriteEntryToAsync(outputStream); + break; // Just test one entry + } + } + } + + [Fact] + public async ValueTask Archive_Entry_Async_Open_Stream() + { + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz"); + await using var archive = await GZipArchive.OpenAsyncArchive( + new AsyncOnlyStream(File.OpenRead(testArchive)) + ); + + await foreach (var entry in archive.EntriesAsync.Where(e => !e.IsDirectory).Take(1)) + { +#if NETFRAMEWORK + using var entryStream = await entry.OpenEntryStreamAsync(); +#else + await using var entryStream = await entry.OpenEntryStreamAsync(); +#endif + Assert.NotNull(entryStream); + Assert.True(entryStream.CanRead); + + // Read some data to verify it works + var buffer = new byte[1024]; + var read = await entryStream.ReadAsync(buffer, 0, buffer.Length); + Assert.True(read > 0); + } + } + + [Fact] + public async ValueTask Writer_Async_Write_Single_File() + { + var outputPath = Path.Combine(SCRATCH_FILES_PATH, "async_test.zip"); + +#if NETFRAMEWORK + using (var stream = File.Create(outputPath)) +#else + await using (var stream = File.Create(outputPath)) +#endif + await using ( + var writer = await WriterFactory.OpenAsyncWriter( + new AsyncOnlyStream(stream), + ArchiveType.Zip, + new WriterOptions(CompressionType.Deflate) { LeaveStreamOpen = false } + ) + ) + { + var testFile = Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz"); + +#if NETFRAMEWORK + using var fileStream = File.OpenRead(testFile); +#else + await using var fileStream = File.OpenRead(testFile); +#endif + await writer.WriteAsync("test_entry.bin", fileStream, new DateTime(2023, 1, 1)); + } + + // Verify the archive was created and contains the entry + Assert.True(File.Exists(outputPath)); + await using var archive = await ZipArchive.OpenAsyncArchive(outputPath); + Assert.Single(await archive.EntriesAsync.Where(e => !e.IsDirectory).ToListAsync()); + } + + [Fact] + public async ValueTask Async_With_Cancellation_Token() + { + using var cts = new CancellationTokenSource(); + cts.CancelAfter(10000); // 10 seconds should be plenty + + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz"); +#if NETFRAMEWORK + using var stream = File.OpenRead(testArchive); +#else + await using var stream = File.OpenRead(testArchive); +#endif + await using var reader = await ReaderFactory.OpenAsyncReader( + new AsyncOnlyStream(stream), + cancellationToken: cts.Token + ); + + await reader.WriteAllToDirectoryAsync(SCRATCH_FILES_PATH, cancellationToken: cts.Token); + + // Just verify some files were extracted + var extractedFiles = Directory.GetFiles( + SCRATCH_FILES_PATH, + "*", + SearchOption.AllDirectories + ); + Assert.True(extractedFiles.Length > 0, "No files were extracted"); + } + + [Fact] + public async ValueTask Stream_Extensions_Async() + { + var testFile = Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz"); + var outputPath = Path.Combine(SCRATCH_FILES_PATH, "async_copy.bin"); +#if NETFRAMEWORK + using var inputStream = File.OpenRead(testFile); + using var outputStream = File.Create(outputPath); +#else + await using var inputStream = File.OpenRead(testFile); + await using var outputStream = File.Create(outputPath); +#endif + + // Test the async extension method + var buffer = new byte[8192]; + int bytesRead; + while ((bytesRead = await inputStream.ReadAsync(buffer, 0, buffer.Length)) > 0) + { + await outputStream.WriteAsync(buffer, 0, bytesRead); + } + + Assert.True(File.Exists(outputPath)); + Assert.True(new FileInfo(outputPath).Length > 0); + } + + [Fact] + public async ValueTask EntryStream_ReadAsync_Works() + { + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz"); +#if NETFRAMEWORK + using var stream = File.OpenRead(testArchive); +#else + await using var stream = File.OpenRead(testArchive); +#endif + await using var reader = await ReaderFactory.OpenAsyncReader(new AsyncOnlyStream(stream)); + + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { +#if NETFRAMEWORK + using var entryStream = await reader.OpenEntryStreamAsync(); +#else + await using var entryStream = await reader.OpenEntryStreamAsync(); +#endif + var buffer = new byte[4096]; + var totalRead = 0; + int bytesRead; + + // Test ReadAsync on EntryStream + while ((bytesRead = await entryStream.ReadAsync(buffer, 0, buffer.Length)) > 0) + { + totalRead += bytesRead; + } + + Assert.True(totalRead > 0, "Should have read some data from entry stream"); + break; // Test just one entry + } + } + } + + [Fact] + public async ValueTask CompressionStream_Async_ReadWrite() + { + var testData = new byte[1024]; + new Random(42).NextBytes(testData); + + var compressedPath = Path.Combine(SCRATCH_FILES_PATH, "async_compressed.gz"); + + // Test async write with GZipStream +#if NETFRAMEWORK + using (var fileStream = File.Create(compressedPath)) + using (var gzipStream = new GZipStream(fileStream, CompressionMode.Compress)) +#else + await using (var fileStream = File.Create(compressedPath)) + await using (var gzipStream = new GZipStream(fileStream, CompressionMode.Compress)) +#endif + { + await gzipStream.WriteAsync(testData, 0, testData.Length); + await gzipStream.FlushAsync(); + } + + Assert.True(File.Exists(compressedPath)); + Assert.True(new FileInfo(compressedPath).Length > 0); +#if NETFRAMEWORK + using (var fileStream = File.OpenRead(compressedPath)) + using (var gzipStream = new GZipStream(fileStream, CompressionMode.Decompress)) +#else + // Test async read with GZipStream + await using (var fileStream = File.OpenRead(compressedPath)) + await using (var gzipStream = new GZipStream(fileStream, CompressionMode.Decompress)) +#endif + { + var decompressed = new byte[testData.Length]; + var totalRead = 0; + int bytesRead; + while ( + totalRead < decompressed.Length + && ( + bytesRead = await gzipStream.ReadAsync( + decompressed, + totalRead, + decompressed.Length - totalRead + ) + ) > 0 + ) + { + totalRead += bytesRead; + } + + Assert.Equal(testData.Length, totalRead); + Assert.Equal(testData, decompressed); + } + } +} diff --git a/tests/SharpCompress.Test/GZip/GZipArchiveAsyncTests.cs b/tests/SharpCompress.Test/GZip/GZipArchiveAsyncTests.cs new file mode 100644 index 00000000..d7812b57 --- /dev/null +++ b/tests/SharpCompress.Test/GZip/GZipArchiveAsyncTests.cs @@ -0,0 +1,243 @@ +using System; +using System.IO; +using System.Linq; +using System.Threading.Tasks; +using SharpCompress.Archives; +using SharpCompress.Archives.GZip; +using SharpCompress.Archives.Tar; +using SharpCompress.Common; +using SharpCompress.Compressors.Deflate; +using SharpCompress.Readers; +using SharpCompress.Test.Mocks; +using SharpCompress.Writers.GZip; +using Xunit; + +namespace SharpCompress.Test.GZip; + +public class GZipArchiveAsyncTests : ArchiveTests +{ + public GZipArchiveAsyncTests() => UseExtensionInsteadOfNameToVerify = true; + + [Fact] + public async ValueTask GZip_Archive_Generic_Async() + { +#if NETFRAMEWORK + using (Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz"))) +#else + await using (Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz"))) +#endif + await using (var archive = await GZipArchive.OpenAsyncArchive(new AsyncOnlyStream(stream))) + { + var entry = await archive.EntriesAsync.FirstAsync(); + await entry.WriteToFileAsync(Path.Combine(SCRATCH_FILES_PATH, entry.Key.NotNull())); + + var size = entry.Size; + var scratch = new FileInfo(Path.Combine(SCRATCH_FILES_PATH, "Tar.tar")); + var test = new FileInfo(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar")); + + Assert.Equal(size, scratch.Length); + Assert.Equal(size, test.Length); + } + CompareArchivesByPath( + Path.Combine(SCRATCH_FILES_PATH, "Tar.tar"), + Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar") + ); + } + + [Fact] + public async ValueTask GZip_Archive_Async() + { +#if NETFRAMEWORK + using (Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz"))) +#else + await using (Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz"))) +#endif + { + await using ( + var archive = await GZipArchive.OpenAsyncArchive(new AsyncOnlyStream(stream)) + ) + { + var entry = await archive.EntriesAsync.FirstAsync(); + await entry.WriteToFileAsync(Path.Combine(SCRATCH_FILES_PATH, entry.Key.NotNull())); + + var size = entry.Size; + var scratch = new FileInfo(Path.Combine(SCRATCH_FILES_PATH, "Tar.tar")); + var test = new FileInfo(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar")); + + Assert.Equal(size, scratch.Length); + Assert.Equal(size, test.Length); + } + } + CompareArchivesByPath( + Path.Combine(SCRATCH_FILES_PATH, "Tar.tar"), + Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar") + ); + } + + [Fact] + public async ValueTask GZip_Archive_NoAdd_Async() + { + var jpg = Path.Combine(ORIGINAL_FILES_PATH, "jpg", "test.jpg"); +#if NETFRAMEWORK + using Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz")); +#else + await using Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz")); +#endif + await using (var archive = await GZipArchive.OpenAsyncArchive(new AsyncOnlyStream(stream))) + { + await Assert.ThrowsAsync(async () => + await archive.AddEntryAsync("jpg\\test.jpg", File.OpenRead(jpg), closeStream: true) + ); + await archive.SaveToAsync( + Path.Combine(SCRATCH_FILES_PATH, "Tar.tar.gz"), + new GZipWriterOptions() + ); + } + } + + [Fact] + public async ValueTask GZip_Archive_Multiple_Reads_Async() + { + var inputStream = new MemoryStream(); +#if NETFRAMEWORK + using (Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz"))) +#else + await using (Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz"))) +#endif + { + await stream.CopyToAsync(inputStream); + inputStream.Position = 0; + } + + await using var archive = await GZipArchive.OpenAsyncArchive( + new AsyncOnlyStream(inputStream) + ); + var archiveEntry = await archive.EntriesAsync.FirstAsync(); + + MemoryStream tarStream; +#if NETFRAMEWORK + using (var entryStream = await archiveEntry.OpenEntryStreamAsync()) +#else + await using (var entryStream = await archiveEntry.OpenEntryStreamAsync()) +#endif + { + tarStream = new MemoryStream(); + await entryStream.CopyToAsync(tarStream); + } + var size = tarStream.Length; +#if NETFRAMEWORK + using (var entryStream = await archiveEntry.OpenEntryStreamAsync()) +#else + await using (var entryStream = await archiveEntry.OpenEntryStreamAsync()) +#endif + { + tarStream = new MemoryStream(); + await entryStream.CopyToAsync(tarStream); + } + Assert.Equal(size, tarStream.Length); +#if NETFRAMEWORK + using (var entryStream = await archiveEntry.OpenEntryStreamAsync()) +#else + await using (var entryStream = await archiveEntry.OpenEntryStreamAsync()) +#endif + { + var result = await TarArchive.IsTarFileAsync(entryStream); + Assert.True(result); + } + Assert.Equal(size, tarStream.Length); +#if NETFRAMEWORK + using (var entryStream = await archiveEntry.OpenEntryStreamAsync()) +#else + await using (var entryStream = await archiveEntry.OpenEntryStreamAsync()) +#endif + { + tarStream = new MemoryStream(); + await entryStream.CopyToAsync(tarStream); + } + Assert.Equal(size, tarStream.Length); + } + + [Fact] + public async Task TestGzCrcWithMostSignificantBitNotNegative_Async() + { +#if NETFRAMEWORK + using var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz")); +#else + await using var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz")); +#endif + await using var archive = await GZipArchive.OpenAsyncArchive(new AsyncOnlyStream(stream)); + await foreach (var entry in archive.EntriesAsync.Where(entry => !entry.IsDirectory)) + { + Assert.InRange(entry.Crc, 0L, 0xFFFFFFFFL); + } + } + + [Fact] + public async Task TestGzArchiveTypeGzip_Async() + { +#if NETFRAMEWORK + using var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz")); +#else + await using var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz")); +#endif + await using var archive = await GZipArchive.OpenAsyncArchive(new AsyncOnlyStream(stream)); + Assert.Equal(archive.Type, ArchiveType.GZip); + } + + [Fact] + public async ValueTask GZip_Create_New_Async() + { + var scratchPath = Path.Combine(SCRATCH_FILES_PATH, Guid.NewGuid().ToString()); + var filePath = Path.Combine(scratchPath, "test.gz"); + if (!Directory.Exists(scratchPath)) + { + Directory.CreateDirectory(scratchPath); + } + await using (var archive = (GZipArchive)await GZipArchive.CreateAsyncArchive()) + { + await archive.AddEntryAsync("Tar.tar", Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar")); + await archive.SaveToAsync(filePath, new GZipWriterOptions(CompressionLevel.BestSpeed)); + } + var scratchPath2 = Path.Combine(SCRATCH_FILES_PATH, Guid.NewGuid().ToString()); + if (!Directory.Exists(scratchPath2)) + { + Directory.CreateDirectory(scratchPath2); + } + +#if NETFRAMEWORK + using var fileStream = File.OpenRead(filePath); +#else + await using var fileStream = File.OpenRead(filePath); +#endif + await using var archive2 = await GZipArchive.OpenAsyncArchive( + new AsyncOnlyStream(fileStream) + ); + await foreach (var entry in archive2.EntriesAsync) + { + await entry.WriteToDirectoryAsync(scratchPath2); + } + CompareFilesByPath( + Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar"), + Path.Combine(scratchPath2, "Tar.tar") + ); + } + + [Fact] + public async ValueTask GZip_Async_Dispose_Closes_New_Entry_Stream() + { + var entryStream = new TestStream(new MemoryStream(new byte[] { 1, 2, 3 })); + + await using (var archive = await GZipArchive.CreateAsyncArchive()) + { + await archive.AddEntryAsync( + "test.bin", + entryStream, + closeStream: true, + size: entryStream.Length + ); + await archive.SaveToAsync(new MemoryStream(), new GZipWriterOptions()); + } + + Assert.True(entryStream.IsDisposed); + } +} diff --git a/tests/SharpCompress.Test/GZip/GZipArchiveDirectoryTests.cs b/tests/SharpCompress.Test/GZip/GZipArchiveDirectoryTests.cs new file mode 100644 index 00000000..d1d7fbf1 --- /dev/null +++ b/tests/SharpCompress.Test/GZip/GZipArchiveDirectoryTests.cs @@ -0,0 +1,19 @@ +using System; +using System.IO; +using SharpCompress.Archives.GZip; +using Xunit; + +namespace SharpCompress.Test.GZip; + +public class GZipArchiveDirectoryTests : TestBase +{ + [Fact] + public void GZipArchive_AddDirectoryEntry_ThrowsNotSupportedException() + { + using var archive = GZipArchive.CreateArchive(); + + Assert.Throws(() => + archive.AddDirectoryEntry("test-dir", DateTime.Now) + ); + } +} diff --git a/tests/SharpCompress.Test/GZip/GZipArchiveTests.cs b/tests/SharpCompress.Test/GZip/GZipArchiveTests.cs index e47c339c..5f9f5094 100644 --- a/tests/SharpCompress.Test/GZip/GZipArchiveTests.cs +++ b/tests/SharpCompress.Test/GZip/GZipArchiveTests.cs @@ -3,6 +3,10 @@ using System.IO; using System.Linq; using SharpCompress.Archives; using SharpCompress.Archives.GZip; +using SharpCompress.Archives.Tar; +using SharpCompress.Common; +using SharpCompress.Test.Mocks; +using SharpCompress.Writers.GZip; using Xunit; namespace SharpCompress.Test.GZip; @@ -15,10 +19,10 @@ public class GZipArchiveTests : ArchiveTests public void GZip_Archive_Generic() { using (Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz"))) - using (var archive = ArchiveFactory.Open(stream)) + using (var archive = GZipArchive.OpenArchive(stream)) { var entry = archive.Entries.First(); - entry.WriteToFile(Path.Combine(SCRATCH_FILES_PATH, entry.Key)); + entry.WriteToFile(Path.Combine(SCRATCH_FILES_PATH, entry.Key.NotNull())); var size = entry.Size; var scratch = new FileInfo(Path.Combine(SCRATCH_FILES_PATH, "Tar.tar")); @@ -37,10 +41,10 @@ public class GZipArchiveTests : ArchiveTests public void GZip_Archive() { using (Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz"))) - using (var archive = GZipArchive.Open(stream)) + using (var archive = GZipArchive.OpenArchive(stream)) { var entry = archive.Entries.First(); - entry.WriteToFile(Path.Combine(SCRATCH_FILES_PATH, entry.Key)); + entry.WriteToFile(Path.Combine(SCRATCH_FILES_PATH, entry.Key.NotNull())); var size = entry.Size; var scratch = new FileInfo(Path.Combine(SCRATCH_FILES_PATH, "Tar.tar")); @@ -60,9 +64,9 @@ public class GZipArchiveTests : ArchiveTests { var jpg = Path.Combine(ORIGINAL_FILES_PATH, "jpg", "test.jpg"); using Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz")); - using var archive = GZipArchive.Open(stream); - Assert.Throws(() => archive.AddEntry("jpg\\test.jpg", jpg)); - archive.SaveTo(Path.Combine(SCRATCH_FILES_PATH, "Tar.tar.gz")); + using var archive = GZipArchive.OpenArchive(stream); + Assert.Throws(() => archive.AddEntry("jpg\\test.jpg", jpg)); + archive.SaveTo(Path.Combine(SCRATCH_FILES_PATH, "Tar.tar.gz"), new GZipWriterOptions()); } [Fact] @@ -74,7 +78,7 @@ public class GZipArchiveTests : ArchiveTests fileStream.CopyTo(inputStream); inputStream.Position = 0; } - using var archive = GZipArchive.Open(inputStream); + using var archive = GZipArchive.OpenArchive(inputStream); var archiveEntry = archive.Entries.First(); MemoryStream tarStream; @@ -92,7 +96,8 @@ public class GZipArchiveTests : ArchiveTests Assert.Equal(size, tarStream.Length); using (var entryStream = archiveEntry.OpenEntryStream()) { - var result = Archives.Tar.TarArchive.IsTarFile(entryStream); + var result = TarArchive.IsTarFile(entryStream); + Assert.True(result); } Assert.Equal(size, tarStream.Length); using (var entryStream = archiveEntry.OpenEntryStream()) @@ -104,14 +109,95 @@ public class GZipArchiveTests : ArchiveTests } [Fact] - public void TestGzCrcWithMostSignificaltBitNotNegative() + public void TestGzCrcWithMostSignificantBitNotNegative() { using var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz")); - using var archive = GZipArchive.Open(stream); + using var archive = GZipArchive.OpenArchive(stream); //process all entries in solid archive until the one we want to test foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) { Assert.InRange(entry.Crc, 0L, 0xFFFFFFFFL); } } + + [Fact] + public void TestGzArchiveTypeGzip() + { + using var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz")); + using var archive = GZipArchive.OpenArchive(stream); + Assert.Equal(archive.Type, ArchiveType.GZip); + } + + [Fact] + public void GZipArchive_StreamCollection_Throws_On_NonSeekable_Stream() + { + using var nonSeekable = new ForwardOnlyStream(new MemoryStream()); + using var seekable = new MemoryStream(); + + Assert.Throws(() => GZipArchive.OpenArchive([nonSeekable, seekable])); + } + + [Fact] + public void GZipArchive_Stream_Throws_On_Unreadable_Stream() + { + using var unreadable = new TestStream(new MemoryStream(), false, true, true); + + Assert.Throws(() => GZipArchive.OpenArchive(unreadable)); + } + + [Fact] + public void GZip_Archive_NonSeekableStream() + { + // Test that GZip extraction works with non-seekable streams (like HttpBaseStream) + using var fileStream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz")); + var buffer = new MemoryStream(); + fileStream.CopyTo(buffer); + buffer.Position = 0; + + // Create a non-seekable wrapper around the MemoryStream + using var nonSeekableStream = new NonSeekableStream(buffer); + using var reader = SharpCompress.Readers.GZip.GZipReader.OpenReader(nonSeekableStream); + + // Verify we can move to the first entry and read it without exceptions + Assert.True(reader.MoveToNextEntry()); + Assert.NotNull(reader.Entry); + + // Extract and verify the entry can be read + using var outputStream = new MemoryStream(); + reader.WriteEntryTo(outputStream); + + Assert.True(outputStream.Length > 0); + } + + // Helper class to simulate a non-seekable stream like HttpBaseStream + private class NonSeekableStream : Stream + { + private readonly Stream _baseStream; + + public NonSeekableStream(Stream baseStream) => _baseStream = baseStream; + + public override bool CanRead => _baseStream.CanRead; + public override bool CanSeek => false; // Simulate non-seekable stream + public override bool CanWrite => false; + public override long Length => throw new NotSupportedException(); + + public override long Position + { + get => throw new NotSupportedException(); + set => throw new NotSupportedException(); + } + + public override void Flush() => _baseStream.Flush(); + + public override int Read(byte[] buffer, int offset, int count) => + _baseStream.Read(buffer, offset, count); + + public override long Seek(long offset, SeekOrigin origin) => + throw new NotSupportedException(); + + public override void SetLength(long value) => throw new NotSupportedException(); + + public override void Write(byte[] buffer, int offset, int count) => + throw new NotSupportedException(); + } } diff --git a/tests/SharpCompress.Test/GZip/GZipCrcExtractionTests.cs b/tests/SharpCompress.Test/GZip/GZipCrcExtractionTests.cs new file mode 100644 index 00000000..2e98c962 --- /dev/null +++ b/tests/SharpCompress.Test/GZip/GZipCrcExtractionTests.cs @@ -0,0 +1,94 @@ +using System; +using System.Buffers.Binary; +using System.IO; +using System.Linq; +using System.Threading.Tasks; +using SharpCompress.Archives; +using SharpCompress.Archives.GZip; +using SharpCompress.Common; +using SharpCompress.Readers; +using SharpCompress.Readers.GZip; +using SharpCompress.Test.Mocks; +using Xunit; + +namespace SharpCompress.Test.GZip; + +public class GZipCrcExtractionTests : TestBase +{ + [Fact] + public void GZipArchive_WriteToFile_Throws_On_Crc_Mismatch() + { + using var stream = new MemoryStream(ReadCorruptedGZipTrailer(corruptCrc: true)); + using var archive = GZipArchive.OpenArchive(stream); + var entry = archive.Entries.Single(); + var destination = Path.Combine(SCRATCH_FILES_PATH, Guid.NewGuid().ToString()); + + Assert.Throws(() => entry.WriteToFile(destination)); + } + + [Fact] + public void GZipArchive_WriteToFile_Throws_On_Size_Mismatch() + { + using var stream = new MemoryStream(ReadCorruptedGZipTrailer(corruptCrc: false)); + using var archive = GZipArchive.OpenArchive(stream); + var entry = archive.Entries.Single(); + var destination = Path.Combine(SCRATCH_FILES_PATH, Guid.NewGuid().ToString()); + + Assert.Throws(() => entry.WriteToFile(destination)); + } + + [Fact] + public void GZipReader_WriteEntryToFile_Throws_On_NonSeekable_Crc_Mismatch() + { + using var stream = new MemoryStream(ReadCorruptedGZipTrailer(corruptCrc: true)); + using var nonSeekableStream = new ForwardOnlyStream(stream); + using var reader = GZipReader.OpenReader(nonSeekableStream); + var destination = Path.Combine(SCRATCH_FILES_PATH, Guid.NewGuid().ToString()); + + Assert.True(reader.MoveToNextEntry()); + Assert.Throws(() => reader.WriteEntryToFile(destination)); + } + + [Fact] + public void GZipArchive_WriteToFile_Skips_Trailer_Validation_When_CheckCrc_Is_False() + { + using var stream = new MemoryStream(ReadCorruptedGZipTrailer(corruptCrc: true)); + using var archive = GZipArchive.OpenArchive(stream); + var entry = archive.Entries.Single(); + var destination = Path.Combine(SCRATCH_FILES_PATH, Guid.NewGuid().ToString()); + + entry.WriteToFile(destination, new ExtractionOptions { CheckCrc = false }); + + Assert.Equal( + new FileInfo(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar")).Length, + new FileInfo(destination).Length + ); + } + + [Fact] + public async Task GZipArchive_WriteToFileAsync_Throws_On_Crc_Mismatch() + { +#if LEGACY_DOTNET + using var stream = new MemoryStream(ReadCorruptedGZipTrailer(corruptCrc: true)); +#else + await using var stream = new MemoryStream(ReadCorruptedGZipTrailer(corruptCrc: true)); +#endif + await using var archive = await GZipArchive.OpenAsyncArchive(stream); + var entry = await archive.EntriesAsync.SingleAsync(); + var destination = Path.Combine(SCRATCH_FILES_PATH, Guid.NewGuid().ToString()); + + await Assert.ThrowsAsync(async () => + await entry.WriteToFileAsync(destination) + ); + } + + private static byte[] ReadCorruptedGZipTrailer(bool corruptCrc) + { + var bytes = File.ReadAllBytes(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz")); + var trailer = bytes.AsSpan(bytes.Length - 8); + var offset = corruptCrc ? 0 : 4; + var value = BinaryPrimitives.ReadUInt32LittleEndian(trailer[offset..]); + BinaryPrimitives.WriteUInt32LittleEndian(trailer[offset..], value + 1); + return bytes; + } +} diff --git a/tests/SharpCompress.Test/GZip/GZipReaderAsyncTests.cs b/tests/SharpCompress.Test/GZip/GZipReaderAsyncTests.cs new file mode 100644 index 00000000..83d2b352 --- /dev/null +++ b/tests/SharpCompress.Test/GZip/GZipReaderAsyncTests.cs @@ -0,0 +1,40 @@ +using System.IO; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.IO; +using SharpCompress.Readers; +using SharpCompress.Readers.GZip; +using SharpCompress.Test.Mocks; +using Xunit; + +namespace SharpCompress.Test.GZip; + +public class GZipReaderAsyncTests : ReaderTests +{ + public GZipReaderAsyncTests() => UseExtensionInsteadOfNameToVerify = true; + + [Fact] + public async ValueTask GZip_Reader_Generic_Async() => + await ReadAsync("Tar.tar.gz", CompressionType.GZip); + + [Fact] + public async ValueTask GZip_Reader_Generic2_Async() + { + //read only as GZip item + using Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz")); + await using var reader = await GZipReader.OpenAsyncReader(new AsyncOnlyStream(stream)); + while (await reader.MoveToNextEntryAsync()) + { + Assert.NotEqual(0, reader.Entry.Size); + Assert.NotEqual(0, reader.Entry.Crc); + + // Use async overload for reading the entry + if (!reader.Entry.IsDirectory) + { + using var entryStream = await reader.OpenEntryStreamAsync(); + using var ms = new MemoryStream(); + await entryStream.CopyToAsync(ms); + } + } + } +} diff --git a/tests/SharpCompress.Test/GZip/GZipReaderTests.cs b/tests/SharpCompress.Test/GZip/GZipReaderTests.cs index b5a3d501..665b6f8f 100644 --- a/tests/SharpCompress.Test/GZip/GZipReaderTests.cs +++ b/tests/SharpCompress.Test/GZip/GZipReaderTests.cs @@ -1,6 +1,7 @@ using System.IO; using SharpCompress.Common; using SharpCompress.IO; +using SharpCompress.Readers.GZip; using Xunit; namespace SharpCompress.Test.GZip; @@ -17,7 +18,7 @@ public class GZipReaderTests : ReaderTests { //read only as GZip itme using Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz")); - using var reader = Readers.GZip.GZipReader.Open(new RewindableStream(stream)); + using var reader = GZipReader.OpenReader(SharpCompressStream.CreateNonDisposing(stream)); while (reader.MoveToNextEntry()) // Crash here { Assert.NotEqual(0, reader.Entry.Size); diff --git a/tests/SharpCompress.Test/GZip/GZipWriterAsyncTests.cs b/tests/SharpCompress.Test/GZip/GZipWriterAsyncTests.cs new file mode 100644 index 00000000..ba9ecd61 --- /dev/null +++ b/tests/SharpCompress.Test/GZip/GZipWriterAsyncTests.cs @@ -0,0 +1,94 @@ +using System.IO; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Test.Mocks; +using SharpCompress.Writers; +using SharpCompress.Writers.GZip; +using Xunit; + +namespace SharpCompress.Test.GZip; + +public class GZipWriterAsyncTests : WriterTests +{ + public GZipWriterAsyncTests() + : base(ArchiveType.GZip) => UseExtensionInsteadOfNameToVerify = true; + + [Fact] + public async ValueTask GZip_Writer_Generic_Async() + { + using ( + Stream stream = File.Open( + Path.Combine(SCRATCH_FILES_PATH, "Tar.tar.gz"), + FileMode.OpenOrCreate, + FileAccess.Write + ) + ) + await using ( + var writer = await WriterFactory.OpenAsyncWriter( + new AsyncOnlyStream(stream), + ArchiveType.GZip, + new WriterOptions(CompressionType.GZip) + ) + ) + { + await writer.WriteAsync("Tar.tar", Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar")); + } + CompareArchivesByPath( + Path.Combine(SCRATCH_FILES_PATH, "Tar.tar.gz"), + Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz") + ); + } + + [Fact] + public async ValueTask GZip_Writer_Async() + { + using ( + Stream stream = File.Open( + Path.Combine(SCRATCH_FILES_PATH, "Tar.tar.gz"), + FileMode.OpenOrCreate, + FileAccess.Write + ) + ) + await using (var writer = new GZipWriter(new AsyncOnlyStream(stream))) + { + await writer.WriteAsync("Tar.tar", Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar")); + } + CompareArchivesByPath( + Path.Combine(SCRATCH_FILES_PATH, "Tar.tar.gz"), + Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz") + ); + } + + [Fact] + public void GZip_Writer_Generic_Bad_Compression_Async() => + Assert.Throws(() => + { + using Stream stream = File.OpenWrite(Path.Combine(SCRATCH_FILES_PATH, "Tar.tar.gz")); + using var writer = WriterFactory.OpenWriter( + new AsyncOnlyStream(stream), + ArchiveType.GZip, + new WriterOptions(CompressionType.BZip2) + ); + }); + + [Fact] + public async ValueTask GZip_Writer_Entry_Path_With_Dir_Async() + { + using ( + Stream stream = File.Open( + Path.Combine(SCRATCH_FILES_PATH, "Tar.tar.gz"), + FileMode.OpenOrCreate, + FileAccess.Write + ) + ) + await using (var writer = new GZipWriter(new AsyncOnlyStream(stream))) + { + var path = Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar"); + await writer.WriteAsync(path, path); + } + CompareArchivesByPath( + Path.Combine(SCRATCH_FILES_PATH, "Tar.tar.gz"), + Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz") + ); + } +} diff --git a/tests/SharpCompress.Test/GZip/GZipWriterDirectoryTests.cs b/tests/SharpCompress.Test/GZip/GZipWriterDirectoryTests.cs new file mode 100644 index 00000000..5d51c441 --- /dev/null +++ b/tests/SharpCompress.Test/GZip/GZipWriterDirectoryTests.cs @@ -0,0 +1,19 @@ +using System; +using System.IO; +using SharpCompress.Common; +using SharpCompress.Writers.GZip; +using Xunit; + +namespace SharpCompress.Test.GZip; + +public class GZipWriterDirectoryTests : TestBase +{ + [Fact] + public void GZipWriter_WriteDirectory_ThrowsNotSupportedException() + { + using var memoryStream = new MemoryStream(); + using var writer = new GZipWriter(memoryStream, new GZipWriterOptions()); + + Assert.Throws(() => writer.WriteDirectory("test-dir", DateTime.Now)); + } +} diff --git a/tests/SharpCompress.Test/GZip/GZipWriterTests.cs b/tests/SharpCompress.Test/GZip/GZipWriterTests.cs index f4ca13d3..b7594d12 100644 --- a/tests/SharpCompress.Test/GZip/GZipWriterTests.cs +++ b/tests/SharpCompress.Test/GZip/GZipWriterTests.cs @@ -1,4 +1,4 @@ -using System.IO; +using System.IO; using SharpCompress.Common; using SharpCompress.Writers; using SharpCompress.Writers.GZip; @@ -21,7 +21,13 @@ public class GZipWriterTests : WriterTests FileAccess.Write ) ) - using (var writer = WriterFactory.Open(stream, ArchiveType.GZip, CompressionType.GZip)) + using ( + var writer = WriterFactory.OpenWriter( + stream, + ArchiveType.GZip, + new WriterOptions(CompressionType.GZip) + ) + ) { writer.Write("Tar.tar", Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar")); } @@ -56,7 +62,11 @@ public class GZipWriterTests : WriterTests Assert.Throws(() => { using Stream stream = File.OpenWrite(Path.Combine(SCRATCH_FILES_PATH, "Tar.tar.gz")); - using var writer = WriterFactory.Open(stream, ArchiveType.GZip, CompressionType.BZip2); + using var writer = WriterFactory.OpenWriter( + stream, + ArchiveType.GZip, + new WriterOptions(CompressionType.BZip2) + ); }); [Fact] diff --git a/tests/SharpCompress.Test/LazyAsyncReadOnlyCollectionTests.cs b/tests/SharpCompress.Test/LazyAsyncReadOnlyCollectionTests.cs new file mode 100644 index 00000000..aa9c2af4 --- /dev/null +++ b/tests/SharpCompress.Test/LazyAsyncReadOnlyCollectionTests.cs @@ -0,0 +1,358 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Xunit; + +namespace SharpCompress.Test; + +public class LazyAsyncReadOnlyCollectionTests +{ + // Helper class to track how many times items are enumerated from the source + private class TrackingAsyncEnumerable : IAsyncEnumerable + { + private readonly List _items; + public int EnumerationCount { get; private set; } + public int ItemsRequestedCount { get; private set; } + + public TrackingAsyncEnumerable(params T[] items) + { + _items = new List(items); + } + + public IAsyncEnumerator GetAsyncEnumerator(CancellationToken cancellationToken = default) + { + EnumerationCount++; + return new TrackingEnumerator(this, cancellationToken); + } + + private class TrackingEnumerator : IAsyncEnumerator + { + private readonly TrackingAsyncEnumerable _parent; + private readonly CancellationToken _cancellationToken; + private int _index = -1; + + public TrackingEnumerator( + TrackingAsyncEnumerable parent, + CancellationToken cancellationToken + ) + { + _parent = parent; + _cancellationToken = cancellationToken; + } + + public T Current => _parent._items[_index]; + + public async ValueTask MoveNextAsync() + { + _cancellationToken.ThrowIfCancellationRequested(); + await Task.Yield(); // Simulate async behavior + _index++; + if (_index < _parent._items.Count) + { + _parent.ItemsRequestedCount++; + return true; + } + return false; + } + + public ValueTask DisposeAsync() => default; + } + } + + [Fact] + public async Task BasicEnumeration_IteratesThroughAllItems() + { + // Arrange + var source = new TrackingAsyncEnumerable(1, 2, 3, 4, 5); + var collection = new LazyAsyncReadOnlyCollection(source); + + // Act + var results = new List(); + await foreach (var item in collection) + { + results.Add(item); + } + + // Assert + Assert.Equal(5, results.Count); + Assert.Equal(new[] { 1, 2, 3, 4, 5 }, results); + Assert.Equal(1, source.EnumerationCount); + Assert.Equal(5, source.ItemsRequestedCount); + } + + [Fact] + public async Task MultipleEnumerations_UsesCachedBackingList() + { + // Arrange + var source = new TrackingAsyncEnumerable("a", "b", "c"); + var collection = new LazyAsyncReadOnlyCollection(source); + + // Act - First enumeration + var firstResults = new List(); + await foreach (var item in collection) + { + firstResults.Add(item); + } + + var itemsRequestedAfterFirst = source.ItemsRequestedCount; + + // Act - Second enumeration + var secondResults = new List(); + await foreach (var item in collection) + { + secondResults.Add(item); + } + + // Assert + Assert.Equal(firstResults, secondResults); + Assert.Equal(new[] { "a", "b", "c" }, secondResults); + + // Source should only be enumerated once + Assert.Equal(1, source.EnumerationCount); + + // Items should only be requested from source during first enumeration + Assert.Equal(itemsRequestedAfterFirst, source.ItemsRequestedCount); + } + + [Fact] + public async Task EnsureFullyLoaded_LoadsAllItemsIntoBackingList() + { + // Arrange + var source = new TrackingAsyncEnumerable(10, 20, 30, 40); + var collection = new LazyAsyncReadOnlyCollection(source); + + // Act + await collection.EnsureFullyLoaded(); + var loaded = collection.GetLoaded().ToList(); + + // Assert + Assert.Equal(4, loaded.Count); + Assert.Equal(new[] { 10, 20, 30, 40 }, loaded); + Assert.Equal(4, source.ItemsRequestedCount); + } + + [Fact] + public async Task GetLoaded_ReturnsOnlyLoadedItemsBeforeFullEnumeration() + { + // Arrange + var source = new TrackingAsyncEnumerable(1, 2, 3, 4, 5); + var collection = new LazyAsyncReadOnlyCollection(source); + + // Act - Partially enumerate (only first 2 items) + var enumerator = collection.GetAsyncEnumerator(); + await enumerator.MoveNextAsync(); // Load item 1 + await enumerator.MoveNextAsync(); // Load item 2 + + var loadedItems = collection.GetLoaded().ToList(); + + // Continue enumeration + await enumerator.MoveNextAsync(); // Load item 3 + var loadedItemsAfter = collection.GetLoaded().ToList(); + + await enumerator.DisposeAsync(); + + // Assert + Assert.Equal(2, loadedItems.Count); + Assert.Equal(new[] { 1, 2 }, loadedItems); + + Assert.Equal(3, loadedItemsAfter.Count); + Assert.Equal(new[] { 1, 2, 3 }, loadedItemsAfter); + } + + [Fact] + public async Task CancellationToken_PassedToGetAsyncEnumerator_HonorsToken() + { + // Arrange + var cts = new CancellationTokenSource(); + var source = new TrackingAsyncEnumerable(1, 2, 3, 4, 5, 6, 7, 8, 9, 10); + var collection = new LazyAsyncReadOnlyCollection(source); + + // Act & Assert + var results = new List(); + var exception = await Assert.ThrowsAsync(async () => + { + await foreach (var item in collection.WithCancellation(cts.Token)) + { + results.Add(item); + if (item == 3) + { + cts.Cancel(); + } + } + }); + + Assert.Equal(3, results.Count); + Assert.Equal(new[] { 1, 2, 3 }, results); + } + + [Fact] + public async Task CancellationDuringMoveNextAsync_ThrowsOperationCanceledException() + { + // Arrange + var cts = new CancellationTokenSource(); + var source = CreateDelayedAsyncEnumerable( + new[] { 1, 2, 3, 4, 5 }, + TimeSpan.FromMilliseconds(50) + ); + var collection = new LazyAsyncReadOnlyCollection(source); + + // Act & Assert + await Assert.ThrowsAsync(async () => + { + var enumerator = collection.GetAsyncEnumerator(cts.Token); + await enumerator.MoveNextAsync(); + await enumerator.MoveNextAsync(); + + cts.Cancel(); + + await enumerator.MoveNextAsync(); // Should throw + }); + } + + [Fact] + public async Task EmptySourceEnumerable_ReturnsNoItems() + { + // Arrange + var source = new TrackingAsyncEnumerable(); + var collection = new LazyAsyncReadOnlyCollection(source); + + // Act + var results = new List(); + await foreach (var item in collection) + { + results.Add(item); + } + + // Assert + Assert.Empty(results); + Assert.Equal(1, source.EnumerationCount); + } + + [Fact] + public async Task SingleItemSourceEnumerable_ReturnsSingleItem() + { + // Arrange + var source = new TrackingAsyncEnumerable("only"); + var collection = new LazyAsyncReadOnlyCollection(source); + + // Act + var results = new List(); + await foreach (var item in collection) + { + results.Add(item); + } + + // Assert + Assert.Single(results); + Assert.Equal("only", results[0]); + } + + [Fact] + public async Task PartialEnumeration_ThenGetLoaded_ReturnsOnlyEnumeratedItems() + { + // Arrange + var source = new TrackingAsyncEnumerable(10, 20, 30, 40, 50); + var collection = new LazyAsyncReadOnlyCollection(source); + + // Act - Enumerate only first 3 items + await using (var enumerator = collection.GetAsyncEnumerator()) + { + var hasMore = await enumerator.MoveNextAsync(); + Assert.True(hasMore); + hasMore = await enumerator.MoveNextAsync(); + Assert.True(hasMore); + hasMore = await enumerator.MoveNextAsync(); + Assert.True(hasMore); + } + + var loadedItems = collection.GetLoaded().ToList(); + + // Assert + Assert.Equal(3, loadedItems.Count); + Assert.Equal(new[] { 10, 20, 30 }, loadedItems); + Assert.Equal(3, source.ItemsRequestedCount); + } + + [Fact] + public async Task ConcurrentEnumerations_ShareBackingList() + { + // Arrange + var source = new TrackingAsyncEnumerable(1, 2, 3, 4, 5); + var collection = new LazyAsyncReadOnlyCollection(source); + + // Act - Fully load the collection first, then enumerate from two threads + await collection.EnsureFullyLoaded(); + + var task1 = Task.Run(async () => + { + var results = new List(); + await foreach (var item in collection) + { + results.Add(item); + await Task.Delay(5); + } + return results; + }); + + var task2 = Task.Run(async () => + { + var results = new List(); + await foreach (var item in collection) + { + results.Add(item); + await Task.Delay(5); + } + return results; + }); + + var results1 = await task1; + var results2 = await task2; + + // Assert - Both enumerations should see all items from the shared backing list + Assert.Equal(new[] { 1, 2, 3, 4, 5 }, results1); + Assert.Equal(new[] { 1, 2, 3, 4, 5 }, results2); + Assert.Equal(5, source.ItemsRequestedCount); + } + + [Fact] + public async Task DisposeAsync_OnLazyLoader_CompletesSuccessfully() + { + // Arrange + var source = new TrackingAsyncEnumerable(1, 2, 3); + var collection = new LazyAsyncReadOnlyCollection(source); + + // Act + await using (var enumerator = collection.GetAsyncEnumerator()) + { + await enumerator.MoveNextAsync(); + var firstItem = enumerator.Current; + Assert.Equal(1, firstItem); + + // Dispose is called automatically by await using + } + + // Assert - should be able to enumerate again after disposal + var results = new List(); + await foreach (var item in collection) + { + results.Add(item); + } + + Assert.Equal(new[] { 1, 2, 3 }, results); + } + + // Helper method to create an async enumerable with delays + private static async IAsyncEnumerable CreateDelayedAsyncEnumerable( + IEnumerable items, + TimeSpan delay + ) + { + foreach (var item in items) + { + await Task.Delay(delay); + yield return item; + } + } +} diff --git a/tests/SharpCompress.Test/LazyReadOnlyCollectionTests.cs b/tests/SharpCompress.Test/LazyReadOnlyCollectionTests.cs new file mode 100644 index 00000000..e71dc3eb --- /dev/null +++ b/tests/SharpCompress.Test/LazyReadOnlyCollectionTests.cs @@ -0,0 +1,382 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using Xunit; + +namespace SharpCompress.Test; + +public class LazyReadOnlyCollectionTests +{ + // Helper class to track how many times items are enumerated from the source + private class TrackingEnumerable : IEnumerable + { + private readonly List _items; + public int EnumerationCount { get; private set; } + public int ItemsRequestedCount { get; private set; } + + public TrackingEnumerable(params T[] items) + { + _items = new List(items); + } + + public IEnumerator GetEnumerator() + { + EnumerationCount++; + return new TrackingEnumerator(this); + } + + System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator() => + GetEnumerator(); + + private class TrackingEnumerator : IEnumerator + { + private readonly TrackingEnumerable _parent; + private int _index = -1; + + public TrackingEnumerator(TrackingEnumerable parent) + { + _parent = parent; + } + + public T Current => _parent._items[_index]; + + object? System.Collections.IEnumerator.Current => Current; + + public bool MoveNext() + { + _index++; + if (_index < _parent._items.Count) + { + _parent.ItemsRequestedCount++; + return true; + } + return false; + } + + public void Reset() => throw new NotSupportedException(); + + public void Dispose() { } + } + } + + [Fact] + public void BasicEnumeration_IteratesThroughAllItems() + { + // Arrange + var source = new TrackingEnumerable(1, 2, 3, 4, 5); + var collection = new LazyReadOnlyCollection(source); + + // Act + var results = new List(); + foreach (var item in collection) + { + results.Add(item); + } + + // Assert + Assert.Equal(5, results.Count); + Assert.Equal(new[] { 1, 2, 3, 4, 5 }, results); + Assert.Equal(1, source.EnumerationCount); + Assert.Equal(5, source.ItemsRequestedCount); + } + + [Fact] + public void MultipleEnumerations_UsesCachedBackingList() + { + // Arrange + var source = new TrackingEnumerable("a", "b", "c"); + var collection = new LazyReadOnlyCollection(source); + + // Act - First enumeration + var firstResults = new List(); + foreach (var item in collection) + { + firstResults.Add(item); + } + + var itemsRequestedAfterFirst = source.ItemsRequestedCount; + + // Act - Second enumeration + var secondResults = new List(); + foreach (var item in collection) + { + secondResults.Add(item); + } + + // Assert + Assert.Equal(firstResults, secondResults); + Assert.Equal(new[] { "a", "b", "c" }, secondResults); + + // Source should only be enumerated once + Assert.Equal(1, source.EnumerationCount); + + // Items should only be requested from source during first enumeration + Assert.Equal(itemsRequestedAfterFirst, source.ItemsRequestedCount); + } + + [Fact] + public void EnsureFullyLoaded_LoadsAllItemsIntoBackingList() + { + // Arrange + var source = new TrackingEnumerable(10, 20, 30, 40); + var collection = new LazyReadOnlyCollection(source); + + // Act + collection.EnsureFullyLoaded(); + var loaded = collection.GetLoaded().ToList(); + + // Assert + Assert.Equal(4, loaded.Count); + Assert.Equal(new[] { 10, 20, 30, 40 }, loaded); + Assert.Equal(4, source.ItemsRequestedCount); + } + + [Fact] + public void GetLoaded_ReturnsOnlyLoadedItemsBeforeFullEnumeration() + { + // Arrange + var source = new TrackingEnumerable(1, 2, 3, 4, 5); + var collection = new LazyReadOnlyCollection(source); + + // Act - Partially enumerate (only first 2 items) + using var enumerator = collection.GetEnumerator(); + enumerator.MoveNext(); // Load item 1 + enumerator.MoveNext(); // Load item 2 + + var loadedItems = collection.GetLoaded().ToList(); + + // Continue enumeration + enumerator.MoveNext(); // Load item 3 + var loadedItemsAfter = collection.GetLoaded().ToList(); + + // Assert + Assert.Equal(2, loadedItems.Count); + Assert.Equal(new[] { 1, 2 }, loadedItems); + + Assert.Equal(3, loadedItemsAfter.Count); + Assert.Equal(new[] { 1, 2, 3 }, loadedItemsAfter); + } + + [Fact] + public void Count_TriggersFullLoad() + { + // Arrange + var source = new TrackingEnumerable(1, 2, 3, 4, 5); + var collection = new LazyReadOnlyCollection(source); + + // Act + var count = collection.Count; + + // Assert + Assert.Equal(5, count); + Assert.Equal(5, source.ItemsRequestedCount); + Assert.Equal(5, collection.GetLoaded().Count()); + } + + [Fact] + public void Contains_TriggersFullLoadAndSearches() + { + // Arrange + var source = new TrackingEnumerable("apple", "banana", "cherry"); + var collection = new LazyReadOnlyCollection(source); + + // Act + var containsBanana = collection.Contains("banana"); + var containsOrange = collection.Contains("orange"); + + // Assert + Assert.True(containsBanana); + Assert.False(containsOrange); + Assert.Equal(3, source.ItemsRequestedCount); + Assert.Equal(3, collection.GetLoaded().Count()); + } + + [Fact] + public void CopyTo_TriggersFullLoadAndCopiesArray() + { + // Arrange + var source = new TrackingEnumerable(10, 20, 30); + var collection = new LazyReadOnlyCollection(source); + var array = new int[5]; + + // Act + collection.CopyTo(array, 1); + + // Assert + Assert.Equal(0, array[0]); + Assert.Equal(10, array[1]); + Assert.Equal(20, array[2]); + Assert.Equal(30, array[3]); + Assert.Equal(0, array[4]); + Assert.Equal(3, source.ItemsRequestedCount); + } + + [Fact] + public void EmptySourceEnumerable_ReturnsNoItems() + { + // Arrange + var source = new TrackingEnumerable(); + var collection = new LazyReadOnlyCollection(source); + + // Act + var results = new List(); + foreach (var item in collection) + { + results.Add(item); + } + + // Assert + Assert.Empty(results); + Assert.Equal(0, collection.Count); + Assert.Equal(1, source.EnumerationCount); + } + + [Fact] + public void SingleItemSourceEnumerable_ReturnsSingleItem() + { + // Arrange + var source = new TrackingEnumerable("only"); + var collection = new LazyReadOnlyCollection(source); + + // Act + var results = new List(); + foreach (var item in collection) + { + results.Add(item); + } + + // Assert + Assert.Single(results); + Assert.Equal("only", results[0]); + Assert.Equal(1, collection.Count); + } + + [Fact] + public void PartialEnumeration_ThenGetLoaded_ReturnsOnlyEnumeratedItems() + { + // Arrange + var source = new TrackingEnumerable(10, 20, 30, 40, 50); + var collection = new LazyReadOnlyCollection(source); + + // Act - Enumerate only first 3 items + using (var enumerator = collection.GetEnumerator()) + { + var hasMore = enumerator.MoveNext(); + Assert.True(hasMore); + hasMore = enumerator.MoveNext(); + Assert.True(hasMore); + hasMore = enumerator.MoveNext(); + Assert.True(hasMore); + } + + var loadedItems = collection.GetLoaded().ToList(); + + // Assert + Assert.Equal(3, loadedItems.Count); + Assert.Equal(new[] { 10, 20, 30 }, loadedItems); + Assert.Equal(3, source.ItemsRequestedCount); + } + + [Fact] + public void Reset_ThrowsNotSupportedException() + { + // Arrange + var source = new TrackingEnumerable(1, 2, 3); + var collection = new LazyReadOnlyCollection(source); + + // Act & Assert + using var enumerator = collection.GetEnumerator(); + enumerator.MoveNext(); + Assert.Throws(() => enumerator.Reset()); + } + + [Fact] + public void Dispose_OnEnumerator_CompletesSuccessfully() + { + // Arrange + var source = new TrackingEnumerable(1, 2, 3); + var collection = new LazyReadOnlyCollection(source); + + // Act + using (var enumerator = collection.GetEnumerator()) + { + enumerator.MoveNext(); + var firstItem = enumerator.Current; + Assert.Equal(1, firstItem); + + // Dispose is called automatically by using + } + + // Assert - should be able to enumerate again after disposal + var results = new List(); + foreach (var item in collection) + { + results.Add(item); + } + + Assert.Equal(new[] { 1, 2, 3 }, results); + } + + [Fact] + public void IsReadOnly_ReturnsTrue() + { + // Arrange + var source = new TrackingEnumerable(1, 2, 3); + var collection = new LazyReadOnlyCollection(source); + + // Act & Assert + Assert.True(collection.IsReadOnly); + } + + [Fact] + public void Add_ThrowsNotSupportedException() + { + // Arrange + var source = new TrackingEnumerable(1, 2, 3); + var collection = new LazyReadOnlyCollection(source); + + // Act & Assert + Assert.Throws(() => collection.Add(4)); + } + + [Fact] + public void Clear_ThrowsNotSupportedException() + { + // Arrange + var source = new TrackingEnumerable(1, 2, 3); + var collection = new LazyReadOnlyCollection(source); + + // Act & Assert + Assert.Throws(() => collection.Clear()); + } + + [Fact] + public void Remove_ThrowsNotSupportedException() + { + // Arrange + var source = new TrackingEnumerable(1, 2, 3); + var collection = new LazyReadOnlyCollection(source); + + // Act & Assert + Assert.Throws(() => collection.Remove(1)); + } + + [Fact] + public void NonGenericEnumerator_WorksCorrectly() + { + // Arrange + var source = new TrackingEnumerable(1, 2, 3); + var collection = new LazyReadOnlyCollection(source); + + // Act - Use non-generic IEnumerator + var results = new List(); + var enumerable = (System.Collections.IEnumerable)collection; + foreach (var item in enumerable) + { + results.Add((int)item); + } + + // Assert + Assert.Equal(new[] { 1, 2, 3 }, results); + } +} diff --git a/tests/SharpCompress.Test/Lzw/LzwReaderAsyncTests.cs b/tests/SharpCompress.Test/Lzw/LzwReaderAsyncTests.cs new file mode 100644 index 00000000..0b09298b --- /dev/null +++ b/tests/SharpCompress.Test/Lzw/LzwReaderAsyncTests.cs @@ -0,0 +1,43 @@ +using System.IO; +using SharpCompress.Common; +using SharpCompress.Readers; +using SharpCompress.Readers.Lzw; +using Xunit; + +namespace SharpCompress.Test.Lzw; + +public class LzwReaderAsyncTests : ReaderTests +{ + public LzwReaderAsyncTests() => UseExtensionInsteadOfNameToVerify = true; + + [Fact] + public async System.Threading.Tasks.Task Lzw_Reader_Async() + { + await ReadAsync("Tar.tar.Z", CompressionType.Lzw); + } + + [Fact] + public async System.Threading.Tasks.Task Lzw_Reader_Plain_Z_File_Async() + { + // Test async reading of a plain .Z file (not tar-wrapped) using LzwReader directly + using Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "large_test.txt.Z")); + using var reader = LzwReader.OpenReader(stream); + + Assert.Equal(ArchiveType.Lzw, reader.Type); + Assert.True(reader.MoveToNextEntry()); + + var entry = reader.Entry; + Assert.NotNull(entry); + Assert.Equal(CompressionType.Lzw, entry.CompressionType); + + // When opened as FileStream, key should be derived from filename + Assert.Equal("large_test.txt", entry.Key); + + // Decompress asynchronously + using var entryStream = reader.OpenEntryStream(); + using var ms = new MemoryStream(); + await entryStream.CopyToAsync(ms); + + Assert.Equal(22300, ms.Length); + } +} diff --git a/tests/SharpCompress.Test/Lzw/LzwReaderTests.cs b/tests/SharpCompress.Test/Lzw/LzwReaderTests.cs new file mode 100644 index 00000000..7a74967e --- /dev/null +++ b/tests/SharpCompress.Test/Lzw/LzwReaderTests.cs @@ -0,0 +1,94 @@ +using System.IO; +using SharpCompress.Common; +using SharpCompress.IO; +using SharpCompress.Readers; +using SharpCompress.Readers.Lzw; +using Xunit; + +namespace SharpCompress.Test.Lzw; + +public class LzwReaderTests : ReaderTests +{ + public LzwReaderTests() => UseExtensionInsteadOfNameToVerify = true; + + [Fact] + public void Lzw_Reader_Generic() => Read("Tar.tar.Z", CompressionType.Lzw); + + [Fact] + public void Lzw_Reader_Generic2() + { + //read only as Lzw item + using Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.Z")); + using var reader = LzwReader.OpenReader(SharpCompressStream.CreateNonDisposing(stream)); + while (reader.MoveToNextEntry()) + { + // LZW doesn't have CRC or Size in header like GZip, so we just check the entry exists + Assert.NotNull(reader.Entry); + } + } + + [Fact] + public void Lzw_Reader_Factory_Detects_Tar_Wrapper() + { + // Note: Testing with Tar.tar.Z because: + // 1. LzwStream only supports decompression, not compression + // 2. This tests the important tar wrapper detection code path in LzwFactory.TryOpenReader + // 3. Verifies that tar.Z files correctly return TarReader with CompressionType.Lzw + using Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.Z")); + using var reader = ReaderFactory.OpenReader( + stream, + ReaderOptions.ForExternalStream with + { + LeaveStreamOpen = false, + } + ); + + // Should detect as Tar archive with Lzw compression + Assert.Equal(ArchiveType.Tar, reader.Type); + Assert.True(reader.MoveToNextEntry()); + Assert.NotNull(reader.Entry); + Assert.Equal(CompressionType.Lzw, reader.Entry.CompressionType); + } + + [Fact] + public void Lzw_Reader_Plain_Z_File() + { + // Test with a plain .Z file (not tar-wrapped) using LzwReader directly + using Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "large_test.txt.Z")); + using var reader = LzwReader.OpenReader(stream); + + Assert.True(reader.MoveToNextEntry()); + var entry = reader.Entry; + Assert.NotNull(entry); + Assert.Equal(CompressionType.Lzw, entry.CompressionType); + + // Entry key should be "large_test.txt" (stripped .Z extension) when opened via FileStream + Assert.Equal("large_test.txt", entry.Key); + + // Decompress and verify content + using var entryStream = reader.OpenEntryStream(); + using var ms = new MemoryStream(); + entryStream.CopyTo(ms); + var decompressed = System.Text.Encoding.UTF8.GetString(ms.ToArray()); + + Assert.Equal(22300, ms.Length); + Assert.Contains("This is a test file for LZW compression testing", decompressed); + } + + [Fact] + public void Lzw_Reader_Factory_Detects_Plain_Z_File() + { + // Test that ReaderFactory correctly identifies a plain .Z file (not tar-wrapped) + using Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "large_test.txt.Z")); + using var reader = ReaderFactory.OpenReader(stream); + + // Should detect as Lzw archive (not Tar) + Assert.Equal(ArchiveType.Lzw, reader.Type); + Assert.True(reader.MoveToNextEntry()); + Assert.NotNull(reader.Entry); + Assert.Equal(CompressionType.Lzw, reader.Entry.CompressionType); + + // When opened via ReaderFactory with a non-FileStream, key defaults to "data" + Assert.NotNull(reader.Entry.Key); + } +} diff --git a/tests/SharpCompress.Test/MalformedInputTests.cs b/tests/SharpCompress.Test/MalformedInputTests.cs new file mode 100644 index 00000000..d815a410 --- /dev/null +++ b/tests/SharpCompress.Test/MalformedInputTests.cs @@ -0,0 +1,179 @@ +#if !LEGACY_DOTNET +using System; +using System.IO; +using AwesomeAssertions; +using SharpCompress.Common; +using SharpCompress.Readers; +using Xunit; + +namespace SharpCompress.Test; + +/// +/// Tests that malformed compressed input is handled gracefully, throwing library exceptions +/// rather than unhandled IndexOutOfRangeException, DivideByZeroException, or NullReferenceException. +/// +public class MalformedInputTests +{ + private static void VerifyMalformedInputThrowsLibraryException(string hex) + { + var data = Convert.FromHexString(hex); + using var ms = new MemoryStream(data); + var buf = new byte[4096]; + + Action act = () => + { + using var reader = ReaderFactory.OpenReader(ms); + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + using var entryStream = reader.OpenEntryStream(); + while (entryStream.Read(buf, 0, buf.Length) > 0) { } + } + } + }; + + act.Should() + .Throw() + .And.Should() + .BeAssignableTo( + "malformed input should throw a library exception, not a raw system exception" + ); + } + + [Fact] + public void LzwStream_DivideByZero_ThrowsLibraryException() + { + // LZW stream with invalid header that would cause DivideByZero on subsequent reads + VerifyMalformedInputThrowsLibraryException( + "1f9d1a362f20000000130003edd1310a8030f1605ca2b26245c47b97e6d615e29400000000130003edd1310a8030f1605c606060606060606060606060606060606060606060606060007f60606060280000" + ); + } + + [Fact] + public void LzwStream_IndexOutOfRange_ThrowsLibraryException() + { + // LZW stream with maxBits < INIT_BITS causing table size mismatch + VerifyMalformedInputThrowsLibraryException( + "1f9d0836e1553ac4e1ce9ea227000000000000001070b4058faf051127c54144f8bfe54192e141bab6efe8032c41cd64004aef53da4acc8077a5b26245c47b97e6d615e29400000000000003edd1310a8030f1e2ee66ff535d800000000b00000000" + ); + } + + [Fact] + public void BZip2_NullRef_InBsR_ThrowsLibraryException() + { + // BZip2 stream with invalid block size causing null bsStream access + VerifyMalformedInputThrowsLibraryException( + "425a6857575757575768575757575757fff2fff27c007159425a6857ff0f21007159c1e2d5e2" + ); + } + + [Fact] + public void BZip2_IndexOutOfRange_InGetAndMoveToFrontDecode_ThrowsLibraryException() + { + // BZip2 with malformed Huffman tables causing code-too-long or bad perm index + VerifyMalformedInputThrowsLibraryException( + "425a6839314159265359c1c080e2000001410000100244a000305a6839314159265359c1c080e2000001410000100244a00030cd00c3cd00c34629971772c080e2" + ); + } + + [Fact] + public void SqueezeStream_IndexOutOfRange_ThrowsLibraryException() + { + // Squeezed ARC stream with malformed Huffman tree node indices + VerifyMalformedInputThrowsLibraryException( + "1a041a425a081a0000090000606839425a081730765cbb311042265300040000090000606839425a081730765cbb31104226530053" + ); + } + + [Fact] + public void ArcLzwStream_IndexOutOfRange_ThrowsLibraryException() + { + // ARC LZW stream with empty or malformed compressed data + VerifyMalformedInputThrowsLibraryException( + "1a081a1931081a00000000f9ffffff00000000ddff000000000000000000000000000012006068394200000080c431b37fff531042d9ff" + ); + } + + [Fact] + public void ExplodeStream_IndexOutOfRange_ThrowsLibraryException() + { + // ZIP entry using Implode/Explode with invalid Huffman tables + VerifyMalformedInputThrowsLibraryException( + "504b03040a000000060000ff676767676767676767676767676700000000683a36060000676767676767676767676700000000000000000000000000000000000000000000000000000000630000000000800000000000002e7478745554090003a8c8b6696045ac6975780b000104e803000004e803000068656c6c6f0a504b01021e030a0000000000147f6f5c20303a3639314159265359c1c080e2000001410000100244a00030cd00c346299717786975870b000104e8030000780b000104e803000004e8030000504b050600000000010000e74f004040490000000064" + ); + } + + [Fact] + public void Deflate64_IndexOutOfRange_ThrowsLibraryException() + { + // ZIP entry using Deflate64 with invalid Huffman data + VerifyMalformedInputThrowsLibraryException( + "504b03040a00009709001c0068656c6c6f2e807874555409000000000000147f6f5c20303a36060000ff0600000009425a6839314159265359595959595959a481000000000000000000007478925554050001c601003dffff000000000000001e000000001e00000000000000000000e1490000000000" + ); + } + + [Fact] + public void PPMd_NullRef_ThrowsLibraryException() + { + // ZIP entry using PPMd with malformed properties triggering uninitialized model access + VerifyMalformedInputThrowsLibraryException( + "504b03040000007462001c905c206600fa80ffffffffff1f8b0a00000000000003edd1310a80cf0c00090010000b000000e000000000030000002e000000686515e294362f763ac439d493d62a3671081e05c14114b4058faf051127c54144f8bfe541ace141bab6ef643c2ce2000001410000100244a00040cd41bdc76c4aef3977a5b25645c47b97e6d615e294362f763ac439d493d62a367108f1e2ee66ff535efa7f3015e2943601003ac439d493d62a3671081e05c14114b4058faf3a0003edd1310a80cf8597e6d60500140409" + ); + } + + [Fact] + public void LZMA_NullRef_ThrowsLibraryException() + { + // ZIP entry using LZMA with invalid dictionary size (0) causing null window buffer access + VerifyMalformedInputThrowsLibraryException( + "504b03040a0200000e001c0068646c6c6f2e7478745554ac507578000000000000000000000000000000000000000000e80300000000000068030a0000000000147f040020303a360600002e7478745554090003a8c8b6696045ac69f5780b0006ff1d000908180000e8030000000000a4810000109a9a9a8b8b8b8b8b8b8b8b8b8b8b8b8b8b8b8b8b8b8b8b8b9a0000000000000000000000e80300000000000068030a0000009a9a9a504b03440a6fcb486c6c6f2e74ffff" + ); + } + + [Fact] + public void Reduce_DecompressionBomb_Method2_ThrowsLibraryException() + { + // 31-byte ZIP using Reduce method 2 with declared uncompressed size far exceeding the + // actual compressed data - the decompressor must not generate unbounded output. + VerifyMalformedInputThrowsLibraryException( + "504b03040a000000020000000200f7ff0500f7ff05ff200600180700000000" + ); + } + + [Fact] + public void Deflate64_HuffmanTree_IndexOutOfRange_ThrowsLibraryException() + { + // 105-byte ZIP using Deflate64 with invalid Huffman code lengths causing IOOB in CreateTable + VerifyMalformedInputThrowsLibraryException( + "504b03040a00005409000088c8b669757800009ac8b66975783606000000640028b52ffd047fff" + + "02009a888888888820313735303600303132002030007573746172202000757001307230819b75" + + "72756e7475410a000c2000391eeb061ffe391eeb068f0c0a000c20" + ); + } + + [Fact] + public void BZip2_GetAndMoveToFrontDecode_IndexOutOfRange_ThrowsLibraryException() + { + // 93-byte BZip2 stream triggering IOOB deeper in GetAndMoveToFrontDecode + VerifyMalformedInputThrowsLibraryException( + "425a6839314159265359c1c080e2000001410000100244a00100808b640006000775780b2ef2ed" + + "0001393beb06060606060606060606f9050605060606060f0654090003ffffff7f003403" + + "0a0002001f8b7fff0000000000e98b8b3931" + ); + } + + [Fact] + public void Zip_ShrinkOOM_CraftedCompressedSize_ThrowsLibraryException() + { + // 122-byte ZIP with Shrink compression and compressed size set to 0x7FFFFFFF (2 GB). + // The library must not attempt to allocate a 2 GB buffer based on the untrusted header. + VerifyMalformedInputThrowsLibraryException( + "504b03040a0000000100147f6f5c20303a36ffffff7f0600000009001c0068656c6c6f2e747874" + + "5554090003a8c8b6696045ac6975780b01e8303a36060000000600000009001800000001004f2a" + + "2a2a2a0c2000395d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5d000004e8303a360600000006000000" + + "0900180000" + ); + } +} +#endif diff --git a/tests/SharpCompress.Test/MarkingBinaryReaderParityTests.cs b/tests/SharpCompress.Test/MarkingBinaryReaderParityTests.cs new file mode 100644 index 00000000..ff4646b7 --- /dev/null +++ b/tests/SharpCompress.Test/MarkingBinaryReaderParityTests.cs @@ -0,0 +1,258 @@ +using System; +using System.IO; +using System.Linq; +using System.Threading.Tasks; +using SharpCompress.Common.Rar; +using SharpCompress.IO; +using Xunit; + +namespace SharpCompress.Test; + +public class MarkingBinaryReaderParityTests : TestBase +{ + private readonly byte[] _testData; + + public MarkingBinaryReaderParityTests() + { + // Create test data with various patterns + _testData = new byte[256]; + for (int i = 0; i < 256; i++) + { + _testData[i] = (byte)i; + } + } + + [Fact] + public void Mark_Resets_ByteCount() + { + using var stream = new MemoryStream(_testData); + using var reader = new MarkingBinaryReader(stream); + + reader.ReadBytes(10); + Assert.Equal(10, reader.CurrentReadByteCount); + + reader.Mark(); + Assert.Equal(0, reader.CurrentReadByteCount); + + reader.ReadBytes(5); + Assert.Equal(5, reader.CurrentReadByteCount); + } + + [Fact] + public async Task Mark_Resets_ByteCount_Async() + { + using var stream = new MemoryStream(_testData); + var reader = new AsyncMarkingBinaryReader(stream); + + await reader.ReadBytesAsync(10); + Assert.Equal(10, reader.CurrentReadByteCount); + + reader.Mark(); + Assert.Equal(0, reader.CurrentReadByteCount); + + await reader.ReadBytesAsync(5); + Assert.Equal(5, reader.CurrentReadByteCount); + } + + [Fact] + public void ReadByte_Updates_ByteCount() + { + using var stream = new MemoryStream(_testData); + using var reader = new MarkingBinaryReader(stream); + + reader.Mark(); + reader.ReadByte(); + Assert.Equal(1, reader.CurrentReadByteCount); + + reader.ReadByte(); + Assert.Equal(2, reader.CurrentReadByteCount); + } + + [Fact] + public async Task ReadByte_Updates_ByteCount_Async() + { + using var stream = new MemoryStream(_testData); + var reader = new AsyncMarkingBinaryReader(stream); + + reader.Mark(); + await reader.ReadByteAsync(); + Assert.Equal(1, reader.CurrentReadByteCount); + + await reader.ReadByteAsync(); + Assert.Equal(2, reader.CurrentReadByteCount); + } + + [Fact] + public void ReadBytes_Updates_ByteCount() + { + using var stream = new MemoryStream(_testData); + using var reader = new MarkingBinaryReader(stream); + + reader.Mark(); + reader.ReadBytes(16); + Assert.Equal(16, reader.CurrentReadByteCount); + + reader.ReadBytes(8); + Assert.Equal(24, reader.CurrentReadByteCount); + } + + [Fact] + public async Task ReadBytes_Updates_ByteCount_Async() + { + using var stream = new MemoryStream(_testData); + var reader = new AsyncMarkingBinaryReader(stream); + + reader.Mark(); + await reader.ReadBytesAsync(16); + Assert.Equal(16, reader.CurrentReadByteCount); + + await reader.ReadBytesAsync(8); + Assert.Equal(24, reader.CurrentReadByteCount); + } + + [Fact] + public void ReadUInt16_Updates_ByteCount() + { + using var stream = new MemoryStream(_testData); + using var reader = new MarkingBinaryReader(stream); + + reader.Mark(); + reader.ReadUInt16(); + Assert.Equal(2, reader.CurrentReadByteCount); + } + + [Fact] + public async Task ReadUInt16_Updates_ByteCount_Async() + { + using var stream = new MemoryStream(_testData); + var reader = new AsyncMarkingBinaryReader(stream); + + reader.Mark(); + await reader.ReadUInt16Async(); + Assert.Equal(2, reader.CurrentReadByteCount); + } + + [Fact] + public void ReadUInt32_Updates_ByteCount() + { + using var stream = new MemoryStream(_testData); + using var reader = new MarkingBinaryReader(stream); + + reader.Mark(); + reader.ReadUInt32(); + Assert.Equal(4, reader.CurrentReadByteCount); + } + + [Fact] + public async Task ReadUInt32_Updates_ByteCount_Async() + { + using var stream = new MemoryStream(_testData); + var reader = new AsyncMarkingBinaryReader(stream); + + reader.Mark(); + await reader.ReadUInt32Async(); + Assert.Equal(4, reader.CurrentReadByteCount); + } + + [Fact] + public void ReadRarVInt_Updates_ByteCount() + { + // Create valid RAR v-int data: 0x05 (value 5, no continuation bit) + var data = new byte[] { 0x05, 0x85, 0x01, 0x00 }; // 0x05, then 0x85 0x01 (value 5 + 128 = 133) + using var stream = new MemoryStream(data); + using var reader = new MarkingBinaryReader(stream); + + reader.Mark(); + // Read a single-byte v-int (value 5, no continuation bit) + reader.ReadRarVInt(); + Assert.Equal(1, reader.CurrentReadByteCount); + + reader.Mark(); + // Read a two-byte v-int (0x85 means continuation, then 0x01) + reader.ReadRarVInt(); + Assert.Equal(2, reader.CurrentReadByteCount); + } + + [Fact] + public async Task ReadRarVInt_Updates_ByteCount_Async() + { + // Create valid RAR v-int data: 0x05 (value 5, no continuation bit) + var data = new byte[] { 0x05, 0x85, 0x01, 0x00 }; + using var stream = new MemoryStream(data); + var reader = new AsyncMarkingBinaryReader(stream); + + reader.Mark(); + // Read a single-byte v-int (value 5, no continuation bit) + await reader.ReadRarVIntAsync(); + Assert.Equal(1, reader.CurrentReadByteCount); + + reader.Mark(); + // Read a two-byte v-int (0x85 means continuation, then 0x01) + await reader.ReadRarVIntAsync(); + Assert.Equal(2, reader.CurrentReadByteCount); + } + + [Fact] + public async Task Sync_Async_ByteCount_Parity() + { + using var syncStream = new MemoryStream(_testData); + using var asyncStream = new MemoryStream(_testData); + using var syncReader = new MarkingBinaryReader(syncStream); + var asyncReader = new AsyncMarkingBinaryReader(asyncStream); + + syncReader.Mark(); + asyncReader.Mark(); + + // Read bytes with sync + syncReader.ReadByte(); + syncReader.ReadByte(); + syncReader.ReadUInt16(); + syncReader.ReadUInt32(); + syncReader.ReadBytes(8); + var syncCount = syncReader.CurrentReadByteCount; + + // Read bytes with async + await asyncReader.ReadByteAsync(); + await asyncReader.ReadByteAsync(); + await asyncReader.ReadUInt16Async(); + await asyncReader.ReadUInt32Async(); + await asyncReader.ReadBytesAsync(8); + var asyncCount = asyncReader.CurrentReadByteCount; + + Assert.Equal(syncCount, asyncCount); + Assert.Equal(16, syncCount); + Assert.Equal(16, asyncCount); + } + + [Fact] + public async Task Sync_Async_ByteCount_Parity_Alt() + { + using var syncStream = new MemoryStream(_testData); + using var asyncStream = new MemoryStream(_testData); + using var syncReader = new MarkingBinaryReader(syncStream); + var asyncReader = new AsyncMarkingBinaryReader(asyncStream); + + syncReader.Mark(); + asyncReader.Mark(); + + // Read bytes with sync + syncReader.ReadByte(); + syncReader.ReadByte(); + syncReader.ReadUInt16(); + syncReader.ReadUInt32(); + syncReader.ReadBytes(8); + var syncCount = syncReader.CurrentReadByteCount; + + // Read bytes with async + await asyncReader.ReadByteAsync(); + await asyncReader.ReadByteAsync(); + await asyncReader.ReadUInt16Async(); + await asyncReader.ReadUInt32Async(); + await asyncReader.ReadBytesAsync(8); + var asyncCount = asyncReader.CurrentReadByteCount; + + Assert.Equal(syncCount, asyncCount); + Assert.Equal(16, syncCount); + Assert.Equal(16, asyncCount); + } +} diff --git a/tests/SharpCompress.Test/Mocks/AsyncOnlyStream.cs b/tests/SharpCompress.Test/Mocks/AsyncOnlyStream.cs new file mode 100644 index 00000000..232f60ee --- /dev/null +++ b/tests/SharpCompress.Test/Mocks/AsyncOnlyStream.cs @@ -0,0 +1,80 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Test.Mocks; + +public class AsyncOnlyStream(Stream stream, bool disposeStream = true) : Stream +{ + private readonly Stream _stream = stream ?? throw new ArgumentNullException(nameof(stream)); + + public override bool CanRead => _stream.CanRead; + public override bool CanSeek => _stream.CanSeek; + public override bool CanWrite => _stream.CanWrite; + public override long Length => _stream.Length; + public override long Position + { + get => _stream.Position; + set => _stream.Position = value; + } + + public override Task FlushAsync(CancellationToken cancellationToken) => + _stream.FlushAsync(cancellationToken); + + public override void Flush() => + throw new NotSupportedException("Synchronous Flush is not supported"); + + public override int ReadByte() => + throw new NotSupportedException("Synchronous ReadByte is not supported"); + + public override int Read(byte[] buffer, int offset, int count) => + throw new NotSupportedException("Synchronous Read is not supported"); + + public override Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) => _stream.ReadAsync(buffer, offset, count, cancellationToken); + +#if NET8_0_OR_GREATER + public override ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) => _stream.ReadAsync(buffer, cancellationToken); +#endif + + public override long Seek(long offset, SeekOrigin origin) => _stream.Seek(offset, origin); + + public override void SetLength(long value) => _stream.SetLength(value); + + public override Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) => _stream.WriteAsync(buffer, offset, count, cancellationToken); + +#if NET8_0_OR_GREATER + public override ValueTask WriteAsync( + ReadOnlyMemory buffer, + CancellationToken cancellationToken = default + ) => _stream.WriteAsync(buffer, cancellationToken); +#endif + + public override void Write(byte[] buffer, int offset, int count) => + throw new NotSupportedException("Synchronous Write is not supported"); + + public override void WriteByte(byte value) => + throw new NotSupportedException("Synchronous WriteByte is not supported"); + + protected override void Dispose(bool disposing) + { + if (disposing && disposeStream) + { + _stream.Dispose(); + } + base.Dispose(disposing); + } +} diff --git a/tests/SharpCompress.Test/Mocks/FlushOnDisposeStream.cs b/tests/SharpCompress.Test/Mocks/FlushOnDisposeStream.cs index 194e7788..63f20702 100644 --- a/tests/SharpCompress.Test/Mocks/FlushOnDisposeStream.cs +++ b/tests/SharpCompress.Test/Mocks/FlushOnDisposeStream.cs @@ -1,5 +1,6 @@ -using System; +using System; using System.IO; +using System.Threading.Tasks; namespace SharpCompress.Test.Mocks; @@ -7,30 +8,26 @@ namespace SharpCompress.Test.Mocks; // CryptoStream doesn't always trigger the Flush, so this class is used instead // See https://referencesource.microsoft.com/#mscorlib/system/security/cryptography/cryptostream.cs,141 -public class FlushOnDisposeStream : Stream, IDisposable +public class FlushOnDisposeStream(Stream innerStream) : Stream { - private Stream inner; - - public FlushOnDisposeStream(Stream innerStream) => inner = innerStream; - - public override bool CanRead => inner.CanRead; + public override bool CanRead => innerStream.CanRead; public override bool CanSeek => false; public override bool CanWrite => false; - public override long Length => inner.Length; + public override long Length => innerStream.Length; public override long Position { - get => inner.Position; - set => inner.Position = value; + get => innerStream.Position; + set => innerStream.Position = value; } - public override void Flush() => throw new NotImplementedException(); + public override void Flush() { } public override int Read(byte[] buffer, int offset, int count) => - inner.Read(buffer, offset, count); + innerStream.Read(buffer, offset, count); public override long Seek(long offset, SeekOrigin origin) => throw new NotImplementedException(); @@ -44,10 +41,19 @@ public class FlushOnDisposeStream : Stream, IDisposable { if (disposing) { - inner.Flush(); - inner.Close(); + innerStream.Flush(); + innerStream.Close(); } base.Dispose(disposing); } + +#if !LEGACY_DOTNET + public override async ValueTask DisposeAsync() + { + await innerStream.FlushAsync(); + innerStream.Close(); + await base.DisposeAsync(); + } +#endif } diff --git a/tests/SharpCompress.Test/Mocks/ForwardOnlyStream.cs b/tests/SharpCompress.Test/Mocks/ForwardOnlyStream.cs index cef93a38..32961ebc 100644 --- a/tests/SharpCompress.Test/Mocks/ForwardOnlyStream.cs +++ b/tests/SharpCompress.Test/Mocks/ForwardOnlyStream.cs @@ -1,51 +1,164 @@ -using System; +using System; using System.IO; +using System.Threading; +using System.Threading.Tasks; namespace SharpCompress.Test.Mocks; +/// +/// A forward-only stream wrapper that delegates directly to the underlying stream +/// without any buffering. Supports reading and writing but not seeking. +/// public class ForwardOnlyStream : Stream { - private readonly Stream stream; + private readonly Stream _stream; + private bool _isDisposed; - public bool IsDisposed { get; private set; } - - public ForwardOnlyStream(Stream stream) => this.stream = stream; - - protected override void Dispose(bool disposing) + /// + /// Initializes a new instance of the class. + /// + /// The underlying stream to wrap. + /// Buffer size parameter (ignored - this implementation does not buffer). + /// Thrown when is null. + public ForwardOnlyStream(Stream stream, int? bufferSize = null) { - if (!IsDisposed) - { - if (disposing) - { - stream.Dispose(); - IsDisposed = true; - base.Dispose(disposing); - } - } + _stream = stream ?? throw new ArgumentNullException(nameof(stream)); + // bufferSize is ignored - this implementation does not buffer } - public override bool CanRead => true; + public override bool CanRead => !_isDisposed && _stream.CanRead; public override bool CanSeek => false; - public override bool CanWrite => false; - public override void Flush() => throw new NotSupportedException(); + public override bool CanWrite => !_isDisposed && _stream.CanWrite; - public override long Length => throw new NotSupportedException(); + public override long Length + { + get => throw new NotSupportedException("Length is not supported on a forward-only stream."); + } public override long Position { - get => throw new NotSupportedException(); - set => throw new NotSupportedException(); + get => + throw new NotSupportedException("Position is not supported on a forward-only stream."); + set => + throw new NotSupportedException("Position is not supported on a forward-only stream."); } - public override int Read(byte[] buffer, int offset, int count) => - stream.Read(buffer, offset, count); + public override void Flush() + { + ThrowIfDisposed(); + _stream.Flush(); + } - public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); + public override int Read(byte[] buffer, int offset, int count) + { + ThrowIfDisposed(); + return _stream.Read(buffer, offset, count); + } - public override void SetLength(long value) => throw new NotSupportedException(); + public override Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + ThrowIfDisposed(); + return _stream.ReadAsync(buffer, offset, count, cancellationToken); + } - public override void Write(byte[] buffer, int offset, int count) => - throw new NotSupportedException(); +#if !LEGACY_DOTNET + public override ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) + { + ThrowIfDisposed(); + return _stream.ReadAsync(buffer, cancellationToken); + } +#endif + + public override long Seek(long offset, SeekOrigin origin) => + throw new NotSupportedException("Seek is not supported on a forward-only stream."); + + public override void SetLength(long value) => + throw new NotSupportedException("SetLength is not supported on a forward-only stream."); + + public override void Write(byte[] buffer, int offset, int count) + { + ThrowIfDisposed(); + _stream.Write(buffer, offset, count); + } + + public override Task WriteAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) + { + ThrowIfDisposed(); + return _stream.WriteAsync(buffer, offset, count, cancellationToken); + } + +#if !LEGACY_DOTNET + public override ValueTask WriteAsync( + ReadOnlyMemory buffer, + CancellationToken cancellationToken = default + ) + { + ThrowIfDisposed(); + return _stream.WriteAsync(buffer, cancellationToken); + } +#endif + + public override Task FlushAsync(CancellationToken cancellationToken) + { + ThrowIfDisposed(); + return _stream.FlushAsync(cancellationToken); + } + + public override Task CopyToAsync( + Stream destination, + int bufferSize, + CancellationToken cancellationToken + ) + { + ThrowIfDisposed(); + return _stream.CopyToAsync(destination, bufferSize, cancellationToken); + } + + protected override void Dispose(bool disposing) + { + if (!_isDisposed) + { + if (disposing) + { + _stream.Dispose(); + } + _isDisposed = true; + base.Dispose(disposing); + } + } + +#if !LEGACY_DOTNET + public override async ValueTask DisposeAsync() + { + if (!_isDisposed) + { + await _stream.DisposeAsync(); + _isDisposed = true; + } + await base.DisposeAsync(); + } +#endif + + private void ThrowIfDisposed() + { + if (_isDisposed) + { + throw new ObjectDisposedException(nameof(ForwardOnlyStream)); + } + } } diff --git a/tests/SharpCompress.Test/Mocks/TestStream.cs b/tests/SharpCompress.Test/Mocks/TestStream.cs index 66923df3..d3ff48a8 100644 --- a/tests/SharpCompress.Test/Mocks/TestStream.cs +++ b/tests/SharpCompress.Test/Mocks/TestStream.cs @@ -1,24 +1,17 @@ -using System.IO; +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; namespace SharpCompress.Test.Mocks; -public class TestStream : Stream +public class TestStream(Stream stream, bool read, bool write, bool seek) : Stream { - private readonly Stream stream; - public TestStream(Stream stream) : this(stream, stream.CanRead, stream.CanWrite, stream.CanSeek) { } public bool IsDisposed { get; private set; } - public TestStream(Stream stream, bool read, bool write, bool seek) - { - this.stream = stream; - CanRead = read; - CanWrite = write; - CanSeek = seek; - } - protected override void Dispose(bool disposing) { base.Dispose(disposing); @@ -26,11 +19,11 @@ public class TestStream : Stream IsDisposed = true; } - public override bool CanRead { get; } + public override bool CanRead { get; } = read; - public override bool CanSeek { get; } + public override bool CanSeek { get; } = seek; - public override bool CanWrite { get; } + public override bool CanWrite { get; } = write; public override void Flush() => stream.Flush(); @@ -45,6 +38,27 @@ public class TestStream : Stream public override int Read(byte[] buffer, int offset, int count) => stream.Read(buffer, offset, count); + public override Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) => stream.ReadAsync(buffer, offset, count, cancellationToken); + +#if !LEGACY_DOTNET + public override ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) => stream.ReadAsync(buffer, cancellationToken); + + public override async ValueTask DisposeAsync() + { + await base.DisposeAsync(); + await stream.DisposeAsync(); + IsDisposed = true; + } +#endif + public override long Seek(long offset, SeekOrigin origin) => stream.Seek(offset, origin); public override void SetLength(long value) => stream.SetLength(value); diff --git a/tests/SharpCompress.Test/Mocks/ThrowOnFlushStream.cs b/tests/SharpCompress.Test/Mocks/ThrowOnFlushStream.cs new file mode 100644 index 00000000..2cf1a84c --- /dev/null +++ b/tests/SharpCompress.Test/Mocks/ThrowOnFlushStream.cs @@ -0,0 +1,73 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace SharpCompress.Test.Mocks; + +/// +/// A stream wrapper that throws NotSupportedException on Flush() calls. +/// This is used to test that archive iteration handles streams that don't support flushing. +/// +public class ThrowOnFlushStream : Stream +{ + private readonly Stream inner; + + public ThrowOnFlushStream(Stream inner) + { + this.inner = inner; + } + + public override bool CanRead => inner.CanRead; + + public override bool CanSeek => false; + + public override bool CanWrite => false; + + public override long Length => throw new NotSupportedException(); + + public override long Position + { + get => throw new NotSupportedException(); + set => throw new NotSupportedException(); + } + + public override void Flush() => throw new NotSupportedException("Flush not supported"); + + public override Task FlushAsync(CancellationToken cancellationToken) => + throw new NotSupportedException("FlushAsync not supported"); + + public override int Read(byte[] buffer, int offset, int count) => + inner.Read(buffer, offset, count); + + public override Task ReadAsync( + byte[] buffer, + int offset, + int count, + CancellationToken cancellationToken + ) => inner.ReadAsync(buffer, offset, count, cancellationToken); + +#if !NETFRAMEWORK && !NETSTANDARD2_0 + public override ValueTask ReadAsync( + Memory buffer, + CancellationToken cancellationToken = default + ) => inner.ReadAsync(buffer, cancellationToken); +#endif + + public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); + + public override void SetLength(long value) => throw new NotSupportedException(); + + public override void Write(byte[] buffer, int offset, int count) => + throw new NotSupportedException(); + + protected override void Dispose(bool disposing) + { + if (disposing) + { + inner.Dispose(); + } + + base.Dispose(disposing); + } +} diff --git a/tests/SharpCompress.Test/Mocks/TruncatedStream.cs b/tests/SharpCompress.Test/Mocks/TruncatedStream.cs new file mode 100644 index 00000000..8699a430 --- /dev/null +++ b/tests/SharpCompress.Test/Mocks/TruncatedStream.cs @@ -0,0 +1,65 @@ +using System; +using System.IO; + +namespace SharpCompress.Test.Mocks; + +/// +/// A stream wrapper that truncates the underlying stream after reading a specified number of bytes. +/// Used for testing error handling when streams end prematurely. +/// +public class TruncatedStream : Stream +{ + private readonly Stream baseStream; + private readonly long truncateAfterBytes; + private long bytesRead; + + public TruncatedStream(Stream baseStream, long truncateAfterBytes) + { + this.baseStream = baseStream ?? throw new ArgumentNullException(nameof(baseStream)); + this.truncateAfterBytes = truncateAfterBytes; + bytesRead = 0; + } + + public override bool CanRead => baseStream.CanRead; + public override bool CanSeek => baseStream.CanSeek; + public override bool CanWrite => false; + public override long Length => baseStream.Length; + + public override long Position + { + get => baseStream.Position; + set => baseStream.Position = value; + } + + public override int Read(byte[] buffer, int offset, int count) + { + if (bytesRead >= truncateAfterBytes) + { + // Simulate premature end of stream + return 0; + } + + var maxBytesToRead = (int)Math.Min(count, truncateAfterBytes - bytesRead); + var actualBytesRead = baseStream.Read(buffer, offset, maxBytesToRead); + bytesRead += actualBytesRead; + return actualBytesRead; + } + + public override long Seek(long offset, SeekOrigin origin) => baseStream.Seek(offset, origin); + + public override void SetLength(long value) => throw new NotSupportedException(); + + public override void Write(byte[] buffer, int offset, int count) => + throw new NotSupportedException(); + + public override void Flush() => baseStream.Flush(); + + protected override void Dispose(bool disposing) + { + if (disposing) + { + baseStream?.Dispose(); + } + base.Dispose(disposing); + } +} diff --git a/tests/SharpCompress.Test/OperatingSystemExtensions.cs b/tests/SharpCompress.Test/OperatingSystemExtensions.cs new file mode 100644 index 00000000..c8ca768e --- /dev/null +++ b/tests/SharpCompress.Test/OperatingSystemExtensions.cs @@ -0,0 +1,11 @@ +using System; + +namespace SharpCompress.Test; + +public static class OperatingSystemExtensions +{ + public static bool IsWindows(this OperatingSystem os) => + os.Platform == PlatformID.Win32NT + || os.Platform == PlatformID.Win32Windows + || os.Platform == PlatformID.Win32S; +} diff --git a/tests/SharpCompress.Test/OptionsUsabilityTests.cs b/tests/SharpCompress.Test/OptionsUsabilityTests.cs new file mode 100644 index 00000000..1902e4ab --- /dev/null +++ b/tests/SharpCompress.Test/OptionsUsabilityTests.cs @@ -0,0 +1,548 @@ +#if !LEGACY_DOTNET +using System; +using System.IO; +using System.Linq; +using System.Reflection; +using System.Runtime.CompilerServices; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Archives; +using SharpCompress.Common; +using SharpCompress.Readers; +using SharpCompress.Test.Mocks; +using SharpCompress.Writers; +using SharpCompress.Writers.GZip; +using SharpCompress.Writers.Tar; +using SharpCompress.Writers.Zip; +using Xunit; + +namespace SharpCompress.Test; + +public class OptionsUsabilityTests : TestBase +{ + [Fact] + public void ReaderFactory_Stream_Default_Leaves_Stream_Open() + { + using var file = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.zip")); + using var testStream = new TestStream(file); + + using (var reader = ReaderFactory.OpenReader(testStream)) + { + reader.MoveToNextEntry(); + } + + Assert.False(testStream.IsDisposed); + } + + [Fact] + public void ArchiveFactory_Stream_Default_Leaves_Stream_Open() + { + using var file = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.zip")); + using var testStream = new TestStream(file); + + using (var archive = ArchiveFactory.OpenArchive(testStream)) + { + _ = archive.Entries; + } + + Assert.False(testStream.IsDisposed); + } + + [Fact] + public async Task ReaderFactory_Stream_Default_Leaves_Stream_Open_Async() + { + using var file = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.zip")); + using var testStream = new TestStream(file); + + await using ( + var reader = await ReaderFactory.OpenAsyncReader(new AsyncOnlyStream(testStream)) + ) + { + await reader.MoveToNextEntryAsync(); + } + + Assert.False(testStream.IsDisposed); + } + + [Fact] + public void WriterOptions_Invalid_CompressionLevels_Throw() + { + Assert.Throws(() => + new WriterOptions(CompressionType.Deflate, 10) + ); + Assert.Throws(() => + new WriterOptions(CompressionType.ZStandard, 0) + ); + Assert.Throws(() => + new WriterOptions(CompressionType.BZip2, 1) + ); + } + + [Fact] + public void ZipWriterOptions_Invalid_CompressionLevels_Throw() + { + Assert.Throws(() => + new ZipWriterOptions(CompressionType.Deflate, 10) + ); + Assert.Throws(() => + new ZipWriterOptions(CompressionType.ZStandard, 23) + ); + } + + [Fact] + public void GZipWriterOptions_Invalid_Settings_Throw() + { + Assert.Throws(() => new GZipWriterOptions(10)); + Assert.Throws(() => + new GZipWriterOptions { CompressionType = CompressionType.Deflate } + ); + } + + [Fact] + public void ZipWriterEntryOptions_Invalid_CompressionLevel_Throws() + { + using var destination = new MemoryStream(); + using var source = new MemoryStream(new byte[] { 1, 2, 3 }); + using var writer = new ZipWriter( + destination, + new ZipWriterOptions(CompressionType.Deflate) + ); + + var options = new ZipWriterEntryOptions { CompressionLevel = 11 }; + + Assert.Throws(() => + writer.Write("entry.bin", source, options) + ); + } + + [Fact] + public void WriterOptions_Factory_Methods_Create_Valid_Options() + { + // ForZip + var zipOptions = WriterOptions.ForZip(); + Assert.Equal(CompressionType.Deflate, zipOptions.CompressionType); + Assert.True(zipOptions.LeaveStreamOpen); + + // ForTar + var tarOptions = WriterOptions.ForTar(); + Assert.Equal(CompressionType.None, tarOptions.CompressionType); + + // ForGZip + var gzipOptions = WriterOptions.ForGZip(); + Assert.Equal(CompressionType.GZip, gzipOptions.CompressionType); + } + + [Fact] + public void WriterOptions_Fluent_Methods_Modify_Correctly() + { + var options = WriterOptions + .ForZip() + .WithLeaveStreamOpen(false) + .WithCompressionLevel(9) + .WithBufferSize(65536); + + Assert.Equal(CompressionType.Deflate, options.CompressionType); + Assert.Equal(9, options.CompressionLevel); + Assert.False(options.LeaveStreamOpen); + Assert.Equal(65536, options.BufferSize); + } + + [Fact] + public void WriterOptions_Factory_And_Fluent_Equivalent_To_Constructor() + { + // Factory + fluent approach + var factoryApproach = WriterOptions + .ForZip() + .WithLeaveStreamOpen(false) + .WithCompressionLevel(9); + + // Traditional constructor approach + var constructorApproach = new WriterOptions(CompressionType.Deflate) + { + CompressionLevel = 9, + LeaveStreamOpen = false, + }; + + Assert.Equal(factoryApproach.CompressionType, constructorApproach.CompressionType); + Assert.Equal(factoryApproach.CompressionLevel, constructorApproach.CompressionLevel); + Assert.Equal(factoryApproach.LeaveStreamOpen, constructorApproach.LeaveStreamOpen); + } + + [Fact] + public void WriterOptions_Default_BufferSize_Uses_Constants_BufferSize() + { + Assert.Equal(Constants.BufferSize, WriterOptions.ForZip().BufferSize); + Assert.Equal( + Constants.BufferSize, + new ZipWriterOptions(CompressionType.Deflate).BufferSize + ); + Assert.Equal( + Constants.BufferSize, + new TarWriterOptions(CompressionType.None, true).BufferSize + ); + Assert.Equal(Constants.BufferSize, new GZipWriterOptions().BufferSize); + } + + [Fact] + public void Format_WriterOptions_Copy_BufferSize() + { + var options = WriterOptions.ForZip().WithBufferSize(12345); + + Assert.Equal(12345, new ZipWriterOptions(options).BufferSize); + Assert.Equal(12345, new TarWriterOptions(options).BufferSize); + Assert.Equal(12345, new GZipWriterOptions(options).BufferSize); + } + + [Fact] + public void ZipWriter_Uses_WriterOptions_BufferSize() + { + using var source = new TrackingReadStream(new byte[100]); + using var destination = new MemoryStream(); + using var writer = new ZipWriter( + destination, + new ZipWriterOptions(CompressionType.None) { BufferSize = 17 } + ); + + writer.Write("buffer-size.txt", source, DateTime.Now); + + Assert.Equal(17, source.CopyBufferSize); + } + + [Fact] + public void GZipWriter_Uses_WriterOptions_BufferSize() + { + using var source = new TrackingReadStream(new byte[100]); + using var destination = new MemoryStream(); + using var writer = new GZipWriter(destination, new GZipWriterOptions { BufferSize = 19 }); + + writer.Write("buffer-size.txt", source, DateTime.Now); + + Assert.Equal(19, source.CopyBufferSize); + } + + [Fact] + public void TarWriter_Uses_WriterOptions_BufferSize_ForTransfer() + { + using var source = new MemoryStream(new byte[100]); + using var destination = new MemoryStream(); + using var writer = new TarWriter( + destination, + new TarWriterOptions(CompressionType.None, true) { BufferSize = 0 } + ); + + Assert.Throws(() => + writer.Write("buffer-size.txt", source, DateTime.Now) + ); + } + + [Fact] + public void ReaderOptions_Fluent_Methods_Modify_Correctly() + { + var options = ReaderOptions + .ForExternalStream.WithLeaveStreamOpen(false) + .WithPassword("secret") + .WithLookForHeader(true) + .WithBufferSize(65536); + + Assert.False(options.LeaveStreamOpen); + Assert.Equal("secret", options.Password); + Assert.True(options.LookForHeader); + Assert.Equal(65536, options.BufferSize); + } + + [Fact] + public void ReaderOptions_Fluent_And_Initializer_Equivalent() + { + // Fluent approach + var fluentApproach = ReaderOptions + .ForExternalStream.WithLeaveStreamOpen(false) + .WithPassword("secret") + .WithLookForHeader(true) + .WithBufferSize(65536) + .WithDisableCheckIncomplete(true); + + // Preset + with-expression approach + var initializerApproach = ReaderOptions.ForExternalStream with + { + LeaveStreamOpen = false, + Password = "secret", + LookForHeader = true, + BufferSize = 65536, + DisableCheckIncomplete = true, + }; + + Assert.Equal(fluentApproach.LeaveStreamOpen, initializerApproach.LeaveStreamOpen); + Assert.Equal(fluentApproach.Password, initializerApproach.Password); + Assert.Equal(fluentApproach.LookForHeader, initializerApproach.LookForHeader); + Assert.Equal(fluentApproach.BufferSize, initializerApproach.BufferSize); + Assert.Equal( + fluentApproach.DisableCheckIncomplete, + initializerApproach.DisableCheckIncomplete + ); + } + + [Fact] + public void ReaderOptions_Presets_Have_Correct_Defaults() + { + var external = ReaderOptions.ForExternalStream; + Assert.True(external.LeaveStreamOpen); + + var owned = ReaderOptions.ForFilePath; + Assert.False(owned.LeaveStreamOpen); + } + + [Fact] + public void ExtractionOptions_Presets_Have_Correct_Defaults() + { + var safe = ExtractionOptions.SafeExtract; + Assert.False(safe.Overwrite); + + var flat = ExtractionOptions.FlatExtract; + Assert.False(flat.ExtractFullPath); + Assert.True(flat.Overwrite); + + var preserveMetadata = ExtractionOptions.PreserveMetadata; + Assert.True(preserveMetadata.PreserveFileTime); + Assert.True(preserveMetadata.PreserveAttributes); + + Assert.Equal(Constants.BufferSize, new ExtractionOptions().BufferSize); + Assert.True(new ExtractionOptions().CheckCrc); + } + + [Fact] + public void Reader_WriteEntryToFile_Uses_ExtractionOptions_BufferSize() + { + using var reader = new TrackingReader(); + var destination = Path.Combine(SCRATCH_FILES_PATH, "reader-buffer-size.txt"); + + reader.WriteEntryToFile(destination, new ExtractionOptions { BufferSize = 11 }); + + Assert.Equal(11, reader.EntryStreamCopyBufferSize); + } + + [Fact] + public async Task Reader_WriteEntryToFileAsync_Uses_ExtractionOptions_BufferSize() + { + await using var reader = new TrackingReader(); + var destination = Path.Combine(SCRATCH_FILES_PATH, "reader-buffer-size-async.txt"); + + await reader.WriteEntryToFileAsync(destination, new ExtractionOptions { BufferSize = 13 }); + + Assert.Equal(13, reader.EntryStreamCopyBufferSize); + } + + [Fact] + public void Public_Api_Does_Not_Expose_CSharp_9_Required_Metadata() + { + var assembly = typeof(ReaderOptions).Assembly; + const string RequiredMemberAttributeName = + "System.Runtime.CompilerServices.RequiredMemberAttribute"; + const string SetsRequiredMembersAttributeName = + "System.Diagnostics.CodeAnalysis.SetsRequiredMembersAttribute"; + var initOnlyProperties = assembly + .GetExportedTypes() + .SelectMany(type => + type.GetProperties(BindingFlags.Instance | BindingFlags.Public) + .Where(property => property.SetMethod?.IsPublic == true) + .Where(property => + property + .SetMethod!.ReturnParameter.GetRequiredCustomModifiers() + .Contains(typeof(IsExternalInit)) + ) + .Select(property => $"{type.FullName}.{property.Name}") + ) + .ToArray(); + + Assert.Empty(initOnlyProperties); + + var requiredMembers = assembly + .GetExportedTypes() + .SelectMany(type => + type.GetMembers(BindingFlags.Instance | BindingFlags.Static | BindingFlags.Public) + .Where(member => + member + .GetCustomAttributesData() + .Any(attribute => + attribute.AttributeType.FullName == RequiredMemberAttributeName + ) + ) + .Select(member => $"{type.FullName}.{member.Name}") + ) + .ToArray(); + + Assert.Empty(requiredMembers); + + var constructorsWithRequiredMembers = assembly + .GetExportedTypes() + .SelectMany(type => + type.GetConstructors(BindingFlags.Instance | BindingFlags.Public) + .Where(constructor => + constructor + .GetCustomAttributesData() + .Any(attribute => + attribute.AttributeType.FullName == SetsRequiredMembersAttributeName + ) + ) + .Select(_ => $"{type.FullName}.ctor") + ) + .ToArray(); + + Assert.Empty(constructorsWithRequiredMembers); + } + + [Fact] + public void ReaderOptions_Factory_ForEncryptedArchive_Sets_Password() + { + var options = ReaderOptions.ForEncryptedArchive("myPassword"); + Assert.Equal("myPassword", options.Password); + + var noPassword = ReaderOptions.ForEncryptedArchive(); + Assert.Null(noPassword.Password); + } + + [Fact] + public void ReaderOptions_Factory_ForEncoding_Sets_Encoding() + { + var encoding = new ArchiveEncoding { Default = System.Text.Encoding.UTF8 }; + var options = ReaderOptions.ForEncoding(encoding); + Assert.Equal(encoding, options.ArchiveEncoding); + } + + [Fact] + public void ReaderOptions_Factory_ForSelfExtractingArchive_Configures_Correctly() + { + var options = ReaderOptions.ForSelfExtractingArchive("password"); + Assert.True(options.LookForHeader); + Assert.Equal("password", options.Password); + Assert.Equal(1_048_576, options.RewindableBufferSize); + + var noPassword = ReaderOptions.ForSelfExtractingArchive(); + Assert.True(noPassword.LookForHeader); + Assert.Null(noPassword.Password); + Assert.Equal(1_048_576, noPassword.RewindableBufferSize); + } + + private sealed class TestArchiveEntry(Stream source) : IArchiveEntry + { + public CompressionType CompressionType => CompressionType.None; + public DateTime? ArchivedTime => null; + public long CompressedSize => source.Length; + public long Crc => 0; + public DateTime? CreatedTime => null; + public string? Key => "buffer-size.txt"; + public string? LinkTarget => null; + public bool IsDirectory => false; + public bool IsEncrypted => false; + public bool IsSplitAfter => false; + public bool IsSolid => false; + public int VolumeIndexFirst => 0; + public int VolumeIndexLast => 0; + public DateTime? LastAccessedTime => null; + public DateTime? LastModifiedTime => null; + public long Size => source.Length; + public int? Attrib => null; + public SharpCompress.Common.Options.IReaderOptions Options => + ReaderOptions.ForExternalStream; + public bool IsComplete => true; + public IArchive Archive => throw new NotSupportedException(); + + public Stream OpenEntryStream() + { + source.Position = 0; + return source; + } + + public ValueTask OpenEntryStreamAsync(CancellationToken cancellationToken = default) + { + source.Position = 0; + return new ValueTask(source); + } + } + + private sealed class TrackingReadStream(byte[] data) : MemoryStream(data) + { + public int? CopyBufferSize { get; private set; } + + public override void CopyTo(Stream destination, int bufferSize) + { + CopyBufferSize = bufferSize; + base.CopyTo(destination, bufferSize); + } + + public override Task CopyToAsync( + Stream destination, + int bufferSize, + CancellationToken cancellationToken + ) + { + CopyBufferSize = bufferSize; + return base.CopyToAsync(destination, bufferSize, cancellationToken); + } + } + + private sealed class TrackingReader : IReader, IAsyncReader + { + public ArchiveType Type => ArchiveType.Zip; + public MemoryStream Source { get; } = new(new byte[100]); + public IEntry Entry => new TestArchiveEntry(Source); + public bool Cancelled => false; + public int? EntryStreamCopyBufferSize { get; private set; } + + public void Dispose() { } + + public ValueTask DisposeAsync() => default; + + public void WriteEntryTo(Stream writableStream) => throw new NotSupportedException(); + + public ValueTask WriteEntryToAsync( + Stream writableStream, + CancellationToken cancellationToken = default + ) => throw new NotSupportedException(); + + public void Cancel() { } + + public bool MoveToNextEntry() => false; + + public ValueTask MoveToNextEntryAsync( + CancellationToken cancellationToken = default + ) => new(false); + + public EntryStream OpenEntryStream() + { + Source.Position = 0; + return new TrackingEntryStream( + this, + Source, + bufferSize => EntryStreamCopyBufferSize = bufferSize + ); + } + + public ValueTask OpenEntryStreamAsync( + CancellationToken cancellationToken = default + ) => new(OpenEntryStream()); + } + + private sealed class TrackingEntryStream( + IReader reader, + Stream stream, + Action copyBufferSize + ) : EntryStream(reader, stream) + { + public override void CopyTo(Stream destination, int bufferSize) + { + copyBufferSize(bufferSize); + base.CopyTo(destination, bufferSize); + } + + public override Task CopyToAsync( + Stream destination, + int bufferSize, + CancellationToken cancellationToken + ) + { + copyBufferSize(bufferSize); + return base.CopyToAsync(destination, bufferSize, cancellationToken); + } + } +} +#endif diff --git a/tests/SharpCompress.Test/ProgressReportTests.cs b/tests/SharpCompress.Test/ProgressReportTests.cs new file mode 100644 index 00000000..ff79a41c --- /dev/null +++ b/tests/SharpCompress.Test/ProgressReportTests.cs @@ -0,0 +1,612 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Archives; +using SharpCompress.Archives.Zip; +using SharpCompress.Common; +using SharpCompress.IO; +using SharpCompress.Readers; +using SharpCompress.Test.Mocks; +using SharpCompress.Writers; +using SharpCompress.Writers.Tar; +using SharpCompress.Writers.Zip; +using Xunit; + +namespace SharpCompress.Test; + +/// +/// A synchronous progress implementation for testing. +/// Unlike Progress<T>, this captures reports immediately without SynchronizationContext. +/// +internal sealed class TestProgress : IProgress +{ + private readonly List _reports = new(); + + public IReadOnlyList Reports => _reports; + + public void Report(T value) => _reports.Add(value); +} + +public class ProgressReportTests : TestBase +{ + private static byte[] CreateTestData(int size, byte fillValue) + { + var data = new byte[size]; + for (var i = 0; i < size; i++) + { + data[i] = fillValue; + } + return data; + } + + [Fact] + public void Zip_Write_ReportsProgress() + { + var progress = new TestProgress(); + + using var archiveStream = new MemoryStream(); + var options = new ZipWriterOptions(CompressionType.Deflate) { Progress = progress }; + + using (var writer = new ZipWriter(archiveStream, options)) + { + var testData = CreateTestData(10000, (byte)'A'); + using var sourceStream = new MemoryStream(testData); + writer.Write("test.txt", sourceStream, DateTime.Now); + } + + Assert.NotEmpty(progress.Reports); + Assert.All(progress.Reports, p => Assert.Equal("test.txt", p.EntryPath)); + Assert.All(progress.Reports, p => Assert.Equal(10000, p.TotalBytes)); + + var lastReport = progress.Reports[progress.Reports.Count - 1]; + Assert.Equal(10000, lastReport.BytesTransferred); + Assert.Equal(100.0, lastReport.PercentComplete); + } + + [Fact] + public void Tar_Write_ReportsProgress() + { + var progress = new TestProgress(); + + using var archiveStream = new MemoryStream(); + var options = new TarWriterOptions(CompressionType.None, true) { Progress = progress }; + + using (var writer = new TarWriter(archiveStream, options)) + { + var testData = CreateTestData(10000, (byte)'A'); + using var sourceStream = new MemoryStream(testData); + writer.Write("test.txt", sourceStream, DateTime.Now); + } + + Assert.NotEmpty(progress.Reports); + Assert.All(progress.Reports, p => Assert.Equal("test.txt", p.EntryPath)); + Assert.All(progress.Reports, p => Assert.Equal(10000, p.TotalBytes)); + + var lastReport = progress.Reports[progress.Reports.Count - 1]; + Assert.Equal(10000, lastReport.BytesTransferred); + Assert.Equal(100.0, lastReport.PercentComplete); + } + + [Fact] + public void Zip_Read_ReportsProgress() + { + var progress = new TestProgress(); + + // First create a zip archive + using var archiveStream = new MemoryStream(); + using ( + var writer = new ZipWriter(archiveStream, new ZipWriterOptions(CompressionType.Deflate)) + ) + { + var testData = CreateTestData(10000, (byte)'A'); + using var sourceStream = new MemoryStream(testData); + writer.Write("test.txt", sourceStream, DateTime.Now); + } + + // Now read it with progress reporting + archiveStream.Position = 0; + var readerOptions = ReaderOptions.ForExternalStream.WithProgress(progress); + + using (var reader = ReaderFactory.OpenReader(archiveStream, readerOptions)) + { + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + using var extractedStream = new MemoryStream(); + reader.WriteEntryTo(extractedStream); + } + } + } + + Assert.NotEmpty(progress.Reports); + Assert.All(progress.Reports, p => Assert.Equal("test.txt", p.EntryPath)); + + var lastReport = progress.Reports[progress.Reports.Count - 1]; + Assert.Equal(10000, lastReport.BytesTransferred); + } + + [Fact] + public void ZipArchive_Entry_WriteTo_ReportsProgress() + { + var progress = new TestProgress(); + + // First create a zip archive + using var archiveStream = new MemoryStream(); + using ( + var writer = new ZipWriter(archiveStream, new ZipWriterOptions(CompressionType.Deflate)) + ) + { + var testData = CreateTestData(10000, (byte)'A'); + using var sourceStream = new MemoryStream(testData); + writer.Write("test.txt", sourceStream, DateTime.Now); + } + + // Now open as archive and extract entry with progress as parameter + archiveStream.Position = 0; + + using var archive = ZipArchive.OpenArchive(archiveStream); + foreach (var entry in archive.Entries) + { + if (!entry.IsDirectory) + { + using var extractedStream = new MemoryStream(); + entry.WriteTo(extractedStream, progress); + } + } + + Assert.NotEmpty(progress.Reports); + Assert.All(progress.Reports, p => Assert.Equal("test.txt", p.EntryPath)); + + var lastReport = progress.Reports[progress.Reports.Count - 1]; + Assert.Equal(10000, lastReport.BytesTransferred); + } + + [Fact] + public async ValueTask ZipArchive_Entry_WriteToAsync_ReportsProgress() + { + var progress = new TestProgress(); + + // First create a zip archive + using var archiveStream = new MemoryStream(); + using ( + var writer = new ZipWriter(archiveStream, new ZipWriterOptions(CompressionType.Deflate)) + ) + { + var testData = CreateTestData(10000, (byte)'A'); + using var sourceStream = new MemoryStream(testData); + writer.Write("test.txt", sourceStream, DateTime.Now); + } + + // Now open as archive and extract entry async with progress as parameter + archiveStream.Position = 0; + + using var archive = ZipArchive.OpenArchive(archiveStream); + foreach (var entry in archive.Entries) + { + if (!entry.IsDirectory) + { + using var extractedStream = new MemoryStream(); + await entry.WriteToAsync(extractedStream, progress, CancellationToken.None); + } + } + + Assert.NotEmpty(progress.Reports); + Assert.All(progress.Reports, p => Assert.Equal("test.txt", p.EntryPath)); + + var lastReport = progress.Reports[progress.Reports.Count - 1]; + Assert.Equal(10000, lastReport.BytesTransferred); + } + + [Fact] + public void WriterOptions_WithoutProgress_DoesNotThrow() + { + using var archiveStream = new MemoryStream(); + var options = new ZipWriterOptions(CompressionType.Deflate); + Assert.Null(options.Progress); + + using (var writer = new ZipWriter(archiveStream, options)) + { + var testData = CreateTestData(100, (byte)'A'); + using var sourceStream = new MemoryStream(testData); + writer.Write("test.txt", sourceStream, DateTime.Now); + } + + Assert.True(archiveStream.Length > 0); + } + + [Fact] + public void ReaderOptions_WithoutProgress_DoesNotThrow() + { + // First create a zip archive + using var archiveStream = new MemoryStream(); + using ( + var writer = new ZipWriter(archiveStream, new ZipWriterOptions(CompressionType.Deflate)) + ) + { + var testData = CreateTestData(100, (byte)'A'); + using var sourceStream = new MemoryStream(testData); + writer.Write("test.txt", sourceStream, DateTime.Now); + } + + // Read without progress + archiveStream.Position = 0; + var readerOptions = ReaderOptions.ForExternalStream; + Assert.Null(readerOptions.Progress); + + using (var reader = ReaderFactory.OpenReader(archiveStream, readerOptions)) + { + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + using var extractedStream = new MemoryStream(); + reader.WriteEntryTo(extractedStream); + } + } + } + } + + [Fact] + public void ZipArchive_WithoutProgress_DoesNotThrow() + { + // First create a zip archive + using var archiveStream = new MemoryStream(); + using ( + var writer = new ZipWriter(archiveStream, new ZipWriterOptions(CompressionType.Deflate)) + ) + { + var testData = CreateTestData(100, (byte)'A'); + using var sourceStream = new MemoryStream(testData); + writer.Write("test.txt", sourceStream, DateTime.Now); + } + + // Open archive and extract without progress + archiveStream.Position = 0; + + using var archive = ZipArchive.OpenArchive(archiveStream); + foreach (var entry in archive.Entries) + { + if (!entry.IsDirectory) + { + using var extractedStream = new MemoryStream(); + entry.WriteTo(extractedStream); + } + } + } + + [Fact] + public void ProgressReport_PercentComplete_WithUnknownTotalBytes_ReturnsNull() + { + var progress = new ProgressReport("test.txt", 100, null); + Assert.Null(progress.PercentComplete); + } + + [Fact] + public void ProgressReport_PercentComplete_WithZeroTotalBytes_ReturnsNull() + { + var progress = new ProgressReport("test.txt", 0, 0); + Assert.Null(progress.PercentComplete); + } + + [Fact] + public void ProgressReport_Properties_AreSetCorrectly() + { + var progress = new ProgressReport("path/to/file.txt", 500, 1000); + + Assert.Equal("path/to/file.txt", progress.EntryPath); + Assert.Equal(500, progress.BytesTransferred); + Assert.Equal(1000, progress.TotalBytes); + Assert.Equal(50.0, progress.PercentComplete); + } + + [Fact] + public void Tar_Read_ReportsProgress() + { + var progress = new TestProgress(); + + // Create a tar archive first + using var archiveStream = new MemoryStream(); + using ( + var writer = new TarWriter( + archiveStream, + new TarWriterOptions(CompressionType.None, true) + ) + ) + { + var testData = CreateTestData(10000, (byte)'B'); + using var sourceStream = new MemoryStream(testData); + writer.Write("data.bin", sourceStream, DateTime.Now); + } + + // Now read it with progress reporting + archiveStream.Position = 0; + var readerOptions = ReaderOptions.ForExternalStream.WithProgress(progress); + + using (var reader = ReaderFactory.OpenReader(archiveStream, readerOptions)) + { + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + using var extractedStream = new MemoryStream(); + reader.WriteEntryTo(extractedStream); + } + } + } + + Assert.NotEmpty(progress.Reports); + Assert.All(progress.Reports, p => Assert.Equal("data.bin", p.EntryPath)); + + var lastReport = progress.Reports[progress.Reports.Count - 1]; + Assert.Equal(10000, lastReport.BytesTransferred); + } + + [Fact] + public void TarArchive_Entry_WriteTo_ReportsProgress() + { + var progress = new TestProgress(); + + // Create a tar archive first + using var archiveStream = new MemoryStream(); + using ( + var writer = new TarWriter( + archiveStream, + new TarWriterOptions(CompressionType.None, true) + ) + ) + { + var testData = CreateTestData(10000, (byte)'C'); + using var sourceStream = new MemoryStream(testData); + writer.Write("file.dat", sourceStream, DateTime.Now); + } + + // Now open as archive and extract entry with progress as parameter + archiveStream.Position = 0; + + using var archive = SharpCompress.Archives.Tar.TarArchive.OpenArchive(archiveStream); + foreach (var entry in archive.Entries) + { + if (!entry.IsDirectory) + { + using var extractedStream = new MemoryStream(); + entry.WriteTo(extractedStream, progress); + } + } + + Assert.NotEmpty(progress.Reports); + Assert.All(progress.Reports, p => Assert.Equal("file.dat", p.EntryPath)); + + var lastReport = progress.Reports[progress.Reports.Count - 1]; + Assert.Equal(10000, lastReport.BytesTransferred); + } + + [Fact] + public async ValueTask TarArchive_Entry_WriteToAsync_ReportsProgress() + { + var progress = new TestProgress(); + + // Create a tar archive first + using var archiveStream = new MemoryStream(); + using ( + var writer = new TarWriter( + archiveStream, + new TarWriterOptions(CompressionType.None, true) + ) + ) + { + var testData = CreateTestData(10000, (byte)'D'); + using var sourceStream = new MemoryStream(testData); + writer.Write("async.dat", sourceStream, DateTime.Now); + } + + // Now open as archive and extract entry async with progress as parameter + archiveStream.Position = 0; + + using var archive = SharpCompress.Archives.Tar.TarArchive.OpenArchive(archiveStream); + foreach (var entry in archive.Entries) + { + if (!entry.IsDirectory) + { + using var extractedStream = new MemoryStream(); + await entry.WriteToAsync(extractedStream, progress, CancellationToken.None); + } + } + + Assert.NotEmpty(progress.Reports); + Assert.All(progress.Reports, p => Assert.Equal("async.dat", p.EntryPath)); + + var lastReport = progress.Reports[progress.Reports.Count - 1]; + Assert.Equal(10000, lastReport.BytesTransferred); + } + + [Fact] + public void Zip_Read_MultipleEntries_ReportsProgress() + { + var progress = new TestProgress(); + + // Create a zip archive with multiple entries + using var archiveStream = new MemoryStream(); + using ( + var writer = new ZipWriter(archiveStream, new ZipWriterOptions(CompressionType.Deflate)) + ) + { + var testData1 = CreateTestData(5000, (byte)'A'); + using var sourceStream1 = new MemoryStream(testData1); + writer.Write("file1.txt", sourceStream1, DateTime.Now); + + var testData2 = CreateTestData(8000, (byte)'B'); + using var sourceStream2 = new MemoryStream(testData2); + writer.Write("file2.txt", sourceStream2, DateTime.Now); + } + + // Now read it with progress reporting + archiveStream.Position = 0; + var readerOptions = ReaderOptions.ForExternalStream.WithProgress(progress); + + using (var reader = ReaderFactory.OpenReader(archiveStream, readerOptions)) + { + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + using var extractedStream = new MemoryStream(); + reader.WriteEntryTo(extractedStream); + } + } + } + + Assert.NotEmpty(progress.Reports); + + // Should have reports for both files + var file1Reports = progress.Reports.Where(p => p.EntryPath == "file1.txt").ToList(); + var file2Reports = progress.Reports.Where(p => p.EntryPath == "file2.txt").ToList(); + + Assert.NotEmpty(file1Reports); + Assert.NotEmpty(file2Reports); + + // Verify final bytes for each file + Assert.Equal(5000, file1Reports[file1Reports.Count - 1].BytesTransferred); + Assert.Equal(8000, file2Reports[file2Reports.Count - 1].BytesTransferred); + } + + [Fact] + public void ZipArchive_MultipleEntries_WriteTo_ReportsProgress() + { + var progress = new TestProgress(); + + // Create a zip archive with multiple entries + using var archiveStream = new MemoryStream(); + using ( + var writer = new ZipWriter(archiveStream, new ZipWriterOptions(CompressionType.Deflate)) + ) + { + var testData1 = CreateTestData(5000, (byte)'A'); + using var sourceStream1 = new MemoryStream(testData1); + writer.Write("entry1.txt", sourceStream1, DateTime.Now); + + var testData2 = CreateTestData(7000, (byte)'B'); + using var sourceStream2 = new MemoryStream(testData2); + writer.Write("entry2.txt", sourceStream2, DateTime.Now); + } + + // Now open as archive and extract entries with progress as parameter + archiveStream.Position = 0; + + using var archive = ZipArchive.OpenArchive(archiveStream); + foreach (var entry in archive.Entries) + { + if (!entry.IsDirectory) + { + using var extractedStream = new MemoryStream(); + entry.WriteTo(extractedStream, progress); + } + } + + Assert.NotEmpty(progress.Reports); + + // Should have reports for both files + var entry1Reports = progress.Reports.Where(p => p.EntryPath == "entry1.txt").ToList(); + var entry2Reports = progress.Reports.Where(p => p.EntryPath == "entry2.txt").ToList(); + + Assert.NotEmpty(entry1Reports); + Assert.NotEmpty(entry2Reports); + + // Verify final bytes for each entry + Assert.Equal(5000, entry1Reports[entry1Reports.Count - 1].BytesTransferred); + Assert.Equal(7000, entry2Reports[entry2Reports.Count - 1].BytesTransferred); + } + + [Fact] + public async ValueTask Zip_ReadAsync_ReportsProgress() + { + var progress = new TestProgress(); + + // Create a zip archive + using var archiveStream = new MemoryStream(); + using ( + var writer = new ZipWriter(archiveStream, new ZipWriterOptions(CompressionType.Deflate)) + ) + { + var testData = CreateTestData(10000, (byte)'E'); + using var sourceStream = new MemoryStream(testData); + writer.Write("async_read.txt", sourceStream, DateTime.Now); + } + + // Now read it with progress reporting + archiveStream.Position = 0; + var readerOptions = ReaderOptions.ForExternalStream.WithProgress(progress); + + await using ( + var reader = await ReaderFactory.OpenAsyncReader( + new AsyncOnlyStream(archiveStream), + readerOptions + ) + ) + { + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + using var extractedStream = new MemoryStream(); + await reader.WriteEntryToAsync(extractedStream); + } + } + } + + Assert.NotEmpty(progress.Reports); + Assert.All(progress.Reports, p => Assert.Equal("async_read.txt", p.EntryPath)); + + var lastReport = progress.Reports[progress.Reports.Count - 1]; + Assert.Equal(10000, lastReport.BytesTransferred); + } + + [Fact] + public void GZip_Write_ReportsProgress() + { + var progress = new TestProgress(); + + using var archiveStream = new MemoryStream(); + var options = new SharpCompress.Writers.GZip.GZipWriterOptions { Progress = progress }; + + using (var writer = new SharpCompress.Writers.GZip.GZipWriter(archiveStream, options)) + { + var testData = CreateTestData(10000, (byte)'G'); + using var sourceStream = new MemoryStream(testData); + writer.Write("gzip_test.txt", sourceStream, DateTime.Now); + } + + Assert.NotEmpty(progress.Reports); + Assert.All(progress.Reports, p => Assert.Equal("gzip_test.txt", p.EntryPath)); + Assert.All(progress.Reports, p => Assert.Equal(10000, p.TotalBytes)); + + var lastReport = progress.Reports[progress.Reports.Count - 1]; + Assert.Equal(10000, lastReport.BytesTransferred); + Assert.Equal(100.0, lastReport.PercentComplete); + } + + [Fact] + public async ValueTask Tar_WriteAsync_ReportsProgress() + { + var progress = new TestProgress(); + + using var archiveStream = new MemoryStream(); + var options = new TarWriterOptions(CompressionType.None, true) { Progress = progress }; + + using (var writer = new TarWriter(archiveStream, options)) + { + var testData = CreateTestData(10000, (byte)'A'); + using var sourceStream = new MemoryStream(testData); + await writer.WriteAsync("test.txt", sourceStream, DateTime.Now); + } + + Assert.NotEmpty(progress.Reports); + Assert.All(progress.Reports, p => Assert.Equal("test.txt", p.EntryPath)); + + var lastReport = progress.Reports[progress.Reports.Count - 1]; + Assert.Equal(10000, lastReport.BytesTransferred); + } +} diff --git a/tests/SharpCompress.Test/Rar/RarArchiveAsyncTests.cs b/tests/SharpCompress.Test/Rar/RarArchiveAsyncTests.cs new file mode 100644 index 00000000..b3cd79bd --- /dev/null +++ b/tests/SharpCompress.Test/Rar/RarArchiveAsyncTests.cs @@ -0,0 +1,787 @@ +using System.IO; +using System.Linq; +using System.Threading.Tasks; +using SharpCompress.Archives; +using SharpCompress.Archives.Rar; +using SharpCompress.Common; +using SharpCompress.Compressors.LZMA.Utilities; +using SharpCompress.Readers; +using SharpCompress.Test.Mocks; +using Xunit; + +namespace SharpCompress.Test.Rar; + +public class RarArchiveAsyncTests : ArchiveTests +{ + [Theory] + [InlineData("Rar15.rar")] + [InlineData("Rar2.rar")] + [InlineData("Rar.rar")] + [InlineData("Rar.Audio_program.rar")] + [InlineData("Rar5.rar")] + [InlineData("Rar5.solid.rar")] + public async ValueTask Rar_Archive_Recently_Changed_Unpackers_Async(string filename) + { + var extractedEntries = 0; + await using var archive = await RarArchive.OpenAsyncArchive( + Path.Combine(TEST_ARCHIVES_PATH, filename), + new ReaderOptions { LookForHeader = true } + ); + + await foreach (var entry in archive.EntriesAsync.Where(entry => !entry.IsDirectory)) + { + using var output = new AsyncOnlyStream(new MemoryStream()); + await entry.WriteToAsync(output); + extractedEntries++; + } + + Assert.True(extractedEntries > 0); + } + + [Fact] + public async ValueTask Rar_EncryptedFileAndHeader_Archive_Async() => + await ReadRarPasswordAsync("Rar.encrypted_filesAndHeader.rar", "test"); + + [Fact] + public async ValueTask Rar_EncryptedFileAndHeader_NoPasswordExceptionTest_Async() => + await Assert.ThrowsAsync( + typeof(CryptographicException), + async () => await ReadRarPasswordAsync("Rar.encrypted_filesAndHeader.rar", null) + ); + + [Fact] + public async ValueTask Rar5_EncryptedFileAndHeader_Archive_Async() => + await ReadRarPasswordAsync("Rar5.encrypted_filesAndHeader.rar", "test"); + + [Fact] + public async ValueTask Rar5_EncryptedFileAndHeader_Archive_Err_Async() => + await Assert.ThrowsAsync( + typeof(CryptographicException), + async () => await ReadRarPasswordAsync("Rar5.encrypted_filesAndHeader.rar", "failed") + ); + + [Fact] + public async ValueTask Rar5_EncryptedFileAndHeader_NoPasswordExceptionTest_Async() => + await Assert.ThrowsAsync( + typeof(CryptographicException), + async () => await ReadRarPasswordAsync("Rar5.encrypted_filesAndHeader.rar", null) + ); + + [Fact] + public async ValueTask Rar_EncryptedFileOnly_Archive_Async() => + await ReadRarPasswordAsync("Rar.encrypted_filesOnly.rar", "test"); + + [Fact] + public async ValueTask Rar_EncryptedFileOnly_Archive_Err_Async() => + await Assert.ThrowsAsync( + typeof(CryptographicException), + async () => await ReadRarPasswordAsync("Rar5.encrypted_filesOnly.rar", "failed") + ); + + [Fact] + public async ValueTask Rar5_EncryptedFileOnly_Archive_Async() => + await ReadRarPasswordAsync("Rar5.encrypted_filesOnly.rar", "test"); + + [Fact] + public async ValueTask Rar_Encrypted_Archive_Async() => + await ReadRarPasswordAsync("Rar.Encrypted.rar", "test"); + + [Fact] + public async ValueTask Rar5_Encrypted_Archive_Async() => + await ReadRarPasswordAsync("Rar5.encrypted_filesAndHeader.rar", "test"); + + private async ValueTask ReadRarPasswordAsync(string testArchive, string? password) + { + using (Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, testArchive))) + await using ( + var archive = await RarArchive.OpenAsyncArchive( + stream, + ReaderOptions.ForExternalStream with + { + Password = password, + LeaveStreamOpen = true, + } + ) + ) + { + await foreach (var entry in archive.EntriesAsync) + { + if (!entry.IsDirectory) + { + Assert.Equal(CompressionType.Rar, entry.CompressionType); + await entry.WriteToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + } + VerifyFiles(); + } + + [Fact] + public async ValueTask Rar_Multi_Archive_Encrypted_Async() => + await Assert.ThrowsAsync(async () => + await ArchiveFileReadPasswordAsync("Rar.EncryptedParts.part01.rar", "test") + ); + + protected async Task ArchiveFileReadPasswordAsync(string archiveName, string password) + { + using ( + var archive = RarArchive.OpenArchive( + Path.Combine(TEST_ARCHIVES_PATH, archiveName), + ReaderOptions.ForFilePath with + { + Password = password, + LeaveStreamOpen = true, + } + ) + ) + { + foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) + { + await entry.WriteToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + VerifyFiles(); + } + + [Fact] + public async ValueTask Rar_None_ArchiveStreamRead_Async() => + await ArchiveStreamReadAsync("Rar.none.rar"); + + [Fact] + public async ValueTask Rar5_None_ArchiveStreamRead_Async() => + await ArchiveStreamReadAsync("Rar5.none.rar"); + + [Fact] + public async ValueTask Rar_ArchiveStreamRead_Async() => await ArchiveStreamReadAsync("Rar.rar"); + + [Fact] + public async ValueTask Rar5_ArchiveStreamRead_Async() => + await ArchiveStreamReadAsync("Rar5.rar"); + + [Fact] + public async ValueTask Rar_test_invalid_exttime_ArchiveStreamRead_Async() => + await DoRar_test_invalid_exttime_ArchiveStreamReadAsync("Rar.test_invalid_exttime.rar"); + + private async ValueTask DoRar_test_invalid_exttime_ArchiveStreamReadAsync(string filename) + { + using var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, filename)); + using var archive = ArchiveFactory.OpenArchive(stream); + foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) + { + await entry.WriteToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + + [Fact] + public async ValueTask Rar_Jpg_ArchiveStreamRead_Async() + { + using var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Rar.jpeg.jpg")); + using ( + var archive = RarArchive.OpenArchive( + stream, + ReaderOptions.ForExternalStream with + { + LookForHeader = true, + } + ) + ) + { + foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) + { + await entry.WriteToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + VerifyFiles(); + } + + [Fact] + public async ValueTask Rar_IsSolidArchiveCheck_Async() => + await DoRar_IsSolidArchiveCheckAsync("Rar.rar"); + + [Fact] + public async ValueTask Rar5_IsSolidArchiveCheck_Async() => + await DoRar_IsSolidArchiveCheckAsync("Rar5.rar"); + + private async ValueTask DoRar_IsSolidArchiveCheckAsync(string filename) + { + using (var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, filename))) + { + using var archive = RarArchive.OpenArchive(stream); + Assert.False(archive.IsSolid); + foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) + { + await entry.WriteToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + VerifyFiles(); + } + + [Fact] + public async ValueTask Rar_IsSolidEntryStreamCheck_Async() => + await DoRar_IsSolidEntryStreamCheckAsync("Rar.solid.rar"); + + private async ValueTask DoRar_IsSolidEntryStreamCheckAsync(string filename) + { + using var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, filename)); + using var archive = RarArchive.OpenArchive(stream); + Assert.True(archive.IsSolid); + IArchiveEntry[] entries = archive.Entries.Where(a => !a.IsDirectory).ToArray(); + Assert.NotInRange(entries.Length, 0, 1); + Assert.False(entries[0].IsSolid); + var testEntry = entries[1]; + Assert.True(testEntry.IsSolid); + + foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) + { + using (var crcStream = new CrcCheckStream((uint)entry.Crc)) + { + using var eStream = await entry.OpenEntryStreamAsync(); + await eStream.CopyToAsync(crcStream); + } + if (entry == testEntry) + { + break; + } + } + } + + [Fact] + public async ValueTask Rar_Solid_ArchiveStreamRead_Async() => + await ArchiveStreamReadAsync("Rar.solid.rar"); + + [Fact] + public async ValueTask Rar5_Solid_ArchiveStreamRead_Async() => + await ArchiveStreamReadAsync("Rar5.solid.rar"); + + [Fact] + public async ValueTask Rar_Solid_StreamRead_Extract_All_Async() => + await ArchiveStreamReadExtractAllAsync("Rar.solid.rar", CompressionType.Rar); + + [Fact] + public async ValueTask Rar5_Solid_StreamRead_Extract_All_Async() => + await ArchiveStreamReadExtractAllAsync("Rar5.solid.rar", CompressionType.Rar); + + [Fact] + public async ValueTask Rar_Multi_ArchiveStreamRead_Async() => + await DoRar_Multi_ArchiveStreamReadAsync( + [ + "Rar.multi.part01.rar", + "Rar.multi.part02.rar", + "Rar.multi.part03.rar", + "Rar.multi.part04.rar", + "Rar.multi.part05.rar", + "Rar.multi.part06.rar", + ], + false + ); + + [Fact] + public async ValueTask Rar5_Multi_ArchiveStreamRead_Async() => + await DoRar_Multi_ArchiveStreamReadAsync( + [ + "Rar5.multi.part01.rar", + "Rar5.multi.part02.rar", + "Rar5.multi.part03.rar", + "Rar5.multi.part04.rar", + "Rar5.multi.part05.rar", + "Rar5.multi.part06.rar", + ], + false + ); + + private async ValueTask DoRar_Multi_ArchiveStreamReadAsync(string[] archives, bool isSolid) + { + using var archive = RarArchive.OpenArchive( + archives + .Select(s => Path.Combine(TEST_ARCHIVES_PATH, s)) + .Select(File.OpenRead) + .ToArray() + ); + Assert.Equal(archive.IsSolid, isSolid); + foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) + { + await entry.WriteToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + + [Fact] + public async ValueTask Rar5_MultiSolid_ArchiveStreamRead_Async() => + await DoRar_Multi_ArchiveStreamReadAsync( + [ + "Rar.multi.solid.part01.rar", + "Rar.multi.solid.part02.rar", + "Rar.multi.solid.part03.rar", + "Rar.multi.solid.part04.rar", + "Rar.multi.solid.part05.rar", + "Rar.multi.solid.part06.rar", + ], + true + ); + + [Fact] + public async ValueTask RarNoneArchiveFileRead_Async() => + await ArchiveFileReadAsync("Rar.none.rar"); + + [Fact] + public async ValueTask Rar5NoneArchiveFileRead_Async() => + await ArchiveFileReadAsync("Rar5.none.rar"); + + [Fact] + public async ValueTask Rar_ArchiveFileRead_Async() => await ArchiveFileReadAsync("Rar.rar"); + + [Fact] + public async ValueTask Rar5_ArchiveFileRead_Async() => await ArchiveFileReadAsync("Rar5.rar"); + + [Fact] + public async ValueTask Rar_ArchiveFileRead_HasDirectories_Async() => + await DoRar_ArchiveFileRead_HasDirectoriesAsync("Rar.rar"); + + [Fact] + public async ValueTask Rar5_ArchiveFileRead_HasDirectories_Async() => + await DoRar_ArchiveFileRead_HasDirectoriesAsync("Rar5.rar"); + + private Task DoRar_ArchiveFileRead_HasDirectoriesAsync(string filename) + { + using var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, filename)); + using var archive = RarArchive.OpenArchive(stream); + Assert.False(archive.IsSolid); + Assert.Contains(true, archive.Entries.Select(entry => entry.IsDirectory)); + return Task.CompletedTask; + } + + [Fact] + public async ValueTask Rar_Jpg_ArchiveFileRead_Async() + { + using ( + var archive = RarArchive.OpenArchive( + Path.Combine(TEST_ARCHIVES_PATH, "Rar.jpeg.jpg"), + ReaderOptions.ForFilePath with + { + LookForHeader = true, + } + ) + ) + { + foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) + { + await entry.WriteToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + VerifyFiles(); + } + + [Fact] + public async ValueTask Rar_Solid_ArchiveFileRead_Async() => + await ArchiveFileReadAsync("Rar.solid.rar"); + + [Fact] + public async ValueTask Rar5_Solid_ArchiveFileRead_Async() => + await ArchiveFileReadAsync("Rar5.solid.rar"); + + [Fact] + public async ValueTask Rar2_Multi_ArchiveStreamRead_Async() => + await DoRar_Multi_ArchiveStreamReadAsync( + [ + "Rar2.multi.rar", + "Rar2.multi.r00", + "Rar2.multi.r01", + "Rar2.multi.r02", + "Rar2.multi.r03", + "Rar2.multi.r04", + "Rar2.multi.r05", + ], + false + ); + + [Fact] + public async ValueTask Rar2_Multi_ArchiveFileRead_Async() => + await ArchiveFileReadAsync("Rar2.multi.rar"); + + [Fact] + public async ValueTask Rar2_ArchiveFileRead_Async() => await ArchiveFileReadAsync("Rar2.rar"); + + [Fact] + public async ValueTask Rar15_ArchiveFileRead_Async() + { + UseExtensionInsteadOfNameToVerify = true; + UseCaseInsensitiveToVerify = true; + await ArchiveFileReadAsync("Rar15.rar"); + } + + [Fact] + public void Rar15_ArchiveVersionTest_Async() + { + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "Rar15.rar"); + + using var archive = RarArchive.OpenArchive(testArchive); + Assert.Equal(1, archive.MinVersion); + Assert.Equal(1, archive.MaxVersion); + } + + [Fact] + public void Rar2_ArchiveVersionTest_Async() + { + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "Rar2.rar"); + + using var archive = RarArchive.OpenArchive(testArchive); + Assert.Equal(2, archive.MinVersion); + Assert.Equal(2, archive.MaxVersion); + } + + [Fact] + public void Rar4_ArchiveVersionTest_Async() + { + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "Rar4.multi.part01.rar"); + + using var archive = RarArchive.OpenArchive(testArchive); + Assert.Equal(3, archive.MinVersion); + Assert.Equal(4, archive.MaxVersion); + } + + [Fact] + public void Rar5_ArchiveVersionTest_Async() + { + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "Rar5.solid.rar"); + + using var archive = RarArchive.OpenArchive(testArchive); + Assert.Equal(5, archive.MinVersion); + Assert.Equal(6, archive.MaxVersion); + } + + [Fact] + public async ValueTask Rar4_Multi_ArchiveFileRead_Async() => + await ArchiveFileReadAsync("Rar4.multi.part01.rar"); + + [Fact] + public async ValueTask Rar4_ArchiveFileRead_Async() => await ArchiveFileReadAsync("Rar4.rar"); + + [Fact] + public void Rar_GetPartsSplit_Async() => + ArchiveGetParts( + new[] + { + "Rar4.split.001", + "Rar4.split.002", + "Rar4.split.003", + "Rar4.split.004", + "Rar4.split.005", + "Rar4.split.006", + } + ); + + [Fact] + public void Rar_GetPartsOld_Async() => + ArchiveGetParts( + new[] + { + "Rar2.multi.rar", + "Rar2.multi.r00", + "Rar2.multi.r01", + "Rar2.multi.r02", + "Rar2.multi.r03", + "Rar2.multi.r04", + "Rar2.multi.r05", + } + ); + + [Fact] + public void Rar_GetPartsNew_Async() => + ArchiveGetParts( + new[] + { + "Rar4.multi.part01.rar", + "Rar4.multi.part02.rar", + "Rar4.multi.part03.rar", + "Rar4.multi.part04.rar", + "Rar4.multi.part05.rar", + "Rar4.multi.part06.rar", + "Rar4.multi.part07.rar", + } + ); + + [Fact] + public async ValueTask Rar4_Multi_ArchiveStreamRead_Async() => + await DoRar_Multi_ArchiveStreamReadAsync( + [ + "Rar4.multi.part01.rar", + "Rar4.multi.part02.rar", + "Rar4.multi.part03.rar", + "Rar4.multi.part04.rar", + "Rar4.multi.part05.rar", + "Rar4.multi.part06.rar", + "Rar4.multi.part07.rar", + ], + false + ); + + [Fact] + public async ValueTask Rar4_Split_ArchiveStreamRead_Async() => + await ArchiveStreamMultiReadAsync( + null, + [ + "Rar4.split.001", + "Rar4.split.002", + "Rar4.split.003", + "Rar4.split.004", + "Rar4.split.005", + "Rar4.split.006", + ] + ); + + [Fact] + public async ValueTask Rar4_Multi_ArchiveFirstFileRead_Async() => + await ArchiveFileReadAsync("Rar4.multi.part01.rar"); + + [Fact] + public async ValueTask Rar4_Split_ArchiveFirstFileRead_Async() => + await ArchiveFileReadAsync("Rar4.split.001"); + + [Fact] + public async ValueTask Rar4_Split_ArchiveStreamFirstFileRead_Async() => + await ArchiveStreamMultiReadAsync(null, ["Rar4.split.001"]); + + [Fact] + public async ValueTask Rar4_Split_ArchiveOpen_Async() => + await ArchiveOpenStreamReadAsync( + null, + "Rar4.split.001", + "Rar4.split.002", + "Rar4.split.003", + "Rar4.split.004", + "Rar4.split.005", + "Rar4.split.006" + ); + + [Fact] + public async ValueTask Rar4_Multi_ArchiveOpen_Async() => + await ArchiveOpenStreamReadAsync( + null, + "Rar4.multi.part01.rar", + "Rar4.multi.part02.rar", + "Rar4.multi.part03.rar", + "Rar4.multi.part04.rar", + "Rar4.multi.part05.rar", + "Rar4.multi.part06.rar", + "Rar4.multi.part07.rar" + ); + + [Fact] + public void Rar4_Multi_ArchiveOpenEntryVolumeIndexTest_Async() => + ArchiveOpenEntryVolumeIndexTest( + [ + [0, 1], + [1, 5], + [5, 6], + ], + null, + "Rar4.multi.part01.rar", + "Rar4.multi.part02.rar", + "Rar4.multi.part03.rar", + "Rar4.multi.part04.rar", + "Rar4.multi.part05.rar", + "Rar4.multi.part06.rar", + "Rar4.multi.part07.rar" + ); + + [Fact] + public async ValueTask Rar_Multi_ArchiveFileRead_Async() => + await ArchiveFileReadAsync("Rar.multi.part01.rar"); + + [Fact] + public async ValueTask Rar5_Multi_ArchiveFileRead_Async() => + await ArchiveFileReadAsync("Rar5.multi.part01.rar"); + + [Fact] + public void Rar_IsFirstVolume_True_Async() => DoRar_IsFirstVolume_True("Rar.multi.part01.rar"); + + [Fact] + public void Rar5_IsFirstVolume_True_Async() => + DoRar_IsFirstVolume_True("Rar5.multi.part01.rar"); + + private void DoRar_IsFirstVolume_True(string firstFilename) + { + using var archive = RarArchive.OpenArchive(Path.Combine(TEST_ARCHIVES_PATH, firstFilename)); + Assert.True(archive.IsMultipartVolume()); + Assert.True(archive.IsFirstVolume()); + } + + [Fact] + public void Rar_IsFirstVolume_False_Async() => + DoRar_IsFirstVolume_False("Rar.multi.part03.rar"); + + [Fact] + public void Rar5_IsFirstVolume_False_Async() => + DoRar_IsFirstVolume_False("Rar5.multi.part03.rar"); + + private void DoRar_IsFirstVolume_False(string notFirstFilename) + { + using var archive = RarArchive.OpenArchive( + Path.Combine(TEST_ARCHIVES_PATH, notFirstFilename) + ); + Assert.True(archive.IsMultipartVolume()); + Assert.False(archive.IsFirstVolume()); + } + + [Fact] + public async ValueTask Rar5_CRC_Blake2_Archive_Async() => + await ArchiveFileReadAsync("Rar5.crc_blake2.rar"); + + [Fact] + void Rar_Iterate_Archive_Async() => + ArchiveFileSkip("Rar.rar", "Failure jpg exe Empty jpg\\test.jpg exe\\test.exe тест.txt"); + + [Fact] + public void Rar2_Iterate_Archive_Async() => + ArchiveFileSkip("Rar2.rar", "Failure Empty тест.txt jpg\\test.jpg exe\\test.exe jpg exe"); + + [Fact] + public void Rar4_Iterate_Archive_Async() => + ArchiveFileSkip("Rar4.rar", "Failure Empty jpg exe тест.txt jpg\\test.jpg exe\\test.exe"); + + [Fact] + public void Rar5_Iterate_Archive_Async() => + ArchiveFileSkip("Rar5.rar", "Failure jpg exe Empty тест.txt jpg\\test.jpg exe\\test.exe"); + + [Fact] + public void Rar_Encrypted_Iterate_Archive_Async() => + ArchiveFileSkip( + "Rar.encrypted_filesOnly.rar", + "Failure jpg exe Empty тест.txt jpg\\test.jpg exe\\test.exe" + ); + + [Fact] + public void Rar5_Encrypted_Iterate_Archive_Async() => + ArchiveFileSkip( + "Rar5.encrypted_filesOnly.rar", + "Failure jpg exe Empty тест.txt jpg\\test.jpg exe\\test.exe" + ); + + private async ValueTask ArchiveStreamReadAsync(string testArchive) + { + testArchive = Path.Combine(TEST_ARCHIVES_PATH, testArchive); + using var stream = File.OpenRead(testArchive); + using var archive = ArchiveFactory.OpenArchive(stream); + foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) + { + await entry.WriteToDirectoryAsync(SCRATCH_FILES_PATH); + } + VerifyFiles(); + } + + private async ValueTask ArchiveStreamReadExtractAllAsync( + string testArchive, + CompressionType compression + ) + { + testArchive = Path.Combine(TEST_ARCHIVES_PATH, testArchive); + using var stream = File.OpenRead(testArchive); + await using var archive = await ArchiveFactory.OpenAsyncArchive( + new AsyncOnlyStream(stream) + ); + Assert.True(await archive.IsSolidAsync()); + await using (var reader = await archive.ExtractAllEntriesAsync()) + { + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + Assert.Equal(compression, reader.Entry.CompressionType); + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + } + VerifyFiles(); + + await foreach (var entry in archive.EntriesAsync.Where(entry => !entry.IsDirectory)) + { + await entry.WriteToDirectoryAsync(SCRATCH_FILES_PATH); + } + VerifyFiles(); + } + + private async ValueTask ArchiveFileReadAsync(string testArchive) + { + testArchive = Path.Combine(TEST_ARCHIVES_PATH, testArchive); + using var archive = ArchiveFactory.OpenArchive(testArchive); + foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) + { + await entry.WriteToDirectoryAsync(SCRATCH_FILES_PATH); + } + VerifyFiles(); + } + + private async ValueTask ArchiveStreamMultiReadAsync( + ReaderOptions? readerOptions, + params string[] testArchives + ) + { + var paths = testArchives.Select(x => Path.Combine(TEST_ARCHIVES_PATH, x)); + using var archive = ArchiveFactory.OpenArchive( + paths.Select(a => new FileInfo(a)).ToArray(), + readerOptions + ); + foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) + { + await entry.WriteToDirectoryAsync(SCRATCH_FILES_PATH); + } + VerifyFiles(); + } + + /// + /// Tests for Issue #1050 - RAR extraction with WriteToDirectoryAsync creates folders + /// but places all files at the top level instead of in their subdirectories. + /// + [Fact] + public async ValueTask Rar_Issue1050_WriteToDirectoryAsync_ExtractsToSubdirectories() + { + var testFile = "Rar.issue1050.rar"; + using var fileStream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, testFile)); + await using var archive = await RarArchive.OpenAsyncArchive(fileStream); + + // Extract using archive.WriteToDirectoryAsync without explicit options + await archive.WriteToDirectoryAsync(SCRATCH_FILES_PATH); + + // Verify files are in their subdirectories, not at the root + Assert.True( + File.Exists(Path.Combine(SCRATCH_FILES_PATH, "PhysicsBraid", "263825.tr11dtp")), + "File should be in PhysicsBraid subdirectory" + ); + Assert.True( + File.Exists(Path.Combine(SCRATCH_FILES_PATH, "Animations", "15441.tr11anim")), + "File should be in Animations subdirectory" + ); + Assert.True( + File.Exists(Path.Combine(SCRATCH_FILES_PATH, "Braid", "766728.tr11dtp")), + "File should be in Braid subdirectory" + ); + Assert.True( + File.Exists(Path.Combine(SCRATCH_FILES_PATH, "Braid", "766832.tr11dtp")), + "File should be in Braid subdirectory" + ); + Assert.True( + File.Exists(Path.Combine(SCRATCH_FILES_PATH, "HeadBraid", "321353.tr11modeldata")), + "File should be in HeadBraid subdirectory" + ); + + // NOTE: The file size check is omitted because there's a separate pre-existing bug + // in the async RAR stream implementation that causes incorrect file sizes. + // This test only verifies the directory structure fix. + } + + private async ValueTask ArchiveOpenStreamReadAsync( + ReaderOptions? readerOptions, + params string[] testArchives + ) + { + var paths = testArchives.Select(x => Path.Combine(TEST_ARCHIVES_PATH, x)); + using var archive = ArchiveFactory.OpenArchive( + paths.Select(f => new FileInfo(f)).ToArray(), + readerOptions + ); + foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) + { + await entry.WriteToDirectoryAsync(SCRATCH_FILES_PATH); + } + VerifyFiles(); + } +} diff --git a/tests/SharpCompress.Test/Rar/RarArchiveTests.cs b/tests/SharpCompress.Test/Rar/RarArchiveTests.cs index 6496e15e..65f5a0d2 100644 --- a/tests/SharpCompress.Test/Rar/RarArchiveTests.cs +++ b/tests/SharpCompress.Test/Rar/RarArchiveTests.cs @@ -1,66 +1,102 @@ +using System; using System.IO; using System.Linq; using SharpCompress.Archives; using SharpCompress.Archives.Rar; using SharpCompress.Common; -using SharpCompress.Compressors.LZMA.Utilites; +using SharpCompress.Compressors.LZMA.Utilities; using SharpCompress.Readers; +using SharpCompress.Test.Mocks; using Xunit; namespace SharpCompress.Test.Rar; public class RarArchiveTests : ArchiveTests { + [Theory] + [InlineData("Rar15.rar")] + [InlineData("Rar2.rar")] + [InlineData("Rar.rar")] + [InlineData("Rar.Audio_program.rar")] + [InlineData("Rar5.rar")] + [InlineData("Rar5.solid.rar")] + public void Rar_Archive_Recently_Changed_Unpackers_Sync(string filename) + { + var extractedEntries = 0; + using var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, filename)); + using var archive = RarArchive.OpenArchive( + stream, + new ReaderOptions { LookForHeader = true } + ); + + foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) + { + using var output = new MemoryStream(); + entry.WriteTo(output); + extractedEntries++; + } + + Assert.True(extractedEntries > 0); + } + [Fact] public void Rar_EncryptedFileAndHeader_Archive() => ReadRarPassword("Rar.encrypted_filesAndHeader.rar", "test"); [Fact] public void Rar_EncryptedFileAndHeader_NoPasswordExceptionTest() => - Assert.Throws( - typeof(CryptographicException), - () => ReadRarPassword("Rar.encrypted_filesAndHeader.rar", null) + Assert.Throws(() => + ReadRarPassword("Rar.encrypted_filesAndHeader.rar", null) ); - /*[Fact] - public void Rar5_EncryptedFileAndHeader_Archive() - { + [Fact] + public void Rar5_EncryptedFileAndHeader_Archive() => ReadRarPassword("Rar5.encrypted_filesAndHeader.rar", "test"); - }*/ + + [Fact] + public void Rar5_EncryptedFileAndHeader_Archive_Err() => + Assert.Throws(() => + ReadRarPassword("Rar5.encrypted_filesAndHeader.rar", "failed") + ); [Fact] public void Rar5_EncryptedFileAndHeader_NoPasswordExceptionTest() => - Assert.Throws( - typeof(CryptographicException), - () => ReadRarPassword("Rar5.encrypted_filesAndHeader.rar", null) + Assert.Throws(() => + ReadRarPassword("Rar5.encrypted_filesAndHeader.rar", null) ); [Fact] public void Rar_EncryptedFileOnly_Archive() => ReadRarPassword("Rar.encrypted_filesOnly.rar", "test"); - /*[Fact] - public void Rar5_EncryptedFileOnly_Archive() - { + [Fact] + public void Rar_EncryptedFileOnly_Archive_Err() => + Assert.Throws(() => + ReadRarPassword("Rar5.encrypted_filesOnly.rar", "failed") + ); + + [Fact] + public void Rar5_EncryptedFileOnly_Archive() => ReadRarPassword("Rar5.encrypted_filesOnly.rar", "test"); - }*/ [Fact] public void Rar_Encrypted_Archive() => ReadRarPassword("Rar.Encrypted.rar", "test"); - /*[Fact] - public void Rar5_Encrypted_Archive() - { + [Fact] + public void Rar5_Encrypted_Archive() => ReadRarPassword("Rar5.encrypted_filesAndHeader.rar", "test"); - }*/ private void ReadRarPassword(string testArchive, string? password) { using (Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, testArchive))) using ( - var archive = RarArchive.Open( + var archive = RarArchive.OpenArchive( stream, - new ReaderOptions() { Password = password, LeaveStreamOpen = true } + ReaderOptions.ForExternalStream with + { + Password = password, + LeaveStreamOpen = true, + } ) ) { @@ -69,10 +105,7 @@ public class RarArchiveTests : ArchiveTests if (!entry.IsDirectory) { Assert.Equal(CompressionType.Rar, entry.CompressionType); - entry.WriteToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + entry.WriteToDirectory(SCRATCH_FILES_PATH); } } } @@ -81,25 +114,26 @@ public class RarArchiveTests : ArchiveTests [Fact] public void Rar_Multi_Archive_Encrypted() => - Assert.Throws( - () => ArchiveFileReadPassword("Rar.EncryptedParts.part01.rar", "test") + Assert.Throws(() => + ArchiveFileReadPassword("Rar.EncryptedParts.part01.rar", "test") ); protected void ArchiveFileReadPassword(string archiveName, string password) { using ( - var archive = RarArchive.Open( + var archive = RarArchive.OpenArchive( Path.Combine(TEST_ARCHIVES_PATH, archiveName), - new ReaderOptions() { Password = password, LeaveStreamOpen = true } + ReaderOptions.ForFilePath with + { + Password = password, + LeaveStreamOpen = true, + } ) ) { foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) { - entry.WriteToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + entry.WriteToDirectory(SCRATCH_FILES_PATH); } } VerifyFiles(); @@ -114,6 +148,23 @@ public class RarArchiveTests : ArchiveTests [Fact] public void Rar_ArchiveStreamRead() => ArchiveStreamRead("Rar.rar"); + [Fact] + public void RarArchive_StreamCollection_Throws_On_NonSeekable_Stream() + { + using var nonSeekable = new ForwardOnlyStream(new MemoryStream()); + using var seekable = new MemoryStream(); + + Assert.Throws(() => RarArchive.OpenArchive([nonSeekable, seekable])); + } + + [Fact] + public void RarArchive_Stream_Throws_On_Unreadable_Stream() + { + using var unreadable = new TestStream(new MemoryStream(), false, true, true); + + Assert.Throws(() => RarArchive.OpenArchive(unreadable)); + } + [Fact] public void Rar5_ArchiveStreamRead() => ArchiveStreamRead("Rar5.rar"); @@ -124,13 +175,10 @@ public class RarArchiveTests : ArchiveTests private void DoRar_test_invalid_exttime_ArchiveStreamRead(string filename) { using var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, filename)); - using var archive = ArchiveFactory.Open(stream); + using var archive = ArchiveFactory.OpenArchive(stream); foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) { - entry.WriteToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + entry.WriteToDirectory(SCRATCH_FILES_PATH); } } @@ -138,14 +186,19 @@ public class RarArchiveTests : ArchiveTests public void Rar_Jpg_ArchiveStreamRead() { using var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Rar.jpeg.jpg")); - using (var archive = RarArchive.Open(stream, new ReaderOptions() { LookForHeader = true })) + using ( + var archive = RarArchive.OpenArchive( + stream, + ReaderOptions.ForExternalStream with + { + LookForHeader = true, + } + ) + ) { foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) { - entry.WriteToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + entry.WriteToDirectory(SCRATCH_FILES_PATH); } } VerifyFiles(); @@ -161,14 +214,11 @@ public class RarArchiveTests : ArchiveTests { using (var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, filename))) { - using var archive = RarArchive.Open(stream); + using var archive = RarArchive.OpenArchive(stream); Assert.False(archive.IsSolid); foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) { - entry.WriteToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + entry.WriteToDirectory(SCRATCH_FILES_PATH); } } VerifyFiles(); @@ -181,7 +231,7 @@ public class RarArchiveTests : ArchiveTests private void DoRar_IsSolidEntryStreamCheck(string filename) { using var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, filename)); - using var archive = RarArchive.Open(stream); + using var archive = RarArchive.OpenArchive(stream); Assert.True(archive.IsSolid); IArchiveEntry[] entries = archive.Entries.Where(a => !a.IsDirectory).ToArray(); Assert.NotInRange(entries.Length, 0, 1); @@ -221,60 +271,57 @@ public class RarArchiveTests : ArchiveTests [Fact] public void Rar_Multi_ArchiveStreamRead() => DoRar_Multi_ArchiveStreamRead( - new[] - { + [ "Rar.multi.part01.rar", "Rar.multi.part02.rar", "Rar.multi.part03.rar", "Rar.multi.part04.rar", "Rar.multi.part05.rar", - "Rar.multi.part06.rar" - }, + "Rar.multi.part06.rar", + ], false ); [Fact] public void Rar5_Multi_ArchiveStreamRead() => DoRar_Multi_ArchiveStreamRead( - new[] - { + [ "Rar5.multi.part01.rar", "Rar5.multi.part02.rar", "Rar5.multi.part03.rar", "Rar5.multi.part04.rar", "Rar5.multi.part05.rar", - "Rar5.multi.part06.rar" - }, + "Rar5.multi.part06.rar", + ], false ); private void DoRar_Multi_ArchiveStreamRead(string[] archives, bool isSolid) { - using var archive = RarArchive.Open( - archives.Select(s => Path.Combine(TEST_ARCHIVES_PATH, s)).Select(File.OpenRead) + using var archive = RarArchive.OpenArchive( + archives + .Select(s => Path.Combine(TEST_ARCHIVES_PATH, s)) + .Select(File.OpenRead) + .ToArray() ); Assert.Equal(archive.IsSolid, isSolid); foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) { - entry.WriteToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + entry.WriteToDirectory(SCRATCH_FILES_PATH); } } [Fact] public void Rar5_MultiSolid_ArchiveStreamRead() => DoRar_Multi_ArchiveStreamRead( - new[] - { + [ "Rar.multi.solid.part01.rar", "Rar.multi.solid.part02.rar", "Rar.multi.solid.part03.rar", "Rar.multi.solid.part04.rar", "Rar.multi.solid.part05.rar", - "Rar.multi.solid.part06.rar" - }, + "Rar.multi.solid.part06.rar", + ], true ); @@ -301,7 +348,7 @@ public class RarArchiveTests : ArchiveTests private void DoRar_ArchiveFileRead_HasDirectories(string filename) { using var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, filename)); - using var archive = RarArchive.Open(stream); + using var archive = RarArchive.OpenArchive(stream); Assert.False(archive.IsSolid); Assert.Contains(true, archive.Entries.Select(entry => entry.IsDirectory)); } @@ -310,18 +357,18 @@ public class RarArchiveTests : ArchiveTests public void Rar_Jpg_ArchiveFileRead() { using ( - var archive = RarArchive.Open( + var archive = RarArchive.OpenArchive( Path.Combine(TEST_ARCHIVES_PATH, "Rar.jpeg.jpg"), - new ReaderOptions() { LookForHeader = true } + ReaderOptions.ForFilePath with + { + LookForHeader = true, + } ) ) { foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) { - entry.WriteToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + entry.WriteToDirectory(SCRATCH_FILES_PATH); } } VerifyFiles(); @@ -336,16 +383,15 @@ public class RarArchiveTests : ArchiveTests [Fact] public void Rar2_Multi_ArchiveStreamRead() => DoRar_Multi_ArchiveStreamRead( - new[] - { + [ "Rar2.multi.rar", "Rar2.multi.r00", "Rar2.multi.r01", "Rar2.multi.r02", "Rar2.multi.r03", "Rar2.multi.r04", - "Rar2.multi.r05" - }, + "Rar2.multi.r05", + ], false ); @@ -368,7 +414,7 @@ public class RarArchiveTests : ArchiveTests { var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "Rar15.rar"); - using var archive = RarArchive.Open(testArchive); + using var archive = RarArchive.OpenArchive(testArchive); Assert.Equal(1, archive.MinVersion); Assert.Equal(1, archive.MaxVersion); } @@ -378,7 +424,7 @@ public class RarArchiveTests : ArchiveTests { var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "Rar2.rar"); - using var archive = RarArchive.Open(testArchive); + using var archive = RarArchive.OpenArchive(testArchive); Assert.Equal(2, archive.MinVersion); Assert.Equal(2, archive.MaxVersion); } @@ -388,7 +434,7 @@ public class RarArchiveTests : ArchiveTests { var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "Rar4.multi.part01.rar"); - using var archive = RarArchive.Open(testArchive); + using var archive = RarArchive.OpenArchive(testArchive); Assert.Equal(3, archive.MinVersion); Assert.Equal(4, archive.MaxVersion); } @@ -398,7 +444,7 @@ public class RarArchiveTests : ArchiveTests { var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "Rar5.solid.rar"); - using var archive = RarArchive.Open(testArchive); + using var archive = RarArchive.OpenArchive(testArchive); Assert.Equal(5, archive.MinVersion); Assert.Equal(6, archive.MaxVersion); } @@ -420,7 +466,7 @@ public class RarArchiveTests : ArchiveTests "Rar4.split.003", "Rar4.split.004", "Rar4.split.005", - "Rar4.split.006" + "Rar4.split.006", } ); @@ -436,7 +482,7 @@ public class RarArchiveTests : ArchiveTests "Rar2.multi.r02", "Rar2.multi.r03", "Rar2.multi.r04", - "Rar2.multi.r05" + "Rar2.multi.r05", } ); @@ -452,23 +498,22 @@ public class RarArchiveTests : ArchiveTests "Rar4.multi.part04.rar", "Rar4.multi.part05.rar", "Rar4.multi.part06.rar", - "Rar4.multi.part07.rar" + "Rar4.multi.part07.rar", } ); [Fact] public void Rar4_Multi_ArchiveStreamRead() => DoRar_Multi_ArchiveStreamRead( - new[] - { + [ "Rar4.multi.part01.rar", "Rar4.multi.part02.rar", "Rar4.multi.part03.rar", "Rar4.multi.part04.rar", "Rar4.multi.part05.rar", "Rar4.multi.part06.rar", - "Rar4.multi.part07.rar" - }, + "Rar4.multi.part07.rar", + ], false ); @@ -477,15 +522,14 @@ public class RarArchiveTests : ArchiveTests public void Rar4_Split_ArchiveStreamRead() => ArchiveStreamMultiRead( null, - new[] - { + [ "Rar4.split.001", "Rar4.split.002", "Rar4.split.003", "Rar4.split.004", "Rar4.split.005", - "Rar4.split.006" - } + "Rar4.split.006", + ] ); //will detect and load other files @@ -512,15 +556,14 @@ public class RarArchiveTests : ArchiveTests public void Rar4_Split_ArchiveStreamFirstFileRead() => ArchiveStreamMultiRead( null, - new[] - { + [ "Rar4.split.001", //"Rar4.split.002", //"Rar4.split.003", //"Rar4.split.004", //"Rar4.split.005", //"Rar4.split.006" - } + ] ); //open with ArchiveFactory.Open and stream @@ -553,12 +596,11 @@ public class RarArchiveTests : ArchiveTests [Fact] public void Rar4_Multi_ArchiveOpenEntryVolumeIndexTest() => ArchiveOpenEntryVolumeIndexTest( - new[] - { - new[] { 0, 1 }, //exe - Rar4.multi.part01.rar to Rar4.multi.part02.rar - new[] { 1, 5 }, //jpg - Rar4.multi.part02.rar to Rar4.multi.part06.rar - new[] { 5, 6 } //txt - Rar4.multi.part06.rar to Rar4.multi.part07.rar - }, + [ + [0, 1], //exe - Rar4.multi.part01.rar to Rar4.multi.part02.rar + [1, 5], //jpg - Rar4.multi.part02.rar to Rar4.multi.part06.rar + [5, 6], //txt - Rar4.multi.part06.rar to Rar4.multi.part07.rar + ], null, "Rar4.multi.part01.rar", "Rar4.multi.part02.rar", @@ -583,7 +625,7 @@ public class RarArchiveTests : ArchiveTests private void DoRar_IsFirstVolume_True(string firstFilename) { - using var archive = RarArchive.Open(Path.Combine(TEST_ARCHIVES_PATH, firstFilename)); + using var archive = RarArchive.OpenArchive(Path.Combine(TEST_ARCHIVES_PATH, firstFilename)); Assert.True(archive.IsMultipartVolume()); Assert.True(archive.IsFirstVolume()); } @@ -596,8 +638,199 @@ public class RarArchiveTests : ArchiveTests private void DoRar_IsFirstVolume_False(string notFirstFilename) { - using var archive = RarArchive.Open(Path.Combine(TEST_ARCHIVES_PATH, notFirstFilename)); + using var archive = RarArchive.OpenArchive( + Path.Combine(TEST_ARCHIVES_PATH, notFirstFilename) + ); Assert.True(archive.IsMultipartVolume()); Assert.False(archive.IsFirstVolume()); } + + [Fact] + public void Rar5_CRC_Blake2_Archive() => ArchiveFileRead("Rar5.crc_blake2.rar"); + + [Fact] + void Rar_Iterate_Archive() => + ArchiveFileSkip("Rar.rar", "Failure jpg exe Empty jpg\\test.jpg exe\\test.exe тест.txt"); + + [Fact] + public void Rar2_Iterate_Archive() => + ArchiveFileSkip("Rar2.rar", "Failure Empty тест.txt jpg\\test.jpg exe\\test.exe jpg exe"); + + [Fact] + public void Rar4_Iterate_Archive() => + ArchiveFileSkip("Rar4.rar", "Failure Empty jpg exe тест.txt jpg\\test.jpg exe\\test.exe"); + + [Fact] + public void Rar5_Iterate_Archive() => + ArchiveFileSkip("Rar5.rar", "Failure jpg exe Empty тест.txt jpg\\test.jpg exe\\test.exe"); + + [Fact] + public void Rar_Encrypted_Iterate_Archive() => + ArchiveFileSkip( + "Rar.encrypted_filesOnly.rar", + "Failure jpg exe Empty тест.txt jpg\\test.jpg exe\\test.exe" + ); + + [Fact] + public void Rar5_Encrypted_Iterate_Archive() => + ArchiveFileSkip( + "Rar5.encrypted_filesOnly.rar", + "Failure jpg exe Empty тест.txt jpg\\test.jpg exe\\test.exe" + ); + + [Fact] + public void Rar_TestEncryptedDetection() + { + using var passwordProtectedFilesArchive = RarArchive.OpenArchive( + Path.Combine(TEST_ARCHIVES_PATH, "Rar.encrypted_filesOnly.rar") + ); + Assert.True(passwordProtectedFilesArchive.IsEncrypted); + } + + /// + /// Test for issue: InvalidOperationException when extracting RAR files. + /// This test verifies the fix for the validation logic that was changed from + /// (_position != Length) to (_position < Length). + /// The old logic would throw an exception when position exceeded expected length, + /// but the new logic only throws when decompression ends prematurely (position < expected). + /// + [Fact] + public void Rar_StreamValidation_OnlyThrowsOnPrematureEnd() + { + // Test normal extraction - should NOT throw InvalidOperationException + // even if actual decompressed size differs from header + var testFiles = new[] { "Rar.rar", "Rar5.rar", "Rar4.rar", "Rar2.rar" }; + + foreach (var testFile in testFiles) + { + using var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, testFile)); + using var archive = RarArchive.OpenArchive(stream); + + // Extract all entries and read them completely + foreach (var entry in archive.Entries.Where(e => !e.IsDirectory)) + { + using var entryStream = entry.OpenEntryStream(); + using var ms = new MemoryStream(); + + // This should complete without throwing InvalidOperationException + // The fix ensures we only throw when position < expected length, not when position >= expected + entryStream.CopyTo(ms); + + // Verify we read some data + Assert.True( + ms.Length > 0, + $"Failed to extract data from {entry.Key} in {testFile}" + ); + } + } + } + + /// + /// Negative test case: Verifies that InvalidOperationException IS thrown when + /// a RAR stream ends prematurely (position < expected length). + /// This tests the validation condition (_position < Length) works correctly. + /// + [Fact] + public void Rar_StreamValidation_ThrowsOnTruncatedStream() + { + // This test verifies the exception is thrown when decompression ends prematurely + // by using a truncated stream that stops reading after a small number of bytes + var testFile = "Rar.rar"; + using var fileStream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, testFile)); + + // Wrap the file stream with a truncated stream that will stop reading early + // This simulates a corrupted or truncated RAR file + using var truncatedStream = new TruncatedStream(fileStream, 1000); + + // Opening the archive should work, but extracting should throw + // when we try to read beyond the truncated data + var exception = Assert.Throws(() => + { + using var archive = RarArchive.OpenArchive(truncatedStream); + foreach (var entry in archive.Entries.Where(e => !e.IsDirectory)) + { + using var entryStream = entry.OpenEntryStream(); + using var ms = new MemoryStream(); + // This should throw InvalidOperationException when it can't read all expected bytes + entryStream.CopyTo(ms); + } + }); + + // Verify the exception message matches our expectation + Assert.Contains("unpacked file size does not match header", exception.Message); + } + + /// + /// Tests for Issue #1050 - RAR extraction with WriteToDirectory creates folders + /// but places all files at the top level instead of in their subdirectories. + /// + [Fact] + public void Rar_Issue1050_WriteToDirectory_ExtractsToSubdirectories() + { + var testFile = "Rar.issue1050.rar"; + using var fileStream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, testFile)); + using var archive = RarArchive.OpenArchive(fileStream); + + // Extract using archive.WriteToDirectory without explicit options + archive.WriteToDirectory(SCRATCH_FILES_PATH); + + // Verify files are in their subdirectories, not at the root + Assert.True( + File.Exists(Path.Combine(SCRATCH_FILES_PATH, "PhysicsBraid", "263825.tr11dtp")), + "File should be in PhysicsBraid subdirectory" + ); + Assert.True( + File.Exists(Path.Combine(SCRATCH_FILES_PATH, "Animations", "15441.tr11anim")), + "File should be in Animations subdirectory" + ); + Assert.True( + File.Exists(Path.Combine(SCRATCH_FILES_PATH, "Braid", "766728.tr11dtp")), + "File should be in Braid subdirectory" + ); + Assert.True( + File.Exists(Path.Combine(SCRATCH_FILES_PATH, "Braid", "766832.tr11dtp")), + "File should be in Braid subdirectory" + ); + Assert.True( + File.Exists(Path.Combine(SCRATCH_FILES_PATH, "HeadBraid", "321353.tr11modeldata")), + "File should be in HeadBraid subdirectory" + ); + + // Verify the exact file size of 766832.tr11dtp matches the archive entry size + var fileInfo = new FileInfo(Path.Combine(SCRATCH_FILES_PATH, "Braid", "766832.tr11dtp")); + Assert.Equal(4867620, fileInfo.Length); // Expected: 4,867,620 bytes + } + + /// + /// Test case for malformed RAR archives that previously caused infinite loops. + /// This test verifies that attempting to read entries from a potentially malformed + /// 512-byte RAR archive throws an InvalidOperationException instead of looping infinitely. + /// See: https://github.com/adamhathcock/sharpcompress/issues/1176 + /// + [Fact] + public void Rar_MalformedArchive_NoInfiniteLoop() + { + var testFile = "Rar.malformed_512byte.rar"; + var readerOptions = ReaderOptions.ForExternalStream.WithLookForHeader(true); + + // This should throw InvalidOperationException, not hang in an infinite loop + var exception = Assert.Throws(() => + { + using var fileStream = File.Open( + Path.Combine(TEST_ARCHIVES_PATH, testFile), + FileMode.Open + ); + using var archive = RarArchive.OpenArchive(fileStream, readerOptions); + + // Attempting to enumerate entries should throw an exception + // instead of looping infinitely + foreach (var entry in archive.Entries.Where(e => !e.IsDirectory)) + { + // This line should not be reached due to the exception + } + }); + + // Verify that the exception is related to seeking beyond available data + Assert.Contains("Cannot seek to position", exception.Message); + } } diff --git a/tests/SharpCompress.Test/Rar/RarCRCTest.cs b/tests/SharpCompress.Test/Rar/RarCRCTest.cs new file mode 100644 index 00000000..6a9860d5 --- /dev/null +++ b/tests/SharpCompress.Test/Rar/RarCRCTest.cs @@ -0,0 +1,361 @@ +using System; +using SharpCompress.Compressors.Rar; +using Xunit; + +namespace SharpCompress.Test.Compressors.Rar; + +public class RarCRCTest +{ + [Fact] + public void CheckCrc_SingleByte_ReturnsCorrectCrc() + { + // Arrange + uint startCrc = 0; + byte testByte = 0x42; + + // Act + var result = RarCRC.CheckCrc(startCrc, testByte); + + // Assert + Assert.NotEqual(0u, result); + } + + [Fact] + public void CheckCrc_SingleByte_WithNonZeroStartCrc() + { + // Arrange + uint startCrc = 0x12345678; + byte testByte = 0xAB; + + // Act + var result = RarCRC.CheckCrc(startCrc, testByte); + + // Assert + Assert.NotEqual(startCrc, result); + } + + [Fact] + public void CheckCrc_SingleByte_DifferentBytesProduceDifferentCrcs() + { + // Arrange + uint startCrc = 0; + byte byte1 = 0x01; + byte byte2 = 0x02; + + // Act + var result1 = RarCRC.CheckCrc(startCrc, byte1); + var result2 = RarCRC.CheckCrc(startCrc, byte2); + + // Assert + Assert.NotEqual(result1, result2); + } + + [Fact] + public void CheckCrc_EmptySpan_ReturnsStartCrc() + { + // Arrange + uint startCrc = 0x12345678; + ReadOnlySpan data = ReadOnlySpan.Empty; + + // Act + var result = RarCRC.CheckCrc(startCrc, data, 0, 0); + + // Assert + Assert.Equal(startCrc, result); + } + + [Fact] + public void CheckCrc_SingleByteSpan_MatchesSingleByteMethod() + { + // Arrange + uint startCrc = 0; + byte testByte = 0x42; + ReadOnlySpan data = stackalloc byte[] { testByte }; + + // Act + var resultSingleByte = RarCRC.CheckCrc(startCrc, testByte); + var resultSpan = RarCRC.CheckCrc(startCrc, data, 0, 1); + + // Assert + Assert.Equal(resultSingleByte, resultSpan); + } + + [Fact] + public void CheckCrc_MultipleBytes_ProducesConsistentResult() + { + // Arrange + uint startCrc = 0; + ReadOnlySpan data = stackalloc byte[] { 0x01, 0x02, 0x03, 0x04 }; + + // Act + var result1 = RarCRC.CheckCrc(startCrc, data, 0, 4); + var result2 = RarCRC.CheckCrc(startCrc, data, 0, 4); + + // Assert + Assert.Equal(result1, result2); + } + + [Fact] + public void CheckCrc_MultipleBytes_IncrementalMatchesComplete() + { + // Arrange + uint startCrc = 0; + ReadOnlySpan data = stackalloc byte[] { 0x01, 0x02, 0x03, 0x04 }; + + // Act - calculate incrementally + var crc1 = RarCRC.CheckCrc(startCrc, data[0]); + var crc2 = RarCRC.CheckCrc(crc1, data[1]); + var crc3 = RarCRC.CheckCrc(crc2, data[2]); + var crc4 = RarCRC.CheckCrc(crc3, data[3]); + + // Act - calculate all at once + var crcComplete = RarCRC.CheckCrc(startCrc, data, 0, 4); + + // Assert + Assert.Equal(crc4, crcComplete); + } + + [Fact] + public void CheckCrc_WithOffset_ProcessesCorrectBytes() + { + // Arrange + uint startCrc = 0; + ReadOnlySpan data = stackalloc byte[] { 0xFF, 0xFF, 0x01, 0x02, 0x03, 0xFF, 0xFF }; + + // Act + var result = RarCRC.CheckCrc(startCrc, data, 2, 3); + var expected = RarCRC.CheckCrc(startCrc, stackalloc byte[] { 0x01, 0x02, 0x03 }, 0, 3); + + // Assert + Assert.Equal(expected, result); + } + + [Fact] + public void CheckCrc_WithCountSmallerThanData_ProcessesOnlyCount() + { + // Arrange + uint startCrc = 0; + ReadOnlySpan data = stackalloc byte[] { 0x01, 0x02, 0x03, 0x04, 0x05 }; + + // Act + var result = RarCRC.CheckCrc(startCrc, data, 0, 3); + var expected = RarCRC.CheckCrc(startCrc, stackalloc byte[] { 0x01, 0x02, 0x03 }, 0, 3); + + // Assert + Assert.Equal(expected, result); + } + + [Fact] + public void CheckCrc_CountLargerThanRemainingData_ProcessesOnlyAvailableData() + { + // Arrange + uint startCrc = 0; + ReadOnlySpan data = stackalloc byte[] { 0x01, 0x02, 0x03 }; + + // Act + var result = RarCRC.CheckCrc(startCrc, data, 0, 100); + var expected = RarCRC.CheckCrc(startCrc, data, 0, 3); + + // Assert + Assert.Equal(expected, result); + } + + [Fact] + public void CheckCrc_KnownTestVector_HelloWorld() + { + // Arrange - "Hello, World!" in ASCII + uint startCrc = 0xFFFFFFFF; // CRC32 typically starts with inverted bits + var data = System.Text.Encoding.ASCII.GetBytes("Hello, World!"); + + // Act + var result = RarCRC.CheckCrc(startCrc, data, 0, data.Length); + + // Assert - verify it produces a result (exact value depends on CRC32 variant) + Assert.NotEqual(startCrc, result); + Assert.NotEqual(0u, result); + } + + [Fact] + public void CheckCrc_AllZeros_ProducesConsistentResult() + { + // Arrange + uint startCrc = 0; + ReadOnlySpan data = stackalloc byte[10]; // all zeros + + // Act + var result = RarCRC.CheckCrc(startCrc, data, 0, 10); + + // Assert - verify it's deterministic + var result2 = RarCRC.CheckCrc(startCrc, data, 0, 10); + Assert.Equal(result, result2); + // CRC of all zeros from startCrc=0 can be 0, that's valid + } + + [Fact] + public void CheckCrc_AllOnes_ProducesConsistentResult() + { + // Arrange + uint startCrc = 0; + Span data = stackalloc byte[10]; + data.Fill(0xFF); + + // Act + var result = RarCRC.CheckCrc(startCrc, data, 0, 10); + + // Assert + var result2 = RarCRC.CheckCrc(startCrc, data, 0, 10); + Assert.Equal(result, result2); + Assert.NotEqual(0u, result); + } + + [Fact] + public void CheckCrc_OrderMatters() + { + // Arrange + uint startCrc = 0; + ReadOnlySpan data1 = stackalloc byte[] { 0x01, 0x02 }; + ReadOnlySpan data2 = stackalloc byte[] { 0x02, 0x01 }; + + // Act + var result1 = RarCRC.CheckCrc(startCrc, data1, 0, 2); + var result2 = RarCRC.CheckCrc(startCrc, data2, 0, 2); + + // Assert - different order should produce different CRC + Assert.NotEqual(result1, result2); + } + + [Fact] + public void CheckCrc_LargeData_ProcessesCorrectly() + { + // Arrange + uint startCrc = 0; + var data = new byte[1024]; + for (int i = 0; i < data.Length; i++) + { + data[i] = (byte)(i % 256); + } + + // Act + var result = RarCRC.CheckCrc(startCrc, data, 0, data.Length); + + // Assert + Assert.NotEqual(0u, result); + // Verify it's deterministic + var result2 = RarCRC.CheckCrc(startCrc, data, 0, data.Length); + Assert.Equal(result, result2); + } + + [Fact] + public void CheckCrc_PartialSpan_WithOffsetAndCount() + { + // Arrange + uint startCrc = 0; + var data = new byte[100]; + for (int i = 0; i < data.Length; i++) + { + data[i] = (byte)(i % 256); + } + + // Act - process middle section + var result = RarCRC.CheckCrc(startCrc, data, 25, 50); + + // Assert - verify it processes exactly 50 bytes starting at offset 25 + var middleSection = new byte[50]; + Array.Copy(data, 25, middleSection, 0, 50); + var expected = RarCRC.CheckCrc(startCrc, middleSection, 0, 50); + Assert.Equal(expected, result); + } + + [Fact] + public void CheckCrc_ZeroCount_ReturnsStartCrc() + { + // Arrange + uint startCrc = 0x12345678; + ReadOnlySpan data = stackalloc byte[] { 0x01, 0x02, 0x03 }; + + // Act + var result = RarCRC.CheckCrc(startCrc, data, 0, 0); + + // Assert + Assert.Equal(startCrc, result); + } + + [Fact] + public void CheckCrc_MaxByteValue_HandlesCorrectly() + { + // Arrange + uint startCrc = 0; + byte maxByte = 0xFF; + + // Act + var result = RarCRC.CheckCrc(startCrc, maxByte); + + // Assert + Assert.NotEqual(0u, result); + } + + [Fact] + public void CheckCrc_MinByteValue_HandlesCorrectly() + { + // Arrange + uint startCrc = 0; + byte minByte = 0x00; + + // Act + var result = RarCRC.CheckCrc(startCrc, minByte); + + // Assert - CRC of 0x00 from startCrc=0 can be 0, that's mathematically valid + // What matters is that it's deterministic and doesn't crash + var result2 = RarCRC.CheckCrc(startCrc, minByte); + Assert.Equal(result, result2); + } + + [Fact] + public void CheckCrc_ChainedCalls_ProduceCorrectResult() + { + // Arrange + uint startCrc = 0; + ReadOnlySpan part1 = stackalloc byte[] { 0x01, 0x02 }; + ReadOnlySpan part2 = stackalloc byte[] { 0x03, 0x04 }; + ReadOnlySpan combined = stackalloc byte[] { 0x01, 0x02, 0x03, 0x04 }; + + // Act + var crc1 = RarCRC.CheckCrc(startCrc, part1, 0, 2); + var crc2 = RarCRC.CheckCrc(crc1, part2, 0, 2); + var crcCombined = RarCRC.CheckCrc(startCrc, combined, 0, 4); + + // Assert - chained calculation should equal combined calculation + Assert.Equal(crc2, crcCombined); + } + + [Theory] + [InlineData(0x00000000)] + [InlineData(0xFFFFFFFF)] + [InlineData(0x12345678)] + [InlineData(0xABCDEF01)] + public void CheckCrc_VariousStartCrcs_ProduceDifferentResults(uint startCrc) + { + // Arrange + ReadOnlySpan data = stackalloc byte[] { 0x01, 0x02, 0x03 }; + + // Act + var result = RarCRC.CheckCrc(startCrc, data, 0, 3); + + // Assert - result should be different from start (unless by extreme coincidence) + Assert.NotEqual(0u, result); + } + + [Fact] + public void CheckCrc_OffsetAtEndOfData_ReturnsStartCrc() + { + // Arrange + uint startCrc = 0x12345678; + ReadOnlySpan data = stackalloc byte[] { 0x01, 0x02, 0x03 }; + + // Act - offset is at the end, so no bytes to process + var result = RarCRC.CheckCrc(startCrc, data, 3, 5); + + // Assert + Assert.Equal(startCrc, result); + } +} diff --git a/tests/SharpCompress.Test/Rar/RarCrcExtractionTests.cs b/tests/SharpCompress.Test/Rar/RarCrcExtractionTests.cs new file mode 100644 index 00000000..f2dc8f4f --- /dev/null +++ b/tests/SharpCompress.Test/Rar/RarCrcExtractionTests.cs @@ -0,0 +1,36 @@ +using System.IO; +using System.Linq; +using SharpCompress.Archives; +using SharpCompress.Archives.Rar; +using SharpCompress.Common; +using Xunit; + +namespace SharpCompress.Test.Rar; + +public class RarCrcExtractionTests : ArchiveTests +{ + [Theory] + [InlineData("Rar.rar")] + [InlineData("Rar5.rar")] + public void Rar_Archive_WriteToFile_Throws_On_Crc_Mismatch(string archiveName) + { + using var archive = RarArchive.OpenArchive(Path.Combine(TEST_ARCHIVES_PATH, archiveName)); + var entry = CorruptFirstFileCrc(archive); + var destination = Path.Combine(SCRATCH_FILES_PATH, $"{archiveName}-crc-mismatch.txt"); + + Assert.Throws(() => entry.WriteToFile(destination)); + } + + private static RarArchiveEntry CorruptFirstFileCrc(IArchive archive) + { + var entry = archive.Entries.OfType().First(e => !e.IsDirectory); + CorruptCrc(entry); + return entry; + } + + private static void CorruptCrc(RarArchiveEntry entry) + { + var crc = entry.FileHeader.FileCrc.NotNull(); + crc[0] ^= 0xFF; + } +} diff --git a/tests/SharpCompress.Test/Rar/RarHeaderFactoryTest.cs b/tests/SharpCompress.Test/Rar/RarHeaderFactoryTest.cs index f2b1913c..6fd42ba9 100644 --- a/tests/SharpCompress.Test/Rar/RarHeaderFactoryTest.cs +++ b/tests/SharpCompress.Test/Rar/RarHeaderFactoryTest.cs @@ -11,12 +11,15 @@ namespace SharpCompress.Test.Rar; /// public class RarHeaderFactoryTest : TestBase { - private readonly RarHeaderFactory rarHeaderFactory; + private readonly RarHeaderFactory _rarHeaderFactory; public RarHeaderFactoryTest() => - rarHeaderFactory = new RarHeaderFactory( + _rarHeaderFactory = new RarHeaderFactory( StreamingMode.Seekable, - new ReaderOptions { LeaveStreamOpen = true } + ReaderOptions.ForExternalStream with + { + LeaveStreamOpen = true, + } ); [Fact] @@ -40,11 +43,11 @@ public class RarHeaderFactoryTest : TestBase FileMode.Open, FileAccess.Read ); - foreach (var header in rarHeaderFactory.ReadHeaders(stream)) + foreach (var header in _rarHeaderFactory.ReadHeaders(stream)) { if (header.HeaderType == HeaderType.Archive || header.HeaderType == HeaderType.Crypt) { - Assert.Equal(isEncrypted, rarHeaderFactory.IsEncrypted); + Assert.Equal(isEncrypted, _rarHeaderFactory.IsEncrypted); break; } } diff --git a/tests/SharpCompress.Test/Rar/RarReaderAsyncTests.cs b/tests/SharpCompress.Test/Rar/RarReaderAsyncTests.cs new file mode 100644 index 00000000..3cec11e2 --- /dev/null +++ b/tests/SharpCompress.Test/Rar/RarReaderAsyncTests.cs @@ -0,0 +1,416 @@ +using System; +using System.Collections; +using System.IO; +using System.Linq; +using System.Threading.Tasks; +using SharpCompress.Archives.Rar; +using SharpCompress.Common; +using SharpCompress.Readers; +using SharpCompress.Readers.Rar; +using SharpCompress.Test.Mocks; +using Xunit; + +namespace SharpCompress.Test.Rar; + +public class RarReaderAsyncTests : ReaderTests +{ + [Theory] + [InlineData("Rar15.rar")] + [InlineData("Rar.rar")] + [InlineData("Rar.Audio_program.rar")] + [InlineData("Rar5.rar")] + [InlineData("Rar5.solid.rar")] + public async ValueTask Rar_Reader_Async_Uses_Only_Async_Stream_Operations(string filename) + { + using var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, filename)); + await using var reader = await ReaderFactory.OpenAsyncReader( + new AsyncOnlyStream(stream), + new ReaderOptions { LookForHeader = true } + ); + + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + using var output = new AsyncOnlyStream(new MemoryStream()); + await reader.WriteEntryToAsync(output); + } + } + } + + [Fact] + public async ValueTask Rar_Multi_Reader_Async() => + await DoRar_Multi_Reader_Async([ + "Rar.multi.part01.rar", + "Rar.multi.part02.rar", + "Rar.multi.part03.rar", + "Rar.multi.part04.rar", + "Rar.multi.part05.rar", + "Rar.multi.part06.rar", + ]); + + [Fact] + public async ValueTask Rar5_Multi_Reader_Async() => + await DoRar_Multi_Reader_Async([ + "Rar5.multi.part01.rar", + "Rar5.multi.part02.rar", + "Rar5.multi.part03.rar", + "Rar5.multi.part04.rar", + "Rar5.multi.part05.rar", + "Rar5.multi.part06.rar", + ]); + + private async ValueTask DoRar_Multi_Reader_Async(string[] archives) + { + using ( + IReader baseReader = RarReader.OpenReader( + archives + .Select(s => Path.Combine(TEST_ARCHIVES_PATH, s)) + .Select(p => File.OpenRead(p)) + ) + ) + { + IAsyncReader reader = (IAsyncReader)baseReader; + while (await reader.MoveToNextEntryAsync()) + { + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + VerifyFiles(); + } + + [Fact] + public async ValueTask Rar_Multi_Reader_Encrypted_Async() => + await Assert.ThrowsAsync(async () => + { + string[] archives = + [ + "Rar.EncryptedParts.part01.rar", + "Rar.EncryptedParts.part02.rar", + "Rar.EncryptedParts.part03.rar", + "Rar.EncryptedParts.part04.rar", + "Rar.EncryptedParts.part05.rar", + "Rar.EncryptedParts.part06.rar", + ]; + using ( + IReader baseReader = RarReader.OpenReader( + archives + .Select(s => Path.Combine(TEST_ARCHIVES_PATH, s)) + .Select(p => File.OpenRead(p)), + ReaderOptions.ForExternalStream with + { + Password = "test", + } + ) + ) + { + IAsyncReader reader = (IAsyncReader)baseReader; + while (await reader.MoveToNextEntryAsync()) + { + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + VerifyFiles(); + }); + + [Fact] + public async ValueTask Rar_Multi_Reader_Delete_Files_Async() => + await DoRar_Multi_Reader_Delete_Files_Async([ + "Rar.multi.part01.rar", + "Rar.multi.part02.rar", + "Rar.multi.part03.rar", + "Rar.multi.part04.rar", + "Rar.multi.part05.rar", + "Rar.multi.part06.rar", + ]); + + [Fact] + public async ValueTask Rar5_Multi_Reader_Delete_Files_Async() => + await DoRar_Multi_Reader_Delete_Files_Async([ + "Rar5.multi.part01.rar", + "Rar5.multi.part02.rar", + "Rar5.multi.part03.rar", + "Rar5.multi.part04.rar", + "Rar5.multi.part05.rar", + "Rar5.multi.part06.rar", + ]); + + private async ValueTask DoRar_Multi_Reader_Delete_Files_Async(string[] archives) + { + foreach (var file in archives) + { + File.Copy( + Path.Combine(TEST_ARCHIVES_PATH, file), + Path.Combine(SCRATCH2_FILES_PATH, file) + ); + } + var streams = archives + .Select(s => Path.Combine(SCRATCH2_FILES_PATH, s)) + .Select(File.OpenRead) + .ToList(); + using (IReader baseReader = RarReader.OpenReader(streams)) + { + IAsyncReader reader = (IAsyncReader)baseReader; + while (await reader.MoveToNextEntryAsync()) + { + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + foreach (var stream in streams) + { + stream.Dispose(); + } + VerifyFiles(); + + foreach (var file in archives.Select(s => Path.Combine(SCRATCH2_FILES_PATH, s))) + { + File.Delete(file); + } + } + + [Fact] + public async ValueTask Rar_None_Reader_Async() => + await ReadAsync("Rar.none.rar", CompressionType.Rar); + + [Fact] + public async ValueTask Rar5_None_Reader_Async() => + await ReadAsync("Rar5.none.rar", CompressionType.Rar); + + [Fact] + public async ValueTask Rar_Reader_Async() => await ReadAsync("Rar.rar", CompressionType.Rar); + + [Fact] + public async ValueTask Rar5_Reader_Async() => await ReadAsync("Rar5.rar", CompressionType.Rar); + + [Fact] + public async ValueTask Rar5_CRC_Blake2_Reader_Async() => + await ReadAsync("Rar5.crc_blake2.rar", CompressionType.Rar); + + [Fact] + public async ValueTask Rar_EncryptedFileAndHeader_Reader_Async() => + await ReadRar_Async("Rar.encrypted_filesAndHeader.rar", "test"); + + [Fact] + public async ValueTask Rar5_EncryptedFileAndHeader_Reader_Async() => + await ReadRar_Async("Rar5.encrypted_filesAndHeader.rar", "test"); + + [Fact] + public async ValueTask Rar_EncryptedFileOnly_Reader_Async() => + await ReadRar_Async("Rar.encrypted_filesOnly.rar", "test"); + + [Fact] + public async ValueTask Rar5_EncryptedFileOnly_Reader_Async() => + await ReadRar_Async("Rar5.encrypted_filesOnly.rar", "test"); + + [Fact] + public async ValueTask Rar_Encrypted_Reader_Async() => + await ReadRar_Async("Rar.Encrypted.rar", "test"); + + [Fact] + public async ValueTask Rar5_Encrypted_Reader_Async() => + await ReadRar_Async("Rar5.encrypted_filesOnly.rar", "test"); + + private async ValueTask ReadRar_Async(string testArchive, string password) => + await ReadAsync( + testArchive, + CompressionType.Rar, + ReaderOptions.ForFilePath with + { + Password = password, + } + ); + + [Fact] + public async ValueTask Rar_Entry_Stream_Async() => await DoRar_Entry_Stream_Async("Rar.rar"); + + [Fact] + public async ValueTask Rar5_Entry_Stream_Async() => await DoRar_Entry_Stream_Async("Rar5.rar"); + + private async ValueTask DoRar_Entry_Stream_Async(string filename) + { + using (Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, filename))) + await using (var reader = await ReaderFactory.OpenAsyncReader(new AsyncOnlyStream(stream))) + { + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + Assert.Equal(CompressionType.Rar, reader.Entry.CompressionType); + var entryStream = await reader.OpenEntryStreamAsync(); + try + { + var file = Path.GetFileName(reader.Entry.Key).NotNull(); + var folder = + Path.GetDirectoryName(reader.Entry.Key) + ?? throw new InvalidOperationException( + "Entry key must have a directory name." + ); + var destdir = Path.Combine(SCRATCH_FILES_PATH, folder); + if (!Directory.Exists(destdir)) + { + Directory.CreateDirectory(destdir); + } + var destinationFileName = Path.Combine(destdir, file); + + using var fs = File.OpenWrite(destinationFileName); + await entryStream.CopyToAsync(fs); + } + finally + { +#if NETCOREAPP2_1_OR_GREATER || NETSTANDARD2_1_OR_GREATER + await entryStream.DisposeAsync(); +#else + entryStream.Dispose(); +#endif + } + } + } + } + VerifyFiles(); + } + + [Fact] + public async ValueTask Rar_Reader_Audio_program_Async() + { + using ( + var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Rar.Audio_program.rar")) + ) + await using ( + var reader = await ReaderFactory.OpenAsyncReader( + new AsyncOnlyStream(stream), + ReaderOptions.ForExternalStream with + { + LookForHeader = true, + } + ) + ) + { + while (await reader.MoveToNextEntryAsync()) + { + Assert.Equal(CompressionType.Rar, reader.Entry.CompressionType); + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + CompareFilesByPath( + Path.Combine(SCRATCH_FILES_PATH, "test.dat"), + Path.Combine(MISC_TEST_FILES_PATH, "test.dat") + ); + } + + [Fact] + public async ValueTask Rar_Jpg_Reader_Async() + { + using (var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Rar.jpeg.jpg"))) + using ( + IReader baseReader = RarReader.OpenReader( + stream, + ReaderOptions.ForExternalStream with + { + LookForHeader = true, + } + ) + ) + { + IAsyncReader reader = (IAsyncReader)baseReader; + while (await reader.MoveToNextEntryAsync()) + { + Assert.Equal(CompressionType.Rar, reader.Entry.CompressionType); + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + VerifyFiles(); + } + + [Fact] + public async ValueTask Rar_Solid_Reader_Async() => + await ReadAsync("Rar.solid.rar", CompressionType.Rar); + + [Fact] + public async ValueTask Rar_Comment_Reader_Async() => + await ReadAsync("Rar.comment.rar", CompressionType.Rar); + + [Fact] + public async ValueTask Rar5_Comment_Reader_Async() => + await ReadAsync("Rar5.comment.rar", CompressionType.Rar); + + [Fact] + public async ValueTask Rar5_Solid_Reader_Async() => + await ReadAsync("Rar5.solid.rar", CompressionType.Rar); + + [Fact] + public async ValueTask Rar_Solid_Skip_Reader_Async() => + await DoRar_Solid_Skip_Reader_Async("Rar.solid.rar"); + + [Fact] + public async ValueTask Rar5_Solid_Skip_Reader_Async() => + await DoRar_Solid_Skip_Reader_Async("Rar5.solid.rar"); + + private async ValueTask DoRar_Solid_Skip_Reader_Async(string filename) + { + using var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, filename)); + await using var reader = await ReaderFactory.OpenAsyncReader( + new AsyncOnlyStream(stream), + ReaderOptions.ForExternalStream with + { + LookForHeader = true, + } + ); + while (await reader.MoveToNextEntryAsync()) + { + if (reader.Entry.Key.NotNull().Contains("jpg")) + { + Assert.Equal(CompressionType.Rar, reader.Entry.CompressionType); + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + } + + [Fact] + public async ValueTask Rar_Reader_Skip_Async() => await DoRar_Reader_Skip_Async("Rar.rar"); + + [Fact] + public async ValueTask Rar5_Reader_Skip_Async() => await DoRar_Reader_Skip_Async("Rar5.rar"); + + private async ValueTask DoRar_Reader_Skip_Async(string filename) + { + using var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, filename)); + await using var reader = await ReaderFactory.OpenAsyncReader( + new AsyncOnlyStream(stream), + ReaderOptions.ForExternalStream with + { + LookForHeader = true, + } + ); + while (await reader.MoveToNextEntryAsync()) + { + if (reader.Entry.Key.NotNull().Contains("jpg")) + { + Assert.Equal(CompressionType.Rar, reader.Entry.CompressionType); + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + } + + private async ValueTask ReadAsync( + string testArchive, + CompressionType expectedCompression, + ReaderOptions? readerOptions = null + ) + { + testArchive = Path.Combine(TEST_ARCHIVES_PATH, testArchive); + using Stream stream = File.OpenRead(testArchive); + await using var reader = await ReaderFactory.OpenAsyncReader( + new AsyncOnlyStream(stream), + readerOptions ?? ReaderOptions.ForExternalStream + ); + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + Assert.Equal(expectedCompression, reader.Entry.CompressionType); + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + VerifyFiles(); + } +} diff --git a/tests/SharpCompress.Test/Rar/RarReaderTests.cs b/tests/SharpCompress.Test/Rar/RarReaderTests.cs index 097b0fc8..a037f9bb 100644 --- a/tests/SharpCompress.Test/Rar/RarReaderTests.cs +++ b/tests/SharpCompress.Test/Rar/RarReaderTests.cs @@ -1,6 +1,8 @@ using System; +using System.Collections; using System.IO; using System.Linq; +using SharpCompress.Archives.Rar; using SharpCompress.Common; using SharpCompress.Readers; using SharpCompress.Readers.Rar; @@ -12,36 +14,30 @@ public class RarReaderTests : ReaderTests { [Fact] public void Rar_Multi_Reader() => - DoRar_Multi_Reader( - new[] - { - "Rar.multi.part01.rar", - "Rar.multi.part02.rar", - "Rar.multi.part03.rar", - "Rar.multi.part04.rar", - "Rar.multi.part05.rar", - "Rar.multi.part06.rar" - } - ); + DoRar_Multi_Reader([ + "Rar.multi.part01.rar", + "Rar.multi.part02.rar", + "Rar.multi.part03.rar", + "Rar.multi.part04.rar", + "Rar.multi.part05.rar", + "Rar.multi.part06.rar", + ]); [Fact] public void Rar5_Multi_Reader() => - DoRar_Multi_Reader( - new[] - { - "Rar5.multi.part01.rar", - "Rar5.multi.part02.rar", - "Rar5.multi.part03.rar", - "Rar5.multi.part04.rar", - "Rar5.multi.part05.rar", - "Rar5.multi.part06.rar" - } - ); + DoRar_Multi_Reader([ + "Rar5.multi.part01.rar", + "Rar5.multi.part02.rar", + "Rar5.multi.part03.rar", + "Rar5.multi.part04.rar", + "Rar5.multi.part05.rar", + "Rar5.multi.part06.rar", + ]); private void DoRar_Multi_Reader(string[] archives) { using ( - var reader = RarReader.Open( + var reader = RarReader.OpenReader( archives .Select(s => Path.Combine(TEST_ARCHIVES_PATH, s)) .Select(p => File.OpenRead(p)) @@ -50,10 +46,7 @@ public class RarReaderTests : ReaderTests { while (reader.MoveToNextEntry()) { - reader.WriteEntryToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + reader.WriteEntryToDirectory(SCRATCH_FILES_PATH); } } VerifyFiles(); @@ -61,36 +54,33 @@ public class RarReaderTests : ReaderTests [Fact] public void Rar_Multi_Reader_Encrypted() => - DoRar_Multi_Reader_Encrypted( - new[] - { - "Rar.EncryptedParts.part01.rar", - "Rar.EncryptedParts.part02.rar", - "Rar.EncryptedParts.part03.rar", - "Rar.EncryptedParts.part04.rar", - "Rar.EncryptedParts.part05.rar", - "Rar.EncryptedParts.part06.rar" - } - ); + DoRar_Multi_Reader_Encrypted([ + "Rar.EncryptedParts.part01.rar", + "Rar.EncryptedParts.part02.rar", + "Rar.EncryptedParts.part03.rar", + "Rar.EncryptedParts.part04.rar", + "Rar.EncryptedParts.part05.rar", + "Rar.EncryptedParts.part06.rar", + ]); private void DoRar_Multi_Reader_Encrypted(string[] archives) => Assert.Throws(() => { using ( - var reader = RarReader.Open( + var reader = RarReader.OpenReader( archives .Select(s => Path.Combine(TEST_ARCHIVES_PATH, s)) .Select(p => File.OpenRead(p)), - new ReaderOptions() { Password = "test" } + ReaderOptions.ForExternalStream with + { + Password = "test", + } ) ) { while (reader.MoveToNextEntry()) { - reader.WriteEntryToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + reader.WriteEntryToDirectory(SCRATCH_FILES_PATH); } } VerifyFiles(); @@ -98,31 +88,25 @@ public class RarReaderTests : ReaderTests [Fact] public void Rar_Multi_Reader_Delete_Files() => - DoRar_Multi_Reader_Delete_Files( - new[] - { - "Rar.multi.part01.rar", - "Rar.multi.part02.rar", - "Rar.multi.part03.rar", - "Rar.multi.part04.rar", - "Rar.multi.part05.rar", - "Rar.multi.part06.rar" - } - ); + DoRar_Multi_Reader_Delete_Files([ + "Rar.multi.part01.rar", + "Rar.multi.part02.rar", + "Rar.multi.part03.rar", + "Rar.multi.part04.rar", + "Rar.multi.part05.rar", + "Rar.multi.part06.rar", + ]); [Fact] public void Rar5_Multi_Reader_Delete_Files() => - DoRar_Multi_Reader_Delete_Files( - new[] - { - "Rar5.multi.part01.rar", - "Rar5.multi.part02.rar", - "Rar5.multi.part03.rar", - "Rar5.multi.part04.rar", - "Rar5.multi.part05.rar", - "Rar5.multi.part06.rar" - } - ); + DoRar_Multi_Reader_Delete_Files([ + "Rar5.multi.part01.rar", + "Rar5.multi.part02.rar", + "Rar5.multi.part03.rar", + "Rar5.multi.part04.rar", + "Rar5.multi.part05.rar", + "Rar5.multi.part06.rar", + ]); private void DoRar_Multi_Reader_Delete_Files(string[] archives) { @@ -137,14 +121,11 @@ public class RarReaderTests : ReaderTests .Select(s => Path.Combine(SCRATCH2_FILES_PATH, s)) .Select(File.OpenRead) .ToList(); - using (var reader = RarReader.Open(streams)) + using (var reader = RarReader.OpenReader(streams)) { while (reader.MoveToNextEntry()) { - reader.WriteEntryToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + reader.WriteEntryToDirectory(SCRATCH_FILES_PATH); } } foreach (var stream in streams) @@ -171,36 +152,38 @@ public class RarReaderTests : ReaderTests [Fact] public void Rar5_Reader() => Read("Rar5.rar", CompressionType.Rar); + [Fact] + public void Rar5_CRC_Blake2_Reader() => Read("Rar5.crc_blake2.rar", CompressionType.Rar); + [Fact] public void Rar_EncryptedFileAndHeader_Reader() => ReadRar("Rar.encrypted_filesAndHeader.rar", "test"); - /*[Fact] - public void Rar5_EncryptedFileAndHeader_Reader() - { + [Fact] + public void Rar5_EncryptedFileAndHeader_Reader() => ReadRar("Rar5.encrypted_filesAndHeader.rar", "test"); - }*/ [Fact] public void Rar_EncryptedFileOnly_Reader() => ReadRar("Rar.encrypted_filesOnly.rar", "test"); - /*[Fact] - public void Rar5_EncryptedFileOnly_Reader() - { - ReadRar("Rar5.encrypted_filesOnly.rar", "test"); - }*/ + [Fact] + public void Rar5_EncryptedFileOnly_Reader() => ReadRar("Rar5.encrypted_filesOnly.rar", "test"); [Fact] public void Rar_Encrypted_Reader() => ReadRar("Rar.Encrypted.rar", "test"); - /*[Fact] - public void Rar5_Encrypted_Reader() - { - ReadRar("Rar5.encrypted_filesOnly.rar", "test"); - }*/ + [Fact] + public void Rar5_Encrypted_Reader() => ReadRar("Rar5.encrypted_filesOnly.rar", "test"); private void ReadRar(string testArchive, string password) => - Read(testArchive, CompressionType.Rar, new ReaderOptions { Password = password }); + Read( + testArchive, + CompressionType.Rar, + ReaderOptions.ForFilePath with + { + Password = password, + } + ); [Fact] public void Rar_Entry_Stream() => DoRar_Entry_Stream("Rar.rar"); @@ -211,7 +194,7 @@ public class RarReaderTests : ReaderTests private void DoRar_Entry_Stream(string filename) { using (Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, filename))) - using (var reader = ReaderFactory.Open(stream)) + using (var reader = ReaderFactory.OpenReader(stream)) { while (reader.MoveToNextEntry()) { @@ -219,10 +202,12 @@ public class RarReaderTests : ReaderTests { Assert.Equal(CompressionType.Rar, reader.Entry.CompressionType); using var entryStream = reader.OpenEntryStream(); - var file = Path.GetFileName(reader.Entry.Key); + var file = Path.GetFileName(reader.Entry.Key).NotNull(); var folder = Path.GetDirectoryName(reader.Entry.Key) - ?? throw new ArgumentNullException(); + ?? throw new InvalidOperationException( + "Entry key must have a directory name." + ); var destdir = Path.Combine(SCRATCH_FILES_PATH, folder); if (!Directory.Exists(destdir)) { @@ -231,7 +216,7 @@ public class RarReaderTests : ReaderTests var destinationFileName = Path.Combine(destdir, file); using var fs = File.OpenWrite(destinationFileName); - entryStream.TransferTo(fs); + entryStream.CopyTo(fs); } } } @@ -245,16 +230,19 @@ public class RarReaderTests : ReaderTests var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Rar.Audio_program.rar")) ) using ( - var reader = ReaderFactory.Open(stream, new ReaderOptions() { LookForHeader = true }) + var reader = ReaderFactory.OpenReader( + stream, + ReaderOptions.ForExternalStream with + { + LookForHeader = true, + } + ) ) { while (reader.MoveToNextEntry()) { Assert.Equal(CompressionType.Rar, reader.Entry.CompressionType); - reader.WriteEntryToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + reader.WriteEntryToDirectory(SCRATCH_FILES_PATH); } } CompareFilesByPath( @@ -267,15 +255,20 @@ public class RarReaderTests : ReaderTests public void Rar_Jpg_Reader() { using (var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Rar.jpeg.jpg"))) - using (var reader = RarReader.Open(stream, new ReaderOptions() { LookForHeader = true })) + using ( + var reader = RarReader.OpenReader( + stream, + ReaderOptions.ForExternalStream with + { + LookForHeader = true, + } + ) + ) { while (reader.MoveToNextEntry()) { Assert.Equal(CompressionType.Rar, reader.Entry.CompressionType); - reader.WriteEntryToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + reader.WriteEntryToDirectory(SCRATCH_FILES_PATH); } } VerifyFiles(); @@ -284,6 +277,12 @@ public class RarReaderTests : ReaderTests [Fact] public void Rar_Solid_Reader() => Read("Rar.solid.rar", CompressionType.Rar); + [Fact] + public void Rar_Comment_Reader() => Read("Rar.comment.rar", CompressionType.Rar); + + [Fact] + public void Rar5_Comment_Reader() => Read("Rar5.comment.rar", CompressionType.Rar); + [Fact] public void Rar5_Solid_Reader() => Read("Rar5.solid.rar", CompressionType.Rar); @@ -296,16 +295,19 @@ public class RarReaderTests : ReaderTests private void DoRar_Solid_Skip_Reader(string filename) { using var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, filename)); - using var reader = ReaderFactory.Open(stream, new ReaderOptions() { LookForHeader = true }); + using var reader = ReaderFactory.OpenReader( + stream, + ReaderOptions.ForExternalStream with + { + LookForHeader = true, + } + ); while (reader.MoveToNextEntry()) { - if (reader.Entry.Key.Contains("jpg")) + if (reader.Entry.Key.NotNull().Contains("jpg")) { Assert.Equal(CompressionType.Rar, reader.Entry.CompressionType); - reader.WriteEntryToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + reader.WriteEntryToDirectory(SCRATCH_FILES_PATH); } } } @@ -319,62 +321,118 @@ public class RarReaderTests : ReaderTests private void DoRar_Reader_Skip(string filename) { using var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, filename)); - using var reader = ReaderFactory.Open(stream, new ReaderOptions() { LookForHeader = true }); + using var reader = ReaderFactory.OpenReader( + stream, + ReaderOptions.ForExternalStream with + { + LookForHeader = true, + } + ); while (reader.MoveToNextEntry()) { - if (reader.Entry.Key.Contains("jpg")) + if (reader.Entry.Key.NotNull().Contains("jpg")) { Assert.Equal(CompressionType.Rar, reader.Entry.CompressionType); - reader.WriteEntryToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + reader.WriteEntryToDirectory(SCRATCH_FILES_PATH); } } } [Fact] - public void Rar_NullReference() + public void Rar_SkipEncryptedFilesWithoutPassword() { - { - var archives = new[] + using var stream = File.OpenRead( + Path.Combine(TEST_ARCHIVES_PATH, "Rar.encrypted_filesOnly.rar") + ); + using var reader = ReaderFactory.OpenReader( + stream, + ReaderOptions.ForExternalStream with { - "Rar.EncryptedParts.part01.rar", - "Rar.EncryptedParts.part02.rar", - "Rar.EncryptedParts.part03.rar", - "Rar.EncryptedParts.part04.rar", - "Rar.EncryptedParts.part05.rar", - "Rar.EncryptedParts.part06.rar" - }; - - using ( - var reader = RarReader.Open( - archives - .Select(s => Path.Combine(TEST_ARCHIVES_PATH, s)) - .Select(p => File.OpenRead(p)), - new ReaderOptions() { Password = "test" } - ) - ) - { - while (reader.MoveToNextEntry()) - { - // - } + LookForHeader = true, } + ); + while (reader.MoveToNextEntry()) + { + // } + } - { - using var stream = File.OpenRead( - Path.Combine(TEST_ARCHIVES_PATH, "Rar.encrypted_filesOnly.rar") - ); - using var reader = ReaderFactory.Open( - stream, - new ReaderOptions() { LookForHeader = true } - ); - while (reader.MoveToNextEntry()) + [Fact] + public void Rar_Iterate_Reader() => + Iterate( + "Rar.rar", + "Failure jpg exe Empty jpg\\test.jpg exe\\test.exe тест.txt", + CompressionType.Rar + ); + + [Fact] + public void Rar2_Iterate_Archive() => + Iterate( + "Rar2.rar", + "Failure Empty тест.txt jpg\\test.jpg exe\\test.exe jpg exe", + CompressionType.Rar + ); + + [Fact] + public void Rar4_Iterate_Archive() => + Iterate( + "Rar4.rar", + "Failure Empty jpg exe тест.txt jpg\\test.jpg exe\\test.exe", + CompressionType.Rar + ); + + [Fact] + public void Rar5_Iterate_Archive() => + Iterate( + "Rar5.rar", + "Failure jpg exe Empty тест.txt jpg\\test.jpg exe\\test.exe", + CompressionType.Rar + ); + + [Fact] + public void Rar_Encrypted_Iterate_Archive() => + Iterate( + "Rar.encrypted_filesOnly.rar", + "Failure jpg exe Empty тест.txt jpg\\test.jpg exe\\test.exe", + CompressionType.Rar + ); + + [Fact] + public void Rar5_Encrypted_Iterate_Archive() => + Assert.Throws(() => + Iterate( + "Rar5.encrypted_filesOnly.rar", + "Failure jpg exe Empty тест.txt jpg\\test.jpg exe\\test.exe", + CompressionType.Rar + ) + ); + + [Fact] + public void Rar_Iterate_Multipart() + { + var expectedOrder = new Stack( + new[] { - // + "Failure", + "jpg", + "exe", + "Empty", + "тест.txt", + Path.Combine("jpg", "test.jpg"), + Path.Combine("exe", "test.exe"), } + ); + using var reader = RarReader.OpenReader([ + Path.Combine(TEST_ARCHIVES_PATH, "Rar.multi.part01.rar"), + Path.Combine(TEST_ARCHIVES_PATH, "Rar.multi.part02.rar"), + Path.Combine(TEST_ARCHIVES_PATH, "Rar.multi.part03.rar"), + Path.Combine(TEST_ARCHIVES_PATH, "Rar.multi.part04.rar"), + Path.Combine(TEST_ARCHIVES_PATH, "Rar.multi.part05.rar"), + Path.Combine(TEST_ARCHIVES_PATH, "Rar.multi.part06.rar"), + ]); + while (reader.MoveToNextEntry()) + { + Assert.Equal(expectedOrder.Pop(), reader.Entry.Key); } } } diff --git a/tests/SharpCompress.Test/ReaderFactoryTests.cs b/tests/SharpCompress.Test/ReaderFactoryTests.cs new file mode 100644 index 00000000..2c5236c8 --- /dev/null +++ b/tests/SharpCompress.Test/ReaderFactoryTests.cs @@ -0,0 +1,41 @@ +using System; +using System.IO; +using System.Threading.Tasks; +using SharpCompress.Readers; +using SharpCompress.Readers.Rar; +using SharpCompress.Test.Mocks; +using Xunit; + +namespace SharpCompress.Test; + +public class ReaderFactoryTests +{ + [Fact] + public void OpenReader_Stream_Throws_On_Unreadable_Stream() + { + using var unreadable = new TestStream(new MemoryStream(), false, true, true); + + Assert.Throws(() => ReaderFactory.OpenReader(unreadable)); + } + + [Fact] + public async ValueTask OpenAsyncReader_Stream_Throws_On_Unreadable_Stream() + { + using var unreadable = new TestStream(new MemoryStream(), false, true, true); + + await Assert.ThrowsAsync(() => + ReaderFactory.OpenAsyncReader(unreadable).AsTask() + ); + } + + [Fact] + public void RarReader_StreamCollection_Throws_On_Unreadable_Stream() + { + using var unreadable = new TestStream(new MemoryStream(), false, true, true); + using var readable = new MemoryStream(); + + Assert.Throws(() => + RarReader.OpenReader([unreadable, readable]).MoveToNextEntry() + ); + } +} diff --git a/tests/SharpCompress.Test/ReaderTests.cs b/tests/SharpCompress.Test/ReaderTests.cs index be1dbe07..4c9024c9 100644 --- a/tests/SharpCompress.Test/ReaderTests.cs +++ b/tests/SharpCompress.Test/ReaderTests.cs @@ -1,5 +1,12 @@ +using System; +using System.Collections.Generic; using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using AwesomeAssertions; using SharpCompress.Common; +using SharpCompress.Factories; using SharpCompress.IO; using SharpCompress.Readers; using SharpCompress.Test.Mocks; @@ -9,62 +16,263 @@ namespace SharpCompress.Test; public abstract class ReaderTests : TestBase { + protected void Read(string testArchive, ReaderOptions? options = null) => + ReadCore(testArchive, options, ReadImpl); + protected void Read( string testArchive, CompressionType expectedCompression, ReaderOptions? options = null + ) => ReadCore(testArchive, options, (path, opts) => ReadImpl(path, expectedCompression, opts)); + + private void ReadCore( + string testArchive, + ReaderOptions? options, + Action readImpl ) { testArchive = Path.Combine(TEST_ARCHIVES_PATH, testArchive); + options ??= ReaderOptions.ForFilePath.WithBufferSize(0x20000); - options ??= new ReaderOptions(); + var optionsWithStreamOpen = options.WithLeaveStreamOpen(true); + readImpl(testArchive, optionsWithStreamOpen); - options.LeaveStreamOpen = true; - ReadImpl(testArchive, expectedCompression, options); + var optionsWithStreamClosed = options.WithLeaveStreamOpen(false); + readImpl(testArchive, optionsWithStreamClosed); - options.LeaveStreamOpen = false; - ReadImpl(testArchive, expectedCompression, options); VerifyFiles(); } + private void ReadImpl(string testArchive, ReaderOptions options) => + ReadImplCore(testArchive, options, UseReader); + private void ReadImpl( string testArchive, CompressionType expectedCompression, ReaderOptions options - ) + ) => ReadImplCore(testArchive, options, r => UseReader(r, expectedCompression)); + + private void ReadImplCore(string testArchive, ReaderOptions options, Action useReader) { using var file = File.OpenRead(testArchive); - using var protectedStream = NonDisposingStream.Create( - new ForwardOnlyStream(file), - throwOnDispose: true + using var protectedStream = SharpCompressStream.CreateNonDisposing( + new ForwardOnlyStream(file, options.BufferSize) ); using var testStream = new TestStream(protectedStream); - using (var reader = ReaderFactory.Open(testStream, options)) + using (var reader = ReaderFactory.OpenReader(testStream, options)) { - UseReader(reader, expectedCompression); - protectedStream.ThrowOnDispose = false; - Assert.False(testStream.IsDisposed, "{nameof(testStream)} prematurely closed"); + useReader(reader); + Assert.False(testStream.IsDisposed, $"{nameof(testStream)} prematurely closed"); } - // Boolean XOR -- If the stream should be left open (true), then the stream should not be diposed (false) - // and if the stream should be closed (false), then the stream should be disposed (true) var message = $"{nameof(options.LeaveStreamOpen)} is set to '{options.LeaveStreamOpen}', so {nameof(testStream.IsDisposed)} should be set to '{!testStream.IsDisposed}', but is set to {testStream.IsDisposed}"; Assert.True(options.LeaveStreamOpen != testStream.IsDisposed, message); } - public void UseReader(IReader reader, CompressionType expectedCompression) + protected void UseReader(IReader reader, CompressionType expectedCompression) { while (reader.MoveToNextEntry()) { if (!reader.Entry.IsDirectory) { Assert.Equal(expectedCompression, reader.Entry.CompressionType); - reader.WriteEntryToDirectory( + reader.WriteEntryToDirectory(SCRATCH_FILES_PATH); + } + } + } + + private void UseReader(IReader reader) + { + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + reader.WriteEntryToDirectory(SCRATCH_FILES_PATH); + } + } + } + + protected async Task AssertArchiveAsync( + string testArchive, + CancellationToken cancellationToken = default + ) + { + testArchive = Path.Combine(TEST_ARCHIVES_PATH, testArchive); + var factory = new TarFactory(); + ( + await factory.IsArchiveAsync( + new FileInfo(testArchive).OpenRead(), + ReaderOptions.ForExternalStream, + cancellationToken + ) + ) + .Should() + .BeTrue(); + ( + await factory.IsArchiveAsync( + new FileInfo(testArchive).OpenRead(), + ReaderOptions.ForExternalStream, + cancellationToken: cancellationToken + ) + ) + .Should() + .BeTrue(); + } + + protected async Task ReadAsync( + string testArchive, + CompressionType? expectedCompression = null, + ReaderOptions? options = null, + CancellationToken cancellationToken = default + ) + { + testArchive = Path.Combine(TEST_ARCHIVES_PATH, testArchive); + + options ??= ReaderOptions.ForExternalStream.WithBufferSize(0x20000); + + var optionsWithStreamOpen = options.WithLeaveStreamOpen(true); + await ReadImplAsync( + testArchive, + expectedCompression, + optionsWithStreamOpen, + cancellationToken + ); + + var optionsWithStreamClosed = options.WithLeaveStreamOpen(false); + await ReadImplAsync( + testArchive, + expectedCompression, + optionsWithStreamClosed, + cancellationToken + ); + + VerifyFiles(); + } + + private async ValueTask ReadImplAsync( + string testArchive, + CompressionType? expectedCompression, + ReaderOptions options, + CancellationToken cancellationToken = default + ) + { + using var file = File.OpenRead(testArchive); + +#if !LEGACY_DOTNET + await using var protectedStream = SharpCompressStream.CreateNonDisposing( + new ForwardOnlyStream(file, options.BufferSize) + ); + await using var testStream = new TestStream(protectedStream); +#else + + using var protectedStream = SharpCompressStream.CreateNonDisposing( + new ForwardOnlyStream(file, options.BufferSize) + ); + using var testStream = new TestStream(protectedStream); +#endif + await using ( + var reader = await ReaderFactory.OpenAsyncReader( + new AsyncOnlyStream(testStream), + options, + cancellationToken + ) + ) + { + await UseReaderAsync(reader, expectedCompression, cancellationToken); + Assert.False(testStream.IsDisposed, $"{nameof(testStream)} prematurely closed"); + } + + var message = + $"{nameof(options.LeaveStreamOpen)} is set to '{options.LeaveStreamOpen}', so {nameof(testStream.IsDisposed)} should be set to '{!testStream.IsDisposed}', but is set to {testStream.IsDisposed}"; + Assert.True(options.LeaveStreamOpen != testStream.IsDisposed, message); + } + + public async ValueTask UseReaderAsync( + IAsyncReader reader, + CompressionType? expectedCompression, + CancellationToken cancellationToken = default + ) + { + while (await reader.MoveToNextEntryAsync(cancellationToken)) + { + if (!reader.Entry.IsDirectory) + { + if (expectedCompression.HasValue) + { + Assert.Equal(expectedCompression, reader.Entry.CompressionType); + } + + await reader.WriteEntryToDirectoryAsync( SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } + cancellationToken: cancellationToken ); } } } + + protected void ReadForBufferBoundaryCheck(string fileName, CompressionType compressionType) + { + using var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, fileName)); + using var reader = ReaderFactory.OpenReader( + stream, + ReaderOptions.ForExternalStream with + { + LookForHeader = true, + } + ); + + while (reader.MoveToNextEntry()) + { + Assert.Equal(compressionType, reader.Entry.CompressionType); + + reader.WriteEntryToDirectory(SCRATCH_FILES_PATH); + } + + CompareFilesByPath( + Path.Combine(SCRATCH_FILES_PATH, "alice29.txt"), + Path.Combine(MISC_TEST_FILES_PATH, "alice29.txt") + ); + } + + protected void Iterate( + string testArchive, + string fileOrder, + CompressionType expectedCompression, + ReaderOptions? options = null + ) + { + if (!Environment.OSVersion.IsWindows()) + { + fileOrder = fileOrder.Replace('\\', '/'); + } + var expected = new Stack(fileOrder.Split(' ')); + + testArchive = Path.Combine(TEST_ARCHIVES_PATH, testArchive); + using var file = File.OpenRead(testArchive); + using var forward = new ForwardOnlyStream(file); + using var reader = ReaderFactory.OpenReader(forward, options); + while (reader.MoveToNextEntry()) + { + Assert.Equal(expectedCompression, reader.Entry.CompressionType); + Assert.Equal(expected.Pop(), reader.Entry.Key); + } + } + + protected void DoMultiReader( + string[] archives, + Func, IReader> readerFactory + ) + { + using var reader = readerFactory( + archives.Select(s => Path.Combine(TEST_ARCHIVES_PATH, s)).Select(File.OpenRead) + ); + + while (reader.MoveToNextEntry()) + { + reader.WriteEntryToDirectory(SCRATCH_FILES_PATH); + } + + VerifyFiles(); + } } diff --git a/tests/SharpCompress.Test/RemainingCrcExtractionTests.cs b/tests/SharpCompress.Test/RemainingCrcExtractionTests.cs new file mode 100644 index 00000000..45034123 --- /dev/null +++ b/tests/SharpCompress.Test/RemainingCrcExtractionTests.cs @@ -0,0 +1,75 @@ +using System; +using System.IO; +using SharpCompress.Common; +using SharpCompress.Compressors.LZMA; +using SharpCompress.Readers; +using Xunit; + +namespace SharpCompress.Test; + +public class RemainingCrcExtractionTests : TestBase +{ + [Theory] + [InlineData("Arj.store.arj", "This")] + [InlineData("Ace.store.ace", "This")] + [InlineData("Arc.uncompressed.arc", "This")] + public void Reader_WriteEntryToFile_Throws_On_Checksum_Mismatch( + string archiveName, + string payloadMarker + ) + { + using var stream = new MemoryStream(ReadCorruptedArchive(archiveName, payloadMarker)); + using var reader = ReaderFactory.OpenReader(stream); + var destination = Path.Combine(SCRATCH_FILES_PATH, Guid.NewGuid().ToString()); + Directory.CreateDirectory(destination); + + Assert.Throws(() => reader.WriteAllToDirectory(destination)); + } + + [Theory] + [InlineData("Arj.store.arj", "This")] + [InlineData("Ace.store.ace", "This")] + [InlineData("Arc.uncompressed.arc", "This")] + public void Reader_WriteEntryToFile_Skips_Checksum_When_CheckCrc_Is_False( + string archiveName, + string payloadMarker + ) + { + using var stream = new MemoryStream(ReadCorruptedArchive(archiveName, payloadMarker)); + using var reader = ReaderFactory.OpenReader(stream); + var destination = Path.Combine(SCRATCH_FILES_PATH, Guid.NewGuid().ToString()); + Directory.CreateDirectory(destination); + + reader.WriteAllToDirectory(destination, new ExtractionOptions { CheckCrc = false }); + + Assert.True(Directory.GetFiles(destination, "*", SearchOption.AllDirectories).Length > 0); + } + + [Fact] + public void LZipStream_Throws_On_Trailer_Crc_Mismatch() + { + var bytes = File.ReadAllBytes(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.lz")); + bytes[^20] ^= 1; + using var stream = LZipStream.Create( + new MemoryStream(bytes), + SharpCompress.Compressors.CompressionMode.Decompress + ); + using var output = new MemoryStream(); + + Assert.Throws(() => stream.CopyTo(output)); + } + + private static byte[] ReadCorruptedArchive(string archiveName, string payloadMarker) + { + var bytes = File.ReadAllBytes(Path.Combine(TEST_ARCHIVES_PATH, archiveName)); + var marker = System.Text.Encoding.ASCII.GetBytes(payloadMarker); + var offset = bytes.AsSpan().IndexOf(marker); + if (offset < 0) + { + throw new InvalidOperationException($"Payload marker '{payloadMarker}' was not found."); + } + + bytes[offset] ^= 1; + return bytes; + } +} diff --git a/tests/SharpCompress.Test/Security/ExtractionPathTraversalTests.cs b/tests/SharpCompress.Test/Security/ExtractionPathTraversalTests.cs new file mode 100644 index 00000000..69ebddd8 --- /dev/null +++ b/tests/SharpCompress.Test/Security/ExtractionPathTraversalTests.cs @@ -0,0 +1,238 @@ +#if NET8_0_OR_GREATER +using System; +using System.IO; +using System.Linq; +using System.Threading.Tasks; +using SharpCompress.Archives; +using SharpCompress.Common; +using SharpCompress.Readers; +using Xunit; +using SysZip = System.IO.Compression.ZipArchive; +using SysZipMode = System.IO.Compression.ZipArchiveMode; + +namespace SharpCompress.Test.Security; + +public class ExtractionPathTraversalTests : TestBase +{ + [Theory] + [InlineData("ReaderAll")] + [InlineData("ReaderEntry")] + [InlineData("Archive")] + [InlineData("ArchiveEntry")] + [InlineData("AsyncReaderAll")] + [InlineData("AsyncReaderEntry")] + [InlineData("AsyncArchive")] + [InlineData("AsyncArchiveEntry")] + public async Task DirectoryTraversalToExistingOutsideDirectory_ShouldThrow(string api) + { + var extractDir = Path.Combine(SCRATCH_FILES_PATH, "extract"); + Directory.CreateDirectory(extractDir); + var escapedDirectory = Path.GetFullPath(Path.Combine(extractDir, "../../escaped_existing")); + Directory.CreateDirectory(escapedDirectory); + var archivePath = Path.Combine(SCRATCH2_FILES_PATH, $"{api}.zip"); + BuildZip(archivePath, "../../escaped_existing/"); + + var exception = await RecordExtractionExceptionAsync(api, archivePath, extractDir); + + var extractionException = Assert.IsType(exception); + Assert.Contains("outside of the destination", extractionException.Message); + } + + [Theory] + [InlineData("ReaderAll")] + [InlineData("ReaderEntry")] + [InlineData("Archive")] + [InlineData("ArchiveEntry")] + [InlineData("AsyncReaderAll")] + [InlineData("AsyncReaderEntry")] + [InlineData("AsyncArchive")] + [InlineData("AsyncArchiveEntry")] + public async Task FileTraversalToSiblingDirectory_ShouldThrow(string api) + { + var extractDir = Path.Combine(SCRATCH_FILES_PATH, "extract"); + Directory.CreateDirectory(extractDir); + var siblingDirectory = Path.Combine(SCRATCH_FILES_PATH, "extract2"); + Directory.CreateDirectory(siblingDirectory); + var archivePath = Path.Combine(SCRATCH2_FILES_PATH, $"{api}.zip"); + BuildZip(archivePath, "../extract2/evil.txt"); + + var exception = await RecordExtractionExceptionAsync(api, archivePath, extractDir); + + var extractionException = Assert.IsType(exception); + Assert.Contains("outside of the destination", extractionException.Message); + Assert.False(File.Exists(Path.Combine(siblingDirectory, "evil.txt"))); + } + + private static void BuildZip(string path, string entryName) + { + using var fs = File.Create(path); + using var zip = new SysZip(fs, SysZipMode.Create); + var entry = zip.CreateEntry(entryName); + + if (entryName.EndsWith('/')) + { + return; + } + + using var writer = new StreamWriter(entry.Open()); + writer.Write("evil"); + } + + private static async Task RecordExtractionExceptionAsync( + string api, + string archivePath, + string extractDir + ) + { + var options = new ExtractionOptions { ExtractFullPath = true, Overwrite = true }; + + return api switch + { + "ReaderAll" => RecordException(() => + ExtractWithReaderAll(archivePath, extractDir, options) + ), + "ReaderEntry" => RecordException(() => + ExtractWithReaderEntry(archivePath, extractDir, options) + ), + "Archive" => RecordException(() => + ExtractWithArchive(archivePath, extractDir, options) + ), + "ArchiveEntry" => RecordException(() => + ExtractWithArchiveEntry(archivePath, extractDir, options) + ), + "AsyncReaderAll" => await RecordExceptionAsync(() => + ExtractWithAsyncReaderAllAsync(archivePath, extractDir, options) + ), + "AsyncReaderEntry" => await RecordExceptionAsync(() => + ExtractWithAsyncReaderEntryAsync(archivePath, extractDir, options) + ), + "AsyncArchive" => await RecordExceptionAsync(() => + ExtractWithAsyncArchiveAsync(archivePath, extractDir, options) + ), + "AsyncArchiveEntry" => await RecordExceptionAsync(() => + ExtractWithAsyncArchiveEntryAsync(archivePath, extractDir, options) + ), + _ => throw new ArgumentOutOfRangeException(nameof(api), api, null), + }; + } + + private static Exception? RecordException(Action action) + { + try + { + action(); + return null; + } + catch (Exception exception) + { + return exception; + } + } + + private static async Task RecordExceptionAsync(Func action) + { + try + { + await action(); + return null; + } + catch (Exception exception) + { + return exception; + } + } + + private static void ExtractWithReaderAll( + string archivePath, + string extractDir, + ExtractionOptions options + ) + { + using var stream = File.OpenRead(archivePath); + using var reader = ReaderFactory.OpenReader(stream); + reader.WriteAllToDirectory(extractDir, options); + } + + private static void ExtractWithReaderEntry( + string archivePath, + string extractDir, + ExtractionOptions options + ) + { + using var stream = File.OpenRead(archivePath); + using var reader = ReaderFactory.OpenReader(stream); + Assert.True(reader.MoveToNextEntry()); + reader.WriteEntryToDirectory(extractDir, options); + } + + private static void ExtractWithArchive( + string archivePath, + string extractDir, + ExtractionOptions options + ) + { + using var archive = ArchiveFactory.OpenArchive(archivePath); + archive.WriteToDirectory(extractDir, options); + } + + private static void ExtractWithArchiveEntry( + string archivePath, + string extractDir, + ExtractionOptions options + ) + { + using var archive = ArchiveFactory.OpenArchive(archivePath); + archive.Entries.Single().WriteToDirectory(extractDir, options); + } + + private static async Task ExtractWithAsyncReaderAllAsync( + string archivePath, + string extractDir, + ExtractionOptions options + ) + { + using var stream = File.OpenRead(archivePath); + await using var reader = await ReaderFactory.OpenAsyncReader(stream); + await reader.WriteAllToDirectoryAsync(extractDir, options); + } + + private static async Task ExtractWithAsyncReaderEntryAsync( + string archivePath, + string extractDir, + ExtractionOptions options + ) + { + using var stream = File.OpenRead(archivePath); + await using var reader = await ReaderFactory.OpenAsyncReader(stream); + Assert.True(await reader.MoveToNextEntryAsync()); + await reader.WriteEntryToDirectoryAsync(extractDir, options); + } + + private static async Task ExtractWithAsyncArchiveAsync( + string archivePath, + string extractDir, + ExtractionOptions options + ) + { + await using var archive = await ArchiveFactory.OpenAsyncArchive(archivePath); + await archive.WriteToDirectoryAsync(extractDir, options); + } + + private static async Task ExtractWithAsyncArchiveEntryAsync( + string archivePath, + string extractDir, + ExtractionOptions options + ) + { + await using var archive = await ArchiveFactory.OpenAsyncArchive(archivePath); + + await foreach (var entry in archive.EntriesAsync) + { + await entry.WriteToDirectoryAsync(extractDir, options); + return; + } + + throw new InvalidOperationException("Archive did not contain an entry."); + } +} +#endif diff --git a/tests/SharpCompress.Test/Security/ZipSlip.cs b/tests/SharpCompress.Test/Security/ZipSlip.cs new file mode 100644 index 00000000..208c3192 --- /dev/null +++ b/tests/SharpCompress.Test/Security/ZipSlip.cs @@ -0,0 +1,134 @@ +#if NET8_0_OR_GREATER +using System; +using System.IO; +using System.Threading.Tasks; +using AwesomeAssertions; +using SharpCompress.Archives; +using SharpCompress.Common; +using Xunit; +using SysZip = System.IO.Compression.ZipArchive; +using SysZipMode = System.IO.Compression.ZipArchiveMode; + +namespace SharpCompress.Test.Security; + +public class ZipSlip : TestBase +{ + [Fact] + public void RunSync() + { + Console.WriteLine("--- Sync: archive.WriteToDirectory() ---"); + var (extractDir, parentDir) = SetupDirs("sync"); + Directory.CreateDirectory(extractDir); + var archivePath = Path.Combine(parentDir, "malicious.zip"); + + BuildMaliciousZip(archivePath); + + using (var archive = ArchiveFactory.OpenArchive(archivePath)) + { + var ex = Assert.Throws(() => + archive.WriteToDirectory( + extractDir, + new ExtractionOptions { ExtractFullPath = true } + ) + ); + ex.Message.Should() + .Contain( + "Entry is trying to create a directory outside of the destination directory" + ); + } + + CheckResults(archivePath, parentDir, extractDir); + } + + [Fact] + public async Task RunAsync() + { + Console.WriteLine("--- Async: archive.WriteToDirectoryAsync() ---"); + var (extractDir, parentDir) = SetupDirs("async"); + Directory.CreateDirectory(extractDir); + var archivePath = Path.Combine(parentDir, "malicious.zip"); + + BuildMaliciousZip(archivePath); + + var archive = await ArchiveFactory.OpenAsyncArchive(archivePath); + await using (archive) + { + var ex = await Assert.ThrowsAsync(async () => + await archive.WriteToDirectoryAsync( + extractDir, + new ExtractionOptions { ExtractFullPath = true } + ) + ); + ex.Message.Should() + .Contain( + "Entry is trying to create a directory outside of the destination directory" + ); + } + + CheckResults(archivePath, parentDir, extractDir); + } + + // Craft a ZIP with malicious directory entries using System.IO.Compression + // so we bypass any SharpCompress write-side normalisation. + static void BuildMaliciousZip(string path) + { + using var fs = File.Create(path); + using var zip = new SysZip(fs, SysZipMode.Create); + + // 1. Relative traversal: two levels up, then "escaped_relative/" + zip.CreateEntry("../../escaped_relative/"); + + // 2. Absolute Unix path (Path.Combine discards the base when second arg is rooted) + zip.CreateEntry("/tmp/escaped_absolute/"); + + // 3. A legitimate entry for contrast + zip.CreateEntry("safe_subdir/"); + } + + private (string extractDir, string parentDir) SetupDirs(string label) + { + var parentDir = Path.Combine( + SCRATCH_FILES_PATH, + $"sc_poc_{label}_{Path.GetRandomFileName()}" + ); + Directory.CreateDirectory(parentDir); + var extractDir = Path.Combine(parentDir, "extract_target"); + + Console.WriteLine($" Parent : {parentDir}"); + Console.WriteLine($" Target : {extractDir}"); + return (extractDir, parentDir); + } + + static void CheckResults(string archivePath, string parentDir, string extractDir) + { + Console.WriteLine(" Directories created after extraction:"); + foreach (var d in Directory.GetDirectories(parentDir, "*", SearchOption.AllDirectories)) + { + var relative = Path.GetRelativePath(parentDir, d); + var escaped = !d.StartsWith(extractDir, StringComparison.Ordinal); + Console.WriteLine($" {(escaped ? "[ESCAPED]" : "[ ok ]")} {relative}"); + } + + // Relative traversal "../../escaped_relative/" escapes two levels above extractDir + // (which is parentDir/extract_target), landing in Path.GetTempPath() + var relTarget = Path.GetFullPath(Path.Combine(extractDir, "../../escaped_relative")); + if (Directory.Exists(relTarget)) + { + Console.WriteLine($" [ESCAPED] relative traversal created: {relTarget}"); + Directory.Delete(relTarget); + } + File.Delete(archivePath); + if (Directory.Exists(extractDir)) + { + Directory.Delete(extractDir); + } + + var absTarget = "/tmp/escaped_absolute"; + if (Directory.Exists(absTarget)) + { + Console.WriteLine($" [ESCAPED] absolute path created: {absTarget}"); + Directory.Delete(absTarget); + } + } +} +#endif diff --git a/tests/SharpCompress.Test/SevenZip/SevenZipArchiveAsyncTests.cs b/tests/SharpCompress.Test/SevenZip/SevenZipArchiveAsyncTests.cs new file mode 100644 index 00000000..d28b6d54 --- /dev/null +++ b/tests/SharpCompress.Test/SevenZip/SevenZipArchiveAsyncTests.cs @@ -0,0 +1,373 @@ +using System; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Archives; +using SharpCompress.Archives.SevenZip; +using SharpCompress.Common; +using SharpCompress.Readers; +using SharpCompress.Test.Mocks; +using Xunit; + +namespace SharpCompress.Test.SevenZip; + +public class SevenZipArchiveAsyncTests : ArchiveTests +{ + [Fact] + public async Task SevenZipArchive_LZMA_AsyncStreamExtraction() + { + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "7Zip.LZMA.7z"); +#if NETFRAMEWORK + using var stream = File.OpenRead(testArchive); +#else + await using var stream = File.OpenRead(testArchive); +#endif + await using var archive = await ArchiveFactory.OpenAsyncArchive( + new AsyncOnlyStream(stream) + ); + + await foreach (var entry in archive.EntriesAsync.Where(entry => !entry.IsDirectory)) + { + var targetPath = Path.Combine(SCRATCH_FILES_PATH, entry.Key!); + var targetDir = Path.GetDirectoryName(targetPath); + + if (!string.IsNullOrEmpty(targetDir) && !Directory.Exists(targetDir)) + { + Directory.CreateDirectory(targetDir); + } + +#if NETFRAMEWORK + using var sourceStream = await entry.OpenEntryStreamAsync(CancellationToken.None); +#else + await using var sourceStream = await entry.OpenEntryStreamAsync(CancellationToken.None); +#endif +#if NETFRAMEWORK + using var targetStream = File.Create(targetPath); +#else + await using var targetStream = File.Create(targetPath); +#endif +#if NETFRAMEWORK + await sourceStream.CopyToAsync(targetStream, 81920, CancellationToken.None); +#else + await sourceStream.CopyToAsync(targetStream, CancellationToken.None); +#endif + } + + VerifyFiles(); + } + + //[Fact] + public async Task SevenZipArchive_LZMA2_AsyncStreamExtraction() + { + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "7Zip.LZMA2.7z"); +#if NETFRAMEWORK + using var stream = File.OpenRead(testArchive); +#else + await using var stream = File.OpenRead(testArchive); +#endif + await using var archive = await ArchiveFactory.OpenAsyncArchive( + new AsyncOnlyStream(stream) + ); + + await foreach (var entry in archive.EntriesAsync.Where(entry => !entry.IsDirectory)) + { + var targetPath = Path.Combine(SCRATCH_FILES_PATH, entry.Key!); + var targetDir = Path.GetDirectoryName(targetPath); + + if (!string.IsNullOrEmpty(targetDir) && !Directory.Exists(targetDir)) + { + Directory.CreateDirectory(targetDir); + } + +#if NETFRAMEWORK + using var sourceStream = await entry.OpenEntryStreamAsync(CancellationToken.None); +#else + await using var sourceStream = await entry.OpenEntryStreamAsync(CancellationToken.None); +#endif +#if NETFRAMEWORK + using var targetStream = File.Create(targetPath); +#else + await using var targetStream = File.Create(targetPath); +#endif +#if NETFRAMEWORK + await sourceStream.CopyToAsync(targetStream, 81920, CancellationToken.None); +#else + await sourceStream.CopyToAsync(targetStream, CancellationToken.None); +#endif + } + + VerifyFiles(); + } + + [Fact] + public async Task SevenZipArchive_Solid_AsyncStreamExtraction() + { + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "7Zip.solid.7z"); +#if NETFRAMEWORK + using var stream = File.OpenRead(testArchive); +#else + await using var stream = File.OpenRead(testArchive); +#endif + await using var archive = await ArchiveFactory.OpenAsyncArchive( + new AsyncOnlyStream(stream) + ); + + await foreach (var entry in archive.EntriesAsync.Where(entry => !entry.IsDirectory)) + { + var targetPath = Path.Combine(SCRATCH_FILES_PATH, entry.Key!); + var targetDir = Path.GetDirectoryName(targetPath); + + if (!string.IsNullOrEmpty(targetDir) && !Directory.Exists(targetDir)) + { + Directory.CreateDirectory(targetDir); + } + +#if NETFRAMEWORK + using var sourceStream = await entry.OpenEntryStreamAsync(CancellationToken.None); +#else + await using var sourceStream = await entry.OpenEntryStreamAsync(CancellationToken.None); +#endif +#if NETFRAMEWORK + using var targetStream = File.Create(targetPath); +#else + await using var targetStream = File.Create(targetPath); +#endif +#if NETFRAMEWORK + await sourceStream.CopyToAsync(targetStream, 81920, CancellationToken.None); +#else + await sourceStream.CopyToAsync(targetStream, CancellationToken.None); +#endif + } + + VerifyFiles(); + } + + [Fact] + public async Task SevenZipArchive_Solid_WriteToDirectoryAsync_WithProgress() + { + var progressReports = new System.Collections.Generic.List(); + var progress = new SynchronousProgress(report => + progressReports.Add(report) + ); + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "7Zip.solid.7z"); +#if NETFRAMEWORK + using var stream = File.OpenRead(testArchive); +#else + await using var stream = File.OpenRead(testArchive); +#endif + await using var archive = await ArchiveFactory.OpenAsyncArchive( + new AsyncOnlyStream(stream) + ); + + await archive.WriteToDirectoryAsync(SCRATCH_FILES_PATH, progress: progress); + + VerifyFiles(); + Assert.True(progressReports.Count > 0, "Progress reports should be generated"); + } + + [Fact] + public async Task SevenZipArchive_BZip2_AsyncStreamExtraction() + { + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "7Zip.BZip2.7z"); +#if NETFRAMEWORK + using var stream = File.OpenRead(testArchive); +#else + await using var stream = File.OpenRead(testArchive); +#endif + await using var archive = await ArchiveFactory.OpenAsyncArchive( + new AsyncOnlyStream(stream) + ); + + await foreach (var entry in archive.EntriesAsync.Where(entry => !entry.IsDirectory)) + { + var targetPath = Path.Combine(SCRATCH_FILES_PATH, entry.Key!); + var targetDir = Path.GetDirectoryName(targetPath); + + if (!string.IsNullOrEmpty(targetDir) && !Directory.Exists(targetDir)) + { + Directory.CreateDirectory(targetDir); + } + +#if NETFRAMEWORK + using var sourceStream = await entry.OpenEntryStreamAsync(CancellationToken.None); +#else + await using var sourceStream = await entry.OpenEntryStreamAsync(CancellationToken.None); +#endif +#if NETFRAMEWORK + using var targetStream = File.Create(targetPath); +#else + await using var targetStream = File.Create(targetPath); +#endif +#if NETFRAMEWORK + await sourceStream.CopyToAsync(targetStream, 81920, CancellationToken.None); +#else + await sourceStream.CopyToAsync(targetStream, CancellationToken.None); +#endif + } + + VerifyFiles(); + } + + [Fact] + public async Task SevenZipArchive_PPMd_AsyncStreamExtraction() + { + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "7Zip.PPMd.7z"); +#if NETFRAMEWORK + using var stream = File.OpenRead(testArchive); +#else + await using var stream = File.OpenRead(testArchive); +#endif + await using var archive = await ArchiveFactory.OpenAsyncArchive( + new AsyncOnlyStream(stream) + ); + + await foreach (var entry in archive.EntriesAsync.Where(entry => !entry.IsDirectory)) + { + var targetPath = Path.Combine(SCRATCH_FILES_PATH, entry.Key!); + var targetDir = Path.GetDirectoryName(targetPath); + + if (!string.IsNullOrEmpty(targetDir) && !Directory.Exists(targetDir)) + { + Directory.CreateDirectory(targetDir); + } + +#if NETFRAMEWORK + using var sourceStream = await entry.OpenEntryStreamAsync(CancellationToken.None); +#else + await using var sourceStream = await entry.OpenEntryStreamAsync(CancellationToken.None); +#endif +#if NETFRAMEWORK + using var targetStream = File.Create(targetPath); +#else + await using var targetStream = File.Create(targetPath); +#endif +#if NETFRAMEWORK + await sourceStream.CopyToAsync(targetStream, 81920, CancellationToken.None); +#else + await sourceStream.CopyToAsync(targetStream, CancellationToken.None); +#endif + } + + VerifyFiles(); + } + + [Fact] + public async Task SevenZipArchive_TestSolidDetectionAsync() + { + await using var oneBlockSolidArchive = await SevenZipArchive.OpenAsyncArchive( + Path.Combine(TEST_ARCHIVES_PATH, "7Zip.solid.1block.7z") + ); + Assert.True(await oneBlockSolidArchive.IsSolidAsync()); + + await using var solidArchive = await SevenZipArchive.OpenAsyncArchive( + Path.Combine(TEST_ARCHIVES_PATH, "7Zip.solid.7z") + ); + Assert.True(await solidArchive.IsSolidAsync()); + + await using var nonSolidArchive = await SevenZipArchive.OpenAsyncArchive( + Path.Combine(TEST_ARCHIVES_PATH, "7Zip.nonsolid.7z") + ); + Assert.False(await nonSolidArchive.IsSolidAsync()); + } + + [Fact] + public async Task SevenZipArchive_Solid_ExtractAllEntries_Contiguous_Async() + { + // This test verifies that solid archives iterate entries as contiguous streams + // rather than recreating the decompression stream for each entry + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "7Zip.solid.7z"); + await using var archive = await SevenZipArchive.OpenAsyncArchive(testArchive); + Assert.True(await archive.IsSolidAsync()); + + await using var reader = await archive.ExtractAllEntriesAsync(); + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + + VerifyFiles(); + } + + [Fact] + public async Task SevenZipArchive_Solid_VerifyStreamReuse() + { + // This test verifies that the folder stream is reused within each folder + // and not recreated for each entry in solid archives + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "7Zip.solid.7z"); + await using var archive = await SevenZipArchive.OpenAsyncArchive(testArchive); + Assert.True(await archive.IsSolidAsync()); + + await using var reader = await archive.ExtractAllEntriesAsync(); + + var sevenZipReader = Assert.IsType(reader); + sevenZipReader.DiagnosticsEnabled = true; + + Stream? currentFolderStreamInstance = null; + object? currentFolder = null; + var entryCount = 0; + var entriesInCurrentFolder = 0; + var streamRecreationsWithinFolder = 0; + + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + // Extract the entry to trigger GetEntryStream + using var entryStream = await reader.OpenEntryStreamAsync(); + var buffer = new byte[4096]; + while (entryStream.Read(buffer, 0, buffer.Length) > 0) + { + // Read the stream to completion + } + + entryCount++; + + var folderStream = sevenZipReader.DiagnosticsCurrentFolderStream; + var folder = sevenZipReader.DiagnosticsCurrentFolder; + + Assert.NotNull(folderStream); // Folder stream should exist + + // Check if we're in a new folder + if (currentFolder == null || !ReferenceEquals(currentFolder, folder)) + { + // Starting a new folder + currentFolder = folder; + currentFolderStreamInstance = folderStream; + entriesInCurrentFolder = 1; + } + else + { + // Same folder - verify stream wasn't recreated + entriesInCurrentFolder++; + + if (!ReferenceEquals(currentFolderStreamInstance, folderStream)) + { + // Stream was recreated within the same folder - this is the bug we're testing for! + streamRecreationsWithinFolder++; + } + + currentFolderStreamInstance = folderStream; + } + } + } + + // Verify we actually tested multiple entries + Assert.True(entryCount > 1, "Test should have multiple entries to verify stream reuse"); + + // The critical check: within a single folder, the stream should NEVER be recreated + Assert.Equal(0, streamRecreationsWithinFolder); // Folder stream should remain the same for all entries in the same folder + } + + private sealed class SynchronousProgress : IProgress + { + private readonly Action _handler; + + public SynchronousProgress(Action handler) => _handler = handler; + + public void Report(T value) => _handler(value); + } +} diff --git a/tests/SharpCompress.Test/SevenZip/SevenZipArchiveTests.cs b/tests/SharpCompress.Test/SevenZip/SevenZipArchiveTests.cs index e77e288c..0fc86c3e 100644 --- a/tests/SharpCompress.Test/SevenZip/SevenZipArchiveTests.cs +++ b/tests/SharpCompress.Test/SevenZip/SevenZipArchiveTests.cs @@ -4,7 +4,10 @@ using System.Linq; using SharpCompress.Archives; using SharpCompress.Archives.SevenZip; using SharpCompress.Common; +using SharpCompress.Common.SevenZip; +using SharpCompress.Factories; using SharpCompress.Readers; +using SharpCompress.Test.Mocks; using Xunit; namespace SharpCompress.Test.SevenZip; @@ -23,19 +26,49 @@ public class SevenZipArchiveTests : ArchiveTests [Fact] public void SevenZipArchive_LZMA_PathRead() => ArchiveFileRead("7Zip.LZMA.7z"); + [Fact] + public void SevenZipArchive_StreamCollection_Throws_On_NonSeekable_Stream() + { + using var nonSeekable = new ForwardOnlyStream(new MemoryStream()); + using var seekable = new MemoryStream(); + + Assert.Throws(() => + SevenZipArchive.OpenArchive([nonSeekable, seekable]) + ); + } + + [Fact] + public void SevenZipArchive_Stream_Throws_On_Unreadable_Stream() + { + using var unreadable = new TestStream(new MemoryStream(), false, true, true); + + Assert.Throws(() => SevenZipArchive.OpenArchive(unreadable)); + } + [Fact] public void SevenZipArchive_LZMAAES_StreamRead() => - ArchiveStreamRead("7Zip.LZMA.Aes.7z", new ReaderOptions() { Password = "testpassword" }); + ArchiveStreamRead( + "7Zip.LZMA.Aes.7z", + ReaderOptions.ForExternalStream with + { + Password = "testpassword", + } + ); [Fact] public void SevenZipArchive_LZMAAES_PathRead() => - ArchiveFileRead("7Zip.LZMA.Aes.7z", new ReaderOptions() { Password = "testpassword" }); + ArchiveFileRead( + "7Zip.LZMA.Aes.7z", + ReaderOptions.ForFilePath with + { + Password = "testpassword", + } + ); [Fact] public void SevenZipArchive_LZMAAES_NoPasswordExceptionTest() => - Assert.Throws( - typeof(CryptographicException), - () => ArchiveFileRead("7Zip.LZMA.Aes.7z", new ReaderOptions() { Password = null }) + Assert.Throws(() => + ArchiveFileRead("7Zip.LZMA.Aes.7z", ReaderOptions.ForFilePath.WithPassword(null)) ); //was failing with ArgumentNullException not CryptographicException like rar [Fact] @@ -54,13 +87,41 @@ public class SevenZipArchiveTests : ArchiveTests [Fact] public void SevenZipArchive_LZMA2_PathRead() => ArchiveFileRead("7Zip.LZMA2.7z"); + [Fact] + public void SevenZipArchive_LZMA2_EXE_StreamRead() => + ArchiveStreamRead( + new SevenZipFactory(), + "7Zip.LZMA2.exe", + ReaderOptions.ForExternalStream.WithLookForHeader(true) + ); + + [Fact] + public void SevenZipArchive_LZMA2_EXE_PathRead() => + ArchiveFileRead( + "7Zip.LZMA2.exe", + ReaderOptions.ForFilePath.WithLookForHeader(true), + new SevenZipFactory() + ); + [Fact] public void SevenZipArchive_LZMA2AES_StreamRead() => - ArchiveStreamRead("7Zip.LZMA2.Aes.7z", new ReaderOptions { Password = "testpassword" }); + ArchiveStreamRead( + "7Zip.LZMA2.Aes.7z", + ReaderOptions.ForExternalStream with + { + Password = "testpassword", + } + ); [Fact] public void SevenZipArchive_LZMA2AES_PathRead() => - ArchiveFileRead("7Zip.LZMA2.Aes.7z", new ReaderOptions { Password = "testpassword" }); + ArchiveFileRead( + "7Zip.LZMA2.Aes.7z", + ReaderOptions.ForFilePath with + { + Password = "testpassword", + } + ); [Fact] public void SevenZipArchive_BZip2_StreamRead() => ArchiveStreamRead("7Zip.BZip2.7z"); @@ -74,32 +135,35 @@ public class SevenZipArchiveTests : ArchiveTests [Fact] public void SevenZipArchive_BZip2_Split() => - Assert.Throws( - () => - ArchiveStreamRead( - null, - "Original.7z.001", - "Original.7z.002", - "Original.7z.003", - "Original.7z.004", - "Original.7z.005", - "Original.7z.006", - "Original.7z.007" - ) + Assert.Throws(() => + ArchiveStreamRead( + ".001", + null, + "Original.7z.001", + "Original.7z.002", + "Original.7z.003", + "Original.7z.004", + "Original.7z.005", + "Original.7z.006", + "Original.7z.007" + ) ); //Same as archive as Original.7z.001 ... 007 files without the root directory 'Original\' in the archive - this caused the verify to fail [Fact] public void SevenZipArchive_BZip2_Split_Working() => - ArchiveStreamMultiRead( - null, - "7Zip.BZip2.split.001", - "7Zip.BZip2.split.002", - "7Zip.BZip2.split.003", - "7Zip.BZip2.split.004", - "7Zip.BZip2.split.005", - "7Zip.BZip2.split.006", - "7Zip.BZip2.split.007" + Assert.Throws(() => + ArchiveStreamRead( + ".001", + null, + "7Zip.BZip2.split.001", + "7Zip.BZip2.split.002", + "7Zip.BZip2.split.003", + "7Zip.BZip2.split.004", + "7Zip.BZip2.split.005", + "7Zip.BZip2.split.006", + "7Zip.BZip2.split.007" + ) ); //will detect and load other files @@ -114,6 +178,25 @@ public class SevenZipArchiveTests : ArchiveTests //"7Zip.BZip2.split.006", //"7Zip.BZip2.split.007" + [Fact] + public void SevenZipArchive_Copy_StreamRead() => ArchiveStreamRead("7Zip.Copy.7z"); + + [Fact] + public void SevenZipArchive_Copy_PathRead() => ArchiveFileRead("7Zip.Copy.7z"); + + [Fact] + public void SevenZipArchive_Copy_CompressionType() + { + using (Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "7Zip.Copy.7z"))) + using (var archive = SevenZipArchive.OpenArchive(stream)) + { + foreach (var entry in archive.Entries.Where(e => !e.IsDirectory)) + { + Assert.Equal(CompressionType.None, entry.CompressionType); + } + } + } + [Fact] public void SevenZipArchive_ZSTD_StreamRead() => ArchiveStreamRead("7Zip.ZSTD.7z"); @@ -122,19 +205,19 @@ public class SevenZipArchiveTests : ArchiveTests [Fact] public void SevenZipArchive_ZSTD_Split() => - Assert.Throws( - () => - ArchiveStreamRead( - null, - "7Zip.ZSTD.Split.7z.001", - "7Zip.ZSTD.Split.7z.002", - "7Zip.ZSTD.Split.7z.003", - "7Zip.ZSTD.Split.7z.004", - "7Zip.ZSTD.Split.7z.005", - "7Zip.ZSTD.Split.7z.006" - ) + ArchiveStreamMultiRead( + null, + "7Zip.ZSTD.Split.7z.001", + "7Zip.ZSTD.Split.7z.002", + "7Zip.ZSTD.Split.7z.003", + "7Zip.ZSTD.Split.7z.004", + "7Zip.ZSTD.Split.7z.005", + "7Zip.ZSTD.Split.7z.006" ); + [Fact] + public void SevenZipArchive_EOS_FileRead() => ArchiveFileRead("7Zip.eos.7z"); + [Fact] public void SevenZipArchive_Delta_FileRead() => ArchiveFileRead("7Zip.delta.7z"); @@ -159,6 +242,12 @@ public class SevenZipArchiveTests : ArchiveTests [Fact] public void SevenZipArchive_SPARC_FileRead() => ArchiveFileRead("7Zip.SPARC.7z"); + [Fact] + public void SevenZipArchive_ARM64_FileRead() => ArchiveFileRead("7Zip.ARM64.7z"); + + [Fact] + public void SevenZipArchive_RISCV_FileRead() => ArchiveFileRead("7Zip.RISCV.7z"); + [Fact] public void SevenZipArchive_Filters_FileRead() => ArchiveFileRead("7Zip.Filters.7z"); @@ -170,10 +259,10 @@ public class SevenZipArchiveTests : ArchiveTests public void SevenZipArchive_Tar_PathRead() { using (Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "7Zip.Tar.tar.7z"))) - using (var archive = SevenZipArchive.Open(stream)) + using (var archive = SevenZipArchive.OpenArchive(stream)) { var entry = archive.Entries.First(); - entry.WriteToFile(Path.Combine(SCRATCH_FILES_PATH, entry.Key)); + entry.WriteToFile(Path.Combine(SCRATCH_FILES_PATH, entry.Key.NotNull())); var size = entry.Size; var scratch = new FileInfo(Path.Combine(SCRATCH_FILES_PATH, "7Zip.Tar.tar")); @@ -188,4 +277,172 @@ public class SevenZipArchiveTests : ArchiveTests Path.Combine(TEST_ARCHIVES_PATH, "7Zip.Tar.tar") ); } + + [Fact] + public void SevenZipArchive_TestEncryptedDetection() + { + using var passwordProtectedFilesArchive = SevenZipArchive.OpenArchive( + Path.Combine(TEST_ARCHIVES_PATH, "7Zip.encryptedFiles.7z") + ); + Assert.True(passwordProtectedFilesArchive.IsEncrypted); + } + + [Fact] + public void SevenZipArchive_TestSolidDetection() + { + using var oneBlockSolidArchive = SevenZipArchive.OpenArchive( + Path.Combine(TEST_ARCHIVES_PATH, "7Zip.solid.1block.7z") + ); + Assert.True(oneBlockSolidArchive.IsSolid); + + using var solidArchive = SevenZipArchive.OpenArchive( + Path.Combine(TEST_ARCHIVES_PATH, "7Zip.solid.7z") + ); + Assert.True(solidArchive.IsSolid); + + using var nonSolidArchive = SevenZipArchive.OpenArchive( + Path.Combine(TEST_ARCHIVES_PATH, "7Zip.nonsolid.7z") + ); + Assert.False(nonSolidArchive.IsSolid); + } + + [Fact] + public void SevenZipArchive_Solid_ExtractAllEntries_Contiguous() + { + // This test verifies that solid archives iterate entries as contiguous streams + // rather than recreating the decompression stream for each entry + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "7Zip.solid.7z"); + using var archive = SevenZipArchive.OpenArchive(testArchive); + Assert.True(archive.IsSolid); + + using var reader = archive.ExtractAllEntries(); + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + reader.WriteEntryToDirectory(SCRATCH_FILES_PATH); + } + } + + VerifyFiles(); + } + + [Fact] + public void SevenZipArchive_Solid_VerifyStreamReuse() + { + // This test verifies that the folder stream is reused within each folder + // and not recreated for each entry in solid archives + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "7Zip.solid.7z"); + using var archive = SevenZipArchive.OpenArchive(testArchive); + Assert.True(archive.IsSolid); + + using var reader = archive.ExtractAllEntries(); + + var sevenZipReader = Assert.IsType(reader); + sevenZipReader.DiagnosticsEnabled = true; + + Stream? currentFolderStreamInstance = null; + object? currentFolder = null; + var entryCount = 0; + var entriesInCurrentFolder = 0; + var streamRecreationsWithinFolder = 0; + + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + // Extract the entry to trigger GetEntryStream + using var entryStream = reader.OpenEntryStream(); + var buffer = new byte[4096]; + while (entryStream.Read(buffer, 0, buffer.Length) > 0) + { + // Read the stream to completion + } + + entryCount++; + + var folderStream = sevenZipReader.DiagnosticsCurrentFolderStream; + var folder = sevenZipReader.DiagnosticsCurrentFolder; + + Assert.NotNull(folderStream); // Folder stream should exist + + // Check if we're in a new folder + if (currentFolder == null || !ReferenceEquals(currentFolder, folder)) + { + // Starting a new folder + currentFolder = folder; + currentFolderStreamInstance = folderStream; + entriesInCurrentFolder = 1; + } + else + { + // Same folder - verify stream wasn't recreated + entriesInCurrentFolder++; + + if (!ReferenceEquals(currentFolderStreamInstance, folderStream)) + { + // Stream was recreated within the same folder - this is the bug we're testing for! + streamRecreationsWithinFolder++; + } + + currentFolderStreamInstance = folderStream; + } + } + } + + // Verify we actually tested multiple entries + Assert.True(entryCount > 1, "Test should have multiple entries to verify stream reuse"); + + // The critical check: within a single folder, the stream should NEVER be recreated + Assert.Equal(0, streamRecreationsWithinFolder); // Folder stream should remain the same for all entries in the same folder + } + + [Fact] + public void SevenZipArchive_EmptyStream_WriteToDirectory() + { + // This test specifically verifies that archives with empty-stream entries + // (files with size 0 and no compressed data) can be extracted without throwing + // NullReferenceException. This was previously failing because the folder was null + // for empty-stream entries. + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "7Zip.EmptyStream.7z"); + using var archive = SevenZipArchive.OpenArchive(testArchive); + + var emptyStreamFileCount = 0; + foreach (var entry in archive.Entries) + { + if (!entry.IsDirectory) + { + // Verify this is actually an empty-stream entry (HasStream == false) + var sevenZipEntry = entry as SevenZipEntry; + if (sevenZipEntry?.FilePart.Header.HasStream == false) + { + emptyStreamFileCount++; + } + + // This should not throw NullReferenceException + entry.WriteToDirectory(SCRATCH_FILES_PATH); + } + } + + // Ensure we actually tested empty-stream entries + Assert.True( + emptyStreamFileCount > 0, + "Test archive should contain at least one empty-stream entry" + ); + + // Verify that empty files were created + var extractedFiles = Directory.GetFiles( + SCRATCH_FILES_PATH, + "*", + SearchOption.AllDirectories + ); + Assert.NotEmpty(extractedFiles); + + // All extracted files should be empty (0 bytes) + foreach (var file in extractedFiles) + { + var fileInfo = new FileInfo(file); + Assert.Equal(0, fileInfo.Length); + } + } } diff --git a/tests/SharpCompress.Test/SevenZip/SevenZipCrcExtractionTests.cs b/tests/SharpCompress.Test/SevenZip/SevenZipCrcExtractionTests.cs new file mode 100644 index 00000000..3261f16a --- /dev/null +++ b/tests/SharpCompress.Test/SevenZip/SevenZipCrcExtractionTests.cs @@ -0,0 +1,81 @@ +using System.IO; +using System.Linq; +using System.Threading.Tasks; +using SharpCompress.Archives; +using SharpCompress.Archives.SevenZip; +using SharpCompress.Common; +using SharpCompress.Common.SevenZip; +using Xunit; + +namespace SharpCompress.Test.SevenZip; + +public class SevenZipCrcExtractionTests : ArchiveTests +{ + [Fact] + public void SevenZip_Archive_WriteToFile_Throws_On_Crc_Mismatch() + { + using var archive = SevenZipArchive.OpenArchive( + Path.Combine(TEST_ARCHIVES_PATH, "7Zip.LZMA.7z") + ); + var entry = CorruptFirstFileCrc(archive); + var destination = Path.Combine(SCRATCH_FILES_PATH, "7zip-crc-mismatch.txt"); + + var exception = Assert.Throws(() => entry.WriteToFile(destination)); + + Assert.Contains(entry.Key!, exception.Message); + } + + [Fact] + public void SevenZip_Archive_WriteToFile_Skips_Crc_Mismatch_When_Disabled() + { + using var archive = SevenZipArchive.OpenArchive( + Path.Combine(TEST_ARCHIVES_PATH, "7Zip.LZMA.7z") + ); + var entry = CorruptFirstFileCrc(archive); + var destination = Path.Combine(SCRATCH_FILES_PATH, "7zip-crc-disabled.txt"); + + entry.WriteToFile(destination, new ExtractionOptions { CheckCrc = false }); + + Assert.True(new FileInfo(destination).Length > 0); + } + + [Fact] + public async Task SevenZip_Archive_WriteToFileAsync_Throws_On_Crc_Mismatch() + { + await using var archive = await SevenZipArchive.OpenAsyncArchive( + Path.Combine(TEST_ARCHIVES_PATH, "7Zip.LZMA.7z") + ); + var entries = await archive.EntriesAsync.ToListAsync(); + var entry = CorruptFirstFileCrc(entries); + var destination = Path.Combine(SCRATCH_FILES_PATH, "7zip-crc-mismatch-async.txt"); + + var exception = await Assert.ThrowsAsync(async () => + await entry.WriteToFileAsync(destination) + ); + + Assert.Contains(entry.Key!, exception.Message); + } + + private static IArchiveEntry CorruptFirstFileCrc(IArchive archive) + { + var entry = archive.Entries.First(e => !e.IsDirectory); + CorruptCrc(entry); + return entry; + } + + private static IArchiveEntry CorruptFirstFileCrc( + System.Collections.Generic.IEnumerable entries + ) + { + var entry = entries.First(e => !e.IsDirectory); + CorruptCrc(entry); + return entry; + } + + private static void CorruptCrc(IArchiveEntry entry) + { + var sevenZipEntry = Assert.IsAssignableFrom(entry); + var crc = sevenZipEntry.FilePart.Header.Crc.NotNull(); + sevenZipEntry.FilePart.Header.Crc = crc ^ 0xFFFFFFFF; + } +} diff --git a/tests/SharpCompress.Test/SevenZip/SevenZipWriterAsyncTests.cs b/tests/SharpCompress.Test/SevenZip/SevenZipWriterAsyncTests.cs new file mode 100644 index 00000000..e61b77d7 --- /dev/null +++ b/tests/SharpCompress.Test/SevenZip/SevenZipWriterAsyncTests.cs @@ -0,0 +1,159 @@ +using System; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Archives.SevenZip; +using SharpCompress.Common; +using SharpCompress.Test.Mocks; +using SharpCompress.Writers; +using SharpCompress.Writers.SevenZip; +using Xunit; + +namespace SharpCompress.Test.SevenZip; + +public class SevenZipWriterAsyncTests : TestBase +{ + [Fact] + public async ValueTask SevenZipWriter_Async_SingleFile_RoundTrip() + { + var content = "Hello, async 7z world!"u8.ToArray(); + + using var archiveStream = new MemoryStream(); + + await using ( + var writer = new SevenZipWriter( + new AsyncOnlyStream(archiveStream), + new SevenZipWriterOptions() + ) + ) + { + await writer.WriteAsync("test.txt", new MemoryStream(content), DateTime.UtcNow); + } + + archiveStream.Position = 0; + using var archive = (SevenZipArchive)SevenZipArchive.OpenArchive(archiveStream); + var entries = archive.Entries.Where(e => !e.IsDirectory).ToList(); + Assert.Single(entries); + Assert.Equal("test.txt", entries[0].Key); + Assert.Equal(content.Length, (int)entries[0].Size); + + using var output = new MemoryStream(); + using (var entryStream = entries[0].OpenEntryStream()) + { + entryStream.CopyTo(output); + } + + Assert.Equal(content, output.ToArray()); + } + + [Fact] + public async ValueTask SevenZipWriter_Async_WithDirectory_RoundTrip() + { + using var archiveStream = new MemoryStream(); + + await using ( + var writer = new SevenZipWriter( + new AsyncOnlyStream(archiveStream), + new SevenZipWriterOptions(CompressionType.LZMA2) + ) + ) + { + await writer.WriteDirectoryAsync("mydir", DateTime.UtcNow); + await writer.WriteAsync( + "mydir/file1.txt", + new MemoryStream(Encoding.UTF8.GetBytes("file one")), + DateTime.UtcNow + ); + await writer.WriteAsync( + "mydir/file2.txt", + new MemoryStream(Encoding.UTF8.GetBytes("file two")), + DateTime.UtcNow + ); + } + + archiveStream.Position = 0; + using var archive = (SevenZipArchive)SevenZipArchive.OpenArchive(archiveStream); + var entries = archive.Entries.ToList(); + + Assert.Equal(3, entries.Count); + Assert.Contains(entries, e => e.IsDirectory && e.Key == "mydir"); + Assert.Contains(entries, e => !e.IsDirectory && e.Key == "mydir/file1.txt"); + Assert.Contains(entries, e => !e.IsDirectory && e.Key == "mydir/file2.txt"); + } + + [Fact] + public async ValueTask SevenZipWriter_Async_ViaWriterFactory() + { + var content = "Factory-created async archive"u8.ToArray(); + + using var archiveStream = new MemoryStream(); + + await using ( + var writer = await WriterFactory.OpenAsyncWriter( + new AsyncOnlyStream(archiveStream), + ArchiveType.SevenZip, + new SevenZipWriterOptions() + ) + ) + { + await writer.WriteAsync("factory.txt", new MemoryStream(content), DateTime.UtcNow); + } + + archiveStream.Position = 0; + using var archive = (SevenZipArchive)SevenZipArchive.OpenArchive(archiveStream); + var entry = archive.Entries.Single(e => !e.IsDirectory); + + using var output = new MemoryStream(); + using (var entryStream = entry.OpenEntryStream()) + { + entryStream.CopyTo(output); + } + + Assert.Equal("factory.txt", entry.Key); + Assert.Equal(content, output.ToArray()); + } + + [Fact] + public async ValueTask SevenZipWriter_Async_UsesAsyncSourceReads() + { + var content = "source stream supports async reads only"u8.ToArray(); + + using var archiveStream = new MemoryStream(); + + await using (var writer = new SevenZipWriter(archiveStream, new SevenZipWriterOptions())) + { + using var source = new AsyncOnlyStream(new MemoryStream(content)); + await writer.WriteAsync("async-source.txt", source, DateTime.UtcNow); + } + + archiveStream.Position = 0; + using var archive = (SevenZipArchive)SevenZipArchive.OpenArchive(archiveStream); + var entry = archive.Entries.Single(e => !e.IsDirectory); + + using var output = new MemoryStream(); + using (var entryStream = entry.OpenEntryStream()) + { + entryStream.CopyTo(output); + } + + Assert.Equal("async-source.txt", entry.Key); + Assert.Equal(content, output.ToArray()); + } + + [Fact] + public async ValueTask SevenZipWriter_Async_Cancelled_Throws() + { + using var archiveStream = new MemoryStream(); + await using var writer = new SevenZipWriter(archiveStream, new SevenZipWriterOptions()); + + using var source = new MemoryStream("cancel me"u8.ToArray()); + using var cts = new CancellationTokenSource(); + cts.Cancel(); + + await Assert.ThrowsAnyAsync(() => + writer.WriteAsync("cancel.txt", source, DateTime.UtcNow, cts.Token).AsTask() + ); + } +} diff --git a/tests/SharpCompress.Test/SevenZip/SevenZipWriterTests.cs b/tests/SharpCompress.Test/SevenZip/SevenZipWriterTests.cs new file mode 100644 index 00000000..8209db4c --- /dev/null +++ b/tests/SharpCompress.Test/SevenZip/SevenZipWriterTests.cs @@ -0,0 +1,463 @@ +using System; +using System.IO; +using System.Linq; +using System.Text; +using SharpCompress.Archives.SevenZip; +using SharpCompress.Common; +using SharpCompress.Writers; +using SharpCompress.Writers.SevenZip; +using Xunit; + +namespace SharpCompress.Test.SevenZip; + +public class SevenZipWriterTests : TestBase +{ + [Fact] + public void SevenZipWriter_SingleFile_RoundTrip() + { + var content = "Hello, 7z world! This is a test of the SevenZipWriter."u8.ToArray(); + + using var archiveStream = new MemoryStream(); + + // Write archive + using (var writer = new SevenZipWriter(archiveStream, new SevenZipWriterOptions())) + { + using var source = new MemoryStream(content); + writer.Write("test.txt", source, DateTime.UtcNow); + } + + // Read back and verify + archiveStream.Position = 0; + using (var archive = (SevenZipArchive)SevenZipArchive.OpenArchive(archiveStream)) + { + var entries = archive.Entries.Where(e => !e.IsDirectory).ToList(); + Assert.Single(entries); + Assert.Equal("test.txt", entries[0].Key); + Assert.Equal(content.Length, (int)entries[0].Size); + + using var output = new MemoryStream(); + using (var entryStream = entries[0].OpenEntryStream()) + { + entryStream.CopyTo(output); + } + Assert.Equal(content, output.ToArray()); + } + } + + [Fact] + public void SevenZipWriter_MultipleFiles_RoundTrip() + { + var files = new[] + { + ("file1.txt", "Content of file 1"), + ("subdir/file2.txt", "Content of file 2 in subdirectory"), + ("file3.bin", "Some binary-ish content with special bytes"), + }; + + using var archiveStream = new MemoryStream(); + + // Write archive + using (var writer = new SevenZipWriter(archiveStream, new SevenZipWriterOptions())) + { + foreach (var (name, text) in files) + { + using var source = new MemoryStream(Encoding.UTF8.GetBytes(text)); + writer.Write(name, source, DateTime.UtcNow); + } + } + + // Read back and verify + archiveStream.Position = 0; + using (var archive = (SevenZipArchive)SevenZipArchive.OpenArchive(archiveStream)) + { + var entries = archive.Entries.Where(e => !e.IsDirectory).ToList(); + Assert.Equal(files.Length, entries.Count); + + for (var i = 0; i < files.Length; i++) + { + var entry = entries.First(e => e.Key == files[i].Item1); + using var output = new MemoryStream(); + using (var entryStream = entry.OpenEntryStream()) + { + entryStream.CopyTo(output); + } + var extractedText = Encoding.UTF8.GetString(output.ToArray()); + Assert.Equal(files[i].Item2, extractedText); + } + } + } + + [Fact] + public void SevenZipWriter_WithDirectory_RoundTrip() + { + using var archiveStream = new MemoryStream(); + + // Write archive with directory and file + using (var writer = new SevenZipWriter(archiveStream, new SevenZipWriterOptions())) + { + writer.WriteDirectory("mydir", DateTime.UtcNow); + + using var source = new MemoryStream("file inside dir"u8.ToArray()); + writer.Write("mydir/data.txt", source, DateTime.UtcNow); + } + + // Read back and verify + archiveStream.Position = 0; + using (var archive = (SevenZipArchive)SevenZipArchive.OpenArchive(archiveStream)) + { + var allEntries = archive.Entries.ToList(); + Assert.Equal(2, allEntries.Count); + + var dirEntry = allEntries.FirstOrDefault(e => e.IsDirectory); + Assert.NotNull(dirEntry); + + var fileEntry = allEntries.FirstOrDefault(e => !e.IsDirectory); + Assert.NotNull(fileEntry); + Assert.Equal("mydir/data.txt", fileEntry!.Key); + + using var output = new MemoryStream(); + using (var entryStream = fileEntry.OpenEntryStream()) + { + entryStream.CopyTo(output); + } + Assert.Equal("file inside dir", Encoding.UTF8.GetString(output.ToArray())); + } + } + + [Fact] + public void SevenZipWriter_EmptyFile_RoundTrip() + { + using var archiveStream = new MemoryStream(); + + // Write archive with an empty file + using (var writer = new SevenZipWriter(archiveStream, new SevenZipWriterOptions())) + { + using var source = new MemoryStream(); + writer.Write("empty.txt", source, DateTime.UtcNow); + + using var source2 = new MemoryStream("not empty"u8.ToArray()); + writer.Write("notempty.txt", source2, DateTime.UtcNow); + } + + // Read back and verify + archiveStream.Position = 0; + using (var archive = (SevenZipArchive)SevenZipArchive.OpenArchive(archiveStream)) + { + var entries = archive.Entries.Where(e => !e.IsDirectory).ToList(); + Assert.Equal(2, entries.Count); + + var emptyEntry = entries.First(e => e.Key == "empty.txt"); + Assert.Equal(0, (int)emptyEntry.Size); + + var nonEmptyEntry = entries.First(e => e.Key == "notempty.txt"); + using var output = new MemoryStream(); + using (var entryStream = nonEmptyEntry.OpenEntryStream()) + { + entryStream.CopyTo(output); + } + Assert.Equal("not empty", Encoding.UTF8.GetString(output.ToArray())); + } + } + + [Fact] + public void SevenZipWriter_LZMA2_SingleFile_RoundTrip() + { + var content = + "Hello, LZMA2 world! This is a test of LZMA2 encoding in the SevenZipWriter."u8.ToArray(); + + using var archiveStream = new MemoryStream(); + + using ( + var writer = new SevenZipWriter( + archiveStream, + new SevenZipWriterOptions(CompressionType.LZMA2) + ) + ) + { + using var source = new MemoryStream(content); + writer.Write("test.txt", source, DateTime.UtcNow); + } + + archiveStream.Position = 0; + using (var archive = (SevenZipArchive)SevenZipArchive.OpenArchive(archiveStream)) + { + var entries = archive.Entries.Where(e => !e.IsDirectory).ToList(); + Assert.Single(entries); + Assert.Equal("test.txt", entries[0].Key); + Assert.Equal(content.Length, (int)entries[0].Size); + + using var output = new MemoryStream(); + using (var entryStream = entries[0].OpenEntryStream()) + { + entryStream.CopyTo(output); + } + Assert.Equal(content, output.ToArray()); + } + } + + [Fact] + public void SevenZipWriter_LZMA2_MultipleFiles_RoundTrip() + { + var files = new[] + { + ("file1.txt", "Content of file 1 for LZMA2 testing"), + ("subdir/file2.txt", "Content of file 2 in subdirectory for LZMA2"), + ("file3.bin", "Some binary-ish content with special bytes for LZMA2 testing"), + }; + + using var archiveStream = new MemoryStream(); + + using ( + var writer = new SevenZipWriter( + archiveStream, + new SevenZipWriterOptions(CompressionType.LZMA2) + ) + ) + { + foreach (var (name, text) in files) + { + using var source = new MemoryStream(Encoding.UTF8.GetBytes(text)); + writer.Write(name, source, DateTime.UtcNow); + } + } + + archiveStream.Position = 0; + using (var archive = (SevenZipArchive)SevenZipArchive.OpenArchive(archiveStream)) + { + var entries = archive.Entries.Where(e => !e.IsDirectory).ToList(); + Assert.Equal(files.Length, entries.Count); + + for (var i = 0; i < files.Length; i++) + { + var entry = entries.First(e => e.Key == files[i].Item1); + using var output = new MemoryStream(); + using (var entryStream = entry.OpenEntryStream()) + { + entryStream.CopyTo(output); + } + var extractedText = Encoding.UTF8.GetString(output.ToArray()); + Assert.Equal(files[i].Item2, extractedText); + } + } + } + + [Fact] + public void SevenZipWriter_LZMA2_LargerFile_RoundTrip() + { + // Create 3MB of repeating pattern data - forces multi-chunk in LZMA2 + var content = new byte[3 * 1024 * 1024]; + var pattern = Encoding.UTF8.GetBytes( + "This is a repeating pattern for LZMA2 compression testing. " + ); + for (var i = 0; i < content.Length; i++) + { + content[i] = pattern[i % pattern.Length]; + } + + using var archiveStream = new MemoryStream(); + + using ( + var writer = new SevenZipWriter( + archiveStream, + new SevenZipWriterOptions(CompressionType.LZMA2) + ) + ) + { + using var source = new MemoryStream(content); + writer.Write("large.bin", source, DateTime.UtcNow); + } + + Assert.True( + archiveStream.Length < content.Length, + "Archive should be smaller than uncompressed data" + ); + + archiveStream.Position = 0; + using (var archive = (SevenZipArchive)SevenZipArchive.OpenArchive(archiveStream)) + { + var entries = archive.Entries.Where(e => !e.IsDirectory).ToList(); + Assert.Single(entries); + Assert.Equal(content.Length, (int)entries[0].Size); + + using var output = new MemoryStream(); + using (var entryStream = entries[0].OpenEntryStream()) + { + entryStream.CopyTo(output); + } + Assert.Equal(content, output.ToArray()); + } + } + + [Fact] + public void SevenZipWriter_LZMA2_IncompressibleData_RoundTrip() + { + // Random bytes - forces uncompressed fallback in LZMA2 + var content = new byte[100 * 1024]; + var rng = new Random(42); + rng.NextBytes(content); + + using var archiveStream = new MemoryStream(); + + using ( + var writer = new SevenZipWriter( + archiveStream, + new SevenZipWriterOptions(CompressionType.LZMA2) + ) + ) + { + using var source = new MemoryStream(content); + writer.Write("random.bin", source, DateTime.UtcNow); + } + + archiveStream.Position = 0; + using (var archive = (SevenZipArchive)SevenZipArchive.OpenArchive(archiveStream)) + { + var entries = archive.Entries.Where(e => !e.IsDirectory).ToList(); + Assert.Single(entries); + Assert.Equal(content.Length, (int)entries[0].Size); + + using var output = new MemoryStream(); + using (var entryStream = entries[0].OpenEntryStream()) + { + entryStream.CopyTo(output); + } + Assert.Equal(content, output.ToArray()); + } + } + + [Fact] + public void SevenZipWriter_UnsupportedCompressionType_Throws() + { + Assert.Throws(() => new SevenZipWriterOptions(CompressionType.Deflate)); + } + + [Fact] + public void SevenZipWriter_UncompressedHeader_RoundTrip() + { + var content = "Testing with uncompressed header"u8.ToArray(); + + using var archiveStream = new MemoryStream(); + + // Write archive with uncompressed header + using ( + var writer = new SevenZipWriter( + archiveStream, + new SevenZipWriterOptions { CompressHeader = false } + ) + ) + { + using var source = new MemoryStream(content); + writer.Write("rawheader.txt", source, DateTime.UtcNow); + } + + // Read back and verify + archiveStream.Position = 0; + using (var archive = (SevenZipArchive)SevenZipArchive.OpenArchive(archiveStream)) + { + var entries = archive.Entries.Where(e => !e.IsDirectory).ToList(); + Assert.Single(entries); + + using var output = new MemoryStream(); + using (var entryStream = entries[0].OpenEntryStream()) + { + entryStream.CopyTo(output); + } + Assert.Equal(content, output.ToArray()); + } + } + + [Fact] + public void SevenZipWriter_ViaWriterFactory() + { + var content = "Factory-created archive"u8.ToArray(); + + using var archiveStream = new MemoryStream(); + + // Write via WriterFactory + using ( + var writer = WriterFactory.OpenWriter( + archiveStream, + ArchiveType.SevenZip, + new SevenZipWriterOptions() + ) + ) + { + using var source = new MemoryStream(content); + writer.Write("factory.txt", source, DateTime.UtcNow); + } + + // Read back and verify + archiveStream.Position = 0; + using (var archive = (SevenZipArchive)SevenZipArchive.OpenArchive(archiveStream)) + { + var entries = archive.Entries.Where(e => !e.IsDirectory).ToList(); + Assert.Single(entries); + + using var output = new MemoryStream(); + using (var entryStream = entries[0].OpenEntryStream()) + { + entryStream.CopyTo(output); + } + Assert.Equal(content, output.ToArray()); + } + } + + [Fact] + public void SevenZipWriter_LargerFile_RoundTrip() + { + // Create 100KB of repeating pattern data (compresses well) + var content = new byte[100 * 1024]; + var pattern = Encoding.UTF8.GetBytes( + "This is a repeating pattern for compression testing. " + ); + for (var i = 0; i < content.Length; i++) + { + content[i] = pattern[i % pattern.Length]; + } + + using var archiveStream = new MemoryStream(); + + // Write archive + using (var writer = new SevenZipWriter(archiveStream, new SevenZipWriterOptions())) + { + using var source = new MemoryStream(content); + writer.Write("large.bin", source, DateTime.UtcNow); + } + + // Verify compressed size is smaller than original + Assert.True( + archiveStream.Length < content.Length, + "Archive should be smaller than uncompressed data" + ); + + // Read back and verify + archiveStream.Position = 0; + using (var archive = (SevenZipArchive)SevenZipArchive.OpenArchive(archiveStream)) + { + var entries = archive.Entries.Where(e => !e.IsDirectory).ToList(); + Assert.Single(entries); + Assert.Equal(content.Length, (int)entries[0].Size); + + using var output = new MemoryStream(); + using (var entryStream = entries[0].OpenEntryStream()) + { + entryStream.CopyTo(output); + } + Assert.Equal(content, output.ToArray()); + } + } + + [Fact] + public void SevenZipWriter_RequiresSeekableStream() + { + var nonSeekable = new NonSeekableStream(); + Assert.Throws(() => + new SevenZipWriter(nonSeekable, new SevenZipWriterOptions()) + ); + } + + private class NonSeekableStream : MemoryStream + { + public override bool CanSeek => false; + } +} diff --git a/tests/SharpCompress.Test/SharpCompress.Test.csproj b/tests/SharpCompress.Test/SharpCompress.Test.csproj index c20664a7..68ff64ae 100644 --- a/tests/SharpCompress.Test/SharpCompress.Test.csproj +++ b/tests/SharpCompress.Test/SharpCompress.Test.csproj @@ -1,23 +1,30 @@ - + - net7.0;net462 + net10.0;net48 + Exe SharpCompress.Test SharpCompress.Test + SharpCompress.Test.snk + true + + + $(DefineConstants);LEGACY_DOTNET + AnyCPU + false + + + $(DefineConstants);WINDOWS + + + $(DefineConstants);LINUX - - - - all - runtime; build; native; contentfiles; analyzers; buildtransitive - - - - - - + + + + diff --git a/tests/SharpCompress.Test/SharpCompress.Test.snk b/tests/SharpCompress.Test/SharpCompress.Test.snk new file mode 100644 index 00000000..5a9ce097 Binary files /dev/null and b/tests/SharpCompress.Test/SharpCompress.Test.snk differ diff --git a/tests/SharpCompress.Test/Streams/DisposalTests.cs b/tests/SharpCompress.Test/Streams/DisposalTests.cs new file mode 100644 index 00000000..3ddb687e --- /dev/null +++ b/tests/SharpCompress.Test/Streams/DisposalTests.cs @@ -0,0 +1,205 @@ +using System; +using System.IO; +using SharpCompress.Common; +using SharpCompress.Compressors; +using SharpCompress.Compressors.BZip2; +using SharpCompress.Compressors.Deflate; +using SharpCompress.Compressors.LZMA; +using SharpCompress.Compressors.Lzw; +using SharpCompress.Compressors.PPMd; +using SharpCompress.Compressors.Reduce; +using SharpCompress.Compressors.ZStandard; +using SharpCompress.IO; +using SharpCompress.Readers; +using SharpCompress.Test.Mocks; +using Xunit; + +namespace SharpCompress.Test.Streams; + +public class DisposalTests +{ + private void VerifyStreamDisposal( + Func createStream, + bool supportsLeaveOpen = true + ) + { + // 1. Test Dispose behavior (should dispose inner stream) + { + using var innerStream = new TestStream(new MemoryStream()); + // createStream(stream, leaveOpen: false) + var stream = createStream(innerStream, false); + stream.Dispose(); + + // Some streams might not support disposal of inner stream (e.g. PpmdStream apparently) + // But for those that satisfy the pattern, we assert true. + Assert.True( + innerStream.IsDisposed, + "Stream should have been disposed when leaveOpen=false" + ); + } + + // 2. Test LeaveOpen behavior (should NOT dispose inner stream) + if (supportsLeaveOpen) + { + using var innerStream = new TestStream(new MemoryStream()); + // createStream(stream, leaveOpen: true) + var stream = createStream(innerStream, true); + stream.Dispose(); + Assert.False( + innerStream.IsDisposed, + "Stream should NOT have been disposed when leaveOpen=true" + ); + } + } + + private void VerifyAlwaysDispose(Func createStream) + { + using var innerStream = new TestStream(new MemoryStream()); + var stream = createStream(innerStream); + stream.Dispose(); + Assert.True(innerStream.IsDisposed, "Stream should have been disposed (AlwaysDispose)"); + } + + private void VerifyNeverDispose(Func createStream) + { + using var innerStream = new TestStream(new MemoryStream()); + var stream = createStream(innerStream); + stream.Dispose(); + Assert.False(innerStream.IsDisposed, "Stream should NOT have been disposed (NeverDispose)"); + } + + [Fact] + public void SourceStream_Disposal() + { + VerifyStreamDisposal( + (stream, leaveOpen) => + new SourceStream( + stream, + i => null, + ReaderOptions.ForExternalStream with + { + LeaveStreamOpen = leaveOpen, + } + ) + ); + } + + [Fact] + public void ProgressReportingStream_Disposal() + { + VerifyStreamDisposal( + (stream, leaveOpen) => + new ProgressReportingStream( + stream, + new Progress(), + "", + 0, + leaveOpen: leaveOpen + ) + ); + } + + [Fact] + public void DataDescriptorStream_Disposal() + { + // DataDescriptorStream DOES dispose inner stream + VerifyAlwaysDispose(stream => new DataDescriptorStream(stream)); + } + + [Fact] + public void DeflateStream_Disposal() + { + // DeflateStream in SharpCompress always disposes inner stream + VerifyAlwaysDispose(stream => new DeflateStream(stream, CompressionMode.Compress)); + } + + [Fact] + public void GZipStream_Disposal() + { + // GZipStream in SharpCompress always disposes inner stream + VerifyAlwaysDispose(stream => new GZipStream(stream, CompressionMode.Compress)); + } + + [Fact] + public void LzwStream_Disposal() + { + VerifyStreamDisposal( + (stream, leaveOpen) => + { + var lzw = new LzwStream(stream); + lzw.IsStreamOwner = !leaveOpen; + return lzw; + } + ); + } + + [Fact] + public void PpmdStream_Disposal() + { + // PpmdStream seems to not dispose inner stream based on code analysis + // It takes PpmdProperties which we need to mock or create. + var props = new PpmdProperties(); + VerifyNeverDispose(stream => PpmdStream.Create(props, stream, false)); + } + + [Fact] + public void LzmaStream_Disposal() + { + // LzmaStream always disposes inner stream + // Need to provide valid properties to avoid crash in constructor (invalid window size) + // 5 bytes: 1 byte properties + 4 bytes dictionary size (little endian) + // Dictionary size = 1024 (0x400) -> 00 04 00 00 + var lzmaProps = new byte[] { 0, 0, 4, 0, 0 }; + VerifyAlwaysDispose(stream => LzmaStream.Create(lzmaProps, stream)); + } + + [Fact] + public void LZipStream_Disposal() + { + // LZipStream now supports leaveOpen parameter + // Use Compress mode to avoid need for valid input header + VerifyStreamDisposal( + (stream, leaveOpen) => LZipStream.Create(stream, CompressionMode.Compress, leaveOpen) + ); + } + + [Fact] + public void BZip2Stream_Disposal() + { + // BZip2Stream now supports leaveOpen parameter + VerifyStreamDisposal( + (stream, leaveOpen) => + BZip2Stream.Create(stream, CompressionMode.Compress, false, leaveOpen) + ); + } + + [Fact] + public void ReduceStream_Disposal() + { + // ReduceStream does not dispose inner stream + VerifyNeverDispose(stream => ReduceStream.Create(stream, 0, 0, 1)); + } + + [Fact] + public void ZStandard_CompressionStream_Disposal() + { + VerifyStreamDisposal( + (stream, leaveOpen) => + new CompressionStream(stream, level: 0, bufferSize: 0, leaveOpen: leaveOpen) + ); + } + + [Fact] + public void ZStandard_DecompressionStream_Disposal() + { + VerifyStreamDisposal( + (stream, leaveOpen) => + new DecompressionStream( + stream, + bufferSize: 0, + checkEndOfStream: false, + leaveOpen: leaveOpen + ) + ); + } +} diff --git a/tests/SharpCompress.Test/Streams/LeaveOpenBehaviorTests.cs b/tests/SharpCompress.Test/Streams/LeaveOpenBehaviorTests.cs new file mode 100644 index 00000000..f78cd989 --- /dev/null +++ b/tests/SharpCompress.Test/Streams/LeaveOpenBehaviorTests.cs @@ -0,0 +1,225 @@ +using System; +using System.IO; +using System.Text; +using SharpCompress.Compressors; +using SharpCompress.Compressors.BZip2; +using SharpCompress.Compressors.LZMA; +using SharpCompress.Test.Mocks; +using Xunit; + +namespace SharpCompress.Test.Streams; + +public class LeaveOpenBehaviorTests +{ + private static byte[] CreateTestData() => + Encoding.UTF8.GetBytes("The quick brown fox jumps over the lazy dog"); + + [Fact] + public void BZip2Stream_Compress_LeaveOpen_False() + { + using var innerStream = new TestStream(new MemoryStream()); + using (var bzip2 = BZip2Stream.Create(innerStream, CompressionMode.Compress, false, false)) + { + bzip2.Write(CreateTestData(), 0, CreateTestData().Length); + bzip2.Finish(); + } + + Assert.True(innerStream.IsDisposed, "Inner stream should be disposed when leaveOpen=false"); + } + + [Fact] + public void BZip2Stream_Compress_LeaveOpen_True() + { + using var innerStream = new TestStream(new MemoryStream()); + byte[] compressed; + using ( + var bzip2 = BZip2Stream.Create( + innerStream, + CompressionMode.Compress, + false, + leaveOpen: true + ) + ) + { + bzip2.Write(CreateTestData(), 0, CreateTestData().Length); + bzip2.Finish(); + } + + Assert.False( + innerStream.IsDisposed, + "Inner stream should NOT be disposed when leaveOpen=true" + ); + + // Should be able to read the compressed data + innerStream.Position = 0; + compressed = new byte[innerStream.Length]; + innerStream.Read(compressed, 0, compressed.Length); + Assert.True(compressed.Length > 0); + } + + [Fact] + public void BZip2Stream_Decompress_LeaveOpen_False() + { + // First compress some data + var memStream = new MemoryStream(); + using (var bzip2 = BZip2Stream.Create(memStream, CompressionMode.Compress, false, true)) + { + bzip2.Write(CreateTestData(), 0, CreateTestData().Length); + bzip2.Finish(); + } + + memStream.Position = 0; + using var innerStream = new TestStream(memStream); + var decompressed = new byte[CreateTestData().Length]; + + using ( + var bzip2 = BZip2Stream.Create( + innerStream, + CompressionMode.Decompress, + false, + leaveOpen: false + ) + ) + { + bzip2.Read(decompressed, 0, decompressed.Length); + } + + Assert.True(innerStream.IsDisposed, "Inner stream should be disposed when leaveOpen=false"); + Assert.Equal(CreateTestData(), decompressed); + } + + [Fact] + public void BZip2Stream_Decompress_LeaveOpen_True() + { + // First compress some data + var memStream = new MemoryStream(); + using (var bzip2 = BZip2Stream.Create(memStream, CompressionMode.Compress, false, true)) + { + bzip2.Write(CreateTestData(), 0, CreateTestData().Length); + bzip2.Finish(); + } + + memStream.Position = 0; + using var innerStream = new TestStream(memStream); + var decompressed = new byte[CreateTestData().Length]; + + using ( + var bzip2 = BZip2Stream.Create( + innerStream, + CompressionMode.Decompress, + false, + leaveOpen: true + ) + ) + { + bzip2.Read(decompressed, 0, decompressed.Length); + } + + Assert.False( + innerStream.IsDisposed, + "Inner stream should NOT be disposed when leaveOpen=true" + ); + Assert.Equal(CreateTestData(), decompressed); + + // Should still be able to use the stream + innerStream.Position = 0; + Assert.True(innerStream.CanRead); + } + + [Fact] + public void LZipStream_Compress_LeaveOpen_False() + { + using var innerStream = new TestStream(new MemoryStream()); + using ( + var lzip = LZipStream.Create(innerStream, CompressionMode.Compress, leaveOpen: false) + ) + { + lzip.Write(CreateTestData(), 0, CreateTestData().Length); + lzip.Finish(); + } + + Assert.True(innerStream.IsDisposed, "Inner stream should be disposed when leaveOpen=false"); + } + + [Fact] + public void LZipStream_Compress_LeaveOpen_True() + { + using var innerStream = new TestStream(new MemoryStream()); + byte[] compressed; + using (var lzip = LZipStream.Create(innerStream, CompressionMode.Compress, leaveOpen: true)) + { + lzip.Write(CreateTestData(), 0, CreateTestData().Length); + lzip.Finish(); + } + + Assert.False( + innerStream.IsDisposed, + "Inner stream should NOT be disposed when leaveOpen=true" + ); + + // Should be able to read the compressed data + innerStream.Position = 0; + compressed = new byte[innerStream.Length]; + innerStream.Read(compressed, 0, compressed.Length); + Assert.True(compressed.Length > 0); + } + + [Fact] + public void LZipStream_Decompress_LeaveOpen_False() + { + // First compress some data + var memStream = new MemoryStream(); + using (var lzip = LZipStream.Create(memStream, CompressionMode.Compress, true)) + { + lzip.Write(CreateTestData(), 0, CreateTestData().Length); + lzip.Finish(); + } + + memStream.Position = 0; + using var innerStream = new TestStream(memStream); + var decompressed = new byte[CreateTestData().Length]; + + using ( + var lzip = LZipStream.Create(innerStream, CompressionMode.Decompress, leaveOpen: false) + ) + { + lzip.Read(decompressed, 0, decompressed.Length); + } + + Assert.True(innerStream.IsDisposed, "Inner stream should be disposed when leaveOpen=false"); + Assert.Equal(CreateTestData(), decompressed); + } + + [Fact] + public void LZipStream_Decompress_LeaveOpen_True() + { + // First compress some data + var memStream = new MemoryStream(); + using (var lzip = LZipStream.Create(memStream, CompressionMode.Compress, true)) + { + lzip.Write(CreateTestData(), 0, CreateTestData().Length); + lzip.Finish(); + } + + memStream.Position = 0; + using var innerStream = new TestStream(memStream); + var decompressed = new byte[CreateTestData().Length]; + + using ( + var lzip = LZipStream.Create(innerStream, CompressionMode.Decompress, leaveOpen: true) + ) + { + lzip.Read(decompressed, 0, decompressed.Length); + } + + Assert.False( + innerStream.IsDisposed, + "Inner stream should NOT be disposed when leaveOpen=true" + ); + Assert.Equal(CreateTestData(), decompressed); + + // Should still be able to use the stream + innerStream.Position = 0; + Assert.True(innerStream.CanRead); + } +} diff --git a/tests/SharpCompress.Test/Streams/LzmaStreamAsyncTests.cs b/tests/SharpCompress.Test/Streams/LzmaStreamAsyncTests.cs new file mode 100644 index 00000000..cc41e831 --- /dev/null +++ b/tests/SharpCompress.Test/Streams/LzmaStreamAsyncTests.cs @@ -0,0 +1,628 @@ +using System; +using System.Buffers; +using System.IO; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Compressors.LZMA; +using SharpCompress.Compressors.Xz; +using SharpCompress.Test.Mocks; +using Xunit; + +namespace SharpCompress.Test.Streams; + +public class LzmaStreamAsyncTests : TestBase +{ + [Fact] + public async ValueTask TestLzma2Decompress() + { + using var stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "bad-1-lzma2-7.xz")); + + using var xz = new XZStream(stream); + await Assert.ThrowsAnyAsync(async () => + await xz.TransferToAsync(Stream.Null, long.MaxValue) + ); + } + + [Fact] + public async ValueTask TestLzma2Decompress1ByteAsync() + { + var properties = new byte[] { 0x01 }; + var compressedData = new byte[] { 0x01, 0x00, 0x00, 0x58, 0x00 }; + var lzma2Stream = new MemoryStream(compressedData); + + var decompressor = LzmaStream.Create(properties, lzma2Stream, 5, 1); + var buffer = new byte[1]; + var bytesRead = await decompressor.ReadAsync(buffer, 0, 1).ConfigureAwait(false); + Assert.Equal(1, bytesRead); + Assert.Equal((byte)'X', buffer[0]); + } + + private static byte[] LzmaData { get; } = + [ + 0x5D, + 0x00, + 0x20, + 0x00, + 0x00, + 0x48, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x80, + 0x24, + 0x18, + 0x2F, + 0xEB, + 0x20, + 0x78, + 0xBA, + 0x78, + 0x70, + 0xDC, + 0x43, + 0x2C, + 0x32, + 0xC9, + 0xC3, + 0x97, + 0x4D, + 0x10, + 0x74, + 0xE2, + 0x20, + 0xBF, + 0x5A, + 0xB4, + 0xB3, + 0xC4, + 0x31, + 0x80, + 0x26, + 0x3E, + 0x6A, + 0xEA, + 0x51, + 0xFC, + 0xE4, + 0x8D, + 0x54, + 0x96, + 0x05, + 0xCC, + 0x78, + 0x59, + 0xAC, + 0xD4, + 0x21, + 0x65, + 0x8F, + 0xA9, + 0xC8, + 0x0D, + 0x9B, + 0xE2, + 0xC2, + 0xF9, + 0x7C, + 0x3C, + 0xDD, + 0x4D, + 0x38, + 0x04, + 0x0B, + 0xF8, + 0x0B, + 0x68, + 0xA5, + 0x93, + 0x6C, + 0x64, + 0xAC, + 0xCF, + 0x71, + 0x68, + 0xE8, + 0x69, + 0x25, + 0xC6, + 0x17, + 0x28, + 0xF1, + 0x7C, + 0xF1, + 0xDC, + 0x47, + 0x51, + 0x4D, + 0x1E, + 0x0E, + 0x0B, + 0x80, + 0x37, + 0x24, + 0x58, + 0x80, + 0xF7, + 0xB4, + 0xAC, + 0x54, + 0xF1, + 0x0F, + 0x7F, + 0x0F, + 0x0F, + 0xF5, + 0x9C, + 0xDE, + 0x54, + 0x4F, + 0xA3, + 0x7B, + 0x20, + 0xC5, + 0xA8, + 0x18, + 0x3B, + 0xED, + 0xDC, + 0x04, + 0xF6, + 0xFB, + 0x86, + 0xE0, + 0xAB, + 0xB6, + 0x87, + 0x99, + 0x92, + 0x43, + 0x7B, + 0x2C, + 0xCC, + 0x31, + 0x83, + 0x90, + 0xFF, + 0xF1, + 0x76, + 0x03, + 0x90, + ]; + + /// + /// The decoded data for . + /// + private static byte[] LzmaResultData { get; } = + [ + 0x01, + 0x00, + 0xFD, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0xFA, + 0x61, + 0x18, + 0x5F, + 0x02, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x02, + 0x00, + 0x00, + 0x00, + 0x03, + 0x00, + 0x00, + 0x00, + 0x01, + 0x00, + 0x00, + 0x00, + 0x02, + 0x00, + 0xB4, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0x3D, + 0x61, + 0xE5, + 0x5E, + 0x03, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x12, + 0x00, + 0x00, + 0x00, + 0x02, + 0x00, + 0xB4, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0xE2, + 0x61, + 0x18, + 0x5F, + 0x04, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x12, + 0x00, + 0x00, + 0x00, + 0x29, + 0x00, + 0x00, + 0x00, + 0x01, + 0x00, + 0xFD, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0x14, + 0x62, + 0x18, + 0x5F, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x03, + 0x00, + 0x00, + 0x00, + 0x40, + 0x00, + 0x00, + 0x00, + 0x09, + 0x00, + 0x00, + 0x00, + 0x02, + 0x00, + 0xB4, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0x7F, + 0x61, + 0xE5, + 0x5E, + 0x05, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x3B, + 0x00, + 0x00, + 0x00, + 0xCB, + 0x15, + 0x00, + 0x00, + 0x02, + 0x00, + 0xB4, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0x7F, + 0x61, + 0xE5, + 0x5E, + 0x06, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x3B, + 0x00, + 0x00, + 0x00, + 0xCB, + 0x15, + 0x00, + 0x00, + 0x02, + 0x00, + 0xB4, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0x3D, + 0x61, + 0xE5, + 0x5E, + 0x07, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x12, + 0x00, + 0x00, + 0x00, + 0x02, + 0x00, + 0xB4, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0xFC, + 0x96, + 0x40, + 0x5C, + 0x08, + 0x00, + 0x00, + 0x00, + 0x60, + 0x00, + 0x00, + 0x00, + 0xFF, + 0xFF, + 0xFF, + 0xFF, + 0x00, + 0x00, + 0x00, + 0x00, + 0xF8, + 0x83, + 0x12, + 0x00, + 0xD4, + 0x99, + 0x00, + 0x00, + 0x43, + 0x95, + 0x00, + 0x00, + 0xEB, + 0x7A, + 0x00, + 0x00, + 0x40, + 0x6F, + 0x00, + 0x00, + 0xD2, + 0x6F, + 0x00, + 0x00, + 0x67, + 0x74, + 0x00, + 0x00, + 0x02, + 0x69, + 0x00, + 0x00, + 0x76, + 0x79, + 0x00, + 0x00, + 0x98, + 0x66, + 0x00, + 0x00, + 0x23, + 0x25, + 0x00, + 0x00, + 0x01, + 0x00, + 0xFD, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0x3B, + 0x2F, + 0xC0, + 0x5F, + 0x09, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x03, + 0x00, + 0x00, + 0x00, + 0x69, + 0x00, + 0x3D, + 0x00, + 0x0A, + 0x00, + 0x00, + 0x00, + ]; + + [Fact] + public async ValueTask TestLzmaBufferAsync() + { + var input = new MemoryStream(LzmaData); + using var output = new MemoryStream(); + var properties = new byte[5]; + await input.ReadAsync(properties, 0, 5).ConfigureAwait(false); + + var fileLengthBytes = new byte[8]; + await input.ReadAsync(fileLengthBytes, 0, 8).ConfigureAwait(false); + var fileLength = BitConverter.ToInt64(fileLengthBytes, 0); + + var coder = new Decoder(); + coder.SetDecoderProperties(properties); + coder.Code(input, output, input.Length, fileLength, null); + + Assert.Equal(output.ToArray(), LzmaResultData); + } + + [Fact] + public async ValueTask TestLzmaStreamEncodingWritesDataAsync() + { + using var inputStream = new MemoryStream(LzmaResultData); + using MemoryStream outputStream = new(); + await using var lzmaStream = LzmaStream.Create( + LzmaEncoderProperties.Default, + false, + new AsyncOnlyStream(outputStream, disposeStream: false) + ); + await inputStream.CopyToAsync(lzmaStream).ConfigureAwait(false); + await lzmaStream.DisposeAsync().ConfigureAwait(false); + Assert.NotEqual(0, outputStream.Length); + } + + [Fact] + public async ValueTask TestLzmaEncodingAccuracyAsync() + { + var input = new MemoryStream(LzmaResultData); + var compressed = new MemoryStream(); + var lzmaEncodingStream = LzmaStream.Create( + LzmaEncoderProperties.Default, + false, + new AsyncOnlyStream(compressed, disposeStream: false) + ); + await input.CopyToAsync(lzmaEncodingStream).ConfigureAwait(false); + await lzmaEncodingStream.DisposeAsync().ConfigureAwait(false); + compressed.Position = 0; + + var output = new MemoryStream(); + await DecompressLzmaStreamAsync( + lzmaEncodingStream.Properties, + compressed, + compressed.Length, + output, + LzmaResultData.LongLength + ) + .ConfigureAwait(false); + + Assert.Equal(output.ToArray(), LzmaResultData); + } + + private static async Task DecompressLzmaStreamAsync( + byte[] properties, + Stream compressedStream, + long compressedSize, + Stream decompressedStream, + long decompressedSize + ) + { + var lzmaStream = await LzmaStream.CreateAsync( + properties, + compressedStream, + compressedSize, + -1, + null, + false + ); + + var buffer = new byte[1024]; + long totalRead = 0; + while (totalRead < decompressedSize) + { + var toRead = (int)Math.Min(buffer.Length, decompressedSize - totalRead); + var read = await lzmaStream.ReadAsync(buffer, 0, toRead).ConfigureAwait(false); + if (read > 0) + { + await decompressedStream.WriteAsync(buffer, 0, read).ConfigureAwait(false); + totalRead += read; + } + else + { + break; + } + } + } +} diff --git a/tests/SharpCompress.Test/Streams/LzmaStreamTests.cs b/tests/SharpCompress.Test/Streams/LzmaStreamTests.cs index 2f336329..8b7ebea0 100644 --- a/tests/SharpCompress.Test/Streams/LzmaStreamTests.cs +++ b/tests/SharpCompress.Test/Streams/LzmaStreamTests.cs @@ -1,6 +1,5 @@ using System; using System.Buffers; -using System.Buffers.Binary; using System.IO; using SharpCompress.Compressors.LZMA; using Xunit; @@ -16,509 +15,507 @@ public class LzmaStreamTests var compressedData = new byte[] { 0x01, 0x00, 0x00, 0x58, 0x00 }; var lzma2Stream = new MemoryStream(compressedData); - var decompressor = new LzmaStream(properties, lzma2Stream, 5, 1); + var decompressor = LzmaStream.Create(properties, lzma2Stream, 5, 1); Assert.Equal('X', decompressor.ReadByte()); } - private static byte[] lzmaData { get; } = - new byte[] - { - 0x5D, - 0x00, - 0x20, - 0x00, - 0x00, - 0x48, - 0x01, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x80, - 0x24, - 0x18, - 0x2F, - 0xEB, - 0x20, - 0x78, - 0xBA, - 0x78, - 0x70, - 0xDC, - 0x43, - 0x2C, - 0x32, - 0xC9, - 0xC3, - 0x97, - 0x4D, - 0x10, - 0x74, - 0xE2, - 0x20, - 0xBF, - 0x5A, - 0xB4, - 0xB3, - 0xC4, - 0x31, - 0x80, - 0x26, - 0x3E, - 0x6A, - 0xEA, - 0x51, - 0xFC, - 0xE4, - 0x8D, - 0x54, - 0x96, - 0x05, - 0xCC, - 0x78, - 0x59, - 0xAC, - 0xD4, - 0x21, - 0x65, - 0x8F, - 0xA9, - 0xC8, - 0x0D, - 0x9B, - 0xE2, - 0xC2, - 0xF9, - 0x7C, - 0x3C, - 0xDD, - 0x4D, - 0x38, - 0x04, - 0x0B, - 0xF8, - 0x0B, - 0x68, - 0xA5, - 0x93, - 0x6C, - 0x64, - 0xAC, - 0xCF, - 0x71, - 0x68, - 0xE8, - 0x69, - 0x25, - 0xC6, - 0x17, - 0x28, - 0xF1, - 0x7C, - 0xF1, - 0xDC, - 0x47, - 0x51, - 0x4D, - 0x1E, - 0x0E, - 0x0B, - 0x80, - 0x37, - 0x24, - 0x58, - 0x80, - 0xF7, - 0xB4, - 0xAC, - 0x54, - 0xF1, - 0x0F, - 0x7F, - 0x0F, - 0x0F, - 0xF5, - 0x9C, - 0xDE, - 0x54, - 0x4F, - 0xA3, - 0x7B, - 0x20, - 0xC5, - 0xA8, - 0x18, - 0x3B, - 0xED, - 0xDC, - 0x04, - 0xF6, - 0xFB, - 0x86, - 0xE0, - 0xAB, - 0xB6, - 0x87, - 0x99, - 0x92, - 0x43, - 0x7B, - 0x2C, - 0xCC, - 0x31, - 0x83, - 0x90, - 0xFF, - 0xF1, - 0x76, - 0x03, - 0x90 - }; + private static byte[] LzmaData { get; } = + [ + 0x5D, + 0x00, + 0x20, + 0x00, + 0x00, + 0x48, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x80, + 0x24, + 0x18, + 0x2F, + 0xEB, + 0x20, + 0x78, + 0xBA, + 0x78, + 0x70, + 0xDC, + 0x43, + 0x2C, + 0x32, + 0xC9, + 0xC3, + 0x97, + 0x4D, + 0x10, + 0x74, + 0xE2, + 0x20, + 0xBF, + 0x5A, + 0xB4, + 0xB3, + 0xC4, + 0x31, + 0x80, + 0x26, + 0x3E, + 0x6A, + 0xEA, + 0x51, + 0xFC, + 0xE4, + 0x8D, + 0x54, + 0x96, + 0x05, + 0xCC, + 0x78, + 0x59, + 0xAC, + 0xD4, + 0x21, + 0x65, + 0x8F, + 0xA9, + 0xC8, + 0x0D, + 0x9B, + 0xE2, + 0xC2, + 0xF9, + 0x7C, + 0x3C, + 0xDD, + 0x4D, + 0x38, + 0x04, + 0x0B, + 0xF8, + 0x0B, + 0x68, + 0xA5, + 0x93, + 0x6C, + 0x64, + 0xAC, + 0xCF, + 0x71, + 0x68, + 0xE8, + 0x69, + 0x25, + 0xC6, + 0x17, + 0x28, + 0xF1, + 0x7C, + 0xF1, + 0xDC, + 0x47, + 0x51, + 0x4D, + 0x1E, + 0x0E, + 0x0B, + 0x80, + 0x37, + 0x24, + 0x58, + 0x80, + 0xF7, + 0xB4, + 0xAC, + 0x54, + 0xF1, + 0x0F, + 0x7F, + 0x0F, + 0x0F, + 0xF5, + 0x9C, + 0xDE, + 0x54, + 0x4F, + 0xA3, + 0x7B, + 0x20, + 0xC5, + 0xA8, + 0x18, + 0x3B, + 0xED, + 0xDC, + 0x04, + 0xF6, + 0xFB, + 0x86, + 0xE0, + 0xAB, + 0xB6, + 0x87, + 0x99, + 0x92, + 0x43, + 0x7B, + 0x2C, + 0xCC, + 0x31, + 0x83, + 0x90, + 0xFF, + 0xF1, + 0x76, + 0x03, + 0x90, + ]; /// - /// The decoded data for . + /// The decoded data for . /// - private static byte[] lzmaResultData { get; } = - new byte[] - { - 0x01, - 0x00, - 0xFD, - 0x01, - 0x00, - 0x00, - 0x00, - 0x00, - 0xFA, - 0x61, - 0x18, - 0x5F, - 0x02, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x02, - 0x00, - 0x00, - 0x00, - 0x03, - 0x00, - 0x00, - 0x00, - 0x01, - 0x00, - 0x00, - 0x00, - 0x02, - 0x00, - 0xB4, - 0x01, - 0x00, - 0x00, - 0x00, - 0x00, - 0x3D, - 0x61, - 0xE5, - 0x5E, - 0x03, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x12, - 0x00, - 0x00, - 0x00, - 0x02, - 0x00, - 0xB4, - 0x01, - 0x00, - 0x00, - 0x00, - 0x00, - 0xE2, - 0x61, - 0x18, - 0x5F, - 0x04, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x12, - 0x00, - 0x00, - 0x00, - 0x29, - 0x00, - 0x00, - 0x00, - 0x01, - 0x00, - 0xFD, - 0x01, - 0x00, - 0x00, - 0x00, - 0x00, - 0x14, - 0x62, - 0x18, - 0x5F, - 0x01, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x03, - 0x00, - 0x00, - 0x00, - 0x40, - 0x00, - 0x00, - 0x00, - 0x09, - 0x00, - 0x00, - 0x00, - 0x02, - 0x00, - 0xB4, - 0x01, - 0x00, - 0x00, - 0x00, - 0x00, - 0x7F, - 0x61, - 0xE5, - 0x5E, - 0x05, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x3B, - 0x00, - 0x00, - 0x00, - 0xCB, - 0x15, - 0x00, - 0x00, - 0x02, - 0x00, - 0xB4, - 0x01, - 0x00, - 0x00, - 0x00, - 0x00, - 0x7F, - 0x61, - 0xE5, - 0x5E, - 0x06, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x3B, - 0x00, - 0x00, - 0x00, - 0xCB, - 0x15, - 0x00, - 0x00, - 0x02, - 0x00, - 0xB4, - 0x01, - 0x00, - 0x00, - 0x00, - 0x00, - 0x3D, - 0x61, - 0xE5, - 0x5E, - 0x07, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x12, - 0x00, - 0x00, - 0x00, - 0x02, - 0x00, - 0xB4, - 0x01, - 0x00, - 0x00, - 0x00, - 0x00, - 0xFC, - 0x96, - 0x40, - 0x5C, - 0x08, - 0x00, - 0x00, - 0x00, - 0x60, - 0x00, - 0x00, - 0x00, - 0xFF, - 0xFF, - 0xFF, - 0xFF, - 0x00, - 0x00, - 0x00, - 0x00, - 0xF8, - 0x83, - 0x12, - 0x00, - 0xD4, - 0x99, - 0x00, - 0x00, - 0x43, - 0x95, - 0x00, - 0x00, - 0xEB, - 0x7A, - 0x00, - 0x00, - 0x40, - 0x6F, - 0x00, - 0x00, - 0xD2, - 0x6F, - 0x00, - 0x00, - 0x67, - 0x74, - 0x00, - 0x00, - 0x02, - 0x69, - 0x00, - 0x00, - 0x76, - 0x79, - 0x00, - 0x00, - 0x98, - 0x66, - 0x00, - 0x00, - 0x23, - 0x25, - 0x00, - 0x00, - 0x01, - 0x00, - 0xFD, - 0x01, - 0x00, - 0x00, - 0x00, - 0x00, - 0x3B, - 0x2F, - 0xC0, - 0x5F, - 0x09, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x00, - 0x03, - 0x00, - 0x00, - 0x00, - 0x69, - 0x00, - 0x3D, - 0x00, - 0x0A, - 0x00, - 0x00, - 0x00 - }; + private static byte[] LzmaResultData { get; } = + [ + 0x01, + 0x00, + 0xFD, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0xFA, + 0x61, + 0x18, + 0x5F, + 0x02, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x02, + 0x00, + 0x00, + 0x00, + 0x03, + 0x00, + 0x00, + 0x00, + 0x01, + 0x00, + 0x00, + 0x00, + 0x02, + 0x00, + 0xB4, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0x3D, + 0x61, + 0xE5, + 0x5E, + 0x03, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x12, + 0x00, + 0x00, + 0x00, + 0x02, + 0x00, + 0xB4, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0xE2, + 0x61, + 0x18, + 0x5F, + 0x04, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x12, + 0x00, + 0x00, + 0x00, + 0x29, + 0x00, + 0x00, + 0x00, + 0x01, + 0x00, + 0xFD, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0x14, + 0x62, + 0x18, + 0x5F, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x03, + 0x00, + 0x00, + 0x00, + 0x40, + 0x00, + 0x00, + 0x00, + 0x09, + 0x00, + 0x00, + 0x00, + 0x02, + 0x00, + 0xB4, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0x7F, + 0x61, + 0xE5, + 0x5E, + 0x05, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x3B, + 0x00, + 0x00, + 0x00, + 0xCB, + 0x15, + 0x00, + 0x00, + 0x02, + 0x00, + 0xB4, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0x7F, + 0x61, + 0xE5, + 0x5E, + 0x06, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x3B, + 0x00, + 0x00, + 0x00, + 0xCB, + 0x15, + 0x00, + 0x00, + 0x02, + 0x00, + 0xB4, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0x3D, + 0x61, + 0xE5, + 0x5E, + 0x07, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x12, + 0x00, + 0x00, + 0x00, + 0x02, + 0x00, + 0xB4, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0xFC, + 0x96, + 0x40, + 0x5C, + 0x08, + 0x00, + 0x00, + 0x00, + 0x60, + 0x00, + 0x00, + 0x00, + 0xFF, + 0xFF, + 0xFF, + 0xFF, + 0x00, + 0x00, + 0x00, + 0x00, + 0xF8, + 0x83, + 0x12, + 0x00, + 0xD4, + 0x99, + 0x00, + 0x00, + 0x43, + 0x95, + 0x00, + 0x00, + 0xEB, + 0x7A, + 0x00, + 0x00, + 0x40, + 0x6F, + 0x00, + 0x00, + 0xD2, + 0x6F, + 0x00, + 0x00, + 0x67, + 0x74, + 0x00, + 0x00, + 0x02, + 0x69, + 0x00, + 0x00, + 0x76, + 0x79, + 0x00, + 0x00, + 0x98, + 0x66, + 0x00, + 0x00, + 0x23, + 0x25, + 0x00, + 0x00, + 0x01, + 0x00, + 0xFD, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0x3B, + 0x2F, + 0xC0, + 0x5F, + 0x09, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x00, + 0x03, + 0x00, + 0x00, + 0x00, + 0x69, + 0x00, + 0x3D, + 0x00, + 0x0A, + 0x00, + 0x00, + 0x00, + ]; [Fact] public void TestLzmaBuffer() { - var input = new MemoryStream(lzmaData); + var input = new MemoryStream(LzmaData); using var output = new MemoryStream(); var properties = new byte[5]; input.Read(properties, 0, 5); @@ -531,15 +528,15 @@ public class LzmaStreamTests coder.SetDecoderProperties(properties); coder.Code(input, output, input.Length, fileLength, null); - Assert.Equal(output.ToArray(), lzmaResultData); + Assert.Equal(output.ToArray(), LzmaResultData); } [Fact] public void TestLzmaStreamEncodingWritesData() { - using MemoryStream inputStream = new MemoryStream(lzmaResultData); + using var inputStream = new MemoryStream(LzmaResultData); using MemoryStream outputStream = new(); - using LzmaStream lzmaStream = new LzmaStream( + using var lzmaStream = LzmaStream.Create( LzmaEncoderProperties.Default, false, outputStream @@ -552,9 +549,9 @@ public class LzmaStreamTests [Fact] public void TestLzmaEncodingAccuracy() { - var input = new MemoryStream(lzmaResultData); + var input = new MemoryStream(LzmaResultData); var compressed = new MemoryStream(); - LzmaStream lzmaEncodingStream = new LzmaStream( + var lzmaEncodingStream = LzmaStream.Create( LzmaEncoderProperties.Default, false, compressed @@ -569,10 +566,10 @@ public class LzmaStreamTests compressed, compressed.Length, output, - lzmaResultData.LongLength + LzmaResultData.LongLength ); - Assert.Equal(output.ToArray(), lzmaResultData); + Assert.Equal(output.ToArray(), LzmaResultData); } private static void DecompressLzmaStream( @@ -583,7 +580,7 @@ public class LzmaStreamTests long decompressedSize ) { - LzmaStream lzmaStream = new LzmaStream( + var lzmaStream = LzmaStream.Create( properties, compressedStream, compressedSize, @@ -592,12 +589,12 @@ public class LzmaStreamTests false ); - byte[] buffer = ArrayPool.Shared.Rent(1024); + var buffer = new byte[1024]; long totalRead = 0; while (totalRead < decompressedSize) { - int toRead = (int)Math.Min(buffer.Length, decompressedSize - totalRead); - int read = lzmaStream.Read(buffer, 0, toRead); + var toRead = (int)Math.Min(buffer.Length, decompressedSize - totalRead); + var read = lzmaStream.Read(buffer, 0, toRead); if (read > 0) { decompressedStream.Write(buffer, 0, read); @@ -608,6 +605,5 @@ public class LzmaStreamTests break; } } - ArrayPool.Shared.Return(buffer); } } diff --git a/tests/SharpCompress.Test/Streams/LzwStreamAsyncTests.cs b/tests/SharpCompress.Test/Streams/LzwStreamAsyncTests.cs new file mode 100644 index 00000000..e91089cd --- /dev/null +++ b/tests/SharpCompress.Test/Streams/LzwStreamAsyncTests.cs @@ -0,0 +1,175 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Compressors.Lzw; +using Xunit; + +namespace SharpCompress.Test.Streams; + +public class LzwStreamAsyncTests : TestBase +{ + [Fact] + public async Task LzwStream_ReadAsync_ByteArray() + { + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.Z"); + using var stream = File.OpenRead(testArchive); + using var lzwStream = new LzwStream(stream); + + var buffer = new byte[4096]; + int bytesRead = await lzwStream.ReadAsync(buffer, 0, buffer.Length).ConfigureAwait(false); + + Assert.True(bytesRead > 0, "Should read at least some data"); + } + +#if !LEGACY_DOTNET + [Fact] + public async Task LzwStream_ReadAsync_Memory() + { + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.Z"); + using var stream = File.OpenRead(testArchive); + using var lzwStream = new LzwStream(stream); + + var buffer = new byte[4096]; + int bytesRead = await lzwStream.ReadAsync(new Memory(buffer)).ConfigureAwait(false); + + Assert.True(bytesRead > 0, "Should read at least some data"); + } +#endif + + [Fact] + public async Task LzwStream_ReadAsync_ProducesSameResultAsSync() + { + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.Z"); + + byte[] syncResult; + byte[] asyncResult; + + using (var stream = File.OpenRead(testArchive)) + using (var lzwStream = new LzwStream(stream)) + { + syncResult = ReadAllSync(lzwStream); + } + + using (var stream = File.OpenRead(testArchive)) + using (var lzwStream = new LzwStream(stream)) + { + asyncResult = await ReadAllAsync(lzwStream).ConfigureAwait(false); + } + + Assert.Equal(syncResult, asyncResult); + } + + [Fact] + public async Task LzwStream_ReadAsync_MultipleReads() + { + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.Z"); + using var stream = File.OpenRead(testArchive); + using var lzwStream = new LzwStream(stream); + + var totalData = new List(); + var buffer = new byte[1024]; + int bytesRead; + + while ( + (bytesRead = await lzwStream.ReadAsync(buffer, 0, buffer.Length).ConfigureAwait(false)) + > 0 + ) + { + for (int i = 0; i < bytesRead; i++) + { + totalData.Add(buffer[i]); + } + } + + Assert.True(totalData.Count > 0, "Should have read some data"); + } + + [Fact] + public async Task LzwStream_ReadAsync_Cancellation() + { + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.Z"); + using var stream = File.OpenRead(testArchive); + using var lzwStream = new LzwStream(stream); + + var cts = new CancellationTokenSource(); + var buffer = new byte[4096]; + + cts.Cancel(); + + await Assert.ThrowsAsync(async () => + await lzwStream.ReadAsync(buffer, 0, buffer.Length, cts.Token) + ); + } + + [Fact] + public async Task LzwStream_ReadAsync_EmptyBuffer() + { + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.Z"); + using var stream = File.OpenRead(testArchive); + using var lzwStream = new LzwStream(stream); + + var buffer = Array.Empty(); + int bytesRead = await lzwStream.ReadAsync(buffer, 0, 0).ConfigureAwait(false); + + Assert.Equal(0, bytesRead); + } + + [Fact] + public async Task LzwStream_ReadAsync_ReturnsZeroAtEndOfStream() + { + var testArchive = Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.Z"); + using var stream = File.OpenRead(testArchive); + using var lzwStream = new LzwStream(stream); + + var buffer = new byte[4096]; + + int bytesRead; + while ( + (bytesRead = await lzwStream.ReadAsync(buffer, 0, buffer.Length).ConfigureAwait(false)) + > 0 + ) { } + + Assert.Equal(0, bytesRead); + + bytesRead = await lzwStream.ReadAsync(buffer, 0, buffer.Length).ConfigureAwait(false); + Assert.Equal(0, bytesRead); + } + + private static async Task ReadAllAsync(LzwStream stream) + { + var result = new List(); + var buffer = new byte[4096]; + int bytesRead; + + while ( + (bytesRead = await stream.ReadAsync(buffer, 0, buffer.Length).ConfigureAwait(false)) > 0 + ) + { + for (int i = 0; i < bytesRead; i++) + { + result.Add(buffer[i]); + } + } + + return result.ToArray(); + } + + private static byte[] ReadAllSync(LzwStream stream) + { + var result = new List(); + var buffer = new byte[4096]; + int bytesRead; + + while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) > 0) + { + for (int i = 0; i < bytesRead; i++) + { + result.Add(buffer[i]); + } + } + + return result.ToArray(); + } +} diff --git a/tests/SharpCompress.Test/Streams/PooledMemoryStreamTests.cs b/tests/SharpCompress.Test/Streams/PooledMemoryStreamTests.cs new file mode 100644 index 00000000..d2b0996c --- /dev/null +++ b/tests/SharpCompress.Test/Streams/PooledMemoryStreamTests.cs @@ -0,0 +1,356 @@ +using System; +using System.Buffers; +using System.IO; +using SharpCompress.IO; +using Xunit; + +namespace SharpCompress.Test.Streams; + +public class PooledMemoryStreamTests +{ + [Fact] + public void GrowsUsingFixedSizeBlocks() + { + var pool = new TrackingArrayPool(); + + using var stream = new PooledMemoryStream(capacity: 0, blockSize: 8, arrayPool: pool); + stream.Write(new byte[20], 0, 20); + + Assert.Equal(3, pool.RentRequests.Count); + Assert.All(pool.RentRequests, requested => Assert.Equal(8, requested)); + } + + [Fact] + public void DisposeReturnsRentedBlocksToPool() + { + var pool = new TrackingArrayPool(); + var stream = new PooledMemoryStream(capacity: 0, blockSize: 8, arrayPool: pool); + + stream.Write(new byte[17], 0, 17); + stream.Dispose(); + + Assert.Equal(pool.RentRequests.Count, pool.ReturnedLengths.Count); + Assert.All(pool.ReturnedLengths, length => Assert.Equal(8, length)); + } + + [Fact] + public void OverRentedBlocksUseLogicalBlockSize() + { + var pool = new FilledOverRentingArrayPool(extraLength: 8, fillValue: 0x5A); + + using var stream = new PooledMemoryStream(capacity: 0, blockSize: 8, arrayPool: pool); + stream.Write(new byte[] { 1, 2, 3, 4, 5 }, 0, 5); + + stream.Position = 10; + stream.Write(new byte[] { 42, 43, 44, 45, 46, 47, 48, 49, 50, 51 }, 0, 10); + + Assert.Equal(3, pool.RentRequests.Count); + Assert.All(pool.RentRequests, requested => Assert.Equal(8, requested)); + Assert.All(pool.RentedLengths, length => Assert.Equal(16, length)); + + var expected = new byte[] + { + 1, + 2, + 3, + 4, + 5, + 0, + 0, + 0, + 0, + 0, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + }; + + Assert.Equal(expected, stream.ToArray()); + + stream.Position = 0; + var roundTrip = new byte[expected.Length]; + Assert.Equal(expected.Length, stream.Read(roundTrip, 0, roundTrip.Length)); + Assert.Equal(expected, roundTrip); + } + + [Fact] + public void GetBufferReturnsArraySizedToCapacityWithoutTouchingPool() + { + var pool = new OverRentingArrayPool(extraLength: 8); + + using var stream = new PooledMemoryStream(capacity: 0, blockSize: 8, arrayPool: pool); + stream.Write(new byte[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 }, 0, 10); + + var rentsBefore = pool.RentRequests.Count; + var returnsBefore = pool.ReturnedLengths.Count; + + var buffer = stream.GetBuffer(); + Assert.Equal(16, buffer.Length); + Assert.Equal(1, buffer[0]); + Assert.Equal(10, buffer[9]); + Assert.Equal(0, buffer[10]); + Assert.Equal(0, buffer[15]); + Assert.Equal(rentsBefore, pool.RentRequests.Count); + Assert.Equal(returnsBefore, pool.ReturnedLengths.Count); + + buffer[0] = 255; + stream.Position = 0; + Assert.Equal(1, stream.ReadByte()); + } + + [Fact] + public void TryGetBufferReturnsSegmentWhenOpen() + { + using var stream = new PooledMemoryStream(capacity: 0, blockSize: 8); + stream.Write(new byte[] { 1, 2, 3, 4 }, 0, 4); + + Assert.True(stream.TryGetBuffer(out var segment)); + Assert.Equal(0, segment.Offset); + Assert.Equal(4, segment.Count); + Assert.Equal(1, segment.Array![0]); + } + + [Fact] + public void TryGetBufferReturnsArraySizedToCapacityWithoutTouchingPool() + { + var pool = new OverRentingArrayPool(extraLength: 8); + + using var stream = new PooledMemoryStream(capacity: 0, blockSize: 8, arrayPool: pool); + stream.Write(new byte[] { 1, 2, 3, 4 }, 0, 4); + + var rentsBefore = pool.RentRequests.Count; + var returnsBefore = pool.ReturnedLengths.Count; + + Assert.True(stream.TryGetBuffer(out var segment)); + Assert.Equal(0, segment.Offset); + Assert.Equal(4, segment.Count); + Assert.Equal(8, segment.Array!.Length); + Assert.Equal(1, segment.Array[0]); + Assert.Equal(4, segment.Array[3]); + Assert.Equal(0, segment.Array[4]); + Assert.Equal(0, segment.Array[7]); + Assert.Equal(rentsBefore, pool.RentRequests.Count); + Assert.Equal(returnsBefore, pool.ReturnedLengths.Count); + + segment.Array[0] = 255; + stream.Position = 0; + Assert.Equal(1, stream.ReadByte()); + } + + [Fact] + public void CapacitySetterCanGrowAndShrinkWithinLength() + { + using var stream = new PooledMemoryStream(capacity: 16, blockSize: 8); + stream.Write(new byte[6], 0, 6); + + stream.Capacity = 24; + Assert.Equal(24, stream.Capacity); + + stream.Capacity = 8; + Assert.Equal(8, stream.Capacity); + } + + [Fact] + public void SetLengthExtendingClearsGap() + { + using var stream = new PooledMemoryStream(capacity: 0, blockSize: 8); + stream.Position = 5; + stream.WriteByte(42); + stream.Position = 0; + + var data = stream.ToArray(); + Assert.Equal(6, data.Length); + Assert.Equal(0, data[0]); + Assert.Equal(0, data[4]); + Assert.Equal(42, data[5]); + } + + [Fact] + public void MethodsThrowAfterDispose() + { + using var stream = new PooledMemoryStream(capacity: 0, blockSize: 8); + stream.WriteByte(1); + stream.Dispose(); + + Assert.Throws(() => stream.ReadByte()); + Assert.Throws(() => stream.ToArray()); + Assert.Throws(() => stream.GetBuffer()); + } + + [Fact] + public void MultipleGetBufferCallsReturnDifferentArrays() + { + using var stream = new PooledMemoryStream(capacity: 0, blockSize: 8); + stream.Write(new byte[] { 1, 2, 3 }, 0, 3); + + var buffer1 = stream.GetBuffer(); + var buffer2 = stream.GetBuffer(); + + Assert.NotSame(buffer1, buffer2); + Assert.Equal(buffer1, buffer2); + } + + [Fact] + public void SeekBeyondMaxLengthThrows() + { + using var stream = new PooledMemoryStream(); + Assert.Throws(() => + stream.Seek(int.MaxValue + 1L, SeekOrigin.Begin) + ); + } + + [Fact] + public void DisposeAfterGetBufferDoesNotReturnExposedArrayToPool() + { + var pool = new OverRentingArrayPool(extraLength: 8); + byte[] buffer; + + using (var stream = new PooledMemoryStream(capacity: 0, blockSize: 8, pool)) + { + stream.Write(new byte[] { 1, 2, 3 }, 0, 3); + buffer = stream.GetBuffer(); + + Assert.NotNull(buffer); + Assert.NotEmpty(pool.RentRequests); + } + + Assert.DoesNotContain(buffer, pool.ReturnedArrays); + Assert.Equal(1, buffer[0]); + Assert.Equal(2, buffer[1]); + Assert.Equal(3, buffer[2]); + } + + [Fact] + public void DisposeAfterTryGetBufferDoesNotReturnExposedArrayToPool() + { + var pool = new OverRentingArrayPool(extraLength: 8); + ArraySegment segment; + + using (var stream = new PooledMemoryStream(capacity: 0, blockSize: 8, pool)) + { + stream.Write(new byte[] { 1, 2, 3 }, 0, 3); + + Assert.True(stream.TryGetBuffer(out segment)); + Assert.NotNull(segment.Array); + Assert.NotEmpty(pool.RentRequests); + } + + Assert.DoesNotContain(segment.Array!, pool.ReturnedArrays); + Assert.Equal(1, segment.Array![segment.Offset]); + Assert.Equal(2, segment.Array[segment.Offset + 1]); + Assert.Equal(3, segment.Array[segment.Offset + 2]); + } + + [Fact] + public void SetLengthNearIntMaxValueThrowsIOExceptionWhenBlockRoundingOverflows() + { + using var stream = new PooledMemoryStream(capacity: 0, blockSize: 8); + var length = int.MaxValue - 1L; + + Assert.Throws(() => stream.SetLength(length)); + Assert.Equal(0, stream.Length); + } + + private sealed class TrackingArrayPool : ArrayPool + { + private const byte RentedBufferFillValue = 0x5A; + + public readonly System.Collections.Generic.List RentRequests = new(); + public readonly System.Collections.Generic.List ReturnedLengths = new(); + + public override byte[] Rent(int minimumLength) + { + RentRequests.Add(minimumLength); + + var array = new byte[minimumLength]; + for (var i = 0; i < array.Length; i++) + { + array[i] = RentedBufferFillValue; + } + + return array; + } + + public override void Return(byte[] array, bool clearArray = false) + { + ReturnedLengths.Add(array.Length); + if (clearArray) + { + Array.Clear(array, 0, array.Length); + } + } + } + + private sealed class OverRentingArrayPool : ArrayPool + { + private readonly int _extraLength; + + public OverRentingArrayPool(int extraLength) + { + _extraLength = extraLength; + } + + public readonly System.Collections.Generic.List RentRequests = new(); + public readonly System.Collections.Generic.List ReturnedLengths = new(); + public readonly System.Collections.Generic.List ReturnedArrays = new(); + + public override byte[] Rent(int minimumLength) + { + RentRequests.Add(minimumLength); + return new byte[minimumLength + _extraLength]; + } + + public override void Return(byte[] array, bool clearArray = false) + { + ReturnedLengths.Add(array.Length); + ReturnedArrays.Add(array); + if (clearArray) + { + Array.Clear(array, 0, array.Length); + } + } + } + + private sealed class FilledOverRentingArrayPool : ArrayPool + { + private readonly int _extraLength; + private readonly byte _fillValue; + + public FilledOverRentingArrayPool(int extraLength, byte fillValue) + { + _extraLength = extraLength; + _fillValue = fillValue; + } + + public readonly System.Collections.Generic.List RentRequests = new(); + public readonly System.Collections.Generic.List RentedLengths = new(); + + public override byte[] Rent(int minimumLength) + { + RentRequests.Add(minimumLength); + + var array = new byte[minimumLength + _extraLength]; + RentedLengths.Add(array.Length); + for (var i = 0; i < array.Length; i++) + { + array[i] = _fillValue; + } + return array; + } + + public override void Return(byte[] array, bool clearArray = false) + { + if (clearArray) + { + Array.Clear(array, 0, array.Length); + } + } + } +} diff --git a/tests/SharpCompress.Test/Streams/RewindableStreamTest.cs b/tests/SharpCompress.Test/Streams/RewindableStreamTest.cs deleted file mode 100644 index ab18da45..00000000 --- a/tests/SharpCompress.Test/Streams/RewindableStreamTest.cs +++ /dev/null @@ -1,82 +0,0 @@ -using System.IO; -using SharpCompress.IO; -using Xunit; - -namespace SharpCompress.Test.Streams; - -public class RewindableStreamTest -{ - [Fact] - public void TestRewind() - { - var ms = new MemoryStream(); - var bw = new BinaryWriter(ms); - bw.Write(1); - bw.Write(2); - bw.Write(3); - bw.Write(4); - bw.Write(5); - bw.Write(6); - bw.Write(7); - bw.Flush(); - ms.Position = 0; - var stream = new RewindableStream(ms); - stream.StartRecording(); - var br = new BinaryReader(stream); - Assert.Equal(1, br.ReadInt32()); - Assert.Equal(2, br.ReadInt32()); - Assert.Equal(3, br.ReadInt32()); - Assert.Equal(4, br.ReadInt32()); - stream.Rewind(true); - stream.StartRecording(); - Assert.Equal(1, br.ReadInt32()); - Assert.Equal(2, br.ReadInt32()); - Assert.Equal(3, br.ReadInt32()); - Assert.Equal(4, br.ReadInt32()); - Assert.Equal(5, br.ReadInt32()); - Assert.Equal(6, br.ReadInt32()); - Assert.Equal(7, br.ReadInt32()); - stream.Rewind(true); - stream.StartRecording(); - Assert.Equal(1, br.ReadInt32()); - Assert.Equal(2, br.ReadInt32()); - Assert.Equal(3, br.ReadInt32()); - Assert.Equal(4, br.ReadInt32()); - } - - [Fact] - public void TestIncompleteRewind() - { - var ms = new MemoryStream(); - var bw = new BinaryWriter(ms); - bw.Write(1); - bw.Write(2); - bw.Write(3); - bw.Write(4); - bw.Write(5); - bw.Write(6); - bw.Write(7); - bw.Flush(); - ms.Position = 0; - var stream = new RewindableStream(ms); - stream.StartRecording(); - var br = new BinaryReader(stream); - Assert.Equal(1, br.ReadInt32()); - Assert.Equal(2, br.ReadInt32()); - Assert.Equal(3, br.ReadInt32()); - Assert.Equal(4, br.ReadInt32()); - stream.Rewind(true); - Assert.Equal(1, br.ReadInt32()); - Assert.Equal(2, br.ReadInt32()); - stream.StartRecording(); - Assert.Equal(3, br.ReadInt32()); - Assert.Equal(4, br.ReadInt32()); - Assert.Equal(5, br.ReadInt32()); - stream.Rewind(true); - Assert.Equal(3, br.ReadInt32()); - Assert.Equal(4, br.ReadInt32()); - Assert.Equal(5, br.ReadInt32()); - Assert.Equal(6, br.ReadInt32()); - Assert.Equal(7, br.ReadInt32()); - } -} diff --git a/tests/SharpCompress.Test/Streams/RingBufferTests.cs b/tests/SharpCompress.Test/Streams/RingBufferTests.cs new file mode 100644 index 00000000..e03e37ac --- /dev/null +++ b/tests/SharpCompress.Test/Streams/RingBufferTests.cs @@ -0,0 +1,486 @@ +using System; +using SharpCompress.IO; +using Xunit; + +namespace SharpCompress.Test.Streams; + +public class RingBufferTests +{ + #region Constructor Tests + + [Fact] + public void Constructor_ValidCapacity_SuccessfullyCreates() + { + var buffer = new RingBuffer(100); + Assert.Equal(100, buffer.Capacity); + Assert.Equal(0, buffer.Length); + buffer.Dispose(); + } + + [Fact] + public void Constructor_ZeroCapacity_ThrowsArgumentOutOfRangeException() + { + var ex = Assert.Throws(() => new RingBuffer(0)); + Assert.Contains("Capacity must be positive", ex.Message); + } + + [Fact] + public void Constructor_NegativeCapacity_ThrowsArgumentOutOfRangeException() + { + var ex = Assert.Throws(() => new RingBuffer(-10)); + Assert.Contains("Capacity must be positive", ex.Message); + } + + #endregion + + #region Write Tests + + [Fact] + public void Write_DataWithinCapacity_UpdatesLengthCorrectly() + { + var buffer = new RingBuffer(100); + var data = new byte[] { 1, 2, 3, 4, 5 }; + + buffer.Write(data, 0, 5); + Assert.Equal(5, buffer.Length); + + buffer.Dispose(); + } + + [Fact] + public void Write_ZeroBytes_ReturnsWithoutChangingBuffer() + { + var buffer = new RingBuffer(100); + var data = new byte[] { 1, 2, 3, 4, 5 }; + + buffer.Write(data, 0, 0); + Assert.Equal(0, buffer.Length); + + buffer.Dispose(); + } + + [Fact] + public void Write_DataExceedsCapacity_KeepsLastNBytes() + { + var buffer = new RingBuffer(5); + var data = new byte[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 }; + + buffer.Write(data, 0, 10); + Assert.Equal(5, buffer.Length); + + var readBuffer = new byte[5]; + buffer.ReadFromEnd(5, readBuffer, 0, 5); + Assert.Equal(new byte[] { 6, 7, 8, 9, 10 }, readBuffer); + + buffer.Dispose(); + } + + [Fact] + public void Write_DataEqualToCapacity_FillsCompleteBuffer() + { + var buffer = new RingBuffer(5); + var data = new byte[] { 1, 2, 3, 4, 5 }; + + buffer.Write(data, 0, 5); + Assert.Equal(5, buffer.Length); + + var readBuffer = new byte[5]; + buffer.ReadFromEnd(5, readBuffer, 0, 5); + Assert.Equal(new byte[] { 1, 2, 3, 4, 5 }, readBuffer); + + buffer.Dispose(); + } + + [Fact] + public void Write_MultipleWrites_TracksCumulativeLength() + { + var buffer = new RingBuffer(100); + var data1 = new byte[] { 1, 2, 3 }; + var data2 = new byte[] { 4, 5, 6 }; + + buffer.Write(data1, 0, 3); + Assert.Equal(3, buffer.Length); + + buffer.Write(data2, 0, 3); + Assert.Equal(6, buffer.Length); + + buffer.Dispose(); + } + + [Fact] + public void Write_WrapAround_CircularWritePosition() + { + var buffer = new RingBuffer(5); + var data1 = new byte[] { 1, 2, 3, 4, 5 }; + var data2 = new byte[] { 6, 7 }; + + buffer.Write(data1, 0, 5); + buffer.Write(data2, 0, 2); + Assert.Equal(5, buffer.Length); + + var readBuffer = new byte[5]; + buffer.ReadFromEnd(5, readBuffer, 0, 5); + // After writing 5 bytes (full), writePos=0. After writing 2 more bytes, + // writePos=2, and buffer contains [6, 7, 3, 4, 5] (last 5 bytes total) + Assert.Equal(new byte[] { 3, 4, 5, 6, 7 }, readBuffer); + + buffer.Dispose(); + } + + [Fact] + public void Write_LargeDataWhenBufferFull_ReplacesOldData() + { + var buffer = new RingBuffer(10); + var data1 = new byte[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 }; + var data2 = new byte[] { 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 }; + + buffer.Write(data1, 0, 10); + buffer.Write(data2, 0, 10); + + var readBuffer = new byte[10]; + buffer.ReadFromEnd(10, readBuffer, 0, 10); + Assert.Equal(data2, readBuffer); + + buffer.Dispose(); + } + + #endregion + + #region ReadFromEnd Tests + + [Fact] + public void ReadFromEnd_ValidPosition_ReturnsCorrectData() + { + var buffer = new RingBuffer(100); + var data = new byte[] { 1, 2, 3, 4, 5 }; + buffer.Write(data, 0, 5); + + var readBuffer = new byte[5]; + int read = buffer.ReadFromEnd(5, readBuffer, 0, 5); + + Assert.Equal(5, read); + Assert.Equal(new byte[] { 1, 2, 3, 4, 5 }, readBuffer); + + buffer.Dispose(); + } + + [Fact] + public void ReadFromEnd_WithWrapAround_ReturnsCorrectData() + { + var buffer = new RingBuffer(5); + var data1 = new byte[] { 1, 2, 3, 4, 5 }; + var data2 = new byte[] { 6, 7 }; + + buffer.Write(data1, 0, 5); + buffer.Write(data2, 0, 2); + + var readBuffer = new byte[5]; + buffer.ReadFromEnd(5, readBuffer, 0, 5); + // After writing 5 bytes, writePos=0. After writing 2 more bytes, + // writePos=2, buffer=[6, 7, 3, 4, 5] + Assert.Equal(new byte[] { 3, 4, 5, 6, 7 }, readBuffer); + + buffer.Dispose(); + } + + [Fact] + public void ReadFromEnd_PartialRead_ReturnsAvailableBytes() + { + var buffer = new RingBuffer(100); + var data = new byte[] { 1, 2, 3, 4, 5 }; + buffer.Write(data, 0, 5); + + var readBuffer = new byte[10]; + int read = buffer.ReadFromEnd(3, readBuffer, 0, 10); + + Assert.Equal(3, read); + Assert.Equal(new byte[] { 3, 4, 5, 0, 0, 0, 0, 0, 0, 0 }, readBuffer); + + buffer.Dispose(); + } + + [Fact] + public void ReadFromEnd_FullCapacity_ReadsAllData() + { + var buffer = new RingBuffer(10); + var data = new byte[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 }; + buffer.Write(data, 0, 10); + + var readBuffer = new byte[10]; + buffer.ReadFromEnd(10, readBuffer, 0, 10); + + Assert.Equal(data, readBuffer); + + buffer.Dispose(); + } + + [Fact] + public void ReadFromEnd_ZeroBytesFromEnd_ReturnsZero() + { + var buffer = new RingBuffer(100); + var data = new byte[] { 1, 2, 3, 4, 5 }; + buffer.Write(data, 0, 5); + + var readBuffer = new byte[5]; + int read = buffer.ReadFromEnd(0, readBuffer, 0, 5); + + Assert.Equal(0, read); + + buffer.Dispose(); + } + + [Fact] + public void ReadFromEnd_ExceedsBufferLength_ThrowsArgumentOutOfRangeException() + { + var buffer = new RingBuffer(100); + var data = new byte[] { 1, 2, 3, 4, 5 }; + buffer.Write(data, 0, 5); + + var readBuffer = new byte[10]; + var ex = Assert.Throws(() => + buffer.ReadFromEnd(6, readBuffer, 0, 10) + ); + Assert.Contains("outside buffer range", ex.Message); + + buffer.Dispose(); + } + + [Fact] + public void ReadFromEnd_NegativeBytesFromEnd_ReturnsZero() + { + var buffer = new RingBuffer(100); + var data = new byte[] { 1, 2, 3, 4, 5 }; + buffer.Write(data, 0, 5); + + var readBuffer = new byte[5]; + int read = buffer.ReadFromEnd(-1, readBuffer, 0, 5); + + Assert.Equal(0, read); + + buffer.Dispose(); + } + + [Fact] + public void ReadFromEnd_CountZero_ReturnsZero() + { + var buffer = new RingBuffer(100); + var data = new byte[] { 1, 2, 3, 4, 5 }; + buffer.Write(data, 0, 5); + + var readBuffer = new byte[5]; + int read = buffer.ReadFromEnd(5, readBuffer, 0, 0); + + Assert.Equal(0, read); + + buffer.Dispose(); + } + + #endregion + + #region CanReadFromEnd Tests + + [Fact] + public void CanReadFromEnd_ValidPosition_ReturnsTrue() + { + var buffer = new RingBuffer(100); + var data = new byte[] { 1, 2, 3, 4, 5 }; + buffer.Write(data, 0, 5); + + Assert.True(buffer.CanReadFromEnd(1)); + Assert.True(buffer.CanReadFromEnd(5)); + + buffer.Dispose(); + } + + [Fact] + public void CanReadFromEnd_PositionExceedsLength_ReturnsFalse() + { + var buffer = new RingBuffer(100); + var data = new byte[] { 1, 2, 3, 4, 5 }; + buffer.Write(data, 0, 5); + + Assert.False(buffer.CanReadFromEnd(6)); + Assert.False(buffer.CanReadFromEnd(100)); + + buffer.Dispose(); + } + + [Fact] + public void CanReadFromEnd_NegativePosition_ReturnsFalse() + { + var buffer = new RingBuffer(100); + var data = new byte[] { 1, 2, 3, 4, 5 }; + buffer.Write(data, 0, 5); + + Assert.False(buffer.CanReadFromEnd(-1)); + + buffer.Dispose(); + } + + [Fact] + public void CanReadFromEnd_Zero_ReturnsTrue() + { + var buffer = new RingBuffer(100); + var data = new byte[] { 1, 2, 3, 4, 5 }; + buffer.Write(data, 0, 5); + + Assert.True(buffer.CanReadFromEnd(0)); + + buffer.Dispose(); + } + + #endregion + + #region Properties Tests + + [Fact] + public void Capacity_ReturnsSetCapacity() + { + var buffer = new RingBuffer(42); + Assert.Equal(42, buffer.Capacity); + buffer.Dispose(); + } + + [Fact] + public void Length_InitiallyZero() + { + var buffer = new RingBuffer(100); + Assert.Equal(0, buffer.Length); + buffer.Dispose(); + } + + [Fact] + public void Length_UpdatesAfterWrite() + { + var buffer = new RingBuffer(100); + var data = new byte[] { 1, 2, 3, 4, 5 }; + + Assert.Equal(0, buffer.Length); + buffer.Write(data, 0, 5); + Assert.Equal(5, buffer.Length); + + buffer.Dispose(); + } + + [Fact] + public void Length_CapsAtCapacity() + { + var buffer = new RingBuffer(5); + var data = new byte[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 }; + + buffer.Write(data, 0, 10); + Assert.Equal(5, buffer.Length); + + buffer.Dispose(); + } + + #endregion + + #region Dispose Tests + + [Fact] + public void Dispose_ReturnsArrayToPool() + { + var buffer = new RingBuffer(100); + buffer.Dispose(); + } + + [Fact] + public void Dispose_WriteAfterDispose_ThrowsObjectDisposedException() + { + var buffer = new RingBuffer(100); + buffer.Dispose(); + + var data = new byte[] { 1, 2, 3 }; + Assert.Throws(() => buffer.Write(data, 0, 3)); + } + + [Fact] + public void Dispose_ReadFromEndAfterDispose_ThrowsObjectDisposedException() + { + var buffer = new RingBuffer(100); + var data = new byte[] { 1, 2, 3, 4, 5 }; + buffer.Write(data, 0, 5); + buffer.Dispose(); + + var readBuffer = new byte[5]; + Assert.Throws(() => buffer.ReadFromEnd(5, readBuffer, 0, 5)); + } + + [Fact] + public void Dispose_IdempotentDispose_NoException() + { + var buffer = new RingBuffer(100); + buffer.Dispose(); + buffer.Dispose(); + } + + #endregion + + #region Integration Tests + + [Fact] + public void Integration_SequentialReadsWithMultipleWrites() + { + var buffer = new RingBuffer(10); + + buffer.Write(new byte[] { 1, 2, 3 }, 0, 3); + var readBuffer1 = new byte[3]; + buffer.ReadFromEnd(3, readBuffer1, 0, 3); + Assert.Equal(new byte[] { 1, 2, 3 }, readBuffer1); + + buffer.Write(new byte[] { 4, 5, 6 }, 0, 3); + var readBuffer2 = new byte[6]; + buffer.ReadFromEnd(6, readBuffer2, 0, 6); + Assert.Equal(new byte[] { 1, 2, 3, 4, 5, 6 }, readBuffer2); + + buffer.Dispose(); + } + + [Fact] + public void Integration_ComplexWrapAroundScenario() + { + var buffer = new RingBuffer(5); + + buffer.Write(new byte[] { 1, 2, 3, 4, 5 }, 0, 5); + // Buffer: [1, 2, 3, 4, 5], writePos=0 + buffer.Write(new byte[] { 6, 7, 8 }, 0, 3); + // After writing 3 more bytes: Buffer = [6, 7, 8, 4, 5], writePos=3 + Assert.Equal(5, buffer.Length); + + var readBuffer = new byte[5]; + buffer.ReadFromEnd(5, readBuffer, 0, 5); + Assert.Equal(new byte[] { 4, 5, 6, 7, 8 }, readBuffer); + + buffer.Write(new byte[] { 9 }, 0, 1); + // After writing 1 more byte: Buffer = [6, 7, 8, 9, 5], writePos=4 + buffer.ReadFromEnd(5, readBuffer, 0, 5); + Assert.Equal(new byte[] { 5, 6, 7, 8, 9 }, readBuffer); + + buffer.Dispose(); + } + + [Fact] + public void Integration_PartialReadsMultipleTimes() + { + var buffer = new RingBuffer(100); + var data = new byte[20]; + for (byte i = 0; i < 20; i++) + { + data[i] = i; + } + buffer.Write(data, 0, 20); + + var readBuffer1 = new byte[5]; + buffer.ReadFromEnd(10, readBuffer1, 0, 5); + Assert.Equal(new byte[] { 10, 11, 12, 13, 14 }, readBuffer1); + + var readBuffer2 = new byte[10]; + buffer.ReadFromEnd(20, readBuffer2, 0, 10); + Assert.Equal(new byte[] { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }, readBuffer2); + + buffer.Dispose(); + } + + #endregion +} diff --git a/tests/SharpCompress.Test/Streams/SeekableSharpCompressStreamAsyncTest.cs b/tests/SharpCompress.Test/Streams/SeekableSharpCompressStreamAsyncTest.cs new file mode 100644 index 00000000..d39b82ad --- /dev/null +++ b/tests/SharpCompress.Test/Streams/SeekableSharpCompressStreamAsyncTest.cs @@ -0,0 +1,198 @@ +using System; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.IO; +using Xunit; + +namespace SharpCompress.Test.Streams; + +public class SeekableSharpCompressStreamAsyncTest +{ + [Fact] + public async Task ReadAsync_Buffers() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = new SeekableSharpCompressStream(ms); + var buffer = new byte[5]; + int bytesRead = await stream.ReadAsync(buffer, 0, buffer.Length).ConfigureAwait(false); + Assert.Equal(5, bytesRead); + Assert.Equal(new byte[] { 1, 2, 3, 4, 5 }, buffer); + } + + [Fact] + public async Task ReadAsync_WithCancellation() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = new SeekableSharpCompressStream(ms); + var buffer = new byte[5]; + var cts = new CancellationTokenSource(); + int bytesRead = await stream + .ReadAsync(buffer, 0, buffer.Length, cts.Token) + .ConfigureAwait(false); + Assert.Equal(5, bytesRead); + Assert.Equal(new byte[] { 1, 2, 3, 4, 5 }, buffer); + } + + [Fact] + public async Task ReadAsync_PartialRead() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = new SeekableSharpCompressStream(ms); + var buffer = new byte[10]; + int bytesRead = await stream.ReadAsync(buffer, 0, 10).ConfigureAwait(false); + Assert.Equal(5, bytesRead); + Assert.Equal(new byte[] { 1, 2, 3, 4, 5 }, buffer.Take(5).ToArray()); + } + + [Fact] + public async Task WriteAsync_Buffers() + { + var ms = new MemoryStream(); + var stream = new SeekableSharpCompressStream(ms); + var data = new byte[] { 1, 2, 3, 4, 5 }; + await stream.WriteAsync(data, 0, data.Length).ConfigureAwait(false); + Assert.Equal(data, ms.ToArray()); + } + + [Fact] + public async Task WriteAsync_WithCancellation() + { + var ms = new MemoryStream(); + var stream = new SeekableSharpCompressStream(ms); + var data = new byte[] { 1, 2, 3, 4, 5 }; + var cts = new CancellationTokenSource(); + await stream.WriteAsync(data, 0, data.Length, cts.Token).ConfigureAwait(false); + Assert.Equal(data, ms.ToArray()); + } + + [Fact] + public async Task FlushAsync_DelegatesToUnderlyingStream() + { + var ms = new MemoryStream(); + var stream = new SeekableSharpCompressStream(ms); + var data = new byte[] { 1, 2, 3, 4, 5 }; + await stream.WriteAsync(data, 0, data.Length).ConfigureAwait(false); + await stream.FlushAsync().ConfigureAwait(false); + Assert.Equal(5, ms.Length); + } + + [Fact] + public async Task CopyToAsync_CopiesAllData() + { + var sourceMs = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = new SeekableSharpCompressStream(sourceMs); + var destinationMs = new MemoryStream(); + await stream.CopyToAsync(destinationMs, 4096).ConfigureAwait(false); + Assert.Equal(new byte[] { 1, 2, 3, 4, 5 }, destinationMs.ToArray()); + } + + [Fact] + public async Task ReadAsyncAndSeek_MultipleOperations() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 }); + var stream = new SeekableSharpCompressStream(ms); + + var buffer = new byte[3]; + await stream.ReadAsync(buffer, 0, 3).ConfigureAwait(false); + Assert.Equal(new byte[] { 1, 2, 3 }, buffer); + Assert.Equal(3, stream.Position); + + stream.Seek(7, SeekOrigin.Begin); + Assert.Equal(7, stream.Position); + + Array.Clear(buffer, 0, buffer.Length); + await stream.ReadAsync(buffer, 0, 2).ConfigureAwait(false); + Assert.Equal(new byte[] { 8, 9, 0 }, buffer); + Assert.Equal(9, stream.Position); + } + + [Fact] + public async Task WriteAsyncAndReadAsync_WrittenDataIsReadable() + { + var ms = new MemoryStream(); + var stream = new SeekableSharpCompressStream(ms); + + var writeData = new byte[] { 1, 2, 3, 4, 5 }; + await stream.WriteAsync(writeData, 0, writeData.Length).ConfigureAwait(false); + + stream.Position = 0; + var readBuffer = new byte[5]; + await stream.ReadAsync(readBuffer, 0, 5).ConfigureAwait(false); + Assert.Equal(writeData, readBuffer); + } +} + +#if !LEGACY_DOTNET +public partial class SeekableSharpCompressStreamMemoryAsyncTest +{ + [Fact] + public async ValueTask ReadAsync_Memory() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = new SeekableSharpCompressStream(ms); + var buffer = new byte[5]; + int bytesRead = await stream.ReadAsync(buffer).ConfigureAwait(false); + Assert.Equal(5, bytesRead); + Assert.Equal(new byte[] { 1, 2, 3, 4, 5 }, buffer); + } + + [Fact] + public async ValueTask ReadAsync_Memory_WithCancellation() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = new SeekableSharpCompressStream(ms); + var buffer = new byte[5]; + var cts = new CancellationTokenSource(); + int bytesRead = await stream.ReadAsync(buffer, cts.Token).ConfigureAwait(false); + Assert.Equal(5, bytesRead); + Assert.Equal(new byte[] { 1, 2, 3, 4, 5 }, buffer); + } + + [Fact] + public async ValueTask WriteAsync_Memory() + { + var ms = new MemoryStream(); + var stream = new SeekableSharpCompressStream(ms); + var data = new byte[] { 1, 2, 3, 4, 5 }; + await stream.WriteAsync(data).ConfigureAwait(false); + Assert.Equal(data, ms.ToArray()); + } + + [Fact] + public async ValueTask WriteAsync_Memory_WithCancellation() + { + var ms = new MemoryStream(); + var stream = new SeekableSharpCompressStream(ms); + var data = new byte[] { 1, 2, 3, 4, 5 }; + var cts = new CancellationTokenSource(); + await stream.WriteAsync(data, cts.Token).ConfigureAwait(false); + Assert.Equal(data, ms.ToArray()); + } + + [Fact] + public async ValueTask ReadMemoryAndWriteMemory_MemoryOperations() + { + var ms = new MemoryStream(); + var stream = new SeekableSharpCompressStream(ms); + + var writeData = new byte[] { 1, 2, 3, 4, 5 }; + await stream.WriteAsync(writeData).ConfigureAwait(false); + + stream.Position = 0; + var readBuffer = new byte[5]; + await stream.ReadAsync(readBuffer).ConfigureAwait(false); + Assert.Equal(writeData, readBuffer); + } + + [Fact] + public async ValueTask DisposeAsync_DisposesUnderlyingStream() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = new SeekableSharpCompressStream(ms); + await stream.DisposeAsync().ConfigureAwait(false); + Assert.Throws(() => ms.Read(new byte[1], 0, 1)); + } +} +#endif diff --git a/tests/SharpCompress.Test/Streams/SeekableSharpCompressStreamTest.cs b/tests/SharpCompress.Test/Streams/SeekableSharpCompressStreamTest.cs new file mode 100644 index 00000000..0071d420 --- /dev/null +++ b/tests/SharpCompress.Test/Streams/SeekableSharpCompressStreamTest.cs @@ -0,0 +1,284 @@ +using System; +using System.IO; +using System.Linq; +using SharpCompress.IO; +using SharpCompress.Test.Mocks; +using Xunit; + +namespace SharpCompress.Test.Streams; + +public class SeekableSharpCompressStreamTest +{ + [Fact] + public void Constructor_ThrowsOnNullStream() + { + Assert.Throws(() => new SeekableSharpCompressStream(null!)); + } + + [Fact] + public void Constructor_ThrowsOnNonSeekableStream() + { + var nonSeekable = new ForwardOnlyStream(new MemoryStream()); + Assert.Throws(() => new SeekableSharpCompressStream(nonSeekable)); + } + + [Fact] + public void Constructor_AcceptsSeekableStream() + { + var ms = new MemoryStream(); + var stream = new SeekableSharpCompressStream(ms); + Assert.NotNull(stream); + } + + [Fact] + public void CanRead_DelegatesToUnderlyingStream() + { + var ms = new MemoryStream(); + var stream = new SeekableSharpCompressStream(ms); + Assert.Equal(ms.CanRead, stream.CanRead); + } + + [Fact] + public void CanSeek_DelegatesToUnderlyingStream() + { + var ms = new MemoryStream(); + var stream = new SeekableSharpCompressStream(ms); + Assert.Equal(ms.CanSeek, stream.CanSeek); + } + + [Fact] + public void CanWrite_DelegatesToUnderlyingStream() + { + var ms = new MemoryStream(); + var stream = new SeekableSharpCompressStream(ms); + Assert.Equal(ms.CanWrite, stream.CanWrite); + } + + [Fact] + public void Length_DelegatesToUnderlyingStream() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = new SeekableSharpCompressStream(ms); + Assert.Equal(5, stream.Length); + } + + [Fact] + public void Position_Getter_DelegatesToUnderlyingStream() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + ms.Position = 2; + var stream = new SeekableSharpCompressStream(ms); + Assert.Equal(2, stream.Position); + } + + [Fact] + public void Position_Setter_DelegatesToUnderlyingStream() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = new SeekableSharpCompressStream(ms); + stream.Position = 3; + Assert.Equal(3, ms.Position); + Assert.Equal(3, stream.Position); + } + + [Fact] + public void IsRecording_AlwaysFalse() + { + var ms = new MemoryStream(); + var stream = new SeekableSharpCompressStream(ms); + Assert.False(stream.IsRecording); + } + + [Fact] + public void Read_Buffers() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = new SeekableSharpCompressStream(ms); + var buffer = new byte[5]; + int bytesRead = stream.Read(buffer, 0, buffer.Length); + Assert.Equal(5, bytesRead); + Assert.Equal(new byte[] { 1, 2, 3, 4, 5 }, buffer); + } + + [Fact] + public void Seek_DelegatesToUnderlyingStream() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = new SeekableSharpCompressStream(ms); + long result = stream.Seek(3, SeekOrigin.Begin); + Assert.Equal(3, result); + Assert.Equal(3, ms.Position); + } + + [Fact] + public void SetLength_DelegatesToUnderlyingStream() + { + var ms = new MemoryStream(); + var stream = new SeekableSharpCompressStream(ms); + stream.SetLength(20); + Assert.Equal(20, stream.Length); + Assert.Equal(20, ms.Length); + } + + [Fact] + public void Write_DelegatesToUnderlyingStream() + { + var ms = new MemoryStream(); + var stream = new SeekableSharpCompressStream(ms); + var data = new byte[] { 1, 2, 3, 4, 5 }; + stream.Write(data, 0, data.Length); + Assert.Equal(data, ms.ToArray()); + } + + [Fact] + public void Rewind_IsNoOp() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = new SeekableSharpCompressStream(ms); + stream.Rewind(); + Assert.Equal(0, stream.Position); + ms.Position = 2; + stream.Rewind(true); + Assert.Equal(2, stream.Position); + } + + [Fact] + public void StartRecording_IsNotNoOp() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = new SeekableSharpCompressStream(ms); + stream.StartRecording(); + Assert.True(stream.IsRecording); + } + + [Fact] + public void StopRecording_IsNoOp() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = new SeekableSharpCompressStream(ms); + stream.StopRecording(); + Assert.False(stream.IsRecording); + } + + [Fact] + public void Dispose_DisposesUnderlyingStream() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = new SeekableSharpCompressStream(ms); + stream.Dispose(); + Assert.Throws(() => ms.Read(new byte[1], 0, 1)); + } + + [Fact] + public void ReadAndSeek_MultipleOperations() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 }); + var stream = new SeekableSharpCompressStream(ms); + + var buffer = new byte[3]; + stream.Read(buffer, 0, 3); + Assert.Equal(new byte[] { 1, 2, 3 }, buffer); + Assert.Equal(3, stream.Position); + + stream.Seek(7, SeekOrigin.Begin); + Assert.Equal(7, stream.Position); + + Array.Clear(buffer, 0, buffer.Length); + stream.Read(buffer, 0, 2); + Assert.Equal(new byte[] { 8, 9, 0 }, buffer); + Assert.Equal(9, stream.Position); + } + + [Fact] + public void SeekWithDifferentOrigins() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 }); + var stream = new SeekableSharpCompressStream(ms); + + stream.Seek(3, SeekOrigin.Begin); + Assert.Equal(3, stream.Position); + + stream.Seek(2, SeekOrigin.Current); + Assert.Equal(5, stream.Position); + + stream.Seek(-3, SeekOrigin.End); + Assert.Equal(7, stream.Position); + } + + [Fact] + public void WriteAndRead_WrittenDataIsReadable() + { + var ms = new MemoryStream(); + var stream = new SeekableSharpCompressStream(ms); + + var writeData = new byte[] { 1, 2, 3, 4, 5 }; + stream.Write(writeData, 0, writeData.Length); + + stream.Position = 0; + var readBuffer = new byte[5]; + stream.Read(readBuffer, 0, 5); + Assert.Equal(writeData, readBuffer); + } + + [Fact] + public void RecordingOperationsDoAffectStream() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = new SeekableSharpCompressStream(ms); + + stream.StartRecording(); + var buffer = new byte[3]; + stream.Read(buffer, 0, 3); + Assert.Equal(new byte[] { 1, 2, 3 }, buffer); + Assert.Equal(3, stream.Position); + + stream.Rewind(true); + Assert.Equal(0, stream.Position); + + var buffer2 = new byte[2]; + stream.Read(buffer2, 0, 2); + Assert.Equal(new byte[] { 1, 2 }, buffer2); + Assert.Equal(2, stream.Position); + } +} + +#if !LEGACY_DOTNET +public partial class SeekableRewindableSpanTest +{ + [Fact] + public void Read_Span() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = new SeekableSharpCompressStream(ms); + var buffer = new byte[5]; + int bytesRead = stream.Read(buffer); + Assert.Equal(5, bytesRead); + Assert.Equal(new byte[] { 1, 2, 3, 4, 5 }, buffer); + } + + [Fact] + public void Write_Span() + { + var ms = new MemoryStream(); + var stream = new SeekableSharpCompressStream(ms); + var data = new byte[] { 1, 2, 3, 4, 5 }; + stream.Write(data); + Assert.Equal(data, ms.ToArray()); + } + + [Fact] + public void ReadAndWrite_SpanOperations() + { + var ms = new MemoryStream(); + var stream = new SeekableSharpCompressStream(ms); + + var writeData = new byte[] { 1, 2, 3, 4, 5 }; + stream.Write(writeData); + + stream.Position = 0; + var readBuffer = new byte[5]; + stream.Read(readBuffer); + Assert.Equal(writeData, readBuffer); + } +} +#endif diff --git a/tests/SharpCompress.Test/Streams/SharpCompressStreamEdgeAsyncTest.cs b/tests/SharpCompress.Test/Streams/SharpCompressStreamEdgeAsyncTest.cs new file mode 100644 index 00000000..1c3b7d18 --- /dev/null +++ b/tests/SharpCompress.Test/Streams/SharpCompressStreamEdgeAsyncTest.cs @@ -0,0 +1,120 @@ +using System; +using System.IO; +using System.Threading.Tasks; +using SharpCompress.IO; +using SharpCompress.Test.Mocks; +using Xunit; + +namespace SharpCompress.Test.Streams; + +public class SharpCompressStreamEdgeAsyncTest +{ +#if !LEGACY_DOTNET + + [Fact] + public async ValueTask DisposeAsync_WithLeaveStreamOpenTrue_DoesNotDisposeUnderlying() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + await stream.DisposeAsync().ConfigureAwait(false); + Assert.Equal(0, ms.Position); + Assert.True(ms.CanRead); + } + + [Fact] + public async ValueTask DisposeAsync_WithLeaveStreamOpenFalse_DisposesUnderlying() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new ForwardOnlyStream(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + await stream.DisposeAsync().ConfigureAwait(false); + Assert.Throws(() => ms.Read(new byte[1], 0, 1)); + } +#endif + + [Fact] + public async ValueTask ReadAsync_ZeroCount_ReturnsZero() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + var buffer = new byte[10]; + int bytesRead = await stream.ReadAsync(buffer, 0, 0).ConfigureAwait(false); + Assert.Equal(0, bytesRead); + } + + [Fact] + public async ValueTask ReadAsync_AtEndOfStream_ReturnsZero() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + var buffer = new byte[10]; + await stream.ReadAsync(buffer, 0, buffer.Length).ConfigureAwait(false); + int bytesRead = await stream.ReadAsync(buffer, 0, buffer.Length).ConfigureAwait(false); + Assert.Equal(0, bytesRead); + } + +#if !LEGACY_DOTNET + [Fact] + public async ValueTask ReadAsyncMemory_ZeroCount_ReturnsZero() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + var buffer = new byte[10]; + int bytesRead = await stream.ReadAsync(buffer.AsMemory(0, 0)).ConfigureAwait(false); + Assert.Equal(0, bytesRead); + } + + [Fact] + public async ValueTask ReadAsyncMemory_AtEndOfStream_ReturnsZero() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + var buffer = new byte[10]; + await stream.ReadAsync(buffer.AsMemory()).ConfigureAwait(false); + int bytesRead = await stream.ReadAsync(buffer.AsMemory()).ConfigureAwait(false); + Assert.Equal(0, bytesRead); + } +#endif + + [Fact] + public async ValueTask CopyToAsync_DelegatesToUnderlyingStream() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + var destination = new MemoryStream(); + await stream.CopyToAsync(destination).ConfigureAwait(false); + Assert.Equal(new byte[] { 1, 2, 3, 4, 5 }, destination.ToArray()); + } + + [Fact] + public async ValueTask CopyToAsync_WithBufferSize_WorksCorrectly() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + var destination = new MemoryStream(); + await stream.CopyToAsync(destination, 2).ConfigureAwait(false); + Assert.Equal(new byte[] { 1, 2, 3, 4, 5 }, destination.ToArray()); + } + +#if !LEGACY_DOTNET + [Fact] + public async ValueTask WriteAsyncMemory_DelegatesToUnderlying() + { + var ms = new MemoryStream(); + var stream = SharpCompressStream.CreateNonDisposing(ms); + var data = new byte[] { 1, 2, 3, 4, 5 }; + await stream.WriteAsync(data.AsMemory()).ConfigureAwait(false); + Assert.Equal(data, ms.ToArray()); + } + + [Fact] + public async ValueTask FlushAsyncMemory_DelegatesToUnderlying() + { + var ms = new MemoryStream(); + var stream = SharpCompressStream.CreateNonDisposing(ms); + await stream.WriteAsync(new byte[] { 1, 2, 3 }).ConfigureAwait(false); + await stream.FlushAsync().ConfigureAwait(false); + Assert.Equal(3, ms.Length); + } +#endif +} diff --git a/tests/SharpCompress.Test/Streams/SharpCompressStreamEdgeTest.cs b/tests/SharpCompress.Test/Streams/SharpCompressStreamEdgeTest.cs new file mode 100644 index 00000000..fa53e8e3 --- /dev/null +++ b/tests/SharpCompress.Test/Streams/SharpCompressStreamEdgeTest.cs @@ -0,0 +1,100 @@ +using System; +using System.IO; +using SharpCompress.Common; +using SharpCompress.IO; +using SharpCompress.Test.Mocks; +using Xunit; + +namespace SharpCompress.Test.Streams; + +public class SharpCompressStreamEdgeTest +{ + [Fact] + public void Dispose_WithLeaveStreamOpenTrue_DoesNotDisposeUnderlying() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + stream.Dispose(); + Assert.Equal(0, ms.Position); + Assert.True(ms.CanRead); + } + + [Fact] + public void Dispose_WithLeaveStreamOpenFalse_DisposesUnderlying() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new ForwardOnlyStream(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + stream.Dispose(); + Assert.Throws(() => ms.Read(new byte[1], 0, 1)); + } + + [Fact] + public void Dispose_WithThrowOnDisposeTrue_ThrowsInvalidOperation() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + stream.ThrowOnDispose = true; + Assert.Throws(() => stream.Dispose()); + } + + [Fact] + public void Read_ZeroCount_ReturnsZero() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + var buffer = new byte[10]; + int bytesRead = stream.Read(buffer, 0, 0); + Assert.Equal(0, bytesRead); + } + + [Fact] + public void Read_AtEndOfStream_ReturnsZero() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + var buffer = new byte[10]; + int bytesRead = stream.Read(buffer, 0, buffer.Length); + Assert.Equal(5, bytesRead); + bytesRead = stream.Read(buffer, 0, buffer.Length); + Assert.Equal(0, bytesRead); + } + + [Fact] + public void Position_InitialValue_IsZero() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new ForwardOnlyStream(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + Assert.Equal(0, stream.Position); + } + + [Fact] + public void CanRead_AlwaysReturnsTrue() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + Assert.True(stream.CanRead); + } + + [Fact] + public void CanSeek_PassthroughMode_DelegatesToUnderlying() + { + var seekableMs = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new ForwardOnlyStream(new MemoryStream(new byte[] { 1, 2, 3, 4, 5 })); + + var seekableStream = SharpCompressStream.CreateNonDisposing(seekableMs); + var nonSeekableStream = SharpCompressStream.CreateNonDisposing(nonSeekableMs); + + Assert.True(seekableStream.CanSeek); + Assert.False(nonSeekableStream.CanSeek); + } + + [Fact] + public void BaseStream_ReturnsUnderlyingStream() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + Assert.Same(ms, stream.BaseStream()); + } +} diff --git a/tests/SharpCompress.Test/Streams/SharpCompressStreamErrorAsyncTest.cs b/tests/SharpCompress.Test/Streams/SharpCompressStreamErrorAsyncTest.cs new file mode 100644 index 00000000..cf90a566 --- /dev/null +++ b/tests/SharpCompress.Test/Streams/SharpCompressStreamErrorAsyncTest.cs @@ -0,0 +1,150 @@ +using System; +using System.IO; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.IO; +using Xunit; + +namespace SharpCompress.Test.Streams; + +public class SharpCompressStreamErrorAsyncTest +{ + private class NonSeekableStreamWrapper : Stream + { + private readonly Stream _baseStream; + + public NonSeekableStreamWrapper(Stream baseStream) => _baseStream = baseStream; + + public override bool CanRead => _baseStream.CanRead; + public override bool CanSeek => false; + public override bool CanWrite => _baseStream.CanWrite; + public override long Length => _baseStream.Length; + public override long Position + { + get => throw new NotSupportedException(); + set => throw new NotSupportedException(); + } + + public override void Flush() => _baseStream.Flush(); + + public override int Read(byte[] buffer, int offset, int count) => + _baseStream.Read(buffer, offset, count); + + public override long Seek(long offset, SeekOrigin origin) => + throw new NotSupportedException(); + + public override void SetLength(long value) => _baseStream.SetLength(value); + + public override void Write(byte[] buffer, int offset, int count) => + _baseStream.Write(buffer, offset, count); + + protected override void Dispose(bool disposing) + { + if (disposing) + { + _baseStream.Dispose(); + } + + base.Dispose(disposing); + } + } + +#if !LEGACY_DOTNET + [Fact] + public async ValueTask DisposeAsync_WithThrowOnDisposeTrue_ThrowsInvalidOperation() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + stream.ThrowOnDispose = true; + await Assert + .ThrowsAsync(async () => + await stream.DisposeAsync().ConfigureAwait(false) + ) + .ConfigureAwait(false); + } +#endif + + [Fact] + public async ValueTask CreateNonDisposing_ReadAsync_ZeroCount_ReturnsZero() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + var buffer = new byte[10]; + int bytesRead = await stream.ReadAsync(buffer, 0, 0).ConfigureAwait(false); + Assert.Equal(0, bytesRead); + } + + [Fact] + public async ValueTask CreateNonDisposing_ReadAsync_AtEndOfStream_ReturnsZero() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + var buffer = new byte[10]; + await stream.ReadAsync(buffer, 0, buffer.Length).ConfigureAwait(false); + int bytesRead = await stream.ReadAsync(buffer, 0, buffer.Length).ConfigureAwait(false); + Assert.Equal(0, bytesRead); + } + + [Fact] + public async ValueTask Create_AsyncReadWithRecording_WorksCorrectly() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + stream.StartRecording(); + var buffer = new byte[4]; + await stream.ReadAsync(buffer, 0, 4).ConfigureAwait(false); + Assert.Equal(4, stream.Position); + } + + [Fact] + public async ValueTask Create_AsyncReadWithBufferOverflow_ThrowsInvalidOperation() + { + var ms = new MemoryStream(new byte[256]); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 64); + stream.StartRecording(); + var buffer = new byte[32]; + for (int i = 0; i < 3; i++) + { + await stream.ReadExactAsync(buffer, 0, 32).ConfigureAwait(false); + } + Assert.Throws(() => stream.Rewind()); + } + + [Fact] + public async ValueTask FlushAsync_NonPassthrough_ThrowsNotSupported() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + await Assert + .ThrowsAsync(async () => + await stream.FlushAsync().ConfigureAwait(false) + ) + .ConfigureAwait(false); + } + + [Fact] + public async ValueTask WriteAsync_NonPassthrough_ThrowsNotSupported() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + await Assert + .ThrowsAsync(async () => + await stream.WriteAsync(new byte[] { 1 }, 0, 1).ConfigureAwait(false) + ) + .ConfigureAwait(false); + } + + [Fact] + public async ValueTask CopyToAsync_WithPassthrough_CopiesAllData() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + var destination = new MemoryStream(); + await stream.CopyToAsync(destination).ConfigureAwait(false); + Assert.Equal(new byte[] { 1, 2, 3, 4, 5 }, destination.ToArray()); + } +} diff --git a/tests/SharpCompress.Test/Streams/SharpCompressStreamErrorTest.cs b/tests/SharpCompress.Test/Streams/SharpCompressStreamErrorTest.cs new file mode 100644 index 00000000..7b844ff4 --- /dev/null +++ b/tests/SharpCompress.Test/Streams/SharpCompressStreamErrorTest.cs @@ -0,0 +1,188 @@ +using System; +using System.IO; +using SharpCompress.Common; +using SharpCompress.IO; +using SharpCompress.Test.Mocks; +using Xunit; + +namespace SharpCompress.Test.Streams; + +public class SharpCompressStreamErrorTest +{ + private class NonSeekableStreamWrapper : Stream + { + private readonly Stream _baseStream; + + public NonSeekableStreamWrapper(Stream baseStream) + { + _baseStream = baseStream; + } + + public override bool CanRead => _baseStream.CanRead; + + public override bool CanSeek => false; + + public override bool CanWrite => _baseStream.CanWrite; + + public override long Length => _baseStream.Length; + + public override long Position + { + get => throw new NotSupportedException(); + set => throw new NotSupportedException(); + } + + public override void Flush() => _baseStream.Flush(); + + public override int Read(byte[] buffer, int offset, int count) => + _baseStream.Read(buffer, offset, count); + + public override long Seek(long offset, SeekOrigin origin) => + throw new NotSupportedException(); + + public override void SetLength(long value) => _baseStream.SetLength(value); + + public override void Write(byte[] buffer, int offset, int count) => + _baseStream.Write(buffer, offset, count); + + protected override void Dispose(bool disposing) + { + if (disposing) + { + _baseStream.Dispose(); + } + base.Dispose(disposing); + } + } + + [Fact] + public void Rewind_WithoutStartRecording_ThrowsInvalidOperation() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + Assert.Throws(() => stream.Rewind()); + } + + [Fact] + public void Rewind_PassthroughMode_ThrowsInvalidOperation() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + Assert.Throws(() => stream.Rewind()); + } + + [Fact] + public void StartRecording_Twice_ThrowsInvalidOperation() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + stream.StartRecording(); + Assert.Throws(() => stream.StartRecording()); + } + + [Fact] + public void StartRecording_PassthroughMode_ThrowsInvalidOperation() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + Assert.Throws(() => stream.StartRecording()); + } + + [Fact] + public void StopRecording_WithoutRecording_ThrowsInvalidOperation() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + Assert.Throws(() => stream.StopRecording()); + } + + [Fact] + public void StopRecording_PassthroughMode_ThrowsInvalidOperation() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + Assert.Throws(() => stream.StopRecording()); + } + + [Fact] + public void StopRecording_Twice_ThrowsInvalidOperation() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + stream.StartRecording(); + stream.Read(new byte[4], 0, 4); + stream.StopRecording(); + Assert.Throws(() => stream.StopRecording()); + } + + [Fact] + public void Seek_BeyondRecordedRange_ThrowsNotSupported() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + stream.StartRecording(); + stream.Read(new byte[4], 0, 4); + Assert.Throws(() => stream.Position = 100); + } + + [Fact] + public void Seek_FromEnd_ThrowsNotSupported() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + Assert.Throws(() => stream.Seek(-1, SeekOrigin.End)); + } + + [Fact] + public void Position_SetNegative_Throws() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + stream.StartRecording(); + stream.Read(new byte[4], 0, 4); + Assert.Throws(() => stream.Position = -1); + } + + [Fact] + public void Flush_NonPassthrough_ThrowsNotSupported() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + Assert.Throws(() => stream.Flush()); + } + + [Fact] + public void Write_NonPassthrough_ThrowsNotSupported() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + Assert.Throws(() => stream.Write(new byte[] { 1 }, 0, 1)); + } + + [Fact] + public void SetLength_NonPassthrough_ThrowsNotSupported() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + Assert.Throws(() => stream.SetLength(100)); + } + + [Fact] + public void Length_NonPassthroughWithoutBuffer_ThrowsNotSupported() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new ForwardOnlyStream(ms); + var stream = new SharpCompressStream(nonSeekableMs); + Assert.Throws(() => stream.Length); + } +} diff --git a/tests/SharpCompress.Test/Streams/SharpCompressStreamFactoryAsyncTest.cs b/tests/SharpCompress.Test/Streams/SharpCompressStreamFactoryAsyncTest.cs new file mode 100644 index 00000000..f5ba1909 --- /dev/null +++ b/tests/SharpCompress.Test/Streams/SharpCompressStreamFactoryAsyncTest.cs @@ -0,0 +1,56 @@ +using System; +using System.IO; +using System.Threading.Tasks; +using SharpCompress.IO; +using SharpCompress.Test.Mocks; +using Xunit; + +namespace SharpCompress.Test.Streams; + +public class SharpCompressStreamFactoryAsyncTest +{ + [Fact] + public async ValueTask Create_AsyncReadWithSeekableStream_WorksCorrectly() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.Create(ms); + var buffer = new byte[5]; + int bytesRead = await stream.ReadAsync(buffer, 0, buffer.Length).ConfigureAwait(false); + Assert.Equal(5, bytesRead); + Assert.Equal(new byte[] { 1, 2, 3, 4, 5 }, buffer); + } + + [Fact] + public async ValueTask Create_AsyncReadWithNonSeekableStream_BufferedCorrectly() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new ForwardOnlyStream(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + var buffer = new byte[5]; + int bytesRead = await stream.ReadAsync(buffer, 0, buffer.Length).ConfigureAwait(false); + Assert.Equal(5, bytesRead); + Assert.Equal(new byte[] { 1, 2, 3, 4, 5 }, buffer); + } + + [Fact] + public async ValueTask Create_WithBufferAsync_WorksCorrectly() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new ForwardOnlyStream(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + stream.StartRecording(); + var buffer = new byte[4]; + await stream.ReadAsync(buffer, 0, 4).ConfigureAwait(false); + Assert.Equal(4, stream.Position); + } + + [Fact] + public async ValueTask Create_AsyncReadSeekable_PositionUpdatesCorrectly() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5, 6, 7, 8 }); + var stream = SharpCompressStream.Create(ms); + var buffer = new byte[4]; + await stream.ReadAsync(buffer, 0, 4).ConfigureAwait(false); + Assert.Equal(4, stream.Position); + } +} diff --git a/tests/SharpCompress.Test/Streams/SharpCompressStreamFactoryTest.cs b/tests/SharpCompress.Test/Streams/SharpCompressStreamFactoryTest.cs new file mode 100644 index 00000000..4a5d74a9 --- /dev/null +++ b/tests/SharpCompress.Test/Streams/SharpCompressStreamFactoryTest.cs @@ -0,0 +1,148 @@ +using System; +using System.IO; +using SharpCompress.IO; +using SharpCompress.Test.Mocks; +using Xunit; + +namespace SharpCompress.Test.Streams; + +public class SharpCompressStreamFactoryTest +{ + private class IStreamStackMock : Stream, IStreamStack + { + private readonly Stream _baseStream; + + public IStreamStackMock(Stream baseStream) + { + _baseStream = baseStream; + } + + public Stream BaseStream() => _baseStream; + + public override bool CanRead => _baseStream.CanRead; + + public override bool CanSeek => _baseStream.CanSeek; + + public override bool CanWrite => _baseStream.CanWrite; + + public override long Length => _baseStream.Length; + + public override long Position + { + get => _baseStream.Position; + set => _baseStream.Position = value; + } + + public override void Flush() => _baseStream.Flush(); + + public override int Read(byte[] buffer, int offset, int count) => + _baseStream.Read(buffer, offset, count); + + public override long Seek(long offset, SeekOrigin origin) => + _baseStream.Seek(offset, origin); + + public override void SetLength(long value) => _baseStream.SetLength(value); + + public override void Write(byte[] buffer, int offset, int count) => + _baseStream.Write(buffer, offset, count); + } + + [Fact] + public void Create_WithSeekableStream_ReturnsSeekableSharpCompressStream() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.Create(ms); + Assert.IsType(stream); + } + + [Fact] + public void Create_WithNonSeekableStream_ReturnsSharpCompressStreamWithBuffer() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new ForwardOnlyStream(ms); + var stream = SharpCompressStream.Create(nonSeekableMs); + Assert.IsType(stream); + Assert.NotNull(stream); + } + + [Fact] + public void Create_WithSharpCompressStreamPassthrough_UnwrapsAndCreatesNew() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var passthroughStream = SharpCompressStream.CreateNonDisposing(ms); + var stream = SharpCompressStream.Create(passthroughStream); + Assert.NotSame(passthroughStream, stream); + Assert.IsType(stream); + } + + [Fact] + public void Create_WithSharpCompressStreamNonPassthrough_ReturnsSameInstance() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new ForwardOnlyStream(ms); + var sharpStream = SharpCompressStream.Create(nonSeekableMs, 128); + var stream = SharpCompressStream.Create(sharpStream); + Assert.Same(sharpStream, stream); + } + + [Fact] + public void Create_WithIStreamStack_UnwrapsSharpCompressStream() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var sharpStream = SharpCompressStream.CreateNonDisposing(ms); + var wrappedStream = new IStreamStackMock(sharpStream); + var stream = SharpCompressStream.Create(wrappedStream); + Assert.Same(sharpStream, stream); + } + + [Fact] + public void Create_WithBufferSize_UsesCustomBufferSize() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new ForwardOnlyStream(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + Assert.NotNull(stream); + stream.StartRecording(); + var buffer = new byte[4]; + stream.Read(buffer, 0, 4); + Assert.Equal(4, stream.Position); + } + + [Fact] + public void Create_WithLeaveStreamOpenTrue_PreservesSetting() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var passthroughStream = SharpCompressStream.CreateNonDisposing(ms); + var stream = SharpCompressStream.Create(passthroughStream); + Assert.True(stream.LeaveStreamOpen); + } + + [Fact] + public void Create_WithSeekablePassthroughStream_CreatesSeekableWrapper() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var passthroughStream = SharpCompressStream.CreateNonDisposing(ms); + var stream = SharpCompressStream.Create(passthroughStream); + Assert.IsType(stream); + } + + [Fact] + public void Create_WithIStreamStack_ReturnsUnderlyingSharpCompressStream() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var sharpStream = SharpCompressStream.Create(ms); + var wrappedStream = new IStreamStackMock(sharpStream); + var result = SharpCompressStream.Create(wrappedStream); + Assert.Same(sharpStream, result); + } + + [Fact] + public void Create_WithNonSeekablePassthroughStream_CreatesBufferedWrapper() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new ForwardOnlyStream(ms); + var passthroughStream = SharpCompressStream.CreateNonDisposing(nonSeekableMs); + var stream = SharpCompressStream.Create(passthroughStream); + Assert.IsType(stream); + } +} diff --git a/tests/SharpCompress.Test/Streams/SharpCompressStreamPassthroughAsyncTest.cs b/tests/SharpCompress.Test/Streams/SharpCompressStreamPassthroughAsyncTest.cs new file mode 100644 index 00000000..521bac6a --- /dev/null +++ b/tests/SharpCompress.Test/Streams/SharpCompressStreamPassthroughAsyncTest.cs @@ -0,0 +1,111 @@ +using System; +using System.IO; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.IO; +using SharpCompress.Test.Mocks; +using Xunit; + +namespace SharpCompress.Test.Streams; + +public class SharpCompressStreamPassthroughAsyncTest +{ + [Fact] + public async ValueTask CreateNonDisposing_ReadAsync_DelegatesDirectly() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + var buffer = new byte[5]; + int bytesRead = await stream.ReadAsync(buffer, 0, buffer.Length).ConfigureAwait(false); + Assert.Equal(5, bytesRead); + Assert.Equal(new byte[] { 1, 2, 3, 4, 5 }, buffer); + } + + [Fact] + public async ValueTask CreateNonDisposing_WriteAsync_DelegatesDirectly() + { + var ms = new MemoryStream(); + var stream = SharpCompressStream.CreateNonDisposing(ms); + var data = new byte[] { 1, 2, 3, 4, 5 }; + await stream.WriteAsync(data, 0, data.Length).ConfigureAwait(false); + Assert.Equal(data, ms.ToArray()); + } + + [Fact] + public async ValueTask CreateNonDisposing_FlushAsync_DelegatesDirectly() + { + var ms = new MemoryStream(); + var stream = SharpCompressStream.CreateNonDisposing(ms); + await stream.WriteAsync(new byte[] { 1, 2, 3 }, 0, 3).ConfigureAwait(false); + await stream.FlushAsync().ConfigureAwait(false); + Assert.Equal(3, ms.Length); + } + + [Fact] + public async ValueTask CreateNonDisposing_CanReadAsync_ReturnsTrue() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + Assert.True(stream.CanRead); + } + + [Fact] + public async ValueTask CreateNonDisposing_ReadAsync_WithCancellationToken() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + var buffer = new byte[5]; + var cts = new System.Threading.CancellationTokenSource(); + int bytesRead = await stream + .ReadAsync(buffer, 0, buffer.Length, cts.Token) + .ConfigureAwait(false); + Assert.Equal(5, bytesRead); + } + + [Fact] + public async ValueTask CreateNonDisposing_WriteAsync_WithCancellationToken() + { + var ms = new MemoryStream(); + var stream = SharpCompressStream.CreateNonDisposing(ms); + var data = new byte[] { 1, 2, 3, 4, 5 }; + var cts = new System.Threading.CancellationTokenSource(); + await stream.WriteAsync(data, 0, data.Length, cts.Token).ConfigureAwait(false); + Assert.Equal(data, ms.ToArray()); + } + + [Fact] + public async ValueTask CreateNonDisposing_FlushAsync_WithCancellationToken() + { + var ms = new MemoryStream(); + var stream = SharpCompressStream.CreateNonDisposing(ms); + await stream.WriteAsync(new byte[] { 1, 2, 3 }, 0, 3).ConfigureAwait(false); + var cts = new System.Threading.CancellationTokenSource(); + await stream.FlushAsync(cts.Token).ConfigureAwait(false); + Assert.Equal(3, ms.Length); + } + +#if !LEGACY_DOTNET + [Fact] + public async ValueTask CreateNonDisposing_DoesNotDisposeUnderlying_Async() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + await stream.DisposeAsync().ConfigureAwait(false); + Assert.Equal(0, ms.Position); + Assert.True(ms.CanRead); + } + + [Fact] + public async ValueTask CreateNonDisposing_DisposeAsync_WithThrowOnDisposeTrue_ThrowsInvalidOperation() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + stream.ThrowOnDispose = true; + await Assert + .ThrowsAsync(async () => + await stream.DisposeAsync().ConfigureAwait(false) + ) + .ConfigureAwait(false); + } +#endif +} diff --git a/tests/SharpCompress.Test/Streams/SharpCompressStreamPassthroughTest.cs b/tests/SharpCompress.Test/Streams/SharpCompressStreamPassthroughTest.cs new file mode 100644 index 00000000..6e346be1 --- /dev/null +++ b/tests/SharpCompress.Test/Streams/SharpCompressStreamPassthroughTest.cs @@ -0,0 +1,186 @@ +using System; +using System.IO; +using SharpCompress.Common; +using SharpCompress.IO; +using SharpCompress.Test.Mocks; +using Xunit; + +namespace SharpCompress.Test.Streams; + +public class SharpCompressStreamPassthroughTest +{ + [Fact] + public void CreateNonDisposing_LeaveStreamOpen_ReturnsTrue() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + Assert.True(stream.LeaveStreamOpen); + } + + [Fact] + public void CreateNonDisposing_IsPassthrough_ReturnsTrue() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + Assert.True(stream.IsPassthrough); + } + + [Fact] + public void CreateNonDisposing_CanRead_ReturnsTrue() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + Assert.True(stream.CanRead); + } + + [Fact] + public void CreateNonDisposing_CanSeek_DelegatesToUnderlyingSeekableStream() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + Assert.Equal(ms.CanSeek, stream.CanSeek); + Assert.True(stream.CanSeek); + } + + [Fact] + public void CreateNonDisposing_CanSeek_DelegatesToUnderlyingNonSeekableStream() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new ForwardOnlyStream(ms); + var stream = SharpCompressStream.CreateNonDisposing(nonSeekableMs); + Assert.Equal(nonSeekableMs.CanSeek, stream.CanSeek); + Assert.False(stream.CanSeek); + } + + [Fact] + public void CreateNonDisposing_CanWrite_DelegatesToUnderlyingStream() + { + var ms = new MemoryStream(); + var stream = SharpCompressStream.CreateNonDisposing(ms); + Assert.Equal(ms.CanWrite, stream.CanWrite); + Assert.True(stream.CanWrite); + } + + [Fact] + public void CreateNonDisposing_Read_DelegatesDirectlyWithoutBuffering() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + var buffer = new byte[5]; + int bytesRead = stream.Read(buffer, 0, buffer.Length); + Assert.Equal(5, bytesRead); + Assert.Equal(new byte[] { 1, 2, 3, 4, 5 }, buffer); + } + + [Fact] + public void CreateNonDisposing_PositionGet_DelegatesToUnderlyingStream() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + ms.Position = 2; + var stream = SharpCompressStream.CreateNonDisposing(ms); + Assert.Equal(ms.Position, stream.Position); + Assert.Equal(2, stream.Position); + } + + [Fact] + public void CreateNonDisposing_PositionSet_DelegatesToUnderlyingStream() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + stream.Position = 3; + Assert.Equal(3, ms.Position); + Assert.Equal(3, stream.Position); + } + + [Fact] + public void CreateNonDisposing_DoesNotDisposeUnderlyingStream() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + stream.Dispose(); + Assert.Equal(0, ms.Position); + Assert.True(ms.CanRead); + } + + [Fact] + public void CreateNonDisposing_Length_DelegatesToUnderlyingStream() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + Assert.Equal(ms.Length, stream.Length); + Assert.Equal(5, stream.Length); + } + + [Fact] + public void CreateNonDisposing_Seek_DelegatesToUnderlyingStream() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + long result = stream.Seek(3, SeekOrigin.Begin); + Assert.Equal(3, result); + Assert.Equal(3, ms.Position); + Assert.Equal(3, stream.Position); + } + + [Fact] + public void CreateNonDisposing_Flush_DelegatesToUnderlyingStream() + { + var ms = new MemoryStream(); + var stream = SharpCompressStream.CreateNonDisposing(ms); + stream.Write(new byte[] { 1, 2, 3 }, 0, 3); + stream.Flush(); + Assert.Equal(3, ms.Length); + } + + [Fact] + public void CreateNonDisposing_SetLength_DelegatesToUnderlyingStream() + { + var ms = new MemoryStream(); + var stream = SharpCompressStream.CreateNonDisposing(ms); + stream.SetLength(20); + Assert.Equal(20, stream.Length); + Assert.Equal(20, ms.Length); + } + + [Fact] + public void CreateNonDisposing_Write_DelegatesToUnderlyingStream() + { + var ms = new MemoryStream(); + var stream = SharpCompressStream.CreateNonDisposing(ms); + var data = new byte[] { 1, 2, 3, 4, 5 }; + stream.Write(data, 0, data.Length); + Assert.Equal(data, ms.ToArray()); + } + + [Fact] + public void CreateNonDisposing_StartRecording_ThrowsInvalidOperation() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + Assert.Throws(() => stream.StartRecording()); + } + + [Fact] + public void CreateNonDisposing_Rewind_ThrowsInvalidOperation() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + Assert.Throws(() => stream.Rewind()); + } + + [Fact] + public void CreateNonDisposing_StopRecording_ThrowsInvalidOperation() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + Assert.Throws(() => stream.StopRecording()); + } + + [Fact] + public void CreateNonDisposing_IsRecording_AlwaysFalse() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + Assert.False(stream.IsRecording); + } +} diff --git a/tests/SharpCompress.Test/Streams/SharpCompressStreamPropertyTest.cs b/tests/SharpCompress.Test/Streams/SharpCompressStreamPropertyTest.cs new file mode 100644 index 00000000..2262960d --- /dev/null +++ b/tests/SharpCompress.Test/Streams/SharpCompressStreamPropertyTest.cs @@ -0,0 +1,177 @@ +using System; +using System.IO; +using SharpCompress.IO; +using SharpCompress.Test.Mocks; +using Xunit; + +namespace SharpCompress.Test.Streams; + +public class SharpCompressStreamPropertyTest +{ + [Fact] + public void BaseStream_ReturnsUnderlyingStream() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + Assert.Same(ms, stream.BaseStream()); + } + + [Fact] + public void IsPassthrough_CreateNonDisposing_ReturnsTrue() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + Assert.True(stream.IsPassthrough); + } + + [Fact] + public void IsPassthrough_CreateWithBuffer_ReturnsFalse() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new ForwardOnlyStream(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + Assert.False(stream.IsPassthrough); + } + + [Fact] + public void IsPassthrough_CreateSeekable_ReturnsFalse() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.Create(ms); + Assert.False(stream.IsPassthrough); + } + + [Fact] + public void IsRecording_AfterStartRecording_ReturnsTrue() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new ForwardOnlyStream(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + stream.StartRecording(); + Assert.True(stream.IsRecording); + } + + [Fact] + public void IsRecording_AfterStopRecording_ReturnsFalse() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new ForwardOnlyStream(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + stream.StartRecording(); + stream.Read(new byte[4], 0, 4); + stream.StopRecording(); + Assert.False(stream.IsRecording); + } + + [Fact] + public void IsRecording_AfterRewindWithStopRecording_ReturnsFalse() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new ForwardOnlyStream(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + stream.StartRecording(); + stream.Read(new byte[4], 0, 4); + stream.Rewind(true); + Assert.False(stream.IsRecording); + } + + [Fact] + public void LeaveStreamOpen_CreateNonDisposing_ReturnsTrue() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + Assert.True(stream.LeaveStreamOpen); + } + + [Fact] + public void LeaveStreamOpen_CreateWithBuffer_ReturnsFalse() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new ForwardOnlyStream(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + Assert.False(stream.LeaveStreamOpen); + } + + [Fact] + public void LeaveStreamOpen_CreateSeekable_ReturnsFalse() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.Create(ms); + Assert.False(stream.LeaveStreamOpen); + } + + [Fact] + public void CanRead_AlwaysTrue() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + Assert.True(stream.CanRead); + } + + [Fact] + public void CanSeek_PassthroughWithSeekable_DelegatesTrue() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var stream = SharpCompressStream.CreateNonDisposing(ms); + Assert.True(stream.CanSeek); + } + + [Fact] + public void CanSeek_PassthroughWithNonSeekable_DelegatesFalse() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new ForwardOnlyStream(ms); + var stream = SharpCompressStream.CreateNonDisposing(nonSeekableMs); + Assert.False(stream.CanSeek); + } + + [Fact] + public void CanWrite_PassthroughWithWritable_DelegatesTrue() + { + var ms = new MemoryStream(); + var stream = SharpCompressStream.CreateNonDisposing(ms); + Assert.True(stream.CanWrite); + } + + [Fact] + public void CanWrite_PassthroughWithReadOnly_DelegatesFalse() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var readOnlyMs = new ReadOnlyStreamWrapper(ms); + var stream = SharpCompressStream.CreateNonDisposing(readOnlyMs); + Assert.False(stream.CanWrite); + } + + private class ReadOnlyStreamWrapper : Stream + { + private readonly Stream _baseStream; + + public ReadOnlyStreamWrapper(Stream baseStream) + { + _baseStream = baseStream; + } + + public override bool CanRead => _baseStream.CanRead; + public override bool CanSeek => _baseStream.CanSeek; + public override bool CanWrite => false; + public override long Length => _baseStream.Length; + public override long Position + { + get => _baseStream.Position; + set => _baseStream.Position = value; + } + + public override void Flush() => _baseStream.Flush(); + + public override int Read(byte[] buffer, int offset, int count) => + _baseStream.Read(buffer, offset, count); + + public override long Seek(long offset, SeekOrigin origin) => + _baseStream.Seek(offset, origin); + + public override void SetLength(long value) => _baseStream.SetLength(value); + + public override void Write(byte[] buffer, int offset, int count) => + throw new NotSupportedException(); + } +} diff --git a/tests/SharpCompress.Test/Streams/SharpCompressStreamSeekAsyncTest.cs b/tests/SharpCompress.Test/Streams/SharpCompressStreamSeekAsyncTest.cs new file mode 100644 index 00000000..3a93960f --- /dev/null +++ b/tests/SharpCompress.Test/Streams/SharpCompressStreamSeekAsyncTest.cs @@ -0,0 +1,144 @@ +using System; +using System.IO; +using System.Threading.Tasks; +using SharpCompress.IO; +using Xunit; + +namespace SharpCompress.Test.Streams; + +public class SharpCompressStreamSeekAsyncTest +{ + private class NonSeekableStreamWrapper : Stream + { + private readonly Stream _baseStream; + + public NonSeekableStreamWrapper(Stream baseStream) => _baseStream = baseStream; + + public override bool CanRead => _baseStream.CanRead; + public override bool CanSeek => false; + public override bool CanWrite => _baseStream.CanWrite; + public override long Length => _baseStream.Length; + public override long Position + { + get => throw new NotSupportedException(); + set => throw new NotSupportedException(); + } + + public override void Flush() => _baseStream.Flush(); + + public override int Read(byte[] buffer, int offset, int count) => + _baseStream.Read(buffer, offset, count); + + public override long Seek(long offset, SeekOrigin origin) => + throw new NotSupportedException(); + + public override void SetLength(long value) => _baseStream.SetLength(value); + + public override void Write(byte[] buffer, int offset, int count) => + _baseStream.Write(buffer, offset, count); + + protected override void Dispose(bool disposing) + { + if (disposing) + { + _baseStream.Dispose(); + } + + base.Dispose(disposing); + } + } + + [Fact] + public async ValueTask SeekAsync_AfterReadAsync_MaintainsPosition() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5, 6, 7, 8 }); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + stream.StartRecording(); + var buffer = new byte[4]; + await stream.ReadAsync(buffer, 0, 4).ConfigureAwait(false); + Assert.Equal(4, stream.Position); + + stream.Seek(-2, SeekOrigin.Current); + Assert.Equal(2, stream.Position); + + await stream.ReadAsync(buffer, 0, 2).ConfigureAwait(false); + Assert.Equal(3, buffer[0]); + Assert.Equal(4, buffer[1]); + } + + [Fact] + public async ValueTask Position_Set_AfterAsyncRead_WorksCorrectly() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5, 6, 7, 8 }); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + stream.StartRecording(); + var buffer = new byte[8]; + await stream.ReadAsync(buffer, 0, 8).ConfigureAwait(false); + + stream.Position = 2; + Assert.Equal(2, stream.Position); + + var readBuffer = new byte[2]; + await stream.ReadAsync(readBuffer, 0, 2).ConfigureAwait(false); + Assert.Equal(3, readBuffer[0]); + Assert.Equal(4, readBuffer[1]); + } + + [Fact] + public async ValueTask SeekAsync_ToRecordingStart_AfterAsyncRead_WorksCorrectly() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5, 6, 7, 8 }); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + stream.StartRecording(); + var buffer = new byte[4]; + await stream.ReadAsync(buffer, 0, 4).ConfigureAwait(false); + + stream.Position = 0; + Assert.Equal(0, stream.Position); + + await stream.ReadAsync(buffer, 0, 4).ConfigureAwait(false); + Assert.Equal(1, buffer[0]); + Assert.Equal(2, buffer[1]); + Assert.Equal(3, buffer[2]); + Assert.Equal(4, buffer[3]); + } + + [Fact] + public async ValueTask SeekAsync_ZeroCurrentOrigin_DoesNotMove() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5, 6, 7, 8 }); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + stream.StartRecording(); + var buffer = new byte[4]; + await stream.ReadAsync(buffer, 0, 4).ConfigureAwait(false); + Assert.Equal(4, stream.Position); + + stream.Seek(0, SeekOrigin.Current); + Assert.Equal(4, stream.Position); + } + + [Fact] + public async ValueTask SeekAsync_NegativeCurrent_MovesBackward() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5, 6, 7, 8 }); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + stream.StartRecording(); + var buffer = new byte[6]; + await stream.ReadAsync(buffer, 0, 6).ConfigureAwait(false); + Assert.Equal(6, stream.Position); + + stream.Seek(-3, SeekOrigin.Current); + Assert.Equal(3, stream.Position); + + var readBuffer = new byte[3]; + await stream.ReadAsync(readBuffer, 0, 3).ConfigureAwait(false); + Assert.Equal(4, readBuffer[0]); + Assert.Equal(5, readBuffer[1]); + Assert.Equal(6, readBuffer[2]); + } +} diff --git a/tests/SharpCompress.Test/Streams/SharpCompressStreamSeekTest.cs b/tests/SharpCompress.Test/Streams/SharpCompressStreamSeekTest.cs new file mode 100644 index 00000000..04c86284 --- /dev/null +++ b/tests/SharpCompress.Test/Streams/SharpCompressStreamSeekTest.cs @@ -0,0 +1,201 @@ +using System; +using System.IO; +using SharpCompress.Common; +using SharpCompress.IO; +using SharpCompress.Test.Mocks; +using Xunit; + +namespace SharpCompress.Test.Streams; + +public class SharpCompressStreamSeekTest +{ + private class NonSeekableStreamWrapper : Stream + { + private readonly Stream _baseStream; + + public NonSeekableStreamWrapper(Stream baseStream) + { + _baseStream = baseStream; + } + + public override bool CanRead => _baseStream.CanRead; + public override bool CanSeek => false; + public override bool CanWrite => _baseStream.CanWrite; + public override long Length => _baseStream.Length; + public override long Position + { + get => throw new NotSupportedException(); + set => throw new NotSupportedException(); + } + + public override void Flush() => _baseStream.Flush(); + + public override int Read(byte[] buffer, int offset, int count) => + _baseStream.Read(buffer, offset, count); + + public override long Seek(long offset, SeekOrigin origin) => + throw new NotSupportedException(); + + public override void SetLength(long value) => _baseStream.SetLength(value); + + public override void Write(byte[] buffer, int offset, int count) => + _baseStream.Write(buffer, offset, count); + + protected override void Dispose(bool disposing) + { + if (disposing) + { + _baseStream.Dispose(); + } + + base.Dispose(disposing); + } + } + + [Fact] + public void Seek_CurrentOrigin_MovesRelativeToCurrent() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5, 6, 7, 8 }); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + stream.StartRecording(); + var buffer = new byte[4]; + stream.Read(buffer, 0, 4); + Assert.Equal(4, stream.Position); + + stream.Seek(-2, SeekOrigin.Current); + Assert.Equal(2, stream.Position); + + stream.Read(buffer, 0, 2); + Assert.Equal(3, buffer[0]); + Assert.Equal(4, buffer[1]); + } + + [Fact] + public void Seek_BeginOrigin_MovesToAbsolutePosition() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5, 6, 7, 8 }); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + stream.StartRecording(); + var buffer = new byte[8]; + stream.Read(buffer, 0, 8); + + stream.Seek(2, SeekOrigin.Begin); + Assert.Equal(2, stream.Position); + + var readBuffer = new byte[2]; + stream.Read(readBuffer, 0, 2); + Assert.Equal(3, readBuffer[0]); + Assert.Equal(4, readBuffer[1]); + } + + [Fact] + public void Seek_ToExactBufferBoundary_Succeeds() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5, 6, 7, 8 }); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + stream.StartRecording(); + var buffer = new byte[4]; + stream.Read(buffer, 0, 4); + + stream.Seek(4, SeekOrigin.Begin); + Assert.Equal(4, stream.Position); + + stream.Read(buffer, 0, 4); + Assert.Equal(5, buffer[0]); + Assert.Equal(6, buffer[1]); + Assert.Equal(7, buffer[2]); + Assert.Equal(8, buffer[3]); + } + + [Fact] + public void Position_SetWithinRecordedRange_Succeeds() + { + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5, 6, 7, 8 }); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 128); + stream.StartRecording(); + var buffer = new byte[8]; + stream.Read(buffer, 0, 8); + + stream.Position = 2; + Assert.Equal(2, stream.Position); + + var readBuffer = new byte[2]; + stream.Read(readBuffer, 0, 2); + Assert.Equal(3, readBuffer[0]); + Assert.Equal(4, readBuffer[1]); + } + + [Fact] + public void StartRecording_WithLargerMinBufferSize_AllowsLargeRewind() + { + // Simulates the BZip2 scenario: the ring buffer must be large enough + // from the moment StartRecording is called so that a large probe read + // (up to 900 KB for BZip2) can be rewound without buffer overflow. + const int largeSize = 100_000; + const int largeReadSize = 80_000; + + var data = new byte[largeSize]; + for (var i = 0; i < data.Length; i++) + { + data[i] = (byte)(i + 1); + } + + var ms = new MemoryStream(data); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, largeSize); + + // Pass the required size upfront — no expansion needed later + stream.StartRecording(largeSize); + + // Read a large amount (simulating BZip2 block decompression during IsTarFile probe) + var largeBuffer = new byte[largeReadSize]; + stream.Read(largeBuffer, 0, largeReadSize); + + // Rewind must succeed because the buffer was large enough from the start + stream.Rewind(); + + var verifyBuffer = new byte[largeReadSize]; + stream.Read(verifyBuffer, 0, largeReadSize); + Assert.Equal(data[0], verifyBuffer[0]); + Assert.Equal(data[largeReadSize - 1], verifyBuffer[largeReadSize - 1]); + } + + [Fact] + public void StartRecording_DefaultSize_UsesConstantsRewindableBufferSize() + { + // When no minimum is specified StartRecording uses the global default. + var ms = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 }); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs); + stream.StartRecording(); + + var buffer = new byte[5]; + stream.Read(buffer, 0, 5); + stream.Rewind(); + + var readBuffer = new byte[5]; + stream.Read(readBuffer, 0, 5); + Assert.Equal(1, readBuffer[0]); + Assert.Equal(5, readBuffer[4]); + } + + [Fact] + public void StartRecording_WithExistingSmallerRingBuffer_Throws() + { + var ms = new MemoryStream(new byte[131_072]); + var nonSeekableMs = new NonSeekableStreamWrapper(ms); + var stream = SharpCompressStream.Create(nonSeekableMs, 32_768); + + var exception = Assert.Throws(() => + stream.StartRecording(131_072) + ); + + Assert.Contains("ring buffer", exception.Message); + Assert.Contains("131072", exception.Message); + Assert.Contains("32768", exception.Message); + } +} diff --git a/tests/SharpCompress.Test/Streams/WinzipAesCryptoStreamTests.cs b/tests/SharpCompress.Test/Streams/WinzipAesCryptoStreamTests.cs new file mode 100644 index 00000000..578f6212 --- /dev/null +++ b/tests/SharpCompress.Test/Streams/WinzipAesCryptoStreamTests.cs @@ -0,0 +1,257 @@ +using System; +using System.Buffers.Binary; +using System.Diagnostics.CodeAnalysis; +using System.IO; +using System.Linq; +using System.Security.Cryptography; +using System.Threading.Tasks; +using SharpCompress.Common.Zip; +using Xunit; + +namespace SharpCompress.Test.Streams; + +public class WinzipAesCryptoStreamTests +{ + [Fact] + public void Read_Decrypts_Data_For_Aligned_Buffer_Size() + { + const string password = "sample-password"; + byte[] plainText = Enumerable.Range(0, 64).Select(i => (byte)i).ToArray(); + byte[] salt = [0x10, 0x21, 0x32, 0x43, 0x54, 0x65, 0x76, 0x87]; + using var stream = CreateStream(plainText, password, salt); + + byte[] actual = new byte[plainText.Length]; + int bytesRead = stream.Read(actual, 0, actual.Length); + + Assert.Equal(plainText.Length, bytesRead); + Assert.Equal(plainText, actual); + } + + [Fact] + public void Read_Preserves_Keystream_Between_NonAligned_Reads() + { + const string password = "sample-password"; + byte[] plainText = Enumerable.Range(0, 97).Select(i => (byte)i).ToArray(); + byte[] salt = [0x10, 0x21, 0x32, 0x43, 0x54, 0x65, 0x76, 0x87]; + using var stream = CreateStream(plainText, password, salt); + + byte[] actual = ReadWithChunkPattern( + (buffer, offset, count) => stream.Read(buffer, offset, count), + plainText.Length, + [13, 5, 29, 7, 43] + ); + + Assert.Equal(plainText, actual); + } + + [Fact] + public async Task ReadAsync_Preserves_Keystream_Between_NonAligned_Reads() + { + const string password = "sample-password"; + byte[] plainText = Enumerable + .Range(0, 113) + .Select(i => unchecked((byte)(255 - i))) + .ToArray(); + byte[] salt = [0x91, 0x82, 0x73, 0x64, 0x55, 0x46, 0x37, 0x28]; + using var stream = CreateStream(plainText, password, salt); + + byte[] actual = await ReadWithChunkPatternAsync( + (buffer, offset, count) => stream.ReadAsync(buffer, offset, count), + plainText.Length, + [11, 3, 17, 5, 41] + ); + + Assert.Equal(plainText, actual); + } + + [Fact] + public async Task ReadAsync_Memory_Preserves_Keystream_Between_NonAligned_Reads() + { + const string password = "sample-password"; + byte[] plainText = Enumerable + .Range(0, 113) + .Select(i => unchecked((byte)(255 - i))) + .ToArray(); + byte[] salt = [0x91, 0x82, 0x73, 0x64, 0x55, 0x46, 0x37, 0x28]; + using var stream = CreateStream(plainText, password, salt); + + byte[] actual = await ReadWithChunkPatternMemoryAsync( + stream, + plainText.Length, + [11, 3, 17, 5, 41] + ); + + Assert.Equal(plainText, actual); + } + + [Fact] + public void Read_Stops_At_Encrypted_Payload_Length() + { + const string password = "sample-password"; + byte[] plainText = Enumerable.Range(0, 31).Select(i => (byte)(i * 3)).ToArray(); + byte[] salt = [0xA1, 0xB2, 0xC3, 0xD4, 0x01, 0x12, 0x23, 0x34]; + using var stream = CreateStream(plainText, password, salt); + + byte[] actual = new byte[plainText.Length + 16]; + int bytesRead = stream.Read(actual, 0, actual.Length); + int eofRead = stream.Read(actual, bytesRead, actual.Length - bytesRead); + + Assert.Equal(plainText.Length, bytesRead); + Assert.Equal(0, eofRead); + Assert.Equal(plainText, actual.Take(bytesRead).ToArray()); + } + + private static WinzipAesCryptoStream CreateStream( + byte[] plainText, + string password, + byte[] salt + ) + { + var encryptionData = CreateEncryptionData(password, salt); + byte[] cipherText = EncryptCtr(plainText, encryptionData.KeyBytes); + byte[] archiveBytes = cipherText.Concat(new byte[10]).ToArray(); + return new WinzipAesCryptoStream( + new MemoryStream(archiveBytes, writable: false), + encryptionData, + cipherText.Length + ); + } + + [SuppressMessage( + "Security", + "CA5379:Rfc2898DeriveBytes might be using a weak hash algorithm", + Justification = "WinZip AES interop requires PBKDF2 with SHA-1." + )] + private static WinzipAesEncryptionData CreateEncryptionData(string password, byte[] salt) + { +#pragma warning disable SYSLIB0060 // Rfc2898DeriveBytes might be using a weak hash algorithm + using var deriveBytes = new Rfc2898DeriveBytes( + password, + salt, + 1000, + HashAlgorithmName.SHA1 + ); +#pragma warning restore SYSLIB0060 + deriveBytes.GetBytes(16); + deriveBytes.GetBytes(16); + byte[] passwordVerifyValue = deriveBytes.GetBytes(2); + + return new WinzipAesEncryptionData( + WinzipAesKeySize.KeySize128, + salt, + passwordVerifyValue, + password + ); + } + + private static byte[] EncryptCtr(byte[] plainText, byte[] keyBytes) + { + using var aes = Aes.Create(); + aes.BlockSize = 128; + aes.KeySize = keyBytes.Length * 8; + aes.Mode = CipherMode.ECB; + aes.Padding = PaddingMode.None; + + using var encryptor = aes.CreateEncryptor(keyBytes, new byte[16]); + byte[] counter = new byte[16]; + byte[] counterOut = new byte[16]; + byte[] cipherText = new byte[plainText.Length]; + int nonce = 1; + int offset = 0; + + while (offset < plainText.Length) + { + BinaryPrimitives.WriteInt32LittleEndian(counter, nonce++); + encryptor.TransformBlock(counter, 0, counter.Length, counterOut, 0); + + int blockLength = Math.Min(counterOut.Length, plainText.Length - offset); + for (int i = 0; i < blockLength; i++) + { + cipherText[offset + i] = (byte)(plainText[offset + i] ^ counterOut[i]); + } + + offset += blockLength; + } + + return cipherText; + } + + private static byte[] ReadWithChunkPattern( + Func read, + int totalLength, + int[] chunkPattern + ) + { + byte[] actual = new byte[totalLength]; + int offset = 0; + int chunkIndex = 0; + + while (offset < totalLength) + { + int requested = Math.Min( + chunkPattern[chunkIndex % chunkPattern.Length], + totalLength - offset + ); + int bytesRead = read(actual, offset, requested); + Assert.True(bytesRead > 0); + offset += bytesRead; + chunkIndex++; + } + + return actual; + } + + private static async Task ReadWithChunkPatternAsync( + Func> readAsync, + int totalLength, + int[] chunkPattern + ) + { + byte[] actual = new byte[totalLength]; + int offset = 0; + int chunkIndex = 0; + + while (offset < totalLength) + { + int requested = Math.Min( + chunkPattern[chunkIndex % chunkPattern.Length], + totalLength - offset + ); + int bytesRead = await readAsync(actual, offset, requested); + Assert.True(bytesRead > 0); + offset += bytesRead; + chunkIndex++; + } + + return actual; + } + + private static async Task ReadWithChunkPatternMemoryAsync( + Stream stream, + int totalLength, + int[] chunkPattern + ) + { + byte[] actual = new byte[totalLength]; + int offset = 0; + int chunkIndex = 0; + + while (offset < totalLength) + { + int requested = Math.Min( + chunkPattern[chunkIndex % chunkPattern.Length], + totalLength - offset + ); +#if NET48 + int bytesRead = await stream.ReadAsync(actual, offset, requested); +#else + int bytesRead = await stream.ReadAsync(actual.AsMemory(offset, requested)); +#endif + Assert.True(bytesRead > 0); + offset += bytesRead; + chunkIndex++; + } + + return actual; + } +} diff --git a/tests/SharpCompress.Test/Streams/ZLibBaseStreamAsyncTests.cs b/tests/SharpCompress.Test/Streams/ZLibBaseStreamAsyncTests.cs new file mode 100644 index 00000000..e2f1506c --- /dev/null +++ b/tests/SharpCompress.Test/Streams/ZLibBaseStreamAsyncTests.cs @@ -0,0 +1,112 @@ +using System.IO; +using System.Text; +using System.Threading.Tasks; +using AwesomeAssertions; +using SharpCompress.Compressors; +using SharpCompress.Compressors.Deflate; +using SharpCompress.IO; +using Xunit; + +namespace SharpCompress.Test.Streams; + +public class ZLibBaseStreamAsyncTests +{ + [Fact] + public async ValueTask TestChunkedZlibCompressesEverythingAsync() + { + var plainData = new byte[] + { + 0xf7, + 0x1b, + 0xda, + 0x0f, + 0xb6, + 0x2b, + 0x3d, + 0x91, + 0xd7, + 0xe1, + 0xb5, + 0x11, + 0x34, + 0x5a, + 0x51, + 0x3f, + 0x8b, + 0xce, + 0x49, + 0xd2, + }; + var buf = new byte[plainData.Length * 2]; + + var plainStream1 = new MemoryStream(plainData); + var compressor1 = new DeflateStream(plainStream1, CompressionMode.Compress); + // This is enough to read the entire data + var realCompressedSize = await compressor1 + .ReadAsync(buf, 0, plainData.Length * 2) + .ConfigureAwait(false); + + var plainStream2 = new MemoryStream(plainData); + var compressor2 = new DeflateStream(plainStream2, CompressionMode.Compress); + var total = 0; + var r = -1; // Jumpstart + while (r != 0) + { + // Reading in chunks + r = await compressor2.ReadAsync(buf, 0, plainData.Length).ConfigureAwait(false); + total += r; + } + + Assert.Equal(total, realCompressedSize); + } + + [Fact] + public async ValueTask Zlib_should_read_the_previously_written_message_async() + { + var message = new string('a', 131073); // 131073 causes the failure, but 131072 (-1) doesn't + var bytes = Encoding.ASCII.GetBytes(message); + + using var inputStream = new MemoryStream(bytes); + using var compressedStream = new MemoryStream(); + using var byteBufferStream = new BufferedStream(inputStream); // System.IO + await CompressAsync(byteBufferStream, compressedStream, compressionLevel: 1) + .ConfigureAwait(false); + compressedStream.Position = 0; + + using var decompressedStream = new MemoryStream(); + await DecompressAsync(compressedStream, decompressedStream).ConfigureAwait(false); + + byteBufferStream.Position = 0; + var result = Encoding.ASCII.GetString( + await GetBytesAsync(byteBufferStream).ConfigureAwait(false) + ); + result.Should().Be(message); + } + + private async ValueTask CompressAsync(Stream input, Stream output, int compressionLevel) + { + using var zlibStream = new ZlibStream( + SharpCompressStream.CreateNonDisposing(output), + CompressionMode.Compress, + (CompressionLevel)compressionLevel + ); + zlibStream.FlushMode = FlushType.Sync; + await input.CopyToAsync(zlibStream).ConfigureAwait(false); + } + + private async ValueTask DecompressAsync(Stream input, Stream output) + { + using var zlibStream = new ZlibStream( + SharpCompressStream.CreateNonDisposing(input), + CompressionMode.Decompress + ); + await zlibStream.CopyToAsync(output).ConfigureAwait(false); + } + + private async ValueTask GetBytesAsync(BufferedStream stream) + { + var bytes = new byte[stream.Length]; + await stream.ReadAsync(bytes, 0, (int)stream.Length).ConfigureAwait(false); + return bytes; + } +} diff --git a/tests/SharpCompress.Test/Streams/ZlibBaseStreamTests.cs b/tests/SharpCompress.Test/Streams/ZlibBaseStreamTests.cs index 56f934d6..0fd53d8f 100644 --- a/tests/SharpCompress.Test/Streams/ZlibBaseStreamTests.cs +++ b/tests/SharpCompress.Test/Streams/ZlibBaseStreamTests.cs @@ -1,6 +1,6 @@ using System.IO; using System.Text; -using FluentAssertions; +using AwesomeAssertions; using SharpCompress.Compressors; using SharpCompress.Compressors.Deflate; using SharpCompress.IO; @@ -34,7 +34,7 @@ public class ZLibBaseStreamTests 0x8b, 0xce, 0x49, - 0xd2 + 0xd2, }; var buf = new byte[plainData.Length * 2]; @@ -80,7 +80,7 @@ public class ZLibBaseStreamTests private void Compress(Stream input, Stream output, int compressionLevel) { using var zlibStream = new ZlibStream( - NonDisposingStream.Create(output), + SharpCompressStream.CreateNonDisposing(output), CompressionMode.Compress, (CompressionLevel)compressionLevel ); @@ -91,7 +91,7 @@ public class ZLibBaseStreamTests private void Decompress(Stream input, Stream output) { using var zlibStream = new ZlibStream( - NonDisposingStream.Create(input), + SharpCompressStream.CreateNonDisposing(input), CompressionMode.Decompress ); zlibStream.CopyTo(output); diff --git a/tests/SharpCompress.Test/Tar/TarArchiveAsyncTests.cs b/tests/SharpCompress.Test/Tar/TarArchiveAsyncTests.cs new file mode 100644 index 00000000..28947e97 --- /dev/null +++ b/tests/SharpCompress.Test/Tar/TarArchiveAsyncTests.cs @@ -0,0 +1,481 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using SharpCompress.Archives; +using SharpCompress.Archives.Tar; +using SharpCompress.Common; +using SharpCompress.Readers; +using SharpCompress.Readers.Tar; +using SharpCompress.Test.Mocks; +using SharpCompress.Writers; +using SharpCompress.Writers.Tar; +using Xunit; + +namespace SharpCompress.Test.Tar; + +public class TarArchiveAsyncTests : ArchiveTests +{ + public TarArchiveAsyncTests() => UseExtensionInsteadOfNameToVerify = true; + + [Fact] + public async ValueTask TarArchiveStreamRead_Async() => await ArchiveStreamReadAsync("Tar.tar"); + + [Fact] + public async ValueTask TarArchiveOpenAsyncStream_Throws_On_NonSeekable_Stream() + { + using Stream stream = new ForwardOnlyStream( + File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar")) + ); + + await Assert.ThrowsAsync(async () => + await TarArchive.OpenAsyncArchive(new AsyncOnlyStream(stream)) + ); + } + + [Fact] + public async ValueTask TarArchiveOpenAsyncStream_Throws_On_Unreadable_Stream() + { + using var stream = new TestStream(new MemoryStream(), false, true, true); + + await Assert.ThrowsAsync(() => + TarArchive.OpenAsyncArchive(stream).AsTask() + ); + } + + [Fact] + public async ValueTask Tar_FileName_Exactly_100_Characters_Async() + { + var archive = "Tar_FileName_Exactly_100_Characters.tar"; + + // create the 100 char filename + var filename = + "filename_with_exactly_100_characters_______________________________________________________________X"; + + // Step 1: create a tar file containing a file with the test name + using (Stream stream = File.OpenWrite(Path.Combine(SCRATCH2_FILES_PATH, archive))) + { + await using ( + var writer = await WriterFactory.OpenAsyncWriter( + new AsyncOnlyStream(stream), + ArchiveType.Tar, + new WriterOptions(CompressionType.None) { LeaveStreamOpen = false } + ) + ) + using (Stream inputStream = new MemoryStream()) + { + var sw = new StreamWriter(inputStream); + await sw.WriteAsync("dummy filecontent"); + await sw.FlushAsync(); + + inputStream.Position = 0; + await writer.WriteAsync(filename, inputStream, null); + } + } + + // Step 2: check if the written tar file can be read correctly + var unmodified = Path.Combine(SCRATCH2_FILES_PATH, archive); + await using ( + var archive2 = await TarArchive.OpenAsyncArchive( + new AsyncOnlyStream(File.OpenRead(unmodified)), + ReaderOptions.ForExternalStream.WithLeaveStreamOpen(false) + ) + ) + { + Assert.Equal(1, await archive2.EntriesAsync.CountAsync()); + Assert.Contains( + filename, + await archive2.EntriesAsync.Select(entry => entry.Key).ToListAsync() + ); + + await foreach (var entry in archive2.EntriesAsync) + { + using (var sr = new StreamReader(await entry.OpenEntryStreamAsync())) + { + Assert.Equal("dummy filecontent", await sr.ReadLineAsync()); + } + } + } + } + + [Fact] + public async ValueTask Tar_VeryLongFilepathReadback_Async() + { + var archive = "Tar_VeryLongFilepathReadback.tar"; + + // create a very long filename + var longFilename = ""; + for (var i = 0; i < 600; i = longFilename.Length) + { + longFilename += i.ToString("D10") + "-"; + } + + longFilename += ".txt"; + + // Step 1: create a tar file containing a file with a long name + using (Stream stream = File.OpenWrite(Path.Combine(SCRATCH2_FILES_PATH, archive))) + await using ( + var writer = await WriterFactory.OpenAsyncWriter( + new AsyncOnlyStream(stream), + ArchiveType.Tar, + new WriterOptions(CompressionType.None) { LeaveStreamOpen = false } + ) + ) + using (Stream inputStream = new MemoryStream()) + { + var sw = new StreamWriter(inputStream); + await sw.WriteAsync("dummy filecontent"); + await sw.FlushAsync(); + + inputStream.Position = 0; + await writer.WriteAsync(longFilename, inputStream, null); + } + + // Step 2: check if the written tar file can be read correctly + var unmodified = Path.Combine(SCRATCH2_FILES_PATH, archive); + await using ( + var archive2 = await TarArchive.OpenAsyncArchive( + new AsyncOnlyStream(File.OpenRead(unmodified)), + ReaderOptions.ForExternalStream.WithLeaveStreamOpen(false) + ) + ) + { + Assert.Equal(1, await archive2.EntriesAsync.CountAsync()); + Assert.Contains( + longFilename, + await archive2.EntriesAsync.Select(entry => entry.Key).ToListAsync() + ); + + await foreach (var entry in archive2.EntriesAsync) + { + using (var sr = new StreamReader(await entry.OpenEntryStreamAsync())) + { + Assert.Equal("dummy filecontent", await sr.ReadLineAsync()); + } + } + } +#if LEGACY_DOTNET + //add a delay because old .net sucks on DisposeAsync + await Task.Delay(TimeSpan.FromSeconds(1)); +#endif + } + + [Fact] + public async ValueTask Tar_Create_New_Async() + { + var scratchPath = Path.Combine(SCRATCH_FILES_PATH, "Tar.tar"); + var unmodified = Path.Combine(TEST_ARCHIVES_PATH, "Tar.noEmptyDirs.tar"); + + await using (var archive = await TarArchive.CreateAsyncArchive()) + { + await archive.AddAllFromDirectoryAsync(ORIGINAL_FILES_PATH); + var twopt = new TarWriterOptions(CompressionType.None, true) + { + ArchiveEncoding = new ArchiveEncoding { Default = Encoding.GetEncoding(866) }, + }; + await archive.SaveToAsync(scratchPath, twopt); + } + CompareArchivesByPath(unmodified, scratchPath); + } + + [Fact] + public async ValueTask Tar_Async_Dispose_Closes_New_Entry_Stream() + { + var entryStream = new TestStream(new MemoryStream(Encoding.UTF8.GetBytes("test"))); + + await using (var archive = await TarArchive.CreateAsyncArchive()) + { + await archive.AddEntryAsync( + "test.txt", + entryStream, + closeStream: true, + size: entryStream.Length + ); + await archive.SaveToAsync( + new MemoryStream(), + new TarWriterOptions(CompressionType.None, true) + ); + } + + Assert.True(entryStream.IsDisposed); + } + + [Fact] + public async ValueTask Tar_Random_Write_Add_Async() + { + var jpg = Path.Combine(ORIGINAL_FILES_PATH, "jpg", "test.jpg"); + var scratchPath = Path.Combine(SCRATCH_FILES_PATH, "Tar.mod.tar"); + var unmodified = Path.Combine(TEST_ARCHIVES_PATH, "Tar.mod.tar"); + var modified = Path.Combine(TEST_ARCHIVES_PATH, "Tar.noEmptyDirs.tar"); + + await using (var archive = await TarArchive.OpenAsyncArchive(unmodified)) + { + await archive.AddEntryAsync("jpg\\test.jpg", jpg); + await archive.SaveToAsync( + scratchPath, + new TarWriterOptions(CompressionType.None, true) + ); + } + CompareArchivesByPath(modified, scratchPath); + } + + [Fact] + public async ValueTask Tar_Random_Write_Remove_Async() + { + var scratchPath = Path.Combine(SCRATCH_FILES_PATH, "Tar.mod.tar"); + var modified = Path.Combine(TEST_ARCHIVES_PATH, "Tar.mod.tar"); + var unmodified = Path.Combine(TEST_ARCHIVES_PATH, "Tar.noEmptyDirs.tar"); + + await using (var archive = await TarArchive.OpenAsyncArchive(unmodified)) + { + var entry = await archive.EntriesAsync.SingleAsync(x => + x.Key.NotNull().EndsWith("jpg", StringComparison.OrdinalIgnoreCase) + ); + await archive.RemoveEntryAsync(entry); + await archive.SaveToAsync( + scratchPath, + new TarWriterOptions(CompressionType.None, true) + ); + } + CompareArchivesByPath(modified, scratchPath); + } + + [Theory] + [InlineData(10)] + [InlineData(128)] + public async ValueTask Tar_Japanese_Name_Async(int length) + { + using var mstm = new MemoryStream(); + var enc = new ArchiveEncoding { Default = Encoding.UTF8 }; + var twopt = new TarWriterOptions(CompressionType.None, true) { ArchiveEncoding = enc }; + var fname = new string((char)0x3042, length); + using (var tw = new TarWriter(mstm, twopt)) + using (var input = new MemoryStream(new byte[32])) + { + await tw.WriteAsync(fname, input, null); + } + using (var inputMemory = new MemoryStream(mstm.ToArray())) + { + var tropt = ReaderOptions.ForExternalStream.WithArchiveEncoding(enc); + await using ( + var tr = await ReaderFactory.OpenAsyncReader( + new AsyncOnlyStream(inputMemory), + tropt + ) + ) + { + while (await tr.MoveToNextEntryAsync()) + { + Assert.Equal(fname, tr.Entry.Key); + } + } + } + } + + [Fact] + public async ValueTask Tar_Read_One_At_A_Time_Async() + { + var archiveEncoding = new ArchiveEncoding { Default = Encoding.UTF8 }; + var tarWriterOptions = new TarWriterOptions(CompressionType.None, true) + { + ArchiveEncoding = archiveEncoding, + }; + var testBytes = Encoding.UTF8.GetBytes("This is a test."); + + using var memoryStream = new MemoryStream(); + using (var tarWriter = new TarWriter(memoryStream, tarWriterOptions)) + using (var testFileStream = new MemoryStream(testBytes)) + { + await tarWriter.WriteAsync("test1.txt", testFileStream, null); + testFileStream.Position = 0; + await tarWriter.WriteAsync("test2.txt", testFileStream, null); + } + + memoryStream.Position = 0; + + var numberOfEntries = 0; + + await using ( + var archiveFactory = await ArchiveFactory.OpenAsyncArchive( + new AsyncOnlyStream(memoryStream) + ) + ) + { + await foreach (var entry in archiveFactory.EntriesAsync) + { + ++numberOfEntries; + +#if LEGACY_DOTNET + using var tarEntryStream = await entry.OpenEntryStreamAsync(); +#else + await using var tarEntryStream = await entry.OpenEntryStreamAsync(); +#endif + using var testFileStream = new MemoryStream(); + await tarEntryStream.CopyToAsync(testFileStream); + Assert.Equal(testBytes.Length, testFileStream.Length); + } + } + + Assert.Equal(2, numberOfEntries); + } + + [Fact] + public async ValueTask Tar_PaxLocalHeader_Archive_Async() + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.PaxLocalHeader.tar"); + await using var archive = await TarArchive.OpenAsyncArchive( + new AsyncOnlyStream(File.OpenRead(archivePath)) + ); + + var firstEntry = (TarArchiveEntry) + await archive.EntriesAsync.SingleAsync(entry => entry.Key == "pax/overridden-name.txt"); + Assert.Equal(10, firstEntry.Size); + Assert.Equal(1234, firstEntry.UserID); + Assert.Equal(2345, firstEntry.GroupId); + Assert.Equal(Convert.ToInt64("640", 8), firstEntry.Mode); + + var expectedTime = DateTimeOffset.FromUnixTimeSeconds(1700000000).LocalDateTime; + Assert.Equal(expectedTime, firstEntry.LastModifiedTime); + + var secondEntry = (TarArchiveEntry) + await archive.EntriesAsync.SingleAsync(entry => entry.Key == "second.txt"); + Assert.Equal(2, secondEntry.Size); + Assert.Equal(11, secondEntry.UserID); + Assert.Equal(22, secondEntry.GroupId); + Assert.Equal(Convert.ToInt64("644", 8), secondEntry.Mode); + } + + [Fact] + public async ValueTask Tar_PaxLocalHeader_Link_Archive_Async() + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.PaxLocalHeader.Link.tar"); + await using var archive = await TarArchive.OpenAsyncArchive( + new AsyncOnlyStream(File.OpenRead(archivePath)) + ); + + var entry = (TarArchiveEntry)await archive.EntriesAsync.SingleAsync(); + Assert.Equal("pax/link-entry", entry.Key); + Assert.Equal("pax/target-entry", entry.LinkTarget); + } + + [Fact] + public async ValueTask Tar_PaxGlobalHeader_Archive_Async() + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.PaxGlobalHeader.tar"); + await using var archive = await TarArchive.OpenAsyncArchive( + new AsyncOnlyStream(File.OpenRead(archivePath)) + ); + + var globalTime = DateTimeOffset.FromUnixTimeSeconds(1700000100).LocalDateTime; + var localOverrideTime = DateTimeOffset.FromUnixTimeSeconds(1700000200).LocalDateTime; + + var firstEntry = (TarArchiveEntry) + await archive.EntriesAsync.SingleAsync(entry => entry.Key == "global-one.txt"); + Assert.Equal(4000, firstEntry.UserID); + Assert.Equal(5000, firstEntry.GroupId); + Assert.Equal(Convert.ToInt64("640", 8), firstEntry.Mode); + Assert.Equal(globalTime, firstEntry.LastModifiedTime); + + var secondEntry = (TarArchiveEntry) + await archive.EntriesAsync.SingleAsync(entry => + entry.Key == "global-local-override.txt" + ); + Assert.Equal(4010, secondEntry.UserID); + Assert.Equal(5010, secondEntry.GroupId); + Assert.Equal(Convert.ToInt64("600", 8), secondEntry.Mode); + Assert.Equal(localOverrideTime, secondEntry.LastModifiedTime); + + var thirdEntry = (TarArchiveEntry) + await archive.EntriesAsync.SingleAsync(entry => entry.Key == "global-three.txt"); + Assert.Equal(4000, thirdEntry.UserID); + Assert.Equal(5000, thirdEntry.GroupId); + Assert.Equal(Convert.ToInt64("640", 8), thirdEntry.Mode); + Assert.Equal(globalTime, thirdEntry.LastModifiedTime); + } + + [Fact] + public async ValueTask Tar_PaxGlobalHeader_Link_Archive_Async() + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.PaxGlobalHeader.Link.tar"); + await using var archive = await TarArchive.OpenAsyncArchive( + new AsyncOnlyStream(File.OpenRead(archivePath)) + ); + + var globalLink = (TarArchiveEntry) + await archive.EntriesAsync.SingleAsync(entry => entry.Key == "global-link"); + Assert.Equal("global-target", globalLink.LinkTarget); + Assert.Equal(4100, globalLink.UserID); + Assert.Equal(5100, globalLink.GroupId); + Assert.Equal(Convert.ToInt64("777", 8), globalLink.Mode); + + var localOverrideLink = (TarArchiveEntry) + await archive.EntriesAsync.SingleAsync(entry => entry.Key == "local-link-override"); + Assert.Equal("local-target", localOverrideLink.LinkTarget); + Assert.Equal(4100, localOverrideLink.UserID); + Assert.Equal(5100, localOverrideLink.GroupId); + Assert.Equal(Convert.ToInt64("777", 8), localOverrideLink.Mode); + } + + [Fact] + public async ValueTask Tar_Read_One_At_A_Time_Without_Disposing_Entry_Stream_Async() + { + var archiveEncoding = new ArchiveEncoding { Default = Encoding.UTF8 }; + var tarWriterOptions = new TarWriterOptions(CompressionType.None, true) + { + ArchiveEncoding = archiveEncoding, + }; + var testBytes = Encoding.UTF8.GetBytes("This is a test."); + + using var memoryStream = new MemoryStream(); + using (var tarWriter = new TarWriter(memoryStream, tarWriterOptions)) + using (var testFileStream = new MemoryStream(testBytes)) + { + await tarWriter.WriteAsync("file0.txt", testFileStream, null); + testFileStream.Position = 0; + await tarWriter.WriteAsync("file1.txt", testFileStream, null); + tarWriter.WriteDirectory("folder0", null); + testFileStream.Position = 0; + await tarWriter.WriteAsync("folder0/file_in_folder0.txt", testFileStream, null); + } + + memoryStream.Position = 0; + + var entryKeys = new List(); + var openEntryStreams = new List(); + + await using ( + var archive = await TarArchive.OpenAsyncArchive( + new AsyncOnlyStream(memoryStream), + ReaderOptions.ForExternalStream + ) + ) + { + await foreach (var entry in archive.EntriesAsync) + { + entryKeys.Add(entry.Key); + if (entry.IsDirectory) + { + continue; + } + + var tarEntryStream = await entry.OpenEntryStreamAsync(); + openEntryStreams.Add(tarEntryStream); + + using var testFileStream = new MemoryStream(); + await tarEntryStream.CopyToAsync(testFileStream); + Assert.Equal(testBytes.Length, testFileStream.Length); + } + + Assert.Equal(4, await archive.EntriesAsync.CountAsync()); + } + + openEntryStreams.ForEach(stream => stream.Dispose()); + + Assert.Equal( + ["file0.txt", "file1.txt", "folder0/", "folder0/file_in_folder0.txt"], + entryKeys + ); + } +} diff --git a/tests/SharpCompress.Test/Tar/TarArchiveDirectoryTests.cs b/tests/SharpCompress.Test/Tar/TarArchiveDirectoryTests.cs new file mode 100644 index 00000000..08120d13 --- /dev/null +++ b/tests/SharpCompress.Test/Tar/TarArchiveDirectoryTests.cs @@ -0,0 +1,113 @@ +using System; +using System.IO; +using System.Linq; +using SharpCompress.Archives.Tar; +using SharpCompress.Common; +using SharpCompress.Writers.Tar; +using Xunit; + +namespace SharpCompress.Test.Tar; + +public class TarArchiveDirectoryTests : TestBase +{ + [Fact] + public void TarArchive_AddDirectoryEntry_CreatesDirectoryEntry() + { + using var archive = TarArchive.CreateArchive(); + + archive.AddDirectoryEntry("test-dir", DateTime.Now); + + var entries = archive.Entries.ToList(); + Assert.Single(entries); + Assert.Equal("test-dir", entries[0].Key); + Assert.True(entries[0].IsDirectory); + } + + [Fact] + public void TarArchive_AddDirectoryEntry_MultipleDirectories() + { + using var archive = TarArchive.CreateArchive(); + + archive.AddDirectoryEntry("dir1", DateTime.Now); + archive.AddDirectoryEntry("dir2", DateTime.Now); + archive.AddDirectoryEntry("dir1/subdir", DateTime.Now); + + var entries = archive.Entries.OrderBy(e => e.Key).ToList(); + Assert.Equal(3, entries.Count); + Assert.True(entries.All(e => e.IsDirectory)); + } + + [Fact] + public void TarArchive_AddDirectoryEntry_MixedWithFiles() + { + using var archive = TarArchive.CreateArchive(); + + archive.AddDirectoryEntry("dir1", DateTime.Now); + + using var contentStream = new MemoryStream( + System.Text.Encoding.UTF8.GetBytes("test content") + ); + archive.AddEntry("dir1/file.txt", contentStream, false, contentStream.Length, DateTime.Now); + + archive.AddDirectoryEntry("dir2", DateTime.Now); + + var entries = archive.Entries.OrderBy(e => e.Key).ToList(); + Assert.Equal(3, entries.Count); + Assert.True(entries[0].IsDirectory); + Assert.False(entries[1].IsDirectory); + Assert.True(entries[2].IsDirectory); + } + + [Fact] + public void TarArchive_AddDirectoryEntry_SaveAndReload() + { + var scratchPath = Path.Combine(SCRATCH_FILES_PATH, "tar-directory-test.tar"); + + using (var archive = TarArchive.CreateArchive()) + { + archive.AddDirectoryEntry("dir1", DateTime.Now); + archive.AddDirectoryEntry("dir2", DateTime.Now); + + using var contentStream = new MemoryStream( + System.Text.Encoding.UTF8.GetBytes("test content") + ); + archive.AddEntry( + "dir1/file.txt", + contentStream, + false, + contentStream.Length, + DateTime.Now + ); + + using (var fileStream = File.Create(scratchPath)) + { + archive.SaveTo(fileStream, new TarWriterOptions(CompressionType.None, true)); + } + } + + using (var archive = TarArchive.OpenArchive(scratchPath)) + { + var entries = archive.Entries.OrderBy(e => e.Key).ToList(); + Assert.Equal(3, entries.Count); + + Assert.Equal("dir1/", entries[0].Key); + Assert.True(entries[0].IsDirectory); + + Assert.Equal("dir1/file.txt", entries[1].Key); + Assert.False(entries[1].IsDirectory); + + Assert.Equal("dir2/", entries[2].Key); + Assert.True(entries[2].IsDirectory); + } + } + + [Fact] + public void TarArchive_AddDirectoryEntry_DuplicateKey_ThrowsException() + { + using var archive = TarArchive.CreateArchive(); + + archive.AddDirectoryEntry("test-dir", DateTime.Now); + + Assert.Throws(() => archive.AddDirectoryEntry("test-dir", DateTime.Now)); + } +} diff --git a/tests/SharpCompress.Test/Tar/TarArchiveTests.cs b/tests/SharpCompress.Test/Tar/TarArchiveTests.cs index dddb32a6..7959967e 100644 --- a/tests/SharpCompress.Test/Tar/TarArchiveTests.cs +++ b/tests/SharpCompress.Test/Tar/TarArchiveTests.cs @@ -1,15 +1,17 @@ using System; +using System.Collections.Generic; using System.IO; using System.Linq; +using System.Text; using SharpCompress.Archives; using SharpCompress.Archives.Tar; using SharpCompress.Common; -using SharpCompress.Writers; -using Xunit; -using System.Text; using SharpCompress.Readers; -using SharpCompress.Writers.Tar; using SharpCompress.Readers.Tar; +using SharpCompress.Test.Mocks; +using SharpCompress.Writers; +using SharpCompress.Writers.Tar; +using Xunit; namespace SharpCompress.Test.Tar; @@ -23,21 +25,59 @@ public class TarArchiveTests : ArchiveTests [Fact] public void TarArchivePathRead() => ArchiveFileRead("Tar.tar"); + [Fact] + public void TarArchiveStreamRead_Throws_On_NonSeekable_Stream() + { + using Stream stream = new ForwardOnlyStream( + File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar")) + ); + + Assert.Throws(() => ArchiveFactory.OpenArchive(stream)); + } + + [Fact] + public void TarArchiveStreamRead_Throws_On_Unreadable_Stream() + { + using var unreadable = new TestStream( + File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar")), + false, + true, + true + ); + + Assert.Throws(() => TarArchive.OpenArchive(unreadable)); + } + + [Fact] + public void TarArchive_StreamCollection_Throws_On_NonSeekable_Stream() + { + using var nonSeekable = new ForwardOnlyStream(new MemoryStream()); + using var seekable = new MemoryStream(); + + Assert.Throws(() => TarArchive.OpenArchive([nonSeekable, seekable])); + } + [Fact] public void Tar_FileName_Exactly_100_Characters() { - string archive = "Tar_FileName_Exactly_100_Characters.tar"; + var archive = "Tar_FileName_Exactly_100_Characters.tar"; // create the 100 char filename - string filename = + var filename = "filename_with_exactly_100_characters_______________________________________________________________X"; // Step 1: create a tar file containing a file with the test name using (Stream stream = File.OpenWrite(Path.Combine(SCRATCH2_FILES_PATH, archive))) - using (var writer = WriterFactory.Open(stream, ArchiveType.Tar, CompressionType.None)) + using ( + var writer = WriterFactory.OpenWriter( + stream, + ArchiveType.Tar, + new WriterOptions(CompressionType.None) + ) + ) using (Stream inputStream = new MemoryStream()) { - StreamWriter sw = new StreamWriter(inputStream); + var sw = new StreamWriter(inputStream); sw.Write("dummy filecontent"); sw.Flush(); @@ -46,18 +86,18 @@ public class TarArchiveTests : ArchiveTests } // Step 2: check if the written tar file can be read correctly - string unmodified = Path.Combine(SCRATCH2_FILES_PATH, archive); - using (var archive2 = TarArchive.Open(unmodified)) + var unmodified = Path.Combine(SCRATCH2_FILES_PATH, archive); + using (var archive2 = ArchiveFactory.OpenArchive(unmodified)) { - Assert.Equal(1, archive2.Entries.Count); + Assert.Equal(1, archive2.Entries.Count()); Assert.Contains(filename, archive2.Entries.Select(entry => entry.Key)); foreach (var entry in archive2.Entries) { - Assert.Equal( - "dummy filecontent", - new StreamReader(entry.OpenEntryStream()).ReadLine() - ); + using (var sr = new StreamReader(entry.OpenEntryStream())) + { + Assert.Equal("dummy filecontent", sr.ReadLine()); + } } } } @@ -65,29 +105,27 @@ public class TarArchiveTests : ArchiveTests [Fact] public void Tar_NonUstarArchiveWithLongNameDoesNotSkipEntriesAfterTheLongOne() { - string unmodified = Path.Combine(TEST_ARCHIVES_PATH, "very long filename.tar"); - using (var archive = TarArchive.Open(unmodified)) - { - Assert.Equal(5, archive.Entries.Count); - Assert.Contains("very long filename/", archive.Entries.Select(entry => entry.Key)); - Assert.Contains( - "very long filename/very long filename very long filename very long filename very long filename very long filename very long filename very long filename very long filename very long filename very long filename.jpg", - archive.Entries.Select(entry => entry.Key) - ); - Assert.Contains("z_file 1.txt", archive.Entries.Select(entry => entry.Key)); - Assert.Contains("z_file 2.txt", archive.Entries.Select(entry => entry.Key)); - Assert.Contains("z_file 3.txt", archive.Entries.Select(entry => entry.Key)); - } + var unmodified = Path.Combine(TEST_ARCHIVES_PATH, "very long filename.tar"); + using var archive = ArchiveFactory.OpenArchive(unmodified); + Assert.Equal(5, archive.Entries.Count()); + Assert.Contains("very long filename/", archive.Entries.Select(entry => entry.Key)); + Assert.Contains( + "very long filename/very long filename very long filename very long filename very long filename very long filename very long filename very long filename very long filename very long filename very long filename.jpg", + archive.Entries.Select(entry => entry.Key) + ); + Assert.Contains("z_file 1.txt", archive.Entries.Select(entry => entry.Key)); + Assert.Contains("z_file 2.txt", archive.Entries.Select(entry => entry.Key)); + Assert.Contains("z_file 3.txt", archive.Entries.Select(entry => entry.Key)); } [Fact] public void Tar_VeryLongFilepathReadback() { - string archive = "Tar_VeryLongFilepathReadback.tar"; + var archive = "Tar_VeryLongFilepathReadback.tar"; // create a very long filename - string longFilename = ""; - for (int i = 0; i < 600; i = longFilename.Length) + var longFilename = ""; + for (var i = 0; i < 600; i = longFilename.Length) { longFilename += i.ToString("D10") + "-"; } @@ -96,10 +134,16 @@ public class TarArchiveTests : ArchiveTests // Step 1: create a tar file containing a file with a long name using (Stream stream = File.OpenWrite(Path.Combine(SCRATCH2_FILES_PATH, archive))) - using (var writer = WriterFactory.Open(stream, ArchiveType.Tar, CompressionType.None)) + using ( + var writer = WriterFactory.OpenWriter( + stream, + ArchiveType.Tar, + new WriterOptions(CompressionType.None) + ) + ) using (Stream inputStream = new MemoryStream()) { - StreamWriter sw = new StreamWriter(inputStream); + var sw = new StreamWriter(inputStream); sw.Write("dummy filecontent"); sw.Flush(); @@ -108,18 +152,18 @@ public class TarArchiveTests : ArchiveTests } // Step 2: check if the written tar file can be read correctly - string unmodified = Path.Combine(SCRATCH2_FILES_PATH, archive); - using (var archive2 = TarArchive.Open(unmodified)) + var unmodified = Path.Combine(SCRATCH2_FILES_PATH, archive); + using (var archive2 = ArchiveFactory.OpenArchive(unmodified)) { - Assert.Equal(1, archive2.Entries.Count); + Assert.Equal(1, archive2.Entries.Count()); Assert.Contains(longFilename, archive2.Entries.Select(entry => entry.Key)); foreach (var entry in archive2.Entries) { - Assert.Equal( - "dummy filecontent", - new StreamReader(entry.OpenEntryStream()).ReadLine() - ); + using (var sr = new StreamReader(entry.OpenEntryStream())) + { + Assert.Equal("dummy filecontent", sr.ReadLine()); + } } } } @@ -127,47 +171,134 @@ public class TarArchiveTests : ArchiveTests [Fact] public void Tar_UstarArchivePathReadLongName() { - string unmodified = Path.Combine(TEST_ARCHIVES_PATH, "ustar with long names.tar"); - using (var archive = TarArchive.Open(unmodified)) - { - Assert.Equal(6, archive.Entries.Count); - Assert.Contains("Directory/", archive.Entries.Select(entry => entry.Key)); - Assert.Contains( - "Directory/Some file with veeeeeeeeeery loooooooooong name", - archive.Entries.Select(entry => entry.Key) - ); - Assert.Contains( - "Directory/Directory with veeeeeeeeeery loooooooooong name/", - archive.Entries.Select(entry => entry.Key) - ); - Assert.Contains( - "Directory/Directory with veeeeeeeeeery loooooooooong name/Some file with veeeeeeeeeery loooooooooong name", - archive.Entries.Select(entry => entry.Key) - ); - Assert.Contains( - "Directory/Directory with veeeeeeeeeery loooooooooong name/Directory with veeeeeeeeeery loooooooooong name/", - archive.Entries.Select(entry => entry.Key) - ); - Assert.Contains( - "Directory/Directory with veeeeeeeeeery loooooooooong name/Directory with veeeeeeeeeery loooooooooong name/Some file with veeeeeeeeeery loooooooooong name", - archive.Entries.Select(entry => entry.Key) - ); - } + var unmodified = Path.Combine(TEST_ARCHIVES_PATH, "ustar with long names.tar"); + using var archive = ArchiveFactory.OpenArchive(unmodified); + Assert.Equal(6, archive.Entries.Count()); + Assert.Contains("Directory/", archive.Entries.Select(entry => entry.Key)); + Assert.Contains( + "Directory/Some file with veeeeeeeeeery loooooooooong name", + archive.Entries.Select(entry => entry.Key) + ); + Assert.Contains( + "Directory/Directory with veeeeeeeeeery loooooooooong name/", + archive.Entries.Select(entry => entry.Key) + ); + Assert.Contains( + "Directory/Directory with veeeeeeeeeery loooooooooong name/Some file with veeeeeeeeeery loooooooooong name", + archive.Entries.Select(entry => entry.Key) + ); + Assert.Contains( + "Directory/Directory with veeeeeeeeeery loooooooooong name/Directory with veeeeeeeeeery loooooooooong name/", + archive.Entries.Select(entry => entry.Key) + ); + Assert.Contains( + "Directory/Directory with veeeeeeeeeery loooooooooong name/Directory with veeeeeeeeeery loooooooooong name/Some file with veeeeeeeeeery loooooooooong name", + archive.Entries.Select(entry => entry.Key) + ); + } + + [Fact] + public void Tar_PaxLocalHeader_Archive() + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.PaxLocalHeader.tar"); + using var archive = TarArchive.OpenArchive(archivePath); + + var firstEntry = (TarArchiveEntry) + archive.Entries.Single(entry => entry.Key == "pax/overridden-name.txt"); + Assert.Equal(10, firstEntry.Size); + Assert.Equal(1234, firstEntry.UserID); + Assert.Equal(2345, firstEntry.GroupId); + Assert.Equal(Convert.ToInt64("640", 8), firstEntry.Mode); + + var expectedTime = DateTimeOffset.FromUnixTimeSeconds(1700000000).LocalDateTime; + Assert.Equal(expectedTime, firstEntry.LastModifiedTime); + + var secondEntry = (TarArchiveEntry) + archive.Entries.Single(entry => entry.Key == "second.txt"); + Assert.Equal(2, secondEntry.Size); + Assert.Equal(11, secondEntry.UserID); + Assert.Equal(22, secondEntry.GroupId); + Assert.Equal(Convert.ToInt64("644", 8), secondEntry.Mode); + } + + [Fact] + public void Tar_PaxLocalHeader_Link_Archive() + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.PaxLocalHeader.Link.tar"); + using var archive = TarArchive.OpenArchive(archivePath); + + var entry = (TarArchiveEntry)archive.Entries.Single(); + Assert.Equal("pax/link-entry", entry.Key); + Assert.Equal("pax/target-entry", entry.LinkTarget); + } + + [Fact] + public void Tar_PaxGlobalHeader_Archive() + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.PaxGlobalHeader.tar"); + using var archive = TarArchive.OpenArchive(archivePath); + + var globalTime = DateTimeOffset.FromUnixTimeSeconds(1700000100).LocalDateTime; + var localOverrideTime = DateTimeOffset.FromUnixTimeSeconds(1700000200).LocalDateTime; + + var firstEntry = (TarArchiveEntry) + archive.Entries.Single(entry => entry.Key == "global-one.txt"); + Assert.Equal(4000, firstEntry.UserID); + Assert.Equal(5000, firstEntry.GroupId); + Assert.Equal(Convert.ToInt64("640", 8), firstEntry.Mode); + Assert.Equal(globalTime, firstEntry.LastModifiedTime); + + var secondEntry = (TarArchiveEntry) + archive.Entries.Single(entry => entry.Key == "global-local-override.txt"); + Assert.Equal(4010, secondEntry.UserID); + Assert.Equal(5010, secondEntry.GroupId); + Assert.Equal(Convert.ToInt64("600", 8), secondEntry.Mode); + Assert.Equal(localOverrideTime, secondEntry.LastModifiedTime); + + var thirdEntry = (TarArchiveEntry) + archive.Entries.Single(entry => entry.Key == "global-three.txt"); + Assert.Equal(4000, thirdEntry.UserID); + Assert.Equal(5000, thirdEntry.GroupId); + Assert.Equal(Convert.ToInt64("640", 8), thirdEntry.Mode); + Assert.Equal(globalTime, thirdEntry.LastModifiedTime); + } + + [Fact] + public void Tar_PaxGlobalHeader_Link_Archive() + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.PaxGlobalHeader.Link.tar"); + using var archive = TarArchive.OpenArchive(archivePath); + + var globalLink = (TarArchiveEntry) + archive.Entries.Single(entry => entry.Key == "global-link"); + Assert.Equal("global-target", globalLink.LinkTarget); + Assert.Equal(4100, globalLink.UserID); + Assert.Equal(5100, globalLink.GroupId); + Assert.Equal(Convert.ToInt64("777", 8), globalLink.Mode); + + var localOverrideLink = (TarArchiveEntry) + archive.Entries.Single(entry => entry.Key == "local-link-override"); + Assert.Equal("local-target", localOverrideLink.LinkTarget); + Assert.Equal(4100, localOverrideLink.UserID); + Assert.Equal(5100, localOverrideLink.GroupId); + Assert.Equal(Convert.ToInt64("777", 8), localOverrideLink.Mode); } [Fact] public void Tar_Create_New() { - string scratchPath = Path.Combine(SCRATCH_FILES_PATH, "Tar.tar"); - string unmodified = Path.Combine(TEST_ARCHIVES_PATH, "Tar.noEmptyDirs.tar"); + var scratchPath = Path.Combine(SCRATCH_FILES_PATH, "Tar.tar"); + var unmodified = Path.Combine(TEST_ARCHIVES_PATH, "Tar.noEmptyDirs.tar"); // var aropt = new Ar - using (var archive = TarArchive.Create()) + using (var archive = TarArchive.CreateArchive()) { archive.AddAllFromDirectory(ORIGINAL_FILES_PATH); - var twopt = new TarWriterOptions(CompressionType.None, true); - twopt.ArchiveEncoding = new ArchiveEncoding() { Default = Encoding.GetEncoding(866) }; + var twopt = new TarWriterOptions(CompressionType.None, true) + { + ArchiveEncoding = new ArchiveEncoding { Default = Encoding.GetEncoding(866) }, + }; archive.SaveTo(scratchPath, twopt); } CompareArchivesByPath(unmodified, scratchPath); @@ -176,15 +307,15 @@ public class TarArchiveTests : ArchiveTests [Fact] public void Tar_Random_Write_Add() { - string jpg = Path.Combine(ORIGINAL_FILES_PATH, "jpg", "test.jpg"); - string scratchPath = Path.Combine(SCRATCH_FILES_PATH, "Tar.mod.tar"); - string unmodified = Path.Combine(TEST_ARCHIVES_PATH, "Tar.mod.tar"); - string modified = Path.Combine(TEST_ARCHIVES_PATH, "Tar.noEmptyDirs.tar"); + var jpg = Path.Combine(ORIGINAL_FILES_PATH, "jpg", "test.jpg"); + var scratchPath = Path.Combine(SCRATCH_FILES_PATH, "Tar.mod.tar"); + var unmodified = Path.Combine(TEST_ARCHIVES_PATH, "Tar.mod.tar"); + var modified = Path.Combine(TEST_ARCHIVES_PATH, "Tar.noEmptyDirs.tar"); - using (var archive = TarArchive.Open(unmodified)) + using (var archive = TarArchive.OpenArchive(unmodified)) { archive.AddEntry("jpg\\test.jpg", jpg); - archive.SaveTo(scratchPath, CompressionType.None); + archive.SaveTo(scratchPath, new TarWriterOptions(CompressionType.None, true)); } CompareArchivesByPath(modified, scratchPath); } @@ -192,17 +323,17 @@ public class TarArchiveTests : ArchiveTests [Fact] public void Tar_Random_Write_Remove() { - string scratchPath = Path.Combine(SCRATCH_FILES_PATH, "Tar.mod.tar"); - string modified = Path.Combine(TEST_ARCHIVES_PATH, "Tar.mod.tar"); - string unmodified = Path.Combine(TEST_ARCHIVES_PATH, "Tar.noEmptyDirs.tar"); + var scratchPath = Path.Combine(SCRATCH_FILES_PATH, "Tar.mod.tar"); + var modified = Path.Combine(TEST_ARCHIVES_PATH, "Tar.mod.tar"); + var unmodified = Path.Combine(TEST_ARCHIVES_PATH, "Tar.noEmptyDirs.tar"); - using (var archive = TarArchive.Open(unmodified)) + using (var archive = TarArchive.OpenArchive(unmodified)) { - var entry = archive.Entries.Single( - x => x.Key.EndsWith("jpg", StringComparison.OrdinalIgnoreCase) + var entry = archive.Entries.Single(x => + x.Key.NotNull().EndsWith("jpg", StringComparison.OrdinalIgnoreCase) ); archive.RemoveEntry(entry); - archive.SaveTo(scratchPath, CompressionType.None); + archive.SaveTo(scratchPath, new TarWriterOptions(CompressionType.None, true)); } CompareArchivesByPath(modified, scratchPath); } @@ -210,23 +341,19 @@ public class TarArchiveTests : ArchiveTests [Fact] public void Tar_Containing_Rar_Archive() { - string archiveFullPath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.ContainsRar.tar"); - using (Stream stream = File.OpenRead(archiveFullPath)) - using (IArchive archive = ArchiveFactory.Open(stream)) - { - Assert.True(archive.Type == ArchiveType.Tar); - } + var archiveFullPath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.ContainsRar.tar"); + using Stream stream = File.OpenRead(archiveFullPath); + using var archive = ArchiveFactory.OpenArchive(stream); + Assert.True(archive.Type == ArchiveType.Tar); } [Fact] public void Tar_Empty_Archive() { - string archiveFullPath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.Empty.tar"); - using (Stream stream = File.OpenRead(archiveFullPath)) - using (IArchive archive = ArchiveFactory.Open(stream)) - { - Assert.True(archive.Type == ArchiveType.Tar); - } + var archiveFullPath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.Empty.tar"); + using Stream stream = File.OpenRead(archiveFullPath); + using var archive = ArchiveFactory.OpenArchive(stream); + Assert.True(archive.Type == ArchiveType.Tar); } [Theory] @@ -234,26 +361,23 @@ public class TarArchiveTests : ArchiveTests [InlineData(128)] public void Tar_Japanese_Name(int length) { - using (var mstm = new MemoryStream()) + using var mstm = new MemoryStream(); + var enc = new ArchiveEncoding { Default = Encoding.UTF8 }; + var twopt = new TarWriterOptions(CompressionType.None, true) { ArchiveEncoding = enc }; + var fname = new string((char)0x3042, length); + using (var tw = new TarWriter(mstm, twopt)) + using (var input = new MemoryStream(new byte[32])) { - var enc = new ArchiveEncoding() { Default = Encoding.UTF8 }; - var twopt = new TarWriterOptions(CompressionType.None, true); - twopt.ArchiveEncoding = enc; - var fname = new string((char)0x3042, length); - using (var tw = new TarWriter(mstm, twopt)) - using (var input = new MemoryStream(new byte[32])) + tw.Write(fname, input, null); + } + using (var inputMemory = new MemoryStream(mstm.ToArray())) + { + var tropt = ReaderOptions.ForExternalStream.WithArchiveEncoding(enc); + using (var tr = TarReader.OpenReader(inputMemory, tropt)) { - tw.Write(fname, input, null); - } - using (var inputMemory = new MemoryStream(mstm.ToArray())) - { - var tropt = new ReaderOptions() { ArchiveEncoding = enc }; - using (var tr = TarReader.Open(inputMemory, tropt)) + while (tr.MoveToNextEntry()) { - while (tr.MoveToNextEntry()) - { - Assert.Equal(fname, tr.Entry.Key); - } + Assert.Equal(fname, tr.Entry.Key); } } } @@ -262,43 +386,123 @@ public class TarArchiveTests : ArchiveTests [Fact] public void Tar_Read_One_At_A_Time() { - var archiveEncoding = new ArchiveEncoding { Default = Encoding.UTF8, }; + var archiveEncoding = new ArchiveEncoding { Default = Encoding.UTF8 }; var tarWriterOptions = new TarWriterOptions(CompressionType.None, true) { ArchiveEncoding = archiveEncoding, }; var testBytes = Encoding.UTF8.GetBytes("This is a test."); - using (var memoryStream = new MemoryStream()) + using var memoryStream = new MemoryStream(); + using (var tarWriter = new TarWriter(memoryStream, tarWriterOptions)) + using (var testFileStream = new MemoryStream(testBytes)) { - using (var tarWriter = new TarWriter(memoryStream, tarWriterOptions)) - using (var testFileStream = new MemoryStream(testBytes)) - { - tarWriter.Write("test1.txt", testFileStream); - testFileStream.Position = 0; - tarWriter.Write("test2.txt", testFileStream); - } - - memoryStream.Position = 0; - - var numberOfEntries = 0; - - using (var archiveFactory = TarArchive.Open(memoryStream)) - { - foreach (var entry in archiveFactory.Entries) - { - ++numberOfEntries; - - using (var tarEntryStream = entry.OpenEntryStream()) - using (var testFileStream = new MemoryStream()) - { - tarEntryStream.CopyTo(testFileStream); - Assert.Equal(testBytes.Length, testFileStream.Length); - } - } - } - - Assert.Equal(2, numberOfEntries); + tarWriter.Write("test1.txt", testFileStream); + testFileStream.Position = 0; + tarWriter.Write("test2.txt", testFileStream); } + + memoryStream.Position = 0; + + var numberOfEntries = 0; + + using (var archive = ArchiveFactory.OpenArchive(memoryStream)) + { + foreach (var entry in archive.Entries) + { + ++numberOfEntries; + + using var tarEntryStream = entry.OpenEntryStream(); + using var testFileStream = new MemoryStream(); + tarEntryStream.CopyTo(testFileStream); + Assert.Equal(testBytes.Length, testFileStream.Length); + } + } + + Assert.Equal(2, numberOfEntries); + } + + [Fact] + public void Tar_Read_One_At_A_Time_Without_Disposing_Entry_Stream() + { + var archiveEncoding = new ArchiveEncoding { Default = Encoding.UTF8 }; + var tarWriterOptions = new TarWriterOptions(CompressionType.None, true) + { + ArchiveEncoding = archiveEncoding, + }; + var testBytes = Encoding.UTF8.GetBytes("This is a test."); + + using var memoryStream = new MemoryStream(); + using (var tarWriter = new TarWriter(memoryStream, tarWriterOptions)) + using (var testFileStream = new MemoryStream(testBytes)) + { + tarWriter.Write("file0.txt", testFileStream); + testFileStream.Position = 0; + tarWriter.Write("file1.txt", testFileStream); + tarWriter.WriteDirectory("folder0", null); + testFileStream.Position = 0; + tarWriter.Write("folder0/file_in_folder0.txt", testFileStream); + } + + memoryStream.Position = 0; + + var entryKeys = new List(); + var openEntryStreams = new List(); + + using (var archive = ArchiveFactory.OpenArchive(memoryStream)) + { + foreach (var entry in archive.Entries) + { + entryKeys.Add(entry.Key); + if (entry.IsDirectory) + { + continue; + } + + var tarEntryStream = entry.OpenEntryStream(); + openEntryStreams.Add(tarEntryStream); + + using var testFileStream = new MemoryStream(); + tarEntryStream.CopyTo(testFileStream); + Assert.Equal(testBytes.Length, testFileStream.Length); + } + + Assert.Equal(4, archive.Entries.Count()); + } + + openEntryStreams.ForEach(stream => stream.Dispose()); + + Assert.Equal( + ["file0.txt", "file1.txt", "folder0/", "folder0/file_in_folder0.txt"], + entryKeys + ); + } + + [Fact] + public void Tar_Detect_Test() + { + var isTar = TarArchive.IsTarFile(Path.Combine(TEST_ARCHIVES_PATH, "false.positive.tar")); + + Assert.False(isTar); + } + + [Fact] + public void TarArchiveStreamRead_Autodetect_CompressedTar() + { + using Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz")); + using var archive = ArchiveFactory.OpenArchive(stream); + + Assert.Equal(ArchiveType.Tar, archive.Type); + Assert.NotEmpty(archive.Entries); + } + + [Fact] + public void TarReaderStreamRead_Autodetect_CompressedTar() + { + using Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.gz")); + using var reader = ReaderFactory.OpenReader(stream); + + Assert.Equal(ArchiveType.Tar, reader.Type); + Assert.True(reader.MoveToNextEntry()); } } diff --git a/tests/SharpCompress.Test/Tar/TarReaderAsyncTests.cs b/tests/SharpCompress.Test/Tar/TarReaderAsyncTests.cs new file mode 100644 index 00000000..d0104cbb --- /dev/null +++ b/tests/SharpCompress.Test/Tar/TarReaderAsyncTests.cs @@ -0,0 +1,424 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Text; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Common.Tar; +using SharpCompress.Factories; +using SharpCompress.Readers; +using SharpCompress.Readers.Tar; +using SharpCompress.Test.Mocks; +using SharpCompress.Writers.Tar; +using Xunit; + +namespace SharpCompress.Test.Tar; + +public class TarReaderAsyncTests : ReaderTests +{ + public TarReaderAsyncTests() => UseExtensionInsteadOfNameToVerify = true; + + [Fact] + public async ValueTask Tar_Reader_Async() => await ReadAsync("Tar.tar", CompressionType.None); + + [Fact] + public async ValueTask Tar_Skip_Async() + { + using Stream stream = new ForwardOnlyStream( + File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar")) + ); + await using var reader = await ReaderFactory.OpenAsyncReader(new AsyncOnlyStream(stream)); + var x = 0; + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + x++; + if (x % 2 == 0) + { + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + } + } + + [Fact] + public async ValueTask Tar_Z_Reader_Async() => + await ReadAsync("Tar.tar.Z", CompressionType.Lzw); + + [Fact] + public async ValueTask Tar_Async_Assert() => await AssertArchiveAsync("Tar.tar"); + + [Fact] + public async ValueTask Tar_BZip2_Reader_Async() => + await ReadAsync("Tar.tar.bz2", CompressionType.BZip2); + + [Fact] + public async ValueTask Tar_GZip_Reader_Async() => + await ReadAsync("Tar.tar.gz", CompressionType.GZip); + + [Fact] + public async ValueTask Tar_ZStandard_Reader_Async() => + await ReadAsync("Tar.tar.zst", CompressionType.ZStandard); + + [Fact] + public async ValueTask Tar_LZip_Reader_Async() => + await ReadAsync("Tar.tar.lz", CompressionType.LZip); + + [Fact] + public async ValueTask Tar_Xz_Reader_Async() => + await ReadAsync("Tar.tar.xz", CompressionType.Xz); + + [Fact] + public async ValueTask Tar_GZip_OldGnu_Reader_Async() => + await ReadAsync("Tar.oldgnu.tar.gz", CompressionType.GZip); + + [Fact] + public async ValueTask Tar_BZip2_Entry_Stream_Async() + { + using Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.bz2")); + await using var reader = await TarReader.OpenAsyncReader(stream); + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + Assert.Equal(CompressionType.BZip2, reader.Entry.CompressionType); + using var entryStream = await reader.OpenEntryStreamAsync(); + var file = Path.GetFileName(reader.Entry.Key); + var folder = + Path.GetDirectoryName(reader.Entry.Key) ?? throw new ArgumentNullException(); + var destdir = Path.Combine(SCRATCH_FILES_PATH, folder); + if (!Directory.Exists(destdir)) + { + Directory.CreateDirectory(destdir); + } + var destinationFileName = Path.Combine(destdir, file.NotNull()); + + using var fs = File.OpenWrite(destinationFileName); + await entryStream.CopyToAsync(fs); + } + } + VerifyFiles(); + } + + [Fact] + public void Tar_LongNamesWithLongNameExtension_Async() + { + var filePaths = new List(); + + using ( + Stream stream = File.OpenRead( + Path.Combine(TEST_ARCHIVES_PATH, "Tar.LongPathsWithLongNameExtension.tar") + ) + ) + using (var reader = TarReader.OpenReader(stream)) + { + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + filePaths.Add(reader.Entry.Key.NotNull("Entry Key is null")); + } + } + } + + Assert.Equal(3, filePaths.Count); + Assert.Contains("a.txt", filePaths); + Assert.Contains( + "wp-content/plugins/gravityformsextend/lib/Aws/Symfony/Component/ClassLoader/Tests/Fixtures/Apc/beta/Apc/ApcPrefixCollision/A/B/Bar.php", + filePaths + ); + Assert.Contains( + "wp-content/plugins/gravityformsextend/lib/Aws/Symfony/Component/ClassLoader/Tests/Fixtures/Apc/beta/Apc/ApcPrefixCollision/A/B/Foo.php", + filePaths + ); + } + + [Fact] + public async ValueTask Tar_BZip2_Skip_Entry_Stream_Async() + { + using Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.bz2")); + await using var reader = await TarReader.OpenAsyncReader(stream); + var names = new List(); + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + Assert.Equal(CompressionType.BZip2, reader.Entry.CompressionType); + using var entryStream = await reader.OpenEntryStreamAsync(); + await entryStream.SkipEntryAsync(); + names.Add(reader.Entry.Key.NotNull()); + } + } + Assert.Equal(3, names.Count); + } + + [Fact] + public async ValueTask Tar_PaxLocalHeader_Reader_Async() + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.PaxLocalHeader.tar"); + + using Stream stream = File.OpenRead(archivePath); + await using var reader = await TarReader.OpenAsyncReader(stream); + + Assert.True(await reader.MoveToNextEntryAsync()); + var firstEntry = (TarEntry)reader.Entry; + Assert.Equal("pax/overridden-name.txt", firstEntry.Key); + Assert.Equal(10, firstEntry.Size); + Assert.Equal(1234, firstEntry.UserID); + Assert.Equal(2345, firstEntry.GroupId); + Assert.Equal(Convert.ToInt64("640", 8), firstEntry.Mode); + + var expectedTime = DateTimeOffset.FromUnixTimeSeconds(1700000000).LocalDateTime; + Assert.Equal(expectedTime, firstEntry.LastModifiedTime); + + using (var entryStream = await reader.OpenEntryStreamAsync()) + using (var memoryStream = new MemoryStream()) + { + await entryStream.CopyToAsync(memoryStream); + Assert.Equal(10, memoryStream.Length); + } + + Assert.True(await reader.MoveToNextEntryAsync()); + var secondEntry = (TarEntry)reader.Entry; + Assert.Equal("second.txt", secondEntry.Key); + Assert.Equal(11, secondEntry.UserID); + Assert.Equal(22, secondEntry.GroupId); + Assert.Equal(Convert.ToInt64("644", 8), secondEntry.Mode); + Assert.Equal(2, secondEntry.Size); + + Assert.False(await reader.MoveToNextEntryAsync()); + } + + [Fact] + public async ValueTask Tar_PaxLocalHeader_Link_Reader_Async() + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.PaxLocalHeader.Link.tar"); + + using Stream stream = File.OpenRead(archivePath); + await using var reader = await TarReader.OpenAsyncReader(stream); + + Assert.True(await reader.MoveToNextEntryAsync()); + Assert.Equal("pax/link-entry", reader.Entry.Key); + Assert.Equal("pax/target-entry", reader.Entry.LinkTarget); + Assert.False(reader.Entry.IsDirectory); + Assert.False(await reader.MoveToNextEntryAsync()); + } + + [Fact] + public async ValueTask Tar_PaxGlobalHeader_Reader_Async() + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.PaxGlobalHeader.tar"); + + using Stream stream = File.OpenRead(archivePath); + await using var reader = await TarReader.OpenAsyncReader(stream); + + var globalTime = DateTimeOffset.FromUnixTimeSeconds(1700000100).LocalDateTime; + var localOverrideTime = DateTimeOffset.FromUnixTimeSeconds(1700000200).LocalDateTime; + + Assert.True(await reader.MoveToNextEntryAsync()); + var firstEntry = (TarEntry)reader.Entry; + Assert.Equal("global-one.txt", firstEntry.Key); + Assert.Equal(4000, firstEntry.UserID); + Assert.Equal(5000, firstEntry.GroupId); + Assert.Equal(Convert.ToInt64("640", 8), firstEntry.Mode); + Assert.Equal(globalTime, firstEntry.LastModifiedTime); + + Assert.True(await reader.MoveToNextEntryAsync()); + var secondEntry = (TarEntry)reader.Entry; + Assert.Equal("global-local-override.txt", secondEntry.Key); + Assert.Equal(4010, secondEntry.UserID); + Assert.Equal(5010, secondEntry.GroupId); + Assert.Equal(Convert.ToInt64("600", 8), secondEntry.Mode); + Assert.Equal(localOverrideTime, secondEntry.LastModifiedTime); + + Assert.True(await reader.MoveToNextEntryAsync()); + var thirdEntry = (TarEntry)reader.Entry; + Assert.Equal("global-three.txt", thirdEntry.Key); + Assert.Equal(4000, thirdEntry.UserID); + Assert.Equal(5000, thirdEntry.GroupId); + Assert.Equal(Convert.ToInt64("640", 8), thirdEntry.Mode); + Assert.Equal(globalTime, thirdEntry.LastModifiedTime); + + Assert.False(await reader.MoveToNextEntryAsync()); + } + + [Fact] + public async ValueTask Tar_PaxGlobalHeader_Link_Reader_Async() + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.PaxGlobalHeader.Link.tar"); + + using Stream stream = File.OpenRead(archivePath); + await using var reader = await TarReader.OpenAsyncReader(stream); + + Assert.True(await reader.MoveToNextEntryAsync()); + var firstEntry = (TarEntry)reader.Entry; + Assert.Equal("global-link", firstEntry.Key); + Assert.Equal("global-target", firstEntry.LinkTarget); + Assert.Equal(4100, firstEntry.UserID); + Assert.Equal(5100, firstEntry.GroupId); + Assert.Equal(Convert.ToInt64("777", 8), firstEntry.Mode); + + Assert.True(await reader.MoveToNextEntryAsync()); + var secondEntry = (TarEntry)reader.Entry; + Assert.Equal("local-link-override", secondEntry.Key); + Assert.Equal("local-target", secondEntry.LinkTarget); + Assert.Equal(4100, secondEntry.UserID); + Assert.Equal(5100, secondEntry.GroupId); + Assert.Equal(Convert.ToInt64("777", 8), secondEntry.Mode); + + Assert.False(await reader.MoveToNextEntryAsync()); + } + + [Fact] + public async ValueTask Tar_WithSymlink_Reader_SurfacesLinkTargets_Async() + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, "TarWithSymlink.tar.gz"); + + using Stream stream = File.OpenRead(archivePath); + await using var reader = await TarReader.OpenAsyncReader(stream); + + var foundVulkanToolsLink = false; + var foundVulkanSamplesLink = false; + + while (await reader.MoveToNextEntryAsync()) + { + if (reader.Entry.Key == "MoltenVK-1.0.21/Demos/LunarG-VulkanSamples/Vulkan-Tools") + { + foundVulkanToolsLink = true; + Assert.Equal("../../External/Vulkan-Tools", reader.Entry.LinkTarget); + } + + if (reader.Entry.Key == "MoltenVK-1.0.21/Demos/LunarG-VulkanSamples/VulkanSamples") + { + foundVulkanSamplesLink = true; + Assert.Equal("../../External/VulkanSamples", reader.Entry.LinkTarget); + } + } + + Assert.True(foundVulkanToolsLink); + Assert.True(foundVulkanSamplesLink); + } + + [Fact] + public void Tar_Containing_Rar_Reader_Async() + { + var archiveFullPath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.ContainsRar.tar"); + using Stream stream = File.OpenRead(archiveFullPath); + using var reader = ReaderFactory.OpenReader(stream); + Assert.True(reader.Type == ArchiveType.Tar); + } + + [Fact] + public async ValueTask Tar_With_TarGz_With_Flushed_EntryStream_Async() + { + var archiveFullPath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.ContainsTarGz.tar"); + using Stream stream = File.OpenRead(archiveFullPath); + await using var reader = await ReaderFactory.OpenAsyncReader(stream); + Assert.True(await reader.MoveToNextEntryAsync()); + Assert.Equal("inner.tar.gz", reader.Entry.Key); + +#if !LEGACY_DOTNET + await using var entryStream = await reader.OpenEntryStreamAsync(); + await using var flushingStream = new FlushOnDisposeStream(entryStream); +#else + using var entryStream = await reader.OpenEntryStreamAsync(); + using var flushingStream = new FlushOnDisposeStream(entryStream); +#endif + + // Extract inner.tar.gz + await using var innerReader = await ReaderFactory.OpenAsyncReader(flushingStream); + Assert.True(await innerReader.MoveToNextEntryAsync()); + Assert.Equal("test", innerReader.Entry.Key); + } + + [Fact] + public async ValueTask Tar_Broken_Stream_Async() + { + var archiveFullPath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar"); + using Stream stream = File.OpenRead(archiveFullPath); + await using var reader = await ReaderFactory.OpenAsyncReader(new AsyncOnlyStream(stream)); + var memoryStream = new MemoryStream(); + + Assert.True(await reader.MoveToNextEntryAsync()); + Assert.True(await reader.MoveToNextEntryAsync()); + await reader.WriteEntryToAsync(memoryStream); + stream.Close(); + await Assert.ThrowsAsync(async () => + await reader.MoveToNextEntryAsync() + ); + } + + [Fact] + public async ValueTask Tar_Read_One_At_A_Time_Without_Disposing_Entry_Stream_Async() + { + var archiveEncoding = new ArchiveEncoding { Default = Encoding.UTF8 }; + var tarWriterOptions = new TarWriterOptions(CompressionType.None, true) + { + ArchiveEncoding = archiveEncoding, + }; + var testBytes = Encoding.UTF8.GetBytes("This is a test."); + + using var memoryStream = new MemoryStream(); + using (var tarWriter = new TarWriter(memoryStream, tarWriterOptions)) + using (var testFileStream = new MemoryStream(testBytes)) + { + await tarWriter.WriteAsync("file0.txt", testFileStream, null); + testFileStream.Position = 0; + await tarWriter.WriteAsync("file1.txt", testFileStream, null); + tarWriter.WriteDirectory("folder0", null); + testFileStream.Position = 0; + await tarWriter.WriteAsync("folder0/file_in_folder0.txt", testFileStream, null); + } + + memoryStream.Position = 0; + + var entryKeys = new List(); + var openEntryStreams = new List(); + + await using ( + var reader = await TarReader.OpenAsyncReader(new AsyncOnlyStream(memoryStream)) + ) + { + while (await reader.MoveToNextEntryAsync()) + { + entryKeys.Add(reader.Entry.Key); + if (reader.Entry.IsDirectory) + { + continue; + } + + var entryStream = await reader.OpenEntryStreamAsync(); + openEntryStreams.Add(entryStream); + + using var testFileStream = new MemoryStream(); + await entryStream.CopyToAsync(testFileStream); + Assert.Equal(testBytes.Length, testFileStream.Length); + } + } + + openEntryStreams.ForEach(stream => stream.Dispose()); + + Assert.Equal( + ["file0.txt", "file1.txt", "folder0/", "folder0/file_in_folder0.txt"], + entryKeys + ); + } + + [Fact] + public async ValueTask Tar_Corrupted_Async() + { + var archiveFullPath = Path.Combine(TEST_ARCHIVES_PATH, "TarCorrupted.tar"); + using Stream stream = File.OpenRead(archiveFullPath); + await using var reader = await ReaderFactory.OpenAsyncReader(new AsyncOnlyStream(stream)); + var memoryStream = new MemoryStream(); + + Assert.True(await reader.MoveToNextEntryAsync()); + Assert.True(await reader.MoveToNextEntryAsync()); + await reader.WriteEntryToAsync(memoryStream); + stream.Close(); + await Assert.ThrowsAsync(async () => + await reader.MoveToNextEntryAsync() + ); + } +} diff --git a/tests/SharpCompress.Test/Tar/TarReaderTests.cs b/tests/SharpCompress.Test/Tar/TarReaderTests.cs index 76629363..39c1ba15 100644 --- a/tests/SharpCompress.Test/Tar/TarReaderTests.cs +++ b/tests/SharpCompress.Test/Tar/TarReaderTests.cs @@ -1,10 +1,15 @@ using System; using System.Collections.Generic; using System.IO; +using System.Text; using SharpCompress.Common; +using SharpCompress.Common.Tar; +using SharpCompress.Compressors.BZip2; +using SharpCompress.Factories; using SharpCompress.Readers; using SharpCompress.Readers.Tar; using SharpCompress.Test.Mocks; +using SharpCompress.Writers.Tar; using Xunit; namespace SharpCompress.Test.Tar; @@ -19,72 +24,116 @@ public class TarReaderTests : ReaderTests [Fact] public void Tar_Skip() { - using ( - Stream stream = new ForwardOnlyStream( - File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar")) - ) - ) - using (IReader reader = ReaderFactory.Open(stream)) + using Stream stream = new ForwardOnlyStream( + File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar")) + ); + using var reader = ReaderFactory.OpenReader(stream); + var x = 0; + while (reader.MoveToNextEntry()) { - int x = 0; - while (reader.MoveToNextEntry()) + if (!reader.Entry.IsDirectory) { - if (!reader.Entry.IsDirectory) + x++; + if (x % 2 == 0) { - x++; - if (x % 2 == 0) - { - reader.WriteEntryToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); - } + reader.WriteEntryToDirectory(SCRATCH_FILES_PATH); } } } } + [Fact] + public void Tar_Z_Reader() => Read("Tar.tar.Z", CompressionType.Lzw); + [Fact] public void Tar_BZip2_Reader() => Read("Tar.tar.bz2", CompressionType.BZip2); [Fact] public void Tar_GZip_Reader() => Read("Tar.tar.gz", CompressionType.GZip); + [Fact] + public void Tar_ZStandard_Reader() => Read("Tar.tar.zst", CompressionType.ZStandard); + [Fact] public void Tar_LZip_Reader() => Read("Tar.tar.lz", CompressionType.LZip); [Fact] public void Tar_Xz_Reader() => Read("Tar.tar.xz", CompressionType.Xz); + [Fact] + public void Tar_GZip_OldGnu_Reader() => Read("Tar.oldgnu.tar.gz", CompressionType.GZip); + + [Fact] + public void Tar_BZip2_Reader_NonSeekable() + { + // Regression test for: Dynamic default RingBuffer for BZip2 + // Opening a .tar.bz2 from a non-seekable stream should succeed + // because the ring buffer is sized to hold the BZip2 block before calling IsTarFile. + using var fs = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.bz2")); + using var nonSeekable = new ForwardOnlyStream(fs); + using var reader = ReaderFactory.OpenReader(nonSeekable); + var entryCount = 0; + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + entryCount++; + } + } + Assert.True(entryCount > 0); + } + + [Fact] + public void TarWrapper_BZip2_MinimumRewindBufferSize_IsMaxBZip2BlockSize() + { + // The BZip2 TarWrapper must declare a MinimumRewindBufferSize large enough + // to hold an entire maximum-size compressed BZip2 block (9 × 100 000 bytes). + var bzip2Wrapper = Array.Find( + TarWrapper.Wrappers, + w => w.CompressionType == CompressionType.BZip2 + ); + Assert.NotNull(bzip2Wrapper); + Assert.Equal(BZip2Constants.baseBlockSize * 9, bzip2Wrapper.MinimumRewindBufferSize); + } + + [Fact] + public void TarWrapper_Default_MinimumRewindBufferSize_Is_DefaultRewindableBufferSize() + { + // Non-BZip2 wrappers that don't specify a custom size default to + // Constants.RewindableBufferSize so existing behaviour is unchanged. + var noneWrapper = Array.Find( + TarWrapper.Wrappers, + w => w.CompressionType == CompressionType.None + ); + Assert.NotNull(noneWrapper); + Assert.Equal(Common.Constants.RewindableBufferSize, noneWrapper.MinimumRewindBufferSize); + } + [Fact] public void Tar_BZip2_Entry_Stream() { using (Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.bz2"))) - using (var reader = TarReader.Open(stream)) + using (var reader = TarReader.OpenReader(stream)) { while (reader.MoveToNextEntry()) { if (!reader.Entry.IsDirectory) { Assert.Equal(CompressionType.BZip2, reader.Entry.CompressionType); - using (var entryStream = reader.OpenEntryStream()) + using var entryStream = reader.OpenEntryStream(); + var file = Path.GetFileName(reader.Entry.Key); + var folder = + Path.GetDirectoryName(reader.Entry.Key) + ?? throw new ArgumentNullException(); + var destdir = Path.Combine(SCRATCH_FILES_PATH, folder); + if (!Directory.Exists(destdir)) { - string file = Path.GetFileName(reader.Entry.Key); - string folder = - Path.GetDirectoryName(reader.Entry.Key) - ?? throw new ArgumentNullException(); - string destdir = Path.Combine(SCRATCH_FILES_PATH, folder); - if (!Directory.Exists(destdir)) - { - Directory.CreateDirectory(destdir); - } - string destinationFileName = Path.Combine(destdir, file); - - using (FileStream fs = File.OpenWrite(destinationFileName)) - { - entryStream.TransferTo(fs); - } + Directory.CreateDirectory(destdir); } + var destinationFileName = Path.Combine(destdir, file.NotNull()); + + using var fs = File.OpenWrite(destinationFileName); + entryStream.CopyTo(fs); } } } @@ -101,13 +150,13 @@ public class TarReaderTests : ReaderTests Path.Combine(TEST_ARCHIVES_PATH, "Tar.LongPathsWithLongNameExtension.tar") ) ) - using (var reader = TarReader.Open(stream)) + using (var reader = TarReader.OpenReader(stream)) { while (reader.MoveToNextEntry()) { if (!reader.Entry.IsDirectory) { - filePaths.Add(reader.Entry.Key); + filePaths.Add(reader.Entry.Key.NotNull("Entry Key is null")); } } } @@ -124,150 +173,334 @@ public class TarReaderTests : ReaderTests ); } + [Fact] + public void Tar_PaxLocalHeader_Reader() + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.PaxLocalHeader.tar"); + + using Stream stream = File.OpenRead(archivePath); + using var reader = TarReader.OpenReader(stream); + + Assert.True(reader.MoveToNextEntry()); + var firstEntry = (TarEntry)reader.Entry; + Assert.Equal("pax/overridden-name.txt", firstEntry.Key); + Assert.Equal(10, firstEntry.Size); + Assert.Equal(1234, firstEntry.UserID); + Assert.Equal(2345, firstEntry.GroupId); + Assert.Equal(Convert.ToInt64("640", 8), firstEntry.Mode); + + var expectedTime = DateTimeOffset.FromUnixTimeSeconds(1700000000).LocalDateTime; + Assert.Equal(expectedTime, firstEntry.LastModifiedTime); + + using (var entryStream = reader.OpenEntryStream()) + using (var memoryStream = new MemoryStream()) + { + entryStream.CopyTo(memoryStream); + Assert.Equal(10, memoryStream.Length); + } + + Assert.True(reader.MoveToNextEntry()); + var secondEntry = (TarEntry)reader.Entry; + Assert.Equal("second.txt", secondEntry.Key); + Assert.Equal(11, secondEntry.UserID); + Assert.Equal(22, secondEntry.GroupId); + Assert.Equal(Convert.ToInt64("644", 8), secondEntry.Mode); + Assert.Equal(2, secondEntry.Size); + + Assert.False(reader.MoveToNextEntry()); + } + + [Fact] + public void Tar_PaxLocalHeader_Link_Reader() + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.PaxLocalHeader.Link.tar"); + + using Stream stream = File.OpenRead(archivePath); + using var reader = TarReader.OpenReader(stream); + + Assert.True(reader.MoveToNextEntry()); + Assert.Equal("pax/link-entry", reader.Entry.Key); + Assert.Equal("pax/target-entry", reader.Entry.LinkTarget); + Assert.False(reader.Entry.IsDirectory); + Assert.False(reader.MoveToNextEntry()); + } + + [Fact] + public void Tar_PaxGlobalHeader_Reader() + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.PaxGlobalHeader.tar"); + + using Stream stream = File.OpenRead(archivePath); + using var reader = TarReader.OpenReader(stream); + + var globalTime = DateTimeOffset.FromUnixTimeSeconds(1700000100).LocalDateTime; + var localOverrideTime = DateTimeOffset.FromUnixTimeSeconds(1700000200).LocalDateTime; + + Assert.True(reader.MoveToNextEntry()); + var firstEntry = (TarEntry)reader.Entry; + Assert.Equal("global-one.txt", firstEntry.Key); + Assert.Equal(4000, firstEntry.UserID); + Assert.Equal(5000, firstEntry.GroupId); + Assert.Equal(Convert.ToInt64("640", 8), firstEntry.Mode); + Assert.Equal(globalTime, firstEntry.LastModifiedTime); + + Assert.True(reader.MoveToNextEntry()); + var secondEntry = (TarEntry)reader.Entry; + Assert.Equal("global-local-override.txt", secondEntry.Key); + Assert.Equal(4010, secondEntry.UserID); + Assert.Equal(5010, secondEntry.GroupId); + Assert.Equal(Convert.ToInt64("600", 8), secondEntry.Mode); + Assert.Equal(localOverrideTime, secondEntry.LastModifiedTime); + + Assert.True(reader.MoveToNextEntry()); + var thirdEntry = (TarEntry)reader.Entry; + Assert.Equal("global-three.txt", thirdEntry.Key); + Assert.Equal(4000, thirdEntry.UserID); + Assert.Equal(5000, thirdEntry.GroupId); + Assert.Equal(Convert.ToInt64("640", 8), thirdEntry.Mode); + Assert.Equal(globalTime, thirdEntry.LastModifiedTime); + + Assert.False(reader.MoveToNextEntry()); + } + + [Fact] + public void Tar_PaxGlobalHeader_Link_Reader() + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.PaxGlobalHeader.Link.tar"); + + using Stream stream = File.OpenRead(archivePath); + using var reader = TarReader.OpenReader(stream); + + Assert.True(reader.MoveToNextEntry()); + var firstEntry = (TarEntry)reader.Entry; + Assert.Equal("global-link", firstEntry.Key); + Assert.Equal("global-target", firstEntry.LinkTarget); + Assert.Equal(4100, firstEntry.UserID); + Assert.Equal(5100, firstEntry.GroupId); + Assert.Equal(Convert.ToInt64("777", 8), firstEntry.Mode); + + Assert.True(reader.MoveToNextEntry()); + var secondEntry = (TarEntry)reader.Entry; + Assert.Equal("local-link-override", secondEntry.Key); + Assert.Equal("local-target", secondEntry.LinkTarget); + Assert.Equal(4100, secondEntry.UserID); + Assert.Equal(5100, secondEntry.GroupId); + Assert.Equal(Convert.ToInt64("777", 8), secondEntry.Mode); + + Assert.False(reader.MoveToNextEntry()); + } + + [Fact] + public void Tar_WithSymlink_Reader_SurfacesLinkTargets() + { + var archivePath = Path.Combine(TEST_ARCHIVES_PATH, "TarWithSymlink.tar.gz"); + + using Stream stream = File.OpenRead(archivePath); + using var reader = TarReader.OpenReader(stream); + + var foundVulkanToolsLink = false; + var foundVulkanSamplesLink = false; + + while (reader.MoveToNextEntry()) + { + if (reader.Entry.Key == "MoltenVK-1.0.21/Demos/LunarG-VulkanSamples/Vulkan-Tools") + { + foundVulkanToolsLink = true; + Assert.Equal("../../External/Vulkan-Tools", reader.Entry.LinkTarget); + } + + if (reader.Entry.Key == "MoltenVK-1.0.21/Demos/LunarG-VulkanSamples/VulkanSamples") + { + foundVulkanSamplesLink = true; + Assert.Equal("../../External/VulkanSamples", reader.Entry.LinkTarget); + } + } + + Assert.True(foundVulkanToolsLink); + Assert.True(foundVulkanSamplesLink); + } + [Fact] public void Tar_BZip2_Skip_Entry_Stream() { - using (Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.bz2"))) - using (var reader = TarReader.Open(stream)) + using Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar.bz2")); + using var reader = TarReader.OpenReader(stream); + var names = new List(); + while (reader.MoveToNextEntry()) { - List names = new List(); - while (reader.MoveToNextEntry()) + if (!reader.Entry.IsDirectory) { - if (!reader.Entry.IsDirectory) - { - Assert.Equal(CompressionType.BZip2, reader.Entry.CompressionType); - using (var entryStream = reader.OpenEntryStream()) - { - entryStream.SkipEntry(); - names.Add(reader.Entry.Key); - } - } + Assert.Equal(CompressionType.BZip2, reader.Entry.CompressionType); + using var entryStream = reader.OpenEntryStream(); + entryStream.SkipEntry(); + names.Add(reader.Entry.Key.NotNull()); } - Assert.Equal(3, names.Count); } + Assert.Equal(3, names.Count); } [Fact] public void Tar_Containing_Rar_Reader() { - string archiveFullPath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.ContainsRar.tar"); - using (Stream stream = File.OpenRead(archiveFullPath)) - using (IReader reader = ReaderFactory.Open(stream)) - { - Assert.True(reader.ArchiveType == ArchiveType.Tar); - } + var archiveFullPath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.ContainsRar.tar"); + using Stream stream = File.OpenRead(archiveFullPath); + using var reader = ReaderFactory.OpenReader(stream); + Assert.True(reader.Type == ArchiveType.Tar); } [Fact] public void Tar_With_TarGz_With_Flushed_EntryStream() { - string archiveFullPath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.ContainsTarGz.tar"); - using (Stream stream = File.OpenRead(archiveFullPath)) - using (IReader reader = ReaderFactory.Open(stream)) - { - Assert.True(reader.MoveToNextEntry()); - Assert.Equal("inner.tar.gz", reader.Entry.Key); + var archiveFullPath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.ContainsTarGz.tar"); + using Stream stream = File.OpenRead(archiveFullPath); + using var reader = ReaderFactory.OpenReader(stream); + Assert.True(reader.MoveToNextEntry()); + Assert.Equal("inner.tar.gz", reader.Entry.Key); - using (var entryStream = reader.OpenEntryStream()) - { - using (FlushOnDisposeStream flushingStream = new FlushOnDisposeStream(entryStream)) - { - // Extract inner.tar.gz - using (var innerReader = ReaderFactory.Open(flushingStream)) - { - Assert.True(innerReader.MoveToNextEntry()); - Assert.Equal("test", innerReader.Entry.Key); - } - } - } - } + using var entryStream = reader.OpenEntryStream(); + using var flushingStream = new FlushOnDisposeStream(entryStream); + + // Extract inner.tar.gz + using var innerReader = ReaderFactory.OpenReader(flushingStream); + Assert.True(innerReader.MoveToNextEntry()); + Assert.Equal("test", innerReader.Entry.Key); } [Fact] public void Tar_Broken_Stream() { - string archiveFullPath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar"); - using (Stream stream = File.OpenRead(archiveFullPath)) - using (IReader reader = ReaderFactory.Open(stream)) - { - var memoryStream = new MemoryStream(); + var archiveFullPath = Path.Combine(TEST_ARCHIVES_PATH, "Tar.tar"); + using Stream stream = File.OpenRead(archiveFullPath); + using var reader = ReaderFactory.OpenReader(stream); + var memoryStream = new MemoryStream(); - Action action = () => reader.MoveToNextEntry(); - var exception = Record.Exception(action); - Assert.Null(exception); - reader.MoveToNextEntry(); - reader.WriteEntryTo(memoryStream); - stream.Close(); - Assert.Throws(action); - } + Assert.True(reader.MoveToNextEntry()); + Assert.True(reader.MoveToNextEntry()); + reader.WriteEntryTo(memoryStream); + stream.Close(); + Assert.Throws(() => reader.MoveToNextEntry()); } -#if !NETFRAMEWORK [Fact] - public void Tar_GZip_With_Symlink_Entries() + public void Tar_Read_One_At_A_Time_Without_Disposing_Entry_Stream() { - var isWindows = System.Runtime.InteropServices.RuntimeInformation.IsOSPlatform( - System.Runtime.InteropServices.OSPlatform.Windows - ); - using ( - Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "TarWithSymlink.tar.gz")) - ) - using (var reader = TarReader.Open(stream)) + var archiveEncoding = new ArchiveEncoding { Default = Encoding.UTF8 }; + var tarWriterOptions = new TarWriterOptions(CompressionType.None, true) + { + ArchiveEncoding = archiveEncoding, + }; + var testBytes = Encoding.UTF8.GetBytes("This is a test."); + + using var memoryStream = new MemoryStream(); + using (var tarWriter = new TarWriter(memoryStream, tarWriterOptions)) + using (var testFileStream = new MemoryStream(testBytes)) + { + tarWriter.Write("file0.txt", testFileStream, null); + testFileStream.Position = 0; + tarWriter.Write("file1.txt", testFileStream, null); + tarWriter.WriteDirectory("folder0", null); + testFileStream.Position = 0; + tarWriter.Write("folder0/file_in_folder0.txt", testFileStream, null); + } + + memoryStream.Position = 0; + + var entryKeys = new List(); + var openEntryStreams = new List(); + + using (var reader = TarReader.OpenReader(memoryStream)) { - List names = new List(); while (reader.MoveToNextEntry()) { + entryKeys.Add(reader.Entry.Key); if (reader.Entry.IsDirectory) { continue; } - reader.WriteEntryToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() - { - ExtractFullPath = true, - Overwrite = true, - WriteSymbolicLink = (sourcePath, targetPath) => - { - if (!isWindows) - { - var link = new Mono.Unix.UnixSymbolicLinkInfo(sourcePath); - if (File.Exists(sourcePath)) - { - link.Delete(); // equivalent to ln -s -f - } - link.CreateSymbolicLinkTo(targetPath); - } - } - } - ); - if (!isWindows) - { - if (reader.Entry.LinkTarget != null) - { - var path = System.IO.Path.Combine(SCRATCH_FILES_PATH, reader.Entry.Key); - var link = new Mono.Unix.UnixSymbolicLinkInfo(path); - if (link.HasContents) - { - // need to convert the link to an absolute path for comparison - var target = reader.Entry.LinkTarget; - var realTarget = System.IO.Path.GetFullPath( - System.IO.Path.Combine( - $"{System.IO.Path.GetDirectoryName(path)}", - target - ) - ); - Assert.Equal(realTarget, link.GetContents().ToString()); - } - else - { - Assert.True(false, "Symlink has no target"); - } - } - } + var entryStream = reader.OpenEntryStream(); + openEntryStreams.Add(entryStream); + + using var testFileStream = new MemoryStream(); + entryStream.CopyTo(testFileStream); + Assert.Equal(testBytes.Length, testFileStream.Length); } } + + openEntryStreams.ForEach(stream => stream.Dispose()); + + Assert.Equal( + ["file0.txt", "file1.txt", "folder0/", "folder0/file_in_folder0.txt"], + entryKeys + ); + } + + [Fact] + public void Tar_Corrupted() + { + var archiveFullPath = Path.Combine(TEST_ARCHIVES_PATH, "TarCorrupted.tar"); + using Stream stream = File.OpenRead(archiveFullPath); + using var reader = ReaderFactory.OpenReader(stream); + var memoryStream = new MemoryStream(); + + Assert.True(reader.MoveToNextEntry()); + Assert.True(reader.MoveToNextEntry()); + reader.WriteEntryTo(memoryStream); + stream.Close(); + Assert.Throws(() => reader.MoveToNextEntry()); + } + + [Fact] + public void Tar_Malformed_LongName_Excessive_Size() + { + // Create a malformed TAR header with an excessively large LongName size + // This simulates what happens during auto-detection of compressed files + var buffer = new byte[512]; + + // Set up a basic TAR header structure + // Name field (offset 0, 100 bytes) - set to "././@LongLink" which is typical for LongName + var nameBytes = System.Text.Encoding.ASCII.GetBytes("././@LongLink"); + Array.Copy(nameBytes, 0, buffer, 0, nameBytes.Length); + + // Set entry type to LongName (offset 156) + buffer[156] = (byte)'L'; // EntryType.LongName + + // Set an excessively large size (offset 124, 12 bytes, octal format) + // This simulates a corrupted/misinterpreted size field + // Using "77777777777" (octal) = 8589934591 bytes (~8GB) + var sizeBytes = System.Text.Encoding.ASCII.GetBytes("77777777777 "); + Array.Copy(sizeBytes, 0, buffer, 124, sizeBytes.Length); + + // Calculate and set checksum (offset 148, 8 bytes) + // Set checksum field to spaces first + for (var i = 148; i < 156; i++) + { + buffer[i] = (byte)' '; + } + + // Calculate checksum + var checksum = 0; + foreach (var b in buffer) + { + checksum += b; + } + + var checksumStr = Convert.ToString(checksum, 8).PadLeft(6, '0') + "\0 "; + var checksumBytes = System.Text.Encoding.ASCII.GetBytes(checksumStr); + Array.Copy(checksumBytes, 0, buffer, 148, checksumBytes.Length); + + // Create a stream with this malformed header + using var stream = new MemoryStream(); + stream.Write(buffer, 0, buffer.Length); + stream.Position = 0; + + // Attempt to read this malformed archive + // The InvalidFormatException from the validation gets caught and converted to IncompleteArchiveException + // The important thing is it doesn't cause OutOfMemoryException + Assert.Throws(() => + { + using var reader = TarReader.OpenReader(stream); + reader.MoveToNextEntry(); + }); } -#endif } diff --git a/tests/SharpCompress.Test/Tar/TarWriterAsyncTests.cs b/tests/SharpCompress.Test/Tar/TarWriterAsyncTests.cs new file mode 100644 index 00000000..3a516a04 --- /dev/null +++ b/tests/SharpCompress.Test/Tar/TarWriterAsyncTests.cs @@ -0,0 +1,172 @@ +using System.IO; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using SharpCompress.Archives.Tar; +using SharpCompress.Common; +using SharpCompress.Common.Tar.Headers; +using SharpCompress.Test.Mocks; +using SharpCompress.Writers.Tar; +using Xunit; + +namespace SharpCompress.Test.Tar; + +public class TarWriterAsyncTests : WriterTests +{ + static TarWriterAsyncTests() + { +#if !NETFRAMEWORK + //fix issue where these tests could not be ran in isolation + System.Text.Encoding.RegisterProvider(System.Text.CodePagesEncodingProvider.Instance); +#endif + } + + public TarWriterAsyncTests() + : base(ArchiveType.Tar) => UseExtensionInsteadOfNameToVerify = true; + + [Fact] + public async ValueTask Tar_Writer_Async() => + await WriteAsync( + CompressionType.None, + "Tar.noEmptyDirs.tar", + "Tar.noEmptyDirs.tar", + Encoding.GetEncoding(866) + ); + + [Fact] + public async ValueTask Tar_BZip2_Writer_Async() => + await WriteAsync( + CompressionType.BZip2, + "Tar.noEmptyDirs.tar.bz2", + "Tar.noEmptyDirs.tar.bz2", + Encoding.GetEncoding(866) + ); + + [Fact] + public async ValueTask Tar_LZip_Writer_Async() => + await WriteAsync( + CompressionType.LZip, + "Tar.noEmptyDirs.tar.lz", + "Tar.noEmptyDirs.tar.lz", + Encoding.GetEncoding(866) + ); + + [Fact] + public async ValueTask Tar_Rar_Write_Async() => + await Assert.ThrowsAsync(async () => + await WriteAsync( + CompressionType.Rar, + "Zip.ppmd.noEmptyDirs.zip", + "Zip.ppmd.noEmptyDirs.zip" + ) + ); + + [Theory] + [InlineData(CompressionType.Xz)] + [InlineData(CompressionType.ZStandard)] + [InlineData(CompressionType.Lzw)] + public async ValueTask Tar_UnsupportedWrapperCompression_Write_Async( + CompressionType compressionType + ) => + await Assert.ThrowsAsync(async () => + await WriteAsync( + compressionType, + "Zip.ppmd.noEmptyDirs.zip", + "Zip.ppmd.noEmptyDirs.zip" + ) + ); + + [Theory] + [InlineData(true)] + [InlineData(false)] + public async ValueTask Tar_Finalize_Archive_Async(bool finalizeArchive) + { + using var stream = new MemoryStream(); + using Stream content = File.OpenRead(Path.Combine(ORIGINAL_FILES_PATH, "jpg", "test.jpg")); + await using ( + var writer = new TarWriter( + new AsyncOnlyStream(stream, false), + new TarWriterOptions(CompressionType.None, finalizeArchive) + ) + ) + { + await writer.WriteAsync("doesn't matter", content, null); + } + + var paddedContentWithHeader = (content.Length / 512 * 512) + 512 + 512; + var expectedStreamLength = finalizeArchive + ? paddedContentWithHeader + (512 * 2) + : paddedContentWithHeader; + Assert.Equal(expectedStreamLength, stream.Length); + } + + [Fact] + public async ValueTask Tar_Ustar_HeaderFormat_WritesShortPath_Async() + { + using var stream = new MemoryStream(); + var options = new TarWriterOptions(CompressionType.None, true, TarHeaderWriteFormat.USTAR); + await using (var writer = new TarWriter(new AsyncOnlyStream(stream), options)) + using (var content = new MemoryStream(Encoding.UTF8.GetBytes("hello"))) + { + await writer.WriteAsync("dir/file.txt", content, null); + } + + stream.Position = 0; + using var archive = TarArchive.OpenArchive(stream); + Assert.Single(archive.Entries); + Assert.Equal("dir/file.txt", archive.Entries.Single().Key); + } + + [Fact] + public async ValueTask Tar_Ustar_HeaderFormat_ThrowsForLongPath_Async() + { + var longName = new string('a', 160) + ".txt"; + + using var stream = new MemoryStream(); + var options = new TarWriterOptions(CompressionType.None, true, TarHeaderWriteFormat.USTAR); + await using var writer = new TarWriter(new AsyncOnlyStream(stream), options); + using var content = new MemoryStream(Encoding.UTF8.GetBytes("hello")); + + await Assert.ThrowsAsync(async () => + await writer.WriteAsync(longName, content, null) + ); + } + + [Fact] + public async ValueTask Tar_GnuLongLink_HeaderFormat_WritesLongPath_Async() + { + var longName = new string('a', 160) + ".txt"; + + using var stream = new MemoryStream(); + var options = new TarWriterOptions( + CompressionType.None, + true, + TarHeaderWriteFormat.GNU_TAR_LONG_LINK + ); + + await using (var writer = new TarWriter(new AsyncOnlyStream(stream), options)) + using (var content = new MemoryStream(Encoding.UTF8.GetBytes("hello"))) + { + await writer.WriteAsync(longName, content, null); + } + + stream.Position = 0; + using var archive = TarArchive.OpenArchive(stream); + Assert.Single(archive.Entries); + Assert.Equal(longName, archive.Entries.Single().Key); + } + + [Fact] + public async ValueTask Tar_Ustar_HeaderFormat_ThrowsForLongDirectory_Async() + { + var longDirectory = new string('a', 170); + + using var stream = new MemoryStream(); + var options = new TarWriterOptions(CompressionType.None, true, TarHeaderWriteFormat.USTAR); + await using var writer = new TarWriter(new AsyncOnlyStream(stream), options); + + await Assert.ThrowsAsync(async () => + await writer.WriteDirectoryAsync(longDirectory, null) + ); + } +} diff --git a/tests/SharpCompress.Test/Tar/TarWriterDirectoryTests.cs b/tests/SharpCompress.Test/Tar/TarWriterDirectoryTests.cs new file mode 100644 index 00000000..279fb511 --- /dev/null +++ b/tests/SharpCompress.Test/Tar/TarWriterDirectoryTests.cs @@ -0,0 +1,207 @@ +using System; +using System.IO; +using System.Linq; +using SharpCompress.Archives.Tar; +using SharpCompress.Common; +using SharpCompress.Common.Tar.Headers; +using SharpCompress.Writers.Tar; +using Xunit; + +namespace SharpCompress.Test.Tar; + +public class TarWriterDirectoryTests : TestBase +{ + [Fact] + public void TarWriter_WriteDirectory_CreatesDirectoryEntry() + { + using var memoryStream = new MemoryStream(); + using ( + var writer = new TarWriter( + memoryStream, + new TarWriterOptions(CompressionType.None, true) + ) + ) + { + writer.WriteDirectory("test-dir", DateTime.Now); + } + + memoryStream.Position = 0; + using var archive = TarArchive.OpenArchive(memoryStream); + var entries = archive.Entries.ToList(); + + Assert.Single(entries); + Assert.Equal("test-dir/", entries[0].Key); + Assert.True(entries[0].IsDirectory); + } + + [Fact] + public void TarWriter_WriteDirectory_WithTrailingSlash() + { + using var memoryStream = new MemoryStream(); + using ( + var writer = new TarWriter( + memoryStream, + new TarWriterOptions(CompressionType.None, true) + ) + ) + { + writer.WriteDirectory("test-dir/", DateTime.Now); + } + + memoryStream.Position = 0; + using var archive = TarArchive.OpenArchive(memoryStream); + var entries = archive.Entries.ToList(); + + Assert.Single(entries); + Assert.Equal("test-dir/", entries[0].Key); + Assert.True(entries[0].IsDirectory); + } + + [Fact] + public void TarWriter_WriteDirectory_WithBackslash() + { + using var memoryStream = new MemoryStream(); + using ( + var writer = new TarWriter( + memoryStream, + new TarWriterOptions(CompressionType.None, true) + ) + ) + { + writer.WriteDirectory("test-dir\\subdir", DateTime.Now); + } + + memoryStream.Position = 0; + using var archive = TarArchive.OpenArchive(memoryStream); + var entries = archive.Entries.ToList(); + + Assert.Single(entries); + Assert.Equal("test-dir/subdir/", entries[0].Key); + Assert.True(entries[0].IsDirectory); + } + + [Fact] + public void TarWriter_WriteDirectory_EmptyString_IsSkipped() + { + using var memoryStream = new MemoryStream(); + using ( + var writer = new TarWriter( + memoryStream, + new TarWriterOptions(CompressionType.None, true) + ) + ) + { + writer.WriteDirectory("", DateTime.Now); + } + + memoryStream.Position = 0; + using var archive = TarArchive.OpenArchive(memoryStream); + + Assert.Empty(archive.Entries); + } + + [Fact] + public void TarWriter_WriteDirectory_MultipleDirectories() + { + using var memoryStream = new MemoryStream(); + using ( + var writer = new TarWriter( + memoryStream, + new TarWriterOptions(CompressionType.None, true) + ) + ) + { + writer.WriteDirectory("dir1", DateTime.Now); + writer.WriteDirectory("dir2", DateTime.Now); + writer.WriteDirectory("dir1/subdir", DateTime.Now); + } + + memoryStream.Position = 0; + using var archive = TarArchive.OpenArchive(memoryStream); + var entries = archive.Entries.OrderBy(e => e.Key).ToList(); + + Assert.Equal(3, entries.Count); + Assert.Equal("dir1/", entries[0].Key); + Assert.True(entries[0].IsDirectory); + Assert.Equal("dir1/subdir/", entries[1].Key); + Assert.True(entries[1].IsDirectory); + Assert.Equal("dir2/", entries[2].Key); + Assert.True(entries[2].IsDirectory); + } + + [Fact] + public void TarWriter_WriteDirectory_MixedWithFiles() + { + using var memoryStream = new MemoryStream(); + using ( + var writer = new TarWriter( + memoryStream, + new TarWriterOptions(CompressionType.None, true) + ) + ) + { + writer.WriteDirectory("dir1", DateTime.Now); + + using var contentStream = new MemoryStream( + System.Text.Encoding.UTF8.GetBytes("test content") + ); + writer.Write("dir1/file.txt", contentStream, DateTime.Now); + + writer.WriteDirectory("dir2", DateTime.Now); + } + + memoryStream.Position = 0; + using var archive = TarArchive.OpenArchive(memoryStream); + var entries = archive.Entries.OrderBy(e => e.Key).ToList(); + + Assert.Equal(3, entries.Count); + Assert.Equal("dir1/", entries[0].Key); + Assert.True(entries[0].IsDirectory); + Assert.Equal("dir1/file.txt", entries[1].Key); + Assert.False(entries[1].IsDirectory); + Assert.Equal("dir2/", entries[2].Key); + Assert.True(entries[2].IsDirectory); + } + + [Fact] + public void TarWriter_WriteDirectory_Ustar_ThrowsForLongDirectoryName() + { + using var memoryStream = new MemoryStream(); + using var writer = new TarWriter( + memoryStream, + new TarWriterOptions(CompressionType.None, true, TarHeaderWriteFormat.USTAR) + ); + + var longDirectoryName = new string('a', 170); + Assert.Throws(() => + writer.WriteDirectory(longDirectoryName, DateTime.Now) + ); + } + + [Fact] + public void TarWriter_WriteDirectory_GnuLongLink_WritesLongDirectoryName() + { + var longDirectoryName = new string('a', 170); + + using var memoryStream = new MemoryStream(); + using ( + var writer = new TarWriter( + memoryStream, + new TarWriterOptions( + CompressionType.None, + true, + TarHeaderWriteFormat.GNU_TAR_LONG_LINK + ) + ) + ) + { + writer.WriteDirectory(longDirectoryName, DateTime.Now); + } + + memoryStream.Position = 0; + using var archive = TarArchive.OpenArchive(memoryStream); + var entry = archive.Entries.Single(); + Assert.Equal(longDirectoryName + "/", entry.Key); + Assert.True(entry.IsDirectory); + } +} diff --git a/tests/SharpCompress.Test/Tar/TarWriterTests.cs b/tests/SharpCompress.Test/Tar/TarWriterTests.cs index 4f4464c1..b4cada92 100644 --- a/tests/SharpCompress.Test/Tar/TarWriterTests.cs +++ b/tests/SharpCompress.Test/Tar/TarWriterTests.cs @@ -1,6 +1,9 @@ -using System.IO; +using System.IO; +using System.Linq; using System.Text; +using SharpCompress.Archives.Tar; using SharpCompress.Common; +using SharpCompress.Common.Tar.Headers; using SharpCompress.Writers.Tar; using Xunit; @@ -8,6 +11,14 @@ namespace SharpCompress.Test.Tar; public class TarWriterTests : WriterTests { + static TarWriterTests() + { +#if !NETFRAMEWORK + //fix issue where these tests could not be ran in isolation + System.Text.Encoding.RegisterProvider(System.Text.CodePagesEncodingProvider.Instance); +#endif + } + public TarWriterTests() : base(ArchiveType.Tar) => UseExtensionInsteadOfNameToVerify = true; @@ -40,8 +51,17 @@ public class TarWriterTests : WriterTests [Fact] public void Tar_Rar_Write() => - Assert.Throws( - () => Write(CompressionType.Rar, "Zip.ppmd.noEmptyDirs.zip", "Zip.ppmd.noEmptyDirs.zip") + Assert.Throws(() => + Write(CompressionType.Rar, "Zip.ppmd.noEmptyDirs.zip", "Zip.ppmd.noEmptyDirs.zip") + ); + + [Theory] + [InlineData(CompressionType.Xz)] + [InlineData(CompressionType.ZStandard)] + [InlineData(CompressionType.Lzw)] + public void Tar_UnsupportedWrapperCompression_Write(CompressionType compressionType) => + Assert.Throws(() => + Write(compressionType, "Zip.ppmd.noEmptyDirs.zip", "Zip.ppmd.noEmptyDirs.zip") ); [Theory] @@ -67,4 +87,58 @@ public class TarWriterTests : WriterTests : paddedContentWithHeader; Assert.Equal(expectedStreamLength, stream.Length); } + + [Fact] + public void Tar_Ustar_HeaderFormat_WritesShortPath() + { + using var stream = new MemoryStream(); + var options = new TarWriterOptions(CompressionType.None, true, TarHeaderWriteFormat.USTAR); + using (var writer = new TarWriter(stream, options)) + using (var content = new MemoryStream(Encoding.UTF8.GetBytes("hello"))) + { + writer.Write("dir/file.txt", content, null); + } + + stream.Position = 0; + using var archive = TarArchive.OpenArchive(stream); + Assert.Single(archive.Entries); + Assert.Equal("dir/file.txt", archive.Entries.Single().Key); + } + + [Fact] + public void Tar_Ustar_HeaderFormat_ThrowsForLongPath() + { + var longName = new string('a', 160) + ".txt"; + + using var stream = new MemoryStream(); + var options = new TarWriterOptions(CompressionType.None, true, TarHeaderWriteFormat.USTAR); + using var writer = new TarWriter(stream, options); + using var content = new MemoryStream(Encoding.UTF8.GetBytes("hello")); + + Assert.Throws(() => writer.Write(longName, content, null)); + } + + [Fact] + public void Tar_GnuLongLink_HeaderFormat_WritesLongPath() + { + var longName = new string('a', 160) + ".txt"; + + using var stream = new MemoryStream(); + var options = new TarWriterOptions( + CompressionType.None, + true, + TarHeaderWriteFormat.GNU_TAR_LONG_LINK + ); + + using (var writer = new TarWriter(stream, options)) + using (var content = new MemoryStream(Encoding.UTF8.GetBytes("hello"))) + { + writer.Write(longName, content, null); + } + + stream.Position = 0; + using var archive = TarArchive.OpenArchive(stream); + Assert.Single(archive.Entries); + Assert.Equal(longName, archive.Entries.Single().Key); + } } diff --git a/tests/SharpCompress.Test/TempDirectory.cs b/tests/SharpCompress.Test/TempDirectory.cs new file mode 100644 index 00000000..e155210f --- /dev/null +++ b/tests/SharpCompress.Test/TempDirectory.cs @@ -0,0 +1,83 @@ +using System; +using System.IO; +using System.Threading.Tasks; + +namespace SharpCompress.Test; + +internal sealed class TempDirectory : IAsyncDisposable +{ + private const int MaxDeleteAttempts = 5; + private static readonly TimeSpan DeleteRetryDelay = TimeSpan.FromMilliseconds(100); + + public TempDirectory(string prefix) + { + Path = System.IO.Path.Combine(System.IO.Path.GetTempPath(), $"{prefix}.{Guid.NewGuid():N}"); + Directory.CreateDirectory(Path); + } + + public string Path { get; } + + public string GetDirectory(string name) + { + var path = System.IO.Path.Combine(Path, name); + Directory.CreateDirectory(path); + return path; + } + + public string CreateDirectory(string name) + { + var path = System.IO.Path.Combine(Path, name, System.IO.Path.GetRandomFileName()); + Directory.CreateDirectory(path); + return path; + } + + public void ResetDirectory(string name) + { + DeleteDirectory(System.IO.Path.Combine(Path, name)); + Directory.CreateDirectory(System.IO.Path.Combine(Path, name)); + } + + public async ValueTask DisposeAsync() + { + for (var attempt = 1; attempt <= MaxDeleteAttempts; attempt++) + { + try + { + DeleteDirectory(Path); + if (Directory.Exists(Path)) + { + throw new IOException( + $"Temp test directory '{Path}' still exists after deletion." + ); + } + + return; + } + catch (Exception ex) + when (IsRetryableDeleteException(ex) && attempt < MaxDeleteAttempts) + { + await Task.Delay(DeleteRetryDelay).ConfigureAwait(false); + } + catch (Exception ex) when (IsRetryableDeleteException(ex)) + { + throw new InvalidOperationException( + $"Failed to clean up temp test directory '{Path}'.", + ex + ); + } + } + + throw new InvalidOperationException($"Temp test directory '{Path}' was not cleaned up."); + } + + private static void DeleteDirectory(string path) + { + if (Directory.Exists(path)) + { + Directory.Delete(path, true); + } + } + + private static bool IsRetryableDeleteException(Exception ex) => + ex is IOException or UnauthorizedAccessException; +} diff --git a/tests/SharpCompress.Test/TestBase.cs b/tests/SharpCompress.Test/TestBase.cs index 3ad1fe79..1c244ac8 100644 --- a/tests/SharpCompress.Test/TestBase.cs +++ b/tests/SharpCompress.Test/TestBase.cs @@ -3,22 +3,20 @@ using System.Collections.Generic; using System.IO; using System.Linq; using System.Text; +using System.Threading.Tasks; using SharpCompress.Readers; using Xunit; namespace SharpCompress.Test; -public class TestBase : IDisposable +public class TestBase : IAsyncDisposable { - private string SOLUTION_BASE_PATH; - protected string TEST_ARCHIVES_PATH; - protected string ORIGINAL_FILES_PATH; - protected string MISC_TEST_FILES_PATH; - private string SCRATCH_BASE_PATH; - public string SCRATCH_FILES_PATH; - protected string SCRATCH2_FILES_PATH; + private static readonly string SOLUTION_BASE_PATH; + public static readonly string TEST_ARCHIVES_PATH; + public static readonly string ORIGINAL_FILES_PATH; + public static readonly string MISC_TEST_FILES_PATH; - public TestBase() + static TestBase() { var index = AppDomain.CurrentDomain.BaseDirectory.IndexOf( "SharpCompress.Test", @@ -30,20 +28,42 @@ public class TestBase : IDisposable TEST_ARCHIVES_PATH = Path.Combine(SOLUTION_BASE_PATH, "TestArchives", "Archives"); ORIGINAL_FILES_PATH = Path.Combine(SOLUTION_BASE_PATH, "TestArchives", "Original"); MISC_TEST_FILES_PATH = Path.Combine(SOLUTION_BASE_PATH, "TestArchives", "MiscTest"); - - SCRATCH_BASE_PATH = Path.Combine( - SOLUTION_BASE_PATH, - "TestArchives", - Guid.NewGuid().ToString() - ); - SCRATCH_FILES_PATH = Path.Combine(SCRATCH_BASE_PATH, "Scratch"); - SCRATCH2_FILES_PATH = Path.Combine(SCRATCH_BASE_PATH, "Scratch2"); - - Directory.CreateDirectory(SCRATCH_FILES_PATH); - Directory.CreateDirectory(SCRATCH2_FILES_PATH); } - public void Dispose() => Directory.Delete(SCRATCH_BASE_PATH, true); + private readonly TempDirectory _tempDirectory; + protected readonly string SCRATCH_FILES_PATH; + protected readonly string SCRATCH2_FILES_PATH; + + protected TestBase() + { + _tempDirectory = new TempDirectory("SharpCompress.Test"); + SCRATCH_FILES_PATH = _tempDirectory.GetDirectory("Scratch"); + SCRATCH2_FILES_PATH = _tempDirectory.GetDirectory("Scratch2"); + } + + // Always use async dispose since we have I/O and sync Dispose doesn't wait when using xunit. + public ValueTask DisposeAsync() => _tempDirectory.DisposeAsync(); + + public void CleanScratch() + { + _tempDirectory.ResetDirectory("Scratch"); + _tempDirectory.ResetDirectory("Scratch2"); + } + + protected string CreateScratchDirectory(string name) => + _tempDirectory.CreateDirectory(Path.Combine("Scratch", name)); + + protected string CreateScratch2Directory(string name) => + _tempDirectory.CreateDirectory(Path.Combine("Scratch2", name)); + + protected string GetScratchPath(params string[] parts) => + CombinePath(SCRATCH_FILES_PATH, parts); + + protected string GetScratch2Path(params string[] parts) => + CombinePath(SCRATCH2_FILES_PATH, parts); + + private static string CombinePath(string root, string[] parts) => + parts.Length == 0 ? root : Path.Combine(root, Path.Combine(parts)); public void VerifyFiles() { @@ -110,14 +130,14 @@ public class TestBase : IDisposable Assert.True(extracted.Contains(orig.Key)); CompareFilesByPath(orig.Single(), extracted[orig.Key].Single()); - CompareFilesByTimeAndAttribut(orig.Single(), extracted[orig.Key].Single()); + CompareFilesByTimeAndAttribute(orig.Single(), extracted[orig.Key].Single()); } } /// /// Verifies the files by extension also check modified time and attributes. /// - protected void VerifyFilesByExtensionEx() + private void VerifyFilesByExtensionEx() { var extracted = Directory .EnumerateFiles(SCRATCH_FILES_PATH, "*.*", SearchOption.AllDirectories) @@ -133,7 +153,7 @@ public class TestBase : IDisposable Assert.True(extracted.Contains(orig.Key)); CompareFilesByPath(orig.Single(), extracted[orig.Key].Single()); - CompareFilesByTimeAndAttribut(orig.Single(), extracted[orig.Key].Single()); + CompareFilesByTimeAndAttribute(orig.Single(), extracted[orig.Key].Single()); } } @@ -159,7 +179,7 @@ public class TestBase : IDisposable .EnumerateFiles(ORIGINAL_FILES_PATH, "*.*", SearchOption.AllDirectories) .ToLookup(path => Path.GetExtension(path)); - Assert.Equal(extracted.Count, original.Count); + Assert.Equal(original.Count, extracted.Count); foreach (var orig in original) { @@ -194,7 +214,7 @@ public class TestBase : IDisposable } } - protected void CompareFilesByTimeAndAttribut(string file1, string file2) + private void CompareFilesByTimeAndAttribute(string file1, string file2) { var fi1 = new FileInfo(file1); var fi2 = new FileInfo(file2); @@ -204,20 +224,20 @@ public class TestBase : IDisposable protected void CompareArchivesByPath(string file1, string file2, Encoding? encoding = null) { - var readerOptions = new ReaderOptions { LeaveStreamOpen = false }; + var readerOptions = ReaderOptions.ForExternalStream.WithLeaveStreamOpen(false); readerOptions.ArchiveEncoding.Default = encoding ?? Encoding.Default; //don't compare the order. OS X reads files from the file system in a different order therefore makes the archive ordering different var archive1Entries = new List(); var archive2Entries = new List(); - using (var archive1 = ReaderFactory.Open(File.OpenRead(file1), readerOptions)) - using (var archive2 = ReaderFactory.Open(File.OpenRead(file2), readerOptions)) + using (var archive1 = ReaderFactory.OpenReader(File.OpenRead(file1), readerOptions)) + using (var archive2 = ReaderFactory.OpenReader(File.OpenRead(file2), readerOptions)) { while (archive1.MoveToNextEntry()) { Assert.True(archive2.MoveToNextEntry()); - archive1Entries.Add(archive1.Entry.Key); - archive2Entries.Add(archive2.Entry.Key); + archive1Entries.Add(archive1.Entry.Key.NotNull()); + archive2Entries.Add(archive2.Entry.Key.NotNull()); } Assert.False(archive2.MoveToNextEntry()); } diff --git a/tests/SharpCompress.Test/UtilityTests.cs b/tests/SharpCompress.Test/UtilityTests.cs new file mode 100644 index 00000000..4dbe2f83 --- /dev/null +++ b/tests/SharpCompress.Test/UtilityTests.cs @@ -0,0 +1,853 @@ +using System; +using System.Buffers; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using SharpCompress.Common; +using Xunit; + +namespace SharpCompress.Test; + +public class UtilityTests +{ + #region URShift Tests + + [Fact] + public void URShift_Int_PositiveNumber_ShiftsCorrectly() + { + var result = Utility.URShift(16, 2); + Assert.Equal(4, result); + } + + [Fact] + public void URShift_Int_NegativeNumber_PerformsUnsignedShift() + { + // -1 in binary is all 1s (0xFFFFFFFF), shifted right by 1 should be 0x7FFFFFFF + var result = Utility.URShift(-1, 1); + Assert.Equal(int.MaxValue, result); + } + + [Fact] + public void URShift_Int_Zero_ReturnsZero() + { + var result = Utility.URShift(0, 5); + Assert.Equal(0, result); + } + + [Fact] + public void URShift_Long_PositiveNumber_ShiftsCorrectly() + { + var result = Utility.URShift(32L, 3); + Assert.Equal(4L, result); + } + + [Fact] + public void URShift_Long_NegativeNumber_PerformsUnsignedShift() + { + var result = Utility.URShift(-1L, 1); + Assert.Equal(long.MaxValue, result); + } + + [Fact] + public void URShift_Long_Zero_ReturnsZero() + { + var result = Utility.URShift(0L, 10); + Assert.Equal(0L, result); + } + + #endregion + + #region ReadFully Tests + + [Fact] + public void ReadFully_ByteArray_ReadsExactlyRequiredBytes() + { + var data = new byte[] { 1, 2, 3, 4, 5 }; + using var stream = new MemoryStream(data); + var buffer = new byte[5]; + + var result = stream.ReadFully(buffer); + + Assert.True(result); + Assert.Equal(data, buffer); + } + + [Fact] + public void ReadFully_ByteArray_ReturnsFalseWhenNotEnoughData() + { + var data = new byte[] { 1, 2, 3 }; + using var stream = new MemoryStream(data); + var buffer = new byte[5]; + + var result = stream.ReadFully(buffer); + + Assert.False(result); + } + + [Fact] + public void ReadFully_ByteArray_EmptyStream_ReturnsFalse() + { + using var stream = new MemoryStream(); + var buffer = new byte[5]; + + var result = stream.ReadFully(buffer); + + Assert.False(result); + } + + [Fact] + public void ReadFully_ByteArray_EmptyBuffer_ReturnsTrue() + { + var data = new byte[] { 1, 2, 3 }; + using var stream = new MemoryStream(data); + var buffer = Array.Empty(); + + var result = stream.ReadFully(buffer); + + Assert.True(result); + } + + [Fact] + public void ReadFully_Span_ReadsExactlyRequiredBytes() + { + var data = new byte[] { 1, 2, 3, 4, 5 }; + using var stream = new MemoryStream(data); + Span buffer = new byte[5]; + + var result = stream.ReadFully(buffer); + + Assert.True(result); + Assert.Equal(data, buffer.ToArray()); + } + + [Fact] + public void ReadFully_Span_ReturnsFalseWhenNotEnoughData() + { + var data = new byte[] { 1, 2, 3 }; + using var stream = new MemoryStream(data); + Span buffer = new byte[5]; + + var result = stream.ReadFully(buffer); + + Assert.False(result); + } + + [Fact] + public void ReadFully_Span_EmptyStream_ReturnsFalse() + { + using var stream = new MemoryStream(); + Span buffer = new byte[5]; + + var result = stream.ReadFully(buffer); + + Assert.False(result); + } + + [Fact] + public void ReadFully_Span_EmptyBuffer_ReturnsTrue() + { + var data = new byte[] { 1, 2, 3 }; + using var stream = new MemoryStream(data); + Span buffer = Array.Empty(); + + var result = stream.ReadFully(buffer); + + Assert.True(result); + } + + #endregion + + #region ReadByteAsync Tests + + [Fact] + public async ValueTask ReadByteAsync_ReadsOneByte() + { + var data = new byte[] { 42, 1, 2, 3 }; + using var stream = new MemoryStream(data); + using var reader = new BinaryReader(stream); + + var result = await reader.ReadByteAsync(); + + Assert.Equal(42, result); + Assert.Equal(1, stream.Position); + } + + [Fact] + public async ValueTask ReadByteAsync_EmptyStream_ThrowsIncompleteArchiveException() + { + using var stream = new MemoryStream(); + using var reader = new BinaryReader(stream); + + await Assert.ThrowsAsync(async () => + await reader.ReadByteAsync() + ); + } + + [Fact] + public async ValueTask ReadByteAsync_MultipleReads_ReadsSequentially() + { + var data = new byte[] { 1, 2, 3 }; + using var stream = new MemoryStream(data); + using var reader = new BinaryReader(stream); + + var first = await reader.ReadByteAsync(); + var second = await reader.ReadByteAsync(); + var third = await reader.ReadByteAsync(); + + Assert.Equal(1, first); + Assert.Equal(2, second); + Assert.Equal(3, third); + } + + #endregion + + #region ReadBytesAsync Tests + + [Fact] + public async ValueTask ReadBytesAsync_ReadsExactlyRequiredBytes() + { + var data = new byte[] { 1, 2, 3, 4, 5 }; + using var stream = new MemoryStream(data); + using var reader = new BinaryReader(stream); + + var result = await reader.ReadBytesAsync(3); + + Assert.Equal(new byte[] { 1, 2, 3 }, result); + Assert.Equal(3, stream.Position); + } + + [Fact] + public async ValueTask ReadBytesAsync_NotEnoughData_ThrowsIncompleteArchiveException() + { + var data = new byte[] { 1, 2, 3 }; + using var stream = new MemoryStream(data); + using var reader = new BinaryReader(stream); + + await Assert.ThrowsAsync(async () => + await reader.ReadBytesAsync(5) + ); + } + + [Fact] + public async ValueTask ReadBytesAsync_EmptyStream_ThrowsIncompleteArchiveException() + { + using var stream = new MemoryStream(); + using var reader = new BinaryReader(stream); + + await Assert.ThrowsAsync(async () => + await reader.ReadBytesAsync(1) + ); + } + + [Fact] + public async ValueTask ReadBytesAsync_ZeroBytes_ReturnsEmptyArray() + { + var data = new byte[] { 1, 2, 3 }; + using var stream = new MemoryStream(data); + using var reader = new BinaryReader(stream); + + var result = await reader.ReadBytesAsync(0); + + Assert.Empty(result); + Assert.Equal(0, stream.Position); + } + + #endregion + + #region Skip Tests + + [Fact] + public void Skip_SeekableStream_UsesSeek() + { + var data = new byte[] { 1, 2, 3, 4, 5 }; + using var stream = new MemoryStream(data); + + stream.Skip(3); + + Assert.Equal(3, stream.Position); + } + + [Fact] + public void Skip_SeekableStream_SkipsCorrectAmount() + { + var data = new byte[] { 1, 2, 3, 4, 5 }; + using var stream = new MemoryStream(data); + + stream.Skip(2); + var buffer = new byte[2]; + stream.Read(buffer); + + Assert.Equal(new byte[] { 3, 4 }, buffer); + } + + [Fact] + public void Skip_NonSeekableStream_SkipsCorrectAmount() + { + var data = new byte[] { 1, 2, 3, 4, 5 }; + using var seekableStream = new MemoryStream(data); + using var nonSeekableStream = new NonSeekableStream(seekableStream); + + nonSeekableStream.Skip(2); + var buffer = new byte[2]; + nonSeekableStream.Read(buffer); + + Assert.Equal(new byte[] { 3, 4 }, buffer); + } + + [Fact] + public void Skip_NonSeekableStream_SkipsBeyondStreamEnd() + { + var data = new byte[] { 1, 2, 3 }; + using var seekableStream = new MemoryStream(data); + using var nonSeekableStream = new NonSeekableStream(seekableStream); + + // Should not throw, just skip what's available + nonSeekableStream.Skip(10); + + Assert.Equal(-1, nonSeekableStream.ReadByte()); + } + + [Fact] + public void Skip_Parameterless_SkipsEntireStream() + { + var data = new byte[] { 1, 2, 3, 4, 5 }; + using var stream = new MemoryStream(data); + + stream.Skip(); + + Assert.Equal(-1, stream.ReadByte()); + } + + [Fact] + public void Skip_Zero_DoesNotMove() + { + var data = new byte[] { 1, 2, 3, 4, 5 }; + using var stream = new MemoryStream(data); + stream.Position = 2; + + stream.Skip(0); + + Assert.Equal(2, stream.Position); + } + + #endregion + + #region SetSize Tests + + [Fact] + public void SetSize_GrowsList_AddsZeroBytes() + { + var list = new List { 1, 2, 3 }; + + Utility.SetSize(list, 5); + + Assert.Equal(5, list.Count); + Assert.Equal(new byte[] { 1, 2, 3, 0, 0 }, list); + } + + [Fact] + public void SetSize_ShrinksListByOne() + { + var list = new List { 1, 2, 3, 4, 5 }; + + Utility.SetSize(list, 3); + + Assert.Equal(3, list.Count); + Assert.Equal(new byte[] { 1, 2, 3 }, list); + } + + [Fact] + public void SetSize_ToZero_ClearsAllItems() + { + var list = new List { 1, 2, 3 }; + + Utility.SetSize(list, 0); + + Assert.Empty(list); + } + + [Fact] + public void SetSize_SameSize_NoChange() + { + var list = new List { 1, 2, 3 }; + + Utility.SetSize(list, 3); + + Assert.Equal(3, list.Count); + Assert.Equal(new byte[] { 1, 2, 3 }, list); + } + + #endregion + + #region ForEach Tests + + [Fact] + public void ForEach_ExecutesActionForEachItem() + { + var items = new[] { 1, 2, 3, 4, 5 }; + var results = new List(); + + items.ForEach(x => results.Add(x)); + + Assert.Equal(items, results); + } + + [Fact] + public void ForEach_EmptyCollection_NoExecutions() + { + var items = Array.Empty(); + var count = 0; + + items.ForEach(x => count++); + + Assert.Equal(0, count); + } + + #endregion + + #region AsEnumerable Tests + + [Fact] + public void AsEnumerable_SingleItem_YieldsItem() + { + var item = 42; + + var result = item.AsEnumerable().ToList(); + + Assert.Single(result); + Assert.Equal(42, result[0]); + } + + [Fact] + public void AsEnumerable_String_YieldsString() + { + var item = "test"; + + var result = item.AsEnumerable().ToList(); + + Assert.Single(result); + Assert.Equal("test", result[0]); + } + + #endregion + + #region DosDateToDateTime Tests + + [Fact] + public void DosDateToDateTime_ValidDate_ConvertsCorrectly() + { + // DOS date format: year (7 bits) | month (4 bits) | day (5 bits) + // DOS time format: hour (5 bits) | minute (6 bits) | second (5 bits, in 2-second increments) + // This represents: 2020-01-15 10:30:20 (approximately) + ushort dosDate = (ushort)(((2020 - 1980) << 9) | (1 << 5) | 15); // 2020-01-15 + ushort dosTime = (ushort)((10 << 11) | (30 << 5) | 10); // 10:30:20 + + var result = Utility.DosDateToDateTime(dosDate, dosTime); + + Assert.Equal(2020, result.Year); + Assert.Equal(1, result.Month); + Assert.Equal(15, result.Day); + Assert.Equal(10, result.Hour); + Assert.Equal(30, result.Minute); + Assert.Equal(20, result.Second); + } + + [Fact] + public void DosDateToDateTime_InvalidDate_DefaultsTo1980_01_01() + { + ushort dosDate = ushort.MaxValue; + ushort dosTime = (ushort)((10 << 11) | (30 << 5) | 10); + + var result = Utility.DosDateToDateTime(dosDate, dosTime); + + Assert.Equal(1980, result.Year); + Assert.Equal(1, result.Month); + Assert.Equal(1, result.Day); + } + + [Fact] + public void DosDateToDateTime_InvalidTime_DefaultsToMidnight() + { + ushort dosDate = (ushort)(((2020 - 1980) << 9) | (1 << 5) | 15); + ushort dosTime = ushort.MaxValue; + + var result = Utility.DosDateToDateTime(dosDate, dosTime); + + Assert.Equal(0, result.Hour); + Assert.Equal(0, result.Minute); + Assert.Equal(0, result.Second); + } + + [Fact] + public void DosDateToDateTime_FromUint_ConvertsCorrectly() + { + ushort dosDate = (ushort)(((2020 - 1980) << 9) | (6 << 5) | 20); // 2020-06-20 + ushort dosTime = (ushort)((14 << 11) | (45 << 5) | 15); // 14:45:30 + uint combined = (uint)(dosDate << 16) | dosTime; + + var result = Utility.DosDateToDateTime(combined); + + Assert.Equal(2020, result.Year); + Assert.Equal(6, result.Month); + Assert.Equal(20, result.Day); + Assert.Equal(14, result.Hour); + Assert.Equal(45, result.Minute); + } + + #endregion + + #region DateTimeToDosTime Tests + + [Fact] + public void DateTimeToDosTime_ValidDateTime_ConvertsCorrectly() + { + //always do local time + var dt = new DateTime(2020, 6, 15, 14, 30, 20, DateTimeKind.Local); + + var result = Utility.DateTimeToDosTime(dt); + + // Verify we can convert back + var reversed = Utility.DosDateToDateTime(result); + Assert.Equal(2020, reversed.Year); + Assert.Equal(6, reversed.Month); + Assert.Equal(15, reversed.Day); + Assert.Equal(14, reversed.Hour); + Assert.Equal(30, reversed.Minute); + // Seconds are rounded down to nearest even number in DOS format + Assert.True(reversed.Second == 20 || reversed.Second == 18); + } + + [Fact] + public void DateTimeToDosTime_NullDateTime_ReturnsZero() + { + DateTime? dt = null; + + var result = Utility.DateTimeToDosTime(dt); + + Assert.Equal(0u, result); + } + + #endregion + + #region UnixTimeToDateTime Tests + + [Fact] + public void UnixTimeToDateTime_Zero_Returns1970_01_01() + { + var result = Utility.UnixTimeToDateTime(0); + + Assert.Equal(1970, result.Year); + Assert.Equal(1, result.Month); + Assert.Equal(1, result.Day); + Assert.Equal(0, result.Hour); + Assert.Equal(0, result.Minute); + Assert.Equal(0, result.Second); + } + + [Fact] + public void UnixTimeToDateTime_ValidTimestamp_ConvertsCorrectly() + { + // January 1, 2000 00:00:00 UTC is 946684800 seconds after epoch + var result = Utility.UnixTimeToDateTime(946684800); + + Assert.Equal(2000, result.Year); + Assert.Equal(1, result.Month); + Assert.Equal(1, result.Day); + } + + [Fact] + public void UnixTimeToDateTime_NegativeTimestamp_ReturnsBeforeEpoch() + { + // -86400 is one day before epoch + var result = Utility.UnixTimeToDateTime(-86400); + + Assert.Equal(1969, result.Year); + Assert.Equal(12, result.Month); + Assert.Equal(31, result.Day); + } + + #endregion + + #region TransferTo Tests + + [Fact] + public void TransferTo_WithMaxLength_TransfersCorrectAmount() + { + var sourceData = new byte[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 }; + using var source = new MemoryStream(sourceData); + using var destination = new MemoryStream(); + + var transferred = source.TransferTo(destination, 5, null); + + Assert.Equal(5, transferred); + Assert.Equal(new byte[] { 1, 2, 3, 4, 5 }, destination.ToArray()); + } + + [Fact] + public void TransferTo_SourceSmallerThanMax_TransfersAll() + { + var sourceData = new byte[] { 1, 2, 3 }; + using var source = new MemoryStream(sourceData); + using var destination = new MemoryStream(); + + var transferred = source.TransferTo(destination, 100, null); + + Assert.Equal(3, transferred); + Assert.Equal(sourceData, destination.ToArray()); + } + + [Fact] + public void TransferTo_EmptySource_TransfersNothing() + { + using var source = new MemoryStream(); + using var destination = new MemoryStream(); + + var transferred = source.TransferTo(destination, 100, null); + + Assert.Equal(0, transferred); + Assert.Empty(destination.ToArray()); + } + + #endregion + + #region SwapUINT32 Tests + + [Fact] + public void SwapUINT32_SimpleValue_SwapsEndianness() + { + uint value = 0x12345678; + + var result = Utility.SwapUINT32(value); + + Assert.Equal(0x78563412u, result); + } + + [Fact] + public void SwapUINT32_Zero_ReturnsZero() + { + var result = Utility.SwapUINT32(0); + + Assert.Equal(0u, result); + } + + [Fact] + public void SwapUINT32_MaxValue_SwapsCorrectly() + { + var result = Utility.SwapUINT32(uint.MaxValue); + + Assert.Equal(uint.MaxValue, result); + } + + [Fact] + public void SwapUINT32_Involution_SwappingTwiceReturnsOriginal() + { + uint value = 0x12345678; + + var result = Utility.SwapUINT32(Utility.SwapUINT32(value)); + + Assert.Equal(value, result); + } + + #endregion + + #region SetLittleUInt32 Tests + + [Fact] + public void SetLittleUInt32_InsertsValueCorrectly() + { + byte[] buffer = new byte[10]; + uint value = 0x12345678; + + Utility.SetLittleUInt32(ref buffer, value, 2); + + Assert.Equal(0x78, buffer[2]); + Assert.Equal(0x56, buffer[3]); + Assert.Equal(0x34, buffer[4]); + Assert.Equal(0x12, buffer[5]); + } + + [Fact] + public void SetLittleUInt32_AtOffset_InsertsBehindOffset() + { + byte[] buffer = new byte[10]; + uint value = 0xDEADBEEF; + + Utility.SetLittleUInt32(ref buffer, value, 5); + + Assert.Equal(0xEF, buffer[5]); + Assert.Equal(0xBE, buffer[6]); + Assert.Equal(0xAD, buffer[7]); + Assert.Equal(0xDE, buffer[8]); + } + + #endregion + + #region SetBigUInt32 Tests + + [Fact] + public void SetBigUInt32_InsertsValueCorrectly() + { + byte[] buffer = new byte[10]; + uint value = 0x12345678; + + Utility.SetBigUInt32(ref buffer, value, 2); + + Assert.Equal(0x12, buffer[2]); + Assert.Equal(0x34, buffer[3]); + Assert.Equal(0x56, buffer[4]); + Assert.Equal(0x78, buffer[5]); + } + + [Fact] + public void SetBigUInt32_AtOffset_InsertsBehindOffset() + { + byte[] buffer = new byte[10]; + uint value = 0xDEADBEEF; + + Utility.SetBigUInt32(ref buffer, value, 5); + + Assert.Equal(0xDE, buffer[5]); + Assert.Equal(0xAD, buffer[6]); + Assert.Equal(0xBE, buffer[7]); + Assert.Equal(0xEF, buffer[8]); + } + + #endregion + + #region ReplaceInvalidFileNameChars Tests + +#if WINDOWS + [Theory] + [InlineData("valid_filename.txt", "valid_filename.txt")] + [InlineData("filetest.txt", "file_name_test.txt")] + [InlineData("<>:\"|?*", "_______")] + public void ReplaceInvalidFileNameChars_Windows(string fileName, string expected) + { + var result = Utility.ReplaceInvalidFileNameChars(fileName); + + Assert.Equal(expected, result); + } + +#else + [Theory] + [InlineData("valid_filename.txt", "valid_filename.txt")] + [InlineData("filetest.txt", "filetest.txt")] + [InlineData("<>:\"|?*", "<>:\"|?*")] + public void ReplaceInvalidFileNameChars_NonWindows(string fileName, string expected) + { + var result = Utility.ReplaceInvalidFileNameChars(fileName); + + Assert.Equal(expected, result); + } +#endif + + #endregion + + #region ToReadOnly Tests + + [Fact] + public void ToReadOnly_IList_ReturnsReadOnlyCollection() + { + var list = new List { 1, 2, 3, 4, 5 }; + + var result = list.ToReadOnly(); + + Assert.Equal(5, result.Count); + Assert.Equal(1, result[0]); + Assert.Equal(5, result[4]); + } + + [Fact] + public void ToReadOnly_EmptyList_ReturnsEmptyReadOnlyCollection() + { + var list = new List(); + + var result = list.ToReadOnly(); + + Assert.Empty(result); + } + + #endregion + + #region TrimNulls Tests + + [Fact] + public void TrimNulls_StringWithNulls_ReplacesAndTrims() + { + var input = " hello\0world\0 "; + + var result = Utility.TrimNulls(input); + + Assert.Equal("hello world", result); + } + + [Fact] + public void TrimNulls_StringWithoutNulls_TrimsWhitespace() + { + var input = " hello world "; + + var result = Utility.TrimNulls(input); + + Assert.Equal("hello world", result); + } + + [Fact] + public void TrimNulls_OnlyNulls_ReturnsEmpty() + { + var input = "\0\0\0"; + + var result = Utility.TrimNulls(input); + + Assert.Empty(result); + } + + #endregion +} + +/// +/// Helper class for testing non-seekable streams +/// +internal class NonSeekableStream : Stream +{ + private readonly Stream _inner; + + public NonSeekableStream(Stream inner) + { + _inner = inner; + } + + public override bool CanRead => _inner.CanRead; + public override bool CanSeek => false; // Force non-seekable + public override bool CanWrite => _inner.CanWrite; + public override long Length => _inner.Length; + public override long Position + { + get => _inner.Position; + set => throw new NotSupportedException("Stream is not seekable"); + } + + public override void Flush() => _inner.Flush(); + + public override int Read(byte[] buffer, int offset, int count) => + _inner.Read(buffer, offset, count); + + public override long Seek(long offset, SeekOrigin origin) => + throw new NotSupportedException("Stream is not seekable"); + + public override void SetLength(long value) => + throw new NotSupportedException("Stream is not seekable"); + + public override void Write(byte[] buffer, int offset, int count) => + _inner.Write(buffer, offset, count); + + protected override void Dispose(bool disposing) + { + if (disposing) + { + _inner.Dispose(); + } + base.Dispose(disposing); + } +} diff --git a/tests/SharpCompress.Test/WriterFactoryTests.cs b/tests/SharpCompress.Test/WriterFactoryTests.cs new file mode 100644 index 00000000..ca7a45a5 --- /dev/null +++ b/tests/SharpCompress.Test/WriterFactoryTests.cs @@ -0,0 +1,34 @@ +using System; +using System.IO; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Test.Mocks; +using SharpCompress.Writers; +using Xunit; + +namespace SharpCompress.Test; + +public class WriterFactoryTests +{ + [Fact] + public void OpenWriter_Stream_Throws_On_Unwritable_Stream() + { + using var unwritable = new TestStream(new MemoryStream(), true, false, true); + + Assert.Throws(() => + WriterFactory.OpenWriter(unwritable, ArchiveType.Zip, WriterOptions.ForZip()) + ); + } + + [Fact] + public async ValueTask OpenAsyncWriter_Stream_Throws_On_Unwritable_Stream() + { + using var unwritable = new TestStream(new MemoryStream(), true, false, true); + + await Assert.ThrowsAsync(() => + WriterFactory + .OpenAsyncWriter(unwritable, ArchiveType.Zip, WriterOptions.ForZip()) + .AsTask() + ); + } +} diff --git a/tests/SharpCompress.Test/WriterTests.cs b/tests/SharpCompress.Test/WriterTests.cs index 64ed6d2c..1142f7ac 100644 --- a/tests/SharpCompress.Test/WriterTests.cs +++ b/tests/SharpCompress.Test/WriterTests.cs @@ -1,17 +1,20 @@ using System.IO; using System.Text; +using System.Threading; +using System.Threading.Tasks; using SharpCompress.Common; using SharpCompress.IO; using SharpCompress.Readers; +using SharpCompress.Test.Mocks; using SharpCompress.Writers; namespace SharpCompress.Test; public class WriterTests : TestBase { - private readonly ArchiveType type; + private readonly ArchiveType _type; - protected WriterTests(ArchiveType type) => this.type = type; + protected WriterTests(ArchiveType type) => _type = type; protected void Write( CompressionType compressionType, @@ -22,11 +25,11 @@ public class WriterTests : TestBase { using (Stream stream = File.OpenWrite(Path.Combine(SCRATCH2_FILES_PATH, archive))) { - var writerOptions = new WriterOptions(compressionType) { LeaveStreamOpen = true, }; + var writerOptions = new WriterOptions(compressionType) { LeaveStreamOpen = true }; writerOptions.ArchiveEncoding.Default = encoding ?? Encoding.Default; - using var writer = WriterFactory.Open(stream, type, writerOptions); + using var writer = WriterFactory.OpenWriter(stream, _type, writerOptions); writer.WriteAll(ORIGINAL_FILES_PATH, "*", SearchOption.AllDirectories); } CompareArchivesByPath( @@ -36,14 +39,69 @@ public class WriterTests : TestBase using (Stream stream = File.OpenRead(Path.Combine(SCRATCH2_FILES_PATH, archive))) { - var readerOptions = new ReaderOptions(); + var readerOptions = ReaderOptions.ForExternalStream; readerOptions.ArchiveEncoding.Default = encoding ?? Encoding.Default; - using var reader = ReaderFactory.Open(NonDisposingStream.Create(stream), readerOptions); - reader.WriteAllToDirectory( + using var reader = ReaderFactory.OpenReader( + SharpCompressStream.CreateNonDisposing(stream), + readerOptions + ); + reader.WriteAllToDirectory(SCRATCH_FILES_PATH); + } + VerifyFiles(); + } + + protected async Task WriteAsync( + CompressionType compressionType, + string archive, + string archiveToVerifyAgainst, + Encoding? encoding = null, + CancellationToken cancellationToken = default + ) + { + using ( + Stream stream = new AsyncOnlyStream( + File.OpenWrite(Path.Combine(SCRATCH2_FILES_PATH, archive)) + ) + ) + { + var writerOptions = new WriterOptions(compressionType) { LeaveStreamOpen = true }; + + writerOptions.ArchiveEncoding.Default = encoding ?? Encoding.Default; + + await using var writer = await WriterFactory.OpenAsyncWriter( + stream, + _type, + writerOptions, + cancellationToken + ); + await writer.WriteAllAsync( + ORIGINAL_FILES_PATH, + "*", + SearchOption.AllDirectories, + cancellationToken + ); + } + CompareArchivesByPath( + Path.Combine(SCRATCH2_FILES_PATH, archive), + Path.Combine(TEST_ARCHIVES_PATH, archiveToVerifyAgainst) + ); + + using (Stream stream = File.OpenRead(Path.Combine(SCRATCH2_FILES_PATH, archive))) + { + var readerOptions = ReaderOptions.ForExternalStream; + + readerOptions.ArchiveEncoding.Default = encoding ?? Encoding.Default; + + await using var reader = await ReaderFactory.OpenAsyncReader( + new AsyncOnlyStream(SharpCompressStream.CreateNonDisposing(stream)), + readerOptions, + cancellationToken + ); + await reader.WriteAllToDirectoryAsync( SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true } + cancellationToken: cancellationToken ); } VerifyFiles(); diff --git a/tests/SharpCompress.Test/Xz/Crc32Tests.cs b/tests/SharpCompress.Test/Xz/Crc32Tests.cs index 52b71078..eaa0b100 100644 --- a/tests/SharpCompress.Test/Xz/Crc32Tests.cs +++ b/tests/SharpCompress.Test/Xz/Crc32Tests.cs @@ -6,16 +6,16 @@ namespace SharpCompress.Test.Xz; public class Crc32Tests { - private const string SimpleString = @"The quick brown fox jumps over the lazy dog."; - private readonly byte[] SimpleBytes = Encoding.ASCII.GetBytes(SimpleString); - private const string SimpleString2 = + private const string SIMPLE_STRING = @"The quick brown fox jumps over the lazy dog."; + private readonly byte[] _simpleBytes = Encoding.ASCII.GetBytes(SIMPLE_STRING); + private const string SIMPLE_STRING2 = @"Life moves pretty fast. If you don't stop and look around once in a while, you could miss it."; - private readonly byte[] SimpleBytes2 = Encoding.ASCII.GetBytes(SimpleString2); + private readonly byte[] _simpleBytes2 = Encoding.ASCII.GetBytes(SIMPLE_STRING2); [Fact] public void ShortAsciiString() { - var actual = Crc32.Compute(SimpleBytes); + var actual = Crc32.Compute(_simpleBytes); Assert.Equal((uint)0x519025e9, actual); } @@ -23,7 +23,7 @@ public class Crc32Tests [Fact] public void ShortAsciiString2() { - var actual = Crc32.Compute(SimpleBytes2); + var actual = Crc32.Compute(_simpleBytes2); Assert.Equal((uint)0x6ee3ad88, actual); } diff --git a/tests/SharpCompress.Test/Xz/Crc64Tests.cs b/tests/SharpCompress.Test/Xz/Crc64Tests.cs index 79912e48..e2338c64 100644 --- a/tests/SharpCompress.Test/Xz/Crc64Tests.cs +++ b/tests/SharpCompress.Test/Xz/Crc64Tests.cs @@ -6,16 +6,16 @@ namespace SharpCompress.Test.Xz; public class Crc64Tests { - private const string SimpleString = @"The quick brown fox jumps over the lazy dog."; - private readonly byte[] SimpleBytes = Encoding.ASCII.GetBytes(SimpleString); - private const string SimpleString2 = + private const string SIMPLE_STRING = @"The quick brown fox jumps over the lazy dog."; + private readonly byte[] _simpleBytes = Encoding.ASCII.GetBytes(SIMPLE_STRING); + private const string SIMPLE_STRING2 = @"Life moves pretty fast. If you don't stop and look around once in a while, you could miss it."; - private readonly byte[] SimpleBytes2 = Encoding.ASCII.GetBytes(SimpleString2); + private readonly byte[] _simpleBytes2 = Encoding.ASCII.GetBytes(SIMPLE_STRING2); [Fact] public void ShortAsciiString() { - var actual = Crc64.Compute(SimpleBytes); + var actual = Crc64.Compute(_simpleBytes); Assert.Equal((ulong)0x7E210EB1B03E5A1D, actual); } @@ -23,8 +23,16 @@ public class Crc64Tests [Fact] public void ShortAsciiString2() { - var actual = Crc64.Compute(SimpleBytes2); + var actual = Crc64.Compute(_simpleBytes2); Assert.Equal((ulong)0x416B4150508661EE, actual); } + + [Fact] + public void XzCheckString() + { + var actual = Crc64.ComputeXz(Encoding.ASCII.GetBytes("123456789")); + + Assert.Equal(0x995DC9BBDF1939FAUL, actual); + } } diff --git a/tests/SharpCompress.Test/Xz/Filters/BCJTests.cs b/tests/SharpCompress.Test/Xz/Filters/BCJTests.cs index 427a7b98..a53565b7 100644 --- a/tests/SharpCompress.Test/Xz/Filters/BCJTests.cs +++ b/tests/SharpCompress.Test/Xz/Filters/BCJTests.cs @@ -3,62 +3,62 @@ * */ -using System.IO; +using SharpCompress.Common; using SharpCompress.Compressors.Xz.Filters; using Xunit; namespace SharpCompress.Test.Xz.Filters; -public class BCJTests : XZTestsBase +public class BcjTests : XzTestsBase { - private readonly ArmFilter armFilter; - private readonly ArmThumbFilter armtFilter; - private readonly IA64Filter ia64Filter; - private readonly PowerPCFilter ppcFilter; - private readonly SparcFilter sparcFilter; - private readonly X86Filter x86Filter; + private readonly ArmFilter _armFilter; + private readonly ArmThumbFilter _armtFilter; + private readonly IA64Filter _ia64Filter; + private readonly PowerPCFilter _ppcFilter; + private readonly SparcFilter _sparcFilter; + private readonly X86Filter _x86Filter; - public BCJTests() + public BcjTests() { - armFilter = new ArmFilter(); - armtFilter = new ArmThumbFilter(); - ia64Filter = new IA64Filter(); - ppcFilter = new PowerPCFilter(); - sparcFilter = new SparcFilter(); - x86Filter = new X86Filter(); + _armFilter = new ArmFilter(); + _armtFilter = new ArmThumbFilter(); + _ia64Filter = new IA64Filter(); + _ppcFilter = new PowerPCFilter(); + _sparcFilter = new SparcFilter(); + _x86Filter = new X86Filter(); } [Fact] public void IsOnlyAllowedLast() { - Assert.False(armFilter.AllowAsLast); - Assert.True(armFilter.AllowAsNonLast); + Assert.False(_armFilter.AllowAsLast); + Assert.True(_armFilter.AllowAsNonLast); - Assert.False(armtFilter.AllowAsLast); - Assert.True(armtFilter.AllowAsNonLast); + Assert.False(_armtFilter.AllowAsLast); + Assert.True(_armtFilter.AllowAsNonLast); - Assert.False(ia64Filter.AllowAsLast); - Assert.True(ia64Filter.AllowAsNonLast); + Assert.False(_ia64Filter.AllowAsLast); + Assert.True(_ia64Filter.AllowAsNonLast); - Assert.False(ppcFilter.AllowAsLast); - Assert.True(ppcFilter.AllowAsNonLast); + Assert.False(_ppcFilter.AllowAsLast); + Assert.True(_ppcFilter.AllowAsNonLast); - Assert.False(sparcFilter.AllowAsLast); - Assert.True(sparcFilter.AllowAsNonLast); + Assert.False(_sparcFilter.AllowAsLast); + Assert.True(_sparcFilter.AllowAsNonLast); - Assert.False(x86Filter.AllowAsLast); - Assert.True(x86Filter.AllowAsNonLast); + Assert.False(_x86Filter.AllowAsLast); + Assert.True(_x86Filter.AllowAsNonLast); } [Fact] public void ChangesStreamSize() { - Assert.False(armFilter.ChangesDataSize); - Assert.False(armtFilter.ChangesDataSize); - Assert.False(ia64Filter.ChangesDataSize); - Assert.False(ppcFilter.ChangesDataSize); - Assert.False(sparcFilter.ChangesDataSize); - Assert.False(x86Filter.ChangesDataSize); + Assert.False(_armFilter.ChangesDataSize); + Assert.False(_armtFilter.ChangesDataSize); + Assert.False(_ia64Filter.ChangesDataSize); + Assert.False(_ppcFilter.ChangesDataSize); + Assert.False(_sparcFilter.ChangesDataSize); + Assert.False(_x86Filter.ChangesDataSize); } [Theory] @@ -66,23 +66,23 @@ public class BCJTests : XZTestsBase [InlineData(new byte[] { 0, 0, 0, 0, 0 })] public void OnlyAcceptsOneByte(byte[] bytes) { - InvalidDataException ex; - ex = Assert.Throws(() => armFilter.Init(bytes)); + InvalidFormatException ex; + ex = Assert.Throws(() => _armFilter.Init(bytes)); Assert.Equal("ARM properties unexpected length", ex.Message); - ex = Assert.Throws(() => armtFilter.Init(bytes)); + ex = Assert.Throws(() => _armtFilter.Init(bytes)); Assert.Equal("ARM Thumb properties unexpected length", ex.Message); - ex = Assert.Throws(() => ia64Filter.Init(bytes)); + ex = Assert.Throws(() => _ia64Filter.Init(bytes)); Assert.Equal("IA64 properties unexpected length", ex.Message); - ex = Assert.Throws(() => ppcFilter.Init(bytes)); + ex = Assert.Throws(() => _ppcFilter.Init(bytes)); Assert.Equal("PPC properties unexpected length", ex.Message); - ex = Assert.Throws(() => sparcFilter.Init(bytes)); + ex = Assert.Throws(() => _sparcFilter.Init(bytes)); Assert.Equal("SPARC properties unexpected length", ex.Message); - ex = Assert.Throws(() => x86Filter.Init(bytes)); + ex = Assert.Throws(() => _x86Filter.Init(bytes)); Assert.Equal("X86 properties unexpected length", ex.Message); } } diff --git a/tests/SharpCompress.Test/Xz/Filters/Lzma2Tests.cs b/tests/SharpCompress.Test/Xz/Filters/Lzma2Tests.cs index 83eed21c..e6ad48d6 100644 --- a/tests/SharpCompress.Test/Xz/Filters/Lzma2Tests.cs +++ b/tests/SharpCompress.Test/Xz/Filters/Lzma2Tests.cs @@ -1,25 +1,25 @@ using System; -using Xunit; -using System.IO; +using SharpCompress.Common; using SharpCompress.Compressors.Xz.Filters; +using Xunit; namespace SharpCompress.Test.Xz.Filters; -public class Lzma2Tests : XZTestsBase +public class Lzma2Tests : XzTestsBase { - private readonly Lzma2Filter filter; + private readonly Lzma2Filter _filter; - public Lzma2Tests() => filter = new Lzma2Filter(); + public Lzma2Tests() => _filter = new Lzma2Filter(); [Fact] public void IsOnlyAllowedLast() { - Assert.True(filter.AllowAsLast); - Assert.False(filter.AllowAsNonLast); + Assert.True(_filter.AllowAsLast); + Assert.False(_filter.AllowAsNonLast); } [Fact] - public void ChangesStreamSize() => Assert.True(filter.ChangesDataSize); + public void ChangesStreamSize() => Assert.True(_filter.ChangesDataSize); [Theory] [InlineData(0, (uint)4 * 1024)] @@ -31,18 +31,18 @@ public class Lzma2Tests : XZTestsBase [InlineData(40, (uint)(1024 * 1024 * 1024 - 1) * 4 + 3)] public void CalculatesDictionarySize(byte inByte, uint dicSize) { - filter.Init(new[] { inByte }); - Assert.Equal(filter.DictionarySize, dicSize); + _filter.Init([inByte]); + Assert.Equal(_filter.DictionarySize, dicSize); } [Fact] public void CalculatesDictionarySizeError() { uint temp; - filter.Init(new byte[] { 41 }); - var ex = Assert.Throws(() => + _filter.Init([41]); + var ex = Assert.Throws(() => { - temp = filter.DictionarySize; + temp = _filter.DictionarySize; }); Assert.Equal("Dictionary size greater than UInt32.Max", ex.Message); } @@ -52,14 +52,14 @@ public class Lzma2Tests : XZTestsBase [InlineData(new byte[] { 0, 0 })] public void OnlyAcceptsOneByte(byte[] bytes) { - var ex = Assert.Throws(() => filter.Init(bytes)); + var ex = Assert.Throws(() => _filter.Init(bytes)); Assert.Equal("LZMA properties unexpected length", ex.Message); } [Fact] public void ReservedBytesThrow() { - var ex = Assert.Throws(() => filter.Init(new byte[] { 0xC0 })); + var ex = Assert.Throws(() => _filter.Init([0xC0])); Assert.Equal("Reserved bits used in LZMA properties", ex.Message); } } diff --git a/tests/SharpCompress.Test/Xz/XZBlockAsyncTests.cs b/tests/SharpCompress.Test/Xz/XZBlockAsyncTests.cs new file mode 100644 index 00000000..9d59d6d4 --- /dev/null +++ b/tests/SharpCompress.Test/Xz/XZBlockAsyncTests.cs @@ -0,0 +1,125 @@ +using System.IO; +using System.Text; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Compressors.Xz; +using Xunit; + +namespace SharpCompress.Test.Xz; + +public class XzBlockAsyncTests : XzTestsBase +{ + protected override void Rewind(Stream stream) => stream.Position = 12; + + protected override void RewindIndexed(Stream stream) => stream.Position = 12; + + private static async Task ReadBytesAsync(XZBlock block, int bytesToRead) + { + var buffer = new byte[bytesToRead]; + var read = await block.ReadAsync(buffer, 0, bytesToRead).ConfigureAwait(false); + if (read != bytesToRead) + { + throw new EndOfStreamException(); + } + + return buffer; + } + + [Fact] + public async ValueTask OnFindIndexBlockThrowAsync() + { + var bytes = new byte[] { 0 }; + using Stream indexBlockStream = new MemoryStream(bytes); + using var xzBlock = new XZBlock(indexBlockStream, CheckType.CRC64, 8); + await Assert.ThrowsAsync(async () => + { + await ReadBytesAsync(xzBlock, 1).ConfigureAwait(false); + }); + } + + [Fact] + public async ValueTask CrcIncorrectThrowsAsync() + { + var bytes = (byte[])Compressed.Clone(); + bytes[20]++; + using Stream badCrcStream = new MemoryStream(bytes); + Rewind(badCrcStream); + using var xzBlock = new XZBlock(badCrcStream, CheckType.CRC64, 8); + var ex = await Assert.ThrowsAsync(async () => + { + await ReadBytesAsync(xzBlock, 1).ConfigureAwait(false); + }); + Assert.Equal("Block header corrupt", ex.Message); + } + + [Fact] + public async ValueTask CanReadMAsync() + { + using var xzBlock = new XZBlock(CompressedStream, CheckType.CRC64, 8); + Assert.Equal( + Encoding.ASCII.GetBytes("M"), + await ReadBytesAsync(xzBlock, 1).ConfigureAwait(false) + ); + } + + [Fact] + public async ValueTask CanReadMaryAsync() + { + using var xzBlock = new XZBlock(CompressedStream, CheckType.CRC64, 8); + Assert.Equal( + Encoding.ASCII.GetBytes("M"), + await ReadBytesAsync(xzBlock, 1).ConfigureAwait(false) + ); + Assert.Equal( + Encoding.ASCII.GetBytes("a"), + await ReadBytesAsync(xzBlock, 1).ConfigureAwait(false) + ); + Assert.Equal( + Encoding.ASCII.GetBytes("ry"), + await ReadBytesAsync(xzBlock, 2).ConfigureAwait(false) + ); + } + + [Fact] + public async ValueTask CanReadPoemWithStreamReaderAsync() + { + using var xzBlock = new XZBlock(CompressedStream, CheckType.CRC64, 8); + using var sr = new StreamReader(xzBlock); + Assert.Equal(Original, await sr.ReadToEndAsync().ConfigureAwait(false)); + } + + [Fact] + public async ValueTask NoopWhenNoPaddingAsync() + { + // CompressedStream's only block has no padding. + using var xzBlock = new XZBlock(CompressedStream, CheckType.CRC64, 8); + using var sr = new StreamReader(xzBlock); + await sr.ReadToEndAsync().ConfigureAwait(false); + Assert.Equal(0L, CompressedStream.Position % 4L); + } + + [Fact] + public async ValueTask SkipsPaddingWhenPresentAsync() + { + // CompressedIndexedStream uses CRC32 checks. + using var xzBlock = new XZBlock(CompressedIndexedStream, CheckType.CRC32, 4); + using var sr = new StreamReader(xzBlock); + await sr.ReadToEndAsync().ConfigureAwait(false); + Assert.Equal(0L, CompressedIndexedStream.Position % 4L); + } + + [Fact] + public async ValueTask HandlesPaddingInUnalignedBlockAsync() + { + var compressedUnaligned = new byte[Compressed.Length + 1]; + Compressed.CopyTo(compressedUnaligned, 1); + var compressedUnalignedStream = new MemoryStream(compressedUnaligned); + compressedUnalignedStream.Position = 13; + + // Compressed's only block has no padding. + using var xzBlock = new XZBlock(compressedUnalignedStream, CheckType.CRC64, 8); + using var sr = new StreamReader(xzBlock); + await sr.ReadToEndAsync().ConfigureAwait(false); + Assert.Equal(1L, compressedUnalignedStream.Position % 4L); + } +} diff --git a/tests/SharpCompress.Test/Xz/XZBlockTests.cs b/tests/SharpCompress.Test/Xz/XZBlockTests.cs index 87e68d93..9b6efe6e 100644 --- a/tests/SharpCompress.Test/Xz/XZBlockTests.cs +++ b/tests/SharpCompress.Test/Xz/XZBlockTests.cs @@ -1,11 +1,12 @@ using System.IO; using System.Text; +using SharpCompress.Common; using SharpCompress.Compressors.Xz; using Xunit; namespace SharpCompress.Test.Xz; -public class XZBlockTests : XZTestsBase +public class XzBlockTests : XzTestsBase { protected override void Rewind(Stream stream) => stream.Position = 12; @@ -28,10 +29,10 @@ public class XZBlockTests : XZTestsBase { var bytes = new byte[] { 0 }; using Stream indexBlockStream = new MemoryStream(bytes); - var XZBlock = new XZBlock(indexBlockStream, CheckType.CRC64, 8); + using var xzBlock = new XZBlock(indexBlockStream, CheckType.CRC64, 8); Assert.Throws(() => { - ReadBytes(XZBlock, 1); + ReadBytes(xzBlock, 1); }); } @@ -42,10 +43,10 @@ public class XZBlockTests : XZTestsBase bytes[20]++; using Stream badCrcStream = new MemoryStream(bytes); Rewind(badCrcStream); - var XZBlock = new XZBlock(badCrcStream, CheckType.CRC64, 8); - var ex = Assert.Throws(() => + using var xzBlock = new XZBlock(badCrcStream, CheckType.CRC64, 8); + var ex = Assert.Throws(() => { - ReadBytes(XZBlock, 1); + ReadBytes(xzBlock, 1); }); Assert.Equal("Block header corrupt", ex.Message); } @@ -53,33 +54,33 @@ public class XZBlockTests : XZTestsBase [Fact] public void CanReadM() { - var XZBlock = new XZBlock(CompressedStream, CheckType.CRC64, 8); - Assert.Equal(Encoding.ASCII.GetBytes("M"), ReadBytes(XZBlock, 1)); + using var xzBlock = new XZBlock(CompressedStream, CheckType.CRC64, 8); + Assert.Equal(Encoding.ASCII.GetBytes("M"), ReadBytes(xzBlock, 1)); } [Fact] public void CanReadMary() { - var XZBlock = new XZBlock(CompressedStream, CheckType.CRC64, 8); - Assert.Equal(Encoding.ASCII.GetBytes("M"), ReadBytes(XZBlock, 1)); - Assert.Equal(Encoding.ASCII.GetBytes("a"), ReadBytes(XZBlock, 1)); - Assert.Equal(Encoding.ASCII.GetBytes("ry"), ReadBytes(XZBlock, 2)); + using var xzBlock = new XZBlock(CompressedStream, CheckType.CRC64, 8); + Assert.Equal(Encoding.ASCII.GetBytes("M"), ReadBytes(xzBlock, 1)); + Assert.Equal(Encoding.ASCII.GetBytes("a"), ReadBytes(xzBlock, 1)); + Assert.Equal(Encoding.ASCII.GetBytes("ry"), ReadBytes(xzBlock, 2)); } [Fact] public void CanReadPoemWithStreamReader() { - var XZBlock = new XZBlock(CompressedStream, CheckType.CRC64, 8); - var sr = new StreamReader(XZBlock); - Assert.Equal(sr.ReadToEnd(), Original); + var xzBlock = new XZBlock(CompressedStream, CheckType.CRC64, 8); + using var sr = new StreamReader(xzBlock); + Assert.Equal(Original, sr.ReadToEnd()); } [Fact] public void NoopWhenNoPadding() { // CompressedStream's only block has no padding. - var XZBlock = new XZBlock(CompressedStream, CheckType.CRC64, 8); - var sr = new StreamReader(XZBlock); + using var xzBlock = new XZBlock(CompressedStream, CheckType.CRC64, 8); + using var sr = new StreamReader(xzBlock); sr.ReadToEnd(); Assert.Equal(0L, CompressedStream.Position % 4L); } @@ -87,10 +88,25 @@ public class XZBlockTests : XZTestsBase [Fact] public void SkipsPaddingWhenPresent() { - // CompressedIndexedStream's first block has 1-byte padding. - var XZBlock = new XZBlock(CompressedIndexedStream, CheckType.CRC64, 8); - var sr = new StreamReader(XZBlock); + // CompressedIndexedStream uses CRC32 checks. + using var xzBlock = new XZBlock(CompressedIndexedStream, CheckType.CRC32, 4); + using var sr = new StreamReader(xzBlock); sr.ReadToEnd(); Assert.Equal(0L, CompressedIndexedStream.Position % 4L); } + + [Fact] + public void HandlesPaddingInUnalignedBlock() + { + var compressedUnaligned = new byte[Compressed.Length + 1]; + Compressed.CopyTo(compressedUnaligned, 1); + var compressedUnalignedStream = new MemoryStream(compressedUnaligned); + compressedUnalignedStream.Position = 13; + + // Compressed's only block has no padding. + using var xzBlock = new XZBlock(compressedUnalignedStream, CheckType.CRC64, 8); + using var sr = new StreamReader(xzBlock); + sr.ReadToEnd(); + Assert.Equal(1L, compressedUnalignedStream.Position % 4L); + } } diff --git a/tests/SharpCompress.Test/Xz/XZHeaderAsyncTests.cs b/tests/SharpCompress.Test/Xz/XZHeaderAsyncTests.cs new file mode 100644 index 00000000..61296b91 --- /dev/null +++ b/tests/SharpCompress.Test/Xz/XZHeaderAsyncTests.cs @@ -0,0 +1,83 @@ +using System.IO; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Compressors.Xz; +using Xunit; + +namespace SharpCompress.Test.Xz; + +public class XzHeaderAsyncTests : XzTestsBase +{ + [Fact] + public async ValueTask ChecksMagicNumberAsync() + { + var bytes = (byte[])Compressed.Clone(); + bytes[3]++; + using Stream badMagicNumberStream = new MemoryStream(bytes); + using var br = new BinaryReader(badMagicNumberStream); + var header = new XZHeader(br); + var ex = await Assert.ThrowsAsync(async () => + { + await header.ProcessAsync().ConfigureAwait(false); + }); + Assert.Equal("Invalid XZ Stream", ex.Message); + } + + [Fact] + public async ValueTask CorruptHeaderThrowsAsync() + { + var bytes = (byte[])Compressed.Clone(); + bytes[8]++; + using Stream badCrcStream = new MemoryStream(bytes); + using var br = new BinaryReader(badCrcStream); + var header = new XZHeader(br); + var ex = await Assert.ThrowsAsync(async () => + { + await header.ProcessAsync().ConfigureAwait(false); + }); + Assert.Equal("Stream header corrupt", ex.Message); + } + + [Fact] + public async ValueTask BadVersionIfCrcOkButStreamFlagUnknownAsync() + { + var bytes = (byte[])Compressed.Clone(); + byte[] streamFlags = [0x00, 0xF4]; + var crc = Crc32.Compute(streamFlags).ToLittleEndianBytes(); + streamFlags.CopyTo(bytes, 6); + crc.CopyTo(bytes, 8); + using Stream badFlagStream = new MemoryStream(bytes); + using var br = new BinaryReader(badFlagStream); + var header = new XZHeader(br); + var ex = await Assert.ThrowsAsync(async () => + { + await header.ProcessAsync().ConfigureAwait(false); + }); + Assert.Equal("Unknown XZ Stream Version", ex.Message); + } + + [Fact] + public async ValueTask ProcessesBlockCheckTypeAsync() + { + using var br = new BinaryReader(CompressedStream); + var header = new XZHeader(br); + await header.ProcessAsync().ConfigureAwait(false); + Assert.Equal(CheckType.CRC64, header.BlockCheckType); + } + + [Fact] + public async ValueTask CanCalculateBlockCheckSizeAsync() + { + using var br = new BinaryReader(CompressedStream); + var header = new XZHeader(br); + await header.ProcessAsync().ConfigureAwait(false); + Assert.Equal(8, header.BlockCheckSize); + } + + [Fact] + public async ValueTask ProcessesStreamHeaderFromFactoryAsync() + { + var header = await XZHeader.FromStreamAsync(CompressedStream).ConfigureAwait(false); + Assert.Equal(CheckType.CRC64, header.BlockCheckType); + } +} diff --git a/tests/SharpCompress.Test/Xz/XZHeaderTests.cs b/tests/SharpCompress.Test/Xz/XZHeaderTests.cs index 3fa9dc86..ab9bd315 100644 --- a/tests/SharpCompress.Test/Xz/XZHeaderTests.cs +++ b/tests/SharpCompress.Test/Xz/XZHeaderTests.cs @@ -1,26 +1,25 @@ -using SharpCompress.Compressors.Xz; using System.IO; +using SharpCompress.Common; +using SharpCompress.Compressors.Xz; using Xunit; namespace SharpCompress.Test.Xz; -public class XZHeaderTests : XZTestsBase +public class XzHeaderTests : XzTestsBase { [Fact] public void ChecksMagicNumber() { var bytes = (byte[])Compressed.Clone(); bytes[3]++; - using (Stream badMagicNumberStream = new MemoryStream(bytes)) + using Stream badMagicNumberStream = new MemoryStream(bytes); + using var br = new BinaryReader(badMagicNumberStream); + var header = new XZHeader(br); + var ex = Assert.Throws(() => { - BinaryReader br = new BinaryReader(badMagicNumberStream); - var header = new XZHeader(br); - var ex = Assert.Throws(() => - { - header.Process(); - }); - Assert.Equal("Invalid XZ Stream", ex.Message); - } + header.Process(); + }); + Assert.Equal("Invalid XZ Stream", ex.Message); } [Fact] @@ -28,42 +27,38 @@ public class XZHeaderTests : XZTestsBase { var bytes = (byte[])Compressed.Clone(); bytes[8]++; - using (Stream badCrcStream = new MemoryStream(bytes)) + using Stream badCrcStream = new MemoryStream(bytes); + using var br = new BinaryReader(badCrcStream); + var header = new XZHeader(br); + var ex = Assert.Throws(() => { - BinaryReader br = new BinaryReader(badCrcStream); - var header = new XZHeader(br); - var ex = Assert.Throws(() => - { - header.Process(); - }); - Assert.Equal("Stream header corrupt", ex.Message); - } + header.Process(); + }); + Assert.Equal("Stream header corrupt", ex.Message); } [Fact] public void BadVersionIfCrcOkButStreamFlagUnknown() { var bytes = (byte[])Compressed.Clone(); - byte[] streamFlags = { 0x00, 0xF4 }; - byte[] crc = Crc32.Compute(streamFlags).ToLittleEndianBytes(); + byte[] streamFlags = [0x00, 0xF4]; + var crc = Crc32.Compute(streamFlags).ToLittleEndianBytes(); streamFlags.CopyTo(bytes, 6); crc.CopyTo(bytes, 8); - using (Stream badFlagStream = new MemoryStream(bytes)) + using Stream badFlagStream = new MemoryStream(bytes); + using var br = new BinaryReader(badFlagStream); + var header = new XZHeader(br); + var ex = Assert.Throws(() => { - BinaryReader br = new BinaryReader(badFlagStream); - var header = new XZHeader(br); - var ex = Assert.Throws(() => - { - header.Process(); - }); - Assert.Equal("Unknown XZ Stream Version", ex.Message); - } + header.Process(); + }); + Assert.Equal("Unknown XZ Stream Version", ex.Message); } [Fact] public void ProcessesBlockCheckType() { - BinaryReader br = new BinaryReader(CompressedStream); + using var br = new BinaryReader(CompressedStream); var header = new XZHeader(br); header.Process(); Assert.Equal(CheckType.CRC64, header.BlockCheckType); @@ -72,7 +67,7 @@ public class XZHeaderTests : XZTestsBase [Fact] public void CanCalculateBlockCheckSize() { - BinaryReader br = new BinaryReader(CompressedStream); + using var br = new BinaryReader(CompressedStream); var header = new XZHeader(br); header.Process(); Assert.Equal(8, header.BlockCheckSize); diff --git a/tests/SharpCompress.Test/Xz/XZIndexAsyncTests.cs b/tests/SharpCompress.Test/Xz/XZIndexAsyncTests.cs new file mode 100644 index 00000000..b740ddcd --- /dev/null +++ b/tests/SharpCompress.Test/Xz/XZIndexAsyncTests.cs @@ -0,0 +1,97 @@ +using System.IO; +using System.Threading.Tasks; +using SharpCompress.Common; +using SharpCompress.Compressors.Xz; +using Xunit; + +namespace SharpCompress.Test.Xz; + +public class XzIndexAsyncTests : XzTestsBase +{ + protected override void RewindEmpty(Stream stream) => stream.Position = 12; + + protected override void Rewind(Stream stream) => stream.Position = 356; + + protected override void RewindIndexed(Stream stream) => stream.Position = 612; + + [Fact] + public void RecordsStreamStartOnInit() + { + using Stream badStream = new MemoryStream([1, 2, 3, 4, 5]); + using var br = new BinaryReader(badStream); + var index = new XZIndex(br, false); + Assert.Equal(0, index.StreamStartPosition); + } + + [Fact] + public async ValueTask ThrowsIfHasNoIndexMarkerAsync() + { + using Stream badStream = new MemoryStream([1, 2, 3, 4, 5]); + using var br = new BinaryReader(badStream); + var index = new XZIndex(br, false); + await Assert.ThrowsAsync(async () => + await index.ProcessAsync().ConfigureAwait(false) + ); + } + + [Fact] + public async ValueTask ReadsNoRecordAsync() + { + using var br = new BinaryReader(CompressedEmptyStream); + var index = new XZIndex(br, false); + await index.ProcessAsync().ConfigureAwait(false); + Assert.Equal((ulong)0, index.NumberOfRecords); + } + + [Fact] + public async ValueTask ReadsOneRecordAsync() + { + using var br = new BinaryReader(CompressedStream); + var index = new XZIndex(br, false); + await index.ProcessAsync().ConfigureAwait(false); + Assert.Equal((ulong)1, index.NumberOfRecords); + } + + [Fact] + public async ValueTask ReadsMultipleRecordsAsync() + { + using var br = new BinaryReader(CompressedIndexedStream); + var index = new XZIndex(br, false); + await index.ProcessAsync().ConfigureAwait(false); + Assert.Equal((ulong)2, index.NumberOfRecords); + } + + [Fact] + public async ValueTask ReadsFirstRecordAsync() + { + using var br = new BinaryReader(CompressedStream); + var index = new XZIndex(br, false); + await index.ProcessAsync().ConfigureAwait(false); + Assert.Equal((ulong)OriginalBytes.Length, index.Records[0].UncompressedSize); + } + + [Fact] + public async ValueTask SkipsPaddingAsync() + { + // Index with 3-byte padding. + using Stream badStream = new MemoryStream([ + 0x00, + 0x01, + 0x10, + 0x80, + 0x01, + 0x00, + 0x00, + 0x00, + 0xB1, + 0x01, + 0xD9, + 0xC9, + 0xFF, + ]); + using var br = new BinaryReader(badStream); + var index = new XZIndex(br, false); + await index.ProcessAsync().ConfigureAwait(false); + Assert.Equal(0L, badStream.Position % 4L); + } +} diff --git a/tests/SharpCompress.Test/Xz/XZIndexTests.cs b/tests/SharpCompress.Test/Xz/XZIndexTests.cs index 9c9d15bf..022dcd97 100644 --- a/tests/SharpCompress.Test/Xz/XZIndexTests.cs +++ b/tests/SharpCompress.Test/Xz/XZIndexTests.cs @@ -1,10 +1,11 @@ -using SharpCompress.Compressors.Xz; using System.IO; +using SharpCompress.Common; +using SharpCompress.Compressors.Xz; using Xunit; namespace SharpCompress.Test.Xz; -public class XZIndexTests : XZTestsBase +public class XzIndexTests : XzTestsBase { protected override void RewindEmpty(Stream stream) => stream.Position = 12; @@ -15,29 +16,25 @@ public class XZIndexTests : XZTestsBase [Fact] public void RecordsStreamStartOnInit() { - using (Stream badStream = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 })) - { - BinaryReader br = new BinaryReader(badStream); - var index = new XZIndex(br, false); - Assert.Equal(0, index.StreamStartPosition); - } + using Stream badStream = new MemoryStream([1, 2, 3, 4, 5]); + using var br = new BinaryReader(badStream); + var index = new XZIndex(br, false); + Assert.Equal(0, index.StreamStartPosition); } [Fact] public void ThrowsIfHasNoIndexMarker() { - using (Stream badStream = new MemoryStream(new byte[] { 1, 2, 3, 4, 5 })) - { - BinaryReader br = new BinaryReader(badStream); - var index = new XZIndex(br, false); - Assert.Throws(() => index.Process()); - } + using Stream badStream = new MemoryStream([1, 2, 3, 4, 5]); + using var br = new BinaryReader(badStream); + var index = new XZIndex(br, false); + Assert.Throws(() => index.Process()); } [Fact] public void ReadsNoRecord() { - BinaryReader br = new BinaryReader(CompressedEmptyStream); + using var br = new BinaryReader(CompressedEmptyStream); var index = new XZIndex(br, false); index.Process(); Assert.Equal((ulong)0, index.NumberOfRecords); @@ -46,7 +43,7 @@ public class XZIndexTests : XZTestsBase [Fact] public void ReadsOneRecord() { - BinaryReader br = new BinaryReader(CompressedStream); + using var br = new BinaryReader(CompressedStream); var index = new XZIndex(br, false); index.Process(); Assert.Equal((ulong)1, index.NumberOfRecords); @@ -55,7 +52,7 @@ public class XZIndexTests : XZTestsBase [Fact] public void ReadsMultipleRecords() { - BinaryReader br = new BinaryReader(CompressedIndexedStream); + using var br = new BinaryReader(CompressedIndexedStream); var index = new XZIndex(br, false); index.Process(); Assert.Equal((ulong)2, index.NumberOfRecords); @@ -64,7 +61,7 @@ public class XZIndexTests : XZTestsBase [Fact] public void ReadsFirstRecord() { - BinaryReader br = new BinaryReader(CompressedStream); + using var br = new BinaryReader(CompressedStream); var index = new XZIndex(br, false); index.Process(); Assert.Equal((ulong)OriginalBytes.Length, index.Records[0].UncompressedSize); @@ -74,31 +71,24 @@ public class XZIndexTests : XZTestsBase public void SkipsPadding() { // Index with 3-byte padding. - using ( - Stream badStream = new MemoryStream( - new byte[] - { - 0x00, - 0x01, - 0x10, - 0x80, - 0x01, - 0x00, - 0x00, - 0x00, - 0xB1, - 0x01, - 0xD9, - 0xC9, - 0xFF - } - ) - ) - { - BinaryReader br = new BinaryReader(badStream); - var index = new XZIndex(br, false); - index.Process(); - Assert.Equal(0L, badStream.Position % 4L); - } + using Stream badStream = new MemoryStream([ + 0x00, + 0x01, + 0x10, + 0x80, + 0x01, + 0x00, + 0x00, + 0x00, + 0xB1, + 0x01, + 0xD9, + 0xC9, + 0xFF, + ]); + using var br = new BinaryReader(badStream); + var index = new XZIndex(br, false); + index.Process(); + Assert.Equal(0L, badStream.Position % 4L); } } diff --git a/tests/SharpCompress.Test/Xz/XZStreamAsyncTests.cs b/tests/SharpCompress.Test/Xz/XZStreamAsyncTests.cs new file mode 100644 index 00000000..8f8e59b8 --- /dev/null +++ b/tests/SharpCompress.Test/Xz/XZStreamAsyncTests.cs @@ -0,0 +1,58 @@ +using System.IO; +using System.Threading.Tasks; +using SharpCompress.Compressors.Xz; +using SharpCompress.IO; +using SharpCompress.Test.Mocks; +using Xunit; + +namespace SharpCompress.Test.Xz; + +public class XzStreamAsyncTests : XzTestsBase +{ + [Fact] + public async ValueTask CanReadEmptyStreamAsync() + { + using var xz = new XZStream(CompressedEmptyStream); + using var sr = new StreamReader(new AsyncOnlyStream(xz)); + var uncompressed = await sr.ReadToEndAsync().ConfigureAwait(false); + Assert.Equal(OriginalEmpty, uncompressed); + } + + [Fact] + public async ValueTask CanReadStreamAsync() + { + using var xz = new XZStream(CompressedStream); + using var sr = new StreamReader(new AsyncOnlyStream(xz)); + var uncompressed = await sr.ReadToEndAsync().ConfigureAwait(false); + Assert.Equal(Original, uncompressed); + } + + [Fact] + public async ValueTask CanReadIndexedStreamAsync() + { + using var xz = new XZStream(CompressedIndexedStream); + using var sr = new StreamReader(new AsyncOnlyStream(xz)); + var uncompressed = await sr.ReadToEndAsync().ConfigureAwait(false); + Assert.Equal(OriginalIndexed, uncompressed); + } + + [Fact] + public async ValueTask CanReadNonSeekableStreamAsync() + { + var nonSeekable = new ForwardOnlyStream(new MemoryStream(Compressed)); + var xz = new XZStream(SharpCompressStream.Create(nonSeekable)); + using var sr = new StreamReader(new AsyncOnlyStream(xz)); + var uncompressed = await sr.ReadToEndAsync().ConfigureAwait(false); + Assert.Equal(Original, uncompressed); + } + + [Fact] + public async ValueTask CanReadNonSeekableEmptyStreamAsync() + { + var nonSeekable = new ForwardOnlyStream(new MemoryStream(CompressedEmpty)); + var xz = new XZStream(SharpCompressStream.Create(nonSeekable)); + using var sr = new StreamReader(new AsyncOnlyStream(xz)); + var uncompressed = await sr.ReadToEndAsync().ConfigureAwait(false); + Assert.Equal(OriginalEmpty, uncompressed); + } +} diff --git a/tests/SharpCompress.Test/Xz/XZStreamTests.cs b/tests/SharpCompress.Test/Xz/XZStreamTests.cs index 1f50fb40..00cd2f27 100644 --- a/tests/SharpCompress.Test/Xz/XZStreamTests.cs +++ b/tests/SharpCompress.Test/Xz/XZStreamTests.cs @@ -1,41 +1,69 @@ -using SharpCompress.Compressors.Xz; -using System.IO; +using System.IO; +using SharpCompress.Common; +using SharpCompress.Compressors.Xz; +using SharpCompress.IO; +using SharpCompress.Test.Mocks; using Xunit; namespace SharpCompress.Test.Xz; -public class XZStreamTests : XZTestsBase +public class XzStreamTests : XzTestsBase { [Fact] public void CanReadEmptyStream() { - XZStream xz = new XZStream(CompressedEmptyStream); - using (var sr = new StreamReader(xz)) - { - string uncompressed = sr.ReadToEnd(); - Assert.Equal(OriginalEmpty, uncompressed); - } + var xz = new XZStream(CompressedEmptyStream); + using var sr = new StreamReader(xz); + var uncompressed = sr.ReadToEnd(); + Assert.Equal(OriginalEmpty, uncompressed); } [Fact] public void CanReadStream() { - XZStream xz = new XZStream(CompressedStream); - using (var sr = new StreamReader(xz)) - { - string uncompressed = sr.ReadToEnd(); - Assert.Equal(Original, uncompressed); - } + var xz = new XZStream(CompressedStream); + using var sr = new StreamReader(xz); + var uncompressed = sr.ReadToEnd(); + Assert.Equal(Original, uncompressed); } [Fact] public void CanReadIndexedStream() { - XZStream xz = new XZStream(CompressedIndexedStream); - using (var sr = new StreamReader(xz)) - { - string uncompressed = sr.ReadToEnd(); - Assert.Equal(OriginalIndexed, uncompressed); - } + var xz = new XZStream(CompressedIndexedStream); + using var sr = new StreamReader(xz); + var uncompressed = sr.ReadToEnd(); + Assert.Equal(OriginalIndexed, uncompressed); + } + + [Fact] + public void CanReadNonSeekableStream() + { + var nonSeekable = new ForwardOnlyStream(new MemoryStream(Compressed)); + var xz = new XZStream(SharpCompressStream.Create(nonSeekable)); + using var sr = new StreamReader(xz); + var uncompressed = sr.ReadToEnd(); + Assert.Equal(Original, uncompressed); + } + + [Fact] + public void CanReadNonSeekableEmptyStream() + { + var nonSeekable = new ForwardOnlyStream(new MemoryStream(CompressedEmpty)); + var xz = new XZStream(SharpCompressStream.Create(nonSeekable)); + using var sr = new StreamReader(xz); + var uncompressed = sr.ReadToEnd(); + Assert.Equal(OriginalEmpty, uncompressed); + } + + [Fact] + public void Throws_On_Corrupt_Block_Check() + { + var compressed = (byte[])Compressed.Clone(); + compressed[compressed.Length - 29] ^= 1; + using var xz = new XZStream(new MemoryStream(compressed)); + using var output = new MemoryStream(); + + Assert.Throws(() => xz.CopyTo(output)); } } diff --git a/tests/SharpCompress.Test/Xz/XZTestsBase.cs b/tests/SharpCompress.Test/Xz/XZTestsBase.cs index 5a1b5cbb..a260f172 100644 --- a/tests/SharpCompress.Test/Xz/XZTestsBase.cs +++ b/tests/SharpCompress.Test/Xz/XZTestsBase.cs @@ -4,9 +4,9 @@ using System.Text; namespace SharpCompress.Test.Xz; -public abstract class XZTestsBase : IDisposable +public abstract class XzTestsBase : IDisposable { - public XZTestsBase() + public XzTestsBase() { RewindEmpty(CompressedEmptyStream); Rewind(CompressedStream); @@ -29,41 +29,40 @@ public abstract class XZTestsBase : IDisposable protected Stream CompressedEmptyStream { get; } = new MemoryStream(CompressedEmpty); protected static byte[] CompressedEmpty { get; } = - new byte[] - { - 0xfd, - 0x37, - 0x7a, - 0x58, - 0x5a, - 0x00, - 0x00, - 0x01, - 0x69, - 0x22, - 0xde, - 0x36, - 0x00, - 0x00, - 0x00, - 0x00, - 0x1c, - 0xdf, - 0x44, - 0x21, - 0x90, - 0x42, - 0x99, - 0x0d, - 0x01, - 0x00, - 0x00, - 0x00, - 0x00, - 0x01, - 0x59, - 0x5a - }; + [ + 0xfd, + 0x37, + 0x7a, + 0x58, + 0x5a, + 0x00, + 0x00, + 0x01, + 0x69, + 0x22, + 0xde, + 0x36, + 0x00, + 0x00, + 0x00, + 0x00, + 0x1c, + 0xdf, + 0x44, + 0x21, + 0x90, + 0x42, + 0x99, + 0x0d, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0x01, + 0x59, + 0x5a, + ]; protected static byte[] OriginalEmptyBytes => Encoding.ASCII.GetBytes(OriginalEmpty); @@ -72,389 +71,388 @@ public abstract class XZTestsBase : IDisposable protected Stream CompressedStream { get; } = new MemoryStream(Compressed); protected static byte[] Compressed { get; } = - new byte[] - { - 0xfd, - 0x37, - 0x7a, - 0x58, - 0x5a, - 0x00, - 0x00, - 0x04, - 0xe6, - 0xd6, - 0xb4, - 0x46, - 0x02, - 0x00, - 0x21, - 0x01, - 0x16, - 0x00, - 0x00, - 0x00, - 0x74, - 0x2f, - 0xe5, - 0xa3, - 0xe0, - 0x01, - 0xe4, - 0x01, - 0x3c, - 0x5d, - 0x00, - 0x26, - 0x98, - 0x4a, - 0x47, - 0xc6, - 0x6a, - 0x27, - 0xd7, - 0x36, - 0x7a, - 0x05, - 0xb9, - 0x4f, - 0xd7, - 0xde, - 0x52, - 0x4c, - 0xca, - 0x26, - 0x4f, - 0x23, - 0x60, - 0x4d, - 0xf3, - 0x1f, - 0xa3, - 0x67, - 0x49, - 0x53, - 0xd0, - 0xf5, - 0xc7, - 0xa9, - 0x3e, - 0xd6, - 0xb5, - 0x3d, - 0x2b, - 0x02, - 0xbe, - 0x83, - 0x27, - 0xe2, - 0xa6, - 0xc3, - 0x13, - 0x4a, - 0x31, - 0x14, - 0x33, - 0xed, - 0x9a, - 0x85, - 0x1d, - 0x05, - 0x6e, - 0x7e, - 0xa4, - 0x91, - 0xbf, - 0x46, - 0x71, - 0x7d, - 0xa7, - 0xfb, - 0x12, - 0x10, - 0xdf, - 0x21, - 0x73, - 0x75, - 0xd8, - 0xd9, - 0xab, - 0x8f, - 0x1f, - 0x8b, - 0xb0, - 0xb9, - 0x3f, - 0x9a, - 0xa5, - 0x1e, - 0xd4, - 0x2f, - 0xdf, - 0x09, - 0xb3, - 0xfe, - 0x45, - 0xef, - 0x16, - 0xec, - 0x95, - 0x68, - 0x64, - 0xbb, - 0x42, - 0x0c, - 0x8b, - 0x96, - 0x27, - 0x30, - 0x62, - 0x42, - 0x91, - 0x7c, - 0xf3, - 0x6e, - 0x4d, - 0x03, - 0xc5, - 0x00, - 0x04, - 0x73, - 0xdd, - 0xee, - 0xb0, - 0xaa, - 0xd6, - 0x0b, - 0x11, - 0x90, - 0x81, - 0xd4, - 0xaa, - 0x69, - 0x63, - 0xfa, - 0x2f, - 0xb4, - 0x25, - 0x0a, - 0x7f, - 0xf9, - 0x47, - 0x77, - 0xb1, - 0x1f, - 0xc3, - 0xb4, - 0x4d, - 0x51, - 0xf8, - 0x23, - 0x3a, - 0x7c, - 0x44, - 0xc8, - 0xcc, - 0xca, - 0x72, - 0x09, - 0xae, - 0xc9, - 0x7b, - 0x7e, - 0x91, - 0x5d, - 0xff, - 0xc4, - 0xeb, - 0xfd, - 0xa1, - 0x9b, - 0xd4, - 0x8d, - 0xd7, - 0xd3, - 0x57, - 0xac, - 0x7e, - 0x3b, - 0x97, - 0x2e, - 0xe4, - 0xc2, - 0x2e, - 0x93, - 0x3d, - 0xb0, - 0x16, - 0x64, - 0x78, - 0x45, - 0xb1, - 0xc9, - 0x40, - 0x96, - 0xcf, - 0x5b, - 0xc2, - 0x2f, - 0xaa, - 0xba, - 0xcf, - 0x98, - 0x38, - 0x21, - 0x3d, - 0x1a, - 0x13, - 0xe8, - 0xa6, - 0xa6, - 0xdf, - 0xf4, - 0x3d, - 0x01, - 0xa1, - 0x9d, - 0xc1, - 0x3e, - 0x37, - 0xac, - 0x20, - 0xc4, - 0xef, - 0x18, - 0xb1, - 0xeb, - 0x35, - 0xf4, - 0x66, - 0x9a, - 0x47, - 0x3c, - 0xce, - 0x7c, - 0xad, - 0xdb, - 0x2e, - 0x39, - 0xf5, - 0x8d, - 0x4a, - 0x1d, - 0x65, - 0xc2, - 0x0f, - 0xa4, - 0x40, - 0x7e, - 0xe6, - 0xa7, - 0x17, - 0xce, - 0x75, - 0x7f, - 0xd9, - 0xa3, - 0xf9, - 0x27, - 0x42, - 0xd7, - 0x98, - 0x54, - 0x17, - 0xa7, - 0x7a, - 0x7c, - 0x82, - 0xdf, - 0xeb, - 0x08, - 0x28, - 0x86, - 0xdd, - 0x57, - 0x77, - 0x92, - 0x80, - 0x5f, - 0x7b, - 0x3b, - 0xce, - 0x77, - 0x72, - 0xff, - 0xa3, - 0x85, - 0xd8, - 0x5c, - 0x8a, - 0xb7, - 0x83, - 0x58, - 0xfa, - 0xbd, - 0x72, - 0xe3, - 0x66, - 0x9d, - 0x3b, - 0xff, - 0x13, - 0x5b, - 0x0b, - 0xf1, - 0x6c, - 0xa6, - 0xb1, - 0x3b, - 0x85, - 0x3b, - 0x47, - 0x91, - 0xc8, - 0x7c, - 0x38, - 0xe2, - 0xe5, - 0x54, - 0xf8, - 0x27, - 0xee, - 0x00, - 0xff, - 0xd3, - 0x68, - 0xf1, - 0xc6, - 0xc7, - 0xd7, - 0x24, - 0x00, - 0x01, - 0xd8, - 0x02, - 0xe5, - 0x03, - 0x00, - 0x00, - 0xac, - 0x16, - 0x1f, - 0xa4, - 0xb1, - 0xc4, - 0x67, - 0xfb, - 0x02, - 0x00, - 0x00, - 0x00, - 0x00, - 0x04, - 0x59, - 0x5a - }; + [ + 0xfd, + 0x37, + 0x7a, + 0x58, + 0x5a, + 0x00, + 0x00, + 0x04, + 0xe6, + 0xd6, + 0xb4, + 0x46, + 0x02, + 0x00, + 0x21, + 0x01, + 0x16, + 0x00, + 0x00, + 0x00, + 0x74, + 0x2f, + 0xe5, + 0xa3, + 0xe0, + 0x01, + 0xe4, + 0x01, + 0x3c, + 0x5d, + 0x00, + 0x26, + 0x98, + 0x4a, + 0x47, + 0xc6, + 0x6a, + 0x27, + 0xd7, + 0x36, + 0x7a, + 0x05, + 0xb9, + 0x4f, + 0xd7, + 0xde, + 0x52, + 0x4c, + 0xca, + 0x26, + 0x4f, + 0x23, + 0x60, + 0x4d, + 0xf3, + 0x1f, + 0xa3, + 0x67, + 0x49, + 0x53, + 0xd0, + 0xf5, + 0xc7, + 0xa9, + 0x3e, + 0xd6, + 0xb5, + 0x3d, + 0x2b, + 0x02, + 0xbe, + 0x83, + 0x27, + 0xe2, + 0xa6, + 0xc3, + 0x13, + 0x4a, + 0x31, + 0x14, + 0x33, + 0xed, + 0x9a, + 0x85, + 0x1d, + 0x05, + 0x6e, + 0x7e, + 0xa4, + 0x91, + 0xbf, + 0x46, + 0x71, + 0x7d, + 0xa7, + 0xfb, + 0x12, + 0x10, + 0xdf, + 0x21, + 0x73, + 0x75, + 0xd8, + 0xd9, + 0xab, + 0x8f, + 0x1f, + 0x8b, + 0xb0, + 0xb9, + 0x3f, + 0x9a, + 0xa5, + 0x1e, + 0xd4, + 0x2f, + 0xdf, + 0x09, + 0xb3, + 0xfe, + 0x45, + 0xef, + 0x16, + 0xec, + 0x95, + 0x68, + 0x64, + 0xbb, + 0x42, + 0x0c, + 0x8b, + 0x96, + 0x27, + 0x30, + 0x62, + 0x42, + 0x91, + 0x7c, + 0xf3, + 0x6e, + 0x4d, + 0x03, + 0xc5, + 0x00, + 0x04, + 0x73, + 0xdd, + 0xee, + 0xb0, + 0xaa, + 0xd6, + 0x0b, + 0x11, + 0x90, + 0x81, + 0xd4, + 0xaa, + 0x69, + 0x63, + 0xfa, + 0x2f, + 0xb4, + 0x25, + 0x0a, + 0x7f, + 0xf9, + 0x47, + 0x77, + 0xb1, + 0x1f, + 0xc3, + 0xb4, + 0x4d, + 0x51, + 0xf8, + 0x23, + 0x3a, + 0x7c, + 0x44, + 0xc8, + 0xcc, + 0xca, + 0x72, + 0x09, + 0xae, + 0xc9, + 0x7b, + 0x7e, + 0x91, + 0x5d, + 0xff, + 0xc4, + 0xeb, + 0xfd, + 0xa1, + 0x9b, + 0xd4, + 0x8d, + 0xd7, + 0xd3, + 0x57, + 0xac, + 0x7e, + 0x3b, + 0x97, + 0x2e, + 0xe4, + 0xc2, + 0x2e, + 0x93, + 0x3d, + 0xb0, + 0x16, + 0x64, + 0x78, + 0x45, + 0xb1, + 0xc9, + 0x40, + 0x96, + 0xcf, + 0x5b, + 0xc2, + 0x2f, + 0xaa, + 0xba, + 0xcf, + 0x98, + 0x38, + 0x21, + 0x3d, + 0x1a, + 0x13, + 0xe8, + 0xa6, + 0xa6, + 0xdf, + 0xf4, + 0x3d, + 0x01, + 0xa1, + 0x9d, + 0xc1, + 0x3e, + 0x37, + 0xac, + 0x20, + 0xc4, + 0xef, + 0x18, + 0xb1, + 0xeb, + 0x35, + 0xf4, + 0x66, + 0x9a, + 0x47, + 0x3c, + 0xce, + 0x7c, + 0xad, + 0xdb, + 0x2e, + 0x39, + 0xf5, + 0x8d, + 0x4a, + 0x1d, + 0x65, + 0xc2, + 0x0f, + 0xa4, + 0x40, + 0x7e, + 0xe6, + 0xa7, + 0x17, + 0xce, + 0x75, + 0x7f, + 0xd9, + 0xa3, + 0xf9, + 0x27, + 0x42, + 0xd7, + 0x98, + 0x54, + 0x17, + 0xa7, + 0x7a, + 0x7c, + 0x82, + 0xdf, + 0xeb, + 0x08, + 0x28, + 0x86, + 0xdd, + 0x57, + 0x77, + 0x92, + 0x80, + 0x5f, + 0x7b, + 0x3b, + 0xce, + 0x77, + 0x72, + 0xff, + 0xa3, + 0x85, + 0xd8, + 0x5c, + 0x8a, + 0xb7, + 0x83, + 0x58, + 0xfa, + 0xbd, + 0x72, + 0xe3, + 0x66, + 0x9d, + 0x3b, + 0xff, + 0x13, + 0x5b, + 0x0b, + 0xf1, + 0x6c, + 0xa6, + 0xb1, + 0x3b, + 0x85, + 0x3b, + 0x47, + 0x91, + 0xc8, + 0x7c, + 0x38, + 0xe2, + 0xe5, + 0x54, + 0xf8, + 0x27, + 0xee, + 0x00, + 0xff, + 0xd3, + 0x68, + 0xf1, + 0xc6, + 0xc7, + 0xd7, + 0x24, + 0x00, + 0x01, + 0xd8, + 0x02, + 0xe5, + 0x03, + 0x00, + 0x00, + 0xac, + 0x16, + 0x1f, + 0xa4, + 0xb1, + 0xc4, + 0x67, + 0xfb, + 0x02, + 0x00, + 0x00, + 0x00, + 0x00, + 0x04, + 0x59, + 0x5a, + ]; protected static byte[] OriginalBytes => Encoding.ASCII.GetBytes(Original); protected static string Original { get; } = @@ -481,649 +479,648 @@ public abstract class XZTestsBase : IDisposable protected Stream CompressedIndexedStream { get; } = new MemoryStream(CompressedIndexed); protected static byte[] CompressedIndexed { get; } = - new byte[] - { - 0xfd, - 0x37, - 0x7a, - 0x58, - 0x5a, - 0x00, - 0x00, - 0x01, - 0x69, - 0x22, - 0xde, - 0x36, - 0x03, - 0xc0, - 0xe3, - 0x02, - 0x80, - 0x20, - 0x21, - 0x01, - 0x00, - 0x00, - 0x00, - 0x00, - 0x7e, - 0xe5, - 0xd7, - 0x32, - 0xe0, - 0x0f, - 0xff, - 0x01, - 0x5b, - 0x5d, - 0x00, - 0x26, - 0x98, - 0x4a, - 0x47, - 0xc6, - 0x6a, - 0x27, - 0xd7, - 0x36, - 0x7a, - 0x05, - 0xb9, - 0x4f, - 0xd7, - 0xde, - 0x3a, - 0x0e, - 0xee, - 0x1b, - 0xd7, - 0x81, - 0xe9, - 0xf5, - 0x90, - 0x1e, - 0xd5, - 0x9e, - 0x88, - 0x32, - 0x1c, - 0x7b, - 0x43, - 0x84, - 0x02, - 0x58, - 0x92, - 0xcf, - 0x97, - 0xfc, - 0xae, - 0x01, - 0x83, - 0x23, - 0x48, - 0x93, - 0xc6, - 0x56, - 0xcc, - 0x6d, - 0xb1, - 0x23, - 0x10, - 0x24, - 0x3b, - 0x9e, - 0x06, - 0xaa, - 0xc0, - 0xce, - 0x86, - 0x0a, - 0xb7, - 0x9f, - 0x99, - 0x61, - 0xbe, - 0x3b, - 0x6d, - 0xfe, - 0x60, - 0xef, - 0x14, - 0x35, - 0x7f, - 0x21, - 0xe8, - 0x96, - 0x0e, - 0xbd, - 0x41, - 0x7c, - 0x65, - 0x89, - 0x96, - 0x28, - 0x5e, - 0x85, - 0xa6, - 0x4b, - 0xf3, - 0xf9, - 0xf8, - 0x25, - 0x31, - 0x4a, - 0xbb, - 0x72, - 0xce, - 0xcf, - 0x53, - 0xdf, - 0x13, - 0x42, - 0x2d, - 0xbc, - 0x95, - 0xa5, - 0x6d, - 0xc4, - 0x8c, - 0x72, - 0x99, - 0xe8, - 0x9a, - 0xcf, - 0x80, - 0xd4, - 0xc4, - 0x3f, - 0x55, - 0xc3, - 0x9b, - 0x00, - 0xce, - 0x65, - 0x27, - 0x6e, - 0xbf, - 0xb2, - 0x88, - 0xab, - 0xc0, - 0x5f, - 0xf9, - 0xd0, - 0xc8, - 0xbb, - 0xd7, - 0x48, - 0xd7, - 0x2e, - 0x5e, - 0xbb, - 0x23, - 0x35, - 0x6e, - 0x62, - 0xb6, - 0x13, - 0xd4, - 0x06, - 0xd1, - 0x5b, - 0x97, - 0xee, - 0x5b, - 0x89, - 0x78, - 0x07, - 0x24, - 0x74, - 0x59, - 0x06, - 0x1e, - 0x7f, - 0x8c, - 0xb0, - 0x48, - 0xff, - 0x0a, - 0x76, - 0xb2, - 0x07, - 0xa0, - 0x99, - 0xf5, - 0x4b, - 0x68, - 0xd4, - 0x55, - 0x32, - 0xb3, - 0x17, - 0x7b, - 0xb6, - 0x26, - 0xdb, - 0x1c, - 0xc3, - 0x0b, - 0xda, - 0x3e, - 0x46, - 0xba, - 0x1a, - 0x67, - 0x23, - 0xb7, - 0x2a, - 0x40, - 0xdc, - 0xc9, - 0xa2, - 0xe4, - 0xb5, - 0x68, - 0x5c, - 0x81, - 0x60, - 0xa7, - 0xad, - 0xe6, - 0xba, - 0xbb, - 0x0d, - 0x82, - 0x8a, - 0xe2, - 0x03, - 0xa9, - 0x22, - 0x09, - 0x5e, - 0xd8, - 0x69, - 0xfa, - 0x29, - 0xd1, - 0x32, - 0xa1, - 0xf0, - 0x9b, - 0x3c, - 0xc3, - 0x0b, - 0x9a, - 0x53, - 0xf0, - 0x3e, - 0xf3, - 0x1b, - 0x77, - 0xee, - 0x8f, - 0xa6, - 0x15, - 0x02, - 0x77, - 0x14, - 0x54, - 0x60, - 0xae, - 0xbe, - 0x91, - 0x9e, - 0xe6, - 0x8b, - 0x87, - 0x6e, - 0x46, - 0x44, - 0x64, - 0xc7, - 0x58, - 0x90, - 0x62, - 0x25, - 0x32, - 0xf9, - 0xcd, - 0xd2, - 0x73, - 0x2e, - 0x3f, - 0xd7, - 0x5d, - 0x3c, - 0x86, - 0x1c, - 0xa8, - 0x35, - 0xa9, - 0xc2, - 0xcb, - 0x59, - 0xcb, - 0xac, - 0xb3, - 0x03, - 0x12, - 0xd4, - 0x8a, - 0xde, - 0xd5, - 0xc1, - 0xd8, - 0x0c, - 0x32, - 0x49, - 0x87, - 0x97, - 0x62, - 0x4f, - 0x32, - 0x39, - 0x63, - 0x5b, - 0x8b, - 0xd1, - 0x6c, - 0x5c, - 0x90, - 0xd9, - 0x93, - 0x13, - 0xae, - 0x70, - 0xf5, - 0x2f, - 0x40, - 0xaf, - 0x01, - 0x95, - 0x01, - 0x0c, - 0xc5, - 0xfa, - 0x82, - 0xf8, - 0x71, - 0x9d, - 0x53, - 0xe6, - 0x47, - 0x6e, - 0x99, - 0x54, - 0x57, - 0x41, - 0x72, - 0xea, - 0xf5, - 0x78, - 0xdd, - 0x86, - 0xbd, - 0x00, - 0x00, - 0x00, - 0x72, - 0x6a, - 0xf2, - 0x47, - 0x03, - 0xc0, - 0xcb, - 0x01, - 0x8d, - 0x02, - 0x21, - 0x01, - 0x00, - 0x00, - 0x00, - 0x00, - 0xfb, - 0xa7, - 0xf7, - 0x94, - 0xe0, - 0x01, - 0x0c, - 0x00, - 0xc3, - 0x5d, - 0x00, - 0x06, - 0x82, - 0xca, - 0x9b, - 0x77, - 0x93, - 0x57, - 0xb3, - 0x76, - 0xbd, - 0x8b, - 0xcb, - 0xee, - 0xf4, - 0x2c, - 0xff, - 0x7f, - 0x95, - 0x33, - 0x15, - 0x10, - 0xa5, - 0xf9, - 0xfd, - 0xa6, - 0xbb, - 0x9e, - 0xf9, - 0x75, - 0x67, - 0xee, - 0xec, - 0x8b, - 0x40, - 0xea, - 0x32, - 0x47, - 0x3d, - 0x26, - 0xbe, - 0x11, - 0x9c, - 0xa6, - 0x40, - 0xbe, - 0x84, - 0x1f, - 0x1b, - 0x35, - 0x1a, - 0x66, - 0x10, - 0x9c, - 0xf4, - 0x12, - 0x1a, - 0x95, - 0x81, - 0xb5, - 0x55, - 0x6b, - 0xc5, - 0x42, - 0xfd, - 0x37, - 0x70, - 0xc5, - 0x08, - 0xa4, - 0x27, - 0x67, - 0x11, - 0x0b, - 0x1f, - 0xcc, - 0xdb, - 0x54, - 0x9b, - 0x5a, - 0x5f, - 0xee, - 0x21, - 0x63, - 0xdd, - 0x4b, - 0xbc, - 0x49, - 0x95, - 0x6d, - 0xf4, - 0xcb, - 0x9a, - 0x9a, - 0x5e, - 0xe4, - 0x7d, - 0x0f, - 0x02, - 0x22, - 0xa9, - 0x42, - 0x46, - 0x1a, - 0x04, - 0x87, - 0x43, - 0x72, - 0x59, - 0xa4, - 0xd6, - 0xeb, - 0x69, - 0x36, - 0xde, - 0xea, - 0x53, - 0x8c, - 0x89, - 0xd7, - 0x22, - 0xa6, - 0xf7, - 0xa8, - 0x4c, - 0x72, - 0x6c, - 0x80, - 0x69, - 0x01, - 0xb2, - 0xa7, - 0xe8, - 0x8b, - 0x94, - 0xaf, - 0x0e, - 0x47, - 0x58, - 0x1d, - 0x0e, - 0x5c, - 0x7c, - 0x33, - 0x9f, - 0x21, - 0x17, - 0x2c, - 0x4f, - 0x3d, - 0x72, - 0xff, - 0xcf, - 0x7a, - 0x4f, - 0x82, - 0x5b, - 0x85, - 0x28, - 0x70, - 0xf4, - 0x8c, - 0x81, - 0x41, - 0xb8, - 0x20, - 0x5c, - 0x3e, - 0x02, - 0x5e, - 0x5a, - 0x61, - 0xbb, - 0x2f, - 0x64, - 0xc5, - 0x4e, - 0x53, - 0xe4, - 0xca, - 0xe4, - 0xd9, - 0x75, - 0xaf, - 0x15, - 0x4d, - 0xff, - 0x01, - 0xec, - 0x13, - 0x4a, - 0x70, - 0x00, - 0x04, - 0xf9, - 0xfa, - 0x00, - 0x00, - 0x99, - 0x57, - 0xc4, - 0x96, - 0x00, - 0x02, - 0xf7, - 0x02, - 0x80, - 0x20, - 0xdf, - 0x01, - 0x8d, - 0x02, - 0x00, - 0x00, - 0x4c, - 0x41, - 0xe6, - 0xa1, - 0x9b, - 0xe3, - 0x51, - 0x40, - 0x03, - 0x00, - 0x00, - 0x00, - 0x00, - 0x01, - 0x59, - 0x5a - }; + [ + 0xfd, + 0x37, + 0x7a, + 0x58, + 0x5a, + 0x00, + 0x00, + 0x01, + 0x69, + 0x22, + 0xde, + 0x36, + 0x03, + 0xc0, + 0xe3, + 0x02, + 0x80, + 0x20, + 0x21, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0x7e, + 0xe5, + 0xd7, + 0x32, + 0xe0, + 0x0f, + 0xff, + 0x01, + 0x5b, + 0x5d, + 0x00, + 0x26, + 0x98, + 0x4a, + 0x47, + 0xc6, + 0x6a, + 0x27, + 0xd7, + 0x36, + 0x7a, + 0x05, + 0xb9, + 0x4f, + 0xd7, + 0xde, + 0x3a, + 0x0e, + 0xee, + 0x1b, + 0xd7, + 0x81, + 0xe9, + 0xf5, + 0x90, + 0x1e, + 0xd5, + 0x9e, + 0x88, + 0x32, + 0x1c, + 0x7b, + 0x43, + 0x84, + 0x02, + 0x58, + 0x92, + 0xcf, + 0x97, + 0xfc, + 0xae, + 0x01, + 0x83, + 0x23, + 0x48, + 0x93, + 0xc6, + 0x56, + 0xcc, + 0x6d, + 0xb1, + 0x23, + 0x10, + 0x24, + 0x3b, + 0x9e, + 0x06, + 0xaa, + 0xc0, + 0xce, + 0x86, + 0x0a, + 0xb7, + 0x9f, + 0x99, + 0x61, + 0xbe, + 0x3b, + 0x6d, + 0xfe, + 0x60, + 0xef, + 0x14, + 0x35, + 0x7f, + 0x21, + 0xe8, + 0x96, + 0x0e, + 0xbd, + 0x41, + 0x7c, + 0x65, + 0x89, + 0x96, + 0x28, + 0x5e, + 0x85, + 0xa6, + 0x4b, + 0xf3, + 0xf9, + 0xf8, + 0x25, + 0x31, + 0x4a, + 0xbb, + 0x72, + 0xce, + 0xcf, + 0x53, + 0xdf, + 0x13, + 0x42, + 0x2d, + 0xbc, + 0x95, + 0xa5, + 0x6d, + 0xc4, + 0x8c, + 0x72, + 0x99, + 0xe8, + 0x9a, + 0xcf, + 0x80, + 0xd4, + 0xc4, + 0x3f, + 0x55, + 0xc3, + 0x9b, + 0x00, + 0xce, + 0x65, + 0x27, + 0x6e, + 0xbf, + 0xb2, + 0x88, + 0xab, + 0xc0, + 0x5f, + 0xf9, + 0xd0, + 0xc8, + 0xbb, + 0xd7, + 0x48, + 0xd7, + 0x2e, + 0x5e, + 0xbb, + 0x23, + 0x35, + 0x6e, + 0x62, + 0xb6, + 0x13, + 0xd4, + 0x06, + 0xd1, + 0x5b, + 0x97, + 0xee, + 0x5b, + 0x89, + 0x78, + 0x07, + 0x24, + 0x74, + 0x59, + 0x06, + 0x1e, + 0x7f, + 0x8c, + 0xb0, + 0x48, + 0xff, + 0x0a, + 0x76, + 0xb2, + 0x07, + 0xa0, + 0x99, + 0xf5, + 0x4b, + 0x68, + 0xd4, + 0x55, + 0x32, + 0xb3, + 0x17, + 0x7b, + 0xb6, + 0x26, + 0xdb, + 0x1c, + 0xc3, + 0x0b, + 0xda, + 0x3e, + 0x46, + 0xba, + 0x1a, + 0x67, + 0x23, + 0xb7, + 0x2a, + 0x40, + 0xdc, + 0xc9, + 0xa2, + 0xe4, + 0xb5, + 0x68, + 0x5c, + 0x81, + 0x60, + 0xa7, + 0xad, + 0xe6, + 0xba, + 0xbb, + 0x0d, + 0x82, + 0x8a, + 0xe2, + 0x03, + 0xa9, + 0x22, + 0x09, + 0x5e, + 0xd8, + 0x69, + 0xfa, + 0x29, + 0xd1, + 0x32, + 0xa1, + 0xf0, + 0x9b, + 0x3c, + 0xc3, + 0x0b, + 0x9a, + 0x53, + 0xf0, + 0x3e, + 0xf3, + 0x1b, + 0x77, + 0xee, + 0x8f, + 0xa6, + 0x15, + 0x02, + 0x77, + 0x14, + 0x54, + 0x60, + 0xae, + 0xbe, + 0x91, + 0x9e, + 0xe6, + 0x8b, + 0x87, + 0x6e, + 0x46, + 0x44, + 0x64, + 0xc7, + 0x58, + 0x90, + 0x62, + 0x25, + 0x32, + 0xf9, + 0xcd, + 0xd2, + 0x73, + 0x2e, + 0x3f, + 0xd7, + 0x5d, + 0x3c, + 0x86, + 0x1c, + 0xa8, + 0x35, + 0xa9, + 0xc2, + 0xcb, + 0x59, + 0xcb, + 0xac, + 0xb3, + 0x03, + 0x12, + 0xd4, + 0x8a, + 0xde, + 0xd5, + 0xc1, + 0xd8, + 0x0c, + 0x32, + 0x49, + 0x87, + 0x97, + 0x62, + 0x4f, + 0x32, + 0x39, + 0x63, + 0x5b, + 0x8b, + 0xd1, + 0x6c, + 0x5c, + 0x90, + 0xd9, + 0x93, + 0x13, + 0xae, + 0x70, + 0xf5, + 0x2f, + 0x40, + 0xaf, + 0x01, + 0x95, + 0x01, + 0x0c, + 0xc5, + 0xfa, + 0x82, + 0xf8, + 0x71, + 0x9d, + 0x53, + 0xe6, + 0x47, + 0x6e, + 0x99, + 0x54, + 0x57, + 0x41, + 0x72, + 0xea, + 0xf5, + 0x78, + 0xdd, + 0x86, + 0xbd, + 0x00, + 0x00, + 0x00, + 0x72, + 0x6a, + 0xf2, + 0x47, + 0x03, + 0xc0, + 0xcb, + 0x01, + 0x8d, + 0x02, + 0x21, + 0x01, + 0x00, + 0x00, + 0x00, + 0x00, + 0xfb, + 0xa7, + 0xf7, + 0x94, + 0xe0, + 0x01, + 0x0c, + 0x00, + 0xc3, + 0x5d, + 0x00, + 0x06, + 0x82, + 0xca, + 0x9b, + 0x77, + 0x93, + 0x57, + 0xb3, + 0x76, + 0xbd, + 0x8b, + 0xcb, + 0xee, + 0xf4, + 0x2c, + 0xff, + 0x7f, + 0x95, + 0x33, + 0x15, + 0x10, + 0xa5, + 0xf9, + 0xfd, + 0xa6, + 0xbb, + 0x9e, + 0xf9, + 0x75, + 0x67, + 0xee, + 0xec, + 0x8b, + 0x40, + 0xea, + 0x32, + 0x47, + 0x3d, + 0x26, + 0xbe, + 0x11, + 0x9c, + 0xa6, + 0x40, + 0xbe, + 0x84, + 0x1f, + 0x1b, + 0x35, + 0x1a, + 0x66, + 0x10, + 0x9c, + 0xf4, + 0x12, + 0x1a, + 0x95, + 0x81, + 0xb5, + 0x55, + 0x6b, + 0xc5, + 0x42, + 0xfd, + 0x37, + 0x70, + 0xc5, + 0x08, + 0xa4, + 0x27, + 0x67, + 0x11, + 0x0b, + 0x1f, + 0xcc, + 0xdb, + 0x54, + 0x9b, + 0x5a, + 0x5f, + 0xee, + 0x21, + 0x63, + 0xdd, + 0x4b, + 0xbc, + 0x49, + 0x95, + 0x6d, + 0xf4, + 0xcb, + 0x9a, + 0x9a, + 0x5e, + 0xe4, + 0x7d, + 0x0f, + 0x02, + 0x22, + 0xa9, + 0x42, + 0x46, + 0x1a, + 0x04, + 0x87, + 0x43, + 0x72, + 0x59, + 0xa4, + 0xd6, + 0xeb, + 0x69, + 0x36, + 0xde, + 0xea, + 0x53, + 0x8c, + 0x89, + 0xd7, + 0x22, + 0xa6, + 0xf7, + 0xa8, + 0x4c, + 0x72, + 0x6c, + 0x80, + 0x69, + 0x01, + 0xb2, + 0xa7, + 0xe8, + 0x8b, + 0x94, + 0xaf, + 0x0e, + 0x47, + 0x58, + 0x1d, + 0x0e, + 0x5c, + 0x7c, + 0x33, + 0x9f, + 0x21, + 0x17, + 0x2c, + 0x4f, + 0x3d, + 0x72, + 0xff, + 0xcf, + 0x7a, + 0x4f, + 0x82, + 0x5b, + 0x85, + 0x28, + 0x70, + 0xf4, + 0x8c, + 0x81, + 0x41, + 0xb8, + 0x20, + 0x5c, + 0x3e, + 0x02, + 0x5e, + 0x5a, + 0x61, + 0xbb, + 0x2f, + 0x64, + 0xc5, + 0x4e, + 0x53, + 0xe4, + 0xca, + 0xe4, + 0xd9, + 0x75, + 0xaf, + 0x15, + 0x4d, + 0xff, + 0x01, + 0xec, + 0x13, + 0x4a, + 0x70, + 0x00, + 0x04, + 0xf9, + 0xfa, + 0x00, + 0x00, + 0x99, + 0x57, + 0xc4, + 0x96, + 0x00, + 0x02, + 0xf7, + 0x02, + 0x80, + 0x20, + 0xdf, + 0x01, + 0x8d, + 0x02, + 0x00, + 0x00, + 0x4c, + 0x41, + 0xe6, + 0xa1, + 0x9b, + 0xe3, + 0x51, + 0x40, + 0x03, + 0x00, + 0x00, + 0x00, + 0x00, + 0x01, + 0x59, + 0x5a, + ]; protected static byte[] OriginalIndexedBytes => Encoding.ASCII.GetBytes(OriginalIndexed); diff --git a/tests/SharpCompress.Test/Zip/TestPseudoTextStream.cs b/tests/SharpCompress.Test/Zip/TestPseudoTextStream.cs new file mode 100644 index 00000000..20f1c9ce --- /dev/null +++ b/tests/SharpCompress.Test/Zip/TestPseudoTextStream.cs @@ -0,0 +1,99 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Text; + +namespace SharpCompress.Test.Zip; + +/// +/// Generates pseudo English-style text for testing - Nanook +/// +internal class TestPseudoTextStream : Stream +{ + private static readonly char[] _vowels = { 'a', 'e', 'i', 'o', 'u' }; + private static readonly char[] _consonants = "bcdfghjklmnpqrstvwxyz".ToCharArray(); + + private long _position = 0; + + public override bool CanRead => true; + public override bool CanSeek => false; + public override bool CanWrite => false; + public override long Length => throw new NotSupportedException(); + public override long Position + { + get => _position; + set => throw new NotSupportedException(); + } + + public override void Flush() => throw new NotSupportedException(); + + public static byte[] Create(int size) + { + byte[] data = new byte[size]; + using (TestPseudoTextStream rts = new TestPseudoTextStream()) + { + int bufferSize = 64 * 1024; // 64k blocks + byte[] buffer = new byte[bufferSize]; + int bytesRead = 0; + int totalBytesRead = 0; + + while (totalBytesRead < size) + { + bytesRead = rts.Read(buffer, 0, Math.Min(bufferSize, size - totalBytesRead)); + Array.Copy(buffer, 0, data, totalBytesRead, bytesRead); + totalBytesRead += bytesRead; + } + } + + return data; + } + + public override int Read(byte[] buffer, int offset, int count) + { + int bytesRead = 0; + while (bytesRead < count) + { + string word = GenerateDeterministicWord(_position + bytesRead); + byte[] wordBytes = System.Text.Encoding.ASCII.GetBytes(word + " "); + + int bytesToCopy = Math.Min(wordBytes.Length, count - bytesRead); + Array.Copy(wordBytes, 0, buffer, offset + bytesRead, bytesToCopy); + + bytesRead += bytesToCopy; + _position += bytesToCopy; + } + + return bytesRead; + } + + private string GenerateDeterministicWord(long seed) + { + int length = (int)(seed % 7) + 2; // 2 to 8 letters + int vowelCount = (int)((seed / 7) % 4) + 2; // 2 to 5 vowels + + System.Text.StringBuilder word = new System.Text.StringBuilder(length); + int vowelsAdded = 0; + + for (int i = 0; i < length; i++) + { + if (vowelsAdded < vowelCount && ((seed >> i) & 1) == 0) + { + word.Append(_vowels[(int)(seed >> (i + 1)) % _vowels.Length]); + vowelsAdded++; + } + else + { + word.Append(_consonants[(int)(seed >> (i + 1)) % _consonants.Length]); + } + } + + return word.ToString(); + } + + public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); + + public override void SetLength(long value) => throw new NotSupportedException(); + + public override void Write(byte[] buffer, int offset, int count) => + throw new NotSupportedException(); +} diff --git a/tests/SharpCompress.Test/Zip/Zip64AsyncTests.cs b/tests/SharpCompress.Test/Zip/Zip64AsyncTests.cs new file mode 100644 index 00000000..5124b6b0 --- /dev/null +++ b/tests/SharpCompress.Test/Zip/Zip64AsyncTests.cs @@ -0,0 +1,377 @@ +using System; +using System.IO; +using System.Linq; +using System.Threading.Tasks; +using SharpCompress.Archives; +using SharpCompress.Common; +using SharpCompress.Common.Zip; +using SharpCompress.Compressors.Deflate; +using SharpCompress.Readers; +using SharpCompress.Readers.Zip; +using SharpCompress.Test.Mocks; +using SharpCompress.Writers; +using SharpCompress.Writers.Zip; +using Xunit; + +namespace SharpCompress.Test.Zip; + +public class Zip64AsyncTests : WriterTests +{ + public Zip64AsyncTests() + : base(ArchiveType.Zip) { } + + // 4GiB + 1 + private const long FOUR_GB_LIMIT = ((long)uint.MaxValue) + 1; + + //[Fact] + [Trait("format", "zip64")] + public async ValueTask Zip64_Single_Large_File_Async() => + await RunSingleTestAsync(1, FOUR_GB_LIMIT, setZip64: true, forwardOnly: false); + + //[Fact] + [Trait("format", "zip64")] + public async ValueTask Zip64_Two_Large_Files_Async() => + await RunSingleTestAsync(2, FOUR_GB_LIMIT, setZip64: true, forwardOnly: false); + + //[Fact] + [Trait("format", "zip64")] + public async ValueTask Zip64_Two_Small_files_Async() => + // Multiple files, does not require zip64 + await RunSingleTestAsync(2, FOUR_GB_LIMIT / 2, setZip64: false, forwardOnly: false); + + // [Fact] + [Trait("format", "zip64")] + public async ValueTask Zip64_Two_Small_files_stream_Async() => + await RunSingleTestAsync(2, FOUR_GB_LIMIT / 2, setZip64: false, forwardOnly: true); + + // [Fact] + [Trait("format", "zip64")] + public async ValueTask Zip64_Two_Small_Files_Zip64_Async() => + // Multiple files, use zip64 even though it is not required + await RunSingleTestAsync(2, FOUR_GB_LIMIT / 2, setZip64: true, forwardOnly: false); + + // [Fact] + [Trait("format", "zip64")] + public async ValueTask Zip64_Single_Large_File_Fail_Async() + { + try + { + // One single file, should fail + await RunSingleTestAsync(1, FOUR_GB_LIMIT, setZip64: false, forwardOnly: false); + throw new InvalidOperationException("Test did not fail?"); + } + catch (NotSupportedException) { } + } + + // [Fact] + [Trait("zip64", "true")] + public async ValueTask Zip64_Single_Large_File_Zip64_Streaming_Fail_Async() + { + try + { + // One single file, should fail (fast) with zip64 + await RunSingleTestAsync(1, FOUR_GB_LIMIT, setZip64: true, forwardOnly: true); + throw new InvalidOperationException("Test did not fail?"); + } + catch (NotSupportedException) { } + } + + // [Fact] + [Trait("zip64", "true")] + public async ValueTask Zip64_Single_Large_File_Streaming_Fail_Async() + { + try + { + // One single file, should fail once the write discovers the problem + await RunSingleTestAsync(1, FOUR_GB_LIMIT, setZip64: false, forwardOnly: true); + throw new InvalidOperationException("Test did not fail?"); + } + catch (NotSupportedException) { } + } + + // Regression test for reading a Zip64 archive over a *non-seekable* async stream, as + // happens when extracting directly from a network download. When a >=4GB (Zip64) entry + // is followed by another entry, the streaming reader probes a few bytes past the big + // entry's data to locate the next header and must rewind them. For non-seekable streams + // it previously failed to do so (the rewind was gated on SeekableSharpCompressStream), + // leaving the reader misaligned so the *following* local header was parsed from garbage + // and extraction threw near the very end. A seekable stream rewinds correctly and works, + // which is exactly the "works seekable, fails non-seekable" symptom that was reported. + // + // NOTE: heavy (~4GB) like the other Zip64 large-file tests in this file, hence disabled + // by default. Enable to verify the fix. + //[Fact] + [Trait("format", "zip64")] + public async ValueTask Zip64_Large_File_Then_Small_File_NonSeekable_Async() + { + var filename = Path.Combine(SCRATCH2_FILES_PATH, "zip64-nonseekable-async.zip"); + + // A small trailing entry with recognizable content. Its bytes can only be read back + // correctly if the reader stays byte-aligned after the preceding >=4GB Zip64 entry. + var smallContent = new byte[64 * 1024]; + for (var i = 0; i < smallContent.Length; i++) + { + smallContent[i] = (byte)(i % 251); + } + + try + { + if (File.Exists(filename)) + { + File.Delete(filename); + } + + CreateLargeThenSmallZip(filename, FOUR_GB_LIMIT, smallContent); + + var (count, lastKey, lastContent) = await ReadLargeThenSmallNonSeekableAsync(filename); + + // The reader must reach the second (small) entry without throwing, identify it + // correctly, and read its bytes verbatim. + Assert.Equal(2, count); + Assert.Equal("small", lastKey); + Assert.NotNull(lastContent); + Assert.Equal(smallContent, lastContent!); + } + finally + { + if (File.Exists(filename)) + { + File.Delete(filename); + } + } + } + + private void CreateLargeThenSmallZip(string filename, long largeSize, byte[] smallContent) + { + var chunk = new byte[1024 * 1024]; + + // Force Zip64 and store (level 0) so the large entry's compressed size also exceeds + // 4GiB, which is what marks the entry as Zip64 for the streaming reader. + var opts = new ZipWriterOptions(CompressionType.Deflate) { UseZip64 = true }; + var eo = new ZipWriterEntryOptions { CompressionLevel = 0 }; + + using var zip = File.OpenWrite(filename); + using var zipWriter = (ZipWriter)WriterFactory.OpenWriter(zip, ArchiveType.Zip, opts); + + using (var str = zipWriter.WriteToStream("large", eo)) + { + var left = largeSize; + while (left > 0) + { + var b = (int)Math.Min(left, chunk.Length); + str.Write(chunk, 0, b); + left -= b; + } + } + + using (var str = zipWriter.WriteToStream("small", eo)) + { + str.Write(smallContent, 0, smallContent.Length); + } + } + + private async ValueTask<( + long Count, + string? LastKey, + byte[]? LastContent + )> ReadLargeThenSmallNonSeekableAsync(string filename) + { + long count = 0; + string? lastKey = null; + byte[]? lastContent = null; + + using var fs = File.OpenRead(filename); + // ForwardOnlyStream reports CanSeek == false; AsyncOnlyStream forces async reads. + // Together they emulate a non-seekable, async-only source (e.g. a network download). + // + // IMPORTANT: use default ReaderOptions (LeaveStreamOpen == false), exactly as the + // reporting user did. With LeaveStreamOpen == true the Volume wraps the stream in a + // passthrough that Create() later unwraps into a SeekableSharpCompressStream, which + // happens to take the working seek-back path and hides the bug. The default keeps a + // plain ring-buffer SharpCompressStream, which is where the streaming reader fails. + await using var rd = await ReaderFactory.OpenAsyncReader( + new AsyncOnlyStream(new ForwardOnlyStream(fs)), + new ReaderOptions { LookForHeader = false } + ); + while (await rd.MoveToNextEntryAsync()) + { + count++; + lastKey = rd.Entry.Key; + +#if LEGACY_DOTNET + using var entryStream = await rd.OpenEntryStreamAsync(); +#else + await using var entryStream = await rd.OpenEntryStreamAsync(); +#endif + if (rd.Entry.Key == "small") + { + using var ms = new MemoryStream(); + await entryStream.CopyToAsync(ms); + lastContent = ms.ToArray(); + } + else + { + await entryStream.SkipEntryAsync(); + } + } + + return (count, lastKey, lastContent); + } + + public async ValueTask RunSingleTestAsync( + long files, + long filesize, + bool setZip64, + bool forwardOnly, + long writeChunkSize = 1024 * 1024, + string filename = "zip64-test-async.zip" + ) + { + filename = Path.Combine(SCRATCH2_FILES_PATH, filename); + + try + { + if (File.Exists(filename)) + { + File.Delete(filename); + } + + if (!File.Exists(filename)) + { + await CreateZipArchiveAsync( + filename, + files, + filesize, + writeChunkSize, + setZip64, + forwardOnly + ); + } + + var resForward = await ReadForwardOnlyAsync(filename); + if (resForward.Item1 != files) + { + throw new InvalidOperationException( + $"Incorrect number of items reported: {resForward.Item1}, should have been {files}" + ); + } + + if (resForward.Item2 != files * filesize) + { + throw new InvalidOperationException( + $"Incorrect combined size reported: {resForward.Item2}, should have been {files * filesize}" + ); + } + + var resArchive = ReadArchive(filename); + if (resArchive.Item1 != files) + { + throw new InvalidOperationException( + $"Incorrect number of items reported: {resArchive.Item1}, should have been {files}" + ); + } + + if (resArchive.Item2 != files * filesize) + { + throw new InvalidOperationException( + $"Incorrect number of items reported: {resArchive.Item2}, should have been {files * filesize}" + ); + } + } + finally + { + if (File.Exists(filename)) + { + File.Delete(filename); + } + } + } + + public async ValueTask CreateZipArchiveAsync( + string filename, + long files, + long filesize, + long chunksize, + bool setZip64, + bool forwardOnly + ) + { + var data = new byte[chunksize]; + + // Use deflate for speed + var opts = new ZipWriterOptions(CompressionType.Deflate) { UseZip64 = setZip64 }; + + // Use no compression to ensure we hit the limits (actually inflates a bit, but seems better than using method==Store) + var eo = new ZipWriterEntryOptions { CompressionLevel = 0 }; + + using var zip = File.OpenWrite(filename); + using var st = forwardOnly ? (Stream)new ForwardOnlyStream(zip) : zip; + using var zipWriter = (ZipWriter)WriterFactory.OpenWriter(st, ArchiveType.Zip, opts); + for (var i = 0; i < files; i++) + { + using var str = zipWriter.WriteToStream(i.ToString(), eo); + var left = filesize; + while (left > 0) + { + var b = (int)Math.Min(left, data.Length); + // Use synchronous Write to match the sync version and avoid ForwardOnlyStream issues + await str.WriteAsync(data, 0, b); + left -= b; + } + } + } + + public async ValueTask> ReadForwardOnlyAsync(string filename) + { + long count = 0; + long size = 0; + ZipEntry? prev = null; + using (var fs = File.OpenRead(filename)) + { + await using var rd = await ReaderFactory.OpenAsyncReader( + new AsyncOnlyStream(fs), + ReaderOptions.ForExternalStream with + { + LookForHeader = false, + } + ); + while (await rd.MoveToNextEntryAsync()) + { +#if LEGACY_DOTNET + using (var entryStream = await rd.OpenEntryStreamAsync()) + { + await entryStream.SkipEntryAsync(); + } +#else + await using (var entryStream = await rd.OpenEntryStreamAsync()) + { + await entryStream.SkipEntryAsync(); + } +#endif + count++; + if (prev != null) + { + size += prev.Size; + } + + prev = (ZipEntry)rd.Entry; + } + } + + if (prev != null) + { + size += prev.Size; + } + + return new Tuple(count, size); + } + + public Tuple ReadArchive(string filename) + { + using var archive = ArchiveFactory.OpenArchive(filename); + return new Tuple( + archive.Entries.Count(), + archive.Entries.Select(x => x.Size).Sum() + ); + } +} diff --git a/tests/SharpCompress.Test/Zip/Zip64Tests.cs b/tests/SharpCompress.Test/Zip/Zip64Tests.cs index 739867f7..e1bac5bd 100644 --- a/tests/SharpCompress.Test/Zip/Zip64Tests.cs +++ b/tests/SharpCompress.Test/Zip/Zip64Tests.cs @@ -3,6 +3,8 @@ using System.IO; using System.Linq; using SharpCompress.Archives; using SharpCompress.Common; +using SharpCompress.Common.Zip; +using SharpCompress.Compressors.Deflate; using SharpCompress.Readers; using SharpCompress.Readers.Zip; using SharpCompress.Test.Mocks; @@ -20,62 +22,70 @@ public class Zip64Tests : WriterTests // 4GiB + 1 private const long FOUR_GB_LIMIT = ((long)uint.MaxValue) + 1; + //[Fact] [Trait("format", "zip64")] public void Zip64_Single_Large_File() => // One single file, requires zip64 - RunSingleTest(1, FOUR_GB_LIMIT, set_zip64: true, forward_only: false); + RunSingleTest(1, FOUR_GB_LIMIT, setZip64: true, forwardOnly: false); + //[Fact] [Trait("format", "zip64")] public void Zip64_Two_Large_Files() => // One single file, requires zip64 - RunSingleTest(2, FOUR_GB_LIMIT, set_zip64: true, forward_only: false); + RunSingleTest(2, FOUR_GB_LIMIT, setZip64: true, forwardOnly: false); + //[Fact] [Trait("format", "zip64")] public void Zip64_Two_Small_files() => // Multiple files, does not require zip64 - RunSingleTest(2, FOUR_GB_LIMIT / 2, set_zip64: false, forward_only: false); + RunSingleTest(2, FOUR_GB_LIMIT / 2, setZip64: false, forwardOnly: false); + //[Fact] [Trait("format", "zip64")] public void Zip64_Two_Small_files_stream() => // Multiple files, does not require zip64, and works with streams - RunSingleTest(2, FOUR_GB_LIMIT / 2, set_zip64: false, forward_only: true); + RunSingleTest(2, FOUR_GB_LIMIT / 2, setZip64: false, forwardOnly: true); + //[Fact] [Trait("format", "zip64")] public void Zip64_Two_Small_Files_Zip64() => // Multiple files, use zip64 even though it is not required - RunSingleTest(2, FOUR_GB_LIMIT / 2, set_zip64: true, forward_only: false); + RunSingleTest(2, FOUR_GB_LIMIT / 2, setZip64: true, forwardOnly: false); + // [Fact] [Trait("format", "zip64")] public void Zip64_Single_Large_File_Fail() { try { // One single file, should fail - RunSingleTest(1, FOUR_GB_LIMIT, set_zip64: false, forward_only: false); + RunSingleTest(1, FOUR_GB_LIMIT, setZip64: false, forwardOnly: false); throw new InvalidOperationException("Test did not fail?"); } catch (NotSupportedException) { } } + //[Fact] [Trait("zip64", "true")] public void Zip64_Single_Large_File_Zip64_Streaming_Fail() { try { // One single file, should fail (fast) with zip64 - RunSingleTest(1, FOUR_GB_LIMIT, set_zip64: true, forward_only: true); + RunSingleTest(1, FOUR_GB_LIMIT, setZip64: true, forwardOnly: true); throw new InvalidOperationException("Test did not fail?"); } catch (NotSupportedException) { } } + // [Fact] [Trait("zip64", "true")] public void Zip64_Single_Large_File_Streaming_Fail() { try { // One single file, should fail once the write discovers the problem - RunSingleTest(1, FOUR_GB_LIMIT, set_zip64: false, forward_only: true); + RunSingleTest(1, FOUR_GB_LIMIT, setZip64: false, forwardOnly: true); throw new InvalidOperationException("Test did not fail?"); } catch (NotSupportedException) { } @@ -84,9 +94,9 @@ public class Zip64Tests : WriterTests public void RunSingleTest( long files, long filesize, - bool set_zip64, - bool forward_only, - long write_chunk_size = 1024 * 1024, + bool setZip64, + bool forwardOnly, + long writeChunkSize = 1024 * 1024, string filename = "zip64-test.zip" ) { @@ -99,7 +109,7 @@ public class Zip64Tests : WriterTests if (!File.Exists(filename)) { - CreateZipArchive(filename, files, filesize, write_chunk_size, set_zip64, forward_only); + CreateZipArchive(filename, files, filesize, writeChunkSize, setZip64, forwardOnly); } var resForward = ReadForwardOnly(filename); @@ -138,24 +148,21 @@ public class Zip64Tests : WriterTests long files, long filesize, long chunksize, - bool set_zip64, - bool forward_only + bool setZip64, + bool forwardOnly ) { var data = new byte[chunksize]; // Use deflate for speed - var opts = new ZipWriterOptions(CompressionType.Deflate) { UseZip64 = set_zip64 }; + var opts = new ZipWriterOptions(CompressionType.Deflate) { UseZip64 = setZip64 }; // Use no compression to ensure we hit the limits (actually inflates a bit, but seems better than using method==Store) - var eo = new ZipWriterEntryOptions() - { - DeflateCompressionLevel = Compressors.Deflate.CompressionLevel.None - }; + var eo = new ZipWriterEntryOptions { CompressionLevel = 0 }; using var zip = File.OpenWrite(filename); - using var st = forward_only ? (Stream)new ForwardOnlyStream(zip) : zip; - using var zipWriter = (ZipWriter)WriterFactory.Open(st, ArchiveType.Zip, opts); + using var st = forwardOnly ? (Stream)new ForwardOnlyStream(zip) : zip; + using var zipWriter = (ZipWriter)WriterFactory.OpenWriter(st, ArchiveType.Zip, opts); for (var i = 0; i < files; i++) { using var str = zipWriter.WriteToStream(i.ToString(), eo); @@ -173,9 +180,17 @@ public class Zip64Tests : WriterTests { long count = 0; long size = 0; - Common.Zip.ZipEntry? prev = null; + IEntry? prev = null; using (var fs = File.OpenRead(filename)) - using (var rd = ZipReader.Open(fs, new ReaderOptions() { LookForHeader = false })) + using ( + var rd = ZipReader.OpenReader( + fs, + ReaderOptions.ForExternalStream with + { + LookForHeader = false, + } + ) + ) { while (rd.MoveToNextEntry()) { @@ -201,7 +216,7 @@ public class Zip64Tests : WriterTests public Tuple ReadArchive(string filename) { - using var archive = ArchiveFactory.Open(filename); + using var archive = ArchiveFactory.OpenArchive(filename); return new Tuple( archive.Entries.Count(), archive.Entries.Select(x => x.Size).Sum() diff --git a/tests/SharpCompress.Test/Zip/Zip64VersionConsistencyTests.cs b/tests/SharpCompress.Test/Zip/Zip64VersionConsistencyTests.cs new file mode 100644 index 00000000..9a06560f --- /dev/null +++ b/tests/SharpCompress.Test/Zip/Zip64VersionConsistencyTests.cs @@ -0,0 +1,441 @@ +using System; +using System.Buffers.Binary; +using System.IO; +using SharpCompress.Archives; +using SharpCompress.Archives.Zip; +using SharpCompress.Common; +using SharpCompress.Writers; +using SharpCompress.Writers.Zip; +using Xunit; + +namespace SharpCompress.Test.Zip; + +/// +/// Tests for verifying version consistency between Local File Header (LFH) +/// and Central Directory File Header (CDFH) when using Zip64. +/// +public class Zip64VersionConsistencyTests : WriterTests +{ + public Zip64VersionConsistencyTests() + : base(ArchiveType.Zip) { } + + [Fact] + public void Zip64_Small_File_With_UseZip64_Should_Have_Matching_Versions() + { + // Create a zip with UseZip64=true but with a small file + var filename = Path.Combine(SCRATCH2_FILES_PATH, "zip64_version_test.zip"); + + if (File.Exists(filename)) + { + File.Delete(filename); + } + + // Create archive with UseZip64=true + var writerOptions = new ZipWriterOptions(CompressionType.Deflate) + { + LeaveStreamOpen = false, + UseZip64 = true, + }; + + var zipArchive = ZipArchive.CreateArchive(); + zipArchive.AddEntry("empty", new MemoryStream()); + zipArchive.SaveTo(filename, writerOptions); + + // Now read the raw bytes to verify version consistency + using var fs = File.OpenRead(filename); + using var br = new BinaryReader(fs); + + // Read Local File Header + var lfhSignature = br.ReadUInt32(); + Assert.Equal(0x04034b50u, lfhSignature); // Local file header signature + + var lfhVersion = br.ReadUInt16(); + + // Skip to Central Directory + // Find Central Directory by searching from the end + fs.Seek(-22, SeekOrigin.End); // Min EOCD size + var eocdSignature = br.ReadUInt32(); + + if (eocdSignature != 0x06054b50u) + { + // Might have Zip64 EOCD, search backwards + fs.Seek(-100, SeekOrigin.End); + var buffer = new byte[100]; + fs.Read(buffer, 0, 100); + + // Find EOCD signature + for (int i = buffer.Length - 4; i >= 0; i--) + { + if (BinaryPrimitives.ReadUInt32LittleEndian(buffer.AsSpan(i)) == 0x06054b50u) + { + fs.Seek(-100 + i, SeekOrigin.End); + break; + } + } + } + + // Read EOCD + fs.Seek(-22, SeekOrigin.End); + br.ReadUInt32(); // EOCD signature + br.ReadUInt16(); // disk number + br.ReadUInt16(); // disk with central dir + br.ReadUInt16(); // entries on this disk + br.ReadUInt16(); // total entries + br.ReadUInt32(); // central directory size (unused) + var cdOffset = br.ReadUInt32(); + + // If Zip64, need to read from Zip64 EOCD + if (cdOffset == 0xFFFFFFFF) + { + // Find Zip64 EOCD Locator + fs.Seek(-22 - 20, SeekOrigin.End); + var z64eocdlSig = br.ReadUInt32(); + if (z64eocdlSig == 0x07064b50u) + { + br.ReadUInt32(); // disk number + var z64eocdOffset = br.ReadUInt64(); + br.ReadUInt32(); // total disks + + // Read Zip64 EOCD + fs.Seek((long)z64eocdOffset, SeekOrigin.Begin); + br.ReadUInt32(); // signature + br.ReadUInt64(); // size of EOCD64 + br.ReadUInt16(); // version made by + br.ReadUInt16(); // version needed + br.ReadUInt32(); // disk number + br.ReadUInt32(); // disk with CD + br.ReadUInt64(); // entries on disk + br.ReadUInt64(); // total entries + br.ReadUInt64(); // CD size + cdOffset = (uint)br.ReadUInt64(); // CD offset + } + } + + // Read Central Directory Header + fs.Seek(cdOffset, SeekOrigin.Begin); + var cdhSignature = br.ReadUInt32(); + Assert.Equal(0x02014b50u, cdhSignature); // Central directory header signature + + br.ReadUInt16(); // version made by + var cdhVersionNeeded = br.ReadUInt16(); + + // The versions should match when UseZip64 is true + Assert.Equal(lfhVersion, cdhVersionNeeded); + } + + [Fact] + public void Zip64_Small_File_Without_UseZip64_Should_Have_Version_20() + { + // Create a zip without UseZip64 + var filename = Path.Combine(SCRATCH2_FILES_PATH, "no_zip64_version_test.zip"); + + if (File.Exists(filename)) + { + File.Delete(filename); + } + + // Create archive without UseZip64 + var writerOptions = new ZipWriterOptions(CompressionType.Deflate) + { + LeaveStreamOpen = false, + UseZip64 = false, + }; + + var zipArchive = ZipArchive.CreateArchive(); + zipArchive.AddEntry("empty", new MemoryStream()); + zipArchive.SaveTo(filename, writerOptions); + + // Read the raw bytes + using var fs = File.OpenRead(filename); + using var br = new BinaryReader(fs); + + // Read Local File Header version + var lfhSignature = br.ReadUInt32(); + Assert.Equal(0x04034b50u, lfhSignature); + var lfhVersion = br.ReadUInt16(); + + // Read Central Directory Header version + fs.Seek(-22, SeekOrigin.End); + br.ReadUInt32(); // EOCD signature + br.ReadUInt16(); // disk number + br.ReadUInt16(); // disk with central dir + br.ReadUInt16(); // entries on this disk + br.ReadUInt16(); // total entries + br.ReadUInt32(); // CD size + var cdOffset = br.ReadUInt32(); + + fs.Seek(cdOffset, SeekOrigin.Begin); + var cdhSignature = br.ReadUInt32(); + Assert.Equal(0x02014b50u, cdhSignature); + br.ReadUInt16(); // version made by + var cdhVersionNeeded = br.ReadUInt16(); + + // Both should be version 20 (or less) + Assert.True(lfhVersion <= 20); + Assert.Equal(lfhVersion, cdhVersionNeeded); + } + + [Fact] + public void LZMA_Compression_Should_Use_Version_63() + { + // Create a zip with LZMA compression + var filename = Path.Combine(SCRATCH2_FILES_PATH, "lzma_version_test.zip"); + + if (File.Exists(filename)) + { + File.Delete(filename); + } + + var writerOptions = new ZipWriterOptions(CompressionType.LZMA) + { + LeaveStreamOpen = false, + UseZip64 = false, + }; + + var zipArchive = ZipArchive.CreateArchive(); + var data = new byte[100]; + new Random(42).NextBytes(data); + zipArchive.AddEntry("test.bin", new MemoryStream(data)); + zipArchive.SaveTo(filename, writerOptions); + + // Read the raw bytes + using var fs = File.OpenRead(filename); + using var br = new BinaryReader(fs); + + // Read Local File Header version + var lfhSignature = br.ReadUInt32(); + Assert.Equal(0x04034b50u, lfhSignature); + var lfhVersion = br.ReadUInt16(); + + // Read Central Directory Header version + fs.Seek(-22, SeekOrigin.End); + br.ReadUInt32(); // EOCD signature + br.ReadUInt16(); // disk number + br.ReadUInt16(); // disk with central dir + br.ReadUInt16(); // entries on this disk + br.ReadUInt16(); // total entries + br.ReadUInt32(); // CD size + var cdOffset = br.ReadUInt32(); + + fs.Seek(cdOffset, SeekOrigin.Begin); + var cdhSignature = br.ReadUInt32(); + Assert.Equal(0x02014b50u, cdhSignature); + br.ReadUInt16(); // version made by + var cdhVersionNeeded = br.ReadUInt16(); + + // Both should be version 63 for LZMA + Assert.Equal(63, lfhVersion); + Assert.Equal(lfhVersion, cdhVersionNeeded); + } + + [Fact] + public void PPMd_Compression_Should_Use_Version_63() + { + // Create a zip with PPMd compression + var filename = Path.Combine(SCRATCH2_FILES_PATH, "ppmd_version_test.zip"); + + if (File.Exists(filename)) + { + File.Delete(filename); + } + + var writerOptions = new ZipWriterOptions(CompressionType.PPMd) + { + LeaveStreamOpen = false, + UseZip64 = false, + }; + + var zipArchive = ZipArchive.CreateArchive(); + var data = new byte[100]; + new Random(42).NextBytes(data); + zipArchive.AddEntry("test.bin", new MemoryStream(data)); + zipArchive.SaveTo(filename, writerOptions); + + // Read the raw bytes + using var fs = File.OpenRead(filename); + using var br = new BinaryReader(fs); + + // Read Local File Header version + var lfhSignature = br.ReadUInt32(); + Assert.Equal(0x04034b50u, lfhSignature); + var lfhVersion = br.ReadUInt16(); + + // Read Central Directory Header version + fs.Seek(-22, SeekOrigin.End); + br.ReadUInt32(); // EOCD signature + br.ReadUInt16(); // disk number + br.ReadUInt16(); // disk with central dir + br.ReadUInt16(); // entries on this disk + br.ReadUInt16(); // total entries + br.ReadUInt32(); // CD size + var cdOffset = br.ReadUInt32(); + + fs.Seek(cdOffset, SeekOrigin.Begin); + var cdhSignature = br.ReadUInt32(); + Assert.Equal(0x02014b50u, cdhSignature); + br.ReadUInt16(); // version made by + var cdhVersionNeeded = br.ReadUInt16(); + + // Both should be version 63 for PPMd + Assert.Equal(63, lfhVersion); + Assert.Equal(lfhVersion, cdhVersionNeeded); + } + + [Fact] + public void Zip64_Multiple_Small_Files_With_UseZip64_Should_Have_Matching_Versions() + { + // Create a zip with UseZip64=true but with multiple small files + var filename = Path.Combine(SCRATCH2_FILES_PATH, "zip64_version_multiple_test.zip"); + + if (File.Exists(filename)) + { + File.Delete(filename); + } + + var writerOptions = new ZipWriterOptions(CompressionType.Deflate) + { + LeaveStreamOpen = false, + UseZip64 = true, + }; + + var zipArchive = ZipArchive.CreateArchive(); + for (int i = 0; i < 5; i++) + { + var data = new byte[100]; + new Random(i).NextBytes(data); + zipArchive.AddEntry($"file{i}.bin", new MemoryStream(data)); + } + zipArchive.SaveTo(filename, writerOptions); + + // Verify that all entries have matching versions + using var fs = File.OpenRead(filename); + using var br = new BinaryReader(fs); + + // Read all LFH versions + var lfhVersions = new System.Collections.Generic.List(); + while (true) + { + var sig = br.ReadUInt32(); + if (sig == 0x04034b50u) // LFH signature + { + var version = br.ReadUInt16(); + lfhVersions.Add(version); + + // Skip rest of LFH + br.ReadUInt16(); // flags + br.ReadUInt16(); // compression + br.ReadUInt32(); // mod time + br.ReadUInt32(); // crc + br.ReadUInt32(); // compressed size + br.ReadUInt32(); // uncompressed size + var fnLen = br.ReadUInt16(); + var extraLen = br.ReadUInt16(); + fs.Seek(fnLen + extraLen, SeekOrigin.Current); + + // Skip compressed data by reading compressed size from extra field if zip64 + // For simplicity in this test, we'll just find the next signature + var found = false; + + while (fs.Position < fs.Length - 4) + { + var b = br.ReadByte(); + if (b == 0x50) + { + var nextBytes = br.ReadBytes(3); + if ( + (nextBytes[0] == 0x4b && nextBytes[1] == 0x03 && nextBytes[2] == 0x04) + || // LFH + (nextBytes[0] == 0x4b && nextBytes[1] == 0x01 && nextBytes[2] == 0x02) + ) // CDH + { + fs.Seek(-4, SeekOrigin.Current); + found = true; + break; + } + } + } + + if (!found) + { + break; + } + } + else if (sig == 0x02014b50u) // CDH signature + { + break; // Reached central directory + } + else + { + break; // Unknown signature + } + } + + // Find Central Directory + fs.Seek(-22, SeekOrigin.End); + br.ReadUInt32(); // EOCD signature + br.ReadUInt16(); // disk number + br.ReadUInt16(); // disk with central dir + br.ReadUInt16(); // entries on this disk + var totalEntries = br.ReadUInt16(); + br.ReadUInt32(); // CD size + var cdOffset = br.ReadUInt32(); + + // Check if we need Zip64 EOCD + if (cdOffset == 0xFFFFFFFF) + { + fs.Seek(-22 - 20, SeekOrigin.End); + var z64eocdlSig = br.ReadUInt32(); + if (z64eocdlSig == 0x07064b50u) + { + br.ReadUInt32(); // disk number + var z64eocdOffset = br.ReadUInt64(); + fs.Seek((long)z64eocdOffset, SeekOrigin.Begin); + br.ReadUInt32(); // signature + br.ReadUInt64(); // size + br.ReadUInt16(); // version made by + br.ReadUInt16(); // version needed + br.ReadUInt32(); // disk number + br.ReadUInt32(); // disk with CD + br.ReadUInt64(); // entries on disk + totalEntries = (ushort)br.ReadUInt64(); // total entries + br.ReadUInt64(); // CD size + cdOffset = (uint)br.ReadUInt64(); // CD offset + } + } + + // Read CDH versions + fs.Seek(cdOffset, SeekOrigin.Begin); + var cdhVersions = new System.Collections.Generic.List(); + for (int i = 0; i < totalEntries; i++) + { + var sig = br.ReadUInt32(); + Assert.Equal(0x02014b50u, sig); + br.ReadUInt16(); // version made by + var version = br.ReadUInt16(); + cdhVersions.Add(version); + + // Skip rest of CDH + br.ReadUInt16(); // flags + br.ReadUInt16(); // compression + br.ReadUInt32(); // mod time + br.ReadUInt32(); // crc + br.ReadUInt32(); // compressed size + br.ReadUInt32(); // uncompressed size + var fnLen = br.ReadUInt16(); + var extraLen = br.ReadUInt16(); + var commentLen = br.ReadUInt16(); + br.ReadUInt16(); // disk number start + br.ReadUInt16(); // internal attributes + br.ReadUInt32(); // external attributes + br.ReadUInt32(); // LFH offset + fs.Seek(fnLen + extraLen + commentLen, SeekOrigin.Current); + } + + // Verify all versions match + Assert.Equal(lfhVersions.Count, cdhVersions.Count); + for (int i = 0; i < lfhVersions.Count; i++) + { + Assert.Equal(lfhVersions[i], cdhVersions[i]); + } + } +} diff --git a/tests/SharpCompress.Test/Zip/ZipArchiveAsyncTests.cs b/tests/SharpCompress.Test/Zip/ZipArchiveAsyncTests.cs new file mode 100644 index 00000000..420ce481 --- /dev/null +++ b/tests/SharpCompress.Test/Zip/ZipArchiveAsyncTests.cs @@ -0,0 +1,274 @@ +using System; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using SharpCompress; +using SharpCompress.Archives; +using SharpCompress.Archives.Zip; +using SharpCompress.Common; +using SharpCompress.Compressors.Deflate; +using SharpCompress.Test.Mocks; +using SharpCompress.Writers; +using SharpCompress.Writers.Zip; +using Xunit; + +namespace SharpCompress.Test.Zip; + +public class ZipArchiveAsyncTests : ArchiveTests +{ + public ZipArchiveAsyncTests() => UseExtensionInsteadOfNameToVerify = true; + + [Fact] + public async ValueTask Zip_ZipX_ArchiveStreamRead_Async() => + await ArchiveStreamReadAsync("Zip.zipx"); + + [Fact] + public async ValueTask Zip_BZip2_Streamed_ArchiveStreamRead_Async() => + await ArchiveStreamReadAsync("Zip.bzip2.dd.zip"); + + [Fact] + public async ValueTask Zip_BZip2_ArchiveStreamRead_Async() => + await ArchiveStreamReadAsync("Zip.bzip2.zip"); + + [Fact] + public async ValueTask Zip_Deflate_Streamed2_ArchiveStreamRead_Async() => + await ArchiveStreamReadAsync("Zip.deflate.dd-.zip"); + + [Fact] + public async ValueTask Zip_Deflate_Streamed_ArchiveStreamRead_Async() => + await ArchiveStreamReadAsync("Zip.deflate.dd.zip"); + + [Fact] + public async ValueTask Zip_Deflate_ArchiveStreamRead_Async() => + await ArchiveStreamReadAsync("Zip.deflate.zip"); + + [Fact] + public async ValueTask Zip_Deflate64_ArchiveStreamRead_Async() => + await ArchiveStreamReadAsync("Zip.deflate64.zip"); + + [Fact] + public async ValueTask Zip_LZMA_Streamed_ArchiveStreamRead_Async() => + await ArchiveStreamReadAsync("Zip.lzma.dd.zip"); + + [Fact] + public async ValueTask Zip_LZMA_ArchiveStreamRead_Async() => + await ArchiveStreamReadAsync("Zip.lzma.zip"); + + [Fact] + public async ValueTask Zip_PPMd_Streamed_ArchiveStreamRead_Async() => + await ArchiveStreamReadAsync("Zip.ppmd.dd.zip"); + + [Fact] + public async ValueTask Zip_PPMd_ArchiveStreamRead_Async() => + await ArchiveStreamReadAsync("Zip.ppmd.zip"); + + [Fact] + public async ValueTask Zip_None_ArchiveStreamRead_Async() => + await ArchiveStreamReadAsync("Zip.none.zip"); + + [Fact] + public async ValueTask Zip_Zip64_ArchiveStreamRead_Async() => + await ArchiveStreamReadAsync("Zip.zip64.zip"); + + [Fact] + public async ValueTask Zip_Shrink_ArchiveStreamRead_Async() + { + UseExtensionInsteadOfNameToVerify = true; + UseCaseInsensitiveToVerify = true; + await ArchiveStreamReadAsync("Zip.shrink.zip"); + } + + [Fact] + public async ValueTask Zip_Implode_ArchiveStreamRead_Async() + { + UseExtensionInsteadOfNameToVerify = true; + UseCaseInsensitiveToVerify = true; + await ArchiveStreamReadAsync("Zip.implode.zip"); + } + + [Fact] + public async ValueTask Zip_Reduce1_ArchiveStreamRead_Async() + { + UseExtensionInsteadOfNameToVerify = true; + UseCaseInsensitiveToVerify = true; + await ArchiveStreamReadAsync("Zip.reduce1.zip"); + } + + [Fact] + public async ValueTask Zip_Reduce2_ArchiveStreamRead_Async() + { + UseExtensionInsteadOfNameToVerify = true; + UseCaseInsensitiveToVerify = true; + await ArchiveStreamReadAsync("Zip.reduce2.zip"); + } + + [Fact] + public async ValueTask Zip_Reduce3_ArchiveStreamRead_Async() + { + UseExtensionInsteadOfNameToVerify = true; + UseCaseInsensitiveToVerify = true; + await ArchiveStreamReadAsync("Zip.reduce3.zip"); + } + + [Fact] + public async ValueTask Zip_Reduce4_ArchiveStreamRead_Async() + { + UseExtensionInsteadOfNameToVerify = true; + UseCaseInsensitiveToVerify = true; + await ArchiveStreamReadAsync("Zip.reduce4.zip"); + } + + [Fact] + public async ValueTask Zip_Random_Write_Remove_Async() + { + var scratchPath = Path.Combine(SCRATCH_FILES_PATH, "Zip.deflate.mod.zip"); + var unmodified = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.noEmptyDirs.zip"); + var modified = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.mod.zip"); + + await using (var archive = await ZipArchive.OpenAsyncArchive(unmodified)) + { + var entry = await archive.EntriesAsync.SingleAsync(x => + x.Key.NotNull().EndsWith("jpg", StringComparison.OrdinalIgnoreCase) + ); + await archive.RemoveEntryAsync(entry); + + var writerOptions = new ZipWriterOptions(CompressionType.Deflate) + { + ArchiveEncoding = new ArchiveEncoding { Default = Encoding.GetEncoding(866) }, + }; + + await archive.SaveToAsync(scratchPath, writerOptions); + } + CompareArchivesByPath(modified, scratchPath, Encoding.GetEncoding(866)); + } + + [Fact] + public async ValueTask Zip_Random_Write_Add_Async() + { + var jpg = Path.Combine(ORIGINAL_FILES_PATH, "jpg", "test.jpg"); + var scratchPath = Path.Combine(SCRATCH_FILES_PATH, "Zip.deflate.mod.zip"); + var unmodified = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.mod.zip"); + var modified = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.noEmptyDirs.zip"); + + await using (var archive = await ZipArchive.OpenAsyncArchive(unmodified)) + { + await archive.AddEntryAsync("jpg\\test.jpg", jpg); + + var writerOptions = new ZipWriterOptions(CompressionType.Deflate) + { + ArchiveEncoding = new ArchiveEncoding { Default = Encoding.GetEncoding(866) }, + }; + + await archive.SaveToAsync(scratchPath, writerOptions); + } + CompareArchivesByPath(modified, scratchPath, Encoding.GetEncoding(866)); + } + + [Fact] + public async ValueTask Zip_Create_New_Async() + { + var scratchPath = Path.Combine(SCRATCH_FILES_PATH, "Zip.deflate.noEmptyDirs.zip"); + var unmodified = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.noEmptyDirs.zip"); + + await using (var archive = (ZipArchive)await ZipArchive.CreateAsyncArchive()) + { + archive.DeflateCompressionLevel = CompressionLevel.BestSpeed; + archive.AddAllFromDirectory(ORIGINAL_FILES_PATH); + + var writerOptions = new ZipWriterOptions(CompressionType.Deflate) + { + ArchiveEncoding = new ArchiveEncoding { Default = Encoding.UTF8 }, + }; + + await archive.SaveToAsync(scratchPath, writerOptions); + } + CompareArchivesByPath(unmodified, scratchPath); + } + + [Fact] + public async ValueTask Zip_Async_Dispose_Closes_New_Entry_Stream() + { + var entryStream = new TestStream(new MemoryStream(Encoding.UTF8.GetBytes("test"))); + + await using (var archive = await ZipArchive.CreateAsyncArchive()) + { + await archive.AddEntryAsync( + "test.txt", + entryStream, + closeStream: true, + size: entryStream.Length + ); + await archive.SaveToAsync( + new MemoryStream(), + new ZipWriterOptions(CompressionType.Deflate) + ); + } + + Assert.True(entryStream.IsDisposed); + } + + [Fact] + public async ValueTask Zip_Deflate_Entry_Stream_Async() + { + using (Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.zip"))) + { + IAsyncArchive archive = await ZipArchive.OpenAsyncArchive(new AsyncOnlyStream(stream)); + try + { + await foreach (var entry in archive.EntriesAsync.Where(entry => !entry.IsDirectory)) + { + await entry.WriteToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + finally + { + await archive.DisposeAsync(); + } + } + VerifyFiles(); + } + + [Fact] + public async ValueTask Zip_Deflate_Archive_WriteToDirectoryAsync() + { + using (Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.zip"))) + { + IAsyncArchive archive = await ZipArchive.OpenAsyncArchive(new AsyncOnlyStream(stream)); + try + { + await archive.WriteToDirectoryAsync(SCRATCH_FILES_PATH); + } + finally + { + await archive.DisposeAsync(); + } + } + VerifyFiles(); + } + + [Fact] + public async ValueTask Zip_Deflate_Archive_WriteToDirectoryAsync_WithProgress() + { + var progressReports = new System.Collections.Generic.List(); + var progress = new Progress(report => progressReports.Add(report)); + +#if NETFRAMEWORK + using (Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.zip"))) +#else + await using ( + Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.zip")) + ) +#endif + { + await using IAsyncArchive archive = await ZipArchive.OpenAsyncArchive( + new AsyncOnlyStream(stream) + ); + await archive.WriteToDirectoryAsync(SCRATCH_FILES_PATH, progress: progress); + } + + await Task.Delay(1000); + VerifyFiles(); + Assert.True(progressReports.Count > 0, "Progress reports should be generated"); + } +} diff --git a/tests/SharpCompress.Test/Zip/ZipArchiveDirectoryTests.cs b/tests/SharpCompress.Test/Zip/ZipArchiveDirectoryTests.cs new file mode 100644 index 00000000..7b6dc981 --- /dev/null +++ b/tests/SharpCompress.Test/Zip/ZipArchiveDirectoryTests.cs @@ -0,0 +1,113 @@ +using System; +using System.IO; +using System.Linq; +using SharpCompress.Archives.Zip; +using SharpCompress.Common; +using SharpCompress.Writers.Zip; +using Xunit; + +namespace SharpCompress.Test.Zip; + +public class ZipArchiveDirectoryTests : TestBase +{ + [Fact] + public void ZipArchive_AddDirectoryEntry_CreatesDirectoryEntry() + { + using var archive = ZipArchive.CreateArchive(); + + archive.AddDirectoryEntry("test-dir", DateTime.Now); + + var entries = archive.Entries.ToList(); + Assert.Single(entries); + Assert.Equal("test-dir", entries[0].Key); + Assert.True(entries[0].IsDirectory); + } + + [Fact] + public void ZipArchive_AddDirectoryEntry_MultipleDirectories() + { + using var archive = ZipArchive.CreateArchive(); + + archive.AddDirectoryEntry("dir1", DateTime.Now); + archive.AddDirectoryEntry("dir2", DateTime.Now); + archive.AddDirectoryEntry("dir1/subdir", DateTime.Now); + + var entries = archive.Entries.OrderBy(e => e.Key).ToList(); + Assert.Equal(3, entries.Count); + Assert.True(entries.All(e => e.IsDirectory)); + } + + [Fact] + public void ZipArchive_AddDirectoryEntry_MixedWithFiles() + { + using var archive = ZipArchive.CreateArchive(); + + archive.AddDirectoryEntry("dir1", DateTime.Now); + + using var contentStream = new MemoryStream( + System.Text.Encoding.UTF8.GetBytes("test content") + ); + archive.AddEntry("dir1/file.txt", contentStream, false, contentStream.Length, DateTime.Now); + + archive.AddDirectoryEntry("dir2", DateTime.Now); + + var entries = archive.Entries.OrderBy(e => e.Key).ToList(); + Assert.Equal(3, entries.Count); + Assert.True(entries[0].IsDirectory); + Assert.False(entries[1].IsDirectory); + Assert.True(entries[2].IsDirectory); + } + + [Fact] + public void ZipArchive_AddDirectoryEntry_SaveAndReload() + { + var scratchPath = Path.Combine(SCRATCH_FILES_PATH, "zip-directory-test.zip"); + + using (var archive = ZipArchive.CreateArchive()) + { + archive.AddDirectoryEntry("dir1", DateTime.Now); + archive.AddDirectoryEntry("dir2", DateTime.Now); + + using var contentStream = new MemoryStream( + System.Text.Encoding.UTF8.GetBytes("test content") + ); + archive.AddEntry( + "dir1/file.txt", + contentStream, + false, + contentStream.Length, + DateTime.Now + ); + + using (var fileStream = File.Create(scratchPath)) + { + archive.SaveTo(fileStream, new ZipWriterOptions(CompressionType.Deflate)); + } + } + + using (var archive = ZipArchive.OpenArchive(scratchPath)) + { + var entries = archive.Entries.OrderBy(e => e.Key).ToList(); + Assert.Equal(3, entries.Count); + + Assert.Equal("dir1/", entries[0].Key); + Assert.True(entries[0].IsDirectory); + + Assert.Equal("dir1/file.txt", entries[1].Key); + Assert.False(entries[1].IsDirectory); + + Assert.Equal("dir2/", entries[2].Key); + Assert.True(entries[2].IsDirectory); + } + } + + [Fact] + public void ZipArchive_AddDirectoryEntry_DuplicateKey_ThrowsException() + { + using var archive = ZipArchive.CreateArchive(); + + archive.AddDirectoryEntry("test-dir", DateTime.Now); + + Assert.Throws(() => archive.AddDirectoryEntry("test-dir", DateTime.Now)); + } +} diff --git a/tests/SharpCompress.Test/Zip/ZipArchiveTests.cs b/tests/SharpCompress.Test/Zip/ZipArchiveTests.cs index fde87dc6..ebee095a 100644 --- a/tests/SharpCompress.Test/Zip/ZipArchiveTests.cs +++ b/tests/SharpCompress.Test/Zip/ZipArchiveTests.cs @@ -5,7 +5,9 @@ using System.Text; using SharpCompress.Archives; using SharpCompress.Archives.Zip; using SharpCompress.Common; +using SharpCompress.Common.Zip; using SharpCompress.Readers; +using SharpCompress.Test.Mocks; using SharpCompress.Writers; using SharpCompress.Writers.Zip; using Xunit; @@ -14,6 +16,8 @@ namespace SharpCompress.Test.Zip; public class ZipArchiveTests : ArchiveTests { + private const long GeneratedZip64EntrySize = (long)uint.MaxValue + 1; + public ZipArchiveTests() => UseExtensionInsteadOfNameToVerify = true; [Fact] @@ -25,6 +29,23 @@ public class ZipArchiveTests : ArchiveTests [Fact] public void Zip_BZip2_ArchiveStreamRead() => ArchiveStreamRead("Zip.bzip2.zip"); + [Fact] + public void ZipArchive_StreamCollection_Throws_On_NonSeekable_Stream() + { + using var nonSeekable = new NonSeekableMemoryStream(); + using var seekable = new MemoryStream(); + + Assert.Throws(() => ZipArchive.OpenArchive([nonSeekable, seekable])); + } + + [Fact] + public void ZipArchive_Stream_Throws_On_Unreadable_Stream() + { + using var unreadable = new TestStream(new MemoryStream(), false, true, true); + + Assert.Throws(() => ZipArchive.OpenArchive(unreadable)); + } + [Fact] public void Zip_Deflate_Streamed2_ArchiveStreamRead() => ArchiveStreamRead("Zip.deflate.dd-.zip"); @@ -79,6 +100,19 @@ public class ZipArchiveTests : ArchiveTests [Fact] public void WinZip27_X_XZ_ArchiveFileRead() => ArchiveFileRead("WinZip27_XZ.zipx"); + [Fact] + public void WinZip27_X_XZ_Reports_CompressionType_Xz() + { + using var archive = ArchiveFactory.OpenArchive( + Path.Combine(TEST_ARCHIVES_PATH, "WinZip27_XZ.zipx") + ); + + foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) + { + Assert.Equal(CompressionType.Xz, entry.CompressionType); + } + } + [Fact] public void Zip_Deflate_Streamed2_ArchiveFileRead() => ArchiveFileRead("Zip.deflate.dd-.zip"); @@ -88,6 +122,10 @@ public class ZipArchiveTests : ArchiveTests [Fact] public void Zip_Deflate_ArchiveFileRead() => ArchiveFileRead("Zip.deflate.zip"); + [Fact] + public void Zip_Deflate_ArchiveExtractToDirectory() => + ArchiveExtractToDirectory("Zip.deflate.zip"); + //will detect and load other files [Fact] public void Zip_Deflate_Multi_ArchiveFirstFileRead() => @@ -121,7 +159,7 @@ public class ZipArchiveTests : ArchiveTests "Zip.deflate.split.003", "Zip.deflate.split.004", "Zip.deflate.split.005", - "Zip.deflate.split.006" + "Zip.deflate.split.006", } ); @@ -175,22 +213,72 @@ public class ZipArchiveTests : ArchiveTests [Fact] public void Zip_Zip64_ArchiveFileRead() => ArchiveFileRead("Zip.zip64.zip"); + [Fact] + public void Zip_Shrink_ArchiveStreamRead() + { + UseExtensionInsteadOfNameToVerify = true; + UseCaseInsensitiveToVerify = true; + ArchiveStreamRead("Zip.shrink.zip"); + } + + [Fact] + public void Zip_Implode_ArchiveStreamRead() + { + UseExtensionInsteadOfNameToVerify = true; + UseCaseInsensitiveToVerify = true; + ArchiveStreamRead("Zip.implode.zip"); + } + + [Fact] + public void Zip_Reduce1_ArchiveStreamRead() + { + UseExtensionInsteadOfNameToVerify = true; + UseCaseInsensitiveToVerify = true; + ArchiveStreamRead("Zip.reduce1.zip"); + } + + [Fact] + public void Zip_Reduce2_ArchiveStreamRead() + { + UseExtensionInsteadOfNameToVerify = true; + UseCaseInsensitiveToVerify = true; + ArchiveStreamRead("Zip.reduce2.zip"); + } + + [Fact] + public void Zip_Reduce3_ArchiveStreamRead() + { + UseExtensionInsteadOfNameToVerify = true; + UseCaseInsensitiveToVerify = true; + ArchiveStreamRead("Zip.reduce3.zip"); + } + + [Fact] + public void Zip_Reduce4_ArchiveStreamRead() + { + UseExtensionInsteadOfNameToVerify = true; + UseCaseInsensitiveToVerify = true; + ArchiveStreamRead("Zip.reduce4.zip"); + } + [Fact] public void Zip_Random_Write_Remove() { - string scratchPath = Path.Combine(SCRATCH_FILES_PATH, "Zip.deflate.mod.zip"); - string unmodified = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.noEmptyDirs.zip"); - string modified = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.mod.zip"); + var scratchPath = Path.Combine(SCRATCH_FILES_PATH, "Zip.deflate.mod.zip"); + var unmodified = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.noEmptyDirs.zip"); + var modified = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.mod.zip"); - using (var archive = ZipArchive.Open(unmodified)) + using (var archive = ZipArchive.OpenArchive(unmodified)) { - var entry = archive.Entries.Single( - x => x.Key.EndsWith("jpg", StringComparison.OrdinalIgnoreCase) + var entry = archive.Entries.Single(x => + x.Key.NotNull().EndsWith("jpg", StringComparison.OrdinalIgnoreCase) ); archive.RemoveEntry(entry); - WriterOptions writerOptions = new ZipWriterOptions(CompressionType.Deflate); - writerOptions.ArchiveEncoding.Default = Encoding.GetEncoding(866); + var writerOptions = new ZipWriterOptions(CompressionType.Deflate) + { + ArchiveEncoding = new ArchiveEncoding { Default = Encoding.GetEncoding(866) }, + }; archive.SaveTo(scratchPath, writerOptions); } @@ -200,17 +288,19 @@ public class ZipArchiveTests : ArchiveTests [Fact] public void Zip_Random_Write_Add() { - string jpg = Path.Combine(ORIGINAL_FILES_PATH, "jpg", "test.jpg"); - string scratchPath = Path.Combine(SCRATCH_FILES_PATH, "Zip.deflate.mod.zip"); - string unmodified = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.mod.zip"); - string modified = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.mod2.zip"); + var jpg = Path.Combine(ORIGINAL_FILES_PATH, "jpg", "test.jpg"); + var scratchPath = Path.Combine(SCRATCH_FILES_PATH, "Zip.deflate.mod.zip"); + var unmodified = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.mod.zip"); + var modified = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.mod2.zip"); - using (var archive = ZipArchive.Open(unmodified)) + using (var archive = ZipArchive.OpenArchive(unmodified)) { archive.AddEntry("jpg\\test.jpg", jpg); - WriterOptions writerOptions = new ZipWriterOptions(CompressionType.Deflate); - writerOptions.ArchiveEncoding.Default = Encoding.GetEncoding(866); + var writerOptions = new ZipWriterOptions(CompressionType.Deflate) + { + ArchiveEncoding = new ArchiveEncoding { Default = Encoding.GetEncoding(866) }, + }; archive.SaveTo(scratchPath, writerOptions); } @@ -220,16 +310,16 @@ public class ZipArchiveTests : ArchiveTests [Fact] public void Zip_Save_Twice() { - string scratchPath1 = Path.Combine(SCRATCH_FILES_PATH, "a.zip"); - string scratchPath2 = Path.Combine(SCRATCH_FILES_PATH, "b.zip"); + var scratchPath1 = Path.Combine(SCRATCH_FILES_PATH, "a.zip"); + var scratchPath2 = Path.Combine(SCRATCH_FILES_PATH, "b.zip"); - using (var arc = ZipArchive.Create()) + using (var arc = ZipArchive.CreateArchive()) { - string str = "test.txt"; + var str = "test.txt"; var source = new MemoryStream(Encoding.UTF8.GetBytes(str)); arc.AddEntry("test.txt", source, true, source.Length); - arc.SaveTo(scratchPath1, CompressionType.Deflate); - arc.SaveTo(scratchPath2, CompressionType.Deflate); + arc.SaveTo(scratchPath1, new ZipWriterOptions(CompressionType.Deflate)); + arc.SaveTo(scratchPath2, new ZipWriterOptions(CompressionType.Deflate)); } Assert.Equal(new FileInfo(scratchPath1).Length, new FileInfo(scratchPath2).Length); @@ -238,51 +328,47 @@ public class ZipArchiveTests : ArchiveTests [Fact] public void Zip_Removal_Poly() { - string scratchPath = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.noEmptyDirs.zip"); + var scratchPath = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.noEmptyDirs.zip"); - using (ZipArchive vfs = (ZipArchive)ArchiveFactory.Open(scratchPath)) - { - var e = vfs.Entries.First( - v => v.Key.EndsWith("jpg", StringComparison.OrdinalIgnoreCase) - ); - vfs.RemoveEntry(e); - Assert.Null( - vfs.Entries.FirstOrDefault( - v => v.Key.EndsWith("jpg", StringComparison.OrdinalIgnoreCase) - ) - ); - Assert.Null( - ((IArchive)vfs).Entries.FirstOrDefault( - v => v.Key.EndsWith("jpg", StringComparison.OrdinalIgnoreCase) - ) - ); - } + using var vfs = (ZipArchive)ArchiveFactory.OpenArchive(scratchPath); + var e = vfs.Entries.First(v => + v.Key.NotNull().EndsWith("jpg", StringComparison.OrdinalIgnoreCase) + ); + vfs.RemoveEntry(e); + Assert.Null( + vfs.Entries.FirstOrDefault(v => + v.Key.NotNull().EndsWith("jpg", StringComparison.OrdinalIgnoreCase) + ) + ); + Assert.Null( + ((IArchive)vfs).Entries.FirstOrDefault(v => + v.Key.NotNull().EndsWith("jpg", StringComparison.OrdinalIgnoreCase) + ) + ); } [Fact] public void Zip_Create_NoDups() { - using (var arc = ZipArchive.Create()) - { - arc.AddEntry("1.txt", new MemoryStream()); - Assert.Throws(() => arc.AddEntry("\\1.txt", new MemoryStream())); - } + using var arc = ZipArchive.CreateArchive(); + arc.AddEntry("1.txt", new MemoryStream()); + Assert.Throws(() => arc.AddEntry("\\1.txt", new MemoryStream())); } [Fact] public void Zip_Create_Same_Stream() { - string scratchPath1 = Path.Combine(SCRATCH_FILES_PATH, "a.zip"); - string scratchPath2 = Path.Combine(SCRATCH_FILES_PATH, "b.zip"); + var scratchPath1 = Path.Combine(SCRATCH_FILES_PATH, "a.zip"); + var scratchPath2 = Path.Combine(SCRATCH_FILES_PATH, "b.zip"); - using (var arc = ZipArchive.Create()) + using (var arc = ZipArchive.CreateArchive()) { using (var stream = new MemoryStream(Encoding.UTF8.GetBytes("qwert"))) { arc.AddEntry("1.txt", stream, false, stream.Length); arc.AddEntry("2.txt", stream, false, stream.Length); - arc.SaveTo(scratchPath1, CompressionType.Deflate); - arc.SaveTo(scratchPath2, CompressionType.Deflate); + arc.SaveTo(scratchPath1, new ZipWriterOptions(CompressionType.Deflate)); + arc.SaveTo(scratchPath2, new ZipWriterOptions(CompressionType.Deflate)); } } @@ -318,20 +404,21 @@ public class ZipArchiveTests : ArchiveTests } File.Copy(file, newFileName); } - string scratchPath = Path.Combine(SCRATCH2_FILES_PATH, "Zip.deflate.noEmptyDirs.zip"); - string unmodified = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.noEmptyDirs.zip"); + var scratchPath = Path.Combine(SCRATCH2_FILES_PATH, "Zip.deflate.noEmptyDirs.zip"); + var unmodified = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.noEmptyDirs.zip"); - using (var archive = ZipArchive.Create()) + using (var archive = ZipArchive.CreateArchive()) { archive.AddAllFromDirectory(SCRATCH_FILES_PATH); - WriterOptions writerOptions = new ZipWriterOptions(CompressionType.Deflate); - writerOptions.ArchiveEncoding.Default = Encoding.GetEncoding(866); + var writerOptions = new ZipWriterOptions(CompressionType.Deflate) + { + ArchiveEncoding = new ArchiveEncoding { Default = Encoding.GetEncoding(866) }, + }; archive.SaveTo(scratchPath, writerOptions); } CompareArchivesByPath(unmodified, scratchPath, Encoding.GetEncoding(866)); - Directory.Delete(SCRATCH_FILES_PATH, true); } /// @@ -341,15 +428,15 @@ public class ZipArchiveTests : ArchiveTests [Fact] public void Zip_Create_Empty_And_Read() { - var archive = ZipArchive.Create(); + var archive = ZipArchive.CreateArchive(); var archiveStream = new MemoryStream(); - archive.SaveTo(archiveStream, CompressionType.LZMA); + archive.SaveTo(archiveStream, new ZipWriterOptions(CompressionType.LZMA)); archiveStream.Position = 0; - var readArchive = ArchiveFactory.Open(archiveStream); + var readArchive = ArchiveFactory.OpenArchive(archiveStream); var count = readArchive.Entries.Count(); @@ -385,41 +472,40 @@ public class ZipArchiveTests : ArchiveTests } File.Copy(file, newFileName); } - string scratchPath = Path.Combine(SCRATCH2_FILES_PATH, "Zip.deflate.noEmptyDirs.zip"); + var scratchPath = Path.Combine(SCRATCH2_FILES_PATH, "Zip.deflate.noEmptyDirs.zip"); - using (var archive = ZipArchive.Create()) + using (var archive = ZipArchive.CreateArchive()) { archive.AddAllFromDirectory(SCRATCH_FILES_PATH); archive.RemoveEntry( - archive.Entries.Single( - x => x.Key.EndsWith("jpg", StringComparison.OrdinalIgnoreCase) + archive.Entries.Single(x => + x.Key.NotNull().EndsWith("jpg", StringComparison.OrdinalIgnoreCase) ) ); Assert.Null( - archive.Entries.FirstOrDefault( - x => x.Key.EndsWith("jpg", StringComparison.OrdinalIgnoreCase) + archive.Entries.FirstOrDefault(x => + x.Key.NotNull().EndsWith("jpg", StringComparison.OrdinalIgnoreCase) ) ); } - Directory.Delete(SCRATCH_FILES_PATH, true); } [Fact] public void Zip_Deflate_WinzipAES_Read() { using ( - var reader = ZipArchive.Open( + var reader = ZipArchive.OpenArchive( Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.WinzipAES.zip"), - new ReaderOptions() { Password = "test" } + ReaderOptions.ForFilePath with + { + Password = "test", + } ) ) { foreach (var entry in reader.Entries.Where(x => !x.IsDirectory)) { - entry.WriteToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + entry.WriteToDirectory(SCRATCH_FILES_PATH); } } VerifyFiles(); @@ -428,58 +514,174 @@ public class ZipArchiveTests : ArchiveTests [Fact] public void Zip_Deflate_WinzipAES_MultiOpenEntryStream() { - using ( - var reader = ZipArchive.Open( - Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.WinzipAES2.zip"), - new ReaderOptions() { Password = "test" } - ) - ) - { - foreach (var entry in reader.Entries.Where(x => !x.IsDirectory)) + using var reader = ZipArchive.OpenArchive( + Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.WinzipAES2.zip"), + ReaderOptions.ForFilePath with { - var stream = entry.OpenEntryStream(); - Assert.NotNull(stream); - var ex = Record.Exception(() => stream = entry.OpenEntryStream()); - Assert.Null(ex); + Password = "test", } + ); + foreach (var entry in reader.Entries.Where(x => !x.IsDirectory)) + { + var stream = entry.OpenEntryStream(); + Assert.NotNull(stream); + var ex = Record.Exception(() => stream = entry.OpenEntryStream()); + Assert.Null(ex); + } + } + + [Fact] + public void Zip_WinzipAES_CompressionType() + { + // Test that WinZip AES encrypted entries correctly report their compression type + using var deflateArchive = ZipArchive.OpenArchive( + Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.WinzipAES.zip"), + ReaderOptions.ForFilePath with + { + Password = "test", + } + ); + foreach (var entry in deflateArchive.Entries.Where(x => !x.IsDirectory)) + { + Assert.True(entry.IsEncrypted); + Assert.Equal(CompressionType.Deflate, entry.CompressionType); + } + + using var lzmaArchive = ZipArchive.OpenArchive( + Path.Combine(TEST_ARCHIVES_PATH, "Zip.lzma.WinzipAES.zip"), + ReaderOptions.ForFilePath with + { + Password = "test", + } + ); + foreach (var entry in lzmaArchive.Entries.Where(x => !x.IsDirectory)) + { + Assert.True(entry.IsEncrypted); + Assert.Equal(CompressionType.LZMA, entry.CompressionType); + } + } + + [Fact] + public void Zip_Zstandard_WinzipAES_Mixed_ArchiveFileRead() + { + using var archive = ZipArchive.OpenArchive( + Path.Combine(TEST_ARCHIVES_PATH, "Zip.zstd.WinzipAES.mixed.zip"), + ReaderOptions.ForFilePath with + { + Password = "test", + } + ); + + VerifyMixedZstandardArchive(archive); + } + + [Fact] + public void Zip_Zstandard_WinzipAES_Mixed_ArchiveStreamRead() + { + using var stream = File.OpenRead( + Path.Combine(TEST_ARCHIVES_PATH, "Zip.zstd.WinzipAES.mixed.zip") + ); + using var archive = ZipArchive.OpenArchive( + stream, + ReaderOptions.ForExternalStream with + { + Password = "test", + } + ); + + VerifyMixedZstandardArchive(archive); + } + + [Fact(Explicit = true)] + [Trait("zip64", "generated")] + public void Zip_Zip64_GeneratedArchive_StreamIsBoundedToEntryLength() + { + var zipPath = Path.Combine(SCRATCH2_FILES_PATH, "generated.zip64.large.zip"); + CreateZip64Archive(zipPath); + + using var archive = ZipArchive.OpenArchive(zipPath); + var entries = archive + .Entries.Where(x => !x.IsDirectory) + .OrderByDescending(x => x.Size) + .ToArray(); + + Assert.Equal(2, entries.Length); + Assert.Equal(GeneratedZip64EntrySize, entries[0].Size); + Assert.Equal(1, entries[1].Size); + + using var firstStream = entries[0].OpenEntryStream(); + Assert.Equal(entries[0].Size, CountBytes(firstStream)); + + using var secondStream = entries[1].OpenEntryStream(); + Assert.Equal(0x42, secondStream.ReadByte()); + Assert.Equal(-1, secondStream.ReadByte()); + } + + [Fact] + public void Zip_Pkware_CompressionType() + { + // Test that Pkware encrypted entries correctly report their compression type + using var deflateArchive = ZipArchive.OpenArchive( + Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.pkware.zip"), + ReaderOptions.ForFilePath with + { + Password = "test", + } + ); + foreach (var entry in deflateArchive.Entries.Where(x => !x.IsDirectory)) + { + Assert.True(entry.IsEncrypted); + Assert.Equal(CompressionType.Deflate, entry.CompressionType); + } + + using var bzip2Archive = ZipArchive.OpenArchive( + Path.Combine(TEST_ARCHIVES_PATH, "Zip.bzip2.pkware.zip"), + ReaderOptions.ForFilePath with + { + Password = "test", + } + ); + foreach (var entry in bzip2Archive.Entries.Where(x => !x.IsDirectory)) + { + Assert.True(entry.IsEncrypted); + Assert.Equal(CompressionType.BZip2, entry.CompressionType); } } [Fact] public void Zip_Read_Volume_Comment() { - using ( - var reader = ZipArchive.Open( - Path.Combine(TEST_ARCHIVES_PATH, "Zip.zip64.zip"), - new ReaderOptions() { Password = "test" } - ) - ) - { - var isComplete = reader.IsComplete; - Assert.Equal(1, reader.Volumes.Count); + using var reader = ZipArchive.OpenArchive( + Path.Combine(TEST_ARCHIVES_PATH, "Zip.zip64.zip"), + ReaderOptions.ForFilePath with + { + Password = "test", + } + ); + var isComplete = reader.IsComplete; + Assert.Equal(1, reader.Volumes.Count()); - string expectedComment = - "Encoding:utf-8 || Compression:Deflate levelDefault || Encrypt:None || ZIP64:Always\r\nCreated at 2017-Jan-23 14:10:43 || DotNetZip Tool v1.9.1.8\r\nTest zip64 archive"; - Assert.Equal(expectedComment, reader.Volumes.First().Comment); - } + var expectedComment = + "Encoding:utf-8 || Compression:Deflate levelDefault || Encrypt:None || ZIP64:Always\r\nCreated at 2017-Jan-23 14:10:43 || DotNetZip Tool v1.9.1.8\r\nTest zip64 archive"; + Assert.Equal(expectedComment, ((ZipVolume)reader.Volumes.First()).Comment); } [Fact] public void Zip_BZip2_Pkware_Read() { using ( - var reader = ZipArchive.Open( + var reader = ZipArchive.OpenArchive( Path.Combine(TEST_ARCHIVES_PATH, "Zip.bzip2.pkware.zip"), - new ReaderOptions() { Password = "test" } + ReaderOptions.ForFilePath with + { + Password = "test", + } ) ) { foreach (var entry in reader.Entries.Where(x => !x.IsDirectory)) { - entry.WriteToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + entry.WriteToDirectory(SCRATCH_FILES_PATH); } } VerifyFiles(); @@ -488,10 +690,10 @@ public class ZipArchiveTests : ArchiveTests [Fact] public void Zip_Random_Entry_Access() { - string unmodified = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.noEmptyDirs.zip"); + var unmodified = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.noEmptyDirs.zip"); - ZipArchive a = ZipArchive.Open(unmodified); - int count = 0; + var a = ZipArchive.OpenArchive(unmodified); + var count = 0; foreach (var e in a.Entries) { count++; @@ -501,8 +703,8 @@ public class ZipArchiveTests : ArchiveTests Assert.Equal(3, count); a.Dispose(); - a = ZipArchive.Open(unmodified); - int count2 = 0; + a = ZipArchive.OpenArchive(unmodified); + var count2 = 0; foreach (var e in a.Entries) { @@ -518,7 +720,7 @@ public class ZipArchiveTests : ArchiveTests } } - int count3 = 0; + var count3 = 0; foreach (var e in a.Entries) { count3++; @@ -530,55 +732,44 @@ public class ZipArchiveTests : ArchiveTests [Fact] public void Zip_Deflate_PKWear_Multipy_Entry_Access() { - string zipFile = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.pkware.zip"); + var zipFile = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.pkware.zip"); - using (FileStream fileStream = File.Open(zipFile, FileMode.Open)) - { - using ( - IArchive archive = ArchiveFactory.Open( - fileStream, - new ReaderOptions { Password = "12345678" } - ) - ) + using var fileStream = File.OpenRead(zipFile); + using var archive = ArchiveFactory.OpenArchive( + fileStream, + ReaderOptions.ForExternalStream with { - var entries = archive.Entries.Where(entry => !entry.IsDirectory); - foreach (IArchiveEntry entry in entries) - { - for (var i = 0; i < 100; i++) - { - using (var memoryStream = new MemoryStream()) - using (Stream entryStream = entry.OpenEntryStream()) - { - entryStream.CopyTo(memoryStream); - } - } - } + Password = "12345678", + } + ); + var entries = archive.Entries.Where(entry => !entry.IsDirectory); + foreach (var entry in entries) + { + for (var i = 0; i < 100; i++) + { + using var memoryStream = new MemoryStream(); + using var entryStream = entry.OpenEntryStream(); + entryStream.CopyTo(memoryStream); } } } - [SkippableFact] - public void Zip_Evil_Throws_Exception() +#if WINDOWS + [Fact] + public void Zip_Evil_Throws_Exception_Windows() { - //windows only because of the paths - Skip.IfNot(Environment.OSVersion.Platform == PlatformID.Win32NT); - - string zipFile = Path.Combine(TEST_ARCHIVES_PATH, "Zip.Evil.zip"); + var zipFile = Path.Combine(TEST_ARCHIVES_PATH, "Zip.Evil.zip"); Assert.ThrowsAny(() => { - using (var archive = ZipArchive.Open(zipFile)) + using var archive = ZipArchive.OpenArchive(zipFile); + foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) { - foreach (var entry in archive.Entries.Where(entry => !entry.IsDirectory)) - { - entry.WriteToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); - } + entry.WriteToDirectory(SCRATCH_FILES_PATH); } }); } +#endif private class NonSeekableMemoryStream : MemoryStream { @@ -591,7 +782,11 @@ public class ZipArchiveTests : ArchiveTests MemoryStream stream = new NonSeekableMemoryStream(); using ( - IWriter zipWriter = WriterFactory.Open(stream, ArchiveType.Zip, CompressionType.Deflate) + var zipWriter = WriterFactory.OpenWriter( + stream, + ArchiveType.Zip, + new ZipWriterOptions(CompressionType.Deflate) + ) ) { zipWriter.Write("foo.txt", new MemoryStream(Array.Empty())); @@ -601,20 +796,18 @@ public class ZipArchiveTests : ArchiveTests stream = new MemoryStream(stream.ToArray()); File.WriteAllBytes(Path.Combine(SCRATCH_FILES_PATH, "foo.zip"), stream.ToArray()); - using (var zipArchive = ZipArchive.Open(stream)) + using (var zipArchive = ZipArchive.OpenArchive(stream)) { foreach (var entry in zipArchive.Entries) { - using (var entryStream = entry.OpenEntryStream()) + using var entryStream = entry.OpenEntryStream(); + var tempStream = new MemoryStream(); + const int bufSize = 0x1000; + var buf = new byte[bufSize]; + var bytesRead = 0; + while ((bytesRead = entryStream.Read(buf, 0, bufSize)) > 0) { - MemoryStream tempStream = new MemoryStream(); - const int bufSize = 0x1000; - byte[] buf = new byte[bufSize]; - int bytesRead = 0; - while ((bytesRead = entryStream.Read(buf, 0, bufSize)) > 0) - { - tempStream.Write(buf, 0, bytesRead); - } + tempStream.Write(buf, 0, bytesRead); } } } @@ -623,119 +816,104 @@ public class ZipArchiveTests : ArchiveTests [Fact] public void Zip_BadLocalExtra_Read() { - string zipPath = Path.Combine(TEST_ARCHIVES_PATH, "Zip.badlocalextra.zip"); + var zipPath = Path.Combine(TEST_ARCHIVES_PATH, "Zip.badlocalextra.zip"); - using (ZipArchive za = ZipArchive.Open(zipPath)) + using var za = ZipArchive.OpenArchive(zipPath); + var ex = Record.Exception(() => { - var ex = Record.Exception(() => - { - var firstEntry = za.Entries.First(x => x.Key == "first.txt"); - var buffer = new byte[4096]; + var firstEntry = za.Entries.First(x => x.Key == "first.txt"); + var buffer = new byte[4096]; - using (var memoryStream = new MemoryStream()) - using (var firstStream = firstEntry.OpenEntryStream()) - { - firstStream.CopyTo(memoryStream); - Assert.Equal(199, memoryStream.Length); - } - }); + using var memoryStream = new MemoryStream(); + using var firstStream = firstEntry.OpenEntryStream(); + firstStream.CopyTo(memoryStream); + Assert.Equal(199, memoryStream.Length); + }); - Assert.Null(ex); - } + Assert.Null(ex); } [Fact] public void Zip_NoCompression_DataDescriptors_Read() { - string zipPath = Path.Combine(TEST_ARCHIVES_PATH, "Zip.none.datadescriptors.zip"); + var zipPath = Path.Combine(TEST_ARCHIVES_PATH, "Zip.none.datadescriptors.zip"); - using (ZipArchive za = ZipArchive.Open(zipPath)) + using var za = ZipArchive.OpenArchive(zipPath); + var firstEntry = za.Entries.First(x => x.Key == "first.txt"); + var buffer = new byte[4096]; + + using (var memoryStream = new MemoryStream()) + using (var firstStream = firstEntry.OpenEntryStream()) { - var firstEntry = za.Entries.First(x => x.Key == "first.txt"); - var buffer = new byte[4096]; - - using (var memoryStream = new MemoryStream()) - using (var firstStream = firstEntry.OpenEntryStream()) - { - firstStream.CopyTo(memoryStream); - Assert.Equal(199, memoryStream.Length); - } - - var len1 = 0; - var buffer1 = new byte[firstEntry.Size + 256]; - - using (var firstStream = firstEntry.OpenEntryStream()) - { - len1 = firstStream.Read(buffer1, 0, buffer.Length); - } - - Assert.Equal(199, len1); - -#if !NETFRAMEWORK && !NETSTANDARD2_0 - var len2 = 0; - var buffer2 = new byte[firstEntry.Size + 256]; - - using (var firstStream = firstEntry.OpenEntryStream()) - { - len2 = firstStream.Read(buffer2.AsSpan()); - } - Assert.Equal(len1, len2); - Assert.Equal(buffer1, buffer2); -#endif + firstStream.CopyTo(memoryStream); + Assert.Equal(199, memoryStream.Length); } + + var len1 = 0; + var buffer1 = new byte[firstEntry.Size + 256]; + + using (var firstStream = firstEntry.OpenEntryStream()) + { + len1 = firstStream.Read(buffer1, 0, buffer.Length); + } + + Assert.Equal(199, len1); + +#if !NETFRAMEWORK + var len2 = 0; + var buffer2 = new byte[firstEntry.Size + 256]; + + using (var firstStream = firstEntry.OpenEntryStream()) + { + len2 = firstStream.Read(buffer2.AsSpan()); + } + Assert.Equal(len1, len2); + Assert.Equal(buffer1, buffer2); +#endif } [Fact] public void Zip_LongComment_Read() { - string zipPath = Path.Combine(TEST_ARCHIVES_PATH, "Zip.LongComment.zip"); + var zipPath = Path.Combine(TEST_ARCHIVES_PATH, "Zip.LongComment.zip"); - using (ZipArchive za = ZipArchive.Open(zipPath)) - { - var count = za.Entries.Count; - Assert.Equal(1, count); - } + using var za = ZipArchive.OpenArchive(zipPath); + var count = za.Entries.Count(); + Assert.Equal(1, count); } [Fact] public void Zip_Zip64_CompressedSizeExtraOnly_Read() { - string zipPath = Path.Combine(TEST_ARCHIVES_PATH, "Zip.zip64.compressedonly.zip"); + var zipPath = Path.Combine(TEST_ARCHIVES_PATH, "Zip.zip64.compressedonly.zip"); - using (ZipArchive za = ZipArchive.Open(zipPath)) - { - var firstEntry = za.Entries.First(x => x.Key == "test/test.txt"); + using var za = ZipArchive.OpenArchive(zipPath); + var firstEntry = za.Entries.First(x => x.Key == "test/test.txt"); - using (var memoryStream = new MemoryStream()) - using (var firstStream = firstEntry.OpenEntryStream()) - { - firstStream.CopyTo(memoryStream); - Assert.Equal(15, memoryStream.Length); - } - } + using var memoryStream = new MemoryStream(); + using var firstStream = firstEntry.OpenEntryStream(); + firstStream.CopyTo(memoryStream); + Assert.Equal(15, memoryStream.Length); } [Fact] public void Zip_Uncompressed_Read_All() { - string zipPath = Path.Combine(TEST_ARCHIVES_PATH, "Zip.uncompressed.zip"); - using (var stream = File.Open(zipPath, FileMode.Open, FileAccess.Read)) + var zipPath = Path.Combine(TEST_ARCHIVES_PATH, "Zip.uncompressed.zip"); + using var stream = File.OpenRead(zipPath); + var reader = ReaderFactory.OpenReader(stream); + var entries = 0; + while (reader.MoveToNextEntry()) { - IArchive archive = ArchiveFactory.Open(stream); - IReader reader = archive.ExtractAllEntries(); - int entries = 0; - while (reader.MoveToNextEntry()) + using (var entryStream = reader.OpenEntryStream()) + using (var target = new MemoryStream()) { - using (var entryStream = reader.OpenEntryStream()) - using (var target = new MemoryStream()) - { - entryStream.CopyTo(target); - } - - entries++; + entryStream.CopyTo(target); } - Assert.Equal(4, entries); + + entries++; } + Assert.Equal(4, entries); } [Fact] @@ -747,59 +925,190 @@ public class ZipArchiveTests : ArchiveTests "Folder/File2.rtf", "Folder2/File1.txt", "Folder2/File2.txt", - "DEADBEEF" + "DEADBEEF", }; var zipPath = Path.Combine(TEST_ARCHIVES_PATH, "Zip.uncompressed.zip"); - using (var stream = File.Open(zipPath, FileMode.Open, FileAccess.Read)) + using var stream = File.OpenRead(zipPath); + var reader = ReaderFactory.OpenReader(stream); + var x = 0; + while (reader.MoveToNextEntry()) { - IArchive archive = ArchiveFactory.Open(stream); - IReader reader = archive.ExtractAllEntries(); - int x = 0; - while (reader.MoveToNextEntry()) - { - Assert.Equal(keys[x], reader.Entry.Key); - x++; - } - - Assert.Equal(4, x); + Assert.Equal(keys[x], reader.Entry.Key); + x++; } + + Assert.Equal(4, x); } [Fact] public void Zip_Forced_Ignores_UnicodePathExtra() { var zipPath = Path.Combine(TEST_ARCHIVES_PATH, "Zip.UnicodePathExtra.zip"); - using (var stream = File.Open(zipPath, FileMode.Open, FileAccess.Read)) + using (var stream = File.OpenRead(zipPath)) { - IArchive archive = ArchiveFactory.Open( + var reader = ReaderFactory.OpenReader( stream, - new ReaderOptions + ReaderOptions.ForExternalStream with { ArchiveEncoding = new ArchiveEncoding { Default = Encoding.GetEncoding("shift_jis"), - } + }, } ); - IReader reader = archive.ExtractAllEntries(); reader.MoveToNextEntry(); Assert.Equal("궖귛궖귙귪궖귗귪궖귙_wav.frq", reader.Entry.Key); } - using (var stream = File.Open(zipPath, FileMode.Open, FileAccess.Read)) + using (var stream = File.OpenRead(zipPath)) { - IArchive archive = ArchiveFactory.Open( + var reader = ReaderFactory.OpenReader( stream, - new ReaderOptions + ReaderOptions.ForExternalStream with { ArchiveEncoding = new ArchiveEncoding { Forced = Encoding.GetEncoding("shift_jis"), - } + }, } ); - IReader reader = archive.ExtractAllEntries(); reader.MoveToNextEntry(); Assert.Equal("きょきゅんきゃんきゅ_wav.frq", reader.Entry.Key); } } + + [Fact] + public void TestDataDescriptorRead() + { + using var archive = ArchiveFactory.OpenArchive( + Path.Combine(TEST_ARCHIVES_PATH, "Zip.none.datadescriptors.zip") + ); + var firstEntry = archive.Entries.First(); + Assert.Equal(199, firstEntry.Size); + using var _ = firstEntry.OpenEntryStream(); + Assert.Equal(199, firstEntry.Size); + } + + [Fact] + public void Zip_EntryCommentAfterEntryRead() + { + using var archive = ZipArchive.OpenArchive( + Path.Combine(TEST_ARCHIVES_PATH, "Zip.EntryComment.zip") + ); + var firstEntry = (ZipArchiveEntry)archive.Entries.First(); + Assert.Equal(29, firstEntry.Comment!.Length); + using var _ = firstEntry.OpenEntryStream(); + Assert.Equal(29, firstEntry.Comment.Length); + } + + [Fact] + public void Zip_FilePermissions() + { + using var archive = ArchiveFactory.OpenArchive( + Path.Combine(TEST_ARCHIVES_PATH, "Zip.644.zip") + ); + + var firstEntry = archive.Entries.First(); + const int S_IFREG = 0x8000; + const int expected = (S_IFREG | 0b110_100_100) << 16; // 0644 mode regular file + Assert.Equal(expected, firstEntry.Attrib); + } + + [Fact] + public void Zip_LZMA_ZeroSizeEntry_CanExtract() + { + using var archive = ArchiveFactory.OpenArchive( + Path.Combine(TEST_ARCHIVES_PATH, "Zip.lzma.empty.zip") + ); + var entries = archive.Entries.Where(x => !x.IsDirectory).ToList(); + Assert.Single(entries); + Assert.Equal(0, entries[0].Size); + var outStream = new MemoryStream(); + entries[0].WriteTo(outStream); + Assert.Equal(0, outStream.Length); + } + + private static void VerifyMixedZstandardArchive(IArchive archive) + { + var entries = archive.Entries.Where(x => !x.IsDirectory).ToArray(); + Assert.Equal(4, entries.Length); + Assert.Equal(2, entries.Count(x => x.IsEncrypted)); + Assert.Equal( + [".signature", "encrypted-zstd-entry.bin", "plain-zstd-entry.bin", "tables.db"], + entries.Select(x => x.Key.NotNull()).OrderBy(x => x).ToArray() + ); + Assert.All( + entries, + entry => Assert.Equal(CompressionType.ZStandard, entry.CompressionType) + ); + + var expectedSizes = new long[] { 160, 64 * 1024, 64 * 1024, 192 * 1024 }; + Assert.Equal(expectedSizes, entries.Select(x => x.Size).OrderBy(x => x).ToArray()); + + foreach (var entry in entries) + { + using var entryStream = entry.OpenEntryStream(); + using var target = new MemoryStream(); + entryStream.CopyTo(target); + Assert.Equal(entry.Size, target.Length); + } + } + + private static long CountBytes(Stream stream) + { + var buffer = new byte[8 * 1024 * 1024]; + long total = 0; + int read; + while ((read = stream.Read(buffer, 0, buffer.Length)) > 0) + { + total += read; + } + + return total; + } + + private static void CreateZip64Archive(string path) + { + if (File.Exists(path)) + { + File.Delete(path); + } + + var writerOptions = new ZipWriterOptions(CompressionType.None) { UseZip64 = true }; + using var fileStream = File.OpenWrite(path); + using var zipWriter = (ZipWriter) + WriterFactory.OpenWriter(fileStream, ArchiveType.Zip, writerOptions); + + using ( + var largeEntryStream = zipWriter.WriteToStream( + "large-entry.bin", + new ZipWriterEntryOptions() + ) + ) + { + WriteZeroes(largeEntryStream, GeneratedZip64EntrySize); + } + + using ( + var trailingEntryStream = zipWriter.WriteToStream( + "trailing-entry.bin", + new ZipWriterEntryOptions() + ) + ) + { + trailingEntryStream.WriteByte(0x42); + } + } + + private static void WriteZeroes(Stream stream, long length) + { + byte[] buffer = new byte[8 * 1024 * 1024]; + long remaining = length; + + while (remaining > 0) + { + int chunk = (int)Math.Min(buffer.Length, remaining); + stream.Write(buffer, 0, chunk); + remaining -= chunk; + } + } } diff --git a/tests/SharpCompress.Test/Zip/ZipCrcExtractionTests.cs b/tests/SharpCompress.Test/Zip/ZipCrcExtractionTests.cs new file mode 100644 index 00000000..cd0e610c --- /dev/null +++ b/tests/SharpCompress.Test/Zip/ZipCrcExtractionTests.cs @@ -0,0 +1,260 @@ +using System; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Archives; +using SharpCompress.Archives.Zip; +using SharpCompress.Common; +using SharpCompress.Readers; +using SharpCompress.Writers; +using SharpCompress.Writers.Zip; +using Xunit; + +namespace SharpCompress.Test.Zip; + +public class ZipCrcExtractionTests : ArchiveTests +{ + private const string EntryName = "crc.txt"; + private static readonly byte[] EntryData = Encoding.UTF8.GetBytes("crc validation payload"); + + [Fact] + public void Zip_Archive_WriteToFile_Throws_On_Crc_Mismatch() + { + using var zipStream = CreateZipWithInvalidCrc(useDataDescriptor: false); + using var archive = ZipArchive.OpenArchive(zipStream); + var entry = archive.Entries.Single(e => !e.IsDirectory); + var destination = Path.Combine(SCRATCH_FILES_PATH, "zip-crc-mismatch.txt"); + + var exception = Assert.Throws(() => entry.WriteToFile(destination)); + + Assert.Contains(EntryName, exception.Message); + } + + [Fact] + public void Zip_Archive_WriteToFile_Skips_Crc_Mismatch_When_Disabled() + { + using var zipStream = CreateZipWithInvalidCrc(useDataDescriptor: false); + using var archive = ZipArchive.OpenArchive(zipStream); + var entry = archive.Entries.Single(e => !e.IsDirectory); + var destination = Path.Combine(SCRATCH_FILES_PATH, "zip-crc-disabled.txt"); + + entry.WriteToFile(destination, new ExtractionOptions { CheckCrc = false }); + + Assert.Equal(EntryData, File.ReadAllBytes(destination)); + } + + [Fact] + public void Zip_Archive_WriteTo_Throws_On_Crc_Mismatch_When_Enabled() + { + using var zipStream = CreateZipWithInvalidCrc(useDataDescriptor: false); + using var archive = ZipArchive.OpenArchive(zipStream); + var entry = archive.Entries.Single(e => !e.IsDirectory); + using var destination = new MemoryStream(); + + var exception = Assert.Throws(() => + entry.WriteTo(destination, new ExtractionOptions { CheckCrc = true }) + ); + + Assert.Contains(EntryName, exception.Message); + } + + [Fact] + public void Zip_Archive_WriteTo_Skips_Crc_Mismatch_When_Disabled() + { + using var zipStream = CreateZipWithInvalidCrc(useDataDescriptor: false); + using var archive = ZipArchive.OpenArchive(zipStream); + var entry = archive.Entries.Single(e => !e.IsDirectory); + using var destination = new MemoryStream(); + + entry.WriteTo(destination, new ExtractionOptions { CheckCrc = false }); + + Assert.Equal(EntryData, destination.ToArray()); + } + + [Fact] + public void Zip_Reader_WriteEntryToFile_Throws_On_Crc_Mismatch() + { + using var zipStream = CreateZipWithInvalidCrc(useDataDescriptor: false); + using var reader = ReaderFactory.OpenReader(zipStream); + var destination = Path.Combine(SCRATCH_FILES_PATH, "zip-reader-crc-mismatch.txt"); + + Assert.True(reader.MoveToNextEntry()); + var exception = Assert.Throws(() => + reader.WriteEntryToFile(destination) + ); + + Assert.Contains(EntryName, exception.Message); + } + + [Fact] + public void Zip_Reader_WriteEntryToFile_Skips_Crc_Mismatch_When_Disabled() + { + using var zipStream = CreateZipWithInvalidCrc(useDataDescriptor: false); + using var reader = ReaderFactory.OpenReader(zipStream); + var destination = Path.Combine(SCRATCH_FILES_PATH, "zip-reader-crc-disabled.txt"); + + Assert.True(reader.MoveToNextEntry()); + reader.WriteEntryToFile(destination, new ExtractionOptions { CheckCrc = false }); + + Assert.Equal(EntryData, File.ReadAllBytes(destination)); + } + + [Fact] + public async Task Zip_Archive_WriteToFileAsync_Throws_On_Crc_Mismatch() + { + using var zipStream = CreateZipWithInvalidCrc(useDataDescriptor: false); + using var archive = ZipArchive.OpenArchive(zipStream); + var entry = archive.Entries.Single(e => !e.IsDirectory); + var destination = Path.Combine(SCRATCH_FILES_PATH, "zip-crc-mismatch-async.txt"); + + var exception = await Assert.ThrowsAsync(async () => + await entry.WriteToFileAsync(destination) + ); + + Assert.Contains(EntryName, exception.Message); + } + + [Fact] + public async Task Zip_Archive_WriteToAsync_Throws_On_Crc_Mismatch_When_Enabled() + { + using var zipStream = CreateZipWithInvalidCrc(useDataDescriptor: false); + using var archive = ZipArchive.OpenArchive(zipStream); + var entry = archive.Entries.Single(e => !e.IsDirectory); +#if LEGACY_DOTNET + using var destination = new MemoryStream(); +#else + await using var destination = new MemoryStream(); +#endif + + var exception = await Assert.ThrowsAsync(async () => + await entry.WriteToAsync(destination, new ExtractionOptions { CheckCrc = true }) + ); + + Assert.Contains(EntryName, exception.Message); + } + + [Fact] + public async Task Zip_Archive_WriteToAsync_Skips_Crc_Mismatch_When_Disabled() + { + using var zipStream = CreateZipWithInvalidCrc(useDataDescriptor: false); + using var archive = ZipArchive.OpenArchive(zipStream); + var entry = archive.Entries.Single(e => !e.IsDirectory); +#if LEGACY_DOTNET + using var destination = new MemoryStream(); +#else + await using var destination = new MemoryStream(); +#endif + + await entry.WriteToAsync(destination, new ExtractionOptions { CheckCrc = false }); + + Assert.Equal(EntryData, destination.ToArray()); + } + + [Fact] + public async Task Zip_Reader_WriteEntryToFileAsync_Throws_On_Crc_Mismatch() + { + using var zipStream = CreateZipWithInvalidCrc(useDataDescriptor: false); + await using var reader = await ReaderFactory.OpenAsyncReader(zipStream); + var destination = Path.Combine(SCRATCH_FILES_PATH, "zip-reader-crc-mismatch-async.txt"); + + Assert.True(await reader.MoveToNextEntryAsync()); + var exception = await Assert.ThrowsAsync(async () => + await reader.WriteEntryToFileAsync(destination) + ); + + Assert.Contains(EntryName, exception.Message); + } + + [Fact] + public void Zip_Archive_WriteToFile_Throws_On_DataDescriptor_Crc_Mismatch() + { + using var zipStream = CreateZipWithInvalidCrc(useDataDescriptor: true); + using var archive = ZipArchive.OpenArchive(zipStream); + var entry = archive.Entries.Single(e => !e.IsDirectory); + var destination = Path.Combine(SCRATCH_FILES_PATH, "zip-dd-crc-mismatch.txt"); + + var exception = Assert.Throws(() => entry.WriteToFile(destination)); + + Assert.Contains(EntryName, exception.Message); + } + + private static MemoryStream CreateZipWithInvalidCrc(bool useDataDescriptor) + { + var zipStream = new MemoryStream(); + Stream writerStream = useDataDescriptor ? new NonSeekableWriteStream(zipStream) : zipStream; + using ( + var writer = WriterFactory.OpenWriter( + writerStream, + ArchiveType.Zip, + new ZipWriterOptions(CompressionType.Deflate) { LeaveStreamOpen = true } + ) + ) + { + writer.Write(EntryName, new MemoryStream(EntryData)); + } + + var bytes = zipStream.ToArray(); + CorruptCrc(bytes, ZipHeaderFactoryEntrySignature, 14); + CorruptCrc(bytes, ZipHeaderFactoryDirectorySignature, 16); + return new MemoryStream(bytes); + } + + private const uint ZipHeaderFactoryEntrySignature = 0x04034b50; + private const uint ZipHeaderFactoryDirectorySignature = 0x02014b50; + + private static void CorruptCrc(byte[] bytes, uint signature, int crcOffset) + { + var offset = FindSignature(bytes, signature); + var crcIndex = offset + crcOffset; + bytes[crcIndex] ^= 0xFF; + } + + private static int FindSignature(byte[] bytes, uint signature) + { + var signatureBytes = BitConverter.GetBytes(signature); + for (var i = 0; i <= bytes.Length - signatureBytes.Length; i++) + { + if (bytes.AsSpan(i, signatureBytes.Length).SequenceEqual(signatureBytes)) + { + return i; + } + } + + throw new InvalidOperationException($"ZIP signature 0x{signature:X8} was not found."); + } + + private sealed class NonSeekableWriteStream(Stream stream) : Stream + { + public override bool CanRead => false; + public override bool CanSeek => false; + public override bool CanWrite => true; + public override long Length => throw new NotSupportedException(); + public override long Position + { + get => throw new NotSupportedException(); + set => throw new NotSupportedException(); + } + + public override void Flush() => stream.Flush(); + + public override Task FlushAsync(CancellationToken cancellationToken) => + stream.FlushAsync(cancellationToken); + + public override int Read(byte[] buffer, int offset, int count) => + throw new NotSupportedException(); + + public override long Seek(long offset, SeekOrigin origin) => + throw new NotSupportedException(); + + public override void SetLength(long value) => throw new NotSupportedException(); + + public override void Write(byte[] buffer, int offset, int count) => + stream.Write(buffer, offset, count); + +#if !LEGACY_DOTNET + public override void Write(ReadOnlySpan buffer) => stream.Write(buffer); +#endif + } +} diff --git a/tests/SharpCompress.Test/Zip/ZipFilePartTests.cs b/tests/SharpCompress.Test/Zip/ZipFilePartTests.cs new file mode 100644 index 00000000..a299cfbf --- /dev/null +++ b/tests/SharpCompress.Test/Zip/ZipFilePartTests.cs @@ -0,0 +1,59 @@ +using System.IO; +using SharpCompress.Common; +using SharpCompress.Common.Zip; +using SharpCompress.Common.Zip.Headers; +using SharpCompress.IO; +using SharpCompress.Providers; +using Xunit; + +namespace SharpCompress.Test.Zip; + +public class ZipFilePartTests +{ + [Fact] + public void GetCryptoStream_Bounds_Known_Size_Zip64_Entries() + { + var header = new DirectoryEntryHeader(new ArchiveEncoding()) + { + Name = "entry.bin", + CompressionMethod = ZipCompressionMethod.None, + CompressedSize = uint.MaxValue, + UncompressedSize = uint.MaxValue, + }; + + using var backingStream = new MemoryStream([1, 2, 3, 4, 5], writable: false); + var part = new TestZipFilePart(header, backingStream); + + using var cryptoStream = part.OpenCryptoStream(); + + Assert.IsType(cryptoStream); + } + + [Fact] + public void GetCryptoStream_Leaves_DataDescriptor_Entries_Unbounded_When_Size_Is_Unknown() + { + var header = new DirectoryEntryHeader(new ArchiveEncoding()) + { + Name = "entry.bin", + CompressionMethod = ZipCompressionMethod.None, + CompressedSize = 0, + UncompressedSize = 0, + Flags = HeaderFlags.UsePostDataDescriptor, + }; + + using var backingStream = new MemoryStream([1, 2, 3, 4, 5], writable: false); + var part = new TestZipFilePart(header, backingStream); + + using var cryptoStream = part.OpenCryptoStream(); + + Assert.IsNotType(cryptoStream); + } + + private sealed class TestZipFilePart(ZipFileEntry header, Stream stream) + : ZipFilePart(header, stream, CompressionProviderRegistry.Default) + { + public Stream OpenCryptoStream() => GetCryptoStream(CreateBaseStream()); + + protected override Stream CreateBaseStream() => BaseStream; + } +} diff --git a/tests/SharpCompress.Test/Zip/ZipMemoryArchiveWithCrcAsyncTests.cs b/tests/SharpCompress.Test/Zip/ZipMemoryArchiveWithCrcAsyncTests.cs new file mode 100644 index 00000000..478aaafe --- /dev/null +++ b/tests/SharpCompress.Test/Zip/ZipMemoryArchiveWithCrcAsyncTests.cs @@ -0,0 +1,296 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading.Tasks; +using SharpCompress.Archives.Zip; +using SharpCompress.Common; +using SharpCompress.Compressors.Deflate; +using SharpCompress.Compressors.Xz; +using SharpCompress.Crypto; +using SharpCompress.Test.Mocks; +using SharpCompress.Writers; +using SharpCompress.Writers.Zip; +using Xunit; + +namespace SharpCompress.Test.Zip; + +public class ZipTypesLevelsWithCrcRatioAsyncTests : ArchiveTests +{ + public ZipTypesLevelsWithCrcRatioAsyncTests() => UseExtensionInsteadOfNameToVerify = true; + + [Theory] + [InlineData(CompressionType.Deflate, 1, 1, 0.11f)] // was 0.8f, actual 0.104 + [InlineData(CompressionType.Deflate, 3, 1, 0.08f)] // was 0.8f, actual 0.078 + [InlineData(CompressionType.Deflate, 6, 1, 0.05f)] // was 0.8f, actual ~0.042 + [InlineData(CompressionType.Deflate, 9, 1, 0.04f)] // was 0.7f, actual 0.038 + [InlineData(CompressionType.ZStandard, 1, 1, 0.025f)] // was 0.8f, actual 0.023 + [InlineData(CompressionType.ZStandard, 3, 1, 0.015f)] // was 0.7f, actual 0.013 + [InlineData(CompressionType.ZStandard, 9, 1, 0.006f)] // was 0.7f, actual 0.005 + [InlineData(CompressionType.ZStandard, 22, 1, 0.005f)] // was 0.7f, actual 0.004 + [InlineData(CompressionType.BZip2, 0, 1, 0.035f)] // was 0.8f, actual 0.033 + [InlineData(CompressionType.LZMA, 0, 1, 0.005f)] // was 0.8f, actual 0.004 + [InlineData(CompressionType.None, 0, 1, 1.001f)] // was 1.1f, actual 1.000 + [InlineData(CompressionType.Deflate, 6, 2, 0.045f)] // was 0.8f, actual 0.042 + [InlineData(CompressionType.ZStandard, 3, 2, 0.012f)] // was 0.7f, actual 0.010 + [InlineData(CompressionType.BZip2, 0, 2, 0.035f)] // was 0.8f, actual 0.032 + [InlineData(CompressionType.Deflate, 9, 3, 0.04f)] // was 0.7f, actual 0.038 + [InlineData(CompressionType.ZStandard, 9, 3, 0.003f)] // was 0.7f, actual 0.002 + public async ValueTask Zip_Create_Archive_With_3_Files_Crc32_Test_Async( + CompressionType compressionType, + int compressionLevel, + int sizeMb, + float expectedRatio + ) + { + const int OneMiB = 1024 * 1024; + var baseSize = sizeMb * OneMiB; + + // Generate test content for files with sizes based on the sizeMb parameter + var file1Data = TestPseudoTextStream.Create(baseSize); + var file2Data = TestPseudoTextStream.Create(baseSize * 2); + var file3Data = TestPseudoTextStream.Create(baseSize * 3); + + var expectedFiles = new Dictionary + { + [$"file1_{sizeMb}MiB.txt"] = (file1Data, CalculateCrc32(file1Data)), + [$"data/file2_{sizeMb * 2}MiB.txt"] = (file2Data, CalculateCrc32(file2Data)), + [$"deep/nested/file3_{sizeMb * 3}MiB.txt"] = (file3Data, CalculateCrc32(file3Data)), + }; + + // Create zip archive in memory + using var zipStream = new MemoryStream(); + await using ( + var writer = await CreateWriterWithLevelAsync( + zipStream, + compressionType, + compressionLevel + ) + ) + { + await writer.WriteAsync($"file1_{sizeMb}MiB.txt", new MemoryStream(file1Data)); + await writer.WriteAsync($"data/file2_{sizeMb * 2}MiB.txt", new MemoryStream(file2Data)); + await writer.WriteAsync( + $"deep/nested/file3_{sizeMb * 3}MiB.txt", + new MemoryStream(file3Data) + ); + } + + // Calculate and output actual compression ratio + var originalSize = file1Data.Length + file2Data.Length + file3Data.Length; + var actualRatio = (double)zipStream.Length / originalSize; + + // Verify compression occurred (except for None compression type) + if (compressionType != CompressionType.None) + { + Assert.True( + zipStream.Length < originalSize, + $"Compression failed: compressed={zipStream.Length}, original={originalSize}" + ); + } + + // Verify compression ratio + VerifyCompressionRatio( + originalSize, + zipStream.Length, + expectedRatio, + $"{compressionType} level {compressionLevel}" + ); + + // Verify archive content and CRC32 + await VerifyArchiveContentAsync(zipStream, expectedFiles); + + // Verify compression type is correctly set + VerifyCompressionType(zipStream, compressionType); + } + + [Theory] + [InlineData(CompressionType.Deflate, 1, 4, 0.11f)] // was 0.8, actual 0.105 + [InlineData(CompressionType.Deflate, 3, 4, 0.08f)] // was 0.8, actual 0.077 + [InlineData(CompressionType.Deflate, 6, 4, 0.045f)] // was 0.8, actual 0.042 + [InlineData(CompressionType.Deflate, 9, 4, 0.04f)] // was 0.8, actual 0.037 + [InlineData(CompressionType.ZStandard, 1, 4, 0.025f)] // was 0.8, actual 0.022 + [InlineData(CompressionType.ZStandard, 3, 4, 0.012f)] // was 0.8, actual 0.010 + [InlineData(CompressionType.ZStandard, 9, 4, 0.003f)] // was 0.8, actual 0.002 + [InlineData(CompressionType.ZStandard, 22, 4, 0.003f)] // was 0.8, actual 0.002 + [InlineData(CompressionType.BZip2, 0, 4, 0.035f)] // was 0.8, actual 0.032 + [InlineData(CompressionType.LZMA, 0, 4, 0.003f)] // was 0.8, actual 0.002 + public async ValueTask Zip_WriterFactory_Crc32_Test_Async( + CompressionType compressionType, + int compressionLevel, + int sizeMb, + float expectedRatio + ) + { + var fileSize = sizeMb * 1024 * 1024; + + var testData = TestPseudoTextStream.Create(fileSize); + var expectedCrc = CalculateCrc32(testData); + + // Create archive with specified compression level + using var zipStream = new MemoryStream(); + var writerOptions = new ZipWriterOptions(compressionType) + { + CompressionLevel = compressionLevel, + }; + + await using ( + var writer = await WriterFactory.OpenAsyncWriter( + new AsyncOnlyStream(zipStream), + ArchiveType.Zip, + writerOptions + ) + ) + { + await writer.WriteAsync( + $"{compressionType}_level_{compressionLevel}_{sizeMb}MiB.txt", + new MemoryStream(testData) + ); + } + + // Calculate and output actual compression ratio + var actualRatio = (double)zipStream.Length / testData.Length; + + VerifyCompressionRatio( + testData.Length, + zipStream.Length, + expectedRatio, + $"{compressionType} level {compressionLevel}" + ); + + // Verify the archive + zipStream.Position = 0; + using var archive = ZipArchive.OpenArchive(zipStream); + + var entry = archive.Entries.Single(e => !e.IsDirectory); + using var entryStream = await entry.OpenEntryStreamAsync(); + using var extractedStream = new MemoryStream(); + await entryStream.CopyToAsync(extractedStream); + + var extractedData = extractedStream.ToArray(); + var actualCrc = CalculateCrc32(extractedData); + + Assert.Equal(compressionType, entry.CompressionType); + Assert.Equal(expectedCrc, actualCrc); + Assert.Equal(testData.Length, extractedData.Length); + Assert.Equal(testData, extractedData); + } + + [Theory] + [InlineData(CompressionType.Deflate, 1, 2, 0.11f)] // was 0.8, actual 0.104 + [InlineData(CompressionType.Deflate, 3, 2, 0.08f)] // was 0.8, actual 0.077 + [InlineData(CompressionType.Deflate, 6, 2, 0.045f)] // was 0.8, actual 0.042 + [InlineData(CompressionType.Deflate, 9, 2, 0.04f)] // was 0.7, actual 0.038 + [InlineData(CompressionType.ZStandard, 1, 2, 0.025f)] // was 0.8, actual 0.023 + [InlineData(CompressionType.ZStandard, 3, 2, 0.015f)] // was 0.7, actual 0.012 + [InlineData(CompressionType.ZStandard, 9, 2, 0.006f)] // was 0.7, actual 0.005 + [InlineData(CompressionType.ZStandard, 22, 2, 0.005f)] // was 0.7, actual 0.004 + [InlineData(CompressionType.BZip2, 0, 2, 0.035f)] // was 0.8, actual 0.032 + [InlineData(CompressionType.LZMA, 0, 2, 0.005f)] // was 0.8, actual 0.004 + public async ValueTask Zip_ZipArchiveOpen_Crc32_Test_Async( + CompressionType compressionType, + int compressionLevel, + int sizeMb, + float expectedRatio + ) + { + var fileSize = sizeMb * 1024 * 1024; + + var testData = TestPseudoTextStream.Create(fileSize); + var expectedCrc = CalculateCrc32(testData); + + // Create archive with specified compression and level + using var zipStream = new MemoryStream(); + await using ( + var writer = await CreateWriterWithLevelAsync( + zipStream, + compressionType, + compressionLevel + ) + ) + { + await writer.WriteAsync( + $"{compressionType}_{compressionLevel}_{sizeMb}MiB.txt", + new MemoryStream(testData) + ); + } + + // Calculate and output actual compression ratio + var actualRatio = (double)zipStream.Length / testData.Length; + + // Verify the archive + zipStream.Position = 0; + using var archive = ZipArchive.OpenArchive(zipStream); + + var entry = archive.Entries.Single(e => !e.IsDirectory); + using var entryStream = await entry.OpenEntryStreamAsync(); + using var extractedStream = new MemoryStream(); + await entryStream.CopyToAsync(extractedStream); + + var extractedData = extractedStream.ToArray(); + var actualCrc = CalculateCrc32(extractedData); + + Assert.Equal(compressionType, entry.CompressionType); + Assert.Equal(expectedCrc, actualCrc); + Assert.Equal(testData.Length, extractedData.Length); + + // For smaller files, verify full content; for larger, spot check + if (testData.Length <= sizeMb * 2) + { + Assert.Equal(testData, extractedData); + } + else + { + VerifyDataSpotCheck(testData, extractedData); + } + + VerifyCompressionRatio( + testData.Length, + zipStream.Length, + expectedRatio, + $"{compressionType} Level {compressionLevel}" + ); + } + + // Helper method for async archive content verification + private async ValueTask VerifyArchiveContentAsync( + MemoryStream zipStream, + Dictionary expectedFiles + ) + { + zipStream.Position = 0; + using var archive = ZipArchive.OpenArchive(zipStream); + + foreach (var entry in archive.Entries.Where(e => !e.IsDirectory)) + { + Assert.True( + expectedFiles.ContainsKey(entry.Key!), + $"Unexpected file in archive: {entry.Key}" + ); + + var expected = expectedFiles[entry.Key!]; + using var entryStream = await entry.OpenEntryStreamAsync(); + using var extractedStream = new MemoryStream(); + await entryStream.CopyToAsync(extractedStream); + + var extractedData = extractedStream.ToArray(); + var actualCrc = CalculateCrc32(extractedData); + + Assert.Equal(expected.crc, actualCrc); + Assert.Equal(expected.data.Length, extractedData.Length); + + // For larger files, just spot check, for smaller verify full content + var expectedData = expected.data; + if (expectedData.Length <= 2 * 1024 * 1024) + { + Assert.Equal(expectedData, extractedData); + } + else + { + VerifyDataSpotCheck(expectedData, extractedData); + } + } + + Assert.Equal(expectedFiles.Count, archive.Entries.Count(e => !e.IsDirectory)); + } +} diff --git a/tests/SharpCompress.Test/Zip/ZipMemoryArchiveWithCrcTests.cs b/tests/SharpCompress.Test/Zip/ZipMemoryArchiveWithCrcTests.cs new file mode 100644 index 00000000..03ce4e1e --- /dev/null +++ b/tests/SharpCompress.Test/Zip/ZipMemoryArchiveWithCrcTests.cs @@ -0,0 +1,231 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using SharpCompress.Archives.Zip; +using SharpCompress.Common; +using SharpCompress.Compressors.Deflate; +using SharpCompress.Compressors.Xz; +using SharpCompress.Crypto; +using SharpCompress.Writers; +using SharpCompress.Writers.Zip; +using Xunit; + +namespace SharpCompress.Test.Zip; + +public class ZipTypesLevelsWithCrcRatioTests : ArchiveTests +{ + public ZipTypesLevelsWithCrcRatioTests() => UseExtensionInsteadOfNameToVerify = true; + + [Theory] + [InlineData(CompressionType.Deflate, 1, 1, 0.11f)] // was 0.8f, actual 0.104 + [InlineData(CompressionType.Deflate, 3, 1, 0.08f)] // was 0.8f, actual 0.078 + [InlineData(CompressionType.Deflate, 6, 1, 0.05f)] // was 0.8f, actual ~0.042 + [InlineData(CompressionType.Deflate, 9, 1, 0.04f)] // was 0.7f, actual 0.038 + [InlineData(CompressionType.ZStandard, 1, 1, 0.025f)] // was 0.8f, actual 0.023 + [InlineData(CompressionType.ZStandard, 3, 1, 0.015f)] // was 0.7f, actual 0.013 + [InlineData(CompressionType.ZStandard, 9, 1, 0.006f)] // was 0.7f, actual 0.005 + [InlineData(CompressionType.ZStandard, 22, 1, 0.005f)] // was 0.7f, actual 0.004 + [InlineData(CompressionType.BZip2, 0, 1, 0.035f)] // was 0.8f, actual 0.033 + [InlineData(CompressionType.LZMA, 0, 1, 0.005f)] // was 0.8f, actual 0.004 + [InlineData(CompressionType.None, 0, 1, 1.001f)] // was 1.1f, actual 1.000 + [InlineData(CompressionType.Deflate, 6, 2, 0.045f)] // was 0.8f, actual 0.042 + [InlineData(CompressionType.ZStandard, 3, 2, 0.012f)] // was 0.7f, actual 0.010 + [InlineData(CompressionType.BZip2, 0, 2, 0.035f)] // was 0.8f, actual 0.032 + [InlineData(CompressionType.Deflate, 9, 3, 0.04f)] // was 0.7f, actual 0.038 + [InlineData(CompressionType.ZStandard, 9, 3, 0.003f)] // was 0.7f, actual 0.002 + public void Zip_Create_Archive_With_3_Files_Crc32_Test( + CompressionType compressionType, + int compressionLevel, + int sizeMb, + float expectedRatio + ) + { + const int OneMiB = 1024 * 1024; + var baseSize = sizeMb * OneMiB; + + // Generate test content for files with sizes based on the sizeMb parameter + var file1Data = TestPseudoTextStream.Create(baseSize); + var file2Data = TestPseudoTextStream.Create(baseSize * 2); + var file3Data = TestPseudoTextStream.Create(baseSize * 3); + + var expectedFiles = new Dictionary + { + [$"file1_{sizeMb}MiB.txt"] = (file1Data, CalculateCrc32(file1Data)), + [$"data/file2_{sizeMb * 2}MiB.txt"] = (file2Data, CalculateCrc32(file2Data)), + [$"deep/nested/file3_{sizeMb * 3}MiB.txt"] = (file3Data, CalculateCrc32(file3Data)), + }; + + // Create zip archive in memory + using var zipStream = new MemoryStream(); + using (var writer = CreateWriterWithLevel(zipStream, compressionType, compressionLevel)) + { + writer.Write($"file1_{sizeMb}MiB.txt", new MemoryStream(file1Data)); + writer.Write($"data/file2_{sizeMb * 2}MiB.txt", new MemoryStream(file2Data)); + writer.Write($"deep/nested/file3_{sizeMb * 3}MiB.txt", new MemoryStream(file3Data)); + } + + // Calculate and output actual compression ratio + var originalSize = file1Data.Length + file2Data.Length + file3Data.Length; + var actualRatio = (double)zipStream.Length / originalSize; + + // Verify compression occurred (except for None compression type) + if (compressionType != CompressionType.None) + { + Assert.True( + zipStream.Length < originalSize, + $"Compression failed: compressed={zipStream.Length}, original={originalSize}" + ); + } + + // Verify compression ratio + VerifyCompressionRatio( + originalSize, + zipStream.Length, + expectedRatio, + $"{compressionType} level {compressionLevel}" + ); + + // Verify archive content and CRC32 + VerifyArchiveContent(zipStream, expectedFiles); + + // Verify compression type is correctly set + VerifyCompressionType(zipStream, compressionType); + } + + [Theory] + [InlineData(CompressionType.Deflate, 1, 4, 0.11f)] // was 0.8, actual 0.105 + [InlineData(CompressionType.Deflate, 3, 4, 0.08f)] // was 0.8, actual 0.077 + [InlineData(CompressionType.Deflate, 6, 4, 0.045f)] // was 0.8, actual 0.042 + [InlineData(CompressionType.Deflate, 9, 4, 0.04f)] // was 0.8, actual 0.037 + [InlineData(CompressionType.ZStandard, 1, 4, 0.025f)] // was 0.8, actual 0.022 + [InlineData(CompressionType.ZStandard, 3, 4, 0.012f)] // was 0.8, actual 0.010 + [InlineData(CompressionType.ZStandard, 9, 4, 0.003f)] // was 0.8, actual 0.002 + [InlineData(CompressionType.ZStandard, 22, 4, 0.003f)] // was 0.8, actual 0.002 + [InlineData(CompressionType.BZip2, 0, 4, 0.035f)] // was 0.8, actual 0.032 + [InlineData(CompressionType.LZMA, 0, 4, 0.003f)] // was 0.8, actual 0.002 + public void Zip_WriterFactory_Crc32_Test( + CompressionType compressionType, + int compressionLevel, + int sizeMb, + float expectedRatio + ) + { + var fileSize = sizeMb * 1024 * 1024; + + var testData = TestPseudoTextStream.Create(fileSize); + var expectedCrc = CalculateCrc32(testData); + + // Create archive with specified compression level + using var zipStream = new MemoryStream(); + var writerOptions = new ZipWriterOptions(compressionType) + { + CompressionLevel = compressionLevel, + }; + + using (var writer = WriterFactory.OpenWriter(zipStream, ArchiveType.Zip, writerOptions)) + { + writer.Write( + $"{compressionType}_level_{compressionLevel}_{sizeMb}MiB.txt", + new MemoryStream(testData) + ); + } + + // Calculate and output actual compression ratio + var actualRatio = (double)zipStream.Length / testData.Length; + + VerifyCompressionRatio( + testData.Length, + zipStream.Length, + expectedRatio, + $"{compressionType} level {compressionLevel}" + ); + + // Verify the archive + zipStream.Position = 0; + using var archive = ZipArchive.OpenArchive(zipStream); + + var entry = archive.Entries.Single(e => !e.IsDirectory); + using var entryStream = entry.OpenEntryStream(); + using var extractedStream = new MemoryStream(); + entryStream.CopyTo(extractedStream); + + var extractedData = extractedStream.ToArray(); + var actualCrc = CalculateCrc32(extractedData); + + Assert.Equal(compressionType, entry.CompressionType); + Assert.Equal(expectedCrc, actualCrc); + Assert.Equal(testData.Length, extractedData.Length); + Assert.Equal(testData, extractedData); + } + + [Theory] + [InlineData(CompressionType.Deflate, 1, 2, 0.11f)] // was 0.8, actual 0.104 + [InlineData(CompressionType.Deflate, 3, 2, 0.08f)] // was 0.8, actual 0.077 + [InlineData(CompressionType.Deflate, 6, 2, 0.045f)] // was 0.8, actual 0.042 + [InlineData(CompressionType.Deflate, 9, 2, 0.04f)] // was 0.7, actual 0.038 + [InlineData(CompressionType.ZStandard, 1, 2, 0.025f)] // was 0.8, actual 0.023 + [InlineData(CompressionType.ZStandard, 3, 2, 0.015f)] // was 0.7, actual 0.012 + [InlineData(CompressionType.ZStandard, 9, 2, 0.006f)] // was 0.7, actual 0.005 + [InlineData(CompressionType.ZStandard, 22, 2, 0.005f)] // was 0.7, actual 0.004 + [InlineData(CompressionType.BZip2, 0, 2, 0.035f)] // was 0.8, actual 0.032 + [InlineData(CompressionType.LZMA, 0, 2, 0.005f)] // was 0.8, actual 0.004 + public void Zip_ZipArchiveOpen_Crc32_Test( + CompressionType compressionType, + int compressionLevel, + int sizeMb, + float expectedRatio + ) + { + var fileSize = sizeMb * 1024 * 1024; + + var testData = TestPseudoTextStream.Create(fileSize); + var expectedCrc = CalculateCrc32(testData); + + // Create archive with specified compression and level + using var zipStream = new MemoryStream(); + using (var writer = CreateWriterWithLevel(zipStream, compressionType, compressionLevel)) + { + writer.Write( + $"{compressionType}_{compressionLevel}_{sizeMb}MiB.txt", + new MemoryStream(testData) + ); + } + + // Calculate and output actual compression ratio + var actualRatio = (double)zipStream.Length / testData.Length; + + // Verify the archive + zipStream.Position = 0; + using var archive = ZipArchive.OpenArchive(zipStream); + + var entry = archive.Entries.Single(e => !e.IsDirectory); + using var entryStream = entry.OpenEntryStream(); + using var extractedStream = new MemoryStream(); + entryStream.CopyTo(extractedStream); + + var extractedData = extractedStream.ToArray(); + var actualCrc = CalculateCrc32(extractedData); + + Assert.Equal(compressionType, entry.CompressionType); + Assert.Equal(expectedCrc, actualCrc); + Assert.Equal(testData.Length, extractedData.Length); + + // For smaller files, verify full content; for larger, spot check + if (testData.Length <= sizeMb * 2) + { + Assert.Equal(testData, extractedData); + } + else + { + VerifyDataSpotCheck(testData, extractedData); + } + + VerifyCompressionRatio( + testData.Length, + zipStream.Length, + expectedRatio, + $"{compressionType} Level {compressionLevel}" + ); + } +} diff --git a/tests/SharpCompress.Test/Zip/ZipReaderAsyncTests.cs b/tests/SharpCompress.Test/Zip/ZipReaderAsyncTests.cs new file mode 100644 index 00000000..4c7cd627 --- /dev/null +++ b/tests/SharpCompress.Test/Zip/ZipReaderAsyncTests.cs @@ -0,0 +1,405 @@ +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using SharpCompress.Archives; +using SharpCompress.Common; +using SharpCompress.IO; +using SharpCompress.Readers; +using SharpCompress.Readers.Zip; +using SharpCompress.Test.Mocks; +using Xunit; + +namespace SharpCompress.Test.Zip; + +public class ZipReaderAsyncTests : ReaderTests +{ + public ZipReaderAsyncTests() => UseExtensionInsteadOfNameToVerify = true; + + [Fact] + public async ValueTask Issue_269_Double_Skip_Async() + { + var path = Path.Combine(TEST_ARCHIVES_PATH, "PrePostHeaders.zip"); + using Stream stream = new ForwardOnlyStream(File.OpenRead(path)); + await using var reader = await ReaderFactory.OpenAsyncReader(new AsyncOnlyStream(stream)); + var count = 0; + while (await reader.MoveToNextEntryAsync()) + { + count++; + if (!reader.Entry.IsDirectory) + { + if (count % 2 != 0) + { + await reader.WriteEntryToAsync(Stream.Null); + } + } + } + } + + [Fact] + public async ValueTask Zip_Zip64_Streamed_Read_Async() => + await ReadAsync("Zip.zip64.zip", CompressionType.Deflate); + + [Fact] + public async ValueTask Zip_ZipX_Streamed_Read_Async() => + await ReadAsync("Zip.zipx", CompressionType.LZMA); + + [Fact] + public async ValueTask Zip_BZip2_Streamed_Read_Async() => + await ReadAsync("Zip.bzip2.dd.zip", CompressionType.BZip2); + + [Fact] + public async ValueTask Zip_BZip2_Read_Async() => + await ReadAsync("Zip.bzip2.zip", CompressionType.BZip2); + + [Fact] + public async ValueTask Zip_Deflate_Streamed2_Read_Async() => + await ReadAsync("Zip.deflate.dd-.zip", CompressionType.Deflate); + + [Fact] + public async ValueTask Zip_Deflate_Streamed_Read_Async() => + await ReadAsync("Zip.deflate.dd.zip", CompressionType.Deflate); + + [Fact] + public async ValueTask Zip_Deflate_Streamed_Skip_Async() + { + using Stream stream = new ForwardOnlyStream( + File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.dd.zip")) + ); + await using var reader = await ReaderFactory.OpenAsyncReader(new AsyncOnlyStream(stream)); + var x = 0; + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + x++; + if (x % 2 == 0) + { + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + } + } + + [Fact] + public async ValueTask Zip_Deflate_Streamed2_Skip_Async() + { + using Stream stream = new ForwardOnlyStream( + File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.dd-.zip")) + ); + await using var reader = await ReaderFactory.OpenAsyncReader(new AsyncOnlyStream(stream)); + var x = 0; + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + x++; + if (x % 2 == 0) + { + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + } + } + + [Fact] + public async ValueTask Zip_Deflate_Read_Async() => + await ReadAsync("Zip.deflate.zip", CompressionType.Deflate); + + [Fact] + public async ValueTask Zip_Deflate64_Read_Async() => + await ReadAsync("Zip.deflate64.zip", CompressionType.Deflate64); + + [Fact] + public async ValueTask Zip_LZMA_Streamed_Read_Async() => + await ReadAsync("Zip.lzma.dd.zip", CompressionType.LZMA); + + [Fact] + public async ValueTask Zip_LZMA_Read_Async() => + await ReadAsync("Zip.lzma.zip", CompressionType.LZMA); + + [Fact] + public async ValueTask Zip_PPMd_Streamed_Read_Async() => + await ReadAsync("Zip.ppmd.dd.zip", CompressionType.PPMd); + + [Fact] + public async ValueTask Zip_PPMd_Read_Async() => + await ReadAsync("Zip.ppmd.zip", CompressionType.PPMd); + + [Fact] + public async ValueTask Zip_None_Read_Async() => + await ReadAsync("Zip.none.zip", CompressionType.None); + + [Fact] + public async ValueTask Zip_Deflate_NoEmptyDirs_Read_Async() => + await ReadAsync("Zip.deflate.noEmptyDirs.zip", CompressionType.Deflate); + + [Fact] + public async ValueTask Zip_BZip2_PkwareEncryption_Read_Async() + { + using ( + Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Zip.bzip2.pkware.zip")) + ) + using ( + IReader baseReader = ZipReader.OpenReader( + stream, + ReaderOptions.ForExternalStream with + { + Password = "test", + } + ) + ) + { + IAsyncReader reader = (IAsyncReader)baseReader; + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + Assert.Equal(CompressionType.BZip2, reader.Entry.CompressionType); + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + } + VerifyFiles(); + } + + [Fact] + public async ValueTask Zip_Reader_Disposal_Test_Async() + { + using var stream = new TestStream( + File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.dd.zip")) + ); + await using ( + var reader = await ReaderFactory.OpenAsyncReader( + new AsyncOnlyStream(stream), + ReaderOptions.ForExternalStream.WithLeaveStreamOpen(false) + ) + ) + { + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + } + Assert.True(stream.IsDisposed); + } + + [Fact] + public async ValueTask Zip_Reader_Disposal_Test2_Async() + { + using var stream = new TestStream( + new AsyncOnlyStream( + File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.dd.zip")) + ) + ); + await using var reader = await ReaderFactory.OpenAsyncReader(stream); + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + Assert.False(stream.IsDisposed); + } + + [Fact] + public async ValueTask Zip_LZMA_WinzipAES_Read_Async() => + await Assert.ThrowsAsync(async () => + { + using ( + Stream stream = new AsyncOnlyStream( + File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Zip.lzma.WinzipAES.zip")) + ) + ) + using ( + IReader baseReader = ZipReader.OpenReader( + stream, + ReaderOptions.ForExternalStream with + { + Password = "test", + } + ) + ) + { + IAsyncReader reader = (IAsyncReader)baseReader; + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + Assert.Equal(CompressionType.LZMA, reader.Entry.CompressionType); + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + } + VerifyFiles(); + }); + + [Fact] + public async ValueTask Zip_Deflate_WinzipAES_Read_Async() + { + using ( + Stream stream = new AsyncOnlyStream( + File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.WinzipAES.zip")) + ) + ) + + await using ( + var reader = await ReaderFactory.OpenAsyncReader( + stream, + ReaderOptions.ForExternalStream with + { + Password = "test", + } + ) + ) + { + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + Assert.Equal(CompressionType.Deflate, reader.Entry.CompressionType); + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + } + } + } + VerifyFiles(); + } + + [Fact] + public async ValueTask Zip_Deflate_ZipCrypto_Read_Async() + { + var count = 0; + using ( + Stream stream = new AsyncOnlyStream( + File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "zipcrypto.zip")) + ) + ) + await using ( + var reader = await ReaderFactory.OpenAsyncReader( + stream, + ReaderOptions.ForExternalStream with + { + Password = "test", + } + ) + ) + { + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + Assert.Equal(CompressionType.None, reader.Entry.CompressionType); + await reader.WriteEntryToDirectoryAsync(SCRATCH_FILES_PATH); + count++; + } + } + } + Assert.Equal(8, count); + } + + [Fact] + public async ValueTask EntryStream_Dispose_DoesNotThrow_OnNonSeekableStream_Deflate_Async() + { + // Since version 0.41.0: EntryStream.DisposeAsync() should not throw NotSupportedException + // when FlushAsync() fails on non-seekable streams (Deflate compression) + var path = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.dd.zip"); + using Stream stream = new ForwardOnlyStream(File.OpenRead(path)); + await using var reader = await ReaderFactory.OpenAsyncReader(new AsyncOnlyStream(stream)); + + // This should not throw, even if internal FlushAsync() fails + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { +#if LEGACY_DOTNET + using var entryStream = await reader.OpenEntryStreamAsync(); +#else + await using var entryStream = await reader.OpenEntryStreamAsync(); +#endif + // Read some data + var buffer = new byte[1024]; + await entryStream.ReadAsync(buffer, 0, buffer.Length); + // DisposeAsync should not throw NotSupportedException + } + } + } + + [Fact] + public async ValueTask EntryStream_Dispose_DoesNotThrow_OnNonSeekableStream_LZMA_Async() + { + // Since version 0.41.0: EntryStream.DisposeAsync() should not throw NotSupportedException + // when FlushAsync() fails on non-seekable streams (LZMA compression) + var path = Path.Combine(TEST_ARCHIVES_PATH, "Zip.lzma.dd.zip"); + using Stream stream = new ForwardOnlyStream(File.OpenRead(path)); + await using var reader = await ReaderFactory.OpenAsyncReader(new AsyncOnlyStream(stream)); + + // This should not throw, even if internal FlushAsync() fails + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { +#if LEGACY_DOTNET + using var entryStream = await reader.OpenEntryStreamAsync(); +#else + await using var entryStream = await reader.OpenEntryStreamAsync(); +#endif + // Read some data + var buffer = new byte[1024]; + await entryStream.ReadAsync(buffer, 0, buffer.Length); + // DisposeAsync should not throw NotSupportedException + } + } + } + + [Fact] + public async ValueTask Archive_Iteration_DoesNotBreak_WhenFlushThrows_Deflate_Async() + { + // Regression test: since 0.41.0, archive iteration would silently break + // when the input stream throws NotSupportedException in Flush(). + // Only the first entry would be returned, then iteration would stop without exception. + var path = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.dd.zip"); + using var fileStream = File.OpenRead(path); + using Stream stream = new ThrowOnFlushStream(fileStream); + await using var reader = await ReaderFactory.OpenAsyncReader(new AsyncOnlyStream(stream)); + + var count = 0; + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + count++; + } + } + + // Should iterate through all entries, not just the first one + Assert.True(count > 1, $"Expected more than 1 entry, but got {count}"); + } + + [Fact] + public async ValueTask Archive_Iteration_DoesNotBreak_WhenFlushThrows_LZMA_Async() + { + // Regression test: since 0.41.0, archive iteration would silently break + // when the input stream throws NotSupportedException in Flush(). + // Only the first entry would be returned, then iteration would stop without exception. + var path = Path.Combine(TEST_ARCHIVES_PATH, "Zip.lzma.dd.zip"); + using var fileStream = File.OpenRead(path); + using Stream stream = new ThrowOnFlushStream(fileStream); + await using var reader = await ReaderFactory.OpenAsyncReader(new AsyncOnlyStream(stream)); + + var count = 0; + while (await reader.MoveToNextEntryAsync()) + { + if (!reader.Entry.IsDirectory) + { + count++; + } + } + + // Should iterate through all entries, not just the first one + Assert.True(count > 1, $"Expected more than 1 entry, but got {count}"); + } +} diff --git a/tests/SharpCompress.Test/Zip/ZipReaderTests.cs b/tests/SharpCompress.Test/Zip/ZipReaderTests.cs index e5254c19..473c862e 100644 --- a/tests/SharpCompress.Test/Zip/ZipReaderTests.cs +++ b/tests/SharpCompress.Test/Zip/ZipReaderTests.cs @@ -1,5 +1,8 @@ using System; +using System.Collections.Generic; using System.IO; +using System.Linq; +using SharpCompress.Archives; using SharpCompress.Common; using SharpCompress.IO; using SharpCompress.Readers; @@ -19,7 +22,7 @@ public class ZipReaderTests : ReaderTests { var path = Path.Combine(TEST_ARCHIVES_PATH, "PrePostHeaders.zip"); using Stream stream = new ForwardOnlyStream(File.OpenRead(path)); - using var reader = ReaderFactory.Open(stream); + using var reader = ReaderFactory.OpenReader(stream); var count = 0; while (reader.MoveToNextEntry()) { @@ -59,7 +62,7 @@ public class ZipReaderTests : ReaderTests using Stream stream = new ForwardOnlyStream( File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.dd.zip")) ); - using var reader = ReaderFactory.Open(stream); + using var reader = ReaderFactory.OpenReader(stream); var x = 0; while (reader.MoveToNextEntry()) { @@ -68,10 +71,28 @@ public class ZipReaderTests : ReaderTests x++; if (x % 2 == 0) { - reader.WriteEntryToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + reader.WriteEntryToDirectory(SCRATCH_FILES_PATH); + } + } + } + } + + [Fact] + public void Zip_Deflate_Streamed2_Skip() + { + using Stream stream = new ForwardOnlyStream( + File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.dd-.zip")) + ); + using var reader = ReaderFactory.OpenReader(stream); + var x = 0; + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + x++; + if (x % 2 == 0) + { + reader.WriteEntryToDirectory(SCRATCH_FILES_PATH); } } } @@ -108,17 +129,22 @@ public class ZipReaderTests : ReaderTests using ( Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Zip.bzip2.pkware.zip")) ) - using (var reader = ZipReader.Open(stream, new ReaderOptions() { Password = "test" })) + using ( + var reader = ZipReader.OpenReader( + stream, + ReaderOptions.ForExternalStream with + { + Password = "test", + } + ) + ) { while (reader.MoveToNextEntry()) { if (!reader.Entry.IsDirectory) { Assert.Equal(CompressionType.BZip2, reader.Entry.CompressionType); - reader.WriteEntryToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + reader.WriteEntryToDirectory(SCRATCH_FILES_PATH); } } } @@ -131,16 +157,18 @@ public class ZipReaderTests : ReaderTests using var stream = new TestStream( File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.dd.zip")) ); - using (var reader = ReaderFactory.Open(stream)) + using ( + var reader = ReaderFactory.OpenReader( + stream, + ReaderOptions.ForExternalStream.WithLeaveStreamOpen(false) + ) + ) { while (reader.MoveToNextEntry()) { if (!reader.Entry.IsDirectory) { - reader.WriteEntryToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + reader.WriteEntryToDirectory(SCRATCH_FILES_PATH); } } } @@ -153,15 +181,12 @@ public class ZipReaderTests : ReaderTests using var stream = new TestStream( File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.dd.zip")) ); - var reader = ReaderFactory.Open(stream); + var reader = ReaderFactory.OpenReader(stream); while (reader.MoveToNextEntry()) { if (!reader.Entry.IsDirectory) { - reader.WriteEntryToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + reader.WriteEntryToDirectory(SCRATCH_FILES_PATH); } } Assert.False(stream.IsDisposed); @@ -176,17 +201,22 @@ public class ZipReaderTests : ReaderTests Path.Combine(TEST_ARCHIVES_PATH, "Zip.lzma.WinzipAES.zip") ) ) - using (var reader = ZipReader.Open(stream, new ReaderOptions() { Password = "test" })) + using ( + var reader = ZipReader.OpenReader( + stream, + ReaderOptions.ForExternalStream with + { + Password = "test", + } + ) + ) { while (reader.MoveToNextEntry()) { if (!reader.Entry.IsDirectory) { - Assert.Equal(CompressionType.Unknown, reader.Entry.CompressionType); - reader.WriteEntryToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + Assert.Equal(CompressionType.LZMA, reader.Entry.CompressionType); + reader.WriteEntryToDirectory(SCRATCH_FILES_PATH); } } } @@ -201,17 +231,22 @@ public class ZipReaderTests : ReaderTests Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.WinzipAES.zip") ) ) - using (var reader = ZipReader.Open(stream, new ReaderOptions() { Password = "test" })) + using ( + var reader = ZipReader.OpenReader( + stream, + ReaderOptions.ForExternalStream with + { + Password = "test", + } + ) + ) { while (reader.MoveToNextEntry()) { if (!reader.Entry.IsDirectory) { - Assert.Equal(CompressionType.Unknown, reader.Entry.CompressionType); - reader.WriteEntryToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + Assert.Equal(CompressionType.Deflate, reader.Entry.CompressionType); + reader.WriteEntryToDirectory(SCRATCH_FILES_PATH); } } } @@ -223,17 +258,22 @@ public class ZipReaderTests : ReaderTests { var count = 0; using (Stream stream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "zipcrypto.zip"))) - using (var reader = ZipReader.Open(stream, new ReaderOptions() { Password = "test" })) + using ( + var reader = ZipReader.OpenReader( + stream, + ReaderOptions.ForExternalStream with + { + Password = "test", + } + ) + ) { while (reader.MoveToNextEntry()) { if (!reader.Entry.IsDirectory) { Assert.Equal(CompressionType.None, reader.Entry.CompressionType); - reader.WriteEntryToDirectory( - SCRATCH_FILES_PATH, - new ExtractionOptions() { ExtractFullPath = true, Overwrite = true } - ); + reader.WriteEntryToDirectory(SCRATCH_FILES_PATH); count++; } } @@ -247,13 +287,19 @@ public class ZipReaderTests : ReaderTests var expected = new[] { new Tuple("foo.txt", Array.Empty()), - new Tuple("foo2.txt", new byte[10]) + new Tuple("foo2.txt", new byte[10]), }; using var memory = new MemoryStream(); Stream stream = new TestStream(memory, read: true, write: true, seek: false); - using (var zipWriter = WriterFactory.Open(stream, ArchiveType.Zip, CompressionType.Deflate)) + using ( + var zipWriter = WriterFactory.OpenWriter( + stream, + ArchiveType.Zip, + new WriterOptions(CompressionType.Deflate) + ) + ) { zipWriter.Write(expected[0].Item1, new MemoryStream(expected[0].Item2)); zipWriter.Write(expected[1].Item1, new MemoryStream(expected[1].Item2)); @@ -262,7 +308,9 @@ public class ZipReaderTests : ReaderTests stream = new MemoryStream(memory.ToArray()); File.WriteAllBytes(Path.Combine(SCRATCH_FILES_PATH, "foo.zip"), memory.ToArray()); - using IReader zipReader = ZipReader.Open(NonDisposingStream.Create(stream, true)); + using IReader zipReader = ZipReader.OpenReader( + SharpCompressStream.CreateNonDisposing(stream) + ); var i = 0; while (zipReader.MoveToNextEntry()) { @@ -292,7 +340,7 @@ public class ZipReaderTests : ReaderTests using Stream stream = File.OpenRead( Path.Combine(TEST_ARCHIVES_PATH, "Zip.none.issue86.zip") ); - using var reader = ZipReader.Open(stream); + using var reader = ZipReader.OpenReader(stream); foreach (var key in keys) { reader.MoveToNextEntry(); @@ -314,7 +362,7 @@ public class ZipReaderTests : ReaderTests var keys = new[] { "version", "sizehint", "data/0/metadata", "data/0/records" }; using var fileStream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "test_477.zip")); - using var reader = ZipReader.Open(fileStream); + using var reader = ZipReader.OpenReader(fileStream); foreach (var key in keys) { reader.MoveToNextEntry(); @@ -328,7 +376,7 @@ public class ZipReaderTests : ReaderTests { var count = 0; using var fileStream = File.OpenRead(Path.Combine(TEST_ARCHIVES_PATH, "Issue_685.zip")); - using var reader = ZipReader.Open(fileStream); + using var reader = ZipReader.OpenReader(fileStream); while (reader.MoveToNextEntry()) { count++; @@ -341,8 +389,8 @@ public class ZipReaderTests : ReaderTests public void Zip_ReaderFactory_Uncompressed_Read_All() { var zipPath = Path.Combine(TEST_ARCHIVES_PATH, "Zip.uncompressed.zip"); - using var stream = File.Open(zipPath, FileMode.Open, FileAccess.Read); - using var reader = ReaderFactory.Open(stream); + using var stream = File.OpenRead(zipPath); + using var reader = ReaderFactory.OpenReader(stream); while (reader.MoveToNextEntry()) { var target = new MemoryStream(); @@ -354,20 +402,22 @@ public class ZipReaderTests : ReaderTests public void Zip_ReaderFactory_Uncompressed_Skip_All() { var zipPath = Path.Combine(TEST_ARCHIVES_PATH, "Zip.uncompressed.zip"); - using var stream = File.Open(zipPath, FileMode.Open, FileAccess.Read); - using var reader = ReaderFactory.Open(stream); + using var stream = File.OpenRead(zipPath); + using var reader = ReaderFactory.OpenReader(stream); while (reader.MoveToNextEntry()) { } } + //this test uses a large 7zip file containing a zip file inside it to test zip64 support + // we probably shouldn't be allowing ExtractAllEntries here but it works for now. [Fact] public void Zip_Uncompressed_64bit() { var zipPath = Path.Combine(TEST_ARCHIVES_PATH, "64bitstream.zip.7z"); - using var stream = File.Open(zipPath, FileMode.Open, FileAccess.Read); - var archive = Archives.ArchiveFactory.Open(stream); + using var stream = File.OpenRead(zipPath); + var archive = ArchiveFactory.OpenArchive(stream); var reader = archive.ExtractAllEntries(); reader.MoveToNextEntry(); - var zipReader = ZipReader.Open(reader.OpenEntryStream()); + var zipReader = ZipReader.OpenReader(reader.OpenEntryStream()); var x = 0; while (zipReader.MoveToNextEntry()) { @@ -376,4 +426,177 @@ public class ZipReaderTests : ReaderTests Assert.Equal(4, x); } + + [Fact] + public void Zip_Uncompressed_Encrypted_Read() + { + using var reader = ReaderFactory.OpenReader( + Path.Combine(TEST_ARCHIVES_PATH, "Zip.none.encrypted.zip"), + ReaderOptions.ForFilePath with + { + Password = "test", + } + ); + reader.MoveToNextEntry(); + Assert.Equal("first.txt", reader.Entry.Key); + Assert.Equal(199, reader.Entry.Size); + reader.OpenEntryStream().Dispose(); + reader.MoveToNextEntry(); + Assert.Equal("second.txt", reader.Entry.Key); + Assert.Equal(197, reader.Entry.Size); + } + + [Fact] + public void ZipReader_Returns_Same_Entries_As_ZipArchive() + { + // Verifies that ZipReader and ZipArchive return the same entries + // for standard single-volume ZIP files. ZipReader processes LocalEntry + // headers sequentially, while ZipArchive uses DirectoryEntry headers + // from the central directory and seeks to LocalEntry headers for data. + var testFiles = new[] { "Zip.none.zip", "Zip.deflate.zip", "Zip.none.issue86.zip" }; + + foreach (var testFile in testFiles) + { + var path = Path.Combine(TEST_ARCHIVES_PATH, testFile); + + var readerKeys = new List(); + using (var stream = File.OpenRead(path)) + using (var reader = ZipReader.OpenReader(stream)) + { + while (reader.MoveToNextEntry()) + { + readerKeys.Add(reader.Entry.Key!); + } + } + + var archiveKeys = new List(); + using (var archive = Archives.Zip.ZipArchive.OpenArchive(path)) + { + foreach (var entry in archive.Entries) + { + archiveKeys.Add(entry.Key!); + } + } + + Assert.Equal(archiveKeys.Count, readerKeys.Count); + Assert.Equal(archiveKeys.OrderBy(k => k), readerKeys.OrderBy(k => k)); + } + } + + [Fact] + public void EntryStream_Dispose_DoesNotThrow_OnNonSeekableStream_Deflate() + { + // Since version 0.41.0: EntryStream.Dispose() should not throw NotSupportedException + // when Flush() fails on non-seekable streams (Deflate compression) + var path = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.dd.zip"); + using Stream stream = new ForwardOnlyStream(File.OpenRead(path)); + using var reader = ReaderFactory.OpenReader(stream); + + // This should not throw, even if internal Flush() fails + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + using var entryStream = reader.OpenEntryStream(); + // Read some data + var buffer = new byte[1024]; + entryStream.Read(buffer, 0, buffer.Length); + // Dispose should not throw NotSupportedException + } + } + } + + [Fact] + public void EntryStream_Dispose_DoesNotThrow_OnNonSeekableStream_LZMA() + { + // Since version 0.41.0: EntryStream.Dispose() should not throw NotSupportedException + // when Flush() fails on non-seekable streams (LZMA compression) + var path = Path.Combine(TEST_ARCHIVES_PATH, "Zip.lzma.dd.zip"); + using Stream stream = new ForwardOnlyStream(File.OpenRead(path)); + using var reader = ReaderFactory.OpenReader(stream); + + // This should not throw, even if internal Flush() fails + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + using var entryStream = reader.OpenEntryStream(); + // Read some data + var buffer = new byte[1024]; + entryStream.Read(buffer, 0, buffer.Length); + // Dispose should not throw NotSupportedException + } + } + } + + [Fact] + public void Archive_Iteration_DoesNotBreak_WhenFlushThrows_Deflate() + { + // Regression test: since 0.41.0, archive iteration would silently break + // when the input stream throws NotSupportedException in Flush(). + // Only the first entry would be returned, then iteration would stop without exception. + var path = Path.Combine(TEST_ARCHIVES_PATH, "Zip.deflate.dd.zip"); + using var fileStream = File.OpenRead(path); + using Stream stream = new ThrowOnFlushStream(fileStream); + using var reader = ReaderFactory.OpenReader(stream); + + var count = 0; + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + count++; + } + } + + // Should iterate through all entries, not just the first one + Assert.True(count > 1, $"Expected more than 1 entry, but got {count}"); + } + + [Fact] + public void Archive_Iteration_DoesNotBreak_WhenFlushThrows_LZMA() + { + // Regression test: since 0.41.0, archive iteration would silently break + // when the input stream throws NotSupportedException in Flush(). + // Only the first entry would be returned, then iteration would stop without exception. + var path = Path.Combine(TEST_ARCHIVES_PATH, "Zip.lzma.dd.zip"); + using var fileStream = File.OpenRead(path); + using Stream stream = new ThrowOnFlushStream(fileStream); + using var reader = ReaderFactory.OpenReader(stream); + + var count = 0; + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + count++; + } + } + + // Should iterate through all entries, not just the first one + Assert.True(count > 1, $"Expected more than 1 entry, but got {count}"); + } + + [Fact] + public void Zip_LZMA_ZeroSizeEntry_CanExtract_Streaming() + { + var path = Path.Combine(TEST_ARCHIVES_PATH, "Zip.lzma.empty.zip"); + using var fileStream = File.OpenRead(path); + using Stream stream = new ForwardOnlyStream(fileStream); + using var reader = ReaderFactory.OpenReader(stream); + + var count = 0; + while (reader.MoveToNextEntry()) + { + if (!reader.Entry.IsDirectory) + { + count++; + Assert.Equal(0, reader.Entry.Size); + var outStream = new MemoryStream(); + reader.WriteEntryTo(outStream); + Assert.Equal(0, outStream.Length); + } + } + Assert.Equal(1, count); + } } diff --git a/tests/SharpCompress.Test/Zip/ZipShortReadTests.cs b/tests/SharpCompress.Test/Zip/ZipShortReadTests.cs new file mode 100644 index 00000000..f9f1d72a --- /dev/null +++ b/tests/SharpCompress.Test/Zip/ZipShortReadTests.cs @@ -0,0 +1,123 @@ +using System; +using System.Collections.Generic; +using System.IO; +using SharpCompress.Readers; +using Xunit; + +namespace SharpCompress.Test.Zip; + +/// +/// Tests for ZIP reading with streams that return short reads. +/// Reproduces the regression where ZIP parsing fails depending on Stream.Read chunking patterns. +/// +public class ZipShortReadTests : ReaderTests +{ + /// + /// A non-seekable stream that returns controlled short reads. + /// Simulates real-world network/multipart streams that legally return fewer bytes than requested. + /// + private sealed class PatternReadStream : Stream + { + private readonly MemoryStream _inner; + private readonly int _firstReadSize; + private readonly int _chunkSize; + private bool _firstReadDone; + + public PatternReadStream(byte[] bytes, int firstReadSize, int chunkSize) + { + _inner = new MemoryStream(bytes, writable: false); + _firstReadSize = firstReadSize; + _chunkSize = chunkSize; + } + + public override int Read(byte[] buffer, int offset, int count) + { + int limit = !_firstReadDone ? _firstReadSize : _chunkSize; + _firstReadDone = true; + + int toRead = Math.Min(count, limit); + return _inner.Read(buffer, offset, toRead); + } + + public override bool CanRead => true; + public override bool CanSeek => false; + public override bool CanWrite => false; + public override long Length => throw new NotSupportedException(); + public override long Position + { + get => throw new NotSupportedException(); + set => throw new NotSupportedException(); + } + + public override void Flush() => throw new NotSupportedException(); + + public override long Seek(long offset, SeekOrigin origin) => + throw new NotSupportedException(); + + public override void SetLength(long value) => throw new NotSupportedException(); + + public override void Write(byte[] buffer, int offset, int count) => + throw new NotSupportedException(); + } + + /// + /// Test that ZIP reading works correctly with short reads on non-seekable streams. + /// Uses a test archive and different chunking patterns. + /// + [Theory] + [InlineData("Zip.deflate.zip", 1000, 4096)] + [InlineData("Zip.deflate.zip", 999, 4096)] + [InlineData("Zip.deflate.zip", 100, 4096)] + [InlineData("Zip.deflate.zip", 50, 512)] + [InlineData("Zip.deflate.zip", 1, 1)] // Extreme case: 1 byte at a time + [InlineData("Zip.deflate.dd.zip", 1000, 4096)] + [InlineData("Zip.deflate.dd.zip", 999, 4096)] + [InlineData("Zip.zip64.zip", 3816, 4096)] + [InlineData("Zip.zip64.zip", 3815, 4096)] // Similar to the issue pattern + public void Zip_Reader_Handles_Short_Reads(string zipFile, int firstReadSize, int chunkSize) + { + // Use an existing test ZIP file + var zipPath = Path.Combine(TEST_ARCHIVES_PATH, zipFile); + if (!File.Exists(zipPath)) + { + return; // Skip if file doesn't exist + } + + var bytes = File.ReadAllBytes(zipPath); + + // Baseline with MemoryStream (seekable, no short reads) + var baseline = ReadEntriesFromStream(new MemoryStream(bytes, writable: false)); + Assert.NotEmpty(baseline); + + // Non-seekable stream with controlled short read pattern + var chunked = ReadEntriesFromStream(new PatternReadStream(bytes, firstReadSize, chunkSize)); + Assert.Equal(baseline, chunked); + } + + private List ReadEntriesFromStream(Stream stream) + { + var names = new List(); + using var reader = ReaderFactory.OpenReader( + stream, + ReaderOptions.ForExternalStream with + { + LeaveStreamOpen = true, + } + ); + + while (reader.MoveToNextEntry()) + { + if (reader.Entry.IsDirectory) + { + continue; + } + + names.Add(reader.Entry.Key!); + + using var entryStream = reader.OpenEntryStream(); + entryStream.CopyTo(Stream.Null); + } + + return names; + } +} diff --git a/tests/SharpCompress.Test/Zip/ZipWriterAsyncTests.cs b/tests/SharpCompress.Test/Zip/ZipWriterAsyncTests.cs new file mode 100644 index 00000000..897e95bd --- /dev/null +++ b/tests/SharpCompress.Test/Zip/ZipWriterAsyncTests.cs @@ -0,0 +1,67 @@ +using System.Text; +using System.Threading.Tasks; +using SharpCompress.Common; +using Xunit; + +namespace SharpCompress.Test.Zip; + +public class ZipWriterAsyncTests : WriterTests +{ + public ZipWriterAsyncTests() + : base(ArchiveType.Zip) { } + + [Fact] + public async ValueTask Zip_Deflate_Write_Async() => + await WriteAsync( + CompressionType.Deflate, + "Zip.deflate.noEmptyDirs.zip", + "Zip.deflate.noEmptyDirs.zip", + Encoding.UTF8 + ); + + [Fact] + public async ValueTask Zip_BZip2_Write_Async() => + await WriteAsync( + CompressionType.BZip2, + "Zip.bzip2.noEmptyDirs.zip", + "Zip.bzip2.noEmptyDirs.zip", + Encoding.UTF8 + ); + + [Fact] + public async ValueTask Zip_None_Write_Async() => + await WriteAsync( + CompressionType.None, + "Zip.none.noEmptyDirs.zip", + "Zip.none.noEmptyDirs.zip", + Encoding.UTF8 + ); + + [Fact] + public async ValueTask Zip_LZMA_Write_Async() => + await WriteAsync( + CompressionType.LZMA, + "Zip.lzma.noEmptyDirs.zip", + "Zip.lzma.noEmptyDirs.zip", + Encoding.UTF8 + ); + + [Fact] + public async ValueTask Zip_PPMd_Write_Async() => + await WriteAsync( + CompressionType.PPMd, + "Zip.ppmd.noEmptyDirs.zip", + "Zip.ppmd.noEmptyDirs.zip", + Encoding.UTF8 + ); + + [Fact] + public async ValueTask Zip_Rar_Write_Async() => + await Assert.ThrowsAsync(async () => + await WriteAsync( + CompressionType.Rar, + "Zip.ppmd.noEmptyDirs.zip", + "Zip.ppmd.noEmptyDirs.zip" + ) + ); +} diff --git a/tests/SharpCompress.Test/Zip/ZipWriterDirectoryTests.cs b/tests/SharpCompress.Test/Zip/ZipWriterDirectoryTests.cs new file mode 100644 index 00000000..14bc143c --- /dev/null +++ b/tests/SharpCompress.Test/Zip/ZipWriterDirectoryTests.cs @@ -0,0 +1,146 @@ +using System; +using System.IO; +using System.Linq; +using SharpCompress.Archives.Zip; +using SharpCompress.Common; +using SharpCompress.Writers.Zip; +using Xunit; + +namespace SharpCompress.Test.Zip; + +public class ZipWriterDirectoryTests : TestBase +{ + [Fact] + public void ZipWriter_WriteDirectory_CreatesDirectoryEntry() + { + using var memoryStream = new MemoryStream(); + using ( + var writer = new ZipWriter(memoryStream, new ZipWriterOptions(CompressionType.Deflate)) + ) + { + writer.WriteDirectory("test-dir", DateTime.Now); + } + + memoryStream.Position = 0; + using var archive = ZipArchive.OpenArchive(memoryStream); + var entries = archive.Entries.ToList(); + + Assert.Single(entries); + Assert.Equal("test-dir/", entries[0].Key); + Assert.True(entries[0].IsDirectory); + } + + [Fact] + public void ZipWriter_WriteDirectory_WithTrailingSlash() + { + using var memoryStream = new MemoryStream(); + using ( + var writer = new ZipWriter(memoryStream, new ZipWriterOptions(CompressionType.Deflate)) + ) + { + writer.WriteDirectory("test-dir/", DateTime.Now); + } + + memoryStream.Position = 0; + using var archive = ZipArchive.OpenArchive(memoryStream); + var entries = archive.Entries.ToList(); + + Assert.Single(entries); + Assert.Equal("test-dir/", entries[0].Key); + Assert.True(entries[0].IsDirectory); + } + + [Fact] + public void ZipWriter_WriteDirectory_WithBackslash() + { + using var memoryStream = new MemoryStream(); + using ( + var writer = new ZipWriter(memoryStream, new ZipWriterOptions(CompressionType.Deflate)) + ) + { + writer.WriteDirectory("test-dir\\subdir", DateTime.Now); + } + + memoryStream.Position = 0; + using var archive = ZipArchive.OpenArchive(memoryStream); + var entries = archive.Entries.ToList(); + + Assert.Single(entries); + Assert.Equal("test-dir/subdir/", entries[0].Key); + Assert.True(entries[0].IsDirectory); + } + + [Fact] + public void ZipWriter_WriteDirectory_EmptyString_IsSkipped() + { + using var memoryStream = new MemoryStream(); + using ( + var writer = new ZipWriter(memoryStream, new ZipWriterOptions(CompressionType.Deflate)) + ) + { + writer.WriteDirectory("", DateTime.Now); + } + + memoryStream.Position = 0; + using var archive = ZipArchive.OpenArchive(memoryStream); + + Assert.Empty(archive.Entries); + } + + [Fact] + public void ZipWriter_WriteDirectory_MultipleDirectories() + { + using var memoryStream = new MemoryStream(); + using ( + var writer = new ZipWriter(memoryStream, new ZipWriterOptions(CompressionType.Deflate)) + ) + { + writer.WriteDirectory("dir1", DateTime.Now); + writer.WriteDirectory("dir2", DateTime.Now); + writer.WriteDirectory("dir1/subdir", DateTime.Now); + } + + memoryStream.Position = 0; + using var archive = ZipArchive.OpenArchive(memoryStream); + var entries = archive.Entries.OrderBy(e => e.Key).ToList(); + + Assert.Equal(3, entries.Count); + Assert.Equal("dir1/", entries[0].Key); + Assert.True(entries[0].IsDirectory); + Assert.Equal("dir1/subdir/", entries[1].Key); + Assert.True(entries[1].IsDirectory); + Assert.Equal("dir2/", entries[2].Key); + Assert.True(entries[2].IsDirectory); + } + + [Fact] + public void ZipWriter_WriteDirectory_MixedWithFiles() + { + using var memoryStream = new MemoryStream(); + using ( + var writer = new ZipWriter(memoryStream, new ZipWriterOptions(CompressionType.Deflate)) + ) + { + writer.WriteDirectory("dir1", DateTime.Now); + + using var contentStream = new MemoryStream( + System.Text.Encoding.UTF8.GetBytes("test content") + ); + writer.Write("dir1/file.txt", contentStream, DateTime.Now); + + writer.WriteDirectory("dir2", DateTime.Now); + } + + memoryStream.Position = 0; + using var archive = ZipArchive.OpenArchive(memoryStream); + var entries = archive.Entries.OrderBy(e => e.Key).ToList(); + + Assert.Equal(3, entries.Count); + Assert.Equal("dir1/", entries[0].Key); + Assert.True(entries[0].IsDirectory); + Assert.Equal("dir1/file.txt", entries[1].Key); + Assert.False(entries[1].IsDirectory); + Assert.Equal("dir2/", entries[2].Key); + Assert.True(entries[2].IsDirectory); + } +} diff --git a/tests/SharpCompress.Test/Zip/ZipWriterTests.cs b/tests/SharpCompress.Test/Zip/ZipWriterTests.cs index 530397bd..70c2632d 100644 --- a/tests/SharpCompress.Test/Zip/ZipWriterTests.cs +++ b/tests/SharpCompress.Test/Zip/ZipWriterTests.cs @@ -1,6 +1,7 @@ +using System.IO; using System.Text; - using SharpCompress.Common; +using SharpCompress.Writers; using Xunit; namespace SharpCompress.Test.Zip; @@ -10,6 +11,42 @@ public class ZipWriterTests : WriterTests public ZipWriterTests() : base(ArchiveType.Zip) { } + [Fact] + public void Zip_BZip2_Write_EmptyFile() + { + // Test that writing an empty file with BZip2 compression doesn't throw DivideByZeroException + using var memoryStream = new MemoryStream(); + var options = new WriterOptions(CompressionType.BZip2) + { + ArchiveEncoding = new ArchiveEncoding { Default = new UTF8Encoding(false) }, + }; + + using (var writer = WriterFactory.OpenWriter(memoryStream, ArchiveType.Zip, options)) + { + writer.Write("test-folder/zero-byte-file.txt", Stream.Null); + } + + Assert.True(memoryStream.Length > 0); + } + + [Fact] + public void Zip_BZip2_Write_EmptyFolder() + { + // Test that writing an empty folder entry with BZip2 compression doesn't throw DivideByZeroException + using var memoryStream = new MemoryStream(); + var options = new WriterOptions(CompressionType.BZip2) + { + ArchiveEncoding = new ArchiveEncoding { Default = new UTF8Encoding(false) }, + }; + + using (var writer = WriterFactory.OpenWriter(memoryStream, ArchiveType.Zip, options)) + { + writer.Write("test-empty-folder/", Stream.Null); + } + + Assert.True(memoryStream.Length > 0); + } + [Fact] public void Zip_Deflate_Write() => Write( @@ -57,7 +94,7 @@ public class ZipWriterTests : WriterTests [Fact] public void Zip_Rar_Write() => - Assert.Throws( - () => Write(CompressionType.Rar, "Zip.ppmd.noEmptyDirs.zip", "Zip.ppmd.noEmptyDirs.zip") + Assert.Throws(() => + Write(CompressionType.Rar, "Zip.ppmd.noEmptyDirs.zip", "Zip.ppmd.noEmptyDirs.zip") ); } diff --git a/tests/SharpCompress.Test/packages.lock.json b/tests/SharpCompress.Test/packages.lock.json new file mode 100644 index 00000000..2f685ff1 --- /dev/null +++ b/tests/SharpCompress.Test/packages.lock.json @@ -0,0 +1,560 @@ +{ + "version": 2, + "dependencies": { + ".NETFramework,Version=v4.8": { + "AwesomeAssertions": { + "type": "Direct", + "requested": "[9.4.0, )", + "resolved": "9.4.0", + "contentHash": "dJxkWiQ8D+xT6Gr2sSL83+Mar+Vpy2JTcUPxFcckpPJ8VYBfSgnk+zqpS6t7kcGnjz8NLyF14qfuoL4bKzzoew==", + "dependencies": { + "System.Threading.Tasks.Extensions": "4.5.4" + } + }, + "Microsoft.NET.Test.Sdk": { + "type": "Direct", + "requested": "[18.6.0, )", + "resolved": "18.6.0", + "contentHash": "kAIBt0MsYR0o2RULmlW5BhQ1ha50aGEgLKG4f1p0kePBGLJCprqs3S+NxRrYN8UH7mSQRPKpeiH9mwPMEKUObQ==", + "dependencies": { + "Microsoft.CodeCoverage": "18.6.0" + } + }, + "Microsoft.NETFramework.ReferenceAssemblies": { + "type": "Direct", + "requested": "[1.0.3, )", + "resolved": "1.0.3", + "contentHash": "vUc9Npcs14QsyOD01tnv/m8sQUnGTGOw1BCmKcv77LBJY7OxhJ+zJF7UD/sCL3lYNFuqmQEVlkfS4Quif6FyYg==", + "dependencies": { + "Microsoft.NETFramework.ReferenceAssemblies.net48": "1.0.3" + } + }, + "Microsoft.SourceLink.GitHub": { + "type": "Direct", + "requested": "[10.0.300, )", + "resolved": "10.0.300", + "contentHash": "QzCtLkXVb3l4IxcpvJCbzUwMLihAmLN6vVLjQGSzYSF8d2dvXxqJAZk83RV3gYnp2egz8jRMgSR2woY3vOahTA==", + "dependencies": { + "Microsoft.Build.Tasks.Git": "10.0.300", + "Microsoft.SourceLink.Common": "10.0.300", + "System.IO.Hashing": "10.0.8" + } + }, + "Microsoft.VisualStudio.Threading.Analyzers": { + "type": "Direct", + "requested": "[17.14.15, )", + "resolved": "17.14.15", + "contentHash": "mXQPJsbuUD2ydq4/ffd8h8tSOFCXec+2xJOVNCvXjuMOq/+5EKHq3D2m2MC2+nUaXeFMSt66VS/J4HdKBixgcw==" + }, + "PolySharp": { + "type": "Direct", + "requested": "[1.16.0, )", + "resolved": "1.16.0", + "contentHash": "3kdIIceBPumwjw279FuiVMfVENT2cGASXJgcigdySsbX2dJB8ofUgG6i47yqF/k1qu6fvNR3csrSekZPviR6kQ==" + }, + "xunit.runner.visualstudio": { + "type": "Direct", + "requested": "[3.1.5, )", + "resolved": "3.1.5", + "contentHash": "tKi7dSTwP4m5m9eXPM2Ime4Kn7xNf4x4zT9sdLO/G4hZVnQCRiMTWoSZqI/pYTVeI27oPPqHBKYI/DjJ9GsYgA==", + "dependencies": { + "Microsoft.TestPlatform.ObjectModel": "17.13.0" + } + }, + "xunit.v3": { + "type": "Direct", + "requested": "[3.2.2, )", + "resolved": "3.2.2", + "contentHash": "L+4/4y0Uqcg8/d6hfnxhnwh4j9FaeULvefTwrk30rr1o4n/vdPfyUQ8k0yzH8VJx7bmFEkDdcRfbtbjEHlaYcA==", + "dependencies": { + "xunit.v3.mtp-v1": "[3.2.2]" + } + }, + "Microsoft.ApplicationInsights": { + "type": "Transitive", + "resolved": "2.23.0", + "contentHash": "nWArUZTdU7iqZLycLKWe0TDms48KKGE6pONH2terYNa8REXiqixrMOkf1sk5DHGMaUTqONU2YkS4SAXBhLStgw==", + "dependencies": { + "System.Diagnostics.DiagnosticSource": "5.0.0" + } + }, + "Microsoft.Build.Tasks.Git": { + "type": "Transitive", + "resolved": "10.0.300", + "contentHash": "P0kaQwVZx4xIUe2FtrLyBadYNXuAljttJUPvjBYRuHhPE8L77L42KakLDkaADRiUrGspoLcMwayjrbQhYTr0zA==", + "dependencies": { + "System.IO.Hashing": "10.0.8" + } + }, + "Microsoft.CodeCoverage": { + "type": "Transitive", + "resolved": "18.6.0", + "contentHash": "bkmCXn/65Cd0LdO2zTb/ValGAJ1H8y/CgYOiBb3jsDyHI3Y1ljKx6RBvhvn3e5D/4R4I00RRwLf+Bd2Sn6bJjA==" + }, + "Microsoft.NETFramework.ReferenceAssemblies.net48": { + "type": "Transitive", + "resolved": "1.0.3", + "contentHash": "zMk4D+9zyiEWByyQ7oPImPN/Jhpj166Ky0Nlla4eXlNL8hI/BtSJsgR8Inldd4NNpIAH3oh8yym0W2DrhXdSLQ==" + }, + "Microsoft.SourceLink.Common": { + "type": "Transitive", + "resolved": "10.0.300", + "contentHash": "0jlkXaUGjYlWTIVPve5MftjKHnT3SlAtq9BCLV4J9IjdPrxV/+4rMlBSjfr1khG8/GC6KGojjola8E1VvWF0qQ==" + }, + "Microsoft.Testing.Extensions.Telemetry": { + "type": "Transitive", + "resolved": "1.9.1", + "contentHash": "No5AudZMmSb+uNXjlgL2y3/stHD2IT4uxqc5yHwkE+/nNux9jbKcaJMvcp9SwgP4DVD8L9/P3OUz8mmmcvEIdQ==", + "dependencies": { + "Microsoft.ApplicationInsights": "2.23.0", + "Microsoft.Testing.Platform": "1.9.1", + "System.Diagnostics.DiagnosticSource": "6.0.0" + } + }, + "Microsoft.Testing.Extensions.TrxReport.Abstractions": { + "type": "Transitive", + "resolved": "1.9.1", + "contentHash": "AL46Xe1WBi85Ntd4mNPvat5ZSsZ2uejiVqoKCypr8J3wK0elA5xJ3AN4G/Q4GIwzUFnggZoH/DBjnr9J18IO/g==", + "dependencies": { + "Microsoft.Testing.Platform": "1.9.1" + } + }, + "Microsoft.Testing.Platform": { + "type": "Transitive", + "resolved": "1.9.1", + "contentHash": "QafNtNSmEI0zazdebnsIkDKmFtTSpmx/5PLOjURWwozcPb3tvRxzosQSL8xwYNM1iPhhKiBksXZyRSE2COisrA==" + }, + "Microsoft.Testing.Platform.MSBuild": { + "type": "Transitive", + "resolved": "1.9.1", + "contentHash": "oTUtyR4X/s9ytuiNA29FGsNCCH0rNmY5Wdm14NCKLjTM1cT9edVSlA+rGS/mVmusPqcP0l/x9qOnMXg16v87RQ==", + "dependencies": { + "Microsoft.Testing.Platform": "1.9.1" + } + }, + "Microsoft.TestPlatform.ObjectModel": { + "type": "Transitive", + "resolved": "17.13.0", + "contentHash": "bt0E0Dx+iqW97o4A59RCmUmz/5NarJ7LRL+jXbSHod72ibL5XdNm1Ke+UO5tFhBG4VwHLcSjqq9BUSblGNWamw==", + "dependencies": { + "System.Reflection.Metadata": "1.6.0" + } + }, + "Microsoft.Win32.Registry": { + "type": "Transitive", + "resolved": "5.0.0", + "contentHash": "dDoKi0PnDz31yAyETfRntsLArTlVAVzUzCIvvEDsDsucrl33Dl8pIJG06ePTJTI3tGpeyHS9Cq7Foc/s4EeKcg==", + "dependencies": { + "System.Security.AccessControl": "5.0.0", + "System.Security.Principal.Windows": "5.0.0" + } + }, + "System.Buffers": { + "type": "Transitive", + "resolved": "4.6.1", + "contentHash": "N8GXpmiLMtljq7gwvyS+1QvKT/W2J8sNAvx+HVg4NGmsG/H+2k/y9QI23auLJRterrzCiDH+IWAw4V/GPwsMlw==" + }, + "System.Collections.Immutable": { + "type": "Transitive", + "resolved": "6.0.0", + "contentHash": "l4zZJ1WU2hqpQQHXz1rvC3etVZN+2DLmQMO79FhOTZHMn8tDRr+WU287sbomD0BETlmKDn0ygUgVy9k5xkkJdA==", + "dependencies": { + "System.Memory": "4.5.4", + "System.Runtime.CompilerServices.Unsafe": "6.0.0" + } + }, + "System.Diagnostics.DiagnosticSource": { + "type": "Transitive", + "resolved": "6.0.0", + "contentHash": "frQDfv0rl209cKm1lnwTgFPzNigy2EKk1BS3uAvHvlBVKe5cymGyHO+Sj+NLv5VF/AhHsqPIUUwya5oV4CHMUw==", + "dependencies": { + "System.Memory": "4.5.4", + "System.Runtime.CompilerServices.Unsafe": "6.0.0" + } + }, + "System.IO.Hashing": { + "type": "Transitive", + "resolved": "10.0.8", + "contentHash": "+dJsbPJ3FyUbTZNplFj0RCKePFizmv6ewDV46JE9q/IVH4c3xTCftHfHelLsAKf0jryIPqgMb5GpS0x7TAY3mg==", + "dependencies": { + "System.Buffers": "4.6.1", + "System.Memory": "4.6.3" + } + }, + "System.Memory": { + "type": "Transitive", + "resolved": "4.6.3", + "contentHash": "qdcDOgnFZY40+Q9876JUHnlHu7bosOHX8XISRoH94fwk6hgaeQGSgfZd8srWRZNt5bV9ZW2TljcegDNxsf+96A==", + "dependencies": { + "System.Buffers": "4.6.1", + "System.Numerics.Vectors": "4.6.1", + "System.Runtime.CompilerServices.Unsafe": "6.1.2" + } + }, + "System.Numerics.Vectors": { + "type": "Transitive", + "resolved": "4.6.1", + "contentHash": "sQxefTnhagrhoq2ReR0D/6K0zJcr9Hrd6kikeXsA1I8kOCboTavcUC4r7TSfpKFeE163uMuxZcyfO1mGO3EN8Q==" + }, + "System.Reflection.Metadata": { + "type": "Transitive", + "resolved": "1.6.0", + "contentHash": "COC1aiAJjCoA5GBF+QKL2uLqEBew4JsCkQmoHKbN3TlOZKa2fKLz5CpiRQKDz0RsAOEGsVKqOD5bomsXq/4STQ==", + "dependencies": { + "System.Collections.Immutable": "1.5.0" + } + }, + "System.Runtime.CompilerServices.Unsafe": { + "type": "Transitive", + "resolved": "6.1.2", + "contentHash": "2hBr6zdbIBTDE3EhK7NSVNdX58uTK6iHW/P/Axmm9sl1xoGSLqDvMtpecn226TNwHByFokYwJmt/aQQNlO5CRw==" + }, + "System.Security.AccessControl": { + "type": "Transitive", + "resolved": "5.0.0", + "contentHash": "dagJ1mHZO3Ani8GH0PHpPEe/oYO+rVdbQjvjJkBRNQkX4t0r1iaeGn8+/ybkSLEan3/slM0t59SVdHzuHf2jmw==", + "dependencies": { + "System.Security.Principal.Windows": "5.0.0" + } + }, + "System.Security.Principal.Windows": { + "type": "Transitive", + "resolved": "5.0.0", + "contentHash": "t0MGLukB5WAVU9bO3MGzvlGnyJPgUlcwerXn1kzBRjwLKixT96XV0Uza41W49gVd8zEMFu9vQEFlv0IOrytICA==" + }, + "System.Threading.Tasks.Extensions": { + "type": "Transitive", + "resolved": "4.5.4", + "contentHash": "zteT+G8xuGu6mS+mzDzYXbzS7rd3K6Fjb9RiZlYlJPam2/hU7JCBZBVEcywNuR+oZ1ncTvc/cq0faRr3P01OVg==", + "dependencies": { + "System.Runtime.CompilerServices.Unsafe": "4.5.3" + } + }, + "xunit.analyzers": { + "type": "Transitive", + "resolved": "1.27.0", + "contentHash": "y/pxIQaLvk/kxAoDkZW9GnHLCEqzwl5TW0vtX3pweyQpjizB9y3DXhb9pkw2dGeUqhLjsxvvJM1k89JowU6z3g==" + }, + "xunit.v3.assert": { + "type": "Transitive", + "resolved": "3.2.2", + "contentHash": "BPciBghgEEaJN/JG00QfCYDfEfnLgQhfnYEy+j1izoeHVNYd5+3Wm8GJ6JgYysOhpBPYGE+sbf75JtrRc7jrdA==", + "dependencies": { + "System.Collections.Immutable": "6.0.0", + "System.Memory": "4.5.5" + } + }, + "xunit.v3.common": { + "type": "Transitive", + "resolved": "3.2.2", + "contentHash": "Hj775PEH6GTbbg0wfKRvG2hNspDCvTH9irXhH4qIWgdrOSV1sQlqPie+DOvFeigsFg2fxSM3ZAaaCDQs+KreFA==", + "dependencies": { + "Microsoft.Bcl.AsyncInterfaces": "6.0.0" + } + }, + "xunit.v3.core.mtp-v1": { + "type": "Transitive", + "resolved": "3.2.2", + "contentHash": "Ga5aA2Ca9ktz+5k3g5ukzwfexwoqwDUpV6z7atSEUvqtd6JuybU1XopHqg1oFd78QdTfZgZE9h5sHpO4qYIi5w==", + "dependencies": { + "Microsoft.Testing.Extensions.Telemetry": "1.9.1", + "Microsoft.Testing.Extensions.TrxReport.Abstractions": "1.9.1", + "Microsoft.Testing.Platform": "1.9.1", + "Microsoft.Testing.Platform.MSBuild": "1.9.1", + "xunit.v3.extensibility.core": "[3.2.2]", + "xunit.v3.runner.inproc.console": "[3.2.2]" + } + }, + "xunit.v3.extensibility.core": { + "type": "Transitive", + "resolved": "3.2.2", + "contentHash": "srY8z/oMPvh/t8axtO2DwrHajhFMH7tnqKildvYrVQIfICi8fOn3yIBWkVPAcrKmHMwvXRJ/XsQM3VMR6DOYfQ==", + "dependencies": { + "xunit.v3.common": "[3.2.2]" + } + }, + "xunit.v3.mtp-v1": { + "type": "Transitive", + "resolved": "3.2.2", + "contentHash": "O41aAzYKBT5PWqATa1oEWVNCyEUypFQ4va6K0kz37dduV3EKzXNMaV2UnEhufzU4Cce1I33gg0oldS8tGL5I0A==", + "dependencies": { + "xunit.analyzers": "1.27.0", + "xunit.v3.assert": "[3.2.2]", + "xunit.v3.core.mtp-v1": "[3.2.2]" + } + }, + "xunit.v3.runner.common": { + "type": "Transitive", + "resolved": "3.2.2", + "contentHash": "/hkHkQCzGrugelOAehprm7RIWdsUFVmIVaD6jDH/8DNGCymTlKKPTbGokD5czbAfqfex47mBP0sb0zbHYwrO/g==", + "dependencies": { + "Microsoft.Win32.Registry": "[5.0.0]", + "xunit.v3.common": "[3.2.2]" + } + }, + "xunit.v3.runner.inproc.console": { + "type": "Transitive", + "resolved": "3.2.2", + "contentHash": "ulWOdSvCk+bPXijJZ73bth9NyoOHsAs1ZOvamYbCkD4DNLX/Bd29Ve2ZNUwBbK0MqfIYWXHZViy/HKrdEC/izw==", + "dependencies": { + "xunit.v3.extensibility.core": "[3.2.2]", + "xunit.v3.runner.common": "[3.2.2]" + } + }, + "sharpcompress": { + "type": "Project", + "dependencies": { + "Microsoft.Bcl.AsyncInterfaces": "[8.0.0, )", + "System.Text.Encoding.CodePages": "[8.0.0, )" + } + }, + "Microsoft.Bcl.AsyncInterfaces": { + "type": "CentralTransitive", + "requested": "[8.0.0, )", + "resolved": "8.0.0", + "contentHash": "3WA9q9yVqJp222P3x1wYIGDAkpjAku0TMUaaQV22g6L67AI0LdOIrVS7Ht2vJfLHGSPVuqN94vIr15qn+HEkHw==", + "dependencies": { + "System.Threading.Tasks.Extensions": "4.5.4" + } + }, + "System.Text.Encoding.CodePages": { + "type": "CentralTransitive", + "requested": "[8.0.0, )", + "resolved": "8.0.0", + "contentHash": "OZIsVplFGaVY90G2SbpgU7EnCoOO5pw1t4ic21dBF3/1omrJFpAGoNAVpPyMVOC90/hvgkGG3VFqR13YgZMQfg==", + "dependencies": { + "System.Memory": "4.5.5", + "System.Runtime.CompilerServices.Unsafe": "6.0.0" + } + } + }, + "net10.0": { + "AwesomeAssertions": { + "type": "Direct", + "requested": "[9.4.0, )", + "resolved": "9.4.0", + "contentHash": "dJxkWiQ8D+xT6Gr2sSL83+Mar+Vpy2JTcUPxFcckpPJ8VYBfSgnk+zqpS6t7kcGnjz8NLyF14qfuoL4bKzzoew==" + }, + "Microsoft.NET.Test.Sdk": { + "type": "Direct", + "requested": "[18.6.0, )", + "resolved": "18.6.0", + "contentHash": "kAIBt0MsYR0o2RULmlW5BhQ1ha50aGEgLKG4f1p0kePBGLJCprqs3S+NxRrYN8UH7mSQRPKpeiH9mwPMEKUObQ==", + "dependencies": { + "Microsoft.CodeCoverage": "18.6.0", + "Microsoft.TestPlatform.TestHost": "18.6.0" + } + }, + "Microsoft.NETFramework.ReferenceAssemblies": { + "type": "Direct", + "requested": "[1.0.3, )", + "resolved": "1.0.3", + "contentHash": "vUc9Npcs14QsyOD01tnv/m8sQUnGTGOw1BCmKcv77LBJY7OxhJ+zJF7UD/sCL3lYNFuqmQEVlkfS4Quif6FyYg==", + "dependencies": { + "Microsoft.NETFramework.ReferenceAssemblies.net461": "1.0.3" + } + }, + "Microsoft.SourceLink.GitHub": { + "type": "Direct", + "requested": "[10.0.300, )", + "resolved": "10.0.300", + "contentHash": "QzCtLkXVb3l4IxcpvJCbzUwMLihAmLN6vVLjQGSzYSF8d2dvXxqJAZk83RV3gYnp2egz8jRMgSR2woY3vOahTA==", + "dependencies": { + "Microsoft.Build.Tasks.Git": "10.0.300", + "Microsoft.SourceLink.Common": "10.0.300", + "System.IO.Hashing": "10.0.8" + } + }, + "Microsoft.VisualStudio.Threading.Analyzers": { + "type": "Direct", + "requested": "[17.14.15, )", + "resolved": "17.14.15", + "contentHash": "mXQPJsbuUD2ydq4/ffd8h8tSOFCXec+2xJOVNCvXjuMOq/+5EKHq3D2m2MC2+nUaXeFMSt66VS/J4HdKBixgcw==" + }, + "PolySharp": { + "type": "Direct", + "requested": "[1.16.0, )", + "resolved": "1.16.0", + "contentHash": "3kdIIceBPumwjw279FuiVMfVENT2cGASXJgcigdySsbX2dJB8ofUgG6i47yqF/k1qu6fvNR3csrSekZPviR6kQ==" + }, + "xunit.runner.visualstudio": { + "type": "Direct", + "requested": "[3.1.5, )", + "resolved": "3.1.5", + "contentHash": "tKi7dSTwP4m5m9eXPM2Ime4Kn7xNf4x4zT9sdLO/G4hZVnQCRiMTWoSZqI/pYTVeI27oPPqHBKYI/DjJ9GsYgA==" + }, + "xunit.v3": { + "type": "Direct", + "requested": "[3.2.2, )", + "resolved": "3.2.2", + "contentHash": "L+4/4y0Uqcg8/d6hfnxhnwh4j9FaeULvefTwrk30rr1o4n/vdPfyUQ8k0yzH8VJx7bmFEkDdcRfbtbjEHlaYcA==", + "dependencies": { + "xunit.v3.mtp-v1": "[3.2.2]" + } + }, + "Microsoft.ApplicationInsights": { + "type": "Transitive", + "resolved": "2.23.0", + "contentHash": "nWArUZTdU7iqZLycLKWe0TDms48KKGE6pONH2terYNa8REXiqixrMOkf1sk5DHGMaUTqONU2YkS4SAXBhLStgw==" + }, + "Microsoft.Build.Tasks.Git": { + "type": "Transitive", + "resolved": "10.0.300", + "contentHash": "P0kaQwVZx4xIUe2FtrLyBadYNXuAljttJUPvjBYRuHhPE8L77L42KakLDkaADRiUrGspoLcMwayjrbQhYTr0zA==", + "dependencies": { + "System.IO.Hashing": "10.0.8" + } + }, + "Microsoft.CodeCoverage": { + "type": "Transitive", + "resolved": "18.6.0", + "contentHash": "bkmCXn/65Cd0LdO2zTb/ValGAJ1H8y/CgYOiBb3jsDyHI3Y1ljKx6RBvhvn3e5D/4R4I00RRwLf+Bd2Sn6bJjA==" + }, + "Microsoft.NETFramework.ReferenceAssemblies.net461": { + "type": "Transitive", + "resolved": "1.0.3", + "contentHash": "AmOJZwCqnOCNp6PPcf9joyogScWLtwy0M1WkqfEQ0M9nYwyDD7EX9ZjscKS5iYnyvteX7kzSKFCKt9I9dXA6mA==" + }, + "Microsoft.SourceLink.Common": { + "type": "Transitive", + "resolved": "10.0.300", + "contentHash": "0jlkXaUGjYlWTIVPve5MftjKHnT3SlAtq9BCLV4J9IjdPrxV/+4rMlBSjfr1khG8/GC6KGojjola8E1VvWF0qQ==" + }, + "Microsoft.Testing.Extensions.Telemetry": { + "type": "Transitive", + "resolved": "1.9.1", + "contentHash": "No5AudZMmSb+uNXjlgL2y3/stHD2IT4uxqc5yHwkE+/nNux9jbKcaJMvcp9SwgP4DVD8L9/P3OUz8mmmcvEIdQ==", + "dependencies": { + "Microsoft.ApplicationInsights": "2.23.0", + "Microsoft.Testing.Platform": "1.9.1" + } + }, + "Microsoft.Testing.Extensions.TrxReport.Abstractions": { + "type": "Transitive", + "resolved": "1.9.1", + "contentHash": "AL46Xe1WBi85Ntd4mNPvat5ZSsZ2uejiVqoKCypr8J3wK0elA5xJ3AN4G/Q4GIwzUFnggZoH/DBjnr9J18IO/g==", + "dependencies": { + "Microsoft.Testing.Platform": "1.9.1" + } + }, + "Microsoft.Testing.Platform": { + "type": "Transitive", + "resolved": "1.9.1", + "contentHash": "QafNtNSmEI0zazdebnsIkDKmFtTSpmx/5PLOjURWwozcPb3tvRxzosQSL8xwYNM1iPhhKiBksXZyRSE2COisrA==" + }, + "Microsoft.Testing.Platform.MSBuild": { + "type": "Transitive", + "resolved": "1.9.1", + "contentHash": "oTUtyR4X/s9ytuiNA29FGsNCCH0rNmY5Wdm14NCKLjTM1cT9edVSlA+rGS/mVmusPqcP0l/x9qOnMXg16v87RQ==", + "dependencies": { + "Microsoft.Testing.Platform": "1.9.1" + } + }, + "Microsoft.TestPlatform.ObjectModel": { + "type": "Transitive", + "resolved": "18.6.0", + "contentHash": "gQTW4BIfM2ZLxixo9ITXoulLKjn20FiiHtqTsx9PENqTrX7368ZeJ5L0QZJyReXDWORPRV8jXwZR6Aar8JOyaA==" + }, + "Microsoft.TestPlatform.TestHost": { + "type": "Transitive", + "resolved": "18.6.0", + "contentHash": "em1eLz5Q46+hsCtAXdXggWAPd9gQyT4ngdsQ7k1eWvQgpsjtS/wAOJ/5TteieFdiAvrEq1iVn00LtusAxRaVmQ==", + "dependencies": { + "Microsoft.TestPlatform.ObjectModel": "18.6.0", + "Newtonsoft.Json": "13.0.3" + } + }, + "Microsoft.Win32.Registry": { + "type": "Transitive", + "resolved": "5.0.0", + "contentHash": "dDoKi0PnDz31yAyETfRntsLArTlVAVzUzCIvvEDsDsucrl33Dl8pIJG06ePTJTI3tGpeyHS9Cq7Foc/s4EeKcg==" + }, + "Newtonsoft.Json": { + "type": "Transitive", + "resolved": "13.0.3", + "contentHash": "HrC5BXdl00IP9zeV+0Z848QWPAoCr9P3bDEZguI+gkLcBKAOxix/tLEAAHC+UvDNPv4a2d18lOReHMOagPa+zQ==" + }, + "System.IO.Hashing": { + "type": "Transitive", + "resolved": "10.0.8", + "contentHash": "+dJsbPJ3FyUbTZNplFj0RCKePFizmv6ewDV46JE9q/IVH4c3xTCftHfHelLsAKf0jryIPqgMb5GpS0x7TAY3mg==" + }, + "xunit.analyzers": { + "type": "Transitive", + "resolved": "1.27.0", + "contentHash": "y/pxIQaLvk/kxAoDkZW9GnHLCEqzwl5TW0vtX3pweyQpjizB9y3DXhb9pkw2dGeUqhLjsxvvJM1k89JowU6z3g==" + }, + "xunit.v3.assert": { + "type": "Transitive", + "resolved": "3.2.2", + "contentHash": "BPciBghgEEaJN/JG00QfCYDfEfnLgQhfnYEy+j1izoeHVNYd5+3Wm8GJ6JgYysOhpBPYGE+sbf75JtrRc7jrdA==" + }, + "xunit.v3.common": { + "type": "Transitive", + "resolved": "3.2.2", + "contentHash": "Hj775PEH6GTbbg0wfKRvG2hNspDCvTH9irXhH4qIWgdrOSV1sQlqPie+DOvFeigsFg2fxSM3ZAaaCDQs+KreFA==", + "dependencies": { + "Microsoft.Bcl.AsyncInterfaces": "6.0.0" + } + }, + "xunit.v3.core.mtp-v1": { + "type": "Transitive", + "resolved": "3.2.2", + "contentHash": "Ga5aA2Ca9ktz+5k3g5ukzwfexwoqwDUpV6z7atSEUvqtd6JuybU1XopHqg1oFd78QdTfZgZE9h5sHpO4qYIi5w==", + "dependencies": { + "Microsoft.Testing.Extensions.Telemetry": "1.9.1", + "Microsoft.Testing.Extensions.TrxReport.Abstractions": "1.9.1", + "Microsoft.Testing.Platform": "1.9.1", + "Microsoft.Testing.Platform.MSBuild": "1.9.1", + "xunit.v3.extensibility.core": "[3.2.2]", + "xunit.v3.runner.inproc.console": "[3.2.2]" + } + }, + "xunit.v3.extensibility.core": { + "type": "Transitive", + "resolved": "3.2.2", + "contentHash": "srY8z/oMPvh/t8axtO2DwrHajhFMH7tnqKildvYrVQIfICi8fOn3yIBWkVPAcrKmHMwvXRJ/XsQM3VMR6DOYfQ==", + "dependencies": { + "xunit.v3.common": "[3.2.2]" + } + }, + "xunit.v3.mtp-v1": { + "type": "Transitive", + "resolved": "3.2.2", + "contentHash": "O41aAzYKBT5PWqATa1oEWVNCyEUypFQ4va6K0kz37dduV3EKzXNMaV2UnEhufzU4Cce1I33gg0oldS8tGL5I0A==", + "dependencies": { + "xunit.analyzers": "1.27.0", + "xunit.v3.assert": "[3.2.2]", + "xunit.v3.core.mtp-v1": "[3.2.2]" + } + }, + "xunit.v3.runner.common": { + "type": "Transitive", + "resolved": "3.2.2", + "contentHash": "/hkHkQCzGrugelOAehprm7RIWdsUFVmIVaD6jDH/8DNGCymTlKKPTbGokD5czbAfqfex47mBP0sb0zbHYwrO/g==", + "dependencies": { + "Microsoft.Win32.Registry": "[5.0.0]", + "xunit.v3.common": "[3.2.2]" + } + }, + "xunit.v3.runner.inproc.console": { + "type": "Transitive", + "resolved": "3.2.2", + "contentHash": "ulWOdSvCk+bPXijJZ73bth9NyoOHsAs1ZOvamYbCkD4DNLX/Bd29Ve2ZNUwBbK0MqfIYWXHZViy/HKrdEC/izw==", + "dependencies": { + "xunit.v3.extensibility.core": "[3.2.2]", + "xunit.v3.runner.common": "[3.2.2]" + } + }, + "sharpcompress": { + "type": "Project" + }, + "Microsoft.Bcl.AsyncInterfaces": { + "type": "CentralTransitive", + "requested": "[8.0.0, )", + "resolved": "8.0.0", + "contentHash": "3WA9q9yVqJp222P3x1wYIGDAkpjAku0TMUaaQV22g6L67AI0LdOIrVS7Ht2vJfLHGSPVuqN94vIr15qn+HEkHw==" + } + } + } +} \ No newline at end of file diff --git a/tests/SharpCompress.Test/xunit.runner.json b/tests/SharpCompress.Test/xunit.runner.json new file mode 100644 index 00000000..f78bc2f0 --- /dev/null +++ b/tests/SharpCompress.Test/xunit.runner.json @@ -0,0 +1,3 @@ +{ + "$schema": "https://xunit.net/schema/current/xunit.runner.schema.json" +} \ No newline at end of file diff --git a/tests/TestArchives/Archives/7Zip.ARM64.7z b/tests/TestArchives/Archives/7Zip.ARM64.7z new file mode 100644 index 00000000..2e9d6293 Binary files /dev/null and b/tests/TestArchives/Archives/7Zip.ARM64.7z differ diff --git a/tests/TestArchives/Archives/7Zip.Copy.7z b/tests/TestArchives/Archives/7Zip.Copy.7z new file mode 100644 index 00000000..6143ec0d Binary files /dev/null and b/tests/TestArchives/Archives/7Zip.Copy.7z differ diff --git a/tests/TestArchives/Archives/7Zip.EmptyStream.7z b/tests/TestArchives/Archives/7Zip.EmptyStream.7z new file mode 100644 index 00000000..c2d9bfa4 Binary files /dev/null and b/tests/TestArchives/Archives/7Zip.EmptyStream.7z differ diff --git a/tests/TestArchives/Archives/7Zip.Filters.7z b/tests/TestArchives/Archives/7Zip.Filters.7z index ea15f3b2..d066779b 100644 Binary files a/tests/TestArchives/Archives/7Zip.Filters.7z and b/tests/TestArchives/Archives/7Zip.Filters.7z differ diff --git a/tests/TestArchives/Archives/7Zip.LZMA2.exe b/tests/TestArchives/Archives/7Zip.LZMA2.exe new file mode 100644 index 00000000..889c8318 Binary files /dev/null and b/tests/TestArchives/Archives/7Zip.LZMA2.exe differ diff --git a/tests/TestArchives/Archives/7Zip.RISCV.7z b/tests/TestArchives/Archives/7Zip.RISCV.7z new file mode 100644 index 00000000..b65c517b Binary files /dev/null and b/tests/TestArchives/Archives/7Zip.RISCV.7z differ diff --git a/tests/TestArchives/Archives/7Zip.encryptedFiles.7z b/tests/TestArchives/Archives/7Zip.encryptedFiles.7z new file mode 100644 index 00000000..4c15bb2c Binary files /dev/null and b/tests/TestArchives/Archives/7Zip.encryptedFiles.7z differ diff --git a/tests/TestArchives/Archives/7Zip.eos.7z b/tests/TestArchives/Archives/7Zip.eos.7z new file mode 100644 index 00000000..2fe951ad Binary files /dev/null and b/tests/TestArchives/Archives/7Zip.eos.7z differ diff --git a/tests/TestArchives/Archives/7Zip.solid.1block.7z b/tests/TestArchives/Archives/7Zip.solid.1block.7z new file mode 100644 index 00000000..410e4f37 Binary files /dev/null and b/tests/TestArchives/Archives/7Zip.solid.1block.7z differ diff --git a/tests/TestArchives/Archives/Ace.encrypted.ace b/tests/TestArchives/Archives/Ace.encrypted.ace new file mode 100644 index 00000000..d0b86b4a Binary files /dev/null and b/tests/TestArchives/Archives/Ace.encrypted.ace differ diff --git a/tests/TestArchives/Archives/Ace.method1-solid.ace b/tests/TestArchives/Archives/Ace.method1-solid.ace new file mode 100644 index 00000000..9cc4145e Binary files /dev/null and b/tests/TestArchives/Archives/Ace.method1-solid.ace differ diff --git a/tests/TestArchives/Archives/Ace.method1.ace b/tests/TestArchives/Archives/Ace.method1.ace new file mode 100644 index 00000000..5a8abc30 Binary files /dev/null and b/tests/TestArchives/Archives/Ace.method1.ace differ diff --git a/tests/TestArchives/Archives/Ace.method2-solid.ace b/tests/TestArchives/Archives/Ace.method2-solid.ace new file mode 100644 index 00000000..1b991acd Binary files /dev/null and b/tests/TestArchives/Archives/Ace.method2-solid.ace differ diff --git a/tests/TestArchives/Archives/Ace.method2.ace b/tests/TestArchives/Archives/Ace.method2.ace new file mode 100644 index 00000000..7d093e97 Binary files /dev/null and b/tests/TestArchives/Archives/Ace.method2.ace differ diff --git a/tests/TestArchives/Archives/Ace.store.ace b/tests/TestArchives/Archives/Ace.store.ace new file mode 100644 index 00000000..8503b872 Binary files /dev/null and b/tests/TestArchives/Archives/Ace.store.ace differ diff --git a/tests/TestArchives/Archives/Ace.store.largefile.ace b/tests/TestArchives/Archives/Ace.store.largefile.ace new file mode 100644 index 00000000..8df2e743 Binary files /dev/null and b/tests/TestArchives/Archives/Ace.store.largefile.ace differ diff --git a/tests/TestArchives/Archives/Ace.store.split.ace b/tests/TestArchives/Archives/Ace.store.split.ace new file mode 100644 index 00000000..df3e5255 Binary files /dev/null and b/tests/TestArchives/Archives/Ace.store.split.ace differ diff --git a/tests/TestArchives/Archives/Ace.store.split.c00 b/tests/TestArchives/Archives/Ace.store.split.c00 new file mode 100644 index 00000000..4f1d89a4 Binary files /dev/null and b/tests/TestArchives/Archives/Ace.store.split.c00 differ diff --git a/tests/TestArchives/Archives/Arc.crunched.arc b/tests/TestArchives/Archives/Arc.crunched.arc new file mode 100644 index 00000000..d4138960 Binary files /dev/null and b/tests/TestArchives/Archives/Arc.crunched.arc differ diff --git a/tests/TestArchives/Archives/Arc.crunched.largefile.arc b/tests/TestArchives/Archives/Arc.crunched.largefile.arc new file mode 100644 index 00000000..e0ace5ab Binary files /dev/null and b/tests/TestArchives/Archives/Arc.crunched.largefile.arc differ diff --git a/tests/TestArchives/Archives/Arc.squashed.arc b/tests/TestArchives/Archives/Arc.squashed.arc new file mode 100644 index 00000000..83f54a49 Binary files /dev/null and b/tests/TestArchives/Archives/Arc.squashed.arc differ diff --git a/tests/TestArchives/Archives/Arc.squashed.largefile.arc b/tests/TestArchives/Archives/Arc.squashed.largefile.arc new file mode 100644 index 00000000..768ac000 Binary files /dev/null and b/tests/TestArchives/Archives/Arc.squashed.largefile.arc differ diff --git a/tests/TestArchives/Archives/Arc.squeezed.arc b/tests/TestArchives/Archives/Arc.squeezed.arc new file mode 100644 index 00000000..cba07e30 Binary files /dev/null and b/tests/TestArchives/Archives/Arc.squeezed.arc differ diff --git a/tests/TestArchives/Archives/Arc.squeezed.largefile.arc b/tests/TestArchives/Archives/Arc.squeezed.largefile.arc new file mode 100644 index 00000000..a4c54f77 Binary files /dev/null and b/tests/TestArchives/Archives/Arc.squeezed.largefile.arc differ diff --git a/tests/TestArchives/Archives/Arc.uncompressed.arc b/tests/TestArchives/Archives/Arc.uncompressed.arc new file mode 100644 index 00000000..78cff2bc Binary files /dev/null and b/tests/TestArchives/Archives/Arc.uncompressed.arc differ diff --git a/tests/TestArchives/Archives/Arc.uncompressed.largefile.arc b/tests/TestArchives/Archives/Arc.uncompressed.largefile.arc new file mode 100644 index 00000000..f8945dc6 Binary files /dev/null and b/tests/TestArchives/Archives/Arc.uncompressed.largefile.arc differ diff --git a/tests/TestArchives/Archives/Arj.encrypted.arj b/tests/TestArchives/Archives/Arj.encrypted.arj new file mode 100644 index 00000000..682c509f Binary files /dev/null and b/tests/TestArchives/Archives/Arj.encrypted.arj differ diff --git a/tests/TestArchives/Archives/Arj.method1.arj b/tests/TestArchives/Archives/Arj.method1.arj new file mode 100644 index 00000000..8dba1434 Binary files /dev/null and b/tests/TestArchives/Archives/Arj.method1.arj differ diff --git a/tests/TestArchives/Archives/Arj.method1.largefile.arj b/tests/TestArchives/Archives/Arj.method1.largefile.arj new file mode 100644 index 00000000..ea0098b4 Binary files /dev/null and b/tests/TestArchives/Archives/Arj.method1.largefile.arj differ diff --git a/tests/TestArchives/Archives/Arj.method2.arj b/tests/TestArchives/Archives/Arj.method2.arj new file mode 100644 index 00000000..0e759b8a Binary files /dev/null and b/tests/TestArchives/Archives/Arj.method2.arj differ diff --git a/tests/TestArchives/Archives/Arj.method2.largefile.arj b/tests/TestArchives/Archives/Arj.method2.largefile.arj new file mode 100644 index 00000000..8ebc2e9c Binary files /dev/null and b/tests/TestArchives/Archives/Arj.method2.largefile.arj differ diff --git a/tests/TestArchives/Archives/Arj.method3.arj b/tests/TestArchives/Archives/Arj.method3.arj new file mode 100644 index 00000000..7969683b Binary files /dev/null and b/tests/TestArchives/Archives/Arj.method3.arj differ diff --git a/tests/TestArchives/Archives/Arj.method3.largefile.arj b/tests/TestArchives/Archives/Arj.method3.largefile.arj new file mode 100644 index 00000000..cb8d57c8 Binary files /dev/null and b/tests/TestArchives/Archives/Arj.method3.largefile.arj differ diff --git a/tests/TestArchives/Archives/Arj.method4.arj b/tests/TestArchives/Archives/Arj.method4.arj new file mode 100644 index 00000000..5f3bbbf6 Binary files /dev/null and b/tests/TestArchives/Archives/Arj.method4.arj differ diff --git a/tests/TestArchives/Archives/Arj.method4.largefile.arj b/tests/TestArchives/Archives/Arj.method4.largefile.arj new file mode 100644 index 00000000..65e49c37 Binary files /dev/null and b/tests/TestArchives/Archives/Arj.method4.largefile.arj differ diff --git a/tests/TestArchives/Archives/Arj.store.arj b/tests/TestArchives/Archives/Arj.store.arj new file mode 100644 index 00000000..b6fa515c Binary files /dev/null and b/tests/TestArchives/Archives/Arj.store.arj differ diff --git a/tests/TestArchives/Archives/Arj.store.largefile.arj b/tests/TestArchives/Archives/Arj.store.largefile.arj new file mode 100644 index 00000000..e39fb32e Binary files /dev/null and b/tests/TestArchives/Archives/Arj.store.largefile.arj differ diff --git a/tests/TestArchives/Archives/Arj.store.split.a01 b/tests/TestArchives/Archives/Arj.store.split.a01 new file mode 100644 index 00000000..a27d4f00 Binary files /dev/null and b/tests/TestArchives/Archives/Arj.store.split.a01 differ diff --git a/tests/TestArchives/Archives/Arj.store.split.a02 b/tests/TestArchives/Archives/Arj.store.split.a02 new file mode 100644 index 00000000..0d40578c Binary files /dev/null and b/tests/TestArchives/Archives/Arj.store.split.a02 differ diff --git a/tests/TestArchives/Archives/Arj.store.split.a03 b/tests/TestArchives/Archives/Arj.store.split.a03 new file mode 100644 index 00000000..15062609 Binary files /dev/null and b/tests/TestArchives/Archives/Arj.store.split.a03 differ diff --git a/tests/TestArchives/Archives/Arj.store.split.a04 b/tests/TestArchives/Archives/Arj.store.split.a04 new file mode 100644 index 00000000..c54f83e0 Binary files /dev/null and b/tests/TestArchives/Archives/Arj.store.split.a04 differ diff --git a/tests/TestArchives/Archives/Arj.store.split.a05 b/tests/TestArchives/Archives/Arj.store.split.a05 new file mode 100644 index 00000000..a75c4ed3 Binary files /dev/null and b/tests/TestArchives/Archives/Arj.store.split.a05 differ diff --git a/tests/TestArchives/Archives/Arj.store.split.arj b/tests/TestArchives/Archives/Arj.store.split.arj new file mode 100644 index 00000000..eb17059f Binary files /dev/null and b/tests/TestArchives/Archives/Arj.store.split.arj differ diff --git a/tests/TestArchives/Archives/Rar.comment.rar b/tests/TestArchives/Archives/Rar.comment.rar new file mode 100644 index 00000000..b0a909c7 Binary files /dev/null and b/tests/TestArchives/Archives/Rar.comment.rar differ diff --git a/tests/TestArchives/Archives/Rar.issue1050.rar b/tests/TestArchives/Archives/Rar.issue1050.rar new file mode 100644 index 00000000..704a0188 Binary files /dev/null and b/tests/TestArchives/Archives/Rar.issue1050.rar differ diff --git a/tests/TestArchives/Archives/Rar.malformed_512byte.rar b/tests/TestArchives/Archives/Rar.malformed_512byte.rar new file mode 100644 index 00000000..914693f8 Binary files /dev/null and b/tests/TestArchives/Archives/Rar.malformed_512byte.rar differ diff --git a/tests/TestArchives/Archives/Rar5.comment.rar b/tests/TestArchives/Archives/Rar5.comment.rar new file mode 100644 index 00000000..d3a65de9 Binary files /dev/null and b/tests/TestArchives/Archives/Rar5.comment.rar differ diff --git a/tests/TestArchives/Archives/Tar.PaxGlobalHeader.Link.tar b/tests/TestArchives/Archives/Tar.PaxGlobalHeader.Link.tar new file mode 100644 index 00000000..acab3275 Binary files /dev/null and b/tests/TestArchives/Archives/Tar.PaxGlobalHeader.Link.tar differ diff --git a/tests/TestArchives/Archives/Tar.PaxGlobalHeader.tar b/tests/TestArchives/Archives/Tar.PaxGlobalHeader.tar new file mode 100644 index 00000000..63c1e6a9 Binary files /dev/null and b/tests/TestArchives/Archives/Tar.PaxGlobalHeader.tar differ diff --git a/tests/TestArchives/Archives/Tar.PaxLocalHeader.Link.tar b/tests/TestArchives/Archives/Tar.PaxLocalHeader.Link.tar new file mode 100644 index 00000000..8b68c57f Binary files /dev/null and b/tests/TestArchives/Archives/Tar.PaxLocalHeader.Link.tar differ diff --git a/tests/TestArchives/Archives/Tar.PaxLocalHeader.tar b/tests/TestArchives/Archives/Tar.PaxLocalHeader.tar new file mode 100644 index 00000000..94ab4b02 Binary files /dev/null and b/tests/TestArchives/Archives/Tar.PaxLocalHeader.tar differ diff --git a/tests/TestArchives/Archives/Tar.oldgnu.tar.gz b/tests/TestArchives/Archives/Tar.oldgnu.tar.gz new file mode 100644 index 00000000..fbffb7a2 Binary files /dev/null and b/tests/TestArchives/Archives/Tar.oldgnu.tar.gz differ diff --git a/tests/TestArchives/Archives/Tar.tar.Z b/tests/TestArchives/Archives/Tar.tar.Z new file mode 100644 index 00000000..0eb90fb1 Binary files /dev/null and b/tests/TestArchives/Archives/Tar.tar.Z differ diff --git a/tests/TestArchives/Archives/Tar.tar.zst b/tests/TestArchives/Archives/Tar.tar.zst new file mode 100644 index 00000000..1443b669 Binary files /dev/null and b/tests/TestArchives/Archives/Tar.tar.zst differ diff --git a/tests/TestArchives/Archives/TarCorrupted.tar b/tests/TestArchives/Archives/TarCorrupted.tar new file mode 100644 index 00000000..a21c4fe5 Binary files /dev/null and b/tests/TestArchives/Archives/TarCorrupted.tar differ diff --git a/tests/TestArchives/Archives/Zip.644.zip b/tests/TestArchives/Archives/Zip.644.zip new file mode 100644 index 00000000..71a786be Binary files /dev/null and b/tests/TestArchives/Archives/Zip.644.zip differ diff --git a/tests/TestArchives/Archives/Zip.EntryComment.zip b/tests/TestArchives/Archives/Zip.EntryComment.zip new file mode 100644 index 00000000..7e4ab419 Binary files /dev/null and b/tests/TestArchives/Archives/Zip.EntryComment.zip differ diff --git a/tests/TestArchives/Archives/Zip.implode.zip b/tests/TestArchives/Archives/Zip.implode.zip new file mode 100644 index 00000000..bd4354f0 Binary files /dev/null and b/tests/TestArchives/Archives/Zip.implode.zip differ diff --git a/tests/TestArchives/Archives/Zip.lzma.empty.zip b/tests/TestArchives/Archives/Zip.lzma.empty.zip new file mode 100644 index 00000000..81c13678 Binary files /dev/null and b/tests/TestArchives/Archives/Zip.lzma.empty.zip differ diff --git a/tests/TestArchives/Archives/Zip.none.encrypted.zip b/tests/TestArchives/Archives/Zip.none.encrypted.zip new file mode 100644 index 00000000..915a2be8 Binary files /dev/null and b/tests/TestArchives/Archives/Zip.none.encrypted.zip differ diff --git a/tests/TestArchives/Archives/Zip.reduce1.zip b/tests/TestArchives/Archives/Zip.reduce1.zip new file mode 100644 index 00000000..61af465b Binary files /dev/null and b/tests/TestArchives/Archives/Zip.reduce1.zip differ diff --git a/tests/TestArchives/Archives/Zip.reduce2.zip b/tests/TestArchives/Archives/Zip.reduce2.zip new file mode 100644 index 00000000..c1bb8fdc Binary files /dev/null and b/tests/TestArchives/Archives/Zip.reduce2.zip differ diff --git a/tests/TestArchives/Archives/Zip.reduce3.zip b/tests/TestArchives/Archives/Zip.reduce3.zip new file mode 100644 index 00000000..41773e57 Binary files /dev/null and b/tests/TestArchives/Archives/Zip.reduce3.zip differ diff --git a/tests/TestArchives/Archives/Zip.reduce4.zip b/tests/TestArchives/Archives/Zip.reduce4.zip new file mode 100644 index 00000000..d21129af Binary files /dev/null and b/tests/TestArchives/Archives/Zip.reduce4.zip differ diff --git a/tests/TestArchives/Archives/Zip.shrink.zip b/tests/TestArchives/Archives/Zip.shrink.zip new file mode 100644 index 00000000..697c7e00 Binary files /dev/null and b/tests/TestArchives/Archives/Zip.shrink.zip differ diff --git a/tests/TestArchives/Archives/Zip.zstd.WinzipAES.mixed.zip b/tests/TestArchives/Archives/Zip.zstd.WinzipAES.mixed.zip new file mode 100644 index 00000000..506e49c9 Binary files /dev/null and b/tests/TestArchives/Archives/Zip.zstd.WinzipAES.mixed.zip differ diff --git a/tests/TestArchives/Archives/bad-1-lzma2-7.xz b/tests/TestArchives/Archives/bad-1-lzma2-7.xz new file mode 100644 index 00000000..8cc711c1 Binary files /dev/null and b/tests/TestArchives/Archives/bad-1-lzma2-7.xz differ diff --git a/tests/TestArchives/Archives/false.positive.tar b/tests/TestArchives/Archives/false.positive.tar new file mode 100644 index 00000000..db91a2ff Binary files /dev/null and b/tests/TestArchives/Archives/false.positive.tar differ diff --git a/tests/TestArchives/Archives/large_test.txt.Z b/tests/TestArchives/Archives/large_test.txt.Z new file mode 100644 index 00000000..64ae2af1 Binary files /dev/null and b/tests/TestArchives/Archives/large_test.txt.Z differ diff --git a/tests/TestArchives/MiscTest/alice29.txt b/tests/TestArchives/MiscTest/alice29.txt new file mode 100644 index 00000000..f1156864 --- /dev/null +++ b/tests/TestArchives/MiscTest/alice29.txt @@ -0,0 +1,3609 @@ + + + + + ALICE'S ADVENTURES IN WONDERLAND + + Lewis Carroll + + THE MILLENNIUM FULCRUM EDITION 2.9 + + + + + CHAPTER I + + Down the Rabbit-Hole + + + Alice was beginning to get very tired of sitting by her sister +on the bank, and of having nothing to do: once or twice she had +peeped into the book her sister was reading, but it had no +pictures or conversations in it, `and what is the use of a book,' +thought Alice `without pictures or conversation?' + + So she was considering in her own mind (as well as she could, +for the hot day made her feel very sleepy and stupid), whether +the pleasure of making a daisy-chain would be worth the trouble +of getting up and picking the daisies, when suddenly a White +Rabbit with pink eyes ran close by her. + + There was nothing so VERY remarkable in that; nor did Alice +think it so VERY much out of the way to hear the Rabbit say to +itself, `Oh dear! Oh dear! I shall be late!' (when she thought +it over afterwards, it occurred to her that she ought to have +wondered at this, but at the time it all seemed quite natural); +but when the Rabbit actually TOOK A WATCH OUT OF ITS WAISTCOAT- +POCKET, and looked at it, and then hurried on, Alice started to +her feet, for it flashed across her mind that she had never +before seen a rabbit with either a waistcoat-pocket, or a watch to +take out of it, and burning with curiosity, she ran across the +field after it, and fortunately was just in time to see it pop +down a large rabbit-hole under the hedge. + + In another moment down went Alice after it, never once +considering how in the world she was to get out again. + + The rabbit-hole went straight on like a tunnel for some way, +and then dipped suddenly down, so suddenly that Alice had not a +moment to think about stopping herself before she found herself +falling down a very deep well. + + Either the well was very deep, or she fell very slowly, for she +had plenty of time as she went down to look about her and to +wonder what was going to happen next. First, she tried to look +down and make out what she was coming to, but it was too dark to +see anything; then she looked at the sides of the well, and +noticed that they were filled with cupboards and book-shelves; +here and there she saw maps and pictures hung upon pegs. She +took down a jar from one of the shelves as she passed; it was +labelled `ORANGE MARMALADE', but to her great disappointment it +was empty: she did not like to drop the jar for fear of killing +somebody, so managed to put it into one of the cupboards as she +fell past it. + + `Well!' thought Alice to herself, `after such a fall as this, I +shall think nothing of tumbling down stairs! How brave they'll +all think me at home! Why, I wouldn't say anything about it, +even if I fell off the top of the house!' (Which was very likely +true.) + + Down, down, down. Would the fall NEVER come to an end! `I +wonder how many miles I've fallen by this time?' she said aloud. +`I must be getting somewhere near the centre of the earth. Let +me see: that would be four thousand miles down, I think--' (for, +you see, Alice had learnt several things of this sort in her +lessons in the schoolroom, and though this was not a VERY good +opportunity for showing off her knowledge, as there was no one to +listen to her, still it was good practice to say it over) `--yes, +that's about the right distance--but then I wonder what Latitude +or Longitude I've got to?' (Alice had no idea what Latitude was, +or Longitude either, but thought they were nice grand words to +say.) + + Presently she began again. `I wonder if I shall fall right +THROUGH the earth! How funny it'll seem to come out among the +people that walk with their heads downward! The Antipathies, I +think--' (she was rather glad there WAS no one listening, this +time, as it didn't sound at all the right word) `--but I shall +have to ask them what the name of the country is, you know. +Please, Ma'am, is this New Zealand or Australia?' (and she tried +to curtsey as she spoke--fancy CURTSEYING as you're falling +through the air! Do you think you could manage it?) `And what +an ignorant little girl she'll think me for asking! No, it'll +never do to ask: perhaps I shall see it written up somewhere.' + + Down, down, down. There was nothing else to do, so Alice soon +began talking again. `Dinah'll miss me very much to-night, I +should think!' (Dinah was the cat.) `I hope they'll remember +her saucer of milk at tea-time. Dinah my dear! I wish you were +down here with me! There are no mice in the air, I'm afraid, but +you might catch a bat, and that's very like a mouse, you know. +But do cats eat bats, I wonder?' And here Alice began to get +rather sleepy, and went on saying to herself, in a dreamy sort of +way, `Do cats eat bats? Do cats eat bats?' and sometimes, `Do +bats eat cats?' for, you see, as she couldn't answer either +question, it didn't much matter which way she put it. She felt +that she was dozing off, and had just begun to dream that she +was walking hand in hand with Dinah, and saying to her very +earnestly, `Now, Dinah, tell me the truth: did you ever eat a +bat?' when suddenly, thump! thump! down she came upon a heap of +sticks and dry leaves, and the fall was over. + + Alice was not a bit hurt, and she jumped up on to her feet in a +moment: she looked up, but it was all dark overhead; before her +was another long passage, and the White Rabbit was still in +sight, hurrying down it. There was not a moment to be lost: +away went Alice like the wind, and was just in time to hear it +say, as it turned a corner, `Oh my ears and whiskers, how late +it's getting!' She was close behind it when she turned the +corner, but the Rabbit was no longer to be seen: she found +herself in a long, low hall, which was lit up by a row of lamps +hanging from the roof. + + There were doors all round the hall, but they were all locked; +and when Alice had been all the way down one side and up the +other, trying every door, she walked sadly down the middle, +wondering how she was ever to get out again. + + Suddenly she came upon a little three-legged table, all made of +solid glass; there was nothing on it except a tiny golden key, +and Alice's first thought was that it might belong to one of the +doors of the hall; but, alas! either the locks were too large, or +the key was too small, but at any rate it would not open any of +them. However, on the second time round, she came upon a low +curtain she had not noticed before, and behind it was a little +door about fifteen inches high: she tried the little golden key +in the lock, and to her great delight it fitted! + + Alice opened the door and found that it led into a small +passage, not much larger than a rat-hole: she knelt down and +looked along the passage into the loveliest garden you ever saw. +How she longed to get out of that dark hall, and wander about +among those beds of bright flowers and those cool fountains, but +she could not even get her head though the doorway; `and even if +my head would go through,' thought poor Alice, `it would be of +very little use without my shoulders. Oh, how I wish +I could shut up like a telescope! I think I could, if I only +know how to begin.' For, you see, so many out-of-the-way things +had happened lately, that Alice had begun to think that very few +things indeed were really impossible. + + There seemed to be no use in waiting by the little door, so she +went back to the table, half hoping she might find another key on +it, or at any rate a book of rules for shutting people up like +telescopes: this time she found a little bottle on it, (`which +certainly was not here before,' said Alice,) and round the neck +of the bottle was a paper label, with the words `DRINK ME' +beautifully printed on it in large letters. + + It was all very well to say `Drink me,' but the wise little +Alice was not going to do THAT in a hurry. `No, I'll look +first,' she said, `and see whether it's marked "poison" or not'; +for she had read several nice little histories about children who +had got burnt, and eaten up by wild beasts and other unpleasant +things, all because they WOULD not remember the simple rules +their friends had taught them: such as, that a red-hot poker +will burn you if you hold it too long; and that if you cut your +finger VERY deeply with a knife, it usually bleeds; and she had +never forgotten that, if you drink much from a bottle marked +`poison,' it is almost certain to disagree with you, sooner or +later. + + However, this bottle was NOT marked `poison,' so Alice ventured +to taste it, and finding it very nice, (it had, in fact, a sort +of mixed flavour of cherry-tart, custard, pine-apple, roast +turkey, toffee, and hot buttered toast,) she very soon finished +it off. + + * * * * * * * + + * * * * * * + + * * * * * * * + + `What a curious feeling!' said Alice; `I must be shutting up +like a telescope.' + + And so it was indeed: she was now only ten inches high, and +her face brightened up at the thought that she was now the right +size for going though the little door into that lovely garden. +First, however, she waited for a few minutes to see if she was +going to shrink any further: she felt a little nervous about +this; `for it might end, you know,' said Alice to herself, `in my +going out altogether, like a candle. I wonder what I should be +like then?' And she tried to fancy what the flame of a candle is +like after the candle is blown out, for she could not remember +ever having seen such a thing. + + After a while, finding that nothing more happened, she decided +on going into the garden at once; but, alas for poor Alice! when +she got to the door, she found he had forgotten the little golden +key, and when she went back to the table for it, she found she +could not possibly reach it: she could see it quite plainly +through the glass, and she tried her best to climb up one of the +legs of the table, but it was too slippery; and when she had +tired herself out with trying, the poor little thing sat down and +cried. + + `Come, there's no use in crying like that!' said Alice to +herself, rather sharply; `I advise you to leave off this minute!' +She generally gave herself very good advice, (though she very +seldom followed it), and sometimes she scolded herself so +severely as to bring tears into her eyes; and once she remembered +trying to box her own ears for having cheated herself in a game +of croquet she was playing against herself, for this curious +child was very fond of pretending to be two people. `But it's no +use now,' thought poor Alice, `to pretend to be two people! Why, +there's hardly enough of me left to make ONE respectable +person!' + + Soon her eye fell on a little glass box that was lying under +the table: she opened it, and found in it a very small cake, on +which the words `EAT ME' were beautifully marked in currants. +`Well, I'll eat it,' said Alice, `and if it makes me grow larger, +I can reach the key; and if it makes me grow smaller, I can creep +under the door; so either way I'll get into the garden, and I +don't care which happens!' + + She ate a little bit, and said anxiously to herself, `Which +way? Which way?', holding her hand on the top of her head to +feel which way it was growing, and she was quite surprised to +find that she remained the same size: to be sure, this generally +happens when one eats cake, but Alice had got so much into the +way of expecting nothing but out-of-the-way things to happen, +that it seemed quite dull and stupid for life to go on in the +common way. + + So she set to work, and very soon finished off the cake. + + * * * * * * * + + * * * * * * + + * * * * * * * + + + + + CHAPTER II + + The Pool of Tears + + + `Curiouser and curiouser!' cried Alice (she was so much +surprised, that for the moment she quite forgot how to speak good +English); `now I'm opening out like the largest telescope that +ever was! Good-bye, feet!' (for when she looked down at her +feet, they seemed to be almost out of sight, they were getting so +far off). `Oh, my poor little feet, I wonder who will put on +your shoes and stockings for you now, dears? I'm sure _I_ shan't +be able! I shall be a great deal too far off to trouble myself +about you: you must manage the best way you can; --but I must be +kind to them,' thought Alice, `or perhaps they won't walk the +way I want to go! Let me see: I'll give them a new pair of +boots every Christmas.' + + And she went on planning to herself how she would manage it. +`They must go by the carrier,' she thought; `and how funny it'll +seem, sending presents to one's own feet! And how odd the +directions will look! + + ALICE'S RIGHT FOOT, ESQ. + HEARTHRUG, + NEAR THE FENDER, + (WITH ALICE'S LOVE). + +Oh dear, what nonsense I'm talking!' + + Just then her head struck against the roof of the hall: in +fact she was now more than nine feet high, and she at once took +up the little golden key and hurried off to the garden door. + + Poor Alice! It was as much as she could do, lying down on one +side, to look through into the garden with one eye; but to get +through was more hopeless than ever: she sat down and began to +cry again. + + `You ought to be ashamed of yourself,' said Alice, `a great +girl like you,' (she might well say this), `to go on crying in +this way! Stop this moment, I tell you!' But she went on all +the same, shedding gallons of tears, until there was a large pool +all round her, about four inches deep and reaching half down the +hall. + + After a time she heard a little pattering of feet in the +distance, and she hastily dried her eyes to see what was coming. +It was the White Rabbit returning, splendidly dressed, with a +pair of white kid gloves in one hand and a large fan in the +other: he came trotting along in a great hurry, muttering to +himself as he came, `Oh! the Duchess, the Duchess! Oh! won't she +be savage if I've kept her waiting!' Alice felt so desperate +that she was ready to ask help of any one; so, when the Rabbit +came near her, she began, in a low, timid voice, `If you please, +sir--' The Rabbit started violently, dropped the white kid +gloves and the fan, and skurried away into the darkness as hard +as he could go. + + Alice took up the fan and gloves, and, as the hall was very +hot, she kept fanning herself all the time she went on talking: +`Dear, dear! How queer everything is to-day! And yesterday +things went on just as usual. I wonder if I've been changed in +the night? Let me think: was I the same when I got up this +morning? I almost think I can remember feeling a little +different. But if I'm not the same, the next question is, Who in +the world am I? Ah, THAT'S the great puzzle!' And she began +thinking over all the children she knew that were of the same age +as herself, to see if she could have been changed for any of +them. + + `I'm sure I'm not Ada,' she said, `for her hair goes in such +long ringlets, and mine doesn't go in ringlets at all; and I'm +sure I can't be Mabel, for I know all sorts of things, and she, +oh! she knows such a very little! Besides, SHE'S she, and I'm I, +and--oh dear, how puzzling it all is! I'll try if I know all the +things I used to know. Let me see: four times five is twelve, +and four times six is thirteen, and four times seven is--oh dear! +I shall never get to twenty at that rate! However, the +Multiplication Table doesn't signify: let's try Geography. +London is the capital of Paris, and Paris is the capital of Rome, +and Rome--no, THAT'S all wrong, I'm certain! I must have been +changed for Mabel! I'll try and say "How doth the little--"' +and she crossed her hands on her lap as if she were saying lessons, +and began to repeat it, but her voice sounded hoarse and +strange, and the words did not come the same as they used to do:-- + + `How doth the little crocodile + Improve his shining tail, + And pour the waters of the Nile + On every golden scale! + + `How cheerfully he seems to grin, + How neatly spread his claws, + And welcome little fishes in + With gently smiling jaws!' + + `I'm sure those are not the right words,' said poor Alice, and +her eyes filled with tears again as she went on, `I must be Mabel +after all, and I shall have to go and live in that poky little +house, and have next to no toys to play with, and oh! ever so +many lessons to learn! No, I've made up my mind about it; if I'm +Mabel, I'll stay down here! It'll be no use their putting their +heads down and saying "Come up again, dear!" I shall only look +up and say "Who am I then? Tell me that first, and then, if I +like being that person, I'll come up: if not, I'll stay down +here till I'm somebody else"--but, oh dear!' cried Alice, with a +sudden burst of tears, `I do wish they WOULD put their heads +down! I am so VERY tired of being all alone here!' + + As she said this she looked down at her hands, and was +surprised to see that she had put on one of the Rabbit's little +white kid gloves while she was talking. `How CAN I have done +that?' she thought. `I must be growing small again.' She got up +and went to the table to measure herself by it, and found that, +as nearly as she could guess, she was now about two feet high, +and was going on shrinking rapidly: she soon found out that the +cause of this was the fan she was holding, and she dropped it +hastily, just in time to avoid shrinking away altogether. + +`That WAS a narrow escape!' said Alice, a good deal frightened at +the sudden change, but very glad to find herself still in +existence; `and now for the garden!' and she ran with all speed +back to the little door: but, alas! the little door was shut +again, and the little golden key was lying on the glass table as +before, `and things are worse than ever,' thought the poor child, +`for I never was so small as this before, never! And I declare +it's too bad, that it is!' + + As she said these words her foot slipped, and in another +moment, splash! she was up to her chin in salt water. He first +idea was that she had somehow fallen into the sea, `and in that +case I can go back by railway,' she said to herself. (Alice had +been to the seaside once in her life, and had come to the general +conclusion, that wherever you go to on the English coast you find +a number of bathing machines in the sea, some children digging in +the sand with wooden spades, then a row of lodging houses, and +behind them a railway station.) However, she soon made out that +she was in the pool of tears which she had wept when she was nine +feet high. + + `I wish I hadn't cried so much!' said Alice, as she swam about, +trying to find her way out. `I shall be punished for it now, I +suppose, by being drowned in my own tears! That WILL be a queer +thing, to be sure! However, everything is queer to-day.' + + Just then she heard something splashing about in the pool a +little way off, and she swam nearer to make out what it was: at +first she thought it must be a walrus or hippopotamus, but then +she remembered how small she was now, and she soon made out that +it was only a mouse that had slipped in like herself. + + `Would it be of any use, now,' thought Alice, `to speak to this +mouse? Everything is so out-of-the-way down here, that I should +think very likely it can talk: at any rate, there's no harm in +trying.' So she began: `O Mouse, do you know the way out of +this pool? I am very tired of swimming about here, O Mouse!' +(Alice thought this must be the right way of speaking to a mouse: +she had never done such a thing before, but she remembered having +seen in her brother's Latin Grammar, `A mouse--of a mouse--to a +mouse--a mouse--O mouse!' The Mouse looked at her rather +inquisitively, and seemed to her to wink with one of its little +eyes, but it said nothing. + + `Perhaps it doesn't understand English,' thought Alice; `I +daresay it's a French mouse, come over with William the +Conqueror.' (For, with all her knowledge of history, Alice had +no very clear notion how long ago anything had happened.) So she +began again: `Ou est ma chatte?' which was the first sentence in +her French lesson-book. The Mouse gave a sudden leap out of the +water, and seemed to quiver all over with fright. `Oh, I beg +your pardon!' cried Alice hastily, afraid that she had hurt the +poor animal's feelings. `I quite forgot you didn't like cats.' + + `Not like cats!' cried the Mouse, in a shrill, passionate +voice. `Would YOU like cats if you were me?' + + `Well, perhaps not,' said Alice in a soothing tone: `don't be +angry about it. And yet I wish I could show you our cat Dinah: +I think you'd take a fancy to cats if you could only see her. +She is such a dear quiet thing,' Alice went on, half to herself, +as she swam lazily about in the pool, `and she sits purring so +nicely by the fire, licking her paws and washing her face--and +she is such a nice soft thing to nurse--and she's such a capital +one for catching mice--oh, I beg your pardon!' cried Alice again, +for this time the Mouse was bristling all over, and she felt +certain it must be really offended. `We won't talk about her any +more if you'd rather not.' + + `We indeed!' cried the Mouse, who was trembling down to the end +of his tail. `As if I would talk on such a subject! Our family +always HATED cats: nasty, low, vulgar things! Don't let me hear +the name again!' + + `I won't indeed!' said Alice, in a great hurry to change the +subject of conversation. `Are you--are you fond--of--of dogs?' +The Mouse did not answer, so Alice went on eagerly: `There is +such a nice little dog near our house I should like to show you! +A little bright-eyed terrier, you know, with oh, such long curly +brown hair! And it'll fetch things when you throw them, and +it'll sit up and beg for its dinner, and all sorts of things--I +can't remember half of them--and it belongs to a farmer, you +know, and he says it's so useful, it's worth a hundred pounds! +He says it kills all the rats and--oh dear!' cried Alice in a +sorrowful tone, `I'm afraid I've offended it again!' For the +Mouse was swimming away from her as hard as it could go, and +making quite a commotion in the pool as it went. + + So she called softly after it, `Mouse dear! Do come back +again, and we won't talk about cats or dogs either, if you don't +like them!' When the Mouse heard this, it turned round and swam +slowly back to her: its face was quite pale (with passion, Alice +thought), and it said in a low trembling voice, `Let us get to +the shore, and then I'll tell you my history, and you'll +understand why it is I hate cats and dogs.' + + It was high time to go, for the pool was getting quite crowded +with the birds and animals that had fallen into it: there were a +Duck and a Dodo, a Lory and an Eaglet, and several other curious +creatures. Alice led the way, and the whole party swam to the +shore. + + + + CHAPTER III + + A Caucus-Race and a Long Tale + + + They were indeed a queer-looking party that assembled on the +bank--the birds with draggled feathers, the animals with their +fur clinging close to them, and all dripping wet, cross, and +uncomfortable. + + The first question of course was, how to get dry again: they +had a consultation about this, and after a few minutes it seemed +quite natural to Alice to find herself talking familiarly with +them, as if she had known them all her life. Indeed, she had +quite a long argument with the Lory, who at last turned sulky, +and would only say, `I am older than you, and must know better'; +and this Alice would not allow without knowing how old it was, +and, as the Lory positively refused to tell its age, there was no +more to be said. + + At last the Mouse, who seemed to be a person of authority among +them, called out, `Sit down, all of you, and listen to me! I'LL +soon make you dry enough!' They all sat down at once, in a large +ring, with the Mouse in the middle. Alice kept her eyes +anxiously fixed on it, for she felt sure she would catch a bad +cold if she did not get dry very soon. + + `Ahem!' said the Mouse with an important air, `are you all ready? +This is the driest thing I know. Silence all round, if you please! +"William the Conqueror, whose cause was favoured by the pope, was +soon submitted to by the English, who wanted leaders, and had been +of late much accustomed to usurpation and conquest. Edwin and +Morcar, the earls of Mercia and Northumbria--"' + + `Ugh!' said the Lory, with a shiver. + + `I beg your pardon!' said the Mouse, frowning, but very +politely: `Did you speak?' + + `Not I!' said the Lory hastily. + + `I thought you did,' said the Mouse. `--I proceed. "Edwin and +Morcar, the earls of Mercia and Northumbria, declared for him: +and even Stigand, the patriotic archbishop of Canterbury, found +it advisable--"' + + `Found WHAT?' said the Duck. + + `Found IT,' the Mouse replied rather crossly: `of course you +know what "it" means.' + + `I know what "it" means well enough, when I find a thing,' said +the Duck: `it's generally a frog or a worm. The question is, +what did the archbishop find?' + + The Mouse did not notice this question, but hurriedly went on, +`"--found it advisable to go with Edgar Atheling to meet William +and offer him the crown. William's conduct at first was +moderate. But the insolence of his Normans--" How are you +getting on now, my dear?' it continued, turning to Alice as it +spoke. + + `As wet as ever,' said Alice in a melancholy tone: `it doesn't +seem to dry me at all.' + + `In that case,' said the Dodo solemnly, rising to its feet, `I +move that the meeting adjourn, for the immediate adoption of more +energetic remedies--' + + `Speak English!' said the Eaglet. `I don't know the meaning of +half those long words, and, what's more, I don't believe you do +either!' And the Eaglet bent down its head to hide a smile: +some of the other birds tittered audibly. + + `What I was going to say,' said the Dodo in an offended tone, +`was, that the best thing to get us dry would be a Caucus-race.' + + `What IS a Caucus-race?' said Alice; not that she wanted much +to know, but the Dodo had paused as if it thought that SOMEBODY +ought to speak, and no one else seemed inclined to say anything. + + `Why,' said the Dodo, `the best way to explain it is to do it.' +(And, as you might like to try the thing yourself, some winter +day, I will tell you how the Dodo managed it.) + + First it marked out a race-course, in a sort of circle, (`the +exact shape doesn't matter,' it said,) and then all the party +were placed along the course, here and there. There was no `One, +two, three, and away,' but they began running when they liked, +and left off when they liked, so that it was not easy to know +when the race was over. However, when they had been running half +an hour or so, and were quite dry again, the Dodo suddenly called +out `The race is over!' and they all crowded round it, panting, +and asking, `But who has won?' + + This question the Dodo could not answer without a great deal of +thought, and it sat for a long time with one finger pressed upon +its forehead (the position in which you usually see Shakespeare, +in the pictures of him), while the rest waited in silence. At +last the Dodo said, `EVERYBODY has won, and all must have +prizes.' + + `But who is to give the prizes?' quite a chorus of voices +asked. + + `Why, SHE, of course,' said the Dodo, pointing to Alice with +one finger; and the whole party at once crowded round her, +calling out in a confused way, `Prizes! Prizes!' + + Alice had no idea what to do, and in despair she put her hand +in her pocket, and pulled out a box of comfits, (luckily the salt +water had not got into it), and handed them round as prizes. +There was exactly one a-piece all round. + + `But she must have a prize herself, you know,' said the Mouse. + + `Of course,' the Dodo replied very gravely. `What else have +you got in your pocket?' he went on, turning to Alice. + + `Only a thimble,' said Alice sadly. + + `Hand it over here,' said the Dodo. + + Then they all crowded round her once more, while the Dodo +solemnly presented the thimble, saying `We beg your acceptance of +this elegant thimble'; and, when it had finished this short +speech, they all cheered. + + Alice thought the whole thing very absurd, but they all looked +so grave that she did not dare to laugh; and, as she could not +think of anything to say, she simply bowed, and took the thimble, +looking as solemn as she could. + + The next thing was to eat the comfits: this caused some noise +and confusion, as the large birds complained that they could not +taste theirs, and the small ones choked and had to be patted on +the back. However, it was over at last, and they sat down again +in a ring, and begged the Mouse to tell them something more. + + `You promised to tell me your history, you know,' said Alice, +`and why it is you hate--C and D,' she added in a whisper, half +afraid that it would be offended again. + + `Mine is a long and a sad tale!' said the Mouse, turning to +Alice, and sighing. + + `It IS a long tail, certainly,' said Alice, looking down with +wonder at the Mouse's tail; `but why do you call it sad?' And +she kept on puzzling about it while the Mouse was speaking, so +that her idea of the tale was something like this:-- + + `Fury said to a + mouse, That he + met in the + house, + "Let us + both go to + law: I will + prosecute + YOU. --Come, + I'll take no + denial; We + must have a + trial: For + really this + morning I've + nothing + to do." + Said the + mouse to the + cur, "Such + a trial, + dear Sir, + With + no jury + or judge, + would be + wasting + our + breath." + "I'll be + judge, I'll + be jury," + Said + cunning + old Fury: + "I'll + try the + whole + cause, + and + condemn + you + to + death."' + + + `You are not attending!' said the Mouse to Alice severely. +`What are you thinking of?' + + `I beg your pardon,' said Alice very humbly: `you had got to +the fifth bend, I think?' + + `I had NOT!' cried the Mouse, sharply and very angrily. + + `A knot!' said Alice, always ready to make herself useful, and +looking anxiously about her. `Oh, do let me help to undo it!' + + `I shall do nothing of the sort,' said the Mouse, getting up +and walking away. `You insult me by talking such nonsense!' + + `I didn't mean it!' pleaded poor Alice. `But you're so easily +offended, you know!' + + The Mouse only growled in reply. + + `Please come back and finish your story!' Alice called after +it; and the others all joined in chorus, `Yes, please do!' but +the Mouse only shook its head impatiently, and walked a little +quicker. + + `What a pity it wouldn't stay!' sighed the Lory, as soon as it +was quite out of sight; and an old Crab took the opportunity of +saying to her daughter `Ah, my dear! Let this be a lesson to you +never to lose YOUR temper!' `Hold your tongue, Ma!' said the +young Crab, a little snappishly. `You're enough to try the +patience of an oyster!' + + `I wish I had our Dinah here, I know I do!' said Alice aloud, +addressing nobody in particular. `She'd soon fetch it back!' + + `And who is Dinah, if I might venture to ask the question?' +said the Lory. + + Alice replied eagerly, for she was always ready to talk about +her pet: `Dinah's our cat. And she's such a capital one for +catching mice you can't think! And oh, I wish you could see her +after the birds! Why, she'll eat a little bird as soon as look +at it!' + + This speech caused a remarkable sensation among the party. +Some of the birds hurried off at once: one the old Magpie began +wrapping itself up very carefully, remarking, `I really must be +getting home; the night-air doesn't suit my throat!' and a Canary +called out in a trembling voice to its children, `Come away, my +dears! It's high time you were all in bed!' On various pretexts +they all moved off, and Alice was soon left alone. + + `I wish I hadn't mentioned Dinah!' she said to herself in a +melancholy tone. `Nobody seems to like her, down here, and I'm +sure she's the best cat in the world! Oh, my dear Dinah! I +wonder if I shall ever see you any more!' And here poor Alice +began to cry again, for she felt very lonely and low-spirited. +In a little while, however, she again heard a little pattering of +footsteps in the distance, and she looked up eagerly, half hoping +that the Mouse had changed his mind, and was coming back to +finish his story. + + + + CHAPTER IV + + The Rabbit Sends in a Little Bill + + + It was the White Rabbit, trotting slowly back again, and +looking anxiously about as it went, as if it had lost something; +and she heard it muttering to itself `The Duchess! The Duchess! +Oh my dear paws! Oh my fur and whiskers! She'll get me +executed, as sure as ferrets are ferrets! Where CAN I have +dropped them, I wonder?' Alice guessed in a moment that it was +looking for the fan and the pair of white kid gloves, and she +very good-naturedly began hunting about for them, but they were +nowhere to be seen--everything seemed to have changed since her +swim in the pool, and the great hall, with the glass table and +the little door, had vanished completely. + + Very soon the Rabbit noticed Alice, as she went hunting about, +and called out to her in an angry tone, `Why, Mary Ann, what ARE +you doing out here? Run home this moment, and fetch me a pair of +gloves and a fan! Quick, now!' And Alice was so much frightened +that she ran off at once in the direction it pointed to, without +trying to explain the mistake it had made. + + `He took me for his housemaid,' she said to herself as she ran. +`How surprised he'll be when he finds out who I am! But I'd +better take him his fan and gloves--that is, if I can find them.' +As she said this, she came upon a neat little house, on the door +of which was a bright brass plate with the name `W. RABBIT' +engraved upon it. She went in without knocking, and hurried +upstairs, in great fear lest she should meet the real Mary Ann, +and be turned out of the house before she had found the fan and +gloves. + + `How queer it seems,' Alice said to herself, `to be going +messages for a rabbit! I suppose Dinah'll be sending me on +messages next!' And she began fancying the sort of thing that +would happen: `"Miss Alice! Come here directly, and get ready +for your walk!" "Coming in a minute, nurse! But I've got to see +that the mouse doesn't get out." Only I don't think,' Alice went +on, `that they'd let Dinah stop in the house if it began ordering +people about like that!' + + By this time she had found her way into a tidy little room with +a table in the window, and on it (as she had hoped) a fan and two +or three pairs of tiny white kid gloves: she took up the fan and +a pair of the gloves, and was just going to leave the room, when +her eye fell upon a little bottle that stood near the looking- +glass. There was no label this time with the words `DRINK ME,' +but nevertheless she uncorked it and put it to her lips. `I know +SOMETHING interesting is sure to happen,' she said to herself, +`whenever I eat or drink anything; so I'll just see what this +bottle does. I do hope it'll make me grow large again, for +really I'm quite tired of being such a tiny little thing!' + + It did so indeed, and much sooner than she had expected: +before she had drunk half the bottle, she found her head pressing +against the ceiling, and had to stoop to save her neck from being +broken. She hastily put down the bottle, saying to herself +`That's quite enough--I hope I shan't grow any more--As it is, I +can't get out at the door--I do wish I hadn't drunk quite so +much!' + + Alas! it was too late to wish that! She went on growing, and +growing, and very soon had to kneel down on the floor: in +another minute there was not even room for this, and she tried +the effect of lying down with one elbow against the door, and the +other arm curled round her head. Still she went on growing, and, +as a last resource, she put one arm out of the window, and one +foot up the chimney, and said to herself `Now I can do no more, +whatever happens. What WILL become of me?' + + Luckily for Alice, the little magic bottle had now had its full +effect, and she grew no larger: still it was very uncomfortable, +and, as there seemed to be no sort of chance of her ever getting +out of the room again, no wonder she felt unhappy. + + `It was much pleasanter at home,' thought poor Alice, `when one +wasn't always growing larger and smaller, and being ordered about +by mice and rabbits. I almost wish I hadn't gone down that +rabbit-hole--and yet--and yet--it's rather curious, you know, +this sort of life! I do wonder what CAN have happened to me! +When I used to read fairy-tales, I fancied that kind of thing +never happened, and now here I am in the middle of one! There +ought to be a book written about me, that there ought! And when +I grow up, I'll write one--but I'm grown up now,' she added in a +sorrowful tone; `at least there's no room to grow up any more +HERE.' + + `But then,' thought Alice, `shall I NEVER get any older than I +am now? That'll be a comfort, one way--never to be an old woman- +-but then--always to have lessons to learn! Oh, I shouldn't like +THAT!' + + `Oh, you foolish Alice!' she answered herself. `How can you +learn lessons in here? Why, there's hardly room for YOU, and no +room at all for any lesson-books!' + + And so she went on, taking first one side and then the other, +and making quite a conversation of it altogether; but after a few +minutes she heard a voice outside, and stopped to listen. + + `Mary Ann! Mary Ann!' said the voice. `Fetch me my gloves +this moment!' Then came a little pattering of feet on the +stairs. Alice knew it was the Rabbit coming to look for her, and +she trembled till she shook the house, quite forgetting that she +was now about a thousand times as large as the Rabbit, and had no +reason to be afraid of it. + + Presently the Rabbit came up to the door, and tried to open it; +but, as the door opened inwards, and Alice's elbow was pressed +hard against it, that attempt proved a failure. Alice heard it +say to itself `Then I'll go round and get in at the window.' + + `THAT you won't' thought Alice, and, after waiting till she +fancied she heard the Rabbit just under the window, she suddenly +spread out her hand, and made a snatch in the air. She did not +get hold of anything, but she heard a little shriek and a fall, +and a crash of broken glass, from which she concluded that it was +just possible it had fallen into a cucumber-frame, or something +of the sort. + + Next came an angry voice--the Rabbit's--`Pat! Pat! Where are +you?' And then a voice she had never heard before, `Sure then +I'm here! Digging for apples, yer honour!' + + `Digging for apples, indeed!' said the Rabbit angrily. `Here! +Come and help me out of THIS!' (Sounds of more broken glass.) + + `Now tell me, Pat, what's that in the window?' + + `Sure, it's an arm, yer honour!' (He pronounced it `arrum.') + + `An arm, you goose! Who ever saw one that size? Why, it +fills the whole window!' + + `Sure, it does, yer honour: but it's an arm for all that.' + + `Well, it's got no business there, at any rate: go and take it +away!' + + There was a long silence after this, and Alice could only hear +whispers now and then; such as, `Sure, I don't like it, yer +honour, at all, at all!' `Do as I tell you, you coward!' and at +last she spread out her hand again, and made another snatch in +the air. This time there were TWO little shrieks, and more +sounds of broken glass. `What a number of cucumber-frames there +must be!' thought Alice. `I wonder what they'll do next! As for +pulling me out of the window, I only wish they COULD! I'm sure I +don't want to stay in here any longer!' + + She waited for some time without hearing anything more: at +last came a rumbling of little cartwheels, and the sound of a +good many voice all talking together: she made out the words: +`Where's the other ladder?--Why, I hadn't to bring but one; +Bill's got the other--Bill! fetch it here, lad!--Here, put 'em up +at this corner--No, tie 'em together first--they don't reach half +high enough yet--Oh! they'll do well enough; don't be particular- +-Here, Bill! catch hold of this rope--Will the roof bear?--Mind +that loose slate--Oh, it's coming down! Heads below!' (a loud +crash)--`Now, who did that?--It was Bill, I fancy--Who's to go +down the chimney?--Nay, I shan't! YOU do it!--That I won't, +then!--Bill's to go down--Here, Bill! the master says you're to +go down the chimney!' + + `Oh! So Bill's got to come down the chimney, has he?' said +Alice to herself. `Shy, they seem to put everything upon Bill! +I wouldn't be in Bill's place for a good deal: this fireplace is +narrow, to be sure; but I THINK I can kick a little!' + + She drew her foot as far down the chimney as she could, and +waited till she heard a little animal (she couldn't guess of what +sort it was) scratching and scrambling about in the chimney close +above her: then, saying to herself `This is Bill,' she gave one +sharp kick, and waited to see what would happen next. + + The first thing she heard was a general chorus of `There goes +Bill!' then the Rabbit's voice along--`Catch him, you by the +hedge!' then silence, and then another confusion of voices--`Hold +up his head--Brandy now--Don't choke him--How was it, old fellow? +What happened to you? Tell us all about it!' + + Last came a little feeble, squeaking voice, (`That's Bill,' +thought Alice,) `Well, I hardly know--No more, thank ye; I'm +better now--but I'm a deal too flustered to tell you--all I know +is, something comes at me like a Jack-in-the-box, and up I goes +like a sky-rocket!' + + `So you did, old fellow!' said the others. + + `We must burn the house down!' said the Rabbit's voice; and +Alice called out as loud as she could, `If you do. I'll set +Dinah at you!' + + There was a dead silence instantly, and Alice thought to +herself, `I wonder what they WILL do next! If they had any +sense, they'd take the roof off.' After a minute or two, they +began moving about again, and Alice heard the Rabbit say, `A +barrowful will do, to begin with.' + + `A barrowful of WHAT?' thought Alice; but she had not long to +doubt, for the next moment a shower of little pebbles came +rattling in at the window, and some of them hit her in the face. +`I'll put a stop to this,' she said to herself, and shouted out, +`You'd better not do that again!' which produced another dead +silence. + + Alice noticed with some surprise that the pebbles were all +turning into little cakes as they lay on the floor, and a bright +idea came into her head. `If I eat one of these cakes,' she +thought, `it's sure to make SOME change in my size; and as it +can't possibly make me larger, it must make me smaller, I +suppose.' + + So she swallowed one of the cakes, and was delighted to find +that she began shrinking directly. As soon as she was small +enough to get through the door, she ran out of the house, and +found quite a crowd of little animals and birds waiting outside. +The poor little Lizard, Bill, was in the middle, being held up by +two guinea-pigs, who were giving it something out of a bottle. +They all made a rush at Alice the moment she appeared; but she +ran off as hard as she could, and soon found herself safe in a +thick wood. + + `The first thing I've got to do,' said Alice to herself, as she +wandered about in the wood, `is to grow to my right size again; +and the second thing is to find my way into that lovely garden. +I think that will be the best plan.' + + It sounded an excellent plan, no doubt, and very neatly and +simply arranged; the only difficulty was, that she had not the +smallest idea how to set about it; and while she was peering +about anxiously among the trees, a little sharp bark just over +her head made her look up in a great hurry. + + An enormous puppy was looking down at her with large round +eyes, and feebly stretching out one paw, trying to touch her. +`Poor little thing!' said Alice, in a coaxing tone, and she tried +hard to whistle to it; but she was terribly frightened all the +time at the thought that it might be hungry, in which case it +would be very likely to eat her up in spite of all her coaxing. + + Hardly knowing what she did, she picked up a little bit of +stick, and held it out to the puppy; whereupon the puppy jumped +into the air off all its feet at once, with a yelp of delight, +and rushed at the stick, and made believe to worry it; then Alice +dodged behind a great thistle, to keep herself from being run +over; and the moment she appeared on the other side, the puppy +made another rush at the stick, and tumbled head over heels in +its hurry to get hold of it; then Alice, thinking it was very +like having a game of play with a cart-horse, and expecting every +moment to be trampled under its feet, ran round the thistle +again; then the puppy began a series of short charges at the +stick, running a very little way forwards each time and a long +way back, and barking hoarsely all the while, till at last it sat +down a good way off, panting, with its tongue hanging out of its +mouth, and its great eyes half shut. + + This seemed to Alice a good opportunity for making her escape; +so she set off at once, and ran till she was quite tired and out +of breath, and till the puppy's bark sounded quite faint in the +distance. + + `And yet what a dear little puppy it was!' said Alice, as she +leant against a buttercup to rest herself, and fanned herself +with one of the leaves: `I should have liked teaching it tricks +very much, if--if I'd only been the right size to do it! Oh +dear! I'd nearly forgotten that I've got to grow up again! Let +me see--how IS it to be managed? I suppose I ought to eat or +drink something or other; but the great question is, what?' + + The great question certainly was, what? Alice looked all round +her at the flowers and the blades of grass, but she did not see +anything that looked like the right thing to eat or drink under +the circumstances. There was a large mushroom growing near her, +about the same height as herself; and when she had looked under +it, and on both sides of it, and behind it, it occurred to her +that she might as well look and see what was on the top of it. + + She stretched herself up on tiptoe, and peeped over the edge of +the mushroom, and her eyes immediately met those of a large +caterpillar, that was sitting on the top with its arms folded, +quietly smoking a long hookah, and taking not the smallest notice +of her or of anything else. + + + + CHAPTER V + + Advice from a Caterpillar + + + The Caterpillar and Alice looked at each other for some time in +silence: at last the Caterpillar took the hookah out of its +mouth, and addressed her in a languid, sleepy voice. + + `Who are YOU?' said the Caterpillar. + + This was not an encouraging opening for a conversation. Alice +replied, rather shyly, `I--I hardly know, sir, just at present-- +at least I know who I WAS when I got up this morning, but I think +I must have been changed several times since then.' + + `What do you mean by that?' said the Caterpillar sternly. +`Explain yourself!' + + `I can't explain MYSELF, I'm afraid, sir' said Alice, `because +I'm not myself, you see.' + + `I don't see,' said the Caterpillar. + + `I'm afraid I can't put it more clearly,' Alice replied very +politely, `for I can't understand it myself to begin with; and +being so many different sizes in a day is very confusing.' + + `It isn't,' said the Caterpillar. + + `Well, perhaps you haven't found it so yet,' said Alice; `but +when you have to turn into a chrysalis--you will some day, you +know--and then after that into a butterfly, I should think you'll +feel it a little queer, won't you?' + + `Not a bit,' said the Caterpillar. + + `Well, perhaps your feelings may be different,' said Alice; +`all I know is, it would feel very queer to ME.' + + `You!' said the Caterpillar contemptuously. `Who are YOU?' + + Which brought them back again to the beginning of the +conversation. Alice felt a little irritated at the Caterpillar's +making such VERY short remarks, and she drew herself up and said, +very gravely, `I think, you ought to tell me who YOU are, first.' + + `Why?' said the Caterpillar. + + Here was another puzzling question; and as Alice could not +think of any good reason, and as the Caterpillar seemed to be in +a VERY unpleasant state of mind, she turned away. + + `Come back!' the Caterpillar called after her. `I've something +important to say!' + + This sounded promising, certainly: Alice turned and came back +again. + + `Keep your temper,' said the Caterpillar. + + `Is that all?' said Alice, swallowing down her anger as well as +she could. + + `No,' said the Caterpillar. + + Alice thought she might as well wait, as she had nothing else +to do, and perhaps after all it might tell her something worth +hearing. For some minutes it puffed away without speaking, but +at last it unfolded its arms, took the hookah out of its mouth +again, and said, `So you think you're changed, do you?' + + `I'm afraid I am, sir,' said Alice; `I can't remember things as +I used--and I don't keep the same size for ten minutes together!' + + `Can't remember WHAT things?' said the Caterpillar. + + `Well, I've tried to say "HOW DOTH THE LITTLE BUSY BEE," but it +all came different!' Alice replied in a very melancholy voice. + + `Repeat, "YOU ARE OLD, FATHER WILLIAM,"' said the Caterpillar. + + Alice folded her hands, and began:-- + + `You are old, Father William,' the young man said, + `And your hair has become very white; + And yet you incessantly stand on your head-- + Do you think, at your age, it is right?' + + `In my youth,' Father William replied to his son, + `I feared it might injure the brain; + But, now that I'm perfectly sure I have none, + Why, I do it again and again.' + + `You are old,' said the youth, `as I mentioned before, + And have grown most uncommonly fat; + Yet you turned a back-somersault in at the door-- + Pray, what is the reason of that?' + + `In my youth,' said the sage, as he shook his grey locks, + `I kept all my limbs very supple + By the use of this ointment--one shilling the box-- + Allow me to sell you a couple?' + + `You are old,' said the youth, `and your jaws are too weak + For anything tougher than suet; + Yet you finished the goose, with the bones and the beak-- + Pray how did you manage to do it?' + + `In my youth,' said his father, `I took to the law, + And argued each case with my wife; + And the muscular strength, which it gave to my jaw, + Has lasted the rest of my life.' + + `You are old,' said the youth, `one would hardly suppose + That your eye was as steady as ever; + Yet you balanced an eel on the end of your nose-- + What made you so awfully clever?' + + `I have answered three questions, and that is enough,' + Said his father; `don't give yourself airs! + Do you think I can listen all day to such stuff? + Be off, or I'll kick you down stairs!' + + + `That is not said right,' said the Caterpillar. + + `Not QUITE right, I'm afraid,' said Alice, timidly; `some of the +words have got altered.' + + `It is wrong from beginning to end,' said the Caterpillar +decidedly, and there was silence for some minutes. + + The Caterpillar was the first to speak. + + `What size do you want to be?' it asked. + + `Oh, I'm not particular as to size,' Alice hastily replied; +`only one doesn't like changing so often, you know.' + + `I DON'T know,' said the Caterpillar. + + Alice said nothing: she had never been so much contradicted in +her life before, and she felt that she was losing her temper. + + `Are you content now?' said the Caterpillar. + + `Well, I should like to be a LITTLE larger, sir, if you +wouldn't mind,' said Alice: `three inches is such a wretched +height to be.' + + `It is a very good height indeed!' said the Caterpillar +angrily, rearing itself upright as it spoke (it was exactly three +inches high). + + `But I'm not used to it!' pleaded poor Alice in a piteous tone. +And she thought of herself, `I wish the creatures wouldn't be so +easily offended!' + + `You'll get used to it in time,' said the Caterpillar; and it +put the hookah into its mouth and began smoking again. + + This time Alice waited patiently until it chose to speak again. +In a minute or two the Caterpillar took the hookah out of its +mouth and yawned once or twice, and shook itself. Then it got +down off the mushroom, and crawled away in the grass, merely +remarking as it went, `One side will make you grow taller, and +the other side will make you grow shorter.' + + `One side of WHAT? The other side of WHAT?' thought Alice to +herself. + + `Of the mushroom,' said the Caterpillar, just as if she had +asked it aloud; and in another moment it was out of sight. + + Alice remained looking thoughtfully at the mushroom for a +minute, trying to make out which were the two sides of it; and as +it was perfectly round, she found this a very difficult question. +However, at last she stretched her arms round it as far as they +would go, and broke off a bit of the edge with each hand. + + `And now which is which?' she said to herself, and nibbled a +little of the right-hand bit to try the effect: the next moment +she felt a violent blow underneath her chin: it had struck her +foot! + + She was a good deal frightened by this very sudden change, but +she felt that there was no time to be lost, as she was shrinking +rapidly; so she set to work at once to eat some of the other bit. +Her chin was pressed so closely against her foot, that there was +hardly room to open her mouth; but she did it at last, and +managed to swallow a morsel of the lefthand bit. + + + * * * * * * * + + * * * * * * + + * * * * * * * + + `Come, my head's free at last!' said Alice in a tone of +delight, which changed into alarm in another moment, when she +found that her shoulders were nowhere to be found: all she could +see, when she looked down, was an immense length of neck, which +seemed to rise like a stalk out of a sea of green leaves that lay +far below her. + + `What CAN all that green stuff be?' said Alice. `And where +HAVE my shoulders got to? And oh, my poor hands, how is it I +can't see you?' She was moving them about as she spoke, but no +result seemed to follow, except a little shaking among the +distant green leaves. + + As there seemed to be no chance of getting her hands up to her +head, she tried to get her head down to them, and was delighted +to find that her neck would bend about easily in any direction, +like a serpent. She had just succeeded in curving it down into a +graceful zigzag, and was going to dive in among the leaves, which +she found to be nothing but the tops of the trees under which she +had been wandering, when a sharp hiss made her draw back in a +hurry: a large pigeon had flown into her face, and was beating +her violently with its wings. + + `Serpent!' screamed the Pigeon. + + `I'm NOT a serpent!' said Alice indignantly. `Let me alone!' + + `Serpent, I say again!' repeated the Pigeon, but in a more +subdued tone, and added with a kind of sob, `I've tried every +way, and nothing seems to suit them!' + + `I haven't the least idea what you're talking about,' said +Alice. + + `I've tried the roots of trees, and I've tried banks, and I've +tried hedges,' the Pigeon went on, without attending to her; `but +those serpents! There's no pleasing them!' + + Alice was more and more puzzled, but she thought there was no +use in saying anything more till the Pigeon had finished. + + `As if it wasn't trouble enough hatching the eggs,' said the +Pigeon; `but I must be on the look-out for serpents night and +day! Why, I haven't had a wink of sleep these three weeks!' + + `I'm very sorry you've been annoyed,' said Alice, who was +beginning to see its meaning. + + `And just as I'd taken the highest tree in the wood,' continued +the Pigeon, raising its voice to a shriek, `and just as I was +thinking I should be free of them at last, they must needs come +wriggling down from the sky! Ugh, Serpent!' + + `But I'm NOT a serpent, I tell you!' said Alice. `I'm a--I'm +a--' + + `Well! WHAT are you?' said the Pigeon. `I can see you're +trying to invent something!' + + `I--I'm a little girl,' said Alice, rather doubtfully, as she +remembered the number of changes she had gone through that day. + + `A likely story indeed!' said the Pigeon in a tone of the +deepest contempt. `I've seen a good many little girls in my +time, but never ONE with such a neck as that! No, no! You're a +serpent; and there's no use denying it. I suppose you'll be +telling me next that you never tasted an egg!' + + `I HAVE tasted eggs, certainly,' said Alice, who was a very +truthful child; `but little girls eat eggs quite as much as +serpents do, you know.' + + `I don't believe it,' said the Pigeon; `but if they do, why +then they're a kind of serpent, that's all I can say.' + + This was such a new idea to Alice, that she was quite silent +for a minute or two, which gave the Pigeon the opportunity of +adding, `You're looking for eggs, I know THAT well enough; and +what does it matter to me whether you're a little girl or a +serpent?' + + `It matters a good deal to ME,' said Alice hastily; `but I'm +not looking for eggs, as it happens; and if I was, I shouldn't +want YOURS: I don't like them raw.' + + `Well, be off, then!' said the Pigeon in a sulky tone, as it +settled down again into its nest. Alice crouched down among the +trees as well as she could, for her neck kept getting entangled +among the branches, and every now and then she had to stop and +untwist it. After a while she remembered that she still held the +pieces of mushroom in her hands, and she set to work very +carefully, nibbling first at one and then at the other, and +growing sometimes taller and sometimes shorter, until she had +succeeded in bringing herself down to her usual height. + + It was so long since she had been anything near the right size, +that it felt quite strange at first; but she got used to it in a +few minutes, and began talking to herself, as usual. `Come, +there's half my plan done now! How puzzling all these changes +are! I'm never sure what I'm going to be, from one minute to +another! However, I've got back to my right size: the next +thing is, to get into that beautiful garden--how IS that to be +done, I wonder?' As she said this, she came suddenly upon an +open place, with a little house in it about four feet high. +`Whoever lives there,' thought Alice, `it'll never do to come +upon them THIS size: why, I should frighten them out of their +wits!' So she began nibbling at the righthand bit again, and did +not venture to go near the house till she had brought herself +down to nine inches high. + + + + CHAPTER VI + + Pig and Pepper + + + For a minute or two she stood looking at the house, and +wondering what to do next, when suddenly a footman in livery came +running out of the wood--(she considered him to be a footman +because he was in livery: otherwise, judging by his face only, +she would have called him a fish)--and rapped loudly at the door +with his knuckles. It was opened by another footman in livery, +with a round face, and large eyes like a frog; and both footmen, +Alice noticed, had powdered hair that curled all over their +heads. She felt very curious to know what it was all about, and +crept a little way out of the wood to listen. + + The Fish-Footman began by producing from under his arm a great +letter, nearly as large as himself, and this he handed over to +the other, saying, in a solemn tone, `For the Duchess. An +invitation from the Queen to play croquet.' The Frog-Footman +repeated, in the same solemn tone, only changing the order of the +words a little, `From the Queen. An invitation for the Duchess +to play croquet.' + + Then they both bowed low, and their curls got entangled +together. + + Alice laughed so much at this, that she had to run back into +the wood for fear of their hearing her; and when she next peeped +out the Fish-Footman was gone, and the other was sitting on the +ground near the door, staring stupidly up into the sky. + + Alice went timidly up to the door, and knocked. + + `There's no sort of use in knocking,' said the Footman, `and +that for two reasons. First, because I'm on the same side of the +door as you are; secondly, because they're making such a noise +inside, no one could possibly hear you.' And certainly there was +a most extraordinary noise going on within--a constant howling +and sneezing, and every now and then a great crash, as if a dish +or kettle had been broken to pieces. + + `Please, then,' said Alice, `how am I to get in?' + + `There might be some sense in your knocking,' the Footman went +on without attending to her, `if we had the door between us. For +instance, if you were INSIDE, you might knock, and I could let +you out, you know.' He was looking up into the sky all the time +he was speaking, and this Alice thought decidedly uncivil. `But +perhaps he can't help it,' she said to herself; `his eyes are so +VERY nearly at the top of his head. But at any rate he might +answer questions.--How am I to get in?' she repeated, aloud. + + `I shall sit here,' the Footman remarked, `till tomorrow--' + + At this moment the door of the house opened, and a large plate +came skimming out, straight at the Footman's head: it just +grazed his nose, and broke to pieces against one of the trees +behind him. + + `--or next day, maybe,' the Footman continued in the same tone, +exactly as if nothing had happened. + + `How am I to get in?' asked Alice again, in a louder tone. + + `ARE you to get in at all?' said the Footman. `That's the +first question, you know.' + + It was, no doubt: only Alice did not like to be told so. +`It's really dreadful,' she muttered to herself, `the way all the +creatures argue. It's enough to drive one crazy!' + + The Footman seemed to think this a good opportunity for +repeating his remark, with variations. `I shall sit here,' he +said, `on and off, for days and days.' + + `But what am I to do?' said Alice. + + `Anything you like,' said the Footman, and began whistling. + + `Oh, there's no use in talking to him,' said Alice desperately: +`he's perfectly idiotic!' And she opened the door and went in. + + The door led right into a large kitchen, which was full of +smoke from one end to the other: the Duchess was sitting on a +three-legged stool in the middle, nursing a baby; the cook was +leaning over the fire, stirring a large cauldron which seemed to +be full of soup. + + `There's certainly too much pepper in that soup!' Alice said to +herself, as well as she could for sneezing. + + There was certainly too much of it in the air. Even the +Duchess sneezed occasionally; and as for the baby, it was +sneezing and howling alternately without a moment's pause. The +only things in the kitchen that did not sneeze, were the cook, +and a large cat which was sitting on the hearth and grinning from +ear to ear. + + `Please would you tell me,' said Alice, a little timidly, for +she was not quite sure whether it was good manners for her to +speak first, `why your cat grins like that?' + + `It's a Cheshire cat,' said the Duchess, `and that's why. +Pig!' + + She said the last word with such sudden violence that Alice +quite jumped; but she saw in another moment that it was addressed +to the baby, and not to her, so she took courage, and went on +again:-- + + `I didn't know that Cheshire cats always grinned; in fact, I +didn't know that cats COULD grin.' + + `They all can,' said the Duchess; `and most of 'em do.' + + `I don't know of any that do,' Alice said very politely, +feeling quite pleased to have got into a conversation. + + `You don't know much,' said the Duchess; `and that's a fact.' + + Alice did not at all like the tone of this remark, and thought +it would be as well to introduce some other subject of +conversation. While she was trying to fix on one, the cook took +the cauldron of soup off the fire, and at once set to work +throwing everything within her reach at the Duchess and the baby +--the fire-irons came first; then followed a shower of saucepans, +plates, and dishes. The Duchess took no notice of them even when +they hit her; and the baby was howling so much already, that it +was quite impossible to say whether the blows hurt it or not. + + `Oh, PLEASE mind what you're doing!' cried Alice, jumping up +and down in an agony of terror. `Oh, there goes his PRECIOUS +nose'; as an unusually large saucepan flew close by it, and very +nearly carried it off. + + `If everybody minded their own business,' the Duchess said in a +hoarse growl, `the world would go round a deal faster than it +does.' + + `Which would NOT be an advantage,' said Alice, who felt very +glad to get an opportunity of showing off a little of her +knowledge. `Just think of what work it would make with the day +and night! You see the earth takes twenty-four hours to turn +round on its axis--' + + `Talking of axes,' said the Duchess, `chop off her head!' + + Alice glanced rather anxiously at the cook, to see if she meant +to take the hint; but the cook was busily stirring the soup, and +seemed not to be listening, so she went on again: `Twenty-four +hours, I THINK; or is it twelve? I--' + + `Oh, don't bother ME,' said the Duchess; `I never could abide +figures!' And with that she began nursing her child again, +singing a sort of lullaby to it as she did so, and giving it a +violent shake at the end of every line: + + `Speak roughly to your little boy, + And beat him when he sneezes: + He only does it to annoy, + Because he knows it teases.' + + CHORUS. + + (In which the cook and the baby joined):-- + + `Wow! wow! wow!' + + While the Duchess sang the second verse of the song, she kept +tossing the baby violently up and down, and the poor little thing +howled so, that Alice could hardly hear the words:-- + + `I speak severely to my boy, + I beat him when he sneezes; + For he can thoroughly enjoy + The pepper when he pleases!' + + CHORUS. + + `Wow! wow! wow!' + + `Here! you may nurse it a bit, if you like!' the Duchess said +to Alice, flinging the baby at her as she spoke. `I must go and +get ready to play croquet with the Queen,' and she hurried out of +the room. The cook threw a frying-pan after her as she went out, +but it just missed her. + + Alice caught the baby with some difficulty, as it was a queer- +shaped little creature, and held out its arms and legs in all +directions, `just like a star-fish,' thought Alice. The poor +little thing was snorting like a steam-engine when she caught it, +and kept doubling itself up and straightening itself out again, +so that altogether, for the first minute or two, it was as much +as she could do to hold it. + + As soon as she had made out the proper way of nursing it, +(which was to twist it up into a sort of knot, and then keep +tight hold of its right ear and left foot, so as to prevent its +undoing itself,) she carried it out into the open air. `IF I +don't take this child away with me,' thought Alice, `they're sure +to kill it in a day or two: wouldn't it be murder to leave it +behind?' She said the last words out loud, and the little thing +grunted in reply (it had left off sneezing by this time). `Don't +grunt,' said Alice; `that's not at all a proper way of expressing +yourself.' + + The baby grunted again, and Alice looked very anxiously into +its face to see what was the matter with it. There could be no +doubt that it had a VERY turn-up nose, much more like a snout +than a real nose; also its eyes were getting extremely small for +a baby: altogether Alice did not like the look of the thing at +all. `But perhaps it was only sobbing,' she thought, and looked +into its eyes again, to see if there were any tears. + + No, there were no tears. `If you're going to turn into a pig, +my dear,' said Alice, seriously, `I'll have nothing more to do +with you. Mind now!' The poor little thing sobbed again (or +grunted, it was impossible to say which), and they went on for +some while in silence. + + Alice was just beginning to think to herself, `Now, what am I +to do with this creature when I get it home?' when it grunted +again, so violently, that she looked down into its face in some +alarm. This time there could be NO mistake about it: it was +neither more nor less than a pig, and she felt that it would be +quite absurd for her to carry it further. + + So she set the little creature down, and felt quite relieved to +see it trot away quietly into the wood. `If it had grown up,' +she said to herself, `it would have made a dreadfully ugly child: +but it makes rather a handsome pig, I think.' And she began +thinking over other children she knew, who might do very well as +pigs, and was just saying to herself, `if one only knew the right +way to change them--' when she was a little startled by seeing +the Cheshire Cat sitting on a bough of a tree a few yards off. + + The Cat only grinned when it saw Alice. It looked good- +natured, she thought: still it had VERY long claws and a great +many teeth, so she felt that it ought to be treated with respect. + + `Cheshire Puss,' she began, rather timidly, as she did not at +all know whether it would like the name: however, it only +grinned a little wider. `Come, it's pleased so far,' thought +Alice, and she went on. `Would you tell me, please, which way I +ought to go from here?' + + `That depends a good deal on where you want to get to,' said +the Cat. + + `I don't much care where--' said Alice. + + `Then it doesn't matter which way you go,' said the Cat. + + `--so long as I get SOMEWHERE,' Alice added as an explanation. + + `Oh, you're sure to do that,' said the Cat, `if you only walk +long enough.' + + Alice felt that this could not be denied, so she tried another +question. `What sort of people live about here?' + + `In THAT direction,' the Cat said, waving its right paw round, +`lives a Hatter: and in THAT direction,' waving the other paw, +`lives a March Hare. Visit either you like: they're both mad.' + + `But I don't want to go among mad people,' Alice remarked. + + `Oh, you can't help that,' said the Cat: `we're all mad here. +I'm mad. You're mad.' + + `How do you know I'm mad?' said Alice. + + `You must be,' said the Cat, `or you wouldn't have come here.' + + Alice didn't think that proved it at all; however, she went on +`And how do you know that you're mad?' + + `To begin with,' said the Cat, `a dog's not mad. You grant +that?' + + `I suppose so,' said Alice. + + `Well, then,' the Cat went on, `you see, a dog growls when it's +angry, and wags its tail when it's pleased. Now I growl when I'm +pleased, and wag my tail when I'm angry. Therefore I'm mad.' + + `I call it purring, not growling,' said Alice. + + `Call it what you like,' said the Cat. `Do you play croquet +with the Queen to-day?' + + `I should like it very much,' said Alice, `but I haven't been +invited yet.' + + `You'll see me there,' said the Cat, and vanished. + + Alice was not much surprised at this, she was getting so used +to queer things happening. While she was looking at the place +where it had been, it suddenly appeared again. + + `By-the-bye, what became of the baby?' said the Cat. `I'd +nearly forgotten to ask.' + + `It turned into a pig,' Alice quietly said, just as if it had +come back in a natural way. + + `I thought it would,' said the Cat, and vanished again. + + Alice waited a little, half expecting to see it again, but it +did not appear, and after a minute or two she walked on in the +direction in which the March Hare was said to live. `I've seen +hatters before,' she said to herself; `the March Hare will be +much the most interesting, and perhaps as this is May it won't be +raving mad--at least not so mad as it was in March.' As she said +this, she looked up, and there was the Cat again, sitting on a +branch of a tree. + + `Did you say pig, or fig?' said the Cat. + + `I said pig,' replied Alice; `and I wish you wouldn't keep +appearing and vanishing so suddenly: you make one quite giddy.' + + `All right,' said the Cat; and this time it vanished quite +slowly, beginning with the end of the tail, and ending with the +grin, which remained some time after the rest of it had gone. + + `Well! I've often seen a cat without a grin,' thought Alice; +`but a grin without a cat! It's the most curious thing I ever +say in my life!' + + She had not gone much farther before she came in sight of the +house of the March Hare: she thought it must be the right house, +because the chimneys were shaped like ears and the roof was +thatched with fur. It was so large a house, that she did not +like to go nearer till she had nibbled some more of the lefthand +bit of mushroom, and raised herself to about two feet high: even +then she walked up towards it rather timidly, saying to herself +`Suppose it should be raving mad after all! I almost wish I'd +gone to see the Hatter instead!' + + + + CHAPTER VII + + A Mad Tea-Party + + + There was a table set out under a tree in front of the house, +and the March Hare and the Hatter were having tea at it: a +Dormouse was sitting between them, fast asleep, and the other two +were using it as a cushion, resting their elbows on it, and the +talking over its head. `Very uncomfortable for the Dormouse,' +thought Alice; `only, as it's asleep, I suppose it doesn't mind.' + + The table was a large one, but the three were all crowded +together at one corner of it: `No room! No room!' they cried +out when they saw Alice coming. `There's PLENTY of room!' said +Alice indignantly, and she sat down in a large arm-chair at one +end of the table. + + `Have some wine,' the March Hare said in an encouraging tone. + + Alice looked all round the table, but there was nothing on it +but tea. `I don't see any wine,' she remarked. + + `There isn't any,' said the March Hare. + + `Then it wasn't very civil of you to offer it,' said Alice +angrily. + + `It wasn't very civil of you to sit down without being +invited,' said the March Hare. + + `I didn't know it was YOUR table,' said Alice; `it's laid for a +great many more than three.' + + `Your hair wants cutting,' said the Hatter. He had been +looking at Alice for some time with great curiosity, and this was +his first speech. + + `You should learn not to make personal remarks,' Alice said +with some severity; `it's very rude.' + + The Hatter opened his eyes very wide on hearing this; but all +he SAID was, `Why is a raven like a writing-desk?' + + `Come, we shall have some fun now!' thought Alice. `I'm glad +they've begun asking riddles.--I believe I can guess that,' she +added aloud. + + `Do you mean that you think you can find out the answer to it?' +said the March Hare. + + `Exactly so,' said Alice. + + `Then you should say what you mean,' the March Hare went on. + + `I do,' Alice hastily replied; `at least--at least I mean what +I say--that's the same thing, you know.' + + `Not the same thing a bit!' said the Hatter. `You might just +as well say that "I see what I eat" is the same thing as "I eat +what I see"!' + + `You might just as well say,' added the March Hare, `that "I +like what I get" is the same thing as "I get what I like"!' + + `You might just as well say,' added the Dormouse, who seemed to +be talking in his sleep, `that "I breathe when I sleep" is the +same thing as "I sleep when I breathe"!' + + `It IS the same thing with you,' said the Hatter, and here the +conversation dropped, and the party sat silent for a minute, +while Alice thought over all she could remember about ravens and +writing-desks, which wasn't much. + + The Hatter was the first to break the silence. `What day of +the month is it?' he said, turning to Alice: he had taken his +watch out of his pocket, and was looking at it uneasily, shaking +it every now and then, and holding it to his ear. + + Alice considered a little, and then said `The fourth.' + + `Two days wrong!' sighed the Hatter. `I told you butter +wouldn't suit the works!' he added looking angrily at the March +Hare. + + `It was the BEST butter,' the March Hare meekly replied. + + `Yes, but some crumbs must have got in as well,' the Hatter +grumbled: `you shouldn't have put it in with the bread-knife.' + + The March Hare took the watch and looked at it gloomily: then +he dipped it into his cup of tea, and looked at it again: but he +could think of nothing better to say than his first remark, `It +was the BEST butter, you know.' + + Alice had been looking over his shoulder with some curiosity. +`What a funny watch!' she remarked. `It tells the day of the +month, and doesn't tell what o'clock it is!' + + `Why should it?' muttered the Hatter. `Does YOUR watch tell +you what year it is?' + + `Of course not,' Alice replied very readily: `but that's +because it stays the same year for such a long time together.' + + `Which is just the case with MINE,' said the Hatter. + + Alice felt dreadfully puzzled. The Hatter's remark seemed to +have no sort of meaning in it, and yet it was certainly English. +`I don't quite understand you,' she said, as politely as she +could. + + `The Dormouse is asleep again,' said the Hatter, and he poured +a little hot tea upon its nose. + + The Dormouse shook its head impatiently, and said, without +opening its eyes, `Of course, of course; just what I was going to +remark myself.' + + `Have you guessed the riddle yet?' the Hatter said, turning to +Alice again. + + `No, I give it up,' Alice replied: `what's the answer?' + + `I haven't the slightest idea,' said the Hatter. + + `Nor I,' said the March Hare. + + Alice sighed wearily. `I think you might do something better +with the time,' she said, `than waste it in asking riddles that +have no answers.' + + `If you knew Time as well as I do,' said the Hatter, `you +wouldn't talk about wasting IT. It's HIM.' + + `I don't know what you mean,' said Alice. + + `Of course you don't!' the Hatter said, tossing his head +contemptuously. `I dare say you never even spoke to Time!' + + `Perhaps not,' Alice cautiously replied: `but I know I have to +beat time when I learn music.' + + `Ah! that accounts for it,' said the Hatter. `He won't stand +beating. Now, if you only kept on good terms with him, he'd do +almost anything you liked with the clock. For instance, suppose +it were nine o'clock in the morning, just time to begin lessons: +you'd only have to whisper a hint to Time, and round goes the +clock in a twinkling! Half-past one, time for dinner!' + + (`I only wish it was,' the March Hare said to itself in a +whisper.) + + `That would be grand, certainly,' said Alice thoughtfully: +`but then--I shouldn't be hungry for it, you know.' + + `Not at first, perhaps,' said the Hatter: `but you could keep +it to half-past one as long as you liked.' + + `Is that the way YOU manage?' Alice asked. + + The Hatter shook his head mournfully. `Not I!' he replied. +`We quarrelled last March--just before HE went mad, you know--' +(pointing with his tea spoon at the March Hare,) `--it was at the +great concert given by the Queen of Hearts, and I had to sing + + "Twinkle, twinkle, little bat! + How I wonder what you're at!" + +You know the song, perhaps?' + + `I've heard something like it,' said Alice. + + `It goes on, you know,' the Hatter continued, `in this way:-- + + "Up above the world you fly, + Like a tea-tray in the sky. + Twinkle, twinkle--"' + +Here the Dormouse shook itself, and began singing in its sleep +`Twinkle, twinkle, twinkle, twinkle--' and went on so long that +they had to pinch it to make it stop. + + `Well, I'd hardly finished the first verse,' said the Hatter, +`when the Queen jumped up and bawled out, "He's murdering the +time! Off with his head!"' + + `How dreadfully savage!' exclaimed Alice. + + `And ever since that,' the Hatter went on in a mournful tone, +`he won't do a thing I ask! It's always six o'clock now.' + + A bright idea came into Alice's head. `Is that the reason so +many tea-things are put out here?' she asked. + + `Yes, that's it,' said the Hatter with a sigh: `it's always +tea-time, and we've no time to wash the things between whiles.' + + `Then you keep moving round, I suppose?' said Alice. + + `Exactly so,' said the Hatter: `as the things get used up.' + + `But what happens when you come to the beginning again?' Alice +ventured to ask. + + `Suppose we change the subject,' the March Hare interrupted, +yawning. `I'm getting tired of this. I vote the young lady +tells us a story.' + + `I'm afraid I don't know one,' said Alice, rather alarmed at +the proposal. + + `Then the Dormouse shall!' they both cried. `Wake up, +Dormouse!' And they pinched it on both sides at once. + + The Dormouse slowly opened his eyes. `I wasn't asleep,' he +said in a hoarse, feeble voice: `I heard every word you fellows +were saying.' + + `Tell us a story!' said the March Hare. + + `Yes, please do!' pleaded Alice. + + `And be quick about it,' added the Hatter, `or you'll be asleep +again before it's done.' + + `Once upon a time there were three little sisters,' the +Dormouse began in a great hurry; `and their names were Elsie, +Lacie, and Tillie; and they lived at the bottom of a well--' + + `What did they live on?' said Alice, who always took a great +interest in questions of eating and drinking. + + `They lived on treacle,' said the Dormouse, after thinking a +minute or two. + + `They couldn't have done that, you know,' Alice gently +remarked; `they'd have been ill.' + + `So they were,' said the Dormouse; `VERY ill.' + + Alice tried to fancy to herself what such an extraordinary ways +of living would be like, but it puzzled her too much, so she went +on: `But why did they live at the bottom of a well?' + + `Take some more tea,' the March Hare said to Alice, very +earnestly. + + `I've had nothing yet,' Alice replied in an offended tone, `so +I can't take more.' + + `You mean you can't take LESS,' said the Hatter: `it's very +easy to take MORE than nothing.' + + `Nobody asked YOUR opinion,' said Alice. + + `Who's making personal remarks now?' the Hatter asked +triumphantly. + + Alice did not quite know what to say to this: so she helped +herself to some tea and bread-and-butter, and then turned to the +Dormouse, and repeated her question. `Why did they live at the +bottom of a well?' + + The Dormouse again took a minute or two to think about it, and +then said, `It was a treacle-well.' + + `There's no such thing!' Alice was beginning very angrily, but +the Hatter and the March Hare went `Sh! sh!' and the Dormouse +sulkily remarked, `If you can't be civil, you'd better finish the +story for yourself.' + + `No, please go on!' Alice said very humbly; `I won't interrupt +again. I dare say there may be ONE.' + + `One, indeed!' said the Dormouse indignantly. However, he +consented to go on. `And so these three little sisters--they +were learning to draw, you know--' + + `What did they draw?' said Alice, quite forgetting her promise. + + `Treacle,' said the Dormouse, without considering at all this +time. + + `I want a clean cup,' interrupted the Hatter: `let's all move +one place on.' + + He moved on as he spoke, and the Dormouse followed him: the +March Hare moved into the Dormouse's place, and Alice rather +unwillingly took the place of the March Hare. The Hatter was the +only one who got any advantage from the change: and Alice was a +good deal worse off than before, as the March Hare had just upset +the milk-jug into his plate. + + Alice did not wish to offend the Dormouse again, so she began +very cautiously: `But I don't understand. Where did they draw +the treacle from?' + + `You can draw water out of a water-well,' said the Hatter; `so +I should think you could draw treacle out of a treacle-well--eh, +stupid?' + + `But they were IN the well,' Alice said to the Dormouse, not +choosing to notice this last remark. + + `Of course they were', said the Dormouse; `--well in.' + + This answer so confused poor Alice, that she let the Dormouse +go on for some time without interrupting it. + + `They were learning to draw,' the Dormouse went on, yawning and +rubbing its eyes, for it was getting very sleepy; `and they drew +all manner of things--everything that begins with an M--' + + `Why with an M?' said Alice. + + `Why not?' said the March Hare. + + Alice was silent. + + The Dormouse had closed its eyes by this time, and was going +off into a doze; but, on being pinched by the Hatter, it woke up +again with a little shriek, and went on: `--that begins with an +M, such as mouse-traps, and the moon, and memory, and muchness-- +you know you say things are "much of a muchness"--did you ever +see such a thing as a drawing of a muchness?' + + `Really, now you ask me,' said Alice, very much confused, `I +don't think--' + + `Then you shouldn't talk,' said the Hatter. + + This piece of rudeness was more than Alice could bear: she got +up in great disgust, and walked off; the Dormouse fell asleep +instantly, and neither of the others took the least notice of her +going, though she looked back once or twice, half hoping that +they would call after her: the last time she saw them, they were +trying to put the Dormouse into the teapot. + + `At any rate I'll never go THERE again!' said Alice as she +picked her way through the wood. `It's the stupidest tea-party I +ever was at in all my life!' + + Just as she said this, she noticed that one of the trees had a +door leading right into it. `That's very curious!' she thought. +`But everything's curious today. I think I may as well go in at +once.' And in she went. + + Once more she found herself in the long hall, and close to the +little glass table. `Now, I'll manage better this time,' she +said to herself, and began by taking the little golden key, and +unlocking the door that led into the garden. Then she went to +work nibbling at the mushroom (she had kept a piece of it in her +pocked) till she was about a foot high: then she walked down the +little passage: and THEN--she found herself at last in the +beautiful garden, among the bright flower-beds and the cool +fountains. + + + + CHAPTER VIII + + The Queen's Croquet-Ground + + + A large rose-tree stood near the entrance of the garden: the +roses growing on it were white, but there were three gardeners at +it, busily painting them red. Alice thought this a very curious +thing, and she went nearer to watch them, and just as she came up +to them she heard one of them say, `Look out now, Five! Don't go +splashing paint over me like that!' + + `I couldn't help it,' said Five, in a sulky tone; `Seven jogged +my elbow.' + + On which Seven looked up and said, `That's right, Five! Always +lay the blame on others!' + + `YOU'D better not talk!' said Five. `I heard the Queen say only +yesterday you deserved to be beheaded!' + + `What for?' said the one who had spoken first. + + `That's none of YOUR business, Two!' said Seven. + + `Yes, it IS his business!' said Five, `and I'll tell him--it +was for bringing the cook tulip-roots instead of onions.' + + Seven flung down his brush, and had just begun `Well, of all +the unjust things--' when his eye chanced to fall upon Alice, as +she stood watching them, and he checked himself suddenly: the +others looked round also, and all of them bowed low. + + `Would you tell me,' said Alice, a little timidly, `why you are +painting those roses?' + + Five and Seven said nothing, but looked at Two. Two began in a +low voice, `Why the fact is, you see, Miss, this here ought to +have been a RED rose-tree, and we put a white one in by mistake; +and if the Queen was to find it out, we should all have our heads +cut off, you know. So you see, Miss, we're doing our best, afore +she comes, to--' At this moment Five, who had been anxiously +looking across the garden, called out `The Queen! The Queen!' +and the three gardeners instantly threw themselves flat upon +their faces. There was a sound of many footsteps, and Alice +looked round, eager to see the Queen. + + First came ten soldiers carrying clubs; these were all shaped +like the three gardeners, oblong and flat, with their hands and +feet at the corners: next the ten courtiers; these were +ornamented all over with diamonds, and walked two and two, as the +soldiers did. After these came the royal children; there were +ten of them, and the little dears came jumping merrily along hand +in hand, in couples: they were all ornamented with hearts. Next +came the guests, mostly Kings and Queens, and among them Alice +recognised the White Rabbit: it was talking in a hurried nervous +manner, smiling at everything that was said, and went by without +noticing her. Then followed the Knave of Hearts, carrying the +King's crown on a crimson velvet cushion; and, last of all this +grand procession, came THE KING AND QUEEN OF HEARTS. + + Alice was rather doubtful whether she ought not to lie down on +her face like the three gardeners, but she could not remember +every having heard of such a rule at processions; `and besides, +what would be the use of a procession,' thought she, `if people +had all to lie down upon their faces, so that they couldn't see +it?' So she stood still where she was, and waited. + + When the procession came opposite to Alice, they all stopped +and looked at her, and the Queen said severely `Who is this?' +She said it to the Knave of Hearts, who only bowed and smiled in +reply. + + `Idiot!' said the Queen, tossing her head impatiently; and, +turning to Alice, she went on, `What's your name, child?' + + `My name is Alice, so please your Majesty,' said Alice very +politely; but she added, to herself, `Why, they're only a pack of +cards, after all. I needn't be afraid of them!' + + `And who are THESE?' said the Queen, pointing to the three +gardeners who were lying round the rosetree; for, you see, as +they were lying on their faces, and the pattern on their backs +was the same as the rest of the pack, she could not tell whether +they were gardeners, or soldiers, or courtiers, or three of her +own children. + + `How should I know?' said Alice, surprised at her own courage. +`It's no business of MINE.' + + The Queen turned crimson with fury, and, after glaring at her +for a moment like a wild beast, screamed `Off with her head! +Off--' + + `Nonsense!' said Alice, very loudly and decidedly, and the +Queen was silent. + + The King laid his hand upon her arm, and timidly said +`Consider, my dear: she is only a child!' + + The Queen turned angrily away from him, and said to the Knave +`Turn them over!' + + The Knave did so, very carefully, with one foot. + + `Get up!' said the Queen, in a shrill, loud voice, and the +three gardeners instantly jumped up, and began bowing to the +King, the Queen, the royal children, and everybody else. + + `Leave off that!' screamed the Queen. `You make me giddy.' +And then, turning to the rose-tree, she went on, `What HAVE you +been doing here?' + + `May it please your Majesty,' said Two, in a very humble tone, +going down on one knee as he spoke, `we were trying--' + + `I see!' said the Queen, who had meanwhile been examining the +roses. `Off with their heads!' and the procession moved on, +three of the soldiers remaining behind to execute the unfortunate +gardeners, who ran to Alice for protection. + + `You shan't be beheaded!' said Alice, and she put them into a +large flower-pot that stood near. The three soldiers wandered +about for a minute or two, looking for them, and then quietly +marched off after the others. + + `Are their heads off?' shouted the Queen. + + `Their heads are gone, if it please your Majesty!' the soldiers +shouted in reply. + + `That's right!' shouted the Queen. `Can you play croquet?' + + The soldiers were silent, and looked at Alice, as the question +was evidently meant for her. + + `Yes!' shouted Alice. + + `Come on, then!' roared the Queen, and Alice joined the +procession, wondering very much what would happen next. + + `It's--it's a very fine day!' said a timid voice at her side. +She was walking by the White Rabbit, who was peeping anxiously +into her face. + + `Very,' said Alice: `--where's the Duchess?' + + `Hush! Hush!' said the Rabbit in a low, hurried tone. He +looked anxiously over his shoulder as he spoke, and then raised +himself upon tiptoe, put his mouth close to her ear, and +whispered `She's under sentence of execution.' + + `What for?' said Alice. + + `Did you say "What a pity!"?' the Rabbit asked. + + `No, I didn't,' said Alice: `I don't think it's at all a pity. +I said "What for?"' + + `She boxed the Queen's ears--' the Rabbit began. Alice gave a +little scream of laughter. `Oh, hush!' the Rabbit whispered in a +frightened tone. `The Queen will hear you! You see, she came +rather late, and the Queen said--' + + `Get to your places!' shouted the Queen in a voice of thunder, +and people began running about in all directions, tumbling up +against each other; however, they got settled down in a minute or +two, and the game began. Alice thought she had never seen such a +curious croquet-ground in her life; it was all ridges and +furrows; the balls were live hedgehogs, the mallets live +flamingoes, and the soldiers had to double themselves up and to +stand on their hands and feet, to make the arches. + + The chief difficulty Alice found at first was in managing her +flamingo: she succeeded in getting its body tucked away, +comfortably enough, under her arm, with its legs hanging down, +but generally, just as she had got its neck nicely straightened +out, and was going to give the hedgehog a blow with its head, it +WOULD twist itself round and look up in her face, with such a +puzzled expression that she could not help bursting out laughing: +and when she had got its head down, and was going to begin again, +it was very provoking to find that the hedgehog had unrolled +itself, and was in the act of crawling away: besides all this, +there was generally a ridge or furrow in the way wherever she +wanted to send the hedgehog to, and, as the doubled-up soldiers +were always getting up and walking off to other parts of the +ground, Alice soon came to the conclusion that it was a very +difficult game indeed. + + The players all played at once without waiting for turns, +quarrelling all the while, and fighting for the hedgehogs; and in +a very short time the Queen was in a furious passion, and went +stamping about, and shouting `Off with his head!' or `Off with +her head!' about once in a minute. + + Alice began to feel very uneasy: to be sure, she had not as +yet had any dispute with the Queen, but she knew that it might +happen any minute, `and then,' thought she, `what would become of +me? They're dreadfully fond of beheading people here; the great +wonder is, that there's any one left alive!' + + She was looking about for some way of escape, and wondering +whether she could get away without being seen, when she noticed a +curious appearance in the air: it puzzled her very much at +first, but, after watching it a minute or two, she made it out to +be a grin, and she said to herself `It's the Cheshire Cat: now I +shall have somebody to talk to.' + + `How are you getting on?' said the Cat, as soon as there was +mouth enough for it to speak with. + + Alice waited till the eyes appeared, and then nodded. `It's no +use speaking to it,' she thought, `till its ears have come, or at +least one of them.' In another minute the whole head appeared, +and then Alice put down her flamingo, and began an account of the +game, feeling very glad she had someone to listen to her. The +Cat seemed to think that there was enough of it now in sight, and +no more of it appeared. + + `I don't think they play at all fairly,' Alice began, in rather +a complaining tone, `and they all quarrel so dreadfully one can't +hear oneself speak--and they don't seem to have any rules in +particular; at least, if there are, nobody attends to them--and +you've no idea how confusing it is all the things being alive; +for instance, there's the arch I've got to go through next +walking about at the other end of the ground--and I should have +croqueted the Queen's hedgehog just now, only it ran away when it +saw mine coming!' + + `How do you like the Queen?' said the Cat in a low voice. + + `Not at all,' said Alice: `she's so extremely--' Just then +she noticed that the Queen was close behind her, listening: so +she went on, `--likely to win, that it's hardly worth while +finishing the game.' + + The Queen smiled and passed on. + + `Who ARE you talking to?' said the King, going up to Alice, and +looking at the Cat's head with great curiosity. + + `It's a friend of mine--a Cheshire Cat,' said Alice: `allow me +to introduce it.' + + `I don't like the look of it at all,' said the King: `however, +it may kiss my hand if it likes.' + + `I'd rather not,' the Cat remarked. + + `Don't be impertinent,' said the King, `and don't look at me +like that!' He got behind Alice as he spoke. + + `A cat may look at a king,' said Alice. `I've read that in +some book, but I don't remember where.' + + `Well, it must be removed,' said the King very decidedly, and +he called the Queen, who was passing at the moment, `My dear! I +wish you would have this cat removed!' + + The Queen had only one way of settling all difficulties, great +or small. `Off with his head!' she said, without even looking +round. + + `I'll fetch the executioner myself,' said the King eagerly, and +he hurried off. + + Alice thought she might as well go back, and see how the game +was going on, as she heard the Queen's voice in the distance, +screaming with passion. She had already heard her sentence three +of the players to be executed for having missed their turns, and +she did not like the look of things at all, as the game was in +such confusion that she never knew whether it was her turn or +not. So she went in search of her hedgehog. + + The hedgehog was engaged in a fight with another hedgehog, +which seemed to Alice an excellent opportunity for croqueting one +of them with the other: the only difficulty was, that her +flamingo was gone across to the other side of the garden, where +Alice could see it trying in a helpless sort of way to fly up +into a tree. + + By the time she had caught the flamingo and brought it back, +the fight was over, and both the hedgehogs were out of sight: +`but it doesn't matter much,' thought Alice, `as all the arches +are gone from this side of the ground.' So she tucked it away +under her arm, that it might not escape again, and went back for +a little more conversation with her friend. + + When she got back to the Cheshire Cat, she was surprised to +find quite a large crowd collected round it: there was a dispute +going on between the executioner, the King, and the Queen, who +were all talking at once, while all the rest were quite silent, +and looked very uncomfortable. + + The moment Alice appeared, she was appealed to by all three to +settle the question, and they repeated their arguments to her, +though, as they all spoke at once, she found it very hard indeed +to make out exactly what they said. + + The executioner's argument was, that you couldn't cut off a +head unless there was a body to cut it off from: that he had +never had to do such a thing before, and he wasn't going to begin +at HIS time of life. + + The King's argument was, that anything that had a head could be +beheaded, and that you weren't to talk nonsense. + + The Queen's argument was, that if something wasn't done about +it in less than no time she'd have everybody executed, all round. +(It was this last remark that had made the whole party look so +grave and anxious.) + + Alice could think of nothing else to say but `It belongs to the +Duchess: you'd better ask HER about it.' + + `She's in prison,' the Queen said to the executioner: `fetch +her here.' And the executioner went off like an arrow. + + The Cat's head began fading away the moment he was gone, and, +by the time he had come back with the Dutchess, it had entirely +disappeared; so the King and the executioner ran wildly up and +down looking for it, while the rest of the party went back to the game. + + + + CHAPTER IX + + The Mock Turtle's Story + + + `You can't think how glad I am to see you again, you dear old +thing!' said the Duchess, as she tucked her arm affectionately +into Alice's, and they walked off together. + + Alice was very glad to find her in such a pleasant temper, and +thought to herself that perhaps it was only the pepper that had +made her so savage when they met in the kitchen. + + `When I'M a Duchess,' she said to herself, (not in a very +hopeful tone though), `I won't have any pepper in my kitchen AT +ALL. Soup does very well without--Maybe it's always pepper that +makes people hot-tempered,' she went on, very much pleased at +having found out a new kind of rule, `and vinegar that makes them +sour--and camomile that makes them bitter--and--and barley-sugar +and such things that make children sweet-tempered. I only wish +people knew that: then they wouldn't be so stingy about it, you +know--' + + She had quite forgotten the Duchess by this time, and was a +little startled when she heard her voice close to her ear. +`You're thinking about something, my dear, and that makes you +forget to talk. I can't tell you just now what the moral of that +is, but I shall remember it in a bit.' + + `Perhaps it hasn't one,' Alice ventured to remark. + + `Tut, tut, child!' said the Duchess. `Everything's got a +moral, if only you can find it.' And she squeezed herself up +closer to Alice's side as she spoke. + + Alice did not much like keeping so close to her: first, +because the Duchess was VERY ugly; and secondly, because she was +exactly the right height to rest her chin upon Alice's shoulder, +and it was an uncomfortably sharp chin. However, she did not +like to be rude, so she bore it as well as she could. + + `The game's going on rather better now,' she said, by way of +keeping up the conversation a little. + + `'Tis so,' said the Duchess: `and the moral of that is--"Oh, +'tis love, 'tis love, that makes the world go round!"' + + `Somebody said,' Alice whispered, `that it's done by everybody +minding their own business!' + + `Ah, well! It means much the same thing,' said the Duchess, +digging her sharp little chin into Alice's shoulder as she added, +`and the moral of THAT is--"Take care of the sense, and the +sounds will take care of themselves."' + + `How fond she is of finding morals in things!' Alice thought to +herself. + + `I dare say you're wondering why I don't put my arm round your +waist,' the Duchess said after a pause: `the reason is, that I'm +doubtful about the temper of your flamingo. Shall I try the +experiment?' + + `HE might bite,' Alice cautiously replied, not feeling at all +anxious to have the experiment tried. + + `Very true,' said the Duchess: `flamingoes and mustard both +bite. And the moral of that is--"Birds of a feather flock +together."' + + `Only mustard isn't a bird,' Alice remarked. + + `Right, as usual,' said the Duchess: `what a clear way you +have of putting things!' + + `It's a mineral, I THINK,' said Alice. + + `Of course it is,' said the Duchess, who seemed ready to agree +to everything that Alice said; `there's a large mustard-mine near +here. And the moral of that is--"The more there is of mine, the +less there is of yours."' + + `Oh, I know!' exclaimed Alice, who had not attended to this +last remark, `it's a vegetable. It doesn't look like one, but it +is.' + + `I quite agree with you,' said the Duchess; `and the moral of +that is--"Be what you would seem to be"--or if you'd like it put +more simply--"Never imagine yourself not to be otherwise than +what it might appear to others that what you were or might have +been was not otherwise than what you had been would have appeared +to them to be otherwise."' + + `I think I should understand that better,' Alice said very +politely, `if I had it written down: but I can't quite follow it +as you say it.' + + `That's nothing to what I could say if I chose,' the Duchess +replied, in a pleased tone. + + `Pray don't trouble yourself to say it any longer than that,' +said Alice. + + `Oh, don't talk about trouble!' said the Duchess. `I make you +a present of everything I've said as yet.' + + `A cheap sort of present!' thought Alice. `I'm glad they don't +give birthday presents like that!' But she did not venture to +say it out loud. + + `Thinking again?' the Duchess asked, with another dig of her +sharp little chin. + + `I've a right to think,' said Alice sharply, for she was +beginning to feel a little worried. + + `Just about as much right,' said the Duchess, `as pigs have to +fly; and the m--' + + But here, to Alice's great surprise, the Duchess's voice died +away, even in the middle of her favourite word `moral,' and the +arm that was linked into hers began to tremble. Alice looked up, +and there stood the Queen in front of them, with her arms folded, +frowning like a thunderstorm. + + `A fine day, your Majesty!' the Duchess began in a low, weak +voice. + + `Now, I give you fair warning,' shouted the Queen, stamping on +the ground as she spoke; `either you or your head must be off, +and that in about half no time! Take your choice!' + + The Duchess took her choice, and was gone in a moment. + + `Let's go on with the game,' the Queen said to Alice; and Alice +was too much frightened to say a word, but slowly followed her +back to the croquet-ground. + + The other guests had taken advantage of the Queen's absence, +and were resting in the shade: however, the moment they saw her, +they hurried back to the game, the Queen merely remarking that a +moment's delay would cost them their lives. + + All the time they were playing the Queen never left off +quarrelling with the other players, and shouting `Off with his +head!' or `Off with her head!' Those whom she sentenced were +taken into custody by the soldiers, who of course had to leave +off being arches to do this, so that by the end of half an hour +or so there were no arches left, and all the players, except the +King, the Queen, and Alice, were in custody and under sentence of +execution. + + Then the Queen left off, quite out of breath, and said to +Alice, `Have you seen the Mock Turtle yet?' + + `No,' said Alice. `I don't even know what a Mock Turtle is.' + + `It's the thing Mock Turtle Soup is made from,' said the Queen. + + `I never saw one, or heard of one,' said Alice. + + `Come on, then,' said the Queen, `and he shall tell you his +history,' + + As they walked off together, Alice heard the King say in a low +voice, to the company generally, `You are all pardoned.' `Come, +THAT'S a good thing!' she said to herself, for she had felt quite +unhappy at the number of executions the Queen had ordered. + + They very soon came upon a Gryphon, lying fast asleep in the +sun. (IF you don't know what a Gryphon is, look at the picture.) +`Up, lazy thing!' said the Queen, `and take this young lady to +see the Mock Turtle, and to hear his history. I must go back and +see after some executions I have ordered'; and she walked off, +leaving Alice alone with the Gryphon. Alice did not quite like +the look of the creature, but on the whole she thought it would +be quite as safe to stay with it as to go after that savage +Queen: so she waited. + + The Gryphon sat up and rubbed its eyes: then it watched the +Queen till she was out of sight: then it chuckled. `What fun!' +said the Gryphon, half to itself, half to Alice. + + `What IS the fun?' said Alice. + + `Why, SHE,' said the Gryphon. `It's all her fancy, that: they +never executes nobody, you know. Come on!' + + `Everybody says "come on!" here,' thought Alice, as she went +slowly after it: `I never was so ordered about in all my life, +never!' + + They had not gone far before they saw the Mock Turtle in the +distance, sitting sad and lonely on a little ledge of rock, and, +as they came nearer, Alice could hear him sighing as if his heart +would break. She pitied him deeply. `What is his sorrow?' she +asked the Gryphon, and the Gryphon answered, very nearly in the +same words as before, `It's all his fancy, that: he hasn't got +no sorrow, you know. Come on!' + + So they went up to the Mock Turtle, who looked at them with +large eyes full of tears, but said nothing. + + `This here young lady,' said the Gryphon, `she wants for to +know your history, she do.' + + `I'll tell it her,' said the Mock Turtle in a deep, hollow +tone: `sit down, both of you, and don't speak a word till I've +finished.' + + So they sat down, and nobody spoke for some minutes. Alice +thought to herself, `I don't see how he can EVEN finish, if he +doesn't begin.' But she waited patiently. + + `Once,' said the Mock Turtle at last, with a deep sigh, `I was +a real Turtle.' + + These words were followed by a very long silence, broken only +by an occasional exclamation of `Hjckrrh!' from the Gryphon, and +the constant heavy sobbing of the Mock Turtle. Alice was very +nearly getting up and saying, `Thank you, sir, for your +interesting story,' but she could not help thinking there MUST be +more to come, so she sat still and said nothing. + + `When we were little,' the Mock Turtle went on at last, more +calmly, though still sobbing a little now and then, `we went to +school in the sea. The master was an old Turtle--we used to call +him Tortoise--' + + `Why did you call him Tortoise, if he wasn't one?' Alice asked. + + `We called him Tortoise because he taught us,' said the Mock +Turtle angrily: `really you are very dull!' + + `You ought to be ashamed of yourself for asking such a simple +question,' added the Gryphon; and then they both sat silent and +looked at poor Alice, who felt ready to sink into the earth. At +last the Gryphon said to the Mock Turtle, `Drive on, old fellow! +Don't be all day about it!' and he went on in these words: + + `Yes, we went to school in the sea, though you mayn't believe +it--' + + `I never said I didn't!' interrupted Alice. + + `You did,' said the Mock Turtle. + + `Hold your tongue!' added the Gryphon, before Alice could speak +again. The Mock Turtle went on. + + `We had the best of educations--in fact, we went to school +every day--' + + `I'VE been to a day-school, too,' said Alice; `you needn't be +so proud as all that.' + + `With extras?' asked the Mock Turtle a little anxiously. + + `Yes,' said Alice, `we learned French and music.' + + `And washing?' said the Mock Turtle. + + `Certainly not!' said Alice indignantly. + + `Ah! then yours wasn't a really good school,' said the Mock +Turtle in a tone of great relief. `Now at OURS they had at the +end of the bill, "French, music, AND WASHING--extra."' + + `You couldn't have wanted it much,' said Alice; `living at the +bottom of the sea.' + + `I couldn't afford to learn it.' said the Mock Turtle with a +sigh. `I only took the regular course.' + + `What was that?' inquired Alice. + + `Reeling and Writhing, of course, to begin with,' the Mock +Turtle replied; `and then the different branches of Arithmetic-- +Ambition, Distraction, Uglification, and Derision.' + + `I never heard of "Uglification,"' Alice ventured to say. `What +is it?' + + The Gryphon lifted up both its paws in surprise. `What! Never +heard of uglifying!' it exclaimed. `You know what to beautify +is, I suppose?' + + `Yes,' said Alice doubtfully: `it means--to--make--anything-- +prettier.' + + `Well, then,' the Gryphon went on, `if you don't know what to +uglify is, you ARE a simpleton.' + + Alice did not feel encouraged to ask any more questions about +it, so she turned to the Mock Turtle, and said `What else had you +to learn?' + + `Well, there was Mystery,' the Mock Turtle replied, counting +off the subjects on his flappers, `--Mystery, ancient and modern, +with Seaography: then Drawling--the Drawling-master was an old +conger-eel, that used to come once a week: HE taught us +Drawling, Stretching, and Fainting in Coils.' + + `What was THAT like?' said Alice. + + `Well, I can't show it you myself,' the Mock Turtle said: `I'm +too stiff. And the Gryphon never learnt it.' + + `Hadn't time,' said the Gryphon: `I went to the Classics +master, though. He was an old crab, HE was.' + + `I never went to him,' the Mock Turtle said with a sigh: `he +taught Laughing and Grief, they used to say.' + + `So he did, so he did,' said the Gryphon, sighing in his turn; +and both creatures hid their faces in their paws. + + `And how many hours a day did you do lessons?' said Alice, in a +hurry to change the subject. + + `Ten hours the first day,' said the Mock Turtle: `nine the +next, and so on.' + + `What a curious plan!' exclaimed Alice. + + `That's the reason they're called lessons,' the Gryphon +remarked: `because they lessen from day to day.' + + This was quite a new idea to Alice, and she thought it over a +little before she made her next remark. `Then the eleventh day +must have been a holiday?' + + `Of course it was,' said the Mock Turtle. + + `And how did you manage on the twelfth?' Alice went on eagerly. + + `That's enough about lessons,' the Gryphon interrupted in a +very decided tone: `tell her something about the games now.' + + + + CHAPTER X + + The Lobster Quadrille + + + The Mock Turtle sighed deeply, and drew the back of one flapper +across his eyes. He looked at Alice, and tried to speak, but for +a minute or two sobs choked his voice. `Same as if he had a bone +in his throat,' said the Gryphon: and it set to work shaking him +and punching him in the back. At last the Mock Turtle recovered +his voice, and, with tears running down his cheeks, he went on +again:-- + + `You may not have lived much under the sea--' (`I haven't,' +said Alice)--`and perhaps you were never even introduced to a lobster--' +(Alice began to say `I once tasted--' but checked herself hastily, +and said `No, never') `--so you can have no idea what a delightful +thing a Lobster Quadrille is!' + + `No, indeed,' said Alice. `What sort of a dance is it?' + + `Why,' said the Gryphon, `you first form into a line along the +sea-shore--' + + `Two lines!' cried the Mock Turtle. `Seals, turtles, salmon, +and so on; then, when you've cleared all the jelly-fish out of +the way--' + + `THAT generally takes some time,' interrupted the Gryphon. + + `--you advance twice--' + + `Each with a lobster as a partner!' cried the Gryphon. + + `Of course,' the Mock Turtle said: `advance twice, set to +partners--' + + `--change lobsters, and retire in same order,' continued the +Gryphon. + + `Then, you know,' the Mock Turtle went on, `you throw the--' + + `The lobsters!' shouted the Gryphon, with a bound into the air. + + `--as far out to sea as you can--' + + `Swim after them!' screamed the Gryphon. + + `Turn a somersault in the sea!' cried the Mock Turtle, +capering wildly about. + + `Back to land again, and that's all the first figure,' said the +Mock Turtle, suddenly dropping his voice; and the two creatures, +who had been jumping about like mad things all this time, sat +down again very sadly and quietly, and looked at Alice. + + `It must be a very pretty dance,' said Alice timidly. + + `Would you like to see a little of it?' said the Mock Turtle. + + `Very much indeed,' said Alice. + + `Come, let's try the first figure!' said the Mock Turtle to the +Gryphon. `We can do without lobsters, you know. Which shall +sing?' + + `Oh, YOU sing,' said the Gryphon. `I've forgotten the words.' + + So they began solemnly dancing round and round Alice, every now +and then treading on her toes when they passed too close, and +waving their forepaws to mark the time, while the Mock Turtle +sang this, very slowly and sadly:-- + + +`"Will you walk a little faster?" said a whiting to a snail. +"There's a porpoise close behind us, and he's treading on my + tail. +See how eagerly the lobsters and the turtles all advance! +They are waiting on the shingle--will you come and join the +dance? + +Will you, won't you, will you, won't you, will you join the +dance? +Will you, won't you, will you, won't you, won't you join the +dance? + + +"You can really have no notion how delightful it will be +When they take us up and throw us, with the lobsters, out to + sea!" +But the snail replied "Too far, too far!" and gave a look + askance-- +Said he thanked the whiting kindly, but he would not join the + dance. + Would not, could not, would not, could not, would not join + the dance. + Would not, could not, would not, could not, could not join + the dance. + +`"What matters it how far we go?" his scaly friend replied. +"There is another shore, you know, upon the other side. +The further off from England the nearer is to France-- +Then turn not pale, beloved snail, but come and join the dance. + + Will you, won't you, will you, won't you, will you join the + dance? + Will you, won't you, will you, won't you, won't you join the + dance?"' + + + + `Thank you, it's a very interesting dance to watch,' said +Alice, feeling very glad that it was over at last: `and I do so +like that curious song about the whiting!' + + `Oh, as to the whiting,' said the Mock Turtle, `they--you've +seen them, of course?' + + `Yes,' said Alice, `I've often seen them at dinn--' she +checked herself hastily. + + `I don't know where Dinn may be,' said the Mock Turtle, `but +if you've seen them so often, of course you know what they're +like.' + + `I believe so,' Alice replied thoughtfully. `They have their +tails in their mouths--and they're all over crumbs.' + + `You're wrong about the crumbs,' said the Mock Turtle: +`crumbs would all wash off in the sea. But they HAVE their tails +in their mouths; and the reason is--' here the Mock Turtle +yawned and shut his eyes.--`Tell her about the reason and all +that,' he said to the Gryphon. + + `The reason is,' said the Gryphon, `that they WOULD go with +the lobsters to the dance. So they got thrown out to sea. So +they had to fall a long way. So they got their tails fast in +their mouths. So they couldn't get them out again. That's all.' + + `Thank you,' said Alice, `it's very interesting. I never knew +so much about a whiting before.' + + `I can tell you more than that, if you like,' said the +Gryphon. `Do you know why it's called a whiting?' + + `I never thought about it,' said Alice. `Why?' + + `IT DOES THE BOOTS AND SHOES.' the Gryphon replied very +solemnly. + + Alice was thoroughly puzzled. `Does the boots and shoes!' she +repeated in a wondering tone. + + `Why, what are YOUR shoes done with?' said the Gryphon. `I +mean, what makes them so shiny?' + + Alice looked down at them, and considered a little before she +gave her answer. `They're done with blacking, I believe.' + + `Boots and shoes under the sea,' the Gryphon went on in a deep +voice, `are done with a whiting. Now you know.' + + `And what are they made of?' Alice asked in a tone of great +curiosity. + + `Soles and eels, of course,' the Gryphon replied rather +impatiently: `any shrimp could have told you that.' + + `If I'd been the whiting,' said Alice, whose thoughts were +still running on the song, `I'd have said to the porpoise, "Keep +back, please: we don't want YOU with us!"' + + `They were obliged to have him with them,' the Mock Turtle +said: `no wise fish would go anywhere without a porpoise.' + + `Wouldn't it really?' said Alice in a tone of great surprise. + + `Of course not,' said the Mock Turtle: `why, if a fish came +to ME, and told me he was going a journey, I should say "With +what porpoise?"' + + `Don't you mean "purpose"?' said Alice. + + `I mean what I say,' the Mock Turtle replied in an offended +tone. And the Gryphon added `Come, let's hear some of YOUR +adventures.' + + `I could tell you my adventures--beginning from this morning,' +said Alice a little timidly: `but it's no use going back to +yesterday, because I was a different person then.' + + `Explain all that,' said the Mock Turtle. + + `No, no! The adventures first,' said the Gryphon in an +impatient tone: `explanations take such a dreadful time.' + + So Alice began telling them her adventures from the time when +she first saw the White Rabbit. She was a little nervous about +it just at first, the two creatures got so close to her, one on +each side, and opened their eyes and mouths so VERY wide, but she +gained courage as she went on. Her listeners were perfectly +quiet till she got to the part about her repeating `YOU ARE OLD, +FATHER WILLIAM,' to the Caterpillar, and the words all coming +different, and then the Mock Turtle drew a long breath, and said +`That's very curious.' + + `It's all about as curious as it can be,' said the Gryphon. + + `It all came different!' the Mock Turtle repeated +thoughtfully. `I should like to hear her try and repeat +something now. Tell her to begin.' He looked at the Gryphon as +if he thought it had some kind of authority over Alice. + + `Stand up and repeat "'TIS THE VOICE OF THE SLUGGARD,"' said +the Gryphon. + + `How the creatures order one about, and make one repeat +lessons!' thought Alice; `I might as well be at school at once.' +However, she got up, and began to repeat it, but her head was so +full of the Lobster Quadrille, that she hardly knew what she was +saying, and the words came very queer indeed:-- + + `'Tis the voice of the Lobster; I heard him declare, + "You have baked me too brown, I must sugar my hair." + As a duck with its eyelids, so he with his nose + Trims his belt and his buttons, and turns out his toes.' + + [later editions continued as follows + When the sands are all dry, he is gay as a lark, + And will talk in contemptuous tones of the Shark, + But, when the tide rises and sharks are around, + His voice has a timid and tremulous sound.] + + `That's different from what I used to say when I was a child,' +said the Gryphon. + + `Well, I never heard it before,' said the Mock Turtle; `but it +sounds uncommon nonsense.' + + Alice said nothing; she had sat down with her face in her +hands, wondering if anything would EVER happen in a natural way +again. + + `I should like to have it explained,' said the Mock Turtle. + + `She can't explain it,' said the Gryphon hastily. `Go on with +the next verse.' + + `But about his toes?' the Mock Turtle persisted. `How COULD +he turn them out with his nose, you know?' + + `It's the first position in dancing.' Alice said; but was +dreadfully puzzled by the whole thing, and longed to change the +subject. + + `Go on with the next verse,' the Gryphon repeated impatiently: +`it begins "I passed by his garden."' + + Alice did not dare to disobey, though she felt sure it would +all come wrong, and she went on in a trembling voice:-- + + `I passed by his garden, and marked, with one eye, + How the Owl and the Panther were sharing a pie--' + + [later editions continued as follows + The Panther took pie-crust, and gravy, and meat, + While the Owl had the dish as its share of the treat. + When the pie was all finished, the Owl, as a boon, + Was kindly permitted to pocket the spoon: + While the Panther received knife and fork with a growl, + And concluded the banquet--] + + `What IS the use of repeating all that stuff,' the Mock Turtle +interrupted, `if you don't explain it as you go on? It's by far +the most confusing thing I ever heard!' + + `Yes, I think you'd better leave off,' said the Gryphon: and +Alice was only too glad to do so. + + `Shall we try another figure of the Lobster Quadrille?' the +Gryphon went on. `Or would you like the Mock Turtle to sing you +a song?' + + `Oh, a song, please, if the Mock Turtle would be so kind,' +Alice replied, so eagerly that the Gryphon said, in a rather +offended tone, `Hm! No accounting for tastes! Sing her "Turtle +Soup," will you, old fellow?' + + The Mock Turtle sighed deeply, and began, in a voice sometimes +choked with sobs, to sing this:-- + + + `Beautiful Soup, so rich and green, + Waiting in a hot tureen! + Who for such dainties would not stoop? + Soup of the evening, beautiful Soup! + Soup of the evening, beautiful Soup! + Beau--ootiful Soo--oop! + Beau--ootiful Soo--oop! + Soo--oop of the e--e--evening, + Beautiful, beautiful Soup! + + `Beautiful Soup! Who cares for fish, + Game, or any other dish? + Who would not give all else for two p + ennyworth only of beautiful Soup? + Pennyworth only of beautiful Soup? + Beau--ootiful Soo--oop! + Beau--ootiful Soo--oop! + Soo--oop of the e--e--evening, + Beautiful, beauti--FUL SOUP!' + + `Chorus again!' cried the Gryphon, and the Mock Turtle had +just begun to repeat it, when a cry of `The trial's beginning!' +was heard in the distance. + + `Come on!' cried the Gryphon, and, taking Alice by the hand, +it hurried off, without waiting for the end of the song. + + `What trial is it?' Alice panted as she ran; but the Gryphon +only answered `Come on!' and ran the faster, while more and more +faintly came, carried on the breeze that followed them, the +melancholy words:-- + + `Soo--oop of the e--e--evening, + Beautiful, beautiful Soup!' + + + + CHAPTER XI + + Who Stole the Tarts? + + + The King and Queen of Hearts were seated on their throne when +they arrived, with a great crowd assembled about them--all sorts +of little birds and beasts, as well as the whole pack of cards: +the Knave was standing before them, in chains, with a soldier on +each side to guard him; and near the King was the White Rabbit, +with a trumpet in one hand, and a scroll of parchment in the +other. In the very middle of the court was a table, with a large +dish of tarts upon it: they looked so good, that it made Alice +quite hungry to look at them--`I wish they'd get the trial done,' +she thought, `and hand round the refreshments!' But there seemed +to be no chance of this, so she began looking at everything about +her, to pass away the time. + + Alice had never been in a court of justice before, but she had +read about them in books, and she was quite pleased to find that +she knew the name of nearly everything there. `That's the +judge,' she said to herself, `because of his great wig.' + + The judge, by the way, was the King; and as he wore his crown +over the wig, (look at the frontispiece if you want to see how he +did it,) he did not look at all comfortable, and it was certainly +not becoming. + + `And that's the jury-box,' thought Alice, `and those twelve +creatures,' (she was obliged to say `creatures,' you see, because +some of them were animals, and some were birds,) `I suppose they +are the jurors.' She said this last word two or three times over +to herself, being rather proud of it: for she thought, and +rightly too, that very few little girls of her age knew the +meaning of it at all. However, `jury-men' would have done just +as well. + + The twelve jurors were all writing very busily on slates. +`What are they doing?' Alice whispered to the Gryphon. `They +can't have anything to put down yet, before the trial's begun.' + + `They're putting down their names,' the Gryphon whispered in +reply, `for fear they should forget them before the end of the +trial.' + + `Stupid things!' Alice began in a loud, indignant voice, but +she stopped hastily, for the White Rabbit cried out, `Silence in +the court!' and the King put on his spectacles and looked +anxiously round, to make out who was talking. + + Alice could see, as well as if she were looking over their +shoulders, that all the jurors were writing down `stupid things!' +on their slates, and she could even make out that one of them +didn't know how to spell `stupid,' and that he had to ask his +neighbour to tell him. `A nice muddle their slates'll be in +before the trial's over!' thought Alice. + + One of the jurors had a pencil that squeaked. This of course, +Alice could not stand, and she went round the court and got +behind him, and very soon found an opportunity of taking it +away. She did it so quickly that the poor little juror (it was +Bill, the Lizard) could not make out at all what had become of +it; so, after hunting all about for it, he was obliged to write +with one finger for the rest of the day; and this was of very +little use, as it left no mark on the slate. + + `Herald, read the accusation!' said the King. + + On this the White Rabbit blew three blasts on the trumpet, and +then unrolled the parchment scroll, and read as follows:-- + + `The Queen of Hearts, she made some tarts, + All on a summer day: + The Knave of Hearts, he stole those tarts, + And took them quite away!' + + `Consider your verdict,' the King said to the jury. + + `Not yet, not yet!' the Rabbit hastily interrupted. `There's +a great deal to come before that!' + + `Call the first witness,' said the King; and the White Rabbit +blew three blasts on the trumpet, and called out, `First +witness!' + + The first witness was the Hatter. He came in with a teacup in +one hand and a piece of bread-and-butter in the other. `I beg +pardon, your Majesty,' he began, `for bringing these in: but I +hadn't quite finished my tea when I was sent for.' + + `You ought to have finished,' said the King. `When did you +begin?' + + The Hatter looked at the March Hare, who had followed him into +the court, arm-in-arm with the Dormouse. `Fourteenth of March, I +think it was,' he said. + + `Fifteenth,' said the March Hare. + + `Sixteenth,' added the Dormouse. + + `Write that down,' the King said to the jury, and the jury +eagerly wrote down all three dates on their slates, and then +added them up, and reduced the answer to shillings and pence. + + `Take off your hat,' the King said to the Hatter. + + `It isn't mine,' said the Hatter. + + `Stolen!' the King exclaimed, turning to the jury, who +instantly made a memorandum of the fact. + + `I keep them to sell,' the Hatter added as an explanation; +`I've none of my own. I'm a hatter.' + + Here the Queen put on her spectacles, and began staring at the +Hatter, who turned pale and fidgeted. + + `Give your evidence,' said the King; `and don't be nervous, or +I'll have you executed on the spot.' + + This did not seem to encourage the witness at all: he kept +shifting from one foot to the other, looking uneasily at the +Queen, and in his confusion he bit a large piece out of his +teacup instead of the bread-and-butter. + + Just at this moment Alice felt a very curious sensation, which +puzzled her a good deal until she made out what it was: she was +beginning to grow larger again, and she thought at first she +would get up and leave the court; but on second thoughts she +decided to remain where she was as long as there was room for +her. + + `I wish you wouldn't squeeze so.' said the Dormouse, who was +sitting next to her. `I can hardly breathe.' + + `I can't help it,' said Alice very meekly: `I'm growing.' + + `You've no right to grow here,' said the Dormouse. + + `Don't talk nonsense,' said Alice more boldly: `you know +you're growing too.' + + `Yes, but I grow at a reasonable pace,' said the Dormouse: +`not in that ridiculous fashion.' And he got up very sulkily +and crossed over to the other side of the court. + + All this time the Queen had never left off staring at the +Hatter, and, just as the Dormouse crossed the court, she said to +one of the officers of the court, `Bring me the list of the +singers in the last concert!' on which the wretched Hatter +trembled so, that he shook both his shoes off. + + `Give your evidence,' the King repeated angrily, `or I'll have +you executed, whether you're nervous or not.' + + `I'm a poor man, your Majesty,' the Hatter began, in a +trembling voice, `--and I hadn't begun my tea--not above a week +or so--and what with the bread-and-butter getting so thin--and +the twinkling of the tea--' + + `The twinkling of the what?' said the King. + + `It began with the tea,' the Hatter replied. + + `Of course twinkling begins with a T!' said the King sharply. +`Do you take me for a dunce? Go on!' + + `I'm a poor man,' the Hatter went on, `and most things +twinkled after that--only the March Hare said--' + + `I didn't!' the March Hare interrupted in a great hurry. + + `You did!' said the Hatter. + + `I deny it!' said the March Hare. + + `He denies it,' said the King: `leave out that part.' + + `Well, at any rate, the Dormouse said--' the Hatter went on, +looking anxiously round to see if he would deny it too: but the +Dormouse denied nothing, being fast asleep. + + `After that,' continued the Hatter, `I cut some more bread- +and-butter--' + + `But what did the Dormouse say?' one of the jury asked. + + `That I can't remember,' said the Hatter. + + `You MUST remember,' remarked the King, `or I'll have you +executed.' + + The miserable Hatter dropped his teacup and bread-and-butter, +and went down on one knee. `I'm a poor man, your Majesty,' he +began. + + `You're a very poor speaker,' said the King. + + Here one of the guinea-pigs cheered, and was immediately +suppressed by the officers of the court. (As that is rather a +hard word, I will just explain to you how it was done. They had +a large canvas bag, which tied up at the mouth with strings: +into this they slipped the guinea-pig, head first, and then sat +upon it.) + + `I'm glad I've seen that done,' thought Alice. `I've so often +read in the newspapers, at the end of trials, "There was some +attempts at applause, which was immediately suppressed by the +officers of the court," and I never understood what it meant +till now.' + + `If that's all you know about it, you may stand down,' +continued the King. + + `I can't go no lower,' said the Hatter: `I'm on the floor, as +it is.' + + `Then you may SIT down,' the King replied. + + Here the other guinea-pig cheered, and was suppressed. + + `Come, that finished the guinea-pigs!' thought Alice. `Now we +shall get on better.' + + `I'd rather finish my tea,' said the Hatter, with an anxious +look at the Queen, who was reading the list of singers. + + `You may go,' said the King, and the Hatter hurriedly left the +court, without even waiting to put his shoes on. + + `--and just take his head off outside,' the Queen added to one +of the officers: but the Hatter was out of sight before the +officer could get to the door. + + `Call the next witness!' said the King. + + The next witness was the Duchess's cook. She carried the +pepper-box in her hand, and Alice guessed who it was, even before +she got into the court, by the way the people near the door began +sneezing all at once. + + `Give your evidence,' said the King. + + `Shan't,' said the cook. + + The King looked anxiously at the White Rabbit, who said in a +low voice, `Your Majesty must cross-examine THIS witness.' + + `Well, if I must, I must,' the King said, with a melancholy +air, and, after folding his arms and frowning at the cook till +his eyes were nearly out of sight, he said in a deep voice, `What +are tarts made of?' + + `Pepper, mostly,' said the cook. + + `Treacle,' said a sleepy voice behind her. + + `Collar that Dormouse,' the Queen shrieked out. `Behead that +Dormouse! Turn that Dormouse out of court! Suppress him! Pinch +him! Off with his whiskers!' + + For some minutes the whole court was in confusion, getting the +Dormouse turned out, and, by the time they had settled down +again, the cook had disappeared. + + `Never mind!' said the King, with an air of great relief. +`Call the next witness.' And he added in an undertone to the +Queen, `Really, my dear, YOU must cross-examine the next witness. +It quite makes my forehead ache!' + + Alice watched the White Rabbit as he fumbled over the list, +feeling very curious to see what the next witness would be like, +`--for they haven't got much evidence YET,' she said to herself. +Imagine her surprise, when the White Rabbit read out, at the top +of his shrill little voice, the name `Alice!' + + + + CHAPTER XII + + Alice's Evidence + + + `Here!' cried Alice, quite forgetting in the flurry of the +moment how large she had grown in the last few minutes, and she +jumped up in such a hurry that she tipped over the jury-box with +the edge of her skirt, upsetting all the jurymen on to the heads +of the crowd below, and there they lay sprawling about, reminding +her very much of a globe of goldfish she had accidentally upset +the week before. + + `Oh, I BEG your pardon!' she exclaimed in a tone of great +dismay, and began picking them up again as quickly as she could, +for the accident of the goldfish kept running in her head, and +she had a vague sort of idea that they must be collected at once +and put back into the jury-box, or they would die. + + `The trial cannot proceed,' said the King in a very grave +voice, `until all the jurymen are back in their proper places-- +ALL,' he repeated with great emphasis, looking hard at Alice as +he said do. + + Alice looked at the jury-box, and saw that, in her haste, she +had put the Lizard in head downwards, and the poor little thing +was waving its tail about in a melancholy way, being quite unable +to move. She soon got it out again, and put it right; `not that +it signifies much,' she said to herself; `I should think it +would be QUITE as much use in the trial one way up as the other.' + + As soon as the jury had a little recovered from the shock of +being upset, and their slates and pencils had been found and +handed back to them, they set to work very diligently to write +out a history of the accident, all except the Lizard, who seemed +too much overcome to do anything but sit with its mouth open, +gazing up into the roof of the court. + + `What do you know about this business?' the King said to +Alice. + + `Nothing,' said Alice. + + `Nothing WHATEVER?' persisted the King. + + `Nothing whatever,' said Alice. + + `That's very important,' the King said, turning to the jury. +They were just beginning to write this down on their slates, when +the White Rabbit interrupted: `UNimportant, your Majesty means, +of course,' he said in a very respectful tone, but frowning and +making faces at him as he spoke. + + `UNimportant, of course, I meant,' the King hastily said, and +went on to himself in an undertone, `important--unimportant-- +unimportant--important--' as if he were trying which word +sounded best. + + Some of the jury wrote it down `important,' and some +`unimportant.' Alice could see this, as she was near enough to +look over their slates; `but it doesn't matter a bit,' she +thought to herself. + + At this moment the King, who had been for some time busily +writing in his note-book, cackled out `Silence!' and read out +from his book, `Rule Forty-two. ALL PERSONS MORE THAN A MILE +HIGH TO LEAVE THE COURT.' + + Everybody looked at Alice. + + `I'M not a mile high,' said Alice. + + `You are,' said the King. + + `Nearly two miles high,' added the Queen. + + `Well, I shan't go, at any rate,' said Alice: `besides, +that's not a regular rule: you invented it just now.' + + `It's the oldest rule in the book,' said the King. + + `Then it ought to be Number One,' said Alice. + + The King turned pale, and shut his note-book hastily. +`Consider your verdict,' he said to the jury, in a low, trembling +voice. + + `There's more evidence to come yet, please your Majesty,' said +the White Rabbit, jumping up in a great hurry; `this paper has +just been picked up.' + + `What's in it?' said the Queen. + + `I haven't opened it yet,' said the White Rabbit, `but it seems +to be a letter, written by the prisoner to--to somebody.' + + `It must have been that,' said the King, `unless it was +written to nobody, which isn't usual, you know.' + + `Who is it directed to?' said one of the jurymen. + + `It isn't directed at all,' said the White Rabbit; `in fact, +there's nothing written on the OUTSIDE.' He unfolded the paper +as he spoke, and added `It isn't a letter, after all: it's a set +of verses.' + + `Are they in the prisoner's handwriting?' asked another of +they jurymen. + + `No, they're not,' said the White Rabbit, `and that's the +queerest thing about it.' (The jury all looked puzzled.) + + `He must have imitated somebody else's hand,' said the King. +(The jury all brightened up again.) + + `Please your Majesty,' said the Knave, `I didn't write it, and +they can't prove I did: there's no name signed at the end.' + + `If you didn't sign it,' said the King, `that only makes the +matter worse. You MUST have meant some mischief, or else you'd +have signed your name like an honest man.' + + There was a general clapping of hands at this: it was the +first really clever thing the King had said that day. + + `That PROVES his guilt,' said the Queen. + + `It proves nothing of the sort!' said Alice. `Why, you don't +even know what they're about!' + + `Read them,' said the King. + + The White Rabbit put on his spectacles. `Where shall I begin, +please your Majesty?' he asked. + + `Begin at the beginning,' the King said gravely, `and go on +till you come to the end: then stop.' + + These were the verses the White Rabbit read:-- + + `They told me you had been to her, + And mentioned me to him: + She gave me a good character, + But said I could not swim. + + He sent them word I had not gone + (We know it to be true): + If she should push the matter on, + What would become of you? + + I gave her one, they gave him two, + You gave us three or more; + They all returned from him to you, + Though they were mine before. + + If I or she should chance to be + Involved in this affair, + He trusts to you to set them free, + Exactly as we were. + + My notion was that you had been + (Before she had this fit) + An obstacle that came between + Him, and ourselves, and it. + + Don't let him know she liked them best, + For this must ever be + A secret, kept from all the rest, + Between yourself and me.' + + `That's the most important piece of evidence we've heard yet,' +said the King, rubbing his hands; `so now let the jury--' + + `If any one of them can explain it,' said Alice, (she had +grown so large in the last few minutes that she wasn't a bit +afraid of interrupting him,) `I'll give him sixpence. _I_ don't +believe there's an atom of meaning in it.' + + The jury all wrote down on their slates, `SHE doesn't believe +there's an atom of meaning in it,' but none of them attempted to +explain the paper. + + `If there's no meaning in it,' said the King, `that saves a +world of trouble, you know, as we needn't try to find any. And +yet I don't know,' he went on, spreading out the verses on his +knee, and looking at them with one eye; `I seem to see some +meaning in them, after all. "--SAID I COULD NOT SWIM--" you +can't swim, can you?' he added, turning to the Knave. + + The Knave shook his head sadly. `Do I look like it?' he said. +(Which he certainly did NOT, being made entirely of cardboard.) + + `All right, so far,' said the King, and he went on muttering +over the verses to himself: `"WE KNOW IT TO BE TRUE--" that's +the jury, of course-- "I GAVE HER ONE, THEY GAVE HIM TWO--" why, +that must be what he did with the tarts, you know--' + + `But, it goes on "THEY ALL RETURNED FROM HIM TO YOU,"' said +Alice. + + `Why, there they are!' said the King triumphantly, pointing to +the tarts on the table. `Nothing can be clearer than THAT. +Then again--"BEFORE SHE HAD THIS FIT--" you never had fits, my +dear, I think?' he said to the Queen. + + `Never!' said the Queen furiously, throwing an inkstand at the +Lizard as she spoke. (The unfortunate little Bill had left off +writing on his slate with one finger, as he found it made no +mark; but he now hastily began again, using the ink, that was +trickling down his face, as long as it lasted.) + + `Then the words don't FIT you,' said the King, looking round +the court with a smile. There was a dead silence. + + `It's a pun!' the King added in an offended tone, and +everybody laughed, `Let the jury consider their verdict,' the +King said, for about the twentieth time that day. + + `No, no!' said the Queen. `Sentence first--verdict afterwards.' + + `Stuff and nonsense!' said Alice loudly. `The idea of having +the sentence first!' + + `Hold your tongue!' said the Queen, turning purple. + + `I won't!' said Alice. + + `Off with her head!' the Queen shouted at the top of her voice. +Nobody moved. + + `Who cares for you?' said Alice, (she had grown to her full +size by this time.) `You're nothing but a pack of cards!' + + At this the whole pack rose up into the air, and came flying +down upon her: she gave a little scream, half of fright and half +of anger, and tried to beat them off, and found herself lying on +the bank, with her head in the lap of her sister, who was gently +brushing away some dead leaves that had fluttered down from the +trees upon her face. + + `Wake up, Alice dear!' said her sister; `Why, what a long +sleep you've had!' + + `Oh, I've had such a curious dream!' said Alice, and she told +her sister, as well as she could remember them, all these strange +Adventures of hers that you have just been reading about; and +when she had finished, her sister kissed her, and said, `It WAS a +curious dream, dear, certainly: but now run in to your tea; it's +getting late.' So Alice got up and ran off, thinking while she +ran, as well she might, what a wonderful dream it had been. + + But her sister sat still just as she left her, leaning her +head on her hand, watching the setting sun, and thinking of +little Alice and all her wonderful Adventures, till she too began +dreaming after a fashion, and this was her dream:-- + + First, she dreamed of little Alice herself, and once again the +tiny hands were clasped upon her knee, and the bright eager eyes +were looking up into hers--she could hear the very tones of her +voice, and see that queer little toss of her head to keep back +the wandering hair that WOULD always get into her eyes--and +still as she listened, or seemed to listen, the whole place +around her became alive the strange creatures of her little +sister's dream. + + The long grass rustled at her feet as the White Rabbit hurried +by--the frightened Mouse splashed his way through the +neighbouring pool--she could hear the rattle of the teacups as +the March Hare and his friends shared their never-ending meal, +and the shrill voice of the Queen ordering off her unfortunate +guests to execution--once more the pig-baby was sneezing on the +Duchess's knee, while plates and dishes crashed around it--once +more the shriek of the Gryphon, the squeaking of the Lizard's +slate-pencil, and the choking of the suppressed guinea-pigs, +filled the air, mixed up with the distant sobs of the miserable +Mock Turtle. + + So she sat on, with closed eyes, and half believed herself in +Wonderland, though she knew she had but to open them again, and +all would change to dull reality--the grass would be only +rustling in the wind, and the pool rippling to the waving of the +reeds--the rattling teacups would change to tinkling sheep- +bells, and the Queen's shrill cries to the voice of the shepherd +boy--and the sneeze of the baby, the shriek of the Gryphon, and +all thy other queer noises, would change (she knew) to the +confused clamour of the busy farm-yard--while the lowing of the +cattle in the distance would take the place of the Mock Turtle's +heavy sobs. + + Lastly, she pictured to herself how this same little sister of +hers would, in the after-time, be herself a grown woman; and how +she would keep, through all her riper years, the simple and +loving heart of her childhood: and how she would gather about +her other little children, and make THEIR eyes bright and eager +with many a strange tale, perhaps even with the dream of +Wonderland of long ago: and how she would feel with all their +simple sorrows, and find a pleasure in all their simple joys, +remembering her own child-life, and the happy summer days. + + THE END + \ No newline at end of file diff --git a/tests/TestArchives/MiscTest/news.txt b/tests/TestArchives/MiscTest/news.txt new file mode 100644 index 00000000..df56febe --- /dev/null +++ b/tests/TestArchives/MiscTest/news.txt @@ -0,0 +1,10059 @@ +#! rnews 1312 +Path: alberta!mnetor!uunet!mcvax!ukc!dcl-cs!strath-cs!jml +From: jml@cs.strath.ac.uk (Joseph McLean) +Newsgroups: sci.math +Subject: the extendability of digit sequences into primes +Message-ID: <753@stracs.cs.strath.ac.uk> +Date: 2 Dec 87 10:36:33 GMT +Reply-To: jml@cs.strath.ac.uk (Joseph McLean) +Organization: Comp. Sci. Dept., Strathclyde Univ., Scotland. +Lines: 19 + +Is the following conjecture reasonable and/or provable? : + +Given a sequence of digits, starting with a non-zero digit, of arbitrary +but finite length, is it always possible to extend this sequence by +appending more digits, in such a way as to form a prime? + +e.g. the sequence 1 can be extended into a prime in an infinite number +of ways, as in 13, 17, 19, 101, 1231, 1579, etc (there an infinite +number of primes beginning with a 1 by Bertrand's postulate). +However, it is far more difficult to try and locate a prime which +starts with the sequence 1528296922945708 (although at least one is known). + +My personal opinion is that the conjecture is reasonable, simply because +one can keep adding digits at the end and checking for primality ad +infinitum, and the law of averages will do the rest. Of course this is +totally groundless mathematically, so can anyone provide a heuristic +argument with more weight? + + jml, the mad mathematician. +#! rnews 3077 +Path: alberta!mnetor!uunet!husc6!psuvax1!burdvax!bigburd!fritzson +From: fritzson@bigburd.PRC.Unisys.COM (Richard Fritzson) +Newsgroups: comp.editors +Subject: Re: lisp environments (Structure vs. text editors) +Message-ID: <3375@bigburd.PRC.Unisys.COM> +Date: 14 Dec 87 02:11:18 GMT +References: <487@PT.CS.CMU.EDU> <460@cresswell.quintus.UUCP> <499@PT.CS.CMU.EDU> +Sender: news@bigburd.PRC.Unisys.COM +Organization: Unisys Corporation, Paoli Research Center; Paoli, PA +Lines: 56 + +In article <499@PT.CS.CMU.EDU> ralphw@IUS2.CS.CMU.EDU (Ralph Hyre) writes: +>In article <460@cresswell.quintus.UUCP> pds@quintus.UUCP (Peter Schachte) writes: +>>Text editors CANNOT simulate structure editors. They can do a rather +>>feeble job of it. Text editors fall down when context information is +> +>I disagree - a PROGRAMMABLE text editor can do anything you want. This is +>because it's programmable. Whether you're happy with the performance or a + +Sure it can do anything. The best way for a programmable text editor to +simulate a structure editor would be for it to build an internal +representation (or structure) or what was really being edited and then +use its text manipulating primitives to show the user the effect of his +editing commands on the structure that is "really" being edited. Now you've +shown that mocklisp (for example) is a language in which you can implement +a structure editor. I doubt if it is the best way to do it though. + +>>...For example: a structure editor can supply different commands, different +>>facilities, for editing comments and code. +>Seems like there's the potential here for moby modefulness. I can't see +>why I would want different commands when I edit code compared with comments. + +I don't know about "commands", but Common Lisp comments are nothing +like Common Lisp code (much to the shame of Common Lisp). I want the +characters I type in as comments treated differently than those I type in +as parts of S-expressions. + +>My interest is in an pseudo-WYSIWYG editor which gives you the option +>of entering/editing text without formatting attributes, then optionally +>displaying the text with them. <...>This sort of decoupling between editing a +>document and a representation of a document could even be used to great +>advantage in many environments: + +You're right. An editor which is really editing the structure underlying +the visual presentation of it IS a useful thing. + +> A program code editor might actually be showing you variable names, +> statements, and S-expressions while it is really writing the P-code +> (or .lbin file) on the fly. +> This could result in 'instant' language interpreter facilities and +> fast compilers. +> [I admit that this might be hairy to program in MockLisp.] + +But it is one of the reasons Xerox structure editor fans are fans. + +>[disclaimer: I've never used a 'structure editor' + +No offense intended, but I could tell. If you write any Lisp you should +look for an opportunity to try SEdit on a D-machine. + + + + +-- + -Rich Fritzson + ARPA: fritzson@prc.unisys.com + UUCP: {sdcrdcf,psuvax1,cbmvax}!burdvax!fritzson +#! rnews 3135 +Path: alberta!mnetor!uunet!husc6!cmcl2!brl-adm!umd5!ames!sdcsvax!sdcc6!loral!dml +From: dml@loral.UUCP (Dave Lewis) +Newsgroups: rec.arts.movies +Subject: Re: Live Action Amber Films +Summary: Use Zelazny's descriptions! +Message-ID: <1496@loral.UUCP> +Date: 14 Dec 87 06:41:04 GMT +References: <349@morningdew.BBN.COM> <2620001@hpcvlx.HP.COM> +Reply-To: dml@loral.UUCP (Dave Lewis) +Followup-To: rec.arts.movies +Distribution: na +Organization: Loral Instrumentation, San Diego +Lines: 59 + +In article <2620001@hpcvlx.HP.COM> markc@hpcvlx.HP.COM (Mark Cook) writes: +>>/ hpcvlx:rec.arts.movies / dkovar@lf-server-2.BBN.COM (David Kovar) / 7:07 am Dec 9, 1987 / +>> +>> Well, someone else was wondering who would be the actors in a Tolkien +>>film which brought to mind a favorite question of mine from a few years +>>back: Who would play the parts of a Amber film? I used to have the + +>>Corwin: Mel Gibson + +Jonathan Pryce. From "Something Wicked This Way Comes". + +> even better, how about Timothy Dalton (James Bond isn't the only thing he + +>>Brand: (Who's the guy from Kiss who was in Runaway?) + +> You mean Gene Simmons. Well, he could play the part but he has to look like + + No way. Brand is "a figure both like Bleys and myself. My features, though +smaller, my eyes, Bleys' hair. There was a quality of both strength and weak- +ness, questing and abandonment about him." This is Corwin speaking, of course. + + And Bleys is "a fiery bearded, flame-crowned man, dressed all in red and +orange, mainly of silk stuff, and he held a sword in his right hand and a +glass of wine in his left, and the devil himself danced behind his eyes, as +blue as Flora's, or Eric's. His chin was slight, but the beard covered it." + + I can't think of anyone offhand for either part, but I nominate Gene Simmons +to play Caine: "Then came the swarthy, dark-eyed countenance of Caine, dressed +all in satin that was black and green, wearing a dark three-cornered hat set +at a rakish angle, a green plume of feathers trailing down the back." (Yeah, +I got "Nine Princes in Amber" lying right next to the keyboard here) + + Random: "a wily-looking little man, with a sharp nose and a laughing mouth +and a shock of straw-colored hair." How about Dudley Moore (with his hair +bleached, of course). + + Dierdre: "a black-haired girl with [Flora's] blue eyes, and her hair hung +long and she was dressed all in black, with a girdle of silver about her +waist." Lee Meriwether or Kate Jackson. + + Fiona: "with hair like Bleys or Brand, [Corwin's] eyes, and a complexion +like mother of pearl. Ann-Margret! + + That's all for now; if people are interested I can type in the whole 2-1/2 +pages of descriptions so we'll REALLY have something to argue over. + +------------------------------- + Dave Lewis Loral Instrumentation San Diego + + hp-sdd --\ ihnp4 --\ + sdcrdcf --\ bang --\ kontron -\ + csndvax ---\ calmasd -->-->!crash --\ + celerity --->------->!sdcsvax!sdcc3 --->--->!loral!dml (uucp) + dcdwest ---/ gould9 --/ + + "I'm alive and he's dead and that's the way I wanted it." + -- Corwin, about Borel + +------------------------------- +#! rnews 2421 +Path: alberta!mnetor!uunet!mcvax!ukc!dcl-cs!strath-cs!jim +From: jim@cs.strath.ac.uk (Jim Reid) +Newsgroups: comp.mail.headers +Subject: Re: RFC976 vs. the real world... +Message-ID: <754@stracs.cs.strath.ac.uk> +Date: 2 Dec 87 12:51:51 GMT +References: <18533@amdahl.amdahl.com> +Reply-To: jim@cs.strath.ac.uk +Organization: Comp. Sci. Dept., Strathclyde Univ., Scotland. +Lines: 40 + +In article <18533@amdahl.amdahl.com> tron@uts.amdahl.com (Ronald S. Karr) writes: +>Some Introduction: +>However, we have conflicting ideas concerning what to do with sender +>addresses in headers. We do, now, support the idea that a pure !-path +>coming in can be left as a !-path, with the current hostname prepended +>(this is optional and is a function of the destination). However, +>should I ever produce, in mail originated locally, a From: line in the +>following form? +> +> From: localhost!username + +The answer is perhaps. In an ideal world, everyone will adhere to one +standard for mail headers - RFC822 possibly, but X.400 is more likely. +Until that glorious day arrives (if it ever does), mailers at the mail +'gateways' between networks will have little option but to munge +addresses because of incompatible mail headers and addressing formats. + +What you mail system should do is rewrite mail headers into the +appropriate form for transmission to a given host. In short, if your +uucp neighbours only understand bang-style addresses, you mailer should +only present bang-style paths to these sites. If some sites understand +RFC822 (user@host.domain), then you should send them RFC822 style mail. +What would be less easy for the mailer is separating your bang-stlye +uucp neighbours from those who understand RFC822. + +The best mailers (MMDF or sendmail - no flames please!) take an input +address, convert it to a canonical form and then rewrite the address in +the appropriate style for the message transfer agent. This is the most +sensible way of dealing with hybrid addresses like A!B@C. [Does that +mean send by uucp to A for relaying to user B on host C or does it mean +send to C for them to relay to user B on uucp host A? Then what if C +(or A) doesn't like addresses with '!' (or '@') signs in them?] + + Jim +-- +ARPA: jim%cs.strath.ac.uk@ucl-cs.arpa, jim@cs.strath.ac.uk +UUCP: jim@strath-cs.uucp, ...!seismo!mcvax!ukc!strath-cs!jim +JANET: jim@uk.ac.strath.cs + +"JANET domain ordering is swapped around so's there'd be some use for rev(1)!" +#! rnews 3873 +Path: alberta!mnetor!uunet!husc6!cmcl2!brl-adm!umd5!ames!sdcsvax!sdcc6!loral!dml +From: dml@loral.UUCP (Dave Lewis) +Newsgroups: rec.arts.sf-lovers +Subject: Re: One more long-gone show +Summary: What S. F. movies should be +Keywords: Questor +Message-ID: <1497@loral.UUCP> +Date: 14 Dec 87 06:45:22 GMT +References: <1672@bsu-cs.UUCP> +Reply-To: dml@loral.UUCP (Dave Lewis) +Followup-To: rec.arts.sf-lovers +Distribution: na +Organization: Loral Instrumentation, San Diego +Lines: 64 + +In article <1672@bsu-cs.UUCP> cfchiesa@bsu-cs.UUCP (Christopher F. Chiesa) writes: +>Anyone remember a movie called _The_Questor_Tapes_ ? Basic premise: gov't +>project constructs an android according to eccentric scientist's specs; and- + +>C.Chiesa + + Yea, verily, I recall The Questor Tapes. I've forgotten the scientist's +name, but he was a very rich and secretive genius known for several major +advances in robotics and cybernetics. About 2 years previous to the start +of the movie, he had disappeared, leaving only a partially completed project +he called Questor. Much of the work was complete, including a small fusion +reactor, most of the brain, and a lot of the support machinery. He also left +a BIG mag tape of programs, which some government idiot had partially erased +while trying to decode it. Questor, when activated, did nothing; the team +that assembled him figured it was because of the bad tape. + + Late that night, Questor got up, used the 'finishing' molds to give himself +human features, and walked out. The scientist had known one member of the +Questor-assembly team and put his name and address on the program tape; by +good fortune it had survived the attempted decoding. Questor knows only that +he must find `a boat' -- other details have been erased. + + The government catches up with them in a playground and some fool shoots +Questor. Apparently the shock knocks some bits loose because when he sees +a jungle gym that looks like Noah's Ark he remembers, "the boat, the boat +of legend. [whatsisname] is waiting for me there." He also remembers that +if he doesn't find the scientist within about two days, his fusion power +supply is programmed to overload and blow up. + + They patch him up and he leads them a merry chase to Mt. Ararat where he +finds his creator in a cave hidden by a force barrier/hologram projection. +There is a long row of metallic slabs suspended about a meter above the +floor; on each lies a defunct robot. Each one wears clothing from a time +far earlier than the next. Questor's creator lies on the second to last +slab, still conscious but unable to move. + + These robots have been watching over the human race for more than ten +thousand years. Each one lasts two hundred years, then builds his successor. +Questor's predecessor was brought to an early end by some combination of +pollution and radiation exposure; he has provided Questor with extra +shielding so he will last the full two centuries. + + Questor is the last. By the end of his term, the human race will have +reached a point where we can make our own decisions without guidance. +The robots were placed here by some advanced aliens to see us through our +racial childhood, to allow us a chance to mature and achieve whatever +potential we have. + + The Questor Tapes was an excellent movie, one makers of more recent films +should take a lesson from. Very few other movies have impressed me as much +as "2001: A Space Odyssey" and "The Questor Tapes". They show up the likes +of"Close Encounters of the Third Kind" and "E.T." for the vapid silliness +they are. + +------------------------------- + Dave Lewis Loral Instrumentation San Diego + + hp-sdd --\ ihnp4 --\ + sdcrdcf --\ bang --\ kontron -\ + csndvax ---\ calmasd -->-->!crash --\ + celerity --->------->!sdcsvax!sdcc3 --->--->!loral!dml (uucp) + dcdwest ---/ gould9 --/ + +------------------------------- +#! rnews 1384 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!hwcs!jack +From: jack@cs.hw.ac.uk (Jack Campin) +Newsgroups: rec.arts.sf-lovers +Subject: Re: M. John Harrison +Message-ID: <1560@brahma.cs.hw.ac.uk> +Date: 2 Dec 87 18:20:17 GMT +References: <1950@charon.unm.edu> +Reply-To: jack@cs.glasgow.ac.uk (Jack Campin) +Organization: PISA Project, Glesga Yoonie +Lines: 23 +Summary: + +Expires: + +Sender: + +Followup-To: + + + + +[ignore the above address and use my signature] + + +By far the best thing I have read by MJH is a long short story called +"Running Down", about a man with unwanted psychic powers that cause things +to malfunction, decay and fall apart around him. It is set in a Britain +in the near future of when the story was written (i.e. about now) in which +the whole society reflects a similar dingy, pointless chaos - remarkably +like Britain after 8 years of Thatcher, in fact. +He's very good at describing that sort of situation - his novel "The Centauri +Device" does it at length, though his suggested political solution is bloody +stupid. His understanding of anarchism is about on a level with Robert Anton +Wilson's. + +- jack + +-- +ARPA: jack%cs.glasgow.ac.uk@nss.cs.ucl.ac.uk +JANET:jack@uk.ac.glasgow.cs USENET: ...mcvax!ukc!cs.glasgow.ac.uk!jack +Mail: Jack Campin, Computing Science Department, University of Glasgow, + 17 Lilybank Gardens, Glasgow G12 8QQ, Scotland (041 339 8855 x 6045) +#! rnews 1188 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!hwcs!jack +From: jack@cs.hw.ac.uk (Jack Campin) +Newsgroups: rec.music.classical +Subject: Re: Tippett +Message-ID: <1561@brahma.cs.hw.ac.uk> +Date: 2 Dec 87 18:38:20 GMT +References: <1950@bath63.ux63.bath.ac.uk> +Reply-To: jack@cs.glasgow.ac.uk (Jack Campin) +Organization: PISA Project, Glesga Yoonie +Lines: 15 +Summary: + +Expires: + +Sender: + +Followup-To: + + + +[ignore the above email address and use my signature] +Tippett moved on a LONG way musically after "A Child Of Our Time". +I believe his masterpiece is the Triple Concerto for violin, viola and cello. +There is a wonderful recording of it by Pauk, Imai and Kirschbaum with the LSO +under Davis. +A problem I find with a lot of his music is the silly words. The man really +shouldn't have tried writing his own libretti that often. +I believe he's got another opera in the pipeline, due for its premiere in the +next few months. +-- +ARPA: jack%cs.glasgow.ac.uk@nss.cs.ucl.ac.uk +JANET:jack@uk.ac.glasgow.cs USENET: ...mcvax!ukc!cs.glasgow.ac.uk!jack +Mail: Jack Campin, Computing Science Department, University of Glasgow, + 17 Lilybank Gardens, Glasgow G12 8QQ, Scotland (041 339 8855 x 6045) +#! rnews 894 +Path: alberta!mnetor!uunet!mcvax!ukc!eagle!jg +From: jg@eagle.ukc.ac.uk (J.Grant) +Newsgroups: comp.sys.mac +Subject: The Spinning watch cursor +Message-ID: <4023@eagle.ukc.ac.uk> +Date: 3 Dec 87 14:59:09 GMT +Reply-To: jg@ukc.ac.uk (J.Grant) +Organization: Computing Lab, University of Kent at Canterbury, UK. +Lines: 11 + +OK - I've changed my spinning watch back into the lovely sand-timer +(remember the good old days?); I've changed the CURS resource in the +Finder and also in the System so that I have various quantities +of sand in the top & bottom, but there is still a watch lurking! + +More precisely, where does the watch that says 9 o'clock live, as +now I get the magic watch followed by the sand1->7, then the watch +again as the cycle repeats. This only happens in the Finder, so I +suspect that there must be a watch lurking elsewhere, but where? + +Ps. system 4.2b(5?) & Finder 6.0 (Mac 512Ke) +#! rnews 3539 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!csnjr +From: csnjr@its63b.ed.ac.uk (Nick Rothwell) +Newsgroups: rec.music.synth,rec.music.makers +Subject: Casio MG500, Roland MT-32, MIDI bug? [LONG] +Summary: Where's the MIDI bug in this lot?: +Keywords: MG500 MT-32 MIDI +Message-ID: <805@its63b.ed.ac.uk> +Date: 3 Dec 87 13:10:56 GMT +Reply-To: nick%ed.lfcs@uk.ac.ucl.cs.nss (Nick Rothwell) +Organization: LFCS, University of Edinburgh +Lines: 46 +Xref: alberta rec.music.synth:1879 rec.music.makers:1070 + +Last weekend a friend and I strolled into a music shop and ended up playing +with the new Casio MG500 MIDI guitar linked into a Roland MT-32. I don't +play guitar, and was just along for the curiosity, but I've got a few comments +to make and a question about what I consider to be a MIDI bug in one of the +instruments. + Firstly - the performance of the MG500. I wasn't actually playing it (I was +just pushing buttons on the MT-32 instead), but I was impressed with its +speed and tracking ability - it was fast and followed pitch accurately, +responding to pitch bend and so on; it generally sounded pretty tight. +There were a couple of things I didn't like - but maybe it's a generic +weakness of all guitar-to-MIDI systems. Firstly, the guitar transmits +velocity information (hit the string harder -> louder/brighter note), but +gives no control (other than pitch-bend) once a note's sounding - there's +nothing equivalent to aftertouch/modulation so once a note sounds you're +at the mercy of the synth until you stop the string. +Point two - You've got six strings, so you can only sound six synth voices. +This is probably obvious, but playing a guitar patch through MIDI doesn't +sound like a real guitar, because each touch of a string retriggers the voice +on that string, sometimes in a rather distracting way. On a real (classical) +guitar you have the resonance of the soundbox to hang on to notes so you +aren't aware of this (I presume - comments?) +Now for what is (in my opinion) a MIDI Bug! Play two different notes on +two strings and you get two voices - ok so far. Play the same note on two +different strings and you get one voice. Humm. Play two different notes on +two strings and slide one note up to the other, and one of the voices is +chopped off. I think this is a bug - something somewhere doesn't want to +the same note more than once. Needless to say, this completely screws up +a number of guitar chords. + We mentioned this to the guy in the shop. He seemed convinced that it's +a problem with the MIDI spec. itself - if you play a keyboard synth, you +have to release the middle C key to play it again, don't you? I think this +is a load of dingos kidneys - if I send my D-50 two separate middle C +note on messages, then I'll get two voices cycling through the envelopes at +middle C pitch. This is what happens with the sustain pedal on, as well. + What's the verdict, net people? I think the guy was wrong (quite adamant, +but wrong...) and there's a bug in one of the boxes. I suspect the MG500. +If the MT-32 is anything like the D-50, then it doesn't care about playing +the same note twice. (A quick note in passing that synths with less voices +(Juno106 for instance) often won't double a voice, in an attempt to play +chords properly without running out.) +-- +Nick Rothwell, Laboratory for Foundations of Computer Science, Edinburgh. + nick%lfcs.ed.ac.uk@nss.cs.ucl.ac.uk + !mcvax!ukc!lfcs!nick +~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ +"Nothing's forgotten. Nothing is ever forgotten." - Herne +#! rnews 1505 +Path: alberta!mnetor!uunet!mcvax!ukc!dcl-cs!bath63!pes +From: pes@ux63.bath.ac.uk (Smee) +Newsgroups: comp.sys.atari.st +Subject: Re: Resource file question +Keywords: resource mwc rcs .rsc dri c +Message-ID: <1963@bath63.ux63.bath.ac.uk> +Date: 3 Dec 87 10:33:26 GMT +References: <1592@wiley.UUCP> +Reply-To: pes@ux63.bath.ac.uk (Smee) +Organization: AUCC c/o University of Bath +Lines: 19 + + +You might try looking to see if K-Resource is still available (by Kuma Software, +who else?). It's been out a long while. It's now available bundled with some +of the MetaComCo stuff (in particular the new Lattice C) but I believe that +Kuma still do it separately as well. Don't have a clue what it costs, but +must be cheaper than a new compiler. + +It produces (by switch option) appropriate 'include' type files for C, +FORTRAN, and 2 other languages which I've conveniently forgotten -- in +addition to the expected .RSC file. Will also produce a 'non-specific +structured description' file (they say, I've never tried this) which is +alleged to be pretty easy to massage into an appropriate 'include' for +any unsupported language you might like. + +The documentation is written in a bit of a 'too-folksy' style for my liking, +but the program is pretty intuitive to use which makes up for some of that. +It does, however, assume that you have some sort of a clue as to what the +various resource items/flags mean and do -- it doesn't teach you how to use +RSC files or what they mean, but rather gives a handle for making them. +#! rnews 1258 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!hwcs!neil +From: neil@cs.hw.ac.uk (Neil Forsyth) +Newsgroups: comp.sys.atari.st +Subject: Bug in bets test Gulam +Keywords: none +Message-ID: <1562@brahma.cs.hw.ac.uk> +Date: 3 Dec 87 09:46:32 GMT +Organization: Computer Science, Heriot-Watt U., Scotland +Lines: 28 + + +I think I have found a bug in the latest version of Gulam. + + alias test 'echo $<' + +produces a couple of spurious charcters on the input line. + + $<%& + +The characters are usually above $80. The alpha version didn't do this. +I just delete them by backspacing anyway. + + echo $< + +by itself works fine. + +------------------------------------------------------------------------------- +"I think all right thinking people in this country are sick and tired of being +told that ordinary decent people are fed up in this country with being sick and +tired. I'm certainly not and I'm sick and tired of being told that I am!" +- Monty Python + + Neil Forsyth JANET: neil@uk.ac.hw.cs + Dept. of Computer Science ARPA: neil@cs.hw.ac.uk + Heriot-Watt University UUCP: ..!ukc!cs.hw.ac.uk!neil + Edinburgh + Scotland +------------------------------------------------------------------------------- +#! rnews 1009 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!hwcs!jack +From: jack@cs.hw.ac.uk (Jack Campin) +Newsgroups: comp.sys.mac +Subject: Re: how strong of a magnet? +Message-ID: <1564@brahma.cs.hw.ac.uk> +Date: 3 Dec 87 18:59:42 GMT +References: <9554@shemp.UCLA.EDU> +Reply-To: jack@cs.glasgow.ac.uk (Jack Campin) +Organization: PISA Project, Glesga Yoonie +Lines: 12 +Summary: + +Expires: + +Sender: + +Followup-To: + + + +[ignore the above email address and use my signature] +This may be an FOAF story (urban folklore) but I have heard that the mag-lev +train at Birmingham Airport lets enough field into the passenger compartment +to wipe floppies. +Then again, I have also heard that story about ordinary underground railways +and it certainly isn't true of them. +-- +ARPA: jack%cs.glasgow.ac.uk@nss.cs.ucl.ac.uk +JANET:jack@uk.ac.glasgow.cs USENET: ...mcvax!ukc!cs.glasgow.ac.uk!jack +Mail: Jack Campin, Computing Science Department, University of Glasgow, + 17 Lilybank Gardens, Glasgow G12 8QQ, Scotland (041 339 8855 x 6045) +#! rnews 988 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!csan +From: csan@its63b.ed.ac.uk (Andie) +Newsgroups: comp.sys.atari.st +Subject: Re: Resource file question +Keywords: Kuma +Message-ID: <808@its63b.ed.ac.uk> +Date: 3 Dec 87 23:08:12 GMT +References: <1592@wiley.UUCP> <1298@saturn.ucsc.edu> +Reply-To: csan@its63b.ed.ac.uk (Andie) +Organization: Computer Science Department, Edinburgh University +Lines: 14 + +In article <1298@saturn.ucsc.edu> koreth@ssyx.ucsc.edu (Steven Grimm) writes: +> +>Kuma Software makes the best resource editor I've seen. It's called +>"K-Resource" and is a really friendly, well-thought-out piece of software. +> +I am in total agreement here. I use it in preference to any others I have. + +Andie Ness . Department of Computer Science ,Edinburgh University. + +ARPA: csan%ed.itspna@nss.cs.ucl.ac.uk UUCP: ...!uunet!mcvax!ukc!itspna!csan + JANET: csan@uk.ac.ed.itspna + +% These are my own views and any resemblance to any coherent reasoning is +% probably a typo. +#! rnews 852 +Path: alberta!mnetor!uunet!mcvax!ukc!stc!idec!howellg +From: howellg@idec.stc.co.uk (Gareth Howell) +Newsgroups: rec.ham-radio.packet,comp.protocols.tcp-ip +Subject: NEEDED: KISS for TNC220 +Message-ID: <869@idec.stc.co.uk> +Date: 1 Dec 87 09:05:59 GMT +Organization: ICL Network Systems, Stevenage, Herts. UK +Lines: 12 +Xref: alberta rec.ham-radio.packet:767 comp.protocols.tcp-ip:1918 + +I have a Pacomm TNC220 on which I want to run KISS and thence the KA9Q +tcp/ip package. Unfortunately I don't have a KISS for the TNC. +Can anybody help. I would prefer the co-resident bootstrap with a +downloaded KISS module if possible. +ta Gareth +==== + +-- +Gareth Howell G6KVK @ IO91VX +ICL NS PNBC, England, SG1 1YB Tel:+44 (0)438 738294 +howellg%idec%ukc@mcvax.uucp, mcvax!ukc!idec!howellg@uunet.uu.net +G6KVK @ G4SPV (uk packet 144.650MHz) +#! rnews 710 +Path: alberta!mnetor!uunet!mcvax!ukc!stc!hilda +From: hilda@tcom.stc.co.uk ( Jeff Tracey ) +Newsgroups: rec.arts.sf-lovers +Subject: Thunderbirds are GO!!! +Keywords: FAB +Message-ID: <1503@arran.tcom.stc.co.uk> +Date: 2 Dec 87 10:54:39 GMT +Organization: STC Telecoms, London N11 1HB. +Lines: 14 + +A few quick trivia questions on Thunderbirds :- + +1) Does anybody know what the phrase 'FAB' stands for ??? + +2) What's the first mission that International Rescue accomplished ? + +3) What's the Butler's name on the Island AND who is his daughter ? + + +Regards, + +Steve Hillyer. || ...uunet!mcvax!ukc!stc!hilda +STC Telecommunications, Oakleigh Rd South, London N11 1HB. +Phone : +44 1 368 1234 x3358 +#! rnews 1159 +Path: alberta!mnetor!uunet!mcvax!ukc!stc!btnix!crouch +From: crouch@btnix.axion.bt.co.uk (Chris Rouch) +Newsgroups: comp.os.vms +Subject: callable TPU? +Keywords: TPU callable editor +Message-ID: <632@btnix.axion.bt.co.uk> +Date: 3 Dec 87 11:33:58 GMT +Organization: British Telecom Research Labs, Martlesham Heath, IPSWICH, UK +Lines: 17 + +I read somewhere that there is a callable version of EDT, available by using +EDT$EDIT(...). Does anyone know if there is a similar function for the TPU +editor and/or other commands such as MAIL, PRINT etc. If somebody could +also point me in the direction of the VMS manual which contains this +information (assuming there is one), I would be very grateful. + + Chris Rouch + +-------------------------------------------------------------------------------- +vax to vax (UUCP) CRouch@axion.bt.co.uk (...!ukc!btnix!crouch) +desk to desk RT3124, 310 SSTF, + British Telecom Research Laboratories, + Martlesham Heath, IPSWICH, IP5 7RE, UK. +voice to voice +44 473 646093 + + "Ours is not to look back, ours to continue the crack." +-------------------------------------------------------------------------------- +#! rnews 1090 +Path: alberta!mnetor!uunet!mcvax!ukc!stc!pete +From: pete@tcom.stc.co.uk (Peter Kendell) +Newsgroups: rec.music.classical +Subject: Durufle virgin seeks advice +Message-ID: <483@stc-f.tcom.stc.co.uk> +Date: 3 Dec 87 11:50:35 GMT +Organization: STC Telecoms, London N11 1HB. +Lines: 25 + + + Being curious, as the name was completely new to me, I borrowed + the Hyperion CD of Durufle's Requiem from my local public + library. I enjoyed it very much and would like to find out more + about him, so :- + + - What else has he written? (I believe he's not been very prolific) + + - What else has been recorded? + + - Is his other work similar to the Requiem; it is better, worse or + just different? + + - I thought I heard a Holst influence; is this typical? + + - Are there other 20th Century composers in a similar vein that I + should try? + + + +-- +------------------------------------------------------------------------------ +| Peter Kendell | +| ...{uunet!}mcvax!ukc!stc!pete | +------------------------------------------------------------------------------ +#! rnews 1235 +Path: alberta!mnetor!uunet!mcvax!ukc!stc!praxis!gauss!drb +From: drb@praxis.co.uk (David Brownbridge) +Newsgroups: comp.unix.wizards +Subject: Re: //host vs "mount point" +Message-ID: <1606@newton.praxis.co.uk> +Date: 3 Dec 87 12:42:36 GMT +References: <648@tut.cis.ohio-state.edu> <1668@tut.cis.ohio-state.edu> <38c15248.4580@hi-csc.UUCP> <9559@mimsy.UUCP> <411@PT.CS.CMU.EDU> +Sender: nobody@praxis.co.uk +Reply-To: drb%praxis.uucp@ukc.ac.uk(David Brownbridge) +Organization: Praxis Systems plc, Bath, UK +Lines: 19 + +In article <411@PT.CS.CMU.EDU> jgm@K.GP.CS.CMU.EDU (John Myers) writes: +>Just to add to the confusion, let me put in a plug in for the Carnegie-Mellon +>University Computer Science Department's syntax: +> +>/../host + +We built a system which also allowed super-super-roots and so on ad infinitum. + + /../NearbyHost + /../../OtherSite/host + /../../../OtherCountry/AnotherSite/host + +"/.." makes sense to me which is why I promoted it as the "University of +Newcastle upon Tyne Computing Laboratory's syntax" :-) Some old-timers must +remember the "Newcastle Connection" distributed UNIX system which Lindsay +Marshall and I wrote in 1981-2. + +"Not for the iron fist but for the helping hand" +[Billy Bragg/Oyster Band "Between The Wars"] +#! rnews 1785 +Path: alberta!mnetor!uunet!mcvax!ukc!stc!datlog!slxsys!jpp +From: jpp@slxsys.specialix.co.uk (John Pettitt) +Newsgroups: comp.unix.xenix +Subject: Re: smail2.5 +Summary: smail on xenix without writing new programs +Keywords: At last, a 'real' mailer for Xenix (are you listening SCO) :-) +Message-ID: <106@slxsys.specialix.co.uk> +Date: 3 Dec 87 06:44:07 GMT +References: <484@rel.eds.com> +Reply-To: jpp@slxsys.UUCP (John Pettitt) +Organization: Specialix International, London, UK. +Lines: 27 + +In article <484@rel.eds.com> bob@rel.eds.com (Bob Leffler) writes: +>During the last several weeks there have been numerous solutions posted to +>the net to resolved the interface problem with Xenix and smail 2.5. I +>have tried all the solutions that I am aware of and my conclusion for the +>best approach is a combination of two. + + lots of stuff about how to install smail deleted. + +I have just installed smail 2.5 on Xenix 386. The solution I used +here was to replace /usr/lib/mail/execmail with a link to (copy of) +/bin/smail. I also moved the old sco execmail to execmail.sco and used +it as the local delivery agent. The above will not work as it stands +because the command syntax for execmail is not the same as smail. This +can be corrected by swapping the meaning of the -F and -f switches in +smail (main.c and defs.h). The local delivery macro in defs.h should +be set to give /usr/lib/mail/execmail.com -f from to. With this +setup you get the sco mailer (mailx) and smail with both From and From: +lines correct. Also as execmail is still used for 'local' delivery +micnet (sco's RS232 "LAN") still works. + + + + +-- +John Pettitt G6KCQ, CIX jpettitt, Voice +44 1 398 9422 +UUCP: ...uunet!mcvax!ukc!pyrltd!slxsys!jpp (jpp@slxsys.co.uk) +Disclaimer: I don't even own a cat to share my views ! +#! rnews 1287 +Path: alberta!mnetor!uunet!mcvax!lambert +From: lambert@cwi.nl (Lambert Meertens) +Newsgroups: sci.math +Subject: Re: Least-squares fitting +Message-ID: <135@piring.cwi.nl> +Date: 5 Dec 87 14:41:10 GMT +References: <1823@culdev1.UUCP> +Organization: CWI, Amsterdam +Lines: 28 + +In article <1823@culdev1.UUCP> drw@culdev1.UUCP (Dale Worley) writes: +) Is is known how to perform least-squares fitting where the "error" is +) the perpendicular distance between the point and the line? + +This least-squares fit still passes through the "center of gravity" of the +data points, so assume that the data has been reduced such that the +averages of the x- and y-coordinates are both zero. Let the equation of +the line to be determined be + + x*(sin phi) - y*(cos phi) = 0, + +that is, it is the line making an angle phi with the x-axis. Put + + XX = SUM_i x[i]^2, + XY = SUM_i x[i]*y[i], + YY = SUM_i y[i]^2. + +Then tan(2*phi) = 2*XY/(XX-YY). + +This gives two solutions for phi. Take the one such that the point +(XX-YY, 2*XY) lies on the ray through the origin with angle 2*phi. +(Remark. It is possible to solve the coefficients for x and y +algebraically, without going through the arctan routine, but it is harder +then to get the signs correct.) + +-- + +Lambert Meertens, CWI, Amsterdam; lambert@cwi.nl +#! rnews 852 +Path: alberta!mnetor!uunet!mcvax!inria!imag!bordier +From: bordier@imag.UUCP (Jerome Bordier) +Newsgroups: comp.sys.mac +Subject: Re: Arabic Wordprocessing / Publishing +Message-ID: <2285@imag.UUCP> +Date: 4 Dec 87 10:24:35 GMT +Reply-To: bordier@imag.UUCP (Jerome Bordier) +Organization: IMAG, University of Grenoble, France +Lines: 14 + +"Winsoft", a small firm developping and selling software for the Macintosh, +has made "Wintext", a word processor fully compatible with the "Arabic +Macintosh+" (you have to obtain the Arabic keyboard distributed by Apple). +Their address is: + Winsoft + 34 boulevard de l'Esplanade + 38000 GRENOBLE France +Phone no.: 76.87.56.01 + +-- +Jerome BORDIER Laboratoire Structures Discretes Institut IMAG + B.P.68 - 38402 SAINT MARTIN D'HERES CEDEX France +E.Mail: +bordier@imag.imag.fr or {uunet.uu.net|mcvax}!imag!bordier +#! rnews 1182 +Path: alberta!mnetor!uunet!mcvax!inria!rouaix +From: rouaix@inria.UUCP (Francois Rouaix) +Newsgroups: comp.sys.amiga +Subject: POPCLI III Another Bug +Keywords: left-amiga-esc timing +Message-ID: <587@inria.UUCP> +Date: 5 Dec 87 17:45:19 GMT +Organization: INRIA, Rocquencourt. France +Lines: 20 + + + Well, it seems there is another bug in Popcli III. + Just try + 1> run popcli 30 + and then press Left-Amiga-Esc: the drive (where c: is) spins for + a moment and nothing happens. + The new 'screen-blanker' works all right but the automatic launch + is defeated. + Same for values of 10 and 40 seconds. + I didn't have time to figure out the limit value for which Popcli will + work (it works with default value and 240s). + Anyway, despite I *love* the new feature (let's keep the secret :-), + I'd rather have the old screen-blanker : at least I can sleep while + the Amiga is still on and working, and also it won't eat CPU-time I + need for Ray-tracing !! +-- + +*- Francois Rouaix / When the going gets tough, * +*- USENET:rouaix@inria.inria.fr \/ the guru goes meditating...* +* SYSOP of Sgt. Flam's Lonely Amigas Club. (33) (1) 39-55-84-59 (Videotext) * +#! rnews 539 +Path: alberta!mnetor!uunet!husc6!uwvax!rutgers!lll-lcc!ames!sdcsvax!ucsdhub!hp-sdd!ncr-sd!crash!pnet01!hhaller +From: hhaller@pnet01.cts.com (Harry Haller) +Newsgroups: comp.dcom.modems +Subject: Re: Facsimile on PC +Message-ID: <2140@crash.cts.com> +Date: 14 Dec 87 04:36:13 GMT +Sender: news@crash.cts.com +Organization: People-Net [pnet01], El Cajon, CA +Lines: 4 + +There is a board you can plug into the backplane that purports to give you +full FAX capability with editing. Of course, I forget the name, but if you +look in the literature... +() +#! rnews 1436 +Path: alberta!mnetor!uunet!husc6!uwvax!rutgers!lll-lcc!ames!sdcsvax!ucsdhub!hp-sdd!ncr-sd!crash!pnet01!dm +From: dm@pnet01.cts.com (Dan Melson) +Newsgroups: rec.aviation +Subject: Re: ARSA transition phraseology +Message-ID: <2141@crash.cts.com> +Date: 14 Dec 87 06:16:11 GMT +Sender: news@crash.cts.com +Organization: People-Net [pnet01], El Cajon, CA +Lines: 21 + +The question was asked why an ARSA controller might want to know your +destination. + +Actually, what they really want to know is where you're going *now*, like +'direct PMD' or 'following I5 northbound' (I have *no* idea of what type of +airspace that will take you through at any given altitude) or whatever course, +heading, or whatever you intend to take through the ARSA. + +Now, if you're going to get flight following, the controller is going to want +to know your complete route of flight for which you want flight following, +so that it can be entered into the machine and the autumated handoffs can be +used between sectors and facilities. + +As for why, that's very simple. For purposes of calling traffic, which I +consider to be sufficient, if no one else does. The same reason the +controller at the VFR tower asks your direction of departure. If nothing +else, the controller can always tell the left downwind departures 'traffic a +(whatever) reported 6 SE for a left base entry', or whatever is appropriate. + +MY opinions ONLY! + DM +#! rnews 2759 +Path: alberta!mnetor!uunet!mcvax!diku!iesd!jacob +From: jacob@iesd.uucp (Jacob stergaard B{kke) +Newsgroups: comp.ai +Subject: job search, Comp. eng. +Summary: I'm looking for a job +Keywords: Job, Computer. eng., Computer. sci., M.S. +Message-ID: <152@iesd.uucp> +Date: 2 Dec 87 13:20:07 GMT +Reply-To: jacob@iesd.UUCP (Jacob \stergaard B{kke) +Organization: Dept. of Comp. Sci., Aalborg University, Denmark (student) +Lines: 69 + +I'm looking for a job in Computer Engineering to begin around July +1988. I'm getting my Master of Science in Computer Engineering June +1988 and at present holding a degree equal to BS in Electronic +Engineering. My BS studies have included: + + Computer hardware (hands-on knowledge with mc68k), + Analog electronic + Control engineering (analog and digital control) + +My MS studies have included: + + Software development (man-machine interface, what people want + from programs) + Compiler construction (an expertsystem shell) + Program environment (for CCS programming) + Distributed operating systems (in UNIX) + Compiler mapping object-orinted language on parallel computers + +Furthermore I do have experience in conventional programming (PASCAL, +C, postscript, UNIX (awk, shell-scripts(C-shell) and yacc/lex) (and Basic)), +functional programming (LISP and ML) and logical programming (Prolog) +and knowledge about object-oriented programming. And I +have also attended courses in VLSI design, databases, etc. I have been +working with CDC under NOS/Telex, VAX 11/750 under Ultrix, SUN 3 under +Sun OS 4.3 (UNIX), MacIntosh (LISA) under Finder and IBM S36 under IBM +property operating system. + +My spoken English is excellent and my written English is satisfactory, +good knowledge of the Scandinavian languages (Danish (of course), +Swedish and Norwegian), some speaking and reading knowledge of German +and limited knowledge of French and Spanish (and Latin). + +I have 5 years experience in group project work in engineering and +computer scinence areas, broad social interest, good health. + +My interest include computer hardware and software, operating system +design, expertsystems, distributed, concurrency and teaching. + +I'm open on location (outside Denmark) but I have relatives or other +reasons to be especially intereted in: + + Canada (British Colombia or Toronto) + USA (New England or Pacific Coast) + Pacific (New Zealand or Oceania) + Thailand + Scotland (Highlands) + +I'll look forward to any reponds. + + Yours sincerely + + Jacob Baekke, Denmark + +For further information: + +Reply to: jacob@iesd.uucp, {...}!mcvax!diku!ised!jacob or + +at Univ: Jacob Baekke + S9D (in spring S10) + Strandvejen 19 + AUC + DK--9000 Aalborg + Denmark + +private: Jacob Baekke + Davids Alle 48 + DK--9000 Aalborg + Denmark + Tel. 45-(0)8102673 +#! rnews 2425 +Path: alberta!mnetor!uunet!husc6!bbn!rochester!cornell!uw-beaver!uw-june!uw-entropy!dataio!suvax1!hirayama +From: hirayama@suvax1.UUCP (Pat Hirayama) +Newsgroups: rec.arts.anime +Subject: Re: Speed Racer and the Mach 5 +Message-ID: <810@suvax1.UUCP> +Date: 14 Dec 87 05:28:25 GMT +References: <1103@jumbo.dec.com> +Organization: Seattle University, Seattle, WA. +Lines: 45 + +in article <1103@jumbo.dec.com>, schubert@jumbo.dec.com (Ann Schubert) says: +> Posted: Thu Dec 10 15:26:21 1987 +> +> +> THIS IS A RE-POST FROM REC.ARTS.TV +> +> +> In article <4540011@wdl1.UUCP> (James Y. Nakamura) writes: +> +> I have a question about Speed. We can't figure out all the neato gadgets his +> car had I think it went like: +> 1: Jacks that also made the car able to jump. +> 2: ??? +> 3: Saw blades that cut through stuff +> 4: Closes off the top so the Mach 5 becomes a sub.. +> 5: Homing pidgeon on a rope. +> + + Don't forget the special treads which would appear on his tires + to allow for climbing up rough ground or driving near vertical. + + Every now and then, I remember that they would add a new option + (boy, don't you wish your friendly neighborhood dealership would + offer some of these for your car?). Unfortunately, it has been + many years since I last saw Speed Racer, but I do remember one + episode which added little winglets which would come out from under- + neath the doors. This added a little gliding ability. Any one + else remember any? + + +> Also why did Speed have a G on his shirt? Go doesn't really wash with me and +> I don't know enough Japanese to equate letters. +> + + I used to know this but I can't remember anymore, though I + suspect that it might have to do with the original name of + the character/title of the show in Nihongo. Help anyone? + +******************************************************************************* +* --Pat Hirayama * +* --Seattle University * +* * +* "Yamato Hasshin!" - Kodai Susumu * +* * +******************************************************************************* +#! rnews 684 +Path: alberta!mnetor!uunet!mcvax!inria!axis!matra!godefroy +From: godefroy@matra.UUCP (Eric Godefroy) +Newsgroups: comp.unix.wizards +Subject: 8 bits on a pseudo-tty +Message-ID: <252@matra.matra.UUCP> +Date: 3 Dec 87 13:33:27 GMT +Reply-To: godefroy@matra.UUCP (Eric Godefroy) +Organization: Matra Datasysteme +Lines: 9 + +On 4.2 bsd, it seems difficult to set a pseudo-tty (ptyp / ttyp) in +the pass8 mode. Is it impossible really or how can I do that ? + +---------------------------------------------------------- + + Eric Godefroy UUCP: mcvax!inria!matra!godefroy + Matra Datasysteme Tel: (33-1) 30 58 98 00 + 1, av Niepce Fax: (33-1) 30 45 41 59 + 78180 Montigny-le-Bretonneux France +#! rnews 2663 +Path: alberta!mnetor!uunet!husc6!bbn!rochester!cornell!uw-beaver!uw-june!uw-entropy!dataio!suvax1!hirayama +From: hirayama@suvax1.UUCP (Pat Hirayama) +Newsgroups: rec.arts.sf-lovers,rec.arts.anime +Subject: Re: Old TV shows +Message-ID: <811@suvax1.UUCP> +Date: 14 Dec 87 05:54:39 GMT +References: <4254@dandelion.CI.COM> +Organization: Seattle University, Seattle, WA. +Lines: 40 +Xref: alberta rec.arts.sf-lovers:9224 rec.arts.anime:249 + +in article <4254@dandelion.CI.COM>, david@dandelion.CI.COM (David M. Watson) says: +> Xref: suvax1 rec.arts.sf-lovers:7102 rec.arts.anime:235 +> +> +> I have foggy but pleasant memories of three other converted Japanese +> - (not anime, but...) Ultraman! (Was it: "Hiyata! The beta capsule!"?) +> He was a large silver "good-monster" with a red light +> mounted on his chest that would blink whenever his batteries +> were getting low. And in his valiant, exhausting fights +> against the dinosaur types that frequently showed up to +> menace the World, he almost always came close to running out! +> And I remember a obligatory post-crisis trip to the jewelery +> store for Hiyata and friend! +> +> Would anyone like to refresh my memory about any of these three? +> + - Ultraman was one of several incredibly popular shows in Japan during + the late 60s/early 70s/early 80s. Actually, there were several shows + each featuring one or more of the "Ultra" brothers, of whom Ultraman + was the "leader/head/eldest (you get the idea)". There was also + Ultra 5 and a bunch of others which I can't remember and it would + take a long time to dig out the books. There was something of a + revival when UltraMan 80 (?) was released in Japan. + + Of these, I believe that only the original Ultraman was released + and dubbed for the American market. + + - By the way, Hayata would be the way to spell his name (though it + would be more accurately pronounced by you gaijin as "Hiyata". + + - Of course, there is nary a trace of him now in Japan. Programs + have this incredible tendency of grabbing hold of everyone, then + they drop it for something new. + +***************************************************************************** +* -Pat Hirayama * +* -Seattle University * +* * +* > No messages or quotes right now < * +***************************************************************************** +#! rnews 1910 +Path: alberta!mnetor!uunet!mcvax!enea!tut!santra!nispa +From: nispa@hutcs.hut.fi (Tapani Lindgren) +Newsgroups: comp.unix.wizards,comp.unix.questions +Subject: Unattended dumps (BSD4.3) +Message-ID: <9032@santra.UUCP> +Date: 4 Dec 87 15:19:19 GMT +Sender: news@santra.UUCP +Followup-To: comp.unix.wizards +Organization: Helsinki University of Technology, Finland +Lines: 28 +Xref: alberta comp.unix.wizards:5739 comp.unix.questions:4767 + +I have encountered a problem trying to make a shell script that would +make incremental backups at nighttime without operator attendance. +The problem results from dump(8) program requiring occasional +responses from the operator through /dev/tty. The script is +run from another script, /usr/adm/daily, under cron control and has no +controlling terminal, so it just hangs trying to read /dev/tty. It would +be ok if dump just aborted when facing a situation that would require +operator intervention. The script should never hang in a loop under +any circumstances, because /usr/adm/daily must do other things too +and finish after a reasonable time. + +Currently I have the dump script run a background subshell that sleeps for +an hour and then kills the dump script (if it still runs) and all dump +processes. This is very complicated, however, and the watchdog process +is almost 50% of the whole script. It is also very slow - I would +like it to stop immediately if it finds an error, report it to log file, +rewind the tape, and let /usr/adm/daily continue its work. + +Has anyone out there in the Netland have any suggestions of what to do? +Can yes(1) somehow be piped to a program that reads /dev/tty? +Could dump(8) be modified to abort at errors without any questions? +What kind of unattended backup systems do you have? + +--- +Tapani Lindgren, Helsinki Univ. of Technology, CS dept. +INTERNET: nispa@hutcs.hut.fi +UUCP: mcvax!santra!hutcs!nispa +BITNET: nispa%hutcs.UUCP@fingate.BITNET +#! rnews 2315 +Path: alberta!mnetor!uunet!husc6!bbn!oberon!pollux.usc.edu!kurtzman +From: kurtzman@pollux.usc.edu (Stephen Kurtzman) +Newsgroups: rec.food.cooking +Subject: Re: Cooking Wines +Message-ID: <5698@oberon.USC.EDU> +Date: 14 Dec 87 11:38:44 GMT +References: <4628@pyr.gatech.EDU> <10722@sri-unix.ARPA> <2028@ttrdc.UUCP> +Sender: nobody@oberon.USC.EDU +Reply-To: kurtzman@pollux.usc.edu (Stephen Kurtzman) +Organization: University of Southern California, Los Angeles, CA +Lines: 37 + +In article <2028@ttrdc.UUCP> levy@ttrdc.UUCP (Daniel R. Levy) writes: +> +>2) (more seriously) I've seen bottles of "wine for cooking" that have had +> salt (and vinegar?) added. These might be OK for sauces (yeah, the +> snootier gourmets wouldn't want anything to do with them) but they +> would obviously be horrible to drink. + +I think that these wines would be particularly bad for sauces that require +wine as a major component and require reducing the wine. There are two +reasons that come to mind: + +1) What is normally labeled as cooking wine is usually wine that is not good +enough to sell as table wine. If the taste is not the best, reducing it will +only concentrate its flaws. + +2) Cooking wines contain salt. Reducing a cooking wine will concentrate the +salt. This could really ruin the sauce. + +There best reason I have seen for using a good wine to cook with was given +by Alexis Bespaloff in the "New Signet Book of Wine", which states + + "Furthermore, it is actually uneconomical to buy cheap wine for cooking. + Say that an elaborate lobster dish calls for a spoonful or two of sherry + to heighten its flavor. A cook who runs out to buy a bottle of cheap + sherry will diminish the taste of an expensive and time-consuming dish with + a quarter's worth of wine. What's more, because the wine is a poor example + of its type, it may not be enjoyable to drink, so the spoonful of wine has, + in fact, cost the full price of the bottle." + +That is fairly sound reasoning. Of course, the last sentence does not +necessarily follow. You could keep the cheap wine around to diminish several +meals. + +BTW, I recommend the "New Signet Book of Wine" to anyone who wants to learn +more about wine. It is available for $4.50 as a paperback. Quite a value +when you compare it to the $20-or-more, glossy coffee-table wine books out +on the market. +#! rnews 1809 +Path: alberta!mnetor!uunet!mcvax!enea!luth!d2c-usg +From: d2c-usg@sm.luth.se (Ulrik"Rick"Sandberg) +Newsgroups: rec.music.misc +Subject: Re: Yes and ELP questions...... +Keywords: Tales from Topographic Oceans +Message-ID: <435@psi.luth.se> +Date: 4 Dec 87 18:28:34 GMT +References: <748@augusta.UUCP> <434@psi.luth.se> +Reply-To: Ulrik"Rick"Sandberg +Organization: University of Lulea, Sweden +Lines: 28 +UUCP-Path: {uunet,mcvax}!enea!psi.luth.se!d2c-usg + + +In article <434@psi.luth.se> I wrote: +>In article <748@augusta.UUCP> bs@augusta.UUCP (Burch Seymour) writes: +>>been looking for Tales on CD without success. To get to the point, is +>>it (Tales) on CD? +> +>One of my friends ordered it from a Recordshop in Gothenburg, but got +>the answer that it was sold out. However, they didn't say that the +>record isn't existing on CD. He was supposed to recieve it later. +>Any wiser of that? +> + +Correction: + +Received is spelled received, not recieved. :-) + +My friend told me that they said "Tales.. is not on CD." That's why he didn't +get it. Sorry for the confusing information. + + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~ ~ +~ Ulrik 'Rick' Sandberg d2c-usg@luth.UUCP (or) ~ +~ Computer Technology d2c-usg@psi.luth.se (or) ~ +~ University of Lulea {uunet,mcvax}!enea!psi.luth.se!d2c-usg ~ +~ Sweden ~ +~ phone: (0920)-977 90 (home) "I feel lost in the city..." ~ +~ -- Jon Anderson -- ~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +#! rnews 1128 +Path: alberta!mnetor!uunet!husc6!rutgers!lll-lcc!pyramid!decwrl!cssaus.dec.com!bell +From: bell@cssaus.dec.com (Peter Bell, SNA-2, Sydney) +Newsgroups: rec.music.classical +Subject: Hogwood +Message-ID: <8712141101.AA04882@decwrl.dec.com> +Date: 15 Dec 87 05:23:00 GMT +Organization: Digital Equipment Corporation +Lines: 15 + +I have just finished singing (in choir) under Hogwood, it was an experience. We +sang Schuberts Mass in G, (as Schubert wrote it, missing a few phrases of the +Credo). Hogwood knew exactly what he wanted, and worked till we did it right. +Then as we tidied up the last few problems, he would let us sing through whole +sections, then go back and point out all the problems. + +We also sang the Messiah (not with Hogwood unfortunately) the delight of +those performances was Elizabeth Cambells singing "He was despised..." + +In this performance the two trumpeters waited off stage until just before their +appearances in each half, the first trumpet parts were played by a large +trumpeter (in nice to see that Sydney musicains are not starving) on what +looked like a very small valved trumpet (trumpet in F??). + +Peter. +#! rnews 1252 +Path: alberta!mnetor!uunet!mcvax!dutrun!winffhp +From: winffhp@dutrun.UUCP (Frits Post and/or Andrew Glassner) +Newsgroups: comp.graphics +Subject: abstracts wanted +Keywords: ray tracing, abstracts +Message-ID: <190@dutrun.UUCP> +Date: 2 Dec 87 09:14:55 GMT +Organization: Delft University of Technology,The Netherlands +Lines: 21 + +I am preparing a list of technical memos, technical notes, +internal reports, and other such low-circulation documents +that deal with ray tracing. I'm interested in documents +both large and small. The documents need not be expressly +about ray tracing; the criterion is that the information in +the document be useful to ray tracing researchers in some way. + +If you have prepared such a document, please send me enough +information to digest it. That would at least include your +name and organization, the document's title, perhaps a reference +number, and (very important!) an abstract. + +All contributors will receive a complete copy of the final list. + +-Andrew Glassner + email until 15 December: uunet!mcvax!dutrun!frits + email after 15 December: glassner@unc.cs.edu , unc!glassner +-- + ...mcvax!dutrun!frits + Faculty of Mathematics and Informatics + Delft University of Technology +#! rnews 593 +Path: alberta!mnetor!uunet!mcvax!lambert +From: lambert@cwi.nl (Lambert Meertens) +Newsgroups: sci.lang +Subject: Re: Acquiring native accents +Message-ID: <136@piring.cwi.nl> +Date: 5 Dec 87 22:56:44 GMT +Organization: CWI, Amsterdam +Lines: 8 + +When I speak English I hear no Dutch accent in my voice. But if my voice +is recorded and played back to me I find the Dutch accent unmistakable. If +this phenomenon is a general one, it goes a good deal towards explaining +why adult learners of a new language do not fully master the native accent. + +-- + +Lambert Meertens, CWI, Amsterdam; lambert@cwi.nl +#! rnews 6735 +Path: alberta!mnetor!uunet!mcvax!philmds!leffe!janpo +From: janpo@leffe.UUCP (janpo) +Newsgroups: rec.music.misc +Subject: Re: Ideas for improving the debate (was: Digital vs. Analog music) +Summary: Digital versus Analog +Keywords: CDs expensive audiophile equip. fourier analysis +Message-ID: <43@leffe.UUCP> +Date: 4 Dec 87 13:55:04 GMT +References: <574@ucdavis.ucdavis.edu> <522@altura.srcsip.UUCP> <3051@batcomputer.tn.cornell.edu> +Organization: Philips I&E DTS Eindhoven +Lines: 123 + + + +1) Mr. Konar, press the 'n' key immediately! There's another arrogant + audiophile going to pollute the net with his view on the Digital vs. + Analog issue. + +2) I'm not very much acquainted with the news stuff on the net, but it + seems we don't receive the rec.audio newsgroup here in Europe. Can + something be done about that? + +3) Now let me come to the point. + In article <3051@batcomputer.tn.cornell.edu> eacj@batcomputer.tn.cornell + (Julian Vrieslander) he writes: +>Konar than goes on to comment about the "arrogance" of audiophiles who still +>prefer analog recordings to digital. He says the issue should be laid to rest +>, the implicit assumption being that the case has been proven that analog +>recording is obsolete. + +>I for one think that the issue is still an open (and interesting) one, but I +>am a bit surprised at how polarized and closed the recent comments to this +>thread have been. + +I agree with him, so let me do my bit now. A technology not being perfect +, or getting close to that, is still worth a discussion. Remember that it +took about a 100 years of thorough research from Edison's first grammophone +to the modern high quality turntables. Don't expect digital audio to be +perfect now only a few years after its introduction, no matter what the +commercial guys say. They are only interested in your hard-earned $$$$. + +Until now I have only been in the opportunity to make a good comparison +between a high-end turntable and some first genaration. I'll summarize +the pros and cons of which I think are important and which I can think of +now. Many of them are well known, others may not. + +PROS OF ANALOG: +- Cheap records. +- As John Vrieslander mentioned: More real, more spatious, more delicate, + more emotionally involving. I won't try to find other words for this + description 'cause I can't think of a better one. Unfortunately, this + can only be heard on good, say > $2k-$3k systems without an infinite + number of knobs, lights and other gadgets normally found in aeroplane + cockpits. + +CONS OF ANALOG: +- More hissy, rumble,scratches,sound degrading after many times of + playing the record. This counts less when you have good records + (Japanese ones are most often excellent but hard to get now.) and take + good care of them. +- No flat frequency response, especially at the low and high end. +- Phase distortion. +- Harmonic distortion increases with amplitude. + +PROS OF DIGITAL: +- Longer durability than records (?). Less hissy, no rumble or ticks of + scratches. +- Almost no phase distortion, flat frequency response within the audio + range. +- Easy to use. +- Slightly (!) more dynamic. Why only slightly? Well, the 96 dB dynamic + range theoretically possible with a CD is not very practical. In reality + it is compressed, as far as I know, to some 40-60 dB depending on the + music (Pop, Jazz, Classic) because: + a) No one wants to run continuously to his volume knob to adjust the + volume.If not compressed the music will either be banging through your + living room and of your neighbours or it will drown in the inevitable + background noise. + b) Studio equipment has a dynamic range of less than, say, 70 to 80 dB + when you assume the Signal to Noise ratio being equal to the dynamic + range. + c) Sound gets to distorted at low levels. (See also cons) + d) A dynamic headroom of 10 dB is desired. + With all this limitations the dynamic range of CD's is not much different + with that of a good record. +- Excellent bass response. Deep and well defined. +- Very stable stereo image. + +CONS OF DIGITAL: +- Expensive records. +- When listening to a CD, it seems as if there is no "space" around around + the instruments and voices. It sounds cold and not very lively. +- First generation players and the cheaper CD players nowadays suffer from + very distorted high tones. They sound harsh. Cymbals for instance sound + like someone is sawing them into pieces instead giving it a gentle hit + with a drum-stick. They do not sound crisp and clear. +- Distortion increases dramatically with lower amplitudes (!). It can be + more than 1.5 % at low levels. And it's a very nasty kind of distortion. +- CD players produce (digital) noise above the audio range. This noise itself + can not be heard but other audio equipment may suffer from intermodulation + distortion which brings this noise back in the audio range. +- Many CD players, especially the first ones, do not seem to be very reliable + mechanically. +- I've heard that digital audio recording is quite different from analog. + I mean in terms of how it has to be done properly. I don't mean the + equipment needed, that's quite obvious. Not all studio crew seem to know + how to make a good digital recording. Does anyone know more about that? + +Well, this must be enough stuff to think and talk about. + +The above mentioned cons of digital audio may be overcome in the latest +players but I have not had the opportunity until now to carefully listen +to them and to compare them. According to some serious audio magazines +available here in Holland they seem to be improved. Many top-of-the-line +models now have separated power supplys for the digital and analog +circuitry, opto-couplers between those circuits, an additional analog filter +to filter out > 20 kHz noise, 16 bit with with n times oversampling, stable +and rigid chassis and improved error correction. All this may have solved +(some) of the cons I mentioned but I'm not sure. Players which have one or +more of these improvements and thus may be of interest (At least for Julian +Vrieslander and me) are: Philips (Magnavox in the USA I believe) CD 650 and +CD 960, Nakamichi, Mission, Meridian and if memory serves me well, Acoustical +Research (Or Audio Research. Don't know anymore). The latter two may be +difficult to get in the USA, they are made in the UK. No doubt that there +are more good CD players but can't think of others now. These are the +players I consider buying when normal LP's are no longer available. + +Pooh! That was more than I intended to write but still far less than I can +tell about this stuff. + + Kind regards from + Jan Postma + + +And on the seventh day, God went surfing! +#! rnews 1424 +Path: alberta!mnetor!uunet!mcvax!cernvax!ethz!heiser +From: heiser@ethz.UUCP (Gernot Heiser) +Newsgroups: comp.emacs +Subject: Setting terminal-emulator's environment +Keywords: GNU Emacs function `terminal-emulator' +Message-ID: <261@bernina.UUCP> +Date: 5 Dec 87 13:42:37 GMT +Reply-To: heiser@ethz.UUCP (Gernot Heiser) +Organization: ETH Zuerich, Switzerland +Lines: 18 + + +Using the GNU emacs terminal-emulator to run interactive programs would be +quite limited if the parent emacs can't be used for editing (when the program +run under the emulator starts up an editor). While some programs (like `rn') +allow to explicitely specify the editor, a general solution would require to +specify `emacsclient' in the `EDITOR' environment variable of the process +running under the terminal emulator. + +Naturally this could be done by running the shell under the emulator, setting +the environment of the shell, and then running the program we are really +interested in. A better way would be to set the environment from the +`terminal-mode-hook'. Is there any means to achieve this???? (I'm running GNU +Emacs version 18.49.) +-- +Gernot Heiser Phone: +41 1/256 23 48 +Integrated Systems Laboratory CSNET/ARPA: heiser%ifi.ethz.ch@relay.cs.net +ETH Zuerich EARN/BITNET: GRIDFILE@CZHETH5A +CH-8092 Zuerich, Switzerland EUNET/UUCP: {uunet,...}!mcvax!ethz!heiser +#! rnews 1617 +Path: alberta!mnetor!uunet!mcvax!enea!sommar +From: sommar@enea.UUCP (Erland Sommarskog) +Newsgroups: rec.music.misc +Subject: Re: another net.question +Message-ID: <2496@enea.UUCP> +Date: 5 Dec 87 16:46:48 GMT +References: <251@ho7cad.ATT.COM> +Reply-To: sommar@enea.UUCP(Erland Sommarskog) +Followup-To: rec.music.misc +Organization: ENEA DATA Svenska AB, Sweden +Lines: 27 + +P.CLARK (prc@ho7cad.ATT.COM) writes: +> Should a band play the entire new album when they do a concert? + + +No, why should they? There may be songs on the album that are +very good listening to at home, but just doesn't make it live, +just as there are songs with the opposite character; good live, +but just a bore on disc. + +Deep Purple and Marillion and good example of extremes in both +ends. When I saw D.P. in February this year, they played three +of the ten songs from "The House of Blue Light", their latest +album. That is a quite decent product, but I didn't miss those +songs anyway. (I, and everyone else, would have been much more +disappointed if they had left out "Smoke on the Water".) + Marillion on the other hand; on the two tours they made after +"Misplaced Childhood", they insisted on playing entire album +as one long song. There are many parts on that album that just +becomes dead passages where nothing happens when they are played +live. ("Bitter Suite" and "Blind Curve" for instance.) Marillion +is no good live band, and playing obsolete material does not +make things better. +-- +Erland Sommarskog +ENEA Data, Stockholm +sommar@enea.UUCP + C, it's a 3rd class language, you can tell by the name. +#! rnews 4410 +Path: alberta!mnetor!uunet!mcvax!enea!sommar +From: sommar@enea.UUCP (Erland Sommarskog) +Newsgroups: rec.music.misc +Subject: Re: More than Yes +Message-ID: <2502@enea.UUCP> +Date: 5 Dec 87 19:18:59 GMT +References: <22034@ucbvax.BERKELEY.EDU> +Reply-To: sommar@enea.UUCP(Erland Sommarskog) +Followup-To: rec.music.misc +Organization: ENEA DATA Svenska AB, Sweden +Lines: 72 + +Grady Toss (ebm@ernie.Berkeley.EDU) writes: +>Whenever this newsgroup gets around to discussing 70's/80's fusion (the +>current go-round sparked by the proof that Yes is Best), the content +>seems to be limited to the same 5 or 6 groups (Yes, Rush, ELP, King +>Crimson, Pink Floyd, Genesis, etc.). + +Grady seems to be confusing the issue a bit here. He talks about fusion and +the mentions groups that belong(ed) to the symphonic-rock genre. (I prefer +that term instead of "progressive") For me "fusion" is a synonym with +jazz-rock. Anyway, that is more of question of semantics, the two genres +have a lot in common. (The main difference maybe being that symphony-rock +is European and fusion American.) + +The reason why these groups are being discussed the most is probably that +they have gained the greatest commercial succes. This may or may not +be correlated to the fact they are the best. + +>Doesn't (didn't) anyone listen to +>some of the (apparently) lesser-known fusion greats? Bands and artists +>like Arti + Mestieri, Brand X, Arthur Brown & Kingdom Come, Egg, Gilgamesh, +>Hatfield & The North, Henry Cow, Alain Markusfeld, National Health, PFM, +>Quiet Sun, Return to Forever, Seventh Wave, The Soft Machine, UK and +>Weather Report. + +Being quite fond of this kind of music, I feel obliged to comment. It's +a real mixture Grady presents and I must admit there are names I have +never heard. Anyway, I think he is a bit unfair, some of them have +certainly been discussed on the net. For instance, I posted a discograhpy +on Brand X some month ago. Some comments to the other names: + PFM (Premiata Forneria Marconi) have been mentioned from time to time, +the Italian answer on Genesis, which developed in a different way. Now +disbanded, I believe. One day or another may be I'll post a discography. + Quiet Sun. The band in which Phil Manzanera played before he joined +Roxy Music. Their "Mainstream" is rather like jazz, but not mainstream. + Seventh Wave. This is the name I never expected to see on the net! I +bought their "Things to Come" when I was 15 and I was really fond of it +then. These days I don't find that amount of synthesizers so exciting as +I did then. + Wheather Report. Quite well-known. But really, you do only need "Heavy +Wheather", the one with "Birdland". May be some more, "Mysterious Traveller" +perhaps, but then you'll find that they all sound the same. + +>As I said before, I find much of Yes and ELP to be very dull, and un- +>affecting. I like Rush, though more live than on record. + +As you can guess, I don't share Grady's view here. Yes has made good +music, yet never really touched my soul, probably due to their utterly +stupid and semi-religious lyrics. "Brain Salad Surgery" is a very good +record, the rest of what ELP have done is so-so. Rush don't turn me on +at all, on the other hand. The net discussion inspired me to try +"A Farewell to Kings" (A random choice). May be I would have liked them +10 years ago, but not today with those lyrics and that voice. + +>So, were Yes, ELP, Pink Floyd, King Crimson, Genesis and Rush really "it" +>as far as most progrock fans go, or did some of these "lesser known" artists +>(and all the others I forgot or never knew) filter out to larger audiences? + +Depends on how you define your terms here, but I can easily think +of more groups, some of them succesful, some of them not, some of them +good, some them not so good: +Kansas, Saga, Asia, Jethro Tull, Roxy Music, Gentle Giant, Van Der Graaf +Generator, George Duke, Billy Cobham, Al DiMeola, Herbie Hancock, Dixie +Dregs, Ange, (Mahavishnu) John McLaughlin, Santana, Bill Bruford etc + I think that most of these people have had their share of the discussion +on the net. So to conclude, I do not really share Grady's initial obser- +vation. However some particular groups are certainly being over-discussed, +namely Rush, Yes and recently also Pink Floyd. +-- +Erland Sommarskog +ENEA Data, Stockholm +sommar@enea.UUCP + C, it's a 3rd class language, you can tell by the name. +#! rnews 916 +Path: alberta!mnetor!uunet!mcvax!enea!tut!santra!clinet!waldo +From: waldo@clinet.FI (Tuomas Siltala) +Newsgroups: rec.music.synth +Subject: Siel DK80 sequenceer +Keywords: How to use? +Message-ID: <553@clinet.FI> +Date: 5 Dec 87 21:36:23 GMT +Reply-To: waldo@clinet.UUCP (Tuomas Siltala) +Organization: City Lines Oy, Helsinki, Finland +Lines: 17 + +My friend bought a Siel DK80 synthesizer and now he is wondering how +the sequencer in that machine works. + + +Unfortunately we don't have any manuals for it. + +Could somebody kindly send me information concerning this problem? + +Thank you! + +------------------------------------------------------------------------------ + + Tuomas Siltala Internet: waldo@clinet.FI + Kalevankatu 51 B 37 + SF-00180 Helsinki, Finland Telephone: +358-0-6947735 + +------------------------------------------------------------------------------ +#! rnews 1655 +Path: alberta!mnetor!uunet!mcvax!enea!luth!d2c-czl +From: d2c-czl@sm.luth.se (Caj Zell) +Newsgroups: rec.music.misc +Subject: Re: Black Sabbath songs +Message-ID: <437@psi.luth.se> +Date: 6 Dec 87 01:38:40 GMT +References: <1208@gumby.wisc.edu> <3590@h.cc.purdue.edu> +Reply-To: Caj Zell +Organization: University of Lulea, Sweden +Lines: 25 +UUCP-Path: {uunet,mcvax}!enea!psi.luth.se!d2c-czl + + +In article <3590@h.cc.purdue.edu> acu@h.cc.purdue.edu.UUCP (Floyd McWilliams) +writes: + +> While we're talking about Sabbath, does anyone know who does the +>vocals on "Solitute" (from the Master of Reality album) and "It's All Right" +>(from Technical Ecstasy)? It sure doesn't sound like the Oz... + +I would like to add another song:"Swinging The Chain" on _Never Say Die!_. +Who the hell does the vocals here? + +By the way,has anybody heard the new album? + + + XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX + X X + X X + X Caj Zell ________________________ X + X University of Lulea : : X + X Sweden : Jazz is not dead, : X + X : it just smells funny : X + X mail: d2c-czl@psi.luth.se : -Frank Zappa : X + X : : X + X -----------------------: X + X X + XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX +#! rnews 760 +Path: alberta!mnetor!uunet!mcvax!unido!gmdka!florin +From: florin@gmdka.UUCP +Newsgroups: comp.windows.x +Subject: C++ re-hacks of X11 include files - (nf) +Message-ID: <2800001@gmdka.UUCP> +Date: 3 Dec 87 13:16:00 GMT +Lines: 14 +Nf-ID: #N:gmdka:2800001:000:458 +Nf-From: gmdka!florin Dec 3 14:16:00 1987 + +Hi there, + +I'm actually working on C++ re-hacks of the X11 include files. +There are some problems with Xlib.h. In structures +Visual, XWindowAttributes and XColormapEvent there are variables named +``class'' and ``new'' which cause serious problems (C++ keywords) ! + +For the moment I've changed the names, but this is an awful hack. Does anybody +know a better solution ? + + -- Florin + +UUCP: ...!uunet!unido!gmdka!florin +X.400: florin@karlsruhe.gmd.dbp.de +#! rnews 4069 +Path: alberta!mnetor!uunet!mcvax!ukc!reading!onion!riddle!domo +From: domo@riddle.UUCP (Dominic Dunlop) +Newsgroups: comp.unix.xenix,comp.sys.att,comp.sys.intel +Subject: How to load AT&T 6300 Plus packages to generic UNIX V.3 +Summary: Here's a shell script to do it for you +Keywords: Intel, 386/ix, Microport, Prime +Message-ID: <522@riddle.UUCP> +Date: 4 Dec 87 17:47:52 GMT +Reply-To: domo@riddle.UUCP (Dominic Dunlop) +Followup-To: comp.unix.xenix +Organization: Sphinx Ltd., Maidenhead, England +Lines: 106 +Xref: alberta comp.unix.xenix:1170 comp.sys.att:1825 comp.sys.intel:379 + +[If there's a Microport newsgroup, it doesn't come here] + + Background + +AT&T's generic UNIX V.3 for the 80386 (as sold in binary form by AT&T, +Bell Technologies, Intel, Interactive Systems, Microport, Prime etc.) +will run binaries created for UNIX V.2 on the 80286. A large number of +packages exists for AT&T's 6300 Plus, an 80286-based system running V.2. +These can be run on 80386-based systems while you're waiting for +software authors to come up with native 80386 ports of their products. + + Problem + +You are supposed to load packages onto your 6300 Plus using the system's +administration procedures. These handle weird multi-volume cpio diskette +sets, which are a pig to load unless you have the installation software. +Which you don't if you're trying to load the software onto an 80386-based +system running 386/ix, Microport, or whatever. + + Solution + +Here's a shell script which does the job. If you want to know the details, +it reads 350k, starting at offset 9k, from each 360k diskette in the +installation set, piping the result into cpio -c. It the fires off the +Install program which should be part of the application package. As the +comments remark, there's not a lot of error checking, as it's essentially +a quick hack. Also, testing is about at the ``worked twice in a row'' +level. Despite all that, I hope it's useful to somebody out there. + +Dominic Dunlop +domo@sphinx.co.uk domo@riddle.uucp + +++++cut here++++++++cut here++++++++cut here++++++++cut here++++ +: +# load_script +# +# Shell script to load software packages delivered in AT&T PC +# 6300+ UNIX V.2 format on systems where the PC 6300+ +# installation procedure is not available (eg 386/ix). +# The script can be executed by any user who can read the raw +# diskette device. However, the root password is requested +# before files are moved to their final destinations if this +# script is not run by the super-user. +# +# Note that this script does NOT check that sufficient space is +# available to load the package. In general, your /usr file +# system should have at least (700 * diskettes_in_package) +# blocks free before installation. Note also that there is no +# check that the diskettes are in the correct format, or that +# they are inserted in the correct order. +# +# 871204 DFD Created + +# Change the following device assignment if the 360kB raw +# diskette device on your system has a different name. +DEV=${DEV-/dev/rdsk/f0d9dt} + +if [ ! -r $DEV -o ! -c $DEV ] +then + cat << E_O_F +Can't read $DEV. Check raw diskette device name and/or your +access permissions. +E_O_F +exit 1 +fi + +cd /usr/tmp +mkdir install 2>/dev/null +cd install +IT="the first diskette of the package" + +trap "echo Installation aborted.; rm -r /usr/tmp/install; exit 1" 2 15 +( + while echo "Insert $IT and hit return >\c" 1>&2 \ + && read ANS + do + IT="next diskette" + echo "The following files are being loaded:" 1>&2 + dd if=$DEV ibs=1k obs=5k skip=9 count=350 2>/dev/null + done +) | cpio -icvmudB 1>&2 + +chmod +x Install + +trap 2 15 + +cat << E_O_F +Files read from diskettes. You may remove the last diskette from +the drive. If you are not already logged in as the super-user, +Please enter the root password to continue with installation. +E_O_F +if su root -c ./Install +then + cat << E_O_F +Installation complete. You should execute + rm -r /usr/tmp/install +to remove installation scratch files at a convenient time. +E_O_F +else + cat << E_O_F +Installation failed. To retry, + su + cd /usr/tmp/install + ./Install +E_O_F +fi +#! rnews 1801 +Path: alberta!mnetor!uunet!mcvax!ukc!reading!onion!bru-me!ralph +From: ralph@me.brunel.ac.uk (Ralph Mitchell) +Newsgroups: comp.graphics,sci.space,sci.space.shuttle +Subject: Re: 3d digitized shuttle data +Message-ID: <338@Pluto.me.brunel.ac.uk> +Date: 4 Dec 87 09:49:43 GMT +References: <509@otto.cvedc.UUCP> +Reply-To: ralph@me.brunel.ac.uk (Ralph Mitchell) +Organization: Brunel University, Uxbridge, UK +Lines: 26 +Xref: alberta comp.graphics:1381 sci.space:3674 sci.space.shuttle:445 + +In article <509@otto.cvedc.UUCP> billa@otto.UUCP (Bill Anderson) writes: +>In article <> apollo@ecf.toronto.edu (Vince Pugliese) writes: +>> +>>As well I will be include a very simple C program, hacked together by fellow group member +>> [...] +> +>If anyone out there in netland converts this C program so that it can be +>run on suns, please post the results of your work to the net. + +It has already been done. The program should be in /usr/demo/SRC/shaded.c, +the shuttle data is in /usr/demo/DATA/space.dat. There are notes on running +it in /usr/demo/README. The program displays 2 windows with cursor lines, to +enable you to select the 3d viewpoint, and there's a pop-up menu for setting +fill style and colour, &c. For monochrome you need to select the "edges" (I +think) fill style or it'll look pretty wierd. Also, if your display surface +doesn't support hidden surface removal, you'll get a wireframe effect that +can be confusing to the eye. + +/usr/demo/DATA also contains data files for an icosahedron, a pyramid, a +ball and a Klein bottle. + +-- + From: Ralph Mitchell at Brunel University, Uxbridge, UB8, 3PH, UK + JANET: ralph@uk.ac.brunel.cc ARPA: ralph%cc.brunel.ac.uk@cwi.nl + UUCP: ...ukc!cc.brunel!ralph PHONE: +44 895 74000 x2561 + "There's so many different worlds, so many different Suns" -- Dire Straits +#! rnews 1156 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!bob +From: bob@its63b.ed.ac.uk (ERCF08 Bob Gray) +Newsgroups: rec.arts.sf-lovers +Subject: Re: SPACE WAR BLUES (was Re: Gibson) +Message-ID: <809@its63b.ed.ac.uk> +Date: 4 Dec 87 12:49:58 GMT +References: <8711211710.AA02986@decwrl.dec.com> +Reply-To: bob@its63b.ed.ac.uk (ERCF08 Bob Gray) +Organization: I.T. School, Univ. of Edinburgh, U.K. +Lines: 17 + +In article <8711211710.AA02986@decwrl.dec.com> boyajian@akov68.dec.com (JERRY BOYAJIAN) writes: +>(Oh, before anyone asks the obvious question, the author was Richard +>Lupoff, who is one of the best unknown science fiction writers around.) + +I find this statement hard to believe, based on the quality +of his book "Circumpolar". It is full of characters which +barely qualify as two dimensional, offensive racial stereotypes +and various other assorted characters whose collective IQ doesn't get +into double figures. I rated this book as -****. + +I cannot believe that someone who turned out such complete +drivel could improve enough in other books to even qualify +as average. + +I am however, willing to be surprised. What other books of +his would people recommend? + Bob. +#! rnews 1599 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!bob +From: bob@its63b.ed.ac.uk (ERCF08 Bob Gray) +Newsgroups: sci.misc +Subject: Re: Grey Goo that's too smart for its own good +Keywords: nanotechnology foresight drexler +Message-ID: <810@its63b.ed.ac.uk> +Date: 4 Dec 87 13:10:07 GMT +References: <799@sbcs.sunysb.edu> <2698@drivax.UUCP> <1063@sugar.UUCP> <2411@watcgl.waterloo.edu> <1445@m-net.UUCP> <1526@mmm.UUCP> <2783@drivax.UUCP> +Reply-To: bob@its63b.ed.ac.uk (ERCF08 Bob Gray) +Organization: I.T. School, Univ. of Edinburgh, U.K. +Lines: 23 + +In article <2783@drivax.UUCP> macleod@drivax.UUCP (MacLeod) writes: +>In article <1526@mmm.UUCP> cipher@mmm.UUCP (Andre Guirard) writes: +>>In article <1445@m-net.UUCP> russ@m-net.UUCP (Russ Cage) writes: +>>>In <2411@watcgl.waterloo.edu> kdmoen@watcgl.waterloo.edu (Doug Moen) writes: +>>>>[...] If it *does* turn out to be possible to build Grey Goo, +>>>>then by the time fabrication technology catches up, perhaps we can have +>>>>a wide spectrum of Goo killing techniques already available. +> +>Goo seems almost inevitable. It should not be a big problem, of itself; +>the definition of Goo (for those not familiar with the problem) is that +>of a nanomachine that will use any available energy and raw material to +>reproduce itself periodically. If it reproduces at 2x per year you have +>one problem, relatively minor; if it reproduces at 512x per minute, you have +>quite another. + +I can hear the squeals from the anti-nuclear type lobby already + + Can you PROVE it is safe? + Campaign against the Grey Goo! + prevent Nano-technology! + +and not a :-> in sight. + Bob. +#! rnews 3115 +Path: alberta!mnetor!uunet!mcvax!ukc!cheviot!robert +From: robert@cheviot.newcastle.ac.uk (Robert Stroud) +Newsgroups: comp.unix.wizards +Subject: Re: //host vs "mount point" +Message-ID: <2584@cheviot.newcastle.ac.uk> +Date: 4 Dec 87 16:22:51 GMT +References: <648@tut.cis.ohio-state.edu> <1668@tut.cis.ohio-state.edu> <38c15248.4580@hi-csc.UUCP> <9559@mimsy.UUCP> <411@PT.CS.CMU.EDU> <6769@brl-smoke.ARPA> +Reply-To: robert@cheviot (Robert Stroud) +Organization: Computing Laboratory, U of Newcastle upon Tyne, UK NE17RU +Lines: 62 + +In article <6769@brl-smoke.ARPA> gwyn@brl.arpa (Doug Gwyn (VLD/VMB) ) writes: +>In article <411@PT.CS.CMU.EDU> jgm@K.GP.CS.CMU.EDU (John Myers) writes: +>>Just to add to the confusion, let me put in a plug in for the Carnegie-Mellon +>>University Computer Science Department's syntax: +>>/../host +> +>Stolen from the Newcastle Connection. +> +>>"/.." is known as the "super-root". It seems logically consistent to me... +> +>So, what is the result of +> $ cd /.. +> $ pwd + +/.. of course!! + +If you add directories above root (and remember that with the Newcastle +Connection, /.. was just a directory rather than some mysterious +"super-root") so that it is possible for your current directory to +be in an uncle or cousin relationship with root (rather than a direct +descendent), then you have to modify the pwd algorithm accordingly. + +pwd assumes that if you go up the tree with ".." enough times you will +get to root. If your current directory is in a sideways relationship +to root, this assumption will no longer be valid. + +The modified pwd algorithm should work like this: + +(1) Go up the tree with .. from your current directory until you +find / or reach the base of the tree (a directory which is its own +parent). + +(2) If you didn't reach / in (1), then starting from / go up to +the base of the tree with .. and prefix the appropriate number of +/..'s to the string from (1). + +For example, after cd /../../C/D, step (1) will give /C/D and step (2) +will give /../.. so the answer is /../../C/D. + +This is relatively straightforward to implement. I've made the necessary +modifications to the System V /bin/pwd and sh (which has a built-in pwd) +for use with a kernel implementation of the Newcastle Connection. + +The tricky bit is getting the shortest possible pathname. For example, +if / corresponds to /../../A/B in the global naming tree, then after +cd /../C, the modified pwd algorithm would give /../../A/C which is +correct but redundant. (/../../A is the same as /.. if / is /../../A/B). + +This can be fixed if you keep a record of everywhere you visit in (1) and +stop in (2) when you reach somewhere you've visited before, but since in +an infinite naming tree this would require an infinite amount of storage +and isn't very efficient in any case, it is easier to simply implement +the algorithm given (which also requires an infinite amount of storage +in the general case of course!) and ignore this problem. + +Robert J Stroud, +Computing Laboratory, +University of Newcastle upon Tyne. + +ARPA robert%cheviot.newcastle@nss.cs.ucl.ac.uk +UUCP ...!ukc!cheviot!robert +JANET robert@newcastle.cheviot +#! rnews 835 +Path: alberta!mnetor!uunet!mcvax!ukc!dcl-cs!nott-cs!pyr1.cs.ucl.ac.uk!awylie +From: awylie@pyr1.cs.ucl.ac.uk +Newsgroups: comp.sys.ibm.pc +Subject: Re: Standard date bug +Message-ID: <39500002@pyr1.cs.ucl.ac.uk> +Date: 4 Dec 87 13:58:00 GMT +References: <7457@eddie.MIT.EDU> +Lines: 12 +Nf-ID: #R:eddie.MIT.EDU:7457:pyr1.cs.ucl.ac.uk:39500002:000:432 +Nf-From: pyr1.cs.ucl.ac.uk!awylie Dec 4 13:58:00 1987 + + +I have a Taiwanese XT clone with some strange BIOS and MSDOS 3.2 and the +bug has annoyed me some time. This is NOT the 'subtle' bug mentioned in +another reply, but a simple non-increment of the date at midnight. This +wreaks havoc with MAKE! + I shall try CLOCKFIX.SYS tonight. Thanks very much to the poster, his +was the only really useful solution proposed. + +Andrew Wylie +University of London Computer Centre + +awylie@uk.ac.ucl.cs +#! rnews 847 +Path: alberta!mnetor!uunet!mcvax!ukc!dcl-cs!bath63!sc_dra +From: sc_dra@ux63.bath.ac.uk (Dave Allum) +Newsgroups: comp.sys.atari.st +Subject: Hard Disk Optimisers +Summary: Recommendations wanted +Message-ID: <1972@bath63.ux63.bath.ac.uk> +Date: 4 Dec 87 15:53:49 GMT +Reply-To: sc_dra@ux63.bath.ac.uk (Dave Allum) +Organization: SWURCC, University of Bath, U.K. +Lines: 13 + + +Does anyone have any recommendations for and/or experience of hard disk +optimisers for the ST? + +The only ones I have come across are Simon Poole's DLII and Michtron's +Tune Up! (their exclamation mark, not mine). + +I have tried neither (DLII did some strange things with a ram disk I +tested it on, and I'd rather not pay for Tune Up! until I have some +favorable reports on it) and would be very interested in anyone's +experiences with the above or any other such beasts. + +Thanks. +#! rnews 1573 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!bob +From: bob@its63b.ed.ac.uk (ERCF08 Bob Gray) +Newsgroups: sci.physics +Subject: Re: GR question +Message-ID: <811@its63b.ed.ac.uk> +Date: 4 Dec 87 17:31:49 GMT +References: <4688@cit-vax.Caltech.Edu> <895@ubc-vision.UUCP> +Reply-To: bob@its63b.ed.ac.uk (ERCF08 Bob Gray) +Organization: I.T. School, Univ. of Edinburgh, U.K. +Lines: 28 + +In article <895@ubc-vision.UUCP> majka@ubc-vision.UUCP (Marc Majka) writes: +>would see the "poor fellow's" delta-t getting longer. The poor fellow +>crosses the Absolute Event Horizon in a finite amount of (his) time. +>The observer sees the poor fellow falling more and more slowly (while +>also seeing him getting exponentially red-shifted) toward r=2M, but +>never getting there. I liked the presentation of this in my GR textbook: + +The observer, if he waited around long enough, would also +see the black hole evaporate by Hawkins' radiation. + +But, from the point of view of the observer, the "poor fellow" +can never cross the event horizon before the hole evaporates +away from under him. + +Therefore, the "poor fellow" must observe one of two things. +Either he crosses the event horizon in a finite amount of +time, or he will observe the black hole to vanish as he +approaches. + +1. sets up a paradox, but 2. implies that anything falling +into a black hole can't get into the black hole before it +evaporates. i.e. the black hole can't form in the first +place. It just get very close to it. + + +Would someone please comment on the above. I am sure I must +be missing something. (I'm no physicist) + Bob. +#! rnews 2122 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!hwcs!zen!frank +From: frank@zen.UUCP (Frank Wales) +Newsgroups: news.config +Subject: Updated map entry for zen +Keywords: new host computer +Message-ID: <787@zen.UUCP> +Date: 3 Dec 87 22:20:52 GMT +Organization: Zengrange Limited, Leeds, England +Lines: 47 + + +It's a bit late again, we've been running the new system for about 3 +months now, but here is our updated map entry: + +#N zen +#S HP 9000 Model 840; HP-UX 1.1 (V.2) +#O Zengrange Limited +#C Julian Perry, Frank Wales +#E jules@zen.co.uk ...!mcvax!ukc!zen.co.uk!jules +#T +44 532 489048 +#P Greenfield Road, Leeds, West Yorkshire, England, LS9 8DB +#L 01 31 22 W / 53 47 42 N +#R +# +zen hwcs(DAILY) + + +Who we are and what we do: + +As a company, we produce custom solutions on hand-held and portable +equipment, primarily customising Hewlett-Packard hand-helds. For +example, we recently installed almost 6 000 HP-71 hand-held computers as +networked terminals in 430 DHSS offices as part of a Document Tracking +System developed by us to a DHSS specification. + +We're not just a software house, but also develop custom packaging and +electronics where necessary too. Our customers are primarily government +departments (here and abroad), but we have also produced products for +individual sale through dealers (such as the Zenwand-71 barcode wand for +the HP-71, which span off of the DHSS contract). + +Although our products are almost exclusively related to hand-helds, our +expertise stretches through to custom chip design and mainframe-hosted +software packages (mainly under Unix). As a consequence, we regard +ourselves as a solutions house, rather than being specific to software, +hardware, design or whatever. + +We have one office [in Leeds], have been around for seven years and +employ over 40 people at present. Is that a reasonable summary? + +Jules & Frank + +Julian Perry [ jules@zen.co.uk ...!mcvax!ukc!zen.co.uk!jules ] +Frank Wales [ frank@zen.co.uk ...!mcvax!ukc!zen.co.uk!frank ] +System Managers +Zengrange Limited Phone: +44 532 489048 ext 217 +Leeds, England. +#! rnews 1158 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!hwcs!jack +From: jack@cs.hw.ac.uk (Jack Campin) +Newsgroups: rec.music.classical +Subject: Re: re repeat repeating pieces +Message-ID: <1567@brahma.cs.hw.ac.uk> +Date: 4 Dec 87 18:18:28 GMT +References: <8712011820.AA18589@decwrl.dec.com> +Reply-To: jack@cs.glasgow.ac.uk (Jack Campin) +Organization: PISA Project, Glesga Yoonie +Lines: 13 +Summary: + +Expires: + +Sender: + +Followup-To: + + + +[ignore the above email address and use my signature] +I may have missed some of this thread, but I haven't heard anyone mention +Satie yet. His Vexations for piano is meant to be repeated 840 times +(it takes about 18 hours to perform). He also wrote some pieces of music +to be played in particular spaces - "Music for a Boardroom" is one +that comes to mind - which go round and round in circles. (I think that one +would produce some #@$% aggressive board meetings). +-- +ARPA: jack%cs.glasgow.ac.uk@nss.cs.ucl.ac.uk +JANET:jack@uk.ac.glasgow.cs USENET: ...mcvax!ukc!cs.glasgow.ac.uk!jack +Mail: Jack Campin, Computing Science Department, University of Glasgow, + 17 Lilybank Gardens, Glasgow G12 8QQ, Scotland (041 339 8855 x 6045) +#! rnews 1124 +Path: alberta!mnetor!uunet!mcvax!ukc!stl!dww +From: dww@stl.stc.co.uk (David Wright) +Newsgroups: comp.os.vms +Subject: Problem with VMS 4.6 if your uVAX has EMULEX CS02's +Message-ID: <596@acer.stl.stc.co.uk> +Date: 4 Dec 87 21:54:38 GMT +Reply-To: dww@stl.UUCP (David Wright) +Organization: STL,Harlow,UK. +Lines: 16 + +Our System Manager has reported that there is a problem with using EMULEX CS02 +QBUS comms cards which are not at the latest revision level, under VMS 4.6. +These cards appeared to work fine under VMS 4.5 and earlier. + +The EMULEX CS02 card, configured as two DHV-11 8-line muxs, gives phantom +devices when running SHOW DEVICE. For example, TXC0 to TXC7 become TXC0 to +TXC15. There are problems in using the lines - for example Control-Y acts +on the group of lines not just one! There are other problems known to EMULEX. + +The solution is to upgrade the firmware PROM on the card to at least +revision P. Emulex may make a charge for this. + +-- +Regards, + David Wright STL, London Road, Harlow, Essex CM17 9NA, UK +dww@stl.stc.co.uk ...uunet!mcvax!ukc!stl!dww PSI%234237100122::DWW +#! rnews 384 +Path: alberta!mnetor!uunet!mcvax!ukc!stc!root44!jgh +From: jgh@root.co.uk (Jeremy G Harris) +Newsgroups: comp.sys.amiga +Subject: New Kickstart +Keywords: Kickstart workbench janus +Message-ID: <489@root44.co.uk> +Date: 4 Dec 87 19:05:27 GMT +Organization: Root Computers Ltd., London, England +Lines: 3 + +Will the Workbench-less Kickstart initialise Janus? +-- +Jeremy Harris jgh@root.co.uk +#! rnews 1018 +Path: alberta!mnetor!uunet!mcvax!unido!stollco!til +From: til@stollco.UUCP (tilgner) +Newsgroups: sci.astro +Subject: The current state of Hubble Constant? +Keywords: Cosmology +Message-ID: <142@stollco.UUCP> +Date: 5 Dec 87 18:32:42 GMT +Organization: Stollmann Gmbh, D 2000 Hamburg 50 +Lines: 17 + +I am just preparing a 'semi-popular' lecture on how the +value of the Hubble Constant is determined. + +As is generally +known, the values of different authors fluctuates between +ca. 50 to 100 km/(sec Mpc). The latest discussion of this +problem which I know of is M. Rowan-Robinson's book +"The Cosmological Distance Ladder" (Freeman 1985). He +advocates 67 km/(sec Mpc) after a detailed discussion of +the different distance indicators. + +Now I would like to know: What is the current state of +affairs? The responses of the advocates of the various +values, for example by Sandage & Tammann or de Vaucouleurs +(= the grand old men of this topic)? Somehow I missed +their reactions. Can anybody give me a hint via e-mail? +I'll summarize. +#! rnews 2734 +Path: alberta!mnetor!uunet!mcvax!enea!ttds!draken!zap +From: zap@draken.nada.kth.se (Svante Lindahl) +Newsgroups: comp.unix.wizards,comp.emacs +Subject: Re: Emacs csh alias +Message-ID: <235@draken.nada.kth.se> +Date: 6 Dec 87 07:00:31 GMT +References: <10672@brl-adm.ARPA> +Reply-To: zap@nada.kth.se (Svante Lindahl) +Followup-To: comp.emacs +Organization: The Royal Inst. of Techn., Stockholm +Lines: 49 +Xref: alberta comp.unix.wizards:5741 comp.emacs:2402 + +[Warning: Extensive inclusion, but I have included a new newsgroup in + the newsgroups-line, and directed followups to it (comp.emacs)] + +In article <10672@brl-adm.ARPA> dsill@NSWC-OAS.arpa (Dave Sill) writes: +>I've been trying to set up a C-Shell (4.2 BSD) alias for Emacs (GNU +>17.64, not that it matters) which, when run the first time will +>actually run Emacs, but after suspending Emacs with C-z, will bring +>the background Emacs job to the foreground. The catch is that I'd +>also like the alias to re-load emacs if I exit with C-x C-c. Simply +>stated, I want an alias named "emacs" which will load Emacs if it +>isn't already loaded, but will foreground a background Emacs if one +>exists. +> +>I know I could do this with a script (if I assume the Emacs job is +>always job %1), but I'd prefer an alias since they're faster. It +>would be especially nice to determine which background job was the +>Emacs job and foreground *it*, instead of just assuming job %1. +> +>Any ideas or alternate approaches? Should I just put up with the +>occasional "fg: No such job." message? + +Here is something which should do part of what you want. It doesn't +accomplish to start a new emacs process if you exited the last one +with C-x C-c - unless the first one had never been suspended! +Whenever you get "fg: No such job" just type ``i!!'', reinvoking the +commandline prefixed with an "i", "iemacs" standing for "init emacs". + +alias emacs iemacs +alias iemacs 'alias emacs remacs; "emacs" \!* ; alias emacs iemacs' +alias remacs fg %emacs + +Here we use a special version of suspend-emacs, that will look for a +file ".emacs_pause" in the user's home directory when emacs is +resumed. In this file suspend-emacs expects to find the current +working directory and an optional "command line" that is parsed like +the initial command line. Very useful! +This could be done using "suspend-resume-hook", but the hook wasn't +available in 17.?? when this was first implemented here. + +These are the aliases I use together with the special version of +suspend-emacs. + +alias emacs iemacs +alias remacs 'echo `pwd` \!* >\! ~/.emacs_pause ; %emacs' +alias iemacs 'alias emacs remacs; "emacs" \!* ; alias emacs iemacs' +alias kemacs 'alias emacs iemacs; remacs -kill' + + +Svante Lindahl zap@nada.kth.se uunet!nada.kth.se!zap +#! rnews 2721 +Path: alberta!mnetor!uunet!mcvax!diku!iesd!jpc +From: jpc@iesd.uucp (Jens P. Christensen) +Newsgroups: comp.unix.questions,comp.unix.wizards,sci.math.stat +Subject: Problems with S statistical package +Summary: Cannot make S work properly on Sun-3 +Keywords: S AT&T Sun-3 SunOS 3.4 +Message-ID: <162@iesd.uucp> +Date: 5 Dec 87 19:41:09 GMT +Reply-To: jpc@iesd.UUCP (Jens P. Christensen) +Followup-To: comp.unix.questions +Organization: Dept. of Comp. Sci., Aalborg University, Denmark +Lines: 58 +Xref: alberta comp.unix.questions:4768 comp.unix.wizards:5742 sci.math.stat:213 + +Could anyone please shed light on a problem I have in compiling the S +statistical package from AT&T on our Sun-3 system: + +System specifics: Sun 3/260 under SunOS 3.4 using the m4 macro +processor supplied with the S system. S version date: Fri Feb 28 1986 + +Using the hints on compiling with BSD4.2 systems I only get apparently +harmless warnings under the compilation. This could for example be: + +Warning on line 84 of hcp.f: local variable i never used +Warning on line 96 of stems.f: statement cannot be reached +f77: Warning: File with unknown suffix (/usr/local/src/s/S/newfun/lib/grz) + passed to ld +or +"dprint.c", line 20: warning: illegal combination of pointer and integer, op = + +Furthermore there are problems with the utility routine scandata.C, which +fails with error: too many local variables. This is fixed by making the +declaration of "table" global. Not pretty, but it works. + +These are all the kinds of problems that appear during the +compilation, and it *will* result in an executable, except.... +The f...ing system doesn't even know how to add two numbers, as seen in +the following: + +One-time initialization for new S user in /usr.MC68020/iesd/tap/jpc ... +Directories swork and sdata created +> 1 + 2 +Bad operator: + +Error in + +> + +Running the tests supplied with the system ($A/DOTEST ALL) will not +give better results. This is an excerpt from $TEST/current/apply: + +> prefix("apply.") # test of apply and multivariate stuff, some time-series +> $Random.seed_c(57,0,3,0,0,0,49,16,0,0,0,0) # to initialize at same spot +> matr_matrix(rnorm(100),20,5) +Invalid distribution: rnorm +Error in rnorm +Dumped +> print(cm_apply(matr,2,"mean")); apply(matr,2,"var") +apply.matr not found +Dumped + . + . +and more depressing errors... +Why does the prefix command work, while the matr_matrix(rnorm... stuff don't? + +So, have *anybody* made this run on a Sun system, and how did you do it? +All suggestions or pointers to which direction I should go, are welcome. + +regards, +-- +Jens Peter Christensen jpc@iesd.uucp +Department of Math. and Computer Science {...}!mcvax!diku!iesd!jpc +Aalborg University Centre +Denmark +#! rnews 1496 +Path: alberta!mnetor!uunet!mcvax!lambert +From: lambert@cwi.nl (Lambert Meertens) +Newsgroups: sci.math.symbolic +Subject: Bug in Macsyma SOLVE +Message-ID: <137@piring.cwi.nl> +Date: 6 Dec 87 21:50:53 GMT +Organization: CWI, Amsterdam +Lines: 39 + +This is UNIX MACSYMA Release 309.2. + +(c1) x^12-12*x^11+48*x^10-40*x^9-193*x^8+392*x^7+44*x^6+8*x^5-977*x^4 + -604*x^3+2108*x^2+4913; + + 12 11 10 9 8 7 6 5 4 +(d1) x - 12 x + 48 x - 40 x - 193 x + 392 x + 44 x + 8 x - 977 x + 3 2 + - 604 x + 2108 x + 4913 + +(c2) solve(%); + 6 5 4 3 2 +(d2) [0 = - x + 12 x - 47 x + 188 x - 527 x - 4913] + +That looks wrong, but let's check if it factors (d1): + +(c3) part(%,1,2); + 6 5 4 3 2 +(d3) - x + 12 x - 47 x + 188 x - 527 x - 4913 + +(c4) gcd(%,d1); + +(d4) 1 + +No, it does not. Let's have a look at the real roots of (d1) and (d3): + +(c5) realroots(d1)$ %,numer; + +(d6) [x = - 1.960768669843674, x = - 1.544090360403061, x = 3.544090360403061, + x = 3.960768669843674] +(c7) realroots(d3)$ %,numer; + +(d8) [x = 5.472395747900009, x = 7.766151040792465] + +Way off. + +-- + +Lambert Meertens, CWI, Amsterdam; lambert@cwi.nl +#! rnews 989 +Path: alberta!mnetor!uunet!mcvax!unido!tub!actisb!bernd +From: bernd@actisb.UUCP (Gunter Nitzler) +Newsgroups: comp.sources.bugs +Subject: Re: Starchart printing problem +Message-ID: <116@actisb.UUCP> +Date: 6 Dec 87 15:55:23 GMT +References: <3554@ames.arpa> +Reply-To: bernd@actisb.UUCP (Bernd-Gunter Nitzler) +Organization: Actis in Berlin GmbH, W. Germany +Lines: 19 + +In article <3554@ames.arpa> yee@ames.UUCP (Peter E. Yee) writes: +>I compiled and ran the starchart program. The starpost version prints out +>the outline of the chart and the legend. Nothing more. No stars, no planets, +>no nebulas. Nothing. Is it just me, or has anyone else had this problem? + +I had the same problem and have found two bugs: + +In starchart.c, line 243 old: + char ras[2], .... +new: + char ras[20], ... + +In starchart.c, line 757 old: + sscanf(cbuf, "%*5s%f%f%f %[^\n]", &ra, &de, &sc, legend); +new: + sscanf(cbuf, "%*5s%lf%lf%lf %[^\n]", &ra, &de, &sc, legend); + +This two changes fixes the bugs. +Bernd. +#! rnews 2244 +Path: alberta!mnetor!uunet!mcvax!enea!tut!santra!jmunkki +From: jmunkki@santra.UUCP (Juri Munkki) +Newsgroups: comp.sys.mac +Subject: Color CopyBits Is Too Slow! +Keywords: Mac II Color QuickDraw Animation Speed Optimization +Message-ID: <9130@santra.UUCP> +Date: 6 Dec 87 21:13:10 GMT +Organization: Helsinki University of Technology, Finland +Lines: 76 + + +I experimented with offscreen pixmaps today. It seems that Color +Quickdraw is very flexible, but too slow for good animation. Most of the +overhead comes from color matching and conversion. I guess I could write +my own color matching routine, but I think there should be a fast way to +do a simple copy operation. + +In most painting programs the actual painting could be done on an +offscreen bitmap with the same color table as the best gDevice. + +It takes about twice as much time to do a copybits in srcCopy mode than +it takes in the srcXor mode. Below is a short program that draws to an +offscreen pixmap and then copies it back to the screen. Try different +transfer modes and note the speed difference. The code is written in LS +C 2.13. Even srcXor, which is the fastest usable mode, is too slow for +really high quality animation. + +How can it be done faster? + +#include +#include +#include +#include + +WindowPtr onScreen; +CGrafPtr offS; +RGBColor temp; +PixMapPtr offP; + +void main() +{ + int i; + + InitGraf(&thePort); InitCursor(); + InitFonts(); InitWindows(); + + onScreen=GetNewWindow(1000,0L,-1); + + offS=(CGrafPtr)NewPtr(sizeof(*offS)); + + OpenCPort(offS); + HLock(offS->portPixMap); + offP=*(offS->portPixMap); + + SetRect(&offP->bounds,0,0,256,256); + PortSize(256,256); + offP->rowBytes=32768L+256; + + offP->baseAddr=NewPtr(65536L); + + EraseRect(&offS->portRect); + temp.blue=65535; + temp.red=0; + temp.green=0; + RGBForeColor(&temp); + for(i=0;i<256;i+=4) + { MoveTo(i,0); + LineTo(255-i,255); + } + + SysBeep(10); + HideCursor(); + for(i=100;i;i--) + CopyBits(&((GrafPtr)offS)->portBits,&onScreen->portBits, + &offS->portRect,&offS->portRect,srcXor,0); + SysBeep(10); + while(!Button()); +} + +Juri Munkki +jmunkki@santra.hut.fi +jmunkki@fingate.bitnet +lk-jmu@finhut.bitnet + +P.S. The window is longword aligned and a color table was copied from the + system file. +#! rnews 617 +Path: alberta!mnetor!uunet!mcvax!enea!chalmers!benke +From: benke@chalmers.UUCP (Bengt-Eric Ericson) +Newsgroups: comp.sys.ibm.pc +Subject: Re: WARNING! FASTBACK may corrupt your hard disk! +Message-ID: <2239@chalmers.UUCP> +Date: 6 Dec 87 21:16:48 GMT +References: <703@vaxine.UUCP> <3225@bnrmtv.UUCP> <7024@sunybcs.UUCP> +Reply-To: benke@chalmers.UUCP (Bengt-Eric Ericson) +Organization: Dept. of CS, Chalmers, Sweden +Lines: 3 +Keywords:Computer Shopper + + +In some article in this group there is said something about +"Computer Shopper". Is this a magazine or what? Please +enlight us guys here in the land of Polar bears. :-) +#! rnews 2432 +Path: alberta!mnetor!uunet!mcvax!ukc!eagle!kjws +From: kjws@eagle.ukc.ac.uk (K.J.W.Smithers) +Newsgroups: comp.sys.amiga +Subject: Re: A2090A HD controller +Message-ID: <4038@eagle.ukc.ac.uk> +Date: 6 Dec 87 14:46:00 GMT +References: <5474@oberon.USC.EDU> <6575@ccicpg.UUCP> <2903@cbmvax.UUCP> +Reply-To: kjws@ukc.ac.uk (K.J.W.Smithers) +Organization: Computing Lab, University of Kent at Canterbury, UK. +Lines: 56 +Summary: + +Expires: + +Sender: + +Followup-To: + + +In article <2903@cbmvax.UUCP> you write: +> +>This is one of the things that the updated hddisk device I announced +>awhile ago (and will mail to people over usenet) fixes. If you don't +>have have a 2090 card, the software that comes with your 2090 is +>the new driver, so it will work fine in overscan. +>-- +>andy finkel {ihnp4|seismo|allegra}!cbmvax!andy +>Commodore-Amiga, Inc. +> + +I have an A2090 card and a CSA68020/68881 board with no 32 bit ram. + + They will Not work together. (but both work seperately) + +I think the driver (hddisk) is dated 1986 , is this the latest driver? +(If not could you please e-mail me the latest version) + +The problem is when I run binddrivers that task stops, (binddrivers +never exits). It seems to fallover on a particular call to execbase. +The last instruction (displayed by MetaScope) is mov a2,(a0) + +If i move the hddisk from expansion draw , to hddisk.device in the +devs draw, i can mount the harddisk (dh0:) , but when i do a +cd dh0: , the cd command displays 'Cant find dh0:' + +I am running morerows, 672*266 on a B2000 rev 4.0 board (pal) with +2Mbytes expansion ram , 2*3.5inch drives, and (hopefully) A2090 + +20 Mbyte hard disk, and a CSA 68020/68881 board. + +I have also done the wire-link modification to the main B2000 board, +as required by CSA for the 68020 board on Rev4.0 and later boards. + +Slots are as follows :- + + I I E E E M H 6 + B B M M M E A 8 + M M P P P M R 0 + T T T O D 2 + Y Y Y R D 0 + Y I C + S P + K U + + Thanks in advance for any help + + Kit Smithers + +____________________________________________________________________________ + Kit Smithers kjws@ukc.ac.uk + kjws@ukc.UUCP + !mcvax!ukc!kjws + +The man who can not stay fast and hard at the same time ! +Live for ever, or die in the attempt. +______________________________________________________________________________ +#! rnews 1572 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!ajcd +From: ajcd@its63b.ed.ac.uk (Angus Duggan, Department of Computer Science, University of Edinburgh,) +Newsgroups: rec.games.hack +Subject: pickup option - suggestion +Keywords: pickup HACKOPTIONS +Message-ID: <813@its63b.ed.ac.uk> +Date: 6 Dec 87 11:47:56 GMT +Reply-To: ajcd@its63b.ed.ac.uk (Angus Duggan) +Organization: I.T. School, Univ. of Edinburgh, U.K. +Lines: 23 + +Here's a suggestion for an improvement (at least I think it is :-) to the +"pickup" option in nethack, which someone who is familiar with the source +code might like to implement - + +Make the "pickup" option a composite option like "packorder", and re-write +the picking up code so that the types of objects specified will be +automatically picked up. All other objects could still be picked up by ','. + +e.g. "pickup:?+/=!)" would pick up scrolls, spellbooks, wands, rings, + potions, and weapons. + +This would be useful for those of us who don't like carrying hoards of +gold around, and also to prevent picking up dead cockatrices while still +picking up other objects. + +BTW, does anyone know what the options "null" and "news" do? +-- +Angus Duggan, Department of Computer Science, University of Edinburgh, +James Clerk Maxwell Building, The King's Buildings, Mayfield Road, +Edinburgh, EH9 3JZ, Scotland, U.K. +JANET: ajcd@uk.ac.ed.ecsvax ARPA: ajcd%ecsvax.ed.ac.uk@cs.ucl.ac.uk +USENET: ajcd@ecsvax.ed.ac.uk UUCP: ...!seismo!mcvax!ukc!ecsvax.ed.ac.uk!ajcd +BITNET: psuvax1!ecsvax.ed.ac.uk!ajcd or ajcd%ecsvax.ed.ac.uk@earn.rl.ac.uk +#! rnews 4243 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!simon +From: simon@its63b.ed.ac.uk (ECSC68 S Brown CS) +Newsgroups: comp.lang.c +Subject: Re: stdio error detection +Message-ID: <814@its63b.ed.ac.uk> +Date: 6 Dec 87 17:35:07 GMT +References: <10649@brl-adm.ARPA> +Reply-To: simon%lfcs.ed.ac.uk@nss.cs.ucl.ac.uk (Simon Brown) +Organization: LFCS, University of Edinburgh +Lines: 87 + +In article <10649@brl-adm.ARPA> dsill@NSWC-OAS.arpa (Dave Sill) writes: +>>I used to be rather fond of C, but this error stuff is quite +>>incredibly bad. The problem isn't really the language; it's +>>the libraries. +> +>Rather than messing with errno, I think a new variable, say, liberr, +>should be used. An include file, say liberr.h, could contain macro +>definitions for the various types of errors. A macro named LIBERR +>could also be defined in liberr.h so code could be written that would +>take advantage of liberr if it was available or handle errors in the +>usual way if it's not. Even better would be to have LIBERR be a +>predefined macro like ANSI, unix, vax, et cetera. +> + +This still has the same problem as with "errno"- namely that you're trying +to describe a general ``error condition'' using a single number! I'm told +that VMS (but it's a good idea for all that...) provides a stack of error +values which allows a program to search backward to find out what the "real" +error was, depending on what kind of detail is required. If you have several +levels of library calls between you and the system call that failed, this +can be extremely useful- it's not really much use having an error-value +if you can't even tell what system call it came from (let alone what parameters +were *passed* to that system call to cause it to fail!). + +A *decent* error-returning mechanism would describe: + + 1. What call (syscall or library call) failed. + This could be a number- you could use something like internet + addressing to put some kind of structure into it: + libc.stdio.fopen + 2. Why it failed. + Simple E-numbers will do for this (although I suppose they'd + have to be grouped for different libraries): + E_STDIO.E_CANNOT_OPEN_FILE + 3. What value it returned. + (FILE *)NULL + 3. What parameters were passed to it. + This is the most difficult one, because it would have to have + some kind of idea as to the types involved. It could (I suppose) + deal only with string types (and convert any other type into + "printable" form by doing the equivalent of sprintf()'ing it). + It also has to be a "list", which means it would probably have + to be done using something like "argc,argv": + argc: 2 + argv: "mumble.splat", "r" + +If the error is not "dealt with", then this information should propogate +down (together with the info from the callee's failure), and so on... + +So, If you do a + fopen("mumble.splat","r") +and it fails, then the following would be left on the stack (in some format +or other) to be dealt with by some error-diagnosing function: + + kernel.open: + param 1: "mumble.splat" [string] + param 2: 0 [int] + returns: -1 [int] + error: E_KERNEL.ENOENT + libc.stdio.fopen: + param 1: "mumble.splat" [string] + param 2: "r" [string] + returns: 0 [FILE *] + error: E_LIBC.E_STDIO.E_CANNOT_OPEN_FILE + +The error-diagnosing stuff could then print something *useful* such as + stdio fopen: couldn't open file "mumble.splat" for reading, because: + kernel open: no file or directory "mumble.splat" + +(and of course the format of these messages could be user-configurable, so +that noddies would just get the information they need, whereas people who +understand what they're doing could get reams and reams of info- just by setting +some environment parameter to the appropriate value). + +Of course, all this stuff would have to be known by the compiler, and I'm sure +it'd be dead slow to execute! + +-- +-------------------------------------------------- +| Simon Brown | +| Laboratory for Foundations of Computer Science | +| Department of Computer Science | +| University of Edinburgh, Scotland, UK. | +-------------------------------------------------- + UUCP: uunet!mcvax!ukc!lfcs!simon + ARPA: simon%lfcs.ed@nss.cs.ucl.ac.uk "Life's like that, you know" + JANET: simon@uk.ac.ed.lfcs +#! rnews 665 +Path: alberta!mnetor!uunet!mcvax!henk +From: henk@cwi.nl (Henk Schouten) +Newsgroups: rec.games.board +Subject: diplomacy +Keywords: pbm +Message-ID: <138@piring.cwi.nl> +Date: 7 Dec 87 08:36:16 GMT +Organization: CWI, Amsterdam +Lines: 9 + +A local group is going to start a diplomacy game by mail. We have +only few players so I would like to take part in the game myself. +To do so, I would like to have the moves evaluated by a +program. Before writing such a program myself, I would like to +ask if anyone has or knows of such a program in the public +domain, preferrably written in C. Code or pointers to it will be +greatly appreciated. + Henk Schouten + ..!nl!cwi!henk +#! rnews 1298 +Path: alberta!mnetor!uunet!mcvax!varol +From: varol@cwi.nl (Varol Akman) +Newsgroups: sci.crypt +Subject: Re: NSA advertisment +Summary: Somewhat naive, huh? +Message-ID: <139@piring.cwi.nl> +Date: 7 Dec 87 08:59:02 GMT +References: <4781@cit-vax.Caltech.Edu> +Organization: CWI, Amsterdam +Lines: 22 + +palmer@tybalt.caltech.edu.UUCP (David Palmer) writes: +>I just read a magazine add seeking people to work at the NSA (pg. 80R of +>Dec. 1987 IEEE Spectrum) +>The graphic is 10,000,0... (100 zeros) written on three lines. The first +>paragraph of the text reads: +> You're looking at a "googol." Ten raised to the 100th power. +> One followed by 100 zeros. Counting 24 hours a day, you would +> need 120 years to reach a googol. Two lifetimes. It's a +> number that's impossible to grasp. A number beyond our imagination. +>... material deleted ... + +This strikes me as quite odd. I mean, if something can be done in two lifetimes +then, darn it, it is well within my imagination. +If it can be done within 20 lifetimes +I can still grasp how difficult it should be. A real difficult thing would +be something that takes say 10^100 lifetimes. + +In short, I find the above ad quite naive. NSA guys should probably +have something better than this for the inspring encryption student. +What do you say? + +-Varol Akman +#! rnews 1520 +Path: alberta!mnetor!uunet!mcvax!prlb2!ronse +From: ronse@prlb2.UUCP (Christian Ronse) +Newsgroups: sci.math +Subject: Re: Least-squares fitting +Summary: see Duda & Hart, Chapter 9, for a solution +Keywords: ``eigenvector line fitting'' +Message-ID: <387@prlb2.UUCP> +Date: 7 Dec 87 09:22:11 GMT +References: <1823@culdev1.UUCP> <528@amethyst.ma.arizona.edu> +Organization: Philips Research Laboratory, Brussels +Lines: 21 + +From article <528@amethyst.ma.arizona.edu> by hdunne@amethyst.ma.arizona.edu: +< In article <1823@culdev1.UUCP> drw@culdev1.UUCP (Dale Worley) writes: + [deleted ...] +< }Is is known how to perform least-squares fitting where the "error" is +< }the perpendicular distance between the point and the line? +< } +< If the point is (x_i,y_i) and the line is y = a*x + b, then the square of the +< perpendicular distance is [(y_i - a*x_i - b)^2]/(1 + a^2) (assuming the line +< isn't vertical). Taking the sum of the squared distances and setting the +< partial derivatives wrt. a and b equal to zero, you get the same equations +< for a and b as you get from the usual least-squares procedure. + +See the book ``Pattern Classification and Scene Analysis'' by R.O. Duda & P.E. +Hart, Chapter 9. Section 9.2.1 introduces the usual least square fitting +(``minimum-squared-error line fitting''), and 9.2.2 the one asked by Dale +(``eigenvector line fitting''). There the problem is solved. + +Christian Ronse maldoror@prlb2.UUCP +{uunet|philabs|mcvax|...}!prlb2!{maldoror|ronse} + + STAT ROSA PRISTINA NOMINE, NOMINA NUDA TENEMUS +#! rnews 977 +Path: alberta!mnetor!uunet!mcvax!ukc!stl!stc!idec!camcon!mb +From: mb@camcon.uucp (Mike Bell) +Newsgroups: comp.sys.ibm.pc +Subject: Re: Neat voice|gag program +Summary: How does HELPME work? +Message-ID: <1107@titan.camcon.uucp> +Date: 2 Dec 87 14:25:07 GMT +References: <3692@uwmcsd1.UUCP> +Distribution: all +Organization: Cambridge Consultants Ltd., Cambridge, UK +Lines: 15 + +in article <3692@uwmcsd1.UUCP>, cmaag@csd4.milw.wisc.edu +(posting to comp.binaries.ibm.pc) says: + +> Here is a neat little program I found on a local bbs. It uses the speaker +> to generate a very-realistic (the best I've heard on a PC!) voice that +> says something to the effect of "Help! I'm locked in this computer! +> Let me out! Help!". + +I just played it, and was much impressed. Given the rudimentary +nature of IBM PC's, can anybody explain how it achieves its +effect? +-- +--------------- UUCP: ...mcvax!ukc!camcon!mb +-- Mike Bell -- or: mb%camcon.uucp +--------------- Phone: +44 223 358855 +#! rnews 710 +Path: alberta!mnetor!uunet!mcvax!ukc!stl!stc!idec!camcon!mb +From: mb@camcon.uucp (Mike Bell) +Newsgroups: comp.sources.bugs +Subject: Re: v12i071: StarChart program (Minor correction) +Message-ID: <1114@titan.camcon.uucp> +Date: 4 Dec 87 15:48:15 GMT +References: <1110@artemis3.camcon.uucp> +Organization: Cambridge Consultants Ltd., Cambridge, UK +Lines: 10 + +in article <1110@artemis3.camcon.uucp>, mb@camcon.uucp (Mike Bell) says: +> (Problem found on Sun 4.3 BSD Unix) + +Sorry, that should have been Sun Release 3.4 of 4.2 BSD... (well it +was correct within an order of magnitude:-) + +-- +--------------- UUCP: ...mcvax!ukc!camcon!mb +-- Mike Bell -- or: mb%camcon.uucp +--------------- Phone: +44 223 358855 +#! rnews 2447 +Path: alberta!mnetor!uunet!mcvax!tuvie!rcvie +From: rcvie@tuvie (ELIN Forsch.z.) +Newsgroups: comp.lang.c +Subject: Re: Autoincrement question +Message-ID: <548@tuvie> +Date: 7 Dec 87 10:00:58 GMT +References: <1507@ogcvax.UUCP> +Organization: TU Vienna EDP-Center, Vienna, AUSTRIA +Lines: 58 + +In article <1507@ogcvax.UUCP>, schaefer@ogcvax.UUCP (Barton E. Schaefer) writes: +> (I realize this might be similar to another question asked recently, but ...) +> +> Another student here at OGC recently came to me with a question about the +> C autoincrement operator. The following program is representative of the +> code he wrote, which did not do what he expected: +> +> struct foo { struct foo *tmp; char junk[32]; } foolist[4]; +> +> main () +> { +> struct foo *bar; +> +> bar = foolist; +> /* Do something with bar */ +> bar->tmp = bar++; /* This is the problem line */ +> /* Do something else */ +> } +> + +This is really dangerous programming. The points where the left and where the +right "bar" are evaluated are implementation defined. The problem is similar to +another one, which a friend of mine had some time ago. He tried to pack as much +as possible into the control part of a while loop using the following statement: + +while (a[i]=b[i++]) + ; + +Things were even worse here, as the program behaved even differently depending +on whether it was compiled with the optimization option or not. Non optimized +everything worked as expected but in the optimized version only for the first +assignment "i" was incremented after the assignment, for all the following +assignments it was incremented after the evaluation of "b[i]" but before the +assignment. Nevertheless this behaviour was in the sense of both K&R and ANSI. +The only thing you can trust on, is that the *operand* of the increment +operator is evaluated before its incrementation. One way to achieve the desired +behaviour is, as you suggested yourself, to write: + +> What he really wanted was the equivalent of +> bar->tmp = bar; +> bar++; + +and not (for the same reasons stated above): + +> (bar++)->tmp = bar; + +If there is any necessity to have the whole semantic in one *expression*, use +the comma operator, as + +bar->tmp = bar, bar++; + +This operator *guarantees* the sequential evaluation of its operands from +left to right. + +In real life: Dipl.Ing. Dietmar Weickert + ALCATEL Austria - ELIN Research Center + Floridusg. 50 + A - 1210 Vienna / Austria +#! rnews 1822 +Path: alberta!mnetor!uunet!mcvax!steven +From: steven@cwi.nl (Steven Pemberton) +Newsgroups: comp.sys.atari.st +Subject: Re: Alcyon C Bug N++ +Message-ID: <140@piring.cwi.nl> +Date: 7 Dec 87 14:59:48 GMT +References: <8712051307.AA12109@ucbvax.Berkeley.EDU> +Reply-To: steven@cwi.nl (or try mcvax!steven.uucp) +Organization: CWI, Amsterdam +Lines: 38 + +For people interested, here are a couple of bugs in the Alcyon +compiler that we've been hitting our heads against for the last few +weeks: + + 1) The compiler doesn't seem able to cope with nested + initialisations. For instance, a struct with an array in + the middle: + static struct foo table[] = { + { ...... {.....} ......}, + ... + } + The compiler complains about mismatched braces. + Cure: 'unwrap' the struct declaration, so it's all at the + same level. + + 2) In a construct like + bar *p = (expression1, expression2); + the result of expression2 gets coerced to int, and then + back to bar *, meaning basically that you get bombs on the + screen when you try to use p, due to a wrong address. + Cure: use + bar *p = (expression1, (bar *) expression2); + + 3) We believe that 'complicated' initialisations to auto + variables in functions (for instance where the + initialisation involves a call to another function) often + come out wrong. However, by this point, we despaired, and + stopped using the compiler, so we never followed up on it. + +I might point out that we're trying to compile a BIG program: 30,000 +lines of C, so just trying to trace bug 2 took us a LOT of time. + +By the way, just for interest: to compile the lot from scratch, using +a ram disk for temporaries would take 4 hours. When we reinitialised +the disk partition, and copied the files back, a recompile only took +1.5 hours! + +Steven Pemberton, CWI, Amsterdam; steven@cwi.nl +#! rnews 1265 +Path: alberta!mnetor!uunet!mcvax!mhres!jv +From: jv@mhres.mh.nl (Johan Vromans) +Newsgroups: comp.sys.hp +Subject: Re: syslogd on HP-UX +Summary: I have one +Message-ID: <1495@mhres.mh.nl> +Date: 7 Dec 87 12:19:02 GMT +References: <641@ucdavis.ucdavis.edu> +Sender: jv@mhres.mh.nl +Reply-To: jv@mhres.mh.nl (Johan Vromans) +Organization: Multihouse N.V., The Netherlands +Lines: 20 + +In article <641@ucdavis.ucdavis.edu> arons@iris.ucdavis.edu (Tom Arons) writes: +>Has anyone successfully ported syslog(3) and syslogd from 4.2 or +>4.3 BSD to HP-UX 5.3 running on a 9000 series 300? +> +>It doesn't look like it would be too hard to do, but I don't want to +>reinvent the wheel. + +I once implemented a syslogd for HP-UX using message queues. I have posted +it to comp.sources.unix some time ago, but I can mail it if you cannot find +it. + +Features: (almost) BSD compatible, no network support, runs as a daemon, +communicates with message queues. +If no daemon is running, calling 'syslog' is effectivily a no-op. +I have used it when I tried to get sendmail running. + + + +-- +Johan Vromans | jv@mh.nl via European backbone +Multihouse N.V., Gouda, the Netherlands | uucp: ..{uunet!}mcvax!mh.nl!jv +"It is better to light a candle than to curse the darkness" +#! rnews 1036 +Path: alberta!mnetor!uunet!mcvax!botter!wundt!michael +From: michael@wundt.psy.vu.nl (M.A.M. Michael) +Newsgroups: comp.sys.mac +Subject: Address for update of VersaTerm requested +Message-ID: <164@wundt.psy.vu.nl> +Date: 7 Dec 87 16:30:39 GMT +Reply-To: michael@psy.vu.nl.UUCP (M.A.M. Felt) +Organization: VU Psychologie, Amsterdam +Lines: 24 + +!!!!!!!!!!!!!!!!!!!!!!!!! +Please reply via e-mail. +!!!!!!!!!!!!!!!!!!!!!!!!! + +When I purchased VersaTerm 2+ years ago I didn't bother to register. +Now I wish I had. It's about time for an update. + +The manual lists the address: +Peripherals Computers & Supplies Inc +2232 Perkiomen Avenue +Mt. Penn, PA 19606 + +Is this still current (other VersaTerm Users)? + +In either case, an e-mail reply will be appreciated. +The dealer (I bought it from) here is still selling +the same version of two years ago. (1.42) + +Thanks, michael felt +-- +Michael Felt Psychology Dept, Vrije Universiteit, Amsterdam, Netherlands +InterNet: michael@psy.vu.nl +UUCP: ...!mcvax!vupsy!michael , michael@vupsy.UUCP +AppleLink: HOL0038 +#! rnews 600 +Path: alberta!mnetor!uunet!mcvax!inria!axis!alastair +From: alastair@axis.fr (Alastair Adamson) +Newsgroups: comp.text +Subject: To break or not to break +Summary: br command in [nt]roff +Message-ID: <348@axis.fr> +Date: 7 Dec 87 08:33:25 GMT +Organization: Axis Digital, Paris +Lines: 9 + +I have long wondered at the ubiquitous [nt]roff request + 'br +found in the mm macros and elsewhere. Could someone +please elucidate the use of the break request with +the no-break command character ' used? + +Thanks in advance, Alastair Adamson, + alastair@axis.fr + Axis Digital, 135 rue d'Aguesseau, 92100, Boulogne, France +#! rnews 8193 +Path: alberta!mnetor!uunet!mcvax!botter!ast +From: ast@cs.vu.nl (Andy Tanenbaum) +Newsgroups: comp.os.minix +Subject: New program: treecmp.c +Message-ID: <1774@botter.cs.vu.nl> +Date: 7 Dec 87 20:53:16 GMT +Reply-To: ast@cs.vu.nl (Andy Tanenbaum) +Organization: VU Informatica, Amsterdam +Lines: 321 + + +I have written a program to recursively compare the contents of two given +directories, file for file. The program descends the tree and reports about +files that are missing or different. Some day, if I ever get around to +producing V1.3 of MINIX, I will make a tree of the current version next to +the V1.2 tree, and then run this program to get a list of all files that +are different. Then I can make diff listings etc. In reality, the reason +I wrote it however, is that I had just copied my MINIX tree from one part +of the disk to another, and I wanted to make sure nothing was forgotten. +I am sure there are other uses as well. One could no doubt write a shell +script to do this same thing, or perhaps use find, but this program is +much faster, being able to compare two 8 megabyte trees in about 12 +minutes on a Z-248. + +Please post any bugs you find. + +Andy Tanenbaum (ast@cs.vu.nl) + +----------------------------- treecmp.c --------------------------------- +/* treecmp - compare two trees Author: Andy Tanenbaum */ + +/* This program recursively compares two trees and reports on differences. + * It can be used, for example, when a project consists of a large number + * of files and directories. When a new release (i.e., a new tree) has been + * prepared, the old and new tree can be compared to give a list of what has + * changed. The algorithm used is that the first tree is recursively + * descended and for each file or directory found, the corresponding one in + * the other tree checked. The two arguments are not completely symmetric + * because the first tree is descended, not the second one, but reversing + * the arguments will still detect all the differences, only they will be + * printed in a different order. The program needs lots of stack space + * because routines with local arrays are called recursively. The call is + * treecmp [-v] dir1 dir2 + * The -v flag (verbose) prints the directory names as they are processed. + */ + +#include + +#define BUFSIZE 4096 /* size of file buffers */ +#define MAXPATH 128 /* longest acceptable path */ +#define DIRENTLEN 14 /* number of characters in a file name */ + +struct dirstruct { /* layout of a directory entry */ + unsigned inum; + char fname[DIRENTLEN]; +}; + +struct stat stat1, stat2; /* stat buffers */ + +char buf1[BUFSIZE]; /* used for comparing bufs */ +char buf2[BUFSIZE]; /* used for comparing bufs */ + +int verbose; /* set if mode is verbose */ + +main(argc, argv) +int argc; +char *argv[]; +{ + char *p; + + if (argc < 3 || argc > 4) usage(); + p = argv[1]; + if (argc == 4) { + if (*p == '-' && *(p+1) == 'v') + verbose++; + else + usage(); + } + + if (argc == 3) + compare(argv[1], argv[2]); + else + compare(argv[2], argv[3]); + + exit(0); +} + +compare(f1, f2) +char *f1, *f2; +{ +/* This is the main comparision routine. It gets two path names as arguments + * and stats them both. Depending on the results, it calls other routines + * to compare directories or files. + */ + + int type1, type2; + + if (stat(f1, &stat1) < 0) { + printf("Cannot stat %s\n", f1); + return; + } + + if (stat(f2, &stat2) < 0) { + printf("Missing file: %s\n", f2); + return; + } + + /* Examine the types of the files. */ + type1 = stat1.st_mode & S_IFMT; + type2 = stat2.st_mode & S_IFMT; + if (type1 != type2) { + printf("Type diff: %s and %s\n", f1, f2); + return; + } + + /* The types are the same. */ + switch(type1) { + case S_IFREG: regular(f1, f2); + break; + + case S_IFDIR: directory(f1, f2); + break; + + case S_IFCHR: + case S_IFBLK: break; + + default: printf("Unknown file type %o\n", type1); + } + return; +} + +regular(f1, f2) +char *f1, *f2; +{ +/* Compare to regular files. If they are different, complain. */ + + int fd1, fd2, n1, n2, i; + unsigned bytes; + long count; + char *p1, *p2; + + if (stat1.st_size != stat2.st_size) { + printf("Size diff: %s and %s\n", f1, f2); + return; + } + + /* The sizes are the same. We actually have to read the files now. */ + fd1 = open(f1, 0); + if (fd1 < 0) { + printf("Cannot open %s for reading\n", f1); + return; + } + + fd2 = open(f2, 0); + if (fd2 < 0) { + printf("Cannot open %s for reading\n", f2); + return; + } + + count = stat1.st_size; + while (count > 0L) { + bytes = (unsigned) (count > BUFSIZE ? BUFSIZE : count); /* rd count */ + n1 = read(fd1, buf1, bytes); + n2 = read(fd2, buf2, bytes); + if (n1 != n2) { + printf("Length diff: %s and %s\n", f1, f2); + close(fd1); + close(fd2); + return; + } + + /* Compare the buffers. */ + i = n1; + p1 = buf1; + p2 = buf2; + while (i--) { + if (*p1++ != *p2++) { + printf("File diff: %s and %s\n", f1, f2); + close(fd1); + close(fd2); + return; + } + } + count -= n1; + } + close(fd1); + close(fd2); +} + +directory(f1, f2) +char *f1, *f2; +{ +/* Recursively compare two directories by reading them and comparing their + * contents. The order of the entries need not be the same. + */ + + int fd1, fd2, n1, n2, ent1, ent2, i, used1 = 0, used2 = 0; + char *dir1buf, *dir2buf; + char name1buf[MAXPATH], name2buf[MAXPATH]; + struct dirstruct *dp1, *dp2; + unsigned dir1bytes, dir2bytes; + extern char *malloc(); + + /* Allocate space to read in the directories */ + dir1bytes = (unsigned) stat1.st_size; + dir1buf = malloc(dir1bytes); + if (dir1buf == 0) { + printf("Cannot process directory %s: out of memory\n", f1); + return; + } + + dir2bytes = (unsigned) stat2.st_size; + dir2buf = malloc(dir2bytes); + if (dir2buf == 0) { + printf("Cannot process directory %s: out of memory\n", f2); + free(dir1buf); + return; + } + + /* Read in the directories. */ + fd1 = open(f1, 0); + if (fd1 > 0) n1 = read(fd1, dir1buf, dir1bytes); + if (fd1 < 0 || n1 != dir1bytes) { + printf("Cannot read directory %s\n", f1); + free(dir1buf); + free(dir2buf); + if (fd1 > 0) close(fd1); + return; + } + close(fd1); + + fd2 = open(f2, 0); + if (fd2 > 0) n2 = read(fd2, dir2buf, dir2bytes); + if (fd2 < 0 || n2 != dir2bytes) { + printf("Cannot read directory %s\n", f2); + free(dir1buf); + free(dir2buf); + close(fd1); + if (fd2 > 0) close(fd2); + return; + } + close(fd2); + + /* Linearly search directories */ + ent1 = dir1bytes/sizeof(struct dirstruct); + dp1 = (struct dirstruct *) dir1buf; + for (i = 0; i < ent1; i++) { + if (dp1->inum != 0) used1++; + dp1++; + } + + ent2 = dir2bytes/sizeof(struct dirstruct); + dp2 = (struct dirstruct *) dir2buf; + for (i = 0; i < ent2; i++) { + if (dp2->inum != 0) used2++; + dp2++; + } + + if (verbose) printf("Directory %s: %d entries\n", f1, used1); + + /* Check to see if any entries in dir2 are missing from dir1. */ + dp1 = (struct dirstruct *) dir1buf; + dp2 = (struct dirstruct *) dir2buf; + for (i = 0; i < ent2; i++) { + if (dp2->inum == 0 || strcmp(dp2->fname, ".") == 0 || + strcmp(dp2->fname, "..") == 0) { + dp2++; + continue; + } + check(dp2->fname, dp1, ent1, f1); + dp2++; + } + + /* Recursively process all the entries in dir1. */ + dp1 = (struct dirstruct *) dir1buf; + for (i = 0; i < ent1; i++) { + if (dp1->inum == 0 || strcmp(dp1->fname, ".") == 0 || + strcmp(dp1->fname, "..") == 0) { + dp1++; + continue; + } + if (strlen(f1) + DIRENTLEN >= MAXPATH) { + printf("Path too long: %s\n", f1); + free(dir1buf); + free(dir2buf); + return; + } + if (strlen(f2) + DIRENTLEN >= MAXPATH) { + printf("Path too long: %s\n", f2); + free(dir1buf); + free(dir2buf); + return; + } + + strcpy(name1buf, f1); + strcat(name1buf, "/"); + strncat(name1buf, dp1->fname, DIRENTLEN); + strcpy(name2buf, f2); + strcat(name2buf, "/"); + strncat(name2buf, dp1->fname, DIRENTLEN); + + /* Here is the recursive call to process an entry. */ + compare(name1buf, name2buf); /* recursive call */ + dp1++; + } + + free(dir1buf); + free(dir2buf); +} + +check(s, dp1, ent1, f1) +char *s; +struct dirstruct *dp1; +int ent1; +char *f1; +{ +/* See if the file name 's' is present in the directory 'dirbuf'. */ + int i; + + for (i = 0; i < ent1; i++) { + if (strncmp(dp1->fname, s, DIRENTLEN) == 0) return; + dp1++; + } + printf("Missing file: %s/%s\n", f1, s); +} + +usage() +{ + printf("Usage: treecmp [-v] dir1 dir2\n"); + exit(0); +} +#! rnews 1196 +Path: alberta!mnetor!uunet!mcvax!prlb2!kulcs!kdv +From: kdv@kulcs.UUCP (Karel De Vlaminck) +Newsgroups: comp.text +Subject: Laserprinters for troff on NCR Tower +Message-ID: <1066@kulcs.UUCP> +Date: 7 Dec 87 19:18:01 GMT +Reply-To: kdv@kulcs.UUCP () +Organization: Katholieke Universiteit Leuven, Dept. Computer Science +Lines: 22 + + +1) We want to connect a laserprinter for use with troff +on a NCR Tower System. Has anyone experience with this? + +2) We will have access to a KYOCERA F-1000 or F-1200 laser printer. +Does anyone know about the existence of a filter for the +troff output to the laserprinter (which uses 'Prescribe'). + +3) This laserprinter also has an HP Laserjet Plus emulation. +Another solution would then be to use a troff output filter +for the HP Laserjet. So I will ask the same question +about the existence for this filter. + +Please mail responses directly to me. If there are usefull +responses, I will post a summary to the net. + +Karel De Vlaminck + + | K. U. Leuven + kdv@kulcs.uucp | Department of Computer Science + or ...!mcvax!prlb2!kulcs!kdv | Celestijnenlaan 200 A + Phone: +(32) 16-200656 x3565 | B-3030 Leuven (Heverlee), Belgium +#! rnews 685 +Path: alberta!mnetor!uunet!mcvax!cernvax!ethz!solaris!wyle +From: wyle@solaris.ifi.ethz.ch@relay.cs.net (Mitchell Wyle) +Newsgroups: comp.unix.questions,comp.text +Subject: Scribe, GML +Keywords: Generalized Mark-up Languages, Scribe +Message-ID: <194@A14A.solaris.ifi.ethz.ch@relay.cs.net> +Date: 7 Dec 87 17:14:05 GMT +Organization: SOT sun cluster, ETH Zuerich +Lines: 7 +Xref: alberta comp.unix.questions:4769 comp.text:1344 + +Where can I buy Scribe? Are there other implementations of +a standard Markup Language on BSD Unix? What is Scribe? + +Please respond via e-mail; if there are enough "me too's," +I'll post. + +-Mitch Wyle (wyle@solaris.uucp | wyle@ethz.uucp | ...!cernvax!ethz!wyle +#! rnews 1896 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!simon +From: simon@its63b.ed.ac.uk (ECSC68 S Brown CS) +Newsgroups: comp.unix.wizards +Subject: Re: Setting process groups +Message-ID: <815@its63b.ed.ac.uk> +Date: 7 Dec 87 10:30:32 GMT +References: <1765@unc.cs.unc.edu> <910@mcgill-vision.UUCP> <1261@saturn.ucsc.edu> <3134@psuvax1.psu.edu> <2990@hcr.UUCP> +Reply-To: simon@lfcs.ed.ac.uk (Simon Brown) +Organization: LFCS, University of Edinburgh +Lines: 29 + +In article <2990@hcr.UUCP> writes: +>Actually SVID setpgrp() has an "extra feature" that Berkeley setpgrp(getpid()) +>does not have - it detaches the process from its controlling terminal. This +>does tend to make it "difficult" to create a pipeline attached to your terminal +>but with its own process group. + +Well, you can do that by making each such pipeline belong to it's own SXT +device, and have all these SXT's multiplexed onto your *real* terminal. +Instant job-control! + +BTW, SVR2 (and 3?) setpgrp() doesn't fully detach a process from its +controlling tty if this process has already done a setpgrp() previously +(as is the case for a login-shell -- this comes from init and getty). +What it does in this case is to "partially" detach -- so that if you try +to set up a new controlling terminal, it's not actually a controlling terminal +at all -- things like terminal-generated signals don't get sent to the process. +Presumably this is just a cretinous bug, and not something more sophisticated. + + +-- +-------------------------------------------------- +| Simon Brown | +| Laboratory for Foundations of Computer Science | +| Department of Computer Science | +| University of Edinburgh, Scotland, UK. | +-------------------------------------------------- + UUCP: uunet!mcvax!ukc!lfcs!simon + ARPA: simon%lfcs.ed@nss.cs.ucl.ac.uk "Life's like that, you know" + JANET: simon@uk.ac.ed.lfcs +#! rnews 1126 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!hwcs!adrian +From: adrian@cs.hw.ac.uk (Adrian Hurt) +Newsgroups: rec.arts.sf-lovers +Subject: Re: NCC, USS, Klingons, etc... +Summary: She was a Klingon +Message-ID: <1568@brahma.cs.hw.ac.uk> +Date: 7 Dec 87 10:33:54 GMT +References: <8712011928.AA04370@topaz.rutgers.edu> <1632@bsu-cs.UUCP> <19321@teknowledge-vaxc.ARPA> +Organization: Computer Science, Heriot-Watt U., Scotland +Lines: 15 + +In article <19321@teknowledge-vaxc.ARPA>, hshiffma@teknowledge-vaxc.ARPA (Hank Shiffman) writes: +> +> Why do you think she was a Klingon? As I recall, she looked human. +> You weren't assuming that she was a Klingon just because she had +> something going with the Christoper Lloyd character, were you? For +> shame! + +In the book of the film, Valkris was definitely a Klingon, out to do something +valiant to redeem her family's honour. She became very friendly with another +alien on board that ship because of that alien's warrior traditions. +-- + "Keyboard? Tis quaint!" - M. Scott + + Adrian Hurt | JANET: adrian@uk.ac.hw.cs + UUCP: ..!ukc!cs.hw.ac.uk!adrian | ARPA: adrian@cs.hw.ac.uk +#! rnews 1332 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!hwcs!adrian +From: adrian@cs.hw.ac.uk (Adrian Hurt) +Newsgroups: rec.games.frp +Subject: Re: Star Wars: the RPG +Summary: Pictures +Message-ID: <1569@brahma.cs.hw.ac.uk> +Date: 7 Dec 87 10:42:12 GMT +References: <1570@cup.portal.com> <13450021@acf4.UUCP> <1676@cup.portal.com> <1799@cup.portal.com> +Organization: Computer Science, Heriot-Watt U., Scotland +Lines: 21 + +In article <1799@cup.portal.com>, Nightstalker@cup.portal.com writes: +> +> Hi! Does anyone know if the force skills can be learned by any PC like +> a smuggler or outlaw for example, or can they only be taught to the +> jedi classes and NPCs? Thank you. +> Jason Wallace +> + +Any character may learn the Force skills from a master, and the rulebook even +encourages players using the Jedi characters to do some teaching, provided that +the pupil hasn't got any Dark Side points. Remember, Luke Skywalker was a +"Brash Pilot" type until Obi-Wan (OB1? :-) got to him. + +Now for my question. There are some really nice pictures in the rulebook. Can I +get separate copies of these? They would be great posters, especially the +Imperial Navy recruiting poster and the R2 advert. +-- + "Keyboard? Tis quaint!" - M. Scott + + Adrian Hurt | JANET: adrian@uk.ac.hw.cs + UUCP: ..!ukc!cs.hw.ac.uk!adrian | ARPA: adrian@cs.hw.ac.uk +#! rnews 1047 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!bob +From: bob@its63b.ed.ac.uk (ERCF08 Bob Gray) +Newsgroups: rec.arts.sf-lovers +Subject: Re: Klingon females +Message-ID: <816@its63b.ed.ac.uk> +Date: 7 Dec 87 12:36:36 GMT +References: <8712042225.AA03829@topaz.rutgers.edu> <3490@hoptoad.uucp> +Reply-To: bob@its63b.ed.ac.uk (ERCF08 Bob Gray) +Organization: I.T. School, Univ. of Edinburgh, U.K. +Lines: 13 + +In article <3490@hoptoad.uucp> tim@hoptoad.UUCP (Tim Maroney) writes: +>I like the fact that the Klingons are portrayed as sexist scumbags, but it +>disturbs me that all major sentient races except humans and Romulans put +>women in a subservient role (Klingons, Vulcans, Ferrengi). It almost seems +>as if we are being told that female subservience is part of the natural +>order of sentience. There are no major female-dominated sentient races, two +>semi-egalitarian races, and three male-dominated races, a clear imbalance in +>favor of male dominance. + +Then who was T'pau supposed to be? + +She was vulcan, and very obviously in charge of things. + Bob. +#! rnews 1313 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!bob +From: bob@its63b.ed.ac.uk (ERCF08 Bob Gray) +Newsgroups: rec.arts.sf-lovers +Subject: Re: Max Headroom +Message-ID: <817@its63b.ed.ac.uk> +Date: 7 Dec 87 13:10:09 GMT +References: <82*quale@si.uninett> <3333@ihlpl.ATT.COM> +Reply-To: bob@its63b.ed.ac.uk (ERCF08 Bob Gray) +Distribution: rec.arts.sf-lovers +Organization: I.T. School, Univ. of Edinburgh, U.K. +Lines: 18 + +In article <3333@ihlpl.ATT.COM> barth@ihlpl.UUCP (BARTH RICHARDS) writes: +>The problem is that the first few episodes were *re*made by an American +>production company for broadcast on ABC (not the Australian ABC). As I +>understand it, the first ABC run of six shows (winter/spring of 1987) were +>all reworkings of episodes already done by the British. The second run +>(fall 1987) were stories newly developed by the American producers. + +Sorry, there was only ever one original Max Headroom +programme. That was a one-off TV film made by the BBC. +Any episodes beyond the original story did not originate +with the BBC, although Maxs' creators may have been involved. + +Max then re-appeared on Channel 4 as host of a chat show for two +short seasons. (interviewing guest stars about their views +on Golf, music, life and, most importantly, Golf :->) + +He then crossed the atlantic to be re-made by ABC. + Bob. +#! rnews 1152 +Path: alberta!mnetor!uunet!mcvax!ukc!warwick!jeff +From: jeff@warwick.UUCP (Jeff Smith) +Newsgroups: comp.lang.c++ +Subject: cfront runs too fast (and fix) +Keywords: cfront fix +Message-ID: <586@ubu.warwick.UUCP> +Date: 7 Dec 87 14:49:10 GMT +Organization: Computer Science, Warwick University, UK +Lines: 27 + +If you can persuade cfront to finish in less than a second with the ++S option on, then the calculation of the number of lines processed +per second generates a divide-by-zero! On a SUN-3 with 1.2.1, +typing + cfront +S 0 ? + Nline/(stop_time-start_time) : Nline); +#else !CFRONTTOOFASTFIX + stop_time-start_time, Nline/(stop_time-start_time) ); +#endif CFRONTTOOFASTFIX + fflush(stderr); + + +Jeff +warwick!jeff + +PS. Does anyone have a fix to simpl.c for the null dereference +on Pfct f = Pfct(Pptr(q->tp)->typ) caused by the pointer to member function +problem? The problem's been noted a couple of times in comp.lang.c++, by +Paul Calder and others.. +#! rnews 1114 +Path: alberta!mnetor!uunet!mcvax!ukc!warwick!strgh +From: strgh@daisy.warwick.ac.uk (J E H Shaw) +Newsgroups: rec.music.misc +Subject: Re: More than Yes (really Egg) +Message-ID: <357@daisy.warwick.ac.uk> +Date: 7 Dec 87 17:57:56 GMT +References: <22034@ucbvax.BERKELEY.EDU> <19826@yale-celray.yale.UUCP> +Reply-To: strgh@daisy.warwick.ac.uk (J E H Shaw) +Organization: Computing Services, Warwick University, UK +Lines: 14 + +---------- +Egg released at least one other album before `Civil Surface', I think it +was called `the Polite Force'. They were very good. + +Their drummer (Clive Brooks?) joined the Groundhogs. +Their bassist (Mont Campbell?) played sometimes with some of the other + Canterbury scene people: National Health, U.K. or similar (mid 70's). +Their organist, Dave Stewart, became a pop star (`It's My Party'), and + also played with National Health, Hatfield & the North, etc. + +Apologies for any wrong names - the above is all based on memory. +-- +J.E.H.Shaw Department of Statistics, University of Warwick, Coventry CV4 7AL +$$\times\times\qquad\top\gamma\alpha\omega\exists\qquad{\odot\odot\atop\smile}$$ +#! rnews 1231 +Path: alberta!mnetor!uunet!mcvax!ukc!eagle!icdoc!qmc-cs!nickd +From: nickd@cs.qmc.ac.uk (Nick Dunlavey) +Newsgroups: comp.cog-eng +Subject: Touch-screen research +Message-ID: <348@sequent.cs.qmc.ac.uk> +Date: 4 Dec 87 10:52:55 GMT +References: <19@gollum.Columbia.NCR.COM> <290@rd1632.Dayton.NCR.COM> +Reply-To: nickd@qmc.ac.uk (Nick Dunlavey) +Organization: Sch Of C+IT, Thames Polytechnic, Woolwich, London, UK +Lines: 19 +Summary: + +Expires: + +Sender: + +Followup-To: + +Distribution: + +Keywords: + + +I know that the CEGB (for those outside the UK, this is the +UK's Central Electricity Generating Board) has done some work +on this in the Scientific Services Department in its +North-eastern Region. A report was produced called: + +"A Touch-Sensitive Screen As An Interface For On-Line Control", +by Sutherland, Pringle and Carlin. + +It documents the use of an upgraded VT103 in a power station +for operator control. +-- +------------- +Nick Dunlavey ARPA: nickd@cs.qmc.ac.uk (gw: cs.ucl.edu) +School Of Computing & IT UUCP: nickd@qmc-cs.UUCP +Thames Polytechnic Tel: 01-854 2030 Ext 339 +Wellington Street +Woolwich Thanks to Queen Mary College for +LONDON net access +SE18 6PF +#! rnews 1563 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!hwcs!adrian +From: adrian@cs.hw.ac.uk (Adrian Hurt) +Newsgroups: rec.arts.sf-lovers +Subject: Re: ST:TNG posters +Summary: Tolerance, please +Message-ID: <1570@brahma.cs.hw.ac.uk> +Date: 7 Dec 87 13:24:11 GMT +References: <5226@zen.berkeley.edu> +Organization: Computer Science, Heriot-Watt U., Scotland +Lines: 27 + +In article <5226@zen.berkeley.edu>, iverson@cory.Berkeley.EDU (Tim Iverson) writes: +> +> ... Not only that, but these article made no mention of +> ST:TNG in the subject line or header, so I couldn't kill them easily. +> +> ... The simple fact is that there is +> newsgroup for all of you to communicate in, and if the rest of us wanted to +> listen, then we would. +> + +Oh no, not again. Remember last time, when the number of articles complaining +about ST articles outnumbered the articles concerned (and every other single +type of article as well)? + +There is a ST group, but not for "all of us". Some of us can't get at it. But +your point about headers is valid. In the interests of preventing Flame War III +I suggest that those of us who wish to put ST (and Dr. Who, etc) articles here +make sure that "ST" (or Dr. Who, etc) or some similar warning appears in the +header. And those who wish to complain about such postings should also always +put some clear warning in the header, so those of us who aren't interested can +kill their articles easily. + +-- + "Keyboard? Tis quaint!" - M. Scott + + Adrian Hurt | JANET: adrian@uk.ac.hw.cs + UUCP: ..!ukc!cs.hw.ac.uk!adrian | ARPA: adrian@cs.hw.ac.uk +#! rnews 782 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!hwcs!jack +From: jack@cs.hw.ac.uk (Jack Campin) +Newsgroups: comp.os.misc +Subject: incorporating processes into file systems +Message-ID: <1572@brahma.cs.hw.ac.uk> +Date: 7 Dec 87 20:36:40 GMT +Organization: Computer Science, Heriot-Watt U., Scotland +Lines: 9 + +I believe there has been at least one OS that manages the naming of +processes and files in the same way - so 'ps' would become yet another +option to 'ls'. I forget which. Can anyone enlighten me? References? + +-- +ARPA: jack%cs.glasgow.ac.uk@nss.cs.ucl.ac.uk +JANET:jack@uk.ac.glasgow.cs USENET: ...mcvax!ukc!cs.glasgow.ac.uk!jack +Mail: Jack Campin, Computing Science Department, University of Glasgow, + 17 Lilybank Gardens, Glasgow G12 8QQ, Scotland (041 339 8855 x 6045) +#! rnews 1538 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!hwcs!jack +From: jack@cs.hw.ac.uk (Jack Campin) +Newsgroups: sci.physics,rec.games.programmer,comp.sys.mac +Subject: simulating relativistic motion +Keywords: relativity, graphics, flight simulators +Message-ID: <1573@brahma.cs.hw.ac.uk> +Date: 7 Dec 87 21:00:08 GMT +Organization: Computer Science, Heriot-Watt U., Scotland +Lines: 18 +Xref: alberta sci.physics:2410 rec.games.programmer:44 comp.sys.mac:10006 + +A long time ago I read about a program developed at MIT that produced +images of the way ordinary scenes (a street) would look at speeds nearing +c. I don't know if it used a plotter or calligraphic display, but it was +so long ago that whatever it did should surely be possible now in real time +on a Mac or equivalent. Does anything like that exist? - a sort of flight +simulator for cosmic ray particles, that would let you define a scene +with a 3D graphics editor and then look at it at various fractions of c. +(Colour would be a nice optional extra). The MIT program produced weirdly +drooping lampposts. +More ambitiously: what about general relativity? Here I am thinking about +some of the descriptions in Kaufmann's "The Cosmic Frontiers of General +Relativity" about how the world would look from near a black hole. + +-- +ARPA: jack%cs.glasgow.ac.uk@nss.cs.ucl.ac.uk +JANET:jack@uk.ac.glasgow.cs USENET: ...mcvax!ukc!cs.glasgow.ac.uk!jack +Mail: Jack Campin, Computing Science Department, University of Glasgow, + 17 Lilybank Gardens, Glasgow G12 8QQ, Scotland (041 339 8855 x 6045) +#! rnews 737 +Path: alberta!mnetor!uunet!mcvax!enea!kuling!nicke +From: nicke@kuling.UUCP (Niclas Holm) +Newsgroups: comp.lang.c++ +Subject: Anyone ported c++ to UNISYS 50xx ? +Message-ID: <569@kuling.UUCP> +Date: 6 Dec 87 16:33:36 GMT +Reply-To: nicke@kuling.UUCP (Niclas Holm) +Organization: Dept. of Computer Systems, Uppsala University, Sweden +Lines: 7 + +I am interested in running c++ on a UNISYS 50xx (read NCR Tower ..). +Has someone successfully ported it, or need I do it myself ? + +-- + Niclas F. Holm | UUCP: nicke@kuling ({seismo!mcvax}!enea!kuling!nicke) + Idrottsg. 21 II | or nicke@umecs ({seismo!mcvax}!enea!umecs!nicke) + S-753 35 Uppsala | Phone: +46 - 18 13 36 + SWEDEN | Famous Last Words: Look, no hands! +#! rnews 1584 +Path: alberta!mnetor!uunet!mcvax!ukc!stc!datlog!slxsys!jpp +From: jpp@slxsys.specialix.co.uk (John Pettitt) +Newsgroups: comp.sys.ibm.pc +Subject: Re: anyone have info on "multilink"? +Summary: Get it from: TSL, Atlanta Ga. +Keywords: remote modem multilink +Message-ID: <108@slxsys.specialix.co.uk> +Date: 7 Dec 87 20:01:09 GMT +References: <167@iisat.UUCP> +Reply-To: jpp@slxsys.UUCP (John Pettitt) +Organization: Specialix International, London, UK. +Lines: 31 + +In article <167@iisat.UUCP> iis@iisat.UUCP (Paul Gauthier) writes: +>i am trying to locate information on a program called mutlilink. i have +>heard that it permits one to run software on an ibm from a remote (dumb) +>terminal. is this correct? does anyone know of other software that will +>accomplish the same? any and all help would be appreciated. thank you. + +Multilink will allow several serial screens to run dos programs. + +More info from: + The Software Link + 3577 Parkway Lane + Atlanta GA 30092 + (404) 448 5465 + +They also have a product call PC-MOS that does the same thing on +386 boxes. + +Other software that lets you run multi user dos includes QNX, +Concurrent DOS (from Digital Research - remember CP/M :-), +Xenix (vp/ix comming soon), Unix V (ISC and Microport, with +vp/ix and locus merge respectivly). + +Disclaimer: I don't sell any of the above - just write serial +driver's for them - not easy in some cases :-( + + + +-- +John Pettitt - 144.5 MHz: G6KCQ, CIX: jpettitt, Voice: +44 1 398 9422 +UUCP: ...uunet!mcvax!ukc!pyrltd!slxsys!jpp (jpp@slxsys.specialix.co.uk) +Disclaimer: I don't even own a cat to share my views ! +#! rnews 1792 +Path: alberta!mnetor!uunet!mcvax!targon!wim +From: wim@targon.UUCP (Wim C. J. van Eerdt) +Newsgroups: comp.lang.c++ +Subject: Bug bug? solved (?) +Keywords: inline local variables +Message-ID: <367@targon.UUCP> +Date: 8 Dec 87 09:22:14 GMT +Reply-To: wim@targon.UUCP (Wim C. J. van Eerdt) +Organization: Nixdorf Computer BV., OSP, P.O. Box 29,Vianen, The Netherlands +Lines: 34 + +As long as the department does not have an uucp-feed, +you can e-mail me, the poster. +Success! + + Wim van Eerdt E-mail: mcvax!targon!wim + OSP, Nixdorf Computer Bv, Postbus 29, 4130 EA Vianen + Nederland. Tel.: +31 3473 62211. + +----------------News article got:------------------------------------- +Author: Gerard van Dorth +Subject: Bug bug? solved (?) +Keywords: inline local variables + +> ... Redeclaration of "_au2__Xt_val_global" + +The conditional statement on the lines 161/162 "if ( base == BLOCK && + n->lex_level < ( (Pfct(expand_fn->tp)->memof) ? 3 : 2 ) )" +in file expand.c has to be changed in: +"if ( base == BLOCK && n->lex_level < 'function-defined-in-class' ? 3 : 2 )". + +For a function defined in a class the lex_level is raised by the curly brace +of the class itself. Not only member functions (memof = member of) can be +defined inline, friends can also. +(Note that funny declarations of local variables did appear in case a member +function which needs locals is declared inline but not defined in the class +itself). + +The most simple way to tell whether a function is defined in a class is the +use of a global variable (the more globals the more fun), set and reset +(embracing the first loop) in the routine classdef::simpl() in file simpl.c +-- + Wim van Eerdt E-mail: mcvax!targon!wim + OSP, Nixdorf Computer Bv, Postbus 29, 4130 EA Vianen + Nederland. Tel.: +31 3473 62211. +#! rnews 1196 +Path: alberta!mnetor!uunet!mcvax!unido!infbs!hild +From: hild@infbs +Newsgroups: rec.music.classical +Subject: Re: K. u. K. - (nf) +Message-ID: <24200003@infbs.UUCP> +Date: 7 Dec 87 10:26:00 GMT +References: <19123@amdahl.UUCP> +Lines: 17 +Nf-ID: #R:amdahl:19123:infbs:24200003:000:873 +Nf-From: infbs!hild Dec 7 11:26:00 1987 + +This is only partly true. + +"K.u.K." is short for "Kaiserlich und Koeniglich", that's right. +But it has nothing to do with the king of prussia. + +At the time "K.u.K." was used, the king of Austria was also the +king of Hungary and the emperor of "Oestreich-Ungarn" (Austria and +Hungary. When thinking of K.u.K., I have the picture of +Kaiser Franz Josef, a fatherly man who kept his nation in a long +period of prosperous (sp?) peace, especially good for the arts. + +BTW, Otto von Bismarck is remembered as a man who united Germany +(with an iron hand, that's true), which at that time was divided +into many small parts, all of them having a duke, different legislation +and borders between them. This meant having to pay customs very often, +thus disallowing free trade, which in turn was necessary for the +upcoming industrial revolution. So you might regard OvB a good statesman. +#! rnews 1564 +Path: alberta!mnetor!uunet!mcvax!nikhefh!t68 +From: t68@nikhefh.UUCP (Jos Vermaseren) +Newsgroups: comp.sys.atari.st +Subject: Re: FOLDERXXXXX +Summary: FOLDRXXX may not do the job either. +Message-ID: <410@nikhefh.UUCP> +Date: 8 Dec 87 10:44:50 GMT +References: <637@aucs.UUCP> +Organization: Nikhef-H, Amsterdam (the Netherlands). +Lines: 22 + +In article <637@aucs.UUCP>, 870646c@aucs.UUCP (barry comer) writes: +> After I posted my message about the GEMBOOT prg. not working properly, I +> received a message stating that GEMBOOT will not work properly with the new ROMS,well he also stated that there is a prg. call something like "FOLDRXXX.TOS", +> will this prg. work with the new ROMS? If it will do the trick could someone +> that has it please sent it to me in a reply msg. PLEASE do not send it via +> the binaries section I will never get it. +> Thanx in advance +> Barry + +FOLDRXXX starts up with a little table of ROM versions and corresponding +to each version an address. At that address it inserts a list of memory +pieces to be used. If you use new ROM's these addresses have been changed +so you cannot use FOLDRXXX unless you figure out the new address you need +and substitute the necessary information into the binary of FOLDRXXX ( or +a disassembly ). On the other hand: the new version of the ROMs for the +Mega has a much larger OSpool from which these memory blocks are taken. +It used to be 6000 bytes, but the new size is 16000 bytes. I don't know +whether this makes FOLDRXXX superfluous. Maybe Allan Pratt can comment +on that. + +Jos Vermaseren +T68@nikhefh.uucp +#! rnews 793 +Path: alberta!mnetor!uunet!mcvax!nikhefk!marcel +From: marcel@nikhefk.UUCP (Marcel Corbeek) +Newsgroups: rec.music.classical +Subject: Question +Message-ID: <291@nikhefk.UUCP> +Date: 8 Dec 87 11:19:26 GMT +Reply-To: marcel@nikhefk.UUCP (Marcel Corbeek) +Organization: Nikhef-K, Amsterdam (the Netherlands). +Lines: 15 + +In the film "Once upon a time in America" an ouverture of Rossini is played. +Is there anyone who can tell me which one this is ? + +Marcel Corbeek, Arpanet : marcel@nikhefk.uucp +NIKHEF-K, Amsterdam. Bitnet : v59u0002@hasara11.bitnet +Home address : +Aletta Jacobsstraat 48, +1628 NP Hoorn, +The Netherlands. +Marcel Corbeek, Arpanet : marcel@nikhefk.uucp +NIKHEF-K, Amsterdam. Bitnet : v59u0002@hasara11.bitnet +Home address : +Aletta Jacobsstraat 48, +1628 NP Hoorn, +The Netherlands. +#! rnews 2545 +Path: alberta!mnetor!uunet!mcvax!prlb2!ronse +From: ronse@prlb2.UUCP (Christian Ronse) +Newsgroups: sci.math +Subject: Re: Putnam Exam (SPOILER) +Summary: another proof for the x<25 solution +Keywords: Putnam +Message-ID: <388@prlb2.UUCP> +Date: 8 Dec 87 10:03:24 GMT +References: <16863@topaz.rutgers.edu> <16864@topaz.rutgers.edu> <3482@husc6.harvard.edu> +Organization: Philips Research Laboratory, Brussels +Lines: 69 + +In article <3482@husc6.harvard.edu>, elkies@huma1.HARVARD.EDU (Noam Elkies) writes: +< [Problem A-6 of the 48th Annual W.L.Putnam Contest, Dec. 5, 1987: ] +< >> For each positive integer n, let a(n) be the number of zeros in the +< >> base 3 representation of n. For which positive real numbers x does +< >> the series +< >> +< >> inf +< >> ----- x^a(n) +< >> \ ------ +< >> / n^3 +< >> ----- +< >> n = 1 +< >> +< >> converge? + +> Actually the correct interval of convergence is x<25. Indeed, in the +> partial sum corresponding to 3^k<=n<3^(k+1), the coefficients n^(-3) are +> within a factor of 27 of 27^(-k), and the sum of x^a(n) is easily seen to +> be 2(x+2)^k, so by comparison with the geometric series sum(r^k,k,0,inf) +> with r=(x+2)/27 we find that the series converges if and only if r<1, +> i.e. x<25. + +This is correct, but the way the proof is written is not easy to understand. I +give below another proof. + +For n>0 let + +T(n) = x^a(n)/n^3 and U(n) = T(3n) + T(3n+1) + T(3n+2) + +and for k>=0 let + +Z(k) = sum {n=3^k to 3^(k+1)-1} T(n) + +We have + +Z(k+1) = sum {n=3^(k+1) to 3^(k+2)-1} T(n) + = sum {n=3^k to 3^(k+1)-1} [T(3n) + T(3n+1) + T(3n+2)] + = sum {n=3^k to 3^(k+1)-1} U(n) + +Let us compare U(n) to T(n). We have a(3n)=a(n)+1 and a(3n+1)=a(3n+2)=a(n). +Thus + +U(n) = x^[a(n)+1]/(3n)^3 + x^a(n)/(3n+1)^3 + x^a(n)/(3n+2)^3 + +and so U(n) has as upper bound + +x^a(n) * (x+2)/(3n)^3 = T(n) * (x+2)/27 + +and as lower bound + +x^a(n) * (x+2)/(3n+2)^3 = T(n) * (x+2)/(3+2/n)^3 + +in other words U(n) = T(n) * (x+2)/(27+e(n)), where e(n)<(3+2/n)^3-27 tends to +0 when n tends to infinity. It follows then that + +Z(k+1)= Z(k)*(x+2)/(27+f(k)) + +where f(k)<(3+2/3^k)^3-27 tends to 0 for n tending to infinity. + +Now the series is the sum of all Z(k). Thus for x>25 we have Z(k+1)>Z(k) for k +large enough, and the series diverges; for x<25 we have Z(k+1)< r * Z(k) (with +r=(x+2)/27<1) for every k, and the series converges. For x=25 the series +diverges too (I think so), because Z(k+1)/Z(k) tends to 1 for k tending to +infinity. + +Christian Ronse maldoror@prlb2.UUCP +{uunet|philabs|mcvax|...}!prlb2!{maldoror|ronse} + + Time is Mona Lisa +#! rnews 1248 +Path: alberta!mnetor!uunet!mcvax!botter!star!sater +From: sater@cs.vu.nl (Hans van Staveren) +Newsgroups: comp.dcom.lans,comp.sys.ibm.pc +Subject: Need info on hardware Western Digital EtherCard PLUS +Keywords: moron suppliers, Ethernet, IBM PC's +Message-ID: <608@sater.cs.vu.nl> +Date: 8 Dec 87 14:11:10 GMT +Organization: V.U. Informatica, Amsterdam, the Netherlands +Lines: 18 +Xref: alberta comp.dcom.lans:906 comp.sys.ibm.pc:9572 + +We recently acquired some Western Digital EtherCard PLUS cards for IBM PC's. +We were planning to write MINIX drivers for them and we wanted the hardware +documentation from the supplier. We were indeed promised that. +However, as one might expect, we only got the documentation that stated +where to plug in the cable, and we are more interested in which IO-ports there +are, and what they do. Our supplier is not very helpful at the moment. + +We will continue to nag our supplier, but in the meantime, does anyone have +the hardware info on this board? +We know there is a NatSemi DP8390 on there, and we have the datasheet on that +one, but there should also be an Ethernet Address Rom, plus some other things +on the board. + +As they say, thanks in advance. + + Hans van Staveren + Vrije Universiteit + Amsterdam, Holland +#! rnews 742 +Path: alberta!mnetor!uunet!mcvax!nikhefk!marcel +From: marcel@nikhefk.UUCP (Marcel Corbeek) +Newsgroups: rec.music.synth +Subject: Question +Message-ID: <292@nikhefk.UUCP> +Date: 8 Dec 87 15:59:15 GMT +Reply-To: marcel@nikhefk.UUCP (Marcel Corbeek) +Organization: Nikhef-K, Amsterdam (the Netherlands). +Lines: 15 + +Is there anybody who can give me some information about the WERSI +stageperformer? + +Marcel Corbeek, Arpanet : marcel@nikhefk.uucp +NIKHEF-K, Amsterdam. Bitnet : v59u0002@hasara11.bitnet +Home address : +Aletta Jacobsstraat 48, +1628 NP Hoorn, +The Netherlands. +Marcel Corbeek, Arpanet : marcel@nikhefk.uucp +NIKHEF-K, Amsterdam. Bitnet : v59u0002@hasara11.bitnet +Home address : +Aletta Jacobsstraat 48, +1628 NP Hoorn, +The Netherlands. +#! rnews 1193 +Path: alberta!mnetor!uunet!mcvax!targon!wim +From: wim@targon.UUCP (Wim C. J. van Eerdt) +Newsgroups: comp.lang.c++ +Subject: Another C++ problem, solved (?) +Message-ID: <368@targon.UUCP> +Date: 8 Dec 87 15:33:45 GMT +Reply-To: wim@targon.UUCP (Wim C. J. van Eerdt) +Organization: Nixdorf Computer BV., OSP, P.O. Box 29,Vianen, The Netherlands +Lines: 27 + +I did get yet another file from my colleague Gerard. +As in other articles stated send he is not reachable by e-mail. +I shall forward your mail! +Success and have fun! + + Wim +--------Fix--------------------------------------------------------- +Author: Gerard van Dorth +Subject: Another C++ problem, solved (?) + +> Yet another crazy C++ problem +> ... +> The below code is a generalization of a problem we are seeing with C++ +> ... + +Substitute the line + Pfct f = Pfct(Pptr(q->tp)->typ); +in routine call::simpl of the file simpl.c by + Ptype pt = q->tp; + while (pt->base == TYPE) pt = Pbase(pt)->b_name->tp; + Pfct f = Pfct(Pptr(pt)->typ); // for basic type only. + +(Simpl(e) turns out to be hard). +-- + Wim van Eerdt E-mail: mcvax!targon!wim + OSP, Nixdorf Computer Bv, Postbus 29, 4130 EA Vianen + Nederland. Tel.: +31 3473 62211. +#! rnews 1046 +Path: alberta!mnetor!uunet!mcvax!cogpsi!tom +From: tom@cogpsi.UUCP (Tom Vijlbrief) +Newsgroups: comp.unix.wizards +Subject: Re: Unattended dumps (BSD4.3) +Message-ID: <327@cogpsi.UUCP> +Date: 8 Dec 87 15:51:57 GMT +References: <9032@santra.UUCP> +Reply-To: tom@cogpsi.UUCP (Tom Vijlbrief) +Organization: TNO Institute for Perception, Soesterberg, The Netherlands +Lines: 21 + +In article <9032@santra.UUCP> nispa@hutcs.hut.fi (Tapani Lindgren) writes: +>Can yes(1) somehow be piped to a program that reads /dev/tty? +>Could dump(8) be modified to abort at errors without any questions? + +If you want dump to read the output from e.g. yes(1) +then you'll have to use a pty(4). + +You should arrange that this pty is the control terminal of the +dump program and then write (redirect) the output of yes(1) to the pty. + +Setting the control terminal of dump is done by writing a program which: + +A) Removes the association with its control terminal by: + + ioctl(f, TIOCNOTTY, 0); + +B) Opens the pty. + +C) Exec's the dump program. + +The above applies to Berkeley Unix 4.X +#! rnews 1034 +Path: alberta!mnetor!uunet!mcvax!botter!ark!maart +From: maart@cs.vu.nl (Maarten Litmaath) +Newsgroups: comp.unix.wizards +Subject: Re: Emacs csh alias -- better solution than the first posted (2) +Summary: this time really faster +Keywords: this time really faster +Message-ID: <1160@ark.cs.vu.nl> +Date: 8 Dec 87 16:55:49 GMT +References: <1508@ogcvax.UUCP> <1159@ark.cs.vu.nl> +Reply-To: maart@cs.vu.nl (Maarten Litmaath) +Organization: VU Informatica, Amsterdam +Lines: 17 + +Of course the alias had to be: + +alias emacs \ +'jobs > /tmp/jobs; grep emacs /tmp/jobs > /dev/null && fg %?emacs || /bin/emacs' + ^ ^^^^^ + ! !!!!! +or + + !! + vv +alias em \ +'jobs > /tmp/jobs; grep emacs /tmp/jobs > /dev/null && fg %emacs || emacs' + +Sorry. +-- +Time flies like an arrow, fruit flies |Maarten Litmaath @ Free U Amsterdam: +like an orange. (seen elsewhere) |maart@cs.vu.nl, mcvax!botter!ark!maart +#! rnews 691 +Path: alberta!mnetor!uunet!mcvax!botter!ark!maart +From: maart@cs.vu.nl (Maarten Litmaath) +Newsgroups: comp.unix.wizards +Subject: Re: Emacs csh alias -- better solution than the first posted +Summary: faster +Keywords: faster +Message-ID: <1159@ark.cs.vu.nl> +Date: 8 Dec 87 15:41:32 GMT +References: <1508@ogcvax.UUCP> +Reply-To: maart@cs.vu.nl (Maarten Litmaath) +Organization: VU Informatica, Amsterdam +Lines: 7 + +alias emacs \ +'jobs > /tmp/jobs; grep emacs /tmp/jobs > /dev/null && fg %emacs || emacs' + +BTW, long live vi! +-- +Time flies like an arrow, fruit flies |Maarten Litmaath @ Free U Amsterdam: +like an orange. (seen elsewhere) |maart@cs.vu.nl, mcvax!botter!ark!maart +#! rnews 987 +Path: alberta!mnetor!uunet!mcvax!mhres!jv +From: jv@mhres.mh.nl (Johan Vromans) +Newsgroups: comp.unix.questions +Subject: Re: UCB 2.9 LISP goes illegal +Summary: sysmac.sml? RT-11 +Message-ID: <1498@mhres.mh.nl> +Date: 8 Dec 87 21:16:13 GMT +References: <10712@brl-adm.ARPA> +Organization: Multihouse N.V., The Netherlands +Lines: 12 + +In article <10712@brl-adm.ARPA> PAAAAAR%CALSTATE.BITNET@CUNYVM.CUNY.EDU writes: +>We are trying to make LISP run on an 11/24 (yes they still exist) +>What is sysmac.sml, for instance? + +That reminds me to the goold old days, when PDP-11's ran only RSX, +RT-11 or RSTS. Sysmac.sml is a macro library, which contains the definitions +for the RT-11 "Programmed Requests" (nowadays known as system calls). +Don't think it's equivalent exists on Unix ... +-- +Johan Vromans | jv@mh.nl via European backbone +Multihouse N.V., Gouda, the Netherlands | uucp: ..{uunet!}mcvax!mh.nl!jv +"It is better to light a candle than to curse the darkness" +#! rnews 6100 +Path: alberta!mnetor!uunet!mcvax!cernvax!ethz!srp +From: srp@ethz.UUCP (Scott Presnell) +Newsgroups: rec.games.hack,comp.sources.d +Subject: Re: Compilation of Nethack 2.2 +Keywords: AAARGH. +Message-ID: <262@bernina.UUCP> +Date: 8 Dec 87 06:13:59 GMT +References: <9714@shemp.UCLA.EDU> +Reply-To: srp@bernina.UUCP (Scott Presnell) +Organization: Chem. Dept., Swiss Federal Inst. of Tech. (ETH-Zurich) +Lines: 211 +Xref: alberta rec.games.hack:1746 comp.sources.d:1578 + +In article <9714@shemp.UCLA.EDU> claus@CS.UCLA.EDU (Claus Giloi) writes: + +>I just downloaded Nethack 2.2 from the net and compiled it on my AT +>at home. +>There were only a few small problems, then it came to linking the +>monster. An executable was produced, but I get a "Stack Overflow" +>error when I try to run the 350K executable, and changing the +>value of (STACK:) to outlandish figures (8000, 3fff) didn't change +>that. Someone out there must have gotten it to run, please tell me +>what value you used to link it. (I am using MSC 4.0) + +Here's the makefile that I used to get Nethack up under MSC 4.0... NB: the +CFLAGS macro and the link command. I was able to play a couple of levels +without stack errors or hangups, however there are some problems, +(everything seems to be identified, inventory not displayed correctly, +color not quite right (but overall it works)) so i did not "install" it. + +"good luck, jim" + +Scott Presnell Organic Chemistry +Swiss Federal Institute of Technology (ETH-Zentrum) +CH-8092 Zurich, Switzerland. +uucp:seismo!mcvax!cernvax!ethz!srp (srp@ethz.uucp); bitnet:Benner@CZHETH5A + + + +# SCCS Id: @(#)Makefile.pc 2.2 87/11/11 +# Makefile for NetHack (PC) version 1.0 written using +# Microsoft(tm) "C" v3.0 or better. +# +# Large memory model, register bug, remove stack probes: +WIZARD= +V = 22 +#CFLAGS = -A$(MODEL) -DREGBUG -DLINT_ARGS -DVER=$V $(WIZARD) -Ot -Gs -Gt100 +CFLAGS = -nologo -A$(MODEL) -DLINT_ARGS -DVER=$V -Ox -Gt10 +CC = cl +LIBS = +LFLAGS = +MODEL = L +SETARGV = #$(LIB)\$(MODEL)SETARGV +.SUFFIXES: .exe .obj .c +.c.obj:; cl $(CFLAGS) -c $*.c +.c.exe:; + cl $(CFLAGS) -c $*.c + link $*.obj $(SETARGV), $@,, $(LIBS) $(LFLAGS); + +# The game name +GAME = hack.exe + +# The game directory +GAMEDIR = \h + +# All object modules +OBJS = decl.obj apply.obj bones.obj cmd.obj do.obj dothrow.obj\ + do_name.obj do_wear.obj dog.obj dogmove.obj eat.obj end.obj \ + engrave.obj fight.obj fountain.obj hack.obj invent.obj \ + lev.obj main.obj makemon.obj mhitu.obj mklev.obj \ + mkmaze.obj mkobj.obj mkshop.obj mon.obj monmove.obj\ + monst.obj o_init.obj objnam.obj options.obj \ + pager.obj polyself.obj potion.obj pray.obj pri.obj prisym.obj\ + read.obj rip.obj rumors.obj save.obj \ + search.obj shk.obj shknam.obj sit.obj spell.obj steal.obj \ + termcap.obj timeout.obj topl.obj topten.obj track.obj trap.obj \ + tty.obj unix.obj u_init.obj vault.obj wield.obj \ + wizard.obj worm.obj worn.obj write.obj zap.obj \ + version.obj rnd.obj alloc.obj msdos.obj + +# The main target - you may want to try both of these alternatives. +# +$(GAME) : $(OBJS) +# link $(OBJS), $(GAME) /NOIG /STACK:4000 /CP:1; + link $(OBJS), $(GAME) /NOIG /STACK:10000 /SEG:512; + + +# variable auxilary files. +# +VARAUX = data rumors + +install : $(GAME) $(VARAUX) + - exepack $(GAME) $(GAMEDIR)\$(GAME) + - exemod $(GAMEDIR)\$(GAME) /max 1 + +clean : + erase $(GAME) + +spotless: clean + erase *.obj + erase main.c + erase tty.c + erase unix.c + +srcs : + copy makefile \tmp + copy *.c \tmp + copy *.h \tmp + copy \local\make\make.doc \tmp + copy \local\make\make.ini \tmp + copy \bin\make.exe \tmp + cd \tmp + time + touch *.* + arc m hack$Vs * *.* + cd $(CWD) + + +# Other dependencies +# +RUMORFILES= rumors.bas rumors.kaa rumors.mrx + +makedefs.exe: makedefs.c alloc.obj config.h + cl -AL makedefs.c alloc.obj + + +rumors : config.h $(RUMORFILES) makedefs.exe + makedefs.exe -r + +data : config.h data.bas makedefs.exe + makedefs.exe -d + +onames.h : config.h objects.h makedefs.exe + makedefs.exe -o + +# Below is a kluge. date.h should actually depend on any source +# module being changed. (but hack.h is close enough for most). +# +date.h : hack.h makedefs.exe + makedefs.exe -D + +trap.h : config.h makedefs.exe + makedefs.exe -t + +main.obj : pcmain.c hack.h + $(CC) $(CFLAGS) -Fo$@ -c pcmain.c + +tty.obj : pctty.c hack.h msdos.h + $(CC) $(CFLAGS) -Fo$@ -c pctty.c + +unix.obj : pcunix.c hack.h mkroom.h + $(CC) $(CFLAGS) -Fo$@ -c pcunix.c + +decl.obj : hack.h mkroom.h +apply.obj : hack.h edog.h mkroom.h +bones.obj : hack.h +hack.obj : hack.h +cmd.obj : hack.h func_tab.h +do.obj : hack.h +do_name.obj : hack.h +do_wear.obj : hack.h +dog.obj : hack.h edog.h mkroom.h +dogmove.obj : hack.h mfndpos.h edog.h mkroom.h +dothrow.obj : hack.h +eat.obj : hack.h +end.obj : hack.h +engrave.obj : hack.h +fight.obj : hack.h +fountain.obj : hack.h mkroom.h +invent.obj : hack.h wseg.h +ioctl.obj : config.h +lev.obj : hack.h mkroom.h wseg.h +makemon.obj : hack.h +mhitu.obj : hack.h +mklev.obj : hack.h mkroom.h +mkmaze.obj : hack.h mkroom.h +mkobj.obj : hack.h +mkshop.obj : hack.h mkroom.h eshk.h +mon.obj : hack.h mfndpos.h +monmove.obj : hack.h mfndpos.h +monst.obj : hack.h eshk.h +msdos.obj : msdos.h +o_init.obj : config.h objects.h onames.h +objnam.obj : hack.h +options.obj : hack.h +pager.obj : hack.h +polyself.obj : hack.h +potion.obj : hack.h +pray.obj : hack.h +pri.obj : hack.h +prisym.obj : hack.h wseg.h +read.obj : hack.h +rip.obj : hack.h +rumors.obj : hack.h +save.obj : hack.h +search.obj : hack.h +shk.obj : hack.h mfndpos.h mkroom.h eshk.h +shknam.obj : hack.h +sit.obj : hack.h +spell.obj : hack.h +steal.obj : hack.h +termcap.obj : hack.h +timeout.obj : hack.h +topl.obj : hack.h +topten.obj : hack.h +track.obj : hack.h +trap.obj : hack.h edog.h mkroom.h +u_init.obj : hack.h +vault.obj : hack.h mkroom.h +wield.obj : hack.h +wizard.obj : hack.h +worm.obj : hack.h wseg.h +worn.obj : hack.h +write.obj : hack.h +zap.obj : hack.h +version.obj : hack.h date.h +extern.h: config.h spell.h obj.h + touch extern.h +hack.h: extern.h flag.h gold.h monst.h objclass.h rm.h trap.h you.h + touch hack.h +objects.h: config.h objclass.h + touch objects.h +you.h: config.h onames.h permonst.h + touch you.h +#! rnews 3561 +Path: alberta!mnetor!uunet!mcvax!cernvax!jmg +From: jmg@cernvax.UUCP (jmg) +Newsgroups: comp.protocols.appletalk +Subject: Kinetics/NCSA problems +Message-ID: <581@cernvax.UUCP> +Date: 8 Dec 87 10:01:07 GMT +Reply-To: jmg@cernvax.UUCP () +Organization: CERN European Laboratory for Particle Physics, CH-1211 Geneva, Switzerland +Lines: 58 + +This is a bit of a flame, which I hope does not upset some people +too much. I have tried sending the comments privately, but have had +no reply. +I got a Kinetics internal Ethernet interface for a Mac SE, plus the +ethernet driver, test software and NCSA telnet version 1.12. +In order to try out this software in a safe manner I created a mini- +-Ethernet with the Mac and an Ethernet monitor. Am I glad that I did +this! +The test software, when run, tends to throw out a large number of +broadcast packets in a very short space of time. Sometimes one can +control the frequency, other times not. At least one test threw out +about 200 broadcast packets in much less than one second. If I had +been on the real CERN Ethernet then a few hundred users would have +had to deal with these! +FLAME ON +When will people writing test software avoid the intensive use of +broadcast packets? Multicast would be slightly better, but even then +the software should establish the address of those other computers +with which it can run a test, and then address them directly. +FLAME OFF (for a while) +I then tried to run NCSA telnet. This also started out with about +70 immediate broadcasts. These started out with a set of three +types of broadcast: + 1. arp with source ip address 0.0.0.255, looking for 0.0.0.127 + 2. something with type field 80f3 (what the hell is this?) + 3. some other arp-type (type field 809b) with sender as 0.0.0.127 +These three are repeated about 20 times at intervals of about +10 milliseconds (yes, milliseconds!). There are then a few more type +809b broadcasts at reasonable (a few hundred milliseconds!) intervals +before telnet starts to arp for the real host that I asked for. +FLAME ON +Why does software often insist on repeating packets at very short +intervals on vey reliable LANs (and have you seen the Sun lately!)? +FLAME OFF +Despite all the above, I waited for a quiet moment before connecting +onto the real Ethernet. I then tried telnet to our Ultrix Vax. +Immediate remark: keyboard in application mode does not work for us. +I then thought to run the vt100 test program (which some of you might +also have picked up off usenet). What a disaster: the emulation fails +all over the place! +Never mind, let us see if I can connect to our IBM VM system. Of course, +I have to go via a Spartacus KNET, because there is no NCSA tn3270 +(is anyone working on this?). Complete failure: Spartacus has a bit of +a peculiar telnet setup (though Ultrix, bsd4.2 and FTP Inc. telnet on +a PC work fine) which seems to screw NCSA telnet. +Final try: go through an IBM 7171 front-end, which has 3270 to VT100 +built in. Sort of works (using ESC n for PF key n), but since the +application keypad mode fails there is no way that I could get PA2 +for clear screen. Merde (which the French will understand. +FLAME ON +I know that NCSA is now at version 2.0. Why did I get version 1.12 +from Kinetics? (and why must only a Kinetics agent modify their Mac SCSI +box for a European power supply?). How do I get an updated version +quickly (no, I cannot do anonymous FTP!). Why have these simple tests +never been reported before? etc. etc. +FLAME OFF +I would be delighted if someone could tell me that all the above problems +are fixed in the current release! +#! rnews 1102 +Path: alberta!mnetor!uunet!mcvax!cernvax!ethz!wanner +From: wanner@ethz.UUCP (Juerg Wanner) +Newsgroups: rec.games.misc +Subject: Re: The Pawn help +Message-ID: <263@bernina.UUCP> +Date: 8 Dec 87 14:48:50 GMT +References: <2884@cbmvax.UUCP> <2299@killer.UUCP> <2910@cbmvax.UUCP> +Reply-To: wanner@owf.UUCP (Juerg Wanner) +Organization: OWF AG, Switzerland +Lines: 17 +Keywords: + + +In article <2910@cbmvax.UUCP> daveb@cbmvax.UUCP (Dave Berezowski) writes: +>How does one assure that they get the chest? Wait around for Kronos at the +>beginning of the game after you've delivered the note? + +After delivering the note? Hmmm... that might be too late. + +>I've been told that there is a bug in the game such that you must get to +>the pedestal asap else the blue key won't be there (this is what has happended +>to me). If I do go to the pedestal first, will I miss the Adventuer and +>Kronos? ie. should I wait around for Kronos, give the adventuer (with the +>chest I guess), and then go for the blue key? + +I've neither encountered that bug, nor did I first get the key. There's a lot +one can do before. + + +Juerg Wanner +#! rnews 2241 +Path: alberta!mnetor!uunet!mcvax!nikhefk!paulm +From: paulm@nikhefk.UUCP (Paul Molenaar) +Newsgroups: comp.sys.mac +Subject: Re: HyperCard Find +Summary: Here's the solution (well...) +Message-ID: <293@nikhefk.UUCP> +Date: 8 Dec 87 23:45:24 GMT +References: <1262@runx.ips.oz> +Organization: Nikhef-K, Amsterdam (the Netherlands). +Lines: 50 + +In article <1262@runx.ips.oz>, clubmac@runx.ips.oz (Macintosh Users Group) writes: +> +> I was asked this question by a guy on the weekend, and was unable to help +> him. Any of you Hypercard gurus able to answer?? +> +> "I want to have a BACKGROUND button which has a script that tries to FIND +> an arbitrary text. However, when I try it, it only finds the text in +> BACKGROUND fields, not FOREGROUND. The FIND works properly when you use +> the MESSAGE box.. how come?" +> +> Jeff Laing (where for art thou comp.sys.mac.hypercard?) +> +Same problem here. I noticed that strange Find bug too. My +solution is a real kludge, but it works. + +Instead of issuing the FIND command in script, TYPE the FIND command +with all the arguments into the message box and then (again +from script) add a return. Like: + +on mouseUp + type "FIND" && quote & key & quote && "in background field id" && number & + return +end mouseUp + +This also makes the repeated FIND easier. + +I made a stack that needed a search option on partial keys. So I wanted +HC to keep on looking when the user stated that the item found wasn't +the right one. + +I made a script to do this (if interested I can mail/post it) that +expects a second field for every field to be looked in. The item found +is put in the second field (named something like showName). When +the user says he wants to keep on searching, the next item found is +compared to the contents of showName. If it's the same, my script +says that 'it's all there is'. And cancels the search. Otherwise +a repeated search would be impossible. + +If you like I can upload the lot. To comp.sys.mac.hypercard maybe? + +To Apple: +Why do you reply to all the easy answers in comp.sys.mac.hypercard +bu happily skip all the possibly difficult ones? Seems like +the HyperCard group chooses the easy way out. Too many bugs in HC +perhaps? +-- + Paul Molenaar + + "Just checking the walls" + - Basil Fawlty - +#! rnews 704 +Path: alberta!mnetor!uunet!mcvax!enea!tut!santra!kolvi!jku +From: jku@kolvi.UUCP (Juha Kuusama) +Newsgroups: comp.sys.ibm.pc +Subject: Screen dump from Hercules to Laserjet wanted +Message-ID: <31@kolvi.UUCP> +Date: 8 Dec 87 12:19:56 GMT +Reply-To: jku@kolvi.UUCP (Juha Kuusama) +Organization: Helsinki University of Technology, Finland +Lines: 10 + +Could some kind soul over there send me a program/a reference to a program, +that would allow me to print a graphics dump from a Hercules screen to a +HP Laserjet printer It should + + a) not distort the image (circles as circles, not ovals) + + b) send its output to a file (so I can import it to my text). + +-- +Juha Kuusama, jku@kolvi.UUCP ( ...!mcvax!tut!kolvi!jku ) +#! rnews 1093 +Path: alberta!mnetor!uunet!mcvax!enea!tut!santra!jmunkki +From: jmunkki@santra.UUCP (Juri Munkki) +Newsgroups: comp.sys.mac +Subject: Re: Development Environment Advice Wanted +Keywords: Development, MacII Debuggers +Message-ID: <9206@santra.UUCP> +Date: 8 Dec 87 16:51:43 GMT +References: <687@howtek.UUCP> <3456@husc6.harvard.edu> +Reply-To: jmunkki@santra.UUCP (Juri Munkki) +Organization: Helsinki University of Technology, Finland +Lines: 16 + +In article <3456@husc6.harvard.edu> singer@endor.UUCP (THINK Technologies) writes: +>The current version of MacsBug, version 5.5, works fine on a Mac II - +>even disassembles 68020 and 68881 opwords, and works with or without +And it slows down the 68881 by about 50%. Can anyone else verify this? +I moved to TMON mainly because it does not affect the speed of my Mac. + +I hope none of the Byte or MacTutor benchmarks were run under MacsBug. + +Still, ES works better in MacsBug than it does in TMON. + +Juri Munkki +jmunkki@santra.hut.fi +jmunkki@fingate.bitnet +lk-jmu@finhut.bitnet + +Disclaimer: I'm just a freelance programmer, you shouldn't listen to me anyway. +#! rnews 1288 +Path: alberta!mnetor!uunet!mcvax!diku!rancke +From: rancke@diku.UUCP (Hans Rancke-Madsen.) +Newsgroups: rec.games.frp +Subject: Re: Re: Characters with two classes +Message-ID: <3567@diku.UUCP> +Date: 7 Dec 87 15:31:25 GMT +References: <26561S9S@PSUVMA> <81800077@uiucdcsp> +Organization: DIKU, U of Copenhagen, DK +Lines: 23 + +In article <81800077@uiucdcsp> jenks@uiucdcsp.cs.uiuc.edu writes: + +> The PHB doesn't specifically forbid doing this +>more than once, nor does it say what the "prime stat" is for Paladinks, +>Rangers, Monks, etc. + +I seem to recall having seen a statement like "since has no prime requisite, +you can't switch to/from it." The implication being that any of +the sub-classes that require more than one minimum is out as +regards dual-class characters. So you could be a "fighter-turned- +magician" but not a "ranger-turned-magician". I think it was in +one of THE BOOKS, but I'm not certain. One thing you could do +is to require 15 or 17 in ALL the requisites with minimums. +That will restrict the number of assasin/illusionists!!! + + Hans Rancke, University of Copenhagen + ..mcvax!diku!rancke + +--=-=-=-=-=-=-=-=-=-=-=-=-=-=- + +- I hate it when people call me paranoid. + It makes me feel persecuted. +#! rnews 456 +Path: alberta!mnetor!uunet!mcvax!diku!iesd!torbennr +From: torbennr@iesd.uucp (Torben N. Rasmussen) +Newsgroups: comp.sources.wanted +Subject: Wanted: Microemacs part 8 +Message-ID: <166@iesd.uucp> +Date: 7 Dec 87 08:15:25 GMT +Reply-To: torbennr@neumann.UUCP (Torben N. Rasmussen) +Organization: Dept. of Comp. Sci., Aalborg University, Denmark +Lines: 7 + + +Could someone please send me part 8 of the sources for Microemacs. + +-- + + + Torben Rasmussen (torbennr) +#! rnews 1138 +Path: alberta!mnetor!uunet!mcvax!diku!dde!jk +From: jk@dde.uucp (Jens Kjerte) +Newsgroups: comp.sources.wanted,comp.text +Subject: Sourcecode for dca2troff wanted. +Keywords: DCA conversion. +Message-ID: <277@Aragorn.dde.uucp> +Date: 8 Dec 87 09:11:32 GMT +Organization: Dansk Data Elektronik A/S, Herlev, Denmark +Lines: 18 +Xref: alberta comp.sources.wanted:2716 comp.text:1345 + + + We are right now starting a project, that involves translating + IBM DCA documents to and from a wordprocessing package. + A program called dca2troff was posted sometime ago. + This program, as the name says, was able to + convert from DCA format to troff format. + Would somebody having that source, please e-mail it to me. + + Other information about software regarding DCA conversion, + Public Domain or not, would be appreciated. + + Thanks in advance + +-- ++---------------------------------------------------------------------------+ +| Jens Kjerte @ Dansk Data Elektronik A/S, Systems Software Department | +| E-mail: ..!uunet!mcvax!diku!dde!jk or jk@dde.uucp | ++---------------------------------------------------------------------------+ +#! rnews 1323 +Path: alberta!mnetor!uunet!mcvax!diku!dde!ct +From: ct@dde.uucp (Claus Tondering) +Newsgroups: sci.physics +Subject: Maxwell's daemon +Message-ID: <279@Aragorn.dde.uucp> +Date: 8 Dec 87 13:57:45 GMT +Organization: Dansk Data Elektronik A/S, Herlev, Denmark +Lines: 26 + +Consider the following variant of Maxwell's daemon: + +You have the following two items: + 1) a metal block, + 2) a bowl with a liquid. +Both items have the same temperature and are placed close together, they +may, however, be thermally isolated from one another. + +Now into the bowl you drop a very small magnet. The motion of the +molecules in the liquid will cause the magnet to move slightly. This +will induce a (very small) current in the metal block. This current will +cause the temperature of the metal block to rise. The current will also +try to stop the movements of the magnet; this will in turn slow down the +motion of the molecules, and the liquid will cool. + +The result: The metal block will grow warmer and warmer, and the liquid +will grow colder and colder. + +This contradicts the second law of thermodynamics, and has the "advantage" +over Maxwell's daemon that no intelligence is involved. + +What is wrong with the above argument? +-- +Claus Tondering +Dansk Data Elektronik A/S, Herlev, Denmark +E-mail: ct@dde.uucp or ...!uunet!mcvax!diku!dde!ct +#! rnews 2476 +Path: alberta!mnetor!uunet!mcvax!enea!sommar +From: sommar@enea.UUCP (Erland Sommarskog) +Newsgroups: rec.music.misc +Subject: Swedish prog-rock (was Re: More than Yes) +Message-ID: <2505@enea.UUCP> +Date: 8 Dec 87 23:22:11 GMT +References: <19949@yale-celray.yale.UUCP> +Reply-To: sommar@enea.UUCP(Erland Sommarskog) +Followup-To: rec.music.misc +Organization: ENEA DATA Svenska AB, Sweden +Lines: 42 + +No isn't that an obscure subject line? But I must correct my +fellow-countryman here. + +Bjorn Lisper (lisper@yale-celray.UUCP) writes: +>Bo Hansson, to be correct. Gee, I didn't know that he was known outside +>Sweden. This guy was a keyboard player who was active mainly in the late +>sixties and early seventies. He is remembered for having made the very +>first record for the first Swedish independent non-profit label "Silence". +>Unexpectedly the record became a hit and the income helped financing a lot +>of records with early Swedish prog-rock that would otherwise not have been +>economically possible to make. Thus his importance for Swedish rock music +>cannot be overestimated. + +So he is the one being guilty to it all. Grr. You see, in Sweden +"progressive" music had nothing to do with the music. When we speak - +or spoke at that time - of "progressive" groups, we talked of groups +that played quite regular rock or pop. There were just one difference +to the ordinary hit music, the lyrics. They were naive, trivial and +uttermost boring political texts of a communistic nature. (Which does +not imply that they were paid by KGB or something.) I must admit I +didn't listen to much to them, their proganda was too much for me. + +Now, this kind of people dominated this non-profit companies that Bjorn +talked of. For them ideological purity was much more important than +interesting than good music. Not to be denied, *some* good music was +actually released on Silence and MNW (the other big non-profit), but +also a lot of true crap. And I can easily imagine that groups with +interesting music was refused beacuse they voted with the wrong party. +(They would never have released Yes, that are right wing if anything.) + +Finally, I should admit that despite the poorness of Silence, they +had the most interesting music in Sweden at that time. But that more +gives an indication of bad the rest was. (Abba, do you remember?) + + + + +-- +Erland Sommarskog +ENEA Data, Stockholm +sommar@enea.UUCP + C, it's a 3rd class language, you can tell by the name. +#! rnews 2310 +Path: alberta!mnetor!uunet!mcvax!cernvax!ethz!forty2!poole +From: poole@forty2.UUCP (Simon Poole) +Newsgroups: comp.sys.atari.st +Subject: Re: GEMBOOT and the Megas +Message-ID: <122@forty2.UUCP> +Date: 8 Dec 87 14:19:44 GMT +References: <608@aucs.UUCP> <900@atari.UUCP> +Reply-To: poole@forty2.UUCP (Simon Poole) +Organization: Exp. Physics University Zuerich +Lines: 39 + + +In article <900@atari.UUCP> apratt@atari.UUCP (Allan Pratt) writes: +>in article <608@aucs.UUCP>, 870646c@aucs.UUCP (barry comer) says: +>> +>> Hi all, well my Mega2 just landed on my desk, really nice. I've got a question for all other Mega owners using the hard disks, I have been using GEMBOOT with +>> my 1040ST all along, when I boot up the Mega two bombs appear then disappear +>> after GEMBOOT has done its thing. +> +>DO NOT USE GEMBOOT. Use FOLDRXXX from Atari. HINSTALL should be available, +>too... It makes your hard disk bootable (no "boot floppy" needed). +> +The lastest version of GEMBOOT which was distributed something like +half a year ago, allows you to set the location of the sole undocumented +variable that Konrad uses in GEMBOOT. Matter of fact I used GEMBOOT +without problems on one of the first Mega's that arrived in Switzerland +after changing the GEMBOOT startup file. + +>patches the appropriate location in the OS. In the case of the Mega +>ROMs, he actually added a pointer in the OS header which points to +>the necessary spot, so FOLDRXXX will work for all future ROM releases. + ^^^^^^^^^^^^^^ +Didn't Atari claim it was working on a new '40 folder bug'less OS? + +>Even old TOS ROM users should probably not use GEMBOOT... I certainly +>wouldn't trust it, and with FOLDRXXX and HINSTALL available, you just +>don't need it. +Hmmmm, as Landon Dyer once said (a long time ago) FOLDRXXX does NOT fix +the other problem with GEMDOS management of the internal directory +list (mutiple bad copies of the same block), GEMBOOT does provide +a workaround for this problem (so I wouldn't trust FOLDRXXX) plus +a lot of other nice things. + + + Simon Poole + UUCP: ....mcvax!cernvax!forty2!poole + Bitnet: K538915@CZHRZU1A + +* +***************When will Atari annouce PC-6 to PC-10?**************** +* +#! rnews 1572 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!bob +From: bob@its63b.ed.ac.uk (ERCF08 Bob Gray) +Newsgroups: rec.arts.sf-lovers +Subject: Re: Blake's, all 7 of them! +Message-ID: <818@its63b.ed.ac.uk> +Date: 8 Dec 87 10:08:31 GMT +References: <6320@ihlpa.ATT.COM> <1572@cup.portal.com> <1372@aurora.UUCP> +Reply-To: bob@its63b.ed.ac.uk (ERCF08 Bob Gray) +Organization: I.T. School, Univ. of Edinburgh, U.K. +Lines: 28 + +In article <1372@aurora.UUCP> timelord@aurora.UUCP (G. "Murdock" Helms) writes: +>In article <1572@cup.portal.com>, Isaac_K_Rabinovitch@cup.portal.com writes: +>> Whoops. After the Star One episode, the actor who played +>> Blake got a job with the National Shakespeare Company, so Blake essentially +>> disappears until the "last" episode. +> +>The second Travis, the one with the really thick Cockney accent, +>was spotted in the BBC movie "Edge of Darkness" recently broadcast +>in California. + +Something else to watch out for. The recently concluded +series "Knights of God" on independant television was +notable only for having Gareth Thomas (Blake himself) playing +the part of the leader of a band of rebels trying to +overthrow the harsh Goverment sometime in the future UK. +Almost a reprise of his part as blake, but he isn't even +one of the major characters. His name comes about eighth +on the credits. + +Now we know what he was doing while he was missing from +Blake's Seven. :-> + +Also look out for the second Dr Who, Patrick Troughton, in a +supporting role. + +Note: I do Not recommend this series for any other reson +than the above mentioned curiosity value. + Bob +#! rnews 2656 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!csnjr +From: csnjr@its63b.ed.ac.uk (Nick Rothwell) +Newsgroups: comp.lang.lisp,comp.lang.scheme,comp.lang.misc +Subject: Re: Applicative languages? Anyone? +Keywords: ML interpreter typechecker +Message-ID: <819@its63b.ed.ac.uk> +Date: 8 Dec 87 12:40:13 GMT +References: <1409@mind.UUCP> <584@zippy.eecs.umich.edu> <1202@uoregon.UUCP> +Reply-To: nick%ed.lfcs@uk.ac.ucl.cs.nss (Nick Rothwell) +Organization: LFCS, University of Edinburgh +Lines: 37 +Xref: alberta comp.lang.lisp:566 comp.lang.scheme:85 comp.lang.misc:886 + +In article <1202@uoregon.UUCP> markv@drizzle.UUCP (Mark VandeWettering) writes: +>In article <584@zippy.eecs.umich.edu> dwt@zippy.eecs.umich.edu (David West) writes: +>>Applicativity has its advantages, but it needs +>>1) ... +>>2) Some syntactic means for preventing argumentsfrom getting unreadably +>> numerous just to pass something down to where it's finally used. +> +> Hmmm, not a bad idea. I have just acquired "Implementation of +> Functional Programming Languages by Simon L. Peyton Jones, and +> am much impressed by the depth/level of the text. Seeing as I +> have to do a final thesis/project sometime :-) I might be +> tempted to try a hand at an ML interpreter/compiler. I would +> like to hear from anyone who is trying/has tried similar +> projects. + +ML gives you objects with modifiable state, so that you don't need to +pass a state structure around with you. The disadvantage, of course, is +that you smash the applicative behaviour of the language - +whether it's worth it depends what you're trying to do. + Another way around this is to use type abstraction. That way, your +state structure is an abstract object with a few access functions to get +at the bits you need. I've always used the former approach, so I don't know +how far the latter approach gets you. It's quite possible to take non- +applicative features like assignment and abstract over them to build +structured objects with varying state, a la Smalltalk perhaps. This isn't +"dirty" functional programming - it's just using a functional language as if +it were a language of a different kind. I recently dedicated a lecture to the +structured use of side-effects in ML. + By the way, I have various little typecheckers and interpreters for tiny +functional languages lying around on-line somewhere, if you're interested. +All written in ML, of course. +-- +Nick Rothwell, Laboratory for Foundations of Computer Science, Edinburgh. + nick%lfcs.ed.ac.uk@nss.cs.ucl.ac.uk + !mcvax!ukc!lfcs!nick +~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ +"Nothing's forgotten. Nothing is ever forgotten." - Herne +#! rnews 1368 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!csnjr +From: csnjr@its63b.ed.ac.uk (Nick Rothwell) +Newsgroups: rec.music.synth +Subject: D-50, D-550, MT-32, ??? +Message-ID: <820@its63b.ed.ac.uk> +Date: 8 Dec 87 13:01:30 GMT +References: <633@elxsi.UUCP> <5470012@hplsla.HP.COM> +Reply-To: nick%ed.lfcs@uk.ac.ucl.cs.nss (Nick Rothwell) +Organization: LFCS, University of Edinburgh +Lines: 17 + +In article <5470012@hplsla.HP.COM> steveb@hplsla.HP.COM (Steve Bye) writes: +>The MT-32 is not a product of Roland's professional music products group. +>It is a product of their home keyboards (upscale toys) department. It uses +>technology develped for the D-50 and D-550. There is no comparison in +>actual ussuage between a D-550 and an MT-32. + +I recently read a report from a British music journalist visiting Roland in +Japan. Apparently (but *don't* quote me on this :-)) Roland are working on +a rack-mount box with the same sorts of features as the MT-32 but aimed a +bit more at the Pro market - presumably related to the MT-32 as the TX81Z is +to the FB01. I'm keeping my wallet closed and my eyes open... +-- +Nick Rothwell, Laboratory for Foundations of Computer Science, Edinburgh. + nick%lfcs.ed.ac.uk@nss.cs.ucl.ac.uk + !mcvax!ukc!lfcs!nick +~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ +"Nothing's forgotten. Nothing is ever forgotten." - Herne +#! rnews 1563 +Path: alberta!mnetor!uunet!mcvax!ukc!eagle!icdoc!doc.ic.ac.uk!aw +From: aw@doc.ic.ac.uk (Andrew Weeks) +Newsgroups: comp.emacs +Subject: uEmacs 3.9 - Function keys on Suns +Message-ID: <144@gould.doc.ic.ac.uk> +Date: 8 Dec 87 17:16:50 GMT +Sender: aw@doc.ic.ac.uk +Reply-To: aw@doc.ic.ac.uk (Andrew Weeks) +Organization: Dept. of Computing, Imperial College, London, UK. +Lines: 40 + +I have implemented, as an extension to the "VT100" option, some extra +code to allow uEmacs to recognise the top, left and right function keys +on Sun 3 consoles. ( I imagine they will work on Sun 2s as well). + +These keys, except for the cursor keys (R8,R10,R12 & R14), return a +string of the form [ followed by 3 digits followed by 'z'. By +interpreting the digits as an integer, and subtracting 128 to get a +character, all the function keys can be made to simulate 'FN?' keys. +Which they return depends on how the Sun keyboard is set up (with +setkeys(1)). + +They won't work if you use Sun-windows and have a .ttyswrc file. + +Anyway - Here are the diffs: + +*** input.c Mon Nov 30 12:57:21 1987 +--- input.c.orig Mon Nov 30 12:54:37 1987 +*************** +*** 364,376 **** + #if VT100 + if (c == '[' || c == 'O') { + c = get1key(); +! if ( c >= 'A' ) +! return(SPEC | c); +! c = c - 48; +! c = (c*10) + get1key() - 48; +! c = (c*10) + get1key() - 176; +! get1key(); +! return ( SPEC | c ); + } + #endif + return(META | c); +--- 364,370 ---- + #if VT100 + if (c == '[' || c == 'O') { + c = get1key(); +! return(SPEC | c); + } + #endif + return(META | c); +#! rnews 1248 +Path: alberta!mnetor!uunet!mcvax!ukc!dcl-cs!jam +From: jam@comp.lancs.ac.uk (John A. Mariani) +Newsgroups: rec.arts.sf-lovers +Subject: Re: Max Headroom +Message-ID: <454@dcl-csvax.comp.lancs.ac.uk> +Date: 8 Dec 87 18:47:18 GMT +References: <82*quale@si.uninett> <3333@ihlpl.ATT.COM> +Reply-To: jam@comp.lancs.ac.uk (John A. Mariani) +Distribution: rec.arts.sf-lovers +Organization: Department of Computing at Lancaster University, UK. +Lines: 16 + +Having observed chat about the American Max series and comparisons with the +UK series, I would like to point out that we (in the +UK) have only seen the Pilot in +terms of an action/adventure episode. Our Max series have really featured +Max as a video DJ, and later as a talk show host. + +So, I have kept silent till now, but I reckon the action/adventure series +you guys in the US of A are discussing must be worth watching! Anyone care +to hazard a guess as to why we in the UK don't get your Max show; and +do you get ours? + +-- +"You see me now a veteran of a thousand psychic wars .. " +UUCP: ...!seismo!mcvax!ukc!dcl-cs!jam | DARPA: jam%lancs.comp@ucl-cs +JANET: jam@uk.ac.lancs.comp | Post : University of Lancaster, Department of +Phone: +44 524 65201 ext 4467 | Computing, Bailrigg, Lancaster, LA1 4YR, UK. +#! rnews 1017 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!hwcs!jack +From: jack@cs.hw.ac.uk (Jack Campin) +Newsgroups: rec.music.classical +Subject: Minimalist recorder music, anyone? +Message-ID: <1575@brahma.cs.hw.ac.uk> +Date: 8 Dec 87 19:29:14 GMT +Organization: Computer Science, Heriot-Watt U., Scotland +Lines: 14 + +What minimalist music is performable by a recorder consort? Terry Riley's +In C is the one and only thing I've found so far (almost no published +minimal music is available in the UK - I have drawn a virtually complete +blank at every major library and music shop in Scotland). + +I guess this resolves into two questions: does it exist, and if it does, +can I get it? Do Glass et al have the same attitude to scores that AT&T +does to source code? + +-- +ARPA: jack%cs.glasgow.ac.uk@nss.cs.ucl.ac.uk +JANET:jack@uk.ac.glasgow.cs USENET: ...mcvax!ukc!cs.glasgow.ac.uk!jack +Mail: Jack Campin, Computing Science Department, University of Glasgow, + 17 Lilybank Gardens, Glasgow G12 8QQ, Scotland (041 339 8855 x 6045) +#! rnews 1165 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!hwcs!jack +From: jack@cs.hw.ac.uk (Jack Campin) +Newsgroups: comp.sys.mac +Subject: mathematical laser fonts +Keywords: font, logic, PostScript, laser printer, symbols +Message-ID: <1576@brahma.cs.hw.ac.uk> +Date: 8 Dec 87 19:43:07 GMT +Organization: Computer Science, Heriot-Watt U., Scotland +Lines: 21 + + +What mathematical laser fonts are available? + +What I need is: + + - logic and theoretical computer science symbols (like the old Ophir + bitmap font, but with the squared-off set theory symbols used in + domain theory); + + - symbols for the better known algebraic structures (N, Z, Q, A, R, C) + (is there a font that looks like these do as usually printed?); + + - subscripts and superscripts with little enough leading not to + sabotage inter-line spacing in programs like WriteNow; + + - maybe some of the more useful German capital letters. +-- +ARPA: jack%cs.glasgow.ac.uk@nss.cs.ucl.ac.uk +JANET:jack@uk.ac.glasgow.cs USENET: ...mcvax!ukc!cs.glasgow.ac.uk!jack +Mail: Jack Campin, Computing Science Department, University of Glasgow, + 17 Lilybank Gardens, Glasgow G12 8QQ, Scotland (041 339 8855 x 6045) +#! rnews 1030 +Path: alberta!mnetor!uunet!mcvax!ukc!cheviot!eas +From: eas@cheviot.newcastle.ac.uk (Edward Scott) +Newsgroups: rec.arts.sf-lovers +Subject: Re: Obscure TV SF shows +Message-ID: <2588@cheviot.newcastle.ac.uk> +Date: 8 Dec 87 15:19:25 GMT +References: <871201124327980.ABWD@Mars.UCC.UMass.EDU> <4100001@hpcllf.HP.COM> +Reply-To: eas@cheviot (Edward Scott) +Organization: Computing Laboratory, U of Newcastle upon Tyne, UK NE17RU +Lines: 12 + +In article <4100001@hpcllf.HP.COM> jws@hpcllf.HP.COM (John Stafford x75743) writes: +>Re: UFO +> The wigs worn by the women on moonbase were of a purple hue and were +> described (at least in the books the followed the series if not +> actually on the air) as "anti-static wigs". + +About ten years ago I got a second hand copy of "UFO 1: Flesh Hunters" by +Robert Miall. It is a Warner Paperback Library edition, printed with +permission from Pan books (who presumably did the UK edition). I have't seen +any since then. +How many of these UFO novels were there? +Did Robert Miall write anything else? +#! rnews 542 +Path: alberta!mnetor!uunet!mcvax!ukc!stc!pete +From: pete@tcom.stc.co.uk (Peter Kendell) +Newsgroups: rec.arts.sf-lovers +Subject: No More Mel +Message-ID: <488@stc-f.tcom.stc.co.uk> +Date: 8 Dec 87 09:14:20 GMT +Organization: STC Telecoms, London N11 1HB. +Lines: 7 + + + Hurrah, Hurrah!! +-- +------------------------------------------------------------------------------ +| Peter Kendell | +| ...{uunet!}mcvax!ukc!stc!pete | +------------------------------------------------------------------------------ +#! rnews 660 +Path: alberta!mnetor!uunet!mcvax!ukc!stc!btnix!psanders +From: psanders@btnix.axion.bt.co.uk (Bob-Cut Maniac) +Newsgroups: comp.sys.mac +Subject: SMALLTALK wanted +Keywords: Mac SMALLTALK +Message-ID: <635@btnix.axion.bt.co.uk> +Date: 8 Dec 87 12:53:54 GMT +Organization: British Telecom Research Labs, Martlesham Heath, IPSWICH, UK +Lines: 10 + + +Does anyone know of a PD SMALLTALK system for the Mac ?? + +Answers to me and I'll summarise on the Net. + +Paul. +-- +E-mail (UUCP) PSanders@axion.bt.co.uk (...!ukc!btnix!psanders) +Organisation British Telecom Research Laboratories, Ipswich UK. +"This mime of mortal life, in which we are apportioned roles we misinterpret..." +#! rnews 628 +Path: alberta!mnetor!uunet!mcvax!ukc!stc!root44!hrc63!trw +From: trw@hrc63.co.uk (Trevor Wright Marconi Baddow) +Newsgroups: comp.sys.ibm.pc +Subject: M.Magee AUTOMENU - any knowledge +Message-ID: <475@hrc63.co.uk> +Date: 8 Dec 87 10:39:57 GMT +Organization: GEC Hirst Research Centre, Wembley, England. +Lines: 10 + + +We have seen a demo of a tiny MS-DOS utility called AUTOMENU which +makes building menus for PC users simple. We want to find who is the +vendor of this utility, the cost, and any details of the command characters +for the menu definition file. + +Any help appreciated. + +Trevor Wright +yc23%a.gec-mrc.co.uk@nss.cs.ucl.ac.uk +#! rnews 2646 +Path: alberta!mnetor!uunet!mcvax!ukc!stc!datlog!dlhpedg!cl +From: cl@dlhpedg.co.uk (Charles Lambert) +Newsgroups: comp.lang.c +Subject: Re: Address of array +Message-ID: <329@dlhpedg.co.uk> +Date: 8 Dec 87 13:30:45 GMT +References: <126@citcom.UUCP> <163@mccc.UUCP> <422@xyzzy.UUCP> +Sender: news@dlhpedg.co.uk +Reply-To: cl@.co.uk (Charles Lambert) +Organization: FSG@Data Logic Ltd, Queens House, Greenhill Way, Harrow, London. +Lines: 56 + +In article <422@xyzzy.UUCP> throopw@xyzzy.UUCP (Wayne A. Throop) writes: +>> pjh@mccc.UUCP (Peter J. Holsberg) +>> OK - perhaps you had better tell us neophytes what you mean by the +>> address of an array! +> +>Same as address of anything else. It is an address which, when +>indirected, yields an array, and when "N" is added to it, yields the +>address of an array which is itself a member of an array "N" elements +>away from the array yielded by an indirection. +> +> [ several abstruse observations ] +> +>What could be simpler? + +Well, several other forms of explanation, I guess. This one confused me, +and I *understand* the address of an array. (Just teasing) + +To put it another way.... + +Any object, of any type (integer, structure, array, etc.), has an address. +Usually, if it is an object that occupies several words of memory, it is the +address at which it begins. (Compiler theorists may be itching to tell me it +might mean something else entirely; let's keep this simple.) The address of +an object is the compiler's handle for manipulating it. You think of an +object by its name; the compiler "thinks" of it by its address. + +The "address of an array" is the address that the compiler uses to access +that array and to calculate the position of any element in the array. + +In C, the address of an array is the same as the address of its first +element (array[0]). If you want to set up a pointer to the array, you +get its address simply by naming it. Hence: + + pa = array; /* pa now contains the address of "array" */ + +which is exactly the same as + + pa = &array[0]; /* "&" means "address of", so pa contains the + address of element [0] of "array" */ + +Now this is a slight quirk in C - the name of the array being a synonym for +its address; for any other object (notably a struct) that is not true. If +you want the address of a structure you must write + + ps = &mystruct; /* NOT ps = mystruct */ + +So we get back to the discussion from whence we came: why can't we be +consistent and get the address of an array by + + pa = &array; ? + +To which the answer is: you can, with some compilers. + +[Further reading: The C Programming Language; Kernighan & Ritchie; pp.93-95] +-------------------------- +Charles Lambert +#! rnews 1652 +Path: alberta!mnetor!uunet!mcvax!unido!iaoobelix!vogt +From: vogt@iaoobelix +Newsgroups: comp.sys.dec +Subject: Bug in BASIC-PLUS for RSTS V8.0? - (nf) +Message-ID: <9900003@iaoobelix.UUCP> +Date: 8 Dec 87 18:36:00 GMT +Lines: 43 +Nf-ID: #N:iaoobelix:9900003:000:1342 +Nf-From: iaoobelix!vogt Dec 8 19:36:00 1987 + +I think I found a bug in BASIC-PLUS of RSTS V8.0. The following program +isn't working in the right way. I tried to read some records from a file +and to store them in an array. But after I read and stored all records, +the array was completely empty. + +> 10 ON ERROR GOTO 1000 +> 15 DIM IN$(100%) +> 20 FIELD #1%, 3% as a$, 20% as i$, 15% as q$ +> 30 OPEN 'foobar' as file #1%, recordsize 38% +> 40 Z% = 0% +> 50 Z% = Z% + 1% +> 60 GET #15%, RECORD Z% +> 70 IN$(Z%) = I$ +> 75 PRINT IN$(Z%) +> 80 GOTO 50 +> 90 CLOSE #1% +> 100 PRINT IN$(I%) FOR I% = 1% TO Z% - 1% +> 110 GOTO 32767 +> 1000 IF ERR = 11 THEN RESUME 90 +> 1010 ON ERROR GOTO 0 +> 32767 END + +The outputs in line 75 are alright, but those in line 100 aren't. +Only blank lines appear there. + +I found out that if you change line 70 to 'IN$(Z%) = LEFT$(I$, 20%)' +- which does nearly nothing different - it works correctly. + +Does anybody know a patch for this bug? Or does anybody know how to +avoid this in an other way? + +Thanks in advance + +Gerald Vogt + +-------------------------------------------------------------------------- +Fraunhofer Institut fuer Arbeitswirtschaft und Organisation +Holzgartenstrasse 17 +D-7000 Stuttgart 1 UUCP: ...{uunet!unido,pyramid}!iaoobel!vogt +W-Germany + +Phone: (W-Germany) 711 6648191 +-------------------------------------------------------------------------- +#! rnews 3127 +Path: alberta!mnetor!uunet!mcvax!hafro!gst!gunnar +From: gunnar@gst.UUCP (Gunnar Stefnsson) +Newsgroups: sci.math +Subject: Re: Least-squares fitting +Message-ID: <428@gst.UUCP> +Date: 8 Dec 87 15:25:35 GMT +References: <1823@culdev1.UUCP> <22191@cca.CCA.COM> <2301@utastro.UUCP> +Reply-To: gunnar@gst.UUCP (Gunnar Stefansson) +Organization: Marine Research Institute, Reykjavik +Lines: 56 + +In article <2301@utastro.UUCP> bill@astro.UUCP (William H. Jefferys) writes: +>In article <22191@cca.CCA.COM> g-rh@CCA.CCA.COM.UUCP (Richard Harter) writes: +>~In article <1823@culdev1.UUCP> drw@culdev1.UUCP (Dale Worley) writes: +>~>The normal least-squares fitting of a line to a set of points in the +>~>plane assumes that the x-coordinates of the points are known to be +>~>exact, and the y-coordinates have all the error. That is, chi^2 is +>~>the sum of the squares of the distances from the points to the line in +>~>a vertical direction. This introduces assymetry between the +>~>coordinates. +>~> +>~>Is is known how to perform least-squares fitting where the "error" is +>~>the perpendicular distance between the point and the line? +> +> +>Actually, if both coordinates have error, it is essential that this +>fact be taken into account. If you fail to do this, the result will be +>*biased* -- the slope will be systematically underestimated, and +>this bias will not go to zero as you take more and more points + +Hold on, isn't this statement a bit too strong? The answer to which method +should be used ultimately depends on what the purpose of the estimations +is. + +In fact, if the purpose is to estimate y for a given x, then ordinary +least squares will do. In this case one is not really interested in +getting the best estimates of the parameters but only in getting a good +prediction. + +I claim that there are very few regression examples where one really +cares whether or not the parameters are biased. In the large majority of +cases one is much more interested in the goodness of prediction. In this +case, one is interested in E[Y|X]. So if we model this quantity as +linear in X, then the OLS estimates are BLUE. This will also give +variances etc, all valid conditionally on X. + +It is my feeling that a lot of books overemphasize the so-called bias, +since that is very often totally irrelevant. For example, some +textbooks talk about biased parameter estimates when some variables +are missing in a multiple regression. In reality OLS is estimating a +better set of parameters than would the corresponding "unbiased" +estimator (OLS in this case will give an unbiased estimate of the best +surface based on the reduced set of variables). Certainly in this case, +one can make a strong argument that all the talk about biasses is +totally irrelevant. + +Of course if the true purpose is to estimate parameters, e.g. to assess +the effect of a change in X on Y, then indeed one needs to worry a bit +about the effects of X being random. + +Gunnar + +-- + +----------------------------------------------------------------------------- +Gunnar Stefansson {mcvax,enea}!hafro!gunnar +Marine Research Institute, Reykjavik gunnar@hafro.UUCP +#! rnews 528 +Path: alberta!mnetor!uunet!mcvax!unido!tub!ao +From: ao@tub.UUCP (Arnfried Ossen) +Newsgroups: comp.mail.misc +Subject: Path to UMass Amherst +Message-ID: <318@tub.UUCP> +Date: 7 Dec 87 13:29:49 GMT +Reply-To: ao@tub.UUCP (Arnfried Ossen) +Organization: Technical University of Berlin, Germany +Lines: 7 + +Anybody out there who knows the PATH to + + University of Massachusetts, Amherst Campus, COINS Department + +It should allow access from USENET or BITNET. + +Arnfried, ao@tub.UUCP, ao@db0tui6.BITNET, TU Berlin, Berlin, Fed.Rep.Germany +#! rnews 2902 +Path: alberta!mnetor!uunet!mcvax!varol +From: varol@cwi.nl (Varol Akman) +Newsgroups: sci.math +Subject: Re: computational geometry / finding segment intersections +Summary: Try adaptive grid ... +Keywords: segment intersection +Message-ID: <141@piring.cwi.nl> +Date: 9 Dec 87 11:14:57 GMT +References: <4369@sdcsvax.UCSD.EDU> +Organization: CWI, Amsterdam +Lines: 50 + +<4369@sdcsvax.UCSD.EDU> maiden@sdcsvax.UCSD.EDU (VLSI Layout Project) writes: +> +>Consider a path embedded into the Cartesian plane, where for convenience +>all vertices of the path are lattice points in the positive quadrant. +>All edges are line segments. +>So, the path will look like < (x1,y1) , (x2,y2) , ... , (xn,yn) >. +>Question: What is the fastest method of determining *ALL* self- +> intersections of this path? +>This may have been beaten to death by computational geometers, so I'll +>append some extra conditions: +>Suppose there are **many** vertices in the path, and that edges are +>for the most part very short. For example, there could be 10000 +>points in a 200 by 200 square, with most edges less than 3 units long. +>Furthermore, assume that there are not very many self-intersections +>to be found. Now, what would the fastest method be??? Any ideas +>welcome. + +There are, as you've guessed several papers in computational geometry +on line segment intersections. You may look at the books by Shamos +and Preparata, and also the book by Edelsbrunner for references. + +My favorite method to solve your problem though is an excellent +method invented by Randolph Franklin at RPI. It is called ''adaptive +grid'' and works as follows. First you overlay a regular, say G by G +integer grid on your scene. Then you enter your edges into respective +cells of the grid (similar to the bucketing idea!) Then you make a pass +thru all the cells and find the intersections in each cell. If an +intersection falls on a grid cell boundary you should be careful to +treat it so the integrity is kept intact. + +I'm not very good in describing things in a hurry (especially Email) +but let me tell that I've wide experience with this stuff and it works +very well. It is especially excellent for a scene made of short edges +with a rather homogeneous distribution. Write me for details. +Also you may try Franklin at franklin@csv.rpi.edu. Here is a short bibl. + +W.R. Franklin An exact hidden sphere algorithm that operates + in real time COMP. GRAPHICS AND IMAGE PROC. 15(4), 1981 + +------------- A linear time exact hidden surface algorithm SIGGRAPH'80 + +------------- and V. Akman A simple and efficient haloed line algorithm + for hidden line elimination COMPUTER GRAPHICS + FORUM, 1987 + +-------------------------- Adaptive grid for polyhedral visibility in + object space: an implementation BJC 1987, to appear + +-Varol Akman +CWI, Amsterdam +#! rnews 2392 +Path: alberta!mnetor!uunet!mcvax!jack +From: jack@cwi.nl (Jack Jansen) +Newsgroups: comp.os.misc,comp.unix.wizards +Subject: Re: Command interfaces +Message-ID: <142@piring.cwi.nl> +Date: 9 Dec 87 15:41:45 GMT +References: <1257@boulder.Colorado.EDU> <6840002@hpcllmv.HP.COM> <9555@mimsy.UUCP> <798@rocky.STANFORD.EDU> <432@cresswell.quintus.UUCP> <3161@psuvax1.psu.edu> <5565@oberon.USC.EDU> +Organization: AMOEBA project, CWI, Amsterdam +Lines: 43 +Xref: alberta comp.os.misc:339 comp.unix.wizards:5747 + +In article <5565@oberon.USC.EDU> blarson@skat.usc.edu (Bob Larson) writes: +> [Discussing primos wildcards versus unix wildcards] +>For example, how would you do the equivelent of this in unix: +> +>cmpf *>old>@@.(c,h) == -report ==.+cmpf -file +> +>(Explanation: compare all files in the old sub-directory ending in .c or +>.h with the file of the same name in the current directory, and put +>the output in the file of the same name with .cmpf appended. Non-files +>(directories and segment directories) ending in .c or .h are ignored. +>[I do prefer the output of diff -c to that of cmpf, but that isn't +>what I'm talking about here.] + +Uhm, yes, unfortunately I find the 'feature' quite unusable. +I *never* come up with the correct sequence of == and @@, so I have to type +the command three times before I get it right. (really retype, that is. +'History mechanism' is something primos has never heard about). + +I definitely prefer +for i in *.[ch]; do + diff old/$i $i >$i.diff +done + +(and you can add an 'if [ -d $i ]' if you really care about directories +ending in .c or .h. I don't, because I don't *have* directories ending +in .c or .h). + +And, to continue some gripes on primos wildcards: +- I would expect them to work *always*. I.e. if I do + TYPE @@ + (TYPE is primos echo) I would expect a list of all files, *not* '@@'. +- If I want all arguments on one line, and I use [WILD @@.TMP], and the + result doesn't fit in 80 characters, I DO DEFINITELY NOT WANT IT TO TRUNCATE + IT AT EIGHTY CHARS! I lost an important file that way: it was trying + to generate a list containing PRECIOUSFILE.TMP, but, unfortunately, + the .TMP started at position 81. So, it removed PRECIOUSFILE in stead. + sigh. + +Sorry, there are some neat ideas in primos, but the command processor and +it's wildcards is definitely *not* one of them. +-- + Jack Jansen, jack@cwi.nl (or jack@mcvax.uucp) + The shell is my oyster. +#! rnews 1552 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!csbg +From: csbg@its63b.ed.ac.uk (Andie) +Newsgroups: comp.windows.news +Subject: Windows and menus through the CPS interface +Keywords: NeWS windowing, menus, CPS interface +Message-ID: <821@its63b.ed.ac.uk> +Date: 9 Dec 87 00:15:36 GMT +Reply-To: csbg@its63b.ed.ac.uk (Bruce) +Organization: Computer Science Department, Edinburgh University +Lines: 26 + +Hi everybody ! + +I'm a final year student at Edinburgh University and as part of my final +year project I am using NeWS to build up a document composition system. +Alas, I'm new to NeWS and the NeWS manual does seem to be rather sketchy, +especially when it comes to using the CPS interface. + +Having had a look at the stuff that is floating around in this newsgroup +I think that someone out there will be able to help me. + +Point 1 : How can I control the litewin.ps and litemenu.ps packages through + the CPS interface - especially, how do I get notification to the + C program that something is happening ? + +Point 2 : This may be trivial, but when I create an overlay for the purposes + of rubber-banding, using the getclick family of operators, I can + never get the overlay to disappear again. What is happening and + how should it be done ? + +If these points have already been raised in the past then I will be happy to +receive direct e-mail from anybody who can answer any part of the above +queries. + +As they say: When the going gets tough, I get the hell out of it ! + +Bruce Gilmour (CS4 student at Edinburgh University) +#! rnews 1134 +Path: alberta!mnetor!uunet!mcvax!botter!ast +From: ast@cs.vu.nl (Andy Tanenbaum) +Newsgroups: comp.os.minix +Subject: Re: Problems with serial TTY driver +Message-ID: <1778@botter.cs.vu.nl> +Date: 9 Dec 87 15:36:52 GMT +References: <2314@encore.UUCP> +Reply-To: ast@cs.vu.nl (Andy Tanenbaum) +Organization: VU Informatica, Amsterdam +Lines: 16 + +In article <2314@encore.UUCP> paradis@encore.UUCP (Jim Paradis) writes: +>Is there some limit to how fast MINIX will take interrupts? +>If one takes them too fast, will messages get lost? +> +If you try to force feed MINIX from an Ethernet at 10 Mbps it will probably +drop stuff. There is undoubtedly a limit on how many interrupts per second +it can handle, but an AT it should be over 1000 per second. + +The original tty driver was very carefully written to deal with exactly +this issue. When characters come in, they are buffered, even if it is +not possible to send a message to the tty task. This code is on lines +3528 to 3552 of the book. Assuming you are still using this mechanism, +you ought to be able to accept characters at say 2400 baud without losing +any. + +Andy Tanenbaum (ast@cs.vu.nl) +#! rnews 915 +Path: alberta!mnetor!uunet!mcvax!botter!ast +From: ast@cs.vu.nl (Andy Tanenbaum) +Newsgroups: comp.os.minix +Subject: P-H has MINIX in stock (finally) +Message-ID: <1779@botter.cs.vu.nl> +Date: 9 Dec 87 15:46:14 GMT +Reply-To: ast@cs.vu.nl (Andy Tanenbaum) +Organization: VU Informatica, Amsterdam +Lines: 11 + + +I talked to P-H yesterday. Version 1.2 of MINIX in 256K & 640K PC, 512K AT, +mag tape, and the IBM slipcase version with the abridged book are all +in stock. If it is of any consolation to the people who have had to wait +and wait and wait, one of the corporate vice presidents was so unhappy +about the poor service to customers that he fired the person who was in charge +of managing the MINIX inventory. He has been replaced by someone else who has +clear instructions to make sure it doesn't go out of stock again. They are now +shipping to everyone whose order got backlogged. + +Andy Tanenbaum (ast@cs.vu.nl) +#! rnews 841 +Path: alberta!mnetor!uunet!mcvax!prlb2!lln-cs!gf +From: gf@lln-cs.UUCP (Frank Grognet) +Newsgroups: rec.games.misc,rec.games.frp,rec.games.board +Subject: WARGAMING! +Keywords: wargame,rule,figurine,game +Message-ID: <796@lln-cs.UUCP> +Date: 9 Dec 87 15:19:13 GMT +Organization: Computer Science Dept., Louvain-la-Neuve Belgium +Lines: 11 +Xref: alberta rec.games.misc:1150 rec.games.frp:1652 rec.games.board:543 + + + I want to start wargaming but I don't know how! + +I won't be playing wargames on a board, but with 15mm or 25mm +figurines. +I would like to find addresses in Europe (especially Belgium) +of good figurine manufacturers and also references to rule +books for the Napoleonic period. +I am also interested in rules contained on the net or in files at +other sites, if they exist! +I anybody can help me, please reply to ..!mcvax!prlb2!lln-cs!gf +#! rnews 1656 +Path: alberta!mnetor!uunet!mcvax!nikhefk!frankg +From: frankg@nikhefk.UUCP (Frank Geerling) +Newsgroups: comp.sys.atari.st +Subject: Re: the perfect ram disk +Keywords: ramdisk, resizeable, reset-survivable +Message-ID: <294@nikhefk.UUCP> +Date: 9 Dec 87 19:53:23 GMT +References: <427@dukempd.UUCP> +Reply-To: frankg@nikhefk.UUCP (Frank Geerling) +Organization: Nikhef-K, Amsterdam (the Netherlands). +Lines: 42 + +In article <427@dukempd.UUCP> gpm@dukempd.UUCP (Guy Metcalfe) writes: +>I have Mike's Ramdisk v. .95, and like the idea of what it's trying to do. +>It has a dialogue box as if it were resizable, but it's very buggy. Could +>someone send me a later version that works like it's dialogue implies it +>should. What I would like best of all is an eternal ram disk that I can +>size up and down as I see fit, but which sizes down without letting me +>destroy any data I may have on the disk. If anybody has and would send me +>or knows where I could get such a beast, I would be grateful. Thanks. +>-- +> Guy Metcalfe gpm@dukempd.uucp + + +Please send it to me too, I also have Mike's Ramdisk and the resize doesn't +work it doesn't return allocated memory when you resize to a smaller amount +of memory. + +Thanx in advance + + + Frank Geerling + (frankg@nikhefk.uucp) + + +Usenet: {seismo, philabs, decvax}!mcvax!frankg@nikhefk + +Normal mail: Frank Geerling + NIKHEF-K (DIGEL) + Postbus 4395 + 1009 AJ Amsterdam + The Netherlands + + Frank Geerling + (frankg@nikhefk.uucp) + + +Usenet: {seismo, philabs, decvax}!mcvax!frankg@nikhefk + +Normal mail: Frank Geerling + NIKHEF-K (PIMU) + Postbus 4395 + 1009 AJ Amsterdam + The Netherlands +#! rnews 2666 +Path: alberta!mnetor!uunet!mcvax!prlb2!kulcs!luc +From: luc@kulcs.UUCP (Luc Van Braekel) +Newsgroups: comp.lang.pascal +Subject: Re: self-replicating programs? +Summary: here is a self-replicating pascal program +Message-ID: <1070@kulcs.UUCP> +Date: 9 Dec 87 08:31:27 GMT +References: <1400@tulum.swatsun.UUCP> +Organization: Kath.Univ.Leuven, Comp. Sc., Belgium +Lines: 37 + +In article <1400@tulum.swatsun.UUCP>, hirai@swatsun (Eiji "A.G." Hirai) writes: +> In our recent ACM programming contest (regionals), one of the +> problems was to write a self-replicating program. That is, we had to +> write a program whose output was itself, the source code. No alterations +> of the original code during execution was allowed (I think). +> Does anyone have any code for this problem? We have one but +> it looks inelegant. I've also see bery bery short Prolog code for this. +> Help, we are looking for good codes to study! And yes, the contest is +> over (we ain't cheating). + +Here is a self-replicating Pascal program I wrote a few years ago. +The program looks dirty but it works ! + +program self (output); +var i,j: integer; + a: array[1..8] of packed array[1..59] of char; begin + a[1] := 'program self (output); '; + a[2] := 'var i,j: integer; '; + a[3] := ' a: array[1..8] of packed array[1..59] of char; begin '; + a[4] := 'for i := 1 to 3 do writeln(a[i]); '; + a[5] := 'for i := 1 to 8 do begin write('' a['',i:0,''] := '',chr(39));'; + a[6] := 'for j := 1 to 59 do begin write(a[i][j]);if a[i][j]=chr(39)'; + a[7] := 'then write(a[i][j]) end; writeln(chr(39),'';'') end; '; + a[8] := 'for i := 4 to 8 do writeln(a[i]) end. '; +for i := 1 to 3 do writeln(a[i]); +for i := 1 to 8 do begin write(' a[',i:0,'] := ',chr(39)); +for j := 1 to 59 do begin write(a[i][j]);if a[i][j]=chr(39) +then write(a[i][j]) end; writeln(chr(39),';') end; +for i := 4 to 8 do writeln(a[i]) end. + ++-----------------------------------+------------------------------------+ +| Name : Luc Van Braekel | Katholieke Universiteit Leuven | +| UUCP : luc@kulcs.UUCP | Department of Computer Science | +| BITNET : luc@blekul60.bitnet | Celestijnenlaan 200 A | +| Phone : +(32) 16 20 0656 x3563 | B-3030 Leuven (Heverlee) | +| Telex : 23674 kuleuv b | Belgium | ++-----------------------------------+------------------------------------+ +#! rnews 1678 +Path: alberta!mnetor!uunet!mcvax!cernvax!ethz!zu +From: zu@ethz.UUCP (Urs Zurbuchen) +Newsgroups: comp.sys.ibm.pc +Subject: Re: Oooh Yeccheo. How Does This One Really Work?!? +Message-ID: <264@bernina.UUCP> +Date: 9 Dec 87 12:16:51 GMT +References: <164300022@uiucdcsb> <412@wa3wbu.UUCP> <13091@beta.UUCP> <1269@phoenix.Princeton.EDU> +Reply-To: zu@bernina.UUCP (Urs Zurbuchen) +Organization: ETH Zuerich, CS Department, Switzerland +Lines: 41 + +In article <1269@phoenix.Princeton.EDU> rjchen@phoenix.Princeton.EDU (Raymond Juimong Chen) writes: +>In article <13091@beta.UUCP> it was written: +>What you'd probably want is something like +> +>AUTOEXEC.BAT: +> doit +> +>DOIT.BAT: +> copy \autoexec.ddd \autoexec.bat +> del \autoexec.ddd +> do other stuff +> reboot. +> +>AUTOEXEC.DDD: +> same as before + +You could the same thing without changing your AUTOEXEC.BAT. With the solution +presented above you will execute the same second version of AUTOEXEC.BAT each +time you reboot your machine (perhaps that's really what you want, but my +imagination doesn't go that far. If so, just disregard this article). + +My solution: In the startup file you include the following: + +if exist goto second + +echo gaga > +:second + + +That's it. If you want to toggle between the two boot modes just add a line +like: + +del + + + I hope this will help anybody :-) + + ...urs + + +UUCP: ...seismo!mcvax!cernvax!ethz!zu +#! rnews 420 +Path: alberta!mnetor!uunet!mcvax!inria!irisa!michaud +From: michaud@irisa.UUCP (Michaud Franck INSA BN205) +Newsgroups: comp.protocols.tcp-ip +Subject: virtual circuit +Keywords: tcp, socket +Message-ID: <202@irisa.UUCP> +Date: 9 Dec 87 20:05:21 GMT +Organization: IRISA, Rennes (Fr) +Lines: 7 + + + I'd like to have a good definition of : +- virtual circuit. + + If you have a good definition, send me a mail. + thanck you. + franck +#! rnews 758 +Path: alberta!mnetor!uunet!mcvax!enea!liuida!dat08 +From: dat08@butterix.liu.se +Newsgroups: rec.games.frp +Subject: Re: New rules for AD&D +Message-ID: <686@butterix.liu.se> +Date: 9 Dec 87 03:53:09 GMT +References: <26788S9S@PSUVMA> +Organization: CIS Dept, Univ of Linkoping, Sweden +Lines: 11 + +In article <26788S9S@PSUVMA> S9S@PSUVMA.BITNET (Steven A. Schrader) writes: +>New Rules for TSR. [...] Does anyone know when these rules will be out +>and how much they will cost? + +According to Harold Johnson of TSR (at a local convention in Sweden) the new +rules will be out in 89. + +BTW -- Any reactions about the new (again!) Gamma World? I haven't tried it +yet but I like their idea of one-table-system for everything. + +Per Westling dat08@majestix.liu.se +#! rnews 968 +Path: alberta!mnetor!uunet!mcvax!enea!tut!tolsun!reini +From: reini@tolsun.oulu.fi (Jukka Reinikainen) +Newsgroups: comp.sys.ibm.pc,comp.sources.wanted +Subject: Hercules graphic characters +Keywords: hercules, text, MASM, MSC +Message-ID: <246@tolsun.oulu.fi> +Date: 8 Dec 87 15:32:30 GMT +Organization: University of Oulu, Finland +Lines: 14 +Xref: alberta comp.sys.ibm.pc:9576 comp.sources.wanted:2717 + + + +Help wanted: how to create text in Hercules graphic mode? + +I have a program written in MSC (parts coded with MASM) which does +quite nice things with grapichs but suffers lack of characters. +According to my knowledge the only way to get characters in Herc graphic +mode is to draw them on screen by lightning a set of pixels, right? + +Somebody *must* have written a program which draws characters and +other symbols, so please help me. C and/or ASM sources and/or ideas +will be *very* appreciated. + + > Jukka Reinikainen reini@tolsun.oulu.fi < +#! rnews 935 +Path: alberta!mnetor!uunet!mcvax!enea!liuida!andka +From: andka@smidefix.liu.se (Andreas K}gedal) +Newsgroups: rec.music.synth +Subject: Yamaha CLP - pf question +Keywords: Yamaha pf85 CLP300 +Message-ID: <687@smidefix.liu.se> +Date: 9 Dec 87 15:36:48 GMT +Organization: CIS Dept, Univ of Linkoping, Sweden +Lines: 13 + + + I'm thinking of getting one of those new sampled pianos and would like +to get som info. From the net and from my own experience in my local +piano store, I've understood that the Yamaha Clavinova CLP 300 is +a pretty good choise. But I seem to remember a rumor about something +called Yamaha pf85 wich would be some kind of stageversion of the CLP 300. +Has anyone seen it, played it, compared it with the CLP 300? What are the +differences in price, sound, keyboard? + +My local pianopusher here in Sweden hadn't heard of it. Is this because +it is so new or because it is a local phenomenon in the states? + + /Andreas Kagedal +#! rnews 2816 +Path: alberta!mnetor!uunet!mcvax!enea!ttds!draken!sics!erikn +From: erikn@sics.se (Erik Nordmark) +Newsgroups: comp.unix.questions +Subject: Re: Need help with interprocess communications +Keywords: Pipes, Ptys, Buffering, I/O +Message-ID: <1639@sics.se> +Date: 9 Dec 87 21:21:43 GMT +References: <8117@steinmetz.steinmetz.UUCP> +Reply-To: erikn@sics.UUCP (Erik Nordmark) +Organization: Swedish Institute of Computer Science, Kista +Lines: 60 + +[[ I tried sending this as mail using different addresses, but failed! ]] + +In article <8117@steinmetz.steinmetz.UUCP> you write: +> +> +>I have tried using "fcntl(fd,F_SETFL,FASYNC)" as well as setting up an +>interrupt handler to handle SIGIO signals (via "sigvec(2)"), and this works +>fine when I'm reading from the terminal, but does not seem to work at all +>when I try it from a pipe. +> +> +>Well, the SIGIO handler works fine to detect input from places like stdin, but +>never sees anything coming down the pipe. When it gets invoked (generally +>by me banging on the key causing an interrupt from stdin), it +>does find that there is data available in the pipe (as well as stdin) and +>has no problem reading it. +> +> +>Does anyone out there know how I can fix this problem? +> + +>From looking at the BSD4.3 sources I found out the following: +When a tty is opened the associated process group is set to +that of the creator. The signals that the tty driver generate (e.g. caused +by ^C) are sent to this process group. + +However, for sockets (a pipe is implemented as a pair of sockets in BSD4.3 +and maybe elsewhere!) the associated process group is not set automatically. + +So what you have to do is to set it before you can get ant SIGIO's! Use + int pgrp = getpid(); + if (fcntl(fd, F_SETOWN, pgrp) == -1) { + perror("fnctl"); + exit(1); + } +or + ioctl(fd, SIOCSPGRP, &pgrp) /* note: & */ + +I think this should work even if pipes aren't implemented as a pair of +sockets, but I haven't tried any of it. + +>Also: Is there a way that I can determine WHICH file descriptor caused +>a SIGIO interrupt to be invoked, or by which I can set up a different +>interrupt handler for each descriptor? +> + +See select(2). (Just a detail: select will tell you that there is data +to read if there actually is data to read or if the other end(s) have +closed the pipe. In the latter case read() will return an EOF - this +stuff caused me some trouble before I read the *real* documentation - +the OS source code!!) + +------------------------------------------------------------------------- +Erik Nordmark +Swedish Institute of Computer Science, Box 1263, S-163 13 SPANGA, Sweden +Phone: +46 8 750 79 70 Ttx: 812 61 54 SICS S Fax: +46 8 751 72 30 + +uucp: erikn@sics.UUCP or {seismo,mcvax}!enea!sics!erikn +Domain: erikn@sics.se +------------------------------------------------------------------------- +#! rnews 2508 +Path: alberta!mnetor!uunet!mcvax!enea!luth!d2c-czl +From: d2c-czl@sm.luth.se (Caj Zell) +Newsgroups: rec.music.misc +Subject: Re: Ace-Screamingest Guitar Solos on Record +Keywords: guitar, flames (regrettably) +Message-ID: <438@psi.luth.se> +Date: 9 Dec 87 14:51:22 GMT +References: <1725@s.cc.purdue.edu> +Reply-To: Caj Zell +Organization: University of Lulea, Sweden +Lines: 44 +UUCP-Path: {uunet,mcvax}!enea!psi.luth.se!d2c-czl + + +In article <1725@s.cc.purdue.edu> rsk@s.cc.purdue.edu (Rich Kulawiec) writes: +>I thought I'd make up a very hasty list of what I +>thought were some of the best solos I've heard, and then ask y'all to +>contribute further. + +Good idea,I love making up lists! + +>Money (Pink Floyd), David Gilmour +>Cracked Actor (David Bowie), Earl Slick +>Don't Take Me Alive (Steely Dan), Jeff 'Skunk' Baxter? +>All Along the Watchtower Jimi Hendrix +>Aqualung (Jethro Tull), Martin Barre +>Highway 61, Johnny Winter + +Agree,but how about these: + +Muffin Man (Frank Zappa) (I think FZ was the most underrated) +Son of Mr. Green Genes (Frank Zappa) (guitarist there ever has been.But,) +Son of Orange County (Frank Zappa) (he can't play anymore,too bad. ) +Push Comes To Show (Van Halen) Eddie Van Halen +Crossroads (Cream) Eric Clapton (The 2nd solo,of course) +Astronomy (Blue \yster Cult) Donald Roeser (on "Some Enchanted Evening") +Lazy (Deep Purple) Ritchie Blackmore +Fat Time (Miles Davis) Mike Stern + +I know that when I get home I will kill myself for not adding more solos, +but these are the ones I can think of without looking at my records. +But maybe that's a good sign indicating that these are really my favourites. + +I'd be very glad to see some reactions on the list. + + + XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX + X X + X X + X Caj Zell ________________________ X + X University of Lulea : : X + X Sweden : Jazz is not dead, : X + X : it just smells funny : X + X mail: d2c-czl@psi.luth.se : -Frank Zappa : X + X : : X + X -----------------------: X + X X + XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX +#! rnews 1389 +Path: alberta!mnetor!uunet!mcvax!enea!kuling!peterf +From: peterf@kuling.UUCP (Peter Fagerberg) +Newsgroups: comp.sys.mac +Subject: More memory for Mac+...? +Message-ID: <570@kuling.UUCP> +Date: 9 Dec 87 15:08:48 GMT +Organization: DoCS, Uppsala University, Sweden +Lines: 23 + + +Hello. I've been wondering how to get a little extra memory for my +Macintosh Plus (needed in these days of Hypercard and Multifinder). + +I was wondering if the normal brute-force method could be used; + Just solder 1M memory chips on top of the existing one (piggyback) + and attach CS (chip-select) and whatever else is needed from the + adressbus to select the appropiate chip. I haven't really checked + out the memorychips but maybe an inverter is needed for some signals. + + If I'm correctly informed there are 22 bit defining the adress on + a MC68000, making it possible to have 4M of memory. + +*If* this is possibly, would programs take advantage of it? + +Well, maybe this is one of the most stupid questions asked to USENET +since it all began and if so - please forgive my ignorance... + + Peter-- +============================================================================== +Peter Fagerberg UUCP: {seismo,enea,mcvax,decwrl,...}!kuling!peterf +Applied Computer Science ARPA: kuling!peterf@seismo.css.gov +Uppsala University Analog: +46 18-128286 or 8-102927 +#! rnews 1429 +Path: alberta!mnetor!uunet!mcvax!botter!klipper!biep +From: biep@cs.vu.nl (J. A. "Biep" Durieux) +Newsgroups: soc.culture.jewish +Subject: Re: Jews in soc.culture.jewish? +Message-ID: <958@klipper.cs.vu.nl> +Date: 10 Dec 87 09:07:28 GMT +References: <4765@spool.wisc.edu> <2086@ucbcad.berkeley.edu> <2264@encore.UUCP> <5779@cisunx.UUCP> <2872@sphinx.uchicago.edu> <5861@cisunx.UUCP> +Reply-To: biep@cs.vu.nl (J. A. "Biep" Durieux) +Organization: VU Informatica, Amsterdam +Lines: 23 + +In article <5861@cisunx.UUCP> dlhst@unix.cis.pittsburgh.edu.UUCP, + (David L. Heyman) writes: +>Don't kid yourself. the Constitution is one thing but reality is +>another. National Christmas tree, etc. + ^^^^^^^^^^^^^^ + +You are not trying to say that the US are German-mythological qua +religion, are you? :-) + +No, but seriously: what does that tree have to do with Christianity? +(Or, what does the mean US Christmas have to do with it at all - but +that's another story) +Is Santa Claus Christian? The Easter Bunny and its eggs? + +While I agree that the dates of these festivities originally come from +the church, the things which are generally celebrated have no origin in +Christian doctrine, and no one pretends so. + +Sorry if I offended anyone by this - I am not commenting on those who do +use those times for prayer and as memorial days. +-- + Biep. (biep@cs.vu.nl via mcvax) + To be the question or not to be the question, that is. +#! rnews 1323 +Path: alberta!mnetor!uunet!mcvax!ukc!eagle!icdoc!ivax!shb +From: shb@ivax.doc.ic.ac.uk (Simon Brock) +Newsgroups: comp.sys.mac +Subject: Re: uw/Multifinder? +Message-ID: <146@gould.doc.ic.ac.uk> +Date: 9 Dec 87 10:08:38 GMT +References: <174400085@uxc.cso.uiuc.edu> +Sender: news@doc.ic.ac.uk +Reply-To: shb@doc.ic.ac.uk (Simon Brock) +Organization: Dept. of Computing, Imperial College, London, UK. +Lines: 22 + +In article <174400085@uxc.cso.uiuc.edu> dorner@uxc.cso.uiuc.edu writes: +> +>I can't get uw to work under Multifinder. ... +>I have an SE, and am running the latest system software (obviously). +>I'm using uw version 4.1. +> +>Is anybody successfully using uw under Multifinder? +Yes. I'm using uw4.1 on an SE with System 4.1/Finder 6.0 and a beta version +of MF (1.0b6). (As an aside, we can't get System Tools 5.0 in the UK until +early next year, unless you know different to me !) + +UW runs but I do character losses at 9600 baud. I can't work out why, and +I'm not convinced its UW's fault. I wrote to John Bruner, the author, who +says other people were reporting the same problem. + + Simon. + +Simon H Brock, Dept. of Computing, Imperial College, London SW7 2AZ +Tel : 01 589 5111 x4993 +BitNet : shb@doc.ic.ac.uk (or shb%uk.ac.ic.doc@AC.UK) +UUCP : shb@icdoc.uucp (...siesmo!mcvax!ukc!icdoc!shb) +JANET : shb@uk.ac.ic.doc +#! rnews 1446 +Path: alberta!mnetor!uunet!mcvax!ukc!eagle!icdoc!cam-cl!am +From: am@cl.cam.ac.uk (Alan Mycroft) +Newsgroups: comp.lang.c +Subject: Re: closing stdout +Keywords: Yes it IS a buggy library +Message-ID: <1115@jenny.cl.cam.ac.uk> +Date: 9 Dec 87 10:38:55 GMT +References: <442@cresswell.quintus.UUCP> +Reply-To: am@cl.cam.ac.uk (Alan Mycroft) +Organization: U of Cambridge Comp Lab, UK +Lines: 19 + +In article <442@cresswell.quintus.UUCP> ok@quintus.UUCP (Richard A. O'Keefe) writes: +>There's an old joke with the punch-line "We've already established what +>you are, madam. Now we're just haggling over the price." +> result = getchar(); +> errno = 0; +> result = putc(result, stdin); +> printf("result = %d, errno = %d\n", result, errno); +>The bug is that depending on where you are in the buffer, putc() MIGHT +>notice the mistake, but it usually won't. +>... the bug is a pretty fundamental one in the UNIX stdio implementation, +Richard, The bug is not in the slightest bit fundamental and could be fixed +in less than 1 day once and for all. I have done it for a ANSI unix-like I/O +library: +Merely separate the _cnt field +of struct FILE into a _icnt and an _ocnt, change getc/putc to use _icnt/_ocnt. +Fix _filbuf/_flsbuf to use the right one, and to whinge when _icnt/_ocnt +goes -ve when you expect the other one to. +This for free also enables the library to police the "fflush/fseek between +change of direction for I/O" restriction and avoids chaos there. +#! rnews 1094 +Path: alberta!mnetor!uunet!mcvax!ukc!eagle!icdoc!cam-cl!lg +From: lg@cl.cam.ac.uk (Li Gong) +Newsgroups: soc.culture.china +Subject: Change of Policy After Beginnig Signing Contrct ? +Message-ID: <1114@jenny.cl.cam.ac.uk> +Date: 9 Dec 87 10:37:41 GMT +Organization: U of Cambridge Comp Lab, UK +Lines: 19 + + + Is there anybody out there who has info about whether the Chinese +government has changed the policy regarding students aboard and how +it is changed, because from this April, all students sent by the +government are asked to sign contracts between him/her and his/her +institution. + + What do these contracts mean ? Does this imply that those who came +out before this April (thus did not sign) then have a somewhat different +status (for example, can not be asked to go back to carry out a certain +contract) ? + + E-mail to me and I'll summurize OR post to the newsgroup. I believe +there are other people who are also interested in this issue. + + Martin +----------------------------------------------------------------------- +lg@uk.ac.cam.cl +--------------- +#! rnews 1315 +Path: alberta!mnetor!uunet!mcvax!ukc!stc!root44!cdwf +From: cdwf@root.co.uk (Clive D.W. Feather) +Newsgroups: rec.arts.sf-lovers +Subject: Eric Frank Russell - was Re: Misc questionings +Message-ID: <492@root44.co.uk> +Date: 9 Dec 87 15:03:01 GMT +References: <362@n8emr.UUCP> <2481@pbhyf.UUCP> +Reply-To: cdwf@root44.UUCP (Clive D.W. Feather) +Organization: Root Computers Ltd, London, England +Lines: 23 + +In article <2481@pbhyf.UUCP> djl@pbhyf.UUCP (Dave Lampe) writes: +>In article <362@n8emr.UUCP> lwv@n8emr.UUCP (Larry W. Virden) writes: +>> +>>5. Finally, and perhaps most important. I am looking for author and +>>anthology names for a short story (perhaps longer than thtat?) called I +>>believe "MYOB". +>>The title stands for "Mind Your Own Business". +> +>The story is in a book called "The Great Explosion" by Eric Frank +>Russell in 1962. It is a collection of 3 or 4 stories telling +>of an attempt by Earth to recontact colonies that had been lost +>for a long time and that had evolved into unusual societies. + +I have come across "The Great Explosion", but I also have this part of it +in a collection whose name I have forgotten, under the title "And then there +were none.". Great story. THE BEST AUTHOR EVER. + +[Kill the line counter] +[Kill Mel] +[Keep Adric Dead] +[Kill the line counter] +[Kill Mel] +[Keep Adric Dead] +#! rnews 849 +Path: alberta!mnetor!uunet!mcvax!ukc!stc!root44!cdwf +From: cdwf@root.co.uk (Clive D.W. Feather) +Newsgroups: sci.misc +Subject: Re: Color +Message-ID: <493@root44.co.uk> +Date: 9 Dec 87 15:46:24 GMT +References: <162300002@uiucdcsb> <162300004@uiucdcsb> +Reply-To: cdwf@root44.UUCP (Clive D.W. Feather) +Organization: Root Computers Ltd, London, England +Lines: 13 + + +Carl Kadie +Inductive Learning Group +University of Illinois at Urbana-Champaign +writes: +>ii. There is "no such color" as purple! Mixing red and blue ink +> causes your eye to react in a way which is not reproducible +> by any single wavelength of light. + +The eye can see colours (for example, in afterimages) that cannot be +reproduced by any combination of wavelengths of light ! +There was an article in Scientific American c.1970 entitled "Phosphenes" +that went into this. +#! rnews 795 +Path: alberta!mnetor!uunet!mcvax!ukc!stc!datlog!dlhpedg!cl +From: cl@dlhpedg.co.uk (Charles Lambert) +Newsgroups: rec.games.empire,comp.sources.bugs +Subject: Re: conquest newsletter #3 +Message-ID: <330@dlhpedg.co.uk> +Date: 9 Dec 87 14:27:16 GMT +References: <4886@mhuxd.UUCP> <6899@apple.UUCP> +Sender: news@dlhpedg.co.uk +Reply-To: cl@.co.uk (Charles Lambert) +Organization: FSG@Data Logic Ltd, Queens House, Greenhill Way, Harrow, London. +Lines: 8 +Xref: alberta rec.games.empire:292 comp.sources.bugs:563 + +>In article <4886@mhuxd.UUCP>, smile@mhuxd.UUCP (Edward Barlow) writes: +>> 3) Still have not thought of a new name for the game. Best so far is +>> (need to check spelling). Comments? + +I've missed something here; what was wrong with "conquest"? + +--------------- +Charlie Lambert +#! rnews 819 +Path: alberta!mnetor!uunet!mcvax!weijers +From: weijers@cwi.nl (Eric Weijers) +Newsgroups: comp.lang.c++ +Subject: another error in vector.h 1.3 +Message-ID: <143@piring.cwi.nl> +Date: 10 Dec 87 13:27:06 GMT +Organization: CWI, Amsterdam +Lines: 22 + +In "vector.h 1.3" the following definition of the X(X&) constructor +is given: + +vector(type).vector(type)(vector(type)& a) +{ + register i = a.sz; + sz = a.sz; /* ADD THIS LINE */ + v = new type[i]; + register type* vv = &v[i]; + register type* av = &a.v[i]; + while (i--) *--vv = *--av; +} + +You should add the indicated line in order to set the size of +the new vector. If that is not done you get "vector index out of +range" errors. + +I found two other errors in this header file, I posted +earlier. If you are interested in them just send a reply (r). + +Eric Weijers. +weijers@cwi.nl +#! rnews 830 +Path: alberta!mnetor!uunet!mcvax!botter!klipper!biep +From: biep@cs.vu.nl (J. A. "Biep" Durieux) +Newsgroups: soc.culture.jewish +Subject: Anything positive about Jewish genes? (Was: Jewish genetic diseases) +Message-ID: <959@klipper.cs.vu.nl> +Date: 10 Dec 87 09:50:15 GMT +References: <4362@ig.ig.com> <4374@ig.ig.com> +Reply-To: biep@cs.vu.nl (J. A. "Biep" Durieux) +Organization: VU Informatica, Amsterdam +Lines: 12 + +I suppose the exclusive intermarriage among Jews must also have +spared them for many genetic diseases found among "the rest of us". +Does anyone have any data on that? + +~~~ +I understand nobody is interested in discussing the Dead Sea scrolls? + +And nobody knows what the "Jewish region" in the far SE of Siberia is? +~~~ +-- + Biep. (biep@cs.vu.nl via mcvax) + To be the question or not to be the question, that is. +#! rnews 960 +Path: alberta!mnetor!uunet!mcvax!unido!ecrcvax!johng +From: johng@ecrcvax.UUCP (John Gregor) +Newsgroups: rec.arts.sf-lovers +Subject: Re: Old SF Shows +Summary: Yet another show I can't remember the name of... +Message-ID: <463@ecrcvax.UUCP> +Date: 9 Dec 87 13:57:35 GMT +References: <04.Dec.87.11:29:45.GMT.ZZASSGL@UK.AC.UMRCC.CMS> <18784@linus.UUCP> <1046@bc-cis.UUCP> <19026@linus.UUCP> +Reply-To: johng@ecrcvax.UUCP (John Gregor) +Organization: ECRC, Munich 81, West Germany +Lines: 10 + +There was a show on sometime between the late 70's and early 80's (1 season). +And I can't remember the name. It was actually two (or more) shows in one +with each sub-show taking a fraction of the time slot. One part was a +modern day dracula. Another dealt with a society living underground. They +couldn't come up to the surface without special filters due to dust/pollution +or some such. Ring any bells? It was NBC, I think. + + John + + johng%ecrcvax.UUCP@germany.CSNET +#! rnews 756 +Path: alberta!mnetor!uunet!mcvax!botter!ark!maart +From: maart@cs.vu.nl (Maarten Litmaath) +Newsgroups: comp.bugs.4bsd +Subject: Re: 4.3BSD: using control-m in .exrc file +Summary: More ^V's are needed (won't the editor get enough of it ? :-) +Keywords: 4.3bsd .exrc control-m ^V +Message-ID: <1161@ark.cs.vu.nl> +Date: 10 Dec 87 18:44:07 GMT +References: <133@telesoft.UUCP> +Reply-To: maart@cs.vu.nl (Maarten Litmaath) +Organization: VU Informatica, Amsterdam +Lines: 8 + +Try preceding each ^M by *another* ^V (which in turn is escaped by ^V) ! +Type: + map , ^V^V^V^M^V^V^V^M^V^V^V^M + +BTW, death to emacs ! +-- +Time flies like an arrow, fruit flies |Maarten Litmaath @ Free U Amsterdam: +like an orange. (seen elsewhere) |maart@cs.vu.nl, mcvax!botter!ark!maart +#! rnews 1079 +Path: alberta!mnetor!uunet!mcvax!inria!shapiro +From: shapiro@inria.UUCP (Marc Shapiro) +Newsgroups: comp.lang.c++ +Subject: Re: Is there a "real" C++ compiler available? +Summary: There is a native C++, with debugger support +Message-ID: <589@inria.UUCP> +Date: 10 Dec 87 17:55:27 GMT +References: <2097@ucbcad.berkeley.edu> +Organization: INRIA, Rocquencourt. France +Lines: 14 + +In article <2097@ucbcad.berkeley.edu>, faustus@ic.Berkeley.EDU (Wayne A. Christopher) writes: +> [...]. Is there a C++ +> compiler available now that will compile directly into asm +> code, instead of into C? Alternatively, is there a good way +> to use dbx with C++ programs (i.e, using the c++ source instead +> of the c files)? + +The answer to both questions is yes. The Free Software Foundation (you +know, the GNU Emacs people) will distribute (soon?) a modified version of +their C compiler which does C++. Their debugger GDB (a dbx-lookalike) knows +how to handle it. + +I haven't used either of these so I have no opinions to whether they are +in any way adequate. Just passing useful information along. +#! rnews 1269 +Path: alberta!mnetor!uunet!mcvax!enea!ttds!draken!zap +From: zap@draken.nada.kth.se (Svante Lindahl) +Newsgroups: comp.os.misc,comp.unix.wizards +Subject: Re: Command interfaces +Message-ID: <239@draken.nada.kth.se> +Date: 10 Dec 87 04:54:11 GMT +References: <1257@boulder.Colorado.EDU> <6840002@hpcllmv.HP.COM> <9555@mimsy.UUCP> <798@rocky.STANFORD.EDU> <432@cresswell.quintus.UUCP> <3161@psuvax1.psu.edu> <5565@oberon.USC.EDU> +Reply-To: zap@nada.kth.se (Svante Lindahl) +Organization: The Royal Inst. of Techn., Stockholm +Lines: 21 +Xref: alberta comp.os.misc:340 comp.unix.wizards:5748 + +In article <5565@oberon.USC.EDU> blarson@skat.usc.edu (Bob Larson) writes: +#For example, how would you do the equivelent of this in unix: +# +#cmpf *>old>@@.(c,h) == -report ==.+cmpf -file + +I can do it using either /bin/sh or csh, but it does require more +typing than in Primos. The test for existence of the file is not +necessary so these examples could be simplified at the expense of +risking a few error messages to the terminal. + +C-shell: +% foreach i (`cd old; ls *.[ch]`) +> if (-r $i) diff -c old $i > $i.cmpf +> end + +Bourne-shell: +$ for i in `cd old; ls *.[ch]` ; do +> if [ -r $i ] ; then diff -c old $i > $i.cmpf ; fi +> done + +Svante Lindahl zap@nada.kth.se uunet!nada.kth.se!zap +#! rnews 2030 +Path: alberta!mnetor!uunet!mcvax!enea!ttds!draken!sics!lhe +From: lhe@sics.se (Lars-Henrik Eriksson) +Newsgroups: rec.arts.sf-lovers +Subject: Re: ST:TNG posters, GET OUT! +Keywords: Why +Message-ID: <1640@sics.se> +Date: 10 Dec 87 11:50:40 GMT +References: <5226@zen.berkeley.edu> <2011@charon.unm.edu> +Reply-To: lhe@sics.se (Lars-Henrik Eriksson) +Organization: Swedish Institute of Computer Science, Kista +Lines: 32 + +In article <2011@charon.unm.edu> cs3631cg@hydra.UUCP (Mark Giaquinto) writes: +>Two points here, interesting is a *very* relative term, what is +>interesting to you may not be to me and visa versa. Secondly I +>agree, that if you have a ST posting put it in the header, for people +>who don't want to read this stuff. +> +>>If there was no group for star trek fans to converse in without pestering +>>the rest of the sf world, I would just have to sit here and suffer, but +>>that's not the case. Rec.arts.startrek is alive and well. There is no +>>reason beyond sheer orneryness to post to sf-lovers as well. Arguments that +>>star trek is sci-fi as well are pointless. The simple fact is that there is +>>newsgroup for all of you to communicate in, and if the rest of us wanted to +>>listen, then we would. +> +>Well startrek is sf and I don't see how that arguement is pointless. + +I have only the faintest interest in the ST stuff and I would prefer it +to be posted elsewhere, although I am not particularly bothered either. + +I think the interesting question is: WHY DO WE HAVE DIFFERENT NEWSGROUPS?? + +I always thought it was to organize postings by subject and because different +people are interested in different things. + +If you argue that ST postings could as well be made to rec.arts.sf-lovers +rather than to the special ST newsgroup, you could just as well argue +that we only need one newsgroup on the entire net: general.general.general. + +Lars-Henrik Eriksson Internet: lhe@sics.se +Swedish Institute of Computer Science Phone (intn'l): +46 8 750 79 70 +Box 1263 Telefon (nat'l): 08 - 750 79 70 +S-164 28 KISTA +#! rnews 1007 +Path: alberta!mnetor!uunet!mcvax!enea!tut!mk59200 +From: mk59200@tut.fi (Kolkka Markku Olavi) +Newsgroups: comp.sources.bugs +Subject: Re: PC Nethack 2.2 bugs + help wanted linking +Summary: Inventory display problems +Message-ID: <522@fuksi.tut.fi> +Date: 10 Dec 87 13:32:40 GMT +References: <492@silver.bacs.indiana.edu> <5253@zen.berkeley.edu> +Reply-To: mk59200@fuksi.UUCP (Kolkka Markku Olavi) +Organization: Tampere University of Technology, Finland +Lines: 13 + +I have successfully compiled and linked Nethack using MSC 4.0 +and it looks great, exept in a few points. The inventory +display is spread all over the screen if there aren't enough +items to force a full-screen display. It seems that after +printing each line the cursor is moved one step down, but +it doesn't move left to the right place. + +Also, when I teleport away from an unlit room, some quote characters +are left behind around the place I was in. + +Markku Kolkka at Tampere University of Technology, Finland +mk59200@tut.fi +...mcvax!tut!mk59200 +#! rnews 811 +Path: alberta!mnetor!uunet!mcvax!enea!tut!tolsun!jto +From: jto@tolsun.oulu.fi (Jarkko Oikarinen) +Newsgroups: comp.sys.amiga,rec.games.misc +Subject: 'Real' controllers for Flight Simulator II +Keywords: Controllers, Flight Simulator +Message-ID: <247@tolsun.oulu.fi> +Date: 10 Dec 87 16:47:22 GMT +Organization: University of Oulu, Finland +Lines: 15 +Xref: alberta comp.sys.amiga:11680 rec.games.misc:1151 + + + I am interested in finding any information about 'real' controllers +for Amiga's Flight Simulator II program. ie. similar controllers +that are used in real airplanes. + +Please mail your responses because I don't read this group regularly. + +-- +======================================== +Jarkko Oikarinen mcvax!tut!oulu!jarkko + jarkko@tolsun.oulu.fi +======================================== +#! rnews 913 +Path: alberta!mnetor!uunet!mcvax!inria!imag!pierre +From: pierre@imag.UUCP (Pierre LAFORGUE) +Newsgroups: comp.protocols.appletalk +Subject: NCSA TELNET bug with foreign MacSE or MacII keyboards +Message-ID: <2331@imag.UUCP> +Date: 10 Dec 87 08:08:19 GMT +Reply-To: pierre@imag.UUCP (Pierre LAFORGUE) +Organization: IMAG, University of Grenoble, France +Lines: 11 + +NCSA Telnet is really a must, but ... +on a Mac SE and a Mac II, NCSA Telnet 2.0 forces an american keyboard, in a +permanent manner (it remains after exiting telnet, until the next Macintosh +reboot). It is very painful when you use, for instance, a french keyboard: +not only you have to remember to type Q for A, and so on, but you cannot +type for example a Control-Z under telnet. +[On a Macintosh +, one do not loss its keyboard] + +Is this bug fixed in the last version ? +-- +Pierre Laforgue pierre@imag.imag.fr {uunet.uu.net|mcvax}!imag!pierre +#! rnews 490 +Path: alberta!mnetor!uunet!mcvax!diku!sergej +From: sergej@diku.UUCP (S|ren O. Jensen) +Newsgroups: sci.math.stat +Subject: The SAS package +Message-ID: <3570@diku.UUCP> +Date: 10 Dec 87 14:03:31 GMT +Organization: DIKU, U of Copenhagen, DK +Lines: 7 + + +Is the SAS package available for UNIX-systems? We are currently using the +package on a old IBM machine but would like to change this machine to +something newer - preferably a UNIX-machine. +-- +---- +S|ren Oskar Jensen ({sergej,postmaster}@diku) +#! rnews 2766 +Path: alberta!mnetor!uunet!mcvax!diku!iesd!jacob +From: jacob@iesd.uucp (Jacob stergaard B{kke) +Newsgroups: comp.arch +Subject: job search, Comp. eng. +Summary: I'm looking for a job +Keywords: Job, Computer. eng., Computer. sci., M.S. +Message-ID: <172@iesd.uucp> +Date: 10 Dec 87 12:00:17 GMT +Reply-To: jaaob@iesd.UUCP (Jacob \stergaard B{kke) +Organization: Dept. of Comp. Sci., Aalborg University, Denmark (student) +Lines: 68 + +I'm looking for a job in Computer Engineering to begin around July +1988. I'm getting my Master of Science in Computer Engineering June +1988 and at present holding a degree equal to BS in Electronic +Engineering. My BS studies have included: + + Computer hardware (hands-on knowledge with mc68k), + Analog electronic + Control engineering (analog and digital control) + +My MS studies have included: + + Software development (man-machine interface, what people want + from programs) + Compiler construction (an expertsystem shell) + Program environment (for CCS programming) + Distributed operating systems (in UNIX) + Compiler mapping object-oriented language on parallel computers + +Furthermore I do have experience in conventional programming (PASCAL, +C, postscript, UNIX (awk, shell-scripts(C-shell) and yacc/lex) (and Basic)), +functional programming (LISP and ML) and logical programming (Prolog) +and knowledge about object-oriented programming. And I have also attended +courses in VLSI design, databases, etc. I have been working with CDC under +NOS/Telex, VAX 11/750 under Ultrix, SUN 3 under Sun OS 4.3 (UNIX), MacIntosh +(LISA) under Finder and IBM S36 under IBM property operating system. + +My spoken English is excellent and my written English is satisfactory, +good knowledge of the Scandinavian languages (Danish (of course), +Swedish and Norwegian), some speaking and reading knowledge of German +and limited knowledge of French and Spanish (and Latin). + +I have 5 years experience in group project work in engineering and +computer scinence areas, broad social interest, good health. + +My interest include computer hardware and software, operating system +design, expertsystems, distributed, concurrency and teaching. + +I'm open on location (outside Denmark) but I have relatives or other +reasons to be especially intereted in: + + Canada (British Colombia or Toronto) + USA (New England or Pacific Coast) + Pacific (New Zealand or Oceania) + Thailand + Scotland (Highlands) + +I'll look forward to any reponds. + + Yours sincerely + + Jacob Baekke, Denmark + +For further information: + +Reply to: jacob@iesd.uucp, {...}!mcvax!diku!iesd!jacob or + +at Univ: Jacob Baekke + S9D (in spring S10) + Strandvejen 19 + AUC + DK--9000 Aalborg + Denmark + +private: Jacob Baekke + Davids Alle 48 + DK--9000 Aalborg + Denmark + Tel. 45-(0)8102673 +#! rnews 867 +Path: alberta!mnetor!uunet!mcvax!diku!dde!jk +From: jk@dde.uucp (Jens Kjerte) +Newsgroups: comp.sources.wanted +Subject: Re: Wanted: Microemacs part 8 +Message-ID: <281@Aragorn.dde.uucp> +Date: 10 Dec 87 09:27:24 GMT +References: <166@iesd.uucp> +Reply-To: jk@dde.uucp (Jens Kjerte) +Organization: Dansk Data Elektronik A/S, Herlev, Denmark +Lines: 15 + +In article <166@iesd.uucp> torbennr@neumann.UUCP (Torben N. Rasmussen) writes: +> +>Could someone please send me part 8 of the sources for Microemacs. +> + +Me too! + +It seems as if part8 never reached Denmark. + +-- + ++---------------------------------------------------------------------------+ +| Jens Kjerte @ Dansk Data Elektronik A/S, Systems Software Department | +| E-mail: ..!uunet!mcvax!diku!dde!jk or jk@dde.uucp | ++---------------------------------------------------------------------------+ +#! rnews 512 +Path: alberta!mnetor!uunet!mcvax!cernvax!ethz!solaris!wyle +From: wyle@solaris.ifi.ethz.ch@relay.cs.net (Mitchell Wyle) +Newsgroups: comp.lang.modula2 +Subject: modula-2 pretty-printer +Keywords: pretty-printer +Message-ID: <195@solaris.ifi.ethz.ch@relay.cs.net> +Date: 9 Dec 87 21:56:57 GMT +Organization: SOT sun cluster, ETH Zuerich +Lines: 7 + +Did anyone ever get the m2pp program to work on Sun Modula-2? + +Does anyone have a different Modula-2 pretty-printer (perhaps better)? + +Thanks, + +Mitch Wyle (wyle@ethz.uucp) +#! rnews 1762 +Path: alberta!mnetor!uunet!mcvax!cernvax!ethz!zu +From: zu@ethz.UUCP (Urs Zurbuchen) +Newsgroups: comp.emacs +Subject: Re: Has uemacs 3.9 solved the file save bug? +Message-ID: <265@bernina.UUCP> +Date: 10 Dec 87 07:21:02 GMT +References: <3056@pegasus.UUCP> +Reply-To: zu@bernina.UUCP (Urs Zurbuchen) +Organization: ETH Zuerich, CS Department, Switzerland +Lines: 30 + +In article <3056@pegasus.UUCP> avi@pegasus.UUCP (XMPE40000-Avi E. Gross;LZ 3C-314;6241) writes: +> +>I haven't compiled the new micro emacs since I have a MSC compiler, which is +>not fully supported. + +This is simply NOT TRUE. I am also working with MSC (version 4.0) and had only +one minor problem when I compiled MicroEmacs 3.9e (the latest version which +was posted on Usenet). This problem relates to the Subshell spawning. But if +you know just a little bit of C, there is no problem to fix it (add a routine +specific to MSC). Some time ago, there was even a posting in comp.sources.bugs +describing all the necessary steps to do that. + +>I have been having a very annoying problem with the +>older version, and am wondering if it has been fixed, or if someone has a +>work around. I am used to saving my files regularly with ^X^S, and then +>sometimes quiting with ^X^C. Unfortunately, uemacs will quit before +>completing the writing of the file, leaving me with only a small piece of +>the file. + +I am sure you enable breaking with ^C (either in config.sys or in autoexec.bat) +Turn this off, and all your problems have gone :-) +I know this is not the solution to this problem we all want to have. Perhaps +you can do it with signal(). If not you have to included a function of your own +which intercepts the break vector of MS-DOS. + + + Have a nice day, + ...urs + +UUCP: ...seismo!mcvax!cernvax!ethz!zu +#! rnews 2164 +Path: alberta!mnetor!uunet!mcvax!cernvax!ethz!ceb +From: ceb@ethz.UUCP (Charles Buckley) +Newsgroups: comp.lang.lisp +Subject: Re: lisp environments summary -- program storage methods +Message-ID: <266@bernina.UUCP> +Date: 10 Dec 87 23:08:38 GMT +References: <613@umbc3.UMD.EDU> <325@siemens.UUCP> <323@spar.SPAR.SLB.COM> <329@siemens.UUCP> <13253@think.UUCP> +Organization: ETH Zuerich, Switzerland +Lines: 30 +In-reply-to: barmar@think.COM's message of 9 Dec 87 03:18:01 GMT + +Posting-Front-End: GNU Emacs 18.41.2 of Mon Sep 14 1987 on bernina (berkeley-unix) + + +In article <329@siemens.UUCP> steve@siemens.UUCP (Steve Clark) writes: +> I maintain that the non-Interlisp systems are wrong, however. It +>is clearly more advanced to treat a file as a database of definitions of +>functions, data, structures, etc. than to treat it as a string of characters +>that might have been typed at the keyboard. However, since the rest of the +>world hasn't caught up yet, there are bound to be incompatibilities. + +(Character) file storage is simply more flexible. The form in which +information is stored must be the most flexible possible, or you lose +information. The D-crate's pitching of conditionals is simply the +manifestation of this. + +Proponents of restrictive protocols for information storage really ask +"the world" to change to fit the protocol model. In science, models +change to fit the data, not the other way round (unless you cheat). +To me, browbeating eventual non-conformists into "catching up" by +labeling the a model as "advanced" is just a form of negative +motivation. All the lousy places I have ever worked ran on negative +motivation, none of the good ones. If your model *is* really worth +using, and you can communicate its value, you will not need such +tactics. + +Interactively defined functions? Haven't typed one in *years* - +that's what scratch buffers are for (in case I want to change a +*character* or two, or later save it.). + +Any mouse-based gadgets you can point to in Interlisp can be recreated +for a text editor working on correctly parsed Lisp code. May take +execution time, but if this is prohibitive, your function is probably +too large. +#! rnews 2319 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!jha +From: jha@its63b.ed.ac.uk (J Andrews) +Newsgroups: rec.games.frp +Subject: Fantasy Philosophy +Keywords: wackafoo +Message-ID: <824@its63b.ed.ac.uk> +Date: 10 Dec 87 14:30:15 GMT +Reply-To: jha@lfcs.ed.ac.uk (J Andrews) +Organization: Univ. of Edinburgh Dept. of Computer Science +Lines: 38 +God: Kate Bush + +Least-favourite-subject: domain theory + + + + Those interested in the issues surrounding the mechanics and +philosophy of fantasy worlds should read Tolkien's (non-fiction) +essay "On Fairy-Stories". It appears in the collections _Tree and +Leaf_ and _The Tolkien Reader_. + + One of the main ideas behind it is that the fantasy author or +story-teller is a "sub-creator", who tries to create a "secondary +belief" (rather than exactly a "willing suspension of disbelief") +in the reader. In the fantasy that works, the reader should be +able to enter the world every time she picks up the book, and not +be aware of the world as being constructed by the author. This +involves not only internal consistency, but a lack of gimmickry. + + For instance, in _Lord of the Rings_ I was never aware of +anything being in the world gratuitously. (Others may differ! :-)) +In _The Sword of Sha-Na-Na_ (sic)(sick?), on the other hand, I was +very aware of the Elfstones as being just a gimmick to get the +characters out of tight spots. Sure it was internally consistent +(the Elfstones only had any effect in times of direst need for +their holders), but the hand of the author was clearly visible. + + Similarly, applying it to FRPG's, the magic system in AD&D is +certainly internally consistent (to the extent that it is described), +but just doesn't "work" for me. Having MU's able to remember several +copies of a spell, but forgetting it when the last copy is cast, is +obviously a gimmick to limit the number of spells an MU can use. + + So I guess the moral of all this for FRPG or module designers +is that it's best to start out with a few basic assumptions and build +up your world from them by fairly believable steps, and if you can't +avoid ending up with something really hairy, then change one of your +assumptions rather than put in quick kludges. (Gee, sounds like +software engineering! :=)) + +--Jamie. + jha@uk.ac.ed.lfcs +"Switch off the mind and let the heart decide" +#! rnews 1818 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!db +From: db@its63b.ed.ac.uk (D Berry) +Newsgroups: comp.windows.x +Subject: Questions about implementing the X toolkit. +Message-ID: <825@its63b.ed.ac.uk> +Date: 10 Dec 87 17:28:06 GMT +Reply-To: db@lfcs.ed.ac.uk (Dave Berry) +Organization: LFCS, University of Edinburgh +Lines: 25 + +1) Does anyone, preferably in the UK or Europe, have a copy of the new +X toolkit interface definition I can get by ftp? + +2) I'm considering implementing the X toolkit in Standard ML. Are there any +constraints on what I should include or exclude? The documentation mentions +implementation in different languages, but doesn't say much about what this +means. Is the idea to provide the same functions, with the same names and +functionality, in each language? What about languages that have automatic +storage management or automatic creation of objects, etc? How far can I +deviate from the documentation & still use the name "X Toolkit"? + +3) Is the toolkit definition limited to the intrinsics, or are toolkits +expected to provide a standard class hierarchy? + +4) Is there any relation between the InterViews toolkit, the Xr, Sx & +DEC toolkits provided with X version 10R4, and the current X toolkit? + +5) If I go ahead, my first implementation will be a prototype, on top of X +version 10R4. This is because someone else is working on porting X version 11 +to Standard ML, and I want a simple windowing system I can use fairly quickly. +I hope the prototype will make implementing a full version reasonably +straightforward. I will probably ignore the resource manager, since I'll get +that for free when the full Xlib is implemented. I'll also ignore colour for +the time being, and only implement devices (widgets) I'm immediately interested +in. Is there anything else I can obviously ignore? +#! rnews 1113 +Path: alberta!mnetor!uunet!mcvax!ukc!eagle!csw +From: csw@eagle.ukc.ac.uk (C.S.Welch) +Newsgroups: rec.arts.sf-lovers +Subject: Re: Word processors are: [was Re: Pournelle's Problems] +Message-ID: <4065@eagle.ukc.ac.uk> +Date: 10 Dec 87 18:42:09 GMT +References: <1915@haddock.ISC.COM> +Reply-To: csw@ukc.ac.uk (C.S.Welch) +Organization: Computing Lab, University of Kent at Canterbury, UK. +Lines: 20 +Summary: + +Expires: + +Sender: + +Followup-To: + + + +Some (possibly) timely information from a course entitled "The Art of +Communication for Engineers" that I'm on this week. + +From one of the handouts :- + +"Word processors: research has shown that when writers use pen and paper + alone, their thoughts and information tend to have better planning and + organisation. When using word processors alone, writers tend to plan + on a more surface level, focussing on such aspects as word choice, sentence + structure, and spelling" + +It goes on to recommend starting with pen and paper and graduating to WP's +after the first draft has been written. + +I trust that this may have been of some interest. + +Chris Welch +Cranfield Institute +U.K. +#! rnews 1286 +Path: alberta!mnetor!uunet!mcvax!ukc!eagle!icdoc!qmc-cs!pd +From: pd@cs.qmc.ac.uk (Paul Davison) +Newsgroups: rec.music.misc +Subject: Re: Another Day : by Peter Gabriel and Kate Bush +Message-ID: <352@sequent.cs.qmc.ac.uk> +Date: 10 Dec 87 12:58:25 GMT +References: <1987Dec8.154517.11828@gpu.utcs.toronto.edu> +Reply-To: pd@qmc.ac.uk (Paul Davison) +Organization: Computer Science Dept, Queen Mary College, University of London, UK. +Lines: 22 + + +I've heard of this as well, but I have never found it. It's a pity +because I would really like to hear it, so if anyone has got it please +let me know as well!! + +As an aside, Roy has a new album out early next year, probably January. + +Paul. + +PS Your internal newsgroup "tor.general" shouldn't have been on the +newsgroups line really, because nobody else has heard of it! +-- +-- +Paul Davison + +UUCP: pd@qmc-cs.uucp or ...seismo!mcvax!ukc!qmc-cs!pd +Internet: pd@cs.qmc.ac.uk Post: Dept of Computer Science +JANET: pd@uk.ac.qmc.cs Queen Mary College +Easylink: 19019285 University of London +Telex: 893750 QMCUOL G Mile End Road +Fax: +44 1 981 7517 London E1 4NS +Voice: +44 1 980 4811 x3950 England +#! rnews 786 +Path: alberta!mnetor!uunet!mcvax!ukc!dcl-cs!craig +From: craig@comp.lancs.ac.uk (Craig) +Newsgroups: comp.sys.mac +Subject: MAC II Debuggers +Keywords: Development, MacII Debuggers +Message-ID: <457@dcl-csvax.comp.lancs.ac.uk> +Date: 9 Dec 87 13:36:01 GMT +References: <687@howtek.UUCP> <3456@husc6.harvard.edu> +Reply-To: craig@comp.lancs.ac.uk (Craig) +Organization: Department of Computing at Lancaster University, UK. +Lines: 11 + +Having found out that Macsbug 5.5 works well with the MAC II, +how do I get a copy ? + + +Craig. + +-- +UUCP: ...!seismo!mcvax!ukc!dcl-cs!craig| Post: University of Lancaster, +DARPA: craig%lancs.comp@ucl-cs | Department of Computing, +JANET: craig@uk.ac.lancs.comp | Bailrigg, Lancaster, UK. +Phone: +44 524 65201 Ext. 4476 | LA1 4YR +#! rnews 1070 +Path: alberta!mnetor!uunet!mcvax!ukc!dcl-cs!strath-cs!jml +From: jml@cs.strath.ac.uk (Joseph McLean) +Newsgroups: sci.math +Subject: concatenation making primes +Message-ID: <756@stracs.cs.strath.ac.uk> +Date: 9 Dec 87 12:47:19 GMT +Reply-To: jml@cs.strath.ac.uk (Joseph McLean) +Organization: Comp. Sci. Dept., Strathclyde Univ., Scotland. +Lines: 14 + + +tege@nada.kth.se replied by e-mail to my original posting which asked +if it is always possible to append digits to a positive number in order +to make a prime. Unfortunately, his address is one of those I can't +reach, and so I thought I'd kill two birds with one stone and post +another article. + His argument is very simple, using the Prime Number Theorem to give +an approximation to the number of primes between x.10^n and +x.10^n+10^n-1 (which is the same problem I asked but translated to +mathematics) which shows that as n -> inf, this number of primes also +goes to infinity. A very simple argument that proves you can always +append digits to make any number into a prime. Great stuff. + + jml, the mad mathematician. +#! rnews 1275 +Path: alberta!mnetor!uunet!mcvax!ukc!stc!root44!miduet!misoft!tait +From: tait@gec-mi-at.co.uk (Philip Tait) +Newsgroups: comp.sys.ibm.pc,comp.sources.wanted +Subject: Re: Wanted: PC Checkbook Software +Summary: Continental Software's Home Accountant Plus +Keywords: Checkbook +Message-ID: <800@gec-mi-at.co.uk> +Date: 9 Dec 87 17:34:03 GMT +References: <985@mhuxh.UUCP> +Sender: news@gec-mi-at.co.uk +Reply-To: tait@gec-mi-at.co.uk (Philip Tait) +Organization: Marconi Instruments Ltd., St. Albans, UK +Lines: 15 +Xref: alberta comp.sys.ibm.pc:9577 comp.sources.wanted:2719 + +In article <985@mhuxh.UUCP> vxb@mhuxh.UUCP (Vern Bradner) writes: +> +>Can anyone suggest a PC checkbook program? + +I use Home Accountant Plus by Continental Software. The (legit.) version I use +was originally bundled with the Columbia MPC, so it had to be 'unprotected' +and altered to remove some hardware dependencies. (Incidentally, this made +it possible to compile it with QuickBasic - essential if you're impatient +like me!) + +I've found it reasonably secure and well-featured. + +| Philip J. Tait, Marconi Instruments Ltd. | St. Albans, Herts. AL4 0JN, U.K. | +| UUCP: ...mcvax!ukc!hrc63!miduet!tait | NRS : tait@gec-mi-at.co.uk | +| Voice: +44 727 36421 x4549 Telex: 297221 | Fax: +44 727 39447 | +#! rnews 1059 +Path: alberta!mnetor!uunet!mcvax!ukc!stc!idec!kbsc!yorick +From: yorick@kbsc.UUCP (Yorick Phoenix) +Newsgroups: comp.os.cpm,comp.sources.wanted +Subject: Kermit for MP/M +Message-ID: <888@kbsc.UUCP> +Date: 7 Dec 87 17:23:21 GMT +Organization: The Knowledge-Based Systems Centre, London, UK +Lines: 16 +Xref: alberta comp.os.cpm:1030 comp.sources.wanted:2720 + +I have a friend who is trying to transfer some files off of an Micromation +MP/M system. + +He has so far moved the standard "Generic" CP/M Kermit (slowly) to the MP/M +machine but it doesn't seem to work correctly. + +Has anybody ever managed to get Kermit to work under M/PM? Is there a simple +set of differences between C/PM kermit and M/PM Kermit. We have the full +source code for C/PM Kermit. + + Yorick Phoenix +-- ++------------------------------------------+ The Knowledge-Based Systems Center +| yorick@kbsc.UUCP | 58 Northside, Clapham Common +| ..mcvax!ukc!{idec,hrc63}!kbsc!yorick | LONDON SW4 9RZ England ++------------------------------------------+ Voice: +44 1 350 1622 +#! rnews 1946 +Path: alberta!mnetor!uunet!mcvax!ukc!stc!root44!gwc +From: gwc@root.co.uk (Geoff Clare) +Newsgroups: comp.unix.questions +Subject: Re: rmail under HP-UX (was Re: Using RMAIL under HPUX) +Summary: RISC architecture +Keywords: RISC, HP-UX +Message-ID: <495@root44.co.uk> +Date: 10 Dec 87 13:58:20 GMT +References: <8711251805.AA02481@mitre-bedford.ARPA> <3720010@hpsemc.UUCP> <3631@xanth.cs.odu.edu> +Reply-To: gwc@root44.UUCP (Geoff Clare) +Organization: Root Computers Ltd, London, England +Lines: 31 + +>In article <3720010@hpsemc.UUCP>, bd@hpsemc.UUCP (bob desinger) writes: +>> Here's how it is on our HP-UX system, a model 840: + +>> drwxrwxr-x 2 bin mail 1024 Nov 25 18:45 /usr/mail +>> -rwxr-sr-x 2 root mail 137216 Oct 2 00:00 /bin/rmail + +>Wow! Why is rmail so BIG? What does HP-UX rmail do that SMAIL 2.5 +>doesn't? Contrast the size of this rmail with various executables +>found on our 4.3 BSD system. + +>-rwxr-xr-x 2 root staff 35840 Nov 3 07:02 /bin/rmail (SMAIL 2.5) +>-rwxr-xr-x 1 root staff 104448 Jun 5 1986 /lib/ccom (C compiler) +>-rwxr-xr-x 1 root staff 97280 Dec 5 05:17 /usr/local/carmen (Lisp) +>-rwsr-xr-x 1 root staff 100352 Apr 5 1987 /usr/lib/sendmail + +The HP840 is a RISC architecture machine. Reduced instruction set implies +more instructions required to do the same job than on a 'complex' +instruction set machine, hence the proportionately larger executable files. +Presumably your 4.3BSD machine is a VAX-alike (i.e. complex instruction set). + +The only other file from your list which exists on our HP840 system is +the C compiler, and look at the size of that beast!! + +-rwxrwxr-x 1 bin bin 1097728 Mar 5 1987 /lib/ccom + +(No, that's not a typo - it really is more than 1 Megabyte!) + +Geoff Clare gwc@root.co.uk seismo!mcvax!ukc!root44!gwc +-- + +Geoff Clare gwc@root.co.uk seismo!mcvax!ukc!root44!gwc +#! rnews 1904 +Path: alberta!mnetor!uunet!mcvax!ukc!stc!datlog!slxsys!jpp +From: jpp@slxsys.specialix.co.uk (John Pettitt) +Newsgroups: comp.unix.xenix +Subject: Re: 16-bit versus 32-bit memory performance +Summary: 32 bit cpu on 16 bit ram is a waste of money +Message-ID: <109@slxsys.specialix.co.uk> +Date: 10 Dec 87 14:17:13 GMT +References: <388@ddsw1.UUCP> <620@omen.UUCP> <435@spdcc.COM> +Reply-To: jpp@slxsys.UUCP (John Pettitt) +Organization: Specialix International, London, UK. +Lines: 29 + +This should perhaps belong in comp.arch + +It would appear that most 8088,8086,186 and 286 systems are +limited by the number of cycles taken to execute instructions +(I.E the clock speed). However the 80386 (at 16 and esp at 20 Mhz) +is limited by its memory bus bandwidth. That is the memory subsystem +on most 286 boxes is fast enough have little or no real effect on +performance compared to a change in clock speed. An 80386 +however is largly limited by the rate that it can be 'fed' data +and instructions. + +16 Bit memory subsystems have a devestating effect on the 80386 +for 2 reasons. Firstly 2 memory accesses are required rather than +one thus doubling the access time. Secondly most 16 bit memory cards +are designed for 8 or 10 Mhz operation not 16 Mhz so a significant +number of wait states are needed when used with a 386. It would +appear that a 'cache miss' on the Intel Inboard(tm) generates beteween +10 and 12 wait states thus making access to 16 bit ram slower than +from the original 286. + +In conclustion - if you want a 32 bit CPU use 32 bit ram. If you +just want the instruction set use the P9 (80388) - if it ever appears. + +(This posting written on a Dell 386 with 6 MB of 0 wait static 32 bit ram) + +-- +John Pettitt - 144.5 MHz: G6KCQ, CIX: jpettitt, Voice: +44 1 398 9422 +UUCP: ...uunet!mcvax!ukc!pyrltd!slxsys!jpp (jpp@slxsys.specialix.co.uk) +Disclaimer: I don't even own a cat to share my views ! +#! rnews 1704 +Path: alberta!mnetor!uunet!mcvax!unido!iaoobelix!woerz +From: woerz@iaoobelix +Newsgroups: comp.unix.wizards +Subject: Re: Request for human interface design a - (nf) +Message-ID: <8300012@iaoobelix.UUCP> +Date: 3 Dec 87 01:35:00 GMT +References: <10559@brl-adm.UUCP> +Lines: 32 +Nf-ID: #R:brl-adm:10559:iaoobelix:8300012:000:1331 +Nf-From: iaoobelix!woerz Dec 3 02:35:00 1987 + +> /***** iaoobelix:comp.unix.wiz / oberon!blarson / 5:40 pm Nov 28, 1987*/ +> In article <7995@steinmetz.steinmetz.UUCP> dawn!stpeters@steinmetz.UUCP (Dick St.Peters) writes: +> >(The VMS interface is not always so friendly to novices: name the file +> >"junk" instead of "junk.txt", and a novice may never figure out how to +> >read it. As for expert interfaces, rename the expert's .emacs file to +> >sav.emacs and watch him/her try to recover.) +> +> I'm no VMS expert and I know a way to recover. Use a gun to put a few +> bullets in the aproprate disk drive. (When it is replaced and the +> backups restored, my .emacs reappears. :-) + +And if you're out of luck, a backup has been done between the time +you changed your .emacs file and the shooting of the disk and you +will get your changed file. :-( + +> -- +> Bob Larson Arpa: Blarson@Ecla.Usc.Edu +> Uucp: {sdcrdcf,cit-vax}!oberon!skat!blarson blarson@skat.usc.edu +> Prime mailing list (requests): info-prime-request%fns1@ecla.usc.edu +> /* ---------- */ + +------------------------------------------------------------------------------ + +Dieter Woerz +Fraunhofer Institut fuer Arbeitswirtschaft und Organisation +Abt. 453 +Holzgartenstrasse 17 +D-7000 Stuttgart 1 +W-Germany + +BITNET: iaoobel.uucp!woerz@unido.bitnet +UUCP: ...{uunet!unido, pyramid}!iaoobel!woerz +#! rnews 1992 +Path: alberta!mnetor!uunet!mcvax!unido!tub!actisb!federico +From: federico@actisb.UUCP (Federico Heinz) +Newsgroups: comp.sys.atari.st +Subject: Re: Hard disk boot??? +Keywords: Hard disk, GEMBOOT +Message-ID: <122@actisb.UUCP> +Date: 8 Dec 87 19:34:12 GMT +References: <624@aucs.UUCP> +Reply-To: federico@actisb.UUCP (Federico Heinz) +Organization: Actis in Berlin GmbH, W. Germany +Lines: 39 + +[The line eater was sleeping again ...] + +In article <624@aucs.UUCP> 870646c@aucs.UUCP (barry comer) writes: +>I have a few questions for anyone using a SH204 with a Mega ST. I have a Meag2 +>with a SH204, I have being auto booting from the hard disk using HDB_V2.3, I +>used to be able to auto boot from the floppy when the CTRL,SHIFT, and ALT. +>keys were held down, well since I started using the Mega, the machine always +>boots from the hard disk with the keys down or up?????????????? + +I didn't know of the CTRL-SHIFT-ALT trick, but I had a problem similar +to yours: there was no way my Mega would boot from floppy, and that +turned out to be quite a problem when a desk accessory I had downloded +from somewhere was turned unusable because of line noise. My "solution" +was not to boot from hard disk at all, which I now find better since it +allows me to choose different configurations (desk accesories and such) +depending on the job I'm going to do. + +>I am also using GEMBOOT to overcome the 40 folder limit in TOS(has it been +>fixed with the new ROMS?). + +I'm also interested on this question, and it has been already asked a couple +of times with no visible answer. I've never used the old ROMs, so I don't +know what the infamous "40 folder limit" means. I've had more than 40 folders +on my hard disk and nothing happened. Does this mean that the problem is +fixed? Or is it 40 folders DEEP? + + + + + /////// + //____ // + Federico // // + // __ // + // / / // + /////// + + +UUCP: ...!mcvax!unido!tub!actisb +BIX: fheinz +#! rnews 888 +Path: alberta!mnetor!uunet!mcvax!varol +From: varol@cwi.nl (Varol Akman) +Newsgroups: sci.physics +Subject: Texts a la Feynman +Summary: I would like to read them +Message-ID: <144@piring.cwi.nl> +Date: 11 Dec 87 10:59:47 GMT +Organization: CWI, Amsterdam +Lines: 12 + +I've been re-reading recently Feynman's excellent volumes and enjoying +myself. The question is: Are there physics books of similar style? +One thing that I like about Feynman is that he tries to ``demystify'' +stuff instead of giving cookbook formulas. Since I do this as a +leisurely activity, the absence of too many formulas and long +mathematical analyses (at least in Vol. I) are also appreciated. +I'm especially interested in classical mechanics. Philosophical +implications of physics laws such as causality, etc. are also interesting. + +Send me individual replies and I'll post a summary to the net. Thanks! + +-Varol Akman +#! rnews 1649 +Path: alberta!mnetor!uunet!mcvax!ukc!dcl-cs!bath63!pes +From: pes@ux63.bath.ac.uk (Smee) +Newsgroups: rec.games.misc +Subject: Re: The Pawn help +Keywords: ** EXPLICIT SPOILERS ** +Message-ID: <2011@bath63.ux63.bath.ac.uk> +Date: 9 Dec 87 11:24:10 GMT +References: <2884@cbmvax.UUCP> <2299@killer.UUCP> <2910@cbmvax.UUCP> +Reply-To: pes@ux63.bath.ac.uk (Smee) +Organization: AUCC c/o University of Bath +Lines: 22 + +In article <2910@cbmvax.UUCP> daveb@cbmvax.UUCP (Dave Berezowski) writes: +> +>I've been told that there is a bug in the game such that you must get to +>the pedestal asap else the blue key won't be there (this is what has happended +>to me)... + +The story I've heard is that this is not a bug. Rather (as warned in the +manual) the other characters you meet are also poking around, and can have +effects even while they are not in the same location as you. + +In particular, as I've heard it, if the adventurer gets to the pedestal before +you do then he will take the key. (And allegedly you then can recover it when +you kill him.) I haven't tried this line of play yet, so can't vouch for it, +but it sounds plausible. + +There's a cute bug in the ST version, though, to do with the pedestal. If +you move the pedestal and then type 'take all' you end up carrying the pedestal, +a duplicate of which remains in place. (If you just try to 'take pedestal', +you are told that it is too heavy to lift.) I'm told that this results from +a bug in the relevant object definition table entry, so it might have propagated +to other versions. (I'd doubt that the driving data undergoes as much analysis +as the executable code during porting to other machines.) +#! rnews 1317 +Path: alberta!mnetor!uunet!mcvax!ukc!dcl-cs!nott-cs!pyr1.cs.ucl.ac.uk!awylie +From: awylie@pyr1.cs.ucl.ac.uk +Newsgroups: comp.sys.ibm.pc +Subject: Zorland/Datalight C INT86 problem +Message-ID: <39500003@pyr1.cs.ucl.ac.uk> +Date: 8 Dec 87 13:23:00 GMT +Lines: 24 +Nf-ID: #N:pyr1.cs.ucl.ac.uk:39500003:000:954 +Nf-From: pyr1.cs.ucl.ac.uk!awylie Dec 8 13:23:00 1987 + + +Hi, + I have a problem with the Zorland C compiler, aka Datalight-C or +NorthWest-C which I wondered if any netlander had previously encountered +and solved. + I have a program which works fine in small model but recently I had +to go to the data model (small code, large data) whereupon it hung my +XT clone. Tracing execution seems to indicate that the DOS software +interrupt routine INT86 may be the source of the trouble. + Has anyone seen problems with INT86 in D or L model programs? The +prospect of DEBUGging the interface between C and assembler does not +appeal to me. + BTW I have deliberately not given details of the program. I do not + want to debug it on the net. Please e-mail me only if you have + solid evidence of problems in the INT86 area. + + thanks for any help you can give, + Andrew + +Andrew Wylie +University of London Computer Centre, London, England + +uucp: awylie@uk.ac.ucl.cs +JANET: andrew@ulcc.ncdlab +#! rnews 644 +Path: alberta!mnetor!uunet!mcvax!ukc!dcl-cs!nott-cs!pyr1.cs.ucl.ac.uk!awylie +From: awylie@pyr1.cs.ucl.ac.uk +Newsgroups: comp.sys.ibm.pc +Subject: Re: Virus program warning +Message-ID: <39500004@pyr1.cs.ucl.ac.uk> +Date: 8 Dec 87 17:12:00 GMT +References: <6146@jade.BERKELEY.EDU> +Lines: 8 +Nf-ID: #R:jade.BERKELEY.EDU:-614600:pyr1.cs.ucl.ac.uk:39500004:000:227 +Nf-From: pyr1.cs.ucl.ac.uk!awylie Dec 8 17:12:00 1987 + + +Presumably it would be relatively easy to modify the virus program to +make it into an 'antibody' which would automatically overwrite the +virus on any infected floppy which was used on the PC. + +Andrew Wylie + +awylie@uk.ac.ucl.cs +#! rnews 541 +Path: alberta!mnetor!uunet!mcvax!ukc!dcl-cs!nott-cs!pyr1.cs.ucl.ac.uk!awylie +From: awylie@pyr1.cs.ucl.ac.uk +Newsgroups: rec.games.hack +Subject: NetHack 2.2 part 18 +Message-ID: <42700005@pyr1.cs.ucl.ac.uk> +Date: 10 Dec 87 09:51:00 GMT +Lines: 8 +Nf-ID: #N:pyr1.cs.ucl.ac.uk:42700005:000:193 +Nf-From: pyr1.cs.ucl.ac.uk!awylie Dec 10 09:51:00 1987 + + +People in the UK and Europe who need NetHack 2.2 part18 can get it by +sending me e-mail, preferably to my Janet address. + +Andrew Wylie + +Janet: andrew@ulcc.ncdlab +uucp: awylie@uk.ac.ucl.cs +#! rnews 892 +Path: alberta!mnetor!uunet!mcvax!ukc!dcl-cs!nott-cs!smb!dave +From: dave@smb.co.uk (Dave Settle) +Newsgroups: comp.sources.wanted +Subject: B-tree routines required. +Keywords: b-tree index rmcobol +Message-ID: <18@oscar.smb.co.uk> +Date: 8 Dec 87 11:17:39 GMT +Organization: SMB Business Software, Mansfield, UK +Lines: 21 + +I'm looking for a set of routines which can handle B-trees, as part of +a program which I'm writing to recover RM-COBOL indexed files. + +If anyone knows of any routines which might be helpful (or any hints about +how to go about it), I'd be very grateful to hear about them. + +Please reply to me directly by mail, as I don't (yet) get this newsgroup +directly. + +Thanks in advance, + Dave Settle. +--- + +Dave Settle, + SMB Business Software, Thorn EMI Datasolve, High St, Mansfield, UK + +UUCP: dave@smb.co.uk + ...!mcvax!ukc!nott-cs!smb!dave + + <--- This way to point of view ---> + +#! rnews 3785 +Path: alberta!mnetor!uunet!mcvax!ukc!eagle!sph +From: sph@eagle.ukc.ac.uk (S.P.Holmes) +Newsgroups: rec.games.misc,rec.games.frp,rec.games.board +Subject: Re: WARGAMING! +Message-ID: <4067@eagle.ukc.ac.uk> +Date: 11 Dec 87 10:24:28 GMT +References: <796@lln-cs.UUCP> +Reply-To: sph@ukc.ac.uk (S.P.Holmes) +Organization: Computing Lab, University of Kent at Canterbury, UK. +Lines: 76 +Xref: alberta rec.games.misc:1153 rec.games.frp:1655 rec.games.board:544 +Summary: + +Expires: + +Sender: + +Followup-To: + + +In article <796@lln-cs.UUCP> gf@lln-cs.UUCP (Frank Grognet) writes: +> +> I want to start wargaming but I don't know how! +> +>I won't be playing wargames on a board, but with 15mm or 25mm +>figurines. +>I would like to find addresses in Europe (especially Belgium) +>of good figurine manufacturers and also references to rule +>books for the Napoleonic period. + +The best set which I've found are the Wargames Research Group 1685 - 1850 +rules. Although the time period sounds a bit long these rules have the +following advantages (My opinions only). + +- Wide ranges of troops covered (You can fight outside Europe) +- Wide range of weapons covered (Pikes for those Moscow Militiamen etc) +- Simple solution for combat - This is what I really like, There@s + No nonsense evaluating every 20th of a casualty, or evaluating + grenadier companies firing separate from the rest of their battallion. +- All weapons are handled simply. Just a different entry in one table. +- Movement is alternate, not simultaneous, things move much quicker. +- Hand to hand combat is decided very quickly, (Just like reality). +- Morale tests are also quite fast to do, and give specific tests for + different situations. (This avoids an old problem where eg Horsemen test + morale before charging, Test fails horribly, Horsemen rout off the + field.) To make you go away, the opponent actually has to do + something. +- European regulars have "National characteristics". + ie British are disciplined infantry and rash cavalry. + Russians are stoical Infantry (Won@t retreat easily) + Spanish are easily panicked + Highlanders charge aggressively + French columns are impetuous and frighten the enemy. + Austrian and Prussian cavalry are Bold + Austrian, Spanish and Dutch Generals are Cautious. + +Together with these rules I would recommend the army lists published by +Table Top Games. + +These cover the European armies for most of the big campaigns of +1805-1815 and ensure a balanced army is selected (Although the +1000 point armies don@t always work too well. +eg My russians need 12 Gun Artillery Batteries (6 pieces on the table) + This leaves me few points for infantry or cavalry + (In practice a Russian 1000 point army has two of Inf, Cav & Art) + +The lists also help to enhance the National Flavour of an army +ie British get few Cavalry, but some veteran Infantry. + French after 1812 have Raw Infantry or Guards. + Austrians Have Very Large numbers Of infantry. + +I can summarise some of the +/- points of each of the armies I've seen +if you mail me. + +I'd recommend 15mm scale troops (Much cheaper and more transportable) + They'll fit on your table too. + +I actually use the 6mm scale which is cheaper, lighter and requires +about 60cm x 100 cm for a medium game. +However the Job of painting, mounting and moving the little guys is +much harder. + + +>I am also interested in rules contained on the net or in files at +>other sites, if they exist! + +Copyright makes this difficult. + +>I anybody can help me, please reply to ..!mcvax!prlb2!lln-cs!gf + + +-- + Steve Holmes | Noel Coward : "Would you object if I smoked" + Room 109a | +E-mail sph | Sarah Bernhardt : "I wouldn't care if you burned" +Phone ext 7681 or 3682 | +#! rnews 1737 +Path: alberta!mnetor!uunet!mcvax!ukc!pyrltd!lucifer!rob +From: rob@lucifer.UUCP ( 237) +Newsgroups: rec.arts.sf-lovers +Subject: Re: Origin of Hithchiker's Guide +Message-ID: <6@lucifer.UUCP> +Date: 11 Dec 87 10:03:57 GMT +References: <909WDMCU@CUNYVM> <1240001@otter.HP.COM> +Reply-To: rob@lucifer.UUCP (Rob Clive - 237) +Organization: Lucas Micos, Phoenix Way, Cirencester, Glos, UK (0285 67981) +Lines: 24 + +In article <1240001@otter.HP.COM> kers@otter.HP.COM (Christopher Dollin) writes: +>> I have recently been told be someone that The Hitchhiker's Guide to the +>> Galaxy originated as a radio program rather than as a book. +> +>The radio series "The Hitch-hikers Guide to the Galaxy" was broadcast in +>Britain for the first time between 1976..1979 (sorry for the range but all I + +It was 1978. Episode 1 of the first series was a pilot production for the +whole thing and as such is slightly different in flavour to the others. The +first series (6 episodes) covered the ground of the TV version and books 1 +and 2. Then came the Christmas (1978) show to make a link to the second +series which was broadcast in 1979 and consisted of 5 episodes. + +> For my money, the show (and scripts) are MUCH funnier than the books. + +True. The radio shows left much more to the imagination with the assistance +of some very good sound effects. I thought the TV series spoiled it. For +instance at the end of the first radio series you hear the song 'What a +Wonderful World' amid the sound of burning trees on prehistoric Earth; can't +you just imagine it? + +----------------------------------------------------------------------------- +Rob Clive. UUCP: ...!mcvax!ukc!lucifer!rob +Lucas Micos Ltd., Cirencester, GL7 1QG, UK. Now read on.... +#! rnews 1160 +Path: alberta!mnetor!uunet!mcvax!botter!tjalk!rblieva +From: rblieva@cs.vu.nl (Roemer Lievaart) +Newsgroups: rec.music.classical +Subject: Re: The range of the male voice. +Message-ID: <918@tjalk.cs.vu.nl> +Date: 11 Dec 87 13:06:10 GMT +References: <1280@phoenix.Princeton.EDU> <1597@faline.bellcore.com> <3999@pucc.Princeton.EDU> +Reply-To: rblieva@cs.vu.nl (Roemer B. Lievaart) +Organization: VU Informatica, Amsterdam +Lines: 15 + +Q2816@pucc.Princeton.EDU (Roger Lustig) typed: ++--------------------------------------- +| Choral music is generally written for a fairly restricted range (note +| the two qualifications in that sentence) in order to allow choirs, not +| individuals, to sing it. There are choral high Bb's (in Singet dem +| Herrn, for instance) and even C's for the sopranos (end of Kodaly's +| Laudes Organi), and the incredible stuff Beethoven asked for in the +| Missa Solemnis and Ninth. But they are the exception, and are generally +| intended to sound like an exception. ++--------------------------------------- + +We're playing Mahler's 2nd, and so I noticed last wednesday that +the Basses have to sing as deep as (at least ?) the low B. + + -- Roemer. +#! rnews 871 +Path: alberta!mnetor!uunet!mcvax!botter!ast +From: ast@cs.vu.nl (Andy Tanenbaum) +Newsgroups: comp.os.minix +Subject: Re: scanf() +Message-ID: <1782@botter.cs.vu.nl> +Date: 11 Dec 87 14:40:54 GMT +References: <782@louie.udel.EDU> +Reply-To: ast@cs.vu.nl (Andy Tanenbaum) +Organization: VU Informatica, Amsterdam +Lines: 10 + +In article <782@louie.udel.EDU> KIMMEL%ecs.umass.edu@relay.cs.net (Matt Kimmel) writes: +>I just got Minix v1.2, and I like it a lot. However, when I try to +>compile a C program that calls scanf(), I get a message to the effect +>of " _scanf not resolved". Am I missing something? Or is there no scanf() + +There is a scanf in libsrc.a, but it is not included in libc.a. You have to +compile it yourself with cc -LIB -c scanf.c and put in in the library. +It was omitted from libc.a because there was no room on that diskette! + +Andy Tanenbaum (ast@cs.vu.nl) +#! rnews 1614 +Path: alberta!mnetor!uunet!mcvax!guido +From: guido@cwi.nl (Guido van Rossum) +Newsgroups: comp.windows.x +Subject: X and different IPC protocols +Summary: Surely feasible; but how useful? +Message-ID: <145@piring.cwi.nl> +Date: 11 Dec 87 22:15:13 GMT +Reply-To: guido@cwi.nl (Guido van Rossum) +Organization: "The Amoeba Project", CWI, Amsterdam +Lines: 22 + +Although X as distributed uses TCP/IP to connect clients and server, it +is possible use other network protocols by relatively small changes to +the lowest levels of library and server. We have almost gotten the +server half of such a set-up running using Amoeba (a distributed +operating system with its own, capability-based RPC mechanism). +The library half should be working as soon as we solve problems with the +C compiler. + +The question is, how much does this buy us? Since Amoeba is not Unix, X +clients requiring advanced Unix features won't run under vanilla Amoeba. +What percentage of the available client applications will be convertable +to a different operating system, where, e.g., one will have +available, but not select(2)? I would assume that there will be VMS +support for X, so that one might expect clients to be OS-independent, +but then again, you can never know what hacks a performance-driven +application programmer may use... (including VAX assembly :-) + +Can anybody comment on this? It would also be interesting to know if +third-party software for X would come binary or source. +-- +Guido van Rossum, Centre for Mathematics and Computer Science (CWI), Amsterdam +guido@cwi.nl or mcvax!guido or (from ARPAnet) guido%cwi.nl@uunet.uu.net +#! rnews 1102 +Path: alberta!mnetor!uunet!mcvax!prlb2!kulcs!wim +From: wim@kulcs.UUCP (Wim De Bisschop) +Newsgroups: comp.lang.ada +Subject: Ada-interface to Termcap(3) +Keywords: termcap +Message-ID: <1075@kulcs.UUCP> +Date: 11 Dec 87 10:56:28 GMT +Organization: Kath.Univ.Leuven, Comp. Sc., Belgium +Lines: 15 + +Has anyone an Ada interface to the C routines from the termcap +library? We would have a package for terminal independent +screen oriented output in Ada. The most natural way to do this, +is to make use of the C-routines of termcap. +We were wondering whether someone else has already defined an +interface package, preferably for a Verdix 5.41 compiler to +run under 4.3BSD. + + ++----------------------------------------------------------------------+ +| Name: Wim De Bisschop | Katholieke Universiteit Leuven | +| E-mail: wim@kulcs.UUCP or | Department of Computer Science | +| ...!mcvax!prlb2!kulcs!wim | Celestijnenlaan 200 A | +| Phone: +(32) 16-200656 x3596 | B-3030 Leuven (Heverlee), Belgium| ++----------------------------------------------------------------------+ +#! rnews 835 +Path: alberta!mnetor!uunet!mcvax!enea!erix!erialfa!afr +From: afr@erialfa.UUCP (Anders Fredrikson ZX/DRG) +Newsgroups: rec.music.misc +Subject: Re: Ace-Screamingest Guitar Solos on Record +Message-ID: <172@erialfa.UUCP> +Date: 10 Dec 87 12:31:18 GMT +References: <1725@s.cc.purdue.edu> <2455@sfsup.UUCP> +Reply-To: afr@erialfa.UUCP (Anders Fredrikson ZX/DRG) +Organization: Ericsson Information Systems AB, Kista, Stockholm, SWEDEN +Lines: 17 + +In article <2455@sfsup.UUCP> mingus@sfsup.UUCP (Damballah Wedo) writes: +>> rsk@s.cc.purdue.edu.UUCP (in <1725@s.cc.purdue.edu>): +>> [ lists some excellent guitar solos ] +> +>Sure, I'll play that game: +> +>...... +>---cut +>She'a a Woman (Jeff Beck, BLOW BY BLOW) +This tune is even better on the "Jeff Beck & Jan Hammer group LIVE" +>---Cut +>..... +You might also add +Europa (Santana, MOONFLOWER) + + +/Anders +#! rnews 1046 +Path: alberta!mnetor!uunet!mcvax!enea!pvab!robert +From: robert@pvab.UUCP (Robert Claeson) +Newsgroups: comp.lang.c +Subject: Re: Making re-#includes harmless--a simple solution? +Message-ID: <339@pvab.UUCP> +Date: 11 Dec 87 10:23:09 GMT +References: <13395@think.UUCP> +Reply-To: robert@pvab.UUCP (Robert Claeson) +Organization: Statskonsult Programvaruhuset AB, Sweden +Lines: 16 + +In article <13395@think.UUCP> rlk@THINK.COM writes: + +>1) The same file may have multiple names (symlinks and/or hard +>links). How do you KNOW whether a file has been included? The only +>way is by defining an attribute that only that file will have. The +>easiest way to do this (aside from checking device/inumbers, which is +>not portable and may not work in some bizarre cases, or other system +>dependent hacks) is to #define a unique name. + +How can you be sure that the name you choose is unique, especially if +you use links or symlinks? + +-- +Robert Claeson, System Administrator, PVAB, Box 4040, S-171 04 Solna, Sweden +eunet: robert@pvab +uucp: sun!enea!pvab!robert +#! rnews 1812 +Path: alberta!mnetor!uunet!mcvax!enea!ttds!draken!sics!lhe +From: lhe@sics.se (Lars-Henrik Eriksson) +Newsgroups: rec.arts.sf-lovers +Subject: Re: Houston SF Opera +Message-ID: <1642@sics.se> +Date: 11 Dec 87 10:31:40 GMT +References: <8168@ism780c.UUCP> +Reply-To: lhe@sics.se (Lars-Henrik Eriksson) +Organization: Swedish Institute of Computer Science, Kista +Lines: 32 + +In article <8168@ism780c.UUCP> jimh@ism780c.UUCP (Jim Hori) writes: +>The Lessing is probably Doris who has +>written several futurist/SF novels ... + +>Her SF novels are serialized, and from what +>I recall from scanning them in bookstores, +>reminiscent of Marge Piercy's enjoyable, +>though somewhat stiff, feminist SF. +> +>The series is called "Canopus and Argos: Archives", +Should be Canopus IN Argos: Archives + +The five books are quite different in character. The second one +("The marriages between zones 3, 4 and 5") could possibly be called +"feminist SF" - it is very different from the other four in most ways. +The third ("The Sirian Experiments") is at times rather funny, and the +fifth ("The sentimental agents in the Volyen empire") is among the funniest +books I've read. + +On the other hand, number 4 ("The making of the representative of planet 8") +was rather depressing. While reading it I thought that "it can't get any +worse than this". It could, of course. (I don't refer to the quality of the +book, but to the events in the story). + +I should mention the title of the first one also: "Shikasta" This is +the most "important" of the five, in some sense. It is also the one that +could perhaps be called "stiff". All the books are well worth reading. + +Lars-Henrik Eriksson Internet: lhe@sics.se +Swedish Institute of Computer Science Phone (Intn'l): +46 8 750 79 70 +Box 1263 Telefon (nat'l): 08 - 750 79 70 +S-164 28 KISTA +#! rnews 768 +Path: alberta!mnetor!uunet!mcvax!enea!tut!santra!kolvi!jku +From: jku@kolvi.UUCP (Juha Kuusama) +Newsgroups: comp.sys.ibm.pc +Subject: Re: EVALuation of Shareware Word Processors - Version 1 +Message-ID: <32@kolvi.UUCP> +Date: 11 Dec 87 07:40:17 GMT +References: <3610@dhw68k.UUCP> +Reply-To: jku@kolvi.UUCP (Juha Kuusama) +Organization: Helsinki University of Technology, Finland +Lines: 9 + +I'm not at all questioning the value of the comparision, but (as a VERY +satisfied and registered) user of PC-Write, I'd like to point out that: + +- PC-Write does support the ega in 43-line mode + +- PC-Write can remind you to do backups at specified time intervals or + when you have entered a specified number of characters. +--- +Juha Kuusama, jku@kolvi.UUCP ( ...!mcvax!tut!kolvi!jku ) +#! rnews 904 +Path: alberta!mnetor!uunet!mcvax!diku!daimi!jnp +From: jnp@daimi.UUCP (J|rgen N|rgaard) +Newsgroups: comp.sys.mac +Subject: Re: Conjecture: why several tech notes failed +Message-ID: <1248@daimi.UUCP> +Date: 10 Dec 87 08:43:14 GMT +References: <9827@ut-sally.UUCP> +Reply-To: jnp@titan.UUCP (J|rgen N|rgaard) +Organization: DAIMI: Computer Science Department, Aarhus University, Denmark +Lines: 16 + + +Earlier this year there has been trouble with tech-notes, that would +not binhex correctly (the Mac program). +Then the problem could be solved with a similiar program on unix-machines. +The problem seemed to show up when the file-names where extremely long +(28 might be the number). + +It seemed not to be so sensitive about file-names. + +Unfortunately I have lost the sources. + + +-- + Regards J|rgen N|rgaard + e-mail: jnp@daimi.dk +------------------------------------------------------------------------------- +#! rnews 785 +Path: alberta!mnetor!uunet!mcvax!diku!iesd!jacob +From: jacob@iesd.uucp (Jacob stergaard B{kke) +Newsgroups: sci.misc +Subject: A request on the Ozone layer +Keywords: More information wanted about the Ozone layer. +Message-ID: <174@iesd.uucp> +Date: 11 Dec 87 13:38:23 GMT +Reply-To: jacob@iesd.UUCP (Jacob \stergaard B{kke) +Organization: Dept. of Comp. Sci., Aalborg University, Denmark +Lines: 12 + +Today I read an posting from rhorn@infinet.UUCP about the problems +with the Ozone layer. So I got interested and now wanted more +information about it and the problems with the Ozone layer in +Switzerland present. I would like any information and I'll look +forward to any reponds. + + Yours sincerely + + Jacob Baekke, Denmark + + +Reply to: jacob@iesd.uucp, {...}!mcvax!diku!iesd!jacob +#! rnews 1246 +Path: alberta!mnetor!uunet!mcvax!inria!imag!jarwa +From: jarwa@imag.UUCP (Jarwa Sahar) +Newsgroups: comp.software-eng +Subject: LOOKING FOR DOCUMENTS ON SOFTWARE DOCUMENTATION +Message-ID: <2336@imag.UUCP> +Date: 11 Dec 87 09:15:21 GMT +Reply-To: jarwa@imag.UUCP (Jarwa Sahar) +Organization: IMAG, University of Grenoble, France +Lines: 26 + + + I am very interested in all publications concerning Documents + Related to Software Documentation and to Maitenance Environment. + + What I am interested in are papers on different types + of these documents, their formalism and their structure. + + If this area also interest you, I'd be very pleased if you could + contact me, or send me your papers and/or what you have found + interesting pertaining to this area. This will help me making a + preliminary study on it. + + Looking forward to your answer, and thank you for your help. + Sahar JARWA + + My adress is + Sahar JARWAH + Equipe "Systemes Intelligents de Recherche d'Informations" + Laboratoire de Genie Informatique - IMAG + BP 68 + 38462 St Martin d'Heres Cedex + FRANCE + + my phone is 76-51-46-00 extension 5182 + + my electronic adress is jarwa@imag.imag.fr + on UUCP: jarwa@imag +#! rnews 1217 +Path: alberta!mnetor!uunet!mcvax!inria!imag!jarwa +From: jarwa@imag.UUCP (Jarwa Sahar) +Newsgroups: comp.databases +Subject: LOOKING FOR DOCUMENTS +Message-ID: <2337@imag.UUCP> +Date: 11 Dec 87 09:18:16 GMT +Reply-To: jarwa@imag.UUCP (Jarwa Sahar) +Organization: IMAG, University of Grenoble, France +Lines: 26 + + + I am very interested in all publications concerning Documents + Related to Software Documentation and to Maitenance Environment. + + What I am interested in are papers on different types + of these documents, their formalism and their structure. + + If this area also interest you, I'd be very pleased if you could + contact me, or send me your papers and/or what you have found + interesting pertaining to this area. This will help me making a + preliminary study on it. + + Looking forward to your answer, and thank you for your help. + Sahar JARWA + + My adress is + Sahar JARWAH + Equipe "Systemes Intelligents de Recherche d'Informations" + Laboratoire de Genie Informatique - IMAG + BP 68 + 38462 St Martin d'Heres Cedex + FRANCE + + my phone is 76-51-46-00 extension 5182 + + my electronic adress is jarwa@imag.imag.fr + on UUCP: jarwa@imag +#! rnews 2496 +Path: alberta!mnetor!uunet!mcvax!unido!laura!hmm +From: hmm@laura.UUCP (Hans-Martin Mosner) +Newsgroups: comp.lang.smalltalk +Subject: User Survey +Keywords: survey smalltalk curiosity +Message-ID: <165@laura.UUCP> +Date: 10 Dec 87 21:31:51 GMT +Organization: University of Dortmund, W-Germany +Lines: 59 + +To stir up some unrest, we have decided to post a smalltalk user survey. +Where are you, all you happy smalltalk hackers ? There must be life +in other parts of the world, too... :-) +Anyway, we would like you to fill in this questionnaire and give us some +feedback. Of course we would also like if you would post your experiences +and questions to this group. After all, that's it's purpose... + + Hans-Martin Mosner & Andreas Toenne + Smalltalk hackers at the University of Dortmund + ++------------------------------- +|1. What kind of hardware/software do you use: +|1.1. Hardware +|1.1.1. Processor type: _____ +|1.1.2. Physical memory size: _____ +|1.1.3. Display size: _____ +|1.2. Software +|1.2.1. Operating system: _____ +|1.2.2. Virtual machine: _____ +|1.2.3. Virtual image version: _____ +|1.3 Overall performance: _____ % Dorado (if you know that) +|2. For what purposes do you use smalltalk ? +| (FillInThisBlank) +|3. Do you think that the system meets your requirements ? +| If not, why ? +|4. If you are a programmer: +|4.1. What kind of applications have you written ? +|4.2. If those applications were not written for your employer, +| why didn't you share them with the Usenet community ? :-) +|5. How do you like smalltalk ? +|5.1. How long have you been using smalltalk ? +|5.2. How familiar are you with smalltalk ? ++------------------------------- +Thank you for being so cooperative. +Now that you have answered all those questions, please +send the whole thing back to: + + hmm@unido.uucp +or hmm@unido.bitnet +or ...!uunet!unido!hmm +or hmm%unido.uucp@uunet.uu.net + +If everything fails, just post it to this group... + +If even that does not work, then send it via snail mail to: + Hans-Martin Mosner + Informatik-Rechner-Betriebsgruppe + Universitaet Dortmund + Postfac` 500500 +D-4600 Dortmund + West Germany + +Disclaimer: these opinions are not opinions but just random bits & bytes +and therefore I don't need to disclaim anything... +-- +Hans-Martin Mosner | Don't tell Borland about Smalltalk - | +hmm@unido.{uucp,bitnet} | they might invent TurboSmalltalk ! | +------------------------------------------------------------------------ +Disclaimer: TurboSmalltalk may already be a trademark of Borland... +D +#! rnews 14600 +Path: alberta!mnetor!uunet!mcvax!unido!laura!atoenne +From: atoenne@laura.UUCP (Andreas Toenne) +Newsgroups: comp.lang.smalltalk +Subject: A small IconEditor for Smalltalk 80, VI2.2 +Keywords: smalltalk icons goodie +Message-ID: <166@laura.UUCP> +Date: 10 Dec 87 21:48:21 GMT +Organization: University of Dortmund, W-Germany +Lines: 525 + +Here is a little IconEditor I wrote. +This goodie works on Smalltalk 80 VI2.2 VM1.1 +It comes in two parts. +The first part 'Icon menu.st' adds knowledge about icons to the +StandardSystemController's blueButtonMenu. +You should file in this one first. +The second part 'Icon Editor.st' is the editor himself. + +Some notes about icons: +The icon's textRectangle is clipped with the icon's boundingBox. +To cancel a given textRectangle simply move it outside the outlined box. +The method storeOn: in class OpaqueForm is buggy. +You should add enclosing round brackets to the output. Otherwise +you won't be able to read the saved icon definitions back. + + Have fun + + Andreas Toenne + atoenne@unido.uucp + atoenne@unido.bitnet + ...!uunet!unido!atoenne + atoenne%unido.uucp@uunet.uu.net + +~~~~~~~~~~~~~~~~~~ cut here for best results ~~~~~~~~~~~~~~~~~~~~~~~~~~ +#! /bin/sh +# This is a shell archive, meaning: +# 1. Remove everything above the #! /bin/sh line. +# 2. Save the resulting text in a file. +# 3. Execute the file with /bin/sh (not csh) to create: +# Icon Editor.st +# Icon Menu.st +# This archive created: Thu Dec 10 22:36:04 1987 +export PATH; PATH=/bin:/usr/bin:$PATH +if test -f 'Icon Editor.st' +then + echo shar: "will not over-write existing file 'Icon Editor.st'" +else +cat << \SHAR_EOF > 'Icon Editor.st' +MouseMenuController subclass: #IconDisplayController + instanceVariableNames: '' + classVariableNames: '' + poolDictionaries: '' + category: 'Icon Editor'! + + +!IconDisplayController methodsFor: 'controller default'! + +isControlActive + ^ super isControlActive and: [sensor blueButtonPressed not]! ! + +!IconDisplayController methodsFor: 'menu messages'! + +yellowButtonActivity + | index menu | + menu _ view yellowButtonMenu. + menu == nil + ifTrue: + [view flash. + super controlActivity] + ifFalse: + [index _ menu startUpYellowButton. + index ~= 0 + ifTrue: + [self controlTerminate. + view perform: (menu selectorAt: index). + self controlInitialize]]! ! + +View subclass: #IconDisplayView + instanceVariableNames: 'icon aspect iconMsg iconMenu ' + classVariableNames: '' + poolDictionaries: '' + category: 'Icon Editor'! +IconDisplayView comment: +'I am a stupid view used to display the edited icon'! + + +!IconDisplayView methodsFor: 'displaying'! + +displayView + "display icon centered in my insetBox" + + | r iconRect rec | + Display white: self insetDisplayBox. + (icon isKindOf: Icon) + ifTrue: + [r _ self insetDisplayBox. + icon form displayOn: Display at: r topLeft + r bottomRight - icon form extent // 2. + iconRect _ icon form computeBoundingBox. + iconRect _ iconRect translateBy: r topLeft + r bottomRight - iconRect extent // 2. + (iconRect areasOutside: (iconRect insetBy: 1 @ 1)) + do: [:edge | Display fill: edge mask: Form gray]. + rec _ icon textRect. + rec = nil + ifFalse: + [rec _ rec translateBy: r topLeft + r bottomRight - icon form computeBoundingBox extent // 2. + (rec areasOutside: (rec insetBy: 1 @ 1)) + do: [:edge | Display fill: edge mask: Form gray]]]! ! + +!IconDisplayView methodsFor: 'updating'! + +update: anAspect + "update the view" + + anAspect == aspect + ifTrue: + [icon _ model perform: iconMsg. + self displayView]! ! + +!IconDisplayView methodsFor: 'menu messages'! + +allBlack + "make the selected icon all black" + | figure shape | + figure _ icon form figure. + shape _ icon form shape. + figure fill: figure computeBoundingBox rule: Form over mask: Form black. + shape fill: figure computeBoundingBox rule: Form over mask: Form black. + model changed: #iconView! + +allGray + "make the selected icon all transparent" + | figure shape | + figure _ icon form figure. + shape _ icon form shape. + figure fill: figure computeBoundingBox rule: Form over mask: Form white. + shape fill: figure computeBoundingBox rule: Form over mask: Form white. + model changed: #iconView! + +allWhite + "make the selected icon all white" + | figure shape | + figure _ icon form figure. + shape _ icon form shape. + figure fill: figure computeBoundingBox rule: Form over mask: Form white. + shape fill: figure computeBoundingBox rule: Form over mask: Form black. + model changed: #iconView! + +editIcon + "edit the selected icon" + + | figure shape opaqueForm iconExtent bitView viewPoint savedForm | + (icon = nil and: [model iconSymbol ~= #default]) + ifTrue: + [iconExtent _ Rectangle fromUser extent. + figure _ Form extent: iconExtent. + shape _ Form extent: iconExtent. + opaqueForm _ OpaqueForm figure: figure shape: shape. + model icon: (Icon form: opaqueForm textRect: nil)]. + icon = nil + ifFalse: + [viewPoint _ (BitEditor locateMagnifiedView: icon form scale: 4 @ 4) topLeft. + bitView _ BitEditor + bitEdit: icon form + at: viewPoint + scale: 4 @ 4 + remoteView: nil. + savedForm _ Form fromDisplay: (bitView displayBox merge: bitView labelDisplayBox). + bitView controller startUp. + savedForm displayOn: Display at: bitView labelDisplayBox topLeft. + bitView release. + model changed: #iconView]! + +textRect + "let the user specify a rectangle that will hold the icon's text" + + | rec r| + rec _ Rectangle fromUser. + r _ self insetDisplayBox. + rec _ rec translateBy: 0@0 - (r topLeft + r bottomRight - icon form computeBoundingBox extent //2). + icon form: icon form textRect: rec. + model changed: #iconView! ! + +!IconDisplayView methodsFor: 'controller access'! + +defaultControllerClass + ^IconDisplayController! ! + +!IconDisplayView methodsFor: 'private'! + +on: anIcon aspect: m1 icon: m2 menu: m3 + self model: anIcon. + aspect _ m1. + iconMsg _ m2. + iconMenu _ m3! ! + +!IconDisplayView methodsFor: 'adaptor'! + +yellowButtonMenu + ^ self model perform: iconMenu! ! +"-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- "! + +IconDisplayView class + instanceVariableNames: ''! + + +!IconDisplayView class methodsFor: 'instance creation'! + +on: anIcon aspect: m1 icon: m2 menu: m3 + "create a new view for anIcon with aspect m1" + + ^self new + on: anIcon + aspect: m1 + icon: m2 + menu: m3! ! + +Model subclass: #IconEditor + instanceVariableNames: 'icon iconSymbol iconBuffer ' + classVariableNames: 'IconMenu ListMenu ' + poolDictionaries: '' + category: 'Icon Editor'! +IconEditor comment: +'I am a bit editor for system icons. + +Instance Variables : + icon "The selected icon" + iconSymbol "The symbol for the selected icon" + +Class Variables: + ListMenu "The action menu for the SelectionInListView over all icons"'! + + +!IconEditor methodsFor: 'accessing'! + +icon + "return the selected icon" + + ^icon! + +icon: anIcon + "change the selected Icon to anIcon" + + icon _ anIcon. + Icon constantNamed: iconSymbol put: anIcon. + self changed: #iconView " aspect for the IconDisplayView"! + +icon: anIcon named: aSymbol + " store anIcon at position aSymbol" + + Icon constantNamed: aSymbol put: anIcon. + icon _ anIcon. + iconSymbol _ aSymbol. + self changed: #iconSymbol. "aspect for SelectionInListView" + self changed: #iconView "aspect for iconDisplayView "! + +iconSymbol + "return the symbol for the selected icon" + + ^iconSymbol! + +iconSymbol: aSymbol + "change the symbol for the selected icon to aSymbol" + + iconSymbol _ aSymbol. + icon _ Icon constantNamed: aSymbol. + self changed: #iconView "aspect for the IconDisplayView"! ! + +!IconEditor methodsFor: 'removing'! + +removeIcon + " remove the currently selected icon " + + Icon constantDictionary removeKey: iconSymbol ifAbsent: [^nil]. + iconSymbol _ icon _ nil. + self changed: #iconSymbol. + self changed: #iconView! ! + +!IconEditor methodsFor: 'list display'! + +iconList + "return the list of icon symbols" + + | list | + list _ OrderedCollection new. + Icon constantDictionary keysDo: [:i | list add: i]. + ^list! + +initialSymbol + "get the initial symbol selection" + "this method is used every time the SelectionInListView receives an update mesage " + + ^iconSymbol! + +listMenu + "return the menu for the icon list" + + ^ListMenu! ! + +!IconEditor methodsFor: 'icon display'! + +iconMenu + "return the menu for the iconDisplayController" + + ^IconMenu! ! + +!IconEditor methodsFor: 'menu messages'! + +copy + " save a (deep) copy of the currently selected icon" + + icon = nil ifFalse: [iconBuffer _ icon deepCopy]! + +cut + " remove the currently selected icon from the icon dictionary and + save it in iconBuffer" + + (icon ~= nil or: [iconSymbol ~= #default]) + ifTrue: + [iconBuffer _ icon. + self removeIcon]! + +loadIcon + "override the current icon with a definition from a file" + + | aFileName anIcon aStream | + (icon ~= nil or: [iconSymbol ~= #default]) + ifTrue: + [aFileName _ FileDirectory + requestFileName: 'file : ' + default: iconSymbol asString , '.icn' + version: #old + ifFail: [^'']. + aFileName ~= '' + ifTrue: + [aStream _ FileStream oldFileNamed: aFileName. + anIcon _ Object readFrom: aStream. + aStream close. + self icon: anIcon]]! + +newIcon + " create a new clean icon" + + | iconName | + iconName _ FillInTheBlank request: 'Icon Name ?'. + iconName = '' ifFalse: [self icon: nil named: iconName asSymbol]! + +paste + " change the currently selected icon to the icon held in iconBuffer" + " invoke newIcon if none is selected" + + iconSymbol = nil + ifTrue: + ["add a new icon" + self newIcon. + iconSymbol = nil ifFalse: [self icon: iconBuffer]] + ifFalse: ["override old icon" + self icon: iconBuffer]! + +renameIcon + " change the name of an icon" + + | key value newName | + (icon ~= nil or: [iconSymbol ~= #default]) + ifTrue: + [key _ iconSymbol. + value _ icon. + newName _ FillInTheBlank request: 'Change icon name' initialAnswer: key. + newName ~= '' + ifTrue: + [self removeIcon. + self icon: value named: newName asSymbol]]! + +saveIcon + "store the selected icon to a file" + + | aFileName aStream | + icon = nil + ifFalse: + [aFileName _ FileDirectory + requestFileName: 'file : ' + default: iconSymbol asString , '.icn' + version: #any + ifFail: [^'']. + aFileName ~= '' + ifTrue: + [aStream _ FileStream newFileNamed: aFileName. + icon storeOn: aStream. + aStream close]]! ! + +!IconEditor methodsFor: 'view creation'! + +open + "open the views" + + | topView | + topView _ StandardSystemView + model: self + label: 'Icon Editor' + minimumSize: 256 @ 300. + topView + addSubView: (SelectionInListView + on: self + aspect: #iconSymbol + change: #iconSymbol: + list: #iconList + menu: #listMenu + initialSelection: #initialSymbol) + in: (0 @ 0 corner: 1.0 @ 0.3) + borderWidth: 1. + topView + addSubView: (IconDisplayView + on: self + aspect: #iconView + icon: #icon + menu: #iconMenu) + in: (0.0 @ 0.3 corner: 1.0 @ 1.0) + borderWidth: 1. + topView controller open! ! +"-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- "! + +IconEditor class + instanceVariableNames: ''! + + +!IconEditor class methodsFor: 'class initialization'! + +initialize + "Initialize the class IconEditor" + "IconEditor initialize" + + ListMenu _ ActionMenu labelList: #((copy cut paste ) (newIcon renameIcon ) (saveIcon loadIcon ) ) selectors: #(copy cut paste newIcon renameIcon saveIcon loadIcon ). + IconMenu _ ActionMenu labelList: #((editIcon textRect ) (allWhite allBlack allGray) ) selectors: #(editIcon textRect allWhite allBlack allGray)! ! + +!IconEditor class methodsFor: 'instance creation'! + +open + "create on schedule a new Icon Editor" + + self new open! ! + +IconEditor initialize! +SHAR_EOF +fi +if test -f 'Icon Menu.st' +then + echo shar: "will not over-write existing file 'Icon Menu.st'" +else +cat << \SHAR_EOF > 'Icon Menu.st' +!MouseMenuController methodsFor: 'menu messages'! + +blueButtonActivity + "Determine which item in the blue button pop-up menu is selected. + If one is selected, then send the corresponding message to the object + designated as the menu message receiver." + "Enhanced to use HierarchicalMenus by atoenne@unido.uucp" + + | index | + blueButtonMenu ~~ nil + ifTrue: + [index _ blueButtonMenu startUpBlueButton. + index ~= 0 ifTrue: [blueButtonMenu class = HierarchicalMenu + ifTrue: [self menuMessageReceiver perform: (blueButtonMenu selectorAt: index)] + ifFalse: [self menuMessageReceiver perform: (blueButtonMessages at: index)]]] + ifFalse: [super controlActivity]! ! + +!StandardSystemController class methodsFor: 'class initialization'! + +initialize + "Initialize the class variables." + "StandardSystemController initialize. + StandardSystemController allInstances do: [:sc | sc + initializeBlueButtonMenu] " + + ScheduledBlueButtonMenu _ (MenuBuilder parseFrom: (ReadStream on: 'newLabel[newLabel] +(under[under] move[move] frame[frame]) (collapse[collapse] +icon: ((selectIcon[selectIcon] editIcon[editIcon]) (loadIcons[loadIcons] saveIcons[saveIcons]))) +(close[close])')) menu. + MenuWhenCollapsed _ ActionMenu + labels: 'new label\under\move\expand\close' withCRs + lines: #(1 4 ) + selectors: #(newLabel under move expand close )! ! + +!StandardSystemController methodsFor: 'menu messages'! + +editIcon + " call an icon editor " + + IconEditor open! + +loadIcons + "load new constant definitions for icons" + + | aFileName | + aFileName _ FileDirectory + requestFileName: 'file:' + default: '*.icn' + version: #old + ifFail: [^'']. + aFileName ~= '' ifTrue: [Icon constantsFromFile: aFileName]! + +saveIcons + "write current icon constants to a file" + + | aFileName | + aFileName _ FileDirectory + requestFileName: 'file:' + default: '*.icn' + version: #any + ifFail: [^'']. + aFileName ~= '' ifTrue: [Icon constantsToFile: aFileName]! + +selectIcon + "let the user choose from the current icons" + + | nameList iconList selection selectedIcon | + nameList _ OrderedCollection new. + Icon constantDictionary keysDo: [:key | nameList add: key]. + iconList _ Array with: nameList asArray. + selection _ (PopUpMenu labelList: iconList) startUp. + selection ~= 0 + ifTrue: + [selectedIcon _ (Icon constantNamed: (nameList at: selection) asSymbol) copy. + self view icon: selectedIcon. "change the icon" + self view iconView lock. "essential. see below" + self view iconView text: self view label. "set new icon text" + self view iconView newIcon] "compute new icon" +"lock is needed to perform the newIcon computation. Otherwise insetDisplayBox would be garbled. Text setting is merely needed at the first change. (The standard label has no iconText) "! ! + +!StandardSystemController initialize. +StandardSystemController allInstances do: [:sc | sc +initializeBlueButtonMenu]! +SHAR_EOF +fi +exit 0 +# End of shell archive +D +#! rnews 813 +Path: alberta!mnetor!uunet!mcvax!unido!laura!atoenne +From: atoenne@laura.UUCP (Andreas Toenne) +Newsgroups: rec.games.hack +Subject: Re: Nethack 2.2: You stop to avoid hitting. +Keywords: I have this bug too. +Message-ID: <167@laura.UUCP> +Date: 10 Dec 87 21:53:23 GMT +References: <7515@alice.UUCP> +Reply-To: atoenne@unido.UUCP (Andreas Toenne) +Organization: University of Dortmund, W-Germany +Lines: 9 + +In article <7515@alice.UUCP> wilber@alice.UUCP writes: +>I have nethack running on my 3B1. So far the only bug I've encountered +>is the message "You stop to avoid hitting." (Which sometimes comes out as +>"You stop to avoid hitting .") I haven't hit the plethora + +You have defined DOGNAME but you are missing the dog's name :-) +Simply add 'dogname:...' to your nethack options. + + Andreas Toenne +D +#! rnews 1201 +Path: alberta!mnetor!uunet!mcvax!unido!rmi!dg2kk!dg2kk +From: dg2kk@dg2kk.UUCP (Walter) +Newsgroups: rec.ham-radio.packet +Subject: Problems with WA8DED 2.1 and TNC-2 clones (+possible solution) +Summary: PTT line is released too early +Message-ID: <174@dg2kk.UUCP> +Date: 10 Dec 87 23:09:52 GMT +Reply-To: dg2kk@dg2kk.UUCP +Organization: dg2kk, W Germany, (JO30FT) +Lines: 20 + +Some TNC-2's have problems with the WA8DED software (version 2.1). +Most of the outgoing frames cannot be docoded by other stations because +the software turns off the transmitter before all bits have been transmitted. + +There are two solutions to this problem: + +Hardware: connect a small (~2.2uf) capacitor from the base of the PTT keying + transistor to ground. (Note: you may have to increase TXDELAY) + +Software: the code that turns off the transmitter starts at location $037B + (3E 05...). It's possible to insert a short delay loop, so that the + transmitter remains keyed for a few milliseconds longer. + (I haven't tried this yet.) + + +73s, Walter dg2kk@dg2kk.UUCP + + +PS: Does anyone know if WA8DED is on USENET/Bitnet/ARPANET/anynet??? + What is his email address? Please let me know. Thanks. +#! rnews 1319 +Path: alberta!mnetor!uunet!mcvax!4gl!honzo +From: honzo@4gl.UUCP (Honzo Svasek) +Newsgroups: comp.unix.xenix,comp.os.misc,comp.unix.questions,comp.unix.wizards +Subject: Re: Venix Users? +Message-ID: <253@4gl.UUCP> +Date: 11 Dec 87 18:23:50 GMT +References: <2439@sputnik.COM> +Organization: 4GL Consultants b.v., the Netherlands +Lines: 27 +Xref: alberta comp.unix.xenix:1172 comp.os.misc:341 comp.unix.questions:4773 comp.unix.wizards:5750 + +in article <2439@sputnik.COM>, dbb@tc.fluke.COM (Dave Bartley) says: +> +> The Great OS Search continues ... +> +> What about Venix? + +I am using Venix for several years now and have the folowing comments. + +1. it IS System V UNIX. + +2. It has a faster 'feel' for the interactive user than Xenix or Microport + +3. It seems to be bug free. This system is running news and I am doing most + of the development on it. I have had no problems for at least a year now, + and the system is on the air 24 hours a day. + + A few times I had to remove the -O options when compiling, but same + counts for 3B2 UNIX. + +4. Venturecom claims it to be REAL TIME. I have no experience with + REAL real-time on this system, and don't know if the venix system calls + are interruptable. + +Honzo Svasek, + +PS. Anyone out there has a way to install 2.2 on a Seagate ST4096 disk? + (on an AT) +#! rnews 1252 +Path: alberta!mnetor!uunet!mcvax!cernvax!ethz!forty2!vogel +From: vogel@forty2.UUCP (Stefan Vogel) +Newsgroups: comp.sources.bugs +Subject: bug in sush +Message-ID: <123@forty2.UUCP> +Date: 11 Dec 87 17:02:58 GMT +Reply-To: vogel@forty2.UUCP (Stefan Vogel) +Organization: Exp. Physics University Zuerich +Lines: 33 + +We found the following bug in sushperm.c of the sush distribution: + +In routine addgroup the pointer gpmem was incremented before it was used. +So, the first member of the group was never found, and the reference to +the last member lead to an illegal memory reference (NULL pointer!). + +original code: + + gpmem = gpt->gr_mem; + while(*gpmem++) { <------------------gpmem is incremented + if(!strcmp(user,*gpmem)) <---gpmem is used + ok++; + } + + /* auth failed - return */ + +corrected code: + + gpmem = gpt->gr_mem; + while(*gpmem) { + if(!strcmp(user,*gpmem++)) + ok++; + } + + /* auth failed - return */ + + Stefan Vogel, Simon Poole + Inst. for Theoretical Physics + University of Zuerich + Switzerland + + UUCP: ....mcvac!cernvax!forty2!vogel + BITNET: k524911@czhrzu1a +#! rnews 626 +Path: alberta!mnetor!uunet!mcvax!prlb2!vub!leo +From: leo@vub.UUCP (Leo Smekens) +Newsgroups: comp.sys.mac +Subject: 4th Dimension vs. dBase Mac +Keywords: 4th Dimension,dBase Mac,Macintosh +Message-ID: <506@vub.UUCP> +Date: 11 Dec 87 13:40:05 GMT +Organization: Vrije Universiteit Brussel, Brussels +Lines: 14 + +What can 4th Dimension do what dBase Mac can't? +What can dBase Mac do what 4th Dimension can't? + +Who should invest in which program? +If you don`t like answering on the net, +please mail direct to: +leo@vub.vub.uucp + +Leo Smekens +Metabolism & Endocrinology +Free University of Brussels +Laarbeeklaan 103 +B-1090 BRUSSELS +BELGIUM +#! rnews 783 +Path: alberta!mnetor!uunet!mcvax!prlb2!vub!leo +From: leo@vub.UUCP (Leo Smekens) +Newsgroups: comp.sys.mac +Subject: Latest SE's shipped +Keywords: Mac,Mac SE,Macintosh,Macintosh SE,hardware +Message-ID: <507@vub.UUCP> +Date: 11 Dec 87 13:50:46 GMT +Organization: Vrije Universiteit Brussel, Brussels +Lines: 15 + +We noticed that the last Macintosh SE's we received at our +university are equipped with a new type of mouse,and, +apparently,with another internal disk drive (at least,it +sounds differently and beeps upon activation). +What has been changed on the new Mac SE compared to the first version? +If you don't like to answer via the net,please mail direct to: + +leo@vub.vub.uucp + +Leo Smekens +Metabolism & Endocrinology +Free University of Brussels +Laarbeeklaan 103 +B-1090 BRUSSELS +BELGIUM +#! rnews 1432 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!csnjr +From: csnjr@its63b.ed.ac.uk (Nick Rothwell) +Newsgroups: rec.music.misc +Subject: Re: Ace-Screamingest Guitar Solos on Record +Keywords: guitar, flames (regrettably) +Message-ID: <826@its63b.ed.ac.uk> +Date: 11 Dec 87 13:06:12 GMT +References: <1725@s.cc.purdue.edu> <1349@saturn.ucsc.edu> <6480@ihlpa.ATT.COM> +Reply-To: nick@lfcs.ed.ac.uk (Nick Rothwell) +Organization: LFCS, University of Edinburgh +Lines: 21 + +In article <6480@ihlpa.ATT.COM> rjp1@ihlpa.ATT.COM writes: +>>C'mon people, you can't omit: +>... +>Edgar Froese - Underwater Twilight, Riding The Ray, Le Parc and +> Heartbreakers tunes, etc, etc. + +Froese's best guitar solo, by most accounts, is on Cloudburst Flight +on the Force Majeure album, back in '79. He starts with slow chords +and fingering on a 12 string acoustic, then some "power chords" (!) on +the 12 string, and then onto the electric (Fender Strat I think). +Some of the recent live work's been good, as well - Franke holding down +a rhythm, with Froese and Haslinger both firing off screaming guitar riffs. + +>Bob Pietkivitch ( e - x - p - o - s - u - r - e ) UUCP: ihnp4!ihlpa!rjp1 + +-- +Nick Rothwell, Laboratory for Foundations of Computer Science, Edinburgh. + nick%lfcs.ed.ac.uk@nss.cs.ucl.ac.uk + !mcvax!ukc!lfcs!nick +~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ ~~ +"Nothing's forgotten. Nothing is ever forgotten." - Herne +#! rnews 818 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!gvw +From: gvw@its63b.ed.ac.uk (G Wilson) +Newsgroups: comp.sys.transputer +Subject: Meiko email contact +Message-ID: <827@its63b.ed.ac.uk> +Date: 11 Dec 87 13:23:42 GMT +Reply-To: gvw@its63b.ed.ac.uk (G Wilson) +Organization: I.T. School, Univ. of Edinburgh, U.K. +Lines: 18 + +In response to several queries --- Meiko Ltd. is not +connected to any electronic mail network at present. +However, both myself and Dr. Duncan Roweth, who are +Meiko employees working on the Edinburgh Concurrent +Supercomputer Project, are connected to various networks. +I can be reached at: + + gvw@itspna.ed.ac.uk (usual) + gvw@its63b.ed.ac.uk (alternative) + +while Duncan is: + + egnp36@meiko.ed.ac.uk + +If you want more information on Meiko, please include +a telephone number and a physical mail address. + +Greg +#! rnews 733 +Path: alberta!mnetor!uunet!mcvax!ukc!dcl-cs!nott-cs!pyr1.cs.ucl.ac.uk!awylie +From: awylie@pyr1.cs.ucl.ac.uk +Newsgroups: rec.games.misc +Subject: Re: Does anyone remember Zork1? (*S +Message-ID: <42800002@pyr1.cs.ucl.ac.uk> +Date: 11 Dec 87 09:42:00 GMT +References: <22039@ucbvax.BERKELEY.EDU> +Lines: 8 +Nf-ID: #R:ucbvax.BERKELEY.EDU:-2203900:pyr1.cs.ucl.ac.uk:42800002:000:300 +Nf-From: pyr1.cs.ucl.ac.uk!awylie Dec 11 09:42:00 1987 + + +Its a looooong time since I played Zork, but I believe that you can get +to the INSIDE of the grate in the woods by which time you should have +obtained a key which will open it. This gives you an alternative entrance/ +exit to the dungeon, but is not actually much help. + Andrew + +awylie@uk.ac.ucl.cs +#! rnews 1324 +Path: alberta!mnetor!uunet!mcvax!ukc!eagle!icdoc!ivax!mst +From: mst@ivax.doc.ic.ac.uk (Martin Taylor) +Newsgroups: rec.games.trivia +Subject: Re: words to a song (old lady who swallowed a fly) +Message-ID: <148@gould.doc.ic.ac.uk> +Date: 11 Dec 87 10:55:47 GMT +References: <2170@homxc.UUCP> <12270004@hpldola.HP.COM> <1053@mtuxo.UUCP> +Sender: news@doc.ic.ac.uk +Reply-To: mst@doc.ic.ac.uk (Martin Taylor) +Organization: Dept. of Computing, Imperial College, London, UK. +Lines: 26 + +In article <1053@mtuxo.UUCP> gertler@mtuxo.UUCP (xm960-D.GERTLER) writes: + +>As I recall, the sequence is as follows (more or less): +> +> 1) Fly Perhaps she'll die. +> 2) Spider That wriggled and jiggled and tickled inside her. +> 3) Bird How absurd to swallow a bird! +> 4) Cat Imagine that, to swallow a cat! +> 5) Dog What a hog, to swallow a dog! +> 6) Horse She's dead, of course! +> +>I seem to remember a goat at about 5.5, but I don't +>recall it's associated comment. Sorry. +> + +It's "She just opened her throat, and swallowed a goat" + +Also heard at an informal church social group, this alternative ending: + + 6) Horse Not easy, of course, but she swallowed a horse + 7) Minister That finished her! + + +Martin S Taylor Department of Computing +JANET/ARPANET : mst@doc.ic.ac.uk Imperial College ++44 589 5111 X4996 LONDON SW7 2BZ +#! rnews 1060 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!aiva!ken +From: ken@aiva.ed.ac.uk (Ken Johnson) +Newsgroups: comp.edu,comp.lang.misc +Subject: Free audio tape about Logo +Message-ID: <209@aiva.ed.ac.uk> +Date: 11 Dec 87 11:33:10 GMT +Reply-To: ken@aiva.ed.ac.uk (Ken Johnson) +Followup-To: comp.lang.misc +Organization: Dept. of AI, Univ. of Edinburgh, UK +Lines: 26 +Xref: alberta comp.edu:745 comp.lang.misc:887 + + +Logotron Limited have prepared an audio tape called "Logo comes of age". + +Although it is basically a plug for the Logotron product, (it contains a +reference to the mythical "LCSI standard", for example) there is a lot +of interesting chat about how Logo is actually used. + +Playing time 45 minutes. + +Free from: + Logotron Limited, + Dales Brewery, + Gwydir Street, + CAMBRIDGE, + England CB1 2LJ + + Phone (0223) 323656 +-- + +From Ken Johnson | Phone 031-225 4464 Ext 212 + AI Applications Institute | Email k.johnson@ed.ac.uk + 80 South Bridge | + The University | + EDINBURGH, Scotland EH1 1HN | + +"Things will get worse before they get worse." +#! rnews 2806 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!hwcs!tom +From: tom@cs.hw.ac.uk (Tom Kane) +Newsgroups: comp.ai +Subject: Probability Bounds from Bayes Theory: (A Problem). +Keywords: Bayes Theorem, Probability, Expert Systems, Uncertainty +Message-ID: <1578@brahma.cs.hw.ac.uk> +Date: 11 Dec 87 14:10:21 GMT +Organization: Computer Science, Heriot-Watt U., Scotland +Lines: 65 + + +I am sending this letter out to the network to ask for solutions to a +particular problem of Bayesian Inference. Below is the text of the +problem, and at the end is the mathematical statement of the information +given. Simply, I am asking the questions: + +1) Can you find bounds on the final result. If so, how? +2) If not, why is it not possible to do so? + What is missing in the specification of the problem? +3) If you get nowhere with this problem, would you be able to solve it + if you were given the information: p(pv|t or l)=0.9? + +I am interested in the problem of providing probability bounds for events +specified in a Bayesian setting when not all the necessary conditional +probabilities are provided in setting up the problem. + +PROBLEM +~~~~~~~ +(A problem relevant to the handling of Uncertainty in Expert Systems.) +We want to know the probability of a patient having both lung cancer and +tuberculosis based on the fact that this person has had a positive reading +in a chest X-ray. We are given the following pieces of information: + +1. The probability that a person with lung cancer will have a positive + chest X-ray is 0.9. + +2. The probability that a person with tuberculosis will have a positive + chest X-ray is 0.95. + +3. The probability that a person with neither lung cancer nor tuberculosis + will have a positive chest X-ray is 0.07. + +4. In the town of interest, 4 percent of the population have lung cancer, + and three percent have tuberculosis. + +EVENTS +~~~~~~ +l = lung cancer; t = tuberculosis; pv = positive chest X-ray + +SETUP +~~~~~ +In the statement of the problem below:- + +~l means 'not l'. +~l, ~t means 'not l and not t'. +t or l means 't or l' +where 'not', 'and' , and 'or' are logical operators. +so that: p(~l, ~t) means probability( not l and not t). +Also, +p(pv|l) means the conditional probability of event pv, given event l. +PRIORS +~~~~~~ +p(l) = 0.04; p(t) = 0.03; p(~l, ~t) = 0.95 +CONDITIONALS +~~~~~~~~~~~~ +p(pv|l) = 0.9; p(pv|t) = 0.95; p(pv| ~t,~l) = 0.07 + +(You are not given p(pv| t or l) ) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Please mail all solutions or comments to me, and I will let interested parties +know what the results are. +(I will specially treasure attempts which don't use independence assumptions.) +Thanks in advance to anyone who will spend time on this problem... +Regards, +Tom Kane. +#! rnews 3109 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!hwcs!adrian +From: adrian@cs.hw.ac.uk (Adrian Hurt) +Newsgroups: sci.space +Subject: Re: SPACE Digest V8 #68 +Summary: First submarines +Message-ID: <1580@brahma.cs.hw.ac.uk> +Date: 11 Dec 87 15:43:08 GMT +References: <8712091350.AA00806@angband.s1.gov> +Organization: Computer Science, Heriot-Watt U., Scotland +Lines: 52 + +In article <8712091350.AA00806@angband.s1.gov>, ESC1361@DDAESA10.BITNET (Rupert Williams) writes: +> +> In fact the British must have +> been the most war-like nation in the world, fighting with more countries than +> anyone I can think off. Is this the reason why the English language is so +> popular ( hello America!! )???!!!! + +I assume you refer to the British Empire - prior to that, Britain (and before +the rest joined/were conquered by it, England) fought mostly against either +France, Spain or both at once. The wide domain of the English language is +directly due to the Empire, just as the wide use of Spanish throughout South +and Central America is due to the Spanish Empire. + +> I think also that ALL countries train their armies in ice and snow??!! + +Including the Arabs? :-) + +> As for the Submarine....well I dont know about that, I thought that was an +> English invention too, like the Tank and the Jet-plane??! Maybe I'm wrong??! + +There are a number of ancient submarine designs, including one which was a +rowing boat with a watertight cover! The first practical submarine was (I +believe) designed by a Mr. Holland, resident of Ireland, for use against the +Royal Navy. The Royal Navy took over the design, but regarded such concealed +warfare as ungentlemanly, and didn't make much use of them until Germany +showed the way. + +The jet plane was invented practically at the same time by Britain, Germany +and the U.S.A. Germany had the first flying jet aircraft, followed closely by +Britain. Britain would have had a jet fighter not long after the Battle of +Britain except for government intervention. Fortunately, Hitler was equally +stupid. The Nazis believed they would win the war in a couple of months, and +gave little interest to projects which would bear no short term military +results. When they did get the world's first jet fighter (the Me262) it was +pretty devastating, albeit rare, until Hitler decided that it would make a +great fighter-bomber. Two bombs were fitted under the nose, at the expense +of two cannon and much speed and agility. Fortunately, Nazi policy was "if +it doesn't work, stomp on whoever says so." The first American jet was too +late for WW2, and the first Russian jet had a captured German engine. + +> As for the NASA/Space shuttle saga, the sooner they pull their fingers out +> the better. Arianne is having a field day over this one. + +Now, for those who say "Why is this in sci.space?", read the above and apply +the lessons of history to the shuttle, Hermes, HOTOL, or whatever craft your +country should be sponsoring. + +-- + "Keyboard? Tis quaint!" - M. Scott + + Adrian Hurt | JANET: adrian@uk.ac.hw.cs + UUCP: ..!ukc!cs.hw.ac.uk!adrian | ARPA: adrian@cs.hw.ac.uk +#! rnews 1128 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!hwcs!jack +From: jack@cs.hw.ac.uk (Jack Campin) +Newsgroups: sci.space +Subject: Re: Remote Sensing Fascism +Message-ID: <1583@brahma.cs.hw.ac.uk> +Date: 11 Dec 87 17:36:36 GMT +References: <566084060.amon@H.GP.CS.CMU.EDU> +Reply-To: jack@cs.glasgow.ac.uk (Jack Campin) +Organization: PISA Project, Glesga Yoonie +Lines: 18 +Summary: + +Expires: + +Sender: + +Followup-To: + + + +[ignore the above email address and use my signature] + +>'Our' (I use the term VERY loosely since I'm not really sure which side +>they are on) people have obviously learned how to lie about the +>existance of things which are common knowledge +>PS: Is it now appropriate to address members of the DOD and the various spook +> agencies as Comrade? + +How about Right Honourable? (or have the Zircon and Spycatcher affairs not +made the news over there?) + + +-- +ARPA: jack%cs.glasgow.ac.uk@nss.cs.ucl.ac.uk +JANET:jack@uk.ac.glasgow.cs USENET: ...mcvax!ukc!cs.glasgow.ac.uk!jack +Mail: Jack Campin, Computing Science Department, University of Glasgow, + 17 Lilybank Gardens, Glasgow G12 8QQ, Scotland (041 339 8855 x 6045) +#! rnews 1825 +Path: alberta!mnetor!uunet!mcvax!ukc!its63b!hwcs!jack +From: jack@cs.hw.ac.uk (Jack Campin) +Newsgroups: rec.arts.sf-lovers +Subject: Re: Houston SF Opera +Message-ID: <1584@brahma.cs.hw.ac.uk> +Date: 11 Dec 87 19:10:14 GMT +References: <8168@ism780c.UUCP> +Reply-To: jack@cs.glasgow.ac.uk (Jack Campin) +Organization: PISA Project, Glesga Yoonie +Lines: 32 +Summary: + +Expires: + +Sender: + +Followup-To: + + + +[ignore the above email address and use my signature] + +In article <8168@ism780c.UUCP> jimh@ism780c.UUCP (Jim Hori) writes: +> +>I expected somebody to respond by now to the +>question about a SF opera being co-written +>by Philip Glass and somebody named Lessing, +> +>Any other news on this opera? + +It's "the Making Of The Representative From Planet 8", if I remember right. +This is from the announcements to a Radio 3 broadcast of Glass's new orchestral +piece "The Light" - a tone poem about the Michelson-Morley experiments. + +Incidentally, it's not the first SF opera. I heard a broadcast in New Zealand +of a Swedish opera called "Aniara", based on an epic poem about a colonizer +spaceship on its way to oblivion. I can remember neither the poet's nor the +composer's name. + +I've only read the first of Lessing's series and didn't like it much. I felt I +was being preached at (Lessing is a Sufi - I don't know whether her having +been born in Iran has anything to with that - and it shows in her more recent +writing). OK, the content of the sermon may not have been as obnoxious as +Heinlein, Tolkien or Pournelle, but it was still gratuitous in literary terms. + + +-- +ARPA: jack%cs.glasgow.ac.uk@nss.cs.ucl.ac.uk +JANET:jack@uk.ac.glasgow.cs USENET: ...mcvax!ukc!cs.glasgow.ac.uk!jack +Mail: Jack Campin, Computing Science Department, University of Glasgow, + 17 Lilybank Gardens, Glasgow G12 8QQ, Scotland (041 339 8855 x 6045) +#! rnews 1184 +Path: alberta!mnetor!uunet!mcvax!ukc!stl!stc!root44!cdwf +From: cdwf@root.co.uk (Clive D.W. Feather) +Newsgroups: rec.arts.sf-lovers +Subject: Asimov, UFO, and others +Summary: Where you can find them +Message-ID: <497@root44.co.uk> +Date: 11 Dec 87 08:48:49 GMT +Reply-To: cdwf@root44.UUCP (Clive D.W. Feather) +Organization: Root Computers Ltd, London, England +Lines: 17 + +Readers in the UK, and those elsewhere with UK contacts, may like to know... + +(1) W.H.Smiths are stocking Asimov's "Fantastic Voyage II" in hardback, +UKL10.95. + +(2) An organisation called Channel 5 Video, available at least in W.H.Smiths +and Woolworths, produces tapes of UFO, Thunderbirds, Captain Scarlet (under the +title "Captain Scarlet and the Mysterons", Stingray (yuk), and, of course, +the Prisoner. Each tape that I have seen contains two episodes of the +appropriate program. All cost less than UKL10. +What proportion of the total output of these programmes is available I can't +say, except for the Prisoner (100%). + +Warning for foreign readers: +These tapes are VHS-PAL. According to "Which?" they work in Australia, +New Zealand, Europe except France, South Africa, and the Middle East, but not +North America. +#! rnews 1077 +Path: alberta!mnetor!uunet!mcvax!ukc!stl!stc!root44!jgh +From: jgh@root.co.uk (Jeremy G Harris) +Newsgroups: comp.protocols.tcp-ip +Subject: Subnetting questions +Keywords: subnet ethernet +Message-ID: <498@root44.co.uk> +Date: 11 Dec 87 10:25:23 GMT +Organization: Root Computers Ltd., London, England +Lines: 27 + +A whole bunch of questions: + + +Does anybody run multiple subnets on a single Ethernet? + + If so, do you use subnet broadcasts or net broadcasts? + Do you find it worthwhile to use ethernet multicast for + subnet broadcasts? How do you assign the multicast addresses? + For what purposes do you still use net broadcast? + + Should redirects be provided by an inter-subnet gateway, + when both subnets are on the same Ethernet? + + +What are the semantics of 'ICMP redirect to net' in a subnettted environment? + + +Does anybody run multiple classes of subnet on a single net? + + Does the mechanism proposed in rfc950 ( ICMP broadcasts to + discover the subnet mask ) still work? Do you use it? + + +Thanks for your time + Jeremy +-- +Jeremy Harris jgh@root.co.uk +#! rnews 1006 +Path: alberta!mnetor!uunet!mcvax!ukc!stl!stc!root44!hrc63!nwh +From: nwh@hrc63.co.uk (Nigel Holder Marconi) +Newsgroups: comp.unix.wizards +Subject: Re: /dev/swap - possibility of it being a ramdisk +Summary: depends on your system ? +Keywords: /dev/swap +Message-ID: <476@hrc63.co.uk> +Date: 11 Dec 87 10:09:05 GMT +References: <712@qetzal.UUCP> <16869@topaz.rutgers.edu> +Organization: GEC Hirst Research Centre, Wembley, England. +Lines: 12 + + +I have just added some extra memory to a Sun 3. Unfortunately, it did +not increase the usable amount of virtual memory. I have been informed +(not by Sun I hasten to add), that 4.x will only allocate memory up to +the disk swap space size. Adding more memory will speed things up but will +not increase your total usable virtual memory size (this is achieved by +increasing the swap space). I was also informed that system V does not +inforce this type of restriction. + + +Nigel Holder UK JANET: yf21@uk.co.gec-mrc.u + ARPA: yf21%u.gec-mrc.co.uk@ucl-cs +#! rnews 781 +Path: alberta!mnetor!uunet!mcvax!enea!luth!jem +From: jem@sm.luth.se (Jan Erik Mostr|m) +Newsgroups: comp.sys.mac,comp.sys.mac.hypercard +Subject: Hypercard/CD-ROM +Message-ID: <902@luth.luth.se> +Date: 11 Dec 87 11:31:13 GMT +Reply-To: Jan Erik Mostr|m +Organization: University of Lulea, Sweden +Lines: 8 +Xref: alberta comp.sys.mac:10017 comp.sys.mac.hypercard:200 +UUCP-Path: {uunet,mcvax}!enea!luth.luth.se!jem + + + + +Is there someone out there who has experience with Hypercard and CD-ROM. +I would appreciate any information (and especially about Mac II/CD-ROM). +-- +Jan Erik Mostrom | {uunet,mcvax}!enea!luth!jem | Mors certa, +University of Lulea | jem@sm.luth.se | vita incerta +Sweden | jem@luth.UUCP | +#! rnews 1535 +Path: alberta!mnetor!uunet!mcvax!enea!diab!pf +From: pf@diab.UUCP (Per Fogelstrom) +Newsgroups: comp.arch +Subject: Re: Why is SPARC so slow? +Summary: Yet another "super processor". +Message-ID: <344@ma.diab.UUCP> +Date: 11 Dec 87 13:59:12 GMT +References: <1078@quacky.UUCP> <8809@sgi.SGI.COM> <6964@apple.UUCP> +Reply-To: pf@ma.UUCP (Per Fogelstrom) +Organization: Diab Data AB, Taby, Sweden +Lines: 16 + +Well, the history repeats once again. A new RISC chip is launched and peopels +expectations reaches new "high scores". A few years ago there was another risc +chip set brougth to the market, called the Clipper. This processors performence +was climed to sweep all competitors off the sceene. Often compared to the +DEC 8x00 computers. For this chip set the picture has cleared now. The perfor- +mence range is not much more than can be achived with a 16-20 Mhz 68020. The +most i have seen of the 33Mhz versions is one running at room temprature. +Intergraph is one of the companys who is still using the Clipper (They recently +bought the rights for the chip set from NS/Fairchild) . From what i recall they +throw out the NS32032 for the Clipper. Well they could have had 2-3 times the +clipper performance with the NS32532 today. And they called the buy a bargin ! +It's not suprising that the MIPS 2000 gives most power/Mhz, The architecture has +evolved during many years, without a hard pressure from the marketing such as +'We must have it NOW!!!'. (John Mashey mayby has another opinion, only my guess) + +SO: Why is everybody so suprised ????! +#! rnews 1169 +Path: alberta!mnetor!uunet!mcvax!mhres!jv +From: jv@mhres.mh.nl (Johan Vromans) +Newsgroups: comp.unix.questions +Subject: Re: Finding Files +Summary: looking everywhere +Message-ID: <1503@mhres.mh.nl> +Date: 12 Dec 87 16:08:12 GMT +References: <205700003@prism> <4441@ihlpg.ATT.COM> +Organization: Multihouse N.V., The Netherlands +Lines: 21 + +In article <205700003@prism> billc@prism.UUCP writes: +> +> Right now, to find a file somewhere under my current directory, +> I use the following alias: +> +> alias where "find \$cwd -name \!* -exec echo {} \;" +> .. etc .. + +On our systems, a small cron script executes every night the following +command: + + find / -print > /dirfile + +Finding a file somewhere can be done by grepping in the /dirfile. +Of course, the contents of /dirfile are not really up-to-date, but this is +just a minor drawback. "find" on the whole system (including mounted disks) +takes more than an hour, a grep in /dirfile much less than a minute. +-- +Johan Vromans | jv@mh.nl via European backbone +Multihouse N.V., Gouda, the Netherlands | uucp: ..{uunet!}mcvax!mh.nl!jv +"It is better to light a candle than to curse the darkness" +#! rnews 904 +Path: alberta!mnetor!uunet!mcvax!mhres!jv +From: jv@mhres.mh.nl (Johan Vromans) +Newsgroups: comp.os.vms +Subject: Re: Are VMS and VAX synonymous? +Summary: NO +Message-ID: <1504@mhres.mh.nl> +Date: 12 Dec 87 16:55:05 GMT +References: <8712111910.AA18210@ucbvax.Berkeley.EDU> +Organization: Multihouse N.V., The Netherlands +Lines: 11 + +In article <8712111910.AA18210@ucbvax.Berkeley.EDU> "ERI::SMITH" writes: +>But someone who thinks VAX and VMS are synonymous +>MAY POSSIBLY also be expressing a philosophical stance. + +The only thing you can do between "#ifdef vax" and its corresponding "#endif" +is conclude that you are running on a big-endian machine .... + +-- +Johan Vromans | jv@mh.nl via European backbone +Multihouse N.V., Gouda, the Netherlands | uucp: ..{uunet!}mcvax!mh.nl!jv +"It is better to light a candle than to curse the darkness" +#! rnews 691 +Path: alberta!mnetor!uunet!mcvax!enea!tut!jh +From: jh@tut.fi (Juha Hein{nen) +Newsgroups: comp.lang.scheme +Subject: Re: Request for MacScheme source for SCOOPS +Message-ID: <2108@korppi.tut.fi> +Date: 12 Dec 87 07:32:39 GMT +References: <8712101554.AA15940@ucbvax.Berkeley.EDU> +Reply-To: jh@korppi.UUCP (Juha Hein{nen) +Organization: Tampere University of Technology, Finland +Lines: 10 + +MacScheme doesn't have enviroments (atleast my version doesn't). It +would be straightforward to port SCOOPS if somebody first provides +environments. The hacks provided with MacScheme distribution are not +enough. + +-- + Juha Heinanen + Tampere Univ. of Technology + Finland + jh@tut.fi (Internet), tut!jh (UUCP) +#! rnews 643 +Path: alberta!mnetor!uunet!mcvax!enea!diab!pf +From: pf@diab.UUCP (Per Fogelstrom) +Newsgroups: comp.arch +Subject: Re: Zilog Z320 32-bit chip +Keywords: 80,000 vaporware model +Message-ID: <345@ma.diab.UUCP> +Date: 12 Dec 87 11:15:23 GMT +References: <1911@ho95e.ATT.COM> <9071@utzoo.UUCP> <3521@aw.sei.cmu.edu> <485@PT.CS.CMU.EDU> +Reply-To: pf@ma.UUCP (Per Fogelstrom) +Organization: Diab Data AB, Taby, Sweden +Lines: 3 + +The Z80,000 was put on market just about 8 months ago. It newer reached the +target specification (e.g. clock speed) and the performence was not impressive. +It has some nice things, but as someone pointed out, to late .......... +#! rnews 884 +Path: alberta!mnetor!uunet!mcvax!diku!daimi!erja +From: erja@daimi.UUCP (Erik Jacobsen) +Newsgroups: comp.lang.modula2 +Subject: Re: Modula II on IBM PC with HALO graphics +Keywords: Modula IBM HALO graphics +Message-ID: <1253@daimi.UUCP> +Date: 12 Dec 87 13:13:50 GMT +References: <17237@glacier.STANFORD.EDU> +Reply-To: erja@daimi.UUCP (Erik Jacobsen) +Organization: DAIMI: Computer Science Department, Aarhus University, Denmark +Lines: 10 + +jbn@glacier.STANFORD.EDU (John B. Nagle) asks in <17237@glacier.STANFORD.EDU> +> Some questions on Logitec Modula II: +> +> 1. Are subranges assigned space appropriately? In particular, +> does 0..255 occupy only one byte? + +No, subranges occupy the same amount of space as the type they are +a subrange of. E.g. 0..255 will occupy two bytes. You may use a +CHAR or a BYTE, and convert to and from CARDINAL everytime you need +to do some caluculations. +#! rnews 869 +Path: alberta!mnetor!uunet!mcvax!unido!tub!stx +From: stx@tub.UUCP (Stefan Taxhet) +Newsgroups: comp.text,comp.sources.wanted +Subject: MS-WORD to Q-ONE +Keywords: MS-WORD Q-ONE DCA +Message-ID: <319@tub.UUCP> +Date: 11 Dec 87 18:23:35 GMT +Organization: Technical University of Berlin, Germany +Lines: 19 +Xref: alberta comp.text:1346 comp.sources.wanted:2722 + + +We're looking for a document conversion program. +It should translate MS-Word- to Q-ONE-documents. + +Q-ONE offers conversions to several formats as: +Fortune:Word, Wang, IBM's DCA (RFT,FFT) +Therefor programs to interchange documents between +MS-Word and these format would also help us. + +Thanks in advance + +Stefan Taxhet, +Communications and Operating Systems Research Group +Technical University of Berlin + +UUCP: ...!pyramid!tub!stx (From the US) + ...!mcvax!unido!tub!stx (From Europe) + +BITNET: stx@db0tui6.BITNET +#! rnews 1235 +Path: alberta!mnetor!uunet!mcvax!unido!rmi!kkaempf +From: kkaempf@rmi.UUCP (Klaus Kaempf) +Newsgroups: comp.sys.amiga +Subject: Breaking the 54MB limit on HardDisks +Keywords: BitMap, Blocksize, filehandler.h +Message-ID: <821@rmi.UUCP> +Date: 12 Dec 87 12:19:32 GMT +Reply-To: kkaempf@rmi.UUCP (Klaus Kaempf) +Organization: RMI Net, Aachen, W.Germany +Lines: 19 + + + +Well, maybe that i've overlooked something really important, but i don't +see the 54MB limit with the AmigaDOS. +About a yaer ago, when there was no mount command, somebody from CATS +posted a sample device driver that mounted itself. It set up a device +structure which described the layout of the device. This structure is +now documented in dos/filehandler.h. One field in this structure holds +the number of longwords per block of this device. This is always set +to 128, giving 512 Bytes per Block. +Now, if i set this to 256 (1024 Bytes per Block), i should be able to +increase the disk limit to 108MB. +Apparently, AmigaDOS supports larger blocksizes. Just have a look into +the AmigaDOS Manual from Bantam. All block-layouts are described relative +to a 'SIZE', nowhere is said that SIZE is fixed to 128 ! + +So where is the problem ??? (Please, send no flames, only facts !) + +Klaus +#! rnews 2545 +Path: alberta!mnetor!uunet!mcvax!botter!ast +From: ast@cs.vu.nl (Andy Tanenbaum) +Newsgroups: comp.os.minix +Subject: Getting rid of _cleanup (finally) +Message-ID: <1783@botter.cs.vu.nl> +Date: 13 Dec 87 11:56:59 GMT +Reply-To: ast@cs.vu.nl (Andy Tanenbaum) +Organization: VU Informatica, Amsterdam +Lines: 97 + +There was a lot of discussion about how to get rid of my calls to _cleanup +earlier. Here is the solution that I finally adopted. The following commands +should do the job. + cc -c -LIB exit.c putc.c + ar r /usr/lib/libc.a exit.c putc.c + ar x /usr/lib/libc.a cleanup.s + ar d /usr/lib/libc.a cleanup.s + ar bfork.s /usr/lib/libc.a cleanup.s + +This requires the new archiver posted a while back (for the b option). +It also assumes that putting cleanup before fork.s will include cleanup.s +after exit.s and putc.s (check this). + +Andy Tanenbaum (ast@cs.vu.nl) + + +: This is a shar archive. Extract with sh, not csh. +: This archive ends with exit, so do not worry about trailing junk. +: --------------------------- cut here -------------------------- +PATH=/bin:/usr/bin +echo Extracting \e\x\i\t\.\c +sed 's/^X//' > \e\x\i\t\.\c << '+ END-OF-FILE '\e\x\i\t\.\c +X#include "../include/lib.h" +X +XPUBLIC int (*__cleanup)(); +X +XPUBLIC int exit(status) +Xint status; +X{ +X if (__cleanup) (*__cleanup)(); +X return callm1(MM, EXIT, status, 0, 0, NIL_PTR, NIL_PTR, NIL_PTR); +X} ++ END-OF-FILE exit.c +chmod 'u=rw,g=r,o=r' \e\x\i\t\.\c +set `sum \e\x\i\t\.\c` +sum=$1 +case $sum in +11315) :;; +*) echo 'Bad sum in '\e\x\i\t\.\c >&2 +esac +echo Extracting \p\u\t\c\.\c +sed 's/^X//' > \p\u\t\c\.\c << '+ END-OF-FILE '\p\u\t\c\.\c +X#include "../include/stdio.h" +X +Xextern int (*__cleanup)(); +Xextern int _cleanup(); +X +Xputc(ch, iop) +Xchar ch; +XFILE *iop; +X{ +X int n, +X didwrite = 0; +X +X if (testflag(iop, (_ERR | _EOF))) +X return (EOF); +X +X if ( !testflag(iop,WRITEMODE)) +X return(EOF); +X +X if ( testflag(iop,UNBUFF)){ +X n = write(iop->_fd,&ch,1); +X iop->_count = 1; +X didwrite++; +X } +X else{ +X __cleanup = _cleanup; +X *iop->_ptr++ = ch; +X if ((++iop->_count) >= BUFSIZ && !testflag(iop,STRINGS) ){ +X n = write(iop->_fd,iop->_buf,iop->_count); +X iop->_ptr = iop->_buf; +X didwrite++; +X } +X } +X +X if (didwrite){ +X if (n<=0 || iop->_count != n){ +X if (n < 0) +X iop->_flags |= _ERR; +X else +X iop->_flags |= _EOF; +X return (EOF); +X } +X iop->_count=0; +X } +X return(ch & CMASK); +X} +X ++ END-OF-FILE putc.c +chmod 'u=rw,g=r,o=r' \p\u\t\c\.\c +set `sum \p\u\t\c\.\c` +sum=$1 +case $sum in +49878) :;; +*) echo 'Bad sum in '\p\u\t\c\.\c >&2 +esac +exit 0 +#! rnews 1120 +Path: alberta!mnetor!uunet!husc6!mit-eddie!uw-beaver!cornell!svax!beck +From: beck@svax.cs.cornell.edu (Micah Beck) +Newsgroups: comp.windows.x +Subject: Document previewing using Xps +Message-ID: <1898@svax.cs.cornell.edu> +Date: 14 Dec 87 13:25:54 GMT +Reply-To: beck@svax.cs.cornell.edu (Micah Beck) +Distribution: comp +Organization: Cornell Univ. CS Dept, Ithaca NY +Lines: 18 + +In article <6224@jade.BERKELEY.EDU> shipley@web1d.berkeley.edu () writes +on the subject of troff previewing under X: + +>The other thing to try is some version of TROFF which can speak PostScript(tm) +>which you can then feed through one of the several Xps programs floating +>around -- these are PostScript(tm) interpreter/previewers for Xwindows. + +I've not been very successful in getting Goswell's Xps to preview documents. +The Postscript file generated from TeX DVI files by dvi2ps and from Ditroff +files by the Transcript psdit program both cause it to choke, although in +different ways. + +Is anyone using Xps successfully for TeX or Ditroff previewing? Is there some +trick? + +Micah Beck +Cornell Dept of Computer Science +beck@svax.cs.cornell.edu +#! rnews 1332 +Path: alberta!mnetor!uunet!mcvax!botter!ast +From: ast@cs.vu.nl (Andy Tanenbaum) +Newsgroups: comp.os.minix +Subject: Re: Hard disk partitions? +Message-ID: <1784@botter.cs.vu.nl> +Date: 13 Dec 87 12:13:15 GMT +References: <5500001@ucf-cs.ucf.edu> +Reply-To: ast@cs.vu.nl (Andy Tanenbaum) +Organization: VU Informatica, Amsterdam +Lines: 21 + +In article <5500001@ucf-cs.ucf.edu> tony@ucf-cs.ucf.edu writes: +>If partition 1 is set up for DOS and 2 for Minix, with #2 mounted under +>/usr, Minix crashes unpredictably. + +One thing to remember is that the partition size for partition 1 is one +smaller than for partition 2. + +Another possibility is that the MINIX fdisk and the DOS fdisk don't agree +on the meaning of the partition table. If everyone would create their +partitions from lowest cylinder to highest there would be no ambiguity. +However, if the order in the partition table is different from the cylinder +order, there are at least three interpretations. + 1. Table slot 1 is partition 1 + 2. Innermost cylinder is partition 1 + 3. Outermost cylinder is partition 1 +I believe that the combination of MINIX, DOS, XENIX and Microport together +exhaust the entire list of possibilities. I don'know if this is related +to your problem (which I otherwise can't understand), but it is worth +keeping in mind. + +Andy Tanenbaum (ast@cs.vu.nl) +#! rnews 873 +Path: alberta!mnetor!uunet!mcvax!lambert +From: lambert@cwi.nl (Lambert Meertens) +Newsgroups: sci.math +Subject: Re: Fixed Points +Message-ID: <146@piring.cwi.nl> +Date: 13 Dec 87 12:09:54 GMT +References: <2269@ihuxv.ATT.COM> +Organization: CWI, Amsterdam +Lines: 14 + +In article <2269@ihuxv.ATT.COM> eklhad@ihuxv.ATT.COM (K. A. Dahlke) writes: +) If a continuous function maps the unit square into itself, must it have a +) fixed point? [...] +) I seem to remember there is some theorem in topology, +) without resorting to snakes, that says there is always a fixed point +) whenever a closed region in a metric space is continuously mapped into itself. + +Brouwer's Fixed Point Theorem states that a continuous mapping of an n-cube +into itself has a fixed point. This extends, obviously, to any region +homeomorphic to an n-cube. + +-- + +Lambert Meertens, CWI, Amsterdam; lambert@cwi.nl +#! rnews 1075 +Path: alberta!mnetor!uunet!mcvax!unido!rmi!zentrale +From: zentrale@rmi.UUCP (RMI Net) +Newsgroups: rec.ham-radio +Subject: Re: My PC generates RFI +Message-ID: <822@rmi.UUCP> +Date: 13 Dec 87 10:03:45 GMT +References: <12354296992.20.QUALCOMM@A.ISI.EDU> +Reply-To: dl3no@rmi.UUCP (Rupert Mohr) +Organization: RMI Net, Aachen, W.Germany +Lines: 21 + +In article <12354296992.20.QUALCOMM@A.ISI.EDU> QUALCOMM@A.ISI.EDU (Franklin Antonio) writes: +: > I'd like to know of some ways to reduce interference to my... +: +: All PCs generate RFI to some degree. In general, the "clones" are worse +: than the brand name "IBM", "COMPAQ", etc. The Macintosh is relatively +: quiet. +: + +In general: I would not believe that... (But it may be, that some +IBM's are as quiet as a clone...) + +We have a good mixture of various PCs here... + +Regarding my recent posting on RFI of my PK-232: +The PK-232 was innocent. It was the old power supply which interfered +exactly on 80m and 40m with S9 and 20m with S6... + +-rm + +P.S. nevertheless: PC's nowadays are much more quiet than those times +of TRS-80 (sigh). +#! rnews 1629 +Path: alberta!mnetor!uunet!mcvax!unido!rmi!zentrale +From: zentrale@rmi.UUCP (RMI Net) +Newsgroups: rec.ham-radio +Subject: Re: some SWL questions +Message-ID: <823@rmi.UUCP> +Date: 13 Dec 87 10:18:23 GMT +References: <38c9774f.44e6@apollo.uucp> <871201110223.1.ED@BLACK-BIRD.SCRC.Symbolics.COM> +Reply-To: dl3no@rmi.UUCP (Rupert Mohr) +Organization: RMI Net, Aachen, W.Germany +Lines: 38 + +In article <871201110223.1.ED@BLACK-BIRD.SCRC.Symbolics.COM> Ed@MEAD.SCRC.SYMBOLICS.COM (Ed Schwalenberg) writes: +: +: Date: 30 Nov 87 15:30:00 GMT +: From: apollo!nelson_p%apollo.uucp@eddie.mit.edu +: +: Is there a detailed single-source of info on what I might +: hear as I tune around the bands? The much vaunted World +: Radio and TV Handbook just covers broadcasting, which I +: have little interest in. +: +: The second source is the Klingenfuss Guide to Utility Stations. +: This is harder to come by, but is advertised in RDI. + +I just got the 6th edition (1988), which is VERY good. You can +get it directly : + +Klingenfuss, Guide to Utility Stations, 6th Edition + +Klningenfuss Publications +Hagenloher Str. 14 +D-7400 Tuebingen +Fed.Rep.Germany +Tel. (+41) 7071 62830 + +Price: DM 60 (abt. $ 35) maybe plus handling. +They are very fast! I received it two days after ordering by telephone. +They also have an quarterly update Service. + +You find a complete listing sorted by frequency an different listings +sorted by different services: +press by time, +fax alphebetically with time schedule + +addresses, codes, commercial call signs, telegram formats etc. + +ALL in English, 500 pages with correct entries...... + +Rupert +#! rnews 1503 +Path: alberta!mnetor!uunet!husc6!cmcl2!rutgers!orstcs!mist!koff +From: koff@mist.cs.orst.edu (Caroline N. Koff) +Newsgroups: rec.arts.startrek +Subject: Troi's outfit +Message-ID: <1501@orstcs.CS.ORST.EDU> +Date: 14 Dec 87 13:35:02 GMT +References: <1008@percival.UUCP> <275@hi3.aca.mcc.com.UUCP> <2032@charon.unm.edu> <2432@homxc.UUCP> <1987Dec12.230124.16416@gpu.utcs.toronto.edu> <2216@nicmad.UUCP> +Sender: netnews@orstcs.CS.ORST.EDU +Reply-To: koff@mist.UUCP (Caroline N. Koff) +Distribution: na +Organization: Oregon State Universtiy - CS - Corvallis, Oregon +Lines: 17 + +If people are noticing and mentioning about Yar's breasts, why not +also mention about Troi's low cut outfit!! Why does it need to be +so low cut that it shows her crevice? Who is she trying to impress? +Do you think that the women in the future, working with men, will be +trying to dress sexy? If so, what about the men? Why not let them +show off their body too to make things even? I think that the +producers, or whoever is in charge of outfits, and character development +is making a contemporary decision regarding the issue of how people +will dress in the future. I.e. he/she thinks that female will be +trying to attract males' attention by bringing out her femininity, +but not vice versa, which is the current social behavior. +Or, perhaps the producers are just being comformists with bunch of +other tv shows + movie producers by keeping females attractive +towards men... + +--Caroline Koff +koff!cs.orst.edu@cs.net.relay +#! rnews 1028 +Path: alberta!mnetor!uunet!mcvax!hafro!krafla!frisk +From: frisk@rhi.is (Fridrik Skulason) +Newsgroups: comp.sys.ibm.pc +Subject: Identifying VGA +Message-ID: <100@krafla.rhi.is> +Date: 13 Dec 87 11:39:28 GMT +Reply-To: frisk@rhi.UUCP (Fridrik Skulason) +Organization: University of Iceland (RHI) +Lines: 19 + +In the november issue of Dr.Dobb's Journal there is an article on how to +identify the video adaptor in your PC. They cover EGA,CGA,MDA,Compaq and +Hercules(mono). + +What I need is information on how to find out if a VGA (or a PGA) adaptor +is installed. + +Also - can someone tell me how to obtain the current cursor position directly +from these adaptors. That is - I need the location of the 6845 registers. + +The reason I can not use the INT10 function provided is that my program has +to work with some TSR programs that access the hardware directly. + +Thanks... +-- + Fridrik Skulason University of Iceland + UUCP frisk@rhi.uucp BIX frisk + + This line intentionally left blank ................... +#! rnews 805 +Path: alberta!mnetor!uunet!husc6!cmcl2!rutgers!orstcs!mist!koff +From: koff@mist.cs.orst.edu (Caroline N. Koff) +Newsgroups: rec.arts.startrek +Subject: Requesting ST:TOS episode directors and writers guide +Message-ID: <1502@orstcs.CS.ORST.EDU> +Date: 14 Dec 87 13:38:05 GMT +References: <1008@percival.UUCP> <275@hi3.aca.mcc.com.UUCP> <2032@charon.unm.edu> <2432@homxc.UUCP> <1987Dec12.230124.16416@gpu.utcs.toronto.edu> <2216@nicmad.UUCP> +Sender: netnews@orstcs.CS.ORST.EDU +Reply-To: koff@mist.UUCP (Caroline N. Koff) +Distribution: na +Organization: Oregon State Universtiy - CS - Corvallis, Oregon +Lines: 6 + +Has anybody ever posted or have a complete list of directors and writers +for each of the ST:TOS episodes? If so, may I have a copy? Thanks in +advance. + +--Caroline Koff +koff!cs.orst.edu@cs.net.relay +#! rnews 1125 +Path: alberta!mnetor!uunet!mcvax!enea!sems!olof +From: olof@sems.SE (Olof Backing) +Newsgroups: comp.emacs +Subject: Problems with uEmacs 3.9e and OS-9/68K C. +Message-ID: <207@sems.SE> +Date: 13 Dec 87 13:00:39 GMT +Organization: Sems AB, Stockholm, Sweden +Lines: 32 + +I have a problem when I try to compile the latest version of +microEmacs, ie. 3.9e. The problem occurs in file 'bind.c' at lines +602, 609 and 642, 650 respectively. + +It's the following lines that causes the error: + +600: int (*getbind(c))() +601: +602: int c; +603: +604: { + +The compiler reports an error at line 602 with 'not an argument'. The +same thing happens at line 642; + +639: int (*fncmatch(fname))() +640: +641: +642: int fname; +643: +644: { + +Since my experiences aren't the very best i C sofar, I would like to +get some hints on what to do. Maybe Kim Kempf at Microware has the +answer for me. Feel free to overwelm me with hints. Until then (when I +recieve the hints...), CU all! + + +-- + ADDRESS: Havrevagen 14, S-175 43 Jarfalla, Sweden + PHONE : (46) 758 33941, 35516 home + UUCP : ...{uunet,mcvax,ukc,unido}!enea!sems!olof +#! rnews 1708 +Path: alberta!mnetor!uunet!husc6!yale!dwald +From: dwald@yale-zoo-suned..arpa (David Wald) +Newsgroups: rec.arts.startrek +Subject: Re: Hide & Q notes and comments and notes and comments and....< +Date: 14 Dec 87 04:29:38 GMT +References: <19962@yale-celray.yale.UUCP> <17300072@silver> <1838@leadsv.UUCP> +Sender: root@yale.UUCP +Reply-To: dwald@yale-zoo-suned.UUCP (David Wald) +Distribution: na +Organization: Yale University Computer Science Dept, New Haven CT +Lines: 27 + +In article <1838@leadsv.UUCP> lilly@leadsv.UUCP (Harriette Lilly) writes: +> +>In article <17300072@silver>, sl131008@silver.bacs.indiana.edu writes: +>> /* Written 7:19 pm Dec 7, 1987 by sl131008@silver.UUCP in silver:rec.arts.startrek */ +>> /* ..ditto x 7..... +>> /* Written 9:12 pm Dec 6, 1987 by dwald@yale in silver:rec.arts.startrek */ +>> /* ---------- "Re: Hide & Q notes and comments <> In article <2328@homxc.UUCP> scott@homxc.UUCP (Scott Berry) writes: +... +>> David Wald dwald@yale.UUCP +... +>> /* End of text from silver:rec.arts.startrek */ +>> /* ditto x 8 +> +> +> Ummm, are you lost?... + +I was a bit puzzled by this too, since I didn't think my article so +brilliant that anyone would want to repost it eight times. If anyone +finds out what happened, could they please send me mail? + + +We now return you to your regularly scheduled nonsense... +============================================================================ +David Wald dwald@yale.UUCP + waldave@yalevmx.bitnet +============================================================================ +#! rnews 761 +Path: alberta!mnetor!uunet!mcvax!enea!sems!olof +From: olof@sems.SE (Olof Backing) +Newsgroups: rec.games.misc +Subject: Larn at dungeon level 10. +Message-ID: <208@sems.SE> +Date: 13 Dec 87 18:51:50 GMT +Organization: Sems AB, Stockholm, Sweden +Lines: 12 + +Well folks, I've reached to master warlord (lvl 17, ~550000 Exp). To +my great dis-something, I haven't found any ladder down to level 11. +Somewhere back in my human brain, I recall that I've read something +about how to further down in the dungeon. What do I do ?!. Please give +me a hint. + + +-- +WHOAMI : Olof Backing ! +WHERE : Havrevagen 14, S-175 43 Jarfalla, Sweden ! +PHONE : + (46) 758 33941, 35516 ! +UUCP : ...{uunet,mcvax,ukc,unido}!enea!sems!olof ! +#! rnews 1631 +Path: alberta!mnetor!uunet!husc6!yale!dwald +From: dwald@yale-zoo-suned..arpa (David Wald) +Newsgroups: rec.arts.startrek +Subject: Re: Terralian Ship in "Haven" +Keywords: ST:TNG +Message-ID: <20253@yale-celray.yale.UUCP> +Date: 14 Dec 87 04:36:39 GMT +References: <5243@zen.berkeley.edu> <9615@ufcsv.cis.ufl.EDU> +Sender: root@yale.UUCP +Reply-To: dwald@yale-zoo-suned.UUCP (David Wald) +Distribution: na +Organization: Yale University Computer Science Dept, New Haven CT +Lines: 19 + +In article <9615@ufcsv.cis.ufl.EDU> jco@beach.cis.ufl.edu () writes: +>In article <5243@zen.berkeley.edu> timlee@cory.Berkeley.EDU (Timothy J. Lee) writes: +>>Did anyone think that the Terralian ship was pretty big for something that +>>was built by a group of people whose technology approximated late 20th +>>century Earth? +> +>It was my understanding from the show that the people of 20th century +>earth could build a virus that could wipe out a planet. This did NOT +>mean that they (the Terralians) where of the 20th century tech level. + +There was more to the 20th century reference than that, however. +Dr. Crusher made the point that, since they were only at the technology +level of ~20th century Earth, it was easy for the disease to get out of +control and spread over the planet. The implication was that if they +were more advanced the disease would not have wiped out the entire world. +============================================================================ +David Wald dwald@yale.UUCP + waldave@yalevmx.bitnet +============================================================================