Lazy field should only be applicable to full message, whose `mergeFrom(Message)` supports merging from a different message but with the same descriptor. Therefore, this change abandons the default instance check in the lazy field impl.
Also note that InternalLazyField currently belongs to the lite runtime, so direct access to descriptors is infeasible, thus depending on the mergeFrom overrides to invoke the full merging method.
PiperOrigin-RevId: 926112976
We have to break up the descriptors into multiple strings because of the 64k cap on string literals and then join them back up at runtime. This change we avoid building the intermediate joined string before extracting it back as byte[], and instead pre-allocate the total byte[] array first and turns each 64kb segment to byte[] and uses System.arraycopy to copy them directly.
This avoids some pointless work, but also removes garbage copies. Note that its a garbage copy that should be escape-analysis by the JVM and so should be very cheap, but still cheaper not to.
PiperOrigin-RevId: 925969602
By outlining the exception handling path, this notably shrinks this method to below 35 bytes in .class bytecode representation, which is one of the magic arbitrary limits that influences JIT inlining behaviors.
PiperOrigin-RevId: 925530732
In `AbstractMessageLite.Builder.mergeFrom(MessageLite)`, it either goes into (full runtime) `AbstractMessage` which supports merge from a different instance but the same *descriptor* type, or into the lite runtime path which should not support it in the absence of descriptors .e.g in case of `DynamicMessage`.
PiperOrigin-RevId: 924941126
Today, equals() causes all lazy extensions to be parsed before the equality check starts. After this change, equals() will try to avoid hitting lazy parsing if possible
Notably:
- When the count of extensions don't match, or if "which extensions are set" don't match, we can return false before looking at any extension values.
- When actually comparing extensions, it goes iteratively which means it'll only parse the extensions as it goes; if it finds any one ext which is not-equal it will will bail out and not parse any of the rest.
Note that when the case where two messages are equal, this unavoidably still cause all extensions to parse since it has to examine all fields.
To achieve this does require adding a protected `extensionsEquals()` method for the gencode to call, since the current base class API isn't otherwise powerful enough to do so.
PiperOrigin-RevId: 924903438
This exception is put at the spot where we will typically first notice that we cannot continue due to being on Lite gencode flows and that sun.misc.Unsafe is unavailable.
Add corresponding note on lite.md that it requires sun.misc.Unsafe
PiperOrigin-RevId: 919813377
These paths are 100% unreachable today via our public APIs (they're only reachable by our tests which pierce-the-veil of the package-protected api). The only entry point is via GeneratedMessageLite, and this code already checks for GeneratedMessageLite types and always uses the Lite types.
This behavior means that even if someone in OSS uses Lite gencode on Full runtime this path is never reachable today.
PiperOrigin-RevId: 917409414
The only field order used in practice is ASCENDING. DESCENDING was added for the experimental JavaProto runtime which is now defunct.
PiperOrigin-RevId: 917271023
When toBuilder() is called on an immutable ExtendableMessage whose extensions field is the singleton FieldSet.emptySet(), generated code calls ExtendableBuilder.mergeExtensionFields(other).
Because other.extensions is FieldSet.emptySet() (which is not null), mergeExtensionFields previously invoked ensureExtensionsIsMutable(), which unnecessarily allocated a brand new temporary FieldSet.Builder and SmallSortedMap. Later, when build() executed, FieldSet.Builder.buildImpl recognized that the builder's map was empty and correctly returned the singleton FieldSet.emptySet().
By checking !other.extensions.isEmpty() before mutating the builder, we completely eliminate these wasted temporary allocations.
PiperOrigin-RevId: 916174058
The refactor shrinks the method's bytecode to be below 35 bytes instead of above, and 35 bytes is one of the relevant default threshold conditions for inlining.
The theory behind this change is to improve a very hot case of:
- gencode always calls readMessage with the concrete builder types of fields
- That means readMessage itself is called with all reachable Builder types
- If readMessage is not inlined, it will be highly megamorphic (= too many different types for the JIT to try to monomorphize it), which makes it true dynamic dispatch at runtime. But if it is inlined instead, then it will trivially dispatch to one specific Builder type instead.
It isn't so easy to benchmark how much this will impact the fleet in reality, but since this change is otherwise very benign we can just make the change and then post-hoc see if it looks like it improved things.
PiperOrigin-RevId: 915625779
The last usages of UnsafeUtil from the UnsafeProcessor were removed in a previous change.
The name 'UnsafeProcessor' was already a bit stale, and is now very stale. But not renaming it now to reduce churn if this has to be rolled back. After this sticks, we should come back and do a pass to rename them to 'StandardProcessor' versus 'AndroidOnDeviceProcessor'
PiperOrigin-RevId: 914424161
This use was somewhat buggy: it would have broken if used on a non-Direct Buffer, or if it was executed in a context where sun.misc.Unsafe was not available.
This path is empirically extremely rarely reached, and so we will just use the 'natural' API as part of getting off of the terminally deprecated sun.misc.Unsafe
PiperOrigin-RevId: 914423267
This change effectively has the runtime behavior of hardcoding HAS_UNSAFE_ARRAY_OPERATIONS to false, which was measured as a net performance improvement after the recent loop unrolling changes.
This is one of the last steps towards no longer using the terminally deprecated sun.misc.Unsafe from JavaFull.
PiperOrigin-RevId: 914398839
This change is in preparation for a subsequent change which will disable sun.misc.Unsafe usage. Simply turning off s.m.Unsafe would result in some small but measurably regressions; unrolling these loops by contrast provides a significant speedup, which should more than outweighs any regression we will get from the disabling Unsafe. This will make it easier for us to roll out the change to stop using Unsafe without problems from regression.
PiperOrigin-RevId: 913906295
Right now we delegate negative int32s to the int64 encoder, which has to still figure out how many bytes its going to take, which is wasted work since we already checked the top bit and know it will be exactly 10 byte varint.
PiperOrigin-RevId: 913824216
This was changed in 2020 to an unrolled loop with Unsafe but was rolled back after external reports that it was a regression (https://github.com/protocolbuffers/protobuf/issues/6977). However, not only has JIT behavior evolved in that amount of time, the main external claim appears to have been that the `int position = this.position;` local variable optimization was lost at that time, with one the first comments on that issue noting that restoring that also fixed the regression.
Today, all available benchmarks suggest that this should be good change to make (with the local variable position optimization included), ~twice as fast for direct Uint32 writes usages.
PiperOrigin-RevId: 913780730
This per-processor path static method implementation in a previous change.
This is incremental progress towards migrating off of sun.misc.Unsafe which is terminally deprecated.
PiperOrigin-RevId: 912643167
This path uses sun.misc.Unsafe which is terminally deprecated. Although it is theoretically 'good' that we can optimize a Direct buffer faster than the basic APIs, this is empirically never reached in all of Google's internal use. With sun.misc.Unsafe going away and no easy replacement, this is turning down the corner case optimization.
PiperOrigin-RevId: 912177627
In the cases where we historically didn't enforce a depth limit (C++ and Python) document that the default behavior having no depth limit is an intentional decision.
PiperOrigin-RevId: 903216142
## Fix unguarded recursion DoS in Protobuf Lite skipField
This PR addresses a missing recursion depth limit check in `ArrayDecoders.skipField()` that causes a `StackOverflowError` and subsequent Denial of Service.
### Vulnerability Context
The initial patch for CVE-2024-7254 correctly added recursion limits to the standard `CodedInputStream` and `decodeUnknownField` in the Lite fast-path array decoders. However, `ArrayDecoders.skipField()` handles the `WIRETYPE_START_GROUP` recursively without checking or incrementing `Registers.recursionDepth`.
This omission exposes any Protobuf Java Lite application that accepts unauthenticated binary input to a remote DoS via simple stack overflow payloads.
**The bug is reachable via two independent paths from Public APIs:**
1. **Legacy Vectors:** `message_set_wire_format`. A payload containing thousands of nested unknown groups inside a MessageSet forces `MessageSetSchema.mergeFrom` to invoke `skipField()`.
2. **Universal Vectors:** Standard Maps! `MessageSchema.decodeMapEntry` handles unrecognized tags inside a map entry by calling `ArrayDecoders.skipField()`. Nesting thousands of groups inside a standard map entry instantly crashes the JVM process.
### Fix
Added `registers.recursionDepth++` and `checkRecursionLimit()` to the `WIRETYPE_START_GROUP` branch in `skipField()`, aligning it securely with `decodeUnknownField()`.
### Issue Link
Additional discussion and reproduction code is documented internally at:
https://issuetracker.google.com/issues/498542124Closes#26670
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/26670 from VenkatKwest:fix-unguarded-recursion-lite 3660dd6bce
PiperOrigin-RevId: 899005419
these loops are quite hot, so do simple iteration. also skip the overhead of getFields(). this means we skip 1) creating a list from the internal fields array, 2) creating an unmodifiable list 3) creating an iterator for that list 4) the overhead of calling the iterator's hasNext() and next() methods (which in turn have to call the wrapped iterator). i skipped doing the same in GeneratedMessage since the generated messages have optimized isInitialized implementations already.
here is a screenshot from a cpu flamechart (i believe this change would remove all of the pink and then some):
<img width="1612" height="478" alt="image" src="https://github.com/user-attachments/assets/73ce5629-ecce-4f0b-a7de-4906fcd68533" />
here is an excerpt from the memory flamechart:
<img width="1614" height="397" alt="image" src="https://github.com/user-attachments/assets/7e3fa66a-a2c8-420f-a3d4-a84788af65ad" />
background: i am using kafka connect to process a lot of protobuf messages and that uses their schema registry which uses protobuf's DynamicMessage.
Closes#25277
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/25277 from yuzawa-san:dynamic-message-is-initialized fa1e1f7c17
PiperOrigin-RevId: 893693587
Exception throwing is expensive: it increases inlining cost and
generates lots of assembly. If we can hide it away in its own function,
that makes the other functions in here more tempting for Android's
inliner; e.g. `set` might be small enough to inline now.
PiperOrigin-RevId: 888897688
This communicates to readers that it's not for overriding.
This method is hot; it indicates to the compiler that it can inline or
optimise this further.
In practice, R8 will turn all not-overriden methods to be final, so this will only improve performance in unoptimised dev builds.
PiperOrigin-RevId: 888885902
