Avoid calling ByteSize() separately in serialize_length_prefixed(). The serialized payload already carries the exact length to prefix, so this keeps behavior intact while avoiding a duplicate serialization-sized pass.
Closes#27252
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/27252 from Zaczero:python-serialize-length-prefixed-once 8a4f60b4c7
PiperOrigin-RevId: 923539199
This test highlights a behavioral difference where upb allows equality comparison between a RepeatedCompositeContainer and a Python list, while the cpp and python implementations raise a TypeError.
PiperOrigin-RevId: 912754358
Fixes#26389
`AssureWritable()` returns `-1` on failure (OOM in `MutableMessage()`), but 13 call sites were ignoring this return value.
When `AssureWritable()` fails, `self->message` still points to the shared read-only default instance. Subsequent mutations (`MergeFrom`, `CopyFrom`, `Clear`, `SetField`, etc.) then corrupt the shared global default instance, affecting all messages of the same type.
Same bug class as the recent `FixupMessageAfterMerge` fix (ab14c0f8a).
Closes#26390
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/26390 from Oblivionsage:fix/check-assure-writable-return-value adbd20477e
PiperOrigin-RevId: 912044414
This avoids calling `GetPrototype()` every time a message is constructed, which would acquire a lock.
There is no need to look it up every time, because it will never change as long as the MessageFactory is alive.
PiperOrigin-RevId: 910918341
Add test coverage of 'assign string to enum-typed field' in general (including this case). This appears to be accepted only Py-upb and not otherwise.
Fixes https://github.com/protocolbuffers/protobuf/issues/27106
PiperOrigin-RevId: 908148411
This change introduces a FreeThreadingMutex to PyDescriptorPool to guard access to the descriptor_options and descriptor_features hash maps.
PiperOrigin-RevId: 908115178
The test demonstrates that modifying the message options returned by GetOptions() is currently possible, which is an unintended side effect.
PiperOrigin-RevId: 903414364
In the cases where we historically didn't enforce a depth limit (C++ and Python) document that the default behavior having no depth limit is an intentional decision.
PiperOrigin-RevId: 903216142
The previous implementation of RepeatedContainer_pop would clamp the index to the last element if it was out of bounds. This change removes the index clamping, ensuring that pop() raises an IndexError when given an index that is out of range, matching the other 2 pyproto implementation as well as standard Python list behavior. Tests are added to confirm this corrected behavior.
This is the 5th bug listed in https://github.com/protocolbuffers/protobuf/issues/26596
PiperOrigin-RevId: 902737839
## Summary
- fix Python `text_format` by adding an optional recursion depth limit
- add `max_recursion_depth` to `Parse`, `Merge`, `ParseLines`, and `MergeLines`
- keep the default behavior unchanged for compatibility
- enforce the configured limit consistently across root parsing, nested submessages, and expanded `Any`
- add regression tests for explicit opt-in depth enforcement and the default compatibility path
## Context
Python `text_format` parsing recursively descends through `_MergeField()` and `_MergeMessageField()`, and can also recurse through expanded `Any` payloads. Without a shared parser-side depth guard, deeply nested input can recurse until Python raises `RecursionError` instead of a controlled protobuf `ParseError`.
For applications that parse untrusted textproto, that makes deep nesting a denial-of-service primitive.
## Fix
This change adds an optional `max_recursion_depth` parameter to the public Python text format parsing entry points.
When the option is set, parsing tracks message depth through a shared helper and raises `ParseError` once the configured limit is exceeded. When the option is not set, parsing keeps the historical unbounded behavior so existing callers remain compatible.
## Testing
- `BAZEL_NO_APPLE_CPP_TOOLCHAIN=1 bazel test //python:text_format_test --test_output=errors`
Closes#26604
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/26604 from Sakura-501:fix-python-text-format-unbounded-recursion 235abcd492
PiperOrigin-RevId: 898014428
This change introduces a `LazyUniquePtr` class to manage the lazy initialization of `composite_fields` and `child_submessages` within `CMessage`. When Python's GIL is disabled, `LazyUniquePtr` uses atomic operations to ensure thread-safe initialization of these maps, preventing data races when multiple threads concurrently access fields that trigger their creation. A new test case is added to reproduce and verify the fix for a race condition in `GetFieldValue`.
We are forced to use placement new and placement delete for the `LazyUniquePtr` members, because the `CMessage` struct is currently not properly constructed or destroyed. This makes the code a bit awkward, but changing the construction/destruction of CMessage seemed like too much to bite off in this CL.
PiperOrigin-RevId: 891703488
This CL concerns the following functions in C++:
```c++
class Reflection {
const Message& GetRepeatedMessage(const Message& message,
const FieldDescriptor* field,
int index) const;
Message* MutableRepeatedMessage(Message* message,
const FieldDescriptor* field,
int index) const;
}
```
Suppose a Python program contains the following code:
```
def foo(msg: MyMessage):
submsg = msg.repeated_foo[2] # (A)
if also_mutate:
submsg.abc = 1 # (B)
```
For line (A), we currently call `MutableRepeatedMessage()` in C++ to
obtain the submessage pointer, because the resulting `submsg` object in
Python is a conceptually mutable object that will require a non-const
pointer if/when the program hits line (B).
The `MutableRepeatedMessage(Message* msg, ...)` function in C++
currently mutates `msg` by setting the hasbit of the repeated field. But
this seems unnecessary, as the function requires that the requested
sub-message already exists; it does not create a new message. So it
should not be necessary to touch the hasbit of the message, and a TGP
confirms that all tests pass if we remove this call.
Once the `SetHasBitForRepeated()` call is removed, the `MutableRepeatedMessage()`
function no longer actually mutates the `msg` argument. This makes it
effectively safe to call concurrently (ie. it will no longer trigger
Undefined Behavior in C++, and the TSAN errors will go away), which
is why it fixes the free threading test. But it leaves us in an odd
state where we are still passing a non-`const` pointer to the same
message to two functions concurrently, which is not allowed under a
normal thread-compatible contract.
An alternate solution would be to call `GetRepeatedMessage()` instead,
and `const_cast<>` away the const in the returned `const Message&`.
But this is also violates the contract: users should not be casting
away `const`.
The root cause of this odd situation is that
`MutableRepeatedMessage(Message* msg)` requires a non-`const` `msg` not
because it actually *mutates* `msg`, but because it is trying to
propagate the const-ness of `msg` to all of its children. The proto API
generally guarantees that a const message prevents mutation of not just
the top-level message, but of the entire tree of messages. If someone
passes you a `const` message pointer, that is supposed to render the
entire tree of messages under it immutable through that pointer.
What we wish we could express in `MutableRepeatedMessage(Message* msg)`
is: "this function requires that `msg` is mutable, but this function
will not actually mutate it, and is therefore safe to call concurrently."
But there is no way of expressing this in C++.
PiperOrigin-RevId: 886830933
This change adds type annotations to the methods and instance variables within the DescriptorDatabase class, as well as to the helper function _ExtractSymbols. Forward references are used for types from descriptor_pb2.
PiperOrigin-RevId: 884793323
In `FixupMessageAfterMerge()`, the loop over fields was exiting the entire function if a mutable field was found, but this skips other fields that may need fixing up.
PiperOrigin-RevId: 882766354
* Remove unneeded "google/protobuf/stubs/common.h" includes
* Migrate uses of kuint64max over to the C++ standard std::numeric_limits<uint64_t>::max()
PiperOrigin-RevId: 875122434
Prior to this CL, it was possible for the following sequence to occur:
|Thread 1|Thread 2|
|--------|--------|
|`obj = NewDescriptor(desc)`||
|`InsertCache(desc, obj)`||
|`Py_DECREF(obj)` (to 0)||
|`Dealloc(obj) {`||
||`LookupCache(desc) -> obj`|
||`Py_INCREF(obj)`|
|` DeleteFromCache(obj)`||
|`}`||
||`Py_DECREF(obj)`|
||`Dealloc(obj)`|
This could lead to double-`Dealloc()` calls on a single object. These calls could race, leading to TSAN failures.
We should look deeper into whether `GcTraverse()` and `GcClear()` still need critical sections.
PiperOrigin-RevId: 874084218
This fixes a security vulnerability where nested google.protobuf.Any messages could bypass the max_recursion_depth limit, potentially leading to denial of service via stack overflow.
The root cause was that _ConvertAnyMessage() was calling itself recursively via methodcaller() for nested well-known types, bypassing the recursion depth tracking in ConvertMessage().
The fix routes well-known type parsing through ConvertMessage() to ensure proper recursion depth accounting for all message types including nested Any.
Fixes#25070Closes#25239
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/25239 from aviralgarg05:fix-any-recursion-depth-bypass 3cbbcbea142593d3afd2ceba2db14b05660f62f4
PiperOrigin-RevId: 862740421
The pyi generator now includes Kythe annotations for:
* Extension field constants (e.g., `EXTENSION_FIELD`).
* Field number constants (e.g., `STRING_FIELD_FIELD_NUMBER`).
* `Create` methods in generated Stubby client classes.
These annotations allow the Python indexer to link these generated symbols back to their definitions in the `.proto` files.
PiperOrigin-RevId: 852971617
This covers two types of failures:
* Methods that are logically const and failure to consume the result indicates a bug
(an unnecessary call, etc.)
* Methods that return significant errors (failure to parse, etc.) that should not be
unintentionally ignored.
PiperOrigin-RevId: 852313694
