## Summary
- fix Python `text_format` by adding an optional recursion depth limit
- add `max_recursion_depth` to `Parse`, `Merge`, `ParseLines`, and `MergeLines`
- keep the default behavior unchanged for compatibility
- enforce the configured limit consistently across root parsing, nested submessages, and expanded `Any`
- add regression tests for explicit opt-in depth enforcement and the default compatibility path
## Context
Python `text_format` parsing recursively descends through `_MergeField()` and `_MergeMessageField()`, and can also recurse through expanded `Any` payloads. Without a shared parser-side depth guard, deeply nested input can recurse until Python raises `RecursionError` instead of a controlled protobuf `ParseError`.
For applications that parse untrusted textproto, that makes deep nesting a denial-of-service primitive.
## Fix
This change adds an optional `max_recursion_depth` parameter to the public Python text format parsing entry points.
When the option is set, parsing tracks message depth through a shared helper and raises `ParseError` once the configured limit is exceeded. When the option is not set, parsing keeps the historical unbounded behavior so existing callers remain compatible.
## Testing
- `BAZEL_NO_APPLE_CPP_TOOLCHAIN=1 bazel test //python:text_format_test --test_output=errors`
Closes#26604
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/26604 from Sakura-501:fix-python-text-format-unbounded-recursion 235abcd492
PiperOrigin-RevId: 898014428
This change introduces a `LazyUniquePtr` class to manage the lazy initialization of `composite_fields` and `child_submessages` within `CMessage`. When Python's GIL is disabled, `LazyUniquePtr` uses atomic operations to ensure thread-safe initialization of these maps, preventing data races when multiple threads concurrently access fields that trigger their creation. A new test case is added to reproduce and verify the fix for a race condition in `GetFieldValue`.
We are forced to use placement new and placement delete for the `LazyUniquePtr` members, because the `CMessage` struct is currently not properly constructed or destroyed. This makes the code a bit awkward, but changing the construction/destruction of CMessage seemed like too much to bite off in this CL.
PiperOrigin-RevId: 891703488
This CL concerns the following functions in C++:
```c++
class Reflection {
const Message& GetRepeatedMessage(const Message& message,
const FieldDescriptor* field,
int index) const;
Message* MutableRepeatedMessage(Message* message,
const FieldDescriptor* field,
int index) const;
}
```
Suppose a Python program contains the following code:
```
def foo(msg: MyMessage):
submsg = msg.repeated_foo[2] # (A)
if also_mutate:
submsg.abc = 1 # (B)
```
For line (A), we currently call `MutableRepeatedMessage()` in C++ to
obtain the submessage pointer, because the resulting `submsg` object in
Python is a conceptually mutable object that will require a non-const
pointer if/when the program hits line (B).
The `MutableRepeatedMessage(Message* msg, ...)` function in C++
currently mutates `msg` by setting the hasbit of the repeated field. But
this seems unnecessary, as the function requires that the requested
sub-message already exists; it does not create a new message. So it
should not be necessary to touch the hasbit of the message, and a TGP
confirms that all tests pass if we remove this call.
Once the `SetHasBitForRepeated()` call is removed, the `MutableRepeatedMessage()`
function no longer actually mutates the `msg` argument. This makes it
effectively safe to call concurrently (ie. it will no longer trigger
Undefined Behavior in C++, and the TSAN errors will go away), which
is why it fixes the free threading test. But it leaves us in an odd
state where we are still passing a non-`const` pointer to the same
message to two functions concurrently, which is not allowed under a
normal thread-compatible contract.
An alternate solution would be to call `GetRepeatedMessage()` instead,
and `const_cast<>` away the const in the returned `const Message&`.
But this is also violates the contract: users should not be casting
away `const`.
The root cause of this odd situation is that
`MutableRepeatedMessage(Message* msg)` requires a non-`const` `msg` not
because it actually *mutates* `msg`, but because it is trying to
propagate the const-ness of `msg` to all of its children. The proto API
generally guarantees that a const message prevents mutation of not just
the top-level message, but of the entire tree of messages. If someone
passes you a `const` message pointer, that is supposed to render the
entire tree of messages under it immutable through that pointer.
What we wish we could express in `MutableRepeatedMessage(Message* msg)`
is: "this function requires that `msg` is mutable, but this function
will not actually mutate it, and is therefore safe to call concurrently."
But there is no way of expressing this in C++.
PiperOrigin-RevId: 886830933
This change adds type annotations to the methods and instance variables within the DescriptorDatabase class, as well as to the helper function _ExtractSymbols. Forward references are used for types from descriptor_pb2.
PiperOrigin-RevId: 884793323
In `FixupMessageAfterMerge()`, the loop over fields was exiting the entire function if a mutable field was found, but this skips other fields that may need fixing up.
PiperOrigin-RevId: 882766354
This doesn't flip the default bazel version, but instead makes this repo
compatible with the current version and bazel 9.x. The primary changes
for 9.x support are adding new load statements for things that were
previously built in. This cascaded into a few dep updates to pull in
their missing load statement fixes.
Closes#26201
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/26201 from keith:ks/add-support-for-bazel-9.x e8192d86b3c838f6c88d4a4adfebd07162debc0d
PiperOrigin-RevId: 878654773
* Remove unneeded "google/protobuf/stubs/common.h" includes
* Migrate uses of kuint64max over to the C++ standard std::numeric_limits<uint64_t>::max()
PiperOrigin-RevId: 875122434
Prior to this CL, it was possible for the following sequence to occur:
|Thread 1|Thread 2|
|--------|--------|
|`obj = NewDescriptor(desc)`||
|`InsertCache(desc, obj)`||
|`Py_DECREF(obj)` (to 0)||
|`Dealloc(obj) {`||
||`LookupCache(desc) -> obj`|
||`Py_INCREF(obj)`|
|` DeleteFromCache(obj)`||
|`}`||
||`Py_DECREF(obj)`|
||`Dealloc(obj)`|
This could lead to double-`Dealloc()` calls on a single object. These calls could race, leading to TSAN failures.
We should look deeper into whether `GcTraverse()` and `GcClear()` still need critical sections.
PiperOrigin-RevId: 874084218
Bazel 8 no longer supports legacy toolchains, so all of our release toolchains have been updated to use platforms.
#test-continuous
PiperOrigin-RevId: 870017275
This fixes a security vulnerability where nested google.protobuf.Any messages could bypass the max_recursion_depth limit, potentially leading to denial of service via stack overflow.
The root cause was that _ConvertAnyMessage() was calling itself recursively via methodcaller() for nested well-known types, bypassing the recursion depth tracking in ConvertMessage().
The fix routes well-known type parsing through ConvertMessage() to ensure proper recursion depth accounting for all message types including nested Any.
Fixes#25070Closes#25239
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/25239 from aviralgarg05:fix-any-recursion-depth-bypass 3cbbcbea142593d3afd2ceba2db14b05660f62f4
PiperOrigin-RevId: 862740421
#test-continuous
While we'd like to burn these down, they're still necessary at least for WKT to avoid cyclic dependencies. This solution at least enabled prebuilt protoc for languages other than Java/Kotlin.
Fixes#25453
PiperOrigin-RevId: 862290462
We are absorbing its contents into other directories. This will reduce the 3:1 merge to `upb/bazel` in GitHub to 2:1 (soon to be 1:1).
PiperOrigin-RevId: 860299666
(The crashes happen only since Python 3.13)
In particular:
* Skip adding to the weak map if `Py_IsFinalizing()` is true. Calling `PyUpb_WeakMap_Add` may be caused by user code that creates protos inside `__del__`!
* `PyUpb_DescriptorPool_Get` and subsequently `PyUpb_ObjCache_Get` may *also* be called if users cause `PyUpb_RepeatedContainer_GetOrCreateWrapper` to be called. So make it nullable and skip the assertion if currently shutting down.
This follows a prior attempted fix d57d2708b3 but expands the scope of it.
PiperOrigin-RevId: 856589514
The pyi generator now includes Kythe annotations for:
* Extension field constants (e.g., `EXTENSION_FIELD`).
* Field number constants (e.g., `STRING_FIELD_FIELD_NUMBER`).
* `Create` methods in generated Stubby client classes.
These annotations allow the Python indexer to link these generated symbols back to their definitions in the `.proto` files.
PiperOrigin-RevId: 852971617
This covers two types of failures:
* Methods that are logically const and failure to consume the result indicates a bug
(an unnecessary call, etc.)
* Methods that return significant errors (failure to parse, etc.) that should not be
unintentionally ignored.
PiperOrigin-RevId: 852313694
