We previously had a hack in Python/C++ Protobuf to account for the fact that LazyField did not properly remember a custom DescriptorPool or MessageFactory that was set in the ParseContext at parse time.
The code has since been fixed to properly handle the case where ParseContext contains a custom DescriptorPool/MessageFactory. Removing the hack removes the data race under free threading.
PiperOrigin-RevId: 967236272
The unit tests added in the initial implementation in cl/925012500 and
cl/922729754 were fairly minimal. I took some inspiration from our C++
and Java unit tests for JSON's custom enum name parsing to come up with
these new test cases.
This change also adds a python Benchmark that we can use to verify that
our optimizations in cl/952170251 actually works.
Benchmark results across all three proto implementations (cpp, python, upb):
=== C++ Extension Protos (json_format_benchmark_cpp_protos) ===
[BENCHMARK] ParseJsonDefault: med 8.70 us/op | p99 9.56 us/op | mean 8.74 ± 0.15 us/op
[BENCHMARK] ParseJsonCustom: med 17.39 us/op | p99 21.64 us/op | mean 17.43 ± 0.51 us/op
[BENCHMARK] ParseJsonUnknownIgnored: med 26.97 us/op | p99 32.04 us/op | mean 27.06 ± 0.65 us/op
[BENCHMARK] ParseRepeatedJsonDefault: med 1.71 us/item | p99 1.83 us/item | mean 1.71 ± 0.02 us/item
[BENCHMARK] ParseRepeatedJsonCustom: med 9.18 us/item | p99 9.88 us/item | mean 9.20 ± 0.11 us/item
[BENCHMARK] ParseRepeatedJsonUnknownIgnored: med 18.79 us/item | p99 19.19 us/item | mean 18.81 ± 0.13 us/item
=== UPB Extension Protos (json_format_benchmark_upb_protos) ===
[BENCHMARK] ParseJsonDefault: med 8.70 us/op | p99 9.56 us/op | mean 8.74 ± 0.15 us/op
[BENCHMARK] ParseJsonCustom: med 17.39 us/op | p99 21.64 us/op | mean 17.43 ± 0.51 us/op
[BENCHMARK] ParseJsonUnknownIgnored: med 26.97 us/op | p99 32.04 us/op | mean 27.06 ± 0.65 us/op
[BENCHMARK] ParseRepeatedJsonDefault: med 1.71 us/item | p99 1.83 us/item | mean 1.71 ± 0.02 us/item
[BENCHMARK] ParseRepeatedJsonCustom: med 9.18 us/item | p99 9.88 us/item | mean 9.20 ± 0.11 us/item
[BENCHMARK] ParseRepeatedJsonUnknownIgnored: med 18.79 us/item | p99 19.19 us/item | mean 18.81 ± 0.13 us/item
=== Pure Python Protos (json_format_benchmark_python_protos) ===
[BENCHMARK] ParseJsonDefault: med 10.26 us/op | p99 11.20 us/op | mean 10.30 ± 0.25 us/op
[BENCHMARK] ParseJsonCustom: med 16.59 us/op | p99 18.35 us/op | mean 16.70 ± 0.45 us/op
[BENCHMARK] ParseJsonUnknownIgnored: med 25.09 us/op | p99 28.22 us/op | mean 25.27 ± 0.65 us/op
[BENCHMARK] ParseRepeatedJsonDefault: med 1.94 us/item | p99 3.25 us/item | mean 1.97 ± 0.14 us/item
[BENCHMARK] ParseRepeatedJsonCustom: med 7.16 us/item | p99 10.35 us/item | mean 7.24 ± 0.39 us/item
[BENCHMARK] ParseRepeatedJsonUnknownIgnored: med 15.43 us/item | p99 19.54 us/item | mean 15.54 ± 0.49 us/item
PiperOrigin-RevId: 966820541
Update _message to use rules_python's py_extension macro (which
automatically injects required toolchain headers and manages shared library
extensions across platforms), and update _api_implementation.so to depend
directly on `@rules_python//python/cc:current_py_cc_headers` instead of
`@system_python`.
Using `py_extension` and `current_py_cc_headers` decouples Python C
extensions from the custom system_python.bzl script while maintaining
current host-interpreter behavior during the migration.
#test-continuous
PiperOrigin-RevId: 966345187
MessageSet is a legacy affordance holdover from pre-proto2 days. Unlike the rest of Protobuf, it has first-wins instead of last-wins semantic in the case of duplicate ids or values within the same message set entry.
No serializer would ever write such sequences, but the same hypothetical byte sequence being parsed differently by two implementations is undesirable.
Note that other nonconformance fixes would commonly be held to breaking change releases to avoid any possible disruption of single-language users who may be relying on it as a load-bearing bug. However, the nature of this particular case is such that there is little reason to hold it back.
PiperOrigin-RevId: 964043226
* `protoc`: Included in zips alongside binary
* Java: Generated classes have been added to the Maven `util` artifact
* Ruby: Generated classes have been added to gems
* CMake: `.proto` files are included in installation
PiperOrigin-RevId: 957382561
We parse into a temporary message first to detect oneof switches before modifying the target message, and release the wrappers for switching oneof fields in the target message.
PiperOrigin-RevId: 954778098
In 'PythonMessageMutator::~PythonMessageMutator()', 'PyBytes_FromStringAndSize' can return 'nullptr' if memory allocation fails or string length overflows. Calling 'Py_DECREF(py_wire)' and 'PyObject_CallMethod' when 'py_wire' is 'nullptr' results in a NULL-pointer dereference crash.
Add a null check for 'py_wire' before calling 'PyObject_CallMethod' and 'Py_DECREF'.
PiperOrigin-RevId: 953616147
In 'PyUpb_Message_MergeFromString', 'upb_Decode' may partially mutate or populate sub-messages on the parent message before failing with a decode error status. Skipping 'PyUpb_Message_SyncSubobjs' when 'status != kUpb_DecodeStatus_Ok' leaves stub sub-object wrappers unsynced in 'unset_subobj_map', causing duplicate keys in 'ObjCache' on subsequent access and leading to heap-use-after-free.
Move 'PyUpb_Message_SyncSubobjs(self)' before the decode status check in 'PyUpb_Message_MergeFromString' so any sub-message wrappers modified during decoding are synced properly.
Add 'testMergeFromStringDecodeErrorSync' to 'third_party/py/google/protobuf/internal/message_test.py' to verify that stub sub-objects remain synced and intact after a failed 'MergeFromString'.
PiperOrigin-RevId: 953615725
This change modifies all remaining `Dealloc()` functions to use `EraseIfEqual` if they were not already. This prevents the same race that was fixed for descriptors in cl/874084218.
PiperOrigin-RevId: 952273589
This change adds API and implementation to read and write the option_dependency field from/to proto2.FileDescriptorProto in upb reflection.
PiperOrigin-RevId: 947120641
* Updates edition to 2026 in the unittest protos.
* Removes the obsolete target compile option `cc_enable_arenas` since C++
options are moved and arenas are enabled by default.
* Updates `maximum_edition` target constraints to edition 2026 in Java,
Python, C#, and upb build targets to allow loading and validation of
Edition 2026 files.
* Regenerates internal_options_bootstrap compiler files.
PiperOrigin-RevId: 944424868
Fixes https://github.com/protocolbuffers/protobuf/issues/28261 by setting symbol visibility in `setup.py`, similar to `py_extension.bzl`, using the existing branching logic to bypass the flag on MSVC.
This issue manifests in the [NixOS/nixpkgs](https://github.com/NixOS/nixpkgs) build of the `protobuf` python package, which contains upb symbols, and causes fatal failures when loaded in a process that separately links/loads libprotobuf. We'll supply a patch there as well, until this can be backported or released in future protobuf versions.
Closes#28264
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/28264 from mitchallain:mallain/fix-python-upb-symbols 6e118a62bf
PiperOrigin-RevId: 942223178
## Bug
`Clear()` in `map_container.cc` (line 294-302) calls
`reflection->ClearField()` which destroys all underlying map nodes via
`ClearTable(reset=true)`, but does not increment `self->version`.
All other mutators (ScalarMapSetItem, MessageMapSetItem, MergeFrom, etc.)
increment `self->version` after mutation. `IterNext()` relies on version
mismatch to detect concurrent modification and raise `RuntimeError`.
Without the version bump, a live iterator proceeds to dereference the
freed `NodeBase*` via `SetMapIteratorValue` → `UntypedMapIterator::PlusPlus`.
**ASAN confirmed:** heap-use-after-free, READ size 8 at
`UntypedMapIterator::PlusPlus` (map.h:599), freed by `ClearTable`
(map.h:345), allocated by `ScalarMapSetItem` (map_container.cc:416).
## Fix
Add `self->version++` after `ClearField` in `Clear()`, matching every
other mutator in the same file.
## Reproducer
```python
msg = M() # proto3 with map<string, int32> mp
for k in ("a","b","c","d"): msg.mp[k] = 1
it = iter(msg.mp)
next(it)
msg.mp.clear() # frees nodes, version NOT bumped
next(it) # heap-use-after-free
```
Closes#27257
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/27257 from vhullto:fix/python-map-clear-uaf fb35225110
PiperOrigin-RevId: 939482747
When cpython_bits.type_getattro(self, name) returns NULL due to an exception raised in a descriptor (such as KeyboardInterrupt, MemoryError, or SystemExit), PyUpb_MessageMeta_GetAttr previously cleared the error and raised AttributeError.
Check PyErr_ExceptionMatches(PyExc_AttributeError) before clearing the error to ensure non-AttributeError exceptions are properly propagated.
PiperOrigin-RevId: 938246969
When PyObject_GetAttrString returns NULL (e.g., because __name__ or __module__ attribute access raises an exception on metaclass), PyUpb_GetStrData returns NULL. Passing NULL to strcmp previously caused a segmentation fault.
Check for NULL before calling PyUpb_GetStrData and strcmp, and call PyErr_Clear() when attribute lookup fails.
PiperOrigin-RevId: 938039167
PyTuple_Pack increments the reference count of its arguments. When PyTuple_Pack(2, start, end) was passed directly to PyList_SetItem, the start and end PyLong objects were leaked on every call.
Store the tuple created by PyTuple_Pack and DECREF start and end before inserting into the list.
PiperOrigin-RevId: 937601625
The pure Python implementation of maps had a bug where looking up an entry in a `map<string, ValueType>` field using a `bytes` key would cause silent data corruption.
**Bug:**
1. `msg.my_map['foo'] = 100` stores `{'foo': 100}`.
2. `msg.my_map[b'foo']` attempts a lookup.
3. `self._values[b'foo']` raises `KeyError` as `b'foo' != 'foo'`.
4. The `except KeyError` block normalizes `b'foo'` to `'foo'`.
5. Crucially, it then inserts a *default value* for the value type, overwriting the existing entry: `self._values['foo'] = 0`.
6. The lookup returns 0, and the original value of 100 is lost.
**Fix:**
The key is now normalized using `self._key_checker.CheckValue(key)` *at the beginning* of `__getitem__`, `__contains__`, `get`, `__delitem__`, and `setdefault`. This ensures the key is in the canonical `str` format *before* any dictionary access, preventing the erroneous write-on-miss.
This change makes the behavior consistent with the C++ and upb implementations.
PiperOrigin-RevId: 930498495
