This implementation is similar to our Java idea in cl/949670944,
basically the idea is that we will have a local map (dictionary) that
will be populated the very first time we try to parse into an enum
field. The lifetime of this dictionary will be tied to the Parser
instance, similar to the Java implementation.
BENCHMARKS (baseline stats are recorded at cl/952170250):
Python:
```
[BENCHMARK] ParseJsonDefault: med 10.06 us/op | p99 11.22 us/op | mean 10.10 ± 0.20 us/op
[BENCHMARK] ParseJsonCustom: med 20.33 us/op | p99 22.23 us/op | mean 20.40 ± 0.34 us/op
[BENCHMARK] ParseJsonUnknownIgnored: med 24.21 us/op | p99 26.25 us/op | mean 24.31 ± 0.44 us/op
[BENCHMARK] ParseRepeatedJsonDefault: med 1.94 us/item | p99 2.05 us/item | mean 1.95 ± 0.03 us/item
[BENCHMARK] ParseRepeatedJsonCustom: med 2.26 us/item | p99 2.45 us/item | mean 2.27 ± 0.03 us/item
[BENCHMARK] ParseRepeatedJsonUnknownIgnored: med 5.31 us/item | p99 5.52 us/item | mean 5.32 ± 0.05 us/item
```
Cpp:
```
[BENCHMARK] ParseJsonDefault: med 8.66 us/op | p99 9.21 us/op | mean 8.70 ± 0.13 us/op
[BENCHMARK] ParseJsonCustom: med 26.33 us/op | p99 29.71 us/op | mean 26.44 ± 0.69 us/op
[BENCHMARK] ParseJsonUnknownIgnored: med 28.46 us/op | p99 31.80 us/op | mean 28.59 ± 0.67 us/op
[BENCHMARK] ParseRepeatedJsonDefault: med 1.72 us/item | p99 2.19 us/item | mean 1.74 ± 0.06 us/item
[BENCHMARK] ParseRepeatedJsonCustom: med 2.12 us/item | p99 2.38 us/item | mean 2.13 ± 0.04 us/item
[BENCHMARK] ParseRepeatedJsonUnknownIgnored: med 4.61 us/item | p99 4.92 us/item | mean 4.62 ± 0.07 us/item
```
UPB:
```
[BENCHMARK] ParseJsonDefault: med 11.22 us/op | p99 12.29 us/op | mean 11.27 ± 0.21 us/op
[BENCHMARK] ParseJsonCustom: med 37.54 us/op | p99 39.25 us/op | mean 37.65 ± 0.46 us/op
[BENCHMARK] ParseJsonUnknownIgnored: med 41.18 us/op | p99 48.45 us/op | mean 41.30 ± 0.89 us/op
[BENCHMARK] ParseRepeatedJsonDefault: med 2.10 us/item | p99 2.19 us/item | mean 2.10 ± 0.02 us/item
[BENCHMARK] ParseRepeatedJsonCustom: med 2.58 us/item | p99 2.73 us/item | mean 2.59 ± 0.03 us/item
[BENCHMARK] ParseRepeatedJsonUnknownIgnored: med 5.70 us/item | p99 5.90 us/item | mean 5.71 ± 0.04 us/item
```
Seeing these benchmarks, we can see a very noticeable improvement in
`ParseRepeatedJsonCustom` and `ParseRepeatedJsonUnknownIgnored` cases --
which correspond to the pathological cases that we are worried about.
Singular fields are taking longer to parse in our microbenchmark because
the cache that we've added is tied to each Parser instance and must be
re-instantiated across every `Parse` call.
PiperOrigin-RevId: 970528683
We previously had a hack in Python/C++ Protobuf to account for the fact that LazyField did not properly remember a custom DescriptorPool or MessageFactory that was set in the ParseContext at parse time.
The code has since been fixed to properly handle the case where ParseContext contains a custom DescriptorPool/MessageFactory. Removing the hack removes the data race under free threading.
PiperOrigin-RevId: 967236272
The unit tests added in the initial implementation in cl/925012500 and
cl/922729754 were fairly minimal. I took some inspiration from our C++
and Java unit tests for JSON's custom enum name parsing to come up with
these new test cases.
This change also adds a python Benchmark that we can use to verify that
our optimizations in cl/952170251 actually works.
Benchmark results across all three proto implementations (cpp, python, upb):
=== C++ Extension Protos (json_format_benchmark_cpp_protos) ===
[BENCHMARK] ParseJsonDefault: med 8.70 us/op | p99 9.56 us/op | mean 8.74 ± 0.15 us/op
[BENCHMARK] ParseJsonCustom: med 17.39 us/op | p99 21.64 us/op | mean 17.43 ± 0.51 us/op
[BENCHMARK] ParseJsonUnknownIgnored: med 26.97 us/op | p99 32.04 us/op | mean 27.06 ± 0.65 us/op
[BENCHMARK] ParseRepeatedJsonDefault: med 1.71 us/item | p99 1.83 us/item | mean 1.71 ± 0.02 us/item
[BENCHMARK] ParseRepeatedJsonCustom: med 9.18 us/item | p99 9.88 us/item | mean 9.20 ± 0.11 us/item
[BENCHMARK] ParseRepeatedJsonUnknownIgnored: med 18.79 us/item | p99 19.19 us/item | mean 18.81 ± 0.13 us/item
=== UPB Extension Protos (json_format_benchmark_upb_protos) ===
[BENCHMARK] ParseJsonDefault: med 8.70 us/op | p99 9.56 us/op | mean 8.74 ± 0.15 us/op
[BENCHMARK] ParseJsonCustom: med 17.39 us/op | p99 21.64 us/op | mean 17.43 ± 0.51 us/op
[BENCHMARK] ParseJsonUnknownIgnored: med 26.97 us/op | p99 32.04 us/op | mean 27.06 ± 0.65 us/op
[BENCHMARK] ParseRepeatedJsonDefault: med 1.71 us/item | p99 1.83 us/item | mean 1.71 ± 0.02 us/item
[BENCHMARK] ParseRepeatedJsonCustom: med 9.18 us/item | p99 9.88 us/item | mean 9.20 ± 0.11 us/item
[BENCHMARK] ParseRepeatedJsonUnknownIgnored: med 18.79 us/item | p99 19.19 us/item | mean 18.81 ± 0.13 us/item
=== Pure Python Protos (json_format_benchmark_python_protos) ===
[BENCHMARK] ParseJsonDefault: med 10.26 us/op | p99 11.20 us/op | mean 10.30 ± 0.25 us/op
[BENCHMARK] ParseJsonCustom: med 16.59 us/op | p99 18.35 us/op | mean 16.70 ± 0.45 us/op
[BENCHMARK] ParseJsonUnknownIgnored: med 25.09 us/op | p99 28.22 us/op | mean 25.27 ± 0.65 us/op
[BENCHMARK] ParseRepeatedJsonDefault: med 1.94 us/item | p99 3.25 us/item | mean 1.97 ± 0.14 us/item
[BENCHMARK] ParseRepeatedJsonCustom: med 7.16 us/item | p99 10.35 us/item | mean 7.24 ± 0.39 us/item
[BENCHMARK] ParseRepeatedJsonUnknownIgnored: med 15.43 us/item | p99 19.54 us/item | mean 15.54 ± 0.49 us/item
PiperOrigin-RevId: 966820541
MessageSet is a legacy affordance holdover from pre-proto2 days. Unlike the rest of Protobuf, it has first-wins instead of last-wins semantic in the case of duplicate ids or values within the same message set entry.
No serializer would ever write such sequences, but the same hypothetical byte sequence being parsed differently by two implementations is undesirable.
Note that other nonconformance fixes would commonly be held to breaking change releases to avoid any possible disruption of single-language users who may be relying on it as a load-bearing bug. However, the nature of this particular case is such that there is little reason to hold it back.
PiperOrigin-RevId: 964043226
We parse into a temporary message first to detect oneof switches before modifying the target message, and release the wrappers for switching oneof fields in the target message.
PiperOrigin-RevId: 954778098
In 'PyUpb_Message_MergeFromString', 'upb_Decode' may partially mutate or populate sub-messages on the parent message before failing with a decode error status. Skipping 'PyUpb_Message_SyncSubobjs' when 'status != kUpb_DecodeStatus_Ok' leaves stub sub-object wrappers unsynced in 'unset_subobj_map', causing duplicate keys in 'ObjCache' on subsequent access and leading to heap-use-after-free.
Move 'PyUpb_Message_SyncSubobjs(self)' before the decode status check in 'PyUpb_Message_MergeFromString' so any sub-message wrappers modified during decoding are synced properly.
Add 'testMergeFromStringDecodeErrorSync' to 'third_party/py/google/protobuf/internal/message_test.py' to verify that stub sub-objects remain synced and intact after a failed 'MergeFromString'.
PiperOrigin-RevId: 953615725
This change modifies all remaining `Dealloc()` functions to use `EraseIfEqual` if they were not already. This prevents the same race that was fixed for descriptors in cl/874084218.
PiperOrigin-RevId: 952273589
* Updates edition to 2026 in the unittest protos.
* Removes the obsolete target compile option `cc_enable_arenas` since C++
options are moved and arenas are enabled by default.
* Updates `maximum_edition` target constraints to edition 2026 in Java,
Python, C#, and upb build targets to allow loading and validation of
Edition 2026 files.
* Regenerates internal_options_bootstrap compiler files.
PiperOrigin-RevId: 944424868
## Bug
`Clear()` in `map_container.cc` (line 294-302) calls
`reflection->ClearField()` which destroys all underlying map nodes via
`ClearTable(reset=true)`, but does not increment `self->version`.
All other mutators (ScalarMapSetItem, MessageMapSetItem, MergeFrom, etc.)
increment `self->version` after mutation. `IterNext()` relies on version
mismatch to detect concurrent modification and raise `RuntimeError`.
Without the version bump, a live iterator proceeds to dereference the
freed `NodeBase*` via `SetMapIteratorValue` → `UntypedMapIterator::PlusPlus`.
**ASAN confirmed:** heap-use-after-free, READ size 8 at
`UntypedMapIterator::PlusPlus` (map.h:599), freed by `ClearTable`
(map.h:345), allocated by `ScalarMapSetItem` (map_container.cc:416).
## Fix
Add `self->version++` after `ClearField` in `Clear()`, matching every
other mutator in the same file.
## Reproducer
```python
msg = M() # proto3 with map<string, int32> mp
for k in ("a","b","c","d"): msg.mp[k] = 1
it = iter(msg.mp)
next(it)
msg.mp.clear() # frees nodes, version NOT bumped
next(it) # heap-use-after-free
```
Closes#27257
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/27257 from vhullto:fix/python-map-clear-uaf fb35225110
PiperOrigin-RevId: 939482747
When cpython_bits.type_getattro(self, name) returns NULL due to an exception raised in a descriptor (such as KeyboardInterrupt, MemoryError, or SystemExit), PyUpb_MessageMeta_GetAttr previously cleared the error and raised AttributeError.
Check PyErr_ExceptionMatches(PyExc_AttributeError) before clearing the error to ensure non-AttributeError exceptions are properly propagated.
PiperOrigin-RevId: 938246969
When PyObject_GetAttrString returns NULL (e.g., because __name__ or __module__ attribute access raises an exception on metaclass), PyUpb_GetStrData returns NULL. Passing NULL to strcmp previously caused a segmentation fault.
Check for NULL before calling PyUpb_GetStrData and strcmp, and call PyErr_Clear() when attribute lookup fails.
PiperOrigin-RevId: 938039167
PyTuple_Pack increments the reference count of its arguments. When PyTuple_Pack(2, start, end) was passed directly to PyList_SetItem, the start and end PyLong objects were leaked on every call.
Store the tuple created by PyTuple_Pack and DECREF start and end before inserting into the list.
PiperOrigin-RevId: 937601625
The pure Python implementation of maps had a bug where looking up an entry in a `map<string, ValueType>` field using a `bytes` key would cause silent data corruption.
**Bug:**
1. `msg.my_map['foo'] = 100` stores `{'foo': 100}`.
2. `msg.my_map[b'foo']` attempts a lookup.
3. `self._values[b'foo']` raises `KeyError` as `b'foo' != 'foo'`.
4. The `except KeyError` block normalizes `b'foo'` to `'foo'`.
5. Crucially, it then inserts a *default value* for the value type, overwriting the existing entry: `self._values['foo'] = 0`.
6. The lookup returns 0, and the original value of 100 is lost.
**Fix:**
The key is now normalized using `self._key_checker.CheckValue(key)` *at the beginning* of `__getitem__`, `__contains__`, `get`, `__delitem__`, and `setdefault`. This ensures the key is in the canonical `str` format *before* any dictionary access, preventing the erroneous write-on-miss.
This change makes the behavior consistent with the C++ and upb implementations.
PiperOrigin-RevId: 930498495
Avoid calling ByteSize() separately in serialize_length_prefixed(). The serialized payload already carries the exact length to prefix, so this keeps behavior intact while avoiding a duplicate serialization-sized pass.
Closes#27252
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/27252 from Zaczero:python-serialize-length-prefixed-once 8a4f60b4c7
PiperOrigin-RevId: 923539199
This test highlights a behavioral difference where upb allows equality comparison between a RepeatedCompositeContainer and a Python list, while the cpp and python implementations raise a TypeError.
PiperOrigin-RevId: 912754358
Add test coverage of 'assign string to enum-typed field' in general (including this case). This appears to be accepted only Py-upb and not otherwise.
Fixes https://github.com/protocolbuffers/protobuf/issues/27106
PiperOrigin-RevId: 908148411
This change introduces a FreeThreadingMutex to PyDescriptorPool to guard access to the descriptor_options and descriptor_features hash maps.
PiperOrigin-RevId: 908115178
