This implementation is similar to our Java idea in cl/949670944,
basically the idea is that we will have a local map (dictionary) that
will be populated the very first time we try to parse into an enum
field. The lifetime of this dictionary will be tied to the Parser
instance, similar to the Java implementation.
BENCHMARKS (baseline stats are recorded at cl/952170250):
Python:
```
[BENCHMARK] ParseJsonDefault: med 10.06 us/op | p99 11.22 us/op | mean 10.10 ± 0.20 us/op
[BENCHMARK] ParseJsonCustom: med 20.33 us/op | p99 22.23 us/op | mean 20.40 ± 0.34 us/op
[BENCHMARK] ParseJsonUnknownIgnored: med 24.21 us/op | p99 26.25 us/op | mean 24.31 ± 0.44 us/op
[BENCHMARK] ParseRepeatedJsonDefault: med 1.94 us/item | p99 2.05 us/item | mean 1.95 ± 0.03 us/item
[BENCHMARK] ParseRepeatedJsonCustom: med 2.26 us/item | p99 2.45 us/item | mean 2.27 ± 0.03 us/item
[BENCHMARK] ParseRepeatedJsonUnknownIgnored: med 5.31 us/item | p99 5.52 us/item | mean 5.32 ± 0.05 us/item
```
Cpp:
```
[BENCHMARK] ParseJsonDefault: med 8.66 us/op | p99 9.21 us/op | mean 8.70 ± 0.13 us/op
[BENCHMARK] ParseJsonCustom: med 26.33 us/op | p99 29.71 us/op | mean 26.44 ± 0.69 us/op
[BENCHMARK] ParseJsonUnknownIgnored: med 28.46 us/op | p99 31.80 us/op | mean 28.59 ± 0.67 us/op
[BENCHMARK] ParseRepeatedJsonDefault: med 1.72 us/item | p99 2.19 us/item | mean 1.74 ± 0.06 us/item
[BENCHMARK] ParseRepeatedJsonCustom: med 2.12 us/item | p99 2.38 us/item | mean 2.13 ± 0.04 us/item
[BENCHMARK] ParseRepeatedJsonUnknownIgnored: med 4.61 us/item | p99 4.92 us/item | mean 4.62 ± 0.07 us/item
```
UPB:
```
[BENCHMARK] ParseJsonDefault: med 11.22 us/op | p99 12.29 us/op | mean 11.27 ± 0.21 us/op
[BENCHMARK] ParseJsonCustom: med 37.54 us/op | p99 39.25 us/op | mean 37.65 ± 0.46 us/op
[BENCHMARK] ParseJsonUnknownIgnored: med 41.18 us/op | p99 48.45 us/op | mean 41.30 ± 0.89 us/op
[BENCHMARK] ParseRepeatedJsonDefault: med 2.10 us/item | p99 2.19 us/item | mean 2.10 ± 0.02 us/item
[BENCHMARK] ParseRepeatedJsonCustom: med 2.58 us/item | p99 2.73 us/item | mean 2.59 ± 0.03 us/item
[BENCHMARK] ParseRepeatedJsonUnknownIgnored: med 5.70 us/item | p99 5.90 us/item | mean 5.71 ± 0.04 us/item
```
Seeing these benchmarks, we can see a very noticeable improvement in
`ParseRepeatedJsonCustom` and `ParseRepeatedJsonUnknownIgnored` cases --
which correspond to the pathological cases that we are worried about.
Singular fields are taking longer to parse in our microbenchmark because
the cache that we've added is tied to each Parser instance and must be
re-instantiated across every `Parse` call.
PiperOrigin-RevId: 970528683
The unit tests added in the initial implementation in cl/925012500 and
cl/922729754 were fairly minimal. I took some inspiration from our C++
and Java unit tests for JSON's custom enum name parsing to come up with
these new test cases.
This change also adds a python Benchmark that we can use to verify that
our optimizations in cl/952170251 actually works.
Benchmark results across all three proto implementations (cpp, python, upb):
=== C++ Extension Protos (json_format_benchmark_cpp_protos) ===
[BENCHMARK] ParseJsonDefault: med 8.70 us/op | p99 9.56 us/op | mean 8.74 ± 0.15 us/op
[BENCHMARK] ParseJsonCustom: med 17.39 us/op | p99 21.64 us/op | mean 17.43 ± 0.51 us/op
[BENCHMARK] ParseJsonUnknownIgnored: med 26.97 us/op | p99 32.04 us/op | mean 27.06 ± 0.65 us/op
[BENCHMARK] ParseRepeatedJsonDefault: med 1.71 us/item | p99 1.83 us/item | mean 1.71 ± 0.02 us/item
[BENCHMARK] ParseRepeatedJsonCustom: med 9.18 us/item | p99 9.88 us/item | mean 9.20 ± 0.11 us/item
[BENCHMARK] ParseRepeatedJsonUnknownIgnored: med 18.79 us/item | p99 19.19 us/item | mean 18.81 ± 0.13 us/item
=== UPB Extension Protos (json_format_benchmark_upb_protos) ===
[BENCHMARK] ParseJsonDefault: med 8.70 us/op | p99 9.56 us/op | mean 8.74 ± 0.15 us/op
[BENCHMARK] ParseJsonCustom: med 17.39 us/op | p99 21.64 us/op | mean 17.43 ± 0.51 us/op
[BENCHMARK] ParseJsonUnknownIgnored: med 26.97 us/op | p99 32.04 us/op | mean 27.06 ± 0.65 us/op
[BENCHMARK] ParseRepeatedJsonDefault: med 1.71 us/item | p99 1.83 us/item | mean 1.71 ± 0.02 us/item
[BENCHMARK] ParseRepeatedJsonCustom: med 9.18 us/item | p99 9.88 us/item | mean 9.20 ± 0.11 us/item
[BENCHMARK] ParseRepeatedJsonUnknownIgnored: med 18.79 us/item | p99 19.19 us/item | mean 18.81 ± 0.13 us/item
=== Pure Python Protos (json_format_benchmark_python_protos) ===
[BENCHMARK] ParseJsonDefault: med 10.26 us/op | p99 11.20 us/op | mean 10.30 ± 0.25 us/op
[BENCHMARK] ParseJsonCustom: med 16.59 us/op | p99 18.35 us/op | mean 16.70 ± 0.45 us/op
[BENCHMARK] ParseJsonUnknownIgnored: med 25.09 us/op | p99 28.22 us/op | mean 25.27 ± 0.65 us/op
[BENCHMARK] ParseRepeatedJsonDefault: med 1.94 us/item | p99 3.25 us/item | mean 1.97 ± 0.14 us/item
[BENCHMARK] ParseRepeatedJsonCustom: med 7.16 us/item | p99 10.35 us/item | mean 7.24 ± 0.39 us/item
[BENCHMARK] ParseRepeatedJsonUnknownIgnored: med 15.43 us/item | p99 19.54 us/item | mean 15.54 ± 0.49 us/item
PiperOrigin-RevId: 966820541
* `protoc`: Included in zips alongside binary
* Java: Generated classes have been added to the Maven `util` artifact
* Ruby: Generated classes have been added to gems
* CMake: `.proto` files are included in installation
PiperOrigin-RevId: 957382561
(cherry picked from commit 713b6a4a24)
* Updates edition to 2026 in the unittest protos.
* Removes the obsolete target compile option `cc_enable_arenas` since C++
options are moved and arenas are enabled by default.
* Updates `maximum_edition` target constraints to edition 2026 in Java,
Python, C#, and upb build targets to allow loading and validation of
Edition 2026 files.
* Regenerates internal_options_bootstrap compiler files.
PiperOrigin-RevId: 944424868
Fixes https://github.com/protocolbuffers/protobuf/issues/28261 by setting symbol visibility in `setup.py`, similar to `py_extension.bzl`, using the existing branching logic to bypass the flag on MSVC.
This issue manifests in the [NixOS/nixpkgs](https://github.com/NixOS/nixpkgs) build of the `protobuf` python package, which contains upb symbols, and causes fatal failures when loaded in a process that separately links/loads libprotobuf. We'll supply a patch there as well, until this can be backported or released in future protobuf versions.
Closes#28264
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/28264 from mitchallain:mallain/fix-python-upb-symbols 6e118a62bf
PiperOrigin-RevId: 942223178
## Bug
`Clear()` in `map_container.cc` (line 294-302) calls
`reflection->ClearField()` which destroys all underlying map nodes via
`ClearTable(reset=true)`, but does not increment `self->version`.
All other mutators (ScalarMapSetItem, MessageMapSetItem, MergeFrom, etc.)
increment `self->version` after mutation. `IterNext()` relies on version
mismatch to detect concurrent modification and raise `RuntimeError`.
Without the version bump, a live iterator proceeds to dereference the
freed `NodeBase*` via `SetMapIteratorValue` → `UntypedMapIterator::PlusPlus`.
**ASAN confirmed:** heap-use-after-free, READ size 8 at
`UntypedMapIterator::PlusPlus` (map.h:599), freed by `ClearTable`
(map.h:345), allocated by `ScalarMapSetItem` (map_container.cc:416).
## Fix
Add `self->version++` after `ClearField` in `Clear()`, matching every
other mutator in the same file.
## Reproducer
```python
msg = M() # proto3 with map<string, int32> mp
for k in ("a","b","c","d"): msg.mp[k] = 1
it = iter(msg.mp)
next(it)
msg.mp.clear() # frees nodes, version NOT bumped
next(it) # heap-use-after-free
```
Closes#27257
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/27257 from vhullto:fix/python-map-clear-uaf fb35225110
PiperOrigin-RevId: 939482747
When cpython_bits.type_getattro(self, name) returns NULL due to an exception raised in a descriptor (such as KeyboardInterrupt, MemoryError, or SystemExit), PyUpb_MessageMeta_GetAttr previously cleared the error and raised AttributeError.
Check PyErr_ExceptionMatches(PyExc_AttributeError) before clearing the error to ensure non-AttributeError exceptions are properly propagated.
PiperOrigin-RevId: 938246969
When PyObject_GetAttrString returns NULL (e.g., because __name__ or __module__ attribute access raises an exception on metaclass), PyUpb_GetStrData returns NULL. Passing NULL to strcmp previously caused a segmentation fault.
Check for NULL before calling PyUpb_GetStrData and strcmp, and call PyErr_Clear() when attribute lookup fails.
PiperOrigin-RevId: 938039167
PyTuple_Pack increments the reference count of its arguments. When PyTuple_Pack(2, start, end) was passed directly to PyList_SetItem, the start and end PyLong objects were leaked on every call.
Store the tuple created by PyTuple_Pack and DECREF start and end before inserting into the list.
PiperOrigin-RevId: 937601625
The pure Python implementation of maps had a bug where looking up an entry in a `map<string, ValueType>` field using a `bytes` key would cause silent data corruption.
**Bug:**
1. `msg.my_map['foo'] = 100` stores `{'foo': 100}`.
2. `msg.my_map[b'foo']` attempts a lookup.
3. `self._values[b'foo']` raises `KeyError` as `b'foo' != 'foo'`.
4. The `except KeyError` block normalizes `b'foo'` to `'foo'`.
5. Crucially, it then inserts a *default value* for the value type, overwriting the existing entry: `self._values['foo'] = 0`.
6. The lookup returns 0, and the original value of 100 is lost.
**Fix:**
The key is now normalized using `self._key_checker.CheckValue(key)` *at the beginning* of `__getitem__`, `__contains__`, `get`, `__delitem__`, and `setdefault`. This ensures the key is in the canonical `str` format *before* any dictionary access, preventing the erroneous write-on-miss.
This change makes the behavior consistent with the C++ and upb implementations.
PiperOrigin-RevId: 930498495
Avoid calling ByteSize() separately in serialize_length_prefixed(). The serialized payload already carries the exact length to prefix, so this keeps behavior intact while avoiding a duplicate serialization-sized pass.
Closes#27252
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/27252 from Zaczero:python-serialize-length-prefixed-once 8a4f60b4c7
PiperOrigin-RevId: 923539199
Previously we had a lock in the global object cache (an instance of `PyUpb_WeakMap`), but this CL refactors `PyUpb_WeakMap` to be internally synchronized.
We also add code to handle a tricky edge case that can occur when a dealloc operation races with lookup. This implementation mirrors the code previously implemented in C++ (`PyWeakValueMap`).
PiperOrigin-RevId: 921534474
