Commit graph

1090 commits

Author SHA1 Message Date
Joshua Haberman
feaa31c4d7 [Py/FreeThreading] Fixed remaining race conditions in Dealloc()
This change modifies all remaining `Dealloc()` functions to use `EraseIfEqual` if they were not already. This prevents the same race that was fixed for descriptors in cl/874084218.

PiperOrigin-RevId: 952273589
2026-07-22 12:43:18 -07:00
Samuel Benzaquen
9bb4aebaf9 Replace calls of Reflection::GetEnum/GetRepeatedEnum with GetEnumValue/GetRepeatedEnumValue when we only care about the value.
It is more efficient this way.

PiperOrigin-RevId: 951514413
2026-07-21 08:49:12 -07:00
Runze Wang
4db27e1f33 Add UPB support for FileDescriptorProto.option_dependency.
This change adds API and implementation to read and write the option_dependency field from/to proto2.FileDescriptorProto in upb reflection.

PiperOrigin-RevId: 947120641
2026-07-13 10:43:20 -07:00
Runze Wang
52e82c810c Emit future warning when mutating GetOptions() in OSS
PiperOrigin-RevId: 946657065
2026-07-12 13:41:24 -07:00
Runze Wang
927c504eea Emit deprecation warning when comparing descriptor containers with unrecognized types in OSS
PiperOrigin-RevId: 945830705
2026-07-10 12:16:37 -07:00
rgoldfinger6
4b269eb436 Updating version.json and repo version numbers to: 37.0-dev (#28425)
NOTE: This should be reviewed and imported via Copybara per the normal PR review process.

Closes #28425

PiperOrigin-RevId: 945675238
2026-07-10 06:22:59 -07:00
Jason Aragorn Tobias Lunn
271ca53b50 Migrate top-level protobuf unittest protos from Edition 2024 to Edition 2026.
* Updates edition to 2026 in the unittest protos.
* Removes the obsolete target compile option `cc_enable_arenas` since C++
  options are moved and arenas are enabled by default.
* Updates `maximum_edition` target constraints to edition 2026 in Java,
  Python, C#, and upb build targets to allow loading and validation of
  Edition 2026 files.
* Regenerates internal_options_bootstrap compiler files.

PiperOrigin-RevId: 944424868
2026-07-08 04:38:22 -07:00
Charlie Beattie
4f5350ef98 Internal change.
PiperOrigin-RevId: 943935338
2026-07-07 09:27:46 -07:00
Protobuf Team Bot
27a0b8b687 Automated Code Change
PiperOrigin-RevId: 943864493
2026-07-07 06:47:10 -07:00
Runze Wang
11c34350bb Format descriptor_pool.cc to eliminate pre-existing lint issues.
PiperOrigin-RevId: 943655751
2026-07-06 21:38:41 -07:00
Hana Joo
ebedfb958e Automated Code Change
PiperOrigin-RevId: 941625858
2026-07-02 04:39:38 -07:00
Protobuf Team Bot
eaa8c8cca5 Automated Code Change
PiperOrigin-RevId: 941496740
2026-07-01 23:28:07 -07:00
Charlie Beattie
dc4d738909 Internal change.
PiperOrigin-RevId: 941222035
2026-07-01 12:07:16 -07:00
vhulto
9e9dbae858 Python: fix heap-use-after-free in MapIterator after map.clear() (#27257)
## Bug

`Clear()` in `map_container.cc` (line 294-302) calls
`reflection->ClearField()` which destroys all underlying map nodes via
`ClearTable(reset=true)`, but does not increment `self->version`.

All other mutators (ScalarMapSetItem, MessageMapSetItem, MergeFrom, etc.)
increment `self->version` after mutation. `IterNext()` relies on version
mismatch to detect concurrent modification and raise `RuntimeError`.
Without the version bump, a live iterator proceeds to dereference the
freed `NodeBase*` via `SetMapIteratorValue` → `UntypedMapIterator::PlusPlus`.

**ASAN confirmed:** heap-use-after-free, READ size 8 at
`UntypedMapIterator::PlusPlus` (map.h:599), freed by `ClearTable`
(map.h:345), allocated by `ScalarMapSetItem` (map_container.cc:416).

## Fix

Add `self->version++` after `ClearField` in `Clear()`, matching every
other mutator in the same file.

## Reproducer

```python
msg = M()  # proto3 with map<string, int32> mp
for k in ("a","b","c","d"): msg.mp[k] = 1
it = iter(msg.mp)
next(it)
msg.mp.clear()   # frees nodes, version NOT bumped
next(it)         # heap-use-after-free
```

Closes #27257

COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/27257 from vhullto:fix/python-map-clear-uaf fb35225110
PiperOrigin-RevId: 939482747
2026-06-28 13:43:09 -07:00
Runze Wang
abdf212e92 Add PyDescriptorPool_FromSharedPool(std::shared_ptr) API
This new API overload enables safe true co-ownership between the taken C++ pointer and the returned python pointer

PiperOrigin-RevId: 938697698
2026-06-26 11:48:30 -07:00
Runze Wang
9ac0b22526 Propagate non-AttributeError exceptions in PyUpb_MessageMeta_GetAttr
When cpython_bits.type_getattro(self, name) returns NULL due to an exception raised in a descriptor (such as KeyboardInterrupt, MemoryError, or SystemExit), PyUpb_MessageMeta_GetAttr previously cleared the error and raised AttributeError.

Check PyErr_ExceptionMatches(PyExc_AttributeError) before clearing the error to ensure non-AttributeError exceptions are properly propagated.

PiperOrigin-RevId: 938246969
2026-06-25 16:13:45 -07:00
Runze Wang
ae4f98fe70 Fix segfault via strcmp(NULL) in numpy type detection
When PyObject_GetAttrString returns NULL (e.g., because __name__ or __module__ attribute access raises an exception on metaclass), PyUpb_GetStrData returns NULL. Passing NULL to strcmp previously caused a segmentation fault.

Check for NULL before calling PyUpb_GetStrData and strcmp, and call PyErr_Clear() when attribute lookup fails.

PiperOrigin-RevId: 938039167
2026-06-25 09:52:49 -07:00
Runze Wang
3d31b0f786 Fix memory leak in PyUpb_Descriptor_GetExtensionRanges
PyTuple_Pack increments the reference count of its arguments. When PyTuple_Pack(2, start, end) was passed directly to PyList_SetItem, the start and end PyLong objects were leaked on every call.

Store the tuple created by PyTuple_Pack and DECREF start and end before inserting into the list.

PiperOrigin-RevId: 937601625
2026-06-24 15:49:13 -07:00
Charlie Beattie
a9c1c48f23 Internal change.
PiperOrigin-RevId: 937266383
2026-06-24 05:13:39 -07:00
Charlie Beattie
10046e9157 Internal Change.
PiperOrigin-RevId: 936784975
2026-06-23 11:20:48 -07:00
Charlie Beattie
2bda7ee751 Internal change
PiperOrigin-RevId: 933111478
2026-06-16 08:49:46 -07:00
Protobuf Team Bot
a7ab28ae44 Restore recursion limit in testRecursionMap to prevent test pollution
Without this, the test fails if it runs the test cases in a certain order.

PiperOrigin-RevId: 932553469
2026-06-15 10:56:42 -07:00
Charlie Beattie
ea40f1d32a Support assigning repeated scalar fields using Python Buffer API.
PiperOrigin-RevId: 932284662
2026-06-15 00:51:15 -07:00
Charlie Beattie
156a0b8dfc Fix Pure Python map<string, ...> key handling for bytes lookups.
The pure Python implementation of maps had a bug where looking up an entry in a `map<string, ValueType>` field using a `bytes` key would cause silent data corruption.

**Bug:**

1.  `msg.my_map['foo'] = 100` stores `{'foo': 100}`.
2.  `msg.my_map[b'foo']` attempts a lookup.
3.  `self._values[b'foo']` raises `KeyError` as `b'foo' != 'foo'`.
4.  The `except KeyError` block normalizes `b'foo'` to `'foo'`.
5.  Crucially, it then inserts a *default value* for the value type, overwriting the existing entry: `self._values['foo'] = 0`.
6.  The lookup returns 0, and the original value of 100 is lost.

**Fix:**

The key is now normalized using `self._key_checker.CheckValue(key)` *at the beginning* of `__getitem__`, `__contains__`, `get`, `__delitem__`, and `setdefault`. This ensures the key is in the canonical `str` format *before* any dictionary access, preventing the erroneous write-on-miss.

This change makes the behavior consistent with the C++ and upb implementations.

PiperOrigin-RevId: 930498495
2026-06-11 06:49:03 -07:00
Protobuf Team Bot
7ab792b969 Add link to the page which explains that deterministic != canonical on our apis which let you set determinsitic serialization
PiperOrigin-RevId: 930485402
2026-06-11 06:20:08 -07:00
Charlie Beattie
cdcf21dbf7 Support C++ Immutable map lookup.
PiperOrigin-RevId: 930421591
2026-06-11 03:36:58 -07:00
Runze Wang
be53faf37d internal change
PiperOrigin-RevId: 930072020
2026-06-10 14:17:37 -07:00
Jie Luo
e86e536349 Change Python DescriptorDatabase FindFileContainingSymbol() to accept leading "." to match DescriptorPool and other languages
PiperOrigin-RevId: 929451066
2026-06-09 15:58:00 -07:00
Israel Blancas
b6548fc510 python: mark proto_builder sha1 as non-security use (#27473)
Fixes #27472

Closes #27473

COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/27473 from iblancasa:27472 0ab662a73b
PiperOrigin-RevId: 929221750
2026-06-09 08:53:29 -07:00
Charlie Beattie
593ab176d6 Faster Python cpp protos assignment to bytes/string fields.
PiperOrigin-RevId: 928459028
2026-06-08 03:32:33 -07:00
Charlie Beattie
2906b1e81b Internal change.
PiperOrigin-RevId: 927241088
2026-06-05 05:35:17 -07:00
Runze Wang
eebe9b7737 [py/C++] Implement Make GetOptions() return immutable options. C++ will raise a TypeError when options returned GetOptions() by is mutated.
PiperOrigin-RevId: 926215827
2026-06-03 13:10:50 -07:00
Runze Wang
f5738bb5f0 [py/pure python] Implement Make GetOptions() return immutable options. Raise a TypeError when options returned GetOptions() by is mutated.
PiperOrigin-RevId: 926114776
2026-06-03 10:20:31 -07:00
Mike Kruskal
ad842574d8 Automated Code Change
PiperOrigin-RevId: 925070778
2026-06-01 20:28:56 -07:00
Hong Shin
99efd19cb0 json/python: remove pool arg from _GetJsonEnumValueOption
PiperOrigin-RevId: 925012500
2026-06-01 17:54:28 -07:00
Charlie Beattie
524871b9d1 Add Python repeated fields benchmarks.
PiperOrigin-RevId: 924877292
2026-06-01 13:22:33 -07:00
Kamil Monicz
e090f8ae77 Serialize length-prefixed protos once (#27252)
Avoid calling ByteSize() separately in serialize_length_prefixed(). The serialized payload already carries the exact length to prefix, so this keeps behavior intact while avoiding a duplicate serialization-sized pass.

Closes #27252

COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/27252 from Zaczero:python-serialize-length-prefixed-once 8a4f60b4c7
PiperOrigin-RevId: 923539199
2026-05-29 12:19:19 -07:00
Runze Wang
12baad3caa internal change
PiperOrigin-RevId: 922907524
2026-05-28 12:21:38 -07:00
Hong Shin
7800571253 json/python: add support for custom json strings for enum values
PiperOrigin-RevId: 922729754
2026-05-28 06:17:20 -07:00
Charlie Beattie
6dac2de5b6 Remove possible use after free via Python Buffer Objects in MergeFromString.
PiperOrigin-RevId: 921681380
2026-05-26 14:27:25 -07:00
Charlie Beattie
f7c66578cc Add assign_bytes Benchmarks.
PiperOrigin-RevId: 919863834
2026-05-22 14:36:41 -07:00
Runze Wang
403d429eab [py/upb] Make GetOptions() return immutable options for scalar type. UPB will raise a TypeError when options returned GetOptions() by is mutated.
PiperOrigin-RevId: 918559519
2026-05-20 11:33:26 -07:00
Amer Elsheikh
5f77738be4 Create an enum_type_wrapper.pyi stub with DESCRIPTOR: EnumDescriptor
PiperOrigin-RevId: 914243299
2026-05-12 06:14:01 -07:00
Joshua Haberman
ac90873fac Add test for RepeatedCompositeContainer comparison with list.
This test highlights a behavioral difference where upb allows equality comparison between a RepeatedCompositeContainer and a Python list, while the cpp and python implementations raise a TypeError.

PiperOrigin-RevId: 912754358
2026-05-08 16:43:32 -07:00
Joshua Haberman
6cbc7593bf Fixed race in GetMessageClass/RegisterMessageClass under free threading.
The race condition was resolved by adding a mutex lock to all accesses of the cache.

PiperOrigin-RevId: 912595698
2026-05-08 10:47:54 -07:00
Oblivionsage
94df512b87 fix: check AssureWritable() return value at all call sites (#26390)
Fixes #26389

`AssureWritable()` returns `-1` on failure (OOM in `MutableMessage()`), but 13 call sites were ignoring this return value.

When `AssureWritable()` fails, `self->message` still points to the shared read-only default instance. Subsequent mutations (`MergeFrom`, `CopyFrom`, `Clear`, `SetField`, etc.) then corrupt the shared global default instance, affecting all messages of the same type.

Same bug class as the recent `FixupMessageAfterMerge` fix (ab14c0f8a).

Closes #26390

COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/26390 from Oblivionsage:fix/check-assure-writable-return-value adbd20477e
PiperOrigin-RevId: 912044414
2026-05-07 10:59:13 -07:00
Protobuf Team Bot
66ee7113cf Automated rollback of commit 62074624a9.
PiperOrigin-RevId: 911144902
2026-05-06 00:07:32 -07:00
Joshua Haberman
62074624a9 [Py/C++ Optimization]: cache prototype in the CMessageClass
This avoids calling `GetPrototype()` every time a message is constructed, which would acquire a lock.

There is no need to look it up every time, because it will never change as long as the MessageFactory is alive.

PiperOrigin-RevId: 910918341
2026-05-05 14:35:54 -07:00
Protobuf Team Bot
af47fbb369 Improve serialization performance in the case of recursive maps in pure Python Protobuf.
PiperOrigin-RevId: 910887857
2026-05-05 13:45:18 -07:00
Protobuf Team Bot
5be0a43cd6 Avoid NULL pointer dereference in PyUpb_PyToUpbEnum if run with a malformed UTF-8 str (e.g. "\ud800")
Add test coverage of 'assign string to enum-typed field' in general (including this case). This appears to be accepted only Py-upb and not otherwise.

Fixes https://github.com/protocolbuffers/protobuf/issues/27106

PiperOrigin-RevId: 908148411
2026-04-30 07:01:58 -07:00