Commit graph

87 commits

Author SHA1 Message Date
Runze Wang
be53faf37d internal change
PiperOrigin-RevId: 930072020
2026-06-10 14:17:37 -07:00
Runze Wang
f5738bb5f0 [py/pure python] Implement Make GetOptions() return immutable options. Raise a TypeError when options returned GetOptions() by is mutated.
PiperOrigin-RevId: 926114776
2026-06-03 10:20:31 -07:00
Protobuf Team Bot
af47fbb369 Improve serialization performance in the case of recursive maps in pure Python Protobuf.
PiperOrigin-RevId: 910887857
2026-05-05 13:45:18 -07:00
Protobuf Team Bot
6cd6ca4116 Auto-format Py Proto python files.
PiperOrigin-RevId: 907755617
2026-04-29 13:40:58 -07:00
Jie Luo
b4c3fecb85 Add recursion guards for the following nested messages:
- map field for pure Python
- message_set_extension for Pure Python
- message_set_extension for UPB Python
https://github.com/protocolbuffers/protobuf/issues/25335

PiperOrigin-RevId: 868863633
2026-02-11 15:13:37 -08:00
Jie Luo
1eb4e5290a Python Proto scalar repeated numpy binding.
Add __array__ magic method to scalar repeated field. The default returned dtype of numpy/tensorflow usages like np.arrary() np.asarray() will be change.
Cpp_Type | np.array() dtype | Old tf.reshape() dtype | New dtype
float.   | float64/float64  | float32/float32.       | float32
double.  | float64/float64. | float32/float32.       | float64
int32.   | int64/float64.   | int32/float32.         | int32
int64.   | int64/float64.   | int32/float32.         | int64
uint32.  | int64/float64.   | int32/float32.         | uint32
uint64.  | int64/float64.   | int32/float32.         | uint64
bool.    | bool/float64.    | bool/float32.          | bool
enum     | int64/float64    | int32/float32          | int32
string   | U/float64        | string/float32         | U (Unicode string)
bytes    | S/float64        | string/float32.        | S (Byte string)

Performance improvement:
For non-string repeated:
  -upb/cpp are about 50-100 times faster.
  -pure python is ~3x times faster.
For string repeated: numbers are noise, no regression (maybe a slight win)

PiperOrigin-RevId: 836264273
2025-11-24 09:59:08 -08:00
Jie Luo
5f6c0139a7 Add construction support for repeated Timestamp/Duration/Struct/ListValue.
-Users can now assign list datetime/temedelta to repeated Timestamp/Duration in construction.
-Can assign list dictionary to repeated Struct.
-Can assign list of list to repeated ListValue.

https://github.com/protocolbuffers/protobuf/issues/21541

PiperOrigin-RevId: 804523563
2025-09-08 11:47:20 -07:00
Mike Kruskal
1af7fd407c Extend gencode compatibility support back to 3.20.0
#test-continuous

This adds 2 shims for some previous breakages, and compatibility tests to avoid further accidents.  This will be backported to 25.x and 29.x.

PiperOrigin-RevId: 776414924
2025-06-26 21:48:03 -07:00
Protobuf Team Bot
f69180a6c1 Remove calls to Label within Protobuf Python.
PiperOrigin-RevId: 756880197
2025-05-09 13:16:44 -07:00
Aaron Jacobs
ac4a50be6c Add Python deprecation warnings for Descriptor Label. Also, minimize calls to Label within Protobuf.
PiperOrigin-RevId: 756720481
2025-05-09 05:15:24 -07:00
Protobuf Team Bot
4f2ccc207d Add Python deprecation warnings for Descriptor Label. Also, minimize calls to Label within Protobuf.
PiperOrigin-RevId: 756631443
2025-05-09 00:09:00 -07:00
Protobuf Team Bot
5bd7d02ed8 Deprecate Descriptor Label. As an alternative, add helper methods for checking whether a field is required or repeated.
This is for C++, Java, and Python.

PiperOrigin-RevId: 740152256
2025-03-24 19:12:18 -07:00
Protobuf Team Bot
a6fa5799ff Internal pure python fixes
PiperOrigin-RevId: 733441339
2025-03-04 13:30:02 -08:00
Protobuf Team Bot
6235687246 Python protobuf: kwargs initialization now allows dict() for proto map values.
It is now possible to initialize a field of type `map<string, Message>` with nested dict objects:
```
storage_pb2.Directory(
  name='/home/user',
  owner='user',
  content={
    '.bashrc': dict(    # <=== dict() allowed here
      size=1234,
      permissions='PRIVATE',
    ),
  },
)
```

PiperOrigin-RevId: 719437601
2025-01-24 14:42:07 -08:00
Mike Kruskal
bc16fe8921 Treat -0.0 as consistently distinct from 0.0 in pure python
The previous treatment was a conformance violation, where implicit present float fields with a non-default value of -0.0 could get dropped.

PiperOrigin-RevId: 705728806
2024-12-12 20:34:44 -08:00
Mike Kruskal
1b0b1c9ba0 Remove decoder.SkipField.
This is only used in some narrow edge cases, and is a less-safe version of our unknown field decoders.  Switching to those reduces some duplication and improves error handling.

PiperOrigin-RevId: 704899254
2024-12-10 17:21:38 -08:00
Mike Kruskal
f69ea1cc48 Make Pure Python reject unmatched end-group tag in unknown fields
This brings it into conformance with our spec and other languages.

PiperOrigin-RevId: 704518974
2024-12-09 20:12:03 -08:00
Jie Luo
47613cf7ad Fix a python bug when assign an empty Struct at creation.
PiperOrigin-RevId: 689211445
2024-10-23 20:34:59 -07:00
Nathan Baulch
e4cbc79ab4 Fix minor typos (#17682)
Just thought I'd contribute some typo fixes that keep tripping up CI/CD checks in my projects. Nothing controversial (hopefully), just 174 simple fixes.

Use the following command to get a quick and dirty summary of the specific corrections made:
```shell
git diff HEAD^! --word-diff-regex='\w+' -U0 \
  | grep -E '\[\-.*\-\]\{\+.*\+\}' \
  | sed -r 's/.*\[\-(.*)\-\]\{\+(.*)\+\}.*/\1 \2/' \
  | sort | uniq -c | sort -n
```

FWIW, the top typos are:
* trimed → trimmed (37)
* substract → subtract (7)
* qualifed → qualified (7)
* extesion → extension (6)
* mising → missing (5)
* btye → byte (4)
* likey → likely (4)
* candicate → candidate (3)
* decriptor → descriptor (3)
* inherting → inheriting (3)
* colletion → collection (3)
* caluclated → calculated (3)
* unititialized → uninitialized (3)
* implemting → implementing (3)
* binrary → binary (3)
* descripor → descriptor (3)
* negitive → negative (3)

Closes #17682

COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/17682 from NathanBaulch:typos d41762d137
PiperOrigin-RevId: 677074418
2024-09-20 20:50:06 -07:00
Jie Luo
c37dd57a70 Remove deprecated GetPrototype usages in google/protobuf/ to git rid of the
warinings

PiperOrigin-RevId: 654836977
2024-07-22 11:42:37 -07:00
Jie Luo
e17821cac1 Nextgen Proto Pythonic API: Struct/ListValue assignment and creation
Python dict is now able to be assigned (by create and copy, not reference) and compared with the Protobuf Struct field.
Python list is now able to be assigned (by create and copy, not reference) and compared with the Protobuf ListValue field.

example usage:
  dictionary = {'key1': 5.0, 'key2': {'subkey': 11.0, 'k': False},}
  list_value = [6, 'seven', True, False, None, dictionary]
  msg = more_messages_pb2.WKTMessage(
      optional_struct=dictionary, optional_list_value=list_value
  )
  self.assertEqual(msg.optional_struct, dictionary)
  self.assertEqual(msg.optional_list_value, list_value)

PiperOrigin-RevId: 646099987
2024-06-24 08:41:33 -07:00
Jie Luo
b690e729eb Nextgen Proto Pythonic API: Timestamp/Duration assignment, creation and calculation
Timestamp and Duration are now have more support with datetime and timedelta:
- Allows assign python datetime to protobuf DateTime field in addition to current FromDatetime/ToDatetime (Note: will throw exceptions for the differences in supported ranges)
- Allows assign python timedelta to protobuf Duration field in addition to current FromTimedelta/ToTimedelta
- Calculation between Timestamp, Duration, datetime and timedelta will also be supported.

example usage:

from datetime import datetime, timedelta
from event_pb2 import Event
e = Event(start_time=datetime(year=2112, month=2, day=3),
          duration=timedelta(hours=10))
duration = timedelta(hours=10))
end_time = e.start_time + timedelta(hours=4)
e.duration = end_time - e.start_time
PiperOrigin-RevId: 640639168
2024-06-05 13:55:39 -07:00
Jie Luo
24f27c3b88 Nextgen Proto Pythonic API: Add 'in' operator
(Second attempt. The first attempt missed ListValue)

The “in” operator will be consistent with HasField but a little different with Proto Plus.

The detail behavior of “in” operator in Nextgen

* For WKT Struct (to be consist with old Struct behavior):
    -Raise TypeError if not pass a string
    -Check if the key is in the struct.fields

* For WKT ListValue (to be consist with old behavior):
    -Check if the key is in the list_value.values

* For other messages:
    -Raise ValueError if not pass a string
    -Raise ValueError if the string is not a field
    -For Oneof: Check any field under the oneof is set
    -For has-presence field: check if set
    -For non-has-presence field (include repeated fields): raise ValueError

PiperOrigin-RevId: 631143378
2024-05-06 12:13:09 -07:00
Jie Luo
baf211bf63 Automated rollback of commit de8e550e90.
PiperOrigin-RevId: 621326794
2024-04-02 16:28:26 -07:00
Jie Luo
de8e550e90 Nextgen Proto Pythonic API: Add 'in' operator
The “in” operator will be consistent with HasField but a little different with Proto Plus.

The detail behavior of “in” operator in Nextgen for Struct (to be consist with old Struct behavior):
-Raise TypeError if not pass a string
-Check if the key is in the struct.fields

The detail behavior of “in” operator in Nextgen(for other message):
-Raise ValueError if not pass a string
-Raise ValueError if the string is not a field
-For Oneof: Check any field under the oneof is set
-For has-presence field: check if set
-For non-has-presence field (include repeated fields): raise ValueError

PiperOrigin-RevId: 621240977
2024-04-02 11:32:18 -07:00
Jie Luo
eee53125c4 internal change
PiperOrigin-RevId: 613982814
2024-03-08 10:55:53 -08:00
Jie Luo
4ebba684c7 BREAKING CHANGE in v26: Remove RegisterExtension in message class
PiperOrigin-RevId: 595989309
2024-01-05 07:25:56 -08:00
Jie Luo
0eac77c2c6 Breaking change in 26 release: Remove msg.UnknownFields() support in pure python and cpp extension.
Users should use the add-on unknown_fields.py support.
Old usage example:
unknown_field_set = msg.UnknownFields()

New usage should be:
from google.protobuf import unknown_fields
unknown_field_set = unknown_fields.UnknownFieldSet(msg)
PiperOrigin-RevId: 589969095
2023-12-11 15:25:41 -08:00
Sandy Zhang
81068e8e8c Internal change
PiperOrigin-RevId: 566426899
2023-09-18 15:17:56 -07:00
Joshua Haberman
9ef8f5cffc Shorten our license headers into an abbreviated form that references LICENSE instead of including it in full.
PiperOrigin-RevId: 563897888
2023-09-08 18:50:27 -07:00
Jie Luo
3e966f1dc9 Add lock to pure python's field decoders
Fix data race that may increase flake rate of some tests.

PiperOrigin-RevId: 562044867
2023-09-01 14:15:10 -07:00
Protobuf Team Bot
12d4f418a7 Comparing a proto message with an object of unknown returns NotImplemented
The Python comparison protocol requires that if an object doesn't know how to
compare itself to an object of a different type, it returns NotImplemented
rather than False. The interpreter will then try performing the comparison using
the other operand. This translates, for protos, to:
If a proto message doesn't know how to compare itself to an object of
non-message type, it returns NotImplemented. This way, the interpreter will then
try performing the comparison using the comparison methods of the other object,
which may know how to compare itself to a message. If not, then Python will
return the combined result (e.g., if both objects don't know how to perform
__eq__, then the equality operator `==` return false).
This change allows one to compare a proto with custom matchers such as mock.ANY
that the message doesn't know how to compare to, regardless of whether
mock.ANY is on the right-hand side or left-hand side of the equality (prior to
this change, it only worked with mock.ANY on the left-hand side).

Fixes https://github.com/protocolbuffers/protobuf/issues/9173

PiperOrigin-RevId: 561728156
2023-08-31 12:32:18 -07:00
Jie Luo
abf5dfbfbc Fix a bug that strips options from descriptor.proto in Pure Python.
GetOptions on fields (which parse the _serialized_options) will be called for the first time of parse or serialize instead of Build time.

Note: GetOptions on messages are still called in Build time because of message_set_wire_format. If message options are needed in descriptor.proto, a parse error will be raised in GetOptions(). We can check the file to not invoke GetOptions() for descriptor.proto as long as message_set_wire_format not needed in descriptor.proto.

Other options except message options do not invoke GetOptions() in Build time

PiperOrigin-RevId: 560741182
2023-08-28 10:00:45 -07:00
Jie Luo
161989080b Fix wrong warning message info
PiperOrigin-RevId: 554629697
2023-08-07 16:43:40 -07:00
Protobuf Team Bot
0cf1ea48d4 Fix typo in warning about the deprecated use of message.UnknownFields().
The message.UnknownFields() support will be removed soon.

PiperOrigin-RevId: 554598474
2023-08-07 14:46:22 -07:00
Jie Luo
c988759a9b Raise warings for message.UnknownFields() usages and navigate to the new add
one API

The message.UnknownFields() support will be removed soon

PiperOrigin-RevId: 553871453
2023-08-04 11:59:29 -07:00
Jie Luo
862a2281ce Remove AddHelpersToExtenions() in generated code
PiperOrigin-RevId: 536439101
2023-05-30 10:20:55 -07:00
Jie Luo
86fc32cedc Removed message._extensions_by_number
PiperOrigin-RevId: 534514738
2023-05-23 12:21:02 -07:00
Jie Luo
e737c8108a Removed message._extensions_by_name
PiperOrigin-RevId: 533501166
2023-05-19 11:10:25 -07:00
Jie Luo
f75fd051d6 Change extension_handle to field_descriptor in python HasExtension() and ClearExtension()
PiperOrigin-RevId: 499349711
2023-01-03 17:07:15 -08:00
Jie Luo
3d3bebf8ab Internal change only
PiperOrigin-RevId: 491993057
2022-11-30 12:50:31 -08:00
Jie Luo
da9de8d4d4 Added is_closed to EnumDescriptor in protobuf python
PiperOrigin-RevId: 486274963
2022-11-04 18:34:08 -07:00
Jie Luo
9b725fb9a3 use FieldDescriptor.has_presence instead of file.syntax in python_message.py
PiperOrigin-RevId: 486254134
2022-11-04 16:17:40 -07:00
Jie Luo
5fc9c37208 Internal change
PiperOrigin-RevId: 485680539
2022-11-02 13:32:35 -07:00
Mike Kruskal
a29e0e3f44 Sync from Piper @439400376
PROTOBUF_SYNC_PIPER
2022-04-04 13:42:37 -07:00
Joshua Haberman
e5c570bb57 Sync from Piper @395706834
PROTOBUF_SYNC_PIPER
2021-09-09 08:21:42 -07:00
Alexander Shadchin
87dd07b436
Switch on "new" buffer API (#8339)
"Old" buffer API will removed in Python 3.10. This is also fix #7930.
2021-03-04 11:00:41 -08:00
Joshua Haberman
f77065d4eb Sync from Piper @316511779
PROTOBUF_SYNC_PIPER
2020-06-15 11:48:47 -07:00
Joshua Haberman
b7742c51fd Sync from Piper @305505267
PROTOBUF_SYNC_PIPER
2020-04-08 10:30:17 -07:00
Joshua Haberman
c649397029
Set execute bit on files if and only if they begin with (#!). (#7347)
* Set execute bit on files if and only if they begin with (#!).

Git only tracks the 'x' (executable) bit on each file. Prior to this
CL, our files were a random mix of executable and non-executable.
This change imposes some order by making files executable if and only
if they have shebang (#!) lines at the beginning.

We don't have any executable binaries checked into the repo, so
we shouldn't need to worry about that case.

* Added fix_permissions.sh script to set +x iff a file begins with (#!).
2020-04-01 15:28:25 -07:00