8c1a9a4b01
This CL concerns the following functions in C++:
```c++
class Reflection {
const Message& GetRepeatedMessage(const Message& message,
const FieldDescriptor* field,
int index) const;
Message* MutableRepeatedMessage(Message* message,
const FieldDescriptor* field,
int index) const;
}
```
Suppose a Python program contains the following code:
```
def foo(msg: MyMessage):
submsg = msg.repeated_foo[2] # (A)
if also_mutate:
submsg.abc = 1 # (B)
```
For line (A), we currently call `MutableRepeatedMessage()` in C++ to
obtain the submessage pointer, because the resulting `submsg` object in
Python is a conceptually mutable object that will require a non-const
pointer if/when the program hits line (B).
The `MutableRepeatedMessage(Message* msg, ...)` function in C++
currently mutates `msg` by setting the hasbit of the repeated field. But
this seems unnecessary, as the function requires that the requested
sub-message already exists; it does not create a new message. So it
should not be necessary to touch the hasbit of the message, and a TGP
confirms that all tests pass if we remove this call.
Once the `SetHasBitForRepeated()` call is removed, the `MutableRepeatedMessage()`
function no longer actually mutates the `msg` argument. This makes it
effectively safe to call concurrently (ie. it will no longer trigger
Undefined Behavior in C++, and the TSAN errors will go away), which
is why it fixes the free threading test. But it leaves us in an odd
state where we are still passing a non-`const` pointer to the same
message to two functions concurrently, which is not allowed under a
normal thread-compatible contract.
An alternate solution would be to call `GetRepeatedMessage()` instead,
and `const_cast<>` away the const in the returned `const Message&`.
But this is also violates the contract: users should not be casting
away `const`.
The root cause of this odd situation is that
`MutableRepeatedMessage(Message* msg)` requires a non-`const` `msg` not
because it actually *mutates* `msg`, but because it is trying to
propagate the const-ness of `msg` to all of its children. The proto API
generally guarantees that a const message prevents mutation of not just
the top-level message, but of the entire tree of messages. If someone
passes you a `const` message pointer, that is supposed to render the
entire tree of messages under it immutable through that pointer.
What we wish we could express in `MutableRepeatedMessage(Message* msg)`
is: "this function requires that `msg` is mutable, but this function
will not actually mutate it, and is therefore safe to call concurrently."
But there is no way of expressing this in C++.
PiperOrigin-RevId: 886830933
|
||
|---|---|---|
| .. | ||
| dist | ||
| docs | ||
| protobuf_distutils | ||
| .repo-metadata.json | ||
| BUILD.bazel | ||
| build_targets.bzl | ||
| convert.c | ||
| convert.h | ||
| descriptor.c | ||
| descriptor.h | ||
| descriptor_containers.c | ||
| descriptor_containers.h | ||
| descriptor_pool.c | ||
| descriptor_pool.h | ||
| extension_dict.c | ||
| extension_dict.h | ||
| internal.bzl | ||
| MANIFEST.in | ||
| map.c | ||
| map.h | ||
| message.c | ||
| message.h | ||
| minimal_test.py | ||
| protobuf.c | ||
| protobuf.h | ||
| py_extension.bzl | ||
| python_api.h | ||
| python_version_test.py | ||
| README.md | ||
| repeated.c | ||
| repeated.h | ||
| requirements.txt | ||
| unknown_fields.c | ||
| unknown_fields.h | ||
| version_script.lds | ||
Protocol Buffers Python
This directory contains the Protobuf library for Python.
For user documentation about how to use Protobuf Python, see https://protobuf.dev/getting-started/pythontutorial/
Installation
In most cases you should install the library using pip or another package
manager:
$ pip install protobuf
The packages released on https://pypi.org/project/protobuf/#files include both a source distribution and binary wheels.
Building packages from this repo
If for some reason you wish to build the packages directly from this repo, you can use the following Bazel commands:
$ bazel build //python/dist:source_wheel
$ bazel build //python/dist:binary_wheel
The binary wheel will build against whatever version of Python is installed on your system. The source package is always the same and does not depend on a local version of Python.
Building from setup.py
We support building from setup.py, but only from a Python source package.
You cannot build from setup.py using the GitHub repo or the GitHub source
tarball.
To build a source package from this repo, see the instructions in the previous section.
Implementation backends
There are three separate implementations of Python Protobuf. All of them offer the same API and are thus functionally the same, though they have very different performance characteristics.
The runtime library contains a switching layer that can choose between these
backends at runtime. Normally it will choose between them automatically, using
priority-ordered list, and skipping any backends that are not available. However
you can request a specific backend by setting the
PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION environment variable to one of the
following values:
- upb: Built on the upb C library, this is a new extension module released in 4.21.0. It offers better performance than any of the previous backends, and it is now the default. It is distributed in our PyPI packages, and requires no special installation. The code for this module lives in this directory.
- cpp: This extension module wraps the C++ protobuf library. It is deprecated and is no longer released in our PyPI packages, however it is still used in some legacy cases where apps want to perform zero-copy message sharing between Python and C++. It must be installed separately before it can be used. The code for this module lives in google/protobuf/pyext.
- python: The pure-Python backend, this does not require any extension module to be present on the system. The code for the pure-Python backend lives in google/protobuf/internal
The order given above is the general priority order, with upb being preferred
the most and the python backend preferred the least. However this ordering can
be overridden by the presence of a
google.protobuf.internal._api_implementation module. See the logic in
api_implementation.py
for details.
You can check which backend you are using with the following snippet:
$ python
Python 3.10.9 (main, Dec 7 2022, 13:47:07) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from google.protobuf.internal import api_implementation
>>> print(api_implementation.Type())
upb
This is not an officially-supported or stable API, but it is useful for ad hoc diagnostics.
More information about sharing messages between Python and C++ is available here: https://protobuf.dev/reference/python/python-generated/#sharing-messages
Code generator
The code for the Protobuf Python code generator lives in
//src/google/protobuf/compiler/python.
The code generator can output two different files for each proto foo.proto:
- foo_pb2.py: The module you import to actually use the protos.
- foo_pb2.pyi: A stub file that describes the interface of the protos.
The foo_pb2.pyi file is useful for IDEs or for users who want to read the
output file. The foo_pb2.py file is optimized for fast loading and is not
readable at all.
Note that the pyi file is only generated if you pass the pyi_out option to
protoc:
$ protoc --python_out=pyi_out:output_dir