Commit graph

1056 commits

Author SHA1 Message Date
Mike Kruskal
d5543304d5 Fix has_presence to account for proto3 extensions in Python.
PiperOrigin-RevId: 613361074
2024-03-06 15:32:16 -08:00
Protobuf Team Bot
554a00c40a Introduce Python Protobuf runtime version.
PiperOrigin-RevId: 613224059
2024-03-06 08:26:18 -08:00
Sandy Zhang
5b32936822 Register Scalar/MessageMapContainerTypes as virtual subclasses of MutableMapping instead of inheriting directly.
This prevents these from using abc.ABCMeta metaclass to avoid deprecation warning:
```
DeprecationWarning: Type google._upb._message.MessageMapContainer uses PyType_Spec with a metaclass that has custom tp_new. This is deprecated and will no longer be allowed in Python 3.14.
```

Fixes #15077
Fixes #12186

PiperOrigin-RevId: 613029479
2024-03-05 17:56:32 -08:00
aiuto
3ab1276bbb Update Bazel references to rules_pkg to the supported file locations. (#15780)
WIP:  I want to see this pass CI first.

Fixes: #15779

3 scripted transforms:
```
  -load("@rules_pkg//:mappings.bzl", ...)
  +load("@rules_pkg//pkg:mappings.bzl", ...)
```
```
  -load("@rules_pkg//:pkg.bzl", "pkg_tar")
  +load("@rules_pkg//pkg:tar.bzl", "pkg_tar")
```
```
  -load("@rules_pkg//:pkg.bzl", "pkg_zip")
  +load("@rules_pkg//pkg:zip.bzl", "pkg_zip")
```

Closes #15780

COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/15780 from aiuto:rules_pkg a630276341
PiperOrigin-RevId: 606641398
2024-02-13 09:08:42 -08:00
Joshua Haberman
776d072fa1 Removed obsolete TODO.
PiperOrigin-RevId: 606317655
2024-02-12 11:48:35 -08:00
Eric Salo
c69ed472cd upb: implement minitable-based field iterators
PiperOrigin-RevId: 605718863
2024-02-09 13:39:18 -08:00
Protobuf Team Bot
2fc7022879 Adds string view fields whose accessors are absl::string_view based.
PiperOrigin-RevId: 605368710
2024-02-08 11:04:02 -08:00
Protobuf Team Bot
2699579875 Breaking change: Remove the deprecated always_print_primitive_fields option from Java, Python and C++ JSON parsers.
The replacement always_print_without_presence_fields should be used instead, which is very similar but has consistent handling of optional fields by not affecting them.

PiperOrigin-RevId: 604381178
2024-02-05 11:35:16 -08:00
Protobuf Team Bot
7d43131a0a Rename the 'includingDefaultValueWithoutPresenceFields' and 'always_print_without_presence_fields' to 'alwaysPrintFieldsWithNoPresence' in the Cpp, Py and Java JSON serializers for consistency.
PiperOrigin-RevId: 604292220
2024-02-05 05:53:59 -08:00
Protobuf Team Bot
461b50b180 Stop using including_default_value_fields as a positional argument in json_format_test.py
PiperOrigin-RevId: 603713528
2024-02-02 09:57:56 -08:00
Joshua Haberman
8135fca851 Breaking Change: Removed obsolete/duplicate setup.py from Python.
This copy of `setup.py` is obsolete, as it builds the old C++ backend for Protobuf, which has been deprecated and
unused since 4.21.0 when
[the backend was switch to upb](https://protobuf.dev/news/2022-05-06/#python-updates).

The `setup.py` that we actually distribute in our source packages is located in [`python/dist/setup.py`](https://github.com/protocolbuffers/protobuf/blob/main/python/dist/setup.py).  It is not possible to build this `setup.py` directly from the GitHub repo or GitHub release tarball, because it depends on the file layout of our Python source package ([as distributed on PyPI](https://pypi.org/project/protobuf/#files)).  The Python source package uses a layout that pulls together all of the things Python needs:

|Python Source Package Path|GitHub Repo Path|Description|
|-----|-----|-----|
|`setup.py`|`python/dist/setup.py`|
|`google/protobuf/*`|`python/google/protobuf/*`|pure Python sources|
|`python/*`|`python/*`|C extension sources|
|`utf8_range/*`|`third_party/utf8_range`|C UTF-8 Validation Library|
|`upb/*`|`upb/upb/*`|C Protobuf Library|

Users who want to build their own Python packages should build from our source package on PyPI, not from our GitHub repo or our GitHub release tarball.

It is also possible to build our source package from GitHub using the following command (this requires Bazel):

```
$ bazel build //python/dist:source_wheel
```

PiperOrigin-RevId: 603162788
2024-01-31 14:40:41 -08:00
Mike Kruskal
9a020c4a7b Unify behavior of deprecated_legacy_json_field_conflicts across upb and syntax.
This disables all checks of json_name for upb and protoc under both proto2 and proto3.  This option is deprecated that will be removed in future versions, and is only meant as a temporary solution.  This also fixes a latent bug in the calculation of camelcase name in Python/upb.

Fixes #12525

PiperOrigin-RevId: 603158026
2024-01-31 14:21:54 -08:00
Protobuf Team Bot
25c6d34d4e Add new including_default_value_without_presence_fields to Python JSON serializer.
This flag has consistent behavior between proto2 and proto3 optionals (by not including either one), unlike including_default_value_fields which does include proto2 optional but excludes proto3 optionals.

including_default_value_fields is now deprecated and will be removed in an upcoming release.

PiperOrigin-RevId: 603156447
2024-01-31 14:16:54 -08:00
Dennis Shao
9146ce6ddb Updating version.json and repo version numbers to: 27-dev (#15590)
Closes #15590

PiperOrigin-RevId: 601468713
2024-01-25 09:18:51 -08:00
Joshua Haberman
6cf7f9b5f6 Automated rollback of commit f7dbd3841f.
PiperOrigin-RevId: 601300590
2024-01-24 18:57:17 -08:00
Joshua Haberman
ca5ca62fa6 Fixed a bug when deep copying a map field with a closed enum value.
PiperOrigin-RevId: 601195914
2024-01-24 12:20:01 -08:00
Joshua Haberman
af339fad1e Fixed a bug in cl/600990001 -- we need to propagate the as_utf8 parameter that was passed.
PiperOrigin-RevId: 601109776
2024-01-24 07:08:16 -08:00
Joshua Haberman
bf00034493 Breaking Change: Made text_format output default to UTF-8.
Also hardened the text format printer against invalid UTF-8 in string fields.  The output string will always be valid UTF-8, even if string fields contain invalid UTF-8.

PiperOrigin-RevId: 600990001
2024-01-23 20:34:54 -08:00
Joshua Haberman
b9e4894462 Fixed a SEGV when deep copying a non-reified sub-message.
PiperOrigin-RevId: 600951523
2024-01-23 17:13:47 -08:00
Jie Luo
f7dbd3841f internal change
PiperOrigin-RevId: 600609318
2024-01-22 17:01:02 -08:00
Sandy Zhang
65c65c2d04 Breaking Change: Use Editions features in Java full runtimes.
This change breaks compatibility with old generated code from previous major versions per the Cross Version Runtime policy: https://protobuf.dev/support/cross-version-runtime-guarantee. This includes old gencode from <4.26.x, which does not resolve features.

See https://protobuf.dev/news/2023-12-05/

PiperOrigin-RevId: 600487923
2024-01-22 09:57:26 -08:00
Eric Salo
8d0bfb0a5a upb: delete upb:upb, upb:collections, upb/upb.hpp, upb/collections/
PiperOrigin-RevId: 597923328
2024-01-12 12:44:04 -08:00
Joshua Haberman
f2a91b33e1 Make str(msg) in Python print raw UTF-8 strings. Only invalid UTF-8 is escaped.
PiperOrigin-RevId: 597917280
2024-01-12 12:22:53 -08:00
Dmitri Gribenko
321b817147 Automated Code Change
PiperOrigin-RevId: 596091728
2024-01-05 15:06:22 -08:00
Jie Luo
0905ba3462 Add "extend repeated with nothing" tests back for upb python
PiperOrigin-RevId: 596017378
2024-01-05 09:46:52 -08:00
Mike Kruskal
3acd85e0a9 Remove edition getter from python descriptor APIs
We've never released these, so this is not a breaking change.

PiperOrigin-RevId: 596007719
2024-01-05 09:01:37 -08:00
Jie Luo
4ebba684c7 BREAKING CHANGE in v26: Remove RegisterExtension in message class
PiperOrigin-RevId: 595989309
2024-01-05 07:25:56 -08:00
Jie Luo
1658213ba8 BREAKING CHANGE in v26: Reject extend repeated field with none iterable (Raise TypeError)
For example m.repeated_int32.extend(None) will be rejected

PiperOrigin-RevId: 595840357
2024-01-04 17:14:41 -08:00
Jie Luo
c51f111022 BREAKING CHANGE in v26: Remove Deprecated APIs that add non top descriptor.
Include AddFileDescriptor, AddDescriptor, AddEnumDescriptor,
AddExtensionDescriptor, AddServiceDescriptor.

Those Deprecated APIs may add unlinked descriptors to descriptor_pool which is
is wrong. Should use Add() or AddSerializedFile() instead. Those APIs were
raising deprecated warnings since 2019

PiperOrigin-RevId: 595831718
2024-01-04 16:31:09 -08:00
Marcel
fbbe681409 Do no longer depend on deprecated //external:python_headers (#15236)
`bind()` targets are deprecated and unsupported with Bzlmod. Specifying the dependency directly as a mitigation.

Alternative: Define an `alias()` within `/third_party/BUILD`.

Closes #15236

COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/15236 from mering:alias-python-headers 801ac73313
PiperOrigin-RevId: 595737575
2024-01-04 10:19:23 -08:00
Adam Cozzette
a7b0421c78 Breaking change: make protobuf comply with the C++ layering check
This check enforces that each C++ build target has the correct dependencies for
all headers that it includes. We have many targets that were not correct with
respect to this check, so I fixed them up.

I also cleaned up the C++ targets related to the well-known types. I created a
cc_proto_library() target for each one and removed the :wkt_cc_protos target,
since this was necessary to satisfy the layering check. I deleted the
//src/google/protobuf:protobuf_nowkt target and deprecated :protobuf_nowkt,
because the distinction between the :protobuf and :protobuf_nowkt targets was
not really correct. Neither one exposed the headers for the well-known types in
a way that was valid with respect to the layering check, and the idea of
bundling all the well-known types together is not idiomatic in Bazel anyway.
This is a breaking change, because the //:protobuf target no longer bundles the
well-known types. From now on they should be accessed through the new
//:*_cc_proto aliases in our top-level package.

I renamed the :port_def target to :port, which simplifies things a bit by
matching our internal name.

The original motivation for this change was that to move utf8_range onto our CI
infrastructure, we needed to make its dependency rules_fuzzing compatible with
Bazel 6. The rules_fuzzing project builds with the layering check, and I found
that the process of upgrading it to Bazel 6 made it take a dependency on
protobuf, which caused it to break due to layering violations. I was able to
work around this, but it would still be nice to comply with the layering check
so that we don't have to worry about this kind of thing in the future.

PiperOrigin-RevId: 595516736
2024-01-03 15:20:05 -08:00
Eric Salo
64dbf0dba4 upb: delete the aliases for upb:hash, upb:lex, upb:reflection_internal
PiperOrigin-RevId: 594992691
2024-01-01 18:21:58 -08:00
Adam Cozzette
4ec9170bcd Fix layering check for usage of gtest
To satisfy the layering check, we need to depend on :gtest for the headers, in
addition to :gtest_main which provides the main() function.

There are a bunch of formatting changes as a side effect of this, but they
should be harmless.

PiperOrigin-RevId: 594318263
2023-12-28 14:41:48 -08:00
Jie Luo
4f77929203 BREAKING CHANGE in v26: check if Timestamp is valid.
Seconds should be in range [-62135596800, 253402300799]
Nanos should be in range [0, 999999999]

PiperOrigin-RevId: 594119545
2023-12-27 16:38:51 -08:00
Eric Salo
2fb0b93d9d upb: tighten up and lock down upb/wire/
PiperOrigin-RevId: 593821827
2023-12-26 10:58:17 -08:00
Eric Salo
b997cb6d8d upb: add a non-void typedef for upb_Message
PiperOrigin-RevId: 592863926
2023-12-21 08:12:16 -08:00
Jie Luo
706c0d03f8 Automated rollback of commit 1250d5f6cc.
PiperOrigin-RevId: 592707509
2023-12-20 18:21:25 -08:00
Eric Salo
33cb42e33b upb: add 'static' to several Python map functions
PiperOrigin-RevId: 592391399
2023-12-19 17:52:06 -08:00
Jie Luo
1250d5f6cc BREAKING CHANGE in v26: check if Timestamp is valid.
Seconds should be in range [-62135596800, 253402300799]
Nanos should be in range [0, 999999999]

PiperOrigin-RevId: 592365636
2023-12-19 15:49:41 -08:00
Protobuf Team Bot
9c7d2b9d83 Make the utf8_range implementation just in C
PiperOrigin-RevId: 590961088
2023-12-14 09:25:06 -08:00
Joshua Haberman
de52944f38 [Python/upb] Fixed SEGV when attempting to delete a message attribute
Deleting an attribute is not allowed in any Proto Python implementation, but upb was not checking for this case.

PiperOrigin-RevId: 589995449
2023-12-11 16:57:49 -08:00
Jie Luo
0eac77c2c6 Breaking change in 26 release: Remove msg.UnknownFields() support in pure python and cpp extension.
Users should use the add-on unknown_fields.py support.
Old usage example:
unknown_field_set = msg.UnknownFields()

New usage should be:
from google.protobuf import unknown_fields
unknown_field_set = unknown_fields.UnknownFieldSet(msg)
PiperOrigin-RevId: 589969095
2023-12-11 15:25:41 -08:00
Mike Kruskal
cf2d6965dc Breaking change: Remove C++ legacy syntax descriptor APIs
PiperOrigin-RevId: 589879506
2023-12-11 11:09:26 -08:00
Mike Kruskal
fd40c87bef Breaking change: Remove deprecated syntax accessor.
This has been replaced by edition and feature getters.  Any code depending on syntax will likely be broken by editions files, and should be migrated to the finer-grained feature helpers.

PiperOrigin-RevId: 589866745
2023-12-11 10:30:26 -08:00
Joshua Haberman
c05b320d9c Fixed a bug where FileDescriptor maps like message_types_by_name could return descriptors from other files.
Fixes: https://github.com/protocolbuffers/protobuf/issues/13740
PiperOrigin-RevId: 587135054
2023-12-01 15:12:41 -08:00
Protobuf Team Bot
0fab773c1d Update remainder of upb to new short license style.
PiperOrigin-RevId: 584116886
2023-11-20 13:43:32 -08:00
Richard Levasseur
06776d20a5 Add load() statements for builtin Python rules/providers
PiperOrigin-RevId: 584089414
2023-11-20 11:54:29 -08:00
Protobuf Team Bot
00e783d113 Apply clang-tidy fixes to change static_cast to proto2::DownCastToGenerated in sr_www dependencies.
As a prerequisite of enabling go/protobuf-weak-speed-messages downcasts using built-in cast operations need to be changed to cast functions recommended by protobuf team (go/protobuf-downcast-recommendation)

**More information**: go/protobuf-stripping-lsc

#codehealth

PiperOrigin-RevId: 582977861
2023-11-16 03:21:10 -08:00
Mike Kruskal
c23d5333c3 Expose editions in Python/upb.
This also fixes a few minor bugs in the editions implementation that were caught in python/conformance tests, and adds a new SetFeatureSetDefaults API to the def pool for consistency with C++ and other python implementations.

PiperOrigin-RevId: 581384108
2023-11-10 15:20:41 -08:00
Mike Kruskal
e8377f12e1 Expose editions in Python/C++.
This was already fully implemented in C++, but we need to expose features methods to the Python runtime to fully enable it.  This also enables conformance and unit-testing for C++.

PiperOrigin-RevId: 581364355
2023-11-10 14:14:05 -08:00