Commit graph

1068 commits

Author SHA1 Message Date
Protobuf Team Bot
83ec410d30 Internal change
PiperOrigin-RevId: 619377672
2024-03-27 05:50:55 -07:00
Anton Grbin
86abf35ef5 Python JSON parser: Ignore invalid enum string values if ignore_unknown_fields is set (#15887)
# Motivation

This PR fixes failing conformance tests for python with name `IgnoreUnknownEnumStringValue`.

The JSON parsing spec was discussed in https://github.com/protocolbuffers/protobuf/issues/7392.

Recent equivalent changes for other languages:
* Swift: https://github.com/apple/swift-protobuf/pull/1345
* C#: https://github.com/protocolbuffers/protobuf/pull/15758

# Changes

- 1st commit is a noop  refactoring to make relevant _ConvertScalarFieldValue invocations localized
- 2nd commit introduces the child exception of `ParseError` named `EnumStringValueParseError` which is suppressed if `ignore_unknown_fields` is set
- 3rd commit updates the conformance test failure lists

Closes #15887

COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/15887 from noom:anton/7392/fix-python-test fbcc93a232
PiperOrigin-RevId: 619288323
2024-03-26 13:22:12 -07:00
Jie Luo
7e033c0be1 internal chnge
PiperOrigin-RevId: 619287040
2024-03-26 13:17:46 -07:00
Jie Luo
a2cc9f7b62 internal change
PiperOrigin-RevId: 619038132
2024-03-25 20:02:03 -07:00
Jie Luo
7cf02389b3 fix upb python: DO NOT create empty optional field after sorting empty repeated field
PiperOrigin-RevId: 618350866
2024-03-22 20:00:05 -07:00
Eric Salo
d452a16b8c upb: fix textformat encoder to print group name instead of field name
PiperOrigin-RevId: 618228797
2024-03-22 10:59:29 -07:00
Mike Kruskal
0e167e796f Disallow all feature types except enum and boolean.
Features are designed as temporary migration tools, and any unbounded type leaves the system open to unexpected use.  Features should have a fixed set of values, with well defined behaviors.

PiperOrigin-RevId: 617933544
2024-03-21 12:51:15 -07:00
Protobuf Team Bot
0442ed62cb Enable Protobuf Python version check.
PiperOrigin-RevId: 617626372
2024-03-20 14:35:37 -07:00
Tiago Quelhas
774627235e Automated Code Change
PiperOrigin-RevId: 617586334
2024-03-20 12:09:39 -07:00
Protobuf Team Bot
dca05ea1cd Add a go/pytype annotation on Duration.toTimedelta.
PiperOrigin-RevId: 616938980
2024-03-18 14:03:02 -07:00
Eric Salo
e1a19ba8bb upb: add 'options' arg to upb_Message_IsEqual()
PiperOrigin-RevId: 614707625
2024-03-11 10:16:13 -07:00
Jie Luo
eee53125c4 internal change
PiperOrigin-RevId: 613982814
2024-03-08 10:55:53 -08:00
Mike Kruskal
d5543304d5 Fix has_presence to account for proto3 extensions in Python.
PiperOrigin-RevId: 613361074
2024-03-06 15:32:16 -08:00
Protobuf Team Bot
554a00c40a Introduce Python Protobuf runtime version.
PiperOrigin-RevId: 613224059
2024-03-06 08:26:18 -08:00
Sandy Zhang
5b32936822 Register Scalar/MessageMapContainerTypes as virtual subclasses of MutableMapping instead of inheriting directly.
This prevents these from using abc.ABCMeta metaclass to avoid deprecation warning:
```
DeprecationWarning: Type google._upb._message.MessageMapContainer uses PyType_Spec with a metaclass that has custom tp_new. This is deprecated and will no longer be allowed in Python 3.14.
```

Fixes #15077
Fixes #12186

PiperOrigin-RevId: 613029479
2024-03-05 17:56:32 -08:00
aiuto
3ab1276bbb Update Bazel references to rules_pkg to the supported file locations. (#15780)
WIP:  I want to see this pass CI first.

Fixes: #15779

3 scripted transforms:
```
  -load("@rules_pkg//:mappings.bzl", ...)
  +load("@rules_pkg//pkg:mappings.bzl", ...)
```
```
  -load("@rules_pkg//:pkg.bzl", "pkg_tar")
  +load("@rules_pkg//pkg:tar.bzl", "pkg_tar")
```
```
  -load("@rules_pkg//:pkg.bzl", "pkg_zip")
  +load("@rules_pkg//pkg:zip.bzl", "pkg_zip")
```

Closes #15780

COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/15780 from aiuto:rules_pkg a630276341
PiperOrigin-RevId: 606641398
2024-02-13 09:08:42 -08:00
Joshua Haberman
776d072fa1 Removed obsolete TODO.
PiperOrigin-RevId: 606317655
2024-02-12 11:48:35 -08:00
Eric Salo
c69ed472cd upb: implement minitable-based field iterators
PiperOrigin-RevId: 605718863
2024-02-09 13:39:18 -08:00
Protobuf Team Bot
2fc7022879 Adds string view fields whose accessors are absl::string_view based.
PiperOrigin-RevId: 605368710
2024-02-08 11:04:02 -08:00
Protobuf Team Bot
2699579875 Breaking change: Remove the deprecated always_print_primitive_fields option from Java, Python and C++ JSON parsers.
The replacement always_print_without_presence_fields should be used instead, which is very similar but has consistent handling of optional fields by not affecting them.

PiperOrigin-RevId: 604381178
2024-02-05 11:35:16 -08:00
Protobuf Team Bot
7d43131a0a Rename the 'includingDefaultValueWithoutPresenceFields' and 'always_print_without_presence_fields' to 'alwaysPrintFieldsWithNoPresence' in the Cpp, Py and Java JSON serializers for consistency.
PiperOrigin-RevId: 604292220
2024-02-05 05:53:59 -08:00
Protobuf Team Bot
461b50b180 Stop using including_default_value_fields as a positional argument in json_format_test.py
PiperOrigin-RevId: 603713528
2024-02-02 09:57:56 -08:00
Joshua Haberman
8135fca851 Breaking Change: Removed obsolete/duplicate setup.py from Python.
This copy of `setup.py` is obsolete, as it builds the old C++ backend for Protobuf, which has been deprecated and
unused since 4.21.0 when
[the backend was switch to upb](https://protobuf.dev/news/2022-05-06/#python-updates).

The `setup.py` that we actually distribute in our source packages is located in [`python/dist/setup.py`](https://github.com/protocolbuffers/protobuf/blob/main/python/dist/setup.py).  It is not possible to build this `setup.py` directly from the GitHub repo or GitHub release tarball, because it depends on the file layout of our Python source package ([as distributed on PyPI](https://pypi.org/project/protobuf/#files)).  The Python source package uses a layout that pulls together all of the things Python needs:

|Python Source Package Path|GitHub Repo Path|Description|
|-----|-----|-----|
|`setup.py`|`python/dist/setup.py`|
|`google/protobuf/*`|`python/google/protobuf/*`|pure Python sources|
|`python/*`|`python/*`|C extension sources|
|`utf8_range/*`|`third_party/utf8_range`|C UTF-8 Validation Library|
|`upb/*`|`upb/upb/*`|C Protobuf Library|

Users who want to build their own Python packages should build from our source package on PyPI, not from our GitHub repo or our GitHub release tarball.

It is also possible to build our source package from GitHub using the following command (this requires Bazel):

```
$ bazel build //python/dist:source_wheel
```

PiperOrigin-RevId: 603162788
2024-01-31 14:40:41 -08:00
Mike Kruskal
9a020c4a7b Unify behavior of deprecated_legacy_json_field_conflicts across upb and syntax.
This disables all checks of json_name for upb and protoc under both proto2 and proto3.  This option is deprecated that will be removed in future versions, and is only meant as a temporary solution.  This also fixes a latent bug in the calculation of camelcase name in Python/upb.

Fixes #12525

PiperOrigin-RevId: 603158026
2024-01-31 14:21:54 -08:00
Protobuf Team Bot
25c6d34d4e Add new including_default_value_without_presence_fields to Python JSON serializer.
This flag has consistent behavior between proto2 and proto3 optionals (by not including either one), unlike including_default_value_fields which does include proto2 optional but excludes proto3 optionals.

including_default_value_fields is now deprecated and will be removed in an upcoming release.

PiperOrigin-RevId: 603156447
2024-01-31 14:16:54 -08:00
Dennis Shao
9146ce6ddb Updating version.json and repo version numbers to: 27-dev (#15590)
Closes #15590

PiperOrigin-RevId: 601468713
2024-01-25 09:18:51 -08:00
Joshua Haberman
6cf7f9b5f6 Automated rollback of commit f7dbd3841f.
PiperOrigin-RevId: 601300590
2024-01-24 18:57:17 -08:00
Joshua Haberman
ca5ca62fa6 Fixed a bug when deep copying a map field with a closed enum value.
PiperOrigin-RevId: 601195914
2024-01-24 12:20:01 -08:00
Joshua Haberman
af339fad1e Fixed a bug in cl/600990001 -- we need to propagate the as_utf8 parameter that was passed.
PiperOrigin-RevId: 601109776
2024-01-24 07:08:16 -08:00
Joshua Haberman
bf00034493 Breaking Change: Made text_format output default to UTF-8.
Also hardened the text format printer against invalid UTF-8 in string fields.  The output string will always be valid UTF-8, even if string fields contain invalid UTF-8.

PiperOrigin-RevId: 600990001
2024-01-23 20:34:54 -08:00
Joshua Haberman
b9e4894462 Fixed a SEGV when deep copying a non-reified sub-message.
PiperOrigin-RevId: 600951523
2024-01-23 17:13:47 -08:00
Jie Luo
f7dbd3841f internal change
PiperOrigin-RevId: 600609318
2024-01-22 17:01:02 -08:00
Sandy Zhang
65c65c2d04 Breaking Change: Use Editions features in Java full runtimes.
This change breaks compatibility with old generated code from previous major versions per the Cross Version Runtime policy: https://protobuf.dev/support/cross-version-runtime-guarantee. This includes old gencode from <4.26.x, which does not resolve features.

See https://protobuf.dev/news/2023-12-05/

PiperOrigin-RevId: 600487923
2024-01-22 09:57:26 -08:00
Eric Salo
8d0bfb0a5a upb: delete upb:upb, upb:collections, upb/upb.hpp, upb/collections/
PiperOrigin-RevId: 597923328
2024-01-12 12:44:04 -08:00
Joshua Haberman
f2a91b33e1 Make str(msg) in Python print raw UTF-8 strings. Only invalid UTF-8 is escaped.
PiperOrigin-RevId: 597917280
2024-01-12 12:22:53 -08:00
Dmitri Gribenko
321b817147 Automated Code Change
PiperOrigin-RevId: 596091728
2024-01-05 15:06:22 -08:00
Jie Luo
0905ba3462 Add "extend repeated with nothing" tests back for upb python
PiperOrigin-RevId: 596017378
2024-01-05 09:46:52 -08:00
Mike Kruskal
3acd85e0a9 Remove edition getter from python descriptor APIs
We've never released these, so this is not a breaking change.

PiperOrigin-RevId: 596007719
2024-01-05 09:01:37 -08:00
Jie Luo
4ebba684c7 BREAKING CHANGE in v26: Remove RegisterExtension in message class
PiperOrigin-RevId: 595989309
2024-01-05 07:25:56 -08:00
Jie Luo
1658213ba8 BREAKING CHANGE in v26: Reject extend repeated field with none iterable (Raise TypeError)
For example m.repeated_int32.extend(None) will be rejected

PiperOrigin-RevId: 595840357
2024-01-04 17:14:41 -08:00
Jie Luo
c51f111022 BREAKING CHANGE in v26: Remove Deprecated APIs that add non top descriptor.
Include AddFileDescriptor, AddDescriptor, AddEnumDescriptor,
AddExtensionDescriptor, AddServiceDescriptor.

Those Deprecated APIs may add unlinked descriptors to descriptor_pool which is
is wrong. Should use Add() or AddSerializedFile() instead. Those APIs were
raising deprecated warnings since 2019

PiperOrigin-RevId: 595831718
2024-01-04 16:31:09 -08:00
Marcel
fbbe681409 Do no longer depend on deprecated //external:python_headers (#15236)
`bind()` targets are deprecated and unsupported with Bzlmod. Specifying the dependency directly as a mitigation.

Alternative: Define an `alias()` within `/third_party/BUILD`.

Closes #15236

COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/15236 from mering:alias-python-headers 801ac73313
PiperOrigin-RevId: 595737575
2024-01-04 10:19:23 -08:00
Adam Cozzette
a7b0421c78 Breaking change: make protobuf comply with the C++ layering check
This check enforces that each C++ build target has the correct dependencies for
all headers that it includes. We have many targets that were not correct with
respect to this check, so I fixed them up.

I also cleaned up the C++ targets related to the well-known types. I created a
cc_proto_library() target for each one and removed the :wkt_cc_protos target,
since this was necessary to satisfy the layering check. I deleted the
//src/google/protobuf:protobuf_nowkt target and deprecated :protobuf_nowkt,
because the distinction between the :protobuf and :protobuf_nowkt targets was
not really correct. Neither one exposed the headers for the well-known types in
a way that was valid with respect to the layering check, and the idea of
bundling all the well-known types together is not idiomatic in Bazel anyway.
This is a breaking change, because the //:protobuf target no longer bundles the
well-known types. From now on they should be accessed through the new
//:*_cc_proto aliases in our top-level package.

I renamed the :port_def target to :port, which simplifies things a bit by
matching our internal name.

The original motivation for this change was that to move utf8_range onto our CI
infrastructure, we needed to make its dependency rules_fuzzing compatible with
Bazel 6. The rules_fuzzing project builds with the layering check, and I found
that the process of upgrading it to Bazel 6 made it take a dependency on
protobuf, which caused it to break due to layering violations. I was able to
work around this, but it would still be nice to comply with the layering check
so that we don't have to worry about this kind of thing in the future.

PiperOrigin-RevId: 595516736
2024-01-03 15:20:05 -08:00
Eric Salo
64dbf0dba4 upb: delete the aliases for upb:hash, upb:lex, upb:reflection_internal
PiperOrigin-RevId: 594992691
2024-01-01 18:21:58 -08:00
Adam Cozzette
4ec9170bcd Fix layering check for usage of gtest
To satisfy the layering check, we need to depend on :gtest for the headers, in
addition to :gtest_main which provides the main() function.

There are a bunch of formatting changes as a side effect of this, but they
should be harmless.

PiperOrigin-RevId: 594318263
2023-12-28 14:41:48 -08:00
Jie Luo
4f77929203 BREAKING CHANGE in v26: check if Timestamp is valid.
Seconds should be in range [-62135596800, 253402300799]
Nanos should be in range [0, 999999999]

PiperOrigin-RevId: 594119545
2023-12-27 16:38:51 -08:00
Eric Salo
2fb0b93d9d upb: tighten up and lock down upb/wire/
PiperOrigin-RevId: 593821827
2023-12-26 10:58:17 -08:00
Eric Salo
b997cb6d8d upb: add a non-void typedef for upb_Message
PiperOrigin-RevId: 592863926
2023-12-21 08:12:16 -08:00
Jie Luo
706c0d03f8 Automated rollback of commit 1250d5f6cc.
PiperOrigin-RevId: 592707509
2023-12-20 18:21:25 -08:00
Eric Salo
33cb42e33b upb: add 'static' to several Python map functions
PiperOrigin-RevId: 592391399
2023-12-19 17:52:06 -08:00