This implementation is similar to our Java idea in cl/949670944,
basically the idea is that we will have a local map (dictionary) that
will be populated the very first time we try to parse into an enum
field. The lifetime of this dictionary will be tied to the Parser
instance, similar to the Java implementation.
BENCHMARKS (baseline stats are recorded at cl/952170250):
Python:
```
[BENCHMARK] ParseJsonDefault: med 10.06 us/op | p99 11.22 us/op | mean 10.10 ± 0.20 us/op
[BENCHMARK] ParseJsonCustom: med 20.33 us/op | p99 22.23 us/op | mean 20.40 ± 0.34 us/op
[BENCHMARK] ParseJsonUnknownIgnored: med 24.21 us/op | p99 26.25 us/op | mean 24.31 ± 0.44 us/op
[BENCHMARK] ParseRepeatedJsonDefault: med 1.94 us/item | p99 2.05 us/item | mean 1.95 ± 0.03 us/item
[BENCHMARK] ParseRepeatedJsonCustom: med 2.26 us/item | p99 2.45 us/item | mean 2.27 ± 0.03 us/item
[BENCHMARK] ParseRepeatedJsonUnknownIgnored: med 5.31 us/item | p99 5.52 us/item | mean 5.32 ± 0.05 us/item
```
Cpp:
```
[BENCHMARK] ParseJsonDefault: med 8.66 us/op | p99 9.21 us/op | mean 8.70 ± 0.13 us/op
[BENCHMARK] ParseJsonCustom: med 26.33 us/op | p99 29.71 us/op | mean 26.44 ± 0.69 us/op
[BENCHMARK] ParseJsonUnknownIgnored: med 28.46 us/op | p99 31.80 us/op | mean 28.59 ± 0.67 us/op
[BENCHMARK] ParseRepeatedJsonDefault: med 1.72 us/item | p99 2.19 us/item | mean 1.74 ± 0.06 us/item
[BENCHMARK] ParseRepeatedJsonCustom: med 2.12 us/item | p99 2.38 us/item | mean 2.13 ± 0.04 us/item
[BENCHMARK] ParseRepeatedJsonUnknownIgnored: med 4.61 us/item | p99 4.92 us/item | mean 4.62 ± 0.07 us/item
```
UPB:
```
[BENCHMARK] ParseJsonDefault: med 11.22 us/op | p99 12.29 us/op | mean 11.27 ± 0.21 us/op
[BENCHMARK] ParseJsonCustom: med 37.54 us/op | p99 39.25 us/op | mean 37.65 ± 0.46 us/op
[BENCHMARK] ParseJsonUnknownIgnored: med 41.18 us/op | p99 48.45 us/op | mean 41.30 ± 0.89 us/op
[BENCHMARK] ParseRepeatedJsonDefault: med 2.10 us/item | p99 2.19 us/item | mean 2.10 ± 0.02 us/item
[BENCHMARK] ParseRepeatedJsonCustom: med 2.58 us/item | p99 2.73 us/item | mean 2.59 ± 0.03 us/item
[BENCHMARK] ParseRepeatedJsonUnknownIgnored: med 5.70 us/item | p99 5.90 us/item | mean 5.71 ± 0.04 us/item
```
Seeing these benchmarks, we can see a very noticeable improvement in
`ParseRepeatedJsonCustom` and `ParseRepeatedJsonUnknownIgnored` cases --
which correspond to the pathological cases that we are worried about.
Singular fields are taking longer to parse in our microbenchmark because
the cache that we've added is tied to each Parser instance and must be
re-instantiated across every `Parse` call.
PiperOrigin-RevId: 970528683
The unit tests added in the initial implementation in cl/925012500 and
cl/922729754 were fairly minimal. I took some inspiration from our C++
and Java unit tests for JSON's custom enum name parsing to come up with
these new test cases.
This change also adds a python Benchmark that we can use to verify that
our optimizations in cl/952170251 actually works.
Benchmark results across all three proto implementations (cpp, python, upb):
=== C++ Extension Protos (json_format_benchmark_cpp_protos) ===
[BENCHMARK] ParseJsonDefault: med 8.70 us/op | p99 9.56 us/op | mean 8.74 ± 0.15 us/op
[BENCHMARK] ParseJsonCustom: med 17.39 us/op | p99 21.64 us/op | mean 17.43 ± 0.51 us/op
[BENCHMARK] ParseJsonUnknownIgnored: med 26.97 us/op | p99 32.04 us/op | mean 27.06 ± 0.65 us/op
[BENCHMARK] ParseRepeatedJsonDefault: med 1.71 us/item | p99 1.83 us/item | mean 1.71 ± 0.02 us/item
[BENCHMARK] ParseRepeatedJsonCustom: med 9.18 us/item | p99 9.88 us/item | mean 9.20 ± 0.11 us/item
[BENCHMARK] ParseRepeatedJsonUnknownIgnored: med 18.79 us/item | p99 19.19 us/item | mean 18.81 ± 0.13 us/item
=== UPB Extension Protos (json_format_benchmark_upb_protos) ===
[BENCHMARK] ParseJsonDefault: med 8.70 us/op | p99 9.56 us/op | mean 8.74 ± 0.15 us/op
[BENCHMARK] ParseJsonCustom: med 17.39 us/op | p99 21.64 us/op | mean 17.43 ± 0.51 us/op
[BENCHMARK] ParseJsonUnknownIgnored: med 26.97 us/op | p99 32.04 us/op | mean 27.06 ± 0.65 us/op
[BENCHMARK] ParseRepeatedJsonDefault: med 1.71 us/item | p99 1.83 us/item | mean 1.71 ± 0.02 us/item
[BENCHMARK] ParseRepeatedJsonCustom: med 9.18 us/item | p99 9.88 us/item | mean 9.20 ± 0.11 us/item
[BENCHMARK] ParseRepeatedJsonUnknownIgnored: med 18.79 us/item | p99 19.19 us/item | mean 18.81 ± 0.13 us/item
=== Pure Python Protos (json_format_benchmark_python_protos) ===
[BENCHMARK] ParseJsonDefault: med 10.26 us/op | p99 11.20 us/op | mean 10.30 ± 0.25 us/op
[BENCHMARK] ParseJsonCustom: med 16.59 us/op | p99 18.35 us/op | mean 16.70 ± 0.45 us/op
[BENCHMARK] ParseJsonUnknownIgnored: med 25.09 us/op | p99 28.22 us/op | mean 25.27 ± 0.65 us/op
[BENCHMARK] ParseRepeatedJsonDefault: med 1.94 us/item | p99 3.25 us/item | mean 1.97 ± 0.14 us/item
[BENCHMARK] ParseRepeatedJsonCustom: med 7.16 us/item | p99 10.35 us/item | mean 7.24 ± 0.39 us/item
[BENCHMARK] ParseRepeatedJsonUnknownIgnored: med 15.43 us/item | p99 19.54 us/item | mean 15.54 ± 0.49 us/item
PiperOrigin-RevId: 966820541
This fixes a security vulnerability where nested google.protobuf.Any messages could bypass the max_recursion_depth limit, potentially leading to denial of service via stack overflow.
The root cause was that _ConvertAnyMessage() was calling itself recursively via methodcaller() for nested well-known types, bypassing the recursion depth tracking in ConvertMessage().
The fix routes well-known type parsing through ConvertMessage() to ensure proper recursion depth accounting for all message types including nested Any.
Fixes#25070Closes#25239
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/25239 from aviralgarg05:fix-any-recursion-depth-bypass 3cbbcbea142593d3afd2ceba2db14b05660f62f4
PiperOrigin-RevId: 862740421
For historical reasons, our internal copy of the codebase uses
`proto2_unittest` while the open source one uses `protobuf_unittest`. This adds
a lot of friction to syncing between the two, and there's no great reason to
keep maintaining this difference, so let's just go with `proto2_unittest`.
PiperOrigin-RevId: 721951543
Found this bug where if `_ConvertFieldValuePair` fails while parsing an
Any from a dictionary value then the `@type` field gets removed from
the original object. Adding a simple `try/finally` to ensure we always
restore the original object.
Closes#20111
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/20111 from atulmerchia:fix-destructive-anyparser 58aea1911f
PiperOrigin-RevId: 721850901
The replacement always_print_without_presence_fields should be used instead, which is very similar but has consistent handling of optional fields by not affecting them.
PiperOrigin-RevId: 604381178
This flag has consistent behavior between proto2 and proto3 optionals (by not including either one), unlike including_default_value_fields which does include proto2 optional but excludes proto3 optionals.
including_default_value_fields is now deprecated and will be removed in an upcoming release.
PiperOrigin-RevId: 603156447
assertRaisesRegexp has been deprecated in favor of the slightly
different spelling assertRaisesRegex. Let's fix this just to clean up a
bunch of deprecation warnings in the tests.
* Fix a typo
* Fix lots of spelling errors
* Fix a few more spelling mistakes
* s/parsable/parseable/
* Don't touch the third party files
* Cloneable is the preferred C# term
* Copyable is the preferred C++ term
* Revert "s/parsable/parseable/"
This reverts commit 534ecf7675.
* Revert unparseable->unparsable corrections
* Set execute bit on files if and only if they begin with (#!).
Git only tracks the 'x' (executable) bit on each file. Prior to this
CL, our files were a random mix of executable and non-executable.
This change imposes some order by making files executable if and only
if they have shebang (#!) lines at the beginning.
We don't have any executable binaries checked into the repo, so
we shouldn't need to worry about that case.
* Added fix_permissions.sh script to set +x iff a file begins with (#!).
