No wire format should ever contain an overlong varint, so the topic here is only how to react to non-standard and potentially corrupted data.
The situation today is that there's 4 main ways that implementations deal when parsing tags:
1) parse up to 10 bytes, cast to uint32
2) parse up to 10 bytes, reject if it is above uint32_max
3) parse up to 5 bytes, cast to uint32
4) parse up to 5 bytes, reject if it is above uint32_max
Of our primary supported implementations, these four strategies are used by Java, Go, C++ and upb correspondingly.
Based on examining the situation, the decision taken is that:
- Coercing down silently ignoring bits in the tag is dangerous to interpretation-confusion / silent misparsing, which means Java approach is dangerous.
- Needing to support parsing up to 10 bytes (even when they may just be all 0x80 and no content) would have real performance implications on the upb and C++ parsers. Since it should really never happen taking any performance hit on all parses based on a hypothetical is considered undesirable.
For that reason, the conformance test is set to match upb's behavior, which is slight mismatch to C++ and Go behavior today (in different ways), and larger mismatch to the Java behavior today.
Because fixing this 'bug' may be disruptive to a customer in theory (though it would probably mean they have some bad data that was accidentally parsing), we may hold back fixing the behavior to a breaking change release; this change to the conformance suite only establishes the decision on preferred behavior.
PiperOrigin-RevId: 841856475
Adds test coverage for invalid empty strings (e.g. ""), non-numeric strings (e.g. "abc"), and partially-numeric strings (e.g. "12abc"), as well as valid exponential numeric strings ("1e5)
We will target enforcing non-conformant cases that should have failed to parse but didn't in upb in v30.x (our ~annual breaking release in some languages). Conformance failures to accept input we previously failed on can be landed at any time.
PiperOrigin-RevId: 694269337
This change only covers pure python, and follow-up changes will handle C++/upb variants and actually enable editions support. The C++ one works (as evident from the conformance tests), but needs some APIs added to allow for testing.
PiperOrigin-RevId: 580304039
* Down-integrate internal changes to github.
* fix python conformance test
* fix csharp conformance test
* add back java map_lite_test.proto's optimize for option
* fix php conformance test
Add note about JSON tests maybe being wrong.
- Add note about the JSON test maybe not being correct yet.
- Add test to checks the generated names for double underscores to be sure they
are what is expected.
