The spec has been relaxed such that the preferred behavior is last-wins, and parse-failure if last-wins is not implementable. The conformance test is updated correspondingly to pass under either of those permissible behaviors.
https://github.com/protocolbuffers/protobuf/issues/28791
PiperOrigin-RevId: 957979215
In general, runtimes also expose Timestamp parsing by utils but it is conformance-testable via JSON.
Timestamp format is RFC3339 which looks like `1970-01-01T00:00:00Z` (for UTC timestamps) or `1970-01-02T01:00:00+02:00` (with timezone).
There's actually two different reasons why a Timestamp format might be invalid.
RFC3339 spec has some bright line rules as follows:
- All of the segments must be the expected fixed width (eg: month is always 2 digits, 1970-01-01 and not 1970-1-1).
- Year must be [0001 .. 9999]
- Month must be [01 .. 12]
- Day must be valid for the relevant month, including that Feb 29 is only legal in leap years.
- The literal `T` comes next (spec ambiguous if `t` is also allowed, not tested in this change)
- Hour must be [00..23]
- Minute must be [00..59]
- Second must be [00..59]
- Subseconds are unconstrained precision
- Timezone must be Z for UTC or else formatted with HH:MM where HH is [00..23] and MM is [00..59].
Protobuf has a few extra effective constraints compared to RFC3339 though, which are _mostly_ academic:
- We disallow subsecond resolution has more than 9 decimal digits
- Some RFC3339 timestamps which are not-UTC-timezone in year 1 or year 9999 are not valid in Protobuf [*reason explained below]
Note that GoProto uses a std RFC3339 parser and does implement these bonus restrictions rules on top.
* The reason is that `0001-01-01T00:00:00+00:01` is a legal RFC3339 timestamp, but is BCE in UTC timezone. This is an oddity: we otherwise say parsers should accept RFC3339 timestamps with any timezone, but must serialize as UTC. This timestamp is not legally representable in RFC3339 in UTC, and the google.protobuf.Timestamp field secs is documented that `-62135596800` is the minimum legal value, but this timestamp is earlier than that. Because this situation only occurs at the years 1 and 9999 its not plausibly a concern on real user data, its only an oddity that you must either reject this legal RFC3339 timestamp at parse time, or else accept it but produce an in-memory google.protobuf.Timestamp which is officially 'invalid' and cannot be printed back out.
PiperOrigin-RevId: 954603113
No wire format should ever contain an overlong varint, so the topic here is only how to react to non-standard and potentially corrupted data.
The situation today is that there's 4 main ways that implementations deal when parsing tags:
1) parse up to 10 bytes, cast to uint32
2) parse up to 10 bytes, reject if it is above uint32_max
3) parse up to 5 bytes, cast to uint32
4) parse up to 5 bytes, reject if it is above uint32_max
Of our primary supported implementations, these four strategies are used by Java, Go, C++ and upb correspondingly.
Based on examining the situation, the decision taken is that:
- Coercing down silently ignoring bits in the tag is dangerous to interpretation-confusion / silent misparsing, which means Java approach is dangerous.
- Needing to support parsing up to 10 bytes (even when they may just be all 0x80 and no content) would have real performance implications on the upb and C++ parsers. Since it should really never happen taking any performance hit on all parses based on a hypothetical is considered undesirable.
For that reason, the conformance test is set to match upb's behavior, which is slight mismatch to C++ and Go behavior today (in different ways), and larger mismatch to the Java behavior today.
Because fixing this 'bug' may be disruptive to a customer in theory (though it would probably mean they have some bad data that was accidentally parsing), we may hold back fixing the behavior to a breaking change release; this change to the conformance suite only establishes the decision on preferred behavior.
PiperOrigin-RevId: 841856475
Adds test coverage for invalid empty strings (e.g. ""), non-numeric strings (e.g. "abc"), and partially-numeric strings (e.g. "12abc"), as well as valid exponential numeric strings ("1e5)
We will target enforcing non-conformant cases that should have failed to parse but didn't in upb in v30.x (our ~annual breaking release in some languages). Conformance failures to accept input we previously failed on can be landed at any time.
PiperOrigin-RevId: 694269337
This change only covers pure python, and follow-up changes will handle C++/upb variants and actually enable editions support. The C++ one works (as evident from the conformance tests), but needs some APIs added to allow for testing.
PiperOrigin-RevId: 580304039
* Down-integrate internal changes to github.
* fix python conformance test
* fix csharp conformance test
* add back java map_lite_test.proto's optimize for option
* fix php conformance test
Add note about JSON tests maybe being wrong.
- Add note about the JSON test maybe not being correct yet.
- Add test to checks the generated names for double underscores to be sure they
are what is expected.
