It is common to want to match only a subset of fields in large protobufs, and doing that in rust right now requires manually clearing out the fields in the `actual` message that we don't care about. This is a pain, and a worse experience when compared to C++.
Like in C++, when doing partial matching, only fields set in the expected protobuf message are compared, while extra fields set only in the actual message are ignored.
When using the C++ kernel we leverage `proto2::util::MessageDifferencer`, and for the UPB kernel we use the `kUpb_CompareOption_Partial` option.
PiperOrigin-RevId: 964125682
`New` is more efficient on ClassData than on a prototype, as `New` on the prototype has to first load the class data through the prototype.
There's also a small optimization in `LazyField::MergeFromMessage` which avoids loading the class data twice. The redundancy was made more obvious by this refactor.
PiperOrigin-RevId: 953889072
This code makes use of some Rust features that are not supported in Rust 1.85 (our currently supported version), so we cannot enable the codegen for this feature yet.
This CL introduces support for [Protobuf Extensions](https://engdoc.corp.google.com/eng/doc/devguide/proto/programming-guides/proto2.html#extensions) to the Rust Protobuf library, a key feature for full Proto2 API parity.
Key changes include:
1. **Code Generation:**
* Added `compiler/rust/extension.cc` and `extension.h` to handle the generation of Rust code for extension definitions.
* Updated `generator.cc` and `message.cc` to invoke extension generation for both top-level and nested extensions.
* Generates `ExtensionId` constants for each extension.
2. **Runtime API (`rust/extension.rs`):**
* Introduced the generic `ExtensionId<Extendee, T>` type to identify and type-check extensions.
* Defined traits for extension operations: `ExtHas`, `ExtGet`, `ExtSet`, `ExtClear`, and `ExtGetMut`.
* Implemented methods on `ExtensionId` that dispatch to the trait implementations provided by the active kernel.
3. **Kernel Implementations:**
* **C++ Kernel (`rust/cpp_kernel/extension.rs`):** Implemented extension traits by calling into new C++ FFI functions (defined in `rust/cpp_kernel/extension.cc`) that interact with the C++ `ExtensionSet`.
* **UPB Kernel (`rust/upb_kernel/extension.rs`):** Implemented extension traits using UPB's MiniTable-based extension API. Extension MiniTables are registered at static initialization time using the `linkme` crate.
4. **Build System:**
* Updated `protobuf/rust/BUILD` and `protobuf/compiler/rust/BUILD` to include new files.
* Added a dependency on `//third_party/rust/linkme/v0_3` for the UPB kernel's extension registration.
5. **Testing:**
* Added `test/extensions.proto` and `test/extensions_separate_file.proto` to define various extension types.
* Added `test/shared/extensions_test.rs` with comprehensive tests covering presence, getters, setters, defaults, nested extensions, submessage extensions, repeated extensions, serialization, and `MergeFrom`.
This change enables Rust users to define and interact with Protobuf extensions in a type-safe manner, compatible with both the C++ and UPB runtimes.
PiperOrigin-RevId: 896506594
This change moves the `OwnedMessageInterop`, `MessageViewInterop`, and `MessageMutInterop` traits from `codegen_traits.rs` into new `interop.rs` files within both the `cpp_kernel` and `upb_kernel` directories.
The `KernelMessage`, `KernelMessageView`, and `KernelMessageMut` traits in each kernel's `message.rs` are updated to include the respective kernel-specific interop traits. This ensures that the interop functionality is tied to the specific kernel implementation.
The `shared.rs` module now re-exports the interop traits from `crate::__internal::runtime::interop`, which will resolve to the correct kernel's interop module.
It would be possible to take this farther by making the interop traits return strongly typed pointers to `upb_Message*` and `proto2::MessageLite*` instead of `c_void`, but that would be a breaking change.
PiperOrigin-RevId: 892470084
MessageMutInner was Copy+Clone, which had been convenient for some internal use. But, with them being .into() SomeMessageMut this was a potential footgun, since it was very easy to into() the same one twice which would be formally unsound if it was reachable (including that you could send each of them to a different thread and concurrently modify them).
As these were implementation detail types this is only a defensive fix and no actual soundness issues were identified.
PiperOrigin-RevId: 888615057
The `Enum` trait now requires several other traits, including `Singular`, `Into<i32>`, `Copy`, `Proxied`, `EntityType`, and `SealedInternal`. This simplifies bounds in functions that operate on types implementing `Enum`, as they no longer need to explicitly list `+ Singular` and other common requirements.
PiperOrigin-RevId: 888286136
This CL contains no functional change, it is a pure splitting of the existing files.
I elected to move the C++ kernel files to `cpp_kernel/` which already contained the C++ API for the C++ kernel. Unfortunately this required removing the existing `third_party/protobuf/rust/cpp_kernel/BUILD` file, because, the Rust parts of the cpp kernel have circular dependencies with the shared code in `third_party/protobuf/rust`.
PiperOrigin-RevId: 886884124
Make a new trait "MapKey" to represent the types that are allowed as a Map Key. Without this, right now our Map<> bounds is over generalized.
Rename "ProxiedInMapValue" to "MapValue" to correspondingly represent the types that are allowed as a Map Value.
PiperOrigin-RevId: 860118222
Note that we can't change any public-facing API's that need an arena pointer, as this would be a breaking change. Additionally, assignment operators need the arena pointer, but can't take additional arguments.
This is in preparation for removing the arena_ member from maps.
PiperOrigin-RevId: 818979467
This change also makes all the necessary fixes to get us compatible with
edition 2024.
I set things up so that we test edition 2024 with Bazel but keep Cargo on
edition 2021. This seems like a nice way to go since it allows us to get full
test coverage of both editions without us having to increment our minimum
supported Rust version.
PiperOrigin-RevId: 816241556
The logic to create new elements in `RepeatedPtrField` was spread across an assorted collection of specializations of `GenericTypeHandler`, individual `GenericTypeHandler` static functions, and external member and non-member functions, all living in multiple sources. Now all that logic is concentrated in three symmetric `GenericTypeHandler` specializations.
PiperOrigin-RevId: 754049107
Users still using the old `WORKSPACE` system may need to do the same or use `repo_mapping` on repositories that still use the old names. See Abseil's release notes for details: https://github.com/abseil/abseil-cpp/releases/tag/20250127.
#test-continuous
PiperOrigin-RevId: 725247849
It uses the same nodes that Map<> would use for the respectice types.
This makes the type compatible with visitation and removes some custom types
supporting DynamicMapField.
Remove most of the vtable, since this can now be done directly in MapFieldBase without derived type support.
PiperOrigin-RevId: 717984824
The invariant is that PtrAndLen may hold ptr+len values which are legal for _either_ C++ string_view or Rust slices: constructing one from either Rust or C++ permits the laxest constraints, and care must be taken when converting a PtrAndLen into either type.
For "into Rust slice" case to handle is that len=0 ptr must be non-null, so len=0 ptr=null gets turned into an arbitrary non-null pointer as provided by std::ptr::NonNull::dangling() (any preexisting non-null ptr is kept in that direction).
For the "into C++ slice" the risk is more obscure that a Rust non-null pointer could potentially be an illegal pointer (such that ptr+0 is not a legal operation in C++), so when going into C++ we map any len=0 cases into ptr=null to avoid this as a possible risk.
PiperOrigin-RevId: 704776210
The data is not currently used. This change is what we need to hook it up from Rust. In a future change we will start using the data to simplify the interface.
No expected performance change. The data inserted via static typing is constant evaluated, and fits in the existing padding (in 64-bit builds).
Also, fix Rust bindings to take float/double into account now that the enum lists them.
PiperOrigin-RevId: 698780360
This CL deletes the per-message C++ functions for operating on repeated fields
and replaces them with functions in the runtime that can work with arbitrary
messages.
Similar to what we did with maps, this required refactoring the code to make it
work with `RepeatedPtrFieldBase`, the untyped base class of
`RepeatedPtrField<T>`. I added a `RustRepeatedMessageHelper` class to allow us
access to the protected methods we need.
This should save a bit of linker input code size, but I think more importantly
we are going to need this eventually to enable tree shaking to work well.
PiperOrigin-RevId: 693394959
This CL migrates messages, enums, and primitive types all onto the same blanket
implementation of the `ProxiedInMapValue` trait. This gets us to the point
where messages and enums no longer need to generate any significant amount of
extra code just in case they might be used as a map value.
There are a few big pieces to this:
- I generalized the message-specific FFI endpoints in `rust/cpp_kernel/map.cc`
to be able to additionally handle enums and primitive types as values. This
mostly consisted of replacing `MessageLite*` parameters with a new `MapValue`
tagged union.
- On the Rust side, I added a new blanket implementation of
`ProxiedInMapValue` in rust/cpp.rs. It relies on its value type to implement
a new `CppMapTypeConversions` trait so that it can convert to and from the
`MapValue` tagged union used for FFI.
- In the Rust generated code, I deleted the generated `ProxiedInMapValue`
implementations for messages and enums and replaced them with
implementations of the `CppMapTypeConversions` trait.
PiperOrigin-RevId: 687355817
We generate these constants to enable map operations, but this is no longer
necessary now that we can get the relevant size and alignment information for
each message through its vtable.
PiperOrigin-RevId: 680712939
We have been relying on a per-message generated `placement_new` function for
implementing map insertion, but this CL simplifies things by removing that.
Instead, we do a reflective swap if possible, or else fall back on a copy.
This will probably make insertions a bit slower, but I think it may be worth it
because it should make it much simpler to have a blanket implementation for
ProxedInMapValue that works for all map types.
It looks like it should be possible to make this faster in the future by
implementing a bitwise move that will work for any message.
PiperOrigin-RevId: 676495920
`UntypedMapIterator::next_unchecked` currently expects to be passed a pointer
to a C++ function that it can use to dereference an iterator. However, this is
awkward because it's not natural for this C++ function to have the same
signature for every map type. Maps with a message as value need a
`MapNodeSizeInfo`, but other map types do not. We are working around this by
sometimes passing an ignored placeholder constant, but this is messy.
This CL replaces the `extern "C"` functions with closures. This way, we can
capture the `MapNodeSizeInfo` in the closure in cases where we need it, but
otherwise we no longer need to pass around placeholder values.
(Note: `MapNodeSizeInfo` is going away soon, but this is still relevant because
it will likely need to be replaced by a message default instance pointer.)
PiperOrigin-RevId: 676438640
Putting it into BUILD files unintentionally forces it on all our downstream users. Instead, we just want to enable this during testing and let them choose for themselves in their builds.
Note, that this expands the scope of -Werror to our entire repo for CI, so a bunch of fixes and opt-outs had to be applied to get this change passing.
Closed#14714
PiperOrigin-RevId: 666903224
This change adds delete, clear, serialize, parse, copy_from, and merge_from
operations to the runtime. Since these operations can all be implemented easily
on the `MessageLite` interface, we can use a common implementation in the
runtime instead of generating per-message thunks for all of these.
I suspect this will also make it possible to remove some of our generated trait
implementations and replace them with blanket implementations, but I will leave
that for a future change.
PiperOrigin-RevId: 665910927
With the C++ kernel for Rust, we currently need to generate quite a few C++
thunks for operations on map fields. For each message we generate, we generate
these thunks for all possible map types that could have that message as a
value. These operations are for things such as insertion, removal, clearing,
iterating, etc.
The reason we do this is that templated types don't play well with FFI, so we
effectively need separate FFI endpoints for every possible combination of key
and value types used (or even potentially used) as a map field.
This CL fixes the problem by replacing the generated thunks with functions in
the runtime that can operate on `proto2::MessageLite*` without needing to care
about the specific message type.
The way it works is that we implement the operations using either
`UntypedMapBase` (the base class of all map types, which knows nothing about
the key and value types) or `KeyMapBase`, which knows the key type but not the
value type. I roughly followed the example of the table-driven parser, which
has a similar problem of needing to operate generically on maps without having
access to the concrete types.
I removed 54 thunks per message (that's 6 key types times 9 operations per
key), but had to add two new thunks per message:
- The `size_info` thunk looks up the `MapNodeSizeInfoT`, which is stored in a
small constant table. The important thing here is an offset indicating where
to look for the value in each map entry. This offset can be different for
every pair of key and value types, but we can safely assume that the result
does not depend on the signedness of the key. As a result we only need to
store four entries per message: one each for i32, i64, bool, and string.
- The `placement_new` thunk move-constructs a message in place. We need this
to be able to efficiently implement map insertion.
There are two big things that this CL does not address yet but which I plan to
follow up on:
- Enums still generate many map-related C++ thunks that could be replaced with
a common implementation. This should actually be much easier to handle than
messages, because every enum has the same representation as an i32.
- We still generate six `ProxiedInMapValue` implementations for every message,
but it should be possible to replace these with a blanket implementation that
works for all message types.
PiperOrigin-RevId: 657681421
By changing the return type to `auto`, we can handle `std::string` and other
types in a single definition without needing a separate overload.
PiperOrigin-RevId: 653272253
Our ASAN test runs have not had the heap checker enabled, so this has allowed a
few memory leaks to slip in. This CL fixes all of them so that we can turn on
the heap checker.
The first one takes place whenever we add an entry into a string-valued map
using the C++ kernel. The problem is that `InnerProtoString::into_raw()` gives
up ownership of the raw `std::string` pointer it holds, but then we never
delete that pointer. This CL fixes the problem by deleting the pointer in C++
right after we perform the map insertion. To simplify things, I created a
`MakeCleanup()` helper function that we always call in our map insertion
thunks, but it's a no-op in the cases where we don't need to free anything.
There were a couple similar memory leaks related to repeated field accessors in
the C++ kernel, and those were simple to fix just by adding the necessary
`delete` call.
Finally, there were two benign memory leaks in the upb kernel involving global
variables used for empty repeated fields and maps. It turned out that we did
not need to use `Box` at all here, so removing that simplified things and fixed
the leaks.
PiperOrigin-RevId: 652947042
