dep-protobuf/regenerate_stale_files.sh

47 lines
1.6 KiB
Bash
Raw Normal View History

#!/bin/bash
# This script runs the staleness tests and uses them to update any stale
# generated files.
set -ex
echo "::group::Regenerate stale files"
# Cd to the repo root.
cd $(dirname -- "$0")
readonly BazelBin="${BAZEL:-bazel} ${BAZEL_STARTUP_FLAGS}"
STALENESS_TESTS=(
"csharp:generated_csharp_defaults_staleness_test"
"java/core:generated_java_defaults_staleness_test"
"upb/reflection:bootstrap_upb_defaults_staleness_test"
"cmake:test_dependencies_staleness"
"src:cmake_lists_staleness_test"
"src/google/protobuf:well_known_types_staleness_test"
"objectivec:well_known_types_staleness_test"
"php:test_amalgamation_staleness"
"php:proto_staleness_test"
"ruby/ext/google/protobuf_c:test_amalgamation_staleness"
"upb/reflection:descriptor_upb_proto_staleness_test"
"upb/reflection:json_enumvalue_options_upb_proto_staleness_test"
"upb_generator:plugin_upb_proto_staleness_test"
)
# Run and fix all staleness tests.
for test in ${STALENESS_TESTS[@]}; do
protobuf: improve staleness test regeneration to reduce confusion This change updates `regenerate_stale_files.sh` to always execute a direct fix and refactors `staleness_test_lib.py` to support cleaner output during regeneration without sacrificing debugging info. Motivation: Automatic staleness tests are run to regenerate stale files on non-bazel environments (like GHA). When these files change, the tests "fail" to trigger regeneration. This causes constant confusion for developers as they look like test failures rather than "Working As Intended" regeneration triggers. Details: 1. **`regenerate_stale_files.sh`**: Updated to always build the staleness tests via `bazel build` and execute them directly with `--fix --print-diffs`, avoiding the confusing "FAIL" output of `bazel test`. This script is primarily used by automation (like GHA) where this behavior is desired by default. 2. **`staleness_test_lib.py`**: Refactored diff generation into a shared helper `_GetDiffErrors`. Both `CheckFilesMatch` (testing) and `FixFiles` (fixing) use this helper. Added a `print_diffs` parameter to `FixFiles` so diffs are only printed when explicitly requested (e.g., in `regenerate_stale_files.sh`), preventing double-printing if run manually. 3. **Tone Improvement**: Updated the message for missing files during fixing to be less alarming ("Creating missing file" instead of "File does not exist"). ### Output Comparison **New Automation Output**: Clean execution without `AssertionError` boilerplate, but preserves actionable diffs. ``` + ./bazel-bin/upb/reflection/descriptor_upb_proto_staleness_test --fix --print-diffs Creating missing file upb/reflection/stage0/descriptor_upb_proto.h File upb/reflection/descriptor_upb_proto.c is out of date: --- upb/reflection/descriptor_upb_proto.c.generated +++ upb/reflection/descriptor_upb_proto.c @@ -45,6 +45,7 @@ #include "upb/reflection/descriptor_upb_proto.h" UPB_NOINLINE const upb_MiniTable* google_protobuf_FileDescriptorSet_msg_init() { + // Added a new field return &google_protobuf_FileDescriptorSet_msg_init_table; } ``` **Traditional Test Failure (Manual Runs)**: Verbose, traditional test failure (for manual runs via `bazel test`). ``` + bazel test upb/reflection:descriptor_upb_proto_staleness_test ... FAIL: upb/reflection:descriptor_upb_proto_staleness_test (Exit 1) ================================================================================ AssertionError: Files out of date! To fix run THIS command: bazel-bin/upb/reflection/descriptor_upb_proto_staleness_test --fix Errors: File upb/reflection/descriptor_upb_proto.c is out of date: --- upb/reflection/descriptor_upb_proto.c.generated +++ upb/reflection/descriptor_upb_proto.c ... ``` PiperOrigin-RevId: 963179635
2026-08-11 21:00:54 -07:00
${BazelBin} build $test "$@"
./bazel-bin/${test%%:*}/${test#*:} --fix --print-diffs
done
# Generate C# code.
# This doesn't currently have Bazel staleness tests, but there's an existing
# shell script that generates everything required. The output files are stable,
# so just regenerating in place should be harmless.
${BazelBin} build src/google/protobuf/compiler:protoc "$@"
Fix for performance penalty in deep dependencies (#20392) The goal of this PR is to fix #20371 (closed) #24267 and #24330 and improve the performance of csharp runtime when dealing with deep nested dependencies of .proto files. - The fix consists in caching the recursive calls for the search of extensions. - Two identical datasets (nested proto files) are created automatically using Bazel: one to test it with caching enabled, and another one to test it with no caching. Note: the same dataset cannot be used because once it is loaded inside the test, it cannot be unloaded. - The created datasets have a width of 6 and a depth of 6, enough to showcase the dependency impact and to not impact the testing time for unit tests. This could be configured. - In the dataset there is only one proto file with a message `Example` whose descriptor is loaded, and by doing so all the dependencies (and sub dependencies) that it has. - A set of benchmark metrics have been created to evaluate the performance impact before and after the fix. - The assertion inside the unit test makes sure that when using metrics the number of traversed extensions (dependencies of the dependencies) when using caching, is reduced by a factor of 1000x. - The second assertion also makes sure that the time used to load the descriptor when using cache is less than the one without using cache. Closes #20392 COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/20392 from fgarciacorona:fix_20371_csharp_deep_dependencies b1353f6fd1ca02a0ade27b7974a0ac640eb36c35 PiperOrigin-RevId: 919713760
2026-05-22 09:23:52 -07:00
${BazelBin} build //csharp/protos/unittest_deep_dependencies:generate_cached_protos "$@"
${BazelBin} build //csharp/protos/unittest_deep_dependencies:generate_notcached_protos "$@"
(export PROTOC=$PWD/bazel-bin/protoc && cd csharp && ./generate_protos.sh)
echo "::endgroup::"