dep-protobuf/conformance

Protocol Buffers - Google's data interchange format

Copyright 2008 Google LLC.

This directory contains conformance tests for testing completeness and correctness of Protocol Buffers implementations. These tests are designed to be easy to run against any Protocol Buffers implementation.

This directory contains the tester process conformance-test, which contains all of the tests themselves. Then separate programs written in whatever language you want to test communicate with the tester program over a pipe.

If you're not using Bazel to run these tests, make sure you build the C++ tester code beforehand, e.g. from the base directory:

$ cmake . -Dprotobuf_BUILD_CONFORMANCE=ON && cmake --build .

This will produce a conformance_test_runner binary that can be used to run conformance tests on any executable. Pass it --help for more information.

Running the tests for C++

To run the tests against the C++ implementation, run:

$ bazel test //src:conformance_test

Or alternatively with CMake:

$ ctest -R conformance_cpp_test

Running the tests for other languages

All of the languages in the Protobuf source tree are set up to run conformance tests using similar patterns. You can either use Bazel to run the conformance_test target defined in the language's root BUILD.bazel file, or create an executable for a custom test and pass it to conformance_test_runner.

Note: CMake can be used to build the conformance test runner, but not any of the conformance test executables outside C++. So if you aren't using Bazel you'll need to create the executable you pass to conformance_test_runner via some alternate build system.

While we plan to model all our supported languages more completely in Bazel, today some of them are a bit tricky to run. Below is a list of the commands (and prerequisites) to run each language's conformance tests.

C#:

$ which dotnet || echo "You must have dotnet installed!"
$ bazel test //csharp:conformance_test \
    --action_env=DOTNET_CLI_TELEMETRY_OPTOUT=1 --test_env=DOTNET_CLI_HOME=~ \
    --action_env=DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1

Java:

$ bazel test //java/core:conformance_test //java/lite:conformance_test

Objective-C (Mac only):

$ bazel test //objectivec:conformance_test --macos_minimum_os=12.0

PHP:

$ bazel test //php:conformance_test

PHP (C):

$ which gcc     || echo "gcc is required!"
$ which libtool || echo "libtool is required!"
$ which make    || echo "make is required!"
$ which pear    || echo "pear is required! It might require a development version of PHP such as a php-dev package"
$ which pecl    || echo "pecl is required! It might require a development version of PHP such as a php-dev package"
$ which phpize  || echo "phpize is required! It might require a development version of PHP such as a php-dev package"
$ bazel test //php:conformance_test_c

Python:

$ bazel test //python:conformance_test

Python (C++):

$ bazel test //python:conformance_test_cpp --define=use_fast_cpp_protos=true

Ruby:

$ [[ $(ruby --version) == "ruby"* ]] || echo "Select a C Ruby!"
$ bazel test //ruby:conformance_test --define=ruby_platform=c \
    --action_env=PATH --action_env=GEM_PATH --action_env=GEM_HOME

Ruby (JRuby):

$ [[ $(ruby --version) == "jruby"* ]] || echo "Switch to Java Ruby!"
$ bazel test //ruby:conformance_test_jruby --define=ruby_platform=java \
    --action_env=PATH --action_env=GEM_PATH --action_env=GEM_HOME

Testing other Protocol Buffer implementations

To run these tests against a new Protocol Buffers implementation, write a program in your language that uses the protobuf implementation you want to test. This program should implement the testing protocol defined in conformance.proto. This is designed to be as easy as possible: the C++ version is only 150 lines and is a good example for what this program should look like (see conformance_cpp.cc). The program only needs to be able to read from stdin and write to stdout.

Portability

Note that the test runner currently does not work on Windows. Patches to fix this are welcome! (But please get in touch first to settle on a general implementation strategy).