renderdiff: add a filter option for local testing (#9257)

This option will allow for rendering and comparing a single image or files that match a wildcard-style string. README.md has been updated
2025-09-25 14:18:35 -07:00
parent a1b825b5b4
commit f233b20427
5 changed files with 137 additions and 95 deletions
--- a/test/renderdiff/README.md
+++ b/test/renderdiff/README.md
@@ -3,122 +3,128 @@
 We created a few scripts to run `gltf_viewer` and produce headless renderings.

 This is mainly useful for continuous integration where GPUs are generally not available on cloud
-machines. To perform software rasterization, these scripts are centered around [Mesa]'s software
-rasterizers, but nothing bars us from using another rasterizer like [SwiftShader]. Additionally,
-we should be able to use GPUs where available (though this is more of a future work).
+machines. To perform software rasterization, these scripts are centered around [Mesa]'s
+software rasterizers, but nothing bars us from using another rasterizer like [SwiftShader].
+Additionally, we should be able to use GPUs where available (though this is more of a future
+work).

 The script `render.py` contains the core logic for taking input parameters (such as the test
 description file) and then running gltf_viewer to produce the renderings.

-In the `test` directory is a list of test descriptions that are specified in json.  Please see
+In the `test` directory is a list of test descriptions that are specified in json. Please see
 `sample.json` to parse the structure.

 ## Setting up python
+
 The `renderdiff` project uses `python` extensively. To install the dependencies for producing
 renderings, do the following step
- - Set up a virtual environment (from the root directory)
-   ```
-     python3 -m venv venv
-     . ./venv/bin/activate
-   ```
- - Install the rendering dependencies
-   ```
-     pip install -r test/renderdiff/src/rendering_requirements.txt
-   ```
- - Install the viewer depdencies
-   ```
-     pip install -r test/renderdiff/src/viewer_requirements.txt
-   ```
- - For the commands in the following section, do not exit the virtual environment. Once you've
-   completed all your work, you can exit with
-   ```
-     deactivate
-   ```
+- Set up a virtual environment (from the root directory)
+  ```
+  python3 -m venv venv
+  . ./venv/bin/activate
+  ```
+- Install the rendering dependencies
+  ```
+  pip install -r test/renderdiff/src/rendering_requirements.txt
+  ```
+- Install the viewer depdencies
+  ```
+  pip install -r test/renderdiff/src/viewer_requirements.txt
+  ```
+- For the commands in the following section, do not exit the virtual environment. Once you've
+  completed all your work, you can exit with
+  ```
+  deactivate
+  ```

 ## Running the test locally
- - To run the same presbumit as [`test-renderdiff`](presubmit-renderdiff), you can do

-   ```
-   bash test/renderdiff/local_test.sh
-   ```
+- To run the same presbumit as [`test-renderdiff`](presubmit-renderdiff), you can do
+  ```
+  bash test/renderdiff/local_test.sh
+  ```
+- This script will generate the renderings based on the current state of your repo.
+  Additionally, it will also compare the generated images with corresponding images in the
+  golden repo.
+- To just render without running the test, you could use the following script
+  ```
+  bash test/renderdiff/generate.sh
+  ```

- - This script will generate the renderings based on the current state of your repo.
-   Additionally, it will also compare the generated images with corresponding images in the
-   golden repo.
- - To just render without running the test, you could use the following script
+## Filtering Tests

-   ```
-   bash test/renderdiff/generate.sh
-   ```
+You can run a subset of the tests by passing the `--test_filter` flag to `local_test.sh`. The
+filter supports wildcards (`*`) to match test names. For example, to run all tests in the
+`ClearCoat` suite, you can use the following command:
+
+```
+bash test/renderdiff/local_test.sh --test_filter='ClearCoat.*.*'
+```

 ## Update the golden images
-The golden images are stored in a github repository: https://github.com/google/filament-assets.
-Filament team members should have access to write to the repository. A typical flow for updating
-the goldens is to upload your changed images into **branch** of `filament-assets`. This branch is
-paired with a PR or commit on the `filament` repo.

-As an example, imagine I am working on a PR, and I've uploaded my change, which is in a branch
-called `my-pr-branch`, to `filament`. This PR requires updating the golden. We would do it
-in the following fashion
+The golden images are stored in a github repository:
+https://github.com/google/filament-assets. Filament team members should have access to write
+to the repository. A typical flow for updating the goldens is to upload your changed images
+into **branch** of `filament-assets`. This branch is paired with a PR or commit on the
+`filament` repo.
+
+As an example, imagine I am working on a PR, and I've uploaded my change, which is in a
+branch called `my-pr-branch`, to `filament`. This PR requires updating the golden. We would do
+it in the following fashion

 ### Using a script to update the golden repo

- - Run interactive mode in the `update_golden.py` script.
-
-   ```
-   python3 test/renderdiff/src/update_golden.py
-   ```
-
- - This will guide you through a series of steps to push the changes to a remote branch
-   on `filament-assets`.
+- Run interactive mode in the `update_golden.py` script.
+  ```
+  python3 test/renderdiff/src/update_golden.py
+  ```
+- This will guide you through a series of steps to push the changes to a remote branch on
+  `filament-assets`.

 ### Manually updating the golden repo

- - Check out the golden repo
-
-   ```
-   git clone git@github.com:google/filament-assets.git
-   ```
-
- - Create a branch on the golden repo
-
-   ```
-   cd filament-assets
-   git switch -c my-pr-branch-golden
-   ```
-
- - Copy the new images to their appropriate place in `filament-assets`
- - Push the `filament-assets` working branch to remote
-
-   ```
-   git push origin my-pr-branch-golden
-   ```
-
- - In the commit message of your working branch on `filament`, add the following line
-
-   ```
-   RDIFF_BBRANCH=my-pr-branch-golden
-   ```
+- Check out the golden repo
+  ```
+  git clone git@github.com:google/filament-assets.git
+  ```
+- Create a branch on the golden repo
+  ```
+  cd filament-assets
+  git switch -c my-pr-branch-golden
+  ```
+- Copy the new images to their appropriate place in `filament-assets`
+- Push the `filament-assets` working branch to remote
+  ```
+  git push origin my-pr-branch-golden
+  ```
+- In the commit message of your working branch on `filament`, add the following line
+  ```
+  RDIFF_BBRANCH=my-pr-branch-golden
+  ```

 Doing the above has multiple effects:
- - The presubmit test [`test-renderdiff`][presubmit-renderdiff] will test against the provided
-   branch of the golden repo (i.e. `my-pr-branch-golden`).
- - If the PR is merged, then there is another workflow that will merge `my-pr-branch-golden` to
-   the `main` branch of the golden repo.
+- The presubmit test [`test-renderdiff`][presubmit-renderdiff] will test against the provided
+  branch of the golden repo (i.e. `my-pr-branch-golden`).
+- If the PR is merged, then there is another workflow that will merge `my-pr-branch-golden`
+  to the `main` branch of the golden repo.

 ## Viewing test results
-We provide a viewer for looking at the result of a test run. The viewer is a webapp that can be used by
-pointing your browser to a localhost port. If you input the viewer with a PR or a directory, it will
-parse the test result and show the results and the rendered and/or golden images.
+
+We provide a viewer for looking at the result of a test run. The viewer is a webapp that can
+be used by pointing your browser to a localhost port. If you input the viewer with a PR or a
+directory, it will parse the test result and show the results and the rendered and/or golden
+images.

 ![Viewer](docs/images/renderdiff_example.png)

-To run the viewer of a test output directory that has been generated locally, you would run the
-following
+To run the viewer of a test output directory that has been generated locally, you would run
+the following

 ```
 python3 test/renderdiff/src/viewer.py --diff=[test output]
 ```
+
 where `[test output]` is a directory containing the `compare_results.json` of the test run.
 For example, it could be `out/renderdiff/diffs/presubmit` for the standard path to the
 `presubmit` test output.
@@ -129,10 +135,10 @@ To see the results of a Pull Request initiated test run, you would do the follow
 python3 test/renderdiff/src/viewer.py --pr_number=[PR #] --github_token=[github token]
 ```

-where `[PR #]` is the numeric ID of your pull request, and the `[github token]` is an acess token
-that you (as a github user) needs to generate ([reference][github_token_ref]).
+where `[PR #]` is the numeric ID of your pull request, and the `[github token]` is an acess
+token that you (as a github user) needs to generate ([reference][github_token_ref]).

 [github_token_ref]: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens
 [Mesa]: https://docs.mesa3d.org
 [SwiftShader]: https://github.com/google/swiftshader
-[presubmit-renderdiff]: https://github.com/google/filament/blob/e85dfe75c86106a05019e13ccdbef67e030af675/.github/workflows/presubmit.yml#L118
+[presubmit-renderdiff]: https://github.com/google/filament/blob/e85dfe75c86106a05019e13ccdbef67e030af675/.github/workflows/presubmit.yml#L118
--- a/test/renderdiff/generate.sh
+++ b/test/renderdiff/generate.sh
@@ -56,11 +56,26 @@ function end_render_() {
 #  - Build gltf_viewer
 #  - Run a test

+for i in "$@"
+do
+case $i in
+    --test_filter=*)
+    TEST_FILTER="${i#*=}"
+    shift # past argument=value
+    ;;
+    *)
+          # unknown option
+    ;;
+esac
+done
+
+
 start_render_ && \
    python3 ${RENDERDIFF_TEST_DIR}/src/render.py \
            --gltf_viewer="$(pwd)/out/cmake-debug/samples/gltf_viewer" \
            --test="${RENDERDIFF_TEST_DIR}/tests/presubmit.json" \
            --output_dir="${RENDER_OUTPUT_DIR}" \
            --opengl_lib="${MESA_LIB_DIR}" \
-            --vk_icd="${MESA_VK_ICD_PATH}" && \
+            --vk_icd="${MESA_VK_ICD_PATH}" \
+            ${TEST_FILTER:+--test_filter="$TEST_FILTER"} && \
    end_render_
--- a/test/renderdiff/local_test.sh
+++ b/test/renderdiff/local_test.sh
@@ -25,12 +25,13 @@ else
    GOLDEN_BRANCH=$(git log -1 | python3 test/renderdiff/src/commit_msg.py)
 fi

-bash `dirname $0`/generate.sh && \
+bash `dirname $0`/generate.sh "$@" && \
    python3 ${RENDERDIFF_TEST_DIR}/src/golden_manager.py \
            --branch=${GOLDEN_BRANCH} \
            --output=${GOLDEN_OUTPUT_DIR} && \
+    # Pass arguments to compare.py, e.g. --test_filter
    python3 ${RENDERDIFF_TEST_DIR}/src/compare.py \
            --src=${GOLDEN_OUTPUT_DIR} \
            --dest=${RENDER_OUTPUT_DIR} \
-            --out=${DIFF_OUTPUT_DIR}
+            --out=${DIFF_OUTPUT_DIR} "$@"
 end_
--- a/test/renderdiff/src/compare.py
+++ b/test/renderdiff/src/compare.py
@@ -3,13 +3,19 @@ import os
 import sys
 import pprint
 import json
+import fnmatch

 from utils import execute, ArgParseImpl, important_print, mkdir_p
 from image_diff import same_image, output_image_diff
 from results import RESULT_OK, RESULT_FAILED, RESULT_MISSING, GOLDEN_MISSING

-def _compare_goldens(base_dir, comparison_dir, out_dir=None):
+def _compare_goldens(base_dir, comparison_dir, out_dir=None, test_filter=None):
+  def test_name(p):
+    return p.replace('.tif', '')
+
  all_files = glob.glob(os.path.join(base_dir, "./**/*.tif"), recursive=True)
+  all_files = [os.path.abspath(f) for f in all_files \
+               if not test_filter or fnmatch.fnmatch(test_name(os.path.basename(f)), test_filter)]
  test_dirs = set(os.path.abspath(os.path.dirname(f)).replace(os.path.abspath(base_dir) + '/', '') \
                  for f in all_files)
  all_results = []
@@ -43,13 +49,17 @@ def _compare_goldens(base_dir, comparison_dir, out_dir=None):
    comp_test_dir = os.path.abspath(os.path.join(comparison_dir, test_dir))
    results = [
      single_test(base_test_dir, comp_test_dir, golden_file) \
-      for golden_file in glob.glob(os.path.join(base_test_dir, "*.tif"))
+      for golden_file in all_files if os.path.dirname(golden_file) == base_test_dir
    ]
    seen_test_cases = set([r['name'] for r in results])

    # For files that are rendered but not in the golden directory
-    for base_file in \
-        glob.glob(os.path.join(comp_test_dir, "*.tif")):
+    comparison_files = glob.glob(os.path.join(comp_test_dir, "*.tif"))
+    if test_filter:
+      comparison_files = [f for f in comparison_files \
+                          if fnmatch.fnmatch(test_name(os.path.basename(f)), test_filter)]
+
+    for base_file in comparison_files:
      src_fname = os.path.abspath(base_file)
      test_case = base_file.replace(f'{comp_test_dir}/', '')
      if test_case not in seen_test_cases:
@@ -76,6 +86,7 @@ if __name__ == '__main__':
  parser.add_argument('--src', help='Directory of the base of the diff.', required=True)
  parser.add_argument('--dest', help='Directory of the comparison of the diff.')
  parser.add_argument('--out', help='Directory of output for the result of the diff.')
+  parser.add_argument('--test_filter', help='Filter for the tests to run')

  args, _ = parser.parse_known_args(sys.argv[1:])

@@ -85,7 +96,7 @@ if __name__ == '__main__':
    dest = os.path.join(os.getcwd(), './out/renderdiff')
  assert os.path.exists(dest), f"Destination folder={dest} does not exist."

-  results = _compare_goldens(args.src, dest, out_dir=args.out)
+  results = _compare_goldens(args.src, dest, out_dir=args.out, test_filter=args.test_filter)

  failed = [f"   {k['name']} ({k['result']})" for k in results if k['result'] != RESULT_OK]
  success_count = len(results) - len(failed)
--- a/test/renderdiff/src/render.py
+++ b/test/renderdiff/src/render.py
@@ -18,6 +18,8 @@ import json
 import glob
 import shutil
 import concurrent.futures
+import fnmatch
+

 from utils import execute, ArgParseImpl, mkdir_p, mv_f, important_print

@@ -80,7 +82,8 @@ def _render_test_config(gltf_viewer,
            output_dir,
            local_only=False,
            opengl_lib=None,
-            vk_icd=None):
+            vk_icd=None,
+            test_filter=None):
  assert os.path.isdir(output_dir), f"output directory {output_dir} does not exist"
  assert os.access(gltf_viewer, os.X_OK)

@@ -103,6 +106,10 @@ def _render_test_config(gltf_viewer,
        if backend == 'vulkan':
          assert vk_icd, "VK ICD must be specified when testing vulkan backend"
        for model in test.models:
+          test_name = f'{test.name}.{backend}.{model}'
+          if test_filter and not fnmatch.fnmatch(test_name, test_filter):
+            print(f'Skipping {test_name} because it does not match filter')
+            continue
          model_path = os.path.abspath(test_config.models[model])
          futures.append(
            executor.submit(_render_single_model, gltf_viewer_abs,
@@ -122,6 +129,7 @@ if __name__ == "__main__":
  parser.add_argument('--output_dir', help='Output Directory', required=True)
  parser.add_argument('--opengl_lib', help='Path to the folder containing OpenGL driver lib (for LD_LIBRARY_PATH)')
  parser.add_argument('--vk_icd', help='Path to VK ICD file')
+  parser.add_argument('--test_filter', help='Filter for the tests to run')

  args, _ = parser.parse_known_args(sys.argv[1:])
  test = test_config.parse_from_path(args.test)
@@ -131,7 +139,8 @@ if __name__ == "__main__":
                        test,
                        args.output_dir,
                        opengl_lib=args.opengl_lib,
-                        vk_icd=args.vk_icd)
+                        vk_icd=args.vk_icd,
+                        test_filter=args.test_filter)

  with open(f'{output_dir}/render_results.json', 'w') as f:
    f.write(json.dumps(results, indent=2))