# Rendering Difference Test

This tool (`/test/renderdiff`) is a collections of scripts to run `gltf_viewer` and produce headless
renderings.

This is mainly useful for continuous integration where GPUs are generally not available on cloud
machines. To perform software rasterization, these scripts are centered around [Mesa]'s
software rasterizers, but nothing bars us from using another rasterizer like [SwiftShader].
Additionally, we should be able to use GPUs where available (though this is more of a future
work).

The script `render.py` contains the core logic for taking input parameters (such as the test
description file) and then running `gltf_viewer` to produce the renderings.

In the `test` directory is a list of test descriptions that are specified in json. Please see
`sample.json` to glean the structure.

## Setting up python

The `renderdiff` project uses `python` extensively. To install the dependencies for producing
renderings, do the following step
- Set up a virtual environment (from the root directory)
  ```
  python3 -m venv venv
  . ./venv/bin/activate
  ```
- Install the rendering dependencies
  ```
  pip install -r test/renderdiff/src/rendering_requirements.txt
  ```
- Install the viewer depdencies
  ```
  pip install -r test/renderdiff/src/viewer_requirements.txt
  ```
- For the commands in the following section, do not exit the virtual environment. Once you've
  completed all your work, you can exit with
  ```
  deactivate
  ```

## Running the test locally

- To run the same presbumit as [`test-renderdiff`](presubmit-renderdiff), you can do
  ```
  bash test/renderdiff/local_test.sh
  ```
- This script will generate the renderings based on the current state of your repo.
  Additionally, it will also compare the generated images with corresponding images in the
  golden repo.
- To just render without running the test, you could use the following script
  ```
  bash test/renderdiff/generate.sh
  ```

## Command-line Options

You can control the behavior of the test scripts with the following flags passed to `local_test.sh`:

- `--test_filter=<filter>`: Run a subset of tests. The filter supports wildcards (`*`) to match test names.
- `--no_rebuild`: Skip rebuilding the `gltf_viewer` executable.
- `--num_threads=<number>`: Set the number of threads for rendering. If not set, the system's default is used.

For example, to run all `MSAA` tests without rebuilding and using 8 threads:

```
bash test/renderdiff/local_test.sh --test_filter='MSAA.*.*' --no_rebuild --num_threads=8
```

## Update the golden images

The golden images are stored in a github repository:
https://github.com/google/filament-assets. Filament team members should have access to write
to the repository. A typical flow for updating the goldens is to upload your changed images
into **branch** of `filament-assets`. This branch is paired with a PR or commit on the
`filament` repo.

As an example, imagine I am working on a PR, and I've uploaded my change, which is in a
branch called `my-pr-branch`, to `filament`. This PR requires updating the golden. We would do
it in the following fashion on a macOS machine:

### Using a script to update the golden repo

- Make sure you've completed the steps in 'Setting up python'
- Run interactive mode in the `update_golden.py` script.
  ```
  python3 test/renderdiff/src/update_golden.py
  ```
- This will guide you through a series of steps to push the changes to a remote branch on
  `filament-assets`.

### Manually updating the golden repo

- Check out the golden repo
  ```
  git clone git@github.com:google/filament-assets.git
  ```
- Create a branch on the golden repo
  ```
  cd filament-assets
  git switch -c my-pr-branch-golden
  ```
- Copy the new images to their appropriate place in `filament-assets`
- Push the `filament-assets` working branch to remote
  ```
  git push origin my-pr-branch-golden
  ```
- In the commit message of your working branch on `filament`, add the following line
  ```
  RDIFF_BRANCH=my-pr-branch-golden
  ```

Doing the above has multiple effects:
- The presubmit test [`test-renderdiff`][presubmit-renderdiff] will test against the provided
  branch of the golden repo (i.e. `my-pr-branch-golden`).
- If the PR is merged, then there is another workflow that will merge `my-pr-branch-golden`
  to the `main` branch of the golden repo.

### Automated update via commit message

Alternatively, if you are confident in your changes and want the CI to handle the update
for you, you can use the following tag in your commit message:

- In the commit message of your working branch on `filament`, add the following line:
  ```
  RDIFF_ACCEPT_NEW_GOLDENS
  ```

This has the following effects:
- The presubmit test `test-renderdiff` will be bypassed (it will not perform rendering or
  comparison).
- When the PR is merged, the postsubmit CI will automatically:
    1. Build Filament and generate the new images.
    2. Upload them to a temporary branch in `filament-assets`.
    3. Merge that branch into `main`.

## Viewing test results

We provide a viewer for looking at the result of a test run. The viewer is a webapp that can
be used by pointing your browser to a localhost port. If you input the viewer with a PR or a
directory, it will parse the test result and show the results and the rendered and/or golden
images.

![Viewer](docs/images/renderdiff_example.png)

To run the viewer of a test output directory that has been generated locally, you would run
the following

```
python3 test/renderdiff/src/viewer.py --diff=[test output]
```

where `[test output]` is a directory containing the `compare_results.json` of the test run.
For example, it could be `out/renderdiff/diffs/presubmit` for the standard path to the
`presubmit` test output.

To see the results of a Pull Request initiated test run, you would do the following

```
python3 test/renderdiff/src/viewer.py --pr_number=[PR #] --github_token=[github token]
```

where `[PR #]` is the numeric ID of your pull request, and the `[github token]` is an acess
token that you (as a github user) needs to generate ([reference][github_token_ref]).

To see the results of a specific run, you would do the following

```
python3 test/renderdiff/src/viewer.py --run_number=[RUN #] --github_token=[github token]
```

where `[RUN #]` is the numeric ID of the run. You can find the run number in the URL of the
GitHub Actions page. For example, in the URL
`https://github.com/google/filament/actions/runs/18023632663/job/51286323708?pr=9264`,
the run number is `18023632663`.

[github_token_ref]: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens
[Mesa]: https://docs.mesa3d.org
[SwiftShader]: https://github.com/google/swiftshader
[presubmit-renderdiff]: https://github.com/google/filament/blob/e85dfe75c86106a05019e13ccdbef67e030af675/.github/workflows/presubmit.yml#L118