Files
Mathias Agopian 167a91efaa Optimize Shader LineDictionary (#9814)
* Optimize Shader LineDictionary with Variable-Length 3-Streams

This commit completely reorganizes the string dictionary compression
pipeline used by compiled Material Text Chunks and improves matinfo
dictionary output.

1. Multi-Base Variable-Length Scaling:
We replaced the static 16-bit indices overhead with a
bounded payload. Now, indices scale dynamically:
- 0 to 239 evaluate in 1 byte.
- 240 to 3584 leverage 0xF0-0xFD escapes evaluating in 2 bytes.
- 3584+ are locked behind a 0xFF marker to 3 bytes.
This natively eradicated the massive monolithic lengths and zero-padding
issues previously dominating shader packages.

2. Variable-Length 3-Stream Decoding:
To solve the Zstandard/Zlib entropy fragmentation that conventionally
plagues interleaved variable byte lengths (which previously inflated
our `filament.aar` boundary constraint by +2KB), we segregated the encoded
payloads.

By grouping high-entropy string boundaries into a `Base Stream`
and isolating offset digits inside an `Extension Stream`,
predictive LZ77 ZIP sliding-windows perfectly map
over both arrays independently without disruption.

3. Optimize Numeric Stream using LEB128
Prior to this change, numerical suffixes split from shader
variables (e.g., `param_1024` -> `param_` + `1024`) were fed back
into the localized String Dictionary. Because high-frequency numbers were
assigned disjointed localized IDs per shader variant, LZ77
failed to cross-reference their repetitive structures across
shipped `.aar` archives, fracturing compression sequences.

This patch implements a unified 3-Stream topology. It
extracts numerical primitives (< 32768) away from the baseline
String Dictionary, writing them into an isolated, contiguous
LEB128 array.

By using a dedicated `[254]` Escape Token within the primary stream, numerical
variables maintain exact 1-byte (`< 128`) or 2-byte (`>= 128`)
geometric layouts across all permutations.

The resulting deterministic alignment guarantees that Zlib sliding windows
can deduplicate highly repetitive variables across the
entire application binary block.

4. We use the ShaderStage information to create distinct index ranges, which
further help use 1-byte indices.

Verification Metrics:
 `filament-android.aar`: -7,938 B
 `gltfio-android.aar`: -290,939 B
 `libfilament.a`: -18,464 B

* Optimize shader dictionary by decoding '_' for numeric literals

Most numbers extracted from the shader text are preceded by an 
underscore (e.g., from `_`, `hp_copy_`), which previously caused 
standalone `_` strings to heavily pollute the LineDictionary.

This change removes the standalone `_` from the dictionary index:
- `MaterialChunk` rehydrates the `_` prefix when decoding these numeric
literals.

This frees up dictionary indices, yielding massive byte savings across 
uncompressed binaries (e.g., -28.4 KB for volume_masked.filamat).

* Optimize ShaderMinifier to strip explicit spacing

Spirv-cross outputs GLSL with explicit spacing around generic
operators (e.g., ` = `, `, `, ` ) * `). This padding consumes a 
significant amount of uncompressed bytes across large ubershaders. 

By applying targeted string replacements at the end of the `ShaderMinifier` 
pass, we strip this extraneous padding down to its raw tokens 
(e.g., `a=b`, `a,b`, `a*b`). 

This optimization preserves isolating spaces where valuable, ensuring 
line-dictionary tokens (such as raw `=` or `,`)  remain deduplicated 
instead of fusing into unpredictable variables. 

Impact:
This saves roughly ~9.1 KB in `libfilament.a` and ~3.2 KB in 
`volume_masked.filamat` uncompressed, with proportional gains across 
the downstream LZ4 compressed archives.
2026-03-26 22:41:36 -07:00
..

Filamat

Filamat allows for generating materials programatically on the device as opposed to with the matc tool on the host machine. The cost is a binary size increase of your app due to the relatively larger size of the filamat library.

For a smaller-sized library, see filamat_lite. It has no dependencies on glslang, but can only compile materials for OpenGL and does no shader code optimization.

The filamat package is included in the releases available on GitHub.

Libraries

Filamat is distributed as a set of static libraries you must link against:

  • filamat, Filamat main library
  • filabridge, Support library for Filament / Filamat
  • shaders, Shader text for material generation
  • utils, Support library for Filament / Filamat
  • smol-v, SPIR-V compression library

To use Filamat from Java you must use the following two libraries instead:

  • filamat-java.jar, Contains Filamat's Java classes
  • filamat-jni, Filamat's JNI bindings

Linking against Filamat

This walkthrough will get you successfully compiling and linking native code against Filamat with minimum dependencies.

To start, download Filament's latest binary release and extract into a directory of your choosing. Binary releases are suffixed with the platform name, for example, filament-20181009-linux.tgz.

Create a file, main.cpp, in the same directory with the following contents:

#include <filamat/MaterialBuilder.h>

#include <iostream>

using namespace filamat;

int main(int argc, char** argv)
{
    // Must be called before any materials can be built.
    MaterialBuilder::init();

    MaterialBuilder builder;
    builder
        .name("My material")
        .material("void material (inout MaterialInputs material) {"
                  "  prepareMaterial(material);"
                  "  material.baseColor.rgb = float3(1.0, 0.0, 0.0);"
                  "}")
        .shading(MaterialBuilder::Shading::LIT)
        .targetApi(MaterialBuilder::TargetApi::ALL)
        .platform(MaterialBuilder::Platform::ALL);

    Package package = builder.build();
    if (package.isValid()) {
        std::cout << "Success!" << std::endl;
    }

    // Call when finished building all materials to release internal MaterialBuilder resources.
    MaterialBuilder::shutdown();
    return 0;
}

The directory should look like:

|-- README.md
|-- bin
|-- docs
|-- include
|-- lib
|-- main.cpp

We'll use a platform-specific Makefile to compile and link main.cpp with Filamat's libraries. Copy your platform's Makefile below into a Makefile inside the same directory.

Linux

FILAMENT_LIBS=-lfilamat -lfilabridge -lshaders -lutils -lsmol-v
CC=clang++

main: main.o
	$(CC) -Llib/x86_64/ -stdlib=libc++ main.o $(FILAMENT_LIBS) -lpthread -ldl -o main

main.o: main.cpp
	$(CC) -Iinclude/ -std=c++20 -stdlib=libc++ -pthread -c main.cpp

clean:
	rm -f main main.o

.PHONY: clean

macOS

FILAMENT_LIBS=-lfilamat -lfilabridge -lshaders -lutils -lsmol-v
CC=clang++

main: main.o
	$(CC) -Llib/x86_64/ main.o $(FILAMENT_LIBS) -o main

main.o: main.cpp
	$(CC) -Iinclude/ -std=c++20 -c main.cpp

clean:
	rm -f main main.o

.PHONY: clean

Windows

Note that the static libraries distributed for Windows include several variants: mt, md, mtd, mdd. These correspond to the run-time library flags /MT, /MD, /MTd, and /MDd, respectively. Here we use the mt variant.

When building Filamat from source, the USE_STATIC_CRT CMake option can be used to change the run-time library version.

FILAMENT_LIBS=lib/x86_64/mt/filamat.lib lib/x86_64/mt/filabridge.lib lib/x86_64/mt/shaders.lib \
              lib/x86_64/mt/utils.lib lib/x86_64/mt/smol-v.lib
CC=clang-cl.exe

main.exe: main.obj
	$(CC) main.obj $(FILAMENT_LIBS) gdi32.lib user32.lib opengl32.lib

main.obj: main.cpp
	$(CC) /MT /Iinclude/ /std:c++20 /c main.cpp

clean:
	del main.exe main.obj

.PHONY: clean

Compiling

You should be able to invoke make and run the executable successfully:

$ make
$ ./main
Success!

On Windows, you'll need to open up a Visual Studio Native Tools Command Prompt and invoke nmake instead of make.

Using the Material with Filament

For simplicity, this demo doesn't do anything useful with the built material package. To use the material with Filament, pass the material package's data into a Filament Material builder:

    Package package = builder.build();
    filament::Material* myMaterial = Material::Builder()
        .package(package.getData(), package.getSize())
        .build(*engine);

Note that this will require linking against Filament's libraries in addition to Filamat's.

Filamat Lite

The filamat_lite library is interchangeable with filamat, with a few caveats:

  1. Material compilation is only supported for the OpenGL backend.
  2. No shader-level optimization is performed.
  3. GLSL correctness is not checked.

In addition, filamat_lite only performs a simple text match to determine which properties on the MaterialInputs structure are set. The material input variable must also always be refered to by the name material.

void anotherFunction(inout MaterialInputs m) {
    // Incorrect! The MaterialInputs is being referred to by the name "m".
    m.metallic = 0.0;
}

void aFunction(inout MaterialInputs material) {
    // Works, but only because the variable name "material" is used.
    material.reflectance = 0.5;
}

// The MaterialInputs variable must be named material.
void material(inout MaterialInputs material) {
    prepareMaterial(material);

    // Good.
    material.roughness = materialParams.roughness;
    material.baseColor.rgb = vec3(1.0, 0.0, 1.0);

    aFunction(material);
    anotherFunction(material);
}