167a91efaa
* Optimize Shader LineDictionary with Variable-Length 3-Streams This commit completely reorganizes the string dictionary compression pipeline used by compiled Material Text Chunks and improves matinfo dictionary output. 1. Multi-Base Variable-Length Scaling: We replaced the static 16-bit indices overhead with a bounded payload. Now, indices scale dynamically: - 0 to 239 evaluate in 1 byte. - 240 to 3584 leverage 0xF0-0xFD escapes evaluating in 2 bytes. - 3584+ are locked behind a 0xFF marker to 3 bytes. This natively eradicated the massive monolithic lengths and zero-padding issues previously dominating shader packages. 2. Variable-Length 3-Stream Decoding: To solve the Zstandard/Zlib entropy fragmentation that conventionally plagues interleaved variable byte lengths (which previously inflated our `filament.aar` boundary constraint by +2KB), we segregated the encoded payloads. By grouping high-entropy string boundaries into a `Base Stream` and isolating offset digits inside an `Extension Stream`, predictive LZ77 ZIP sliding-windows perfectly map over both arrays independently without disruption. 3. Optimize Numeric Stream using LEB128 Prior to this change, numerical suffixes split from shader variables (e.g., `param_1024` -> `param_` + `1024`) were fed back into the localized String Dictionary. Because high-frequency numbers were assigned disjointed localized IDs per shader variant, LZ77 failed to cross-reference their repetitive structures across shipped `.aar` archives, fracturing compression sequences. This patch implements a unified 3-Stream topology. It extracts numerical primitives (< 32768) away from the baseline String Dictionary, writing them into an isolated, contiguous LEB128 array. By using a dedicated `[254]` Escape Token within the primary stream, numerical variables maintain exact 1-byte (`< 128`) or 2-byte (`>= 128`) geometric layouts across all permutations. The resulting deterministic alignment guarantees that Zlib sliding windows can deduplicate highly repetitive variables across the entire application binary block. 4. We use the ShaderStage information to create distinct index ranges, which further help use 1-byte indices. Verification Metrics: `filament-android.aar`: -7,938 B `gltfio-android.aar`: -290,939 B `libfilament.a`: -18,464 B * Optimize shader dictionary by decoding '_' for numeric literals Most numbers extracted from the shader text are preceded by an underscore (e.g., from `_`, `hp_copy_`), which previously caused standalone `_` strings to heavily pollute the LineDictionary. This change removes the standalone `_` from the dictionary index: - `MaterialChunk` rehydrates the `_` prefix when decoding these numeric literals. This frees up dictionary indices, yielding massive byte savings across uncompressed binaries (e.g., -28.4 KB for volume_masked.filamat). * Optimize ShaderMinifier to strip explicit spacing Spirv-cross outputs GLSL with explicit spacing around generic operators (e.g., ` = `, `, `, ` ) * `). This padding consumes a significant amount of uncompressed bytes across large ubershaders. By applying targeted string replacements at the end of the `ShaderMinifier` pass, we strip this extraneous padding down to its raw tokens (e.g., `a=b`, `a,b`, `a*b`). This optimization preserves isolating spaces where valuable, ensuring line-dictionary tokens (such as raw `=` or `,`) remain deduplicated instead of fusing into unpredictable variables. Impact: This saves roughly ~9.1 KB in `libfilament.a` and ~3.2 KB in `volume_masked.filamat` uncompressed, with proportional gains across the downstream LZ4 compressed archives.
215 lines
7.8 KiB
C++
215 lines
7.8 KiB
C++
/*
|
|
* Copyright (C) 2025 The Android Open Source Project
|
|
*
|
|
* Licensed under the Apache License, Version 2.0 (the "License");
|
|
* you may not use this file except in compliance with the License.
|
|
* You may obtain a copy of the License at
|
|
*
|
|
* http://www.apache.org/licenses/LICENSE-2.0
|
|
*
|
|
* Unless required by applicable law or agreed to in writing, software
|
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
* See the License for the specific language governing permissions and
|
|
* limitations under the License.
|
|
*/
|
|
|
|
#include <gtest/gtest.h>
|
|
|
|
#include "eiff/LineDictionary.h"
|
|
#include <private/filament/LineDictionaryUtils.h>
|
|
|
|
#include <string>
|
|
|
|
using namespace filamat;
|
|
using namespace ::filament::backend;
|
|
|
|
TEST(LineDictionary, splitString) {
|
|
LineDictionary dictionary;
|
|
const std::string text = "first line hp_copy_123456 second line";
|
|
dictionary.addText(ShaderStage::FRAGMENT, text);
|
|
EXPECT_EQ(dictionary.size(), 4);
|
|
EXPECT_EQ(dictionary[0], "first line ");
|
|
EXPECT_EQ(dictionary[1], "hp_copy");
|
|
EXPECT_EQ(dictionary[2], "123456"); // Bypassing fails because 123456 > 16383
|
|
EXPECT_EQ(dictionary[3], " second line");
|
|
}
|
|
|
|
TEST(LineDictionary, Empty) {
|
|
LineDictionary const dictionary;
|
|
EXPECT_TRUE(dictionary.empty());
|
|
EXPECT_EQ(dictionary.size(), 0);
|
|
}
|
|
|
|
TEST(LineDictionary, AddTextSimple) {
|
|
LineDictionary dictionary;
|
|
dictionary.addText(ShaderStage::FRAGMENT, "Hello world\n");
|
|
EXPECT_FALSE(dictionary.empty());
|
|
EXPECT_EQ(dictionary.size(), 1);
|
|
EXPECT_EQ(dictionary[0], "Hello world\n");
|
|
}
|
|
|
|
TEST(LineDictionary, AddTextMultipleLines) {
|
|
LineDictionary dictionary;
|
|
dictionary.addText(ShaderStage::FRAGMENT, "First line\nSecond line\n");
|
|
EXPECT_EQ(dictionary.size(), 2);
|
|
EXPECT_EQ(dictionary[0], "First line\n");
|
|
EXPECT_EQ(dictionary[1], "Second line\n");
|
|
}
|
|
|
|
TEST(LineDictionary, AddTextDuplicateLines) {
|
|
LineDictionary dictionary;
|
|
dictionary.addText(ShaderStage::FRAGMENT, "Same line\nSame line\n");
|
|
EXPECT_EQ(dictionary.size(), 1);
|
|
EXPECT_EQ(dictionary[0], "Same line\n");
|
|
}
|
|
|
|
TEST(LineDictionary, SplitLogicNoPattern) {
|
|
LineDictionary dictionary;
|
|
dictionary.addText(ShaderStage::FRAGMENT, "A simple line with no patterns.");
|
|
EXPECT_EQ(dictionary.size(), 1);
|
|
EXPECT_EQ(dictionary[0], "A simple line with no patterns.");
|
|
}
|
|
|
|
TEST(LineDictionary, SplitLogicHpPattern) {
|
|
LineDictionary dictionary;
|
|
dictionary.addText(ShaderStage::FRAGMENT, "some_var = hp_copy_123;");
|
|
EXPECT_EQ(dictionary.size(), 3);
|
|
EXPECT_EQ(dictionary[0], "some_var = ");
|
|
EXPECT_EQ(dictionary[1], "hp_copy");
|
|
EXPECT_EQ(dictionary[2], ";"); // 123 is bypassed!
|
|
}
|
|
|
|
TEST(LineDictionary, SplitLogicMpPattern) {
|
|
LineDictionary dictionary;
|
|
dictionary.addText(ShaderStage::FRAGMENT, "another_var = mp_copy_4567;");
|
|
EXPECT_EQ(dictionary.size(), 3);
|
|
EXPECT_EQ(dictionary[0], "another_var = ");
|
|
EXPECT_EQ(dictionary[1], "mp_copy");
|
|
EXPECT_EQ(dictionary[2], ";");
|
|
}
|
|
|
|
TEST(LineDictionary, SplitLogicUnderscorePattern) {
|
|
LineDictionary dictionary;
|
|
dictionary.addText(ShaderStage::FRAGMENT, "var_1 = 0;");
|
|
EXPECT_EQ(dictionary.size(), 1);
|
|
EXPECT_EQ(dictionary[0], "var_1 = 0;"); // Underscore preceded by a word char is not considered a pattern boundary
|
|
}
|
|
|
|
TEST(LineDictionary, SplitLogicMultiplePatterns) {
|
|
LineDictionary dictionary;
|
|
dictionary.addText(ShaderStage::FRAGMENT, "hp_copy_1 mp_copy_2 _3");
|
|
EXPECT_EQ(dictionary.size(), 3);
|
|
EXPECT_EQ(dictionary[0], "hp_copy");
|
|
EXPECT_EQ(dictionary[1], " ");
|
|
EXPECT_EQ(dictionary[2], "mp_copy");
|
|
}
|
|
|
|
TEST(LineDictionary, SplitLogicInvalidPattern) {
|
|
LineDictionary dictionary;
|
|
dictionary.addText(ShaderStage::FRAGMENT, "hp_copy_ a_b_c");
|
|
EXPECT_EQ(dictionary.size(), 1);
|
|
EXPECT_EQ(dictionary[0], "hp_copy_ a_b_c");
|
|
}
|
|
|
|
TEST(LineDictionary, SplitLogicPatternFollowedByWordChar) {
|
|
LineDictionary dictionary;
|
|
dictionary.addText(ShaderStage::FRAGMENT, "hp_copy_99rest");
|
|
EXPECT_EQ(dictionary.size(), 1);
|
|
EXPECT_EQ(dictionary[0], "hp_copy_99rest"); // Invalid word boundary on right side
|
|
}
|
|
|
|
TEST(LineDictionary, SplitLogicPatternPrecededByWordChar) {
|
|
LineDictionary dictionary;
|
|
dictionary.addText(ShaderStage::FRAGMENT, "rest_of_it_hp_copy_99");
|
|
EXPECT_EQ(dictionary.size(), 1);
|
|
EXPECT_EQ(dictionary[0], "rest_of_it_hp_copy_99"); // Preceded by word char
|
|
}
|
|
|
|
TEST(LineDictionary, SplitLogicPatternNotFollowedByWordChar) {
|
|
LineDictionary dictionary;
|
|
dictionary.addText(ShaderStage::FRAGMENT, "hp_copy_99;");
|
|
EXPECT_EQ(dictionary.size(), 2);
|
|
EXPECT_EQ(dictionary[0], "hp_copy");
|
|
EXPECT_EQ(dictionary[1], ";");
|
|
}
|
|
|
|
TEST(LineDictionary, AddEmptyText) {
|
|
LineDictionary dictionary;
|
|
dictionary.addText(ShaderStage::FRAGMENT, "");
|
|
EXPECT_TRUE(dictionary.empty());
|
|
}
|
|
|
|
TEST(LineDictionary, GetIndicesMultiple) {
|
|
LineDictionary dictionary;
|
|
dictionary.addText(ShaderStage::FRAGMENT, "A _1 B _2");
|
|
dictionary.addText(ShaderStage::FRAGMENT, "A _1");
|
|
dictionary.resolve();
|
|
auto const [indices, numerics] = dictionary.tokenize("A _1"); // String is in dictionary
|
|
ASSERT_EQ(indices.size(), 2);
|
|
EXPECT_EQ(indices[0], 0); // "A "
|
|
EXPECT_EQ(indices[1], filament::LineDictionaryUtils::DICTIONARY_NUMERIC_FLAG); // 1
|
|
ASSERT_EQ(numerics.size(), 1);
|
|
EXPECT_EQ(numerics[0], 1);
|
|
}
|
|
|
|
TEST(LineDictionary, GetIndicesMultiplePatternsInARow) {
|
|
LineDictionary dictionary;
|
|
dictionary.addText(ShaderStage::FRAGMENT, "hp_copy_1 hp_copy_2");
|
|
dictionary.resolve();
|
|
auto const [indices, numerics] = dictionary.tokenize("hp_copy_1 hp_copy_2");
|
|
ASSERT_EQ(indices.size(), 5);
|
|
EXPECT_EQ(indices[0], 0); // hp_copy
|
|
EXPECT_EQ(indices[1], filament::LineDictionaryUtils::DICTIONARY_NUMERIC_FLAG);
|
|
EXPECT_EQ(indices[2], 1); // " "
|
|
EXPECT_EQ(indices[3], 0); // hp_copy
|
|
EXPECT_EQ(indices[4], filament::LineDictionaryUtils::DICTIONARY_NUMERIC_FLAG);
|
|
ASSERT_EQ(numerics.size(), 2);
|
|
EXPECT_EQ(numerics[0], 1);
|
|
EXPECT_EQ(numerics[1], 2);
|
|
}
|
|
|
|
TEST(LineDictionary, GetIndicesSamePatternMultipleTimes) {
|
|
LineDictionary dictionary;
|
|
dictionary.addText(ShaderStage::FRAGMENT, "hp_copy_1 hp_copy_1");
|
|
dictionary.resolve();
|
|
auto const [indices, numerics] = dictionary.tokenize("hp_copy_1 hp_copy_1");
|
|
ASSERT_EQ(indices.size(), 5);
|
|
EXPECT_EQ(indices[0], 0); // hp_copy
|
|
EXPECT_EQ(indices[1], filament::LineDictionaryUtils::DICTIONARY_NUMERIC_FLAG);
|
|
EXPECT_EQ(indices[2], 1); // " "
|
|
EXPECT_EQ(indices[3], 0); // hp_copy
|
|
EXPECT_EQ(indices[4], filament::LineDictionaryUtils::DICTIONARY_NUMERIC_FLAG);
|
|
ASSERT_EQ(numerics.size(), 2);
|
|
EXPECT_EQ(numerics[0], 1);
|
|
EXPECT_EQ(numerics[1], 1);
|
|
}
|
|
|
|
TEST(LineDictionary, GetIndicesWithAdjacentPatterns) {
|
|
LineDictionary dictionary;
|
|
dictionary.addText(ShaderStage::FRAGMENT, "hp_copy_1hp_copy_2");
|
|
dictionary.resolve();
|
|
auto const [indices, numerics] = dictionary.tokenize("hp_copy_1hp_copy_2");
|
|
ASSERT_EQ(indices.size(), 1);
|
|
EXPECT_EQ(indices[0], 0);
|
|
}
|
|
|
|
TEST(LineDictionary, GetIndicesWithAdjacentPatternsNotInDictionary) {
|
|
LineDictionary dictionary;
|
|
dictionary.addText(ShaderStage::FRAGMENT, "hp_copy_1");
|
|
dictionary.addText(ShaderStage::FRAGMENT, "hp_copy_2");
|
|
dictionary.resolve();
|
|
auto const [indices, numerics] = dictionary.tokenize("hp_copy_1hp_copy_2");
|
|
ASSERT_EQ(indices.size(), 0);
|
|
}
|
|
|
|
TEST(LineDictionary, GetIndicesWithMixedContent) {
|
|
LineDictionary dictionary;
|
|
dictionary.addText(ShaderStage::FRAGMENT, "hp_copy_1");
|
|
dictionary.addText(ShaderStage::FRAGMENT, " ");
|
|
dictionary.addText(ShaderStage::FRAGMENT, "mp_copy_2");
|
|
dictionary.resolve();
|
|
auto const [indices, numerics] = dictionary.tokenize("prefix hp_copy_1 mp_copy_2 suffix");
|
|
ASSERT_EQ(indices.size(), 0);
|
|
}
|