docs: new page with mdbook (#8342)

- Use mdbook to generate a book from collection of documents
 - documents are
     - copied and processed from READMEs
     - original content stored in the docs_src folder
     - markdeep docs transformed into embedded html
 - Main logic is in script docs_src/build/run.py

docs_src/.gitignore

@@ -0,0 +1,2 @@
+book
+src/dup/*.md

docs_src/README.md

@@ -0,0 +1,89 @@
+# Documentation
+
+Filament's documentation (which you are reading) is a collection of pages created with [`mdBook`].
+
+## How the book is created and updated {#how-to-create}
+### Prerequisites
+ - Install [`mdBook`] for your platform
+ - `selenium` package for python
+   ```shell
+   python3 -m pip install selenium
+   ```
+
+### Generate  {#how-to-generate}
+We wrote a python script to gather and transform the different documents in the project tree into a
+single book. This script can be found in [`docs_src/build/run.py`]. In addition,
+[`docs_src/build/duplicates.json`] is used to describe the markdown files that are copied and
+transformed from the source tree. These copies are placed into `docs_src/src/dup`.
+
+To collect the pages and generate the book, run the following
+```shell
+cd docs_src
+python3 build/run.py
+```
+
+### Copy to `docs`
+`docs` is the github-specfic directory for producing a web frontend (i.e. documentation) for a
+project.
+
+(To be completed)
+
+## Document sources
+We list the different document sources and how they are copied and processed into the collection
+of markdown files that are then processed with `mdBook`.
+
+### Introductory docs {#introductory-doc}
+The [github landing page] for Filament displays an extensive introduction to Filament. It
+links to `BUILDING.md` and `CONTRIBUTING.md`, which are conventional pages for building or
+contributing to the project. We copy these pages from their respective locations in the project
+tree into `docs_src/src/dup`. Moreover, to restore valid linkage between the pages, we need
+to perform a number of URL replacements in addition to the copy. These replacements are
+described in [`docs_src/build/duplicates.json`].
+
+### Core concept docs
+The primary design of Filament as a phyiscally-based renderer and details of its materials
+system are described in `Filament.md.html` and `Materials.md.html`, respectively. These two
+documents are written in [`markdeep`]. To embed them into our book, we
+ 1. Convert the markdeep into html
+ 2. Embed the html output in a markdown file
+ 3. Place the markdown file in `docs_src/src/main`
+
+We describe step 1 in detail for the sake of record:
+ - Start a local-only server to serve the markdeep file (e.g. `Filament.md.html`)
+ - Start a `selenium` driver (essentially run chromium in headless mode)
+ - Visit the local page through the driver (i.e. open url `http://localhost:xx/Filament.md.html?export`)
+ - Parse out the exported output in the retrieved html (note that the output of the markdeep
+   export is an html with the output captured in a `<pre>` tag).
+ - Replace css styling in the exported output as needed (so they don't interfere with the book's css.
+ - Replace resource urls to refer to locations relative to the mdbook structure.
+
+### READMEs
+Filament depends on a number of libraries, which reside in the directory `libs`. These individual
+libaries often have README.md in their root to describe itself. We collect these descriptions into our
+book. In addition, client usage of Filament also requires using a set of binary tools, which are
+located in `tools`. Some of tools also have README.md as description. We also collect them into the book.
+
+The process for copying and processing these READMEs is outlined in [Introductory docs](#introductory-doc).
+
+### Other technical notes
+These are technical documents that do not fit into a library, tool, or directory of the
+Filament source tree. We collect them into the `docs_src/src/notes` directory. No additional
+processing are needed for these documents.
+
+## Adding more documents
+To add any documentation, first consider the type of the document you like to add. If it
+belongs to any of the above sources, then simply place the document in the appropriate place,
+add a link in `SUMMARY.md`, and perform the steps outlined in
+[how-to create section](#how-to-create).
+
+For example, if you are adding a general technical note, then you would
+ - Place the document (file with extension `.md`) in `docs_src/src/notes`
+ - Add a link in [`docs_src/src/SUMMARY.md`]
+ - Run the commands in the [Generate](#how-to-generate) section
+
+[github landing page]: https://google.github.io/filament
+[`mdBook`]: https://rust-lang.github.io/mdBook/
+[`markdeep`]: https://casual-effects.com/markdeep/
+[`docs_src/build/run.py`]: https://github.com/google/filament/blob/main/docs_src/build/run.py
+[`docs_src/build/duplicates.json`]: https://github.com/google/filament/blob/main/docs_src/build/duplicates.json
+[`docs_src/src/SUMMARY.md`]: https://github.com/google/filament/blob/main/docs_src/src/SUMMARY.md

docs_src/book.toml

@@ -0,0 +1,20 @@
+[book]
+authors = []
+language = "en"
+multilingual = false
+src = "src"
+title = "Filament"
+
+[build]
+create-missing = false
+
+[output.html]
+mathjax-support = true
+default-theme = "light"
+preferred-dark-theme = "light"
+
+[output.html.print]
+enable = false
+
+[output.html.fold]
+enable = false

docs_src/build/duplicates.json

@@ -0,0 +1,74 @@
+{
+    "README.md": {
+        "dest": "dup/intro.md",
+        "link_transforms": {
+            "BUILDING.md": "building.md",
+            "/CONTRIBUTING.md": "contributing.md",
+            "/CODE_STYLE.md": "code_style.md",
+            "docs/images/samples": "../images/samples"
+        }
+    },
+    "BUILDING.md": {
+        "dest": "dup/building.md"
+    },
+    "CONTRIBUTING.md": {
+        "dest": "dup/contributing.md"
+    },
+    "CODE_STYLE.md": {
+        "dest": "dup/code_style.md"
+    },
+    "libs/uberz/README.md": {
+        "dest": "dup/uberz.md"
+    },
+    "libs/bluegl/README.md": {
+        "dest": "dup/bluegl.md"
+    },
+    "libs/bluevk/README.md": {
+        "dest": "dup/bluevk.md"
+    },
+    "libs/gltfio/README.md": {
+        "dest": "dup/gltfio.md"
+    },
+    "libs/filamat/README.md": {
+        "dest": "dup/filamat.md"
+    },
+    "libs/iblprefilter/README.md": {
+        "dest": "dup/iblprefilter.md"
+    },
+    "libs/matdbg/README.md": {
+        "dest": "dup/matdbg.md"
+    },
+    "tools/normal-blending/README.md": {
+        "dest": "dup/normal_blending.md"
+    },
+    "tools/filamesh/README.md": {
+        "dest": "dup/filamesh.md"
+    },
+    "tools/beamsplitter/README.md": {
+        "dest": "dup/beamsplitter.md"
+    },
+    "tools/cmgen/README.md": {
+        "dest": "dup/cmgen.md"
+    },
+    "tools/mipgen/README.md": {
+        "dest": "dup/mipgen.md"
+    },
+    "tools/matinfo/README.md": {
+        "dest": "dup/matinfo.md"
+    },
+    "tools/roughness-prefilter/README.md": {
+        "dest": "dup/roughness_prefilter.md"
+    },
+    "tools/zbloat/README.md": {
+        "dest": "dup/zbloat.md"
+    },
+    "tools/cso-lut/README.md": {
+        "dest": "dup/cso_lut.md"
+    },
+    "tools/specular-color/README.md": {
+        "dest": "dup/specular_color.md"
+    },
+    "docs_src/README.md": {
+        "dest": "dup/docs.md"
+    }
+}

docs_src/build/run.py

@@ -0,0 +1,128 @@
+# Copyright (C) 2025 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import json
+import os
+import re
+from utils import execute, ArgParseImpl
+
+CUR_DIR = os.path.dirname(os.path.abspath(__file__))
+DOCS_SRC_DIR = os.path.join(CUR_DIR, '../')
+ROOT_DIR = os.path.join(CUR_DIR, '../../')
+SRC_DIR = os.path.join(CUR_DIR, '../src')
+MARKDEEP_DIR = os.path.join(CUR_DIR, '../markdeep')
+DUP_DIR = os.path.join(SRC_DIR, 'dup')
+MAIN_DIR = os.path.join(SRC_DIR, 'main')
+
+def transform_dup_file_link(line, transforms):
+  URL_CONTENT = '[-a-zA-Z0-9()@:%_\+.~#?&//=]+'
+  res = re.findall(f'\[(.+)\]\(({URL_CONTENT})\)', line)
+  for text, url in  res:
+    word = f'[{text}]({url})'
+    for tkey in transforms.keys():
+      if url.startswith(tkey):
+        nurl = url.replace(tkey, transforms[tkey])
+        line = line.replace(word, f'[{text}]({nurl})')
+        break
+  return line
+
+def pull_duplicates():
+  if not os.path.exists(DUP_DIR):
+    os.mkdir(DUP_DIR)
+
+  config = {}
+  with open(f'{CUR_DIR}/duplicates.json') as config_txt:
+    config = json.loads(config_txt.read())
+
+  for fin in config.keys():
+    new_name = config[fin]['dest']
+    link_transforms = config[fin].get('link_transforms', {})
+    fpath = os.path.join(ROOT_DIR, fin)
+    new_fpath = os.path.join(SRC_DIR, new_name)
+
+    with open(fpath, 'r') as in_file:
+      with open(new_fpath, 'w') as out_file:
+        for line in in_file.readlines():
+          out_file.write(transform_dup_file_link(line, link_transforms))
+
+def pull_markdeep_docs():
+  import http.server
+  import socketserver
+  import threading
+  from selenium import webdriver
+  from selenium.webdriver.chrome.options import Options
+  from selenium.webdriver.common.by import By
+  import time
+
+  class Server(socketserver.ThreadingMixIn, http.server.HTTPServer):
+    """Handle requests in a separate thread."""
+
+  class Handler(http.server.SimpleHTTPRequestHandler):
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, directory=MARKDEEP_DIR, **kwargs)
+
+  def start_server(port):
+    """Starts the web server in a separate thread."""
+    httpd = Server(("", port), Handler)
+    server_thread = threading.Thread(target=httpd.serve_forever)
+    server_thread.daemon = True  # Allow main thread to exit
+    server_thread.start()
+    print(f"Server started on port {port}...")
+    return httpd
+
+  PORT = 12345
+  httpd = start_server(PORT)
+
+  # Set up Chrome options for headless mode
+  chrome_options = Options()
+  chrome_options.add_argument("--headless")
+
+  # This option is necessary for running on some VMs
+  chrome_options.add_argument("--no-sandbox")
+
+  # Create a new Chrome instance in headless mode
+  driver = webdriver.Chrome(options=chrome_options)
+
+  for doc in ['Filament', 'Materials']:
+    # Open the URL with ?export, which markdeep will export the resulting html.
+    driver.get(f"http://localhost:{PORT}/{doc}.md.html?export")
+
+    time.sleep(3)
+    # We extract the html from the resulting "page" (an html output itself).
+    text = driver.find_elements(By.TAG_NAME, "pre")[0].text
+
+    # 1. Remove the double empty lines.  These make the following text seem like markdown text as oppose to embedded html.
+    # 2. Remove the max-width styling from the body tag.
+    # 3. Remove the font-family styling from the body tag.
+    text = text.replace("\n\n","\n")\
+               .replace("max-width:680px;", "")\
+               .replace("font-family:Palatino", "--font-family:Palatino")\
+               .replace("\"./images", "\"../images")\
+               .replace("\"images/", "\"../images/")
+
+    # Save the page source as .md with embedded html
+    with open(f'{MAIN_DIR}/{doc.lower()}.md', "w", encoding="utf-8") as f:
+      f.write(text)
+
+  # Close the browser
+  driver.quit()
+  # Shutdown the server
+  httpd.shutdown()
+
+if __name__ == "__main__":
+  pull_duplicates()
+  pull_markdeep_docs()
+
+  res, err = execute('mdbook build', cwd=DOCS_SRC_DIR)
+  assert res == 0, f"failed to execute `mdbook`. return-code={res} err=\"{err}\""

docs_src/build/utils.py

@@ -0,0 +1,68 @@
+# Copyright (C) 2025 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import subprocess
+import os
+import argparse
+import sys
+
+def execute(cmd,
+            cwd=None,
+            capture_output=True,
+            stdin=None,
+            env=None,
+            raise_errors=False):
+  in_env = os.environ
+  in_env.update(env if env else {})
+  home = os.environ['HOME']
+  if f'{home}/bin' not in in_env['PATH']:
+    in_env['PATH'] = in_env['PATH'] + f':{home}/bin'
+
+  stdout = subprocess.PIPE if capture_output else sys.stdout
+  stderr = subprocess.PIPE if capture_output else sys.stdout
+  output = ''
+  err_output = ''
+  return_code = -1
+  kwargs = {
+      'cwd': cwd,
+      'env': in_env,
+      'stdout': stdout,
+      'stderr': stderr,
+      'stdin': stdin,
+      'universal_newlines': True
+  }
+  if capture_output:
+    process = subprocess.Popen(cmd.split(' '), **kwargs)
+    output, err_output = process.communicate()
+    return_code = process.returncode
+  else:
+    return_code = subprocess.call(cmd.split(' '), **kwargs)
+
+  if return_code:
+    # Error
+    if raise_errors:
+      raise subprocess.CalledProcessError(return_code, cmd)
+  if output:
+    if type(output) != str:
+      try:
+        output = output.decode('utf-8').strip()
+      except UnicodeDecodeError as e:
+        print('cannot decode ', output, file=sys.stderr)
+  return return_code, (output if return_code == 0 else err_output)
+
+class ArgParseImpl(argparse.ArgumentParser):
+  def error(self, message):
+    sys.stderr.write('error: %s\n' % message)
+    self.print_help()
+    sys.exit(1)

docs_src/markdeep/README.md

@@ -0,0 +1,10 @@
+# Markdeep documents
+
+Markdeep documents require special processing before they can be compiled into the group.
+The "original" are stored in this folder. The processing part takes place in
+`docs_src/build/run.py`.
+
+## Editing
+While editing the file, you might consider doing the following
+ - `python3 -m http.server 8001`
+ - visit `http://localhost:8001/Filament.md.html` in the browser to view the result

docs_src/src/SUMMARY.md

@@ -0,0 +1,39 @@
+# Summary
+
+- [Introduction](./dup/intro.md)
+  - [Build](./dup/building.md)
+    - [Build for Android on Windows](./build/windows_android.md)
+  - [Contribute](./dup/contributing.md)
+  - [Coding Style](./dup/code_style.md)
+- [Core Concepts](./main/README.md)
+  - [Filament](./main/filament.md)
+  - [Materials](./main/materials.md)
+- [Tutorials and Samples](./samples/README.md)
+  - [iOS Tutorial](./samples/ios.md)
+  - [Web Tutorial](./samples/web.md)
+- [Technical Notes](./notes/README.md)
+  - [Versioning](./notes/versioning.md)
+  - [Documentation](./dup/docs.md)
+  - [Debugging](./notes/debugging.md)
+    - [Metal](./notes/metal_debugging.md)
+    - [Vulkan](./notes/vulkan_debugging.md)
+    - [SPIR-V](./notes/spirv_debugging.md)
+  - [Libraries](./notes/libs.md)
+    - [bluegl](./dup/bluegl.md)
+    - [bluevk](./dup/bluevk.md)
+    - [filamat](./dup/filamat.md)
+    - [gltfio](./dup/gltfio.md)
+    - [iblprefilter](./dup/iblprefilter.md)
+    - [matdbg](./dup/matdbg.md)
+    - [uberz](./dup/uberz.md)
+  - [Tools](./notes/tools.md)
+    - [beamsplitter](./dup/beamsplitter.md)
+    - [cmgen](./dup/cmgen.md)
+    - [cso-lut](./dup/cso_lut.md)
+    - [filamesh](./dup/filamesh.md)
+    - [normal-blending](./dup/normal_blending.md)
+    - [mipgen](./dup/mipgen.md)
+    - [matinfo](./dup/matinfo.md)
+    - [roughness-prefilter](./dup/roughness_prefilter.md)
+    - [specular-color](./dup/specular_color.md)
+    - [zbloat](./dup/zbloat.md)

docs_src/src/dup/README.txt

@@ -0,0 +1,2 @@
+Do not manually edit any file in this folder. They have been autogenerated
+by a script.