graph { graph [engine:dot rankdir:LR applyCss:true clusterize:true intention:semi grayscale:false desaturate:true] ast_lake -> ast_lake__commands_ [reverse: false] ast_lake__commands_ -> ast_lake__commands____ast_lake__info [reverse: false] ast_lake__commands_ -> ast_lake__commands____ast_lake__init [reverse: false] ast_lake__commands_ -> ast_lake__commands____ast_lake__reset [reverse: false] ast_lake__commands_ -> ast_lake__commands____ast_lake__cleanup [reverse: false] ast_lake__commands_ -> ast_lake__commands____ast_lake__restart [reverse: false] ast_lake__commands____ast_lake__restart -> ast_lake__commands____ast_lake__restart___ast_lake__commands____ast_lake__reset_nodes [reverse: false] ast_lake *= ast_lake__commands_ [reverse: false] ast_lake__commands_ *= ast_lake__commands____ast_lake__add [reverse: false] ast_lake__commands_ *= ast_lake__commands____ast_lake__populate [reverse: false] ast_lake__commands_ *= ast_lake__commands____ast_lake__query_TERM [reverse: false] ast_lake__commands_ *= ast_lake__commands____ast_lake__queryraw_TERM [reverse: false] ast_lake__commands_ *= ast_lake__commands____ast_lake__flast [reverse: false] ast_lake__commands_ *= ast_lake__commands____ast_lake__nodes [reverse: false] ast_lake__commands_ *= ast_lake__commands____ast_lake__mdg [reverse: false] ast_lake__commands_ *= ast_lake__commands____ast_lake__mosaic [reverse: false] ast_lake__commands_ *= ast_lake__commands____ast_lake__mprocs [reverse: false] ast_lake *= ast_lake__recent_updates [reverse: false] ast_lake__recent_updates *= ast_lake__recent_updates___ast_lake__2025_12_01 [reverse: false] db_tables -> db_tables__The_pipeline__how_data_flows_ [reverse: false] db_tables -> db_tables__cache_ast_lake_flast____cache_ast_lake_nodes____cache_ast_lake_nodes_ref____cache_ast_lake_nodes_backup_ [reverse: false] db_tables__cache_ast_lake_flast____cache_ast_lake_nodes____cache_ast_lake_nodes_ref____cache_ast_lake_nodes_backup_ -> db_tables__cache_ast_lake_flast____cache_ast_lake_nodes____cache_ast_lake_nodes_ref____cache_ast_lake_nodes_backup____db_tables__cache_ast_lake_flast_ [reverse: false] db_tables__cache_ast_lake_flast____cache_ast_lake_nodes____cache_ast_lake_nodes_ref____cache_ast_lake_nodes_backup_ -> db_tables__cache_ast_lake_flast____cache_ast_lake_nodes____cache_ast_lake_nodes_ref____cache_ast_lake_nodes_backup____db_tables__cache_ast_lake_nodes_ [reverse: false] db_tables__cache_ast_lake_flast____cache_ast_lake_nodes____cache_ast_lake_nodes_ref____cache_ast_lake_nodes_backup_ -> db_tables__cache_ast_lake_flast____cache_ast_lake_nodes____cache_ast_lake_nodes_ref____cache_ast_lake_nodes_backup____db_tables__cache_ast_lake_nodes_ref_ [reverse: false] db_tables__cache_ast_lake_flast____cache_ast_lake_nodes____cache_ast_lake_nodes_ref____cache_ast_lake_nodes_backup_ -> db_tables__cache_ast_lake_flast____cache_ast_lake_nodes____cache_ast_lake_nodes_ref____cache_ast_lake_nodes_backup____db_tables__cache_ast_lake_nodes_backup_ [reverse: false] db_tables *= db_tables__linetime_ [reverse: false] db_tables *= db_tables__repo_voting_ [reverse: false] ast_lake [z: 9.5 @html: "

ast-lake
usage: ast-lake 【command】【args...】

• 123⸨123⸩

• 【xxx

" nomen: "ast-lake"] ast_lake__commands_ [z: 4.75 @html: "

commands:

" nomen: "commands:"] ast_lake__commands____ast_lake__info [z: 2.375 @html: "

info

• info on the state of ast-lake

" nomen: "info"] ast_lake__commands____ast_lake__init [z: 2.375 @html: "

init

• precreate the cache db

" nomen: "init"] ast_lake__commands____ast_lake__reset [z: 2.375 @html: "

reset

• delete list

• delete all cached data

" nomen: "reset"] ast_lake__commands____ast_lake__cleanup [z: 2.375 @html: "

cleanup

• remove non existent files from sqlite
cache

" nomen: "cleanup"] ast_lake__commands____ast_lake__restart [z: 2.375 @html: "

restart

• keep list

• delete all cached data

" nomen: "restart"] ast_lake__commands____ast_lake__restart___ast_lake__commands____ast_lake__reset_nodes [z: 1.1875 @html: "

reset-nodes

• cleanup only cached nodes (dev)

• and recreated them

" nomen: "reset-nodes"] ast_lake__commands____ast_lake__add [z: 2.375 @html: "

add

• add files to list

• dbPopulateList is a temporary list file,
listing absolute file paths

• dbPopulateList is usually /tmp/ast-list.txt

• add absolute file paths:

• note: does use fd . honors .gitignore

• usage:

ast-lake add '*.mdd'
'*.json'

• will add **/*.mdd and *.json to list

• or manually:

• you may edit $dbPopulateList

# --base-directory $dir fd
--absolute-path -tf
".+【.】(mdd)$" | rg -v test
> $dbPopulateList fd -tf
--absolute-path
"$filterFilesPattern" >
$dbPopulateList | rg -v node_modules |
rg -v test > $dbPopulateList # or
only add arbitrary files: rg TERM
--files-with-matches | xargs realpath {}
>> $dbPopulateList

• see also ast-lake populate

" nomen: "add"] ast_lake__commands____ast_lake__populate [z: 2.375 @html: "

populate

• ensure all ast's (from on the list) are
up to date

" nomen: "populate"] ast_lake__commands____ast_lake__query_TERM [z: 2.375 @html: "

query TERM

• ast-lake query term

• TODO term patterns ?

• TODO regex: ^cs.+$

" nomen: "query TERM"] ast_lake__commands____ast_lake__queryraw_TERM [z: 2.375 @html: "

queryraw TERM

• same as query, but returns raw JSON

" nomen: "queryraw TERM"] ast_lake__commands____ast_lake__flast [z: 2.375 @html: "

flast

• basically a dump

• print all raw flASTs

" nomen: "flast"] ast_lake__commands____ast_lake__nodes [z: 2.375 @html: "

nodes

• print raw nodes

" nomen: "nodes"] ast_lake__commands____ast_lake__mdg [z: 2.375 @html: "

mdg

• TODO really?

• make one big mdg graph from all mdd's

" nomen: "mdg"] ast_lake__commands____ast_lake__mosaic [z: 2.375 @html: "

mosaic

• mosaic

• TODO really?

" nomen: "mosaic"] ast_lake__commands____ast_lake__mprocs [z: 2.375 @html: "

mprocs

• TODO support mprocs

" nomen: "mprocs"] ast_lake__recent_updates [z: 4.75 @html: "

recent updates

" nomen: "recent updates"] ast_lake__recent_updates___ast_lake__2025_12_01 [z: 2.375 @html: "

2025-12-01

• actualizeNodes: chunked parallel
processing

• chunk size: CPU count / 2

• faster database query processing

" nomen: "2025-12-01"] db_tables [z: 9.5 @html: "

db tables
These tables are in our ast-lake SQLite
database. They hold the state derived
from parsing mdd/md/js files into AST
nodes. Two parallel implementations exist:

• Shell-based ( ast-lake.sh ) — older approach using local SQLite +
nljson files

• Node.js-based ( ast-lake.mjs ) — newer approach using remote SQLite
via RemoteSqlite

Both follow the same pipeline: extract
flast → normalize → store nodes.

" nomen: "db tables"] db_tables__The_pipeline__how_data_flows_ [z: 4.75 @html: "

The pipeline how data flows

File on disk │ ▼ astHole() →
astNavigator() — reads file, calls
language-specific extractor │
(extractFlastMarkdown,
extractFlastJavascript, │
extractFlastJson, extractFlastBash, ...)
▼ cache_ast_lake_flast — one
row per file; id=filepath, value=JSON
flast array │ ▼ actualizeNodes() →
normalizeNodes() — extracts symbolic
types from flast │
(heading, link, comment,
FunctionDeclaration, Identifier) ▼
cache_ast_lake_nodes — one row
per AST node; id=fn:offset
cache_ast_lake_nodes_ref — same
rows, different ref format (readable
refId) cache_ast_lake_nodes_backup —
backup of old nodes before update

" nomen: "The pipeline how data flows"] db_tables__cache_ast_lake_flast____cache_ast_lake_nodes____cache_ast_lake_nodes_ref____cache_ast_lake_nodes_backup_ [z: 4.75 @html: "

cache_ast_lake_flast
Schema from sqlCreateTableCache(tblName) in ast-lake.mjs:51 . All four share the same column layout
but each column has different semantics per table :

Column	Type	Description
id	TEXT NOT NULL	Primary key
value	TEXT	JSON payload of the node or flast
mt	TEXT	Modification timestamp or mime-type
sem	TEXT	Semantic type
num1	INTEGER	First numeric slot (position or line)
num2	INTEGER	Second numeric slot (position or column)
ref	TEXT	Reference ID (trail or refId)
body	TEXT	Node body text

Shared indexes: num1 , num2 , sem , ref (each named {table}_idx_{col} ).

" nomen: "`cache_ast_lake_flast`, `cache_ast_lake_nodes`, `cache_ast_lake_nodes_ref`, `cache_ast_lake_nodes_backup`"] db_tables__cache_ast_lake_flast____cache_ast_lake_nodes____cache_ast_lake_nodes_ref____cache_ast_lake_nodes_backup____db_tables__cache_ast_lake_flast_ [z: 2.375 @html: "

cache_ast_lake_flast
Role: Raw flattened AST cache — one row per
file. Stores the complete AST produced
by the language-specific extractor.

Column	How it's populated (from actualizeNodes() )	Semantics
id	row【0】 — the file path (fn)	Full filesystem path to the source file
value	row【1】 — JSON.parse(row【1】) → data.flast	Full JSON array of all AST nodes extracted from the file (the "flattened AST")
mt	row【2】 — file modification timestamp	ISO timestamp of last file modification

How it's populated: Written by astNavigator.mjs via cacheFlast(fnIn, data) (line 1571), using initSqliteCache from sqlito.mjs . The data.value is the array of fragments produced by
extractors like extractFlastMarkdown , extractFlastJavascript , etc. Query pattern (in actualizeNodes() ):
SELECT id, value, mt FROM
cache_ast_lake_flast WHERE unixepoch(mt)
> <lastSync>
Only changed files (newer mtime) are
reprocessed. Node value structure (each element in the JSON flast array):
{ "id":
"node-id", "dim":
【"file",
"listItem.0",
"property.key"】,
"type":
"heading|link|code|paragraph|list|...",
"code": "raw text
content", "loc": {
"line": { "s": 1,
"e": 3 }, "column":
{ "s": 1, "e": 80 }
}, "symbName": null,
"symbRangeL": null }

" nomen: "`cache_ast_lake_flast`"] db_tables__cache_ast_lake_flast____cache_ast_lake_nodes____cache_ast_lake_nodes_ref____cache_ast_lake_nodes_backup____db_tables__cache_ast_lake_nodes_ [z: 2.375 @html: "

cache_ast_lake_nodes
Role: Individual AST node storage — one row
per symbolic node. Produced by normalizeNodes() in actualizeNodes() (ast-lake.mjs:183-266), which extracts
only symbolicTypes (heading, link, comment,
FunctionDeclaration, Identifier) from
the flast.

Column	How it's populated (ast-lake.mjs:516-561)	Semantics
id	`${node.fn}:${node.offset}`	File path + offset position → buildNodeAbsoluteId()
value	JSON.stringify(node)	Full normalized node as JSON (see structure below)
mt	node.mt	Modification time (symbMt from git blame, or file mtime)
sem	node.type	Node type: "heading" , "link" , "comment" , "FunctionDeclaration" , "Identifier"
num1	node.position【0】	Line number in source file (1-based)
num2	node.position【1】	Column number in source file (1-based)
ref	buildNodeRefId(node)	For headings: last trail segment (heading text). For links: nomen (link label). Defines the short ref used in voting queries.
body	node.outerCode \|\| node.code \|\| ''	Full text content of the node. For headings: the entire heading subtree text. For links: the markdown link syntax.

Normalized node value structure (the JSON in the value column):
{ "fn":
"/home/.../repo/repolero/paper/file.mdd",
"trail":
【"getting-started",
"installation"】,
"type": "heading",
"nomen":
"Installation",
"code": "##
Installation",
"symbName": null,
"symbBlame":
【"abc123def",
"456ghi789"】,
"outerCode": "full
subtree text...",
"position": 【42, 1】,
"lineRange": 【42, 85】,
"offset": 42,
"symbMt":
"2025-06-15T10:30:00Z",
"symbUser":
"author@example.com",
"mt":
"2025-06-15T10:30:00Z" }
Key normalization details (from normalizeNodes() at ast-lake.mjs:183):

• headings : processHeading() extracts level and pure text; breadcrumb
trail tracks hierarchy; currentLevel > lastLevel pushes, currentLevel <= lastLevel pops

• links : parseMarkdownLink() extracts 【label】(url) ; outerCode = url; nomen = label

• git blame : symbBlame array of commit hashes for lines in symbRangeL ; symbMt = most recent blame mtime; symbUser = most recent author

• Only nodes with types in symbolicTypes are kept; all others (paragraph, list,
code without heading context, etc.) are
filtered out

Query pattern (voting dashboard):
SELECT n.id, n.body, n.num1 as
headingLevel, r.ref FROM
cache_ast_lake_nodes n INNER JOIN
cache_ast_lake_nodes_ref r USING(id)
INNER JOIN repo_voting v ON v.ref =
r.ref WHERE n.sem = 'heading'

" nomen: "`cache_ast_lake_nodes`"] db_tables__cache_ast_lake_flast____cache_ast_lake_nodes____cache_ast_lake_nodes_ref____cache_ast_lake_nodes_backup____db_tables__cache_ast_lake_nodes_ref_ [z: 2.375 @html: "

cache_ast_lake_nodes_ref
Role: Readable refId mirror of nodes. Stores
the user-visible fragment reference
string (the one used in URLs and voting)
instead of the last-trail-segment ref.

Column	How it's populated (ast-lake.mjs:560)	Semantics
id	Same as nodes: fn:offset	Joins to cache_ast_lake_nodes.id
sem	node.type	Node type (same as nodes.sem)
ref	buildNodeAbsoluteId(node)	Readable refId — format: relative/path/file.mdd::sanitized/trail/path . For links: ...::trail#nomen:tinyHash(url) . Built by buildNodeAbsoluteId() (ast-lake.mjs:29)
body	linkUrl for links, repoName for others	For links: the URL target. For headings: the repo name.

[object Object] (ast-lake.mjs:29-44):

• Strips /repo/ prefix from fn → relative/path/file.mdd

• Appends :: + sanitized trail segments joined by /

• For links: appends #sanitizeName(nomen):tinyHash(url) anchor

• Result example: paper/visualization.mdd::visualization/graphviz#osage:ab12cd

[object Object] (ast-lake.mjs:46-48):

• For headings: returns last trail segment
(e.g. "installation" )

• For links: returns nomen (link label,
e.g. "click here" )

• This is the short ref stored in cache_ast_lake_nodes.ref

Why two ref tables? cache_ast_lake_nodes.ref has the short, per-node ref (last trail
segment). cache_ast_lake_nodes_ref.ref has the full, human-readable fragment
path including the file prefix. The
voting system uses the full refId from nodes_ref .

" nomen: "`cache_ast_lake_nodes_ref`"] db_tables__cache_ast_lake_flast____cache_ast_lake_nodes____cache_ast_lake_nodes_ref____cache_ast_lake_nodes_backup____db_tables__cache_ast_lake_nodes_backup_ [z: 2.375 @html: "

cache_ast_lake_nodes_backup
Role: Rollback copy. Before updating a file's
nodes in cache_ast_lake_nodes , the old rows are copied here
(ast-lake.mjs:500-514):
INSERT OR REPLACE INTO
cache_ast_lake_nodes_backup (id,
value, mt, sem, num1, num2, ref, body)
SELECT id, value, mt, sem, num1, num2,
ref, body FROM cache_ast_lake_nodes
WHERE id LIKE
'<escapedFn>:%'
Then the old rows are deleted from both cache_ast_lake_nodes and cache_ast_lake_nodes_ref before inserting the new normalized
nodes. If a sync fails, the backup can
be restored.Note: cache_ast_lake_nodes_ref does NOT have a backup table — the ref
table is rebuilt from scratch each time.

" nomen: "`cache_ast_lake_nodes_backup`"] db_tables__linetime_ [z: 4.75 @html: "

linetime
Role: Git blame line-history cache. Maps each
line in each file to its git blame
metadata (commit hash, author,
timestamp). Used by normalizeNodes() to attach symbBlame / symbMt / symbUser to each node.Schema (commented inline in ast-lake.mjs:122-131 ):

Column	Type	Description
id	INTEGER	Primary key, autoincrement
fn	TEXT NOT NULL	File name / path (matches flast.id / nodes fn prefix)
line	INTEGER NOT NULL	Line number in the file (1-based)
hash	TEXT NOT NULL	Git commit hash from blame
mt	TEXT NOT NULL	Commit author timestamp
user	TEXT	Author name or email
json	TEXT	Additional metadata as JSON blob

Query pattern (in actualizeNodes() , ast-lake.mjs:473):
SELECT line, mt, user, hash FROM
linetime WHERE fn='<filepath>' AND
hash!="" ORDER BY line
The result is used to compute symbRangeL → symbBlame for each node: all commit hashes for
lines within the node's source range.See linetime-parse-blame.jq for the processing logic that parses git blame --line-porcelain output into linetime rows.

" nomen: "`linetime`"] db_tables__repo_voting_ [z: 4.75 @html: "

repo_voting
Role: Voting event log. Records voting actions
(cast, uncast, hot, pin, weak, veto) on
fragments within campaigns. Schema from sqlCreateTableEventSource(tblName) in ast-lake.mjs:85 .

Column	Type	Description
id	INTEGER	Primary key, autoincrement
d	TEXT NOT NULL	ISO timestamp of the vote event
user	TEXT	Voter identifier (username or email)
ref	TEXT NOT NULL	Fragment refId being voted on (matches cache_ast_lake_nodes_ref.ref )
campaign	TEXT NOT NULL	Campaign scope — determines which vote UI is active (e.g. "eisenhower" , "debate" , "heading-changes" )
operation	TEXT NOT NULL	Vote operation — quadrant or action (e.g. "hot" , "pin" , "weak" , "veto" , "cast" , "uncast" )
num1	INTEGER NOT NULL	Generic numeric slot — used for vote weight or count
num2	INTEGER NOT NULL	Generic numeric slot — used for secondary count
json	TEXT	Additional payload as JSON blob

Indexes: num1 , num2 , campaign , operation , ref . Key query patterns: Dashboard voting (count per ref):
SELECT ref, operation, COUNT(*) as
count, SUM(num1) as total FROM
repo_voting WHERE campaign =
'eisenhower' GROUP BY ref,
operation
Heading-changes metadata check ( hasMetaData ):
SELECT DISTINCT ref FROM
repo_voting WHERE ref IN ('ref1',
'ref2', ...)
ref format: Must match cache_ast_lake_nodes_ref.ref — the readable refId from buildNodeAbsoluteId() . Format: relative/path/file.mdd::sanitized/trail/path . For links: ...::trail#nomen:tinyHash(url) . Cleanup on file deletion (ast-lake.mjs cleanup() at line 305):

• Does NOT delete repo_voting rows when a file is removed (votes are
preserved)

• Does DELETE from cache_ast_lake_flast , cache_ast_lake_nodes , cache_ast_lake_nodes_ref , cache_ast_lake_nodes_backup , and linetime

• Sweeps orphan refs: DELETE FROM
cache_ast_lake_nodes_ref WHERE id NOT IN
(SELECT id FROM
cache_ast_lake_nodes)

" nomen: "`repo_voting`"] }