| mdt | ||||||
|
| transclusion | |||||||||||||||||||||||||||||||
|
| ideas | ||||||||
|
| mdt — Markdown Construction Pseudo-Code Spec Pure JavaScript library for a markdown construction pseudo-code language . Markdown is the surface syntax. # ${...} headings are extructions — labeled markers that produce no output; bodies use ```javascript code blocks for eval.The library follows a compile / runner split: | ||
| ||
| All functions are pure — no mutation of inputs, no side effects, no classes, all external dependencies passed as arguments. |
| The idea | |||||
| |||||
| The # ${...} construct is called an extruction — a coined term for a labeled heading marker that produces no output; the body uses ```javascript code blocks for evaluation.The name evolved through several candidates during design: | |||||
| |||||
| Other ideas considered: hatch, vault, pocket, slot, well, lens, scope, portal, embed, injection, graft, splice, yield, emit, render. |
| Goals | ||||
|
| mdt as Markdown Every .mdd file is also valid .md . Extructions ( # ${label} ) render as ordinary visible headings. Standard markdown renderers see no special syntax — the mdt semantics are invisible to them. |
| compile compile(mdtMd, { remark }) → Runner Single entry point. Takes raw mdt markdown text and a remark instance (for .parse() ). Returns a Runner — no evaluation happens yet. import { compile } from './mdt/mdt.js' import { remark } from 'remark' const runner = compile(sourceMd, { remark }) Compile-time errors (thrown synchronously): | |
| |
| During compilation, headings whose text starts with ${ are marked as extructions. They are tracked separately but no transform is applied — the remark AST is kept as-is. |
| Runner runner(context, opts?) → Document The runner is a function. Call it with context and options to get a Document — the entry point for navigating the document tree. No processing happens until you pull from the iterable or call navigate. opts carries run-time dependencies: opts = { sanitizeName: (str) => str.toLowerCase().replace(/\\W+/g, '-').replace(/^-+|-+$/g, '') } sanitizeName defaults to the function shown (lowercase, non-word chars to - , leading/trailing dashes trimmed). Callers can override. opts.loadRefBody : | |||
|
| Document A Document is both an async iterable (yields root-level Fragments) and a navigation hub (find fragments by trail-id): doc【Symbol.asyncIterator】() → AsyncIterable<Fragment> doc.find(trail) → Fragment | undefined doc.children(trail) → AsyncIterable<Fragment> doc.preamble → string | |||
| |||
| A Document is stateless and re-iterable — each call to the runner produces a fresh Document, and each iteration re-derives from the compiled tree. |
| Usage — Iteration const doc = runner({ user }); for await (const section of doc) { // section.heading → "# Chapter 1" // section.body → "Some text..." // section.toString() → "# Chapter 1\\n\\nSome text..." for await (const child of section.expand()) { // child.heading → "## Section 1.1" // child.headingLevel → 2 // child.body → "Details..." } } |
| Usage — Trail navigation const doc = runner( { user }, { sanitizeName: (s) => s.toLowerCase().replace(/\\W+/g, "-"), }, ); // Find a heading by trail-id const section = doc.find("getting-started/installation"); for await (const step of section.expand()) { // immediate children of ## Installation } // Or shortcut: get children directly for await (const step of doc.children("getting-started/installation")) { // same result } // Preamble text before the first heading console.log(doc.preamble); |
| Trail-id A trail-id is a / -separated path of sanitized heading names that uniquely identifies a heading in the document hierarchy: | ||||||||||||
| ||||||||||||
| The trail is constructed with the same stack algorithm used by getHeadingTrail in the existing codebase: | ||||||||||||
| ||||||||||||
| Extructions ( # ${label} ) are skipped by the trail algorithm — they produce no output and don't contribute to the stack. A ## Details after an extruction ## ${sidebar} at the same level gets trail "intro/details" , not "intro/sidebar/details" .Traversal stops at the first match — find() and children() return the section at the exact trail without pre-processing the entire document. Fragments past the match are not materialized. |
| Usage — Extruction evaluation with adapters When evalFn is provided, extruction bodies run as JavaScript and can produce output via the insert protocol: import { compile } from './mdt/mdt.js' import { evalBody } from './mdt/eval-body.js' import { remark } from 'remark' const md = `# ${greeting} \\`\\`\\`javascript const name = _mdt_label return insert(\\`Hello **\\${name}**\\`) \\`\\`\\` # Results ## ${search mdd} \\`\\`\\`javascript const items = await search("mdd") return insert(items.map(i => i.uri).join("\\\\n")) \\`\\`\\` ## Total \\`\\`\\`javascript return insert(String(total)) \\`\\`\\` ` const search = async (q) => 【 { name: "file1", uri: "#/paper/file1" }, { name: "file2", uri: "#/paper/file2" }, 】 const total = 42 const runner = compile(md, { remark }) const doc = runner({ search, total }, { evalFn: evalBody }) for await (const section of doc) { console.log(section.toString()) // "${greeting}" → "<!-- greeting -->\\n\\nHello **world**" // "Results" → normal heading, expanded below for await (const child of section.expand()) { // "${search mdd}" → "#/paper/file1\\n#/paper/file2" // "Total" → "42" } } The extruction body return insert(value) yields one or more Fragment-like objects directly into the output. Any await -able function in context is an adapter — search , total , and _mdt_label all coexist as named bindings. |
| Usage — Error recovery When an extruction body throws, onExtructionError lets you log and skip instead of crashing the iteration: const doc = runner({ search }, { evalFn: evalBody, onExtructionError: (err, headingNode) => { console.warn( \\`Extruction "\\${headingNode.data?.label}" failed:\\`, err.message, ) }, }) for await (const section of doc) { // Sections after the failing extruction still appear } Without the callback, errors propagate to the consumer's for await loop. With the callback, the failing extruction is silently dropped and iteration continues with the next heading. The heading node gives access to the position ( headingNode.position ) for source-mapped diagnostics. |
| Usage — Adapter with The _mdt_label binding lets one adapter serve multiple extruction variants: const md = `# ${search mdd} \\`\\`\\`javascript const items = await search(_mdt_label) return insert( items.map(i => "- " + i.uri). join("\\\\n")) \\`\\`\\` # ${search js} \\`\\`\\`javascript const items = await search(_mdt_label) return insert( items.map(i => "- " + i.name). join("\\\\n")) \\`\\`\\` ` const search = async (q) => { if (q === "search mdd") return 【{ name: "readme", uri: "#/readme" }】 return 【{ name: "main.js", uri: "#/main.js" }】 } const runner = compile(md, { remark }) const doc = runner({ search }, { evalFn: evalBody }) The same search adapter is called with the label as its argument — no need to hardcode adapter names per extruction. |
| Usage — State across extructions The runner automatically injects mdtState — a plain object that persists across extruction evaluations within the same document: const md = `# ${init} \\`\\`\\`javascript mdtState.counter = 0 mdtState.items = 【"a", "b", "c"】 \\`\\`\\` # ${first} \\`\\`\\`javascript mdtState.counter++ return insert( \\`Item \\${mdtState.counter}: \\${mdtState.items【0】}\\` ) \\`\\`\\` # ${second} \\`\\`\\`javascript mdtState.counter++ return insert( \\`Item \\${mdtState.counter}: \\${mdtState.items【1】}\\` ) \\`\\`\\` `; const runner = compile(md, { remark }); const doc = runner({}, { evalFn: evalBody }); for await (const section of doc) { console.log(section.toString()); // "${init}" → transparent (no return/insert) // "${first}" → "Item 1: a" // "${second}" → "Item 2: b" } mdtState is just a {} — the extruction body sets properties on it, and subsequent evaluations read them back. It's automatically available in every extruction body without being added to the runner context.Callers can pre-populate mdtState by passing it in the context: const doc = runner( { mdtState: { repo: "my-repo", branch: "main" } }, { evalFn: evalBody }, ); ## ${header} \\`\\`\\`javascript return insert( \\`Repo: \\${mdtState.repo}, branch: \\${mdtState.branch}\\` ) \\`\\`\\` This is useful when extructions need shared initialization or cross-section communication without resorting to global variables. Why this works: mdtState is a single object stored on runnerContext . Each eval call spreads runnerContext into the function parameters, but the spread copies the reference — all evaluations share the same mdtState object. Property mutations (set/add/delete) persist; reassigning mdtState = ... would only affect the local parameter. |
| Phases The runner materializes the document in phases: | |||||||||
| |||||||||
| No phase happens until the consumer pulls. |
| Fragment A heading + its immediate body content. A fragment is the core unit the runner yields and the consumer navigates. { trail: "getting-started/installation", // trail-id identifying this heading heading: "# Chapter 1", // raw markdown heading string headingLevel: 1, // number of # characters body: "Some introductory text.", // canonicalized markdown body (no children) hasChildren: true, // does this fragment have expandable children? expand(): AsyncIterable<Fragment>, // yields child fragments toString(): "# Chapter 1\\n\\nSome introductory text." // heading + body } | |||||||
| |||||||
| AST source: currently the fragment is materialized from remark's parsed AST. In the future it could come from the ast-nodes database ( cache_ast_lake_nodes with sem = 'heading' ), where each row carries { id, mt, sem, num1, num2, ref } and nomen is derived from ref . The fragment shape is designed to be mappable to/from that schema: trail ↔ id , heading ↔ ref , headingLevel ↔ sem . |
| expand traversal expand() walks the remark AST child heading nodes: | ||||
| ||||
| Body boundary rule: content before the first child heading belongs to the parent's body ; content between child headingNand the next heading belongs to childN's body . |
| Lazy guarantees | ||
|
| Extruction ## ${label} \\`\\`\\`javascript // body code — only ```javascript blocks are evaluated \\`\\`\\` An extruction is a # ${...} heading. When evalFn is provided, the body is evaluated as JavaScript — but only code inside is extracted. Any other markdown content in the body is ignored. Without evalFn , the extruction and its body are silently dropped. | ||||||||
| ||||||||
| The data.label (text between ${} ) is available on the heading node for future processing but has no current effect. |
| Transparency semantics Extructions are fully transparent — they produce no output and their body content is silently dropped, but non-extruction headings nested under an extruction are promoted to the nearest non-extruction ancestor's expand() output. Their trail is computed as if the extruction doesn't exist.Implementation: skipExtructionBody(startIdx, rootChildren) advances past an extruction's non-heading content but stops at any heading (a promoted child), rather than skipping the entire subtree. This is used by expandChildren , collectBodyNodes , and hasNonExtructionChild to maintain consistency. |
| Error Handling Compile-time (thrown by compile() ): | ||||||
| ||||||
| Runtime (caught by onExtructionError callback): | ||||||
| ||||||
| When an extruction body throws during evaluation, the behavior depends on the presence of onExtructionError : | ||||||
| ||||||
| In children resolution, an errored child extruction follows the same rule — treated as transparent, its children promoted into the parent's children output.All errors include the source position ( node.position ) for debugging. |
| Open Questions |
| 1. What is Resolved: context is state — a bag of global variables that the document can reference. With evalFn , extruction bodies can access context keys as named parameters. Without evalFn , context is accepted but unused.The runner signature stays runner(context, opts?) . With no active extructions, context is accepted but unused — a forward-looking parameter. |
| 2. Extruction label semantics Deferred. data.label is a free-form string — the text between ${} . Its semantics are intentionally undefined until extruction evaluation is designed. Currently just stored, no effect. |
| 3. When will extruction bodies activate? Resolved. Extruction bodies are evaluated as JavaScript when evalFn is provided. Only ```javascript code blocks within the body are extracted — non-javascript code blocks and other markdown content are ignored. Without evalFn , the body remains inert (silently dropped). |
| 4. Verbatim vs canonicalized body Resolved. body is canonicalized by default (re-stringified remark nodes). Source position ( node.position ) is the escape hatch for verbatim access. No default flip — canonicalized is the correct default because consumers should get consistent, predictable markdown output. If verbatim is needed, slice the original text using source offsets. |
| 5. Resolved — extructions are fully transparent with child promotion. Extructions are skipped from both output and navigation. Non-extruction headings nested under an extruction are promoted to the parent's expand() output: | ||||||
|
| App Integration The MDT library is integrated into player-paper.js at the "mdt" case of the extension switch (line 876). When a .mdt file is opened: | |||||||
| |||||||
| The current integration uses the browser's dynamic import() for remark (same CDN source as mdd.mjs ). The context parameter passes the app's STATE object, with adapters mixed in for extruction evaluation. |
| Extruction Evaluation Extruction bodies can be evaluated as JavaScript at runtime when the evalFn option is passed to the runner. This enables # ${...} headings to produce dynamic content. |
| evalBody mdt/eval-body.js exports the default evaluation function: evalBody(bodyText, context) → Promise<any> It uses the AsyncFunction constructor (same pattern as evalJsFilterWithContext in filter-base.js ) to evaluate the body text as JS code with the context keys available as named parameters. import { evalBody } from "./mdt/eval-body.js"; const doc = runner({ search, STATE }, { evalFn: evalBody }); Inside an extruction body, any key from the context is directly accessible: ## ${the list} \\`\\`\\`javascript const x = await search("mdd") return insert( x.map(i => i.uri).join("\\n")) \\`\\`\\` |
| Extruction return value — When evalFn is provided, the extruction body has access to auto-injected helpers and data (like _mdt_label , mdtState , and log ): | |||
|
| insertchildren Takes one or more Fragment-like objects and yields each as-is into the output stream. No wrapping, no heading comment — the caller has full control: ## ${search results} \\`\\`\\`javascript const items = await search("mdd") return insert(items.map(r => ({ trail: _mdt_label + "/" + r.id, heading: "### " + r.name, headingLevel: 3, body: r.description, hasChildren: false, expand: () => (async function* {})(), toString: () => "### " + r.name + "\\n\\n" + r.description, }))) \\`\\`\\` Pass a single fragment or an array — insert() handles both: return insert(singleFrag); return insert(【fragA, fragB, fragC】); |
| injecttext Takes a string and yields a single raw-body Fragment with no heading, no trail, no wrapper: ## ${notice} \\`\\`\\`javascript return inject("> **Note:** generated from live data.") \\`\\`\\` The Fragment has heading: "" , headingLevel: 0 , trail: "" , and toString() returns the raw body. |
| children The children variable holds the resolved output of the extruction's child subtree — all headings between this extruction and the next heading at the same or higher depth. Non-heading body text after the extruction heading is not included (that's the bodyText passed to evalFn ).Resolution is recursive — children is computed by walking the child tree and processing each node: | ||||||||||||
| ||||||||||||
| This means extructions at any depth are fully evaluated — a ##### ${...} deep under a regular #### heading will still produce its resolved output.A common pattern is to pipe children through insert() : ## ${list of todos} \\`\\`\\`javascript return 【inject("> Generated list:\\n\\n"), insert(children)】 \\`\\`\\` children is an empty string "" when: | ||||||||||||
| ||||||||||||
| Non-extruction headings are included as original markdown (source positions preserve formatting). Extruction headings themselves never appear in the output — they're transparent, only their resolved content is included. |
| insertRefsAsSubtreeitems, opts? Turn an array of fragment refs (typically await search(...) results) into child-depth heading Fragments with lazy body-fetch : ## ${search fragments; do} \\`\\`\\`javascript return 【insertRefsAsSubtree(await search(_mdt_label))】 \\`\\`\\` Each item becomes ONE Fragment one level below the extruction ( extruction.depth + 1 ), so the results nest as children of the current level. The Fragment's body is empty at yield-time; the fetch happens only inside its expand() — i.e. only when the render pipeline walks into that subtree. Depth is clamped at 6 (markdown's maximum heading level). ## insertRefsAsSubtree ← depth 2, visible parent ### ${insertRefsAsSubtree} ← depth 3, extruction (filtered from output) #### auth ← depth 4, one Fragment per item ##### …transcluded body… ← depth 5+, from loadRefBody This is the only verb whose heading is real markdown — every other verb emits an HTML-comment heading, so its depth is invisible. Item contract (minimum): | ||||||||||
| ||||||||||
| Items missing name / ref , or without both fn and ( trail or num1 ), are skipped with console.warn . If every item is skipped, a visible\nblockquote is emitted explaining why — the verb never fails silently.The common cause is feeding it the wrong search source: files results ( {name, uri, fn, type:"file"} ) carry no trail / num1 , so there is no subtree to resolve. Use a fragments query, whose items carry nomen / trail / num1 / fn . opts: | ||||||||||
| ||||||||||
| Runner opt required: runner(ctx, { evalFn, loadRefBody }) . If loadRefBody is not provided, each Fragment renders heading-only. |
| insertNljsoncollection, opts? Serialize a collection as newline-delimited JSON inside an ```nljson fence — one JSON object per line: ## ${rows} \\`\\`\\`javascript return 【insertNljson(【{ a: 1 }, { b: 2 }】)】 \\`\\`\\` {"a":1} {"b":2} A single non-array value is wrapped. This is a raw passthrough — values are serialized as given, so nested objects and arrays survive. That makes it unsuitable for feeding a table directly: insertNljson(await search(...)) emits trail arrays, and Tabulator's html formatter throws Formatter has returned a type of object . Use insertRefsAsNljson for table-bound ref data, or pick scalar fields yourself. |
| insertRefsAsListitems, opts? Render an array of refs as a markdown bullet list — a flat alternative to insertRefsAsSubtree with no lazy fetch: ## ${links} \\`\\`\\`javascript return 【insertRefsAsList(await search(_mdt_label))】 \\`\\`\\` - 【auth】(#/paper/todo.mdd/auth) {{"platba":{"suma":42}}} - 【login】(#/paper/a.mdd) - plain Labels resolve with the same 4-step rule as insertRefsAsSubtree . An item with uri becomes a markdown link; without one it stays plain text. Items with no resolvable label are skipped with console.warn . | ||||||||
|
| insertRefsAsNljsonitems, optsOrFn? Render an array of refs as nljson rows — reuses insertNljson 's fence, but builds each row from the ref and guarantees table-safe scalar cells : ## ${table} \\`\\`\\`javascript return 【insertRefsAsNljson(await search(_mdt_label))】 \\`\\`\\` {"link":"<a HREF=\\"#/paper/todo.mdd/auth\\">auth</a>","data":"{\\"platba\\":{\\"suma\\":42}}"} link is an HTML anchor (not a markdown link) because nljson usually feeds a table — the table needs columnDefaults: { formatter: 'html' } to render it. The uri is attribute-escaped ( & → & , " → " ).Every row value is flattened before output: any object or array becomes a JSON string. This is what keeps Tabulator's html formatter from throwing on trail arrays or nested data . Second argument — object or function. A bare function is shorthand for { extend: fn } : \\`\\`\\`javascript return 【 insertRefsAsNljson(mdtState.items, function addAdditionalProperties(i) { const data = i.data ? JSON.parse(i.data) : undefined return { suma: data?.platba?.suma, data: JSON.stringify(data), } }), 】 \\`\\`\\` extend(item, row) receives the raw item first (so item.data is the untouched string) plus the base row, and its returned props are merged over the auto-built ones — the example above replaces the auto data . Keys whose value is undefined are dropped from the row rather than emitted as null , so ragged rows are normal. | ||||||||||||
|
| buildUrlcontent, mimeType? Not a command — a plain helper returning a base64 data URI via btoa() . Defaults to text/plain : \\`\\`\\`javascript return 【inject(`【download】(${buildUrl(JSON.stringify(rows), "application/json")})`)】 \\`\\`\\` |
| Mixed output Return an array of calls to produce multiple items in sequence: ## ${mixed} \\`\\`\\`javascript const items = await search("mdd") const cards = items.map(r => ({ /* fragment shape */ })) return 【inject("> Preview below:\\n\\n"), insert(cards)】 \\`\\`\\` Each item in the array is a command object produced by any of the verbs — insert() , inject() , insertNljson() , insertRefsAsList() , insertRefsAsNljson() , or insertRefsAsSubtree() — mixable in any order. |
| Return nothing | ||
|
| State still via The mdtState object is mutated directly through property assignment, not through helpers: ## ${init} \\`\\`\\`javascript mdtState.counter = 0 \\`\\`\\` ## ${count} \\`\\`\\`javascript mdtState.counter++ return inject(String(mdtState.counter)) \\`\\`\\` |
| Adapters — Adapters are not commands. They are async functions injected into the eval context by createAdapters() ( adapters.js ) and used toobtainitems, which the insert* verbs then render. All three are await -ed. | ||||||||||||
| ||||||||||||
| searchVotes queries the v_voting_campaign view. repo defaults to STATE.repoName . campaign accepts '*' (all campaigns), a prefix, or an array of prefixes — matching is by prefix, not exact name : | ||||||||||||
| ||||||||||||
| This mirrors campaignPrefix in tagCloudByVotingsFromView() . A consequence worth remembering: an exact campaign name matches only if something sits below it, so pass the parent prefix rather than the full campaign.Rows come back as objects: repo campaign nomen aliasRef id num1 voteCount maxCount rn score is not selected — the deployed view may have been generated with withScore: false , and its LN() also needs a SQLite built with SQLITE_ENABLE_MATH_FUNCTIONS . It is computed locally instead, from voteCount / maxCount , and added to each row: 1 + Math.round(Math.log1p((voteCount / maxCount) * 100)); Verified identical to the view's SQL expression across the real vote rows. votesAsRefs is a pure conversion — vote rows carry aliasRef , id and num1 , which is everything a ref item needs. It builds uri the same way a fragments search does ( #/paper/${aliasRef} , falling back to legacyPaperUrl ), sets nomen for the label, and derives fn by stripping the :NNNN node-seq suffix off id so buildRefId() resolves. Vote data ( campaign , voteCount , maxCount , score , rn ) rides along, so insertRefsAsNljson can surface counts without a second query.It is async despite doing no I/O today — the signature is the contract, so a later version can enrich from the DB without breaking callers. Example — list voted fragments: ## ${init} \\`\\`\\`javascript mdtState.queryVotes = { campaign: '*' } mdtState.votes = await searchVotes(mdtState.queryVotes) \\`\\`\\` ### ${list} \\`\\`\\`javascript return 【 insertRefsAsList(await votesAsRefs(mdtState.votes)), 】 \\`\\`\\` Both are wired in adapters.js exactly as search is, so anything that builds a runner context gets them for free. |
| Command contract — all verbs | ||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||
| buildUrl(content, mimeType?) is a helper, not a command — it returns a data: URI string for use inside any of the above. [object Object] Every other verb yields exactly one leaf Fragment ( hasChildren: false , inert expand() ) whose heading is an invisible HTML comment. insertRefsAsSubtree fans out to one Fragmentper item, each with a real visible heading, hasChildren: true , and a real expand() that calls loadRefBody — so the content fetch is deferred until the render pipeline walks into that subtree. It also dedupes colliding trails with -2 / -3 suffixes. [object Object] (conversion-tree provenance) rides on insert , insertNljson , insertRefsAsList , and insertRefsAsNljson . inject never carries it; insertRefsAsSubtree derives sourceFragment itself from buildRefId(item) . Two dispatch sites handle these: processExtructionResult yields real Fragments, while the array walker in resolveChildTree stringifies commands into a parent's children text. insertRefsAsSubtree is deliberately absent from the second — nested inside a children resolution there is no lazy expansion in a flat string context, so it contributes nothing there.Under the hood every helper produces a command object ( { insert: 【...】 } / { inject: "..." } / …) that the runner processes. The extruction must return an array 【cmd1, cmd2, ...】 to yield fragments. A bare non-array object yields nothing — only undefined or an array is valid. Example — injecting a preamble: ## ${notice} \\`\\`\\`javascript return inject("> **Note:** this document is generated from live data.") \\`\\`\\` This produces a Fragment whose toString() is just the blockquote — no heading comment wrapping it. The consumer sees clean markdown without synthetic HTML comments. Implementation notes: | ||||||||||||||||||||||||||||
|
| hasChildren & extruction evaluation When evalFn is active, any extruction child heading causes the parent's hasChildren to be true , since the extruction might produce an insert . This ensures rebuildMd() -style collectors expand to find evaluated content. Extructions that evaluate to undefined yield no children (the expansion returns empty immediately). |
| Error behavior | |||
| |||
| The snapshot test "syntax error in extruction body" documents the current behavior without evalFn (silently dropped). When evalFn is added to that test, it should throw. |
| buildInsertFragment serialization buildInsertFragment(insertValue, ...) handles the { insert } value: | |||
| |||
| This prevents 【object Object】 output when extruction bodies return arrays or objects (e.g. search results). |
| Probes Two console.log probes are placed at the extruction result handling points: | ||
| ||
| These are the frontend equivalent of the backend probe pattern ( PROXY.remoteState?.log({ label }) ). The MDT library is a pure frontend module without PROXY access, so console.log is used directly. |
| Search Adapter The MDT library provides a search adapter that wraps the app's glassSearchRun() with proper async completion detection, emitting per-source events and a final allCompletedDone event. |
| glassSearchRunAsync mdt/glass-search-run.js exports an async wrapper around the app's glassSearchRun() : glassSearchRunAsync(queryString, ssss, state, STATE, route, prevHashRoute, proxy) → { onSource(fn), onComplete(fn), then(resolve, reject) } The wrapper: | |||||
| |||||
| Returns a thenable object — supports both event-based and Promise-based usage: // Event-based const search = glassSearchRunAsync( query, ssss, state, STATE, route, prevHashRoute, proxy, ); search.onSource((items) => console.log("received", items.length, "results")); search.onComplete((allResults) => console.log("all done", allResults.length)); // Promise-based const allResults = await glassSearchRunAsync( query, ssss, state, STATE, route, prevHashRoute, proxy, ); |
| search adapter mdt/search-adapter.js exports a thin convenience function: search(query, ssss, state, STATE, route, prevHashRoute, proxy) → thenable Returns empty results for empty/whitespace queries. Otherwise delegates to glassSearchRunAsync . |
| Completion detection The "tiny issue" with glassSearchRun() is that it returns synchronously but kicks off async SQLite fragment searches (debounced at 5ms). The result list ( resultList from glass-search.js ) is populated incrementally: | |||
| |||
| The wrapper intercepts proxy.addResultItems to know when async results arrive. A 50ms batch window absorbs cascaded calls, then onComplete fires with the full, deduplicated result list. |
| Adapter Pattern Adapters are functions injected into the runner context that extruction bodies can call as if they were local variables. The mechanism is simple: | |||
| |||
runner(context, { evalFn: evalBody }) // ^— keys here become parameter names in extruction bodies |
| How it works Given this setup: const doc = runner( { search: mySearchFn, getUser: myGetUserFn }, { evalFn: evalBody }, ); An extruction body like: ## ${find stuff} \\`\\`\\`javascript const results = await search("mdd") return insert( results.map(r => r.name).join("\\n")) \\`\\`\\` ...is compiled to something like AsyncFunction(search, getUser, bodyText) , so search and getUser are directly accessible in the body without any import. |
| Naming rules | ||||
|
| Return protocol Adapters can return anything — there's no adapter-specific protocol. The extruction body is responsible for handling the return value and deciding what to do with it via the insert protocol: | |||
| |||
| This means adapters can return raw data (arrays, objects, strings) and the extruction body formats it into markdown. |
| Adapter conventions | |||
| |||
## ${search mdd} \\`\\`\\`javascript return insert( await search(_mdt_label)) \\`\\`\\` This is how the same adapter can be driven by different extruction labels without hardcoding the query string. |
| Example adapters |
| 1. Simple lookup sync const repoInfo = { ssss: { stars: 42, description: "The ssss project" }, mdt: { stars: 12, description: "Markdown construction pseudo-code" }, }; const doc = runner({ repoInfo }, { evalFn: evalBody }); ## ${repo info} \\`\\`\\`javascript const r = repoInfo【"ssss"】 return insert( \\`**${r.stars}** stars — ${r.description}\\` ) \\`\\`\\` |
| 2. Search adapter Already documented in#search-adapter⸨#search-adapter⸩. The pattern: import { search } from "./mdt/search-adapter.js"; const doc = runner( { search: (q) => search(q, ssss, state, STATE, route, prevHashRoute, proxy) }, { evalFn: evalBody }, ); ## ${results} \\`\\`\\`javascript const items = await search("mdd") return insert( items.map(i => \\`- 【\\${i.name}】(${i.uri})\\`).join("\\n")) \\`\\`\\` The key insight: the adapter wraps the app's async search with completion detection, but the extruction body just sees a function it can await . |
| 3. HTTP fetch const fetchJson = async (url) => { const res = await fetch(url); if (!res.ok) throw new Error(`fetch ${url}: ${res.status}`); return res.json(); }; const doc = runner( { fetchJson }, { evalFn: evalBody, onExtructionError: handleError }, ); ## ${github stats} \\`\\`\\`javascript const data = await fetchJson("https:/repo") return insert( \\`\\${data.stargazers_count} stars, \\${data.forks_count} forks\\` ) \\`\\`\\` The adapter is a thin wrapper around fetch() with error handling. The extruction body destructures the response and formats it as markdown. |
| 4. Database query const queryDb = async (sql) => { const db = await getDatabase(); return db.exec(sql); }; const doc = runner({ queryDb }, { evalFn: evalBody }); ## ${active users} \\`\\`\\`javascript const rows = await queryDb("SELECT name, email FROM users WHERE active = 1") return insert( rows.map(r => \\`- \\${r.name} <\\${r.email}>\\`).join("\\n")) \\`\\`\\` |
| 5. State access When the runner context includes the app's state object, extructions can read from it directly: const doc = runner({ STATE, currentUser: "bebo" }, { evalFn: evalBody }); ## ${welcome} \\`\\`\\`javascript return insert( \\`Hello **\\${currentUser}**, you have \\${STATE.notifications.length} notifications\\` ) \\`\\`\\` This is how the app passes its reactive state into extruction bodies. |
| 6. Composition — multiple adapters Adapters compose naturally since they're just JS functions: const doc = runner({ repoInfo, fetchJson, currentUser }, { evalFn: evalBody }); ## ${dashboard} \\`\\`\\`javascript const user = currentUser const repos = await fetchJson(\\`https:/repos\\`) const summary = repos.map(r => \\`- \\${r.name}: \\${repoInfo【r.name】?.description || "unknown"}\\`).join("\\n") return insert( \\`### \\${user}'s repos\\n\\n\\${summary}\\` ) \\`\\`\\` Here repoInfo is a sync lookup, fetchJson is async, and currentUser is a plain string — all coexist as named parameters. |
| 7. Using The label (text between ${} ) is injected as _mdt_label automatically. This lets a single adapter serve multiple extruction variants: ## ${fetch todos} \\`\\`\\`javascript return insert( await fetchJson("/api/todos")) \\`\\`\\` ## ${fetch users} \\`\\`\\`javascript return insert( await fetchJson("/api/users") ) \\`\\`\\` Without hardcoding the path in each body — although in this case you'd still need to map the label to the path. A more practical use: ## ${search mdd} \\`\\`\\`javascript const items = await search(_mdt_label) return insert( items.map(i => i.uri).join("\\n")) \\`\\`\\` ## ${search js} \\`\\`\\`javascript const items = await search(_mdt_label) return insert( items.map(i => i.name).join("\\n")) \\`\\`\\` The same search adapter is called with different labels. |
| Key constraints | ||||||||||||
|
| 6. E2E tests Test the full player-paper.js integration: .mdt file fetch → compile → run with evalBody + adapters → rebuild clean md → render. | |
| Conversion tree — transclusion provenance When an mdt document transcludes content from source fragments (via extructions), the produced fragments have virtual trail positions in the generated document. To resolve these back to the real source fragments, the mdt runner produces a conversion tree alongside the generated text. |
| sourceFragment field Each Fragment now carries an optional sourceFragment : { trail: "a/x", heading: "## <!-- ... -->", body: "hello", sourceFragment: null | { fn: "paper/real.mdd", refId: "paper/real.mdd::real/heading" } } | |||
|
| Extruction protocol — insert extended insert() accepts an optional second argument { source } : // without source (existing behavior) return 【insert(children)】; // with source tag (new) return 【 insert(children, { source: { fn: "paper/real.mdd", refId: "paper/real.mdd::real/heading" }, }), 】; The source object flows from the command → buildInsertFragment → Fragment → consumer's conversion tree. |
| Consumer collects conversion tree The consumer iterates fragments and builds a Map<trail, sourceFragment> : const conversionTree = new Map(); for await (const frag of doc) { if (frag.sourceFragment) { conversionTree.set(frag.trail, frag.sourceFragment); } } |
| Rendering uses conversion tree renderFractalCollageOfPaper accepts { conversionTree } in opts. In renderAfterEffect , the rendering trail (semicolons) is converted to / -separated key and looked up in the tree. If found, the source refId is used as data-fragmentRef instead of the computed virtual refId. |
| Trail format conversion | |||||||||||||||
|
| Changed files | ||||
| ||||