| mdt | ||||||
|
| transclusion | |||||||||||||||||||||||||||||||
|
| ideas | ||||||||
|
| mdt — Markdown Construction Pseudo-Code Spec Pure JavaScript library for a markdown construction pseudo-code language . Markdown is the surface syntax. # ${...} headings are extructions — labeled markers that produce no output; bodies use ```javascript code blocks for eval.The library follows a compile / runner split: | ||
| ||
| All functions are pure — no mutation of inputs, no side effects, no classes, all external dependencies passed as arguments. |
| The idea | |||||
| |||||
| The # ${...} construct is called an extruction — a coined term for a labeled heading marker that produces no output; the body uses ```javascript code blocks for evaluation.The name evolved through several candidates during design: | |||||
| |||||
| Other ideas considered: hatch, vault, pocket, slot, well, lens, scope, portal, embed, injection, graft, splice, yield, emit, render. |
| Goals | ||||
|
| mdt as Markdown Every .mdd file is also valid .md . Extructions ( # ${label} ) render as ordinary visible headings. Standard markdown renderers see no special syntax — the mdt semantics are invisible to them. |
| compile compile(mdtMd, { remark }) → Runner Single entry point. Takes raw mdt markdown text and a remark instance (for .parse() ). Returns a Runner — no evaluation happens yet. import { compile } from './mdt/mdt.js' import { remark } from 'remark' const runner = compile(sourceMd, { remark }) Compile-time errors (thrown synchronously): | |
| |
| During compilation, headings whose text starts with ${ are marked as extructions. They are tracked separately but no transform is applied — the remark AST is kept as-is. |
| Runner runner(context, opts?) → Document The runner is a function. Call it with context and options to get a Document — the entry point for navigating the document tree. No processing happens until you pull from the iterable or call navigate. opts carries run-time dependencies: opts = { sanitizeName: (str) => str.toLowerCase().replace(/\W+/g, '-').replace(/^-+|-+$/g, '') } sanitizeName defaults to the function shown (lowercase, non-word chars to - , leading/trailing dashes trimmed). Callers can override. opts.loadRefBody : | |||
|
| Fragment A heading + its immediate body content. A fragment is the core unit the runner yields and the consumer navigates. { trail: "getting-started/installation", // trail-id identifying this heading heading: "# Chapter 1", // raw markdown heading string headingLevel: 1, // number of # characters body: "Some introductory text.", // canonicalized markdown body (no children) hasChildren: true, // does this fragment have expandable children? expand(): AsyncIterable<Fragment>, // yields child fragments toString(): "# Chapter 1\n\nSome introductory text." // heading + body } | |||||||
| |||||||
| AST source: currently the fragment is materialized from remark's parsed AST. In the future it could come from the ast-nodes database ( cache_ast_lake_nodes with sem = 'heading' ), where each row carries { id, mt, sem, num1, num2, ref } and nomen is derived from ref . The fragment shape is designed to be mappable to/from that schema: trail ↔ id , heading ↔ ref , headingLevel ↔ sem . |
| Extruction ## ${label} \`\`\`javascript // body code — only ```javascript blocks are evaluated \`\`\` An extruction is a # ${...} heading. When evalFn is provided, the body is evaluated as JavaScript — but only code inside is extracted. Any other markdown content in the body is ignored. Without evalFn , the extruction and its body are silently dropped. | ||||||||
| ||||||||
| The data.label (text between ${} ) is available on the heading node for future processing but has no current effect. |
| Error Handling Compile-time (thrown by compile() ): | ||||||
| ||||||
| Runtime (caught by onExtructionError callback): | ||||||
| ||||||
| When an extruction body throws during evaluation, the behavior depends on the presence of onExtructionError : | ||||||
| ||||||
| In children resolution, an errored child extruction follows the same rule — treated as transparent, its children promoted into the parent's children output.All errors include the source position ( node.position ) for debugging. |
| Open Questions |
| App Integration The MDT library is integrated into player-paper.js at the "mdt" case of the extension switch (line 876). When a .mdt file is opened: | |||||||
| |||||||
| The current integration uses the browser's dynamic import() for remark (same CDN source as mdd.mjs ). The context parameter passes the app's STATE object, with adapters mixed in for extruction evaluation. |
| Extruction Evaluation Extruction bodies can be evaluated as JavaScript at runtime when the evalFn option is passed to the runner. This enables # ${...} headings to produce dynamic content. |
| Document A Document is both an async iterable (yields root-level Fragments) and a navigation hub (find fragments by trail-id): doc【Symbol.asyncIterator】() → AsyncIterable<Fragment> doc.find(trail) → Fragment | undefined doc.children(trail) → AsyncIterable<Fragment> doc.preamble → string | |||
| |||
| A Document is stateless and re-iterable — each call to the runner produces a fresh Document, and each iteration re-derives from the compiled tree. |
| Usage — Iteration const doc = runner({ user }); for await (const section of doc) { // section.heading → "# Chapter 1" // section.body → "Some text..." // section.toString() → "# Chapter 1\n\nSome text..." for await (const child of section.expand()) { // child.heading → "## Section 1.1" // child.headingLevel → 2 // child.body → "Details..." } } |
| Usage — Trail navigation const doc = runner( { user }, { sanitizeName: (s) => s.toLowerCase().replace(/\W+/g, "-"), }, ); // Find a heading by trail-id const section = doc.find("getting-started/installation"); for await (const step of section.expand()) { // immediate children of ## Installation } // Or shortcut: get children directly for await (const step of doc.children("getting-started/installation")) { // same result } // Preamble text before the first heading console.log(doc.preamble); |
| Trail-id A trail-id is a / -separated path of sanitized heading names that uniquely identifies a heading in the document hierarchy: | ||||||||||||
| ||||||||||||
| The trail is constructed with the same stack algorithm used by getHeadingTrail in the existing codebase: | ||||||||||||
| ||||||||||||
| Extructions ( # ${label} ) are skipped by the trail algorithm — they produce no output and don't contribute to the stack. A ## Details after an extruction ## ${sidebar} at the same level gets trail "intro/details" , not "intro/sidebar/details" .Traversal stops at the first match — find() and children() return the section at the exact trail without pre-processing the entire document. Fragments past the match are not materialized. |
| Usage — Extruction evaluation with adapters When evalFn is provided, extruction bodies run as JavaScript and can produce output via the insert protocol: import { compile } from './mdt/mdt.js' import { evalBody } from './mdt/eval-body.js' import { remark } from 'remark' const md = `# ${greeting} \`\`\`javascript const name = _mdt_label return insert(\`Hello **\${name}**\`) \`\`\` # Results ## ${search mdd} \`\`\`javascript const items = await search("mdd") return insert(items.map(i => i.uri).join("\\n")) \`\`\` ## Total \`\`\`javascript return insert(String(total)) \`\`\` ` const search = async (q) => 【 { name: "file1", uri: "#/paper/file1" }, { name: "file2", uri: "#/paper/file2" }, 】 const total = 42 const runner = compile(md, { remark }) const doc = runner({ search, total }, { evalFn: evalBody }) for await (const section of doc) { console.log(section.toString()) // "${greeting}" → "<!-- greeting -->\n\nHello **world**" // "Results" → normal heading, expanded below for await (const child of section.expand()) { // "${search mdd}" → "#/paper/file1\n#/paper/file2" // "Total" → "42" } } The extruction body return insert(value) yields one or more Fragment-like objects directly into the output. Any await -able function in context is an adapter — search , total , and _mdt_label all coexist as named bindings. |
| Usage — Error recovery When an extruction body throws, onExtructionError lets you log and skip instead of crashing the iteration: const doc = runner({ search }, { evalFn: evalBody, onExtructionError: (err, headingNode) => { console.warn( \`Extruction "\${headingNode.data?.label}" failed:\`, err.message, ) }, }) for await (const section of doc) { // Sections after the failing extruction still appear } Without the callback, errors propagate to the consumer's for await loop. With the callback, the failing extruction is silently dropped and iteration continues with the next heading. The heading node gives access to the position ( headingNode.position ) for source-mapped diagnostics. |
| Usage — Adapter with The _mdt_label binding lets one adapter serve multiple extruction variants: const md = `# ${search mdd} \`\`\`javascript const items = await search(_mdt_label) return insert( items.map(i => "- " + i.uri). join("\\n")) \`\`\` # ${search js} \`\`\`javascript const items = await search(_mdt_label) return insert( items.map(i => "- " + i.name). join("\\n")) \`\`\` ` const search = async (q) => { if (q === "search mdd") return 【{ name: "readme", uri: "#/readme" }】 return 【{ name: "main.js", uri: "#/main.js" }】 } const runner = compile(md, { remark }) const doc = runner({ search }, { evalFn: evalBody }) The same search adapter is called with the label as its argument — no need to hardcode adapter names per extruction. |
| Usage — State across extructions The runner automatically injects mdtState — a plain object that persists across extruction evaluations within the same document: const md = `# ${init} \`\`\`javascript mdtState.counter = 0 mdtState.items = 【"a", "b", "c"】 \`\`\` # ${first} \`\`\`javascript mdtState.counter++ return insert( \`Item \${mdtState.counter}: \${mdtState.items【0】}\` ) \`\`\` # ${second} \`\`\`javascript mdtState.counter++ return insert( \`Item \${mdtState.counter}: \${mdtState.items【1】}\` ) \`\`\` `; const runner = compile(md, { remark }); const doc = runner({}, { evalFn: evalBody }); for await (const section of doc) { console.log(section.toString()); // "${init}" → transparent (no return/insert) // "${first}" → "Item 1: a" // "${second}" → "Item 2: b" } mdtState is just a {} — the extruction body sets properties on it, and subsequent evaluations read them back. It's automatically available in every extruction body without being added to the runner context.Callers can pre-populate mdtState by passing it in the context: const doc = runner( { mdtState: { repo: "my-repo", branch: "main" } }, { evalFn: evalBody }, ); ## ${header} \`\`\`javascript return insert( \`Repo: \${mdtState.repo}, branch: \${mdtState.branch}\` ) \`\`\` This is useful when extructions need shared initialization or cross-section communication without resorting to global variables. Why this works: mdtState is a single object stored on runnerContext . Each eval call spreads runnerContext into the function parameters, but the spread copies the reference — all evaluations share the same mdtState object. Property mutations (set/add/delete) persist; reassigning mdtState = ... would only affect the local parameter. |
| Phases The runner materializes the document in phases: | |||||||||
| |||||||||
| No phase happens until the consumer pulls. |
| expand traversal expand() walks the remark AST child heading nodes: | ||||
| ||||
| Body boundary rule: content before the first child heading belongs to the parent's body ; content between child headingNand the next heading belongs to childN's body . |
| Lazy guarantees | ||
|
| Transparency semantics Extructions are fully transparent — they produce no output and their body content is silently dropped, but non-extruction headings nested under an extruction are promoted to the nearest non-extruction ancestor's expand() output. Their trail is computed as if the extruction doesn't exist.Implementation: skipExtructionBody(startIdx, rootChildren) advances past an extruction's non-heading content but stops at any heading (a promoted child), rather than skipping the entire subtree. This is used by expandChildren , collectBodyNodes , and hasNonExtructionChild to maintain consistency. |
| 1. What is Resolved: context is state — a bag of global variables that the document can reference. With evalFn , extruction bodies can access context keys as named parameters. Without evalFn , context is accepted but unused.The runner signature stays runner(context, opts?) . With no active extructions, context is accepted but unused — a forward-looking parameter. |
| 2. Extruction label semantics Deferred. data.label is a free-form string — the text between ${} . Its semantics are intentionally undefined until extruction evaluation is designed. Currently just stored, no effect. |
| 3. When will extruction bodies activate? Resolved. Extruction bodies are evaluated as JavaScript when evalFn is provided. Only ```javascript code blocks within the body are extracted — non-javascript code blocks and other markdown content are ignored. Without evalFn , the body remains inert (silently dropped). |
| 4. Verbatim vs canonicalized body Resolved. body is canonicalized by default (re-stringified remark nodes). Source position ( node.position ) is the escape hatch for verbatim access. No default flip — canonicalized is the correct default because consumers should get consistent, predictable markdown output. If verbatim is needed, slice the original text using source offsets. |
| 5. Resolved — extructions are fully transparent with child promotion. Extructions are skipped from both output and navigation. Non-extruction headings nested under an extruction are promoted to the parent's expand() output: | ||||||
|
| evalBody mdt/eval-body.js exports the default evaluation function: evalBody(bodyText, context) → Promise<any> It uses the AsyncFunction constructor (same pattern as evalJsFilterWithContext in filter-base.js ) to evaluate the body text as JS code with the context keys available as named parameters. import { evalBody } from "./mdt/eval-body.js"; const doc = runner({ search, STATE }, { evalFn: evalBody }); Inside an extruction body, any key from the context is directly accessible: ## ${the list} \`\`\`javascript const x = await search("mdd") return insert( x.map(i => i.uri).join("\n")) \`\`\` |