[{"id":"/root","type":"ROOT","loc":{"start":0,"end":6014,"column":{"s":0,"e":24},"line":{"s":0,"e":281}},"dim":[""],"code":null},{"id":"/root/children/0","type":"paragraph","loc":{"start":0,"end":44,"line":{"s":1,"e":1,"code":[";{ engine: dot rankdir:LR clusterize: true }"]},"column":{"s":0,"e":44}},"dim":["","paragraph.0"],"code":";{ engine: dot rankdir:LR clusterize: true }"},{"id":"/root/children/0/children/0","type":"text","loc":{"start":0,"end":44,"line":{"s":1,"e":1,"code":[";{ engine: dot rankdir:LR clusterize: true }"]},"column":{"s":0,"e":44}},"dim":["","paragraph.0","text.0"],"code":";{ engine: dot rankdir:LR clusterize: true }"},{"id":"/root/children/1","type":"heading","loc":{"start":46,"end":58,"line":{"s":3,"e":3,"code":["# LLM Tagger"]},"column":{"s":0,"e":12}},"dim":["","heading.1"],"code":"# LLM Tagger","symbName":"heading","symbRange":[60,118],"symbRangeL":[3,6],"outerCode":"\nDocument clustering and analysis using embeddings and LLM.","outerHtml":"\n<p>Document clustering and analysis using embeddings and LLM.</p>"},{"id":"/root/children/1/children/0","type":"text","loc":{"start":48,"end":58,"line":{"s":3,"e":3,"code":["# LLM Tagger"]},"column":{"s":2,"e":12}},"dim":["","heading.1","text.0"],"code":"LLM Tagger"},{"id":"/root/children/2","type":"paragraph","loc":{"start":60,"end":118,"line":{"s":5,"e":5,"code":["Document clustering and analysis using embeddings and LLM."]},"column":{"s":0,"e":58}},"dim":["","paragraph.2"],"code":"Document clustering and analysis using embeddings and LLM."},{"id":"/root/children/2/children/0","type":"text","loc":{"start":60,"end":118,"line":{"s":5,"e":5,"code":["Document clustering and analysis using embeddings and LLM."]},"column":{"s":0,"e":58}},"dim":["","paragraph.2","text.0"],"code":"Document clustering and analysis using embeddings and LLM."},{"id":"/root/children/3","type":"heading","loc":{"start":120,"end":134,"line":{"s":7,"e":7,"code":["## Quick Start"]},"column":{"s":0,"e":14}},"dim":["","heading.3"],"code":"## Quick Start","symbName":"heading","symbRange":[136,644],"symbRangeL":[7,27],"outerCode":"\n```bash\ncd docker/llm-tagger\n\n# step 1: cache embeddings (always first)\n./llm-tagger.sh init total.ndjson\n\n# step 2: cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100\n\n# optional: hierarchical tagging\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n\n# optional: analysis tools\n./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```","outerHtml":"\n<p>```bash\ncd docker/llm-tagger</p>\n\n<h1>step 1: cache embeddings (always first)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>step 2: cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100</p>\n\n<h1>optional: hierarchical tagging</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3</p>\n\n<h1>optional: analysis tools</h1><p>./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```</p>"},{"id":"/root/children/3/children/0","type":"text","loc":{"start":123,"end":134,"line":{"s":7,"e":7,"code":["## Quick Start"]},"column":{"s":3,"e":14}},"dim":["","heading.3","text.0"],"code":"Quick Start"},{"id":"/root/children/4","type":"code","loc":{"start":136,"end":644,"line":{"s":9,"e":26,"code":["```bash","cd docker/llm-tagger","","# step 1: cache embeddings (always first)","./llm-tagger.sh init total.ndjson","","# step 2: cluster and tag","./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100","","# optional: hierarchical tagging","./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3","","# optional: analysis tools","./llm-tagger.sh search data.ndjson 'query' 10","./llm-tagger.sh similar data.ndjson 0.95","./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'","./llm-tagger.sh visualize tagged.ndjson output.html","```"]},"column":{"s":0,"e":3}},"dim":["","code.4"],"code":"```bash\ncd docker/llm-tagger\n\n# step 1: cache embeddings (always first)\n./llm-tagger.sh init total.ndjson\n\n# step 2: cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100\n\n# optional: hierarchical tagging\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n\n# optional: analysis tools\n./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```","symbName":"code","symbRange":[646,1270],"symbRangeL":[null,45],"outerCode":";{ engine: dot rankdir:LR clusterize: true }\n\n# LLM Tagger\n\nDocument clustering and analysis using embeddings and LLM.\n\n## Quick Start\n\n```bash\ncd docker/llm-tagger\n\n# step 1: cache embeddings (always first)\n./llm-tagger.sh init total.ndjson\n\n# step 2: cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100\n\n# optional: hierarchical tagging\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n\n# optional: analysis tools\n./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```\n\n## All Commands\n\n| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |\n\n## Workflows\n\n### Workflow 1: Single-Level Tagging\n\nOne level of clusters with tags.","outerHtml":"<p>;{ engine: dot rankdir:LR clusterize: true }</p>\n\n<h1>LLM Tagger</h1>\n\n<p>Document clustering and analysis using embeddings and LLM.</p>\n\n<h2>Quick Start</h2>\n\n<p>```bash\ncd docker/llm-tagger</p>\n\n<h1>step 1: cache embeddings (always first)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>step 2: cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100</p>\n\n<h1>optional: hierarchical tagging</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3</p>\n\n<h1>optional: analysis tools</h1><p>./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```</p>\n\n<h2>All Commands</h2>\n\n<p>| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |</p>\n\n<h2>Workflows</h2>\n\n<h3>Workflow 1: Single-Level Tagging</h3>\n\n<p>One level of clusters with tags.</p>"},{"id":"/root/children/5","type":"heading","loc":{"start":646,"end":661,"line":{"s":28,"e":28,"code":["## All Commands"]},"column":{"s":0,"e":15}},"dim":["","heading.5"],"code":"## All Commands","symbName":"heading","symbRange":[663,1184],"symbRangeL":[28,39],"outerCode":"\n| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |","outerHtml":"\n<p>| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |</p>"},{"id":"/root/children/5/children/0","type":"text","loc":{"start":649,"end":661,"line":{"s":28,"e":28,"code":["## All Commands"]},"column":{"s":3,"e":15}},"dim":["","heading.5","text.0"],"code":"All Commands"},{"id":"/root/children/6","type":"paragraph","loc":{"start":663,"end":1184,"line":{"s":30,"e":38,"code":["| Command         | Purpose                             |","| --------------- | ----------------------------------- |","| `init`          | Cache embeddings from total dataset |","| `educe-tags`    | Cluster subset and generate tags    |","| `hierarchical`  | Create two-level taxonomy           |","| `search`        | Semantic search by query            |","| `similar`       | Find duplicate/similar documents    |","| `custom-prompt` | Ask custom questions to clusters    |","| `visualize`     | Interactive 2D visualization        |"]},"column":{"s":0,"e":57}},"dim":["","paragraph.6"],"code":"| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |"},{"id":"/root/children/6/children/0","type":"text","loc":{"start":663,"end":781,"line":{"s":30,"e":32,"code":["| Command         | Purpose                             |","| --------------- | ----------------------------------- |","| `init`          | Cache embeddings from total dataset |"]},"column":{"s":0,"e":2}},"dim":["","paragraph.6","text.0"],"code":"| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| "},{"id":"/root/children/6/children/1","type":"inlineCode","loc":{"start":781,"end":787,"line":{"s":32,"e":32,"code":["| `init`          | Cache embeddings from total dataset |"]},"column":{"s":2,"e":8}},"dim":["","paragraph.6","inlineCode.1"],"code":"`init`"},{"id":"/root/children/6/children/2","type":"text","loc":{"start":787,"end":839,"line":{"s":32,"e":33,"code":["| `init`          | Cache embeddings from total dataset |","| `educe-tags`    | Cluster subset and generate tags    |"]},"column":{"s":8,"e":2}},"dim":["","paragraph.6","text.2"],"code":"          | Cache embeddings from total dataset |\n| "},{"id":"/root/children/6/children/3","type":"inlineCode","loc":{"start":839,"end":851,"line":{"s":33,"e":33,"code":["| `educe-tags`    | Cluster subset and generate tags    |"]},"column":{"s":2,"e":14}},"dim":["","paragraph.6","inlineCode.3"],"code":"`educe-tags`"},{"id":"/root/children/6/children/4","type":"text","loc":{"start":851,"end":897,"line":{"s":33,"e":34,"code":["| `educe-tags`    | Cluster subset and generate tags    |","| `hierarchical`  | Create two-level taxonomy           |"]},"column":{"s":14,"e":2}},"dim":["","paragraph.6","text.4"],"code":"    | Cluster subset and generate tags    |\n| "},{"id":"/root/children/6/children/5","type":"inlineCode","loc":{"start":897,"end":911,"line":{"s":34,"e":34,"code":["| `hierarchical`  | Create two-level taxonomy           |"]},"column":{"s":2,"e":16}},"dim":["","paragraph.6","inlineCode.5"],"code":"`hierarchical`"},{"id":"/root/children/6/children/6","type":"text","loc":{"start":911,"end":955,"line":{"s":34,"e":35,"code":["| `hierarchical`  | Create two-level taxonomy           |","| `search`        | Semantic search by query            |"]},"column":{"s":16,"e":2}},"dim":["","paragraph.6","text.6"],"code":"  | Create two-level taxonomy           |\n| "},{"id":"/root/children/6/children/7","type":"inlineCode","loc":{"start":955,"end":963,"line":{"s":35,"e":35,"code":["| `search`        | Semantic search by query            |"]},"column":{"s":2,"e":10}},"dim":["","paragraph.6","inlineCode.7"],"code":"`search`"},{"id":"/root/children/6/children/8","type":"text","loc":{"start":963,"end":1013,"line":{"s":35,"e":36,"code":["| `search`        | Semantic search by query            |","| `similar`       | Find duplicate/similar documents    |"]},"column":{"s":10,"e":2}},"dim":["","paragraph.6","text.8"],"code":"        | Semantic search by query            |\n| "},{"id":"/root/children/6/children/9","type":"inlineCode","loc":{"start":1013,"end":1022,"line":{"s":36,"e":36,"code":["| `similar`       | Find duplicate/similar documents    |"]},"column":{"s":2,"e":11}},"dim":["","paragraph.6","inlineCode.9"],"code":"`similar`"},{"id":"/root/children/6/children/10","type":"text","loc":{"start":1022,"end":1071,"line":{"s":36,"e":37,"code":["| `similar`       | Find duplicate/similar documents    |","| `custom-prompt` | Ask custom questions to clusters    |"]},"column":{"s":11,"e":2}},"dim":["","paragraph.6","text.10"],"code":"       | Find duplicate/similar documents    |\n| "},{"id":"/root/children/6/children/11","type":"inlineCode","loc":{"start":1071,"end":1086,"line":{"s":37,"e":37,"code":["| `custom-prompt` | Ask custom questions to clusters    |"]},"column":{"s":2,"e":17}},"dim":["","paragraph.6","inlineCode.11"],"code":"`custom-prompt`"},{"id":"/root/children/6/children/12","type":"text","loc":{"start":1086,"end":1129,"line":{"s":37,"e":38,"code":["| `custom-prompt` | Ask custom questions to clusters    |","| `visualize`     | Interactive 2D visualization        |"]},"column":{"s":17,"e":2}},"dim":["","paragraph.6","text.12"],"code":" | Ask custom questions to clusters    |\n| "},{"id":"/root/children/6/children/13","type":"inlineCode","loc":{"start":1129,"end":1140,"line":{"s":38,"e":38,"code":["| `visualize`     | Interactive 2D visualization        |"]},"column":{"s":2,"e":13}},"dim":["","paragraph.6","inlineCode.13"],"code":"`visualize`"},{"id":"/root/children/6/children/14","type":"text","loc":{"start":1140,"end":1184,"line":{"s":38,"e":38,"code":["| `visualize`     | Interactive 2D visualization        |"]},"column":{"s":13,"e":57}},"dim":["","paragraph.6","text.14"],"code":"     | Interactive 2D visualization        |"},{"id":"/root/children/7","type":"heading","loc":{"start":1186,"end":1198,"line":{"s":40,"e":40,"code":["## Workflows"]},"column":{"s":0,"e":12}},"dim":["","heading.7"],"code":"## Workflows","symbName":"heading","symbRange":[1200,6013],"symbRangeL":[40,41],"outerCode":"","outerHtml":""},{"id":"/root/children/7/children/0","type":"text","loc":{"start":1189,"end":1198,"line":{"s":40,"e":40,"code":["## Workflows"]},"column":{"s":3,"e":12}},"dim":["","heading.7","text.0"],"code":"Workflows"},{"id":"/root/children/8","type":"heading","loc":{"start":1200,"end":1236,"line":{"s":42,"e":42,"code":["### Workflow 1: Single-Level Tagging"]},"column":{"s":0,"e":36}},"dim":["","heading.8"],"code":"### Workflow 1: Single-Level Tagging","symbName":"heading","symbRange":[1238,1550],"symbRangeL":[42,59],"outerCode":"\nOne level of clusters with tags.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```\n\n**Result:** Documents with cluster and tags\n\n```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```","outerHtml":"\n<p>One level of clusters with tags.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```</p>\n\n<p><b>Result:</b> Documents with cluster and tags</p>\n\n<p>```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```</p>"},{"id":"/root/children/8/children/0","type":"text","loc":{"start":1204,"end":1236,"line":{"s":42,"e":42,"code":["### Workflow 1: Single-Level Tagging"]},"column":{"s":4,"e":36}},"dim":["","heading.8","text.0"],"code":"Workflow 1: Single-Level Tagging"},{"id":"/root/children/9","type":"paragraph","loc":{"start":1238,"end":1270,"line":{"s":44,"e":44,"code":["One level of clusters with tags."]},"column":{"s":0,"e":32}},"dim":["","paragraph.9"],"code":"One level of clusters with tags."},{"id":"/root/children/9/children/0","type":"text","loc":{"start":1238,"end":1270,"line":{"s":44,"e":44,"code":["One level of clusters with tags."]},"column":{"s":0,"e":32}},"dim":["","paragraph.9","text.0"],"code":"One level of clusters with tags."},{"id":"/root/children/10","type":"code","loc":{"start":1272,"end":1426,"line":{"s":46,"e":52,"code":["```bash","# 1. cache embeddings (once)","./llm-tagger.sh init total.ndjson","","# 2. cluster and tag","./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20","```"]},"column":{"s":0,"e":3}},"dim":["","code.10"],"code":"```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```","symbName":"code","symbRange":[1428,1471],"symbRangeL":[null,55],"outerCode":";{ engine: dot rankdir:LR clusterize: true }\n\n# LLM Tagger\n\nDocument clustering and analysis using embeddings and LLM.\n\n## Quick Start\n\n```bash\ncd docker/llm-tagger\n\n# step 1: cache embeddings (always first)\n./llm-tagger.sh init total.ndjson\n\n# step 2: cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100\n\n# optional: hierarchical tagging\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n\n# optional: analysis tools\n./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```\n\n## All Commands\n\n| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |\n\n## Workflows\n\n### Workflow 1: Single-Level Tagging\n\nOne level of clusters with tags.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```\n\n**Result:** Documents with cluster and tags","outerHtml":"<p>;{ engine: dot rankdir:LR clusterize: true }</p>\n\n<h1>LLM Tagger</h1>\n\n<p>Document clustering and analysis using embeddings and LLM.</p>\n\n<h2>Quick Start</h2>\n\n<p>```bash\ncd docker/llm-tagger</p>\n\n<h1>step 1: cache embeddings (always first)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>step 2: cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100</p>\n\n<h1>optional: hierarchical tagging</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3</p>\n\n<h1>optional: analysis tools</h1><p>./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```</p>\n\n<h2>All Commands</h2>\n\n<p>| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |</p>\n\n<h2>Workflows</h2>\n\n<h3>Workflow 1: Single-Level Tagging</h3>\n\n<p>One level of clusters with tags.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```</p>\n\n<p><b>Result:</b> Documents with cluster and tags</p>"},{"id":"/root/children/11","type":"paragraph","loc":{"start":1428,"end":1471,"line":{"s":54,"e":54,"code":["**Result:** Documents with cluster and tags"]},"column":{"s":0,"e":43}},"dim":["","paragraph.11"],"code":"**Result:** Documents with cluster and tags"},{"id":"/root/children/11/children/0","type":"strong","loc":{"start":1428,"end":1439,"line":{"s":54,"e":54,"code":["**Result:** Documents with cluster and tags"]},"column":{"s":0,"e":11}},"dim":["","paragraph.11","strong.0"],"code":"**Result:**"},{"id":"/root/children/11/children/0/children/0","type":"text","loc":{"start":1430,"end":1437,"line":{"s":54,"e":54,"code":["**Result:** Documents with cluster and tags"]},"column":{"s":2,"e":9}},"dim":["","paragraph.11","strong.0","text.0"],"code":"Result:"},{"id":"/root/children/11/children/1","type":"text","loc":{"start":1439,"end":1471,"line":{"s":54,"e":54,"code":["**Result:** Documents with cluster and tags"]},"column":{"s":11,"e":43}},"dim":["","paragraph.11","text.1"],"code":" Documents with cluster and tags"},{"id":"/root/children/12","type":"code","loc":{"start":1473,"end":1550,"line":{"s":56,"e":58,"code":["```json","{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }","```"]},"column":{"s":0,"e":3}},"dim":["","code.12"],"code":"```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```","symbName":"code","symbRange":[1552,1634],"symbRangeL":[null,63],"outerCode":";{ engine: dot rankdir:LR clusterize: true }\n\n# LLM Tagger\n\nDocument clustering and analysis using embeddings and LLM.\n\n## Quick Start\n\n```bash\ncd docker/llm-tagger\n\n# step 1: cache embeddings (always first)\n./llm-tagger.sh init total.ndjson\n\n# step 2: cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100\n\n# optional: hierarchical tagging\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n\n# optional: analysis tools\n./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```\n\n## All Commands\n\n| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |\n\n## Workflows\n\n### Workflow 1: Single-Level Tagging\n\nOne level of clusters with tags.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```\n\n**Result:** Documents with cluster and tags\n\n```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```\n\n### Workflow 2: Hierarchical Tagging\n\nTwo levels: fine-grained + broad categories.","outerHtml":"<p>;{ engine: dot rankdir:LR clusterize: true }</p>\n\n<h1>LLM Tagger</h1>\n\n<p>Document clustering and analysis using embeddings and LLM.</p>\n\n<h2>Quick Start</h2>\n\n<p>```bash\ncd docker/llm-tagger</p>\n\n<h1>step 1: cache embeddings (always first)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>step 2: cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100</p>\n\n<h1>optional: hierarchical tagging</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3</p>\n\n<h1>optional: analysis tools</h1><p>./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```</p>\n\n<h2>All Commands</h2>\n\n<p>| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |</p>\n\n<h2>Workflows</h2>\n\n<h3>Workflow 1: Single-Level Tagging</h3>\n\n<p>One level of clusters with tags.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```</p>\n\n<p><b>Result:</b> Documents with cluster and tags</p>\n\n<p>```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```</p>\n\n<h3>Workflow 2: Hierarchical Tagging</h3>\n\n<p>Two levels: fine-grained + broad categories.</p>"},{"id":"/root/children/13","type":"heading","loc":{"start":1552,"end":1588,"line":{"s":60,"e":60,"code":["### Workflow 2: Hierarchical Tagging"]},"column":{"s":0,"e":36}},"dim":["","heading.13"],"code":"### Workflow 2: Hierarchical Tagging","symbName":"heading","symbRange":[1590,2060],"symbRangeL":[60,86],"outerCode":"\nTwo levels: fine-grained + broad categories.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. fine-grained clustering\n./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5\n\n# 3. cluster the clusters\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```\n\n**Result:** Documents with two levels\n\n```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 12,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 2,\n  \"tags_l2\": \"hardware\"\n}\n```","outerHtml":"\n<p>Two levels: fine-grained + broad categories.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. fine-grained clustering</h1><p>./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5</p>\n\n<h1>3. cluster the clusters</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```</p>\n\n<p><b>Result:</b> Documents with two levels</p>\n\n<p>```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 12,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 2,\n  \"tags_l2\": \"hardware\"\n}\n```</p>"},{"id":"/root/children/13/children/0","type":"text","loc":{"start":1556,"end":1588,"line":{"s":60,"e":60,"code":["### Workflow 2: Hierarchical Tagging"]},"column":{"s":4,"e":36}},"dim":["","heading.13","text.0"],"code":"Workflow 2: Hierarchical Tagging"},{"id":"/root/children/14","type":"paragraph","loc":{"start":1590,"end":1634,"line":{"s":62,"e":62,"code":["Two levels: fine-grained + broad categories."]},"column":{"s":0,"e":44}},"dim":["","paragraph.14"],"code":"Two levels: fine-grained + broad categories."},{"id":"/root/children/14/children/0","type":"text","loc":{"start":1590,"end":1634,"line":{"s":62,"e":62,"code":["Two levels: fine-grained + broad categories."]},"column":{"s":0,"e":44}},"dim":["","paragraph.14","text.0"],"code":"Two levels: fine-grained + broad categories."},{"id":"/root/children/15","type":"code","loc":{"start":1636,"end":1883,"line":{"s":64,"e":73,"code":["```bash","# 1. cache embeddings (once)","./llm-tagger.sh init total.ndjson","","# 2. fine-grained clustering","./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5","","# 3. cluster the clusters","./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3","```"]},"column":{"s":0,"e":3}},"dim":["","code.15"],"code":"```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. fine-grained clustering\n./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5\n\n# 3. cluster the clusters\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```","symbName":"code","symbRange":[1885,1922],"symbRangeL":[null,76],"outerCode":";{ engine: dot rankdir:LR clusterize: true }\n\n# LLM Tagger\n\nDocument clustering and analysis using embeddings and LLM.\n\n## Quick Start\n\n```bash\ncd docker/llm-tagger\n\n# step 1: cache embeddings (always first)\n./llm-tagger.sh init total.ndjson\n\n# step 2: cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100\n\n# optional: hierarchical tagging\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n\n# optional: analysis tools\n./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```\n\n## All Commands\n\n| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |\n\n## Workflows\n\n### Workflow 1: Single-Level Tagging\n\nOne level of clusters with tags.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```\n\n**Result:** Documents with cluster and tags\n\n```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```\n\n### Workflow 2: Hierarchical Tagging\n\nTwo levels: fine-grained + broad categories.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. fine-grained clustering\n./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5\n\n# 3. cluster the clusters\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```\n\n**Result:** Documents with two levels","outerHtml":"<p>;{ engine: dot rankdir:LR clusterize: true }</p>\n\n<h1>LLM Tagger</h1>\n\n<p>Document clustering and analysis using embeddings and LLM.</p>\n\n<h2>Quick Start</h2>\n\n<p>```bash\ncd docker/llm-tagger</p>\n\n<h1>step 1: cache embeddings (always first)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>step 2: cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100</p>\n\n<h1>optional: hierarchical tagging</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3</p>\n\n<h1>optional: analysis tools</h1><p>./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```</p>\n\n<h2>All Commands</h2>\n\n<p>| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |</p>\n\n<h2>Workflows</h2>\n\n<h3>Workflow 1: Single-Level Tagging</h3>\n\n<p>One level of clusters with tags.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```</p>\n\n<p><b>Result:</b> Documents with cluster and tags</p>\n\n<p>```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```</p>\n\n<h3>Workflow 2: Hierarchical Tagging</h3>\n\n<p>Two levels: fine-grained + broad categories.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. fine-grained clustering</h1><p>./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5</p>\n\n<h1>3. cluster the clusters</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```</p>\n\n<p><b>Result:</b> Documents with two levels</p>"},{"id":"/root/children/16","type":"paragraph","loc":{"start":1885,"end":1922,"line":{"s":75,"e":75,"code":["**Result:** Documents with two levels"]},"column":{"s":0,"e":37}},"dim":["","paragraph.16"],"code":"**Result:** Documents with two levels"},{"id":"/root/children/16/children/0","type":"strong","loc":{"start":1885,"end":1896,"line":{"s":75,"e":75,"code":["**Result:** Documents with two levels"]},"column":{"s":0,"e":11}},"dim":["","paragraph.16","strong.0"],"code":"**Result:**"},{"id":"/root/children/16/children/0/children/0","type":"text","loc":{"start":1887,"end":1894,"line":{"s":75,"e":75,"code":["**Result:** Documents with two levels"]},"column":{"s":2,"e":9}},"dim":["","paragraph.16","strong.0","text.0"],"code":"Result:"},{"id":"/root/children/16/children/1","type":"text","loc":{"start":1896,"end":1922,"line":{"s":75,"e":75,"code":["**Result:** Documents with two levels"]},"column":{"s":11,"e":37}},"dim":["","paragraph.16","text.1"],"code":" Documents with two levels"},{"id":"/root/children/17","type":"code","loc":{"start":1924,"end":2060,"line":{"s":77,"e":85,"code":["```json","{","  \"description\": \"...\",","  \"cluster_l1\": 12,","  \"tags_l1\": \"screws, fasteners\",","  \"cluster_l2\": 2,","  \"tags_l2\": \"hardware\"","}","```"]},"column":{"s":0,"e":3}},"dim":["","code.17"],"code":"```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 12,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 2,\n  \"tags_l2\": \"hardware\"\n}\n```","symbName":"code","symbRange":[2062,2135],"symbRangeL":[null,90],"outerCode":";{ engine: dot rankdir:LR clusterize: true }\n\n# LLM Tagger\n\nDocument clustering and analysis using embeddings and LLM.\n\n## Quick Start\n\n```bash\ncd docker/llm-tagger\n\n# step 1: cache embeddings (always first)\n./llm-tagger.sh init total.ndjson\n\n# step 2: cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100\n\n# optional: hierarchical tagging\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n\n# optional: analysis tools\n./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```\n\n## All Commands\n\n| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |\n\n## Workflows\n\n### Workflow 1: Single-Level Tagging\n\nOne level of clusters with tags.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```\n\n**Result:** Documents with cluster and tags\n\n```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```\n\n### Workflow 2: Hierarchical Tagging\n\nTwo levels: fine-grained + broad categories.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. fine-grained clustering\n./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5\n\n# 3. cluster the clusters\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```\n\n**Result:** Documents with two levels\n\n```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 12,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 2,\n  \"tags_l2\": \"hardware\"\n}\n```\n\n### Workflow 3: Multiple Subsets\n\nAnalyze different slices independently.","outerHtml":"<p>;{ engine: dot rankdir:LR clusterize: true }</p>\n\n<h1>LLM Tagger</h1>\n\n<p>Document clustering and analysis using embeddings and LLM.</p>\n\n<h2>Quick Start</h2>\n\n<p>```bash\ncd docker/llm-tagger</p>\n\n<h1>step 1: cache embeddings (always first)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>step 2: cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100</p>\n\n<h1>optional: hierarchical tagging</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3</p>\n\n<h1>optional: analysis tools</h1><p>./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```</p>\n\n<h2>All Commands</h2>\n\n<p>| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |</p>\n\n<h2>Workflows</h2>\n\n<h3>Workflow 1: Single-Level Tagging</h3>\n\n<p>One level of clusters with tags.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```</p>\n\n<p><b>Result:</b> Documents with cluster and tags</p>\n\n<p>```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```</p>\n\n<h3>Workflow 2: Hierarchical Tagging</h3>\n\n<p>Two levels: fine-grained + broad categories.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. fine-grained clustering</h1><p>./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5</p>\n\n<h1>3. cluster the clusters</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```</p>\n\n<p><b>Result:</b> Documents with two levels</p>\n\n<p>```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 12,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 2,\n  \"tags_l2\": \"hardware\"\n}\n```</p>\n\n<h3>Workflow 3: Multiple Subsets</h3>\n\n<p>Analyze different slices independently.</p>"},{"id":"/root/children/18","type":"heading","loc":{"start":2062,"end":2094,"line":{"s":87,"e":87,"code":["### Workflow 3: Multiple Subsets"]},"column":{"s":0,"e":32}},"dim":["","heading.18"],"code":"### Workflow 3: Multiple Subsets","symbName":"heading","symbRange":[2096,2524],"symbRangeL":[87,102],"outerCode":"\nAnalyze different slices independently.\n\n```bash\n# 1. cache embeddings (once for all)\n./llm-tagger.sh init total.ndjson\n\n# 2. tag each subset (different patterns)\n./llm-tagger.sh educe-tags january.ndjson jan-tagged.ndjson 15\n./llm-tagger.sh educe-tags hardware.ndjson hw-tagged.ndjson 20\n./llm-tagger.sh educe-tags vendor-x.ndjson vx-tagged.ndjson 10\n```\n\n**Benefit:** Each subset gets unique clustering reflecting its patterns","outerHtml":"\n<p>Analyze different slices independently.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once for all)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. tag each subset (different patterns)</h1><p>./llm-tagger.sh educe-tags january.ndjson jan-tagged.ndjson 15\n./llm-tagger.sh educe-tags hardware.ndjson hw-tagged.ndjson 20\n./llm-tagger.sh educe-tags vendor-x.ndjson vx-tagged.ndjson 10\n```</p>\n\n<p><b>Benefit:</b> Each subset gets unique clustering reflecting its patterns</p>"},{"id":"/root/children/18/children/0","type":"text","loc":{"start":2066,"end":2094,"line":{"s":87,"e":87,"code":["### Workflow 3: Multiple Subsets"]},"column":{"s":4,"e":32}},"dim":["","heading.18","text.0"],"code":"Workflow 3: Multiple Subsets"},{"id":"/root/children/19","type":"paragraph","loc":{"start":2096,"end":2135,"line":{"s":89,"e":89,"code":["Analyze different slices independently."]},"column":{"s":0,"e":39}},"dim":["","paragraph.19"],"code":"Analyze different slices independently."},{"id":"/root/children/19/children/0","type":"text","loc":{"start":2096,"end":2135,"line":{"s":89,"e":89,"code":["Analyze different slices independently."]},"column":{"s":0,"e":39}},"dim":["","paragraph.19","text.0"],"code":"Analyze different slices independently."},{"id":"/root/children/20","type":"code","loc":{"start":2137,"end":2451,"line":{"s":91,"e":99,"code":["```bash","# 1. cache embeddings (once for all)","./llm-tagger.sh init total.ndjson","","# 2. tag each subset (different patterns)","./llm-tagger.sh educe-tags january.ndjson jan-tagged.ndjson 15","./llm-tagger.sh educe-tags hardware.ndjson hw-tagged.ndjson 20","./llm-tagger.sh educe-tags vendor-x.ndjson vx-tagged.ndjson 10","```"]},"column":{"s":0,"e":3}},"dim":["","code.20"],"code":"```bash\n# 1. cache embeddings (once for all)\n./llm-tagger.sh init total.ndjson\n\n# 2. tag each subset (different patterns)\n./llm-tagger.sh educe-tags january.ndjson jan-tagged.ndjson 15\n./llm-tagger.sh educe-tags hardware.ndjson hw-tagged.ndjson 20\n./llm-tagger.sh educe-tags vendor-x.ndjson vx-tagged.ndjson 10\n```","symbName":"code","symbRange":[2453,2596],"symbRangeL":[null,106],"outerCode":";{ engine: dot rankdir:LR clusterize: true }\n\n# LLM Tagger\n\nDocument clustering and analysis using embeddings and LLM.\n\n## Quick Start\n\n```bash\ncd docker/llm-tagger\n\n# step 1: cache embeddings (always first)\n./llm-tagger.sh init total.ndjson\n\n# step 2: cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100\n\n# optional: hierarchical tagging\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n\n# optional: analysis tools\n./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```\n\n## All Commands\n\n| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |\n\n## Workflows\n\n### Workflow 1: Single-Level Tagging\n\nOne level of clusters with tags.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```\n\n**Result:** Documents with cluster and tags\n\n```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```\n\n### Workflow 2: Hierarchical Tagging\n\nTwo levels: fine-grained + broad categories.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. fine-grained clustering\n./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5\n\n# 3. cluster the clusters\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```\n\n**Result:** Documents with two levels\n\n```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 12,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 2,\n  \"tags_l2\": \"hardware\"\n}\n```\n\n### Workflow 3: Multiple Subsets\n\nAnalyze different slices independently.\n\n```bash\n# 1. cache embeddings (once for all)\n./llm-tagger.sh init total.ndjson\n\n# 2. tag each subset (different patterns)\n./llm-tagger.sh educe-tags january.ndjson jan-tagged.ndjson 15\n./llm-tagger.sh educe-tags hardware.ndjson hw-tagged.ndjson 20\n./llm-tagger.sh educe-tags vendor-x.ndjson vx-tagged.ndjson 10\n```\n\n**Benefit:** Each subset gets unique clustering reflecting its patterns\n\n### Workflow 4: Multiple Subsets + Hierarchy\n\nCombine both approaches.","outerHtml":"<p>;{ engine: dot rankdir:LR clusterize: true }</p>\n\n<h1>LLM Tagger</h1>\n\n<p>Document clustering and analysis using embeddings and LLM.</p>\n\n<h2>Quick Start</h2>\n\n<p>```bash\ncd docker/llm-tagger</p>\n\n<h1>step 1: cache embeddings (always first)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>step 2: cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100</p>\n\n<h1>optional: hierarchical tagging</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3</p>\n\n<h1>optional: analysis tools</h1><p>./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```</p>\n\n<h2>All Commands</h2>\n\n<p>| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |</p>\n\n<h2>Workflows</h2>\n\n<h3>Workflow 1: Single-Level Tagging</h3>\n\n<p>One level of clusters with tags.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```</p>\n\n<p><b>Result:</b> Documents with cluster and tags</p>\n\n<p>```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```</p>\n\n<h3>Workflow 2: Hierarchical Tagging</h3>\n\n<p>Two levels: fine-grained + broad categories.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. fine-grained clustering</h1><p>./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5</p>\n\n<h1>3. cluster the clusters</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```</p>\n\n<p><b>Result:</b> Documents with two levels</p>\n\n<p>```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 12,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 2,\n  \"tags_l2\": \"hardware\"\n}\n```</p>\n\n<h3>Workflow 3: Multiple Subsets</h3>\n\n<p>Analyze different slices independently.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once for all)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. tag each subset (different patterns)</h1><p>./llm-tagger.sh educe-tags january.ndjson jan-tagged.ndjson 15\n./llm-tagger.sh educe-tags hardware.ndjson hw-tagged.ndjson 20\n./llm-tagger.sh educe-tags vendor-x.ndjson vx-tagged.ndjson 10\n```</p>\n\n<p><b>Benefit:</b> Each subset gets unique clustering reflecting its patterns</p>\n\n<h3>Workflow 4: Multiple Subsets + Hierarchy</h3>\n\n<p>Combine both approaches.</p>"},{"id":"/root/children/21","type":"paragraph","loc":{"start":2453,"end":2524,"line":{"s":101,"e":101,"code":["**Benefit:** Each subset gets unique clustering reflecting its patterns"]},"column":{"s":0,"e":71}},"dim":["","paragraph.21"],"code":"**Benefit:** Each subset gets unique clustering reflecting its patterns"},{"id":"/root/children/21/children/0","type":"strong","loc":{"start":2453,"end":2465,"line":{"s":101,"e":101,"code":["**Benefit:** Each subset gets unique clustering reflecting its patterns"]},"column":{"s":0,"e":12}},"dim":["","paragraph.21","strong.0"],"code":"**Benefit:**"},{"id":"/root/children/21/children/0/children/0","type":"text","loc":{"start":2455,"end":2463,"line":{"s":101,"e":101,"code":["**Benefit:** Each subset gets unique clustering reflecting its patterns"]},"column":{"s":2,"e":10}},"dim":["","paragraph.21","strong.0","text.0"],"code":"Benefit:"},{"id":"/root/children/21/children/1","type":"text","loc":{"start":2465,"end":2524,"line":{"s":101,"e":101,"code":["**Benefit:** Each subset gets unique clustering reflecting its patterns"]},"column":{"s":12,"e":71}},"dim":["","paragraph.21","text.1"],"code":" Each subset gets unique clustering reflecting its patterns"},{"id":"/root/children/22","type":"heading","loc":{"start":2526,"end":2570,"line":{"s":103,"e":103,"code":["### Workflow 4: Multiple Subsets + Hierarchy"]},"column":{"s":0,"e":44}},"dim":["","heading.22"],"code":"### Workflow 4: Multiple Subsets + Hierarchy","symbName":"heading","symbRange":[2572,2970],"symbRangeL":[103,119],"outerCode":"\nCombine both approaches.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. subset 1 with hierarchy\n./llm-tagger.sh educe-tags january.ndjson jan-l1.ndjson 5\n./llm-tagger.sh hierarchical jan-l1.ndjson jan-final.ndjson 3\n\n# 3. subset 2 with hierarchy\n./llm-tagger.sh educe-tags hardware.ndjson hw-l1.ndjson 5\n./llm-tagger.sh hierarchical hw-l1.ndjson hw-final.ndjson 3\n```","outerHtml":"\n<p>Combine both approaches.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. subset 1 with hierarchy</h1><p>./llm-tagger.sh educe-tags january.ndjson jan-l1.ndjson 5\n./llm-tagger.sh hierarchical jan-l1.ndjson jan-final.ndjson 3</p>\n\n<h1>3. subset 2 with hierarchy</h1><p>./llm-tagger.sh educe-tags hardware.ndjson hw-l1.ndjson 5\n./llm-tagger.sh hierarchical hw-l1.ndjson hw-final.ndjson 3\n```</p>"},{"id":"/root/children/22/children/0","type":"text","loc":{"start":2530,"end":2570,"line":{"s":103,"e":103,"code":["### Workflow 4: Multiple Subsets + Hierarchy"]},"column":{"s":4,"e":44}},"dim":["","heading.22","text.0"],"code":"Workflow 4: Multiple Subsets + Hierarchy"},{"id":"/root/children/23","type":"paragraph","loc":{"start":2572,"end":2596,"line":{"s":105,"e":105,"code":["Combine both approaches."]},"column":{"s":0,"e":24}},"dim":["","paragraph.23"],"code":"Combine both approaches."},{"id":"/root/children/23/children/0","type":"text","loc":{"start":2572,"end":2596,"line":{"s":105,"e":105,"code":["Combine both approaches."]},"column":{"s":0,"e":24}},"dim":["","paragraph.23","text.0"],"code":"Combine both approaches."},{"id":"/root/children/24","type":"code","loc":{"start":2598,"end":2970,"line":{"s":107,"e":118,"code":["```bash","# 1. cache embeddings (once)","./llm-tagger.sh init total.ndjson","","# 2. subset 1 with hierarchy","./llm-tagger.sh educe-tags january.ndjson jan-l1.ndjson 5","./llm-tagger.sh hierarchical jan-l1.ndjson jan-final.ndjson 3","","# 3. subset 2 with hierarchy","./llm-tagger.sh educe-tags hardware.ndjson hw-l1.ndjson 5","./llm-tagger.sh hierarchical hw-l1.ndjson hw-final.ndjson 3","```"]},"column":{"s":0,"e":3}},"dim":["","code.24"],"code":"```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. subset 1 with hierarchy\n./llm-tagger.sh educe-tags january.ndjson jan-l1.ndjson 5\n./llm-tagger.sh hierarchical jan-l1.ndjson jan-final.ndjson 3\n\n# 3. subset 2 with hierarchy\n./llm-tagger.sh educe-tags hardware.ndjson hw-l1.ndjson 5\n./llm-tagger.sh hierarchical hw-l1.ndjson hw-final.ndjson 3\n```","symbName":"code","symbRange":[2972,3909],"symbRangeL":[null,174],"outerCode":";{ engine: dot rankdir:LR clusterize: true }\n\n# LLM Tagger\n\nDocument clustering and analysis using embeddings and LLM.\n\n## Quick Start\n\n```bash\ncd docker/llm-tagger\n\n# step 1: cache embeddings (always first)\n./llm-tagger.sh init total.ndjson\n\n# step 2: cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100\n\n# optional: hierarchical tagging\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n\n# optional: analysis tools\n./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```\n\n## All Commands\n\n| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |\n\n## Workflows\n\n### Workflow 1: Single-Level Tagging\n\nOne level of clusters with tags.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```\n\n**Result:** Documents with cluster and tags\n\n```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```\n\n### Workflow 2: Hierarchical Tagging\n\nTwo levels: fine-grained + broad categories.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. fine-grained clustering\n./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5\n\n# 3. cluster the clusters\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```\n\n**Result:** Documents with two levels\n\n```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 12,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 2,\n  \"tags_l2\": \"hardware\"\n}\n```\n\n### Workflow 3: Multiple Subsets\n\nAnalyze different slices independently.\n\n```bash\n# 1. cache embeddings (once for all)\n./llm-tagger.sh init total.ndjson\n\n# 2. tag each subset (different patterns)\n./llm-tagger.sh educe-tags january.ndjson jan-tagged.ndjson 15\n./llm-tagger.sh educe-tags hardware.ndjson hw-tagged.ndjson 20\n./llm-tagger.sh educe-tags vendor-x.ndjson vx-tagged.ndjson 10\n```\n\n**Benefit:** Each subset gets unique clustering reflecting its patterns\n\n### Workflow 4: Multiple Subsets + Hierarchy\n\nCombine both approaches.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. subset 1 with hierarchy\n./llm-tagger.sh educe-tags january.ndjson jan-l1.ndjson 5\n./llm-tagger.sh hierarchical jan-l1.ndjson jan-final.ndjson 3\n\n# 3. subset 2 with hierarchy\n./llm-tagger.sh educe-tags hardware.ndjson hw-l1.ndjson 5\n./llm-tagger.sh hierarchical hw-l1.ndjson hw-final.ndjson 3\n```\n\n## Performance Changes\n\n### 1. Parallel Tag Extraction\n\n- 8 concurrent workers\n- removed sleep delays\n- 8x faster\n\n### 2. Connection Pooling\n\n- requests.Session() for reuse\n- all API calls\n\n### 3. Parallel Embedding\n\n- 4 concurrent workers\n- batch size: 32 → 64\n- maintains ordering\n\n### 4. HDBSCAN Multi-core\n\n- core_dist_n_jobs=-1\n- uses all cores\n\n**Overall speedup: 5-8x**\n\n## Two-Phase Architecture\n\n### Phase 1: init total.ndjson\n\n- pre-compute all embeddings\n- save to cache\n- no clustering yet\n\n### Phase 2: educe-tags subset.ndjson\n\n- load cached embeddings\n- cluster subset only\n- generate subset-specific tags\n- each subset: unique patterns\n\n### Benefits\n\n- embeddings computed once\n- each subset independently clustered\n- tags reflect subset context\n- january.ndjson → january patterns\n- hardware.ndjson → hardware categories\n\n## Hierarchical Tagging\n\nCreate two-level taxonomy: fine-grained + broad categories.\n\n### Workflow","outerHtml":"<p>;{ engine: dot rankdir:LR clusterize: true }</p>\n\n<h1>LLM Tagger</h1>\n\n<p>Document clustering and analysis using embeddings and LLM.</p>\n\n<h2>Quick Start</h2>\n\n<p>```bash\ncd docker/llm-tagger</p>\n\n<h1>step 1: cache embeddings (always first)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>step 2: cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100</p>\n\n<h1>optional: hierarchical tagging</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3</p>\n\n<h1>optional: analysis tools</h1><p>./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```</p>\n\n<h2>All Commands</h2>\n\n<p>| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |</p>\n\n<h2>Workflows</h2>\n\n<h3>Workflow 1: Single-Level Tagging</h3>\n\n<p>One level of clusters with tags.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```</p>\n\n<p><b>Result:</b> Documents with cluster and tags</p>\n\n<p>```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```</p>\n\n<h3>Workflow 2: Hierarchical Tagging</h3>\n\n<p>Two levels: fine-grained + broad categories.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. fine-grained clustering</h1><p>./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5</p>\n\n<h1>3. cluster the clusters</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```</p>\n\n<p><b>Result:</b> Documents with two levels</p>\n\n<p>```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 12,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 2,\n  \"tags_l2\": \"hardware\"\n}\n```</p>\n\n<h3>Workflow 3: Multiple Subsets</h3>\n\n<p>Analyze different slices independently.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once for all)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. tag each subset (different patterns)</h1><p>./llm-tagger.sh educe-tags january.ndjson jan-tagged.ndjson 15\n./llm-tagger.sh educe-tags hardware.ndjson hw-tagged.ndjson 20\n./llm-tagger.sh educe-tags vendor-x.ndjson vx-tagged.ndjson 10\n```</p>\n\n<p><b>Benefit:</b> Each subset gets unique clustering reflecting its patterns</p>\n\n<h3>Workflow 4: Multiple Subsets + Hierarchy</h3>\n\n<p>Combine both approaches.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. subset 1 with hierarchy</h1><p>./llm-tagger.sh educe-tags january.ndjson jan-l1.ndjson 5\n./llm-tagger.sh hierarchical jan-l1.ndjson jan-final.ndjson 3</p>\n\n<h1>3. subset 2 with hierarchy</h1><p>./llm-tagger.sh educe-tags hardware.ndjson hw-l1.ndjson 5\n./llm-tagger.sh hierarchical hw-l1.ndjson hw-final.ndjson 3\n```</p>\n\n<h2>Performance Changes</h2>\n\n<h3>1. Parallel Tag Extraction</h3>\n\n<ul><li>8 concurrent workers</li><li>removed sleep delays</li><li>8x faster</li></ul>\n\n<h3>2. Connection Pooling</h3>\n\n<ul><li>requests.Session() for reuse</li><li>all API calls</li></ul>\n\n<h3>3. Parallel Embedding</h3>\n\n<ul><li>4 concurrent workers</li><li>batch size: 32 → 64</li><li>maintains ordering</li></ul>\n\n<h3>4. HDBSCAN Multi-core</h3>\n\n<ul><li>core_dist_n_jobs=-1</li><li>uses all cores</li></ul>\n\n<p><b>Overall speedup: 5-8x</b></p>\n\n<h2>Two-Phase Architecture</h2>\n\n<h3>Phase 1: init total.ndjson</h3>\n\n<ul><li>pre-compute all embeddings</li><li>save to cache</li><li>no clustering yet</li></ul>\n\n<h3>Phase 2: educe-tags subset.ndjson</h3>\n\n<ul><li>load cached embeddings</li><li>cluster subset only</li><li>generate subset-specific tags</li><li>each subset: unique patterns</li></ul>\n\n<h3>Benefits</h3>\n\n<ul><li>embeddings computed once</li><li>each subset independently clustered</li><li>tags reflect subset context</li><li>january.ndjson → january patterns</li><li>hardware.ndjson → hardware categories</li></ul>\n\n<h2>Hierarchical Tagging</h2>\n\n<p>Create two-level taxonomy: fine-grained + broad categories.</p>\n\n<h3>Workflow</h3>"},{"id":"/root/children/25","type":"heading","loc":{"start":2972,"end":2994,"line":{"s":120,"e":120,"code":["## Performance Changes"]},"column":{"s":0,"e":22}},"dim":["","heading.25"],"code":"## Performance Changes","symbName":"heading","symbRange":[2996,6013],"symbRangeL":[120,121],"outerCode":"","outerHtml":""},{"id":"/root/children/25/children/0","type":"text","loc":{"start":2975,"end":2994,"line":{"s":120,"e":120,"code":["## Performance Changes"]},"column":{"s":3,"e":22}},"dim":["","heading.25","text.0"],"code":"Performance Changes"},{"id":"/root/children/26","type":"heading","loc":{"start":2996,"end":3026,"line":{"s":122,"e":122,"code":["### 1. Parallel Tag Extraction"]},"column":{"s":0,"e":30}},"dim":["","heading.26"],"code":"### 1. Parallel Tag Extraction","symbName":"heading","symbRange":[3028,3085],"symbRangeL":[122,127],"outerCode":"\n- 8 concurrent workers\n- removed sleep delays\n- 8x faster","outerHtml":"\n<ul><li>8 concurrent workers</li><li>removed sleep delays</li><li>8x faster</li></ul>"},{"id":"/root/children/26/children/0","type":"text","loc":{"start":3000,"end":3026,"line":{"s":122,"e":122,"code":["### 1. Parallel Tag Extraction"]},"column":{"s":4,"e":30}},"dim":["","heading.26","text.0"],"code":"1. Parallel Tag Extraction"},{"id":"/root/children/27","type":"list","loc":{"start":3028,"end":3085,"line":{"s":124,"e":126,"code":["- 8 concurrent workers","- removed sleep delays","- 8x faster"]},"column":{"s":0,"e":11}},"dim":["","list.27"],"code":"- 8 concurrent workers\n- removed sleep delays\n- 8x faster","symbName":"list","symbRange":[3087,3112],"symbRangeL":[124,129],"outerCode":"- removed sleep delays\n- 8x faster\n\n### 2. Connection Pooling","outerHtml":"<ul><li>removed sleep delays</li><li>8x faster</li></ul>\n\n<h3>2. Connection Pooling</h3>"},{"id":"/root/children/27/children/0","type":"listItem","loc":{"start":3028,"end":3050,"line":{"s":124,"e":124,"code":["- 8 concurrent workers"]},"column":{"s":0,"e":22}},"dim":["","list.27","listItem.0"],"code":"- 8 concurrent workers"},{"id":"/root/children/27/children/0/children/0","type":"paragraph","loc":{"start":3030,"end":3050,"line":{"s":124,"e":124,"code":["- 8 concurrent workers"]},"column":{"s":2,"e":22}},"dim":["","list.27","listItem.0","paragraph.0"],"code":"8 concurrent workers"},{"id":"/root/children/27/children/0/children/0/children/0","type":"text","loc":{"start":3030,"end":3050,"line":{"s":124,"e":124,"code":["- 8 concurrent workers"]},"column":{"s":2,"e":22}},"dim":["","list.27","listItem.0","paragraph.0","text.0"],"code":"8 concurrent workers"},{"id":"/root/children/27/children/1","type":"listItem","loc":{"start":3051,"end":3073,"line":{"s":125,"e":125,"code":["- removed sleep delays"]},"column":{"s":0,"e":22}},"dim":["","list.27","listItem.1"],"code":"- removed sleep delays"},{"id":"/root/children/27/children/1/children/0","type":"paragraph","loc":{"start":3053,"end":3073,"line":{"s":125,"e":125,"code":["- removed sleep delays"]},"column":{"s":2,"e":22}},"dim":["","list.27","listItem.1","paragraph.0"],"code":"removed sleep delays"},{"id":"/root/children/27/children/1/children/0/children/0","type":"text","loc":{"start":3053,"end":3073,"line":{"s":125,"e":125,"code":["- removed sleep delays"]},"column":{"s":2,"e":22}},"dim":["","list.27","listItem.1","paragraph.0","text.0"],"code":"removed sleep delays"},{"id":"/root/children/27/children/2","type":"listItem","loc":{"start":3074,"end":3085,"line":{"s":126,"e":126,"code":["- 8x faster"]},"column":{"s":0,"e":11}},"dim":["","list.27","listItem.2"],"code":"- 8x faster"},{"id":"/root/children/27/children/2/children/0","type":"paragraph","loc":{"start":3076,"end":3085,"line":{"s":126,"e":126,"code":["- 8x faster"]},"column":{"s":2,"e":11}},"dim":["","list.27","listItem.2","paragraph.0"],"code":"8x faster"},{"id":"/root/children/27/children/2/children/0/children/0","type":"text","loc":{"start":3076,"end":3085,"line":{"s":126,"e":126,"code":["- 8x faster"]},"column":{"s":2,"e":11}},"dim":["","list.27","listItem.2","paragraph.0","text.0"],"code":"8x faster"},{"id":"/root/children/28","type":"heading","loc":{"start":3087,"end":3112,"line":{"s":128,"e":128,"code":["### 2. Connection Pooling"]},"column":{"s":0,"e":25}},"dim":["","heading.28"],"code":"### 2. Connection Pooling","symbName":"heading","symbRange":[3114,3160],"symbRangeL":[128,132],"outerCode":"\n- requests.Session() for reuse\n- all API calls","outerHtml":"\n<ul><li>requests.Session() for reuse</li><li>all API calls</li></ul>"},{"id":"/root/children/28/children/0","type":"text","loc":{"start":3091,"end":3112,"line":{"s":128,"e":128,"code":["### 2. Connection Pooling"]},"column":{"s":4,"e":25}},"dim":["","heading.28","text.0"],"code":"2. Connection Pooling"},{"id":"/root/children/29","type":"list","loc":{"start":3114,"end":3160,"line":{"s":130,"e":131,"code":["- requests.Session() for reuse","- all API calls"]},"column":{"s":0,"e":15}},"dim":["","list.29"],"code":"- requests.Session() for reuse\n- all API calls","symbName":"list","symbRange":[3162,3187],"symbRangeL":[130,134],"outerCode":"- all API calls\n\n### 3. Parallel Embedding","outerHtml":"<ul><li>all API calls</li></ul>\n\n<h3>3. Parallel Embedding</h3>"},{"id":"/root/children/29/children/0","type":"listItem","loc":{"start":3114,"end":3144,"line":{"s":130,"e":130,"code":["- requests.Session() for reuse"]},"column":{"s":0,"e":30}},"dim":["","list.29","listItem.0"],"code":"- requests.Session() for reuse"},{"id":"/root/children/29/children/0/children/0","type":"paragraph","loc":{"start":3116,"end":3144,"line":{"s":130,"e":130,"code":["- requests.Session() for reuse"]},"column":{"s":2,"e":30}},"dim":["","list.29","listItem.0","paragraph.0"],"code":"requests.Session() for reuse"},{"id":"/root/children/29/children/0/children/0/children/0","type":"text","loc":{"start":3116,"end":3144,"line":{"s":130,"e":130,"code":["- requests.Session() for reuse"]},"column":{"s":2,"e":30}},"dim":["","list.29","listItem.0","paragraph.0","text.0"],"code":"requests.Session() for reuse"},{"id":"/root/children/29/children/1","type":"listItem","loc":{"start":3145,"end":3160,"line":{"s":131,"e":131,"code":["- all API calls"]},"column":{"s":0,"e":15}},"dim":["","list.29","listItem.1"],"code":"- all API calls"},{"id":"/root/children/29/children/1/children/0","type":"paragraph","loc":{"start":3147,"end":3160,"line":{"s":131,"e":131,"code":["- all API calls"]},"column":{"s":2,"e":15}},"dim":["","list.29","listItem.1","paragraph.0"],"code":"all API calls"},{"id":"/root/children/29/children/1/children/0/children/0","type":"text","loc":{"start":3147,"end":3160,"line":{"s":131,"e":131,"code":["- all API calls"]},"column":{"s":2,"e":15}},"dim":["","list.29","listItem.1","paragraph.0","text.0"],"code":"all API calls"},{"id":"/root/children/30","type":"heading","loc":{"start":3162,"end":3187,"line":{"s":133,"e":133,"code":["### 3. Parallel Embedding"]},"column":{"s":0,"e":25}},"dim":["","heading.30"],"code":"### 3. Parallel Embedding","symbName":"heading","symbRange":[3189,3254],"symbRangeL":[133,138],"outerCode":"\n- 4 concurrent workers\n- batch size: 32 → 64\n- maintains ordering","outerHtml":"\n<ul><li>4 concurrent workers</li><li>batch size: 32 → 64</li><li>maintains ordering</li></ul>"},{"id":"/root/children/30/children/0","type":"text","loc":{"start":3166,"end":3187,"line":{"s":133,"e":133,"code":["### 3. Parallel Embedding"]},"column":{"s":4,"e":25}},"dim":["","heading.30","text.0"],"code":"3. Parallel Embedding"},{"id":"/root/children/31","type":"list","loc":{"start":3189,"end":3254,"line":{"s":135,"e":137,"code":["- 4 concurrent workers","- batch size: 32 → 64","- maintains ordering"]},"column":{"s":0,"e":20}},"dim":["","list.31"],"code":"- 4 concurrent workers\n- batch size: 32 → 64\n- maintains ordering","symbName":"list","symbRange":[3256,3281],"symbRangeL":[135,140],"outerCode":"- batch size: 32 → 64\n- maintains ordering\n\n### 4. HDBSCAN Multi-core","outerHtml":"<ul><li>batch size: 32 → 64</li><li>maintains ordering</li></ul>\n\n<h3>4. HDBSCAN Multi-core</h3>"},{"id":"/root/children/31/children/0","type":"listItem","loc":{"start":3189,"end":3211,"line":{"s":135,"e":135,"code":["- 4 concurrent workers"]},"column":{"s":0,"e":22}},"dim":["","list.31","listItem.0"],"code":"- 4 concurrent workers"},{"id":"/root/children/31/children/0/children/0","type":"paragraph","loc":{"start":3191,"end":3211,"line":{"s":135,"e":135,"code":["- 4 concurrent workers"]},"column":{"s":2,"e":22}},"dim":["","list.31","listItem.0","paragraph.0"],"code":"4 concurrent workers"},{"id":"/root/children/31/children/0/children/0/children/0","type":"text","loc":{"start":3191,"end":3211,"line":{"s":135,"e":135,"code":["- 4 concurrent workers"]},"column":{"s":2,"e":22}},"dim":["","list.31","listItem.0","paragraph.0","text.0"],"code":"4 concurrent workers"},{"id":"/root/children/31/children/1","type":"listItem","loc":{"start":3212,"end":3233,"line":{"s":136,"e":136,"code":["- batch size: 32 → 64"]},"column":{"s":0,"e":21}},"dim":["","list.31","listItem.1"],"code":"- batch size: 32 → 64"},{"id":"/root/children/31/children/1/children/0","type":"paragraph","loc":{"start":3214,"end":3233,"line":{"s":136,"e":136,"code":["- batch size: 32 → 64"]},"column":{"s":2,"e":21}},"dim":["","list.31","listItem.1","paragraph.0"],"code":"batch size: 32 → 64"},{"id":"/root/children/31/children/1/children/0/children/0","type":"text","loc":{"start":3214,"end":3233,"line":{"s":136,"e":136,"code":["- batch size: 32 → 64"]},"column":{"s":2,"e":21}},"dim":["","list.31","listItem.1","paragraph.0","text.0"],"code":"batch size: 32 → 64"},{"id":"/root/children/31/children/2","type":"listItem","loc":{"start":3234,"end":3254,"line":{"s":137,"e":137,"code":["- maintains ordering"]},"column":{"s":0,"e":20}},"dim":["","list.31","listItem.2"],"code":"- maintains ordering"},{"id":"/root/children/31/children/2/children/0","type":"paragraph","loc":{"start":3236,"end":3254,"line":{"s":137,"e":137,"code":["- maintains ordering"]},"column":{"s":2,"e":20}},"dim":["","list.31","listItem.2","paragraph.0"],"code":"maintains ordering"},{"id":"/root/children/31/children/2/children/0/children/0","type":"text","loc":{"start":3236,"end":3254,"line":{"s":137,"e":137,"code":["- maintains ordering"]},"column":{"s":2,"e":20}},"dim":["","list.31","listItem.2","paragraph.0","text.0"],"code":"maintains ordering"},{"id":"/root/children/32","type":"heading","loc":{"start":3256,"end":3281,"line":{"s":139,"e":139,"code":["### 4. HDBSCAN Multi-core"]},"column":{"s":0,"e":25}},"dim":["","heading.32"],"code":"### 4. HDBSCAN Multi-core","symbName":"heading","symbRange":[3283,3348],"symbRangeL":[139,145],"outerCode":"\n- core_dist_n_jobs=-1\n- uses all cores\n\n**Overall speedup: 5-8x**","outerHtml":"\n<ul><li>core_dist_n_jobs=-1</li><li>uses all cores</li></ul>\n\n<p><b>Overall speedup: 5-8x</b></p>"},{"id":"/root/children/32/children/0","type":"text","loc":{"start":3260,"end":3281,"line":{"s":139,"e":139,"code":["### 4. HDBSCAN Multi-core"]},"column":{"s":4,"e":25}},"dim":["","heading.32","text.0"],"code":"4. HDBSCAN Multi-core"},{"id":"/root/children/33","type":"list","loc":{"start":3283,"end":3321,"line":{"s":141,"e":142,"code":["- core_dist_n_jobs=-1","- uses all cores"]},"column":{"s":0,"e":16}},"dim":["","list.33"],"code":"- core_dist_n_jobs=-1\n- uses all cores","symbName":"list","symbRange":[3323,3407],"symbRangeL":[141,149],"outerCode":"- uses all cores\n\n**Overall speedup: 5-8x**\n\n## Two-Phase Architecture\n\n### Phase 1: init total.ndjson","outerHtml":"<ul><li>uses all cores</li></ul>\n\n<p><b>Overall speedup: 5-8x</b></p>\n\n<h2>Two-Phase Architecture</h2>\n\n<h3>Phase 1: init total.ndjson</h3>"},{"id":"/root/children/33/children/0","type":"listItem","loc":{"start":3283,"end":3304,"line":{"s":141,"e":141,"code":["- core_dist_n_jobs=-1"]},"column":{"s":0,"e":21}},"dim":["","list.33","listItem.0"],"code":"- core_dist_n_jobs=-1"},{"id":"/root/children/33/children/0/children/0","type":"paragraph","loc":{"start":3285,"end":3304,"line":{"s":141,"e":141,"code":["- core_dist_n_jobs=-1"]},"column":{"s":2,"e":21}},"dim":["","list.33","listItem.0","paragraph.0"],"code":"core_dist_n_jobs=-1"},{"id":"/root/children/33/children/0/children/0/children/0","type":"text","loc":{"start":3285,"end":3304,"line":{"s":141,"e":141,"code":["- core_dist_n_jobs=-1"]},"column":{"s":2,"e":21}},"dim":["","list.33","listItem.0","paragraph.0","text.0"],"code":"core_dist_n_jobs=-1"},{"id":"/root/children/33/children/1","type":"listItem","loc":{"start":3305,"end":3321,"line":{"s":142,"e":142,"code":["- uses all cores"]},"column":{"s":0,"e":16}},"dim":["","list.33","listItem.1"],"code":"- uses all cores"},{"id":"/root/children/33/children/1/children/0","type":"paragraph","loc":{"start":3307,"end":3321,"line":{"s":142,"e":142,"code":["- uses all cores"]},"column":{"s":2,"e":16}},"dim":["","list.33","listItem.1","paragraph.0"],"code":"uses all cores"},{"id":"/root/children/33/children/1/children/0/children/0","type":"text","loc":{"start":3307,"end":3321,"line":{"s":142,"e":142,"code":["- uses all cores"]},"column":{"s":2,"e":16}},"dim":["","list.33","listItem.1","paragraph.0","text.0"],"code":"uses all cores"},{"id":"/root/children/34","type":"paragraph","loc":{"start":3323,"end":3348,"line":{"s":144,"e":144,"code":["**Overall speedup: 5-8x**"]},"column":{"s":0,"e":25}},"dim":["","paragraph.34"],"code":"**Overall speedup: 5-8x**"},{"id":"/root/children/34/children/0","type":"strong","loc":{"start":3323,"end":3348,"line":{"s":144,"e":144,"code":["**Overall speedup: 5-8x**"]},"column":{"s":0,"e":25}},"dim":["","paragraph.34","strong.0"],"code":"**Overall speedup: 5-8x**"},{"id":"/root/children/34/children/0/children/0","type":"text","loc":{"start":3325,"end":3346,"line":{"s":144,"e":144,"code":["**Overall speedup: 5-8x**"]},"column":{"s":2,"e":23}},"dim":["","paragraph.34","strong.0","text.0"],"code":"Overall speedup: 5-8x"},{"id":"/root/children/35","type":"heading","loc":{"start":3350,"end":3375,"line":{"s":146,"e":146,"code":["## Two-Phase Architecture"]},"column":{"s":0,"e":25}},"dim":["","heading.35"],"code":"## Two-Phase Architecture","symbName":"heading","symbRange":[3377,6013],"symbRangeL":[146,147],"outerCode":"","outerHtml":""},{"id":"/root/children/35/children/0","type":"text","loc":{"start":3353,"end":3375,"line":{"s":146,"e":146,"code":["## Two-Phase Architecture"]},"column":{"s":3,"e":25}},"dim":["","heading.35","text.0"],"code":"Two-Phase Architecture"},{"id":"/root/children/36","type":"heading","loc":{"start":3377,"end":3407,"line":{"s":148,"e":148,"code":["### Phase 1: init total.ndjson"]},"column":{"s":0,"e":30}},"dim":["","heading.36"],"code":"### Phase 1: init total.ndjson","symbName":"heading","symbRange":[3409,3473],"symbRangeL":[148,153],"outerCode":"\n- pre-compute all embeddings\n- save to cache\n- no clustering yet","outerHtml":"\n<ul><li>pre-compute all embeddings</li><li>save to cache</li><li>no clustering yet</li></ul>"},{"id":"/root/children/36/children/0","type":"text","loc":{"start":3381,"end":3407,"line":{"s":148,"e":148,"code":["### Phase 1: init total.ndjson"]},"column":{"s":4,"e":30}},"dim":["","heading.36","text.0"],"code":"Phase 1: init total.ndjson"},{"id":"/root/children/37","type":"list","loc":{"start":3409,"end":3473,"line":{"s":150,"e":152,"code":["- pre-compute all embeddings","- save to cache","- no clustering yet"]},"column":{"s":0,"e":19}},"dim":["","list.37"],"code":"- pre-compute all embeddings\n- save to cache\n- no clustering yet","symbName":"list","symbRange":[3475,3512],"symbRangeL":[150,155],"outerCode":"- save to cache\n- no clustering yet\n\n### Phase 2: educe-tags subset.ndjson","outerHtml":"<ul><li>save to cache</li><li>no clustering yet</li></ul>\n\n<h3>Phase 2: educe-tags subset.ndjson</h3>"},{"id":"/root/children/37/children/0","type":"listItem","loc":{"start":3409,"end":3437,"line":{"s":150,"e":150,"code":["- pre-compute all embeddings"]},"column":{"s":0,"e":28}},"dim":["","list.37","listItem.0"],"code":"- pre-compute all embeddings"},{"id":"/root/children/37/children/0/children/0","type":"paragraph","loc":{"start":3411,"end":3437,"line":{"s":150,"e":150,"code":["- pre-compute all embeddings"]},"column":{"s":2,"e":28}},"dim":["","list.37","listItem.0","paragraph.0"],"code":"pre-compute all embeddings"},{"id":"/root/children/37/children/0/children/0/children/0","type":"text","loc":{"start":3411,"end":3437,"line":{"s":150,"e":150,"code":["- pre-compute all embeddings"]},"column":{"s":2,"e":28}},"dim":["","list.37","listItem.0","paragraph.0","text.0"],"code":"pre-compute all embeddings"},{"id":"/root/children/37/children/1","type":"listItem","loc":{"start":3438,"end":3453,"line":{"s":151,"e":151,"code":["- save to cache"]},"column":{"s":0,"e":15}},"dim":["","list.37","listItem.1"],"code":"- save to cache"},{"id":"/root/children/37/children/1/children/0","type":"paragraph","loc":{"start":3440,"end":3453,"line":{"s":151,"e":151,"code":["- save to cache"]},"column":{"s":2,"e":15}},"dim":["","list.37","listItem.1","paragraph.0"],"code":"save to cache"},{"id":"/root/children/37/children/1/children/0/children/0","type":"text","loc":{"start":3440,"end":3453,"line":{"s":151,"e":151,"code":["- save to cache"]},"column":{"s":2,"e":15}},"dim":["","list.37","listItem.1","paragraph.0","text.0"],"code":"save to cache"},{"id":"/root/children/37/children/2","type":"listItem","loc":{"start":3454,"end":3473,"line":{"s":152,"e":152,"code":["- no clustering yet"]},"column":{"s":0,"e":19}},"dim":["","list.37","listItem.2"],"code":"- no clustering yet"},{"id":"/root/children/37/children/2/children/0","type":"paragraph","loc":{"start":3456,"end":3473,"line":{"s":152,"e":152,"code":["- no clustering yet"]},"column":{"s":2,"e":19}},"dim":["","list.37","listItem.2","paragraph.0"],"code":"no clustering yet"},{"id":"/root/children/37/children/2/children/0/children/0","type":"text","loc":{"start":3456,"end":3473,"line":{"s":152,"e":152,"code":["- no clustering yet"]},"column":{"s":2,"e":19}},"dim":["","list.37","listItem.2","paragraph.0","text.0"],"code":"no clustering yet"},{"id":"/root/children/38","type":"heading","loc":{"start":3475,"end":3512,"line":{"s":154,"e":154,"code":["### Phase 2: educe-tags subset.ndjson"]},"column":{"s":0,"e":37}},"dim":["","heading.38"],"code":"### Phase 2: educe-tags subset.ndjson","symbName":"heading","symbRange":[3514,3623],"symbRangeL":[154,160],"outerCode":"\n- load cached embeddings\n- cluster subset only\n- generate subset-specific tags\n- each subset: unique patterns","outerHtml":"\n<ul><li>load cached embeddings</li><li>cluster subset only</li><li>generate subset-specific tags</li><li>each subset: unique patterns</li></ul>"},{"id":"/root/children/38/children/0","type":"text","loc":{"start":3479,"end":3512,"line":{"s":154,"e":154,"code":["### Phase 2: educe-tags subset.ndjson"]},"column":{"s":4,"e":37}},"dim":["","heading.38","text.0"],"code":"Phase 2: educe-tags subset.ndjson"},{"id":"/root/children/39","type":"list","loc":{"start":3514,"end":3623,"line":{"s":156,"e":159,"code":["- load cached embeddings","- cluster subset only","- generate subset-specific tags","- each subset: unique patterns"]},"column":{"s":0,"e":30}},"dim":["","list.39"],"code":"- load cached embeddings\n- cluster subset only\n- generate subset-specific tags\n- each subset: unique patterns","symbName":"list","symbRange":[3625,3637],"symbRangeL":[156,162],"outerCode":"- cluster subset only\n- generate subset-specific tags\n- each subset: unique patterns\n\n### Benefits","outerHtml":"<ul><li>cluster subset only</li><li>generate subset-specific tags</li><li>each subset: unique patterns</li></ul>\n\n<h3>Benefits</h3>"},{"id":"/root/children/39/children/0","type":"listItem","loc":{"start":3514,"end":3538,"line":{"s":156,"e":156,"code":["- load cached embeddings"]},"column":{"s":0,"e":24}},"dim":["","list.39","listItem.0"],"code":"- load cached embeddings"},{"id":"/root/children/39/children/0/children/0","type":"paragraph","loc":{"start":3516,"end":3538,"line":{"s":156,"e":156,"code":["- load cached embeddings"]},"column":{"s":2,"e":24}},"dim":["","list.39","listItem.0","paragraph.0"],"code":"load cached embeddings"},{"id":"/root/children/39/children/0/children/0/children/0","type":"text","loc":{"start":3516,"end":3538,"line":{"s":156,"e":156,"code":["- load cached embeddings"]},"column":{"s":2,"e":24}},"dim":["","list.39","listItem.0","paragraph.0","text.0"],"code":"load cached embeddings"},{"id":"/root/children/39/children/1","type":"listItem","loc":{"start":3539,"end":3560,"line":{"s":157,"e":157,"code":["- cluster subset only"]},"column":{"s":0,"e":21}},"dim":["","list.39","listItem.1"],"code":"- cluster subset only"},{"id":"/root/children/39/children/1/children/0","type":"paragraph","loc":{"start":3541,"end":3560,"line":{"s":157,"e":157,"code":["- cluster subset only"]},"column":{"s":2,"e":21}},"dim":["","list.39","listItem.1","paragraph.0"],"code":"cluster subset only"},{"id":"/root/children/39/children/1/children/0/children/0","type":"text","loc":{"start":3541,"end":3560,"line":{"s":157,"e":157,"code":["- cluster subset only"]},"column":{"s":2,"e":21}},"dim":["","list.39","listItem.1","paragraph.0","text.0"],"code":"cluster subset only"},{"id":"/root/children/39/children/2","type":"listItem","loc":{"start":3561,"end":3592,"line":{"s":158,"e":158,"code":["- generate subset-specific tags"]},"column":{"s":0,"e":31}},"dim":["","list.39","listItem.2"],"code":"- generate subset-specific tags"},{"id":"/root/children/39/children/2/children/0","type":"paragraph","loc":{"start":3563,"end":3592,"line":{"s":158,"e":158,"code":["- generate subset-specific tags"]},"column":{"s":2,"e":31}},"dim":["","list.39","listItem.2","paragraph.0"],"code":"generate subset-specific tags"},{"id":"/root/children/39/children/2/children/0/children/0","type":"text","loc":{"start":3563,"end":3592,"line":{"s":158,"e":158,"code":["- generate subset-specific tags"]},"column":{"s":2,"e":31}},"dim":["","list.39","listItem.2","paragraph.0","text.0"],"code":"generate subset-specific tags"},{"id":"/root/children/39/children/3","type":"listItem","loc":{"start":3593,"end":3623,"line":{"s":159,"e":159,"code":["- each subset: unique patterns"]},"column":{"s":0,"e":30}},"dim":["","list.39","listItem.3"],"code":"- each subset: unique patterns"},{"id":"/root/children/39/children/3/children/0","type":"paragraph","loc":{"start":3595,"end":3623,"line":{"s":159,"e":159,"code":["- each subset: unique patterns"]},"column":{"s":2,"e":30}},"dim":["","list.39","listItem.3","paragraph.0"],"code":"each subset: unique patterns"},{"id":"/root/children/39/children/3/children/0/children/0","type":"text","loc":{"start":3595,"end":3623,"line":{"s":159,"e":159,"code":["- each subset: unique patterns"]},"column":{"s":2,"e":30}},"dim":["","list.39","listItem.3","paragraph.0","text.0"],"code":"each subset: unique patterns"},{"id":"/root/children/40","type":"heading","loc":{"start":3625,"end":3637,"line":{"s":161,"e":161,"code":["### Benefits"]},"column":{"s":0,"e":12}},"dim":["","heading.40"],"code":"### Benefits","symbName":"heading","symbRange":[3639,3809],"symbRangeL":[161,168],"outerCode":"\n- embeddings computed once\n- each subset independently clustered\n- tags reflect subset context\n- january.ndjson → january patterns\n- hardware.ndjson → hardware categories","outerHtml":"\n<ul><li>embeddings computed once</li><li>each subset independently clustered</li><li>tags reflect subset context</li><li>january.ndjson → january patterns</li><li>hardware.ndjson → hardware categories</li></ul>"},{"id":"/root/children/40/children/0","type":"text","loc":{"start":3629,"end":3637,"line":{"s":161,"e":161,"code":["### Benefits"]},"column":{"s":4,"e":12}},"dim":["","heading.40","text.0"],"code":"Benefits"},{"id":"/root/children/41","type":"list","loc":{"start":3639,"end":3809,"line":{"s":163,"e":167,"code":["- embeddings computed once","- each subset independently clustered","- tags reflect subset context","- january.ndjson → january patterns","- hardware.ndjson → hardware categories"]},"column":{"s":0,"e":39}},"dim":["","list.41"],"code":"- embeddings computed once\n- each subset independently clustered\n- tags reflect subset context\n- january.ndjson → january patterns\n- hardware.ndjson → hardware categories","symbName":"list","symbRange":[3811,4135],"symbRangeL":[163,184],"outerCode":"- each subset independently clustered\n- tags reflect subset context\n- january.ndjson → january patterns\n- hardware.ndjson → hardware categories\n\n## Hierarchical Tagging\n\nCreate two-level taxonomy: fine-grained + broad categories.\n\n### Workflow\n\n```bash\n# level 1: fine-grained (many small clusters)\n./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5\n\n# level 2: cluster the clusters\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```\n\n### How It Works","outerHtml":"<ul><li>each subset independently clustered</li><li>tags reflect subset context</li><li>january.ndjson → january patterns</li><li>hardware.ndjson → hardware categories</li></ul>\n\n<h2>Hierarchical Tagging</h2>\n\n<p>Create two-level taxonomy: fine-grained + broad categories.</p>\n\n<h3>Workflow</h3>\n\n<p>```bash</p><h1>level 1: fine-grained (many small clusters)</h1><p>./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5</p>\n\n<h1>level 2: cluster the clusters</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```</p>\n\n<h3>How It Works</h3>"},{"id":"/root/children/41/children/0","type":"listItem","loc":{"start":3639,"end":3665,"line":{"s":163,"e":163,"code":["- embeddings computed once"]},"column":{"s":0,"e":26}},"dim":["","list.41","listItem.0"],"code":"- embeddings computed once"},{"id":"/root/children/41/children/0/children/0","type":"paragraph","loc":{"start":3641,"end":3665,"line":{"s":163,"e":163,"code":["- embeddings computed once"]},"column":{"s":2,"e":26}},"dim":["","list.41","listItem.0","paragraph.0"],"code":"embeddings computed once"},{"id":"/root/children/41/children/0/children/0/children/0","type":"text","loc":{"start":3641,"end":3665,"line":{"s":163,"e":163,"code":["- embeddings computed once"]},"column":{"s":2,"e":26}},"dim":["","list.41","listItem.0","paragraph.0","text.0"],"code":"embeddings computed once"},{"id":"/root/children/41/children/1","type":"listItem","loc":{"start":3666,"end":3703,"line":{"s":164,"e":164,"code":["- each subset independently clustered"]},"column":{"s":0,"e":37}},"dim":["","list.41","listItem.1"],"code":"- each subset independently clustered"},{"id":"/root/children/41/children/1/children/0","type":"paragraph","loc":{"start":3668,"end":3703,"line":{"s":164,"e":164,"code":["- each subset independently clustered"]},"column":{"s":2,"e":37}},"dim":["","list.41","listItem.1","paragraph.0"],"code":"each subset independently clustered"},{"id":"/root/children/41/children/1/children/0/children/0","type":"text","loc":{"start":3668,"end":3703,"line":{"s":164,"e":164,"code":["- each subset independently clustered"]},"column":{"s":2,"e":37}},"dim":["","list.41","listItem.1","paragraph.0","text.0"],"code":"each subset independently clustered"},{"id":"/root/children/41/children/2","type":"listItem","loc":{"start":3704,"end":3733,"line":{"s":165,"e":165,"code":["- tags reflect subset context"]},"column":{"s":0,"e":29}},"dim":["","list.41","listItem.2"],"code":"- tags reflect subset context"},{"id":"/root/children/41/children/2/children/0","type":"paragraph","loc":{"start":3706,"end":3733,"line":{"s":165,"e":165,"code":["- tags reflect subset context"]},"column":{"s":2,"e":29}},"dim":["","list.41","listItem.2","paragraph.0"],"code":"tags reflect subset context"},{"id":"/root/children/41/children/2/children/0/children/0","type":"text","loc":{"start":3706,"end":3733,"line":{"s":165,"e":165,"code":["- tags reflect subset context"]},"column":{"s":2,"e":29}},"dim":["","list.41","listItem.2","paragraph.0","text.0"],"code":"tags reflect subset context"},{"id":"/root/children/41/children/3","type":"listItem","loc":{"start":3734,"end":3769,"line":{"s":166,"e":166,"code":["- january.ndjson → january patterns"]},"column":{"s":0,"e":35}},"dim":["","list.41","listItem.3"],"code":"- january.ndjson → january patterns"},{"id":"/root/children/41/children/3/children/0","type":"paragraph","loc":{"start":3736,"end":3769,"line":{"s":166,"e":166,"code":["- january.ndjson → january patterns"]},"column":{"s":2,"e":35}},"dim":["","list.41","listItem.3","paragraph.0"],"code":"january.ndjson → january patterns"},{"id":"/root/children/41/children/3/children/0/children/0","type":"text","loc":{"start":3736,"end":3769,"line":{"s":166,"e":166,"code":["- january.ndjson → january patterns"]},"column":{"s":2,"e":35}},"dim":["","list.41","listItem.3","paragraph.0","text.0"],"code":"january.ndjson → january patterns"},{"id":"/root/children/41/children/4","type":"listItem","loc":{"start":3770,"end":3809,"line":{"s":167,"e":167,"code":["- hardware.ndjson → hardware categories"]},"column":{"s":0,"e":39}},"dim":["","list.41","listItem.4"],"code":"- hardware.ndjson → hardware categories"},{"id":"/root/children/41/children/4/children/0","type":"paragraph","loc":{"start":3772,"end":3809,"line":{"s":167,"e":167,"code":["- hardware.ndjson → hardware categories"]},"column":{"s":2,"e":39}},"dim":["","list.41","listItem.4","paragraph.0"],"code":"hardware.ndjson → hardware categories"},{"id":"/root/children/41/children/4/children/0/children/0","type":"text","loc":{"start":3772,"end":3809,"line":{"s":167,"e":167,"code":["- hardware.ndjson → hardware categories"]},"column":{"s":2,"e":39}},"dim":["","list.41","listItem.4","paragraph.0","text.0"],"code":"hardware.ndjson → hardware categories"},{"id":"/root/children/42","type":"heading","loc":{"start":3811,"end":3834,"line":{"s":169,"e":169,"code":["## Hierarchical Tagging"]},"column":{"s":0,"e":23}},"dim":["","heading.42"],"code":"## Hierarchical Tagging","symbName":"heading","symbRange":[3836,3895],"symbRangeL":[169,172],"outerCode":"\nCreate two-level taxonomy: fine-grained + broad categories.","outerHtml":"\n<p>Create two-level taxonomy: fine-grained + broad categories.</p>"},{"id":"/root/children/42/children/0","type":"text","loc":{"start":3814,"end":3834,"line":{"s":169,"e":169,"code":["## Hierarchical Tagging"]},"column":{"s":3,"e":23}},"dim":["","heading.42","text.0"],"code":"Hierarchical Tagging"},{"id":"/root/children/43","type":"paragraph","loc":{"start":3836,"end":3895,"line":{"s":171,"e":171,"code":["Create two-level taxonomy: fine-grained + broad categories."]},"column":{"s":0,"e":59}},"dim":["","paragraph.43"],"code":"Create two-level taxonomy: fine-grained + broad categories."},{"id":"/root/children/43/children/0","type":"text","loc":{"start":3836,"end":3895,"line":{"s":171,"e":171,"code":["Create two-level taxonomy: fine-grained + broad categories."]},"column":{"s":0,"e":59}},"dim":["","paragraph.43","text.0"],"code":"Create two-level taxonomy: fine-grained + broad categories."},{"id":"/root/children/44","type":"heading","loc":{"start":3897,"end":3909,"line":{"s":173,"e":173,"code":["### Workflow"]},"column":{"s":0,"e":12}},"dim":["","heading.44"],"code":"### Workflow","symbName":"heading","symbRange":[3911,4117],"symbRangeL":[173,182],"outerCode":"\n```bash\n# level 1: fine-grained (many small clusters)\n./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5\n\n# level 2: cluster the clusters\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```","outerHtml":"\n<p>```bash</p><h1>level 1: fine-grained (many small clusters)</h1><p>./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5</p>\n\n<h1>level 2: cluster the clusters</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```</p>"},{"id":"/root/children/44/children/0","type":"text","loc":{"start":3901,"end":3909,"line":{"s":173,"e":173,"code":["### Workflow"]},"column":{"s":4,"e":12}},"dim":["","heading.44","text.0"],"code":"Workflow"},{"id":"/root/children/45","type":"code","loc":{"start":3911,"end":4117,"line":{"s":175,"e":181,"code":["```bash","# level 1: fine-grained (many small clusters)","./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5","","# level 2: cluster the clusters","./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3","```"]},"column":{"s":0,"e":3}},"dim":["","code.45"],"code":"```bash\n# level 1: fine-grained (many small clusters)\n./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5\n\n# level 2: cluster the clusters\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```","symbName":"code","symbRange":[4119,4378],"symbRangeL":[null,191],"outerCode":";{ engine: dot rankdir:LR clusterize: true }\n\n# LLM Tagger\n\nDocument clustering and analysis using embeddings and LLM.\n\n## Quick Start\n\n```bash\ncd docker/llm-tagger\n\n# step 1: cache embeddings (always first)\n./llm-tagger.sh init total.ndjson\n\n# step 2: cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100\n\n# optional: hierarchical tagging\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n\n# optional: analysis tools\n./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```\n\n## All Commands\n\n| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |\n\n## Workflows\n\n### Workflow 1: Single-Level Tagging\n\nOne level of clusters with tags.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```\n\n**Result:** Documents with cluster and tags\n\n```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```\n\n### Workflow 2: Hierarchical Tagging\n\nTwo levels: fine-grained + broad categories.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. fine-grained clustering\n./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5\n\n# 3. cluster the clusters\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```\n\n**Result:** Documents with two levels\n\n```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 12,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 2,\n  \"tags_l2\": \"hardware\"\n}\n```\n\n### Workflow 3: Multiple Subsets\n\nAnalyze different slices independently.\n\n```bash\n# 1. cache embeddings (once for all)\n./llm-tagger.sh init total.ndjson\n\n# 2. tag each subset (different patterns)\n./llm-tagger.sh educe-tags january.ndjson jan-tagged.ndjson 15\n./llm-tagger.sh educe-tags hardware.ndjson hw-tagged.ndjson 20\n./llm-tagger.sh educe-tags vendor-x.ndjson vx-tagged.ndjson 10\n```\n\n**Benefit:** Each subset gets unique clustering reflecting its patterns\n\n### Workflow 4: Multiple Subsets + Hierarchy\n\nCombine both approaches.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. subset 1 with hierarchy\n./llm-tagger.sh educe-tags january.ndjson jan-l1.ndjson 5\n./llm-tagger.sh hierarchical jan-l1.ndjson jan-final.ndjson 3\n\n# 3. subset 2 with hierarchy\n./llm-tagger.sh educe-tags hardware.ndjson hw-l1.ndjson 5\n./llm-tagger.sh hierarchical hw-l1.ndjson hw-final.ndjson 3\n```\n\n## Performance Changes\n\n### 1. Parallel Tag Extraction\n\n- 8 concurrent workers\n- removed sleep delays\n- 8x faster\n\n### 2. Connection Pooling\n\n- requests.Session() for reuse\n- all API calls\n\n### 3. Parallel Embedding\n\n- 4 concurrent workers\n- batch size: 32 → 64\n- maintains ordering\n\n### 4. HDBSCAN Multi-core\n\n- core_dist_n_jobs=-1\n- uses all cores\n\n**Overall speedup: 5-8x**\n\n## Two-Phase Architecture\n\n### Phase 1: init total.ndjson\n\n- pre-compute all embeddings\n- save to cache\n- no clustering yet\n\n### Phase 2: educe-tags subset.ndjson\n\n- load cached embeddings\n- cluster subset only\n- generate subset-specific tags\n- each subset: unique patterns\n\n### Benefits\n\n- embeddings computed once\n- each subset independently clustered\n- tags reflect subset context\n- january.ndjson → january patterns\n- hardware.ndjson → hardware categories\n\n## Hierarchical Tagging\n\nCreate two-level taxonomy: fine-grained + broad categories.\n\n### Workflow\n\n```bash\n# level 1: fine-grained (many small clusters)\n./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5\n\n# level 2: cluster the clusters\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```\n\n### How It Works\n\n1. **Level 1** creates detailed clusters (e.g., \"screws\", \"bolts\", \"cables\")\n1. **Level 2** computes cluster centroids\n1. Clusters the centroids into super-clusters\n1. Generates broad tags (e.g., \"hardware\", \"electronics\")\n\n### Output Format","outerHtml":"<p>;{ engine: dot rankdir:LR clusterize: true }</p>\n\n<h1>LLM Tagger</h1>\n\n<p>Document clustering and analysis using embeddings and LLM.</p>\n\n<h2>Quick Start</h2>\n\n<p>```bash\ncd docker/llm-tagger</p>\n\n<h1>step 1: cache embeddings (always first)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>step 2: cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100</p>\n\n<h1>optional: hierarchical tagging</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3</p>\n\n<h1>optional: analysis tools</h1><p>./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```</p>\n\n<h2>All Commands</h2>\n\n<p>| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |</p>\n\n<h2>Workflows</h2>\n\n<h3>Workflow 1: Single-Level Tagging</h3>\n\n<p>One level of clusters with tags.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```</p>\n\n<p><b>Result:</b> Documents with cluster and tags</p>\n\n<p>```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```</p>\n\n<h3>Workflow 2: Hierarchical Tagging</h3>\n\n<p>Two levels: fine-grained + broad categories.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. fine-grained clustering</h1><p>./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5</p>\n\n<h1>3. cluster the clusters</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```</p>\n\n<p><b>Result:</b> Documents with two levels</p>\n\n<p>```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 12,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 2,\n  \"tags_l2\": \"hardware\"\n}\n```</p>\n\n<h3>Workflow 3: Multiple Subsets</h3>\n\n<p>Analyze different slices independently.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once for all)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. tag each subset (different patterns)</h1><p>./llm-tagger.sh educe-tags january.ndjson jan-tagged.ndjson 15\n./llm-tagger.sh educe-tags hardware.ndjson hw-tagged.ndjson 20\n./llm-tagger.sh educe-tags vendor-x.ndjson vx-tagged.ndjson 10\n```</p>\n\n<p><b>Benefit:</b> Each subset gets unique clustering reflecting its patterns</p>\n\n<h3>Workflow 4: Multiple Subsets + Hierarchy</h3>\n\n<p>Combine both approaches.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. subset 1 with hierarchy</h1><p>./llm-tagger.sh educe-tags january.ndjson jan-l1.ndjson 5\n./llm-tagger.sh hierarchical jan-l1.ndjson jan-final.ndjson 3</p>\n\n<h1>3. subset 2 with hierarchy</h1><p>./llm-tagger.sh educe-tags hardware.ndjson hw-l1.ndjson 5\n./llm-tagger.sh hierarchical hw-l1.ndjson hw-final.ndjson 3\n```</p>\n\n<h2>Performance Changes</h2>\n\n<h3>1. Parallel Tag Extraction</h3>\n\n<ul><li>8 concurrent workers</li><li>removed sleep delays</li><li>8x faster</li></ul>\n\n<h3>2. Connection Pooling</h3>\n\n<ul><li>requests.Session() for reuse</li><li>all API calls</li></ul>\n\n<h3>3. Parallel Embedding</h3>\n\n<ul><li>4 concurrent workers</li><li>batch size: 32 → 64</li><li>maintains ordering</li></ul>\n\n<h3>4. HDBSCAN Multi-core</h3>\n\n<ul><li>core_dist_n_jobs=-1</li><li>uses all cores</li></ul>\n\n<p><b>Overall speedup: 5-8x</b></p>\n\n<h2>Two-Phase Architecture</h2>\n\n<h3>Phase 1: init total.ndjson</h3>\n\n<ul><li>pre-compute all embeddings</li><li>save to cache</li><li>no clustering yet</li></ul>\n\n<h3>Phase 2: educe-tags subset.ndjson</h3>\n\n<ul><li>load cached embeddings</li><li>cluster subset only</li><li>generate subset-specific tags</li><li>each subset: unique patterns</li></ul>\n\n<h3>Benefits</h3>\n\n<ul><li>embeddings computed once</li><li>each subset independently clustered</li><li>tags reflect subset context</li><li>january.ndjson → january patterns</li><li>hardware.ndjson → hardware categories</li></ul>\n\n<h2>Hierarchical Tagging</h2>\n\n<p>Create two-level taxonomy: fine-grained + broad categories.</p>\n\n<h3>Workflow</h3>\n\n<p>```bash</p><h1>level 1: fine-grained (many small clusters)</h1><p>./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5</p>\n\n<h1>level 2: cluster the clusters</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```</p>\n\n<h3>How It Works</h3>\n\n<ol><li><b>Level 1</b> creates detailed clusters (e.g., \"screws\", \"bolts\", \"cables\")</li><li><b>Level 2</b> computes cluster centroids</li><li>Clusters the centroids into super-clusters</li><li>Generates broad tags (e.g., \"hardware\", \"electronics\")</li></ol>\n\n<h3>Output Format</h3>"},{"id":"/root/children/46","type":"heading","loc":{"start":4119,"end":4135,"line":{"s":183,"e":183,"code":["### How It Works"]},"column":{"s":0,"e":16}},"dim":["","heading.46"],"code":"### How It Works","symbName":"heading","symbRange":[4137,4359],"symbRangeL":[183,189],"outerCode":"\n1. **Level 1** creates detailed clusters (e.g., \"screws\", \"bolts\", \"cables\")\n1. **Level 2** computes cluster centroids\n1. Clusters the centroids into super-clusters\n1. Generates broad tags (e.g., \"hardware\", \"electronics\")","outerHtml":"\n<ol><li><b>Level 1</b> creates detailed clusters (e.g., \"screws\", \"bolts\", \"cables\")</li><li><b>Level 2</b> computes cluster centroids</li><li>Clusters the centroids into super-clusters</li><li>Generates broad tags (e.g., \"hardware\", \"electronics\")</li></ol>"},{"id":"/root/children/46/children/0","type":"text","loc":{"start":4123,"end":4135,"line":{"s":183,"e":183,"code":["### How It Works"]},"column":{"s":4,"e":16}},"dim":["","heading.46","text.0"],"code":"How It Works"},{"id":"/root/children/47","type":"list","loc":{"start":4137,"end":4359,"line":{"s":185,"e":188,"code":["1. **Level 1** creates detailed clusters (e.g., \"screws\", \"bolts\", \"cables\")","1. **Level 2** computes cluster centroids","1. Clusters the centroids into super-clusters","1. Generates broad tags (e.g., \"hardware\", \"electronics\")"]},"column":{"s":0,"e":57}},"dim":["","list.47"],"code":"1. **Level 1** creates detailed clusters (e.g., \"screws\", \"bolts\", \"cables\")\n1. **Level 2** computes cluster centroids\n1. Clusters the centroids into super-clusters\n1. Generates broad tags (e.g., \"hardware\", \"electronics\")","symbName":"list","symbRange":[4361,4537],"symbRangeL":[185,203],"outerCode":"1. **Level 2** computes cluster centroids\n1. Clusters the centroids into super-clusters\n1. Generates broad tags (e.g., \"hardware\", \"electronics\")\n\n### Output Format\n\n```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 5,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 1,\n  \"tags_l2\": \"hardware, tools\"\n}\n```\n\n### Use Cases","outerHtml":"<ol><li><b>Level 2</b> computes cluster centroids</li><li>Clusters the centroids into super-clusters</li><li>Generates broad tags (e.g., \"hardware\", \"electronics\")</li></ol>\n\n<h3>Output Format</h3>\n\n<p>```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 5,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 1,\n  \"tags_l2\": \"hardware, tools\"\n}\n```</p>\n\n<h3>Use Cases</h3>"},{"id":"/root/children/47/children/0","type":"listItem","loc":{"start":4137,"end":4213,"line":{"s":185,"e":185,"code":["1. **Level 1** creates detailed clusters (e.g., \"screws\", \"bolts\", \"cables\")"]},"column":{"s":0,"e":76}},"dim":["","list.47","listItem.0"],"code":"1. **Level 1** creates detailed clusters (e.g., \"screws\", \"bolts\", \"cables\")"},{"id":"/root/children/47/children/0/children/0","type":"paragraph","loc":{"start":4140,"end":4213,"line":{"s":185,"e":185,"code":["1. **Level 1** creates detailed clusters (e.g., \"screws\", \"bolts\", \"cables\")"]},"column":{"s":3,"e":76}},"dim":["","list.47","listItem.0","paragraph.0"],"code":"**Level 1** creates detailed clusters (e.g., \"screws\", \"bolts\", \"cables\")"},{"id":"/root/children/47/children/0/children/0/children/0","type":"strong","loc":{"start":4140,"end":4151,"line":{"s":185,"e":185,"code":["1. **Level 1** creates detailed clusters (e.g., \"screws\", \"bolts\", \"cables\")"]},"column":{"s":3,"e":14}},"dim":["","list.47","listItem.0","paragraph.0","strong.0"],"code":"**Level 1**"},{"id":"/root/children/47/children/0/children/0/children/0/children/0","type":"text","loc":{"start":4142,"end":4149,"line":{"s":185,"e":185,"code":["1. **Level 1** creates detailed clusters (e.g., \"screws\", \"bolts\", \"cables\")"]},"column":{"s":5,"e":12}},"dim":["","list.47","listItem.0","paragraph.0","strong.0","text.0"],"code":"Level 1"},{"id":"/root/children/47/children/0/children/0/children/1","type":"text","loc":{"start":4151,"end":4213,"line":{"s":185,"e":185,"code":["1. **Level 1** creates detailed clusters (e.g., \"screws\", \"bolts\", \"cables\")"]},"column":{"s":14,"e":76}},"dim":["","list.47","listItem.0","paragraph.0","text.1"],"code":" creates detailed clusters (e.g., \"screws\", \"bolts\", \"cables\")"},{"id":"/root/children/47/children/1","type":"listItem","loc":{"start":4214,"end":4255,"line":{"s":186,"e":186,"code":["1. **Level 2** computes cluster centroids"]},"column":{"s":0,"e":41}},"dim":["","list.47","listItem.1"],"code":"1. **Level 2** computes cluster centroids"},{"id":"/root/children/47/children/1/children/0","type":"paragraph","loc":{"start":4217,"end":4255,"line":{"s":186,"e":186,"code":["1. **Level 2** computes cluster centroids"]},"column":{"s":3,"e":41}},"dim":["","list.47","listItem.1","paragraph.0"],"code":"**Level 2** computes cluster centroids"},{"id":"/root/children/47/children/1/children/0/children/0","type":"strong","loc":{"start":4217,"end":4228,"line":{"s":186,"e":186,"code":["1. **Level 2** computes cluster centroids"]},"column":{"s":3,"e":14}},"dim":["","list.47","listItem.1","paragraph.0","strong.0"],"code":"**Level 2**"},{"id":"/root/children/47/children/1/children/0/children/0/children/0","type":"text","loc":{"start":4219,"end":4226,"line":{"s":186,"e":186,"code":["1. **Level 2** computes cluster centroids"]},"column":{"s":5,"e":12}},"dim":["","list.47","listItem.1","paragraph.0","strong.0","text.0"],"code":"Level 2"},{"id":"/root/children/47/children/1/children/0/children/1","type":"text","loc":{"start":4228,"end":4255,"line":{"s":186,"e":186,"code":["1. **Level 2** computes cluster centroids"]},"column":{"s":14,"e":41}},"dim":["","list.47","listItem.1","paragraph.0","text.1"],"code":" computes cluster centroids"},{"id":"/root/children/47/children/2","type":"listItem","loc":{"start":4256,"end":4301,"line":{"s":187,"e":187,"code":["1. Clusters the centroids into super-clusters"]},"column":{"s":0,"e":45}},"dim":["","list.47","listItem.2"],"code":"1. Clusters the centroids into super-clusters"},{"id":"/root/children/47/children/2/children/0","type":"paragraph","loc":{"start":4259,"end":4301,"line":{"s":187,"e":187,"code":["1. Clusters the centroids into super-clusters"]},"column":{"s":3,"e":45}},"dim":["","list.47","listItem.2","paragraph.0"],"code":"Clusters the centroids into super-clusters"},{"id":"/root/children/47/children/2/children/0/children/0","type":"text","loc":{"start":4259,"end":4301,"line":{"s":187,"e":187,"code":["1. Clusters the centroids into super-clusters"]},"column":{"s":3,"e":45}},"dim":["","list.47","listItem.2","paragraph.0","text.0"],"code":"Clusters the centroids into super-clusters"},{"id":"/root/children/47/children/3","type":"listItem","loc":{"start":4302,"end":4359,"line":{"s":188,"e":188,"code":["1. Generates broad tags (e.g., \"hardware\", \"electronics\")"]},"column":{"s":0,"e":57}},"dim":["","list.47","listItem.3"],"code":"1. Generates broad tags (e.g., \"hardware\", \"electronics\")"},{"id":"/root/children/47/children/3/children/0","type":"paragraph","loc":{"start":4305,"end":4359,"line":{"s":188,"e":188,"code":["1. Generates broad tags (e.g., \"hardware\", \"electronics\")"]},"column":{"s":3,"e":57}},"dim":["","list.47","listItem.3","paragraph.0"],"code":"Generates broad tags (e.g., \"hardware\", \"electronics\")"},{"id":"/root/children/47/children/3/children/0/children/0","type":"text","loc":{"start":4305,"end":4359,"line":{"s":188,"e":188,"code":["1. Generates broad tags (e.g., \"hardware\", \"electronics\")"]},"column":{"s":3,"e":57}},"dim":["","list.47","listItem.3","paragraph.0","text.0"],"code":"Generates broad tags (e.g., \"hardware\", \"electronics\")"},{"id":"/root/children/48","type":"heading","loc":{"start":4361,"end":4378,"line":{"s":190,"e":190,"code":["### Output Format"]},"column":{"s":0,"e":17}},"dim":["","heading.48"],"code":"### Output Format","symbName":"heading","symbRange":[4380,4522],"symbRangeL":[190,201],"outerCode":"\n```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 5,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 1,\n  \"tags_l2\": \"hardware, tools\"\n}\n```","outerHtml":"\n<p>```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 5,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 1,\n  \"tags_l2\": \"hardware, tools\"\n}\n```</p>"},{"id":"/root/children/48/children/0","type":"text","loc":{"start":4365,"end":4378,"line":{"s":190,"e":190,"code":["### Output Format"]},"column":{"s":4,"e":17}},"dim":["","heading.48","text.0"],"code":"Output Format"},{"id":"/root/children/49","type":"code","loc":{"start":4380,"end":4522,"line":{"s":192,"e":200,"code":["```json","{","  \"description\": \"...\",","  \"cluster_l1\": 5,","  \"tags_l1\": \"screws, fasteners\",","  \"cluster_l2\": 1,","  \"tags_l2\": \"hardware, tools\"","}","```"]},"column":{"s":0,"e":3}},"dim":["","code.49"],"code":"```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 5,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 1,\n  \"tags_l2\": \"hardware, tools\"\n}\n```","symbName":"code","symbRange":[4524,4759],"symbRangeL":[null,215],"outerCode":";{ engine: dot rankdir:LR clusterize: true }\n\n# LLM Tagger\n\nDocument clustering and analysis using embeddings and LLM.\n\n## Quick Start\n\n```bash\ncd docker/llm-tagger\n\n# step 1: cache embeddings (always first)\n./llm-tagger.sh init total.ndjson\n\n# step 2: cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100\n\n# optional: hierarchical tagging\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n\n# optional: analysis tools\n./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```\n\n## All Commands\n\n| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |\n\n## Workflows\n\n### Workflow 1: Single-Level Tagging\n\nOne level of clusters with tags.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```\n\n**Result:** Documents with cluster and tags\n\n```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```\n\n### Workflow 2: Hierarchical Tagging\n\nTwo levels: fine-grained + broad categories.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. fine-grained clustering\n./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5\n\n# 3. cluster the clusters\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```\n\n**Result:** Documents with two levels\n\n```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 12,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 2,\n  \"tags_l2\": \"hardware\"\n}\n```\n\n### Workflow 3: Multiple Subsets\n\nAnalyze different slices independently.\n\n```bash\n# 1. cache embeddings (once for all)\n./llm-tagger.sh init total.ndjson\n\n# 2. tag each subset (different patterns)\n./llm-tagger.sh educe-tags january.ndjson jan-tagged.ndjson 15\n./llm-tagger.sh educe-tags hardware.ndjson hw-tagged.ndjson 20\n./llm-tagger.sh educe-tags vendor-x.ndjson vx-tagged.ndjson 10\n```\n\n**Benefit:** Each subset gets unique clustering reflecting its patterns\n\n### Workflow 4: Multiple Subsets + Hierarchy\n\nCombine both approaches.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. subset 1 with hierarchy\n./llm-tagger.sh educe-tags january.ndjson jan-l1.ndjson 5\n./llm-tagger.sh hierarchical jan-l1.ndjson jan-final.ndjson 3\n\n# 3. subset 2 with hierarchy\n./llm-tagger.sh educe-tags hardware.ndjson hw-l1.ndjson 5\n./llm-tagger.sh hierarchical hw-l1.ndjson hw-final.ndjson 3\n```\n\n## Performance Changes\n\n### 1. Parallel Tag Extraction\n\n- 8 concurrent workers\n- removed sleep delays\n- 8x faster\n\n### 2. Connection Pooling\n\n- requests.Session() for reuse\n- all API calls\n\n### 3. Parallel Embedding\n\n- 4 concurrent workers\n- batch size: 32 → 64\n- maintains ordering\n\n### 4. HDBSCAN Multi-core\n\n- core_dist_n_jobs=-1\n- uses all cores\n\n**Overall speedup: 5-8x**\n\n## Two-Phase Architecture\n\n### Phase 1: init total.ndjson\n\n- pre-compute all embeddings\n- save to cache\n- no clustering yet\n\n### Phase 2: educe-tags subset.ndjson\n\n- load cached embeddings\n- cluster subset only\n- generate subset-specific tags\n- each subset: unique patterns\n\n### Benefits\n\n- embeddings computed once\n- each subset independently clustered\n- tags reflect subset context\n- january.ndjson → january patterns\n- hardware.ndjson → hardware categories\n\n## Hierarchical Tagging\n\nCreate two-level taxonomy: fine-grained + broad categories.\n\n### Workflow\n\n```bash\n# level 1: fine-grained (many small clusters)\n./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5\n\n# level 2: cluster the clusters\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```\n\n### How It Works\n\n1. **Level 1** creates detailed clusters (e.g., \"screws\", \"bolts\", \"cables\")\n1. **Level 2** computes cluster centroids\n1. Clusters the centroids into super-clusters\n1. Generates broad tags (e.g., \"hardware\", \"electronics\")\n\n### Output Format\n\n```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 5,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 1,\n  \"tags_l2\": \"hardware, tools\"\n}\n```\n\n### Use Cases\n\n- multi-level taxonomy\n- drill-down navigation\n- broad overview + details\n\n## Additional Tools\n\nBeyond clustering, embeddings enable powerful analysis.\n\n### 1. Semantic Search\n\nFind documents matching query semantically.","outerHtml":"<p>;{ engine: dot rankdir:LR clusterize: true }</p>\n\n<h1>LLM Tagger</h1>\n\n<p>Document clustering and analysis using embeddings and LLM.</p>\n\n<h2>Quick Start</h2>\n\n<p>```bash\ncd docker/llm-tagger</p>\n\n<h1>step 1: cache embeddings (always first)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>step 2: cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100</p>\n\n<h1>optional: hierarchical tagging</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3</p>\n\n<h1>optional: analysis tools</h1><p>./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```</p>\n\n<h2>All Commands</h2>\n\n<p>| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |</p>\n\n<h2>Workflows</h2>\n\n<h3>Workflow 1: Single-Level Tagging</h3>\n\n<p>One level of clusters with tags.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```</p>\n\n<p><b>Result:</b> Documents with cluster and tags</p>\n\n<p>```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```</p>\n\n<h3>Workflow 2: Hierarchical Tagging</h3>\n\n<p>Two levels: fine-grained + broad categories.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. fine-grained clustering</h1><p>./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5</p>\n\n<h1>3. cluster the clusters</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```</p>\n\n<p><b>Result:</b> Documents with two levels</p>\n\n<p>```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 12,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 2,\n  \"tags_l2\": \"hardware\"\n}\n```</p>\n\n<h3>Workflow 3: Multiple Subsets</h3>\n\n<p>Analyze different slices independently.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once for all)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. tag each subset (different patterns)</h1><p>./llm-tagger.sh educe-tags january.ndjson jan-tagged.ndjson 15\n./llm-tagger.sh educe-tags hardware.ndjson hw-tagged.ndjson 20\n./llm-tagger.sh educe-tags vendor-x.ndjson vx-tagged.ndjson 10\n```</p>\n\n<p><b>Benefit:</b> Each subset gets unique clustering reflecting its patterns</p>\n\n<h3>Workflow 4: Multiple Subsets + Hierarchy</h3>\n\n<p>Combine both approaches.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. subset 1 with hierarchy</h1><p>./llm-tagger.sh educe-tags january.ndjson jan-l1.ndjson 5\n./llm-tagger.sh hierarchical jan-l1.ndjson jan-final.ndjson 3</p>\n\n<h1>3. subset 2 with hierarchy</h1><p>./llm-tagger.sh educe-tags hardware.ndjson hw-l1.ndjson 5\n./llm-tagger.sh hierarchical hw-l1.ndjson hw-final.ndjson 3\n```</p>\n\n<h2>Performance Changes</h2>\n\n<h3>1. Parallel Tag Extraction</h3>\n\n<ul><li>8 concurrent workers</li><li>removed sleep delays</li><li>8x faster</li></ul>\n\n<h3>2. Connection Pooling</h3>\n\n<ul><li>requests.Session() for reuse</li><li>all API calls</li></ul>\n\n<h3>3. Parallel Embedding</h3>\n\n<ul><li>4 concurrent workers</li><li>batch size: 32 → 64</li><li>maintains ordering</li></ul>\n\n<h3>4. HDBSCAN Multi-core</h3>\n\n<ul><li>core_dist_n_jobs=-1</li><li>uses all cores</li></ul>\n\n<p><b>Overall speedup: 5-8x</b></p>\n\n<h2>Two-Phase Architecture</h2>\n\n<h3>Phase 1: init total.ndjson</h3>\n\n<ul><li>pre-compute all embeddings</li><li>save to cache</li><li>no clustering yet</li></ul>\n\n<h3>Phase 2: educe-tags subset.ndjson</h3>\n\n<ul><li>load cached embeddings</li><li>cluster subset only</li><li>generate subset-specific tags</li><li>each subset: unique patterns</li></ul>\n\n<h3>Benefits</h3>\n\n<ul><li>embeddings computed once</li><li>each subset independently clustered</li><li>tags reflect subset context</li><li>january.ndjson → january patterns</li><li>hardware.ndjson → hardware categories</li></ul>\n\n<h2>Hierarchical Tagging</h2>\n\n<p>Create two-level taxonomy: fine-grained + broad categories.</p>\n\n<h3>Workflow</h3>\n\n<p>```bash</p><h1>level 1: fine-grained (many small clusters)</h1><p>./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5</p>\n\n<h1>level 2: cluster the clusters</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```</p>\n\n<h3>How It Works</h3>\n\n<ol><li><b>Level 1</b> creates detailed clusters (e.g., \"screws\", \"bolts\", \"cables\")</li><li><b>Level 2</b> computes cluster centroids</li><li>Clusters the centroids into super-clusters</li><li>Generates broad tags (e.g., \"hardware\", \"electronics\")</li></ol>\n\n<h3>Output Format</h3>\n\n<p>```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 5,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 1,\n  \"tags_l2\": \"hardware, tools\"\n}\n```</p>\n\n<h3>Use Cases</h3>\n\n<ul><li>multi-level taxonomy</li><li>drill-down navigation</li><li>broad overview + details</li></ul>\n\n<h2>Additional Tools</h2>\n\n<p>Beyond clustering, embeddings enable powerful analysis.</p>\n\n<h3>1. Semantic Search</h3>\n\n<p>Find documents matching query semantically.</p>"},{"id":"/root/children/50","type":"heading","loc":{"start":4524,"end":4537,"line":{"s":202,"e":202,"code":["### Use Cases"]},"column":{"s":0,"e":13}},"dim":["","heading.50"],"code":"### Use Cases","symbName":"heading","symbRange":[4539,4612],"symbRangeL":[202,207],"outerCode":"\n- multi-level taxonomy\n- drill-down navigation\n- broad overview + details","outerHtml":"\n<ul><li>multi-level taxonomy</li><li>drill-down navigation</li><li>broad overview + details</li></ul>"},{"id":"/root/children/50/children/0","type":"text","loc":{"start":4528,"end":4537,"line":{"s":202,"e":202,"code":["### Use Cases"]},"column":{"s":4,"e":13}},"dim":["","heading.50","text.0"],"code":"Use Cases"},{"id":"/root/children/51","type":"list","loc":{"start":4539,"end":4612,"line":{"s":204,"e":206,"code":["- multi-level taxonomy","- drill-down navigation","- broad overview + details"]},"column":{"s":0,"e":26}},"dim":["","list.51"],"code":"- multi-level taxonomy\n- drill-down navigation\n- broad overview + details","symbName":"list","symbRange":[4614,4903],"symbRangeL":[204,223],"outerCode":"- drill-down navigation\n- broad overview + details\n\n## Additional Tools\n\nBeyond clustering, embeddings enable powerful analysis.\n\n### 1. Semantic Search\n\nFind documents matching query semantically.\n\n```bash\n./llm-tagger.sh search data.ndjson 'hardware repair' 10\n```\n\n**Output:** Top matching documents with similarity scores\n\n**Use cases:**","outerHtml":"<ul><li>drill-down navigation</li><li>broad overview + details</li></ul>\n\n<h2>Additional Tools</h2>\n\n<p>Beyond clustering, embeddings enable powerful analysis.</p>\n\n<h3>1. Semantic Search</h3>\n\n<p>Find documents matching query semantically.</p>\n\n<p>```bash\n./llm-tagger.sh search data.ndjson 'hardware repair' 10\n```</p>\n\n<p><b>Output:</b> Top matching documents with similarity scores</p>\n\n<p><b>Use cases:</b></p>"},{"id":"/root/children/51/children/0","type":"listItem","loc":{"start":4539,"end":4561,"line":{"s":204,"e":204,"code":["- multi-level taxonomy"]},"column":{"s":0,"e":22}},"dim":["","list.51","listItem.0"],"code":"- multi-level taxonomy"},{"id":"/root/children/51/children/0/children/0","type":"paragraph","loc":{"start":4541,"end":4561,"line":{"s":204,"e":204,"code":["- multi-level taxonomy"]},"column":{"s":2,"e":22}},"dim":["","list.51","listItem.0","paragraph.0"],"code":"multi-level taxonomy"},{"id":"/root/children/51/children/0/children/0/children/0","type":"text","loc":{"start":4541,"end":4561,"line":{"s":204,"e":204,"code":["- multi-level taxonomy"]},"column":{"s":2,"e":22}},"dim":["","list.51","listItem.0","paragraph.0","text.0"],"code":"multi-level taxonomy"},{"id":"/root/children/51/children/1","type":"listItem","loc":{"start":4562,"end":4585,"line":{"s":205,"e":205,"code":["- drill-down navigation"]},"column":{"s":0,"e":23}},"dim":["","list.51","listItem.1"],"code":"- drill-down navigation"},{"id":"/root/children/51/children/1/children/0","type":"paragraph","loc":{"start":4564,"end":4585,"line":{"s":205,"e":205,"code":["- drill-down navigation"]},"column":{"s":2,"e":23}},"dim":["","list.51","listItem.1","paragraph.0"],"code":"drill-down navigation"},{"id":"/root/children/51/children/1/children/0/children/0","type":"text","loc":{"start":4564,"end":4585,"line":{"s":205,"e":205,"code":["- drill-down navigation"]},"column":{"s":2,"e":23}},"dim":["","list.51","listItem.1","paragraph.0","text.0"],"code":"drill-down navigation"},{"id":"/root/children/51/children/2","type":"listItem","loc":{"start":4586,"end":4612,"line":{"s":206,"e":206,"code":["- broad overview + details"]},"column":{"s":0,"e":26}},"dim":["","list.51","listItem.2"],"code":"- broad overview + details"},{"id":"/root/children/51/children/2/children/0","type":"paragraph","loc":{"start":4588,"end":4612,"line":{"s":206,"e":206,"code":["- broad overview + details"]},"column":{"s":2,"e":26}},"dim":["","list.51","listItem.2","paragraph.0"],"code":"broad overview + details"},{"id":"/root/children/51/children/2/children/0/children/0","type":"text","loc":{"start":4588,"end":4612,"line":{"s":206,"e":206,"code":["- broad overview + details"]},"column":{"s":2,"e":26}},"dim":["","list.51","listItem.2","paragraph.0","text.0"],"code":"broad overview + details"},{"id":"/root/children/52","type":"heading","loc":{"start":4614,"end":4633,"line":{"s":208,"e":208,"code":["## Additional Tools"]},"column":{"s":0,"e":19}},"dim":["","heading.52"],"code":"## Additional Tools","symbName":"heading","symbRange":[4635,4690],"symbRangeL":[208,211],"outerCode":"\nBeyond clustering, embeddings enable powerful analysis.","outerHtml":"\n<p>Beyond clustering, embeddings enable powerful analysis.</p>"},{"id":"/root/children/52/children/0","type":"text","loc":{"start":4617,"end":4633,"line":{"s":208,"e":208,"code":["## Additional Tools"]},"column":{"s":3,"e":19}},"dim":["","heading.52","text.0"],"code":"Additional Tools"},{"id":"/root/children/53","type":"paragraph","loc":{"start":4635,"end":4690,"line":{"s":210,"e":210,"code":["Beyond clustering, embeddings enable powerful analysis."]},"column":{"s":0,"e":55}},"dim":["","paragraph.53"],"code":"Beyond clustering, embeddings enable powerful analysis."},{"id":"/root/children/53/children/0","type":"text","loc":{"start":4635,"end":4690,"line":{"s":210,"e":210,"code":["Beyond clustering, embeddings enable powerful analysis."]},"column":{"s":0,"e":55}},"dim":["","paragraph.53","text.0"],"code":"Beyond clustering, embeddings enable powerful analysis."},{"id":"/root/children/54","type":"heading","loc":{"start":4692,"end":4714,"line":{"s":212,"e":212,"code":["### 1. Semantic Search"]},"column":{"s":0,"e":22}},"dim":["","heading.54"],"code":"### 1. Semantic Search","symbName":"heading","symbRange":[4716,4974],"symbRangeL":[212,227],"outerCode":"\nFind documents matching query semantically.\n\n```bash\n./llm-tagger.sh search data.ndjson 'hardware repair' 10\n```\n\n**Output:** Top matching documents with similarity scores\n\n**Use cases:**\n\n- find relevant documents\n- content discovery\n- query-based filtering","outerHtml":"\n<p>Find documents matching query semantically.</p>\n\n<p>```bash\n./llm-tagger.sh search data.ndjson 'hardware repair' 10\n```</p>\n\n<p><b>Output:</b> Top matching documents with similarity scores</p>\n\n<p><b>Use cases:</b></p>\n\n<ul><li>find relevant documents</li><li>content discovery</li><li>query-based filtering</li></ul>"},{"id":"/root/children/54/children/0","type":"text","loc":{"start":4696,"end":4714,"line":{"s":212,"e":212,"code":["### 1. Semantic Search"]},"column":{"s":4,"e":22}},"dim":["","heading.54","text.0"],"code":"1. Semantic Search"},{"id":"/root/children/55","type":"paragraph","loc":{"start":4716,"end":4759,"line":{"s":214,"e":214,"code":["Find documents matching query semantically."]},"column":{"s":0,"e":43}},"dim":["","paragraph.55"],"code":"Find documents matching query semantically."},{"id":"/root/children/55/children/0","type":"text","loc":{"start":4716,"end":4759,"line":{"s":214,"e":214,"code":["Find documents matching query semantically."]},"column":{"s":0,"e":43}},"dim":["","paragraph.55","text.0"],"code":"Find documents matching query semantically."},{"id":"/root/children/56","type":"code","loc":{"start":4761,"end":4828,"line":{"s":216,"e":218,"code":["```bash","./llm-tagger.sh search data.ndjson 'hardware repair' 10","```"]},"column":{"s":0,"e":3}},"dim":["","code.56"],"code":"```bash\n./llm-tagger.sh search data.ndjson 'hardware repair' 10\n```","symbName":"code","symbRange":[4830,5044],"symbRangeL":[null,231],"outerCode":";{ engine: dot rankdir:LR clusterize: true }\n\n# LLM Tagger\n\nDocument clustering and analysis using embeddings and LLM.\n\n## Quick Start\n\n```bash\ncd docker/llm-tagger\n\n# step 1: cache embeddings (always first)\n./llm-tagger.sh init total.ndjson\n\n# step 2: cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100\n\n# optional: hierarchical tagging\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n\n# optional: analysis tools\n./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```\n\n## All Commands\n\n| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |\n\n## Workflows\n\n### Workflow 1: Single-Level Tagging\n\nOne level of clusters with tags.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```\n\n**Result:** Documents with cluster and tags\n\n```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```\n\n### Workflow 2: Hierarchical Tagging\n\nTwo levels: fine-grained + broad categories.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. fine-grained clustering\n./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5\n\n# 3. cluster the clusters\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```\n\n**Result:** Documents with two levels\n\n```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 12,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 2,\n  \"tags_l2\": \"hardware\"\n}\n```\n\n### Workflow 3: Multiple Subsets\n\nAnalyze different slices independently.\n\n```bash\n# 1. cache embeddings (once for all)\n./llm-tagger.sh init total.ndjson\n\n# 2. tag each subset (different patterns)\n./llm-tagger.sh educe-tags january.ndjson jan-tagged.ndjson 15\n./llm-tagger.sh educe-tags hardware.ndjson hw-tagged.ndjson 20\n./llm-tagger.sh educe-tags vendor-x.ndjson vx-tagged.ndjson 10\n```\n\n**Benefit:** Each subset gets unique clustering reflecting its patterns\n\n### Workflow 4: Multiple Subsets + Hierarchy\n\nCombine both approaches.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. subset 1 with hierarchy\n./llm-tagger.sh educe-tags january.ndjson jan-l1.ndjson 5\n./llm-tagger.sh hierarchical jan-l1.ndjson jan-final.ndjson 3\n\n# 3. subset 2 with hierarchy\n./llm-tagger.sh educe-tags hardware.ndjson hw-l1.ndjson 5\n./llm-tagger.sh hierarchical hw-l1.ndjson hw-final.ndjson 3\n```\n\n## Performance Changes\n\n### 1. Parallel Tag Extraction\n\n- 8 concurrent workers\n- removed sleep delays\n- 8x faster\n\n### 2. Connection Pooling\n\n- requests.Session() for reuse\n- all API calls\n\n### 3. Parallel Embedding\n\n- 4 concurrent workers\n- batch size: 32 → 64\n- maintains ordering\n\n### 4. HDBSCAN Multi-core\n\n- core_dist_n_jobs=-1\n- uses all cores\n\n**Overall speedup: 5-8x**\n\n## Two-Phase Architecture\n\n### Phase 1: init total.ndjson\n\n- pre-compute all embeddings\n- save to cache\n- no clustering yet\n\n### Phase 2: educe-tags subset.ndjson\n\n- load cached embeddings\n- cluster subset only\n- generate subset-specific tags\n- each subset: unique patterns\n\n### Benefits\n\n- embeddings computed once\n- each subset independently clustered\n- tags reflect subset context\n- january.ndjson → january patterns\n- hardware.ndjson → hardware categories\n\n## Hierarchical Tagging\n\nCreate two-level taxonomy: fine-grained + broad categories.\n\n### Workflow\n\n```bash\n# level 1: fine-grained (many small clusters)\n./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5\n\n# level 2: cluster the clusters\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```\n\n### How It Works\n\n1. **Level 1** creates detailed clusters (e.g., \"screws\", \"bolts\", \"cables\")\n1. **Level 2** computes cluster centroids\n1. Clusters the centroids into super-clusters\n1. Generates broad tags (e.g., \"hardware\", \"electronics\")\n\n### Output Format\n\n```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 5,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 1,\n  \"tags_l2\": \"hardware, tools\"\n}\n```\n\n### Use Cases\n\n- multi-level taxonomy\n- drill-down navigation\n- broad overview + details\n\n## Additional Tools\n\nBeyond clustering, embeddings enable powerful analysis.\n\n### 1. Semantic Search\n\nFind documents matching query semantically.\n\n```bash\n./llm-tagger.sh search data.ndjson 'hardware repair' 10\n```\n\n**Output:** Top matching documents with similarity scores\n\n**Use cases:**\n\n- find relevant documents\n- content discovery\n- query-based filtering\n\n### 2. Find Similar Documents\n\nDetect duplicates or near-duplicates.","outerHtml":"<p>;{ engine: dot rankdir:LR clusterize: true }</p>\n\n<h1>LLM Tagger</h1>\n\n<p>Document clustering and analysis using embeddings and LLM.</p>\n\n<h2>Quick Start</h2>\n\n<p>```bash\ncd docker/llm-tagger</p>\n\n<h1>step 1: cache embeddings (always first)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>step 2: cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100</p>\n\n<h1>optional: hierarchical tagging</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3</p>\n\n<h1>optional: analysis tools</h1><p>./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```</p>\n\n<h2>All Commands</h2>\n\n<p>| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |</p>\n\n<h2>Workflows</h2>\n\n<h3>Workflow 1: Single-Level Tagging</h3>\n\n<p>One level of clusters with tags.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```</p>\n\n<p><b>Result:</b> Documents with cluster and tags</p>\n\n<p>```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```</p>\n\n<h3>Workflow 2: Hierarchical Tagging</h3>\n\n<p>Two levels: fine-grained + broad categories.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. fine-grained clustering</h1><p>./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5</p>\n\n<h1>3. cluster the clusters</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```</p>\n\n<p><b>Result:</b> Documents with two levels</p>\n\n<p>```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 12,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 2,\n  \"tags_l2\": \"hardware\"\n}\n```</p>\n\n<h3>Workflow 3: Multiple Subsets</h3>\n\n<p>Analyze different slices independently.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once for all)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. tag each subset (different patterns)</h1><p>./llm-tagger.sh educe-tags january.ndjson jan-tagged.ndjson 15\n./llm-tagger.sh educe-tags hardware.ndjson hw-tagged.ndjson 20\n./llm-tagger.sh educe-tags vendor-x.ndjson vx-tagged.ndjson 10\n```</p>\n\n<p><b>Benefit:</b> Each subset gets unique clustering reflecting its patterns</p>\n\n<h3>Workflow 4: Multiple Subsets + Hierarchy</h3>\n\n<p>Combine both approaches.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. subset 1 with hierarchy</h1><p>./llm-tagger.sh educe-tags january.ndjson jan-l1.ndjson 5\n./llm-tagger.sh hierarchical jan-l1.ndjson jan-final.ndjson 3</p>\n\n<h1>3. subset 2 with hierarchy</h1><p>./llm-tagger.sh educe-tags hardware.ndjson hw-l1.ndjson 5\n./llm-tagger.sh hierarchical hw-l1.ndjson hw-final.ndjson 3\n```</p>\n\n<h2>Performance Changes</h2>\n\n<h3>1. Parallel Tag Extraction</h3>\n\n<ul><li>8 concurrent workers</li><li>removed sleep delays</li><li>8x faster</li></ul>\n\n<h3>2. Connection Pooling</h3>\n\n<ul><li>requests.Session() for reuse</li><li>all API calls</li></ul>\n\n<h3>3. Parallel Embedding</h3>\n\n<ul><li>4 concurrent workers</li><li>batch size: 32 → 64</li><li>maintains ordering</li></ul>\n\n<h3>4. HDBSCAN Multi-core</h3>\n\n<ul><li>core_dist_n_jobs=-1</li><li>uses all cores</li></ul>\n\n<p><b>Overall speedup: 5-8x</b></p>\n\n<h2>Two-Phase Architecture</h2>\n\n<h3>Phase 1: init total.ndjson</h3>\n\n<ul><li>pre-compute all embeddings</li><li>save to cache</li><li>no clustering yet</li></ul>\n\n<h3>Phase 2: educe-tags subset.ndjson</h3>\n\n<ul><li>load cached embeddings</li><li>cluster subset only</li><li>generate subset-specific tags</li><li>each subset: unique patterns</li></ul>\n\n<h3>Benefits</h3>\n\n<ul><li>embeddings computed once</li><li>each subset independently clustered</li><li>tags reflect subset context</li><li>january.ndjson → january patterns</li><li>hardware.ndjson → hardware categories</li></ul>\n\n<h2>Hierarchical Tagging</h2>\n\n<p>Create two-level taxonomy: fine-grained + broad categories.</p>\n\n<h3>Workflow</h3>\n\n<p>```bash</p><h1>level 1: fine-grained (many small clusters)</h1><p>./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5</p>\n\n<h1>level 2: cluster the clusters</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```</p>\n\n<h3>How It Works</h3>\n\n<ol><li><b>Level 1</b> creates detailed clusters (e.g., \"screws\", \"bolts\", \"cables\")</li><li><b>Level 2</b> computes cluster centroids</li><li>Clusters the centroids into super-clusters</li><li>Generates broad tags (e.g., \"hardware\", \"electronics\")</li></ol>\n\n<h3>Output Format</h3>\n\n<p>```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 5,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 1,\n  \"tags_l2\": \"hardware, tools\"\n}\n```</p>\n\n<h3>Use Cases</h3>\n\n<ul><li>multi-level taxonomy</li><li>drill-down navigation</li><li>broad overview + details</li></ul>\n\n<h2>Additional Tools</h2>\n\n<p>Beyond clustering, embeddings enable powerful analysis.</p>\n\n<h3>1. Semantic Search</h3>\n\n<p>Find documents matching query semantically.</p>\n\n<p>```bash\n./llm-tagger.sh search data.ndjson 'hardware repair' 10\n```</p>\n\n<p><b>Output:</b> Top matching documents with similarity scores</p>\n\n<p><b>Use cases:</b></p>\n\n<ul><li>find relevant documents</li><li>content discovery</li><li>query-based filtering</li></ul>\n\n<h3>2. Find Similar Documents</h3>\n\n<p>Detect duplicates or near-duplicates.</p>"},{"id":"/root/children/57","type":"paragraph","loc":{"start":4830,"end":4887,"line":{"s":220,"e":220,"code":["**Output:** Top matching documents with similarity scores"]},"column":{"s":0,"e":57}},"dim":["","paragraph.57"],"code":"**Output:** Top matching documents with similarity scores"},{"id":"/root/children/57/children/0","type":"strong","loc":{"start":4830,"end":4841,"line":{"s":220,"e":220,"code":["**Output:** Top matching documents with similarity scores"]},"column":{"s":0,"e":11}},"dim":["","paragraph.57","strong.0"],"code":"**Output:**"},{"id":"/root/children/57/children/0/children/0","type":"text","loc":{"start":4832,"end":4839,"line":{"s":220,"e":220,"code":["**Output:** Top matching documents with similarity scores"]},"column":{"s":2,"e":9}},"dim":["","paragraph.57","strong.0","text.0"],"code":"Output:"},{"id":"/root/children/57/children/1","type":"text","loc":{"start":4841,"end":4887,"line":{"s":220,"e":220,"code":["**Output:** Top matching documents with similarity scores"]},"column":{"s":11,"e":57}},"dim":["","paragraph.57","text.1"],"code":" Top matching documents with similarity scores"},{"id":"/root/children/58","type":"paragraph","loc":{"start":4889,"end":4903,"line":{"s":222,"e":222,"code":["**Use cases:**"]},"column":{"s":0,"e":14}},"dim":["","paragraph.58"],"code":"**Use cases:**"},{"id":"/root/children/58/children/0","type":"strong","loc":{"start":4889,"end":4903,"line":{"s":222,"e":222,"code":["**Use cases:**"]},"column":{"s":0,"e":14}},"dim":["","paragraph.58","strong.0"],"code":"**Use cases:**"},{"id":"/root/children/58/children/0/children/0","type":"text","loc":{"start":4891,"end":4901,"line":{"s":222,"e":222,"code":["**Use cases:**"]},"column":{"s":2,"e":12}},"dim":["","paragraph.58","strong.0","text.0"],"code":"Use cases:"},{"id":"/root/children/59","type":"list","loc":{"start":4905,"end":4974,"line":{"s":224,"e":226,"code":["- find relevant documents","- content discovery","- query-based filtering"]},"column":{"s":0,"e":23}},"dim":["","list.59"],"code":"- find relevant documents\n- content discovery\n- query-based filtering","symbName":"list","symbRange":[4976,5154],"symbRangeL":[224,239],"outerCode":"- content discovery\n- query-based filtering\n\n### 2. Find Similar Documents\n\nDetect duplicates or near-duplicates.\n\n```bash\n./llm-tagger.sh similar data.ndjson 0.95\n```\n\n**Output:** Pairs of similar documents\n\n**Use cases:**","outerHtml":"<ul><li>content discovery</li><li>query-based filtering</li></ul>\n\n<h3>2. Find Similar Documents</h3>\n\n<p>Detect duplicates or near-duplicates.</p>\n\n<p>```bash\n./llm-tagger.sh similar data.ndjson 0.95\n```</p>\n\n<p><b>Output:</b> Pairs of similar documents</p>\n\n<p><b>Use cases:</b></p>"},{"id":"/root/children/59/children/0","type":"listItem","loc":{"start":4905,"end":4930,"line":{"s":224,"e":224,"code":["- find relevant documents"]},"column":{"s":0,"e":25}},"dim":["","list.59","listItem.0"],"code":"- find relevant documents"},{"id":"/root/children/59/children/0/children/0","type":"paragraph","loc":{"start":4907,"end":4930,"line":{"s":224,"e":224,"code":["- find relevant documents"]},"column":{"s":2,"e":25}},"dim":["","list.59","listItem.0","paragraph.0"],"code":"find relevant documents"},{"id":"/root/children/59/children/0/children/0/children/0","type":"text","loc":{"start":4907,"end":4930,"line":{"s":224,"e":224,"code":["- find relevant documents"]},"column":{"s":2,"e":25}},"dim":["","list.59","listItem.0","paragraph.0","text.0"],"code":"find relevant documents"},{"id":"/root/children/59/children/1","type":"listItem","loc":{"start":4931,"end":4950,"line":{"s":225,"e":225,"code":["- content discovery"]},"column":{"s":0,"e":19}},"dim":["","list.59","listItem.1"],"code":"- content discovery"},{"id":"/root/children/59/children/1/children/0","type":"paragraph","loc":{"start":4933,"end":4950,"line":{"s":225,"e":225,"code":["- content discovery"]},"column":{"s":2,"e":19}},"dim":["","list.59","listItem.1","paragraph.0"],"code":"content discovery"},{"id":"/root/children/59/children/1/children/0/children/0","type":"text","loc":{"start":4933,"end":4950,"line":{"s":225,"e":225,"code":["- content discovery"]},"column":{"s":2,"e":19}},"dim":["","list.59","listItem.1","paragraph.0","text.0"],"code":"content discovery"},{"id":"/root/children/59/children/2","type":"listItem","loc":{"start":4951,"end":4974,"line":{"s":226,"e":226,"code":["- query-based filtering"]},"column":{"s":0,"e":23}},"dim":["","list.59","listItem.2"],"code":"- query-based filtering"},{"id":"/root/children/59/children/2/children/0","type":"paragraph","loc":{"start":4953,"end":4974,"line":{"s":226,"e":226,"code":["- query-based filtering"]},"column":{"s":2,"e":23}},"dim":["","list.59","listItem.2","paragraph.0"],"code":"query-based filtering"},{"id":"/root/children/59/children/2/children/0/children/0","type":"text","loc":{"start":4953,"end":4974,"line":{"s":226,"e":226,"code":["- query-based filtering"]},"column":{"s":2,"e":23}},"dim":["","list.59","listItem.2","paragraph.0","text.0"],"code":"query-based filtering"},{"id":"/root/children/60","type":"heading","loc":{"start":4976,"end":5005,"line":{"s":228,"e":228,"code":["### 2. Find Similar Documents"]},"column":{"s":0,"e":29}},"dim":["","heading.60"],"code":"### 2. Find Similar Documents","symbName":"heading","symbRange":[5007,5214],"symbRangeL":[228,243],"outerCode":"\nDetect duplicates or near-duplicates.\n\n```bash\n./llm-tagger.sh similar data.ndjson 0.95\n```\n\n**Output:** Pairs of similar documents\n\n**Use cases:**\n\n- duplicate detection\n- data quality\n- identify redundancy","outerHtml":"\n<p>Detect duplicates or near-duplicates.</p>\n\n<p>```bash\n./llm-tagger.sh similar data.ndjson 0.95\n```</p>\n\n<p><b>Output:</b> Pairs of similar documents</p>\n\n<p><b>Use cases:</b></p>\n\n<ul><li>duplicate detection</li><li>data quality</li><li>identify redundancy</li></ul>"},{"id":"/root/children/60/children/0","type":"text","loc":{"start":4980,"end":5005,"line":{"s":228,"e":228,"code":["### 2. Find Similar Documents"]},"column":{"s":4,"e":29}},"dim":["","heading.60","text.0"],"code":"2. Find Similar Documents"},{"id":"/root/children/61","type":"paragraph","loc":{"start":5007,"end":5044,"line":{"s":230,"e":230,"code":["Detect duplicates or near-duplicates."]},"column":{"s":0,"e":37}},"dim":["","paragraph.61"],"code":"Detect duplicates or near-duplicates."},{"id":"/root/children/61/children/0","type":"text","loc":{"start":5007,"end":5044,"line":{"s":230,"e":230,"code":["Detect duplicates or near-duplicates."]},"column":{"s":0,"e":37}},"dim":["","paragraph.61","text.0"],"code":"Detect duplicates or near-duplicates."},{"id":"/root/children/62","type":"code","loc":{"start":5046,"end":5098,"line":{"s":232,"e":234,"code":["```bash","./llm-tagger.sh similar data.ndjson 0.95","```"]},"column":{"s":0,"e":3}},"dim":["","code.62"],"code":"```bash\n./llm-tagger.sh similar data.ndjson 0.95\n```","symbName":"code","symbRange":[5100,5275],"symbRangeL":[null,247],"outerCode":";{ engine: dot rankdir:LR clusterize: true }\n\n# LLM Tagger\n\nDocument clustering and analysis using embeddings and LLM.\n\n## Quick Start\n\n```bash\ncd docker/llm-tagger\n\n# step 1: cache embeddings (always first)\n./llm-tagger.sh init total.ndjson\n\n# step 2: cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100\n\n# optional: hierarchical tagging\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n\n# optional: analysis tools\n./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```\n\n## All Commands\n\n| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |\n\n## Workflows\n\n### Workflow 1: Single-Level Tagging\n\nOne level of clusters with tags.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```\n\n**Result:** Documents with cluster and tags\n\n```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```\n\n### Workflow 2: Hierarchical Tagging\n\nTwo levels: fine-grained + broad categories.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. fine-grained clustering\n./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5\n\n# 3. cluster the clusters\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```\n\n**Result:** Documents with two levels\n\n```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 12,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 2,\n  \"tags_l2\": \"hardware\"\n}\n```\n\n### Workflow 3: Multiple Subsets\n\nAnalyze different slices independently.\n\n```bash\n# 1. cache embeddings (once for all)\n./llm-tagger.sh init total.ndjson\n\n# 2. tag each subset (different patterns)\n./llm-tagger.sh educe-tags january.ndjson jan-tagged.ndjson 15\n./llm-tagger.sh educe-tags hardware.ndjson hw-tagged.ndjson 20\n./llm-tagger.sh educe-tags vendor-x.ndjson vx-tagged.ndjson 10\n```\n\n**Benefit:** Each subset gets unique clustering reflecting its patterns\n\n### Workflow 4: Multiple Subsets + Hierarchy\n\nCombine both approaches.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. subset 1 with hierarchy\n./llm-tagger.sh educe-tags january.ndjson jan-l1.ndjson 5\n./llm-tagger.sh hierarchical jan-l1.ndjson jan-final.ndjson 3\n\n# 3. subset 2 with hierarchy\n./llm-tagger.sh educe-tags hardware.ndjson hw-l1.ndjson 5\n./llm-tagger.sh hierarchical hw-l1.ndjson hw-final.ndjson 3\n```\n\n## Performance Changes\n\n### 1. Parallel Tag Extraction\n\n- 8 concurrent workers\n- removed sleep delays\n- 8x faster\n\n### 2. Connection Pooling\n\n- requests.Session() for reuse\n- all API calls\n\n### 3. Parallel Embedding\n\n- 4 concurrent workers\n- batch size: 32 → 64\n- maintains ordering\n\n### 4. HDBSCAN Multi-core\n\n- core_dist_n_jobs=-1\n- uses all cores\n\n**Overall speedup: 5-8x**\n\n## Two-Phase Architecture\n\n### Phase 1: init total.ndjson\n\n- pre-compute all embeddings\n- save to cache\n- no clustering yet\n\n### Phase 2: educe-tags subset.ndjson\n\n- load cached embeddings\n- cluster subset only\n- generate subset-specific tags\n- each subset: unique patterns\n\n### Benefits\n\n- embeddings computed once\n- each subset independently clustered\n- tags reflect subset context\n- january.ndjson → january patterns\n- hardware.ndjson → hardware categories\n\n## Hierarchical Tagging\n\nCreate two-level taxonomy: fine-grained + broad categories.\n\n### Workflow\n\n```bash\n# level 1: fine-grained (many small clusters)\n./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5\n\n# level 2: cluster the clusters\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```\n\n### How It Works\n\n1. **Level 1** creates detailed clusters (e.g., \"screws\", \"bolts\", \"cables\")\n1. **Level 2** computes cluster centroids\n1. Clusters the centroids into super-clusters\n1. Generates broad tags (e.g., \"hardware\", \"electronics\")\n\n### Output Format\n\n```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 5,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 1,\n  \"tags_l2\": \"hardware, tools\"\n}\n```\n\n### Use Cases\n\n- multi-level taxonomy\n- drill-down navigation\n- broad overview + details\n\n## Additional Tools\n\nBeyond clustering, embeddings enable powerful analysis.\n\n### 1. Semantic Search\n\nFind documents matching query semantically.\n\n```bash\n./llm-tagger.sh search data.ndjson 'hardware repair' 10\n```\n\n**Output:** Top matching documents with similarity scores\n\n**Use cases:**\n\n- find relevant documents\n- content discovery\n- query-based filtering\n\n### 2. Find Similar Documents\n\nDetect duplicates or near-duplicates.\n\n```bash\n./llm-tagger.sh similar data.ndjson 0.95\n```\n\n**Output:** Pairs of similar documents\n\n**Use cases:**\n\n- duplicate detection\n- data quality\n- identify redundancy\n\n### 3. Custom Prompts\n\nAsk different questions to clusters.","outerHtml":"<p>;{ engine: dot rankdir:LR clusterize: true }</p>\n\n<h1>LLM Tagger</h1>\n\n<p>Document clustering and analysis using embeddings and LLM.</p>\n\n<h2>Quick Start</h2>\n\n<p>```bash\ncd docker/llm-tagger</p>\n\n<h1>step 1: cache embeddings (always first)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>step 2: cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100</p>\n\n<h1>optional: hierarchical tagging</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3</p>\n\n<h1>optional: analysis tools</h1><p>./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```</p>\n\n<h2>All Commands</h2>\n\n<p>| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |</p>\n\n<h2>Workflows</h2>\n\n<h3>Workflow 1: Single-Level Tagging</h3>\n\n<p>One level of clusters with tags.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```</p>\n\n<p><b>Result:</b> Documents with cluster and tags</p>\n\n<p>```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```</p>\n\n<h3>Workflow 2: Hierarchical Tagging</h3>\n\n<p>Two levels: fine-grained + broad categories.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. fine-grained clustering</h1><p>./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5</p>\n\n<h1>3. cluster the clusters</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```</p>\n\n<p><b>Result:</b> Documents with two levels</p>\n\n<p>```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 12,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 2,\n  \"tags_l2\": \"hardware\"\n}\n```</p>\n\n<h3>Workflow 3: Multiple Subsets</h3>\n\n<p>Analyze different slices independently.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once for all)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. tag each subset (different patterns)</h1><p>./llm-tagger.sh educe-tags january.ndjson jan-tagged.ndjson 15\n./llm-tagger.sh educe-tags hardware.ndjson hw-tagged.ndjson 20\n./llm-tagger.sh educe-tags vendor-x.ndjson vx-tagged.ndjson 10\n```</p>\n\n<p><b>Benefit:</b> Each subset gets unique clustering reflecting its patterns</p>\n\n<h3>Workflow 4: Multiple Subsets + Hierarchy</h3>\n\n<p>Combine both approaches.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. subset 1 with hierarchy</h1><p>./llm-tagger.sh educe-tags january.ndjson jan-l1.ndjson 5\n./llm-tagger.sh hierarchical jan-l1.ndjson jan-final.ndjson 3</p>\n\n<h1>3. subset 2 with hierarchy</h1><p>./llm-tagger.sh educe-tags hardware.ndjson hw-l1.ndjson 5\n./llm-tagger.sh hierarchical hw-l1.ndjson hw-final.ndjson 3\n```</p>\n\n<h2>Performance Changes</h2>\n\n<h3>1. Parallel Tag Extraction</h3>\n\n<ul><li>8 concurrent workers</li><li>removed sleep delays</li><li>8x faster</li></ul>\n\n<h3>2. Connection Pooling</h3>\n\n<ul><li>requests.Session() for reuse</li><li>all API calls</li></ul>\n\n<h3>3. Parallel Embedding</h3>\n\n<ul><li>4 concurrent workers</li><li>batch size: 32 → 64</li><li>maintains ordering</li></ul>\n\n<h3>4. HDBSCAN Multi-core</h3>\n\n<ul><li>core_dist_n_jobs=-1</li><li>uses all cores</li></ul>\n\n<p><b>Overall speedup: 5-8x</b></p>\n\n<h2>Two-Phase Architecture</h2>\n\n<h3>Phase 1: init total.ndjson</h3>\n\n<ul><li>pre-compute all embeddings</li><li>save to cache</li><li>no clustering yet</li></ul>\n\n<h3>Phase 2: educe-tags subset.ndjson</h3>\n\n<ul><li>load cached embeddings</li><li>cluster subset only</li><li>generate subset-specific tags</li><li>each subset: unique patterns</li></ul>\n\n<h3>Benefits</h3>\n\n<ul><li>embeddings computed once</li><li>each subset independently clustered</li><li>tags reflect subset context</li><li>january.ndjson → january patterns</li><li>hardware.ndjson → hardware categories</li></ul>\n\n<h2>Hierarchical Tagging</h2>\n\n<p>Create two-level taxonomy: fine-grained + broad categories.</p>\n\n<h3>Workflow</h3>\n\n<p>```bash</p><h1>level 1: fine-grained (many small clusters)</h1><p>./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5</p>\n\n<h1>level 2: cluster the clusters</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```</p>\n\n<h3>How It Works</h3>\n\n<ol><li><b>Level 1</b> creates detailed clusters (e.g., \"screws\", \"bolts\", \"cables\")</li><li><b>Level 2</b> computes cluster centroids</li><li>Clusters the centroids into super-clusters</li><li>Generates broad tags (e.g., \"hardware\", \"electronics\")</li></ol>\n\n<h3>Output Format</h3>\n\n<p>```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 5,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 1,\n  \"tags_l2\": \"hardware, tools\"\n}\n```</p>\n\n<h3>Use Cases</h3>\n\n<ul><li>multi-level taxonomy</li><li>drill-down navigation</li><li>broad overview + details</li></ul>\n\n<h2>Additional Tools</h2>\n\n<p>Beyond clustering, embeddings enable powerful analysis.</p>\n\n<h3>1. Semantic Search</h3>\n\n<p>Find documents matching query semantically.</p>\n\n<p>```bash\n./llm-tagger.sh search data.ndjson 'hardware repair' 10\n```</p>\n\n<p><b>Output:</b> Top matching documents with similarity scores</p>\n\n<p><b>Use cases:</b></p>\n\n<ul><li>find relevant documents</li><li>content discovery</li><li>query-based filtering</li></ul>\n\n<h3>2. Find Similar Documents</h3>\n\n<p>Detect duplicates or near-duplicates.</p>\n\n<p>```bash\n./llm-tagger.sh similar data.ndjson 0.95\n```</p>\n\n<p><b>Output:</b> Pairs of similar documents</p>\n\n<p><b>Use cases:</b></p>\n\n<ul><li>duplicate detection</li><li>data quality</li><li>identify redundancy</li></ul>\n\n<h3>3. Custom Prompts</h3>\n\n<p>Ask different questions to clusters.</p>"},{"id":"/root/children/63","type":"paragraph","loc":{"start":5100,"end":5138,"line":{"s":236,"e":236,"code":["**Output:** Pairs of similar documents"]},"column":{"s":0,"e":38}},"dim":["","paragraph.63"],"code":"**Output:** Pairs of similar documents"},{"id":"/root/children/63/children/0","type":"strong","loc":{"start":5100,"end":5111,"line":{"s":236,"e":236,"code":["**Output:** Pairs of similar documents"]},"column":{"s":0,"e":11}},"dim":["","paragraph.63","strong.0"],"code":"**Output:**"},{"id":"/root/children/63/children/0/children/0","type":"text","loc":{"start":5102,"end":5109,"line":{"s":236,"e":236,"code":["**Output:** Pairs of similar documents"]},"column":{"s":2,"e":9}},"dim":["","paragraph.63","strong.0","text.0"],"code":"Output:"},{"id":"/root/children/63/children/1","type":"text","loc":{"start":5111,"end":5138,"line":{"s":236,"e":236,"code":["**Output:** Pairs of similar documents"]},"column":{"s":11,"e":38}},"dim":["","paragraph.63","text.1"],"code":" Pairs of similar documents"},{"id":"/root/children/64","type":"paragraph","loc":{"start":5140,"end":5154,"line":{"s":238,"e":238,"code":["**Use cases:**"]},"column":{"s":0,"e":14}},"dim":["","paragraph.64"],"code":"**Use cases:**"},{"id":"/root/children/64/children/0","type":"strong","loc":{"start":5140,"end":5154,"line":{"s":238,"e":238,"code":["**Use cases:**"]},"column":{"s":0,"e":14}},"dim":["","paragraph.64","strong.0"],"code":"**Use cases:**"},{"id":"/root/children/64/children/0/children/0","type":"text","loc":{"start":5142,"end":5152,"line":{"s":238,"e":238,"code":["**Use cases:**"]},"column":{"s":2,"e":12}},"dim":["","paragraph.64","strong.0","text.0"],"code":"Use cases:"},{"id":"/root/children/65","type":"list","loc":{"start":5156,"end":5214,"line":{"s":240,"e":242,"code":["- duplicate detection","- data quality","- identify redundancy"]},"column":{"s":0,"e":21}},"dim":["","list.65"],"code":"- duplicate detection\n- data quality\n- identify redundancy","symbName":"list","symbRange":[5216,5386],"symbRangeL":[240,253],"outerCode":"- data quality\n- identify redundancy\n\n### 3. Custom Prompts\n\nAsk different questions to clusters.\n\n```bash\n./llm-tagger.sh custom-prompt tagged.ndjson 'Zhrň hlavné témy:\\n{examples}'\n```\n\n**Prompt examples:**","outerHtml":"<ul><li>data quality</li><li>identify redundancy</li></ul>\n\n<h3>3. Custom Prompts</h3>\n\n<p>Ask different questions to clusters.</p>\n\n<p>```bash\n./llm-tagger.sh custom-prompt tagged.ndjson 'Zhrň hlavné témy:\\n{examples}'\n```</p>\n\n<p><b>Prompt examples:</b></p>"},{"id":"/root/children/65/children/0","type":"listItem","loc":{"start":5156,"end":5177,"line":{"s":240,"e":240,"code":["- duplicate detection"]},"column":{"s":0,"e":21}},"dim":["","list.65","listItem.0"],"code":"- duplicate detection"},{"id":"/root/children/65/children/0/children/0","type":"paragraph","loc":{"start":5158,"end":5177,"line":{"s":240,"e":240,"code":["- duplicate detection"]},"column":{"s":2,"e":21}},"dim":["","list.65","listItem.0","paragraph.0"],"code":"duplicate detection"},{"id":"/root/children/65/children/0/children/0/children/0","type":"text","loc":{"start":5158,"end":5177,"line":{"s":240,"e":240,"code":["- duplicate detection"]},"column":{"s":2,"e":21}},"dim":["","list.65","listItem.0","paragraph.0","text.0"],"code":"duplicate detection"},{"id":"/root/children/65/children/1","type":"listItem","loc":{"start":5178,"end":5192,"line":{"s":241,"e":241,"code":["- data quality"]},"column":{"s":0,"e":14}},"dim":["","list.65","listItem.1"],"code":"- data quality"},{"id":"/root/children/65/children/1/children/0","type":"paragraph","loc":{"start":5180,"end":5192,"line":{"s":241,"e":241,"code":["- data quality"]},"column":{"s":2,"e":14}},"dim":["","list.65","listItem.1","paragraph.0"],"code":"data quality"},{"id":"/root/children/65/children/1/children/0/children/0","type":"text","loc":{"start":5180,"end":5192,"line":{"s":241,"e":241,"code":["- data quality"]},"column":{"s":2,"e":14}},"dim":["","list.65","listItem.1","paragraph.0","text.0"],"code":"data quality"},{"id":"/root/children/65/children/2","type":"listItem","loc":{"start":5193,"end":5214,"line":{"s":242,"e":242,"code":["- identify redundancy"]},"column":{"s":0,"e":21}},"dim":["","list.65","listItem.2"],"code":"- identify redundancy"},{"id":"/root/children/65/children/2/children/0","type":"paragraph","loc":{"start":5195,"end":5214,"line":{"s":242,"e":242,"code":["- identify redundancy"]},"column":{"s":2,"e":21}},"dim":["","list.65","listItem.2","paragraph.0"],"code":"identify redundancy"},{"id":"/root/children/65/children/2/children/0/children/0","type":"text","loc":{"start":5195,"end":5214,"line":{"s":242,"e":242,"code":["- identify redundancy"]},"column":{"s":2,"e":21}},"dim":["","list.65","listItem.2","paragraph.0","text.0"],"code":"identify redundancy"},{"id":"/root/children/66","type":"heading","loc":{"start":5216,"end":5237,"line":{"s":244,"e":244,"code":["### 3. Custom Prompts"]},"column":{"s":0,"e":21}},"dim":["","heading.66"],"code":"### 3. Custom Prompts","symbName":"heading","symbRange":[5239,5671],"symbRangeL":[244,264],"outerCode":"\nAsk different questions to clusters.\n\n```bash\n./llm-tagger.sh custom-prompt tagged.ndjson 'Zhrň hlavné témy:\\n{examples}'\n```\n\n**Prompt examples:**\n\n- `'Zhrň hlavné témy:\\n{examples}'` - summaries\n- `'Identifikuj riziká:\\n{examples}'` - risk analysis\n- `'Aké patterny vidíš:\\n{examples}'` - insights\n- `'Prečo sú zoskupené:\\n{examples}'` - reasoning\n\n**Use cases:**\n\n- different perspectives\n- specialized analysis\n- custom insights","outerHtml":"\n<p>Ask different questions to clusters.</p>\n\n<p>```bash\n./llm-tagger.sh custom-prompt tagged.ndjson 'Zhrň hlavné témy:\\n{examples}'\n```</p>\n\n<p><b>Prompt examples:</b></p>\n\n<ul><li>`'Zhrň hlavné témy:\\n{examples}'` - summaries</li><li>`'Identifikuj riziká:\\n{examples}'` - risk analysis</li><li>`'Aké patterny vidíš:\\n{examples}'` - insights</li><li>`'Prečo sú zoskupené:\\n{examples}'` - reasoning</li></ul>\n\n<p><b>Use cases:</b></p>\n\n<ul><li>different perspectives</li><li>specialized analysis</li><li>custom insights</li></ul>"},{"id":"/root/children/66/children/0","type":"text","loc":{"start":5220,"end":5237,"line":{"s":244,"e":244,"code":["### 3. Custom Prompts"]},"column":{"s":4,"e":21}},"dim":["","heading.66","text.0"],"code":"3. Custom Prompts"},{"id":"/root/children/67","type":"paragraph","loc":{"start":5239,"end":5275,"line":{"s":246,"e":246,"code":["Ask different questions to clusters."]},"column":{"s":0,"e":36}},"dim":["","paragraph.67"],"code":"Ask different questions to clusters."},{"id":"/root/children/67/children/0","type":"text","loc":{"start":5239,"end":5275,"line":{"s":246,"e":246,"code":["Ask different questions to clusters."]},"column":{"s":0,"e":36}},"dim":["","paragraph.67","text.0"],"code":"Ask different questions to clusters."},{"id":"/root/children/68","type":"code","loc":{"start":5277,"end":5364,"line":{"s":248,"e":250,"code":["```bash","./llm-tagger.sh custom-prompt tagged.ndjson 'Zhrň hlavné témy:\\n{examples}'","```"]},"column":{"s":0,"e":3}},"dim":["","code.68"],"code":"```bash\n./llm-tagger.sh custom-prompt tagged.ndjson 'Zhrň hlavné témy:\\n{examples}'\n```","symbName":"code","symbRange":[5366,5749],"symbRangeL":[null,268],"outerCode":";{ engine: dot rankdir:LR clusterize: true }\n\n# LLM Tagger\n\nDocument clustering and analysis using embeddings and LLM.\n\n## Quick Start\n\n```bash\ncd docker/llm-tagger\n\n# step 1: cache embeddings (always first)\n./llm-tagger.sh init total.ndjson\n\n# step 2: cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100\n\n# optional: hierarchical tagging\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n\n# optional: analysis tools\n./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```\n\n## All Commands\n\n| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |\n\n## Workflows\n\n### Workflow 1: Single-Level Tagging\n\nOne level of clusters with tags.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```\n\n**Result:** Documents with cluster and tags\n\n```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```\n\n### Workflow 2: Hierarchical Tagging\n\nTwo levels: fine-grained + broad categories.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. fine-grained clustering\n./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5\n\n# 3. cluster the clusters\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```\n\n**Result:** Documents with two levels\n\n```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 12,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 2,\n  \"tags_l2\": \"hardware\"\n}\n```\n\n### Workflow 3: Multiple Subsets\n\nAnalyze different slices independently.\n\n```bash\n# 1. cache embeddings (once for all)\n./llm-tagger.sh init total.ndjson\n\n# 2. tag each subset (different patterns)\n./llm-tagger.sh educe-tags january.ndjson jan-tagged.ndjson 15\n./llm-tagger.sh educe-tags hardware.ndjson hw-tagged.ndjson 20\n./llm-tagger.sh educe-tags vendor-x.ndjson vx-tagged.ndjson 10\n```\n\n**Benefit:** Each subset gets unique clustering reflecting its patterns\n\n### Workflow 4: Multiple Subsets + Hierarchy\n\nCombine both approaches.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. subset 1 with hierarchy\n./llm-tagger.sh educe-tags january.ndjson jan-l1.ndjson 5\n./llm-tagger.sh hierarchical jan-l1.ndjson jan-final.ndjson 3\n\n# 3. subset 2 with hierarchy\n./llm-tagger.sh educe-tags hardware.ndjson hw-l1.ndjson 5\n./llm-tagger.sh hierarchical hw-l1.ndjson hw-final.ndjson 3\n```\n\n## Performance Changes\n\n### 1. Parallel Tag Extraction\n\n- 8 concurrent workers\n- removed sleep delays\n- 8x faster\n\n### 2. Connection Pooling\n\n- requests.Session() for reuse\n- all API calls\n\n### 3. Parallel Embedding\n\n- 4 concurrent workers\n- batch size: 32 → 64\n- maintains ordering\n\n### 4. HDBSCAN Multi-core\n\n- core_dist_n_jobs=-1\n- uses all cores\n\n**Overall speedup: 5-8x**\n\n## Two-Phase Architecture\n\n### Phase 1: init total.ndjson\n\n- pre-compute all embeddings\n- save to cache\n- no clustering yet\n\n### Phase 2: educe-tags subset.ndjson\n\n- load cached embeddings\n- cluster subset only\n- generate subset-specific tags\n- each subset: unique patterns\n\n### Benefits\n\n- embeddings computed once\n- each subset independently clustered\n- tags reflect subset context\n- january.ndjson → january patterns\n- hardware.ndjson → hardware categories\n\n## Hierarchical Tagging\n\nCreate two-level taxonomy: fine-grained + broad categories.\n\n### Workflow\n\n```bash\n# level 1: fine-grained (many small clusters)\n./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5\n\n# level 2: cluster the clusters\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```\n\n### How It Works\n\n1. **Level 1** creates detailed clusters (e.g., \"screws\", \"bolts\", \"cables\")\n1. **Level 2** computes cluster centroids\n1. Clusters the centroids into super-clusters\n1. Generates broad tags (e.g., \"hardware\", \"electronics\")\n\n### Output Format\n\n```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 5,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 1,\n  \"tags_l2\": \"hardware, tools\"\n}\n```\n\n### Use Cases\n\n- multi-level taxonomy\n- drill-down navigation\n- broad overview + details\n\n## Additional Tools\n\nBeyond clustering, embeddings enable powerful analysis.\n\n### 1. Semantic Search\n\nFind documents matching query semantically.\n\n```bash\n./llm-tagger.sh search data.ndjson 'hardware repair' 10\n```\n\n**Output:** Top matching documents with similarity scores\n\n**Use cases:**\n\n- find relevant documents\n- content discovery\n- query-based filtering\n\n### 2. Find Similar Documents\n\nDetect duplicates or near-duplicates.\n\n```bash\n./llm-tagger.sh similar data.ndjson 0.95\n```\n\n**Output:** Pairs of similar documents\n\n**Use cases:**\n\n- duplicate detection\n- data quality\n- identify redundancy\n\n### 3. Custom Prompts\n\nAsk different questions to clusters.\n\n```bash\n./llm-tagger.sh custom-prompt tagged.ndjson 'Zhrň hlavné témy:\\n{examples}'\n```\n\n**Prompt examples:**\n\n- `'Zhrň hlavné témy:\\n{examples}'` - summaries\n- `'Identifikuj riziká:\\n{examples}'` - risk analysis\n- `'Aké patterny vidíš:\\n{examples}'` - insights\n- `'Prečo sú zoskupené:\\n{examples}'` - reasoning\n\n**Use cases:**\n\n- different perspectives\n- specialized analysis\n- custom insights\n\n### 4. Visualize Embeddings\n\nInteractive 2D visualization of document space.","outerHtml":"<p>;{ engine: dot rankdir:LR clusterize: true }</p>\n\n<h1>LLM Tagger</h1>\n\n<p>Document clustering and analysis using embeddings and LLM.</p>\n\n<h2>Quick Start</h2>\n\n<p>```bash\ncd docker/llm-tagger</p>\n\n<h1>step 1: cache embeddings (always first)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>step 2: cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100</p>\n\n<h1>optional: hierarchical tagging</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3</p>\n\n<h1>optional: analysis tools</h1><p>./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```</p>\n\n<h2>All Commands</h2>\n\n<p>| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |</p>\n\n<h2>Workflows</h2>\n\n<h3>Workflow 1: Single-Level Tagging</h3>\n\n<p>One level of clusters with tags.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```</p>\n\n<p><b>Result:</b> Documents with cluster and tags</p>\n\n<p>```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```</p>\n\n<h3>Workflow 2: Hierarchical Tagging</h3>\n\n<p>Two levels: fine-grained + broad categories.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. fine-grained clustering</h1><p>./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5</p>\n\n<h1>3. cluster the clusters</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```</p>\n\n<p><b>Result:</b> Documents with two levels</p>\n\n<p>```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 12,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 2,\n  \"tags_l2\": \"hardware\"\n}\n```</p>\n\n<h3>Workflow 3: Multiple Subsets</h3>\n\n<p>Analyze different slices independently.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once for all)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. tag each subset (different patterns)</h1><p>./llm-tagger.sh educe-tags january.ndjson jan-tagged.ndjson 15\n./llm-tagger.sh educe-tags hardware.ndjson hw-tagged.ndjson 20\n./llm-tagger.sh educe-tags vendor-x.ndjson vx-tagged.ndjson 10\n```</p>\n\n<p><b>Benefit:</b> Each subset gets unique clustering reflecting its patterns</p>\n\n<h3>Workflow 4: Multiple Subsets + Hierarchy</h3>\n\n<p>Combine both approaches.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. subset 1 with hierarchy</h1><p>./llm-tagger.sh educe-tags january.ndjson jan-l1.ndjson 5\n./llm-tagger.sh hierarchical jan-l1.ndjson jan-final.ndjson 3</p>\n\n<h1>3. subset 2 with hierarchy</h1><p>./llm-tagger.sh educe-tags hardware.ndjson hw-l1.ndjson 5\n./llm-tagger.sh hierarchical hw-l1.ndjson hw-final.ndjson 3\n```</p>\n\n<h2>Performance Changes</h2>\n\n<h3>1. Parallel Tag Extraction</h3>\n\n<ul><li>8 concurrent workers</li><li>removed sleep delays</li><li>8x faster</li></ul>\n\n<h3>2. Connection Pooling</h3>\n\n<ul><li>requests.Session() for reuse</li><li>all API calls</li></ul>\n\n<h3>3. Parallel Embedding</h3>\n\n<ul><li>4 concurrent workers</li><li>batch size: 32 → 64</li><li>maintains ordering</li></ul>\n\n<h3>4. HDBSCAN Multi-core</h3>\n\n<ul><li>core_dist_n_jobs=-1</li><li>uses all cores</li></ul>\n\n<p><b>Overall speedup: 5-8x</b></p>\n\n<h2>Two-Phase Architecture</h2>\n\n<h3>Phase 1: init total.ndjson</h3>\n\n<ul><li>pre-compute all embeddings</li><li>save to cache</li><li>no clustering yet</li></ul>\n\n<h3>Phase 2: educe-tags subset.ndjson</h3>\n\n<ul><li>load cached embeddings</li><li>cluster subset only</li><li>generate subset-specific tags</li><li>each subset: unique patterns</li></ul>\n\n<h3>Benefits</h3>\n\n<ul><li>embeddings computed once</li><li>each subset independently clustered</li><li>tags reflect subset context</li><li>january.ndjson → january patterns</li><li>hardware.ndjson → hardware categories</li></ul>\n\n<h2>Hierarchical Tagging</h2>\n\n<p>Create two-level taxonomy: fine-grained + broad categories.</p>\n\n<h3>Workflow</h3>\n\n<p>```bash</p><h1>level 1: fine-grained (many small clusters)</h1><p>./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5</p>\n\n<h1>level 2: cluster the clusters</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```</p>\n\n<h3>How It Works</h3>\n\n<ol><li><b>Level 1</b> creates detailed clusters (e.g., \"screws\", \"bolts\", \"cables\")</li><li><b>Level 2</b> computes cluster centroids</li><li>Clusters the centroids into super-clusters</li><li>Generates broad tags (e.g., \"hardware\", \"electronics\")</li></ol>\n\n<h3>Output Format</h3>\n\n<p>```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 5,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 1,\n  \"tags_l2\": \"hardware, tools\"\n}\n```</p>\n\n<h3>Use Cases</h3>\n\n<ul><li>multi-level taxonomy</li><li>drill-down navigation</li><li>broad overview + details</li></ul>\n\n<h2>Additional Tools</h2>\n\n<p>Beyond clustering, embeddings enable powerful analysis.</p>\n\n<h3>1. Semantic Search</h3>\n\n<p>Find documents matching query semantically.</p>\n\n<p>```bash\n./llm-tagger.sh search data.ndjson 'hardware repair' 10\n```</p>\n\n<p><b>Output:</b> Top matching documents with similarity scores</p>\n\n<p><b>Use cases:</b></p>\n\n<ul><li>find relevant documents</li><li>content discovery</li><li>query-based filtering</li></ul>\n\n<h3>2. Find Similar Documents</h3>\n\n<p>Detect duplicates or near-duplicates.</p>\n\n<p>```bash\n./llm-tagger.sh similar data.ndjson 0.95\n```</p>\n\n<p><b>Output:</b> Pairs of similar documents</p>\n\n<p><b>Use cases:</b></p>\n\n<ul><li>duplicate detection</li><li>data quality</li><li>identify redundancy</li></ul>\n\n<h3>3. Custom Prompts</h3>\n\n<p>Ask different questions to clusters.</p>\n\n<p>```bash\n./llm-tagger.sh custom-prompt tagged.ndjson 'Zhrň hlavné témy:\\n{examples}'\n```</p>\n\n<p><b>Prompt examples:</b></p>\n\n<ul><li>`'Zhrň hlavné témy:\\n{examples}'` - summaries</li><li>`'Identifikuj riziká:\\n{examples}'` - risk analysis</li><li>`'Aké patterny vidíš:\\n{examples}'` - insights</li><li>`'Prečo sú zoskupené:\\n{examples}'` - reasoning</li></ul>\n\n<p><b>Use cases:</b></p>\n\n<ul><li>different perspectives</li><li>specialized analysis</li><li>custom insights</li></ul>\n\n<h3>4. Visualize Embeddings</h3>\n\n<p>Interactive 2D visualization of document space.</p>"},{"id":"/root/children/69","type":"paragraph","loc":{"start":5366,"end":5386,"line":{"s":252,"e":252,"code":["**Prompt examples:**"]},"column":{"s":0,"e":20}},"dim":["","paragraph.69"],"code":"**Prompt examples:**"},{"id":"/root/children/69/children/0","type":"strong","loc":{"start":5366,"end":5386,"line":{"s":252,"e":252,"code":["**Prompt examples:**"]},"column":{"s":0,"e":20}},"dim":["","paragraph.69","strong.0"],"code":"**Prompt examples:**"},{"id":"/root/children/69/children/0/children/0","type":"text","loc":{"start":5368,"end":5384,"line":{"s":252,"e":252,"code":["**Prompt examples:**"]},"column":{"s":2,"e":18}},"dim":["","paragraph.69","strong.0","text.0"],"code":"Prompt examples:"},{"id":"/root/children/70","type":"list","loc":{"start":5388,"end":5588,"line":{"s":254,"e":257,"code":["- `'Zhrň hlavné témy:\\n{examples}'` - summaries","- `'Identifikuj riziká:\\n{examples}'` - risk analysis","- `'Aké patterny vidíš:\\n{examples}'` - insights","- `'Prečo sú zoskupené:\\n{examples}'` - reasoning"]},"column":{"s":0,"e":49}},"dim":["","list.70"],"code":"- `'Zhrň hlavné témy:\\n{examples}'` - summaries\n- `'Identifikuj riziká:\\n{examples}'` - risk analysis\n- `'Aké patterny vidíš:\\n{examples}'` - insights\n- `'Prečo sú zoskupené:\\n{examples}'` - reasoning","symbName":"list","symbRange":[5590,5604],"symbRangeL":[254,260],"outerCode":"- `'Identifikuj riziká:\\n{examples}'` - risk analysis\n- `'Aké patterny vidíš:\\n{examples}'` - insights\n- `'Prečo sú zoskupené:\\n{examples}'` - reasoning\n\n**Use cases:**","outerHtml":"<ul><li>`'Identifikuj riziká:\\n{examples}'` - risk analysis</li><li>`'Aké patterny vidíš:\\n{examples}'` - insights</li><li>`'Prečo sú zoskupené:\\n{examples}'` - reasoning</li></ul>\n\n<p><b>Use cases:</b></p>"},{"id":"/root/children/70/children/0","type":"listItem","loc":{"start":5388,"end":5435,"line":{"s":254,"e":254,"code":["- `'Zhrň hlavné témy:\\n{examples}'` - summaries"]},"column":{"s":0,"e":47}},"dim":["","list.70","listItem.0"],"code":"- `'Zhrň hlavné témy:\\n{examples}'` - summaries"},{"id":"/root/children/70/children/0/children/0","type":"paragraph","loc":{"start":5390,"end":5435,"line":{"s":254,"e":254,"code":["- `'Zhrň hlavné témy:\\n{examples}'` - summaries"]},"column":{"s":2,"e":47}},"dim":["","list.70","listItem.0","paragraph.0"],"code":"`'Zhrň hlavné témy:\\n{examples}'` - summaries"},{"id":"/root/children/70/children/0/children/0/children/0","type":"inlineCode","loc":{"start":5390,"end":5423,"line":{"s":254,"e":254,"code":["- `'Zhrň hlavné témy:\\n{examples}'` - summaries"]},"column":{"s":2,"e":35}},"dim":["","list.70","listItem.0","paragraph.0","inlineCode.0"],"code":"`'Zhrň hlavné témy:\\n{examples}'`"},{"id":"/root/children/70/children/0/children/0/children/1","type":"text","loc":{"start":5423,"end":5435,"line":{"s":254,"e":254,"code":["- `'Zhrň hlavné témy:\\n{examples}'` - summaries"]},"column":{"s":35,"e":47}},"dim":["","list.70","listItem.0","paragraph.0","text.1"],"code":" - summaries"},{"id":"/root/children/70/children/1","type":"listItem","loc":{"start":5436,"end":5489,"line":{"s":255,"e":255,"code":["- `'Identifikuj riziká:\\n{examples}'` - risk analysis"]},"column":{"s":0,"e":53}},"dim":["","list.70","listItem.1"],"code":"- `'Identifikuj riziká:\\n{examples}'` - risk analysis"},{"id":"/root/children/70/children/1/children/0","type":"paragraph","loc":{"start":5438,"end":5489,"line":{"s":255,"e":255,"code":["- `'Identifikuj riziká:\\n{examples}'` - risk analysis"]},"column":{"s":2,"e":53}},"dim":["","list.70","listItem.1","paragraph.0"],"code":"`'Identifikuj riziká:\\n{examples}'` - risk analysis"},{"id":"/root/children/70/children/1/children/0/children/0","type":"inlineCode","loc":{"start":5438,"end":5473,"line":{"s":255,"e":255,"code":["- `'Identifikuj riziká:\\n{examples}'` - risk analysis"]},"column":{"s":2,"e":37}},"dim":["","list.70","listItem.1","paragraph.0","inlineCode.0"],"code":"`'Identifikuj riziká:\\n{examples}'`"},{"id":"/root/children/70/children/1/children/0/children/1","type":"text","loc":{"start":5473,"end":5489,"line":{"s":255,"e":255,"code":["- `'Identifikuj riziká:\\n{examples}'` - risk analysis"]},"column":{"s":37,"e":53}},"dim":["","list.70","listItem.1","paragraph.0","text.1"],"code":" - risk analysis"},{"id":"/root/children/70/children/2","type":"listItem","loc":{"start":5490,"end":5538,"line":{"s":256,"e":256,"code":["- `'Aké patterny vidíš:\\n{examples}'` - insights"]},"column":{"s":0,"e":48}},"dim":["","list.70","listItem.2"],"code":"- `'Aké patterny vidíš:\\n{examples}'` - insights"},{"id":"/root/children/70/children/2/children/0","type":"paragraph","loc":{"start":5492,"end":5538,"line":{"s":256,"e":256,"code":["- `'Aké patterny vidíš:\\n{examples}'` - insights"]},"column":{"s":2,"e":48}},"dim":["","list.70","listItem.2","paragraph.0"],"code":"`'Aké patterny vidíš:\\n{examples}'` - insights"},{"id":"/root/children/70/children/2/children/0/children/0","type":"inlineCode","loc":{"start":5492,"end":5527,"line":{"s":256,"e":256,"code":["- `'Aké patterny vidíš:\\n{examples}'` - insights"]},"column":{"s":2,"e":37}},"dim":["","list.70","listItem.2","paragraph.0","inlineCode.0"],"code":"`'Aké patterny vidíš:\\n{examples}'`"},{"id":"/root/children/70/children/2/children/0/children/1","type":"text","loc":{"start":5527,"end":5538,"line":{"s":256,"e":256,"code":["- `'Aké patterny vidíš:\\n{examples}'` - insights"]},"column":{"s":37,"e":48}},"dim":["","list.70","listItem.2","paragraph.0","text.1"],"code":" - insights"},{"id":"/root/children/70/children/3","type":"listItem","loc":{"start":5539,"end":5588,"line":{"s":257,"e":257,"code":["- `'Prečo sú zoskupené:\\n{examples}'` - reasoning"]},"column":{"s":0,"e":49}},"dim":["","list.70","listItem.3"],"code":"- `'Prečo sú zoskupené:\\n{examples}'` - reasoning"},{"id":"/root/children/70/children/3/children/0","type":"paragraph","loc":{"start":5541,"end":5588,"line":{"s":257,"e":257,"code":["- `'Prečo sú zoskupené:\\n{examples}'` - reasoning"]},"column":{"s":2,"e":49}},"dim":["","list.70","listItem.3","paragraph.0"],"code":"`'Prečo sú zoskupené:\\n{examples}'` - reasoning"},{"id":"/root/children/70/children/3/children/0/children/0","type":"inlineCode","loc":{"start":5541,"end":5576,"line":{"s":257,"e":257,"code":["- `'Prečo sú zoskupené:\\n{examples}'` - reasoning"]},"column":{"s":2,"e":37}},"dim":["","list.70","listItem.3","paragraph.0","inlineCode.0"],"code":"`'Prečo sú zoskupené:\\n{examples}'`"},{"id":"/root/children/70/children/3/children/0/children/1","type":"text","loc":{"start":5576,"end":5588,"line":{"s":257,"e":257,"code":["- `'Prečo sú zoskupené:\\n{examples}'` - reasoning"]},"column":{"s":37,"e":49}},"dim":["","list.70","listItem.3","paragraph.0","text.1"],"code":" - reasoning"},{"id":"/root/children/71","type":"paragraph","loc":{"start":5590,"end":5604,"line":{"s":259,"e":259,"code":["**Use cases:**"]},"column":{"s":0,"e":14}},"dim":["","paragraph.71"],"code":"**Use cases:**"},{"id":"/root/children/71/children/0","type":"strong","loc":{"start":5590,"end":5604,"line":{"s":259,"e":259,"code":["**Use cases:**"]},"column":{"s":0,"e":14}},"dim":["","paragraph.71","strong.0"],"code":"**Use cases:**"},{"id":"/root/children/71/children/0/children/0","type":"text","loc":{"start":5592,"end":5602,"line":{"s":259,"e":259,"code":["**Use cases:**"]},"column":{"s":2,"e":12}},"dim":["","paragraph.71","strong.0","text.0"],"code":"Use cases:"},{"id":"/root/children/72","type":"list","loc":{"start":5606,"end":5671,"line":{"s":261,"e":263,"code":["- different perspectives","- specialized analysis","- custom insights"]},"column":{"s":0,"e":17}},"dim":["","list.72"],"code":"- different perspectives\n- specialized analysis\n- custom insights","symbName":"list","symbRange":[5673,5930],"symbRangeL":[261,278],"outerCode":"- specialized analysis\n- custom insights\n\n### 4. Visualize Embeddings\n\nInteractive 2D visualization of document space.\n\n```bash\n./llm-tagger.sh visualize tagged.ndjson output.html\n```\n\n**Output:** Interactive HTML with plotly\n\n**Requirements:** `pip install umap-learn` in Dockerfile\n\n**Use cases:**","outerHtml":"<ul><li>specialized analysis</li><li>custom insights</li></ul>\n\n<h3>4. Visualize Embeddings</h3>\n\n<p>Interactive 2D visualization of document space.</p>\n\n<p>```bash\n./llm-tagger.sh visualize tagged.ndjson output.html\n```</p>\n\n<p><b>Output:</b> Interactive HTML with plotly</p>\n\n<p><b>Requirements:</b> `pip install umap-learn` in Dockerfile</p>\n\n<p><b>Use cases:</b></p>"},{"id":"/root/children/72/children/0","type":"listItem","loc":{"start":5606,"end":5630,"line":{"s":261,"e":261,"code":["- different perspectives"]},"column":{"s":0,"e":24}},"dim":["","list.72","listItem.0"],"code":"- different perspectives"},{"id":"/root/children/72/children/0/children/0","type":"paragraph","loc":{"start":5608,"end":5630,"line":{"s":261,"e":261,"code":["- different perspectives"]},"column":{"s":2,"e":24}},"dim":["","list.72","listItem.0","paragraph.0"],"code":"different perspectives"},{"id":"/root/children/72/children/0/children/0/children/0","type":"text","loc":{"start":5608,"end":5630,"line":{"s":261,"e":261,"code":["- different perspectives"]},"column":{"s":2,"e":24}},"dim":["","list.72","listItem.0","paragraph.0","text.0"],"code":"different perspectives"},{"id":"/root/children/72/children/1","type":"listItem","loc":{"start":5631,"end":5653,"line":{"s":262,"e":262,"code":["- specialized analysis"]},"column":{"s":0,"e":22}},"dim":["","list.72","listItem.1"],"code":"- specialized analysis"},{"id":"/root/children/72/children/1/children/0","type":"paragraph","loc":{"start":5633,"end":5653,"line":{"s":262,"e":262,"code":["- specialized analysis"]},"column":{"s":2,"e":22}},"dim":["","list.72","listItem.1","paragraph.0"],"code":"specialized analysis"},{"id":"/root/children/72/children/1/children/0/children/0","type":"text","loc":{"start":5633,"end":5653,"line":{"s":262,"e":262,"code":["- specialized analysis"]},"column":{"s":2,"e":22}},"dim":["","list.72","listItem.1","paragraph.0","text.0"],"code":"specialized analysis"},{"id":"/root/children/72/children/2","type":"listItem","loc":{"start":5654,"end":5671,"line":{"s":263,"e":263,"code":["- custom insights"]},"column":{"s":0,"e":17}},"dim":["","list.72","listItem.2"],"code":"- custom insights"},{"id":"/root/children/72/children/2/children/0","type":"paragraph","loc":{"start":5656,"end":5671,"line":{"s":263,"e":263,"code":["- custom insights"]},"column":{"s":2,"e":17}},"dim":["","list.72","listItem.2","paragraph.0"],"code":"custom insights"},{"id":"/root/children/72/children/2/children/0/children/0","type":"text","loc":{"start":5656,"end":5671,"line":{"s":263,"e":263,"code":["- custom insights"]},"column":{"s":2,"e":17}},"dim":["","list.72","listItem.2","paragraph.0","text.0"],"code":"custom insights"},{"id":"/root/children/73","type":"heading","loc":{"start":5673,"end":5700,"line":{"s":265,"e":265,"code":["### 4. Visualize Embeddings"]},"column":{"s":0,"e":27}},"dim":["","heading.73"],"code":"### 4. Visualize Embeddings","symbName":"heading","symbRange":[5702,5930],"symbRangeL":[265,278],"outerCode":"\nInteractive 2D visualization of document space.\n\n```bash\n./llm-tagger.sh visualize tagged.ndjson output.html\n```\n\n**Output:** Interactive HTML with plotly\n\n**Requirements:** `pip install umap-learn` in Dockerfile\n\n**Use cases:**","outerHtml":"\n<p>Interactive 2D visualization of document space.</p>\n\n<p>```bash\n./llm-tagger.sh visualize tagged.ndjson output.html\n```</p>\n\n<p><b>Output:</b> Interactive HTML with plotly</p>\n\n<p><b>Requirements:</b> `pip install umap-learn` in Dockerfile</p>\n\n<p><b>Use cases:</b></p>"},{"id":"/root/children/73/children/0","type":"text","loc":{"start":5677,"end":5700,"line":{"s":265,"e":265,"code":["### 4. Visualize Embeddings"]},"column":{"s":4,"e":27}},"dim":["","heading.73","text.0"],"code":"4. Visualize Embeddings"},{"id":"/root/children/74","type":"paragraph","loc":{"start":5702,"end":5749,"line":{"s":267,"e":267,"code":["Interactive 2D visualization of document space."]},"column":{"s":0,"e":47}},"dim":["","paragraph.74"],"code":"Interactive 2D visualization of document space."},{"id":"/root/children/74/children/0","type":"text","loc":{"start":5702,"end":5749,"line":{"s":267,"e":267,"code":["Interactive 2D visualization of document space."]},"column":{"s":0,"e":47}},"dim":["","paragraph.74","text.0"],"code":"Interactive 2D visualization of document space."},{"id":"/root/children/75","type":"code","loc":{"start":5751,"end":5814,"line":{"s":269,"e":271,"code":["```bash","./llm-tagger.sh visualize tagged.ndjson output.html","```"]},"column":{"s":0,"e":3}},"dim":["","code.75"],"code":"```bash\n./llm-tagger.sh visualize tagged.ndjson output.html\n```","symbName":"code","symbRange":[5816,5930],"symbRangeL":[null,278],"outerCode":";{ engine: dot rankdir:LR clusterize: true }\n\n# LLM Tagger\n\nDocument clustering and analysis using embeddings and LLM.\n\n## Quick Start\n\n```bash\ncd docker/llm-tagger\n\n# step 1: cache embeddings (always first)\n./llm-tagger.sh init total.ndjson\n\n# step 2: cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100\n\n# optional: hierarchical tagging\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n\n# optional: analysis tools\n./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```\n\n## All Commands\n\n| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |\n\n## Workflows\n\n### Workflow 1: Single-Level Tagging\n\nOne level of clusters with tags.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. cluster and tag\n./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```\n\n**Result:** Documents with cluster and tags\n\n```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```\n\n### Workflow 2: Hierarchical Tagging\n\nTwo levels: fine-grained + broad categories.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. fine-grained clustering\n./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5\n\n# 3. cluster the clusters\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```\n\n**Result:** Documents with two levels\n\n```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 12,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 2,\n  \"tags_l2\": \"hardware\"\n}\n```\n\n### Workflow 3: Multiple Subsets\n\nAnalyze different slices independently.\n\n```bash\n# 1. cache embeddings (once for all)\n./llm-tagger.sh init total.ndjson\n\n# 2. tag each subset (different patterns)\n./llm-tagger.sh educe-tags january.ndjson jan-tagged.ndjson 15\n./llm-tagger.sh educe-tags hardware.ndjson hw-tagged.ndjson 20\n./llm-tagger.sh educe-tags vendor-x.ndjson vx-tagged.ndjson 10\n```\n\n**Benefit:** Each subset gets unique clustering reflecting its patterns\n\n### Workflow 4: Multiple Subsets + Hierarchy\n\nCombine both approaches.\n\n```bash\n# 1. cache embeddings (once)\n./llm-tagger.sh init total.ndjson\n\n# 2. subset 1 with hierarchy\n./llm-tagger.sh educe-tags january.ndjson jan-l1.ndjson 5\n./llm-tagger.sh hierarchical jan-l1.ndjson jan-final.ndjson 3\n\n# 3. subset 2 with hierarchy\n./llm-tagger.sh educe-tags hardware.ndjson hw-l1.ndjson 5\n./llm-tagger.sh hierarchical hw-l1.ndjson hw-final.ndjson 3\n```\n\n## Performance Changes\n\n### 1. Parallel Tag Extraction\n\n- 8 concurrent workers\n- removed sleep delays\n- 8x faster\n\n### 2. Connection Pooling\n\n- requests.Session() for reuse\n- all API calls\n\n### 3. Parallel Embedding\n\n- 4 concurrent workers\n- batch size: 32 → 64\n- maintains ordering\n\n### 4. HDBSCAN Multi-core\n\n- core_dist_n_jobs=-1\n- uses all cores\n\n**Overall speedup: 5-8x**\n\n## Two-Phase Architecture\n\n### Phase 1: init total.ndjson\n\n- pre-compute all embeddings\n- save to cache\n- no clustering yet\n\n### Phase 2: educe-tags subset.ndjson\n\n- load cached embeddings\n- cluster subset only\n- generate subset-specific tags\n- each subset: unique patterns\n\n### Benefits\n\n- embeddings computed once\n- each subset independently clustered\n- tags reflect subset context\n- january.ndjson → january patterns\n- hardware.ndjson → hardware categories\n\n## Hierarchical Tagging\n\nCreate two-level taxonomy: fine-grained + broad categories.\n\n### Workflow\n\n```bash\n# level 1: fine-grained (many small clusters)\n./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5\n\n# level 2: cluster the clusters\n./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```\n\n### How It Works\n\n1. **Level 1** creates detailed clusters (e.g., \"screws\", \"bolts\", \"cables\")\n1. **Level 2** computes cluster centroids\n1. Clusters the centroids into super-clusters\n1. Generates broad tags (e.g., \"hardware\", \"electronics\")\n\n### Output Format\n\n```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 5,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 1,\n  \"tags_l2\": \"hardware, tools\"\n}\n```\n\n### Use Cases\n\n- multi-level taxonomy\n- drill-down navigation\n- broad overview + details\n\n## Additional Tools\n\nBeyond clustering, embeddings enable powerful analysis.\n\n### 1. Semantic Search\n\nFind documents matching query semantically.\n\n```bash\n./llm-tagger.sh search data.ndjson 'hardware repair' 10\n```\n\n**Output:** Top matching documents with similarity scores\n\n**Use cases:**\n\n- find relevant documents\n- content discovery\n- query-based filtering\n\n### 2. Find Similar Documents\n\nDetect duplicates or near-duplicates.\n\n```bash\n./llm-tagger.sh similar data.ndjson 0.95\n```\n\n**Output:** Pairs of similar documents\n\n**Use cases:**\n\n- duplicate detection\n- data quality\n- identify redundancy\n\n### 3. Custom Prompts\n\nAsk different questions to clusters.\n\n```bash\n./llm-tagger.sh custom-prompt tagged.ndjson 'Zhrň hlavné témy:\\n{examples}'\n```\n\n**Prompt examples:**\n\n- `'Zhrň hlavné témy:\\n{examples}'` - summaries\n- `'Identifikuj riziká:\\n{examples}'` - risk analysis\n- `'Aké patterny vidíš:\\n{examples}'` - insights\n- `'Prečo sú zoskupené:\\n{examples}'` - reasoning\n\n**Use cases:**\n\n- different perspectives\n- specialized analysis\n- custom insights\n\n### 4. Visualize Embeddings\n\nInteractive 2D visualization of document space.\n\n```bash\n./llm-tagger.sh visualize tagged.ndjson output.html\n```\n\n**Output:** Interactive HTML with plotly\n\n**Requirements:** `pip install umap-learn` in Dockerfile\n\n**Use cases:**","outerHtml":"<p>;{ engine: dot rankdir:LR clusterize: true }</p>\n\n<h1>LLM Tagger</h1>\n\n<p>Document clustering and analysis using embeddings and LLM.</p>\n\n<h2>Quick Start</h2>\n\n<p>```bash\ncd docker/llm-tagger</p>\n\n<h1>step 1: cache embeddings (always first)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>step 2: cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100</p>\n\n<h1>optional: hierarchical tagging</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3</p>\n\n<h1>optional: analysis tools</h1><p>./llm-tagger.sh search data.ndjson 'query' 10\n./llm-tagger.sh similar data.ndjson 0.95\n./llm-tagger.sh custom-prompt tagged.ndjson 'prompt'\n./llm-tagger.sh visualize tagged.ndjson output.html\n```</p>\n\n<h2>All Commands</h2>\n\n<p>| Command         | Purpose                             |\n| --------------- | ----------------------------------- |\n| `init`          | Cache embeddings from total dataset |\n| `educe-tags`    | Cluster subset and generate tags    |\n| `hierarchical`  | Create two-level taxonomy           |\n| `search`        | Semantic search by query            |\n| `similar`       | Find duplicate/similar documents    |\n| `custom-prompt` | Ask custom questions to clusters    |\n| `visualize`     | Interactive 2D visualization        |</p>\n\n<h2>Workflows</h2>\n\n<h3>Workflow 1: Single-Level Tagging</h3>\n\n<p>One level of clusters with tags.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. cluster and tag</h1><p>./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20\n```</p>\n\n<p><b>Result:</b> Documents with cluster and tags</p>\n\n<p>```json\n{ \"description\": \"...\", \"cluster\": 5, \"tags\": \"hardware, tools\" }\n```</p>\n\n<h3>Workflow 2: Hierarchical Tagging</h3>\n\n<p>Two levels: fine-grained + broad categories.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. fine-grained clustering</h1><p>./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5</p>\n\n<h1>3. cluster the clusters</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```</p>\n\n<p><b>Result:</b> Documents with two levels</p>\n\n<p>```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 12,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 2,\n  \"tags_l2\": \"hardware\"\n}\n```</p>\n\n<h3>Workflow 3: Multiple Subsets</h3>\n\n<p>Analyze different slices independently.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once for all)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. tag each subset (different patterns)</h1><p>./llm-tagger.sh educe-tags january.ndjson jan-tagged.ndjson 15\n./llm-tagger.sh educe-tags hardware.ndjson hw-tagged.ndjson 20\n./llm-tagger.sh educe-tags vendor-x.ndjson vx-tagged.ndjson 10\n```</p>\n\n<p><b>Benefit:</b> Each subset gets unique clustering reflecting its patterns</p>\n\n<h3>Workflow 4: Multiple Subsets + Hierarchy</h3>\n\n<p>Combine both approaches.</p>\n\n<p>```bash</p><h1>1. cache embeddings (once)</h1><p>./llm-tagger.sh init total.ndjson</p>\n\n<h1>2. subset 1 with hierarchy</h1><p>./llm-tagger.sh educe-tags january.ndjson jan-l1.ndjson 5\n./llm-tagger.sh hierarchical jan-l1.ndjson jan-final.ndjson 3</p>\n\n<h1>3. subset 2 with hierarchy</h1><p>./llm-tagger.sh educe-tags hardware.ndjson hw-l1.ndjson 5\n./llm-tagger.sh hierarchical hw-l1.ndjson hw-final.ndjson 3\n```</p>\n\n<h2>Performance Changes</h2>\n\n<h3>1. Parallel Tag Extraction</h3>\n\n<ul><li>8 concurrent workers</li><li>removed sleep delays</li><li>8x faster</li></ul>\n\n<h3>2. Connection Pooling</h3>\n\n<ul><li>requests.Session() for reuse</li><li>all API calls</li></ul>\n\n<h3>3. Parallel Embedding</h3>\n\n<ul><li>4 concurrent workers</li><li>batch size: 32 → 64</li><li>maintains ordering</li></ul>\n\n<h3>4. HDBSCAN Multi-core</h3>\n\n<ul><li>core_dist_n_jobs=-1</li><li>uses all cores</li></ul>\n\n<p><b>Overall speedup: 5-8x</b></p>\n\n<h2>Two-Phase Architecture</h2>\n\n<h3>Phase 1: init total.ndjson</h3>\n\n<ul><li>pre-compute all embeddings</li><li>save to cache</li><li>no clustering yet</li></ul>\n\n<h3>Phase 2: educe-tags subset.ndjson</h3>\n\n<ul><li>load cached embeddings</li><li>cluster subset only</li><li>generate subset-specific tags</li><li>each subset: unique patterns</li></ul>\n\n<h3>Benefits</h3>\n\n<ul><li>embeddings computed once</li><li>each subset independently clustered</li><li>tags reflect subset context</li><li>january.ndjson → january patterns</li><li>hardware.ndjson → hardware categories</li></ul>\n\n<h2>Hierarchical Tagging</h2>\n\n<p>Create two-level taxonomy: fine-grained + broad categories.</p>\n\n<h3>Workflow</h3>\n\n<p>```bash</p><h1>level 1: fine-grained (many small clusters)</h1><p>./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5</p>\n\n<h1>level 2: cluster the clusters</h1><p>./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3\n```</p>\n\n<h3>How It Works</h3>\n\n<ol><li><b>Level 1</b> creates detailed clusters (e.g., \"screws\", \"bolts\", \"cables\")</li><li><b>Level 2</b> computes cluster centroids</li><li>Clusters the centroids into super-clusters</li><li>Generates broad tags (e.g., \"hardware\", \"electronics\")</li></ol>\n\n<h3>Output Format</h3>\n\n<p>```json\n{\n  \"description\": \"...\",\n  \"cluster_l1\": 5,\n  \"tags_l1\": \"screws, fasteners\",\n  \"cluster_l2\": 1,\n  \"tags_l2\": \"hardware, tools\"\n}\n```</p>\n\n<h3>Use Cases</h3>\n\n<ul><li>multi-level taxonomy</li><li>drill-down navigation</li><li>broad overview + details</li></ul>\n\n<h2>Additional Tools</h2>\n\n<p>Beyond clustering, embeddings enable powerful analysis.</p>\n\n<h3>1. Semantic Search</h3>\n\n<p>Find documents matching query semantically.</p>\n\n<p>```bash\n./llm-tagger.sh search data.ndjson 'hardware repair' 10\n```</p>\n\n<p><b>Output:</b> Top matching documents with similarity scores</p>\n\n<p><b>Use cases:</b></p>\n\n<ul><li>find relevant documents</li><li>content discovery</li><li>query-based filtering</li></ul>\n\n<h3>2. Find Similar Documents</h3>\n\n<p>Detect duplicates or near-duplicates.</p>\n\n<p>```bash\n./llm-tagger.sh similar data.ndjson 0.95\n```</p>\n\n<p><b>Output:</b> Pairs of similar documents</p>\n\n<p><b>Use cases:</b></p>\n\n<ul><li>duplicate detection</li><li>data quality</li><li>identify redundancy</li></ul>\n\n<h3>3. Custom Prompts</h3>\n\n<p>Ask different questions to clusters.</p>\n\n<p>```bash\n./llm-tagger.sh custom-prompt tagged.ndjson 'Zhrň hlavné témy:\\n{examples}'\n```</p>\n\n<p><b>Prompt examples:</b></p>\n\n<ul><li>`'Zhrň hlavné témy:\\n{examples}'` - summaries</li><li>`'Identifikuj riziká:\\n{examples}'` - risk analysis</li><li>`'Aké patterny vidíš:\\n{examples}'` - insights</li><li>`'Prečo sú zoskupené:\\n{examples}'` - reasoning</li></ul>\n\n<p><b>Use cases:</b></p>\n\n<ul><li>different perspectives</li><li>specialized analysis</li><li>custom insights</li></ul>\n\n<h3>4. Visualize Embeddings</h3>\n\n<p>Interactive 2D visualization of document space.</p>\n\n<p>```bash\n./llm-tagger.sh visualize tagged.ndjson output.html\n```</p>\n\n<p><b>Output:</b> Interactive HTML with plotly</p>\n\n<p><b>Requirements:</b> `pip install umap-learn` in Dockerfile</p>\n\n<p><b>Use cases:</b></p>"},{"id":"/root/children/76","type":"paragraph","loc":{"start":5816,"end":5856,"line":{"s":273,"e":273,"code":["**Output:** Interactive HTML with plotly"]},"column":{"s":0,"e":40}},"dim":["","paragraph.76"],"code":"**Output:** Interactive HTML with plotly"},{"id":"/root/children/76/children/0","type":"strong","loc":{"start":5816,"end":5827,"line":{"s":273,"e":273,"code":["**Output:** Interactive HTML with plotly"]},"column":{"s":0,"e":11}},"dim":["","paragraph.76","strong.0"],"code":"**Output:**"},{"id":"/root/children/76/children/0/children/0","type":"text","loc":{"start":5818,"end":5825,"line":{"s":273,"e":273,"code":["**Output:** Interactive HTML with plotly"]},"column":{"s":2,"e":9}},"dim":["","paragraph.76","strong.0","text.0"],"code":"Output:"},{"id":"/root/children/76/children/1","type":"text","loc":{"start":5827,"end":5856,"line":{"s":273,"e":273,"code":["**Output:** Interactive HTML with plotly"]},"column":{"s":11,"e":40}},"dim":["","paragraph.76","text.1"],"code":" Interactive HTML with plotly"},{"id":"/root/children/77","type":"paragraph","loc":{"start":5858,"end":5914,"line":{"s":275,"e":275,"code":["**Requirements:** `pip install umap-learn` in Dockerfile"]},"column":{"s":0,"e":56}},"dim":["","paragraph.77"],"code":"**Requirements:** `pip install umap-learn` in Dockerfile"},{"id":"/root/children/77/children/0","type":"strong","loc":{"start":5858,"end":5875,"line":{"s":275,"e":275,"code":["**Requirements:** `pip install umap-learn` in Dockerfile"]},"column":{"s":0,"e":17}},"dim":["","paragraph.77","strong.0"],"code":"**Requirements:**"},{"id":"/root/children/77/children/0/children/0","type":"text","loc":{"start":5860,"end":5873,"line":{"s":275,"e":275,"code":["**Requirements:** `pip install umap-learn` in Dockerfile"]},"column":{"s":2,"e":15}},"dim":["","paragraph.77","strong.0","text.0"],"code":"Requirements:"},{"id":"/root/children/77/children/1","type":"text","loc":{"start":5875,"end":5876,"line":{"s":275,"e":275,"code":["**Requirements:** `pip install umap-learn` in Dockerfile"]},"column":{"s":17,"e":18}},"dim":["","paragraph.77","text.1"],"code":" "},{"id":"/root/children/77/children/2","type":"inlineCode","loc":{"start":5876,"end":5900,"line":{"s":275,"e":275,"code":["**Requirements:** `pip install umap-learn` in Dockerfile"]},"column":{"s":18,"e":42}},"dim":["","paragraph.77","inlineCode.2"],"code":"`pip install umap-learn`"},{"id":"/root/children/77/children/3","type":"text","loc":{"start":5900,"end":5914,"line":{"s":275,"e":275,"code":["**Requirements:** `pip install umap-learn` in Dockerfile"]},"column":{"s":42,"e":56}},"dim":["","paragraph.77","text.3"],"code":" in Dockerfile"},{"id":"/root/children/78","type":"paragraph","loc":{"start":5916,"end":5930,"line":{"s":277,"e":277,"code":["**Use cases:**"]},"column":{"s":0,"e":14}},"dim":["","paragraph.78"],"code":"**Use cases:**"},{"id":"/root/children/78/children/0","type":"strong","loc":{"start":5916,"end":5930,"line":{"s":277,"e":277,"code":["**Use cases:**"]},"column":{"s":0,"e":14}},"dim":["","paragraph.78","strong.0"],"code":"**Use cases:**"},{"id":"/root/children/78/children/0/children/0","type":"text","loc":{"start":5918,"end":5928,"line":{"s":277,"e":277,"code":["**Use cases:**"]},"column":{"s":2,"e":12}},"dim":["","paragraph.78","strong.0","text.0"],"code":"Use cases:"},{"id":"/root/children/79","type":"list","loc":{"start":5932,"end":6013,"line":{"s":279,"e":281,"code":["- explore document landscape","- verify clustering quality","- find outliers visually"]},"column":{"s":0,"e":24}},"dim":["","list.79"],"code":"- explore document landscape\n- verify clustering quality\n- find outliers visually","symbName":"list","symbRange":[null,null],"symbRangeL":[279,281],"outerCode":"- verify clustering quality","outerHtml":"<ul><li>verify clustering quality</li></ul>"},{"id":"/root/children/79/children/0","type":"listItem","loc":{"start":5932,"end":5960,"line":{"s":279,"e":279,"code":["- explore document landscape"]},"column":{"s":0,"e":28}},"dim":["","list.79","listItem.0"],"code":"- explore document landscape"},{"id":"/root/children/79/children/0/children/0","type":"paragraph","loc":{"start":5934,"end":5960,"line":{"s":279,"e":279,"code":["- explore document landscape"]},"column":{"s":2,"e":28}},"dim":["","list.79","listItem.0","paragraph.0"],"code":"explore document landscape"},{"id":"/root/children/79/children/0/children/0/children/0","type":"text","loc":{"start":5934,"end":5960,"line":{"s":279,"e":279,"code":["- explore document landscape"]},"column":{"s":2,"e":28}},"dim":["","list.79","listItem.0","paragraph.0","text.0"],"code":"explore document landscape"},{"id":"/root/children/79/children/1","type":"listItem","loc":{"start":5961,"end":5988,"line":{"s":280,"e":280,"code":["- verify clustering quality"]},"column":{"s":0,"e":27}},"dim":["","list.79","listItem.1"],"code":"- verify clustering quality"},{"id":"/root/children/79/children/1/children/0","type":"paragraph","loc":{"start":5963,"end":5988,"line":{"s":280,"e":280,"code":["- verify clustering quality"]},"column":{"s":2,"e":27}},"dim":["","list.79","listItem.1","paragraph.0"],"code":"verify clustering quality"},{"id":"/root/children/79/children/1/children/0/children/0","type":"text","loc":{"start":5963,"end":5988,"line":{"s":280,"e":280,"code":["- verify clustering quality"]},"column":{"s":2,"e":27}},"dim":["","list.79","listItem.1","paragraph.0","text.0"],"code":"verify clustering quality"},{"id":"/root/children/79/children/2","type":"listItem","loc":{"start":5989,"end":6013,"line":{"s":281,"e":281,"code":["- find outliers visually"]},"column":{"s":0,"e":24}},"dim":["","list.79","listItem.2"],"code":"- find outliers visually"},{"id":"/root/children/79/children/2/children/0","type":"paragraph","loc":{"start":5991,"end":6013,"line":{"s":281,"e":281,"code":["- find outliers visually"]},"column":{"s":2,"e":24}},"dim":["","list.79","listItem.2","paragraph.0"],"code":"find outliers visually"},{"id":"/root/children/79/children/2/children/0/children/0","type":"text","loc":{"start":5991,"end":6013,"line":{"s":281,"e":281,"code":["- find outliers visually"]},"column":{"s":2,"e":24}},"dim":["","list.79","listItem.2","paragraph.0","text.0"],"code":"find outliers visually"}]