| LLM Tagger Document clustering and analysis using embeddings and LLM. |
| Quick Start cd docker/llm-tagger # step 1: cache embeddings (always first) ./llm-tagger.sh init total.ndjson # step 2: cluster and tag ./llm-tagger.sh educe-tags subset.ndjson output.ndjson 100 # optional: hierarchical tagging ./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3 # optional: analysis tools ./llm-tagger.sh search data.ndjson 'query' 10 ./llm-tagger.sh similar data.ndjson 0.95 ./llm-tagger.sh custom-prompt tagged.ndjson 'prompt' ./llm-tagger.sh visualize tagged.ndjson output.html |
| All Commands | ||||||||||||||||
|
| Workflows |
| Workflow 1: Single-Level Tagging One level of clusters with tags. # 1. cache embeddings (once) ./llm-tagger.sh init total.ndjson # 2. cluster and tag ./llm-tagger.sh educe-tags subset.ndjson output.ndjson 20 Result: Documents with cluster and tags { "description": "...", "cluster": 5, "tags": "hardware, tools" } |
| Workflow 2: Hierarchical Tagging Two levels: fine-grained + broad categories. # 1. cache embeddings (once) ./llm-tagger.sh init total.ndjson # 2. fine-grained clustering ./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5 # 3. cluster the clusters ./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3 Result: Documents with two levels { "description": "...", "cluster_l1": 12, "tags_l1": "screws, fasteners", "cluster_l2": 2, "tags_l2": "hardware" } |
| Workflow 3: Multiple Subsets Analyze different slices independently. # 1. cache embeddings (once for all) ./llm-tagger.sh init total.ndjson # 2. tag each subset (different patterns) ./llm-tagger.sh educe-tags january.ndjson jan-tagged.ndjson 15 ./llm-tagger.sh educe-tags hardware.ndjson hw-tagged.ndjson 20 ./llm-tagger.sh educe-tags vendor-x.ndjson vx-tagged.ndjson 10 Benefit: Each subset gets unique clustering reflecting its patterns |
| Workflow 4: Multiple Subsets + Hierarchy Combine both approaches. # 1. cache embeddings (once) ./llm-tagger.sh init total.ndjson # 2. subset 1 with hierarchy ./llm-tagger.sh educe-tags january.ndjson jan-l1.ndjson 5 ./llm-tagger.sh hierarchical jan-l1.ndjson jan-final.ndjson 3 # 3. subset 2 with hierarchy ./llm-tagger.sh educe-tags hardware.ndjson hw-l1.ndjson 5 ./llm-tagger.sh hierarchical hw-l1.ndjson hw-final.ndjson 3 |
| Performance Changes |
| 1. Parallel Tag Extraction | |||
|
| 2. Connection Pooling | ||
|
| 3. Parallel Embedding | |||
|
| 4. HDBSCAN Multi-core | ||
| ||
| **Overall speedup: 5-8x |
| Two-Phase Architecture |
| Phase 1: init total.ndjson | |||
|
| Phase 2: educe-tags subset.ndjson | ||||
|
| Benefits | |||||
|
| Hierarchical Tagging Create two-level taxonomy: fine-grained + broad categories. |
| Workflow # level 1: fine-grained (many small clusters) ./llm-tagger.sh educe-tags subset.ndjson level1.ndjson 5 # level 2: cluster the clusters ./llm-tagger.sh hierarchical level1.ndjson level2.ndjson 3 |
| How It Works | ||||
|
| Output Format { "description": "...", "cluster_l1": 5, "tags_l1": "screws, fasteners", "cluster_l2": 1, "tags_l2": "hardware, tools" } |
| Use Cases | |||
|
| Additional Tools Beyond clustering, embeddings enable powerful analysis. |
| 1. Semantic Search Find documents matching query semantically. ./llm-tagger.sh search data.ndjson 'hardware repair' 10 Output: Top matching documents with similarity scores Use cases: | |||
|
| 2. Find Similar Documents Detect duplicates or near-duplicates. ./llm-tagger.sh similar data.ndjson 0.95 Output: Pairs of similar documents Use cases: | |||
|
| 3. Custom Prompts Ask different questions to clusters. ./llm-tagger.sh custom-prompt tagged.ndjson 'Zhrň hlavné témy:\\n{examples}' Prompt examples: | ||||
| ||||
| Use cases: | ||||
|
| 4. Visualize Embeddings Interactive 2D visualization of document space. ./llm-tagger.sh visualize tagged.ndjson output.html Output: Interactive HTML with plotly Requirements: pip install umap-learn in Dockerfile Use cases: | |||
|