graph { graph [engine:dot rankdir:TD clusterize:false applyCss:dark intention:full] pattern__orchestrator -> pattern__orchestrator__experimental [reverse: false] pattern__orchestrator -> pattern__orchestrator__Recommended_minimal_clean_setup__3_queues_ [reverse: false] pattern__orchestrator__Recommended_minimal_clean_setup__3_queues_ -> pattern__orchestrator__Recommended_minimal_clean_setup__3_queues____pattern__orchestrator__queues [reverse: false] pattern__orchestrator *= pattern__orchestrator__Workflow_Orchestration [reverse: false] pattern__orchestrator *= pattern__orchestrator__problem__wait_for_bag_of_tasks [reverse: false] pattern__orchestrator__problem__wait_for_bag_of_tasks *= pattern__orchestrator__problem__wait_for_bag_of_tasks___pattern__orchestrator__Redis_Pub__Sub_notification [reverse: false] pattern__orchestrator__problem__wait_for_bag_of_tasks *= pattern__orchestrator__problem__wait_for_bag_of_tasks___pattern__orchestrator__join_barrier [reverse: false] pattern__orchestrator__problem__wait_for_bag_of_tasks___pattern__orchestrator__join_barrier *= pattern__orchestrator__problem__wait_for_bag_of_tasks___pattern__orchestrator__join_barrier___pattern__orchestrator__problem__wait_for_bag_of_tasks___pattern__orchestrator__Barrier_Counter_in_Redis__recommended_ [reverse: false] pattern__orchestrator__problem__wait_for_bag_of_tasks___pattern__orchestrator__join_barrier *= pattern__orchestrator__problem__wait_for_bag_of_tasks___pattern__orchestrator__join_barrier___pattern__orchestrator__problem__wait_for_bag_of_tasks___pattern__orchestrator__Barrier_via__completion_messages__queue [reverse: false] pattern__orchestrator *= pattern__orchestrator__problem__parallel_programs_ [reverse: false] pattern__orchestrator__problem__parallel_programs_ *= pattern__orchestrator__problem__parallel_programs____pattern__orchestrator__sharding [reverse: false] pattern__orchestrator *= pattern__orchestrator__conclusion [reverse: false] pattern__orchestrator__conclusion *= pattern__orchestrator__conclusion___pattern__orchestrator__state [reverse: false] pattern__orchestrator__conclusion *= pattern__orchestrator__conclusion___pattern__orchestrator__task_output [reverse: false] pattern__orchestrator__conclusion *= pattern__orchestrator__conclusion___pattern__orchestrator__completion_message [reverse: false] pattern__orchestrator [z: 9.5 @html: "

pattern: orchestrator

• uses 3 queues

• structure: serial, exclusive+parallel,
serial

• example:

	• 【 serial tasks 】
	• 【 bag of parallel tasks 】
	• 【 untiloop 】

" nomen: "pattern: orchestrator"] pattern__orchestrator__experimental [z: 4.75 @html: "

experimental

• TODO: could also support eval code ?

• holds program state in namespaced redis
keys: flat json

• state can collect output of tasks in
adjacent structure

" nomen: "experimental"] pattern__orchestrator__Recommended_minimal_clean_setup__3_queues_ [z: 4.75 @html: "

Recommended minimal clean setup 3 queues

• /queue/orchestrator

	• 100% serial queue: prefetch = 1, 1 consumer only
	• State machine messages only (continue workflow, step done, parallel done, start workflow, etc.)

• /queue/serial-tasks

	• 100% serial queue: prefetch = 1, 1 consumer only
	• tasks run sequentially by nature

• /queue/parallel-tasks

	• many consumers
	• tasks run concurrently

• each task updates its state after
finished

	• TODO: but how?
	• completion message

This is the simplest clean design that
gives you:

• controlled sequential execution

• unlimited parallelism

• simple, readable routing

• non-interfering logic

" nomen: "Recommended minimal clean setup 3 queues"] pattern__orchestrator__Recommended_minimal_clean_setup__3_queues____pattern__orchestrator__queues [z: 2.375 @html: "

queues

• queue_orchestrator

• queue_tasks_serial

• queue_tasks_parallel

" nomen: "queues"] pattern__orchestrator__Workflow_Orchestration [z: 4.75 @html: "

Workflow Orchestration
RabbitMQ is only the transport; the
orchestration logic lives in the
program-json.This is a “Workflow Orchestration”
(a.k.a. Saga Orchestration) pattern, a
classic Saga orchestration where every
“step” is either:

• a sequential action, or

• a parallel fan-out group with a join
barrier.

Orchestrated Workflow with Barrier
Synchronization

• Reads the workflow definition (your
array).

• For each step:

• If it’s a single task → publish to a
serial queue, wait for ack.

• If it’s a parallel block → fan-out
messages to a parallel queue, track N
acks, wait until all are done
(barrier),continue to next step.

" nomen: "Workflow Orchestration"] pattern__orchestrator__problem__wait_for_bag_of_tasks [z: 4.75 @html: "

problem: wait for bag of tasks

• its called Barrier of orchestration

• or logical cut

• its how to wait all parallel tasks have
finished, and we shall proceed with next
tasks

" nomen: "problem: wait for bag of tasks"] pattern__orchestrator__problem__wait_for_bag_of_tasks___pattern__orchestrator__Redis_Pub__Sub_notification [z: 2.375 @html: "

Redis Pub/Sub notification

• Redis: (best).

• Setup

When you publish the parallel tasks:
Set counter: SET
barrier:<wf>:<step> N
Workers, when done: DECR
barrier:<wf>:<step> If
result == 0 → PUBLISH barrier:done
"<wf>:<step>"
Orchestrator waits:
redisSub.subscribe("barrier:done");
redisSub.on("message",
(channel, msg) => { const 【wf,
step】 = msg.split(":");
continueWorkflow(wf, step); });
✅ zero polling ✅ instant reaction ✅ very
scalableThis is the cleanest.

" nomen: "Redis Pub/Sub notification"] pattern__orchestrator__problem__wait_for_bag_of_tasks___pattern__orchestrator__join_barrier [z: 2.375 @html: "

join barrier

" nomen: "join barrier"] pattern__orchestrator__problem__wait_for_bag_of_tasks___pattern__orchestrator__join_barrier___pattern__orchestrator__problem__wait_for_bag_of_tasks___pattern__orchestrator__Barrier_Counter_in_Redis__recommended_ [z: 1.1875 @html: "

Barrier Counter in Redis recommended
Simple, robust, works great with
RabbitMQ.How it works:

• Before publishing N parallel tasks → SET
barrier:<workflowId>:pending = N

• Each worker, after finishing, runs: DECR
barrier:<workflowId>:pending

• When the value hits 0, the orchestrator
continues to next step.

Why this is best:

• Atomic

• No race conditions

• Survives crashes

Very easy in Node.js + STOMP

" nomen: "Barrier Counter in Redis recommended"] pattern__orchestrator__problem__wait_for_bag_of_tasks___pattern__orchestrator__join_barrier___pattern__orchestrator__problem__wait_for_bag_of_tasks___pattern__orchestrator__Barrier_via__completion_messages__queue [z: 1.1875 @html: "

Barrier via “completion messages” queue
RabbitMQ-only solution. Pattern:

• Each parallel task sends a completion
message to a dedicated queue: workflow.<id>.step.<n>.completed

• Orchestrator counts how many
"completed" messages arrived.

• When it reaches N → continue.

Cons:

• Requires the orchestrator to stay online
and track counts in memory or storage.

• Not safe without persistent storage.

• Use only when you cannot add
Redis/Postgres.

" nomen: "Barrier via “completion messages” queue"] pattern__orchestrator__problem__parallel_programs_ [z: 4.75 @html: "

problem: parallel programs?

• can multiple programs run in the same
mq-space?

• will a slow program not hold up a fast
one?

• possible solution: sharding into few
queue's

You said:

• fast workflow: occurs often

• slow workflow: occurs rarely

• slow workflow takes long time

You want maximum simplicity, so:✅ Choose Option A: multiple orchestrator
queuesYou only need two:/queue/orchestrator.fast
/queue/orchestrator.slowThen route based on workflow type:const destination = workflow.type ===
'fast' ? '/queue/orchestrator.fast' :
'/queue/orchestrator.slow'Each queue has its own orchestrator
consumer.✅ Fast workflows never wait behind slow
ones ✅ No complexity ✅ No risk ✅ Perfect
isolation

" nomen: "problem: parallel programs?"] pattern__orchestrator__problem__parallel_programs____pattern__orchestrator__sharding [z: 2.375 @html: "

sharding
Multiple orchestrator queues, each one
handles many workflows, but each
workflow stays in exactly one shard
based on hash(workflowId)% N. Gives you
isolation, parallelism, and ordering
with bounded queue count.

" nomen: "sharding"] pattern__orchestrator__conclusion [z: 4.75 @html: "

conclusion

• we use redis. is redisJSON needed? no.

• main queue: queue_orchestrator

• no single long-running program

• program is executed in steps, steps are
a disconnected sequence of messages

" nomen: "conclusion"] pattern__orchestrator__conclusion___pattern__orchestrator__state [z: 2.375 @html: "

state

• central state of the program

• who holds the program?

	• one big redis json program:id ?
	• or flat scalar json namespaced like folders:

/program:id/plan
/program:id/plan/step:0
/program:id/outout/step:0
/program:id/plan/step:1/bag:0
/program:id/output/step:1/bag:0

• redis prefers small values well
granulated into namespaced keys

" nomen: "state"] pattern__orchestrator__conclusion___pattern__orchestrator__task_output [z: 2.375 @html: "

task output

• each task when finished, can set its
output to /program:id/plan/step:1/{bag:0}

• this is done by the consumer, on ACK

" nomen: "task output"] pattern__orchestrator__conclusion___pattern__orchestrator__completion_message [z: 2.375 @html: "

completion message

• pure rabbit-mq message program:next

• advances queue_orchestrator

• must identify which step of program is
saying: move on

• if a bag of tasks:

		• decr counter with redis eval. fires only when counter===0
		• may postprocess its output

" nomen: "completion message"] ```subgraphs 'conclusion*|': '.' '*problem*|': '.' 'pattern:*|': '.' ``` #```ranks # - [ '*funda*' ] # - [ '...' ] # - [ '*problem*' ] #``` }