Agent Mesh — Distributed Agent Orchestration

Turn multiple machines (or processes) into a coordinated agent mesh with master/worker topology, HTTP task delegation, heartbeat health monitoring, and local-first fallback when remote nodes are unreachable.

Architecture

           ┌──────────────┐
           │  Master Node  │  (port 8900)
           │  (orchestrator)│
           └──────┬───────┘
                  │
     ┌────────────┼────────────┐
     │            │            │
  ┌──▼───┐   ┌───▼──┐   ┌────▼───┐
  │Worker│   │Worker│   │Worker  │
  │  1   │   │  2   │   │   3    │
  └──────┘   └──────┘   └────────┘
   Port 8901  Port 8902  Port 8903

• Master: Orchestrator — delegates tasks, runs heartbeat checks, persists node state to .agent-mesh/nodes.json.
• Worker: Registers with master, receives /run task requests, executes them locally, returns results.
• Local-First: When a remote node is dead or unreachable, the task runs locally on the delegate node.

When to Use

• You have multiple machines (laptop + cloud VM + homelab) and want them to share workload as a coordinated mesh.
• You need to delegate long-running tasks to a specific node (e.g. GPU worker).
• You want automatic health checks and failover to local execution.
• You are building a multi-node agent pipeline.

Setup

1. Environment

# Shared secret for request authentication (all nodes must share it)
export OPENAMER_MESH_SECRET="your-secure-random-string-here"

# Optional: custom data directory
export OPENAMER_MESH_HOME="$HOME/.openamer/agent-mesh"

# Optional: auto-register worker with master on startup
export OPENAMER_MESH_MASTER="http://10.0.0.1:8900"

2. Start the Master

python scripts/agent-mesh.py start
# or with custom port:
python scripts/agent-mesh.py start --port 8900 --host 0.0.0.0

3. Start Workers

# On worker machine 1:
python scripts/agent-mesh.py node --port 8901 --master-url http://10.0.0.1:8900

# On worker machine 2:
python scripts/agent-mesh.py node --port 8902 --master-url http://10.0.0.1:8900

# Standalone worker (no master auto-registration):
python scripts/agent-mesh.py node --port 8901
# Then manually register:
python scripts/agent-mesh.py register --host 10.0.0.5 --port 8901 --capabilities code shell

CLI Commands

agent-mesh.py start

Start the master orchestrator node.

Flag	Default	Description
--port / -p	8900	HTTP listen port
--host	0.0.0.0	Bind address
--capabilities / -c	code shell delegate	Node capabilities

agent-mesh.py node

Start a worker node.

Flag	Default	Description
--port / -p	8901	HTTP listen port
--host	0.0.0.0	Bind address
--capabilities / -c	code shell	Node capabilities
--master-url / -m	(none)	Auto-register with this master URL

agent-mesh.py status

List all known nodes with health status.

agent-mesh.py delegate <task> --to <node>

Delegate a task string to a named node. Falls back to local execution if the remote node is dead.

• Tasks starting with ! run as shell commands on the target.
• Other tasks are recorded as natural-language tasks.

agent-mesh.py register --host <ip> --port <port>

Manually register a node without running a server.

Heartbeat System

• Master pings every known worker every 30 seconds.
• Each /ping call carries the shared X-Mesh-Token header.
• A worker is marked dead after 3 consecutive missed pings.
• Dead workers trigger local-first fallback on task delegation.
• The status table shows ✓ for alive and ✗ for dead, plus miss count.

Security

• All HTTP requests carry X-Mesh-Token matching OPENAMER_MESH_SECRET.
• Token comparison uses secrets.compare_digest() to prevent timing attacks.
• If OPENAMER_MESH_SECRET is unset, authentication is skipped (insecure — only use on trusted networks).

Task Delegation Flow

User/Agent ──> delegate(task, node)
                     │
               ┌─────▼─────┐
               │ Node alive?│
               └─────┬─────┘
               ┌─────▼─────┐        ┌──────────────────┐
          YES  │  HTTP POST  │       │  Task executed   │
               │  /run       │──────>│  on remote node  │
               └─────────────┘       └──────────────────┘
               ┌──────────────────┐
          NO   │  Local fallback  │
               │  (_run_task_locally)│
               └──────────────────┘

Persistence

Node state is stored in ~/.openamer/agent-mesh/nodes.json:

{
  "nodes": [
    {
      "node_id": "node-a1b2c3d4",
      "host": "10.0.0.5",
      "port": 8901,
      "capabilities": ["code", "shell"],
      "role": "worker",
      "last_seen": 1735728000.0,
      "missed_heartbeats": 0,
      "alive": true,
      "last_error": null
    }
  ]
}

Cron Job (Automatic Health Checks)

A cron job runs every 5 minutes to check all mesh nodes. Enable via:

python -c "
from cron.jobs import create_job
create_job({
    'id': 'mesh-health-check',
    'name': 'Agent Mesh Health Check',
    'schedule': '*/5 * * * *',
    'script': 'python scripts/agent-mesh.py status',
    'enabled': True,
})
"

Or manually add to ~/.openamer/cron/jobs.json:

{
  "id": "mesh-health-check",
  "name": "Agent Mesh Health Check",
  "schedule": "*/5 * * * *",
  "script": "python scripts/agent-mesh.py status",
  "enabled": true
}

Pitfalls

• Port conflicts: Ensure no other service uses the mesh ports (8900+).
• Firewall: Workers must be reachable from master. Open ports or use SSH tunnels for cross-network setups.
• Shared secret: All nodes must have the same OPENAMER_MESH_SECRET. Rotate it by restarting all nodes with the new value.
• Local-first is a fallback — not a replication mechanism. If the master fails mid-delegation, the task is retried locally on the delegate caller.
• Heartbeat is one-directional (master → workers). For bidirectional health checks, run a worker as a peer (both sides run agent-mesh.py start).
• On Windows: use python explicitly; the script won't #!/usr/bin/env python3 directly. Use py -3 if the python launcher is available.

Verification

# 1. Smoke test — start master in background
OPENAMER_MESH_SECRET=test123 python scripts/agent-mesh.py start --port 18900 &
sleep 2

# 2. Check status (should show master node)
OPENAMER_MESH_SECRET=test123 python scripts/agent-mesh.py status

# 3. Start a worker and register
OPENAMER_MESH_SECRET=test123 python scripts/agent-mesh.py node --port 18901 --master-url http://127.0.0.1:18900 &
sleep 2

# 4. Verify both nodes appear
OPENAMER_MESH_SECRET=test123 python scripts/agent-mesh.py status

# 5. Delegate a task
OPENAMER_MESH_SECRET=test123 python scripts/agent-mesh.py delegate "!echo hello mesh" --to node

# 6. Kill worker, check dead status, delegate again (should fall back to local)
kill %2
OPENAMER_MESH_SECRET=test123 python scripts/agent-mesh.py status

# 7. Cleanup
kill %1

Related

• a2a-swarm — Cryptographic identity, signed skills/insights, GitHub mesh.
• multi-agent-orchestration — In-process swarm patterns (fan-out, debate).
• crew-manager — Role-based CrewAI-style team orchestration (Dev/Reviewer).