Fix Merkle Root Mismatch

When to Use

Use this skill AFTER running /investigate-merkle-root-mismatch to investigate the issue.

1. User request triggers:
   • "Fix the merkle root mismatch on [chain]"
   • "Apply the merkle tree fixes"
   • "Update the relayer database with correct message IDs"

Input Parameters

Parameter	Required	Default	Description
origin	Yes	-	The origin chain to fix (e.g., polygonzkevm, ethereum)
domain_id	Yes	-	Domain ID of the chain
environment	No	mainnet3	mainnet3 or testnet4
merkle_tree_insertions	Yes	-	List of corrections: [{leaf_index, message_id, block_number}]

Prerequisites

1. Investigation completed - Run /investigate-merkle-root-mismatch first to identify mismatched entries
2. kubectl access to the relayer pods
3. Port-forward already established to relayer (port 9090)

Fix Workflow

Step 0: Find Monorepo Root

MONOREPO_ROOT=$(git rev-parse --show-toplevel)

Step 1: Verify Port-Forward is Active

Check if port 9090 is already in use:

lsof -i :9090

If not in use, start port-forward in background:

kubectl port-forward omniscient-relayer-hyperlane-agent-relayer-0 9090:9090 -n [environment] &

Wait a few seconds, then verify it's working:

curl -s "localhost:9090/merkle_tree_insertions?domain_id=[domain_id]&leaf_index_start=0&leaf_index_end=1"

Step 2: Present Changes for Confirmation

IMPORTANT: User confirmation required before executing.

Before making any changes to the relayer database, you MUST:

1. Present a summary of ALL changes to be made in a clear table format:

   Leaf Index	Correct Message ID	Block Number
   1477	0x4bb3e20db45366a6a360ad2639c5421ea622a69b89b2edb045aa97e2051529b7	9891832
   1478	0xa66dbdc1874acfddf14e75e6a20dd1632e9e3206c5f5855884da0d26d8ca12fa	9891872

2. Ask the user to get explicit confirmation:

   • Question: "Do you want to apply these merkle tree fixes to the relayer database?"
   • Options: "Yes, apply fixes" / "No, cancel"

3. Only proceed if the user explicitly confirms.

Step 3: Apply Merkle Tree Fixes

Once confirmed, insert the correct merkle tree insertions via the relayer API:

curl -X POST \
    -H 'Content-type: application/json' \
    'localhost:9090/merkle_tree_insertions' \
    -d '{
    "merkle_tree_insertions": [
        {
            "chain": [domain_id],
            "insertion_block_number": [block_number],
            "leaf_index": [index],
            "message_id": "0x..."
        }
    ]
}'

Request schema:

{
  "merkle_tree_insertions": [
    {
      "chain": "<domain_id as number>",
      "insertion_block_number": "<block number as number>",
      "leaf_index": "<leaf index as number>",
      "message_id": "<0x-prefixed H256 hash>"
    }
  ]
}

Verify response: The API should return a success response with the count of inserted entries.

Step 4: Restart Relayer

IMPORTANT: User confirmation required before executing.

Ask the user to confirm:

• Question: "Do you want to restart the relayer to apply the database fixes?"
• Options: "Yes, restart relayer" / "No, I'll restart manually later"

Only proceed if the user explicitly confirms.

Once confirmed, restart the relayer to rebuild its in-memory merkle tree:

pnpm --dir typescript/infra exec tsx ./scripts/agents/restart-agents.ts -e [environment] --context hyperlane --role relayer

Step 5: Validate Fix

After the relayer restarts, verify the fix:

5.1: Get Latest Index

Query Grafana for the current tree size:

Use mcp__grafana__query_prometheus with:
- datasourceUid: grafanacloud-prom
- expr: hyperlane_latest_tree_insertion_index{origin="[origin]", hyperlane_deployment="[environment]"}
- startTime: now-1h
- queryType: instant

5.2: Spot-Check Roots at Key Indices

Compare validator and relayer roots at three points: first fixed index, middle, and latest.

index=[INDEX]
validator_root=$(curl -s "https://hyperlane-[environment]-[origin]-validator-0.s3.us-east-1.amazonaws.com/checkpoint_${index}_with_id.json" | jq -r '.value.checkpoint.root')
relayer_root=$(curl -s "localhost:9090/merkle_proofs?domain_id=[domain_id]&leaf_index=${index}&root_index=${index}" | jq -r '.root')
echo "Index $index:"
echo "  Validator: $validator_root"
echo "  Relayer:   0x$relayer_root"
if [ "$validator_root" = "0x$relayer_root" ]; then echo "  ✓ Match"; else echo "  ❌ MISMATCH"; fi

Run this for:

1. First fixed index (e.g., 37352)
2. Middle index (e.g., 39000)
3. Latest index (e.g., 41172)

5.3: Verify Mismatch Metric is Clear

Use mcp__grafana__query_prometheus with:
- datasourceUid: grafanacloud-prom
- expr: hyperlane_merkle_root_mismatch{origin="[origin]"}
- startTime: now-1h
- queryType: instant

Expected result: Empty result or value of 0.

5.4: Report Results

Present validation results in a table:

Index	Location	Result
37352	First fixed	✓ Match
39000	Middle	✓ Match
41172	Latest	✓ Match

If still mismatched:

• Inform the user that additional entries may need fixing
• Suggest re-running /investigate-merkle-root-mismatch to investigate further

API Reference

Relayer Endpoints

Endpoint	Method	Description
/merkle_tree_insertions	GET	List merkle tree insertions from DB
/merkle_tree_insertions	POST	Insert/update merkle tree entries

Request/Response Examples

POST /merkle_tree_insertions

Request:

{
  "merkle_tree_insertions": [
    {
      "chain": 1101,
      "insertion_block_number": 9891832,
      "leaf_index": 1477,
      "message_id": "0x4bb3e20db45366a6a360ad2639c5421ea622a69b89b2edb045aa97e2051529b7"
    }
  ]
}

Response (success):

{
  "status": "success",
  "data": {
    "count": 1
  }
}

Common Issues

1. Port-forward disconnected: Re-establish before applying fixes
2. API returns error: Check the error message, verify domain_id and message_id format
3. Relayer restart fails: Check kubectl access and pod status
4. Fix didn't work: May have missed some mismatched indices, re-investigate

Runbook Reference

Full runbook: https://www.notion.so/hyperlanexyz/Merkle-Root-Mismatch-26a6d35200d680a2857dcd0b228d4ab7