Cognitive Systems Engineering State of Practice

Apply insights from cognitive systems engineering research to design resilient agent architectures, diagnose coordination failures, and encode expert knowledge that performs under pressure.

Decision Points

Agent Architecture Design

START: Need to design multi-agent system
│
├─ Is this a well-defined, stable task sequence?
│  ├─ YES → Use pipeline architecture BUT build 3 failure recovery paths
│  └─ NO → Use goal-oriented architecture with alternative methods
│
├─ Does task require expertise under time pressure?
│  ├─ YES → Implement Recognition-Primed Decision Making pattern
│  │        (situation recognition → rapid simulation → action)
│  └─ NO → Standard deliberative architecture acceptable
│
└─ Will humans supervise or intervene?
   ├─ YES → Mandatory: mode transparency + shared state representation
   └─ NO → Focus on agent-to-agent coordination interfaces

Task Decomposition Strategy

Given complex task to decompose:
│
├─ Can expert describe complete process reliably?
│  ├─ YES → Verify with Critical Decision Method anyway
│  └─ NO → Use structured cognitive task analysis FIRST
│
├─ Are there natural failure/degradation points?
│  ├─ YES → Design alternative paths for each failure mode
│  └─ NO → Suspicious - dig deeper for hidden failure modes
│
└─ Will agents need to adapt methods to context?
   ├─ YES → Separate goals from methods in specification
   └─ NO → Fixed sequence acceptable (rare case)

Failure Diagnosis Protocol

System failing unexpectedly:
│
├─ Does it work in demos but fail in production?
│  └─ YES → Invariant sequence assumption violated
│
├─ Are humans surprised by agent actions?
│  └─ YES → Automation surprise - check mode transparency
│
├─ Do agents fail when tools/data unavailable?
│  └─ YES → Missing alternative paths in decomposition
│
└─ Does agent understand but can't execute effectively?
   └─ YES → Knowing-doing gap - check situated context

Failure Modes

1. Pipeline Worship

Symptoms: System works perfectly in happy path, crashes at first unexpected condition Detection Rule: If you hear "we need to handle the edge case" more than once, you're in this anti-pattern Root Cause: Designed for idealized sequence, no alternative paths Fix: Redesign with goal/method separation, build 3 recovery paths for most common failures

2. Behavioral Specification Fallacy

Symptoms: Agent mimics expert actions but can't adapt to novel situations Detection Rule: If expert says "I don't know how I knew that," but system specification doesn't capture cue recognition Root Cause: Encoded surface behavior without underlying reasoning structure Fix: Use Critical Decision Method to elicit tacit knowledge and situation assessment patterns

3. Silent Mode Transitions

Symptoms: Humans/agents surprised when system changes behavior or strategy Detection Rule: If stakeholders say "I had no idea it was doing that," automation surprise is occurring Root Cause: State changes not communicated across coordination boundaries Fix: Make every mode transition an explicit coordination event with shared state updates

4. Representation Divergence

Symptoms: Agent handoffs produce errors despite individual agents working correctly Detection Rule: If output from Agent A is misinterpreted by Agent B consistently Root Cause: Agents maintain different models of task/world state Fix: Explicit shared ontology and interface state verification

5. Novice Architecture for Expert Tasks

Symptoms: System follows rules perfectly but fails under pressure or novel conditions Detection Rule: If system can't explain WHY it chose an action, only WHAT rule it followed Root Cause: Rule-based architecture deployed for expertise-requiring task Fix: Upgrade to recognition-primed or case-based reasoning architecture

Worked Examples

Example 1: Multi-Agent Code Review System

Scenario: Design system where Agent A writes code, Agent B reviews, Agent C handles deployment

Initial Design (Flawed):

• Linear pipeline: Code → Review → Deploy
• Binary review outcome: Pass/Fail
• Fixed criteria checklist

CSE Analysis Reveals:

• Expert code reviewers don't use checklists - they recognize code smells and risk patterns
• Reviews adapt to code complexity, author experience, deployment criticality
• Real reviewers often iterate with authors, not just reject

Improved Design:

Agent A: Code Generation
├─ Includes intention metadata (what problem solving, why this approach)
├─ Context flags (urgency, risk level, author confidence)

Agent B: Recognition-Primed Review
├─ Situation assessment (code type, risk factors, author patterns)
├─ Pattern matching against failure libraries
├─ Graduated response: approve/iterate/escalate/reject

Agent C: Context-Sensitive Deployment  
├─ Deployment strategy adapts to review confidence + context flags
├─ Rollback paths pre-planned based on risk assessment

Key Decision Points Applied:

• Separated goals (ensure code quality) from methods (checklist vs pattern recognition)
• Added alternative paths (iteration loop, escalation)
• Made handoff state explicit (intention metadata, confidence levels)

Example 2: Diagnosing Customer Service Agent Failures

Problem: AI customer service agent handles routine queries well but escalates too often on complex issues

Diagnosis Process:

1. Check for Invariant Sequence: Agent follows script linearly, can't adapt when customer deviates
2. Examine Situation Recognition: Agent uses keyword matching, not contextual assessment
3. Test Alternative Paths: Agent has no recovery strategies when first approach fails

Root Cause: Behavioral specification fallacy - system trained on successful interaction transcripts but missing expert reasoning about when/how to adapt

Solution:

• Interview expert human agents using Critical Decision Method
• Identify situation types and recognition cues
• Build case library of adaptation strategies
• Add confidence scoring to enable graduated escalation

Quality Gates

Task completion checklist for CSE-informed agent design:

[ ] Alternative Path Coverage: System has defined recovery paths for 3 most likely failure modes [ ] Situation Recognition: Agent can classify situation type, not just process inputs [ ] Mode Transparency: All state changes are observable by supervisors/coordinators
[ ] Representation Alignment: Agent handoffs use explicit, shared state models [ ] Expertise Stage Match: Architecture complexity matches required expertise level [ ] Tacit Knowledge Elicitation: Used structured methods (not just self-report) for expert knowledge [ ] Context Sensitivity: System adapts methods to situational factors [ ] Knowing-Doing Verification: Tested execution capability, not just comprehension [ ] Coordination Failure Recovery: System handles representational divergence gracefully [ ] Automation Surprise Prevention: Mode changes communicated across all coordination boundaries

NOT-FOR Boundaries

This skill is NOT for:

• Pure UI/UX design problems → Use interaction design skills instead
• Simple deterministic tasks with no failure modes → Use standard pipeline architecture
• Tasks where behavioral observation captures all relevant expertise → Use direct behavioral modeling
• Systems with no human supervision or agent coordination → Use individual agent optimization

Use OTHER skills for:

• Interface layout and visual design → UI/UX skills
• Mathematical optimization problems → Operations research skills
• Data pipeline engineering → Data engineering skills
• Individual agent prompt optimization → Prompt engineering skills

This skill IS specifically for:

• Multi-agent coordination architecture
• Expert knowledge elicitation and encoding
• Failure mode prediction and mitigation
• Human-AI handoff design
• Complex task decomposition under uncertainty