Building an Event-Driven AWS Agent with Amazon Bedrock for Automated CloudWatch Incident Investigation

2. Event-Driven DevOps Agent Architecture

The architecture follows a strictly event-driven decoupled model designed to operate within milliseconds of alarm state transitions:

3. EventBridge & Lambda Ingestion Pipeline

To capture CloudWatch Alarm state changes, configure an EventBridge rule that targets specific CloudWatch Alarm namespace events. Below is an example AWS CDK / CloudFormation event pattern definition in Python:

import json
import boto3

# EventBridge pattern filter for CloudWatch Alarm State Changes
EVENT_PATTERN = {
    "source": ["aws.cloudwatch"],
    "detail-type": ["CloudWatch Alarm State Change"],
    "detail": {
        "state": {
            "value": ["ALARM"]
        }
    }
}

def lambda_handler(event, context):
    alarm_detail = event.get("detail", {})
    alarm_name = alarm_detail.get("alarmName", "UnknownAlarm")
    metric_name = alarm_detail.get("configuration", {}).get("metrics", [{}])[0].get("metricStat", {}).get("metric", {}).get("metricName", "")

    print(f"✓ Ingested ALARM event for: {alarm_name} (Metric: {metric_name})")

    # Invoke Amazon Bedrock Agent Runtime
    bedrock_runtime = boto3.client("bedrock-agent-runtime", region_name="us-east-1")
    response = bedrock_runtime.invoke_agent(
        agentId="AGENT_ID_PLACEHOLDER",
        agentAliasId="TSTALIASID",
        sessionId=f"incident-{alarm_name}",
        inputText=f"Investigate incident for CloudWatch alarm {alarm_name}. Analyze logs and trace metrics for root cause."
    )

    return {"statusCode": 200, "body": json.dumps("Agent Investigation Triggered")}

4. Bedrock Agent Reasoning & Telemetry Analysis

When the Bedrock agent receives the investigation request, it uses OpenAPI schema action groups to query Amazon CloudWatch Logs Insights and AWS X-Ray trace streams dynamically.

By utilizing Anthropic Claude 3.5 Sonnet on Amazon Bedrock, the agent synthesizes raw log output into an executive summary:

• Log Correlation: Searches for 5xx HTTP status codes and unhandled exceptions within a 15-minute window surrounding the alert time.
• Dependency Graph Mapping: Pinpoints downstream database timeouts or third-party rate limiting endpoints.
• IaC Diff Identification: Identifies if recent Terraform applies introduced low memory limits or tight connection pool thresholds.

5. Production Boto3 & Terraform Auto-Patching

Once the root cause is established (for example, ECS task memory limit undersized at 512MB causing OOMKilled events), the agent triggers a secondary Lambda worker with elevated Git credentials to create a fix branch and propose an infrastructure update:

# Autonomous HCL Patch Generator for ECS Memory Allocation
def apply_iac_remediation_patch(repo_path: str, service_name: str, recommended_memory_mb: int):
    patch_hcl = f'''
# Patch generated by AIConnect Autonomous DevOps Agent
resource "aws_ecs_task_definition" "{service_name}" {{
  family                   = "{service_name}"
  cpu                      = "1024"
  memory                   = "{recommended_memory_mb}" # Auto-increased from 512
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
}}
'''
    print(f"✓ Generated HCL patch for {service_name}: memory = {recommended_memory_mb}MB")
    return patch_hcl

6. Security, Cost & Production Guardrails

Autonomous operations require uncompromising security and cost boundaries:

• Least-Privilege IAM Roles: Ensure investigation Lambda workers hold read-only permissions (cloudwatch:Get*, logs:FilterLogEvents, xray:Get*). Read more in our DevSecOps & FinOps Engineering Guide.
• Human-In-The-Loop Approval: Never let autonomous agents auto-apply state-changing Terraform modifications without human approval on PR review.
• Rate Limits & Token Caps: Set Max Token outputs on Amazon Bedrock requests to prevent runaway agent loops during massive cloud outages.