Implementation Plan for Simplified Agent Observability Showcase

This document outlines a detailed implementation plan to create a simplified version of the Agent Observability Showcase project. The focus is on removing PII masking, simplifying dependencies, and implementing a basic agent with telemetry features.

📋 Comprehensive Implementation Plan

Follow this plan and ask clarifying questions if needed.

Repository Structure (After Simplification)

agent-observability/
├── src/
│   ├── main.py                    # FastAPI application (NEW)
│   ├── agent.py                   # Simple agent implementation (NEW)
│   └── backend/core/
│       ├── telemetry.py           # ✅ Keep (remove PII refs)
│       ├── telemetry_decorators.py # ✅ Keep (remove mask_pii)
│       ├── telemetry_api_dependencies.py # ✅ Keep
│       └── logging.py             # ✅ Keep (remove PIIMaskingFilter)
├── scripts/
│   ├── provision.sh               # Azure infra script (NEW)
│   ├── generate_test_telemetry.py # ✅ Update (remove Terraform)
│   ├── restore.sh                 # ✅ Keep
│   └── utils.sh                   # ✅ Keep
├── pyproject.toml                 # ✅ Massive cleanup
├── Makefile                       # ✅ Simplify targets
├── README.md                      # ✅ Complete rewrite
└── .env.example                   # NEW

---

Phase 1. Simple Agent Implementation Design

Agent Architecture

Use Azure AI Foundry with GPT-4.1 deployment (following ai-foundry-demo pattern)

Agent Features (src/agent.py)

class SimpleAgent:
    """
    Azure AI Foundry agent demonstrating telemetry in multi-step workflows.
    
    Uses Azure AI Agents SDK with GPT-4.1 deployment for realistic agent behavior.
    
    Steps:
    1. Planning (@trace_agent_step) - Analyze user query, decide tools
    2. Tool Execution (@track_tool_execution) - Execute selected tools with Azure Functions
    3. Response Generation (@track_tokens) - Generate response using GPT-4.1
    
    Tools (as Azure Functions):
    - web_search: Simulates web search with random results
    - calculator: Simple math operations  
    - weather: Mock weather data
    
    Configuration:
    - Uses AgentsClient from azure.ai.agents
    - Connects to AI Foundry project endpoint
    - Configured via settings (ai_foundry_project_endpoint, gpt_4_1_deployment_name)
    """

Demo Endpoints (src/main.py)

POST /api/chat          - Full agent conversation with tool execution
POST /api/plan          - Just the planning step (shows @trace_agent_step)
POST /api/tool/search   - Individual tool execution (shows @track_tool_execution)
POST /api/generate      - LLM response using GPT-4.1 (shows @track_tokens)
GET  /health            - Health check with APITelemetry
GET  /api/agent/status  - Agent configuration and status

---

Phase 2. PII Masking Removal Strategy

Files to Modify

telemetry_decorators.py

• Remove from src.backend.core.pii_masking import mask_pii import
• In trace_agent_step: Remove mask_pii() calls, pass kwargs directly to span
• In track_tool_execution: Remove mask_pii() calls, pass values directly
• Remove pii_params parameter from track_tool_execution (no longer needed)
• Update docstrings to remove PII references

logging.py

• Remove entire PIIMaskingFilter class (lines 66-136)
• Remove pii_mask_enabled and pii_mask_level parameters from setup_logging()
• Remove filter setup code (lines 388-395)

telemetry.py

• Remove docstring references to pii_masking module

---

Phase 3. Dependencies Cleanup

pyproject.toml - KEEP ONLY

[tool.poetry.dependencies]
python = ">=3.13,<3.14"

# Core FastAPI
fastapi = {version = "^0.119.0", extras = ["standard"]}
pydantic = "^2.11.9"
pydantic-settings = "^2.10.1"
httpx = "^0.28.1"

# Azure AI Foundry & Agents
azure-identity = "^1.25.0"
azure-ai-agents = ">=1.2.0b2"

# Telemetry
opentelemetry-sdk = "^1.37.0"
opentelemetry-api = "^1.37.0"
azure-core-tracing-opentelemetry = "^1.0.0b12"

# Utilities
orjson = "^3.11.3"
python-dotenv = "^1.1.0"

[tool.poetry.group.dev.dependencies]
ruff = "^0.11.6"
mypy = "^1.8.0"
pytest = "^8.4.0"
pytest-asyncio = "^1.0.0"
pytest-cov = "^6.1.1"
pytest-httpx = "^0.35.0"

REMOVE EVERYTHING ELSE: 39+ unnecessary dependencies!

---

Phase 4. provision.sh Script Design

Interactive Prompts

# Prompt with defaults
read -p "Project name [agent-observability]: " PROJECT_NAME
PROJECT_NAME=${PROJECT_NAME:-agent-observability}

read -p "Azure region [uaenorth]: " REGION
REGION=${REGION:-uaenorth}

read -p "Environment [dev]: " ENVIRONMENT
ENVIRONMENT=${ENVIRONMENT:-dev}

Azure CAF Naming Convention

# Resource names
RG_NAME="rg-${PROJECT_NAME}-${ENVIRONMENT}"
LOG_NAME="log-${PROJECT_NAME}-${ENVIRONMENT}"
APPI_NAME="appi-${PROJECT_NAME}-${ENVIRONMENT}"

Idempotent Operations

# Check if resource group exists
if az group exists --name "$RG_NAME" | grep -q "true"; then
    echo "✓ Resource group already exists"
else
    echo "Creating resource group..."
    az group create --name "$RG_NAME" --location "$REGION"
fi

Output to .env

# Create or update .env file
cat > .env << EOF
# Azure Application Insights
APPLICATIONINSIGHTS_CONNECTION_STRING="${CONN_STRING}"

# Azure AI Foundry (user must configure manually)
AI_FOUNDRY_NAME=
AI_FOUNDRY_PROJECT_NAME=
GPT_4_1_DEPLOYMENT_NAME=

# Service Configuration
LOG_LEVEL=INFO
ENVIRONMENT=${ENVIRONMENT}
SERVICE_NAME=${PROJECT_NAME}-api
EOF

echo ""
echo "✅ .env file created successfully"
echo "⚠️  Please configure AI Foundry settings in .env:"
echo "   - AI_FOUNDRY_NAME (your AI Foundry resource name)"
echo "   - AI_FOUNDRY_PROJECT_NAME (your project name)"
echo "   - GPT_4_1_DEPLOYMENT_NAME (your GPT-4.1 deployment name)"

---

Phase 5. generate_test_telemetry.py Updates

Remove

• All Terraform fallback logic (lines 371-414)
• Subprocess imports and logic
• References to "make provision" (replace with "./scripts/provision.sh")

Simplify get_connection_string()

def get_connection_string() -> str | None:
    """Get Application Insights connection string from .env file."""
    # Load from .env file
    from dotenv import load_dotenv
    load_dotenv()
    
    conn_str = os.getenv("APPLICATIONINSIGHTS_CONNECTION_STRING")
    if conn_str:
        return conn_str
    
    print("⚠️  Connection string not found in .env file")
    print("   Run: ./scripts/provision.sh to create Azure resources")
    return None

---

Phase 6. Makefile Simplification

KEEP THESE TARGETS

help                    # Keep
restore                 # Keep

# Infrastructure
provision               # Keep - runs ./scripts/provision.sh

# Code Quality  
format format-check     # Keep
lint fix                # Keep

# Testing
test test-cov           # Keep

# Application
run-api                 # Keep - runs uvicorn

# Observability
test-telemetry          # Keep
test-telemetry-normal   # Keep
test-telemetry-errors   # Keep
test-telemetry-spikes   # Keep
test-telemetry-batch    # Keep
test-telemetry-load     # Keep

REMOVE THESE TARGETS

# Remove ALL of these
register-providers
setup-remote-state
enable-remote-state-access
generate-env
enable-ai-foundry-access
setup-github-sp
configure-github
verify-sp
rotate-secrets
setup-cd
run-frontend
chat
test-stt
transcribe*
rag-*
config-*
compliance-*
eval-*

---

Phase 7. README.md Structure

# Agent Observability Showcase

Simple FastAPI application demonstrating OpenTelemetry integration with Azure Monitor.

## Features
- ✅ FastAPI with OpenTelemetry decorators
- ✅ Simple agent with multi-step workflows
- ✅ Azure Application Insights integration
- ✅ Automatic trace, metrics, and logs correlation
- ✅ Test telemetry data generator

## Quick Start

### 1. Provision Azure Infrastructure
./scripts/provision.sh

### 2. Install Dependencies  
make restore

### 3. Run the API
make run-api

### 4. Generate Test Data
make test-telemetry

### 5. View in Azure Portal
[Link to Application Insights]

## API Endpoints
[Document endpoints]

## Architecture
[Simple diagram]

---

Phase 8. .env.example

# Azure Application Insights
APPLICATIONINSIGHTS_CONNECTION_STRING=

# Azure AI Foundry Configuration
AI_FOUNDRY_NAME=
AI_FOUNDRY_PROJECT_NAME=
GPT_4_1_DEPLOYMENT_NAME=

# Logging Configuration
LOG_LEVEL=INFO

# Service Configuration
ENVIRONMENT=dev
SERVICE_NAME=agent-observability-api

---

Implementation Order & Estimates

1. Clean pyproject.toml ✅ - Added azure-ai-agents dependency
2. Remove PII masking - Clean 3 files
3. Create provision.sh - Interactive Azure CLI script
4. Update generate_test_telemetry.py - Remove Terraform logic
5. Create simple agent - Azure AI Foundry agent with 3 tools
   • Create src/core/config.py for AI Foundry settings
   • Create src/agent.py using AgentsClient pattern
   • Implement 3 tools as function definitions
6. Create FastAPI app - 6 endpoints with telemetry
7. Simplify Makefile - Remove 25+ targets
8. Create .env.example - 7 variables (added AI Foundry config)
9. Rewrite README - Complete documentation with AI Foundry setup
10. Test end-to-end - Verify everything works

AI Foundry Implementation Notes

Key Components from ai-foundry-demo

1. Settings/Config (src/core/config.py):

   class Settings(BaseSettings):
       ai_foundry_name: str
       ai_foundry_project_name: str
       gpt_4_1_deployment_name: str
       
       @property
       def ai_foundry_project_endpoint(self) -> str:
           return f"https://{self.ai_foundry_name}.services.ai.azure.com/api/projects/{self.ai_foundry_project_name}"
   

2. Agent Client (src/agent.py):

   from azure.ai.agents import AgentsClient
   from azure.identity import DefaultAzureCredential
   
   credential = DefaultAzureCredential()
   agents_client = AgentsClient(
       endpoint=settings.ai_foundry_project_endpoint,
       credential=credential
   )
   

3. Agent Creation:

   agent = agents_client.create_agent(
       model=settings.gpt_4_1_deployment_name,
       name="Simple_Agent",
       instructions="System prompt here...",
       tools=[tool_definitions]  # Functions for web_search, calculator, weather
   )
   

4. Thread & Execution:

   thread = agents_client.threads.create()
   agents_client.messages.create(thread_id=thread.id, role="user", content=query)
   
   with agents_client.runs.stream(thread_id=thread.id, agent_id=agent.id) as stream:
       stream.until_done()
   

---

Key Benefits of This Approach

✅ Simple: One agent, clear purpose
✅ Realistic: Uses actual Azure AI Foundry with GPT-4.1 deployment
✅ Complete: Demonstrates all telemetry features with real agent interactions
✅ Production-ready: Proper Azure CAF naming and AI Foundry integration
✅ Idempotent: Safe to run provision.sh multiple times
✅ No Terraform: Pure Azure CLI
✅ No PII complexity: Clean and focused
✅ Real AI: Actual GPT-4.1 responses with Azure AI Agents SDK