Implementation Plan for Simplified Agent Observability Showcase

# Implementation Plan for Simplified Agent Observability Showcase

This document outlines a detailed implementation plan to create a simplified version of the Agent Observability Showcase project. The focus is on removing PII masking, simplifying dependencies, and implementing a basic agent with telemetry features.

## 📋 **Comprehensive Implementation Plan**

Follow this plan and ask clarifying questions if needed.

### **Repository Structure (After Simplification)**

```
agent-observability/
├── src/
│   ├── main.py                    # FastAPI application (NEW)
│   ├── agent.py                   # Simple agent implementation (NEW)
│   └── backend/core/
│       ├── telemetry.py           # ✅ Keep (remove PII refs)
│       ├── telemetry_decorators.py # ✅ Keep (remove mask_pii)
│       ├── telemetry_api_dependencies.py # ✅ Keep
│       └── logging.py             # ✅ Keep (remove PIIMaskingFilter)
├── scripts/
│   ├── provision.sh               # Azure infra script (NEW)
│   ├── generate_test_telemetry.py # ✅ Update (remove Terraform)
│   ├── restore.sh                 # ✅ Keep
│   └── utils.sh                   # ✅ Keep
├── pyproject.toml                 # ✅ Massive cleanup
├── Makefile                       # ✅ Simplify targets
├── README.md                      # ✅ Complete rewrite
└── .env.example                   # NEW
```

---

## **Phase 1. Simple Agent Implementation Design**

### **Agent Architecture**

Use **Azure AI Foundry** with **GPT-4.1 deployment** (following [ai-foundry-demo](https://github.com/macromania/ai-foundry-demo) pattern)

### **Agent Features** (src/agent.py)

```python
class SimpleAgent:
    """
    Azure AI Foundry agent demonstrating telemetry in multi-step workflows.
    
    Uses Azure AI Agents SDK with GPT-4.1 deployment for realistic agent behavior.
    
    Steps:
    1. Planning (@trace_agent_step) - Analyze user query, decide tools
    2. Tool Execution (@track_tool_execution) - Execute selected tools with Azure Functions
    3. Response Generation (@track_tokens) - Generate response using GPT-4.1
    
    Tools (as Azure Functions):
    - web_search: Simulates web search with random results
    - calculator: Simple math operations  
    - weather: Mock weather data
    
    Configuration:
    - Uses AgentsClient from azure.ai.agents
    - Connects to AI Foundry project endpoint
    - Configured via settings (ai_foundry_project_endpoint, gpt_4_1_deployment_name)
    """
```

### **Demo Endpoints** (src/main.py)

```text
POST /api/chat          - Full agent conversation with tool execution
POST /api/plan          - Just the planning step (shows @trace_agent_step)
POST /api/tool/search   - Individual tool execution (shows @track_tool_execution)
POST /api/generate      - LLM response using GPT-4.1 (shows @track_tokens)
GET  /health            - Health check with APITelemetry
GET  /api/agent/status  - Agent configuration and status
```

---

## **Phase 2. PII Masking Removal Strategy**

### **Files to Modify**

#### **telemetry_decorators.py**

- Remove `from src.backend.core.pii_masking import mask_pii` import
- In `trace_agent_step`: Remove `mask_pii()` calls, pass kwargs directly to span
- In `track_tool_execution`: Remove `mask_pii()` calls, pass values directly
- Remove `pii_params` parameter from `track_tool_execution` (no longer needed)
- Update docstrings to remove PII references

#### **logging.py**

- Remove entire `PIIMaskingFilter` class (lines 66-136)
- Remove `pii_mask_enabled` and `pii_mask_level` parameters from `setup_logging()`
- Remove filter setup code (lines 388-395)

#### **telemetry.py**

- Remove docstring references to `pii_masking` module

---

## **Phase 3. Dependencies Cleanup**

### **pyproject.toml - KEEP ONLY**

```toml
[tool.poetry.dependencies]
python = ">=3.13,<3.14"

# Core FastAPI
fastapi = {version = "^0.119.0", extras = ["standard"]}
pydantic = "^2.11.9"
pydantic-settings = "^2.10.1"
httpx = "^0.28.1"

# Azure AI Foundry & Agents
azure-identity = "^1.25.0"
azure-ai-agents = ">=1.2.0b2"

# Telemetry
opentelemetry-sdk = "^1.37.0"
opentelemetry-api = "^1.37.0"
azure-core-tracing-opentelemetry = "^1.0.0b12"

# Utilities
orjson = "^3.11.3"
python-dotenv = "^1.1.0"

[tool.poetry.group.dev.dependencies]
ruff = "^0.11.6"
mypy = "^1.8.0"
pytest = "^8.4.0"
pytest-asyncio = "^1.0.0"
pytest-cov = "^6.1.1"
pytest-httpx = "^0.35.0"
```

**REMOVE EVERYTHING ELSE**: 39+ unnecessary dependencies!

---

## **Phase 4. provision.sh Script Design**

### **Interactive Prompts**

```bash
# Prompt with defaults
read -p "Project name [agent-observability]: " PROJECT_NAME
PROJECT_NAME=${PROJECT_NAME:-agent-observability}

read -p "Azure region [uaenorth]: " REGION
REGION=${REGION:-uaenorth}

read -p "Environment [dev]: " ENVIRONMENT
ENVIRONMENT=${ENVIRONMENT:-dev}
```

### **Azure CAF Naming Convention**

```bash
# Resource names
RG_NAME="rg-${PROJECT_NAME}-${ENVIRONMENT}"
LOG_NAME="log-${PROJECT_NAME}-${ENVIRONMENT}"
APPI_NAME="appi-${PROJECT_NAME}-${ENVIRONMENT}"
```

### **Idempotent Operations**

```bash
# Check if resource group exists
if az group exists --name "$RG_NAME" | grep -q "true"; then
    echo "✓ Resource group already exists"
else
    echo "Creating resource group..."
    az group create --name "$RG_NAME" --location "$REGION"
fi
```

### **Output to .env**

```bash
# Create or update .env file
cat > .env << EOF
# Azure Application Insights
APPLICATIONINSIGHTS_CONNECTION_STRING="${CONN_STRING}"

# Azure AI Foundry (user must configure manually)
AI_FOUNDRY_NAME=
AI_FOUNDRY_PROJECT_NAME=
GPT_4_1_DEPLOYMENT_NAME=

# Service Configuration
LOG_LEVEL=INFO
ENVIRONMENT=${ENVIRONMENT}
SERVICE_NAME=${PROJECT_NAME}-api
EOF

echo ""
echo "✅ .env file created successfully"
echo "⚠️  Please configure AI Foundry settings in .env:"
echo "   - AI_FOUNDRY_NAME (your AI Foundry resource name)"
echo "   - AI_FOUNDRY_PROJECT_NAME (your project name)"
echo "   - GPT_4_1_DEPLOYMENT_NAME (your GPT-4.1 deployment name)"
```

---

## **Phase 5. generate_test_telemetry.py Updates**

### **Remove**

- All Terraform fallback logic (lines 371-414)
- Subprocess imports and logic
- References to "make provision" (replace with "./scripts/provision.sh")

### **Simplify get_connection_string()**

```python
def get_connection_string() -> str | None:
    """Get Application Insights connection string from .env file."""
    # Load from .env file
    from dotenv import load_dotenv
    load_dotenv()
    
    conn_str = os.getenv("APPLICATIONINSIGHTS_CONNECTION_STRING")
    if conn_str:
        return conn_str
    
    print("⚠️  Connection string not found in .env file")
    print("   Run: ./scripts/provision.sh to create Azure resources")
    return None
```

---

## **Phase 6. Makefile Simplification**

### **KEEP THESE TARGETS**

```makefile
help                    # Keep
restore                 # Keep

# Infrastructure
provision               # Keep - runs ./scripts/provision.sh

# Code Quality  
format format-check     # Keep
lint fix                # Keep

# Testing
test test-cov           # Keep

# Application
run-api                 # Keep - runs uvicorn

# Observability
test-telemetry          # Keep
test-telemetry-normal   # Keep
test-telemetry-errors   # Keep
test-telemetry-spikes   # Keep
test-telemetry-batch    # Keep
test-telemetry-load     # Keep
```

### **REMOVE THESE TARGETS**

```makefile
# Remove ALL of these
register-providers
setup-remote-state
enable-remote-state-access
generate-env
enable-ai-foundry-access
setup-github-sp
configure-github
verify-sp
rotate-secrets
setup-cd
run-frontend
chat
test-stt
transcribe*
rag-*
config-*
compliance-*
eval-*
```

---

## **Phase 7. README.md Structure**

```markdown
# Agent Observability Showcase

Simple FastAPI application demonstrating OpenTelemetry integration with Azure Monitor.

## Features
- ✅ FastAPI with OpenTelemetry decorators
- ✅ Simple agent with multi-step workflows
- ✅ Azure Application Insights integration
- ✅ Automatic trace, metrics, and logs correlation
- ✅ Test telemetry data generator

## Quick Start

### 1. Provision Azure Infrastructure
./scripts/provision.sh

### 2. Install Dependencies  
make restore

### 3. Run the API
make run-api

### 4. Generate Test Data
make test-telemetry

### 5. View in Azure Portal
[Link to Application Insights]

## API Endpoints
[Document endpoints]

## Architecture
[Simple diagram]
```

---

## **Phase 8. .env.example**

```bash
# Azure Application Insights
APPLICATIONINSIGHTS_CONNECTION_STRING=

# Azure AI Foundry Configuration
AI_FOUNDRY_NAME=
AI_FOUNDRY_PROJECT_NAME=
GPT_4_1_DEPLOYMENT_NAME=

# Logging Configuration
LOG_LEVEL=INFO

# Service Configuration
ENVIRONMENT=dev
SERVICE_NAME=agent-observability-api
```

---

## **Implementation Order & Estimates**

1. **Clean pyproject.toml** ✅ - Added azure-ai-agents dependency
2. **Remove PII masking** - Clean 3 files
3. **Create provision.sh** - Interactive Azure CLI script
4. **Update generate_test_telemetry.py** - Remove Terraform logic
5. **Create simple agent** - Azure AI Foundry agent with 3 tools
   - Create src/core/config.py for AI Foundry settings
   - Create src/agent.py using AgentsClient pattern
   - Implement 3 tools as function definitions
6. **Create FastAPI app** - 6 endpoints with telemetry
7. **Simplify Makefile** - Remove 25+ targets
8. **Create .env.example** - 7 variables (added AI Foundry config)
9. **Rewrite README** - Complete documentation with AI Foundry setup
10. **Test end-to-end** - Verify everything works

## **AI Foundry Implementation Notes**

### **Key Components from ai-foundry-demo**

1. **Settings/Config** (src/core/config.py):

   ```python
   class Settings(BaseSettings):
       ai_foundry_name: str
       ai_foundry_project_name: str
       gpt_4_1_deployment_name: str
       
       @property
       def ai_foundry_project_endpoint(self) -> str:
           return f"https://{self.ai_foundry_name}.services.ai.azure.com/api/projects/{self.ai_foundry_project_name}"
   ```

2. **Agent Client** (src/agent.py):

   ```python
   from azure.ai.agents import AgentsClient
   from azure.identity import DefaultAzureCredential
   
   credential = DefaultAzureCredential()
   agents_client = AgentsClient(
       endpoint=settings.ai_foundry_project_endpoint,
       credential=credential
   )
   ```

3. **Agent Creation**:

   ```python
   agent = agents_client.create_agent(
       model=settings.gpt_4_1_deployment_name,
       name="Simple_Agent",
       instructions="System prompt here...",
       tools=[tool_definitions]  # Functions for web_search, calculator, weather
   )
   ```

4. **Thread & Execution**:

   ```python
   thread = agents_client.threads.create()
   agents_client.messages.create(thread_id=thread.id, role="user", content=query)
   
   with agents_client.runs.stream(thread_id=thread.id, agent_id=agent.id) as stream:
       stream.until_done()
   ```


---

## **Key Benefits of This Approach**

✅ **Simple**: One agent, clear purpose  
✅ **Realistic**: Uses actual Azure AI Foundry with GPT-4.1 deployment  
✅ **Complete**: Demonstrates all telemetry features with real agent interactions  
✅ **Production-ready**: Proper Azure CAF naming and AI Foundry integration  
✅ **Idempotent**: Safe to run provision.sh multiple times  
✅ **No Terraform**: Pure Azure CLI  
✅ **No PII complexity**: Clean and focused  
✅ **Real AI**: Actual GPT-4.1 responses with Azure AI Agents SDK