System Architecture

# System Architecture

## Overview

This is a serverless video automation system built on AWS that generates AI video content and publishes it to YouTube on a schedule.

## Architecture Diagram

```
┌─────────────────────────────────────────────────────────────────┐
│                        EventBridge Scheduler                     │
│  ┌──────────────┐  ┌──────────────┐  ┌────────────────────┐   │
│  │ Idea Gen     │  │ Video Check  │  │ Upload Schedule    │   │
│  │ (Daily 8am)  │  │ (Every 15min)│  │ (9am,1pm,5pm,9pm)  │   │
│  └──────┬───────┘  └──────┬───────┘  └─────────┬──────────┘   │
└─────────┼──────────────────┼────────────────────┼──────────────┘
          │                  │                    │
          ▼                  ▼                    ▼
    ┌──────────────────────────────────────────────────┐
    │           Orchestrator Lambda                     │
    │  (Coordinates all services)                       │
    └────┬─────────────────┬─────────────────┬─────────┘
         │                 │                 │
         ▼                 ▼                 ▼
┌─────────────────┐ ┌──────────────┐ ┌─────────────────┐
│ Idea Generator  │ │   Video      │ │    YouTube      │
│    Lambda       │ │  Generator   │ │    Uploader     │
│                 │ │   Lambda     │ │     Lambda      │
│  - GPT-4 API    │ │              │ │                 │
│  - Creates      │ │ - HeyGen/DID │ │ - Downloads     │
│    scripts      │ │ - Async gen  │ │   video         │
│                 │ │ - Status     │ │ - Uploads to    │
│                 │ │   tracking   │ │   YouTube       │
└────────┬────────┘ └──────┬───────┘ └────────┬────────┘
         │                 │                  │
         │                 │                  │
         └─────────────────┼──────────────────┘
                          │
                          ▼
                ┌──────────────────┐
                │    DynamoDB      │
                │  ┌────────────┐  │
                │  │   Ideas    │  │
                │  │   Videos   │  │
                │  │  Uploads   │  │
                │  │   Config   │  │
                │  └────────────┘  │
                └──────────────────┘

┌──────────────────────────────────────────────┐
│         API Gateway (HTTP API)                │
│                                               │
│  GET  /health                                 │
│  GET  /status                                 │
│  GET  /config/topic                           │
│  POST /config/topic                           │
│  POST /trigger/ideas                          │
│  POST /trigger/videos                         │
│  POST /trigger/upload                         │
└───────────────┬──────────────────────────────┘
                │
                ▼
          ┌────────────┐
          │ API Lambda │
          └────────────┘
```

## Data Models

### Video Idea
```python
{
  "id": "uuid",
  "topic": "artificial intelligence",
  "title": "AI Will Change Everything in 2024",
  "script": "Hey everyone! Today I want to talk about...",
  "description": "Discover how AI is transforming...",
  "tags": ["ai", "technology", "future"],
  "status": "pending|generating_video|video_ready|uploaded|failed",
  "created_at": "2024-01-10T12:00:00Z",
  "updated_at": "2024-01-10T12:00:00Z",
  "retry_count": 0
}
```

### Generated Video
```python
{
  "id": "uuid",
  "idea_id": "uuid",
  "video_url": "https://...",
  "video_provider": "heygen",
  "provider_video_id": "abc123",
  "duration_seconds": 60,
  "status": "pending|in_progress|completed|failed",
  "created_at": "2024-01-10T12:00:00Z",
  "updated_at": "2024-01-10T12:05:00Z",
  "retry_count": 0,
  "error_message": null
}
```

### YouTube Upload
```python
{
  "id": "uuid",
  "video_id": "uuid",
  "idea_id": "uuid",
  "youtube_video_id": "dQw4w9WgXcQ",
  "youtube_url": "https://youtube.com/watch?v=...",
  "status": "pending|uploading|published|failed",
  "scheduled_time": "2024-01-10T13:00:00Z",
  "uploaded_at": "2024-01-10T13:02:34Z",
  "created_at": "2024-01-10T12:00:00Z",
  "updated_at": "2024-01-10T13:02:34Z",
  "retry_count": 0,
  "error_message": null
}
```

## Workflow Details

### 1. Idea Generation Flow

```
EventBridge (8am daily)
  → Orchestrator Lambda
    → Idea Generator Lambda
      → OpenAI GPT-4 API
        ← Returns 8 video ideas with scripts
      → Save to DynamoDB (ideas table)
        - Status: "pending"
```

### 2. Video Generation Flow

```
EventBridge (every 2 hours)
  → Orchestrator Lambda
    → Video Generator Lambda
      → Query DynamoDB for pending ideas
      → For each idea:
        → Call HeyGen/D-ID API
          ← Returns video job ID
        → Update DynamoDB
          - Video status: "in_progress"
          - Idea status: "generating_video"

Separately (every 15 minutes):
EventBridge
  → Orchestrator Lambda
    → Video Generator Lambda (check_status=true)
      → Query videos with status "in_progress"
      → For each video:
        → Check status with provider API
        → If completed:
          - Download video URL
          - Update status: "completed"
          - Update idea status: "video_ready"
```

### 3. YouTube Upload Flow

```
EventBridge (at scheduled times: 9am, 1pm, 5pm, 9pm)
  → Orchestrator Lambda
    → YouTube Uploader Lambda
      → Query DynamoDB for completed videos
      → For each video (limit 1 per trigger):
        → Download video from provider URL
        → Upload to YouTube API
          - Set title, description, tags
          - Set privacy: public
        → Update DynamoDB
          - Upload status: "published"
          - Idea status: "uploaded"
        → Delete temporary file
```

## Error Handling

### Retry Strategy

- **Max Retries**: 3 (configurable)
- **Retry Tracking**: `retry_count` field in each table
- **Exponential Backoff**: Implemented in scheduled checks

### Error States

1. **Idea Generation Failure**
   - Logged to CloudWatch
   - Status: "failed"
   - Does not block other ideas

2. **Video Generation Failure**
   - Retry up to max_retries
   - If provider fails, status: "failed"
   - Idea status: "failed"
   - Error message stored

3. **Upload Failure**
   - Retry up to max_retries
   - Common issues:
     - Download failure
     - YouTube API quota exceeded
     - OAuth token expired
   - Error message stored

## Scalability

### Current Limits

- **Ideas per day**: 8 (1 batch)
- **Videos per day**: ~4-8 (depends on generation time)
- **Uploads per day**: 4 (scheduled times)

### Scaling Considerations

1. **Increase Frequency**
   - Modify EventBridge schedules
   - More posting times

2. **Batch Size**
   - Increase ideas per batch
   - Process more videos per trigger

3. **Concurrent Processing**
   - Lambda can process multiple videos simultaneously
   - DynamoDB auto-scales

4. **Cost Optimization**
   - Use reserved capacity for DynamoDB if traffic is predictable
   - Batch video generation to reduce API calls

## Security

### IAM Permissions

- **Lambda Role**: Minimal permissions
  - DynamoDB: Read/Write to specific tables
  - Lambda: Invoke other functions
  - Logs: Write to CloudWatch

### Secrets Management

- API keys stored in Lambda environment variables
- Terraform variables marked as `sensitive`
- Never logged or exposed in responses

### YouTube OAuth

- Refresh token stored securely
- Scope limited to upload only
- Token rotation supported

## Monitoring

### CloudWatch Metrics

1. **Lambda Invocations**
   - Success/failure rates
   - Duration
   - Concurrent executions

2. **DynamoDB**
   - Read/write capacity
   - Item counts
   - Throttling

3. **API Gateway**
   - Request count
   - Latency
   - Error rates

### Logging Strategy

All logs use structured JSON format:
```json
{
  "timestamp": "2024-01-10T12:00:00Z",
  "level": "INFO",
  "message": "Video generation started",
  "video_id": "...",
  "provider_video_id": "..."
}
```

### Alarms (Recommended)

1. Lambda errors > 5 in 5 minutes
2. DynamoDB throttling
3. API Gateway 5xx errors
4. Upload failures

## Cost Breakdown

### AWS Infrastructure

| Service | Usage | Monthly Cost |
|---------|-------|--------------|
| Lambda | ~1000 invocations, 10 GB-seconds | $0.50 |
| DynamoDB | On-demand, ~1000 operations/day | $2.00 |
| API Gateway | ~100 requests/day | $0.10 |
| CloudWatch | Logs retention 14 days | $1.00 |
| EventBridge | ~50 rules invocations/day | $0.00 |
| **Total** | | **~$5/month** |

### External APIs (4 videos/day)

| Service | Cost per Video | Monthly (120 videos) |
|---------|----------------|----------------------|
| OpenAI GPT-4 | $0.03-0.06 | $3.60-7.20 |
| HeyGen | $0.50-2.00 | $60-240 |
| D-ID | $0.30-1.00 | $36-120 |
| YouTube | Free | $0 |

**Total Monthly**: $65-365 depending on provider and plan

## Deployment

### Prerequisites

1. AWS Account with CLI configured
2. Terraform >= 1.0
3. API keys obtained
4. YouTube OAuth completed

### Deployment Steps

```bash
# 1. Install dependencies
pip install -r requirements.txt

# 2. Get YouTube token
python scripts/get_youtube_token.py

# 3. Configure variables
cp terraform/terraform.tfvars.example terraform/terraform.tfvars
# Edit with your values

# 4. Deploy
./scripts/deploy.sh

# 5. Test
./scripts/test_system.sh
```

### Infrastructure Components

- **4 DynamoDB Tables**: Ideas, Videos, Uploads, Config
- **5 Lambda Functions**: Idea Gen, Video Gen, Upload, Orchestrator, API
- **1 API Gateway**: HTTP API with 7 routes
- **7+ EventBridge Rules**: Scheduling automation
- **5 CloudWatch Log Groups**: Per-function logging
- **1 IAM Role**: Lambda execution role with minimal permissions

## Maintenance

### Regular Tasks

1. **Monitor Logs** (weekly)
   - Check for repeated errors
   - Review success rates

2. **Review Costs** (monthly)
   - DynamoDB usage
   - Lambda duration
   - API provider costs

3. **Update Topics** (as needed)
   - Via API: `POST /config/topic`
   - Keep content fresh

4. **Rotate Credentials** (quarterly)
   - YouTube OAuth token
   - API keys
   - Update Terraform variables

### Cleanup Old Data

```bash
# Delete old ideas (optional)
aws dynamodb scan --table-name video-automation-ideas \
  --filter-expression "created_at < :date" | ...

# Archive uploads to S3
# ... custom script
```

## Future Enhancements

### Potential Features

1. **Multi-language Support**
   - Generate videos in different languages
   - Use different avatars per language

2. **A/B Testing**
   - Multiple video variants
   - Track performance metrics

3. **Content Calendar**
   - Schedule specific topics for specific dates
   - Holiday/event-based content

4. **Analytics Integration**
   - Pull YouTube analytics
   - Optimize based on performance

5. **Multiple YouTube Channels**
   - Support posting to different channels
   - Different topics per channel

6. **Video Templates**
   - Predefined formats
   - Consistent branding

7. **Thumbnail Generation**
   - AI-generated thumbnails
   - A/B test thumbnails

## Troubleshooting Guide

### Common Issues

1. **"YouTube OAuth Error"**
   - Regenerate refresh token
   - Verify scopes include youtube.upload

2. **"Video Generation Timeout"**
   - Increase Lambda timeout
   - Check provider API status

3. **"DynamoDB Throttling"**
   - Switch to provisioned capacity
   - Reduce concurrent operations

4. **"No Videos Being Generated"**
   - Check if ideas exist with status "pending"
   - Review video generator logs
   - Verify API key is valid

5. **"Uploads Failing"**
   - Check YouTube API quota
   - Verify video file is valid
   - Test OAuth credentials

For more help, see README.md or check CloudWatch logs.