Data Structures and Operations Reference

# Data Structures and Operations Reference

This document describes all data structures in the Growth Marketing Experimentation System and the operations performed on each entity. Use this as a reference for backend database design and API endpoint planning.

---

## Table of Contents

1. [Entity Relationship Overview](#entity-relationship-overview)
2. [Core Entities](#core-entities)
   - [Product](#product)
   - [Hypothesis](#hypothesis)
   - [CreativeVariant](#creativevariant)
   - [MetricsSnapshot](#metricssnapshot)
   - [Insight](#insight)
   - [Belief](#belief)
3. [Supporting Entities](#supporting-entities)
   - [ProductDefinition](#productdefinition)
   - [ProductRegistry](#productregistry)
   - [EvidenceReference](#evidencereference)
   - [KernelState](#kernelstate)
   - [PlatformSpecs](#platformspecs)
4. [Enums](#enums)
5. [Operations by Entity](#operations-by-entity)
6. [API Endpoint Suggestions](#api-endpoint-suggestions)

---

## Entity Relationship Overview

```
┌─────────────────────────────────────────────────────────────────┐
│                         PRODUCT                                  │
│  (Aggregate Root - isolated state per product)                   │
├─────────────────────────────────────────────────────────────────┤
│  ├── ProductDefinition (embedded)                                │
│  │                                                               │
│  └── KernelState (per-product)                                  │
│       ├── Hypotheses (1:N)                                      │
│       │    └── CreativeVariants (1:N, platform-specific)        │
│       │                                                          │
│       ├── Insights (1:N)                                        │
│       │                                                          │
│       └── Beliefs (1:N)                                         │
│            └── EvidenceReferences (1:N)                         │
│                                                                  │
│  MetricsSnapshots (linked to Hypothesis, platform-specific)     │
└─────────────────────────────────────────────────────────────────┘

Platform (lookup table - not per-product)
  └── PlatformSpecs
```

---

## Core Entities

### Product

The top-level aggregate root. Each product has isolated state (hypotheses, variants, beliefs).

**Fields:**

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `product_id` | string | Yes | Unique identifier (UUID) |
| `definition` | ProductDefinition | Yes | Synthesized product info (embedded) |
| `source_urls` | string[] | No | URLs used to extract product info |
| `additional_text` | string | No | User-provided additional context |
| `enabled_channels` | string[] | No | Marketing channels enabled |
| `created_at` | ISO datetime | Yes | Creation timestamp |
| `updated_at` | ISO datetime | Yes | Last update timestamp |

**Storage:** `products/<product_id>/context.json`

---

### Hypothesis

The atomic unit of learning. A structured, testable marketing claim.

**Fields:**

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `hypothesis_id` | string | Yes | Unique ID (format: "H-{timestamp}") |
| `statement` | string | Yes | The testable claim (max 500 chars) |
| `independent_variable` | string | Yes | Variable being tested (e.g., "headline", "image") |
| `dependent_metric` | string | Yes | Metric to measure (e.g., "ctr", "cpa") |
| `audience_scope` | string | Yes | Target audience definition |
| `expected_direction` | enum | Yes | "increase", "decrease", "change" |
| `confidence_level` | enum | Yes | "low", "medium", "high" |
| `status` | enum | Yes | See HypothesisStatus enum |
| `created_at` | ISO datetime | Yes | Creation timestamp |
| `updated_at` | ISO datetime | No | Last update timestamp |
| `expected_magnitude` | string | No | Expected size of effect (e.g., "20-30%") |
| `conclusion` | enum | No | "confirmed", "refuted", "inconclusive" |
| `abandonment_reason` | enum | No | Reason if abandoned |
| `evidence_summary` | string | No | Summary of test results |
| `rationale` | string | No | Why hypothesis was proposed |
| `psychological_trigger` | string | No | Psychological principle being tested |
| `risk_factors` | string | No | Potential risks |
| `success_criteria` | string | No | What success looks like |
| `test_duration_suggestion` | string | No | Suggested test duration |
| `budget_suggestion` | string | No | Suggested budget |
| `creative_brief` | string | No | Brief for creative development |
| `data_quality_flags` | string[] | No | Quality flags (e.g., "low_sample_size") |
| `creative_variants` | CreativeVariant[] | No | Associated variants (embedded) |

**Business Rules:**
- Cannot be deleted (soft delete only via abandonment)
- Cannot activate without >= 3 variants per platform
- Conclusion is immutable once set
- Status transitions: proposed → approved → active → concluded/abandoned

**Storage:** Part of `products/<product_id>/state.json` under `hypotheses` dict

---

### CreativeVariant

A specific ad creative used in an experiment. Platform-specific.

**Fields:**

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `variant_id` | string | Yes | Unique ID (format: "V-{sequence}") |
| `asset_type` | string | Yes | Ad format (e.g., "single_image", "video") |
| `asset_reference` | string | Yes | Reference to creative asset |
| `description` | string | Yes | Brief description of variant approach |
| `created_at` | ISO datetime | Yes | Creation timestamp |
| `platform_id` | string | No | Platform identifier ("facebook", "linkedin") |
| `primary_text` | string | No | Main ad copy (platform: intro_text for LinkedIn) |
| `headline` | string | No | Ad headline |
| `link_description` | string | No | Description below headline |
| `cta_button` | string | No | Call-to-action button text |
| `hook` | string | No | Attention-grabbing opening |
| `angle` | string | No | Persuasion angle |
| `rationale` | string | No | Why variant was created |
| `psychological_angle` | string | No | Psychological lever used |
| `target_emotion` | string | No | Emotion being evoked |
| `differentiation` | string | No | How it differs from others |
| `image_description` | string | No | Ideal image description |
| `image_style` | string | No | Visual style direction |
| `image_mood` | string | No | Mood/atmosphere |

**Business Rules:**
- Platform_id is required for multi-platform support
- Each platform must have >= 3 variants for hypothesis approval
- Embedded within Hypothesis (not a separate table)

---

### MetricsSnapshot

Performance data for a hypothesis over a time period.

**Fields:**

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `hypothesis_id` | string | Yes | FK to Hypothesis |
| `period_start` | ISO datetime | Yes | Start of measurement period |
| `period_end` | ISO datetime | Yes | End of measurement period |
| `impressions` | integer | Yes | Number of impressions |
| `clicks` | integer | Yes | Number of clicks |
| `conversions` | integer | Yes | Number of conversions |
| `spend` | float | Yes | Ad spend in dollars |
| `platform_id` | string | No | Platform the metrics are from |

**Business Rules:**
- Immutable (frozen dataclass)
- clicks <= impressions
- conversions <= clicks
- period_end > period_start

**Derived Metrics (computed, not stored):**
- CTR = clicks / impressions
- CPA = spend / conversions
- CPC = spend / clicks

---

### Insight

A validated learning extracted from concluded hypotheses.

**Fields:**

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `insight_id` | string | Yes | Unique ID (format: "I-{timestamp}") |
| `statement` | string | Yes | The insight claim |
| `insight_class` | enum | Yes | "messaging", "audience", "channel_mechanics" |
| `evidence_hypothesis_ids` | string[] | Yes | References to supporting hypotheses |
| `confidence` | enum | Yes | "low", "medium", "high" |
| `reusability` | enum | Yes | "single_use", "limited", "broad" |
| `status` | enum | Yes | "proposed", "confirmed", "retired" |
| `scope` | string | Yes | Applicability scope |
| `discovered_at` | ISO datetime | Yes | Discovery timestamp |
| `last_validated_at` | ISO datetime | No | Last validation timestamp |
| `decay_flag` | boolean | No | Whether insight is decaying |

**Business Rules:**
- Must reference at least one concluded hypothesis

**Storage:** Part of `products/<product_id>/state.json` under `insights` dict

---

### Belief

A persistent claim that guides future decisions.

**Fields:**

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `belief_id` | string | Yes | Unique ID (format: "B-{timestamp}") |
| `claim` | string | Yes | The belief statement |
| `confidence` | float | Yes | 0.1 to 1.0 |
| `source` | enum | Yes | "insight", "seed", "inherited" |
| `scope` | enum | Yes | "global", "segment_specific", "creative_specific" |
| `lifecycle_state` | enum | Yes | See BeliefLifecycleState enum |
| `formed_at` | ISO datetime | Yes | Formation timestamp |
| `last_updated_at` | ISO datetime | Yes | Last update timestamp |
| `scope_detail` | string | No | Additional scope info |
| `evidence_for` | EvidenceReference[] | No | Supporting evidence |
| `evidence_against` | EvidenceReference[] | No | Contradicting evidence |

**Business Rules:**
- Confidence must be in [0.1, 1.0]
- Evidence references are append-only
- Decays over time without new evidence
- No untraceable beliefs (source is required)

**Storage:** Part of `products/<product_id>/state.json` under `beliefs` dict

---

## Supporting Entities

### ProductDefinition

Synthesized product information (embedded in Product).

| Field | Type | Description |
|-------|------|-------------|
| `name` | string | Product name |
| `tagline` | string | Short tagline |
| `description` | string | Product description |
| `target_audience` | string | Target audience description |
| `value_propositions` | string[] | Value propositions |
| `key_benefits` | string[] | Key benefits |
| `brand_voice` | string | Brand voice/tone |
| `unique_selling_points` | string[] | USPs |
| `pain_points_addressed` | string[] | Pain points addressed |
| `price_positioning` | string | Price positioning |
| `call_to_action_suggestions` | string[] | Suggested CTAs |

---

### ProductRegistry

Tracks all products and active selection.

| Field | Type | Description |
|-------|------|-------------|
| `products` | Dict[string, string] | product_id → display_name |
| `active_product_id` | string | Currently active product |
| `version` | string | Registry format version |

**Storage:** `products/registry.json`

---

### EvidenceReference

Reference to evidence from a hypothesis (embedded in Belief).

| Field | Type | Description |
|-------|------|-------------|
| `hypothesis_id` | string | FK to Hypothesis |
| `strength` | enum | "strong", "weak" |
| `added_at` | ISO datetime | When evidence was added |

---

### KernelState

Container for all per-product state.

| Field | Type | Description |
|-------|------|-------------|
| `hypotheses` | Dict[string, Hypothesis] | All hypotheses |
| `insights` | Dict[string, Insight] | All insights |
| `beliefs` | Dict[string, Belief] | All beliefs |
| `is_halted` | boolean | Emergency halt flag |
| `halt_reason` | string | Reason for halt |

**Storage:** `products/<product_id>/state.json`

---

### PlatformSpecs

Platform-specific constraints (lookup table).

| Field | Type | Description |
|-------|------|-------------|
| `name` | string | Display name (e.g., "Facebook Ads") |
| `platform_id` | string | Unique ID (e.g., "facebook") |
| `character_limits` | Dict | Field → {visible, max} limits |
| `image_specs` | Dict | Image requirements |
| `cta_options` | string[] | Available CTAs |
| `ad_formats` | string[] | Supported formats |
| `objectives` | string[] | Campaign objectives |
| `audience_types` | string[] | Targeting options |
| `key_metrics` | Dict | Metric benchmarks |

**Note:** This is config/lookup data, not stored per-product.

---

## Enums

### HypothesisStatus
```
PROPOSED → APPROVED → ACTIVE → CONCLUDED
                           └→ ABANDONED
```
Values: `proposed`, `approved`, `active`, `concluded`, `abandoned`

### HypothesisConclusion
Values: `confirmed`, `refuted`, `inconclusive`

### ConfidenceLevel
Values: `low`, `medium`, `high`

### InsightClass
Values: `messaging`, `audience`, `channel_mechanics`

### InsightConfidence
Values: `low`, `medium`, `high`

### BeliefScope
Values: `global`, `segment_specific`, `creative_specific`

### BeliefLifecycleState
Values: `emerging`, `established`, `strong`, `challenged`, `retired`

### EvidenceStrength
Values: `strong`, `weak`

### AbandonmentReason
Values: `spend_cap`, `time_limit`, `early_stop`, `policy_block`, `human_override`

---

## Operations by Entity

### Product Operations

| Operation | CLI Option | Method | Description |
|-----------|------------|--------|-------------|
| Create | 0 | `create_product()` | Create new product from URLs/text |
| List | 0 | `list_products()` | Get all products |
| Get | 0 | `get_product()` | Get single product details |
| Switch | 0 | `switch_product()` | Change active product |
| Update | 0 | `update_product_context()` | Update product definition |
| Delete | 0 | `delete_product()` | Delete product |

### Hypothesis Operations

| Operation | CLI Option | Method | Description |
|-----------|------------|--------|-------------|
| Create | 1 | `create_hypothesis()` | Strategy agent proposes hypothesis |
| List | 8 | `list_hypotheses()` | Get all hypotheses (with filters) |
| Get | 8 | `get_hypothesis()` | Get single hypothesis with variants |
| Approve | 3 | `approve_hypothesis()` | Transition proposed → approved |
| Activate | 4 | `activate_hypothesis()` | Transition approved → active |
| Conclude | 7 | `conclude_hypothesis()` | Set conclusion and transition to concluded |
| Abandon | 8 | `abandon_hypothesis()` | Mark as abandoned with reason |
| Delete | 14 | `delete_hypothesis()` | Soft delete (mark as deleted) |
| Set Flag | 11 | `set_data_quality_flag()` | Add quality flag |
| Clear Flag | 11 | `clear_data_quality_flag()` | Remove quality flag |

### CreativeVariant Operations

| Operation | CLI Option | Method | Description |
|-----------|------------|--------|-------------|
| Generate | 2 | `generate_variants()` | Creative agent generates for platform |
| List | 9 | (via get_hypothesis) | Get all variants for hypothesis |
| Get | 9 | (via get_hypothesis) | Get single variant details |
| Edit | 9 | `update_variant()` | Edit variant fields |
| Delete | 9 | `delete_variant()` | Remove variant from hypothesis |
| Regenerate | 9 | `regenerate_variants()` | Generate fresh variants |
| Export | 15 | `export_variants_for_ads()` | Export for platform (with filter) |
| Validate | - | `validate_creative()` | Validate against platform specs |

### MetricsSnapshot Operations

| Operation | CLI Option | Method | Description |
|-----------|------------|--------|-------------|
| Ingest | 5 | `ingest_metrics_manual()` | Record metrics for platform |
| Get Latest | - | (via harness) | Get most recent snapshot |
| Aggregate | - | `merge_metrics_snapshots()` | Combine multiple snapshots |

### Insight Operations

| Operation | CLI Option | Method | Description |
|-----------|------------|--------|-------------|
| Create | - | (future) | Create from concluded hypothesis |
| List | - | (future) | Get all insights |
| Confirm | - | (future) | Confirm proposed insight |
| Retire | - | (future) | Mark as retired |

### Belief Operations

| Operation | CLI Option | Method | Description |
|-----------|------------|--------|-------------|
| List | 9 | `list_beliefs()` | Get all beliefs |
| Get | 9 | `get_belief()` | Get belief details |
| Update | - | (future) | Update confidence based on evidence |
| Decay | - | (automatic) | Confidence decays over time |

### System Operations

| Operation | CLI Option | Method | Description |
|-----------|------------|--------|-------------|
| Halt | 12 | `halt_system()` | Emergency halt |
| Resume | 12 | `resume_system()` | Resume from halt |
| Summary | 13 | `generate_daily_summary()` | Generate report |
| Events | 10 | `get_events()` | Get event log |

---

## API Endpoint Suggestions

Based on the entities and operations above, here are suggested REST API endpoints:

### Products
```
POST   /api/products                    # Create product
GET    /api/products                    # List products
GET    /api/products/:id                # Get product
PUT    /api/products/:id                # Update product
DELETE /api/products/:id                # Delete product
POST   /api/products/:id/activate       # Set as active product
```

### Hypotheses
```
POST   /api/products/:pid/hypotheses                    # Create hypothesis
GET    /api/products/:pid/hypotheses                    # List hypotheses
GET    /api/products/:pid/hypotheses/:hid               # Get hypothesis
POST   /api/products/:pid/hypotheses/:hid/approve       # Approve
POST   /api/products/:pid/hypotheses/:hid/activate      # Activate
POST   /api/products/:pid/hypotheses/:hid/conclude      # Conclude
POST   /api/products/:pid/hypotheses/:hid/abandon       # Abandon
DELETE /api/products/:pid/hypotheses/:hid               # Delete
PUT    /api/products/:pid/hypotheses/:hid/flags         # Update flags
```

### Variants
```
POST   /api/products/:pid/hypotheses/:hid/variants          # Generate variants
GET    /api/products/:pid/hypotheses/:hid/variants          # List variants
GET    /api/products/:pid/hypotheses/:hid/variants/:vid     # Get variant
PUT    /api/products/:pid/hypotheses/:hid/variants/:vid     # Update variant
DELETE /api/products/:pid/hypotheses/:hid/variants/:vid     # Delete variant
POST   /api/products/:pid/hypotheses/:hid/variants/export   # Export variants
```

Query params for variants:
- `?platform_id=facebook` - Filter by platform

### Metrics
```
POST   /api/products/:pid/hypotheses/:hid/metrics       # Ingest metrics
GET    /api/products/:pid/hypotheses/:hid/metrics       # Get metrics history
```

### Analysis
```
POST   /api/products/:pid/hypotheses/:hid/analyze       # Analyst propose conclusion
```

### System
```
POST   /api/products/:pid/halt                          # Halt system
POST   /api/products/:pid/resume                        # Resume system
GET    /api/products/:pid/summary                       # Get summary report
GET    /api/products/:pid/events                        # Get event log
```

### Platforms (lookup/config)
```
GET    /api/platforms                                   # List available platforms
GET    /api/platforms/:id                               # Get platform specs
```

---

## Database Schema Suggestions

### Relational (PostgreSQL)

```sql
-- Products
CREATE TABLE products (
    product_id UUID PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    tagline TEXT,
    description TEXT,
    target_audience TEXT,
    brand_voice TEXT,
    price_positioning TEXT,
    source_urls JSONB,
    additional_text TEXT,
    enabled_channels JSONB,
    value_propositions JSONB,
    key_benefits JSONB,
    unique_selling_points JSONB,
    pain_points_addressed JSONB,
    call_to_action_suggestions JSONB,
    is_active BOOLEAN DEFAULT FALSE,
    created_at TIMESTAMPTZ NOT NULL,
    updated_at TIMESTAMPTZ NOT NULL
);

-- Hypotheses
CREATE TABLE hypotheses (
    hypothesis_id VARCHAR(50) PRIMARY KEY,
    product_id UUID REFERENCES products(product_id),
    statement TEXT NOT NULL,
    independent_variable VARCHAR(100) NOT NULL,
    dependent_metric VARCHAR(100) NOT NULL,
    audience_scope TEXT NOT NULL,
    expected_direction VARCHAR(20) NOT NULL,
    confidence_level VARCHAR(20) NOT NULL,
    status VARCHAR(20) NOT NULL,
    expected_magnitude VARCHAR(100),
    conclusion VARCHAR(20),
    abandonment_reason VARCHAR(50),
    evidence_summary TEXT,
    rationale TEXT,
    psychological_trigger TEXT,
    risk_factors TEXT,
    success_criteria TEXT,
    test_duration_suggestion VARCHAR(100),
    budget_suggestion VARCHAR(100),
    creative_brief TEXT,
    data_quality_flags JSONB,
    created_at TIMESTAMPTZ NOT NULL,
    updated_at TIMESTAMPTZ
);

-- Creative Variants
CREATE TABLE creative_variants (
    variant_id VARCHAR(50) PRIMARY KEY,
    hypothesis_id VARCHAR(50) REFERENCES hypotheses(hypothesis_id),
    platform_id VARCHAR(50),
    asset_type VARCHAR(50) NOT NULL,
    asset_reference TEXT,
    description TEXT NOT NULL,
    primary_text TEXT,
    headline TEXT,
    link_description TEXT,
    cta_button VARCHAR(100),
    hook TEXT,
    angle TEXT,
    rationale TEXT,
    psychological_angle TEXT,
    target_emotion VARCHAR(100),
    differentiation TEXT,
    image_description TEXT,
    image_style VARCHAR(100),
    image_mood VARCHAR(100),
    created_at TIMESTAMPTZ NOT NULL
);

-- Metrics
CREATE TABLE metrics_snapshots (
    id SERIAL PRIMARY KEY,
    hypothesis_id VARCHAR(50) REFERENCES hypotheses(hypothesis_id),
    platform_id VARCHAR(50),
    period_start TIMESTAMPTZ NOT NULL,
    period_end TIMESTAMPTZ NOT NULL,
    impressions INTEGER NOT NULL,
    clicks INTEGER NOT NULL,
    conversions INTEGER NOT NULL,
    spend DECIMAL(12,2) NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Platforms (lookup)
CREATE TABLE platforms (
    platform_id VARCHAR(50) PRIMARY KEY,
    name VARCHAR(100) NOT NULL,
    character_limits JSONB,
    image_specs JSONB,
    cta_options JSONB,
    ad_formats JSONB,
    objectives JSONB,
    audience_types JSONB,
    key_metrics JSONB
);

-- Indexes
CREATE INDEX idx_hypotheses_product ON hypotheses(product_id);
CREATE INDEX idx_hypotheses_status ON hypotheses(status);
CREATE INDEX idx_variants_hypothesis ON creative_variants(hypothesis_id);
CREATE INDEX idx_variants_platform ON creative_variants(platform_id);
CREATE INDEX idx_metrics_hypothesis ON metrics_snapshots(hypothesis_id);
```

---

## Notes for Backend Implementation

1. **Product Isolation**: Each product's state is fully isolated. The current implementation uses file-based storage with separate directories per product.

2. **Hypothesis-Variant Relationship**: Variants are currently embedded in hypotheses. For a database, normalize to a separate table with FK.

3. **Metrics History**: Store all metrics snapshots for historical analysis and trend detection.

4. **Platform is Config**: Platform specs are configuration data, not user data. Can be stored in code or a config table.

5. **Event Sourcing**: Consider event sourcing for hypothesis state changes to maintain complete audit trail.

6. **Soft Deletes**: Hypotheses should use soft delete (abandonment) rather than hard delete.

7. **Concurrency**: Multiple users may work on same product - consider optimistic locking on hypothesis updates.