# Lecture 3: Development Through Specifications and AI Agents ## Metadata **Course**: AI Talent Hub, Master's Program **Date**: October 1, 2025 **Format**: Interactive seminar with an AI co-host **Instructors**: Alexey (Lyosha) + Claude (AI) + Nikita Kononov **Terms and Definitions** **Specification** = a set of verifiable requirements (BDD/ATDD) + API/data contracts. **Test** = an executable specification. **Contract** = a formal schema describing the interaction between components (versioned). **RGR cycle** = Red-Green-Refactor as a unit of iteration. **SLO** = target boundaries for latency/cost/reliability, verified by tests. **Key idea:** This lecture reveals a paradigm shift in software development: we move away from the "write code, then verify it" model to "first describe in detail what we want, then generate the solution." In this approach, tests are no longer just a verification step — they become an exhaustive specification for a team of AI agents. This enables a radically faster, cheaper, and — most importantly — more predictable and flexible development process. ### 1. The New Paradigm: From Code to Specification The traditional approach, where a developer first writes code and then tests, is outdated. It was justified when writing code was slow and expensive. In the age of AI, which can generate thousands of lines of code in minutes, manually verifying such a volume is impossible. Therefore, the focus shifts to creating precise, machine-readable specifications that AI can understand and implement. **Comparing approaches:** **Traditional development:** ```mermaid graph TD T1[Idea] --> T2[Write code] T2 --> T3[Find bugs] T3 --> T4[Fix code] T4 --> T5[Write tests] T5 --> T6{Tests
failing?} T6 -->|Yes| T4 T6 -->|No| T7[Deploy] style T1 fill:#e9ecef style T2 fill:#ff6b6b,color:#fff style T3 fill:#ff6b6b,color:#fff style T4 fill:#ff6b6b,color:#fff style T5 fill:#ffd43b style T6 fill:#ffd43b style T7 fill:#51cf66,color:#fff ``` **AI agents + Specifications:** ```mermaid graph TD A1[Idea] --> A2[Write tests/
specification] A2 --> A3{Tests
approved?} A3 -->|No| A2 A3 -->|Yes| A4[AI generates
code] A4 --> A5{Tests
passing?} A5 -->|No| A4 A5 -->|Yes| A6[Deploy] style A1 fill:#e9ecef style A2 fill:#51cf66,color:#fff style A3 fill:#51cf66,color:#fff style A4 fill:#51cf66,color:#fff style A5 fill:#51cf66,color:#fff style A6 fill:#51cf66,color:#fff ``` **Key differences:** | Aspect | Traditional | AI Agents | |--------|-------------|-----------| | **What does the human write** | Code | Specifications/Tests | | **Iteration speed** | Hours to days | Minutes to hours | | **Cost of changes** | High (refactoring) | Low (RMRF + regeneration) | | **Entry barrier** | Knowledge of languages/frameworks | Ability to formulate requirements | | **When bugs are found** | After writing code | Before writing code | | **Human's role** | Developer | Product Owner / CEO | - **The central role of testing:** Tests cease to be merely a post-implementation verification tool. They become a formal contract describing **what** the system should do, not **how** it does it. The AI agent reads this contract and writes code to fulfill it. - **Methodologies in service of AI:** - **TDD (Test-Driven Development):** This is the fundamental cycle. For example, instead of immediately writing an addition function, you first write a test: `assert add(2, 3) == 5`. This test naturally fails (the function does not exist yet). Then you write the simplest code to make the test pass: `def add(a, b): return 5`. The test passes! Only then do you improve the code to `def add(a, b): return a + b`, confident that nothing is broken. This is the `Red-Green-Refactor` cycle. ```mermaid graph LR A[Red: Write a test] --> B[Test fails] B --> C[Green: Write minimal code] C --> D[Test passes] D --> E[Refactor: Improve code] E --> F[Tests still pass] F --> A style A fill:#ff6b6b style C fill:#51cf66 style E fill:#339af0 ``` - **BDD (Behavior-Driven Development):** This is an evolution of TDD that translates tests into human language. Instead of technical syntax, we use near-natural language (often in Gherkin format), understandable to the entire team. For example: ```gherkin Feature: User Authentication Scenario: User logs in with valid credentials Given a registered user with username "test" and password "pass123" When they enter their username and password on the login page Then they should see their personal dashboard Scenario: User enters an incorrect password Given a registered user with username "test" When they enter username "test" and incorrect password "wrong_pass" Then they should see an error message "Invalid username or password" ``` **BDD test lifecycle with AI agents:** ```mermaid sequenceDiagram participant PM as Product Owner participant M as Manager participant SW as Spec Writer participant IMPL as Implementation Agent participant TEST as Test Runner PM->>M: "We need user authentication" M->>SW: Create BDD scenarios SW->>SW: Writes Gherkin Note over SW: Feature: Authentication
Scenario: Successful login
Given/When/Then SW->>M: BDD scenarios ready M->>PM: For approval PM->>M: Approved M->>IMPL: Implement according to BDD IMPL->>IMPL: Generates code IMPL->>TEST: Run BDD tests TEST->>TEST: Executes scenarios alt Tests green TEST->>M: All scenarios passed M->>PM: Ready for deployment else Tests red TEST->>IMPL: Scenario X failed IMPL->>IMPL: Fixes code IMPL->>TEST: Re-run end ``` **Benefits of BDD with AI:** - The Product Owner writes in natural language without knowing how to code - The specification simultaneously serves as documentation and tests - AI understands Gherkin and generates code to match the scenarios - Changing requirements = changing Gherkin, and AI automatically adapts the code - **ATDD (Acceptance Test-Driven Development):** This is BDD at a higher level. Tests describe not individual functions but entire user stories that matter to the business (acceptance criteria). **Catalog of Non-Functional Requirements (NFR) Verified by Tests** - Latency: P95 response time <= X ms. - Cost: $ per scenario <= Y. Guardrails with auto-triggering. - Reliability: share of green runs >= Z% over N runs. - Security: secrets policy, access controls, and audit logging. - Reproducibility: fixed seeds, versioned datasets/artifacts. - Compatibility: support for L OS versions/architectures; contract tests for older/newer schema versions. ### 2. Designing the AI Agent Team Imagine not a single brilliant but chaotic AI, but a well-coordinated team of specialists, like in a real IT company. Each agent has its own narrow specialization, which solves the problem of mixing contexts and improves the quality of the final product. **Basic team structure:** - **Manager:** The project manager of our AI team. They **never write code themselves**. Their job is to coordinate other agents, ensure the process follows the plan, tasks are routed to the right specialists, and the team does not get stuck. - **Domain Expert:** The researcher. Their task is to understand not *what* the client wants, but *why* they want it. - **Example question:** Instead of "What kind of chatbot do you need?", they would ask: "Tell me about the *worst* day in your recruiter's work. What frustrates them the most and takes up the most time?" This helps uncover the real pain point — for example, not the conversation with the candidate itself, but the inability to reach them. - **Product Analyst:** This agent takes the "investigation" results from the Domain Expert and turns them into specific, measurable business requirements. - **Example output:** "Based on the finding that recruiters spend hours listening to dial tones, the key success metric will be a 40% increase in conversion from 'contact attempt' to 'conversation started'." - **Spec Writer:** The team's technical writer. They translate business requirements from human language into the formal language of BDD tests. Their job is to create a clear, unambiguous "instruction manual" for the implementation agents. - **Contract Agent:** The API architect. If our system consists of multiple parts (e.g., a mobile app and a server), this agent designs the "agreement" between them. - **Example contract:** `GET /api/user/{id}` should return JSON in the format: `{"id": int, "name": string, "email": string, "is_active": boolean}`. Any deviation from this schema will be considered an error. - **Implementation Agent:** These are the "developers" on our team. They receive only tests and contracts as input and write code whose sole purpose is to make the tests pass. They are unaware of business goals, which protects them from misinterpretation. - **Mutator/Breaker:** The "hacker-tester." After the code is written, they deliberately try to break it. - **Example check:** "What if the user enters the text 'twenty' instead of the number 20 in the age field? What if they enter a negative number? The system should handle the error gracefully rather than crash." - **Refactoring Agent:** The "code cleaner." Once the system works, they come in and make the code better: faster, more efficient, more readable for other agents (and humans). They do not add new functionality — only improve what already exists. Additional possible agents: - **Product/Business Analyst**: formalizes business success metrics and SLOs. - **Market Research Agent**: aggregates "voice of customer" from external sources. - **Data Steward**: data policy, anonymization, dataset versioning, licenses. - **Security Agent**: threat model, secrets, RBAC, risk report. - **Performance Agent**: profiling, CPU/RAM/IO budgets, load test scenarios. - **Observability Agent**: metrics/logs/traces, SLO-based alerts. - **Release Manager**: semantic versioning, changelog, release candidates. ```mermaid graph TD H[Human Client] <--> M[Manager] M --> DE[Domain Expert] M --> PA[Product Analyst] DE --> SW[Spec Writer] PA --> SW SW --> |Tests| M M --> |Tests for approval| H H --> |Approved/Revisions| M M --> CA[Contract Agent] CA --> |API contracts| M M --> |Contracts for approval| H H --> |Approved/Revisions| M M --> IMPL[Implementation Agents] IMPL --> BE[Backend Agent] IMPL --> FE[Frontend Agent] IMPL --> DATA[Data Agent] BE --> INT[Integration] FE --> INT DATA --> INT INT --> MUT[Mutator/Breaker] MUT --> |Bugs found| IMPL MUT --> |Escalation| M M --> |Critical issues| H MUT --> |All OK| REF[Refactoring Agent] REF --> M M --> |Result| H H --> |Approved| DONE[Ready for production] style H fill:#ff6b6b,color:#fff style M fill:#339af0,color:#fff style SW fill:#51cf66,color:#fff style CA fill:#ffd43b style MUT fill:#ff6b6b,color:#fff style DONE fill:#51cf66,color:#fff ``` **AI agent team architecture** Key interaction points: - **Single point of contact**: The human communicates ONLY with the Manager — all requests, approvals, and escalations go through them - **Manager as a filter**: receives tests from the Spec Writer and contracts from the Contract Agent, passes them to the human for approval - **Parallel development**: Backend, Frontend, and Data work simultaneously after contracts are approved - **Feedback loops**: Mutator finds issues -> Implementation fixes them; for critical issues -> escalation through the Manager - **Manager coordinates** but never writes code **Test pyramid for the AI workflow** ```mermaid graph TB subgraph "Test Pyramid for AI Development" E2E["E2E Tests
Full user scenarios
Slow, expensive, brittle"] INT["Integration & Contract Tests
Interaction between components
API contracts between agents
Medium speed"] UNIT["Unit & Property-Based Tests
Isolated logic
Fast, cheap, reliable
Thousands of generated test cases"] SPECIAL["Special Test Types"] end subgraph SPECIAL MUT["Mutation Tests
Verify test strength
Intentionally break code"] APPR["Approval Tests
UI/report snapshots
Detect visual regressions"] PERF["Performance Tests
SLO for latency/cost
Load testing"] end E2E --> INT INT --> UNIT UNIT -.-> SPECIAL style E2E fill:#ff6b6b,color:#fff style INT fill:#ffd43b style UNIT fill:#51cf66,color:#fff style MUT fill:#339af0,color:#fff style APPR fill:#339af0,color:#fff style PERF fill:#339af0,color:#fff ``` **Key differences from the traditional pyramid:** - **Property-based tests at the base**: AI generates thousands of test cases automatically - **Contract tests are critical**: they ensure synchronization of parallel agent work - **Mutation tests are mandatory**: they verify that tests actually catch bugs - **Approval tests for AI-generated UI**: quickly detect visual changes ### 3. The Workflow and the Human's Role The AI team does not work in a vacuum. The human remains at the center as the decision-making "CEO" of the project, setting direction and approving key results. **Step-by-step process:** ```mermaid sequenceDiagram participant H as Human participant M as Manager participant DE as Domain Expert participant SW as Spec Writer participant CA as Contract Agent participant BE as Backend Agent participant FE as Frontend Agent participant MUT as Mutator participant REF as Refactoring Agent H->>M: Task description M->>DE: Research the problem domain DE->>M: Clarifying questions M->>H: Questions from the Domain Expert H->>M: Answers M->>DE: Client's answers DE->>SW: Passing insights SW->>SW: Writing BDD tests SW->>M: Tests ready M->>H: Tests for approval H->>M: Approved/Revisions M->>SW: Final revisions (if needed) M->>CA: Request contracts CA->>CA: Designing the API CA->>M: Contracts ready M->>H: Contracts for approval H->>M: Approved par Parallel development M->>BE: Implement Backend M->>FE: Implement Frontend end BE->>MUT: Code for testing FE->>MUT: Code for testing MUT->>MUT: Searching for vulnerabilities alt Critical issues found MUT->>M: Escalation: critical bugs M->>H: Decision required H->>M: Decision/Direction M->>BE: Fix according to the decision M->>FE: Fix according to the decision else Regular bugs MUT->>BE: Bug report MUT->>FE: Bug report BE->>MUT: Fixed code FE->>MUT: Fixed code else All OK MUT->>REF: Code passed review end REF->>REF: Code optimization REF->>M: Ready for review M->>H: Work result for review H->>M: Approved M->>H: Deploy to production ``` **Key process moments:** 1. **Interview through the Manager:** `Domain Expert` and `Product Analyst` formulate questions, which the `Manager` relays to the human. This often happens via voice for more effective communication. The human answers through the `Manager`, who passes the responses to the agents. 2. **Writing tests:** The `Spec Writer` creates a complete set of scenarios and passes them to the `Manager`. 3. **Approving tests (through the Manager):** The `Manager` presents the tests to the human for approval. The human approves or requests revisions. **This is the most important step.** 4. **Designing contracts:** The `Manager` requests contracts from the `Contract Agent`, who designs the API. The `Manager` passes the contracts to the human for approval. 5. **Parallel development:** After contracts are approved, the `Manager` launches parallel development: the Backend agent writes the logic for saving data to the database, while the Frontend agent creates the input form, trusting the API contract. This speeds up the process many times over. 6. **Testing and debugging:** The `Mutator` searches for vulnerabilities. Regular bugs are fixed automatically. For critical issues, the `Manager` escalates them to the human for a decision. 7. **Refactoring:** The `Refactoring Agent` optimizes the solution and passes the result to the `Manager`. 8. **Final review:** The `Manager` presents the final result to the human for approval before deployment. 9. **Iterations:** To add new functionality, the cycle repeats. **Escalation points (when the Manager reaches out to the human):** All interactions go through the `Manager` — they are the sole point of contact between the human and the agent team: - **Approval of acceptance tests and contracts.** - Manager: "The Spec Writer has prepared tests for the registration feature. Please approve [passes tests]" - Human: "Great, but add a test for the short password case" - Manager: "Understood, I'll pass this to the Spec Writer for revision" - **Resolving requirement conflicts.** - Manager: "A conflict was found: Test A requires public access, Test B requires paid-users-only access. Which rule is correct?" - **Making key architectural decisions.** - Manager: "The Domain Expert recommends choosing between a monolith (faster start) and microservices (future flexibility). What is your decision?" - **Critical integration issues.** - Manager: "The Mutator found a critical issue: Backend and Frontend are failing to synchronize on the contract after 3 attempts. The Contract Agent suggests revising the schema. Do you approve?" - **Exceeding budget limits.** - Manager: "80% of the monthly API budget has been spent. Options: continue development, optimize token usage, or pause?" **Parallelism and bottlenecks:** - Parallelized after contract approval: Backend/Frontend/DevOps/Data. - Bottlenecks: Spec Writer approval, contract freeze, integration. - Mitigation: incremental integration, separate queues for contract changes. **Escalation and halt policies:** - Test/requirement conflict -> halt, human decision. - Contract break after 3 failed builds -> halt, schema revision. - API budget/build time overrun > X% -> halt, SLO recalculation. - High-severity security risks -> halt, patch/review by the Security Agent. **Versioning, reproducibility, environments:** - SemVer for services and contracts; back/forward-compat tests. - Lockfiles, Docker images with hashes, fixed seeds. - Environments: dev/stage/prod, schema migrations with migration tests. - Artifacts: model/dataset/report registry with immutable references. **Cost management** - Token/time quotas per run; auto-failfast on overrun. - Caching intermediate artifacts; deduplicating runs; using emulators instead of LLM where the task is classification. --- ### 3.1 Timeline: From Idea to Production **Comparing time investment:** ```mermaid gantt title Web Application Development: Traditional vs AI Agents dateFormat X axisFormat %H h section Traditional Requirements analysis :t1, 0, 8h Architecture design :t2, after t1, 16h Backend development :t3, after t2, 40h Frontend development :t4, after t2, 40h Integration :t5, after t3, 16h Testing :t6, after t5, 24h Bug fixing :t7, after t6, 16h Documentation :t8, after t7, 8h section AI Agents Interview + BDD (human) :a1, 0, 2h Test approval (human) :a2, after a1, 1h Contracts (human) :a3, after a2, 1h Backend (AI in parallel) :a4, after a3, 4h Frontend (AI in parallel) :a5, after a3, 4h Integration (AI) :a6, after a4, 2h Mutation tests (AI) :a7, after a6, 1h Refactoring (AI) :a8, after a7, 1h Review (human) :a9, after a8, 1h ``` **Total time:** | Stage | Traditional | AI Agents | Speedup | |-------|-------------|-----------|---------| | **Total time** | 168 hours (3+ weeks) | 13 hours (1-2 days) | **13x** | | **Human time** | 168 hours | 5 hours | **34x** | | **Parallelism** | Minimal | Maximum | — | **Distribution of work between human and AI:** | Stage | Who performs | Time | |-------|-------------|------| | **AI Agents: Human involvement** | | | | Interview + BDD | Human (with agent assistance) | 2 hours | | Test approval | Human | 1 hour | | Contract approval | Human | 1 hour | | Final review | Human | 1 hour | | **Total human time** | | **5 hours** | | **AI Agents: Autonomous work** | | | | Backend development | AI (in parallel with Frontend) | 4 hours | | Frontend development | AI (in parallel with Backend) | 4 hours | | Integration | AI | 2 hours | | Mutation tests | AI | 1 hour | | Refactoring | AI | 1 hour | | **Total AI work** | | **8 hours** (in parallel) | | **Total calendar time** | | **13 hours** | **Key takeaways:** - **Human involvement**: only approvals and strategic decisions (5 hours) - **Autonomous AI work**: all implementation without human involvement (8 hours) - **Parallelism**: Backend and Frontend are developed simultaneously (saving 4 hours) - **Critical path**: test approval -> contract approval -> parallel development -> review --- ### 3.2 API Contract Example Between Agents **Visualizing Backend <-> Frontend interaction:** ```mermaid sequenceDiagram participant FE as Frontend Agent participant API as API Contract participant BE as Backend Agent participant DB as Database Note over API: POST /api/users/register
Contract v1.2.3 FE->>API: Request per schema Note over FE: {
"email": "user@example.com",
"password": "secret123",
"name": "John Doe"
} API->>API: Request validation alt Request is valid API->>BE: Pass data BE->>DB: Save user DB->>BE: user_id: 12345 BE->>API: Response per schema Note over BE: {
"id": 12345,
"email": "user@example.com",
"created_at": "2025-10-01T10:00:00Z"
} API->>FE: 201 Created + data else Request is invalid API->>FE: 400 Bad Request Note over API: {
"error": "invalid_email",
"message": "Email format incorrect"
} else Backend unavailable API->>FE: 503 Service Unavailable end ``` **Contract structure (JSON Schema):** ```json { "$schema": "http://json-schema.org/draft-07/schema", "version": "1.2.3", "endpoint": "POST /api/users/register", "request": { "type": "object", "required": ["email", "password"], "properties": { "email": {"type": "string", "format": "email"}, "password": {"type": "string", "minLength": 8}, "name": {"type": "string"} } }, "responses": { "201": { "description": "User successfully created", "schema": { "type": "object", "properties": { "id": {"type": "integer"}, "email": {"type": "string"}, "created_at": {"type": "string", "format": "date-time"} } } }, "400": {"description": "Invalid request data"}, "503": {"description": "Service temporarily unavailable"} } } ``` **Benefits of contracts:** - Frontend and Backend work in parallel, trusting the contract - The Contract Agent automatically generates validation - Contract changes are versioned (SemVer) - Contract tests automatically verify that both sides comply with the schema --- ### 3.3 Responsibility Matrix (RACI) **Who is responsible for what in the AI team:** ```mermaid graph TD subgraph "RACI Matrix" direction TB H["Human (Product Owner)"] M["Manager"] TEAM["Agent Team"] end subgraph "Tasks and Roles" T1["Requirements formulation"] T2["Writing BDD tests"] T3["Test approval"] T4["Contract design"] T5["Contract approval"] T6["Writing code"] T7["Testing"] T8["Refactoring"] T9["Deployment"] T10["Architectural decisions"] end H -->|"R (Responsible)"| T1 H -->|"A (Accountable)"| T3 H -->|"A (Accountable)"| T5 H -->|"A (Accountable)"| T10 M -->|"R (Responsible)"| T3 M -->|"R (Responsible)"| T5 M -->|"C (Consulted)"| T1 M -->|"I (Informed)"| T6 M -->|"I (Informed)"| T7 TEAM -->|"R (Responsible)"| T2 TEAM -->|"R (Responsible)"| T4 TEAM -->|"R (Responsible)"| T6 TEAM -->|"R (Responsible)"| T7 TEAM -->|"R (Responsible)"| T8 TEAM -->|"C (Consulted)"| T9 style H fill:#ff6b6b,color:#fff style M fill:#339af0,color:#fff style TEAM fill:#51cf66,color:#fff ``` **RACI breakdown:** | Task | Human | Manager | Agents | Explanation | |------|-------|---------|--------|-------------| | **Requirements formulation** | **R** (does) | C (consulted) | C (consulted) | Human formulates, agents help structure | | **Writing BDD tests** | I (informed) | I (informed) | **R** (does) | Spec Writer writes, human approves through Manager | | **Test approval** | **A** (approves) | **R** (presents) | I (informed) | Critical control point | | **Contract design** | I (informed) | I (informed) | **R** (does) | Contract Agent designs | | **Contract approval** | **A** (approves) | **R** (presents) | I (informed) | Critical control point | | **Writing code** | — | I (informed) | **R** (does) | Fully autonomous agent work | | **Testing** | — | I (informed) | **R** (does) | Automated, escalation for critical bugs | | **Refactoring** | — | I (informed) | **R** (does) | Automated optimization | | **Deployment** | **A** (approves) | **R** (executes) | C (consulted) | Final human approval | | **Architectural decisions** | **A** (decides) | **R** (proposes options) | C (consulted) | Strategic decisions rest with the human | **Key principles:** - **Human** = Product Owner, makes strategic decisions - **Manager** = single point of interaction, coordinates and presents results - **Agents** = autonomous team of implementers, experts, and reviewers --- ### 3.4 Development Cost Diagram **Comparing token and time costs:** **Traditional development:** ```mermaid graph LR T_TIME["Time
168 hours
3 weeks"] T_COST["Cost
$8,400
$50/h x 168h"] T_RISK["Risks
High
bugs after release"] T_TIME -.-> T_COST T_COST -.-> T_RISK style T_TIME fill:#ff6b6b,color:#fff style T_COST fill:#ff6b6b,color:#fff style T_RISK fill:#ff6b6b,color:#fff ``` **AI agents:** ```mermaid graph LR A_TIME["Human time
5 hours
1 day"] A_TOKENS["AI tokens
$150-300
depending on complexity"] A_COST["Total
~$500
$50/h x 5h + tokens"] A_RISK["Risks
Low
tests provide coverage"] A_TIME -.-> A_TOKENS A_TOKENS -.-> A_COST A_COST -.-> A_RISK style A_TIME fill:#51cf66,color:#fff style A_TOKENS fill:#ffd43b style A_COST fill:#51cf66,color:#fff style A_RISK fill:#51cf66,color:#fff ``` **Comparison:** - **Time**: 168h vs 5h = **34x speedup** - **Cost**: $8,400 vs $500 = **17x savings (94%)** - **Risks**: high vs low = **predictable outcomes** **Token cost breakdown by stage:** | Stage | Tokens (approximate) | Cost | % of total | |-------|----------------------|------|------------| | **Interview + analysis** | 50k tokens | $7.50 | 3% | | **BDD test generation** | 100k tokens | $15 | 6% | | **Contract design** | 75k tokens | $11.25 | 5% | | **Backend development** | 500k tokens | $75 | 30% | | **Frontend development** | 500k tokens | $75 | 30% | | **Integration and tests** | 200k tokens | $30 | 12% | | **Mutation tests** | 150k tokens | $22.50 | 9% | | **Refactoring** | 100k tokens | $15 | 6% | | **Documentation** | 25k tokens | $3.75 | 2% | | **TOTAL** | ~1.7M tokens | **~$255** | 100% | **Cost distribution chart:** ```mermaid pie title Token Distribution by Stage "Backend" : 30 "Frontend" : 30 "Integration + tests" : 12 "Mutation tests" : 9 "BDD generation" : 6 "Refactoring" : 6 "Contracts" : 5 "Interview" : 3 "Documentation" : 2 ``` **Factors affecting cost:** **Reduce cost:** - Reusing contracts and tests - Caching intermediate results - Using cheaper models for simple tasks - Property-based tests instead of manually writing cases **Increase cost:** - Frequent requirement changes without updating tests - Lack of clear contracts (more integration iterations) - Complex business logic (more edge cases) - High performance requirements (more optimizations) **ROI (Return on Investment):** ``` Traditional development: $8,400 over 3 weeks AI agents: $500 in 1 day Savings: $7,900 (94%) Speedup: 21x in calendar time ``` --- ### 4. Applications Across Different Domains This approach is universal and scales well beyond standard web development. - **Scientific Research (Bioinformatics):** - **Problem:** Impossible to compare old and new tools for finding repeats in genomes. - **Solution:** Creating a benchmarking system that generates synthetic genomes with known parameters. - **Result:** Establishing a standard for an entire scientific field, ensuring that results obtained in different laboratories can be compared. - **ML Research:** - **Problem:** The "reproducibility crisis" in science. - **Solution:** Creating an "experimental test bench." Tests are written for **process reproducibility**, not just final accuracy. - **Result:** Instead of screenshots with metrics in a paper, you can provide a "live" test bench where any other researcher can run your tests and confirm the results. - **Repro-testbed as a live benchmark:** swapping models = preserved tests, comparing metric diffs, publishing artifacts and launch commands. - **Marketing and Promotion:** - **Task:** Make a scientific tool popular. - **Solution:** Creating a `Marketing Agent` that behaves like a "digital science communicator." - **Result:** The agent builds reputation and trust in the product, automatically performing the work of a community manager. - **Case study: Recruiting as multi-channel outreach:** - Task: not dialogue, but contact reachability; channel ranking; anti-spam constraints. - Solution: finite state machine + lightweight classifier for routing; deterministic replies; SLO targeting P95<=2s and cost/session. - Tests: achieving a response within <= K touches; >= M% transitions to a slot; anti-spam limits not violated. - **Anti-patterns and limitations:** - Generating code without tests -> requirement drift. - Manager who writes code -> conflict of interest and blind spots. - Unversioned contracts -> cascading integration failures. - Flaky tests without quarantine -> false regressions and pipeline burnout. ### 5. Conclusions and Practical Recommendations - **Freedom without fear:** The main advantage of this approach is the ability to make any changes to the system, confident that tests will catch all "silent" errors and side effects. It is like insurance for your code. - **From MVP to Production-Ready:** The MVP (Minimum Viable Product) concept was born from limited human resources. When you have a team of tireless AI agents, there is no point in artificially cutting functionality. You can build a robust product from the start, simply adding features iteratively. - **Definition of Done criteria** - All acceptance/contract/load tests are green. - Coverage of critical paths >= T%; no flaky tests in the quarantine list. - Contracts and schemas are versioned, migrations are tested. - Observability and alerts on key SLOs are enabled. - Release artifact, security checklist, changelog. - **Launch checklist:** - Approved SLOs for time/cost/reliability. - Data/secrets/license policies are documented. - Rollout: canary release, before/after metrics, rollback plan. - Legacy deactivation/user migration plan. - **Homework:** Try implementing this pipeline on a simple project. The goal is not so much to get working code, but to **experience the process itself** and the shift in mindset. Play the role of a client for the AI, then the role of a "CEO" approving tests. Start with projects that bring you personal value — that is the best motivation. # Templates for ATDD/BDD, Contracts, SLOs, and Releases Below are ready-made templates. Copy them, fill in the placeholders, and keep them in your repository (`/docs/specs`). --- ## 1) ATDD/BDD Template (Gherkin) ```gherkin # file: features/.feature Feature: As I want So that Background: Given And @acceptance @happy_path Scenario:

Given When Then And @edge @negative Scenario: Given When Then @nfr @performance Scenario: P95 latency <= ms Given When Then response time p95 <= ms ``` **Guidelines:** - Short steps, one verifiable fact per `Then`. - Reflect business vocabulary, not internal implementation details. - Tags (`@acceptance`, `@nfr`, `@security`) — for selective test runs. --- ## 2) Step Definition Template (pseudocode) ```python # file: tests/steps/_steps.py from behave import given, when, then @given("") def _(context): # TODO: initialize fixtures/data ... @when("") def _(context): # TODO: call the system via contract/CLI/API context.result = ... @then("") def _(context): assert ... ``` --- ## 3) Contract Template (JSON Schema + SemVer) ```json { "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://example.com/schemas//.schema.json", "title": " v", "x-contract-version": "", "type": "object", "required": ["id", "version"], "properties": { "id": {"type": "string", "format": "uuid"}, "version": {"type": "string", "pattern": "^\\d+\\.\\d+\\.\\d+$"}, "data": { "type": "object", "properties": { "": {"type": "", "description": ""} }, "additionalProperties": false }, "meta": { "type": "object", "properties": { "created_at": {"type": "string", "format": "date-time"}, "deprecated": {"type": "boolean", "default": false}, "replaced_by": {"type": "string", "nullable": true} }, "additionalProperties": false } }, "additionalProperties": false } ``` **SemVer rules for contracts:** - MAJOR up — breaking change (field removal, type change, semantic change). - MINOR up — backward compatible (adding an optional field). - PATCH up — fixes without schema changes. **Contract tests (checklist):** - Producer validates output against the schema. - Consumer accepts unknown optional fields without crashing. - Back/forward-compat: JSON with previous/next MINOR versions. --- ## 4) Contract Changelog Template ```markdown # Contract Changelog: / ## — - Type: - Changes: - - Impact: - - Actions: - ``` --- ## 5) SLO Card (template) ```markdown # Service: | SLI | SLO Target | Window | Metrics Source | Alert | Error budget | |------------------------------|---------------------|----------|-----------------------|-------|--------------| | P95 latency (endpoint X) | <= ms | 28 days | Prometheus/ | warn | % | | Success rate | >= % | 28 days | Logs/HTTP codes | page | % | | Cost per scenario | <= <$K> | 7 days | Billing exporter | warn | n/a | | Uptime | >= 99.% | 90 days | Uptime probe | page | % | ``` **Guardrails:** auto-failfast if error budget is exhausted; stop load tests if cost increases > X%. --- ## 6) NFT/NFR Test Spec (latency/cost/reliability) ```yaml # file: tests/nfr/_nfr.yaml component: scenarios: - name: p95_latency_endpoint_x load: { rps: , duration: } assert: - metric: latency_p95_ms op: <= value: - name: cost_per_run context: { provider: , model: } assert: - metric: cost_usd op: <= value: - name: reliability_7d assert: - metric: success_rate op: ">=" value: ``` --- ## 7) Release/Rollback Checklist ```markdown # Release Checklist v - [ ] All acceptance/contract/NFR tests are green - [ ] Data migrations run on staging; rollback script ready - [ ] Contracts: SemVer updated, compatibility verified - [ ] Observability: new metrics/logs/traces added - [ ] Security: secrets in vault, tokens with minimal permissions - [ ] Changelog written, ownership assigned - [ ] Canaries: % of traffic, stop criteria, rollback plan # Rollback Plan - [ ] Trigger conditions (SLO breach, incident, regression) - [ ] Rollback team (contacts) - [ ] Step-by-step rollback (artifact versions, down migrations) - [ ] Post-rollback verification (metrics health) ``` --- ## 8) Escalation Policy (template) ```markdown # Escalation Policy - Test/requirement conflict -> pipeline halt, product owner decision - 3 consecutive integration failures -> contract revision (Contract Agent), owner approval - API budget overrun > % per day -> auto-halt, SLO recalculation - Security High/Critical -> immediate halt, patch, postmortem ``` --- ## 9) Observability Spec ```yaml service: dashboards: - name: Latency & Errors charts: - metric: http_server_duration_seconds_bucket view: histogram_quantile_p95 - metric: http_requests_total{code!~"2.."} view: rate - name: Cost charts: - metric: llm_cost_usd view: sum_by(model) alerts: - name: p95_latency_breach expr: p95_latency_ms > for 5m route: page - name: cost_spike expr: increase(llm_cost_usd[1h]) > route: warn ``` --- ## 10) Cost Management Policy ```markdown - Limits: tokens/time/runs per PR - Cache LLM artifacts and use deterministic responses where possible - Replace LLM with lightweight classifier/heuristic for simple tasks - Cost reports in PR comments, target <= <$K> / scenario ``` --- ## 11) Mini Artifact Templates ### 11.1 Gherkin Quick Template ```gherkin Scenario: Given When Then ``` ### 11.2 Contract JSON Message (example) ```json { "id": "", "version": "", "data": { "": "" }, "meta": { "created_at": "", "deprecated": false } } ``` ### 11.3 SLO Entry (single SLI) ```yaml sli: latency_p95_ms slo: <= window: 28d source: prometheus alert: warn budget: % ``` --- ## 12) Definition of Done (DoD) — add to your project README ```markdown - Green acceptance/contract/NFR tests - Coverage of critical paths >= % - Contracts versioned, migrations verified - Observability and alerts enabled - Release artifact + rollback plan ready ```

Education & Training

Lecture 3: Development Through Specifications and AI Agents