**Project Requirements Document: News Scraper & Semantic Search with GenAI**

# **Project Requirements Document: News Scraper & Semantic Search with GenAI**

The following table outlines the detailed functional requirements of the News Scraper & Semantic Search application.

| Requirement ID | Description                                      | User Story                                                                                                           | Expected Behavior/Outcome                                                                                                                                         |
|----------------|--------------------------------------------------|----------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| FR001          | Extracting News Articles                        | As a user, I want the system to fetch full news articles from provided URLs so that I can process and store them.   | The system should scrape the article's headline and full text and return structured data for each URL provided. Implement robust error handling with retries, logging, and skip invalid URLs.                                                   |
| FR002          | Summarizing News Articles                       | As a user, I want the system to summarize the articles so I can quickly understand the key points of each one.      | The system should use a GenAI model to generate concise, informative summaries (100-300 words) for each scraped article.                                                          |
| FR003          | Identifying Topics in Articles                  | As a user, I want the system to identify main topics of each article so I can categorize and search them effectively.| The system should extract and list key topics or keywords (3-10 per article) relevant to each article using a GenAI tool. Topics should come from predefined categories for consistency.                                                           |
| FR004          | Storing Data in a Vector Database               | As a developer, I want to store articles, summaries, and topics in a vector DB to enable fast semantic search.      | The system should generate embeddings (using models like text-embedding-ada-002) and save them along with metadata (URL, headline, summary, topics) in a vector database (e.g., FAISS, Pinecone, Qdrant).    |
| FR005          | Semantic Search Feature                         | As a user, I want to search articles semantically using natural language queries so I can find relevant content.     | The system should convert user queries to embeddings, perform similarity searches in the vector DB, and return the top relevant articles ranked by similarity.                         |
| FR006          | Handling Synonyms and Context in Search         | As a user, I want the search to understand synonyms and context so that I get meaningful results.                   | The semantic search should match conceptually similar queries and article content (e.g., "AI" and "Artificial Intelligence" are treated as the same topic).       |
| FR007          | Robust Error Handling for Scraping              | As a developer, I want the system to handle scraping errors gracefully so that the pipeline is stable.              | The system should implement retries, log errors, and skip invalid or unreachable URLs while continuing processing. When partial data is available, it should still store articles with appropriate fallback values for missing fields.                                                 |
| FR008          | API Key Management and Security                 | As a developer, I want to manage API keys securely so the system is safe from unauthorized use.                     | The system should load API keys from environment variables and ensure they are not hardcoded or exposed in logs.                                                   |
| FR009          | Local Development Setup                         | As a developer, I want clear setup instructions so I can run the app locally.                                       | The repository should include a `README.MD` detailing environment setup, dependencies, and how to run the full pipeline end-to-end.                               |
| FR010          | Testing                                         | As a developer, I want to test the system's modules to ensure each component works as expected.                     | The project should include unit tests and integration tests covering scraping, summarization, topic extraction, and semantic search modules.                      |
| FR011          | Python 3.12 Compatibility                       | As a developer, I want the application to use Python 3.12 so it leverages the latest language features.            | All code should be compatible with Python 3.12 and take advantage of its features where appropriate.                                                              |
| FR012          | Poetry Dependency Management                    | As a developer, I want to use Poetry for dependency management so the project has reproducible builds.              | The project should use Poetry for dependency management, with a complete `pyproject.toml` file defining all dependencies and development requirements.            |
| FR013          | Offline Mode Support                           | As a user, I want to use the system without internet connectivity after initial setup so that I can work offline.    | The system should provide an offline mode that uses local models and cached data when internet connectivity is unavailable.                                      |
| FR014          | Text-based Matching                           | As a user, I want to complement semantic search with text-based matching so I can find exact phrases when needed.     | The system should allow hybrid searches combining semantic similarity and exact text matching for more precise results when required.                           |
| FR015          | Docker Containerization                        | As a developer, I want the application containerized so it can be easily deployed in various environments.           | The system should include a Dockerfile and docker-compose configuration for easy setup and deployment with all dependencies managed within the container.       |
| FR016          | Interactive Web UI                             | As a user, I want a web interface to interact with the system so I can use it without command line knowledge.        | The system should provide a Streamlit-based web interface for uploading URLs, viewing article summaries, and performing semantic searches with visualizations. |
| FR017          | Data Quality & Fallbacks                       | As a user, I want a consistent experience even when scraping encounters partial failures.                         | The system should provide appropriate fallbacks and default values for missing data (topics, summary, content), ensuring a consistent experience in the UI. |
| FR018          | Topic Normalization                           | As a user, I want consistent topic categorization across articles for better organization and searching.             | The system should map extracted topics to a predefined set of categories, ensuring consistency in topic naming and hierarchy. |

## Technical Requirements

| Requirement ID | Description                                       | Details                                                                                                                |
|----------------|---------------------------------------------------|------------------------------------------------------------------------------------------------------------------------|
| TR001          | Programming Language                              | Python 3.12+                                                                                                           |
| TR002          | Package Management                                | Poetry for dependency management and virtual environments                                                              |
| TR003          | Web Scraping Libraries                            | newspaper3k and/or BeautifulSoup for article extraction                                                                |
| TR004          | GenAI Integration                                 | OpenAI GPT models or equivalent for summarization and topic extraction                                                 |
| TR005          | Vector Database                                   | FAISS, Pinecone, or Qdrant for storing and searching embeddings                                                        |
| TR006          | Embedding Models                                  | OpenAI embedding models (e.g., text-embedding-ada-002) or equivalent                                                   |
| TR007          | Testing Framework                                 | pytest for unit and integration testing                                                                                |
| TR008          | Configuration Management                          | Environment variables via python-dotenv for sensitive data                                                             |
| TR009          | Code Quality                                      | Adherence to PEP8, use of type hints, comprehensive docstrings                                                         |
| TR010          | Error Handling                                    | Robust exception handling, graceful degradation, and meaningful error messages                                          |
| TR011          | Docker Containerization                           | Docker container configuration for easy deployment and environment isolation                                           |
| TR012          | UI Framework                                      | Streamlit for creating an interactive web interface                                                                    |

## Project Structure

```
src/
  ├── scraper.py      # Article scraping logic
  ├── summarizer.py   # GenAI summarization implementation
  ├── topics.py       # Topic extraction functionality
  ├── embedder.py     # Embedding generation code
  ├── vector_store.py # Vector DB interaction logic
  ├── search.py       # Semantic search implementation
  ├── config.py       # Configuration and settings management
  ├── utils.py        # Utility functions and helpers
  ├── ui/             # Streamlit UI components
  │   ├── app.py      # Main Streamlit application
  │   ├── pages/      # UI pages and components
  │   └── assets/     # UI static assets
  └── main.py         # Pipeline orchestration
tests/                # Unit and integration tests
Dockerfile            # Container configuration
docker-compose.yml    # Multi-container setup
```

## Evaluation Criteria

- Effectiveness of GenAI integration for summarization and topic identification
- Quality and clarity of code and documentation
- Performance of the semantic search functionality
- Robustness of error handling and edge case management
- User experience and intuitiveness of the web interface
- Reliability of offline functionality
- Quality of containerization and ease of deployment