Project Requirements Document: News Scraper & Semantic Search with GenAI

The following table outlines the detailed functional requirements of the News Scraper & Semantic Search application.

Requirement ID	Description	User Story	Expected Behavior/Outcome
FR001	Extracting News Articles	As a user, I want the system to fetch full news articles from provided URLs so that I can process and store them.	The system should scrape the article's headline and full text and return structured data for each URL provided. Implement robust error handling with retries, logging, and skip invalid URLs.
FR002	Summarizing News Articles	As a user, I want the system to summarize the articles so I can quickly understand the key points of each one.	The system should use a GenAI model to generate concise, informative summaries (100-300 words) for each scraped article.
FR003	Identifying Topics in Articles	As a user, I want the system to identify main topics of each article so I can categorize and search them effectively.	The system should extract and list key topics or keywords (3-10 per article) relevant to each article using a GenAI tool. Topics should come from predefined categories for consistency.
FR004	Storing Data in a Vector Database	As a developer, I want to store articles, summaries, and topics in a vector DB to enable fast semantic search.	The system should generate embeddings (using models like text-embedding-ada-002) and save them along with metadata (URL, headline, summary, topics) in a vector database (e.g., FAISS, Pinecone, Qdrant).
FR005	Semantic Search Feature	As a user, I want to search articles semantically using natural language queries so I can find relevant content.	The system should convert user queries to embeddings, perform similarity searches in the vector DB, and return the top relevant articles ranked by similarity.
FR006	Handling Synonyms and Context in Search	As a user, I want the search to understand synonyms and context so that I get meaningful results.	The semantic search should match conceptually similar queries and article content (e.g., "AI" and "Artificial Intelligence" are treated as the same topic).
FR007	Robust Error Handling for Scraping	As a developer, I want the system to handle scraping errors gracefully so that the pipeline is stable.	The system should implement retries, log errors, and skip invalid or unreachable URLs while continuing processing. When partial data is available, it should still store articles with appropriate fallback values for missing fields.
FR008	API Key Management and Security	As a developer, I want to manage API keys securely so the system is safe from unauthorized use.	The system should load API keys from environment variables and ensure they are not hardcoded or exposed in logs.
FR009	Local Development Setup	As a developer, I want clear setup instructions so I can run the app locally.	The repository should include a README.MD detailing environment setup, dependencies, and how to run the full pipeline end-to-end.
FR010	Testing	As a developer, I want to test the system's modules to ensure each component works as expected.	The project should include unit tests and integration tests covering scraping, summarization, topic extraction, and semantic search modules.
FR011	Python 3.12 Compatibility	As a developer, I want the application to use Python 3.12 so it leverages the latest language features.	All code should be compatible with Python 3.12 and take advantage of its features where appropriate.
FR012	Poetry Dependency Management	As a developer, I want to use Poetry for dependency management so the project has reproducible builds.	The project should use Poetry for dependency management, with a complete pyproject.toml file defining all dependencies and development requirements.
FR013	Offline Mode Support	As a user, I want to use the system without internet connectivity after initial setup so that I can work offline.	The system should provide an offline mode that uses local models and cached data when internet connectivity is unavailable.
FR014	Text-based Matching	As a user, I want to complement semantic search with text-based matching so I can find exact phrases when needed.	The system should allow hybrid searches combining semantic similarity and exact text matching for more precise results when required.
FR015	Docker Containerization	As a developer, I want the application containerized so it can be easily deployed in various environments.	The system should include a Dockerfile and docker-compose configuration for easy setup and deployment with all dependencies managed within the container.
FR016	Interactive Web UI	As a user, I want a web interface to interact with the system so I can use it without command line knowledge.	The system should provide a Streamlit-based web interface for uploading URLs, viewing article summaries, and performing semantic searches with visualizations.
FR017	Data Quality & Fallbacks	As a user, I want a consistent experience even when scraping encounters partial failures.	The system should provide appropriate fallbacks and default values for missing data (topics, summary, content), ensuring a consistent experience in the UI.
FR018	Topic Normalization	As a user, I want consistent topic categorization across articles for better organization and searching.	The system should map extracted topics to a predefined set of categories, ensuring consistency in topic naming and hierarchy.

Technical Requirements

Requirement ID	Description	Details
TR001	Programming Language	Python 3.12+
TR002	Package Management	Poetry for dependency management and virtual environments
TR003	Web Scraping Libraries	newspaper3k and/or BeautifulSoup for article extraction
TR004	GenAI Integration	OpenAI GPT models or equivalent for summarization and topic extraction
TR005	Vector Database	FAISS, Pinecone, or Qdrant for storing and searching embeddings
TR006	Embedding Models	OpenAI embedding models (e.g., text-embedding-ada-002) or equivalent
TR007	Testing Framework	pytest for unit and integration testing
TR008	Configuration Management	Environment variables via python-dotenv for sensitive data
TR009	Code Quality	Adherence to PEP8, use of type hints, comprehensive docstrings
TR010	Error Handling	Robust exception handling, graceful degradation, and meaningful error messages
TR011	Docker Containerization	Docker container configuration for easy deployment and environment isolation
TR012	UI Framework	Streamlit for creating an interactive web interface

Project Structure

src/
  ├── scraper.py      # Article scraping logic
  ├── summarizer.py   # GenAI summarization implementation
  ├── topics.py       # Topic extraction functionality
  ├── embedder.py     # Embedding generation code
  ├── vector_store.py # Vector DB interaction logic
  ├── search.py       # Semantic search implementation
  ├── config.py       # Configuration and settings management
  ├── utils.py        # Utility functions and helpers
  ├── ui/             # Streamlit UI components
  │   ├── app.py      # Main Streamlit application
  │   ├── pages/      # UI pages and components
  │   └── assets/     # UI static assets
  └── main.py         # Pipeline orchestration
tests/                # Unit and integration tests
Dockerfile            # Container configuration
docker-compose.yml    # Multi-container setup

Evaluation Criteria

• Effectiveness of GenAI integration for summarization and topic identification
• Quality and clarity of code and documentation
• Performance of the semantic search functionality
• Robustness of error handling and edge case management
• User experience and intuitiveness of the web interface
• Reliability of offline functionality
• Quality of containerization and ease of deployment