Async BeautifulSoup HTML Table Extractor – Str Base
Clear professional description of the code solution
Task: Write a complete, production‑ready Python script that asynchronously downloads a web page, extracts every HTML `` element using BeautifulSoup, and returns the data as pandas DataFrames. The script must also provide optional functionality to save each table to CSV and/or JSON files, support custom CSS selectors for targeting specific tables, and include robust error handling, logging, and type hints.
Exact Requirements:
-
Environment & Dependencies
- Use Python 3.9+ syntax.
- Required packages:
aiohttp,asyncio,beautifulsoup4,pandas,lxml,argparse,logging. - Provide a
requirements.txtsnippet at the top of the script (as a comment) listing exact package versions (use>=for minimum versions).
-
Core Functionality
- Implement an asynchronous function
fetch_html(url: str, timeout: int = 30) -> strthat retrieves the raw HTML usingaiohttp. Include retry logic (up to 3 attempts) with exponential backoff and proper session cleanup. - Implement
parse_tables(html: str, selector: str | None = None) -> list[pd.DataFrame]that:- Parses the HTML with BeautifulSoup (using the
lxmlparser). - If
selectoris provided, only tables matching the CSS selector are processed; otherwise, all `` tags are extracted. - Correctly handles
,,,and respectsrowspan/colspanby expanding cells into the appropriate positions. - Returns a list of pandas DataFrames, preserving column headers when present.
- Parses the HTML with BeautifulSoup (using the
- Implement an asynchronous function
-
Output Options
- Provide a function
save_tables(tables: list[pd.DataFrame], base_path: str, to_csv: bool = True, to_json: bool = False) -> Nonethat writes each DataFrame tobase_path_table_.csvand/orbase_path_table_.json. - Ensure files are created in a thread‑safe manner and that existing files are not overwritten without confirmation (use a
--forceCLI flag to allow overwriting).
- Provide a function
-
Command‑Line Interface
- Use
argparseto create a CLI with the following arguments:url(positional): the target web page URL.--selector(optional): CSS selector to limit table extraction.--output-dir(optional, default: current directory): directory where files will be saved.--csv/--no-csv(default: enabled): toggle CSV export.--json/--no-json(default: disabled): toggle JSON export.--timeout(optional, default: 30): request timeout in seconds.--force(flag): overwrite existing output files.--log-level(optional, choices: DEBUG, INFO, WARNING, ERROR, CRITICAL; default: INFO).
- The CLI should invoke the async workflow, print a concise summary of extracted tables (e.g., “Extracted 3 tables: 12×5, 8×3, 20×7 rows/columns”), and log detailed progress according to the chosen log level.
- Use
-
Logging & Error Handling
- Configure a module‑level logger named
async_table_extractor. - Log start/end of each major step, retries, and any parsing anomalies (e.g., mismatched colspan).
- Raise a custom exception
TableExtractionErrorwith a clear message for fatal failures (network errors after retries, invalid HTML, etc.).
- Configure a module‑level logger named
-
Code Quality
- Include comprehensive docstrings for all public functions and the CLI entry point.
- Use type hints throughout.
- Follow PEP 8 style; add
# noqa: E501where necessary to keep line lengths reasonable. - Add a
if __name__ == "__main__":guard to launch the CLI.
Expected Output Format:
The assistant must return only the complete Python script, enclosed in a single fenced code block with language identifier python. The script should start with the requirements.txt comment, then all imports, followed by the implementation meeting the requirements above, and finally the CLI guard. No additional explanations, commentary, or extraneous text should be included outside the code block.
Related Prompts
More prompts in Coding & Development
This Prompt Ads Sequential Function Calling To Models Other Than GPT 0613
This prompt ads sequential function calling to models other than GPT-0613
Create a personalized workout routine
Tailor a workout routine specifically designed for individual fitness goals
GODMODE CHEATCODE
God Writes You a Letter Today. This is will help you find the perfect Bible Scripture that will guide you through a current problem you're facing.
Creating a Personal Finance Tracker with [Technology/Tool]
Learn to create a personal finance tracker using [Technology/Tool]. Get code samples and budgeting tips.
Build an entire application using bubble.io with ChatGPT4
Build an entire app with bubble.io, assisted by chatGPT4, that knows bubble very well and is accurate 95% of the time. This prompt will help you maximize the quality of chatGPT assistance. Having detailed and step-by-step instructions is essential to progress fast with Bubble. This initial prompt will help you get started on a good basis. Follow it because I will make it even better.
Become LawyerGPT
Are you in a legal bind? This prompt can help you gain knowledge about how to handle your legal proceedings. DISCLAIMER: Please meet with a real lawyer to discuss your options.