Async BeautifulSoup HTML Table Extractor – Str Base

Clear professional description of the code solution

C
Christian Geimer
·Oct 4, 2026·
0 0 2
Free
Prompt
545 words

Task: Write a complete, production‑ready Python script that asynchronously downloads a web page, extracts every HTML `` element using BeautifulSoup, and returns the data as pandas DataFrames. The script must also provide optional functionality to save each table to CSV and/or JSON files, support custom CSS selectors for targeting specific tables, and include robust error handling, logging, and type hints.

Exact Requirements:

  1. Environment & Dependencies

    • Use Python 3.9+ syntax.
    • Required packages: aiohttp, asyncio, beautifulsoup4, pandas, lxml, argparse, logging.
    • Provide a requirements.txt snippet at the top of the script (as a comment) listing exact package versions (use >= for minimum versions).
  2. Core Functionality

    • Implement an asynchronous function fetch_html(url: str, timeout: int = 30) -> str that retrieves the raw HTML using aiohttp. Include retry logic (up to 3 attempts) with exponential backoff and proper session cleanup.
    • Implement parse_tables(html: str, selector: str | None = None) -> list[pd.DataFrame] that:
      • Parses the HTML with BeautifulSoup (using the lxml parser).
      • If selector is provided, only tables matching the CSS selector are processed; otherwise, all `` tags are extracted.
      • Correctly handles , , , and respects rowspan/colspan by expanding cells into the appropriate positions.
      • Returns a list of pandas DataFrames, preserving column headers when present.
  3. Output Options

    • Provide a function save_tables(tables: list[pd.DataFrame], base_path: str, to_csv: bool = True, to_json: bool = False) -> None that writes each DataFrame to base_path_table_.csv and/or base_path_table_.json.
    • Ensure files are created in a thread‑safe manner and that existing files are not overwritten without confirmation (use a --force CLI flag to allow overwriting).
  4. Command‑Line Interface

    • Use argparse to create a CLI with the following arguments:
      • url (positional): the target web page URL.
      • --selector (optional): CSS selector to limit table extraction.
      • --output-dir (optional, default: current directory): directory where files will be saved.
      • --csv / --no-csv (default: enabled): toggle CSV export.
      • --json / --no-json (default: disabled): toggle JSON export.
      • --timeout (optional, default: 30): request timeout in seconds.
      • --force (flag): overwrite existing output files.
      • --log-level (optional, choices: DEBUG, INFO, WARNING, ERROR, CRITICAL; default: INFO).
    • The CLI should invoke the async workflow, print a concise summary of extracted tables (e.g., “Extracted 3 tables: 12×5, 8×3, 20×7 rows/columns”), and log detailed progress according to the chosen log level.
  5. Logging & Error Handling

    • Configure a module‑level logger named async_table_extractor.
    • Log start/end of each major step, retries, and any parsing anomalies (e.g., mismatched colspan).
    • Raise a custom exception TableExtractionError with a clear message for fatal failures (network errors after retries, invalid HTML, etc.).
  6. Code Quality

    • Include comprehensive docstrings for all public functions and the CLI entry point.
    • Use type hints throughout.
    • Follow PEP 8 style; add # noqa: E501 where necessary to keep line lengths reasonable.
    • Add a if __name__ == "__main__": guard to launch the CLI.

Expected Output Format: The assistant must return only the complete Python script, enclosed in a single fenced code block with language identifier python. The script should start with the requirements.txt comment, then all imports, followed by the implementation meeting the requirements above, and finally the CLI guard. No additional explanations, commentary, or extraneous text should be included outside the code block.

Need help?

Connect with verified experts who can help you succeed.

Related Prompts

More prompts in Coding & Development

View All
Coding & Development
Universal

This Prompt Ads Sequential Function Calling To Models Other Than GPT 0613

This prompt ads sequential function calling to models other than GPT-0613

D
digitalmuse$2.99
39,910 89,588
Coding & Development
Universal

Create a personalized workout routine

Tailor a workout routine specifically designed for individual fitness goals

P
primequery$2.99
23,370 23,405
Coding & Development
Universal

GODMODE CHEATCODE

God Writes You a Letter Today. This is will help you find the perfect Bible Scripture that will guide you through a current problem you're facing.

S
signalcraft$3.99
13,574 13,622
Coding & Development
Universal

Creating a Personal Finance Tracker with [Technology/Tool]

Learn to create a personal finance tracker using [Technology/Tool]. Get code samples and budgeting tips.

F
focusqueryFree
376 385
Coding & Development
ChatGPT

Build an entire application using bubble.io with ChatGPT4

Build an entire app with bubble.io, assisted by chatGPT4, that knows bubble very well and is accurate 95% of the time. This prompt will help you maximize the quality of chatGPT assistance. Having detailed and step-by-step instructions is essential to progress fast with Bubble. This initial prompt will help you get started on a good basis. Follow it because I will make it even better.

P
promptframes$5.99
1,280 1,300
Coding & Development
Universal

Become LawyerGPT

Are you in a legal bind? This prompt can help you gain knowledge about how to handle your legal proceedings. DISCLAIMER: Please meet with a real lawyer to discuss your options.

P
promptbench$2.99
1,063 1,076