Async BeautifulSoup HTML Table Extractor – Def Str

Task: Create a production‑ready Python script that asynchronously downloads one or more web pages, extracts every HTML `` element using BeautifulSoup, and converts each table into a clean pandas.DataFrame. The script must correctly handle merged cells (rowspan and colspan), preserve column headers, and allow the user to export the results as CSV, JSON, or Excel files. The solution should be fully typed, include comprehensive inline documentation, robust error handling, and a command‑line interface (CLI) for easy execution.

Exact Requirements (follow in order):

1. Environment & Dependencies

   • Use Python 3.10+ syntax.
   • Required packages: aiohttp, beautifulsoup4, pandas, openpyxl (for Excel), tqdm, typer.
   • Provide a requirements.txt snippet at the end of the script.

2. Asynchronous HTTP Fetching

   • Implement an async def fetch(session: aiohttp.ClientSession, url: str) -> str that returns the raw HTML.
   • Include a configurable timeout (default 15 seconds) and retry logic (up to 3 attempts with exponential back‑off).
   • Use aiohttp.TCPConnector(limit_per_host=CONCURRENCY_LIMIT) where CONCURRENCY_LIMIT is a CLI option (default 5).

3. HTML Parsing & Table Extraction

   • Write def parse_tables(html: str) -> List[pd.DataFrame] that: a. Parses the HTML with BeautifulSoup(html, "html.parser"). b. Locates all `` tags (including those nested). c. For each table, builds a 2‑D list respecting rowspan and colspan. d. Converts the 2‑D list into a pandas.DataFrame, using the first row as column headers when appropriate.
   • Clean cell contents: strip whitespace, replace multiple spaces with a single space, and optionally convert numeric strings to proper numeric dtypes.

4. Data Export Functions

   • Implement three helper functions:

     def save_csv(dfs: List[pd.DataFrame], prefix: str) -> None
     def save_json(dfs: List[pd.DataFrame], prefix: str) -> None
     def save_excel(dfs: List[pd.DataFrame], prefix: str) -> None
     

   • Files should be named {prefix}_table_{index}.{ext} where index starts at 1.
   • Ensure UTF‑8 encoding and handle any filesystem errors gracefully.

5. CLI with Typer

   • Provide a main entry point using typer that accepts:
     • urls: List[str] – one or more target URLs (positional arguments).
     • --output-dir / -o – directory for saved files (default: current working directory).
     • --format / -f – output format: csv, json, excel (can be given multiple times; default is csv).
     • --concurrency / -c – integer concurrency limit (default 5).
     • --timeout / -t – request timeout in seconds (default 15).
   • The CLI must display a progress bar (via tqdm) for the total number of URLs processed.

6. Error Handling & Logging

   • Use the standard logging module; default log level is INFO.
   • Log successes, retries, and any parsing failures with clear messages.
   • If a URL fails after all retries, continue processing remaining URLs and report a summary at the end.

7. Documentation & Usage Example

   • At the top of the script, include a comprehensive docstring that explains: purpose, installation (pip install -r requirements.txt), usage examples, and limitations (e.g., JavaScript‑rendered tables are not supported).
   • Provide a short README section at the bottom of the script (as a multi‑line comment) showing typical command‑line invocations.

8. Testing Stub (Optional but Preferred)

   • Add a minimal if __name__ == "__main__": block that calls typer.run(main).
   • Include a placeholder for unit tests (e.g., a comment indicating where pytest tests could be added).

Expected Output Format: The assistant must return a single, copy‑paste‑ready Python file that fulfills all the requirements above. The file should be organized as follows:

"""[Docstring with overview, installation, usage, limitations]"""

# ==================== IMPORTS ====================
import asyncio
import logging
from pathlib import Path
from typing import List

import aiohttp
import pandas as pd
from bs4 import BeautifulSoup
from tqdm.asyncio import tqdm
import typer

# ==================== CONFIGURATION ====================
# (set up logger, default constants, etc.)

# ==================== ASYNC FETCH FUNCTION ====================
async def fetch(...):
    ...

# ==================== TABLE PARSING FUNCTION ====================
def parse_tables(...):
    ...

# ==================== EXPORT HELPERS ====================
def save_csv(...):
    ...

def save_json(...):
    ...

def save_excel(...):
    ...

# ==================== MAIN ASYNC WORKFLOW ====================
async def process_url(...):
    ...

async def main_async(urls: List[str], ...):
    ...

# ==================== CLI ENTRYPOINT ====================
def main(...):
    ...

if __name__ == "__main__":
    typer.run(main)

# ==================== REQUIREMENTS ====================
# aiohttp
# beautifulsoup4
# pandas
# openpyxl
# tqdm
# typer

# ==================== README SECTION (as comment) ====================
"""
Example usage:
    python async_table_extractor.py https://example.com/page1 https://example.org/page2 -o ./tables -f csv json -c 8

...
"""

The assistant must only output the complete prompt text (i.e., the instructions above) without any additional commentary. Do not include any explanatory notes outside the prompt.