Async BeautifulSoup HTML Table Extractor – Url Data

Task: Create a complete, production‑ready Python script that asynchronously downloads one or more web pages, extracts all HTML `` elements using BeautifulSoup, and returns the data in a clean, structured format. The script must be usable as a command‑line tool and as an importable module.

Requirements:

1. Environment & Dependencies

   • Use Python 3.9+ syntax.
   • Required libraries: aiohttp, asyncio, beautifulsoup4, pandas, lxml, tqdm, and loguru.
   • Provide a requirements.txt snippet at the end of the output.

2. Asynchronous Fetching

   • Implement an async def fetch(session: aiohttp.ClientSession, url: str) -> str that returns the raw HTML.
   • Include timeout handling (default 15 seconds) and retry logic (up to 3 attempts with exponential back‑off).
   • Log each request, response status, and any errors using loguru.

3. HTML Table Parsing

   • Write a function def parse_tables(html: str, url: str) -> List[pd.DataFrame] that: a. Parses the HTML with BeautifulSoup (using the "lxml" parser). b. Finds every element, regardless of nesting level. c. For each table, extracts header rows () and data rows (``), handling:
     • rowspan and colspan (expand cells appropriately).
     • Missing cells (fill with NaN).
     • Mixed data types (convert numeric strings to int/float where possible). d. Returns a list of pandas DataFrames, one per table, each with a column "_source_url" containing the originating URL.

4. Pagination / Multiple URLs

   • Accept either a single URL or a path to a text file containing one URL per line.
   • If a file is provided, fetch and parse all URLs concurrently (limit concurrency to 5 simultaneous requests).

5. CLI Interface

   • Use argparse to expose the following arguments:
     • url (positional): a single URL or path to a file with URLs.
     • --output-dir (optional, default "./output"): directory where CSV files will be saved.
     • --format (optional, choices=["csv","xlsx","json"], default="csv"): output file format.
     • --max-concurrency (optional, default=5): maximum number of concurrent fetches.
     • --log-level (optional, choices=["DEBUG","INFO","WARNING","ERROR"], default="INFO").
   • For each extracted table, save a file named {domain}_{table_index}.{ext} (e.g., example.com_1.csv).

6. Error Handling & Logging

   • All exceptions must be caught and logged without crashing the entire run.
   • If a URL fails after retries, record it in a failed_urls.txt file inside the output directory.

7. Documentation & Type Hints

   • Include module‑level docstring explaining purpose, usage, and limitations.
   • Add docstrings to every function with parameter/return type descriptions.
   • Use typing module for all type hints.

8. Testing Stub

   • Provide a minimal if __name__ == "__main__": block that parses arguments and runs the asynchronous main routine.

9. Output Format

   • Return the entire script enclosed in a single Markdown code fence with language identifier python.
   • After the code block, include a separate fenced block titled requirements.txt listing the exact pip packages and versions (use >= for major versions, e.g., aiohttp>=3.9).
   • Finally, provide a short “How to run” section (plain text) with example commands for both a single URL and a URL list file.

Expected Output:

```python
# (Full, ready‑to‑run Python script meeting all requirements)
```

```txt
# requirements.txt
aiohttp>=3.9
beautifulsoup4>=4.12
pandas>=2.0
lxml>=5.0
tqdm>=4.66
loguru>=0.7
```

How to run:
# Example for a single URL
python async_table_extractor.py https://example.com/page.html --output-dir ./data --format csv

# Example for a list of URLs
python async_table_extractor.py urls.txt --max-concurrency 3 --log-level DEBUG