Async BeautifulSoup HTML Table Extractor – Data Output

Task: Create a complete, production‑ready Python script that asynchronously downloads one or more web pages, extracts all HTML `` elements using BeautifulSoup, cleans and normalizes the data, and saves the results to CSV and JSON files. The script must be configurable via command‑line arguments, include robust error handling, logging, type hints, and be ready for immediate use in a real‑world data‑scraping project.

Exact Requirements:

1. Environment & Dependencies

   • Use Python 3.10+ syntax.
   • Required packages: aiohttp, asyncio, beautifulsoup4, pandas, lxml, typer, loguru.
   • Provide a requirements.txt snippet at the end of the output.

2. CLI Interface (Typer)

   • --url (repeatable) – one or more page URLs to process.
   • --output-dir – directory where results will be written (default: ./output).
   • --selector – optional CSS selector to limit tables (e.g., .data-table).
   • --concurrency – maximum number of concurrent HTTP requests (default: 5).
   • --timeout – request timeout (default: 30).
   • --include-index – flag to include the original row index in the output files.

3. Asynchronous Fetching

   • Implement an async fetch(session, url) that respects the timeout and returns the raw HTML or raises a descriptive exception.
   • Use an aiohttp.ClientSession with a connection pool limited by the --concurrency value.
   • Retry failed requests up to 2 times with exponential back‑off (e.g., 1 s, 2 s).

4. HTML Table Extraction

   • Parse each page with BeautifulSoup (lxml parser).
   • Locate tables using the provided --selector or fall back to all `` tags.
   • For each table:
     • Extract header rows (or first with ``).
     • Extract data rows, handling rowspan/colspan correctly.
     • Strip whitespace, convert numeric strings to int/float where possible, and keep dates as ISO‑8601 strings if they match common date patterns.
     • Preserve the source URL and an auto‑generated table identifier (table_1, table_2, …) as additional columns.

5. Data Consolidation

   • Combine all extracted tables into a single pandas.DataFrame.
   • If tables have differing column sets, align them by column name, filling missing values with NaN.
   • Optionally (if --include-index is set) add a column original_row_index reflecting the row’s position in its source table.

6. Output Generation

   • Write the consolidated DataFrame to two files in --output-dir:
     • tables_combined.csv (UTF‑8, comma‑separated).
     • tables_combined.json (records orientation, UTF‑8).
   • Additionally, create a sub‑folder individual/ containing one CSV file per extracted table named _table_.csv.

7. Logging & Error Reporting

   • Use loguru for structured logs.
   • Log start/end of each fetch, success/failure status, number of tables extracted per page, and summary statistics (total tables, rows, columns).
   • All uncaught exceptions should be logged with stack trace and cause the script to exit with a non‑zero status code.

8. Code Quality

   • Include module‑level docstring explaining purpose, usage, and dependencies.
   • Provide function‑level docstrings with type hints.
   • Follow PEP 8 style; use black‑compatible formatting.
   • Add a if __name__ == "__main__": block that invokes the Typer app.

9. README & Usage Example

   • At the end of the output, include a concise README.md section that explains:
     • Installation (pip install -r requirements.txt).
     • Example command line usage with multiple URLs and custom selector.
     • Expected directory structure after execution.

Expected Output Format:

• A single markdown code block containing the full Python script (scrape_tables_async.py).
• Immediately following, another markdown code block with requirements.txt content.
• Finally, a third markdown code block with the README.md excerpt.

All code blocks must be properly fenced with triple backticks and language identifiers (python, text). No additional explanation outside these blocks is needed.