Scrapling: Web Scraping Framework with Anti-Bot Bypass

Scrapling is a web scraping framework that provides HTTP fetching, dynamic JavaScript rendering, stealth browser automation for anti-bot bypass, and a spider framework for multi-page crawling. It offers a CLI for extraction and a Python API for programmatic control.

When to Use Scrapling

Use Scrapling when you need to:

• Scrape static HTML pages faster than browser-based tools.
• Scrape JavaScript-rendered pages that require a real browser.
• Bypass Cloudflare Turnstile or other bot detection systems.
• Crawl multiple pages with a spider that follows links.
• Extract data that the built-in web_extract tool does not return.

Installation

Full Installation (All Features)

pip install "scrapling[all]"
scrapling install

The scrapling install command downloads browser binaries required by the dynamic and stealth fetchers.

Minimal Installation (HTTP Only, No Browser)

pip install scrapling

Installation with Browser Automation Only

pip install "scrapling[fetchers]"
scrapling install

Prerequisites

• Python 3.10 or higher.
• pip installed.
• For browser-based fetchers: run scrapling install after pip install.

CLI Usage

Extract Static Page

scrapling extract get 'https://example.com' output.md

With optional CSS selector and browser impersonation:

scrapling extract get 'https://example.com' output.md \
  --css-selector '.content' \
  --impersonate 'chrome'

Extract JS-Rendered Page

scrapling extract fetch 'https://example.com' output.md \
  --css-selector '.dynamic-content' \
  --disable-resources \
  --network-idle

Extract Cloudflare-Protected Page

scrapling extract stealthy-fetch 'https://protected-site.com' output.html \
  --solve-cloudflare \
  --block-webrtc \
  --hide-canvas

POST Request

scrapling extract post 'https://example.com/api' output.json \
  --json '{"query": "search term"}'

The output file extension determines the format: .html (raw HTML), .md (Markdown), .txt (plain text), .json or .jsonl (JSON/JSONL). The extension is required.

Python API

HTTP Fetching

Single Request

from scrapling.fetchers import Fetcher

page = Fetcher.get('https://quotes.toscrape.com/')
quotes = page.css('.quote .text::text').getall()
for q in quotes:
    print(q)

Session with Persistent Cookies

from scrapling.fetchers import FetcherSession

with FetcherSession(impersonate='chrome') as session:
    page = session.get('https://example.com/', stealthy_headers=True)
    links = page.css('a::attr(href)').getall()
    for link in links[:5]:
        sub = session.get(link)
        print(sub.css('h1::text').get())

POST, PUT, DELETE

page = Fetcher.post('https://api.example.com/data', json={"key": "value"})
page = Fetcher.put('https://api.example.com/item/1', data={"name": "updated"})
page = Fetcher.delete('https://api.example.com/item/1')

Proxy Support

page = Fetcher.get('https://example.com', proxy='http://user:pass@proxy:8080')

Dynamic Fetching (JS-Rendered Pages)

Basic Dynamic Fetch

from scrapling.fetchers import DynamicFetcher

page = DynamicFetcher.fetch('https://example.com', headless=True)
data = page.css('.js-loaded-content::text').getall()

Wait for Element

page = DynamicFetcher.fetch(
    'https://example.com',
    wait_selector=('.results', 'visible'),
    network_idle=True,
)

Dynamic Session with Disabled Resources

from scrapling.fetchers import DynamicSession

with DynamicSession(headless=True, disable_resources=True, network_idle=True) as session:
    page = session.fetch('https://example.com')
    items = page.css('.item::text').getall()

Custom Page Automation (Playwright)

from playwright.sync_api import Page
from scrapling.fetchers import DynamicFetcher

def scroll_and_click(page: Page):
    page.mouse.wheel(0, 3000)
    page.wait_for_timeout(1000)
    page.click('button.load-more')
    page.wait_for_selector('.extra-results')

page = DynamicFetcher.fetch('https://example.com', page_action=scroll_and_click)
results = page.css('.extra-results .item::text').getall()

Stealth Fetching (Anti-Bot Bypass)

Single Stealth Fetch

from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.fetch(
    'https://protected-site.com',
    headless=True,
    solve_cloudflare=True,
    block_webrtc=True,
    hide_canvas=True,
)
content = page.css('.protected-content::text').getall()

Stealth Session

from scrapling.fetchers import StealthySession

with StealthySession(headless=True, solve_cloudflare=True) as session:
    page1 = session.fetch('https://protected-site.com/page1')
    page2 = session.fetch('https://protected-site.com/page2')

Element Selection and Navigation

Scrapling provides multiple ways to select elements from a page.

CSS Selectors

page.css('h1::text').get()              # First h1 text
page.css('a::attr(href)').getall()      # All link hrefs
page.css('.quote .text::text').getall() # Nested selection

XPath

page.xpath('//div[@class="content"]/text()').getall()
page.xpath('//a/@href').getall()

Find Methods

page.find_all('div', class_='quote')       # By tag + attribute
page.find_by_text('Read more', tag='a')    # By text content
page.find_by_regex(r'\$\d+\.\d{2}')       # By regex pattern

Find Similar Elements

first_product = page.css('.product')[0]
all_similar = first_product.find_similar()

Navigation

el = page.css('.target')[0]
el.parent                # Parent element
el.children              # Child elements
el.next_sibling          # Next sibling
el.prev_sibling          # Previous sibling

Spider Framework

Basic Spider

from scrapling.spiders import Spider, Request, Response

class QuotesSpider(Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]
    concurrent_requests = 10
    download_delay = 1

    async def parse(self, response: Response):
        for quote in response.css('.quote'):
            yield {
                "text": quote.css('.text::text').get(),
                "author": quote.css('.author::text').get(),
                "tags": quote.css('.tag::text').getall(),
            }

        next_page = response.css('.next a::attr(href)').get()
        if next_page:
            yield response.follow(next_page)

result = QuotesSpider().start()
print(f"Scraped {len(result.items)} quotes")
result.items.to_json("quotes.json")

Multi-Session Spider

Route different requests to different fetcher types.

from scrapling.fetchers import FetcherSession, AsyncStealthySession

class SmartSpider(Spider):
    name = "smart"
    start_urls = ["https://example.com/"]

    def configure_sessions(self, manager):
        manager.add("fast", FetcherSession(impersonate="chrome"))
        manager.add("stealth", AsyncStealthySession(headless=True), lazy=True)

    async def parse(self, response: Response):
        for link in response.css('a::attr(href)').getall():
            if "protected" in link:
                yield Request(link, sid="stealth")
            else:
                yield Request(link, sid="fast", callback=self.parse)

Pause and Resume Crawling

spider = QuotesSpider(crawldir="./crawl_checkpoint")
spider.start()  # Ctrl+C to pause, re-run to resume from checkpoint

Parameters Reference

Parameter	Meaning	Required
impersonate	Browser to impersonate (e.g., 'chrome')	No
stealthy_headers	Add stealth headers to request	No
proxy	Proxy URL (e.g., 'http://user:pass@proxy:8080')	No
headless	Run browser in headless mode	No
wait_selector	Tuple (selector, state) to wait for element	No
network_idle	Wait for network to become idle	No
disable_resources	Block fonts, images, media, stylesheets	No
page_action	Custom function on Playwright Page object	No
solve_cloudflare	Attempt to bypass Cloudflare protection	No
block_webrtc	Block WebRTC to prevent IP leak	No
hide_canvas	Hide canvas fingerprinting	No
concurrent_requests	Number of concurrent requests for spider	No
download_delay	Delay between requests in seconds	No
crawldir	Directory for checkpoint data	No
sid	Session ID for routing request in multi-session spider	No
callback	Callback function for request in spider	No
lazy	Initialize session lazily	No
json	JSON body for POST request (CLI)	No
css-selector	CSS selector to filter content (CLI)	No

Constraints and Caveats

• Scrapling is for educational and research purposes only. Users must comply with local and international data scraping laws and respect website Terms of Service. Always check robots.txt and website ToS before scraping.
• Browser install is required: run scrapling install after pip install. Without it, DynamicFetcher and StealthyFetcher will fail.
• DynamicFetcher and StealthyFetcher timeout is in milliseconds (default 30000). Fetcher timeout is in seconds.
• Setting solve_cloudflare=True adds 5 to 15 seconds to fetch time. Only enable when needed.
• StealthyFetcher runs a real browser. Limit concurrent usage to avoid high resource consumption.

Failure Modes

• DynamicFetcher and StealthyFetcher fail if browser is not installed (missing scrappling install).
• Timeouts occur if the timeout value is too low. Note the unit difference: milliseconds for browser fetchers, seconds for Fetcher.
• Cloudflare bypass may fail or be slow (5 to 15 seconds added).
• High resource usage if too many StealthyFetcher instances run concurrently.
• Legal issues if scraping violates Terms of Service or laws.