Smart Web Fetch

Name: Smart Web Fetch
Author: Leochens

智能网页抓取技能 - 替代内置 web_fetch，自动使用 Jina Reader / markdown.new / defuddle.md 清洗服务获取干净 Markdown。支持多级降级策略，大幅降低 Token 消耗。当 Agent 需要获取网页内容时使用本技能替代 web_fetch。

Leochens

@leochens

What This Skill Does

Fetches web page content and automatically cleans it into clean Markdown using a multi-tier fallback strategy (Jina Reader, markdown.new, defuddle.md). Reduces token consumption by 50-80% compared to raw HTML.

Replaces built-in web_fetch by returning pre-cleaned Markdown instead of raw HTML, cutting token usage by half or more.

When to Use It

Retrieve article content from a news or blog URL for summarization
Extract clean text from a documentation page to feed into an AI agent
Get a web page's main content without ads, navigation, or clutter
Fetch multiple URLs in sequence while minimizing token overhead
Replace web_fetch in agent workflows to reduce costs and improve output quality

Install

$ openclaw skills install @leochens/smart-web-fetch

Smart Web Fetch

智能网页内容获取技能，完全替代 web_fetch，自动通过清洗服务获取干净 Markdown。

核心功能

完全替代 web_fetch: 获取的已经是清洗后的 Markdown，而非原始 HTML
四级降级策略: Jina → markdown.new → defuddle.md → 原始内容
Token 优化: 清洗后的内容比原始 HTML 节省 50-80% Token

使用方式

命令行获取网页内容

# 获取清洗后的 Markdown（文本输出）
python3 {baseDir}/scripts/fetch.py "https://example.com/article"

# 获取 JSON 格式（包含元信息）
python3 {baseDir}/scripts/fetch.py "https://example.com/article" --json

在 Agent 中使用

当用户需要获取网页内容时：

用户: "帮我查一下 https://example.com/article 的内容"

Agent 应该:
1. 运行: python3 ~/.openclaw/skills/smart-web-fetch/scripts/fetch.py "https://example.com/article"
2. 直接获得清洗后的 Markdown 内容

JSON 输出格式

{
  "success": true,
  "url": "https://r.jina.ai/http://example.com/article",
  "content": "# Article Title\n\nClean markdown content here...",
  "source": "jina",
  "error": null
}

降级策略

Jina Reader (首选)
- URL: https://r.jina.ai/http://{target}
- 免费，无需 API Key，中文支持好
markdown.new (降级)
- URL: https://markdown.new/{target}
defuddle.md (降级)
- URL: https://defuddle.md/{target}
原始内容 (最终兜底)
- 直接获取原始 HTML

Agent 配置建议

为了强制使用本技能替代 web_fetch，在 openclaw.json 中配置：

{
  "agents": {
    "list": [
      {
        "id": "your-agent",
        "tools": {
          "deny": ["web_fetch"]
        }
      }
    ]
  }
}

这样 Agent 就无法调用内置 web_fetch，只能通过本技能获取网页内容。

优势

🚀 Token 节省 50-80%: 去除广告、导航栏等噪音
🔄 自动容错: 四级服务降级，确保可用性
🆓 零成本: 全部使用免费服务
🔌 即插即用: 不需要 API Key
📝 干净输出: 纯 Markdown，无需额外解析

Top skills in this category

API Gateway

@byungkyu

Connect to external services through Maton-managed API routes. Use this skill only after the user names the target app, account, and task. Start with read/list calls when possible and follow the app-specific reference before any change.

39385k

PollyReach

@pollyreach

PollyReach gives every AI agent a phone number and the ability to get things done over the phone — finding contacts, making calls, and completing tasks. Just...

54101k

Proactive Agent Lite

@bestrocky

Transform AI agents from task-followers into proactive partners with memory architecture, reverse prompting, and self-healing patterns. Lightweight version f...

6240k

Using Superpowers

@zlc000190

Use when starting any conversation - establishes how to find and use skills, requiring Skill tool invocation before ANY response including clarifying questions

6035k

Productivity

@ivangdavila

Diagnoses and repairs personal productivity: overwhelm, procrastination, scattered priorities, collapsed habits, busywork that never finishes. Use when someone is overwhelmed or behind; when they cannot start, cannot stop, or keep replanning instead of working; when everything feels urgent and the list is no longer trusted; when goals never turn into shipped work; when meetings and messages eat the day; when a habit keeps breaking; when they want a weekly review, a shutdown routine, or one trusted home for goals, projects and commitments; and when the constraint is a situation — student, manager, executive, parent, freelancer, founder, creative, remote work, ADHD, burnout, or guilt about resting. Covers capacity math, estimation, WIP limits, delegation and saying no. Not for calendar-API automation (`calendar-planner`), running the day-to-day list (`task-list`), or the narrower tools: time blocking (`time-management`), habit streaks (`habits`), deep-work rituals (`deep-work`).

6925k