AI Digest Platform - Source Research & Configuration Guide

Comprehensive catalog of feeds, APIs, and sources for the AI Digest aggregation platform. Researched: February 2026

---

Table of Contents

1. RSS Feed Sources
2. GitHub Trending Configuration
3. ArXiv Categories & Configuration
4. Hacker News Configuration
5. HuggingFace Hub
6. Reddit Subreddits
7. Product Hunt
8. Additional Sources Not Yet Implemented
9. Source Quality Scoring Framework
10. Deduplication Strategies
11. Recommended Seed Configuration

---

1. RSS Feed Sources

1.1 Major AI Labs

Source	RSS Feed URL	Frequency	Quality	Notes
OpenAI Blog	https://openai.com/news/rss.xml	2-5x/week	Very High	Official announcements, research
OpenAI (alt)	https://openai.com/feed.xml?format=xml	2-5x/week	Very High	Alternate format
Anthropic News	No official RSS; use community feed	1-3x/week	Very High	Use RSSHub or scraping fallback
Anthropic Engineering	https://raw.githubusercontent.com/conoro/anthropic-engineering-rss-feed/main/anthropic_engineering_rss.xml	1-2x/week	Very High	Community-maintained
Google DeepMind	https://deepmind.google/blog/rss.xml	2-4x/week	Very High	Research + product updates
Google AI Blog	https://blog.google/technology/ai/rss/	3-5x/week	Very High	Broader Google AI coverage
Google Research	https://research.google/blog/rss/	1-2x/week	Very High	Deep research posts
Meta AI (Engineering)	https://engineering.fb.com/feed/	2-3x/week	High	AI research category at /category/ai-research/
Microsoft Research	https://www.microsoft.com/en-us/research/blog/feed/	3-5x/week	Very High	Broad research coverage
xAI	No official RSS available	Sporadic	High	Monitor https://x.ai/news via scraping
NVIDIA AI Blog	https://blogs.nvidia.com/feed/	3-5x/week	High	Hardware + software AI
NVIDIA Developer Blog	https://developer.nvidia.com/blog/feed	2-3x/week	High	Technical deep dives
AWS Machine Learning	https://aws.amazon.com/blogs/machine-learning/feed/	3-5x/week	Medium-High	Cloud ML, SageMaker
Intel AI Blog	https://ai.intel.com/blog/feed	1-2x/week	Medium	Hardware-focused AI

1.2 AI News Sites

Source	RSS Feed URL	Frequency	Quality	Notes
The Verge (AI)	https://www.theverge.com/rss/ai-artificial-intelligence/index.xml	5-10x/day	High	Mainstream AI coverage
Ars Technica (AI)	https://arstechnica.com/ai/feed/	2-5x/day	High	Technical depth
TechCrunch (AI)	https://techcrunch.com/category/artificial-intelligence/feed/	5-10x/day	Medium-High	Startup/industry focus
VentureBeat (AI)	https://venturebeat.com/category/ai/feed/	5-10x/day	Medium-High	Enterprise AI focus
MIT Technology Review	https://www.technologyreview.com/topic/artificial-intelligence/feed	2-5x/day	Very High	Deep analysis, paywalled
Wired (AI)	https://www.wired.com/feed/tag/ai/latest/rss	2-5x/day	High	Broad tech coverage
Wired (AI Category)	https://www.wired.com/feed/category/artificial-intelligence/rss	2-5x/day	High	Category-specific
IEEE Spectrum	https://spectrum.ieee.org/feeds/topic/artificial-intelligence.rss	1-3x/day	Very High	Engineering perspective
Nature (ML)	https://www.nature.com/natmachintell.rss	Weekly	Very High	Premier research journal

1.3 Research & Technical Blogs

Source	RSS Feed URL	Frequency	Quality	Notes
Towards Data Science	https://towardsdatascience.com/feed	10+/day	Medium	Community posts, variable quality
Machine Learning Mastery	https://machinelearningmastery.com/blog/feed	2-3x/week	High	Tutorials, accessible
fast.ai	https://www.fast.ai/atom.xml	1-2x/month	Very High	Jeremy Howard's blog
Distill.pub	https://distill.pub/rss.xml	Sporadic	Very High	Interactive ML explanations (inactive since 2021)
BAIR Blog (Berkeley)	https://bair.berkeley.edu/blog/feed.xml	1-2x/month	Very High	Academic research
Lil'Log (Lilian Weng)	https://lilianweng.github.io/index.xml	Monthly	Very High	OpenAI researcher, deep tutorials
Jay Alammar	https://jalammar.github.io/feed.xml	Monthly	Very High	Visual ML explanations
Colah's Blog	https://colah.github.io/rss.xml	Rare	Very High	Classic neural net explanations
The Gradient	https://thegradientpub.substack.com/feed	1-2x/week	Very High	In-depth AI perspectives
WildML	https://www.wildml.com/feed/	Inactive	High	Historical value

1.4 Newsletters (with RSS)

Source	RSS Feed URL	Frequency	Quality	Notes
The Batch (Andrew Ng)	https://www.deeplearning.ai/the-batch/feed/	Weekly	Very High	Curated AI news
Import AI (Jack Clark)	https://importai.substack.com/feed	Weekly	Very High	AI policy + research, Anthropic co-founder
The Gradient	https://thegradientpub.substack.com/feed	1-2x/week	Very High	Research perspectives
Ahead of AI (Raschka)	https://magazine.sebastianraschka.com/feed	Biweekly	Very High	Deep ML research analysis
Last Week in AI	https://lastweekin.ai/feed	Weekly	High	Comprehensive AI news roundup
AI Weekly	https://aiweekly.co/issues.rss	Weekly	Medium-High	Curated links
TLDR AI	https://tldr.tech/ai/rss	Daily	Medium-High	Concise daily digest
Interconnects (Lambert)	https://www.interconnects.ai/feed	1-2x/week	Very High	RLHF, alignment research
Davis Summarizes Papers	https://dblalock.substack.com/feed	Weekly	High	Paper summaries

1.5 Industry / VC Blogs

Source	RSS Feed URL	Frequency	Quality	Notes
a16z Blog	https://a16z.com/feed/	3-5x/week	High	Venture capital AI perspective
a16z AI Podcast	https://feeds.simplecast.com/Hb_IuXOo	Weekly	High	Audio content
Y Combinator Blog	https://www.ycombinator.com/blog/feed/	1-2x/week	High	Startup ecosystem
Sequoia Capital	https://www.sequoiacap.com/feed/	1-2x/month	High	Investment perspective

1.6 Company Engineering Blogs (AI-relevant)

Source	RSS Feed URL	Frequency	Quality
Netflix TechBlog	https://netflixtechblog.com/feed	1-2x/week	High
Uber Engineering	https://www.uber.com/blog/engineering/rss/	1-2x/week	High
Spotify Engineering	https://engineering.atspotify.com/feed/	1-2x/week	Medium-High
LinkedIn Engineering	https://engineering.linkedin.com/blog.rss.html	1-2x/week	High
Airbnb Tech	https://medium.com/feed/airbnb-engineering	1-2x/week	High

---

2. GitHub Trending Configuration

2.1 Fetcher Strategy

GitHub does not have an official "trending" API. Use a combination of:

1. GitHub Search API (https://api.github.com/search/repositories) with date-filtered star queries
2. GitHub RSS Feeds for release tracking
3. Unofficial trending scrapers like https://github.com/trending (HTML scraping)

2.2 Best Topics/Tags for AI Repos

{
  "primary_topics": [
    "artificial-intelligence",
    "machine-learning",
    "deep-learning",
    "large-language-models",
    "llm",
    "natural-language-processing",
    "computer-vision",
    "generative-ai",
    "transformer",
    "reinforcement-learning"
  ],
  "trending_topics_2025_2026": [
    "ai-agents",
    "rag",
    "retrieval-augmented-generation",
    "fine-tuning",
    "mlops",
    "text-to-image",
    "text-to-video",
    "multimodal",
    "diffusion-models",
    "vector-database",
    "function-calling",
    "mcp",
    "model-context-protocol"
  ]
}

2.3 Key Repos to Track for Releases

Use the GitHub Releases API: https://api.github.com/repos/{owner}/{repo}/releases

Or the Releases RSS feed: https://github.com/{owner}/{repo}/releases.atom

Repo	Stars (approx)	Category	Release Feed
huggingface/transformers	140k+	NLP/ML Framework	https://github.com/huggingface/transformers/releases.atom
pytorch/pytorch	85k+	ML Framework	https://github.com/pytorch/pytorch/releases.atom
langchain-ai/langchain	100k+	LLM Framework	https://github.com/langchain-ai/langchain/releases.atom
run-llama/llama_index	40k+	RAG Framework	https://github.com/run-llama/llama_index/releases.atom
ollama/ollama	110k+	Local LLM Runner	https://github.com/ollama/ollama/releases.atom
vllm-project/vllm	40k+	LLM Inference	https://github.com/vllm-project/vllm/releases.atom
ggerganov/llama.cpp	75k+	LLM Inference (C++)	https://github.com/ggerganov/llama.cpp/releases.atom
openai/openai-python	25k+	OpenAI SDK	https://github.com/openai/openai-python/releases.atom
anthropics/anthropic-sdk-python	5k+	Anthropic SDK	https://github.com/anthropics/anthropic-sdk-python/releases.atom
microsoft/autogen	40k+	Multi-agent	https://github.com/microsoft/autogen/releases.atom
crewAIInc/crewAI	25k+	AI Agents	https://github.com/crewAIInc/crewAI/releases.atom
AUTOMATIC1111/stable-diffusion-webui	145k+	Image Gen UI	https://github.com/AUTOMATIC1111/stable-diffusion-webui/releases.atom
comfyanonymous/ComfyUI	65k+	Image Gen UI	https://github.com/comfyanonymous/ComfyUI/releases.atom
lm-sys/FastChat	38k+	LLM Chat/Eval	https://github.com/lm-sys/FastChat/releases.atom
tensorflow/tensorflow	187k+	ML Framework	https://github.com/tensorflow/tensorflow/releases.atom
modelcontextprotocol/servers	15k+	MCP Servers	https://github.com/modelcontextprotocol/servers/releases.atom
dagger/dagger	12k+	AI Pipelines	https://github.com/dagger/dagger/releases.atom
open-webui/open-webui	60k+	LLM Web UI	https://github.com/open-webui/open-webui/releases.atom

2.4 GitHub Search API: Trending Detection Strategy

GET https://api.github.com/search/repositories
  ?q=topic:machine-learning+created:>2026-02-01+stars:>50
  &sort=stars
  &order=desc
  &per_page=30

Rate Limits:

• Unauthenticated: 10 requests/minute
• Authenticated: 30 requests/minute for search API
• General API: 5,000 requests/hour (authenticated)

Recommended queries (rotate daily):

stars:>100 created:>{7_days_ago} topic:artificial-intelligence
stars:>50 created:>{7_days_ago} topic:llm
stars:>50 pushed:>{1_day_ago} topic:machine-learning sort:stars
language:python topic:deep-learning stars:>20 created:>{30_days_ago}

---

3. ArXiv Categories & Configuration

3.1 RSS Feed Format

Base URL: https://rss.arxiv.org/rss/{category}
ATOM URL: https://rss.arxiv.org/atom/{category}
Combined: https://rss.arxiv.org/rss/{cat1}+{cat2}+{cat3}
Max results: 2000 per feed request

Update Schedule: Daily at midnight Eastern US time. No updates on Saturday/Sunday (papers announced Mon-Thu and Sun at 8pm ET).

Feed Status: https://rss.arxiv.org/feed/status

3.2 Most Relevant Categories

Category	Name	Daily Volume (est.)	Relevance	Priority
cs.AI	Artificial Intelligence	~80-120/day	Very High	P0
cs.CL	Computation & Language (NLP)	~150-250/day	Very High	P0
cs.CV	Computer Vision	~200-300/day	High	P0
cs.LG	Machine Learning	~250-400/day	Very High	P0
stat.ML	Statistics - Machine Learning	~50-80/day	High	P1
cs.IR	Information Retrieval	~30-50/day	Medium-High	P1
cs.RO	Robotics	~40-60/day	Medium	P2
cs.NE	Neural & Evolutionary Computing	~20-30/day	Medium	P2
cs.MA	Multiagent Systems	~10-20/day	Medium	P2
cs.HC	Human-Computer Interaction	~20-30/day	Low-Medium	P3
cs.CR	Cryptography (AI security)	~30-50/day	Low-Medium	P3
eess.AS	Audio & Speech Processing	~20-30/day	Medium	P2

Total estimated daily volume for P0 categories: ~700-1100 papers/day

3.3 Recommended Combined Feed

# Primary feed (core AI/ML):
https://rss.arxiv.org/rss/cs.AI+cs.CL+cs.CV+cs.LG

# Secondary feed (supporting areas):
https://rss.arxiv.org/rss/stat.ML+cs.IR+cs.RO+cs.NE

3.4 Quality Filtering Strategies

ArXiv does not have a built-in quality signal, so apply post-fetch filtering:

1. Citation velocity - Cross-reference with Semantic Scholar API (https://api.semanticscholar.org/graph/v1/paper/arXiv:{id}) to check citation counts
2. Author authority - Track known prolific researchers (h-index proxies)
3. HuggingFace Daily Papers - Cross-reference with HF papers (curated selection)
4. Social signal - Cross-reference with HN/Reddit/Twitter mentions
5. Keyword scoring - Boost papers mentioning key terms (see Section 9)
6. Institutional affiliation - Papers from top labs (Google, Meta, OpenAI, etc.) get priority

3.5 ArXiv Search API (Alternative)

GET http://export.arxiv.org/api/query
  ?search_query=cat:cs.LG
  &start=0
  &max_results=100
  &sortBy=submittedDate
  &sortOrder=descending

Rate Limit: 1 request every 3 seconds. Use bulk downloads for large queries.

---

4. Hacker News Configuration

4.1 API Endpoints

Base URL: https://hn.algolia.com/api/v1

Endpoint	Description	Sort
GET /search?query=...	Search (relevance)	Relevance, then points, then comments
GET /search_by_date?query=...	Search (chronological)	Most recent first
GET /items/:id	Get item details	N/A

4.2 Query Parameters

Parameter	Description	Example
query	Search string	query=large language model
tags	Filter by type (AND by default, OR in parens)	tags=story or tags=(story,poll)
numericFilters	Numeric field filters	numericFilters=points>100,created_at_i>1706745600
hitsPerPage	Results per page (max 1000)	hitsPerPage=50
page	Pagination	page=0

Tag Values:

• story, comment, poll, pollopt, show_hn, ask_hn, front_page
• author_{username} - filter by author

4.3 Recommended Search Queries for AI Content

Run these queries daily via search_by_date with tags=story and numericFilters=points>10:

{
  "primary_queries": [
    "artificial intelligence",
    "machine learning",
    "large language model",
    "LLM",
    "GPT",
    "Claude",
    "Gemini AI",
    "neural network",
    "deep learning",
    "transformer model"
  ],
  "secondary_queries": [
    "OpenAI",
    "Anthropic",
    "DeepMind",
    "computer vision",
    "natural language processing",
    "fine-tuning",
    "RAG retrieval",
    "AI safety",
    "AI alignment",
    "diffusion model",
    "text-to-image",
    "AI agent",
    "prompt engineering",
    "AI regulation"
  ],
  "trending_queries_2025_2026": [
    "AI coding",
    "vibe coding",
    "model context protocol",
    "AI reasoning",
    "open source LLM",
    "local LLM",
    "multimodal AI",
    "AI video generation",
    "AI chips",
    "AI inference"
  ]
}

4.4 Recommended Configuration

const HN_CONFIG = {
  // Minimum points to consider a story
  minPoints: 10,

  // For front-page quality stories
  highQualityMinPoints: 50,

  // Time window: last 24 hours
  timeWindowHours: 24,

  // Max stories per fetch cycle
  hitsPerPage: 50,

  // Polling interval
  fetchIntervalMinutes: 30,

  // Example API call:
  // https://hn.algolia.com/api/v1/search_by_date?query=AI&tags=story&numericFilters=points>10,created_at_i>{unix_24h_ago}&hitsPerPage=50
};

4.5 Rate Limits

The HN Algolia API is generous but undocumented officially. Community consensus:

• ~10,000 requests/hour is safe
• Use reasonable delays (1-2 seconds between requests)
• Cache results aggressively

---

5. HuggingFace Hub

5.1 API Endpoints

Base URL: https://huggingface.co/api

Endpoint	Description	Auth Required
GET /api/models	List/search models	No (rate-limited)
GET /api/models?sort=trending	Trending models	No
GET /api/models?sort=downloads	Most downloaded	No
GET /api/models?sort=likes	Most liked	No
GET /api/models?sort=lastModified	Recently updated	No
GET /api/datasets	List/search datasets	No
GET /api/datasets?sort=trending	Trending datasets	No
GET /api/spaces	List/search spaces	No
GET /api/spaces?sort=trending	Trending spaces	No
GET /api/daily_papers	Daily curated papers	No
GET /api/daily_papers?date=2026-02-07	Papers for specific date	No

OpenAPI Spec: https://huggingface.co/.well-known/openapi.json

5.2 Useful Model Query Parameters

Parameter	Values	Description
sort	trending, downloads, likes, lastModified, created	Sort order
direction	-1 (desc), 1 (asc)	Sort direction
limit	1-100	Results per page
filter	tag filter	e.g., text-generation, image-classification
search	text query	Search in model name/description
author	username	Filter by author
library	framework name	e.g., transformers, diffusers, gguf

5.3 What to Track

Content Type	Endpoint	Frequency	Value
Trending Models	/api/models?sort=trending&limit=30	Every 6 hours	High - shows what's hot
Daily Papers	/api/daily_papers	Daily	Very High - curated research
Trending Spaces	/api/spaces?sort=trending&limit=20	Every 12 hours	High - demos/apps
New Datasets	/api/datasets?sort=lastModified&limit=20	Daily	Medium
Text-gen Models	/api/models?sort=trending&filter=text-generation&limit=20	Every 12 hours	High

5.4 Rate Limits

• Unauthenticated: ~100 requests/hour (estimated)
• Authenticated (free token): Higher limits
• Get a token at https://huggingface.co/settings/tokens
• Pass via header: Authorization: Bearer hf_xxxxx

5.5 Webhooks

HuggingFace supports webhooks for real-time notifications on repo changes:

• Configure at: https://huggingface.co/settings/webhooks
• Triggers: model updates, new models, space deployments, paper additions

---

6. Reddit Subreddits

6.1 Recommended Subreddits

Subreddit	Subscribers	Content Type	Quality	SNR
r/MachineLearning	3M+	Research papers, industry news	Very High	High
r/artificial	500k+	General AI news	Medium	Medium
r/LocalLLaMA	500k+	Open-source LLMs, local inference	High	High
r/ChatGPT	5M+	ChatGPT-specific, consumer AI	Low-Medium	Low
r/singularity	1M+	AGI speculation, AI news	Low-Medium	Low
r/LanguageTechnology	50k+	NLP research	High	High
r/deeplearning	200k+	DL research and tutorials	High	Medium-High
r/reinforcementlearning	50k+	RL research	High	High
r/StableDiffusion	500k+	Image generation	Medium	Medium
r/Oobabooga	50k+	Local text generation	Medium	Medium
r/OpenAI	1M+	OpenAI products/news	Medium	Low-Medium
r/ClaudeAI	200k+	Anthropic products/news	Medium	Low-Medium

6.2 API Configuration

Base URL: https://oauth.reddit.com

Authentication: OAuth 2.0 required (no anonymous API access since 2023)

const REDDIT_CONFIG = {
  // OAuth endpoint
  tokenUrl: "https://www.reddit.com/api/v1/access_token",

  // Listing endpoints
  hotEndpoint: "/r/{subreddit}/hot.json",
  topEndpoint: "/r/{subreddit}/top.json?t=day",
  newEndpoint: "/r/{subreddit}/new.json",

  // Search endpoint
  searchEndpoint: "/r/{subreddit}/search.json?q={query}&sort=top&t=day",

  // Rate limits
  requestsPerMinute: 60, // with OAuth
  requestWindow: "10 minute rolling average",

  // Recommended: fetch top daily from each subreddit
  fetchIntervalMinutes: 60,
  minScore: 50, // for r/MachineLearning
  minScoreGeneral: 100, // for larger subs like r/ChatGPT
};

6.3 Rate Limits (2025-2026)

• OAuth authenticated: 100 queries per minute (QPM) per client ID, averaged over 10-minute window
• Unauthenticated: 10 requests/minute (IP-based) - NOT recommended
• Free tier: Non-commercial use only (personal projects, academic research)
• Commercial: Requires prior approval, may incur fees
• Important: Rate limits apply per OAuth client ID, not per user

6.4 RSS Alternative (No Auth Needed)

Reddit still serves RSS feeds without authentication:

https://www.reddit.com/r/MachineLearning/top/.rss?t=day
https://www.reddit.com/r/LocalLLaMA/hot/.rss
https://www.reddit.com/r/MachineLearning/.rss?limit=25

Limitation: RSS feeds have fewer fields (no score in RSS), limited to 25 items, and may be rate-limited by IP. Best used as a fallback or supplement.

---

7. Product Hunt

7.1 Current API Status

The Product Hunt V2 GraphQL API (https://api.producthunt.com/v2/api/graphql) remains available but with significant limitations:

• V1 REST API is fully deprecated
• V2 responses have redacted maker names/usernames since Feb 2023
• Twitter usernames return None
• Authentication required via OAuth

7.2 GraphQL Query for AI Products

query {
  posts(
    order: RANKING
    topic: "artificial-intelligence"
    postedAfter: "2026-02-06T00:00:00Z"
    first: 20
  ) {
    edges {
      node {
        id
        name
        tagline
        description
        url
        votesCount
        commentsCount
        createdAt
        topics {
          edges {
            node {
              name
            }
          }
        }
        thumbnail {
          url
        }
      }
    }
  }
}

7.3 Relevant Topics

artificial-intelligence, machine-learning, chatgpt, developer-tools,
saas, productivity, open-source, api, no-code, automation

7.4 Rate Limits

• Authenticated: ~450 requests per 15-minute window
• Requires API key from: https://www.producthunt.com/v2/oauth/applications

---

8. Additional Sources Not Yet Implemented

8.1 YouTube AI Channels

YouTube provides RSS feeds per channel: https://www.youtube.com/feeds/videos.xml?channel_id={CHANNEL_ID}

Channel	Channel ID	Subscribers	Content	Feed URL
Two Minute Papers	UCbfYPyITQ-7l4upoX8nvctg	1.6M	AI paper summaries	https://www.youtube.com/feeds/videos.xml?channel_id=UCbfYPyITQ-7l4upoX8nvctg
Yannic Kilcher	UCZHmQk67mSJgfCCTn7xBfew	250k+	Deep paper analysis	https://www.youtube.com/feeds/videos.xml?channel_id=UCZHmQk67mSJgfCCTn7xBfew
AI Explained	UCNJ1Ymd5yFuUPtn21xtRbbw	500k+	AI capability analysis	https://www.youtube.com/feeds/videos.xml?channel_id=UCNJ1Ymd5yFuUPtn21xtRbbw
Matthew Berman	UCMp36FHKAQ-HqVNRFCK6VlA	330k+	AI news & tutorials	https://www.youtube.com/feeds/videos.xml?channel_id=UCMp36FHKAQ-HqVNRFCK6VlA
3Blue1Brown	UCYO_jab_esuFRV4b17AJtAw	6M+	Math/ML explanations	https://www.youtube.com/feeds/videos.xml?channel_id=UCYO_jab_esuFRV4b17AJtAw
Fireship	UCsBjURrPoezykLs9EqgamOA	3M+	Quick tech explainers	https://www.youtube.com/feeds/videos.xml?channel_id=UCsBjURrPoezykLs9EqgamOA
Andrej Karpathy	UCXUPKJO5MZQN11PqgIvyuvQ	1M+	Deep ML tutorials	https://www.youtube.com/feeds/videos.xml?channel_id=UCXUPKJO5MZQN11PqgIvyuvQ
Lex Fridman	UCSHZKyawb77ixDdsGog4iWA	4M+	AI interviews	https://www.youtube.com/feeds/videos.xml?channel_id=UCSHZKyawb77ixDdsGog4iWA

Implementation effort: Low - YouTube RSS feeds are free, reliable, and require no authentication. Would need a new YouTubeFetcher that parses RSS and extracts video metadata.

Recommendation: HIGH PRIORITY - Easy to implement, high-value content.

8.2 Twitter/X Lists

Current State: X API pricing is prohibitive for most use cases:

• Free tier: 1 request per 15 minutes (read), write-only focus
• Basic: $200/month, 10,000 tweets/month read
• Pro: $5,000/month, 1M tweets read

Alternatives:

• TwitterAPI.io: $0.15 per 1,000 tweets (unofficial third-party)
• Apify scraping actors: Pay-per-use, no API keys needed
• Nitter instances: Mostly defunct as of 2024-2025

Key AI Accounts to Track (if budget allows):

@ylecun, @kaboris, @AndrewYNg, @demaboris, @sama,
@elaboris, @hardmaru, @jeffdean, @goodfellow_ian,
@iaboris, @OpenAI, @AnthropicAI, @GoogleDeepMind,
@xaboris, @huggingface, @weights_biases

Recommendation: LOW PRIORITY - Cost prohibitive. Consider implementing later if budget allows, using third-party services.

8.3 Conference Proceedings

Conference	Frequency	Proceedings URL	Access
NeurIPS	Annual (Dec)	https://proceedings.neurips.cc/	Open access
ICML	Annual (Jul)	https://proceedings.mlr.press/	Open access (PMLR)
ICLR	Annual (May)	https://openreview.net/group?id=ICLR.cc	Open access (OpenReview)
AAAI	Annual (Feb)	https://ojs.aaai.org/index.php/AAAI	Open access
ACL	Annual (Jul)	https://aclanthology.org/	Open access
EMNLP	Annual (Dec)	https://aclanthology.org/	Open access
CVPR	Annual (Jun)	https://openaccess.thecvf.com/	Open access

RSS for Conference Papers: The CPR-RSS project (https://github.com/CPR-RSS/CPR-RSS.github.io) provides RSS feeds for NeurIPS, ICML, and ICLR.

Recommendation: MEDIUM PRIORITY - Implement as a seasonal source. Most accepted papers first appear on ArXiv, so there's overlap. Main value is the "accepted at top venue" quality signal.

8.4 Semantic Scholar API

Base URL: https://api.semanticscholar.org/graph/v1

GET /paper/search?query=large+language+model&year=2026&fieldsOfStudy=Computer+Science
GET /paper/{paper_id}?fields=title,abstract,citationCount,influentialCitationCount
GET /paper/arXiv:{arxiv_id}

• Rate Limit: 1 request/second (unauthenticated), 10/second (with API key)
• Free API keys available at: https://www.semanticscholar.org/product/api

Recommendation: HIGH PRIORITY for quality scoring - Use citation counts and influential citation counts as quality signals for ArXiv papers.

8.5 Patent Filings

• Google Patents: https://patents.google.com/ - No official API
• USPTO API: https://developer.uspto.gov/api-catalog - Free, structured data
• Lens.org: https://www.lens.org/ - Scholarly + patent search API

Recommendation: LOW PRIORITY - Patents are lagging indicators, often 18 months behind research.

---

9. Source Quality Scoring Framework

9.1 Source Authority Scoring

Assign each source a base authority score (0-100):

const SOURCE_AUTHORITY: Record<string, number> = {
  // Tier 1: Primary research labs (90-100)
  "openai.com": 98,
  "anthropic.com": 97,
  "deepmind.google": 98,
  "ai.meta.com": 95,
  "microsoft.com/research": 95,
  "arxiv.org": 90, // raw, but foundational

  // Tier 2: Premier publications (80-89)
  "nature.com": 95,
  "technologyreview.com": 88,
  "spectrum.ieee.org": 85,
  "proceedings.neurips.cc": 95,
  "proceedings.mlr.press": 93,

  // Tier 3: Quality tech press (70-79)
  "theverge.com": 75,
  "arstechnica.com": 78,
  "wired.com": 75,
  "techcrunch.com": 72,
  "venturebeat.com": 70,

  // Tier 4: Community/curated (60-69)
  "huggingface.co": 80, // curated papers are high quality
  "reddit.com/r/MachineLearning": 65,
  "news.ycombinator.com": 60,
  "github.com": 65,

  // Tier 5: Aggregated/variable (40-59)
  "towardsdatascience.com": 50,
  "reddit.com/r/ChatGPT": 40,
  "producthunt.com": 55,
};

9.2 Freshness Weighting

function freshnessScore(publishedAt: Date): number {
  const hoursOld = (Date.now() - publishedAt.getTime()) / (1000 * 60 * 60);

  if (hoursOld < 6) return 1.0;      // Breaking
  if (hoursOld < 24) return 0.9;     // Fresh
  if (hoursOld < 48) return 0.75;    // Recent
  if (hoursOld < 72) return 0.5;     // Aging
  if (hoursOld < 168) return 0.3;    // Week old
  return 0.1;                         // Stale
}

9.3 Content Quality Signals

Signal	Weight	Source
Source authority	0.30	Pre-assigned per source
Social engagement (normalized)	0.20	Points/upvotes/stars from HN/Reddit/GitHub
Freshness	0.15	Time since publication
Content depth (word count)	0.10	Longer = deeper analysis
Author authority	0.10	Known researchers/journalists
Cross-source mentions	0.10	Same story from multiple sources
Topic relevance	0.05	Keyword/embedding match to AI topics

9.4 Composite Score Formula

function computeScore(article: Article): number {
  const authority = SOURCE_AUTHORITY[article.sourceDomain] / 100;
  const freshness = freshnessScore(article.publishedAt);
  const engagement = normalizeEngagement(article.points, article.sourceType);
  const depth = Math.min(article.wordCount / 2000, 1.0);
  const authorScore = knownAuthors.has(article.author) ? 1.0 : 0.5;
  const crossRef = crossReferenceCount(article.title) > 1 ? 1.0 : 0.5;
  const relevance = computeTopicRelevance(article.title + article.summary);

  return (
    authority * 0.30 +
    engagement * 0.20 +
    freshness * 0.15 +
    depth * 0.10 +
    authorScore * 0.10 +
    crossRef * 0.10 +
    relevance * 0.05
  );
}

9.5 Signal-to-Noise Ratio by Source

Source Type	Est. SNR	Strategy
AI Lab Blogs	95%	Accept all
Nature/IEEE	90%	Accept all
ArXiv (curated via HF)	85%	Accept all from HF daily papers
ArXiv (raw feed)	20-30%	Heavy filtering needed
HN (points > 50)	70%	Good after point threshold
Reddit r/MachineLearning	60%	Filter by score + flair
Tech press (Verge, etc.)	50%	Filter duplicates, prioritize exclusives
TDS / Medium	30%	Heavy filtering, many low-quality posts
Reddit r/ChatGPT	15%	Very noisy, mostly user complaints
Product Hunt	40%	Many "AI-washed" products

---

10. Deduplication Strategies

10.1 Multi-Layer Approach

Layer 1: URL Normalization
  - Strip query params, tracking codes, UTM tags
  - Normalize www vs non-www
  - Detect URL redirects to canonical

Layer 2: Title Similarity
  - Normalize: lowercase, strip punctuation, remove stop words
  - Levenshtein distance threshold: < 0.15 (85% similar = duplicate)
  - Jaccard similarity on word tokens: > 0.7 = likely duplicate

Layer 3: Content Embedding Similarity
  - Generate embeddings with a small model (e.g., all-MiniLM-L6-v2)
  - Cosine similarity threshold: > 0.95 = duplicate
  - 0.85-0.95 = related/rewrite, flag for review

Layer 4: Entity Extraction
  - Extract key entities (company names, model names, person names)
  - If 3+ entities match AND publication date within 48 hours = likely duplicate

10.2 Cross-Source Deduplication

When the same story appears across multiple sources:

1. Keep the highest authority version as the primary
2. Store other versions as related_sources for citation diversity
3. Boost the composite score (cross-source validation)

10.3 ArXiv-Specific Deduplication

• ArXiv papers have unique IDs (e.g., 2401.12345)
• The same paper may appear in multiple category feeds
• Deduplicate by ArXiv ID, keeping the primary category
• Track paper versions (v1, v2, etc.) - only alert on v1

---

11. Recommended Seed Configuration

11.1 Phase 1: Core Sources (Launch)

Priority sources to populate on day 1:

const SEED_SOURCES = [
  // RSS - AI Labs (5 sources)
  { type: "rss", name: "OpenAI Blog", url: "https://openai.com/news/rss.xml", category: "lab", priority: 1 },
  { type: "rss", name: "Anthropic Engineering", url: "https://raw.githubusercontent.com/conoro/anthropic-engineering-rss-feed/main/anthropic_engineering_rss.xml", category: "lab", priority: 1 },
  { type: "rss", name: "Google DeepMind", url: "https://deepmind.google/blog/rss.xml", category: "lab", priority: 1 },
  { type: "rss", name: "Meta AI Engineering", url: "https://engineering.fb.com/feed/", category: "lab", priority: 1 },
  { type: "rss", name: "Microsoft Research", url: "https://www.microsoft.com/en-us/research/blog/feed/", category: "lab", priority: 1 },

  // RSS - News (5 sources)
  { type: "rss", name: "The Verge AI", url: "https://www.theverge.com/rss/ai-artificial-intelligence/index.xml", category: "news", priority: 1 },
  { type: "rss", name: "Ars Technica AI", url: "https://arstechnica.com/ai/feed/", category: "news", priority: 1 },
  { type: "rss", name: "TechCrunch AI", url: "https://techcrunch.com/category/artificial-intelligence/feed/", category: "news", priority: 1 },
  { type: "rss", name: "MIT Tech Review AI", url: "https://www.technologyreview.com/topic/artificial-intelligence/feed", category: "news", priority: 1 },
  { type: "rss", name: "VentureBeat AI", url: "https://venturebeat.com/category/ai/feed/", category: "news", priority: 1 },

  // RSS - Newsletters (4 sources)
  { type: "rss", name: "The Batch (Andrew Ng)", url: "https://www.deeplearning.ai/the-batch/feed/", category: "newsletter", priority: 1 },
  { type: "rss", name: "Import AI", url: "https://importai.substack.com/feed", category: "newsletter", priority: 1 },
  { type: "rss", name: "Ahead of AI", url: "https://magazine.sebastianraschka.com/feed", category: "newsletter", priority: 1 },
  { type: "rss", name: "Last Week in AI", url: "https://lastweekin.ai/feed", category: "newsletter", priority: 1 },

  // ArXiv (1 combined feed)
  { type: "arxiv", name: "ArXiv AI/ML/NLP/CV", url: "https://rss.arxiv.org/rss/cs.AI+cs.CL+cs.CV+cs.LG", category: "research", priority: 1 },

  // HuggingFace (1 source)
  { type: "huggingface", name: "HF Daily Papers", url: "https://huggingface.co/api/daily_papers", category: "research", priority: 1 },

  // Hacker News (1 source)
  { type: "hackernews", name: "HN AI Stories", url: "https://hn.algolia.com/api/v1/search_by_date?query=AI+OR+LLM+OR+%22machine+learning%22&tags=story&numericFilters=points>20&hitsPerPage=50", category: "community", priority: 1 },

  // GitHub (tracked repos)
  { type: "github", name: "GitHub AI Releases", url: "https://api.github.com/search/repositories?q=topic:machine-learning+stars:>100+pushed:>{yesterday}", category: "code", priority: 1 },

  // Reddit (2 subreddits)
  { type: "reddit", name: "r/MachineLearning", url: "https://www.reddit.com/r/MachineLearning/top/.rss?t=day", category: "community", priority: 1 },
  { type: "reddit", name: "r/LocalLLaMA", url: "https://www.reddit.com/r/LocalLLaMA/hot/.rss", category: "community", priority: 1 },

  // Product Hunt (1 source)
  { type: "producthunt", name: "PH AI Products", url: "https://api.producthunt.com/v2/api/graphql", category: "products", priority: 2 },
];

11.2 Phase 2: Expansion Sources

Add after launch once pipeline is stable:

const EXPANSION_SOURCES = [
  // More RSS feeds
  { type: "rss", name: "Wired AI", url: "https://www.wired.com/feed/tag/ai/latest/rss", category: "news", priority: 2 },
  { type: "rss", name: "IEEE Spectrum AI", url: "https://spectrum.ieee.org/feeds/topic/artificial-intelligence.rss", category: "news", priority: 2 },
  { type: "rss", name: "NVIDIA AI Blog", url: "https://blogs.nvidia.com/feed/", category: "lab", priority: 2 },
  { type: "rss", name: "AWS ML Blog", url: "https://aws.amazon.com/blogs/machine-learning/feed/", category: "lab", priority: 2 },
  { type: "rss", name: "Google AI Blog", url: "https://blog.google/technology/ai/rss/", category: "lab", priority: 2 },
  { type: "rss", name: "The Gradient", url: "https://thegradientpub.substack.com/feed", category: "newsletter", priority: 2 },
  { type: "rss", name: "Interconnects", url: "https://www.interconnects.ai/feed", category: "newsletter", priority: 2 },
  { type: "rss", name: "fast.ai", url: "https://www.fast.ai/atom.xml", category: "research", priority: 2 },
  { type: "rss", name: "BAIR Blog", url: "https://bair.berkeley.edu/blog/feed.xml", category: "research", priority: 2 },
  { type: "rss", name: "Lil'Log", url: "https://lilianweng.github.io/index.xml", category: "research", priority: 2 },
  { type: "rss", name: "Nature Machine Intelligence", url: "https://www.nature.com/natmachintell.rss", category: "research", priority: 2 },

  // More Reddit
  { type: "reddit", name: "r/artificial", url: "https://www.reddit.com/r/artificial/top/.rss?t=day", category: "community", priority: 2 },
  { type: "reddit", name: "r/deeplearning", url: "https://www.reddit.com/r/deeplearning/top/.rss?t=day", category: "community", priority: 2 },

  // ArXiv secondary
  { type: "arxiv", name: "ArXiv Stats/IR/Robotics", url: "https://rss.arxiv.org/rss/stat.ML+cs.IR+cs.RO", category: "research", priority: 2 },

  // GitHub releases (individual repos)
  { type: "github", name: "Transformers Releases", url: "https://github.com/huggingface/transformers/releases.atom", category: "code", priority: 2 },
  { type: "github", name: "Ollama Releases", url: "https://github.com/ollama/ollama/releases.atom", category: "code", priority: 2 },
  { type: "github", name: "LangChain Releases", url: "https://github.com/langchain-ai/langchain/releases.atom", category: "code", priority: 2 },
  { type: "github", name: "vLLM Releases", url: "https://github.com/vllm-project/vllm/releases.atom", category: "code", priority: 2 },
  { type: "github", name: "llama.cpp Releases", url: "https://github.com/ggerganov/llama.cpp/releases.atom", category: "code", priority: 2 },
];

11.3 Phase 3: YouTube & Premium Sources

const PREMIUM_SOURCES = [
  // YouTube channels (new fetcher needed)
  { type: "youtube", name: "Two Minute Papers", url: "https://www.youtube.com/feeds/videos.xml?channel_id=UCbfYPyITQ-7l4upoX8nvctg", category: "video", priority: 3 },
  { type: "youtube", name: "Yannic Kilcher", url: "https://www.youtube.com/feeds/videos.xml?channel_id=UCZHmQk67mSJgfCCTn7xBfew", category: "video", priority: 3 },
  { type: "youtube", name: "AI Explained", url: "https://www.youtube.com/feeds/videos.xml?channel_id=UCNJ1Ymd5yFuUPtn21xtRbbw", category: "video", priority: 3 },
  { type: "youtube", name: "Andrej Karpathy", url: "https://www.youtube.com/feeds/videos.xml?channel_id=UCXUPKJO5MZQN11PqgIvyuvQ", category: "video", priority: 3 },

  // Semantic Scholar (for quality enrichment)
  { type: "semanticscholar", name: "SS Citation Data", url: "https://api.semanticscholar.org/graph/v1", category: "enrichment", priority: 3 },

  // Conference proceedings (seasonal)
  { type: "conference", name: "NeurIPS Proceedings", url: "https://proceedings.neurips.cc/", category: "research", priority: 3 },
  { type: "conference", name: "ICML Proceedings", url: "https://proceedings.mlr.press/", category: "research", priority: 3 },
];

11.4 Fetch Schedule Summary

Source Type	Fetch Interval	Daily API Calls (est.)	Cost
RSS Feeds (~20)	Every 30 min	~960	Free
ArXiv RSS (2)	Every 24 hours	~2	Free
HuggingFace API (5 queries)	Every 6-12 hours	~15	Free
Hacker News Algolia (15 queries)	Every 30 min	~720	Free
GitHub Search API	Every 2 hours	~12	Free
GitHub Release Feeds (~15)	Every 6 hours	~60	Free
Reddit RSS (~5)	Every 60 min	~120	Free
Product Hunt GraphQL	Every 24 hours	~1	Free
YouTube RSS (~4)	Every 6 hours	~16	Free
Total		~1,906/day	Free

All sources in the recommended configuration operate within free-tier API limits.

---

Appendix A: Key GitHub Repositories for Reference

• awesome-AI-feeds: https://github.com/RSS-Renaissance/awesome-AI-feeds - Curated OPML of AI feeds
• awesome-AI-news-feeds: https://github.com/RSS-Renaissance/awesome-AI-news-feeds - News-specific feeds
• allainews_sources: https://github.com/foorilla/allainews_sources - 100+ AI/ML news sources
• awesome_ML_AI_RSS_feed: https://github.com/vishalshar/awesome_ML_AI_RSS_feed - ML/AI/RL RSS feeds
• flexible-arxiv-rss: https://github.com/cschreib/flexible-arxiv-rss - Custom ArXiv RSS tool
• papers-with-code-rss: https://github.com/capjamesg/papers-with-code-rss - PWC RSS feeds
• CPR-RSS: https://github.com/CPR-RSS/CPR-RSS.github.io - Conference paper RSS

Appendix B: Useful External APIs

API	Base URL	Auth	Rate Limit	Free
Semantic Scholar	https://api.semanticscholar.org/graph/v1	API key (free)	1-10 req/sec	Yes
ArXiv Search	http://export.arxiv.org/api/query	None	1 req/3 sec	Yes
HN Algolia	https://hn.algolia.com/api/v1	None	~10k/hour	Yes
HuggingFace Hub	https://huggingface.co/api	Token (optional)	~100/hour unauth	Yes
GitHub Search	https://api.github.com/search	Token (recommended)	30 search/min	Yes
Reddit	https://oauth.reddit.com	OAuth required	100 req/min	Yes (non-commercial)
Product Hunt	https://api.producthunt.com/v2/api/graphql	OAuth required	~450/15 min	Yes