github-scraping-agent-ai

# GitHub Scraping AI

AI-powered GitHub project discovery. Automatically finds interesting repositories matching your interests and sends daily notifications to Discord.

## Features

- **Smart Filtering**: Uses LLM to evaluate repositories against your natural language interests
- **Keyword Search**: Filter GitHub search with custom keywords (OR logic)
- **Fork Exclusion**: Automatically filters out forked repositories
- **Multi-Provider Support**: Works with OpenAI, Anthropic (Gemini), or Google (Gemini)
- **Async & Concurrent**: Fetches READMEs concurrently for faster processing
- **Daily Discovery**: Fetches top starred repos from the last 24 hours
- **Discord Notifications**: Rich embeds with repo details and AI reasoning
- **Rejected Repos Log**: Logs rejected repos with reasons for prompt fine-tuning
- **Deduplication**: Tracks seen repos to avoid duplicates
- **Dry Run Mode**: Test without sending to Discord

## How It Works

```
GitHub API → Fetch top 1000 new repos → LLM evaluates each against your prompt → Discord notification
```

1. Queries GitHub for repositories created in the last 24h, sorted by stars
2. Fetches README excerpts for context
3. Sends each repo + your interests to an LLM for evaluation
4. Posts matching repos to Discord with the AI's reasoning
5. Caches seen repos to prevent duplicates

## Quick Start

### Prerequisites

- Python 3.13+
- [uv](https://docs.astral.sh/uv/) package manager
- API key for one of: OpenAI, Anthropic, or Google
- GitHub personal access token
- Discord webhook URL

### Installation

```bash
# Clone the repository
git clone https://github.com/yourusername/github-scraping-agent-ai.git
cd github-scraping-agent-ai

# Install dependencies
uv sync
```

### Configuration

1. **Copy example files:**
   ```bash
   cp config.example.json config.json
   cp prompt.example.md prompt.md
   ```

2. **Edit `config.json`** with your credentials:
   ```json
   {
     "github": {
       "token": "ghp_your_github_token",
       "key