Neura Market
Giskard logo

Giskard

Free

Testing & evaluation library for LLM applications, in particular RAGs

FreeFree tier
Type
Open Source
Visit Website GitHub (38 stars)

About Giskard

Giskard is an open-source Python library for testing and evaluating agentic systems, including LLM applications and RAGs. The v3 rewrite introduces a modular, lightweight, dynamic, and async-first architecture with focused packages: giskard-checks (evals with scenario API, built-in checks, LLM-as-judge), giskard-scan (agent vulnerability scanner for red teaming, prompt injection, data leakage), and planned giskard-rag (RAG evaluation synthetic data generation). It supports catching regressions, validating RAG quality, enforcing safety rules, and evaluating multi-turn agents. Built-in evals include string matching, comparisons, regex, semantic similarity, and LLM-as-judge checks such as Groundedness and Conformity.

Key Features

Modular, lightweight, async-first v3 architecture with focused packages
Built-in evals include string matching, comparisons, regex, semantic similarity, and LLM-as-judge checks (Groundedness, Conformity, LLMJudge)
Agent vulnerability scanner with red teaming, prompt injection, and data leakage detection
RAG evaluation support via planned giskard-rag package
Catch regressions, validate RAG quality, enforce safety rules, and evaluate multi-turn agents
Scenario API for creating and applying evals to test LLM-based systems
Open-source Python library with pip install giskard

Pros & Cons

Pros
  • Open-source and free to use
  • Modular lightweight packages reduce unnecessary dependencies
  • Async-first design for better performance in dynamic environments
  • Covers evals, red teaming, and RAG evaluation in one library
  • Supports multi-turn agent testing, not just single exchanges
  • Built-in LLM-as-judge checks for flexible evaluation
Cons
  • V3 is still in beta with some packages (e.g., giskard-rag) not yet released
  • Vulnerability scanner and RAG evaluation currently rely on Giskard v2, which is no longer actively maintained
  • Requires Python 3.12+
  • Telemetry may collect optional usage analytics (with opt-out available)

Best For

Testing and evaluating LLM applications and agentic systemsValidating RAG answer groundedness and retrieved context qualityRed teaming and vulnerability scanning for AI agentsCatching regressions after changes to AI pipelinesEnforcing content safety policies and output conformityEvaluating multi-turn conversational agents

FAQ

What is Giskard v3?
Giskard v3 is a complete rewrite of the library, designed for dynamic, multi-turn testing of AI agents. It drops heavy dependencies for better efficiency and introduces a more powerful AI vulnerability scanner and enhanced RAG evaluation capabilities.
What packages are included in Giskard v3?
Currently in beta: giskard-checks (testing evaluation with scenario API and built-in checks) and giskard-scan (agent vulnerability scanner). A giskard-rag package for RAG evaluation synthetic data generation is planned.
Does Giskard collect telemetry?
Libraries built on giskard-core (including giskard-checks) may send optional, aggregated usage analytics. No prompts, model outputs, or scenario text are included. Users can opt out.