Benchmarks

This project uses Criterion for benchmarking.

Running Benchmarks

All Benchmarks

cargo bench --package dependi-lsp --bench benchmarks

Specific Benchmark Group

# Parser benchmarks only
cargo bench --package dependi-lsp --bench benchmarks -- parsers

# Cache benchmarks only
cargo bench --package dependi-lsp --bench benchmarks -- cache

# Version utils benchmarks only
cargo bench --package dependi-lsp --bench benchmarks -- version

Save Baseline for Comparison

# Save current results as baseline
cargo bench --package dependi-lsp --bench benchmarks -- --save-baseline main

# Compare against saved baseline
cargo bench --package dependi-lsp --bench benchmarks -- --baseline main

Using the Helper Script

./run-benchmarks.sh              # Run all benchmarks
./run-benchmarks.sh parsers      # Run parser benchmarks only
./run-benchmarks.sh --baseline   # Save results as baseline
./run-benchmarks.sh --compare    # Compare against baseline

Benchmark Suites

Parsing Benchmarks (parsers)

Measures parsing performance for all supported dependency file formats at different scales (10, 50, 100 dependencies).

Benchmark	Description
cargo_toml/{N}	Parse Cargo.toml with N dependencies
package_json/{N}	Parse package.json with N dependencies
requirements_txt/{N}	Parse requirements.txt with N dependencies
go_mod/{N}	Parse go.mod with N dependencies
composer_json/{N}	Parse composer.json with N dependencies
csproj/{N}	Parse .csproj (NuGet) with N dependencies
pubspec_yaml/{N}	Parse pubspec.yaml (Dart) with N dependencies
gemfile/{N}	Parse Gemfile (Ruby) with N dependencies

Cache Benchmarks (cache)

Measures cache operations at different entry counts (100, 1000, 10000 for memory; 100, 1000 for SQLite).

Benchmark	Description
cache/memory/get_hit/{N}	Memory cache hit with N entries
cache/memory/get_miss/{N}	Memory cache miss with N entries
cache/memory/insert/{N}	Memory cache insert with N entries
cache/sqlite/get_hit/{N}	SQLite cache hit with N entries
cache/sqlite/get_miss/{N}	SQLite cache miss with N entries
cache/sqlite/insert/{N}	SQLite cache insert with N entries

Version Utils Benchmarks (version_utils)

Measures prerelease detection performance across all supported ecosystems.

Benchmark	Description
is_prerelease/rust	Rust prerelease detection (10 versions)
is_prerelease/npm	npm prerelease detection (10 versions)
is_prerelease/python	Python prerelease detection (10 versions)
is_prerelease/go	Go prerelease detection (10 versions)
is_prerelease/php	PHP prerelease detection (10 versions)
is_prerelease/dart	Dart prerelease detection (10 versions)
is_prerelease/nuget	NuGet prerelease detection (10 versions)

VersionInfo Benchmarks (version_info)

Measures operations on the VersionInfo struct.

Benchmark	Description
is_version_yanked_hit	Yanked check with hit (100 yanked versions)
is_version_yanked_miss	Yanked check with miss (100 yanked versions)
is_version_yanked_with_prefix	Yanked check with version prefix (^, ~)

Performance Targets

Based on typical usage patterns:

Operation	Target	Rationale
Parse 50-dep Cargo.toml	<5ms	Should be instant for user
Parse 100-dep package.json	<10ms	Larger files still fast
Memory cache hit	<1µs	HashMap lookup is O(1)
Memory cache miss	<1µs	HashMap lookup is O(1)
SQLite cache hit	<500µs	Connection pool + query
SQLite cache miss	<500µs	Connection pool + query
Prerelease detection (10 versions)	<1µs	Simple string operations

Viewing Results

After running benchmarks, HTML reports are generated at:

target/criterion/report/index.html

Reports include:

• Performance distribution plots
• Comparison with previous runs
• Statistical analysis
• Regression detection

CI Integration

Benchmarks can be integrated into CI workflows. See .github/workflows/benchmarks.yml for an example configuration that:

• Runs benchmarks on push/PR to main
• Stores results as artifacts
• Compares against baseline

References

• Criterion.rs documentation
• Benchmarking best practices