"An open source Devin getting 12.29% on 100% of the SWE Bench test set vs Devin's 13.84% on 25% of the test set!"
FreeOpen-source Devin for solving GitHub issues with GPT-4
FreeFree tier
LinksX
About "An open source Devin getting 12.29% on 100% of the SWE Bench test set vs Devin's 13.84% on 25% of the test set!"
SWE-agent is an open-source system developed by Princeton NLP for autonomously resolving issues in GitHub repositories. It achieves similar accuracy to Devin on the SWE-bench benchmark, scoring 12.29% on 100% of the test set compared to Devin’s 13.84% on 25%. The agent uses GPT-4 and a custom agent-computer interface to edit and run code, taking an average of 93 seconds per issue.
Key Features
Autonomously resolves issues in GitHub repositories
Uses GPT-4 with a custom agent-computer interface to edit and run code
Achieves 12.29% accuracy on 100% of SWE-bench test set
Average resolution time of 93 seconds
Open source and transparent (licensed by Princeton NLP)
Pros & Cons
Pros
- Free and open source with no usage restrictions
- Comparable accuracy to proprietary Devin agent
- Fast execution (average 93 seconds per issue)
- Designed for real-world GitHub repositories
- Uses state-of-the-art language model (GPT-4)
Cons
- Currently focused only on GitHub repository issues, not fully generalized
- Accuracy still lower than human developers (12.29% on SWE-bench)
- Requires GPT-4 API access, which may incur costs for users
- May struggle with complex, multi-step issues requiring deep context
Best For
Automated bug fixing in open-source projectsImplementing features from GitHub issuesCode debugging and refactoringSoftware maintenance and task automation
FAQ
What is SWE-agent?
SWE-agent is an open-source system that autonomously solves issues in GitHub repositories. It uses GPT-4 and a custom interface to edit and run code.
How does SWE-agent compare to Devin?
On the SWE-bench benchmark, SWE-agent achieved 12.29% accuracy on 100% of the test set, while Devin scored 13.84% on 25% of the test set. SWE-agent is open source, whereas Devin is proprietary.
How long does SWE-agent take to solve an issue?
On average, SWE-agent resolves an issue in 93 seconds.
Is SWE-agent free to use?
Yes, SWE-agent is open source and free. However, users need their own GPT-4 API access which may have costs.
Who developed SWE-agent?
SWE-agent was developed by John Yang and team at Princeton NLP.