Neura Market
TaskMatrix logo

TaskMatrix

Free

Combines ChatGPT with Visual Models for enhanced image-chat interactions.

FreeFree tier
Inputs: text, imageOutputs: text, image
Type
Open Source
Visit Website GitHub (40 stars)

About TaskMatrix

TaskMatrix is an open-source tool that connects ChatGPT (or other LLMs) with a series of visual foundation models, enabling interactive image-based conversations. It allows users to send and receive images during chat, and supports a wide range of visual tasks including image editing, object detection, segmentation, inpainting, captioning, and visual question answering. The system leverages GroundingDINO for text-guided bounding box detection, segment-anything for mask generation, and Stable Diffusion for inpainting. It also introduces a template system for defining reusable execution flows that combine multiple foundation models for complex tasks, such as InfinityOutPainting for extending images to arbitrary sizes. TaskMatrix is community-driven and supports both English and Chinese.

Key Features

Connects ChatGPT with multiple visual foundation models
Enables sending and receiving images during chat
Supports GroundingDINO, segment-anything, and Stable Diffusion inpainting
Template system for defining complex multi-model workflows
InfinityOutPainting for extending images to any size without additional training
Supports Chinese language
Leverages both general LLM knowledge and domain-specific visual models
Community-driven with easy contribution and extension

Pros & Cons

Pros
  • Open source and free to use
  • Leverages state-of-the-art visual foundation models
  • Template system simplifies complex multi-model workflows
  • Supports both English and Chinese
  • Actively maintained with community contributions
  • Seamlessly integrates general knowledge from LLMs with specialized visual knowledge
Cons
  • Requires OpenAI API key for ChatGPT, incurring usage costs
  • Requires substantial GPU resources for local deployment of foundation models
  • Setup process may be complex for non-technical users
  • Dependent on external models and services (e.g., OpenAI, Hugging Face)

Best For

Interactive image editing with mask-based inpaintingObject detection and segmentation in imagesImage captioning and visual question answeringExtending images to larger sizes (outpainting)Generating images via text prompts within a chat interfaceCombined visual reasoning and conversation using multiple foundation models

FAQ

What is TaskMatrix?
TaskMatrix is an open-source tool that connects ChatGPT with a series of visual foundation models, allowing users to send and receive images during chatting. It supports tasks like image editing, detection, segmentation, captioning, and visual question answering.
What foundation models does TaskMatrix support?
TaskMatrix supports Text2Box (GroundingDINO), Segmenting (segment-anything), Inpainting, ImageCaptioning, VisualQuestionAnswering, and the InfinityOutPainting template. Users can load specific models using command-line arguments.
How do I install TaskMatrix?
Clone the repository, create a conda environment with Python 3.8, install dependencies from requirements.txt, then install GroundingDINO and segment-anything from their GitHub repos. Set your OpenAI API key as an environment variable and run visual_chatgpt.py with desired models.
Can I use TaskMatrix without a GPU?
The foundation models (e.g., GroundingDINO, segment-anything, Stable Diffusion) typically require a GPU for reasonable performance. CPU-only usage may be very slow or infeasible. The ChatGPT component only requires an API key.