ChatARKit: Using ChatGPT to Create AR Experiences with Natural Language
FreeCreate AR experiences with natural language using ChatGPT
FreeFree tier
Inputs: text
About ChatARKit: Using ChatGPT to Create AR Experiences with Natural Language
ChatARKit is an open-source iOS experiment that combines ChatGPT, Whisper speech-to-text, and ARKit to let users create augmented reality experiences using natural language prompts. By speaking commands, users can place 3D objects on detected planes and manipulate their position, scale, and orientation. The app uses ChatGPT’s code generation abilities to call custom ARKit APIs, and it fetches 3D models from Sketchfab. It is fully self-contained on iOS, requiring only an OpenAI API key and a Sketchfab API token.
Key Features
Generates AR code from natural language prompts using ChatGPT
Speech-to-text input via whisper.cpp for hands-free interaction
Places and manipulates 3D objects (position, scale, rotation) on detected AR planes
Fetches 3D assets from Sketchfab’s library
Self-contained iOS app with direct OpenAI API integration
Sample prompts included (e.g., 'Place a cube on the nearest plane and make it spin')
Pros & Cons
Pros
- Open source and free to use (requires only API keys)
- Integrates multiple AI models (Whisper, ChatGPT) into one AR app
- Low API cost – developer reported $0.08 for 52 queries
- Supports a variety of 3D objects from Sketchfab
- Easy to extend with additional interactions
Cons
- Requires a paid OpenAI API key and a free Sketchfab token
- ChatGPT’s code output is variable – results may be inconsistent
- May generate invalid JavaScript (undefined functions, incorrect identifiers)
- Limited to iOS (ARKit) platform
- Object description to code conversion can break asset fetching (e.g., 'school bus' becomes 'schoolBus')
Best For
Prototyping AR scenes without manual codingEducational demonstrations of AI-driven code generationExploring ChatGPT’s ability to use custom APIs in ARAdding interactive 3D objects to real-world environments via voice commands
FAQ
What is ChatARKit?
It is an open-source iOS app that uses ChatGPT to generate ARKit code from natural language prompts, allowing users to place and manipulate 3D objects in augmented reality.
Does it require paid subscriptions?
The app itself is free and open source, but it requires a paid OpenAI API key for ChatGPT access and a free Sketchfab account for 3D assets.
How do I set it up?
Download the Whisper model weights, add your OpenAI API key in APIKeys.swift, and obtain a Sketchfab API token. Then build and run on an iOS device.
What speech recognition does it use?
It uses whisper.cpp, a fast C++ implementation of OpenAI's Whisper speech-to-text model.
Can I use custom 3D models?
The app fetches models from Sketchfab using their API. You can also use the native 'cube' geometry.