DeepSeek-V3
FreeFirst open-sourced GPT-4o level model.
About DeepSeek-V3
DeepSeek-V3 is a powerful Mixture-of-Experts (MoE) language model with 671 billion total parameters (37 billion activated per token). It adopts Multi-head Latent Attention (MLA) and DeepSeekMoE architectures for efficient inference and cost-effective training. The model pioneers an auxiliary-loss-free load balancing strategy and a multi-token prediction training objective. Pre-trained on 14.8 trillion tokens using a novel FP8 mixed-precision training framework, it requires only 2.788 million H800 GPU hours for full training, with no irrecoverable loss spikes. Post-training incorporates knowledge distillation from DeepSeek-R1 to enhance reasoning. DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models like GPT-4o.
Key Features
Pros & Cons
- Open-source model with performance competitive to leading closed-source models
- Extremely cost-efficient training (2.788M H800 GPU hours)
- Innovative auxiliary-loss-free load balancing reduces performance degradation
- Multi-token prediction improves model performance and enables speculative decoding
- Stable training without rollbacks or irrecoverable loss spikes
- Supports FP8 mixed-precision training for further cost reduction
- Reasoning capabilities enhanced via distillation from DeepSeek-R1
- Large model size (671B parameters) requires substantial computational resources for inference
- Requires specialized hardware (e.g., H800 GPUs) for optimal performance
- Not yet deployed as a widely accessible API; primarily available for self-hosting