Amazon Web Services now offers G7e instances powered by NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs on Amazon SageMaker AI. Developers and companies need flexible, affordable, and strong hardware as generative AI use increases. Users can set up nodes with 1, 2, 4, or 8 RTX PRO 6000 GPUs. Each GPU has 96 GB of GDDR7 memory.
A single-node G7e.2xlarge instance runs open source foundation models such as GPT-OSS-120B, Nemotron-3-Super-120B-A12B in NVFP4 variant, and Qwen3.5-35B-A3B. This setup gives organizations a cheap yet fast choice for inference tasks. It helps cut costs without losing speed.
Main Features of G7e Instances
G7e instances double the GPU memory of G6e instances. This allows FP16 large language models up to 35B parameters on one GPU node with G7e.2xlarge, 150B parameters on four GPUs with G7e.24xlarge, and 300B parameters on eight GPUs with G7e.48xlarge. Networking reaches 1600 Gbps. The largest size, G7e.48xlarge, has 768 GB GPU memory.
Amazon EC2 G7e instances boost GPU-based inference in the cloud. They achieve up to 2.3x inference speed over G6e instances from the prior generation. Each G7e GPU delivers 1,597 GB/s bandwidth. This doubles G6e per-GPU memory and quadruples G5 memory. Networking hits 1,600 Gbps with EFA on the biggest G7e size. That marks a 4x gain over G6e and 16x over G5. It enables low-delay multi-node inference and fine-tuning that earlier G-series instances could not handle well.
Here is a table comparing the 8-GPU tier across generations:
| Spec | G5 (g5.48xlarge) | G6e (g6e.48xlarge) | G7e (g7e.48xlarge) | |, , , , , , , , , , , -|, , , , , , , , , , , |, , , , , , , , , , , |, , , , , , , , , , , , , , , | | GPU | 8x NVIDIA A10G | 8x NVIDIA L40S | 8x NVIDIA RTX PRO 6000 Blackwell | | GPU Memory per GPU | 24 GB GDDR6 | 48 GB GDDR6 | 96 GB GDDR7 | | Total GPU Memory | 192 GB | 384 GB | 768 GB | | GPU Memory Bandwidth | 600 GB/s per GPU | 864 GB/s per GPU | 1,597 GB/s per GPU | | vCPUs | 192 | 192 | 192 | | System Memory | 768 GiB | 1,536 GiB | 2,048 GiB | | Network Bandwidth | 100 Gbps | 400 Gbps | 1,600 Gbps (EFA) | | Local NVMe Storage | 7.6 TB | 7.6 TB | 15.2 TB | | Inference vs. G6e | Baseline | ~1x | Up to 2.3x |
With 768 GB total GPU memory on one instance, G7e handles models that needed multiple nodes on G5 or G6e. This cuts complexity and node-to-node delays. Support for FP4 precision via fifth-generation Tensor Cores and NVIDIA GPUDirect RDMA over EFAv4 makes G7e ideal for LLMs, multimodal AI, and agentic inference on AWS.
Suitable Use Cases
G7e fits many generative AI tasks thanks to its memory, bandwidth, and network features:
Chatbots and conversational AI benefit from low time-to-first-token and high throughput for responsive interactions under heavy loads.
Agentic and tool-calling workflows gain from 4x better CPU-to-GPU bandwidth, aiding RAG pipelines and fast context from retrieval stores.
Text generation, summarization, and long-context inference use the 96 GB per-GPU memory for large KV caches, avoiding cuts in long inputs.
Image generation and vision models avoid memory errors on big multimodal models due to doubled memory.
Physical AI and scientific computing leverage Blackwell compute, FP4 support, DLSS 4.0, and 4th-gen RT cores for digital twins, 3D simulation, and physical AI inference.
Getting Started with Deployment
To use G7e on SageMaker AI, prepare these items:
An AWS account for all resources.
An IAM role for SageMaker AI access. Details appear in Identity and Access Management for Amazon SageMaker AI.
Access to SageMaker Studio, a SageMaker notebook, or an IDE like PyCharm or Visual Studio Code. SageMaker Studio works best for easy setup and inference.
Quota for at least one ml.g7e.2xlarge instance for SageMaker AI endpoints. Request increases via Service Quotas console.
Clone the repository and run the sample notebook for deployment.
Stay ahead of the AI curve
The most important updates, news, and content — delivered weekly.
No spam. Unsubscribe anytime.
Benchmark Results
Benchmarks used Qwen3-32B in BF16 on G6e and G7e with 1,000 input tokens and 560 output tokens per request, like summarization tasks. They used native vLLM container with prefix caching. The suite deploys the model, tests concurrency from 1 to 32, and analyzes results.
G6e baseline on ml.g6e.12xlarge (4x L40S, $13.12/hr):
| C | Success | p50 (s) | p99 (s) | tok/s | RPS | Agg tok/s | $/M tokens | |, , |, , , , -|, , , , -|, , , , -|, , , -|, , , |, , , , , -|, , , , , , | | 1 | 100% | 16.1 | 16.3 | 37.1 | 0.07 | 37 | $38.09 | | 8 | 100% | 19.8 | 20.2 | 30.3 | 0.42 | 242 | $5.85 | | 16 | 100% | 23.1 | 23.5 | 26.0 | 0.73 | 416 | $3.41 | | 32 | 100% | 26.0 | 29.2 | 21.5 | 1.21 | 686 | $2.06 |
G7e on ml.g7e.2xlarge (1x RTX PRO 6000, $4.20/hr):
| C | Success | p50 (s) | p99 (s) | tok/s | RPS | Agg tok/s | $/M tokens | |, , |, , , , -|, , , , -|, , , , -|, , , -|, , , |, , , , , -|, , , , , , | | 1 | 100% | 27.2 | 27.5 | 22.0 | 0.04 | 22 | $21.32 | | 8 | 100% | 28.7 | 28.9 | 20.9 | 0.28 | 167 | $2.81 | | 16 | 100% | 30.3 | 30.6 | 19.9 | 0.53 | 318 | $1.48 | | 32 | 100% | 33.2 | 33.3 | 18.5 | 0.99 | 592 | $0.79 |
At C=32, G7e costs $0.79 per million output tokens, 2.6x less than G6e's $2.06. Lower hourly rate and steady throughput drive this. Single-GPU setup avoids synchronization, all-reduce, KV cache issues, and NVLink limits. Latency rises 22% from C=1 to 32 on G7e, versus 62% on G6e.
G7e with EAGLE Speculative Decoding
Pairing G7e with EAGLE boosts results further. EAGLE predicts multiple tokens from hidden states and verifies them fast, matching output quality. See Amazon SageMaker AI introduces EAGLE based adaptive speculative decoding to accelerate generative AI inference for details.
Benchmarks on Qwen3-32B BF16 with EAGLE3 (num_speculative_tokens=4, ~1.56 GB speculator) show G7e + EAGLE3 gives 2.4x throughput and 75% cost cut over G6e baseline. It costs $0.41 per million tokens, 4x cheaper than G6e + EAGLE3 at $1.72 with better throughput.
SageMaker AI EAGLE toolkit trains custom speculators on user data for better rates.
Pricing Details
SageMaker AI bills G7e at standard inference rates by instance and time. No extra token or request fees.
EAGLE jobs use training instance rates; artifacts in S3 at storage rates. No added inference cost post-deployment.
On-demand prices in US East (N. Virginia):
| Instance | GPUs | GPU Memory | Typical Use Case | |, , , , , , , , , |, , , |, , , , , , |, , , , , , , , , , , , , , , , , -| | ml.g5.2xlarge | 1 | 24 GB | Small LLMs (≤7B FP16); dev/test | | ml.g5.48xlarge | 8 | 192 GB | Large multi-GPU on G5 | | ml.g6e.2xlarge | 1 | 48 GB | Mid-size LLMs (≤14B FP16) | | ml.g6e.12xlarge | 2? | 96 GB | Large LLMs (≤36B FP16) baseline | | ml.g6e.48xlarge | 8 | 384 GB | Very large LLMs (≤90B FP16) | | ml.g7e.2xlarge | 1 | 96 GB | Large LLMs (≤70B FP8) single GPU| | ml.g7e.24xlarge | 4 | 384 GB | Very large LLMs; high throughput | | ml.g7e.48xlarge | 8 | 768 GB | Max throughput; largest models |
SageMaker Savings Plans cut costs up to 64% for steady use.
Cleanup Steps
Delete SageMaker endpoints after tests via console or Python SDK in Amazon SageMaker AI Developer Guide. Remove EAGLE S3 artifacts to stop storage fees.
G7e on SageMaker AI advances cheap generative AI inference. Blackwell brings 2x GPU memory, 1.85x bandwidth, and up to 2.3x speed over G6e. Single-GPU runs former multi-GPU tasks. EAGLE adds gains via better bandwidth and memory for draft heads.
From G5 to G6e to G7e plus EAGLE forms a path of hardware-software advances as models and traffic grow.