We introduce Vera 1.6, a multimodal language model developed by Cortex Research and purpose-built for agentic applications. Vera 1.6 is a hybrid model in which efficient linear attention layers alternate with periodic full attention layers within a Sparse Mixture-of-Experts (SMoE) feed-forward structure, yielding favourable inference efficiency without sacrificing model capacity. The model is trained on a 150B-token synthetic dataset assembled with the NVIDIA NeMo Data Designer framework and further aligned via Reinforcement Learning. Vera 1.6 demonstrates strong performance across instruction following, graduate-level reasoning, multilingual understanding, agentic tool use, and multimodal comprehension. A 1M-token context window, native vision and video processing, and support for more than 201 languages make it well suited as the backbone for production-grade agentic AI systems.
Introduction
Recent advances in large language models have demonstrated remarkable capabilities across reasoning, code generation, and multimodal understanding. Deploying such models in production agentic systems — where a model must autonomously execute multi-step tasks, interact with external tools, navigate web interfaces, and sustain coherence across extremely long contexts — raises architectural and training challenges that general-purpose designs are not optimally equipped to address.
Vera 1.6 is Cortex Research's second-generation multimodal model in the Vera series, designed from the outset for agentic workloads. The model is a hybrid in which linear recurrent attention layers alternate with periodic full quadratic attention layers, coupled throughout with Sparse Mixture-of-Experts (SMoE) feed-forward blocks. Linear attention operates in O(L) time with respect to sequence length L, making long-context inference tractable, while interleaved full attention layers preserve global context integration. SMoE routing further decouples model capacity from per-token compute cost, as only a small subset of experts participates in each forward pass.
Model Architecture
Vera 1.6 is a Causal Language Model augmented with a Vision Encoder, forming a unified Vision-Language backbone. The core language model adopts a hierarchical hybrid design comprising 40 layers organised into 10 macro-blocks. Each macro-block contains four sub-blocks that interleave efficient linear and full quadratic attention with sparse feed-forward routing, in a 3:1 ratio of linear-to-quadratic attention — reducing per-token FLOPs for the majority of layers while periodic full attention layers maintain global context integration across the 1M-token window.
| Sub-block | Repetitions |
|---|---|
| Linear Attention → MoE | 3× |
| Full Attention → MoE | 1× |
Linear Attention
The primary attention mechanism is a linear recurrent formulation that maintains a fixed-size recurrent state rather than an expanding KV cache, achieving O(L) inference complexity with respect to sequence length L, making long-context inference tractable. An asymmetric head configuration — more value heads than query/key heads — increases representational capacity in the value projection while keeping recurrent state updates computationally lean.
| Parameter | Value |
|---|---|
| Linear Attn. Heads (V) | 32 |
| Linear Attn. Heads (QK) | 16 |
| Head Dimension | 128 |
Full Attention
One full quadratic attention sub-block per macro-block employs Grouped-Query Attention (GQA) with a 16:2 query-to-KV head ratio, reducing KV-cache memory by 8× relative to standard multi-head attention while retaining model quality. Rotary Position Embeddings (RoPE) are applied with a compressed head dimension of 64, supporting long-range dependency modelling across extended sequences. This configuration enables single-node deployment practical for enterprise customers.
| Parameter | Value |
|---|---|
| Query Heads | 16 |
| Key / Value Heads | 2 (GQA) |
| Head Dimension | 256 |
| RoPE Dimension | 64 |
Sparse Mixture-of-Experts
Every attention sub-block is followed by a Sparse Mixture-of-Experts (SMoE) feed-forward block. A learned router selects a small number of task-specialised experts per token while one shared expert remains always active, providing a stable general-purpose computation path. The capacity multiplier — the ratio of total to simultaneously active experts — reaches approximately 28×, substantially expanding effective model capacity without a proportional increase in per-token compute cost.
| Parameter | Value |
|---|---|
| Total Experts | 256 |
| Activated / Token | 8 Routed + 1 Shared |
| Expert Interm. Dim. | 512 |
| Routing Strategy | Top-K + shared expert |
Vision Encoder
Vera 1.6 incorporates a Vision Encoder that projects image and video feature maps into the language model embedding space, forming a unified Vision-Language model without separate specialist modules. The encoder handles still images, multi-page documents, and video sequences, with representations aligned to the language backbone through a learned projection layer.
| Property | Value |
|---|---|
| Model Type | Causal LM + Vision Encoder |
| Architecture | Hybrid: Linear Attention + SMoE |
| Hidden Dim. | 2,048 |
| Num. Layers | 40 |
| Token Vocab | 248,320 (padded) |
| Context Window | 1,000,000 tokens |
| Languages | 150+ |
| Modalities | Text, Image |
| Training | Causal LM + RL (agentic) |
| Dataset | 150B token synthetic corpus |
| Data Tool | NVIDIA NeMo Data Designer |
| Hardware | DGX B200 (8× B200, 1,440 GB) |
Training Methodology
Base Model
Vera 1.6 is initialised from a base model with broad multilingual competence, strong mathematical reasoning, and code comprehension. This starting point provides a well-conditioned foundation for the domain-specific alignment stages that follow.
Agentic Supervised Fine-tuning
The supervised fine-tuning stage draws on a 150B-token synthetic dataset assembled using the NVIDIA NeMo Data Designer framework, constructed to span the full breadth of skills required for agentic deployment:
- Multi-step task planning and execution
- API and tool invocation with complex function schemas
- Web browsing and information retrieval in agentic settings
- Terminal and shell command execution sequences
- Long-context document processing and synthesis
- Multimodal instruction following with interleaved vision inputs
- Cross-lingual generalisation across 201+ languages
Reinforcement Learning Alignment
Following SFT, a Reinforcement Learning stage aligns model behaviour with agentic task objectives. Reward signals are derived from task success metrics across code execution outcomes, tool-call accuracy, instruction adherence, and multimodal comprehension quality. This stage yields measurable gains in self-correction, long-horizon planning, and reliable external tool invocation within agentic pipelines.
Training Infrastructure
All training was conducted on a single NVIDIA DGX B200 node.
| Component | Specification |
|---|---|
| GPU | 8× NVIDIA B200 (Blackwell) |
| GPU Memory | 1,440 GB total |
| CPU | 2× Intel Xeon Platinum 8570 |
| System Memory | 2 TB DDR5 |
| Storage | 8× 3.84 TB U.2 NVMe |
| OS | DGX OS |
| Interconnect | NVLink (GPU-to-GPU) |
Benchmark Evaluation
Vera 1.6 was evaluated across twelve diverse benchmarks spanning instruction following, scientific reasoning, mathematics, multilingual understanding, agentic tool use, and multimodal comprehension.
| Benchmark | Score | Category |
|---|---|---|
| HMMT Feb 2025 | 92.0% | Mathematics |
| OmniDocBench v1.5 | 89.3% | Document Understanding |
| Video-MME | 87.3% | Video Reasoning |
| MMMLU | 86.2% | Multilingual |
| GPQA Diamond | 85.9% | Graduate Science |
| IFBench | 76.5% | Instruction Following |
| MMMU-Pro | 75.1% | Visual Reasoning |
| SWE-bench Verified | 72.4% | Agentic Coding |
| BFCL V4 | 69.1% | Tool Use |
| ERQA | 64.7% | Embodied Reasoning |
| BrowseComp | 61.0% | Agentic Search |
| Terminal-Bench 2 | 41.6% | Terminal Coding |
Vera 1.6 achieves particularly strong performance on mathematical reasoning (HMMT Feb 2025: 92.0%), document understanding (OmniDocBench v1.5: 89.3%), and video comprehension (Video-MME: 87.3%). A MMMLU score of 86.2% confirms broad multilingual capability across supported languages. Graduate-level scientific reasoning at 85.9% (GPQA Diamond) reflects the contribution of the RL alignment stage to complex multi-step inference.
Agentic benchmarks exhibit expected task-dependent variance. SWE-bench Verified (72.4%) and BFCL V4 (69.1%) reflect strong capability in code-based and tool-use agentic tasks. Terminal-Bench 2 (41.6%) and BrowseComp (61.0%) identify active development areas in low-level terminal execution and autonomous web navigation; both are prioritised in the subsequent release cycle.
Discussion
Architectural Trade-offs
The hybrid model design — linear attention alternating with full attention within a sparse MoE feed-forward structure — represents a deliberate engineering trade-off targeting production agentic deployment. The 3:1 linear-to-full ratio substantially reduces per-token FLOPs across the majority of layers. GQA (16Q/2KV) in the full attention sub-blocks cuts KV-cache memory by 8× relative to standard multi-head attention, enabling single-node deployment at scales practical for enterprise inference. Within the SMoE layers, the shared expert guarantees a consistent base-level computation path for all inputs, while the learned router assigns specialised experts per token — providing fine-grained, input-conditioned computation without proportionally increasing inference cost.
Limitations
Terminal-Bench 2 performance (41.6%) reveals that low-level shell execution remains challenging, likely owing to the compounding sensitivity of sequential commands to earlier errors and relatively limited coverage of terminal-execution trajectories in the training distribution. BrowseComp (61.0%) similarly reflects the difficulty of sustained autonomous web navigation over long horizons. Both areas are being addressed through targeted synthetic data generation and refined RL reward shaping.
Future Work
Planned work includes: expanded terminal and shell execution data to improve robustness on sequential command tasks; broader tool-use coverage across novel API schemas; enhanced video understanding for extended and streaming inputs; additional modalities beyond image and video; and further RL alignment targeting complex multi-agent coordination and collaborative agentic scenarios.
Conclusion
We have presented Vera 1.6, a multimodal hybrid model combining linear and full attention within a Sparse Mixture-of-Experts feed-forward structure, purpose-built for agentic applications. The model achieves competitive results across mathematical reasoning, document understanding, multilingual knowledge, and graduate-level scientific question answering, while maintaining efficient per-token inference through sparse expert routing.
A 1M-token context window and support for more than 201 languages make Vera 1.6 a capable backbone for production-grade agentic AI systems. This work reflects Cortex Research's commitment to architectures that balance frontier task performance with practical deployment efficiency.
