AI technology timeline

Low-rank joint compression reduces the KV-cache requirements of attention.

Group-relative rewards estimate advantages without a separate value model.

A structured generation language and runtime reuse prefix computation and cached state.

Selective state-space models offer an alternative approach to sequence modeling.

Paged KV-cache management improves memory utilization in language-model serving.

Direct optimization on preference data simplifies language-model alignment.

Low-bit quantization and low-rank adaptation reduce memory requirements for model fine-tuning.

torch.compile adds compilation-based acceleration to the established PyTorch workflow.

A C/C++ implementation runs LLaMA locally with quantized inference.

A small model drafts tokens and a target model verifies them, accelerating generation while preserving the target distribution.

Vector fields along conditional probability paths train continuous-flow generative models.

Reasoning steps and tool actions are interleaved so language models can adapt their plans to environment feedback.

Reduced data movement between GPU memory and on-chip storage accelerates exact attention.

A fixed training-compute budget motivates a different balance of model size and data.

Intermediate reasoning steps in examples improve multi-step language-model tasks.

Diffusion in a compressed latent space reduces the cost of high-resolution image generation.

A Python-like language and compiler simplify the development of optimized GPU kernels.

Frozen base weights and trainable low-rank matrices reduce the cost of model adaptation.

Rotations encode position within attention computation.

Image–text contrastive pretraining enables visual concepts to be recognized from natural-language descriptions.

Images become sequences of patches for Transformer-based visual learning.

Iterative denoising provides a training approach for diffusion-based image generation.

Document retrieval supplies external knowledge to a text-generation model.

Partitioning optimizer state and other training data reduces distributed-training memory use.

A systematic study relates language-model performance to model size, data, and compute.

A unified API makes Transformer architectures and pretrained models accessible for research and applications.

Intra-layer model parallelism distributes large Transformer training across GPUs.

Human comparisons train a reward signal that guides reinforcement learning.

Google researchers introduce the Transformer, an attention-based sequence model, in Attention Is All You Need.

Sparse routing selects a small subset of experts to increase capacity while limiting computation.

Dynamic computation graphs and a Python workflow support deep-learning research.

Normalization statistics are computed within each example rather than across a batch.

Residual connections make very deep networks easier to optimize and train.

Google releases its machine-learning framework as open-source software.

Byte-pair encoding is applied to subword segmentation to address unknown words.

A teacher model’s output distribution trains a smaller student model.

First- and second-moment gradient estimates adapt optimization step sizes.

An encoder and decoder map variable-length input sequences to variable-length outputs.

A GPU deep-learning primitive library provides optimized operations for frameworks.

A translation model learns to attend to different input positions as it generates each word.

Adversarial training between a generator and a discriminator learns a data distribution.

Variational inference and reparameterization learn a latent representation that can be sampled.

Efficient word-vector learning represents relationships in a continuous space.

A deep convolutional network trained on GPUs advances ImageNet image classification.

A public beta of the CUDA toolkit and SDK brings C-based programming to GPU computing.