Unified-IO2 Autoregressive Multimodal Model with Vision, Language, Audio, and Action [Paper Reading] 

PROJECT SITE: unified-io-2.a...
Join the Agora discord community: / discord
Follow me on GitHub: github.com/kye...
Some notes I wrote down:
Encoder to Decoder Transformer: This refers to a transformer-based model architecture where an encoder module is used to encode input data into a hidden representation, and a decoder module is used to generate outputs based on the encoded representation.
Dynamic Packer: Dynamic packing involves grouping tokens from multiple sequences together to form a single packed sequence. This can help optimize memory usage and improve efficiency in processing sequences of varying lengths.
Rotary Embeddings: Rotary embeddings are a technique used to introduce rotational symmetry into the self-attention mechanism of transformers. They help capture positional information and improve the model's ability to process sequential data.
QK Norm: QK Norm refers to applying layer normalization to the queries and keys before performing the dot product operation in the self-attention mechanism. This normalization helps stabilize and improve the attention calculation.
Audio + Audio History to linear or perceiver: This suggests that the audio inputs and their temporal context (audio history) can be processed either using linear transformation methods or the perceiver model. Linear transformation refers to applying linear operations such as linear layers or convolutions to the audio data, while the perceiver model is a class of models that can handle both sequential and non-sequential data.
Transformer Outputs: The transformer model outputs discrete tokens, which can be decoded into text, images, or audio clips. This means that the model can generate output data in various modalities based on the input.
Images: The process for image data involves converting the image into tokens using the VQGAN model, which is a generative model for images. The tokens are then processed by a transformer model, where low and high level visual information is captured by concatenating features from specific layers. The generated features go into another layer for further processing.
Audio: The pre-trained AST (Audio Spectrogram Transformer) is used to create audio embeddings by concatenating features from certain layers. These embeddings are then processed using a vit vqgan model, similar to the image processing approach, to convert the audio into discrete tokens for further transformation.
Vit VQGAN: A vit vqgan model refers to a Vision Transformer (vit) combined with a VQGAN model. The Vision Transformer is used to process image or audio data, and the VQGAN model is used to convert the processed data into discrete tokens.
Training Stability: To enhance the stability of training, various techniques are employed, including 2D rotary embeddings, ROPe (Rotary Position Embeddings), and QK Norm. These techniques help stabilize the learning process and ensure better model performance.
Multi-modal Mixture of Denoisers: This refers to a training approach where different corruption methods, such as span corruption and language modeling, are used to train the model to reconstruct corrupted images or audio. The model is also trained to generate the target modality conditioned on other input modalities.
Autoregressive Dynamic Masking: This technique involves masking tokens in the decoder to prevent information leakage during prediction. By masking the token being predicted, the model can maintain the autoregressive nature and ensure causal prediction.
Packing and Attention Masking: To handle multiple examples with different modalities, tokens from different examples are packed into a single sequence. This is done to prevent cross-attending between samples during the self-attention calculation in the transformer.
Optimizer: The Adafactor optimizer is used with a linear warm-up decay for the first 5,000 steps. Additionally, global gradient norm clipping is applied to manage the gradient updates during training. These techniques help optimize the training process and improve convergence.
Limitations and Weaknesses: The notes mention several limitations and weaknesses, such as the need for bigger encoders, quantized layers for longer audio sequences, larger batch sizes, and better optimizers for improved performance. Limited datasets with instructions and challenges with tokenizers and dataset processing are also mentioned as areas of improvement.