LLaVA 1.5: An open source alternative to GPT-4 Vision
The field of generative artificial intelligence is booming with the emergence of large multimodal models (LMMs) such as OpenAI’s GPT-4 Vision. These models revolutionize our interaction with AI systems by integrating text and images.
However, the closed and commercial nature of some of these technologies may hinder their universal adoption. This is where the open source community comes in, propelling the LLaVA 1.5 model as a promising alternative to GPT-4 Vision.
The mechanics of LMM
LMMs operate using a multi-layer architecture. They combine a pre-trained model to encode visual elements, a large language model (LLM) to decipher and respond to user instructions, and a multimodal connector to connect vision and language.
Their training takes place in two stages: a first phase of vision-language alignment, followed by fine adjustment to respond to visual requests. This process, although efficient, is often computationally intensive and requires a rich and precise database.
The advantages of LLaVA 1.5
LLaVA 1.5 relies on the CLIP model for visual encoding and Vicuna for language. Unlike the original model, LLaVA, which used the text versions of ChatGPT and GPT-4 for visual adjustment, LLaVA 1.5 goes further by connecting the language model and visual encoder via a multi-layer perceptron (MLP). This enriches its training database with visual questions and answers. This update, which includes approximately 600,000 examples, allowed LLaVA 1.5 to outperform other open source LMMs on 11 of 12 multimodal benchmarks.
The future of open source LMMs
The online demo of LLaVA 1.5, accessible to everyone, shows promising results even with a limited budget. However, one caveat remains: the use of data generated by ChatGPT limits its use to non-commercial purposes.
Despite this limitation, LLaVA 1.5 opens a perspective on the future of open source LMMs. Its cost-effectiveness, ability to generate scalable training data, and efficiency in adjusting visual instructions make it a prelude to future innovations.
LLaVA 1.5 is just the first step in a melody that will resonate with the progress of the open source community. By anticipating more efficient and accessible models, we can envision a future where generative AI technology is within everyone’s reach, revealing the limitless potential of artificial intelligence.