Curated on
September 29, 2023
LLMs (Large Language Models), mostly recognized for their capabilities in predicting words, are now viewed from a new perspective as strong data compressors, according to a recent research paper by Google's DeepMind. The authors showed that with some modifications, LLMs can compress data as effectively or even better than popular compression algorithms currently in use. This fresh viewpoint can steer the development and evaluation of LLMs in a new direction. LLMs learn to transform inputs such as images or text into a 'latent space' that encapsulates key features of the data. This 'latent space' typically has fewer dimensions than the input space, which allows LLMs to compress the data into a smaller size.
In their study, DeepMind researchers tested LLMs' data compression capabilities using vanilla transformers and Chinchilla models on text, image, and audio data. Despite originally being designed for text compression, these models accomplished impressive compression rates on image and audio data, even surpassing domain-specific compression algorithms like PNG and FLAC. However, despite these promising results, LLMs are currently not practical tools for data compression compared to existing high-speed models like gzip.
One profound realisation from the research was that bigger LLMs are not necessarily better. Although larger models achieved superior compression rates on larger datasets, their effectiveness decreased on smaller datasets. With these findings, compression can serve as an indicator for evaluating the learning capabilities of these models, potentially impacting the future assessment of LLMs. Yet despite these impressive results, conventional compressors like gzip are not becoming obsolete anytime soon due to their superior speed and size advantages over LLMs.
