Multi-Label Classification<\/div><\/td>
F1-Score<\/div><\/td> 0.5906<\/strong><\/div><\/td> 0.5777<\/div><\/td>\n<\/tr>\n \n\tBinary Classification<\/div><\/td> F1-Score<\/div><\/td> 0.9239<\/div><\/td> 0.9252<\/strong><\/div><\/td>\n<\/tr>\n\n\tQuestion Answering<\/div><\/td> Exact Match<\/div><\/td> 0.5860<\/div><\/td> 0.6341<\/strong><\/div><\/td>\n<\/tr>\n\n\tSummarization<\/div><\/td> ROUGE-L<\/div><\/td> 0.3494<\/strong><\/div><\/td>0.3464<\/div><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n Overview of the evaluation results regarding the most important metric for each task type. For the Multi-Label Classification and Summarization, the best combination was reached through the pure Voting strategy; for the Binary Classification and Question Answering it was reached through the pure Weighting strategy.<\/div><\/span>\n\n The reasons for the decrease in performance in some tasks can be diverse. A possible explanation for this phenomenon could be that, due to increased noise and complexity, as well as potential overfitting, which together contribute to inconsistent outcomes, wrong final answers are given. Another likely reason stems from the models having different foci across the combination, thus automatically yielding worse results and hindering better models from performing at their maximum. Furthermore, consensus strategies like Voting can lead to worse performance if LLMs that give the same wrong prediction prevail. To further boost the performance, especially in tasks where the combination did not outperform the best single model, there are various promising steps to take.<\/p>\n Possible approaches could include covering an even broader spectrum of consensus-building techniques to select optimal responses, utilizing diverse model constellations to enhance response variance and creativity, and experimenting with various shot configurations and prompts to optimize task handling and evaluation outcome.<\/p>\n <\/span>Are Less Complex Strategies Beneficial?<\/span><\/h3>\nOur evaluation reveals that simpler consensus strategies outperform the Multi-Agent Debate approach. Although the evaluation in this context focuses solely on Classification datasets, methods like Voting and Weighting achieve better results. Additionally, they are less computationally intensive and reach consensus more quickly.<\/p>\n\n \n\n\n\tTask Type<\/strong><\/div><\/th>Metric<\/strong><\/div><\/th>Best Strategy<\/strong><\/div><\/th>MAD Approach<\/strong><\/div><\/th>\n<\/tr>\n<\/thead>\n\n\n\tMulti-Label Classification<\/div><\/td> F1-Score<\/div><\/td> 0.5906<\/strong><\/div><\/td> 0.5803<\/div><\/td>\n<\/tr>\n \n\tBinary Classification<\/div><\/td> F1-Score<\/div><\/td> 0.9239<\/strong><\/div><\/td> 0.9208<\/div><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n Comparison of the best consensus strategy and the MAD approach in terms of the most important metric for both Classification tasks. For the Multi-Label Classification, the best strategy was the pure Voting strategy; for the Binary Classification it was the pure Weighting strategy.<\/div><\/span>\n\n <\/span>LLM Quantization Methods<\/span><\/h2>\nIn addition to the pure evaluation of the strategies, our goal is to use two different and widely used quantization methods for LLMs, AWQ and GPTQ. Then, based on the results, we’ll make recommendations for the quantization methods depending on the scenario and their benefits.<\/p>\n In general, LLM Quantization is a technique used to make LLMs faster and more efficient without compromising too much on their accuracy. Essentially, it involves reducing the precision of the parameters\/weights the model uses to represent data. Think of it like rounding off decimals to make calculations quicker. Therefore, this process helps models run faster and use less memory, which is especially important when dealing with large datasets and limited hardware resources. Different quantization methods can be used depending on the task, balancing performance and efficiency to optimize the model’s speed and overall performance.<\/p>\n In our evaluation, we use post-training quantization approaches. As the name suggests, post-training quantization is a technique used to quantize a model after it has already been trained.<\/p>\n <\/span>AWQ<\/span><\/h3>\nAWQ (Activation-Aware Weight Quantization) is a smart way of making a model faster and more efficient by reducing the size of its data. However, it carefully does this by following some principles:<\/p>\n \n- Activation-Aware
\n<\/strong>Traditional quantization methods treat weights and activations independently, meaning they don’t consider how the activations behave when adjusting the weights. In AWQ, the quantization process is aware of the activation patterns \u2014 basically, it looks at how data flows through the model and adjusts the weights accordingly.<\/li>\n- Optimizing Precision
\n<\/strong>AWQ doesn’t just reduce the precision of the weights randomly. Instead, it uses a strategy that considers how much precision is needed in each weight based on the activations it will interact with during processing for each channel. In general, more important weights (in regard to activations) get „protected“ by applying smaller scaling factors. By doing so, the quantization process retains as much useful information as possible while still benefiting from the speed and memory-saving advantages of reduced precision.<\/li>\n<\/ul>\nRegarding the aforementioned points, AWQ can be especially beneficial in large-scale applications, like when deploying LLMs to run on resource-constrained devices or cloud environments, where speed and memory efficiency are critical.<\/p>\n <\/span>GPTQ<\/span><\/h3>\nGPTQ (Generalized Post-Training Quantization) is an optimization technique that applies a gradient-based approach to minimize the loss of information during the quantization of weights. Here’s a breakdown of how GPTQ works:<\/p>\n \n- Optimization of Quantization Error
\n<\/strong>The core of GPTQ is its focus on reducing the quantization error. When you reduce the precision of a model’s weights, you lose some details, which can lead to inaccuracies. GPTQ works by minimizing the loss of important information step by step. In other words, GPTQ carefully decides how to lower precision in a way that causes the least harm to the model’s overall performance.<\/li>\n- Batch Quantization
\n<\/strong>To optimize efficiency, GPTQ processes weights in batches to handle multiple weights at once, reducing the overall computation time. Furthermore, instead of quantizing the entire model at once, GPTQ applies layer-wise compression. Each layer’s weights are analyzed individually, taking into account their specific weight distributions. This tailored approach ensures that each layer is quantized in the most efficient way, preserving the model’s performance across different layers.<\/li>\n<\/ul>\nThrough these techniques, GPTQ optimizes the quantization process by minimizing error, improving efficiency, and maintaining model accuracy. This makes it a highly effective approach for compressing LLMs without significant performance loss.<\/p>\n <\/span>Quantization Results When Combining LLMs<\/span><\/h2>\n<\/span>Which Quantization Method Is Better?<\/span><\/h3>\nOverall, AWQ performs slightly better than GPTQ, although again, performance depends heavily on the task. AWQ models provide significantly faster inference for short predictions and generally yield better results. However, GPTQ models sometimes offer significantly faster inference for long predictions, but they tend to deliver slightly poorer overall results in most cases.
\n\n \n\n\n\tTask Type<\/strong><\/div><\/th>Metric<\/strong><\/div><\/th>Better Method<\/strong><\/div><\/th>Significant<\/strong><\/div><\/th>\n<\/tr>\n<\/thead>\n\n\n\tMulti-Label Classification<\/div><\/td> Accuracy<\/div><\/td> AWQ<\/div><\/td> | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |