Private LLM Local AI for Private, Uncensored Chat on iPhone, iPad, and Mac

The core of Private LLM's superior model performance lies in its use of the state-of-the-art OmniQuant quantization algorithm. While quantizing LLMs for on-device inference, outlier values in LLM weights tend to have a marked adverse effect on text generation quality. Omniquant quantization handles outliers by employing an optimization based learnable weight clipping mechanism, which preserves the model's weight distribution with exceptional precision. RTN (Round to nearest) quantization used by popular open-source LLM inference frameworks and apps based on them, does not handle outlier values during quantization, which leads to inferior text generation quality. OmniQuant quantization paired with optimized model-specific Metal kernels, enables Private LLM to deliver text generation that is not only fast but also of the highest quality, significantly raising the bar for on-device LLMs.