Kavli Affiliate: Huawei Zhang | Summary:Language models (LMs) underpin emerging mobile and embedded AI applications like meeting and video summarization and document analysis, which often require processing multiple long-context inputs. Running an LM locally on-device improves privacy, enables offline use, and reduces cost, but long-context inference quickly hits a emphmemory capacity wall as the key-value […]
Continue.. KVSwap: Disk-aware KV Cache Offloading for Long-Context On-device Inference