Kavli Affiliate: Ran Wang
| First 5 Authors: Ran Wang, Ran Wang, , ,
| Summary:
Structured decoding enables large language models (LLMs) to generate outputs
in formats required by downstream systems, such as HTML or JSON. However,
existing methods suffer from efficiency bottlenecks due to grammar compilation,
state tracking, and mask creation. We observe that many real-world tasks embed
strong prior knowledge about output structure. Leveraging this, we propose a
decomposition of constraints into static and dynamic components — precompiling
static structures offline and instantiating dynamic arguments at runtime using
grammar snippets. Instead of relying on pushdown automata, we employ a
compositional set of operators to model regular formats, achieving lower
transition latency. We introduce wgrammar, a lightweight decoding engine that
integrates domain-aware simplification, constraint decomposition, and mask
caching, achieving up to 250x speedup over existing systems. wgrammar’s source
code is publicly available at https://github.com/wrran/wgrammar.
| Search Query: ArXiv Query: search_query=au:”Ran Wang”&id_list=&start=0&max_results=3