InfoQ HomepageNewsShopify Introduces Gisting: Compressing LLM System Prompts into Learned Tokens<\/p>\n
AI, ML & Data Engineering
\nBeyond the PR: The New Control Plane for Agentic Software Delivery (Webinar Sept 17th)<\/p>\n
Shopify’s engineering introduced Gisting, a novel technique for compressing long LLM prompts into a smaller set of learned “gist” tokens, improving throughput and reducing inference cost<\/p>\n
Spotify emphasizes that replacing lengthy text for concise gist tokens at inference time reduces end-to-end latency, drops infrastructure costs, and boosts token throughput without modifying the model’s core weights<\/p>\n
The company says that gisting reduced the Sidekick GraphQL agent\u2019s system prompt from about 6000 tokens to 1500 gist tokens without sacrificing prediction quality. This implies a 4:1 reduction in context size:<\/p>\n
\nAt 350 requests per minute (RPM), the median time to first token (TTFT) dropped from 438ms to 354ms, the median end-to-end request latency dropped from 6.8s to 4.2s, and throughput rose from 20.2 to 23.4 queries per second (QPS)<\/p>\n<\/blockquote>\n
These improved metrics allowed Spotify to reduce the number of allocated GPUs<\/p>\n
Gisting is based on a technique pioneered in a 2022 paper, “Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models<\/em>“<\/a>, and consists of a two-step process to learn the embeddings of the new compressed gist token. In a first pass, the teacher pass<\/em>, the model is run with the real prompt to derive the teacher logits<\/em> of the response. In the student pass<\/em>, the model is run with the gist tokens to derive the student logits. Finally, the gist are trained to minimize the KL divergence between the teacher logits and the student logits, that is until the student’s predictions closely match the teacher’s.<\/p>\n
\nWhen training finishes, we write the gist embeddings straight into the model’s embedding matrix, and register the new gist tokens as special tokens in the model\u2019s tokenizer. The model loads and runs like any other at inference time: no custom attention mask, extra encoder, or special serving path<\/a><\/p>\n<\/blockquote>\n
The key advantage of gisting is that the model does not process a conventional summary of the original prompt, but rather a learned representation designed to make the LLM to behave as close as possible to how it would if it had seen the original prompt<\/p>\n
Gisting can reduce latency and increase throughput. In Shopify’s case, Time to First Token (TTFT) dropped from 438ms to 354ms, and end-to-end latency fell from 6.8s to 4.2s. At the same time, queries per second (QPS) increased from 20.2 to 23.4, allowing engineering teams to scale down overall GPU allocation<\/p>\n
As a final note, Shopify also emphasizes that gisting is complementary to other optimization techniques, such as prefix caching<\/a>. Prefix caching avoids recomputing the KV tensors for cached prompt sequences, but the model must still process those cached tensors during the decoding phase. Gisting further reduces this overhead by replacing a long prompt with a shorter sequence of learned gist tokens. The two optimizations therefore compound, and Shopify uses them together.<\/p>\n
There is much more to gisting than can be covered here. Make sure to read the original article if you are interested in the full details, which covers topics such as the role of autosearch in tuning the Gisting process and other implementation details that significantly affect performance<\/p>\n
\n\n En rapport:<\/strong>
\n <a href="https:\/\/yoursite.com\/automation-training-benin\/" title="Num\u00e9rique<\/a> Formation en automatisation au B\u00e9nin\u00a0: 5 comp\u00e9tences gagnantes que les employeurs recherchent en 2026
\n Formation en automatisation num\u00e9rique au B\u00e9nin\u00a0: 5 comp\u00e9tences cl\u00e9s recherch\u00e9es par les employeurs en 2026
\n <\/a>\n <\/p>\n\n <a href="https:\/\/yoursite.com\/automation-africa\/" title="WhatsApp<\/a> Marketing Automation Africa : 6 erreurs dangereuses commises par les marques au Nig\u00e9ria\u201d>
\n Automatisation du marketing WhatsApp en Afrique\u00a0: 6 erreurs dangereuses commises par les marques au Nig\u00e9ria
\n <\/a>\n <\/p>\n<\/div>\n\nVous souhaitez apprendre cela de mani\u00e8re pratique ?<\/h3>\n
Rejoindre Justfine Infotech<\/strong> et d\u00e9velopper de v\u00e9ritables comp\u00e9tences num\u00e9riques en IA, automatisation, d\u00e9veloppement web, marketing digital, bureautique, e-commerce, travail ind\u00e9pendant et cybers\u00e9curit\u00e9.<\/p>\n
Programmes disponibles :<\/strong>
\n Certificat de 6 semaines \u2022 Certificat professionnel de 3 mois \u2022 Dipl\u00f4me de 6 mois \u2022 Dipl\u00f4me professionnel complet<\/p>\nWhatsApp :<\/strong>
\n +229 01 57 57 99 15
\n +229 01 66 68 11 60<\/p>\n