vLLM - Turbo Charge your LLM Inference 

Uhh... you already get instant response from GGML/llama.cpp (apart from the model weights loading time, but this is not anything that PagedAttention improves on). The deal with PagedAttention is that it prevents the KV cache from wasting memory by not overallocating the entire context length at once, but rather doing it in chunks as the sequence keeps growing (and possibly sharing chunks among different inference beams or users). This allows the same model to serve more users (throughput) - of course, provided that they generate sequences shorter than the context length. It should not affect the response time for any individual user (if anything, it makes it worse because of the overhead of mapping virtual to physical memory blocks). So if it improves HF in that respect, it just demonstrates that either HF's implementation of KV cache sucks or Sam is comparing non-KV-cached generation with KV-cached one.

AFAIK I don't think they are supporting Bitsandbytes etc.which doesn't surprise me as for what they are mainly using it for is comparing models which is not ideal a low resolution quantization.

Thanks for this video. Although I should mention that at least on my RTX 3090 TI the GPTQ 13B models with exllama loader are absolutely flying. Faster than ChatGPT-3.5 turbo. But I'll definitely take a look

for many of the options I have looked at this compares well for the models that it works with etc.

Yes none of these are flawless. I might make about video about hosting with HF Text-gen-inference as well.

They compared to TGI also which does have Flash-Attention huggingface.co/text-generation-inference and it is still quite a bit faster

no I don't think it will work with that.

My guess is just because their setup is not using that yet and it will come. I actually just checked my Colab and that seems to be running in Cuda 12.0 but maybe that is not optimal.

does ONNX have a streaming ability? I can't see any mention of websocket or http/2

@@s0ckpupp3t Not that I know. I converted bloom-560 to ONNX and got similar latency as with vLLM. I guess with ONNX one could optimise it a bit further, but I'm impressed by vLLM because it's much easier to use.

If you are talking with bitsandbytes I don't hink it does just yet.

Yes totally the colab I show was downloading a model from HuggingFace. Not all of the LLMs are compatible, but most the popular ones are.

@@samwitteveenai In my office setup, these models cannot be downloaded (blocked), so I download them separately and use their weights using huggingface pipelines as LLM for Langchain and other use cases. Will try a similar approach for vLLM hoping that this approach works

@@navneetkrc Yes totally, will just need to load locally etc.

@@samwitteveenai thanks a lot for the quick replies. You are the best 🤗

You can use it as an LLM in Langchain. Whether it will work with tools will depend on which model you serve etc.

@@samwitteveenai I assume it doesn't support quants? Don't see any mention

You are running it on a machine you control. What are the privacy issues ?

@@samwitteveenai i though that it's cloud based 🎩

what model you trying to get to work? It also doesn't support quantized models if you are trying for that.

@@samwitteveenai Hi Sam, thanks for the sharing(life-saver for newbies). Wonder your recommendation for quantized models ?

no so far these are for full resolution models only

they have a docker image. That's what im using right now