Recently, I learned about the concept of continuous batching, where multiple users can interact with a single loaded LLM without significantly decreasing tokens per second. The primary limitation is the KV cache. I am wondering if it is possible to apply continuous batching to a single-user workflow
