Continuous Batching

Continuous Batching (iteration-level scheduling) dynamically schedules LLM requests at the individual token generation step rather than waiting for an entire batch to complete. Finished requests are evicted immediately and new requests are admitted on the fly, eliminating GPU compute idle time caused by uneven sequence lengths and multiplying serving throughput.