Partnership brings TPU to vLLM alongside new Ternary model releases
Google Cloud and Inferact announced a partnership to make TPUs a first-class citizen in the vLLM project. Additionally, developers tested low-bit ternary compressed models like Ternary-Bonsai-2-27B running locally on consumer hardware.
5 independent accounts
19 posts
10,478 interactions
vllmmixture of expertsllamallama.cppunsloth