Comment on
How much gpu do i need to run a 90b model
In case of LLM's you should look at AirLLM. I suppose there is no conviniet integrations to local chat tools, but issue at Ollama already started.
Replies
Comment on
How much gpu do i need to run a 90b model
Reply in thread
You may try Harbor. The description claims to provide an OpenAI-compatible API.