Selecting a mannequin for BYOK mode
If you wish to use an area LLM with VS Code’s bring-your-own-model system, the very first thing you want is a solution to host the mannequin. VS Code lacks a model-hosting mechanism of its personal, though it’s conceivable {that a} VS Code extension could provide one thing like that sooner or later. That mentioned, internet hosting fashions is difficult sufficient {that a} devoted app is actually wanted for the job.
One simple solution to host fashions is through a product like LM Studio, a handy GUI for standing up, serving, and managing LLMs on one’s personal {hardware}. The mannequin host doesn’t should be the identical system you run VS Code on, both. It may be on a server field you management, or on a cloud occasion.
The selection of mannequin can be vital. Many fashions are highly effective however gained’t run properly on commodity {hardware} as a result of they’re just too large. A great rule of thumb is to decide on a mannequin that matches into current VRAM, together with the reminiscence wanted for a large token context (the extra, the higher). Additionally, the mannequin needs to be suited to coding and growth work. Some fashions on this vein that match comfortably into 8GB VRAM embrace: