My Local Environment for AI Testing with LLMs

Hello everyone! Chema here. It’s been a while since I wrote anything, but today I’m back to share with you details about my local environment for AI testing with LLMs.

My Equipment

My rig consists of an Intel Core i7 with 64 GB of DDR5 and two nVidia RTX 3090 Ti GPUs. With a total of 48 GB of VRAM, this setup allows me to run a large number of quantified LLMs locally.

About the nVidia RTX 3090 Ti

The nVidia RTX 3090 Ti GPUs are high-performance graphics cards designed for general-purpose and graphic computing. They feature 24 GB of VRAM each, allowing the simultaneous execution of multiple AI processes, even those requiring large amounts of memory.

Furthermore, the RTX 3090 Ti supports NVIDIA DLSS, an AI rendering technology that increases frame rates and generates beautiful quality images. This makes them an excellent choice for any AI professional or enthusiast.

Quantified LLMs and Finetuning with QLoRa

Among the models I usually test are Mixtral, Llama 3, and the new Codestral, which is absolutely amazing for code development.

Additionally, this environment allows me to do finetuning using the QLoRa technique. This technique is based on quantization and learning of learning rates to refine LLM models. It tends to provide excellent results, especially in terms of efficiency and performance.

Thanks to this robust local environment, I have the ability to test and experiment with multiple models efficiently and effectively. This allows me to develop and optimize models more quickly and safely. See you next time!