New Acquisition for my Cluster: Nvidia Jetson Orin Nano 8GB
As I have mentioned here before, I have a Kubernetes cluster made up of 4 Raspberry Pi 4 units and one Raspberry Pi 5. I mainly use it for CI/CD with Gitea, web scraping with Selenium, and many n8n workflows that heavily rely on OpenAI (simple agents and chains).
Nvidia Jetson Orin Nano 8GB
The thing is, the other day I saw a post from Asier Arranz showing how well the Gemma 2 model, quantized to 4 bits, performs on the Nvidia Jetson Orin Nano with 8 GB. The AI capability of this SBC is 40 TOPS. It’s amazing considering it has a peak consumption of 17W!
Gemma 2 performs very well despite being an SLM. It is supposed to reach 12 tokens/s. Although I mostly use GPT-4o mini in most of my workflows and the cost is “negligible”, I like the idea of not depending on OpenAI and being able to run it locally.
Technical Specifications
For those who are not familiar with the Nvidia Jetson Orin Nano, here is a summary of its technical specifications:
- CPU: 6-core Arm® Cortex®-A78AE v8.2 64-bit CPU
- GPU: 1024-core NVIDIA Ampere architecture GPU with 32 Tensor Cores
- Memory: 8GB 128-bit LPDDR5
- Storage: Support for M.2 SSD
- Interfaces: 1 x USB 3.2 Gen 2 Type-C, 1 x HDMI, 1 x DisplayPort, PCIe Gen4, among others
For more details on the specifications, you can visit the official Nvidia page.
Integration with the Cluster
The Jetson Orin Nano is supposed to arrive this coming week, and I already have a 2TB M.2 SSD waiting to give it the necessary space for Gemma 2 and its fine-tunings (some will fall with the desktop and its 2x RTX 3090 Ti).
I am particularly excited to integrate the Jetson Orin Nano with my Raspberry Pi cluster. The Jetson Orin Nano has the ability to perfectly complement the current nodes and offer an AI processing power I did not have in my local network before.
Tests and Results
Anyway, once the kit arrives and is fully integrated into my cluster, I plan to carry out various tests. From evaluating the performance of the Gemma 2 model with 12 tokens/s to the general operation with my current CI/CD and web scraping workflows.
I will keep you posted on how it goes and what results I get. I hope it’s as promising as it seems!
