In the world of AI, there’s a growing tension between capability and control.
Large Language Models (LLMs) are getting smarter and more accessible – but they often come bundled with dependencies: expensive GPUs, managed cloud services, and opaque systems that make true privacy and autonomy feel out of reach.
At Rubyness, we’ve been asking a simple but radical question:
Can you build and operate a full AI model lifecycle – from training to deployment – using only open tools and CPUs?
Spoiler: yes, you can. And we’re showing how.
Why CPU-First AI Matters
Our CEO, Sergy Sergyenko, has been exploring what it means to design practical, private AI systems for sectors where data sensitivity isn’t negotiable – like healthcare, finance, or government.
At the upcoming Cyber Scotland Connect meetup in Leith (Edinburgh), Sergy will share our hands-on journey with CPU-first LLM stacks, built entirely on open infrastructure.
The talk dives into how tools like Llama.cpp and the GGML ecosystem enable local, high-performance inference on standard hardware – without the need for expensive cloud compute or proprietary APIs.
It’s a story about reclaiming control over AI.
Building a Fully Open LLM Stack
The session walks through what a real, end-to-end LLM development lifecycle looks like when built with openness and efficiency in mind:
- Local inference on CPUs with optimized GGML builds.
- Secure app ↔ model integration that respects privacy laws and data boundaries.
- Continuous fine-tuning done locally – no external servers, no data leaks.
- Production operations with lightweight deployment, observability, and hardware-aware optimization.
This isn’t theory – it’s a practical blueprint that any organization can adapt.
Whether you’re running experiments on a developer laptop or deploying in production on a private network, the same philosophy applies: build small, build open, and stay in control.
What You’ll Take Away
If you attend the talk, you’ll leave with a clear understanding of:
- How llama.cpp/GGML achieve fast, memory-efficient inference on CPUs.
- How to deploy and manage on-prem AI systems for privacy-critical workloads.
- The benefits (and boundaries) of quantization, including int4 formats.
- How to fine-tune models locally and evaluate performance without the cloud.
- Strategies for packaging, routing, and monitoring models in production.
Open AI, Literally
For us, this work is more than technical optimization – it’s about philosophy.
In a landscape where AI infrastructure is increasingly centralized and costly, we see enormous value in open ecosystems that keep innovation transparent and accessible.
By leaning on CPU optimization, quantization, and local-first design, organizations can build AI that’s:
- Private – data never leaves your premises.
- Affordable – no GPU arms race required.
- Reliable – runs on commodity hardware.
- Independent – no vendor lock-in or hidden dependencies.
This approach empowers developers to experiment and deploy confidently, knowing they own the full stack—from code to compute.
Join Us in Leith
If you’re in Edinburgh, come meet us at Season Quayside in Leith for the next Cyber Scotland Connect meetup.
We’ll be discussing how to turn these ideas into practice – alongside the local cybersecurity and AI community.
And yes, there will be refreshments.