Local LLM inference in OCI Ampere A1

In this series, I’m going to walk you through running your own large language models locally in CPU-only setup. Specifically, you will utilize Oracle Cloud Infrastructure’s generous free-tier offering by provisioning an Arm-based Ampere A1 instance, and running the Ampere-optimized llama.cpp container as your inference engine. By using this container, you will be able to leverage an optimized quantization scheme to both reduce model size and speed up inferencing.

We’ll also compare different inference engines on the Ampere platform and look at options to optimize our llama.cpp server.

Let’s go!