3f71ba406a
* add details on kubernetes deployment and separate the testing process * Update examples/kubernetes/README.md thanks for suggesting this change, I agree with you and let's make this project better together ! Co-authored-by: JonZeolla <Zeolla@gmail.com> --------- Co-authored-by: QIN Mélony <MQN1@dsone.3ds.com> Co-authored-by: JonZeolla <Zeolla@gmail.com>
38 lines
No EOL
866 B
Markdown
38 lines
No EOL
866 B
Markdown
# Deploy Ollama to Kubernetes
|
|
|
|
## Prerequisites
|
|
|
|
- Ollama: https://ollama.com/download
|
|
- Kubernetes cluster. This example will use Google Kubernetes Engine.
|
|
|
|
## Steps
|
|
|
|
1. Create the Ollama namespace, deployment, and service
|
|
|
|
```bash
|
|
kubectl apply -f cpu.yaml
|
|
```
|
|
|
|
## (Optional) Hardware Acceleration
|
|
|
|
Hardware acceleration in Kubernetes requires NVIDIA's [`k8s-device-plugin`](https://github.com/NVIDIA/k8s-device-plugin) which is deployed in Kubernetes in form of daemonset. Follow the link for more details.
|
|
|
|
Once configured, create a GPU enabled Ollama deployment.
|
|
|
|
```bash
|
|
kubectl apply -f gpu.yaml
|
|
```
|
|
|
|
## Test
|
|
|
|
1. Port forward the Ollama service to connect and use it locally
|
|
|
|
```bash
|
|
kubectl -n ollama port-forward service/ollama 11434:80
|
|
```
|
|
|
|
1. Pull and run a model, for example `orca-mini:3b`
|
|
|
|
```bash
|
|
ollama run orca-mini:3b
|
|
``` |