⚠️ This notebook has been deprecated and it may require modifications to work correctly.
AWS Elastic Kubernetes Service (EKS) Deep MNIST
In this example we will deploy a tensorflow MNIST model in Amazon Web Services' Elastic Kubernetes Service (EKS).
This tutorial will break down in the following sections:
Train a tensorflow model to predict mnist locally
Containerise the tensorflow model with our docker utility
Send some data to the docker model to test it
Install and configure AWS tools to interact with AWS
Use the AWS tools to create and setup EKS cluster with Seldon
Push and run docker image through the AWS Container Registry
Test our Elastic Kubernetes deployment by sending some data
Let's get started! 🚀🔥
Dependencies:
Helm v3.0.0+
A Kubernetes cluster running v1.13 or above (minkube / docker-for-windows work well if enough RAM)
kubectl v1.14+
EKS CLI v0.1.32
AWS Cli v1.16.163
Python 3.6+
Python DEV requirements
1) Train a tensorflow model to predict mnist locally
We will load the mnist images, together with their labels, and then train a tensorflow model to predict the right labels
2) Containerise the tensorflow model with our docker utility
First you need to make sure that you have added the .s2i/environment configuration file in this folder with the following content:
Now we can build a docker image named "deep-mnist" with the tag 0.1
3) Send some data to the docker model to test it
We first run the docker image we just created as a container called "mnist_predictor"
Send some random features that conform to the contract
png
4) Install and configure AWS tools to interact with AWS
First we install the awscli
Configure aws so it can talk to your server
(if you are getting issues, make sure you have the permmissions to create clusters)
Install EKCTL
IMPORTANT: These instructions are for linux Please follow the official installation of ekctl at: https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html
5) Use the AWS tools to create and setup EKS cluster with Seldon
In this example we will create a cluster with 2 nodes, with a minimum of 1 and a max of 3. You can tweak this accordingly.
If you want to check the status of the deployment you can go to AWS CloudFormation or to the EKS dashboard.
It will take 10-15 minutes (so feel free to go grab a ☕).
IMPORTANT: If you get errors in this step it is most probably IAM role access requirements, which requires you to discuss with your administrator.
Configure local kubectl
We want to now configure our local Kubectl so we can actually reach the cluster we've just created
And we can check if the context has been added to kubectl config (contexts are basically the different k8s cluster connections) You should be able to see the context as "...aws:eks:eu-west-1:27...". If it's not activated you can activate that context with kubectlt config set-context <CONTEXT_NAME>
import matplotlib.pyplot as plt
# This is the variable that was initialised at the beginning of the file
i = [0]
x = mnist.test.images[i]
y = mnist.test.labels[i]
plt.imshow(x.reshape((28, 28)), cmap="gray")
plt.show()
print("Expected label: ", np.sum(range(0, 10) * y), ". One hot encoding: ", y)
Expected label: 7.0 . One hot encoding: [[0. 0. 0. 0. 0. 0. 0. 1. 0. 0.]]
import math
import numpy as np
from seldon_core.seldon_client import SeldonClient
# We now test the REST endpoint expecting the same result
endpoint = "0.0.0.0:5000"
batch = x
payload_type = "ndarray"
sc = SeldonClient(microservice_endpoint=endpoint)
# We use the microservice, instead of the "predict" function
client_prediction = sc.microservice(
data=batch, method="predict", payload_type=payload_type, names=["tfidf"]
)
for proba, label in zip(
client_prediction.response.data.ndarray.values[0].list_value.ListFields()[0][1],
range(0, 10),
):
print(f"LABEL {label}:\t {proba.number_value*100:6.4f} %")
%%bash
# You must make sure that the access key and secret are changed
aws configure << END_OF_INPUTS
YOUR_ACCESS_KEY
YOUR_ACCESS_SECRET
us-west-2
json
END_OF_INPUTS
AWS Access Key ID [****************SF4A]: AWS Secret Access Key [****************WLHu]: Default region name [eu-west-1]: Default output format [json]:
!curl --silent --location "https://github.com/weaveworks/eksctl/releases/download/latest_release/eksctl_$(uname -s)_amd64.tar.gz" | tar xz
%%bash
export AWS_ACCOUNT_ID=""
export AWS_REGION="us-west-2"
if [ -z "$AWS_ACCOUNT_ID" ]; then
echo "ERROR: Please provide a value for the AWS variables"
exit 1
fi
docker tag deep-mnist:0.1 "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/seldon-repository"
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /home/alejandro/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
%%bash
export AWS_ACCOUNT_ID=""
export AWS_REGION="us-west-2"
if [ -z "$AWS_ACCOUNT_ID" ]; then
echo "ERROR: Please provide a value for the AWS variables"
exit 1
fi
docker push "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/seldon-repository"
%%bash
export AWS_ACCOUNT_ID=""
export AWS_REGION="us-west-2"
if [ -z "$AWS_ACCOUNT_ID" ]; then
echo "ERROR: Please provide a value for the AWS variables"
exit 1
fi
sed 's|REPLACE_FOR_IMAGE_AND_TAG|'"$AWS_ACCOUNT_ID"'.dkr.ecr.'"$AWS_REGION"'.amazonaws.com/seldon-repository|g' deep_mnist.json | kubectl apply -f -
error: unable to recognize "STDIN": Get https://461835FD3FF52848655C8F09FBF5EEAA.yl4.us-west-2.eks.amazonaws.com/api?timeout=32s: dial tcp: lookup 461835FD3FF52848655C8F09FBF5EEAA.yl4.us-west-2.eks.amazonaws.com on 1.1.1.1:53: no such host
---------------------------------------------------------------------------
CalledProcessError Traceback (most recent call last)
<ipython-input-165-1129742af2c4> in <module>
----> 1 get_ipython().run_cell_magic('bash', '', 'export AWS_ACCOUNT_ID="2710"\nexport AWS_REGION="us-west-2"\nif [ -z "$AWS_ACCOUNT_ID" ]; then\n echo "ERROR: Please provide a value for the AWS variables"\n exit 1\nfi\n\nsed \'s|REPLACE_FOR_IMAGE_AND_TAG|\'"$AWS_ACCOUNT_ID"\'.dkr.ecr.\'"$AWS_REGION"\'.amazonaws.com/seldon-repository|g\' deep_mnist.json | kubectl apply -f -\n')
~/miniconda3/envs/reddit-classification/lib/python3.7/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
2350 with self.builtin_trap:
2351 args = (magic_arg_s, cell)
-> 2352 result = fn(*args, **kwargs)
2353 return result
2354
~/miniconda3/envs/reddit-classification/lib/python3.7/site-packages/IPython/core/magics/script.py in named_script_magic(line, cell)
140 else:
141 line = script
--> 142 return self.shebang(line, cell)
143
144 # write a basic docstring:
</home/alejandro/miniconda3/envs/reddit-classification/lib/python3.7/site-packages/decorator.py:decorator-gen-110> in shebang(self, line, cell)
~/miniconda3/envs/reddit-classification/lib/python3.7/site-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
185 # but it's overkill for just that one bit of state.
186 def magic_deco(arg):
--> 187 call = lambda f, *a, **k: f(*a, **k)
188
189 if callable(arg):
~/miniconda3/envs/reddit-classification/lib/python3.7/site-packages/IPython/core/magics/script.py in shebang(self, line, cell)
243 sys.stderr.flush()
244 if args.raise_error and p.returncode!=0:
--> 245 raise CalledProcessError(p.returncode, cell, output=out, stderr=err)
246
247 def _run_script(self, p, cell, to_close):
CalledProcessError: Command 'b'export AWS_ACCOUNT_ID="2710"\nexport AWS_REGION="us-west-2"\nif [ -z "$AWS_ACCOUNT_ID" ]; then\n echo "ERROR: Please provide a value for the AWS variables"\n exit 1\nfi\n\nsed \'s|REPLACE_FOR_IMAGE_AND_TAG|\'"$AWS_ACCOUNT_ID"\'.dkr.ecr.\'"$AWS_REGION"\'.amazonaws.com/seldon-repository|g\' deep_mnist.json | kubectl apply -f -\n'' returned non-zero exit status 1.
!kubectl get pods
NAME READY STATUS RESTARTS AGE
ambassador-5475779f98-7bhcw 1/1 Running 0 21m
ambassador-5475779f98-986g5 1/1 Running 0 21m
ambassador-5475779f98-zcd28 1/1 Running 0 21m
deep-mnist-single-model-42ed9d9-fdb557d6b-6xv2h 2/2 Running 0 18m
!kubectl get svc ambassador -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
import matplotlib.pyplot as plt
# This is the variable that was initialised at the beginning of the file
i = [0]
x = mnist.test.images[i]
y = mnist.test.labels[i]
plt.imshow(x.reshape((28, 28)), cmap="gray")
plt.show()
print("Expected label: ", np.sum(range(0, 10) * y), ". One hot encoding: ", y)
Expected label: 7.0 . One hot encoding: [[0. 0. 0. 0. 0. 0. 0. 1. 0. 0.]]
import math
import numpy as np
from seldon_core.seldon_client import SeldonClient
host = "a68bbac487ca611e988060247f81f4c1-707754258.us-west-2.elb.amazonaws.com"
port = "80" # Make sure you use the port above
batch = x
payload_type = "ndarray"
sc = SeldonClient(
gateway="ambassador", ambassador_endpoint=host + ":" + port, namespace="default"
)
client_prediction = sc.predict(
data=batch, deployment_name="deep-mnist", names=["text"], payload_type=payload_type
)
print(client_prediction)
