Quite recently I watched a short course created by DeepLearning.AI titled “Serverless LLM apps with Amazon Bedrock“. It is a very good course for beginners, well crafted, with good examples, and clear explanations by Mike Chambers. He goes and creates a small app using AWS Lambda, AWS S3, and Amazon Bedrock that does text summarisation of recorded conversations. I wanted to add something to the topic and provide a fully working example you can deploy easily using AWS CDK.
DAPR is cool, as stated on the website “APIs for building portable and reliable microservices”, it works with many clouds, and external services. As a result you only need to configure the services and then use DAPR APIs. It is true and I’ll show you. You will find the code for this article here. A must have tool when working with micro services.
Applications can use DAPR as a sidecar container or as a separate process. I’ll show you a local version of the app, where DAPR is configured to use Redis running in a container. The repo will have Azure, AWS, and GCP configuration as well.
Before we can start the adventure you have to install DAPR.
You have to start off on the right foot. Because of that we have to configure secrets store where you can keep passwords and such. I could skip this step but then there is a risk someone will never find out how to do it properly and will ship passwords in plain text. Here we go.
# save under ./components/secrets.yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: local-secret-store
namespace: default
spec:
type: secretstores.local.file
version: v1
metadata:
- name: secretsFile
value: ../secrets.json
The file secrets.json should have all your secrets, like connection strings, user and pass pairs, etc. Don’t commit this file.
{
"redisPass": "just a password"
}
Next file is publish subscribe configuration. Dead simple but I’d recommend going through the docs as there is much more to pub/sub. Here you can reference your secrets as shown.
# save under ./components/pubsub.yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: order_pub_sub
spec:
type: pubsub.redis
version: v1
metadata:
- name: redisHost
value: localhost:6379
- name: redisPass
secretKeyRef:
name: redisPass
key: redisPass
auth:
secretStore: local-secret-store
With config out of the way only thing left are publisher part and subscriber part. As mentioned before the app you write talks to DAPR API. This means you may use http calls or Dapr client. Best part is that no matter what is on the other end, be it Redis or PostgreSQL, your code will not change even when you change your external services.
Here goes publisher that will send events to a topic. Topic can be hosted anywhere, here is a list of supported brokers. The list is long however only 3 are stable. I really like how DAPR is approaching components certification though. There are well defined requirements to pass to advance from Alpha, to Beta, and finally to Stable.
# save under ./publisher/app.py
import logging
from dapr.clients import DaprClient
from fastapi import FastAPI
from pydantic import BaseModel
logging.basicConfig(level=logging.INFO)
app = FastAPI()
class Order(BaseModel):
product: str
@app.post("/orders")
def orders(order: Order):
logging.info("Received order")
with DaprClient() as dapr_client:
dapr_client.publish_event(
pubsub_name="order_pub_sub",
topic_name="orders",
data=order.json(),
data_content_type="application/json",
)
return order
Here is a consumer.
# save under ./consumer/app.py
from dapr.ext.fastapi import DaprApp
from fastapi import FastAPI
from pydantic import BaseModel
app = FastAPI()
dapr_app = DaprApp(app)
class CloudEvent(BaseModel):
datacontenttype: str
source: str
topic: str
pubsubname: str
data: dict
id: str
specversion: str
tracestate: str
type: str
traceid: str
@dapr_app.subscribe(pubsub="order_pub_sub", topic="orders")
def orders_subscriber(event: CloudEvent):
print("Subscriber received : %s" % event.data["product"], flush=True)
return {"success": True}
Now you can run both apps together in separate terminal windows and see how they talk to each other using configured broker. For this example we are using Redis as a broker. You will see how easy is to run them on different platforms.
In the first terminal run the consumer.
$ dapr run --app-id order-processor --components-path ../components/ --app-port 8000 -- uvicorn app:app
In the other terminal run the producer.
$ dapr run --app-id order-processor --components-path ../components/ --app-port 8001 -- uvicorn app:app --port 8001
After you make a HTTP call to a producer you should see both of them producing log messages as follows.
$ http :8001/orders product=falafel
# producer
== APP == INFO:root:Received order
== APP == INFO: 127.0.0.1:49698 - "POST /orders HTTP/1.1" 200 OK
# subscriber
== APP == Subscriber received : falafel
== APP == INFO: 127.0.0.1:49701 - "POST /events/order_pub_sub/orders HTTP/1.1" 200 OK
It took us a bit to reach the clu of this post. We had to build something, and run it so then we can run it in the cloud. Above example will run on cloud with a simple change of configuration.
Simplest configuration is for Azure. Change your pubsub.yaml so it looks as follows, and update your secrets.json as well.
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: order_pub_sub
spec:
type: pubsub.azure.servicebus
version: v1
metadata:
- name: connectionString
secretKeyRef:
name: connectionStrings:azure
key: connectionStrings:azure
Your secrets.json should look like this now
{
"connectionStrings": {
"azure": "YOUR CONNECTION STRING"
}
}
Rerun both commands in terminal and the output will look the same as with local env but the app will run on Azure Service Bus.
Bloody magic if you’d ask me. You can mix and match your dependencies without changing your application. In some cases you may even use features not available to a particular cloud, like message routing based on body in Azure Service Bus. This will be another post though.
Here is the repo for this post, it includes all the providers listed below:
Please remember to update your secrets.json.
Have fun 🙂
I was looking for a way to deploy a custom model to Sagemaker. Unfortunately, my online searches failed to find anything that was not using Jupiter notebooks. I like them but this way of deploying models is not a reproducible way nor it is scalable.
After a couple of hours of looking, I decided to do it myself. Here comes a recipe for deploying a custom model to Sagemaker using AWS CDK.
The following steps assume you have knowledge of CDK and Sagemaker. I’ll try to explain as much as I can but if anything is unclear please refer to the docs.
Unfortunately, AWS CDK does not support higher-level constructs for Sagemaker. You have to use CloudFormation constructs which start with the prefix Cfn. Higher-level constructs for Sagemaker are not on the roadmap as of March 2021.
First thing is to have your app in a container form, so it can be deployed in a predictable way. It’s difficult to help with this step as each model may require different dependencies or actions. What I can recommend is to go over https://docs.aws.amazon.com/sagemaker/latest/dg/build-multi-model-build-container.html. This page explains the steps required to prepare a container that can serve a model on Sagemaker. It may also be helpful to read this part https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-inference-code.html on how your docker image will be used.
Once you have your model in a container form it is time to create a Sagemaker model. There are 3 elements to a Sagemaker model:
Adding container definition to your app is simple (the hard part of creating a docker image is already done). The container definition will be used by the Sagemaker model.
asset = DockerImageAsset(
self,
"MLInferenceImage",
directory="../image")
primary_container_definition = sagemaker.CfnModel.ContainerDefinitionProperty(
image=asset.image_uri,
)Code language: PHP (php)Creating Vpc is pretty straightforward, you have to remember about creating public and private subnets.
vpc = ec2.Vpc(
self,
"VPC",
subnet_configuration=[
ec2.SubnetConfiguration(
name="public-model-subnet", subnet_type=ec2.SubnetType.PUBLIC
),
ec2.SubnetConfiguration(
name="private-model-subnet", subnet_type=ec2.SubnetType.PRIVATE
),
],
)
model_vpc_config =
sagemaker.CfnModel.VpcConfigProperty(
security_group_ids=[vpc.vpc_default_security_group],
subnets=[s.subnet_id for s in vpc.private_subnets],
)Code language: PHP (php)Creating a model is putting all created things together.
model = sagemaker.CfnModel(
self,
"MLInference",
execution_role_arn=role.role_arn,
model_name="my-model",
primary_container=primary_container_definition,
vpc_config=model_vpc_config,
)Code language: PHP (php)At this point, cdk deploy would create Sagemaker model with an ML model of your choice.
We are not done yet as the model has to be exposed. Sagemaker Endpoint is perfect for this and in the next step we create endpoint configuration.
Endpoint configuration describes resources that will serve your model.
model_endpoint_config = sagemaker.CfnEndpointConfig(
self,
"model-endpoint-config",
production_variants=[
sagemaker.CfnEndpointConfig.ProductionVariantProperty(
initial_instance_count=1,
initial_variant_weight=1.0,
instance_type="ml.t2.medium",
model_name=model.model_name,
variant_name="production-medium",
),
],
)Code language: PHP (php)Last step is extremely simple. We take the configuration created earlier and create an endpoint.
model_endpoint = sagemaker.CfnEndpoint(
self,
"model-endpoint", endpoint_config_name=model_endpoint_config.attr_endpoint_config_name,
)Code language: PHP (php)Now you may call cdk deploy and the model is up and running on AWS Sagemaker 🙂