Mocking and emulating AWS and GCP services

Unstopped

Google Cloud Spanner

GCP provides a ready-to-use Docker image for the local Cloud Spanner service. The gcloud CLI commands in Listing 10 execute a quick test against a locally running spanner service. To build a Docker image to further launch a quick test routine against a local Cloud Spanner service, create the file in Listing 11 and execute the command

docker build . -f Dockerfile_GCPCSPTST -t gcpcsptst

Listing 10

gcpemu_csptst.sh

#! /bin/sh
(
echo ' <Start of Cloud Spanner Quick Test>'
dockerize -wait tcp://gcpcspemu:9020
gcloud spanner instances create test-instance --config=emulator-config --description="Test Instance" --nodes=1
gcloud spanner instances list --configuration=emulator-config
gcloud spanner instances delete test-instance --configuration=emulator-config --quiet
echo ' <End of Cloud Spanner Quick Test>'
echo
)

Listing 11

Dockerfile_GCPCSPTST

FROM alpine:3.19 AS dockerize
ENV DOCKERIZE_VERSION v0.7.0
RUN wget https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-alpine-linux-amd64-$DOCKERIZE_VERSION.tar.gz && tar -C /usr/local/bin -xzvf dockerize-alpine-linux-amd64-$DOCKERIZE_VERSION.tar.gz && rm dockerize-alpine-linux-amd64-$DOCKERIZE_VERSION.tar.gz && echo "**** fix for host id mapping error ****" && chown root:root /usr/local/bin/dockerize
FROM google/cloud-sdk:alpine
SHELL ["/bin/ash", "-o", "pipefail", "-c"]
RUN gcloud config configurations create emulator --quiet
COPY --from=dockerize /usr/local/bin/dockerize /usr/local/bin/
COPY gcpemu_csptst.sh /usr/local/bin/run.sh
ENTRYPOINT ["run.sh"]

Finally, create the YAML file in Listing 12 and launch the local Bigtable service along with the quick test routine with the command

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock:ro -v ./gcp_emulators_cspemu.yml:/etc/compose/gcp_emulators_cspemu.yml:ro docker docker compose -f /etc/compose/gcp_emulators_cspemu.yml up -d

Listing 12

gcp_emulators_cspemu.yml

services:
  gcpcspemu:
    image: gcr.io/cloud-spanner-emulator/emulator
    container_name: gcpcspemu
    hostname: gcpcspemu
    ports:
      - "29010:9010"
      - "29020:9020"
    restart: unless-stopped
  gcpcsptst:
    image: gcpcsptst
    container_name: gcpcsptst
    hostname: gcpcsptst
    environment:
      - "SPANNER_EMULATOR_HOST=gcpcspemu:9020"
      - "CLOUDSDK_AUTH_DISABLE_CREDENTIALS=true"
      - "CLOUDSDK_CORE_PROJECT=demo"
      - "CLOUDSDK_API_ENDPOINT_OVERRIDES_SPANNER=http://gcpcspemu:9020/"
    depends_on:
      - gcpcspemu
networks:
  default:
    name: gcpemulators-demo
    external: true

You can see the log messages for the local spanner service and the test logic that uses it with the command

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock:ro -v ./gcp_emulators_cspemu.yml:/etc/compose/gcp_emulators_cspemu.yml:ro docker docker compose -f /etc/compose/gcp_emulators_cspemu.yml logs

Figure 4 was taken on my laptop after executing the above logging command. Now you have everything you need to make use of the spanner service; just point your application to local port 29010/ 29020 and reap the benefits of the emulator. Feel free to explore the Cloud Spanner emulator further in the documentation [4].

Figure 4: Google Cloud Spanner emulator quick test log.

Once done, you can use the command

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock:ro -v ./gcp_emulators_cspemu.yml:/etc/compose/gcp_emulators_cspemu.yml:ro docker docker compose -f /etc/compose/gcp_emulators_cspemu.yml down

to clean up.

Cloud Datastore and Firestore

You're in further luck if you use Google Cloud Datastore, its next-generation Cloud Firestore services, or both in your products. These services can also be launched locally through their respective emulators. The Dockerfiles to create the images to be launched locally are shown in Listings 13 and 14.

Listing 13

Dockerfile_GCPCDSEMU

FROM google/cloud-sdk:alpine
EXPOSE 8081
RUN apk add --no-cache openjdk8-jre && gcloud components install beta cloud-datastore-emulator --quiet
ENTRYPOINT ["gcloud","beta","emulators","datastore","start","--project=demo"]
CMD ["--host-port=0.0.0.0:8081"]

Listing 14

Dockerfile_GCPCFSEMU

FROM google/cloud-sdk:alpine
EXPOSE 8721
RUN apk add --no-cache openjdk8-jre && gcloud components install beta cloud-firestore-emulator --quiet
ENTRYPOINT ["gcloud","beta","emulators","firestore","start"]
CMD ["--host-port=0.0.0.0:8721"]

Local AWS Mocking with Moto

AWS is the most popular public cloud services provider, with hundreds of infrastructure-, platform-, and software-as-a-service (IaaS, PaaS, SaaS) offerings, and more. The need of mocking and emulating AWS services becomes more important from financial and dependency perspectives for any company of any size. Because of the power of free and open source software, you can find solutions to mock and emulate almost any AWS service comfortably on your own terms, without paying a single dime.

The first solution to explore is a Python library and server known as Moto [5], with which you can mock a number of AWS services and features in a simple, straightforward way. To begin, create a Docker network from the terminal to start playing with Moto:

docker network create awsmockemu-demo

Now, create the script in Listing 15 to see Moto's basic capabilities as a library in action. This test code simply runs AWS calls between the mock.start and mock.stop methods (lines 16-29). That's all you need to know for raw AWS mocking with Python code.

Listing 15

moto_python_test.py

import boto3
from moto import mock_aws
class MyBucket:
    def __init__(self, name, value):
        self.name = name
        self.value = value
    def save(self):
        s3 = boto3.client("s3", region_name="us-east-1")
        s3.put_object(Bucket="mybucket", Key=self.name, Body=self.value)
def test_s3_save():
    mock = mock_aws()
    mock.start()
    conn = boto3.resource("s3", region_name="us-east-1")
    conn.create_bucket(Bucket="mybucket")
    bucket = MyBucket("PinkFloyd", "is awesome")
    bucket.save()
    body = conn.Object("mybucket", "PinkFloyd").get()[
        "Body"].read().decode("utf-8")
    assert body == "is awesome"
    mock.stop()
if "__main__" == __name__:
  test_s3_save()

Two other ways to mock AWS calls in your code are to use a decorator (by preceding test_s3_save with @mock_aws) or a context manager (all AWS calls after the line with mock_aws). The diffs shown in Listings 16 and 17 are the necessary code changes needed to mock with the Moto decorator and context manager, respectively. The command

docker run --rm -v ./moto_python_test.py:/etc/motopython/moto_python_test.py:ro -w /etc/motopython --entrypoint=python motoserver/moto moto_python_test.py

Listing 16

Decorator Diffs

12a13
> @mock_aws
14,15d14
<     mock = mock_aws()
<     mock.start()
27,28d25
<
<     mock.stop()

Listing 17

Context manager Diffs

<     mock = mock_aws()
<     mock.start()
---
>   with mock_aws():
27,28d25
<
<     mock.stop()

quickly executes the Moto library mocking example code. If no error is thrown, AWS mocking is successful.

If your application is not programmed in Python or you're not in a position to edit your application or use popular Infrastructure as Code (IaC) tools such as Ansible or Terraform, then Moto's server mode comes into the picture. To launch a Moto server along with an AWS CLI container able to execute some basic actions against your locally running AWS mocking server, create the file in Listing 18. Then execute the commands in Listing 19 to bring up the Moto stack and dump information about your default virtual private cloud (VPC) and subnets. Voila, you can see local AWS mocking by running the Moto server (Figure 5).

Figure 5: AWS CLI output showing the default local VPCs and subnets returned by the Moto server.

Listing 18

moto_server_stack.yml

services:
  motoserver:
    image: motoserver/moto:latest
    container_name: motoserver
    hostname: motoserver
    ports:
      - "9500:5000"
    environment:
      - MOTO_PORT=5000
    healthcheck:
      test: ["CMD", "curl", "-I", "localhost:5000"]
      interval: 5s
      timeout: 3s
      retries: 5
    restart: unless-stopped
  awsclitest:
    image: amazon/aws-cli
    container_name: awsclitest
    hostname: awsclitest
    environment:
      - AWS_ACCESS_KEY_ID=foo
      - AWS_SECRET_ACCESS_KEY=foo
      - AWS_DEFAULT_REGION=us-east-1
      - AWS_ENDPOINT_URL=http://motoserver:5000
    entrypoint: "sh"
    command: "-c 'while true; do sleep 5; done'"
    depends_on:
      motoserver:
        condition: service_healthy
networks:
  default:
    name: awsmockemu-demo
    external: true

Listing 19

Moto in Server Mode

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock:ro -v ./moto_server_stack.yml:/etc/compose/moto_server_stack.yml:ro docker docker compose -f /etc/compose/moto_server_stack.yml up -d
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock:ro -v ./moto_server_stack.yml:/etc/compose/moto_server_stack.yml:ro docker compose -f /etc/compose/moto_server_stack.yml exec awsclitest sh -c 'aws ec2 describe-vpcs && aws ec2 describe-subnets'

Next, dump the IDs and descriptions of Linux Amazon machine images (AMIs) returned by the Moto server and the instance types in the mocked environment (Listing 20). Then create an AWS instance in the mocked environment. Please note that the created instance assumes a default key pair, VPC, security group, and so on, but feel free to experiment with creating those new resources and launching your instance under them. Finally, dump the properties of the created instance (Figure 6) and terminate the created instance, if desired, by providing the instance ID (could be different in your case).

Figure 6: AWS instance properties returned by the Moto server.

Listing 20

AWS Instance

# dump IDs and descriptions of Linux AMIs
docker run --rm v /var/run/docker.sock:/var/run/docker.sock:ro -v ./moto_server_stack.yml:/etc/compose/moto_server_stack.yml:ro docker compose -f /etc/compose/moto_server_stack.yml exec awsclitest sh -c 'aws ec2 describe-images --filters "Name=Description,Values=* Linux *" "Name=root-device-type,Values=ebs" --query "Images[*].[ImageId,Description]" --output text'
# dump instance types
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock:ro -v ./moto_server_stack.yml:/etc/compose/moto_server_stack.yml:ro docker compose -f /etc/compose/moto_server_stack.yml exec awsclitest sh -c 'aws ec2 describe-instance-types --query "InstanceTypes[*].[InstanceType]" --output text'
# create an AWS instance
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock:ro -v ./moto_server_stack.yml:/etc/compose/moto_server_stack.yml:ro docker compose -f /etc/compose/moto_server_stack.yml exec awsclitest sh -c 'aws ec2 run-instances --image-id ami-002068ed284fb165b --count 1'
# dump instance properties
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock:ro -v ./moto_server_stack.yml:/etc/compose/moto_server_stack.yml:ro docker compose -f /etc/compose/moto_server_stack.yml exec awsclitest sh -c 'aws ec2 describe-instances'
# terminate instance
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock:ro -v ./moto_server_stack.yml:/etc/compose/moto_server_stack.yml:ro docker compose -f /etc/compose/moto_server_stack.yml exec awsclitest sh -c 'aws ec2 terminate-instances --instance-ids i-6ce75b7f81a552ce0'

These working examples should be enough to help you get going with AWS mocking, and it's just the tip of the iceberg, because Moto covers a number of AWS services [6].

« Previous 1 2 3 Next »