Deploy the Log Processor

In order for us to handle our log entries in background we need two complimentary applications that we could either build ourselves or deploy from a SAR (Serverless Application Repository) already provided by a partner called Lumigo, which are responsible for auto-subscribing specific CloudWatch Log Groups to a certain Kinesis Stream, and another one that consumes data from this Stream, parses it and pushed to CloudWatch Metrics. These apps are called SAR-Logging and SAR-Async-Lambda-Metrics.

Spare a couple of minutes to understand how these two SAR apps are being deployed by examining the SAM Template on serverless-observability-workshop/code/log-processing/template.yaml file.

By inspecting the serverless-observability-workshop/code/log-processing/template.yaml file, you can understand the patterns that Amazon CloudWatch Logs will attempt to filter and route to this module.

# Only captures Custom Metrics and INFO|WARN|ERROR log events; reduces cost and unnecessary noise
FilterPattern:  "?INFO ?ERROR ?WARN ?REPORT ?MONITORING"

Also, by inspecting the logMetric() and buildStatsDMetricData() methods at serverless-observability-workshop/code/sample-app/src/lib file, you can understand that the log structure that our metric entries will start with MONITORING, which will be filtered together with the REPORT log entry of every function execution to capture further operational data points.

Deploy the Log Processing Utility

  1. Go back to your Cloud9 environment and open a new terminal.
cd ~/environment/serverless-observability-workshop/code/log-processing
sam deploy -g
  1. Enter the following settings when prompted:
        Setting default arguments for 'sam deploy'
        =========================================
        Stack Name [sam-app]: log-processing
        AWS Region [us-east-1]: 
        #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
        Confirm changes before deploy [y/N]: Y
        #SAM needs permission to be able to create roles to connect to the resources in your template
        Allow SAM CLI IAM role creation [Y/n]: Y
        Save arguments to samconfig.toml [Y/n]: Y
  1. Wait a few minutes until the changeset is created and then CANCEL its deployment.

Changeset created successfully. arn:aws:cloudformation:us-east-1:1234567890:changeSet/samcli-deploy1597269838/0752490d-33a9-4995-ae17-4ccca3efbf5d


Previewing CloudFormation changeset before deployment
======================================================
Deploy this changeset? [y/N]: N
  1. Edit the serverless-observability-workshop/code/log-processing/samconfig.toml file and modify the capabilities parameter to give additional permissions to CloudFormation deploy resources on your behalf:
capabilities = "CAPABILITY_IAM CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND"
  1. Save it and run the sam deploy command once again without the -g option. Confirm the deployment and wait a few minutes until it completes.
cd ~/environment/serverless-observability-workshop/code/log-processing
sam deploy


Changeset created successfully. arn:aws:cloudformation:us-east-1:1234567890:changeSet/samcli-deploy1597269838/0752490d-33a9-4995-ae17-4ccca3efbf5d


Previewing CloudFormation changeset before deployment
======================================================
Deploy this changeset? [y/N]: Y