Lambda provides for powerful realtime event based code execution in response to infrastructure and application behavior. A number of different Amazon services can be used as event sources.
CloudWatch Events (CWE) is a general event bus for AWS infrastructure. Currently, it covers several major sources of information:
- CloudTrail API calls over a poll period on CloudTrail delivery,
- real-time instance status events,
- autoscale group notifications, and
- scheduled/periodic events.
CloudTrail provides a very rich data source over the entire range of AWS services exposed via the audit trail that allows Custodian to define effective realtime policies against any AWS product. Additionally, for EC2 instances we can provide mandatory policy compliance - this means the non-compliant resources never became available.
Cloud Custodian Integration¶
Custodian provides for policy level execution against any CWE event stream. Each Custodian policy can be deployed as an independent Lambda function. The only difference between a Custodian policy that runs in Lambda and one that runs directly from the CLI in poll mode is the specification of the subscription of the events in the mode config block of the policy.
Internally Custodian will reconstitute current state for all the resources in the event, execute the policy against them, match against the policy filters, and apply the policy actions to matching resources.
CloudTrail API Calls¶
Lambdas can receive CWE over CloudTrail API calls with delay of 90s at P99.
policies: - name: ec2-tag-running resource: ec2 mode: type: cloudtrail events: - RunInstances actions: - type: mark tag: foo msg: bar
Because the total AWS API surface area is so large most CloudTrail API event subscriptions need two additional fields:
- For CloudTrail events we need to reference the source API call.
- To work transparently with existing resource policies, we also need to specify how to extract the resource IDs from the event via JMESPath so that the resources can be queried.
For very common API calls for policies, some shortcuts
have been defined to allow for easier policy writing as for the
RunInstances API call above, which expands to:
events: - source: ec2.amazonaws.com event: RunInstances ids: "responseElements.instancesSet.items.instanceId"
EC2 Instance State Events¶
Lambdas can receive EC2 instance state events in real time (seconds delay).
policies: - name: ec2-require-encrypted-volumes resource: ec2 mode: type: ec2-instance-state events: - pending filters: - type: ebs key: Encrypted value: False actions: - mark - terminate
AWS Config rules allow you to invoke logic in response to configuration changes in your AWS environment, and Cloud Custodian is the easiest way to write and provision Config rules. Delay here is typically 1-15m (though the SLA on tag-only changes is a bit higher).
In this section we’ll look at how we would deploy the quickstart example using Config. Before you proceed, make sure you’ve
Custodian tag from any EC2 instance left over from the
custodian.yml to specify a mode type of
You’ll also need the ARN of an IAM role to assume when running the Lambda that
Custodian is going to install for you. Sensible policies to add to that role would be
AWSConfigRulesExecutionRole, on top of any permissions
your lambda is going to need to perform the actions you want it to perform.
policies: - name: my-first-policy mode: type: config-rule role: arn:aws:iam::123456789012:role/some-role resource: ec2 filters: - "tag:Custodian": present actions: - stop
Now deploy the policy:
custodian run -s . custodian.yml
That should give you log output like this:
2017-01-25 05:43:01,539: custodian.policy:INFO Provisioning policy lambda my-first-policy 2017-01-25 05:43:04,683: custodian.lambda:INFO Publishing custodian policy lambda function custodian-my-first-policy
Go check the AWS console to see the Lambda as well as the Config rule that
Custodian created. The Config rule should be listed as “Compliant” or “No
results reported” (if not, be sure you removed the
Custodian tag from any
instance left over from the quickstart).
Now for the fun part! With your new policy installed, go ahead and create an
EC2 instance with a
Custodian tag (any non-empty value), and wait (events
from Config are effectively delayed 15m up to 6hrs on tag changes). If all goes
well, you should eventually see that your new custom Config rule notices the
EC2 instance with the
Custodian tag, and stops it according to your policy.
Congratulations! You have now installed your policy to run under Config rather than from your command line.
When running in Lambda you may want policy execution to run using particular options corresponding to those passed to the custodian CLI.
Execution in lambda comes with a default set of configuration which is different from the defaults you might set when running through the command line:
- Metrics are enabled
- Output dir is set to a random /tmp/ directory
- Caching of AWS resource state is disabled
- Account ID is automatically set with info from sts
- Region is automatically set to the region of the lambda (using the AWS_DEFAULT_REGION environment variable in lambda)
When you want to override these settings, you must set ‘execution-options’ with one of the following keys:
One useful thing we can do with these options is to make a policy execute in a different account using assume_role. A policy definition for this looks like:
policies: - name: my-first-policy-cross-account mode: type: periodic schedule: "rate(1 day)" role: arn:aws:iam::123456789012:role/lambda-role execution-options: assume_role: arn:aws:iam::210987654321:role/target-role metrics_enabled: false resource: ec2 filters: - "tag:Custodian": present actions: - stop
A couple of things to note here:
- Metrics are pushed using the assumed role which may or may not be desired
- The mode must be periodic as there are restrictions on where policy
executions can run according to the mode:
Config: May run in a different region but not cross-account Event: Only run in the same region and account Periodic: May run in a different region and different account