Developer Guide

Cloud Custodian is a Python application and supports Python 3 on MacOS, Linux, and Windows. It is recommended using Python 3.7 or higher.

Run the following commands in the root directory after cloning Cloud Custodian:

make install
source bin/activate

This creates a virtual env in your enlistment and installs all packages as editable.

Now you may run custodian with any flags in order to directly test changes to the source files. For example, custodian schema aws.<resource_type> will return schema for resource type.

Adding New AWS Resources

3rd Party Documentation

John Lonergan’s guide to adding an extension.<https://gist.github.com/Johnlon/98e1e4b9aaedfedead428c89e7dab5bb>

Create New AWS Resource

Each class definition will use the @resources.register('<resource_name>') decorator to register that class as a Custodian resource substituting <resource_name> with the new resource name. The name specified in the decorator is how the resource will be referenced within policies.

Register the new resource: @resources.register(‘<resource_name>’)

An outer class defining the reference in resource mapping: class <resource_type>(query.QueryResourceManager)

Interior class that defines the aws metadata for resource class resource_type(query.TypeInfo):

class c7n.query.TypeInfo[source]

Resource Type Metadata

Required

Parameters:

  • id – For resource types that use QueryResourceManager this field names the field in the enum_spec response that contains the identifier to use in calls to other API’s of this service. Therefore, this “id” field might be the “arn” field for some API’s but in other API’s it’s a name or other identifier value.

  • name – Defines the name of a field in the resource that contains the “name” of the resource for report purposes. This name value appears in the “report” command output. By default, the id field is automatically included in the report and if name and id fields are the same field name then it’s only shown once. example: custodian report –format csv -s . my-policy.yml

  • service – Which aws service (per sdk) has the api for this resource. See the “client” info for each service in the boto documentation. https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/index.html #noqa

  • enum_spec –

    Defines the boto3 call used to find at least basic details on all resource instances of the relevant type. The data per resource can be further enriched by a supplying a detail_spec function.

    enum_spec is also used when we’ve received an event in which case the results from enum_spec are filtered to include only those identified by the event. If the enum function API allows a filter param to be specified then the filtering can be done on the server side.

    For instance, ASG uses “describe_auto_scaling_groups” as the enum function and “AutoScalingGroupNames” as a filter param to that function so the API returns only relevant resources. However, it seems that most Cloud Custodian integrations do not use this approach. App mesh list_meshes for instance doesn’t support filtering …

    https://boto3.amazonaws.com/v1/documentation/reference/services/appmesh/client/list_meshes.html

    However, if the enum op doesn’t support filtering then the enum op must return all instances of the resource and cloud custodian will perform client side filtering.

    Params to the enum_spec: - enum_op - the aws api operation - path - JMESPATH path to the field in the response that is the collection of result objects - extra_args - optional eg {‘maxResults’: 100}

Permissions - Optional

Parameters:

  • permission_prefix – Permission string prefix if not service

  • permissions_enum – Permissions for resource enumeration/get. Normally we autogen but in some cases we need to specify statically

  • permissions_augment – Permissions for resource augment

Arn handling / generation metadata - Optional

Parameters:

  • arn –

    Defines a field in the resource definition that contains the ARN value, when the resource has an ARM..

    This value is accessed used by the ‘get_arns(..)’ fn on the super-class QueryResourceManager. This value must be a simple field name and cannot be a path.

    If this value is not defined then ‘get_arns’ contains fallback logic. - First fallback logic is to look at what’s defined in the ‘id’ field of the resource. If the value of the “id” field starts with “arn:” then that value is used as the arn. - Otherwise, an attempt at generating (guessing!) the ARN by assembling it from various fields and runtime values based on a recipe defined in ‘generate_arn()’ on the super-class QueryResourceManager.

    If you aren’t going to define the “arn” field and can’t rely on the “id” to be an ARN then you might get lucky that “generate_arn” works for your resource type. However, failing that then you should override “get_arns” function entirely and implement your own logic.

    Testing: Whatever approach you use (above) you REALLY SHOULD (!!!) include a unit test that verifies that “get_arns” yields the right shape of ARNs for your resources. This test should be implemented as an additional assertion within the unit tests you’ll be already planning to write.

  • arn_type – Type, used for arn construction. also required for universal tag augment Only required when you are NOT providing the ARN value directly via the “arn” cfg field. When arn is not provided then QueryResourceManager.generate_arn uses the arn_type value, plus other fields, to construct an ARN; basically, a best guess but not 100% reliable. If generate_arn() isn’t good enough for your needs then you should override the QueryResourceManager.get_arn() function and do it yourself.

  • arn_separator – How arn type is separated from rest of arn

  • arn_service – For services that need custom labeling for arns

Resource retrieval - Optional

Parameters:

  • filter_name – When fetching a single resource via enum_spec this is technically optional, but effectively required for serverless event policies else we have to enumerate the population

  • filter_type – filter_type, scalar or list

  • detail_spec –

    Used to enrich the resource descriptions returned by enum_spec. In many cases the enum_spec function is one of the describe style functions that return a fullish spec that is sufficient for the user policy. However, in other cases the enum_spec is a list style function then the response to then enum call will be lacking in detail and might even just be a list of id’s. In these cases it is generally necessary to define a “detail_spec” function that may be called for each id returned by the enum_spec which can be used to enrich the values provided by the enum_spec.

    Params to the detail_spec: - detail_op - the boto api call name - param_name - name of the identifier argument in the boto api call - param_key - name of field in enum_spec response tha that will be pushed into the identifier argument of the boto api call. - detail_path - path to extract from the boto response and merge into the resource model. if not provided then whole response is merged into the results

  • batch_detail_spec – Used when the api supports getting resource details enmasse

Misc - Optional

Parameters:

  • default_report_fields – Used for reporting, array of fields

  • date – Latest date associated to resource, generally references either create date or modified date. If this field is defined then it will appear in report output such as you would get from …. example: custodian report –format csv -s . my-policy.yml

  • dimension – Defines that resource has cloud watch metrics and the resource id can be passed as this value. Further customizations of dimensions require subclass metrics filter

  • cfn_type – AWS Cloudformation type. See https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-template-resource-type-ref.html

  • config_type – AWS Config Service resource type name. See https://docs.aws.amazon.com/config/latest/developerguide/resource-config-reference.html Typically cfn_type and config_type will have the sane value, but there are some exceptions, so check. The constants defined will be verified by the PolicyMetaLint tests during the build.

  • config_id – Resource attribute that maps to the resourceId field in AWS Config. Intended for resources which use one ID attribute for service API calls and a different one for AWS Config (example: IAM resources).

  • universal_taggable – Determined whether resource group tagging will be used to augment the resource model, in which case we’ll automatically register tag actions/filters. Note: - values of False will disable tag filters/actions, - values of True will register legacy tag filters/actions, - values of object() will just register current standard tag/filters/actions.

  • global_resource – Denotes if this resource exists across all regions (iam, cloudfront, r53)

  • metrics_namespace – Generally we utilize a service to namespace mapping in the metrics filter. However, some resources have a type specific namespace (ig. ebs)

  • id_prefix – Specific to ec2 service resources used to disambiguate a resource by its id

An example that adds a new resource:

@resources.register('scaling-policies')
class ScalingPolicies(query.QueryResourceManager):

    # interior class that defines the aws metadata for resource
    class resource_type(query.TypeInfo):
        service = 'autoscaling'
        arn_type = "scalingPolicy"
        id = name = 'PolicyName'
        date = 'CreatedTime'

        # this defines the boto3 call for the resource as well as JMESPATH
        # for accessing TL resources
        enum_spec = (
            'describe_policies', 'ScalingPolicies', None
        )
        filter_name = 'PolicyNames'
        filter_type = 'list'
        cfn_type = config_type = 'AWS::AutoScaling::ScalingPolicy'

Load New AWS Resource

If you created a new module for an AWS service (i.e. this was the first resource implemented for this service in Custodian), then import the new service module in resource_map.py:

"aws.<name of resource>": "c7n.resources.<name of file>.<name of resource class>"

Add New Filter

A filter can be added with a decorator and class:

@<New-resource-class>.filter_registry.register('<filter-name>')

class <NewFilterName>(ValueFilter)

An example that adds a new filter for scaling policies to the ASG resource:

@ASG.filter_registry.register('scaling-policies')
class ScalingPoliciesFilter(ValueFilter):
    schema = type_schema(
        'scaling-policies', rinherit=ValueFilter.schema
    )
    schema_alias = False
    permissions = ("autoscaling:DescribePolicies",)

    def process(self, asgs, event=None):
        self.policy_info = PolicyInfo(self.manager).initialize(asgs)
        return super(ScalingPoliciesFilter, self).process(asgs, event)

    def __call__(self, asg):

        asg_policies = self.policy_info.get(asg)
        matched = False
        if asg_policies is not None:
            for policy in asg_policies:
                matched = self.match(policy) or matched
        return matched

Add New Action

An action can be added with a decorator and class:

@<New-resource-class>.action_registry.register('<action-name>')

class <NewActionName>(Action)

An example that adds a new action for deleting to the ASG resource:

@ASG.action_registry.register('delete')
class Delete(Action):

    schema = type_schema('delete', force={'type': 'boolean'})
    permissions = ("autoscaling:DeleteAutoScalingGroup",)

    def process(self, asgs):
        client = local_session(
            self.manager.session_factory).client('autoscaling')
        for asg in asgs:
            self.process_asg(client, asg)

    def process_asg(self, client, asg):
        force_delete = self.data.get('force', False)
        try:
            self.manager.retry(
                client.delete_auto_scaling_group,
                AutoScalingGroupName=asg['AutoScalingGroupName'],
                ForceDelete=force_delete)
        except ClientError as e:
            if e.response['Error']['Code'] == 'ValidationError':
                return
            raise

Testing

For information regarding testing see testing for developers.