Tutorial - Helm Deployment

This is a step-by-step tutorial for deploying the Azure Container Host in Azure Kubernetes Service (AKS) with the provided helm chart and deployment script. This tutorial makes use of the Azure CLI and helm. Make sure that both are installed and you are logged in to your subscription.

1. Create a Resource Group

This will hold all of the Azure resources created in this tutorial. You could use an existing resource group too.

az group create --name c7n-helm-tutorial --location westus2

2. Create a Storage Account

This storage account will hold all of the blobs and queues used by the container host. You could also use an existing resource here or multiple storage accounts.

We will also create two blob containers. One will host the uploaded policies, and the other will store the custodian output files of running policies.

az storage account create --resource-group c7n-helm-tutorial --name c7nstorage

account_key=$(az storage account keys list --account-name c7nstorage --query "[0].value" --output tsv)
az storage container create --account-name c7nstorage --account-key $account_key --name c7n-helm-policies
az storage container create --account-name c7nstorage --account-key $account_key --name c7n-helm-logs

3. Create a Service Principal

This Service Principal will be used as the identity for the container host. You could also use an existing Service Principal. Remember to save the password to use as the client secret.

az ad sp create-for-rbac --name c7n-helm

This Service Principal will need the proper permissions to interact with the storage account and any other azure resources that the Azure Container Host interacts with. This includes the resources that policies will run against.

The simplest way to grant these permissions is to make the identity a Contributor on the target subscription and a Storage Blob Data Contributor and Storage Queue Data Contributor on the storage account created above.

az role assignment create --assignee <Service Principal Client Id> \
    --role "Contributor" --scope <Target Subscription Resource Id>

az role assignment create --assignee <Service Principal Client Id> \
    --role "Storage Blob Data Contributor" --scope <Storage Account Resource Id>

az role assignment create --assignee <Service Principal Client Id> \
    --role "Storage Queue Data Contributor" --scope <Storage Account Resource Id>

4. Create an Application Insights Instance

We will create an Application Insights instance to gather logs and telemetry generated by the running container host.

# You may need to add the application-insights extension
az extension add -n application-insights

az monitor app-insights component create --resource-group c7n-helm-tutorial --app c7n-helm-insights --location westus2

5. Create an AKS Cluster and Install Tiller

This AKS cluster will be used to host the pods running the Container Host. Tiller will also need to be installed with the proper service account. The provided node count and vm size will create a small cluster and should be adjusted to fit your needs outside this tutorial.

az aks create --resource-group c7n-helm-tutorial --name c7n-helm \
    --node-count 1 --node-vm-size Standard_B2s \
    --service-principal <Service Principal ID> --client-secret <Client Secret>

az aks get-credentials --resource-group c7n-helm-tutorial --name c7n-helm

Once the cluster has been created, we can initialize helm and tiller. First, create the Service Account and Cluster Role Binding for Tiller

# rbac-config.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system

kubectl apply -f rbac-config.yaml

helm init --service-account tiller

6. Deploy the Helm Chart

Now we are ready to deploy the helm chart and the container host. Create a file with the helm configuration values for our container host. See the Container Host Documentation for information on filling out the environment variables.

# helm-values.yaml

defaultEnvironment:
  AZURE_TENANT_ID: "<Azure Tenant ID>"
  AZURE_CLIENT_ID: "<Azure Client ID>"
  AZURE_CONTAINER_POLICY_URI: "<Azure Policy Container URI>"
  AZURE_CONTAINER_STORAGE_RESOURCE_ID: "<Storage Account Resource ID>"
  AZURE_CONTAINER_METRICS: "azure://<App Insights Instrumentation Key>"
  AZURE_CONTAINER_LOG_GROUP: "azure://<App Insights Instrumentation Key>"
  AZURE_CONTAINER_OUTPUT_DIR: "<Azure Logs Container URI>"

subscriptionHosts:
  - name: '<Subscription Name>'
    environment:
      AZURE_SUBSCRIPTION_ID: "<Subscription ID>"

Then deploy the chart with the following command. The client secret should come from creating the Service Principal and must be provided in a base64 encoded format.

helm upgrade --install --debug --force --wait \
    --namespace cloud-custodian --values helm-values.yaml \
    --set defaultSecretEnvironment.AZURE_CLIENT_SECRET=<Base 64 Azure Client Secret> \
    helm-tutorial tools/ops/azure/container-host/chart

To verify that the pod is running:

# Check if pod status is "Running"
kubectl get pods --namespace cloud-custodian

# Watch the logs for the pod
kubectl logs <Pod Name> --namespace cloud-custodian --follow

7. Upload a Custodian Policy

Finally, create a custodian policy called find-c7nstorage.yaml. This policy will just find the storage account we made earlier. We’ll set the mode to run every minute for easier testing.

policies:
- name: find-c7nstorage
  resource: azure.storage
  mode:
    type: container-periodic
    schedule: "* * * * *"  # Run every minute as an example
  filters:
  - type: value
    key: name
    op: eq
    value: c7nstorage

Upload this file to the policy storage container. Within a few minutes, the container host should pick it up and begin executing it.

az storage blob upload --account-name c7nstorage --account-key $account_key \
    --container-name c7n-helm-policies --file find-c7nstorage.yaml --name find-c7nstorage.yaml