kitchen-azurerm

kitchen-azurerm is a driver for the popular test harness Test Kitchen that allows Microsoft Azure resources to be provisioned before testing. This driver uses the new Microsoft Azure Resource Management REST API via the azure-sdk-for-ruby.

This version has been tested on Windows, macOS, and Ubuntu. If you encounter a problem on your platform, please raise an issue.

This plugin ships in Chef Workstation out of the box so there is no need to install it when using Chef Workstation.

If you're not using Chef Workstation and need to install the plugin as a gem run:

gem install kitchen-azurerm

For the driver to interact with the Microsoft Azure Resource Management REST API, you need to configure a Service Principal with Contributor rights for a specific subscription. Using an Organizational (AAD) account and related password is no longer supported. To create a Service Principal and apply the correct permissions, see the create an Azure service principal with the Azure CLI and the Azure CLI documentation. Make sure you stay within the section titled 'Password-based authentication'.

If the above is TLDR then try this after az login using your target subscription ID and the desired SP name:

# Create a Service Principal using the desired subscription id from the command above
az ad sp create-for-rbac --name="kitchen-azurerm" --role="Contributor" --scopes="/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"

#Output
#
#{
#  "appId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",    <- Also known as the Client ID
#  "displayName": "azure-cli-2018-12-12-14-15-39",
#  "name": "http://azure-cli-2018-12-12-14-15-39",
#  "password": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
#  "tenant": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
#}

NOTE: Don't forget to save the values from the output -- most importantly the password.

You will also need to ensure you have an active Azure subscription (you can get started for free or use your MSDN Subscription).

You are now ready to configure kitchen-azurerm to use the credentials from the service principal you created above. You will use four elements from the output:

1. Subscription ID: available from the Azure portal
2. Client ID: the appId value from the output.
3. Client Secret/Password: the password from the output.
4. Tenant ID: the tenant from the output.

Using a text editor, open or create the file ~/.azure/credentials and add the following section, noting there is one section per Subscription ID. Make sure you save the file with UTF-8 encoding

[ADD-YOUR-AZURE-SUBSCRIPTION-ID-HERE-IN-SQUARE-BRACKET]
client_id = "your-azure-client-id-here"
client_secret = "your-client-secret-here"
tenant_id = "your-azure-tenant-id-here"

If preferred, you may also set the following environment variables, however this would be incompatible with supporting multiple Azure subscriptions.

AZURE_CLIENT_ID="your-azure-client-id-here"
AZURE_CLIENT_SECRET="your-client-secret-here"
AZURE_TENANT_ID="your-azure-tenant-id-here"

Note that the environment variables, if set, take preference over the values in a configuration file.

After adjusting your ~/.azure/credentials file you will need to adjust your kitchen.yml file to leverage the azurerm driver. Use the following examples to achieve this, then check your configuration with standard kitchen commands. For example,

% kitchen list
Instance            Driver   Provisioner  Verifier  Transport  Last Action    Last Error
wsus-windows-2019   Azurerm  ChefZero     Inspec    Winrm      <Not Created>  <None>
wsus-windows-2016   Azurerm  ChefZero     Inspec    Winrm      <Not Created>  <None>

See the kitchen.ci kitchen-azurem docs for a complete list of configuration options.

Here's an example kitchen.yml file that provisions an Ubuntu Server, using Chef Zero as the provisioner and SSH as the transport. Note that if the key does not exist at the specified location, it will be created. Also note that if ssh_key is supplied, Test Kitchen will use this in preference to any default/configured passwords that are supplied.

---
driver:
  name: azurerm
  subscription_id: 'your-azure-subscription-id-here'
  location: 'West Europe'
  machine_size: 'Standard_D1'

transport:
  ssh_key: ~/.ssh/id_kitchen-azurerm

provisioner:
  name: chef_zero

platforms:
  - name: ubuntu-14.04
    driver:
      image_urn: Canonical:UbuntuServer:14.04.4-LTS:latest
      vm_name: trusty-vm

suites:
  - name: default
    attributes:

Concurrent execution of create/converge/destroy is supported via the --concurrency parameter. Each machine is created in its own Azure Resource Group so it has no shared lifecycle with the other machines in the test run. To take advantage of parallel execution use the following command:

kitchen test --concurrency <n>

Where n is the number of threads to create. Note that any failure (e.g. an AzureOperationError) will cause the whole test to fail, though resources already in creation will continue to be created.

Here's a further example kitchen.yml file that will provision a Windows Server 2019 [smalldisk] instance, using WinRM as the transport. An ephemeral os disk is used. The resource created in Azure will enable itself for remote access at deployment time (it does this by customizing the machine at provisioning time) and tags the Azure Resource Group with metadata using the resource_group_tags property. Notice that the vm_tags and resource_group_tags properties use a simple key : value structure per line:

---
driver:
  name: azurerm
  subscription_id: 'your-subscription-id-here'
  location: 'West Europe'
  machine_size: 'Standard_DS2_v2'

provisioner:
  name: chef_zero

platforms:
  - name: windows2019
    driver:
      image_urn: MicrosoftWindowsServer:WindowsServer:2019-Datacenter-smalldisk:latest
      use_ephemeral_osdisk: true
      resource_group_tags:
        project: 'My Cool Project'
        contact: 'me@somewhere.com'
      vm_tags:
        my_tag: its value
        another_tag: its awesome value
    transport:
      name: winrm
suites:
  - name: default
    attributes:

The following example introduces the pre_deployment_template and pre_deployment_parameters properties in the configuration file. You can use this capability to execute an ARM template containing Azure resources to provision before the system under test is created.

In the example the ARM template in the file predeploy.json would be executed with the parameters that are specified under pre_deployment_parameters. These resources will be created in the same Azure Resource Group as the VM under test, and therefore will be destroyed when you type kitchen destroy.

---
driver:
  name: azurerm
  subscription_id: 'your-azure-subscription-id-here'
  location: 'West Europe'
  machine_size: 'Standard_D1'
  pre_deployment_template: predeploy.json
  pre_deployment_parameters:
    test_parameter: 'This is a test.'

transport:
  ssh_key: ~/.ssh/id_kitchen-azurerm

provisioner:
  name: chef_zero

platforms:
  - name: ubuntu-1404
    driver:
      image_urn: Canonical:UbuntuServer:14.04.4-LTS:latest

suites:
  - name: default
    run_list:
      - recipe[kitchen-azurerm-demo::default]
    attributes:

Example predeploy.json:

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
      "test_parameter": {
        "type": "string",
        "defaultValue": ""
      }
  },
  "variables": {

  },
  "resources": [
      {
        "name": "uniqueinstancenamehere01",
        "type": "Microsoft.Sql/servers",
        "location": "[resourceGroup().location]",
        "apiVersion": "2014-04-01-preview",
        "properties": {
          "version": "12.0",
          "administratorLogin": "azure",
          "administratorLoginPassword": "P2ssw0rd"
        }
      }
  ],
  "outputs": {
      "parameter testing": {
        "type": "string",
        "value": "[parameters('test_parameter')]"
      }
  }
}

The following example introduces the vnet_id and subnet_id properties under "driver" in the configuration file. This can be applied at the top level, or per platform. You can use this capability to create the VM on an existing virtual network and subnet created in a different resource group.

In this case, the public IP address is not used unless public_ip is set to true

---
driver:
  name: azurerm
  subscription_id: 'your-azure-subscription-id-here'
  location: 'West Europe'
  machine_size: 'Standard_D1'

transport:
  ssh_key: ~/.ssh/id_kitchen-azurerm

provisioner:
  name: chef_zero

platforms:
  - name: ubuntu-1404
    driver:
      image_urn: Canonical:UbuntuServer:14.04.4-LTS:latest
      vnet_id: /subscriptions/b6e7eee9-YOUR-GUID-HERE-03ab624df016/resourceGroups/pendrica-infrastructure/providers/Microsoft.Network/virtualNetworks/pendrica-arm-vnet
      subnet_id: subnet-10.1.0

suites:
  - name: default
    attributes:

The following example introduces the vnet_id and subnet_id properties under "driver" in the configuration file. This can be applied at the top level, or per platform. You can use this capability to create the VM on an existing virtual network and subnet created in a different resource group.

This enables scenarios that require a Standard SKU public IP resource, for example when a NAT gateway is present on the target subnet.

---
driver:
  name: azurerm
  subscription_id: 'your-azure-subscription-id-here'
  location: 'West Europe'
  machine_size: 'Standard_D1'

transport:
  ssh_key: ~/.ssh/id_kitchen-azurerm

provisioner:
  name: chef_zero

platforms:
  - name: ubuntu-1404
    driver:
      image_urn: Canonical:UbuntuServer:14.04.4-LTS:latest
      vnet_id: /subscriptions/b6e7eee9-YOUR-GUID-HERE-03ab624df016/resourceGroups/pendrica-infrastructure/providers/Microsoft.Network/virtualNetworks/pendrica-arm-vnet
      subnet_id: subnet-10.1.0
      public_ip: true
      public_ip_sku: Standard

suites:
  - name: default
    attributes:

This example is the same as above, but uses a private managed image to provision the vm.

Note: The image must be available first. On deletion the disk and everything is removed.

---
driver:
  name: azurerm
  subscription_id: 'your-azure-subscription-id-here'
  location: 'West Europe'
  machine_size: 'Standard_D1'

transport:
  ssh_key: ~/.ssh/id_kitchen-azurerm

provisioner:
  name: chef_zero

platforms:
  - name: ubuntu-1404
    driver:
      image_id: /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/RESGROUP/providers/Microsoft.Compute/images/IMAGENAME
      vnet_id: /subscriptions/b6e7eee9-YOUR-GUID-HERE-03ab624df016/resourceGroups/pendrica-infrastructure/providers/Microsoft.Network/virtualNetworks/pendrica-arm-vnet
      subnet_id: subnet-10.1.0
      use_managed_disk: true

suites:
  - name: default
    attributes:

This example a classic Custom VM Image (aka a VHD file) is used. As the Image VHD must be in the same storage account then the disk of the instance, the os disk is created in an existing image account.

Note: When the resource group ís deleted, the os disk is left in the existing storage account blob. You must clean up manually.

This example will:

• use the customized image https://yourstorageaccount.blob.core.windows.net/system/Microsoft.Compute/Images/images/Cent7_P4-osDisk.170dd1b7-7dc3-4496-b248-f47c49f63965.vhd (can be built with packer)
• set the disk url of the vm to https://yourstorageaccount.blob.core.windows.net/vhds/osdisk-kitchen-XXXXX.vhd
• set the os type to linux

---
driver:
  name: azurerm
  subscription_id: 'your-azure-subscription-id-here'
  location: 'West Europe'
  machine_size: 'Standard_D1'

transport:
  ssh_key: ~/.ssh/id_kitchen-azurerm

provisioner:
  name: chef_zero

platforms:
  - name: ubuntu-1404
    driver:
      image_url: https://yourstorageaccount.blob.core.windows.net/system/Microsoft.Compute/Images/images/Cent7_P4-osDisk.170dd1b7-7dc3-4496-b248-f47c49f63965.vhd
      existing_storage_account_blob_url: https://yourstorageaccount.blob.core.windows.net
      os_type: linux
      use_managed_disk: false
      vnet_id: /subscriptions/b6e7eee9-YOUR-GUID-HERE-03ab624df016/resourceGroups/pendrica-infrastructure/providers/Microsoft.Network/virtualNetworks/pendrica-arm-vnet
      subnet_id: subnet-10.1.0

suites:
  - name: default
    attributes:

This is the same as above, but uses custom data to customize the instance.

Note: Custom data can be custom data or a file to custom data. Please also note that if you use winrm communication to non-nano windows servers custom data is not supported, as winrm is enabled via custom data.

---
driver:
  name: azurerm
  subscription_id: 'your-azure-subscription-id-here'
  location: 'West Europe'
  machine_size: 'Standard_D1'

transport:
  ssh_key: ~/.ssh/id_kitchen-azurerm

provisioner:
  name: chef_zero

platforms:
  - name: ubuntu-1404
    driver:
      image_url: https://yourstorageaccount.blob.core.windows.net/system/Microsoft.Compute/Images/images/Cent7_P4-osDisk.170dd1b7-7dc3-4496-b248-f47c49f63965.vhd
      existing_storage_account_blob_url: https://yourstorageaccount.blob.core.windows.net
      os_type: linux
      use_managed_disk: false
      vnet_id: /subscriptions/b6e7eee9-YOUR-GUID-HERE-03ab624df016/resourceGroups/pendrica-infrastructure/providers/Microsoft.Network/virtualNetworks/pendrica-arm-vnet
      subnet_id: subnet-10.1.0
      os_disk_size_gb: 100
      #custom_data: /tmp/customdata.txt
      custom_data: |
        #cloud-config
        fqdn: myhostname
        preserve_hostname: false
        runcmd:
          - yum install -y telnet

suites:
  - name: default
    attributes:

This example demonstrates how to add 3 additional Managed data disks to a Windows Server 2016 VM. Not supported with legacy (pre-managed disk) storage accounts.

Note the availability of a format_data_disks option (default: false). When set to true, a PowerShell script will execute at first boot to initialize and format the disks with an NTFS filesystem. This option does not affect Linux machines.

---
driver:
  name: azurerm
  subscription_id: 'your-azure-subscription-id-here'
  location: 'West Europe'
  machine_size: 'Standard_F2s'

provisioner:
  name: chef_zero

platforms:
- name: windows2016-noformat
  driver:
    image_urn: MicrosoftWindowsServer:WindowsServer:2016-Datacenter:latest
    data_disks:
      - lun: 0
        disk_size_gb: 128
      - lun: 1
        disk_size_gb: 128
      - lun: 2
        disk_size_gb: 128
    # format_data_disks: false

suites:
  - name: default
    attributes:

The following example introduces the post_deployment_template and post_deployment_parameters properties in the configuration file. You can use this capability to execute an ARM template containing Azure resources to provision after the system under test is created.

In the example the ARM template in the file postdeploy.json would be executed with the parameters that are specified under post_deployment_parameters. These resources will be created in the same Azure Resource Group as the VM under test, and therefore will be destroyed when you type kitchen destroy.

---
driver:
  name: azurerm
  subscription_id: 'your-azure-subscription-id-here'
  location: 'West Europe'
  machine_size: 'Standard_D1'
  post_deployment_template: postdeploy.json
  post_deployment_parameters:
    test_parameter: 'This is a test.'

transport:
  ssh_key: ~/.ssh/id_kitchen-azurerm

provisioner:
  name: chef_zero

platforms:
  - name: ubuntu-1404
    driver:
      image_urn: Canonical:UbuntuServer:14.04.4-LTS:latest

suites:
  - name: default
    attributes:

Example postdeploy.json to enable MSI extention on VM:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "vmName": {
            "type": "String"
        },
        "location": {
            "type": "String"
        },
        "msiExtensionName": {
            "type": "String"
        }
    },
    "resources": [
        {
            "type": "Microsoft.Compute/virtualMachines",
            "name": "[parameters('vmName')]",
            "apiVersion": "2017-12-01",
            "location": "[parameters('location')]",
            "identity": {
                "type": "systemAssigned"
            }
        },
        {
            "type": "Microsoft.Compute/virtualMachines/extensions",
            "name": "[concat( parameters('vmName'), '/' , parameters('msiExtensionName') )]",
            "apiVersion": "2017-12-01",
            "location": "[parameters('location')]",
            "properties": {
                "publisher": "Microsoft.ManagedIdentity",
                "type": "[parameters('msiExtensionName')]",
                "typeHandlerVersion": "1.0",
                "autoUpgradeMinorVersion": true,
                "settings": {
                    "port": 50342
                }
            },
            "dependsOn": [
                "[concat('Microsoft.Compute/virtualMachines/', parameters('vmName'))]"
            ]
        }
    ]
}

This example demonstrates how to enable a System Assigned Identity and User Assigned Identities on a Kitchen VM. Any combination of System and User assigned identities may be enabled, and multiple User Assigned Identities can be supplied.

See the Managed identities for Azure resources documentation for more information on using Managed Service Identities.

---
driver:
  name: azurerm
  subscription_id: 'your-azure-subscription-id-here'
  location: 'West Europe'
  machine_size: 'Standard_D1'

transport:
  ssh_key: ~/.ssh/id_kitchen-azurerm

provisioner:
  name: chef_zero

platforms:
  - name: ubuntu-1404
    driver:
      image_urn: Canonical:UbuntuServer:14.04.4-LTS:latest
      system_assigned_identity: true
      user_assigned_identities:
        - /subscriptions/4801fa9d-YOUR-GUID-HERE-b265ff49ce21/resourcegroups/test-kitchen-user/providers/Microsoft.ManagedIdentity/userAssignedIdentities/test-kitchen-user

suites:
  - name: default
    attributes:

This following example introduces secret_url, vault_name, and vault_resource_group properties under "driver" in the configuration file. You can use this capability to create a VM with a specified key vault certificate.

---
driver:
  name: azurerm
  subscription_id: 'your-azure-subscription-id-here'
  location: 'CentralUS'
  machine_size: 'Standard_D2s_v3'
  secret_url: 'https://YOUR-SECRET-PATH'
  vault_name: 'YOUR-VAULT-NAME'
  vault_group_name: 'YOUR-VAULT-GROUP-NAME'
transport:
  name: winrm
  elevated: true
provisioner:
  name: chef_zero
platforms:
  - name: win2012R2-sql2016
    driver:
      image_urn: MicrosoftSQLServer:SQL2016SP2-WS2012R2:SQLDEV:latest

suites:
  - name: default
    attributes:

Starting with v0.9.0 this driver has support for Azure Government and Sovereign Clouds via the use of the azure_environment setting. Valid Azure environments are Azure, AzureUSGovernment, AzureChina and AzureGermanCloud

Note that the use_managed_disks option should be set to false until supported by AzureUSGovernment.

---
driver:
  name: azurerm
  subscription_id: 'your-azure-subscription-id-here'
  azure_environment: 'AzureUSGovernment'
  location: 'US Gov Iowa'
  machine_size: 'Standard_D2_v2_Promo'
  use_managed_disks: false

provisioner:
  name: chef_zero

verifier:
  name: inspec

platforms:
- name: ubuntu1604
  driver:
    image_urn: Canonical:UbuntuServer:16.04-LTS:latest
  transport:
    ssh_key: ~/.ssh/id_kitchen-azurerm

suites:
  - name: default

You can use the azure (azure-cli) command line tools to interrogate for the Urn. All 4 parts of the Urn must be specified, though the last part can be changed to "latest" to indicate you always wish to provision the latest operating system and patches.

$ azure vm image list "West Europe" Canonical UbuntuServer

This will return a list like the following, from which you can derive the Urn. this list has been shortened for readability

data:    Publisher  Offer         Sku                Version          Location    Urn
data:    ---------  ------------  -----------------  ---------------  ----------  --------------------------------------------------------
data:    Canonical  UbuntuServer  12.04.5-LTS        12.04.201507301  westeurope  Canonical:UbuntuServer:12.04.5-LTS:12.04.201507301
data:    Canonical  UbuntuServer  12.04.5-LTS        12.04.201507311  westeurope  Canonical:UbuntuServer:12.04.5-LTS:12.04.201507311
data:    Canonical  UbuntuServer  12.04.5-LTS        12.04.201508190  westeurope  Canonical:UbuntuServer:12.04.5-LTS:12.04.201508190
data:    Canonical  UbuntuServer  12.04.5-LTS        12.04.201509060  westeurope  Canonical:UbuntuServer:12.04.5-LTS:12.04.201509060
data:    Canonical  UbuntuServer  12.04.5-LTS        12.04.201509090  westeurope  Canonical:UbuntuServer:12.04.5-LTS:12.04.201509090
data:    Canonical  UbuntuServer  12.10              12.10.201212180  westeurope  Canonical:UbuntuServer:12.10:12.10.201212180
data:    Canonical  UbuntuServer  14.04.3-DAILY-LTS  14.04.201509110  westeurope  Canonical:UbuntuServer:14.04.3-DAILY-LTS:14.04.201509110
data:    Canonical  UbuntuServer  14.04.3-DAILY-LTS  14.04.201509160  westeurope  Canonical:UbuntuServer:14.04.3-DAILY-LTS:14.04.201509160
data:    Canonical  UbuntuServer  14.04.3-DAILY-LTS  14.04.201509220  westeurope  Canonical:UbuntuServer:14.04.3-DAILY-LTS:14.04.201509220
data:    Canonical  UbuntuServer  14.04.3-LTS        14.04.201508050  westeurope  Canonical:UbuntuServer:14.04.3-LTS:14.04.201508050
data:    Canonical  UbuntuServer  14.04.3-LTS        14.04.201509080  westeurope  Canonical:UbuntuServer:14.04.3-LTS:14.04.201509080
data:    Canonical  UbuntuServer  15.04              15.04.201506161  westeurope  Canonical:UbuntuServer:15.04:15.04.201506161
data:    Canonical  UbuntuServer  15.04              15.04.201507070  westeurope  Canonical:UbuntuServer:15.04:15.04.201507070
data:    Canonical  UbuntuServer  15.04              15.04.201507220  westeurope  Canonical:UbuntuServer:15.04:15.04.201507220
data:    Canonical  UbuntuServer  15.04              15.04.201507280  westeurope  Canonical:UbuntuServer:15.04:15.04.201507280
data:    Canonical  UbuntuServer  15.10-DAILY        15.10.201509170  westeurope  Canonical:UbuntuServer:15.10-DAILY:15.10.201509170
data:    Canonical  UbuntuServer  15.10-DAILY        15.10.201509180  westeurope  Canonical:UbuntuServer:15.10-DAILY:15.10.201509180
data:    Canonical  UbuntuServer  15.10-DAILY        15.10.201509190  westeurope  Canonical:UbuntuServer:15.10-DAILY:15.10.201509190
data:    Canonical  UbuntuServer  15.10-DAILY        15.10.201509210  westeurope  Canonical:UbuntuServer:15.10-DAILY:15.10.201509210
data:    Canonical  UbuntuServer  15.10-DAILY        15.10.201509220  westeurope  Canonical:UbuntuServer:15.10-DAILY:15.10.201509220
info:    vm image list command OK

Contributions to the project are welcome via submitting Pull Requests.

1. Fork it ( https://github.com/test-kitchen/kitchen-azurerm/fork )
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request

Stuart Preston

Copyright 2015-2021, Chef Software, Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.