At the OpenStack summit in Sydney, I ran a workshop about writing a Tempest plugin for cross-service integration testing.
This is the first of a series of posts where I will cover the material from the workshop. To make this more practical for people to use, I will also include examples on how to integrate this kind of test in an OpenStack CI job. I expect to split this into four posts, roughly covering the following topics:
- Introduction and how to set up the test environment using Devstack (this post)
- How to create a Tempest plugin from scratch
- How to write a Tempest scenario test for cross-service integration testing
- Writing an OpenStack CI job in Zuul V3 native format
This series of posts assume you have a basic familiarity with OpenStack testing infrastructure and Tempest.
The infra user manual and Tempest docs can provide more background information.
Introduction
As part of the “Technical Committee Vision for 2019”, the OpenStack TC introduced the idea of “constellations”, which are reference sets of OpenStack projects, defined to help users get started with OpenStack.
The idea of the workshop was to demonstrate what Tempest plugins are and how they can be used and combined together to test specific OpenStack constellation. Since a lot of (testing) code starts as copy/paste from an example, I wanted to have a good reference example for test developers to re-use.
Scenario Description
I set-up a fictitious constellation for users who want to use the compute service with DNS. It includes the standard compute set: keystone, nova, neutron, glance and cinder, plus Designate for DNS services and Heat for Orchestration.
Nova, Neutron and Designate integration is well documented, however, there are no integration tests written to validate it. Using this constellation would give me the opportunity to build the workshop demo and also write something actually useful.
Setting up the test environment
The first thing I needed was a cloud running all the services I wanted, so I could fire tests against it. I needed the smallest possible footprint so that before the workshop I could spawn one cloud for each workshop attendee. I decided to use devstack, which is OpenStack development and test environment. I chose OpenStack version Pike, which at the time of writing was the latest stable OpenStack release.
Configuring devstack for my purposes was easy enough, using a combination of Neutron and Designate configuration.
What I did was:
- Disable Horizon and Swift to reduce the footprint
- Enable heat and designate by loading the corresponding devstack plugins
- Configure the DNS driver and DNS domain in Neutron
- Set Designate specific settings in Neutron
Which resulted in the following changes:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
# Use bind as backend for Designate DESIGNATE_BACKEND_DRIVER: bind9 # Disable swift and horizon disable_service horizon disable_service s-proxy s-object s-container s-account # Enable Heat enable_plugin heat https://git.openstack.org/openstack/heat # Enable Designate enable_plugin designate https://git.openstack.org/openstack/designate [[post-config|$NEUTRON_CONF]] [DEFAULT] dns_domain=my-workshop-domain.org. external_dns_driver=designate [designate] url=http://$SERVICE_HOST:9001/v2 admin_auth_url=http://$SERVICE_HOST/identity admin_username=admin admin_password=*** admin_tenant_name=admin allow_reverse_dns_lookup=True ipv4_ptr_zone_prefix_size=24 ipv6_ptr_zone_prefix_size=116 [[post-config|/$Q_PLUGIN_CONF_FILE]] [ml2] extension_drivers=port_security,dns_domain_ports [[post-config|$Q_L3_CONF_FILE]] [DEFAULT] router_delete_namespaces=True [[post-config|$Q_DHCP_CONF_FILE]] [DEFAULT] dhcp_delete_namespaces=True |
The only issue I encountered was with the Designate backend. The version of PowerDNS on Ubuntu Xenial does not match the existing driver, so it cannot be used until the driver is re-written. The default has since been changed to bind9 on stable/pike, so the extra configuration change should not be needed anymore.
The full configuration file is available here.
With devstack configuration ready, to stand up a test cloud I only need to run stack.sh. Configuration settings are then propagated into service configuration files as well as test configuration files (where relevant). The devstack plugin interface defines a test-config interface that can be used to configure the corresponding plugin or even core Tempest settings.
Create a workshop test image
I wanted the devstack setup to be easily reproducible so I could easily rebuild the test environment at any time. I also wanted to give workshop attendees an easy way to set up the test environment and play re-run the workshop if they wanted to.
I wrote the automation in Ansible – the full code is available on GitHub.
The playbook provisions a virtual machine (VM) in an OpenStack cloud using Ansible OpenStack cloud modules. It then adds the created VM to ansible inventory dynamically:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
--- - hosts: localhost tasks: - name: Delete existing workshop VM os_server: cloud: "{{ cloud_profile }}" name: workshop_base state: absent when: force_new_vm - name: Setup a VM to create the workshop image os_server: state: present cloud: "{{ cloud_profile }}" name: workshop_base image: "{{ base_image_name }}" key_name: "{{ cloud_keyname }}" timeout: 200 flavor: "{{ flavor_name }}" register: vm - name: Add the VM to the inventory add_host: name: "{{ vm.openstack.interface_ip }}" group: cloud - name: Wait for the VM to become ssh-able wait_for: port: 22 host: "{{ vm.openstack.interface_ip }}" search_regex: OpenSSH delay: 10 sleep: 5 timeout: 120 |
Once the VM is provisioned and ssh-able, it runs a number of roles against it to prepare it for Devstack and to prefetch relevant repositories. The script “stack.sh” will be executed at the first boot of the VM thanks to a systemd unit file:
|
1 2 3 4 5 6 7 8 9 10 11 |
[Unit] Description=A one shot service - run stack.sh as stack at boot [Service] Type=oneshot RemainAfterExit=yes ExecStart={{ BASE }}/workshop/stack_service.sh start ExecStop={{ BASE }}/workshop/stack_service.sh stop [Install] WantedBy=multi-user.target |
Finally, a new snapshot is taken of the VM and the old snapshot (if any) is deleted.
Reconfiguring the test environment
I wanted people attending the workshop to be able to configure their own DNS domain for the exercise. However, running stack.sh is time-consuming (20+ minutes) and I did not want to spend that time during the workshop. To solve this I asked attendees to change the configuration directly in Neutron:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
# Edit the config file vim /etc/neutron/neutron.conf # Set dns_domain. The new domain must end with a '.' # Save and close. Restart all neutron services sudo systemctl restart devstack@q-* # check the status of a service, you can: sudo systemctl status devstack@q-svc # check logs, use journactl. `-a` for whole logs, `-f` to tail: sudo journalctl [-a|-f] --unit devstack@q-* # Setup the openstack client to use the test cloud export OS_CLOUD=devstack openstack image list # Perform admin operations openstack --os-cloud devstack-admin service list |
In the next blog post I will explain how to create a new Tempest plugin.