In this post I’ll show how you can build a secure virtualized infrastructure for a basic webapp. We will break the setup into VMs that provide isolated services. You can find below the infrastructure diagram. The followings steps will show how you can set up a bare-metal server running Debian Wheezy to act as a KVM hypervisor and the process of deploying and configuring the VMs and the services they are running.
Install kvm and tools:
Install openvswitch :
Build Open vSwitch datapath kernel module:
The management IP address of the hypervisor and the other public IP addresses are assigned on the same interface by the hosting provider. In order to provide Internet connectivity for the VMs we need to create a bridge containing the physical interface where the public IPs are routed and add the VMs ports to this bridge. The trouble is that since this is also the management link we’ll lose connectivity after adding the physical interface to the bridge. After this operation we need to assign the management IP address to the bridge interface. For doing this we’ll edit the /etc/network/interfaces file.
Add openvswitch bridges:
Edit the /etc/network/interfaces file:
At boot time the openvswitch daemon is started after the network init script so when the network init script is run it won’t find the sw-net interface defined in /etc/network/interfaces file. A dirty workaround for this is to re-run the network init script after all the services are loaded. In order to do this we need to edit the /etc/rc.local file:
Now let’s add the physical interface to the bridge. After this we should either restart the network service from the console or do a hard reset:
The next step is to add the second bridge, where the internal network ports will be connected
I prefer using virt-install for new VMs provisioning. The problem with it is that it currently doesn’t support Open vSwitch bridges so we’ll need to adjust it a little by adding the following line to the /usr/lib/pymodules/python2.7/virtinst/VirtualNetworkInterface.py file. This will add the virtualport tag to the VM xml definition:
Now that we have the networking ready the last thing that we need are the storage files that the VMs will use. For creating the files we use the qemu-img utility. I prefer qcow2 files as they provide thin provision and snapshot capabilities. /var/lib/libvirt/images is the default directory used by libvirt so let’s create the storage files here:
We can now create the VM and start the OS installation. We’ll first install the rtr01 VM as it will provide Internet connectivity for the rest of the VMs in the internal network. The following command will generate a VM called rtr01 with 4 vCPUs, 4GB of ram, storage file located at /var/lib/libvirt/images/rtr01.qcow2 and 2 network interfaces - one in the bridge connected to the Internet and another connected to the internal network, the console is presented over VNC and it will first boot from the cdrom device loaded from the /var/lib/libvirt/images/vyatta-livecd_VC6.6R1_amd64.iso file. The disk and network interface will use paravirtualized drivers to obtain increased I/O performance.
After this command is issued a console windows will pop up and it will prompt the cdrom installation. After finishing the installation we can proceed to configuring the device:
After completing these steps we should have a working router, firewall and VPN server.
Now let’s continue with creating the second VM. We’ll do a network install from minimal CD. First create the storage file:
Next we can start the installation process by using the cdrom file located at /var/lib/libvirt/images/debian-7.5.0-amd64-netinst.iso
After completing the OS installation we have a fresh running Debian Wheezy system. We don’t want to repeat the install process for the other files so we’ll just copy the existing image of the Debian system and modify the IP settings and hostnames. We first copy the base image, then attach it by using qemu-nbd, mount the partition where the file system resides and then edit the files that we need.
We repeat the steps above for the db01.qcow2 file.
Let’s now create the web01 and db01 VMs. Since we already have the base storage files we don’t need to run the OS installation:
Once we have booted al the VMs let’s start configuring the services.
On the http load balancer we’ll install varnish and configure the web server as backend:
On the web server we’ll install nginx and php-fpm and configure the default vhost:
Add the following location block to the first server block:
Create an index file in the document root that will query the database server:
On the database server we’ll install mysql server and create a dummy database and table;
After this final step we have our setup ready and http://app.nullzero.me/ should show the Hello World!