Edited by Christopher Douce, Saturday, 7 Jun 2014, 13:21
This blog post is the second of two that summarises an OpenStack conference that I attended on 4 June in London.
This second half of the conference had two parallel sessions. Delegates could either go to the stream that was intended for novices (which is what I did), or go to a more technical session.
I was quite tempted by the technical session, especially by a presentation that was all about what it means to be an OpenStack developer. One of the key points that I did pick up on was that you need to know the Python language to be an OpenStack developer, which is a language that is used within the OU’s data structures and algorithms module, M269 Algorithms, data structures and computability.
Introduction to OpenStack
The first session of the afternoon was by Kevin Jackson who works at Rackspace.
Kevin said that OpenStack and Linux are sometimes spoken about in similar terms. Both can be created from distributions, and both are supported by companies that can offer consultancy support and help to move products forward. ‘OpenStack is like a pile of nuts’, said Kevin, and the nuts represent different components.
So, what are the nuts? Nova is a compute engine, which hosts a virtual machine running in a Hypervisor. I now understand that a hypervisor can host one or more virtual machine. You might have a web server and your application code running within this bit of OpenStack.
Neutron is all about networking. In some respects, Neutron is a virtual network that has been all written in code. There is more about this in later presentations. If you have different parts of an OpenStack implementation, Neutron allows the different bits to talk to each other; it pretends to be a physical network.
Swift is an object store, which is something that was spoken about during an earlier presentation. Despite my earlier description, Swift isn’t really like a traditional file system. Apparently, it can be ‘rack or cabinet aware’, to take account of the design of your physical data centre.
Cinder is another kind of storage; block storage. As mentioned earlier, to all intents and purposes, Cinder looks like a ‘drive’ to a virtual machine. I understand a situation where you might have multiple virtual machines accessing the same block storage device.
Ceilometer is a component that was described as telemetry. This is a block which can apparently say how much bandwidth is being used. (I don’t know how to describe what ‘bandwidth’ is in this instance – does it relate to the network, the available capacity within a VM, or the whole installation? This is a distinct gap in my understanding).
Heat is all about orchestration. Heat monitors ‘the cloud’, or its environment. Kevin said, ‘if it knows all about your environment and suddenly you have two VMs and not three, it creates a third one’. This orchestration piece was described as a recipe for how your system operates.
All these bits and pieces are controlled by a web interface called Horizon, which I assume makes calls to the APIs of each of these components. You can use Horizon to look at the components of the network, for example. I have to confess to being a bit confused about the distinction between JuJu and this standard piece of OpenStack – this is another question that I need to ask myself.
At the end of Kevin’s presentation, I’ve made a note of a question from the floor which was: ‘why should I go open source and not go for a proprietary solution?’ The answer was interesting: you can get locked into a vendor if you choose a proprietary solution. If you use an open source solution, such as OpenStack you can move your ‘cloud’ different providers, say, from Rackspace to HP. With Amazon web services, you’re stuck with using Amazon web services. In some respects, these arguments echo arguments that are given in favour of Linux and other open source products. The most compelling arguments are, of course, freedom and choice.
A further question was, ‘how mainstream is this going to go?’ The answer was, ‘there’s many companies around the globe who are using OpenStack as a solution’, but I think it was also said that OpenStack is just one of many different solutions that exist.
OpenStack and Storage made easy at Lush Cosmetics
The second presentation of the day was made by Jim Liddle who works for a cloud consultancy called Storage Made Easy.
Jim presented a case study about his work with Lush Cosmetics. I’ve made note of a number of important requirements: the data that is stored to the cloud should be encrypted, and there should be ways to help facilitate auditing and governance (of the cloud).
It’s interesting that the subject of governance was explicitly addressed in this case study. The importance of ‘institutional control’ and the need to carry out checks and balances is one of reasons why organisations might choose private clouds over public clouds. In the case of Lush, two key drivers included the cost per user, and the need to keep their data within the UK.
A new TLA that I heard was OVF (Wikipedia), an abbreviation for Open Virtualization Format, and is a way to package virtual machines in a way that is not tied to any particular hypervisor (VM container), or architecture. Other technologies and terms that were referred to included: MySQL, which is touched on in TT284 Web Technologies (OU), Apache, MemCached (Wikipedia) and CentOS.
Deploying Windows Workloads into OpenStack using JuJu
A lot of the presentations had a strong Linux flavour to them. Linux, of course, isn’t the only platform that can be used to power clouds. Alessandro Pilotti from Cloudbase solutions spoke on the connections between Windows and OpenStack.
Terms that cropped up during his presentation included Hyper-V (a hypervisor from Microsoft), KVM (Kernel based virtual machine, which is Linux hypervisor), MaaS (metal as a service, an Ubuntu term), GRE Tunnels (GRE being an abbreviation for Generic Routing Encapsulation), NVGRE (Network Virtualization using Generic Routing Encapsulation), and RDP (Remote Desktop Protocol).
It was all pretty complicated, and even though I’m reasonably technical, this was at a whole other level of detail. Clicking through some of the above links soon takes me into a world of networking and products that are pretty new to me. This clearly suggests that there is a whole lot of ‘new technology’ out there that I need to try to make a bit of sense of, and this, of course, takes time.
Alessandro also treated us to a live video link that showed a set of four small computers that were all hooked up together (I have no idea what these small desktop computers without screens were called; they used to have a special name). The idea was to show LEDs flashing to demonstrate some remote rebooting going on.
This demo didn’t quite work out as planned, but it did get me thinking: to really learn how to do cloud stuff, a good idea would be to spend time actually playing with bits of physical hardware. This way you can understand the relationships between logical and physical architectures. The challenge, of course, is finding the time to get the kit together, and to do the learning.
Using Swift in Entertaining Ways
This presentation was made by a number of people from Sohonet a company that offers cloud services to the film and TV industry. An interesting application of cloud computing is film and video post-production, the part of production where when recordings are digitally edited and manipulated. An interesting challenge is that when it comes to video post-production we’re talking about huge quantities of data, and data that needs to be kept safe and secure.
Sohonet operates two clusters that are geographically separate. Data needs to be held over different timescales, i.e. short, medium and long-term, depending upon the needs of a particular project.
A number of interesting products and companies were mentioned during this talk. These include Expandrive where an OpenStack Swift component can become a network drive. Panzura was mentioned in terms of Swift as a network appliance. Zmanda and Cloudberrylab were all about backup and recovery. Interesting stuff; again, a lot to take in.
Bridges and Tunnels – a drive through OpenStack networking
Mark McClain from the OpenStack foundation, talked about the networking side of things, specifically, the OpenStack networking component that is called Neutron. Even though I didn’t understand all of it, I really enjoyed this presentation. On a technical level, it was very dense; it contained a lot of detail.
Mark spoke about some of the challenges of using the cloud. These included a high density of servers, the difficulties of scaling and the need for on-demand services. A way to tackle some of these challenges is to use network virtualisation and something called overlay tunnelling (but I’m not quite sure what that means!)
Not only can virtual machines talk to virtual drives (such as the block storage service, Cinder), but they can also talk to a virtual network. The design goals of the network component were to have a small core, and to have a pluggable open architecture which is configurable and extensible. You can have DHCP configuration agents and can specify network traffic rules. Neutron is also (apparently) backed by a database and a message queue. (I also heard that there is a REST interface, if I’ve understood it correctly and my notes haven’t been mangled in the rush to write everything down).
A lot of network hardware can now be encoded within software (which links back nicely to the point about abstraction that I mentioned in the first block). One example is something called Openvswitch (project website). I’ve also noted down that you can have a load balancer as a service, a VPN as a service and a firewall as a service (as highlighted by the earlier vArmour talk).
Hybrid cloud workloads
The final talk of the day was by Monty Taylor who is also from the OpenStack foundation. A hybrid cloud is a cloud that is a combination of public and private clouds (which could, arguably be termed an ‘ecosystem of clouds’). Since it was the end of the day, my brain was full, and I was unable to take a lot more on board.
Reflections
All this was pretty interesting and overwhelming stuff. I remember one delegate saying, ‘this is all very good, but it’s all those stupid names that confuse me’. I certainly understand where he was coming from, but when it comes to talking about technical stuff, the names are pretty important: they allow developers to share understandings. I’m thankful for those names, although each name does take quite a bit of remembering.
One of the first things I did after the conference was to go look on YouTube. I thought, ‘there’s got to be some videos that helps me to get a bit more of an understanding of everything’, and I wasn’t to be disappointed – there are loads. Moving forward, I need to find some time to look through some of these.
One of the things that I’ll be looking for (and something that I would have liked to see in the conference) was a little bit more detail about case studies that explicitly show how parts of the architecture work. We were told that virtual machines can spin up in situations where we need to attend to more demand, but perhaps the detail of the case studies or explanations passed me by.
This is a really important point. Some aspects of software development are changing. I’ve always held the view that good software developers need to have an appreciation of system administration (or the ‘operations’ side of things). When I had a job in industry there was always a separation between the systems administrators and the developers. When the developers are done, they throw software over the wall to the admins who deploy the software.
This conference introduced me to a new term: a devop – part developer, part programmer. Devops need to know systems stuff and programming stuff. This is a reflection of software being used at new levels of abstraction: we now have concepts such as network as a service, and software defined security. Cloud developers (and those who are responsible for keeping clouds running) are system software developers, but they can also be (and have to understand) application development too.
A devop needs a very wide skill set: they need to know about networks, hardware, operating systems, and different types of data store. They might also need to know about a range of different scripting languages, and other languages such as Python. All these skills take time (and effort) to acquire. A devop is a tough and challenging job, not only due to the inherent complexity of different components, but also due to the speed that technologies change and evolve.
When I arrived at the conference, I knew next to nothing about what OpenStack was all about, and who was using it. By the end of the conference I ended up knowing the names of some of its really important components; mists of confusion had started to lift. There is, however, a huge amount of detail to get my head around, and one of the things that I’m also going to do is to look at some user stories (OpenStack foundation). This, I think, will help to consolidate some of my learning.
OpenStack conference, June 2014 (part 2 of 2)
This blog post is the second of two that summarises an OpenStack conference that I attended on 4 June in London.
This second half of the conference had two parallel sessions. Delegates could either go to the stream that was intended for novices (which is what I did), or go to a more technical session.
I was quite tempted by the technical session, especially by a presentation that was all about what it means to be an OpenStack developer. One of the key points that I did pick up on was that you need to know the Python language to be an OpenStack developer, which is a language that is used within the OU’s data structures and algorithms module, M269 Algorithms, data structures and computability.
Introduction to OpenStack
The first session of the afternoon was by Kevin Jackson who works at Rackspace.
Kevin said that OpenStack and Linux are sometimes spoken about in similar terms. Both can be created from distributions, and both are supported by companies that can offer consultancy support and help to move products forward. ‘OpenStack is like a pile of nuts’, said Kevin, and the nuts represent different components.
So, what are the nuts? Nova is a compute engine, which hosts a virtual machine running in a Hypervisor. I now understand that a hypervisor can host one or more virtual machine. You might have a web server and your application code running within this bit of OpenStack.
Neutron is all about networking. In some respects, Neutron is a virtual network that has been all written in code. There is more about this in later presentations. If you have different parts of an OpenStack implementation, Neutron allows the different bits to talk to each other; it pretends to be a physical network.
Swift is an object store, which is something that was spoken about during an earlier presentation. Despite my earlier description, Swift isn’t really like a traditional file system. Apparently, it can be ‘rack or cabinet aware’, to take account of the design of your physical data centre.
Cinder is another kind of storage; block storage. As mentioned earlier, to all intents and purposes, Cinder looks like a ‘drive’ to a virtual machine. I understand a situation where you might have multiple virtual machines accessing the same block storage device.
Ceilometer is a component that was described as telemetry. This is a block which can apparently say how much bandwidth is being used. (I don’t know how to describe what ‘bandwidth’ is in this instance – does it relate to the network, the available capacity within a VM, or the whole installation? This is a distinct gap in my understanding).
Heat is all about orchestration. Heat monitors ‘the cloud’, or its environment. Kevin said, ‘if it knows all about your environment and suddenly you have two VMs and not three, it creates a third one’. This orchestration piece was described as a recipe for how your system operates.
All these bits and pieces are controlled by a web interface called Horizon, which I assume makes calls to the APIs of each of these components. You can use Horizon to look at the components of the network, for example. I have to confess to being a bit confused about the distinction between JuJu and this standard piece of OpenStack – this is another question that I need to ask myself.
At the end of Kevin’s presentation, I’ve made a note of a question from the floor which was: ‘why should I go open source and not go for a proprietary solution?’ The answer was interesting: you can get locked into a vendor if you choose a proprietary solution. If you use an open source solution, such as OpenStack you can move your ‘cloud’ different providers, say, from Rackspace to HP. With Amazon web services, you’re stuck with using Amazon web services. In some respects, these arguments echo arguments that are given in favour of Linux and other open source products. The most compelling arguments are, of course, freedom and choice.
A further question was, ‘how mainstream is this going to go?’ The answer was, ‘there’s many companies around the globe who are using OpenStack as a solution’, but I think it was also said that OpenStack is just one of many different solutions that exist.
OpenStack and Storage made easy at Lush Cosmetics
The second presentation of the day was made by Jim Liddle who works for a cloud consultancy called Storage Made Easy.
Jim presented a case study about his work with Lush Cosmetics. I’ve made note of a number of important requirements: the data that is stored to the cloud should be encrypted, and there should be ways to help facilitate auditing and governance (of the cloud).
It’s interesting that the subject of governance was explicitly addressed in this case study. The importance of ‘institutional control’ and the need to carry out checks and balances is one of reasons why organisations might choose private clouds over public clouds. In the case of Lush, two key drivers included the cost per user, and the need to keep their data within the UK.
A new TLA that I heard was OVF (Wikipedia), an abbreviation for Open Virtualization Format, and is a way to package virtual machines in a way that is not tied to any particular hypervisor (VM container), or architecture. Other technologies and terms that were referred to included: MySQL, which is touched on in TT284 Web Technologies (OU), Apache, MemCached (Wikipedia) and CentOS.
Deploying Windows Workloads into OpenStack using JuJu
A lot of the presentations had a strong Linux flavour to them. Linux, of course, isn’t the only platform that can be used to power clouds. Alessandro Pilotti from Cloudbase solutions spoke on the connections between Windows and OpenStack.
Terms that cropped up during his presentation included Hyper-V (a hypervisor from Microsoft), KVM (Kernel based virtual machine, which is Linux hypervisor), MaaS (metal as a service, an Ubuntu term), GRE Tunnels (GRE being an abbreviation for Generic Routing Encapsulation), NVGRE (Network Virtualization using Generic Routing Encapsulation), and RDP (Remote Desktop Protocol).
It was all pretty complicated, and even though I’m reasonably technical, this was at a whole other level of detail. Clicking through some of the above links soon takes me into a world of networking and products that are pretty new to me. This clearly suggests that there is a whole lot of ‘new technology’ out there that I need to try to make a bit of sense of, and this, of course, takes time.
Alessandro also treated us to a live video link that showed a set of four small computers that were all hooked up together (I have no idea what these small desktop computers without screens were called; they used to have a special name). The idea was to show LEDs flashing to demonstrate some remote rebooting going on.
This demo didn’t quite work out as planned, but it did get me thinking: to really learn how to do cloud stuff, a good idea would be to spend time actually playing with bits of physical hardware. This way you can understand the relationships between logical and physical architectures. The challenge, of course, is finding the time to get the kit together, and to do the learning.
Using Swift in Entertaining Ways
This presentation was made by a number of people from Sohonet a company that offers cloud services to the film and TV industry. An interesting application of cloud computing is film and video post-production, the part of production where when recordings are digitally edited and manipulated. An interesting challenge is that when it comes to video post-production we’re talking about huge quantities of data, and data that needs to be kept safe and secure.
Sohonet operates two clusters that are geographically separate. Data needs to be held over different timescales, i.e. short, medium and long-term, depending upon the needs of a particular project.
A number of interesting products and companies were mentioned during this talk. These include Expandrive where an OpenStack Swift component can become a network drive. Panzura was mentioned in terms of Swift as a network appliance. Zmanda and Cloudberrylab were all about backup and recovery. Interesting stuff; again, a lot to take in.
Bridges and Tunnels – a drive through OpenStack networking
Mark McClain from the OpenStack foundation, talked about the networking side of things, specifically, the OpenStack networking component that is called Neutron. Even though I didn’t understand all of it, I really enjoyed this presentation. On a technical level, it was very dense; it contained a lot of detail.
Mark spoke about some of the challenges of using the cloud. These included a high density of servers, the difficulties of scaling and the need for on-demand services. A way to tackle some of these challenges is to use network virtualisation and something called overlay tunnelling (but I’m not quite sure what that means!)
Not only can virtual machines talk to virtual drives (such as the block storage service, Cinder), but they can also talk to a virtual network. The design goals of the network component were to have a small core, and to have a pluggable open architecture which is configurable and extensible. You can have DHCP configuration agents and can specify network traffic rules. Neutron is also (apparently) backed by a database and a message queue. (I also heard that there is a REST interface, if I’ve understood it correctly and my notes haven’t been mangled in the rush to write everything down).
A lot of network hardware can now be encoded within software (which links back nicely to the point about abstraction that I mentioned in the first block). One example is something called Openvswitch (project website). I’ve also noted down that you can have a load balancer as a service, a VPN as a service and a firewall as a service (as highlighted by the earlier vArmour talk).
Hybrid cloud workloads
The final talk of the day was by Monty Taylor who is also from the OpenStack foundation. A hybrid cloud is a cloud that is a combination of public and private clouds (which could, arguably be termed an ‘ecosystem of clouds’). Since it was the end of the day, my brain was full, and I was unable to take a lot more on board.
Reflections
All this was pretty interesting and overwhelming stuff. I remember one delegate saying, ‘this is all very good, but it’s all those stupid names that confuse me’. I certainly understand where he was coming from, but when it comes to talking about technical stuff, the names are pretty important: they allow developers to share understandings. I’m thankful for those names, although each name does take quite a bit of remembering.
One of the first things I did after the conference was to go look on YouTube. I thought, ‘there’s got to be some videos that helps me to get a bit more of an understanding of everything’, and I wasn’t to be disappointed – there are loads. Moving forward, I need to find some time to look through some of these.
One of the things that I’ll be looking for (and something that I would have liked to see in the conference) was a little bit more detail about case studies that explicitly show how parts of the architecture work. We were told that virtual machines can spin up in situations where we need to attend to more demand, but perhaps the detail of the case studies or explanations passed me by.
This is a really important point. Some aspects of software development are changing. I’ve always held the view that good software developers need to have an appreciation of system administration (or the ‘operations’ side of things). When I had a job in industry there was always a separation between the systems administrators and the developers. When the developers are done, they throw software over the wall to the admins who deploy the software.
This conference introduced me to a new term: a devop – part developer, part programmer. Devops need to know systems stuff and programming stuff. This is a reflection of software being used at new levels of abstraction: we now have concepts such as network as a service, and software defined security. Cloud developers (and those who are responsible for keeping clouds running) are system software developers, but they can also be (and have to understand) application development too.
A devop needs a very wide skill set: they need to know about networks, hardware, operating systems, and different types of data store. They might also need to know about a range of different scripting languages, and other languages such as Python. All these skills take time (and effort) to acquire. A devop is a tough and challenging job, not only due to the inherent complexity of different components, but also due to the speed that technologies change and evolve.
When I arrived at the conference, I knew next to nothing about what OpenStack was all about, and who was using it. By the end of the conference I ended up knowing the names of some of its really important components; mists of confusion had started to lift. There is, however, a huge amount of detail to get my head around, and one of the things that I’m also going to do is to look at some user stories (OpenStack foundation). This, I think, will help to consolidate some of my learning.