Report on Current State and Future Predictions of DevOps Tools and Practices
Introduction
The term DevOps is a portmanteau of the words Development and Operations. As the words suggest it is the combination of software development meets the operational testing and deployment of software. Whilst it bridges the gap between these two distinct spheres of the software development lifecycle (SDLC), it exists not as one concrete entity but as a amalgamation of an organisation’s cultural policies and practices together with sophisticated testing, automation and deployment tools. The correct joining of these should result in appreciable savings in time to deliver new software features and bug fixes which are in closer alignment to the business objectives for that software than they would be without DevOps being employed.
A research paper published in 2017 [1], conducted a survey of organisations using DevOps in their SDLC process, noted that their was clear confusion in organisations over what DevOps is or should be, and how best to quantifiably measure the results of it. The paper concluded that there is no one precise definition of how DevOps should be utilised and that it will vary across organisations in its implementation. Gene Kim, the author of ‘The DevOps Handbook’ sums up the ambiguity surrounding Devops as “Currently, DevOps is more like a philosophical movement, not yet a precise collection of practices, descriptive or prescriptive.” [2] With that though there are some principle guidelines that can be followed both from a technology and company culture perspective to get the most from it which will be examined later in this report.
This report will further attempt to offer as accurate an overview of what DevOps is, form its history and why software organisations realised the need for it, to the current trends in DevOps tools and practices today, to then finally extrapolating where DevOps could possibly evolve into when considering the realms of Machine Learning, ChatOps and Infrastructure as Code.
Background & History
The origin of the term DevOps first emerged in the late 2000’s and naturally grew out of the recent agile methodologies first employed in that decade that had sprung up to tackle the rigid constraints of the Waterfall method. The Waterfall method (a cascade flow model from initial analysis to deployment) had been a staple of the SDLC for decades previously, but had grown increasingly too slow and unresponsive for the quick turnover and changes required of software development as it increasingly moved into the online domain.
Conversely the Agile method allowed for more flexibility throughout the SDLC by responding quicker to changing business demands and requiring more communication back and forth between the various software teams, from requirement analysts to developers to testers to deployment operators.
It was from this increased communication between teams that the process behind the current state of DevOps first flourished. Indeed in this article from DevOps.com (a leading online publication for Devops professionals) which categorically states that “DevOps is just a natural extension of Agile. It takes the guiding principles and best practices that Agile brought to developers and applies them on a much broader scale across the organization.” [3]
So as one part of DevOps was the blurring of lines between developers and testers and deployment operators another essential element of the process lies in the keyword of automation. Allowing tools to perform tasks so that other resources, namely human beings, can provide greater value by performing less mundane work. The other part of DevOps, though, is an organisational cultural shift. It’s a change in the overall mindset of the enterprise that erases the traditional barriers between interconnected teams and their associated processes and enables everyone to function more efficiently without unnecessary overhead or the dreaded “red tape”. Devs will start to think more like Ops personnel and vice versea. This facilitates the fostering of shared responsibility between teams and less victim blaming when something goes wrong in the SDLC. The upshot of that is that teams tend to perform better and more efficiently as a direct result of improved cooperation and empathy with each other.
To sum up DevOps in three terms it would have to be enhanced communication, shared responsibility and increased automation. The increased automation comes mostly from the technology and tools being used, but the other two attributes are mostly human driven, with ChatOps (to be discussed later) being the one exception to technology improving the communication angle.
One early triumph for DevOps adoption came in 2009, when the largest photo sharing website at the time, Flickr, took its agile approach further and started using DevOps which until that point had not been experimented with on such a large production site as this. As a result of applying DevOps tools and practices they were able to speed up their deployments to 10+ on a daily basis which was unheard of at the time. [4]
Current Tools & Trends
As was stressed already DevOps is as much about a change in a software organisation’s mindset and approach to the SDLC and how they communicate internally, as it is with the actual tools and technology leveraged by DevOps practices. While the human element of DevOps was covered in Background & History section, this section will examine the actual technology driving the DevOps revolution.
This online article [5] features a comprehensive list of the most prevalent open source DevOps tools on the market in 2019. Using this list as a reference, the below list was formulated. They are arranged in the order from top to bottom that they would be on in a normal DevOps pipeline.
Build Tool - Gradle: The first step in practical DevOps is the building of our software packages for eventual deployment. Previously the domain of such build tools like Maven and Ant, Gradle has seemingly supplanted them in performance in recent years having first being introduced in 2009. Whilst those aforementioned tools use XML for configuration changes, Gradle employs a Groovy based DSL type language for describing builds which offers more fine-tuned flexibility. It supports deploying code in all the major languages and relies on intelligent incremental build queueing to produce performance increases of nearly 100 times than Maven and Ant before it. [6]
Source Management Tool - Git: Widely popular in the industry with no signs of losing its place as the source control tool to beat, Git has just recently being bought by Microsoft. It allows you to track the progress of your both your own code and other work on your team in code silos known as repositories. Team members can make push and pull requests to these repositories, and can create offshoots of the main code base as branches for them to experiment on or test. Another popular alternative to Git, is Bitbucket which some say is better for very small teams of 2-5 developers.
Automation Tool - Jenkins: As mentioned previously, automation is a critical component of DevOps, so a automation tool of some sort is needed. Of these tools, Jenkins is probably the most popular to automate the deployment pipelines that makes the continuous flow of a typical CI/CD pipeline possible. It also has a thriving plugin eco system to further extend its capabilities if needed and has integration for Docker and Ansible which will be discussed shortly this report. For a non open-source alternative to Jenkins, there is Bamboo by Atlassian, the makers of the popular ticket issue board software Jira.
Containerisation Tool - Docker: Containers isolates applications into isolated logical silos, so that they become more portable and secure as a result. Containerised apps are also operating system and platform independent allowing for much more freedom in where and how they are executed. Docker is the forefront technology in the container sphere and is starting to be embraced by DevOps engineers. Indeed this academic paper [7] goes into detail about deploying a Docker application using a Jenkins automated pipeline. This paper also states that Docker and containers in general are becoming a fundamental part of the DevOps process and will also reduce the need for configuration management tools for code deployments (like Ansible) in the future.
Container Orchestration Tool - Kubernetes: Container Orchestration has become a buzzword in the last few years with Kubernetes being its most prominent name (and to a lesser degree, Docker Swarm). Kubernetes harnesses the power of single containers into co-ordinated clusters to cope with massive scale. Kubernetes automates (stressing again that automation is one of the keystones of DevOps) the distribution and scheduling of containers across the whole cluster under the hierarchy of master and slave nodes.
Configuration Management Tool - Ansible: Ansible is the newest of three of the most popular configuration management tools (the others being Chef and Puppet). Compared to those aforementioned tools, Ansible is the simplest to implement with its easy to read YAML syntax, which might explain its quick traction in the marketplace. It is also the cornerstone of the incremental move towards Infrastructure as Code (IAC) which will be discussed in further detail in the Future Trends section of this report.
Monitoring Tool - Nagios: DevOps requires continual monitoring of the hosting environment in order to respond more quickly to potential issues as they arise. Nagios is a mature (out since 2002) monitoring product that provides an all in one infrastructure monitoring solution. With Nagios, records of any anomalous events, outages, or failures are stored and indexed, before being compiled into statistical analysis sheets and forecast charts to monitor for future trends and predict infrastructure downtime and security issues.
Future Predictions
In attempting to predict the future of DevOps, from reading the recent literature, it would appear that the two big areas that DevOps will likely have a significant impact on in the near future are in Infrastructure as Code (IAC) and Machine Learning Operations (MLOps).
IAC is the concept of managing your infrastructure environment in the same way you do applications or other code. Instead of manually making configuration changes or tediously compiling single use scripts to make infrastructure adjustments, the server infrastructure is managed instead using the same rules that govern code development—for instance when provisioning new resources, or spinning upper down currently provisioned resources. That means that the core technical practices of DevOps—like version control, continuous monitoring and controlled automation—are applied to the underlying code that governs the creation and management of your infrastructure. In other words, your infrastructure is treated the same way that application code would be in typical DevOps, bringing the quick time deployment and responsive nature of that solution also.
Some of the tools of IAC include the already covered Chef, Puppet and Ansible, as well as server provisioning tools like Terraform and AWS CloudFormation. The well regarded software development author Martin Fowler, summed up the advantages of IAC as “Most importantly using configuration code makes changes safer, allowing upgrades of applications and system software with less risk. Faults can be found and fixed more quickly and at worst changes can be reverted to the last working configuration. Having your infrastructure defined as version-controlled code aids with compliance and audit. Every change to your configuration can be logged and isn't susceptible to faulty record keeping.” [8]
The relatively new field of MLOps aims to bring the established practices of a DevOps tools and practices and applying them to how Data Scientists can more effectively collaborate with DevOps engineers to better automate and deploy Machine Learning (ML) algorithms. MLOps captures and expands on previous established DevOps practices while also extending them to manage the unique challenges of ML. MLOps is predicted to grow in importance over the next few years to meet the increased demand by data science professionals, which in turn is driven by the exponential growth in ML and Big Data in general.
Both of the main public cloud providers, Amazon and Azure have in 2019 introduced dedicated MLOps tools to their product range and their is hope that with bringing the DevOps tools and practices that have worked for regular software to ML will in turn hasten and improve the quality of ML products to market. A quote from the CTO of ParallelM, a data science firm confirms this by saying, “A business’ ability to optimize MLOps can determine how quickly it can adapt to changing circumstances relative to competition. We believe MLOps is the next competitive frontier in the rapidly expanding ML business applications space.” [9]
As a natural follow on from discussing MLOps, another new field arising as the natural offspring of DevOps and ML is ChatOps (Chat meets Operations) ChatOps is the next step forward in collaboration and communication within software teams. Jesse Newland, the principal production engineer of Github, coined the term ChatOps and referred to it as “Placing tools directly in the middle of the conversation”. [10] What he is referring to is the idea of using advanced AI and ML assisted chatbots and voice recognition algorithms to better bridge the communication and automation gap between developers and operators.
Because ChatOps helps software teams better communicate than more traditional channels like email or a chat room, it in turn will help smooth out the DevOps experience as collaboration and communication are essential to that process. Professional chat room products from Slack and Microsoft Teams already employ a rudimentary form of ChatOps but it is anticipated that this functionality will grow in the coming years and will integrate itself into the DevOps process in a far more seamless way that it is now. So even though ChatOps is an offshoot of DevOps it is also feeding back into its parent process to improve the whole DevOps experience.
Conclusion
When looking at industry trends, DevOps shows no signs of slowing down in its adoption rate by software organisations. A Technavio research report [11] predicts that between 2016 to 2020, DevOps will have grown in use by 19.42% CAGR (compound annual growth rate - anything over 10% is considered very good). This is what would appear to be a confirmation that DevOps arrived at the right time when the necessary tools were in place and there was a real desire to apply those tools using an agile mindset to bring about real innovation in the marketplace.
Suffice to say that DevOps will be sticking around for quite some time to come at least until something new comes along borne of DevOps in the same way that DevOps was borne of agile methodologies. The final evolution some predict may even be the advent of “NoOps”, namely the automated process underpinning DevOps will become so competent and error free, that they will remove the need for human operators. Others would argue that the human component of DevOps is too nuanced to be replaced entirely even with the increasing processing power of ML algorithms.
But disregarding the technology that makes DevOps possible, it has been an undoubted success in nurturing closer ties between what was previously disparate and completely separate teams in the SDLC. DevOps makes the entire SDLC process not only a more collaborative and therefore productive workplace for software professionals but hopefully a better place to work in general.
References
Erich, F., Amrit, C. and Daneva, M. (2017). A qualitative study of DevOps usage in practice. Journal of Software: Evolution and Process, 29(6), p.e1885.
Kim, G. (2015). 18 Great DevOps Quotes. [online] Www3.dbmaestro.com. Available at: https://www3.dbmaestro.com/blog/18-great-devops-quotes [Accessed 23 Jun. 2019].
Bradley, T. (2015). DevOps is Agile for the Rest of the Company - DevOps.com. [online] DevOps.com. Available at: https://devops.com/devops-is-agile-for-the-rest-of-the-company/ [Accessed 23 Jun. 2019].
Allspaw, J. (2019). 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr. [online] Slideshare.net. Available at: https://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr [Accessed 25 Jun. 2019].
Monus, A. (2019). The 10 best DevOps tools for 2019 | Raygun Blog. [online] Raygun Blog. Available at: https://raygun.com/blog/best-devops-tools/ [Accessed 23 Jun. 2019].
Gradle. (2019). Gradle | Gradle vs Maven: Performance Comparison. [online] Available at: https://gradle.org/gradle-vs-maven-performance/ [Accessed 23 Jun. 2019].
Periyannan, S. (2019). Docker and DevOps: Developing Stateful Applications and Deploying in Docker - DZone Cloud. [online] dzone.com. Available at: https://dzone.com/articles/docker-and-devops-developing-state-full-applicatio [Accessed 23 Jun. 2019].
Fowler, M. (2019). bliki: InfrastructureAsCode. [online] martinfowler.com. Available at: https://martinfowler.com/bliki/InfrastructureAsCode.html [Accessed 24 Jun. 2019].
Talagala, N. (2019). Why MLOps (and not just ML) Is Your Business’ New Competitive Frontier. [online] The World's Number One Portal for Artificial Intelligence in Business. Available at: https://aibusiness.com/mlops-parallelm-competitive-edge/ [Accessed 24 Jun. 2019].
10.Newland, J. (2019). How ChatOps Can Help You DevOps Better. [online] Chatbots Magazine. Available at: https://chatbotsmagazine.com/how-chatops-can-help-you-devops-better-5-minutes-read-507438c156bf [Accessed 25 Jun. 2019].
11.Technavio. (2019). Global DevOps Platform Market 2016-2020. [online] Available at: https://www.technavio.com/report/global-it-spending-region-and-industry-devops-platform-market [Accessed 25 Jun. 2019].
