Context
The Company
A major insurance company with global presence and local offices in NYC.
The Challenge
Previous attempts to implement local Agile enhancements have showed very little success and didn’t show any significant improvement in the overall company velocity. Now is the time to implement a comprehensive DevOps transformation plan considering interdependency of the teams, tools and processes.
The Goal
Implement DevOps to faster deliver value to the market and respond to customers’ demands with the velocity and the quality the customers expect.
Project Phases
The company was looking for major data center operations improvements and to redefine the services offered by the systems team (systems, storage, networking, DBA, InfoSec/security, etc…).
Process Automation: The transformation started with the automation of the core and time consuming processes such as provisioning of the new application deployments, application servers and configuration of their networking connectivity. OpenStack private cloud was then introduced on top of the Xen hypervisors to provide the initial API needed for the automation. We could then automate the deployment orchestration and provide a way to programmatically provision new servers on demand. Further automation of the F5 BigIp load balancers was used to connect the new servers to the existing infrastructure and made the new application servers accessible by other system components. Application and system configuration of the servers was handled by Puppet configuration management. All provisioning and end-to-end deployment processes were now fully automated, codified and versioned. Versioning of the provisioning process allowed for integration of the new process independently without breaking the installed base of processes.
Training and Change Management: Implementation of the new tool chain allowed for simplification of the provisioning process and required additional personnel training. The training was provided to the systems teams. This was critical to create buy-in and led to the whole team’s contribution to the automation effort. Coding skills were also improved with training, and hiring processes were improved to emphasis coding skills in system engineers. This has proved to be fundamental in solidifying the management’s DevOps commitment.
Creation of Self-Service Portals for End Customers: Increased amount of automation and creation of the new tool chains allowed us to automate most of the day-to-day activities. Workflow management tools and continuous integration servers provided richer infrastructure for automation, control and auditing of the new processes. Gradually more and more processes were automated allowing us to create self-service portals for the end-customers. The mindset of the organization shifted which in turn enabled end-customers to become self-serving and freed systems engineers’ time from day-to-day repeatable ticket processing.
Adjustment of Security and Audit Requirements: Security and auditing requirements and processes were adjusted to accommodate the new automation by removing, where appropriate, previously required approvals from the security team. This in turn enabled rule-based automation of the approval and auditing.
Challenges
Challenges faced throughout the project can be broken down into following categories:
- Continuous Improvement: The extra time that became available to the system engineers was invested into velocity improvements. This led to a continuous flow of new activities such as: more automation, research and introduction of new technologies for both the internal customers and the development of the new processes.
- More training: the self-fueling automation mechanism led to more investment into personnel skills which allowed the company to up its game.
Outcome
The whole transformation project was a great success by reducing the provisioning cycle of hardware servers from 3 months to 1 day and deployment of new features from 1 month to 1 hour.
The introduction of self-service made this project a huge company success, both for the customers and the bottom line.