Photo by Nasik Lababan on Unsplash

A DevOps practitioner plays the guitar

A short story of a DevOps Engineer sharing his knowledge, guidelines and useful heuristics to stay out of trouble.

Hello, I’m a DevOps engineer and beginning guitar player. When starting to play, I was amazed at how much there is to learn in music. I realized that it is similar to my learning journey in programming, testing, infrastructure support and DevOps. One of my tasks is to mentor others and teach them how to navigate on this terrain. That’s how I ended up writing this story. I’ve tried to write it in easy to read manner. At the same time I’ve have added a lot of worthwhile links that are there for your amusement and learning — enjoy it.

Introduction

A few months back, I’ve started to learn to play the guitar. I’ve always liked the acoustic guitar and melancholic, romantic songs. One of my favourites is Ry Cooder’s version of Cancion Mixteca — if you haven’t heard it you should look it up. The passion, the emotion, the sounds, that feeling it provokes — I’m always drawn to it.

If you take a look into the world of guitars, you’ll see it’s vast — in every aspect:

• types of guitars — from ukulele to electric, from dreadnought to concert, 4, 6, 8 or more strings, …
• types of music — classical, bluegrass, pop, flamenco, South American, rock, folk, …
• number of chords, styles of playing, strumming or fingerpicking patterns, rhythm, …
• famous players: Jimi Hendrix, Django Reinhardt, Eric Clapton, Joan Jett, Marc Knopfler, Marcus Biancardini, …

So what does an article about DevOps and learning to play the guitar have to do with one another? Simply, I like analogies — it brings an unknown world in better known context so that it’s easier to understand. I live and work in the DevOps world where there is an equal breadth and depth in types of work, problems, solutions and complexity as in music. And it’s equally as much of a “forest” for everybody who enters this domain and tries to find his or her way around. You have the choice:

• cloud, on premises, multicloud, kubernetes, hybrid, docker swarm, API’s, …
• application migration, microservices, monoliths, 3-tier, backend, pipelines, CI/CD, …
• GitHub, gitlab, RTC, Systems management, ITIL, agile, Kanban, lean, …
• Jez Humble, Martin Fowler, Kris Buytaert, Patrick Debois, Gene Kim, Alistair Cockburn, …

Have you ever heard of the 80/20 rule? Simplified: with 20 percent knowledge of the domain, you can solve 80 percent of related problems. So, I’m sharing with you my 20 percent of the DevOps domain — use it to your advantage.

I work more in the “Ops” part of DevOps now but have worked in the whole range: development, analysis, testing, support — been there, done that. I understand applications but have managed infra. My job in the team ranges from architect to project manager, from mentor to tester/developer, from spokesperson of the team to trainer of people who work with our solutions. My day to day work focuses on “getting things done”, coordination and shielding my team from the sometimes over-complex environment in which they have to work. The team consists mainly of Belgian and Romanian DevOps specialists and enthusiasts that are part of the value stream for provisioning and configuring servers and application deployments for our customer, a large bank & insurance company.

The team and the steps

As with any project, we get asked to develop new features, expand existing solutions or rebuild parts of it. It ranges from the simple technical to the more complicated team interactions. But no matter the request, I have a tendency to apply the following steps.

Try to understand the why of a request. Given that someone asks: “Can you make me X?”, your first reaction should be: why? How does it help your team, his team and your customer when you implement it? Most likely, he presents a solution — and it’s up to you to figure out what exactly the problem is. Resist the temptation to jump right in and start developing X. Don’t let System One overtake before System Two has had a look.

Apply systems thinking. See the forest for the trees. Get the bigger picture. Figure out the relations between the teams and the tools. Avoid implementing bottlenecks, pinch points or replacing a fragile (semi)-manual process by an automated one.

“The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency.”

Bill Gates, founder and chairman of Microsoft

These questions can help:

• Would it be a good leverage point? In other words, does it make your workflow or value stream more fluent? Are you sure?
• What will be the consequences if you implement this? Direct advantages and disadvantages are usually easily spotted. Try to assess the impact on the indirect environment of where the solution is implemented.
• How will it affect the teams? Does it make things easier for them?
• What about the customer? Will it influence him? Is it something she would appreciate? In other words, is there a good motivation to do it for your customer?
• Think of the systems/applications and teams it interacts with. Who will be involved? What needs to be adapted there? How will it be maintained? What could break? Is it easy to explain to others?
• What are the alternatives? Any short term solutions?
• If it is a workaround, think about the technical debt it introduces
• What about testing?
• How long do you need it for?
• …

Use mental models extensively. They help in clarifying advantages, processes and thoughts — they’re your secret weapon.

Jump into service blueprinting or design thinking. Talk to the teams on how it’s done now and understand the flow from request until delivery of which X will be a part. Map out what is done using for example service blueprinting or design thinking. Understand why those teams work the way they do. Important: listen to understand first, then bring up and discuss possible improvements or alternatives. It helps and creates a better communication. Sometimes you’ll see the problem or solution from another angle and that can open up other possibilities. And yes, don’t be afraid to draw your view or understanding — just use simple boxes with arrows and sticky figures. Anybody can draw — even you!

Evaluate. With what you know now, is it still viable to do it like that? Will it provide value or is it just another stumbling block? For smaller decisions you could trust yourself and your team. For more complicated decisions, I do recommend How to visualize and evaluate decision options.

Measure whenever possible. Think of ways to measure the actual situation and the improved one. Then apply them to the actual situation while developing the solution. Look up Goodhart’s law or the cobra effect and take them into account.

Create the solution. Design, test, test, test and automate, automate, deploy, test, test, deploy again, re-test, test and don’t forget to measure :-). Our typical solutions involve multiple CI/CD pipelines, integration with other tools or API calls. After a while, you see a pattern emerging that can be reapplied.

Continually refine. You’ve delivered, but now the real work begins. You need to maintain and improve it. It’s like a new piece of equipment that needs maintenance — preventive and curative. Keep in mind that the landscape around this solution will be modified and that it possibly affects it’s working. Change is the only constant!

Besides that, as a team we build our strengths and resilience through several practices:

• We aim to be knowledgeable, to know the terrain in which we work, the customer, the teams, the tools.
• Our team has a mix of backgrounds, skills and expertise — it is a strength.
• We strive to be open and honest. It makes us trustworthy so our customers come back for more.
• We listen to concerns of others.

What do we use to automate and prepare servers?

Just like in the musical world where a guitar is so often prevalent, we rely on a tool set that acts as an important corner-stone for our solutions. It’s versatile, comes with a wealth of tools and is extremely stable. Hereunder, you can find a concept map that shows how our system works. It is a simple representation of what we do and a lot of details are not shown. They’re left out to focus on the interactions of this model, not on technical specifics. And as always: all models are wrong, some are useful.

On the right hand side, the three (purple) objects represent the essential building blocks:

• an application: the most important part that brings value to the customer
• the middleware stack: all the frameworks and interfaces the application needs
• base server: the OS with all the tools to manage it

The three tools (in blue) we use are:

• JFrog Artifactory: which keeps the binaries of all the middleware that’s installed
• Chef Effortless Infrastructure Suite: installs middleware and configures a server and
• UrbanCode Deploy (UCD) ®: orchestrator of deployments and configuration actions.

And here’s a bit of the life cycle of an application with our toolset:

• an application developer changes the application (new feature, bug, optimization, …) and puts the result in Artifactory
• she also creates or updates the ‘deployment actions’ — whatever is necessary to deploy the application.
• after she’s finished, a project lead can trigger the deployment of the application in a test environment.
• if there are no servers yet for that application, UCD can delegate the creation of the servers to ICD/VMware.
• next, UCD instructs Chef to configure the server so that it’s ready to receive the application.
• the original scripts for middleware components come from the company’s experts. Our team converts these scripts into cookbooks and policy files for Chef and takes care of the steps in UCD.
• the binaries for the middleware components are stored in Artifactory
• and finally, UCD deploys the application according to the deployment actions specified by the developers.

The added value

So, what’s the added value for the customer, the teams we work with and for us?

1. Visibility, Openness and transparency. We’ve set up an open structure that the customer and other teams can use to see what we’re doing. Part of it is done with a kanban board that shows what we’re doing. Everybody can have access to the dashboards of Artifactory and Chef to see what’s happening. They now have a far better view and understanding of what’s going on. It enhances the understanding of the process and mutual cooperation of the teams. And we plan to set-up an information radiation board.
2. Speed and lean concepts. We’ve glued together and automated the installation of application related middleware. Chef installs database connectors, scheduling tooling, Websphere, etc. As a result, we shortened the time it takes to install these individual components — in some cases we have astonishing results: from 5 or more days to 15 minutes is the eye-catcher. Wherever we can, we try to apply lean techniques and rules to optimize the value steams.
3. Additionally, the burden on the teams that previously executed or coordinated all the separate actions went down significantly alongside with the error rate during the installations. Consequently, we also have less rework.

And how about our team? They can be considered as part of the central hub in provisioning. We work together with the Dev teams of the customer that deploy applications, the Sec teams for IAM, security and auditing and the Ops specialists of each middleware component (TWS, Elasticsearch, Dimension, …) and operating system. We reach out to them and meet (virtually — because we’re not geo-located together) for new components or in case of problems. Everybody takes pride in what we achieved and how we got there.

But it also works the other way around: we’re contacted quite often because of the overview we have or the understanding of how the company we work for hangs together. I also like to think that it is because we come up with other ideas due to our diversity, expertise, ways of working or knowledge of tools, methods and practices. You might say it strikes a chord ;-)

Either way, we love our job! Have fun and enjoy.