devopsy.kinsta.cloud

Tag: developers

VMware Aims to Advance Application Security

Most of the tasks associated with maintaining IT security assume that IT security professionals should discover and neutralize all potential threats to an application. At the VMworld 2017 conference this week, VMware is seeking to turn that notion of IT security on its head by focusing the efforts of IT security professionals on the known good.

Tom Corn, senior vice president for security products at ware, says instead of chasing all the potential bad things that could adversely impact an application, VMware App Defense employs machine-learning algorithms and data VMware collects from its management console and the host on which the applications are running to determine what elements make up an application workload. Those elements are compared to the manifest that was created when the workload was first deployed and, if they have been altered, VMware App Defense will invoke a preloaded library of incident response routines to enforce policy using VMware network virtualization software. Data collected from the host by VMware App Defense is done by inserting code into the kernel rather than deploying agent software.

Corn says VMware App Defense uniquely provides DevSecOps teams with access to reliable telemetry into the guest operating system environments running on top of VMware. IBM will feed the telemetry data gathered by VMware App Defense into its portfolio of security analytics software, and eventually that data will be fed into the IBM Watson cognitive computing platform to generate application security recommendations. Other vendors pledging support for VMware App Defense include VMware sister companies RSA and SecureWorks, as well as Carbon Black and Puppet.

Longer term, Corn says VMware expects to also be able to apply this capability to any production environment attached to a VMware NSX virtual network overlay.

As potentially compelling as VMware App Defense might be, Corn notes that IT organizations should not attempt to boil the IT security ocean. Rather, they should concentrate on new applications currently under development. Otherwise, managing security policies across hundreds of applications made up on thousands of microservices and communicating over microsegmented network overlays might be too complex to manage.

VMware has been making a case for some time now to employ NSX to take advantage of microsegmentation to limit the flow of east-west traffic in a data center. The idea is that if one node gets compromised by malware, it can only share that malware with nodes participating on the same microsegment of virtual network. VMware is now expanding on that idea in a way that also provides more visibility all the way up the application stack.

Priced at $500 per CPU per year, VMware App Defense will induce more IT security professionals to recommend deploying its stack over any other available option, VMware hopes. The degree to which IT operations teams and cybersecurity professionals can influence developers in that direction remains to be seen. But VMware App Defense does fulfill its promise, it’s more than likely developers will appreciate the not having to spend as much time defending existing applications instead of writing new ones.

— Mike Vizard

August 29, 2017
Simulating DevOps to Discover the ‘Culture Thingy’
DevOps … is a term used to refer to a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment where building, testing, and releasing software can happen rapidly, frequently, and more reliably. — Wikipedia

We are all, I believe, familiar with the fact that culture is an important factor when adopting DevOps.

Groan!

Most engineers and technologists are probably tired of having the “C” word thrown at them: culture.

Because let’s be brutally honest: DevOps value comes from the automation stuff. The latest “State of DevOps” report confirmed “that automation is a key differentiator for organizations.” All the shiny technology that supports the “continuous-es”: continuous integration, continuous deployment, continuous delivery, with infrastructure as code and spinning up servers in the cloud and then collapsing them when no longer needed, with other cool technology concepts such as containers, microservices and a whole array of sexy build, test and monitoring tools. Just when we start getting excited about it and drooling about the new tools we get to play with, somebody else pipes up with that culture. stuff again …

Groan. Just let us get on and do IT!

But equally, probably the most important part of the “continuous-es” is continual learning and improvement, because again, let’s be honest: The real value of DevOps takes years to realize. It is not a one-time, overnight migration with a simple press on the ‘Install DevOps” button. It is an iterative exercise involving people from the end-to-end value stream; therefore, it is all about continual experimentation, learning and improvement—and doing it with people from different siloes who don’t usually like talking to each other. So, it’s a “cultural thingy.”

If it’s not somebody talking about culture and collaboration and communication, it’s one of those “frameworky-geeks” talking about agile, scrum, lean thinking and IT service management (ITSM) and how we need “flow,” “process engineering,” “voice of the customer,” “kaizan,”kanban,” and a whole host of other Japanese-zen-like mysterious terms which only consultants are qualified to talk about and do, as they are the enlightened-ones (probably with their own secret handshake or Doctor Spock type split fingered salute!).

“DevOps is a philosophical movement,” we hear. Now, if there are two things that I wouldn’t normally associate with each other it is technical engineers (I was one!) and philosophy. Although I do remember trying to stare down a server once, hoping by the power of positive thought and sheer will power for it to start working again; the alternative was a sledgehammer …

Where was I? Oh, yes. So what are some of these “culture things” we should care about, apart from the “Wall of Confusion” that everybody keeps talking about? “We’re not confused! It’s the other lot on the other side of the wall that don’t get it!”

Below are some of the key discoveries relating to the culture, or soft skills, issues that teams experience and recognize during the DevOps business simulation, “The Phoenix Project.” Before you start getting all excited again, this is not a computer game in which you can win points by zapping Ops people who get in the way or setting phasers to stun when you see a developer about to release some more untested code or beam up an uppity product owner who tells you that what you coded may have been what they asked for but isn’t what they were thinking. This business simulation is a dynamic, interactive, classroom-based workshop in which people have to get together—face to face—and, well, communicate and collaborate! The simulation workshop has been performed with literally hundreds of DevOps teams across the globe and across different industries, and is used to support DevOps training and to help empower teams. (see a blog posted on this site).

What are Some of These Key Learning Points and Takeaways?

I have mapped these discoveries onto the DASA (DevOps and Agile Skills Association) competence model, particularly the “Skills & Behaviors” area.

DASA has developed the competence model to help organizations assess and develop the right knowledge and skills within their DevOps teams. The competence model is based upon industry feedback from practicing organizations, and helps organizations assess their level in terms of Novice, Competent, Proficient, Expert and Master. It looks at four key Skills & Behavior areas and eight Knowledge areas.

Skills & Behavior Areas: Courage, Team-building, DevOps Leadership, Continuous Improvement.

Simulation exercise discoveries for Courage:
- “Feedback” is often seen as negative. People don’t like or dare to give feedback. It is perceived as personal, as criticism and often as blame. A blame culture makes it even more difficult to create open, honest feedback which is a barrier to learning and improvement.
- “Fail fast” often means “blame fast,” or “pass the blame.” A blame culture creates a lack of trust and willingness to experiment for “fear of failure.” Often in the simulations if things go wrong, people revert to asking,”Whose fault was that?” rather than asking, “What happened? How can we stop that happening again?”
- Sharing knowledge. Still in many organizations, knowledge is power; people don’t like to share knowledge or don’t think others are capable of using knowledge. A critical aspect of DevOps is multi-functional teams with broad shared skills and knowledge.
Simulation exercise discoveries for Team-building:
- Teams are unaware of the need for active listening, confirmation of agreements and decisions and verifying “assumptions”—many mistakes are the results of “assumicide” and poor understanding, which causes disruptions and excessive hand-off moments to gain the right information resulting in waste, delays and rework.
- Ownership, responsibility and being proactive. Many IT people wait to be told what to do. Many people don’t like to admit they don’t know or that they need help. others don’t actively seek out and ask, “How can I help,” asking upstream and downstream, “What do you need from me to be able to do your job?”
- Many teams have an aversion to getting together as they have poor experiences of pointless meetings with lots of talking and then not a lot of decisions being made or action being taken.
- Many teams don’t set for themselves goals for a meeting or learning objectives from a training session. They do not ask themselves, “What do I need to take away from this session?” They often don’t ask, “What do you expect from me?” “What should my contribution be?” They often cannot answer the question, “What problem do you want to be able to solve after attending this?” Many times in a session people say, “I was told to come.”
- Silo thinking: Many teams still think from siloes, e.g the technology team they are in or the development team, and don’t think end-to-end. They are building and supporting a technology solution and not managing end-to-end business value.
Simulation exercise discoveries for DevOps Leadership:
- When there are managers in the simulation, people often sit back and wait to be told or managers feel they must come up with the solution and give instructions.
- Managers have difficulty letting go, and sometimes don’t have the patience to let teams learn and improve themselves.
- Managers fear experimentation and failure, as they may be blamed.
- Managers often blame teams for not becoming self-steering, becoming self-empowered or for not taking ownership, yet they do not help coach and empower teams.
- Often managers are unable to answer the “why” question: “Why are we doing DevOps?” If managers do not know then how can they expect team members to know?
Simulation exercise discoveries for Continuous Improvement:
- Teams are poor at reflecting, analyzing, thinking and coming up with solutions. They lack skills in “problem thinking,” also in terms of Kolb learning cycle. Often they recognize one symptom and then leap to conclusions and solutions.
- Lack of skills in facilitating a stand-up meeting. See the five key team skills mentioned by Jan Schilt on devopsy.kinsta.cloud. These skills are: The stand-up; Facilitating a retrospective; Managing the Kanban (Visual Management System); Managing the Andon: (and swarming problems); and Collaboration.
- Many teams struggle to record issues in their work, such as a CSI (continual service improvement) register. They agree it is a good idea, then nobody does it, or if they do then nobody looks at the list again. If I ask, “Who owns this?“ they usually look at each other the point to the business or IT manager. There is no sense of ownership for improvements. This confirmed our own mini-survey of 650 organizations: Less than 20 percent had formal CSI.
- With improvements, teams wait for somebody to say, “You will do this,” rather than suggest, “I can pick up that.” Improvements mean more work and teams are already busy enough, allocating time for improvements and rewarding improvements are not valued.
- Teams often say they do NOT reserve time to reflect and discuss improvements. They are often too busy and managers do not stimulate, or allow them to reserve time for thinking and acting on improvements to the way of working. Priority is given to completing the ever growing backlog of work.
Knowledge Areas

Business Value Optimization:
- Many technology people do not understand the business they are supporting, the critical business activities and business times, how a system delivers value to the business, what happens at a given time if a critical IT feature, function or technology is not available. There is a technology mindset as opposed to a business mindset.
- Many technology teams have never seen a user or seen how users or customers use and experience the technology. Many think they don’t need to know this. This is a top-scoring ABC (Attitude, Behavior, Culture) card worldwide in global workshops and has been for 15 years. “IT has too little understanding of business impact and priority,” and is also a top concern for business managers.
- In every single simulation and workshop I conduct worldwide, the issue of prioritization is raised, again in relation to business impact and business value, yet all teams recognize a growing pressure on scarce resources, a dilemma between “innovation” and “maintenance work” and a need for clear prioritization.
- Product managers in teams are often unaware of technical debt and its impact. ITSM teams have difficulty quantifying the impact of technical debt and how this impacts current and future business plans.
And finally, the latest “State of DevOps” report also confirmed the importance of skills and behaviors, particularly what it calls “Transformational Leadership”: “By 2020, half of the CIOs who have not transformed their teams’ capabilities will be displaced from their organizations’ digital leadership teams.” The survey continues: “Leaders are the ones who set the tone of the organization, and reinforce the desired cultural norms.” Oh! There is that “C” word again. Are we ever to escape it?

What are the characteristics of a transformational leader, according to the State of DevOps findings?
Vision, inspirational communication, intellectual stimulation (challenges people to think about problems in new ways), supportive leadership, personal recognition. These characteristics are one of the reasons why leadership is one of the four key Skills & Behaviors in the DASA competence model.

Conclusion

A simulation experience helps make the “C” word become a reality. Delegates can translate it from a buzzword into meaningful behavior. The simulation allows delegates to see, feel and experience the benefits of communication and collaboration, while helping them to translate DevOps theory into practice.

As a leader, you can use a simulation exercise as an instrument to communicate your vision in an inspirational way. End-to-end stakeholders can be brought together and stimulated to explore and develop DevOps skills, knowledge and competences together. At the end of the simulation experience teams have been empowered to capture takeaways they want to apply. As a leader, you can support them in translating the captured takeaways into real behavior and helping shape together a DevOps culture for your organization.

About the Author / Paul Wilkinson

Paul has been actively involved in the IT industry for more than 35 years. Fulfilling a wide range of roles ranging from computer operator to IT infrastructure operations manager. He was co-author of the ITIL publication, “Planning to Implement IT Service Management,” and was a member of the ITIL advisory group for ITIL Version 3, and the architects team for ITIL Practitioner. Paul is also co-director and owner of GamingWorks, the company that developed the internationally renowned “Apollo 13 – an ITSM case experience” ITSM simulation game and most recently the “Phoenix Project – DevOps business simulation,” which is based upon the Phoenix Project book . He was also co-author and developer of the “ABC of ICT (The Attitude, Behavior and Culture of ICT)” publications, having conducted ABC workshops and simulation workshops with delegates representing more than 4,000 organizations worldwide. Follow him on Twitter or connect with him on LinkedIn.
August 29, 2017
10 Ways to Make Outage Mitigation Fun
It’s 3:47 a.m. You and the rest of the Ops team have been summoned from your peaceful slumber to mitigate an application delivery outage. As you catch up on the frantic emails, Slack chats and text messages from everyone on your international sales team (Every. Single. One.), your mind races as you switch to problem-solving mode. It’s time to start thinking about how to make this mitigation FUN!
1. Gently remind everyone that the SaaS-delivered automated application delivery strategy you proposed would have prevented this. No need to rub it in, but … if you had turned on a software-defined application delivery platform for your hybrid infrastructure, you’d all be sound asleep right now. Automated real-time delivery decisions and failover would be nice, right? Just sayin’.
2. Drop a generationally divisive Giphy in the emergency Slack channel. Your coworkers’ opinions on gaming consoles, Spiderman movies and music are different than yours! The perfect animated GIF will remind them of your erudite tastes, while you have their attention. Extra credit if you can work in some low-key shade about not listening to your equally sophisticated opinions on optimized outage mitigation.
3. Hold an impromptu cloud health dashboard page customization contest. Oh hey, look at that! Your cloud provider’s health dashboard page says everything is fine … because it’s powered by the services that went down. Help your team vent their creative energy (and frustration) with some fun customized MS Paint updates to the offending page. Bonus points for art that reminds everyone of the value of a multi-cloud strategy powered by a programmable application delivery platform.
4. Use health metrics to cure ills instead of just conducting another really good post-mortem. Tired of “learning lessons” from these emergency room drills? You depend on NGINX for your LLB, but don’t have a way to use those LLB health metrics and data to automate global delivery. Disjointed delivery intelligence means you don’t know how your apps will land with users. You need an end-user-centric approach to app delivery that automates the best delivery path and ensures failover is in place at all times. Micro-outages often fly under your passive monitoring radar, but that doesn’t mean your users don’t notice them. An active, integrated app delivery approach re-routes automatically before you lose business. Post-mortems are fun … but so is making sure your apps survive the last mile. Arguably more so?
5. Open (and then close) some tickets for the sales team. “Sales needs to sell more stuff.” You’ll feel better.
6. Build a countdown clock for when you can finally replace your old ADC hardware. Sure, your Mode1 ADC hardware was a sunk cost so you’re stuck with them for a while. But you’re one unnecessary emergency closer to having a fully software-defined application delivery platform for your hybrid cloud. And now you’re even closer. And closer … Tick. Tock.
7. Leverage real user measurements (RUM) to foresee and fix problems before they happen. Probably best to do this during normal work hours. User experience data from around the world can detect degrading, sluggish resources in real time and user-centric app delivery logic powered by RUM can make quick rerouting decisions automatically. No more getting woken up after the application crashes. While you’re wishing you had RUM on your side, you can look up some fun facts from the countries experiencing app outages. Did you know Luxembourgish is an official language?
8. Send an email. You, too, can be the weird colleague who sends emails with crazy, middle-of-the-night time stamps.
9. Do some virtual window shopping. Browse around to see what you could have purchased with the money you were just forced to spend on unplanned cloud instance provisioning to keep your app running. That desktop Nerf missile launcher (or 700 of them) would have been pretty nice.
10. Swear off the game of chance. You’ve just proven it’s not that much fun after all. Don’t just dump everything onto one cloud and call it done. Clouds go down for so many reasons. Use an application delivery platform control layer to build in the capability to auto-switch to an available resource, while you sleep soundly. Running on multi-cloud without an abstracted control layer removes most of the value of the cloud. Swear off the game of chance. Out loud. Right now.
About the Author / Andrew Marshall

Andrew Marshall is the Director of Product Marketing for Cedexis, and has driven product marketing efforts at New Relic, Amazon Web Services and Microsoft. Connect with him on LinkedIn.
August 25, 2017
Codeless Automation: Shattering the Testing Bottleneck
In his article, “Moving from ‘Tester’ to ‘Quality Enabler’,” Scott Edwards clearly lays out the predicament in which QA testing is currently at: with one foot in the quicksand of manual testing, while the other tries to jump over the hurdle of automation. Ever tried hurdle racing with one foot? You probably won’t reach the finish line.

Edwards mentions the infamous label that is attached to testing—bottleneck—and cites the “DevOps Review 2017” research paper of Computing.co.uk that found two-third of DevOps practitioners report QA testing as the bottleneck of the development process.

Why is that? Why is testing bottlenecking development, and how can it be untangled?

What Creates the Development Bottleneck?

The testing bottleneck is created primarily due to two reasons:
1. The reliance on manual testing, and
2. “All the code that is getting thrown” QA teams’ way.
Manual testing is extremely time-consuming, sometimes to the point that it’s close to impossible for test teams to keep up with the fast pace of DevOps. Manual testing is also error-prone, simply because the testing is performed by humans.

The code issue is related to automation. Due to the pressure imposed upon test teams, automation is being hailed as the great savior. While the word “automation” has the flair of hands-free simplicity, the reality of automated testing is quite the opposite.

Automation relies on code and testers aren’t coders. So instead of simplifying and speeding up testers’ work, the opposite is being achieved. Testers are thrown out of their comfort zone, being bogged down by tasks that aren’t in their “job description.” The frustration of all involved is easy to understand. The development team feels that testing is slowing them down, while the testing team feels they are required to jump through hoops.

Add to that the complex setup and heavy maintenance required of automation, and you start to doubt whether it is indeed the right path for testing.

Well, it is. With the ever-faster pace of DevOps, the pressure to shorten release cycles and the cutthroat competition software and web application companies are facing, test automation is the right path, despite all the challenges.

What must be done to clear that path is remove coding from automation.

Codeless Test Automation to the Rescue

Codeless is a growing force in mass technology. It has liberated the web. Codeless platforms such as Wix and Shopify allowed millions of businesses to create an online presence quickly and effortlessly and led to the incredible expansion of the web. On the enterprise side, codeless platforms are making strides with the likes of Salesforce, FileMaker and Nintex, which offer drag-and-drop solutions to speed up and optimize efficiency of large-scale administrative tasks.

The idea behind codeless is the modern-day “power to the people.” Just a decade ago, a small family business from Minnesota would have never been able to afford and maintain an online presence, but now their 22-year-old can set up an online shop, promote it on Facebook and in no time start selling their goods to a consumer across the ocean.

Applying the same logic to testing instead of coding from scratch test after test—which is time-consuming and costly, and requires acquired skill—codeless test automation allows for quick and simple creation of tests, no coding skills required. It allows QA testers to focus on what they do best: testing.

Automation shouldn’t be a hurdle that testers need to jump above; it should be a tool that pushes them forward. As things stand now, automation in testing is at a counterproductive standstill. Don’t get me wrong: Automation means well, but at the moment it’s like a misguided teenager that brings havoc even when he aims for harmony.

How Does Codeless Test Automation Actually Work?

Codeless test automation functions very similarly to enterprise codeless platforms. Both take complex tasks traditionally achieved by coding and simplify them through automated code generation.

With test automation a tester creates a flow by visually binding elements that represent “clicks” in the software or web application. The tester simply goes through motions of the scenario he wishes to test, and the code for the test is generated under the hood.

We can get more technical than this, but there’s no need. That’s exactly the idea with codeless, isn’t it? Testers could focus their efforts on testing functionality and evaluating the user experience rather than struggling with test creation.

This ties back to Edwards’ main point: Testers get so entangled with test creation that the whole point of testing—quality assurance—gets lost.

Side Note: The proposed job title change suggested by Edwards—Tester to Quality Enabler—might sound merely cosmetic, but it can deliver real value. As illustrated in this collection of QA testers’ venting from a reddit forum, QA testers often harness feelings of underappreciation and of being looked down on by their developing colleagues. Recognizing testers’ valuable contribution to the overall quality of the product will surely elevate their feelings of self-worth and enable them a stronger sense of pride in their position.

Using Codeless for Deeper Integration of Testing Into DevOps

Codeless automation is best utilized in sanity and regression testing, the two most basic and frequently applied tests. Sanity testing ensures that all incorrect inputs a user could enter are accounted for. Regression testing is like inventory counting, in a way: you just need to go over your stockpiles and see that everything is in order.

Thinking on inventory: Let’s say I have a huge warehouse full of thousands of items. Checking inventory by hand is a very time-consuming and error-prone process. How many boxes on a piece of paper (representing real boxes on shelves) can you tick before you’ll make a mistake? Now think of Amazon’s warehouse. Inventory counting is done using automation with the press of a button. It is instantaneous and error-free. It’s an action that’s been done so many times before, there’s no need to plan it—or to code it—all over again. Simply run the same command—“inventory”—and you’re done.

If you’re using codeless automation, all you need for regression testing is to click a button and the scripts run. You are done and able to divert your attention to sanity testing, making sure the new iteration is solid.

For the next iteration, you simply pile the sanity testing of the previous round on top of the regression scripts and, again, you’re done.

Continuous testing is getting a whole new meaning with codeless automation as part of the equation. If regression testing is done with the click of a button, and sanity taken care of simultaneously, the reporting back to development is swift and focused and the bottleneck is shattered.

Codeless enables deeper integration of testing into the development process simply by arming testers with automation tools that simplify tasks, not complicate them.

About the Author / Assaf Dudai

Assaf Dudai is the content marketing manager at TestCraft, a pure SaaS continuous test automation solution for QA professionals. Assaf is mostly looking into the future of testing, how automation, codeless and machine learning can and will help testing evolve. You can connect with Assaf on LinkedIn.
August 25, 2017
Using Application Deltas in Deployments
Developers have been versioning their source files since the early days of Unix. Anyone else remember SCCS? Back in those days, storage was at a premium: Disks were small and very expensive and it was too inefficient to store a complete copy of each version of the source file. So version control tools would simply store deltas—the changes made from one version of the file to another. This was more efficient; if you only changed one line in a 3,000-line file, the system would simply store the new line and some metadata about where it fit in the file and who made the change.

Some tools would save the latest version of the file and use reverse deltas to record the changes if you wanted to revert to an earlier version. This was usually more efficient, since generally developers want the latest version of a file. As such, it was more efficient to apply deltas to get earlier versions than it was to start with a 3-year-old file and roll forward through 472 changes to get the file you wanted.

These days, disk storage is literally 3 million percent cheaper than it was in 1981. (The storage in my phone would have cost $38 million in 1981!) Because of this, storing deltas is not the panacea it used to be. In addition, CPUs now can crunch numbers faster than they used to, so it’s possible now to use math to compress and store a complete copy of the new version of the file. That’s what modern revision control tools such as Git do.

Now that’s all very interesting, but the title of this post is, “Using Application Deltas in Deployments.” So why am I rattling on about source code versioning?

Well, here’s the rub: While deltas may no longer have much of a part to play in tracking changes to source files, they most certainly do when it comes to deploying applications, and for similar reasons that plagued us (or our parents!) 30 years ago.

Applications are Made of Components

Let’s be frank: An application that consists of a single binary on a single server is not going to need deltas. That would be akin to a change to a single-line source file. All you need to do is to replace the old binary with the new and you’ve deployed. But applications are getting more complex:
- The application may be split across multiple servers. An n-Tier application may have different tiers running on different servers (and even different operating systems).
- The application could be wholly or partially containerized. You can have multiple containers each running different microservices.
- There could be database changes associated with an application change. Indeed, the application change could consist solely of a database change. Database updates are pretty much always deployed as deltas; alter scripts are used to add and remove columns, tables and indexes.
In the same way a single binary can be built from lots of different source files (each of which is under its own version control), an application can be made up of lots of different binary components. Clearly, if the application consists of lots of different components and those components are resident on different servers or containers, then deploying every component every time a new application version is deployed would be wasteful, time-consuming and error-prone.

An application version “delta,” therefore, would track which components have changed between the version of the application being deployed and the versions of the components on the target environment. This allows the deployment tool to only deploy the components that have changed. So what do we need to track to make this work?
- Component Versions. Obviously, we need to version each individual component so we know which version of which component is included in any particular application version.
- Which component version is present on any particular endpoint within a target environment. That way, if the application we’re about to deploy has version 5 of a component (a DLL or a WAR file, say) and version 5 is already present in the target environment, then we don’t deploy that particular component.
This approach has a number of advantages. You can deploy to an existing environment and only deploy the components that have changed, minimizing the downtime and reducing the risk of failure. You can deploy to a newly provisioned environment and deploy all the components within the application. Then, if you need to fire up a new virtual machine (or a container), the deployment pushes everything. But the next time you deploy to that environment, only the components that have changed are deployed.

Database Deltas

We may need to consider special cases when it comes to database components. Remember we mentioned that database changes are nearly always delivered as deltas, SQL “alter” scripts that make changes to the previous version of the database schema? Well, if all you’re doing is deploying the next version of the application, then that works fine. However, what if you’re deploying a version that’s several releases ahead of what’s in the environment? This can easily happen if the test and production environments are “pulling” deployments into their environment (as opposed to having them “pushed” via a continuous delivery process). Now, just applying the alter script that is associated with this application version won’t work in any environment that doesn’t contain the previous version of the application.

Picture an application (“My Application”) that is up to version 6. It has two components: a WAR file and a database component that contains alter scripts that roll the database schema forward. Here is what the last four versions look like:

So version 3 has a new version of the WAR file; version 4 has a new version of the WAR file and a database alter script that adds a column to a table. Version 5 only changes the database (it amends a stored procedure which uses this new column). Version 6 applies a new change to the WAR file.

Now, in a continuous delivery process, the test rig being targeted will receive each new version. So when version 4 is deployed, the rig will receive version 4 of the WAR file and the alter script will be executed to add the column to the table. Then, when version 5 is deployed, only the DB alter script will run (alter.sql;2) to amend the stored procedure. The WAR file is not deployed, since the server already contains version 4 of the WAR file. When version 6 is deployed, only version 5 of the WAR file is deployed. This is exactly what we want.

But now what happens when we move to user acceptance testing (UAT)? By this stage, we’ve done as much automated testing as we can get away with; in UAT, real users need to log on to our test system to make sure they’re happy with everything. So we’re not going to do a “push.” In UAT, the test lead does a “pull” when they’re ready to accept the new version of the application for testing. So they pull “My Application Version 6” into their test environment and get their testers lined up with coffee and pizza.

But what happens if the test environment is currently on version 3? Well, for the WAR file, there’s no problem—there’s a difference in the component version for the WAR file so the new version is deployed. But what happens to the database? There are no alter scripts associated with version 6 of the application, but without applying alter.sql;1 and alter.sql;2, the database schema is not going to be valid for use with myapp.war;5. The testing will fail, not so much “falling at the first hurdle” as “falling in the paddock.”

Deltas allow us to cure this issue. What we need to do is identify the components that represent the database changes and get them to be applied in sequence for each interim version. In that way, successive database “deltas” are applied to roll the database forward to the correct schema version before the required version of the application is deployed.

So, in our case, going from version 3 to version 6 would deploy the following components:

So we will deploy (and run) alter.sql;1 to add the new column, then alter.sql;2 to amend the stored procedure and, finally, deploy myapp.war;5. Deltas mean we do the minimum required to get the application version to the desired state.

Conclusion

Versioning the components that make up an application means it is easy to determine the deltas (changes) between one application version and another. Recording what version(s) of the components are on the endpoints in a target environment makes it easy to only deploy the components that have changed, minimizing downtime and making the deployment quicker and less error-prone. Identifying the deltas for a database gives us the ability to roll databases forward and jump between releases, which is invaluable when transitioning from an agile “push” to a waterfall-centric “pull” deployment model.

— Phil Gibbs
August 24, 2017
4 Predictive Analytics Challenges Facing SRE Teams
“Predictive analytics” refers to the many techniques used to analyze data to make predictions about the future, such as data mining, statistics, modeling, machine learning and artificial intelligence. Being able to predict technical and business outcomes is one of the main promises of big data. Indeed, many businesses have invested heavily in infrastructure to anticipate key performance indicators such as demand, pricing or maintenance. The investment required is substantial but, when it works, it yields a very positive ROI.

In this post, we’ll examine why implementing predictive analytics in a site reliability engineering (SRE) context requires a significant investment in both setting up the right data sets and applying the right logic to that data.

SRE should in theory be a prime target area for predictive analytics. Why? Because companies have already widely adopted monitoring tools over the past decade. In doing so, they’ve amassed extensive system health and behavior data that is ripe for analysis. SRE teams have also increased their ability to make sense of this data as data science tools have become more accessible. Basic data science aptitude is increasingly become a desired skill when hiring engineers.

Additionally, the SRE team is responsible for a mission-critical KPI: system availability. This single metric is directly tied to customer SLAs in digitally dependent businesses. SRE teams have a strong incentive to implement predictive analytics. Why? Having the ability to prevent downtime by addressing potential issues before they hit the production environment is hugely valuable. More system availability translates into more revenue, higher customer satisfaction and lower costs of problem mitigation.

However, the reality is that an SREs job is still mostly focused on the diagnostic, mitigation and “fixing” parts, as opposed to more proactive tasks. This is in big part because predictive analytics applied to SRE/DevOps has proven difficult to implement and perform in a disciplined manner. Therefore, it is still an aspirational goal at most companies.

SRE Challenges in Predictive Analytics

Up next, let’s explore the four challenges that need to be overcome to implement high-quality predictive analytics practice in an SRE context, including:
- Data collection
- Data quality
- Data volume
- Model usability
Challenge 1: Data Collection

First, the working data sets need to fit what you are trying to predict. In this case, it is challenging to get the right data set because the SRE team’s mission is very broad; specifically, ensuring the reliability of the entire system. The difficulty lies in the fact that there is no such thing as “full system” data readily available. All the relevant data is distributed across siloed, or overlapping repositories and monitoring tools. Centralizing the data requires either a full system agent (which takes a lot of work to set up) or through APIs, which requires mastering a multitude of API integrations. An additional obstacle in getting the right data lies in the inconsistent nature of events, time series data and logs. These disparate data types can’t be analyzed together unless they can be transformed into a uniform data set.

Challenge 2: Data Quality

SRE environments emit lots of data and some won’t make sense for the purpose of the analysis. Whichever data collection method is used, the information tracked often reflects some level of bias on the part of whoever set up the tracking mechanism in the first place. This leads to capturing both irrelevant data or inadvertently leaving out important information. In other words, the data is often both noisy and lacking at the same time.

The key to successful modeling lies in the selection of prediction variables with data that is obtained before the event predicted happens (in this case, a failure). Defining the variables that lead to the determination of a subsystem failure is critical and requires specific domain knowledge. It is difficult to define and extract a pure data set sanitized of noise. The more data you leverage, the higher the chances that you are actually “feeding the beast” with irrelevant information. Too much irrelevant data leads to inaccurate models and false positives.

Challenge 3: Data Volume

Here you can have two different types of challenges: too little data and too much.

Not enough data

You typically need at least 100 instances of what you are trying to predict and probably at least 100 where it didn’t work to train a model. This assumes the data set is clean and directly relevant to a very narrow, specifically defined problem. In the case of SRE, the scope of issues that can affect a system is such that the data set required is very substantial. Likely, it’s several months worth of a wide variety of system behavior measurements. For example, if you collect data only through passive collection, think about the lag time between t-zero (when you start collecting data) and the time when you would have results. During that time, you have a bootstrapping problem—your model can’t be trained to predict accurately what you want.

Too much data

There are two practical points to make around the notion of “too much data.”
- Lack of standardization: Processing large volumes of data for analysis remains a difficult engineering challenge. There is no standardization for data pipeline; every method has a limit in terms of how many inputs it can ingest on a per-second basis, and ingest rates often end up becoming bottlenecks in large enterprise environments.
- Volume does not equal accuracy: There is a point beyond which you get diminishing returns: statistical theorems show that after a certain point, feeding more data into a predictive analytics model will not provide more accurate results.
Overall, getting the quality and quantity of the training data right is the best investment you can make in building your predictive capabilities.

Challenge 4: Model Usability

From a pure “mechanics” standpoint, the analytical concepts and methods are there to reflect the complex thinking required to predict the behavior of an enterprise’s combination of technology, information and infrastructure. Predictive software relies heavily on advanced algorithms and methodologies such as logic regressions, time series analysis and decision trees. The problem is that the workings of the kinds of systems that SREs deal with are very hard to understand in the first place, let alone predict. It is not uncommon even for SREs to not entirely understand how a system works if they were not the ones who initially set it up. In addition, every company’s system is different.

When data scientists design predictive models without years of experience in an SRE capacity in multiple environments—in other words, without leveraging subject matter expertise—it rarely works perfectly. Often the models they create lack usability, even after the validation process to confirm its quality. A modeler will split the available data into a training set where the model is run to generate predictions and compare performance against the actual outcomes. That will require multiple iterations to generate a model that has high overall predictive accuracy, and 9 out of 10 times the result is overfitting, or perfect for the training data. A subject matter expert in the domain of technical operations can cut the cycles of iteration and get to a predictive model that works with an acceptable level of risk.

In short, implementing predictive analytics in an SRE context takes a significant investment both in setting up the right data set and applying the right logic. Stay tuned for a future blog post that will provide some pointers on how to achieve the best prediction results.

About the Author / JP Emelie Marcos

JP Emelie Marcos is the co-founder and CEO of SignifAI. Previously, JP served as the Chief Operating Officer of Tango, a leading mobile consumer application that reached 400 million regular users in five years. JP’s career spans 20+ years leading teams in technology startups and large companies as an investor, GM and CEO.
August 23, 2017
Puppet Makes Development Kit Available to Advance DevOps

Organizations that embrace DevOps often find themselves moving from having a paucity of IT automation tools to an embarrassment of riches in short order. As a result, many of the teams involved in those DevOps processes wind up making different IT automation choices. Before too long, the organization finds itself trying to bridge too many islands of automation.

Puppet is working to bridge the automation gap between developers and IT operations teams with the release of a Puppet Development Kit that makes it easier for anyone to generate code that can be executed using the Puppet IT automation framework.

Michael Olson, senior product marketing manager for Puppet, says all too often developers automate various tasks using their own favorite tool. When it comes time for IT operations teams that have standardized on Puppet to deploy that code, they have to rewrite various routines. Puppet Development Kit makes it possible for developers to now write code that can be shared seamlessly with IT operations teams that have adopted the Puppet IT automation framework.

In some environments, DevOps has literally become too much of a good thing. Olson says the Puppet Development Kit represents an opportunity to truly unify fragmented processes in a way that developers can easily consume. The Puppet Development Kit includes unit test modules from Windows, OS X and Linux workstations to ensure that Puppet code is creating and managing configuration resources as intended.

The existence of the Puppet Development Kit should also make it easier for organizations that have not embraced DevOps to get started. A recent survey conducted by Puppet finds that organizations that have embraced DevOps are delivering new applications and updates at rates that far exceed other organizations. The longer it takes for organizations to embrace DevOps the wider that gap grows. By making it easier to include developers in more structured approach to DevOps, many companies that are getting started late with DevOps now have an opportunity to close that gap.

Developers don’t always have a lot of respect for IT operations. Many of them have been throwing code over the proverbial data center wall because deploying code in a production environment was always someone else’s problem. Tools such as the Puppet Development Kit essentially steal a page from the same playbook Tom Sawyer used in the book by Mark Twain to get the kids in the neighborhood to help him paint a picket fence.

At the other end of the spectrum, however, are developers who are firmly convinced they can program the IT operations team out of existence now that “infrastructure is code.” In reality, most of those developers quickly discover that the ongoing care and feeding of hundreds of applications deployed in a production environment still requires a fair amount of human intervention. In fact, without some form of IT automation framework in place, deploying applications at that level of scale would not be possible. The challenge is striking the right balance between what can be automated and higher-level tasks that still require human expertise.

— Mike Vizard

August 22, 2017
15 Things Your Competitors Know About DevOps
It’s no secret that DevOps is revolutionizing IT, helping organizations do more, faster. DevOps success is now essential for organizations in their quest to remain competitive. According to the “2015 State of DevOps Report” by Puppet, organizations utilizing DevOps deploy software 30 times more frequently and have lead times that are 200 times shorter. Plus, their failures are limited to 60 times while recovery is 168 times faster.

Here are the 15 things your DevOps-enabled competitors know that you might not be aware of:

Easier Code Evaluations

Code evaluations have become easier than ever. Developers now can move on to tasks and get rid of frequent changes in the code. Successful enterprises quickly react to the changes within the system with ultimate results. This enables greater benefit for the end-user applications.

Quality Software

DevOps training is all about transforming the way businesses think about software.

DevOps elements are meant to be employed in individual teams or projects. But, the real success of the organization lies in the support from the whole. Successful organizations change the way its IT operates, supports and leads the enterprise.

Different Definition

Successful DevOps adoption is different for different organizations. Whether it is continuous delivery, quality or speed, every enterprise defines DevOps differently. It depends on you how you define DevOps that meets your enterprise necessities in a clear and consistent tone. Focus more on the results than the methods.

Continuous Delivery – Streamlining development and avoiding the hurdles to achieve efficient functionality.

Release Speed – Effectively meeting the expectations of consumers.

DevOps is Beyond IT

DevOps has consistently evolved beyond the typical IT concept. It involves software, applications and tech-based services. It brings together the responsibilities of everyone in the company, and not just IT.

This way the whole enterprise collaborates more closely on a defined DevOps platform. Complexities are effectively eliminated due to fewer barriers between IT Operations and Development teams.

Transforming the IT Culture

Transition to a DevOps operating model mainly depends on tools and principles. However, DevOps is all about changing the enterprise IT culture rather than just tools and processes.

DevOps culture is all about removing the barriers and getting the whole team at one place, which further, improves the work efficiency. It is necessary that everyone—within and outside IT—must understand the importance and competitive advantages of a business to become more agile.

Digital Transformation

Digital Transformation is in high demand in recent years. Its foundation lies in applications that constantly create new channels that encourage customers and keep employees productive. DevOps is critical for the success of application strategy and forms an important element of digital transformation.

Holistic Approach Toward Metrics

Metrics are the parameters that guide any kind of organization. Your DevOps-enabled competitors are successful because they effectively measure the metrics and change them accordingly to improve growth in the industry.

The right metrics make sure that DevOps transformation is working. Make a full-dimensional approach of metrics including:
- Mean Time To Resolution (MTTR)
- Mean Time Between Failure (MTBF)
- Process and People Metrics
- Active Users
- Development-to-Deployment Lead Time
Incorporating Automation

Agility plays an important role in all processes of the DevOps operating model. To make the process more agile, it is essential to employ automation for repetitive tasks. This maintains the workflow of the whole process.

Automation has expanded to all stages, including, development, testing, production and operation teams. The kinds of automation are:
- Release Automation – Using tools such as Chef and Puppet.
- Runbook Automation – Application performance management and monitoring solutions.
- Automation achieves desired quality goals by maintaining compliance. It reduces costs and delivers a faster response.
Feedback and Feedforward Cycles

DevOps promotes agility through healthier integration within the organization. Feed forward and Feed backward in the development life cycle is critical in determining this agility.

Different functions of the company interact smoothly only when tools lay their focus on software development process. This provides information on the content that is relevant to each audience. Context relevance enables easy creation of dashboards and present this information to the audience in a way which makes sense.

Real-time Approach to APM

DevOps is more about culture than just tools. Right tools and right actions highly improve odds of success and evade negative impact.

Solutions you choose must provide visibility into future problems. Software development and implementation must be maintained in a volatile, fast-paced environment that requires production-ready APM solutions. Your competitors are highly efficient because they are aware of the trick: monitoring applications down to specific functions. Working together ensures quality and makes it easy to predict the success of applications.

Application Analytics Improve Software Strategy

Analytics plays an important role in improving software strategy. Utilizing analytics solutions prior to quality user expectations achieve competency.

Software is not all about just software; it is about how it connects with customers to generate revenue. Ideal analytics are capable of connecting application performance with user management and business data. Be sure to define the vision and establish organizational priorities within the budget.

Customer-centric and Innovation Cocused

DevOps improves the communication between developers and IT Operations by building customer experience. DevOps provide value under pressure to the enterprises. Having DevOps within an organization is a definite necessity. It always upholds the competitive advantage and effectively delivers the software to the customers.

Study Case Studies

If you are the one aiming to make bigger by employing DevOps in your organization, read case studies and real-world examples. This effectively manages the DevOps transition for any size of the enterprise.

Macro Environment

Budget, roles and responsibilities and departments have a come a long way. Product releases have been reduced from annually to quarterly, monthly, weekly, daily and now to several times a day. Microbuilds, microreleases, microupdates, microtesting and microcheck-ins have now become an easy task.

Welcome Ops Movement

Don’t stop with the existing movements. Go and explore the Ops and its related tools: viz. DevOps, DevSecOps, BizOps, etc.

Final Thoughts…

It is clearly evident that investing in DevOps culture is pure benefit. DevOps is here to transform the way organizations develop, monitor and handle the applications and its underlying infrastructure. It has readily progressed from a niche concept to a business imperative. Every enterprise, small and large, aspires to integrate DevOps tools and principles. Employ the above points to successfully drive the DevOps conversation within your enterprise.

About the Author / Sirisha Paladhi

Sirisha Paladhi is a content writer at Mindmajix. During her career, she has written many articles on technology innovations. Connect with her on LinkedIn.
August 11, 2017
Cloud Capabilities Turn Developers into Operational Experts

The upside-down world of digital transformation can leave you feeling like you’ve fallen down an internet rabbit hole. If you’re a DevOps denizen, the swirling pace and scale of virtualization might give you that “lost in the Technicolor forest” feeling when it’s time to release an app into the wild. “Curiouser and curiouser!” you say as you try to sort through hybrid combinations of private and public clouds, CDNs, and data centers.

As a developer, you worked hard on the design, build and test—now you want to make sure it takes the best, fastest path to microservices and users. It’s folly to trust the answer from any one source—synthetic measurements may purr like a Cheshire Cat over hardware health, meanwhile the White Rabbit of RUM (real user measurements) is spazzing out over latency and traffic jams. The lesson Alice finally learns in Lewis’ timeless tale? She must learn to control the journey herself.

Developing for the Complex Cloud

Fortunately, there’s a way for developers to mix that “just right” solution that will reveal the right path through the internet forest. This used to be the purview of operations—setting up servers and delivery conduits. In the pre-cloud era, developers just wrote code. Somebody in Ops pushed it live, and that part of the process was opaque to the developer. With the proliferation and maturation of the cloud, the Ops team is increasingly focused on dynamic infrastructure and services. Developers routinely spin up their own servers. As DevOps teams focus more on microservices and containers and less on “stacking and racking” servers, performance, control and delivery become the focus areas that matter.

As the division between Dev and Ops experts continues to dissolve, each becomes increasingly aware of the other’s roles, responsibilities and skill sets. The goals of agile development, non-stop testing and optimized delivery become the shared destination; paths merge until everyone is creating momentum around the prized feedback loop. As infrastructure and services become more abstracted, the focus is less on writing original code and building stacks and more on selecting resources. The selection and use of these resources has to be strategically guided by real-time intelligence to meet multiple performance and business objectives. The more abstract and hybridized the infrastructure becomes—and it is moving rapidly in that direction for every business—the harder it is to navigate. If you can’t keep all your resources straight, you can’t optimize them. Centralizing and automating tracking and traffic management is mission-critical.

The Delivery is in the Code

Which brings us back to the developer’s mandate to control and optimize delivery from the application. Cloud-based delivery tools combine real-time user measurement with scriptable application traffic shaping and built-in cost controls. Gone are the days of blaming poor performance on the network. End users expect seamless, instantaneous experiences, no matter what. This is a tall order for modern apps that combine video, microservices and dynamic content. To ensure high quality of experience (QoE), you have to automate and optimize with software-defined delivery. Developers can now contribute to optimization by coding in delivery intelligence.

This isn’t merely another task for the developer; it’s an opportunity. Developers are now empowered to deliver the functionality of their product or service exactly as designed. Developers are now actively part of the process that brings together global and local traffic management data (RUM and infrastructure health), application performance management and cost considerations to ensure the cheapest, fastest path to that microservice. The balance and mix of these factors can be customized and fine-tuned by the developer (a.k.a., the newly minted operations expert).

The Bleeding Edge Becomes the Bedrock

You know the cloud is no fairy tale when 74 percent of tech CFOs claim that this year the most measurable impact on their business will come directly from cloud computing. Spending on cloud computing is set to grow at 19 percent CAGR through 2020, six times the rate of overall IT spending. Developers who can harness the magic of hybrid cloud/CDN for video, web content and application delivery will be rock stars. Picture it: fully automated delivery based on business rules, actionable data from real user and synthetic testing, and self-healing network optimizations. Developers can push their capabilities and control right to the bleeding edge, making them an even more agile contributor to the continuous improvement cycle.

Soon enough, as technology progresses and infrastructure is increasingly software-defined, these skills will become a requirement. Over the next few years, this data-driven approach to app and content delivery will be considered a standard capability for digital businesses and a core responsibility for developers. The forward momentum of organizations that have committed to both cloud-first and DevOps-first strategies is powerful and will quickly boost early adopting competitors out of reach. The number of developer-operators capable of managing application delivery in capacity-aware, consumer-responsive, application-tuned ways will soon build to critical mass. Don’t get left behind, wandering lost in the woods.

About the Author / Andrew Marshall

Andrew Marshall is the Director of Product Marketing for Cedexis, and has driven product marketing efforts at New Relic, Amazon Web Services, and Microsoft. Connect with him on LinkedIn.

August 7, 2017
Why Your Manager Will Want to Send You to Jenkins World 2017

Jenkins World 2017 is right around the corner. The conference will be held in San Francisco Aug. 28-31 and will be the biggest event of the year for Jenkins users, with approximately 2,000 people expected to be in attendance! You will not want to miss out on this event, full of learning and networking opportunities.

Need some help convincing your manager why you should attend? No problem—we have laid out all the reasons why attending would be beneficial to you and your company. Once you have convinced your manager to attend, don’t forget to register today! (As a media sponsor for Jenkins World, we are providing a special offer for our readers. Register with this promotional code and get 20 percent off your registration: JWDEVOPSCUST.

Educational Value

Jenkins World will be offering more than 60 educational sessions given by real Jenkins experts—a chance to learn from the most experienced users. Did we mention that you can also receive a free Jenkins exam with the purchase of a full conference registration? Imagine the benefits for both your résumé and team.

At the conference, you’ll be able to take advantage of great training sessions to better enhance your Jenkins capabilities. The training sessions include Jenkins Certification Training, Introduction to Jenkins, DevOps Leader Certification Training and DevOps Test Engineering Certification Training. We recommend highlighting to your manager what sessions you will want to attend in advance. Check out the full agenda here.

Jenkins World is offering keynotes from many industry leaders. Be sure to emphasize to your manager the importance of hearing from the very best in the Jenkins community, live on stage and in the breakout sessions. From Jenkins founder Kohsuke Kawaguchi to CloudBees CEO Sacha Labourey and featured speaker Jez Humble, your manager will greatly appreciate the knowledge you bring back to the office from listening to such experienced industry experts.

Networking Opportunities

The Jenkins World conference will provide numerous networking opportunities. Learn what similar problems and challenges other organizations are facing in the industry. Make it known to your manager that you will be exposed to people like you, with like business needs and challenges, that you will be able to connect with.

Prove ROI

When convincing your manager to let you attend, it is first important to make them aware of the potential ROI of this opportunity. Detail exactly your expenses including airfare, hotels and meals. This will better help your manager understand the exact cost to send you to Jenkins World. Make sure you also present how this knowledge will help push you and your team forward, ultimately propelling your entire organization forward. The ROI will be evident – make that clear! (And the ROI just got better, because devopsy.kinsta.cloud readers can use this discount code and get 20 percent off: JWDEVOPSCUST.)

We hope this provides enough information to help you convince your manager to let you attend! We have also created an email template to use when sending all this over to your manager for review. Once you have successfully convinced your boss, go register!

— Alan Shimel

August 1, 2017