devopsy.kinsta.cloud

Tag: automation

Predictions for DevOps Security in 2018

Automation continues to be a major driver of DevOps as a whole. In 2018, we’ll continue to see a shift in how and where automation is applied in the DevOps life cycle. More specifically, DevOps will move toward adoption of integrated automation and security orchestration solutions. The need to move away from a reactive DevOps security model will see more teams integrating new solutions that leverage machine learning, analytics and orchestration that can provide incident management insights in real time. Based on that approach, here are three security predictions for DevOps in 2018.

Continuous Response Will Become a Fundamental Part of the CI/CD Process in 2018

DevSecOps will increasingly integrate security into the continuous integration/continuous delivery (CI/CD) process through continuous response using automation that eliminates manual processes. This continuous response as part of the CI/CD chain will go beyond ensuring that an application satisfies certain defined baseline security needs to automated threat hunting that maximizes security team response accuracy. By injecting security monitoring on the code level, continuous response can happen right from development to pre-production to the ongoing use life cycle. When this is done right, responding to security vulnerabilities becomes easier during each step of the process, reducing the time taken to fix or mitigate vulnerabilities.

DevOps Will Transition from Reactive to Predictive Security Processes

As market sophistication increases around cloud security, integration and incident lifecycle management, companies will become much more proactive at managing security within the DevOps life cycle. The need for integrating security orchestration into the development process rather than reacting to issues once a project has been deployed to production is clear based on the huge costs of reactive measures. As these platforms and tools become more available in 2018, companies will begin adjusting their DevSecOps mindset on both a cultural as well as products level to create a more holistic, inclusive process.

We’ll See Solutions that Integrate RASP with Security Orchestration

Having a comprehensive security solution for web applications with heightened detection, visibility and remediation capabilities will be key to response acceleration of web application vulnerabilities in 2018 and beyond. By extending security orchestration to web application security, developers and security teams will be capable of closing the interoperability gap. The key will be an increased visibility into the application and context of which attacks are happening by protecting web applications and API services in real time.

Integrated solutions will enable users to design and deploy custom security rulesets based on the rich data derived from analytics to improve RASP solutions through expanded orchestration capabilities. The result will be a much-improved web application security layer capable of identifying and isolating active attackers, separating breaches from attack attempts and creating and enforcing active protection policies for better application protection. This ability to respond to escalating web-based attacks will eliminate days, weeks and months of manual tasks.

— Rishi Bhargava

January 10, 2018
When is Big Automation Too Big for Comfort?

There’s a joke traveling around the tech-o-sphere that goes like this:

How many software developers does it take to change a lightbulb?

Answer: None, it’s a hardware problem.

In the world of modern DevOps, physical infrastructure has been abstracted to code. I can count on my fingers the number of people I know who have done a physical memory upgrade in the last year. Fortunately or unfortunately, depending on your outlook, I am one of them. Nonetheless, for the most part, hardware left the building a long time ago, no pun intended. All that remains is the laptop we use to move the bits around.

However, for those who don’t sling code, the world of automation is much bigger than our software-centric view. Most of us are aware that today there is hardware that can roam around our living room unassisted and vacuum up dirt from the floor. There’s 3-D printing, which is a technology my dentist uses extensively to fabricate crowns in the privacy of a closet next to an examination room in his office. And of course, the driverless car and truck is predicted to soon be a regular part of our day-to-day landscape.

But, this is the small stuff. There’s a lot of hardware automation that goes well beyond the dimensions of consumer appliances and room-size automation technologies being used in factories. In fact, machine automation is becoming gigantic, as in Godzilla-size, particularly in mining.

Mining: Let’s Eat a Mountain

Equipment has been essential to mining since the first days when a man was sent into a cave with a pick and shovel to extract ore. Mining has come a long way since then—today, it’s a gargantuan undertaking that has become more incredibly machine-intensive. Whereas, in the past, we’d send men into a shaft in the ground to get the goods, these days, we just let a machine eat the ground, literally.

The Bagger 293, shown below, is a bucket-wheel excavator. The 315-foot-high machine can dig a 8.5 million cubic feet of earth per day.

The Bagger 293, with a height of 315 ft, is taller than the Statue of Liberty

To give you a sense of how big this whole operation is, first understand that, 315 feet, the height of the machine, is approximately the equivalent of the height of the Statue of Liberty, from ground to torch top. The amount of dirt that the machine can remove in a day is equivalent to digging an eight-level sub-basement beneath the floor of the Roman Colosseum. The Bagger has been in use since 1995, and no matter how you cut it (again no pun intended), that piece of iron has eaten a lot of dirt.

Presently, the Bagger requires a crew of six to operate. But, if trends keep steady, it’s only a matter of time until machine excavators of this size can operate autonomously.

Sound far fetched? It’s not, really. The Komatsu AHS already has a mind of its own.

The Komatsu AHS is an autonomous earth-moving vehicle that can move a payload of 290 tons. The machine is the size of an average house.

The Komatsu AHS (right) is an autonomous earth moving machine

In the past, it would have taken a small army of men weeks, if not months, to mine raw materials. Today, using technologies such at the Bagger 293 and the Komatsu AHS, the work can be done with a workforce of six. And, in the not-too-distant future, the work will require no human intervention whatsoever.

So we’ll have machines that can get massive amounts of dirt out of the ground and move it around without the need for any human intervention, But, once mined, where does it all go?

Hold that thought. The answer is coming.

There’s More to Container Automation than Kubernetes

Yara, a shipping company out of Oslo, Norway, in partnership with Kongsberg, a maritime engineering group also out of Norway, has created an autonomous container ship, the Yara Birkeland, that is set to hit the high seas in 2018. This ocean-going vessel will be manned by a crew of none. It’s completely driverless.

The Yara Birkeland, to be completed in 2018, is a zero-emission, autonomous container ship that is estimated to remove up to 40,000 truck journeys

According to the Bureau of Transportation Statistics, more than 19,000 vessels entering U.S. ports in 2010 were container ships. That’s 31 percent of total maritime activity on American docks that might very well see the elimination of seafaring crews.

Container shipping is useful because it allows for the easy transport of goods from a variety of shippers. In the past, dockside longshoremen loaded individual bails, crates and barrels into the hold of ships by hand using winches, conveyors and stackers. Container shipping made it so the shipper loads a container and then a machine loads the container onto the ship like stacking Legos. One human does the work of dozens.

Mining transport does not have the burden of having to accommodate a variety of goods in different sizes and shapes. There is only one product: raw ore. Thus, it’s quite conceivable that an autonomous container ship, maybe one the size of four soccer fields, can be repurposed as a giant, seagoing container holding nothing but a single product: unprocessed ore.

So, let’s look at the dynamics of the supply chain overall. In the not-too-distant future, a machine will roll up to a mountain. It will start to eat the mountain, spitting out dirt one gigantic mouthful at a time into the bed of an equally gigantic, autonomous dump truck. And that dump truck will unload its contents into the hold of an autonomous, seaworthy vessel that very well might take the contents to an automated mineral refinery for processing.

Those of us who have had to change a lightbulb recently understand the implications. They’re profound. Those of us still trying to figure out how to program our smart home to do the light bulb-changing might need a bit longer to catch on. But, the notion remains: If we can create a machine the size of a building to act on its own, how long will it be until we are creating building-size data centers that can act on their own? Automation on such a gigantic scale is not only possible, it’s inevitable. Controlling an out-of-control home vacuum is one thing. But, what happens when something on the order of a Bagger 293 goes beserk?

I don’t know. But I can guarantee you that it will not be business as usual. The good news is that the Bagger only moves at about a mile and a half a day. But still, if it learns how to speed up, then might we one day wake up to find our favorite hiking hill gone due to the whims of an independent intelligence that cares not or knows not of our love of nature? What then?

As they say in the terrain, forewarned is forearmed.

— Bob Reselman

December 18, 2017
Is DevOps Transformation a Holy Grail?
Both development and operations teams are popping up in enterprises all over the world, and even though DevOps programs are still in the early stages, our ultra digital world calls for even more software applications. This is why many analyst and consultants alike believe that, in the near future, DevOps will be even more powerful than legacy IT. DevOps will be something businesses of all sizes will not be able to live without and the sooner you initiate the DevOps transformation, the better.

Why DevOps Transformation?

One of the most exciting aspects of DevOps is enacting technology practices and architecture that help organizations create an agile and continuous product flow, from development to testing, operations and all the way to deployment while increasing stability, reliability and security at the same time.

More importantly, adopting DevOps can enact changes in the culture of your business, such as allowing teams to work more quickly with more independence. Back in the days before DevOps, deploying even a minor update, upgrade or bug fix would involve literally hundreds of developers doing weeks worth of work.

Today’s top-performing businesses and DevOps adopters are deploying much faster, and the products they release generally have greater reliability. Still not convinced that DevOps is the holy grail? Consider the following.

Focus on Speed, Not Cost

Do you want to achieve the same capabilities of Amazon and Netflix? Of course! Who doesn’t? It’s easy to make an excuse that they have a lot of money to spend, but, when we look deeper, companies that achieve DevOps success are more focused on optimizing speed, not cost. Organizing and optimizing for cost creates functional silos for networking and database people, server admins and many others.

When you optimize for speed, you are willing to cultivate talent from within, increase engineers’ salary and send them to conferences. You would not necessarily do these things if you were simply optimizing for cost. There are a whole bunch of techniques we can use to optimize for speed, such as using small, cross-functional teams that can deploy value to the customer independently without having to rely on hundreds or, in some cases, thousands of people.

Best of all, you don’t have to be a big conglomerate to make this transition. Even small and medium-sized businesses can choose to optimize for speed, break down silos and get their product to market faster. After all, this is what DevOps is all about.

Software Speed vs. Quality

Businesses that rely on developing and deploying software to support or supply their customers often are faced with choosing speed or quality. If you go to market before your competitor(s), you could be cutting corners in terms of testing and quality assurance, which can ruin your launch. However, if you wait until your code is perfect, you could end up lagging behind in the race for the next big breakthrough. Fortunately, the DevOps approach can resolve this dilemma while improving the bottom line.

Way back in 2015, the “State of DevOps Report” collected the DevOps health and habits of more than 20,000 tech professionals and found that top performers using DevOps show more agility and reliability in their releases and they were twice more likely than non-DevOps users to exceed productivity goals. In the roughly 800 companies that had a stock ticker symbol, the top performers had an almost 50 percent higher market capitalization growth over three years. This indicates that DevOps is great for IT performance as well as organizational performance.

What everything boils down to is that all modern businesses are increasingly more reliant on technology when it comes to acquiring new customers and bringing value to those customers. It is becoming increasingly apparent that agility and speed are key attributes for companies who want to win in today’s marketplace.

DevOps Transformation Guide

With all of the benefits listed above, you are probably eager to get started with DevOps. Here is a simple step-by-step guide to assist you in your DevOps transformation.
1. Conduct an Enterprise Readiness Audit – The highest probability of a successful DevOps transformation is when it is championed from the very top. The participation of business leaders is just as vital as operations, security and development teams. To get everyone on the same page and to ensure that your DevOps transformation gets off to a good start, seek assistance from an experienced, external consultant or coach. The purpose of this first step is to get executives to buy in, create mutual goals and get an understanding of what an Agile-DevOps program looks like.
2. Create a DevOps Center of Excellence (CoE) – One of the most important things you must do here is designate an Executive DevOps lead with enterprise authority. There is no use in creating this Center of Excellence at the incorrect organizational level. It must be spearheaded by someone who has the support and trust from all branches of the organization including delivery and vendors.
3. Create a Program Governance – Such a program will include creating a communications plan, an enablement program as well as establishing program KPIs. In terms of making a cultural impact on your organization, this is perhaps the most important step. The roles and responsibilities of practitioners will change with Agile and DevOps and it is vital that they understand collaborations between structures where historically animosity could have existed so they can break down these organizational silos. The KPIs will also need to be transitioned from individual metrics to holistic customer business results.
4. Create a Project Intake Process – Project intake really sets the tone for how successful each new sprint that comes into the program will be. What matters here is not the tools and process, but rather the proper sizing of the tools and process. There is a need to fit for purpose when selecting accelerators, toolchain and development methods. Once it is created, the communication of the intake process needs to be disseminated across the organization. Keep gathering feedback and and continuously evolve and mature based on this feedback.
5. Identify and Initiate Pilots – To achieve the highest probability of success, apply this step to a specific portfolio of applications or a domain solution. The purpose here is to create a value-stream map of a specific application with company wide participation on all levels. This needs to be done with the necessary amount of detail to identify manual and automated processes, end-to-end process tooling and the people and skills that are involved.
  The next aspect of the intake project is the collaboration between the subject matter experts from the Center of Excellence with the project team. The goal is to advise on “to-be” processes with proper tooling, reusable assets and automation. As the application goes through the development and test cycle, be sure to identify and capture the KPIs for the program in order to compare with existing KPIs.
6. Scale Your DevOps Program – This final step involves taking feedback from the pilot metrics and scaling them by taking different application portfolios and running multiple release trains. It is not enough to merely do automated deployments and daily builds. The last, but by no means least, piece of the DevOps puzzle is constant feedback and optimization.
Key Takeaways
- To keep up with top-performing businesses and stay alive in today’s ultra-competitive, fast-paced environment, implementing DevOps is an absolute must. DevOps is not solely reserved for giant enterprises such as Amazon and Netflix and can be implemented in any-size organizations.
- In the not too distant future, DevOps will become something businesses of all sizes will not be able to live without. To reap all the benefits DevOps has to offer, the sooner you make the transition to DevOps, the better.
- With DevOps, there is no longer a reason to choose between speed and quality when deploying your software. When implemented properly, DevOps will allow you to deploy software with both agility and reliability increasing your customer base and the bottom line.
- The step-by-step approach mentioned above can and should be implemented in any flavor and variety that best suits each individual organization’s needs and constraints. Pay particularly close attention to large gaps or excessive deviations as they are red flags which will need to be corrected.
About the Author / Stanislav Ivaschenko

Stanislav Ivaschenko is a certified AWS solutions architect at Squadex with more than 10 years of professional experience in building, delivering, supporting and optimizing a broad range of software applications. From bare metal to the cloud, from monolith architecture to microservices his expertise and experience help to meet business needs for startups, mid-sized companies and large enterprises. He has experienced a large number of DevOps and infrastructure related tools, services and management practices.
December 11, 2017
CA World 2017: DevOps As Far As the Eye Can See
The show floor for CA World 2017 had one inescapable, very obvious feature: There was DevOps as far as the eye can see. No kidding. It was a stunning reminder that this multibillion-dollar-revenue company has gone all-in on DevOps and related technologies. While we know CA has spent almost a billion and a half dollars acquiring DevOps-related companies in the last year, the effect on this year’s CA world was dramatic.

What Was All That DevOps?

CA World is organized so that most of the sessions except the keynotes are all conducted right on the expo floor. The floor is broken into distinct areas. Each area has its own theater, where sessions on the subject of that area are conducted. It is actually a great way to organize a conference like this and saves you from going to room to room, floor to floor.

CA World 2017’s main floor segments were:
- Automation – Basically CA Automic and all of its related partners, integrations and offerings. Primarily a DevOps area.
- DevOps: APIs and microservices – This has been a staple of CA World now for several years. APIs again are related to DevOps, automation and microservices.
- Mainframe -This area focused on enabling customers to extend DevOps practices to legacy applications, which most critically included a modern mainframe experience for enterprise developers
- DevSecOps – There was a whole area dedicated to DevSecOps. Primarily the Veracode team—they had their own theater and lots of space and attention in the middle of the floor. The DevSecOps section did bleed into a smaller security section with the legacy CA security tools.
- Agile Management – All things agile, based primarily on the Rally business CA bought a few years ago.
- DevOps Agile Operations – APM and things of that nature.
- DevOps CD – This area also included continuous testing and the new CD Director.
- Advanced analytics
- MSP Zone
- CA Agility Zone
- CA Acceleration Zone
As you can see, a lot of DevOps right there.

All Your DevOps Are Belong To Us

But more than the real estate of the show floor, even the messaging coming from CA was “all DevOps, all the time.” The keynotes from the executive team were heavy on DevOps and related topics. The sessions at each theater featured household-name customers mixed with partners and CA people discussing DevOps-related topics.

One presentation I found interesting was from the GM of the DevOps continuous delivery business, Jeff Schaeffer. It was a demo of a new product CA rolled called DevOps Continuous Delivery Director, a SaaS product that is a UI wrapped around a lot of the CD-related solutions from CA and leading partners. All kinds of testing, including testing from Sauce Labs and security testing from Veracode was possible from within the interface.

I moderated a good panel on Continuous Testing: Shifting Left, which was well-attended. I was joined by customer reps from Cigna, a large financial institution, Groupon and others. This was also in the DevOps CD area.

The mainframe theater stage featured a DevOps panel session to showcase a cross-BU perspective. Panelists included Aline Gerew from Automation, Steve Feloney from Continuous Delivery, Scott Morrison from APIs and Microservices, and Jean Louis Vignaud from Mainframe. The video recording is not yet available, but I was able to catch one snippet in a promotional tweet – APIs enabling IAM on Mainframe

What’s Still Needed?

As impressive as the DevOps show by CA was, what is missing is a tighter integration between the DevOps solutions. While some rudimentary integration is in the works, really making a 1 + 1 = 3 equation will provide the enhanced value to customers that such a broad set of DevOps solutions promises. I spoke to executives from just about every DevOps-related business unit and to a person they mentioned that this was in the works. But what was pleasantly surprising was the mandate not to force tighter integration that would stunt the innovation that each of the DevOps companies brought with them when they were acquired by CA.

Of course, we will see if this remains the case. However, I did speak with employees at each of the units and they were all happy with how CA has handled the acquisitions to this point.

Conclusion

It is hard to argue when a software company the size of CA Technologies makes such a bold statement regarding DevOps. The company is clearly betting its future on DevOps being the backbone of the modern software factory.

— Alan Shimel
November 20, 2017
IPsoft’s 1Desk Advances AI Labor Force Reality

IPsoft wants to put a digital face on the future of IT operations that will be infused by artificial intelligence (AI). Its recently launched 1Desk service desk combines the Amelia digital assistant with IPsoft’s Apollo virtual engineer technology to automate multiple backend processes.

Allan Anderson, director of enterprise solutions for IPsoft, says 1Desk effectively combines two longstanding areas of IPsoft research to provide IT organization with an ability to manage IT at scale using digital personal assistants and virtual engineers.

IPsoft has been providing managed services based on its Apollo AI framework for several years. The launch of 1Desk creates an opportunity for IT organizations to not only employ that AI framework on their own, but also combine it with a digital assistant technology christened Amelia that IPsoft has trained to handle a variety of technology and business process requests.

Anderson says the virtual engineer AI technology developed by IPsoft make use of machine learning algorithms to continually monitor how humans complete a task. Armed with that data, those virtual engineers can then incorporate that information into their knowledge base to learn a process or fine-tune an existing one.

A continuing shortage of IT skills in key areas is significantly reducing resistance to AI technologies being applied inside IT organizations—AI technologies may enable organizations of almost any size to operate IT environments at unprecedented levels of scale while minimizing human errors. IPsoft, says Anderson, expects virtual engineers to be employed as a digital labor force capable of performing tasks such as identifying and killing a long-running query to configuring and executing runbooks across the most complicated data center environments. To facilitate the automation of the those processes, IPsoft has developed a Digital Labor Studio tool that organizations can use to create and train a virtual agent.

There’s no doubt that many of the tasks associated with managing an IT environment are repetitive. In fact, any repetitive IT task is arguably a ripe candidate for automation. How those advances will impact DevOps processes remains to be seen. It’s not difficult to imagine that each development team will be assigned its own digital assistant. The role of the IT operations staff then would evolve into one where a smaller group of people train and manage a pool of digital labor.

IT professionals are generally used to the idea that IT will be used to automate a task that used to be part of someone else’s job. But now IT is being used to automate many of the tasks performed by IT professionals. It may take some time for them to first accept that idea and then evolve their skills accordingly. There will also be a need for some level of human involvement when it comes to IT. But instead of thinking in terms of managers and administrators, perhaps the brave new world will be one where everyone in IT is now the manager of their own digital force of labor.

— Mike Vizard

November 17, 2017
Top 5 Best Practices for DevOps Monitoring

Today organizations are required to deliver on higher levels of customer satisfaction for their online services. Many, however, are forced to support these initiatives with an interrupt-driven approach as they react to fix things after they break. However, for a more proactive approach and to manage expected high levels of SLAs, organizations can opt to reduce their amount of unscheduled downtime by implementing a continuous delivery (CD) model to their development efforts.

What is critical in the CD model is the ability to monitor and manage systems in a structured way, getting early detection of problems so organizations can make changes before the system is impacted. While unexpected failures make interrupt-driven work unavoidable at times, organizations can become more proactive by examining their current approach and toolset against business needs to help create a path to continuous service delivery optimization.

Effective CD Models Need Effective Monitoring

As a starting point, organizations need control and visibility into their DevOps environment by collecting and instrumenting everything. Considering the amounts of data, this can be an insurmountable challenge for most organizations. To get started, follow these five best practices to perform DevOps monitoring efficiently and quickly in a measurable and scalable way.

Step 1: Collect the Data. You can’t manage what you don’t measure! It is important to take inventory of what is being collected today and align with business and executive teams to collaborate on the aim of the services delivered. Analyze the inventory collection of metrics with questions such as: “Why aren’t we collecting this? How does this fit into our goals? Have you ever seen a failure in X? How often should we be measuring this? How long should we keep it for? Is that important?” Teams also should evaluate how they are collecting the information and consider the best architectural approach for collecting their metrics, including whether pull, push or pull collection methods are better. Once goals are understood and the inventory of data is collected, look at what else your organization should consider collecting.

Step 2: Correlate and Triage. Correlation of data is needed to understand it, but data comes in at different frequencies, in different time frames and from different sources. Work to normalize the data to understand it, perform comparisons against the different incoming metrics and establish baselines for basic service availability. Since organizations aim to go beyond basic service level agreements (SLAs) and offer a high-performing solution, constantly ask what the organization is missing from a data perspective and how it relates to business initiatives. Asking that question from the collection, correlation and triage perspective is critical.

Step 3: Identify Trends. Organizations need to examine historical data to identify trends and take action before issues arise and customers complain. Establish alerting thresholds by outlining what a normal day looks like from a monitoring performance and customer perspective, and then identifying examples of what makes an abnormal day. This ties in with managing infrastructure inventory and understanding safety thresholds for each of the components that potentially could impact the service offering. It’s critical to communicate these findings with teams and business-line managers to prevent service delivery problems from happening and optimize offerings based on identified trends.

Step 4: Notify and Act (Automation). In manual mode, a notification is delivered and then the team reacts. But teams are continually pushed to do things faster, and automation can help. To get to that point, organizations must understand where best to add more automation. How do you gather the right telemetry that delivers consistent answers that the machine can operate rules against, and do you leverage an automated process or notify a person? The desire to improve to a faster process requires a shift from manual to automated practices.

Step 5: Predict (What-if Analysis). If you don’t take the time to go through the first four steps in a methodical way, it’s very difficult to reach this final step without always remaining in a manual mode. To balance costs and availability, it’s critical to discuss with the executive team how to predict customer service consumption (revenue) against the amount your services are going to cost the business.

For example, developing a service that can alert customers of a potential service disruption due to disk space can be accomplished only by clearly setting business goals, then using the right metrics and events platform leveraging a time series database. In this case, the business goal is to ensure no service disruption due to failures such as low disk space; the metric monitoring is for disk space for each customer instance with the appropriate threshold alert set off by an automated trigger; and the action is an email to the customer letting them know of the situation and what action they can take (reduce load or upgrade for more disk space). In this example, it is important to marry business logic with the monitoring practice to make service a successful experience for customers. Furthermore, this approach will help teams to not only predict user experience but also help to predict OpEx and CapEx better into the future.

That’s the benefit of the ability to predict with the right monitoring approach—to make informed business decisions based on results. Customers appreciate that, and proactive support is much better than being reactive.

Emerging trends such as microservices, containerization, elastic storage, software defined networking and hybrid clouds are pushing the boundaries of DevOps monitoring. The right monitoring plan can identify and resolve problems before they affect critical business processes and enables customers to plan for upgrades before outdated systems begin to cause failures and outages for users.

About the Author / Mark Herring

Mark Herring is CMO at InfluxData. He is a well-rounded silicon-valley executive with proven experience in taking complex technology and making it understandable to the broader audience. His deep developer roots are never far from his mind as he looks at trends and asks the tough questions on whether the technology is here to stay or just another fad. Currently Mark is Chief Marketing Officer at InfluxData. Follow him on Twitter and connect with him on LinkedIn.

November 15, 2017
Automation and Its Impact on Cultures

Let me tell you about my granddaughter. She’s 14 years old and wicked smart. For a while we did pair programming together online in Python. In the old days you took your grandkids fishing. Today we code. Go figure.

These days, we don’t program together as much. She’s too busy preparing for math competitions. As I said, she’s wicked smart. Needless to say, my granddaughter has quite the future in front of her.

A few weeks ago, I had the opportunity to visit her in real time and space. We sat around drinking hot chocolate and cappuccino, talking about Big Picture stuff. That’s what grandparents are supposed to do with their grandkids, talk about matters beyond the day to day. At one point during the conversation I asked her the following question, “Are you friends with anybody who has trouble reading?”

Her reply was, no. All her friends read quite well and were interested in things that interested her. She wasn’t bragging or anything like that. It was just a matter of fact. All her friends are smart, engaged, have cell phones and know how to work a computer.

My question came from a book I had just finished reading, “Coming Apart,” by Charles Murray. One of the key points that Murray makes is that we’re dividing into two cultures and the distance between the two is growing wider. A key way to figure out how far from that “other” culture you are is to tally up the number of friends you have who have trouble reading. One camp will have a lot. The other will have practically none.

Back when I was my granddaughter’s age I had friends who had trouble reading. We met playing pickup basketball. The ability to read wasn’t important in that context. We based friendship on other factors. We played ball and rubbed off on each other. Some of the things that the poor readers were good at rubbed off on me. I like to think some of my better qualities rubbed off on them.

But that was then and this is now.

In the book, “Brave New World,” written in 1932, people are born in test tubes according to genetic recipes. Some grow to be highly intelligent, creative, analytic Alphas. Others grow up to be be Betas, people that can perform highly complex tasks such as mixing volatile chemicals, but without having an inkling of understanding as to why they are doing the task. Gammas were good at adding up numbers and thus, become bookkeepers. Deltas are the truck drivers and elevator operators. Epsilons are grown to be three feet tall and abhor the outdoors. They do the dregs of factory work, climbing in and under machines.

Now, hold that thought while I tell you about my daughter.

My daughter is a high school teacher. Most of her students are from what is termed, the wrong side of the tracks. They don’t read that well and are not terribly engaged in school. But, they do have cell phones and they know how to use them—mostly to text back and forth to home and with friends in classes in other parts of the building. To use the transportation analogy, for all intents and purposes they use of technology not so much to explore but to get from one place to another, something a driverless vehicle will do better than the human counterpart in the not-too-distant future.

So, what does this have to do with DevOps?

We in DevOps are the people who are going to drive most, if not all, the technology that will control the world. We are the Alphas. To get into our club you need to be very, very smart. Our work is complex and requires both commitment and ability to engage in continuous learning. When my granddaughter comes of age, she will mostly likely get into the club, maybe with honors. She lives in a world of increasing returns. Because she is smart, she hangs out with smart people. Because she hangs out with smart people, she will get smarter. To use an analogy: Those who know how to work GitHub have a universe of code at their disposal. Thus, they will make a lot more code for the world to use. Those who can’t do a pull request are stuck dead in their tracks. Access to opportunity provides more access to opportunity.

Using the model in “Brave New World,” those using GitHub creatively as part of the software development life cycle can be considered Alphas. Those who can’t are Betas, Gammas, Deltas and Epsilons. The really bad news is that the work of Betas, Gammas, Delta and Epsilons is low-hanging fruit for replacement by automation and robotics. The work of Alphas is a lot more difficult to emulate with machine intelligence. Corporate employers will still need them. The Alphas will do just fine.

Hopefully, those who are displaced by automation will be assuaged by low-cost television and cell phone plans that offer unlimited data and texting. Otherwise, they’re going to be looking around for something else to do instead of working. Hopefully there will be “something else to do” while the other culture is prospering.

My granddaughter’s world reflects the trend that Murray and others describe. We’re dividing into two cultures and the distance between these cultures is growing. The sad part is that the experiences that used to provide a bridge between cultures are becoming less common. There are fewer common spaces in which we can rub off on each other benignly, with no other agenda than to get the ball through the hoop. If the trend continues, the cultures will grow so far apart that they will become unrecognizable to each other. Once recognition leaves the landscape, understanding and empathy are not far behind. You cannot have affinity for that which you cannot see.

Is it possible to reverse the trend? Is it possible to have an outcome other than the very probable one on the horizon? I don’t know. But, if we don’t try, the ramifications will be dire.

— Bob Reselman

November 14, 2017
Survey Finds DevOps Teams Under Increasing Pressure

The fact that a much larger percentage of companies now recognize they are software companies that happen to specialize in a specific vertical industry is now putting a lot of pressure on developers, according to a new survey of 750 development team leaders in the United States and the UK conducted by Testplant, a provider of tools that automate application testing.

The survey finds that 42 percent of the respondents claim they are expected to design, develop and test apps in an unrealistic amount of time. More than one-third (36 percent) also report they are not given enough time to ensure apps are properly tested before deployment, while 56 percent note that use of outdated techniques and tools is holding them back from meeting the demands of the digital world.

Because of this combination of factors, nearly half the respondents (49 percent) say they are deploying applications before they go through an ideal testing amount of testing, and 45 percent admit to putting applications into production that will perform below initial requirements.

Dr. John Bates, CEO of Testplant, says it’s a matter of time before IT organizations will have to rely more on tools that automate much of the testing process, as pressure on DevOps teams to roll out new applications and update existing ones is only going to increase. The survey respondents (60 percent) say most of the pressure to accelerate software development is being driven from within the organization itself. But nearly half (49 percent) say pressure comes from competitors. As more organizations realize their competitive differentiation comes from the software they develop, there’s now a lot more focus on accelerating application development, says Bates.

The survey also notes that the applications being built are more complex than ever. Over the next 12 to 18 months, half the respondents (50 percent) say their company will develop more apps using components for an internet of things (IoT) environment. Well over half (58 percent) also say their company will develop more apps with artificial intelligence (AI) components, and 62 percent say their company will increase the amount of applications that incorporate machine and deep learning algorithms.

When it comes to deploying applications, automation is ranked as the most critical challenge by 67 percent of the respondents, and 70 percent claim they are now more focused on automated testing today than they were in the past.

In total, 75 percent of the respondents say that with access to better tools, the applications deployed by their team would have a greater impact on the business. A full 81 percent also say that with more time, applications deployed by their team would have a greater impact on the business.

Unfortunately, more time for application development is luxury most business don’t have today. Not only is a wave of digital business innovation is making it easier for startups to not only enter markets, but companies that traditionally have focused in one market segment are now extending their reach to adjacent areas. The difference between surviving and being made obsolete now comes down the level of customer experience an organization can deliver by the applications it makes available to customers.

Unfortunately, when it comes to those applications, customers tend to be unforgiving. Once bad application experience and many of them will move on to try an application developed by a rival. Bates notes that too many organizations still seem to think they can test their applications on end users on the assumption they can fix any issue that comes up rapidly. In reality, rather than participate in what amounts to an extended beta testing process, most end users will simply look for a better alternative application experience.

In short, the difference between success and complete failure now hinges on the level of quality application testing that can be implemented in the shortest amount of time possible.

— Mike Vizard

November 2, 2017
The Future of Continuous Testing: Fail Faster

In the early days of Agile and Lean development practices, a very noticeable gap appeared between dev teams and ops and QA. Development was accelerating at a lightning-fast pace, but testing processes could not keep up, slowing down the entire life cycle and preventing organizations from achieving continuous delivery. Traditional testing methods, which rely heavily on manual processes and frequently updated GUI tests, proved too cumbersome and inadequate for our new lifecycle timelines.

The era of testing as an afterthought is dying out, and continuous testing has emerged as the new standard. The idea behind continuous testing is pretty straightforward: Implement a systematic approach towards improvement that incorporates not only automation, but also all of the other tools and processes that will help reduce risk before software progresses to release and production. Organizations embracing this are rethinking testing as a continuous process that starts from the sandbox, rather than a precursor to release.

Continuous Testing: Achieving ‘Fail Faster’

At xMatters, we strive to a testing strategy that supports our goal of committing to production multiple times per day—one that we’ve been diligently moving closer to, having progressed from releases every quarter to every month to every week to now, twice per week. Our approach is to conduct unit tests at the team level, code quality, sonar and static analysis, and then deploy to production at limited percentages that we bump up in stages. Our overarching mantra is “fail fast,” because the faster and sooner you fail, the faster and sooner you can fix what’s broken, and the less resources and time you will spend troubleshooting and resolving issues. This, of course, translates to real cost savings.

Correlating failure to actual events is the biggest hindrance to the continuous testing dream, as pinpointing root causes is an incredibly time-consuming process. This is where automation will take us to the next level, if implemented in a smart way. The future of continuous testing will be adaptive so as to determine the causes of failures faster. We need to automate as much as possible the process of predicting where code quality issues are and identifying the precise data correlations that will enable us to fix issues rapidly.

Part of this adaptive testing approach will include automating intelligent regression testing. We don’t need to run a black box test in every instance, for example, which can take hours every time. We’ll save incredible amounts of time, money and developer resources if we design tests that can detect new changes and only test those. Smarter regression testing will result in precious time saved that ultimately impacts—and can even make or break—the speed of innovation.

Another critical consideration to smarter testing is making sure to account for a comprehensive understanding of risk. We need to incorporate real business considerations into our understanding of risk and bake those into our tests as well. We should hold our software to not only standards of code quality and integration success, but also to meet real business expectations for areas such as security, compliance and reliability. Testing from the bottom up must be done in tandem with top-down analysis to account for the full spectrum of potential risks that have the potential to impact the bottom line. Machine learning systems can also be extended to learn new business applications as new functionalities are added and assist in creating test cases that serve to enhance the quality of the application.

Developing these intelligent automation tools remains our biggest barrier to achieving the continuous testing dream, and while many organizations are gunning for it, most are still mired by manual processes. A global study conducted earlier this year by CA Technologies, for instance, demonstrated that while 75 percent of organizations identify continuous testing as important, just 20 percent report a high quality of test automation coverage.

It’s worth noting that not every aspect of testing can be successfully automated. The vast majority of functional tests will almost certainly not be manual processes—but what about user experience testing? We have yet to conceive of an approach to this that can truly be automated. Because nothing beats a real human’s assessment of experiential quality, this will be one of the areas of testing that will likely remain manual. Thus, while our goal is to automate 100 percent, realistically this won’t be the case.

The end goal here is not only to achieve the dream of continuous delivery, but also to eliminate distractions as much as possible. The more repeatable processes you can automate, the fewer distractions your team will encounter, and the more they can focus on creativity, strategy and building new features and capabilities instead of fixing old ones. This is what will ultimately determine your ability to innovate, as you’ll no longer have to sacrifice either time, scope or quality.

About the Author / Deepa Guna

Deepa Guna is Senior QA Architect for xMatters. She has been working in technology since 2003. Connect with her on LinkedIn.

October 24, 2017
Oracle on the March Toward Automated CI/CD Services

At its OpenWorld 2017 conference today, Oracle charted a course forward that essentially creates a managed DevOps platform spanning a broad number of application types and styles.

The Oracle Application Development Platform is a subset of the Oracle Cloud Platform optimized for application development. Siddhartha Agarwal, vice president of product management and strategy, says the company is working to automate every aspect of DevOps with an eye toward making it possible for organizations to concentrate all their efforts on developing applications versus managing them.

The latest step in achieving that goal are updates to the Oracle Application Development Platform, which now incorporates the Oracle Container Native Application Development Platform based on the open-source Kubernetes container orchestration engine, and Oracle Container Engines, a container pipeline management service based on the Wercker software the company acquired earlier this year. The company also announced it is adding support for the Go and Java Enterprise Edition programming languages to existing support for Java Standard Edition, Node.js, PHP Ruby and Python runtimes.

Other options being added to the Oracle Application Development Platform include Fn.a, an open source serverless computing project based on software the company gained when it acquired Iron.io, as well as an implementation of the Open Service Broker that enables applications developed for the Cloud Foundry platform-as-a-service (PaaS) environment to be deployed on Oracle Cloud. The company is also making available an instance of a BOSH Cloud Provider Interface to automate the deployment of those Cloud Foundry workloads.

Meanwhile, the Oracle Data Hub Cloud Service now also supports the open-source Cassandra database alongside existing support for MongoDB, Postgres and other databases to be added in the future. However, the core Oracle database remains the primary data engine service. In fact, Oracle Database 18c, announced earlier in the week, employs machine learning algorithms to manage instances of the company’s relational database autonomously with no human intervention required.

Finally, the company is adding a Blockchain Cloud Service that allows developers to access an implementation of open-source Hyperledger blockchain software, as well as an intelligent chatbot platform that can incorporate in mobile computing applications.

Agarwal says his company is committed to making available both commercial and open-source platforms available via an Oracle Cloud Platform that provides access to x86, Sparc and graphical processing units (GPUs). Agarwal says the company expects IT organizations to mix and match back-end services as they see fit. But increasingly all those back-end services will be managed under the auspices of a fully managed continuous integration/continuous development (CI/CD) platform.
Long-term, Agarwal says there may even come a day when application workloads are able to self-describe themselves, which would then enable the company to deploy them automatically on the best-suited back end.

Naturally, it may be a while before the level of DevOps nirvana gets achieved. But as machine learning algorithms and other forms of artificial intelligence (AI) advance, it’s clear that the weight IT organizations are expected to carry is increasingly shifting more toward building applications than managing the underlying infrastructure on which they run.

— Mike Vizard

October 2, 2017