devopsy.kinsta.cloud

Category: Features

Sofico Adds DevOps Approaches, Updates CD, Part Two

We recently geared up with Sofico’s existing CD process and the challenges it presented together with its search for a devops solution. Now read the conclusion as Sofico earns satisfaction through its own devops approaches.

Sofico Adopts DevOps Methods / Tools

Today, Sofico uses the Bamboo CI server to perform builds, deployments, and tests. Bamboo automates builds of the backend server, web interfaces, command line interfaces, and a Windows line. Sofico uses Bamboo to deploy its application builds. Sofico runs tests on software using tools such as Selenium for browser testing and functional tests.

Sofico runs Dynatrace monitoring together with the functional tests to gather performance data. “This gives us correlations between the functional outcomes, the performance outcomes, and the expected outcomes of the testing,” says Jan Swaelens, Software Architect and Member of Solution Management, Sofico.

“We actually created a custom Bamboo plug-in for that together with the people at Dynatrace in order to visualize the outputs of that correlation inside Bamboo,” says Swaelens. Sofico also uses JMeter to launch tests on servers and Dynatrace monitors those as well for performance metrics.

Sofico uses application tools to manage its Agile inner workings and to run reports on that. Sofico uses Stash to manage its code repositories where its developers do feature branching for new features. “Developers use the Dynatrace server to monitor their performance metrics for the features they’re building. It enables proactive monitoring of what they create, even before they commit the change to central testing,” says Swaelens.

So the developer does his regular testing and he also enables the agents that do the monitoring to identify errors. Sofico also does load testing using increasing loads for performance metrics, and performs tests for various use cases.

Results

Using DevOps to expedite builds, tests, and deployments significantly decreases the number of times that something will go into production and need to come back for a fix. “We actually manage to stop software with errors before it gets to our customer. This is very critical because in our sector when, for example, we make a major mistake in billing software and the customer actually has that in their production environment, their customers are impacted,” says Swaelens. This is very expensive for the customer and for Sofico because then they have to fix the issue in as little as a day, which is not always possible.

The bumps in the road that Sofico experienced were in rolling out devops and adjusting to the new way of working. “We found that the impact on systems and processes was even larger than we were expecting. We restructured our old company at every layer from management down to the developers and testers,” says Swaelens.

Sofico redesigned every process so everyone could work in devops teams. All of Sofico’s existing methods and processes were analyzed to determine how the company functioned, why it worked that way, and to make sure the end result was still providing the same or better results.

The changes and improvements that devops made were far-reaching. Rather than making devops come to Sofico, Sofico had to gravitate towards devops, change its business culture, and break down all its silos. “We went all in to get to the next level and do more business globally. Because we’re not just one company, we had to spread devops across all our sub-companies around the world,” says Swaelens.

Sofico can draw a straight line between adopting devops and the speed with which it can now grow globally to increase customer uptake of its offerings. The software quality and timeliness of delivery are where they should be. Sofico’s people are happier, too.

Sofico has relocated its teams together into the same space. Consultants, developers, testers, and operations staff are sitting physically together in the same areas. People are working together and talking with each other to derive the best solutions and to improve efficiency. “We actually have cases of people blossoming because they are seeing whole new parts of the business and working in new ways that we could never have imagined,” says Swaelens. Where Sofico’s people once saw only their piece of the development puzzle, they now see the entire process. They know the full value of their work.

March 17, 2015
Linux Containers & Security Implications

Does the word “container” intimate containment, suggesting that containers are inherently secure? If it does, any such assumption of security may be the broadest Docker vulnerability to date.

“One of the biggest threats I see with Docker is its positioning and the implied security in the language. The reality is that these containers don’t contain anything,” says Aaron Cois, Researcher, CERT Division, Software Engineering Institute, Carnegie Mellon University. Yet, that is the implication.

Just as those who thought Linux or VMs were secure enough on their own were mistaken, so those who believe that containers put a lid on security will be sorely disappointed. Today, Linux environments require network, OS / host security, Internet security, and web application security measures similar to those used with other platforms. Tools like security auditing / PEN testing, firewalls / WAFs, anvi-virus and anti-malware tools, DLP, IDS/IPS, remediation tools, and really the gamut of security measures similar to what Microsoft environments require are increasingly needed to defend data in Linux environments. “Likewise, operations can give developers tools to log into the VSphere console to create and change VMs while limiting their privileges,” says Cois.

And so, containers also require appropriate security measures. “Developers and non-admin operations staff don’t need to log into the host command line to work, and no one in security wants them to,” says Cois. But today’s Docker workflow not only permits but requires it.

The Root of the Problem…

–THIS STORY HAS BEEN MOVED TO CONTAINER JOURNAL–

March 16, 2015
Sofico Adds DevOps Approaches, Updates CD

Yes, you can have continuous delivery without DevOps. To understand why you wouldn’t want to, follow the journey of Sofico from its CD of olden days to its CD rooted in DevOps today.

A Very Brief History of Sofico

Its founders started Sofico 27-years ago as a small family company. Today Sofico is a multinational organization with approximately 170 employees. Sofico creates software for automotive financing, leasing, and fleet and mobility management. “We create software for companies that manage all aspects of leasing a car to, say, an employee for example, in case you get a car from your employer,” says Jan Swaelens, Software Architect & Member of Solution Management, Sofico.

Sofico offers its customers websites where they can configure and quote the car they want from their boss. Sofico also creates back office software that processes orders and contracts and supports billing and accounting functions. “We offer a complete solution, which you could compare to a large ERP system like SAP, but specifically for the leasing sector,” says Swaelens.

Sofico CD, Sans DevOps

Sofico started out using CD / CI approaches about seven years ago, first migrating its old systems to do CI for software delivery. “We were using really ancient things like batch scripts to do our software builds and Windows timers to schedule them,” says Swaelens. Sofico took a lot of manual steps in deploying software.

Sofico used Waterfall development methodologies, moving step-by-step through the development process in small chunks. The process was not interactive. “We had separate departments for the analysts, for the designers, with developers working solely with other developers, testers working only with testers, and business consultants working only with other business consultants,” says Swaelens. Sofico moved the work from one silo to the next, with several silos literally isolated by differing geographical locations.

The Challenges of Manual CD

This configuration of continuous development presented several challenges. “We had problems getting software done on a reasonable time scale. It took quite a while to get software built in the first place. Deployment was another matter,” says Swaelens. The quality assurance processes were manual, which created issues for large projects that required a lot of changes in short time periods. The CD process led to many functional and performance errors. “We had some serious technical regressions that caused malfunctions in the products,” says Swaelens.

Staff in one group would return work to its originating group. Development would return work to the consultants who had created bad specifications. Operations would return work to development because it didn’t work right. “The customer bore the cost of these mistakes because his issue or feature either wasn’t done correctly or wasn’t done on time,” says Swaelens.

These homely CD processes also affected Sofico in its growth into other countries. Sofico had to grow fast to gain momentum and succeed at its new locations. It was difficult for the company to pull all the knowledge necessary to work with those processes together fast enough to keep pace with its large projects to make sure those took off so that its new locations could thrive.

Sofico In Search Of Something Better

Automating stuff is one thing; getting people to work together efficiently is something else, according to Swaelens. Sofico was looking for a way to improve how its people worked as an organization. It was important to make customers happy, and to make employees happy. Working in those silos reduced people’s enjoyment of their work because they were stuck in one kind of job doing only one thing. “We hoped that switching to DevOps would also give them more rewarding jobs,” says Swaelens.

Sofico also wanted to improve software quality by automating an increasing number and types of relevant tests such as testing for software performance. Delivering software functionality was not enough for Sofico. They wanted to guarantee internal and external software product performance to their customer. “In our market, that is very important. It directly impacts their end customer in a very visible way,” says Swaelens.

Sofico sought faster go-to-market time, too. When under the old processes the customer called Sofico asking for a new feature set, Sofico had to reach out to each and every silo more than once with communications back and forth whenever there was not sufficient clarity about project requirements. “The advantage of what we were trying to achieve was that it would reduce the time required when crossing those silos just to get the right answer and to secure the prospect of project delivery,” says Swaelens.

To learn what Sofico did to achieve DevOps nirvana, come back and read part two of “Sofico Adds DevOps Approaches, Updates CD”.

March 12, 2015
Happy Birthday devopsy.kinsta.cloud and what a year it’s been

They say it’s your birthday
We’re gonna have a good time,
I’m glad it’s your birthday
Happy birthday to you

Wow, a whole year. To paraphrase the Grateful Dead, what a strange trip it’s been. We launched devopsy.kinsta.cloud a year ago today with perhaps the naive notion that DevOps was going to really change the way organizations do IT and business. That it was almost inevitable that DevOps was going to move beyond the realm of early adopters and visionaries, beyond startups and Unicorns and see adoption by enterprises and businesses of all sizes and shapes. You know what? We were right!

DevOps has skyrocketed onto the mainstream IT consciousness and we were lucky enough to have a first row seat here at devopsy.kinsta.cloud. We are even proud of the fact that maybe we even played a small part in this adventure.

But let’s be real. It hasn’t been all sprinkles and smiles. We have learned a lot of lessons along the year and usually the hard way. But sometimes the hard way is the best way to learn them. We knew that our audience was going to be a technical group. They would want to hear about and from people who had problems and issues similar to their own and how others solved or addressed the issues. We knew that they liked to find out about different tools that can help you keep the IT machine humming along.

We defined our mission as pitching a big tent for DevOps. I said it in the very first post one year ago today. To me personally pitching that big tent meant that we weren’t just about the existing DevOps community, many of which were early adopters, thought leaders and visionaries in the space. In my mind this group represented only about 15% at most of the eventual DevOps community. I realized that though some people were talking about and instituting DevOps principles for some years already, the majority of our audience was going to be new to DevOps. I wanted devopsy.kinsta.cloud to have a place for everyone interested in DevOps.

That right there has also been the source of some of our biggest mistakes and angst over the year. We definitely took some time finding our legs under us. Finding people with the credibility that others would respect, striking the right tone, fine tuning the mix of articles and topics. At each step along the way we made some missteps. Many in the community didn’t mince words in pointing out what they didn’t like. Others remained silent and just ignored us. We took our lumps for sure, but we just kept putting one foot in front of the other, putting out new fresh content every day.

Many times readers or just people piling on via Twitter took us to task about what people wrote in their articles. We had and still have no problem with readers disagreeing with what an author writes here on devopsy.kinsta.cloud. It is healthy to have debate. But we were surprised that people would think that we wrote the particular post or article. Or that is what we espoused. Just not the case. We don’t agree with what everyone writes here. However we think people have the right to write what they believe in. We think giving people a place to write means not censoring or telling them what to write.

Finally we started catching stride. Since the summer months our entire business has been on a rocket like trajectory. Here are some salient metrics:

1. We are on track for a 100k view month for March. This represents a 100% increase over just November 2014.
2. Our twitter, facebook, Google +, Linkedin following has grown exponentially
3. We have published over 630 articles in the past year, by well over 125 different authors!
4. We have produced or been media sponsor for 10+ events (with many more coming this year)
5. We have produced 8 webinars
6. A DevOps salary survey
7. Fulfilled our goal of becoming the single largest collection of original DevOps related content on the Internet.
8. We published over 50 newsletters and now offer several options on how you want to receive your devopsy.kinsta.cloud news.

At a time when media companies like GigaOm are going under. When other media outlets are laying off paid writers, relying on unpaid bloggers, devopsy.kinsta.cloud is adding to our featured writers. At the same time continuing to publish more fresh, original DevOps content than anyone. Other sites have taken to syndicating blog posts from others, republishing stories from elsewhere. At devopsy.kinsta.cloud you can be sure that when we publish a story it is the first time that content has appeared on the Internet.

We have spent a lot of time thinking and planning for the coming year. We think that 2015 and 2016 will be big years for DevOps. We want to make sure we do our part and continue our growth. Look for many new features and new takes on distributing DevOps content and engaging the community this year.

If you look in our about section today you will see a near final draft of our editorial calendar. Our new media kits is also almost ready. You can see a near final draft of that in our sponsor page. Yes, 2015 promises to be even better than 2014. I am sure there will be some missteps this year as well. Hopefully we have learned from our past and there will be less.

Most of all on behalf of everyone associated with devopsy.kinsta.cloud, I want to thank each and every one of you reading this article for your support this year. Without your support, none of the amazing things that has happened so far would be possible.

So happy birthday devopsy.kinsta.cloud. Here is to many more. Also did I mention the first 50 people who tweet with #happybirthdaydevopsdotcom will receive a special edition t-shirt. Thanks again!

March 11, 2015
Red Hat launches platform to support Docker, Kubernetes and enterprise-grade containerization

Red Hat continued its momentum to capture a piece of the containerization market last week with the official launch of its new Red Hat Enterprise Linux Atomic Host operating system. According to Red Hat executives, this is a natural progression in the Red Hat ecosystem as more of the company’s customers embrace Docker and other containerization technologies.

“Today, with the launch of Red Hat Enterprise Linux 7 Atomic Host, we are doing the same for Linux containers, bridging innovative open technology with the stability and security required by the enterprise,” said Jim Totton, vice president and general manager of the platforms business unit at Red Hat. “More than just an addition to the Red Hat Enterprise Linux portfolio, Red Hat Enterprise Linux 7 Atomic Host showcases the future of the enterprise application, a powerful, flexible application greater than the sum of its parts, and entirely fueled by the power of open innovation.”

–THIS STORY HAS BEEN MOVED TO CONTAINER JOURNAL–

March 10, 2015
DevOps Dogfood: IBM’s internal DevOps transformation

When detractors argue the impossibility of doing DevOps at enterprise scale, IBM Matt Ellis has got the perfect case study to disprove that argument. It comes from looking in the mirror. At the recent IBM InterConnect conference, Ellis explained how Big Blue itself is showing how DevOps can rejuvenate even the largest software development operations by undertaking its own DevOps transformation.

If ever there was a stress-test of scale, it’d be at IBM, which has some 36,000 software engineers developing in 110 locations, Ellis said. They’re running code on top of 44,000 physical servers and 116,000 virtual machines. All told, they’re working on 519 major product lines, 150 of which are SaaS offerings, many of them also supported by mobile apps.

“That is enterprise scale,” Ellis said, explaining that IBM has taken this massive workforce from classic development patterns, through an adoption of Agile and on to the delivery of “a true DevOps experience.”

It was all driven by the need to improve business outcomes. As Ellis explained, enterprises of all types are increasingly required to lock horns with small, disruptive competitors that are better able to react quickly to market demands.

“People are getting interested in DevOps is because of this rate and pace of change that’s being forced on organizations by smaller startups that are potentially causing disintermediation of classic enterprise businesses,” he said.

Ellis detailed some of the example drivers and outcomes that played a part in IBM’s own specific DevOps journey:

Backlog Clearing

Back in 2008, IBM realized it’s method of delivering software was hurting its abililty to deliver features in tune with timely market needs.

“We’d build a project road map that runs maybe 12 to 18 months and we were getting a backlog,” he said, explaining that clients and partners in the audience probably experienced that frustration first-hand.

Project Initiation Times

Something as simple as project initiation to decide on new features and functions could involve up to 30-dayy discussions. Now IBM is able to run two-day iteration of deciding what goes into the next sprint for projects, Ellis explained.

Build Time

Build time was dragging down the process. Now IBM has reduced the process of making sure what’s built stands up and is integrated properly down to under an hour.

“So on the same morning that I run the compilation on build, I can confirm that it is hung together and works as a package,” he said.

Ultimately, the DevOps and continuous integration journey has enabled IBM to shorten release cycles for its on-premises products from a 12-month cycle down to quarterly releases.

“We’ve shortened it to support a release cycle—if we needed it—where the train leaves every three months,” Ellis said.

March 9, 2015
Five Areas Every e-Commerce Business Should Monitor Using Log Data
If you’re responsible for an e-commerce website, I guarantee you’ve been asked about, told about, or wondered about what kind of information and events you should be logging. Important things are happening all the time, and you need to know what’s going on! Right? Are they bad things, good things, or “nobody cares” things?

So what do you do? Log all the things! Right?! If you’ve ever found yourself awake at 3:00am wondering why you decided to become a software engineer instead of [insert any job that lets you sleep during the night here], chances are it’s because you decided to log everything. Having said that, guess who has two thumbs and logs all the things? This guy.

You need to focus. It’s okay to log all the things, but you need to focus your attention on the stuff that matters the most, and a simple tool for aggregating IIS logs isn’t enough. A scalable and easy-to-use service that will alert you and provide an interface for surfacing the events that matter the most is critical to ensuring a constant revenue stream and the best possible experience for your customers. Yes, there are lots of components and potential events that really matter when it comes to developing or hosting an e-commerce website. So where should you start? I’m going to break this down into the top five areas that should be the initial focus of your log data efforts. You should not only log these types of events but also put an alerting system in place so you’re aware about things that require your immediate attention. (We use Loggly for this.)
1. Checkout. We’ll go with the lowest hanging fruit first. If you’re an e-commerce website, your job is to convert customers. Conversion means pushing your customers through the checkout funnel and completing an order. All checkout errors should be logged, and you should instantly be alerted if any part of the checkout process fails.
This seems obvious, but it’s important you focus on each minute piece of the process including:
- Address data collection
- Inventory availability checks
- Payment processing (Even if you use a third-party service, you need to log any events in addition to protecting yourself from service failures – more on this later.)
- Authentication and authorization logic
- Shipping estimation
- Tax calculation
…and the list goes on. Not all checkout failures will prevent your user from physically completing the checkout process if you’re doing it right, but many of them decrease the probability that your customer will complete their order. Any squeak in the cog of the checkout process can result in revenue loss.
1. Add-to-cart. You can’t place an order if you can’t add products to your shopping cart or bag. This is a huge duh(!), but there are cases where shopping cart issues are intermittent and aren’t completely preventing people from adding products to their carts. The bigger danger lies in these occasional or even moderately frequent product adds that fail without your knowledge because orders continue to flow through your website. Causes might include bad inventory availability checks, bad data, or an intermittent bug that’s difficult to reproduce. Protect yourself by logging all add-to-cart failures when they occur, send out an alert, and investigate the problem ASAP. Honorable mention goes out here to logging Wish List and Favorites errors as well. These features are about customer experience, but they’re also proven winners when it comes to converting customers.
1. Online Catalog/Product Page. Your customers can’t add items to their shopping cart if they can’t find your products. Similar to adding to their shopping carts, the danger here is the intermittent problem that doesn’t always rear its head in an obvious manner. The larger your online catalog, the greater the chance you’re going to miss errors with specific product lines, markets, or other logical groups of products. Bad data is a likely suspect here, especially if you’re dealing with very old data and/or integration with legacy software that has forced you to compromise a bit on data storage constraints. In addition to product and category pages themselves, consider logging issues with suggested, related, and recently viewed products. You’re missing out on impressions and potential revenue if any of these processes fail.
1. Email Signup. Any marketing director worth their salt will stress the importance of collecting email addresses at every opportunity. Sometimes an email address is the only thing you know about a potential customer, and sometimes it’s all you need to turn a shopper into a buyer. Any break in this process should be identified and fixed immediately. If your website gets a lot of traffic, it doesn’t take much to lose out on collecting a significant number of email addresses and a significant amount of potential future revenue. The trick here is that much of your business logic for presenting email signup to customers may exist client-side. Don’t forget to catch and log those exceptions as well as server-side issues.
1. Login & Registration. Well, you want to get to know your customers much better, right? Of course you do! All the better to turn wonderful customers into wonderful repeat customers by providing additional features and more relevant content. Failures here can lead to frustration and exits from your website as well as a degradation in your ability to provide a more personalized experience. Pay attention to your authentication and authorization logic as a whole, meaning you need to go beyond evaluating simple form data submission and validation issues. Social media login errors, out-of-sync authentication and authorization cookies, and errors generated by any additional authentication checks should be logged.
I’ll wrap this up with a short discussion about third-party services. They’re not specifically called out in my top five areas listed above, but they can definitely impact the performance and user experience of your website. There’s a good chance you’re using at least one or more third-party services as a solution for different business problems your company needs solved, and it’s likely these third-party services are an integral component of critical website features. Whether you’re using a third-party service for calculating shipping, sending emails, or finding and displaying the cutest cat GIF of the day, the success of your site is depending on something that’s outside your control.

Protect yourself from these services. You should know their unscheduled “maintenance” periods better than they do. Chances are you’re in the position to decide which third-party services your company uses or you have significant influence over that decision. Either way, be informed as to how they perform and have data to illustrate where their services are falling short. Are they reliable and worth the eleventy billion dollars they cost, or should your company be looking elsewhere for that cute animated kitty GIF service? It’s likely you’re already protecting yourself from failures here with some fallback business logic, but make sure you’re logging and alerting yourself when exceptions occur. They can have a serious impact on your customer’s experience and could prevent shoppers from becoming future buyers. That’s especially true if these services play in the five areas I describe above (e.g. shipping, tax, payment, personalization, etc.)

And my last parting thought: The same considerations regarding third-party services apply when your e-commerce platform becomes more of a distributed solution where those distributed services and applications are your own. You will almost certainly have other performance monitoring and metrics in place for your own services, but you still need to log, aggregate, alert, and protect yourself from yourself!

About the author/Aaron Remaklus:

Aaron Remaklus is chief architect and development manager for Speedway Motors, Inc. He leads agile development teams building high availability, extensible e-commerce platforms. A Nebraska Wesleyan alumnus, Remaklus taps into a diverse background in business, marketing, training, and software development in tackling large-scale problems.
March 3, 2015
Q and A with Carmen DeArdo: DevOps at Nationwide

Just before the IBM Interconnect show in Las Vegas last week, we had a chance to speak with Nationwide’s Carmen DeArdo and talk about some of the challenges he has faced at Nationwide as DeArdo and his team work to bring DevOps and Agile principles to their delivery value stream.

DeArdo is no stranger here at devopsy.kinsta.cloud, and recently shared his column Nationwide banks on DevOps to drive towards Continuous Delivery. At Nationwide, DeArdo is the director of application development tools and technologies. DeArdo is responsible for driving continuous delivery utilizing DevOps, Lean and Agile techniques across mobile, distributed, mainframe and other technologies.

Here, we interview DeArdo on the importance of transparency into DevOps processes and the importance of attaining efficiencies throughout the delivery value stream.

devopsy.kinsta.cloud: Talk to us about how, as you optimized certain aspects of your delivery value stream, you found wait states in other areas of the chain created delivery backlogs?

DeArdo: If you actually look at the entire delivery value stream, we started out optimizing the middle because we started with Agile design, development and acceptance testing. We gained a lot of benefit out of that. If you take a step back and think about your entire delivery stream, and your continuous delivery efforts, what you realize is that on the upfront process you end up with wait states and lack of visibility flow at that end.

What is happing is that as your teams get better, they actually run out of things to do. There’s nothing for them to pull forward anymore.

Then, on the downstream side of delivery, you still have the issues through your path to production – especially if you have managed test environments. You still have manual processes, manual testing steps in deployment. Even if you’ve automated some of them, you still have manual processes. And you end up moving at the speed of the lowest common denominator. What you find is that you’re going to get diminishing returns now unless you start to address those wait states or contention points on either side of design, develop, and test.

These inefficiencies, on the planning side, start to bleed back into the business. Many times your business areas still have a fixed mindset in terms of their budget and yearly cycles. They’re not used to determining how to continuously feed high priority work into the backlog of your Agile teams. And then on the other side of the pipeline, how can you take what’s being produced by these teams and move that more quickly into your path to deployment?

devopsy.kinsta.cloud: That’s interesting. Your challenge is taking your continuous deployment and continuous improvement mindset and then feathering that into the annual and quarterly planning mindset of the business?

DeArdo: On one side, what you’d like to have is more of a continuous planning, continuous flow so you’re not dumping a lot of stuff in a bursty way onto your teams. They’re going from wait state to being overburdened. You don’t have that continuous flow. And then on the downstream side, how do you start to automate some of the things and get rid of some of those manual practices? We’re very good at deploying to our continuous integration and deployment to our dev environments.

But then beyond that, things tend to get bogged down. We sort of go slow at the beginning and slow at the end and go very fast in the middle. Well, if you’re going to improve your overall speed, you need to look at those beginning and end spots and say, “How can I try to address those?”

devopsy.kinsta.cloud: To improve throughput throughout, it sounds like you need more feedback, more planning throughout the year with the business.

DeArdo: There are a couple of things we need to do, and part of it is around driving more towards continuous planning. The other side is about of having an automated deployment capability. Giving the business more capability, it goes faster if it makes sense from a risk and value capability perspective.

We’re actually focusing more on that latter part right now. We are building this automated delivery pipeline, which starts with release planning and then goes all the way through deployment.

devopsy.kinsta.cloud: What are some of the high-level challenges that you’ve had optimizing that last mile phase?

DeArdo: I think one of the biggest challenges, although people like to talk about tools, is is around the variance in your own processes. If you have too much variance in your own process, then you don’t want to instantiate that into a tool because all you’re taking is a bad current state and making it a permanent current state.

So we have meetings with vendors. The last one we had, we’re pretty far along this path, but the last one we had, we probably spent 90% of the time just talking about the variance in our own process and the fact that it’s not one-size-fits-all, but you have to have a small number of patterns that you’re trying to instantiate and implement. You have to be able to tell the difference between business value-added variance and waste variance.

I sometimes quote “The Wizard of Oz.” It’s like are you a good witch or a bad witch? Are you good variance or are you bad variance? Everybody wants to think what they’re doing differently has value, but in reality, most of it is arbitrary. There might have been a good reason for it at one point, but at this point, there is no value add in having that variance.

One of the variances we have is our path to production: what environments, what phases and things like that. One pattern might be an urgent need to get a business capability out very quickly. So I have a certain environment, a certain way to get certification in an expedited way. It’s not about going through an environment. It’s about proving that you have the quality that you need to deploy.

That one pattern might have a lot of automated testings run in one environment that you can get to very quickly to certify the quality and then release it. Another pattern might be around a record keeping system which is managing financials. You just have things that you have to go through. The system will have dependencies. So you’re going to go through a more elongated path to production.

That’s just an example of two patterns you would use based on a technology and a business need, not just because I’m in Organization A versus Organization B or Organization C. I don’t care what organization you are, if it fits this pattern and this technology, if it fits this pattern, use that and then if it fits this other one, use that. Those are good variances based on the fact that there’s a business need. There’s a technology driver that defines how you’re going to deploy.

devopsy.kinsta.cloud: That is an excellent explanation. I think many people get caught up in their “one right way” of doing things and have a tough time looking objectively at alternative ways.

Let’s talk about your “hidden factory” metaphor. Is the “hidden factory” a metaphor for a lack of transparency into different areas of IT?

DeArdo: Right, and the fact that you really don’t have a model. You have views. One of the things I point out is that you have a business development view, you have a project planning view, and you have release metrics, project metrics, and you have a release view. You have a path to production. None of that is visible. Some of it is hard to find. Some of it is visible if you’re in the tool of choice. But there is no visibility and no linkage or integration of that as you move from putting in your business needs at one end and seeing the code moving into production at the other end.

DevOps is about shining the lights on that. My view is you can’t manage what you can’t see. Another example sometimes I use is a basketball coach. Let’s assume you’re a basketball coach but you’re not allowed to watch the games. What you get after every game is a score sheet. It has the score. It has maybe your shooting percentage and turnovers, assists, but that’s it. Now you have to practice for the next game to work on improving something. Well, how do you know what to improve? So let’s say your shooting percentage was low, assuming you have enough metrics to know what low is.

Maybe your shooting percentage is bad. Maybe you think, “I’m just going to practice shooting.” Well, maybe that’s not it. Maybe the wrong people took the shots. Maybe you took contested shots, bad shots. There are a number of reasons you can have a bad shooting percentage, none of which have to do with the fact that you have bad shooters.

But that’s sort of the way we manage IT today. We’re asking executives to manage things they can’t see based on a set of metrics, but they don’t really watch the game.

March 2, 2015
Composable Environments for Continuous Delivery
The Continuous Deployment model discussed below requires some very specific properties for its platform substrate. Currently AWS is the the only environment that supports all the features needed to implement this. So when we start talking about implementation details in this, and future articles, we will be targeting AWS.

The title of this article is actually reversed from what it should be. This article isn’t as much about using reproducible Composable Environments for Continuous Delivery (CD) as it is about using Continuous Delivery to build reproducible Composible Environments. The Composable Environment is the important piece. Why? To understand that we need to understand what such an environment is. An Environment is a self contained, full software stack that includes every component needed to support the stack along with a version controllable description of that stack. For example, an Environment for a website might be the all of the machines that run the HTTP Servers, other machines that run the RESTful Web Servers, databases, AMQP providers etc. It would also include any infrastructure required for auto scaling, fault recovery etc. This Environment would be completely described in a declarative way by one or more files that are under version control. That description contains every virtual machine image, database schema, DNS server, DNS entry, etc needed to standup a system in a self contained, replicable form that can be put under version control. This description is versioned in a sane way and reproducible through the execution of a single command. A good example of this concept is Cloud Formation Stacks. Stacks declaratively describe a full environment and the Cloud Formation infrastructure takes those stacks and, on request, vivifies them into full running system. Our concept of environment takes that further with ideas around isolation of environments and accessibility via DNS, but these examples provide a good seed for the concept we are describing.

Reproducible Composable Environments are a simple concept that has profound implications. A few of the most important are:
- A completely reproducible environment is self contained. Multiple instances of that environment don’t interfere with one another.
- A declaratively described environment can be changed algorithmically. That is, it can be easily changed via some automated process.
- An environment can be deployed algorithmically can also be un-deployed algorithmically.
Once we have created an environment with these properties, we can use it to accomplish things that where not previously possible. It opens up new opportunities through a combination of Testability, Composability and Deployability. You might notice that this sounds like the move towards Structured Programming that happened in the programming community in the 1960s. That transition had a profound and continuing impact on how we create software systems and I think this approach to Devops will have a similar impact on how we create and manage environments.

Why Environments are Important

These environments are useful through out the software development process. Start with the individual developer writing and iteratively testing the system. Having such an environment allows him to truly test the changes that he is making by standing up a full environment, including all its participating parts, and testing that system in context. The next big place for using such a system is during Continuous Integration, henceforth CI. At the end of the build cycle the CI system creates an environment, publishes the changes into that environment, stands up the complete system and runs the full test suite for the entire system. The ability to standup the full system in context massively reduces risk in rolling out changes. Being able to roll out, at need, an environment with updated components in an ad-hoc or automated manner improves the testability of our system.

Automated processes are the correct method to build and modify environments, and are required for Continuous Delivery.

Continuous Delivery

I define Continuous Delivery as a idempotent, automated process that is primarily designed to update a reproducible Composable Environment with new, well tested artifacts.

We can think of Continuous Delivery as a process with a set of well-defined inputs and a set of well-defined outputs. The process itself is composed of a set of stages, each of which have its own inputs and outputs. This allows us to compose stages and even whole CD processes into different processes that serve new purposes.

For the simplest CD process, the main input is source code and the main outputs are the services that are stood up as part of that system. To be more specific, the outputs are a set of endpoints to which you or your customers connect. In examples I will provide, these will take the form of DNS entries. Knowing the inputs and the outputs, we can now start to think about the process in between.

The Stages of CD

The standard CI pipeline is composed of five parts: Develop, Test, Publish, Deploy and Promotion

Stage: Develop

The Develop stage is essentially the individual developers development process. Its only input is the output of the CD process that automatically creates and provisions the Development Box as well as the related environment. This is the first example of a CD input being the output of another CD process.

The reason Development box is necessary is so that the developer can test his work in an environment that exactly matches that of the CI server. It should also exactly match the system that every other developer is working on. This is critical, and often overlooked. Currently, developers rarely write code in development environments that match each others, let alone ones that match the CI server.

Stage: Test

The Test stage is where we take the artifacts produced in the Develop stage and test their viability in context. The output of the test phase is usually a more complete set of artifacts published into the environments.

These Stage usually takes the form of taking developer code, compiling it, turning it into some package-like structure and publishing it to an environment specific known location. The structure is usually an actual package: a LXC/Jail container like a Docker image, a virtual machine image like an EC2 AMI, or some combination of the three.

During the test phase, the target environment is stood up with the new artifacts, and a series of integration/release tests are run. Test reveal the viability of the changes within the context of the environment as a whole.

Stage: Publish

The Publish stage takes the output of the Test phase and publishes those artifacts in a way that makes them available to the Deploy stage. This usually involves putting the packages in a publicly accessible place.

You may have noticed our heavy reliance on some packaging model. That is intentional. You need turn your system into a manageable unit in order to make it work correctly as part of the process.

Stage: Deploy

The final stage is Deploy. This is where the published packages are used as inputs to update the environment. The output of this stage is the canonical environment updated and vivified with the newly published artifacts. The running systems, provisioned databases, accessible URLs, etc are all updated in place.

Stage: Promotion

Finally we have the optional step of promotion. This simply involves moving the artifacts outputted from the Deploy state to a new environment. A common example of this is the promotion of a beta environment to production, where some secondary process has been run that has validated the environment making it available to consumers.

Overcoming Technological Constraints

I am not going to lie to you. It is non-trivial to get started with this process. Up until a very few years ago, I would have said it is impossible. But, the world has changed, and it is now possible, even approachable. There are some prerequisites that we have to get it working. You must have the ability to programmatically configure your network, configure your DNS, provision hardware etc. I can accomplish this with a mix of new technologies like OpenStack, OpenFlow and customized DNS among other things. Most of us don’t have the ability to implement those technologies in our data center. Fortunately, we have AWS. Between EC2, Route 53 and especially CloudFormation, we have nearly everything needed to make this happen. Over the next few articles, we are going to take the model described here and apply them on a production system. At the series conclusion, you will be able to apply this model in your own AWS-based environments. You should still be able to adapt and apply parts of this model in non-AWS environments, as well.

And, my team will keep implementing and refining our ideas for composable environments for continuous delivery towards having a more complete solution. Stay tuned!
February 26, 2015
Donnie Berkholz on IoT and the Modern Developer

It’s been nearly four years since Marc Andreessen wrote his famous column Why software is eating the world. It’s proven to be incredibly prescient. While software and the Internet has removed many market frictions from many businesses, software is now just starting to “eat” the physical world as adoption of the so-called Internet of Things accelerates.

Before his talk, IoT and the Modern Developer, at IBM InterConnect, we caught up with RedMonk senior analyst Donnie Berkholz to discuss how software is in fact now eating the physical world and how the Internet of Things is now more accessible than ever before, and like the description of Berkholz’s talk says, “We live in a world of glue code, where open building blocks are available to Developers to snap together the pieces of the next big thing.”

devopsy.kinsta.cloud: Thanks for taking the time Donnie. Can you tell us a bit about your talk, IoT and the Modern Developer?

Berkholz: The idea is that IoT has been really hard to work with for a long time because a lot of it has required working with the Arduino tool set, for example, which is a lot of C++ and a lot of developers these days aren’t very interested in working in low level languages anymore. A lot of them want to work in Python or maybe Go or Node, or some form of JavaScript. The challenge for a lot of these embedded devices has been that they’re just really inaccessible to the way people want to write software these days both from a perspective of languages, but also toolsets. For instance, the library support just is not great.

It all kind of ties together into the same idea of IoT hadn’t taken off for a long time. Not from a developer perspective. Independent developers weren’t doing a lot of playing with, largely because it was a pain in the ass to work with. Whereas more recently we’ve seen stuff coming out like the Raspberry Pi, and the BeagleBone. The Espruino is a really good example and as is the Tessel of this new generation of IoT that is easy to work with and is super affordable, and you’re working with it, in most cases, in Node. Every one of those more current boards supports running Node on the embedded board itself which is a huge change.

The old point of view used to be having JavaScript on the server and there was a central server that was dealing with all this stuff. But the new perspective is that all these embedded boards are powerful enough that you can actually run JavaScript on every single one. There’s never a point at which you have to interface down to some low level language that you’re not interested in working on.

That’s basically the point of the talk and it obviously goes into a lot more detail with lots and lots of examples of what’s happening, how the trends have changed over time.

devopsy.kinsta.cloud: That is very interesting, do you know of real-world examples?

Berkholz: One that I particularly enjoyed is the Internet of toilets. There’s a good company called Hagleitner. They are doing smart restrooms. Big data with washrooms is how they frame it. They’ve Raspberry Pis sitting in these washrooms running, it was at an F50 conference, they’re running SQL Anywhere and integrating with the HANA Cloud platform. And they’ve all the kinds of instrumented paper towel dispensers and that sort of thing so that they can get on-demand maintenance instead of just basically pulling-based maintenance. Obviously it’s the way we want to write software. It’s the same way. It would be great if we could do that in the real world.

devopsy.kinsta.cloud: Where do you think IoT is heading in the next years ahead?

Berkholz: We’ve run a show called IoT at Scale. We ran one last fall. We’re going to run another one early summer in Europe. The last one was in Palo Alto and the basic premise is that the IoT that’s real today, the IoT that’s easily commercializable today is industrial IoT. With enterprises, all you have to do is show that there’s ROI there. You have to show that if they invest in this instrumentation, they’re going to save money. They’re going to lower risks.

Whereas with consumer IoT, it’s basically a guessing game. You’re gambling every time you do and sometimes you win, but a lot of times you lose. And so the talk runs through all kinds of examples of what’s going on with industrial IoT and in retail, in construction, in smart buildings, in connected cars, fleets of trucks, all sorts of different things like this. In every one of those cases, it’s a commercial business that’s making money right now and keeping companies alive.

devopsy.kinsta.cloud: That is great to see. Going back 13 years or so at InformationWeek, we actually had a beat called Telematics and it was about covering all of that. It was mostly transportation, but it was so far ahead of the time that we couldn’t find real use cases so we shut it down.

Berkholz: Exactly. That’s the thing that we’ve been waiting for is all the vendors have been suggesting use cases for years and they think wouldn’t it be cool if somebody did this? That would be awesome, right? But they could never come up with any customers and I think that’s the big change is a lot of these examples are actually customers that have developed their own solutions using vendor products as part of them. GE is a reasonable example. They built their own software stack for IoT and they hit a billion dollars in revenue last year. If that’s not a proof point, I don’t know what is.

devopsy.kinsta.cloud: That’s a great proof point. How do you see this affecting the work load of operations and developer teams? Are they going to have to manage an IT layer of physical devices?

Berkholz: Yes. It’s definitely going to require extending the DevOps ideas of collaboration to field teams as well, to utility workers and that sort of thing that are out there actually maintaining smart meters. The equivalent of your rackers and stackers in the data center are these field techs who may not be well educated in the technology at all, but still have to understand enough to maintain the hardware on premise and to manage to do software updates. I think that’s one of the critical aspects and certainly one that probably resonates with you from a security perspective is all these embedded devices get sold and then either they’re never updated even though they’re connected or they’re updatable.

devopsy.kinsta.cloud: That’s exactly right. A lot of it feels like it’s 1999 and the Web all over again.

Berkholz: Yes, exactly. The Internet was not architected with the idea of hackers in mind. We should learn from that lesson as we’re building up the IoT and as we’re creating and experimenting with protocols so that we can make sure we architect in security from the ground up instead of doing the same thing all over again.

devopsy.kinsta.cloud: What are the challenges in working with IoT management teams and building efficient Ops in this environment?

Berkholz: Yes, I think it’s going to be a very different model. We’re going to have to take a step back in a lot of these cases because on the tech side of things we’re moving forward into more and more technological methods of collaboration, using more and more virtual, using tools like Slack or HipChat or others and relying on that to work together whereas a lot of these people are not at all comfortable with that. In many cases, they may not even use email. I live in Minnesota which is not exactly a software hub, as you, and I’m sure you know many people who may not own computers.

I think a lot of it is going to have to rely on having field teams out there with much smarter devices themselves that are a step up from the deployed IoT stuff. Like the ruggedized laptops and PDAs that they’re carrying around. So there’s going to have to be a lot of intelligence on there that can cope with what’s happening out in the field and on the back end it’s obviously going to require very scalable and intelligent software to figure out what’s going on out there in the field.

Techs need to know if alerts require action be taken, or if everything is all good. Because if I can’t figure out if I have to take action until later, that’s going to cost the company money to send somebody back there in person.

devopsy.kinsta.cloud: That brings up a great point. Mistakes here could be potentially expensive, potentially much more expensive than software mistakes because you’re talking about physical devices. Errors in planning or design can be monumental.

Berkholz: It absolutely increases the risk profile. It’s the same thing that software on things like space shuttles, satellites, airplanes have had to cope with for years and years and years. That’s why in a lot of cases they have to use specialized software development techniques and languages, like Ada is a particularly popular one, to basically prove that they’re going to be reliable, to prove that the architecture is solid, which interestingly is the same kind of approaches that Amazon is taking with building its cloud. I don’t know if you’ve looked into what they’re doing with TLA+?

They’re proving that their architecture works from a mathematical perspective. I think that’s the same approach that’s going to have to happen with IoT is we’re going to have to do a lot better job of building automated testing and of being able to rapidly roll back and roll forward and of using much more strongly protected languages and testing tools.

devopsy.kinsta.cloud: What things should developers who are looking at IoT for the first time now, as a potential platform to develop upon, be looking at or self-teaching themselves?

Berkholz: I think for now, for the time being, the answer is the way we start a conversation which is picking up cheap embedded devices like BeagleBoards or Raspberry Pis and putting Node on them and playing with Node.

That seems, to me, to be the place to be with the Internet of Things right now. Come up with a project that sounds interesting and do it.

February 25, 2015