devopsy.kinsta.cloud

Category: Blogs

Tuning in to the real pain of customers

In my product management and marketing experience, I discovered that I carry a deep bias about our product capabilities and why it’s special — before a launch. On a whiteboard, in a spec, or in a usability lab, our product can and will change the world. Post-launch, when customers get their hands on our product, magical things happen. Intended use cases become irrelevant. Intended benefits change. Intended value changes.

Over the past weeks, I’ve had the privilege of talking to many of our customers and prospects. They educated and helped me reframe the value we deliver to our customers.

Real pain

Much has been written about the DevOps revolution, driven by applications moving to the cloud, mobile computing, and the business need to compete. The journey to an agile build-test-release cycle is real, and it’s hard. It’s hard for customers who are cloud-born, and it’s even harder for traditional operations and development teams. Listening to our enterprise customers who are several years into their transformation, I heard a lot about the explosion of technologies and the cultural challenge. The pain of the change is more excruciating than I realized. One large customer told me that two years ago, he (VP of Cloud Operations) and his team couldn’t stand to be in the same room as the development team. Today, they’re collaborating to the point where they now sit together.

Demanding developers

Secondly, developers are more demanding. Again, it seems obvious, but hearing firsthand how hard it is to empower developers brings it home. Developers want and require more agile build, release and deployment capabilities, but they don’t want to be infrastructure experts. Enterprises are beginning to offer self-service to their private and public clouds so that developers can have autonomous access to environments and resources. A leading biotech customer told me that he can’t dictate policies and processes anymore, that he needs the researchers and scientists to adopt and embrace what the DevOps team is providing. Finding the appropriate balance between freedom and speed along with control is a common challenge across the board for our customers.

Delivering value

At ElasticBox, we talk a lot about the technology and cultural shift in enterprise DevOps teams, and how we can enable greater collaboration by being at the nexus of the transformation. That is what we ultimately deliver to our customers: a new way to deliver applications to the cloud that’s better, faster and more efficient.

It’s a big promise, and we’re excited to fulfill it. Today, we deliver the foundational tools for self-service provisioning, service catalogs and continuous integration and delivery — wrapped with a flexible box architecture that’s reusable and agnostic to the underlying infrastructure and deployment tools. We approach the problem from a developer mindset to drive speed and autonomy, and at the same time, provide the policies and reporting to ensure scale. We’re hungry to continue learning from you, so check us out and let us know how we can help.

March 3, 2015
Digital Transformation with Agile and DevOps – Something “Wicked” This Way Comes
Our world is full of wicked problems, like climate change, urban planning and social injustice. They’re called wicked because they’re hard to recognize, define and solve. Furthermore, complex interdependencies mean solving one aspect will just expose or create other problems.

Today’s digital transformation imperatives are creating wicked problems for IT. So wicked in fact, they make managing technology in the traditional sense look like child’s play.

Suddenly, businesses are faced with complex dilemmas. How can they compete in a world where their customers place less emphasis on physical products and more on the total experience? How can we adjust marketing from driving consumer behavior to responding to it? How can they adapt business models in context of the customers’ world? – a world that’s increasingly dependent and driven by digital technology.

The short answer is of course software, but the problem is our approach to delivering it just isn’t suitable in a wicked business world – why? Well its simple – we keep working on the assumption that we understand the problems we’re trying to solve, when in fact we don’t understand the problems of our customers at all – that is, until we have a solution.

Head scratching stuff huh, but consider this –

Did you know you had an entertainment problem before Apple gave you iTunes? Was booking hotel accommodation such a pain before AirBNB introduced the whacky notion of people renting out their own rooms?

So faced with such wickedness what can we do? Well, we can start by accepting our current IT failings, develop some wickedness of our own, and mitigate against what can derail your efforts.

Small is Beautiful … really it is!

In the past we’ve tried to solve business problems with what I call IT “bigness” – big iron, big software applications, big teams, big projects, and big expectations – oh, and let’s not forget big funding. The results, however, haven’t set the world on fire have they? – Big IT white elephants, big budget blowouts and big executive fallouts (literally and through the revolving door).

But surely we can’t lay the blame squarely on IT. After all, traditional models of software design and project management where we diligently survey the client’s needs are baked in common sense. Add to this the rigor and procedure around sign-offs, budgeting, design, development, testing and deployment – doesn’t this guarantees success?

Of course it doesn’t. Apart from the big fat fails, we also end up with what I call a big “sucking culture”– your software sucks, IT operations sucks, this job sucks, our business sucks – we’ve all been there, haven’t we?

All of this isn’t due to bad people and technology. It’s because we work on the premise that the business understands what it wants – it doesn’t. Much better then to put aside “bigness” and start listening (maybe insisting) that the business first lays all the wicked cards on the table, like – what are our markets and how is this changing? What are our target demographics? What are our barriers to entry?

After all this ‘warts and all’ revelation, it’s time to get busy. Now, with a shared understanding, we can start quickly prototyping elements that perhaps only solve one small problem. Then we continuously develop, test and demo, knowing and accepting other problems will surface that need to be solved. And, as we know from agile development, it’s all a case of ‘rinse-and-repeat’ – hopefully achieving a state of nirvana – giving your customers software answers to problems they don’t know they have. Problems that lead to increased market share, revenue and profitability.

So is a liberal scoop of Agile or its younger DevOps cousin the answer? Not necessarily — that is until some essential toppings and flavors are added to my somewhat vanilla perspective:
- Executive Flag Waving – try selling a methodology or philosophy based on the premise you’re trying to solve problems you don’t yet understand. Good luck with that. If executives simply want faster, better, cheaper, it’ll be much easier (and safer) to hold back grandiose talk about experimentation, “failing fast” and continuous improvement. Understand, however, that without tangible commitment, half-assed agile or DevOps is like a fast melting ice-cream cone – messy and hard to handle.
- Behavioral Hacking – even with executive champions, persistent and rigid organizational behaviors will need to change. Just hiring lots of agile coaches or mandating DevOps won’t work until there’s willingness for people to be engaged around new ideas and fresh thinking. With that in mind, be prepared to do some heavy lifting – socializing and selling new concepts, but also looking for opportunities to show how the smallest of changes (e.g. team refactoring, incentives) lead to improvement.
- Show me the Money! – So how to you fund agile and DevOps? It’s going to be tricky to cost something exactly when you don’t know the exact outcome of a new project, but businesses may initially insist on this to minimize risk. My advice here is to tread carefully. Stakeholders should be more interested in ROI than coming in under budget, so fixed-cost might not be actually what want anyway. Your job therefore is to convince them by continuously demonstrating ROI – not just from a single project, but also by the accrued savings that agile and DevOps deliver the whole organization when they systematically reduce technical debt and waste.
Developing the wicked traits I’ve described doesn’t mean IT has suddenly become an evil entity. Nope, it’s just a case of tapping into fantastic business opportunities in a constantly changing digital world. Agile and DevOps can help you solve and even create some wicked problems for your customers, but that involves confronting and challenging entrenched thinking as it relates to just about everything.
March 3, 2015
Webinar: How Hired helps DevOps engineers discover new opportunities

Rails v. Recruiting: How engineering is changing the way you’ll get your next job.

In an ideal world, your job search would be efficient, transparent, and flexible in scope. You would have a good idea of your value on the job market and you’d be able to leverage that to find the best opportunities available. The traditional job search methods—job boards, recruiters, and web forms—have left a lot to be desired.

Hired decided to tackle this problem with technology and innovation. They created a web application that powers the world’s first career marketplace. On Hired, software developers & DevOps engineers are introduced to over 2,000 companies in just one week. And here’s where it gets interesting—in the marketplace, the companies apply to the candidates. They’ll make introductory offers to users that provide compensation numbers upfront. The open marketplace encourages transparency and leverages the market’s demand for talent, meaning candidates get insight into their value and the most competitive offers. And the efficiencies created by this software-supported marketplace allow job seekers get more support, more services, and more opportunities.

Do you have questions about the job market for DevOps engineers or technical jobs in general? Interested in learning more about Hired and how it can help you on your job search? Join us on March 31st for a webinar and Q&A with the Hired team. And in the meantime, check out the all new devopsy.kinsta.cloud job page, powered by our friends at Hired!

Video

Slides

[seoslides embed_id=”6c907fb7a8bf” script_src=”https://devopsy.kinsta.cloud/embed-script/hired-slides/5972/” overview_src=”https://devopsy.kinsta.cloud/slides/hired-slides/” site_src=”https://devopsy.kinsta.cloud” site_title=”devopsy.kinsta.cloud” title=”hired-slides” /]

Scott Bacon, Talent Advocate

When Scott is not fundraising for his stealth salted meat startup (Bacon as a Service), he works as a Talent Advocate at Hired—the leading marketplace for engineering and sales talent. Prior to joining Hired, Scott worked onsite at Google as a technical recruiter and toured as a professional musician. Scott graduated from Columbia University with a degree in Anthropology or, as its informally known, “the gateway to sales and marketing jobs”.

Kyle Brett, Software Engineer

Kyle Brett was born with only five fingers on each hand. Despite this, he has become a reasonably talented software engineer. He has five years of professional XP and his first company, Virtrue, was acquired by Oracle. Today, he ships more bugs than any other engineer at Hired. Kyle prefers mechanical keyboards and emphatically believes that cats are better than dogs.

March 2, 2015
7 Great Thoughts on Real-world DevOps Transformations

Last week, IBM InterConnect brought together quite the gathering of IT pros engaged in DevOps, continuous delivery and digital transformations of their core businesses. Throughout the week, speakers from the Fortune 500 brought forth stories of real world experimentation, success and lessons learned from their DevOps journeys.

Here are some of the best chestnuts to make it out of the show:

Chris Nowak, senior vice president of DevOps for Bank of America

“We’re in sales. Our job is to sell DevOps inside the organization and unless I have the reporting and the data, I’ll get questions every week from CIOs that I can’t answer.”

Nowak recommends that organizations interested in gaining executive support through incremental shifts to DevOps and small wins along the way do their best to collect baseline metrics so they can offer evidence of the potential for improvement through DevOps.

Rich Moran, vice president of web architecture for Fidelity Investments

“When you automate bad processes, you just do bad things faster.”

A longtime architect, Moran explains he’s never seen a tool solve a process problem. In his opinion, DevOps transformations require organizations to think through processes and processes before implementing an automated pipeline, getting a good understanding of how software will be delivered, which groups are involved and when, what the flow will look like, how code will be checked and how approvals will work. Because when those details aren’t addressed and people just start running tools, that’s when they fail.

Gary Gruver, president of Practical Large Scale Agile

“You will probably write more code for test automation than you will code for running your product. And that is not bad.”

According to Gruver, test automation is one of the most important pieces to a DevOps transformation at enterprise scale and it’s also one of the things most often done wrong. Organizations must strive to getting it reliable, stable and maintainable.

Heather Cox, chief client experience digital and marketing officer for global consumer banking at Citi

“The payments and banking space is being fundamentally transformed. It has been said that people need banking, they don’t necessarily need banks.”

Cox kicked off the conference with this well-Tweeted thought, introducing the idea that large enterprises may be dragged kicking and screaming into DevOps and continuous delivery due to overwhelming competitive pressures. Small companies are now able to operate at previously impossible scale and completely disrupting enterprise dominance within their markets.

Jim Neumyer, senior application developer for HM Health Solutions

“Take small bites. Don’t shove the ice cream cone in our mouth all at one time.”

As Neumyer explains, the smaller the bite, the easier it is to tweak things along the way. Just as continuous delivery emphasizes iterative improvements in software development, the DevOps and Agile journey isn’t meant to be made in one massive leap.

Matt Ellis, vice president and CTO of architecture and technology for IBM Software Group

“You don’t know what the user wants until they see it.”

As one of the lead executives in charge of leading the internal IBM DevOps transformation, Ellis emphasized the importance of fast cycles of end user feedback in driving timely innovation through continuous delivery.

Pascal Eymery, vice president of strategy and business development customer support at Airbus

“The good thing about cloud and mobile is you can develop very quickly. You can scale fast if it works or just try something else if it doesn’t.”

Eymery’s organization has been tasked with helping deliver more value to airline customers through technological innovation. This includes developing iPad apps for pilots for Airbus cockpits that allow them to optimize takeoff and landing performance for fuel economy and baking in better telemetry improve maintenance windows. The fail fast and iterate mantra has been strong at Airbus.

March 2, 2015
Amazon Simple Notification Service (SNS)
Amazon has packed a box full of services and covered nearly every aspect in making cloud a smooth experience for developers to put up exceedingly operational and structurally complex applications on AWS. Large numbers of cloud-based systems feel the dire need of or dependency on sending out emails on a subscription basis to end users. But it is unfortunate that the internet today is brimming with spammers who refuse to stop sending requests and retarded emails from servers with objectionable SPF configuration.

Taking care of the above mentioned issues, AWS publicized an exceptionally great service known as Simple Notification Service (SNS), designed to the needs of genuine notification delivery.

What is Amazon Simple Notification Service (SNS)?

The Amazon Simple Notification Service (SNS) has made it straightforward and trouble-free for you to construct and put together an application in its own way.

Amazon SNS is a speedy, adaptable, flexible, entirely commanded and managed push messaging service that allows you without any difficulty to set up, operate and send notifications to all your subscribers. It brings in the simple and cost-effective push notification service to Apple, Google, Fire OS, and Windows devices even, including all the Android devices in China with Baidu Cloud Push. It can also be used to push notifications to internet connected with all distributed services and smart devices.

It is capable of creating topics, subscribe to topics and even publish topics in an exceedingly scalable, flexible and beneficial manner. By means of Amazon SNS, you can direct event driven messages from your applications to your users. In order to bar and avoid any sort of message loss, all the messages broadcasted across the multiple availability zones to Amazon SNS are stored constantly.

Let’s deep dive into the block displayed in the figure:
1. Topics: These are the access points or named groups of events for each identifying a specific subject, content or event type. Every single topic carries along a Unique Identifier (UI) that helps in identifying the SNS endpoint for publishing and subscribing.
2. Owners: Owners are those who create topics and have a complete admittance and access to the topic.
3. Subscribers: These are the clients usually the end-users, servers, applications or other devices that desire to receive notifications on particular topics of their interest.
4. Publishers: They are responsible for sending messages to topics. Now SNS matches up the topic with the list of subscribers interested in the topic and distributes the message to each one of them.
Features and Benefits:

Amazon SNS has come up with sufficiently great benefits over the complications of developing custom messaging resolutions in support of a system that demands to be supervised, managed and maintained on site. Following are some of the key benefits and features of Amazon SNS.
1. Reliable: Whenever Amazon SNS streams inside Amazon’s verified network infrastructure and data centers, all topics will be available each time applications calls for them. All messages broadcasted to Amazon SNS are stocked redundantly throughout numerous servers and data centers in order to prevent message loss.
2. Scalable: Amazon SNS is modeled to encounter the demand of most needy and demanding applications which permit event driven limitless amount of messages at any time.
3. Simple: For the majority number of cases the developer just by using only three API’S, can get started with the Amazon SNS which are: Create Topic, Subscriber and Publish. To provide you with intuitive functionality additional APIs are also made accessible.
4. Flexible: With Amazon SNS, applications as well as end-users on different devices are able to receive notifications with the help of Mobile Push notification (Apple, Google and Kindle Fire Devices), Email/Email-JSON, HTTP/HTTPS, SMS or Amazon Simple Queue Service (SQS) queues.
5. Secure: Amazon SNS is a secure service and makes sure that topics and messages are secure from any unauthorized access. All the Topic holders have a full authority of setting policies for each topic to put limit on who can publish or subscribe to a topic and can even make sure of the encryption via marking that the delivery procedure ought to be in HTTPS.
6. Inexpensive: Pay-as-you-go method helps in saving a lot of money of Amazon SNS customers. There are no beforehand fees or commitments. It only charges you per-request for the messages, notification delivery and transfer fees that are sent through Amazon SNS.
Amazon Simple Notification Service also supports SMS (Simple Messaging Service)

Amazon SNS leaves no means of notification untouched and comes with SMS feature to subscribe a phone number via an SNS topic. Notifications can directly be addressed as text messages to cell phones, Smartphone as well as on any other SMS-enabled device. Amazon CloudWatch users who are actually responsible for monitoring and controlling their applications that are running on AWS are capable of receiving the timely updates for actions of interest or alarms as they set. Content delivery is also possible to the mobile device through email to the existing Amazon SNS users. Developers can combine mobile applications in conjunction with Amazon SNS to not only send messages but even take delivery of notifications over SMS.

Here’s a figure to better explain how Amazon SNS supports SMS

Pricing:

For Amazon SNS usage you can pay-as-you-go and have no upfront costs. You pay money according to the number of notifications that you are publishing and delivering. You even have to pay if you use additional APIs for managing topics and subscriptions. Price of delivering these notifications varies according to the endpoint type.

Below are the tables that will provide you with full details:

Publishers:

Currently, Amazon SNS only permits maximum limit of 256 KB for published messages. Each 64KB block of published data is billed as 1 request.

Fig: 3

Notification Deliveries:

Fig: 4

Data Transfer: Data transfer ‘IN’ is for $0.000 per GB.

Below is the pricing table for Data transfer ‘OUT’:

Fig: 5

Conclusion:

Amazon SNS is designed to use with other Amazon Web Services such as Amazon SQS and EC2. For example, applications which are running in EC2 have the capability of publishing event/information updates to Amazon SNS and transfer them immediately to all other end-users. In addition to this the subscriber can also choose and select Amazon SQS as a delivery protocol that delivers notifications to multiple SQS queues in parallel format keeping in mind the perseverance of messages and a formal assurance that the messages would be delivered. If you are an Amazon SNS user then use must enable this service to your mobile devices and experience its benefits.
March 2, 2015
Q and A with Carmen DeArdo: DevOps at Nationwide

Just before the IBM Interconnect show in Las Vegas last week, we had a chance to speak with Nationwide’s Carmen DeArdo and talk about some of the challenges he has faced at Nationwide as DeArdo and his team work to bring DevOps and Agile principles to their delivery value stream.

DeArdo is no stranger here at devopsy.kinsta.cloud, and recently shared his column Nationwide banks on DevOps to drive towards Continuous Delivery. At Nationwide, DeArdo is the director of application development tools and technologies. DeArdo is responsible for driving continuous delivery utilizing DevOps, Lean and Agile techniques across mobile, distributed, mainframe and other technologies.

Here, we interview DeArdo on the importance of transparency into DevOps processes and the importance of attaining efficiencies throughout the delivery value stream.

devopsy.kinsta.cloud: Talk to us about how, as you optimized certain aspects of your delivery value stream, you found wait states in other areas of the chain created delivery backlogs?

DeArdo: If you actually look at the entire delivery value stream, we started out optimizing the middle because we started with Agile design, development and acceptance testing. We gained a lot of benefit out of that. If you take a step back and think about your entire delivery stream, and your continuous delivery efforts, what you realize is that on the upfront process you end up with wait states and lack of visibility flow at that end.

What is happing is that as your teams get better, they actually run out of things to do. There’s nothing for them to pull forward anymore.

Then, on the downstream side of delivery, you still have the issues through your path to production – especially if you have managed test environments. You still have manual processes, manual testing steps in deployment. Even if you’ve automated some of them, you still have manual processes. And you end up moving at the speed of the lowest common denominator. What you find is that you’re going to get diminishing returns now unless you start to address those wait states or contention points on either side of design, develop, and test.

These inefficiencies, on the planning side, start to bleed back into the business. Many times your business areas still have a fixed mindset in terms of their budget and yearly cycles. They’re not used to determining how to continuously feed high priority work into the backlog of your Agile teams. And then on the other side of the pipeline, how can you take what’s being produced by these teams and move that more quickly into your path to deployment?

devopsy.kinsta.cloud: That’s interesting. Your challenge is taking your continuous deployment and continuous improvement mindset and then feathering that into the annual and quarterly planning mindset of the business?

DeArdo: On one side, what you’d like to have is more of a continuous planning, continuous flow so you’re not dumping a lot of stuff in a bursty way onto your teams. They’re going from wait state to being overburdened. You don’t have that continuous flow. And then on the downstream side, how do you start to automate some of the things and get rid of some of those manual practices? We’re very good at deploying to our continuous integration and deployment to our dev environments.

But then beyond that, things tend to get bogged down. We sort of go slow at the beginning and slow at the end and go very fast in the middle. Well, if you’re going to improve your overall speed, you need to look at those beginning and end spots and say, “How can I try to address those?”

devopsy.kinsta.cloud: To improve throughput throughout, it sounds like you need more feedback, more planning throughout the year with the business.

DeArdo: There are a couple of things we need to do, and part of it is around driving more towards continuous planning. The other side is about of having an automated deployment capability. Giving the business more capability, it goes faster if it makes sense from a risk and value capability perspective.

We’re actually focusing more on that latter part right now. We are building this automated delivery pipeline, which starts with release planning and then goes all the way through deployment.

devopsy.kinsta.cloud: What are some of the high-level challenges that you’ve had optimizing that last mile phase?

DeArdo: I think one of the biggest challenges, although people like to talk about tools, is is around the variance in your own processes. If you have too much variance in your own process, then you don’t want to instantiate that into a tool because all you’re taking is a bad current state and making it a permanent current state.

So we have meetings with vendors. The last one we had, we’re pretty far along this path, but the last one we had, we probably spent 90% of the time just talking about the variance in our own process and the fact that it’s not one-size-fits-all, but you have to have a small number of patterns that you’re trying to instantiate and implement. You have to be able to tell the difference between business value-added variance and waste variance.

I sometimes quote “The Wizard of Oz.” It’s like are you a good witch or a bad witch? Are you good variance or are you bad variance? Everybody wants to think what they’re doing differently has value, but in reality, most of it is arbitrary. There might have been a good reason for it at one point, but at this point, there is no value add in having that variance.

One of the variances we have is our path to production: what environments, what phases and things like that. One pattern might be an urgent need to get a business capability out very quickly. So I have a certain environment, a certain way to get certification in an expedited way. It’s not about going through an environment. It’s about proving that you have the quality that you need to deploy.

That one pattern might have a lot of automated testings run in one environment that you can get to very quickly to certify the quality and then release it. Another pattern might be around a record keeping system which is managing financials. You just have things that you have to go through. The system will have dependencies. So you’re going to go through a more elongated path to production.

That’s just an example of two patterns you would use based on a technology and a business need, not just because I’m in Organization A versus Organization B or Organization C. I don’t care what organization you are, if it fits this pattern and this technology, if it fits this pattern, use that and then if it fits this other one, use that. Those are good variances based on the fact that there’s a business need. There’s a technology driver that defines how you’re going to deploy.

devopsy.kinsta.cloud: That is an excellent explanation. I think many people get caught up in their “one right way” of doing things and have a tough time looking objectively at alternative ways.

Let’s talk about your “hidden factory” metaphor. Is the “hidden factory” a metaphor for a lack of transparency into different areas of IT?

DeArdo: Right, and the fact that you really don’t have a model. You have views. One of the things I point out is that you have a business development view, you have a project planning view, and you have release metrics, project metrics, and you have a release view. You have a path to production. None of that is visible. Some of it is hard to find. Some of it is visible if you’re in the tool of choice. But there is no visibility and no linkage or integration of that as you move from putting in your business needs at one end and seeing the code moving into production at the other end.

DevOps is about shining the lights on that. My view is you can’t manage what you can’t see. Another example sometimes I use is a basketball coach. Let’s assume you’re a basketball coach but you’re not allowed to watch the games. What you get after every game is a score sheet. It has the score. It has maybe your shooting percentage and turnovers, assists, but that’s it. Now you have to practice for the next game to work on improving something. Well, how do you know what to improve? So let’s say your shooting percentage was low, assuming you have enough metrics to know what low is.

Maybe your shooting percentage is bad. Maybe you think, “I’m just going to practice shooting.” Well, maybe that’s not it. Maybe the wrong people took the shots. Maybe you took contested shots, bad shots. There are a number of reasons you can have a bad shooting percentage, none of which have to do with the fact that you have bad shooters.

But that’s sort of the way we manage IT today. We’re asking executives to manage things they can’t see based on a set of metrics, but they don’t really watch the game.

March 2, 2015
Repex – Regex Management Simplified
Our Versioning Complexity

Cloudify is a Python shop.

Our REST service is Python.

Our Workflow Engine is Python.

Our Plugins are Python.

We have different version formats and different dependencies across different types of files which need to be changed when a version is updated.

This is complex, to say the least. We didn’t want to manage this manually having more than 60 repos to take care of when handling version updates.

Why not sed?
- sed is a bitch, isn’t it? It’s unmaintainable and unmanageable. While this is solely my opinion, handling hundreds of files, each change in a different context without any safety net, makes sed something you should walk away from.
- sed isn’t configurable.
Why not Jinja?

Using Jinja templates, while safe, was out the windows pretty much instantly as we needed the repos to be usable out of the box or we’d be forcing users to change templates so that they can use our product.

The Solution

We turned to Regex. Yes, there is the regex problem. You’re probably familiar with the famous quote:

–Some people, when confronted with a problem, think “I know, I’ll use regular expressions.” Now they have two problems.

While this can be true, it also depends on how you use regex.

I believe we’ve done this well.

Repex

We developed repex.

repex allows you to specify the structure of changes you need to perform in YAML format:
```
variables:
  replace: 3.1.0-m2
  version: 3.1.0-m4

paths:
  - description: demo thing
    path: repex/tests/resources/mock_VERSION
    match: '"version": "3.1.0-m2"'
    replace: "{{ .replace }}"
    with: "{{ .version }}"
    to_file: repex/tests/resources/mock_VERSION.test
    validate_before: true
    must_include:
      - date
      - commit
      - version
  - description: handles core modules version in all core repositories
    type: setup.py
    path: cloudify-.*
    excluded:
      - my/excluded/path
    match: version=('|")(\d+)(\.\d+){1,2}(dev|(\w+\d+)?)('|")
    replace: (\d+)(\.\d+){1,2}(dev|(\w+\d+)?)
    with: "{{ .python_core_version }}"
```
You define a list of paths you’d like to iterate over, provide it with what you’d like to search for, what you’d like to replace, what you’d like to replace with, and some more parameters and off you go.

For example, the first object in the above configuration will:
- Look for a file in the path repex/tests/resources/mock_VERSION (regex)
- In the file, look for `”version”: “3.1.0-m2″` (regex)
- Attempt to replace 3.1.0-m2 with 3.1.0-m3 only if the string was found during the matching stage and if the strings `date`, `commit` and `version` were found in the file.
- Write to the output file repex/tests/resources/mock_VERSION.test
The second object will:
- Look for `setup.py`(regex) files under `cloudify-.*` (regex) and not under `my/excluded/path`.
- Look for `version=(‘|”)(\d+)(\.\d+){1,2}(dev|(\w+\d+)?)(‘|”)`(regex) in the files found.
- Replace `(‘|”)(\d+)(\.\d+){1,2}(dev|(\w+\d+)?)(‘|”)`(regex) with a variable which is not harcoded and therefore must be provided using the API.
Layers of protection

repex attempts to provide layers of protection and comfort:

Safe!
- You can first `match` whatever you want to match so that you’re sure you’re only replacing something in the exact context you wish to address. If `validate_before` is true, replacement will only occur if one or more matches were found.
- You can verify that only files containing very specific strings will be addressed by listing them under `must_include`.
- You can exclude files or directories you don’t want to search in by listing them under `excluded`.
Don’t take this the wrong way though – regex is regex. You make a mistake, you pay for it. All repex grants you is the ability to have everything organized in one place, with some layers of protection to keep you from making some trivial mistakes.

Comfortable

You can declare every single directory or file you’d like to address in one manageable YAML.

Of course, we needed to create some kind of naming convention so that we don’t have to configure each and every file separately.

Accepts Variables, both hardcoded and via an API

One of repex’s strongest features is that it supports using variables for string replacement. The above example contains a {{ .version }} variable.

The API allows you to provide a dict of variables which will be placed in placeholders. You can either hardcode a variable in the variables section or use the API to send a dict of variables to be used. If a variable is hardcoded in the YAML and you use the API to send a variable with the same name, it will overwrite the hardcoded one. In addition, validation will occur upon variable replacement to make sure it was replaced.

Functional API

The API provides three basic functions. I’d rather not rewrite repex’s documentation.. so just refer to this.

Basically, the three functions allow you to decide the level of granularity in which you’d like to handle the changes. It will allow you to perform actions in different stages of the replacement process.

For example, we don’t really use the highest level of the API which just iterates over all objects in the YAML. We wrote a Cloudify specific wrapper which can be used as a reference implementation. It enables validation after every replacement. So, for instance, after every blueprint specific replacement, we run a validation that the blueprint is valid vs. our DSL-parser.

Repex is available on PyPI. You can install it by running:

pip install repex

We might, at some point, provide a CLI for repex. Also a logging feature might be implemented to allow for JSON formatted log messages to be generated so that they can be transported for analysis.

After using repex throughout our entire build process for the last five months or so, it has proven to be extremely useful. We can now follow our entire version update process and easily identify problems by looking at one YAML file and one execution log file. We’re planning on using repex to replace additional types of data (not only versions).

We would appreciate feedback and, even better, pull requests.

Hope you find this useful.
March 2, 2015
DevOps Perspectives from Pink15

I recently had the privilege of covering the Pink15 conference on behalf of devopsy.kinsta.cloud. Pink Elephant is a renowned leader in ITIL and ITSM education, consulting and conferences. Now in its 19th year, the Pink conference brings together attendees from a wide range of vertical markets to learn and share about IT service management and related topics. My mission was to take the pulse of DevOps in the ITSM enterprise community, many of whom represent the operational side of IT.

Let me start by saying that this was a well-organized and executed conference. The attendees that I spoke with unanimously agreed that the sessions were relevant and the peer networking invaluable. And while there was not a specific Devops or Agile track, the core concepts were directly or indirectly present in several of the sessions.

One particularly interesting session was on “Enterprise Change Management in an Agile, Fast Moving World” from a large retail organization. The presenter did a good job of describing how they are adapting their categories of their changes to meet the growing requirements for a more agile change management process. At present, only 5% of changes are categorized as “fast track” – an even more expedited route than what is normally considered a standard change. This organization’s goal is to exponentially increase their ability to fast track more changes in response to increasing demand. This session was a good indicator of how enterprise IT is recognizing the need to adapt for continuous delivery and more automation.

More than one attendee shared their appreciation at being able to interact with other IT organizations about specific ways to bridge the practices between Dev and Ops so as to make service management more agile. One person mentioned that while some of the sessions did not specifically call out DevOps or Agile, presenters often had to address these in depth during the question and answer period. There was particular interest among attendees in learning how to engage developers in processes such as Event, Demand, Capacity, Availability, Change and Release Management.

Some attendees expressed concern about the perception that ITSM and ITIL have no place in a DevOps culture or that DevOps would somehow invalidate the investments that many organizations have already made in establishing better processes for managing services. Kaimar Karu, the ITSM Product owner for Axelos (who owns the IP for ITIL) strongly believes that ITIL is absolutely a key player in the realization of DevOps as a philosophy. ITIL processes are not specific to operations – they enable the entire lifecycle of a service or product from development to deployment and into operations. ITIL was meant to be adopted and adapted; nowhere does the guidance recommend that the processes be complex, bureaucratic, lengthy or manual.

So what does all of this mean? While DevOps is not yet fully understood, organizations are taking tangible steps to make ITSM processes go faster, integrate with other practices and leverage automation in order to deliver more business value. Wouldn’t you call that DevOps?

February 27, 2015
IBM InterConnect: HM Health Solutions shows how continuous delivery helps cure healthcare IT headaches

As a spin off business unit of health insurer Highmark, HM Health Solutions is tasked with helping not only that insurer but also other third-party insurers across the industry to optimize systems that run core functions like benefits design, member enrollment, claims processing and customer service. And according to key members who opened up about the organization’s on-going DevOps transformation at IBM InterConnect this week, continuous delivery will increasingly play a role in HM Health’s ability to do that.

Utilizing IBM Urbancode, the business has been able to improve the application delivery process by giving application teams more control over how they stand up new environments, better visibility into builds and deployment , and much speedier configuration management. As a result, not only are they able to deploy more quickly, they’re keeping their developers happier and better engaged, with a higher rate of productivity.

“We save time and a good amount of it,” says Jim Neumyer, senior application developer for HM Health Solutions. “The (app) teams have the ability to look at their builds, configure their builds how they want to do it, strip down the builds with our standard templates and go very quickly from an SCM check-in to deploying in very few steps.”

As he explained, the teams are able to stand up their own environments, they’re no longer hampered by lengthy approval processes. Streamlining of approvals was perhaps the biggest time saver, Christina Felts, team manager of open solutions engineering, explained that one team said they experienced a savings of an hour of deploy time per app—something that can add up on release nights dealing with dozens of applications.

“One of our app teams was able to setup a new environment in less than one day, which previously we couldn’t do. So it could take a couple days before they got everything set up in that new environment,” Felts. “And it also eliminated the need for them to work with my team and put in a request and have to wait up to a couple days turnaround to get configurations done.”

Additionally, Neumyer says they can also easily roll-back when necessary. Perhaps even more impactful, though, has been the visibility piece in overall enablement of developers, Neumyer said.

“So the teams can check everything. If they have an issue, if they don’t know what’s going on in their environment they can go out there and look. It takes me out of the mix,” he said. “They can see it very clearly and easily and try it again. And most of the issues can be resolved on their own. And we can automate things a lot easier.”

HM Health Solutions says this is only the start of their DevOps journey—as things stand, it’s still a waterfall development shop moving toward Agile—but the early successes have garnered more buy-in from across the organization. Neumyer recommended to colleagues in the audience to follow the baby-steps approach that has worked for his organization when it comes to getting started with continuous delivery.

“The biggest takeaway for me is to take small bites,” he says. “Don’t shove the ice cream cone in your mouth all at one time. When you take small bites you’re able to tweak things very quickly.”

February 26, 2015
How Deploy Frequency Impacts Infrastructure Stability
I recently took some time out from reading the Internet to read a very specific piece of research I found on the Internet from IDC – its “DevOps and the Cost of Downtime” to be exact. Sponsored by AppDynamics, this fascinating read included a variety of statistics that are helpful in understanding the impact of (or a lack of) DevOps on organizations given certain financial and competitive impacts.

I won’t bore you by repeating what you can so easily read yourself, but I have pulled out a few key stats that are worthy of further investigation – that of the failure of stability on the business and the predicted increase in deployment frequency.

The average hourly cost of an infrastructure failure is $100,000.

The average hourly cost of a critical application failure is $500,000 – $1M.

The average number of deployments per month is expected to double in 2 years.

Interestingly, that prediction on deployment frequency is a good thing and carries with it the possibility of making sure the costs associated with the other two rarely (if ever) happen.

We’ve already gotten to a point where “speed” with respect to DevOps is effectively equivalent to “time to market”. But there’s another measure of “speed” that’s just as critical, particularly when we apply it to maintaining stability in the infrastructure. That measure is MTTR (Mean Time to Resolution). This measure is very important when you consider that 60% of production failures are caused by human error or lack of automation according to research from ZeroTurnaround. A goal of DevOps, then, should be to reduce that number as much as possible and introduce more stability into the environment. Frequency of deployment can actually be of assistance in realizing that goal.

So how does frequency of deployments help do that?

First let us consider the number of possible points of failure that might cause downtime – both in the application and core network infrastructure. Then let us consider that each of those possible points of failure also have multiple potential causes of failure. A single service in the logical critical data path within the core service infrastructure, for example, can easily cause “downtime” that, in the case of IDC, would be characterized as “infrastructure failure” and run a tab of $100K per hour.

If that application is a critical business application, the tab runs much higher per hour.

Obviously the speed with which we restore stability (service) in such an event is a factor in the overall impact to the business.

This is where DevOps comes in and where operationalization can play a significant role in mitigating the impact of downtime by aiding in faster time to resolution.

Consider that one of the most difficult tasks in the network is not provisioning or configuration, but troubleshooting. To be fair, troubleshooting in app dev, too, is one of the more time-consuming and frustrating tasks, consuming 75% of a developer’s time (according to research from InitialState). That’s close to the 65% of network engineers who are dedicated to “break and fix” tasks, who spend their time troubleshooting network issues up and down and all along the network stack. Needless to say, if you just deployed ten or fifteen different configuration changes splayed across the core and per-app service infrastructure and suddenly something is “wrong”, it’s going to take some time to figure out exactly which configuration change caused the problem in the first place.

The relationship between number of changes and frequency of deployment is no secret; if you’re deploying changes more frequently then it’s likely there are fewer of them. With an average sprint cycle of 3-4 months, the amount of change is logically not going to exceed that of a cycle that takes 12-18 months. The same holds true for the infrastructure; more frequent deployments should mean fewer changes).

Logic (and math) tells us that the fewer lines of code and configuration that have changed, the less time it should take to identify the culprit should something go wrong, thereby hastening the time to resolution. Assuming that an agile, frequent deploy rate results in fewer changes to the environment it stands to reason that this approach should also reduce both the number of failures and the time to resolve any that do crop up. Frequency is a significant factor in reducing failures by increasing its overall stability.

This is backed up anecdotally by Puppet Labs findings in its “State of DevOps” in which organizations adopting a DevOps approach reported 50% fewer failures and a 12x improvement in time to resolution. That’s stability at work and it’s driven in part by the frequency of deployments and its positive impact on MTTR.

So let’s apply this approach (and logic) to the rest of the services along the critical data path in the data center infrastructure, i.e. apply it to application services like caching, web application security, load balancing, identity management and application access control. Automation to reduce misconfiguration caused by human error and smaller, more frequent changes that enable a faster time to identity the inevitable issues that will be introduced. No methodology or approach is going to completely eliminate errors, after all. There’s a limit to how perfect we can get and it’s pretty much a Zeno’s Paradox kind of problem. While we keep solving current issues, more config and code is being deployed that is statistically likely to contain yet another error, and so and so forth.

But the goal is not necessarily to eliminate all errors (though that’d be nice, it’s simply not practical and it’s the primary reason why our goals for uptime are called “5 9s” and not a “1 with two zeros”) it’s to reduce the number of errors and their impact on downtime. And that is something we can achieve without running afoul of Zeno and his paradox.

Smaller, more frequent deployments can result in a more stable environment in which downtime is reduced to a level consistent with acceptable risk.

DevOps, whether applied directly to infrastructure or not, will play a role in improving its stability which in turn has a direct and very real impact on the business by eliminating (some) downtime thanks to more frequent deployments.
February 26, 2015