— ROELBOB

DevOps.com and Automic partnered up to conduct a survey back in February and March of several hundred respondents regarding their use of development tools, their use of Open Source and their rate of change. The survey report entitled: Managing Your DevOps Tool Chest, The Complexity of Herding Kittens is available from Automic here (fyi, registration is required to download the report).
Scott Wilson of Automic and I already conducted a webinar where we took a deep dive on the findings of the survey. You can access the slides and recording of that webinar here (FYI, this is also behind a registration wall).
If you take a moment to study the analysis you will see what I think are some interesting things in this survey. One is that we had two basic groups who answered. One was made up of our DevOps.com readers who tend to be DevOps adopters. The other was a more General IT group. There was some stark differences between the two groups. None was bigger than frequency of release and how often you would like to release code. Talk about polar opposites!
Another interesting filter we applied was by size of organization. The biggest differences when comparing startup smaller organizations to large enterprises was the amount of tools and coding languages they used. Of course larger organizations used more than smaller ones. But the starker difference was in open source usage. Smaller organizations used a much greater percentage of open source tools than larger enterprises.
Finally we also looked at geographical region and industry vertical to draw distinctions. You can read the report and listen to the webinar to to get all of the details on this and more. I highly recommend you download the report and take a look. Also if you can spare some time to listen to the webinar I think you will pick up some great insights.

I spend a considerable amount of time with clients discussing DevOps and the potential value it can bring to them, but there is one theme that always resonates: the ability of a DevOps approach to eliminate risk. While many of us understand the benefits in terms of being faster (improved productivity), cheaper (reduced cost) and better (improved quality), I find that a conversation based around risk, especially when talking to the business or department heads, usually gets straight to their concerns in a language they understand.
IBM’s definition of DevOps is not simply about connecting development and operations; it is fundamentally about providing a capability that: 1) allows lines of business to connect with their customers (which may be internal employees) quickly, and 2) facilitates rapid feedback from their customers so relevant adjustments can be made. DevOps, therefore, embraces the philosophy of iterative (and incremental) development and delivery, whereby solutions are delivered through a series of incremental releases.

By ensuring all aspects of a solution (requirements, designs, code, executables, tests, etc.) evolve over time as each increment is completed, DevOps is fundamentally focused on addressing risks including:
I could go on. There are so many risks that can be addressed simply by delivering working software iteratively, that it’s surprising how many organizations continue to fall back to a “waterfall” mentality—even in high-risk environments where much is unknown.
The tack I’ve taken here originates from my time at Rational Software Corp. (which IBM acquired in 2003), where we would emphasize the need to deliver software iteratively and progress was measured based on the results achieved (i.e., working software) rather that the documents produced. My favorite graphic from that time is the one shown below, which shows how a modern approach changes the risk profile associated with a delivery project, since it focuses on removing risk as early as possible in the life cycle.

In summary, incremental and continuous delivery evident in a DevOps approach allows us to call out risk early and determine its resolution. DevOps often is used along with “lean startup” thinking (and its build-measure-learn cycle) and monikers such as “fail fast” since these are, at their heart, also focused on addressing risk.
Take the DevOps self-assessment to find out where are you on your DevOps journey

Peter Eeles is an Executive IT Architect in the DevOps Global Tiger Team, IBM Hybrid Cloud, where he helps organizations improve their IT delivery capability in order to drive tangible business value. He has a particular interest in the adoption of architecture, agile, DevOps and cloud as innovation enablers. In the IT industry since 1985, Peter comes from a delivery background and was previously chief architect of IBM Rational’s Worldwide Solution Delivery organization, where he led many large-scale transformations in a variety of industries. Peter has written many papers and several books, including “The Process of Software Architecting” (2009), “Building J2EE Applications with the Rational Unified Process” (2002), and “Building Business Objects” (1998). He runs his own website at www.architecting.co.uk.

Making real-time business decisions based on data from social, mobile and cloud platforms is a requirement in today’s world. That’s why an ecosystem of next-generation operational reporting platforms such as Spark, Apache Cassandra, Kafka, Docker, Mesos, Marathon and more have come to life. Enterprises are using these technologies to re-architect their data storage and processing frameworks to achieve lower costs, scalability and faster response times. However, as enterprises onboard business-critical applications on this next-generation technology stack, they need to prepare for failures or errors that may disrupt customer-facing applications.
One of the key benefits that operational reporting platforms provide is real-time visibility, which allows businesses to make strategic decisions based on actionable insights. However, the new applications generate a large amount of data and archaic operational reporting platforms (those based on traditional data warehouses, data marts and ETL logic) are unable to catch up and, as a result:
These issues have driven many progressive, data-centric enterprises to re-architect their operational reporting platforms. In this new architecture, Kafka serves as a persistent message bus that ingests data from relational stores and provides failure resiliency and message buffering. Spark or Spark Streaming may be used for transformation, aggregation and other lightweight stream processing. These processing nodes may be deployed in Docker containers. Finally, Apache Cassandra serves as a distributed data storage layer that provides linear scalability and high-availability. Kafka also enables the ability to add multiple data consumers such as Hadoop/HBase through Flafka.

However, as enterprises adopt operational reporting platforms, they want to ensure that bad data feeds and human error do not result in data loss. Apache Cassandra natively replicates data but that does not provide the capability to go back in time to recover data efficiently and at scale.
As enterprises onboard their critical applications to achieve development agility, scalability and lower operating costs, they must also explore data protection solutions that are specifically dedicated for scale-out databases.

“DevOps is a lifestyle, not a title.” This is a quote from Ron Melanson, of Amazon, and I hope he won’t mind me sharing it here. It struck a chord with me, and this blog explains why.
I have spoken about DevOps on a few occasions. Some audiences are not from a development background, so I explain the “why” of DevOps to them to help set context.
Chances are you are past the “why” stage and into the “how” stage. Even so, it is always useful to think about the “why” part, or even to revisit it—it is important to make sure you really are getting what you want from the DevOps journey.
In my DevOps presentations I vary between the transactional aspects of the “why” and the emotive aspects of the “why.” The transactional aspects are what the tooling can help with, and so that matters to a degree. However, since emotions influence us the most, especially from a lifestyle perspective, it is the emotional aspects that I want to focus on here.
The picture below captures, for me, the essence of why people care about DevOps: they want to be happy at work, and to sleep soundly at night. If you are not where you want to be with respect to those things, then you need to change the way you lead your life.

I highlight the “change the way you lead your life” aspect to illustrate contrasts among the differing approaches to DevOps that I have witnessed.
Now that DevOps has grabbed the popular imagination, it is quite common to see job advertisements for people to be part of a DevOps team.
One ex-colleague of mine, hired to lead a DevOps team, expressed some concern about whether that notion actually made sense. He saw his team off in a corner focused on tooling and processes, i.e. the transactional, while everyone else in the organization seemed to be doing what they always did.
When his team suggested a change to a process, or tried to explain how a tool could be adopted, they often were at a disadvantage. The people in development and operations, whose working practices would be affected, had not actually bought into the whole idea in practice, and/or simply did not have the bandwidth to think about it.
Consequently, with little or no time to pay attention to what the DevOps team were doing, there was no engagement from the people for whose supposed benefit the DevOps team was laboring. Thus, there was no driver for changes in behavior. They were all missing the emotive “why” part.
The challenge my colleague faced is common. Hiring someone to do DevOps does allow you to focus on getting people with the skills and experience for the transactional and technical aspects of DevOps. That is necessary, but it is far from sufficient.
If the benefits are to accrue to the people working in development and operations—and, indeed, the wider company and business they serve—then it is those people who have to own the idea, top to bottom.
Owning the idea means rethinking what you are doing, and why—in other words, changing your lifestyle. At least at work, anyway.

What would we die for in an integrated DevOps tooling solution—something that would make our operations partners stand up and cheer? Intelligent calendar awareness.
The most fundamental construct of any release orchestrations (RO) tooling software is the calendar. RO software, no matter who the vendor is, uses a form of the calendar, to outline what tasks will get performed in what order from beginning to end. Better RO software allows for parallel tasking, but all RO software uses a calendar. So the most fundamental use of a calendar construct is focused on “when” tasks are performed and completed. Adding intelligence to a calendar construct adds three more dimensions or characteristics to this baseline feature.
First is the concept of ownership. Instead of just marking a calendar with the dates of specific tasks, or clusters of tasks, the addition of ownership marks the calendar with “events” that are “owned” by specific teams of people intended to perform groups of actions or tasks. As an example, instead of just marking the calendar that on April 27 this year, with the specific tasks we need to run a mini-release to upgrade our Oracle database from version “x” to version “y.” With the addition of ownership, we could mark this date as running an “event” of an Oracle database platform upgrade. This new event would be “owned” by the database update team.
The block of time the team outlines for the Oracle upgrade “event” would then take on two additional characteristics for teams in general: inclusionary and exclusionary implications. As an example, the time set aside for the tasks to upgrade the Oracle platform becomes inclusionary for the owning team. Members of this team would be allowed to schedule tasks during this block of time, ideally intended to facilitate this goal of upgrading the platform.
Other teams are (by default) excluded from performing actions during this block of time so they don’t impact what the database team is attempting to accomplish. When other teams pull up their respective calendars to schedule some new task or event, the April 27 block of time is greyed out or presented as unavailable because the database team owns an event listed in that block of time. For the database team the time is inclusionary, for all the other teams the block of time is considered exclusionary.
The benefits to you the customer are immediately visible. By adding ownership to the calendar construct, events could be scheduled in such a way that teams do not inadvertently “step on” each other (i.e., attempt to schedule tasks during the same window of time that would adversely affect the other). It also allows you the customer, to view a dashboard that shows the “kinds” of events that are scheduled throughout the week, month or quarter, and who is doing what.
To avoid being too strict with this idea, the RO tooling vendor need only add concept of granting exceptions to requesting teams by the owning team for events that would not impact their efforts. So as an example, if a different team wanted to work on tooling the SQL Server platform during the same block of time that Oracle platform was being updated, and the servers were different anyway, there is no reason why that team could not schedule a similar event of its own at the same time. If ownership is the only concept added in the RO tool, an exception would have to be granted by the Oracle team, but this would be costly in terms of administration. It is for this reason we introduce the second main construct in the intelligence of calendar management.
The second main concept of intelligent calendar management is component level awareness. This dimension of awareness is based on identifying specific “technologies” as resources or components that are part of the environment you operate at your company. It doesn’t have to be a ground-up exercise, and if you prefer one of those, you could use a standard discovery tool to keep it up to date. But the more basic idea is to simply identify the fundamental technology components you use in your world.
So as an example, I could identify that this company uses Oracle databases (perhaps also tracking the version if there are multiples of it), SQL Server databases, the type of servers it uses, the operating systems, the middleware, perhaps a mainframe platform and the network fundamental technologies, etc. It is highly likely that there are groups of people who are uniquely identified in this company that manage these specific types of technologies. This makes the associations easy for event ownership. The Oracle guys become associated with the Oracle technology components but not with the SQL databases, etc.
Having teams associated with the components they manage allows for better parallel event scheduling. But this is only the first level of intelligence. The next level is to track, by application, the underlying technology components they utilize. This process is done during the build and deploy automation anyway, so the effort is nothing more than what is done today. The beauty of having this spider web map, or this map of technology component dependencies, is that it adds a new level of capability to scheduling in our calendar.
At this point, scheduling an Oracle event does not preclude the SQL team from scheduling another one during the same time window, since those two technology types don’t conflict with each other. Both teams would see open calendars on the same day due to this intelligent calendar awareness of component interdependency, or lack thereof. This dimension of intelligence precludes having to grant needless exceptions to owned events. The only requests for exceptions would come from users or teams that actually would impact the upgrade event, and they could be evaluated for legitimate need and potential real impact.
Let’s return to our Oracle platform upgrade event our mythical company scheduled for April 27. With the addition of component-level awareness, when this event is scheduled by the Oracle database team, every application that uses Oracle is “automatically” notified that this time window is now off-limits for its development or usage activities. This limitation could include automatic notification to application development teams that they could not deploy new code during this window. Or the notification could be expanded to notify users of the impacted application(s) that this list would be down during this window of time. Applications that do not use Oracle would remain unaffected and not notified.
Imagine what this kind of automatic notification could do for both development and the user base of the technologies you operate, all coordinated automatically through the RO tooling you use today in DevOps. Then think about the possibilities from here. The Oracle team could look at a block of time again and examine whether scheduling on April 27 would have a high impact on other teams, or not, before it finalizes it. If the impact to others is high, perhaps the team picks a different date. The possibilities become endless.
The third and final architectural dimension to add to calendar intelligence is the concept of scope. Scope is a simple hierarchy that you could define about your company by logical business subdefinition. As an example, you could outline the scope you define as: global (entire company), business unit (listed by name), product unit (listed by name) or departments (listed by name). Just using these kinds of simple scope definitions begins to insert an impact assessment into the event scheduling that takes place.
As an example, events could be scheduled with a global scope, owned by the change management governance body, that excludes a 10-day window because of a corporate audit by outside regulators. Now, when any development team attempts to schedule deployments or releases, for example, their calendars are automatically greyed out during this 10-day window because of the corporate governance event. Exceptions would only be granted by that governance body. In this case, this prevents accidental changes occurring during the middle of an audit.
If the example cited above is too broad, and the mythical company only needs to limit an event to a particular product line or further down to a particular department, the “scope” characteristic allows them to do so. This final dimension to intelligent calendar management does require a higher level of sophistication from the RO tooling vendor and a higher upfront investment in the administrative costs of providing DevOps services. But for an enterprise-class installation, this is an investment that can take the chaos of corporate communication where it comes to the impact of change to a systemic nirvana of communication on demand.
So why outline an architecture that most RO vendors have yet to implement? If it is candy I created that you can’t have yet, why discuss it? Two reasons: First, when you begin to examine the cost of running your DevOps set of services, you will find that, like many other enterprise-class efforts, it is the administration costs that are heavy at the outset and increase over time. The effort it takes to manage “communication” and avoid having interdependent technologies “step on” each other is no easy task, particularly when we have moved from weekly deployment windows to potentially daily deploys to production across large app portfolios or complex legacy hardware architectures.
Second, for those of you who still maintain data centers, consider how much time you “waste” if your Ops teams continue to require a 12-hour to 48-hour window over the weekends to make updates to their own platforms. Going to an external cloud provider is a way out of this, but for organizations that are unable to fully do so, the weekend downtime remains a significant issue. DevOps will create desire in the Dev teams to want to use this traditionally Ops window for their own purposes as well. Once DevOps has been fully embraced, there is as much a need to automate communications and intelligently ensure we do not step on each other as there ever was. DevOps accelerates this need, as it accelerates the speed of innovation itself.
The proper tooling, or rather the tooling that is thinking about intelligent architectures that will greatly reduce your costs and your overhead and ensure consistency in your world—these are the features (and the thinking) you need your tooling vendor to embrace. The alternative is having to pay your labor base to manage it another way. If your leadership and your vendor partners are not considering these issues, they should be. If you need help, feel free to contact me (I am still looking at the moment). The goal of DevOps to automate process should not just be about moving code; it should extend to how we administer, communicate and inject governance as well. Intelligent architectures accomplish these goals, and any vendor could be there first, if they wanted to.
To continue the conversation, feel free to contact me.