devopsy.kinsta.cloud

Tag: devops

The Human Touch: The Key to Optimizing Your Software Value Stream Network
Software has no intrinsic value until it’s working in production and supporting a customer’s business initiatives. With that in mind, it’s imperative that organizations focus on how they reduce the time to value, not just time to deploy. And that’s achieved through people.

A common misconception is that technology underpins a software delivery value stream. This is only half true. Tools add immense value and help people to be more productive—whether through advanced functionality (such as the ability to visualize and manage workflow) or by automating repetitive activities (such as test cases and releasing code). All of this boosts throughput. But it’s actually the human touch that drives a product.

It’s all about how practitioners work together to understand and address a discerning customer’s needs with software. To that end, organizations must ask themselves, “How do we optimize practitioner collaboration?”

It’s All in the Artifacts

Collaboration optimization lies in how organizations handle product-critical information—artifacts such as features, epics, stories and defects—that are created for the very purpose of sharing with collaborators.

Therefore, the follow-up question that organizations should ask is, “What system do we have in place to ensure these artifacts are flowing in real-time, so the information is always up to date, ensuring that all work is always accurate and adding value?”

Fortune 100 companies have already asked these questions, making a number of significant discoveries:
1. The software value stream is actually a social network of collaborators who are connected by artifacts—hence, “value stream network.”
2. This network consists of two different, but equally important, phases: “plan and design” and “build and deploy.”
3. The network by default is not explicitly connected because the tools collaborators use do not naturally integrate with each other and automatically flow artifacts between them.
4. This disconnect forces collaborators to waste precious time chasing up information via manual means such as email, status meetings, spreadsheets, tool-switching and duplicate entry.
5. This manual communication is non-value work that costs organizations up to $10 million in productivity overhead (based on a typical 1,500-person IT department).
6. Such archaic knowledge-sharing practices significantly compromises the fidelity of data as it goes from one practitioner to the next.
7. The “plan and design” phase—the part of the software delivery process that determines what product/feature is to be built/fixed—is detached from the “build and deploy” phase.
Through these discoveries, the last one in particular, organizations came to an important realization: They have no real way of knowing if they’re building the right product because there are six degrees of separation between the people who design the product, and those who build and deploy it.

The result? Disgruntled customers and a big loss of competitive advantage. To address this, they’ve created a new infrastructure that provides a framework to connect and support both phases.

By fusing the value of people and technology into a defined value stream network that is visible and traceable, they have a seamless flow of work that they can see, measure and control. They’re now in a position to identify where value is created and lost, enabling them to make informed business decisions to make better products.

The Software Value Stream Network

The enterprise software delivery process has long been misunderstood as a traditional manufacturing system that is linear and sequential—thanks in part to the industry’s failed adoption of the Waterfall model. The negative influence and impact of that experiment still lingers today.

The process is linear in the sense that there are two clear bookends: A product/feature/fix is requested (A) and then that product/feature/fix is made (B). There are also established phases across the production line: conception, initiation, analysis, design, construction, testing, deployment and maintenance. But that’s where any real similarities end.

In reality, the process is more of a latent network of critical interactions between collaborators. These bilateral working relationships exist at key stages of the software delivery process, such as developer and tester, developer and operations, etc.

When you actually lift up the floorboards of the process and peak inside, it looks more like a social network connected by the flow of artifacts, which looks something like this:

Most of these relationships have developed over time as organizations have introduced a higher degree of specialty in key stages across the value stream. This evolution has been vital in helping organizations to:
- More deeply understand customer requests/issues
- Clearly articulate the requirements/defects
- Enhance product management/optimization
- Acutely break down requirements into epics and stories
- Write more accurate code faster
- Automate and accelerate higher levels of testing
- Continuously deploy
- Introduce tighter feedback loops
- Continuously improve and innovate
While all these new roles, disciplines, purpose-built tools, processes and methodologies have helped different practitioners to perform better, it’s also introduced huge technical complexities. How do you flow those all-important artifacts if the tools are disconnected by design, and when collaborators are working in silos, trapped in “administrivia” hell? It’s tantamount to having a sports car with no gas and a confused driver. Or an airplane flight network with no air traffic control. It’s a banana republic.

Building the Right Product via Value Stream Integration

Agile and DevOps illustrate just how vital integration and automation is to this flow of work. Agile brought in prioritization, feedback loops and sprints to build the most important features as fast and accurately as possible.

DevOps, meanwhile, has accelerated speed of deployment. In that sense, the build and deploy phase is optimized. Automation has work flowing seamlessly between Dev and Ops teams, and eliminated/mitigated cross-team issues, such as miscommunication regarding what products need to be deployed, and when.

But what’s the point of faster delivery if it’s not what the customer wants?

We need to consider how wrong features are developed. If you turn to look upstream, you realize that the teams planning and designing products are affected by the same misalignment that used to plague DevOps teams.

The disconnections between collaborators and their tools in the creative software design process mean that often what’s implemented isn’t right or is delivered too late. In either case, it doesn’t add value.

But that can be radically changed through sophisticated toolchain automation, which shares artifact data across tools and automates the hand-offs between owners in the “plan and design” stage.

From the sales to product to PMO to business analysts, by bringing these customer-centric, product-focused teams together, you dramatically improve the product articulation process and the speed at which its performed.

You build features closer to the customer’s needs. You contain that precious value. And then through automation you can flow that value downstream to developers to create that value. It is as big a revelation as DevOps.

How is this all done? Enter value stream integration. This process is much more than just connecting two tools, say Jira and HPE QC. It’s about connecting all tools, practitioners, disciplines and processes to define and create a free-flowing network that is automated and scalable. Think of it as the plumbing in a house. If even one pipe isn’t connected, there’s no flow of work (value).

Value stream integration connects the pipes to:
- Automate Information Flow Across Value Stream: It enables the frictionless flow of artifacts, as well as information from events across the value stream. This removes non-value-added work and bottlenecks; increases velocity and capacity; enables automated traceability; and dramatically improves employee satisfaction.
- Enable Value Stream Visibility: When managers want to see metrics and dashboards to understand project status, to optimize the process or to ensure compliance, it has been nearly impossible to get a real-time, holistic view across non-integrated tools. Value stream integration unlocks lifecycle data from these application tool silos by automatically compiling lifecycle activity data into a single database. This data can be used to create consolidated, full-lifecycle reports and dashboards, as well as for traceability reporting.
- Create a Modular, Agile Toolchain: Value stream integration enables organizations to use the products that best support each discipline while getting the benefits of a single, integrated toolchain. This drives more value from each tool and allows organizations to easily add, replace and upgrade them, creating a proactive environment for innovation. It creates that value stream network.
Finally … Manageability

With this network, organizations can finally see, manage and optimize one of the most important processes in their business: the engine that drives their prosperity in a digital-centric world. You cannot underestimate how important it is to have this network, this complete system. This is the state where innovation thrives, and where continuous improvement can be executed. And, of course, where you gain the essential visibility to be confident you’re always building the right product.

— Nicole Bryan
April 20, 2018
Why NoSQL is the New Database Darling of DevOps

If developers are from Mars and testers are from Venus, your database administrator is stuck in Saturn. By nature, the database—and specifically the data it carries—has to remain stable even as the application code on top of it is changing faster than Donald Trump’s staff.

When it comes to the speed of your release cycle, issues with your database are like potholes. You don’t see them up ahead, and you only feel the pain after the damage is already done. The least flexible layer in an application is its database, rendering it the weakest link in any agile development process.

Get Your Database Up to Speed

The right NoSQL database offers the following features that can speed up your release cycle without falling into a pothole.

It’s Schema-less: When you make a new build, or issue a new release, your code will change. Obviously, it will go through QA and check out. However, just because it’s working doesn’t mean it’s working correctly. You may have changed the types of data your application is accepting. You may have changed the way your app works with data. Your schema is the blueprint for how you accept data. For a relational database, you need one. The more complicated the schema, the harder it is to change.

A NoSQL document database is schema-less, offering you the ultimate flexibility and speed in how to rearrange the way you handle your application’s data.

Rapid Scalability: The moment your application is a success, expect greater load. Greater load requires more hardware to handle that load while maintaining peak performance. A relational database demands you scale up, investing large amounts in a new server. NoSQL solutions let you scale out, creating data clusters on commodity hardware.

NoSQL lets you add new nodes to your database topography in minutes, expanding your ability to process requests from users at optimum performance.

Immediate Adaptability: New code doesn’t only change the type of data you are working with; it also can change how you work with the data itself. New queries mean whenever a user clicks a button, your database will have to relearn how to sort your data to answer new requests for specific items or aggregated totals.

A relational database will join tables and scour the data row by row to get your users what they want. If the whole thing changes, the process starts from square one. A good NoSQL solution uses dynamic indexing. It means that it won’t perform a query until after it finds the right index. It will either use a pre-existing index, improve on one or create one from scratch that will be made available for future queries.

Such a system reduces the performance hiccup by releasing new code in that the more you use it, the better indexes are created to maximize the performance of the database. Your Agile DevOps team can release new builds every day and the database will be able to keep up.

Platform testing: Agile without testing is like a lunchbox without the apple. It may not be the most appealing part of the meal, but you gotta have it. The right database lets you make queries in test mode without touching the disk itself. The data in real-time is separated from what you are testing.

Due to its inherent flexibility, a NoSQL database lets you re-create whatever test scenarios you need and measure function and performance to see how every layer is working under the new changes. You can also do this fast, keeping your development process moving right until the end.

Technical support that doesn’t hold you back: One of the biggest pitfalls in Agile is when you hand the fate of your next release over to people who have no idea how urgent it is that you get the issue resolved now. Bad tech support can take an issue that shouldn’t slow you down and freeze you up for hours, even days.

The right NoSQL solution offers tech support with quick response time, and engineers who have experience developing the database itself. It is important that they know how the database works, and can do better by you than reciting a series of standard test scripts.

Databases can also notify you of underperformance or any indications that there are escalating problems that need immediate attention before they become full-fledged issues. Small features such as these save you time in having to resort to tech support in the first place.

Conclusion

There are always risks involved with making changes to your application. The risk databases pose is that some changes can shut down your database and paralyze your application. A NoSQL solution can reduce this risk, and even speed up areas of your DevOps process.

— David Ben Horin

April 19, 2018
Reliance on Cloud Foundry PaaS Increases

A survey of 383 IT professionals published April 18 suggests the open source Cloud Foundry platform-as-a-service (PaaS) environment is gaining traction across the enterprise.

The survey conducted by the Cloud Foundry Foundation (CFF) finds one-third of companies now have at least 100 developers using Cloud Foundry. The number of companies with more than 50 developers using Cloud Foundry rose from 20 percent to 47 percent in the last year. More than half the respondents (53 percent) report they have been using Cloud Foundry for less than two years.

Companies with more than 50 applications on Cloud Foundry rose from 24 percent to 40 percent in just over six months and nearly a third (30 percent) of respondents describe their Cloud Foundry usage as broad.

More than a third of Cloud Foundry users report they now save a few months or more per application development cycle, and 10 percent report saving more than six months. In addition, nearly a quarter report saving $500,000 per application development cycle, with 17 percent saying they have saved more than $1 million.

The survey also finds there has been a sharp rise in interest in containers. A total of 13 percent of respondents already are using both the Cloud Foundry Application Runtime (CFAR) and the Cloud Foundry Container Runtime (CFCR), which is based on an instance of Kubernetes. Another 45 percent are evaluating CFCR.

CFF CTO Chip Childers said he expects the Cloud Foundry PaaS and Kubernetes to be employed side by side for years to come. The CFF will continue to develop technologies that unify the management of Cloud Foundry and Kubernetes before there is an inevitable convergence of the two platforms, he said, adding that for now, a PaaS and container-as-a-service (CaaS) environment are serving different classes of applications. In the case of Cloud Foundry, that historically has been complex 12-factor applications. But Childers noted that the types of applications being deployed on Cloud Foundry is also expanding.

Overall, Cloud Foundry reports more than 3,200 contributors participating in the community, with more than 60,000 commits in the last 12 months alone. According to Gartner, the total market value Cloud Foundry is approximately $3.1 billion.

As application development continues to evolve, it is clear many organizations are embracing multiple platforms. Different business units within organizations want to be able to develop applications in the cloud or on-premises as needed. Ninety percent of ten Cloud Foundry users surveyed cited cross-platform flexibility as important capability, with 67 percent calling it “very important.” One of the core value propositions of Cloud Foundry is that it provides a highly portable platform backed by a consortium that includes IBM, Pivotal, SAP, SUSE and, now, Alibaba.

In general, DevOps teams are embracing PaaS and CaaS environments to take advantage of higher levels of abstraction that don’t require them to manage the underlying IT infrastructure. The theory is that the more automated the environment becomes, the more resources there are to allocate to developing applications versus managing the platform.

The trade-off is that each PaaS and CaaS environment is usually highly opinionated in terms of how the application development process should be managed. Because of that issue, many organizations still prefer to build their own platforms from the ground up. But as PaaS and CaaS platforms continue to mature, it’s likely many more organizations may decide there isn’t enough differentiated value to be gained by building a platform themselves.

— Mike Vizard

April 18, 2018
Time to Rethink Infrastructure Operations in Enterprises

“Business is technology and technology is business” seems to be the new mantra as the rules of corporate strategy are being rewritten across industries. According to IDC’s survey in 2017, “Two-thirds of global CEOs will have digital transformation as part of their corporate strategy.” That’s CEOs—not CIOs—who will have digital transformation as part of their corporate and, not IT, strategy.

While digital transformation is not a new phenomenon, making digitization core to the corporate strategy is a fundamental shift for technology teams. Delivery units, or “change the enterprise” teams, have warmed up to new ways of working over the last few years and are already on the learning journey. Most enterprises today have digital or agile transformation initiatives that tend to focus on the “change” aspect significantly. However, the less glamorous “run the enterprise” dimension tends to be forgotten or side-lined in these initiatives.

The rise of DevOps was meant to bridge the gap between “change” and “run.” However, many DevOps transformation initiatives seem to be taking the same old path of least resistance that agile transformations have historically adopted: by focusing on the “change” and delivery teams.

As a consequence, the infrastructure and operations teams—or, “run the enterprise” units—are now waking up to the new reality and have far less time to adapt than their “change” counterparts. The big question really is, What does it mean to infrastructure and operations teams when the CEO is now embracing digital transformation?

This means that the technology infrastructure required to support business strategy requires a combination of the usual consistency and quality that infrastructure and operations have historically focused on—but only with greater speed, innovation, flexibility and customer intimacy. For the first time, business is looking at infrastructure as equal partners on this journey, not as a subservient IT function.

What Caused This Shift in Thinking?

Since 2001, there have been four game-changers in technology that have caused this shift in thinking: the Agile Manifesto of 2001, adoption of Cloud by enterprises, rise of startups and the DevOps movement. The world of technology as we know has been going through a tectonic shift as a consequence of these overlapping waves of change.

Agile and DevOps

Version One’s 11th “State of Agile” report highlights that 71 percent of organizations have current or planned DevOps initiatives in 2018. While Agile ways focus on effective team structures, value focused mindset and greater business alignment, DevOps has taken the principles to the logical next steps of delivering value by focusing on flow, amplified feedback loops and building a culture of learning and innovation. In addition, full stack automation covering infrastructure, application delivery and release orchestration has really catapulted DevOps into the top initiatives in organizations. In terms of architecture, new constructs such as evolving design, microservices, containers and serverless computing are all redefining technology infrastructure. The rise of these new ways of working is only piling immense pressure on traditional infrastructure organizations to reform and rethink every aspect of their operating model and challenge processes built on a foundation of perceived stability.

The Startup Way

Enterprises increasingly are being influenced by the ways of startups and how they are driving rapid innovation. There is a cross-fertilization of people moving to enterprises who are challenging the thinking in these enterprises, especially infrastructure area. Teams are now embracing once considered absolutely alien, if not impossible, concepts such as chaos engineering and anti-fragility.

Cloud

The recent RightScale “State of Cloud” report reveals that 97 percent of enterprises are using cloud in 2017. However, only a small share of their workload has actually been moved to cloud, indicating the potential for cloud in the enterprise context. This indicates that enterprises are past the quick wins such as digital front-ends and will now struggle with the difficult areas of their applications—the bulk of the enterprise apps where the traditional infrastructure constraints and mindset rules. Enterprises can tackle this by truly embracing change, thinking big but starting small with experiments and continuous refinement of the every single dimension of the way infrastructure teams operate.

“Nothing is permanent but change.” The wise and timeless words of Confucius ring true!

— Aditya Vadaganadam

April 18, 2018
NoSQL Databases: Lessons Learned Growing a Globally Focused Startup

Startups deploying NoSQL databases and global citizens learning new languages actually share something in common: Both want to remove the limitations on how they access information and express themselves. In launching and growing a platform that enables anyone around the world to connect and speak with native English-speaking tutors on-demand, we’ve had the fortune of helping countless people learn new language skills—and we’ve also learned a great deal ourselves about the best ways to handle the requisite user and application data.

NoSQL, No Problem

Here are four NoSQL database strategy takeaways we learned from recent experience:

Keep Your Data Options Open with Flexible Technology

When a startup is just out of the gate, it isn’t immediately clear what shape the final product may take, or what data it will require.

In our case, the relatively broad scope of our project also defined the need for a database able to function globally with 24/7 availability. The concept for Cambly was born out of its founders’ real experiences traveling to other countries and learning foreign languages through conversations with native speakers. Immersive experiences such as these are critical for gaining and retaining conversational language abilities, and provide learning opportunities that no language course or software can really match. However, these language skills can recede rapidly if a person no longer has access to native speakers to practice with. Seeing this as an issue that technology could address, we began building an application to provide global on-demand access to immersive conversations with native speakers, initially concentrating on English as the largest language market.

As a startup beginning to explore how exactly we’d accomplish the task ahead of us, it was critical that we could store whatever types of data we found to be of value as we quickly iterated the application. This meant utilizing a NoSQL database solution to free ourselves from any schema constraints. We selected MongoDB as a flexible, NoSQL solution with which our technical team had some existing familiarity.

Get Help …

Startup success is dependent on managing finite internal resources in a way that maximizes the attention that can be devoted to product development and other growth-critical initiatives. This means having the wisdom to rely on managed services where they’re beneficial in saving time and simplifying operations, versus building up internal headcount. For example, we enlisted Heroku to use as our application platform. And, even though our team was experienced in running databases and managing machines on AWS, we still saw the value in keeping production as hands-off as possible and opted for a database-as-a-service (DBaaS) strategy as well.

… And Make Sure It’s the Right Help

It can be tricky, but ensuring that any and all technology service providers a startup works with are a strong fit for its needs is vital to moving quickly. We actually learned this the hard way when it came to our original MongoDB DBaaS. For us, reliable 24/7 availability and insights into database performance were essential requirements. Unfortunately, our initial provider presented us with two challenges to meeting these requirements in a short time frame: Its tools became unable to provide us with visibility into our database, and it announced a required platform migration that could not be completed without downtime. Naturally, these changes led us to seek a better-fitting option. Soon after, we partnered with mLab in large part due to rolling node replacement capabilities for migrations and 24/7 uptime, as well as support and tooling that could help our team quickly optimize MongoDB instances.

Get the Database Tools You Need to Succeed

From reliable data backups to performance insights and beyond, the right toolset can make or break a burgeoning company’s success. Just as language skills must be constantly attended to, your database requires intelligent upkeep and backups to ensure it delivers a satisfactory user experience and remains available in case of disaster. This is one lesson which—very fortunately—we were able to absorb without hardship. We had an incident in which our team accidentally deleted a trove of valuable data. But, with data backups in place, our DBaaS was able to help us restore that data quickly—and without us stressing over it as we would have otherwise. At the same time, the DBaaS tools allow us to recognize and address database issues while improving indexing and performance.

— Gar Higgins

April 17, 2018
A Unified Route to Bring Mainframes to DevOps Speed

Most businesses have mainframes where their key applications and data reside. In today’s digital environment, customer-facing mobile and web systems retrieve critical data from back-end mainframe systems. It is critical to adapt mainframe applications and data to today’s rapidly changing market demands if businesses want to remain competitive.

This is the question IT and businesses are facing today. How can they integrate their well-established mainframe environments and have a unified cross-platform across the enterprise for DevOps to manage releases across mainframe and non-mainframe applications?

An Innovative Digital Solution

The answer lies in mainframe digital DevOps. This helps enterprises remain competitive in an app-centric economy. Through a unified cross platform, DevOps is integrating mainframe tools with open source DevOps tools.

This approach allows organizations to retain their legacy code and does not require legacy code migration—especially when the back-end core applications are on mainframe. As veteran COBOL programmers retire out of the skill orbit, this becomes even more important. There is an imperative and an expectation for agility and higher productivity from IT—without jeopardizing performance, customer experience, security and other compliance factors.

Transforming the Developer Environment

Most mainframe development, test and ops work is executed in antique “green screen” SPF environments, which calls for specialist knowledge that limit productivity among programmers of Agile and DevOps transformation. Turnaround times of monolithic mainframe applications are painfully slow compared with the new-age agile technology of mobile and web applications. This can be frustrating for developers who wish to make frequent changes to mainframe code to fulfill the urgent needs of business.

Therefore, it’s important to make the shift to modern tools and integrate mainframe with non-mainframe DevOps processes to deliver cross-platform services. This will improve staff productivity across development, testing and production operations and enable inexperienced mainframe professionals to perform application development and enhancements with ease and efficiency.

Large mainframe clients embark on transformation by modernizing the workspace to work on both mainframe and non-mainframe applications. Open system developers are familiar with new-age IDEs such as Eclipse, which have user-friendly interfaces. Similarly, other mainframe tools are available that provide source code editing, debugging, fault diagnosis and browse data to enable developers to write, compile and run code from a modern Eclipse-based IDE. Developers can drag and drop files between LPARs and perform other tasks to improve productivity metrics. Collaboration between the two teams will motivate mainframe developers to use DevOps principles to integrate mainframe with distributed and cloud applications.

Once quick wins are established, adoption levels of Agile and DevOps will improve among mainframe developers and business.

Eliminating Barriers to Unit Testing

Historically, mainframe developers have not been accustomed to unit testing. Automated unit testing is common in Java but not in the mainframe environment. Agile coaches, familiar with mainframe tools, will need to enable collaboration in interactively testing smaller pieces of code.

Unit testing can improve throughput by increasing the frequent code drops in production environment and help to accomplish DevOps key performance indicators (KPIs) with fewer errors in the later stages of the software development lifecycle process. Some tools can facilitate this process by automatically breaking the mainframe code into discrete logical units, and creating tests for such logical units.

Tracking and Monitoring Progress for High Quality

Visualization is an important element of DevOps for improved visibility to track and monitor progress across the enterprise. As most mainframe applications and data structures are large, complex and not well-documented, enterprise IT is highly dependent on the personal and institutional knowledge of senior mainframe staff. Graphical visualization can help quickly “read” existing application logic, program interdependencies, data structures and data relationships. Business rule extraction tools are available in the market to reveal logic and data relationships through graphically intuitive diagrams. This helps developers work independently on unfamiliar programs to better estimate project work, meet delivery deadlines and commit timelines to business.

Most mainframe applications are often associated with core business processes with zero tolerance for error. Any attempt with inadequate mainframe experience can lead to catastrophic non-recoverable error. Continuous integration (CI) is critical in determining application quality issues early and establishing KPIs to deliver continuous quality code in production. Enterprises have started embracing modern tools and techniques to transform from the Waterfall delivery model to Agile. Modern DevOps tools embrace CI, and certain tools can quickly detect application quality issues (such as unbalanced or unmatched working storage and data items, and out-of-range nested conditions and branches).

Training for Effective Enablement

Successful implementation of cross-platform DevOps calls for the right training on Agile tools, processes and methodologies. A single Scrum team can be formed to collaborate with developers across mobile, web and mainframe. The Agile mainframe team will need to focus on stories and epics that capture specific units of business value instead of technical tasks in a project plan. Open system tools such as JIRA and Confluence that can support Scrum and Kanban can be used to manage tasks and improve collaboration through centralized and well-organized web tools. This will incorporate Agile values such as transparency, knowledge-sharing and ideation to improve the developer’s performance.

The move to Agile scrumming represents a transformative shift of mainframe work culture, and herein lies the importance of the right coaching in Agile processes. A “2- n a box” partnering model of mainframe developers and experienced Agile developers from other platforms can create the required collaboration.

Operational Data Analysis to Close the ‘Last Mile’ Loop

Good quality code alone is not enough. Operational data analysis is required to reduce costs by discovering and mitigating avoidable CPU consumption caused by code inefficiency.

Mere monitoring and reporting by triage teams is not enough to support and maintain applications in the world of DevOps. Production data analyses by enabling an auto-feedback loop is imperative to improve productivity of the dev, test and production teams. Tools are available that can quickly pinpoint bad statements and pinpoint application inefficiencies such as bad SQL statements. Tools also help identify the causes of excessive CPU consumption and slow data retrieval.

Automation for Speedy Efficiency

Agile-enabled core source code management (SCM) is a fundamental requirement of DevOps. Traditional SCM cannot support parallel development work on different user stories. SCM migration is therefore essential to increase the speed and frequency of mainframe code drops. Automating the change management process and defining the proper workflows can eliminate manual steps and reduce code approval delays.

Again, tools are available that can fulfill business expectations by seamless SCM migration without disrupting the current work in progress.

Automated release and deployment is pivotal to keeping pace with today’s fast-moving markets. Business needs to quickly see reliable new code in production.

Toward a Unified IT Service Management

Cross-platform continuous delivery is vital for efficient use of mainframe applications and their data for multiplatform customer-facing applications, including mobile, web and cloud components. Mainframe DevOps process should fully synchronize the code delivery process across all platforms. This will establish a centralized deployment control center to provide a unified cross-platform and accurately report progress. Tools can be the single point of control for all changes across z/OS, Windows, Unix and other platforms. REST APIs are flexible to mix and match best-in-class tools and avoid vendor lock-in.

The ultimate destination is to create a de-siloed environment where mainframe is just another platform with high scalability, reliability, performance, security and cost-efficiency. A unified mainframe digital DevOps platform may be the answer to modifying traditional process that fulfills this business demand.

— Ramesh Ganapathy

April 16, 2018
Devops Chat: Interop China Features DevOps and DevSecOps Tracks

Interop China has been growing steadily since its inception in 2013. It is part of the Cloud Connect conference in Shanghai. This year the conference will feature both DevOps and DevSecOps tracks and talks. Jayne Groll of the DevOps Institute is the DevOps track chair and Alan Shimel of DevOps.com and Security Boulevard is speaking at DevSecOps.com. We sat down with Molly Huang and Kelly Yao of UBM China to discuss this year’s event.

Molly and Kelly give us a little of the history of Interop & Cloud Connect China, as well as a behind-the-scenes look at what goes into selecting tracks and putting on a major event such as this. It is also a great look into the tech scene in China.

As usual, the streaming audio of our conversation is below followed by the transcript of our conversation.

Audio

Transcript

Alan Shimel: Hey, everyone. This is Alan Shimel.

Jayne Groll: And this is Jayne Groll.

Shimel: And you’re listening to a combined podcast tonight of DevOps Chat, along with Continuous Chit Chat, which is, of course, the podcast from The DevOps Institute and the Community of Continuous Learning.

Jayne, welcome!

Groll: Hi, Alan, how are you? I don’t think we’ve done a podcast together in a little bit, huh?

Shimel: Well, we—[Laughter] that’s what happens when you travel around the world.

Groll: [Laughter]

Shimel: And I’m traveling as well, but we have certainly been on opposite ends of the world, and that’s actually a great lead-in to tonight’s show, Jayne, where we talk about traveling around the world, because we’re happy and lucky to have two of our colleagues coming to us from China today. We have Kelly Yao, who’s a Project Manager at UBM in China, and Molly Wong, who works with UBM for Interop China, and it was a Cloud event, wasn’t it?

Kelly, Molly—welcome to our podcast.

Molly Wong: Hi! Hello, Jayne. Hello, Alan.

Kelly Yao: Hi, Jayne and Alan.

Shimel: Thank you.

Wong: It’s really nice to help out.

Shimel: Ladies, thank you. Molly, besides Interop China, what was the Cloud event you mentioned?

Wong: Okay. So, the whole name of our event is Interop and Cloud Connect China.

Shimel: Ah, I got it.

Groll: So, it’s one event?

Shimel: Yes, so—

Groll: It’s one single event.

Shimel: Got it.

Wong: Yeah, it’s one single event.

Shimel: Interop and Cloud Event China. Very good. Now, Molly, if you don’t mind me asking you, many people in our audience, I’m sure, are familiar with the Interop name. It’s been one of the top IT shows here in the United States for 20 plus years. But they may not be familiar that China has an Interop event. How long has Interop China been held? How many years?

Wong: So, as we all know, Interop is an international event focused on information technologies and the ecosystems of the cloud community. But when it was first introduced to China in 2013, however, only the deliberation of cloud computing was capped, because at that time, Chinese companies were just at the same point switching from hardware to cloud services for cost reduction.

So, that’s why, initially, the Interop China event is not called Interop, it was called Cloud Connect China.

Shimel: Ah, interesting!

Groll: And I was gonna ask, and it kind of makes some sense, if you think of Interop as an international event, which is really about operations, and Cloud Connect certainly is about operations. So, it sounds like, Molly, that the approach here was really to produce an event that spoke to not only cloud from an operational perspective, but also just IT operations in general. Is that correct?

Wong: Yes, because, you know, UBM, by its nature, is an event organizer. So, we have to keep up with the global trends and add different burning topics into the conference every year so as to make the event more compelling.

Shimel: Absolutely.

Wong: So, this year—and Alan, if you don’t mind, let me ask a little bit about DevOps. Because, as you know, I’m gonna be the track chair for the DevOps track. So, tell us why you decided to add a DevOps track to Interop, and even including DevSecOps, which is a security—obviously, DevOps and Security.

So, what are you hoping to come out of the DevOps track?

Wong: You are asking me, or Kelly?

Groll: Either one. We haven’t heard from Kelly, so maybe Kelly wants to answer.

Wong: Okay, I will answer that for Kelly, because I’m mainly responsible for the content.

Groll: Okay.

Wong: So, I think nowadays, the technological benefits of DevOps is very obvious, and organizations that successfully implement DevOps, implement DevOps tools and practices in general, they tend to be ahead of their competitors across key metrics like profitability and growth. So, that’s why the Chinese DevOps communities are growing so fast, as a lot of professionals are flocking into the DevOps ________ and to learn and adapt.

So, from our part, we think we would love to contribute to build up a platform for these lovers and learners to communicate, to collaborate, and to excel.

Shimel: Excellent. Jayne—so, Molly, if you don’t mind, I’d like to hear from Kelly a little bit. Kelly, in your role of Program Manager, how are you sourcing content? Where is the content for the DevOps and the DevSecOps and so forth? Is it internal to the Chinese community? Are you bringing in also people from around the international DevOps community? You know, I imagine it must be quite a job putting this all together.

Kelly Yao: Yes, yes. Exactly. Like, for us, it’s just a platform. We are an event organizer. So, we are not professionals who are familiar with this content. So, what we do is, we will have a—what is it? Like—

Wong: An advisory board.

Kelly Yao: – yeah, advisory board. So, we will have the board meeting like twice a year to put up all the content. So, this year, we decided to put this DevOps tracks, and so, it will benefit our event, it will benefit the whole content for this event. So, that’s when, but you recommended it to us—you and Jayne recommended it to us by Interop ITX, because it’s like—yeah. We’re like partners, so both Interop ITX and Interop China, we’re working together to deliver the content.

And so, when we decided to make the DevOps tracks and then that we reached out to you at DevOps.com and also for Jayne—

Wong: Yes, for Jayne at the DevOps Institute.

Kelly Yao: – yeah, DevOps Institute. And, of course, you are the expertise that—you are the experts, that you will know how to make the content right and how to recruit speakers for, the most suitable speakers for this content, for this topic.

Shimel: Understood. Fantastic. And ladies, how has—you know, from what we understand, and of course, we’re on the other side of the world, but it’s a small world in today’s Internet, right? We’re all connected.

Wong: Uh huh, yeah.

Shimel: We know that the DevOps community in China is, as you said, Molly, is growing very fast. It’s already very large, and I’m wondering, what kind of have you gotten from the community about adding the DevOps tracks? Has it been a positive reaction?

Wong: I think everybody here are fully aware of the benefits of DevOps. But, you know, the challenges of actual adoption remain huge. Because, you know, for the DevOps design for the architecture, it’s all about the tradeoffs. The development of DevOps in China now stands between the early and middle age, because, for the DevOps professionals, they need to consider what they deem as the most important thing and what are they willing to sacrifice or give up for and, really, how to lower the costs when running in a public cloud.

So, I think there are a lot of concerns. There is a need to figure out the full implementation.

Shimel: Understood. And, you know, Molly, just—Interop ITX in Las Vegas is a large show, thousands and thousands of people. How big is Interop China? How many attendees do you expect?

Wong: I think the estimated number would be around 3,000.

Shimel: That’s a big, that’s a—

Wong: Yeah. Each year, like—

Groll: It’s a nice sized continent, you know?

Wong: – around 3,000 people are coming to—yes, 3,000.

Shimel: Very interesting. And, you know, DevOps is great, but give our listeners a sense—what are some of the other tracks about, besides DevOps?

Wong: Besides DevOps, this year, we also have AI—

Kelly Yao: Artificial intelligence—

Wong: – big data—

Kelly Yao: – big data, and data mining.

Wong: Yeah.

Kelly Yao: Yeah, as well as information security.

Wong: Information security, yes.

Shimel: Mm-hmm. So, I have to tell you ladies that listening to both of you talk about the challenges of DevOps and what other areas are of interest to the crowd, you know, to the community in China, it really is a very small world. Because these are the same issues, the same topics that we see here in the United States, it’s the same thing that we see in London and in the rest of Europe. Jayne, you can talk, you were just at a few conferences in Europe.

There are some things that just seem to be universal in today’s technology scene, and everyone is excited about artificial intelligence and machine learning. Cyber security information or InfoSec is top of mind; every day, we seem to hear of more breaches and hacking. And, of course, DevOps is very exciting.

Jayne, as Molly said, you are the DevOps chair for Interop China. I mean, that’s very exciting. Give us a little idea of what you’ve done in your role there.

Groll: So, I’m also the DevOps Track Chair for Interop ITX U.S. So, it’s been very fascinating working with both teams, because, you know, Alan, as you said, the challenges, the successes, the emerging practices in DevOps in particular are very, very universal. And some are starting to emerge differently. Some are, for example, to start, reliability engineering in the U.S. is starting to gain some momentum. I’m not sure, specifically, around the world, but I do know in Asia-PAC ________, there’s conferences like that as well. So, that’s an emerging practice.

Looking at continuous delivery and how it relates to operations is a theme that’s carrying through from both conferences. So, I’m particularly excited about really visiting with the Chinese community, as you know, DevOps Institute is a very, very global organization, and really having some face to face time, bringing content, bringing learning to Interop China and the Cloud Connect.

The other thing we’re gonna do at this event is, we’re gonna do some DevOps training. So, working with one of our Chinese partners, we’re going to do a half-day Essential DevOps workshop. So, for those that are not even so much, “What is DevOps?” but really, a visit to, “What are these practices?” like continuous integration, continuous delivery. I mentioned site reliability engineering. So, in addition to a traditional conference program, we’re going to also bring a half-day workshop to Interop China and Cloud Connect. So, I’m really very, very excited about that in particular. And we will have one of our Chinese colleagues, one of our Chinese partners deliver it so that it can be localized, and he’ll bring not only being able to deliver it in a native Chinese language, but also bring some very, very good local examples and case studies to the training as well.

Shimel: Excellent, excellent. You know what? We didn’t really get the details that, Kelly and Molly, if you can share with us, what are the dates for Interop China?

Wong: Yeah, it will kick off on May the 9th to 11th this year.

Shimel: Okay, and it’s in Shanghai.

Groll: In Shanghai—in Shanghai.

Shimel: Yes, it is, it’s in Shanghai.

Wong: Yeah, in Shanghai.

Shimel: And for people who may be listening to our podcast who are in the area or maybe would like to attend, where can they go on the web to get information about that?

Wong: Yeah, sure. They can register online through our official web portal, cloudconnectevent.com.

Shimel: Cloudconnectevent.com—excellent.

Wong: Yes.

Shimel: And that is for Interop China. So, this—if I’m understanding, this is the fifth year for the event.

Wong: Actually, it’s the sixth year, yeah.

Shimel: Sixth year—yes. So, Molly and Kelly, how many years have you been involved with the event?

Wong: Yeah, actually, we two are—we’re jumping into this platform, this event from last year, 2017.

Shimel: And so, from ’17 to ’18, do you see a big change?

Wong: Yes. I think the market is changing itself. Like, most of our exhibitors, sponsors, most of the cloud enterprises, they’re changing themselves and also their knowledge, their application on the technology. So, it’s changing themselves, and also ________, it’s like, we’re also changing our content to adapt to the market.

Shimel: Like any good event organizer should, correct? You’ve got to adapt to what the market is there.

Wong: Yes.

Shimel: Well, I have to tell you that I am personally very excited to be coming to Shanghai and delivering a presentation on DevSecOps and why security, it’s vital that security become part of the DevOps equation, we’ll talk more about it. And to be there with Jayne and Jayne will be delivering as well, as well as some DevOps Institute training. It promises to be a great event for the Chinese DevOps community as well.

Molly and Kelly, I want to thank you for joining us. We’re just about out of time, unfortunately. But—

Wong: Okay.

Shimel: – this was great. We’re looking forward to coming out to Shanghai in May, and we’re hoping that this will be a tremendous event, and we hope to see you there soon. Thanks very much for joining us tonight.

Groll: Really enjoyed it.

Wong: Thank you, Alan and Jayne.

Shimel: Thank you.

Groll: And maybe we could do a podcast from the event.

Shimel: That would be great!

Groll: So, let’s—you know, [Cross talk] maybe we can do a live podcast from the event.

Wong: Yeah, yeah. We’re really excited to see those in the event.

Shimel: Fantastic.

Wong: And I’m also looking forward to—yeah, looking forward to the podcast if we can have, on site, the Interop China.

Shimel: That would be fantastic. Jayne, I’m gonna give you the last word tonight.

Groll: [Laughter] Well, thank you. Molly, Kelly—it’s a pleasure to work with you. For those that are listening, if you do have the ability to attend, it really does promise to be not only a fantastic event, but an event where there’s lots of opportunities to learn and really track some of the emerging trends not only in DevOps, but particularly on the operational space. So, if you’re going to be in Shanghai, we all look forward to meeting with you. Molly, Kelly—see you in a few weeks. Alan, see you in a few weeks. And for everyone, thanks for listening.

Shimel: You’ve just listened to a Continuous Chit Chat and DevOps Chat. Have a great day, everyone! Bye bye.

— Alan Shimel

April 16, 2018
IBM Think 2018: DevOps Integrated Everywhere

With the onset of conference season I have been running a bit ragged. But something I noticed at the recent IBM Think 2018 Conference really needs to be called to attention: DevOps was still a prominent theme at IBM Think. But instead of having a dedicated DevOps theater or DevOps section of the show floor, DevOps was integrated into everything—all the various flavors of cloud (hybrid, Cloud Private, public), development, apps, even cognitive and AI … all were integrated with DevOps. I, for one, think this is as it should be.

It is a sign of DevOps maturity, I think, that instead of DevOps being called out as some special beast—a unicorn, even—it is more just the way things get done. I had a chance to talk to many different people on the show floor, attend several track sessions (which were also done on the show floor—much better than having to go into separate rooms) and, as usual, interviewed more than a few people for our DevOpsTV YouTube channel. While all of them were excited by various announcements and exhibits, it seemed as though DevOps was part of all of them.

DevOps integrated everywhere

Two great examples of this are my interview with DevOps Playbook author Sanjeev Sharma:

and my interview with Mark Howell and James McLeod of Lloyds

Another really interesting discussion was how DevOps plays into IBM’s Cloud Private offering. Cloud Private is a Kubernetes-based architecture. We spoke with Andy Gower, product marketing manager for Cloud Private. As you would expect with a Kubernetes-based offering, DevOps is right in the middle of things.

This was the theme time and time again at IBM Think 2018. DevOps was not concentrated in any one area; it was everywhere. Even in the quantum computing area, which I will admit was cool as can be.

Mainframes also were all about DevOps. This is not new for the IBM Enterprise Systems team. They have really led the way with DevOps and mainframes.

This was the first year for IBM Think 2018. It represents the merging of several different IBM conferences into one large one. I enjoyed the format and look forward to future IBM Think events.

Also a note on keynotes: The tone for IBM Think was set by IBM CEO Ginny Rometty. It was plain that IBM as a whole is focused on some key technologies and initiatives. Among these were cognitive and AI, cloud and disruptive technologies on the near horizon such as quantum computing.

There was a lot going on at IBM Think 2018, check out the YouTube channel to get more of our coverage.

— Alan Shimel

April 13, 2018
Of DevOps, Databases and DBAs
If you’re on a DevOps team looking for additional efficiencies, you are likely to find them in the realm of database management. The arrival of DevOps did nothing to diminish the responsibilities traditionally associated with database administrators (DBAs). Rather, some of those duties have been redistributed across the team—sometimes explicitly, but more often implicitly. For example, site reliability engineers (SREs) and even full-stack engineers have taken on many of the responsibilities that once belonged to DBAs. And in these roles, DBAs often hold the keys to unlocking a new dimension of efficiency through consistency, repeatability and operational efficiency.

Enter the “DevOps DBA.” This confluence of roles makes a lot of sense. As with application DevOps, it’s logical to assume that the person who architected the platform and wrote the database code also should have a role in deploying and operating it. DBAs manage every aspect of data infrastructure, from capacity planning, installation, configuration, database design and migration to operational duties such as performance monitoring, security and troubleshooting, as well as backup and data recovery. As a result, the DBA’s responsibility for maintaining existing workloads sometimes conflicts with the need to plan and provision workloads for the next generation of applications. Reconciling these responsibilities can speed up the software development life cycle as a whole.

On the operational side, DBAs are responsible for monitoring database behavior under peak application workloads, ensuring they’re performing according to expectations, and working with Operations to resolve bottlenecks and outages. They manage data access and the overall security of the platform. They also perform release activities in support of the application and troubleshoot any errors that happen during that process or during day-to-day operation.

With all this in mind, we can propose a DevOps DBA checklist:
- Establish a microservices plan with regard to data services.
- Set up integration environments, and provision databases for new projects.
- Dynamically extract, transform and load data from one silo to others, employing TTL strategies to reduce the data or enrich some of it.
- Continuously monitor and optimize performance: Strive for low and consistent tail latency and a joint view of the app and the database.
- Expert zone: Automate scale out and automate schema changes.
- Backup and recovery: Backup to cold storage and restore.
- Plan for downtime and high availability using multiple data center disaster recovery capabilities.
Characteristics of a DevOps-friendly Database

We’ve looked at some of the DBA responsibilities as they apply to DevOps. Now we’ll discuss some of the technical aspects of database technologies that can help, or hinder, this hybrid role. To keep up with applications as they migrate through lifecycle stages, there are a number of organizational and team cultural challenges to be addressed. It’s also important to remember that not all databases are created equal—some databases lend themselves more readily to the DevOps toolchain than others.

It takes a combination of elements for a database to be DevOps-friendly. The ultimate goal is to minimize the fear of change by mitigating risk. Relational databases do not conform particularly well to modern software development practices, since changes to schemas can be time-consuming. NoSQL databases, in contrast, have proved to be more life cycle-friendly because they don’t impose rigid schema management during application development. Over time, however, this free-wheeling approach to data modeling can create headaches. The ideal database strikes a balance between strict schema enforcement and decentralized data-model changes.

A key consideration for a DevOps-friendly database is operational simplicity. Traditional SQL databases, such as Postgres and MySQL, are well-understood and have lots of support, but can be difficult and expensive to scale. The rise of NoSQL databases enabled larger datasets to be managed in a more flexible and scalable manner than relational databases. But even among NoSQL databases, there are some significant differences. The first generation of NoSQL databases—in particular the widely adopted Apache Cassandra—can impose significant operational burdens in their own way.

Additionally, modern cloud-native databases have their limits. They can be opaque to operators, and have been known to scale up and down unpredictably. Operators often lack visibility into the distribution of datasets. In many cases, cloud databases also prove to grow at a rate that’s disproportional to the workload supported by the database. Database instances can proliferate to the point where DevOps has to deal with “node sprawl.” You can think of this problem as the false promise of elastic scale, wherein the solution to increased traffic is, “Just add more nodes!” Lots of small nodes are not necessarily better than a few large ones. (For more on this, I’d encourage you to watch this talk on mythbusting the advantages of small nodes.) In any case, the biggest overall cost isn’t necessarily capacity, though that can be significant. The biggest cost, it turns out, is often administering all those nodes.

Node sprawl can result from an overabundance of caution; overprovisioning is a natural response to scaling challenges. Look for a database architecture that uses resources efficiently and autonomously, ideally one that can dynamically tune itself according to varying workloads. An autonomous database monitors operations and adjusts internal processors to achieve a balance. Not only does an autonomous database greatly reduce administrative burden, it also means DevOps can always push the database to the limits of the underlying hardware resources, ensuring availability and SLA compliance. This again reduces the need to tailor database configurations and topologies to specific applications.

Flexible deployment options are also important for DevOps. Who knows what your footprint will look like in 12 or 18 months? A database needs to be fairly agnostic as to topology on which it is deployed, whether that’s on-premises, a cloud platform or a hybrid configuration across both local and cloud environments. In some deployments, databases may be required to run across two different commercial cloud platforms—for example Azure and AWS—simultaneously. There are a couple of reasons for considering this: The first and more obvious is to maximize availability. Geographical outages are more common than you might expect. The second benefit are the financial advantages to the business using the hybrid framework, as it prevents infrastructure-as-a-service (IaaS) vendor lock-in, while also providing tax benefits for the on-premises expenses. Forward-thinking DevOps groups will keep these potential requirements in mind when designing data platforms today.

Conclusion

Trends in application development and IT infrastructure are pushing traditional DBAs into DevOps, while DevOps (including SREs and full-stack engineers) are often taking on responsibilities traditionally associated with the DBA. By adopting data infrastructure that emphasizes operational simplicity, ease of scale and deployment flexibility, IT organizations can more easily reconcile these roles while building a cloud-friendly infrastructure that extends the DevOps toolchain to the database.

— Eyal Gutkind
April 11, 2018
Early Automation: A Key Requirement for DevSecOps Success

According to the “2017 DevSecOps Community Survey,” by Sonatype, almost 60 percent of the respondents consider security to be an inhibitor to DevOps agility, while more than 50 percent of developers say they do not have sufficient time to allocate to security.

Hence arises the need for enhanced security automation in DevOps. Security automation helps in refining and improving application security by reducing the time taken for detecting an attack and responding to any identified issue.

Tools for Security Automation

Organizations looking to implement DevSecOps by integrating security into their DevOps pipelines have an increasing number of tools such as Tanium, InSpec, Splunk, Checkmarx, Metasploit, FireEye and Contrast Security for security analysis and testing throughout the software development life cycle (SDLC), right from source-code analysis to post-deployment monitoring.

For security to become an integral part of the DevOps workflow, which includes continuous integration and continuous deployment (CI/CD), automation is essential. As most organizations are pushing tens of new versions of code into production for each application every day, it becomes necessary to embed security controls early in the development life cycle, which is possible only through automation.

Around 40 percent of the 2,300 IT professionals surveyed have said that they have deviated from the waterfall development model, where automated security tests are run just ahead of production, by running them throughout the development life cycle.

Using SAST Tools

Static application security testing (SAST) tools help in scanning code and providing instant feedback to developers on security-related issues who, in turn, can remediate the potential vulnerabilities as part of the standard workflow.

However, static analysis may not be alone sufficient for detecting all problems in the code.

Automating DAST

Automated dynamic application security testing (DAST) searches for vulnerabilities in real time while the application is running and is really a major improvement over static analysis that only looks for potential security issues in the code.

Inclusion of automated security analysis helps in limiting the introduction of vulnerable code earlier in the development life cycle. The runtime analysis of the issues detected through automation enables developers to prioritize the code problems that need to be fixed.

Red Teaming and Threat Modeling

Organizations have begun to employ the concept of red teaming, a military terminology, by having separate teams to simultaneously take up attacking the code and performing security testing respectively. This ensures that any potential flaws are uncovered in the production stage itself and code fixes are pushed as soon as possible.

Threat modeling is critical for the success of DevSecOps as it leads developers to look at software from an attacker’s perspective. While the threat modeling process cannot be automated, it helps in identifying flaws in the application design and architecture better than other security approaches.

Are You Ready for Automation?

Organizations can automate the provisioning and commissioning of DevSecOps environments to achieve a predictable, consistent and secure service delivery.

While the goal of DevOps Services automation is to help development teams in faster deployment and monitoring of applications, DevSecOps adds security to the automation and contributes to enhancing the quality and efficiency of the software.

DevSecOps adoption provides organizations with a strong foundation for mitigating risk in a proactive, efficient and streamlined manner.

— Veritis

April 10, 2018