Where is DevOps? Searching for it is like querying a rainbow for pots of gold. No single element in a development team’s lifecycle defines them as a DevOps-enabled organization. Were anything the rainbow and the gold, it would be the delivery pipeline. The deployment pipeline centers the team and the process, functioning as the signature of any DevOps shop.
Category: Blogs
-
Building Security and Compliance into the SDLC
As the concept of DevOps has become more popular in recent years, there are some common themes that always seem to come up in discussions about it. For example, how often do we talk about ways to speed up the software development life cycle (SDLC) in order to bring innovative applications to market faster and meet ever-increasing customer demands?
As important as this and other popular DevOps themes may be, though, they often have the side effect of overshadowing some of the less glamorous topics, such as security and compliance, which is often viewed as a sort of “necessary evil.” That said, the risks of putting an unsecure application into production can create massive challenges for organizations – from breaches to loss of company IP to brand damage – making security and compliance a topic that must be tackled as enthusiastically as accelerating innovation.
Three Critical Areas of Focus
When it comes to considering how security and compliance can become more integrated with DevOps practices, there are three critical areas on which all organizations should focus: shifting security left in the SDLC, building security into policies and creating an audit trail throughout development.
Shifting Security Left in the SDLC
Organizations that incorporate security in the earliest stages of the SDLC will ultimately benefit from products and applications that are secure by design. But in order to accomplish this, they can’t wait until the software hits production to test it for vulnerabilities.
Service virtualization can help by simulating the application delivery run-time environment and its data, allowing developers and testers to focus on security immediately, instead of waiting for access to crucial resources. In addition, a release automation tool can help collapse development cycle times from days to minutes, enabling IT teams to spend more time improving the functionality, quality and security of their applications – and less manually promoting them from stage to stage.
Building Security into Policies
It used to be that every element of application development took place between an organization’s four walls, but with the rise of cloud and software-as-a-service (SaaS) technologies, it’s not uncommon for different stages of the SDLC to occur on different infrastructures. So if the environments are always changing, how do we ensure security is consistently addressed? By building it into the policies that comprise our continuous validation methodologies.
Release automation enables development teams to create policy-driven infrastructure provisioning rules that automatically determine what application elements can and cannot be staged on specific infrastructure types. For example, a common policy would be to never let customer or database information move to a public cloud environment, where sensitive information could be lost or stolen.
Creating an Audit Trail through Development
To this point we’ve centered the conversation mostly on security, but being able to maintain – and prove – compliance throughout the SDLC is an equally important challenge that IT development and operations teams must solve.
So when we think about compliance, we’re really asking questions like “What application moved from this stage to that stage?”, “Who released the application into production?” and “If there was a rollback, what was the cause and who rolled it back?” In other words, we’re talking about audit trails and logging all activities within the SDLC – something that can be difficult when numerous manual steps are involved. However, a release automation tool will capture all of this data as a matter of course as it automates release deployments through orchestration and promotion of applications from development through production. This means you’ll always be able to answer those who, what, where and when questions that pop up during audits.
Eliminating Vulnerabilities and Getting Back to the “Good Stuff”
I’ll admit that security and compliance are not the most exciting topics in DevOps. If application development were a meal, they’d be the pile of lima beans you have to eat before you get to the dessert that is innovative feature development. The good news is, service virtualization and release automation tools can help you speed and simplify the process of building security and compliance into the SDLC, which ensures your applications will hit the market without vulnerabilities – and lets you get back to the fun stuff.
-
A/B Test with Caution
When the morning news starts talking about A/B Testing of course my ears perk up, and I eat my cereal a little slower. But when it becomes a conversation about how ethical it is to use customer usage data to iteratively improve the product, it spawns this blog post.
-
Flexible and Scalable Multi-tier Application Deployments
This is a guest post from Alberto Arias Maestro, CTO and co-founder of ElasticBox
One of the biggest problems facing software engineers since the dawn of the multi-tier application is, well, how to make it multi-tier.
It’s more than just having several supporting applications – it is about connecting the layers correctly and allowing them to communicate with each other. All this is to create a scalable and responsive deployment that can be easily updated and adapted to changing business needs.
It is about ensuring that your infrastructure can co-ordinate the order in which your application tiers are spun up, even when the apps themselves have not been designed to perform these critical dependency checks.
What Are You Looking For?
Intelligently handling your solution’s dependencies is an inherent problem in multi-tier deployments – at whatever scale you are operating. For instance,
As a developer working on a project:
“I want to ensure that my database is up and running before my web-server is deployed.”
As the CTO of a rapidly growing startup:
“I want to bootstrap on basic AWS services (such as managed cache, load balancing, and managed databases), but as the product evolves, I want to give myself the freedom to evolve the services I connect to and consume – experiencing as little downtime as possible.”
As the Head of IT Operations of a large-scale enterprise:
“I need to reign in a heterogenous environment where I have hundreds of highly inter-dependent applications that range from just-released to legacy and everything in between.”
At ElasticBox, we’ve found a way to solve this problem using the concept of Boxes, and now, Bindings, a feature we are launching today.
‘Bindings’
ElasticBox lets you build and deploy complex, multi-tier applications using boxes as basic building blocks. Specifying a ‘binding’ from one box to another is a way to declare that there exists a dependency between the boxes.
It is a construct that allows the different tiers of a multi-tier applications to communicate with each other and exchange data.
Bindings enable users to introduce not just structure, but also flexibility to their multi-tier deployments. For example,
Our developer’s Apache Web-server Box can use a required binding to specify, that to launch correctly it needs a connection to a running MySQL DB instance.
Our startup CTO, who has been trying to make a choice between AWS SQS (Simple Queue Service) and RabbitMQ, can simply switch bindings, with his business application experiencing little to no downtime.
Our Head of IT Operations, can chain his applications so that they come up in the right order. He has the flexibility to modify this chain at anytime – to accommodate new applications or possibly remove old ones, without having to bring the entire system down. If you are thinking about deadlocks – don’t worry, bindings are designed to handle these.
On-Prem? Cloud?
For enterprises, bindings can be a very powerful way to evaluate different cloud strategies for business solution with minimal disruption to the existing deployment. Bindings allow switching between an AWS RDS instance and an on-prem database all with a single click.
As you can see from some of the use-cases above, ‘bindings’ is a powerful concept that solves a very prevalent problem in advanced deployments and provides users a simple way to evaluate different strategies. To learn more about how to incorporate bindings into your deployment, please visit our documentation. For a free consultation about how bindings can benefit your particular use-case, please email us at support@elasticbox.com.
-
Infrastructure-as-code, new rules for the old game
There’s been a lot of discussion regarding DevOps and its role in fostering quality and continuous improvement through increased levels of automation. In turn, this automation is fostered through emerging infrastructure and software architectures that are exposing control plane and management programming interfaces, which has been dubbed in many circles as Infrastructure-as-Code.
This movement allows development and operations personnel to have real-time interactions with the underlying environment—the network, servers, operating systems, and storage—that are used to deliver applications and data. It also means that as the environment becomes more programmatic the business must treat this this effort as a first-class application development process and ensure that the corresponding assets follow common coding practices and meet with a consistent and sustainable architecture.
Those that are unfamiliar with the activities of system administrators and data center operations may have an ill-conceived notion after reading emerging literature on DevOps that operators have been running the business on the backbone of purely manual processes. Even as a life-long IT person, until I started working with infrastructure and operations as part of my role, I was unaware of what happens behind the great curtain that keeps the digital bits flowing through the organization.
The truth is that operations and administrators have been developing scripts for years to automate routine tasks. These scripts have typically been used to dynamically generate or update configuration files that were then applied after a restart. This change to Infrastructure-as-Code is significant because now these configuration changes can be made on the fly and are often immediately acted upon without having to take the system offline. If you’re now putting your hand over your mouth upon the realization of the potential disastrous effects this can have, you understand the impact of this power.
Thanks to Software Defined Networking, hypervisors, and even Software Defined Storage, an operator can programmatically change the entire environment underlying the application without having to notify the application of this change. Architecturally, this is a wonderful thing. Pragmatically? Well, there’s a lot of bad code out there that makes assumptions about the underlying infrastructure and is about to be exposed when migrated to these modern infrastructures.
Hence, building, testing and deploying infrastructure control applications must become part of the overall Software Development Life Cycle (SDLC) and begin to be built, tested, deployed and managed using the same controlled techniques that are hopefully being used for any applications that are being developed by application engineering.
You may be asking, “Wait, doesn’t that mean my QA environment must have the same infrastructure components as my production environment?” To wit I respond, “Maybe!” In some cases this may be advantageous for ensuring that production deployment can support continuous deployment processes, which will ensure limited human intervention from the point of user acceptance through deployment, which will, hopefully, lead to limited opportunity for production failures. However, because we’re moving to a more programmatic infrastructure many vendors can also supply virtual instances of their infrastructure code for purposes of testing and this may be satisfactory for your needs.
Of note, there are many businesses out there that do not develop their own software. They primarily use Commercial-Off-The-Shelf (COTS) applications. Thus, these businesses may not have an established SDLC practice. Likewise, these businesses may not believe DevOps is relative for them since they do not do “development”. If your business falls into this category, the move to Infrastructure-as-Code and Software Define Data Centers should be perceived as a driver for DevOps adoption as well as formally adopting one of the well-defined SDLC approaches. Even if these components won’t impact you for a few years until your next refresh cycle, the advantages DevOps brings around improving quality and consistency in delivery is of definite value to your business now.
Infrastructure-as-Code is a significant change in capability that operations has for delivering compute and storage resources to the business. It will allow them to have greater insight into the hardware allowing them to do predictive maintenance, migrate applications to various resources based on demand, and tune the overall environment to meet the business’ needs at a given moment. In short, it will allow IT to deliver compute services with a high degree of availability and security. It will also expose shortcomings in existing applications that have made assumptions about the underlying environment on which it is operating. Thus, if operations and application engineering are not working together and the entire application environment inclusive of the infrastructure control apps is not tested and deployed as a unit, the likelihood for failure will be high.
-
DevOps Rock Star dinner
I have been so busy running from conference to conference lately. Next on the agenda is the Velocity conference in Santa Clara, Ca. While I am sure the conference is going to be great, I am super psyched about attending a dinner on the 24th that promises to be a blockbuster. It is a “State of DevOps” event with an all-star cast including:
- Gene Kim
- Rajat Bhargava
- Jez Humble
- Cory von Wallenstein
- Roy Rapoport
- Ariel Tseitlin
- Nicole Velasquez
- J. Paul Reed
- John Willis
- Damon Edwards
- Howard Diamond
Sounds like DevOps Rock Star panel doesn’t it?
Part of the agenda is:
- The Why andWhatofDevOps?
- Who is DevOps for? Startup, Enterprise, Unicorn
- Business drivers?
- What will theimpactofDevOps be 10 years from now?
- ROI and business value
- How will DevOps cross the chasm?
- What job functions go away? What methodologies / tools go away?
- DevOps as a title?
- Investment landscape?
- How do we get there?
Sound interesting? I am really looking forward to this one
I will be following up with a report from the dinner later next week.
-
DevOps, Growth Hacking & other ways to rule the world
DevOps will disrupt management styles, speed the end to ancient IT practices, and flatten away processes that don’t focus on doing more with less. But DevOps is not the only dev-led movement that will impact and reshape the High-Tech industry. So called “growth hacking” will also disrupt organizational charts by bringing development tools, methods and practitioners into roles previously filled by marketing, sales, and customer success pros. As DevOps and growth hacking become more mainstream, the inevitable result is developers ruling the world.
-
The web-scale avalanche
Old-school businesses may not like it, but they’re competing with Google, Facebook, Amazon and other Internet powerhouses because those players have grown from day one with an eye on rapid, scalable, cloud-based and continuous IT operations.
Companies have been running application delivery aligned to an aging model based on low frequency release cycles. For decades, the major IT suppliers like Microsoft, Oracle, IBM, SAP and others have released software at the predictable, plodding cadence of annual or bi-annual releases. IT departments have adapted to the vendor release cycles with mirror-like processes: highly structured in the ITIL fashion with processes in place for working around lots of bugs that don’t get readily fixed by the vendors. Yet this delivery management model is now out of pace in today’s always-connected, Internet-based world in which apps and sites morph on a weekly or even daily basis.
Suddenly, companies and vendors alike must change what they’ve been doing for years. They need to transform operations and products from the legacy-based delivery model to one which is faster, dynamic and proactive. But that’s hard for most companies. It’s like driving a 1979 Chevy pickup on the Autobahn. You can’t keep up with the pace of traffic, no matter how hard you try.
Companies such as Facebook, Twitter and Netflix, born as Internet-age companies, don’t have the same problems adapting to modern times in which consumers expect instantaneous response times. They were built from the ground up to handle the relentless pace of technological change: their infrastructure and applications automatically shift according to network and CPU conditions and workload spikes. Such companies are masters at Web-Scale IT, and rely upon a multitude of open source and web-based technologies designed for distributed computing. More than just technology it also takes radically new processes and mindsets to become a Web-Scale IT company.
Operating in the Web-Scale World
Web-Scale IT consists of technologies and methodologies that enable developers to build, change and scale applications 10 to 100 times faster than in the past. Web-Scale IT relies upon many modern concepts, including cloud computing, peer-to-peer architecture, NoSQL, SOA, Agile development, DevOps, Continuous Delivery and Deployment and more. Gartner describes Web-Scale IT as “a pattern of global-class computing that delivers the capabilities of large cloud service providers within an enterprise IT setting.” The research firm predicts that by 2017, Web Scale IT will be an architectural approach found in 50 percent of global enterprises, up from less than 10 percent in 2013. Below are a few areas of change that IT organizations must navigate to make the shift.
1. Reevaluate IT partners. It’s not just CIOs that need to shift operations for Web-scale computing. The legacy IT vendors are struggling to walk away from profitable business models based on licensing to embrace the on-demand, services-based revenue that just a few years ago, catapulted startups like Salesforce.com into billion-dollar enterprises. The legacy vendors are unable to innovate fast enough and are having to rely on serial acquisitions to keep their technology relevant. Yet rapid acquisition comes at a great price and is not sustainable. IT departments will need to look far and wide to find the best partners that can support their Web-Scale IT efforts moving forward.
2. Acquire diverse, open-source based IT operations tools: CIOs will likely need to constantly review their technology sourcing and have a more critical method of choosing their technology partners. Relying on a few trusted long-term partners will likely not achieve the Web-scale IT goal. Consider that Netflix, which now runs its business almost exclusively on Amazon AWS, only a few years ago relied on its own datacenters full of legacy vendor equipment. Today, the company manages its high-availability video streaming business using a collection of inhouse-developed open source tools (coined The Simian Army) that test the fault tolerance of the company’s operations. Being flexible and open to change allows Web Scale companies to keep ahead of the competition. Standing still, even for a short while, is no longer an option.
3. Foster iterative, collaborative processes: Strong DevOps cultures, where developers and IT operations personnel exchange information frequently to stay in step with changing business requirements, is also pivotal to making this shift. Companies with advanced DevOps practices are usually committed to running their businesses differently: more efficiently, more customer-focused and with the keen ability to respond quickly to change. CIOs and CTOs need people with strong Web technology skills yet also an unflinching commitment to collaboration and complex problem-solving. IT people today need to think on their feet and adapt as they go. Twelve-month planning cycles are out the window.
The building of a storm
Just a few years ago, it was considered evolutionary, if not revolutionary, to purchase CRM or HR as a service. That’s commonplace today. The next area for innovation is within core applications such as a retailer’s pricing application or a bank’s market forecasting tool. Those applications must now also run at Web Scale to be responsive, customer focused and speedy. Don’t count on your Oracle or SAP database to shift so easily or quickly—at least not yet. That’s why Internet leaders are managing IT infrastructure using the newer generation of Web Scale technologies including Puppet, Chef, AppDynamics, New Relic and yours truly, Boundary.
If running at Web Scale sounds overly complex, it’s not. Since most of the latest, open source-based operational tools are highly automated, using them is actually simpler. Crowdsourcing is another strategy that companies are exploring for Agile development projects. The bigger hurdle is in moving from the old-fashioned software development style of over-engineering everything to the more intuitive, iterative style that characterizes modern Web application development. With Web-Scale IT, it’s okay to cut corners and take more risks, because the infrastructure will stand behind you.
There are plenty of people who’ll disagree with me, but I can envision a world in which the Oracles and IBMs no longer run most of the companies in the world. — one that is adaptive, business-responsive and always forward-looking. The future is closer than we think. Will your business evolve?
-
The DevOps value stream is not SOP
#devops #lean Don’t confuse doing something faster with greater efficiency.
Much of the focus on DevOps remains at the implementation level – on gaining efficiencies through automation of tasks and orchestration of processes. While this is not a bad thing, such a micro-focus on the moving parts can result in the loss of efficiencies to be gained at the macro-level.
One of the biggest inhibitors of a timely release cycle is not the moving parts, but rather the process in which those moving part are executed. It’s not necessarily the actual push into production that’s problematic, but the length of time it takes to accomplish certain tasks when they cross internal organizational boundaries.
For example, many years ago we began the process of pushing a new application into production – a web application. This required, perhaps obviously, changes to the corporate firewall. The “process” for accomplishing this change – essentially opening a port on the firewall for the intended public IP address – required no less than five different signatures across networking, security and architecture teams. There were only two security roles allowed to approve such a change, and the process was noted as taking up to 15 days to get a signature on a piece of paper and subsequently a rule change.
This was not a task level problem. It was not implementation issues. It wasn’t even difficulty in making the change on the firewall. It was all about the process. Scheduling a release meant taking into consideration not only the readiness of the application itself, but its supporting services – including networking.This is why DevOps should start with a review of the processes used to release an application into an environment (whether test, QA or production). Mapping out the process is the first step to finding where the bottlenecks might be that are causing delays in moving that application to the next phase of its lifecycle.
Simply automating an existing process may not provide the value expected or touted by DevOps supporters, because if the process is poor, you just end up doing something poorly much faster.
In other words, the value of DevOps is not necessarily in simply automating SOP (Standard Operating Procedure).
The initial Value Stream Map captures the current process as it exists. This is known as the “As Is.” It is the actual process flow and should be created through observation of the current flow and with participation of those who are in the process. Do not make the mistake of pulling a process map from the procedural manual (SOP). The best technique for mapping is to be the unit (the “thing”) coming through the process. For example, in the insurance industry it might be the policy. In the auto industry, it may be the car. In each instance, the Value Stream Map begins with the request for the unit i.e. insurance policy or car. [emphasis mine]
http://www.goleansixsigma.com/how-to-visualize-a-process-with-a-value-stream-map/
For application deployment, the “thing” is … an application, and the process should be mapped based on what’s actually happening. Only when you have the process mapped can you evaluate it for bottlenecks and understand the steps of the process at which each group adds value. Value is defined as any step that moves the thing (the application” toward the end goal. Efficiency is then measured as a percentage; specifically it’s the value added time as a percentage of the total time it takes to complete the process (the cycle time).
Don’t confuse doing something faster with greater efficiency. If every step of the cycle is automated, the actual process efficiency hasn’t changed. The ultimate goal of DevOps should be about improving the efficiency of these processes, not just doing what’s always been done faster.
Some of the opportunities for improvement in efficiency that will surface is certainly interfaces with other organizational groups. Perhaps it’s that a process step might not take as long if you had relevant data or information up front. If you’re wasting time waiting for some piece of information, that’s not value added time. It’s just cycle time. More cycle time reduces the efficiency of the overall process by overloading it with wasted time. Better information gathering up front can help reduce the wait time and immediately provide greater efficiency of the process.
It is these opportunities for improvement that will be particularly helpful in the enterprise arena, where the issue is not so much attempting to hit ten deploys a day, but rather the predictability and efficiency of the processes that enable teams to hit a specified change window as desired.
DevOps is not about the tools, or the automation of manual tasks. It’s about the big picture, and that means gaining greater efficiencies through process re-engineering and orchestration as well as automation. Focusing on the micro aspects of DevOps will only ensure that if you’re doing it wrong, you’re doing it wrong faster. Step back and look at the big picture, and find those process steps that are really causing inefficiencies in your organization.
