devopsy.kinsta.cloud

Tag: continuous delivery

  • What is a DevOps ‘Best Practice’?

    What is a DevOps ‘Best Practice’?

    In IT, we love our frameworks. Whether agile, ITIL, lean, cobit, six sigma or others, our thirst for “best practice” guidance seems to be unquenchable.

    The concept of a “best practice” is an enigma. Who decides if a practice is actually the best? And best for whom? While an “adopt and adapt” mentality is advocated in most frameworks, there is still a tendency to benchmark efforts against and conform to published best practices. Whether promoting “best in breed,” “world-class,” or “best practices,” many organizations swing these terms around as some form of competitive advantage.

    Is it really? Shouldn’t business outcomes be the real competitive advantage and measure whether IT’s practices are really the “best” to meet customer requirements?

    The DevOps Institute was founded in 2015 as a global learning community for emerging DevOps practices. Since our launch, we have intentionally avoided any reference to DevOps “best practices.”

    We believe that DevOps practices are constantly emerging and, in many cases, being proven in the wild.

    Through research, interviews, conferences and the experience of our Regents and partners, DOI’s goal is to identify, capture and share emerging DevOps practices through knowledge, education and certification models that best serve the learning community and enterprise IT. We feel it is premature to certify an individual as an “Expert” or “Master” when the expertise and mastery needed for successful DevOps is not clear and may not be appropriate in a single person.

    DevOps is fairly young. It does not have a single body of knowledge. There is disagreement on whether it is a philosophy, movement or framework. Despite that, DevOps is crossing the chasm from innovators to the early majority faster than any other IT framework. Why? Perhaps it is because DevOps speaks to many of the cultural and technical challenges that IT has been amassing for years.

    So instead of focusing on best practices, let’s agree on DevOps’ core principles:

    • Go faster
    • Shorten feedback loops
    • Experiment and learn
    • Cultural transformation
    • Deliver business and customer value constantly and consistently

    No doubt, some practices have clearly risen as being critical to the realization of DevOps’ core principles:

    • Agile software development
    • Continuous integration
    • Continuous delivery pipelines
    • Automated and continuous testing
    • Proactive monitoring
    • Improved communication and collaboration

    Still, over the past year, DOI has seen additional practices gain interest

    • DevSecOps/rugged DevOps
    • ChatOps
    • Agile service management
    • Lean
    • Immersion practices (Garage, Lofts, Dojos)
    • DevOps teams

    What will emerge next year? Who knows—but that’s the exciting part of DevOps. As time passes, there is less contradiction and more supplementation. Each emerging practice seems to complement and fine tune what came before.

    Even without a standard definition, DevOps is already encouraging organizations to examine their proprietary practices, identify gaps, assess their automation and, most importantly, engage in collaborative discussions. Transformations have taken seed without defined best practices. That’s awesome! Let’s not quell the momentum by attaching a static set of best practices.

    Will DevOps’ best practices eventually be agreed? Maybe. DevOps touches almost every aspect of IT management—people, practices and automation. That’s a tall order for a set best practices. Certainly some will try to publish their version of a “definitive” DevOps body of knowledge. Even “The DevOps Handbook,” an important book publishing later this year, was built as a collaborative effort between many of the early founders of the DevOps movement. Their perspective and insight will be awesome but not finite.

    I understand the frustration by the lack of a single body of knowledge and prescribed best practices. In past frameworks, the body of knowledge was both a enabling tool and constraining limitation. Bodies of knowledge are difficult to keep current. For DevOps, let’s embrace the concepts of spirit of sharing, collaboration and continuous innovation by promoting an inclusive and collective body of knowledge of emerging practices.

    If we settle on a set of DevOps best practices, are we actually inhibiting the very values that we are trying to instill? I hope not.

  • Tales of DevOps Discovery Part 2: Definitions Matter

    Tales of DevOps Discovery Part 2: Definitions Matter

    Definitions Matter: Step Two in a DevOps Transformation

    The first post in this series on DevOps Discovery discussed the importance of asking fundamental questions of stakeholders to facilitate alignment on priorities and establish a mission statement when beginning a DevOps transformation. Part 2, below, reviews definitions of common DevOps-related terms and provides examples of how failure to agree on such terms can hinder your transformation.

    The industry is full of terms such as continuous integration, agile, continuous delivery, DevOps, etc. The practices described by these terms often are defined differently by various industry “experts,” resulting in stakeholders in the software delivery process having different understandings of what constitutes these practices. Some will state they are already following the practices; others will state they do not. Some will believe the objectives unachievable, while others will be bullish on the prospects; differing interpretations blocking agreement on the organization’s current and target states. If stakeholders cannot agree on the current state (or the start point) and the target state (or the destination), it is impossible to “map a route” for the journey to DevOps.

    For example, let’s look at how some define continuous delivery:

    Continuous Delivery – Definition 1:

    A software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. It aims at building, testing and releasing software faster and more frequently.

    The approach helps reduce the cost, time and risk of delivering changes by allowing for more incremental updates to applications in production. A straightforward and repeatable deployment process is important for continuous delivery.

    Source: Wikipedia

     

    Continuous Delivery – Definition 2:

    Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time.

    [Continuous Delivery Blog Post by Martin Fowler]

     

    You may read the above and initially think there is no difference in the definitions but, note that the first definition addresses deployment of software with the statement: “A straightforward and repeatable deployment process is important for continuous delivery,” whereas the second definition addresses deployment of software with the statement: “Build software in such a way that the software can be released to production at any time.”

    This distinction may seem trivial, but it is the often the source of confusion between the practices of continuous delivery and the practice that is commonly called continuous deployment. In contrast to continuous delivery, which states that every software build “could be released,” continuous deployment is loosely defined as “the practice of releasing every good build to users.”

    When establishing your target state for the DevOps transformation a seemly slight confusion such as this can result in pushback from stakeholders concerned that continuous deployment is unachievable or too disruptive. These sentiments include:

    • Business: “We cannot gather, spec and track our requirements at that rate.”
    • Operations: “There is no way we can support daily delivery into production, ensure security, uptimes and perform our day-to-day work. It takes us an entire weekend and multiple attempts to deploy today.”
    • Customers: “Our compliance team needs three months to review and approve every new version of our accounting system.”

    The confusion and potential pushback exemplify the need for establishing baseline definitions and alignment at the start of a DevOps transformation, and doing so in the specific context of your organization. The importance of points such as how far it deploys varies with the organization—some organizations can support continuous deployment to production multiple times a week or even multiple times a day, such as Netflix does. Other organizations, such as providers of critical financial systems, may never pursue this cadence of deployment. Be sure to apply these industry practices pragmatically as fit for your organization, not dogmatically based on generic industry definitions.

    Let’s look at a couple of definitions of DevOps:

    DevOps – Definition 1:

    “DevOps (development and operations) is an enterprise software development phrase used to mean a type of agile relationship between Development and IT Operations. The goal of DevOps is to change and improve the relationship by advocating better communication and collaboration between the two business units.”

    [Webopedia]

     

    DevOps Definition 2:

    “DevOps (a clipped compound of “development” and “operations”) is a culture, movement or practice that emphasizes the collaboration and communication of both software developers and other information-technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment where building, testing, and releasing software, can happen rapidly, frequently, and more reliably.”

    [Wikipedia]

     

    Like the example with continuous delivery, these two definitions also are very close. Yet on detailed observation, it is clear the first definition describes DevOps as being focused on relationships (i.e., culture and collaboration). The second definition extends this to include the “automating the process of software delivery and infrastructure changes,” giving consideration to tools and technical practices such as continuous delivery.

    Does this matter? Yes. Often if the executive sponsors and key stakeholders do not account for the need to establish tools and technical practices and only focus on relationships, there is a high chance for unrealistic expectations for timeline, budgets and results. Conversely, if the need for changes in culture and process are not considered and the DevOps transformation focuses only on tools and technical practices, the efforts are also unlikely to yield the expected results. A DevOps transformation must account for the trinity of “People, Process and Tools.”

    When considering DevOps, it objectives and the objective of its component practices it is useful to establish a clear idea of the relationships between theses practices. It is often useful to visualize them as below:

     

    DevOps Transformation Triangle
    The DevOps Transformation Triangle

     

    DevOps is the “What,” the target state as defined by a set of guiding principles. Agile development, continuous integration and continuous delivery are the “How,” a collection of process, practices and tools that together build the foundation for a sustainable DevOps implementation.

    In the next section, we will review some of the initial discovery questions asked when beginning on a DevOps transformation, provide real-world examples of answers from various stakeholders in real organizations, and analyze those answer to better understand the journey to DevOps.

  • Enterprise Cassandra Deployments

    Enterprise Cassandra Deployments

    Successful Cassandra deployment in the cloud takes a certain amount of skill and knowledge. In this blog I will share best practices for a successful Cassandra deployment in the cloud such as Amazon Web Services (AWS) and Google Cloud Platform (GCP). I will also discuss how you can protect and recover the data in your Cassandra cluster when deployed in the public cloud.

    Cassandra Deployments in the Public Cloud

    There are two main ways to have successful enterprise Cassandra deployments in AWS:

    • Use a ‘managed’ Cassandra deployment. This is a reasonable approach; the only downside is the inability to customize data placement beyond what is typically provided by the vendor.
    • Manage the cluster yourself, if you want to have full control over your Cassandra deployment and customize it to scale and perform according your business needs.

    Please refer documentation from Datastax or Apache Cassandra for detailed best practices for Cassandra deployments. The following best practices are reiterated here to simplify deployment of a scalable Cassandra cluster in AWS.

    Storage

    Both AWS and GCP offer two types of local storage options for their compute instances. For AWS, the options are Elastic Block Storage (EBS) and Ephemeral Storage. In GCP, the options are Persistent Disks and Local SSD. Both options have their own trade-offs. Pick the one that best suits your deployment.

    Persistent Storage

    EBS volumes have performance characteristics similar to network storage such as a NAS or SAN. Because storage from EBS volumes is multitenant in nature, it is not suitable for distributed, log structured databases such as Cassandra. If you want to ensure reliable performance through EBS volumes, use “EBS optimized instances” with “provisioned IOPS.” Since the IO traffic to EBS volumes goes over the instance network, EBS optimized will provide reliable performance, but does not guarantee any increased performance. To get the most performance out of EBS volumes, use instances with multiple disks and separate the volumes used for “Data” and “Commitlog.”

    If you have to use EBS Volumes, go with “EBS Optimized Instances,” with “Provisioned IOPS” and split data and Commit Log on different volumes.

    Ephemeral Storage

    Ephemeral storage is a great option to use for downstream Cassandra environments such as test/dev and staging. Since ephemeral volumes are typically SSDs, there is no need to separate the data and commitlog into separate drives. To optimize performance, choose EC2 instances with multiple ephemeral storage drives and stripe them together into one logical volume. Send data and commitlog to the logical volume. Remember that when you stop the EC2 instance, the data in Ephemeral storage will be deleted.

    Stripe all Ephemeral drives into a single volume to get maximum performance.

    Data Distribution

    Cassandra uses Snitch to determine the topology of the cluster. The Snitch is also used to determine the data center and rack for each node in the cluster. Replication strategy is used to determine the placement of data across the different nodes in the cluster.

    AWS

    1. EC2Snitch works well for single region deployments.
    1. Ec2MultiRegionSnitch works well for cluster distributed across multiple AWS regions. Please note, Multi-DC, Active-Active configuration makes your cluster resilient to region failures and for multi-dc cluster, use “local quorum” to improve performance.

    Google Cloud Platform (GCP)

    1. Use GoogleCloudSnitch for Cassandra deployments in GCP.
    1. GoogleCloudSnitch treats regions as Cassandra data centers and availability zones as racks.

    Other Features

    Both AWS and GCP offer a series of operational capabilities that seem to make it easier to maintain high uptime on Apache Cassandra cluster. These capabilities have to be chosen carefully depending your deployment needs.

    1. Load Balancing: Network-level load balancing service is offered by both AWS (Elastic Load Balancer) and GCP. It is not recommended to put a load balancer in front of Cassandra. It is better to use a client connector that can handle this, consider open source utilities such Presto and Astyanax
    1. Autoscaling: Both AWS and GCP offer Autoscaling facility for compute instances. However, although this functionality looks very enticing, putting all C* nodes in an autoscaling group will not produce the desired result. Instead, create a separate autoscaling group of ‘1 node’ for every node in your cluster. This will ensure that anytime a node goes down, it is automatically brought back up. This will ensure the cluster data placement is not disturbed when nodes go down and nodes coming up do not cause excessive data movement.

    Scalable Backups and Reliable Point-in-Time Recovery

    Backing up your data is a must-have for enterprise organizations. The impact of data loss is so high that no enterprise wants to risk relying alone on replication. Protecting data in a public cloud environment may seem daunting, but there are several options available that can make this easier compared to an on-premise deployment. A simple option may look like one of the native snapshots:

    Storage Snapshots

    I have seen few customers use snapshots of EBS volumes as a way to protect the data in their cluster against logical and/or operational errors. This solution will work but with some severe limitations. On top of the overhead of maintaining the scripts to take snapshots, there is a large overhead in storage costs. Taking a snapshot of volumes does not allow for incremental backups of your data. This causes an exponential bloat in the amount of data stored in snapshots, leading to high cost of the backup solution.

    Amazon S3 or Google Cloud Storage

    Using an S3 bucket as the secondary storage enables enterprises to use low-cost, highthroughput storage, while reducing overall operating costs for storing backup data. Here are a few recommendations when using S3:

    1. S3 Region: Amazon S3 buckets are region-specific. A bucket created in one region will provide better performance characteristics to other infrastructure in the same region. Subsequently, it is important to ensure the S3 bucket used as the secondary storage is in the same region as the production database being protected and the Datos IO cluster.
    1. Storage Class and Lifecycle Policy: Amazon S3 allows users to create lifecycle policies on buckets. These policies dictate the performance and response time characteristics of objects in the bucket. STANDARD_1A is more suitable for longer lived and less frequently access data, and is not suitable as secondary storage. Additionally, do not turn on any lifecycle policies on the buckets used for the secondary storage. All buckets should be in the standard storage class.

    Conclusion

    As enterprises leverage the agility provided by public cloud environments, it’s important to understand the various facets of Cassandra deployments to ensure the efficient use of your money and time.

  • Continuous Delivery and Deployment: A Tale of Two Models

    Continuous Delivery and Deployment: A Tale of Two Models

    EnerNOC (Energy Network Operations Center) is an energy intelligence software provider with more than 1,300 employees in its Boston headquarters and international locations. The public firm helps global customers to efficiently consume energy. “Our mission is to change how the world uses energy, helping customers to improve profitability and visibility for better decision-making and sustainability by focusing on how they buy energy and when and how they use it,” says James Nichols, principal engineer and cloud architect at EnerNOC.

    A regular user of Splunk (the company uses other DevOps tools and platforms as well, including AWS and Oracle), EnerNOC has fully automated monitoring in its perpetual drive to move past continuous delivery and on to continuous deployment.

    Embedded in EnerNOC’s story of why it uses continuous development and continuous deployment models is a chapter on its move to fully automated monitoring.

    EnerNOC DevOps Models

    EnerNOC’s DevOps model is in constant transition toward full automation. Currently, it has fully automated deployment, fully automated monitoring and semi-automated quality assurance (QA), which pull the company’s continuous development approaches.

    A change to “builds on commit” of smaller applications within the legacy software monolith at EnerNOC came hand in hand with further automation, targeting what the company calls “trustworthy continuous deployment,” Nichols says. That was the crux of EnerNOC extending itself into deployment automation.

    Wherever the model is not fully automated and continuous in deployment, it remains continuous delivery. “Most of the variance between the approaches is due to aspects of the delivery pipeline that still have manual processes; these parts are usually comprised of the remaining manual testing procedures,” says Nichols.

    Change Control Necessitates Two Models

    EnerNOC develops and maintains mission-critical software that must pass through a change management approval process. This process includes communications within and across the company and customers, as well as user acceptance testing. This is where the manual testing comes in. Only at the end of this approval pipeline can EnerNOC pass the software baton into the hands of production.

    Due to EnerNOC’s tightly yoked systems and the manual testing, change control remains, along with a mix of continuous development and semi-automation with fully automated continuous deployment. “DevOps is a cultural movement as well as a technical one, and so the existing model could evolve with the organization’s quality and maturity evolutions,” says Nichols. If the manual testing and the relationships between systems give way to more fully automated testing and more amenable relationships, QA could more closely embrace full automation.

    Monitoring Challenges/Achievements

    EnerNOC’s application diversity includes hybrid software with a foothold in the Amazon cloud, in EnerNOC’s data centers and on-premises. “We use three tiers on top of Oracle in the data center. In AWS, we use a combination of big data, cloud-friendly and serverless technologies,” says Nichols. Business transaction data traverses many machines that belong to service groups’ stakeholders. “The distributed, decentralized architecture makes troubleshooting, performance management, security operations and production operations challenging,” notes Nichols. Better monitoring was the answer.

    EnerNOC wanted to offer nontechnical users easy access to systems-level monitoring data with transparency into the many systems through a single pane-of-glass dashboard with readily available system notices and troubleshooting abilities. “We would use Splunk to do this and then feedback everything we learned from using Splunk into driving our DevOps and engineering processes into the future,” he says.

    The EnerNOC DevOps performance team built the dashboard that accompanied the Splunk service, ensuring it would register the performance metrics that are of value to the organization. The performance, product management and development teams maintain an awareness of the metrics and receive the related alerts. “The development team incorporates the ongoing metrics early in the development life cycle to improve application performance,” says Nichols.

    EnerNOC first applied Splunk to web logs for operational capacity planning, then used Splunk to give the Ops team transparency into the function of its new real-time data product that was working in production. As new products came on board, the organization added them to Splunk to provide Ops teams with dashboards, at the urgent request of Ops team members.

    Current Status: All Systems Good

    EnerNOC applies Splunk to test and production systems for views into consistency and acceleration. “Splunk is our primary alerting infrastructure for low-level system metrics like CPU utilization as well as for complex business transactions that span multiple systems,” says Nichols. Splunk has eyes on application health, cluster state across machines, expected systems tasks and suspect or known-bad system behavior, as well as the presence or lack of any significant event.

  • Data Protection for Operational Reporting Platforms

    Data Protection for Operational Reporting Platforms

    Making real-time business decisions based on data from social, mobile and cloud platforms is a requirement in today’s world. That’s why an ecosystem of next-generation operational reporting platforms such as Spark, Apache Cassandra, Kafka, Docker, Mesos, Marathon and more have come to life. Enterprises are using these technologies to re-architect their data storage and processing frameworks to achieve lower costs, scalability and faster response times. However, as enterprises onboard business-critical applications on this next-generation technology stack, they need to prepare for failures or errors that may disrupt customer-facing applications.

    One of the key benefits that operational reporting platforms provide is real-time visibility, which allows businesses to make strategic decisions based on actionable insights. However, the new applications generate a large amount of data and archaic operational reporting platforms (those based on traditional data warehouses, data marts and ETL logic) are unable to catch up and, as a result:

    • Business decisions are based on stale data
    • Fragmented data increases the risk of errors
    • Multiple copies of data increase storage costs

    These issues have driven many progressive, data-centric enterprises to re-architect their operational reporting platforms. In this new architecture, Kafka serves as a persistent message bus that ingests data from relational stores and provides failure resiliency and message buffering. Spark or Spark Streaming may be used for transformation, aggregation and other lightweight stream processing. These processing nodes may be deployed in Docker containers. Finally, Apache Cassandra serves as a distributed data storage layer that provides linear scalability and high-availability. Kafka also enables the ability to add multiple data consumers such as Hadoop/HBase through Flafka.

    Datos IO - operational reporting platforms

    However, as enterprises adopt operational reporting platforms, they want to ensure that bad data feeds and human error do not result in data loss. Apache Cassandra natively replicates data but that does not provide the capability to go back in time to recover data efficiently and at scale.

    As enterprises onboard their critical applications to achieve development agility, scalability and lower operating costs, they must also explore data protection solutions that are specifically dedicated for scale-out databases.

  • Preserving Bottom-Up Innovation in DevOps

    Preserving Bottom-Up Innovation in DevOps

    Balancing top-down control and oversight with bottom-up innovation and autonomy

    The driving force behind DevOps adoption in the enterprise is the idea that it will enable faster and more frequent delivery of better quality software. If all goes well, DevOps should result in higher customer satisfaction and lower costs, but the journey to get there can be challenging. Unfortunately, only a minority of organizations that attempt to implement DevOps actually fully deploy it.

    There are many barriers to overcome before organizations can realize the benefits of DevOps. One of the main challenges is retaining sufficient top-down control and oversight without hampering bottom-up innovation. 

    From Start-Up to Enterprise

    The root of this problem lies in the fact that many ideas in DevOps were developed in environments with small teams and flexible organizational structures. In such setups, where everyone can see what everyone else is doing, visibility and control seldom are an issue. Adapting this approach to an organization with hundreds of geographically distributed teams, in industries with a lot of regulatory baggage and diverse legacy systems, is an entirely different prospect.

    How do enterprise organizations find a structure that enables individual teams to unleash their innovation and maintain a feeling of responsibility without losing oversight and control? If DevOps teams are not empowered to make their own decisions and free to move swiftly, then they’ll struggle to realize the benefits that drive adoption in the first place. 

    Walking the Tightrope

    Individual DevOps teams should be free to make local decisions, but these need to be in alignment with business goals at the management level.

    You don’t want to hamper the speed and reliability that automation brings, or the innovation that can come from autonomous decision-making, so you need to find a way to balance tools, processes and team structure to maintain control and visibility. Avoid flying under the radar; instead, aim for maximum transparency in what your DevOps teams are doing. Set goals and measure, because visibility for management will prove that the new approach is working, build confidence and secure more top-down investment.

    Connecting the Dots

    A large enterprise will have a number of DevOps teams, but they shouldn’t be completely isolated. Requirements such as audit trails, security access logs and compliance controls must be met by all of these teams. There’s little sense in making each individual team responsible for solving the same problems over and over again. Offer a set of standards and tools, but be careful not to shut down choice.

    That same logic applies to competence development, so offer your DevOps teams access to best practices and enable them to learn from each other. The size and structure of an enterprise organization does present unique challenges to DevOps adoption, but there’s also opportunity here to benefit from the scale by sharing and collaborating. Sharing and standardizing the right things will eradicate redundancies and free your DevOps teams to focus on improving their products and innovating, so the business derives maximum value from their efforts. 

    Finding What Works

    The right answer will be different for each organization, but there are some things everyone should keep in mind when building or shopping for tooling. It needs to support a large, diverse environment. Look for support for the platforms you use and consider extensibility, so you can add the functionality you need.

    A key factor here for orchestration tools—especially those that define and execute your delivery pipelines—is that they must be accessible for management. You shouldn’t need to be a technical expert to dip in and get an overview, or compile a report that provides the insight you need. Ensure you collect diverse metrics and present information in a digestible format so you can find bottlenecks and identify successes and failures and really deliver the best experience possible for the end user.

    About the Author / Andrew Phillips

    cropAndrewPhillipsAndrew Phillips is VP of DevOps Strategy for XebiaLabs, provider of software for continuous delivery and DevOps. He is a cloud, service delivery and automation expert and has been part of the shift to more automated application delivery platforms. He contributes to a number of open source projects including Apache jclouds, the leading cloud library, and is a co-organizer of the DynamicInfraDays container community events.

  • 5 Prerequisites for a Successful DevOps Initiative

    5 Prerequisites for a Successful DevOps Initiative

    There are certain tools and attitudes that are essential to any successful DevOps transformation.

    The manner in which a DevOps transformation is performed will have huge implications on the level of agility the company is able to achieve. However, before that point is even reached, some vital foundations must be laid. It is from that foundation that any innovation occurs. With that in mind, each of these elements are equally important, and should be regarded as such.

    Consensus

    Your first achievement on the road to successful DevOps should be an agreement on what DevOps means to your company and how it should be applied to your needs. Consensus on a strategy is almost as important as the strategy itself and must be in place before a detailed road map is constructed. Remember that DevOps is subjective and its implementation should vary from company to company. Consensus hopefully will be easy, but if not, it must be built by persuading doubters of the need for certain principles and practices selected.

    This is the first step on a cultural journey; the collaboration between Dev and Ops staff and their consensus on a plan could make or break the DevOps idea. The concepts behind the initiative need to be agreed on from all corners of IT, as well as the business and any other key stakeholders such as suppliers. It is important the cons and pros of switching to DevOps are made clear for a balanced proposition. The argument for a shakeup on this scale needs to be compelling and justified for it to go ahead.

    Consensus also involves handling expectations. To keep everyone happy there must be metrics and approximate timelines set in place so everyone knows what to expect and when, to avoid conflict further down the line. Only once consensus has been achieved and the basics are put in place can a more detailed road map be built.

    Flexibility

    A thorough road map is the skeleton that allows its muscular teams to flex independently toward a common goal. However, flexibility without allegiance to the agreed principles can cause chaos. The DNA that runs through all teams stays the same but applications vary, and so teams need to work autonomously if they are to cater for the specific needs and nature of the app they are working on.

    While the road map gives a structure and handles expectations, DevOps culture needs to be tied to business outcomes and not formal internal milestones. Each DevOps team needs to be trusted to interpret the road map as it suits the specific project they are working on. Agile methodologies must be used, of course, but their interpretation is devolved to the teams themselves. This license and trust is key in allowing DevOps teams to react quickly and mold around the needs of the individual application.

    Teams need freedom in terms of what tools they use and to an extent, how they use them. The methodology used to achieve the agreed core principles should have an element of flexibility. Tool-agnosticism is a must when choosing an automation solution for effective DevOps. Teams are happier and work faster when allowed to choose the tools that suits them best, making them more successful.

    Automation

    As C. Little said, “DevOps isn’t about automation, just as astronomy isn’t about telescopes.” An automated continuous delivery pipeline should underpin any DevOps initiative, adding consistency and reliability to flexible delivery methodologies. Tribal knowledge from Dev and Ops is gathered to create a pipeline that is the sum of the knowledge of its contributors. The pipeline is repeatable, and over time is made more efficient by rooting out inefficient manual processes. Modelling into a workflow helps all involved visualize where a team is in the app life cycle and what next steps need to be taken next and by who.

    Trust in application release automation can be a big deciding factor in how companies achieve successful DevOps practice. Sometimes there is a culture within a company of wanting to have manual control over certain processes that inevitably slow things down. Automating any and every manual task possible is essential to speed up DevOps. If a manual intervention is needed, the relevant member of staff can be notified by the automation solution itself, which automates out the remainder of the task for which manual intervention is not required. Over time, manual interventions will disappear altogether.

    The ability to give DevOps teams freedom in their tool selection keeps staff productive but adds yet more complexity and leads to a highly heterogeneous workplace. It’s important your automation solution has approved integrations to all tools DevOps teams might choose and also releases free integration to new tools quickly, so you can stay on top of the latest trends and continue to be successful.

    Cooperation

    Unless you’re a startup with heavy investment, you’ll likely be changing the structure and workload of your existing staff rather than hiring a new team suited perfectly to their individual roles. The success of DevOps depends on the ability to make positive change in the way team members collaborate.

    Resistance to change may occur due to staff being accustomed to working in one particular way for a long time. However, this could also be down to personality type. As it is Ops and Infrastructure staff who face the biggest changes and are used to limiting rather than encouraging risk, often they can be the most resistant to change. Often devlopers do not have to change the way they work to the same extent, so fewer feathers are ruffled.

    I have seen big corporations that do not have compliance from everyone, so their DevOps is limited in scope and unfortunately they will never reach full agility. Sometimes you must adjust your teams to ensure positivity to DevOps and receptivity change in general.

    Rethink Architecture

    I’m not saying DevOps cannot be implemented in legacy systems, but there are elements of legacy systems that make it hard to instill full automation and continuous delivery. What is important is that the organization is open to a certain degree of architectural change. Monolithic apps can certainly slow down a DevOps initiative—often it’s mainframe apps that are not built with a service-orientated architecture—so it’s important to know how to adapt them to fit your strategy.

    A full architectural revolution is not needed to be successful. Obviously this is where cloud-native startups have an advantage as they can assemble their infrastructure from scratch, but there is no need to rip and replace existing systems for the sake of it and waste money. Certain elements of even the most monolithic systems can be modernized—for example, Siebel deployments can be fully automated so operations staff are free to concentrate on more demanding aspects of the cultural change DevOps involves.

    DevOps applications can interact with legacy systems if we use the right techniques. Make sure your automation solution has integrations with the likes of SAP, Oracle and Siebel, and so enables legacy systems to be enablers on your digital journey rather than obstacles that will stand in your way as you seek to reach full DevOps and end-to-end continuous delivery.

  • Balance: The Key to Successful DevOps

    Balance: The Key to Successful DevOps

    In the realm of IT buzzwords, DevOps is in a league of its own. The pervasiveness of the word is matched only by the variety of its definitions. Over time, much like the parable of The Blind Men and the Elephant, it has slowly taken shape out of the void, albeit in a backwards fashion. Associated terms such continuous delivery, integrated quality assurance and deployment automation have helped backlight the beast’s silhouette, but there is much more to discover.

    Generally, DevOps is defined by how it is used rather than what it is. This is helpful for those already familiar with the amorphous nature of the movement, but can be a confusing introduction for the uninitiated. To achieve a better understanding, it is important to grasp the underlying need driving the shift to a DevOps model.

    Currently, the world of IT is barreling toward a future dominated by increasingly specialized and inherently complex automated solutions. Software is taking a front row seat in ways never before possible. (Insert here)-as-a-service offerings and the demand for instantaneous and consistent deployment turnaround is at an all-time high. The days/weeks required to shuffle a solution between development and engineering is a luxury that most companies can no longer afford and is less tolerated by customers. This level of expectation is something that the traditional IT org chart was not designed to cope with. Tech companies everywhere have felt the strain of this increasing momentum, with many at a loss of how to handle it. Demanding that teams work harder, faster, longer will only get so far. And inevitably, increased intensity results in a rise in mistakes and an inferior product.

    Balance: Dev’s Yin to Ops’ Yang

    Clearly, change was and is needed. The path from development to operations needs to be streamlined at a fundamental level. However, dissolving the barriers between two longstanding departments is not without its perils. As any IT veteran will tell you, development and operations each have their own cultures, quirks and personalities. But the differences in each help create the perfect balance.

    In its purest form, development is the trailblazer, forging into uncharted territory and unearthing hidden gems. Ever the innovator, Dev values progress and change above all. This constant forward momentum is Dev’s greatest asset as well as its Achilles’ heel. The habitual urge to tinker, tweak and create can lead to frivolous or wasted effort. The results of this misguided effort can be seen in retrograde patch releases, hotfixes and other “band-aid” measures needed to undo the damage.

    On the other end of the spectrum, the operations mantra can be summarized with the idiom, “If it ain’t broke, don’t fix it.” Pragmatic to the core, stability and consistency are the guiding forces behind Ops. Change requests are usually met with a Treebeard-esque, “Don’t be hasty.” Taken to the extreme, this mindset can quickly lead to an environment running outdated, unpatched software and end-of-life hardware.

    On the surface, the differences between these two groups appear to be on the verge of irreconcilability. However, in this diversity the beauty of DevOps emerges as the strengths of each team balances out the weaknesses of their counterpart. Development is free to innovate and create knowing that their efforts will be guided by feedback from operations. Like stabilizers on a rocket, operations is able to focus on maintaining a reliable infrastructure with development providing the forward momentum. Working in unison, this duo forms a unit far greater than the sum of its parts. And with it, a company is left with an agile and stable launch pad from which to tackle the latest wave of IT challenges.

    Admittedly, the combining of development and operations into a single cohesive unit is just a piece of the entire DevOps puzzle. However, it is an important piece that is frequently overlooked. Understanding the forces and motivations at work within your teams is a pivotal step on the road to a successful DevOps transition.

    About the Author / Jared Perry

    Jared PerryJared Perry is a cloud engineer at iland, where he is responsible for maintaining the virtual infrastructure and a number of disaster recovery solutions. He holds a Bachelor of Applied Science in technology management and a Bachelor of Science in IT administration and management from Texas A&M University.

  • Continuous Delivery: The Atlassian Way

    Continuous Delivery: The Atlassian Way

    Sonatype’s Mark Miller (@TSWAlliance) caught up with Ian Buchanan (@devpartisan) for our 2016 DevOps Leadership Series. Ian discussed his experiences at Atlassian, including continuous delivery, ChatOps, and use of tools such as Bamboo, Nexus, Puppet and Datadog.

    https://www.youtube.com/watch?v=MgRaYKHGqeQ

    Buchanan: I’m Ian Buchanan. I’m a Developer Partisan at Atlassian, which is developer advocacy for our developer tools.

    Miller: Ian, most people know Atlassian from solutions like JIRA and Bitbucket.

    Buchanan: Yeah, I focus mostly on Bitbucket and Bamboo. Bamboo’s our continuous integration and deployment tool.

    Miller: Good. One of the subjects we talked a little about at dinner last night is how Atlassian is using Nexus. Can you give us some background on that?

    Buchanan: Yeah, that’s right. Well, we’re a big Java shop. We have been for a very long time and have a lot of Maven dependencies for external reasons; we also store a lot of our own internally developed libraries in our Nexus repository.

    _________________

     We’re a big Java shop. We have been for a very long time … we store a lot of our own internally developed libraries in our Nexus repository.

    _________________

     Miller: As far as a binary repository, are you using it for anything other than components?

    Buchanan: Not that I’m aware of. I know that we’ve recently started to get into Docker quite a bit and Nexus is a good fit for that as well. I don’t know the extent which we’re using Nexus for that purpose yet, as Docker’s still quite new for us.

    Miller: What does your continuous integration pipeline look like?

    Buchanan: It certainly varies by product in the details. Of course, we have our own continuous integration product, Bamboo. Bamboo is the most pervasive tool we have.

    For the Java stuff it’s pretty simple pipeline where libraries get built and published into Nexus so that they can be used downstream. We also have some interesting, newer cloud products that are built with Python. A lot of the deployments, whether they are Java or Python stuff, are handled by Bamboo’s deployment projects. They pull artifacts from Nexus and put them into production.

     _________________

    A lot of the deployments, whether they are Java or Python stuff, are handled by Bamboo’s deployment projects. They pull artifacts from Nexus and put them into production.

     _________________

    Miller: Are you guys using continuous delivery in house for your workflow?

    Buchanan: We’re in an interesting position that I think I see a lot of companies in. We have some behind-the-firewall products (our server products) and then we also have cloud products. In the cloud side, continuous delivery is used when we want to go very fast. But we also have to balance that with the approach we take for our server products so that they don’t fork terribly. We have continuous delivery up until there’s a product delivered. At that point, there is another kind of pick up—where the cloud products engage with other things like Puppet and Ansible to configure the environments with products that other people can pick up and run on premises.

    Miller: I was talking to your team in San Francisco. How are you guys using ChatOps with HipChat?

    Buchanan: I’ve been talking about ChatOps for quite some time. We do a lot of very interesting things with ChatOps. They play a very important part of our continuous delivery pipeline in that we publish build results in there. People can see when pull requests are ready and check those out. We get to see what’s happening from production as well. We have integration with Datadog; it tells us some of the monitoring things that are going on. It’s not just information coming into the chat room. There are also commands that folks can issue to make the continuous delivery pipeline move along. There are certain stages where you can type a command to our ChatOps and they will perform the necessary actions. Much of the deployment and change management happens in a ChatOps context.

    _________________

     There are certain stages where you can type a command to our ChatOps and they will perform the necessary actions. Much of the deployment and change management happens in a ChatOps context.

     _________________

     Miller: Anything coming up in the future you guys are working on that’d be fun?

    Buchanan: Well, a lot of the innovation really happens at our quarterly “Ship It” events. That’s where our developers spend 24 hours doing whatever kind of innovation interests them. A lot of time is spent scratching an itch they have. Some of the things that I saw there were about having more information coming back upstream, not just into the chat rooms, but into JIRA where more of the longer-term tracking is going on. They’re experimenting a lot with putting more information into JIRA.

    Miller: Nice. Final question, if you were going to be a superhero, would it be Dev, Sec or Ops?

    Buchanan: I’ve played Dev roles and Ops roles and I almost feel like I know those much more. But from a lot of the sessions at DevNexus, I have to feel like Sec is the superhero. They’re the superheroes because they solve some unsolvable problems in a lot of ways. But I don’t feel ready to play that role at the moment, myself.

    Miller: It’s interesting. I agree with you that they are unsung heroes. Most of the time they just get hammered for what they’re doing.

    Buchanan: That’s true. Yesterday I was in a session about securing REST endpoints. The speaker went through problems with BASIC off , DIGEST off and J2EE—all of these things which we use in our products. At the end we’re left with, “What is the answer?”

    Well, the answer is that for each of these, there are different problems. So we have to identify the right context. We have to understand where some things are broken, know that those things aren’t perfect, and design for that imperfection. That’s a mindset that we have to carry through dev, ops and sec. I hope more security folks step up and start telling those stories.

  • Jenkins 2.0 Advances Continuous Delivery with Native Pipeline Support and Usability Enhancements

    New major release accelerates software delivery with focus on pipeline-as-code, software pipeline stage views and usability essentials for new and existing Jenkins users

    SAN JOSE, Calif.–(BUSINESS WIRE)–The Jenkins project, comprised of the community of practitioners using open source Jenkins, today announced the availability of Jenkins 2.0. The first major release after 10 years and 655 weekly releases, Jenkins 2.0 addresses the demands of teams challenged with matching the pace of continuous software delivery while ensuring quality and providing visibility across the full software delivery process. Updates to core Jenkins capabilities include first-class support for continuous delivery (CD) pipelines, clear visibility of delivery stages and enhancements to help users get started more quickly.

    “These new features will ultimately enable us to implement robust continuous delivery process automation and deliver higher quality software faster than before.”

    Tweet this“As Jenkins user adoption continues to grow into millions of users, the community has responded with core updates to assure continued, long term success,” said Kohsuke Kawaguchi, creator of Jenkins and CloudBees CTO. “The Jenkins 2.0 release helps new and existing users implement continuous delivery with the most flexible way to model, orchestrate and visualize the entire delivery pipeline.”

    Jenkins 2.0 Updates

    Continuous Delivery Pipeline Support:
    The Pipeline plugins provide orchestration of all stages of the software delivery process, with enhanced visibility and feedback. Jenkins 2.0 has several updates to support continuous delivery pipelines.

    • Pipeline-as-Code enables teams to quickly create and collaborate on simple to advanced CD pipelines using code.
    • Stage View enables teams to visualize, collaborate and act on the application delivery value stream in real time (see Figure 1, above).
    • Pipeline Durability ensures that long-running pipelines withstand outages by resuming at the point of interruption, significantly improving productivity.
    • Multibranch provides automatic, zero-configuration of CD pipelines for new branches and pull-requests.

    Usability Enhancements:
    Jenkins 2.0 includes several enhancements that make it easier to use, and enable users to quickly be more effective.

    • Get started more quickly with community-recommended plugins: A new quick start UI provides solution-oriented plugin bundles. These are the most commonly-used plugins, selected from the more than 1,000 available plugins, to enable a fast start for new installations (see Figure 2, above).
    • Better user experience: Jenkins 2.0 includes enhanced configuration pages and other UI updates to make configuring and using Jenkins more efficient.
    • Upgrade confidently: Today millions of users depend on Jenkins, making backwards compatibility a primary concern. Jenkins 2.0 is designed as a drop-in replacement for Jenkins 1.x.
    • New community website: In response to the rapidly growing Jenkins community, the new Jenkins website makes it easier for new users to get started with Jenkins, encourages user contribution and increases visibility into Jenkins events.

    “By providing pipeline-as-code capabilities and visualization of software pipeline status within Jenkins, Jenkins 2.0 allows me to define both simple and complex pipelines,” said Shashikant Jagtap, DevOps and automation engineer, Jenkins user and lead for the London-based Jenkins Area Meetup. “These new features will ultimately enable us to implement robust continuous delivery process automation and deliver higher quality software faster than before.”

    Additional Resources:

    About Jenkins and the Jenkins Community
    Jenkins is the leading open source automation server supported by a large and growing community of developers, testers, designers and other people interested in continuous integration, continuous delivery and other modern software delivery practices. Built on the Java Virtual Machine (JVM), it provides more than 1,000 plugins that extend Jenkins to automate with practically any technology software delivery teams use.

    The Jenkins community advocates the use of Jenkins to the global development community, serving as a central source for tutorials, forums and other helpful resources for Jenkins users of all experience-levels. By recognizing the numerous contributors to the Jenkins project, the Jenkins project creates and fosters a community-powered infrastructure for maintaining and further developing Jenkins. You can follow the Jenkins community on Twitter (@JenkinsCI).

    Contacts

    PAN Communications
    Gaby Berkman / Sarah Grucza, +1-617-502-4300
    cloudbees@pancomm.com