devopsy.kinsta.cloud

Category: Enterprise DevOps

  • Principles Systems and Tools – What Shingo Can Teach Us About DevOps

    Principles Systems and Tools – What Shingo Can Teach Us About DevOps

    In my last post, The Power of a Game: What I Learned from Facilitating the Phoenix Project Simulation, we explored how DevOps and Agile practices are built on underlying Lean principles that go back decades to a time before the Toyota Production System. When you look into the history of Lean, it is almost impossible not to encounter the great thinker Shigeo Shingo. Shingo worked closely with Taiichi Ohno (the father of the Toyota Production System) to advance Lean thinking while building the Lean production system at Toyota.

    As someone who attempted to apply Lean to IT long before DevOps came into existence, I have always said DevOps is about creating an environment where it is safe to look deeply, question what is happening, think crazy things, experiment and learn based on outcomes. Ten years ago, I was invited by The Shingo Institute to serve as a teacher, coach and assessor. I had recently been awarded the Shingo prize for my book, Lean IT – Enabling and Sustaining Your Lean Transformation. What I learned from Shigeo Shingo’s writings heavily deepened my understanding of the role of Lean and flow in IT. These ideas are reflected in the Lean Enterprise Pyramid below and it’s amazing how effective they are with Agile and DevOps.

    Principles and IT Excellence

    A principle is a timeless universal truth that controls outcomes whether you believe in it or now. Think of it as a fundamental truth or proposition that serves as the foundation for a system of beliefs and behaviors. Principles govern consequences and outcomes–both positive and negative. “Create customer value” is a core principle in business success. If you don’t believe in customer value, or don’t deliver it, your prognosis is pretty bleak.

    In Lean IT, I introduced some key principles that drive IT success:

    Let’s begin with a working definition of DevOps—a mindset, tool set and set of practices that combine software development and IT operations to shorten the development/operations life cycle to deliver small iterative features, fixes and updates often and in close alignment with business objectives and value creation.

    Consider the Lean principles pyramid from a DevOps perspective:

    • Foundation principles build an intentional culture of trust, respect, focus and learning.
    • Behavior is how we show up and engage.
    • Perspective is a fresh new way of looking at IT work.
    • Flow is the outcome of effective DevOps in action.
    • Capstone of the pyramid fosters a regenerative culture of trust, safety, experimentation, collaboration and innovation.
    • Outcomes are higher quality IT solutions and services, in less time, with less cost with healthier teams.

    For DevOps, Agile and High-Velocity IT of any flavor to be successful, each of these principles must be addressed. You can’t address them all at once, so I suggest beginning at the base while taking steps to enhance flow. If you focus on these two elements, respect for people and flow, you will always be progressing in the right direction.

    Systems and IT Excellence

    Shingo defined a system as a collection of tools and tasks highly integrated to accomplish a specific outcome. The Shingo Model captures the important relationship between principles, systems, tools and results. It also supports three key insights which clarify how to achieve excellence in IT (or any organization). IT needs to get good and then continuously get better with its systems, tools and results (no kidding). Understanding how these elements are informed and reinforced by principles is essential to making this happen.

    The Insights

    1. Ideal Results Require Ideal Behavior
    2. Beliefs and Systems Drive Behavior
    3. Principles Inform Ideal Behavior

    For DevOps and Agile, the Shingo Model provides a logical sequence for change. Put another way, what are the key questions we need to answer?

    1. What do the ideal behaviors of DevOps look like?
    2. What beliefs do our people have today and what beliefs do they need to embrace DevOps? What systems (think: work systems, procedures, policies and technology) do we have in place today, what behaviors do they drive and what needs to change?
    3. How can our understanding of lean principles clarify our understanding of question #1?

    Take a moment and reflect on the role of principles and systems in your transformation. The more you ground your approach in the stability of principles, the greater your chances of success. If you or your organization does not truly believe in these principles, a conversation around which principles you do believe in would be a good place to start.

    — Mike Orzen

  • Fujitsu: State of Digital Transformation Is…Mixed

    Fujitsu: State of Digital Transformation Is…Mixed

    According to the Fujitsu Future Insights Global Digital Transformation Survey Report 2019, nearly 90% of companies in a recent survey are already deep into their digital transformation efforts. While the survey did find broad movement toward digital transformation, there existed wide differences in the progress of transformation across industries.

    If digital transformation progress were a horse race across industries, financial services and transportation would be running neck and neck. While, overall, companies reported 36% progress in their digital transformations, 47% of financial services firms and 45% of transformation companies said they have moved forward with digital transformations and have delivered positive outcomes. From there, the government reported 35%, manufacturing 32%, healthcare 31% and wholesale/retail 29%.

    Fujitsu’s findings are based on 900 respondents from nine countries. Interestingly, only 13% of those surveyed said their companies have yet to get started with their digital transformations.

    Not surprisingly, online businesses are moving along more quickly with their digital transformations than brick-and-mortar businesses. While online businesses comprised 33% of the survey respondents and brick-and-mortar 67%, online businesses have made more progress in digital transformation. Of the online companies, 91% are implementing a digital transformation, with more than half achieving positive outcomes.

    One of the more concerning findings from the Future Insights Global Digital Transformation Survey Report was the lack of trust toward online data. A considerable 70% of respondents found it difficult to judge if online information is trustworthy, and 59% were concerned that the data they use may have been falsified.

    “Concern about personal data is also high, with 72% of respondents saying that they are worried organizations exploit personal data without their permission,” the report said. “However, more than half of respondents are content to provide personal data to an organization if they can trust the organization and in cases where personal data is used to enhance products and services.”

    In addition to a lack of trust toward data, respondents are also unsure about AI-made decisions. “Sixty-three percent of respondents think that decisions made by AI are fair because AI is not subject to bias. However, 52% of respondents claimed they couldn’t trust AI because data can be incorrect or biased.”

    However, 60% of respondents said they would trust decisions made by AI if the AI shows substantial reasons for reaching the decisions, and 66% indicated they would trust a company that published a code of ethics governing the use of AI.

    “Companies in which management places an emphasis on long-term perspectives, practices empathic leadership by sharing their messages and passion with employees, and empowers their staff tend to achieve greater success in their journeys toward digital transformation,” the survey found.

    While respondents to this survey indicate moving forward toward their digital transformation objectives, as we recently covered in Digital Transformations Fall Short, Survey Claims, the Kony Digital Experience 2019 Index Survey concluded that businesses are “still missing the mark” with consumers despite nearly $5 trillion in digital investment. The report didn’t distinguish how the $5 trillion was being spent and in what areas of the business the investments were made.

    That report found that said organizations in banking (54%), retail (59%) and utilities (51%) are keeping pace with their expectations but are not delivering differentiated digital experiences. The big laggard was health care, with only 48% keeping pace and 20% being behind respondents’ expectations.

    — George V. Hulme

  • Relentless Automation: The Roadmap to Success

    Relentless Automation: The Roadmap to Success

    As digital transformation continues to shape organization-wide initiatives, CIOs are recognizing the benefits and value of automation in reducing errors and process variability, leading to lower operating risks and faster execution and innovation. In fact, according to Gartner’s recent report, by 2023, 40% of I&O teams will use AI-augmented automation in large enterprises, resulting in higher IT productivity.

    Despite this future uptick, many organizations are currently automation deficient as a result of complex infrastructure, fragmented task automation and tool sprawl. A renewed focus on automation will put I&O teams in the driver’s seat as cloud-like capabilities become the norm. To achieve the highest ROI, CIOs should audit their organization first in order to determine the level of need for automation as well as which specific areas to automate.

    Having a clear mandate is essential for any organization to be able to implement automation at scale. CIOs need to ask the right questions of their organization to determine a baseline for automation. These core questions include: Where is the organization experiencing business issues as a result of a lack of automation? What is preventing the organization from automating more? Does the organization have the right foundation of standards, people and processes on which to automate?

    From our experience, on the journey to mature automation, companies typically fall into one of following five stages:

    Stage Zero: Reactive

    Organizations in the reactive phase do not understand the demand for IT services and the importance of measuring IT service results. Their customers don’t have a well-established way of requesting services nor an understanding of the status of their request. Requests are obtained from various sources, completed inconsistently and process documentation is typically piecemeal. Difficult requests are often directed to individuals who have assisted with similar requests in the past, and tooling is procured by teams without strategy or direction.

    Stage One: Focused

    Focused organizations know the limitations of existing IT automation and actively create initiatives to improve service delivery on a small scale. In this stage, customers have a formal method of requesting services, but it isn’t fully followed or enforced. These organizations have limited processes and tools, and are still highly reactive, relying heavily on a specific individual’s knowledge. Automation is defined and repeatable with possible consistent results, documentation exists for all common services and there is awareness that tooling and coding is inconsistent.

    Stage Two: Established

    Organizations in the established stage are more proactive, and provide IT services with a formal approach to processing service requests with an automated triage capability. Workflow is measured for completeness, and tasks-to-complete services are well-understood, with a gap analysis on which automation has not been completed. In this stage, at least one primary IT service has been automated. There is an automation backlog with prioritization in place. Tooling is standardized and the process for code management is uniform across all teams. Though there is an agreed-upon vision for automation, integration into broader services, such as DevOps, are still incomplete or inconsistent.

    Stage Three: Accelerated

    Accelerated organizations actively deliver new services based on input from the business. Service levels and quality metrics are captured and improved upon, and IT has a catalogue of services available to all business units. Over half of the major IT services have been automated, and developers have integrated standard tooling into pipelines with defined outcomes. In this stage, organizations are focused on process execution and excellence, but KPIs are centered around IT, instead of the business as a whole.

    Stage Four: Optimized

    Optimized organizations have a mature and continuous service improvement program with an established cadence for enhancements. The entire suite of IT services has been automated and aligned with business initiatives, directly affecting business outcomes. Service offerings have various options to ensure automation is being utilized, there are teams in place to deliver applications and software, and IT services have comprehensive documentation and adoption of standardized processes with seamless integration.

    Achieving Relentless Automation

    A focused, cross-functional team should be assembled to effectively assess and define the automation roadmap for an organization. The cross-functional team will bring relevant perspectives to IT service offerings with the intent to automate. Once a baseline for automation is established, CIOs should monitor the progress of their automation efforts, including potential for growth and barriers the organization is experiencing within the zones of infrastructure, operations, application development, service delivery and enterprise service management.

    As CIOs begin to implement automation at scale, they will see reduced operational costs and time to deliver on critical, strategic products and offerings. As a result, organizations could recognize increased revenue, boosting their organization’s bottom line and driving them closer to total digital transformation.

    — Derek Wise

  • 3 Ways IoT Developers Can Make Their Applications More Secure

    3 Ways IoT Developers Can Make Their Applications More Secure

    When the IoT was still young, IoT application developers got away with making security an afterthought, as they built prototypes and minimum viable products designed to demonstrate the different ways the IoT could be used to transform the way we work, play and live. But the IoT has matured and grown to the point where today, Gartner estimates there were 8.4 billion connected things in 2017 and forecasts there will be 20.4 billion IoT devices deployed by 2020. With this growth has come a massive increase in number and sophistication of IoT security attacks, and the need for developers to make security not just a priority, but a top priority.

    As the IoT hacks of Trendnet’s SecurView camera, St. Jude’s cardiac device and the Jeep Cherokee have demonstrated, when security is not a top priority, successful IoT attacks can cause significant real-world financial, reputational and even human damage.

    While it is impossible to make IoT applications (or any type of software) 100% secure, there are several best practices that IoT application developers can adopt to strengthen the security of their IoT applications. These best practices include:

    1. Following a set of approved cryptographic standards when building applications.
    2. Using static code analysis programs that analyze application code to identify common coding mistakes that can lead to security vulnerabilities.
    3. Instituting a vulnerability management strategy that identifies and repairs known vulnerabilities in IoT applications on a scheduled, ongoing basis.

    These three best practices will not make IoT applications perfectly protected from malicious actors. However, they can significantly reduce an IoT application’s vulnerabilities and attack surface, making it much less likely a developer’s applications will be breached, generating angry customers and negative headlines.

    Follow Approved Cryptographic Standards

    Most developers understand that, if they hope to keep IoT data safe, they should encrypt this data, especially while it is in transit from device to cloud or vice versa. Yet, faced with tight development deadlines, as well as the need to make the most of their IoT devices’ limited bandwidth and processing capabilities, some developers choose to not encrypt their data or use weak cryptographic algorithms that promise to be faster or more lightweight than others but do not provide proper levels of security.

    Experience shows taking encryption shortcuts can result in customers’ data being stolen, an application disabled or other problems. For this reason, developers need to work closely with their security teams to encrypt IoT application data appropriately, following standards and guidelines approved by internationally recognized experts such as the National Institute of Standards and Technology (NIST).

    Choosing to use such approved standards does not have a major impact on IoT application development time, performance or functionality. NIST provides developers with guidelines on how to streamline integration of encryption standards into their applications. They also offer many different standards to choose from, so developers can find one that will provide their application with an appropriate level of security without sacrificing too much in terms of performance or functionality. Yes, it does require some additional work and effort, but if IoT developers want to make security a top priority, they need to make sure they are following approved cryptographic standards.

    Use Static Code Analysis Programs

    Many developers are familiar with static code analysis programs and their ability to discover common coding errors that can lead to bugs that affect software quality. However, these programs also can identify unsafe coding practices such as the use of improperly terminated strings, pointers to freed memory and buffer indexes allowed to go past the end of the buffer. These common coding mistakes create security vulnerabilities malicious actors can exploit in otherwise functional applications.

    There are a variety of static code analysis tools available, both proprietary and open source that developers can use to alert them to coding errors that could create security vulnerabilities. Developers should research which ones best support their security and other code analysis needs, and then ensure that throughout the application development processes—and especially before deploying an application—they use these static code analysis programs to discover any coding errors that might lead to a security breach down the road.

    Again, many developers might be tempted to skip this step, since the problems revealed by the program might prevent them from deploying and commercializing an otherwise functional application as fast as they would like. But, as the examples of IoT application hacks mentioned above demonstrate, cutting corners with security can often result in major problems down the road. If developers are dedicated to delivering their customers IoT applications with robust security, they need to commit themselves to using static code analysis programs to fix coding errors that might make their applications vulnerable to cyberattacks.

    Monitor For and Repair Known Vulnerabilities

    The media pays a lot of attention to so-called zero-day vulnerabilities—those that a software vendor has no prior knowledge about. However, the low-hanging fruit for most attackers is the ever-growing number of known vulnerabilities present in the millions of software components used in IoT applications that haven’t been fixed, either because the user of the component is unaware of the vulnerability or because they have not taken action to update their application to repair the vulnerability.

    All IoT application developers should be aware of the known vulnerabilities their applications might have as result of incorporating third-party software components into their applications, be they open source components (such as a Linux operating systems and associated software) or proprietary components. However, keeping up to date on all the known vulnerabilities in an IoT application’s device firmware, operating system and other third-party software components is not simple. The enterprises that have built these software components, government agencies and other watchdogs often identify as many as 1,000 different vulnerabilities in these types of software components per month.

    Fortunately, there are resources that can help developers stay up-to-date on new known vulnerabilities. For example, the National Vulnerability Database (NVD) is a public resource operated by NIST that catalogues known vulnerabilities in all types of software components, both proprietary and open source. To ensure their IoT applications do not have any known vulnerabilities, IoT developers can compile a security bill of materials (BOM) for their IoT application—a list of all the software components and versions used in the application—and then monitor the NVD for new known vulnerabilities with these components. Alternatively, there are a number of proprietary and open source tools that leverage free APIs to query the NVD database and then scan applications for vulnerabilities either by evaluating their components’ codebases or by discerning versions of exposed services (such as DNS, HTTP/S, NTP, SSH) on network interfaces.

    However, simply being aware of known vulnerabilities does not fix those vulnerabilities. Which is why developers need to not just check for new known vulnerabilities on an ongoing basis, but also have a plan to regularly repair new known vulnerabilities with security updates. Ideally such a plan should allow these updates to be performed remotely (over the air) and automatically. Today, many consumer IoT applications already include an option to enable remote, automatic updates. The commercial IoT applications built by many developers sometimes need more coordination to enable such remote, automatic updates—but this should be the goal. In any event, even if the updates can’t be done remotely or automatically, it is essential that IoT developers have some sort of plan to deploy security updates for known vulnerabilities in their applications in a timely fashion.

    Robust Security Will Enable Future Growth of IoT

    IoT is at an important inflection point. The market continues to rapidly expand, with exciting new agricultural, supply chain, industrial, public safety, consumer and other IoT applications being launched every day. Yet, additional high-profile IoT application breaches could threaten the tentative level of trust businesses and consumers have placed in the IoT, especially as it increasingly is used to collect sensitive data or complete critical tasks.

    To strengthen trust in IoT, developers need to dedicate themselves to making IoT security a top priority. Lip service is not enough–developers need to implement IoT security best practices, including the adoption of approved cryptographic standards, use of static code analysis programs to discover coding mistakes that can create security vulnerabilities and implementation of vulnerability management strategies that regularly identify and repair known vulnerabilities in their applications. If IoT developers make security a real top priority by implementing these and other IoT security best practices, the world’s trust in the IoT will grow and so will their business.

    — Larry LeBlanc

  • The SRE Pressure Cooker: Balancing Velocity Against Risk

    The SRE Pressure Cooker: Balancing Velocity Against Risk

    Delivering fast, reliable digital services today is a lot like Olympian alpine skiing. These services must deftly maneuver a series of perilous passages en route to end users, all while maintaining the astounding speed we now take for granted.

    In an SRE’s world, those passages are today’s increasingly complex and interconnected internet infrastructure through which they must deliver and support cutting-edge application functionality–all while rolling out updates at a relentless pace. It may appear easy to end users, but SREs are working incredibly hard behind the scenes, and as the internet grows more intertwined and treacherous, the risk for catastrophe increases.

    Judging from the results of our second annual SRE survey, striking this critical balance between supporting development velocity while maintaining site reliability is an onerous and stressful task. Here’s a look at some of the key findings, and takeaways for organizations increasingly reliant on this emerging role.

    The SRE Role Is Relatively New

    Even though Google coined the term in 2003, 64% of respondents noted their SRE disciplines have only been in existence for three years or less. This means many SREs and SRE teams—those professionals dually knowledgeable in software development and adapting IT systems to meet the needs of particular software—are still adjusting to their roles and responsibility. Many SRE teams are also small relative to the size and scope of the infrastructure to be managed, and they remain a scarce resource, with many organizations highlighting skills shortages in this area.

    IT Incidents Are the “New Normal”

    Almost half (49%) of survey respondents reported they had worked on a service incident over the course of the past week. In the month of June alone, Google—which is known for having a very mature SRE practice—had two significant outages: One that hit YouTube, G Suite, and several popular third-party apps relying on Google Cloud for their back-ends (including Discord and Snapchat), and the Google Calendar outage just a few weeks later.

    Catchpoint has detected a significant uptick in outages in recent months, likely the result of growing internet complexity which naturally increases the potential for problems impacting digital service reliability. This is a challenge for all companies and their SRE teams, including the biggest and the strongest—the last few weeks have shown us no one is immune. Organizations can help their SREs maintain a healthy perspective by bearing this in mind.

    Many SREs Don’t Have Clearly Defined Service Level Objectives (SLOs)

    This is problematic, because without SLOs, it is nearly impossible to identify what is an incident in the first place. Twenty-seven percent of SREs reported they don’t have any SLOs, and when flags are raised on everything, this leads to more alerts, a greater number of false positives, and naturally, greater SRE fatigue and stress. Our own conversations with SREs have revealed that even among those with SLOs in place, these are often unrealistic or not correlated to end-user satisfaction and happiness—for example, targeted at 100% availability (impossible perfection) when 98% availability may suit end users just fine. If you’re striving for perfection, every little thing that goes wrong constitutes an emergency, even if customers don’t notice. Setting unrealistic targets and not letting customer satisfaction define SLAs is a recipe for SRE burnout.

    Many SREs Feel Burnt Out and Stressed

    According to our survey, 21% of respondents indicated they never experience post-incident stress. However, it was interesting to note the answers to the following question, where we asked what types of changes SREs notice after an incident—in concentration, ability to sleep, mood and more. One would suspect if an SRE never experienced post-incident stress, he/she would select “none” for the second question—but that wasn’t the case. Rather, one third of SREs said they have noticed such symptoms. Sixty-seven percent of SREs who reported feeling stress after each incident, indicated they don’t believe their company cares about their well-being.

    We are aware of other surveys showing SREs are quite happy, particularly given their impressive compensation. While we don’t believe being an SRE automatically makes someone stressed and unhappy, organizations should never assume these professionals know and understand how appreciated they are. There is often a reluctance to discuss topics such as job-related stress, as evidenced by the fact that 50% of the SREs we targeted opted not to complete our survey—perhaps a result of SREs’ predominant hero culture. In addition to compensation, organizations should consider other ways to show SREs—whether they vocalize their stress levels or not—they care about their well-being, such as offering an extra vacation day or other perks after a particularly trying incident.

    SREs Could Benefit from Automation, but Don’t Have Enough

    Another thing we consistently heard in our conversations with SREs is they feel so busy putting out fires today they don’t get a chance to create a better tomorrow—in other words, they lack the opportunity to be proactive and preemptive in their approaches. A key factor in this is the amount of toil SREs face daily—meaning manual, repetitive, automatable, tactical work. Our survey found 59% of SREs believe there is too much of it in their jobs, and too little automation. Nobody strongly agreed with the statement, “We have used automation to reduce toil,” while almost half disagreed or strongly disagreed.

    Considering SRE teams are often small and the talent pool tight, it becomes critical for organizations to automate as much work as possible, including areas such as application performance monitoring. The SRE team at Alaska Airlines recently automated performance monitoring and deep-dive diagnostics, and as a result has been able to reduce their alert noise by 92%. Ultimately, this has freed up the team to focus on real issues versus wasting time investigating false positives. This has helped them bring down the average mean time to detection for real issues from hours to less than 10 minutes.

    Conclusion

    The stage has been set for SREs to experience significant stress, at a time when organizations need their exceedingly rare skillsets the most. As internet complexity increases, it is critical for organizations to continually nurture, encourage and motivate these valuable team members. This means understanding and empathizing with the intense pressures they’re under; having SLOs well-defined, realistic and meaningful; leveraging soft skills to communicate effectively and non-monetary incentives to show appreciation; and providing them with needed automation so they can focus on proactive improvements that advance the business.

    — Mehdi Daoudi

  • Aim to Be a Top 12% Global Testing Team

    Aim to Be a Top 12% Global Testing Team

    We know that you’re thinking, “Really? Another fluff post about the top people in my profession and what I need to do to be just like them?” Listen, we hear you, but this is not some fluff piece with vague ideas of what might (or might not) work.

    In case you missed it, on May 27th, the 2019 State of Automated Visual Testing was released. Based on independent research sourced from over 350 testing teams around the world, we learned 12% of you are getting much better results than the other 88%. We’re talking four times more successful as measured by the things you (and your boss) really care about–test coverage, release velocity, application quality, overall R&D teamwork and cold hard cash. This is not our opinion. It’s not subjective. It’s objective data and information. Data and information that came from you and your peers.

    In other words, we’re not asking you to take our word for it; we’re asking you to take your word for it.

    What Separates the Top 12% from the Other 88%?

    Digital Transformation. Two words that have been baked into our world over the past 20 years. To such an extent it may be fair to call it fluff? Fluff or not, IDC forecasts worldwide spending on technologies and services that enable digital transformation will reach $1.97 trillion in 2022, per the (IDC) Worldwide Semiannual Digital Transformation Spending Guide.

    That’s a lot of fluff! When your bosses’ boss is spending that kind of cash, it’s always worth paying attention. It could be good for your career. As it turns out, 12% of the world’s testing teams did pay attention, and it’s their approach to managing the challenges of digital transformation that have set them apart.

    The Testing World’s Digital Divide – Digital Transformation Quantified.

    You can quantify it for yourself with some simple math and see how you compare to other R&D and testing teams around the world. Got that calculator ready? Here we go…

    How many applications do you have in production today? (Don’t forget those native mobile apps, they really add up).

    How many pages or screens do you have in production on average for each of these applications? (Single page applications can be tricky but give it your best guess).

    How many viewport breaks do you support? (The market average is six if you are not sure).

    How many human languages do you support? (We’re talking about localization here–English, German, Spanish, Chinese, Japanese–not coding languages in case you’re wondering).

    Now, multiply those four numbers together. Congrats! You have just quantified the digital footprint of your business. People can write about digital transformation all they want, but you’ve just transformed all that fluff into something quite real. It’s probably a big number. Over 90,000 for a typical company and over 624,000 for the largest 30% of companies in the world. Here’s the kicker. You and your R&D team are responsible for managing the visual and functional quality of every single one of those pages and screens.

    When functional test automation first emerged way back in 1989 with the launch of Mercury Interactive XRunner, it was a much different time. Browsers didn’t even exist. Since then, the browser wars have come and gone, and standards are now in place. Applications have grown far more complex with native mobile, dynamic content, single page applications (SPA) and responsive design now an everyday reality. Digital transformation for any size company is officially past tense. We’re not transforming, we’re transformed. And now you have to deal with it, but how?

    What Defines a Top 12% Global Testing Team?

    Top 12% teams have overcome the massive challenge posed by digital transformation. Like any technical challenge we have ever faced, it started with someone on the team who felt the pain and set out to solve for it. And when they did, good things happened:

    • Test coverage increased by 60%
    • Release velocity became 2.8x faster (even though coverage increased 60%)
    • Visual and functional quality improved 3x with far fewer escape bugs (even though they release 2.8x times faster)
    • R&D teams are 4x more satisfied with their visual and functional quality outcome

    All of this despite the 12% of testing teams were managing applications 2.2x larger than the other 88%. In all likelihood, they felt the pain before most of us, and have now led the way for the rest of us. We just need to follow.

    Today, successful continuous management of application visual quality creates competitive advantage. Business leaders know this and are paying attention. As a result, testers are in a better position than ever to be heroes respected by their R&D teams for driving huge value for the companies they work for.

    — James Lamberti

  • Top 5 AWS Security Mistakes: Leaky S3 Buckets

    Top 5 AWS Security Mistakes: Leaky S3 Buckets

    As we get ready to discuss our list of the Top 5 AWS Security Mistakes in the upcoming DevOps.com webinar, we wanted to provide a preview of the type and depth of information we’ll be discussing. Since the most-talked about, and likely the most vulnerable, aspect of AWS security is inevitably those leaky S3 buckets, it’s a given we’ll tackle S3 first.

    In this article, we’ll dig into S3 buckets, explain how they become vulnerable, cover eight different ways they can inherently become open to the public and how to find and protect any public facing S3 data.

    The S3 Bucket

    S3 is one of the oldest services in AWS—so old that parts of it still support XML-based policies instead of the JSON you see everywhere else. S3 also has a lot of features that aren’t as commonly known or used anymore, such as allowing someone else to host put objects in your bucket yet still maintain ownership of them.

    Think of an S3 bucket as a server that can have subdirectories and files. Because S3 buckets can have a different root url (e.g. “dops.s3.amazonaws.com”) and have their own life cycle and settings, we tend to think of buckets as servers, not directories. Also, directories in S3 allow you to organize objects, but don’t have any distinct settings of their own, other than a name.

    AWS has a wide range of use cases it needs to support, which is why we have all these mechanisms. Sharing publicly, within an account, between accounts, hosting websites and so on, creates complexity. The good news is S3 always defaults to secure and private. The bad news is AWS allows human beings to use it (and therefore weaken security) and can be confusing to manage. For example, S3 supports both read and write permissions (and list, for buckets). It’s possible to have no public read but public write permissions, which could lead to people placing bad files in your directories.

    8 Ways AWS Data Becomes Public

    There are at least eight different ways an AWS S3 bucket can inadvertently become open to the public and be exposed to a data breach.

    • Bucket ACLs (Access Control Lists): This is an XML document that defines the first layer of access. By default, only the account owner has access, but this can be opened up to other AWS accounts or the public at large. Amazon recommends against using these at all, but it’s also the easiest way to make something public quickly, so we see it all the time.
    • Bucket Policies: These are super-flexible JSON policies that allow you to set things such as IP-based and other conditional permissions on a bucket. While this should be the primary way for managing public access, ACLs are the first tab and about three clicks to make something public. It’s this flexibility that leads to mistakes such as opening up to a wider range of IP addresses than intended or using a negative to block access to some IPs but inadvertently granting access to everyone else.
    • IAM Policies: These are the normal IAM permissions you use to control access throughout your AWS account. You can’t make a bucket public with them, since they only govern AWS users, but you can open up access to the bucket by authorizing another AWS service access, and then that service exposes the content.
    • Object ACLs: These are the primary object-level controls. It’s just like a bucket ACL and uses XML to allow access from other accounts or the public. Objects do not necessarily inherit bucket ACLs. You can totally make an object public even if the bucket is private.
    • Explicit IAM or Bucket Policy Statements: IAM policies and bucket policies can have explicit statements referencing the object, which will override the object ACL (if you own the object), since they are evaluated first.
    • Pre-Signed URLs: These object-level policies must be created using code (not the console) and provide temporary access for anyone with the URL. It’s used, for example, to share files for a few minutes or an hour for someone using your app to download a media file.
    • CloudFront Origin Access Identity: CloudFront is the AWS content delivery network and can serve as the front end to S3. You can create something called a CloudFront origin access identity to write an IAM policy that allows CloudFront to access the S3 content. If the CloudFront allows public access, it can access the S3 content and that won’t show in a bucket policy or ACL.
    • Cross Origin Resource Sharing (CORS) Policies: CORS is required if you use S3 in a website and don’t want the browser to break due to same site security settings. CORS in S3 won’t override an ACL or bucket policy but could mask public access in limited situations where the data is exposed in the web code through the authorized site.

    All put together, AWS evaluates IAM permissions first. Within those, the only one to make a bucket public over the web is the CloudFront Origin Access Identity. AWS then turns to the bucket ACLs and policies for any explicit deny statements. Then it looks at the object ACLs for public access. If you make an object public and the bucket policy doesn’t have an explicit deny, it is still public, but otherwise a good bucket policy will block everything. Besides, a lot of the problems are open bucket ACLs, not object ACLs.

    4 Ways to Find Your Public S3 Buckets and Protect Your Data

    What’s the best way to find public buckets? It’s relatively easy.

    1. Log into the console, click on S3 and look for the Public tag. AWS uses some advanced back-end math to evaluate all the bucket policies to figure out if something is public, which catches most of the fringe cases, but it does not show if the bucket is private and objects in it are public.
    2. Check CloudFront to see if you use any origin access identities. It’s easier to look here than in all your bucket policies (usually). This saves you from looking through object ACLs as well.
    3. Check any CORS policies, if you use them. This is the least common situation and I even hesitated to complicate this post by mentioning them.
    4. The bad news … there is no easy way to find public objects. Even finding them programmatically can be difficult if you have a large number. Also, changing a bucket ACL doesn’t cascade to the object ACLs so you need to run through and fix them one by one. However, if the bucket is locked down the attacker would need to know the full URL/name of the object to find it. It’s not perfect, but it’s better than nothing.

    That’s it. Although, if you know of any other fringe cases, please don’t hesitate to let us know. While conducting assessments, we find well over 90% of the problems are just basic bucket ACL and policy mistakes. Object ACLs are also an issue, but slightly lower risk since you need the exact URL to find them if your bucket is otherwise secured properly.

    If you are curious about the other 4 Top AWS Security Mistakes, sign up for the Aug 14 webinar.

    — Mike Rothman

  • Software Compliance Teams Can Learn a Lot from DevSecOps

    Software Compliance Teams Can Learn a Lot from DevSecOps

    Many argue that application security should be the responsibility of a security team. However, while security professionals can contribute, developers are usually the only ones with the technical ability to fix software security vulnerabilities. The same goes for software compliance. When it comes down to it, only developers are equipped to build applications in compliance with specific software standards.

    Considering the tight deadlines for development teams, giving them additional duties can present challenges. To help developers deliver secure applications without slowing them down, many teams have adopted DevSecOps—which encourages integrating automated security testing into the DevOps workflows for every release.

    DevSecOps helps teams deliver more secure software faster, but what about more compliant software? Many teams trying to achieve software compliance encounter similar challenges as those trying to achieve software security. Luckily for them, they can use similar strategies.

    What Does DevSecOps Do?

    The fast, iterative software release cycles supported by DevOps workflows present hurdles for software security.

    In traditional approaches to securing software, testing occurs toward the end of the software development life cycle (SDLC), after an application’s construction. As a result, most (if not all) of the application is tested at once, and a long, daunting list of security issues is sent back to developers.

    However, many teams find this methodology incompatible with DevOps. Development teams don’t have the time—not to mention the budget—to stop whatever they’re doing to sort through a massive spreadsheet of issues.

    To solve this problem, organizations can implement DevSecOps by integrating security testing into the shorter, higher-frequency feedback loops of fast DevOps release cycles. Rather than running one big test after developers finish building an application, DevSecOps calls for automated security testing performed early and often. By helping developers check in more secure code, DevSecOps helps reduce the volume of issues QA must identify and send back to them.

    If development teams can integrate security testing into DevOps workflows, why not compliance testing?

    How to Apply DevSecOps Practices to Software Compliance

    While testing for security usually requires a different analysis than testing for compliance, development teams practicing DevSecOps can use a similar approach to compliance testing.

    Like traditional application security testing, compliance testing typically happens in the QA environment. A very rough application development and testing workflow looks something like this:

    1. Developers write the code for an application.
    2. QA tests the code while developers move on to another project.
    3. QA sends a list of compliance violations back to developers, requiring them to pause their current work to fix those problems.

    There is nothing technically wrong with this compliance testing strategy. However, this approach is frustrating for developers and costly if the list of issues is especially long. To resolve more compliance issues before an application is complete, teams can implement practices inspired by DevSecOps to help developers commit compliant code earlier in the SDLC.

    One of these strategies is to integrate automated compliance testing into DevOps release cycles. By testing the application in increments rather than all at once, teams can reduce the number of compliance violations per testing cycle. While it obviously requires higher-frequency testing throughout the SDLC, this approach has proven to be faster than waiting to find issues in QA.

    Another approach is to create a sandbox environment in the integrated development environment (IDE) where developers can run tests themselves. Testing their own code as they write it equips developers to check in cleaner, more compliant code, so compliance violations are less likely to show up in QA—or worse, production. This approach has the added benefit of familiarizing developers with code that leads to compliance violations.

    These strategies for identifying and resolving compliance violations are not meant to replace QA testing. However, by integrating compliance testing into multiple stages in the SDLC, teams will see fewer issues in feedback loops between QA and development—which better supports DevOps workflows.

    What Does This Look Like in the Real World?

    As with security testing, integrating compliance testing into DevOps release cycles requires new technologies and workflows, each with a different learning curve.

    Regarding technology, static analysis is growing as a popular option to help development teams build compliant applications. Coverage varies between static analysis tools, but in general, the technology can find issues in code quality standards, such as MISRA and CERT C/C++, as well as security standards, such as OWASP Top 10 and PCI DSS.

    The static analysis tool or any other technology that teams choose to address, software compliance plays an important role in the workflows they adopt. As discussed earlier, those hoping to keep issues out of QA will want to find solutions that can:

    1. Integrate into the quick feedback loops between testing and development.
    2. Scan code in the IDE as developers write it.

    Creating a more continuous cadence of feedback between testing and development could represent a significant shift in a developer’s day-to-day.

    While “DevCompOps” doesn’t quite have the same ring, development teams in regulated industries will find cost-cutting benefits by integrating compliance testing into DevOps release cycles.

    — Charlie Klein

  • Game On: Gamification for the Win

    Game On: Gamification for the Win

    Staying competitive in today’s fast-moving digital economy is easier said than done. Operating in a digital environment requires a significant shift in company culture and an agile software environment that accelerates time to value. Creating a DevOps environment helps companies keep pace, but it requires a workforce that is empowered, excited and motivated.

    Gamification offers a unique opportunity to continuously engage your workforce by applying game-design elements and game principles into a corporate setting, making work more fun. Companies can develop a team-oriented Pod model that incorporates gamification to improve employee engagement and productivity, enable team members to work more efficiently, accelerate the learning curve and deliver more sophisticated products to market faster.

    When I joined Cognizant Softvision as CEO, I wanted to empower our teams and promote a continuous delivery mindset by developing a delivery model that was customer-centric, differentiated and unique to each client without being prescriptive. Thus, Game of Pods was born.

    The global, Game of Pods gamification program is designed to provide timely and superior customer service. At a high level, Game of Pods inspires and measures team performance and delivery while encouraging and monitoring individual performance. During competition, the Pod and its members are recognized and rewarded for their collective and individual contributions with badges and trophies.

    We typically focus on the output of each pod, rewarding collective behaviors and achievements that help the pod reach higher efficiency and quality targets. Collective badges are awarded to the pod while personal and peer-to-peer badges are awarded to the pod members. Pod badges are earned as a team and are focused on the health of the project, performance of the pod, quality of delivery and alignment with client objectives. A leader board displays the badges each pod earns, their experience points and their trophies. Pod and individual recognitions are celebrated over email and across Facebook Workplace, and shared with all employees.

    While perhaps not as intense as “Game of Thrones,” Game of Pods is the epitome of gamification in action and enables organizations to reach a higher level of efficiency and professionalism while challenging employees to achieve personal goals in a fun, team-oriented process. In addition, Game of Pods offers other benefits such as:

    • A flexible model that can work in any client environment.
    • Team empowerment that incentivizes teams and individuals to achieve more.
    • An approach that encourages diverse thinking that recognizes and rewards teams and individuals for improvement.

    Game of Pods is most effective in organizations that embrace an agile mindset and understand the need for flexibility in today’s fast-moving environment. While gamification has come a long way, successfully incorporating it into the workplace can be challenging. However, integrating gaming into your organization can empower and excite employees, motivating them to overcome obstacles that might otherwise stand in the way of success and sustainable innovation.

    — Andres Angelani

  • Is DevOps in the Enterprise Real?

    Is DevOps in the Enterprise Real?

    Is DevOps in the enterprise real? The short answer is yes, but you may assume it’s only relevant to the world’s leading cloud companies because they’re the ones that make the headlines. DevOps offers the promise of agility, quality and ultimately better-performing software. While DevOps processes can increase agility, and agile approaches are a great help in improving responsiveness to change, the overall process still needs a clear objective, measurable KPIs and a finite scope.

    In other words, unlocking the promise of an agile (product) organization requires a few complementary approaches. If those are ignored or applied incorrectly, DevOps agility can become an impediment to adaptability and agility for your enterprise.

    Yet, I’ve read so many blogs about how the cloud and DevOps processes solve problems. It’s true they can, but the nuance is that, while a few teams can coordinate change amongst themselves, when you attempt to deploy a DevOps practice at scale, challenges are quickly apparent. For example, if I’m working for a startup that has fewer than 50 developers and very few layers of management between the CTO and her team, the management and communication lines are likely short enough that strategy and direction can be conveyed directly to the teams. However, for larger organizations—where there are many often geographically distributed teams with silos of responsibility and layers of management—far more coordination and communication are required.

    In these environments, it is more likely the business will achieve agile teams but not an agile culture. This is because it’s too simple and, therefore, commonplace for these environments to execute in silos not aware of changes to other silos and where environmental constraints and dependencies are not considered throughout the development process.

    The best solution to these challenges is to focus on a cascading framework of objectives and goals. Every company has its mission-critical goals—some call them strategic imperatives, others call them (or include in them) board-level mandates—and those goals should cascade into a more refined product, program and team goals as you traverse the management tiers. This ensures the business leaders can set a strategic direction that will be measured in a certain way, and every subsequent layer below them understands how their initiatives align with supporting that direction.

    The first critical element for building a successful, sustainable DevOps culture is understanding your goals and how you’ll define and measure success.

    The next element is to establish which decisions each level of management can make autonomously. For example, risk tolerance is a strategic decision, architecture and security posture are typically defined for a product, platform-level technology decisions can be delegated to program teams and project teams can then focus on making the more granular technology decisions that will differentiate their product, service or feature.

    Finally, communication is key. This sounds trite, but communicating this cascading decisioning framework, exposing the decisions each group makes that could impact the particulars of an environment and then discussing requirements and challenges of architecture and software choices can ensure that each team, at each level, can remain agile and adapt to change effectively.

    The question is: Do you have the management rigor to dig into these challenges and own them? The answer likely lies in how important the pursuit of agility is to you and your business.

    — Ari Weil