devopsy.kinsta.cloud

Category: DevOps Practice

Government Has a DevOps Lesson for Private Sector
The missions of government organizations and DevOps are well-aligned, according to F5 system engineer Scott Van Kalken, who works with many organizations at various levels of government around Australia.

That’s because government organizations provide services to the community, and DevOps is about providing services to a community of users.

He sees more use of DevOps in the government sector than in corporations.

Increasingly, DevOps is embedding security from the start of a project. “That’s really exciting for me,” he said, as it shows DevOps is working and allows faster iteration with security issues being picked up earlier.

For example, the COVID-19 pandemic has resulted in more people working at home. That makes security considerations even more important than ever.

All levels of government have seen massive increases in remote working capacity and functionality. Applying DevOps principles to security practices has allowed them to adapt rapidly to those increased levels of working from home.

One large department took just three or four days to go from 1,000 to 20,000 people working from home. Van Kalken said this was possible partly because they “automate absolutely everything they are doing,” allowing rapid implementation of security policies. They quickly stood up additional cloud infrastructure with the right security settings by using DevOps practices. Similarly, DevOps teams can use APIs to easily and quickly apply security policies to SaaS applications.

Cross-disciplinary Cooperation and Coordination

One government department that Van Kalken is involved with was new to DevOps but very good at getting representatives of the relevant groups into the same room. They all focused on reaching the desired outcome with “no turf wars” and agreed that establishing appropriate security policies was part of that process.

In his experience, the private sector tends to be more siloed, which can make it more difficult to make the right security-related choices from the outset.

For government organizations with more mature DevOps practices, “it’s just an incremental ramp-up,” he said. The collaboration between the various functions makes a big difference to successful outcomes.

So the role of the security organization changes from being akin to a nightclub bouncer (“No, you can’t”) to that of a partner in the delivery of services.

The private sector can learn from this, Van Kalken suggested. Government organizations have a “laser-like alignment” on service delivery. In contrast, the differing and competing objectives of various departments or functions within corporations may be a handicap.

Most organizations have adopted one of the published security frameworks, but project teams have the responsibility to decide how to comply with them. Government entities are different, he said, in that they must at a minimum meet the requirements of the Australian Government Information Security Manual (ISM). That leads to more alignment between departments and functions within the organization.

DevOps and Security Hygiene

From a more general security perspective, “governments are targeted in various ways” as they increasingly make services available online, he said.

“Government is no different to every retailer, every bank, and so on,” in the sense that all the usual threats apply, said Van Kalken. But some players specifically attack government organizations because of the types of data held about individuals. Fortunately, DevOps practices assist in responding rapidly to changing situations.

As a simple example, organizations can quickly deploy new policies to distributed firewalls by using DevOps principles including automation. That gives skilled individuals more time to deal with any exceptions that occur.

Organizations can apply similar practices to improve security hygiene by more quickly rolling out new versions of commercial software.

“That element of speed is something very new to the security space,” he said.

Recommendations

Van Kalken’s recommendations are:
- Get all the right people—including security—together in the project team from the outset.
- Ensure they all understand the objective.
- “All swim in the same direction.”
- Make sure everyone continues to work collaboratively.
The rest “will just happen if you do these things,” he said.
August 11, 2020
How to Run a Remote GameDay
For those unfamiliar with the term, a GameDay is a dedicated day for running chaos engineering experiments on our systems with our team. The goal is cooperative, proactive testing of our system to enhance reliability. By getting the team together and thinking through your system architecture, you can test hypotheses and evaluate if your systems are resilient to various kinds of failures (and if they are not resilient, fix them before those weaknesses impact customers).

But how can we have productive GameDays in a distributed, remote context? This article highlights the differences that exist between running a GameDay with everyone in the same room at the office or data center versus running a GameDay with everyone working remotely. This post is meant to be a resource now that we’re all working from home as a result of COVID-19. First I outline the differences, then I talk about how to be successful despite these differences, and in some cases enhance your potential for success because of them.

What Makes a Remote GameDay Different?

Because all of the participants are spread out, communication will be a challenge at first, especially for teams that have little to no experience with it. Learning how to hold standup meetings, communicate about system changes and updates and coordinate work such as deployments will be a challenge.

Distractions can also be a big deal. When everyone is locked in a conference room it’s easy to run a distraction-free GameDay. It’s more difficult when we are remote and have kids and pets running around, FedEx deliveries and random family noises. Have a plan to mitigate the best you can. Some find headphones helpful, while others move into the basement or a bedroom and lock the door. Find something that works for you.

You also need to think about the risk of someone’s internet connection going down or their PC/laptop crashing. Create alternate ways to dial in to conference calls. Make sure every team member has a backup team member standing by ready to take over that person’s role if needed. When a required role is being performed by someone remotely, there is always a chance their internet connection could go down. There needs to be a contingency plan for someone to take over their role if that happens. This can be a team member who already has a role and is capable of taking on another concurrently or someone who was only participating as an observer.

For teams that are remote-first and have been working this way for a long time, much of what we outline here may be old hat. But it can be easy to fall into bad habits for anyone, so if your team was used to running GameDays weekly in the office, it’s important to keep that muscle memory even in the new paradigm.

How to Prepare for a Remote GameDay

Here are the ways you must prepare.

Define

Define the GameDay. Specify the exact chaos experiments that the team will run, and exactly which systems the team will target. Define the goal. Is it to:
1. Replay/reproduce circumstances that lead to a past outage?
2. Retest something that caused a problem in a previous test that you believe you have fixed and want to confirm?
3. Ensure that a new system has the proper monitoring set up and working with alerts and metrics?
4. Something else?
Time

Set the date with a beginning and end time. This will likely require coordination with people. Be sure to account for time zones if your team is spread out nation- or worldwide. Set a clear agenda for the GameDay that includes:
1. Assigning which tasks will be done by which team member and the timing involved. This orchestration is especially useful with team members working from a distance because it frees up communication channels for use in dealing with problems or for the leader to give direction without interruption.
2. Whiteboarding of the current system architecture to paint a clear picture of what you are going to attack and what you expect might be impacted. Doing this remotely requires the use of some imagination and forethought. One option is to use a diagramming app while sharing your screen over a video conference.
3. A short team debate on assumptions to get everyone on the same page.
4. Clearly defined chaos experiment test cases and scoping to limit test magnitude, length of time and blast radius. To prevent problems, make sure you have a halt or abort plan and can stop the experiment easily if unexpected problems arise.
5. The execution of the chaos experiment. Ask questions like, “What changes are observed in monitoring data?” and “Is this the behavior we expected?” Collect and document the information.
6. A recap at the end to discuss what worked, what didn’t and how to use what was learned.
People

Identify which team(s) will participate and make sure everyone on those teams is ready. Make sure all stakeholders are available and will be present in the appropriate communications venues.
1. Identify the incident manager on-call (IMOC).
2. Define the on-call rotation.
3. Notify everyone on the on-call team(s).
Comms

Set up dedicated, real-time communication venues for the duration of the event. This is where all activities will be coordinated and where all reports will be sent.
1. Create a dedicated instant messaging channel using Slack, Hangouts, Skype or whatever your company currently uses. Test and make sure all involved are invited and can access the channel.
2. Create a dedicated video conference using Zoom, BlueJeans, Webex or whatever your company currently uses. Test and make sure everyone involved can access it.
Final Steps

The agenda will be the same as in an onsite GameDay. Communicate it in advance. Include information about the schedule, the activities and the roles, specifying who is responsible for what. Typically two hours is more than enough time for a bit of an introduction, a discussion of assumptions and the day’s test design.

During the introduction, the designated test leader will make sure everyone understands what will be tested, the parameters for how the blast radius is set, the magnitude of the attack to be run and the expected outcome. Another team member is assigned to monitor the metrics. Others may be assigned to monitor other systems that may end up being involved.

Communication is key, especially because if the test causes unexpected problems. Everyone needs to know immediately so that the test can be stopped and any damage dealt with right away. It’s actually a good thing if you find unexpected issues because then you can fix them.

It’s also a good thing to take time to celebrate when an experiment yields an expected result. Rejoice together that your team’s implementation of self-healing or recovery mechanisms kicked in and worked as expected. Allow yourself to take the opportunity to call out the immediate wins and not just the issues you find.

Conclusion

Running your first GameDay can be intimidating, but with good preparation, any team can be successful. Doing the same thing with everyone working remote can be even more intimidating. It doesn’t have to be! Keep up good habits and set clear agendas, have concrete start and end times, go in with a plan and execute. Your customers will be happier if your team is figuring out problems before they ever had the chance to experience them.
August 10, 2020
Sumo Logic Expands Observability Platform Scope, Reach

Sumo Logic announced today it has extended the reach of its monitoring platform to provide deeper insights across application and infrastructure logs, metrics, traces and metadata in real-time.

The latest additions to the company’s suite of software-as-a-service (SaaS) offerings include Sumo Logic AWS Observability Solution, which focuses exclusively on AWS services, and additions to the Sumo Logic Software Development Observability Solution that provide support for GitHub, Jenkins and PagerDuty alongside existing support for Jira, Bitbucket and OpsGenie. The Sumo Logic Software Development Observability Solution is based on a key performance indicator (KPI) methodology developed by the DevOps Research and Assessment (DORA) organization, an arm of Google.

In addition, Sumo Logic has launched a closed beta for a Distributed Transaction Tracing service based on the OpenTelemetry standard that will integrate logs, metrics and metadata. Finally, the company has added more tracing capabilities to its existing Microservices Observability Solution.

Sumo Logic got its start by providing a cloud service for analyzing logs as an alternative to Splunk. Since then, the company has extended the scope of its services to address a wider range of observability requirements for DevOps teams.

Ben Newton, director of field evangelism for Sumo Logic, said the company is trying to engage IT organizations at any level of maturity by providing access to a range of offerings from log analytics to observability tools for various platforms including Kubernetes. Those offerings now span the application life cycle from coding to grave, he said.

Sumo Logic also differentiates a pricing model based on data tiers that enables IT teams to analyze logs, metrics and traces across a variety of use cases without monthly limits, peak provisioning or on-demand charges for data volume overages or exceeding custom metrics limits.

In the wake of the economic downturn brought on by the COVID-19 pandemic, the rate at which DevOps tools have been shifting into the cloud has accelerated. With more DevOps teams working from home, cloud-based services are more flexible. At the same time, more organizations prefer to treat IT costs as an operating expense at a time when the overall economic outlook remains uncertain. The entire IT organization needs to be able to respond to rapidly changing business conditions in a more agile manner, noted Newton.

Less clear is whether the economic downturn might also drive consolidation across what today is a spectrum of observability tools spanning log analytics to application performance management (APM) platforms used primarily to sample application traffic. It’s even less clear whether the rise of distributed tracing capabilities might reduce dependency on application sampling altogether.

What is apparent is IT teams have never had more visibility into IT environments that increasingly are more complex. The challenge and opportunity now is determining how best to apply a wide range of observability tools to turn all that insight into actionable intelligence.

August 6, 2020
Serverless: Helping the Enterprise Scale DevOps

Serverless is the paradigm shift that allows engineers to innovate and move faster

DevOps has been around for more than 10 years and it’s still misunderstood by many companies. You can’t buy “a DevOps,” but DevOps is critical to your business. In some ways, serverless is similar: a name that markets well but creates confusion and misunderstanding in the industry.

Like DevOps, serverless is a game-changer. Some say that DevOps is no longer required, this is not true. Serverless has enabled Fortune 100 global insurance company Liberty Mutual to scale the advanced DevOps practices required for a modern enterprise with a global reach. Serverless has been the paradigm shift that enables acceleration.

Liberty Mutual started its cloud journey around 10 years ago, the same time the DevOps discussion started within the engineering teams. Two very important concepts came around over the next few years: For cloud, we should build a foundation on cloud-native, not just “someone else’s data center.” And for DevOps, we should think about CALMS (Culture, Automation, Lean, Measurement, Sharing) as opposed to writing scripts. Both of these concepts started a collision course that would eventually result in launching in a serverless-first strategy for our organization.

One of the key concepts of the cloud strategy is infrastructure as code. Back in 2015, it was a pretty tall order to teach 1,000-plus software engineers how to provision secure and resilient infrastructure using YAML. Of course, some teams will excel, but it was a very steep learning curve for many. Then adding on more advanced requirements such as secure by design, chaos testing, resiliency, observability, performance optimization … phew! However, once teams evolved to creating serverless systems, something really interesting happened: Teams started to create self-healing workloads, blue-green deploys, hit high DORA metrics (throughput and stability) and deliver exceptional business value at low compute costs.

Serverless Considerations

For any business starting its serverless journey, the following principles should be considered. These key building blocks will help teams anchor by focusing on business outcomes and addressing customer needs. Enable teams to explore cloud technologies but help them avoid being lost in it—inertia can kill an organization.

Team Outcomes Are Measured in Business Outcomes

Product thinking, North Star metrics and impact mapping are extremely useful to help teams focus on clear outcomes. What is the most important feature we can deliver next? Yes, it is that simple.

Code Is a Liability

For 40 years, we have known that you cannot measure engineering success by lines of code written. Yet, we sometimes have a sense of pride at the high LOC (Lines of Code) counts. Encourage engineers to create systems with as little code as possible. Managed infrastructure that is event-driven and correctly decomposed will not need as much code. This is a fact. You can’t code your way out of complexity—be pragmatic.

Serverless Is a Spectrum

Start with serverless first [Fig. 1] and then work your way back to IaaS (infrastructure as a service). Insist on high engineering standards. Measurement is difficult in modern software development—it should be based around trends, not absolutes. The “four key metrics” from DORA (deployment frequency, lead time, MTTR and change fail rate) are fantastic indicators. A serverless team with minimal hand-offs can perform very well.

Fig. 1 Serverless is an Operational Construct – from AWS re:Invent 2018 presentation

Clearly, such a strategy requires lots of building blocks in place. You have committed to DevOps; serverless can help scale the advanced practices, but let’s make it real! Here are some key building blocks that you may find useful:

Public Cloud Strategy: Absolute clarity is required on the move to public cloud. This is a transformation and many traditional functions must (effectively) reinvent themselves—security, infrastructure, procurement, engineering, etc. Alignment at the initial stages is critical. What is your collective opinion on cloud-native? If you are a startup, then only serverless-first will give you the velocity you need.

Pipelines: The “pipeline to production” becomes a business-critical capability. Many critical controls must be reimagined in this new world and a low-friction environment is required for teams to move fast safely and securely. Investment in this area is important; otherwise, you will frustrate your engineers.

Patterns: You are now moving toward a commodity platform that welcomes standardized. You will start to notice that many teams are creating the same boilerplate components. Do you have an effective reuse strategy? If it’s too heavy, you will slow everyone down, but if it’s too light, you will create mass duplication. Approaches such as AWS CDK (Cloud Development Kit) and some of the scaffolding approaches available today to enable the creation of “building block patterns” to create components quickly. This is well demonstrated in cdkpatterns.com, which is an open source library of cloud patterns. Liberty Mutual has deployed hundreds of applications using such patterns [Fig 2].

Fig. 2 CDK pattern from cdkpatterns.com

Well-Architected: Finally, in this environment, architecture must happen within the teams. But you still need architects. The AWS Well-Architected Framework provides a very clean approach with five pillars of non-functional requirements that prompt teams around concerns that must be addressed. It’s not a checklist; more like a set of challenges. You’ll find that teams can self-assess, but require an architect to coach through some of the more advanced areas.

Finally, imagine well-architected constructs that are available in your code building blocks and comply with your pipeline controls. This is how DevOps can scale in a serverless environment: You bake the quality controls into the engineering environment.

Fig. 3 – Lambda Power Tuner from Alex Casalboni

After engineers go on this journey, they have power tools at their disposal. They can create world-class systems quickly that run at a low cost. But, be warned, once you set your teams off on this journey, there is no turning back. It is a relentless journey of optimization and improvement; that may be an overhead for companies that do not regard engineering as a differentiating capability.

For the serverless journey, the most important piece of the puzzle is to ensure your engineering teams understand your core business. Create cross-functional teams that are chasing a business KPI. Engineers are engaged when they have a powerful environment that allows them to solve business problems quickly. There is no business-IT divide; you end up with technology teams innovating to meet the company vision. Liberty Mutual has not only observed many of the benefits mentioned here, but also improved engagement from our engineering community. Serverless is the paradigm shift that allows engineers to innovate and move at a pace that has never been possible before.

August 6, 2020
DevOps Unbound: Exploring Acceleration and Automation, Setting DevOps Free

We are launching a new video series Thursday, Aug. 6, called DevOps Unbound. DevOps Unbound will focus on DevOps, automation, CI/CD and testing, featuring leaders in these areas as we explore the challenges and issues software delivery and IT teams face every day. How do we go faster, smarter with better quality? DevOps Unbound!

The series will premier on our TechStrong.tv streaming network and should start around 10:40 a.m. Eastern tomorrow, August 6th. It will be available on Linkedin Live (on the DevOps.com page), Facebook Live (DevOps.com, Container Journal or Security Boulevard pages), YouTube on the DevOpsTV channel and the MediaOps sites. Additionally, the show is available on-demand on our DevOps.com mobile app and our DevOps.com OTT channels on Roku, Apple TV and Amazon Fire TV. Finally, all episodes will be available on-demand on Digital Anarchist, our streaming video platform. The series is sponsored by our friends at Tricentis.

New episodes will run every other week. Also planned is a monthly DevOps Unbound roundtable, open to the public and featuring a revolving panel of experts around a given theme. The first live monthly roundtable will be Aug. 18, around the theme “DevEverythingOps.” Roundtable discussions do require registration; you can register for the first event here.

Premier Episode August 6th

The premiere episode features Dr. Grigori Melnik, CPO of Tricentis, and James Bach, the developer of session-based testing and a highly respected expert in testing. We will explore the history of testing and how it has adapted over the years to allow for more automation, help testers and developers be more efficient and accelerate the software release process.

These are two people renowned in their fields who arrived here from very different paths. Bach is a high school dropout who was bitten fiercely by the computer bug. He was a game developer and then a coder who became one of Apple’s first test engineers in 1987. Dr. Grigori Melnik has a Ph.D. in testing and is a respected authority in both the academic and business worlds. Though they come from different backgrounds, they have more in common than you might think. It was a great discussion and one I think you will really enjoy.

Future episodes of the bi-weekly show will feature topics including DevSecOps, open source, cloud-native, CI/CD, SDM, DataOps, observability and more. We think DevOps Unbound will quickly become an essential part of your DevOps information arsenal.

We are always on the lookout for future topics and guests. If you think you would like to take a giant leap with DevOps Unbound, reach out to me at unbound@devops.com with your idea and we can follow up.

BTW, DevOps Unbound is just one of several new shows we are premiering on TechStrong TV and Digital Anarchist. Another is CISO Talk, featuring Unisys CISO Mathew Newfield and me. That will be on TechStrong TV this coming Monday. More announcements to come about new, recurring shows.

We recognize that people want to see video content and at MediaOps/DevOps.com we are producing lots of exciting new video content. Stay tuned.

August 5, 2020
How to Build an Accessibility-First Design Culture
When it comes to digital experiences, every one of us can recount a frustrating interaction with a website or app. For the 61 million American adults with disabilities, lack of website accessibility can transform a frustrating experience to one that inhibits essential activities such as working, shopping and socializing.

To improve customer experiences and ensure equal access for all users, more businesses are adopting a human-centered, inclusive approach to design. But building a culture of accessibility can be a daunting task. Consider these steps to help build a culture of accessibility in your organization and engrain it as a practice across your development teams.

Photo Credit: Microsoft Inclusive Toolkit

Get Buy-In

While most businesses agree accessibility is important, getting internal buy-in can be a bumpy process. Accessibility likely wasn’t a part of the curriculum for developers and engineers, and it may not be a common term yet among leadership in your organization. The industry and your brand haven’t been intentionally exclusive; rather, teams haven’t been enlightened yet to think about accessibility and its impact. There’s a misconception that inaccessible products only affect a small portion of end users. However, 1 billion people, or 15% of the world’s population, experience some form of disability, and we must include them when designing products.

Additionally, the following points can help champion an accessibility program:
1. Accessible products benefit everyone no matter their physical or cognitive ability because accessible products represent an ideal user experience that is clear and intuitive. For example, many people tend to be power users of keyboard shortcuts. The ability to tab through inputs or use the enter key to submit a form makes all users more productive, not just those who rely on a keyboard instead of a mouse. Similarly, applications with proper focus management and strong color contrast have better navigation, while also empowering those with low vision to perceive changes.
2. Accessibility is an investment in your employees and customers. Measuring the business value of accessibility can be tricky. One way to frame it is to think about the ripple effects of inaccessible products. Not only are you preventing users with disabilities from using your product, you’re also limiting the people who can work at your company and develop your product. That impact is further extended for B2B companies—you’re limiting who your customers can engage with and sell to, as well. While measuring the success of an accessible product isn’t always easy, it’s clear that inaccessible products can create unanticipated and cascading business and legal problems.
Bake It In

Like any program, accessibility requires constant attention to engrain it in an organization’s culture. Consider creating a group of cross-functional Accessibility Champions that own keeping accessibility top of mind, sharing insights and advocating for accessibility on their teams. This group should consist of anyone who touches the UI of the product, such as the entire design team, someone on each engineering team and representation from QA. This approach ensures each team is prioritizing accessibility as they build and test and they have a point person on their team leading the effort. It also redistributes the idea of an accessibility expert, disseminates knowledge throughout your organization and gives everyone the sense that accessibility is a part of their role.

Creating an accessibility engineering plan will help guide your organization toward making accessibility and inclusive design an integral part of your processes across planning, development and QA. This plan can include incremental changes along with broader, long-term goals:
- Conduct an accessibility audit.
- Sync with sales and customer support to determine if customers have experienced friction with the product due to inaccessibility or if anyone has asked when the product will be accessible.
- Determine the top three components to fix first.
- Add Accessibility Epic(s) to Jira to track improvements.
- Add accessibility as acceptance criteria for every project.
- Include accessibility testing in bug bashes.
- Weave accessibility into the onboarding process for all engineering new hires.
- Add an accessibility section to any technical design and product requirement documentation.
- Add a section to the GitHub PR template that surfaces accessibility checks (testing with keyboard, screen reader and linting).
- Showcase/demo accessibility fixes on a regular cadence at engineering all-hands meetings and OKR reviews.
Start With the Components

So, what to do first? A great place to begin is your component library. Identify which components are used the most often and which underlying components underpin other functions. For example, make sure buttons, inputs and links have accessible focus and hover states. It’s a lucrative, efficient way of scaling accessibility fixes because once you make one fix, you’ll see it propagate throughout the organization wherever that component is used.

There are a few key factors to be aware of at this stage. First, create a clear plan for who can make changes and how you’re testing components to ensure accessibility features are not unintentionally removed. Second, your work doesn’t end after creating accessible components. In the UI, individual components are put together like puzzle pieces, and just because each piece is accessible doesn’t mean the entire UI will be. Since the UI involves multiple components talking to each other, you’ll need to ensure that the experience is usable and accessible as a whole.

The goal is to ensure every existing and new component in a library is accessible by default. This way, when developers pull features into their work, they’ll know with certainty it’s designed to be accessible. Get it right once, and you get it right everywhere.

Building a culture of accessibility doesn’t happen overnight, but you can start with a few small changes that have a greater collective impact and build from there to make it a part of your entire development workflow. Most importantly, making an actionable, conscious commitment to accessibility will contribute to building better products and a more inclusive web.

For more information and resources about accessibility and inclusive design, visit the W3C Web Accessibility Initiative documentation, which includes tips to help designers and developers get started. Additionally, the A11Y project provides Web Content Accessibility Guidelines (WCAG) checklists to determine how accessible your site is. Plus, services such as Fable and Access Works allow you to meet with people with disabilities and engage them in research and user testing.
August 5, 2020
Top Pressing Concerns for CI/CD in 2020
The tech landscape is changing rapidly. With the need for increased development velocity, CI/CD has become ubiquitous among most organizations. Yet, with the growing number of microservices, CI/CD may be tough to wrangle without the right strategy.

At the same time, it’s difficult to keep pace with the constant evolution around CI/CD. So, what is the state of the CI/CD space in 2020, especially in these turbulent times? What do stakeholders see as top pressing concerns companies should consider to advance their release fluidity?

For this article, I interviewed two stakeholders on the latest trends and industry cooperation around CI/CD. Read on to discover what two CI/CD thinkers, Tracy Miranda, board chair, Continuous Delivery Foundation (CDF), and Lachlan Donald, CEO, Buildkite, see are pressing concerns for CI/CD in 2020.

More Microservices, More Problems

With the number of microservices rising to the hundreds (or thousands) for some enterprises, maintaining consistent release strategies across the board could easily become a nightmare. If the goal is to replicate a unique CI/CD pipeline per microservice, pipeline automation and configuration management become top concerns.

“Automation will be key,” said Miranda. “We can no longer afford to be writing each CI/CD pipeline by hand; we will need new approaches.” She added that recent industry discussion shows “configuration management is a top concern for CI/CD practitioners.”

Tools like Jenkin X could enable this sort of automation, she noted. Companies should consider evolving their CI/CD pipelines with better configuration management.

Platforms Must Evolve

The recent “State of the Cloud Report” from Flexera found 39% of IT leaders saw implementing CI/CD in the cloud as a top priority. According to Miranda, the global crisis seems only to accelerate the need for cloud and CI/CD automation.

But, traditional CI/CD platforms aren’t always suited for modern development workflows with numerous pipelines. For Donald, an evolution in new tooling is absolutely necessary to help companies keep pace.

“We’re at a point where organizations need to completely rethink their approach to CI/CD,” Donald said. “Traditional CI/CD platforms are not suited for modern development workflows.” He cites slow speed, unreliability, scalability limitations and security risks as common hangups among many CI/CD tools.

Lack of Clarity in CI/CD Market

The market has expanded significantly to address the gap in CI/CD tooling effectiveness. According to Miranda, this has led to a continuous deployment tooling ecosystem wrought with interoperability and competing styles.

“My biggest concern is the proliferation of tools and platforms with a lack of interoperability and lack of clarity of how to best use them or whether they are fit for purpose,” Miranda said.

These differences make it difficult for developers to find the right integrations that work for their environments, leading to wasted time and effort. “Developers will experience growing pains with their CI/CD platforms as they work to keep up with the constant demand for quality software,” acknowledged Donald.

Lack of Standards

Part of the lack of clarity resides in the fact that there are little to no standards emerging in the CI/CD space. Aside from Accelerate rising as the “bible” for software delivery, Miranda sees a lack of standards in the CI/CD space.

“Not having standards or de facto standards is a problem,” she said. “For example, why is there no de facto standard for reporting pipeline test results?”

Advancing Open Source CI/CD Tools

To improve Interoperability throughout the CI/CD industry, bodies such as CDF are working on setting general guidelines. Miranda described how CDF is accomplishing this, by supporting a CDF Interoperability working group, sharing CI/CD practices and maintaining open source CI/CD projects. Two CDF projects she finds quite compelling are Tekton, a pipeline execution engine, and Jenkins X, a new CI/CD platform for pipeline automation.

Cloud vs. On-Premises: A False Dichotomy?

In analyzing the CI/CD tooling ecosystem, Donald recognizes a recurring dichotomy; that companies must choose between the security of on-premises and the speed of cloud-native. In reality, this dichotomy may be false.

“It is possible to have the best of both worlds,” he said. “A hybrid strategy [is] the solution for testing and deploying software at scale without compromising security or performance.’

As opposed to cloud tooling, self-hosted CI/CD control planes could grant developers more code control and help meet security standards. “I believe self-hosted agents are always a good idea,” Donald said.

As we recently covered, a hybrid cloud approach is picking up steam. Similarly, Jenkins X is designed to run on any Kubernetes cluster, regardless of on-premises or cloud, demonstrating industry movement toward tooling neutrality and hybrid motivations.

Moving Beyond PASS or FAIL

There is a reductive tendency in CI/CD to simply check whether a build passed or not. But, this mindset misses an enormous amount of useful information and undervalues the test and release suites themselves.

“Teams should get granular about what worked and what didn’t,” Donald said. DevOps teams must evolve beyond a “PASS or FAIL” mindset to analyze more minute aspects of the build process itself.

For Donald, this also means shifting to treat test code performance “as a first-class citizen, applying modern development and scaling practices to it and allocating the necessary resources to it.”

Future Evolution: Progressive Delivery and Hybrid Approach

In closing, I asked Miranda and Donald their thoughts on the future CI/CD evolution. They shared their perspectives on where CI/CD needs to evolve.

Miranda views progressive delivery as the next milestone. With progressive delivery, teams deploy services in a staged manner, she said. “Tools, technologies and processes that fall under the ‘progressive delivery’ umbrella will be the next milestones for improving deployment speed and efficiency.” Building blocks such as services meshes could help advance this process, she noted.

Donald stressed the need for a hybrid approach to CI/CD. A neutral control plane could grant better control and future-proof continual release strategies.

Assessing the CI/CD Landscape

In summary, our panel shared thoughts on the top concerns for CI/CD in 2020:
- Platforms must evolve: Teams should rethink their approach and emphasize automation in pipelines.
- Lack of clarity: There are few standards and a lack of clarity among tools. This confuses developer adoption.
- Interoperability: A focus on interoperability is critical toward advancing the industry.
- Unhelpful dichotomies: The cloud-native versus on-premises divide may not be helpful for a hybrid strategy; thus, it’s recommended to seek out platform-neutral tools.
- Move beyond PASS or FAIL: Switch the mindset to “Test code = application code.”
High performant organizations must scale their technology tremendously. With this desire comes a mounting strain on how companies adopt a continuous deployment strategy. Without a proper CI/CD plan, business agility is sacrificed.

As an ending note, I absolutely love Donald’s statement: “A good litmus test for how well a company is run is their approach to CI/CD.” Perhaps what all can agree on is that companies must prioritize CI/CD, as it is now business-critical. Otherwise, deployments may be numbered for those organizations that don’t adapt.
August 4, 2020
Shifting to Product-Centric Work in Utilities
Large organizations across the services spectrum are transforming from project-centric operations to product-centric way of working. Customer centricity is driving this transformation, as product-centric organizations put customers at the center of everything they do. Such a transition is largely dependent on how information technology solutions are deployed in the said organization.

This article looks at the importance of a product-centric way of working for utilities organizations, particularly the power sector, which is looking to transform customer interactions by revolutionary use of new digital technologies and solutions such as smart metering and smart homes.

At the onset, power utility organizations should focus on shifting to a product-centric way of working in areas that show promise to deliver innovation and digital enterprise such as customer analytics.

Fig. 1: Current State of Play in Most Power Utilities

Fig. 1 outlines the current state of play at most power utility companies, where organizational silos and application silos lead to customer challenges in understanding the products and services offered, as well as productivity losses and an increase in reaction times to changes demanded by business and regulation.

In most power utilities, silos exist in business units which are reflected in how the applications supporting those business units are organized and managed. This leads to duplication of effort, work and applications, as well as a waste of time and resources. Customers are often confused about multitudes of service offerings and various channels. There is also an inherent delay in responding to challenges thrown by changes to regulation and/or business environment.

This is changing, though, and one can look at the “meter to cash” processes for a reflection of market realities and customer empowerment.

In our experience, there are four key drivers of a move toward product-centric model:
- Digital Core – to evolve, innovate as well as leverage cloud, DevOps and agility to support the business.
  - DevOps practices and agility from ideation to sunset.
  - Encourage innovation.
  - Digital ready systems.
  - Focus on Product value.
  - Lean planning and maintenance.
- Intelligent Enterprise – to optimize the cost and delivery of services and encourage innovation.
  - Automation to support business agility of the power utility.
  - Autonomous operations to support power distribution and customer management.
  - Decisions based on insights drawn from customer and market data.
  - Focus on the innovation of products, services offered and customer engagement.
- Experience Quality focused on improving customer experience.
  - Product centric service delivery.
  - Business agility.
  - DevOps practices.
  - Experience centricity.
  - Product and experience-focused metrics.
  - Command center that is aligned with the business.
- Service Management and integration for scalability and unification of service delivery in the power utility while retaining a multi-supplier ecosystem as needed.
  - Continuous improvements.
  - Standardized deployment and integration management.
  - Collaboration in a multi-supplier scenario.
  - Service lifecycle management.
  - Information technology service delivery performance management.
This transformation to a product-centric organization ensures that processes, people, workflows and decisions power the power utility’s ability to:
- Customize services and offers.
- Improved and faster response to customer requests.
- Consistent and distinguished cross-channel experience for customers.
- Boost productivity.
- Eliminate waste.
Shifting to a Product-Centric Model

The product-centric model of delivery uses Agile, DevOps and Lean principles for strategic planning to ensure uncertainties and lack of information do not derail the delivery of services to end customers. A product-centric model places importance on relatively shorter horizons of budget and effort planning over massive planning at the beginning of a financial year. This is important, as a power utility is likely to be able to plan better for a quarter at a time rather than a complete year, particularly in the current economic climate. This not only ties well with the goal of business agility but also ensures improved planning and budgeting, as there is more information available about business goals a quarter at a time. The new teams organized around business capabilities, as shown in Fig. 2, then plan the product backlog, execute, test and deliver business value
to business and customers continually.

Fig. 2: Shifting from technology to business-aligned value streams or product-centric organization

Power utilities also need to change and improve their technology-centric solutions to help with forecasting and data analytics. The product-centric organization helps with this goal by identifying gaps for closure and aiding the creation of self-organizing teams that then take on the goals of the organization to address all the identified gaps, including in areas of forecasting and analytics.

Fig. 3: Organization design – a product-centric organization

Fig. 3 shows a sample organization design for the product-centric model. The figure also shows the breakdown of the structure from business function to value steams to products or product groups and, finally, applications delivering value—all the while ensuring that the goal of customer-centricity is at the center of every decision.

Governance and Metrics

A very important part of creating and adopting a product-centric model of functioning at a power utility includes focusing on the governance of the transformation process, as well as business results accrued from the transformation. The acceleration and adoption of Agile and DevOps practices on the technology side can be tracked through qualitative adoption measures. This leads to product level performance improvement, resulting in better products and faster delivery, with the product delivery mindset focusing on customer-centricity, leading to trends in:
- Faster time to market.
- Improved business satisfaction.
- Improved quality.
- Reduced information technology cost.
Other than this, following metrics and measures be focused upon:
- Adoption Measures.
  - Adoption maturity.
  - Competency/skill index.
  - Operational excellence.
- Product Team Level Measures – measured at team level per sprint.
  - Productivity measures such as team velocity and throughput.
  - Stakeholder satisfaction measures such as product owner demo satisfaction.
  - Quality measures such as percent automated unit test coverage and test effectiveness.
  - Business effectiveness measures such as business value delivered per feature and requirement readiness cycle time.
- Product Group Level Measures – measured at product family level per month or quarter.
  - Lead time to change.
  - Cost of requirement change.
  - Release frequency.
  - Cost of rework.
  - Mean time to recover.
  - Application stability.
- Portfolio Level Measures – measured at portfolio level per quarter or half-year.
  - Delivery satisfaction index.
  - Customer satisfaction index.
- Team Satisfaction Metrics.
  - Team happiness index.
- Coaching Effectiveness.
  - Coaching feedback.
- Team’s Self Organization Quotient.
  - Product ownership.
  - Sprint health.
  - Team health.
  - Technical health.
Potential Areas of Benefit for a Power Utility

There are at least four core areas in which a power utility can expect to experience quantifiable benefits by transforming to a product-centric model of operations. They are:
- Budget and Plan
  - Cost and time are the most valued commodities in a tightly run power utility.
  - A product-centric model ensures money, effort and time are allocated based on the validation of investment already made in an initiative.
  - Improvement in prediction of timelines, effort and benefits of an initiative because of cadence and timeboxing inherent to the product-centric model.
- Managing the Capacity
  - The key focus is on alignment with business and continuous delivery of business value.
  - The cost of delay and the cost of waiting are also reduced.
  - Focus on value delivery with quantified business benefits.
- Governance of Programs
  - The governance in product-centric organizations is about the exercise of control on the quantification of benefits.
  - The focus in a product-centric organization is on “building the right things the right way.” Rationalization of cost, elimination of waste and focus on business value are direct by-products.
- Managing the Demand
  - Continuous visualization and management of demand is an area where benefits are accrued as needs and ideas flow into the strategy filter continually from several sources.
  - The product-centric model allows the continuous elaboration of light use cases to ensure value is correctly calculated and assigned to each use case.
  - The program, portfolio and product backlogs are continuously evaluated to ensure that there is a cost-benefit sense to each item.
Conclusion

Some of the benefits of a product-centric model usage at power utilities are stated below:
- Customized electricity plans for efficiency, cost or comfortable consumption.
- Improvement in reliability.
- Creation of environmentally sustainable renewable programs of power for the more aware customers.
- Incrementally move toward a “business as usual (BAU)” relationship with regulatory changes.
A product-centric model at power utilities can also help bridge the data divide to improve data and analytics usage and enable creation programs in the following areas:
- Services and products that the customers are most probable to use.
- Optimized collection and prediction mechanisms to create plans to encourage timely payments.
- Create pre-paid services.
- Plans for direct control of load which allowed the utility to reduce air conditioning kind of machinery during peak usage and in return recompense customers.
- Improve pattern recognition for delinquency and fraud management.
- Cross-selling opportunities.
- Targeted sales and marketing campaigns.
This article was co-authored by Anurag Choudhry, Enterprise Architect, Manufacturing & Utilities BG at Tata Consultancy Services.

— Anshu Premchand
August 3, 2020
Survey: DevOps Haves Benefiting Over Have-Nots

A global survey of 400 global IT executives and practitioners conducted by Longitude, an arm of Financial Times Group, on behalf of New Relic suggests organizations that have embraced best DevOps practices are starting to enjoy material financial benefits over rivals that have not yet made similar investments in modernizing application development processes.

The survey finds 25% of respondents are leaders when it comes to application development, with 90% of those respondents identifying the development and deployment of more perfect software as a strategic priority. Over a third of those respondents report their organizations have seen a 10% more increased revenue over the last three years.

Those same survey respondents have also adopted continuous integration/continuous delivery (CI/CD) platforms (98%) and have made observability a key tenet of their software development efforts (94%).

The survey also finds 77% of developer time on average among leaders is spent on innovation rather than on fixing issues. Leaders also find out about system and software interruptions via observability platforms (78%) versus needing to be informed by end users. A full 99% of leaders say they are able to quickly understand the results of changes to software, which also accounts for why leaders have fewer than five major software or systems outages per month (83%).

Leaders are also more likely to have adopted chaos testing (77%) and automated remediation (89%). Three-quarters of leaders (75%) have an average resolution time of less than 30 minutes.

On average, the survey finds leaders run 66% of their software and systems in the cloud, with a total of 96% having adopted a cloud-native approach to application development.

The top barriers to greater cloud adoption for the leaders were reported as being security concerns and regulations.

Buddy Brewer, global vice president and general manager for digital experience monitoring at New Relic, said more organizations today are focusing on applications that are most critical to the business, which includes digital business initiatives that enable organizations to engage customers during the current COVID-19 pandemic.

IT organizations are also breaking up into smaller teams to enable them to collaborate more effectively when building and deploying these applications, he added. As organization dynamics change, Brewer said those small teams need access to a broad range of cloud-based tools to manage highly distributed applications.

There’s obviously time for IT laggards to close the current DevOps gap between themselves and DevOps leaders. However, as leaders move further down the DevOps maturity curve, the gap between leaders and laggards appears to be widening. Many organizations in the wake of the pandemic have doubled-down a DevOps process that makes it easier for small teams to collaborate as needed.

Of course, during the downturn brought on by the pandemic, not every organization may be able to make strategic investments in IT tools and processes. The catch-22 is that it’s even less clear is just how those organizations will recover after the economy rebounds if they are still dependent on archaic tools and processes.

July 29, 2020
A Developer’s Guide to CCPA, GDPR Compliance
Here’s what developers need to know to ensure compliance with the two biggest privacy laws

The digital landscape is continuously evolving, and privacy regulations such as CCPA (California Consumer Privacy Act) and the European Union’s GDPR (General Data Protection Regulation) are in effect to give consumers their fundamental right to data privacy.

These regulations force organizations to revamp their operations to comply. This means all departments within an organization, from marketing to software development and everything in between, have to keep privacy regulations in mind and tweak their workflows accordingly.

In this article, we will discuss the steps developers can take to stay compliant with these regulations.

Understanding Data Rights

With more people concerned about their data rights, giving them complete control over their data is essential in today’s world. Under both GDPR and CCPA, here are all the rights consumers have concerning their data:
- The right to be informed.
- The right of access.
- The right to rectification.
- The right to erasure.
- The right to restrict processing.
- The right to data portability.
- The right to object to processing.
- The rights concerning automated decision-making and profiling.
A consumer can practice these rights at any given time and enterprises are obligated to fulfill these requests as soon as possible.

GDPR and CCPA recognize that more and more consumer data is available online, which increases the possibility of cyberthreats and invites other malicious activities. This is why it’s crucial for these regulations to protect the consumers’ data while dissuading any instances of data breach or sprawl.

The Risks for Non-Compliant Businesses

The European Union (EU) has a history of making an example out of companies that are non-compliant with its regulations. One of the EU’s most recent actions was against Google.

France accused Google of infringement regarding the essential principles of the GDPR: transparency, information and consent. “Enforcement action was geared toward the way Google obtained consent,” said Myriah Jaworski, an attorney at Beckage PLLC.

Google did not present how and why an individual’s data was collected and stored, nor did the company make it easily accessible. Google was fined $57 million by the EU.

Seeing that even an industry giant is not immune to prosecution, it is clear that no company can get away with GDPR or CDPR non-compliance. To stay safe, developers must be well-versed in all the regulations and build their websites, apps and software with compliance in mind.

CCPA vs. GDPR: What’s the Difference?

While both laws serve to protect the rights of the individual, there are some differences between the two regulations. The following are the significant differences between the two laws.

Who Needs to Comply

The GDPR has a broad scope concerning who has to stay compliant with the law. It covers all citizens of the EU and regulates all organizations that collect and store personal information of EU citizens irrespective of their location and size.

In contrast, the CCPA places constraints on the size of organizations that need to comply. It applies to organizations that have $25 million or more in annual revenue; possess the personal data of more than 50,000 “consumers, households, or devices”; or earn more than half of its yearly income selling consumers’ data.

Financial Penalties

The GDPR mandates penalties based on non-compliance and data breaches. These penalties can reach up to 4% of the company’s annual global revenues, or €20 million (whichever amount is higher), with the commitment that administrative levies will be applied proportionately. CCPA fines are not cumulative but instead are applied per violation, which can reach up to $2,500 per unintentional violation and $7,500 per intentional violation, with no upper cap.

Consumer Rights

Both regulations give the consumer specific rights that they can exercise. Some of these rights include the right to have information deleted or accessed. The GDPR specifically focuses on all the data related to European Union consumers, whereas the CCPA considers both consumers and households as identifiable entities. Businesses need to test their processes and ensure they can accommodate these rights.

Use of Encryption

The clauses on encryption in both laws constitute an area that, although similar, still have some differences. Both laws call for access to data encryption, making this an essential part of the privacy protection component for businesses.

Steps to Compliance

Developers are the front-line infantry in this struggle toward compliance because websites and mobile apps are the first interactions a consumer will have with an organization. It is essential to cover all bases from the start to make the compliance workflow as smooth and efficient as possible. Let’s take a look at the steps developers can take to comply with each regulation.

Complying With CCPA

Data Mapping

To stay compliant, developers need to integrate proper data-mapping techniques into their systems. The law dictates that organizations should be fully aware of all the data they collect—this refers to what is collected, where it is stored and how it flows through the organization. Some operational suggestions include designating a single source of truth, maintaining lineage and tracking all data within the organization.

Inform Consumers

To comply with the CCPA, organizations will need the capability to fulfill data subject access requests (DSAR). A company’s website must show the consumer what data it is going to collect and how it will be collected. Developers can work with privacy officers to create a standard privacy notice for the website or an abbreviated pop-up policy at the point the data is collected.

Verify Queries

Organizations will be met with a flurry of requests from consumers exercising their rights under these regulations. Developers need to create a system by which the consumer can be authenticated and the correct information can be given to them. To streamline this process, developers can create a dedicated email account for requests and design workflows for verification purposes.

Data Minimization and Purpose Limiting

When collecting data, organizations need to make sure that the data is only used where necessary. To ensure that, developers can create forms that only require minimum information (data minimization) and organizations can make sure that internally used data is in line with privacy policies (purpose limitation).

Data Security

Under the CCPA, organizations are required to protect the data an organization keeps about a specific individual. Although not explicitly mentioned, it is beneficial for organizations to encrypt data at rest to prevent further compromise after any data breaches.

Developers can ensure security by implementing robust applications that offer end-to-end encryption to protect consumers’ data.

Complying with GDPR

Efficiently Store Data

The way an organization stores data can be the difference between compliance and non-compliance under GDPR. Developers need to ensure that minimal data is being derived from consumers to reduce liability and only store the data that is necessary for their processes. Lastly, developers should implement data subject access rights (DSAR) tools in their storage to efficiently respond to subject data access requests.

Subject Access Requests

Developers need to integrate a system that can map all the data in the data stores and make them easily accessible when consumers request access to the data that the company keeps, even complete deletion.

Contacting Users

Under the GDPR, an organization can not assume consent; it must be asked for. Developers working on a feature that will trigger an email or another message to be sent to users will need to integrate it with their organization’s consent tooling and check if they already have a consent channel for their use case. This will likely take the form of some source-of-truth database and an API that developers can query before sending messages.

Profiling

Profiling is the use of data to personalize a customer’s experience. To be compliant with GDPR, organizations should have a clear way for users to opt-out of profiling. The only important thing for developers going forward is understanding what counts as profiling and respecting a users’ choice before implementing any form of personalization.

Rewrite your Privacy Policy

The GDPR has brought several amendments to the current structure of any organization. The IT group is essential for organizations to revamp their privacy policies according to the GDPR. In this case, developers can integrate the privacy policy into company websites or as a pop-up to comply with the GDPR right to notice.

Key Takeaway

CCPA and GDPR are revolutionizing the data privacy sector, and organizations must comply with these regulations. Developers and marketers alike are going to have to find new ways in which they can efficiently comply with these regulations without hindering their current performance. Developers need to integrate automation to create a streamlined approach to compliance throughout the organization.
July 29, 2020