devopsy.kinsta.cloud

Category: DevOps Practice

Becoming a Data Company: Back to the Future

Transformations are complex, and it’s often difficult to see the big picture when you’re in the middle of one.

The elegance of the end-state of harmonious synergy between software and data belie the complexity and chaos of the transition we find ourselves in the midst of. Not that long ago, there was a general balance between the pace and sophistication of software delivery and the speed and maturity of data analysis. Markets and customer bases grew steadily allowing organizations to keep pace in their software and data competencies to handle increasing mutual scope and complexity.

However, the consumerization of IT—the ability of lay consumers to consume sophisticated software experiences from browsers and mobile devices—unleashed access to a massive, eager and demanding customer base measured in the billions. This gave birth to the current wave of digital transformation. Along with it came the race to get to market, i.e. build and deliver high-quality software exponentially faster, in order to capitalize on massive global opportunity.

Innovation thus started in the software domain. With faster, better software came the need for purpose-built data stores (document databases, graph databases, etc.), the need to manage the velocity, volume and variety of data collected from millions (or billions) of users and the need to extract insights and value fast enough to further fuel the pace of software innovation. Thus, software innovation planted the seeds for the current focus on data innovation.

While the innovators quickly traversed this software journey, most enterprises are only now fully in the midst of embracing new software methodologies and technologies to deliver software quickly and effectively. Those enterprises at the vanguard of the software journey are now embarking on the data journey—to find ways to safely and quickly harness the value locked away in enterprise data assets. This includes everything from analytics to rapid provisioning of copies of production data for software testing. In fact, 451 Research states 86% of companies that consider themselves mature in DevOps journey are planning on increasing their investment in DataOps. That is, companies that have traversed the software competency curve are now embarking on the data competency curve.

The Dawn of the CDO

The growing traction by data science, AI, analytics and DataOps technologies and vendors point to the rising importance of developing a data competency. The recent emergence of the Chief Data Officer (CDO) is the culmination of this trend.

Given the nascent focus on data at most enterprises, most CDOs are still grappling with defining their role, navigating the dynamics of the C-Suite, defending against regulatory and compliance risk and justifying their role by identifying growth opportunities. Despite these immediate challenges, they do have one thing going for them: they know they’re in the midst of mind-bending change, have the benefit of hindsight on the software journey and the many parallels it carries, and are mentally prepared for the organizational, technological and cultural changes that will need to be navigated. Most companies were stunned by the sheer suddenness and velocity of change that came with the DevOps and the software transformation journey. With DataOps and the data journey, they are now aware, prepared and committed to the associated journey and changes for the long haul. They can be proactive and avoid being reactive.

Planning for the DataOps Era

As cycle times for going to market and learning from the markets shorten, and innovation cycles accelerate, CDOs can expect that data-centric organizations will need to drive net-new top and bottom-line results without incurring undue risk. This will come through adoption of new technologies, organizational structures, cultures and workflows. New roles focused on data-centric activities, and new forms of collaboration between different personas will emerge.

In the modern enterprise, production will no longer be the most important data environment in enterprises. There is a growing need for fast access to secure, governed data outside of production—where the work of the rest (non-IT part) of the enterprise is centered. What used to be non-prod for IT will now be prod for the rest of the enterprise. To succeed, IT will need to deliver data quickly and securely to the rest of the organization or even across organizational boundaries to partners and other ecosystem players. Embracing collaborative processes and workflows that support this new pattern of enterprise operation will be critical. Enter DataOps—the unique mix of people, process and technology approaches required across teams in an organization to become agile with data.

Given the similarities outlined earlier of the data journey to the software journey, organizations’ DataOps journey can benefit from the lessons learned in their DevOps journey. For instance, developing a data competency will require new layers of organization and workflow patterns supported by corresponding technologies. Proactively planning and laying down infrastructure for DataOps will yield far better results rather than adopting technologies ad hoc until a data competency blueprint emerges through trial and error.

Building a Data Company Technology Stack

As with any complex shift, it makes sense to deconstruct data competency into a series of layered capabilities, each building on the one that came before, that together help develop a sophisticated competency. For example, when building their software competency, in order to apply software in sophisticated ways, (e.g., cloud native architectures and microservices), organizations needed to manage distributed software elements (e.g., scale, heal, monitor) which in turn relied on the ability to test and deploy incrementally, consistently and rapidly.

A similar approach can be applied to data. Building collaborative processes around appropriate technological tooling can yield layered component capabilities that collectively define a sophisticated data competency. While the layering was emergent in the software journey, CDOs have the opportunity to be deliberate rather than reactive in developing a layered approach as the diagram below illustrates.

At the foundation of a successful data competency lies the capability to deliver data anywhere in the organization and secure it to meet regulatory and compliance mandates. Being able to quickly secure and move any type of enterprise data across a hybrid cloud context is hard work, but also table stakes for basic data management hygiene. These are critical prerequisites and this foundation needs to be solid in order to support the layers above.

Next, there are various data management activities to master, such as the self-service access, governance, data catalogs, data cleansing, collaboration and versioning, etc. These define the next level of data competency and must be mastered before data can be applied in different ways, e.g. in machine learning, test data management, analytics, data science, etc.

A thoughtful, layered approach to building a data company requires building the right abstractions between layers and will avoid tool sprawl consisting of a mesh of ad hoc point solutions that inevitably require human glue to drive data workflows.

CDOs should evaluate their organization’s needs through such a structured lens to evaluate the organization’s needs. Each of the activities will bring with them ecosystems of vendors, engendering further evaluation of vendor capabilities. Critical questions to ask include (among many): Is the solution enterprise ready? Does it work with all your data sources across hybrid cloud? Will it evolve with your changing needs? How will it interoperate with other solutions? Can it be automated?

As CDOs think through these infrastructure investments to connect different parts of the enterprise, the similarity to laying down railroad tracks to connect a large country are unmistakable and perhaps instructive: the resilience and stability of lower layers enable flexibility, variety and innovation at the higher levels just like the simplicity and reliability of simple railroad tracks enables running electric, steam engines, luxury bogies and cargo bogies alike.

Toward a Data Company

The refrain of digital transformation has been that every company is a software company. But to complete the transformation, every company must also become a data company. Software development practices and tools have evolved significantly in the last decade. We’re in the early stages of a similar evolution in data practices and tools that will take us through the next decade. The symmetry and coupling between software and data is unmistakable and there is much to be learned from the software journey.

As CDOs, CIOs and CEOs plan the transformation of their enterprises in this new era, they will need to consider building a foundation for their data competency that is stable, resilient, compatible with their software development processes and adaptable to inevitable change. They should be prepared to deliver their data from where it is produced to where it is used throughout the enterprise.

The data company is an organization where data is treated not just as a byproduct of operations, but as the lifeblood of the organization—that drives software development, improves quality assurance, informs decision making and potentially generates revenue in its own right.

— Karun Bakshi

December 4, 2019
Why AWS Cost Optimization Is About More than Saving Money

Why is it important to track and monitor cloud costs? The obvious answer is it helps you save money—and everyone wants to save money.

Yet, cost plays a much larger role in your cloud journey. If you think about cost only in terms of how much you are paying and what you are getting in return, then you’re missing much of the value that careful cloud cost control can provide.

Let’s take a look at all of the reasons why monitoring and optimizing costs is critical for a successful AWS strategy.

What Is Cost Optimization?

In AWS and other clouds, cost optimization is the art and science of identifying and addressing inefficiencies in the way you are using resources.

Those inefficiencies come in many forms. They could be an EC2 instance that offers more resources than you need for a given workload, which could therefore be replaced by a different, less costly instance.

They could be a result of using one type of application architecture, like workloads running directly on virtual machines, when another, such as serverless or containers, would deliver the same performance for less overall cost.

They could be caused by failing to take advantage of the best opportunities available for hosting a given workload. For example, you might be using S3 Standard for storage when S3 Glacier would meet your needs just as well, at a lower cost.

By identifying these sorts of problems, you can take steps to spend less while maintaining the same level of performance for your cloud workloads. A basic best practice is to use tools such as AWS Trusted Advisor to help ensure that your infrastructure is set up in a cost-efficient way from the start.

Cost Optimization Insights

These tools only go so far, and they are not the be-all, end-all of cost optimization. They are designed simply to help choose the right type of configuration for a given workload. However, cost monitoring as a whole goes further, and delivers several other important types of insight into your AWS workloads and strategy.

Rogue Infrastructure

By its nature, rogue infrastructure—meaning use of infrastructure for purposes not officially authorized by your organization—is hard to detect. If it wasn’t, it wouldn’t be very good at being rogue.

If you monitor cloud costs carefully, however, rogue infrastructure will have a hard time hiding. If employees spin up a virtual server in the cloud for personal use, or create storage buckets for organizational data that is not supposed to exist in the cloud, you are likely to discover it when you perform cost optimization operations designed to help identify workloads that should not be running.

Having a strong IT governance policy in place that requires all cloud-based resources to be tagged is another way to help prevent the creation of rogue infrastructure, and to make unauthorized resources easy to detect. When possible, tagging should be automated to ensure the fastest and most consistent results.

Security Risks

Although cost optimization is certainly not the only tool you should use to help identify security issues in your cloud workloads, it provides some useful insight that other tools might lack.

For example, if you notice that your bills for a certain application or service have increased due to heavier usage without a corresponding increase in legitimate requests, it could be a sign of brute-force attacks or other unauthorized access attempts.

Neglected Projects

When was the last time you evaluated each of your applications to ensure it was running as efficiently and securely as possible?

If you don’t know because you don’t regularly do this, cost optimization can help ensure that you do recognize workloads that could stand to be reconfigured. If you discover that you are paying more than you expect for a given workload, or that one workload’s costs are rising more quickly than those of others without a clear reason, it’s probably time to take a look at that workload and assess whether it should be scaled up or down, migrated to a different type of service, or otherwise adjusted to operate more efficiently.

Operational Inefficiencies

Finally, cost optimization can help you identify inefficiencies in the way your organization is run. That is because fees charged by AWS that you could have avoided but didn’t might reflect larger inefficiencies within the organization.

For instance, consider AWS data storage costs. If your AWS cost analysis shows you are paying early-deletion fees for data stored in S3 Glacier, you should determine why that is happening. It could be simply because you should be using a different storage class, but it could also be because employees are deleting data sooner than intended.

Conclusion

Saving money by being cost-efficient should always be one of your goals. But running your overall business in an efficient, transparent manner is another. AWS cost optimization can help you achieve all of this.

This sponsored article was written on behalf of Eplexity.

— Chris Tozzi

December 3, 2019
DevOps: Get the Basics Right to Avoid Costly Mistakes

It is a truth of both agile and DevOps that speed is of the essence. This is a relative statement—some apps/portfolios/orgs envision speed completely differently than others—but if you are releasing more frequently than you were in the past, thanks to DevOps, that’s kind of the point.

Yet, we see a whole selection of things that fall by the wayside or are underserved. Security and testing are the easy/obvious ones. Considering the frequency with which we see “Unsecured Amazon storage bucket” (one announcement of this type spurred this blog), should we not see this folded into standard procedure or tools? Well, it is folded into standard tools for checking your cloud environment—I’ve used a couple of them and seen this warning come up. So shouldn’t organizations be using these tools more?

One thing about speed: It tends to cut corners. And cut corners lead to lapses. You can be a blameless environment that doesn’t punish people for mistakes, but for an organization, a mistake such as exposing hundreds of thousands of users will be punished from the outside, from regulatory agencies and the market in general. They’re not mistakes we should be making, and yet it still happens.

Another one is rollback versus roll forward. Most DevOps shops focus on the ability to fix an issue and plow forward. This works really well, 90+% of the time, and fails spectacularly sometimes. While it is an added step with some slowing of process, being able to roll back changes is important. And more than, “Well, theoretically, Git will allow us to go back to the last marked release …” because there is far more to an application architecture than just the source stored in a repository.

In short, no matter how fast things are going, have a plan for when things go wrong. Because they inevitably will. And take the little bit of extra time for preventatives such as warning someone other than the creator about buckets with no security.

“But there isn’t time …” is the common refrain to blogs such as this one. I get it. The increase in delivery only creates an increased appetite for delivery. But is there time to respond to disaster, should the worst happen? And I’d argue that it must be IT that plans for disasters. When someone gets fired because there was a security breach or a total systems failure, it is rarely an app owner. That puts the responsibility to make certain one doesn’t happen squarely on IT.

We’re delivering faster, offering increased functionality to users, moving forward on every front. You all are out there rocking it daily. Don’t risk all of that being ignored because you failed to take care of basic issues. Check out the tools to help out; feature flags and cloud security scanners both spring to mind with regard to my examples above. The tools are out there, you just have to work them into your process.

— Don Macvittie

November 27, 2019
DevSecOps Adoption and the Web Security Myth
As DevOps practices have become widespread in the tech community, many people have begun proclaiming the virtues of DevSecOps. As the name implies, DevSecOps is the addition of security into DevOps. Just as DevOps promises better-quality production in less time, DevSecOps promises better security with less time required to achieve and maintain it.

DevSecOps has many benefits. However, many executives are under the impression they can’t embrace DevSecOps across their entire organization.

This idea is false. Most organizations that have already adopted DevOps can, and should, take the next logical step and adopt DevSecOps as well.

‘It Sounds Great, But We Can’t Use It Here’

Adopting DevOps requires far-reaching changes throughout an organization. Going to DevSecOps can be similarly disruptive. However, internal disruption is not usually the (perceived) problem with DevSecOps.

Instead, the problem is usually the network border. Many executives believe that their current web security solutions are incompatible with DevSecOps.

In many cases, this is actually true. Older approaches to web security are indeed hindrances to DevSecOps practices.

Therefore, the problem isn’t actually with DevSecOps. The problem comes from the security solutions currently in use.

Legacy Security Solutions and DevSecOps

Many organizations are still using appliances (whether physical or virtual) as their primary protection against external threats. A decade ago, appliances were good enough. Today, they’re a competitive disadvantage.

There are several ways in which appliances are incompatible with DevSecOps:
- Programmatic control of an appliance is often difficult if it’s even supported at all.
- Configuring an appliance can be challenging. It often requires a high level of security expertise.
- Rolling out settings and configuration changes across a bank of appliances can be time-consuming and potentially error-prone.
- The cost and specialized nature of most appliances make them inconsistent with the DevOps mantra of “cattle, not pets.”
- Physical appliances do not scale.
- External-facing applications will be limited in their scaling as well.
- Appliances are not designed for ephemeral workflows.
- Appliances are not designed to support infrastructure as code (IaC).
Security appliances represent a decades-old approach to security, so it’s not surprising they don’t support modern practices. What is surprising is that even many allegedly “cloud-native” solutions aren’t built for DevSecOps, either.

For example, some require multiple instances to be launched into your environments. Some require additional layers of management just to maintain consistent configuration among them. Many do not provide good support for evolving architectures and expanding deployments. In general, these security products tend to require significant intervention and management, while lacking automated control capability. This all severely limits the use of DevSecOps.

These appliances and incorrectly labeled “cloud-native” solutions are why many executives believe DevSecOps isn’t an option for their organizations. To be clear, these executives are correct—for as long as they continue to use previous-generation approaches to web security.

Not Just a DevSecOps Issue

Older approaches to web security have other problems that go beyond a hindrance to DevSecOps usage.

For example, appliances are marketed as complete web security solutions, but they cannot actually fulfill this role. Among other things, an appliance cannot defend against a volumetric DDoS. The attack can overwhelm the upstream ISP before the appliance even has a chance to scrub the traffic. This can result in the ISP blackholing all incoming traffic, which makes the targeted site and web applications unavailable to users and customers—which is the exact situation the appliance was supposed to prevent.

Solving the Problem

Organizations which are still using older security technologies should re-evaluate this decision. Modern solutions such as cloud web security platforms can provide better protection, along with numerous other benefits. These include full management by the provider, real-time reporting and traffic control, adaptive threat identification based on machine learning and much more.

When organizations continue to use legacy security solutions, they not only prevent themselves from enjoying the benefits of DevSecOps, they are also missing out on many other benefits as well.

— Maksim Blishchik
November 26, 2019
Optimizing AWS Performance Over the Long Term
Ask most folks how to optimize performance on AWS, and they’ll talk about things such as autoscaling and rightsizing.

Those tools are certainly part of an effective AWS performance optimization strategy. On their own, however, they are not enough to guarantee performance and reliability over the long haul.

That’s because tools and processes such as autoscaling and rightsizing only help you optimize the performance of applications that are running currently, or are about to be deployed. They do little to ensure that performance remains optimal—especially as cloud services, cost structures and architectures evolve.

With that challenge in mind, let’s discuss how to get the most out of AWS performance over the long term.

What is AWS Performance?

Before delving into long-term AWS performance strategies, let’s make clear what we mean by performance. A high-performing application is one that:
- Meets or exceeds user expectations regarding responsiveness and speed.
- Meets or exceeds SLA requirements for availability.
- Is able to scale seamlessly as performance requirements change.
- Does all of the above in a cost-efficient manner.
These aspects of a high-performing AWS application are important to emphasize because they reflect a holistic approach to performance. Sometimes performance is defined narrowly, in terms of application responsiveness and/or availability. Those are part of the performance equation, but in order to maximize performance over the long term, you need to think about other dimensions of performance, like the role played by scalability and cloud spend.

Long-Term AWS Performance Considerations

To optimize AWS performance over the long term, you must take a variety of factors into consideration.

Handling AWS Service Changes and Upgrades

AWS is constantly evolving. A performance strategy that works well today may not be as effective tomorrow if AWS introduces a new service or changes the features of an existing service.

The challenge here lies not just in updating your app and your cloud configuration as AWS changes how a particular service works. It also requires constantly evaluating whether the service you are using is the best fit for your application.

For example, think back about five years to when AWS Lambda (AWS’s serverless computing service) appeared. At the time, if you were running your apps inside virtual servers, you might have been able to improve overall performance by moving some of the workloads to serverless functions—not because anything in your virtual servers had become less efficient, but because better efficiency opportunities opened up with the addition of a new AWS service.

Thus, being constantly aware of the various services in AWS and the opportunities they offer is key to optimizing long-term performance.

Plan for Changing Cost Structures

AWS not only changes its menu of service offerings frequently, it also updates pricing. Since cost optimization is one key component of performance optimization, you must ensure that your cloud workloads are running in the most cost-efficient way even as cost structures change.

Staying aware of new cloud services (such as lower-cost S3 storage tiers) when they appear is one way to do this. But you also need visibility into the tremendously complex world of AWS pricing in all of its region-specific details.

Plan Long-Term for OSS Dependencies

Open-source software is something else that is constantly changing. If you rely on open source code to help run your AWS applications, you need to stay on top of changes in the projects you depend on.

If you are using the same version of an open-source application or module that you built into your application when you first deployed it, you may be missing out on performance optimization opportunities that have arisen since then.

Here again, constant awareness of and visibility into the complex open-source ecosystem is essential for ensuring that you are not missing out on optimization opportunities.

Application-Specific Performance Monitoring

Every application is a special snowflake and you need to keep that in mind when it comes to the way you track and optimize application performance.

The metrics or KPIs that work for optimizing the performance of one app may not be ideal for another. The pre-deployment testing routines you use for applications might vary, too.

So, make sure to tailor your monitoring strategy to your applications. Doing so is the only way to ensure you have the greatest, most relevant level of visibility into application performance over the long term.

Backup and Disaster Recovery

No matter how well your AWS workloads perform, sooner or later something will go wrong and they will fail. Whether the failure leads to a critical availability disruption or a mere hiccup hinges in large part on whether you have a backup and disaster recovery plan in place beforehand.

For AWS apps, an effective backup and recovery plan entails more than just backing up data to storage buckets and downloading it to rebuild servers. It should also take advantage of opportunities like the ability to create image-level backups of virtual servers and restore them automatically to EC2 if disaster strikes.

And of course, as AWS changes, you’ll need to keep your backup and recovery strategy in sync.

Changing SLA Requirements

Last but not least, keep your SLAs in mind. The SLAs you have to support today could change in the future—and by extension, so will the levels of availability and responsiveness that your users expect.

For this reason, make sure your AWS performance strategy is capable not only of meeting the SLAs you currently have in place, but also adapting and scaling as SLA requirements become more rigid.

Conclusion

Although we tend to think of performance within the context of the present, it’s really a long-tail game. Sure, you can autoscale your workloads or use rightsizing tools to help choose the best type of EC2 instance for right now. But truly maximizing performance requires you to think longer-term and be prepared to react with agility as new performance challenges and opportunities arise.

This sponsored article was written on behalf of Eplexity.

— Chris Tozzi
November 26, 2019
Untapping the Potential of DevOps, Part 1: Why Is It so Critical Today?

Apps are making the world a simpler, faster place. Mortgages can be verified in minutes. Food shows up at doorsteps with a few clicks. A safe ride is just a pin-drop away. For consumers, having a smooth, flawless and enjoyable app experience is the status quo. However, what goes on behind the scenes to make it happen is infinitely more complex.

From developers to architects and QA to test automation engineers, the software we interact with requires connected teams to ensure that security, reliability, speed and quality are consistently being delivered. For years, technical teams operated in silos that were worlds apart. Application and software delivery processes were slow, painful and error-prone—a dangerous mix for failure.

About a decade ago, DevOps was introduced as a remedy.

While definitions of DevOps can vary, it is essentially about aligning people, tools and processes to make software delivery better, faster, cheaper and more enjoyable for those involved (which tends to lead to happy customers, too). However, with the boom in DevOps technologies and resources, cutting through the noise and honing in on what will make the biggest impact can be something of a struggle.

In this series, we will address why DevOps is a key enabler for success, the trends impacting the software delivery industry, DevOps challenges, tips and more. First, let’s dig into why DevOps should be a part of an overall business strategy.

Why is DevOps Critical Today?

Times are changing, and so are IT best practices. So, what makes DevOps great for business success?

As mentioned, customers expect new product features to be delivered continuously—with quality and security guaranteed. In order to deliver on this promise, the entire IT organization must work in unison toward the same goals, and these goals should be centered around better customer experience. Much of a DevOps transformation is focused on culture, and when everyone from developers and business leads to legal and marketing teams are collaborating and happy, it leads to positive business outcomes. In fact, according to the findings in the 2019 Accelerate State of DevOps Report, organizations that invest in creating a culture of psychological safety and collaboration are more likely to be high performing.

Being a high performing organization sounds like a no-brainer, but what does it really mean? According to the same report, the highest performing organizations have 208 times more frequent code deployments, recover 2,604 times faster from incidents and deploy 106 times faster than low performing organizations. DevOps is the foundation that makes this high performance possible. This means that organizations that aren’t adopting DevOps will be left in the dust.

DevOps is a set of practices that combine software development and information technology operations to shorten the systems development lifecycle while delivering features, fixes and updates frequently in close alignment with business objectives. The practices help deliver features to end-users much faster and with higher quality, testing and documentation. Processes should be reproducible and auditable, making the technical infrastructure more secure and reliable as a result. Ultimately, DevOps leads to happy and productive teams, satisfied customers and keeps the budget in the black.

There’s much more to uncover when it comes to DevOps—in the next part of this series we will look at some of the common mistakes made along the DevOps transformation journey.

— Jonathan Fries

November 25, 2019
How Combining DevOps and Agile Can Ease Black Friday Pressure

Every year, seasonal peaks such as Black Friday and Cyber Monday bring a higher volume and faster pace of work, as well as a considerable amount of pressure. This comes with a number of challenges, including managing traffic influx, ensuring top-notch digital experience, rebuilding catalog infrastructure and carrying out load and performance testing.

Development teams need to navigate increasingly complex tasks in time spans that are shorter than usual. That’s where Agile comes in: With its fast-fail nature, Agile fills the gaps between limited resources and market needs. By prioritizing self-organizing teams and robust feedback, its flexibility allows teams to rapidly refocus as business requirements inevitably shift.

DevOps, on the other hand, offers a more systematic approach. Despite its various overlaps with Agile, its signature benefit is its heavy reliance on the collaboration between development and operations throughout the entire product lifecycle. With continuous testing and delivery, DevOps ensures the delivered product is fully in line with customers’ needs.

Both approaches offer benefits to teams looking to navigate the complexities of the holiday season. Research by Cutter Consortium revealed that combining DevOps and Agile can lead to the reduction of the time needed to get a product to market by 20-40% and to significant cost reductions as well.

Why should teams leverage the marriage of DevOps and Agile to relieve the pressure of the holiday shopping season?

Automation as the Centerpiece

The last quarter of the year is traditionally seen as a trial that often stretches organizations to their full capacity. However, if organizations are still using obsolete manual processes, standards will slip. Exposing high-availability systems to human error during seasonal peaks that bring ten-fold higher traffic than normal may have dire consequences for a business. Automation, on the other hand, ensures that processes are less error-prone and vulnerable to human errors.

DevOps puts emphasis on automation. Not only does it focus on maximum efficiency when deploying software, but it also advocates for close process documentation. Automation facilitates optimal knowledge transfer and frees production teams from time-consuming manual tasks, allowing them to become faster and more adaptable.

It’s common that organizations claim that Agile and SCRUM simply don’t work for them. However, this is mainly because they haven’t achieved advanced enough automation levels or outsourced enough tasks to be fully flexible. But with automation, they can embrace it more effectively–as DevOps aims for the simplification of the entire process, its focus on automation establishes solid foundations for efficient Agile. This means organizations cut down the time to market and are more equipped to face the challenges of high season.

High Flexibility for a Better Product

Seasonal peaks such as Black Friday and Cyber Monday are always bound to agility. The revenue potential of an organization depends largely on the flexibility of the team and the processes and systems that, if maintained effectively, can add significant value.

Agile teams usually work in two-week grinds, and each of these sprints should result in certain features. However, in order to meet the business goals, these features must be usable. Without DevOps, it would be difficult to deliver something that brings incremental value. This is, for example, because agile teams often disregard automatic testing. But incorporating this DevOps principle ensures that the process is always moving in the right direction.

Likewise, if agile teams pay close attention to collaboration with customers, product management, developers and other parties, as DevOps suggests, they can develop a better product that has undergone review from various parties. That’s because DevOps can extend agile principles beyond the code to the entire delivered service.

The two approaches fit great together. Agile enables teams to produce minimum viable software and its features in a swift cycle, while DevOps ensures that the delivered product is of a higher quality. Such a complementary relationship wouldn’t be possible with the waterfall scenario where cycles last months or even years. While DevOps encourages integrational behavior, Agile relies on quick evaluation and incentives, so they come together in flexible solutions. This is fundamental throughout the high season, where priorities often shift fast and the needs of customers become more urgent than ever.

Close Oversight to Avoid Failure

As teams face increased pressure during Black Friday and Cyber Monday, errors become more likely. Deploying Agile together with DevOps guarantees there is a robust architecture that’s able to prevent failures.

DevOps practices allow agile teams to respond faster–not only to business requirements but also to any other problems that arise, too. When combined with the best practices of modularity and microservices, failures can often be isolated to a single system. Such a system can be fixed, replaced or completely removed with ease.

Organizations are increasingly aware of the importance of preventative analytics. For example, Slack integrations throughout the DevOps cycle can go a long way for early diagnostics. Agile teams incorporating DevOps can leverage continuous testing and monitoring that bring both constant progress and quality assurance.

Companies such as Etsy, a global handcraft and vintage marketplace, saw a lower occurrence of errors and disruptions after exchanging the waterfall model for a more agile approach. The organization relied on DevOps structures to deploy a fully automated pipeline and assured that continuous delivery practices secured a higher number of daily deployments–with fewer errors at the same time.

High Levels of Synergy

As mentioned earlier, DevOps has many convergence points with Agile, which is why they form such powerful partners with little chance of clashing. One of the fruits of their synergy has crystallized in the form of Infrastructure as Code (IaC).

IaC in a modular system, as promoted by DevOps, brings organizations remarkable benefits, as significant changes can be rolled out across the entire network typology with a single code change in minutes. In the context of feature requirement (or a failure), such velocity is instrumental to leveraging seasonality. This way, developers can alter entire environment configurations, regardless of the initial stage, and thus avoid tedious rebuilding.

Teams looking to optimize their performance shouldn’t be deciding between Agile or DevOps. In fact, organizations will find that their powerful combination can equip them much better for the seasonal peaks and help them deliver greater results than ever.

— Ain TohVri

November 22, 2019
Designing a Future of Innovative UX, Without UI

The long-fabled future of a world without traditional human-to-computer interactions is no longer science fiction. It’s a reality. Consumers all across the globe are wearing computers on the wrists, talking to artificial intelligence to check the weather and monitoring body functions through micro-devices. This is just the tip of the iceberg of what is possible through creative technology design, and yet the masses still see holograms and implantations as far-off science fiction, rather than the very close and possible future.

Designers and developers are leading the charge into the future, but not without their fair share of challenges and hurdles. With a blank slate of endless potential, how can we design for a simple UX while still having a visually appealing product? Moreover, what if, in the near future visuals are not required at all?

The New Frontier of Design

Holograms, audio controls, wearables … new technologies have the potential to disrupt the screen entirely. Society is teetering on the edge of the uncanny valley of a world without traditional UI, and designers must take the reigns.

Take audio design, for example. Voice technology and virtual assistants are already rendering many screen functions moot. Technology leaders such as Apple, Google, Samsung, Amazon and beyond are pouring resources into voice, creating a screenless interface simple and intuitive enough for any consumer to use. Google, for example, has made creating an app on the Google home quite simple though Google Console. Designers can have fun building the tone, language and personality for the app easily in as little as five minutes. While it’s true that we live in a world where some still battle this technology, it’s also the case that audio assistants are carving out a place for themselves in our home and our lives. The technologies powering audio controls are being developed and refined quickly, so learning how to design for it now is key.

Itcher, a recommendation engine that helps users discover movies, music, television shows, bestselling books, games and more, requires no UI at all. The voice-powered platform allows users to ask their voice assistant for a recommendation, prompts them with questions to better understand his or her tastes and preferences and leverages artificial intelligence to provide optimal recommendations.

Designing for audio navigation is not necessarily fully intuitive. More so than with desktop screens or handheld devices, interactive audio treads on the brink of the uncanny valley—imitating humans in many respects but still not quite convincingly human. Audio designers like the team at Itcher must consider factors such as lexicon, pitch, volume, direction and most importantly, value for users to want to interact with a voice-controlled device.

One contributing factor for the rise of audio is the rise of wearable technology such as smartwatches, AR smart glasses and beyond. Designed to be connected to consumers at all times, wearables are blurring the lines between technology and its users. As the wearables become better optimized for everyday use, the screens will shrink, and likely disappear altogether. This will make audio design more valuable than ever before.

While people are still going to be very dependent on traditional screen interfaces for quite some time (especially for complex tasks), there is no doubt the world is hurtling towards a future where screens are no longer a necessity, but an option. Designers and developers no longer have the luxury of waiting for the next big thing to enter mainstream—it’s here.

Focus on the U (User)

No matter if you are designing for a wearable, a voice assistant or a mobile app, customer feedback is an important aspect of innovation and design that often gets overlooked. Of course, the first thing a designer must consider should always be the end user—know the business and customer requirements so they are making the best product that will fill the needs of both. A designer that does not approach a project by asking many “why” questions of both the company and the target audience will not have the information they need to build the best and simplest interface she or he can.

Innovative thinking is important at all points of the design process, but to ground design in reality rather than getting trapped in science fiction aspirations, designers must have both quantitative and qualitative research to better understand not only the functionality success of a product or platform, but also the user sentiment. This requires heavy interaction with stakeholders (both internal and external) at every stage of the design process.

Bring customers in to soundboard ideas, test products and platforms and engage on a deeper level with the business. At Shutterstock, we speak to customers on a regular basis, hold focus groups, and use user testing platforms to ensure we have customer feedback at many steps along the way. It’s vital for customers to be able to interact with products and platforms along with designers. This allows for greater creativity and most importantly, simplicity in UX.

Focus on Simplicity

Shakespeare’s quote “Brevity is the Soul of Wit,” from Hamlet, is typically interpreted for works of writing, encouraging authors to be concise. However, this is also true of all creative endeavors, writing, art and design. Making things simple (concise) takes hard work, it takes time, it takes skill and giving yourself the space as a designer to do that work is critical to finding success in creating a “simple” design.

Designing something simple is one of the hardest things every designer faces. Sure, it’s easy for the designer to come up with a simple interface that answers the customer’s needs at a cursory level, but often there are additional requirements or stakeholder feedback that one also has to take into account. It’s important to remember that product, UX and engineering are all parts of an equal trifecta. Designer and developer questions, as well as conversations with real customers about how to solve a user’s need, will only make a product better in the long run. The challenge for designers is balancing immediate feedback from their company and users with forward-thinking innovation.

It is often not intuitive for both businesses and consumers alike to imagine a future that pushes the boundaries of what exists. Designers are uniquely positioned to bring abstract innovation and technology to the masses through simple and intuitive UX.

— Erin Essex

November 22, 2019
Why Salesforce?

Over the past 20 years, Salesforce has become the fastest growing enterprise software platform, and a market leader in sales, service, marketing, integration and custom application development among other areas. Although there are many reasons why a company might adopt or consider using Salesforce, the overriding value proposition is that it provides tons of capabilities out of the box, releases new capabilities at an ambitious pace and allows for virtually endless customization and innovation.

Unlike with traditional platforms, there’s no need to manage infrastructure for computing, data, network or security, and no need to invest in generic system administration that adds no value for the organization. Add to this the accessible and supportive community and learning culture based around Salesforce’s Trailhead, which enables rapid onboarding and skill development, and it’s easy to see the attraction of this platform.

Salesforce’s mantra throughout the 2000’s was “Clicks not Code,” and their logo was the “No Software” image which became embodied as SaaSy, Salesforce’s first mascot. You could be a declarative developer using clicks alone and build a data model, business process automation and user interface using drag-and-drop interfaces. This opened the door to an entire new job specialization: Salesforce admins and app builders—citizen developers who were empowered to directly customize the platform to meet their company’s needs.

As Salesforce grew, they recognized the need to enable custom coding on their platform and launched Apex Triggers, followed by Apex Classes and Visualforce in 2008. Admins continued to configure rich business capabilities, while low-code developers and professional developers were empowered to create customizations beyond the scope of what could be built declaratively. Initially, the code developed on Salesforce was rudimentary: a few hundred lines here and there to accommodate unusual calculations and data processing. But gradually, companies began to accumulate tens of thousands of lines of custom code, especially as legacy applications were migrated from other systems. Needless to say, the “No Software” motto no longer strictly applies.

In the decade since the launch of Apex, 150,000 companies have adopted Salesforce and moved a staggering number of legacy systems onto Salesforce—which now processes billions of transactions per day. Many of these transactions are part of the core CRM and customer service applications, but a huge number relate to custom applications, making Salesforce one of the largest Platform as a Service (PaaS) providers.

How is DevOps Different for Salesforce?

As mentioned before, Salesforce provides both SaaS (Software as a Service) and a PaaS (Platform as a Service). By contrast, much of the focus elsewhere in the DevOps world is in moving from on-premise infrastructure to using IaaS (Infrastructure as a Service). This diagram shows an illustration of the differences between these four modes, and how they represent progressive simplifications of what companies themselves have to manage.

Figure 1: On-premise systems require you to manage all of the resources yourself. IaaS, PaaS and SaaS delegate progressively.

Managing the Salesforce development lifecycle requires a unique skill set and approach compared to most other platforms. You can say the DevOps movement was jointly realized by both developers (working in traditional languages such as Java) and system admins or operators (working on traditional infrastructure such as servers and databases). There are thus a vast range of well-established tools and techniques in the DevOps world for developing and deploying code, and for managing and updating infrastructure. Unfortunately, almost none of them can directly be used for Salesforce.

To illustrate this, let’s look at how teams would manage infrastructure using AWS, or deploy an application onto Heroku (a PaaS product also owned by Salesforce). Every aspect of AWS infrastructure can be represented using JSON configuration. JSON can be used to define which AWS services are used, which data centers they are running in and how they’re configured. The AWS CLI (command-line interface) can be used in a continuous integration tool such as GitLab CI to automatically deploy updates to AWS infrastructure every time the JSON configuration changes.

Similarly, Heroku provides a platform to deploy custom application code (in Java, PHP, Ruby, etc.) and to specify which services (such as Postgres databases) are needed to support that application. The Heroku CLI can be used to automatically update the application whenever the codebase changes. Heroku also provides a tool called Pipelines that allows you to visualize and manage the entire development lifecycle without requiring a third party CI tool.

AWS’s infrastructure as code approach is a delight for sysadmins, since it allows them to track changes and automate updates. Similarly, Heroku removes the vast majority of the setup and dependencies that developers would need to deploy their applications. Heroku provides a true PaaS that is ready to receive custom-coded applications built in a traditional way: from the ground up.

Although we can say Salesforce’s Lightning Platform is a PaaS, it actually works extremely different from true PaaS systems such as Heroku. The Lightning Platform allows you to write custom server-side or client-side code, but that code can only run on Salesforce. Although it allows you to define custom database tables and fields, that schema cannot be loaded into any database other than Salesforce’s. These customizations are effectively just changes to the configuration of one Salesforce instance. It’s therefore more accurate to say every customization you can make to Salesforce is basically a Salesforce configuration change.

Salesforce is actually just a big application that happens to allow for infinite customization. But this means the tools used for managing other IaaS and PaaS products cannot be used to customize Salesforce. Fortunately however, the release of Salesforce DX means the techniques and principles used with other technologies can now be ported over to Salesforce. That is the focus of the book Mastering Salesforce DevOps that I’ve recently released.

DevOps in the Salesforce World

Moving from on-premises CRM software to Salesforce removes the need for servers and manual software upgrades. Consolidating customer support and online community management onto the CRM platform removes the need to integrate sales, support and community applications. Migrating legacy applications onto the Salesforce platform allows those applications to share data and processes with the rest of the business.

Because moving to the cloud dramatically simplifies many of the challenges in delivering IT functionality, many Salesforce customers have been able to innovate quickly without having DevOps practices in place. But even without the hassle of managing servers and building software from scratch, the inevitable growth of complexity eventually causes companies to struggle with issues such as organizations getting out of sync and delays in deploying functionality.

Salesforce solves so many IT headaches, it’s almost easy to overlook the fact that it’s created some new ones.

Salesforce has unleashed the ability for companies to focus on their core competencies, instead of struggling to provide basic IT services. Salesforce admins and developers are empowered to spend their time directly creating business value. But with multiple Salesforce admins and developers working at companies year after year, more and more Salesforce customers are finding themselves drowning in all of that business value.

If you have tens of thousands of components, each providing business value, but no systematic way of tracking, managing or deploying them, you now have a new form of business pain and chaos. Far too much of a good thing.

What is Salesforce DX?

Salesforce DX (Development eXperience) is an initiative begun by Salesforce in 2016 and launched publicly at Dreamforce 2017. The goal of the initiative is to re-envision the development experience on Salesforce with a focus on how to empower their development community with the tools and processes needed to develop safely and effectively.

Salesforce DX is a very broad initiative, including several main teams at Salesforce focused on environment management, custom coding, developer tooling, APIs and more. Although the Salesforce DX teams tackle very diverse needs, their main focus (at least initially) is to improve the developer tooling and development lifecycle.

The tools and capabilities included in Salesforce DX are made available for free to all Salesforce customers. In that sense, Salesforce DX is an important complement to Trailhead, Salesforce’s free, self-paced, gamified, interactive learning platform. Trailhead is also a major investment for the corporation. Together these are intended to support the growth of skilled Salesforce professionals and to make it easier for those workers to build and innovate on the platform. These multi-year strategic initiatives help ensure that talent shortages and worker inefficiency will not be the limiting factor on the company’s aggressive 30% annual growth.

Summary

Salesforce is an extremely powerful and versatile platform. But it’s unique in many ways and it hasn’t been easy for professional developers to adapt their tools and techniques to working on this platform.

Salesforce DX is a major strategic initiative to ensure that industry best practices such as DevOps and modular architecture are possible on the platform. But making something possible doesn’t mean the process is straightforward. It’s still early days for most teams working to enable DevOps for Salesforce. Fortunately, there is an enormous amount of research and experience that can be borrowed from the broader IT industry. Combining a flexible, powerful platform like Salesforce with the speed and reliability of DevOps processes can open up entirely new levels of efficiency and effectiveness for teams.

— Andrew Davis

November 21, 2019
The Phoenicorn Project
The Unicorn Project is the sequel to the ground-breaking book The Phoenix Project. Most people know the story about Bill the IT manager at Parts Unlimited, who is tasked with fixing a derailed IT project or having his IT department outsourced. The Unicorn Project takes it a step further, with developer and architect Maxine in the starring role, helping Parts Unlimited to develop new products in new markets–the riskiest combination of all. This requires gambles that are only justified when the organization can experiment rapidly, break the rules and deviate from the processes that govern their current–but diminishing–business.

The content of the book revolves around five so-called Ideals:
1. Locality and simplicity in our code and organization.
2. Focus, flow and joy in our work.
3. Enablement of improvement and achievement.
4. A culture of psychological safety.
5. A relentless focus on our customers.
These are a natural progression of the themes in The Phoenix Project, in particular the first and third of The Three Ways of DevOps:
1. Attending to the performance of the entire system, as opposed to the performance of a specific silo of work or department (related to Ideal 2).
2. Creating short feedback loops, enabling necessary corrections to be continually made.
3. Creating a culture that fosters continual experimentation, taking risks and learning from failure; and repetition and practice as the prerequisite to mastery (related to Ideal 3).
Topics related to the first four Ideals have been explored in other books in the same family of publications, in particular The DevOps Handbook. The last Ideal–customer focus–is for me the most significant addition. It really draws attention to how IT and business have to converge and even merge in order to develop new business models. Meaning that those people-previously-known-as-IT have to up their game and develop a much deeper understanding of the jobs customers need to get done. The business value of DevOps has interested me since I got involved in the movement, and I’ve captured some thoughts in a white paper for the DevOps Agile Skills Association.

It’s another easy-to-read, fast-moving and plausible story that will undoubtedly inspire the DevOps community. So far so good. But as a facilitator of GamingWorks’ The Phoenix Project DevOps game, it got me thinking how I can incorporate the Five Ideals (and more) into this highly-interactive game.

My first analysis is encouraging. Starting with the last Ideal and working backwards, the game–that is based on the book–has strong focus on the customer while at the same time ensuring that technical debt (another key topic in the book) is kept under control. Psychological safety comes up a lot in the game–when it’s clear that somebody isn’t really participating, there’s a time out to discuss the importance of people feeling safe to share their concerns without fear for their reputation and position. Continual improvement is hard-wired into the game, with retrospectives after each sprint, leading to hypotheses and experiments for the next sprint. As for focus, flow and joy in our work, we encourage the participants to visualize the value stream and see how work flows from role to role. There’s also plenty of fun, particularly when they hit their revenue targets.

Locality and simplicity in our code and organization is probably the least dominant Ideal in the game–it’s a non-technical game so there’s no coding, and it’s a small team so almost by definition it’s local. We do discuss, however, the challenge of scaling–which is when this Ideal comes into play. I’ll experiment with the Five Ideals in my next The Phoenix Project games so watch out for further blogs.

The Five Ideals triggered another association. I was lead editor of the upcoming ITIL 4 High velocity IT module. This is about how digitally-enabled organizations place higher demands on IT, requiring different ways of thinking and working. One of my contributions is the concept of five Key Behavior Patterns:
1. Help get customers’ jobs done–helping customers become who they seek to become.
2. Keep raising the bar–taking things to a significantly higher level.
3. Trust and be trusted–as professional knowledge workers in a healthy workplace.
4. Accept ambiguity and uncertainty–not scared of not knowing an answer.
5. Commit to continual learning–all as part of their daily work.
I believe many practitioners want to work in organizations that foster these behaviors. As such, I see them as aspirations. As marketing legend Seth Godin said, “people like us do things like this.” You can see the similarities with the Ideals:
- “Help get customers’ jobs done” equates with “A relentless focus on our customer.”
- “Keep raising the bar” and “Commit to continual learning” are close to “Enablement of improvement and achievement.”
- “Trust and be trusted” is the same as “A culture of psychological safety.”
It is so encouraging to see similar thoughts emerging within different communities. This will hopefully break down the silos that exist in the IT industry and bring us all closer together. All too often, conferences and communities are organized by specialization. They stick safely to themselves and reinforce their stereotypical stories about other disciplines. Praise and power to the rebels who practice what they preach (T-shaping and multi-functionality) and liaise with their industry colleagues, organize joint conferences etc. Evolution thrives on diversity and it would be a shame to see inbreeding in communities, leading to extinction. Let’s not only read The Unicorn Project but also do something useful with it.

— Mark Smalley
November 20, 2019