devopsy.kinsta.cloud

Tag: software development

  • Days of the Week

    Days of the Week

  • How the Pandemic Renewed Respect for the Agile Mindset

    How the Pandemic Renewed Respect for the Agile Mindset

    Agile methodologies were introduced to us around 10 years ago. In that time, we have seen technology advance at breakneck speed and faced many new challenges in the way we work; arguably, never more so than in 2020.

    However, despite the abundance of change we have experienced, the core values of Agile have stood the test of time. Yes, Agile has been agile, adapting to fit our individual needs. But its values and principles are so strong that they ring true for each and every organization touched by this way of working.

    It’s impossible not to mention 2020, and how Agile shifted and adapted to help us through this turbulent time. What did we notice? Well, much like the renewed gratification you feel for your automotive breakdown service when your car gives up the ghost for the first time in years, we noticed that the COVID-19 pandemic changed Agile in a different way: by providing renewed respect for the agile mindset.

    What is the Agile Mindset?

    When we talk about the agile mindset, we’re referring to the core values that combine to establish a set of attitudes held by those embracing agility. There are four of these core values: Make people awesome, deliver value continuously, make safety a prerequisite and experiment and learn rapidly.

    All four values are of equal importance, but they each have a unique impact on the mindset. For example, making safety a prerequisite means providing employees with the confidence to experiment, innovate and not fear mistakes or failure – these are all natural on our path to achieving new things.

    Additionally, delivering value frequently benefits both our customers and employees. It takes away the pressure of bouncing from one milestone to the next and adds value to a consistent level of delivery. Our customers are happy and grateful, seeing constant developments and improvements to what we provide them.

    The final two values (make people awesome and experiment and learn rapidly) are those that were particularly important during the change and adversity of 2020. Let’s dive into them.

    Making People Awesome!

    Agile methodology is massively focused on people: both employees and customers. When we work in an agile way, we want to enrich our people and give them the freedom to express themselves and show off their skills. This, in turn, helps the entire organization to deliver excellent service to our customers.

    With such a focus on people, the mandate for remote work brought about by the COVID-19 pandemic introduced an obstacle to agile ways of working. Our only human interaction with colleagues and customers was through a screen, which, let’s be honest, just isn’t the same as interacting in person.

    Therefore, employee engagement has become more important than ever. Virtual interactions make it harder to truly see how somebody is feeling about work – whether they are happy or unhappy, motivated or unmotivated. It’s important to check in with everybody regularly; ask them directly how they’re doing and encourage honest answers. Keep the Agile team spirit alive by making an effort to break the screen barrier during meetings.

    There are benefits of remote working for Agile, too. For example, it offers a fantastic work-life balance. When employees are less pressured by commuting or less exhausted by the constant energy of the office, they are happier. And happy employees make happy customers!

    So, the value of Agile’s ‘Make people awesome’ tenet became even more important and respected in 2020. We realized that we must optimize our employees so that they can bring their best selves to work, even during uncertain times.

    Experiment and Learn Rapidly

    This been an incredibly important value, hasn’t it? For lots of us, remote working was a brand-new experience. Perhaps we’d spend the odd day working from the sofa to let the boiler engineer in, but five days a week from home – unprecedented!

    Learning and innovating at speed has been key to being able to work effectively from home. We’ve been truly embracing this aspect of the Agile mindset. But how? Essential to agility is the ability to critically review how we operate. Almost half of businesses say they do this every quarter (43%) and 21% say they do this every six months.

    Regularly reviewing and analyzing how we work helps us to identify areas of improvement, areas that we could experiment with innovation, and gaps in our knowledge. Without this, we’d be walking through our working life with a blindfold on, and likely not be able to pivot in the face of change and adversity.

    A great example of the renewed respect given to the ‘experiment and learn rapidly’ value is the COVID-19 vaccine. It normally takes years to develop, test and roll out a vaccine, but we’ve seen this happen in just months.

    You wouldn’t be mocked for not knowing that the vaccine was being developed way before the pandemic of 2020. In fact, scientists began preparing for ‘Disease X’ over four years ago. They analyzed the response to the Ebola outbreak of 2014 and recognized a need to develop a vaccine that could be used quickly, for an infection that did not yet exist.

    Fast-forward to COVID-19, and while scientists still had lots of work to do to prepare a specific vaccine (designing; initiating the correct immune response; testing and more testing and roll out), their agility, started years ago, meant they had a strong foundation on which to build.

    This value has definitely received more recognition in recent months. This is great, because it’s going to be vital for the continued response to our changing world. We must continue to analyze our response to new ways of working and adapt based on what we find. Are you already experimenting and learning to be ready for the new normal?

    An Essential Mindset for Remote Teams

    The core values of agility have had a huge part to play in our adaptation to the changes we saw in 2020. And many of us have embodied this agile mindset without even realizing – we’re calling this ‘unconscious agility.’

    If you’ve tried something new in the face of remote working, made a conscious effort to deliver value continuously rather than in big chunks, checked in more regularly with your colleagues or relayed the message that mistakes and failure are a natural part of innovation, you’ve been working in an agile way. Even if your organization doesn’t consciously work agile, you can still embody this mindset.

    And it’s going to be key for the new normal – we should already be looking ahead to that. We didn’t have time to prepare for what came in 2020 – it was somewhat of a surprise. But we do have time to prepare for what comes next, and the Agile mindset is key for tackling this successfully.

  • Scarf Gateway Tracks Open Source Software Usage

    Scarf Gateway Tracks Open Source Software Usage

    Scarf today exited stealth to launch a gateway that makes it simpler for individuals or organizations to track usage of open source software.

    Avi Press, founder of Scarf, said the Scarf Gateway is an online service through which providers of open source software can redirect downloads from a wide range of repositories and registries. That extra step then provides those entities with data and insights about the type of individuals and organizations using their software.

    Fresh off raising initial seed funding, Press said the Scarf Gateway will make it easier for developers of open source software to employ multiple repositories to distribute their code, because they won’t become overly dependent on metrics shared by the provider of a single repository.

    Most of those registries today only provide download counts for packages. Scarf Gateway will provide insights into, among other attributes, downloads by version, platform, location, company and cloud environment(s).

    The majority of organizations making open source software available don’t have a lot of visibility into how their offerings are being employed until organizations using the software sign up for commercial support. Armed with more insights into how their software is being used, many of those providers would be able to more proactively collect feedback, in addition to promoting their commercial services.

    Many open source project maintainers have a difficult time prioritizing what capabilities to add or fix, simply because they don’t know enough about how their software is being employed, Press said.

    The ability to more granularly track usage comes at a time when tensions over open source monetization and compensation are rising. In many cases, providers of open source software are changing the way they license their software to prevent cloud service providers from reaping most of the benefits of their work. Startups, typically funded by venture capitalists, make their software available as open source code in an effort to build a customer base on the strength of their support services. Cloud service providers have discovered they can take the code these vendors create to launch their own service. Those cloud service providers may contribute code to the project, but the startup vendor doesn’t typically enjoy any of the economic benefits.

    Many individual contributors don’t work for a company that sponsors their contributions, and also want to be paid for their efforts. A survey of 4,440 developers who actively participated in open source projects in 2020 finds more than half (54%) of respondents said they feel individuals should be paid for their contributions to open source projects. The survey, conducted by Digital Ocean, a cloud service provider, found only 14% of respondents are currently paid for their contributions.

    It’s too early to say what impact the ability to track usage of open source software may have. Many individuals using open source software may prefer their current level of anonymity. Others might participate more in a project if engaged. Like it or not, however, large swaths of the open source community want to know the degree to which their efforts are making an actual difference.

  • 3 Common App Modernization Pitfalls to Avoid

    3 Common App Modernization Pitfalls to Avoid

    Application modernization has been a consistent theme for organizations year after year. Businesses today still rely on their legacy applications for the majority of their revenue generation, from mainframes to Java EE. New digital transformation initiatives and mandates have pushed application modernization back to the forefront, as a new generation of automation platforms and best practices are helping these organizations avoid the pitfalls of the past and accelerate their journey to cloud-native architectures.

    Here are the three most common application modernization pitfalls and how to avoid them.

    Lack of a Clear Organizational Destination

    Application modernization journeys need a north star – a destination point that development, infrastructure and DevOps teams can rally around. In order to establish a clear, unified direction, everyone must share a precise understanding of that destination. That journey starts by defining a set of business goals you are trying to accomplish (such as cost savings, business agility, resiliency and/or competitive advantage) that can guide and prioritize the technical decisions.

    Next, you need to decide which application modernization technologies, platforms, and infrastructure will help you deliver on those business goals (e.g., Red Hat OpenShift, Kubernetes, serverless, hybrid, private or public cloud, etc.). Setting common architectural standards will help avoid disparate future stacks and ensure a common shared platform for microservices architectures. For example, if an application starts with Java WebSphere, should the goal be to move to a modern container framework like Spring Boot, or a specific microservices Java framework like Quarkus? To avoid analysis paralysis and internal drag, set tangible objectives and targets that prescribe how teams are expected to prioritize, select and modernize these applications and how their progress should be measured, communicated and shared.

    Taking Shortcuts – Don’t Fall For the Old ‘Lift and Shift’

    The more one modernizes, the more value one gets from the cloud – but not all modernization efforts are created equal. For example, ‘lift and shift,’ putting a monolith into a container and application refactoring into microservices are all called “modernization.” But only refactoring allows enterprises to realize the true benefits of the cloud, including elasticity, regained engineering velocity and cost savings. Don’t fall into the quick-and-easy modernization temptation with lift and shift or brute force containerization. You will still have a monolith, in the end; you’ll simply end up paying more and trading one set of monolithic problems for another.

    Not Bringing Everyone Along – No Developer Left Behind

    Modernization is never just about technology. Modernization requires that your team’s processes (for example, agile practices), structure (vertical domain teams as opposed to horizontal teams – UI team, for instance) and skills must all adapt. The organizational culture needs to grow to adopt DevOps and cloud-native best practices to take advantage of the deep development, build, deployment and operational benefits that are available. Take the time to train and educate so everyone is brought along. Modernizing legacy applications can introduce a whole new set of technologists, responsible for those applications and the supporting infrastructure, into the cloud-native world. This is a win-win for everyone.

    In other words, the organization needs to have crystal clear answers to the where (destination), why (path) and how (team) to ensure success and avoid the three most common pitfalls of application modernization.

  • Dual-Track and Low-Code’s Excellent Adventure

    Dual-Track and Low-Code’s Excellent Adventure

    Digital transformation can be a bogus journey. Here’s how low-code and dual-track transformation efforts can help.

    Large-scale enterprise digital transformation can take a while. Transformation initiatives like cloud migration, mainframe upgrades or refactoring monolithic systems could easily take five to seven years. Yet, this slow pace is at direct odds with today’s digital innovation standards and the pressure to release software rapidly.

    COVID–19 has amplified this drive to innovate, especially complicating matters for non-cloud-native companies, which struggle to attract and retain talent amid a developer shortage. Not to mention, developers are experiencing surmounting pressure to perform, which could result in burnout if not supplemented by new strategies.

    To maintain agility while diligently advancing multi-year digital transformation projects, some proponents argue for a dual-track approach. This approach uses low-code platforms to increase operational agility (the fast track) while digital transformation initiatives (the slow track) progress methodically on the side.

    I recently sat down with Jay Jamison, chief product and technology officer at Quickbase, to discuss today’s digital transformation concerns. According to Jamison, implementing a dual-track approach affords great rewards, but it requires an organizational cultural shift. He recommends establishing centers of excellence (COE) to evangelize low-code practices and encourage citizen development, enabling subject matter experts of lower technical acuity to automate operational workflows and operations.

    Dual-Track Transformation

    “Digital transformation is an important element for businesses of all sizes,” said Jamison. However, mainstream digital transformation projects, like rationalizing an ERP, moving to the cloud, or upgrading a network, can take an exorbitant amount of time and money. It’s hard for IT to meet business requirements. “We really need a second track of digital transformation — one that is more agile.”

    Jamison likens dual-track transformation to chipping out “small rock” projects from the larger digital transformation cliff. This strategy focuses on streamlining realizable goals in shorter timespans, empowering teams of people — with subject matter expertise — with tools to transform and digitize their workflows.

    Low-Code’s Role in Dual-Track

    A recent study conducted by Harvard Business Review found that 80% of executives “strongly agree” that a dual-track approach is essential for digital transformation. Yet, only 31% of companies have adopted said approach.

    To realize a dual-track process, Jamison recommends low-code for its “low-risk, high-throughput capability.” According to Jamison, low-code enables the edge of business to quickly adapt new solutions and run them in parallel with big rock digital transformation. For example, non-engineer government employees in a Texas county recently used a low-code platform to help distribute coronavirus vaccines.

    For a citizen developer culture to flourish, “you really need to have all hands in the pile,” Jamison said. IT stakeholders and business members must collaborate and work together to bring low-code to fruition. This process combines setting goals, outlining guardrails, and forming learning groups with progressive leadership at the helm.

    Introducing Centers of Excellence

    Naturally, many factors can affect IT innovation (leadership commitment, collaboration, organizational expectations, etc.). But arguably, the most common roadblock to digital innovation is culture. In fact, 45% of executives report that organizational structure is a major transformation challenge.

    That’s where centers of excellence (COE) come in. No, centers of excellence is not an archive of Bill & Ted’s adventures. COEs aim to fix the cultural barrier by encouraging progressive leadership, collaboration, and innovation. The Playbook for Centers of Excellence defines COE as “a concentration of expertise and resources in a specific capacity, designed to maximize the focus on a specific project.”

    A COE aims to foster shared understanding between IT leadership and business leadership. In Jamison’s words, “a center of excellence provides the ability to utilize low-code to sustainably build and empower applications in a safe, sustainable way.”

    Building a Most Triumphant Center of Excellence

    The COE goal is to empower business users to innovate. Yet, COE is a governance framework, which seems a bit contradictory. So, how does an organization go about implementing it? Well, there is a range of how companies adopt COE, but Jamison shared some common steps:

    Step one is establishing your goals. This could be increasing productivity, enhancing customer satisfaction or improving service reliability. Step two involves inviting the right people and perspectives — the key personnel to own the COE program.

    Next, develop transparent policies around security. This helps preempt concerns around data security and compliance. Finally, now that goals and rules are established, communicate the initiative across an organization.

    Throughout this process, investing in training is paramount. There are many project management philosophies for encouraging low-code adoption, but Quickbase recommends a training and education team to “roll out information and goals in a structured, targeted way”. They also encourage a community focus — user groups, internal events and knowledge-sharing initiatives can build excitement within the employee base.

    Benefits of Low-Code Enabled COE

    A COE aims to bring disparate parties to the same table. For example, companies could hold what Jamison described as an “appathon,” in which business executives present their goals, like rolling out new installations or driving 5G deployments. Then, priorities are assigned to individual groups, who apply domain knowledge to construct applications.

    Setting up a COE by accelerating ground swell innovation “catalyzes the dual-track innovation that we think is really key,” said Jamison, reducing the mean cycle-time to introducing new features. Low-code could also lower the integration barrier, enabling connectivity with CRM systems, or connecting with big, heavy systems like SAP or Oracle.

    Jamison also described low-code platforms as encompassing “the spirit of microservices,” as they introduce a relatively lightweight application process and easy deployment model. Furthermore, if teams are tapped into the same PaaS, they receive data reusability benefits, he said. “Everyone can utilize the same data in a way that’s relevant for what they need.”

    Citizen Developers: Bogus or Non-Bogus?

    Personally, I think it would be great if the future of application development ushered in a democratizing effect, allowing more people to participate in software creation. Yet, the skepticism around a fully working citizen developer model is hard to shake.

    The “citizen developer” seems to have a negative connotation among coder elites, too. As one reader commented:

    “So top talents are overpaid extravagance and bored secretaries working from home will be driving innovation by writing no-code workflows, am I right with understanding what is written here?”

    I pressed Jamison on the reality of the citizen developer concept in practice and how he responds to naysayers. He jokingly equates it to a witch hunt.

    “Saying that only people that get to participate in software development is professional software developers is ridiculous,” he said, noting the importance of subject matter expertise.

    According to Jamison, companies often fail when they attempt to transmit subject matter expertise to developers. Organizations would rather teach subject matter experts low-code than teach the subject matter to programmers, he noted. It’s not hard to imagine why allowing subject matter experts to build their own workflows for specialized fields, such as accounting, natural science, or mathematics, makes sense.

    Regardless, Jamison adds there’s nothing really to worry about — “The full stack developer isn’t going anywhere.”

    Dual-Track Digital Transformation? Excellent!

    A dual-track approach combines large-scale transformation and rapid-cycle development. By separating “small rock digital transformation” out from the larger puzzle, organizations can increase operational agility to meet business needs a bit quicker.

    Establishing a center of excellence seems to boil down to bringing people together to share ideas, collaborate, and drive better productivity. If anything, this way of thinking will help operationally-intensive industries consider how they can automate parts of their business.

    Ultimately, the goal here is to enable organizations to collaborate better and drive better productivity. By empowering more creators, you can start seeing creativity at the edge of business emerge too, said Jamison. If that involves citizen developers creating event-driven workflow automation, so be it. “We invite people to take on the challenge,” said Jamison.

    Hopefully, then, business and IT can be excellent to each other.

  • Real World Development

    Real World Development

  • The Benefits of Open Source Technologies

    The Benefits of Open Source Technologies

    Over the last 20 years, we have seen the proliferation of open source software and technologies become part of commercial offerings, either as part of the technology stack that powers the product or part of the product itself. Although these two use cases are largely different, both have come to showcase just how useful open source is, not only for the developer community but for businesses as well.

    One of the benefits of working with open source technologies or projects is the free sharing of ideas. Open source brings people together to brainstorm and develop a common piece of technology. The shared love for what they do crosses cultural and geographical boundaries. One of the best-known – and arguably, most important – open source projects is the Linux kernel and operating system. Linus Torvalds released the first version of the kernel to the open source community through a mailing list, sparking worldwide collaborative development and widespread adoption.

    The passion for open source stems from a deeply rooted belief that contributors are solving an important problem, and that they are working towards the betterment of the world as a whole. In many cases, this is demonstrable. The Linux kernel now powers the majority of data centers worldwide, serving billions of users. But it’s not only that subjective passion that has played a key role in establishing open source technologies. The peer-review process of open source projects means that each code commit passes a rigorous, public audit and verification process. It has proven to be a superb way to increase code quality, reduce bugs and regressions and guard against vulnerabilities. Research has shown, objectively, that this process produces higher quality software than the standard, proprietary quality assurance and testing procedures.

    When selecting an open source technology to use in development, there are a couple of issues that need to be considered in order to safeguard the quality of your product. One key consideration is to resist chasing the latest technology fad. Instead, select the right technology for the job. It’s in developers’ DNA to obsess over new and exciting technologies, but this does not always mean that they are appropriate. While Kubernetes may be the right solution if you need to serve millions of users at Netflix-like scale while managing maintenance and cost, it likely isn’t the best choice for a new startup launching the first iteration of their product. In this case, speed of development and flexibility to make changes is more important than scalability (so, perhaps, the startup might be better off using Ruby on Rails or Django). These three projects all share an important characteristic: They have active communities that not only safeguard project features but also maintain them and offer peer-to-peer support.

    When implementing a new feature with a particular technology, this supporting community is very important, since they can greatly accelerate the development of your product. These are people who love what they do and want to see other community members succeed with the technology. In this regard, it’s also important to consider giving back to the projects and communities you benefit from. You can either participate in the communities and help others yourself, one-on-one, or contribute by sharing the improvements that you have made to the open source technology. If you have extended the API, for example, or discovered and fixed bugs, there is no reason not to contribute your changes back and further enrich the ecosystem. The credibility you build in this way can also be a benefit, as people will identify you, your team and your product as supporters of the open source project. It’s a virtuous cycle of collaboration with benefits that should not be underestimated.

    As you and your product become an important part of the open source ecosystem, either because you use open source components or because the core of your product is open-sourced, remember that trust is notoriously hard to build and very easy to lose quickly.

    Whatever one’s feelings are on this matter, the open source community is very careful about who they trust and can easily be alienated by a company that appears not to have their best interests in mind. In order to build trust, it’s important for teams to build credibility. Open source communities are largely merit-based, so unless the community sees proof of expertise and knowledge, they will not automatically trust a self-proclaimed expert. This credibility can even extend to the personal, like the many open source developers who have a large following on social media and lend their credibility to the projects and companies that they work for.

    Finally, when dealing with open source technologies, projects, and communities, it’s key to remember that developers are very sensitive to blatant sales pitches. Authenticity matters in the way that a project, company or brand presents itself. Open source is all about collaboration and community; it’s a relationship built on trust and mutual benefit, not profit.

  • Using IBM ADDI and VTP to Enhance Observability in Mainframe Development

    Using IBM ADDI and VTP to Enhance Observability in Mainframe Development

    Observability has always been a key factor in software development – not so much on the mainframe. However, the rise of the internet as the go-to platform for modern commerce has caused an explosion of transactional activities. There’s much more complexity behind the scenes than the request/response activity that used to be the usual interaction between a client and a single server. That necessitates ubiquitous observability as a mission-critical necessity for any modern application. As computational activities increase, so, too, does the risk of failure. One way to avoid disaster is to rely upon observation technologies that monitor not only runtime behavior, but design-time behavior, as well.

    Analytical Insights are Key to Application Modernization

    IBM observabilityModern computing activity goes well beyond simple request/response interactions. Today, one request can traverse any number of computing resources before resolving to a completed response. These resources range from application logic that’s encapsulated and then exposed to the network at a number of endpoints, to a diverse mixture of data storage mechanisms. And, if the application is part of an event-driven architecture in which messages are passed around to a large number of recipients, ancillary logic can be executed in ways that are unknown to the originating request. This can cause significant side-effects, nonetheless.

    Given the complexities inherent in modern applications running at web-scale, without ubiquitous observability, developers are flying blind. This is true not only for what we have come to know as commodity computing, in which applications are hosted on racks of X86 servers housed in private and public data centers but also for mainframe computing.

    The Emergence of Observation Technologies

    Over the last decade, a growing number of observation and monitoring technologies have emerged for network applications and database environments. Open source projects such as OpenTracing, OpenTelemetry, and Prometheus give developers insights into the runtime behavior of applications. Also, there are a plethora of commercial observability products that companies can buy to meet their specific monitoring and observability needs.

    At the development level, there are open source tools such as JSHint for JavaScript developers, PyLint for Python developers and FindBugs for Java developers, just to name a few. These tools can inspect static code and point out shortcomings. Some tools, such as .NET analyzers, will go as far as to correct code as the developer is writing it. The easiest bugs to fix are those that are never created. Correcting code as it’s being written brings a whole new dimension to design-time observability.

    However, while the X86 development community has benefited from observability tools for a while, their mainframe counterparts haven’t had it so easy. Commodity developers require nothing more than a click on a web page link to access any number of powerful observability tools, but mainframe developers have been starved for these types of tools.

    Until now.

    Applying the Principles of Observability to Modern Mainframe Development

    One of the big challenges of mainframe programming is getting a clear picture of application dependencies. Unlike programming under Java, C# or Node.JS, where external dependencies are defined in a central configuration file, or internal dependencies can be discovered with a code analyst tool, mainframe programming makes knowledge of dependencies a difficult undertaking. This type of design-time observation hadn’t really existed until technologies like IBM’s Application Delivery and Discovery Intelligence (ADDI) came along.

    An important thing to understand about dependencies in most mainframe computing is that they exist at both the function and executable levels. A developer can create a function, which, in mainframe parlance, is called a routine. That one routine might be used by dozens, if not hundreds, of other routines. Sharing code from a common resource is not unusual in X86 development, where binary libraries get shared all the time. However, in mainframe computing, these shared routines, along with file layouts and resource definitions, are physically copied in with the developer’s code using a technology called a copybook. Then, the code is compiled and linked to create an executable file. Once turned into an executable, the code becomes opaque. As a result, observing what’s going at runtime and design-time within the executable can be a herculean task. For many years, it was.

    Fortunately, you can use a tool like ADDI to overcome this obstacle. IBM’s ADDI provides important insights and analysis into the code that makes up a mainframe application. Both developers and system analysts can use ADDI to determine not only application dependencies but also to gather information about the execution environment; for example, scheduler settings and database configuration.

    As with X86 development where ancillary application intelligence, well beyond information about the source code, can be displayed in the developer’s integrated development environment, ADDI also makes it possible to report information in the developer’s IDE. Also, systems analysts can use the additional information provided by ADDI, such as the structure and segmentation of an application, to make decisions about how to refactor code when it comes time to make improvements.
    Providing a broader scope of observability into design-time activity is but one aspect of enhanced observability in the modern mainframe environment. ADDI also reports test coverage metrics. Knowing what and how much code has been tested is an important part of observability, even more so when used with tools that are focused on mainframe testing. One such tool is Virtual Test Platform (VTP).

    Leveraging Observability in the Mainframe Testing Process

    Until VTP came along, mainframe developers were flying blind – they had limited visibility into the transactions that their code used and the dependencies that their code relied upon. They had an enormous amount of work to do to figure out transactional behavior throughout the application. They spent a lot of time doing archeological digging in legacy codebases to get a clear idea of what was happening. And, it didn’t get any easier as testing demands on all types of applications, both X86 and mainframe, increased due to the rise in commercial activities on the internet at a global scale.

    According to Chris Trobridge, senior offering manager at IBM, “The testing load for developers has been increasing tremendously, and with the introduction of agile development processes, developers just can’t keep up with the testing load that they have.”

    VTP addresses many of these testing issues, both in terms of observability and quality assurance. “VTP really is observing what goes on at runtime, but without knowing anything about the application,” Trobridge said.

    VTP can monitor an application and capture and record all its interactions. This type of observability enables developers to make modifications and then run those changes against observed behavior. It’s a novel approach to mainframe development in which the benefits of both design-time and runtime observability are leveraged in a synergistic manner. In order to understand how groundbreaking this capability is, you need to remember that only five years ago, while X86 developers were enjoying a maturing set of monitoring and tracing tools; for mainframe developers, this type of observability wasn’t possible. Now it is. It’s a dramatic step forward.

    “VTP is a tool that enables developers to automate a lot of the testing, and, in turn, enables the recording of a transaction environment without having to know a lot about the environment,” Trobridge explained. “The recordings themselves can be played back rapidly without compromising security. This is an important aspect of VTP. This level of observability and interactivity really is something that mainframe developers have not had access to before. It’s a transformational technology in the mainframe world.”

    Technologies such as ADDI and VTP are a transformational approach to enhancing observability in mainframe environments. In many ways, the focus on exposing the granularity of computing at the routine level is a harbinger of things to come for all developers, in both the X86 and mainframe worlds.

    Taking a Function-Centric Approach to Observability

    Historically, for commodity developers, observing request/response behavior between services at runtime, along with identifying code quality and complexity issues at design time, allows for identification and remediation of most issues in software development.

    For the mainframe developer, request and response behavior is only the tip of the iceberg when analyzing application behavior. In a mainframe environment, one sub-routine might be used by thousands of other functions and subroutines at any given time. Commodity developers have been spared the burden of accommodating the function as the fundamental deployment unit because, until recently, functions were grouped into well-encapsulated deployment units, such as Java’s .jar file, .NET DLLs or, more recently, containers. If something went wrong, the error would generally appear in one of the deployment units.

    However, with the current rise of serverless computing, where the function becomes the primary deployment unit, X86 developers are in the same boat as their mainframe counterparts. In the past, for the X86 developer, one application might use a limited number of components that could have dozens, maybe hundreds of functions. Now in the serverless world, all these functions are exposed as standalone units, in which each function is the responsibility of a distinct team. We’ve gone from an ecosystem that has a limited number of deployment units to an ecosystem that can contain thousands. Fixing bugs can require hours of poring over logs and examining minute details just to determine the root cause of an issue.

    As mentioned above, this depth of dependency at the function level is something that mainframe developers have been dealing with for years. Fortunately, tools such as ADDI and VTP provide a high degree of observability for very complex mainframe applications at both runtime and design-time.

    It’s strange to say, but the large, function-based ecosystem issues that mainframe developers are solving today will become a growing concern for X86 developers in the very near future. In many ways, tools such as ADDI and VTP are setting an example for X86 tool development moving forward. The mantra, “You can’t fix what you can’t see” is as true today as it has ever been. That the mainframe community is at the vanguard of creating new ways to work in massive computing environments is a testament to the power of rejuvenation in the most unexpected ways.

  • The Chaos Mindset: Teaching Your Code to Cope

    The Chaos Mindset: Teaching Your Code to Cope

    Chaos engineering sounds alluring and exciting—it’s fun to experiment, right? But what some misunderstand about this approach is that it’s not about moving fast and breaking things. It’s about designing and introducing disruptions in the software production process that tests the resiliency of the code, much like crash testing in the automotive industry.

    If you think about it, this is the logical extension of the way developers like to think anyway: we’re designing software and systems for the real world. Shouldn’t they be able to handle real-world situations?

    To design effectively for the real world, developer teams don’t necessarily need to become chaos engineers. This isn’t about training for an entirely new set of skills. Rather, it’s about adopting a chaos mindset—a set of systems and perspectives that lets your team engage in controlled experiments to deliver products that can cope with whatever end-users throw at them.

    So, how do you know if your team is ready to adopt chaos engineering principles? And what should you do to help them?

    Chaos Engineering: The Basics

    This approach is all about using distributed systems as a “safe” experimentation environment, so that dev teams can ensure their products will withstand unexpected turbulence in production. Netflix engineers get the credit for coining the term, based on their experiences developing for Netflix on AWS.

    The fun part of chaos engineering is that it lets developers find loopholes in existing systems that allow them to develop for resiliency. By running experiments within existing platforms in a distributed environment, or within a sandbox development environment, teams can supervise and control the experiments without impacting end users.

    Some applications, such as transactional applications in finance or trading, should not be disrupted in production, and developers should test in a sandbox environment instead. Chaos engineering is very easy to put in place if your team uses containerization techniques already – which should be part of any modern DevOps CI/CD process anyway. It is common to setup a DevOps CI/CD pipeline to run a battery of tests on an application before it gets to the publish step. Within that testing step, you can easily add some disruptions; some may be harder to put in place and require a more elaborate environment.

    Dev teams can use existing environments and systems to show each other how unexpected issues could impact production of a software product, allowing them to collaborate and solve the coding mystery before it slows or stops production entirely.

    Applications designed by engineers are subject to common issues, whether the applications run on-premises or in the cloud. We can group these issues into the following categories: hardware limitations, sub system failures, external system failures and software bugs. While we could define more categories, these are the main ones, and the ones that are easy to test.

    Chaos Mindset: The Tao of Chaos

    Like Agile, chaos engineering is more than a set of activities and workflows—it’s also a state of mind. Your people and your culture must be ready and able to adopt chaos principles, as well as chaos processes.

    For the DevOps leader, adopting a new mindset might sound a little, well, vague. But this shift is based on concrete actions, not just philosophical musings.

    Consider an example from the world of cloud infrastructure: a mission-critical application that is hosted within a cloud service could be at risk for failure if, say, that cloud service is centralized in a single location, or within a limited number of microservices within the cloud infrastructure. But if the app is hosted in a distributed way, you can create greater opportunity for application-level availability and resilience, and you can test for that resilience within the existing production environment.

    This kind of distributed architecture isn’t brand-new for most enterprises, and, therefore, the process of developing applications in way that tests for availability in a variety of infrastructure scenarios also shouldn’t be a foreign concept. As a DevOps leader, you can build a culture of resilience-centric thinking by empowering your teams with the tools they need to adopt chaos-style testing, and then showing them how to build that thinking into every sprint and every standup.

    It takes work to train your teams, but you don’t have to do it alone. Netflix’s Simian Army has tons of tools and guidance; Facebook Storm and Amazon Gameday, both war room-type experiences to simulate failures in the cloud, also have helpful examples and ideas.

    Chaos Engineering in Real Life

    Deploying a chaos-inspired experiment strategy isn’t easy. Start with a simple set of operating principles as a guide for designing your chaos strategy:

    • Hypothesis: How we think the system should work, so we know when it doesn’t work.
    • Scenario planning: What are the possible failure events or crisis points that you know can happen in the real world?
    • Set up the experiments: Design and put in production the tests that measure the code’s strength.
    • Configure monitoring: Monitoring will help test the hypothesis and help you see, in real time, the impact and resolution of the disruption.
    • Make the experiment run itself: Automating the tests lets you continuously assess both performance and resilience.
    • Contain the experiment: You don’t want software that’s already out in the wild to get disrupted by your lab experiments, so make sure to design with guardrails in place.
    • Shut-off switch: Even when contained, have a shut-off switch to prevent the disruption from further impacting some users.

    The fundamental irony of chaos engineering is that it’s anything but chaotic. It’s a disciplined approach to breaking things that lets your team get smarter, faster, about how their products and applications will work in the real world. It’s a strategy that harmonizes beautifully with Agile practices. And even though it won’t make your apps bulletproof, it can definitely help you dodge the bullets of failure with greater grace and finesse.

    You must look at the software architecture as a whole to find the most appropriate chaos engineering testing tool for each portion of the architecture. There isn’t a single tool that can test all the components of your architecture; you will have to use different tools for different parts – and don’t forget monitoring! It’s only through proper monitoring that you can assess how the software is behaving in response to a disruption.

    Sometimes, the best approach, and the simplest one, is to build some chaos engineering steps into your software so that the software itself can be disrupted given specific inputs without exposing a security weakness in an obvious way. This is very much like how cloud applications create heartbeat and health check endpoints. We like to think about chaos engineering as a great complement to unit testing and integration testing, except we run it on production systems.

  • ARMO Aims to Simplify Microservices Security

    ARMO Aims to Simplify Microservices Security

    ARMO this week emerged from stealth to launch a platform to secure cloud-native workloads and improve visibility. The ARMO Workload Fabric technology attaches code within the memory invoked by a microservice to enable cybersecurity policies to be applied via a cloud service.

    Company CEO Shauli Rozen said the ARMO Workload Fabric is designed to enable IT organizations to implement a zero-trust architecture more easily within the context of a DevSecOps workflow.

    Fresh from raising $4.5 million in seed funding, Rozen said ARMO’s approach eliminates the need to deploy and manage agent software, container sidecars or runtime application self-protection (RASP) software to enforce security policies across what inevitably will become thousands of microservices distributed across an enterprise IT environment.

    That leaner approach then allows for a centralized management platform that can be deployed on a cloud or on-premises IT environment to apply policies that restrict which microservices can communicate with one another, Rozen said. The approach would ensure that only microservices developed using a continuous integration/continuous delivery (CI/CD) platform are allowed to communicate with other microservices, Rozen said.

    Integration with multiple CI/CD platforms is already provided, added Rozen.

    That approach allows policies to be applied to microservices regardless of how they are constructed, Rozen said. That capability is critical, because developers are building microservices using everything from Python and Java to containers, he said. Many of those microservices are also frequently replaced, Rozen said, which can make updating any associated agent software a significant DevOps challenge. Given all the existing dependencies between microservices, overall application environment security is only as strong as the proverbial weakest link. The ARMO Workload Fabric enables developers to add a small amount of code to implement security policies each time a microservice is rolled out into a production environment.

    As part of an effort to make it simpler for IT organizations to apply those policies, ARMO provides a pre-built set of policies that organizations can immediately apply, in addition to creating their own polices over time, Rozen said. In many cases, IT teams simply don’t have the time or expertise required to build those policies, Rozen added.

    Microservices are being built and deployed at such a breakneck pace that it’s nearly impossible for cybersecurity teams to keep up. The ARMO Workload Fabric provides cybersecurity teams a means to centrally manage security policies that would otherwise be implemented as code by developers. It provides a level of observability for how those cybersecurity policies are being implemented across microservices that typically have a lot of dependencies on one another, Rozen said.

    Most organizations today recognize there is a clear need to shift responsibility for application security further left toward developers. The challenge is finding a practical way to achieve that goal, and in a way that doesn’t adversely impact the rate at which applications are being developed. The key to achieving that goal is finding a way to apply microservice security polices that causes the least amount of friction possible to existing DevOps workflows.