devopsy.kinsta.cloud

Category: Enterprise DevOps

  • Implementing DevSecOps Goes Beyond Technology

    Implementing DevSecOps Goes Beyond Technology

    While technology is crucial for implementing DevSecOps, it is the people, processes and culture that drive it forward. As recently as last year, a survey found 58% of technology leaders cited existing culture and lack of skills as hurdles to being able to embed security testing and evaluation within software development processes. That report found that only about one in four companies strongly agreed their organization’s culture and practices supported collaboration across development, operations and security.

    A separate recent survey on DevSecOps skills, conducted in partnership with DevOps.com, found 70% of respondents said the security education they received is not adequate for what their current positions require.

    Plenty of similar statistics can be found, and they point to a recurring theme–education, training and support for developers and security teams are as critical as modern application security solutions in implementing DevSecOps.

    Automation Can Help Overcome Challenges

    Most developers want to create high quality, secure code. Often, the problem is developers have never had formal training in what secure coding is, in either traditional education or while on the job. The good news is more organizations are starting to provide that training for development teams. As development and release cycles speed up, developers need to find ways to incorporate security testing in less intrusive ways.

    In addition, development teams are gaining some control over budgets directed toward application security tools. Legacy tools for testing code and assessing application security risk simply weren’t built for the speed DevOps testing requires. Whenever evaluating new security analysis tools for a development team, automation should be top of mind. Static analysis tests the application code for a broad range of security flaws, and it can be fully automated into both the IDE and continuous integration (CI) processes. In the IDE, it provides early security guidance and education to software engineers while they are coding by highlighting potential vulnerabilities and suggesting best practices.

    Companies can also reduce friction by ensuring developers don’t have to go out of their way to run the security tool. Development teams should be able to view and triage the results and take advantage of plugins and/or APIs to integrate security tools seamlessly into toolchains they are already using. This means there will likely be no one-size-fits-all approach that can be dropped into any dev team; rather, take the time to understand how that team works currently, and work with them to introduce the security tools in the least disruptive way.

    Shifting Mentalities

    One of the biggest challenges is asking developers to do something they weren’t doing before. With DevSecOps, it may seem like dev teams are being asked to take on more work, but in reality, the organization is replacing unplanned work with planned work. Rather than address defects late in the release cycle, development teams are scanning for and addressing flaws early on. Such a cultural shift can be plodding, but developers and development managers should realize embedding security this way will be more efficient and more secure while also reducing their unplanned work.

    Giving development teams more autonomy over security tasks can also be daunting for security practitioners, but in order to facilitate shorter development iterations, security has to relinquish some control and also reconsider how they think about risk tolerance within the context of the business. Historically, the relationship between developers and security practitioners has been adversarial, but DevOps and DevSecOps methodologies have made it more cooperative. Still, many security teams are uncomfortable with the thought of giving up control.

    Adapting to Avoid Adding New Technical Debt

    Even with the understanding that producing more secure code benefits both development and security, reducing technical debt is gradual and starts with integrating security directly into the development process. It may seem convenient to lump security findings in with technical debt with the intention of getting to it someday, but it’s not practical to let security debt pile up.

    Organizations are decentralizing security work, particularly via security champions: One or two people on each development team who serve as the eyes and ears of product security, taking ownership of some of the day-to-day tasks that would have previously been bottlenecked on the security team. A security champion could help with security code reviews or coach others on their team on how to fix security issues. Sharing responsibility for security in this way can further bridge the gap between security and development teams, reducing the workload of the security team while allowing development teams to be more self-sufficient.

    For developers, context switching has a productivity cost. If they find out about a security defect while they are actively working on that particular feature or section of code, it’s only minimally disruptive and can be fixed quickly. By contrast, if they discover the same security defect two weeks later, after they’ve moved on to a completely different feature, they’ll incur the cost of context switching back to the earlier feature before they can fix the flaw correctly. Over the long run, finding and fixing flaws early in the SDLC should not meaningfully delay time to market.

    Here are some best practices in implementing DevSecOps in your organization.

    1. Embed Security Processes Early in the Development Cycle

    Ensure scans are baked in at every stage of the software development process and work with your development teams to integrate security within existing workflows and tools. By the time software is pushed to production, code should have been scanned multiple times and all vulnerabilities should be fixed. Research has shown the frequency of security scans during the development process correlates to fix velocity. Specifically, businesses that do daily scans fix vulnerabilities 11.5 times faster than the typical organization.

    2. Give Security Responsibility Directly to Development Teams

    Developers need to take ownership of the security of their applications. This is the only way to integrate security seamlessly into their existing workflows. The perception of someone coming in and forcing tweaks to their code can be discouraging, but that isn’t to say security teams should be hands off. They need to take an advisory role, working with developers to ensure secure code. Training is key here. It’s likely developers will not be acquainted with the latest security strategies. Security experts need to educate developers on known vulnerabilities, threats and tools.

    3. Identify a Security Champion on Every Development Team

    It’s equally important to find someone on the development team who can go beyond automated testing tools to serve as the eyes and ears of the security team. Rather than assign the role to a member who already has additional responsibilities, choose someone who shows an interest in security. Because engineers may switch teams often, it’s best to have at least two security champions on each dev team. If one moves on, the institutional knowledge remains on the team and a new security champion can be trained up.

    4. Automate Security Scans

    Security cannot be an inhibitor to development. If it is, developers will fail to implement the proper security policies. With automated tools in place, development teams can efficiently test code throughout the software development lifecycle (SDLC), ensuring more secure applications while delivering software on time.

    5. Take a Data-Driven Approach to Tracking Progress

    Showing progress is crucial, both for developers who want to see the impact security is having on their software and for the management team who needs clear insight into application security’s return on investment. Analytics with compelling visualizations supports this value by providing quick and intuitive data so organizations can assess their current state and take action to improve their risk posture. Sophisticated analytics allows organizations to target critical applications and fix critical findings by defining a custom security policy based on their specific risk tolerance.

    — Chris Eng

  • The Good, The Bad and The Testable: How to Give Your Experiment the Best Foundation

    The Good, The Bad and The Testable: How to Give Your Experiment the Best Foundation

    Whether you’re constructing a house, a new business or a mobile app, a solid foundation is critical. It gives you a strong base from which to build, and it can have massive implications on the life of your project further down the road. The same goes for the experimentation process.

    Before ever beginning a software experiment, it’s important to create a strong foundation by investing time in the experimental-design phase. A well designed experiment will have a clear and testable hypothesis, a set run time calculated via a power analysis and a predetermined plan for the analysis stage. Two key components of planning the analysis are deciding which metrics you’ll be measuring and which statistical test you will use to determine whether or not any changes were statistically significant. Unfortunately, this is easier said than done, and with the wrong statistical test or metric, your experiment is destined to crumble.

    How Do You Spot a Bad Metric or Wrong Test?

    Oftentimes, people don’t realize they’re using a bad metric or a wrong test. Of the two, a bad metric is easier to spot. First, ask yourself, “What does it mean if this metric increases?” With that answer, you should be able to easily explain whether it is a good or bad result and/or which aspect of user behavior or product performance has changed and in what way. If there are multiple different ways to interpret a change to your metric, then it is probably not a good one to use.

    Determining if you’re using the wrong statistical test is harder. There are many different types of tests that are appropriate in different scenarios. For example, whilst a chi-squared test may be appropriate for a proportions metric (e.g. percentage of unique users who convert), this test would not give accurate results if applied to a non-proportions metric such as page load time. Instead, for means, or real-valued, a t-test is often the correct one to apply to your results.

    The most important thing to keep in mind is that the metric should satisfy all the assumptions of the test you are applying to it. For example, both the t-test and the chi-squared test assume the observations are independent and the noise in the data is normally distributed. One way to check this is with AA tests, an experiment where there is no difference at all between the two treatments. If you run a series of AA tests applying your statistical test to the metric, the p-values should be uniformly distributed, and the fraction of the AA tests which appear statistically significant should be roughly equal to your false positive rate, e.g. 5% if using a p-value threshold of 0.05.

    Consequences of a Bad Metric or Wrong Test

    Without the right metrics and test, your experiment is bound to run into trouble. Specifically, using a bad metric can make it very difficult to interpret the results of your experiment, or in the worst cases, it can lead you to unknowingly draw incorrect conclusions. It can mean you are not receiving as much value and learnings out of an experiment as you could have.

    Similarly, using a statistical test which is not appropriate for the metric you’re applying it to can also cause you to draw incorrect conclusions. In this case, it can lead to a much higher false positive rate than you intended when you set the significance level. This can leave you thinking your experiment has had an impact, while it’s just normal noise in the data.

    What Is a Good Metric?

    Ideally, a good metric will be interpretable, meaningful, sensitive and fit for the statistical test you’re applying to it. Interpretable means if the metric changes, you should easily be able to determine if it will have a positive impact or negative impact and what that means in terms of user behavior or system performance. A meaningful metric directly measures something which you care about in your experiment. This could be something that’s a good proxy for business value or customer satisfaction. Sensitive metrics are able to detect smaller changes for a given size of traffic. For example, slow moving metrics such as retention or high variance metrics such as revenue, whilst being important metrics, are not inherently sensitive and hence might not be the best choice for your experiment’s primary metric.

    Finally, fit for the test means the metric should satisfy all the assumptions of the statistical test you will use as previously mentioned. For example, metrics where the denominator is not equal to the randomization unit⎯e.g. per-session metrics when you are randomizing on users⎯would not give independent observations and hence are not fit for use in a standard statistical test such as a t-test or chi-squared test.

    Benefits of a Good Metric

    A good metric is crucial to any successful experiment. It allows you to get the most information and learnings out of an experiment, and it makes experiments easier to interpret as well as more efficient to run. Using a good metric and appropriate test will also avoid the chances of inflated false positive rates and getting misled by the data, which provides the biggest benefit of all: saving time and money.

    — Lizzie Eardley

  • How AI and Microservices Create a Reliable Enterprise of the Future

    How AI and Microservices Create a Reliable Enterprise of the Future

    The Rise of Microservices

    Enterprise adoption of microservices is continuing to rise. Its popularity is so strong that 86% of development professionals internationally expect microservices to become the default application architecture within the next five years. Looking out even longer, nearly all professionals see microservices becoming their company’s default application architecture.

    At its core, a microservice approach centers on the idea of breaking up complex enterprise software into smaller, more manageable pieces, containing everything from the operating system to the platform to the framework to run time and dependencies, packaged as one unit of execution.

    By dividing an application into smaller stand-alone services, each performing a single function and communicating over the network to work together, microservices offer increased modularity and make applications easier to scale and faster to develop. In a world of ever-increasing apps, websites and digital experiences, there are immense benefits to deploying microservices.

    However, the use of microservices is no silver bullet and, for all their benefits, they have their own limitations and challenges. When it comes to deploying microservices, scale, complexity and constant change are the new realities. Microservices architecture introduces complexity with respect to network latency, network communication, load balancing, fault tolerance and message formats. With great flexibility also comes a higher risk of something slipping through the cracks. Managing many services requires understanding deep structures, dependency graphs and more.

    Process scalability is another challenge. An enterprise can run one cloud-based service with relative ease, but simultaneously running dozens of services across multiple clouds quickly becomes a complicated juggling act. In fact, with all the moving parts that come with microservices, it is becoming increasingly difficult to manage these processes manually.

    No wonder that, along with the rise of microservices, we have seen the emergence of auto-remediation/self-healing capabilities. To the untrained eye, the terms auto-remediation and self-healing would seem to be the answer, but don’t be fooled. Many of these features, although making the management of microservices easier, use static thresholds for data points such as waiting time, throughput, message queue size and more. When the thresholds are passed, the system will correct or self-heal as necessary. Although helpful, they’re not necessarily smart or intelligent—and that is where AI can help.

    Advanced Self-Healing and Self-Defense

    AI can help solve some of the hardest problems facing microservices today. Below are a few achievements we’re already seeing AI accomplish:

    • Load Prediction: Understanding the load on the system and acting on it before it happens will allow systems to provision more resources before they’re needed. Provision of resources is not just limited to the network and number of tasks running; AI can trigger customer support representatives to be on hand when—and only when—needed.
    • Decay Detection: Anomaly detection is built into a lot of monitoring services and is efficient at detecting spikes, loads and abnormal behaviors; however, detecting a service decay, even gradual or subtle, is difficult. AI can help systems recognize whether services are reacting differently over time and what the reasons are for this decay. It also has the potential to be more effective in identifying bugs based on studying clusters of microservices having continuous issues.
    • Resource Planning: As the scale and complexity of microservices grows, so does the use of the cloud. The ability to plan ahead using smart resource management could save significant costs while maintaining a stable and robust service.
    • Security: Security is already leveraging AI in many places by discovering attack patterns based on known behaviors. Leveraging these capabilities at the network and database level can improve security for actions done by an authorized person. If you can understand the usage patterns and act on them, you can define a more secure system for all users.

    Benefits aside, using AI is not without pitfalls. For example, despite all its potential, the so-called state-of-the-art AI models are still quite primitive when compared to what they have the potential to achieve. Another issue is how to manage the possibility of bias creeping into AI models to avoid unintended consequences. Technology professionals are already working on ways to solve these complications. However, it will take time to develop the right models, data sets and algorithms.

    The Future

    If the future of application architecture is truly to be dominated by the ever-increasing use of microservices, then those architectures will need to also be dominated by AI. The reality is microservices are simply becoming far too complex for humans to manage on their own effectively. Because of the dynamic nature of their structure, their complex dependencies and their sheer scale, machine learning must be applied to assist with monitoring and management tasks. AI is best suited to meet this challenge and tackle it head on.

    — Martin Henley

  • Upping IT Skills: IT Pros Cite Significant Challenges

    Upping IT Skills: IT Pros Cite Significant Challenges

    Anyone who works within enterprise technology learns very quickly that if they want to stay relevant to the workforce, they’ll need to continuously learn new skills. But it’s certainly hard keeping up with new skill training while also keeping pace with day-to-day work demands.

    This is probably why a survey earlier this year from IT management software provider SolarWinds found that 70% of all technology professionals surveyed do not feel completely confident that they have all the necessary skills they’ll need to successfully manage their IT environments over the next three to five years.

    A certain amount of fear that one’s current skillsets may become antiqued by technological change is worthwhile, but that apprehension is only useful if technology professionals turn that energy into action. According to a more recent SolarWinds survey, 60% of technology professionals haven’t actively pursued learning a new skill or completed a certification in the last six months.

    The survey, Building Confidence for Tech Pros of Tomorrow, is based on an international survey of 177 technology professionals within both the public and private sectors. It was fielded in August 2019.

    The survey found roughly 50% of the technology professionals who did start a certification process did not finish. The primary reason was lack of time.

    Enterprises, aware that there exist challenges when it comes to finding the staff with the skills they need, are making efforts to mitigate their staffing challenges. These efforts predominately involve internal trainings, with 35% of respondents citing such training. Additionally, 54% of respondents said full-day in-person workshops provide the most value for the time spent.

    The survey also found soft skills are becoming increasingly important, with 46% of respondents stating interpersonal communications skills are the most critical for career growth. Other top skills were project management (56%) and public speaking/presentations (52%).

    Interesting, the majority of technology professionals surveyed said, while they are confident speaking with business leaders, they don’t believe they are adequately equipped with the support and tools they need to be effective at their work. According to the survey, the top two requirements for technology professionals to become confident managing current and future environments include increased budget/additional resources (64%) and more support from IT or business leadership (48%).

    Finally, and which could be a sign of the increased reliance of enterprises on software, the survey found application performance management (APM) skills are increasingly necessary for technology professionals. Survey respondents cited application support, troubleshooting and monitoring as tasks they are increasingly responsible. And 53% said user experience monitoring of applications is an additional skill of theirs that they are required to develop.

    — George V. Hulme

  • Online Experimentation: With Great Power Comes Great Responsibility

    Online Experimentation: With Great Power Comes Great Responsibility

    Recently we had the honor of giving a talk at this year’s Turing Festival. The festival, which is held in Edinburgh, is one of Europe’s biggest cross-functional tech conferences and is named after the late Alan Turing, who invented the Turing Machine. This invention cracked the Nazi Germany Enigma code, helped to end World War II and is the conceptual foundation of the modern-day computer.

    So much of the technological advancements we benefit from today are a result of Alan Turing’s amazing achievement. But with the great power that Alan left us comes great responsibility. The responsibility we decided to speak on at the festival is ensuring you are not being misled by data.

    Engineering teams are delivering faster than ever before. In the 1990s, a new feature would be released every few years, whereas today, apps such as Netflix see a new feature released hundreds of times a day. However, we need to make sure the value of the software we are delivering is not suffering at the cost of this increase in speed.

    A great way to build this confidence and put out Turing-worthy features is by utilizing online controlled experimentation. Product managers can now utilize controlled experimentation across the full stack of their codebase. Experimentation can change the way in which we deliver new features. Instead of focusing so much on the speed at which new features are being released, it allows us to fine-tune the feature to meet the needs and demands of users.

    All this speed of delivery doesn’t mean companies are getting it right 100% of the time–apps still crash and websites still fail. For example, we’ve seen companies such as Salesforce, US Bank and even Instagram suffer downtime following new feature releases. Online experimentation allows for DevOps teams to better understand the impact the upcoming feature release will have on their users.

    Online experimentation requires you to split your traffic and randomize this traffic in an unbiased way, serving a new feature to a proportion of this traffic while the rest of your traffic does not experience the new feature. You can then compare the metrics of both these user groups to make an informed decision on whether to kill the feature or ramp it up to 100% of your users. You can run an experiment to a specific user segment, such as a demographic or market by utilizing feature flags within your code.

    Online controlled experimentation also allows for real signals to be differentiated from noise. By noise we mean the randomness that is always apparent in the data due to using a sample to represent the whole population. The statistics from online controlled experimentation allow us to determine if the differences in the metrics are unlikely to be due to noise alone, in which case we can be confident our feature really did have an impact. This is how using online controlled experimentation decreases the chance of mistaking noise for a real, meaningful signal. In other words, it allows us to control and limit the chances of a false positive–a result that falsely indicates the change you introduced impacted your metrics.

    We could spend ages discussing the benefits of using online controlled experimentation for feature releases (and we did at Turing Fest). However, the bottom line is experimentation allows you to be an outcome organization rather than an output organization. Taking the time to conduct an online controlled experiment means you will truly understand a new feature’s impact on your users and therefore you can be more confident in making your product decisions.

    This article was co-authored by Sophie Harpur, product manager at Split. Sophie is passionate about making the complex world of experimentation accessible to everyone with core engineering, growth, design and product principles in mind. After graduating from the University of Strathclyde, Sophie worked for Skyscanner before eventually joining Split.

    — Lizzie Eardley

  • 5 Key Skills Developers Need to Master in Order to Thrive

    5 Key Skills Developers Need to Master in Order to Thrive

    The promise of the cloud, in its essence, allows the IT department to focus on generating business value rather than spinning cycles fixing and managing infrastructure. However, the cloud is not a one-size-fits-all proposition, and many early adopters are finding it challenging to move to the cloud. There are well-documented cases of huge budget overruns, far in excess of prior on-premises costs, making estimating costs in the cloud versus the status quo a key skill of any C-level IT executive nowadays.

    This has ushered in an era of microservices and cloud-native applications. The symbiotic relationship between the two has enabled new architectural patterns such as microservices, while the cloud itself has enabled cheap experimentation and throwaway infrastructure. This means developers today can focus less on the infrastructure itself and more on dedicating additional resources, time and skills to iterating and deploying new code, which was traditionally only available to big organizations with limitless budgets. In addition, this evolution has created opportunities for a crop of new developers who come from non-traditional IT backgrounds to supplement the tech skills gap that’s currently crippling the industry.

    Here are five invaluable skills I believe every developer, whether just entering the workforce or currently in it, should cultivate.

    Adaptability Is Key

    This goes without saying, but developers today must be polyglots. Ten years ago, a developer would only have to reason about one process, in one language, connecting to a database. Today, with the advent of microservices, there are many different stores of data and many applications, written in many languages, that all need to coordinate with each other over the network.

    Different development teams are generally responsible for their own microservices, which in turn interact with other teams’ microservices. Careful thought is required when designing APIs through which they interact. Backward compatibility, availability and graceful failure all become crucial design decisions.

    More than ever, adaptability is a key skill. In the past, developers might only have needed to learn one language and one database, and chances are that skill set would have been all they needed throughout their entire career. Today’s developer will know Java (which is still by far the most popular back-end language), but the rise in newcomers such as Go is driving the development of a new wave of cloud-native software, such as Kubernetes and Docker. It’s critical that developers keep up with new languages and database technologies.

    Communication Is Critical

    Communication is vital. Traditionally, developers have not been known for their communication skills, but with distributed teams and developers working from home becoming more common, working in isolation is no longer an option.

    Because of these changes, it’s easy for a developer to become overwhelmed, which is why it’s also important to learn how to schedule time and how to be able to look after oneself physically. Taking breaks, sleeping well and exercising regularly is vital to preventing burnouts.

    Always Look for Ways to Improve Your Technical Skills

    It’s now easier than ever to improve your technical skills. The most important technologies are open source, meaning you can download everything you need and learn at your own pace. Many online videos and other materials can be leveraged as well; you no longer have to wait for your employer to approve funding for training.

    But crucially, the easiest way to pick up technical skills is to practice them in a business setting. This may mean moving jobs to find the right fit. Another way to become familiar with how distributed teams work is to contribute to an open source project. You can contribute to these projects in many different areas, not just coding. Contributing to documentation, for example, is an area that offers a low barrier to entry.

    Success in Humility

    There are certain traits I always look for when building development teams. First and foremost, I always look for people who are able to clearly express ideas and cooperate with other members of the team.

    Humility is a key requirement, because teams that cannot come to a consensus are doomed to failure. It is no use having an intransigent individual who will insist his or her ideas are best; you will waste many hours coming to conclusions with such people. Quite often, hiring a rocket scientist into your team can upset the entire balance.

    Don’t Be Afraid to Ask Questions

    One particular incident very early on in my career demonstrated another key skill: Never be afraid to ask questions, even if it is the same question you’ve asked previously. At the time, I was working as a mainframe operator for a book sales company. All of the day’s orders were stored in volatile memory; at the end of the day, they would be applied to a batch process that would calculate stock, print invoices and send instructions to the warehouse for customer book deliveries. This was in the time before the internet, so orders were entered manually by the data entry team who worked through order forms sent in the mail. Approximately five people entered these orders, and they usually totaled around 100-200 a day.

    I was working on overseeing the batch process that night and was a few weeks into the job. There was an initial step in the batch where manual input was required to start it. However, I encountered an error message and it offered a number of options with increasingly cryptic descriptions. I had no idea which option to choose. Rather than ask the senior person working with me, I simply made a guess. The option I chose ended up wiping the volatile memory holding all the day’s orders. While this caused the company to lose a significant amount of money, I learned a vital lesson: Never, ever guess and always ask for help.

    Developers are today’s innovators. With the advent of the cloud, concerns around infrastructure are minimized. In most cases, platforms to run an organization’s business software will already be available. The days of a developer having to worry about selecting a database and working out sizes of servers will be gone. Instead, a developer’s key skill will be in writing the business software and identifying which cloud platforms to use in order to deliver it. Armed with that knowledge and the desire to keep learning, today’s new wave of developers will only become more adept at delivering value over existing business processes.

    — David Brimley

  • Should Your Company Use DBaaS?

    Should Your Company Use DBaaS?

    From SaaS to IaaS, companies are rapidly moving to the cloud to gain huge agility and cost benefits. So, is it time for your company to take advantage of database as a service (DBaaS)? Here’s a high-level discussion to help you along your decision path.

    First, it’s important to distinguish DBaaS from simply running a database on a cloud-based server, such as Amazon AWS or Microsoft Azure. While you can certainly gain some cost and agility benefits from running your database on a virtual machine, you’ll still need to provision and maintain the database, that is, you will need server operations expertise.

    With DBaaS, the cloud provider also provisions and maintains the database, eliminating the need for your organization to maintain server operations expertise in-house. DBaaS manages the entire environment—physical and virtual hardware, operating system, networking, replication configuration, etc.—so you can stay focused on software development. With the click of a button you can set up the database and then connect your application with a simple API call. And you’re ready to go.

    The speed and cost advantages of this approach can be huge, but there are also some limitations, so it’s important to assess whether your company’s resources and your project’s requirements align with the current capabilities of the various DBaaS offerings.

    Reasons to Use DBaaS

    Lack of Expertise: As suggested above, if your technical team is mainly composed of software developers who won’t be able to manage the complexity of provisioning a database, you should consider DBaaS. In addition to handling the management of the database, the DBaaS provider will offer additional services, such as automatic backups of your data. In this worry-free environment, you can focus on optimizing your software instead of constantly worrying if the database is functioning properly.

    Short-Lived Databases: If you need databases to serve as back ends for continuous integration testing, DBaaS will allow you to rapidly provision the databases, populate them with test data and then delete them when they are no longer needed. Having the database provisioned for you makes it far faster and simpler to build the required automation for the testing environment.

    A Microservices Design: DBaaS can be particularly helpful, even in a production environment, if you are using a microservices design composed of a very large number of databases. Even if you have the internal expertise, you may not want to invest the time and effort in deploying all those databases. DBaaS can get you into production significantly faster.

    Reasons Not to Use DBaaS

    While the above benefits are appealing, you likely won’t want to use DBaaS in the following scenarios.

    Database-Centric Applications with Complex Topologies: To ensure ease of deployment, DBaaS providers limit the ability to deploy complex environment topologies. This reduces the flexibility DBaaS customers have. For example, let’s say that for disaster recovery, you have a primary synchronous replication cluster in one region and a secondary synchronous replication cluster in a different region, with an asynchronous relationship between them. This relatively complex topology is outside the scope of most DBaaS offerings.

    IO-Bound Workloads: If your database has IO-bound workloads, you may need very powerful hardware, including high-end CPUs, terabytes of memory and very fast storage. In this case, you may not find a DBaaS provider that offers a configuration that will meet your needs.

    Understand Your Options

    If your database requirements align with the capabilities of DBaaS, you now have the ability to eliminate significant operational complexity and develop, test and deliver your applications faster and at a lower total cost. However, make sure you spend time carefully investigating today’s DBaaS offerings, including Amazon RDS, Google Cloud SQL and Azure SQL Database, which all have different features and focus, including deployment models, performance trade-offs, instance types and more.

    If your needs don’t fit current DBaaS offerings, you’ll need to provision your databases yourself. If you don’t have the expertise in-house and can’t afford new hires, a database consultant may be a cost-effective strategy for getting the support you need.

    — Tyler Duzan

  • ‘DevOps in the Multicloud World’ Virtual Summit Puts IBM Z in the Spotlight

    ‘DevOps in the Multicloud World’ Virtual Summit Puts IBM Z in the Spotlight

    Got mainframes on the mind? Wonder how you can unlock the value of the platform for mission critical workloads? Trying to figure out where IBM Z fits in an increasingly cloud native IT environment? Those questions and others will be discussed in an upcoming virtual summit that explores the role of IBM Z —within the context of delivering higher efficiency and greater productivity for hybrid multicloud environments.

    “DevOps in the Multicloud World” virtual summit is scheduled to take place Oct. 24, starting at 10 a.m. Eastern. The day-long event features a passel of IBM experts, partners and customers sharing their knowledge and offering a deep dive into how to build and manage hybrid applications for agility in a multicloud world of continuous delivery. Panels and presentations will be held on a multitude of topics ranging from open source and DevOps to provisioning and user experience—all with a focus on the IBM Z systems.

    In addition, IBM customers will discuss how they’ve successfully included IBM Z in their multicloud strategies to improve their operations and bring real value to their end-user customers.

    Featured speakers include:

      • Barry Baker, Vice President, IBM Z Software
      • Rosalind Radcliffe, Distinguished Engineer, Chief Architect for CLM and DevOps, Member of the Academy of Technology, IBM
      • Mike Fulton, IBM Distinguished Engineer, CTO Enterprise DevOps, IBM Systems
      • Keith Wilson, Principal Technical Architect, USAA
      • Patrick Wawrzoszek, Senior Programmer Analyst, Scotiabank
      • James Gill, Mainframe DevOps Engineer, Triton Consulting (Lloyds Banking Group)
      • Arun Sharma, Product Manager Core Banking, DXC Technology
      • and more….

    In between sessions, attendees can roam the virtual conference hall, where they can chat and socialize with other attendees including IBM staff, and visit the virtual expo floor, featuring virtual solution zone stands equipped with video, downloads and even prizes for attendees.

    Perhaps best of all? Because this is a virtual event, you don’t have to leave the comfort of your own environs. Feel free to log in wearing your pajamas, even—we don’t judge!

    This event is free, but space is limited, so register now to reserve your spot today!

    — Charlene O’Hanlon

  • Why Integration Must Be Hybrid

    Why Integration Must Be Hybrid

    Integration is the central piece that ties your enterprise together. The need to integrate has only increased with the proliferation of SaaS applications, IoT sensors and devices, B2B e-commerce supply chains and mobile app development in addition to the vast number of applications that still reside in on-premises data centers which also host mainframe, legacy and custom applications. One of the biggest drivers today is digital transformation—organizations are rapidly trying to become digital enterprises in order to differentiate themselves from the competition and develop new business models to create new revenue channels.

    In the meantime, APIs are becoming the standard way of connecting applications, data and devices, and providing services to partners and end users. The API economy is the term used for creating new products as part of an organization’s transformation journey. APIs are also rapidly becoming a new channel for B2B partner interaction. Since APIs and integration are two sides of the same coin, this allows organizations to quickly realize the value of integration to unlock data from different silos, drive consumption of this data and leverage business benefits for the application of this data in various end user and consumer apps.

    It’s a Hybrid World

    The most common understanding of hybrid is that organizations have applications both on-premises as well as the cloud. Pure cloud integration is a fantasy since most organizations will have applications and systems that have to stay on-premises for a variety of reasons: the cost and risk of redeveloping core capabilities (eg. An ERP application that has been heavily customized for the business or a core banking application that is mission critical), the lack of critical services in the cloud, concerns about security and sunk cost of on-premises applications and data sources (mainframe and legacy applications that just work and will be extremely expensive to rip and replace with a SaaS application).

    However, the definition of hybrid is much broader and according to Gartner, such a strategy must address a set of challenges that go across personas, domains, endpoints and deployment models. Lets explain each of these in more detail:

    • Personas: Integration development today is not just the purview of IT specialists but extends into ad-hoc developers and business users (often called citizen integrators) who despite not being technical cannot wait for IT to develop integrations for them.
    • Domains: Integration has always been a continuum supporting a variety of use cases that go across applications, B2B partners, data (for the purpose of doing analytics) and process (to support automation and orchestration).
    • Endpoints: The sources and targets for integration started originally with on-premises, then expanded to the cloud along with the need for mobile connectivity and more recently has exploded with IoT with the proliferation of sensors and devices.
    • Deployment Models: This covers the more traditional definition of hybrid and includes on-premises, cloud and hybrid (cloud and on-premises) but is now expanding into a new style called embedded, using a headless deployment model which is required for SaaS applications that need underlying integrations to satisfy their end user requirements.

    The Hybrid Integration Platform

    A true hybrid integration architecture makes integration available across all environments, for all the personas, overlapping all the domains and connecting all the endpoints. This will span on-premises ERM, CRM and mainframe, cloud and SaaS environments, edge devices and sensors and B2B partners–we call this integrating edge to app.

    A key architectural principle in hybrid integration is to allow integrations to be done close to where the applications are in order to reduce latency and increase performance. From an application connectivity perspective, hybrid requires applications hosted on-premises, public cloud and private cloud to all be connected seamlessly so an integration developer does not worry about where they are and only needs to easily wire together the mapping to move data from one application to another.

    From a B2B perspective, hybrid requires data from B2B partners (customer and suppliers) to easily be identified, exchanged and processed within the integration layer. From an IoT perspective, hybrid requires that data from sensors and devices either directly or aggregated after analysis is integrated with back-end applications. Finally, all the integrations need to be exposed via an API layer for internal business units and external customers to consume the data in a secure and privileged way through mobile or custom apps.

    These requirements have created the need for a hybrid integration platform (HIP) that supports hybrid deployment across on-premises and cloud, democratizes integration development across IT and business users, is versatile enough to support all integration patterns and is extensive enough to integrate everything.

    Components of a HIP

    A lot of organizations originally solved the integration problem by adopting enterprise service bus (ESBs) and integration platforms. These are still widely in use and in many cases powering mission critical applications for organizations. These integration products solved the application to application (A2A) problem and in conjunction with message oriented middleware (MOM) provided capabilities to decouple publishers and subscribers and allow new applications to be easily added.

    However, ESBs are undergoing an evolution since they are considered a monolith and organizations are seeing the need to deploy light weight integrations as microservices. This has led to the emergence of a microservices runtime (MSR) that can be scaled very quickly to meet the needs of the business. MSRs are typically deployed using Docker containers and scale using a container orchestration standard such as Kubernetes and are subsequently monitored using open source tools, such as Prometheus.

    A big requirement for deploying microservices is the need for DevOps which provides the discipline to manage the dev/build/test/deploy process using automated techniques. Many of these organizations also adopted B2B gateways to extend the reach of integration to their trading partners for the purpose of exchanging e-commerce documents such as purchase orders and invoices via standards, such as EDI and XML.

    A lot of transactions still occur using files leading to the need for a managed file transfer (MFT) offering to securely manage file interactions. These in conjunction with adapters and connectors provide reach to a wide variety of applications and data sources. A newer component is API management, which allows existing integrations to be exposed as APIs for access to a variety of consumers while providing policy based enforcement and security of the underlying services. APIs needs end-to-end lifecycle management and governance to control the creation to retirement process and also drives the need for asset management to track dependencies and relationships and provide impact analysis.

    IoT integration is becoming a common use case. An IoT platform along with edge capabilities will allow data from sensors and devices to be easily integrated and analyzed for the purpose of moving to other systems or to take action. In-memory is a key embedded capability for ensuring high performance and scale.

    Finally, iPaaS is a pure cloud offering which enables integration in the cloud but in conjunction with an on-premises integration platform, or agent, it can ensure data is seamlessly exchanged between cloud and on-premises with end-to-end visibility and monitoring across the different environments. Here is a quick summary of the components:

    • ESB or integration platform (with broad connectivity capabilities)
    • Microservices and DevOps
    • API management and governance
    • B2B and MFT
    • Messaging
    • In-memory
    • IoT platform
    • iPaaS

    A Practical Approach to Hybrid Integration

    A HIP can be extremely powerful but involves a number of components which can also make it overwhelming to tackle all at once. A practical approach is to add components (using a building blocks approach) as the need arises to address various integration use cases.

    Most customers already have an ESB or integration platform, so that is usually the starting point for integrating on-premises applications. Once an organization starts adopting SaaS apps then there is a decision to be made whether to simply do cloud integration using the on-premises ESB or to start adopting an iPaaS. A critical factor in this decision process is to follow the general guideline and allow integrations to be done close to the applications. The decision matrix for such a scenario can be described as follows:

    • If the use case is mostly cloud to on-premises and most of the applications are still on-premises (80/20 rule) then this can be done within the ESB by adding cloud connectivity options.
    • If the use case involves a number of cloud-to-cloud integration needs then it may be worthwhile to invest in an iPaaS for efficiency reasons (there is no need to go from cloud to ground and then back to cloud).
    • If the application mix gets switched to mostly being SaaS apps then an iPaaS should be adopted since most integrations will be done in the cloud anyway.

    Once you bring together an ESB and iPaaS you are in a great position to start leveraging API management and add B2B capabilities to round out most of your integration needs and start deriving value from these integrations.

    Business Benefits

    Hybrid integration provides a way for customers to move ahead with critical projects and strategic transformation by exploring more flexible solutions. Not long ago, integration had been painted as a back-office capability with little direct impact on the business. However, integration needs have moved to the front-office and addressing these requirements are key for successful digital transformation initiatives.

    The business benefits are as follows:

    • Organizations are seeing more demand from customers for a better customer experience. Customers have many choices, the best experience wins the business.
    • Businesses need to react faster to disruptive changes by increasing the delivery cadence and deploy integrations faster.
    • The technology world is experiencing many disruptions. Businesses need flexible architectures to disrupt instead of being disrupted. They need to be able to implement new projects, scale rapidly to take advantage of short-term opportunities and try new fast-fail ideas to leapfrog the competition.
    • Gain benefits of the cloud. There is constant pressure to reduce TCO, use an opex model instead of capex and be able to access and leverage all data available from new IoT and SaaS sources.
    • Democratize integrations. IT departments are at breaking point and can no longer meet the needs of the business which means business analysts need to develop integrations on their own in a self-service way, which allows more people to contribute to the company’s goals.

    Real World Customer Case Studies

    Retail Pharmacy: Our first case study is a large global enterprise with more than half a million employees and hundreds of distribution centers. One of the biggest changes in the company’s history was the merger of two iconic brands. The company needed a global platform strategy to stay competitive and remain ahead of Amazon and other digital disruptors in the retail space, and coined their version of a HIP: Independent Federation Layer.

    Software AG provided a solution to standardize global operations—a “cook once, serve many” solution. Rather than buy more tools, the company would invest in a unified set of software capabilities using webMethods HIP to integrate all systems. They set up a 200-person integration center of excellence. All retail stores are being migrated onto webMethods for a mission-critical retail transformation program to simplify and standardize sales and inventory processing. The next phase will be standardized processes for warehouse and order management, replenishment management, financial management and more. With webMethods HIP, this retail pharmacy has the agility to react to market challenges and opportunities.

    Transport Company: Our second case study is a multinational public transport company that operates bus, train and tram services in Europe, Canada and the U.S. They needed to lift and shift services to the cloud in order to create a new business offering that could scale easily. They refactored their webMethods integration code base and moved key services to the webMethods integration cloud to enable customers to purchase tickets using mobile and web platforms via third-party websites. They provided secure access to on-premises databases by exposing APIs to the cloud using webMethods API gateway. In the process, they created a reusable ticketing platform that they’re offering to other transportation providers. They now have a new line of business.

    — Subhash Ramachandran

  • Split Software Adds Feature Monitoring to Feature Delivery Toolset

    Split Software Adds Feature Monitoring to Feature Delivery Toolset

    One of the issues that has long plagued both development and test is determining exactly where problem code exists. The increasing rate of change that agile and DevOps have brought to projects makes this problem worse than ever because changes today can impact code from yesterday–or yesteryear.

    The constant flux that modern software is in makes tracking down issues that much harder. This is offset a little bit by the fact that changes are generally smaller and more incremental because there are several iterations to get improvements into place and short sprints are the norm. But not completely offset, because a defect may just now be showing up, even if it was introduced several iterations ago.

    Feature Delivery tools offer the ability to group changes into features, then turn them on and off as needed. In the case of Split, they can also be phased in to see impact upon users. While the name uses “feature”, any code change like “feature enhancement” is included.

    The announcement pertains to new functionality called Feature Monitoring. This new addition to the toolset allows monitoring of features in production to show the DevOps team what is happening with their changes in real time. It also allows for advanced targeting of users so small groups can get the updates and evaluations made on how the changes impact those users in production.

    Feature monitoring captures error data, and feature tagging ties that data to features, meaning defect identification and resolution can both happen at a faster pace. Less looking where there isn’t a problem, and more information about how buggy a new release is.

    Given that I’ve recently written a fair amount about testing and the issues test DevOps team members face with rate of change, this seems custom-designed to address those issues. While there is no silver bullet, the faster issues can be identified, assigned and repaired, the better the overall application quality will be–and that is what this announcement is about.

    Not having used the tools yet, I can imagine some thought has to go into inter-related functionality, but that has always been true–you have to know what works as an overall system and can be placed together in a release versus what has to be held back if a given piece is incomplete. The way I envision Feature Delivery, and more specifically Feature Monitoring, being used would be smaller pieces than an entire release. Even if a new feature is going live, someone knows what inter-related pieces must be complete, so simply shifting that knowledge left will allow effective feature tagging.

    Finally, the ability to turn updates on and off by changing flags means less downtime, and if planning is done well, the virtual or complete elimination of rollback plans. Agile and DevOps talk of fix and re-release, but we’ve all seen scenarios where that simply isn’t an option, and this may be the solution we’ve been looking for–rollback by changing a flag that changes what code is executed. Flag-checking occurs in code with if-then’s though, so after many iterations code clean-up, it would likely need to be done just to keep the code from being overburdened with nested logic for iterations that are known to be stable.

    I think I’ll add this to my list of markets/products to dig deeper into and actually put to the test. We have a set of servers sitting idle at the moment, and a mobile app we use for testing, perhaps we’ll take some time to test out flagging and nesting.

    In the meantime, it is definitely worth a look. Anything that eases test burden and might remove the need for rollback planning is worth a look. Meanwhile, keep kicking out the tools that run business.

    — Don Macvittie