devopsy.kinsta.cloud

Tag: application

  • 6 Transformative Mainframe Predictions for 2024

    6 Transformative Mainframe Predictions for 2024

    The success of organizations in today’s digital economy demands speed — the ability to quickly respond to market trends with new applications and services, instant access to critical data, and resolution of issues even before they affect service. As we move into the new year, the adoption and adaptation of emerging technologies on the mainframe will enable enterprises to embrace changing market forces, turning them into a competitive advantage.

    Below are six predictions of how mainframe organizations will leverage new technologies to advance the platform in 2024.

    Prediction: AI/ML Adoption Grows

    The roles of artificial intelligence (AI) and machine learning (ML) and their adoption across the mainframe, from application development to monitoring to data management, will continue to grow.

    The generative AI revolution is real and here to stay. In the coming year, organizations will continue to explore best-use cases for its integration into mainframe management. Development teams will leverage key AI-recommended insights into automated workflows and code generation, improving developer productivity. Database administrators (DBAs) will look to automate data management processes to efficiently handle the demand for the diverse datasets required by generative AI tools, and mainframe operations teams will increase their reliance on AI-driven predictive analytics and root cause analysis.

    As adoption increases and experience is gained, organizations will work past their AI trust issues and begin to harness the power of automation to become a fully autonomous digital enterprise. At the same time, there will be a growing need to govern AIOps and MLOps to ensure the validity and accuracy of ongoing model development.

    Prediction: DevEx Comes to the Forefront of Companies’ Innovation Strategies

    With the long-awaited shift of mainframe workforces from those with decades of experience to a new generation of stewards now in full effect, organizations will find a new imperative to attract and retain top talent by providing developers with familiar tools and development environments.

    Professionals who have recently graduated from college or are moving from other platforms bring with them an expectation of integrated and automated DevOps practices. The move towards enterprise-wide DevOps and continuous integration and continuous deployment (CI/CD) pipelines will continue, with many companies also looking to employ cross-platform tooling. By giving developers the freedom to use their integrated development environment (IDE) of choice, organizations can decrease onboarding time and get new mainframe developers up to speed faster, enabling them to understand code and its function quickly and easily, and creating a seamless, frictionless way to make changes, debug, and deploy.

    Prediction: Database DevOps Bridges the Gap Between DBAs and Developers

    Changing customer demand and the need for agility will accelerate the drive for developers to seamlessly access and manage data sources. The demand for an agile development environment that extends from applications to databases will increase the inclusion of database DevOps in CI/CD pipelines, including version-controlled database schemas and data, automated database provisioning, and database testing as part of the pipeline.

    By embracing agile practices, such as incremental database changes and rolling deployments, organizations will bridge the gap between DBAs and developers and ensure that mainframe databases evolve alongside application code. This cohesive approach will enhance collaboration, reduce conflicts, and expedite application development, positioning mainframes as agile and adaptable components within the rapidly changing IT landscape.

    Prediction: Democratizing Mainframe Data

    The move ― and the need ― to democratize mainframe data, making it available to AI/ML analytics engines, will gain momentum.

    Organizations have matured their data lakes and gained valuable insights from data housed on distributed and cloud platforms, but the efforts needed to extract and transform mainframe data stored on tape and virtual tape libraries (VTL) has left volumes of business-critical data behind. By operationalizing mainframe integration with the cloud, organizations can make it easier and faster to access mainframe data and transform it into open formats, making it accessible to analytics engines and AI/ML applications.

    The ability to realize the hidden potential of data they already store will make cloud-based storage of mainframe data a prerequisite for enterprises looking to digitally transform their businesses and gain competitive advantage.

    Prediction: Object Storage Becomes Critical for Security

    Mainframe-dependent organizations will seek to integrate hybrid cloud object storage to fortify mainframe data protection against escalating cyberthreats and ransomware attacks.

    In 2024, cyberthreats will almost certainly increase in number and sophistication. The European Union’s Digital Operational Resilience Act (DORA) seeks to ensure that organizations remain vigilant and protect their systems and their customers’ data against increasing threats.

    With DORA set to come into effect in January 2025, and further regulations and guidelines on the horizon, managing mainframe data across diverse locations while prioritizing immutable data copies will be imperative. Organizations will pivot towards proactive strategies, aligning their mainframe infrastructure with hybrid cloud solutions to meet evolving compliance standards and shield against the growing sophistication of cyberthreats.

    Prediction: Innovation Takes the Lead

    Organizations will adopt ruthless prioritization, shifting focus from maintaining services and applications to building new ones for the future.

    Having seen firsthand the success of ongoing mainframe modernization efforts, organizations will place a new focus on building skills and specialties that align with future goals and product roadmaps, putting them in a position to capitalize on coming trends and giving them the ability to quickly respond to changing market forces with innovative new products and services.

    As companies mature their approaches to AI on the mainframe, many will turn to building “applied AI” teams tasked with developing proactive solutions to better analyze data and forecast trends. In short, organizations will think logically about how to embrace AI rather than being embraced by it.

    In 2024, mainframe organizations will accelerate the adoption of emerging technologies like AI and the cloud to increase the adaptability, resilience, and efficiency of the platform while situating it for even greater success in the future. The mainframe will continue to be a platform of innovation, integrating with new technologies and practices borrowed from other platforms to further the evolution of a hybrid IT environment that meets and exceeds the demands of the modern digital economy.

    The role of AI, the impact of data governance regulations, the mainframe skills gap, affordability, and interoperability requirements are all brewing a perfect storm around the mainframe that will mark 2024 as a year of significant transformation that will define the leaders from the laggards.

  • Unlocking Accountability: How Real-Time App Monitoring Empowers Engineering Teams

    Unlocking Accountability: How Real-Time App Monitoring Empowers Engineering Teams

    In today’s fast-paced software development environment, traditional infrastructure monitoring is no longer sufficient. As engineering teams take on more responsibility for code ownership and software quality, the need for comprehensive real-time application monitoring becomes crucial. This isn’t just an addition to standard infrastructure checks; it’s a fundamental shift toward a more accountable and empowering engineering culture.

    A Case for “Real-Time Application Monitoring?”

    Infrastructure monitoring is ubiquitous, and we’ve gotten really good at monitoring the health of our servers, our containers and our data stores. APM tools have come a long way and give us insights into how pieces of our code perform from a speed and error context.

    At the same time, the business side of the house is monitoring things like traffic, leads, conversions and an important performance metric—sales. However, for most businesses, this monitoring is rarely in real-time, and visibility gaps are created in the spaces between where problems are introduced, discovered and resolved.

    Let’s look at a concrete example using an auction marketplace. Core to this business is the ability for users to place, accept and pay for accepted bids. The business probably looks at these metrics on a daily basis, at best. What if a delivered change to their application broke and removed the ability for users to conduct one of these actions?

    Placing a bid probably has a “normal” volume pattern, and a change could impact this in a positive or negative way. If the volume of bids drops after a deployment or even spikes, what happens? How quickly can that change be reported, seen and acted upon, if necessary? How quickly can they get back to “normal” before a customer reports it or before the rest of the business notices?

    If this business is limited to once-daily monitoring, how quickly can they realistically expect their engineering team to be able to identify and remediate such an issue?

    How valuable would it be to the business if their engineering team could instrument these metrics with their own real-time tooling?

    Key Metrics for Trust and Satisfaction

    Customer trust and satisfaction are critical for any software product. The most obvious indicator of customer satisfaction is churn–the retention rate of your customers. This metric, while influenced by various factors, remains a vital sign of how well we are doing. At Stoplight (acquired by SmartBear in August 2023), we focused on reducing customer bug reports. A spike in these reports prompts an immediate investigation into the root cause. It’s about understanding what changed and how to address it.

    Moreover, new customer acquisition isn’t just about retention; it’s also about word-of-mouth referrals. Satisfied customers often become your best advocates. Thus, keeping an eye on churn rates and incoming bug reports and setting realistic targets for these metrics is crucial. It’s about balancing resource allocation without striving for impractical perfection.

    Net Promoter Score (NPS) scores, although often dreaded by customers, are a critical metric. As engineers, we might dislike them, but they offer a holistic view of user satisfaction. If you’re deploying daily and consistently breaking things, expect your NPS scores to reflect that.

    We can also demonstrate real-time ROI of real-time application monitoring in a security context using login successes and failures. Tracking the volume of these activities in our engineering tools allows us to quickly identify and assess the threat of anomalous behavior. Did the rate of successful logins drop? Did we break something with a deploy (I’ve broken logins before). What if either successes or failures spike? Are we under attack? How can we mitigate it to protect our users?

    Employee Retention and Developer Satisfaction: Essential in an Era of Real-Time Monitoring

    In the realm of software development—particularly with the advent of real-time application monitoring—employee retention, especially of developers, is paramount. Their deep understanding of the nuances of our applications and their ability to respond swiftly to the insights provided by real-time monitoring are invaluable.

    Maintaining a team of satisfied, engaged developers is crucial in this context. It’s not just about reducing turnover; it’s about fostering a culture where the engineers feel invested in the continuous improvement and success of our products. When developers are genuinely satisfied with their work and their environment, it reflects in the quality of their output. They become proactive in identifying and addressing issues, often before they escalate, thanks to the real-time data at their fingertips.

    The shift toward more dynamic monitoring practices has underscored the need for a supportive, collaborative environment. A culture where developers are encouraged to share insights and take initiative leads to a more responsive and adaptable team. This environment not only supports the technical aspects of our work but also enhances the overall morale and commitment of our developers.

    Engineers will seek out organizations that adhere to better practices. Interestingly, when good practices are in place, developers are more open to being on call. They accept responsibility for their code and their peers’ code as a trade-off for the autonomy and value they can deliver through frequent, quality deployments.

    Starting Small: Embracing the Journey of Incremental Improvements

    The journey to adopting best practices in engineering, such as real-time application monitoring, often begins with small yet significant steps. Reflecting on my past experiences, particularly at a shipping marketplace, the path to continuous deployment was transformative. Initially, we had no automated tests, and the challenges were substantial. However, the commitment to deploy continuously led to an evolutionary process in our practices.

    This approach isn’t just about technology; it’s about mindset. In my earlier days, inspired by the continuous deployment practices I observed in financial institutions, I realized the power of incremental changes. We began with what we had, deploying continuously, and with each step, we uncovered new areas for improvement. It was a process of building our capabilities piece by piece–from integrating automated tests to refining our deployment strategies.

    The essence of this journey lies in its continuous nature. Much like the principles of agile and DevOps, it’s a perpetual process of evolution. You don’t reach a finish line and stop; it’s about relentlessly pushing for betterment, identifying gaps, and adapting to new challenges. It’s about creating a culture where continuous improvement is the norm, where each small step contributes to a larger goal.

    Building a Culture of Continuous Improvement and Excellence

    Adopting real-time application monitoring is more than implementing a new set of tools; it’s about fostering a culture of continuous improvement, accountability, and excellence. In my experience, this cultural shift has a profound impact on every aspect of our work. It’s about ingraining a mindset that constantly seeks to exceed our standards with each release.

    Creating this culture involves a holistic approach to our engineering practices. It’s not just about the technical side of things; it’s about enhancing the collaborative and innovative spirit of our teams. It’s important for engineering leaders to do their part to foster an environment where every team member feels empowered to contribute ideas and take initiative. The ROI of this approach can lead to significant improvements in your team’s processes, as well as a deeper sense of ownership and pride in their work.

    Moreover, this culture of continuous improvement often extends well beyond the engineering department. It can influence the entire organization, encouraging departments like sales, marketing, and customer support to adopt similar principles. By doing so, we create a unified approach to product development, customer service, and business growth.

    Conclusion

    Embracing real-time application monitoring is not just about adding another layer to our development practices; it’s about fundamentally shifting our mindset toward a culture of accountability and continuous improvement. By focusing on key metrics, understanding the psychology of developers, and starting small, we can drive significant improvements in both product quality and employee satisfaction. As engineers, we are in a unique position to influence this change, and it is imperative that we lead the charge in adopting these essential practices.

  • The Path to App Modernization: Benefits, Pitfalls and the Route to Success

    The Path to App Modernization: Benefits, Pitfalls and the Route to Success

    There are many reasons an organization may find itself on the path to considering the risky journey of application modernization. It could be to gain an edge or to build a more competitive positioning, or a business could have a legacy system running on infrastructure that is no longer viable, leaving them no other choice. But the conundrum for organizations still resurfaces when considering app modernization; they must select one of these two routes: Either “the failure rate is high, and failure is simply not an option” or “Everyone is still doing it, so let’s forge ahead.”

    The Benefits of Choosing Application Modernization

    Whether an organization chooses this path out of necessity or ambition, modernizing an old IT environment can deliver several benefits, ranging from reducing outdated hardware and software to creating new value through innovation. Specifically, the key benefits are:

    ● Reducing technical debt and improving the underlying infrastructure: Improved security, operations, reliability and scalability.
    ● Decreasing expenses: Lowering the cost spent on expensive hardware and software improves the utilization of resources, thereby decreasing the total cost of ownership of technology operations.
    ● Accelerated time to market: Modernized IT and product development gain value from technological improvements that decrease the time it takes to develop and deploy new products and services.
    ● Agility in the market: Modernization enables an organization to react to changes at an accelerated rate, providing a competitive edge in the market.

    Choosing the Best Path to Application Modernization

    There are different paths to modernization, each with its own focus. On the one hand, there’s the infrastructure aspect of building your platform engineering practice versus adopting a service approach with a platform-as-a-service (PaaS). On the other hand, there is the actual process of modernization with options like refactoring, re-platforming or rewriting your systems, either in-house or through outsourced services. It ultimately starts with modernizing the relationship of the application with its infrastructure and changing some assumptions.

    Once you have a baseline, running everything else should come from a cost-benefit analysis. A scalable, well-managed monolith is valuable despite its age. Ensuring horizontal scalability and a robust backup strategy are key to effectively managing mutable data.

    First, understand the blockers in your current implementation that would complicate a shift to the cloud. Or identify where the result may not deliver any value.

    There are very common anti-patterns that remain anti-patterns, whether on the cloud, running on managed hosting or on-premises. Identifying them allows you to start absorbing technical debt regardless of whether a move to the cloud happens as intended.

    Anti-patterns to look out for are:
    ● Weak separation of code and configuration: You want to be able to run many parallel instances of each application as separate environments. Telltale signs are hard-coded URLs and service credentials. The dependency is the capability of running ephemeral preview environments that are quick to launch and quick to destroy, such that you know that you are “cloud ready.”
    ● Weak separation of the read/write domain from the read-only domain: In a cloud deployment, you want a build step that creates immutable artifacts that are not modifiable in production, and you need to know where all the actual data resides. Older applications tend to “believe” they own the whole disk.
    ● Manual configuration of services: You should be able to run “stock” services without any manual configuration in case the need for a specific database configuration arises and signals the likelihood of an application issue that requires debugging.

    Simply packaging a Docker container or a VM image and deploying to an IaaS won’t create any benefit and only increases costs. The first step to success is a thorough clean-up and update of the documentation to start the process.

    The one basic rule to follow is the ability to run an arbitrary number of instances of every element. Achieving this will allow everything else to flow. There is no need to rush to transform all Crons into “separate workers” and break the monolith into microservices. But the proof that it is possible now exists.

    The Best Approach to Application Modernization: In-House or Outsourced

    A mixed approach between in-house and outsourced is often optimal. There are many elements to consider. If you have a minimal in-house understanding of what the cloud imposes as constraints, you may be tempted to go for the simplistic lift-and-shift approach. Outsourcing everything can lead to huge amounts of lock-in.

    There are ways to cut corners that could simplify and accelerate the initial project, but this could increase the costs down the line. There needs to be a level of abstraction where using a PaaS is a winning proposition and can provide quite a bit of abstraction at no cost.

    Without control, the project may end up as something that “runs on the cloud” without the actual benefits. But there are many ways to do that.

    The Right Mindset Will Help Avoid Pitfalls Along the Way

    By neglecting constraints and skipping the necessary preparation, the costs may escalate. Running big chunks of hardware continuously versus using small ephemeral resources as needed typically leads to significantly increased costs over time. Full rewrites often go wrong when trying to obtain all the benefits immediately.

    A simple four-step approach for success is:

    ● Start with modernizing the relationship of the application with its infrastructure and changing assumptions.
    ● Identify and address the blockers in the current implementation–this makes a shift to the cloud less complicated and enables a smoother transition.
    ● Set a baseline that allows everything else to run from a cost-benefit analysis. Ensure horizontal scalability and implement a robust backup strategy to effectively manage mutable data.

    Accepting that app modernization will be a journey and that the initial implementation is a slightly more scalable and resilient version of the original. This is often a better mindset that leads to a more effective process.

  • Four Secure Coding Best Practices for Mobile Apps

    Four Secure Coding Best Practices for Mobile Apps

    Mobile apps have become the primary point of innovation for many companies, and skilled mobile developers are in high demand. The business asks developers to innovate fast using the latest smartphone features and capabilities to drive high download rates, wow their users, capture customers and grow their business. Dev teams facing constant pressure to move faster may push security to the back burner, focusing on required features and delivery dates the business demands. However, many dev teams have figured out how to innovate faster with security built in by ensuring that devs understand key security requirements and coding best practices. Working with hundreds of development teams on securing thousands of mobile apps, we found four common areas of security failure that can be addressed easily.

    Use SSL via HTTPS

    Mobile app devs may instinctually use Hypertext Transfer Protocol (HTTP) for network communications. However, HTTP puts private user information out in the open for threat actors to intercept. Developers can fix this common mistake by using HTTPS instead, which encrypts data sent to and from servers via industry-standard SSL. Android developers can use the available NetworkSecurityConfig file to set up a predetermined configuration for all network connections made within the app or manually incorporate HTTPS. iOS developers can use App Transport Security (ATS) as the default feature that enforces secure communications in iOS apps and prevents any insecure connections between the mobile app and the server from being completed.

    Validate the Contents of the Certificate

    Certificates add an additional layer of security to HTTPS connections by enforcing additional validations when performing a connection. Certificates include the certificate authority (CA) that signed it and the list of hostnames known or accepted by the application. Apps that validate these components when performing connections with the server significantly reduce the risk of a man-in-the-middle (MITM) attack that can steal credentials and sensitive data.

    To verify a certificate has been issued by a valid CA, Android developers can reference the preconfigured list of CAs included on devices running the mobile operating system. Android developers can implement native classes such as HostnameVerifier to achieve proper hostname verification within their app.

    iOS developers can leverage ATS that provides built-in validation, or use methods in the NSURLSession class to manually code these instead.

    Avoid Hardcoding Resources of a Mobile App

    Attackers often use hardcoded info within the source code of a mobile app to take advantage of users. For instance, attackers can use credentials stored within app files to gain access to a user account. Attackers can also find hardcoded API keys or URLs to gather private data or take over an app entirely. Mobile app developers can prevent these areas from being compromised by not hardcoding keys, passwords and URLs into the source code. Encrypting transport and decrypting data on the backend also adds an extra layer of security.

    Use the Latest Cryptography to Protect Mobile Users

    Outdated cryptography algorithms like SHA-2, RC4 and DES allow attackers to easily break a seemingly secure mobile app. Using older algorithms may also make a mobile app non-compliant with industry regulations leaving an organization at risk of fines or legal jeopardy. Devs can avoid this problem for Android and iOS mobile apps by using the latest cryptography algorithms available, selecting those that are suitable for a specific app scenario. When signing your binary before publishing it to the store, use keys with a length of at least 2048 bits (preferably 4096 bits), and on Android use SecureRandom or SecRandomCopyBytes on iOS when generating random values for cryptographic implementations. Android devs can leverage the Keystore class and iOS devs can leverage Keychain services to store highly sensitive data. Devs should avoid insecure modes of operation, improperly generated cryptographic keys and initialization vectors (IVs) to guarantee that the information that is encrypted cannot be decrypted by a malicious actor.

    Today’s mobile app users need confidence that mobile apps are built with security in mind. While building innovative mobile apps, developers have a responsibility to learn a practical set of secure coding best practices to protect their users (and their businesses).

  • The Power of Autoscaling and the Mistake You’ll Never Make Again

    The Power of Autoscaling and the Mistake You’ll Never Make Again

    Many years ago when I was interning as a software engineer, I made a mistake that could have ended my career just as it was getting started …

    It was my birthday, and I was given the task of loading queries to a database. The task was straightforward—even then, when I was only an intern learning my craft. I was using a program to upload the queries automatically. It looked like the program was running well, so I decided to go home and leave the program running overnight.

    The next day at the office I met any intern’s worst nightmare: Everything had stopped. And it was all my fault. The queries were loading with a paralyzing level of inefficiency. In fact, it took up all of the processing power that we had in that company server, making it crash. All operations were brought to screeching halt … just for my measly query project.

    It took several hours for the operations team to have everything running again. Luckily, I didn’t lose my job. In fact, the company ended up handling the situation graciously, and it turned out to be a good learning opportunity for me.

    These days, something like this is entirely avoidable, and it’s all thanks to the magic of autoscaling. As recently as 10 years ago, obtaining more server space was a slow and rigid process owned by a few gatekeepers. Now anyone can acquire more server space with just a few clicks.

    The democratization of processing servers has completely transformed the internet. It has given any organization virtually unlimited processing power at a moment’s notice. When done strategically, autoscaling allows for a more flexible and agile processing infrastructure. If implemented in a way that’s ad hoc or rushed, your organization will end up spending exorbitant sums of money without seeing the full potential of autoscaling.

    Here are three questions you need to answer to develop a winning autoscaling deployment strategy.

    What Should You Autoscale?

    Before the advent of autoscaling, projecting server needs took a lot of guesswork. Imagine, for example, that a timely and effective advertisement suddenly brings a 1000x spike in traffic to a company that sells sunglasses. If the company is not ready for this spike in traffic, everything comes to a screeching halt. Orders are not filled—and some orders can’t even be made because the server space is not ready to handle the unexpected surge in volume.

    With autoscaling, processing power can be offloaded to a new server automatically, based on the conditions set by the system’s administrators. This is valuable not only in the case of spikes in traffic, but also for lag periods. This is a way to avoid paying for processing power that you’re not using.

    In any organization’s computing architecture, there are a number of components that can be autoscaled, whether it’s central processing units (CPU), random-access memory (RAM) or bandwidth. The first step is to collect as much data as possible to understand the current operational loads of each of these items. Any organization using Amazon Web Services (AWS) has access to some data monitoring services, but they tend to be very basic. It’s better than nothing, but there are other options out there.

    One option is black box testing, which allows you to measure various components within your  infrastructure through monitoring. Among many things, it can help you determine where bottlenecks are occurring. Elastisearch is another great resource for word searching and metadata collection within a document or a digital framework.

    Another effective tool to assess autoscaling needs is something called an ‘agent.’ It works by sending pulses to measure processing data throughout an entire system. Collecting data like this will give you crucial insight on the state of your processing infrastructure.

    Once you collect all of this data, a thorough analysis is in order. At present, analyzing this data must be done manually, but before long machine learning will take care of this part of the process. It’s important that you consider all layers of your applications (front end, back end, database) to autoscale effectively. It can be tempting to scale at will, given how easily it can be done. Be vigilant against abusing this capability. Otherwise, it won’t be long before you spend outrageous sums of money on something you don’t need.

    How Should You Autoscale?

    An important concept to understand before you decide on your autoscaling approach is containers. Containers are small pieces of code that are meant to perform a specific function. In this way, a single machine can contain several apps used to perform a variety of functions: CPU, RAM, bandwidth, disk space, etc.

    Containers are a sophisticated way to get creative with the server space you’re working with, and should be considered even before you start building on a vertical or horizontal autoscaling strategy. Better algorithms in existing hardware can help you to avoid making new hardware purchases. In any case, once you do decide to autoscale, containers should most definitely be considered as you strategize and build.

    However, while containers can be great for scaling, it should be taken into account that they have the potential to add a great deal of complexity to your server. For example, if your server can run 20 containers, but suddenly you need three more, buying a new server with more containers is an expensive solution, since the server would be mostly empty.

    In principle, there are two ways to autoscale: vertically or horizontally. Scaling vertically means adding more physical machines with more gigabytes for more processing power. The biggest AWS machine server that’s currently available (at least as far as I’m aware) is 128 CPUs. That’s the most processing growth you can make from just adding one server. The cost of a new machine like this will vary: anywhere from 80 cents to $7 an hour is what you can expect to pay.

    The other approach is to scale horizontally, which would involve a cloud-based solution. For organizations not ready to make the commitment that comes with adding a new physical server, this is the approach worth considering. Plus, with a horizontal approach to autoscaling, it’s possible to adjust processing needs on the fly.

    It’s important to understand that not all parts of an application can scale the same way. Databases, for example, can scale up with ease. But scaling in (horizontally) would take a lot of time. If your data is stored in 30 front-end containers and you want to scale in to 15 containers, you can’t do it in one step. You’d first need to scale from 30 containers to 28, wait for the data to replicate, then scale in from 28 containers to 26, again wait for the data to replicate, then from 26 containers to 24, etc. Otherwise you risk losing all of that data.

    When Should You Autoscale?

    Consider the above example of the sunglasses vendor. While it’s impossible to predict exactly when your business or customer base will reach a tipping point, something such as publishing out an advertisement with a broad reach is certainly an instance where your organization ought to be ready for a sudden spike in traffic. Your business decisions should be made mindfully, so that the potential effects of actions like these—and the strain they create on your organizational architecture—are understood ahead of time.

    Anticipating peaks and troughs in demand is something businesses have been doing for centuries. In Colombia, for example, utility companies have long understood the power of “arepa hour”: This is when Colombians are at home eating breakfast (specifically arepas, a traditional breakfast food in Colombia). Knowing that the majority of the country is likely to be in their homes sometime between 7:30 and 9:30 a.m. consuming power, that is a crucial data point that can allow for utility companies to be ready for this disproportionately high level of demand.

    The dilemma to determine “the arepa hour” of your business’s autoscaling needs often comes down to CPU and memory. Should you be ready to autoscale based on when your site crosses a predetermined CPU load? Or should it be based on the memory running on your servers?

    The answer is really neither. There is no “golden metric” that will point the way for your autoscaling strategy, and you should never decide to scale based on a single data point in isolation. For example, it may make sense for processing data to be offloaded to a new server once the processing capacity is at 90 percent, but it’s also entirely plausible that your application continues to run completely normally with zero issues at this processing load.

    You should be aiming to strike the right balance between efficiency and not compromising the user experience for those who visit your site or use your application. And the only way to do this is to consider all of your data points holistically and how they interact with one another—not in isolation.

    Conclusion

    We are past the time in which a query loading faux pas like the one I made as an intern can completely shut down a company’s processing activities. Given how accessible processing servers have become, some may find it tempting to “brute force” their way through every problem: Make a few clicks and pay for more server space at will. Don’t be goaded into taking the easy way out. Think about autoscaling strategically by answering these deceptively basic questions, and you will in fact make it easier (and more cost efficient) for your organization in the long run.

    — Alejandro Calderon

  • Delphix and Datical Extend Global Partnership, Bring New DataOps Solutions to Modern Enterprises

     One-half of all Fortune 50 companies work with Delphix, Datical to deliver high quality software – faster and more securely – to close Data Velocity Gap

     REDWOOD CITY, Calif., October 23, 2018 – Delphix, the company accelerating innovation through DataOps, today announced it has extended its partnership with Datical, the leading provider of database release automation solutions, to bring new DataOps solutions to market that unlock the potential of data to drive business transformation. Already in use by  one-half of the Fortune 50, Delphix and Datical are driving the emerging DataOps ecosystem that will enable more Global 2000 companies to open up the flow of fast, secure data to move fast and compete in today’s Digital Economy.

    With worldwide enterprise data expected to grow to 163 zettabytes (ZB) by 2025 (Data Age 2025, IDC), software development and DevOps teams are drowning in data; unable to use it to deliver innovation that keeps pace with ever-changing consumer demands. Delphix and Datical offer proven solutions that eliminate delays through automation to achieve 100x faster high-quality software releases, 10x less cost and no production outages.

    “As large enterprises look to accelerate their digital businesses, two of the biggest challenges they face are the incredible volumes of data being generated and the related lack of velocity from their database layer,” said Stephen O’Grady, Principal Analyst with RedMonk. “While application development has been revolutionized by the DevOps phenomenon, database management has lagged behind. This issue is what’s created the demand for an equivalent ‘DataOps’ style movement, and is why Delphix and Datical are focusing on these needs so intently.”

    “Businesses are feeling the debilitating effects of data friction, challenging their ability to move fast and confidently compete in today’s digital economy,” said Tony Orlando, SVP of Global Field Operations, Delphix. “Through our partnership with Datical and the DataOps ecosystem at large, we are enabling the secure, continuous flow of data within the world’s most complex enterprises, empowering them to accelerate innovation, reduce costs, increase quality and lead in their markets for years to come.”  

    Modern Enterprises Rely on DataOps to Unlock the Potential of Data

     “Data remains locked in legacy systems, and is holding [CEOs] back,” said Salesforce CEO Marc Benioff in a recent earnings call. Over the last three years, Delphix and Datical have worked with some of the most important brands in the world to help them unlock data to accelerate software development and close the costly data velocity gap.

     Together, Delphix and Datical enable enterprises to dramatically accelerate software development and testing. The unique combination of these solutions reduces the amount of time it takes to provision production-quality data from days or weeks to just minutes, while also eliminating manual tasks, such as reviewing and validating database schema and logic changes. Integrating these solutions into a software development toolchain is at the heart of the DataOps approach that transforms how businesses deliver new releases.

    A Top 5 investment management firm based in the US (with over $2 trillion in assets under management) uses Delphix and Datical to increase the speed and quality of application releases that are the backbone of its digital transformation efforts. In a short period of time, the global brand achieved:

    • A 100% increase in rate of application deployments;
    • A 90% reduction in data provisioning times;
    • Dramatic improvement in quality from eliminating data-related defects.

    Pioneers in the DataOps movement, Delphix and Datical are at the forefront of building the platforms and technologies that support this emerging discipline. As Gartner recently recognized in its Hype Cycle for Data Management for 2018, DataOps is the next chapter in the DevOps story that will accelerate business and help create the modern data enterprise.

    “Software is reshaping all industries and the pace of change is accelerating. Companies must speed up innovation in order to survive in the digital era,” said Derek Hutson, CEO and President of Datical. “To achieve their goals of faster time-to-market, companies continue adopting faster, more iterative development methodologies to be able to change and adapt more quickly – but data has become the new constraint slowing them down. Datical and Delphix are in the perfect position to help enterprise leaders remove that constraint and modernize the way data is managed throughout the software delivery lifecycle.”

    About Delphix

    Delphix’s mission is to free companies from data friction and accelerate innovation. Fortune 100 companies use the Delphix Dynamic Data Platform to connect, virtualize, secure, and manage data in hybrid cloud environments. For more information visit www.delphix.com.

    Follow Delphix on Twitter, Facebook and LinkedIn.

    About Datical, Inc.

    Datical’s mission is to transform the way businesses build software so they can deliver better customer experiences faster. Datical solutions deliver the database release automation capabilities technology executives need to get the most out of their Agile, Cloud, DevOps and Continuous Delivery investments. With Datical database release automation organizations can shorten the time it takes to bring applications to market while eliminating the security vulnerabilities, costly errors, data loss and downtime often associated with current database deployment methods. For more information, visit www.datical.com, call 737-402-7187 or connect via @datical.

    Follow Datical on Twitter, Facebook and LinkedIn

  • All Day DevOps: Modern Infrastructure Automation

    All Day DevOps: Modern Infrastructure Automation

    “We’re no longer an airline. We’re a software company with wings,” claims Veresh Sita, CIO, Alaska Airlines. The success of today’s businesses rests on software. It is an integral part of infrastructure, so we need to always ask, “How can it be better? More reliable? More secure?”

    Nathen Harvey (@nathenharvey) is the VP of Community Development at Chef and the co-host of the Food Fight podcast. He is also an evangelist of automating software operations, which is what he presented on at the 2016 All Day DevOps conference.

    Nathen started his talk by stating that “automation at scale is required for high velocity IT,” making the point that in today’s environment, you can’t wait months between deployments. Enterprises are moving to continuous integration/continuous deployment, and he advocates for continuous automation.

    harvey pic1.png

    To implement automation at scale, you have to start with a dynamic infrastructure by:

    • Provisioning and setting up environments
    • Implementing dynamic scaling of compute resources
    • Migrating legacy workloads to the cloud
    • Deploy in multi-cloud and hybrid-cloud environments
    • Support heterogeneous environments

    harvey2.png

    Several speakers of All Day DevOps talked about automation, and all of them said a critical step is to treat infrastructure as code. According to Nathen, this means that to programmatically provision and configure components, such as servers, databases, firewalls, etc., you also have to treat it like any other code base. This includes version control and automated testing, but also:

    • Automated, full-stack application policies
    • Package and service installation
    • Versionable, testable and repeatable workflow
    • Scalable application policies
    • Management of interdependencies across nodes

    When you treat your infrastructure as code and deliver at scale, you can scale elegantly from one to tens of thousands of managed nodes across multiple complex environments.

    For software to be effective, users have to trust that it will do what they expect it to do. This is true from handheld calculators to self-driving cars, and it is also true when we automate our development and operations. Nathen advocates that we have to build a pipeline to production that creates faith in us that our code is ready for production.  This pipeline must:

    1. Test the code/verify locally
    2. Verify in pipeline with automated testing
    3. Approve with another engineer (aka code review or Nathen’s “four eye rule”)
    4. Build an artifact
    5. Deploy into acceptance environment to automatically test the artifact
    6. Ask someone else, “Do we want to ship this?”
    7. Ship the code with a click of a button
    8. Union (bring together dependencies and run tests)
    9. Rehearsal
    10. Delivered (ship to customers)

    What does the future hold?

    Infrastructure automation is NOT enough. Why? Auditors show up to audit your infrastructure and applications. The problem is, they use different tools—Excel spreadsheets, notebooks, PDFs, etc. This makes it very difficult to audit. Nathen advocates managing compliance documents as code, too. When we translate compliance into code, it can be executed as part of the pipeline so we can test for compliance throughout the entire life cycle.

    Finally, Nathen stresses the need to have application automation, too. Modern applications look like infrastructure as code.

    harvey3.png

    Developers can make applications that operations can deliver and run in production. This makes it easier to consume, run, and keep applications healty. Operable applications need to be:

    • Isolated
    • Immutable
    • Configurable
    • Built on a common interface for monitoring health
    • Rebuildable from sources
    • Reside in common packaging
    • Maintain runtime independence

    As enterprises implement infrastructure, application, and compliance automation, it creates a freedom for people to focus, and people are the key.

    Nathen notes that, “DevOps is a cultural revolution and professional movement, focused on how we build and operate high velocity organizations, born from the experiences of its practitioners.” He underscores that people are the future of automation. Automation places the priority on people, which enables great companies and great products.

    harvey4.png

    You can watch Nathen’s entire talk online here, where you can dig deeper into this subject. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free of charge here.  Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here.  This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed).  It’s all free and online on Oct. 24.

    — Derek E. Weeks

  • Don’t Make Me Code: Environment Protection

    Don’t Make Me Code: Environment Protection

    Thanks for listening to “Don’t Make Me Code,” a Heavybit podcast that discusses developer experience design and the unique challenges of building developer-facing products. “Don’t Make Me Code” is hosted by Steve Boak, co-founder and CPO at Opsee, and David Dollar, co-founder and CEO at Convox.

    In this episode, Steve and David are joined by Sean Li, lead product designer at Docker. Sean, Steve and David discuss Docker’s acquisition of Kitematic, the friction of attempting to replace a developer’s existing tool, and the increasingly prominent role of design in developer tools.

    This podcast is brought to you by Heavybit, a program that helps developer-focused companies take their product to market.

    [soundcloud url=”https://api.soundcloud.com/tracks/274576622?secret_token=s-QHiyf” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

    Steve: We’re calling this episode of “Don’t Make Me Code” “Environment Protection,” and we’re here with Sean Li, lead product designer at Docker and one of the founders of Kitematic.

    Sean: Thank you.

    Steve: Thanks for being here with us. This is a really exciting conversation for me because we get to talk to a product designer for a company like Docker.

    Sean: It’s good to be here.

    Steve: Kitematic is a real interesting project for a bunch of reasons. I mean, it caught the interest of Docker. It’s also proven to be a really novel bit of development environment technology. And so we’re curious how it all got started.

    Sean: It got started in a university dorm room. I was a student from University of Waterloo in Canada, and the school actually offers a pitch competition. So we had this idea around making development environments easier, and so we pitched the idea and won. That’s how we got enough money to get started.

    Steve: Wow, and was it at all tied to Docker at that point?

    Sean: Docker was actually what we used for the back end. At the time, Docker was version 0.3. And then we’re on Hacker News, and we’re like, “Oh, this is interesting. It could be super useful technology to do what it wanted to do.” Our idea was essentially like Dropbox for your code.

    You get a Dropbox-like icon on your task menu, and then you can put code in it. For example, Django, it would just run it for you. And you can see it on the browser.

    Steve: Oh, nice. And so you have your production environment there, compiling for you and ready to go.

    Sean: Yeah. Then, when you added the code, it will sync to the cloud environment and then update right away.

    Steve: Nice. That’s awesome. It’s like a step beyond GitHub. It’s not just committing the code; all the CI and everything is built into this one workflow.

    Sean: That’s like the future plan right at the time, but at first, our MVP was super simple—just syncing code to a remote server and then seeing the changes.

    Steve: From the beginning it was implemented as a desktop application, you said like a Dropbox tray application.

    Sean: Yeah, because Dropbox was obviously the inspiration.

    Steve: And so why a desktop application?

    Sean: At the time, we actually didn’t realize the power of desktop application yet. At one point we used to pivot a lot, as startups usually do. And then we’re talking to customers, and they start realizing what we’re using underneath.

    We talk to Canadian Tire, the biggest retailer in Canada. They were looking for something like this, and they’re like, “Are you using Docker underneath? We want to use that.”

    At the time, the product we have is in the cloud, like an on-prem web browser setup. And then we started thinking of, instead of abstracting away Docker, why don’t we embrace it and then make it easier for the customers? Then we start brainstorming ideas, and it came across that having a desktop application is actually super beneficial.

    It’s like the best platform for developers, because they can import their code super easily. They can open a directory instead of having to upload files in a browser. You can also get a lot of access to their desktop, so once your tool gets more powerful, there’s more things that you can do.

    It’s also super cool because we use Chrome in an electron shell. And we wrote the UI in React, so that means we only have to worry about how the UI runs in that one browser, instead of having to worry about browser compatibilities across the board.

    Steve: So it’s not something React needed, it’s just running in a web view in the electron browser.

    Sean: And we make it feel as native as possible.

    Steve: All right, nice. So it’s a true desktop app in many ways, but specifically the UI, it’s a web UI.

    Sean: A lot of inspiration comes from the Atom Editor by GitHub. They’re using the same technology.

    Steve: And so this whole bit was open-sourced from the beginning?

    Sean: We were working on it, and then we open-sourced it.

    David: I’m curious what made you decide as you’re releasing this tool. Was open-source service a strategic decision, or you figured you could get more users by doing that? Or is it just wanting contributions from the community? Could you talk through a little bit of your inspiration behind that?

    Sean: I think the biggest part is if it’s something that touches their code, it’s really important that the developer trust it by seeing what the code is like.

    We’re really open about what we do, and also the part on gathering the feedback from the community is super, super helpful. I feel like, in general, open-source projects get more feedback from the community.

    Steve: With developer tools it seems like a great way to encourage participation. If you’ve got an open-source project, you’re inviting people to contribute. Obviously, our Opsee product for example, it’s closed, a SaaS application. So we don’t have the level of community support from developers that we would really like in some ways.

    Sean: We hired an engineer from Docker because he was a maintainer and he was super passionate about the project. It really helps on that front as well.

    Steve: Following from that, you launched this to an open-source community, did you have an idea of the kinds of users you were targeting at that point?

    Sean: We sort of did from the original product that we had, the Dropbox-for-code product. We had a signup list. A lot of these users are more leaning towards willing to use UIs and prefer something simpler. The super active ones are actually consultants that work on multiple projects.

    Steve: That’s an interesting segment. People there, maintaining a lot of projects, and they’re looking for an easy way to keep track of all that stuff. And Kitematic fit that mold.

    Sean: Yeah, and Docker being isolated also provides that value.

    Steve: That too. Now, if I’m maintaining multiple projects, I can have a Docker environment for each one. The app makes it very easy to move between those. And so specifically the React UI and the GUI, what was it like having that be part of the open-source project? Did you get contributions for that as well, or was that entirely internal?

    Sean: The React UI helped us in a way, because React is, according to a Google trend, almost on the same trajectory as Docker. So there are a lot of people willing to work on something that’s built in React. It was in a way accidental, because when we started, it was a really good technology to use.

    Steve: Did you have any designers contributing to the project?

    Sean: We do have design-minded people but not professional designers.

    Steve: For example, you’re a lead product designer at Docker and you, I assume, identify mostly as an engineer, or your background in coding. But there’s definitely this hybrid world in the developer tool space.

    Sean: Definitely. In the beginning we had hackthons. A lot of participants were trying to add features to Kitematic, then make a huge effort on figuring out the right UI right experience to build there. By default, I think what I wanted is sort of like designing an OS.

    A UI is very extensible, and you want to figure out the right points where you can extend on it. You actually focus on per component and really set an example of how you extend the current UI. When people contribute, it can follow that pattern. And you only had to make slight adjustments to it.

    Steve: That seems true of open-source projects, generally, too. You want build a broad base that people can build on top of and customize.

    Sean: Yeah, but the initial effort has to be really strong. Because otherwise you will get swayed by different opinions. And design by committee isn’t always a nice thing.

    David: That’s pretty interesting because you oftentimes see open-source projects that are sort of notorious, especially visually, for bad design or design by committee that ends up being a little too weird. I can think of some Linux desktop managers that I won’t name, but Kitematic is actually a beautiful application.

    Sean: Thank you.

    David: I’m curious how you managed to maintain that, even when you’re doing community-driven design.

    Sean: I think a big part of it is keeping focus and actually be willing to cut things. In the beginning, Kitematic actually had build, you can actually build an image and then push it. But to keep focus and make things simple before we figure out the next steps is actually making it run content only.

    Essentially the focus of Kitematic was helping users install Docker on their computer. You know, in a visual way, it feels very reliable. We solve a lot of underlying issues with VirtualBox in the beginning. And then after that choose a beautiful catalog where you can click one button and run a container. And then that’s the experience we were aiming for and focused on. So it’s super obvious what to do.

    Steve: Did you ever get push-back on those decisions from the community where you had to assert your position strongly there?

    Sean: No, I think the community is mainly supportive. What I had to manage was people trying to add too many things that are not focusing on that road map currently.

    Steve: I think I remember you saying, on the Software Engineering Daily podcast, that you did end up cutting quite a few things from the product after a bit of time. Can you talk more about that? You were trying to clean it up?

    Sean: The one thing that was cut was building images from it. Even though it’s a super important thing, we decided to delay it because we want to do it right. Instead of releasing something super early, and then get sidetracked into different things, we want to keep focus until we figure out this is it.

    And then release that as a whole. Talking about cutting more, the next version of Kitematic is sort of Docker for Mac. It really solved the core issues that Kitematic was trying to solve in the beginning, to install Docker reliably on people’s desktops in a way that it almost feels like it’s seamless.

    You don’t feel like it is running inside a VM at all, and that’s the goal in the beginning. You install Docker for Mac. When you open the terminal you can type “Docker run,” and it just works.

    Steve: Had the Docker for Mac project begun before the Kitematic acquisition?

    Sean: It was after the acquisition.

    Steve: So you were involved with that from the beginning.

    Sean: Yeah, I was the founding member of Docker for Mac.

    David: So it’s interesting to think about this this sort of internal debate. I guess with most design questions it’s customizability and configuration versus limited options, and this curated a beautiful experience. I feel like that happens even more with developer tools.

    It seems like Kitematic and Docker for Mac, some of these tools, are definitely trending towards the latter there, where you’re removing some of the optionality of Docker to make a better user experience. I think that’s really interesting.

    Sean: I think the two tools are, in a way, more complimentary. People usually use Kitematic by having the Kitematic UI side by side with the terminal. So they like the transparency that they’re getting from the Kitematic side, and the sense of it feeling reliable.

    “I see things are running. I can check the logs super easy.” On the other end, the terminal is how they’re used to working before, and that was sort of expected in the beginning. And then you go from there. You never want to displace a tool that they’re already using, one they’re so familiar with.

    So instead we embraced that but made Kitematic more of a complimentary to the Docker CLI.

    Steve: That’s really interesting. I’ve seen similar comparisons made with GitHub and some of their tooling. GitHub has their Mac app and their web UI and stuff like that. And I found myself going to them for things like diffs and comparison, merge states, something where I want to get a lot of text or get some status update.

    But for actions, if I want to do merges, everything I’m actually doing with it I’m still doing through the command line. But that’s a really interesting comparison, because it sounds like you’ve made the same distinction. To view things, maybe you go to the UI. But to actually take action?

    Sean: GitHub desktop is like a huge inspiration for all these decisions as well.

    Steve: I never really heard that distinction made this clearly, that there are certain things that developers are going to continue to want to do through the command line and through terminal. But that we can make the experience better in some ways, like just figuring out whether a container is running or not. That’s really nice to have a UI to see that all the time.

    This podcast is brought to you by Heavybit, a program that helps developer-focused companies take their product to market. Throughout the program, founders work on developer traction, product market fit and customer development. If you like what you hear, be sure to subscribe to Heavybit’s Podcast Network on iTunes.