devopsy.kinsta.cloud

Category: DevOps Practice

  • Google Analytics Adapts to GDPR, But Questions Remain

    Google Analytics Adapts to GDPR, But Questions Remain

    The  General Data Protection Regulation (GDPR) will have an effect on how Google Analytics collects and processes data and how DevOps uses the tool—at least, that is what is known for sure at this time. These changes also should almost certainly impact DevOps for any organization that relies on the tool to collect and analyze user data. It is also known DevOps teams will have to reconfigure their Google Analytics implementations, even if an organization has no users based in the European Union (EU).

    However, while Google has recently offered some guidance about modifications to Google Analytics that impact DevOps, the company has yet to clarify some key points about changes ahead of when the new mandate takes effect May 25. In the immediate, here are some changes Google has communicated so far and actions DevOps will need to take ahead of the regulation’s deadline.

    Immediate Action Required

    The GDPR’s “right to be forgotten” provision offers users options to demand the deletion of personal data third parties store and share. To meet the terms of this provision, Google Analytics will allow organizations to manage how long their users’ data remains on Google’s servers. In this way, Google Analytics will automatically delete user and event data older than the date specified. Organizations can specify that the data remains on Google Analytic servers for 14 months, 26 months, 38 months and so on, for a maximum 50 months. It is also possible to opt for data to not expire automatically. Google Analytics says it will not delete information based on aggregate user data.

    Google Analytics also will allow organizations to delete data for individual users in an EU country that can be used to identify them under the GDPR’s “right be forgotten” clause. This data will include common identifiers that Google Analytics stores—including standard Google Analytics first-party cookies, user ID or App Instance ID.

    However, Google Analytics has yet to specify details about how organizations will be able to delete user data. While the process will be animated, Google says more information “will be available shortly,” presumably ahead of the GDPR deadline.

    Google says it is also drafting a new “EU User Consent Policy” ahead of GDPR. This policy that customers will have to agree covers organizations’ “responsibilities for making disclosures to, and obtaining consent from, end users of your sites and apps in the EU.”

    Another key modification Google has yet to reveal is the interfaces Google Analytics will offer that will allow organizations to make changes to how their users’ data is stored. Whether DevOps will be able to automate these processes or not will depend largely on whether Google Analytics will offer a GUI, an API or both.

    “If Google Analytics offers a GUI with no API, there is not much a DevOps team can do to automate it outside the existing functionality, since you need a person to do that. But if it’s an API, DevOps teams can weave the controls into their existing frameworks,” said John L. Myers, an analyst for Enterprise Management Associates (EMA). “If both are available, then you’ve got the best of both worlds.”

    Additionally, an API would allow the retention and deletion tools to work within existing DevOps frameworks and to become automated as part of business processes, Myers noted.

    The DevOps Challenge

    DevOps will obviously need to be prepare for GDPR in other ways outside of the scope of changes to Google Analytics or face fines up to 4 percent of their annual revenues or €20 million ($25 million), whichever is greater. Indeed, GDPR applies to all organizations worldwide that collect data from users in EU countries. In fact, it is often incorrectly reported the new privacy mandate applies to just EU citizens, while instead, the regulation concerns data collected from any user who is physically located in an EU country, regardless of nationality. Regardless, it is also obviously impossible to determine whether user data protected by GDPR including cookies, names, email addresses, social media posts and other information belongs to a user with an EU IP address who might just be visiting the region or is a full-fledged EU national.

    Given the far-reaching provision of GDPR that places strict controls on how third parties store and share data and protect their online privacy, GDPR likely will fall under an urgent DevOps project category, requiring the input of the change advisory board (CAB), as well an organization’s IT operations, QA, InfoSec and development teams. The degree to which GDPR will change code deployments and processes will depend on the organization’s business model, of course. GDPR compliance should also require review of qualified legal counsel.

    Meanwhile, Google Analytics says its tools can help organizations to comply with GDPR for data collection policies. These features include cookie settings, privacy controls, data sharing settings, data deletion on account termination and IP anonymization.

    However, as mentioned above, Google still need to provide DevOps with more clarification about certain changes to Google Analytics affecting organization ahead of the May 25 deadline. Many in the DevOps community are also very likely waiting to learn about other ways DevOps can use Google Analytics for GDPR compliance.

    — B. Cameron Gain

  • Why NoSQL is the New Database Darling of DevOps

    Why NoSQL is the New Database Darling of DevOps

    If developers are from Mars and testers are from Venus, your database administrator is stuck in Saturn. By nature, the database—and specifically the data it carries—has to remain stable even as the application code on top of it is changing faster than Donald Trump’s staff.

    When it comes to the speed of your release cycle, issues with your database are like potholes. You don’t see them up ahead, and you only feel the pain after the damage is already done. The least flexible layer in an application is its database, rendering it the weakest link in any agile development process.

    Get Your Database Up to Speed

    The right NoSQL database offers the following features that can speed up your release cycle without falling into a pothole.

    It’s Schema-less:  When you make a new build, or issue a new release, your code will change. Obviously, it will go through QA and check out. However, just because it’s working doesn’t mean it’s working correctly. You may have changed the types of data your application is accepting. You may have changed the way your app works with data. Your schema is the blueprint for how you accept data. For a relational database, you need one. The more complicated the schema, the harder it is to change.

    A NoSQL document database is schema-less, offering you the ultimate flexibility and speed in how to rearrange the way you handle your application’s data.

    Rapid Scalability: The moment your application is a success, expect greater load. Greater load requires more hardware to handle that load while maintaining peak performance. A relational database demands you scale up, investing large amounts in a new server. NoSQL solutions let you scale out, creating data clusters on commodity hardware.

    NoSQL lets you add new nodes to your database topography in minutes, expanding your ability to process requests from users at optimum performance.

    Immediate Adaptability: New code doesn’t only change the type of data you are working with; it also can change how you work with the data itself. New queries mean whenever a user clicks a button, your database will have to relearn how to sort your data to answer new requests for specific items or aggregated totals.

    A relational database will join tables and scour the data row by row to get your users what they want. If the whole thing changes, the process starts from square one. A good NoSQL solution uses dynamic indexing. It means that it won’t perform a query until after it finds the right index. It will either use a pre-existing index, improve on one or create one from scratch that will be made available for future queries.

    Such a system reduces the performance hiccup by releasing new code in that the more you use it, the better indexes are created to maximize the performance of the database. Your Agile DevOps team can release new builds every day and the database will be able to keep up.

    Platform testing: Agile without testing is like a lunchbox without the apple. It may not be the most appealing part of the meal, but you gotta have it. The right database lets you make queries in test mode without touching the disk itself. The data in real-time is separated from what you are testing.

    Due to its inherent flexibility, a NoSQL database lets you re-create whatever test scenarios you need and measure function and performance to see how every layer is working under the new changes. You can also do this fast, keeping your development process moving right until the end.

    Technical support that doesn’t hold you back: One of the biggest pitfalls in Agile is when you hand the fate of your next release over to people who have no idea how urgent it is that you get the issue resolved now. Bad tech support can take an issue that shouldn’t slow you down and freeze you up for hours, even days.

    The right NoSQL solution offers tech support with quick response time, and engineers who have experience developing the database itself. It is important that they know how the database works, and can do better by you than reciting a series of standard test scripts.

    Databases can also notify you of underperformance or any indications that there are escalating problems that need immediate attention before they become full-fledged issues. Small features such as these save you time in having to resort to tech support in the first place.

    Conclusion

    There are always risks involved with making changes to your application. The risk databases pose is that some changes can shut down your database and paralyze your application. A NoSQL solution can reduce this risk, and even speed up areas of your DevOps process.

    — David Ben Horin

  • Time to Rethink Infrastructure Operations in Enterprises

    Time to Rethink Infrastructure Operations in Enterprises

    “Business is technology and technology is business” seems to be the new mantra as the rules of corporate strategy are being rewritten across industries. According to IDC’s survey in 2017, “Two-thirds of global CEOs will have digital transformation as part of their corporate strategy.”  That’s CEOs—not CIOs—who will have digital transformation as part of their corporate and, not IT, strategy.

    time for change

    While digital transformation is not a new phenomenon, making digitization core to the corporate strategy is a fundamental shift for technology teams. Delivery units, or “change the enterprise” teams, have warmed up to new ways of working over the last few years and are already on the learning journey. Most enterprises today have digital or agile transformation initiatives that tend to focus on the “change” aspect significantly. However, the less glamorous “run the enterprise” dimension tends to be forgotten or side-lined in these initiatives.

    The rise of DevOps was meant to bridge the gap between “change” and “run.” However, many DevOps transformation initiatives seem to be taking the same old path of least resistance that agile transformations have historically adopted: by focusing on the “change” and delivery teams.

    As a consequence, the infrastructure and operations teams—or, “run the enterprise” units—are now waking up to the new reality and have far less time to adapt than their “change” counterparts. The big question really is, What does it mean to infrastructure and operations teams when the CEO is now embracing digital transformation?

    This means that the technology infrastructure required to support business strategy requires a combination of the usual consistency and quality that infrastructure and operations have historically focused on—but only with greater speed, innovation, flexibility and customer intimacy. For the first time, business is looking at infrastructure as equal partners on this journey, not as a subservient IT function.

    What Caused This Shift in Thinking?

    Since 2001, there have been four game-changers in technology that have caused this shift in thinking: the Agile Manifesto of 2001, adoption of Cloud by enterprises, rise of startups and the DevOps movement. The world of technology as we know has been going through a tectonic shift as a consequence of these overlapping waves of change.

    Agile and DevOps

    Version One’s 11th “State of Agile” report highlights that 71 percent of organizations have current or planned DevOps initiatives in 2018. While Agile ways focus on effective team structures, value focused mindset and greater business alignment, DevOps has taken the principles to the logical next steps of delivering value by focusing on flow, amplified feedback loops and building a culture of learning and innovation. In addition, full stack automation covering infrastructure, application delivery and release orchestration has really catapulted DevOps into the top initiatives in organizations. In terms of architecture, new constructs such as evolving design, microservices, containers and serverless computing are all redefining technology infrastructure. The rise of these new ways of working is only piling immense pressure on traditional infrastructure organizations to reform and rethink every aspect of their operating model and challenge processes built on a foundation of perceived stability.

    The Startup Way

    Enterprises increasingly are being influenced by the ways of startups and how they are driving rapid innovation. There is a cross-fertilization of people moving to enterprises who are challenging the thinking in these enterprises, especially infrastructure area. Teams are now embracing once considered absolutely alien, if not impossible, concepts such as chaos engineering and anti-fragility.

    Cloud

    The recent RightScale “State of Cloud” report reveals that 97 percent of enterprises are using cloud in 2017. However, only a small share of their workload has actually been moved to cloud, indicating the potential for cloud in the enterprise context. This indicates that enterprises are past the quick wins such as digital front-ends and will now struggle with the difficult areas of their applications—the bulk of the enterprise apps where the traditional infrastructure constraints and mindset rules. Enterprises can tackle this by truly embracing change, thinking big but starting small with experiments and continuous refinement of the every single dimension of the way infrastructure teams operate.

    “Nothing is permanent but change.” The wise and timeless words of Confucius ring true!

    — Aditya Vadaganadam

  • NoSQL Databases: Lessons Learned Growing a Globally Focused Startup

    NoSQL Databases: Lessons Learned Growing a Globally Focused Startup

    Startups deploying NoSQL databases and global citizens learning new languages actually share something in common: Both want to remove the limitations on how they access information and express themselves. In launching and growing a platform that enables anyone around the world to connect and speak with native English-speaking tutors on-demand, we’ve had the fortune of helping countless people learn new language skills—and we’ve also learned a great deal ourselves about the best ways to handle the requisite user and application data.

    NoSQL, No Problem

    Here are four NoSQL database strategy takeaways we learned from recent experience:

    Keep Your Data Options Open with Flexible Technology

    When a startup is just out of the gate, it isn’t immediately clear what shape the final product may take, or what data it will require.

    In our case, the relatively broad scope of our project also defined the need for a database able to function globally with 24/7 availability. The concept for Cambly was born out of its founders’ real experiences traveling to other countries and learning foreign languages through conversations with native speakers. Immersive experiences such as these are critical for gaining and retaining conversational language abilities, and provide learning opportunities that no language course or software can really match. However, these language skills can recede rapidly if a person no longer has access to native speakers to practice with. Seeing this as an issue that technology could address, we began building an application to provide global on-demand access to immersive conversations with native speakers, initially concentrating on English as the largest language market.

    As a startup beginning to explore how exactly we’d accomplish the task ahead of us, it was critical that we could store whatever types of data we found to be of value as we quickly iterated the application. This meant utilizing a NoSQL database solution to free ourselves from any schema constraints. We selected MongoDB as a flexible, NoSQL solution with which our technical team had some existing familiarity.

    Get Help …

    Startup success is dependent on managing finite internal resources in a way that maximizes the attention that can be devoted to product development and other growth-critical initiatives. This means having the wisdom to rely on managed services where they’re beneficial in saving time and simplifying operations, versus building up internal headcount. For example, we enlisted Heroku to use as our application platform. And, even though our team was experienced in running databases and managing machines on AWS, we still saw the value in keeping production as hands-off as possible and opted for a database-as-a-service (DBaaS) strategy as well.

    … And Make Sure It’s the Right Help

    It can be tricky, but ensuring that any and all technology service providers a startup works with are a strong fit for its needs is vital to moving quickly. We actually learned this the hard way when it came to our original MongoDB DBaaS. For us, reliable 24/7 availability and insights into database performance were essential requirements. Unfortunately, our initial provider presented us with two challenges to meeting these requirements in a short time frame: Its tools became unable to provide us with visibility into our database, and it announced a required platform migration that could not be completed without downtime. Naturally, these changes led us to seek a better-fitting option. Soon after, we partnered with mLab in large part due to rolling node replacement capabilities for migrations and 24/7 uptime, as well as support and tooling that could help our team quickly optimize MongoDB instances.

    Get the Database Tools You Need to Succeed

    From reliable data backups to performance insights and beyond, the right toolset can make or break a burgeoning company’s success. Just as language skills must be constantly attended to, your database requires intelligent upkeep and backups to ensure it delivers a satisfactory user experience and remains available in case of disaster. This is one lesson which—very fortunately—we were able to absorb without hardship. We had an incident in which our team accidentally deleted a trove of valuable data. But, with data backups in place, our DBaaS was able to help us restore that data quickly—and without us stressing over it as we would have otherwise. At the same time, the DBaaS tools allow us to recognize and address database issues while improving indexing and performance.

    — Gar Higgins

  • A Unified Route to Bring Mainframes to DevOps Speed

    A Unified Route to Bring Mainframes to DevOps Speed

    Most businesses have mainframes where their key applications and data reside. In today’s digital environment, customer-facing mobile and web systems retrieve critical data from back-end mainframe systems. It is critical to adapt mainframe applications and data to today’s rapidly changing market demands if businesses want to remain competitive.

    This is the question IT and businesses are facing today. How can they integrate their well-established mainframe environments and have a unified cross-platform across the enterprise for DevOps to manage releases across mainframe and non-mainframe applications?

    An Innovative Digital Solution

    The answer lies in mainframe digital DevOps. This helps enterprises remain competitive in an app-centric economy. Through a unified cross platform, DevOps is integrating mainframe tools with open source DevOps tools.

    This approach allows organizations to retain their legacy code and does not require legacy code migration—especially when the back-end core applications are on mainframe. As veteran COBOL programmers retire out of the skill orbit, this becomes even more important. There is an imperative and an expectation for agility and higher productivity from IT—without jeopardizing performance, customer experience, security and other compliance factors.

    Transforming the Developer Environment

    Most mainframe development, test and ops work is executed in antique “green screen” SPF environments, which calls for specialist knowledge that limit productivity among programmers of Agile and DevOps transformation. Turnaround times of monolithic mainframe applications are painfully slow compared with the new-age agile technology of mobile and web applications. This can be frustrating for developers who wish to make frequent changes to mainframe code to fulfill the urgent needs of business.

    Therefore, it’s important to make the shift to modern tools and integrate mainframe with non-mainframe DevOps processes to deliver cross-platform services. This will improve staff productivity across development, testing and production operations and enable inexperienced mainframe professionals to perform application development and enhancements with ease and efficiency.

    Large mainframe clients embark on transformation by modernizing the workspace to work on both mainframe and non-mainframe applications. Open system developers are familiar with new-age IDEs such as Eclipse, which have user-friendly interfaces. Similarly, other mainframe tools are available that provide source code editing, debugging, fault diagnosis and browse data to enable developers to write, compile and run code from a modern Eclipse-based IDE. Developers can drag and drop files between LPARs and perform other tasks to improve productivity metrics. Collaboration between the two teams will motivate mainframe developers to use DevOps principles to integrate mainframe with distributed and cloud applications.

    Once quick wins are established, adoption levels of Agile and DevOps will improve among mainframe developers and business.

    Eliminating Barriers to Unit Testing

    Historically, mainframe developers have not been accustomed to unit testing. Automated unit testing is common in Java but not in the mainframe environment. Agile coaches, familiar with mainframe tools, will need to enable collaboration in interactively testing smaller pieces of code.

    Unit testing can improve throughput by increasing the frequent code drops in production environment and help to accomplish DevOps key performance indicators (KPIs) with fewer errors in the later stages of the software development lifecycle process. Some tools can facilitate this process by automatically breaking the mainframe code into discrete logical units, and creating tests for such logical units.

    Tracking and Monitoring Progress for High Quality

    Visualization is an important element of DevOps for improved visibility to track and monitor progress across the enterprise. As most mainframe applications and data structures are large, complex and not well-documented, enterprise IT is highly dependent on the personal and institutional knowledge of senior mainframe staff. Graphical visualization can help quickly “read” existing application logic, program interdependencies, data structures and data relationships. Business rule extraction tools are available in the market to reveal logic and data relationships through graphically intuitive diagrams. This helps developers work independently on unfamiliar programs to better estimate project work, meet delivery deadlines and commit timelines to business.

    Most mainframe applications are often associated with core business processes with zero tolerance for error. Any attempt with inadequate mainframe experience can lead to catastrophic non-recoverable error. Continuous integration (CI) is critical in determining application quality issues early and establishing KPIs to deliver continuous quality code in production. Enterprises have started embracing modern tools and techniques to transform from the Waterfall delivery model to Agile. Modern DevOps tools embrace CI, and certain tools can quickly detect application quality issues (such as unbalanced or unmatched working storage and data items, and out-of-range nested conditions and branches).

    Training for Effective Enablement

    Successful implementation of cross-platform DevOps calls for the right training on Agile tools, processes and methodologies. A single Scrum team can be formed to collaborate with developers across mobile, web and mainframe. The Agile mainframe team will need to focus on stories and epics that capture specific units of business value instead of technical tasks in a project plan. Open system tools such as JIRA and Confluence that can support Scrum and Kanban can be used to manage tasks and improve collaboration through centralized and well-organized web tools. This will incorporate Agile values such as transparency, knowledge-sharing and ideation to improve the developer’s performance.

    The move to Agile scrumming represents a transformative shift of mainframe work culture, and herein lies the importance of the right coaching in Agile processes. A “2- n a box” partnering model of mainframe developers and experienced Agile developers from other platforms can create the required collaboration.

    Operational Data Analysis to Close the ‘Last Mile’ Loop

    Good quality code alone is not enough. Operational data analysis is required to reduce costs by discovering and mitigating avoidable CPU consumption caused by code inefficiency.

    Mere monitoring and reporting by triage teams is not enough to support and maintain applications in the world of DevOps.  Production data analyses by enabling an auto-feedback loop is imperative to improve productivity of the dev, test and production teams. Tools are available that can quickly pinpoint bad statements and pinpoint application inefficiencies such as bad SQL statements. Tools also help identify the causes of excessive CPU consumption and slow data retrieval.

    Automation for Speedy Efficiency

    Agile-enabled core source code management (SCM) is a fundamental requirement of DevOps. Traditional SCM cannot support parallel development work on different user stories. SCM migration is therefore essential to increase the speed and frequency of mainframe code drops. Automating the change management process and defining the proper workflows can eliminate manual steps and reduce code approval delays.

    Again, tools are available that can fulfill business expectations by seamless SCM migration without disrupting the current work in progress.

    Automated release and deployment is pivotal to keeping pace with today’s fast-moving markets. Business needs to quickly see reliable new code in production.

    Toward a Unified IT Service Management

    Cross-platform continuous delivery is vital for efficient use of mainframe applications and their data for multiplatform customer-facing applications, including mobile, web and cloud components. Mainframe DevOps process should fully synchronize the code delivery process across all platforms. This will establish a centralized deployment control center to provide a unified cross-platform and accurately report progress. Tools can be the single point of control for all changes across z/OS, Windows, Unix and other platforms. REST APIs are flexible to mix and match best-in-class tools and avoid vendor lock-in.

    The ultimate destination is to create a de-siloed environment where mainframe is just another platform with high scalability, reliability, performance, security and cost-efficiency. A unified mainframe digital DevOps platform may be the answer to modifying traditional process that fulfills this business demand.

    — Ramesh Ganapathy

  • DevOps: Role Clarity is Important

    DevOps: Role Clarity is Important

    Those who talk a lot about DevOps, particularly those who talk a lot about culture change, sometimes gloss over or “think it is obvious” that some roles change more than others when moving from Dev and Ops to DevOps.

    I was having a lighthearted chat with a friend who works for Compuware and realized that, even though several of us have mentioned it before, it bears repeating for those who have not yet dipped their toe into the DevOps pool too deeply.

    The driver for DevOps is increased agility, and the belief is that Dev and Ops will merge. What is often left unclear is that they may well merge into one team, but every team member will not become an expert at every role. Pundits seem to think this self-evident. I’m not so certain it is, if you read a lot about DevOps.

    By way of example, your Cisco ACI specialist (or ios specialist) will not suddenly become an expert in the intricacies of your development toolset and environment so they can contribute code. Granted, that is an extreme example, but there are two reasons why someone is a specialist in what they do: 1. They know/prefer it, and 2. The company had a need for their specialty. Don’t lose sight of those facts while moving to DevOps.

    A better example is DBAs. They’re not going to become network engineers just because they were put on a DevOps team. They’re still going to be DBAs … because you need DBAs. You need network engineers also, but if you need them, you already have them.

    Remember that increased communications and increased latitude are the social aspects of DevOps. The DBA is still a DBA, but faster access to queries and those people writing the queries means more optimization of some of the busiest sections of code. If your EMC specialist has access to the entire team, conversations along the lines of, “We’re running low on disk on LUN X, and it will cost us a billion dollars to upgrade, what are our options?” can happen quickly and include views from across the DevOps team (normally across the application or application portfolio).

    One of the most thankless jobs in IT is being the single/small shop. Anyone who has been IT for an entire org—particularly one with in-house apps—knows the struggle. My time as a developer hired to work on list serv software but who also inherited the entirety of systems administration (the sys admin quit and changed all the root passwords the day before I started) was not optimal. I learned a ton, which was cool, and I never mastered most of it, which was not.

    The goal of DevOps is not to re-create that environment in a team. It is to optimize team interaction, and make things go more smoothly. Make sure you are not on an implementation plan that leads to burnout or lack of accountability. Keep your eye on the real prize: increased responsiveness to business needs.

    And as always, keep kicking it. I have yet to walk into an environment that wasn’t unique, or one that didn’t have cool stuff to learn about. And you keep that running.

    — Don Macvittie

  • The Road Ahead for Service Mesh

    The Road Ahead for Service Mesh

    If you’re struggling to manage microservices as architectures continue to scale, there’s a good chance you’ve at least heard of service mesh. For the purposes of this blog, I’ll assume you’re familiar with the basic tenets of a service mesh.

    I believe that service mesh is advancing microservice communication to a new level that is unachievable with the one-off solutions that were previously being used. DNS patterns, configuration management and other technologies provide some capabilities such as service discovery, but they don’t provide fast retries, load balancing, tracing and health monitoring. The old approach also requires that you cobble together several things each time rather than bundle everything into a reusable tool.

    While it’s possible to accomplish much of what a service mesh manages with individual tools and processes, the process is a pain in the ass. The image below provides a good idea of how a mesh simplifies the management of microservices.

    Right Around the Corner

    So what’s in the immediate future? I think we’ll see the technology quickly mature and add more capabilities as standard features in response to enterprises realizing the efficiency gains created by a mesh and look to implement them as the standard for managing microservice architectures. Service mesh is likely immature for production deployments, but the road map is progressing quickly and it seems we’ll be to v1 in short order. Security is a feature provided by service mesh, but for most enterprises it’s a major consideration and I see policy enforcement and monitoring options becoming more robust for enterprise production deployments.

    A feature I see on the near horizon—and one that will provide tremendous value—is an analytics dashboard to show insights from the huge amount of telemetry data in a service mesh. I think an emerging value proposition we’ll see is that the mesh allows you to gain and act on data that will allow you to more efficiently manage your entire architecture.

    Further Down the Road

    There is a lot of discussion on what’s on the immediate horizon for service mesh, but what is more interesting is considering what the long term will bring. My guess is that we ultimately come to a mesh being an embedded value-add in a platform. Microservices are clearly the way of the future, so organizations are going to demand an effortless way to manage them. They’ll want something automated, running in the background that never has to be thought about. This is probably years down the road, but I do believe service mesh will eventually be a ubiquitous technology that is a fully managed plug and play config. It will be interesting to see new ways of using the technology to manage infrastructure, services and applications.

    We’re excited to be part of the journey, and are inspired by how users are leveraging service mesh to solve direct problems created by the explosion of microservices and also find new efficiencies with it. I look forward to seeing what the community comes up with that I have not yet even imagined.

    — Zach Jory

  • Of DevOps, Databases and DBAs

    Of DevOps, Databases and DBAs

    If you’re on a DevOps team looking for additional efficiencies, you are likely to find them in the realm of database management. The arrival of DevOps did nothing to diminish the responsibilities traditionally associated with database administrators (DBAs). Rather, some of those duties have been redistributed across the team—sometimes explicitly, but more often implicitly. For example, site reliability engineers (SREs) and even full-stack engineers have taken on many of the responsibilities that once belonged to DBAs. And in these roles, DBAs often hold the keys to unlocking a new dimension of efficiency through consistency, repeatability and operational efficiency.

    Enter the “DevOps DBA.” This confluence of roles makes a lot of sense. As with application DevOps, it’s logical to assume that the person who architected the platform and wrote the database code also should have a role in deploying and operating it. DBAs manage every aspect of data infrastructure, from capacity planning, installation, configuration, database design and migration to operational duties such as performance monitoring, security and troubleshooting, as well as backup and data recovery. As a result, the DBA’s responsibility for maintaining existing workloads sometimes conflicts with the need to plan and provision workloads for the next generation of applications. Reconciling these responsibilities can speed up the software development life cycle as a whole.

    On the operational side, DBAs are responsible for monitoring database behavior under peak application workloads, ensuring they’re performing according to expectations, and working with Operations to resolve bottlenecks and outages. They manage data access and the overall security of the platform. They also perform release activities in support of the application and troubleshoot any errors that happen during that process or during day-to-day operation.

    With all this in mind, we can propose a DevOps DBA checklist:

    • Establish a microservices plan with regard to data services.
    • Set up integration environments, and provision databases for new projects.
    • Dynamically extract, transform and load data from one silo to others, employing TTL strategies to reduce the data or enrich some of it.
    • Continuously monitor and optimize performance: Strive for low and consistent tail latency and a joint view of the app and the database.
    • Expert zone: Automate scale out and automate schema changes.
    • Backup and recovery: Backup to cold storage and restore.
    • Plan for downtime and high availability using multiple data center disaster recovery capabilities.

    Characteristics of a DevOps-friendly Database

    We’ve looked at some of the DBA responsibilities as they apply to DevOps. Now we’ll discuss some of the technical aspects of database technologies that can help, or hinder, this hybrid role. To keep up with applications as they migrate through lifecycle stages, there are a number of organizational and team cultural challenges to be addressed. It’s also important to remember that not all databases are created equal—some databases lend themselves more readily to the DevOps toolchain than others.

    It takes a combination of elements for a database to be DevOps-friendly. The ultimate goal is to minimize the fear of change by mitigating risk. Relational databases do not conform particularly well to modern software development practices, since changes to schemas can be time-consuming. NoSQL databases, in contrast, have proved to be more life cycle-friendly because they don’t impose rigid schema management during application development. Over time, however, this free-wheeling approach to data modeling can create headaches. The ideal database strikes a balance between strict schema enforcement and decentralized data-model changes.

    A key consideration for a DevOps-friendly database is operational simplicity. Traditional SQL databases, such as Postgres and MySQL, are well-understood and have lots of support, but can be difficult and expensive to scale. The rise of NoSQL databases enabled larger datasets to be managed in a more flexible and scalable manner than relational databases. But even among NoSQL databases, there are some significant differences. The first generation of NoSQL databases—in particular the widely adopted Apache Cassandra—can impose significant operational burdens in their own way.

    Additionally, modern cloud-native databases have their limits. They can be opaque to operators, and have been known to scale up and down unpredictably. Operators often lack visibility into the distribution of datasets. In many cases, cloud databases also prove to grow at a rate that’s disproportional to the workload supported by the database. Database instances can proliferate to the point where DevOps has to deal with “node sprawl.” You can think of this problem as the false promise of elastic scale, wherein the solution to increased traffic is, “Just add more nodes!” Lots of small nodes are not necessarily better than a few large ones. (For more on this, I’d encourage you to watch this talk on mythbusting the advantages of small nodes.) In any case, the biggest overall cost isn’t necessarily capacity, though that can be significant. The biggest cost, it turns out, is often administering all those nodes.

    Node sprawl can result from an overabundance of caution; overprovisioning is a natural response to scaling challenges. Look for a database architecture that uses resources efficiently and autonomously, ideally one that can dynamically tune itself according to varying workloads. An autonomous database monitors operations and adjusts internal processors to achieve a balance. Not only does an autonomous database greatly reduce administrative burden, it also means DevOps can always push the database to the limits of the underlying hardware resources, ensuring availability and SLA compliance. This again reduces the need to tailor database configurations and topologies to specific applications.

    Flexible deployment options are also important for DevOps. Who knows what your footprint will look like in 12 or 18 months? A database needs to be fairly agnostic as to topology on which it is deployed, whether that’s on-premises, a cloud platform or a hybrid configuration across both local and cloud environments. In some deployments, databases may be required to run across two different commercial cloud platforms—for example Azure and AWS—simultaneously. There are a couple of reasons for considering this: The first and more obvious is to maximize availability. Geographical outages are more common than you might expect. The second benefit are the financial advantages to the business using the hybrid framework, as it prevents infrastructure-as-a-service (IaaS) vendor lock-in, while also providing tax benefits for the on-premises expenses. Forward-thinking DevOps groups will keep these potential requirements in mind when designing data platforms today.

    Conclusion

    Trends in application development and IT infrastructure are pushing traditional DBAs into DevOps, while DevOps (including SREs and full-stack engineers) are often taking on responsibilities traditionally associated with the DBA. By adopting data infrastructure that emphasizes operational simplicity, ease of scale and deployment flexibility, IT organizations can more easily reconcile these roles while building a cloud-friendly infrastructure that extends the DevOps toolchain to the database.

    — Eyal Gutkind

  • The Need for Rapid Development in a Post-App World

    The Need for Rapid Development in a Post-App World

    Gartner’s May 2017 report, “Preparing for a World Beyond Mobile Apps,” anticipates a post-app environment in which consumers have a wide variety of interfaces to choose from—including wearables, bots, ambient voice interfaces, IoT devices and AR/VR—that not only lend themselves to more immersive and seamless digital experiences, but also upend how businesses need to think about developing and delivering experiences. The post-app world, which we’ve begun to see accelerating in 2018, requires a major shift toward a rapid development framework at a time when agile methodologies are becoming overly costly and time-consuming.

    Examples of the types of experiences requiring rapid development often surface in highly competitive areas such as retail.  Examples include a luxury car dealer greeting customers with a large wall display with depth-sensing 3D camera, gesture-based interface and face recognition. As the customer gestures around the screen, an application recognizes his face, alerts the service manager that he’s there to pick up a car from repair, and harnesses his sales history and its loyalty program to make a special offer if appropriate.

    Then there are highly scalable opportunities to offer better capabilities while lowering costs such as chatbot technologies for interactions with vendors and suppliers and numerous HR functions. Retailers such as Gap, Target and Ralph Lauren have started incorporating digital augmented reality, interactive mirrors, beacons and other advanced interactive interfaces to enhance the physical shopping experience.

    Immersive experiences and physical/digital blending calls for a rapid development framework in which developer resources are utilized on infrastructure but not wasted building custom features that can be easily integrated via pre-built modules. This frees developer resources and allows lines of business to have more influence on the resulting experience.

    Rapid development also demands a unique mix of emerging strategies, tools and technologies to achieve the best immersive user experience fast:

    Design Thinking

    To paraphrase Steve Jobs, design thinking is not about how a device, software product or business process appears, but how it works. It requires thoroughly understanding and empathizing with the user’s needs and continually coming up creative ways to meet them, so the interaction is not only immersive but effortless. Design thinking must involve the user—whether customer or business—and must be a process of continuous exploration, prototyping, feedback, evaluation and optimization. Even then, innovation continues as needs change and evolve.

    The ultimate goal of leading with design is to keep the focus on ensuring a superior user experience.

    Microservices

    Microservice architectures allow teams to continually optimize individual components of a larger application without affecting others, thanks to loose coupling, REST and real-time APIs. Scores of application components may exist on-premises or in the cloud and can interact and pass data back and forth easily but each is limited to one or a few functions. Since changes and optimizations in one component don’t necessitate significant changes in the others, teams can optimize each function continually without having to re-engineer an entire application infrastructure. These components can also be shared with other applications if necessary.

    Agile Engineering

    Agile engineering is one component of rapid development, rather than the primary focus, and the goal should be improving the connection between the rapid process of design thinking and quickly evolving microservices and the challenges of software engineering. Even in an agile process the DevOps cadence required to deliver robust and secure software requires multiple sprints of two to four weeks in length. In successful projects, the majority of the layout and workflow falls onto the rapid development side. Then, the investment in additional time and resources is only required for the remaining elements that need an additional level of control and engineering rigor.  Agile engineering is the perfect partner for design thinking as it is designed around the ability to accept well specified requests into backlog and turn them around with high quality in the minimum number of cycles.

    The Way Forward

    Rapid development tools enable quick prototyping, deployment and easy continuous integration with existing systems. Intuitive editors and extensive libraries of application components and immersive interfaces such as VR, AR, personal assistants and IoT components enable developers, business users and designers to pull together prototypes of post-mobile components in hours or days. This allows quick feedback and evaluation at very low cost.

    Developers can then extend and customize the prototypes further via scripting so they become the actual production system. The best come tools with an ample supply of connectors and APIs to link with existing systems and run natively on many different mobile platforms, devices and displays. The right tools can accelerate innovation and cut costs dramatically.

    Radical digital transformation has arrived and organizations will have to adopt post-mobile technologies to compete. The right mix of tools and strategies will be key to redefining the enterprise development process from application-based to immersive-experience-based regardless of device, operating system or infrastructure.

    — David Brebner

  • DevOps and Low-Code/No-Code Tools

    DevOps and Low-Code/No-Code Tools

    Helping out a friend, I had reason to revisit low-code/no-code solutions and poke around a bit. The industry has been relatively quiet on the topic, but changes in systems and marketing made me feel the need to offer an update on the relation of current solutions to DevOps.

    Low-code/no-code solutions have been a panacea in the eyes of some both in and outside the industry for decades. 4GLs were the earliest widespread attempt to make an environment that eliminated the need for developers, and while some years are quieter than others, the movement has not slackened.

    There are even some things we should consider adjuncts to low-code/no-code environments that normally are not considered. Traditionally the drive for low-code/no-code solutions is to move faster than IT normally turns out solutions. This would force us to consider the spreadsheet in accounting with 50,000 lines of special functions as a companion solution—faster than waiting for IT and more agile.

    The thing is that low-code/no-code environments have increasingly realized that IT is going to be their best customer, adjusting their offerings to focus more in that direction. At the same time, agile development and DevOps have made low-code/no-code solutions more palatable to an IT shop that is looking to continue the gains brought about by these technologies.

    The future appears to trend toward low-code/no-code solutions being an adjunct of agile and DevOps. I don’t say this lightly, but the signs do seem to point that way. Agile shops, having become more responsive to business needs, will hit a limit on what they can deliver, while the appetite for faster and more responsive systems delivery will not abate. Low-code environments that offer the following will become a very real option to extending the gains of agile and DevOps:

    • A clear and understandable way to code when needed in the language of choice for the IT environment.
    • A high level of feedback on what is wrong, possibly including suggestions on how to fix it.
    • Transparency in what is actually happening behind the scenes with the “no-code” part of the system.

    The idea is simple. By handling tasks that we, as developers, do over and over and over, we are freed to spend more time doing the truly innovative and environment specific work. This would seem to offer the next step in the growth of agile.

    Those tools that offer options such as automated deployment will step into ARA land also, making the Ops side of DevOps happy—in shops that haven’t invested a fortune in setting up ARA already, anyway. If low-code can offer the ability to say, “Here is a problem,” then the developer can submit a fix, and it is submitted to test, and if tests pass is sent to production (and some of the tools do), then the only questions are, “Does the code generated by the low-code part perform? and, Is it easy to integrate existing libraries, etc.?” If the answers are yes, then we also have an add-on for DevOps that delivers value.

    We live in interesting times. It will be very cool to see this all play out, and interesting to see where it heads after this. At least one vendor is betting on artificial intelligence (AI) and machine learning (ML) to offer increased productivity.

    Of course, it isn’t a panacea. I read where a vendor said, “We automatically update your queries when a table is modified,” which spawns another dozen questions for me, starting with “Really? What do you do about code that was counting on that field much later in processing? Do you check that screens are displaying defaults, or that variables used to make decisions aren’t always the default value? Do you flag this stuff so devs can ‘fix’ what automated query changes ‘broke’?” I’ll be digging in some more, because now they’ve got my attention. So you’ll probably be hearing more from me about this topic.

    Meanwhile, let the future worry about the future, and keep kicking it today. Just stay aware of what else is going on out there that might be handy in your toolbox.

    — Don Macvittie