devopsy.kinsta.cloud

Tag: DBAs

  • Rethinking Data Management in a DevOps Environment

    Rethinking Data Management in a DevOps Environment

    You often hear data is the new oil. This valuable, ever-changing commodity has begun to play a starring role in many cloud-native applications. Yet, according to a number of DevOps teams, data issues continue to plague their efforts to continuously integrate, test and deploy frequent software releases. More specifically, issues with persistent data (and its underlying database engine) often appear to be the culprit.

    Your organization might be pursuing emerging IoT applications that incorporate and analyze sensor data from multiple sources. Or, your DevOps team may be trying to develop applications that extract further actionable insight about customers. Whatever the use case, there’s no doubt back-end database architectures have gained growing importance to the success of such projects. Yet, in many cases, such database systems appear to be having difficulty just keeping up with the pace of today’s DevOps pipelines.

    According to one report on the state of database deployments in 2019, 46% of DevOps teams on an accelerated release schedule (with weekly, or even daily, releases) found it “extremely or very difficult” to speed their database release cycles accordingly. In a related Redgate report on the 2019 State of Database DevOps, 20% of respondents cited slow development and release cycles as one of the biggest drawbacks to “traditional siloed database development practices.” Another 23% saw a higher risk of deployment failure and extra downtime when database changes were introduced in a traditional database environment.

    Getting to the Heart of the Data Problem

    Are such database issues the result of the wrong choice of underlying database technology–such as the use of an RDBMS, NoSQL or even a NewSQL system? Possibly.

    Maybe database issues are also caused by poor (or non-existent) cross-communication between database experts and their developer counterparts? Undoubtedly, this is true as well.

    Would you be surprised to learn both situations (wrong technology and poor communication) are often to blame? Also, to blame is the urge to solve new data problems in the same old way organizations approached their earlier, legacy use cases.

    Ultimately, as with many things DevOps-related, the answer to the data problem is likely to require change along three, separate fronts: People, process and technology. It will also require a fundamental shift in how such issues are approached at the start.

    Starting with People and Process

    Instead of worrying about database slowdowns, what if you began again with how DevOps teams first approach the many facets of the underlying data layer? This means:

    1. From the start of a project, include DBAs as part of the cross-functional DevOps team. This will help promote healthy cross-communication. They will also positively influence the development of appropriate underlying data layers and infrastructure that support your emerging use cases. In larger organizations that manage many data sets, these may be specialized database administrators (DBAs). In smaller companies, these may be more full-stack engineers who also have high-quality expertise in database operations.
    2. Identify early the different data types, domains, boundaries and optimal cloud-native patterns associated with your data. Once these are established, the team can gain a better understanding about the different ways to develop applications and data architectures for each new use case.

    In effect, organizations need to make a concerted effort to shift left with their overall data architecture discussions. This shift allows more of the right questions about data to be asked, answered and incorporated early on in the design of the overall application.

    Asking the Right Questions About Data

    Part of this early collaborative work may involve asking key questions about data. These include the best ways for an application and its users to interact with the underlying data, such as:

    • How will data be ingested and from where?
    • How will we access the data?
    • How and where will we store it?
    • How do we update and/or maintain legacy databases as we build for new use cases?
    • What types of data will we have? How will the types change or be increased over time?
    • How will we query from it? And what are the most common queries users will want to run?
    • How will we manage and scale the data in use?
    • How will we protect the data from data loss, disaster or corruption?
    • How will we secure the data?

    If these questions sound a lot like features in a data lifecycle, you’re right. But, such basic questions about data are often overlooked in the rush to deliver, integrate, test and deploy application code. So often, what happens instead is an application team creates a design, then hands it off saying, “Okay, now create a data architecture to support the application.”

    This fundamental miss with data and collaboration at the start often leads to poor database architecture decisions later. It can also cause DevOps teams to resort to workarounds to accommodate such wrong architecture choices made at the start–which may cause some development teams to choose to circumvent DBA involvement altogether. In this case, they might even find themselves opting to use public cloud services instead (or some other types of data management) to support their application.

    Instead, application teams should choose the path of early collaboration between developers and all functions of IT (including DBAs). This approach can offer better DevOps outcomes. It also helps move organizations toward a future where data architects provide an effective bridge between all parties critical to the application’s success: From the needs of developers to the platforms provided by DBAs and the underlying infrastructures needed to enable them.

    Changing Skillsets, Changing Mindsets

    As with most things DevOps-related, success at the data layer has as much to do with changing the education, mindset and culture toward database architectures and database deployment methods as it does with the specific steps in the DevOps pipeline.

    For example, if most DBAs in your organization are narrowly focused in one area (such as an Oracle DBA or SQL Server admins), your organization may benefit from investing in education to close skill gaps. Such investment can help expand the knowledge and exposure of that workforce to DevOps practices or emerging database approaches to cloud-native application patterns.

    It can also help some organizations to first identify those individuals in the organization who want to transform, innovate and learn about new tools, methods and practices.

    Hiring those with that expertise already in place is another option. When you hire new skillsets, however, the answer may not be just to hire several NoSQL administrators. This tactic will not necessarily solve your data problem, either. It may be just as beneficial (or more so) to hire someone with little specific NoSQL skills but who is good at thinking about data in a different way.

    Business First, Technology Second

    In our practice, we are often asked to weigh in on technology choices to support DevOps efforts in emerging areas such as IoT, Big Data and advanced analytics. Organizations ask us about using traditional relational databases (SQL Server, Oracle, etc.) versus NoSQL databases (MongoDB, Cassandra, etc.). They ask about the merits of one NoSQL iteration over another. They ask about how to manage persistent data in container environments. They even ask about workarounds when the data choices they’ve made cause other unexpected problems.

    We do our best to answer these questions. But, whenever possible, we also tell organizations to back up a few steps and start, instead, from their business objectives:

    • What are you trying to accomplish?
    • What are your business drivers?
    • How do you hope data will be used in these contexts?

    Answers to these questions can help you make better choices about the best data architecture to support your growing applications.

    Ultimately, new tools and technologies can enable a lot of impressive data architectures. But, just throwing tools at the data problem is not enough. Early effort and investment in people, process and cross-departmental communication is just as important to a successful data outcome for any project.

    Many of the discussions we have now are trying to take everyone back to the steps they skipped. This harks back to the fundamentals of software engineering: Addressing things early so you don’t have problems later.

    Unfortunately, data is the one component in the technology stack that you can’t easily undo once you’ve chosen a specific path with your application’s data architecture. When it comes to planning for data, it really is a pay now or pay later equation.

    Isn’t it best to take the time and choose your data path wisely rather than paying later for slowdowns or other issues with deployment and performance?

    — Jeff Bozic

  • The Impact of DevOps on Database Monitoring and Management

    The Impact of DevOps on Database Monitoring and Management

    The rise of DevOps has broken down silos and brought database administrators (DBAs) and application developers closer together to deliver software faster and in a more agile manner. This near-constant change driven by DevOps is reflected in the results of the Redgate 2019 State of SQL Server Monitoring Report.

    It reveals that 73% of organizations now deploy database changes multiple times per month, and 15% deploy multiple times per day, driven by DevOps. Yet, this is just one of the challenges DBAs face today. The research also identified the size of database estates is growing–45% said they will have more to manage over the next year, with 43% remaining the same.

    Additionally, they are increasingly looking to migrate their databases to the cloud (cited as their biggest challenge by 22%) and are seeking to solve performance issues. Perhaps unsurprisingly, given these greater workloads and more frequent changes, human error is the largest cause of issues (22%), ahead of capacity problems and bad deployments.

    Managing in a DevOps World

    The database is now a core part of DevOps for many organizations and DBAs need to work more closely with their developer colleagues, adopting new technologies and tools to ensure business success.

    This essentially involves DBAs focusing on the following distinct areas.

    Standardizing Approaches Across DevOps Processes

    The monitoring report highlighted human error as the biggest cause of problems for DBAs–and with larger DBA and developer teams now working on databases together, adopting standardized processes is vital to flagging issues early and preventing later problems. Otherwise, what starts as a minor issue can swiftly escalate to become a significant business crisis.

    As well as standardizing processes, organizations should look at rolling out industry-standard coding tools across the entire team. This will ensure a consistent approach, not matter who is writing, reviewing or editing code. Introducing a team style for database code, rather than having the code written in many different ways, for example, will make the code base consistent and clear and minimize the confusion that lead to errors. At the same time, merging code early and often with version control tools will ensure you have a known, shared codebase on which to work, and provide a single source of truth that can always be relied on.

    Automating for Speed and Consistency

    Developers who have adopted DevOps already see the value automated processes such as continuous integration, continuous delivery and continuous deployment provide. Automation not only speeds up development, avoids manual errors and reduces laborious work, but also provides a full audit trail for regulatory compliance.

    Extending this to the database can deliver major benefits for hard-pressed DBAs. Every time a change is committed to version control, for example, a continuous integration process can be triggered to test the change and flag up any errors in the database code. The errors can be fixed immediately and tested again, before the change is then passed along to a release management tool where it can be reviewed before being deployed to production.

    Overall, automation dramatically brings down time to market, but organizations need to ensure they have the right tools and processes in place if it is to be truly effective.

    Stepping Up Monitoring to Drive Continual Improvement

    One of the mantras of DevOps is continual improvement, and that applies equally to database DevOps. Monitoring is a key part of this. On one level, monitoring tools highlight potentially disruptive performance issues (cited by DBAs in the research as a major challenge), enabling them to be fixed quickly.

    Beyond this, they also deliver the insights and trends companies need to improve, helping them optimize operations moving forward. Thanks to a combination of DevOps and the growing importance of the database to business success, many organizations are already sharing this information. For example, 40% give developers access to their monitoring data, according to the research, which helps to enhance performance by highlighting opportunities for improvement.

    Guarding Personal Data and Ensuring Regulatory Compliance

    We’ve all seen the spread of data privacy regulations across the globe. Over 60% of the world’s population is likely to be covered by more stringent legislation in the near future, with California joining the EU in enforcing new regulations from January 2020. Organizations need to realize they are now data guardians, not data owners, when it comes to personal information.

    At the same time, developers like to have realistic information in order to successfully create and test new code. That means organizations need to adopt tools for data masking and anonymization that can protect confidential and personal information, while still providing realistic and usable data for development and testing. This also gives a full audit trail for regulatory requirements, balancing speed and compliance.

    DevOps has broken down the traditional silos between DBAs and developers and the importance of information has put the database center stage. As the latest Redgate research shows, DBA workloads are increasing, meaning they need to adopt new technology and collaborate more closely with developers if they are to successfully manage in a DevOps world.

    — Matt Hilbert

  • Integrating the Database into DevOps for Business Success

    Integrating the Database into DevOps for Business Success

    The benefits of DevOps have been known for some time, but this year’s “Accelerate: State of DevOps” report shows that successful companies are extending the advantages they get from adopting it. The longest-running research of its kind, the report from DevOps Research and Assessment (DORA) represents five years of work surveying more than 30,000 technical professionals worldwide.

    Based on their input, the report previously split organizations into low, medium and high performers. This year it added a new category—elite performers—who are embracing DevOps to raise the bar when it comes to the benefits they gain. Elite organizations are able to deploy code 46 times more frequently than low performers, lower lead times from committing to deploying changes to less than one hour (compared to one to six months in low performers), and recover from incidents 2,604 times faster.

    The Importance of the Database

    For the first time the report also focuses on the vital importance of the database to successful DevOps, echoing research from Redgate. As it points out, deploying changes to the database is often a bottleneck, slowing down DevOps. To overcome this, the Accelerate report recommends integrating database work into the software delivery process, based around four good practices:

    Strong Communication and Cross-Team Collaboration

    Traditionally, the teams involved in application and database development have had different priorities and responsibilities. The biggest concern with the database was to limit change to keep data safe, while application development was all about creating features, often without regard to what impact they would have on the database.

    With DevOps, this has changed. The faster releases of smaller changes it encourages means the database cannot be a logjam at the end of the development pipeline. Instead, it needs to be part of the same workflow, so good, ongoing communication is vital.

    In many organizations this is already happening: Fifty-eight percent of respondents in Redgate’s 2018 “State of Database DevOps” survey said that collaboration between developers and database administrators (DBAs) was either good or great, showing that the processes and teamwork are in place to break down any walls between departments.

    Comprehensive Configuration Management

    Using configuration management tools for software development is a central part of achieving successful DevOps results. To avoid issues, these same practices and tools need to be extended to cover the database as well. Handling changes systematically creates a seamless flow that enables faster releases and dramatically reduces issues—vital when more than a third of respondents to Redgate said they deployed database changes either daily or more than once a week.

    Extending Version Control to the Database 

    As part of taking a holistic view, the Accelerate report found that extending version control from application code to database code delivered major benefits. Elite and high performers store database changes as scripts in version control and manage these changes in the same way as application changes. This is a task that is shared between DBAs and developers—75 percent of respondents to the Redgate survey said that application developers now build database deployment scripts, and were responsible for deploying these database changes to production in nearly half (47 percent) of organizations.

    Building a Strong Culture

    Culture is a strong factor in whether DevOps succeeds or fails in an organization. This is unsurprising given that DevOps is a very different process to those it replaces, focusing on speed, collaboration between teams and a willingness to be open and continually learn and improve.

    Most people are now familiar with the work of sociologist Ron Westrum, who characterized the culture within organizations as one of three types:

    • Pathological (power-oriented) companies work in silos, don’t cooperate and are quick to apportion blame if something goes wrong.
    • Bureaucratic (rule-oriented) companies are more open, but are still suspicious of novelty and tolerate limited co-operation between teams.
    • Generative organizations are most suited to successful DevOps, with a focus on co-operation, surfacing problems, breaking down silos, holding postmortems to understand why things have not gone well and continually experimenting to drive improvement.

    To create the right culture for DevOps, leaders need to build trust with their teams, empower them with autonomy and rely on strong communication to ensure that everyone understands their role and overall objectives.

    Summary

    The latest “Accelerate: State of DevOps” report has a simple, top-line message: Higher software delivery performance enables better business outcomes. Achieving this requires taking a development-wide approach to DevOps, incorporating the database seamlessly into DevOps flows, communicating and collaborating between teams, sharing technical practices, and building a strong, open culture across organizations. Only then can businesses become high or elite performers, with an agile infrastructure that underpins their success.

    — Matt Hilbert

  • Shouldn’t DBAs Be Part of the DevOps Inner Circle?

    Shouldn’t DBAs Be Part of the DevOps Inner Circle?

    DevOps is changing the culture of IT.  This movement is bringing application development and operations teams to the same table and the same time, working to understand and overcome the challenges facing the entire application delivery process—not just their piece. Development and operations teams are forging new bonds and alliances in the spirit of accelerating software release cycles.

    However, while quickness to market is paramount to success, businesses can’t forget about the internal implications of accelerated release cycles. With that in mind, there is one conspicuously empty seat within the “DevOps Inner Circle,” and that’s the database administrator’s (DBA).

    DBAs hold the keys to one of businesses’ most important assets: the database. But database change innovation has not kept pace with agile development methodologies and DevOps tooling that enable continuous delivery of software updates and changes. The vast majority of enterprises are still updating databases the same way they have been for the better part of the last three decades: through manual script review, validation, execution and individual heroics. As a result, application release cycles ultimately slow down at the last mile—the database.

    It’s time for a better approach. It’s time databases and their stewards, DBAs, be part of the DevOps Inner Circle. Including DBAs can create better, more efficient teams; save time and money; and even help companies create better solutions for their customers’ needs or their own.

    Shattering Silos

    One of the founding principles behind DevOps is the breakdown of silos between development and operations teams. Product managers can easily translate customer needs and communicate that feedback to developers in a way they understand—using epics and themes, for example. This helps bring everyone on the same page so that application updates or changes can be quickly coded, tested and delivered. But before application updates can be released to users safely and effectively, they have to be translated into the changes that will update the database. This is where the DevOps breakdown occurs. DBAs often have to rely on manually intensive, timeworn, error-prone processes and tools to make sure the right changes are made to the database.

    By inviting DBAs into the DevOps Inner Circle, database updates that go against business policy and will ultimately lock the database or “break the build” (think inserting a new column with a default value when the row count exceeds a million) can be found faster and addressed by the development team well before that change reaches the database. Having DBAs involved in release cycles sooner can help ensure developers are coding with corporate compliance in mind, ultimately making changes to the database smoother while adhering to guidelines that DBAs have typically policed.

    There was a time when databases needed protection from changes made by creative problem-solving in development, but DBAs simply cannot stand against the deluge of changes necessitated by accelerated cycles. Inviting DBAs into the DevOps Inner Circle enables teams to consider the implications of new changes and updates across the entire technology stack. Breaking down silos and making teams more aware of catchpoints in the last mile of the release process lets them anticipate and avoid database issues—which means faster releases and fewer headaches for everyone.

    Saving Time (and Time is Money)

    Faster release cycles are good for business, and not just because they get applications in front of end users sooner. Continuous delivery cycles are designed to cut costs associated with application development considerably by saving man-hours. Still, while development timelines keep getting shorter, the workload and time to deploy database changes gets longer.

    Enterprises can have multiple database configurations, so each change needs to be checked for compatibility. As the complexity of data repositories grows, so too does the time it takes to review and approve changes. In a 2015 IOUG survey on database manageability, 41 percent of respondents said changes took a week or more to approve. With multiple change requests coming to DBAs each month, each week or maybe even daily, it’s clear that sheer manpower can’t keep up with continuous delivery forever. Relief for DBAs is nonexistent without automation.

    Remember our earlier example, where we ran the risk of locking the database by inserting a new column with a default value when the row count exceeded a million?  Database automation software protects against outcomes like this by forecasting the impact application changes will have on the database. DBAs can use forecast reports generated by automation software to pinpoint any issues without expensive hours or days of searching.

    As part of a DevOps team, the DBA equipped with automation can do more than save time and headaches for themselves. Protecting against errors that could break the build saves teams from spending time to find and fix the source of a breakdown, and also saves a business from losing revenue while the fix is made. If so much time, money and potential frustration ride on DBAs’ ability to implement changes at the speed of continuous delivery, why are they still on the outside of the DevOps Inner Circle?

    Freeing Up Time for Innovation

    Flashes of inspiration rarely come about when you’re just trying to keep your head above water, and for so many DBAs, that’s the reality of work. Today, enterprises fight a war of attrition with the database, throwing manpower at the final hurdle of the release cycle. In addition to the attractive benefit of saving money on man-hours, bringing DBAs into the DevOps inner circle and automating some of their tasks also creates time and space for innovation.

    The relational database is nearly a 50-year-old innovation, one that changed the way people and computers interface through data. After so much time, and with so much on the horizon, the next innovation seems imminent or perhaps even overdue. The advent of quantum computing may bring with it a fundamental change to the way databases interact with one another. It’s certain to bring a shift in the way data is stored and retrieved, though at this point experts expect only the most complex business processes will benefit from quantum computing. So, while future tech remains out of reach, DBAs will be vital in the effort to innovate faster, more streamlined ways for dealing with zeros and ones.

    The first step to innovation, then, is addressing the needs of DBAs alongside those of customers and developers. Breaking down walls and increasing efficiencies between existing DevOps teams and DBAs will give teams more opportunity to create novel solutions to both their users’ problems and their own. Add another seat to the Inner Circle, save time and money, get quality applications to market even faster, and create innovative solutions to the problems of today and tomorrow.

    About the Author

    Derek Hutson HeadshotDerek Hutson joined Datical as CEO in 2015, bringing more than 15 years of leadership experience in enterprise software. Before joining Datical, he led multiple companies through extended periods of rapid growth and also held several executive roles at IBM. There he served on the leadership team for IBM’s acquisition of Telelogic, and was responsible for due diligence and integration of the acquired sales functions globally. He currently serves on the board of Hart InterCivic and Charity Dynamics. Other board or advisor positions have included CoreTrace (acquired by Lumension), Innography, StreamStep (acquired by BMC) and Phurnace (acquired by BMC).