— ROELBOB
Category: Blogs
-
Lean Security: How Better Development Can Protect Your Business
If companies are to reach their strategic goals—reducing time to market, boosting sales, improving product market fit and brand image, and cutting cybersecurity costs—then it’s time for a new outlook on software security.
Today’s business leaders must learn to see security for what it is: A differentiating factor. Companies with reputations for secure developmental processes and infrastructure will rise above those known for data breaches.
Security as a Business Strategy
The road to a more secure company—and realizing the perks associated with it—begins with lean security. This is an approach to information security similar to the Toyota principles of management and production that calls for environmentally aware engineering, simplified coding, automation of security checks and constant incorporation of feedback.
Although some business leaders believe organizational security is inherently expensive, lean security doesn’t mean massive costs. Much like DevOps, lean security champions process improvements and cultural changes above purchasing new tools.
Lean security does, however, require engineers to keep its principles in mind throughout the development process. In this way, security is much like sales: A sales representatives’ best strategy is to consider his goals from the beginning. Just as the salesperson isn’t trying to make any sale, the engineer isn’t trying to create just any product. For the salesperson, this means tweaking the sales process—perhaps by vetting leads before engaging them—to make profitable sales. For the software engineer, it means structuring the development process through lean security to create secure products.
Done right, lean security results in a self-defending, simple system created by people who are aware of a product’s real-world security risks and the company’s business goals. Lean security recognizes your company’s data is always at risk, no matter the size of the company or team, and it works to create value through systemic protection. Here’s how to start:
Improve Environmental Awareness
Lean security approaches risk management as a team effort. Think about a hospital. Everybody working there — from the cooks to the nurses to the executives — has a responsibility to take care of patients.
Software firms must function the same way: Everyone from engineers to salespeople to the CIO must keep data privacy, software security and business needs in mind throughout product development and a product’s life cycle.
Environmental awareness means addressing security with the same sensibilities that you would your own life. Just as you don’t hand your credit card to random people, neither should you hand the keys to a software engineer who doesn’t know the risks of insecure development.
More secure coding begins with a change of mindset. Once team members understand a product’s security risks, they start thinking about how to create a more secure environment. If a developer is creating a mobile app for a car, for instance, then they must recognize the app will be used in conjunction with a device that carries human life.
Simplify Engineering
Complexity is the enemy of security. Simplification, however, cannot just be limited to lines of code: It means organizing your teams—and thus, your entire system—in a way that improves time to market and tightens feedback loops.
DevOps is a great way that business leaders can improve security through sensible reorganization. While the integration of development and operations teams has been lauded as a security boosting measure by software leaders at companies including Dell, CA Technologies recently studied DevOps’ business benefits. It found companies that have implemented DevOps techniques are 2.5 times more likely to improve customer retention, twice as likely to grow their revenues and 3.4 times more likely to improve their market share.
In short, keep teams compact and agile and design with the end in mind. The more lines of code you add, the more complex teams you create and the more times you reinvent the wheel—and, ultimately, the less secure and less profitable your company’s products become.
Automate or Die
Automation is an essential component of lean security, and DevOps engineers are no strangers to automation. Threats are constantly increasing, and no engineer—or team of engineers—can discover and remedy them all manually.
Look to Netflix for a great example of automated security. Once upon a time, Netflix addressed malware alerts by manually creating a help desk ticket and assigning an engineer to investigate the problem. The time from alert generation to eventual resolution, according to Netflix, often spanned more than a week. As Netflix’s security challenges increased in both diversity and complexity, it found itself spending exponentially more time and money combating these threats.
Then, Robert Fry, Netflix’s senior information security architect, came up with a brilliant solution: He engineered FIDO, an orchestration layer that automatically evaluates, assesses and responds to security threats. FIDO detects threats, then scores them based on the attack’s intended target and other factors. FIDO then attempts to mitigate the event by closing a network port, ending a VPN session or disabling the account.
Tools such as Netflix’s FIDO automatically detect risks in seconds, while it could take a human months to stumble across them. Think of security automation like a home security system: ongoing, vigilant security without the need for continued manpower.
Measure Everything
Although its origins are disputed, many credit English mathematician Karl Pearson with quipping, “That which is measured improves. That which is measured and reported improves exponentially.”
Development teams should take Pearson’s wisdom to heart, and measurement must begin with identification of goals. Is your goal to reduce application downtime or decrease event response time? Then it’s essential to measure mean time to repair. Do you want to reduce code defect density? Begin by measuring the number of issues per thousand (or million) lines of code.
Once you’ve identified goals, use software to track progress over time. Etsy’s engineers built a tool they call StatsD. The software helps Etsy monitor everything from login failures to coffee availability. The data is then displayed in handy graphs to help the team make sense of the information.
Once you’ve taken baseline measurements, be sure to review your metrics on a regular basis. For instance, you might realize an application’s deployment times aren’t where you’d like them to be. Keep measuring results—and keep trying new things—to nudge those values into better territory. Just remember: You cannot improve that which you do not measure.
The Business of Lean Security
The value of a secure, simple system cannot be understated: It means getting products to market faster; it means better public perception and press; it means more satisfied customers who feel secure using your product; and it means fewer disruptions in engineers’ work schedules. At its core, lean security lowers a company’s overall exposure to risk and reduces its expenses.
Companies are already staking their futures—and consumers’ safety—on code that facilitates transactions, drives vehicles and manages power plants. The Internet of Things will soon run everything from toasters to jet engines, and security will grow even more important. Protect your company from within through lean security.
-
DevOps: Shift Left to Reduce Failure
DevOps is all about increasing the feedback loop from idea to customer and back again. Tremendous gains have been achieved in automation and tools to drastically reduce the amount of manual labor involved in progressing down the release pipeline. But mission-critical applications in a highly competitive market cannot afford to go down. Complex applications, environments and infrastructure increase risk, which can cause failure.
The failure rate of production deployments is still too large. Studies show that the average cost of a production outage is roughly $5,000 per minute, while the average cost of a critical application failure per hour is $500,000 to $1 million.
Why so much failure? All too often we’re still seeing a lack of consistency across the release pipeline. Many times the provisioning and deployment processes are very different at the opposite ends of the pipeline. But by shifting left the operations concerns, you can achieve much more success in production.
Understanding How to Shift Left
The term “shift left” refers to a practice in software development in which teams focus on quality, work on problem prevention instead of detection, and begin testing earlier than ever before. The goal is to increase quality, shorten long test cycles and reduce the possibility of unpleasant surprises at the end of the development cycle—or, worse, in production.
Shifting left requires two key DevOps practices: continuous testing and continuous deployment. Continuous testing involves automating tests and running those tests as early and often as possible, along with service virtualization to mimic unavailable systems. Continuous deployment automates the provisioning and deployment of new builds, enabling continuous testing to happen quickly and efficiently.
The first and most obvious way to shift left operations is to work side by side with development in creating the deployment and testing processes. Failures observed in production often are not seen earlier in the life cycle. Many times these failures can be attributed directly to differences in deployment procedures. Development may create its own deployment procedures that are very different from those used by operations for production. Sometimes production procedures are much more manual and may even use different tooling. Operations and development need to take ownership in building standard deployment procedures. The deployment process is then practiced potentially hundreds of times in test environments before reaching production. You then have much more confidence that the production deployment will be successful.
Another way to reduce the failure rate is to make all environments in the pipeline look as much like production as possible. This is almost impossible using traditional provisioning processes. Production environments usually are much heavier and detailed than are needed by development. But by using cloud and pattern capabilities, you can get much closer.
The ability to use patterns to define consistent environments eliminates the failures that occur simply by configuration inconsistencies. This again requires operations and development to work together in creating the provisioning process so developers create their test environments the same way they are created in production. Using IBM UrbanCode as an example, this can be done by connecting UrbanCode Deploy to an automated self-service provisioning system such as OpenStack HEAT or IBM Cloud Orchestrator connected to OpenStack.
These standard patterns can be used to create environments for all stages of the pipeline. And by using cloud capabilities you can deprovision these environments as fast as you can create them. This also allows the enterprise to have confidence in their environment patterns. They are tested on a daily basis and, therefore, operations can feel confident that testing is done on production-like environments.
No one wants to experience production failures—and it is in everyone’s best interest to do all they can to prevent them. At the root of DevOps is the removal of those silos between development and operations. By applying these shift left operations principles you are well on your way to reducing your failure rate.
The ingredients to make shift left are real and available: automated testing and deployment tools, service virtualization technology, agile practices and cloud environments. The main obstacle that remains is the mindset of delivery organizations. Learn more from the newly updated Application Deployment and Release for Dummies ebook to find out more.
About the author / Dr. Darrell R. Schrag
Dr. Darrell R. Schrag (@devopsdrs) is an experienced software lifecycle consultant specializing in the IBM cloud solution, helping customers establish their cloud strategy across the enterprise. He’s focused on implementing continuous delivery of software applications through the delivery pipeline incorporating cloud and provisioning solutions. He’s experienced in helping clients transition to “born on the cloud” applications utilizing microservices to take advantage of a flexible and nimble cloud platform.
-
What is DevOps’ Business Value?
Intro: DevOps is About Business Performance
It is important understand that DevOps philosophically has a system-level view of a business’ performance. You don’t hear this much—it’s more difficult to write about than, say, the latest tech, but it’s always there under the covers.
And, granted, the frequent focus for DevOps grassroots discussions is about tooling, about tech, about software pipeline—the “IT” in “ITSM,” so to speak. That’s usually because all of the people in those discussions have Dev or Ops or related functions as a job and already know all-too-well the business runs on the work they’re responsible for. You’ll see this at any DevOps Days you ever go to.As should be clear to everyone: All businesses today are in the IT business. This is regardless of what business they think they’re already in. DevOps is born from a disruptive crucible where the world of business morphed from being “do X” to “do X via computing” (i.e., a bank is no longer a bank, it’s now an IT shop with a banking license).
This is where newcomers can mistake the forest for the trees: The heart of the DevOps movement is unrelenting focus on the “SM” in “ITSM.”
Yes, automation gets involved. And rightfully gets a lot of discussion because it is such a huge force multiplier. However, other DevOps 101 beginner’s basics such as, “Have the developer carry a pager,” clearly are about bigger issues than which configuration automation tool you use.
The numbers I will be reviewing here aren’t new. They aren’t mine. I often get the feeling when discussing these numbers with people that they aren’t getting what these numbers mean, in business context. How huge a powerhouse DevOps drives. So I’m going to try to explain a bit and try to put it all in context.
So What is the Value to Business from DevOps?
I wanted to set the stage a little so that we can properly discuss DevOps return on investment (ROI) in context of every business is in the IT business. The fundamentals of business performance are pretty crisp. We all get lost in new shiny and new toys.
New shiny toys are the best.
Yet DevOps as a philosophy is bigger.
To be blunt, practicing DevOps as a philosophy gets you the latest, actionable answers to fundamental business questions:
- How do we improve product delivery to our customers?
- How do we change product more quickly to better satisfy our customers?
- How do we recover after failing our customers?
- How do we get paid faster by our customers?
All of these directly impact sales, revenue growth, profitability, product line and business viability, customer satisfaction—all of which h
ave direct correlation back to investment in the business.I’m going to use Puppetlabs 2015 industry survey and work from remarkable folks such as Gene Kim, Jez Humble (Continuous Delivery and Lean Enterprise), Nicole Forsgren, plus Mary and Tom Poppendieck. I’m pulling from these few for this ultra-specific focus on ROI, but there’s a lot of excellent other sources and material for potential ROI explanations out there.
So let’s answer our questions.
1. How Do We Improve Product Delivery to our Customers?
The fact is, our product isn’t doing anything valuable until our customers are using it. It isn’t making us any money. It isn’t making them any happier. It only does and can do that when we’ve actually produced it, put it into production, and gotten it into the customers’ hands.
And DevOps-practicing companies do this at the rate that is unparalleled. Some of this is a result of that strong DevOps tendency to use automation as a multiplier, and some of it is a holistic view toward facilitating hand-offs and team collaboration. All of it, however, is DevOps.
Studies show these DevOps companies have an almost supernatural advantage of 30x the deploy frequency of their non-DevOps peers. Put it another way, if you’re not embracing DevOps against someone who is, they can on average do what you do but do it 30 times faster than you. 30x.
That is, if your business is in a footrace, you’re up against The Flash. Good luck with that.
2. How Do We Change Product More Quickly to Better Satisfy Our Customers?
The best product features are the ones the customers actually use. So, you focus on features they like (or want to have) based on their usage and fit. This frequently involves a little A/B testing on a subgroup and increasing it to larger groups until it’s everyone. It involves speed of offering as a way of measuring improvements.Amazingly, as it turns out, companies that adopt current DevOps practices have radically positive advantages here on changing product.
I’m going to cite the simplest metric I’ve heard of. There’s a truly nifty test devised by Mary Poppendieck. It’s super simple. It gets you instant insight into the velocity of your product development pipeline at your company.Answer this basic question: How long does it take your organization to put a single, new line of code into production?
For many companies I know of, the answer will be in units of months. Months. For DevOps-practicing companies, the answer often is as long as a couple of weeks to as short as an hour. Months versus hours?
That is a devastating difference in competitive advantage. If your competitor takes months to deliver something to its customer that you can deliver in an hour … But wait, what if it’s your company that takes the months and the competitor that takes the hour …
They’ve even quantified the difference. The DevOps-enhanced companies have a 200x faster lead time over their non-practicing peers.
So, not only do DevOps companies iterate in production far far more quickly, they can move to production even faster than that.
3. How Do We Recover After Failing Our Customers?
There’s always some joker in a crowd who will say, “Well, don’t fail the customer in the first place.” I’ve encountered this point-of-view in executives who haven’t ever programmed or really understood how computers compute. Those are the 100 percent-uptime folk and they don’t get it.
That’s not real world. You will fail; your front-end systems will fail. Some failures will even be catastrophic. Focusing on the meantime between failure (MTBF) is an old hardware-oriented way of thinking. It’s out of date and dangerous in this new age. It’s how we measured the durability of car parts or hard disks or plumbing. (A handy ACM article from Facebook on Fail at Scale is always a good read.)
Embracing the notion that you will fail—and that you should even practice failing—is a key mental step to focusing on meantime to recovery (MTTR). MTTR is how quickly are you back up and running after that failure that will occur has occurred.
And DevOps-savvy companies are amazing here as well, averaging a 168x reduction. That is, on average they are 168 times faster at recovery than their non-DevOpsian peers.
And what about those failures caused by changes, whether they be configuration or application related? The so-called “change success rate”?
The number there for DevOps enabled is astounding, too. There is a 60x improvement in change success rates. So not only do these companies make changes faster, more frequently, with better recovery, they do so with a phenomenally better change record.
4. How Do We Get Paid Faster by Our Customers?
This last one is subtle because it’s the snowball-rolling-downhill accumulation of all the previous DevOps factors at play. Basically, the business life cycle at work.
(I’m not going into “paid” versus “recognizing revenue,” as recognizing revenue means more than getting paid and there are a bunch of accounting rules around that in the software industry. Nor am I going into the difference between purchased versus SaaS/CapEx versus OpEx. It’s an important topic to understand around revenue models but outside the purview of this.)It basically is gated like this, reverse-chronological order of importance for an ongoing business:
- You get money after you bill.
- You bill after you deliver benefit.
- You deliver benefit after you ship.
- You ship after you’ve made the product.
- You get money to make product.
DevOps-embracing companies iterate thru steps 1 thru 4 of this cycle day in and day out, as we’ve reviewed above, at a pace that will ruthlessly crush all competitive laggards. I mean, flatten.
The rates of change allowed by DevOps practices support such rapid acceleration away from the old norm. It’s startling.
So, I suppose I’m saying not only can your company adopt DevOps changes, I’m telling you that you must. Even better, adopt and you’ll probably thrive.
And since I’m talking about modern business, a Deming tax:
P.S.: What about those corporate tales of “DevOps failures” I hear about?
Over the years I hear the odd pundit or company executive dismiss DevOps as a unicorn thing or something that older, established organizations with years of built-in (anti) patterns of behavior can’t adopt. Or they claim they are in a sui generis industry like ahem banking where, well, they’re going to claim they can’t change because … you know, stuff.
At a big analyst conference a couple of years ago I heard how there are many examples of “DevOps failures” at clients’ firms. In every case I heard about, however, the failure turned out to be unrelated to anything DevOps. They turned out to be related directly to poor analyst advice, or woeful execution, or simple gross misunderstanding.
I’m not going to name names or shame them. I’m just going to say it’s incorrect in every case I’ve seen to cite “DevOps” as “the failure.” When you hear anyone talk about a DevOps failure, dig into it. It won’t be there in the end; it’ll be something else that actually derailed the effort.
Most businesses are still command-and-control model orgs where you find a range of managers, mostly process-oriented, who are micro- to macromanagers. And you have fiefdoms. And the customer is rarely in the room. Culture plays a big role here.If your culture is healthy, you optimize for your people, for your customers and for achieving good customer results with the teams of people who are organized to do that. Curiously, if you do this, based on surveys, your company gets great and also becomes a great place to work!Now, that’s DevOps thinking at work.
-
Embedding Predictive Analytics
For an enterprise with a deep DevOps maturity model, the initial “wow” factor of automation has worn off. At this point, the sexier concepts involve proving continuous improvement. As an example, our first baseline of metrics might have taken the pre-DevOps timings of a typical build and deployment of a given application. We can measure this time against our post-implementation of DevOps automation and show a significant savings in time. This is opportunity time, meaning an employee must be ready to turn around the decision cycle faster to take advantage of the time saved (not be playing solitaire, for example). But once DevOps automation is old hat, the difference between our initial post-DevOps typical build and deployment time and our current one still using DevOps may not be all that significant. But is there more hidden there than meets the eye?
Establishing Context & Baselines
At this point context may be a more meaningful measurement. As an example, if my first version of DevOps automation for a deployment contained 35 steps and my current one contains 160 steps it would stand to reason I am “doing more” now in my deployments than I used to do. These additional steps may involve executing more tests than I originally thought to do. They may be interacting with monitoring systems (turning them on and off as appropriate), which reduces “noise” in the alerts my monitoring teams have to respond to. In short, over time, my deployment processes may become “longer” because I am increasing the capabilities or the sophistication of what I do during these deployments. While from a pure timing perspective this may make them longer, it also makes them more powerful and, so, is worth measuring. To know the delta, however, I must have retained older versions of automation to compare against, not just throw away the old every time I update with new steps.
So let us assume the context of my automation is now fairly stable. I do the same things, more or less, the same way every time I deploy my application. At this point, measuring the functionality of what I do becomes more meaningful. As an example, my deployment automation may complete four main functional tasks to complete, I deploy the code (the app), I deploy the dataset, I deploy the config files and, finally, I run a series of tests on the above. While I may not ever incorporate too many new kinds of things to do in this automation, understanding how long each function takes to complete is relevant to proving continuous improvement. For instance, if my code takes roughly the same amount of time each deployment, as does my config files, but all of the sudden my dataset begins to grow in each deployment (from normal usage), my overall deployments may be getting slower. This is not due to hardware inefficiency or network inefficiency; it is simply due to a larger dataset that keeps growing.
If on the other hand, it is my test set that is getting progressively longer to execute, that is an indicator of test script inefficiency (given same code, same config and same dataset). If the number of tests being executed has not increased, but the length of time it takes to run them has, there likely is inefficiency in “how” the testing is being constructed or executed. Understanding the context for what is going on in a deployment or similar kinds of automation must occur before introducing predictive analytics can occur.
Historical Trends Extended
Predictively evaluating automation is nothing more than taking the data from a historical perspective and extending the trend lines into the immediate future. If for example, my deployment automation is getting four seconds longer each month for the last 12 months, I can reasonably infer the trend will continue, and I can predict the length of time it takes to execute a deployment in the coming months. Understanding “why” this occurs requires a contextual investigation, but predicting its occurrence does not. This may seem inconsequential as we talk about the length of deployment processes of a given app, particularly if my portfolio of apps is small. But if the trend holds true for a portfolio of thousands of applications, the impacts become more consequential. Or if the phenomenon is directly connected to a particular cloud provider and not seen elsewhere, again, the consequences become meaningful.
Where having “ground-up” predictive analytics (build and/or deployment automation) becomes much more useful is in the construction of release events. Often, release managers are asked to supply an estimated time duration for a given event to factor into the construction of a new release. If a vendor were to integrate more predictive analytical tooling into the release orchestration (RO) toolsets, “random” or “best guess” timing estimates could be replaced by extending historical data into the timing estimation fields. The RO tooling could provide a default timing estimation based on examining historical trends for a period defined by the user. Estimation defaults could always be replaced by a user (assuming discreet knowledge of other impacting events), but at least the new overall lengths of releases then could become significantly more accurate.
This has special significance for manual tasks. Let’s say our database admin team traditionally takes 40 minutes to perform a given release event update, but now is running significantly longer than that. Having the RO software aware of the historical trends could alert the release manager, as well as the database team manager, during the event to check out why this is occurring. For manual tasks in particular it may be hard for anyone besides a predictive analytics module to determine what timings “should” be, as compared with what they are in a given release.
Impacts to the Bottom Line
The real questions are, Why should a customer care about this kind of feature set? Why should we be looking at the vendors of our tooling to incorporate forward-thinking architectures such as predictive analytics into the tooling they sell us? Because the efficiency it can bring to our DevOps set of services lowers the costs of administration and increases the speed at which we can deliver innovation in general. Over time, trending combined with context can statistically prove continuous improvement of our DevOps services, which is the promise we made to the executive teams that sponsored and made the investment in DevOps to begin with.
Metrics analysis can occur (usually with a dedicated investment on our part) without a predictive view of the impacts. But this changes the perspective of our analysis to hindsight thinking instead of forward thinking. In essence, we then are simply “reacting” to what has occurred instead of tailoring our environment to alter the picture of future predicted events to be better than the software predicts it to be. We start tweaking the variables to instantly impact future predictions, instead of only examining them in a post mortem.
This perspective is yet another cultural catalyst for change your business needs. When I talk about changing how an organization “thinks” about innovation, this is one of the key objectives. We take an enterprise from knowing little about the costs of innovation, to having an extensive knowledge, to having a forward view of what is—or what should be, possible. All of this is possible despite the loss of waterfall controls and in the midst of an agile construction model. All it takes is the integration of forward-thinking architectures into the products we select. If our vendor does not offer them, we begin to enlighten our vendors on the value they would bring to us in cost reduction and to them in market capture. If you need help in these areas, feel free to contact me, I am looking at the moment 🙂 .
To continue the conversation, feel free to contact me.
-
Backup and Recovery of Next Gen Apps in MongoDB
This blog is the first in a two-part series about MongoDB backup and recovery. In this first part, I will discuss the motivations for protecting data that resides on MongoDB and the existing backup and recovery solutions. In the second part, I will discuss how to solve this problem.
In the era of big data, enterprise applications create a large volume of data that may be structured, semi-structured or unstructured in nature. In addition, application development cycles are much shorter and application availability is a critical requirement. Given these application requirements, enterprises are forced to look beyond traditional relational databases to onboard the next-generation, cloud-native Platform 3 applications—infrastructure-as-a-service (IaaS) or platform-as-a-service (PaaS) applications. NoSQL databases such as MongoDB are now being adopted and evaluated by enterprises for these next-generation applications, which include ecommerce and content management. The benefits of MongoDB are many, including dynamic schema, easy scaling through auto-sharding, tunable consistency for reads and built-in replication.
MongoDB database also has native replication capabilities that satisfy availability requirements. However, data protection requirements for scalable point-in-time backup and recovery must be addressed. For robust data protection, enterprises need both backup and replication; Without point-in-time backups, organizations are at substantial risk of losing data due to human error, logical corruption and other operational failures. Traditional backup solutions were built to address the requirements of structured Platform 2 applications on relational databases that used shared storage and had the ACID (atomicity, consistency, isolation and durability) transaction model. Unfortunately, they fall short of addressing point-in-time backup requirements of Platform 3 applications and distributed databases (local storage, eventual consistency and the elastic nature of infrastructure). There are a few alternate script-based solutions such as Strata that enterprises are using to fill the data protection gap, but these solutions are sub-optimal at best.
Manual Scripted Solutions
Manual scripted solutions (MSS) leverage native MongoDB snapshot utility and scripts to transfer data to secondary storage. The scripts (via mongodump) are customized for each MongoDB cluster and require significant operational effort to scale or adapt to any topology changes (such as addition or removal of nodes to your MongoDB database). Further, these scripts are not resilient to failure scenarios such as failure of a node (primary or secondary) or intermittent network issues. Finally, recovery (the paramount value of “backup”) is a manual process, hence, time-consuming (resulting in very high application downtime) and contains data loss risk due to any bugs in the scripts. Overall, these solutions work when the MongoDB environment is small and some data loss may be permitted in the application. Some of the key issues that these solutions face are:
- Lack of enterprise backup solution for sharded configurations
- Database needs to be offline when the snapshots are taken
- Both backup and recovery fail under node failure and other infrastructure failures
- Recovery process is manual and requires verifications, which increases the recovery time
- Recovery at collection-level requires manual recovery that is time consuming
- Recovery to unlike topologies (sharded → unsharded) for test/dev refresh is not available
Most enterprises use these scripted methods as a temporary quick-fix solution. It is like driving your car with flat tires: You can keep going; however, you can’t go at the speed you want to go, nor are you risk-free from disasters.
MongoDB Paid Backup and Recovery
MongoDB (the company) provides a couple of ways to back up MongoDB databases. Enterprises may choose from either a managed backup offering (MMS) that runs in public cloud or, if they are paid MongoDB customers, they may deploy the backup service on-premise. In addition to being exorbitantly costly, the managed backup service stores customers’ data in public cloud. Backup data transfer over WAN may not work for customers who deploy MongoDB on-premise and for the customers who need to keep their sensitive data in-house. Further, there are significant data limitations per shard to use this service.
Using the MongoDB on-premise backup service is possible but is overly complex to deploy and operationalize (this deployment diagram speaks for itself!). Enterprises need to deploy eight servers, additional databases (with additional licensing) and about 6 to 9 times storage capacity (of the database that is backed up) for enabling on-premise backups.
Overall, on-premise backup service is a theoretical solution that brings with it significant CAPEX and OPEX investments:
- Complexity of deploying multiple databases
- Cost of additional infrastructure (servers and storage)
- Cost of licensing additional MongoDB nodes
- Risk of failed backups when nodes fail (secondary from which backup is taken)
- Siloed backup infrastructure for only MongoDB database
Look for Part two of this discussion to come shortly.
-
The Persistence of Enterprise Tech Failure Rates
During a recent dinner conversion with a CIO friend, I was surprised to learn that his organization was having challenges delivering the applications and IT services on a timely enough manner that the business needs. The business is taking too long to approve the budget for major tech projects to start, then develop and deploy, and the CIO expressed frustration how it is always behind the demand curve and following where it should likely be in the market. His level of frustration as he described the situation was painted all over his brow.
Hearing this was surprising to me; after all, we’ve been writing about how such problems delivering with speed and agility are challenges that are designed to be solved by DevOps, agile development methods, microservices and all of the things we focus on here at DevOps.com and Container Journal.
The discussion we had got me wondering how common this situation remains. How many companies are still struggling with their ability to retool themselves into more agile developers? In the late ’90s it was common for tech projects to fail. I recall surveys I reported on that found 60 percent, or higher, failure rates of large projects. But back then software rollouts were truly big bangs. There were years of development time and hundreds of physical servers and thousands of endpoints that had to all be managed. There existed layers and layers of committees and bureaucracies between the developers, IT admins and business leadership. IT is supposed to have moved beyond all of that.
Unfortunately, however, if recent surveys are any indication, it still seems very common.
A survey conducted last year by portfolio management provider Innotas found that more than half of tech projects are not well aligned with the goals of the business. The survey, the annual “Project and Portfolio Management Landscape survey,” also found that more than 61 percent of IT professionals questioned did not think that they have adequate resources to manage demand for tech projects in their organization.
Part of the challenge may be that there’s inadequate project management in place at many organizations. The survey found that 38.7 percent of respondents didn’t have a project management office at their organization; however, 16 percent are building such offices. Additionally, 25 percent of those surveyed said they don’t have a formal methodology to prioritize projects.
Other areas of concern ranged from project prioritization (26 percent) and alignment (13.8 percent). Attaining benefits and finding the appropriate budget were also challenges at 12 percent or fewer of organizations surveyed.
This is concerning, as more enterprises are doing everything they can to move more swiftly (or, at least say that they are). As we covered in “Digital Transformation Strains Traditional IT,” according to a recent report published by Salesforce’s “2016 State of IT,” IT leaders are focusing on mobile apps, migration to the cloud, building adequate cybersecurity/incident response, customer-facing digital apps and productivity apps.
The Salesforce report found that 68 percent of respondents are increasing their tech budgets on mobile apps, cloud and security. The majority (63 percent) of respondents also said they plan to increase their budgets allocated to cybersecurity, while 62 percent will spend more on employee productivity apps.
Some of the things that has come up, repeatedly, in my conversations with other CIOs is that those companies that are struggling have poor business-to-IT relationships and ongoing feedback. This creates a lot of distrust and poorly aligned priorities.
We covered a lot of this earlier in a number of posts, which are available here:
Five often expressed obstacles to enterprise IT innovation
Enterprises still struggle to leverage IT as a tool to innovate
