devopsy.kinsta.cloud

Tag: data

Speed and Security Can Coexist in Mainframe DevOps

DevOps teams face a constant tug-of-war in their daily work, balancing the need for speedy rollouts of high-performing (fast, reliable) applications that are secure also. If the team moves too quickly, an overlooked security vulnerability may make its way into production. If the team is not nimble enough to identify those security gaps, it can slow down the entire development process, hampering organizational agility.

The need to strike this critical balance has led to the rise of DevSecOps, or the integration of security into DevOps best practices through automated techniques. Much has been written about the importance of bringing the mainframe fully into the DevOps fold, evolving the platform to keep pace with the speed of modern software development. But what about the security of mainframe code and mainframe data specifically—incorporating the mainframe into the third leg of the DevSecOps stool?

A recent survey found that 72 percent of customer-facing applications are completely or very dependent on mainframe processing, which means there’s a lot of personal, sensitive information flowing through mainframe code and into mainframe data repositories. The mainframe platform itself is the most secure system on the planet, due in large part to its unique function and design. IBM’s new pervasive encryption is a groundbreaking development for enhancing data security on the mainframe even further, enabling data at rest (in traditional VSAM files and databases) as well as in transit to be encrypted at a pace of 12 billion transactions per day, all with no impact on system throughput speeds.

Bulletproof Mainframe Security

Still, highly personal and sensitive data can be compromised if steps aren’t implemented to ensure mainframe code and data are as inherently bulletproof as the supporting platform. What steps can help mainframe users achieve the conflicting goals of fast mainframe development combined with superior security?

Privatized Test Data

Speed is critical in all aspects of DevOps, including application testing. Mainframe transactions are the heartbeat of modern applications, making testing critical to ensure these components are working properly. Many organizations prefer using live, mainframe-based production data in testing, as this provides the most realistic view of how an application will work “in the wild.” However, in the rush to pull testing data, many organizations have fallen into sloppy practices including failing to disguise this data, which puts sensitive, personal data at risk of exposure. Companies needing to comply with the EU’s Global Data Protection Regulation (GDPR) face another risk—non-compliance—since they may be using customer data for a purpose (application testing) beyond that for which it was originally procured.

Mainframe users can continue to use real production data in application testing, but they need to take special steps such as using pseudonism, which entails randomly scrambling certain values in a database (for instance, names) so that real customer names are disassociated with other personally identifiable information such as addresses or credit card information. Making it reasonably difficult to associate individual customers with their personally identifiable information is one way to achieve a higher level of information security for production data used in testing. Creating privatized application testing data sets must be a repeatable, automated process that allows testers to quickly get access to the test data they need to expedite testing processes.

Credentialed User Auditing, User Role Management

The DevOps concept entails greater frequency of code changes, which naturally leads to more testing and more developers and testers accessing data and applications. Unfortunately, the involvement of more team members, all working at a faster pace, can sometimes lead to carelessness and inadvertent and unintended security risks. There are techniques available now that allow organizations to see exactly what credentialed users did and what data was accessed. Some may view this as bordering on “Big Brother” type of behavior, but the fact is, with so much sensitive data continuing to reside on the mainframe and growing compliance requirements (GDPR, for example), such surveillance is vital to protecting both the enterprise as well as workers, the majority of whom have no ill intention.

However, user auditing comes with an important prerequisite, and that is better management of user roles. “Least privilege” is a long-held security best practice of restricting access for users only to those resources absolutely required for them to do their jobs. The challenge is when workers change roles, they often are not removed from their previous role definitions and privileged access points, and simply given new ones. Since they may no longer need their old credential sets, this violates the least privilege concept and represents a significant security threat that auditing may not be able to fix. The benefits of auditing—in terms of fostering faster testing and overall development without compromising security—depend on strong user role management.

Automated Rollouts/Rollback

Automated rollouts make the process of releasing code into production faster and more efficient. A dwindling pool of mainframe specialists means that in many cases, workloads spend significantly more time being queued up waiting for available resources, than they do actually being implemented. Removing the need for a developer to manually push new updates into production can be a major benefit in speeding up the mainframe cog. However, approval layer and audit capabilities must be built-in to ensure code security has been tested and verified and the source of any potential problems can be traced back quickly and accurately. Automated rollback functionality is also critical in the event that a security flaw does make its way into production, by enabling developers to immediately roll back to a previous version of the software until the flaw is fixed.

Much progress has been made in recent years when it comes to evolving the mainframe for DevOps, as several examples can attest. But given the sensitive personal data that is often running through, and residing in, mainframe code and data, the scope must expand to DevSecOps, and the steps described here represent a solid initial plan for getting there. Organizations that do this will enjoy the maximum benefits derived from a highly agile, secure mainframe environment delivering unparalleled scalability and reliability for its digital transformation initiatives.

— John Crossno

July 16, 2018
Portability Shmortability: Most Applications are Going Nowhere

I’m a huge fan of the idea of taking our super-duper applications and deploying them where it makes most sense. I always have been. Those of us in application development spent forever bemoaning the fact that our applications weren’t all that portable between operating systems. In fact, looking back through the history of computers, you find there has always been one roadblock or another to stop portability.

First there was byte order and instruction set, and we’ve largely overcome those. Internationalization wreaked havoc on us all for a while, and we overcame that. We’re still pretty mightily struggling with OS-dependent things such as windowing systems, but browser-based apps have largely overcome those also. Java takes a lot of heat for a lot of reasons, but it did improve portability of apps written in it. Maybe not “Write once, run anywhere,” as was often claimed—things such as file locations were still OS-specific—but still a lot closer than compilers that were deeply tied to the operating system they ran on. Virtualization and containers have enhanced portability in another dimension, letting the developer take the OS with the app.

We have overcome a ton of issues, and advancing technology has introduced a ton more. Now we have to either choose mobile targets or find a tool we like that can generate code for wildly different mobile systems from a single source. DevOps really isn’t helping with this process—though it will, eventually. By automating the build process, the target will be switchable at the targeting end, meaning our app—given a tool that can multi-target—can be built for whatever mobile platform the cool kids are playing with today, and the ones that are considered “cost of entry” can be generated in the same run.

So far, so good, right? We find problems, we overcome problems, DevOps helps us overcome them faster with a wider selection of outputs …

And then we hit the data.

Vendors in a variety of spaces would love to paint you a picture where it doesn’t matter. It does. The mass of your data, wherever it resides, is a pull like gravity to new application development. And it should be.

Some will try to explain to you that in the age of APIs, it doesn’t matter where the data actually resides: You can get it.

But it really does matter, and this is not Utopia. Even if you ignore the performance implications of data being latency-infested miles away from applications that need to use it non-stop, those APIs are only a viable option if the data can be secured and protected. By definition, an API is an advertisement that whatever data/service it fronts can be found right here. For applications that traditionally are data center apps, it might be possible to expose your customer file or other critical data via API and then lock down access to the API by IP …

But then we quit ignoring the latency problem. If every bit of data an app needs has to flow from San Francisco (or Tokyo, or Bejing, or London, or wherever) to Seattle before the calling app can do what it needs to and then has to be sent off again to Albuquerque (or Rostov, or Madrid, or wherever) the delays will, by definition, be more than they would be if the app sat with the data and had to transfer over a 1GB (or 10GB) connection between servers before sending it to users.

This means most data-heavy apps, no matter the level of agile or DevOps, aren’t likely to move. Vendors with a stake in the process will trot out the one or two customers they convinced to make such a move to tell you how great it is, but performance math and security sense tell you it is only great until things get busy or that exposed API is used by bad actors to vacuum user data.

Some applications have smaller datasets and can move the data with the app. Other applications can be architected not to care where the data is. A few don’t need a database at all. Move these applications to whatever platform makes sense. Make the CD for them retargetable. Most serious business applications rely too much on core data to fall into these categories. And thus, they aren’t moving, unless someone comes up with a viable way to move a ton of data between platforms that is safe and fast and effectively translates where necessary.

So, that’s what we need to see next in the march of portability. We currently have containers that can be run pretty much anywhere and languages that will work on pretty much any platform. We need data tools that can alleviate the current set of non-portability/latency constraints and get data where it is needed, when it is needed, securely.

Unfortunately, I don’t see that happening soon. So addressing the issue, “Where do we target our new application?” will have to start with the question, “Where is the bulk of the data it needs?” At least for now.

Most of you knew that, because you’re living it. I just thought you might have a use for a third-party article about it. Because someday, you will have this conversation with someone who isn’t living it. Enjoy.

— Don Macvittie

June 29, 2018
NASA/JPL Tech-Incubation CEO Talks Artificial Intelligence

AJ Abdallat is a serial entrepreneur who worked on artificial intelligence (AI) projects and commercializing smart sensors at the NASA Jet Propulsion Laboratory at California Institute of Technology (Caltech) from 1998 until 2011. Later he founded, launched and ran several spinoff companies from Caltech/JPL, doing AI work in support of the space program and various defense agencies. NASA, for one, has used AI in its deep space missions and on the Mars Rover for more than 10 years.

In 2014, Abdallat cofounded Beyond Limits, which secured licenses to NASA AI technology with the goal of commercializing the technologies. Beyond Limits owns exclusive licenses to 42 blocks of sophisticated intellectual property (IP) developed through NASA R&D investment, a $150 million AI-technology head start under the aegis of Caltech’s Office of Technology Transfer and Corporate Partnerships program.

Beyond Limits enhances the AI IP developed for NASA/JPL, and drives new IP and innovations through its mission to bring advanced AI technology to commercial applications. The company is an AI leader in cognitive technology that goes beyond conventional AI, binding deep learning and machine learning tools together with symbolic AIs to emulate human intuition.

We interviewed Abdallat, who is CEO of Beyond Limits, while researching the story, “AI Delivering on the Business Analytics Promise.” He offered a lot of interesting commentary that just didn’t fit that earlier story. With AI poised to potentially revolutionize business, the fact that Abdallat’s career has bridged the gap of advanced AI science and commercializing it makes his insights fascinating on topics including digital transformation, IoT, big data and what AI can and can’t do.

Note: This is Part 1 of a two-part series.

devopsy.kinsta.cloud: What three key things about AI should every C-suite member know?

Abdallat: No. 1 is understanding the transformative power of AI. Most of the AI out there today is not going to replace a human. There is a balance needed between understanding the promise of AI and not being unduly fearful of losing their jobs due to the much-publicized peril. AI is going to advance a lot of progress that’s already in place, not necessarily upend everything. So yes, there are transformative solutions in AI, but I think there are lots of places for it to plug in with what you’re already doing today, in ways that can considerably augment what is done, and free up time for individuals in your organization to turn their minds to the harder problems that AI hasn’t even cracked the shell of yet.

N0. 2 is business practicality. For every AI algorithm, there are 100 articles telling you that it’s a good thing or a bad thing. It either doesn’t matter or it’s going to change your life. I think one of the things that’s important to realize is that AI needs to start with a problem to solve. So, being cognizant of which of your business problems have the biggest potential payoff in their resolution is how you should think about applying AI, as opposed to what AI would be good at solving. The chances are, if you have a problem, AI can shed some light on it. Prioritizing how you describe that problem will be more important than selecting the right AI hammer to swing at a particular nail. Understanding what your problem really is helps artificial intelligence experts devise solutions for you.

AJ Abdallat is co-founder and CEO, Beyond Limits. He’s an AI engineer and entrepreneur.

The third thing the C-suite should know about AI is timing. The time to start thinking about this is not now, it was yesterday. Companies that started experimenting with AI a few years ago are reaping the rewards today. One large, recent study shows that 75 percent of surveyed executives say that AI will be “actively implemented” in their companies within the next three years. So, it’s happening right now.

A lot of people reading the news can’t help but wonder maybe today should be the day to act, because every day I read about something new. In fact, the sooner you can modernize your digital infrastructure, the sooner you can bring AI to address your business issues. You need to level the playing field with potential competitors. Today is the day to upgrade your networks to get more digital, get more instrumentation. The benefit is it will optimize a lot of what you’re doing today, even without AI, but it also makes the AI integration possible. AI can’t work if it can’t see the data.

devopsy.kinsta.cloud: Does AI just pave the cow path or create a whole new business model?

Abdallat: It depends on your perspective. AI can help to automate a lot of things, and that would be paving the cow path. But in general, AI can help humans see problems in new ways. Let’s say I have a bunch of data and I don’t know what’s important in it. This is the kind of thing where we used to have pre-formed opinions, where we would have a group of statisticians help us answer. Now, with an afternoon and an algorithm I can get you a hypothesis of what’s important in your data. Is that automation? I don’t know, it’s automation of ideation, which is certainly more esoteric, and that’s where we’re headed. But I think there’s probably room for both. AI will achieve new business models, but we’re also trying to get AI to do things we’ve been doing forever. It’s really a function of how. That’s what AI changes: how we do things, not so much why. Why you should change things is ultimately something the C-suite should determine.

devopsy.kinsta.cloud: CEOs make decisions about the future. Could artificial intelligence help?

Abdallat: Forty-six percent of executives are fearful that their business may get disrupted by an AI-powered startup, rather than a direct competitor. So, it’s a competitive necessity. Today, big companies are using AI for analytical, predictive, diagnostic and industrial control purposes. Some AI touches ordinary people through services, but they probably aren’t aware. What we’re looking at more and more is: How do I stay aware of a big stream of data and change my hypotheses as that stream comes online? There are companies looking at this, like Beyond Limits. It will become more common, but it’s going to be less about data at rest and more about data in motion. How can I quickly translate the data that I find for my search into some kind of action? That is the next big step for big data, analytics and artificial intelligence.

devopsy.kinsta.cloud: People say lack of data and missing data are issues. Is that true?

Abdallat: Yes. One of the things we bump up against is, What happens when I don’t have all the data, or we missed some of it? A lot of machine learning approaches are bumping into this problem as well, and what they need is human intuition or expertise to guide them over those initial stumbling blocks. That’s when the unconventional comes into play.

IoT is an excellent example of missing data and so on. The internet of things (IoT) brings us new orders of magnitude of data that we need to analyze at higher frequency time scales than we’ve been asked to do before. IoT also forces us to do it out at the edge, away from the super computer that lives in the cloud that can help us answer a bunch of questions efficiently. It’s challenging to make decisions quickly enough at the edge without the ability to go back to the cloud. Because here, the speed of light without exaggeration is a factor. You’re not going to be able to get inside that decision loop if you’re waiting for light to travel back to the server farm and out to the edge again with the information needed to make your decision. You will have to bind some of the intelligence to the edge to be inside the control loop, so to speak.

IoT also relies a good deal on sensor data that has far less redundancy. That means you’re going to get data drop out. On a single installation, some of the sensors just turn off. Others might be generating data that we know, with some prior knowledge, is out of spec; we know not to believe it. These are the kinds of problems you’re going to have when you interface with industrial machinery and lightweight distributed sensors at scale. The challenge is how to deal not only with the new stream of data, but also the inconsistency in its delivery.

Watch for Part 2 of this series, which will cover big data/unstructured data, business process issues, human factors, and where AI is headed.

— Scot Finnie

January 15, 2018
How Data Drives the Business of a Clean Energy Company
Imagine using kerosene lanterns to light your way into the evening, at home so you can cook, read, shower and the kids can do homework, or outside to keep a business or medical center open or play night games in a park or field. This is life in undeveloped nations. Not only is the light source insufficient, but burning kerosene is harmful to individual health and pollutes the air.

These issues are important to the team at BBOXX, a new generation utility company that develops clean renewable solar energy products for developing countries. Its team has developed its own “Battery Box,” a solar-powered battery box that powers lights and appliances for rural Africa—providing clean energy for numerous products and appliances including TVs, lamps and mobile phones and any other item that lights up or needs to be powered.

BBOXX describes itself as a service company, not a technology company, and as such focuses on being a data-driven company, capturing customer data, analyzing and manipulating it to provide better services and to continue to improve the battery box. Customer data is critical to its overall success: Knowing how and why customers buy and use the battery, when they use it, length of time … there are a number of data points that help BBOXX understand the customer. In addition, BBOXX needs to monitor the batteries to ensure they’re working properly, help maintain connectivity and disable units, if need be.

BBOXX’s development team captures data by remotely monitoring units. They are able to access data that is being sent from sensors via a 2G network and through the Amazon Web Services infrastructure into the cloud. Once the raw sensor data is in AWS, it is then stored in a database designed for metrics and events. From here, they are able to analyze it, provide business dashboards, generate predictive maintenance alerts and notifications and predict and find interesting business trends such as energy consumption spikes during soccer events, for example.

In addition, the team at BBOXX can also correlate and aggregate usage information, which helps them upsell additional power capacity based on upcoming events. For example, a football match between Kenya and Zambia increased demand for more energy to support extended viewing times.

Becoming a data-driven business meant that BBOXX needed access to real-time data at its core, which had another set of challenges:
- Securely monitor a distributed set of devices across low bandwidth 2G networks
- Provide real-time 24-7 monitoring to a distributed team who expects the data to always be right and available
- Provide the ability to monitor and react to high-speed time-series data
- Ability to remotely monitor and manage the distributed devices
- Ability to track usage statistics and bill based on these statistics
- Proactively monitor units so they could be repaired before the customer lost power
- Provide insight into customer usage patterns to develop compelling pricing plans
- Gather, harmonize and use three different sets of data from the various components:
  - Raw telemetry data (voltage, current, temperature)
  - Logs from the box firmware
  - Derived data (power, daily energy usages, state of charge)
- Lack of staff to manage another solution
- Keep latency systems at a minimum – their systems were already hosted in AWS and the performance was meeting the needs of the company
BBOXX is now able to gain insight into its data and apply lessons learned from analyzing past data to develop new and exciting products. The company now uses data stored in a whole new way:
- Developers analyze the data for ways to create new data streams
- Analysts look for irregularities in usage and write algorithms to detect alerts
- Business managers look for customer usage patterns
- Technicians analyze time series voltage and current patterns for remote diagnostics
- Centralized call center operators reconcile more technical issues for customers by seeing the reason for the outage (hardware failure, over-usage, non-payment)
In addition, their investors are highly interested in their data collection and analysis as a marker for competitive advantages, reduced delinquencies, growth potential and decreased risk.

BBOXX uses Amazon Web Services (AWS) to receive store and analyze its steadily increasing portfolio of products. By designing its software architecture with scalability in mind, BBOXX takes advantage of a wide range of AWS’s integrated features including load balancing, auto scaling groups and code deploy to maintain fast, reliable and consistent access both internally and externally.

Without access to user data, BBOXX would not be able to offer the unique services it does today, which is a fully vertically integrated service, controlling every part of its customer experience.

About the Author / Mark Herring

Mark is a zealous marketer who believes that the road to marketing success always leads with the developer. Before InfluxData, Mark was VP of corporate marketing and developer marketing at Hortonworks, SVP of Products at Software AG, VP of Middleware, Java and MySQL Marketing at Sun Microsystems, and VP of Marketing at Forte Software. Earlier in his career, Mark was a developer and technical support engineer for Oracle. Mark holds a B.S. Degree from the University of Witwatersrand, South Africa. Connect with him on LinkedIn and follow him on Twitter.
November 29, 2017
Making the Data Work for You: Data Visualization Techniques

When Peter F. Drucker, the father of modern management, performed a two-year in-depth analysis of the General Motors corporation in the 1940s, he undertook what may have been the world’s most ambitious big data analysis ever at that time. However, when he gave the results to GM chairman Alfred Sloan, Al was not happy.

In fact, Sloan refused to acknowledge Drucker’s work. He never mentioned it again and banished his colleagues from referencing the resultant book (Concept of the Corporation) around him.

Alfred Sloan may not have been ready to act upon the harsh reality of his company’s data, but modern businesses thrive on this sort of analysis. Big data has consumed every industry on earth, from automobiles to cosmetics to hotels and far beyond.

Businesses that use the power of data visualization will outperform the ones that don’t, every time. If you are responsible for the success of a B2B business, this article is going to show you why data visualization is one of the most powerful tools at your disposal.

The Goal of Data Visualization

There are two ways you can use big data visualization tools to make your data work for you. You’ll make it easier to parse your own data, identifying trends and finding opportunities to improve your business. You’ll also use it to convey your ideas to other people, improving your conversion rate for B2B sales that can make or break your bottom line.

Understanding Your Data

When it comes to better understanding your own data, visualization is everything. Even a small business has an incredible amount of data at its disposal. Social media statistics, front-end stats such as how long users spend on certain web pages, sales and refund statistics by time of day and product … the list goes on and on.

You have to start by figuring what data is most relevant. Ed Baker, former head of growth at Uber, points out that every great business needs a “north star metric.” This is the single variable that, if it improves, the entire business does better. Once you identify your north star metric, you can see how other variables feed into that metric. Data visualization is crucial for parsing the information and making these big decisions.

Conveying Ideas to Other Businesses

You can use well-visualized data to convey ideas to other people that might be near-impossible to express otherwise. Have you ever sat in a meeting, desperately trying to show somebody why your idea makes sense, and yet completely failing to do so?

I’ve had conversations where my brain is about to melt out of my ears, because I think I’m making perfect sense yet the other party simply does not grasp the idea. In a B2B context, this failure to communicate can cost hundreds of thousands of dollars. That’s a critical failure to get the job done. Data visualization is a very cost-effective way to improve your odds of making the sale.

When you’re trying to sell a premium service or product to another business, don’t just tell them why it makes sense—show them. Use the visualization techniques in the rest of this article to remove all doubt that you’ve got the right idea.

The 7 Basic Tools of Quality

There are dozens of different ways to visualize data. It depends on what kind of information you want to look at. However, the best way to start is by understanding the “seven basic tools of quality.”

Made famous in post-war japan by Kaoru Ishikawa, these tools are simple yet potent data visualization methods that most people can learn in a few hours. While not the most intricate or powerful visualizations available, they are elegant and widely understood. When you use one of these tools, you can rest assured that everybody at the table will be comfortable following along.

You want to use these methods of visualization whenever it is possible. Only move into more esoteric or powerful tools when the situation necessitates it. By the time you are working with complex multi-variate data visualizations, you’ll probably need a dedicated data scientist to make it all work. These tools are an easy and cost-effective way to get started.

Here are the seven basic tools, with a brief synopsis of how each one works:

Cause-and-Effect/Fishbone Diagram

A quick and simple method to use when you are working backwards from a known outcome, asking the question: “Why did it happen?” This tool is often too simple for detailed analysis, but can be great for mid-conversation visualizations on the back of a napkin.

Check Sheet

We’ve all used check sheets in our day-to-day lives. This is a great way for small groups to keep track of simple data, such as, “How often are the following conditions being met?”

An example of a check sheet in action would be a coffee shop that wants to track common issues with customers. You could have the barista check a box whenever something goes wrong: Customer says drink tastes bad, customer leaves before drink is ready, customer gets confused while ordering and becomes agitated, etc.

Control Chart

This is a X & Y axis chart where the horizontal axis is the flow of time and the center of the vertical axis represents the average value of the other variable. The top and bottom of the vertical axis are the outer bounds for that variable. The control chart is a simple way to visualize data for a single variable moving through time.

Histogram

You’ve probably heard the word “Histogram” a lot, but may not know what it is. This is a chart where the horizontal axis consists of columns representing one variable and extending upwards to show the relationship with a second variable on the vertical axis.

Pareto Chart

A chart consisting of columns and a line, sort of like a blend between the Histogram and the Control Chart. However, Pareto charts do not necessarily reflect the passage of time. The columns are always ordered from largest to smallest and the line is the sum of the data represented by the columns.

Scatter Diagram

This is a simple diagram that plots two different variables with lines across an X and Y axis, meant to show correlations between them.

Flow Chart

Also known as an “if-than” chart, this shows the potential sequence of actions or thoughts within a mental or physical process. For example: If the client likes coffee, ask them how they like it. If not, offer them water. If the client says they like the coffee with cream, add cream. If not, prepare it black. This sort of data is awkward to express through pure language, but quite easy to grasp when placed in a flow chart.

Advanced Data Visualization

There are many, many forms of advanced data visualization. This is the stuff that Peter Drucker was trying to do back in the 1940’s—capturing huge amounts of data across an entire corporation and then synthesizing it into the most elegant possible form.

If you want to go beyond the seven basic tools, you’ll need help. This isn’t the sort of thing that an employee or even an ambitious executive can tackle over the course of a few days. A team of data experts will need to spend a lot of time and energy to craft the perfect visualizations for you. With that said, here are a few advanced techniques to consider:

Heat Maps

Heat maps are used to show density of activity across a geographical space. This can be utilized to understand city traffic, or to look at where your employees spend time within an office building.

Beyond the physical world, you can use heat maps on websites to see where users tend to move their mouse, or even where they tend to glance at. For example, video game developers and social media platforms use heat maps of user eye-gazing data to optimize their visual layouts.

Word Clouds

A word cloud is a visualization where many different words are scattered about in a “cloud,” with larger words being more common. You can use these to understand the most common sentiments being expressed in customer reviews, company emails or other text communication.

While cheap word-cloud-generators exist, there are many reasons why they do not work. An effective word cloud needs to deal with factors ranging from color to glyph shape and beyond.

Comparative Graphs

Comparative graphs combine several copies of the basic power tool charts, by for example putting several control charts on top of each other to compare larger numbers of variables and factors. While comparative graphs are simple to understand in theory, they are difficult to put together in a way that is practical.

Using Data Visualization for Your Business

The best way to understand data visualization is to give it a try. You can get started with basic visualizations on your own, but you’ll want to contact an expert to really make your data work for you.

After all, it isn’t enough to put a few variables into a chart. You need to understand how to collect the right data, compile it into useful visualizations, and then act upon that data in an effective way.

About the Author / Alexey Semeney

Alexey Semeney is the CEO of DevTeamSpace, where he leverages his knowledge and expertise in product design and remote team management to build a unique software development platform. Connect with him on LinkedIn.

October 10, 2017
DevOps and Data Portability

You have a classic car. You’re in the middle of restoring it, and it looks mighty sweet. Paint is polished, wheels are stock, hubcaps (or spokes) are shiny, engine is completely restored to original spec.

But it has no gas tank.

A bit of a stark comparison, but that’s where we’re at with application release automation (ARA)—at least in terms of portability. And where we’re likely to stay.

Hold on, I’m a Fan

Before the DevOps aficionados get out the torches and pitchforks, let me be completely clear that I’m a huge fan of ARA, and explain again why.

Tools such as Jenkins allow DevOps teams to automate the entire build/test stack—or, as much of it as their organization requires. That is a huge step forward, for all the reasons we’ve covered in the past, including the one I think is often overlooked or underemphasized. With frequent build/test feedback, the code is still fresh in the developers’ head when they learn of problems. Most of the time, they are fixing before moving on to the next task, which is a huge plus compared to waterfall, where many problems show up long after the developer in question has forgotten what the code is doing because they’re off on other things.

Likewise, tools such as Puppet, Ansible and Chef (to name a few) allow DevOps and security to set standards via script, and use those scripts to deploy over and over. This has all the benefits we’ve discussed, too—speed of delivery, reduction of human error, repeatability that becomes stability, etc. This gives operations the ability to answer, “Is server X set up to do this new thing?” by looking in deploy scripts and quickly updating things if the answer is no, but X will be required going forward. That’s one of my favorite bits of Application Provisioning.

Riding the Gap

For those unfamiliar, ARA simply bridges the gap between DEVops and devOPS. Tying code being released to platforms and configurations it will be released under. It has the ability to rapidly re-target a release to a new environment, and it can do predictable and repeatable releases, offering the benefits of application provisioning for the actual release segment. Most tools do this by tying together your favorite build tools and provisioning tools, and giving Ops a control panel that can manage the process end-to-end.

The benefits are huge. Particularly for Operations and Security (who have another place to standardize and insert essential monitoring), but also for development because it offers one further step in testing and by doing so reduces the surprises six months down the road, when you’ve forgotten what

Bubblegum += 10

Is in your code for.

So don’t take my intro as offensive. I am a fan. But I’m a realist. There is no panacea.

It’s About the Data (as Always)

The two weakest parts of most ARA systems are server provisioning and data issues. Server provisioning is setting up OS/hardware, and many ARA systems tend to assume you magically did elsewhere, but this problem is only an issue in highly physical environments and there are tools to handle the problem; they’re just not integrated well with most ARA systems. Data issues are the bigger problem.

It’s cool that we can drag and drop a couple boxes, and magically switch our deployment from inside the DC to a cloud provider in the time it takes the ARA to build/deploy. But we all know that’s not how the world works. Now you have an application, complete with database that is correctly configured and security that does AAA solidly … and no data.

“Just move the data, then!” the choir sings. No, that’s really not how it works in an enterprise. Data itself is an object of security and compliance, and some source databases are so massive that moving them is a nearly impossible chore in and of itself. Amazon tips its hat to this fact with its import/export service.

This was the reason why I gut reacted to the “No more physical data centers!” crowd back in the midst of the hype with an eye roll. The amount of data stored in data centers is staggering, and often not portable at all. Add to this the fact that such data isn’t going to reduce, and we have a quandary.

What About Moving Forward?

Sometimes, it is necessary to accept that the current state is not changeable, but take steps to improve the future state. I don’t see this as a reasonable approach in this case, because the rules of data aren’t changing in a way to change future state in a positive way. We’re generating more data every day, and that data will end up stored somewhere. Be careful where you store it, think long-term strategic. Having ARA doesn’t magically make that data portable, so make sure that where you store it today, while it’s small, is where you’ll want to store it in five years, when it is a massive, largely immobile asset.

What to do … What to do

Use ARA to multitarget all of those apps that are portable without terabytes of data, and make sure data generation/collection/analysis apps are on the architecture you will want them on moving forward. ARA offers a ton of other benefits, so data portability isn’t a deal breaker—or even a significant negative, since you probably already knew some datasets weren’t going anywhere—but don’t expect miracles out of it. It can’t make your large datasets portable, so don’t expect it will. And use it to target the platform your organization feels is most appropriate for apps that will generate or collect a lot of data.

— Don Macvittie

September 22, 2017
Study: CIOs Want a Chief Data Officer
Quick, what does CDO stand for? Before you say chief digital officer, think again. A new study points to the chief data officer as the more important of the two titles, finding 82 percent of CIOs believe there is a compelling case to hire a CDO in most organizations. The survey, commissioned by Experian Data Quality, was completed by 200 CIOs and 50 chief data officers in June and released in August.

But while the report stresses the importance of the CDO to companies that want to become data-driven, it also reveals severe challenges for some CDOs, including very high expectations, insufficient resources and, despite the C-level title, a lack of authority to get the job done.

The survey states that the core responsibility of the CDO is “to define a data-management strategy and translate it into tactical, [achievable] steps for the business.” To make data actionable, it needs to be accessible, accurate and standardized. Here’s a simplistic example of why that is important: Let’s say a data analyst needs to create a report that isolates customers based in New York City. If some of them are listed as being headquartered in NY, NY and others are listed as based New York, NY and still others are in the database as New York City, NY, that’s not standardized. The data analyst would need to realize the data is splintered and needs to be stitched together to get accurate results.

Making your business data-driven is part of the potential promise of digital transformation. CDOs and their ilk face a daunting task. Transforming data into business intelligence is the alchemy of this age.

Chart courtesy of Experian Data Quality

The study identifies three main drivers for hiring a CDO: capitalizing on big data opportunities, creating a competitive advantage and reducing risk on data-driven projects. These roles go far beyond the data management and data governance toward enabling business innovation. Over the next few years, the CDOs surveyed predict that their top business challenges will concern data security, keeping pace with technology advances and managing increasing user expectations.

Perhaps one of the biggest challenges some CDOs face is the inconsistency across companies of the nature of their role and their stature within the organization. Bottom line: They may not have the authority they need to succeed. The survey reveals, for example, that the average tenure of a CDO is only two years. Many CDOs report that despite their C-level title, they are not considered to be a part of the C-suite. Although 37 percent report directly to the CEO, a larger percentage reports to the CIO, CTO or some other part of IT. That can create some friction because it’s important that CDOs are able to bridge the gap between IT and lines of business—and empower the business.

Tips for Structuring the CDO’s role

In creating a new CDO position, the C-suite should be advised of these common challenges faced by CDOs identified by the survey:
- Staying within budget
- Keeping up with evolving technologies
- Knowing how to interpret information
- Finding/hiring people with data expertise
- Managing ever-increasing user expectations
- Balancing the needs of business users and IT
- Having a lot of responsibility with little room for error
- Getting pulled into the weeds
- Maintaining data security
You should consider extending these responsibilities to your CDO:
- Having significant control of the management and use of company data
- Playing an active role in digital transformation
- The responsibility for enterprise-wide governance and utilization of information as an asset
- Manager of a team of experienced data employees
- The purview to implement different approaches to managing data as an asset
- The purview to generate business intelligence that unlocks and supports business innovation
- The authority to be recognized by the entire company is the person overseeing data strategy and process
- Recommend: At least a dotted-line report to the CEO.
— Scot Finnie
September 14, 2017
Application Release Automation: Why and Not

But what if it has no tank?

You have a classic car. You’re in the middle of restoring it, and it looks mighty sweet. Paint is polished, wheels are stock, hubcaps (or spokes) are shiny, engine is completely restored to original spec.

But it has no gas tank.

A bit of a stark comparison, but that’s where we’re at with application release automation (ARA)—at least in terms of portability. And where we’re likely to stay.

Hold on, I’m a Fan

Before the DevOps aficionados get out the torches and pitchforks, let me be completely clear that I’m a huge fan of ARA, and explain again why.

Tools such as Jenkins allow DevOps teams to automate the entire build/test stack—or as much of it as their organization requires. That is a huge step forward, for all the reasons we’ve covered in the past, including the one I think is often overlooked or underemphasized: With frequent build/test feedback, the code is still fresh in the developers’ head when they learn of problems. Most of the time, they are fixing before moving on to the next task, which is a huge plus compared to waterfall, where many problems show up long after the developer in question has forgotten what the code is doing because they’re off on other things.

Likewise, tools including Puppet, Ansible and Chef (to name the top few) allow DevOps and security to set standards via script, and use those scripts to deploy over and over. This has all the benefits we’ve discussed, too: speed of delivery, reduction of human error, repeatability that becomes stability, etc. This gives operations the ability to answer, “Is server X set up to do this new thing?” by looking in deploy scripts, and quickly updating things if the answer is no, but X will be required going forward. That’s one of my favorite bits of application provisioning.

Riding the Gap

For those unfamiliar, ARA simply bridges the gap between DEVops and devOPS—tying code being released to platforms and configurations it will be released under. It has the ability to rapidly re-target a release to a new environment, and it can do predictable and repeatable releases, offering the benefits of application provisioning for the actual release segment. Most tools do this by tying together your favorite build tools and provisioning tools, and giving ops a control panel that can manage the process end to end.

The benefits are huge, particularly for Operations and Security (which have another place to standardize and insert essential monitoring). And also for Development, because it offers one further step in testing and, by doing so, reduces the surprises six months down the road when you’ve forgotten what

Bubblegum += 10

is in your code for.

So don’t take my intro as offensive. I am a fan. But I’m a realist. There is no panacea.

ARA: It’s About the Data (As Always)

The two weakest parts of most ARA systems are server provisioning and data issues. Server provisioning is setting up OS/hardware, and many ARA systems tend to assume you magically did elsewhere. This problem is only an issue in highly physical environments and there are tools to handle the problem—they’re just not integrated well with most ARA systems. Data issues are the bigger problem.

It’s cool that we can drag and drop a couple boxes and magically switch our deployment from inside the data center to a cloud provider in the time it takes the ARA to build/deploy. But we all know that’s not how the world works. Now you have an application, complete with database that is correctly configured and security that does AAA solidly … and no data.

“Just move the data then!” the choir sings. No, that’s really not how it works in an enterprise. Data itself is an object of security and compliance, and some source databases are so massive that moving them is a nearly impossible chore in and of itself. Amazon tips its hat to this fact with its import/export service.

This was the reason why I gut-reacted with an eye roll to the “No more physical data centers!” crowd back in the midst of the hype. The amount of data stored in data centers is staggering, and often not portable at all. Add to this that such data isn’t going to reduce, and we have a quandary.

What About Moving Forward?

Sometimes, it is necessary to accept that the current state is not changeable, but take steps to improve the future state. I don’t see this as a reasonable approach in this case, because the rules of data aren’t changing in a way to change future state in a positive way. We’re generating more data every day, and that data will end up stored somewhere. Be careful where you store it; think long-term strategic. Having ARA doesn’t magically make that data portable, so make sure that where you store it today while it’s small is where you’ll want to store it in five years, when it is a massive, largely immobile asset.

What to do … What to do

Use ARA to multi-target all of those apps that are portable without terabytes of data, and make sure data generation/collection/analysis apps are on the architecture you will want them on moving forward. ARA offers a ton of other benefits, so data portability isn’t a deal-breaker—or even a significant negative, since you probably already knew some datasets weren’t going anywhere—but don’t expect miracles out of it. It can’t make your large datasets portable, so don’t expect it will. And use it to target the platform your organization feels is most appropriate for apps that will generate or collect a lot of data.

— Don Macvittie

July 3, 2017
Self-Healing Service Management and the Future of DevOps
How can DevOps teams take continuous delivery to the next step? Part of the answer lies in self healing, which could optimize performance management and monitoring and underpin the software delivery pipelines of the future.

Self healing means the ability of systems or environments to detect and resolve problems automatically. It eliminates the need for manual human intervention, upon which most systems today still rely to fix issues.

Self healing is better than depending on humans for several reasons:
- Automated solutions are less likely to miss issues by mistake, or to make errors when resolving them.
- Self-healing technology is better positioned to fix problems faster than humans because it can react instantly based on rules or machine learning. It doesn’t have to stop and think about the issue before taking action.
- A self-healing environment can scale without limit. This is a key advantage over manually managed environments, whose scalability is limited in some respects by the availability of human admins.
Self Healing and Continuous Delivery

DevOps teams already take advantage of many other kinds of automated tools to help achieve continuous delivery.

They use continuous integration servers to automate tests. They adopt release automation tools to streamline deployment. They implement orchestrators and scripted infrastructure to provision environments automatically.

Yet, when it comes to performance management and monitoring, the typical organization is a far way off from achieving a fully automated process. Even the best application performance management (APM) tools of today are primarily capable only of detecting problems and displaying information about them. They depend on humans to interpret and react to that information.

That dependency is a major obstacle for achieving fully continuous delivery. You can’t release code quickly if your process for finding and solving problems related to the code is mostly manual.

By making better use of self-healing technology, DevOps teams can speed continuous delivery in the following ways:
- Developers and admins can spend less time on monitoring and more time completing other tasks that can’t be automated.
- Ideally, teams will be freer to switch between development environments and frameworks without having to establish a new monitoring workflow. Self-healing tools will adapt to whichever environment they have to work with.
- Because problems can be found and fixed faster, infrastructure utilization can increase. The need for a buffer between average usage and overload can become smaller because automated healing solutions will be able to fix problems before they lead to failure.
- Teams can establish better continuous feedback loops because, by removing the human element from performance and monitoring, incidents can be reported instantly, consistently and in a way that makes the data easily accessible to everyone on the team.
To be clear, the technology for self healing does not yet fully exist. Developers over the past several years have made great strides in automating configuration, service discovery and environment provisioning, But perhaps it’s time to make them a more prominent focus. They’ll enable the DevOps innovations of the future.

— Chris Tozzi
April 18, 2017
When Did Safety Nets Become Optional?

There are various reasons why some organizations would not be interested in protecting their data. These reasons could range from the lack of financial resources to the lack of non-financial resources, such as engineers to implement a solution. Regardless of whether there is a good reason to not protect your data, the net result is the same: If organizations don’t do it and something bad happens, they will be in a world of hurt.

Data is the lifeblood of any business. It’s true that not all data is created equal, so some organizations may be comfortable losing some data. And it’s also true that there are some systems or applications that are built to not only tolerate data loss, but also seamlessly absorb the loss of data. The reality is that this is not 100 percent of the systems in your business, nor is it 100 percent of the data. If you were to completely lose the users table in your application or have it suffer from data corruption, how would you explain this situation to your customers? Can you realistically tell them that the money wasn’t there, or you didn’t have the resources to protect their data and yours? What type of trust would that instill in them for you as a provider of services? What type of trust would that instill in the business for your technology team?

For most businesses, data protection should be thought of as a component of the overall security strategy. If you wouldn’t set up a network infrastructure without a firewall for protection, why would you create an application without protecting the data that powers it?

Unfortunately, some form of system failure is inevitable. There will come a time when something bad happens. The best thing you can do is to be prepared as possible. Providing yourself with a safety net when this happens is doable, but shouldn’t be thought of as optional. There are some basics tenets that should be involved in providing a proper safety net. The most important of them is to plan for and failure and test your recovery procedures. This team education is your primary line of defense against disaster. It will help mitigate many of the potential failure scenarios through awareness. Data protection is for what happens when things get past this point. And protecting data should not be optional.

— Jeannie Liou

April 18, 2017