Author: Don Macvittie

  • Is Agile Methodology Right for Your Project?

    Is Agile Methodology Right for Your Project?

    I’ve read a few articles of late that have declared the end of Agile development. They are interesting reads for the most part, and they make some interesting arguments, but I see none of them as compelling. I do, however hope that this is the beginning of applying “Right Tool for the Job” to how projects are handled in development/DevOps.

    Agile officially kicked off in 2001, but comes from 90s concepts such as Extreme Programming. That makes it old. But, age isn’t in and of itself a problem. Adaptability beats age every time, and there have been movements that update parts of Agile to keep up with the times/changing environment. My issue is that “We’re an Agile shop” becomes a mantra that implies “Everything is an Agile project” which is absolutely untrue. You can do any development you can consider in Agile. You can also ride your bike from Main to California. Doesn’t make either one the best option.

    As with any methodology, we’ve also had a couple decades for a burgeoning oversight function that interferes with the goals of Agile. The point was to rapidly iterate to get features out the door, low-hanging fruit first. But the weakness of any high-speed development effort is quality, and as fingers have gotten burned from too much change causing problems, oversight has been built in to temper the volume of change. That oversight inevitably becomes rigid and slowing.

    As others have repeatedly pointed out, Agile works best for small teams with shared focus and geographic proximity. That hasn’t changed much. The opposite problem—a project with large, geographically dispersed teams having divergent foci—are a terrible use case for Agile, no matter the flavor of Agile you are implementing. Incremental development is also not the best answer to all problems.

    Indeed, contrary to popular belief, large initial implementations do best if using traditional development methodologies with an Agile approach to feature inclusion. That is because what undermines large projects is growing the size of the project by throwing every requirement and the kitchen sink into Minimum Viable Product, or whatever you call the actual must-have features for your application. Using Agile mindset as a filtering tool, and development methodologies that get a base level of functionality up and running (rather than incrementally implemented) allows more time for testing complete features and limits the number of features dumped into the must-have bucket.

    I’m happy to see this conversation starting to happen more seriously. Agile is a powerful tool that is the silver bullet for some development efforts. But it is only a tool, not a lifestyle. Using the Agile tool does not preclude choosing other methodologies (even waterfall, though it is currently passe’ to suggest its use) when the project is best suited to it. Most large organizations do some amount of this already, but it should be standard practice to choose methodology based upon project and suitability to task. Hopefully this is the first step toward doing so.

    — Don Macvittie

  • Pendulums and DevOps

    Pendulums and DevOps

    I have long noted the trend of pendulums in IT, particularly in organizations with longer histories. Centralized IT will be achieved, its weaknesses noticed and a movement will begin to decentralize. Decentralized IT is achieved, and people remember its weaknesses, causing centralization.

    The same is true with speed of delivery versus quality. Faster will be more important for a while, then the reality that faster causes quality concerns will surface, and reliable becomes the keyword, causing slower rollouts of better product.

    This has been going on since I started in IT, and while it is more evident in enterprises, any company that lasts long enough will see these cycles run through.

    Agile and DevOps, as the Great Disruptors, stood a chance of redefining the tensions that cause these pendulum swings. Centralized versus decentralized is less important if stakeholders are all truly involved and informed—something that the culture side of the DevOps crowd aimed to achieve. Similarly, fast and quality were promised, as the ability to recover from inevitable mistakes would be rapid, while integrated testing would uncover most issues.

    We have enough years under our belts to know at this point that these issues are, at best, resolved with a resounding asterisk that is clarified sometimes.

    One nice thing about DevOps (more than agile) is that it has not stopped evolving yet, which means there is still hope that rapid delivery and high quality can both be answered while involvement of business owners and all the various IT groups will reduce the centralized/de-centralized cycle. But we’re not there yet.

    Split.io—purveyor of feature flagging/tagging tools—has a pretty good blog about what they discovered when they went out and talked to hundreds of shops. It shows that failures are happening at an alarming rate, and at some number of shops, for your average bug, they will still be fixing this released issue when the next release goes out. That is asking for trouble, in my experience.

    Better integrated testing is certainly part of the answer. TDD is good, but not enough, since it is more granular than some tests require. Split believes that feature flags are part of the answer, and I believe they’re right. But this approach is a band-aid to the problem.

    By the time you flip a feature flag to turn off a bad feature, you’ve already released a bad feature. And you will be spending time researching and fixing it. I believe feature flags should be used to catch it when this (inevitably) happens in a complex coding environment, but at the same time the source of issues should be looked into. An ounce of prevention is indeed worth a pound of cure.

    Figuring out what is allowing bugs to slip through into production rather than counting on remediating them (using precious time to do re-work) should be a priority moving forward. Is there a process/policy that is at the source? Is it simply a training issue? Training issues are repetitive—teaching today does nothing for new algorithms/tools put to use after today, and turnover can create a training gap if you’re not staying on top of it. For some shops, is speed of delivery faster than some piece (normally testing, but some shops suffer for development on certain platforms also) of the delivery chain can adequately respond to? Do you need to re-evaluate toolchain and delivery timelines?

    We are delivering more code, faster than ever, and that is exciting. But quality matters, and anyone who relies on end-users to test is playing a very dangerous game. Find ways to ensure quality without slowing down the process more than necessary. At this point, I honestly would implement runtime-flippable feature flags as a starting point, simply to be more responsive to issues. From there, work to resolve the underlying reasons that bugs are cropping up.

    Until then, keep cranking it. Keep working with the business to get quality systems in front of users, and enjoy doing some great work.

    — Don Macvittie

  • API Security in DevOps: Are We Too Comfortable?

    API Security in DevOps: Are We Too Comfortable?

    Postman performed its annual survey of developers, and as this survey and many other surveys have shown, developers are generally comfortable with the level of API security that their organization has implemented. This has been a consistent trend, with most developers being comfortable that they are protected.

    The problem is most organizations also put API security and management in the hands of developers.

    Knowing the rate of change going on in Agile/DevOps shops, and knowing the complexity of environment, I think this mentality is akin to the pre-WW2 French belief in the Maginot Line.

    While API security is certainly a necessity, it is not managed in a vacuum. You really are only as secure as your weakest link, and there are a lot of links in an API-focused environment. User credentials, tokens, SSL, requesting apps and their authentication, (still) code injection concerns…The list goes on and on.

    I’m never comfortable with the level of API security currently implemented in DevOps environments. And I think it’s dangerous for you to be. It only takes one bad code iteration for one of the client instances to be vulnerable. That vulnerability exposes every API the client in question has access to, no matter what is being used as access control.

    In the end, APIs must trust someone, or they’re useless. And that someone is most often a server app running on an instance that your users connect to. That app most often has its own credentials. How are they stored? How are they protected? What code changes could impact that instance’s security?

    API gateways and management tools, when properly configured, are a big help here, but they are not a panacea. Since incoming connections must be trusted at some point, and outgoing connections must be assumed clean at some point, when one of those two things is a wrong assumption, what do you have in place?

    Here’s the scenario: User A has rights to call the ShowMeCriticalData API. This API is called from a server app running on instance B. The server app authenticates user A, then calls the API for him–either with his credentials (best case) or its own (bad for tracking problems). If instance B receives an update that makes it vulnerable to compromise, user credentials are still needed to call ShowMeCriticalData, which is relatively secure, as long as User A is protecting their credentials adequately. However, if the app running on instance B is proxying requests and using its own credentials, Instance B becomes a weak point and attackers could use the vulnerability to call APIs without authenticating users. Worse, if User A is careless with their authentication info, there is little an organization can do to stop at least some misuse.

    API logging and review can be put in place to notice odd behavior by username, and offer early warning that a user has been compromised. But will only catch most issues.

    Of course, there are smart people working on the whole array of API security threats from cross site scripting to authentication spoofing even into TLS weaknesses. But that work is only useful if we all remember that our security could be better. With knowledge that there are weaknesses, we can look for things to mitigate the risk. As you all know, if you have public APIs the risk will never be eliminated–the purpose of a public API is for people to use it, and that exposes it to hackers that seem to be more ingenious by the week–but minimizing the risk is worth the effort.

    Being secure in your current situation means you’re not focused on making it better. With API security, particularly in the type of dynamic environment DevOps creates, this is a dangerous road to be on. Assume there is a fatal flaw, and work to reduce vulnerabilities at each access point into the overall system.

    This also goes for APIs you don’t own. We rate limited an API that called out to Google Places to match up with our plan/budget for the project. Our app was never going to come anywhere near Google’s caps on usage, but should a portion of our infrastructure be compromised, we could potentially end up with a massive bill from Google because we called the API a zillion times in a month. Likely? Not really. Possible? Not after a few lines of rate-limiting code.

    You all have made some astounding apps. Make sure they are astoundingly protected. Because both business and users are counting on it.

    — Don Macvittie

  • Don’t Expect Miracles

    Don’t Expect Miracles

    Let’s run some fun scenarios to start this one off, shall we?

    • “We’ll decide what language to use at a meeting next month. Until then, get to writing code!”
    • “We might deploy internally, though Azure is a strong option, and we’ve discussed GCP. Now get operations up and running, we’ll tell you when we’ve decided.”
    • “The code is changing constantly, get moving and write the documentation.”

    All three of these scenarios are equivalent. The difference is how we, as an industry, view them. The first two, we (collectively) would never tolerate. The third? We do that to the documentation team every day. In DevOps and agile, we do it every day in a never-ending iteration. And it has to stop.

    Two Different Documentations

    For those not aware, there is documentation for internal use and documentation for external use. While we’re certain you have internal covered with self-documenting code, this blog will focus on external use, because if you expose and API, documentation is mandatory. If you’re not an end-user product, any UI requires documentation, too. So to be clear, for this blog we are talking generally about external documentation for B2B systems, and specifically about APIs, though UIs can have the same issues, depending upon how complex your market is.

    We Generate Docs

    Ever tried to use auto-generated documentation? I’ve yet to try one generated set of API documentation that I liked (they say Google Places has it, but my understanding is devs actually write that documentation and it’s reviewed by tech writers, so maybe we count that as one I liked, but its process is not yours).

    Generated docs are only as good as the developers working on the code–very much like self-documenting code. If someone puts “This parameter is a placeholder for the spline development iteration” into a generatable comment, users are going to be totally clueless what that means. Heck, I just dreamed that statement up, and while we can all see it turning up in generated API docs, even I don’t know what it might mean.

    Documentation Slows Up Dev

    Duh? If you are loathe to commit to completing a project until you know the parameters, why would you expect a documentation team to be any different? Of course they need the final product for review, just as the business wants to see the final app before release. The product your docs team produces is detailed documentation about what is. If you change what is underneath them, they look bad for having inaccurate docs. When really it should be IT that looks bad for not considering the entire value chain when delivering. If you deliver an API that is poorly documented and people don’t use it, you have delivered nothing of value to the organization.

    What to Do?

    Put documentation on your DevOps teams. I mean an actual person whose responsibility is documenting. If you’ve got technical writers, use one of them. If you don’t, find someone who likes helping people use your toys, and give them time to do it. For some organizations, technical evangelists can fulfill this role if you don’t have them too busy helping active users. Have them sit in on meetings, treat them as just another piece of both agile and DevOps. They need to know when functionality, URI or whatever else changes, so they can update the docs. The best place to get that info is at stand-ups or design meetings.

    Just like always, they’ll want to be there for design, but they’ll also want to be there for implementation decisions/changes. Because current documentation is the only worthwhile documentation. Give them sign off on release–not release of code (that’s a done deal) but publication of APIs. Before you let users at the access points, make sure they have solid instructions in how to use them.

    Risks

    Everything is a trade-off, but this one has some pretty easy choices to make–fewer people use your API because your documentation isn’t great (outright sucks in many cases), or you run some risks of slower iterations and you dedicate resources to documentation. Given that if no one uses your API, it is all wasted time, a bit of slow-down and man-hour investment is an easy choice to make.

    Tech writers are generally cautious people who want perfection–because this is their product, and they want it right. This can cause unnecessary slow-downs in an agile/DevOps world. The easy answer here is to look for the same traits in documentation DevOps team members that you look for in other DevOps team members–ability to adapt, understanding that everyone is just trying to make it better and willingness to work through changes quickly to deliver.

    Just like you don’t want the DBA that says “no” to every request and you have to argue with just to move the project forward–I worked with this guy, genius, but really frustrating to have on an agile team–you don’t want the documentation person that adds weeks to timelines so they can get it right. This risk is easy to manage. You know how to choose people for a fast-moving environment, just follow your gut.

    Profit

    Well, okay, maybe not profit, but deliver a solid product quickly with sufficient documentation that users not steeped in your corporate shared knowledge will understand and be able to use. That’s the goal of public-facing APIs, to be used. And it’s the goal of docs people to make certain those calling the APIs understand what they’re getting into and how to make the most of it.

    Keep on cranking out cool solutions and crank out better docs. Your customers are counting on it. I’ve had my team go find alternative solutions based on shoddy API documentation. If I can’t use it, why would I keep trying if there are alternatives?

    — Don Macvittie

  • App Mobility Is the New Use Case

    App Mobility Is the New Use Case

    Selling increased investment in DevOps is not always easy. It takes time to implement, resource away from other projects and inevitably there is cash outlay. If not done carefully, it also appears to introduce chaos, something most of us in IT strive to avoid.

    But the benefits, at this point, are easy to document. Plenty of companies have tried and succeeded in moving to a more agile/DevOps approach to deployment. Yes, companies have failed, but the publicly acknowledged failures have also been dissected for lessons learned, which can help in your transformation.

    Every advice site suggests tying DevOps/digital transformation to business goals, which is a good idea, but is harder to do most of the time.

    Not any more. The truth is the future is mobile. Not as in mobile phones or cross platform (though both have a place in the future), but in the ability to run an app where it makes the most sense today with the knowledge that tomorrow there might be a better place to run it.

    Historically, the architectural choices of yesterday’s teams become a burden on today’s teams. I’ve worked at a major insurer where a mainframe flat file DB system was keeping an application from moving. I’ve worked at a power company where we were not allowed to modify a system running on OS/2 because no one quite knew how to maintain it. I’ve worked at an energy services company where I spent a year trying to re-create Novell IPX solutions in an Ethernet environment…The list goes on.

    What if we could avoid leaving such a legacy to future teams, and be more responsive to business needs/profitability in the future? That is, in a nutshell, what containers offer you. Today, you are running your app internally in a container management system. Tomorrow you’re running it on AWS in a container management system.

    There are things to watch out for–most vendor-sponsored container management makes it very easy to use containers on their platform of choice, but difficult to move containers. IBM, as one of the least intrusive examples, offers integration with their proprietary tools. Once you’ve taken those integrations, your mobility/options are more limited. As I mentioned, IBM is one of the more open companies out there. They offer value for tying you down, but if you don’t need those services, it’s better not to use them so you can move the application at will.

    Situations change, and containers allow for those changes. You may have a huge budget for external cloud apps today, but reasons ranging from internal cost decisions to regulatory changes could force your organization to reconsider options in just a few short months.

    That makes mobility important to business. So does cost management. No matter what the cheapest architecture to deploy on tomorrow is, wherever makes the most sense is going to be a hit.

    But how portable are they? Depends upon the app, and the implementation. An application that is developed as a group, able to run in a single pod (or multiple exact copies in multiple pods) is an easy thing to move about. One with containers spread across a variety of platforms, each doing what it is best at, will be more difficult to move. Persistent storage kind of guarantees that most enterprise apps have limited mobility–at least for the time being.

    Any app that can be moved to the cloud can be containerized first, granting mobility that means no need for massive conversion efforts if the cloud doesn’t work out. Any app pulled back from the cloud to the DC has the same potential, the issue of where the data lives is real no matter what architecture is chosen for implementation. But for any app that can be moved, it can be put into containers and moved far more easily in the future. For many apps, it is simply dropping it into the new environment and configuring networking to route traffic to the new location.

    We’re already going to leave future teams with a hodge-podge of databases, programming languages and networking tools such as WAF/load balancing. Let’s make life a little easier on them, and help increase the use of DevOps, by deploying with containers and targeting whatever makes sense for the org today, secure that tomorrow that can change and our burden will be less than it might have been.

    — Don Macvittie

  • Moving Things Around: The Acquisition of Protego

    Moving Things Around: The Acquisition of Protego

    I received a note from Protego Labs’ vice president of marketing about Check Point Software acquiring Protego. We see a fair number of acquisition announcements, and usually don’t write about them–though we read them and stay abreast of industry happenings.

    I used to follow Check Point, and haven’t looked into what they’re doing in a while, so I followed through links to both organizations and looked at their products.

    I found myself becoming the wizard that some call Tim. But instead of “What is your name? What is your quest? What is your favorite color?” I was looking into “Where is your site? What are your products? What value do you add?” My beard is kept trimmed, more along the lines of a knight’s, so it makes sense I would ask different questions. As long as I’m not spouting “It’s just a flesh wound,” I figure I’m doing okay.

    First off, this merger makes sense. Protego does it for serverless and Check Point is doing their best to be where your code is running. Indeed, Protego probably needed a merger of some kind because all they do is serverless. Check Point’s Cloud Guard Dome 9, by contrast, will protect access rights and data for serverless, but doesn’t do what Protego does–keep an eye on the code. So this is a good fit.

    But it once again brings to light an interesting issue as you move things from DC to cloud A to cloud B to serverless. Where are your support functions? Networking, security, storage…These things need to be where the code is, but architectural options are more than they’ve been for decades. And the architectures are largely incompatible. Consider network functions for a physical-server based DC versus a VM DC, versus a Container DC, versus Cloud…You get the idea. The array of functionality and where it runs is different. Different in implementation and different in maintenance.

    This was a problem I (and others) pointed out when cloud started taking off: more work for ops, even if some of it is automated. Turns out ops, storage management and security all have more work, because all of those items are handled differently in the different architectures. And Protego had to focus on one to get it right, meaning they needed some kind of merger in the long run.

    So do you. Adding more and more work to ops–be it DevOps or traditional ops–is not the answer. A plan for standardizing is necessary.

    That is one of the big promises of containers, in my opinion. Not only are they more agile and easier to use in a DevOps environment, they are relatively standardized. Ops will still have more work if you’ve got apps in several of those architectures, but containers obfuscate some of the differences and allow standardized networking within the pod/node/group, for example.

    Some functionality just doesn’t fit into containers–you still need a physical network in the DC, or to configure the virtual network in a VPC to get traffic to/from your containers–but much of it does fit. Once at the pod/group, networking is virtual and the same regardless of where the containers are running. Which reduces overhead while making changes easier. The same is true for security, and increasingly for storage.

    There are great companies out there tackling these issues, and I’ll look at them on occasion. Right now, I know my former employer F5 Networks is working to solve the networking services/performance/security issues, and taking advantage of NGINX to handle some parts the core org traditionally hasn’t addressed. There are other companies, I just hear about that one more because I am still very close to some F5ers.

    It is worth planning for and certainly worth looking into more portable options, anything that lightens the load and keeps your great apps running. Keep kicking rear and spinning up apps.

    And don’t call me Tim.

    — Don Macvittie

  • The Use of Feature Flags in Code: Too Many Flags?

    The Use of Feature Flags in Code: Too Many Flags?

    It is interesting to watch the growth of an idea over time. The idea of using flags in code to do different things is as old as programming itself. Old-school configuration files used them, and they worked, so they were adapted.

    Thing is, given an idea that works, humanity inevitably takes it too far, and flags to control software behavior seem to have reached that point. If you are using one set of flags for A/B testing, another to segregate code in development at runtime, another to turn functionality on for some users but not others and yet another set of flags to hide data based upon user…You’re buried in unnecessary code branches. If your organization isn’t great about cleaning up after flags are no longer needed, you are building spaghetti code that future developers will not thank you for.

    The best answer to this growing burden is focus. What is the best use of feature flags/toggles in your environment? They certainly speed the delivery process by having the code there, in nightly builds, just not executed until the flag is flipped. This helps in rollout also by allowing operators to turn a feature on when it is ready by changing config rather than redeploying the application.

    For A/B testing, in many cases it makes more sense to handle switching at a load balancer, where again it is a config, but separate instances serve the UI in question. This, like all decisions regarding flags, should be based upon the volume of change. Changing the contents of a list is likely easier to test with flags, but large volume of change is probably better run separately and switched off of a load balancer.

    The same is true with the volume of change in source. A rollout might be the best option for handling large volumes of change, and feature flags may simply create more complexity.

    As with everything, approach the problem with a plan that works best, in this instance, in your environment. There is no denying that the ability to turn on a new feature and then turn it off if something goes wrong is appealing, just use the idea judiciously. And clean up after yourself. No shop that uses flagging extensively will say “yeah, we don’t clean it up,” because the mess of nested checks for flags would slow the code and make it passingly difficult to even read, let alone maintain, over time. So approach it from the beginning with “It is a tool, and when it no longer serves its intended purpose (the feature is fully live or abandoned), we remove it.”

    And keep rocking it. You are the engine of business, and this is what you do. Make it great and keep it running.

    — Don Macvittie

  • DevOps: Get the Basics Right to Avoid Costly Mistakes

    DevOps: Get the Basics Right to Avoid Costly Mistakes

    It is a truth of both agile and DevOps that speed is of the essence. This is a relative statement—some apps/portfolios/orgs envision speed completely differently than others—but if you are releasing more frequently than you were in the past, thanks to DevOps, that’s kind of the point.

    Yet, we see a whole selection of things that fall by the wayside or are underserved. Security and testing are the easy/obvious ones. Considering the frequency with which we see “Unsecured Amazon storage bucket” (one announcement of this type spurred this blog), should we not see this folded into standard procedure or tools? Well, it is folded into standard tools for checking your cloud environment—I’ve used a couple of them and seen this warning come up. So shouldn’t organizations be using these tools more?

    One thing about speed: It tends to cut corners. And cut corners lead to lapses. You can be a blameless environment that doesn’t punish people for mistakes, but for an organization, a mistake such as exposing hundreds of thousands of users will be punished from the outside, from regulatory agencies and the market in general. They’re not mistakes we should be making, and yet it still happens.

    Another one is rollback versus roll forward. Most DevOps shops focus on the ability to fix an issue and plow forward. This works really well, 90+% of the time, and fails spectacularly sometimes. While it is an added step with some slowing of process, being able to roll back changes is important. And more than, “Well, theoretically, Git will allow us to go back to the last marked release …” because there is far more to an application architecture than just the source stored in a repository.

    In short, no matter how fast things are going, have a plan for when things go wrong. Because they inevitably will. And take the little bit of extra time for preventatives such as warning someone other than the creator about buckets with no security.

    “But there isn’t time …” is the common refrain to blogs such as this one. I get it. The increase in delivery only creates an increased appetite for delivery. But is there time to respond to disaster, should the worst happen? And I’d argue that it must be IT that plans for disasters. When someone gets fired because there was a security breach or a total systems failure, it is rarely an app owner. That puts the responsibility to make certain one doesn’t happen squarely on IT.

    We’re delivering faster, offering increased functionality to users, moving forward on every front. You all are out there rocking it daily. Don’t risk all of that being ignored because you failed to take care of basic issues. Check out the tools to help out; feature flags and cloud security scanners both spring to mind with regard to my examples above. The tools are out there, you just have to work them into your process.

    — Don Macvittie

  • Oracle v. Google and Its Impact on APIs and Code

    Oracle v. Google and Its Impact on APIs and Code

    What, if any, impact will the Supreme Court’s rulings on Oracle v. Google have on the industry at large?

    TL/DR: It is unlikely to impact the vast majority of us, vendor and enterprise developer alike.

    Disclaimer: I am a technologist, not a lawyer. I have done some research, and talked to lawyers, but all that I say here is opinion—even my direct quotes of lawyers are their opinion. The Supreme Court can and has caught people off guard and taken cases in different directions than expected. Though I believe this path is the one it will follow, nothing is assured until the Court rules.

    Throughout this blog, references to The Case are actually using Harvard’s case document.

     

    Background

    On Nov. 15, the U.S. Supreme Court agreed to hear a dispute between Oracle and Google. The Oracle v. Google case has been going on for nearly a decade. And its roots go further back than that.

    The origins of the case go back to 2005, when Google and Sun Microsystems discussed the licensing of Java for Android. This will likely be important because it shows that Google knew it was someone else’s intellectual property (IP). More on this later.

    The crux of the case falls into three parts:

    1. Eight files that were decompiled and used. In one court case, these were found to never have made it into distribution and only used for testing (which limits liability).
    2. Two source files from Java used directly. These were found to be infringing, but damages were set to zero with consent of both legal teams.
    3. The part at issue: the copying of the entirety of Java’s interface from 37 modules. Most tech writers imply this as 37 APIs. It is not—it is hundreds of classes and thousands of methods. The 37 is the number of Java packages.

    The case has been through four court hearings/rulings so far. To sum up the relevant parts of these cases:

    1. The first case (split between a jury and district court, running at the same time) included patent violations. Those were defeated and Oracle has not appealed that defeat. On copyright infringement, the court found that Google did violate copyright on the 37 packages and the directly copied source code, but not on the eight decompiled files. The jury could not settle the issue of “fair use” of the 37 packages.
    2. The district court decided that the eight decompiled files and the copied source code for one function were violations, but remanded the rest back to the jury trial to settle the issue of fair use.
    3. In the end, the district court determined that APIs are not protected by copyright. Their decision involved two parts:
      1. That there is only one way to write a given API.
      2. The interface code and hierarchy can be replicated by third parties without violating copyright because it is an interface and not content.
    4. On appeal, the finding was reversed, and APIs were deemed copyrightable, sending the case back again.
    5. At this point, Google asked the U.S. Supreme Court to intervene. The court chose not to.
    6. Back at jury trial, the court found that while APIs may be copyrightable, Google’s use was fair use in the legal sense. This is the one topic the jury could not originally agree upon.
    7. Oracle appealed this decision, and the District Court ruled that it indeed was not fair use to copy the structure and exact interfaces.
      1. Where the jury and the district court appear to disagree is in the idea of APIs. Calling a publicly documented API is obviously fair use, and the jury latched onto that idea. The District Court appears to acknowledge the difference between calling an API and copying it.
    8. And after some tumultuous hearings and years in the court, the Supreme Court has agreed to weigh in on the issue.

    The real problem here is that copyright law is not a great fit for software. It never has been. If I rewrite your Java application in Python, using the same exact structure and naming, is that protected? All of the “words” have changed. It is akin to rewriting your Hindi book in English without a direct translation  … except that in software there is the added issue of it behaving exactly the same.

    But we can’t fix the misapplication of copyright law to software—only Congress can. So what to expect?

    A fair number of people, from the open source community and from businesses that see opportunity in a ruling that favors their world  view, are advocating that affording copyright protection to APIs is a horrible idea because it would hurt independent developers. This is not quite true. Independent developers are not generally copying Java. While this case does have bearing on coding practices and what developers can do, it does not have a huge impact on innovation, just on copying others’ work and how that is achieved—much like the “reverse engineering” court cases of the ’80s.

    There are good arguments on both sides of this case, but it seems to me that fair use will lose. They are public APIs in the sense that anyone can call them; indeed, some of these packages are so core to the system that you could argue every app must call them. But calling a function is not the same as re-creating it. In that sense, they are not in the public domain. Anyone can read a book, but that does not make it public domain.

    But no matter how this turns out, it is all good for the industry. This uncertainty has hung over the IT sector for nearly a decade. Just like when a new boss comes along and there is uncertainty in the department until everyone adjusts to the new demands/style of the boss, a solid ruling will allow settling to start. The industry can adapt once it knows what the rules are and we can move along.

    The real long-term answer is to have an equivalent to copyright and patent that is designed around the unique parts of computer code. Clearly, software is not the same beast as a book or magazine. The considerations are half-patent and half-copyright, and trying to bend those two to meet the needs of software has created some serious issues. Oracle quoted a 1995 case to showcase this problem—the company is quoting it to win the appeal, but it should be quoted as the reason the protections need to be designed for this medium that is not a publication and yet consists of words whose sequence matters: “Applying copyright law to computer programs is like assembling a jigsaw puzzle whose pieces do not quite fit.” (attributed by Oracle to the concurring opinion of J. Boudin in Lotus Dev. Corp. v. Borland Int’l, Inc., 1995).

    One interesting bit that the courts thus far have ignored, and I would love to see the Supreme Court take up, is this: If Google entered into licensing negotiations, the company knew it was Sun/Oracle’s IP. To then go and copy parts of it without licensing appears wrong on its face. It would be interesting to understand the process that ended in the copying and hear whether that bears upon the legal standing of the case. I can think of both malicious and innocent routes to get there; it would be interesting to hear testimony about motivations and thought processes.

    But even without delving into the licensing negotiations, this case stands to redefine copyright with regards to software. J. Michael Keyes, a partner at the international law firm Dorsey & Whitney, had this (and more) to say:

    “One of the issues that the court may decide is the scope of copyright protection for software. Google had early success below in convincing the trial judge that the snippets of copied code were not copyrightable. Google may urge to have the court take a look at this issue anew. Another huge implication is the scope of fair use. The jury found that Google’s use was “fair” and that it was not liable for infringement. The appellate court overturned that jury conclusion and found that Google’s use was not fair as a matter of law. The Supreme Court will likely wade into this thicket and provide guidance on the scope of fair use in the context of software code and what role an appellate court plays in reviewing jury decisions in that regard.”

    What we (all of us) do not want to do is confuse “good for the industry” with “legal standard.” It is not the court’s job to do what we think is best for the industry; it is the court’s job to determine applicable law and what that implies needs to happen. Presumably, the Supreme Court will do just that. And it may not please any of us. But if we think the current iteration of the law is flawed, that is something to take up with elected representatives, not ask a court to “fix” on the fly.

    For the vast majority of us, though, it impacts our development not at all. Using APIs or forking open source projects are all still perfectly acceptable, and no ruling by the Supreme Court is going to change that—because copying someone’s APIs is the issue here, not using them. And open source licenses allow for copying and improving projects. So carry on. Get the job done. And read about the Supreme Court’s final decision with a smile, because either way, the industry will adapt to the new normal and continue forward.

    — Don Macvittie

  • IoT, AIOps and OMG!

    IoT, AIOps and OMG!

    The task of keeping networking straight while expanding the definition of network to include both physical and virtual networking has fallen heavily upon the op-centric side of DevOps–or what I like to call devOPS. What is defined as networking has grown, sometimes at a ridiculous rate. While the terms are familiar, how you manage them in truly virtualized environments is different than traditional hardware management. More agile, but different.

    And that means those who deal with networking in DevOps have to know more, manage more and debug more–all while working on other parts of the DevOps toolchain. It’s been a slog through increasing capability with increasing workload.

    There is a confluence of new and different things that promise to set this lopsided progress right. We’ll look at them individually, then talk about the sum of the parts.

    Internet of Things (IoT)

    For most of us, when we hear IoT, we think of millions of devices slamming data into a centralized service. iPhones, for example, returning information to Apple. Or highly specialized devices such as utility meters that report far more frequently than humans could go out and read them. But IoT is many things, and increasingly IoT is networking devices. They hold a wealth of information about the state of the network at any given moment, and that information is increasingly being funneled into APIs that IT or management apps can take advantage of. Considering that some of those management apps are also getting data from application tooling and server health status, there is a wealth of information coming together in one place.

    AI Operations (AIOps)

    A networking device can be doing a lot in any given second, and generating data as it works. Servers can be on their way offline when they feed the information out, or maybe sitting idle and wanting more work shipped their way. Application bottlenecks might be identified in the wealth of application tooling information, but buried under the wealth of other response data being shipped back. AIOps aims to help the devOPS side deal with these issues by limiting what makes it to human attention and drawing correlations between issues. Finding that an over-burdened network connection is associated with an apps performance degradation is an easy one to talk about, even though it is pretty simplistic. AIOps can make the connection before a human gets involved, and then the staff member can decide what to do about the actual root cause.

    It’s not at all perfect yet–we’re not generally talking about AIOps making a ton of decisions, rerouting traffic or spinning up new instances to resolve issues at this point–but it is definitely in the “useful tool” category. The cost in person-hours to analyze and reduce problems is definitely cut by the use of AIOps as it exists today, and that benefit will only grow moving forward.

    The biggest problem is the one that bedevils all data interchange, and in fact is the biggest issue in the world of big data–data standardization.

    Enter OMG

    No, my OMG in the headline wasn’t what you thought. It was the Object Management Group (OMG). They’ve been around forever, and have done a pretty good job of developing useful standards, particularly in the area of data interchange. I’ve followed them for a long time, and used their standards since … well, anyone remember CORBA? That was my first set of OMG standards.

    Now they’ve started an Artificial Intelligence Platform Task Force (AIPTF). This group will be looking at all areas of AI to set standards, but the interesting bits for this blog will be in data standardization and communication. Standardizing data and how it is communicated is the next big step in this process, and hopefully the OMG will set us down that path. They’re not specifically looking to address infrastructure big data needs, but their work will likely guide AIOps moving forward. And there is huge potential in that movement. If AIOps gets some standards that help it to be more proactive, we may actually end up with auto-fixing algorithms for obvious and easy issues. We already have the systemic ability in container managers to increase the number of instances serving an application if the existing ones get over-loaded, if we could expand that type of auto-growth to include networking issues and consider cores used by instance to find the best solution. Members of the team that are more devOPS will have more time to focus on new development. And that’s a huge plus.

    The Future is Bright

    Whether the work at OMG benefits us will depend upon whom is involved in the various subgroups they have planned to implement, and cooperation with other ongoing efforts at standardization. Yet I’m hopeful that, like with that original CORBA spec, they will provide a roadmap that others can follow to put us on the path to the much-maligned self-healing network. Giving devOPS a chance to keep up, while leaving them ultimately in control of what happens. Current iterations of AIOps do this to a lesser extent, but more inputs and more automated analysis of those inputs will only make AIOps more powerful. Standardization will make AIOps more able to make use of data and better focused on their specific problem at hand. I see the confluence of all three trends to be a huge positive to the industry if we manage to take advantage of it. And some vendor out there will see the potential, taking advantage of it for us, we just have to find them and use their solutions.

    Meanwhile, paw through the data available to you by hand again, and keep knocking those walls down.

    — Don Macvittie