Having been on holiday there has been a bit of an outburst of discussion around new licenses for open source, or less open source code. But in many ways the more interesting piece, that puts it in context was Joseph Jacks’ seventh Open Consensus piece, “OSS Will Eat Cloud Computing”. The Redis arguments about changing some code from the anti cloud provider AGPL to a no commercial exploitation license were only aimed at the cloud providers making open source code available as a service while often not contributing back significantly, and not providing any revenue to upstream open source companies.

But if OSS is going to eat up cloud, and the commercial open source companies are also going to be increasingly succesful, then this starts to seem like just a teething issue, soon to be forgotten. So let us look what the optimistic case looks like. First though I think it helps to look at the open source strategies of the cloud providers themselves, to give some context to the current position.

Open source and the cloud providers

AWS does not produce a lot of open source software of any great interest itself. It has started to show some interest in strategic moves, such as backing the nodeless Kubernetes work but overall, nothing of interest. Plenty of open source to help you use its services of course. On the other hand it is the main co-opter that Redis Labs complains about, shipping often slightly old and modified versions of open source software. There are many examples, from the widely used MySQL service to many other projects, such as Redis sold as “Amazon ElasticCache for Redis”, which is “Redis compatible” but does not confirm or deny being upstream Redis. Another example is Amazon ECS, which bundled open source Docker with an orchestration layer quite different from the way that the open source community was going, as a proprietary way to run Docker that has ultimately been unsuccesful.

Azure produces the largest quantity of open source of any of the three major US based cloud providers, with much of the Azure infrastructure actually being open source, including Service Fabric, the entirety of the Function as a service platform and so on. However the sheer quantity of open source coming out of Microsoft means that it needs curation and most of it has not created a deep community. They urgently need to prioritise a smaller number of core projects, get community contribution or shut them down if this fails, and get them into the CNCF to counteract the Google centric nature of the projects there. With careful management Azure could be very well positioned for an OSS eats the cloud future but it needs the community behind it, or failing that to follow the community instead.

Google is the smallest of the cloud players but has so far used open source the most strategically, and has shown that a large investment, such as that in Kubernetes, can push developer mindshare in the direction you want, and gain mass adoption. This has been combined with other succesful projects, such as Go, which has been one of the most succesful programming language launches of recent times. However it is not clear to me that this success can necessarily be replicated on an ongoing basis, and there are opportunities for other players to disrupt the strategic play. First Google demands extreme control over its projects, and many of its “open source” projects are just code over the wall plays, with no interest in real community involvement. Others offer community involvement very late once the direction is already strongly defined, as is clear in the decision to not donate Istio to the CNCF until well post the 1.0 release. There is a whole strategic roadmap being mapped out, pushing the Google way of doing things, and I predict that much of it will not stick in the end. Not every project is going to be in the right place at the right time that Kubernetes was. Another issue is the suite of “GIFFE” (Google infrastructure for everyone else) companies that ex-Googlers like to start up and Google Ventures likes to fund, which further spread the Google way, have a problem. The main issue is that Google already has an internal project that matches these, so in many cases they do not have any interest in actually buying a company with an open source reimplementation. So there is no real process for an exit to Google, unlike the classic Cisco spin out and then purchase model for startups where the viable companies get bought up by the original company again. The biggest exit in this space has been CoreOS that was purchased to remove a competitor in the Kubernetes market; the Linux distribution it started with added no value to the transaction.

The other impact of all three cloud providers that is important is the hiring. Many engineers who might otherwise be working on other open source projects are being hired by the cloud providers to work on their projects, which are largely not open source. The rapidly growing revenues and high margins mean that hiring by the three is very significant, and both Amazon and Microsoft have profits and stock prices (and hence equity compensation) that are now largely being driven by the growth in cloud. Google is still largely an advertising company, and Google Cloud is very small compared to the other two, so there is less of a direct multiplier there. This adds to pressure on salaries in the rest of the industry and shifts people to working on the cloud providers open and closed source strategic directions.

What the near term would look like

If open source is to eat cloud, cloud has to become the commodity layer. We have an similar recent model in the mobile phone providers in the 1990s. Suddenly there was a huge opportunity for telecom companies who were in a mature low margin business, plus upstart businesses to enter a high growth high margin business. Huge profits were made, new global giant companies such as Vodafone were created, and well in the end mobile became just a commodity business to run the (open source driven) internet over. Margins continue to fall, no new value was captured by the network owners despite many attempts. The details of how this failed are not that relevant perhaps; the important thing is that trillions of dollars of value capture that was hoped for, even expected, did not in the end materialize. The key is the “dumb pipes” phrase that telcos worried about so much.

Dumb cloud pipes

The route to dump cloud pipes involves a fair number of forces converging. First the explosive growth in cloud as a whole, as happened in mobile, removes much price pressure, while there is an explosion of services (especially at AWS that is constantly launching them). With the huge demand, there is initially little price sensitivity, and there is an exploration of services while people discover which are most useful. Pricing is opaque and users do not initially realise exactly what they are going to consume or how much value it has. This is our current phase, with cloud provider margins being very high. Counteracting this comes customer maturity, such as better budget control, standardised feature sets, and better price negotiation. Prices could easily start to fall again, at a rate of 20%-30% a year or higher for long periods. The clouds will try to build moats and lock in at this point, building on the current points of lock in. These especially include IAM, where models and APIs differ substantially between providers, hence the build out of deeper security related services such as cloud HSM and other areas that deepen use of these APIs. Data gravity is another moat, and some people have suggested that data storage might end up being subsidised until it is free, anything to get more data in; transit costs dominate for many use cases anyway, and highly discourage cross cloud use cases. Cloud provider network costs are deliberately high.

In general, like the old saying about the internet, that it sees censorship as something to route around, open source tends to see lock in and moats as something to fill in. We already have a situation where the S3 API (the oldest part of AWS) is the standard for all providers, and has open source implementations such as Minio. Managed Kubernetes is another area where all the providers are being forced by the market to provide a standard interface. Pure compute is not so standardised but is undifferentiated. The next thing we see coming are higher level interfaces over whole classes of API; one example of the type of approach is Pulumi that provides a very different, programming language focused rather than API focused, but designed to work across arbitrary clouds without caring. Note that some of the Google open source efforts promote these type of changes, in order to try to make their cloud more interchangeable with AWS in particular, but they also have a large amount of proprietary technology that they are using to attempt moat building at the same time.

Community of purpose

There are some open source companies already working in this space, including my employer Docker and several other large scale companies, as well as the wealth and diversity of smaller, growing companies that make up the current community. As Joseph points out in his post, these commerical open source companies are growing very rapidly but this is being largely ignored as cloud is more obvious. There is plenty more room of course, and as customers gradually realise that the cloud provision is a dumb pipe and the open source software they run on top is where the actual value is they will want to get it from the real providers, and engage directly with the communities to contribute and pay for changes and support.

Ultimately it is the end customers who swing the situation, realise that pipes and hardware are just utility, and the people there they like we have seen elsewhere continue to move towards and engage with open communities, open source communities, and demand that their organizations do fully engage too. So far we have seen that every enterprise has engaged in the consumption of open source software, but its value is still only tangentially embedded in the majority of organisations. A decade ago we used to sell open source software because people would decide they would have one open source vendor in a pitch, and we would win it. Soon there will be whole areas of software, especially infrastructure, where closed source, including closed source cloud services just won’t be viable

Countering the rosy view that the experience of open source as a better development model will inevitably lead to growth in understanding and use of open source, what if people just like free or cheap and convenient, and cloud delivered proprietary services are good enough? Essentially though that is just an argument that cloud providers are the best at producing convenient interfaces; historically that has been true as it is their business, but it is not an exclusive ability, just one that needs working on. As Sophie Haskins points out, open source companies have often undervalued the work on actually making their code deployable and maintainable in production, which the cloud providers have done instead, in a closed way instead. Taking back ownership of this clearly will help.

Overall the question is will open communities simply fold over in the path of cloud provision, or will they route around blockages to open innovation and co-opt the infrastructure for new purposes and tools. It is hard not to be optimistic given the current rate of innovation.