Back in 2015 – following an executive and shortly before it would be deemed by the International Monetary Fund (IMF) – a small team of engineers in Deutsche Bank’s London office were tasked by their new management with transforming the bank into operating “everything-as-a-service.”
Now, three years on, those engineers have built Fabric, an internal platform-as-a-service (PaaS) that is already being used by thousands of Deutsche Bank employees to run thousands of applications, all with the aim of running 80 percent of workloads on Fabric by 2022. Built on top of Red Hat’s OpenShift PaaS, Fabric incorporates a slew of features specific to the highly regulated banking industry to accelerate application development and deployment.
It’s a rapid success story for a highly leveraged and highly regulated international bank – which is in the midst of a turnaround effort and that registered a loss of €5.7 billion ($7.4 billion) last year – and one that even has management considering whether Fabric is good enough to sell to rival banks to eventually turn its technology investments into a revenue stream.
A key problem Fabric helped solve was one that confronted the bank’s new leadership when it arrived in 2015: a sizeable virtual machine (VM) estate that was only being utilised at a rate of around eight percent.
“The CIOs got together and realised they had a problem to fix because this is just money that’s bleeding out to the organisation,” platform-as-a-service product owner at Deutsche Bank, Emma Williamson, said during a recent Red Hat OpenShift Commons event in London.
So the bank set out to drastically modernise its application estate around cloud native technologies like containers and Kubernetes, all with the aim of cutting this waste tied to its legacy platforms and help drive a broader shift towards the cloud.
to help put management’s minds at ease.
“That’s part of the value add that an enterprise platform gives you over and above having someone just go off and think they can do this on a [managed Kubernetes service] somewhere outside of any governance, because they’re not going to get particularly far,” Crawford said.
This is also why the bank integrated Avi Networks software – which was acquired by VMware in July last year – into the Fabric platform. Avi Networks provides automated services like IP whitelisting and network security rules to help ensure services are only being accessed by authorised personnel.
“Without Avi I don’t think the platform would have launched or been as successful, quickly, as it has been because it’s basically fast tracking that ability to get into production,” Crawford said.
Better utilisation, too much speed
Deutsche did go through some growing pains with the new platform. The bank went big on the training and community building aspects early, bringing in Red Hat to help with training and quickly standing up community forums and Symphony chats for internal troubleshooting to happen.
“We opened it up as a Symphony chat channel, which was supposed to be community help and users helping users like Stack Overflow, Slack, whatever,” Patel explained. “It turned into: ‘Hi Fabric team, I have this incident number, can you solve it for us?’ So we are trying to retrospectively go back and fix all of the habits that have appeared.”
Williamson and her team certainly struggled with the sheer level of adoption early on, as developers proved to be a demanding bunch.
“It feels like we weren’t able to give them enough, quickly enough,” she admits. “We’re a tiny team compared to the rest of the bank. I would have slowed down. I wouldn’t have done everything as fast, if I had my time again.”
“We launched this massive image of a Ferrari, but we had a little Ford Fiesta going: ‘Oh, my God, I don’t know if I can cope’,” she added.
Removing the training wheels
That being said, now that the training wheels are off for the developers, they are taking full advantage.
“The speed that they’re able to do things is vastly improved,” Williamson said. One team, for example, was able to get a new application into production in just three weeks, whereas Williamson said she has been in situations “where I’ve had to wait 14 months for a VM to come in.”
In terms of results the bank is now running 49 percent of applications on PaaS, according to Williamson. Could they ever get to 100 percent?
“No, we will never get to 100 percent,” she said. “But we don’t want to get to 100 percent. There’s always going to be applications that can’t go on Fabric. Applications that can’t go to cloud. Those platinum mainframe applications, rightfully so, because of the capabilities that they need. I think our stretch target is probably around 80 percent by the end of 2022 and I think we can do it.”
Still, the learning curve is steep when it comes to cloud native technology like containers, with teams “that haven’t been Kubernetes or OpenShift users and in a lot of cases are not even out and out developers but support staff tasked to deploy applications onto a developer-centric platform. That’s our clientele, right? We have to work with that,” Patel said.
Documentation is a good start, but of course “as engineers none of us read documentation as it is, even if it’s step by step, they don’t follow it.” he admits.
Next stop: the cloud
Currently Fabric is operating across four regions: US, UK, APAC, and Germany on a variety of infrastructure from three unnamed on-premises providers and Microsoft Azure public cloud, but with a private network. The aim is to move as much to the cloud as possible in the future.
The issue with the old school, on-premises providers hinges on paperwork. “The contracts are a problem. We’ve got reduced SLAs; they certainly can’t compete with [those of] the cloud providers. [Then there’s] the inconsistent configuration, because if you don’t have control over the base image like you do in the cloud, you don’t own the full stack and have problems,” Crawford explained.
No matter what infrastructure Fabric resides on, Crawford and his team have essentially become an internal PaaS provider, with all of the pressure that brings.
“This is why people love the platform – we take away all the pain, and we have to deal with all the subtleties and the differences between the different providers and then you just consume the platform,” Crawford said. “Unfortunately, that does make it a little bit difficult to operate in certain cases.”
The transition to service provider has incurred some major changes to the platform team, bringing in new engineering, operations, product management, professional services, program management, and site reliability expertise.
“We had three guys when we started this, so we built a whole brand new team, whole new way of working, new people, new thoughts,” Patel said.
Finally, Deutsche appears to be serious about the possibility of commercialising its Fabric platform for other financial services companies to procure it as a service.
This follows a broader trend in the industry of banks looking to monetise their investment in cloud native technology, essentially becoming software vendors themselves. This can be seen at Goldman Sachs, which is to sell to other financial services firms, and the UK fintech OakNorth, which for other banks to offer credit decisioning services.
Patel talked about how Deutsche Bank has thought about Fabric as a standalone product and brand from the start, so that “we could potentially, one day, maybe, if everything aligned properly, sell to other [financial services companies]. We could probably package this up and sell it as we’ve got it now, today,” he said.
“You have to aim high and that’s the idea,” Crawford added. “We want to manage this as a product and it needs to be, as a consequence, mature in terms of documentation and defining those boundaries. How you consume it, making it easy to consume, is a big, big push as well.”
Copyright © 2020 IDG Communications, Inc.