Boosting science and engineering in the cloud


It has become cliché to talk about “developers as the new kingmakers,” a point popularized by the analyst firm Redmonk. But the point of gifting power (through infrastructure) to developers was never really about developers. It was about changing how enterprises build their products.

And, ultimately, about who builds those products. Yes, developers are central, but as can be surmised from a recent , with Microsoft, Nvidia, and Samsung all contributing, it’s a good time to inspect what others are building on the platforms developers have built. Specifically, engineers and scientists are among the big beneficiaries, and are dramatically accelerating scientific evolution as a result.

Fueling PhD productivity

But first, back to those developers. With every business scrambling to become a software company, developers are the bright, shiny objects that every enterprise  treasures. No longer cost centers that need to be outsourced, development teams are now seen as top-line revenue generators powering the business. Leadership has learned to invest in developers as growth drivers. The rise of opex cloud computing has fueled this trend.

But developers were never the end goal, per se. Developers were simply paving the way so that others could contribute more fully to enterprise productivity. Specifically, it’s time for engineers and scientists to grab some of that developer spotlight. These highly paid specialists (typically PhDs) have grown accustomed to having to queue to run massive workloads on bespoke on-premises hardware, which costs a fortune and slows the pace of inquiry. Then it’s back in line to iterate what they learned from the last job.

That was then, this is now.

Increasingly these same engineers and scientists can turn to the cloud and accelerate the iterations of their workloads that simulate real-world conditions to get more innovative products to market much faster. It’s the principle first :

(with an API for whatever back-end function you need), , , and all these other spreading across the enterprise software stack free developers to focus on the logic of the new application or service they are developing. They can forget about the infrastructure. Time-to-market cycles speed up. More and better services delivered much faster leads to happier, stickier customers. And more top-line revenue.

, they collectively spent $97 billion on capex last year. That’s not all for their cloud businesses, of course, but a considerable chunk of it serves that market. As Fitzgerald says, this money is “bonkers” big.

Charles Fitzgerald

But size only matters inasmuch as it conveys real benefits on other industries, which, it turns out, it does. Suddenly it’s easier to understand why private industry startups can hurl rockets into space after mere years of development and prototyping (hundreds of space startups were funded in the past five years alone) when it took decades to create Saturn V or the Space Shuttle at orders of magnitude more expense. In aerospace today, the United States’ primary strategic bomber capability, the B52, flies with an airframe that was designed by engineers with slide rulers back in 1948.

The priests in the HPC temple want in on this action now. You can’t do hard digital research and development (rockets, genomics, smart cities, etc.) without smart scientists and engineers. These very smart people watched their developer friends in the enterprise free themselves from the shackles of infrastructure. Now they want their own intelligent software-defined computing connected to fellow researchers, scientists, and engineers for collaboration. Sounds like a cloud use case to me.

HPC cloud brokers like Rescale sit in a unique vantage point for these scientists. They know what job you ran, what software you use, what version under what licensing terms, what interconnect, what data you put in, and how long the job took. They also know exactly how much it cost — and can tell you how much it will cost for all your other jobs. At the same time, they can arbitrage the best price-performance for your workload across their network of partner cloud providers.

They solve a very hard problem for customers solving the world’s hardest problems. Customers can also ask, “What is the best software for the job I want to run?” That is a compelling value proposition for algorithmically-driven workloads and widely diverse use cases.

Best of all? No more PhDs waiting in line to run their jobs.

In the world where engineers and scientists live today — solving mind-boggling challenges of extreme complexity — they are starving for change. They want to solve problems, not understand the complex infrastructure that makes a legacy data center look like a child’s Lego house by comparison. Rescale and its peers promise to give the PhDs full-stack HPC economics/performance optimization, and deliver on that promise continuously.

In sum, there are things that developers increasingly take for granted, like continuous integration and continuous delivery (CI/CD). For HPC engineers and scientists, however, such things are relatively new — and somewhat miraculous. In the hands of the PhDs, they’ll undoubtedly lead to miraculous things.

Copyright © 2021 IDG Communications, Inc.