Based in San Mateo, California, Rakuten Rewards is a shopping rewards company that makes money through affiliate marketing links across the web. In return, members earn reward points every time they make a purchase through a partner retailer and get cash back rewards.

Naturally this drives a lot of user insight data – hundreds of terabytes on active recall with more in cold storage, to be exact.

[  ]

In 2018 the business started to get serious about giving more users access to this insight – without having Python or Scala coding chops – while also reducing its capital expenditure on hardware, and started looking to the cloud.

‘SQL server machines don’t scale elegantly’

Formerly known as Ebates, the business was acquired in 2014 by the Japanese e-commerce giant Rakuten, and has been growing fast since, forcing a drive to modernize its technology stack and become more data-driven in the way it attracts and retains customers.

This starts with the architecture. In the past three years Rakuten Rewards has moved its big data estate from largely on-prem SQL to on-prem Hadoop to, today, a cloud data warehouse courtesy of .

“SQL server machines don’t scale elegantly, so we went on-premises Hadoop with Cloudera, using Spark and Python to run ETL, and got some performance out of that,” VP for analytics at Rakuten Rewards, Mark Stange-Tregear, told InfoWorld.

, the world has moved on from Hadoop after the halcyon days of the early 2010’s.


By leveraging cloud computing, Rakuten is better able to scale up and down for peak shopping times. Snowflake also allows the company to split its data lake into a series of different warehouses of different shapes and sizes to meet the requirements of different teams, even spinning up new ones for one-off projects as required, without teams competing for memory or CPU capacity on a single cluster.

before them, Rakuten Rewards milked as much value out of its Hadoop investment as possible, but when an easier way to maintain that platform emerged – while enabling a much wider range of users to benefit – the rewards far outweighed the costs.

Copyright © 2020 IDG Communications, Inc.