Abstract: We are living in interesting times hardware-wise. CPUs, which are already made of a mix of general and specialized components, will soon have a reconfigurable portion as well; volatile memory won‘t be necessarily so for much longer; flash memory, which has been hidden—and slowed down—by layers of block-device-compatibility logic, is being addressed in increasingly direct ways; networks, which had a 10x boost from 1 to 10Gb not too long ago, have gotten another 10x boost to 100Gb, now challenging internal buses in terms of speed; and there are now new ways to lay interconnecting fabrics that all but blur the notion of where one computer ends and the next one starts.(*)
There was seldom a time like that in the industry when so many technologies reached their commercial debut at the same time.
And that means different tradeoffs and challenges for a systems researcher or practitioner interested in databases. The way in which each piece of datum is organized while in its resting state—if it is allowed to lay still at all—and the trajectory that it follows from there till it reaches a query result set can now be quite different than it used to be. We illustrate this by discussing two distinct use cases from classic database systems design: how to support fast (as in networking speeds fast) journalling and how to provide an elastic, distributed data structure that a query execution engine could be based on.
* Intel Xeon/Altera chip, Intel/Micron Optane, CNexLabs Host-based SSD FTL, Mellanox ConnectX-5, NVMoF/NTB PCIe/OpenCAPI, respectively.
Bio: Alberto Lerner is a consultant based in New York City, home of two very data hungry industries: finance and advertisement. He helps teams in those areas build proprietary data pipelines and storage systems. Before that, he has been part of the teams behind a few different database engines: IBM‘s DB2, working on robustness aspects of the query optimizer, Google‘s Bigtable, on elasticity aspects, and MongoDB, on general architecture. Alberto is formally trained in Computer Science and received his Ph.D. from ENST - Paris (now ParisTech), having done his research work at INRIA/Rocquencourt and NYU.