mmap and the Mortgage Crisis

In the aftermath of the 2008 economic crisis, large banks have been saddled with the prospect of foreclosing on millions of distressed mortgages, at a financial cost of billions of dollars and an incalculable social cost. Crucial to solving this problem is the ability to model and analyze these millions of loans in real time, enabling lenders to price homes so that they can find effective and mutually beneficial alternatives to foreclosure.

In this talk, we'll describe how engineers at Palantir are working on a calculation engine that supports such analyses. We'll outline our design goals of constructing a platform that supports queries against large sets of data at interactive speeds and exposes a high-level object-oriented interface that enables analysts to construct models intuitively without having to worry about the underlying implementation details. We'll describe the different architectures we explored in prototyping the system, demo how to use our product to analyze massive datasets, and discuss how we've ultimately deployed it in the field.