Our Troubles with Linux and Why You Should Care

Linux provides researchers with a full-fledged operating system that is widely used and open source. However, due to its complexity and rapid development, care should be exercised when using Linux for performance experiments, especially in systems research. The size and continual evolution of the Linux code-base makes it difficult to understand, and as a result, decipher and explain the reasons for performance improvements. In addition, the rapid kernel development cycle means that experimental results can be viewed as out of date, or meaningless, very quickly. We demonstrate that this viewpoint is incorrect because kernel changes can and have introduced both bugs and performance degradations.

This talk describes some of our experiences using the Linux kernel as a platform for conducting performance evaluations and some performance regressions we have found. Our results show, these performance regressions can be serious (e.g., repeating identical experiments results in large variability in results) and long lived despite having a large negative impact on performance (one problem appears to have existed for more than 3 years). Based on these experiences, we argue that it is often reasonable to use an older kernel version, experimental results need careful analysis to explain why a change in performance occurs, and publishing papers that validate prior research is essential.

This is joint work with Ashif Harji and Peter Buhr.

This talk will be about 20-25 minutes long with lots of time for questions and discussion afterwards.