Performance and Measurement part 2

Measurement

last time we looked at

Producing Wrong Data Without Doing Anything Obviously Wrong! Todd Mytkowicz, Amer Diwan, Matthias Hauswirth, and Peter F. Sweeney. ASPLOS 2009.

445 references

  1. Measurement bias is significant
  2. Changing aspects of an experimental setup can introduce measurement bias. ​ Measurement bias is unpredictable and there are no obvious ways to avoid it. ​ Prior work in computer system evaluation does not adequately consider measurement bias. ​
  3. The paper discusses two techniques for dealing with measurement bias: experimental setup randomization and causal analysis. ​
  4. Measurement bias occurs for all benchmarks and architectures. ​
  5. Measurement bias due to link order can significantly fluctuate conclusions. ​
  6. Measurement bias due to UNIX environment size can lead to conflicting conclusions. ​
  7. To avoid measurement bias, it is important to use diverse evaluation workloads, randomize the experimental setup, conduct causal analysis, and collect more information from hardware manufacturers. ​ —

A sample blog post about this paper blog

another example

Strangely, Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data!

SIGPLAN Empirical Evaluation Guidelines

checklist

Here are the criteria by Berger, Blackburn, Hauswirth, and Hicks (2018):

  1. Clearly stated claims
    1. Explicit Claims
    2. Appropriately-Scoped Claims
    3. Acknowledges Limitations
  2. Suitable Comparison
    1. Appropriate Baseline for Comparison
    2. Fair Comparison
  3. Principled Benchmark Choice
    1. Appropriate Suite
    2. Non-Standard Suite(s) Justified
    3. Applications, Not (Just) Kernels
  4. Adequate Data Analysis
    1. Sufficient Number of Trials
    2. Appropriate Summary Statistics
    3. Report Data Distribution
  5. Relevant Metrics
    1. Direct or Appropriate Proxy Metric
    2. Measures All Important Effects
  6. Appropriate and Clear Experimental Design
    1. Sufficient Information to Repeat
    2. Reasonable Platform
    3. Explores Key Design Parameters
    4. Open Loop in Workload Generator
    5. Cross-Validation Where Needed
  7. Presentation of Results
    1. Comprehensive Summary Results
    2. Axes Include Zero
    3. Ratios Plotted Correctly
    4. Appropriate Level of Precision