start results chapter

1 files changed, 84 insertions, 6 deletions

diff --git a/thesis/parts/results.tex b/thesis/parts/results.tex
index 47c7c98..17b6088 100644
--- a/thesis/parts/results.tex
+++ b/thesis/parts/results.tex
@@ -1,9 +1,87 @@
-\todo{Selection of benchmarks}
-\todo{Testing setup}
-\todo{Justification of tested elements}
+%% * Testing setup, benchmarking rationale
+\section{Testing setup}
 
-\todo{Produced cost models}
+%% ** Specs and VM setup
+In order to ensure consistent results and reduce the chance of outliers, all benchmarks were run on a KVM virtual machine on server hardware.
+We used 4 cores of an Intel Xeon E5-2687Wv4 CPU, and 4GiB of RAM.
 
-\todo{Estimated costs}
+%% ** Reproducibility
+The VM was managed and provisioned using NixOS, meaning it can be easily reproduced with the exact software we used.
 
-\todo{Real benchmark results}
+\section{Cost models}
+
+We start by looking at our generated cost models, and comparing them both to the observations they are based on, and what we expect from asymptotic analysis.
+As we build a total of 51 cost models from our library, we will not examine all of them.
+We look at ones for the most common operations, and group them by containers that are commonly selected together.
+
+%% ** Insertion operations
+Starting with the \code{insert} operation, Figure \ref{fig:cm_insert} shows how the estimated cost changes with the size of the container.
+To help readability, we group these into regular \code{Container} implementations, and our associative key-value \code{Mapping} implementations.
+
+\begin{figure}[h]
+  \centering
+  \includegraphics[width=10cm]{assets/insert_containers.png}
+  \par\centering\rule{11cm}{0.5pt}
+  \includegraphics[width=10cm]{assets/insert_mappings.png}
+  \caption{Estimated cost of insert operation on \code{Container} implementations and \code{Mapping} implementations}
+  \label{fig:cm_insert}
+\end{figure}
+
+%% ** Contains operations
+
+%% ** Comment on some bad/weird ones
+
+%% ** Conclusion
+
+%% * Predictions
+\section{Selections}
+
+%% ** Chosen benchmarks
+Our test cases broadly fall into two categories: Example cases, which just repeat a few operations many times, and our 'real' cases, which are implementations of common algorithms and solutions to programming puzles.
+We expect the results from our example cases to be relatively unsurprising, while our real cases are more complex and harder to predict.
+
+Most of our real cases are solutions to puzzles from Advent of Code\parencite{wastl_advent_2015}, a popular collection of programming puzzles.
+Table \ref{table:test_cases} lists and briefly describes our test cases.
+
+\begin{table}[h]
+  \centering
+  \begin{tabular}{|c|c|}
+    Name & Description \\
+    \hline
+    example\_sets & Repeated insert and contains on a set. \\
+    example\_stack & Repeated push and pop from a stack. \\
+    example\_mapping & Repeated insert and get from a mapping. \\
+    prime\_sieve & Sieve of eratosthenes algorithm. \\
+    aoc\_2021\_09 & Flood-fill like algorithm (Advent of Code 2021, Day 9) \\
+    aoc\_2022\_08 & Simple 2D raycasting (AoC 2022, Day 8) \\
+    aoc\_2022\_09 & Simple 2D soft-body simulation (AoC 2022, Day 9) \\
+    aoc\_2022\_14 & Simple 2D particle simulation (AoC 2022, Day 14) \\
+  \end{tabular}
+
+  \caption{Our test applications}
+  \label{table:test_cases}
+\end{table}
+
+%% ** Effect of selection on benchmarks (spread in execution time)
+
+%% ** Summarise predicted versus actual
+
+%% ** Evaluate performance
+
+%% ** Comment on distribution of best implementation
+
+%% ** Surprising ones / Explain failures
+
+%% * Performance of adaptive containers
+\section{Adaptive containers}
+
+%% ** Find where adaptive containers get suggested
+
+%% ** Comment on relative performance speedup
+
+%% ** Suggest future improvements?
+
+%% * Selection time / developer experience
+\section{Selection time}
+
+%% ** Mention speedup versus naive brute force