notes on how to reproduce

7 files changed, 112 insertions, 167 deletions

diff --git a/README.md b/README.md
index 4bac41a..fee8ce5 100644
--- a/README.md
+++ b/README.md
@@ -1,7 +1,29 @@
-# dissertation
+# Candelabra
 
-## requirements
+These are the supplementary materials for my thesis, Candelabra.
 
-```
-$ raco pkg install rosette
-```
+  - `src/` is a Rust workspace containing the actual source code
+  - `analysis/` contains the benchmark results, and the Elixir livebook notebook used to process and visualise the data.
+  - `thesis/` contains the LaTeX code used to build the actual thesis, as well as VegaLite JSON files for graphs
+
+Check the `README` in each directory for more information.
+
+## Getting dependencies
+
+The easiest way to install the required dependencies is using Nix.
+After [installing it](https://nixos.org/download/), simply run `nix develop`.
+This environment includes everything necessary for building the program, visualisations, and thesis.
+
+You can also build the program itself with `nix build .#default`.
+
+Alternatively, manually set up the following programs:
+
+  - Rust nightly toolchain 2024-01-22
+  - Racket v8.10, with the `rosette` package installed - `raco pkg install rosette`. You may also need to manually install `z3`.
+  - [Just](https://crates.io/crates/just) 1.24.0
+  - To generate visualisations:
+    - [Livebook](https://livebook.dev/) 0.11.3
+    - [vega-cli](https://www.npmjs.com/package/vega-cli 5.26.0
+  - To build the thesis:
+    - Texlive, with the packages listed in `nix/tex-env.nix`
+    - `latexmk`
diff --git a/analysis/Justfile b/analysis/Justfile
index 8200fed..46cce51 100644
--- a/analysis/Justfile
+++ b/analysis/Justfile
@@ -1,21 +1,18 @@
 default:
   livebook server
 
-fetch-log job outp="current":
-  mkdir -p {{justfile_directory()}}/{{outp}}
-  curl https://git.tardisproject.uk/tcmal/dissertation/-/jobs/{{job}}/raw >> {{justfile_directory()}}/{{outp}}/log
+split-log job outp="current":
   ./split-log.py {{justfile_directory()}}/{{outp}}/log
-  grep -R -H -i 'error' {{justfile_directory()}}/{{outp}}/sections/
 
 fetch-cost-models outp="current":
   mkdir -p {{justfile_directory()}}/{{outp}}
   rm -fr {{justfile_directory()}}/{{outp}}/candelabra
-  rsync -ravP candelabra:/opt/candelabra/src/target/candelabra/ {{justfile_directory()}}/{{outp}}/candelabra
+  rsync -ravP ../src/target/candelabra/ {{justfile_directory()}}/{{outp}}/candelabra
 
 fetch-comparisons outp="current":
   mkdir -p {{justfile_directory()}}/{{outp}}
   rm -fr {{justfile_directory()}}/{{outp}}/criterion
-  rsync -ravP candelabra:/opt/candelabra/src/tests/target/criterion/ {{justfile_directory()}}/{{outp}}/criterion
+  rsync -ravP ../src/tests/target/criterion/ {{justfile_directory()}}/{{outp}}/criterion
   rm -rf {{justfile_directory()}}/{{outp}}/criterion/report
   find {{justfile_directory()}}/{{outp}}/criterion -iname '*.svg' -exec rm '{}' \;
   find {{justfile_directory()}}/{{outp}}/criterion -iname '*.html' -exec rm '{}' \;
diff --git a/analysis/README.md b/analysis/README.md
new file mode 100644
index 0000000..c2311e5
--- /dev/null
+++ b/analysis/README.md
@@ -0,0 +1,22 @@
+# Analysis
+
+This folder contains our raw data from testing Candelabra, and the notebook we used to create visualisations used in the thesis.
+
+To run the notebook, start a [livebook](https://livebook.dev) server with `livebook server`, and open `vis.livemd` using it.
+
+## Using new data
+
+To use new data, first generate some new data locally. In `../src/`:
+
+```
+$ just cost-models # approx 10m
+$ just comparisons 2>&1 | tee ../analysis/current/log # approx 1hr 30m
+```
+
+We need to also write the log output when comparing since we use this during analysis. To put this data in the right place, in `../analysis`:
+
+```
+$ just split-log
+$ just fetch-cost-models
+$ just fetch-comparisons
+```
diff --git a/analysis/package.json b/analysis/package.json
deleted file mode 100644
index 880c06c..0000000
--- a/analysis/package.json
+++ /dev/null
@@ -1,11 +0,0 @@
-{
-  "name": "analysis",
-  "version": "1.0.0",
-  "description": "",
-  "main": "index.js",
-  "scripts": {
-    "test": "echo \"Error: no test specified\" && exit 1"
-  },
-  "author": "",
-  "license": "ISC"
-}
diff --git a/src/Justfile b/src/Justfile
index 0d20a5e..6ed018f 100644
--- a/src/Justfile
+++ b/src/Justfile
@@ -3,8 +3,6 @@ target_dir := justfile_directory() / "target"
 tests_dir := justfile_directory() / "tests"
 tests_manifest := tests_dir / "Cargo.toml"
 
-impls := "std::collections::BTreeSet primrose_library::LazySortedVec std::collections::LinkedList primrose_library::EagerUniqueVec std::collections::HashSet std::collections::BTreeMap std::collections::HashMap primrose_library::LazyUniqueVec primrose_library::EagerSortedVec std::vec::Vec"
-proj := "std::vec::Vec std::collections::LinkedList"
 export RUST_LOG := "debug"
 
 cost-models: rebuild
@@ -24,6 +22,9 @@ cost-model impl: (section-start ("cost-model-" + impl)) && (section-end ("cost-m
 select proj *FLAGS: (section-start ("compare-" + proj)) && (section-end ("compare-" + proj))
   {{candelabra}} -l --manifest-path {{tests_manifest}} -p {{proj}} select {{FLAGS}}
 
+profile proj *FLAGS:
+  {{candelabra}} --manifest-path {{tests_manifest}} -p {{proj}} select {{FLAGS}}
+
 @section-start NAME:
   echo -e "\e[0Ksection_start:`date +%s`:{{replace(NAME, ":", "-")}}\r\e[0K{{replace(NAME, ":", "-")}}"
 
diff --git a/src/README.md b/src/README.md
index 604468b..a2691a3 100644
--- a/src/README.md
+++ b/src/README.md
@@ -1,143 +1,51 @@
-# Getting Start
-
-This is the research artifact for the paper *"Primrose: Selecting Container Data Types by their Properties"*.
-
-For easy of evaluation, the artifact is provided as a VirtualBox virtual machine with all dependencies already pre-installed.
-Below, we provide a guide on how to check that the claims made in the paper are supported by the artifact.
-We encourage the evaluators to experiment themselves with the provided examples.
-
-
-## Logging into the VM and locating the artifact
-- user name: `user`
-- password: `helloworld`
-- artifact location: `/home/user/Documents/PrimroseAE/`
-
-# Overview of the artifact
-This artifact supports the claims made in the paper by containing:
-- The property specifications introduced in section 4 of the paper.
-- An implementation of a parser and type checker of the property specifications accompanied by a type-inference algorithm.
-- An implementation of the container library studied in section 5 of the paper.
-- Library specifications for each container discussed in section 5.
-- An implementation of the selection process described in section 6 of the paper.
-- An implementation of the code generation described in section 6.4 of the paper.
-- Property based testing as described in the section 7.1 of the paper. 
-- A script for measuring the solver times reported in section 7.2 of the paper. We do expect that runtimes measured in the virtual machine might differ from the numbers reported in the paper that have been measured outside the virtual machine. 
-
-# Step-by-Step Instructions
-## Walkthrough code related to features introduced in the paper
-- The *syntactic properties* introduced in section 4.1 can be found in `./src/traits/mod.rs`
-- The *semantic properties* introduced in section 4.2, can be found in:
-  - the `unique` property specification can be found in `./spec_code/example_unique.rs`
-  - the `ascending` property specification as well as the composition of `ascending` and `unique` can be found in `./spec_code/example_comp.rs`.
-- The property specification of the stack example introduced in section 4.3 can be found in `./spec_code/example_stack.rs`
-- The combinators used in property specifications are provided in `./racket_specs/combinators.rkt`
-- All library specifications introduced in section 5 can be found in `./src/library/`
-- All property based tests introduced in section 7 can also be found in `./src/library/`
-
-## Selecting of valid container implementations and Rust code generation via the `primrose` tool
-- To run the *unique container example* from the paper:
-  - Make sure you are in the directory `PrimroseAE`
-  - Run command: `cargo run example_unique.rs unique 3`
-  - Generated code can be found in the directory `./gen_code/unique/`
-  - For this example we expect that four files each with a different container implementation are generated:
-    - `./gen_code/unique/unique0.rs`
-    - `./gen_code/unique/unique1.rs`
-    - `./gen_code/unique/unique2.rs`
-    - `./gen_code/unique/unique3.rs`
-  - To compile the generated Rust code, go to `Cargo.toml`, add at the end for the code you want to compile:
-    ```
-    [[bin]]
-    name = "unique0"
-    path = "gen_code/unique/unique0.rs"
-
-    [[bin]]
-    name = "unique1"
-    path = "gen_code/unique/unique1.rs"
-
-    [[bin]]
-    name = "unique2"
-    path = "gen_code/unique/unique2.rs"
-
-    [[bin]]
-    name = "unique2"
-    path = "gen_code/unique/unique3.rs"
-    ```
-  - Then you can compile and execute the generated file with:
-    - `cargo run --bin unique0`
-    - `cargo run --bin unique1`
-    - `cargo run --bin unique2`
-    - `cargo run --bin unique3`
-
-- To run the *unique and ascending (strictly ascending) container example* from the paper:
-  - Make sure you are in the directory `PrimroseAE`
-  - Run command: `cargo run example_comp.rs comp 3`
-  - Generated code can be found under the directory `./gen_code/comp/`
-  - To compile the generated Rust code, add it to `Cargo.toml` as above and then execute it via `cargo run`
-- To run the *stack example* from the paper:
-  - Make sure you are in the directory `PrimroseAE`
-  - Run command: `cargo run example_stack.rs stack 3`
-  - Generated code can be found under the directory `./gen_code/stack/`
-    - To compile the generated Rust code, add it to `Cargo.toml` as above and then execute it via `cargo run`
-
-## Running property based testing from section 7.1
-- Make sure you are in the directory `PrimroseAE`
-- Run command: `cargo test`
-- if you want to measure how long it takes to execute all tests: `./timetests.sh`
-
-## Producing solver efficiency benchmarks form section 7.2
-- Make sure you are in the directory `PrimroseAE`
-- Run command: `./timeall.sh`
-- **Please note:** we do not expect that the times measured inside the virtual machine and on different hardware will be exactly the same as the times presented in the paper.
-
-# Technical Overview of the Artifact
-
-## Overview of pre-installed dependencies
-- Rust 1.67.0-nightly
-- Racket 8.1 or later
-- Rosette
-  - We used the Z3 backend in this project
-- These dependencies are all pre-installed, to check they are installed correctly:
-  - Type command: `rustc --version` in terminal, you should get:
-    ```
-    rustc 1.67.0-nightly (01fbc6ae7 2022-12-07)
-    ```
-  - Type command: `racket --version` in terminal, you should get:
-    ```
-    Welcome to Racket v8.6 [cs].
-
-## Execution of the `primrose` tool with arbitrary property specification
-- Make sure you are in the `PrimroseAE` directory
-- Make sure the Rust program with embedded property specifications (`input.rs`) is provided in the directory `./spec_code/`
-- Run the tool with command:
-  ```
-  cargo run [input.rs] [output_dir_name] [model_size]
-  ```
-- For most properties, we recommend a model size of `3`
-- The generated file will appear in the directory `[output_dir_name]` and can be compiled with `cargo` after an entry for it has been added at the end of `Cargo.toml` file:
-    ```
-    [[bin]]
-    name = "name_you_like"
-    path = "path/to/file.rs"
-    ```
-- To execute the generated Rust code run:
-  ```
-  cargo run --bin name_you_like
-  ```
-
-## File structure of the `PrimroseAE` directory
-- `./benches/`: containing the code producing the runtime performance benchmarks in section 2
--  `./gen_code/`: containing generated code with selected container implementations
-- `./mem_profiling/`: containing scripts producing the memory consumption benchmarks in section 2
-- `./racket_specs/`: containing scripts for setting up and executing the selection process and generated code during the selection process
-- `./scripts/`: containing the scripts for measuring the solver's selection time introduced in section 7.2
-- `./spec_code/`: containing source code with property specifications introduced in section 4.
-- `./src/library/`: containing container implementations used in this paper, library specifications introduced in section 5 and property based tests introduced in section 7.1
-- `./src/proptest/`: containing code for setting up property based tests.
-- `./src/tools/`: containing the code for generating dataset memory profiling.
-- `./src/traits/`: container syntactic properties introduced in section 4.
-- `./src/main.rs`: the entry point for executing the tool.
-- All other files in the `./src/` directory are the detailed implementation of Primrose, including parsing, type checking and analyzing property specifications, extracting and processing library specifications, executing the selection and generating code.
-- `./runall.sh` is the script for executing all examples
-- `./timeall.sh` is the script running every script located in `./scripts/` measuring the solver's selection time introduced in section 7.2
-- `./timetests.sh` is the script measuring the time for running all property based tests reported in section 7.1
-- `./Cargo.toml`, `./Cargo.lock` and `rust-toolchain.toml` are Rust package and compiler configuration files.
+# Candelabra
+
+This folder contains the source code for Candelabra as a Cargo workspace.
+First, setup the dependencies as detailed in the root of the repository.
+
+## Building
+
+Building is done with Cargo as normal: `cargo build`. This places the executable in `./target/debug/candelabra-cli`.
+
+## Creating cost models
+
+To build and view cost models, first find an implementation to look at:
+
+```
+$ cargo run -- list-library
+[...]
+Available container implementations:
+  primrose_library::VecMap
+  primrose_library::VecSet
+  std::vec::Vec
+  std::collections::BTreeSet
+  std::collections::BTreeMap
+  primrose_library::SortedVecSet
+  std::collections::LinkedList
+  primrose_library::SortedVecMap
+  primrose_library::SortedVec
+  std::collections::HashMap
+  std::collections::HashSet
+```
+
+To view the cost model for a single implementation, run `cargo run -- cost-model <impl>`.
+
+Alternatively, run `just cost-model <impl>` to look at a single implementation, or `just cost-models` to view models for all implementations.
+The latter will clear the cache before running.
+
+Cost models are also saved to `target/candelabra/benchmark_results` as JSON files. To analyse your built cost models, copy them to `../analysis/current/candelabra/benchmark_results` and see the README in `../analysis/`.
+
+## Profiling applications
+
+To profile an application in the `tests/` directory and display the results, run `cargo run -- --manifest-path tests/Cargo.toml -p <project> profile`.
+
+Alternatively, run `just profile <project>`.
+
+Profiling info is also saved to `target/candelabra/profiler_info/` as JSON.
+
+## Selecting containers
+
+To print the estimated cost of using each implementation in a project, run `cargo run -- --manifest-path tests/Cargo.toml -p <project> select`.
+
+Alternatively, run `just select <project>` for a single project, or `just selections` to run selection for all test projects.
+You can add `--compare` to any of these commands to also benchmark the project with every assignment of implementations, and print out the results.
diff --git a/thesis/README.md b/thesis/README.md
new file mode 100644
index 0000000..c419e56
--- /dev/null
+++ b/thesis/README.md
@@ -0,0 +1,6 @@
+# Thesis
+
+First, ensure that your LaTeX environment is setup properly, as described in the `README` at the project's root.
+Then, run `make`.
+
+This converts the graphs from JSON to png using `vl2png` from `vega-cli`, and runs `latexmk`.