Making experimentation terms clearer and well-defined

1

I recently went through the process of setting up and provisioning 8 experiments within a short time frame, and noticed a few things about the Merge experimentation process.

Specifically, it’s not very obvious to the user as to:

  1. why realization and materialization requries two separate steps
  2. what realization/relinquish and materialization/dematerialization actually do and their purpose

(and why is there no parallelism between “realize” and “relinquish” like “materialize” and “dematerialize”?)

For example, a user might ask or wonder (with helpful answers from discussions with @bkocolos):

  • if I dematerialize an experiment, are the expnodes imaged and saved for a quicker rematerialization like on DETERLab?

    (answer: no, dematerialization wipes all the expnodes in an experiment.)

  • why can’t I save a snapshot of my expnodes and restore them?

    (answer: to promote complete automation and minimize compute/storage resources. in practice, the inability to snapshot expnodes means that users leave experiments running indefinitely, especially during the design/iteration/testing phase of experimentation.

    while a minimal base image provides the ultimate customizability, bootstrapping a minimal base image each time an experiment is run adds a non-trivial amount of delay to each run.)

  • why do I have to realize and then materalize if realizing already reserves resources on the testbed?

    (answer: the two are separate steps as you could materalize and dematerialize many times over the length of an experiment. rematerializing on the same hardware reservation means you don’t have to account for hardware differences)

  • why do I have to dematerialize, and then relinquish (“derealize”) an experiment to clean it up?

    (answer: see previous answer)

Some of the concepts are buried in the documentation, and we need to push these concepts to the front and center so that users understand, at a high level, the Merge model of experimentation.

Before we do that, I also think there’s an opportunity to define and/or make the terms clearer.

For example, we could:

  • alias create to (realize + materialize) and destroy to (dematerialize + relinquish)
  • alternatively, instead of realize/relinquish and materialize/dematerialize, we could implement/alias create {realization,materialization} and destroy {materialization,realization}
  • support custom images and snapshots
  • make the “create an experiment” process as simple as possible for the user’s first project, and then introduce customization as needed

Thoughts/suggestions?