xper

xper

Experiment tracking that remembers why, not just what.

It’s git, specialized for research iteration. Every experiment’s name is its ancestry — v1.1.2 is the second thing you tried on top of v1.1, which came from v1. You can read the lineage without looking anything up.

Install GitHub 
$ git clone https://github.com/AliKhudiyev/xper.git
$ cd xper && ./install.sh

01 — the problem

Three weeks later

You ran 40 experiments to get one paper. Later you find a good number in an old log and you can’t reproduce it, because the code that produced it has been overwritten a dozen times since.

The idea survived. The state that made it work didn’t.

02 — how it works

Two ways to run a version

Every version is a git branch, and its number is its address. Each one runs in one of two modes: normal, where it is yours alone, or supervisional, where a group shares it one writer at a time.

Your own lineage

A normal version belongs to whoever made it. Anyone else who builds on it gets their own copy — the original stays read-only for them, so your results cannot be edited out from under you. Step through a real session and watch the tree it builds.

    click any line to jump to that point
    base v1 #MLP v1.1 v1.1.1 v1.1.2 v2
    Fig. 1 — one session, and the tree it produces

    When an idea dead-ends, jump back to the last version that worked and go a different way. Nothing is overwritten, and the number tells you where you are — no naming scheme to invent, and none to abandon by week three.

    03 — positioning

    Why not what you already use

    These tools are good at what they do. They just answer a different question.

    Weights & Biases

    tracks runs, not state

    Metrics, curves and artifacts, thoroughly. But it doesn’t touch your code state, so it can tell you one run scored better than another without telling you what you changed.

    DVC

    pipeline first

    Genuinely versions code and data together, and dvc exp is good. But it is built around defining a pipeline up front — heavy when you just want to try something on a script and see if it’s worth pursuing.

    plain git

    no opinion

    Tracks everything, organizes nothing. Branch names are arbitrary, and by week three you’ve abandoned whatever naming scheme you invented in week one.

    xper

    the name is the lineage

    Opinionated, so you don’t have to invent a scheme. Version numbers encode ancestry, navigation is one command, and it’s git underneath — nothing to migrate into.

    04 — beyond the tree

    What else it does

    The tree gets you organized. These are the things you reach for once you have forty of them.

    Rank by your own metric

    Point xper at a log file and write plain key:value lines. It will order your entire experiment tree by any field in it — then take you straight to the best one. 

    $ xper logfile --path log.txt
$ cat log.txt
accuracy:0.918
loss:0.211
$ xper sort --by accuracy
$ xper jump --last   # best run

    Walk the ordering

    jump traverses that sorted order, not the tree — so after a metric sort, “next” means “next best”. Steps clamp at the ends unless you ask them to wrap. 

    $ xper jump -f 1        # forward one
$ xper jump -b 3 --wrap
$ xper jump --first
$ xper children     # what branched off here

    Freeze a result

    xper finish makes the working tree read-only on disk. A finished experiment physically cannot be edited — by you, your editor, or a stray script — until you unlock it. 

    $ xper finish
$ echo x >> train.py
zsh: permission denied: train.py
$ xper modify       # writable again

    Work with other people

    Point at a remote and push. Branch off a collaborator’s version and xper mirrors their address into your own namespace, recording what you started from. 

    $ xper remote --add git@github.com:lab/runs.git
$ xper backup       # push your work
$ xper update       # see what changed

    05 — recordings

    See it run

    Real terminal sessions. Text is selectable — copy any command straight out of the player.

    Building the tree

    The session from Fig. 1, start to finish.

    $ xper init
$ xper new --tag MLP        # v1
$ vim train.py
$ xper new -y               # v1.1
$ vim train.py
$ xper new -y               # v1.1.1
$ xper jump v1.1
$ vim train.py
$ xper new -y               # v1.1.2
$ xper new --scratch        # v2
    recording pending — drop casts/tree.cast in to replace this

    Sorting by a metric

    Log a number, sort the tree by it, jump to the winner.

    $ xper logfile --path log.txt
$ xper sort --by accuracy
$ xper jump --last
$ cat log.txt
accuracy:0.918
    recording pending — drop casts/sort.cast in to replace this

    Locking a finished run

    finish and modify, and what happens in between.

    $ xper finish
$ echo x >> train.py
zsh: permission denied: train.py
$ xper modify
$ echo x >> train.py       # fine now
    recording pending — drop casts/finish.cast in to replace this
    Fig. 2 — recorded sessions

    06 — reference

    Every command

    The complete surface. xper help prints the same list in your terminal.

    CommandWhat it does
    Setup
    xper initSet up the experiment repo here. Creates a base branch named after your git user, plus a .xper metadata file.
    xper remote --add <url>Set the remote. With no --add, prints the current remotes.
    Creating versions
    xper newCommit your work to the current version, then open a child of it: v1 → v1.1 → v1.1.1.
    xper new --tag <tag>Label the new version. Also -t.
    xper new --scratchStart a new top-level version (v2, v3…) from your base branch instead of from where you are — it carries none of the current lineage’s work. Also -s.
    xper new --yesCreate the version even when the parent shows no committed difference. You will reach for this often — see the note below. Also -y.
    xper new -slCreate the version in supervisional mode instead of normal — shared, and read-only until someone takes the write key. Add --acquire to take it straight away.
    xper deleteDelete the current version and every subversion beneath it, then return to base. Also del.
    Navigating
    xper jump <version>Go to a version by address, e.g. xper jump v1.1. Also goto.
    xper jump <v> -u <user>Jump to another person’s version at that address.
    xper jump -f <n> / -b <n>Move n steps forward or backward through the current ordering.
    xper jump --first / --lastGo to the first or last entry in the ordering.
    xper jump --wrapWrap around the ends instead of stopping at them. Also -w.
    xper jump -gTraverse everyone’s versions, not just your own.
    xper childrenList the versions branching directly off this one.
    xper parentPrint the version this one came from.
    Comparing
    xper diff <version>Show the git diff between the current version and that one.
    xper diff jump [opts]Diff the current version against wherever that jump would land.
    Ordering
    xper logfile --path <file>Set this version’s log file. Bare xper logfile prints it. Format is one key:value per line.
    xper sortOrder versions by version number.
    xper sort --by <field>Order by a numeric field read from each version’s log file. Comma-separated values sort component by component.
    xper sort --only-leafKeep only leaf versions — the ones nothing was built on top of.
    xper index --clearEmpty the ordering.
    xper index --add [<v>] / --remove [<v>]Add or drop one version from the ordering. Defaults to the current one.
    xper index --after|--before|--swap <v>Reorder by hand. Target first, then the version to move (current by default).
    Collaborating
    xper backupPush your work to the remote.
    xper updatePull in what your collaborators did on this version.
    xper update -gPull every version from the remote, tracking branches you don’t have yet, so you can jump to anyone’s work. Also --global.
    Supervisional mode
    xper acquireTake the write key on a supervisional version. Your working tree unlocks, and the holder is recorded in the repo for everyone else to see.
    xper releaseHand the key back. Your work is pushed and your tree locks again, freeing the version for the next person.
    Lifecycle
    xper finishLock the current version: the working tree becomes read-only on disk, and the version is backed up if a remote is set.
    xper modifyUnlock the current version for editing again. Only its owner can.
    xper helpPrint the full command list.

    # worth knowing

    • Every command commits first. Branching, jumping and sorting all save your work before they move — navigating never costs you uncommitted changes.
    • xper new compares committed snapshots, not your working tree. Uncommitted edits don’t register as a difference, so straight after editing a file it will decline. Pass -y — it commits your edits to the current version first, then opens the child.
    • Your edits land on the version you are leaving. xper new snapshots the current version and hands you a fresh copy to work in, so the change you just made belongs to the parent.
    • --scratch branches from base, not from where you are. That’s the point: it detaches from the current lineage. It is a copy of your base branch, so it is empty only if your base was.
    • Version numbers are never reused. Delete v1 and the next --scratch still gives you the next unused number, so an address always means one thing.
    • Every version has a mode. Normal by default — yours alone, and anyone else who builds on it gets their own copy. Supervisional (xper new -sl) makes it a shared workspace with one write key, passed by hand with acquire and release.
    • Your username prefixes every branch. Your git user.name with spaces removed, so v1.1 is really the branch ADALOVELACE_v1.1.

    07 — install

    Three lines

    xper is Bash and git, nothing else. No runtime to install, no account to make, no daemon.

    # install

    $ git clone https://github.com/AliKhudiyev/xper.git
$ cd xper && ./install.sh

    Installs to ~/.local — no sudo, and nothing is written outside the prefix. Use --prefix DIR to put it somewhere else, and --uninstall to remove it later. If ~/.local/bin isn’t on your PATH, the installer prints the line to add.

    # check it

    $ mkdir demo && cd demo
$ xper init
[xper_init] initalized with username [ADALOVELACE]
$ echo "print('hi')" > train.py
$ xper new --tag first
$ git branch --show-current
ADALOVELACE_v1

    Needs git 2.22 or newer, bash, and a user.name in your git config — xper builds branch names from it. macOS and Linux.