Benchmarking in OpenVM

Currently processing Ream’s state transition functions has been implemented in: Pico, Jolt, Risc0, SP1 and OpenVM. We need to obtain common benchmarks for all these implementations because zkVMs will be part of lean consensus (formerly known as beam chain). Most zkVMs include instruction counts of any function in the code which allow easy benchmarking of the following functions: read_pre_state_ssz, deserialize_pre_state_ssz, read_operation_input, process_operation, merkleize, commit. To the best of my knowledge, this is not implemented in OpenVM which instead provides a flamegraph script that samples stack frames during execution of guest code. Here is an example of benchmarks found in OpenVM’s repo: https://github.com/openvm-org/openvm/blob/benchmark-results/index.md. Notice that the benchmarks found in the link provide metrics related to proof generation. This is not useful in my case because generating a proof for state transition functions is not possible in a laptop. What I need is instruction counts on specific functions of guest code. So far, I thought about two ways to overcome this issue:

  1. Modifying OpenVM’s flamegraph.py so that it generates instruction count metrics.
  2. Use a general purpose profiling tool such as perf

I am not sure which option is easier to implement. In both cases, there are many aspects which I have to understand before proceeding. In the first case, I have to understand what the script does, how to modify it and how to include some kind of “flags” that point to the functions that will be benchmarked. In the second case, I managed to get a generic flamegraph but failed to get benchmarks on the functions that I was interested in. Moreover, running perf’s generic profiling on my OpenVM program generated 20 GB of data.

Some additional links:

  1. Full CI workflow for reth by OpenVM:https://github.com/axiom-crypto/openvm-reth-benchmark/blob/main/.github/workflows/reth-benchmark.yml
  2. Instructions for benchmarking in OpenVM: https://github.com/openvm-org/openvm/blob/main/docs/crates/benchmarks.md#circuit-flamegraphs

3 Slot Finality (3SF)

So far, we have implemented and in some cases benchmarked beacon chain state transition functions. While this implementation provides a proof of concept it is time to consider running 3SF-mini inside a zkVM. For the moment, I managed to gather and read material about 3SF.

  1. The original paper: https://arxiv.org/abs/2411.00558.
  2. A very important talk about 3SF. In general the series of those recordings is an excellent introduction to lean consensus.
  3. This ethresearch post that lies between the talk and the full paper.
  4. Ream’s 3SF implementation in rust: https://github.com/ReamLabs/rust-3sf/tree/main
  5. Vitalik’s 3SF implementation in python: https://github.com/ReamLabs/rust-3sf/tree/main