VUnit Phases¶
What You Will Learn¶
The different phases a VUnit testbench traverses during its execution.
How phases can be used by a process, such as a third part verification component (VC), to prevent a premature simulation exit without requiring non-intuitive coding constructs on the user side.
How testbench processes can be made aware of an imminent simulation exit, allowing them to carry out exit tasks such as verifying the correctness of the final DUT state or logging interesting summary information.
Introduction¶
During the course of a simulation, VUnit guides the testbench through a number of distinct phases. These phases are
determined by the structure of the test runner process - the main process controlling the execution of the testbench.
Some phases are completely encapsulated within a VUnit procedure, such as test_runner_setup and
test_runner_cleanup, while others are defined as a code region within the test runner process. The code example
below outlines a test runner process consisting of two test cases. The phases involved are named and described by the
phase procedure calls. This procedure, created specifically for this example, provides a visual representation of
the different phases and their transitions as a VUnit log.
test_runner : process
begin
phase("TEST RUNNER SETUP",
"The testbench is initialized from the runner_cfg generic. This allows for " &
"configuration of features such as coloration of log entries. This phase " &
"call comes before initialization, so it will not be affected by any of the " &
"settings and the resulting log entry will be without special colors."
);
test_runner_setup(runner, runner_cfg);
phase("TEST SUITE SETUP",
"Code common to the entire test suite (set of test cases) that is executed *once* " &
"prior to all test cases. For example, if we want to specify what log levels should " &
"be visible."
);
show(display_handler, debug);
while test_suite loop
phase("TEST CASE SETUP",
"Code executed before *every* test case. For example, if we use the VUnit " &
"run_all_in_same_sim attribute to run all test cases in the same simulation, we " &
"may need to reset the DUT before each test case."
);
-- vunit: run_all_in_same_sim
reset <= '1';
wait for 10 ns;
reset <= '0';
if run("Test case 1") then
phase("TEST CASE",
"This is where we run our test case 1 code."
);
wait for 10 ns; -- The test code is just a wait statement in this dummy example
elsif run("Test case 2") then
phase("TEST CASE",
"This is where we run our test case 2 code."
);
wait for 10 ns; -- The test code is just a wait statement in this dummy example
end if;
phase("TEST CASE CLEANUP",
"Code executed after *every* test case. For example, there may be some DUT status " &
"flags we want to check before ending the test."
);
check_equal(error_flag, '0');
end loop;
phase("TEST SUITE CLEANUP",
"Code common to the entire test suite which is executed *once* after all test " &
"cases have been run. For example, it can be used to check if the desired coverage " &
"metric has been fully achieved."
);
check_true(full_coverage);
phase("TEST RUNNER CLEANUP",
"Housekeeping performed by VUnit before ending the simulation. For example, " &
"if VUnit was configure not to end the simulation upon detecting the first error, " &
"it will fail the simulation during this phase if any errors have been detected."
);
test_runner_cleanup(runner);
end process;
0 fs - tb_phases - PHASE - TEST RUNNER SETUP
The testbench is initialized from the runner_cfg generic. This allows for
configuration of features such as coloration of log entries. This phase
call comes before initialization, so it will not be affected by any of the
settings and the resulting log entry will be without special colors.
0 fs - tb_phases - PHASE - TEST SUITE SETUP
Code common to the entire test suite (set of test cases) that is executed
*once* prior to all test cases. For example, if we want to specify what log
levels should be visible.
0 fs - tb_phases - PHASE - TEST CASE SETUP
Code executed before *every* test case. For example, if we use the VUnit
run_all_in_same_sim attribute to run all test cases in the same simulation,
we may need to reset the DUT before each test case.
10000000 fs - tb_phases - PHASE - TEST CASE
This is where we run our test case 1 code.
20000000 fs - tb_phases - PHASE - TEST CASE CLEANUP
Code executed after *every* test case. For example, there may be some DUT
status flags we want to check before ending the test.
20000000 fs - tb_phases - PHASE - TEST CASE SETUP
Code executed before *every* test case. For example, if we use the VUnit
run_all_in_same_sim attribute to run all test cases in the same simulation,
we may need to reset the DUT before each test case.
30000000 fs - tb_phases - PHASE - TEST CASE
This is where we run our test case 2 code.
40000000 fs - tb_phases - PHASE - TEST CASE CLEANUP
Code executed after *every* test case. For example, there may be some DUT
status flags we want to check before ending the test.
40000000 fs - tb_phases - PHASE - TEST SUITE CLEANUP
Code common to the entire test suite which is executed *once* after all
test cases have been run. For example, it can be used to check if the
desired coverage metric has been fully achieved.
40000000 fs - tb_phases - PHASE - TEST RUNNER CLEANUP
Housekeeping performed by VUnit before ending the simulation. For example,
if VUnit was configure not to end the simulation upon detecting the first
error, it will fail the simulation during this phase if any errors have
been detected.
To infer as little overhead as possible (but not less than that), VUnit permits testbenches without named test cases,
resulting in the elimination of certain phases. However, test_runner_setup and test_runner_cleanup remain
present at all times.
test_runner : process
begin
phase("TEST RUNNER SETUP",
"The testbench is initialized from the runner_cfg generic. This allows for " &
"configuration of features such as coloration of log entries. This phase " &
"call comes before initialization, so it will not be affected by any of the " &
"settings and the resulting log entry will be without special colors."
);
test_runner_setup(runner, runner_cfg);
phase("TEST CASE",
"This is where we run all the test code."
);
reset <= '1';
wait for 10 ns;
reset <= '0';
wait for 10 ns; -- The test code is just a wait statement in this dummy example
check_equal(error_flag, '0');
check_true(full_coverage);
phase("TEST RUNNER CLEANUP",
"Housekeeping performed by VUnit before ending the simulation. For example, " &
"if VUnit was configure not to end the simulation upon detecting the first error, " &
"it will fail the simulation during this phase if any errors have been detected."
);
test_runner_cleanup(runner);
end process;
Of these phases, the test runner cleanup phase is the most useful and the focal point of this blog. To understand how we
can use this phase, we can start by showing the usually hidden trace messages from the runner logger when the
test_runner_cleanup function is invoked:
0 fs - tb_phases - PHASE - TEST RUNNER SETUP
The testbench is initialized from the runner_cfg generic. This allows for
configuration of features such as coloration of log entries. This phase
call comes before initialization, so it will not be affected by any of the
settings and the resulting log entry will be without special colors.
0 fs - tb_phases - PHASE - TEST CASE
This is where we run all the test code.
20000000 fs - tb_phases - PHASE - TEST RUNNER CLEANUP
Housekeeping performed by VUnit before ending the simulation. For example,
if VUnit was configure not to end the simulation upon detecting the first
error, it will fail the simulation during this phase if any errors have
been detected.
20000000 fs - runner - TRACE - Entering test runner cleanup phase.
20000000 fs - runner - TRACE - Passed test runner cleanup phase entry gate.
20000000 fs - runner - TRACE - Passed test runner cleanup phase exit gate.
20000000 fs - runner - TRACE - Entering test runner exit phase.
As we can see VUnit keeps track of the phases internally and there is a concept of a gate when entering and exiting a phase. In this example, VUnit passes through the gates but it’s possible for a user to prevent that by locking a gate. This capability can be used to prevent VUnit from cleaning up and exiting the simulation until all processes are done, resolving the issues that arose with using an event as a barrier in our previous blog:
The phase lock solution scales well.
test_runner_cleanupis already present in every VUnit testbench and it eliminates the need for creating a new event for every process to wait on.There is no race condition as it doesn’t matter in which order processes complete.
test_runner_cleanupwill wait until every process has removed its lock.With the absence of any additional constructs in the test runner, there is no risk of forgetting to wait for another process, and any process can introduce locks without requiring external changes. This is especially important for verification components, as they can start using locks without the necessity of their users having to update their code.
Phase Gate Locks¶
To see the phase gate locks in action we’re going to revisit the example provided in the event blog. In that example our dut_checker process notified its completion with the
dut_checker_done event.
dut_checker : process
begin
if is_empty(queue) then
wait until is_active(new_data_set);
end if;
for i in 1 to pop(queue) loop
wait until (rising_edge(clk) and output_tvalid = '1') or log_active(vunit_error, decorate("while waiting on output data"), logger => dut_checker_logger);
check_equal(output_tdata, calculate_expected_output(pop(queue)));
end loop;
if is_empty(queue) then
notify(dut_checker_done);
end if;
end process;
The test runner process waited for that event before calling test_runner_cleanup.
wait until is_active_msg(dut_checker_done);
test_runner_cleanup(runner);
Before adding the locks, we need to remove the dut_checker_done event, the notification of that event in the
dut_checker, and the wait statement waiting for that event in test_runner. Afterwards, we can update the
dut_checker according to the code listing provided below.
dut_checker : process
constant key : key_t := get_entry_key(test_runner_cleanup);
begin
if is_empty(queue) then
wait until is_active(new_data_set);
end if;
lock(runner, key, dut_checker_logger);
for i in 1 to pop(queue) loop
wait until (rising_edge(clk) and output_tvalid = '1') or log_active(vunit_error, decorate("while waiting on output data"), logger => dut_checker_logger);
check_equal(output_tdata, calculate_expected_output(pop(queue)));
end loop;
if is_empty(queue) then
unlock(runner, key, dut_checker_logger);
end if;
end process;
The first step is to acquire a unique key for the gate we want to control, in this case the entry gate for the
test_runner_cleanup phase. This is done by calling the get_entry_key function with test_runner_cleanup as
the parameter. Each gate has many locks and the returned key fits one of those locks. An alternative design for locking
a gate would be to use a keyless system, wherein locks can be added and removed to/from gates. However, such a design is
prone to a class of bugs where a process unlocks a gate more than it locks it. This will lead to locks previously added
by other processes being removed and the protection against premature termination of the simulation is lost.
The second step is to determine when to lock the gate. Generally, this is done when the process has a task that requires
completion before the end of the simulation. In this case, this is when the queue is not empty. Locking is done by
passing the runner signal, the key, and optionally a logger, to the lock procedure. The runner signal in
VUnit contains several events, of which runner_phase is used to indicate that something occurred related to the
VUnit phases.
The third step is to decide when to unlock. Unlocking should be done when the process has completed a task, provided
there are no more tasks left to complete. In this case we unlock if the queue is empty. If we were to unlock before
checking the queue, and the test runner process has already pushed all remaining data sets to the queue, we allow
test_runner_cleanup to end the simulation and data is lost. When unlocking eventually does take place,
runner_phase is activated and triggers test_runner_cleanup to verify that all locks are unlocked such that it
can proceed.
Let’s take a look at what the log looks like after implementing these updates.
0 fs - dut_checker - TRACE - Locked test runner cleanup phase entry gate.
114000000 fs - dut_checker - TRACE - Locked test runner cleanup phase entry gate.
222000000 fs - dut_checker - TRACE - Locked test runner cleanup phase entry gate.
290000000 fs - runner - TRACE - Entering test runner cleanup phase.
290000000 fs - runner - TRACE - Halting on test runner cleanup phase entry gate.
326000000 fs - dut_checker - TRACE - Unlocked test runner cleanup phase entry gate.
326000000 fs - runner - TRACE - Passed test runner cleanup phase entry gate.
326000000 fs - runner - TRACE - Passed test runner cleanup phase exit gate.
326000000 fs - runner - TRACE - Entering test runner exit phase.
First, we can observe that dut_checker is locking the gate three times in succession without unlocking it in
between. This is perfectly fine as it keeps a locked gate locked and allows for simpler code. The same goes for an
unlocked gate; if it is unlocked multiple times, it will remain unlocked. This is another advantage of having unique
keys, as a keyless design would not accommodate this behaviour.
We can also see how test_runner_cleanup halts on the entry gate and proceeds immediately after the locked gate lock
is unlocked.
An alternative design solution for the dut_checker is to move the last if statement to the top:
dut_checker : process
constant key : key_t := get_entry_key(test_runner_cleanup);
begin
if is_empty(queue) then
unlock(runner, key, dut_checker_logger);
end if;
if is_empty(queue) then
wait until is_active(new_data_set);
end if;
lock(runner, key, dut_checker_logger);
for i in 1 to pop(queue) loop
wait until (rising_edge(clk) and output_tvalid = '1') or log_active(vunit_error, decorate("while waiting on output data"), logger => dut_checker_logger);
check_equal(output_tdata, calculate_expected_output(pop(queue)));
end loop;
end process;
The only difference is that there will be an initial unlock because of the initially empty queue but that is, as mentioned before, allowed.
0 fs - dut_checker - TRACE - Unlocked test runner cleanup phase entry gate.
0 fs - dut_checker - TRACE - Locked test runner cleanup phase entry gate.
114000000 fs - dut_checker - TRACE - Locked test runner cleanup phase entry gate.
222000000 fs - dut_checker - TRACE - Locked test runner cleanup phase entry gate.
290000000 fs - runner - TRACE - Entering test runner cleanup phase.
290000000 fs - runner - TRACE - Halting on test runner cleanup phase entry gate.
326000000 fs - dut_checker - TRACE - Unlocked test runner cleanup phase entry gate.
326000000 fs - runner - TRACE - Passed test runner cleanup phase entry gate.
326000000 fs - runner - TRACE - Passed test runner cleanup phase exit gate.
326000000 fs - runner - TRACE - Entering test runner exit phase.
It may be tempting to combine the two initial if statements into one where we first call unlock and then wait for
the new_data_set event. However, this is a bad idea since it exposes us to a potential Time-Of-Check To Time-Of-Use
bug. When we call unlock, it triggers
runner_phase, which is an operation consuming delta cycles. During this time, a new data set may have been pushed to
the queue and the new_data_set event is activated before we return from the unlock procedure. Consequently,
dut_checker will miss the first event and block on the wait statement until the second event arrives some time after
the DUT responded to the first data set. However, the first data set is still at the front of the queue and it will
cause a failure when it is used to verify the DUT response to the second data set. This is a bug that only occurs under
unfortunate timing circumstances and in this example we’ve added a dummy procedure with a finely tuned delay to showcase
the scenario. Even if the risk of encountering this bug is low, or not even possible with the timing at hand, it would
be unwise to make any assumptions about the test runner timing as that can change in the future.
dut_checker : process
constant key : key_t := get_entry_key(test_runner_cleanup);
begin
a_procedure_adding_some_delay;
if is_empty(queue) then
unlock(runner, key, dut_checker_logger);
wait until is_active(new_data_set);
end if;
lock(runner, key, dut_checker_logger);
for i in 1 to pop(queue) loop
wait until (rising_edge(clk) and output_tvalid = '1') or log_active(vunit_error, decorate("while waiting on output data"), logger => dut_checker_logger);
check_equal(output_tdata, calculate_expected_output(pop(queue)));
end loop;
end process;
0 fs - dut_checker - TRACE - Unlocked test runner cleanup phase entry gate.
78000000 fs - dut_checker - TRACE - Locked test runner cleanup phase entry gate.
82000000 fs - check - ERROR - Equality check failed - Got 0000_0000_1011_1001 (185). Expected 0000_0000_1000_0011 (131).
With the two separate if statements, there is still a potential issue to consider. What if the test runner process
pushes the last data set to the queue, notifies the new_data_set event, and then immediately calls
test_runner_cleanup? If the dut_checker just missed that last data set and unlocks the gate, will it then have
enough time to detect the new data set after the unlocking and lock the gate again, before test_runner_cleanup
terminates the simulation? Fortunately, there is time for that, as long as the presented design is used.
Keeping both of the if statements at the beginning serves no purpose other than explaining the dangers of misplaced code optimizations. We recommend keeping the if statement that unlocks the gate at the bottom, thereby keeping the optimization temptation “out of sight”.
At this point, we have designed a testbench where the dut_checker is solely responsible for carrying out its
intended task and ensuring it is fully completed before the simulation ends (high cohesion). There are no
responsibilities for the test_runner process. We have also ensured that the dut_checker does not make any
assumptions about the timing of the test_runner (low coupling).
Phase Transition Events¶
In the previous chapters, we described how a process can prevent phase transitions by locking gates. However, there are
also use cases that require processes to simply be aware of such transitions without needing to prevent them. To
illustrate, let us take a look at the AXI Stream standard. This standard provides a set of protocol assertions that can be employed to verify if a stream conforms to the
protocol. One such assertion, AXI4STREAM_ERRM_STREAM_ALL_DONE_EOS, states that
“At the end of simulation, all streams have had their corresponding TLAST transfer”
An AXI Stream protocol checker will not prevent a simulation from exiting but it must be given the opportunity to check that all streams have ended when that is about to happen. Let’s see how this is solved in the ARM-provided protocol checker IP:
“The testbench that you are using has a signal called EOS_signal. You must drive EOS_signal HIGH at the end of the simulation for at least one clock cycle.”
This is an example of a scenario in which a user must take a non-obvious step to ensure that a verification component
works correctly. This is due to the fact that the ARM IP lacks a testbench structure to rely upon. A VUnit verification
component, on the other hand, has a known structure in the mandatory presence of the test_runner_cleanup procedure.
This procedure will trigger the runner_phase event when the phase changes to test_runner_cleanup, giving the AXI
Stream protocol checker the option to act. However, remember that runner_phase is activated on all phase changes and
also when there is activity related to phase gate locks. Therefore, before checking stream status, the protocol checker
must verify that the event was caused by an imminent simulation exit and not some other change. This is done by
confirming that the active phase is test_runner_cleanup and that the testbench is within the gates of that phase,
i.e. it has passed the entry gate but not yet the exit gate. The principle for the
AXI4STREAM_ERRM_STREAM_ALL_DONE_EOS assertion is outlined below. After ensuring that the simulation is about to
terminate, the process is able to make its final check, provided it is done within a single delta cycle.
end_of_simulation_process : process
begin
wait until is_active(runner_phase) and is_within_gates_of(test_runner_cleanup);
check_stream_activity;
wait;
end process;
Incorporating this protocol checker into our example testbench generates the following log:
0 fs - dut_checker - TRACE - Unlocked test runner cleanup phase entry gate.
0 fs - dut_checker - TRACE - Locked test runner cleanup phase entry gate.
114000000 fs - dut_checker - TRACE - Locked test runner cleanup phase entry gate.
222000000 fs - dut_checker - TRACE - Locked test runner cleanup phase entry gate.
290000000 fs - runner - TRACE - Entering test runner cleanup phase.
290000000 fs - runner - TRACE - Halting on test runner cleanup phase entry gate.
326000000 fs - dut_checker - TRACE - Unlocked test runner cleanup phase entry gate.
326000000 fs - runner - TRACE - Passed test runner cleanup phase entry gate.
326000000 fs - axis_checker:STREAM_ALL_DONE_EOS - PASS - Equality check passed for number of active streams - Got 0.
326000000 fs - runner - TRACE - Passed test runner cleanup phase exit gate.
326000000 fs - runner - TRACE - Entering test runner exit phase.
As you can see from the pass message, the check is perform just before the simulation comes to an end.
Final Words¶
In this blog we’ve shown how phases can be used to handle the final ticks of your simulation in a robust and reliable way while adhering to two key concepts of good code design: high cohesion and low coupling.
Do you have your own personal use cases for phases, or perhaps use cases that you think are not supported? Feel free to reach out and share them with us!