Evaluation

The evaluation is done using multiple stages.

Speaker: Car’s Position + Groundtruth

Create a simple interpretation of what’s happening.


State Machines

At the evaluation’s core are multiple single-purpose state machines that keep track of what’s happening:

  • Progress: Whether the car is at the beginning/middle/end of the road

  • Overtaking: Whether the car correctly overtakes obstacles

  • …

Example: OvertakingStateMachine


Graph of OvertakingStateMachine

Graph of OvertakingStateMachine

Referee

The output of the state machines is monitored by a referee node that check’s if the

  • state_machines are in valid states -> Referee.DRIVING

  • car reaches the end of the road -> Referee.COMPLETED

  • car makes mistake -> Referee.FAILED

Example: Referee Output


The Complete Picture

digraph EvaluationPipeline { node [style=dotted, shape=box]; groundtruth_services; car_state_topic; node [style=solid, shape=ellipse]; speaker_node; node [shape=box]; speaker_topics; broadcast_topic; node [shape=ellipse]; state_machine_node; node [shape=box]; state_topics; set_topics; node [shape=ellipse]; referee_node; groundtruth_services -> speaker_node [style=dotted, dir=both]; car_state_topic -> speaker_node [style=dotted]; speaker_node -> speaker_topics; speaker_node -> broadcast_topic; speaker_topics -> state_machine_node; broadcast_topic -> referee_node; state_machine_node -> state_topics; set_topics -> state_machine_node; state_topics -> referee_node; referee_node -> set_topics; subgraph speaker_topics { rank="same" speaker_topics broadcast_topic } subgraph referee_topics { rank="same" state_topics set_topics } }

Schema of the Evaluation Pipeline


See simulation_evaluation for more details.