factoriax.analysis.eval

factoriax.analysis.eval#

Hold one eval episode and draw its summary plots.

EvalRollout is the shared format for a single episode: the states, the actions, and the achievement mask at each step. The PPO eval pipeline and the scripted-agent runners both build one and pass it to generate_eval_plots().

That call writes three figures. One shows the item counts over time, one the action totals, and one the timestep at which each achievement unlocked.

class factoriax.analysis.eval.EvalRollout(frames: list[ndarray] | None, actions: ndarray, env_states: list[Any], ach_per_step: ndarray)#

The recorded output of one deterministic eval episode.

frames#

RGB frames for a video, or None when the run rendered none.

Type:

list[numpy.ndarray] | None

actions#

The action taken at each step, as a 1-D int array of length T.

Type:

numpy.ndarray

env_states#

The state at each step. This list holds T + 1 entries, one more than actions, because it records the state before the first action and after the last.

Type:

list[Any]

ach_per_step#

The achievement mask at each step, of shape (T + 1, MAX_ACHIEVEMENTS). The mask is latched, so a bit stays set once it is set. Trailing slots past the achievement count of the scenario are padding.

Type:

numpy.ndarray

property final_ach_mask: ndarray#

The achievement mask at the last recorded step.

Returns:

One row of ach_per_step, of length MAX_ACHIEVEMENTS. Because the mask is latched, this row names every achievement the episode reached.

Return type:

numpy.ndarray

factoriax.analysis.eval.generate_eval_plots(rollout: EvalRollout, out_dir: Any, *, achievement_labels: list[str], num_achievements: int, title_prefix: str = 'Final rollout') → dict[str, Any]#

Render inventory, action-count, and achievement-timing plots.

Each plot is drawn inside its own try. A plot that raises is logged with its traceback and left out of the result, so one failure does not cost the other two.

Parameters:

  • rollout – The recorded episode.

  • out_dir – Where to write the three PNG files. Missing parent directories are created.

  • achievement_labels – The display name of each achievement. The list must hold num_achievements entries.

  • num_achievements – How many leading slots of rollout.ach_per_step name a real achievement. The slots after them are padding up to MAX_ACHIEVEMENTS.

  • title_prefix – Text put in front of each figure title.

Returns:

{"items": path, "actions": path, "achievements": path}. A key is absent when its plot raised, so a caller must not assume all three.

Return type:

dict

Notes

The function selects the Agg matplotlib backend for the whole process. It also pads rollout.actions with one zero, to match the state list, which is one longer. Action 0 is NOOP, so the action-count plot plots one extra no-op.

factoriax.analysis.eval.plot_action_counts(actions: ndarray, out_path: Any, *, title: str = 'Action counts') → Any#

Bar chart of per-action counts over a single eval episode.

Bars are sorted by count, and an action with a count of zero is left out. The figure grows taller with the number of actions kept, so the labels stay readable.

One episode gives a noisy distribution over time, so this plot shows plain totals. For several episodes, a moving average of the distribution says more.

Parameters:

  • actions – Action ids in a 1-D int array. Every entry counts, so a trajectory padded with zeros reports those pad steps as real NOOP actions.

  • out_path – Where to write the PNG. Parent directories are not created.

  • title – The figure title.

Returns:

out_path unchanged, after the write.

Return type:

Any

Notes

The function writes a file and closes its own figure.

factoriax.analysis.eval.plot_item_counts(traj: Any, out_path: Any, *, title: str = 'Item counts over time') → Any#

Line plot of player item counts over time.

The plot holds one line for each item type that rises above zero at some point. An item that stays at zero is left out, which keeps the legend short.

The function reads episode 0 and player 0 only, from player_inventory of shape (1, T + 1, P, N). That field already holds a count for each item, so nothing has to map slots to items.

Parameters:
Returns:

out_path unchanged, after the write.

Return type:

Any

Raises:

ValueError – When the trajectory carries no player_inventory.

Notes

The function writes a file and closes its own figure.