factoriax.engine.rewards

Contents

factoriax.engine.rewards#

Score what changed in one step.

A reward function reads the state before a step and the state after it. It returns one number that says how good the step was. It never changes the state. Every function here takes (prev_state, new_state, params) and returns a scalar float32. An environment can therefore take any of them, and functools.partial() can bind the extra argument of the functions that need one.

The functions are of two kinds, and the difference matters more than the names suggest.

A sparse reward pays only when the target event happens: a player mines an ore, or an achievement unlocks. It is honest and hard to learn from, because most steps score zero.

A dense reward adds shaping terms. Those terms pay a small amount for a position near something useful, or for a quantity that moves in the right direction. A dense reward is easier to learn from and easier to get wrong, because an agent optimises the terms that it receives, and not the goal behind them. Two of the shaping terms here have that problem, and their own docstrings say so.

A proximity term measures Manhattan distance from state.selected_player, and not from a player in an argument. With more than one player it therefore always scores the selected player.

The rewards have no clamp, and several can go negative, because they are differences between two states and a quantity can fall. Read the return description of each function. Do not assume a floor of zero.

factoriax.engine.rewards.achievement_reward(prev_state: EnvState, new_state: EnvState, params: EnvParams, weights: Array) → Array#

Sparse reward for newly unlocked achievements.

The function compares achievements_unlocked in the two EnvState objects, and returns the weighted sum of the slots that the step satisfied. The achievements sit on EnvState itself, and the achievement_fn constructor argument of the environment latches them in every step.

An achievement pays one time. The reward is the weighted count of the bits that are set now and were not set before. A bit that stays set therefore scores nothing on a later step. A bit that returns to False also scores nothing, and never a negative amount.

Parameters:

prev_state – State just before the step.
new_state – State just after the step.
params – The function does not read this argument. It is present so that the signature matches the shared reward signature after functools.partial() binds weights.
weights – Reward amount for each slot, shape (MAX_ACHIEVEMENTS,). Build it with factoriax.engine.achievements.achievement_weights(). This argument has no default, because a bit index means a different achievement in each scenario.

Returns:

Scalar float32. The value is zero on every step that unlocks nothing, which is almost every step. The value is never negative.

Return type:

factoriax.engine.rewards.dense_assembler_reward(prev_state: EnvState, new_state: EnvState, params: EnvParams) → Array#

Dense reward for the assembler_production level.

The function adds proximity to the nearest assembler, 2.0 for each item that arrived in an assembler input slot, and 10.0 for each tier-1 science pack that the player gained.

Both delta terms are clamped at zero. An assembler that consumes its inputs to start a craft therefore costs nothing, although the input count falls. The pack count reads player 0, and not selected_player.

Parameters:

prev_state – State just before the step.
new_state – State just after the step.
params – The function does not read this argument. It is present for the shared reward signature.

Returns:

Scalar float32, always positive.

Return type:

factoriax.engine.rewards.dense_belt_reward(prev_state: EnvState, new_state: EnvState, params: EnvParams) → Array#

Dense reward for the belt_line level.

The function adds 5.0 for each belt placed, and 5.0 for each item that arrived in a pallet.

There is no proximity term. The level asks the player to close a gap in a belt line, and no field in the state marks the tile of the gap. There is therefore no target to measure a distance to.

CAUTION: A belt on any tile pays, and not only a belt that closes the gap.

Parameters:

prev_state – State just before the step.
new_state – State just after the step.
params – The function does not read this argument. It is present for the shared reward signature.

Returns:

Scalar float32. The value is zero on a step that does neither, and negative on a step that empties a pallet.

Return type:

factoriax.engine.rewards.dense_craft_reward(prev_state: EnvState, new_state: EnvState, params: EnvParams) → Array#

Dense reward for crafting levels (craft_pallets, craft_miners).

The function adds ore proximity, ore mined, and 10.0 for each placeable item gained. It counts miners, pallets, belts, and assemblers together.

CAUTION: The craft term does not test that the step spent materials. It is the rise in the total placeable count, clamped at zero. A pickup of a machine from the map therefore pays the same 10.0 as a craft. Use sparse_pallet_crafting_reward() or sparse_miner_crafting_reward() when that difference matters.

The craft term reads player 0, and not selected_player.

Parameters:

prev_state – State just before the step.
new_state – State just after the step.
params – The function does not read this argument. It is present for the shared reward signature.

Returns:

Scalar float32, always positive.

Return type:

factoriax.engine.rewards.dense_deploy_reward(prev_state: EnvState, new_state: EnvState, params: EnvParams) → Array#

Dense reward for the levels that ask a player to place a miner and run it.

The levels deploy_miner, place_and_fuel, and mining_factory use this function. It adds ore proximity, ore mined, and 5.0 times miner_output_reward().

CAUTION: That last term has the fault that its own docstring describes. It pays for ore that stays in a miner. A miner connected to a belt therefore scores 0 on that term, and a miner that nothing empties scores 15. On the deployment levels this pays for a miner that a player places and leaves, which is what those levels ask for. It does not fit a level about the movement of the ore after that.

Parameters:

prev_state – State just before the step.
new_state – State just after the step.
params – The function does not read this argument. It is present for the shared reward signature.

Returns:

Scalar float32. The value is negative on a step that empties the miners faster than the mining term and the proximity term add.

Return type:

factoriax.engine.rewards.dense_deposit_reward(prev_state: EnvState, new_state: EnvState, params: EnvParams) → Array#

Dense reward for the deposit_into_pallets level.

The function adds proximity to the nearest pallet, and 5.0 for each item that arrived in a pallet.

Parameters:

prev_state – State just before the step.
new_state – State just after the step.
params – The function does not read this argument. It is present for the shared reward signature.

Returns:

Scalar float32. The value is negative on a step that empties a pallet.

Return type:

factoriax.engine.rewards.dense_fill_pallet_reward(prev_state: EnvState, new_state: EnvState, params: EnvParams) → Array#

Dense reward for the fill_pallet level.

The function adds ore proximity, pallet proximity, ore mined, and 5.0 for each item that arrived in a pallet.

Parameters:

prev_state – State just before the step.
new_state – State just after the step.
params – The function does not read this argument. It is present for the shared reward signature.

Returns:

Scalar float32. The value can go negative on a step that empties a pallet, because the filling term is a level and the lowest proximity value is small.

Return type:

factoriax.engine.rewards.dense_pickup_reward(prev_state: EnvState, new_state: EnvState, params: EnvParams) → Array#

Dense reward for the pickup_machines level.

The function adds proximity to the nearest machine, and 10.0 for each machine that left the map in this step.

CAUTION: A placement costs nothing, because the count term is clamped at zero. An agent can therefore place a machine and pick it up again for 10.0 every two steps, without end. The level that uses this reward is short, so the loop does not control the score. The term does not fit a longer episode.

Parameters:

prev_state – State just before the step.
new_state – State just after the step.
params – The function does not read this argument. It is present for the shared reward signature.

Returns:

Scalar float32, always positive.

Return type:

factoriax.engine.rewards.dense_withdraw_reward(prev_state: EnvState, new_state: EnvState, params: EnvParams) → Array#

Dense reward for the withdraw_ore level.

The function adds proximity to the nearest miner that still holds output, and 10.0 for each item that the player gained.

CAUTION: The withdraw term counts every inventory gain, and not the withdraws. A mine into the inventory pays the same 10.0.

Parameters:

prev_state – State just before the step.
new_state – State just after the step.
params – The function does not read this argument. It is present for the shared reward signature.

Returns:

Scalar float32, always positive. The gain term is clamped at zero, so a step that spends items costs nothing.

Return type:

factoriax.engine.rewards.miner_output_reward(prev_state: EnvState, new_state: EnvState, params: EnvParams) → Array#

Score the change in the ore that sits in the miner buffers.

This function measures a level, not a flow. It is the rise in ent_buf_count, added over the placed miners. It therefore pays for ore that stays in a miner, and not for ore that a miner produced.

CAUTION: This reward pays for the opposite of automation. Use miner_throughput_reward() for a dense mining signal. A miner that pushes its output into a pallet or a belt ends the step as empty as it started, and scores 0. An idle miner that nothing empties scores 3, the default mining rate. On a one-tile coal patch the measured values are 0.0 for a miner that faces a pallet, and 3.0 for the same miner that faces nothing. A step that empties a miner scores a negative amount, because the buffer falls.

Parameters:

prev_state – State just before the step.
new_state – State just after the step.
params – The function does not read this argument. It is present for the shared reward signature.

Returns:

Scalar float32. The value is negative on a step where the miner buffers lose more than they gain.

Return type:

factoriax.engine.rewards.miner_throughput_reward(prev_state: EnvState, new_state: EnvState, params: EnvParams) → Array#

Reward for the ore that placed miners take out of the ground each tick.

The function measures the fall in block_resources on the tiles that hold a miner. It counts the extraction from the ground itself, and not a change in an output slot. A withdraw from the output slot, or no withdraw at all, therefore does not change the value.

This function measures a flow, and miner_output_reward() measures a level. The value here does not depend on whether something empties the miner. The function clamps the fall on each tile at zero, so a tile with more resources than before cannot cancel a tile that a miner mined.

Parameters:

prev_state – State just before the step.
new_state – State just after the step.
params – The function does not read this argument. It is present for the shared reward signature.

Returns:

Scalar float32, the ore taken out of the ground this step. Never negative.

Return type:

factoriax.engine.rewards.mining_reward(prev_state: EnvState, new_state: EnvState, params: EnvParams) → Array#

Dense reward: proximity to ore, plus a bonus for each ore mined.

The function adds two terms:

Proximity: 0.05 / (1 + d), where d is the Manhattan distance in tiles from the selected player to the nearest block in MINEABLE_BLOCKS. The value is 0.05 at most, and never 0.
Mining bonus: 20.0 for each ore item that this step extracted. This is the rise in items_mined, added over coal, iron ore, and copper ore.

The scale of 400:1 between the two terms is deliberate. Proximity only separates steps that mine nothing. One ore is worth more than any distance to ore.

The two terms disagree about the meaning of ore. Proximity targets every block in MINEABLE_BLOCKS, which includes tin, silicon, and limestone. The bonus counts coal, iron ore, and copper ore only. An agent that follows the proximity term to a silicon patch and mines there gets no ore bonus.

Parameters:

prev_state – State just before the step.
new_state – State just after the step.
params – The function does not read this argument. It is present for the shared reward signature.

Returns:

Scalar float32. The value is always positive, because the lowest proximity value is positive. It is about 0.05 or less on a step that mines nothing.

Return type:

factoriax.engine.rewards.pallet_filling_reward(prev_state: EnvState, new_state: EnvState, params: EnvParams) → Array#

Score the change in how much is sitting in pallet buffers.

The function pays for each item as it arrives and does not wait for a full stack, so one deposit scores at once.

This function measures a level and not a flow, the same as miner_output_reward(). The route that an item took to reach a pallet does not matter, so a player deposit and an arm delivery score the same. A step that empties a pallet scores a negative amount.

Parameters:

prev_state – State just before the step.
new_state – State just after the step.
params – The function does not read this argument. It is present for the shared reward signature.

Returns:

Scalar float32. The value is negative on a step where the pallets lose more than they gain.

Return type:

factoriax.engine.rewards.player_inventory_reward(prev_state: EnvState, new_state: EnvState, params: EnvParams) → Array#

Reward for each item that the selected player gains.

The function adds the rise in the item counts over all inventory slots. The value is positive when the step adds items, as a withdraw or a mine does. It is zero or negative when the step consumes items, as a craft, a deposit, or a placement does.

Every item counts the same. A coal and an assembler are both worth 1, so this reward cannot say that some items are harder to get than others.

Parameters:

prev_state – State just before the step.
new_state – State just after the step.
params – The function does not read this argument. It is present for the shared reward signature.

Returns:

Scalar float32. The value is negative when the player spends or stores more than it takes up. A craft, a placement, and a deposit all do this.

Return type:

factoriax.engine.rewards.sparse_miner_crafting_reward(prev_state: EnvState, new_state: EnvState, params: EnvParams) → Array#

Sparse reward of 1.0 for each miner that a craft produced.

The function finds a craft in three facts together: the inventory of the player gained miners, and it lost both iron and copper in the same step. A move of miners between the inventory and the map, through place or pickup, consumes no resources. Those actions therefore pay nothing.

Parameters:

prev_state – State just before the step.
new_state – State just after the step.
params – The function does not read this argument. It is present for the shared reward signature.

Returns:

Scalar float32, one unit for each miner crafted. The value is zero unless both iron and copper fell in the same step.

Return type:

factoriax.engine.rewards.sparse_mining_reward(prev_state: EnvState, new_state: EnvState, params: EnvParams) → Array#

Sparse reward of 1.0 for each ore item mined during this step.

The function adds the difference in items_mined between the two states, over coal, iron, and copper. The value is zero on every step that extracts nothing. This makes behaviour harder to shape, but the number is easy to read: one unit of reward for one unit of ore.

The function counts coal, iron ore, and copper ore only. A player can also mine tin, silicon, and limestone, and those score nothing.

Parameters:

prev_state – State just before the step.
new_state – State just after the step.
params – The function does not read this argument. It is present for the shared reward signature.

Returns:

Scalar float32, one unit for each ore item extracted. The value is zero on most steps.

Return type:

factoriax.engine.rewards.sparse_pallet_crafting_reward(prev_state: EnvState, new_state: EnvState, params: EnvParams) → Array#

Sparse reward of 1.0 for each pallet that a craft produced.

The function finds a craft in two facts together: the inventory of the player gained pallets, and it lost iron in the same step. A move of pallets between the inventory and a machine changes the pallet count and consumes no iron. A cycle of place, pickup, deposit, or withdraw therefore pays nothing.

Parameters:

prev_state – State just before the step.
new_state – State just after the step.
params – The function does not read this argument. It is present for the shared reward signature.

Returns:

Scalar float32, one unit for each pallet crafted. The value is zero when pallets appeared and the step spent no iron. That test is what refuses the reward for a move of items.

Return type: