Unfortunate set-action behavior while a line is disconnected

[mhassouna95]

Environment.

• Grid2op version: 1.9.8
• System: Rocky Linux 9.4

Bug description

I am not sure whether this a bug per se, but it is something that impacts agents performance significantly. When applying (simulate or step) with a set action while a line is disconnected, all substations that are adjacent to the line are rendered useless. This is the case because the substation reconf. actions always set all buses (inlcuding that of the disconnected powerline) which results in an error. This is particularly bad when a line adjacent to a large substation is disconnected, leading to invalidation of almost the whole action space.

How to reproduce

Code snippet

import grid2op
from lightsim2grid import LightSimBackend
backend = LightSimBackend()
env = grid2op.make("l2rpn_wcci_2022", backend=backend)
all_actions = env.action_space.get_all_unitary_topologies_set(env.action_space) # get all set actions 
obs = env.reset()

# disconnect line 0 (adjacent to substation 0)
change_status = env.action_space()
change_status.line_change_status = [0] # disconnect line 0
obs,_,_,_ = env.step(change_status)

#get set action of substation 0 
act_sub0 = all_actions[0] # a substation 0 action

print(env.simulate(act_sub0)) 
# Grid2OpException IllegalAction IllegalAction('Powerline with ids [0] have been modified illegally (cooldown of [3])'


... # Some code 

Current output

(<grid2op.Space.GridObjects.CompleteObservation_l2rpn_wcci_2022 object at 0x7efba377c2e0>, 0.0, False, {'disc_lines': array([-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
       -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
       -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
       -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
       -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
       -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
       -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
       -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
       -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
       -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
       -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1],
      dtype=int32), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'reason_alarm_illegal': None, 'reason_alert_illegal': None, 'opponent_attack_line': None, 'opponent_attack_sub': None, 'opponent_attack_duration': 0, 'exception': [Grid2OpException IllegalAction IllegalAction('Powerline with ids [0] have been modified illegally (cooldown of [3])')], 'time_series_id': None, 'rewards': {}})

Expected output

Ideally, not an IllegalAction exception.

[mhassouna95]

I implemented the following fix which significantly improved the performance of baseline agents. I will link to an arxiv submission soon, where the results are detailed. The idea is basically to remove the bus assignments of the disconnected lines.

def remove_bus_assignment_of_lines(obs, original_action, lines):
    """
    Removes bus assignments for given lines in the power grid.

    Parameters:
    - obs: Observation object containing grid state information.
    - original_action: Action object representing the action to be modified.
    - lines: List of line indices whose bus assignments should be removed.

    Returns:
    - A modified action object with bus assignments for the specified lines removed.
    """
    if len(lines) == 0:
        return original_action

    # get bus ids of each line
    or_bus = [obs.line_or_pos_topo_vect[i] for i in lines]
    ex_bus = [obs.line_ex_pos_topo_vect[i] for i in lines]
    
    # remove bus set actions of affected buses
    bus_assignment = original_action.set_bus.copy()
    bus_assignment[or_bus + ex_bus] = 0

    logging.debug(f"remove_bus_assignment_of_lines: {lines}")
    act = deepcopy(original_action)
    act.update({"set_bus": bus_assignment})
    return act

#call method before applying the action on the grid

[BDonnot]

Hello,

It is not a bug but a feature.

Grid2op does not (and can't) know what the agent really means. Maybe the agent did mean to reconnect the powerline, which is illegal :-/

This is why we introduced some "AgentWithHeuristics" in l2rpn baselines repository (though I am totally aware that this repo is old, not well maintained and probably not on par with grid2op from doc + test + code) for example here https://l2rpn-baselines.readthedocs.io/en/latest/utils.html#l2rpn_baselines.utils.GymEnvWithHeuristics or https://l2rpn-baselines.readthedocs.io/en/latest/utils.html#l2rpn_baselines.utils.GymEnvWithRecoWithDN which are briefly mentionned in the explantory notebooks (but I totally agree this is far from perfect...)

A method of the action actually exists to clean all these undesired behaviour:
https://grid2op.readthedocs.io/en/latest/user/action.html#grid2op.Action.BaseAction.remove_line_status_from_topo

It is more or less equivalent to what you are proposing but is part of the maintained grid2op API (so should not break if something is changed in the way grid2op process these at the moment)

[BDonnot]

So what you can do, to train an agent "transparently" using a gymnasium environment (which I assume you do) is:

class GymEnvCleanAction(GymEnvWithHeuristics):
    def fix_action(self, grid2op_action, g2op_obs):
        YOUR FUNCTION HERE
         OR
        res = copy.deepcopy(grid2op_action)
        res.remove_line_status_from_topo(g2op_obs)
        return res

And it should solve your issue :-) (this is if you want to use l2rpn-baselines of course) if you don't then you have to be aware that you need to implement twice your fixes:

• one during training when your "agent" is simply a RL agent you train and interacts with a gymnasium environment
• a second one during evaluation when your "agent" is a grid2op agent that will transform the observation to a gym observation, feed your trained RL agent with it, retrieve it's results and convert it to grid2op and then finally "fix" the action right before sending it to the grid2op environment

[mhassouna95]

Thanks a lot @BDonnot, it seems that I missed that, I will checkout the remove_line_status_from_topo method. Right now I am employing an imitation learning/supervised approach so I didnt need a gymnasium env yet. But I will definitely consider using l2rpn-baselines now to avoid doing things twice when I apply RL. Appeciate the help!