Commodity price calculations in models with circularities

[tsmbland]

When using cost-based ("marginal", "full", "marginal_average" and "full_average") pricing strategies, we have to calculate prices in order because the price of one commodity may affect the price of another. For example, if coal is used in the production of electricity, we must know the price of coal before we can calculate the price of electricity. In a directed-acyclic graph, the appropriate order can be found by doing a topological sort. This is what we're currently doing and it seems to be working well.

However, this is a problem for commodities with circular dependencies (for example, electricity and hydrogen). There's no "correct" order to calculate these prices as each one depends on the other. Currently, because of this, we disallow using cost-based pricing strategies for commodities in cycles (at the validation stage), limiting them to the "shadow" pricing strategy (or the lesser-used "scarcity" strategy), which don't have this problem as they are taken directly from the dispatch solution. This is a problem as, at least in some cases, we will want to use cost-based strategies for commodities in cycles.

In this discussion, I'll expand upon the problem of using cost-based pricing strategies for commodities with cyclical dependencies, try to formalise the scenarios where solutions can and cannot be found, and open up a dicussion for what to do in the latter case.

I think it's clearest to start with a simple example, so I'll focus on a simple 2-node cycle to start with, before generalising to any SCC.

Problem feasibility

Let's say we have the following two processes that form a cycle, with no external inputs (just to keep it simple for now, but I'll extend this later):

Process 1: 2A -> B
Process 2: 3B -> A

E.g. 2 units of electricity can produce 1 unit of hydrogen, and 3 units of hydrogen can be converted back to 1 unit of electricity (overall round trip of 6 units of electricity -> 1 unit of electricity).

Using the full-cost or marginal-cost pricing strategies, the prices of A and B (pA and pB) can be expressed as

pA = a + 3pB
pB = b + 2pA

where a/b cover all the determinants of pA and pB that are independent of the price of the other commodity (fixed costs, variable costs, capital costs, levies, flow costs, ...).

Solving this, we get

pA = - (a + 3b) / 5
pB = - (b + 2a) / 5

Which, since a and b must always be >=0, means that the equilibrium prices of A and B must be negative (or zero). Not ideal.

In general, pA = a + ypB, pB = b + xpA solves to

pA = (a + yb) / (1 - xy)
pB = (b + xa) / (1 - xy)

Where x/y are the coefficients in the conversion processes. Specifically x = coeff_in / coeff_out for the process setting pB from pA. (This is a simplified case; in the case where multiple internal/external processes contribute to pA and pB then there isn't such a clear link, as expanded on below in the "External contributions" section).

The above equations show that we'll get positive equilibrium prices for A and B only if xy (the product of the coefficients) is <1.

This is a problem for the electricity/hydrogen example, as any scenario with xy < 1 would be physically impossible without an external energy input (that would imply a round trip of 1 unit of electricity -> more than 1 unit of electricity). I could imagine this working in a different cycle with an external energy input (e.g. a small amount of electricity required to extract coal, which can be used to create more electricity than was put in).

Solving

As shown above, we can solve the system analytically to get prices (which may or may not be positive). Giving an example with a = 100, b = 50, x = 1 and y = 0.5, using the equilibrium equations above, we'd get pA = 250, pB = 300.

Another approach would be to iterate. Starting with pA = pB = 0, iteratively solving pA = a + ypB and pB = b + xpA, we'd get (using Jacobi iteration):

• Iteration 1: pA = 100, pB = 50
• Iteration 2: pA = 125, pB = 150
• Iteration 3: pA = 175, pB = 175
• ...
• Iteration 20: pA = 249.7070313, pB = 299.5117188

converging on the equilibrium prices. We could possibly speed up convergence using better starting conditions (e.g. shadow prices) or Gauss-Seidel iteration.

By comparison, if we changed x to 3 (so xy = 1.5), then the iterative approach would explode prices

• Iteration 1: pA = 100, pB = 50
• Iteration 2: pA = 125, pB = 350
• Iteration 3: pA = 275, pB = 425
• ...
• Iteration 20: pA = 14166.25977, pB = 39665.52734

External contributions

The above assumes that prices are set solely by processes within the cycle. In reality, most cycles will have external processes that may also contribute to the price of commodities within the cycle (e.g. another source of electricity such as wind turbines).

In the case of the "marginal_average" and "full_average" strategies, this depends on the relative contribution of each process to overall production, which may effectively bring down the value of x and/or y to a point where a solution is feasible.

For the "marginal" and "full" strategies, where prices are taken from the most expensive producer, the price of a commodity may end up being entirely independent of the cycle if a more expensive producer exists external to the cycle. Returning to the above example, if pA is a constant set by an external producer (e.g. wind turbines), then pB can be solved directly (pB = b + xpA). This has a positive solution for all b and x > 0 (in essence, y becomes zero, so xy is always zero).

It's impossible to know beforehand the relative contributions of internal/external processes to prices within the cyle, as this depends on run-time information. If both commodities in the cycle are set using cost-based approaches, and the product of the coefficients is >1, then all we can know beforehand is that a solution might be infeasible.

Note: shadow prices are also a case of prices being independent from the cycle, as these are pre-determined from the dispatch optimisation. If pA uses the shadow pricing strategy, then, similar to the above, y effectively becomes zero so a solution can be found. Unlike the case where all commodities use a cost-based strategy, this can be known beforehand.

Generalising to larger cycles

Ths can be extended to an arbitrary strongly-connected-component (SCC): the equilibrium solution will be positive if the spectral radius (max absolute eigenvalue) of the gain matrix is less than 1 (ρ(M) < 1).

The gain matrix is a matrix where $x_{ij}$ represents the (maximum) contribution of the price of i to the price of j. The most conservative approach is to use conversion coefficients of the processes making up the cycle, excluding any shadow-priced commodities.

For a 2-node cycle, ρ(M) = sqrt(xy), so ρ(M) < 1 ⟺ xy < 1

Outlook

For any SCC within MUSE, this gives us one of two scenarios:

1. The spectral radius of the gain matrix (excluding shadow-priced commodities) is < 1. In this case we know that a positive solution will be feasible. We safely can go ahead and calculate prices either by solving the system analytically or by iterating until convergence. Realistically, it would be the latter, as the equations would get complicated for large systems or systems with external inputs. Presumably this could be quite quick with care over starting conditions and the iteration approach, although we'd want to put an upper limit on iterations which would hopefully never be reached.

2. The spectral radius >=1. If there are no external processes then this will not work. Most likely, there will be some external processes, and in this case all be can know is that it might not work. The question is then what to do in this case?

   • We could go ahead anyway, and hope that the pricing strategies end up relying on processes external to the cycle (edit: I think timeslice-level buffering could also help stabilise prices, effectively buffering the gain matrix, but again, it's impossible to know this upfront). In this case we would iterate, and terminate the simulation if prices end up spiralling. This isn't ideal though, as there's a high likelihood of simulations failing part-way through, and ideally we want to be upfront with detecting infeasible models.

   • We could flag this at the validation stage, and instruct users to change the pricing strategy for one or more commodities to "shadow". Downsides are that this could take a few attempts to get something that works (it may not be enough just to change one commodity to shadow pricing), and shadow pricing may not be suitable anyway.

   • We could come up with other, more nuanced pricing strategies. For example, a way to exclude certain processes from setting commodity prices. This seems like a better long-term solution, although the above two may be feasible in the short term.

   • Completely open to other ideas and thoughts! @ahawkes @alexdewar

[ahawkes]

Thanks @tsmbland for the thorough investigation.

Overall, what we want is price formation that reflects reality as much as possible. Cycles that exist in reality are mainly of two types; (1) storage to move energy between time slices, and (2) use of downstream commodities in upstream processes where the aim is not storage. In general (1) is a small cycle like your H2/electricity example (but I guess not always). (2) can be many things, including cycles that encompass large chunks of the overall system, like electricity being used in oil extraction.

Here is a half-baked idea: For both of these cases, can price can be estimated by going around the cycle just once, and then ignoring further feedbacks?

The first step would be to identify the cycles in the graph.

We would then put these cycles in order, from largest (i.e. encompassing most commodities/processes) to smallest.

We would then calculate prices for everything until we get to the first cycle (the largest one), and then go back to the top of that cycle and repeat. Keep going with this process until every cycle has been done just once, and then complete, with prices for all commodities in the system (and missing commodities) as needed for the investment tools.

In your simple cycle above, one would get to the H2 commodity, and then re-calculate electricity price for the timeslices H2 is being used (complication here for storage processes as they consume in the cheap slices and output in the pricier slices). We would stop there (and not recalculate H2 price).

I realise this is no-where near as mathematically rigorous as your approach (eigenvalues!), but I think it's somewhat close to reality in important situations. In reality, agents wouldn't enter cycles where price will spiral. Or rather, if they did, the related commodity would be abandoned by downstream users?

[ahawkes]

P.S. Or, perhaps it would be better to order the cycles from the top downwards. i.e. the one that has the greatest potential to influence prices downstream would be the one with zenith closest to the top of the system. Would that be similar to the reverse order to the topological sort used in investment?

[tsmbland]

Hi @ahawkes, I've put something together to iterate over the commodities in cycles (#1322). Currently hardcoded just to go over once, but flexible enough to iterate multiple times if that's something we want to try. With the current example model (electricity-hydrogen loop) it stabilises after just two iterations (using the default "full_average" strategy for both commodities, with shadow prices as the starting condition), although I'm yet to try with more complex cycles and different strategies.

Thinking about it, I think you're right that agents shouldn't enter cycles where prices will spiral, or if they do then the dispatch shouldn't utilise them. I don't think that we can rely on this in general, as agent's won't always do the "correct" thing (e.g. due to constraints), but it might mean that price spiralling is a relatively niche concern.

In terms of ordering, we already do the topological sort of the whole system with cycles collapsed and calculate prices in that order (reverse of the investment order). I think you're referring about what order to run within each cycle? For investments, we calculate an order designed to minimise dependency mismatches, with a secondary bias to put commodities with flows going out of the cycle to the front. For the price calculation, I'm just doing this in reverse (i.e. commodities with outgoing flows last), which I think is what you're suggesting? This only really matters if we only iterate once - if we iterate multiple times then the order is less important.

[ahawkes]

Thanks @tsmbland that sounds promising.

Re prices cycles spiralling, I think maybe that would result in an dispatch optimisation ignoring the cycle, as any other option is an unbounded problem? If it's constrained to use the cycle, then there will be no cost minimum in dispatch?

Re ordering, not sure I follow. But sounds like a structured approach, which is something. I was just thinking that commodities further upstream generally have more bearing on prices downstream (more than the other way around).

It will be good to experiment with different pricing strategies and more complex models to see how robust this all is.

[tsmbland]

With the current model, "full_average" and "marginal_average" prices are stable, but "full" and "marginal" (prices taken from the most expensive producer) do spiral.

Could I check though - what should "full"/"marginal" do about processes that are active in the year but inactive in a particular timeslice? Should these still be able to contribute to the price in that timeslice or not? As it stands these can contribute to the price (based on the discussion in #1128), which I think may contribute to spiralling, because even if a cycle is inactive in a particular timeslice (presumably because it's not favourable to run in that timeslice) its mere existence can still elevate prices in that timeslice. Wanted to double check this.

Have also been trying to test out more complex cycles (e.g. using electricity in gas production), but not had much luck so far, which I think is ultimately down to limitations with the investment algorithm described in #1089

[ahawkes]

With the current model, "full_average" and "marginal_average" prices are stable, but "full" and "marginal" (prices taken from the most expensive producer) do spiral.

Ah. But not when you only do one cycle around the cycle.

Could I check though - what should "full"/"marginal" do about processes that are active in the year but inactive in a particular timeslice? Should these still be able to contribute to the price in that timeslice or not? As it stands these can contribute to the price (based on the discussion in #1128), which I think may contribute to spiralling, because even if a cycle is inactive in a particular timeslice (presumably because it's not favourable to run in that timeslice) its mere existence can still elevate prices in that timeslice. Wanted to double check this.

I think the discussion in #1128 was correct. i.e. if an asset exists, but is not contributing in a time slice (and can contribute in that time slice according to constraints), then we should take price from that asset (rather than a new candidate asset).

Have also been trying to test out more complex cycles (e.g. using electricity in gas production), but not had much luck so far, which I think is ultimately down to limitations with the investment algorithm described in #1089

OK, maybe not this issue?

[tsmbland]

With the current model, "full_average" and "marginal_average" prices are stable, but "full" and "marginal" (prices taken from the most expensive producer) do spiral.

Ah. But not when you only do one cycle around the cycle.

Yes, sorry should have clarified

Could I check though - what should "full"/"marginal" do about processes that are active in the year but inactive in a particular timeslice? Should these still be able to contribute to the price in that timeslice or not? As it stands these can contribute to the price (based on the discussion in #1128), which I think may contribute to spiralling, because even if a cycle is inactive in a particular timeslice (presumably because it's not favourable to run in that timeslice) its mere existence can still elevate prices in that timeslice. Wanted to double check this.

I think the discussion in #1128 was correct. i.e. if an asset exists, but is not contributing in a time slice (and can contribute in that time slice according to constraints), then we should take price from that asset (rather than a new candidate asset).

Ok, thanks for confirming

Have also been trying to test out more complex cycles (e.g. using electricity in gas production), but not had much luck so far, which I think is ultimately down to limitations with the investment algorithm described in #1089

OK, maybe not this issue?

Yeah, would be nice to test the pricing approach on more complex models to see how robust it really is, but the investment algorithm seems pretty limited in what it can handle for now.

Edit: Although, that said, we don't need investments to work if just looking at prices in the first year, so maybe there's something I can do here