Feature updates and bug fixes for v0.3.2 patch release (#270)

* Remove extraneous 'param' files

* Fix spelling of output file

* Updated the Project.toml, README, and the doc pages to comment out and treat SCIP the same way as Gurobi and CPLEX, especially for Windows environment, until we have a stable SCIP deployment for Windows

* Remove extraneous column Line_Min_Flow_MW

* Fixed #222 by re-scaling network_expansion.csv

Outputs are now in MW and $, not the internal units of GW and M$.

* Use ModelScalingFactor, not two ad-hoc conversions

* CRM: update the zone membership

So that the zone is in the CRM as long as the dfCapRes is not zero.

* Fix JuMP interface in DPP

getdual and getvalue were deprecated.

* Updated instructions for MUMPS_seq_jll error

Auto-populated MUMPS_seq_jll v5.5.1 causes pre-compilation issues with solvers on Windows. Edited run instructions to downgrade MUMPS_seq_jll to v5.4.1 after instantiate.

* Update versions of proprietary solver interfaces

* Updated Project.toml 

To fix SCIP dependency issue breaking at CI

* Add write capacity factor function

* Move multi-stage write functions to their own files

Moved write_capacities_dicharge(), write_costs(), and write_stats() to their own files and renamed multi-stage write files to include "multi_stage" to differentiate them from other write files.

* Rename write_settings to write_multi_stage_settings

To be consistent with other files and to differentiate multi-stage write files from single-stage write files.

* New multi-stage write files

Wrote analogous write files for charge capacity, energy capacity, and transmission capacity based on write_multi_stage_capacities_discharge(). Feel free to expand upon network expansion output writing later.

* Update GenX.jl

To incorporate write_multi_stage_settings.jl instead of write_settings.jl

* Fixed Syntax error in GenX.jl

Gotten rid of the extra "/" in include("multi_stage/write_multi_stage_settings.jl")

Co-authored-by: Jacob Schwartz <jacob@jaschwartz.net>
Co-authored-by: Qingyu Xu <qingyux@princeton.edu>
Co-authored-by: wrgunther <112493531+wrgunther@users.noreply.github.com>
Co-authored-by: GenXProject <84470580+GenXProject@users.noreply.github.com>
Co-authored-by: jfmorris <jfmorris@mit.edu>

Author: 6 people

Project.toml

@@ -35,11 +35,11 @@ Cbc = "1.0.1"
 Clp = "1.0.1"
 Clustering = "0.14.2"
 Combinatorics = "1.0.2"
-#CPLEX ="0.6.1"
+#CPLEX ="0.9.3"
 DataFrames = "1.3.4"
 DataStructures = "0.18.13"
 Distances = "0.10.7"
-#Gurobi ="0.9.14"
+#Gurobi ="0.11.3"
 HiGHS = "1.1.4"
 JuMP = "1.1.1"
 MathOptInterface = "1.6.1"

README.md

@@ -78,12 +78,13 @@ For this tutorial it will be assumed to be within your home directory: `/home/yo
 
 2. Start a terminal and navigate into the `GenX` folder.
 3. Type `julia --project=.` to start an instance of the `julia` kernel with the `project` set to the current folder.
-The `.` indicates the current folder.
+The `.` indicates the current folder. On Windows the location of Julia can also be specified as e.g., 'C:\julia-1.6.0\bin\julia.exe --project=.'
 
 If it's your first time running GenX (or, if you have pulled after some major upgrades/release/version) execute steps 3-6.
 
 4. Type `]` to bring up the package system `(GenX) pkg >` prompt. This indicates that the GenX project was detected. If you see `(@v1.6) pkg>` as the prompt, then the `project` was not successfully set.
 5. Type `instantiate` from the `(GenX) pkg` prompt.
+   On Windows there is an issue with the prepopulated MUMPS_seq_jll v5.5.1 that prevents compilation of the solvers. To avoid this issue type 'add MUMPS_seq_jll@5.4.1' after running instantiate.
 6. Type `st` to check that the dependecies have been installed. If there is no error, it has been successful.
 7. Type the back key to come back to the `julia>` prompt.
 

docs/src/how_to_run_genx.md

@@ -36,12 +36,13 @@ For this tutorial it will be assumed to be within your home directory: `/home/yo
 
 2. Start a terminal and navigate into the `GenX` folder.
 3. Type `julia --project=.` to start an instance of the `julia` kernel with the `project` set to the current folder.
-The `.` indicates the current folder.
+The `.` indicates the current folder. On Windows the location of Julia can also be specified as e.g., 'C:\julia-1.6.0\bin\julia.exe --project=.'
 
 If it's your first time running GenX (or, if you have pulled after some major upgrades/release/version) execute steps 3-6.
 
 4. Type `]` to bring up the package system `(GenX) pkg >` prompt. This indicates that the GenX project was detected. If you see `(@v1.6) pkg>` as the prompt, then the `project` was not successfully set.
 5. Type `instantiate` from the `(GenX) pkg` prompt.
+   On Windows there is an issue with the prepopulated MUMPS_seq_jll v5.5.1 that prevents compilation of the solvers. To avoid this issue type 'add MUMPS_seq_jll@5.4.1' after running instantiate.
 6. Type `st` to check that the dependecies have been installed. If there is no error, it has been successful.
 7. Type the back key to come back to the `julia>` prompt.
 
@@ -106,7 +107,6 @@ julia> <press close-bracket ] to access the package manager>
 -or-
 (GenX) pkg> add CPLEX
 ```
-
 3. At the beginning of the `GenX/src/GenX.jl` file, uncomment `using Gurobi` and/or `using CPLEX`.
 4. Set the appropriate solver in the `genx_settings.yml` file of your case
 

src/GenX.jl

@@ -147,6 +147,7 @@ include("model/solve_model.jl")
 
 include("write_outputs/dftranspose.jl")
 include("write_outputs/write_capacity.jl")
+include("write_outputs/write_capacityfactor.jl")
 include("write_outputs/write_charge.jl")
 include("write_outputs/write_charging_cost.jl")
 include("write_outputs/write_costs.jl")
@@ -197,7 +198,7 @@ include("simple_operation.jl")
 include("multi_stage/dual_dynamic_programming.jl")
 include("multi_stage/configure_multi_stage_inputs.jl")
 include("multi_stage/endogenous_retirement.jl")
-include("multi_stage//write_settings.jl")
+include("multi_stage/write_multi_stage_settings.jl")
 
 include("additional_tools/modeling_to_generate_alternatives.jl")
 include("additional_tools/method_of_morris.jl")

src/model/core/non_served_energy.jl

@@ -87,7 +87,7 @@ function non_served_energy!(EP::Model, inputs::Dict, setup::Dict)
 	# Capacity Reserves Margin policy
 	if setup["CapacityReserveMargin"] > 0
 		if SEG >=2
-			@expression(EP, eCapResMarBalanceNSE[res=1:inputs["NCapacityReserveMargin"], t=1:T], sum(EP[:vNSE][s,t,z] for s in 2:SEG, z in findall(x->x>0,inputs["dfCapRes"][:,res])))
+			@expression(EP, eCapResMarBalanceNSE[res=1:inputs["NCapacityReserveMargin"], t=1:T], sum(EP[:vNSE][s,t,z] for s in 2:SEG, z in findall(x->x!=0,inputs["dfCapRes"][:,res])))
 			EP[:eCapResMarBalance] += eCapResMarBalanceNSE
 		end
 	end

src/model/policies/cap_reserve_margin.jl

@@ -35,6 +35,6 @@ function cap_reserve_margin!(EP::Model, inputs::Dict, setup::Dict)
 
 	@constraint(EP, cCapacityResMargin[res=1:inputs["NCapacityReserveMargin"], t=1:T], EP[:eCapResMarBalance][res, t]
 				>= sum(inputs["pD"][t,z] * (1 + inputs["dfCapRes"][z,res])
-				for z=findall(x->x>0,inputs["dfCapRes"][:,res])))
+				for z=findall(x->x!=0,inputs["dfCapRes"][:,res])))
 
 end

src/multi_stage/dual_dynamic_programming.jl

@@ -65,7 +65,7 @@ end
 @doc raw"""
 	function run_ddp(models_d::Dict, setup::Dict, inputs_d::Dict)
 
-This function run the dual dynamic programming (DDP) algorithm, as described in [Pereira and Pinto (1991)](https://doi.org/10.1007/BF01582895), and more recently, [Lara et al. (2018)](https://doi.org/10.1016/j.ejor.2018.05.039). Note that if the algorithm does not converge within 10,000 (currently hardcoded) iterations, this function will return models with sub-optimal solutions. However, results will still be printed as if the model is finished solving.
+This function run the dual dynamic programming (DDP) algorithm, as described in [Pereira and Pinto (1991)](https://doi.org/10.1007/BF01582895), and more recently, [Lara et al. (2018)](https://doi.org/10.1016/j.ejor.2018.05.039). Note that if the algorithm does not converge within 10,000 (currently hardcoded) iterations, this function will return models with sub-optimal solutions. However, results will still be printed as if the model is finished solving. This sub-optimal termination is noted in the output with the 'Exiting Without Covergence!' message.
 
 inputs:
 
@@ -276,141 +276,13 @@ function write_multi_stage_outputs(stats_d::Dict, outpath::String, settings_d::D
 
     multi_stage_settings_d = settings_d["MultiStageSettingsDict"]
 
-    write_capacities_discharge(outpath, multi_stage_settings_d)
-    #write_capacities_charge(outpath, multi_stage_settings_d)
-    #write_capacities_energy(outpath, multi_stage_settings_d)
-    #write_network_expansion(outpath, multi_stage_settings_d)
-    write_costs(outpath, multi_stage_settings_d, inputs_dict)
-    write_stats(outpath, stats_d)
-    write_settings(outpath, settings_d)
-
-end
-
-@doc raw"""
-	function write_capacities_discharge(outpath::String, settings_d::Dict)
-
-This function writes the file capacities\_multi\_stage.csv to the Results directory. This file contains starting resource capcities from the first model stage and end resource capacities for the first and all subsequent model stages.
-
-inputs:
-
-  * outpath – String which represents the path to the Results directory.
-  * settings\_d - Dictionary containing settings dictionary configured in the multi-stage settings file multi\_stage\_settings.yml.
-"""
-function write_capacities_discharge(outpath::String, settings_d::Dict)
-    # TO DO - DO THIS FOR ENERGY CAPACITY AS WELL
-
-    num_stages = settings_d["NumStages"] # Total number of investment planning stages
-    capacities_d = Dict()
-
-    for p in 1:num_stages
-        inpath = joinpath(outpath, "Results_p$p")
-        capacities_d[p] = DataFrame(CSV.File(joinpath(inpath, "capacity.csv"), header=true), copycols=true)
-    end
-
-    # Set first column of DataFrame as resource names from the first stage
-    df_cap = DataFrame(Resource=capacities_d[1][!, :Resource], Zone=capacities_d[1][!, :Zone])
-
-    # Store starting capacities from the first stage
-    df_cap[!, Symbol("StartCap_p1")] = capacities_d[1][!, :StartCap]
-
-    # Store end capacities for all stages
-    for p in 1:num_stages
-        df_cap[!, Symbol("EndCap_p$p")] = capacities_d[p][!, :EndCap]
-    end
-
-    CSV.write(joinpath(outpath, "capacities_multi_stage.csv"), df_cap)
-
-end
-
-function write_capacities_charge(outpath::String, settings_d::Dict) end
-
-function write_capacities_energy(outpath::String, settings_d::Dict) end
-
-function write_network_expansion(outpath::String, settings_d::Dict)
-    # Include discounted NE costs and capacities for each model period
-end
-
-@doc raw"""
-	function write_costs(outpath::String, settings_d::Dict)
-
-This function writes the file costs\_multi\_stage.csv to the Results directory. This file contains variable, fixed, startup, network expansion, unmet reserve, and non-served energy costs discounted to year zero.
-
-inputs:
-
-  * outpath – String which represents the path to the Results directory.
-  * settings\_d - Dictionary containing settings dictionary configured in the multi-stage settings file multi\_stage\_settings.yml.
-"""
-function write_costs(outpath::String, settings_d::Dict, inputs_dict::Dict)
-
-    num_stages = settings_d["NumStages"] # Total number of DDP stages
-    wacc = settings_d["WACC"] # Interest Rate and also the discount rate unless specified other wise
-    stage_lens = settings_d["StageLengths"]
-    myopic = settings_d["Myopic"] == 1 # 1 if myopic (only one forward pass), 0 if full DDP
-
-    costs_d = Dict()
-    for p in 1:num_stages
-        cur_path = joinpath(outpath, "Results_p$p")
-        costs_d[p] = DataFrame(CSV.File(joinpath(cur_path, "costs.csv"), header=true), copycols=true)
-    end
-
-    OPEXMULTS = [inputs_dict[j]["OPEXMULT"] for j in 1:num_stages] # Stage-wise OPEX multipliers to count multiple years between two model stages
-
-    # Set first column of DataFrame as resource names from the first stage
-    df_costs = DataFrame(Costs=costs_d[1][!, :Costs])
-
-    # Store discounted total costs for each stage in a data frame
-    for p in 1:num_stages
-        if myopic
-            DF = 1 # DF=1 because we do not apply discount factor in myopic case
-        else
-            DF = 1 / (1 + wacc)^(stage_lens[p] * (p - 1))  # Discount factor applied to ALL costs in each stage
-        end
-        df_costs[!, Symbol("TotalCosts_p$p")] = DF .* costs_d[p][!, Symbol("Total")]
-    end
-
-    # For OPEX costs, apply additional discounting
-    for cost in ["cVar", "cNSE", "cStart", "cUnmetRsv"]
-        if cost in df_costs[!, :Costs]
-            df_costs[df_costs[!, :Costs].==cost, 2:end] = transpose(OPEXMULTS) .* df_costs[df_costs[!, :Costs].==cost, 2:end]
-        end
-    end
-
-    # Remove "cTotal" from results (as this includes Cost-to-Go)
-    df_costs = df_costs[df_costs[!, :Costs].!="cTotal", :]
-
-    CSV.write(joinpath(outpath, "costs_multi_stage.csv"), df_costs)
-
-end
-
-@doc raw"""
-	function write_stats(outpath::String, stats_d::Dict)
-
-This function writes the file stats\_multi\_stage.csv. to the Results directory. This file contains the runtime, upper bound, lower bound, and relative optimality gap for each iteration of the DDP algorithm.
-
-inputs:
-
-  * outpath – String which represents the path to the Results directory.
-  * stats\_d – Dictionary which contains the run time, upper bound, and lower bound of each DDP iteration.
-"""
-function write_stats(outpath::String, stats_d::Dict)
-
-    times_a = stats_d["TIMES"] # Time (seconds) of each iteration
-    upper_bounds_a = stats_d["UPPER_BOUNDS"] # Upper bound of each iteration
-    lower_bounds_a = stats_d["LOWER_BOUNDS"] # Lower bound of each iteration
-
-    # Create an array of numbers 1 through total number of iterations
-    iteration_count_a = collect(1:length(times_a))
-
-    realtive_gap_a = (upper_bounds_a .- lower_bounds_a) ./ lower_bounds_a
-
-    # Construct dataframe where first column is iteration number, second is iteration time
-    df_stats = DataFrame(Iteration_Number=iteration_count_a,
-        Seconds=times_a,
-        Upper_Bound=upper_bounds_a,
-        Lower_Bound=lower_bounds_a,
-        Relative_Gap=realtive_gap_a)
-
-    CSV.write(joinpath(outpath, "stats_multi_stage.csv"), df_stats)
+    write_multi_stage_capacities_discharge(outpath, multi_stage_settings_d)
+    write_multi_stage_capacities_charge(outpath, multi_stage_settings_d)
+    write_multi_stage_capacities_energy(outpath, multi_stage_settings_d)
+    write_multi_stage_network_expansion(outpath, multi_stage_settings_d)
+    write_multi_stage_costs(outpath, multi_stage_settings_d, inputs_dict)
+    write_multi_stage_stats(outpath, stats_d)
+    write_multi_stage_settings(outpath, settings_d)
 
 end
 
@@ -582,8 +454,8 @@ function generate_cut_component_track(EP_cur::Model, EP_next::Model, var_name::S
         y = k[1] # Index representing resource
         p = k[2] # Index representing stage
 
-        push!(next_dual_value, getdual(EP_next[constr_name][y, p]))
-        push!(cur_inv_value, getvalue(EP_cur[var_name][y, p]))
+        push!(next_dual_value, dual(EP_next[constr_name][y, p]))
+        push!(cur_inv_value, value(EP_cur[var_name][y, p]))
         push!(cur_inv_var, EP_cur[var_name][y, p])
     end
 

src/multi_stage/write_multi_stage_capacities_charge.jl

@@ -0,0 +1,50 @@
+"""
+GenX: An Configurable Capacity Expansion Model
+Copyright (C) 2021,  Massachusetts Institute of Technology
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+A complete copy of the GNU General Public License v2 (GPLv2) is available
+in LICENSE.txt.  Users uncompressing this from an archive may not have
+received this license file.  If not, see <http://www.gnu.org/licenses/>.
+"""
+
+@doc raw"""
+	function write_multi_stage_capacities_charge(outpath::String, settings_d::Dict)
+
+This function writes the file capacities\_charge\_multi\_stage.csv to the Results directory. This file contains starting resource charge capacities from the first model stage and end resource charge capacities for the first and all subsequent model stages.
+
+inputs:
+
+  * outpath – String which represents the path to the Results directory.
+  * settings\_d - Dictionary containing settings dictionary configured in the multi-stage settings file multi\_stage\_settings.yml.
+"""
+function write_multi_stage_capacities_charge(outpath::String, settings_d::Dict)
+
+    num_stages = settings_d["NumStages"] # Total number of investment planning stages
+    capacities_d = Dict()
+
+    for p in 1:num_stages
+        inpath = joinpath(outpath, "Results_p$p")
+        capacities_d[p] = DataFrame(CSV.File(joinpath(inpath, "capacity.csv"), header=true), copycols=true)
+    end
+
+    # Set first column of DataFrame as resource names from the first stage
+    df_cap = DataFrame(Resource=capacities_d[1][!, :Resource], Zone=capacities_d[1][!, :Zone])
+
+    # Store starting capacities from the first stage
+    df_cap[!, Symbol("StartChargeCap_p1")] = capacities_d[1][!, :StartChargeCap]
+
+    # Store end capacities for all stages
+    for p in 1:num_stages
+        df_cap[!, Symbol("EndChargeCap_p$p")] = capacities_d[p][!, :EndChargeCap]
+    end
+
+    CSV.write(joinpath(outpath, "capacities_charge_multi_stage.csv"), df_cap)
+
+end

src/multi_stage/write_multi_stage_capacities_discharge.jl

@@ -0,0 +1,50 @@
+"""
+GenX: An Configurable Capacity Expansion Model
+Copyright (C) 2021,  Massachusetts Institute of Technology
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+A complete copy of the GNU General Public License v2 (GPLv2) is available
+in LICENSE.txt.  Users uncompressing this from an archive may not have
+received this license file.  If not, see <http://www.gnu.org/licenses/>.
+"""
+
+@doc raw"""
+	function write_multi_stage_capacities_discharge(outpath::String, settings_d::Dict)
+
+This function writes the file capacities\_multi\_stage.csv to the Results directory. This file contains starting resource capacities from the first model stage and end resource capacities for the first and all subsequent model stages.
+
+inputs:
+
+  * outpath – String which represents the path to the Results directory.
+  * settings\_d - Dictionary containing settings dictionary configured in the multi-stage settings file multi\_stage\_settings.yml.
+"""
+function write_multi_stage_capacities_discharge(outpath::String, settings_d::Dict)
+
+    num_stages = settings_d["NumStages"] # Total number of investment planning stages
+    capacities_d = Dict()
+
+    for p in 1:num_stages
+        inpath = joinpath(outpath, "Results_p$p")
+        capacities_d[p] = DataFrame(CSV.File(joinpath(inpath, "capacity.csv"), header=true), copycols=true)
+    end
+
+    # Set first column of DataFrame as resource names from the first stage
+    df_cap = DataFrame(Resource=capacities_d[1][!, :Resource], Zone=capacities_d[1][!, :Zone])
+
+    # Store starting capacities from the first stage
+    df_cap[!, Symbol("StartCap_p1")] = capacities_d[1][!, :StartCap]
+
+    # Store end capacities for all stages
+    for p in 1:num_stages
+        df_cap[!, Symbol("EndCap_p$p")] = capacities_d[p][!, :EndCap]
+    end
+
+    CSV.write(joinpath(outpath, "capacities_multi_stage.csv"), df_cap)
+
+end