{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interdepndent Network Restoration Decision-making (Complete Analysis Dashboard) \n",
    "This notebook finds restoration plans for synthetic or real-world interdependent \n",
    "networks subject to different initial seismic damage scenarios. Various restoration \n",
    "decision-making models are considered here:\n",
    "\n",
    "* Centralized methods: These methods solve one optimization problem for the whole \n",
    "interdependent network, which leads to the optimal restoration plan. Such models \n",
    "imply that the decision-maker is one entity that has the complete information and \n",
    "authority to restore all layers of the interdependent network. These methods include \n",
    "Interdependent Network Design Problem (INDP) [cite] and time-dependent INDP (td-INDP) [cite].\n",
    "\n",
    "* Decentralized methods: These methods model a multi-agent decision-making environment where\n",
    " each agent has the authority to restore a single layer and has complete information about \n",
    " her respective layer and minimal or no information about other layers. It is assumed that \n",
    " agents communicate poorly and not on time. This kind of methods includes Judgment Call (JC)\n",
    " method [cite] with and without Auction-based resource allocations [cite] and Interdependent \n",
    " Network Restoration Simultaneous Games (INRSG) and Bayesian Games (INRBG) [cite]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Directory changed to C:/Users/ht20/Documents/GitHub/td-DINDP/pyindp\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import matplotlib.pyplot as plt\n",
    "import runutils\n",
    "import dindputils\n",
    "import gameutils\n",
    "\n",
    "try:\n",
    "    # Change the current working Directory\n",
    "    DIR_MAIN = 'C:/Users/ht20/Documents/GitHub/td-DINDP/pyindp'\n",
    "    os.chdir(DIR_MAIN)\n",
    "    print(\"Directory changed to \"+DIR_MAIN)\n",
    "except OSError:\n",
    "    print(\"Can't change the Current Working Directory\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " \n",
    "## Run a toy example using different methods "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plt.close('all')\n",
    "# runutils.run_sample_problems()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " \n",
    "## Run different methods for different networks, post-process the outputs, and plot the result\n",
    "### Input/Output file addresses\n",
    "1. `BASE_DIR`: the address of the folder where the basic network information\n",
    "\t(topology, parameters, etc.) is stored\n",
    "2. `DAMAGE_DIR`: the address of the folder where the damage information is stored\n",
    "3. `OUTPUT_DIR`: the address of the folder where the output files should be written\n",
    "4. `FILTER_SCE`(optional): The address of the list of scenarios that should be \n",
    "\tincluded in the analyses. It is used to remove less damaging scenarios from\n",
    "\tthe list of damage scenarios. Set it to *None* if you don't want to use this option.\n",
    "5. `PAYOFF_DIR`(optional, only for Games): The address of the folder that contains the objects\n",
    "\tthat store the payoff values for the game so that they are read from file and\n",
    "\tnot calculated again. Set it to *None* if you don't want to use this option."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "BASE_DIR = '/home/hesam/Desktop/Files/Generated_Network_Dataset_v4/'\n",
    "# '/home/hesam/Desktop/Files/Generated_Network_Dataset_v4/'\n",
    "# \"../data/Extended_Shelby_County/\"\n",
    "# 'C:/Users/ht20/Box Sync/Shelby County Database/Node_arc_info'\n",
    "# \"C:\\\\Users\\\\ht20\\\\Documents\\\\Files\\\\Generated_Network_Dataset_v3.1\\\\\"\n",
    "# \"/home/hesam/Desktop/Files/Generated_Network_Dataset_v3.1\"\n",
    "\n",
    "DAMAGE_DIR = '/home/hesam/Desktop/Files/Generated_Network_Dataset_v4/'\n",
    "# '/home/hesam/Desktop/Files/Generated_Network_Dataset_v4/'\n",
    "# ../data/random_disruption_shelby/\"\n",
    "#\"../data/Wu_Damage_scenarios/\" \n",
    "# \"C:\\\\Users\\\\ht20\\\\Documents\\\\Files\\\\Generated_Network_Dataset_v3.1\\\\\"\n",
    "# \"/home/hesam/Desktop/Files/Generated_Network_Dataset_v3.1\"\n",
    "# 'C:/Users/ht20/Box Sync/Shelby County Database/Damage_scenarios'\n",
    "\n",
    "OUTPUT_DIR = '../results/'\n",
    "# '/home/hesam/Desktop/Files/Game_Shelby_County/results_NE/'\n",
    "# 'C:/Users/ht20/Documents/Files/Game_Shelby_County/results_0.9_perc/'\n",
    "# 'C:/Users/ht20/Documents/Files/Auction_Extended_Shelby_County_Data/results/'\n",
    "# '../results/'\n",
    "# 'C:/Users/ht20/Documents/Files/Auction_synthetic_networks_v3.1/'\n",
    "# 'C:/Users/ht20/Documents/Files/Shelby_data_paper/Restoration_results/'\n",
    "# FAIL_SCE_PARAM['TOPO']+'/results/'\n",
    "\n",
    "FILTER_SCE = '../data/damagedElements_sliceQuantile_0.90.csv'\n",
    "\n",
    "\n",
    "### Directory with objects containing payoff values for games\n",
    "PAYOFF_DIR = OUTPUT_DIR+'General/results/reudced_action_matrix_100/'\n",
    "#'/home/hesam/Desktop/Files/Game_Shelby_County/results_NE_only_objs/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " \n",
    "### Set analysis dictionaries \n",
    "1. `FAIL_SCE_PARAM`: stores information on the type of the damage scenario and \n",
    "    network dataset. This dictionary should have the following items:\n",
    "\t1. `TYPE`: type of the network. Options are `shelby` for the infrastructure network \n",
    "\tof Shelby County, TN, and `synthetic` for the synthetic interdependent dataset.\n",
    "\t2. `MAGS`: the damage scenarios for all datasets come in a two-level format.\n",
    "\tThe implication of each level is different, as explained below. `MAGS` sets \n",
    "\tthe range of the scenarios in the first level that should be included in the \n",
    "\tanalysis.\n",
    "\t3. `SAMPLE_RANGE`: sets the range of the scenarios in the second level that \n",
    "    should be included in the analysis.\n",
    "\t4. `BASE_DIR`: sets the folder where the basic network information is stored.\n",
    "\t5. `DAMAGE_DIR`: sets the folder where the damage information is stored\n",
    "\t6. `FILTER_SCE` (optional): sets a given list of scenarios that should be \n",
    "    included in the analyses and exclude the rest (mostly used with **WU format** below). \n",
    "\t7. `TOPO` (only when `TYPE`=*synthetic*): sets the topology of the synthetic \n",
    "    networks that should be analyzed\n",
    "\t<br><br>\n",
    "\tWhen the network dataset is the infrastructure network of Shelby County, TN,\n",
    "\tthere are three formats for network data and damage scenarios files:\n",
    "\t* **ANDRES**: this the old format that Andres Gonzalez employed during the \n",
    "\tdevelopment of INDP, and is based on the input data for Xpress software.\n",
    "\tThe network data that are available in this format are the first version Shleby \n",
    "\tCounty dataset [cite] and the damage data are 1000 realizations of \n",
    "\thazard maps correponding to hypothetical earthquakes with Magnitudes 6 to 9\n",
    "\toccuring at a specific epicenter [cite] . To use this format, set the dictionary to:<br>\n",
    "\t`{'TYPE':\"ANDRES\", 'SAMPLE_RANGE':range(1, 1001), 'MAGS':[6, 7, 8, 9], 'FILTER_SCE':None,\n",
    "\t  'BASE_DIR':BASE_DIR, 'DAMAGE_DIR':DAMAGE_DIR}`<br>\n",
    "\tHere, the range of magnitudes in the analysis is set by `MAGS`, and for each magnitude,\n",
    "\tthe range of analyzed samples is set by `SAMPLE_RANGE`.\n",
    "\t* **WU**: this is the new format that is designed by Hesam Talebiyan and\n",
    "\tused in the Shelby County data paper [cite]. The damage data for this dataset comes\n",
    "\tin a format similar to the hazard maps from Jason Wu [cite], which consist of N\n",
    "\tsets (`SAMPLE_RANGE`) of M damage scenarios (`MAGS`). For shelby county, for example,\n",
    "\tN=50 and M=96. To use this format, set the dictionary to:<br>\n",
    "\t`{'TYPE':\"WU\", 'SAMPLE_RANGE':range(50), 'MAGS':range(96),'FILTER_SCE':FILTER_SCE,\n",
    "\t  'BASE_DIR':BASE_DIR, 'DAMAGE_DIR':DAMAGE_DIR}`\n",
    "\t* **from_csv**: this type uses the same network data format as the **WU format**.\n",
    "\tHowever, the damage data come in the form of two csv files that contain all damage data\n",
    "\tfor nodes and arcs. This is a more compressed representation of damage data. In this \n",
    "    format, \tthere is only one `MAGS`=0, and `SAMPLE_RANGE` defines all scenarios that \n",
    "    should be analyzed. \tTo use this format, set the dictionary to:<br>\n",
    "\t`{'TYPE':\"from_csv\", 'SAMPLE_RANGE':range(100), 'MAGS':range(0, 1), 'FILTER_SCE':None,\n",
    "\t  'BASE_DIR':BASE_DIR, 'DAMAGE_DIR':DAMAGE_DIR}`\n",
    "\t<br><br>\n",
    "\tWhen the network dataset is synthetic, there is one format for network data and\n",
    "\tdamage scenarios files:<br><br>\n",
    "\t* **synthetic**: in this format, network data and damage data are in the same folder, and\n",
    "\thence, `BASE_DIR`= `DAMAGE_DIR`. Also, `MAGS` represents the range of network configuration, and \n",
    "\t`SAMPLE_RANGE` sets the range of sample networks for each configuration in the analysis.\n",
    "\tTo use this format, set the dictionary to:<br>\n",
    "\t`{'TYPE':\"synthetic\", 'SAMPLE_RANGE':range(0, 1), 'MAGS':range(0, 100), 'FILTER_SCE':None,\n",
    "\t  'TOPO':'General', 'BASE_DIR':BASE_DIR, 'DAMAGE_DIR':DAMAGE_DIR}`\n",
    "\t<br><br>\n",
    "2. `DYNAMIC_PARAMS`: sets the features of the models that incorporate dynamic parameters \n",
    "    into the analysis. Set it to *None* if you want to use static parameters that are \n",
    "    constant for different time steps. So far, we only have one type of dynamic parameters,\n",
    "\twhich is the dynamic demand that is calculated based on population dislocation models, \n",
    "\tfor which, the dictionary should have the following items:\n",
    "\t1. `TYPE`: type of the dislocation data (see below).\n",
    "\t2. `RETURN`: type of the model for the return of the dislocated population. Options\n",
    "\tare *step_function* and *linear*.\n",
    "\t3. `DIR`: sets the folder where the dislocation data are stored.\n",
    "\t4. `TESTBED` (only when `TYPE`=*incore*) : sets the name of the testbed in analysis.\n",
    "\t<br><br>\n",
    "\tThe are two types of dislocation data:\n",
    "\t* **shelby_adopted**: this is a precalculated dictionary that stores stylized \n",
    "\tdislocation data for the Shelby County dataset, and the code reads those files.\n",
    "\tTo use this type, set the dictionary to:<br> \n",
    "\t`{'TYPE': 'shelby_adopted', 'RETURN': 'step_function', \n",
    "       'DIR': 'C:/Users/ht20/Documents/Files/dynamic_demand/'}`\n",
    "\t* **incore**: this type uses the population dislocation models and household\n",
    "\tunit allocation data from IN-CORE (stored locally) to calculate demand values \n",
    "\tin each time step of the analysis. To use this type, set the dictionary to:<br>\n",
    "\t`{'TYPE': 'incore', 'RETURN': 'step_function', 'TESTBED':'Joplin', \n",
    "       'DIR': 'C:/Users/ht20/Documents/GitHub/NIST_testbeds/'}`\n",
    "\t<br><br>\n",
    "3. `STM_MODEL_DICT`: contains information about the statistical models approximating \n",
    "    INDP used for valuation methods in auction-based resource allocation. Set it to *None*\n",
    "\tif `VAL_TYPE` does not include *STM* (see below). Otherwise, the dictionary should have\n",
    "\tthe following items:\n",
    "\t1. `num_pred`: number of model predictions that are used to calculate each valuation.\n",
    "\t2. `model_dir`: the folder that contains the statistical model files.\n",
    "\t3. `param_folder`: the folder that contains the statistical model parameters.<br>\n",
    "\tExample: <br>\n",
    "\t`MODEL_DIR = 'C:/Users/ht20/Documents/Files/STAR_models/Shelby_final_all_Rc'\n",
    "\tSTM_MODEL_DICT = {'num_pred':1, 'model_dir':MODEL_DIR+'/traces', \n",
    "    'param_folder':MODEL_DIR+'/parameters'}`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "FAIL_SCE_PARAM = {'TYPE':\"synthetic\", 'SAMPLE_RANGE':range(0, 1), 'MAGS':range(0, 100),\n",
    "\t\t\t\t  'FILTER_SCE':None, 'TOPO':'General', 'BASE_DIR':BASE_DIR,\n",
    "\t\t\t\t  'DAMAGE_DIR':DAMAGE_DIR}\n",
    "DYNAMIC_PARAMS = None\n",
    "STM_MODEL_DICT = None\n",
    "\n",
    "# Adjust output and base dir for sythetic database based on `FAIL_SCE_PARAM`\n",
    "SYNTH_DIR = None\n",
    "if FAIL_SCE_PARAM['TYPE'] == 'synthetic':\n",
    "\tSYNTH_DIR = BASE_DIR+FAIL_SCE_PARAM['TOPO']+'Networks/'\n",
    "\tOUTPUT_DIR += FAIL_SCE_PARAM['TOPO']+'/results/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " \n",
    "### Set analysis parameters \n",
    "1. `RC`: list of resource caps or the number of available reousrces in each step of the \n",
    "\tanalysis. \n",
    "\t* If `FAIL_SCE_PARAM[TYPE']`=*synthetic*, this item is not necessaary. `R_c` is\n",
    "\tadjusted for each configuration. Set it to to `R_c`=[0]\n",
    "\t* If `FAIL_SCE_PARAM[TYPE']`=*shelby*, you have to options.\n",
    "\t\t* if, for example, `R_c`= [3, 6, 8, 12], then the analysis is done for the cases\n",
    "\t\twhen threr are 3, 6, 8, and 12 resources available (total reource assignment).\n",
    "\t\t* if, for example, `R_c`= [[1, 1], [1, 2], [3, 3]] and given there are 2 layers,\n",
    "\t\tthen the analysis is done for the case where each layer gets 1 resource, AND\n",
    "\t\tthe case where layer 1 gets 1 and layer 2 gets 2 resources, AND \n",
    "\t\tthe case where each layer gets 3 resource (Prescribed resource for each layer).\n",
    "2. `LAYERS`: list of layers in the analysis. \n",
    "\t* If `FAIL_SCE_PARAM[TYPE']`=*synthetic*, this item is not necessaary. `LAYERS` is\n",
    "\tadjusted for each configuration. Set it to to `LAYERS`=[0]\n",
    "3. `JUDGE_TYPE`: list of judgment types that are used in JC method and/or computing valuations\n",
    "\tfor auction-based allocation [cite]. Options are *OPTIMISTIC*, *PESSIMISTIC*, *DEMAND*,\n",
    "\t*DET-DEMAND*, and *RANDOM*. \n",
    "4. `RES_ALLOC_TYPE`: list of resource allocation types that are used in JC method [cite].\n",
    "\tOptions are *MDA*, *MAA*, *MCA*, *UNIFORM*, and *OPTIMAL*. \n",
    "5. `VAL_TYPE`: list of valuation types that are used in auction-based resource allocation\n",
    "\tmethod [cite], i.e. when `RES_ALLOC_TYPE` includes at least one of the options *MDA*,\n",
    "\t*MAA*, or *MCA*.\n",
    "\tOptions are *DTC*, *DTC_uniform*, *MDDN*, *STM*, and *DTC-LP*. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "RC = [0]\n",
    "LAYERS = [1, 2, 3, 4]\n",
    "JUDGE_TYPE = [\"OPTIMISTIC\"]\n",
    "RES_ALLOC_TYPE = ['UNIFORM', 'OPTIMAL']\n",
    "VAL_TYPE = ['DTC']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " \n",
    "### Run method(s)\n",
    "There are five choices of method:\n",
    "1. `INDP`: runs Interdependent Network Restoration Problem (INDP) [cite]. To run this method,\n",
    "\tyou have to call:<br>\n",
    "\t`runutils.run_method(FAIL_SCE_PARAM, RC, LAYERS, method='INDP', output_dir=OUTPUT_DIR,\n",
    "\tmisc = {'DYNAMIC_PARAMS':DYNAMIC_PARAMS})`\n",
    "2. `TDINDP`: runs time-dependent INDP (td-INDP) [cite]. To run this method,\n",
    "\tyou have to call:<br>\n",
    "\t`runutils.run_method(FAIL_SCE_PARAM, RC, LAYERS, method='TDINDP', output_dir=OUTPUT_DIR,\n",
    "\tmisc = {'DYNAMIC_PARAMS':DYNAMIC_PARAMS}))`\n",
    "3. `JC`: runs Judgment Call (JC) method, which is a decentralized version of INDP [cite]. \n",
    "    To run this method,\tyou have to call:<br>\n",
    "\t`runutils.run_method(FAIL_SCE_PARAM, RC, LAYERS, method='JC', judgment_type=JUDGE_TYPE,\n",
    "\tres_alloc_type=RES_ALLOC_TYPE, valuation_type=VAL_TYPE,\n",
    "\toutput_dir=OUTPUT_DIR, dynamic_params=DYNAMIC_PARAMS,\n",
    "\tmisc = {'STM_MODEL':STM_MODEL_DICT, 'DYNAMIC_PARAMS':DYNAMIC_PARAMS})`\n",
    "4. `NORMALGAME`: runs Interdependent Network Restoration Normal Game (INRNG), which is a\n",
    "\tdecentralized version of INDP [cite]. To run this method, you have to call:<br>\n",
    "\t`runutils.run_method(FAIL_SCE_PARAM, RC, LAYERS, method='NORMALGAME', judgment_type=JUDGE_TYPE,\n",
    "\tres_alloc_type=RES_ALLOC_TYPE, valuation_type=VAL_TYPE, output_dir=OUTPUT_DIR, \n",
    "\tmisc = {'PAYOFF_DIR':PAYOFF_DIR, 'DYNAMIC_PARAMS':DYNAMIC_PARAMS,\n",
    "   'REDUCED_ACTIONS':'EDM'}`<br>\n",
    "\tHere, `misc['REDUCED_ACTIONS']` sets the heuristic method to reduce the number of actions of\n",
    "\teach player to add Bounded Rationality to the analysis. Options are *ER* for exhausting resources \n",
    "\tand *EDM* for the expert decision-maker.\n",
    "5. `BAYESGAME`: runs Interdependent Network Restoration Bayesian Game (INRBG), which is a\n",
    "\tdecentralized version of INDP [cite]. To run this method, you have to call:<br>\n",
    "\t`runutils.run_method(FAIL_SCE_PARAM, RC, LAYERS, method='BAYESGAME', judgment_type=JUDGE_TYPE,\n",
    "\tres_alloc_type=RES_ALLOC_TYPE, valuation_type=VAL_TYPE, output_dir=OUTPUT_DIR,\n",
    "\tmisc = {'PAYOFF_DIR':PAYOFF_DIR, 'DYNAMIC_PARAMS':DYNAMIC_PARAMS,\n",
    "\t\"SIGNALS\":{1:'C', 2:'C'}, \"BELIEFS\":{1:'U', 2:'U'},\n",
    "\t'REDUCED_ACTIONS':'EDM'}`<br>\n",
    "\tHere, `misc['SIGNALS']` sets the actual type of each player in the game. Options \n",
    "    are *C* for\tcooperative and *N* for non-cooperative.<br>\n",
    "\tAlso, `misc['BELIEFS']` sets the belief of each player in the game. Options are *U* for\n",
    "\tuniformed belief, *F* for false consensus bias, and *I* for inverse false consensus bias."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# runutils.run_method(FAIL_SCE_PARAM, RC, LAYERS, method='INDP', output_dir=OUTPUT_DIR,\n",
    "# \t\t\t\t\tmisc = {'DYNAMIC_PARAMS':DYNAMIC_PARAMS})\n",
    "# # runutils.run_method(FAIL_SCE_PARAM, RC, LAYERS, method='TDINDP', output_dir=OUTPUT_DIR,\n",
    "# #                     misc = {'DYNAMIC_PARAMS':DYNAMIC_PARAMS})\n",
    "# # runutils.run_method(FAIL_SCE_PARAM, RC, LAYERS, method='JC', judgment_type=JUDGE_TYPE,\n",
    "# #                     res_alloc_type=RES_ALLOC_TYPE, valuation_type=VAL_TYPE,\n",
    "# #                     output_dir=OUTPUT_DIR, dynamic_params=DYNAMIC_PARAMS,\n",
    "# #                     misc = {'STM_MODEL':STM_MODEL_DICT, 'DYNAMIC_PARAMS':DYNAMIC_PARAMS})\n",
    "# runutils.run_method(FAIL_SCE_PARAM, RC, LAYERS, method='NORMALGAME', judgment_type=JUDGE_TYPE,\n",
    "# \t\t\t\t\tres_alloc_type=RES_ALLOC_TYPE, valuation_type=VAL_TYPE, output_dir=OUTPUT_DIR, \n",
    "# \t\t\t\t\tmisc = {'PAYOFF_DIR':PAYOFF_DIR, 'DYNAMIC_PARAMS':DYNAMIC_PARAMS,\n",
    "# \t\t\t\t\t'REDUCED_ACTIONS':'EDM'})\n",
    "# runutils.run_method(FAIL_SCE_PARAM, RC, LAYERS, method='BAYESGAME', judgment_type=JUDGE_TYPE,\n",
    "# \t\t\tres_alloc_type=RES_ALLOC_TYPE, valuation_type=VAL_TYPE, output_dir=OUTPUT_DIR,\n",
    "# \t\t\tmisc = {'PAYOFF_DIR':PAYOFF_DIR, 'DYNAMIC_PARAMS':DYNAMIC_PARAMS,\n",
    "# \t\t\t\t\t\"SIGNALS\":{1:'C', 2:'C'}, \"BELIEFS\":{1:'U', 2:'U'},\n",
    "# \t\t\t\t\t'REDUCED_ACTIONS':'EDM'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " \n",
    "### Post-processing \n",
    "First, you have to set a few parameters and then call functions that read outputs\n",
    "and generate the pandas DataFrames that are needed for plotting the results.\n",
    "\n",
    "##### Post-processing parameters\n",
    "\n",
    "1. `COST_TYPES`: type of cost that should be used in processing the outputs. Options \n",
    "are *Total*, *Under Supply*, *Over Supply*, *Node*, *Arc*, *Flow*, *Space Prep*, *Under Supply Perc*.\n",
    "2. `REF_METHOD`: the method served as the reference in computing the relative performance\n",
    "and allocation gap. Usually, this is an optimal method like `indp` or `tdindp`. However,\n",
    "it can be any other method like `jc`, `ng`, or else.\n",
    "3. `METHOD_NAMES`: methods whose output should be read. Options are `indp`, `tdindp`, `jc`,\n",
    "`ng`, `dp_indp`, `dp_jc`, `bg????` (For example, `bgNCUI` means the Bayesian game with\n",
    "two players where the first player is non-cooperative and uses uninformative belief,\n",
    "and the second one is cooperative and uses the inverse false consensus belief).\n",
    "\n",
    "##### Post-processing functions\n",
    "\n",
    "1. `generate_combinations`: generate all the combination of outputs that should be read and\n",
    "save them in `COMBS` and `OPTIMAL_COMBS` lists.\n",
    "2. `read_results`: read results for combinations in `COMBS` and `OPTIMAL_COMBS` lists.\n",
    "3. `relative_performance`: computes relative performance measures for different combinations.\n",
    "4. `read_resourcec_allocation`: read the resource allocations by different methods and\n",
    "compute allocation gaps for different combinations.\n",
    "5. `read_run_time`: compute run time for different combinations.\n",
    "6. `analyze_NE`: analyze the characteristics of Nash equilibria for different combinations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# COST_TYPES = ['Total'] # 'Under Supply', 'Over Supply'\n",
    "# REF_METHOD = 'indp'\n",
    "# METHOD_NAMES = ['indp','ng', 'bgCCUU'] \n",
    "# # #'ng', 'jc', 'dp_indp', 'tdindp',\n",
    "# # #'bgNNNNUUUU','bgCCCCUUUU', 'bgCCNCUUUU', 'bgCCCCFFFF', 'bgNNNNFFFF', 'bgCCNCFFFF'\n",
    "# # #'bgCCCCIIII','bgNNNNIIII', 'bgCCNCIIII',\n",
    "\n",
    "# COMBS, OPTIMAL_COMBS = dindputils.generate_combinations(FAIL_SCE_PARAM['TYPE'],\n",
    "# \t\t\tFAIL_SCE_PARAM['MAGS'], FAIL_SCE_PARAM['SAMPLE_RANGE'], LAYERS,\n",
    "# \t\t\tRC, METHOD_NAMES, JUDGE_TYPE, RES_ALLOC_TYPE, VAL_TYPE,\n",
    "# \t\t\tlist_high_dam_add=FAIL_SCE_PARAM['FILTER_SCE'],\n",
    "# \t\t\tsynthetic_dir=SYNTH_DIR)\n",
    "\n",
    "# BASE_DF, objs = dindputils.read_results(COMBS, OPTIMAL_COMBS, COST_TYPES,\n",
    "# \t\t\t\t\t\t\t\t\troot_result_dir=OUTPUT_DIR, deaggregate=True)\n",
    "\n",
    "# LAMBDA_DF = dindputils.relative_performance(BASE_DF, COMBS, OPTIMAL_COMBS,\n",
    "# \t\t\t\t\t\t\t\t\t\tref_method=REF_METHOD, cost_type=COST_TYPES[0])\n",
    "# RES_ALLOC_DF, ALLOC_GAP_DF = dindputils.read_resourcec_allocation(BASE_DF, COMBS, OPTIMAL_COMBS,\n",
    "# \t\t\t\t\t\t\t\t\t\t\t\t\t\t\t  objs, root_result_dir=OUTPUT_DIR,\n",
    "# \t\t\t\t\t\t\t\t\t\t\t\t\t\t\t  ref_method=REF_METHOD)\n",
    "# RUN_TIME_DF = dindputils.read_run_time(COMBS, OPTIMAL_COMBS, objs, root_result_dir=OUTPUT_DIR)\n",
    "# ANALYZE_NE_DF = gameutils.analyze_NE(objs, COMBS, OPTIMAL_COMBS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " \n",
    "### Save Variables to file\n",
    "All dictionaries that are made in the postprocessing step are saved here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# OBJ_LIST = [COMBS, OPTIMAL_COMBS, BASE_DF, METHOD_NAMES, LAMBDA_DF,\n",
    "#             RES_ALLOC_DF, ALLOC_GAP_DF, RUN_TIME_DF, COST_TYPES, ANALYZE_NE_DF]\n",
    "\n",
    "# ## Saving the objects ###\n",
    "# with open(OUTPUT_DIR+'postprocess_dicts.pkl', 'wb') as f:\n",
    "#     pickle.dump(OBJ_LIST, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " \n",
    "### Plot results \n",
    "Plot functions use the dictionaries that are made in the postprocessing step to \n",
    "make output figures:\n",
    "1. `plot_performance_curves`: plots costs (in `COST_TYPES`) and unmet demand vs. time.\n",
    "2. `plot_seperated_perform_curves`: plots costs (in `COST_TYPES`) vs. time for each layer sepearately.\n",
    "3. `plot_relative_performance`: plots relative performances.\n",
    "4. `plot_auction_allocation`: plots reosurce allocation vs. time.\n",
    "5. `plot_relative_allocation`: plots allocation gaps.\n",
    "6. `plot_run_time`: plots run time vs. time.\n",
    "7. `plot_ne_analysis`: plots NE analysis measures vs. time (for games only).\n",
    "8. `plot_ne_cooperation`: plots action types vs. time (for games only).\n",
    "9. `plot_payoff_hist`: plots size of the payoff matrix vs. time (for games only)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.close('all')\n",
    "# ### Getting back the objects ###\n",
    "# with open(OUTPUT_DIR+'postprocess_dicts.pkl', 'rb') as f:\n",
    "#     [COMBS, OPTIMAL_COMBS, BASE_DF, METHOD_NAMES, LAMBDA_DF, RES_ALLOC_DF,\n",
    "#       ALLOC_GAP_DF, RUN_TIME_DF, COST_TYPE, ANALYZE_NE_DF] = pickle.load(f)\n",
    "\n",
    "# plots.plot_performance_curves(BASE_DF,\n",
    "#                               cost_type='Total', ci=95,\n",
    "#                               deaggregate=False, plot_resilience=False)\n",
    "\n",
    "# # plots.plot_seperated_perform_curves(BASE_DF, x='t', y='cost', cost_type='Total',\n",
    "# #                                     ci=95, normalize=False)\n",
    "\n",
    "# plots.plot_relative_performance(LAMBDA_DF, lambda_type='U')\n",
    "# # plots.plot_auction_allocation(RES_ALLOC_DF, ci=95)\n",
    "# # plots.plot_relative_allocation(ALLOC_GAP_DF, distance_type='gap')\n",
    "# # plots.plot_run_time(RUN_TIME_DF, ci=95)\n",
    "# plots.plot_ne_analysis(ANALYZE_NE_DF, ci=95)\n",
    "# plots.plot_ne_cooperation(ANALYZE_NE_DF, ci=95)\n",
    "# plots.plot_payoff_hist(ANALYZE_NE_DF, compute_payoff_numbers=True, outlier=False)\n",
    "\n",
    " #[(ANALYZE_NE_DF['auction_type']!='UNIFORM')]"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}