spartan (version 2.3)

Technique 5: SPARTAN and Netlogo: Technique 5: SPARTAN and Netlogo

Description

Versions 1.0-1.2 of SPARTAN focused on performing an analysis of simulations, but such demonstrations were never paired with tools that are used by researchers to develop such simulations. Version 1.3 offered the additional functionality of linking SPARTAN with Netlogo, allowing the user to automatically generate Netlogo experiment scripts using SPARTAN, then analysing the resultant simulation response using the techniques developed above. This makes use of Netlogo's BehaviourSpace feature, which allows the researcher to perform a sweep of potential parameter values, storing the results in a CSV table. The weakness with this feature is that the researcher is left to develop tools to analyse the simulation responses. Now, SPARTAN can process this table, generating a robust sensitivity analysis of the parameters of interest. On our laboratory website (www.ycil.org.uk), we have made available a slightly modified version of the Netlogo Virus model, and a tutorial demonstrating how SPARTAN can be used to develop parameter value samples for the techniques described above, and then used to analyse the resultant Netlogo simulations.

Note: Netlogo BehaviourSpace functionality can produce two forms of output: 'Spreadsheet' or 'Table'. This technique has been designed to work with the later. This is explained fully in the tutorial on our lab website.

The reader should make themselves familiar with the techniques described above, as this is not repeated here. The only difference is in the preparation of the Netlogo data: once this is done the methods described in Techniques 2-4 are applied.

The relevant Netlogo Parameter Robustness Methods (Technique 2) are described below. See Technique 2 for a full description of each method: oat_generate_netlogo_behaviour_space_XML: This generates a Netlogo XML Experiment file, to be used with the BehaviourSpace feature or Headless Netlogo, which perturbs each parameter over a set value space, as described in Technique 2. oat_process_netlogo_result: Takes the Netlogo behaviour space file and extracts the required timepoint information from it, storing this in a Spartan compatible CSV file. This CSV file is then processed using the methods described in Technique 2, with A-Test scores determined for each value assigned to each parameter. Once this method has been called, the researcher should use the oat_graphATestsForSampleSize and oat_plotResultDistribution methods of Technique 2 to graph the results.

The relevant Netlogo Latin-Hypercube Sampling and Analysis methods (Technique 3) are described below. See Technique 3 for a full description of each method: lhc_generate_lhc_sample_netlogo: This generates a specified number of simulation parameters sets using latin-hypercube sampling. These are then processed into Netlogo XML experiment files, one for each set of parameters.

lhc_process_netlogo_result: Takes each parameter value set generated by the hypercube in turn, and analyses the Netlogo simulation results. For each parameter set, there will be n simulation results. This method goes through these results, producing a file containing the median of each output measure for each of the n runs. Thus, if a Netlogo simulation was replicated 10 times, the median file will contain 10 medians for each simulation output measure. Once this has been created, the user should run the lhc_generateLHCSummary, lhc_generatePRCoEffs. and lhc_graphMeasuresForParameterChange methods of Technique 3.

The relevant eFAST methods (Technique 4) are desribed below. See Technique 4 for a full description of each method: efast_generate_sample_netlogo: Creates a set of parameter values, over the specified value space, as described in Technique 4. Then processes each of these into a Netlogo experiment XML file, from which a simulation can be run. efast_process_netlogo_result: Takes each parameter value set generated by eFAST in turn, and analyses the Netlogo simulation results. For each parameter set, there will be n simulation results. This method goes through these results, producing a file containing the median of each output measure for each of the n runs. Thus, if a Netlogo simulation was replicated 10 times, the median file will contain 10 medians for each simulation output measure. The user should then run the efast_get_overall_medians and efast_run_Analysis methods of Technique 4.

Note that for all three methods, it is important to run the PARAMETERS and MEASURES parameters through the method table_header_check first (see examples below), as Netlogo converts any spaces or hyphens in these names to periods. This method will take care of this for you.

Usage

oat_generate_netlogo_behaviour_space_XML(FILEPATH,
	NETLOGO_SETUPFILE_NAME,PARAMETERS,PARAMVALS,
	NETLOGO_SETUP_FUNCTION,NETLOGO_RUN_FUNCTION,MEASURES,
	EXPERIMENT_REPETITIONS,RUNMETRICS_EVERYSTEP)
oat_process_netlogo_result(FILEPATH,NETLOGO_BEHAVIOURSPACEFILE,
	PARAMETERS,BASELINE,PMIN,PMAX,PINC,MEASURES,
	RESULTFILENAME,ATESTRESULTSFILENAME,TIMESTEP)
lhc_generate_lhc_sample_netlogo(FILEPATH,PARAMETERS,
	PARAMVALS,NUMSAMPLES,ALGORITHM,
	EXPERIMENT_REPETITIONS,RUNMETRICS_EVERYSTEP,
	NETLOGO_SETUP_FUNCTION,NETLOGO_RUN_FUNCTION,MEASURES)
lhc_process_netlogo_result(FILEPATH,LHCSAMPLE_RESULTFILENAME,
	SPARTAN_PARAMETER_FILE,NUMSAMPLES,MEASURES,
	LHC_ALL_SIM_RESULTS_FILE,TIMESTEP)
efast_generate_sample_netlogo(FILEPATH,NUMCURVES,NUMSAMPLES,
	MEASURES,PARAMETERS,PARAMVALS,EXPERIMENT_REPETITIONS,
	RUNMETRICS_EVERYSTEP,NETLOGO_SETUP_FUNCTION,
	NETLOGO_RUN_FUNCTION)
efast_process_netlogo_result(FILEPATH,
	EFASTSAMPLE_RESULTFILENAME,PARAMETERS,NUMCURVES,
	NUMSAMPLES,MEASURES,RESULTFILENAME,TIMESTEP)

Arguments

FILEPATH

Directory where either the parameter samples are to be stored, or the simulation runs can be found

NETLOGO_SETUPFILE_NAME

Name to give, or given to, the Netlogo XML experiment file(s) created in sampling. For more than one, a sample number is appended

PARAMETERS

Array containing the names of the parameters of which parameter samples will be generated

PARAMVALS

Array containing either the parameter value (if not of interest in the analysis), or range under which this is being explored (stated as as string e.g. "[5,50,5]" for a range of 5-50, increment of 5. This will differ for each of the three techniques, so the reader should read the relevant tutorial in detail.

NETLOGO_SETUP_FUNCTION

The name of the function in Netlogo that sets up the simulation. Commonly is named setup.

NETLOGO_RUN_FUNCTION

The name of the function in Netlogo that starts the simulation. Commonly named go.

MEASURES

Array containing the names of the Netlogo output measures which are used to analyse the simulation.

EXPERIMENT_REPETITIONS

The number of times Netlogo should repeat the experiment for each set of parameter values.

RUNMETRICS_EVERYSTEP

Boolean stating whether Netlogo should produce output for each timestep.

NETLOGO_BEHAVIOURSPACEFILE

The name of the file produced by Netlogo for Parameter Robustness (Technique 2). This is the result file that is analysed.

BASELINE

Array containing the baseline, or calibrated value, of each parameter.

PMIN

Array containing the minimum value that should be used for each parameter. Sets a lower bound on sampling space.

PMAX

Array containing the maximum value that should be used for each parameter. Sets an upper bound on sampling space.

PINC

Array containing the increment value that should be applied for each parameter. For example, a parameter could have a minimum value of 10, and maximum value of 100, and be incremented by 10.

RESULTFILENAME

In Technique 2, where SPARTAN is producing files showing the modified Netlogo CSV file, the name of the file that should be produced. For the other techniques, this is the name of the result file produced by Netlogo.

TIMESTEP

For Technique 2, the timestep of the Netlogo simulation being analysed.

ATESTRESULTSFILENAME

File name of the ATests result summary file created by oat_netlogo_analyseAllParams.

NUMSAMPLES

The number of parameter subsets that were generated in the LHC or eFAST design.

ALGORITHM

Choice of algorithm to use to generate the hypercube. Can be set to either 'normal' or 'optimum'. Beware optimum can take a long time to generate an optimised parameter set (more than 24 hours in some circumstances).

SPARTAN_PARAMETER_FILE

Location of the file output by the latin-hypercube sampling method.

LHCSAMPLE_RESULTFILENAME

Name of the result file generated by Netlogo, for a LHC parameter sample.

LHC_ALL_SIM_RESULTS_FILE

Name of the LHC Summary file to be generated by lhc_process_netlogo_result. Contains each parameter set alongside the result gained when the simulation was run under that criteria.

NUMCURVES

The number of 'resamples' to perform (see eFAST documentation) - recommend using at least 3.

EFASTSAMPLE_RESULTFILENAME

Name of the result file generated by Netlogo, for a LHC parameter sample.

Examples

# NOT RUN {
# THE CODE IN THIS EXAMPLE IS THE SAME AS THAT USED IN THE TUTORIAL, AND
# THUS YOU NEED TO DOWNLOAD THE TUTORIAL DATA SET AND SET FILEPATH
# CORRECTLY TO RUN THIS
# }
# NOT RUN {
## Example for Robustness Analysis
## INCLUDE THESE TWO LINES FOR NETLOGO - REMOVES ANY PERIODS PLACED IN THE
## PARAMETERS AND MEASURES IN THE SPREADSHEET
PARAMETERS<-table_header_check(PARAMETERS)
MEASURES<-table_header_check(MEASURES)

oat_process_netlogo_result(FILEPATH,NETLOGO_BEHAVIOURSPACEFILE,
	PARAMETERS,BASELINE,PMIN,PMAX,PINC,MEASURES,CSV_FILE_NAME,
	ATESTRESULTSFILENAME,TIMESTEP)


oat_graphATestsForSampleSize(FILEPATH,PARAMETERS,MEASURES,
	ATESTSIGLEVEL,ATESTRESULTSFILENAME,BASELINE,
	PMIN=PMIN,PMAX=PMAX,PINC=PINC,PARAMVALS=NULL)

oat_plotResultDistribution(FILEPATH,PARAMETERS,MEASURES,MEASURE_SCALE,
	CSV_FILE_NAME,BASELINE,PMIN=PMIN,PMAX=PMAX,PINC=PINC,
	PARAMVALS=NULL,TIMEPOINTS,TIMEPOINTSCALE)

## Example for Latin-Hypercube Analysis
lhc_process_netlogo_result(FILEPATH,LHCSAMPLE_RESULTFILENAME,
	SPARTAN_PARAMETER_FILE,NUMSAMPLES,MEASURES,
	LHC_ALL_SIM_RESULTS_FILE,TIMESTEP)

PARAMETERS<-table_header_check(PARAMETERS)
MEASURES<-table_header_check(MEASURES)

lhc_generateLHCSummary(FILEPATH,PARAMETERS,MEASURES,
	LHC_ALL_SIM_RESULTS_FILE,LHCSUMMARYFILENAME,
	SPARTAN_PARAMETER_FILE,TIMEPOINTS,TIMEPOINTSCALE)

lhc_generatePRCoEffs(FILEPATH,PARAMETERS,MEASURES,
	LHCSUMMARYFILENAME,CORCOEFFSOUTPUTFILE,
	TIMEPOINTS,TIMEPOINTSCALE)

lhc_graphMeasuresForParameterChange(FILEPATH,PARAMETERS,
	MEASURES,MEASURE_SCALE,CORCOEFFSOUTPUTFILE,
	LHCSUMMARYFILENAME,TIMEPOINTS,TIMEPOINTSCALE)

## Example for eFAST Analysis
efast_process_netlogo_result(FILEPATH,
	EFASTSAMPLE_RESULTFILENAME,PARAMETERS,NUMCURVES,
	NUMSAMPLES,MEASURES,RESULTFILENAME,TIMESTEP)

PARAMETERS<-table_header_check(PARAMETERS)
MEASURES<-table_header_check(MEASURES)

efast_get_overall_medians(FILEPATH,NUMCURVES,PARAMETERS,NUMSAMPLES,
	MEASURES,TIMEPOINTS,TIMEPOINTSCALE)

efast_run_Analysis(FILEPATH,MEASURES,PARAMETERS,NUMCURVES,
	NUMSAMPLES,OUTPUTMEASURES_TO_TTEST,TTEST_CONF_INT,
	GRAPH_FLAG,EFASTRESULTFILENAME,
	TIMEPOINTS,TIMEPOINTSCALE)
# }