16. Dumping Problem Formulation and Data

16.1. Why to use the dump tool?

Along with its clients, FORCESPRO provides a tool that allows the user to dump the formulation and actual data of an optimization problem. This information allows to exactly reproduce the same solver for a given formulation and to feed it with exactly the same data to yield exactly the same results (provided it is run on the very same target hardware). Problem formulation and data stored in “stand-alone” mat or json files, i.e. there is no need to keep copies of other files that may be used to specify the formulation (such as the dynamic equations), except for formulations relying on external callbacks provided as C code (see External function evaluations in C).

The dump tool may be helpful for a couple of use cases such as:

  • Debugging: a dumped problem allows you to re-run single solver calls without the need to have your full simulation environment up and running.

  • External support: you may send a dumped problem to whomever is in charge of providing support and it will enable that person to exactly reproduce your issue.

  • Testing: keeping dumps of problems that performed as expected can be used to run regression tests to ensure they work as expected after future changes.

Note that, depending on the dump type you choose (see How to use the dump tool?), the dump tool either stores your problem formulation on a symbolic level or keeps a copy of the C code generated by the automatic differentiation tool. Thus, keep the following in mind:

Important

A dumped problem will contain complete information about the solver that you have setup. In particular, it may be used to reverse-engineer your problem formulation (including dynamic model, objective function, constraints etc.). Thus, only share a dumped problem with persons that have a right to obtain this information.

16.2. How to use the dump tool?

The dump tool currently provides two different dump types. Section Legacy dumps describes the so-called legacy dump that is available in the MATLAB client only and does not store the full symbolic formulation. Section Symbolic dumps describes the more recent symbolic dump that is also available in the Python client, stores the full symbolic formulation, but requires CasADi v3.5.1 to work.

16.2.1. Legacy dumps

Legacy dumps are available for the MATLAB client only and store a pre-processed problem formulation including C code generated by the automatic differentiation tool. This variant is somewhat less explicit and is supposed to work with all supported AD tools, and is therefore still the default dump type if not specified otherwise by the user.

Creating a legacy dump of a problem consists of two steps:

  1. Dumping the problem formulation: once a new solver has been generated, a formulation struct, the codeoptions struct and optionally the outputs struct need to be stored.

  2. Dumping problem data: for each problem instance, the problem params struct needs to be stored. It is possible to store data of multiple problem instances for the same problem formulation.

16.2.1.1. Dumping the problem formulation

For dumping the problem formulation, the following three steps need to be taken:

  1. Enabling creation of a formulation dump: This is done by using the option

    codeoptions.dump_formulation = 1;
    
  2. Obtaining the dumped formulation: Calling FORCES_NLP with the before-mentioned code option enabled will make it return a formulation struct as third output argument

    [stages, codeoptions, formulation] = FORCES_NLP( model, codeoptions, outputs );
    
  3. Storing the necessary structs into a file: After calling FORCES_NLP, you should use the following function to store both the formulation and codeoptions struct

    tag = ForcesDumpFormulation( formulation,codeoptions,outputs,label,dumpDirectory );
    

All but the first two arguments are optional. Pass outputs if your problem formulation contains outputs. Moreover, you may pass an additional label used inside the filenames (or pass an empty string) and dumpDirectory for storing the dumped formulation (the default is the current working directory). The function ForcesDumpFormulation will create a mat file in the specified directory containing the passed information. The filename is automatically chosen and will contain the name of your solver, your label, a unique tag, a timestamp as well as the suffix _F, e.g. myFORCESsolver_ABC3DEFGHIJ_20200101120000000_F.mat.

Note that this function returns a tag that is unique for a given formulation and code options. It is strongly recommended to use it when dumping corresponding problem data.

16.2.1.2. Dumping problem data

Assuming your generated FORCESPRO solver is called myFORCESsolver and you are calling it with the following command

[output, exitflag, info] = myFORCESsolver( problem );

then dumping the problem data of any problem instance is as simple as calling

ForcesDumpProblem( problem,tag,dumpDirectory );

Here, you need to provide both the problem parameter struct as well as the unique tag that has been generated when dumping the problem formulation. The third argument dumpDirectory for storing the dumped problem data is optional (with the default being the current working directory). The function ForcesDumpProblem will create a mat file in the specified directory containing the passed information. The filename is automatically chosen and will contain the name of your solver, the unique tag (including any label passed when dumping the formulation), a timestamp as well as the suffix _P, e.g. myFORCESsolver_ABC3DEFGHIJ_20200101120001000_P.mat.

There is no limit on the number of problem instances that you may dump that way.

16.2.1.3. Running a dumped problem

After you have dumped a problem formulation and at least one set of problem data, you can use those mat files to exactly reproduce your solver and problem instances. To do so, you need to perform the following two steps (where we assume you have stored the two files myFORCESsolver_ABC3DEFGHIJ_20200101120000000_F.mat and myFORCESsolver_ABC3DEFGHIJ_20200101120001000_P.mat at a location in your MATLAB path):

  1. Re-generate the FORCESPRO solver by loading the formulation mat file and using its content to call the code generation:

    F = load('myFORCESsolver_ABC3DEFGHIJ_20200101120000000_F.mat');
    FORCES_NLP( F.formulation,F.codeoptions,F.outputs );
    

    This will re-create the solver MEX function myFORCESsolver. Note that the third input struct containing the outputs is only available if you included it into your dump.

  2. Running the solver with dumped problem data by loading the data mat file and using its content to call the generated solver:

    P = load('myFORCESsolver_ABC3DEFGHIJ_20200101120001000_P.mat');
    myFORCESsolver( P.problem );
    

    You may repeat this step for as many problem instances as you have dumped.

16.2.1.4. Limitations of legacy dumps

Legacy dumps have the following limitations:

  • They are only available via the MATLAB client of FORCESPRO.

  • They cannot be used if you pass external functions in form of C code.

These limitations can be overcome by using a symbolic dump.

16.2.2. Symbolic dumps

Symbolic dumps direcly store symbolic expressions of your problem formulation and codeoptions after converting both into the text-based JSON format. This variant thus reveals your complete problem formulation to anybody with whom you share those JSON files! While you should thus handle those symbolic dumps with care, they offer more flexibility than the legacy dumps and are also available via the Python client of FORCESPRO.

Creating a symbolic dump of a problem consists of two steps:

  1. Dumping the problem formulation: you need to store your model struct, the codeoptions struct and optionally the outputs struct, which can be done even before generating the actual solver code.

  2. Dumping problem data: for each problem instance, the problem params struct needs to be stored. It is possible to store data of multiple problem instances for the same problem formulation (in either a single file or multiple files).

Both steps may also be performed at once.

16.2.2.1. Dumping the problem formulation

For dumping the problem formulation in a symbolic way, just call the following function:

tag = ForcesDumpFormulation( model,codeoptions,outputs,...
                             label,dumpDirectory,ForcesDumpType.DumpSymbolics );

The last argument enables the use of a symbolic dump. Pass outputs if your problem formulation contains outputs. Moreover, you may pass an additional label used inside the filenames (or pass an empty string) and dumpDirectory for storing the dumped file (the default is the current working directory). When calling this way, the function ForcesDumpFormulation will create a json file in the specified directory containing the passed information. The filename is automatically chosen and will contain the name of your solver, your label, a unique tag, a timestamp as well as the suffix _F, e.g. myFORCESsolver_ABC3DEFGHIJ_20200101120000000_F.json.

Note that this function returns a tag that is unique for a given formulation and code options. It is strongly recommended to use it when dumping corresponding problem data.

Creating a separate symbolic dump of just the problem formulation is currently not supported in the Python client, rather you need to dump formulation and problem data at once (see Dumping the problem formulation and data at once).

16.2.2.2. Dumping problem data

Assuming your generated FORCESPRO solver is called myFORCESsolver and you are calling it with the following command

[output, exitflag, info] = myFORCESsolver( problem );

then dumping the problem data of any problem instance is as simple as calling

ForcesDumpProblem( problem,tag,dumpDirectory,ForcesDumpType.DumpSymbolics );

Here, you need to provide both the problem parameter struct as well as the unique tag that has been generated when dumping the problem formulation. The third argument dumpDirectory for storing the dumped problem data is optional (with the default being the current working directory). The function ForcesDumpProblem will create a json file in the specified directory containing the passed information. The filename is automatically chosen and will contain the name of your solver, the unique tag (including any label passed when dumping the formulation), a timestamp as well as the suffix _P, e.g. myFORCESsolver_ABC3DEFGHIJ_20200101120001000_P.json.

There is no limit on the number of problem instances that you may dump that way.

Creating a separate symbolic dump of just the problem data is currently not supported in the Python client, rather you need to dump formulation and problem data at once (see Dumping the problem formulation and data at once).

16.2.2.3. Dumping the problem formulation and data at once

For dumping both the problem formulation and all problem data at once in a symbolic way, just call the following function:

tag = ForcesDumpAll( model,codeoptions,outputs,...
                     label,dumpDirectory,problems,ForcesDumpType.DumpSymbolics );
path = forcespro.dump.save(model, codeoptions, outputs, problems, dumpDirectory, label)

The last argument enables the use of a symbolic dump. Pass outputs if your problem formulation contains outputs. Moreover, you may pass an additional label used inside the filenames (or pass an empty string) and dumpDirectory for storing the dumped file (the default is the current working directory). Morevoer, problems may be either a single set of problem data or a (cell) array of many problem data sets that you want to dump along with the problem formulation.

When calling this way, the function ForcesDumpAll will create a json file in the specified directory containing the passed information. The filename is automatically chosen and will contain the name of your solver, your label, a unique tag, a timestamp as well as the suffix _A, e.g. myFORCESsolver_ABC3DEFGHIJ_20200101120000000_A.json.

Note that this function returns a tag that is unique for a given formulation and code options and that you may want to record.

16.2.2.4. Running a dumped problem

After you have dumped a problem formulation and at least one set of problem data, you can use either a matching pair of _F/_P files or any single _A file in JSON format to exactly reproduce your solver and problem instances. To do so, you need to perform the following two steps:

  1. Load problem formulation and data from JSON file or files calling:

[model, codeoptions, outputs, additionalData, problems] = ...
              ForcesLoadSymbolicDump( formulationFilename,problemFilenames );
model, options, outputs, problems, additional = forcespro.dump.load(dumpDirectory, label)

problemFilenames may either be a single file name or a cell array containing all the problem data set that you want to load. In case you have dumped both formulation and problem data set(s) at once within a single file, just pass that one as formulationFilename and do not specify problemFilenames.

  1. Re-generating and running the solver with dumped information by simply calling:

FORCES_NLP( model,codeoptions,outputs );
myFORCESsolver( problems{1} );
// and more problem instances if present
solver = model.generate_solver(codeoptions, outputs)
result, exitflag, info = solver.solve(problem[1])
// and more problem instances if present

16.2.2.5. Limitations of symbolic dumps

Symbolic dumps only work with CasADi v3.5.1 for reasons beyond our control, which is why we currently do not plan to extend support to CasADi v2.4.2 or MathWorks’ Symbolic Math Toolbox.