The IMOD Processor

The IMOD Processor, found in processors/imod_processor.py, is implemented to facilitate processing of the generated simulation dataset with the IMOD software. Specifically, a project directory is created and set up, along with necessary files, for running the batchruntomo program for processing the simulated stacks in batch. The IMOD Processor as well will take its inputs from the configuration YAML passed into ets_process_data.py. Consider the following example:

processors: [
  {
    name: "imod",
    args: {
      start_step: 0.0,
      end_step: 8.0,
      num_fiducials: 10,
      tilt_axis: -90,
      apix: 0.283,
      fiducial_method: "autofidseed",
      reconstruction_method: "tomo3d",
      rotx: true,
      binvol: {
        "binning": 2
      },
      tomo3d_path: ""
      tomo3d_options: {
        "S": "enable",
        "z": 700
      }
    }
  }
]

Reconstruction

As exemplified above, tomogram reconstruction is available as part of the steps handled by the IMOD Processor. This can be done using IMOD’s built-in reconstruction methods (as part of the batchruntomo processing”, or by interfacing with the SIRT implementation provided by tomo3d, if installed. Some common post-processing applied to reconstructed tomograms is also provided - namely, rotation and binning. Note: Even if you are planning to use tomo3d and not a built-in IMOD reconstruction method, you must set the end_step parameter to 14 or higher to let the processor know that you intend to proceed through reconstruction. If you specify “tomo3d” as the reconstruction method, it will be substituted in instead of step 14 for the batchruntomo runs.

Parameters

The IMOD Processor, like all others, have the name argument (“imod”) and an args object filled with parameters. The parameters exposed here are abstracted versions of some of the more common ones when processing with batchruntomo, as well as some others specifying processing options for the IMOD processor itself. Specifically, we have:

  • start_stepfloat

    The batchruntomo processing start step (the list of steps can be found on the IMOD batchruntomo documentation page)

  • end_stepfloat

    The batchruntomo processing end step (the list of steps can be found on the IMOD batchruntomo documentation page)

  • filename_conventionstring

    One of either “new” or “old”. Whether to use the old naming convention for files by IMOD, which uses different file extensions based on the step (i.e. *.preali files), or the new convention which sets all MRC files to use the *.mrc extension. Must be set to match the convention used by your version of IMOD. Defaults to “old”.

  • num_fiducialsinteger

    The number of fiducials to track with either Raptor or the automatic fiducial seeding/tracking for alignment

  • tilt_axisfloat

    The tiltseries tilt axis you specified in the TEM-Simulator configs

  • apixfloat

    The pixel size (in nm)

  • fiducial_methodstring

    Either “autofidseed” or “raptor”, determining how beads are seeded and tracked for alignment

  • custom_templatestring

    (Optional) Note this is not included in the example above; putting this option in overwrites the others) A filepath to a batchruntomo template file (.adoc) to be used instead of the default one generated with the other options. This can be used to control any parameters open to batchruntomo, but will probably require more experimentation/advanced knowledge to get it to work.

  • force_coarse_alignbool

    (Optional) Takes a Boolean true or false value. Allows overriding of default behavior which skips the coarse alignment step of the batchruntomo process. Will likely be enabled when using the processor on real data instead of simulated ones.

  • real_data_modebool

    (Optional) When set to true, the IMOD processor will assume you are trying to set up and run batchruntomo for a real dataset not generated by ets_generate_data.py. In real data mode, pre-oriented particle data will not be looked for and extracted from a sim_metadata.json file. It will be assumed that you have the force_coarse_align enabled along with the real data mode, and the fake coarse alignment output files will not be copied over either. Finally, instead of looking for a “raw_data” folder in the provided root, the processor will just use the root as the raw data folder.

  • data_dirs_start_withstring

    (Optional) By default this will be set to the name parameter for the project, and tells the processor what to consider as data subdirectories within the provided root folder. For simulated data, the subdirectories will be in the form name_particleNum so the default is fine. This option is exposed for when dealing with real data where this may not be the case.

  • reconstruction_method: string

    (Optional) This parameter can be set to either “imod-wbp”, “imod-sirt”, or “tomo3d” and is only relevant if the batchruntomo end_step is set to 14 or higher (the step for reconstruction). Using the “imod” option will run the reconstruction with the built-in IMOD WBP or SIRT reconstruction method, while “tomo3d” will use the WBP or SIRT implemented (specified in the tomo3d_options) by tomo3d, assuming you have tomo3d installed.

  • rotxbool

    (Optional) Set this parameter to true if you wish to apply the common step of running “clip rotx” IMOD program on the reconstructed tomogram. This is run before the flipyz program if both are enabled for some reason.

  • flipyzbool

    (Optional) Set this parameter to true if you wish to apply the common step of running “clip flipyz” IMOD program on the reconstructed tomogram.

  • binvolYAML object

    (Optional) If this parameter is included, reconstructed tomograms will be put through the IMOD “binvol” program. Here, a YAML object containing binvol options as keys should be given, with options that do not require an input value must instead be given the string “enable”. This matches the YAML parameter scheme used by the EMAN2 Processor to handle arbitrary command line arguments for sub-processes.

  • imod_tomogram_thicknessinteger

    (Optional; only considered if using “imod-wbp” or “imod-sirt” reconstructions) The unbinned tomogram thickness to pass along as an argument to the IMOD reconstruction step.

  • tomo3d_pathstring

    (Required only if reconstruction_method set to “tomo3d”) The file path to the tomo3d executable, or just the “tomo3d” command if it is set on your PATH variable.

  • tomo3d_optionsYAML object

    (Optional) This parameter provides a place to feed in any additional options to your tomo3d reconstruction calls not including the .tlt file, the aligned tiltseries, and the output file (those three are required and always handled by the IMOD Processor code). Options that do not require specific input values, such as the “S” option for using SIRT, should be passed the special value “enable”.

An important thing to note is that stacks generated by the simulations will most likely not have enough signal to have the IMOD course alignment with cross-correlation step work. Instead, the cross-correlation is likelier to shift tilts extremely out of alignment than to make any improvements. Thus, the IMOD Processor is designed to automatically skip the coarse alignment (steps 2.0 and 3.0) and fake having done it by moving in fake versions of its outputs like the .rawtlt file. It is possible to override this using the force_coarse_align option. \

Additionally, as mentioned, fake versions of the rough alignment are used in place of doing the cross-correlation. These fake files are located in the templates/imod folder and can be edited if desired. For example, the rawtlt file is for a 2-degree increment, -54 to 54 degrees tilt scheme and should be edited if using a different tilt scheme with your TEM-Simulator.

Using the IMOD Processor on real data

It is possible to use the IMOD Processor to set up and run batchruntomo for real data sets. To do so, something like the processor arguments below should be used:

processors: [
  {
    name: "imod",
    args: {
      start_step: 0.0,
      end_step: 8.0,
      num_fiducials: 10,
      tilt_axis: -11,
      apix: 0.52,
      fiducial_method: "autofidseed",
      force_coarse_align: true,
      real_data_mode: true,
      data_dirs_start_with: "dg"
    }
  }
]

Detailed descriptions for each parameter can be found above. For this to work, organize your real data set within your filesystem so that the raw stacks are already split into their own subdirectories within the root folder you provide in the YAML configs. Each subdirectory should begin with the string provided to the data_dirs_start_with option; should only contain one MRC file inside as this will be taken to be the raw stack to process. It may of course be useful to use the custom_template option to provide personalized .adoc files for the batchruntomo directives when dealing with real data.

A note must be made about the IMOD Processor’s handling of the fiducial alignment step for batchruntomo (step 6, corresponding to the tiltalign IMOD program). By default batchruntomo attempts to dynamically iterate runs of tiltalign, changing parameters such as the tilt AngleOffset and thickness based on past iterations, likely to reduce the need for manual intervention. However, in our experience, the AngleOffset values introduced automatically by batchruntomo for tiltalign throw off the final computed tilt angles significantly. To handle this, we manually run tiltalign for each data sub-directory and skip step 6 of batchruntomo.