Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
auto3dem:history [2015/03/29 15:27]
gcardone [auto3dem v2.03]
auto3dem:history [2015/12/22 12:59] (current)
gcardone
Line 1: Line 1:
 ====== Modification history ====== ====== Modification history ======
  
-This document describes incremental modifications made to the Baker lab software suite. Since the image preprocessing,​ image reconstruction,​ format conversion, and utility programs are so tightly coupled (e.g. shared standards and libraries) and released as a single package, all modifications are described together in a single document. Owing to the complexity of the software, it is impractical to provide a line-by-line listing of alterations. Instead we focus primarily on modifications that impact the end user and on large-scale changes to the code. Starting on version 4.02, more details on some of the listed ​changes can be found in the file MINOR_RELEASE_NOTES,​ included in the software package.+This document describes incremental modifications made to the Baker lab software suite. Since the image preprocessing,​ image reconstruction,​ format conversion, and utility programs are so tightly coupled (e.g. shared standards and libraries) and released as a single package, all modifications are described together in a single document. Owing to the complexity of the software, it is impractical to provide a line-by-line listing of alterations. Instead we focus primarily on modifications that impact the end user and on large-scale changes to the code. Starting on version 4.02, more details on the changes ​in the latest version ​can be found in the file MINOR_RELEASE_NOTES,​ included in the software package. 
 + 
 +==== auto3dem v4.05.2 ==== 
 + 
 +== New programs == 
 +  * **prj**: calculate projections from a reconstruction using orientations either random, uniform or from an input data set 
 +  * **a3dm_icosrotate**:​ rotate an icosahedral map from standard orientation to one with a pair of vertices positioned along the z axis 
 +  * **a3dm_dat_stats**:​ gather information on the results from a run at a given iteration 
 +  * **a3dm_sel_stableparts**:​ select only stable particles, i.e. with unchanged orientation with respect to a previous iteration, and generate a new stack parameter file 
 +  * **a3dm_plot_score**:​ generate a histogram of the particle scores at a given iteration 
 +  * **a3dm_mg_inspect_prepare**:​ generate auxiliary info and images for screening micrographs using //​a3dm_mg_inspect//​ 
 +  * **a3dm_mg_inspect**:​ quickly inspect and screen micrographs by means of a graphical interface 
 +  * **avemaps**:​ compute average of input maps (replaces mergemaps and mergemaps2) 
 + 
 +== General modifications== 
 +  * improved handling of cyclic and dihedral symmetries 
 +  * added program a3dm_snapshot to gather information about a run, to send to developers for troubleshooting 
 +  * particles read from stack files are now background-subtracted 
 + 
 +== Random Model Computation == 
 +  * use inner radius given as input to **setup_rmc** to mask reference maps at each iteration 
 +  * suppressed non significant warnings 
 + 
 +== auto3dem == 
 +  * added option //po2r mask_map// to specify a binary mask to use for mask-focused refinement 
 +  * added option //auto evod_parallel//​ to specify if processing of even- and odd-numbered particles to be done concurrently 
 +  * added option //p3dr cpu_sparse//​ to specify if to spread cpus among different nodes to optimize resources for large maps (works only on multi-node clusters with pbs/sge queue systems) ​  
 +  * modified approach to define radial mask on maps (only ppft and hollow procedures):​ search of outer radius starts from radius with minimum standard deviation 
 + 
 +== autopp == 
 +  * :!: restored option //D// to original interpolation and moved binning procedure to option //H// 
 +  * options 5 and 6: keep all info in first line of .dat files 
 +  * option 6: fixed bug when processing stacks with only one particle 
 +  * option j: when scaling, adjust also pixel size in .dat files 
 + 
 +== PPFT == 
 +  * figure of merit calculated in double precision 
 + 
 +== POR == 
 +  * avoid overflow error with metric/​funcmode 0 
 +  * change metric 1 to normalized version 
 +  * use the same user-defined metric for origin refinement 
 + 
 +== maskmap == 
 +  * added option to apply the inverse of input mask 
 + 
 +== rot_map == 
 +  * input shifts now expected as pixels 
 +  * improved estimate of the background value 
 +  * write the map in a format compatible with **p3dr** 
 + 
 +== a3dm_plot_fsc == 
 +  * estimate resolution at threshold value by linear interpolation 
 +  * added option //​-c/​--cutoff//​ to specify threshold value where to estimate the resolution 
 + 
 +==== auto3dem v4.05.1 ==== 
 + 
 +== Installation == 
 +  * changed installation procedure: **auto3dem_install** is the new command 
 +  * made easy to add support for new compilers  
 + 
 +== New programs == 
 +  * **maskmap**:​ apply an input binary mask after adding a soft edge 
 + 
 +== Random Model Computation == 
 +  * a minimum number of micrographs is used to generate an initial model. This value can be controlled by option //-nmgmin// in **setup_rmc** 
 +  * initial master file contains commented line for limiting number of cpus used at reconstruction 
 + 
 +== General modifications == 
 +  * provided binary for CTFFIND4 in the package (Linux only) 
 +  * at start programs will display if compiled in parallel or serial mode  
 +  * maps are flattened with a soft mask, using full map as reference for masking in gold standard 
 +  * inconsistency of pixel size in map header does not throw an error anymore 
 + 
 +== auto3dem == 
 +  * added parameter to control the soft edge with solvent flattening (//auto flatten_falloff//​) 
 +  * keep input/​output files used by flatten and hollow procedure 
 +  * continue parameter file generated at end of each run has now unique name to avoid overwriting and to keep track of the processing 
 + 
 +== autopp == 
 +  * :!: option //D// modified to perform binning of micrographs instead of interpolation 
 +  * added experimental option 3x in autopp for using ctffind4 
 + 
 +== robem == 
 +  * added option for performing binning in batch mode (//​-bin//​) 
 +  * improved font appearance 
 +  * remove unused read/write menus 
 +  * added warning if can not connect to display 
 +  * minor fixes on size of file names 
 +  * avoid including origin when calibrating maps for subtraction 
 + 
 +==== auto3dem v4.05 ==== 
 + 
 +Implemented the gold standard approach, and set as default 
 + 
 +== Installation == 
 +  * added installation script for Mac OSX 
 +  * added installation instructions for Mac OSX 
 + 
 +== new programs == 
 +  * **a3dm_plot_fsc.py**:​ plot FSC curves from a given set of iterations 
 +  * **a3dm_fsc2xml.py**:​ convert FSC curve to EMDB-compatible XML format 
 + 
 +== auto3dem == 
 +  * changed method to identify score to use for selection of particles 
 +  * setup_rmc now prepares parameter files for gold standard procedure 
 +  * map radial profile is avoided if new_ptles set 
 +  * move the parameter updates outside of even/odd loops 
 +  * make sure that parameter files are complete with particles for gold standard 
 +  * fixed bug when estimating mask radii and changed constant factor for outer radius 
 + 
 +== autopp == 
 +  * added option 6 to split parameters into even and odd numbered particles 
 +  * option j:  handle multistack parameter files 
 +  * option 6: check presence of '​dat'​ in parameter file names 
 +  * option 6: handle files not containing _ character when splitting into even/odd files 
 +  * option 6: make up particle parameters when splitting empty files 
 +  * options F and O: added option to delete input files once processed 
 +  * option 5: fixed split and join of parameter files 
 +  * option K: remove box size from first line of dat files 
 +  * added option for not masking corners when generating/​processing stacks 
 +  * option j: remove box size info from first line of dat files 
 +  * option 3 (ctffind): remove unused variable pave 
 + 
 +== P3DR == 
 +  * increased output format size for pixel count 
 + 
 +== PSF == 
 +  * handle large maps 
 + 
 +== PPFT == 
 +  * added warning messages during initialization 
 +  * fix error messages in get_info module 
 +  * constrained shift values in output dat files 
 +  * improved logs and check controls 
 +  * fixed index bug for compare_mode 1 (mode not currently used) 
 + 
 +== POR ==  
 +  * constrained shift values in output dat files 
 + 
 +== tiff2pif == 
 +  * fixed interface and handle endianess 
 + 
 +== dm32pif == 
 +  * accept unsigned integer format 
 + 
 +== robem == 
 +  * added -fastload to avoid calculation of radial profiles 
 +  * calculate radial profile also for maps loaded for comparison (Subtract 3D Map menu) 
 +  * redefined //​map_1d_rob//​ as  //​calc_map_1d_profile//,​ and simplified its interface 
 +  * allow initialization of large arrays 
 +  * allow opening of large maps 
 +  * allow large indexes when reading integer maps 
 +  * changed window parameters to avoid colored stripes in the gui 
 +  * fix radial projection for large maps 
 +  * increased buffer display 
 +  * open stack images even if in different directory than dat file
  
 ==== auto3dem v4.04 ==== ==== auto3dem v4.04 ====
Line 281: Line 437:
  
 <WRAP center round box 60%> <WRAP center round box 60%>
-OLD:    xbatch robem –pre infile [other options]+OLD:    xbatch robem –pre infile [other options]\\
 NEW: robem –nogui –pre infile [other options] NEW: robem –nogui –pre infile [other options]
 </​WRAP>​ </​WRAP>​
  
  
-  *       Fixed the feature that adds Gaussian noise to an image. This had been done incorrectly in previous versions as evidenced by the fact that adding Gaussian noise with mean=0 changed the mean intensity of the image. ​ +  *       Fixed the feature that adds Gaussian noise to an image. This had been done incorrectly in previous versions as evidenced by the fact that adding Gaussian noise with mean=0 changed the mean intensity of the image. (This feature is found at Manipulate > Fix Values)
-(This feature is found at Manipulate > Fix Values)+
   *       Fixed bug that caused robem to crash when clicking in drawing area of particle processing screen before particles had been loaded.   *       Fixed bug that caused robem to crash when clicking in drawing area of particle processing screen before particles had been loaded.
   *       Fixed bug where display of radial projections would sometimes cause robem to crash.   *       Fixed bug where display of radial projections would sometimes cause robem to crash.
Line 782: Line 937:
 The 'quick omega' feature is enabled by default in auto3dem. Adding the following line to auto3dem input files disables this option The 'quick omega' feature is enabled by default in auto3dem. Adding the following line to auto3dem input files disables this option
  
-ppft quick_omega ​ 0 # auto3dem input file+  ​ppft quick_omega ​ 0 # auto3dem input file
  
 This new capability required the creation of a new subroutine get_phiomega_quick and modifications to the following files in directory PPFT: global.f, global_cc.f,​ get_tpo_g.f,​ pftsearch.f,​ ppft_info.f,​ and key_info.f. Also required changes to include/​infohead.inc,​ init_params.pm,​ and make_program_input.pm This new capability required the creation of a new subroutine get_phiomega_quick and modifications to the following files in directory PPFT: global.f, global_cc.f,​ get_tpo_g.f,​ pftsearch.f,​ ppft_info.f,​ and key_info.f. Also required changes to include/​infohead.inc,​ init_params.pm,​ and make_program_input.pm
Line 788: Line 943:
   *       ​Capabilities added to P3DR to produce an apodized map, with a Gaussian falloff ​ applied to the density in the region boxrad-border ≤ r boxrad. Border has units of pixels and must be an integer. Using a border width of  zero recovers original P3DR behavior. Default value in auto3dem set to 12, but can be overridden by adding the following line to the auto3dem input files   *       ​Capabilities added to P3DR to produce an apodized map, with a Gaussian falloff ​ applied to the density in the region boxrad-border ≤ r boxrad. Border has units of pixels and must be an integer. Using a border width of  zero recovers original P3DR behavior. Default value in auto3dem set to 12, but can be overridden by adding the following line to the auto3dem input files
  
-p3dr apo_border n # auto3dem input file+  ​p3dr apo_border n # auto3dem input file
  
 This new capability required changes to the following files: P3DR/​P3dr.F,​ P3DR/​density_clear.f,​ Commpk/​info.f,​ Compar/​bcast_parameters.f,​ info.inc, init_params.pm,​ and make_program_input.pm This new capability required changes to the following files: P3DR/​P3dr.F,​ P3DR/​density_clear.f,​ Commpk/​info.f,​ Compar/​bcast_parameters.f,​ info.inc, init_params.pm,​ and make_program_input.pm
Line 795: Line 950:
  
 The 'quick search'​ feature is enabled by default in auto3dem. Adding the following line to auto3dem input files disables this option The 'quick search'​ feature is enabled by default in auto3dem. Adding the following line to auto3dem input files disables this option
-  + 
-  +  po2r quick_search 0 # auto3dem input file
-po2r quick_search 0 # auto3dem input file+
  
 This new capability required changes to the following files: POR/Por.f, POR/​cmpt_ort.f,​ Commpk/​info.f,​ Compar/​bcast_parameters.f,​ info.inc, init_params.pm,​ and make_program_input.pm This new capability required changes to the following files: POR/Por.f, POR/​cmpt_ort.f,​ Commpk/​info.f,​ Compar/​bcast_parameters.f,​ info.inc, init_params.pm,​ and make_program_input.pm
Line 803: Line 957:
   *       ​Refinement of CTF parameters using program PCTFR is now disabled by default. To override, add the following line to auto3dem input file   *       ​Refinement of CTF parameters using program PCTFR is now disabled by default. To override, add the following line to auto3dem input file
  
-auto refine_ctf 1+  ​auto refine_ctf 1
    
   *       ​setup_rmc.pl prints comments to the auto3dem input file describing how to set parameters to perform faster reconstructions and reach higher resolutions. The following commented out parameters are now listed   *       ​setup_rmc.pl prints comments to the auto3dem input file describing how to set parameters to perform faster reconstructions and reach higher resolutions. The following commented out parameters are now listed
  
-ppft bin_factor +  ​ppft bin_factor 
-ppft verbose +  ppft verbose 
-ppft annulus_low +  ppft annulus_low 
-ppft_annulus_high +  ppft_annulus_high 
-pcut in_rad +  pcut in_rad 
-pcut out_rad +  pcut out_rad 
-auto freeze_annulus +  auto freeze_annulus 
-auto bin_reduce+  auto bin_reduce
  
   *       Bug fix made to Perl module get_ann_lo_hi.pm so that inner and outer radii of capsid are estimated correctly when binning is used in program PPFT.   *       Bug fix made to Perl module get_ann_lo_hi.pm so that inner and outer radii of capsid are estimated correctly when binning is used in program PPFT.
Line 820: Line 974:
   *       POR modified to keep statistics on both the level of individual micrographs and the entire run for the number of particle images that fall into each of the following categories   *       POR modified to keep statistics on both the level of individual micrographs and the entire run for the number of particle images that fall into each of the following categories
  
-Stable orientation / stable origin +  ​Stable orientation / stable origin 
-New orientation / stable origin +  New orientation / stable origin 
-Flipped hand / stable origin +  Flipped hand / stable origin 
-Stable orientation / new origin +  Stable orientation / new origin 
-New orientation / new origin +  New orientation / new origin 
-Flipped hand/ new origin+  Flipped hand/ new origin
  
 The '​stable'​ designation means that the orientation or origin for a particle is unchanged after running POR.  '​Flipped hand' means that the orientation is the same except for a change in the handedness of the particle. This information is summarized in the POR output file. The '​stable'​ designation means that the orientation or origin for a particle is unchanged after running POR.  '​Flipped hand' means that the orientation is the same except for a change in the handedness of the particle. This information is summarized in the POR output file.
Line 839: Line 993:
   *       ​Command line syntax for auto3dem and setup_rmc more flexible. Key value pairs no longer need to be separated by equal signs. For example, both of the following are now valid   *       ​Command line syntax for auto3dem and setup_rmc more flexible. Key value pairs no longer need to be separated by equal signs. For example, both of the following are now valid
  
-%auto3dem -ncpu=16 -input=input_file +  ​%auto3dem -ncpu=16 -input=input_file 
-%auto3dem -ncpu 16 -input input_file+  %auto3dem -ncpu 16 -input input_file
  
   *       Minor performance enhancements made to (q, |φ|) search routine get_thephi in program PPFT.   *       Minor performance enhancements made to (q, |φ|) search routine get_thephi in program PPFT.
  
   *       Make following modifications to PPFT input routine ppft_info:   *       Make following modifications to PPFT input routine ppft_info:
-  Equal sign in key-value pairs is now optional +    *   Equal sign in key-value pairs is now optional 
-  Strings containing ‘/’ properly handled +    ​* ​  Strings containing ‘/’ properly handled 
-  lg_out initialized to zero +    ​* ​  lg_out initialized to zero 
-  Start of data file list can be specified using both ‘end_of_keys’ (first three characters significant) or ‘inputparameterfiles’ (first nine characters significant).+    ​* ​  Start of data file list can be specified using both ‘end_of_keys’ (first three characters significant) or ‘inputparameterfiles’ (first nine characters significant).
  
   *       Allow blank lines and comments (using ‘#’) in the data file section of input for programs PPFT, POR, P3DR, and PCTFR. Required changes to PPFT/​pfile_info_pft.f and Commpk/​pfile_info.f.   *       Allow blank lines and comments (using ‘#’) in the data file section of input for programs PPFT, POR, P3DR, and PCTFR. Required changes to PPFT/​pfile_info_pft.f and Commpk/​pfile_info.f.
Line 855: Line 1009:
    
   *       ​Cleanup timing functionality throughout code   *       ​Cleanup timing functionality throughout code
-  Add new function elapsed_time.f to library Commpk that calculates the time elapsed between pairs of system_clock readings; get rid of old timer function timer1.c, and update Makefile. +    *   Add new function elapsed_time.f to library Commpk that calculates the time elapsed between pairs of system_clock readings; get rid of old timer function timer1.c, and update Makefile. 
-  Modify mem_time.inc so that all timing variables reside in a common block and get rid of test_memory,​ test_time, and wtime declarations. +    ​* ​  Modify mem_time.inc so that all timing variables reside in a common block and get rid of test_memory,​ test_time, and wtime declarations. 
-  Replace calls to mpi_wtime with Fortran 90 system_clock intrinsic in the following files: Ctffit/​ctffit.f,​ P3DR/​P3dr.F,​ P3DR/​cmpt_intrps.F,​ PCTFR/​Pctfr.f, ​ PCTFR/​cmpt_ctf.f, ​ PCUT/​Pcut.f,​ POR/Por.f, POR/​cmpt_ort.f,​ PPFT/​global.f,​ PPFT/​global_cc.f,​ PSF/Psf.f +    ​* ​  Replace calls to mpi_wtime with Fortran 90 system_clock intrinsic in the following files: Ctffit/​ctffit.f,​ P3DR/​P3dr.F,​ P3DR/​cmpt_intrps.F,​ PCTFR/​Pctfr.f, ​ PCTFR/​cmpt_ctf.f, ​ PCUT/​Pcut.f,​ POR/Por.f, POR/​cmpt_ort.f,​ PPFT/​global.f,​ PPFT/​global_cc.f,​ PSF/Psf.f 
-  Get rid of old commented out code in following files: P3DR/​matrixentries_slab.F,​ PCTFR/​Pctfr.f,​ POR/Por.f, POR/​cmpt_ort.f. +    ​* ​  Get rid of old commented out code in following files: P3DR/​matrixentries_slab.F,​ PCTFR/​Pctfr.f,​ POR/Por.f, POR/​cmpt_ort.f. 
-  Comment out or remove unused variables and labels in following files: PCTFR/​cmpt_ctf.f,​ PPFT/​list_ccs.f,​ PPFT/​pfile_info_pft.f,​ PPFT/​ppft_info.f,​ Ctffit/​ctffit.f,​ Ctffit/​rotate.f+    ​* ​  Comment out or remove unused variables and labels in following files: PCTFR/​cmpt_ctf.f,​ PPFT/​list_ccs.f,​ PPFT/​pfile_info_pft.f,​ PPFT/​ppft_info.f,​ Ctffit/​ctffit.f,​ Ctffit/​rotate.f
  
 ==== auto3dem v2.01 ==== ==== auto3dem v2.01 ====
Line 865: Line 1019:
   *       ​Modify programs PSF, PCUT, P3DR, PCTFR, PPFT and POR to allow specification of input file both as command line argument and through redirection of standard input. For example, both of the following are valid ways to execute P3DR   *       ​Modify programs PSF, PCUT, P3DR, PCTFR, PPFT and POR to allow specification of input file both as command line argument and through redirection of standard input. For example, both of the following are valid ways to execute P3DR
  
-% mpirun –np 8 P3DR p3dr_input > p3dr_output +  ​% mpirun –np 8 P3DR p3dr_input > p3dr_output 
-% mpirun –np 8 P3DR < p3dr_input > p3dr_output+  % mpirun –np 8 P3DR < p3dr_input > p3dr_output
    
 The first syntax is preferred and is used internally by auto3dem when launching the parallel jobs. The reason for making this change is that many MPI implementations cannot handle redirection from standard input if the number of characters is greater than 4096. The second syntax is retained for continuity purposes only and is no longer recommended. The first syntax is preferred and is used internally by auto3dem when launching the parallel jobs. The reason for making this change is that many MPI implementations cannot handle redirection from standard input if the number of characters is greater than 4096. The second syntax is retained for continuity purposes only and is no longer recommended.
Line 885: Line 1039:
  
   *       ​Completely overhauled command line interface for both auto3dem and setup_rmc so that input is done using key-value pairs. For example   *       ​Completely overhauled command line interface for both auto3dem and setup_rmc so that input is done using key-value pairs. For example
-  + 
-setup_rmc -ncpu=4 -seed=123 -list=listfile +  setup_rmc -ncpu=4 -seed=123 -list=listfile 
-setup_rmc –usedefaults +  setup_rmc –usedefaults 
-  + 
-auto3dem -ncpu=12 -input=input_file -nodefile=mynodelist+  auto3dem -ncpu=12 -input=input_file -nodefile=mynodelist
    
 setup_rmc can be run purely with default values using the –usedefaults flag. For auto3dem, values must be provided for the number of CPUs and the name of the input file. The nodefile no longer needs to be specified for either setup_rmc or auto3dem. This is to be contrasted with previous versions where the word '​none'​ had to be explicitly used for those cases where the parallel computing environment did not require a node list. For batch systems running the PBS scheduler, the PBS node file is automatically obtained from the $PBS_NODEFILE environment variable via Perl’s %ENV hash. setup_rmc can be run purely with default values using the –usedefaults flag. For auto3dem, values must be provided for the number of CPUs and the name of the input file. The nodefile no longer needs to be specified for either setup_rmc or auto3dem. This is to be contrasted with previous versions where the word '​none'​ had to be explicitly used for those cases where the parallel computing environment did not require a node list. For batch systems running the PBS scheduler, the PBS node file is automatically obtained from the $PBS_NODEFILE environment variable via Perl’s %ENV hash.
Line 922: Line 1076:
   *       ​Semicolons now allowed, in addition to commas and whitespace, as delimiters in list of email addresses in auto3dem input file. For example,   *       ​Semicolons now allowed, in addition to commas and whitespace, as delimiters in list of email addresses in auto3dem input file. For example,
  
- auto recipient address1 address2, address3; address4+  ​auto recipient address1 address2, address3; address4
  
   *       MPI communications in PPFT/​global.f modified to avoid exceeding MPI buffer limitations. Allows PPFT to be run using a finer spacing between projections of the model and/or larger models.   *       MPI communications in PPFT/​global.f modified to avoid exceeding MPI buffer limitations. Allows PPFT to be run using a finer spacing between projections of the model and/or larger models.
Line 997: Line 1151:
   *       ​setup_rmc no longer specifies the number of models on the command line. New syntax is   *       ​setup_rmc no longer specifies the number of models on the command line. New syntax is
  
-setup_rmc dir list ncpu nodefile [boxrad]+  ​setup_rmc dir list ncpu nodefile [boxrad]
  
   *       ​Streamlined random model calculations – auto3dem.pl,​ init_params.pm,​ write_params.pm,​ setup_rmc.pl,​ and findbest.pl were modified to so that calculations on a particular random model are terminated once the resolution improves to the point where the FSC curve never drops below 0.5. Also, additional random models are not calculated once one of the random models had successfully converged. In the best cases, this results in a 20x reduction in the run time relative to calculating ten full iterations for ten different random models.   *       ​Streamlined random model calculations – auto3dem.pl,​ init_params.pm,​ write_params.pm,​ setup_rmc.pl,​ and findbest.pl were modified to so that calculations on a particular random model are terminated once the resolution improves to the point where the FSC curve never drops below 0.5. Also, additional random models are not calculated once one of the random models had successfully converged. In the best cases, this results in a 20x reduction in the run time relative to calculating ten full iterations for ten different random models.
Line 1017: Line 1171:
   *       Email notification – a new module sendmail.pm was added to AUTO3DEM that provides capabilities for sending email notifications with optional attachments. This feature requires that the standard Linux/UNIX mutt email tool be installed on your system. The sendmail.pm module tests for mutt using the UNIX ‘which’ command and does a cursory check on the email address to make sure that it meets the minimal requirements for being well formed. auto3dem.pl has been modified to send email notifications,​ with a text output of the FSC calculations,​ at the end of each iteration. To enable this feature, add the following line to the AUTO3DEM parameter file, where email is a valid email address.   *       Email notification – a new module sendmail.pm was added to AUTO3DEM that provides capabilities for sending email notifications with optional attachments. This feature requires that the standard Linux/UNIX mutt email tool be installed on your system. The sendmail.pm module tests for mutt using the UNIX ‘which’ command and does a cursory check on the email address to make sure that it meets the minimal requirements for being well formed. auto3dem.pl has been modified to send email notifications,​ with a text output of the FSC calculations,​ at the end of each iteration. To enable this feature, add the following line to the AUTO3DEM parameter file, where email is a valid email address.
  
-auto recipient email+  ​auto recipient email
  
   *       ​Random model computations have been simplified. Modifications have been made to both setup_rmc.pl and findbest.pl so that best map is identified and copied into the directory provided as argument to setup_rmc. Argument list for setup_rmc has been expanded so that the commands file does not need to be hand edited and so that all temporary files and directories associated with random model computations have the ‘RMC’ prefix. This last change makes the cleanup of intermediate files more straightforward. The updated syntax for running setup_rmc is shown below. Note the new ncpu and nodefile arguments. ​   *       ​Random model computations have been simplified. Modifications have been made to both setup_rmc.pl and findbest.pl so that best map is identified and copied into the directory provided as argument to setup_rmc. Argument list for setup_rmc has been expanded so that the commands file does not need to be hand edited and so that all temporary files and directories associated with random model computations have the ‘RMC’ prefix. This last change makes the cleanup of intermediate files more straightforward. The updated syntax for running setup_rmc is shown below. Note the new ncpu and nodefile arguments. ​
  
-setup_rmc dir list nmodels ncpu nodefile [boxrad]+  ​setup_rmc dir list nmodels ncpu nodefile [boxrad]
  
   *       CTF refinement – the program PCTFR (Parallel CTF Refinement) has been integrated into AUTO3DEM. This involved changes to auto3dem.pl,​ init_program.pm,​ make_program.pm,​ and write_param.pm. The subroutine cmpt_ctf called by PCTFR has been overhauled and modified so that it can handle both stigmatic and astigmatic images. PCTFR does not currently handle the transition from a stigmatic to astigmatic images unless a reasonably accurate estimate of the orientation of the astigmatism is provided. The main program Pctfr.f has been cleaned up and modernized.   *       CTF refinement – the program PCTFR (Parallel CTF Refinement) has been integrated into AUTO3DEM. This involved changes to auto3dem.pl,​ init_program.pm,​ make_program.pm,​ and write_param.pm. The subroutine cmpt_ctf called by PCTFR has been overhauled and modified so that it can handle both stigmatic and astigmatic images. PCTFR does not currently handle the transition from a stigmatic to astigmatic images unless a reasonably accurate estimate of the orientation of the astigmatism is provided. The main program Pctfr.f has been cleaned up and modernized.
Line 1091: Line 1245:
   *       ​Improved capabilities added for exiting auto3dem if one of the image processing programs aborts. Depending on operating system and batch queuing system, the return value from launching an MPI job with system command may not be accessible to the auto3dem script. To ensure that the abort is detected, the output from the MPI program is parsed and auto3dem exits if the string MPI_Abort (not case sensitive) is found. Involved modifications to error_stop.f,​ run_mpi_prog.pm,​ and auto3dem.pl.   *       ​Improved capabilities added for exiting auto3dem if one of the image processing programs aborts. Depending on operating system and batch queuing system, the return value from launching an MPI job with system command may not be accessible to the auto3dem script. To ensure that the abort is detected, the output from the MPI program is parsed and auto3dem exits if the string MPI_Abort (not case sensitive) is found. Involved modifications to error_stop.f,​ run_mpi_prog.pm,​ and auto3dem.pl.
   *       Minor performance improvements made to 3D interpolation routine interpl_3d.   *       Minor performance improvements made to 3D interpolation routine interpl_3d.
-  *       ​General cleanup of cmpt_intrps.F,​ fftsynth_1_m_slab.f,​ exchange_2_slab.f,​ P3dr.F +  * General cleanup of cmpt_intrps.F,​ fftsynth_1_m_slab.f,​ exchange_2_slab.f,​ P3dr.FPcut.f, ctf_para.f, Psf.f, Por.f, Pctfr.f, com_01.inc, intlz_arrays.f,​ and intlz_params.f. Specification of argument intents, removal of unused variables, etc.
-Pcut.f, ctf_para.f, Psf.f, Por.f, Pctfr.f, com_01.inc, intlz_arrays.f,​ and intlz_params.f. Specification of argument intents, removal of unused variables, etc.+
   *       ​General cleanup of PSF/​comp_sfactor.f. Move calculations contained within subroutine cplex2ap into comp_sfactor and get rid of unnecessary operations.   *       ​General cleanup of PSF/​comp_sfactor.f. Move calculations contained within subroutine cplex2ap into comp_sfactor and get rid of unnecessary operations.
   *       New control parameters added:   *       New control parameters added:
-  auto switch_mode (see note below) +    *   auto switch_mode (see note below) 
-  auto term_refine (functionality not yet active) +    ​* ​  auto term_refine (functionality not yet active) 
-  auto term_search (functionality not yet active)+    ​* ​  auto term_search (functionality not yet active)
   *       ​Auto3dem now monitors resolution and can automatically switch from search to refine mode if the following conditions are met:   *       ​Auto3dem now monitors resolution and can automatically switch from search to refine mode if the following conditions are met:
-  switch_mode flag is true +    *   switch_mode flag is true 
-  Resolution has not improved by at least 0.25Å over previous iteration +    ​* ​  Resolution has not improved by at least 0.25Å over previous iteration 
-  Mode is currently search+    ​* ​  Mode is currently search
   *       ​Iteration information is now tracked so that numbering from one run to the next is maintained. Restart and continuation files specify the last iteration that had been completed. The restart files also keep track of the number of iterations required to finish the original calculation.   *       ​Iteration information is now tracked so that numbering from one run to the next is maintained. Restart and continuation files specify the last iteration that had been completed. The restart files also keep track of the number of iterations required to finish the original calculation.
   *       For restart or continuation,​ new results are appended to the log and summary files.   *       For restart or continuation,​ new results are appended to the log and summary files.
   *       ​Restart files are written at two points in each iteration: after the completion of origin and orientation refinement and after construction of map. The restart files properly set the auto have_map flag so that map is constructed from particle images if program needs to be restarted after first checkpoint.   *       ​Restart files are written at two points in each iteration: after the completion of origin and orientation refinement and after construction of map. The restart files properly set the auto have_map flag so that map is constructed from particle images if program needs to be restarted after first checkpoint.
   *       ​Output files are named consistently using the name of the auto outfile parameter:   *       ​Output files are named consistently using the name of the auto outfile parameter:
-  outfile_log – detailed output log +    *   outfile_log – detailed output log 
-  outfile_summary – summary information +    ​* ​  outfile_summary – summary information 
-  outfile_restart_na – restart after first checkpoint, iteration n +    ​* ​  outfile_restart_na – restart after first checkpoint, iteration n 
-  outfile_restart_nb – restart after second checkpoint, iteration n +    ​* ​  outfile_restart_nb – restart after second checkpoint, iteration n 
-  outfile_continue – continuation file for successfully completed runs+    ​* ​  outfile_continue – continuation file for successfully completed runs
   *       ​Summary file now lists the number of particles that had been selected to construct the final map and the number of CPUs (MPI processes) used.   *       ​Summary file now lists the number of particles that had been selected to construct the final map and the number of CPUs (MPI processes) used.
   *       ​Format of restart and continuation files has been improved to logically group parameters.   *       ​Format of restart and continuation files has been improved to logically group parameters.
   *       ​Default value (0.1 pixels) specified for po2r dcenter, step size for origin refinement.   *       ​Default value (0.1 pixels) specified for po2r dcenter, step size for origin refinement.
   *       auto final_map no longer used and has been labeled as deprecated.   *       auto final_map no longer used and has been labeled as deprecated.
- 
 ==== auto3dem v1.04 ==== ==== auto3dem v1.04 ====
  
   *       ​Auto3dem version now printed in log and summary files   *       ​Auto3dem version now printed in log and summary files
   *       ​Symbolic links to executable Perl scripts in BIN directory created. If path information is setup correctly, can use name of script with or without .pl extension.   *       ​Symbolic links to executable Perl scripts in BIN directory created. If path information is setup correctly, can use name of script with or without .pl extension.
-  auto3dem ​  ​points to   ​auto3dem.pl +    *   auto3dem ​  ​points to   ​auto3dem.pl 
-  setup_rmc ​  ​points to   ​setup_rmc.pl +    ​* ​  setup_rmc ​  ​points to   ​setup_rmc.pl 
-  handflip ​  ​points to   ​handflip.pl +    ​* ​  handflip ​  ​points to   ​handflip.pl 
-  findbest ​  ​points to   ​findbest.pl+    ​* ​  findbest ​  ​points to   ​findbest.pl
   *       ​auto3dem and setup_rmc executed without any arguments provides usage information and version   *       ​auto3dem and setup_rmc executed without any arguments provides usage information and version
   *       ​Improved format for auto3dem summary data   *       ​Improved format for auto3dem summary data
Line 1139: Line 1291:
 Changes were implemented that make it much easier to run AUTO3DEM. Rather than creating symbolic links to Perl code and binaries, path information is simply added to the computing environment. This required some changes to the Perl code so that binaries and custom modules would always be found. Changes were implemented that make it much easier to run AUTO3DEM. Rather than creating symbolic links to Perl code and binaries, path information is simply added to the computing environment. This required some changes to the Perl code so that binaries and custom modules would always be found.
    
-auto3dem+== auto3dem ​==
   *       ​Custom Perl modules reside in the same directory as the executable Perl scripts. Added the following line to all .pl and .pm files so that custom modules would always be found, regardless of where the scripts are executed   *       ​Custom Perl modules reside in the same directory as the executable Perl scripts. Added the following line to all .pl and .pm files so that custom modules would always be found, regardless of where the scripts are executed
  
-use lib do { __FILE__ =~ m|^(.*?​)[^/​]*$|;​ "​$1";​ };+  ​use lib do { __FILE__ =~ m|^(.*?​)[^/​]*$|;​ "​$1";​ };
  
   *       ​Modified setup_rmc.pl so that executable Perl scripts in the command file are not pre-appended with '​./'​. For example   *       ​Modified setup_rmc.pl so that executable Perl scripts in the command file are not pre-appended with '​./'​. For example
  
-./​auto3dem.pl ​  ​becomes ​  ​auto3dem.pl+  ​./​auto3dem.pl ​  ​becomes ​  ​auto3dem.pl
  
   *       In module init_params.pm,​ get rid of full path information for binaries   *       In module init_params.pm,​ get rid of full path information for binaries
  
-make_all+== make_all ​==
   *       ​Modified so that all binaries are copied into the BIN directory   *       ​Modified so that all binaries are copied into the BIN directory
  
Line 1184: Line 1336:
 setup_rmc setup_rmc
   *       ​Script setup_rmc.pl modified so that default values for PPFT are overridden.   *       ​Script setup_rmc.pl modified so that default values for PPFT are overridden.
-  + 
-delta_theta = 1° (default 0.5°) +  delta_theta = 1° (default 0.5°) 
-verbose = -1 (default 2)+  verbose = -1 (default 2)
  
 auto3dem auto3dem
   *       ​parameter file now accepts po2r for specifying PO2R input. por still works to allow back compatibility with v1.0, but all internal perl variables renamed to use po2r (e.g. %por_ref ​   renamed internally to   ​%po2r_ref)   *       ​parameter file now accepts po2r for specifying PO2R input. por still works to allow back compatibility with v1.0, but all internal perl variables renamed to use po2r (e.g. %por_ref ​   renamed internally to   ​%po2r_ref)