Differences

This shows you the differences between two versions of the page.

--- auto3dem:userguide [2015/07/08 15:22]
admin [Auto3dem keyword default values]
+++ auto3dem:userguide [2016/01/19 13:04] (current)
admin [auto]
@@ Line 192: / Line 192: @@
 ppft bin_factor   2 # Start with 2x2 binning of images
 </code>
+==== Controlling distribution on CPUs in parallel runs ====
+All the programs capable of running on multiple processors use the Message Passing Interface (MPI) library for parallelization. In these case the processors are assigned automatically at launch by means of a machinefile, a file listing all the nodes assigned to the run. In Auto3DEM it is possible to control how these nodes are assigned and their scheduling method.
+One control parameter is
+  auto evod_parallel 0/1
+If set to 1, orientation search and reconstructions from even and odd stacks are done at the same time (assuming that the program is running in gold standard mode). Specifically, the processors are divided into two halves, by affinity (i.e. as many processors in the same nodes), and two instances of the programs are launched concurrently.
+If the size of the reconstructions is large, compared to the memory available in a single node, then there is a risk that the reconstruction routine could freeze because of insufficient memory, especially when multiple processes are running on one node, therefore sharing the same memory. In this case it could be advantageous to spread the CPUs among separate nodes, so that they don't compete with each other for resources. This can be obtained by setting the parameter
+  p3dr cpu_sparse
+to 1, which distributes the processors among the two concurrent reconstructions according to a round-robin scheduling approach. For example, if you are launching auto3dem on 8 nodes, asking all the processors on each of them, while you limit P3DR to use only 4 cpus (''p3dr max_cpu 4''), these 4 cpus will be on separate nodes, and the two reconstructions from even- and odd-numbered particles will run at the same time, but each MPI process will be on a different node. Of course, this is kind of best case scenario.
 =====Full listing of auto3dem keywords=====
@@ Line 211: / Line 226: @@
 **estimate_res**: non-zero value indicates that resolution estimation is performed.
-**flatten_map**: flag specifying whether or not background density should be removed from the reference map.
+**evod_parallel**: non-zero value indicates that the processors are distributed by affinity between the two half sets of particles, which are processed at the same time. This parameter is valid only if the program is compiled for parallel execution (MPI)
 **flatten_falloff**: size (pixels) of soft edge to use when removing background density flag (-1= automatically determined, based on map resolution; 0=hard edge; n=soft edge n pixels width).
+**flatten_map**: flag specifying whether or not background density should be removed from the reference map.
 **freeze_annulus**: non-zero value freezes inner and outer radii of the annulus defining the ordered region of the map. This affects the parameters annulus_low and annulus_high in PPFT and in_rad and out_rad in PCUT.
@@ Line 219: / Line 236: @@
 **freeze_res**: non-zero value freezes the resolutions used in PPFT, and PO2R (resolution used in P3DR will still be based on the results from PSF).
-**fsc_hithresh**: cutoff value for FSC used in estimating map resolution.
+**fsc_cutoff**: cutoff value for FSC used in estimating map resolution. If not specified it is set either to 0.143 (gold_standard set to 1) or to 0.5 (gold_standard set to 0).
-**fsc_lothresh**: cutoff value for FSC used to set resolution limits in P3DR, PSF, and PO2R.
+**fsc_hithresh**: cutoff value for FSC used to set resolution limits in PPFT and PO2R.
+**fsc_lothresh**: cutoff value for FSC used in filtering the map in P3DR.
 **gauss_adj**: parameter used to set width of Gaussian falloff in P3DR (reciprocal angstroms).
+**gold_standard**: flag (0/1) indicating whether or not to use the gold standard procedure. If set to 1 input even and odd maps are required (start_map_even and start_map_odd).
 **generate_map**: controls whether to generate or not the reconstruction in combination with the alignment of the particles. If the map is not calculated, the parameter niter is forced to one; also, in the new continue file the parameter iter_start will not be updated, and generate_map will be set for calculating the map (mode = ‘only’). If only the map is calculated, the filename of the reconstruction can be changed from the default by using the parameter map_suffix. Allowed values = (yes,no,only).
@@ Line 245: / Line 266: @@
 **iter_start**: starting iteration, i.e. number assigned to first iteration.
-**map_suffix**: string suffix to append to the name of the reconstruction. Used only if generate_map is set to ‘only’. Default is ‘none’, a reserved string indicating that the name is assigned according to the standard rules. :!:__Warning__: ‘none’ implies that for multiple runs of only-reconstructions each one will overwrite the previous result.
+**map_suffix**: string suffix to append to the name of the reconstruction. Used only if generate_map is set to ‘only’. Default is ‘none’, a reserved string indicating that the name is assigned according to the standard rules. :!: __Warning__ ‘none’ implies that for multiple runs of only-reconstructions each one will overwrite the previous result.
 **mode**: AUTO3DEM mode of operation. Allowed values = (search, refine).
@@ Line 276: / Line 297: @@
 **start_map**: name of starting map used by AUTO3DEM.
+**start_map_even**: name of starting map from even half set used by AUTO3DEM.
+**start_map_odd**: name of starting map from odd half set used by AUTO3DEM.
 **switch_mode**: if true, allows auto3dem to automatically switch from search to refine mode.
@@ Line 325: / Line 350: @@
 **bin**: name of P3DR binary.
+**cpu_sparse**: a non-zero value indicates that the program tries to spread the processors assigned to each reconstruction among different nodes. This is useful for large maps, so that each process does not have to share the memory on the node with other processes.
 **ctf_ff1**: 1st CTF filter factor.
@@ Line 420: / Line 447: @@
 **magref_reset**: 1 to ‘forget’ magnification assigned to  data, 0 to refine around the current estimated value.
-**magref_step**: size of magnification refinement step (microns).
+**magref_step**: size of magnification refinement step (fraction).
+**mask_map**: name of binary map to use for focused refinement. Only used for modes //local// and //ticos_equiv//. Default is ‘none’, a reserved string indicating that no mask is used.
 **max_cpu**: maximum number of CPUs to be used by PO2R
@@ Line 427: / Line 456: @@
 <WRAP center round box 75%>
 ''local'': local refinement (//nangle// steps of //dangle// degrees along each direction)\\
-''mag'': magnification refinement (//nmagref// steps of //magref//_step microns along each direction)\\
+''mag'': magnification refinement (//nmagref// steps of //magref//_step fraction along each direction)\\
 ''global'': global search in one asymmetric unit (on a grid with step of //gangle// degrees)\\
 ''ticos_equiv'': restricted search to the 60 symmetry related orientations.
@@ Line 436: / Line 465: @@
 **ncenter**: number of spatial steps taken in each direction.
-**nmagf**: number of magnification factor steps along each direction.
+**nmagref**: number of magnification factor steps along each direction.
 **per_ptle_ctf**: apply CTF correction on a per-particle basis.
@@ Line 527: / Line 556: @@
 The following tables list the auto3dem input parameters along with their default values. A missing default value means that no default is used. Required input shown in bold **<fc red>red</fc>**.
+^ identifier  ^ key          ^ value  ^
+| auto | adapt_angle | yes |
+| auto | bin_reduce | 0 |
+| auto | boxrad | extracted from PIF (see note 1) |
+| auto | box_center_offset | boxrad/2 |
+| auto | cmp_cc_fraction |
+| auto | cmp_cc_nstd |
+| auto | delete_maps | 1 |
+| auto | estimate_res | 1 |
+| auto | evod_parallel | 1 |
+| auto | flatten_falloff | -1 (automatic) |
+| auto | flatten_map | 0 |
+| auto | freeze_annulus | 0 |
+| auto | freeze_res | 0 |
+| auto | fsc_cutoff | 0.143 if gold_standard=1, 0.5 otherwise |
+| auto | fsc_hithresh | 0.5 |
+| auto | fsc_lothresh | 0.3 |
+| auto | gauss_adj | 0.01 |
+| auto | generate_map | yes |
+| auto | global_select | 1 |
+| auto | gold_standard | 0 |
+| auto | have_map | 1 |
+| auto | hollow_auto | 1 |
+| auto | hollow_cut_step | 12 |
+| auto | hollow_cut_weight | 0.001 |
+| auto | hollow_in_rad | required if hollow_map=1, updated automatically if hollow_auto=1 |
+| auto | hollow_map | 0 |
+| auto | hollow_out_rad | required if hollow_map=1, updated automatically if hollow_auto=1 |
+| auto | iter_start | 1 |
+| auto | map_suffix | none (reserved string) |
+| **<fc red>auto</fc>** | **<fc red>mode</fc>** |
+| auto | new_ptles | 0 |
+| **<fc red>auto</fc>** | **<fc red>niter</fc>** |
+| auto | noctf | 0 |
+| auto | noise_suppression | 0 |
+| auto | nselect_offset | 0 |
+| auto | omega1 | 0 |
+| auto | omega1_tol | 360 |
+| auto | omega2 | 0 |
+| auto | omega2_tol | 0 |
+| auto | outfile | current working directory |
+| auto | partrad | initially: same as boxrad; later: from map stats (see note 2) |
+| auto | per_ptle_ctf | 0 |
+| auto | pft_cc_fraction | 1 |
+| auto | pft_cc_nstd |
+| auto | phi_reject_lower | 360 |
+| auto | phi_reject_upper | -360 |
+| auto | prj_cc_fraction |
+| auto | prj_cc_nstd |
+| auto | quit_early | 0 |
+| auto | refine_ctf | 0 |
+| auto | res_adj | 0.01 |
+| auto | restart | 0 |
+| auto | rmc | 0 |
+| auto | rundir | dat |
+| auto | score_fraction | 1 |
+| auto | select_delta | 0 |
+| **<fc red>auto</fc>** | **<fc red>start_map</fc>** | see note 3 |
+| **<fc red>auto</fc>** | **<fc red>start_map_even</fc>** | see note 3 |
+| **<fc red>auto</fc>** | **<fc red>start_map_odd</fc>** | see note 3 |
+| auto | switch_mode | 1 |
+| auto | symm_code | overrides values set for po2r, ppft, p3dr |
+| auto | term_refine | 0 (not used) |
+| auto | term_search | 0 (not used) |
+| auto | theta_reject_lower | 360 |
+| auto | theta_reject_upper | -360 |
+||
+| pcut | bin | PCUT |
+| pcut | cut_step | 12 |
+| pcut | cut_weight | 0.001 |
+| pcut | in_rad | boxrad/3 |
+| pcut | max_cpu | 8 |
+| pcut | out_rad | boxrad - 10 |
+||
+| p3dr | apo_border | 12 |
+| p3dr | bin | P3DR |
+| p3dr | cpu_sparse | 0 |
+| p3dr | ctf_ff1 | 1 |
+| p3dr | ctf_ff2 | 0.1 |
+| p3dr | ctfmode | 1 |
+| p3dr | fsc_file_name |
+| p3dr | filter | 1 |
+| p3dr | magfactor | 1.0 |
+| p3dr | map_dim | 0 |
+| p3dr | max_cpu | 256 |
+| p3dr | per_ptle_ctf | 0 |
+| p3dr | res_max | 1/(1/p3dr{res_min} + auto{gauss_adj}) (see note 4) |
+| **<fc red>p3dr</fc>** | **<fc red>res_min</fc>** |
+| p3dr | symm_code | 532 |
+| p3dr | tempfac | 0 |
+| p3dr | zero_fill | 1 |
+||
+| pctfr | anastigm | 0 |
+| pctfr | bin | PCTFR |
+| pctfr | ctf_ff1 | 0.5 |
+| pctfr | ctf_ff2 | 0.1 |
+| pctfr | ctfmode | 1 |
+| pctfr | dangle | 1 (degrees) |
+| pctfr | dfocus | 0.05 (microns) |
+| pctfr | funcmode | 3 (recommended) |
+| pctfr | funcweight | 0 (ignored for funcmode values 4 and 5) |
+| pctfr | max_cpu | 256 |
+| pctfr | nangle | 4 |
+| pctfr | nfocus | 4 |
+| pctfr | per_ptle_ctf | 0 |
+| pctfr | res_max | 1/(1/p3dr{res_min} + auto{res_adj}) |
+| pctfr | res_min | (2∙boxrad∙pixel_size)/5 |
+| pctfr | tempfac | 0 |
+| pctfr | zero_fill | 1 |
+||
+| po2r | bin | POR |
+| po2r | ctfmode | 2 |
+| po2r | dangle | (1/2)∙ppft{delta_theta} - ignored if ‘auto adapt_angle’ set to yes. |
+| po2r | dcenter | 0.1 |
+| po2r | funcmode | 4 |
+| po2r | funcweight | 1 (ignored for funcmode values 4 and 5) |
+| po2r | gangle | 2 |
+| po2r | global_por | 0 (obsolete) |
+| po2r | handtest | 1 |
+| po2r | magref_calibrate | 0 |
+| po2r | magref_reset | 1 |
+| po2r | magref_step | 0.005 |
+| po2r | mask_map | none (reserved string) |
+| po2r | max_cpu | 256 |
+| po2r | nangle | 4 |
+| po2r | ncenter | 4 |
+| po2r | nmagref | 10 |
+| po2r | per_ptle_ctf | 0 |
+| po2r | quick_search | 1 |
+| po2r | res_max | 1/(1/p3dr{res_min} + auto{gauss_adj}) |
+| po2r | res_min | (2∙boxrad∙pixel_size)/5 |
+| po2r | symm_code | 532 |
+| po2r | tempfac | 0 |
+| po2r | ticos_equiv | 0 (obsolete) |
+| po2r | zero_fill | 1 |
+||
+| ppft | annulus_high | boxrad/3 |
+| ppft | annulus_low | boxrad – 10 |
+| ppft | bin | PPFT |
+| ppft | bin_factor | 1 |
+| ppft | ctf_mode | 3 |
+| ppft | delta_theta | 0.5  (degrees, ignored if ‘auto adapt_angle’ set to yes) |
+| ppft | filter_factor_1 | 0.1 |
+| ppft | input_mode | 2 |
+| ppft | jcut | 1 |
+| ppft | mag_cen | 1 |
+| ppft | mag_norm | 1 |
+| ppft | mag_num | 1 |
+| ppft | mag_step | 0 |
+| ppft | max_cpu | 256 |
+| ppft | model_filename | auto{start_map} |
+| ppft | per_ptle_ctf | 0 |
+| ppft | pft_filename | pft.pfts |
+| ppft | pftrad_hi | auto{boxrad} |
+| ppft | pftrad_lo | 1 |
+| ppft | pftrad_step | 1 |
+| ppft | prj_filename | pft.prjs |
+| ppft | quick_omega | 1 |
+| ppft | resolution_high | 1/(1/p3dr{res_min} + auto{res_adj}) |
+| ppft | resolution_low | (2∙boxrad∙pixel_size)/5 |
+| ppft | sigcut | 0 |
+| ppft | symmetry | 532 (obsolete) |
+| ppft | symm_code | 532 |
+| ppft | temperature_factor | 0 |
+| ppft | verbose | -1 |
+||
+| psf | bin | PSF |
+| psf | max_cpu | 8 |
+| psf | res_max | p3dr{res_min} (see note 5) |
+| psf | res_min | 60 |
+| psf | res_step | 50 (see note 6) |
+__Notes__
+) Boxrad is normally extracted from the PIF file header field packRadius. The option to specify boxrad in the auto3dem parameter file is provided for use in those cases where this information is missing from the PIF header.
+) Adaptive value: can be set arbitrarily in the continue file, but that value is only used for the first iteration, while it is changed for the subsequent iterations.
+) If the have_map flag is false, then starting map is not required. start_map_even and start_map_odd are required if gold_standard is set to 1.
+) Subject to Nyquist condition that resolution is not less than twice the pixel size.
+) Subject to condition that value is not more than three times the pixel size.
+) Values defined by user are ignored, since the number of steps are adaptively determined starting from resolution range, pixel size and particle radius.