Klauda Lab Simulation Wiki

User Tools

Site Tools

Trace:
Writing /users/jbklauda/pub/wiki/data/cache/3/36a3922e6748748aba38e43b093b0858.i failed
Unable to save cache file. Hint: disk full; file permissions; safe_mode setting.
Writing /users/jbklauda/pub/wiki/data/cache/3/36a3922e6748748aba38e43b093b0858.metadata failed
Writing /users/jbklauda/pub/wiki/data/cache/6/6193a558dc90ed4c190004bf67ecff70.captchaip failed
gmvae_update

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
gmvae_update [2026/08/02 06:22] – editgmvae_update [2026/08/03 12:43] (current) – edit
Line 6: Line 6:
 This is based off Dr. Ghorbani's GMVAE code available on [[https://github.com/ghorbanimahdi73/GMVAE-MD|Github]]. This has been updated to work with newer versions of python and Zaratan's architecture. Use on other HPC will likely need tweaking from the instructions below. This is based off Dr. Ghorbani's GMVAE code available on [[https://github.com/ghorbanimahdi73/GMVAE-MD|Github]]. This has been updated to work with newer versions of python and Zaratan's architecture. Use on other HPC will likely need tweaking from the instructions below.
  
-Tar file for all code: {{ :gmcave_tf1_update_copy.tgz |}}+Tar file for all code:  {{ :gmvae_tf1_v2.tgz |}}
  
 The detailed instructions were added by Dr. Min-Kang Hsieh. The detailed instructions were added by Dr. Min-Kang Hsieh.
Line 73: Line 73:
 </code> </code>
  
-**Usage note: copy the scripts folder as your project name to begin.**  +**Usage note: copy the scripts folder as your project name to begin.** 
 +  
 +Note: If the servers where no graphical user interface (GUI) or display server is available,  add matplotlib.use("Agg") before importing pyplot to force Matplotlib to use a non-GUI backend, allowing you to generate and save plots without opening a visual window.  
 + 
 +<code> 
 +import matplotlib 
 +matplotlib.use("Agg")  
 +import matplotlib.pyplot as plt 
 +</code> 
 + 
 + 
  
 **Input data preparation** **Input data preparation**
Line 195: Line 205:
  
 Once a suitable set of training hyperparameters has been determined (excluding K and z), perform a systematic grid search to identify the optimal latent-space dimension and number of clusters. Once a suitable set of training hyperparameters has been determined (excluding K and z), perform a systematic grid search to identify the optimal latent-space dimension and number of clusters.
-<code>+
 **Example** **Example**
 +<code>
 python gmvae_pipeline_v2.py --mode grid \ python gmvae_pipeline_v2.py --mode grid \
   --pdb /home/mhsieh/scratch/bst2/orf7a-cov-2_bst2_short/aa/c2_r1_cg2aa/openmm/ca_dcd/ca.pdb \   --pdb /home/mhsieh/scratch/bst2/orf7a-cov-2_bst2_short/aa/c2_r1_cg2aa/openmm/ca_dcd/ca.pdb \
Line 266: Line 277:
 **GMVAE Operation details** **GMVAE Operation details**
  
 +Detailed instructions for using this gmvae script are provided below.  
 +
 +<code>
 +usage: gmvae_pipeline_v2.py [-h] --mode {train,grid,eval} --pdb PDB
 +                            --dataset-npy DATASET_NPY --rmsd-npy RMSD_NPY
 +                            [--mode_chain {single_chain,dimer}]
 +                            [--indices1 INDICES1 INDICES1]
 +                            [--indices2 INDICES2 INDICES2]
 +                            [--batch-size BATCH_SIZE] [--lr LR]
 +                            [--epochs EPOCHS] [--num-layers NUM_LAYERS]
 +                            [--z-dim Z_DIM] [--temp TEMP] [--K K]
 +                            [--restore RESTORE] [--dense-n DENSE_N]
 +                            [--filters FILTERS [FILTERS ...]]
 +                            [--kernel-sizes KERNEL_SIZES [KERNEL_SIZES ...]]
 +                            [--pool-sizes POOL_SIZES [POOL_SIZES ...]]
 +                            [--valid-ratio VALID_RATIO] [--dropout DROPOUT]
 +                            [--r-nent R_NENT] [--r-label R_LABEL]
 +                            [--output-dir OUTPUT_DIR]
 +                            [--grid-ks [GRID_KS [GRID_KS ...]]]
 +                            [--grid-zs [GRID_ZS [GRID_ZS ...]]]
 +                            [--evaluate-after-train] [--num-cpus NUM_CPUS]
 +                            [--model-dir MODEL_DIR]
 +
 +GMVAE training & grid testing pipeline
 +
 +optional arguments:
 +  -h, --help            show this help message and exit
 +  --mode {train,grid,eval}
 +                        Mode to run: train | grid | eval
 +  --pdb PDB             Path to pdb file
 +  --dataset-npy DATASET_NPY
 +                        Path to dataset .npy file
 +  --rmsd-npy RMSD_NPY   Path to RMSD .npy file
 +  --mode_chain {single_chain,dimer}
 +                        Single chain or dimer analysis (affects indices_range2
 +                        selection)
 +  --indices1 INDICES1 INDICES1
 +                        Range for chain 1 indices, e.g., 0 99.
 +  --indices2 INDICES2 INDICES2
 +                        Range for chain 2 indices, e.g., 100 220.
 +  --batch-size BATCH_SIZE
 +  --lr LR
 +  --epochs EPOCHS
 +  --num-layers NUM_LAYERS
 +  --z-dim Z_DIM
 +  --temp TEMP
 +  --K K
 +  --restore RESTORE
 +  --dense-n DENSE_N
 +  --filters FILTERS [FILTERS ...]
 +  --kernel-sizes KERNEL_SIZES [KERNEL_SIZES ...]
 +  --pool-sizes POOL_SIZES [POOL_SIZES ...]
 +  --valid-ratio VALID_RATIO
 +  --dropout DROPOUT
 +  --r-nent R_NENT
 +  --r-label R_LABEL
 +  --output-dir OUTPUT_DIR
 +                        Base output directory
 +  --grid-ks [GRID_KS [GRID_KS ...]]
 +                        List of K values for grid
 +  --grid-zs [GRID_ZS [GRID_ZS ...]]
 +                        List of z-dim values for grid
 +  --evaluate-after-train
 +                        Run evaluation after each training run (CPU)
 +  --num-cpus NUM_CPUS   Number of CPU cores to pass to evaluation
 +  --model-dir MODEL_DIR
 +                        Model output dir to evaluate (used in eval mode)
 + 
 +</code>
 **1. Hyperparameter Optimization (excluding K and z)** **1. Hyperparameter Optimization (excluding K and z)**
  
Line 281: Line 361:
 Systems       # of     # of   Latent   # of     Batch           Kernel   Learning   # of        Pooling Systems       # of     # of   Latent   # of     Batch           Kernel   Learning   # of        Pooling
              layers   neurons  dim.  clusters size Temp. size rate   filters sizes     Ref              layers   neurons  dim.  clusters size Temp. size rate   filters sizes     Ref
-Trp-cage 2 64 5 8 5000 0.1 [3,3]   0.001   [64,64] [1,1]    J. Chem. Phys. 155, 194108 (2021) doi: 10.1063/5.0069708 +Trp-cage 2 64 5 8 5000 0.1 [3,3]   0.001   [64,64] [1,1]    J. Chem. Phys. (2021) 155, 194108  doi: 10.1063/5.0069708 
-BBA         2 64 6 9 5000 0.1 [3,3]   0.001   [64,64] [2,2]    J. Chem. Phys. 155, 194108 (2021) doi: 10.1063/5.0069708 +BBA         2 64 6 9 5000 0.1 [3,3]   0.001   [64,64] [2,2]    J. Chem. Phys. (2021) 155, 194108  doi: 10.1063/5.0069708 
-villin         3 64 5 6 2500 0.05 [3,3,3]   0.001   [64,64,32] [2,2,1]    J. Chem. Phys. 155, 194108 (2021) doi: 10.1063/5.0069708 +villin         3 64 5 6 2500 0.05 [3,3,3]   0.001   [64,64,32] [2,2,1]    J. Chem. Phys. (2021) 155, 194108  doi: 10.1063/5.0069708 
-BST-2/ORF7a 3 64 5 9 2500 0.05 [3,3,3]   0.001   [64,64,32] [2,2,1]    BBA Biomembranes (2023)1865 (6): 184174. doi: 10.1016/j.bbamem.2023.184174.  +BST-2/ORF7a 3 64 5 9 2500 0.05 [3,3,3]   0.001   [64,64,32] [2,2,1]    BBA Biomembranes (2023) 1865(6), 184174. doi: 10.1016/j.bbamem.2023.184174 
-BST-2/ORF7a 3 64 5 13 2500 0.05 [3,3,3]   0.001   [64,64,32] [2,2,1]    BBA Biomembranes (2023)1865 (6): 184174. doi: 10.1016/j.bbamem.2023.184174.  +BST-2/ORF7a 3 64 5 13 2500 0.05 [3,3,3]   0.001   [64,64,32] [2,2,1]    BBA Biomembranes (2023) 1865(6), 184174. doi: 10.1016/j.bbamem.2023.184174  
-ORF7b dimer 3 64 6 10 2500 0.05 [3,3,3]   0.001   [64,64,32] [2,2,1]    J Phys Chem B. (2024) 128(1):150-162. doi: 10.1021/acs.jpcb.3c07105 +ORF7b dimer 3 64 6 10 2500 0.05 [3,3,3]   0.001   [64,64,32] [2,2,1]    J. Phys. Chem. B (2024) 128(1), 150-162. doi: 10.1021/acs.jpcb.3c07105 
-ORF7b dimer 3 64 6 8 2500 0.05 [3,3,3]   0.001   [64,64,32] [2,2,1]    J Phys Chem B. (2024) 128(1):150-162. doi: 10.1021/acs.jpcb.3c07105 +ORF7b dimer 3 64 6 8 2500 0.05 [3,3,3]   0.001   [64,64,32] [2,2,1]    J. Phys. Chem. B (2024) 128(1), 150-162. doi: 10.1021/acs.jpcb.3c07105 
-BST-2/ORF7a 3 64 4 11 1000 0.05 [3,3,3]   0.001   [64,64,32] [2,2,1]    J. Phys. Chem. B (2026) 130 (5): 1553–1567. doi: 10.1021/acs.jpcb.5c07529+BST-2/ORF7a 3 64 4 11 1000 0.05 [3,3,3]   0.001   [64,64,32] [2,2,1]    J. Phys. Chem. B (2026) 130(5), 1553–1567. doi: 10.1021/acs.jpcb.5c07529
  
 </code> </code>
Line 318: Line 398:
 </code> </code>
  
-**Note: If the whole usage of data set is not a case, please specify the --start and --end  in the eval mode. **+**My suggestion is to adjust the number of input data points to an appropriate batch size to ensure 0 leftover.** 
 + 
 +**Note: The cluster label (in neither y tot.npy nor z tot.npy) would not be assigned to the leftovers. If the entire dataset is not used, please specify --start and --end in eval mode. **
  
 **1.4. Recommended Optimization Strategy** **1.4. Recommended Optimization Strategy**
Line 356: Line 438:
  
 After selecting an optimal (K, z) configuration, representative conformations from each cluster can be extracted for structural analysis and visualization. The contact_heatmap_top_structures.py script generates cluster-wise contact maps and identifies representative structures based on their cluster assignment probabilities. After selecting an optimal (K, z) configuration, representative conformations from each cluster can be extracted for structural analysis and visualization. The contact_heatmap_top_structures.py script generates cluster-wise contact maps and identifies representative structures based on their cluster assignment probabilities.
-The script below performs the following tasks:+<code> 
 +usage: contact_heatmap_top_structures.py [-h] --mode {dataset,trajectory} 
 +                                         --mode_chain {single_chain,dimer} 
 +                                         [--A_dataset A_DATASET] 
 +                                         [--B_dataset B_DATASET] 
 +                                         [--AB_dataset AB_DATASET] 
 +                                         [--A_size A_SIZE] [--B_size B_SIZE] 
 +                                         [--pdb PDB] 
 +                                         [--traj_files TRAJ_FILES [TRAJ_FILES ...]] 
 +                                         [--traj_dirs TRAJ_DIRS [TRAJ_DIRS ...]] 
 +                                         [--traj_prefix TRAJ_PREFIX] 
 +                                         [--traj_ext {dcd,trr,xtc}] 
 +                                         [--start START] [--end END] 
 +                                         [--skip SKIP] 
 +                                         [--skip_first_frame SKIP_FIRST_FRAME] 
 +                                         [--skip_after_frame SKIP_AFTER_FRAME] 
 +                                         [--selected_atom SELECTED_ATOM] 
 +                                         [--A_indices A_INDICES A_INDICES] 
 +                                         [--B_indices B_INDICES B_INDICES] 
 +                                         --cluster_folder CLUSTER_FOLDER 
 +                                         [--threshold THRESHOLD] 
 +                                         [--cutoff CUTOFF] 
 +                                         [--output_dir OUTPUT_DIR] 
 +                                         [--save_generated_dataset] 
 +                                         [--generate_top_structures] 
 +                                         [--top_n TOP_N] 
 +                                         [--mean_iterations MEAN_ITERATIONS] 
 +                                         [--representative_output_dir REPRESENTATIVE_OUTPUT_DIR] 
 + 
 +</code> 
 + 
 +The script performs the following tasks:
   * Reads one or more molecular dynamics trajectories; it has to follow exactly the data_collection setting.    * Reads one or more molecular dynamics trajectories; it has to follow exactly the data_collection setting. 
   * Loads GMVAE clustering results (y_tot.npy) from the specified cluster_folder.    * Loads GMVAE clustering results (y_tot.npy) from the specified cluster_folder. 
Line 363: Line 476:
   * Identifies the top representative conformations (highest cluster assignment probability) from each cluster.    * Identifies the top representative conformations (highest cluster assignment probability) from each cluster. 
   * Writes representative structures to individual trajectory files for downstream visualization and analysis.   * Writes representative structures to individual trajectory files for downstream visualization and analysis.
 +
 +**Example script**
 +
 <code> <code>
 #!/bin/bash #!/bin/bash
Line 399: Line 515:
              ${dir6}${file_traj1} ${dir6}${file_traj2} ${dir6}${file_traj3} ${dir6}${file_traj4} ${dir6}${file_traj5}               ${dir6}${file_traj1} ${dir6}${file_traj2} ${dir6}${file_traj3} ${dir6}${file_traj4} ${dir6}${file_traj5} 
 </code> </code>
- 
-data_collection.py 
- 
-Required python (main) packages 
-channels:  
-  * conda-forge 
-  * defaults 
-dependencies: 
-  * python=3.9 
-  * numpy 
-  * mdtraj 
-  * pyemma 
- 
-or using provided pyemma.yml to build conda environment. 
- 
-=== Truncating Trajectories === 
- 
-For large systems, it is recommended to trim the trajectory to include only the heavy atoms of interest (e.g., CA). 
- 
-Data file archive: save pdb in ca_dcd/ (e.g. ca_dcd/ca_dyn0.pdb) 
- 
-Example: 
-<code> 
-python data_collection.py --pdb /home/mhsieh/scratch/bst2/orf7a-cov-2_bst2_short/aa/c2_r1_cg2aa/openmm/ca_dcd/ca.pdb --traj_dirs "/home/mhsieh/scratch/bst2/orf7a-cov-2_bst2_short/aa/c2_r1_cg2aa/openmm/ca_dcd/" "/home/mhsieh/scratch/bst2/orf7a-cov-2_bst2_short/aa/c2_r2_cg2aa/openmm/ca_dcd/" --traj_prefix ca_dyn  --start 501 --end 550 –-dimer –-skip 4 –-output_prefix orf7a_bst2_short </code> 
- 
-=== Starting Data Collection Pipeline === 
- 
-Length of indices1: 100 
-Length of indices2: 121 
-Feature dimension: 12100 
-100%|████████████████████| 100/100 [00:14<00:00,  6.82it/s] 
-100%|████████████████████| 100/100 [00:01<00:00, 97.15it/s] 
-Final all_data length: 10000 
-Computed RMSD for 10000 frames. 
-Saved outputs: orf7a_bst2_short_data.npy and orf7a_bst2_short_rmsd.npy 
- 
-=== Pipeline Completed === 
- 
-Using tf1.15.4.yml to build conda environment (pyemma excluded). 
-GMVAE clustering via Zaratan v100 GPU 
- 
-Example of run.sh 
-<code> 
-#!/bin/bash 
-#SBATCH --job-name=k6_z8_r1 
-#SBATCH -n 8 
-#SBATCH --partition=GPU 
-#SBATCH --gpus=v100:1 
-#SBATCH --time=01:30:00 
- 
-source ~/.bashrc 
-conda init --all 
-conda activate /home/mhsieh/scratch/tf1.15.4/ 
-pip list | grep tensorflow 
-echo "python gmvae_pipeline.py --mode train \ 
-  --pdb /home/mhsieh/scratch/bst2/orf7a-cov-2_bst2_short/aa/c2_r1_cg2aa/openmm/ca_dcd/ca.pdb \ 
-  --dataset-npy orf7a_bst2_short_data.npy \ 
-  --rmsd-npy orf7a_bst2_short_rmsd.npy \ 
-  --mode_chain dimer \ 
-  --indices1 0 99 \ 
-  --indices2 100 220 \ 
-  --K 6 --z-dim 8 \ 
-  --epochs 100 --batch-size 500 \ 
-  --output-dir outputs" 
- 
-python gmvae_pipeline.py --mode train \ 
-  --pdb /home/mhsieh/scratch/bst2/orf7a-cov-2_bst2_short/aa/c2_r1_cg2aa/openmm/ca_dcd/ca.pdb \ 
-  --dataset-npy orf7a_bst2_short_data.npy \ 
-  --rmsd-npy orf7a_bst2_short_rmsd.npy \ 
-  --mode_chain dimer \ 
-  --indices1 0 99 \ 
-  --indices2 100 220 \ 
-  --K 6 --z-dim 8 \ 
-  --epochs 100 --batch-size 500 \ 
-  --output-dir outputs 
-</code> 
- 
-If process evaluation right after clustering, the add   --evaluate_after_train option. 
-<code> 
-python gmvae_pipeline.py --mode train \ 
-  --pdb /home/mhsieh/scratch/bst2/orf7a-cov-2_bst2_short/aa/c2_r1_cg2aa/openmm/ca_dcd/ca.pdb \ 
-  --dataset-npy orf7a_bst2_short_data.npy \ 
-  --rmsd-npy orf7a_bst2_short_rmsd.npy \ 
-  --mode_chain dimer \ 
-  --indices1 0 99 \ 
-  --indices2 100 220 \ 
-  --K 6 --z-dim 4 \ 
-  --epochs 10 --batch-size 500 \ 
-  --output-dir outputs \ 
-  --evaluate_after_train 
-</code> 
- 
-If run evaluation separately, using the following script. 
-<code> 
-#!/bin/bash 
-#SBATCH --job-name=k11_z4_r1 
-#SBATCH -N 1 
-#SBATCH -n 64 
-#SBATCH --partition=standard 
-#SBATCH --time=01:30:00 
- 
- 
-source ~/.bashrc 
-conda init --all 
-conda activate /home/mhsieh/scratch/tf1.15.4/ 
- 
-pip list | grep tensorflow 
- 
-echo "python gmvae_pipeline.py --mode eval \ 
-  --pdb /home/mhsieh/scratch/bst2/orf7a-cov-2_bst2_short/aa/c2_r1_cg2aa/openmm/ca_dcd/ca.pdb \ 
-  --dataset-npy orf7a_bst2_short_data.npy \ 
-  --rmsd-npy orf7a_bst2_short_rmsd.npy \ 
-  --mode_chain dimer \ 
-  --indices1 0 99 \ 
-  --indices2 100 220 \ 
-  --K 6 --z-dim 8 \ 
-  --batch-size 1 \ 
-  --output-dir outputs \ 
-  --model-dir ./outputs/K6_z8 \ 
-  --num-cpus 64" 
- 
-python gmvae_pipeline.py --mode eval \ 
-  --pdb /home/mhsieh/scratch/bst2/orf7a-cov-2_bst2_short/aa/c2_r1_cg2aa/openmm/ca_dcd/ca.pdb \ 
-  --dataset-npy orf7a_bst2_short_data.npy \ 
-  --rmsd-npy orf7a_bst2_short_rmsd.npy \ 
-  --mode_chain dimer \ 
-  --indices1 0 99 \ 
-  --indices2 100 220 \ 
-  --K 6 --z-dim 8 \ 
-  --batch-size 1 \ 
-  --output-dir outputs \ 
-  --model-dir ./outputs/K6_z8 \ 
-  --num-cpus 64 
-</code> 
- 
  
gmvae_update.1785666161.txt.gz · Last modified: 2026/08/02 06:22 by edit