User Tools

Site Tools


Writing /users/jbklauda/pub/wiki/data/cache/3/36a3922e6748748aba38e43b093b0858.i failed
Unable to save cache file. Hint: disk full; file permissions; safe_mode setting.
Writing /users/jbklauda/pub/wiki/data/cache/3/36a3922e6748748aba38e43b093b0858.metadata failed
Writing /users/jbklauda/pub/wiki/data/cache/6/6193a558dc90ed4c190004bf67ecff70.captchaip failed
gmvae_update

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
gmvae_update [2026/08/02 03:34] – editgmvae_update [2026/08/03 12:43] (current) – edit
Line 6: Line 6:
 This is based off Dr. Ghorbani's GMVAE code available on [[https://github.com/ghorbanimahdi73/GMVAE-MD|Github]]. This has been updated to work with newer versions of python and Zaratan's architecture. Use on other HPC will likely need tweaking from the instructions below. This is based off Dr. Ghorbani's GMVAE code available on [[https://github.com/ghorbanimahdi73/GMVAE-MD|Github]]. This has been updated to work with newer versions of python and Zaratan's architecture. Use on other HPC will likely need tweaking from the instructions below.
  
-Tar file for all code: {{ :gmcave_tf1_update_copy.tgz |}}+Tar file for all code:  {{ :gmvae_tf1_v2.tgz |}}
  
-The detailed instruction was added by Dr. Min-Kang Hsieh.+The detailed instructions were added by Dr. Min-Kang Hsieh.
  
 This instruction includes a complete workflow for unsupervised clustering of protein conformations using pairwise Cα (alpha-carbon) distance matrices (or BB atoms in CG modeling) as the input representation for a Gaussian Mixture Variational Autoencoder (GMVAE). The workflow covers dataset preparation, model hyperparameter optimization, systematic grid search over the number of Gaussian mixture components (K) and latent-space dimensions (z), and latent-space evaluation to identify the optimal (K, z) configuration for conformational clustering. This instruction includes a complete workflow for unsupervised clustering of protein conformations using pairwise Cα (alpha-carbon) distance matrices (or BB atoms in CG modeling) as the input representation for a Gaussian Mixture Variational Autoencoder (GMVAE). The workflow covers dataset preparation, model hyperparameter optimization, systematic grid search over the number of Gaussian mixture components (K) and latent-space dimensions (z), and latent-space evaluation to identify the optimal (K, z) configuration for conformational clustering.
- 
  
 In addition to GMVAE training and evaluation, the package provides utilities for generating cluster-wise contact maps, representative conformations, and summary statistics for structural analysis and visualization. In addition to GMVAE training and evaluation, the package provides utilities for generating cluster-wise contact maps, representative conformations, and summary statistics for structural analysis and visualization.
- 
-**Recommended Computing Workflow and Resources** 
  
 The workflow can be executed on either GPU or CPU systems. The following hardware configurations are recommended: The workflow can be executed on either GPU or CPU systems. The following hardware configurations are recommended:
Line 75: Line 72:
  
 </code> </code>
-//File Dependencies// 
-data_collection.py 
  
-Required python (main) packages +**Usage note: copy the scripts folder as your project name to begin.** 
-channels:  +  
-  * conda-forge +Note: If the servers where no graphical user interface (GUI) or display server is available,  add matplotlib.use("Agg") before importing pyplot to force Matplotlib to use a non-GUI backend, allowing you to generate and save plots without opening a visual window. 
-  * defaults +
-dependencies: +
-  * python=3.9 +
-  * numpy +
-  * mdtraj +
-  * pyemma+
  
-or using provided pyemma.yml to build conda environment.+<code> 
 +import matplotlib 
 +matplotlib.use("Agg")  
 +import matplotlib.pyplot as plt 
 +</code>
  
-=== Truncating Trajectories ===+ 
  
-For large systems, it is recommended to trim the trajectory to include only the heavy atoms of interest (e.g., CA).+**Input data preparation**
  
-Data file archive: save pdb in ca_dcd/ (e.g. ca_dcd/ca_dyn0.pdb)+Modify //run_datacoll.csh// or //run_datacoll_list.csh// to suit your input trajectory files. Detailed instructions for using this data collection script are provided below. 
  
-Example: 
 <code> <code>
-python data_collection.py --pdb /home/mhsieh/scratch/bst2/orf7a-cov-2_bst2_short/aa/c2_r1_cg2aa/openmm/ca_dcd/ca.pdb --traj_dirs "/home/mhsieh/scratch/bst2/orf7a-cov-2_bst2_short/aa/c2_r1_cg2aa/openmm/ca_dcd/" "/home/mhsieh/scratch/bst2/orf7a-cov-2_bst2_short/aa/c2_r2_cg2aa/openmm/ca_dcd/" --traj_prefix ca_dyn  --start 501 --end 550 –-dimer –-skip 4 –-output_prefix orf7a_bst2_short </code>+usage: data_collection_v4.py [-h] --pdb PDB 
 +                             [--traj_files TRAJ_FILES [TRAJ_FILES ...]] 
 +                             [--traj_dirs TRAJ_DIRS [TRAJ_DIRS ...]] 
 +                             [--traj_prefix TRAJ_PREFIX] 
 +                             [--traj_ext {dcd,trr,xtc}] [--start START] 
 +                             [--end END] [--single_chain] [--dimer] 
 +                             [--selected_atom SELECTED_ATOM] 
 +                             [--indices1 INDICES1 INDICES1] 
 +                             [--indices2 INDICES2 INDICES2] [--skip SKIP] 
 +                             [--skip_first_frame SKIP_FIRST_FRAME] 
 +                             [--output_prefix OUTPUT_PREFIX]
  
-=== Starting Data Collection Pipeline ===+</code>
  
-Length of indices1: 100 +<code> 
-Length of indices2: 121 +optional arguments: 
-Feature dimension: 12100 +  -h, --help            show this help message and exit 
-100%|████████████████████| 100/100 [00:14<00:00,  6.82it/s] +  --pdb PDB             Path to topology/structure file, usually .pdb or .gro. 
-100%|████████████████████| 100/100 [00:01<00:00, 97.15it/s] +  --traj_files TRAJ_FILES [TRAJ_FILES ...] 
-Final all_data length: 10000 +                        Explicit list of trajectory files, e.g. traj1.trr 
-Computed RMSD for 10000 frames. +                        traj2.trr. 
-Saved outputs: orf7a_bst2_short_data.npy and orf7a_bst2_short_rmsd.npy+  --traj_dirs TRAJ_DIRS [TRAJ_DIRS ...] 
 +                        List of directories containing trajectory files. 
 +  --traj_prefix TRAJ_PREFIX 
 +                        Prefix for trajectory files when using traj_dirs. 
 +  --traj_ext {dcd,trr,xtc} 
 +                        Trajectory extension when using traj_dirs. 
 +  --start START         Starting trajectory index. 
 +  --end END             Ending trajectory index. 
 +  --single_chain        Use the same atom indices for both chains. 
 +  --dimer               Use separate atom indices for two chains. 
 +  --selected_atom SELECTED_ATOM 
 +                        Selected atom name for analysis. 
 +  --indices1 INDICES1 INDICES1 
 +                        Range for chain 1 MDTraj atom indices, e.g. 0 99. 
 +                        Range for chain 2 MDTraj atom indices, e.g. 100 220. 
 +  --skip SKIP           Frame skip interval. 
 +  --skip_first_frame SKIP_FIRST_FRAME 
 +                        Number of initial frames to discard after loading all 
 +                        trajectories. 
 +  --output_prefix OUTPUT_PREFIX 
 +                        Prefix for output .npy files.
  
-=== Pipeline Completed ===+</code>
  
-Using tf1.15.4.yml to build conda environment (pyemma excluded). +**GMVAE Clustering Workflow**
-GMVAE clustering via Zaratan v100 GPU+
  
-Example of run.sh +The GMVAE workflow consists of three major stages:
-<code> +
-#!/bin/bash +
-#SBATCH --job-name=k6_z8_r1 +
-#SBATCH -n 8 +
-#SBATCH --partition=GPU +
-#SBATCH --gpus=v100:1 +
-#SBATCH --time=01:30:00+
  
-source ~/.bashrc +  - Hyperparameter optimization (excluding K and z)  
-conda init --all +  - Grid search over the number of clusters (K) and latent dimensions (z)  
-conda activate /home/mhsieh/scratch/tf1.15.4/ +  - Latent-space evaluation to identify the optimal (K, z) configuration 
-pip list | grep tensorflow +
-echo "python gmvae_pipeline.py --mode train \ +
-  --pdb /home/mhsieh/scratch/bst2/orf7a-cov-2_bst2_short/aa/c2_r1_cg2aa/openmm/ca_dcd/ca.pdb \ +
-  --dataset-npy orf7a_bst2_short_data.npy \ +
-  --rmsd-npy orf7a_bst2_short_rmsd.npy \ +
-  --mode_chain dimer \ +
-  --indices1 0 99 \ +
-  --indices2 100 220 \ +
-  --K 6 --z-dim 8 \ +
-  --epochs 100 --batch-size 500 \ +
-  --output-dir outputs"+
  
-python gmvae_pipeline.py --mode train \+**Quick start** 
 + 
 +Activate the TensorFlow environment before running the GMVAE pipeline. 
 + 
 +<code>conda activate ~/ scratch/tf1.15.4</code> 
 + 
 +The default training hyperparameters are:  
 +--temp 0.05 
 +--dropout 0.3   
 +--lr 0.001  
 +--epochs 300   
 +--batch-size 100   
 +--dense-n 64  
 +--filters 64 64 32    
 +--kernel-sizes 3 3 3   
 +--pool-sizes 2 2 1   
 +--num-layers 3 
 + 
 +The gmvae_pipeline_v2.py script provides a unified interface for GMVAE training, hyperparameter grid search, and evaluation of GMVAE for data clustering. The prepared scripts have been included. 
 + 
 +The program supports three operating modes: 
 +  * **train** – Train a single GMVAE model. (run_train.csh) 
 +  * **grid** – Perform a grid search over multiple values of K and z. (run_grid.csh or run_grid_manually.scr) 
 +  * **eval** – Evaluate a trained model and generate clustering results. (run_eval.csh) 
 + 
 +**Training Mode (train)** 
 + 
 +Train a single GMVAE model using a specified set of hyperparameters. 
 + 
 +**Example: Single Training** 
 +<code> 
 +python gmvae_pipeline_v2.py --mode train \
   --pdb /home/mhsieh/scratch/bst2/orf7a-cov-2_bst2_short/aa/c2_r1_cg2aa/openmm/ca_dcd/ca.pdb \   --pdb /home/mhsieh/scratch/bst2/orf7a-cov-2_bst2_short/aa/c2_r1_cg2aa/openmm/ca_dcd/ca.pdb \
   --dataset-npy orf7a_bst2_short_data.npy \   --dataset-npy orf7a_bst2_short_data.npy \
Line 152: Line 186:
 </code> </code>
  
-If process evaluation right after clustering, the add   --evaluate_after_train option.+**Example: Training with Automatic Evaluation**
 <code> <code>
-python gmvae_pipeline.py --mode train \+python gmvae_pipeline_v2.py --mode train \
   --pdb /home/mhsieh/scratch/bst2/orf7a-cov-2_bst2_short/aa/c2_r1_cg2aa/openmm/ca_dcd/ca.pdb \   --pdb /home/mhsieh/scratch/bst2/orf7a-cov-2_bst2_short/aa/c2_r1_cg2aa/openmm/ca_dcd/ca.pdb \
   --dataset-npy orf7a_bst2_short_data.npy \   --dataset-npy orf7a_bst2_short_data.npy \
Line 162: Line 196:
   --indices2 100 220 \   --indices2 100 220 \
   --K 6 --z-dim 4 \   --K 6 --z-dim 4 \
-  --epochs 10 --batch-size 500 \+  --epochs 100 --batch-size 500 \
   --output-dir outputs \   --output-dir outputs \
   --evaluate_after_train   --evaluate_after_train
 +
 </code> </code>
  
-If run evaluation separately, using the following script. +**Grid Search Mode (grid)**
-<code> +
-#!/bin/bash +
-#SBATCH --job-name=k11_z4_r1 +
-#SBATCH -N 1 +
-#SBATCH -n 64 +
-#SBATCH --partition=standard +
-#SBATCH --time=01:30:00+
  
 +Once a suitable set of training hyperparameters has been determined (excluding K and z), perform a systematic grid search to identify the optimal latent-space dimension and number of clusters.
  
-source ~/.bashrc +**Example** 
-conda init --all +<code> 
-conda activate /home/mhsieh/scratch/tf1.15.4/ +python gmvae_pipeline_v2.py --mode grid \
- +
-pip list | grep tensorflow +
- +
-echo "python gmvae_pipeline.py --mode eval \+
   --pdb /home/mhsieh/scratch/bst2/orf7a-cov-2_bst2_short/aa/c2_r1_cg2aa/openmm/ca_dcd/ca.pdb \   --pdb /home/mhsieh/scratch/bst2/orf7a-cov-2_bst2_short/aa/c2_r1_cg2aa/openmm/ca_dcd/ca.pdb \
   --dataset-npy orf7a_bst2_short_data.npy \   --dataset-npy orf7a_bst2_short_data.npy \
Line 190: Line 215:
   --indices1 0 99 \   --indices1 0 99 \
   --indices2 100 220 \   --indices2 100 220 \
-  --K 6 --z-dim 8 \ +  --grid-ks 3 4 5 6 7 8 \ 
-  --batch-size 1 \ +  --grid-zs 5 6 7 \ 
-  --output-dir outputs \ +  --epochs 100 --batch-size 500 \ 
-  --model-dir ./outputs/K6_z8 \ +  --output-dir grid_outputs 
-  --num-cpus 64"+</code>
  
-python gmvae_pipeline.py --mode eval \+**Evaluation Mode (eval)** 
 + 
 +Evaluation is performed after model training and is primarily CPU-based. The evaluation step generates latent-space statistics and clustering summaries for model selection. 
 + 
 +**Example** 
 +<code> 
 +python gmvae_pipeline_V2.py --mode eval \
   --pdb /home/mhsieh/scratch/bst2/orf7a-cov-2_bst2_short/aa/c2_r1_cg2aa/openmm/ca_dcd/ca.pdb \   --pdb /home/mhsieh/scratch/bst2/orf7a-cov-2_bst2_short/aa/c2_r1_cg2aa/openmm/ca_dcd/ca.pdb \
   --dataset-npy orf7a_bst2_short_data.npy \   --dataset-npy orf7a_bst2_short_data.npy \
Line 204: Line 235:
   --indices2 100 220 \   --indices2 100 220 \
   --K 6 --z-dim 8 \   --K 6 --z-dim 8 \
-  --batch-size 1 \ +  --batch-size 500 \
-  --output-dir outputs \+
   --model-dir ./outputs/K6_z8 \   --model-dir ./outputs/K6_z8 \
-  --num-cpus 64+  --num-cpus 64 > ./outputs/K6_z8/eval.log
 </code> </code>
 +**Output Directory Structure**
  
 +Training results are saved in the directory specified by --output-dir. The output directory contains model checkpoints, training logs, evaluation results, and summary statistics.
 +
 +**Output Files**
 +
 +**Training Outputs**
 +
 +The following files/folders are generated during training:
 +
 +  * **checkpoint/** Saved GMVAE model checkpoints.
 +  * **summary/**: Summary statistics and evaluation results (generated after evaluation).
 +  * **train_dic.pickle**: Training loss history recorded for each epoch.
 +  * **valid_dic.pickle**: Validation loss history recorded for each epoch.
 +  * **train.png**: Training loss versus epoch.
 +  * **validation.png**: Validation loss versus epoch.
 +
 +**Latent Space Visualization**
 +
 +To visualize the learned latent representation, the pipeline generates two-dimensional t-SNE embeddings.
 +  * **learned_labels.png**: Two-dimensional t-SNE projection of the latent space, colored by GMVAE cluster assignments.
 +  * **learned_rmsd.png**: Two-dimensional t-SNE projection of the latent space, colored by RMSD values.
 +  * **var2d_zsel.npy**: Two-dimensional t-SNE coordinates used to generate the visualization.
 + 
 +**Clustering Results**
 +  * **y_tot.npy**: Predicted cluster assignment for each input conformation.
 +  * **Z_tot.npy**: Learned latent-space representation for each conformation.
 +
 +**Evaluation Summary**
 +After running the evaluation step (--mode eval), additional summary files are generated in ./<model-dir>/summary/
 +including:
 +
 +  * **latent_mean_corr_matrix.csv** Correlation matrix of the latent variables.
 +  * **latent_mean_corr_matrix.png** Heatmap visualization of the latent-variable correlation matrix.
 +  * **latent_mean_pca_explained.png** Principal component explained variance of the latent space.
 +
 +**GMVAE Operation details**
 +
 +Detailed instructions for using this gmvae script are provided below.  
 +
 +<code>
 +usage: gmvae_pipeline_v2.py [-h] --mode {train,grid,eval} --pdb PDB
 +                            --dataset-npy DATASET_NPY --rmsd-npy RMSD_NPY
 +                            [--mode_chain {single_chain,dimer}]
 +                            [--indices1 INDICES1 INDICES1]
 +                            [--indices2 INDICES2 INDICES2]
 +                            [--batch-size BATCH_SIZE] [--lr LR]
 +                            [--epochs EPOCHS] [--num-layers NUM_LAYERS]
 +                            [--z-dim Z_DIM] [--temp TEMP] [--K K]
 +                            [--restore RESTORE] [--dense-n DENSE_N]
 +                            [--filters FILTERS [FILTERS ...]]
 +                            [--kernel-sizes KERNEL_SIZES [KERNEL_SIZES ...]]
 +                            [--pool-sizes POOL_SIZES [POOL_SIZES ...]]
 +                            [--valid-ratio VALID_RATIO] [--dropout DROPOUT]
 +                            [--r-nent R_NENT] [--r-label R_LABEL]
 +                            [--output-dir OUTPUT_DIR]
 +                            [--grid-ks [GRID_KS [GRID_KS ...]]]
 +                            [--grid-zs [GRID_ZS [GRID_ZS ...]]]
 +                            [--evaluate-after-train] [--num-cpus NUM_CPUS]
 +                            [--model-dir MODEL_DIR]
 +
 +GMVAE training & grid testing pipeline
 +
 +optional arguments:
 +  -h, --help            show this help message and exit
 +  --mode {train,grid,eval}
 +                        Mode to run: train | grid | eval
 +  --pdb PDB             Path to pdb file
 +  --dataset-npy DATASET_NPY
 +                        Path to dataset .npy file
 +  --rmsd-npy RMSD_NPY   Path to RMSD .npy file
 +  --mode_chain {single_chain,dimer}
 +                        Single chain or dimer analysis (affects indices_range2
 +                        selection)
 +  --indices1 INDICES1 INDICES1
 +                        Range for chain 1 indices, e.g., 0 99.
 +  --indices2 INDICES2 INDICES2
 +                        Range for chain 2 indices, e.g., 100 220.
 +  --batch-size BATCH_SIZE
 +  --lr LR
 +  --epochs EPOCHS
 +  --num-layers NUM_LAYERS
 +  --z-dim Z_DIM
 +  --temp TEMP
 +  --K K
 +  --restore RESTORE
 +  --dense-n DENSE_N
 +  --filters FILTERS [FILTERS ...]
 +  --kernel-sizes KERNEL_SIZES [KERNEL_SIZES ...]
 +  --pool-sizes POOL_SIZES [POOL_SIZES ...]
 +  --valid-ratio VALID_RATIO
 +  --dropout DROPOUT
 +  --r-nent R_NENT
 +  --r-label R_LABEL
 +  --output-dir OUTPUT_DIR
 +                        Base output directory
 +  --grid-ks [GRID_KS [GRID_KS ...]]
 +                        List of K values for grid
 +  --grid-zs [GRID_ZS [GRID_ZS ...]]
 +                        List of z-dim values for grid
 +  --evaluate-after-train
 +                        Run evaluation after each training run (CPU)
 +  --num-cpus NUM_CPUS   Number of CPU cores to pass to evaluation
 +  --model-dir MODEL_DIR
 +                        Model output dir to evaluate (used in eval mode)
 + 
 +</code>
 +**1. Hyperparameter Optimization (excluding K and z)**
 +
 +Optimize the network architecture and training parameters before performing the grid search.
 +
 +**Recommended Network Architecture**
 +
 +As a general guideline:
 +
 +  * 2 convolutional layers are recommended for small single-chain proteins (e.g., Trp-cage, BBA, and proteins with fewer than approximately 50 residues). 
 +  * 3 convolutional layers are recommended for larger proteins or protein dimers. 
 +
 +**1.1. Published Hyperparameter Sets**
 +<code>
 +Systems       # of     # of   Latent   # of     Batch           Kernel   Learning   # of        Pooling
 +             layers   neurons  dim.  clusters size Temp. size rate   filters sizes     Ref
 +Trp-cage 2 64 5 8 5000 0.1 [3,3]   0.001   [64,64] [1,1]    J. Chem. Phys. (2021) 155, 194108  doi: 10.1063/5.0069708
 +BBA         2 64 6 9 5000 0.1 [3,3]   0.001   [64,64] [2,2]    J. Chem. Phys. (2021) 155, 194108  doi: 10.1063/5.0069708
 +villin         3 64 5 6 2500 0.05 [3,3,3]   0.001   [64,64,32] [2,2,1]    J. Chem. Phys. (2021) 155, 194108  doi: 10.1063/5.0069708
 +BST-2/ORF7a 3 64 5 9 2500 0.05 [3,3,3]   0.001   [64,64,32] [2,2,1]    BBA Biomembranes (2023) 1865(6), 184174. doi: 10.1016/j.bbamem.2023.184174
 +BST-2/ORF7a 3 64 5 13 2500 0.05 [3,3,3]   0.001   [64,64,32] [2,2,1]    BBA Biomembranes (2023) 1865(6), 184174. doi: 10.1016/j.bbamem.2023.184174 
 +ORF7b dimer 3 64 6 10 2500 0.05 [3,3,3]   0.001   [64,64,32] [2,2,1]    J. Phys. Chem. B (2024) 128(1), 150-162. doi: 10.1021/acs.jpcb.3c07105
 +ORF7b dimer 3 64 6 8 2500 0.05 [3,3,3]   0.001   [64,64,32] [2,2,1]    J. Phys. Chem. B (2024) 128(1), 150-162. doi: 10.1021/acs.jpcb.3c07105
 +BST-2/ORF7a 3 64 4 11 1000 0.05 [3,3,3]   0.001   [64,64,32] [2,2,1]    J. Phys. Chem. B (2026) 130(5), 1553–1567. doi: 10.1021/acs.jpcb.5c07529
 +
 +</code>
 +**1.2. Batch Size Recommendations**
 +
 +GPU memory limits the maximum batch size during training. For single-channel, three-layer GMVAE models trained on an NVIDIA V100 GPU (32 GB), the recommended batch sizes are summarized below.:
 +
 +<code>
 +chain A chain A    matrix    batch size
 +25 25   [25x25] 5000
 +50 50   [50x50] 2500
 +100 100 [100x100] 1000
 +150 150 [150x150] 200
 +
 +</code>
 +
 +**1.3. Dataset Size Considerations**
 +
 +The current implementation processes only complete batches during training and evaluation. If the total number of samples is not divisible by the batch size, the remaining samples in the final incomplete batch are ignored.
 +
 +To maximize data utilization, select a batch size that minimizes the number of discarded samples.
 +<code>
 +# of     start   end        data batch # of data 
 +input data  frame   frame    included    size   batch   leftovers
 +15000      0     14999      15000 1000 15 0
 +5739      0     5699       5700 100 57 39
 +5739      0     5631       5632 128 44 107
 +5739      0     5695       5696 64 89 43
 +</code>
 +
 +**My suggestion is to adjust the number of input data points to an appropriate batch size to ensure 0 leftover.**
 +
 +**Note: The cluster label (in neither y tot.npy nor z tot.npy) would not be assigned to the leftovers. If the entire dataset is not used, please specify --start and --end in eval mode. **
 +
 +**1.4. Recommended Optimization Strategy**
 +
 +Optimize the hyperparameters in the following order:
 +  - Batch size 
 +  - Number of convolutional layers 
 +  - Remaining network hyperparameters 
 +  - Number of clusters (K) 
 +Based on our experience, z ≥ 4 generally provides sufficient latent-space capacity, with z = 6 serving as a good starting point for many protein systems. An initial K value between 6 and 13 is recommended, consistent with previously published GMVAE studies.
 +
 +After selecting an initial configuration, perform several single-model training runs to refine the hyperparameters before proceeding to the grid search.
 +
 +**2. Grid Search**
 +After optimizing the training hyperparameters (excluding K and z), perform a systematic grid search over candidate values of K and z. Compare the resulting models using reconstruction loss, clustering confidence, latent-space independence, and structural interpretability to identify the optimal (K, z) configuration.
 +
 +**3. Evaluate the Latent Space and Select an Optimal (K, z)**
 +After completing the grid search, evaluate each model using multiple complementary metrics. The goal is to identify the lowest-complexity model that accurately represents the conformational landscape while producing confident, interpretable, and non-redundant clusters.
 +Optimal Model Selection Criteria
 +
 +**3.1 An optimal GMVAE model should satisfy the following criteria:**
 +  * Low reconstruction loss: Accurately reconstructs the input distance matrices while preserving conformational variability. 
 +  * Low cluster assignment entropy: Produces confident, well-separated Gaussian mixture assignments. 
 +  * Low latent multicollinearity (VIF): Latent variables are approximately independent and encode complementary structural information. 
 +  * Meaningful PCA contributions: Each latent dimension contributes measurable variance, with minimal redundancy. 
 +  * Balanced cluster populations: Avoids excessive empty or very small clusters while capturing dominant conformational states. 
 +  * Structural interpretability: Clusters correspond to physically meaningful and biologically relevant conformations. 
 +**3.2 Practical Recommendations**
 +  * Choose the smallest latent dimension (z) after reconstruction loss and cluster entropy have converged. 
 +  * Select the number of clusters (K) that balances structural diversity with sufficient samples per cluster. 
 +  * Verify that latent variables have low VIF and weak inter-latent correlations. 
 +  * Use PCA to confirm that each latent dimension contributes unique information. 
 +  * Inspect representative structures from each cluster to ensure biological interpretability. 
 +  * Prefer the smallest (K, z) configuration that meets all evaluation criteria, balancing reconstruction quality, clustering confidence, latent-space independence, cluster population, and structural interpretability. [the smallest (K, z) configuration]
 +
 +**Cluster Visualization and Structural Interpretation**
 +
 +After selecting an optimal (K, z) configuration, representative conformations from each cluster can be extracted for structural analysis and visualization. The contact_heatmap_top_structures.py script generates cluster-wise contact maps and identifies representative structures based on their cluster assignment probabilities.
 +<code>
 +usage: contact_heatmap_top_structures.py [-h] --mode {dataset,trajectory}
 +                                         --mode_chain {single_chain,dimer}
 +                                         [--A_dataset A_DATASET]
 +                                         [--B_dataset B_DATASET]
 +                                         [--AB_dataset AB_DATASET]
 +                                         [--A_size A_SIZE] [--B_size B_SIZE]
 +                                         [--pdb PDB]
 +                                         [--traj_files TRAJ_FILES [TRAJ_FILES ...]]
 +                                         [--traj_dirs TRAJ_DIRS [TRAJ_DIRS ...]]
 +                                         [--traj_prefix TRAJ_PREFIX]
 +                                         [--traj_ext {dcd,trr,xtc}]
 +                                         [--start START] [--end END]
 +                                         [--skip SKIP]
 +                                         [--skip_first_frame SKIP_FIRST_FRAME]
 +                                         [--skip_after_frame SKIP_AFTER_FRAME]
 +                                         [--selected_atom SELECTED_ATOM]
 +                                         [--A_indices A_INDICES A_INDICES]
 +                                         [--B_indices B_INDICES B_INDICES]
 +                                         --cluster_folder CLUSTER_FOLDER
 +                                         [--threshold THRESHOLD]
 +                                         [--cutoff CUTOFF]
 +                                         [--output_dir OUTPUT_DIR]
 +                                         [--save_generated_dataset]
 +                                         [--generate_top_structures]
 +                                         [--top_n TOP_N]
 +                                         [--mean_iterations MEAN_ITERATIONS]
 +                                         [--representative_output_dir REPRESENTATIVE_OUTPUT_DIR]
 +
 +</code>
 +
 +The script performs the following tasks:
 +  * Reads one or more molecular dynamics trajectories; it has to follow exactly the data_collection setting. 
 +  * Loads GMVAE clustering results (y_tot.npy) from the specified cluster_folder. 
 +  * Assigns each trajectory frame to its corresponding GMVAE cluster. 
 +  * Generates residue-residue contact probability heatmaps for each cluster. 
 +  * Identifies the top representative conformations (highest cluster assignment probability) from each cluster. 
 +  * Writes representative structures to individual trajectory files for downstream visualization and analysis.
 +
 +**Example script**
 +
 +<code>
 +#!/bin/bash
 +#Conda
 +source ~/.bashrc
 +conda init --all
 +#conda activate /scratch/zt1/project/energybio/user/dvan1/plexinA1/gmcave_tf1_update_copy/gmcave
 +conda activate /your/pyemma_env/
 +#Directories
 +dir_pdb="/scratch/zt1/project/energybio/user/mhsieh/bst2/orf7a-cov-2_bst2_short/aa/c2_r1_cg2aa/openmm/ca_dcd/ca.pdb"
 +
 +dir1="/scratch/zt1/project/energybio/user/mhsieh/bst2/orf7a-cov-2_bst2_short/aa/c2_r1_cg2aa/openmm/ca_dcd/"
 +dir2="/scratch/zt1/project/energybio/user/mhsieh/bst2/orf7a-cov-2_bst2_short/aa/c2_r2_cg2aa/openmm/ca_dcd/"
 +dir3="/scratch/zt1/project/energybio/user/mhsieh/bst2/orf7a-cov-2_bst2_short/aa/c2_r3_cg2aa/openmm/ca_dcd/"
 +dir4="/scratch/zt1/project/energybio/user/mhsieh/bst2/orf7a-cov-2_bst2_short/aa/c4_r1_cg2aa/openmm/ca_dcd/"
 +dir5="/scratch/zt1/project/energybio/user/mhsieh/bst2/orf7a-cov-2_bst2_short/aa/c4_r2_cg2aa/openmm/ca_dcd/"
 +dir6="/scratch/zt1/project/energybio/user/mhsieh/bst2/orf7a-cov-2_bst2_short/aa/c4_r3_cg2aa/openmm/ca_dcd/"
 +out_prefix="orf7a_bst2_short"
 +
 +#Files
 +file_traj1="ca_dyn301.dcd"
 +file_traj2="ca_dyn302.dcd"
 +file_traj3="ca_dyn303.dcd"
 +file_traj4="ca_dyn304.dcd"
 +file_traj5="ca_dyn305.dcd"
 +
 +#Commands
 +python3 ../src/utils/data_collection_v4.py \
 +--pdb ${dir_pdb} \
 +--dimer --selected_atom CA --indices1 0 99 --indices2 100 220 --output_prefix ${out_prefix} \
 +--traj_files ${dir1}${file_traj1} ${dir1}${file_traj2} ${dir1}${file_traj3} ${dir1}${file_traj4} ${dir1}${file_traj5} \
 +             ${dir2}${file_traj1} ${dir2}${file_traj2} ${dir2}${file_traj3} ${dir2}${file_traj4} ${dir2}${file_traj5} \
 +             ${dir3}${file_traj1} ${dir3}${file_traj2} ${dir3}${file_traj3} ${dir3}${file_traj4} ${dir3}${file_traj5} \
 +             ${dir4}${file_traj1} ${dir4}${file_traj2} ${dir4}${file_traj3} ${dir4}${file_traj4} ${dir4}${file_traj5} \
 +             ${dir5}${file_traj1} ${dir5}${file_traj2} ${dir5}${file_traj3} ${dir5}${file_traj4} ${dir5}${file_traj5} \
 +             ${dir6}${file_traj1} ${dir6}${file_traj2} ${dir6}${file_traj3} ${dir6}${file_traj4} ${dir6}${file_traj5} 
 +</code>
  
gmvae_update.1785656051.txt.gz · Last modified: 2026/08/02 03:34 by edit