Writing /users/jbklauda/pub/wiki/data/cache/3/36a3922e6748748aba38e43b093b0858.i failed
Unable to save cache file. Hint: disk full; file permissions; safe_mode setting.
Writing /users/jbklauda/pub/wiki/data/cache/3/36a3922e6748748aba38e43b093b0858.metadata failed
Writing /users/jbklauda/pub/wiki/data/cache/6/6193a558dc90ed4c190004bf67ecff70.captchaip failed
gmvae_update
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| gmvae_update [2026/08/02 05:02] – edit | gmvae_update [2026/08/03 12:43] (current) – edit | ||
|---|---|---|---|
| Line 6: | Line 6: | ||
| This is based off Dr. Ghorbani' | This is based off Dr. Ghorbani' | ||
| - | Tar file for all code: {{ :gmcave_tf1_update_copy.tgz |}} | + | Tar file for all code: {{ :gmvae_tf1_v2.tgz |}} |
| - | The detailed | + | The detailed |
| This instruction includes a complete workflow for unsupervised clustering of protein conformations using pairwise Cα (alpha-carbon) distance matrices (or BB atoms in CG modeling) as the input representation for a Gaussian Mixture Variational Autoencoder (GMVAE). The workflow covers dataset preparation, | This instruction includes a complete workflow for unsupervised clustering of protein conformations using pairwise Cα (alpha-carbon) distance matrices (or BB atoms in CG modeling) as the input representation for a Gaussian Mixture Variational Autoencoder (GMVAE). The workflow covers dataset preparation, | ||
| - | |||
| In addition to GMVAE training and evaluation, the package provides utilities for generating cluster-wise contact maps, representative conformations, | In addition to GMVAE training and evaluation, the package provides utilities for generating cluster-wise contact maps, representative conformations, | ||
| - | |||
| - | **Recommended Computing Workflow and Resources** | ||
| The workflow can be executed on either GPU or CPU systems. The following hardware configurations are recommended: | The workflow can be executed on either GPU or CPU systems. The following hardware configurations are recommended: | ||
| Line 76: | Line 73: | ||
| </ | </ | ||
| - | **Usage note: copy the scripts folder as your project name to begin.** | + | **Usage note: copy the scripts folder as your project name to begin.** |
| + | |||
| + | Note: If the servers where no graphical user interface (GUI) or display server is available, | ||
| + | |||
| + | < | ||
| + | import matplotlib | ||
| + | matplotlib.use(" | ||
| + | import matplotlib.pyplot as plt | ||
| + | </ | ||
| + | |||
| + | |||
| **Input data preparation** | **Input data preparation** | ||
| Line 154: | Line 161: | ||
| --num-layers 3 | --num-layers 3 | ||
| - | The gmvae_pipeline_v2.py script provides a unified interface for GMVAE training, hyperparameter grid search, and evaluation of GMVAE for data clustering. | + | The gmvae_pipeline_v2.py script provides a unified interface for GMVAE training, hyperparameter grid search, and evaluation of GMVAE for data clustering. The prepared scripts have been included. |
| The program supports three operating modes: | The program supports three operating modes: | ||
| - | * **train** – Train a single GMVAE model. | + | * **train** – Train a single GMVAE model. |
| - | * **grid** – Perform a grid search over multiple values of K and z. | + | * **grid** – Perform a grid search over multiple values of K and z. (run_grid.csh or run_grid_manually.scr) |
| - | * **eval** – Evaluate a trained model and generate clustering results. | + | * **eval** – Evaluate a trained model and generate clustering results. |
| **Training Mode (train)** | **Training Mode (train)** | ||
| Line 198: | Line 205: | ||
| Once a suitable set of training hyperparameters has been determined (excluding K and z), perform a systematic grid search to identify the optimal latent-space dimension and number of clusters. | Once a suitable set of training hyperparameters has been determined (excluding K and z), perform a systematic grid search to identify the optimal latent-space dimension and number of clusters. | ||
| - | < | + | |
| **Example** | **Example** | ||
| + | < | ||
| python gmvae_pipeline_v2.py --mode grid \ | python gmvae_pipeline_v2.py --mode grid \ | ||
| --pdb / | --pdb / | ||
| Line 235: | Line 243: | ||
| Training results are saved in the directory specified by --output-dir. The output directory contains model checkpoints, | Training results are saved in the directory specified by --output-dir. The output directory contains model checkpoints, | ||
| + | **Output Files** | ||
| - | + | **Training Outputs** | |
| - | //File Dependencies// | + | |
| - | data_collection.py | + | |
| - | Required python (main) packages | + | The following files/ |
| - | channels: | + | |
| - | * conda-forge | + | |
| - | * defaults | + | |
| - | dependencies: | + | |
| - | * python=3.9 | + | |
| - | * numpy | + | |
| - | * mdtraj | + | |
| - | * pyemma | + | |
| - | or using provided pyemma.yml to build conda environment. | + | * **checkpoint/ |
| + | * **summary/ | ||
| + | * **train_dic.pickle**: | ||
| + | * **valid_dic.pickle**: | ||
| + | * **train.png**: | ||
| + | * **validation.png**: | ||
| - | === Truncating Trajectories === | + | **Latent Space Visualization** |
| - | For large systems, it is recommended to trim the trajectory to include only the heavy atoms of interest (e.g., CA). | + | To visualize the learned latent representation, the pipeline generates two-dimensional t-SNE embeddings. |
| + | * **learned_labels.png**: | ||
| + | * **learned_rmsd.png**: | ||
| + | * **var2d_zsel.npy**: | ||
| + | |||
| + | **Clustering Results** | ||
| + | * **y_tot.npy**: | ||
| + | * **Z_tot.npy**: | ||
| - | Data file archive: save pdb in ca_dcd/ (e.g. ca_dcd/ | + | **Evaluation Summary** |
| + | After running the evaluation step (--mode eval), additional summary files are generated | ||
| + | including: | ||
| + | |||
| + | * **latent_mean_corr_matrix.csv** Correlation matrix of the latent variables. | ||
| + | * **latent_mean_corr_matrix.png** Heatmap visualization of the latent-variable correlation matrix. | ||
| + | * **latent_mean_pca_explained.png** Principal component explained variance of the latent space. | ||
| + | |||
| + | **GMVAE Operation details** | ||
| + | |||
| + | Detailed instructions for using this gmvae script are provided below. | ||
| - | Example: | ||
| < | < | ||
| - | python data_collection.py --pdb / | + | usage: gmvae_pipeline_v2.py [-h] --mode {train, |
| + | | ||
| + | [--mode_chain {single_chain, | ||
| + | [--indices1 INDICES1 INDICES1] | ||
| + | [--indices2 INDICES2 INDICES2] | ||
| + | [--batch-size BATCH_SIZE] [--lr LR] | ||
| + | [--epochs EPOCHS] [--num-layers NUM_LAYERS] | ||
| + | [--z-dim Z_DIM] [--temp TEMP] [--K K] | ||
| + | [--restore RESTORE] [--dense-n DENSE_N] | ||
| + | [--filters FILTERS [FILTERS | ||
| + | [--kernel-sizes KERNEL_SIZES [KERNEL_SIZES ...]] | ||
| + | [--pool-sizes POOL_SIZES [POOL_SIZES ...]] | ||
| + | [--valid-ratio VALID_RATIO] [--dropout DROPOUT] | ||
| + | [--r-nent R_NENT] [--r-label R_LABEL] | ||
| + | [--output-dir OUTPUT_DIR] | ||
| + | [--grid-ks [GRID_KS [GRID_KS ...]]] | ||
| + | [--grid-zs [GRID_ZS [GRID_ZS ...]]] | ||
| + | [--evaluate-after-train] [--num-cpus NUM_CPUS] | ||
| + | [--model-dir MODEL_DIR] | ||
| - | === Starting Data Collection Pipeline === | + | GMVAE training & grid testing pipeline |
| - | Length of indices1: 100 | + | optional arguments: |
| - | Length of indices2: 121 | + | -h, --help |
| - | Feature dimension: 12100 | + | |
| - | 100%|████████████████████| 100/100 [00: | + | Mode to run: train | grid | eval |
| - | 100%|████████████████████| 100/ | + | --pdb PDB Path to pdb file |
| - | Final all_data length: 10000 | + | --dataset-npy DATASET_NPY |
| - | Computed RMSD for 10000 frames. | + | Path to dataset .npy file |
| - | Saved outputs: orf7a_bst2_short_data.npy and orf7a_bst2_short_rmsd.npy | + | --rmsd-npy RMSD_NPY |
| + | --mode_chain {single_chain, | ||
| + | Single chain or dimer analysis (affects indices_range2 | ||
| + | selection) | ||
| + | --indices1 INDICES1 INDICES1 | ||
| + | Range for chain 1 indices, e.g., 0 99. | ||
| + | --indices2 INDICES2 INDICES2 | ||
| + | Range for chain 2 indices, e.g., 100 220. | ||
| + | --batch-size BATCH_SIZE | ||
| + | --lr LR | ||
| + | --epochs EPOCHS | ||
| + | --num-layers NUM_LAYERS | ||
| + | --z-dim Z_DIM | ||
| + | --temp TEMP | ||
| + | --K K | ||
| + | --restore RESTORE | ||
| + | --dense-n DENSE_N | ||
| + | --filters FILTERS | ||
| + | --kernel-sizes KERNEL_SIZES [KERNEL_SIZES ...] | ||
| + | | ||
| + | | ||
| + | | ||
| + | --r-nent R_NENT | ||
| + | --r-label R_LABEL | ||
| + | --output-dir OUTPUT_DIR | ||
| + | Base output directory | ||
| + | --grid-ks [GRID_KS [GRID_KS ...]] | ||
| + | List of K values | ||
| + | --grid-zs [GRID_ZS [GRID_ZS | ||
| + | List of z-dim values for grid | ||
| + | --evaluate-after-train | ||
| + | Run evaluation after each training run (CPU) | ||
| + | --num-cpus NUM_CPUS | ||
| + | --model-dir MODEL_DIR | ||
| + | Model output dir to evaluate (used in eval mode) | ||
| + | |||
| + | </ | ||
| + | **1. Hyperparameter Optimization (excluding K and z)** | ||
| - | === Pipeline Completed === | + | Optimize the network architecture and training parameters before performing the grid search. |
| - | Using tf1.15.4.yml to build conda environment (pyemma excluded). | + | **Recommended Network Architecture** |
| - | GMVAE clustering via Zaratan v100 GPU | + | |
| - | Example of run.sh | + | As a general guideline: |
| + | |||
| + | * 2 convolutional layers are recommended for small single-chain proteins (e.g., Trp-cage, BBA, and proteins with fewer than approximately 50 residues). | ||
| + | * 3 convolutional layers are recommended for larger proteins or protein dimers. | ||
| + | |||
| + | **1.1. Published Hyperparameter Sets** | ||
| < | < | ||
| - | #!/bin/bash | + | Systems |
| - | #SBATCH --job-name=k6_z8_r1 | + | |
| - | # | + | Trp-cage 2 64 5 8 5000 0.1 [3, |
| - | # | + | BBA 2 64 6 9 5000 0.1 [3, |
| - | # | + | villin |
| - | # | + | BST-2/ |
| + | BST-2/ | ||
| + | ORF7b dimer 3 64 6 10 2500 0.05 [3, | ||
| + | ORF7b dimer 3 64 6 8 2500 0.05 [3, | ||
| + | BST-2/ | ||
| - | source ~/.bashrc | + | </code> |
| - | conda init --all | + | **1.2. Batch Size Recommendations** |
| - | conda activate / | + | |
| - | pip list | grep tensorflow | + | GPU memory limits the maximum batch size during training. For single-channel, three-layer GMVAE models trained on an NVIDIA V100 GPU (32 GB), the recommended batch sizes are summarized below.: |
| - | echo " | + | |
| - | --pdb / | + | < |
| - | | + | chain A chain A |
| - | | + | 25 25 |
| - | | + | 50 50 |
| - | | + | 100 100 [100x100] 1000 |
| - | | + | 150 150 [150x150] 200 |
| - | --K 6 --z-dim 8 \ | + | |
| - | | + | |
| - | --output-dir outputs" | + | |
| - | python gmvae_pipeline.py --mode train \ | ||
| - | --pdb / | ||
| - | --dataset-npy orf7a_bst2_short_data.npy \ | ||
| - | --rmsd-npy orf7a_bst2_short_rmsd.npy \ | ||
| - | --mode_chain dimer \ | ||
| - | --indices1 0 99 \ | ||
| - | --indices2 100 220 \ | ||
| - | --K 6 --z-dim 8 \ | ||
| - | --epochs 100 --batch-size 500 \ | ||
| - | --output-dir outputs | ||
| </ | </ | ||
| - | If process | + | **1.3. Dataset Size Considerations** |
| + | |||
| + | The current implementation processes only complete batches during training and evaluation. If the total number of samples is not divisible by the batch size, the remaining samples in the final incomplete batch are ignored. | ||
| + | |||
| + | To maximize data utilization, select a batch size that minimizes | ||
| < | < | ||
| - | python gmvae_pipeline.py --mode train \ | + | # of start |
| - | | + | input data frame |
| - | | + | 15000 |
| - | --rmsd-npy orf7a_bst2_short_rmsd.npy \ | + | 5739 |
| - | | + | 5739 0 5631 |
| - | --indices1 | + | 5739 |
| - | --indices2 100 220 \ | + | |
| - | --K 6 --z-dim 4 \ | + | |
| - | | + | |
| - | --output-dir outputs \ | + | |
| - | --evaluate_after_train | + | |
| </ | </ | ||
| - | If run evaluation separately, using the following script. | + | **My suggestion is to adjust the number of input data points to an appropriate batch size to ensure 0 leftover.** |
| + | |||
| + | **Note: The cluster label (in neither y tot.npy nor z tot.npy) would not be assigned to the leftovers. | ||
| + | |||
| + | **1.4. Recommended Optimization Strategy** | ||
| + | |||
| + | Optimize the hyperparameters in the following order: | ||
| + | - Batch size | ||
| + | - Number of convolutional layers | ||
| + | - Remaining network hyperparameters | ||
| + | - Number of clusters (K) | ||
| + | Based on our experience, z ≥ 4 generally provides sufficient latent-space capacity, with z = 6 serving as a good starting point for many protein systems. An initial K value between 6 and 13 is recommended, | ||
| + | |||
| + | After selecting an initial configuration, | ||
| + | |||
| + | **2. Grid Search** | ||
| + | After optimizing the training hyperparameters (excluding K and z), perform a systematic grid search over candidate values of K and z. Compare the resulting models | ||
| + | |||
| + | **3. Evaluate the Latent Space and Select an Optimal (K, z)** | ||
| + | After completing the grid search, evaluate each model using multiple complementary metrics. The goal is to identify the lowest-complexity model that accurately represents the conformational landscape while producing confident, interpretable, | ||
| + | Optimal Model Selection Criteria | ||
| + | |||
| + | **3.1 An optimal GMVAE model should satisfy | ||
| + | * Low reconstruction loss: Accurately reconstructs the input distance matrices while preserving conformational variability. | ||
| + | * Low cluster assignment entropy: Produces confident, well-separated Gaussian mixture assignments. | ||
| + | * Low latent multicollinearity (VIF): Latent variables are approximately independent and encode complementary structural information. | ||
| + | * Meaningful PCA contributions: | ||
| + | * Balanced cluster populations: | ||
| + | * Structural interpretability: | ||
| + | **3.2 Practical Recommendations** | ||
| + | * Choose the smallest latent dimension (z) after reconstruction loss and cluster entropy have converged. | ||
| + | * Select the number of clusters (K) that balances structural diversity with sufficient samples per cluster. | ||
| + | * Verify that latent variables have low VIF and weak inter-latent correlations. | ||
| + | * Use PCA to confirm that each latent dimension contributes unique information. | ||
| + | * Inspect representative structures from each cluster to ensure biological interpretability. | ||
| + | * Prefer the smallest (K, z) configuration that meets all evaluation criteria, balancing reconstruction quality, clustering confidence, latent-space independence, | ||
| + | |||
| + | **Cluster Visualization and Structural Interpretation** | ||
| + | |||
| + | After selecting an optimal (K, z) configuration, | ||
| < | < | ||
| - | #!/bin/bash | + | usage: contact_heatmap_top_structures.py [-h] --mode {dataset, |
| - | # | + | |
| - | # | + | [--A_dataset A_DATASET] |
| - | # | + | [--B_dataset B_DATASET] |
| - | # | + | [--AB_dataset AB_DATASET] |
| - | # | + | [--A_size A_SIZE] [--B_size B_SIZE] |
| + | [--pdb PDB] | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | |||
| + | </ | ||
| + | |||
| + | The script performs the following tasks: | ||
| + | * Reads one or more molecular dynamics trajectories; | ||
| + | * Loads GMVAE clustering results (y_tot.npy) from the specified cluster_folder. | ||
| + | * Assigns each trajectory frame to its corresponding GMVAE cluster. | ||
| + | * Generates residue-residue contact probability heatmaps for each cluster. | ||
| + | * Identifies the top representative conformations (highest cluster assignment probability) from each cluster. | ||
| + | * Writes representative structures to individual trajectory files for downstream visualization and analysis. | ||
| + | **Example script** | ||
| + | < | ||
| + | #!/bin/bash | ||
| + | #Conda | ||
| source ~/.bashrc | source ~/.bashrc | ||
| conda init --all | conda init --all | ||
| - | conda activate /home/mhsieh/scratch/tf1.15.4/ | + | #conda activate /scratch/zt1/ |
| + | conda activate / | ||
| + | # | ||
| + | dir_pdb=" | ||
| - | pip list | grep tensorflow | + | dir1="/ |
| + | dir2="/ | ||
| + | dir3="/ | ||
| + | dir4="/ | ||
| + | dir5="/ | ||
| + | dir6="/ | ||
| + | out_prefix=" | ||
| - | echo "python gmvae_pipeline.py --mode eval \ | + | #Files |
| - | --pdb / | + | file_traj1="ca_dyn301.dcd" |
| - | | + | file_traj2=" |
| - | | + | file_traj3=" |
| - | | + | file_traj4=" |
| - | --indices1 0 99 \ | + | file_traj5=" |
| - | --indices2 100 220 \ | + | |
| - | --K 6 --z-dim 8 \ | + | |
| - | --batch-size 1 \ | + | |
| - | --output-dir outputs \ | + | |
| - | --model-dir | + | |
| - | --num-cpus 64" | + | |
| - | python gmvae_pipeline.py --mode eval \ | + | #Commands |
| - | --pdb /home/mhsieh/scratch/ | + | python3 ../src/utils/data_collection_v4.py \ |
| - | --dataset-npy orf7a_bst2_short_data.npy | + | --pdb ${dir_pdb} |
| - | --rmsd-npy orf7a_bst2_short_rmsd.npy \ | + | --dimer --selected_atom CA --indices1 0 99 --indices2 100 220 --output_prefix ${out_prefix} |
| - | | + | --traj_files ${dir1}${file_traj1} ${dir1}${file_traj2} ${dir1}${file_traj3} ${dir1}${file_traj4} ${dir1}${file_traj5} |
| - | | + | ${dir2}${file_traj1} ${dir2}${file_traj2} ${dir2}${file_traj3} ${dir2}${file_traj4} ${dir2}${file_traj5} |
| - | | + | ${dir3}${file_traj1} ${dir3}${file_traj2} ${dir3}${file_traj3} ${dir3}${file_traj4} ${dir3}${file_traj5} |
| - | --K 6 --z-dim 8 \ | + | ${dir4}${file_traj1} ${dir4}${file_traj2} ${dir4}${file_traj3} ${dir4}${file_traj4} ${dir4}${file_traj5} \ |
| - | --batch-size 1 \ | + | |
| - | | + | |
| - | | + | |
| - | | + | |
| </ | </ | ||
| - | |||
gmvae_update.1785661321.txt.gz · Last modified: 2026/08/02 05:02 by edit