Writing /users/jbklauda/pub/wiki/data/cache/3/36a3922e6748748aba38e43b093b0858.i failed
Unable to save cache file. Hint: disk full; file permissions; safe_mode setting.
Writing /users/jbklauda/pub/wiki/data/cache/3/36a3922e6748748aba38e43b093b0858.metadata failed
Writing /users/jbklauda/pub/wiki/data/cache/6/6193a558dc90ed4c190004bf67ecff70.captchaip failed
gmvae_update
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| gmvae_update [2026/08/02 05:58] – edit | gmvae_update [2026/08/03 12:43] (current) – edit | ||
|---|---|---|---|
| Line 6: | Line 6: | ||
| This is based off Dr. Ghorbani' | This is based off Dr. Ghorbani' | ||
| - | Tar file for all code: {{ :gmcave_tf1_update_copy.tgz |}} | + | Tar file for all code: {{ :gmvae_tf1_v2.tgz |}} |
| - | The detailed | + | The detailed |
| This instruction includes a complete workflow for unsupervised clustering of protein conformations using pairwise Cα (alpha-carbon) distance matrices (or BB atoms in CG modeling) as the input representation for a Gaussian Mixture Variational Autoencoder (GMVAE). The workflow covers dataset preparation, | This instruction includes a complete workflow for unsupervised clustering of protein conformations using pairwise Cα (alpha-carbon) distance matrices (or BB atoms in CG modeling) as the input representation for a Gaussian Mixture Variational Autoencoder (GMVAE). The workflow covers dataset preparation, | ||
| - | |||
| In addition to GMVAE training and evaluation, the package provides utilities for generating cluster-wise contact maps, representative conformations, | In addition to GMVAE training and evaluation, the package provides utilities for generating cluster-wise contact maps, representative conformations, | ||
| - | |||
| - | **Recommended Computing Workflow and Resources** | ||
| The workflow can be executed on either GPU or CPU systems. The following hardware configurations are recommended: | The workflow can be executed on either GPU or CPU systems. The following hardware configurations are recommended: | ||
| Line 76: | Line 73: | ||
| </ | </ | ||
| - | **Usage note: copy the scripts folder as your project name to begin.** | + | **Usage note: copy the scripts folder as your project name to begin.** |
| + | |||
| + | Note: If the servers where no graphical user interface (GUI) or display server is available, | ||
| + | |||
| + | < | ||
| + | import matplotlib | ||
| + | matplotlib.use(" | ||
| + | import matplotlib.pyplot as plt | ||
| + | </ | ||
| + | |||
| + | |||
| **Input data preparation** | **Input data preparation** | ||
| Line 154: | Line 161: | ||
| --num-layers 3 | --num-layers 3 | ||
| - | The gmvae_pipeline_v2.py script provides a unified interface for GMVAE training, hyperparameter grid search, and evaluation of GMVAE for data clustering. | + | The gmvae_pipeline_v2.py script provides a unified interface for GMVAE training, hyperparameter grid search, and evaluation of GMVAE for data clustering. The prepared scripts have been included. |
| The program supports three operating modes: | The program supports three operating modes: | ||
| - | * **train** – Train a single GMVAE model. | + | * **train** – Train a single GMVAE model. |
| - | * **grid** – Perform a grid search over multiple values of K and z. | + | * **grid** – Perform a grid search over multiple values of K and z. (run_grid.csh or run_grid_manually.scr) |
| - | * **eval** – Evaluate a trained model and generate clustering results. | + | * **eval** – Evaluate a trained model and generate clustering results. |
| **Training Mode (train)** | **Training Mode (train)** | ||
| Line 198: | Line 205: | ||
| Once a suitable set of training hyperparameters has been determined (excluding K and z), perform a systematic grid search to identify the optimal latent-space dimension and number of clusters. | Once a suitable set of training hyperparameters has been determined (excluding K and z), perform a systematic grid search to identify the optimal latent-space dimension and number of clusters. | ||
| - | < | + | |
| **Example** | **Example** | ||
| + | < | ||
| python gmvae_pipeline_v2.py --mode grid \ | python gmvae_pipeline_v2.py --mode grid \ | ||
| --pdb / | --pdb / | ||
| Line 269: | Line 277: | ||
| **GMVAE Operation details** | **GMVAE Operation details** | ||
| + | Detailed instructions for using this gmvae script are provided below. | ||
| + | |||
| + | < | ||
| + | usage: gmvae_pipeline_v2.py [-h] --mode {train, | ||
| + | --dataset-npy DATASET_NPY --rmsd-npy RMSD_NPY | ||
| + | [--mode_chain {single_chain, | ||
| + | [--indices1 INDICES1 INDICES1] | ||
| + | [--indices2 INDICES2 INDICES2] | ||
| + | [--batch-size BATCH_SIZE] [--lr LR] | ||
| + | [--epochs EPOCHS] [--num-layers NUM_LAYERS] | ||
| + | [--z-dim Z_DIM] [--temp TEMP] [--K K] | ||
| + | [--restore RESTORE] [--dense-n DENSE_N] | ||
| + | [--filters FILTERS [FILTERS ...]] | ||
| + | [--kernel-sizes KERNEL_SIZES [KERNEL_SIZES ...]] | ||
| + | [--pool-sizes POOL_SIZES [POOL_SIZES ...]] | ||
| + | [--valid-ratio VALID_RATIO] [--dropout DROPOUT] | ||
| + | [--r-nent R_NENT] [--r-label R_LABEL] | ||
| + | [--output-dir OUTPUT_DIR] | ||
| + | [--grid-ks [GRID_KS [GRID_KS ...]]] | ||
| + | [--grid-zs [GRID_ZS [GRID_ZS ...]]] | ||
| + | [--evaluate-after-train] [--num-cpus NUM_CPUS] | ||
| + | [--model-dir MODEL_DIR] | ||
| + | |||
| + | GMVAE training & grid testing pipeline | ||
| + | |||
| + | optional arguments: | ||
| + | -h, --help | ||
| + | --mode {train, | ||
| + | Mode to run: train | grid | eval | ||
| + | --pdb PDB Path to pdb file | ||
| + | --dataset-npy DATASET_NPY | ||
| + | Path to dataset .npy file | ||
| + | --rmsd-npy RMSD_NPY | ||
| + | --mode_chain {single_chain, | ||
| + | Single chain or dimer analysis (affects indices_range2 | ||
| + | selection) | ||
| + | --indices1 INDICES1 INDICES1 | ||
| + | Range for chain 1 indices, e.g., 0 99. | ||
| + | --indices2 INDICES2 INDICES2 | ||
| + | Range for chain 2 indices, e.g., 100 220. | ||
| + | --batch-size BATCH_SIZE | ||
| + | --lr LR | ||
| + | --epochs EPOCHS | ||
| + | --num-layers NUM_LAYERS | ||
| + | --z-dim Z_DIM | ||
| + | --temp TEMP | ||
| + | --K K | ||
| + | --restore RESTORE | ||
| + | --dense-n DENSE_N | ||
| + | --filters FILTERS [FILTERS ...] | ||
| + | --kernel-sizes KERNEL_SIZES [KERNEL_SIZES ...] | ||
| + | --pool-sizes POOL_SIZES [POOL_SIZES ...] | ||
| + | --valid-ratio VALID_RATIO | ||
| + | --dropout DROPOUT | ||
| + | --r-nent R_NENT | ||
| + | --r-label R_LABEL | ||
| + | --output-dir OUTPUT_DIR | ||
| + | Base output directory | ||
| + | --grid-ks [GRID_KS [GRID_KS ...]] | ||
| + | List of K values for grid | ||
| + | --grid-zs [GRID_ZS [GRID_ZS ...]] | ||
| + | List of z-dim values for grid | ||
| + | --evaluate-after-train | ||
| + | Run evaluation after each training run (CPU) | ||
| + | --num-cpus NUM_CPUS | ||
| + | --model-dir MODEL_DIR | ||
| + | Model output dir to evaluate (used in eval mode) | ||
| + | |||
| + | </ | ||
| **1. Hyperparameter Optimization (excluding K and z)** | **1. Hyperparameter Optimization (excluding K and z)** | ||
| Line 284: | Line 361: | ||
| Systems | Systems | ||
| | | ||
| - | Trp-cage 2 64 5 8 5000 0.1 [3, | + | Trp-cage 2 64 5 8 5000 0.1 [3, |
| - | BBA 2 64 6 9 5000 0.1 [3, | + | BBA 2 64 6 9 5000 0.1 [3, |
| - | villin | + | villin |
| - | BST-2/ | + | BST-2/ |
| - | BST-2/ | + | BST-2/ |
| - | ORF7b dimer 3 64 6 10 2500 0.05 [3, | + | ORF7b dimer 3 64 6 10 2500 0.05 [3, |
| - | ORF7b dimer 3 64 6 8 2500 0.05 [3, | + | ORF7b dimer 3 64 6 8 2500 0.05 [3, |
| - | BST-2/ | + | BST-2/ |
| </ | </ | ||
| Line 321: | Line 398: | ||
| </ | </ | ||
| - | **Note: If the whole usage of data set is not a case, please specify | + | **My suggestion is to adjust the number of input data points to an appropriate batch size to ensure 0 leftover.** |
| + | |||
| + | **Note: | ||
| **1.4. Recommended Optimization Strategy** | **1.4. Recommended Optimization Strategy** | ||
| Line 340: | Line 419: | ||
| After completing the grid search, evaluate each model using multiple complementary metrics. The goal is to identify the lowest-complexity model that accurately represents the conformational landscape while producing confident, interpretable, | After completing the grid search, evaluate each model using multiple complementary metrics. The goal is to identify the lowest-complexity model that accurately represents the conformational landscape while producing confident, interpretable, | ||
| Optimal Model Selection Criteria | Optimal Model Selection Criteria | ||
| - | An optimal GMVAE model should satisfy the following criteria: | + | |
| + | **3.1 An optimal GMVAE model should satisfy the following criteria:** | ||
| * Low reconstruction loss: Accurately reconstructs the input distance matrices while preserving conformational variability. | * Low reconstruction loss: Accurately reconstructs the input distance matrices while preserving conformational variability. | ||
| * Low cluster assignment entropy: Produces confident, well-separated Gaussian mixture assignments. | * Low cluster assignment entropy: Produces confident, well-separated Gaussian mixture assignments. | ||
| Line 347: | Line 427: | ||
| * Balanced cluster populations: | * Balanced cluster populations: | ||
| * Structural interpretability: | * Structural interpretability: | ||
| - | **Practical Recommendations** | + | **3.2 Practical Recommendations** |
| * Choose the smallest latent dimension (z) after reconstruction loss and cluster entropy have converged. | * Choose the smallest latent dimension (z) after reconstruction loss and cluster entropy have converged. | ||
| * Select the number of clusters (K) that balances structural diversity with sufficient samples per cluster. | * Select the number of clusters (K) that balances structural diversity with sufficient samples per cluster. | ||
| Line 355: | Line 435: | ||
| * Prefer the smallest (K, z) configuration that meets all evaluation criteria, balancing reconstruction quality, clustering confidence, latent-space independence, | * Prefer the smallest (K, z) configuration that meets all evaluation criteria, balancing reconstruction quality, clustering confidence, latent-space independence, | ||
| + | **Cluster Visualization and Structural Interpretation** | ||
| - | data_collection.py | + | After selecting an optimal |
| - | + | ||
| - | Required python | + | |
| - | channels: | + | |
| - | * conda-forge | + | |
| - | * defaults | + | |
| - | dependencies: | + | |
| - | * python=3.9 | + | |
| - | * numpy | + | |
| - | * mdtraj | + | |
| - | * pyemma | + | |
| - | + | ||
| - | or using provided pyemma.yml to build conda environment. | + | |
| - | + | ||
| - | === Truncating Trajectories === | + | |
| - | + | ||
| - | For large systems, it is recommended to trim the trajectory to include only the heavy atoms of interest (e.g., CA). | + | |
| - | + | ||
| - | Data file archive: save pdb in ca_dcd/ (e.g. ca_dcd/ | + | |
| - | + | ||
| - | Example: | + | |
| < | < | ||
| - | python data_collection.py --pdb / | + | usage: contact_heatmap_top_structures.py [-h] --mode {dataset, |
| + | --mode_chain {single_chain, | ||
| + | [--A_dataset A_DATASET] | ||
| + | [--B_dataset B_DATASET] | ||
| + | [--AB_dataset AB_DATASET] | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | [--traj_prefix | ||
| + | | ||
| + | [--start | ||
| + | [--skip | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | [--representative_output_dir REPRESENTATIVE_OUTPUT_DIR] | ||
| - | === Starting Data Collection Pipeline === | + | </ |
| - | Length of indices1: 100 | + | The script performs the following tasks: |
| - | Length of indices2: 121 | + | * Reads one or more molecular dynamics trajectories; |
| - | Feature dimension: 12100 | + | |
| - | 100%|████████████████████| 100/100 [00: | + | * Assigns each trajectory frame to its corresponding GMVAE cluster. |
| - | 100%|████████████████████| 100/100 [00: | + | * Generates residue-residue contact probability heatmaps |
| - | Final all_data length: 10000 | + | * Identifies the top representative conformations (highest cluster assignment probability) from each cluster. |
| - | Computed RMSD for 10000 frames. | + | * Writes representative structures to individual trajectory files for downstream visualization |
| - | Saved outputs: orf7a_bst2_short_data.npy and orf7a_bst2_short_rmsd.npy | + | |
| - | === Pipeline Completed === | + | **Example script** |
| - | Using tf1.15.4.yml to build conda environment (pyemma excluded). | ||
| - | GMVAE clustering via Zaratan v100 GPU | ||
| - | |||
| - | Example of run.sh | ||
| < | < | ||
| #!/bin/bash | #!/bin/bash | ||
| - | #SBATCH --job-name=k6_z8_r1 | + | #Conda |
| - | #SBATCH -n 8 | + | |
| - | #SBATCH --partition=GPU | + | |
| - | #SBATCH --gpus=v100: | + | |
| - | #SBATCH --time=01: | + | |
| source ~/.bashrc | source ~/.bashrc | ||
| conda init --all | conda init --all | ||
| - | conda activate /home/mhsieh/scratch/tf1.15.4/ | + | #conda activate /scratch/zt1/project/energybio/ |
| - | pip list | grep tensorflow | + | conda activate / |
| - | echo "python gmvae_pipeline.py --mode train \ | + | # |
| - | --pdb /home/mhsieh/scratch/ | + | dir_pdb="/scratch/ |
| - | --dataset-npy orf7a_bst2_short_data.npy \ | + | |
| - | --rmsd-npy orf7a_bst2_short_rmsd.npy \ | + | |
| - | --mode_chain dimer \ | + | |
| - | --indices1 0 99 \ | + | |
| - | --indices2 100 220 \ | + | |
| - | --K 6 --z-dim 8 \ | + | |
| - | --epochs 100 --batch-size 500 \ | + | |
| - | --output-dir outputs" | + | |
| - | python gmvae_pipeline.py --mode train \ | + | dir1="/scratch/zt1/project/ |
| - | --pdb /home/mhsieh/scratch/ | + | dir2="/ |
| - | --dataset-npy orf7a_bst2_short_data.npy \ | + | dir3="/ |
| - | --rmsd-npy orf7a_bst2_short_rmsd.npy \ | + | dir4="/ |
| - | --mode_chain dimer \ | + | dir5="/ |
| - | --indices1 0 99 \ | + | dir6="/ |
| - | | + | out_prefix=" |
| - | | + | |
| - | --epochs 100 --batch-size 500 \ | + | |
| - | --output-dir outputs | + | |
| - | </code> | + | |
| - | If process evaluation right after clustering, the add | + | #Files |
| - | < | + | file_traj1=" |
| - | python gmvae_pipeline.py --mode train \ | + | file_traj2=" |
| - | --pdb / | + | file_traj3=" |
| - | | + | file_traj4=" |
| - | | + | file_traj5=" |
| - | --mode_chain dimer \ | + | |
| - | --indices1 0 99 \ | + | |
| - | --indices2 100 220 \ | + | |
| - | --K 6 --z-dim 4 \ | + | |
| - | --epochs 10 --batch-size 500 \ | + | |
| - | --output-dir outputs \ | + | |
| - | --evaluate_after_train | + | |
| - | </ | + | |
| - | If run evaluation separately, using the following script. | + | #Commands |
| - | < | + | python3 ../src/utils/data_collection_v4.py \ |
| - | #!/bin/bash | + | --pdb ${dir_pdb} |
| - | #SBATCH --job-name=k11_z4_r1 | + | --dimer --selected_atom CA --indices1 0 99 --indices2 100 220 --output_prefix ${out_prefix} |
| - | #SBATCH -N 1 | + | --traj_files ${dir1}${file_traj1} ${dir1}${file_traj2} ${dir1}${file_traj3} ${dir1}${file_traj4} ${dir1}${file_traj5} |
| - | #SBATCH -n 64 | + | ${dir2}${file_traj1} ${dir2}${file_traj2} ${dir2}${file_traj3} ${dir2}${file_traj4} ${dir2}${file_traj5} |
| - | #SBATCH --partition=standard | + | ${dir3}${file_traj1} ${dir3}${file_traj2} ${dir3}${file_traj3} ${dir3}${file_traj4} ${dir3}${file_traj5} |
| - | #SBATCH --time=01: | + | ${dir4}${file_traj1} ${dir4}${file_traj2} ${dir4}${file_traj3} ${dir4}${file_traj4} ${dir4}${file_traj5} |
| - | + | ${dir5}${file_traj1} ${dir5}${file_traj2} ${dir5}${file_traj3} ${dir5}${file_traj4} ${dir5}${file_traj5} | |
| - | + | ${dir6}${file_traj1} ${dir6}${file_traj2} ${dir6}${file_traj3} ${dir6}${file_traj4} ${dir6}${file_traj5} | |
| - | source ~/.bashrc | + | |
| - | conda init --all | + | |
| - | conda activate | + | |
| - | + | ||
| - | pip list | grep tensorflow | + | |
| - | + | ||
| - | echo " | + | |
| - | --pdb / | + | |
| - | --dataset-npy orf7a_bst2_short_data.npy \ | + | |
| - | | + | |
| - | --mode_chain dimer \ | + | |
| - | | + | |
| - | | + | |
| - | --K 6 --z-dim 8 \ | + | |
| - | --batch-size 1 \ | + | |
| - | | + | |
| - | | + | |
| - | | + | |
| - | + | ||
| - | python gmvae_pipeline.py --mode eval \ | + | |
| - | --pdb / | + | |
| - | | + | |
| - | --rmsd-npy orf7a_bst2_short_rmsd.npy \ | + | |
| - | --mode_chain dimer \ | + | |
| - | --indices1 0 99 \ | + | |
| - | --indices2 100 220 \ | + | |
| - | --K 6 --z-dim 8 \ | + | |
| - | --batch-size 1 \ | + | |
| - | --output-dir outputs \ | + | |
| - | --model-dir ./ | + | |
| - | --num-cpus 64 | + | |
| </ | </ | ||
| - | |||
gmvae_update.1785664718.txt.gz · Last modified: 2026/08/02 05:58 by edit