Differences

This shows you the differences between two versions of the page.

--- backup [2017/06/13 09:38] – created admin
+++ backup [2023/09/01 16:06] (current) – admin
@@ Line 1: / Line 1: @@
 [[sidebar]]
+====== Backup Data Instructions ======
-Contained in this wiki are instructions for backing your data on the **Klauda Lab Backup Server**. The name of this server is //klauda-bkup1.umd.edu// and can only be reached on computers on the UMD network.  You'll need to logon to DT1/DT2 first if you need to access this server outside campus without VPN.
-. Create your login and password on backup server
+Contained in this wiki are instructions for backing your data on the **Klauda Lab Backup Server**. The name of this server is //klauda-bkup1.umd.edu// and can only be reached on computers on the UMD network.  Your UMD password/userid is needed to log into this server. You'll need to logon to ZT1 first if you need to access this server outside campus without VPN.
-Use same username as your deepthought account but a different strong password. Myself or Pouyan will
+==== Policies ====
- help you with this in the lab. Please get in touch with one of us to set a time for this.
-. Setting up password-less login to backup server from deepthought
+If you have simulation data, then it **MUST** be backed up in two places in the event of failure of hardware. Data is categorized as either being published or unpublished. As you start in the lab your work is in progress for publication but essential for a potential future publication. Published data is after you've had your work accepted by a scientific journal and no longer requires more simulation or analysis.
-a. login to deepthought
+// Unpublished Data //
-b. From you home on deepthought open .ssh/known_hosts file and delete the line starting with klauda-
-bkup1.umd.edu. Save and exit
-c. Type ssh-keygen
-d. Press y (when asked y/n?) and 'return' key otherwise
-e. ssh-copy-id -i ~/.ssh/id_rsa.pub username@klauda-bkup1.umd.edu (Your username on backup machine)
-f. Try ssh username@klauda-bkup1.umd.edu  and you should not be asked for a password
-. Creating backup directories
+This will be the data you are currently running simulations and analysis on. This will be automatically backed up on a daily basis. If you are a new user (after 7/1/2023), please let Dr. Klauda know so he can add you to the list of users to backup data.  If you run on ACCESS resources, this data needs to be copied to ZT1 for data analysis and backups.
-a. Once you are logged in on the backup server, go to /local0/backup and create a directory of your
+// Published Data //
-username
-b. Inside that create two directories named full and weekly
-c. Inside weekly create 12 directories corresponding to 12 months 01, 02, 03 ... 12
-. Creating backup scripts on deepthought
+Once you have data that is published, you need to reduce the trajectory file sizes and make a single tarred and compressed file (.tar.gz). The name of this file should make it clear to me what publication this is from and be placed in your backup directory on the backup server. Then you will create another backup on an external hard drive (see Dr. Klauda) and remove data from ZT1.  So you still have two locations (external HD and backup server). The DCD/trajectory files should be reduced to save space. If you have 100-400ns of data, then reduce this to have only 10ps frames. If longer, consult with Dr. Klauda on the frame rate.
-a. Create a directory named backup in your home directory on deepthought
+==== File Reduction Instructions ====
-b. Copy backup-full.scr and backup-weekly.scr from /homes/jbklauda/backup to this backup directory
-c. Change the username from 'pypendse' to your username everywhere in the script
-d. There will be two paths - i) Path of your directory on deepthought that will be backed up on the
-server. It should generally be "/export/lustre_1/username"  and (ii) Path of directory on server whe
-re it will be backed up. This will be "/local0/backup/username/full" or "weekly/xx"
-e. Double check whether you have changed the username. Otherwise when you run your backup, my data w
-ill be lost.
-. Setting up crontab
+Download the following directory to reduce dcd files. It is adapted from recenter script from EDP analysis. Adjust crystal parameters in ''cryst.str'' and file range in ''recenter1.inp''. Also change the "skip" option as needed.
-Crontab will schedule your weekly and monthly backups
+{{ ::truncate.tar.gz |}}
-a. When you are on deepthought, type crontab -e, a crontab window will open up
-b. Use the following as a template to create your crontab
-SHELL=/bin/csh
+When ready, mv the files into the dcd directory, which will overwrite the large dcd files.
-# Weekly Incremental Backups
-# Sat at 0X:0X am
-X 0X * * 6 /homes/usrname/backup/backup-weekly.csh
-# Monthly Full Backups
-# 1st day of month at 11:01 am
-X 0X 1 * * /homes/usrname/backup/backup-full.csh
-c. 0X:0X am is the time at which your data will be automatically backed up every Saturday and first
-day of the month. We keep our backup times few hours apart from each other. Once everything else is
-completed, we can decide on these times for everyone.
-d. When you save and exit, it will install crontab
-e. Check whether crontab is installed or not using crontab -l
-Listed below are instructions for setting up accounts on deepthought1 (DT1) and deepthought2 (DT2) located at UMD.
-//Instructions//:
-You will assigned one or both clusters based on your simulation needs by Dr. Klauda. These instructions work for both, so do these on the cluster(s) you are assigned. Dr. Klauda will submit a request to add you as a user to my account on DT1 and/or DT2. Once this is done, you should get a generic email from IT regarding the resource.
-Details of DT1 and DT2 can be found:
-http://www.oit.umd.edu/hpcc/
-Now for the cluster setup:
-If you are not familiar with the use of LINUX, I would first go through some on-line tutorials for commands to use in LINUX and here ([[Linux Guide]]). To login to DT1 (login.deepthought.umd.edu) or DT2 (login.deepthought2.umd.edu) with Windows you will need to have PuTTY ([[Extra Software]]) to connect to deepthought:
-http://www.chiark.greenend.org.uk/%7Esgtatham/putty/
-And the following program will be useful to transfer files between deepthought and your computer
-http://winscp.net/eng/index.php
-Built-in commands for MAC users can be used to connect via the command line (scp) and apps for file transfer (eg, cyberduck).
-Then, take a general look at the website of the HPCC OIT to get a general idea on how to login and submit jobs.  Your jobs should not run in your home directory.  They should instead be in the /lustre directory.  You should first make a subdirectory in this with your username (mkdir is the command). Running simulations should be in organized directories.  So think of some naming scheme for simulations. Before starting a simulation, you will need to make a slight change to you .cshrc file in your home directory.  Using the program pico or vm type the following when you are in your home directory:
-<code commend> pico .cshrc.mine </code>
-At the end of the file add the following line:
-<code commend> unset noclobber </code>
-This is needed for the simulations to allow for files to write over existing files.
-One more recent change is that you will need to allow me to look into your /lustre files.  After you've made the directory (described above), you will need to allow access to the files by typing and replacing username with yours:
-<code commend>cd /lustre/username
-chmod a+r .
-chmod -R a+r *
-find . -type d -exec chmod a+x {} \; </code>