From ab2c2800f1a736498be66409c47a0393c277f9c0 Mon Sep 17 00:00:00 2001 From: Michael Baughn Date: Wed, 20 Jan 2016 22:09:04 -0800 Subject: [PATCH 01/16] a message for commit, I talked about bashrc --- commandline.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/commandline.md b/commandline.md index ccd04eb..dd8d732 100644 --- a/commandline.md +++ b/commandline.md @@ -92,3 +92,14 @@ To install a program from source on a shared cluster, you will often need to spe ./configure --prefix=/projects/ps-yeolab/software make && make install # "make install" will run only if "make" is successful ``` + + +### .bashrc + +This is where you put all of the stuff you want to execute, load, echo, etc when you login to TSCC! + +``` +nano ~/.bashrc +``` + +add your code here! From 41435735249170b59f18ef4fded1f650ae9f0438 Mon Sep 17 00:00:00 2001 From: mbaughn Date: Thu, 18 Feb 2016 09:12:06 -0800 Subject: [PATCH 02/16] added stuff from my text tile --- commandline.md | 33 +++++++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/commandline.md b/commandline.md index dd8d732..314ed26 100644 --- a/commandline.md +++ b/commandline.md @@ -1,15 +1,40 @@ # Command line cheatsheet This is a cheat sheet for "Bourne-again shell" (bash) and GNU/Linux commands. +## The basics: +``` + cd ~ go to home directory + clear clear terminal + source reload this file + mkdir make directory + rm -rf Remove recursively and force + chmod ### 777 is the most generous, order: user, group, everyone + more Contents shows up below printed to terminal + less bring up the document, but remove it from the screen and terminal on Q + cat can view (print to terminal) can also pipe data to other things, concatenate vertically + > Output to a file (can replace the file if it already exists) + >> append to existing file (does not replace existing file, only appends) + mv Cut and Paste (copy but remove source) + cp copy (retain source) + ln -s soft links Source Destination + which find the path to the object specified + ls -lhaH human-readable sizes, permissions, groups, and symbolic links displaying fullpaths + qstat -r hotel | less Show all jobs on the node “hotel” + qstat -u ucsd-train03 show job status list for the user “ucsd-train03” + qsub script.sh submit script (which has PBS flags) to the queue + +``` + ## Text maninpulation -### Skip the first line of a file +## Skip the first line of a file ``` tail -n +2 filename.txt ``` + ## Compressing and decompressing files ### Create a gzipped tarball from a `*` glob command @@ -96,10 +121,14 @@ make && make install # "make install" will run only if "make" is successful ### .bashrc -This is where you put all of the stuff you want to execute, load, echo, etc when you login to TSCC! +####This is where you put all of the stuff you want to execute, load, echo, etc when you login to TSCC! ``` nano ~/.bashrc ``` +#### On your local machine: +``` +nano ~/.bash_preferences +``` add your code here! From d8d951c16bd3fb1424807881c07000644d700dd1 Mon Sep 17 00:00:00 2001 From: mbaughn Date: Thu, 18 Feb 2016 10:07:34 -0800 Subject: [PATCH 03/16] added more stuff from my text file --- commandline.md | 84 +++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 77 insertions(+), 7 deletions(-) diff --git a/commandline.md b/commandline.md index 314ed26..f3cfe46 100644 --- a/commandline.md +++ b/commandline.md @@ -12,7 +12,7 @@ This is a cheat sheet for "Bourne-again shell" (bash) and GNU/Linux commands. more Contents shows up below printed to terminal less bring up the document, but remove it from the screen and terminal on Q cat can view (print to terminal) can also pipe data to other things, concatenate vertically - > Output to a file (can replace the file if it already exists) + > Output to a file (can replace the file if it already exists) Called "redirecting" >> append to existing file (does not replace existing file, only appends) mv Cut and Paste (copy but remove source) cp copy (retain source) @@ -20,14 +20,48 @@ This is a cheat sheet for "Bourne-again shell" (bash) and GNU/Linux commands. which find the path to the object specified ls -lhaH human-readable sizes, permissions, groups, and symbolic links displaying fullpaths qstat -r hotel | less Show all jobs on the node “hotel” - qstat -u ucsd-train03 show job status list for the user “ucsd-train03” - qsub script.sh submit script (which has PBS flags) to the queue - + qstat -u ucsd-train03 show job status list for the user “ucsd-train03” + qsub script.sh submit script (which has PBS flags) to the queue + tee Write to something and submit it to a program (see manual) + who On TSCC this lists all of the logged in users and their IP addresses + pwd -P Print physical directory "what drive am I on?" + top What jobs are running? + screen See: https://kb.iu.edu/d/acuy ``` +## How basic program inputs and outputs work: +Standard in +to +#####Program +outputs two outputs +###### 1) Standard out (stdout) +###### 2) Standard Error (stderr) +You can redirect either or both E.G. ```grep NFKB tfs.bed 1>tfs.nfkb.bed 2>tfs.nfkb.err``` -## Text maninpulation +If you don't specify the stream that you want it will just redirect the standard out and alternatively print the standard error output to the command line (E.G. grep:tfs.bed: No such file or directory). + +###Advanced: +On TSCC if you write +2>&1 +as the last line, you will cat your standard error and out files to standard out + +#TSCC +when you are in your home directory on the login node, you are writing to the login node, the login node has very little available ram per user (5 login nodes * 64GB /#Users) + +when you are in your scratch directory, you are writing to the scratch directory + +when you submit to qsub, you are assigned a host machine based on your request (#PBS scripts) for processors, memory, etc. AT THE MOMENT YOU MAKE THE REQUEST. If other tasks go out of control and start taking up more memory it can fuck things up quickly for everyone on that host. + +If you disconnect from the login node while writing to Scratch, it cuts off whatever you are currently writing and leaves a corrupted file. USING THE HEAD NODE IS RISKY. If they want to install an update and bring down your login node, you're boned. If you've submitted your stuff as a job you are far more protected. + +## Text maninpulation +### Navgiation +``` +Control + E Jump to end of the line +Control + A Jump to start of the line +Alt + ErrorKeys Jump between words on the same line +``` ## Skip the first line of a file ``` @@ -118,17 +152,53 @@ To install a program from source on a shared cluster, you will often need to spe make && make install # "make install" will run only if "make" is successful ``` +### Screens +Always use the same login node! +``` +screen -S try1 +``` +[detached] + +``` +screen -S try2 +``` +[detached] + +``` +screen -x +``` +There are several suitable screens on: + 40357.try1 (Detached) + 40451.try2 (Detached) + +Type +``` +screen [-d] -r [pid.]tty.host" +``` +to resume one of them. +``` +screen -x 40357.try +``` +[screen is terminating] + +``` +screen -x +``` +[screen is terminating] + ### .bashrc -####This is where you put all of the stuff you want to execute, load, echo, etc when you login to TSCC! +#####This is where you put all of the stuff you want to execute, load, echo, etc when you login to TSCC! ``` nano ~/.bashrc ``` -#### On your local machine: +##### On your local machine: ``` nano ~/.bash_preferences ``` +##### For using "screens" +```nano ~/.screenrc add your code here! From a284f8a002047517505027505764dc59a35ebb16 Mon Sep 17 00:00:00 2001 From: mbaughn Date: Thu, 18 Feb 2016 10:08:13 -0800 Subject: [PATCH 04/16] formatting --- commandline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/commandline.md b/commandline.md index f3cfe46..418f506 100644 --- a/commandline.md +++ b/commandline.md @@ -5,7 +5,7 @@ This is a cheat sheet for "Bourne-again shell" (bash) and GNU/Linux commands. ``` cd ~ go to home directory clear clear terminal - source reload this file + source reload this file mkdir make directory rm -rf Remove recursively and force chmod ### 777 is the most generous, order: user, group, everyone From 61a3d69171232b86de125cc00560ba3ec4b70175 Mon Sep 17 00:00:00 2001 From: mbaughn Date: Thu, 18 Feb 2016 12:06:48 -0800 Subject: [PATCH 05/16] added commands --- commandline.md | 51 ++++++++++++++++++++++++++++---------------------- 1 file changed, 29 insertions(+), 22 deletions(-) diff --git a/commandline.md b/commandline.md index 418f506..f276c64 100644 --- a/commandline.md +++ b/commandline.md @@ -3,30 +3,37 @@ This is a cheat sheet for "Bourne-again shell" (bash) and GNU/Linux commands. ## The basics: ``` - cd ~ go to home directory - clear clear terminal - source reload this file - mkdir make directory - rm -rf Remove recursively and force - chmod ### 777 is the most generous, order: user, group, everyone - more Contents shows up below printed to terminal - less bring up the document, but remove it from the screen and terminal on Q - cat can view (print to terminal) can also pipe data to other things, concatenate vertically - > Output to a file (can replace the file if it already exists) Called "redirecting" - >> append to existing file (does not replace existing file, only appends) - mv Cut and Paste (copy but remove source) - cp copy (retain source) - ln -s soft links Source Destination - which find the path to the object specified - ls -lhaH human-readable sizes, permissions, groups, and symbolic links displaying fullpaths - qstat -r hotel | less Show all jobs on the node “hotel” - qstat -u ucsd-train03 show job status list for the user “ucsd-train03” - qsub script.sh submit script (which has PBS flags) to the queue - tee Write to something and submit it to a program (see manual) - who On TSCC this lists all of the logged in users and their IP addresses - pwd -P Print physical directory "what drive am I on?" + cd ~ go to home directory + clear clear terminal + source reload this file + mkdir make directory + rm -rf Remove recursively and force + chmod ### 777 is the most generous, order: user, group, everyone + more Contents shows up below printed to terminal + less bring up the document, but remove it from the screen and terminal on Q + cat can view (print to terminal) can also pipe data to other things, concatenate vertically + > Output to a file (can replace the file if it already exists) Called "redirecting" + >> append to existing file (does not replace existing file, only appends) + mv Cut and Paste (copy but remove source) + cp copy (retain source) + ln -s soft links Source Destination + which find the path to the object specified + ls -lhaH human-readable sizes, permissions, groups, and symbolic links displaying fullpaths + qstat -r hotel | less Show all jobs on the node “hotel” + qstat -u ucsd-train03 show job status list for the user “ucsd-train03” + qsub script.sh submit script (which has PBS flags) to the queue + tee Write to something and submit it to a program (see manual) + who On TSCC this lists all of the logged in users and their IP addresses + pwd -P Print physical directory "what drive am I on?" top What jobs are running? screen See: https://kb.iu.edu/d/acuy + git ls-files Which files are managed by git + git ls-files -u Which files are unmerged or have conflicts + git log show all commits that have happened with these files (use to find commit hash if you want to reset) + git reset (hash) reset to certain commit (find hash with git log) + git checkout --ours use everything that is yours when you merge + git checkout --theirs use everything that is theirs when you merge + ``` ## How basic program inputs and outputs work: Standard in From 3a97ff4b21cd0af3ec3b3442d77d678a04b1f44b Mon Sep 17 00:00:00 2001 From: mbaughn Date: Thu, 18 Feb 2016 12:10:07 -0800 Subject: [PATCH 06/16] Added biotools install command for bowtie, star... --- commandline.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/commandline.md b/commandline.md index f276c64..baa72d7 100644 --- a/commandline.md +++ b/commandline.md @@ -208,4 +208,15 @@ nano ~/.bash_preferences ##### For using "screens" ```nano ~/.screenrc -add your code here! + +### Installing biotools +```mothership:~ mbaughn$ pip install biotools +Collecting biotools + Downloading biotools-1.2.12.tar.gz +Building wheels for collected packages: biotools + Running setup.py bdist_wheel for biotools ... done + Stored in directory: /Users/mbaughn/Library/Caches/pip/wheels/86/0e/ec/57f244b7a14c7847b55636f1900206b5039b38828372cafd87 +Successfully built biotools +Installing collected packages: biotools +Successfully installed biotools-1.2.12 +``` From f9017ad2f6e5b8b11e81cbe502a2427be2efca26 Mon Sep 17 00:00:00 2001 From: mbaughn Date: Thu, 18 Feb 2016 17:20:56 -0800 Subject: [PATCH 07/16] Emoji? :troll: --- commandline.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/commandline.md b/commandline.md index baa72d7..e23fc12 100644 --- a/commandline.md +++ b/commandline.md @@ -1,3 +1,5 @@ +http://www.emoji-cheat-sheet.com/ + # Command line cheatsheet This is a cheat sheet for "Bourne-again shell" (bash) and GNU/Linux commands. From fffdaba04b8dd75ebf8e74f60d97435236e13ed8 Mon Sep 17 00:00:00 2001 From: mbaughn Date: Thu, 18 Feb 2016 18:07:48 -0800 Subject: [PATCH 08/16] added tree command --- commandline.md | 1 + 1 file changed, 1 insertion(+) diff --git a/commandline.md b/commandline.md index e23fc12..7e20a50 100644 --- a/commandline.md +++ b/commandline.md @@ -18,6 +18,7 @@ This is a cheat sheet for "Bourne-again shell" (bash) and GNU/Linux commands. >> append to existing file (does not replace existing file, only appends) mv Cut and Paste (copy but remove source) cp copy (retain source) + tree prints out a directory structure tree (stop doing cd ls recursively!) ln -s soft links Source Destination which find the path to the object specified ls -lhaH human-readable sizes, permissions, groups, and symbolic links displaying fullpaths From 8069a9d6b800a0d151d0516162214415120b2fa5 Mon Sep 17 00:00:00 2001 From: mbaughn Date: Thu, 18 Feb 2016 20:52:01 -0800 Subject: [PATCH 09/16] updated tree with -f option --- commandline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/commandline.md b/commandline.md index 7e20a50..f06602f 100644 --- a/commandline.md +++ b/commandline.md @@ -18,7 +18,7 @@ This is a cheat sheet for "Bourne-again shell" (bash) and GNU/Linux commands. >> append to existing file (does not replace existing file, only appends) mv Cut and Paste (copy but remove source) cp copy (retain source) - tree prints out a directory structure tree (stop doing cd ls recursively!) + tree -f prints out a directory structure tree (stop doing cd ls recursively!) -f option prints full paths on every branch ln -s soft links Source Destination which find the path to the object specified ls -lhaH human-readable sizes, permissions, groups, and symbolic links displaying fullpaths From 4d266e9827fa6fd506e992f24e4eef7dda09dd36 Mon Sep 17 00:00:00 2001 From: mbaughn Date: Fri, 19 Feb 2016 11:36:50 -0800 Subject: [PATCH 10/16] hotel is not a node --- commandline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/commandline.md b/commandline.md index f06602f..1358b0e 100644 --- a/commandline.md +++ b/commandline.md @@ -22,7 +22,7 @@ This is a cheat sheet for "Bourne-again shell" (bash) and GNU/Linux commands. ln -s soft links Source Destination which find the path to the object specified ls -lhaH human-readable sizes, permissions, groups, and symbolic links displaying fullpaths - qstat -r hotel | less Show all jobs on the node “hotel” + qstat -r hotel | less Show all jobs on the group “hotel” qstat -u ucsd-train03 show job status list for the user “ucsd-train03” qsub script.sh submit script (which has PBS flags) to the queue tee Write to something and submit it to a program (see manual) From 248320447bdc79804e65702dc6bac56f13f9365e Mon Sep 17 00:00:00 2001 From: mbaughn Date: Fri, 19 Feb 2016 12:28:00 -0800 Subject: [PATCH 11/16] disk use command added --- commandline.md | 1 + 1 file changed, 1 insertion(+) diff --git a/commandline.md b/commandline.md index 1358b0e..734976c 100644 --- a/commandline.md +++ b/commandline.md @@ -19,6 +19,7 @@ This is a cheat sheet for "Bourne-again shell" (bash) and GNU/Linux commands. mv Cut and Paste (copy but remove source) cp copy (retain source) tree -f prints out a directory structure tree (stop doing cd ls recursively!) -f option prints full paths on every branch + du -ch display disk use (-c sum all subfolders) (-h make human readable) ln -s soft links Source Destination which find the path to the object specified ls -lhaH human-readable sizes, permissions, groups, and symbolic links displaying fullpaths From 1c0b9e3c928543b5c5b303da4c7b979e6d761cce Mon Sep 17 00:00:00 2001 From: mbaughn Date: Mon, 29 Feb 2016 14:56:25 -0800 Subject: [PATCH 12/16] Windows Firewall netsh advfirewall line --- commandline.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/commandline.md b/commandline.md index 734976c..d38799f 100644 --- a/commandline.md +++ b/commandline.md @@ -224,3 +224,8 @@ Successfully built biotools Installing collected packages: biotools Successfully installed biotools-1.2.12 ``` + +##### Windows Firewall -- Block an Individual IP Address: +``` +netsh advfirewall firewall add rule name="Intuit1" dir=out interface=any action=block remoteip=75.98.51.173 +``` From 2e7092a26ecb32fa0cb2f523897a855eb4cff33c Mon Sep 17 00:00:00 2001 From: mbaughn Date: Mon, 29 Feb 2016 14:58:08 -0800 Subject: [PATCH 13/16] Fixed formatting --- commandline.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/commandline.md b/commandline.md index d38799f..dab602a 100644 --- a/commandline.md +++ b/commandline.md @@ -201,20 +201,22 @@ screen -x ### .bashrc #####This is where you put all of the stuff you want to execute, load, echo, etc when you login to TSCC! - ``` nano ~/.bashrc ``` + ##### On your local machine: ``` nano ~/.bash_preferences ``` ##### For using "screens" -```nano ~/.screenrc - +``` +nano ~/.screenrc +``` ### Installing biotools -```mothership:~ mbaughn$ pip install biotools +``` +mothership:~ mbaughn$ pip install biotools Collecting biotools Downloading biotools-1.2.12.tar.gz Building wheels for collected packages: biotools From 5692217d6a4a47f92bc18243eafc9d9fac686e9a Mon Sep 17 00:00:00 2001 From: mbaughn Date: Sun, 24 Apr 2016 19:57:04 -0700 Subject: [PATCH 14/16] Added hdiutil use on mac command --- commandline.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/commandline.md b/commandline.md index dab602a..03380fd 100644 --- a/commandline.md +++ b/commandline.md @@ -38,6 +38,10 @@ This is a cheat sheet for "Bourne-again shell" (bash) and GNU/Linux commands. git checkout --ours use everything that is yours when you merge git checkout --theirs use everything that is theirs when you merge + Mac-Only: hdiutil attach /path Mount a disk image via command line + hdiutil info Tell me about mounted disk images + hdiutil detach /dev/disk1s2 detach image at specified location if you don't know it, use the info command^ + ``` ## How basic program inputs and outputs work: Standard in From 0e4eb66e391f001bc9613127e695fb756b74bd8a Mon Sep 17 00:00:00 2001 From: mbaughn Date: Sun, 24 Apr 2016 19:57:27 -0700 Subject: [PATCH 15/16] fixed formatting error --- commandline.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/commandline.md b/commandline.md index 03380fd..67faa8f 100644 --- a/commandline.md +++ b/commandline.md @@ -38,7 +38,8 @@ This is a cheat sheet for "Bourne-again shell" (bash) and GNU/Linux commands. git checkout --ours use everything that is yours when you merge git checkout --theirs use everything that is theirs when you merge - Mac-Only: hdiutil attach /path Mount a disk image via command line + Mac-Only: + hdiutil attach /path Mount a disk image via command line hdiutil info Tell me about mounted disk images hdiutil detach /dev/disk1s2 detach image at specified location if you don't know it, use the info command^ From 1c0ed414f5d07980bbac1c33f34a8c2c12a8bb9f Mon Sep 17 00:00:00 2001 From: mbaughn Date: Fri, 28 Apr 2017 15:50:56 -0700 Subject: [PATCH 16/16] added command to load pre-installed TSCC modules --- commandline.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/commandline.md b/commandline.md index 67faa8f..5f62e2f 100644 --- a/commandline.md +++ b/commandline.md @@ -70,6 +70,11 @@ when you submit to qsub, you are assigned a host machine based on your request ( If you disconnect from the login node while writing to Scratch, it cuts off whatever you are currently writing and leaves a corrupted file. USING THE HEAD NODE IS RISKY. If they want to install an update and bring down your login node, you're boned. If you've submitted your stuff as a job you are far more protected. +To load pre-installed software on TSCC (biotools, etc https://www.sdsc.edu/support/user_guides/tscc-quick-start.html#applications) use the command: +''' +module load biotools +''' + ## Text maninpulation ### Navgiation