Discussion 2

OS | Bash | Server

If you already know all the stuff which I write in the subtitle, please play around with the following packages rpart, party, partykit.

1. Operation System (OS)

Operation System Wiki We are using different operating systems all days, now just focus on your computer:

  • Windows (Our laptop, you need to pay)
  • Unix and Unix like operation system
    • MacOS (Mac, you need to pay)
    • Linux (Server, FREE!!!)
      1. ubuntu Our server Lunchbox use this version of Linux
      2. mint
      3. Debian
      4. etc.

FAQs of OS

What’s your point to say this?

We need an app/software, which is called Bash, installed into your computer.

Why we need this?

  1. We can use ssh via Bash connect to our lunchbox server.
  2. We can use Bash easily run GUIDE.
  3. We can use other powerful GNU software. eg. grep, sed, awk

Why do we need to use grep, sed, awk? Can we just use excel, modified data by hand?

  1. Yes you can, IN SMALL DATA SET!
  2. One question Valid Number Leetcode, if you have 1,000,000 recorders.
  3. I may teach in the later semester.

Windows User

  1. Do I need to download/install Linux? Do I need to buy a Mac?

No! You don’t need to! Our server is free, you only need a software and connect to the server(lunchbox), then you can try everything! Also, Windows 10 has the bash. Don’t worry I will teach you some basic stuff.

  1. What should I do?

We only need Bash to manager our computer, also can connect to the server. That’s all, the following steps you can do to say whether you can get the bash in your computer

open command prompt;
type `bash`
If you have a problem with bash: 
  if you are using Windows 10:
    see [Windows 10 Installation Guide](https://msdn.microsoft.com/en-us/commandline/wsl/about)
  else if you are using Windows 7:
    try install [UnxUtils](https://sourceforge.net/projects/unxutils/)

Mac User

  1. Do I need to download/install Linux? Do I need to buy a new computer and install Linux?

No! Linux is another type of MacOS, although there is some difference between MacOS and Linux, but you will hard to tell the difference. We are not computer science engineer. Don’t buy a new laptop and install Linux…

  1. What should I do?
  • Mac already have the terminal.app, you can use spotlight to find it. Bash already running inside.
  • [Optional] Mac also has another fancy terminal app call iterm2 I used it.

Other Linux

I think you should have the ability to figure out any questions by yourself. Please use StackOverflow wisely.

2. Bash 101

I hope you already have bash in your hand. Here is a great course

summary (Thanks to Professor Cecile Ane):

  • directory structure, the root is /
  • relative versus absolute paths
  • shortcuts: ., .., ~
    • . current directory
    • .. parent directory
    • ~ your home directory
  • tab completion to get program and file names
  • up/down arrows and ! to repeat commands
whoami who am I? to get your username
pwd print working directory. where am I?
ls list. many options, e.g. -a (all) -l (long) -lrt (reverse-sorted by time)
cd change directory
mkdir make directory
rm remove (forever). -f to force, -i to ask interactively, -r recursively
rmdir remove (delete) directory, if empty
mv move (and rename). can overwrite existing files, unless -i to ask
cp copy. would also overwrite existing files
diff difference
wc word count: lines, words, characters. -l, -w, -c
cat concatenate
less because “less is more”. q to quit.
sort -n for numerical sorting
head first 10 lines. -n 3 for first 3 lines (etc.)
tail last 10 lines. -n 3 for last 3 lines, -n +30 for line 30 and up
uniq filters out repeated lines (consecutive). -c to get counts
cut cut and return column(s). -d, to set the comma as field delimiter (tab otherwise), -f2 to get 2nd field (column)
echo print
history shows the history of all previous commands, numbered
! !76 to re-execute command number 76 in the history, !$ for last word or last command

Things you can do

Question: I want to execute my guide anywhere, not by copy my guide to anywhere.

We all use hyperlink every day, we like to create a desktop icon in windows or drag the program into your mac Dock, so that we can easily execute the program. Why we can do it easily because we know where is the program. How to let our computer know where is guide or in the future, if you need install some software, how to let our computer know.

Bash has a variable call PATH, which specifying a set of directories where executable programs are located. So that when you type guide into terminal or command line, bash will search that directory, if there is a program call guide, it will execute it, if not, they will say command not found

So that we need to tell our computer “next time you need to search an additional directory where guide is located”

The following steps help you do this (Only do it in the first time), and you can execute guide anywhere on your own computer after.

  1. Create the following directory on your computer
    i). ~/local
    ii). ~/local/bin
  2. Add ~/local/bin into your environment PATH variable
mkdir -p ~/local/bin 
echo 'export PATH=$PATH:~/local/bin' >> .bash_profile
source ~/.bash_profile

mkdir -p flag: no error if existing, make parent directories as needed

You can also use man mkdir in bash to see the details

Windows or Linux

If you use bash in Windows command prompt, you need to download Linux version of guide.

Download or move your guide program not zip file into ~/local/bin

cd ~/local/bin
wget http://www.stat.wisc.edu/%7Eloh/treeprogs/guide/linux/64bit/Intel/guide.gz
gunzip guide.gz
chmod a+x guide
guide

Mac

If you want a different version of guide, change the URL.

cd ~/local/bin
curl http://www.stat.wisc.edu/%7Eloh/treeprogs/guide/mac/absoft/guide.gz --output guide.gz
gunzip guide.gz
chmod a+x guide
guide

Try it!!

Go to any directory from bash via cd in your computer, type guide see what’s happening.

Server (Your friend’s computer)

  1. What is lunchbox?

lunchbox is our stat department server. Many of you should already have relative experience with server, just like AWS.
You can take it as your friend’s computer, and we can use your friend’s computer to run your own programs.

  1. Ok, why I need this?

You can definitely ignore it leave if you refuse to learn!

  1. When I need this?

When you need your friend’s computer. For me, I usually need to run some simulations thousands of times, some optimization jobs take weeks. So that I can throw everything into the server.

  • But I need my computer to watch a youtube video.
  • I don’t want to save lots of files into my own computer.
  • I want to play computer game when my simulation is running.
  • I just want to save my electronic bill…
  1. How to use lunchbox?

Change username to your own username! The password will not show anything, but you can keep typing, and use Enter or return to execute.

ssh username@lunchbox.stat.wisc.edu

lunchbox 101

Since it is your friend’s computer, in the beginning, it doesn’t have anything. Even don’t know what’s your homework.

Your friend: Ok, you can use my computer only for homework and computing, you can not play games or watch youtube, promise?
You: Sure, what do you have?
Your friend: I only have a bash terminal…haha, no chrome:P
You: …Ok… Do you have guide?
Your friend: YES!!! Type following in bash

~loh/public/ADDGUIDE

Your friend: You classmates also want to use my computer, so I want to save all your file into one place, OK?
You need to create a directory under /workspace, username is your username to login lunchbox

mkdir -p /workspace/username
cd /workspace/username
guide

Your friend: Let’s try R? DON’T JUST TYPE R!!!!! It will make my computer too slow.

srun R --no-save

You: Hope can I upload my homework?
Your friend: You can install SecureFX, but today let’s use another way to upload your file
Open a new Command Prompt or terminal. You new terminal is not connected to your friend’s computer yet, you can use ls to see everything on your computer not the server.

Your current terminal A already connected to your friend’s computer, then open terminal B.

  1. Open a new terminal B, or command prompt
  2. Upload your file via terminal B use command scp with flag -r, which means directory, to lunchbox
  3. Check your file from terminal A

More details about scp use man scp

Windows

I hope you can find your homework easily by this way. Suppose your homework is in the directory

C:\Users\WinUserName\Desktop\stat479

bash
cd /mnt/c/Users/WinUserName/Desktop
ls # you should see stat479
scp -r stat479 username@lunchbox.stat.wisc.edu:/workspace/username

Mac

Drag your homework directory into your terminal, it will show where is your homework directory.

Suppose your homework is on your Desktop directory

~/Desktop/stat479

cd ~/Desktop
scp -r stat479 username@lunchbox.stat.wisc.edu:/workspace/username

lunchbox 102 submit job

You can follow the Professor Loh’s slide to create a submit file for guide_impute.r, I will just show a simple example to write “hello world” use R.

Still in terminal A.

comments powered by Disqus