Try Out Pig

--Local machine(local mode)

--creates copy of linux.words in local fs

cp /usr/share/dict/linux.words linux.words

-- places you in Grunt shell (Grunt is Pig's interactive shell)
pig -x local

--enter Pig latin statements from Grunt shell
grunt> A = load 'linux.words' using PigStorage() as (word:chararray);
grunt> B = filter A by SIZE(word) > 19;
grunt> dump B;
grunt> quit;

--place Pig latin statements in script
/* myscript.pig */
A = load 'linux.words' using PigStorage() as (word:chararray);
B = filter A by SIZE(word) > 19;
dump B;

-- runs the script
pig -x local myscript.pig


-- check running queue/cluster should use?
mapred queue -showacls

--Gateway machine(mapreduce mode. Change (myqueue) to (unfunded)
pig -Dmapred.job.queue.name=unfunded myscript.pig
pig -Dmapred.job.queue.name=unfunded myscript2.pig
pig -Dmapred.job.queue.name=unfunded Pig_ABF.pig

Try Out HDFS

Usage: hadoop [--config confdir] COMMAND

where COMMAND is one of:

namenode -format format the DFS filesystem

secondarynamenode run the DFS secondary namenode

namenode run the DFS namenode

datanode run a DFS datanode

dfsadmin run a DFS admin client

fsck run a DFS filesystem checking utility

fs run a generic filesystem user client

balancer run a cluster balancing utility

jobtracker run the MapReduce job Tracker node

pipes run a Pipes job

tasktracker run a MapReduce task Tracker node

job manipulate MapReduce jobs

version print the version

jar <jar> run a jar file

distcp <srcurl> <desturl> copy file or directories recursively

archive -archiveName NAME <src>* <dest> create a hadoop archive

daemonlog get/set the log level for each daemon

or

CLASSNAME run the class named CLASSNAME

Most commands print help when invoked w/o parameters.

-- Check the status of Kerberos tickets
klist

-- invoke kinit to get a ticket
kinit li@COM

-- sudo user login to get a ticket
sudo -s -h -u d_pbp
/usr/kerberos/bin/kinit -k -t /homes/dfsload/dfsload.prod.headless.keytab dfsload@COM

-- Drop a ticket
kdestroy
Hadoop HDFS Commands

--View the hadoop commands
hadoop

--View the version of hadoop
hadoop version

--View FS shell commands
hadoop fs

-- List directory
hadoop fs –ls /user/li

-- List content of HDFS files:
hadoop fs -cat wordcount/output/*
hadoop fs -cat file.bz2 | bunzip2
hadoop fs -cat dir/*.bz2 | bacat | cut -d ^A -f 125,126 | cat -v
hadoop fs -test -e dir/*.bz2 | tail -1

-- Move files or directory
hadoop fs –ls /user/li/target /user/li/dest

--Copy file or directory
hadoop fs –cp file1 file2 file3 /user/li/dest/
hadoop fs -cp /user/li/file1.txt .
hadoop fs -cp /HRBlock/hrblock.data.reduced /HRBlock/ part-r-00000-- Create HDFS directoyr
hadoop fs –mkdir /user/li/input

-- Upload from the gateway host to the HDFS home directory
hadoop fs -copyFromLocal test-data/ch1/file1.txt /user/li
hadoop fs -put test-data/ch1/file1.txt /user/li

-- Download HDFS Files
hadoop fs –get /user/li/myfile .
hadoop fs -copyToLocal hdfs://dilithiumred-nn1.red.ygrid.com:8020/projects/prod/user/20130327/SIDEBID/user/part-00998.bz2 .

-- Delete directory/file, and remove revursively(rm -rf)
hadoop fs -rm /user/li/temp
hadoop fs -rmr /user/li/temp

-- Change permissions by chown, chgrp, chmod
hadoop fs -chgrp -R users /user/li
hadoop fs -chmod 755 /user/li

-- Viewing Data from HDFS
hadoop fs -text /user/li/tmp5/000000_0.deflate | tr '\001' ',' | head -1
hadoop fs -cat path_to_data/*.bz2 | bzcat | cut -d ^A -f 125,126  | cat -v
This selects columns 125 and 126 from Ctrl-A separated data.
To create the ^A, type:  CTRL-v CTRL-a
Inside screen, type:  CTRL-v CTRL-a a

--Transfer data between clusters
hadoop distcp -Dmapred.job.queue.name=adhoc -Ddfs.umaskmode=002 -i -m 40 -update webhdfs://ygrid.yahoo.com/user/li/search_20131211 hdfs://ygrid.yahoo.com/user/li/search_20131211

--Kill a job
mapred job -list |tail -n+3 |awk {'print $1" "$4'} |grep 'li'
mapred job -kill job_1399615563645_524816

--View job logs
mapred job -logs job_1374774840603_3324664
mapred job -logs job_1387925060187_4840299

-- Show available queues
mapred queue -showacls
mapred queue -list
mapred queue -info apg_dailymedium_p5
The meanings are percentages or fractions:
Capacity = % of the grid’s total capacity used by this queue under normal usage. If you add up the capacities for all queues you will get 100%.
MaximumCapacity =Fraction of the grid’s total capacity this queue can use. In this example, the p5 queue normally uses 5%, but it’s allowed to go as high as 40% of the total grid capacity.
CurrentCapacity = Current usage relative to Capacity. In this example, p5 is using 144% of its Capacity, or 7.2% of the total grid capacity.

-- Check running job list/Show number of jobs per queue for each user
mapred job -list
mapred job -list |tail -n+3 |awk {'print $5" "$4'} |sort |uniq –c

-- Another way to list jobs, both running and completed:
mapred queue -info apg_d**_p3 -showJobs

-- Check Gateway Quota
quota -u apoqa

-- Check HDFS Quotas, aka get count of objests
hadoop fs -count -q /projects/DSP
hadoop fs -count -q /user/li

-- Displays aggregate length of files contained in the directory.
hadoop fs –dus /user/li
hadoop fs -du hdfs://.com:8020/projects

-- Check the running processes
jps

-- Check is file / is zero /is dir
hadoop fs -test -e /user/li/
hadoop fs -test -z /user/li/
hadoop fs -test -d /user/li/

-- Check group members
/gridtools/generic/bin/showmembers –n GROUPNAME
/gridtools/gneric/bin/showmembers -n awrgroup
showmembers --netgroup cp_pnp_c_sudoers --type user --format comma

-- Launch R
echo USER=***
export INSTALL_ROOT=/homes/$USER/custom_root
/homes/$USER/custom_root/bin/R

-- Launch Pig
pig -Dmapred.job.queue.name=***  \
-Dmapreduce.reduce.memory.mb=3072 \
-Dmapreduce.map.memory.mb=3072 \
-Dmapreduce.map.java.opts="-Xmx2048M" \
-Dmapreduce.map.speculative=true \
-Dmapreduce.job.acl-view-job=* \
-Dmapreduce.task.timeout=1800000 \
-Dmapreduce.reduce.speculative=true \
-Dmapreduce.output.fileoutputformat.compress=true \
-param PARALLEL_ORDER=512  \
***_MB3.pig

-- Hadoop Streaming

hadoop jar $HADOOP_PREFIX/share/hadoop/tools/lib/hadoop-streaming.jar \
        -input /ngrams \
        -output /output-streaming \
        -mapper mapper.py \
        -combiner reducer.py \
        -reducer reducer.py \
        -jobconf stream.num.map.output.key.fields=3 \
        -jobconf stream.num.reduce.output.key.fields=3 \
        -jobconf mapred.reduce.tasks=10 \
        -file mapper.py \
        -file reducer.py

Transform to VW format

import sys
import string

for line in sys.stdin:
    line = line.strip()
    toks = line.split('\t')
    pline =  toks[0].strip() + " | "
    #continuous
#    for i in range(1, sys.argv[3]):
    for i in range(1, 2):
        if len(toks[i].strip()) == 0 :
            continue
        pline = pline + str(i) + ":" + toks[i].strip() + str('\t')
    #categorical
    for i in range(2, len(toks)):
        if len(toks[i].strip()) == 0 :
            continue
        pline = pline + str(i) + "cell" + toks[i].strip() + ":1" + str('\t')
    print pline
 
hadoop  jar hadoop-streaming.jar \
-input $1 \
-output $2 \
-mapper "python csv2vm2.py" \
-reducer NONE \
-file csv2vm2.py \
-jobconf mapred.reduce.tasks=5 \
-jobconf mapred.job.queue.name=***;

./csv2vm.sh /train1/* /vwtrain1/
./csv2vm.sh /test1/* /vwtest1/

Running Mincemeat Example on Windows

Lightweight MapReduce in python: https://github.com/michaelfairley/mincemeatpy.

client

E:\Python27>python example.py

server

E:\Python27>python mincemeat.py -p changeme localhost

Word count example

import glob
import mincemeat

#text_files=glob.glob('hw3data/*')
text_files=glob.glob('hw3data/c0001')
print(text_files)

def file_contents(file_name):
    f=open(file_name,'rb')
    try:
        return f.read()
    finally:
        f.close()

source=dict((file_name,file_contents(file_name))
    for file_name in text_files)

# setup map and reduce functions
def mapfn(key,value):
      for line in value.splitlines():
          for word in line.split():
               yield word.lower(),1

def reducefn(key,value):
       return key,len(value)  
  
# start the server
s =    mincemeat.Server()
s.datasource = source
s.mapfn = mapfn
s.reducefn = reducefn

results = s.run_server(password="changeme")
print results

Install VM for Data Science Class

go here

https://my.vmware.com/web/vmware/free#desktop_end_user_computing/vmware_player/5_0

download and install VMWare Player

then download the .ova file from the spark-public link you posted
it will be approximately 2,451,525 bytes or 2.33 GB or 2,510,something bytes

then start VMWare Player and select Open Virtual Machine
select the .ova file you downloaded

it WILL give you an error

it might say saying "invalid /corrupted file"
in which case the file didn't completely download
so you'll have to try and download it again

but even if it downloads completely and correctly, it will say "file didn't pass .ofv specification, try again"

for the 2nd message, just hit ok and it will try again
it will take a few seconds while it sets up

then a 800 x 600 linux window will appear

and you're good to go.

Since the example won’t work on Windows, you’ll need to resort to using Powershell to get the first 10 lines. Here’s how you do it.
Open Powershell – click Start – in Search for programs and files, type “Powershell”. You’ll see it come up on your list of apps.
Launch Powershell and type the following:

1	Get-Content -Path output.json | select-object -first 20 > output.tx

Hadoop 20S VM

1 Install VM player
- www.vmware.com

2 Copy Hadoop 0.20.S
- copy the Hadoop 0.20.S VM into a location on hard drive

3 VM User Account
id: hadoop-***
psw: ****

4 Get authentication token
kinit

psw: hadoop******

5 Shutting down the VM

sudo poweroff

--View the hadoop commands 
hadoop

--View the version of hadoop 
hadoop version

R with Hadoop

1 R + Streaming
 With this approach, you use MapReduce to execute R scripts in the map and reduce phrases.
The R package needs to be installed on each Data-Node, but packages are available on pubicly available Yum repositories for easy installation.

2 RHipe

 Rhipe is an open source priject which allows MapReduce to be closely integrated with R on the client side.An R package that integrates the R environment with Hadoop, the open source implementation of Google’s MapReduce. 
Using Rhipe, it is possible to write MapReduce algorithms in R, launch and monitor MapReduce jobs from R and interact with the HDFS.
R must be installed on each Data-Node, in conjunction with Protocal Buffers, and Rhipe itself. 

3 RHadoop
RHadoop like Rhipe, provides an R wrapper around Map-Reduce so that they can be seamlessly integrated on the client side.
R must be installed on each Data-Node, and RHadoop has dependencies on other R packages. But these packages can be installled with CRAN, and the RHadoop installlation,, while not via CRAN, is straight-forward.

4 RHive

RHive is an R extension facilitating distributed computing via HIVE query. It provides an easy to use HQL like SQL and R objects and functions in HQL. It requires Hadoop core and Hive system.

5 Segue

An R language segue into parallel processing on Amazon’s Web Serives (in the cloud). Not a full map/reduce framework for R. Currently runs on Mac or Linux.