-appendToFile

This appends files to HDFS. These files can be local or inputs to be written on HDFS.

Have a look at the following example:

This parameter reads the input from stdin.

-cat

This displays the content of a file on stdout. The following is the syntax:

hdfs dfs –cat <file URI>

Here is an example:

hdfs dfs -cat hdfs://<namenode>/file

** -chgrp

This changes the group of a file or directory. –R is used for recursive. The syntax is:

hdfs dfs –chgrp [-R] [owner][:[group]]<URI>

Here are some examples:

hdfs dfs –chgrp –R hadoop:hadoop hdfs://namenode/dir
hdfs dfs –chgrp hadoop:hadoop hdfs://namenode/file

For permission-related information, visit http://hadoop.apache.org/docs/r1.2.1/hdfs_permissions_guide.html.

** -chmod

This changes the access mode of a file or directory. –R is used for recursive.

The syntax is as follows:

hdfs dfs –chmod <mode> <URI>

Here are some examples:

hdfs dfs –chmod 777 hdfs://namenode/filename
hdfs dfs –chmod –R 777 hdfs://namenode/directory

For more on permission and mode, visit http://hadoop.apache.org/docs/r1.2.1/hdfs_permissions_guide.html.

** -chown

This changes the owner of a directory or file, we use –R for recursive.

The syntax is as follows:

hdfs dfs –chown [–R] <user or owner:group><URI>

Here is an example:

hdfs dfs –chown –R shashwat:hadoop hdfs://namenode/directory

-copyFromLocal

This copies files from a local drive to an HDFS file system.

The syntax is as follows:

hdfs dfs –copyFromLocal <Local file/ Local Directory><URI>

Here are some examples:

hdfs dfs –copyFromLocal /user/home/Shashwat/file1 hdfs://namenode/newdir
hdfs dfs –copyFromLocal /user/home/Shashwat/dir hdfs://namenode/newdir

-copyToLocal

This copies files from HDFS to local drives. Adding ignorecrc will force it not to check crc after copying, and the crc option will print the crc details.

This is the syntax:

hdfs dfs –copyToLocal [-ignorecrc] [-crc] <URI><Local file/ Local Directory>

Here are some examples:

hdfs dfs –copyToLocal hdfs://namenode/newdir /user/home/Shashwat
hdfs dfs –copyToLocal hdfs://namenode/newdir /user/home/Shashwat

## -count

This counts the number of directories, files, and bytes under the given path. –q is also added to get quota.

The following is the syntax:

hdfs dfs –count [–q] <path>

Here's an example:

hdfs dfs –count /dir

## -cp

This is used to copy files from one HDFS location to another on the same Hadoop cluster or other Hadoop clusters.

This is the syntax:

hdfs dfs –cp <Source URI> <Destination URI>

Here are some examples:

hdfs dfs –cp /user/file1 /user/dir1/

## -du

This displays the size of the directories and files under the given path.

This is the syntax:

hdfs dfs –du [-S] [-h] <URI>

Adding options –S will give the summarized (aggregated) size, and –h will give it in a human-readable format (in MB, GB, and so on).

Here are some examples:

hdfs dfs –du -s /user/dire
hdfs dfs –du /user/dir
hdfs dfs –du –s –h /user/dir

## -dus

This is equivalent to –du –s and displays the size of directories or files as an aggregated summary.

## -expunge

When we delete files, and trash is enabled on HDFS, deleted files go to trash, but not directly deleted from HDFS. This command enables us to empty the trash.

This is the syntax:

hdfs dfs –expunge

-get

This is equivalent to –copyToLocal.

This is the syntax:

hdfs dfs –get <HDFS location> <local destination>

Here is an example:

hdfs dfs –get hdfs://namenode/dir /tmp

-getmerge

This concatenates the source and destination, and copies them to a local directory.

This is the syntax:

hdfs dfs -getmerge <src> <localdst> [addnl]

-ls

This lists out files and folders in a given path.

Here is the syntax:

hdfs dfs –ls <directory path>

This is the example:

hdfs dfs –ls /

-lsr

This lists out files and folders recursively; the syntax and uses are same as –ls.

-mkdir

This creates a directory on HDFS.

This is the syntax:

hdfs dfs –mkdir <directory path to be created>

Here are some examples:

hdfs dfs -mkdir /user/hadoop/dirtocreate
hdfs dfs -mkdir /user/hadoop/dirtocreate /user/hadoop/dirtocreate1

-moveFromLocal

This copies a file from the local directory and deletes the source file on the source path.

This is the syntax:

dfs -moveFromLocal <localsrc> <dst>

-moveToLocal

This moves the file from HDFS to a local destination.

This is the syntax:

hdfs dfs -moveToLocal [-crc] <src> <dst>

** -mv

This moves a file or directory from one HDFS location to another, either on the same cluster or to different clusters.

This is the syntax:

hdfs dfs -mv <source> <dest>

Here are some examples:

hdfs dfs -mv /user/shashwat/file /user/shashwat/file1
hdfs dfs -mv hdfs://namenode/file hdfs://namenode/file

-put

This copies a single source or multiple sources from the local file system to the destination file system. It also reads the input from stdin and writes to the destination file system.

This is the syntax:

hdfs dfs –put <source local> <destination HDFS>

Here are some examples:

hdfs dfs –put /tmp/userfilelocal hdfs://namenode/dirtarget
hdfs dfs -put - hdfs://nn.example.com/hadoop/hadoopfile

Giving – (hyphen) instead of source path will take input from stdin.

** -rm

This deletes a specified file. If the –skipTrash option is specified, the trash, if enabled, will be bypassed, and the specified file or files will be deleted immediately. This can be useful when it is necessary to delete files from an over-quota directory.

This is the syntax:

hdfs dfs -rm [-skipTrash] URI [URI …]

Here are the examples:

hdfs dfs -rm hdfs://namenode/file
hdfs dfs -rm hdfs://namenode/file hdfs://namenode/file1
hdfs dfs –rm /user/files/file

** -rmr

This deletes files/directories recursively.

This is the syntax:

hdfs dfs -rmr [-skipTrash] URI [URI …]

Here is an example:

hdfs dfs -rmr /user/shashwat/dirTodelete

## -setrep

This is a helpful command for specifying the replication of existing files on HDFS explicitly. –R is added to set the replication factor recursively.

This is the syntax:

hdfs dfs -setrep [-R] <path>

Here is an example:

hdfs dfs -setrep -w 5 -R /user/shashwat/dir

Here, –w will wait until it is replicated and -R will perform the operation recursively.

## -stat

This displays the statistics of the given argument file/directory.

This is the syntax:

hdfs dfs -stat URI [URI …]

Here is an example:

hdfs dfs -stat /user/Shashwat/file

-tail

This displays the trailing content of a file on HDFS, which is the same as the tail command in Linux. –f is added to continuously tail the file content.

This is the syntax:

hdfs dfs -tail [-f] URI

Here are some examples:

hdfs dfs -tail /user/Shashwat/file.log
hdfs dfs –tail -f /user/Shashwat/file.log

-test

This tests the condition with the following options:

This is the syntax:

hdfs dfs -test -[ezd] URI

Here are some examples:

hdfs dfs -test -e /user/Shashwat/file
hdfs dfs -test -z /user/Shashwat/file
hdfs dfs -test -d /user/Shashwat/file

## -text

hdfs dfs –cat will display the file content correctly when the file is text based. If we need to read a sequential binary file or compressed file, cat will not do, so we have to use this command. The allowed formats are ZIP and TextRecordInputStream.

This is the syntax:

hdfs dfs -text <src>

Here is an example:

hdfs dfs -text /user/Shashwat/mr/_part01

-touchz

This command creates a zero-length file on HDFS.

This is the syntax:

hdfs dfs -touchz URI [URI …]

Here are some examples:

hdfs dfs -touchz hdfs://namenode/user/Shashwat/file
hdfs dfs -touchz /user/Shashwat/file

Commands preceding ## are important admin commands.

Commands preceding ** are the commands to be used with caution as they might result in data loss.