Frequently Used HDFS Commands With Examples

June 17, 2022

Frequently Used HDFS Commands With Examples

In this post there is a compilation of some of the frequently used HDFS commands with examples which can be used as reference.

All HDFS commands are invoked by the bin/hdfs script. Running the hdfs script without any arguments prints the description for all commands.

1- HDFS command to create a directory

hdfs dfs -mkdir

Example– To create a new directory input inside the /user directory

hdfs dfs -mkdir /user/input

2- HDFS command to list all the files

hdfs dfs -ls

Example- To list content of the root directory in HDFS

hdfs dfs -ls /

HDFS command to recursively list all the sub directories

hdfs dfs -ls -R /

3- HDFS command To copy file from local to HDFS

hfds dfs -copyFromLocal

Example– Copy file test.txt from local directory /usr/test to /user/input directory in HDFS

hdfs dfs -copyFromLocal /usr/test/aa.txt /user/input/

If you want overwrite existing file use -f option

hdfs dfs -copyFromLocal -f /usr/test/aa.txt /user/input/

4- put command is also used to copy file from local to HDFS

hfds dfs -put

Example– HDFS command to copy the entire directory /usr/test to HDFS directory /user/input

hdfs dfs -put /usr/test /user/input

5- To copy files with in HDFS

hdfs dfs -cp

Example- Copy file /user/input/test/aa.txt in HDFS to /user/output in HDFS

hdfs dfs -cp /user/input/test/aa.txt /user/output/

6- HDFS command to display free space

hdfs dfs -df

Example– With -h option to show output in human readable format

hdfs dfs -df -h

7- HDFS command to copy file to local file system from HDFS

hdfs dfs -copyToLocal

Example– Copy file part-r-00000 in HDFS location /user/output/ to /usr/Test

hdfs dfs -copyToLocal /user/output/part-r-00000 /usr/test

8- You can also use get command to copy file to local file system from HDFS

hdfs dfs -get

Example– Copy file part-r-00000 in HDFS location /user/output/ to /usr/Test with -f option to overwrite if exists.

hdfs dfs -get -f /user/output/part-r-00000 /home/knpcode/Documents/test

9- HDFS command to delete a file in HDFS

hdfs dfs -rm

Example– Deleting all the .txt files in /user/input/test directory

hdfs dfs -rm /user/input/test/*.txt

HDFS command to recursively delete directory /user/input/test using -R (recursive) option

hdfs dfs -rm -R /user/input/test

10- HDFS command to delete a directory.

hdfs dfs -rmdir

It will delete a directory only if it is empty.

11- To view content of a file in HDFS.

Hdfs dfs -cat

Example– HDFS command to display content of aa.txt file in directory /user/input

hdfs dfs -cat /user/input/test/aa.txt

12- To change group association of files.

hdfs dfs -chgrp

Example– Change group of /user/input/test/aa.txt file to acp.

hdfs dfs -chgrp acp /user/input/test/aa.txt

13- To Change the permissions of files in HDFS.

hdfs dfs -chmod

Permissions are same as in Linux. Read, write and execute permissions for user, group and others. if you want to provide read, write and execute permissions for all then you can use 777 as argument.

hdfs dfs -chmod 777 /user/input/test/aa.txt

You can also do the same thing using the following command.

hdfs dfs -chmod a+rwx /user/input/test/aa.txt

14- To Change the owner of files.

hadoop fs -chown

15- HDFS command to permanently delete files.

hdfs dfs -expunge

16- To display the access control lists for file and directory.

hdfs dfs -getfacl

Example– Getting owner, group and permission information for /user/input/test/aa.txt file.

hdfs dfs -getfacl /user/input/test/aa.txt

17– HDFS command to change replication factor of a file.

hdfs dfs -setrep

Example– Change the replication factor to two.

hdfs dfs -setrep 2 /user/input/test/aa.txt

18- Getting help about commands

hdfs dfs -help

19– Command to run the HDFS filesystem checking utility.

hdfs fsck

Example– Running fsck for path /user/input

hdfs fsck /user/input

20- HDFS command to print the version.

hdfs version

That's all for the topic Frequently Used HDFS Commands With Examples. If something is missing or you have something to share about the topic please write a comment.

You may also like

KnpCode

June 17, 2022

Frequently Used HDFS Commands With Examples

No comments:

Post a Comment