edu.iu.dsc.tws.data.hdfs

Class HadoopFileSystem

java.lang.Object

edu.iu.dsc.tws.api.data.FileSystem

edu.iu.dsc.tws.data.hdfs.HadoopFileSystem

All Implemented Interfaces:

java.io.Closeable, java.io.Serializable, java.lang.AutoCloseable

public class HadoopFileSystem
extends FileSystem
implements java.io.Closeable

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class edu.iu.dsc.tws.api.data.FileSystem

FileSystem.WriteMode

Constructor Summary

Constructors

Constructor and Description
HadoopFileSystem(org.apache.hadoop.conf.Configuration hadoopConfig, org.apache.hadoop.fs.FileSystem hadoopfileSystem)

Method Summary

Modifier and Type	Method and Description
HadoopDataOutputStream	append(Path path)
void	close()
HadoopDataOutputStream	create(Path f) Create a file system with the OVERWRITE
FSDataOutputStream	create(Path f, FileSystem.WriteMode writeMode) Create a file system with the specific write mdoe
boolean	delete(Path f, boolean recursive)
boolean	exists(Path f) Check if the given path exsits
long	getDefaultBlockSize()
BlockLocation[]	getFileBlockLocations(FileStatus file, long start, long len) Return an array containing hostnames, offset and size of portions of the given file.
FileStatus	getFileStatus(Path f) It returns the status of the file respective to the path given by the user.
org.apache.hadoop.fs.FileSystem	getHadoopFileSystem()
java.net.URI	getUri() Returns a URI whose scheme and authority identify this file system.
Path	getWorkingDirectory() Get the working Directory
void	initialize(java.net.URI name) Called after a new FileSystem instance is constructed.
boolean	isDistributedFS()
FileStatus[]	listFiles(Path f) List the statuses of the files/directories in the given path if the path is a directory.
FileStatus[]	listStatus(Path f)
boolean	mkdirs(Path f)
HadoopDataInputStream	open(Path f) This method open and return the input stream object respective to the path
HadoopDataInputStream	open(Path f, int bufferSize)
boolean	rename(Path src, Path dst)
void	setWorkingDirectory(Path path1) Set the working Directory

Methods inherited from class edu.iu.dsc.tws.api.data.FileSystem

isDirectory, isFile, isSymlink

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopFileSystem

public HadoopFileSystem(org.apache.hadoop.conf.Configuration hadoopConfig,
                        org.apache.hadoop.fs.FileSystem hadoopfileSystem)

Method Detail

getHadoopFileSystem

public org.apache.hadoop.fs.FileSystem getHadoopFileSystem()

getWorkingDirectory

public Path getWorkingDirectory()

Get the working Directory

Specified by:

getWorkingDirectory in class FileSystem

setWorkingDirectory

public void setWorkingDirectory(Path path1)

Set the working Directory

Specified by:

setWorkingDirectory in class FileSystem

getUri

public java.net.URI getUri()

Description copied from class: FileSystem

Returns a URI whose scheme and authority identify this file system.

Specified by:

getUri in class FileSystem

Returns:

a URI whose scheme and authority identify this file system

initialize

public void initialize(java.net.URI name)

Called after a new FileSystem instance is constructed.

Specified by:

initialize in class FileSystem

Parameters:

name - a URI whose authority section names the host, port, etc. for this file system

getFileStatus

public FileStatus getFileStatus(Path f)
                         throws java.io.IOException

It returns the status of the file respective to the path given by the user.

Specified by:

getFileStatus in class FileSystem

Parameters:

f - The path we want information from

Returns:

a FileStatus object

Throws:

java.io.FileNotFoundException - when the path does not exist; IOException see specific implementation

java.io.IOException

getFileBlockLocations

public BlockLocation[] getFileBlockLocations(FileStatus file,
                                             long start,
                                             long len)
                                      throws java.io.IOException

Description copied from class: FileSystem

Return an array containing hostnames, offset and size of portions of the given file. For a nonexistent file or regions, null will be returned. This call is most helpful with DFS, where it returns hostnames of machines that contain the given file. The FileSystem will simply return an elt containing 'localhost'.

Specified by:

getFileBlockLocations in class FileSystem

Throws:

java.io.IOException

open

public HadoopDataInputStream open(Path f,
                                  int bufferSize)
                           throws java.io.IOException

Specified by:

open in class FileSystem

Throws:

java.io.IOException

open

public HadoopDataInputStream open(Path f)
                           throws java.io.IOException

This method open and return the input stream object respective to the path

Specified by:

open in class FileSystem

Parameters:

f - Open an data input stream at the indicated path

Throws:

java.io.IOException

create

public HadoopDataOutputStream create(Path f)
                              throws java.io.IOException

Description copied from class: FileSystem

Create a file system with the OVERWRITE

Specified by:

create in class FileSystem

Parameters:

f - path

Returns:

the output stream

Throws:

java.io.IOException - if an error occurs

create

public FSDataOutputStream create(Path f,
                                 FileSystem.WriteMode writeMode)
                          throws java.io.IOException

Description copied from class: FileSystem

Create a file system with the specific write mdoe

Specified by:

create in class FileSystem

Parameters:

f - path

writeMode - weather overwrite or not, when creating new files

Returns:

the output stream

Throws:

java.io.IOException - if an error occurs

append

public HadoopDataOutputStream append(Path path)
                              throws java.io.IOException

Throws:

java.io.IOException

delete

public boolean delete(Path f,
                      boolean recursive)
               throws java.io.IOException

Specified by:

delete in class FileSystem

Throws:

java.io.IOException

exists

public boolean exists(Path f)
               throws java.io.IOException

Description copied from class: FileSystem

Check if the given path exsits

Overrides:

exists in class FileSystem

Throws:

java.io.IOException

listStatus

public FileStatus[] listStatus(Path f)
                        throws java.io.IOException

Specified by:

listStatus in class FileSystem

Throws:

java.io.IOException

mkdirs

public boolean mkdirs(Path f)
               throws java.io.IOException

Specified by:

mkdirs in class FileSystem

Throws:

java.io.IOException

rename

public boolean rename(Path src,
                      Path dst)
               throws java.io.IOException

Specified by:

rename in class FileSystem

Throws:

java.io.IOException

getDefaultBlockSize

public long getDefaultBlockSize()

Specified by:

getDefaultBlockSize in class FileSystem

isDistributedFS

public boolean isDistributedFS()

Specified by:

isDistributedFS in class FileSystem

listFiles

public FileStatus[] listFiles(Path f)
                       throws java.io.IOException

List the statuses of the files/directories in the given path if the path is a directory.

Specified by:

listFiles in class FileSystem

Parameters:

f - given path

Returns:

the statuses of the files/directories in the given patch

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Throws:

java.io.IOException