edu.iu.dsc.tws.data.utils

Class FileSystemUtils

java.lang.Object

edu.iu.dsc.tws.data.utils.FileSystemUtils

public final class FileSystemUtils
extends java.lang.Object

Method Summary

Modifier and Type	Method and Description
static FileSystem	get(Path path)
static FileSystem	get(Path path, Config config)
static FileSystem	get(java.net.URI uri) Returns a FileSystem for the given uri TODO: need to think about security (Flink adds a safety net here, that is skipped for now)
static FileSystem	get(java.net.URI uri, Config config)
static FileSystem	getFileSystem(java.net.URI uri) Returns a unsafe filesystem for the given uri
static FileSystem	getFileSystem(java.net.URI uri, Config config) For hadoop file system
static void	setDefaultScheme(Config config) Sets the default filesystem scheme based on the user-specified configuration parameter fs.default-scheme.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

getFileSystem

public static FileSystem getFileSystem(java.net.URI uri,
                                       Config config)
                                throws java.io.IOException

For hadoop file system

Throws:

java.io.IOException

getFileSystem

public static FileSystem getFileSystem(java.net.URI uri)
                                throws java.io.IOException

Returns a unsafe filesystem for the given uri

Throws:

java.io.IOException

setDefaultScheme

public static void setDefaultScheme(Config config)
                             throws java.io.IOException

Sets the default filesystem scheme based on the user-specified configuration parameter fs.default-scheme. By default this is set to file:/// and the local filesystem is used.

As an example, if set to hdfs://localhost:9000/, then an HDFS deployment with the namenode being on the local node and listening to port 9000 is going to be used. In this case, a file path specified as /user/USERNAME/in.txt is going to be transformed into hdfs://localhost:9000/user/USERNAME/in.txt. By default this is set to file:/// which points to the local filesystem.

Parameters:

config - the configuration from where to fetch the parameter.

Throws:

java.io.IOException

get

public static FileSystem get(java.net.URI uri)
                      throws java.io.IOException

Returns a FileSystem for the given uri TODO: need to think about security (Flink adds a safety net here, that is skipped for now)

Throws:

java.io.IOException

get

public static FileSystem get(Path path)
                      throws java.io.IOException

Throws:

java.io.IOException

get

public static FileSystem get(Path path,
                             Config config)
                      throws java.io.IOException

Throws:

java.io.IOException

get

public static FileSystem get(java.net.URI uri,
                             Config config)
                      throws java.io.IOException

Throws:

java.io.IOException