Skip navigation links
A B C D E F G H I M N O P R S T U V W X 

A

abortJob(JobContext, JobStatus.State) - Method in class org.apache.hadoop.tools.mapred.CopyCommitter
ACLS_NOT_SUPPORTED - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
AclsNotSupportedException(String) - Constructor for exception org.apache.hadoop.tools.CopyListing.AclsNotSupportedException
 
addToConf(Configuration, DistCpOptionSwitch, String) - Static method in enum org.apache.hadoop.tools.DistCpOptionSwitch
Helper function to add an option to hadoop configuration object
addToConf(Configuration, DistCpOptionSwitch) - Static method in enum org.apache.hadoop.tools.DistCpOptionSwitch
Helper function to set an option to hadoop configuration object
appendToConf(Configuration) - Method in class org.apache.hadoop.tools.DistCpOptions
Add options to configuration.

B

buildListing(Path, DistCpOptions) - Method in class org.apache.hadoop.tools.CopyListing
Build listing function creates the input listing that distcp uses to perform the copy.

C

checkFileSystemAclSupport(FileSystem) - Static method in class org.apache.hadoop.tools.util.DistCpUtils
Determines if a file system supports ACLs by running a canary getAclStatus request on the file system root.
checkFileSystemXAttrSupport(FileSystem) - Static method in class org.apache.hadoop.tools.util.DistCpUtils
Determines if a file system supports XAttrs by running a getXAttrs request on the file system root.
checkOutputSpecs(JobContext) - Method in class org.apache.hadoop.tools.mapred.CopyOutputFormat
checksumsAreEqual(FileSystem, Path, FileChecksum, FileSystem, Path) - Static method in class org.apache.hadoop.tools.util.DistCpUtils
Utility to compare checksums for the paths specified.
clone() - Method in class org.apache.hadoop.tools.DistCpOptions
 
close() - Method in class org.apache.hadoop.tools.mapred.lib.DynamicRecordReader
Implementation of RecordReader::close().
close() - Method in class org.apache.hadoop.tools.util.ThrottledInputStream
 
commitJob(JobContext) - Method in class org.apache.hadoop.tools.mapred.CopyCommitter
compareFs(FileSystem, FileSystem) - Static method in class org.apache.hadoop.tools.util.DistCpUtils
 
CONF_LABEL_APPEND - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_ATOMIC_COPY - Static variable in class org.apache.hadoop.tools.DistCpConstants
Constants mapping to command line switches/input options
CONF_LABEL_BANDWIDTH_MB - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_COPY_LISTING_CLASS - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_COPY_STRATEGY - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_DELETE_MISSING - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_DIFF - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_DISTCP_JOB_ID - Static variable in class org.apache.hadoop.tools.DistCpConstants
DistCp job id for consumers of the Disctp
CONF_LABEL_IGNORE_FAILURES - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_LISTING_FILE_PATH - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_LOG_PATH - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_MAX_CHUNKS_IDEAL - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_MAX_CHUNKS_TOLERABLE - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_MAX_MAPS - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_META_FOLDER - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_MIN_RECORDS_PER_CHUNK - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_OVERWRITE - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_PRESERVE_RAWXATTRS - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_PRESERVE_STATUS - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_SKIP_CRC - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_SOURCE_LISTING - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_SPLIT_RATIO - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_SSL_CONF - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_SSL_KEY_STORE_LOCATION - Static variable in class org.apache.hadoop.tools.DistCpConstants
Conf label for SSL Key-store location.
CONF_LABEL_SSL_KEYSTORE - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_SSL_TRUST_STORE_LOCATION - Static variable in class org.apache.hadoop.tools.DistCpConstants
Conf label for SSL Trust-store location.
CONF_LABEL_SYNC_FOLDERS - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_TARGET_FINAL_PATH - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_TARGET_PATH_EXISTS - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_TARGET_WORK_PATH - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_TOTAL_BYTES_TO_BE_COPIED - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_TOTAL_NUMBER_OF_RECORDS - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CONF_LABEL_WORK_PATH - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
CopyCommitter - Class in org.apache.hadoop.tools.mapred
The CopyCommitter class is DistCp's OutputCommitter implementation.
CopyCommitter(Path, TaskAttemptContext) - Constructor for class org.apache.hadoop.tools.mapred.CopyCommitter
Create a output committer
CopyListing - Class in org.apache.hadoop.tools
The CopyListing abstraction is responsible for how the list of sources and targets is constructed, for DistCp's copy function.
CopyListing(Configuration, Credentials) - Constructor for class org.apache.hadoop.tools.CopyListing
Protected constructor, to initialize configuration.
CopyListing.AclsNotSupportedException - Exception in org.apache.hadoop.tools
 
CopyListing.XAttrsNotSupportedException - Exception in org.apache.hadoop.tools
 
CopyListingFileStatus - Class in org.apache.hadoop.tools
CopyListingFileStatus is a specialized subclass of FileStatus for attaching additional data members useful to distcp.
CopyListingFileStatus() - Constructor for class org.apache.hadoop.tools.CopyListingFileStatus
Default constructor.
CopyListingFileStatus(FileStatus) - Constructor for class org.apache.hadoop.tools.CopyListingFileStatus
Creates a new CopyListingFileStatus by copying the members of the given FileStatus.
CopyMapper - Class in org.apache.hadoop.tools.mapred
Mapper class that executes the DistCp copy operation.
CopyMapper() - Constructor for class org.apache.hadoop.tools.mapred.CopyMapper
 
CopyMapper.Counter - Enum in org.apache.hadoop.tools.mapred
Hadoop counters for the DistCp CopyMapper.
CopyOutputFormat<K,V> - Class in org.apache.hadoop.tools.mapred
The CopyOutputFormat is the Hadoop OutputFormat used in DistCp.
CopyOutputFormat() - Constructor for class org.apache.hadoop.tools.mapred.CopyOutputFormat
 
CopyReadException(Throwable) - Constructor for exception org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.CopyReadException
 
createAndSubmitJob() - Method in class org.apache.hadoop.tools.DistCp
Create and submit the mapreduce job.
createInputFileListing(Job) - Method in class org.apache.hadoop.tools.DistCp
Create input listing by invoking an appropriate copy listing implementation.
createRecordReader(InputSplit, TaskAttemptContext) - Method in class org.apache.hadoop.tools.mapred.lib.DynamicInputFormat
Implementation of Inputformat::createRecordReader().
createRecordReader(InputSplit, TaskAttemptContext) - Method in class org.apache.hadoop.tools.mapred.UniformSizeInputFormat
Implementation of InputFormat::createRecordReader().

D

DEFAULT_BANDWIDTH_MB - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
DEFAULT_MAPS - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
description - Variable in class org.apache.hadoop.tools.util.RetriableCommand
 
disableUsingDiff() - Method in class org.apache.hadoop.tools.DistCpOptions
 
DistCp - Class in org.apache.hadoop.tools
DistCp is the main driver-class for DistCpV2.
DistCp(Configuration, DistCpOptions) - Constructor for class org.apache.hadoop.tools.DistCp
Public Constructor.
DistCpConstants - Class in org.apache.hadoop.tools
Utility class to hold commonly used constants.
DistCpConstants() - Constructor for class org.apache.hadoop.tools.DistCpConstants
 
DistCpOptions - Class in org.apache.hadoop.tools
The Options class encapsulates all DistCp options.
DistCpOptions(List<Path>, Path) - Constructor for class org.apache.hadoop.tools.DistCpOptions
Constructor, to initialize source/target paths.
DistCpOptions(Path, Path) - Constructor for class org.apache.hadoop.tools.DistCpOptions
Constructor, to initialize source/target paths.
DistCpOptions(DistCpOptions) - Constructor for class org.apache.hadoop.tools.DistCpOptions
Copy constructor.
DistCpOptions.FileAttribute - Enum in org.apache.hadoop.tools
 
DistCpOptionSwitch - Enum in org.apache.hadoop.tools
Enumeration mapping configuration keys to distcp command line options.
DistCpUtils - Class in org.apache.hadoop.tools.util
Utility functions used in DistCp.
DistCpUtils() - Constructor for class org.apache.hadoop.tools.util.DistCpUtils
 
doBuildListing(Path, DistCpOptions) - Method in class org.apache.hadoop.tools.CopyListing
The interface to be implemented by sub-classes, to create the source/target file listing.
doBuildListing(Path, DistCpOptions) - Method in class org.apache.hadoop.tools.FileBasedCopyListing
Implementation of CopyListing::buildListing().
doBuildListing(Path, DistCpOptions) - Method in class org.apache.hadoop.tools.GlobbedCopyListing
Implementation of CopyListing::buildListing().
doBuildListing(Path, DistCpOptions) - Method in class org.apache.hadoop.tools.SimpleCopyListing
The interface to be implemented by sub-classes, to create the source/target file listing.
doBuildListing(SequenceFile.Writer, DistCpOptions) - Method in class org.apache.hadoop.tools.SimpleCopyListing
Collect the list of <sourceRelativePath, sourceFileStatus> to be copied and write to the sequence file.
doExecute(Object...) - Method in class org.apache.hadoop.tools.mapred.RetriableDirectoryCreateCommand
Implementation of RetriableCommand::doExecute().
doExecute(Object...) - Method in class org.apache.hadoop.tools.mapred.RetriableFileCopyCommand
Implementation of RetriableCommand::doExecute().
doExecute(Object...) - Method in class org.apache.hadoop.tools.util.RetriableCommand
Implement this interface-method define the command-logic that will be retried on failure (i.e.
DUPLICATE_INPUT - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
DynamicInputFormat<K,V> - Class in org.apache.hadoop.tools.mapred.lib
DynamicInputFormat implements the "Worker pattern" for DistCp.
DynamicInputFormat() - Constructor for class org.apache.hadoop.tools.mapred.lib.DynamicInputFormat
 
DynamicRecordReader<K,V> - Class in org.apache.hadoop.tools.mapred.lib
The DynamicRecordReader is used in conjunction with the DynamicInputFormat to implement the "Worker pattern" for DistCp.
DynamicRecordReader() - Constructor for class org.apache.hadoop.tools.mapred.lib.DynamicRecordReader
 

E

equals(Object) - Method in class org.apache.hadoop.tools.CopyListingFileStatus
 
execute() - Method in class org.apache.hadoop.tools.DistCp
Implements the core-execution.
execute(Object...) - Method in class org.apache.hadoop.tools.util.RetriableCommand
The execute() method invokes doExecute() until either: 1.

F

FileBasedCopyListing - Class in org.apache.hadoop.tools
FileBasedCopyListing implements the CopyListing interface, to create the copy-listing for DistCp, by iterating over all source paths mentioned in a specified input-file.
FileBasedCopyListing(Configuration, Credentials) - Constructor for class org.apache.hadoop.tools.FileBasedCopyListing
Constructor, to initialize base-class.

G

getAcl(FileSystem, FileStatus) - Static method in class org.apache.hadoop.tools.util.DistCpUtils
Returns a file's full logical ACL.
getAclEntries() - Method in class org.apache.hadoop.tools.CopyListingFileStatus
Returns the full logical ACL.
getAtomicWorkPath() - Method in class org.apache.hadoop.tools.DistCpOptions
Get work path for atomic commit.
getAttribute(char) - Static method in enum org.apache.hadoop.tools.DistCpOptions.FileAttribute
 
getBytesPerSec() - Method in class org.apache.hadoop.tools.util.ThrottledInputStream
Getter for the read-rate from this stream, since creation.
getBytesToCopy() - Method in class org.apache.hadoop.tools.CopyListing
Return the total bytes that distCp should copy for the source paths This doesn't consider whether file is same should be skipped during copy
getBytesToCopy() - Method in class org.apache.hadoop.tools.FileBasedCopyListing
Return the total bytes that distCp should copy for the source paths This doesn't consider whether file is same should be skipped during copy
getBytesToCopy() - Method in class org.apache.hadoop.tools.GlobbedCopyListing
Return the total bytes that distCp should copy for the source paths This doesn't consider whether file is same should be skipped during copy
getBytesToCopy() - Method in class org.apache.hadoop.tools.SimpleCopyListing
Return the total bytes that distCp should copy for the source paths This doesn't consider whether file is same should be skipped during copy
getCommitDirectory(Job) - Static method in class org.apache.hadoop.tools.mapred.CopyOutputFormat
Getter for the final commit-directory.
getConfigLabel() - Method in enum org.apache.hadoop.tools.DistCpOptionSwitch
Get Configuration label for the option
getCopyListing(Configuration, Credentials, DistCpOptions) - Static method in class org.apache.hadoop.tools.CopyListing
Public Factory method with which the appropriate CopyListing implementation may be retrieved.
getCopyStrategy() - Method in class org.apache.hadoop.tools.DistCpOptions
Get the copy strategy to use.
getCredentials() - Method in class org.apache.hadoop.tools.CopyListing
get credentials to update the delegation tokens for accessed FS objects
getCurrentKey() - Method in class org.apache.hadoop.tools.mapred.lib.DynamicRecordReader
Implementation of RecordReader::getCurrentKey().
getCurrentValue() - Method in class org.apache.hadoop.tools.mapred.lib.DynamicRecordReader
Implementation of RecordReader::getCurrentValue().
getFileListingPath() - Method in class org.apache.hadoop.tools.DistCp
Get default name of the copy listing file.
getFileSize(Path, Configuration) - Static method in class org.apache.hadoop.tools.util.DistCpUtils
Retrieves size of the file at the specified path.
getFormatter() - Static method in class org.apache.hadoop.tools.util.DistCpUtils
 
getFromSnapshot() - Method in class org.apache.hadoop.tools.DistCpOptions
 
getInt(Configuration, String) - Static method in class org.apache.hadoop.tools.util.DistCpUtils
Utility to retrieve a specified key from a Configuration.
getLogPath() - Method in class org.apache.hadoop.tools.DistCpOptions
Get output directory for writing distcp logs.
getLong(Configuration, String) - Static method in class org.apache.hadoop.tools.util.DistCpUtils
Utility to retrieve a specified key from a Configuration.
getMapBandwidth() - Method in class org.apache.hadoop.tools.DistCpOptions
Get the map bandwidth in MB
getMaxMaps() - Method in class org.apache.hadoop.tools.DistCpOptions
Get the max number of maps to use for this copy
getNumberOfPaths() - Method in class org.apache.hadoop.tools.CopyListing
Return the total number of paths to distcp, includes directories as well This doesn't consider whether file/dir is already present and should be skipped during copy
getNumberOfPaths() - Method in class org.apache.hadoop.tools.FileBasedCopyListing
Return the total number of paths to distcp, includes directories as well This doesn't consider whether file/dir is already present and should be skipped during copy
getNumberOfPaths() - Method in class org.apache.hadoop.tools.GlobbedCopyListing
Return the total number of paths to distcp, includes directories as well This doesn't consider whether file/dir is already present and should be skipped during copy
getNumberOfPaths() - Method in class org.apache.hadoop.tools.SimpleCopyListing
Return the total number of paths to distcp, includes directories as well This doesn't consider whether file/dir is already present and should be skipped during copy
getOption() - Method in enum org.apache.hadoop.tools.DistCpOptionSwitch
Get CLI Option corresponding to the distcp option
getOutputCommitter(TaskAttemptContext) - Method in class org.apache.hadoop.tools.mapred.CopyOutputFormat
getProgress() - Method in class org.apache.hadoop.tools.mapred.lib.DynamicRecordReader
Implementation of RecordReader::getProgress().
getRelativePath(Path, Path) - Static method in class org.apache.hadoop.tools.util.DistCpUtils
Gets relative path of child path with respect to a root path For ex.
getSourceFileListing() - Method in class org.apache.hadoop.tools.DistCpOptions
File path (hdfs:// or file://) that contains the list of actual files to copy
getSourcePaths() - Method in class org.apache.hadoop.tools.DistCpOptions
Getter for sourcePaths.
getSplits(JobContext) - Method in class org.apache.hadoop.tools.mapred.lib.DynamicInputFormat
Implementation of InputFormat::getSplits().
getSplits(JobContext) - Method in class org.apache.hadoop.tools.mapred.UniformSizeInputFormat
Implementation of InputFormat::getSplits().
getSslConfigurationFile() - Method in class org.apache.hadoop.tools.DistCpOptions
Get path where the ssl configuration file is present to use for hftps://
getStrategy(Configuration, DistCpOptions) - Static method in class org.apache.hadoop.tools.util.DistCpUtils
Returns the class that implements a copy strategy.
getStringDescriptionFor(long) - Static method in class org.apache.hadoop.tools.util.DistCpUtils
 
getSwitch() - Method in enum org.apache.hadoop.tools.DistCpOptionSwitch
Get Switch symbol
getTargetPath() - Method in class org.apache.hadoop.tools.DistCpOptions
Getter for the targetPath.
getTargetPathExists() - Method in class org.apache.hadoop.tools.DistCpOptions
Getter for the targetPathExists.
getToSnapshot() - Method in class org.apache.hadoop.tools.DistCpOptions
 
getTotalBytesRead() - Method in class org.apache.hadoop.tools.util.ThrottledInputStream
Getter for the number of bytes read from this stream, since creation.
getTotalSleepTime() - Method in class org.apache.hadoop.tools.util.ThrottledInputStream
Getter the total time spent in sleep.
getWorkingDirectory(Job) - Static method in class org.apache.hadoop.tools.mapred.CopyOutputFormat
Getter for the working directory.
getXAttrs() - Method in class org.apache.hadoop.tools.CopyListingFileStatus
Returns all xAttrs.
getXAttrs(FileSystem, Path) - Static method in class org.apache.hadoop.tools.util.DistCpUtils
Returns a file's all xAttrs.
GlobbedCopyListing - Class in org.apache.hadoop.tools
GlobbedCopyListing implements the CopyListing interface, to create the copy listing-file by "globbing" all specified source paths (wild-cards and all.)
GlobbedCopyListing(Configuration, Credentials) - Constructor for class org.apache.hadoop.tools.GlobbedCopyListing
Constructor, to initialize the configuration.

H

hashCode() - Method in class org.apache.hadoop.tools.CopyListingFileStatus
 

I

initialize(InputSplit, TaskAttemptContext) - Method in class org.apache.hadoop.tools.mapred.lib.DynamicRecordReader
Implementation for RecordReader::initialize().
INVALID_ARGUMENT - Static variable in class org.apache.hadoop.tools.DistCpConstants
 

M

main(String[]) - Static method in class org.apache.hadoop.tools.DistCp
Main function of the DistCp program.
map(Text, CopyListingFileStatus, Mapper<Text, CopyListingFileStatus, Text, Text>.Context) - Method in class org.apache.hadoop.tools.mapred.CopyMapper
Implementation of the Mapper::map().
MAX_CHUNKS_IDEAL_DEFAULT - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
MAX_CHUNKS_TOLERABLE_DEFAULT - Static variable in class org.apache.hadoop.tools.DistCpConstants
Constants for DistCp default values of configurable values
MIN_RECORDS_PER_CHUNK_DEFAULT - Static variable in class org.apache.hadoop.tools.DistCpConstants
 

N

nextKeyValue() - Method in class org.apache.hadoop.tools.mapred.lib.DynamicRecordReader
Implementation of RecordReader::nextValue().

O

OptionsParser - Class in org.apache.hadoop.tools
The OptionsParser parses out the command-line options passed to DistCp, and interprets those specific to DistCp, to create an Options object.
OptionsParser() - Constructor for class org.apache.hadoop.tools.OptionsParser
 
org.apache.hadoop.tools - package org.apache.hadoop.tools
 
org.apache.hadoop.tools.mapred - package org.apache.hadoop.tools.mapred
 
org.apache.hadoop.tools.mapred.lib - package org.apache.hadoop.tools.mapred.lib
 
org.apache.hadoop.tools.util - package org.apache.hadoop.tools.util
 

P

packAttributes(EnumSet<DistCpOptions.FileAttribute>) - Static method in class org.apache.hadoop.tools.util.DistCpUtils
Pack file preservation attributes into a string, containing just the first character of each preservation attribute
parse(String[]) - Static method in class org.apache.hadoop.tools.OptionsParser
The parse method parses the command-line options, and creates a corresponding Options object.
preserve(DistCpOptions.FileAttribute) - Method in class org.apache.hadoop.tools.DistCpOptions
Add file attributes that need to be preserved.
preserve(FileSystem, Path, CopyListingFileStatus, EnumSet<DistCpOptions.FileAttribute>, boolean) - Static method in class org.apache.hadoop.tools.util.DistCpUtils
Preserve attribute on file matching that of the file status being sent as argument.
PRESERVE_STATUS_DEFAULT - Static variable in enum org.apache.hadoop.tools.DistCpOptionSwitch
 
preserveAttributes() - Method in class org.apache.hadoop.tools.DistCpOptions
Returns an iterator with the list of file attributes to preserve
preserveRawXattrs() - Method in class org.apache.hadoop.tools.DistCpOptions
Indicate that raw.* xattrs should be preserved
publish(Configuration, String, T) - Static method in class org.apache.hadoop.tools.util.DistCpUtils
Utility to publish a value to a configuration.

R

read() - Method in class org.apache.hadoop.tools.util.ThrottledInputStream
read(byte[]) - Method in class org.apache.hadoop.tools.util.ThrottledInputStream
read(byte[], int, int) - Method in class org.apache.hadoop.tools.util.ThrottledInputStream
read(long, byte[], int, int) - Method in class org.apache.hadoop.tools.util.ThrottledInputStream
Read bytes starting from the specified position.
readFields(DataInput) - Method in class org.apache.hadoop.tools.CopyListingFileStatus
 
RetriableCommand - Class in org.apache.hadoop.tools.util
This class represents commands that be retried on failure, in a configurable manner.
RetriableCommand(String) - Constructor for class org.apache.hadoop.tools.util.RetriableCommand
Constructor.
RetriableCommand(String, RetryPolicy) - Constructor for class org.apache.hadoop.tools.util.RetriableCommand
Constructor.
RetriableDirectoryCreateCommand - Class in org.apache.hadoop.tools.mapred
This class extends Retriable command to implement the creation of directories with retries on failure.
RetriableDirectoryCreateCommand(String) - Constructor for class org.apache.hadoop.tools.mapred.RetriableDirectoryCreateCommand
Constructor, taking a description of the action.
RetriableFileCopyCommand - Class in org.apache.hadoop.tools.mapred
This class extends RetriableCommand to implement the copy of files, with retries on failure.
RetriableFileCopyCommand(String, CopyMapper.FileAction) - Constructor for class org.apache.hadoop.tools.mapred.RetriableFileCopyCommand
Constructor, taking a description of the action.
RetriableFileCopyCommand(boolean, String, CopyMapper.FileAction) - Constructor for class org.apache.hadoop.tools.mapred.RetriableFileCopyCommand
Create a RetriableFileCopyCommand.
RetriableFileCopyCommand.CopyReadException - Exception in org.apache.hadoop.tools.mapred
Special subclass of IOException.
run(String[]) - Method in class org.apache.hadoop.tools.DistCp
Implementation of Tool::run().

S

setAclEntries(List<AclEntry>) - Method in class org.apache.hadoop.tools.CopyListingFileStatus
Sets optional ACL entries.
setAppend(boolean) - Method in class org.apache.hadoop.tools.DistCpOptions
Set if we want to append new data to target files.
setAtomicCommit(boolean) - Method in class org.apache.hadoop.tools.DistCpOptions
Set if data need to be committed automatically
setAtomicWorkPath(Path) - Method in class org.apache.hadoop.tools.DistCpOptions
Set the work path for atomic commit
setBlocking(boolean) - Method in class org.apache.hadoop.tools.DistCpOptions
Set if Disctp should run blocking or non-blocking
setCommitDirectory(Job, Path) - Static method in class org.apache.hadoop.tools.mapred.CopyOutputFormat
Setter for the final directory for DistCp (where files copied will be moved, atomically.)
setCopyStrategy(String) - Method in class org.apache.hadoop.tools.DistCpOptions
Set the copy strategy to use.
setCredentials(Credentials) - Method in class org.apache.hadoop.tools.CopyListing
set Credentials store, on which FS delegatin token will be cached
setDeleteMissing(boolean) - Method in class org.apache.hadoop.tools.DistCpOptions
Set if files only present in target should be deleted
setIgnoreFailures(boolean) - Method in class org.apache.hadoop.tools.DistCpOptions
Set if failures during copy be ignored
setLogPath(Path) - Method in class org.apache.hadoop.tools.DistCpOptions
Set the log path where distcp output logs are stored Uses JobStagingDir/_logs by default
setMapBandwidth(int) - Method in class org.apache.hadoop.tools.DistCpOptions
Set per map bandwidth
setMaxMaps(int) - Method in class org.apache.hadoop.tools.DistCpOptions
Set the max number of maps to use for copy
setOverwrite(boolean) - Method in class org.apache.hadoop.tools.DistCpOptions
Set if files should always be overwritten on target
setRetryPolicy(RetryPolicy) - Method in class org.apache.hadoop.tools.util.RetriableCommand
Fluent-interface to change the RetryHandler.
setSkipCRC(boolean) - Method in class org.apache.hadoop.tools.DistCpOptions
Set if checksum comparison should be skipped while determining if source and destination files are identical
setSourcePaths(List<Path>) - Method in class org.apache.hadoop.tools.DistCpOptions
Setter for sourcePaths.
setSslConfigurationFile(String) - Method in class org.apache.hadoop.tools.DistCpOptions
Set the SSL configuration file path to use with hftps:// (local path)
setSyncFolder(boolean) - Method in class org.apache.hadoop.tools.DistCpOptions
Set if source and target folder contents be sync'ed up
setTargetPathExists(boolean) - Method in class org.apache.hadoop.tools.DistCpOptions
Set targetPathExists.
setup(Mapper<Text, CopyListingFileStatus, Text, Text>.Context) - Method in class org.apache.hadoop.tools.mapred.CopyMapper
Implementation of the Mapper::setup() method.
setUseDiff(boolean, String, String) - Method in class org.apache.hadoop.tools.DistCpOptions
 
setWorkingDirectory(Job, Path) - Static method in class org.apache.hadoop.tools.mapred.CopyOutputFormat
Setter for the working directory for DistCp (where files will be copied before they are moved to the final commit-directory.)
setXAttrs(Map<String, byte[]>) - Method in class org.apache.hadoop.tools.CopyListingFileStatus
Sets optional xAttrs.
shouldAppend() - Method in class org.apache.hadoop.tools.DistCpOptions
 
shouldAtomicCommit() - Method in class org.apache.hadoop.tools.DistCpOptions
Should the data be committed atomically?
shouldBlock() - Method in class org.apache.hadoop.tools.DistCpOptions
Should DistCp be running in blocking mode
shouldCopy(Path, DistCpOptions) - Method in class org.apache.hadoop.tools.SimpleCopyListing
Provide an option to skip copy of a path, Allows for exclusion of files such as FileOutputCommitter.SUCCEEDED_FILE_NAME
shouldDeleteMissing() - Method in class org.apache.hadoop.tools.DistCpOptions
Should target files missing in source should be deleted?
shouldIgnoreFailures() - Method in class org.apache.hadoop.tools.DistCpOptions
Should failures be logged and ignored during copy?
shouldOverwrite() - Method in class org.apache.hadoop.tools.DistCpOptions
Should files be overwritten always?
shouldPreserve(DistCpOptions.FileAttribute) - Method in class org.apache.hadoop.tools.DistCpOptions
Checks if the input attribute should be preserved or not
shouldPreserveRawXattrs() - Method in class org.apache.hadoop.tools.DistCpOptions
Return true if raw.* xattrs should be preserved.
shouldSkipCRC() - Method in class org.apache.hadoop.tools.DistCpOptions
Should CRC/checksum check be skipped while checking files are identical
shouldSyncFolder() - Method in class org.apache.hadoop.tools.DistCpOptions
Should the data be sync'ed between source and target paths?
shouldUseDiff() - Method in class org.apache.hadoop.tools.DistCpOptions
 
SimpleCopyListing - Class in org.apache.hadoop.tools
The SimpleCopyListing is responsible for making the exhaustive list of all files/directories under its specified list of input-paths.
SimpleCopyListing(Configuration, Credentials) - Constructor for class org.apache.hadoop.tools.SimpleCopyListing
Protected constructor, to initialize configuration.
sortListing(FileSystem, Configuration, Path) - Static method in class org.apache.hadoop.tools.util.DistCpUtils
Sort sequence file containing FileStatus and Text as key and value respecitvely
SPLIT_RATIO_DEFAULT - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
SUCCESS - Static variable in class org.apache.hadoop.tools.DistCpConstants
Constants for DistCp return code to shell / consumer of ToolRunner's run

T

ThrottledInputStream - Class in org.apache.hadoop.tools.util
The ThrottleInputStream provides bandwidth throttling on a specified InputStream.
ThrottledInputStream(InputStream) - Constructor for class org.apache.hadoop.tools.util.ThrottledInputStream
 
ThrottledInputStream(InputStream, long) - Constructor for class org.apache.hadoop.tools.util.ThrottledInputStream
 
toCopyListingFileStatus(FileSystem, FileStatus, boolean, boolean, boolean) - Static method in class org.apache.hadoop.tools.util.DistCpUtils
Converts a FileStatus to a CopyListingFileStatus.
toString() - Method in class org.apache.hadoop.tools.CopyListingFileStatus
 
toString() - Method in class org.apache.hadoop.tools.DistCpOptions
Utility to easily string-ify Options, for logging.
toString() - Method in enum org.apache.hadoop.tools.DistCpOptionSwitch
 
toString() - Method in class org.apache.hadoop.tools.util.ThrottledInputStream

U

UNIFORMSIZE - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
UniformSizeInputFormat - Class in org.apache.hadoop.tools.mapred
UniformSizeInputFormat extends the InputFormat class, to produce input-splits for DistCp.
UniformSizeInputFormat() - Constructor for class org.apache.hadoop.tools.mapred.UniformSizeInputFormat
 
UNKNOWN_ERROR - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
unpackAttributes(String) - Static method in class org.apache.hadoop.tools.util.DistCpUtils
Unpacks preservation attribute string containing the first character of each preservation attribute back to a set of attributes to preserve
usage() - Static method in class org.apache.hadoop.tools.OptionsParser
 

V

validate(DistCpOptionSwitch, boolean) - Method in class org.apache.hadoop.tools.DistCpOptions
 
validatePaths(DistCpOptions) - Method in class org.apache.hadoop.tools.CopyListing
Validate input and output paths
validatePaths(DistCpOptions) - Method in class org.apache.hadoop.tools.FileBasedCopyListing
Validate input and output paths
validatePaths(DistCpOptions) - Method in class org.apache.hadoop.tools.GlobbedCopyListing
Validate input and output paths
validatePaths(DistCpOptions) - Method in class org.apache.hadoop.tools.SimpleCopyListing
 
valueOf(String) - Static method in enum org.apache.hadoop.tools.DistCpOptions.FileAttribute
Returns the enum constant of this type with the specified name.
valueOf(String) - Static method in enum org.apache.hadoop.tools.DistCpOptionSwitch
Returns the enum constant of this type with the specified name.
valueOf(String) - Static method in enum org.apache.hadoop.tools.mapred.CopyMapper.Counter
Returns the enum constant of this type with the specified name.
values() - Static method in enum org.apache.hadoop.tools.DistCpOptions.FileAttribute
Returns an array containing the constants of this enum type, in the order they are declared.
values() - Static method in enum org.apache.hadoop.tools.DistCpOptionSwitch
Returns an array containing the constants of this enum type, in the order they are declared.
values() - Static method in enum org.apache.hadoop.tools.mapred.CopyMapper.Counter
Returns an array containing the constants of this enum type, in the order they are declared.

W

waitForJobCompletion(Job) - Method in class org.apache.hadoop.tools.DistCp
Wait for the given job to complete.
write(DataOutput) - Method in class org.apache.hadoop.tools.CopyListingFileStatus
 

X

XATTRS_NOT_SUPPORTED - Static variable in class org.apache.hadoop.tools.DistCpConstants
 
XAttrsNotSupportedException(String) - Constructor for exception org.apache.hadoop.tools.CopyListing.XAttrsNotSupportedException
 
A B C D E F G H I M N O P R S T U V W X 
Skip navigation links

Copyright © 2015 Apache Software Foundation. All Rights Reserved.