hadoop配置文件

core-site.xml hdfs核心文件

name	default	description
hadoop.common.configuration.version	0.23.0	此配置文件的版本。
hadoop.tmp.dir	/tmp/hadoop-${user.name}	其他临时目录的根目录.
io.native.lib.available	true	Should native hadoop libraries, if present, be used.
hadoop.http.filter.initializers	org.apache.hadoop.http.lib.StaticUserWebFilter	一个逗号分隔的类名列表。 列表中的每个类都必须 extend org.apache.hadoop.http.FilterInitializer。 相应的过滤器将被初始化。 然后, 该过滤器将应用于所有面向jsp和servlet web页面的用户。 列表的顺序定义了过滤器的顺序。
hadoop.security.authorization	false	服务级别授权是否启用?
hadoop.security.instrumentation.requires.admin	false	Indicates if administrator ACLs are required to access instrumentation servlets (JMX, METRICS, CONF, STACKS).
hadoop.security.authentication	simple	可以设置的值为 simple (无认证) 或者 kerberos（一种安全认证系统）
hadoop.security.group.mapping	org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback	Class for user to group mapping (get groups for a given user) for ACL. The default implementation, org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, will determine if the Java Native Interface (JNI) is available. If JNI is available the implementation will use the API within hadoop to resolve a list of groups for a user. If JNI is not available then the shell implementation, ShellBasedUnixGroupsMapping, is used. This implementation shells out to the Linux/Unix environment with the bash -c groups command to resolve a list of groups for a user.
fs.client.htrace.sampler.classes		The class names of the HTrace Samplers to use for Hadoop filesystem clients.
hadoop.htrace.span.receiver.classes		The class names of the Span Receivers to use for Hadoop.
hadoop.security.groups.cache.secs	300	This is the config controlling the validity of the entries in the cache containing the user->group mapping. When this duration has expired, then the implementation of the group mapping provider is invoked to get the groups of the user and then cached back.
hadoop.security.groups.negative-cache.secs	30	Expiration time for entries in the the negative user-to-group mapping caching, in seconds. This is useful when invalid users are retrying frequently. It is suggested to set a small value for this expiration, since a transient error in group lookup could temporarily lock out a legitimate user. Set this to zero or negative value to disable negative user-to-group caching.
hadoop.security.groups.cache.warn.after.ms	5000	If looking up a single user to group takes longer than this amount of milliseconds, we will log a warning message.
hadoop.security.group.mapping.ldap.url		The URL of the LDAP server to use for resolving user groups when using the LdapGroupsMapping user to group mapping.
hadoop.security.group.mapping.ldap.ssl	false	Whether or not to use SSL when connecting to the LDAP server.
hadoop.security.group.mapping.ldap.ssl.keystore		File path to the SSL keystore that contains the SSL certificate required by the LDAP server.
hadoop.security.group.mapping.ldap.ssl.keystore.password.file		The path to a file containing the password of the LDAP SSL keystore. IMPORTANT: This file should be readable only by the Unix user running the daemons.
hadoop.security.group.mapping.ldap.bind.user		The distinguished name of the user to bind as when connecting to the LDAP server. This may be left blank if the LDAP server supports anonymous binds.
hadoop.security.group.mapping.ldap.bind.password.file		The path to a file containing the password of the bind user. IMPORTANT: This file should be readable only by the Unix user running the daemons.
hadoop.security.group.mapping.ldap.base		The search base for the LDAP connection. This is a distinguished name, and will typically be the root of the LDAP directory.
hadoop.security.group.mapping.ldap.search.filter.user	(&(objectClass=user)(sAMAccountName={0}))	An additional filter to use when searching for LDAP users. The default will usually be appropriate for Active Directory installations. If connecting to an LDAP server with a non-AD schema, this should be replaced with (&(objectClass=inetOrgPerson)(uid={0}). {0} is a special string used to denote where the username fits into the filter. If the LDAP server supports posixGroups, Hadoop can enable the feature by setting the value of this property to “posixAccount” and the value of the hadoop.security.group.mapping.ldap.search.filter.group property to “posixGroup”.
hadoop.security.group.mapping.ldap.search.filter.group	(objectClass=group)	An additional filter to use when searching for LDAP groups. This should be changed when resolving groups against a non-Active Directory installation. See the description of hadoop.security.group.mapping.ldap.search.filter.user to enable posixGroups support.
hadoop.security.group.mapping.ldap.search.attr.member	member	The attribute of the group object that identifies the users that are members of the group. The default will usually be appropriate for any LDAP installation.
hadoop.security.group.mapping.ldap.search.attr.group.name	cn	The attribute of the group object that identifies the group name. The default will usually be appropriate for all LDAP systems.
hadoop.security.group.mapping.ldap.directory.search.timeout	10000	The attribute applied to the LDAP SearchControl properties to set a maximum time limit when searching and awaiting a result. Set to 0 if infinite wait period is desired. Default is 10 seconds. Units in milliseconds.
hadoop.security.service.user.name.key		For those cases where the same RPC protocol is implemented by multiple servers, this configuration is required for specifying the principal name to use for the service when the client wishes to make an RPC call.
hadoop.security.uid.cache.secs	14400	This is the config controlling the validity of the entries in the cache containing the userId to userName and groupId to groupName used by NativeIO getFstat().
hadoop.rpc.protection	authentication	A comma-separated list of protection values for secured sasl connections. Possible values are authentication, integrity and privacy. authentication means authentication only and no integrity or privacy; integrity implies authentication and integrity are enabled; and privacy implies all of authentication, integrity and privacy are enabled. hadoop.security.saslproperties.resolver.class can be used to override the hadoop.rpc.protection for a connection at the server side.
hadoop.security.saslproperties.resolver.class		SaslPropertiesResolver used to resolve the QOP used for a connection. If not specified, the full set of values specified in hadoop.rpc.protection is used while determining the QOP used for the connection. If a class is specified, then the QOP values returned by the class will be used while determining the QOP used for the connection.
hadoop.work.around.non.threadsafe.getpwuid	false	Some operating systems or authentication modules are known to have broken implementations of getpwuid_r and getpwgid_r, such that these calls are not thread-safe. Symptoms of this problem include JVM crashes with a stack trace inside these functions. If your system exhibits this issue, enable this configuration parameter to include a lock around the calls as a workaround. An incomplete list of some systems known to have this issue is available at http://wiki.apache.org/hadoop/KnownBrokenPwuidImplementations
hadoop.kerberos.kinit.command	kinit	Used to periodically renew Kerberos credentials when provided to Hadoop. The default setting assumes that kinit is in the PATH of users running the Hadoop client. Change this to the absolute path to kinit if this is not the case.
hadoop.security.auth_to_local		Maps kerberos principals to local user names
*io.file.buffer.size	4096	用于顺序文件的缓冲区大小。 这个缓冲区的大小应该是hardware page size的倍数。 (4096 on Intel x86), 它决定了在读和写操作期间缓冲了多少数据。
io.bytes.per.checksum	512	The number of bytes per checksum. Must not be larger than io.file.buffer.size.
io.skip.checksum.errors	false	如果是true, when a checksum error is encountered while reading a sequence file, entries are skipped, instead of throwing an exception.
io.compression.codecs		A comma-separated list of the compression codec classes that can be used for compression/decompression. In addition to any classes specified with this property (which take precedence), codec classes on the classpath are discovered using a Java ServiceLoader.
io.compression.codec.bzip2.library	system-native	The native-code library to be used for compression and decompression by the bzip2 codec. This library could be specified either by by name or the full pathname. In the former case, the library is located by the dynamic linker, usually searching the directories specified in the environment variable LD_LIBRARY_PATH. The value of “system-native” indicates that the default system library should be used. To indicate that the algorithm should operate entirely in Java, specify “java-builtin”.
io.serializations	org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization	A list of serialization classes that can be used for obtaining serializers and deserializers.
io.seqfile.local.dir	${hadoop.tmp.dir}/io/local	The local directory where sequence file stores intermediate data files during merge. May be a comma-separated list of directories on different devices in order to spread disk i/o. Directories that do not exist are ignored.
io.map.index.skip	0	Number of index entries to skip between each entry. Zero by default. Setting this to values larger than zero can facilitate opening large MapFiles using less memory.
io.map.index.interval	128	MapFile consist of two files - data file (tuples) and index file (keys). For every io.map.index.interval records written in the data file, an entry (record-key, data-file-position) is written in the index file. This is to allow for doing binary search later within the index file to look up records by their keys and get their closest positions in the data file.
fs.defaultFS	file:///	默认文件系统的名称。A URI whose scheme and authority determine the FileSystem implementation. The uri’s scheme determines the config property (fs.SCHEME.impl) naming the FileSystem implementation class. uri的权限用于确定文件系统的主机、端口等。
fs.default.name	file:///	弃用. 使用(fs.defaultFS)属性
fs.trash.interval	0	trash checkpoints被删除的分钟数。如果是0, trash功能被禁用。 可以在服务器和客户机上配置此选项。如果垃圾是禁用的服务器端，则检查客户端配置。如果在服务器端启用了垃圾，那么将使用服务器上配置的值，忽略客户机配置值。
fs.trash.checkpoint.interval	0	trash checkpoints之间的分钟数。 应该小于或等于 fs.trash.interval 。 如果是0, 该值设置为 fs.trash.interval 的值。 每当checkpointer运行时，就会创建一个新的检查点，并移除超过fs.trash.interval创建的检查点。
fs.AbstractFileSystem.file.impl	org.apache.hadoop.fs.local.LocalFs	The AbstractFileSystem for file: uris.
fs.AbstractFileSystem.har.impl	org.apache.hadoop.fs.HarFs	The AbstractFileSystem for har: uris.
fs.AbstractFileSystem.hdfs.impl	org.apache.hadoop.fs.Hdfs	The FileSystem for hdfs: uris.
fs.AbstractFileSystem.viewfs.impl	org.apache.hadoop.fs.viewfs.ViewFs	The AbstractFileSystem for view file system for viewfs: uris (ie client side mount table:).
fs.ftp.host	0.0.0.0	FTP filesystem connects to this server
fs.ftp.host.port	21	FTP filesystem connects to fs.ftp.host on this port
fs.df.interval	60000	Disk usage statistics refresh interval in msec.
fs.du.interval	600000	File space usage statistics refresh interval in msec.
fs.s3.block.size	67108864	Block size to use when writing files to S3.
fs.s3.buffer.dir	${hadoop.tmp.dir}/s3	Determines where on the local filesystem the S3 filesystem should store files before sending them to S3 (or after retrieving them from S3).
fs.s3.maxRetries	4	The maximum number of retries for reading or writing files to S3, before we signal failure to the application.
fs.s3.sleepTimeSeconds	10	The number of seconds to sleep between each S3 retry.
fs.swift.impl	org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem	The implementation class of the OpenStack Swift Filesystem
fs.automatic.close	true	By default, FileSystem instances are automatically closed at program exit using a JVM shutdown hook. Setting this property to false disables this behavior. This is an advanced option that should only be used by server applications requiring a more carefully orchestrated shutdown sequence.
fs.s3n.block.size	67108864	Block size to use when reading files using the native S3 filesystem (s3n: URIs).
fs.s3n.multipart.uploads.enabled	false	Setting this property to true enables multiple uploads to native S3 filesystem. When uploading a file, it is split into blocks if the size is larger than fs.s3n.multipart.uploads.block.size.
fs.s3n.multipart.uploads.block.size	67108864	The block size for multipart uploads to native S3 filesystem. Default size is 64MB.
fs.s3n.multipart.copy.block.size	5368709120	The block size for multipart copy in native S3 filesystem. Default size is 5GB.
fs.s3n.server-side-encryption-algorithm		Specify a server-side encryption algorithm for S3. The default is NULL, and the only other currently allowable value is AES256.
fs.s3a.awsAccessKeyId		AWS access key ID. Omit for Role-based authentication.
fs.s3a.awsSecretAccessKey		AWS secret key. Omit for Role-based authentication.
fs.s3a.connection.maximum	15	Controls the maximum number of simultaneous connections to S3.
fs.s3a.connection.ssl.enabled	true	Enables or disables SSL connections to S3.
fs.s3a.endpoint		AWS S3 endpoint to connect to. An up-to-date list is provided in the AWS Documentation: regions and endpoints. Without this property, the standard region (s3.amazonaws.com) is assumed.
fs.s3a.proxy.host		Hostname of the (optional) proxy server for S3 connections.
fs.s3a.proxy.port		Proxy server port. If this property is not set but fs.s3a.proxy.host is, port 80 or 443 is assumed (consistent with the value of fs.s3a.connection.ssl.enabled).
fs.s3a.proxy.username		Username for authenticating with proxy server.
fs.s3a.proxy.password		Password for authenticating with proxy server.
fs.s3a.proxy.domain		Domain for authenticating with proxy server.
fs.s3a.proxy.workstation		Workstation for authenticating with proxy server.
fs.s3a.attempts.maximum	20	How many times we should retry commands on transient errors.
fs.s3a.connection.establish.timeout	5000	Socket connection setup timeout in milliseconds.
fs.s3a.connection.timeout	200000	Socket connection timeout in milliseconds.
fs.s3a.paging.maximum	5000	How many keys to request from S3 when doing directory listings at a time.
fs.s3a.threads.max	256	Maximum number of concurrent active (part)uploads, which each use a thread from the threadpool.
fs.s3a.threads.core	15	Number of core threads in the threadpool.
fs.s3a.threads.keepalivetime	60	Number of seconds a thread can be idle before being terminated.
fs.s3a.max.total.tasks	1000	Number of (part)uploads allowed to the queue before blocking additional uploads.
fs.s3a.multipart.size	104857600	How big (in bytes) to split upload or copy operations up into.
fs.s3a.multipart.threshold	2147483647	Threshold before uploads or copies use parallel multipart operations.
fs.s3a.acl.default		Set a canned ACL for newly created and copied objects. Value may be private, public-read, public-read-write, authenticated-read, log-delivery-write, bucket-owner-read, or bucket-owner-full-control.
fs.s3a.multipart.purge	false	True if you want to purge existing multipart uploads that may not have been completed/aborted correctly
fs.s3a.multipart.purge.age	86400	Minimum age in seconds of multipart uploads to purge
fs.s3a.signing-algorithm		Override the default signing algorithm so legacy implementations can still be used
fs.s3a.buffer.dir	${hadoop.tmp.dir}/s3a	Comma separated list of directories that will be used to buffer file uploads to.
fs.s3a.fast.upload	false	Upload directly from memory instead of buffering to disk first. Memory usage and parallelism can be controlled as up to fs.s3a.multipart.size memory is consumed for each (part)upload actively uploading (fs.s3a.threads.max) or queueing (fs.s3a.max.total.tasks)
fs.s3a.fast.buffer.size	1048576	Size of initial memory buffer in bytes allocated for an upload. No effect if fs.s3a.fast.upload is false.
fs.s3a.impl	org.apache.hadoop.fs.s3a.S3AFileSystem	The implementation class of the S3A Filesystem
fs.AbstractFileSystem.s3a.impl	org.apache.hadoop.fs.s3a.S3A	The implementation class of the S3A AbstractFileSystem.
io.seqfile.compress.blocksize	1000000	The minimum block size for compression in block compressed SequenceFiles.
io.seqfile.lazydecompress	true	Should values of block-compressed SequenceFiles be decompressed only when necessary.
io.seqfile.sorter.recordlimit	1000000	The limit on number of records to be kept in memory in a spill in SequenceFiles.Sorter
io.mapfile.bloom.size	1048576	The size of BloomFilter-s used in BloomMapFile. Each time this many keys is appended the next BloomFilter will be created (inside a DynamicBloomFilter). Larger values minimize the number of filters, which slightly increases the performance, but may waste too much space if the total number of keys is usually much smaller than this number.
io.mapfile.bloom.error.rate	0.005	The rate of false positives in BloomFilter-s used in BloomMapFile. As this value decreases, the size of BloomFilter-s increases exponentially. This value is the probability of encountering false positives (default is 0.5%).
hadoop.util.hash.type	murmur	The default implementation of Hash. Currently this can take one of the two values: ‘murmur’ to select MurmurHash and ‘jenkins’ to select JenkinsHash.
ipc.client.idlethreshold	4000	Defines the threshold number of connections after which connections will be inspected for idleness.
ipc.client.kill.max	10	Defines the maximum number of clients to disconnect in one go.
ipc.client.connection.maxidletime	10000	The maximum time in msec after which a client will bring down the connection to the server.
ipc.client.connect.max.retries	10	Indicates the number of retries a client will make to establish a server connection.
ipc.client.connect.retry.interval	1000	Indicates the number of milliseconds a client will wait for before retrying to establish a server connection.
ipc.client.connect.timeout	20000	Indicates the number of milliseconds a client will wait for the socket to establish a server connection.
ipc.client.connect.max.retries.on.timeouts	45	Indicates the number of retries a client will make on socket timeout to establish a server connection.
ipc.client.ping	true	Send a ping to the server when timeout on reading the response, if set to true. If no failure is detected, the client retries until at least a byte is read.
ipc.ping.interval	60000	Timeout on waiting response from server, in milliseconds. The client will send ping when the interval is passed without receiving bytes, if ipc.client.ping is set to true.
ipc.client.rpc-timeout.ms	0	Timeout on waiting response from server, in milliseconds. Currently this timeout works only when ipc.client.ping is set to true because it uses the same facilities with IPC ping. The timeout overrides the ipc.ping.interval and client will throw exception instead of sending ping when the interval is passed.
ipc.server.listen.queue.size	128	Indicates the length of the listen queue for servers accepting client connections.
hadoop.security.impersonation.provider.class		A class which implements ImpersonationProvider interface, used to authorize whether one user can impersonate a specific user. If not specified, the DefaultImpersonationProvider will be used. If a class is specified, then that class will be used to determine the impersonation capability.
hadoop.rpc.socket.factory.class.default	org.apache.hadoop.net.StandardSocketFactory	Default SocketFactory to use. This parameter is expected to be formatted as “package.FactoryClassName”.
hadoop.rpc.socket.factory.class.ClientProtocol		SocketFactory to use to connect to a DFS. If null or empty, use hadoop.rpc.socket.class.default. This socket factory is also used by DFSClient to create sockets to DataNodes.
hadoop.socks.server		Address (host:port) of the SOCKS server to be used by the SocksSocketFactory.
net.topology.node.switch.mapping.impl	org.apache.hadoop.net.ScriptBasedMapping	The default implementation of the DNSToSwitchMapping. It invokes a script specified in net.topology.script.file.name to resolve node names. If the value for net.topology.script.file.name is not set, the default value of DEFAULT_RACK is returned for all node names.
net.topology.impl	org.apache.hadoop.net.NetworkTopology	The default implementation of NetworkTopology which is classic three layer one.
net.topology.script.file.name		The script name that should be invoked to resolve DNS names to NetworkTopology names. Example: the script would take host.foo.bar as an argument, and return /rack1 as the output.
net.topology.script.number.args	100	The max number of args that the script configured with net.topology.script.file.name should be run with. Each arg is an IP address.
net.topology.table.file.name		The file name for a topology file, which is used when the net.topology.node.switch.mapping.impl property is set to org.apache.hadoop.net.TableMapping. The file format is a two column text file, with columns separated by whitespace. The first column is a DNS or IP address and the second column specifies the rack where the address maps. If no entry corresponding to a host in the cluster is found, then /default-rack is assumed.
file.stream-buffer-size	4096	The size of buffer to stream files. The size of this buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines how much data is buffered during read and write operations.
file.bytes-per-checksum	512	The number of bytes per checksum. Must not be larger than file.stream-buffer-size
file.client-write-packet-size	65536	Packet size for clients to write
file.blocksize	67108864	Block size
file.replication	1	Replication factor
s3.stream-buffer-size	4096	The size of buffer to stream files. The size of this buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines how much data is buffered during read and write operations.
s3.bytes-per-checksum	512	The number of bytes per checksum. Must not be larger than s3.stream-buffer-size
s3.client-write-packet-size	65536	Packet size for clients to write
s3.blocksize	67108864	Block size
s3.replication	3	Replication factor
s3native.stream-buffer-size	4096	The size of buffer to stream files. The size of this buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines how much data is buffered during read and write operations.
s3native.bytes-per-checksum	512	The number of bytes per checksum. Must not be larger than s3native.stream-buffer-size
s3native.client-write-packet-size	65536	Packet size for clients to write
s3native.blocksize	67108864	Block size
s3native.replication	3	Replication factor
ftp.stream-buffer-size	4096	The size of buffer to stream files. The size of this buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines how much data is buffered during read and write operations.
ftp.bytes-per-checksum	512	The number of bytes per checksum. Must not be larger than ftp.stream-buffer-size
ftp.client-write-packet-size	65536	Packet size for clients to write
ftp.blocksize	67108864	Block size
ftp.replication	3	Replication factor
tfile.io.chunk.size	1048576	Value chunk size in bytes. Default to 1MB. Values of the length less than the chunk size is guaranteed to have known value length in read time (See also TFile.Reader.Scanner.Entry.isValueLengthKnown()).
tfile.fs.output.buffer.size	262144	Buffer size used for FSDataOutputStream in bytes.
tfile.fs.input.buffer.size	262144	Buffer size used for FSDataInputStream in bytes.
hadoop.http.authentication.type	simple	Defines authentication used for Oozie HTTP endpoint. Supported values are: simple # kerberos # AUTHENTICATION_HANDLER_CLASSNAME#
hadoop.http.authentication.token.validity	36000	Indicates how long (in seconds) an authentication token is valid before it has to be renewed.
hadoop.http.authentication.signature.secret.file	${user.home}/hadoop-http-auth-signature-secret	The signature secret for signing the authentication tokens. The same secret should be used for JT/NN/DN/TT configurations.
hadoop.http.authentication.cookie.domain		The domain to use for the HTTP cookie that stores the authentication token. In order to authentiation to work correctly across all Hadoop nodes web-consoles the domain must be correctly set. IMPORTANT: when using IP addresses, browsers ignore cookies with domain settings. For this setting to work properly all nodes in the cluster must be configured to generate URLs with hostname.domain names on it.
hadoop.http.authentication.simple.anonymous.allowed	true	Indicates if anonymous requests are allowed when using ‘simple’ authentication.
hadoop.http.authentication.kerberos.principal	HTTP/_HOST@LOCALHOST	Indicates the Kerberos principal to be used for HTTP endpoint. The principal MUST start with ‘HTTP/’ as per Kerberos HTTP SPNEGO specification.
hadoop.http.authentication.kerberos.keytab	${user.home}/hadoop.keytab	Location of the keytab file with the credentials for the principal. Referring to the same keytab file Oozie uses for its Kerberos credentials for Hadoop.
dfs.ha.fencing.methods		List of fencing methods to use for service fencing. May contain builtin methods (eg shell and sshfence) or user-defined method.
dfs.ha.fencing.ssh.connect-timeout	30000	SSH connection timeout, in milliseconds, to use with the builtin sshfence fencer.
dfs.ha.fencing.ssh.private-key-files		The SSH private key files to use with the builtin sshfence fencer.
hadoop.http.staticuser.user	dr.who	在呈现内容时，将用户名过滤为静态web过滤器。一个例子是HDFS web UI(用于浏览文件的用户)。The user name to filter as, on static web filters while rendering content. An example use is the HDFS web UI (user to be used for browsing files).
ha.zookeeper.quorum		由ZKFailoverController在自动故障转移中使用的ZooKeeper服务器地址列表，逗号分隔。
ha.zookeeper.session-timeout.ms	5000	The session timeout to use when the ZKFC connects to ZooKeeper. Setting this value to a lower value implies that server crashes will be detected more quickly, but risks triggering failover too aggressively in the case of a transient error or network blip.
ha.zookeeper.parent-znode	/hadoop-ha	The ZooKeeper znode under which the ZK failover controller stores its information. Note that the nameservice ID is automatically appended to this znode, so it is not normally necessary to configure this, even in a federated environment.
ha.zookeeper.acl	world:anyone:rwcda	A comma-separated list of ZooKeeper ACLs to apply to the znodes used by automatic failover. These ACLs are specified in the same format as used by the ZooKeeper CLI. If the ACL itself contains secrets, you may instead specify a path to a file, prefixed with the ‘@’ symbol, and the value of this configuration will be loaded from within.
ha.zookeeper.auth		A comma-separated list of ZooKeeper authentications to add when connecting to ZooKeeper. These are specified in the same format as used by the “addauth” command in the ZK CLI. It is important that the authentications specified here are sufficient to access znodes with the ACL specified in ha.zookeeper.acl. If the auths contain secrets, you may instead specify a path to a file, prefixed with the ‘@’ symbol, and the value of this configuration will be loaded from within.
hadoop.ssl.keystores.factory.class	org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory	The keystores factory to use for retrieving certificates.
hadoop.ssl.require.client.cert	false	Whether client certificates are required
hadoop.ssl.hostname.verifier	DEFAULT	The hostname verifier to provide for HttpsURLConnections. Valid values are: DEFAULT, STRICT, STRICT_I6, DEFAULT_AND_LOCALHOST and ALLOW_ALL
hadoop.ssl.server.conf	ssl-server.xml	Resource file from which ssl server keystore information will be extracted. This file is looked up in the classpath, typically it should be in Hadoop conf/ directory.
hadoop.ssl.client.conf	ssl-client.xml	Resource file from which ssl client keystore information will be extracted This file is looked up in the classpath, typically it should be in Hadoop conf/ directory.
hadoop.ssl.enabled	false	Deprecated. Use dfs.http.policy and yarn.http.policy instead.
hadoop.ssl.enabled.protocols	TLSv1,SSLv2Hello,TLSv1.1,TLSv1.2	The supported SSL protocols.
hadoop.jetty.logs.serve.aliases	true	Enable/Disable aliases serving from jetty
fs.permissions.umask-mode	022	The umask used when creating files and directories. Can be in octal or in symbolic. Examples are: “022” (octal for u=rwx,g=r-x,o=r-x in symbolic), or “u=rwx,g=rwx,o=” (symbolic for 007 in octal).
ha.health-monitor.connect-retry-interval.ms	1000	How often to retry connecting to the service.
ha.health-monitor.check-interval.ms	1000	How often to check the service.
ha.health-monitor.sleep-after-disconnect.ms	1000	How long to sleep after an unexpected RPC error.
ha.health-monitor.rpc-timeout.ms	45000	Timeout for the actual monitorHealth() calls.
ha.failover-controller.new-active.rpc-timeout.ms	60000	Timeout that the FC waits for the new active to become active
ha.failover-controller.graceful-fence.rpc-timeout.ms	5000	Timeout that the FC waits for the old active to go to standby
ha.failover-controller.graceful-fence.connection.retries	1	FC connection retries for graceful fencing
ha.failover-controller.cli-check.rpc-timeout.ms	20000	Timeout that the CLI (manual) FC waits for monitorHealth, getServiceState
ipc.client.fallback-to-simple-auth-allowed	false	When a client is configured to attempt a secure connection, but attempts to connect to an insecure server, that server may instruct the client to switch to SASL SIMPLE (unsecure) authentication. This setting controls whether or not the client will accept this instruction from the server. When false (the default), the client will not allow the fallback to SIMPLE authentication, and will abort the connection.
fs.client.resolve.remote.symlinks	true	Whether to resolve symlinks when accessing a remote Hadoop filesystem. Setting this to false causes an exception to be thrown upon encountering a symlink. This setting does not apply to local filesystems, which automatically resolve local symlinks.
nfs.exports.allowed.hosts	* rw	By default, the export can be mounted by any client. The value string contains machine name and access privilege, separated by whitespace characters. The machine name format can be a single host, a Java regular expression, or an IPv4 address. The access privilege uses rw or ro to specify read/write or read-only access of the machines to exports. If the access privilege is not provided, the default is read-only. Entries are separated by “;”. For example: “192.168.0.0/22 rw ; host.*.example.com ; host1.test.org ro;”. Only the NFS gateway needs to restart after this property is updated.
hadoop.user.group.static.mapping.overrides	dr.who=;	Static mapping of user to groups. This will override the groups if available in the system for the specified user. In otherwords, groups look-up will not happen for these users, instead groups mapped in this configuration will be used. Mapping should be in this format. user1=group1,group2;user2=;user3=group2; Default, “dr.who=;” will consider “dr.who” as user without groups.
rpc.metrics.quantile.enable	false	Setting this property to true and rpc.metrics.percentiles.intervals to a comma-separated list of the granularity in seconds, the 50/75/90/95/99th percentile latency for rpc queue/processing time in milliseconds are added to rpc metrics.
rpc.metrics.percentiles.intervals		A comma-separated list of the granularity in seconds for the metrics which describe the 50/75/90/95/99th percentile latency for rpc queue/processing time. The metrics are outputted if rpc.metrics.quantile.enable is set to true.
hadoop.security.crypto.codec.classes.EXAMPLECIPHERSUITE		The prefix for a given crypto codec, contains a comma-separated list of implementation classes for a given crypto codec (eg EXAMPLECIPHERSUITE). The first implementation will be used if available, others are fallbacks.
hadoop.security.crypto.codec.classes.aes.ctr.nopadding	org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec,org.apache.hadoop.crypto.JceAesCtrCryptoCodec	Comma-separated list of crypto codec implementations for AES/CTR/NoPadding. The first implementation will be used if available, others are fallbacks.
hadoop.security.crypto.cipher.suite	AES/CTR/NoPadding	Cipher suite for crypto codec.
hadoop.security.crypto.jce.provider		The JCE provider name used in CryptoCodec.
hadoop.security.crypto.buffer.size	8192	The buffer size used by CryptoInputStream and CryptoOutputStream.
hadoop.security.java.secure.random.algorithm	SHA1PRNG	The java secure random algorithm.
hadoop.security.secure.random.impl		Implementation of secure random.
hadoop.security.random.device.file.path	/dev/urandom	OS security random device file path.
fs.har.impl.disable.cache	true	Don’t cache ‘har’ filesystem instances.
hadoop.security.kms.client.authentication.retry-count	1	Number of time to retry connecting to KMS on authentication failure
hadoop.security.kms.client.encrypted.key.cache.size	500	Size of the EncryptedKeyVersion cache Queue for each key
hadoop.security.kms.client.encrypted.key.cache.low-watermark	0.3f	If size of the EncryptedKeyVersion cache Queue falls below the low watermark, this cache queue will be scheduled for a refill
hadoop.security.kms.client.encrypted.key.cache.num.refill.threads	2	Number of threads to use for refilling depleted EncryptedKeyVersion cache Queues
hadoop.security.kms.client.encrypted.key.cache.expiry	43200000	Cache expiry time for a Key, after which the cache Queue for this key will be dropped. Default = 12hrs
hadoop.shell.missing.defaultFs.warning	true	Enable hdfs shell commands to display warnings if (fs.defaultFS) property is not set.
hadoop.htrace.spanreceiver.classes		A comma separated list of the fully-qualified class name of classes implementing SpanReceiver. The tracing system works by collecting information in structs called ‘Spans’. It is up to you to choose how you want to receive this information by implementing the SpanReceiver interface.
hadoop.http.logs.enabled	true	Enable the “/logs” endpoint on all Hadoop daemons, which serves local logs, but may be considered a security risk due to it listing the contents of a directory.

hdfs-site.xml HDFS配置文件

name	default	description
hadoop.hdfs.configuration.version	1	version of this configuration file
dfs.namenode.rpc-address	# RPC address that handles all clients requests. In the case of HA/Federation where multiple namenodes exist, the name service id is added to the name e.g. dfs.namenode.rpc-address.ns1 dfs.namenode.rpc-address.EXAMPLENAMESERVICE The value of this property will take the form of nn-host1:rpc-port.
dfs.namenode.rpc-bind-host		The actual address the RPC server will bind to. If this optional address is set, it overrides only the hostname portion of dfs.namenode.rpc-address. It can also be specified per name node or name service for HA/Federation. This is useful for making the name node listen on all interfaces by setting it to 0.0.0.0.
dfs.namenode.servicerpc-address		RPC address for HDFS Services communication. BackupNode, Datanodes and all other services should be connecting to this address if it is configured. In the case of HA/Federation where multiple namenodes exist, the name service id is added to the name e.g. dfs.namenode.servicerpc-address.ns1 dfs.namenode.rpc-address.EXAMPLENAMESERVICE The value of this property will take the form of nn-host1:rpc-port. If the value of this property is unset the value of dfs.namenode.rpc-address will be used as the default.
dfs.namenode.servicerpc-bind-host		The actual address the service RPC server will bind to. If this optional address is set, it overrides only the hostname portion of dfs.namenode.servicerpc-address. It can also be specified per name node or name service for HA/Federation. This is useful for making the name node listen on all interfaces by setting it to 0.0.0.0.
dfs.namenode.secondary.http-address	0.0.0.0:50090	The secondary namenode http server address and port.
dfs.namenode.secondary.https-address	0.0.0.0:50091	The secondary namenode HTTPS server address and port.
dfs.datanode.address	0.0.0.0:50010	The datanode server address and port for data transfer.
dfs.datanode.http.address	0.0.0.0:50075	The datanode http server address and port.
dfs.datanode.ipc.address	0.0.0.0:50020	The datanode ipc server address and port.
*dfs.datanode.handler.count	10	datanode处理RPC的服务线程数。
dfs.namenode.http-address	0.0.0.0:50070	The address and the base port where the dfs namenode web ui will listen on.
dfs.namenode.http-bind-host		The actual adress the HTTP server will bind to. If this optional address is set, it overrides only the hostname portion of dfs.namenode.http-address. It can also be specified per name node or name service for HA/Federation. This is useful for making the name node HTTP server listen on all interfaces by setting it to 0.0.0.0.
dfs.http.policy	HTTP_ONLY	Decide if HTTPS(SSL) is supported on HDFS This configures the HTTP endpoint for HDFS daemons: The following values are supported: - HTTP_ONLY : Service is provided only on http - HTTPS_ONLY : Service is provided only on https - HTTP_AND_HTTPS : Service is provided both on http and https
dfs.client.https.need-auth	false	Whether SSL client certificate authentication is required
dfs.client.cached.conn.retry	3	The number of times the HDFS client will pull a socket from the cache. Once this number is exceeded, the client will try to create a new socket.
dfs.https.server.keystore.resource	ssl-server.xml	Resource file from which ssl server keystore information will be extracted
dfs.client.https.keystore.resource	ssl-client.xml	Resource file from which ssl client keystore information will be extracted
dfs.datanode.https.address	0.0.0.0:50475	The datanode secure http server address and port.
dfs.namenode.https-address	0.0.0.0:50470	The namenode secure http server address and port.
dfs.namenode.https-bind-host		The actual adress the HTTPS server will bind to. If this optional address is set, it overrides only the hostname portion of dfs.namenode.https-address. It can also be specified per name node or name service for HA/Federation. This is useful for making the name node HTTPS server listen on all interfaces by setting it to 0.0.0.0.
dfs.datanode.dns.interface	default	The name of the Network Interface from which a data node should report its IP address.
dfs.datanode.dns.nameserver	default	The host name or IP address of the name server (DNS) which a DataNode should use to determine the host name used by the NameNode for communication and display purposes.
dfs.namenode.backup.address	0.0.0.0:50100	The backup node server address and port. If the port is 0 then the server will start on a free port.
dfs.namenode.backup.http-address	0.0.0.0:50105	The backup node http server address and port. If the port is 0 then the server will start on a free port.
dfs.namenode.replication.considerLoad	true	Decide if chooseTarget considers the target’s load or not
dfs.default.chunk.view.size	32768	The number of bytes to view for a file on the browser.
dfs.datanode.du.reserved	0	Reserved space in bytes per volume. Always leave this much space free for non dfs use.
dfs.namenode.name.dir	file://${hadoop.tmp.dir}/dfs/name	namenode存放fsimage的目录。 如果这是一个逗号分隔的目录列表, 那么将在所有目录中复制名称表, 以进行冗余。
dfs.namenode.name.dir.restore	false	Set to true to enable NameNode to attempt recovering a previously failed dfs.namenode.name.dir. When enabled, a recovery of any failed directory is attempted during checkpoint.
dfs.namenode.fs-limits.max-component-length	255	Defines the maximum number of bytes in UTF-8 encoding in each component of a path. A value of 0 will disable the check.
dfs.namenode.fs-limits.max-directory-items	1048576	Defines the maximum number of items that a directory may contain. A value of 0 will disable the check.
dfs.namenode.fs-limits.min-block-size	1048576	Minimum block size in bytes, enforced by the Namenode at create time. This prevents the accidental creation of files with tiny block sizes (and thus many blocks), which can degrade performance.
dfs.namenode.fs-limits.max-blocks-per-file	1048576	Maximum number of blocks per file, enforced by the Namenode on write. This prevents the creation of extremely large files which can degrade performance.
dfs.namenode.edits.dir	${dfs.namenode.name.dir}	Determines where on the local filesystem the DFS name node should store the transaction (edits) file. If this is a comma-delimited list of directories then the transaction file is replicated in all of the directories, for redundancy. Default value is same as dfs.namenode.name.dir
dfs.namenode.shared.edits.dir		一个HA集群中多个namenodes之间共享存储的目录。 该目录将由NN编写，并由SNN读取，以保持名称空间的同步。 此目录不需要在dfs.namenode.edits.dir中列出。 它应该在非HA集群中保持为空。
dfs.namenode.edits.journal-plugin.qjournal	org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager
dfs.permissions.enabled	true	如果是true, 在HDFS中启用权限检查。如果是false, 权限检查是关闭的, 但是其他的行为都是不变的。 从一个参数值切换到另一个参数值不会改变模式、所有者或文件组或目录。
dfs.permissions.superusergroup	supergroup	The name of the group of super-users.
dfs.namenode.acls.enabled	false	Set to true to enable support for HDFS ACLs (Access Control Lists). By default, ACLs are disabled. When ACLs are disabled, the NameNode rejects all RPCs related to setting or getting ACLs.
dfs.namenode.lazypersist.file.scrub.interval.sec	300	The NameNode periodically scans the namespace for LazyPersist files with missing blocks and unlinks them from the namespace. This configuration key controls the interval between successive scans. Set it to a negative value to disable this behavior.
dfs.block.access.token.enable	false	If “true”, access tokens are used as capabilities for accessing datanodes. If “false”, no access tokens are checked on accessing datanodes.
dfs.block.access.key.update.interval	600	Interval in minutes at which namenode updates its access keys.
dfs.block.access.token.lifetime	600	The lifetime of access tokens in minutes.
*dfs.datanode.data.dir	file://${hadoop.tmp.dir}/dfs/data	datanode存放数据块文件的目录。 如果这是一个逗号分隔的目录列表, 那么数据将存储在所有命名的目录中, 通常在不同的设备上。 不存在的目录将被忽略。
dfs.datanode.data.dir.perm	700	Permissions for the directories on on the local filesystem where the DFS data node store its blocks. The permissions can either be octal or symbolic.
dfs.replication	3	数据副本数。 可以在创建文件时指定复制的实际数量。 如果在创建文件时没有指定复制, 则使用默认值。
dfs.replication.max	512	Maximal block replication.
dfs.namenode.replication.min	1	Minimal block replication.
*dfs.blocksize	134217728	新文件的默认块大小, 以bytes为单位。 您可以使用以下后缀(大小写不敏感) k(kilo), m(mega), g(giga), t(tera), p(peta), e(exa) 指定大小 (比如 128k, 512m, 1g, etc.), 或提供完整的bytes (比如 134217728 for 128 MB).
dfs.client.block.write.retries	3	The number of retries for writing blocks to the data nodes, before we signal failure to the application.
dfs.client.block.write.replace-datanode-on-failure.enable	true	If there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes. As a result, the number of datanodes in the pipeline is decreased. The feature is to add new datanodes to the pipeline. This is a site-wide property to enable/disable the feature. When the cluster size is extremely small, e.g. 3 nodes or less, cluster administrators may want to set the policy to NEVER in the default configuration file or disable this feature. Otherwise, users may experience an unusually high rate of pipeline failures since it is impossible to find new datanodes for replacement. See also dfs.client.block.write.replace-datanode-on-failure.policy
dfs.client.block.write.replace-datanode-on-failure.policy	DEFAULT	This property is used only if the value of dfs.client.block.write.replace-datanode-on-failure.enable is true. ALWAYS: always add a new datanode when an existing datanode is removed. NEVER: never add a new datanode. DEFAULT: Let r be the replication number. Let n be the number of existing datanodes. Add a new datanode only if r is greater than or equal to 3 and either (1) floor(r/2) is greater than or equal to n; or (2) r is greater than n and the block is hflushed/appended.
dfs.client.block.write.replace-datanode-on-failure.best-effort	false	This property is used only if the value of dfs.client.block.write.replace-datanode-on-failure.enable is true. Best effort means that the client will try to replace a failed datanode in write pipeline (provided that the policy is satisfied), however, it continues the write operation in case that the datanode replacement also fails. Suppose the datanode replacement fails. false: An exception should be thrown so that the write will fail. true : The write should be resumed with the remaining datandoes. Note that setting this property to true allows writing to a pipeline with a smaller number of datanodes. As a result, it increases the probability of data loss.
dfs.blockreport.intervalMsec	21600000	Determines block reporting interval in milliseconds.
dfs.blockreport.initialDelay	0	Delay for first block report in seconds.
dfs.blockreport.split.threshold	1000000	If the number of blocks on the DataNode is below this threshold then it will send block reports for all Storage Directories in a single message. If the number of blocks exceeds this threshold then the DataNode will send block reports for each Storage Directory in separate messages. Set to zero to always split.
dfs.namenode.max.full.block.report.leases	6	The maximum number of leases for full block reports that the NameNode will issue at any given time. This prevents the NameNode from being flooded with full block reports that use up all the RPC handler threads. This number should never be more than the number of RPC handler threads or less than 1.
dfs.namenode.full.block.report.lease.length.ms	300000	The number of milliseconds that the NameNode will wait before invalidating a full block report lease. This prevents a crashed DataNode from permanently using up a full block report lease.
dfs.datanode.directoryscan.interval	21600	Interval in seconds for Datanode to scan data directories and reconcile the difference between blocks in memory and on the disk.
dfs.datanode.directoryscan.threads	1	How many threads should the threadpool used to compile reports for volumes in parallel have.
dfs.datanode.directoryscan.throttle.limit.ms.per.sec	1000	The report compilation threads are limited to only running for a given number of milliseconds per second, as configured by the property. The limit is taken per thread, not in aggregate, e.g. setting a limit of 100ms for 4 compiler threads will result in each thread being limited to 100ms, not 25ms. Note that the throttle does not interrupt the report compiler threads, so the actual running time of the threads per second will typically be somewhat higher than the throttle limit, usually by no more than 20%. Setting this limit to 1000 disables compiler thread throttling. Only values between 1 and 1000 are valid. Setting an invalid value will result in the throttle being disbled and an error message being logged. 1000 is the default setting.
dfs.heartbeat.interval	3	datanode 心跳间隔 单位秒.
*dfs.namenode.handler.count	10	namenode或jobtracker中用于处理Namenode RPC server的线程数，默认是10，较大集群，可调大些，比如64。 如果没有配置dfs.namenode.servicerpc-address，那么Namenode RPC服务器线程将监听来自所有节点的请求。
dfs.namenode.service.handler.count	10	监听来自DataNodes和所有其他非客户机节点的请求的Namenode RPC服务器线程的数量。dfs.namenode.service.handler.count 只有当dfs.namenode.servicerpc-address被配置时，dfs.namenode.service.handler.count才有效。
dfs.namenode.safemode.threshold-pct	0.999f	Specifies the percentage of blocks that should satisfy the minimal replication requirement defined by dfs.namenode.replication.min. Values less than or equal to 0 mean not to wait for any particular percentage of blocks before exiting safemode. Values greater than 1 will make safe mode permanent.
dfs.namenode.safemode.min.datanodes	0	Specifies the number of datanodes that must be considered alive before the name node exits safemode. Values less than or equal to 0 mean not to take the number of live datanodes into account when deciding whether to remain in safe mode during startup. Values greater than the number of datanodes in the cluster will make safe mode permanent.
dfs.namenode.safemode.extension	30000	Determines extension of safe mode in milliseconds after the threshold level is reached.
dfs.namenode.resource.check.interval	5000	The interval in milliseconds at which the NameNode resource checker runs. The checker calculates the number of the NameNode storage volumes whose available spaces are more than dfs.namenode.resource.du.reserved, and enters safemode if the number becomes lower than the minimum value specified by dfs.namenode.resource.checked.volumes.minimum.
dfs.namenode.resource.du.reserved	104857600	The amount of space to reserve/require for a NameNode storage directory in bytes. The default is 100MB.
dfs.namenode.resource.checked.volumes		A list of local directories for the NameNode resource checker to check in addition to the local edits directories.
dfs.namenode.resource.checked.volumes.minimum	1	The minimum number of redundant NameNode storage volumes required.
*dfs.datanode.balance.bandwidthPerSec	1048576	指定每个datanode可以利用每秒字节数来平衡目标的最大带宽。
dfs.hosts		Names a file that contains a list of hosts that are permitted to connect to the namenode. The full pathname of the file must be specified. If the value is empty, all hosts are permitted.
dfs.hosts.exclude		Names a file that contains a list of hosts that are not permitted to connect to the namenode. The full pathname of the file must be specified. If the value is empty, no hosts are excluded.
dfs.namenode.max.objects	0	The maximum number of files, directories and blocks dfs supports. A value of zero indicates no limit to the number of objects that dfs supports.
dfs.namenode.datanode.registration.ip-hostname-check	true	If true (the default), then the namenode requires that a connecting datanode’s address must be resolved to a hostname. If necessary, a reverse DNS lookup is performed. All attempts to register a datanode from an unresolvable address are rejected. It is recommended that this setting be left on to prevent accidental registration of datanodes listed by hostname in the excludes file during a DNS outage. Only set this to false in environments where there is no infrastructure to support reverse DNS lookup.
dfs.namenode.decommission.interval	30	Namenode periodicity in seconds to check if decommission is complete.
dfs.namenode.decommission.nodes.per.interval	5	The number of nodes namenode checks if decommission is complete in each dfs.namenode.decommission.interval.
dfs.namenode.replication.interval	3	The periodicity in seconds with which the namenode computes repliaction work for datanodes.
dfs.namenode.accesstime.precision	3600000	The access time for HDFS file is precise upto this value. The default value is 1 hour. Setting a value of 0 disables access times for HDFS.
dfs.datanode.plugins		Comma-separated list of datanode plug-ins to be activated.
dfs.namenode.plugins		Comma-separated list of namenode plug-ins to be activated.
dfs.stream-buffer-size	4096	The size of buffer to stream files. The size of this buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines how much data is buffered during read and write operations.
dfs.bytes-per-checksum	512	The number of bytes per checksum. Must not be larger than dfs.stream-buffer-size
dfs.client-write-packet-size	65536	Packet size for clients to write
dfs.client.write.exclude.nodes.cache.expiry.interval.millis	600000	The maximum period to keep a DN in the excluded nodes list at a client. After this period, in milliseconds, the previously excluded node(s) will be removed automatically from the cache and will be considered good for block allocations again. Useful to lower or raise in situations where you keep a file open for very long periods (such as a Write-Ahead-Log (WAL) file) to make the writer tolerant to cluster maintenance restarts. Defaults to 10 minutes.
dfs.namenode.checkpoint.dir	file://${hadoop.tmp.dir}/dfs/namesecondary	DSecondarynamenode启动时使用, 放置sn做合并的fsimage及 editlog。 如果这是一个逗号分隔的目录列表, 那么该映像将在所有的冗余目录中复制。
dfs.namenode.checkpoint.edits.dir	${dfs.namenode.checkpoint.dir}	Determines where on the local filesystem the DFS secondary name node should store the temporary edits to merge. If this is a comma-delimited list of directoires then teh edits is replicated in all of the directoires for redundancy. Default value is same as dfs.namenode.checkpoint.dir
dfs.namenode.checkpoint.period	3600	The number of seconds between two periodic checkpoints.
dfs.namenode.checkpoint.txns	1000000	The Secondary NameNode or CheckpointNode will create a checkpoint of the namespace every ‘dfs.namenode.checkpoint.txns’ transactions, regardless of whether ‘dfs.namenode.checkpoint.period’ has expired.
dfs.namenode.checkpoint.check.period	60	The SecondaryNameNode and CheckpointNode will poll the NameNode every ‘dfs.namenode.checkpoint.check.period’ seconds to query the number of uncheckpointed transactions.
dfs.namenode.checkpoint.max-retries	3	The SecondaryNameNode retries failed checkpointing. If the failure occurs while loading fsimage or replaying edits, the number of retries is limited by this variable.
dfs.namenode.num.checkpoints.retained	2	The number of image checkpoint files that will be retained by the NameNode and Secondary NameNode in their storage directories. All edit logs necessary to recover an up-to-date namespace from the oldest retained checkpoint will also be retained.
dfs.namenode.num.extra.edits.retained	1000000	The number of extra transactions which should be retained beyond what is minimally necessary for a NN restart. This can be useful for audit purposes or for an HA setup where a remote Standby Node may have been offline for some time and need to have a longer backlog of retained edits in order to start again. Typically each edit is on the order of a few hundred bytes, so the default of 1 million edits should be on the order of hundreds of MBs or low GBs. NOTE: Fewer extra edits may be retained than value specified for this setting if doing so would mean that more segments would be retained than the number configured by dfs.namenode.max.extra.edits.segments.retained.
dfs.namenode.max.extra.edits.segments.retained	10000	The maximum number of extra edit log segments which should be retained beyond what is minimally necessary for a NN restart. When used in conjunction with dfs.namenode.num.extra.edits.retained, this configuration property serves to cap the number of extra edits files to a reasonable value.
dfs.namenode.delegation.key.update-interval	86400000	The update interval for master key for delegation tokens in the namenode in milliseconds.
dfs.namenode.delegation.token.max-lifetime	604800000	The maximum lifetime in milliseconds for which a delegation token is valid.
dfs.namenode.delegation.token.renew-interval	86400000	The renewal interval for delegation token in milliseconds.
dfs.datanode.failed.volumes.tolerated	0	The number of volumes that are allowed to fail before a datanode stops offering service. By default any volume failure will cause a datanode to shutdown.
dfs.image.compress	false	Should the dfs image be compressed?
dfs.image.compression.codec	org.apache.hadoop.io.compress.DefaultCodec	If the dfs image is compressed, how should they be compressed? This has to be a codec defined in io.compression.codecs.
dfs.image.transfer.timeout	60000	Socket timeout for image transfer in milliseconds. This timeout and the related dfs.image.transfer.bandwidthPerSec parameter should be configured such that normal image transfer can complete successfully. This timeout prevents client hangs when the sender fails during image transfer. This is socket timeout during image tranfer.
dfs.image.transfer.bandwidthPerSec	0	Maximum bandwidth used for regular image transfers (instead of bootstrapping the standby namenode), in bytes per second. This can help keep normal namenode operations responsive during checkpointing. The maximum bandwidth and timeout in dfs.image.transfer.timeout should be set such that normal image transfers can complete successfully. A default value of 0 indicates that throttling is disabled. The maximum bandwidth used for bootstrapping standby namenode is configured with dfs.image.transfer-bootstrap-standby.bandwidthPerSec.
dfs.image.transfer-bootstrap-standby.bandwidthPerSec	0	Maximum bandwidth used for transferring image to bootstrap standby namenode, in bytes per second. A default value of 0 indicates that throttling is disabled. This default value should be used in most cases, to ensure timely HA operations. The maximum bandwidth used for regular image transfers is configured with dfs.image.transfer.bandwidthPerSec.
dfs.image.transfer.chunksize	65536	Chunksize in bytes to upload the checkpoint. Chunked streaming is used to avoid internal buffering of contents of image file of huge size.
dfs.namenode.support.allow.format	true	Does HDFS namenode allow itself to be formatted? You may consider setting this to false for any production cluster, to avoid any possibility of formatting a running DFS.
*dfs.datanode.max.transfer.threads	4096	指定用于在DataNode间传输block数据的最大线程数。
dfs.datanode.scan.period.hours	504	If this is positive, the DataNode will not scan any individual block more than once in the specified scan period. If this is negative, the block scanner is disabled. If this is set to zero, then the default value of 504 hours or 3 weeks is used. Prior versions of HDFS incorrectly documented that setting this key to zero will disable the block scanner.
dfs.block.scanner.volume.bytes.per.second	1048576	If this is 0, the DataNode’s block scanner will be disabled. If this is positive, this is the number of bytes per second that the DataNode’s block scanner will try to scan from each volume.
dfs.datanode.readahead.bytes	4193404	While reading block files, if the Hadoop native libraries are available, the datanode can use the posix_fadvise system call to explicitly page data into the operating system buffer cache ahead of the current reader’s position. This can improve performance especially when disks are highly contended. This configuration specifies the number of bytes ahead of the current read position which the datanode will attempt to read ahead. This feature may be disabled by configuring this property to 0. If the native libraries are not available, this configuration has no effect.
dfs.datanode.drop.cache.behind.reads	false	In some workloads, the data read from HDFS is known to be significantly large enough that it is unlikely to be useful to cache it in the operating system buffer cache. In this case, the DataNode may be configured to automatically purge all data from the buffer cache after it is delivered to the client. This behavior is automatically disabled for workloads which read only short sections of a block (e.g HBase random-IO workloads). This may improve performance for some workloads by freeing buffer cache spage usage for more cacheable data. If the Hadoop native libraries are not available, this configuration has no effect.
dfs.datanode.drop.cache.behind.writes	false	In some workloads, the data written to HDFS is known to be significantly large enough that it is unlikely to be useful to cache it in the operating system buffer cache. In this case, the DataNode may be configured to automatically purge all data from the buffer cache after it is written to disk. This may improve performance for some workloads by freeing buffer cache spage usage for more cacheable data. If the Hadoop native libraries are not available, this configuration has no effect.
dfs.datanode.sync.behind.writes	false	If this configuration is enabled, the datanode will instruct the operating system to enqueue all written data to the disk immediately after it is written. This differs from the usual OS policy which may wait for up to 30 seconds before triggering writeback. This may improve performance for some workloads by smoothing the IO profile for data written to disk. If the Hadoop native libraries are not available, this configuration has no effect.
dfs.client.failover.max.attempts	15	Expert only. The number of client failover attempts that should be made before the failover is considered failed.
dfs.client.failover.sleep.base.millis	500	Expert only. The time to wait, in milliseconds, between failover attempts increases exponentially as a function of the number of attempts made so far, with a random factor of +/- 50%. This option specifies the base value used in the failover calculation. The first failover will retry immediately. The 2nd failover attempt will delay at least dfs.client.failover.sleep.base.millis milliseconds. And so on.
dfs.client.failover.sleep.max.millis	15000	Expert only. The time to wait, in milliseconds, between failover attempts increases exponentially as a function of the number of attempts made so far, with a random factor of +/- 50%. This option specifies the maximum value to wait between failovers. Specifically, the time between two failover attempts will not exceed +/- 50% of dfs.client.failover.sleep.max.millis milliseconds.
dfs.client.failover.connection.retries	0	Expert only. Indicates the number of retries a failover IPC client will make to establish a server connection.
dfs.client.failover.connection.retries.on.timeouts	0	Expert only. The number of retry attempts a failover IPC client will make on socket timeout when establishing a server connection.
dfs.client.datanode-restart.timeout	30	Expert only. The time to wait, in seconds, from reception of an datanode shutdown notification for quick restart, until declaring the datanode dead and invoking the normal recovery mechanisms. The notification is sent by a datanode when it is being shutdown using the shutdownDatanode admin command with the upgrade option.
dfs.nameservices		nameservices的逗号分隔列表。
dfs.nameservice.id		The ID of this nameservice. If the nameservice ID is not configured or more than one nameservice is configured for dfs.nameservices it is determined automatically by matching the local node’s address with the configured address.
dfs.internal.nameservices		Comma-separated list of nameservices that belong to this cluster. Datanode will report to all the nameservices in this list. By default this is set to the value of dfs.nameservices.
dfs.ha.namenodes.EXAMPLENAMESERVICE		The prefix for a given nameservice, contains a comma-separated list of namenodes for a given nameservice (eg EXAMPLENAMESERVICE).
dfs.ha.namenode.id		The ID of this namenode. If the namenode ID is not configured it is determined automatically by matching the local node’s address with the configured address.
dfs.ha.log-roll.period	120	How often, in seconds, the StandbyNode should ask the active to roll edit logs. Since the StandbyNode only reads from finalized log segments, the StandbyNode will only be as up-to-date as how often the logs are rolled. Note that failover triggers a log roll so the StandbyNode will be up to date before it becomes active.
dfs.ha.tail-edits.period	60	How often, in seconds, the StandbyNode should check for new finalized log segments in the shared edits log.
dfs.ha.automatic-failover.enabled	false	是否启用了自动故障转移。 有关详细信息,请参阅HDFS HA文档自动HA配置。
dfs.client.use.datanode.hostname	false	Whether clients should use datanode hostnames when connecting to datanodes.
dfs.datanode.use.datanode.hostname	false	Whether datanodes should use datanode hostnames when connecting to other datanodes for data transfer.
dfs.client.local.interfaces		A comma separated list of network interface names to use for data transfer between the client and datanodes. When creating a connection to read from or write to a datanode, the client chooses one of the specified interfaces at random and binds its socket to the IP of that interface. Individual names may be specified as either an interface name (eg “eth0”), a subinterface name (eg “eth0:0”), or an IP address (which may be specified using CIDR notation to match a range of IPs).
dfs.datanode.shared.file.descriptor.paths	/dev/shm,/tmp	A comma-separated list of paths to use when creating file descriptors that will be shared between the DataNode and the DFSClient. Typically we use /dev/shm, so that the file descriptors will not be written to disk. Systems that don’t have /dev/shm will fall back to /tmp by default.
dfs.short.circuit.shared.memory.watcher.interrupt.check.ms	60000	The length of time in milliseconds that the short-circuit shared memory watcher will go between checking for java interruptions sent from other threads. This is provided mainly for unit tests.
dfs.namenode.kerberos.internal.spnego.principal	${dfs.web.authentication.kerberos.principal}
dfs.secondary.namenode.kerberos.internal.spnego.principal	${dfs.web.authentication.kerberos.principal}
dfs.namenode.kerberos.principal.pattern	*	A client-side RegEx that can be configured to control allowed realms to authenticate with (useful in cross-realm env.)
dfs.namenode.avoid.read.stale.datanode	false	Indicate whether or not to avoid reading from “stale” datanodes whose heartbeat messages have not been received by the namenode for more than a specified time interval. Stale datanodes will be moved to the end of the node list returned for reading. See dfs.namenode.avoid.write.stale.datanode for a similar setting for writes.
dfs.namenode.avoid.write.stale.datanode	false	Indicate whether or not to avoid writing to “stale” datanodes whose heartbeat messages have not been received by the namenode for more than a specified time interval. Writes will avoid using stale datanodes unless more than a configured ratio (dfs.namenode.write.stale.datanode.ratio) of datanodes are marked as stale. See dfs.namenode.avoid.read.stale.datanode for a similar setting for reads.
dfs.namenode.stale.datanode.interval	30000	Default time interval for marking a datanode as “stale”, i.e., if the namenode has not received heartbeat msg from a datanode for more than this time interval, the datanode will be marked and treated as “stale” by default. The stale interval cannot be too small since otherwise this may cause too frequent change of stale states. We thus set a minimum stale interval value (the default value is 3 times of heartbeat interval) and guarantee that the stale interval cannot be less than the minimum value. A stale data node is avoided during lease/block recovery. It can be conditionally avoided for reads (see dfs.namenode.avoid.read.stale.datanode) and for writes (see dfs.namenode.avoid.write.stale.datanode).
dfs.namenode.write.stale.datanode.ratio	0.5f	When the ratio of number stale datanodes to total datanodes marked is greater than this ratio, stop avoiding writing to stale nodes so as to prevent causing hotspots.
dfs.namenode.invalidate.work.pct.per.iteration	0.32f	Note: Advanced property. Change with caution. This determines the percentage amount of block invalidations (deletes) to do over a single DN heartbeat deletion command. The final deletion count is determined by applying this percentage to the number of live nodes in the system. The resultant number is the number of blocks from the deletion list chosen for proper invalidation over a single heartbeat of a single DN. Value should be a positive, non-zero percentage in float notation (X.Yf), with 1.0f meaning 100%.
dfs.namenode.replication.work.multiplier.per.iteration	2	Note: Advanced property. Change with caution. This determines the total amount of block transfers to begin in parallel at a DN, for replication, when such a command list is being sent over a DN heartbeat by the NN. The actual number is obtained by multiplying this multiplier with the total number of live nodes in the cluster. The result number is the number of blocks to begin transfers immediately for, per DN heartbeat. This number can be any positive, non-zero integer.
nfs.server.port	2049	Specify the port number used by Hadoop NFS.
nfs.mountd.port	4242	Specify the port number used by Hadoop mount daemon.
nfs.dump.dir	/tmp/.hdfs-nfs	This directory is used to temporarily save out-of-order writes before writing to HDFS. For each file, the out-of-order writes are dumped after they are accumulated to exceed certain threshold (e.g., 1MB) in memory. One needs to make sure the directory has enough space.
nfs.rtmax	1048576	This is the maximum size in bytes of a READ request supported by the NFS gateway. If you change this, make sure you also update the nfs mount’s rsize(add rsize= # of bytes to the mount directive).
nfs.wtmax	1048576	This is the maximum size in bytes of a WRITE request supported by the NFS gateway. If you change this, make sure you also update the nfs mount’s wsize(add wsize= # of bytes to the mount directive).
nfs.keytab.file		Note: Advanced property. Change with caution. This is the path to the keytab file for the hdfs-nfs gateway. This is required when the cluster is kerberized.
nfs.kerberos.principal		Note: Advanced property. Change with caution. This is the name of the kerberos principal. This is required when the cluster is kerberized.It must be of this format: nfs-gateway-user/nfs-gateway-host@kerberos-realm
nfs.allow.insecure.ports	true	When set to false, client connections originating from unprivileged ports (those above 1023) will be rejected. This is to ensure that clients connecting to this NFS Gateway must have had root privilege on the machine where they’re connecting from.
dfs.webhdfs.enabled	true	在Namenodes和Datanodes中启用WebHDFS(REST API)默认端口50070。
hadoop.fuse.connection.timeout	300	The minimum number of seconds that we’ll cache libhdfs connection objects in fuse_dfs. Lower values will result in lower memory consumption; higher values may speed up access by avoiding the overhead of creating new connection objects.
hadoop.fuse.timer.period	5	The number of seconds between cache expiry checks in fuse_dfs. Lower values will result in fuse_dfs noticing changes to Kerberos ticket caches more quickly.
dfs.metrics.percentiles.intervals		Comma-delimited set of integers denoting the desired rollover intervals (in seconds) for percentile latency metrics on the Namenode and Datanode. By default, percentile latency metrics are disabled.
dfs.encrypt.data.transfer	false	Whether or not actual block data that is read/written from/to HDFS should be encrypted on the wire. This only needs to be set on the NN and DNs, clients will deduce this automatically. It is possible to override this setting per connection by specifying custom logic via dfs.trustedchannel.resolver.class.
dfs.encrypt.data.transfer.algorithm		This value may be set to either “3des” or “rc4”. If nothing is set, then the configured JCE default on the system is used (usually 3DES.) It is widely believed that 3DES is more cryptographically secure, but RC4 is substantially faster. Note that if AES is supported by both the client and server then this encryption algorithm will only be used to initially transfer keys for AES. (See dfs.encrypt.data.transfer.cipher.suites.)
dfs.encrypt.data.transfer.cipher.suites		This value may be either undefined or AES/CTR/NoPadding. If defined, then dfs.encrypt.data.transfer uses the specified cipher suite for data encryption. If not defined, then only the algorithm specified in dfs.encrypt.data.transfer.algorithm is used. By default, the property is not defined.
dfs.encrypt.data.transfer.cipher.key.bitlength	128	The key bitlength negotiated by dfsclient and datanode for encryption. This value may be set to either 128, 192 or 256.
dfs.trustedchannel.resolver.class		TrustedChannelResolver is used to determine whether a channel is trusted for plain data transfer. The TrustedChannelResolver is invoked on both client and server side. If the resolver indicates that the channel is trusted, then the data transfer will not be encrypted even if dfs.encrypt.data.transfer is set to true. The default implementation returns false indicating that the channel is not trusted.
dfs.data.transfer.protection		A comma-separated list of SASL protection values used for secured connections to the DataNode when reading or writing block data. Possible values are authentication, integrity and privacy. authentication means authentication only and no integrity or privacy; integrity implies authentication and integrity are enabled; and privacy implies all of authentication, integrity and privacy are enabled. If dfs.encrypt.data.transfer is set to true, then it supersedes the setting for dfs.data.transfer.protection and enforces that all connections must use a specialized encrypted SASL handshake. This property is ignored for connections to a DataNode listening on a privileged port. In this case, it is assumed that the use of a privileged port establishes sufficient trust.
dfs.data.transfer.saslproperties.resolver.class		SaslPropertiesResolver used to resolve the QOP used for a connection to the DataNode when reading or writing block data. If not specified, the value of hadoop.security.saslproperties.resolver.class is used as the default value.
dfs.datanode.hdfs-blocks-metadata.enabled	false	Boolean which enables backend datanode-side support for the experimental DistributedFileSystem#getFileVBlockStorageLocations API.
dfs.client.file-block-storage-locations.num-threads	10	Number of threads used for making parallel RPCs in DistributedFileSystem#getFileBlockStorageLocations().
dfs.client.file-block-storage-locations.timeout.millis	1000	Timeout (in milliseconds) for the parallel RPCs made in DistributedFileSystem#getFileBlockStorageLocations().
dfs.journalnode.rpc-address	0.0.0.0:8485	JournalNode（日志节点）RPC服务器地址和端口。
dfs.journalnode.http-address	0.0.0.0:8480	日志节点(JournalNode)HTTP服务器监听的地址和端口。 如果端口设置为0，则服务器将从一个空闲端口开始。
dfs.journalnode.https-address	0.0.0.0:8481	日志节点(JournalNode)HTTPS服务器监听的地址和端口。 如果端口设置为0，则服务器将从一个空闲端口开始。
dfs.namenode.audit.loggers	default	List of classes implementing audit loggers that will receive audit events. These should be implementations of org.apache.hadoop.hdfs.server.namenode.AuditLogger. The special value “default” can be used to reference the default audit logger, which uses the configured log system. Installing custom audit loggers may affect the performance and stability of the NameNode. Refer to the custom logger’s documentation for more details.
dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold	10737418240	Only used when the dfs.datanode.fsdataset.volume.choosing.policy is set to org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy. This setting controls how much DN volumes are allowed to differ in terms of bytes of free disk space before they are considered imbalanced. If the free space of all the volumes are within this range of each other, the volumes will be considered balanced and block assignments will be done on a pure round robin basis.
dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction	0.75f	Only used when the dfs.datanode.fsdataset.volume.choosing.policy is set to org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy. This setting controls what percentage of new block allocations will be sent to volumes with more available disk space than others. This setting should be in the range 0.0 - 1.0, though in practice 0.5 - 1.0, since there should be no reason to prefer that volumes with less available disk space receive more block allocations.
dfs.namenode.edits.noeditlogchannelflush	false	Specifies whether to flush edit log file channel. When set, expensive FileChannel#force calls are skipped and synchronous disk writes are enabled instead by opening the edit log file with RandomAccessFile(“rws”) flags. This can significantly improve the performance of edit log writes on the Windows platform. Note that the behavior of the “rws” flags is platform and hardware specific and might not provide the same level of guarantees as FileChannel#force. For example, the write will skip the disk-cache on SAS and SCSI devices while it might not on SATA devices. This is an expert level setting, change with caution.
dfs.client.cache.drop.behind.writes		Just like dfs.datanode.drop.cache.behind.writes, this setting causes the page cache to be dropped behind HDFS writes, potentially freeing up more memory for other uses. Unlike dfs.datanode.drop.cache.behind.writes, this is a client-side setting rather than a setting for the entire datanode. If present, this setting will override the DataNode default. If the native libraries are not available to the DataNode, this configuration has no effect.
dfs.client.cache.drop.behind.reads		Just like dfs.datanode.drop.cache.behind.reads, this setting causes the page cache to be dropped behind HDFS reads, potentially freeing up more memory for other uses. Unlike dfs.datanode.drop.cache.behind.reads, this is a client-side setting rather than a setting for the entire datanode. If present, this setting will override the DataNode default. If the native libraries are not available to the DataNode, this configuration has no effect.
dfs.client.cache.readahead		When using remote reads, this setting causes the datanode to read ahead in the block file using posix_fadvise, potentially decreasing I/O wait times. Unlike dfs.datanode.readahead.bytes, this is a client-side setting rather than a setting for the entire datanode. If present, this setting will override the DataNode default. When using local reads, this setting determines how much readahead we do in BlockReaderLocal. If the native libraries are not available to the DataNode, this configuration has no effect.
dfs.namenode.enable.retrycache	true	This enables the retry cache on the namenode. Namenode tracks for non-idempotent requests the corresponding response. If a client retries the request, the response from the retry cache is sent. Such operations are tagged with annotation @AtMostOnce in namenode protocols. It is recommended that this flag be set to true. Setting it to false, will result in clients getting failure responses to retried request. This flag must be enabled in HA setup for transparent fail-overs. The entries in the cache have expiration time configurable using dfs.namenode.retrycache.expirytime.millis.
dfs.namenode.retrycache.expirytime.millis	600000	The time for which retry cache entries are retained.
dfs.namenode.retrycache.heap.percent	0.03f	This parameter configures the heap size allocated for retry cache (excluding the response cached). This corresponds to approximately 4096 entries for every 64MB of namenode process java heap size. Assuming retry cache entry expiration time (configured using dfs.namenode.retrycache.expirytime.millis) of 10 minutes, this enables retry cache to support 7 operations per second sustained for 10 minutes. As the heap size is increased, the operation rate linearly increases.
dfs.client.mmap.enabled	true	If this is set to false, the client won’t attempt to perform memory-mapped reads.
dfs.client.mmap.cache.size	256	When zero-copy reads are used, the DFSClient keeps a cache of recently used memory mapped regions. This parameter controls the maximum number of entries that we will keep in that cache. The larger this number is, the more file descriptors we will potentially use for memory-mapped files. mmaped files also use virtual address space. You may need to increase your ulimit virtual address space limits before increasing the client mmap cache size. Note that you can still do zero-copy reads when this size is set to 0.
dfs.client.mmap.cache.timeout.ms	3600000	The minimum length of time that we will keep an mmap entry in the cache between uses. If an entry is in the cache longer than this, and nobody uses it, it will be removed by a background thread.
dfs.client.mmap.retry.timeout.ms	300000	The minimum amount of time that we will wait before retrying a failed mmap operation.
dfs.client.short.circuit.replica.stale.threshold.ms	1800000	The maximum amount of time that we will consider a short-circuit replica to be valid, if there is no communication from the DataNode. After this time has elapsed, we will re-fetch the short-circuit replica even if it is in the cache.
dfs.namenode.path.based.cache.block.map.allocation.percent	0.25	The percentage of the Java heap which we will allocate to the cached blocks map. The cached blocks map is a hash map which uses chained hashing. Smaller maps may be accessed more slowly if the number of cached blocks is large; larger maps will consume more memory.
dfs.datanode.max.locked.memory	0	The amount of memory in bytes to use for caching of block replicas in memory on the datanode. The datanode’s maximum locked memory soft ulimit (RLIMIT_MEMLOCK) must be set to at least this value, else the datanode will abort on startup. By default, this parameter is set to 0, which disables in-memory caching. If the native libraries are not available to the DataNode, this configuration has no effect.
dfs.namenode.list.cache.directives.num.responses	100	This value controls the number of cache directives that the NameNode will send over the wire in response to a listDirectives RPC.
dfs.namenode.list.cache.pools.num.responses	100	This value controls the number of cache pools that the NameNode will send over the wire in response to a listPools RPC.
dfs.namenode.path.based.cache.refresh.interval.ms	30000	The amount of milliseconds between subsequent path cache rescans. Path cache rescans are when we calculate which blocks should be cached, and on what datanodes. By default, this parameter is set to 30 seconds.
dfs.namenode.path.based.cache.retry.interval.ms	30000	When the NameNode needs to uncache something that is cached, or cache something that is not cached, it must direct the DataNodes to do so by sending a DNA_CACHE or DNA_UNCACHE command in response to a DataNode heartbeat. This parameter controls how frequently the NameNode will resend these commands.
dfs.datanode.fsdatasetcache.max.threads.per.volume	4	The maximum number of threads per volume to use for caching new data on the datanode. These threads consume both I/O and CPU. This can affect normal datanode operations.
dfs.cachereport.intervalMsec	10000	Determines cache reporting interval in milliseconds. After this amount of time, the DataNode sends a full report of its cache state to the NameNode. The NameNode uses the cache report to update its map of cached blocks to DataNode locations. This configuration has no effect if in-memory caching has been disabled by setting dfs.datanode.max.locked.memory to 0 (which is the default). If the native libraries are not available to the DataNode, this configuration has no effect.
dfs.namenode.edit.log.autoroll.multiplier.threshold	2.0	Determines when an active namenode will roll its own edit log. The actual threshold (in number of edits) is determined by multiplying this value by dfs.namenode.checkpoint.txns. This prevents extremely large edit files from accumulating on the active namenode, which can cause timeouts during namenode startup and pose an administrative hassle. This behavior is intended as a failsafe for when the standby or secondary namenode fail to roll the edit log by the normal checkpoint threshold.
dfs.namenode.edit.log.autoroll.check.interval.ms	300000	How often an active namenode will check if it needs to roll its edit log, in milliseconds.
dfs.webhdfs.user.provider.user.pattern	^[A-Za-z_][A-Za-z0-9._-]*[$]?$	Valid pattern for user and group names for webhdfs, it must be a valid java regex.
dfs.client.context	default	The name of the DFSClient context that we should use. Clients that share a context share a socket cache and short-circuit cache, among other things. You should only change this if you don’t want to share with another set of threads.
dfs.client.read.shortcircuit	false	This configuration parameter turns on short-circuit local reads.
dfs.domain.socket.path		Optional. This is a path to a UNIX domain socket that will be used for communication between the DataNode and local HDFS clients. If the string “_PORT” is present in this path, it will be replaced by the TCP port of the DataNode.
dfs.client.read.shortcircuit.skip.checksum	false	If this configuration parameter is set, short-circuit local reads will skip checksums. This is normally not recommended, but it may be useful for special setups. You might consider using this if you are doing your own checksumming outside of HDFS.
dfs.client.read.shortcircuit.streams.cache.size	256	The DFSClient maintains a cache of recently opened file descriptors. This parameter controls the size of that cache. Setting this higher will use more file descriptors, but potentially provide better performance on workloads involving lots of seeks.
dfs.client.read.shortcircuit.streams.cache.expiry.ms	300000	This controls the minimum amount of time file descriptors need to sit in the client cache context before they can be closed for being inactive for too long.
dfs.datanode.shared.file.descriptor.paths	/dev/shm,/tmp	Comma separated paths to the directory on which shared memory segments are created. The client and the DataNode exchange information via this shared memory segment. It tries paths in order until creation of shared memory segment succeeds.
dfs.namenode.audit.log.debug.cmdlist		A comma separated list of NameNode commands that are written to the HDFS namenode audit log only if the audit log level is debug.
dfs.client.use.legacy.blockreader.local	false	Legacy short-circuit reader implementation based on HDFS-2246 is used if this configuration parameter is true. This is for the platforms other than Linux where the new implementation based on HDFS-347 is not available.
dfs.block.local-path-access.user		Comma separated list of the users allowd to open block files on legacy short-circuit local read.
dfs.client.domain.socket.data.traffic	false	This control whether we will try to pass normal data traffic over UNIX domain socket rather than over TCP socket on node-local data transfer. This is currently experimental and turned off by default.
dfs.namenode.reject-unresolved-dn-topology-mapping	false	If the value is set to true, then namenode will reject datanode registration if the topology mapping for a datanode is not resolved and NULL is returned (script defined by net.topology.script.file.name fails to execute). Otherwise, datanode will be registered and the default rack will be assigned as the topology path. Topology paths are important for data resiliency, since they define fault domains. Thus it may be unwanted behavior to allow datanode registration with the default rack if the resolving topology failed.
dfs.client.slow.io.warning.threshold.ms	30000	The threshold in milliseconds at which we will log a slow io warning in a dfsclient. By default, this parameter is set to 30000 milliseconds (30 seconds).
dfs.datanode.slow.io.warning.threshold.ms	300	The threshold in milliseconds at which we will log a slow io warning in a datanode. By default, this parameter is set to 300 milliseconds.
dfs.namenode.xattrs.enabled	true	Whether support for extended attributes is enabled on the NameNode.
dfs.namenode.fs-limits.max-xattrs-per-inode	32	Maximum number of extended attributes per inode.
dfs.namenode.fs-limits.max-xattr-size	16384	The maximum combined size of the name and value of an extended attribute in bytes.
dfs.namenode.startup.delay.block.deletion.sec	0	The delay in seconds at which we will pause the blocks deletion after Namenode startup. By default it’s disabled. In the case a directory has large number of directories and files are deleted, suggested delay is one hour to give the administrator enough time to notice large number of pending deletion blocks and take corrective action.
dfs.namenode.list.encryption.zones.num.responses	100	When listing encryption zones, the maximum number of zones that will be returned in a batch. Fetching the list incrementally in batches improves namenode performance.
dfs.namenode.inotify.max.events.per.rpc	1000	Maximum number of events that will be sent to an inotify client in a single RPC response. The default value attempts to amortize away the overhead for this RPC while avoiding huge memory requirements for the client and NameNode (1000 events should consume no more than 1 MB.)
dfs.user.home.dir.prefix	/user	The directory to prepend to user name to get the user’s home direcotry.
dfs.datanode.cache.revocation.timeout.ms	900000	When the DFSClient reads from a block file which the DataNode is caching, the DFSClient can skip verifying checksums. The DataNode will keep the block file in cache until the client is done. If the client takes an unusually long time, though, the DataNode may need to evict the block file from the cache anyway. This value controls how long the DataNode will wait for the client to release a replica that it is reading without checksums.
dfs.datanode.cache.revocation.polling.ms	500	How often the DataNode should poll to see if the clients have stopped using a replica that the DataNode wants to uncache.
dfs.datanode.block.id.layout.upgrade.threads	12	The number of threads to use when creating hard links from current to previous blocks during upgrade of a DataNode to block ID-based block layout (see HDFS-6482 for details on the layout).
dfs.encryption.key.provider.uri		The KeyProvider to use when interacting with encryption keys used when reading and writing to an encryption zone.
dfs.storage.policy.enabled	true	Allow users to change the storage policy on files and directories.
dfs.namenode.legacy-oiv-image.dir		Determines where to save the namespace in the old fsimage format during checkpointing by standby NameNode or SecondaryNameNode. Users can dump the contents of the old format fsimage by oiv_legacy command. If the value is not specified, old format fsimage will not be saved in checkpoint.
dfs.namenode.top.enabled	true	Enable nntop: reporting top users on namenode
dfs.namenode.top.window.num.buckets	10	Number of buckets in the rolling window implementation of nntop
dfs.namenode.top.num.users	10	Number of top users returned by the top tool
dfs.namenode.top.windows.minutes	1,5,25	comma separated list of nntop reporting periods in minutes
dfs.namenode.blocks.per.postponedblocks.rescan	10000	Number of blocks to rescan for each iteration of postponedMisreplicatedBlocks.
dfs.ha.zkfc.nn.http.timeout.ms	20000	The HTTP connection and read timeout value (unit is ms ) when DFS ZKFC tries to get local NN thread dump after local NN becomes SERVICE_NOT_RESPONDING or SERVICE_UNHEALTHY. If it is set to zero, DFS ZKFC won’t get local NN thread dump.
dfs.datanode.block-pinning.enabled	false	Whether pin blocks on favored DataNode.
dfs.datanode.bp-ready.timeout	20	The maximum wait time for datanode to be ready before failing the received request. Setting this to 0 fails requests right away if the datanode is not yet registered with the namenode. This wait time reduces initial request failures after datanode restart.

mapred-site.xml MapReduce配置文件

name	default	description
mapreduce.jobtracker.jobhistory.location		If job tracker is static the history files are stored in this single well known place. If No value is set here, by default, it is in the local file system at ${hadoop.log.dir}/history.
mapreduce.jobtracker.jobhistory.task.numberprogresssplits	12	Every task attempt progresses from 0.0 to 1.0 [unless it fails or is killed]. We record, for each task attempt, certain statistics over each twelfth of the progress range. You can change the number of intervals we divide the entire range of progress into by setting this property. Higher values give more precision to the recorded data, but costs more memory in the job tracker at runtime. Each increment in this attribute costs 16 bytes per running task.
mapreduce.job.userhistorylocation		User can specify a location to store the history files of a particular job. If nothing is specified, the logs are stored in output directory. The files are stored in “_logs/history/” in the directory. User can stop logging by giving the value “none”.
mapreduce.jobtracker.jobhistory.completed.location		The completed job history files are stored at this single well known location. If nothing is specified, the files are stored at ${mapreduce.jobtracker.jobhistory.location}/done.
mapreduce.job.committer.setup.cleanup.needed	true	true, if job needs job-setup and job-cleanup. false, otherwise
*mapreduce.task.io.sort.factor	10	对文件进行排序时同时合并的流的数量。 这决定了打开文件句柄的数量。 打开的文件越多，不一定就越快，所以要根据数据情况适当的调整。
*mapreduce.task.io.sort.mb	100	在对文件进行排序时使用的缓冲区内存总量，以MB为单位。 默认情况下，给每个合并流1MB应该是最小需求.
*mapreduce.map.sort.spill.percent	0.80	序列化缓冲区中的软限制。 一旦到达指定位置，线程将开始将内容溢出到后台的磁盘。 请注意，如果超出此阈值，则收集将不会被阻塞，而泄漏已经在进行中，因此，当它被设置为小于0.5时，溢出可能大于这个阈值。
mapreduce.jobtracker.address	local	The host and port that the MapReduce job tracker runs at. If “local”, then jobs are run in-process as a single map and reduce task.
mapreduce.local.clientfactory.class.name	org.apache.hadoop.mapred.LocalClientFactory	This the client factory that is responsible for creating local job runner client
mapreduce.jobtracker.http.address	0.0.0.0:50030	The job tracker http server address and port the server will listen on. If the port is 0 then the server will start on a free port.
mapreduce.jobtracker.handler.count	10	JobTracker的服务器线程数。这大约是tasktracker节点数量的4%。
mapreduce.tasktracker.report.address	127.0.0.1:0	The interface and port that task tracker server listens on. Since it is only connected to by the tasks, it uses the local interface. EXPERT ONLY. Should only be changed if your host does not have the loopback interface.
mapreduce.cluster.local.dir	${hadoop.tmp.dir}/mapred/local	The local directory where MapReduce stores intermediate data files. May be a comma-separated list of directories on different devices in order to spread disk i/o. Directories that do not exist are ignored.
mapreduce.jobtracker.system.dir	${hadoop.tmp.dir}/mapred/system	The directory where MapReduce stores control files.
mapreduce.jobtracker.staging.root.dir	${hadoop.tmp.dir}/mapred/staging	The root of the staging area for users’ job files In practice, this should be the directory where users’ home directories are located (usually /user)
mapreduce.cluster.temp.dir	${hadoop.tmp.dir}/mapred/temp	A shared directory for temporary files.
mapreduce.tasktracker.local.dir.minspacestart	0	If the space in mapreduce.cluster.local.dir drops under this, do not ask for more tasks. Value in bytes.
mapreduce.tasktracker.local.dir.minspacekill	0	If the space in mapreduce.cluster.local.dir drops under this, do not ask more tasks until all the current ones have finished and cleaned up. Also, to save the rest of the tasks we have running, kill one of them, to clean up some space. Start with the reduce tasks, then go with the ones that have finished the least. Value in bytes.
mapreduce.jobtracker.expire.trackers.interval	600000	Expert: The time-interval, in miliseconds, after which a tasktracker is declared ‘lost’ if it doesn’t send heartbeats.
mapreduce.tasktracker.instrumentation	org.apache.hadoop.mapred.TaskTrackerMetricsInst	Expert: The instrumentation class to associate with each TaskTracker.
mapreduce.tasktracker.resourcecalculatorplugin		Name of the class whose instance will be used to query resource information on the tasktracker. The class must be an instance of org.apache.hadoop.util.ResourceCalculatorPlugin. If the value is null, the tasktracker attempts to use a class appropriate to the platform. Currently, the only platform supported is Linux.
mapreduce.tasktracker.taskmemorymanager.monitoringinterval	5000	The interval, in milliseconds, for which the tasktracker waits between two cycles of monitoring its tasks’ memory usage. Used only if tasks’ memory management is enabled via mapred.tasktracker.tasks.maxmemory.
mapreduce.tasktracker.tasks.sleeptimebeforesigkill	5000	The time, in milliseconds, the tasktracker waits for sending a SIGKILL to a task, after it has been sent a SIGTERM. This is currently not used on WINDOWS where tasks are just sent a SIGTERM.
mapreduce.job.maps	2	The default number of map tasks per job. Ignored when mapreduce.jobtracker.address is “local”.
mapreduce.job.reduces	1	The default number of reduce tasks per job. Typically set to 99% of the cluster’s reduce capacity, so that if a node fails the reduces can still be executed in a single wave. Ignored when mapreduce.jobtracker.address is “local”.
mapreduce.jobtracker.restart.recover	false	“true” to enable (job) recovery upon restart, “false” to start afresh
mapreduce.jobtracker.jobhistory.block.size	3145728	The block size of the job history file. Since the job recovery uses job history, its important to dump job history to disk as soon as possible. Note that this is an expert level parameter. The default value is set to 3 MB.
mapreduce.job.reducer.preempt.delay.sec	0	The threshold (in seconds) after which an unsatisfied mapper request triggers reducer preemption when there is no anticipated headroom. If set to 0 or a negative value, the reducer is preempted as soon as lack of headroom is detected. Default is 0.
mapreduce.job.reducer.unconditional-preempt.delay.sec	300	The threshold (in seconds) after which an unsatisfied mapper request triggers a forced reducer preemption irrespective of the anticipated headroom. By default, it is set to 5 mins. Setting it to 0 leads to immediate reducer preemption. Setting to -1 disables this preemption altogether.
mapreduce.job.max.split.locations	10	The max number of block locations to store for each split for locality calculation.
mapreduce.job.split.metainfo.maxsize	10000000	The maximum permissible size of the split metainfo file. The JobTracker won’t attempt to read split metainfo files bigger than the configured value. No limits if set to -1.
mapreduce.jobtracker.taskscheduler.maxrunningtasks.perjob		The maximum number of running tasks for a job before it gets preempted. No limits if undefined.
mapreduce.map.maxattempts	4	Expert: The maximum number of attempts per map task. In other words, framework will try to execute a map task these many number of times before giving up on it.
mapreduce.reduce.maxattempts	4	Expert: The maximum number of attempts per reduce task. In other words, framework will try to execute a reduce task these many number of times before giving up on it.
mapreduce.reduce.shuffle.fetch.retry.enabled	${yarn.nodemanager.recovery.enabled}	Set to enable fetch retry during host restart.
mapreduce.reduce.shuffle.fetch.retry.interval-ms	1000	Time of interval that fetcher retry to fetch again when some non-fatal failure happens because of some events like NM restart.
mapreduce.reduce.shuffle.fetch.retry.timeout-ms	30000	Timeout value for fetcher to retry to fetch again when some non-fatal failure happens because of some events like NM restart.
mapreduce.reduce.shuffle.retry-delay.max.ms	60000	The maximum number of ms the reducer will delay before retrying to download map data.
*mapreduce.reduce.shuffle.parallelcopies	5	在复制(shuffle)阶段reduce并行传输的默认数量。对于较大集群，可调整为16~25。
mapreduce.reduce.shuffle.connect.timeout	180000	Expert: The maximum amount of time (in milli seconds) reduce task spends in trying to connect to a tasktracker for getting map output.
mapreduce.reduce.shuffle.read.timeout	180000	Expert: The maximum amount of time (in milli seconds) reduce task waits for map output data to be available for reading after obtaining connection.
mapreduce.shuffle.connection-keep-alive.enable	false	set to true to support keep-alive connections.
mapreduce.shuffle.connection-keep-alive.timeout	5	The number of seconds a shuffle client attempts to retain http connection. Refer “Keep-Alive: timeout=” header in Http specification
mapreduce.task.timeout	600000	The number of milliseconds before a task will be terminated if it neither reads an input, writes an output, nor updates its status string. A value of 0 disables the timeout.
*mapreduce.tasktracker.map.tasks.maximum	2	map任务的最大数量将被一个task tracker同时运行。
*mapreduce.tasktracker.reduce.tasks.maximum	2	reduce任务的最大数量将被一个task tracker同时运行。
mapreduce.map.memory.mb	-1	The amount of memory to request from the scheduler for each map task. If this is not specified or is non-positive, it is inferred from mapreduce.map.java.opts and mapreduce.job.heap.memory-mb.ratio. If java-opts are also not specified, we set it to 1024.
mapreduce.map.cpu.vcores	1	The number of virtual cores to request from the scheduler for each map task.
mapreduce.reduce.memory.mb	-1	The amount of memory to request from the scheduler for each reduce task. If this is not specified or is non-positive, it is inferred from mapreduce.reduce.java.opts and mapreduce.job.heap.memory-mb.ratio. If java-opts are also not specified, we set it to 1024.
mapreduce.reduce.cpu.vcores	1	The number of virtual cores to request from the scheduler for each reduce task.
mapreduce.jobtracker.retiredjobs.cache.size	1000	The number of retired job status to keep in the cache.
*mapreduce.tasktracker.outofband.heartbeat	false	专家:当把这个设置为true时，任务跟踪器在任务完成时发送一个带外心跳，立即通知jobtracker，以便能够马上分配任务以获得更好的延迟。
mapreduce.jobtracker.jobhistory.lru.cache.size	5	The number of job history files loaded in memory. The jobs are loaded when they are first accessed. The cache is cleared based on LRU.
mapreduce.jobtracker.instrumentation	org.apache.hadoop.mapred.JobTrackerMetricsInst	Expert: The instrumentation class to associate with each JobTracker.
*mapred.child.java.opts		用于任务流程的Java相关配置，需从应用程序角度进行配置。 下面的符号，如果存在，将被内插: @taskid@ 被当前的TaskID取代。 任何其他的 ‘@’ 出现将不会改变. 例如, 要启用详细gc日志记录到一个文件名为tmp的文件夹并且配置最大对内存为1G, 通过这样配置: -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc 配置 -Djava.library.path 使用hadoop本地库会导致程序不再起作用。 These values should instead be set as part of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and mapreduce.reduce.env config settings. If -Xmx is not set, it is inferred from mapreduce.{map|reduce}.memory.mb and mapreduce.job.heap.memory-mb.ratio.
mapred.child.env		User added environment variables for the task processes. Example : 1) A=foo This will set the env variable A to foo 2) B=$B:c This is inherit nodemanager’s B env variable on Unix. 3) B=%B%;c This is inherit nodemanager’s B env variable on Windows.
mapreduce.admin.user.env		Expert: Additional execution environment entries for map and reduce task processes. This is not an additive property. You must preserve the original value if you want your map and reduce tasks to have access to native libraries (compression, etc). When this value is empty, the command to set execution envrionment will be OS dependent: For linux, use LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native. For windows, use PATH = %PATH%;%HADOOP_COMMON_HOME%\bin.
mapreduce.task.tmp.dir	./tmp	To set the value of tmp directory for map and reduce tasks. If the value is an absolute path, it is directly assigned. Otherwise, it is prepended with task’s working directory. The java tasks are executed with option -Djava.io.tmpdir=’the absolute path of the tmp dir’. Pipes and streaming are set with environment variable, TMPDIR=’the absolute path of the tmp dir’
mapreduce.map.log.level	INFO	The logging level for the map task. The allowed levels are: OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL.
mapreduce.reduce.log.level	INFO	The logging level for the reduce task. The allowed levels are: OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL.
mapreduce.map.cpu.vcores	1	The number of virtual cores required for each map task.
mapreduce.reduce.cpu.vcores	1	The number of virtual cores required for each reduce task.
mapreduce.reduce.merge.inmem.threshold	1000	The threshold, in terms of the number of files for the in-memory merge process. When we accumulate threshold number of files we initiate the in-memory merge and spill to disk. A value of 0 or less than 0 indicates we want to DON’T have any threshold and instead depend only on the ramfs’s memory consumption to trigger the merge.
mapreduce.reduce.shuffle.merge.percent	0.66	The usage threshold at which an in-memory merge will be initiated, expressed as a percentage of the total memory allocated to storing in-memory map outputs, as defined by mapreduce.reduce.shuffle.input.buffer.percent.
mapreduce.reduce.shuffle.input.buffer.percent	0.70	从最大堆内存中分配的用于在shuffle阶段存储map输出的内存百分比。
mapreduce.reduce.input.buffer.percent	0.0	The percentage of memory- relative to the maximum heap size- to retain map outputs during the reduce. When the shuffle is concluded, any remaining map outputs in memory must consume less than this threshold before the reduce can begin.
mapreduce.reduce.shuffle.memory.limit.percent	0.25	Expert: Maximum percentage of the in-memory limit that a single shuffle can consume
mapreduce.shuffle.ssl.enabled	false	Whether to use SSL for for the Shuffle HTTP endpoints.
mapreduce.shuffle.ssl.file.buffer.size	65536	Buffer size for reading spills from file when using SSL.
mapreduce.shuffle.max.connections	0	最大允许进行shuffle的连接。设置为0表示连接数没有限制。
mapreduce.shuffle.max.threads	0	Max allowed threads for serving shuffle connections. Set to zero to indicate the default of 2 times the number of available processors (as reported by Runtime.availableProcessors()). Netty is used to serve requests, so a thread is not needed for each connection.
mapreduce.shuffle.transferTo.allowed		This option can enable/disable using nio transferTo method in the shuffle phase. NIO transferTo does not perform well on windows in the shuffle phase. Thus, with this configuration property it is possible to disable it, in which case custom transfer method will be used. Recommended value is false when running Hadoop on Windows. For Linux, it is recommended to set it to true. If nothing is set then the default value is false for Windows, and true for Linux.
mapreduce.shuffle.transfer.buffer.size	131072	This property is used only if mapreduce.shuffle.transferTo.allowed is set to false. In that case, this property defines the size of the buffer used in the buffer copy code for the shuffle phase. The size of this buffer determines the size of the IO requests.
mapreduce.reduce.markreset.buffer.percent	0.0	The percentage of memory -relative to the maximum heap size- to be used for caching values when using the mark-reset functionality.
mapreduce.map.speculative	true	如果是true, 表示可以并行执行一些映射任务的多个实例。
mapreduce.reduce.speculative	true	如果是true, 表示一些reduce任务的多个实例可以并行执行。
mapreduce.job.speculative.speculative-cap-running-tasks	0.1	The max percent (0-1) of running tasks that can be speculatively re-executed at any time.
mapreduce.job.speculative.speculative-cap-total-tasks	0.01	The max percent (0-1) of all tasks that can be speculatively re-executed at any time.
mapreduce.job.speculative.minimum-allowed-tasks	10	The minimum allowed tasks that can be speculatively re-executed at any time.
mapreduce.job.speculative.retry-after-no-speculate	1000	The waiting time(ms) to do next round of speculation if there is no task speculated in this round.
mapreduce.job.speculative.retry-after-speculate	15000	The waiting time(ms) to do next round of speculation if there are tasks speculated in this round.
mapreduce.job.map.output.collector.class	org.apache.hadoop.mapred.MapTask$MapOutputBuffer	The MapOutputCollector implementation(s) to use. This may be a comma-separated list of class names, in which case the map task will try to initialize each of the collectors in turn. The first to successfully initialize will be used.
mapreduce.job.speculative.slowtaskthreshold	1.0	The number of standard deviations by which a task’s ave progress-rates must be lower than the average of all running tasks’ for the task to be considered too slow.
mapreduce.job.jvm.numtasks	1	How many tasks to run per jvm. If set to -1, there is no limit.
mapreduce.job.ubertask.enable	false	Whether to enable the small-jobs “ubertask” optimization, which runs “sufficiently small” jobs sequentially within a single JVM. “Small” is defined by the following maxmaps, maxreduces, and maxbytes settings. Note that configurations for application masters also affect the “Small” definition - yarn.app.mapreduce.am.resource.mb must be larger than both mapreduce.map.memory.mb and mapreduce.reduce.memory.mb, and yarn.app.mapreduce.am.resource.cpu-vcores must be larger than both mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores to enable ubertask. Users may override this value.
mapreduce.job.ubertask.maxmaps	9	Threshold for number of maps, beyond which job is considered too big for the ubertasking optimization. Users may override this value, but only downward.
mapreduce.job.ubertask.maxreduces	1	Threshold for number of reduces, beyond which job is considered too big for the ubertasking optimization. CURRENTLY THE CODE CANNOT SUPPORT MORE THAN ONE REDUCE and will ignore larger values. (Zero is a valid max, however.) Users may override this value, but only downward.
mapreduce.job.ubertask.maxbytes		Threshold for number of input bytes, beyond which job is considered too big for the ubertasking optimization. If no value is specified, dfs.block.size is used as a default. Be sure to specify a default value in mapred-site.xml if the underlying filesystem is not HDFS. Users may override this value, but only downward.
mapreduce.job.emit-timeline-data	false	Specifies if the Application Master should emit timeline data to the timeline server. Individual jobs can override this value.
mapreduce.input.fileinputformat.split.minsize	0	The minimum size chunk that map input should be split into. Note that some file formats may have minimum split sizes that take priority over this setting.
mapreduce.input.fileinputformat.list-status.num-threads	1	The number of threads to use to list and fetch block locations for the specified input paths. Note: multiple threads should not be used if a custom non thread-safe path filter is used.
mapreduce.jobtracker.maxtasks.perjob	-1	The maximum number of tasks for a single job. A value of -1 indicates that there is no maximum.
mapreduce.input.lineinputformat.linespermap	1	When using NLineInputFormat, the number of lines of input data to include in each split.
mapreduce.client.submit.file.replication	10	The replication level for submitted job files. This should be around the square root of the number of nodes.
mapreduce.tasktracker.dns.interface	default	The name of the Network Interface from which a task tracker should report its IP address.
mapreduce.tasktracker.dns.nameserver	default	The host name or IP address of the name server (DNS) which a TaskTracker should use to determine the host name used by the JobTracker for communication and display purposes.
*mapreduce.tasktracker.http.threads	40	http服务的工作线程数。 运行在每个TaskTracker上，用于map task输出。 大集群，可以将其设为40~50。
mapreduce.tasktracker.http.address	0.0.0.0:50060	The task tracker http server address and port. If the port is 0 then the server will start on a free port.
mapreduce.task.files.preserve.failedtasks	false	Should the files for failed tasks be kept. This should only be used on jobs that are failing, because the storage is never reclaimed. It also prevents the map outputs from being erased from the reduce directory as they are consumed.
*mapreduce.output.fileoutputformat.compress	false	工作输出是否应该被压缩?
mapreduce.output.fileoutputformat.compress.type	RECORD	If the job outputs are to compressed as SequenceFiles, how should they be compressed? Should be one of NONE, RECORD or BLOCK.
mapreduce.output.fileoutputformat.compress.codec	org.apache.hadoop.io.compress.DefaultCodec	If the job outputs are compressed, how should they be compressed?
mapreduce.map.output.compress	false	Should the outputs of the maps be compressed before being sent across the network. Uses SequenceFile compression.
mapreduce.map.output.compress.codec	org.apache.hadoop.io.compress.DefaultCodec	If the map outputs are compressed, how should they be compressed?
map.sort.class	org.apache.hadoop.util.QuickSort	The default sort class for sorting keys.
mapreduce.task.userlog.limit.kb	0	The maximum size of user-logs of each task in KB. 0 disables the cap.
yarn.app.mapreduce.am.container.log.limit.kb	0	The maximum size of the MRAppMaster attempt container logs in KB. 0 disables the cap.
yarn.app.mapreduce.task.container.log.backups	0	Number of backup files for task logs when using ContainerRollingLogAppender (CRLA). See org.apache.log4j.RollingFileAppender.maxBackupIndex. By default, ContainerLogAppender (CLA) is used, and container logs are not rolled. CRLA is enabled for tasks when both mapreduce.task.userlog.limit.kb and yarn.app.mapreduce.task.container.log.backups are greater than zero.
yarn.app.mapreduce.am.container.log.backups	0	Number of backup files for the ApplicationMaster logs when using ContainerRollingLogAppender (CRLA). See org.apache.log4j.RollingFileAppender.maxBackupIndex. By default, ContainerLogAppender (CLA) is used, and container logs are not rolled. CRLA is enabled for the ApplicationMaster when both mapreduce.task.userlog.limit.kb and yarn.app.mapreduce.am.container.log.backups are greater than zero.
mapreduce.job.userlog.retain.hours	24	The maximum time, in hours, for which the user-logs are to be retained after the job completion.
mapreduce.jobtracker.hosts.filename		Names a file that contains the list of nodes that may connect to the jobtracker. If the value is empty, all hosts are permitted.
mapreduce.jobtracker.hosts.exclude.filename		Names a file that contains the list of hosts that should be excluded by the jobtracker. If the value is empty, no hosts are excluded.
mapreduce.jobtracker.heartbeats.in.second	100	Expert: Approximate number of heart-beats that could arrive at JobTracker in a second. Assuming each RPC can be processed in 10msec, the default value is made 100 RPCs in a second.
mapreduce.jobtracker.tasktracker.maxblacklists	4	The number of blacklists for a taskTracker by various jobs after which the task tracker could be blacklisted across all jobs. The tracker will be given a tasks later (after a day). The tracker will become a healthy tracker after a restart.
mapreduce.job.maxtaskfailures.per.tracker	3	The number of task-failures on a tasktracker of a given job after which new tasks of that job aren’t assigned to it. It MUST be less than mapreduce.map.maxattempts and mapreduce.reduce.maxattempts otherwise the failed task will never be tried on a different node.
mapreduce.client.output.filter	FAILED	The filter for controlling the output of the task’s userlogs sent to the console of the JobClient. The permissible options are: NONE, KILLED, FAILED, SUCCEEDED and ALL.
mapreduce.client.completion.pollinterval	5000	The interval (in milliseconds) between which the JobClient polls the JobTracker for updates about job status. You may want to set this to a lower value to make tests run faster on a single node system. Adjusting this value in production may lead to unwanted client-server traffic.
mapreduce.client.progressmonitor.pollinterval	1000	The interval (in milliseconds) between which the JobClient reports status to the console and checks for job completion. You may want to set this to a lower value to make tests run faster on a single node system. Adjusting this value in production may lead to unwanted client-server traffic.
mapreduce.jobtracker.persist.jobstatus.active	true	Indicates if persistency of job status information is active or not.
mapreduce.jobtracker.persist.jobstatus.hours	1	The number of hours job status information is persisted in DFS. The job status information will be available after it drops of the memory queue and between jobtracker restarts. With a zero value the job status information is not persisted at all in DFS.
mapreduce.jobtracker.persist.jobstatus.dir	/jobtracker/jobsInfo	The directory where the job status information is persisted in a file system to be available after it drops of the memory queue and between jobtracker restarts.
mapreduce.task.profile	false	To set whether the system should collect profiler information for some of the tasks in this job? The information is stored in the user log directory. The value is “true” if task profiling is enabled.
mapreduce.task.profile.maps	0-2	To set the ranges of map tasks to profile. mapreduce.task.profile has to be set to true for the value to be accounted.
mapreduce.task.profile.reduces	0-2	To set the ranges of reduce tasks to profile. mapreduce.task.profile has to be set to true for the value to be accounted.
mapreduce.task.profile.params	-agentlib:hprof=cpu=samples,heap=sites,force=n,thread=y,verbose=n,file=%s	JVM profiler parameters used to profile map and reduce task attempts. This string may contain a single format specifier %s that will be replaced by the path to profile.out in the task attempt log directory. To specify different profiling options for map tasks and reduce tasks, more specific parameters mapreduce.task.profile.map.params and mapreduce.task.profile.reduce.params should be used.
mapreduce.task.profile.map.params	${mapreduce.task.profile.params}	Map-task-specific JVM profiler parameters. See mapreduce.task.profile.params
mapreduce.task.profile.reduce.params	${mapreduce.task.profile.params}	Reduce-task-specific JVM profiler parameters. See mapreduce.task.profile.params
mapreduce.task.skip.start.attempts	2	The number of Task attempts AFTER which skip mode will be kicked off. When skip mode is kicked off, the tasks reports the range of records which it will process next, to the TaskTracker. So that on failures, TT knows which ones are possibly the bad records. On further executions, those are skipped.
mapreduce.job.skip.outdir		If no value is specified here, the skipped records are written to the output directory at _logs/skip. User can stop writing skipped records by giving the value “none”.
mapreduce.map.skip.maxrecords	0	The number of acceptable skip records surrounding the bad record PER bad record in mapper. The number includes the bad record as well. To turn the feature of detection/skipping of bad records off, set the value to 0. The framework tries to narrow down the skipped range by retrying until this threshold is met OR all attempts get exhausted for this task. Set the value to Long.MAX_VALUE to indicate that framework need not try to narrow down. Whatever records(depends on application) get skipped are acceptable.
mapreduce.reduce.skip.maxgroups	0	The number of acceptable skip groups surrounding the bad group PER bad group in reducer. The number includes the bad group as well. To turn the feature of detection/skipping of bad groups off, set the value to 0. The framework tries to narrow down the skipped range by retrying until this threshold is met OR all attempts get exhausted for this task. Set the value to Long.MAX_VALUE to indicate that framework need not try to narrow down. Whatever groups(depends on application) get skipped are acceptable.
mapreduce.ifile.readahead	true	Configuration key to enable/disable IFile readahead.
mapreduce.ifile.readahead.bytes	4194304	Configuration key to set the IFile readahead length in bytes.
mapreduce.jobtracker.taskcache.levels	2	This is the max level of the task cache. For example, if the level is 2, the tasks cached are at the host level and at the rack level.
mapreduce.job.queuename	default	Queue to which a job is submitted. This must match one of the queues defined in mapred-queues.xml for the system. Also, the ACL setup for the queue must allow the current user to submit a job to the queue. Before specifying a queue, ensure that the system is configured with the queue, and access is allowed for submitting jobs to the queue.
mapreduce.job.tags		Tags for the job that will be passed to YARN at submission time. Queries to YARN for applications can filter on these tags.
mapreduce.cluster.acls.enabled	false	Specifies whether ACLs should be checked for authorization of users for doing various queue and job level operations. ACLs are disabled by default. If enabled, access control checks are made by JobTracker and TaskTracker when requests are made by users for queue operations like submit job to a queue and kill a job in the queue and job operations like viewing the job-details (See mapreduce.job.acl-view-job) or for modifying the job (See mapreduce.job.acl-modify-job) using Map/Reduce APIs, RPCs or via the console and web user interfaces. For enabling this flag(mapreduce.cluster.acls.enabled), this is to be set to true in mapred-site.xml on JobTracker node and on all TaskTracker nodes.
mapreduce.job.acl-modify-job		Job specific access-control list for ‘modifying’ the job. It is only used if authorization is enabled in Map/Reduce by setting the configuration property mapreduce.cluster.acls.enabled to true. This specifies the list of users and/or groups who can do modification operations on the job. For specifying a list of users and groups the format to use is “user1,user2 group1,group”. If set to ‘*’, it allows all users/groups to modify this job. If set to ‘ ‘(i.e. space), it allows none. This configuration is used to guard all the modifications with respect to this job and takes care of all the following operations: o killing this job o killing a task of this job, failing a task of this job o setting the priority of this job Each of these operations are also protected by the per-queue level ACL “acl-administer-jobs” configured via mapred-queues.xml. So a caller should have the authorization to satisfy either the queue-level ACL or the job-level ACL. Irrespective of this ACL configuration, (a) job-owner, (b) the user who started the cluster, (c) members of an admin configured supergroup configured via mapreduce.cluster.permissions.supergroup and (d) queue administrators of the queue to which this job was submitted to configured via acl-administer-jobs for the specific queue in mapred-queues.xml can do all the modification operations on a job. By default, nobody else besides job-owner, the user who started the cluster, members of supergroup and queue administrators can perform modification operations on a job.
mapreduce.job.acl-view-job		Job specific access-control list for ‘viewing’ the job. It is only used if authorization is enabled in Map/Reduce by setting the configuration property mapreduce.cluster.acls.enabled to true. This specifies the list of users and/or groups who can view private details about the job. For specifying a list of users and groups the format to use is “user1,user2 group1,group”. If set to ‘*’, it allows all users/groups to modify this job. If set to ‘ ‘(i.e. space), it allows none. This configuration is used to guard some of the job-views and at present only protects APIs that can return possibly sensitive information of the job-owner like o job-level counters o task-level counters o tasks’ diagnostic information o task-logs displayed on the TaskTracker web-UI and o job.xml showed by the JobTracker’s web-UI Every other piece of information of jobs is still accessible by any other user, for e.g., JobStatus, JobProfile, list of jobs in the queue, etc. Irrespective of this ACL configuration, (a) job-owner, (b) the user who started the cluster, (c) members of an admin configured supergroup configured via mapreduce.cluster.permissions.supergroup and (d) queue administrators of the queue to which this job was submitted to configured via acl-administer-jobs for the specific queue in mapred-queues.xml can do all the view operations on a job. By default, nobody else besides job-owner, the user who started the cluster, memebers of supergroup and queue administrators can perform view operations on a job.
mapreduce.tasktracker.indexcache.mb	10	The maximum memory that a task tracker allows for the index cache that is used when serving map outputs to reducers.
mapreduce.job.token.tracking.ids.enabled	false	Whether to write tracking ids of tokens to job-conf. When true, the configuration property “mapreduce.job.token.tracking.ids” is set to the token-tracking-ids of the job
mapreduce.job.token.tracking.ids		When mapreduce.job.token.tracking.ids.enabled is set to true, this is set by the framework to the token-tracking-ids used by the job.
mapreduce.task.merge.progress.records	10000	The number of records to process during merge before sending a progress notification to the TaskTracker.
mapreduce.task.combine.progress.records	10000	The number of records to process during combine output collection before sending a progress notification.
mapreduce.job.reduce.slowstart.completedmaps	0.05	Fraction of the number of maps in the job which should be complete before reduces are scheduled for the job.
mapreduce.job.complete.cancel.delegation.tokens	true	if false - do not unregister/cancel delegation tokens from renewal, because same tokens may be used by spawned jobs
mapreduce.tasktracker.taskcontroller	org.apache.hadoop.mapred.DefaultTaskController	TaskController which is used to launch and manage task execution
mapreduce.tasktracker.group		Expert: Group to which TaskTracker belongs. If LinuxTaskController is configured via mapreduce.tasktracker.taskcontroller, the group owner of the task-controller binary should be same as this group.
mapreduce.shuffle.port	13562	Default port that the ShuffleHandler will run on. ShuffleHandler is a service run at the NodeManager to facilitate transfers of intermediate Map outputs to requesting Reducers.
mapreduce.job.reduce.shuffle.consumer.plugin.class	org.apache.hadoop.mapreduce.task.reduce.Shuffle	Name of the class whose instance will be used to send shuffle requests by reducetasks of this job. The class must be an instance of org.apache.hadoop.mapred.ShuffleConsumerPlugin.
mapreduce.tasktracker.healthchecker.script.path		Absolute path to the script which is periodicallyrun by the node health monitoring service to determine if the node is healthy or not. If the value of this key is empty or the file does not exist in the location configured here, the node health monitoring service is not started.
mapreduce.tasktracker.healthchecker.interval	60000	Frequency of the node health script to be run, in milliseconds
mapreduce.tasktracker.healthchecker.script.timeout	600000	Time after node health script should be killed if unresponsive and considered that the script has failed.
mapreduce.tasktracker.healthchecker.script.args		List of arguments which are to be passed to node health script when it is being launched comma seperated.
mapreduce.job.counters.limit	120	Limit on the number of user counters allowed per job.
mapreduce.framework.name	local	执行MapReduce作业的运行时框架。 可能的值为 local | classic | yarn.
yarn.app.mapreduce.am.staging-dir	/tmp/hadoop-yarn/staging	The staging dir used while submitting jobs.
mapreduce.am.max-attempts	2	The maximum number of application attempts. It is a application-specific setting. It should not be larger than the global number set by resourcemanager. Otherwise, it will be override. The default number is set to 2, to allow at least one retry for AM.
mapreduce.job.end-notification.url		Indicates url which will be called on completion of job to inform end status of job. User can give at most 2 variables with URI : $jobId and $jobStatus. If they are present in URI, then they will be replaced by their respective values.
mapreduce.job.end-notification.retry.attempts	0	The number of times the submitter of the job wants to retry job end notification if it fails. This is capped by mapreduce.job.end-notification.max.attempts
mapreduce.job.end-notification.retry.interval	1000	The number of milliseconds the submitter of the job wants to wait before job end notification is retried if it fails. This is capped by mapreduce.job.end-notification.max.retry.interval
mapreduce.job.end-notification.max.attempts	5	The maximum number of times a URL will be read for providing job end notification. Cluster administrators can set this to limit how long after end of a job, the Application Master waits before exiting. Must be marked as final to prevent users from overriding this.
mapreduce.job.end-notification.max.retry.interval	5000	The maximum amount of time (in milliseconds) to wait before retrying job end notification. Cluster administrators can set this to limit how long the Application Master waits before exiting. Must be marked as final to prevent users from overriding this.
yarn.app.mapreduce.am.env		User added environment variables for the MR App Master processes. Example : 1) A=foo This will set the env variable A to foo 2) B=$B:c This is inherit tasktracker’s B env variable.
yarn.app.mapreduce.am.admin.user.env		Environment variables for the MR App Master processes for admin purposes. These values are set first and can be overridden by the user env (yarn.app.mapreduce.am.env) Example : 1) A=foo This will set the env variable A to foo 2) B=$B:c This is inherit app master’s B env variable.
yarn.app.mapreduce.am.command-opts	-Xmx1024m	Java opts for the MR App Master processes. The following symbol, if present, will be interpolated: @taskid@ is replaced by current TaskID. Any other occurrences of ‘@’ will go unchanged. For example, to enable verbose gc logging to a file named for the taskid in /tmp and to set the heap maximum to be a gigabyte, pass a ‘value’ of: -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc Usage of -Djava.library.path can cause programs to no longer function if hadoop native libraries are used. These values should instead be set as part of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and mapreduce.reduce.env config settings.
yarn.app.mapreduce.am.admin-command-opts		Java opts for the MR App Master processes for admin purposes. It will appears before the opts set by yarn.app.mapreduce.am.command-opts and thus its options can be overridden user. Usage of -Djava.library.path can cause programs to no longer function if hadoop native libraries are used. These values should instead be set as part of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and mapreduce.reduce.env config settings.
yarn.app.mapreduce.am.job.task.listener.thread-count	30	The number of threads used to handle RPC calls in the MR AppMaster from remote tasks
yarn.app.mapreduce.am.job.client.port-range		Range of ports that the MapReduce AM can use when binding. Leave blank if you want all possible ports. For example 50000-50050,50100-50200
yarn.app.mapreduce.am.job.committer.cancel-timeout	60000	The amount of time in milliseconds to wait for the output committer to cancel an operation if the job is killed
yarn.app.mapreduce.am.job.committer.commit-window	10000	Defines a time window in milliseconds for output commit operations. If contact with the RM has occurred within this window then commits are allowed, otherwise the AM will not allow output commits until contact with the RM has been re-established.
mapreduce.fileoutputcommitter.algorithm.version	1	The file output committer algorithm version valid algorithm version number: 1 or 2 default to 1, which is the original algorithm In algorithm version 1, 1. commitTask will rename directory $joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID/ to $joboutput/_temporary/$appAttemptID/$taskID/ 2. recoverTask will also do a rename $joboutput/_temporary/$appAttemptID/$taskID/ to $joboutput/_temporary/($appAttemptID + 1)/$taskID/ 3. commitJob will merge every task output file in $joboutput/_temporary/$appAttemptID/$taskID/ to $joboutput/, then it will delete $joboutput/_temporary/ and write $joboutput/_SUCCESS It has a performance regression, which is discussed in MAPREDUCE-4815. If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. the commit is single-threaded and waits until all tasks have completed before commencing. algorithm version 2 will change the behavior of commitTask, recoverTask, and commitJob. 1. commitTask will rename all files in $joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID/ to $joboutput/ 2. recoverTask actually doesn’t require to do anything, but for upgrade from version 1 to version 2 case, it will check if there are any files in $joboutput/_temporary/($appAttemptID - 1)/$taskID/ and rename them to $joboutput/ 3. commitJob can simply delete $joboutput/_temporary and write $joboutput/_SUCCESS This algorithm will reduce the output commit time for large jobs by having the tasks commit directly to the final output directory as they were completing and commitJob had very little to do.
yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms	1000	MR AppMaster应该向ResourceManager发送以毫秒为单位的心跳时间间隔。
yarn.app.mapreduce.client-am.ipc.max-retries	3	The number of client retries to the AM - before reconnecting to the RM to fetch Application Status.
yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts	3	The number of client retries on socket timeouts to the AM - before reconnecting to the RM to fetch Application Status.
yarn.app.mapreduce.client.max-retries	3	The number of client retries to the RM/HS before throwing exception. This is a layer above the ipc.
yarn.app.mapreduce.am.resource.mb	1536	The amount of memory the MR AppMaster needs.
yarn.app.mapreduce.am.resource.cpu-vcores	1	The number of virtual CPU cores the MR AppMaster needs.
yarn.app.mapreduce.client.job.max-retries	0	The number of retries the client will make for getJob and dependent calls. The default is 0 as this is generally only needed for non-HDFS DFS where additional, high level retries are required to avoid spurious failures during the getJob call. 30 is a good value for WASB
yarn.app.mapreduce.client.job.retry-interval	2000	The delay between getJob retries in ms for retries configured with yarn.app.mapreduce.client.job.max-retries.
mapreduce.application.classpath		CLASSPATH for MR applications. A comma-separated list of CLASSPATH entries. If mapreduce.application.framework is set then this must specify the appropriate classpath for that archive, and the name of the archive must be present in the classpath. If mapreduce.app-submission.cross-platform is false, platform-specific environment vairable expansion syntax would be used to construct the default CLASSPATH entries. For Linux: $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*, $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*. For Windows: %HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/*, %HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/lib/*. If mapreduce.app-submission.cross-platform is true, platform-agnostic default CLASSPATH for MR applications would be used: /share/hadoop/mapreduce/*, /share/hadoop/mapreduce/lib/* Parameter expansion marker will be replaced by NodeManager on container launch based on the underlying OS accordingly.
mapreduce.app-submission.cross-platform	false	If enabled, user can submit an application cross-platform i.e. submit an application from a Windows client to a Linux/Unix server or vice versa.
mapreduce.application.framework.path		Path to the MapReduce framework archive. If set, the framework archive will automatically be distributed along with the job, and this path would normally reside in a public location in an HDFS filesystem. As with distributed cache files, this can be a URL with a fragment specifying the alias to use for the archive name. For example, hdfs:/mapred/framework/hadoop-mapreduce-2.1.1.tar.gz#mrframework would alias the localized archive as “mrframework”. Note that mapreduce.application.classpath must include the appropriate classpath for the specified framework. The base name of the archive, or alias of the archive if an alias is used, must appear in the specified classpath.
mapreduce.job.classloader	false	Whether to use a separate (isolated) classloader for user classes in the task JVM.
mapreduce.job.classloader.system.classes		Used to override the default definition of the system classes for the job classloader. The system classes are a comma-separated list of classes that should be loaded from the system classpath, not the user-supplied JARs, when mapreduce.job.classloader is enabled. Names ending in ‘.’ (period) are treated as package names, and names starting with a ‘-‘ are treated as negative matches.
mapreduce.jobhistory.address	0.0.0.0:10020	MapReduce JobHistory Server IPC host:port
mapreduce.jobhistory.webapp.address	0.0.0.0:19888	MapReduce JobHistory Server Web UI host:port
mapreduce.jobhistory.keytab	/etc/security/keytab/jhs.service.keytab	Location of the kerberos keytab file for the MapReduce JobHistory Server.
mapreduce.jobhistory.principal	jhs/_HOST@REALM.TLD	Kerberos principal name for the MapReduce JobHistory Server.
mapreduce.jobhistory.intermediate-done-dir	${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate
mapreduce.jobhistory.done-dir	${yarn.app.mapreduce.am.staging-dir}/history/done
mapreduce.jobhistory.cleaner.enable	true
mapreduce.jobhistory.cleaner.interval-ms	86400000	How often the job history cleaner checks for files to delete, in milliseconds. Defaults to 86400000 (one day). Files are only deleted if they are older than mapreduce.jobhistory.max-age-ms.
mapreduce.jobhistory.max-age-ms	604800000	Job history files older than this many milliseconds will be deleted when the history cleaner runs. Defaults to 604800000 (1 week).
mapreduce.jobhistory.client.thread-count	10	The number of threads to handle client API requests
mapreduce.jobhistory.datestring.cache.size	200000	Size of the date string cache. Effects the number of directories which will be scanned to find a job.
mapreduce.jobhistory.joblist.cache.size	20000	Size of the job list cache
mapreduce.jobhistory.loadedjobs.cache.size	5	Size of the loaded job cache. This property is ignored if the property mapreduce.jobhistory.loadedtasks.cache.size is set to a positive value.
mapreduce.jobhistory.loadedtasks.cache.size		Change the job history cache limit to be set in terms of total task count. If the total number of tasks loaded exceeds this value, then the job cache will be shrunk down until it is under this limit (minimum 1 job in cache). If this value is empty or nonpositive then the cache reverts to using the property mapreduce.jobhistory.loadedjobs.cache.size as a job cache size. Two recommendations for the mapreduce.jobhistory.loadedtasks.cache.size property: 1) For every 100k of cache size, set the heap size of the Job History Server to 1.2GB. For example, mapreduce.jobhistory.loadedtasks.cache.size=500000, heap size=6GB. 2) Make sure that the cache size is larger than the number of tasks required for the largest job run on the cluster. It might be a good idea to set the value slightly higher (say, 20%) in order to allow for job size growth.
mapreduce.jobhistory.move.interval-ms	180000	Scan for history files to more from intermediate done dir to done dir at this frequency.
mapreduce.jobhistory.move.thread-count	3	The number of threads used to move files.
mapreduce.jobhistory.store.class		The HistoryStorage class to use to cache history data.
mapreduce.jobhistory.minicluster.fixed.ports	false	Whether to use fixed ports with the minicluster
mapreduce.jobhistory.admin.address	0.0.0.0:10033	The address of the History server admin interface.
mapreduce.jobhistory.admin.acl	*	ACL of who can be admin of the History server.
mapreduce.jobhistory.recovery.enable	false	Enable the history server to store server state and recover server state upon startup. If enabled then mapreduce.jobhistory.recovery.store.class must be specified.
mapreduce.jobhistory.recovery.store.class	org.apache.hadoop.mapreduce.v2.hs.HistoryServerFileSystemStateStoreService	The HistoryServerStateStoreService class to store history server state for recovery.
mapreduce.jobhistory.recovery.store.fs.uri	${hadoop.tmp.dir}/mapred/history/recoverystore	The URI where history server state will be stored if HistoryServerFileSystemStateStoreService is configured as the recovery storage class.
mapreduce.jobhistory.http.policy	HTTP_ONLY	This configures the HTTP endpoint for JobHistoryServer web UI. The following values are supported: - HTTP_ONLY : Service is provided only on http - HTTPS_ONLY : Service is provided only on https
mapreduce.jobhistory.jhist.format	json	File format the AM will use when generating the .jhist file. Valid values are “json” for text output and “binary” for faster parsing.
mapreduce.job.heap.memory-mb.ratio	0.8	The ratio of heap-size to container-size. If no -Xmx is specified, it is calculated as (mapreduce.{map|reduce}.memory.mb * mapreduce.heap.memory-mb.ratio). If -Xmx is specified but not mapreduce.{map|reduce}.memory.mb, it is calculated as (heapSize / mapreduce.heap.memory-mb.ratio).
mapreduce.task.exit.timeout	60000	The number of milliseconds before a task will be terminated if it stays in finishing state for too long. After a task attempt completes from TaskUmbilicalProtocol’s point of view, it will be transitioned to finishing state. That will give a chance for the task to exit by itself.
mapreduce.task.exit.timeout.check-interval-ms	20000	The interval in milliseconds between which the MR framework checks if task attempts stay in finishing state for too long.
yarn.app.mapreduce.am.containerlauncher.threadpool-initial-size	10	The initial size of thread pool to launch containers in the app master.

yarn-site.xml yarn配置文件

name	default	description
yarn.ipc.client.factory.class		Factory to create client IPC classes.
yarn.ipc.server.factory.class		Factory to create server IPC classes.
yarn.ipc.record.factory.class		Factory to create serializeable records.
yarn.ipc.rpc.class	org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC	RPC class implementation
yarn.resourcemanager.hostname	0.0.0.0	The hostname of the RM.
yarn.resourcemanager.address	${yarn.resourcemanager.hostname}:8032	The address of the applications manager interface in the RM.
yarn.resourcemanager.bind-host		The actual address the server will bind to. If this optional address is set, the RPC and webapp servers will bind to this address and the port specified in yarn.resourcemanager.address and yarn.resourcemanager.webapp.address, respectively. This is most useful for making RM listen to all interfaces by setting to 0.0.0.0.
yarn.resourcemanager.client.thread-count	50	The number of threads used to handle applications manager requests.
yarn.am.liveness-monitor.expiry-interval-ms	600000	The expiry interval for application master reporting.
yarn.resourcemanager.principal		The Kerberos principal for the resource manager.
yarn.resourcemanager.scheduler.address	${yarn.resourcemanager.hostname}:8030	the scheduler interface 的 IP 地址。
yarn.resourcemanager.scheduler.client.thread-count	50	Number of threads to handle scheduler interface.
yarn.http.policy	HTTP_ONLY	This configures the HTTP endpoint for Yarn Daemons.The following values are supported: - HTTP_ONLY : Service is provided only on http - HTTPS_ONLY : Service is provided only on https
yarn.resourcemanager.webapp.address	${yarn.resourcemanager.hostname}:8088	the RM web application 的 IP 地址。
yarn.resourcemanager.webapp.https.address	${yarn.resourcemanager.hostname}:8090	The https adddress of the RM web application.
yarn.resourcemanager.resource-tracker.address	${yarn.resourcemanager.hostname}:8031
yarn.acl.enable	false	Are acls enabled.
yarn.admin.acl	*	ACL of who can be admin of the YARN cluster.
yarn.resourcemanager.admin.address	${yarn.resourcemanager.hostname}:8033	the RM admin interface的 IP 地址。
yarn.resourcemanager.admin.client.thread-count	1	Number of threads used to handle RM admin interface.
yarn.resourcemanager.connect.max-wait.ms	900000	Maximum time to wait to establish connection to ResourceManager.
yarn.resourcemanager.connect.retry-interval.ms	30000	How often to try connecting to the ResourceManager.
yarn.resourcemanager.am.max-attempts	2	The maximum number of application attempts. It’s a global setting for all application masters. Each application master can specify its individual maximum number of application attempts via the API, but the individual number cannot be more than the global upper bound. If it is, the resourcemanager will override it. The default number is set to 2, to allow at least one retry for AM.
yarn.resourcemanager.container.liveness-monitor.interval-ms	600000	How often to check that containers are still alive.
yarn.resourcemanager.keytab	/etc/krb5.keytab	The keytab for the resource manager.
yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled	true	Flag to enable override of the default kerberos authentication filter with the RM authentication filter to allow authentication using delegation tokens(fallback to kerberos if the tokens are missing). Only applicable when the http authentication type is kerberos.
yarn.nm.liveness-monitor.expiry-interval-ms	600000	How long to wait until a node manager is considered dead.
yarn.resourcemanager.nodes.include-path		Path to file with nodes to include.
yarn.resourcemanager.nodes.exclude-path		Path to file with nodes to exclude.
yarn.resourcemanager.resource-tracker.client.thread-count	50	Number of threads to handle resource tracker calls.
yarn.resourcemanager.scheduler.class	org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler	资源调度器使用的类.
*yarn.scheduler.minimum-allocation-mb	1024	最小容器内存。 内存请求低于此值不会生效, 将至少分配指定的值。
*yarn.scheduler.maximum-allocation-mb	8192	最大容器内存。 内存请求高于此值不会生效, 并且将被限制为这个值。
*yarn.scheduler.minimum-allocation-vcores	1	最小容器虚拟CPU内核数量。 低于此值的请求不会生效，指定的值将被分配到最小值。
*yarn.scheduler.maximum-allocation-vcores	4	最大容器虚拟CPU内核数量。 请求高于此值不会生效，并将被限制为该值。
yarn.resourcemanager.recovery.enabled	false	Enable RM to recover state after starting. 如果是true, then yarn.resourcemanager.store.class must be specified.
yarn.resourcemanager.work-preserving-recovery.enabled	false	Enable RM work preserving recovery. This configuration is private to YARN for experimenting the feature.
yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms	10000	Set the amount of time RM waits before allocating new containers on work-preserving-recovery. Such wait period gives RM a chance to settle down resyncing with NMs in the cluster on recovery, before assigning new containers to applications.
yarn.resourcemanager.store.class	org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore	The class to use as the persistent store. If org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore is used, the store is implicitly fenced; meaning a single ResourceManager is able to use the store at any point in time. More details on this implicit fencing, along with setting up appropriate ACLs is discussed under yarn.resourcemanager.zk-state-store.root-node.acl.
yarn.resourcemanager.state-store.max-completed-applications	${yarn.resourcemanager.max-completed-applications}	The maximum number of completed applications RM state store keeps, less than or equals to ${yarn.resourcemanager.max-completed-applications}. By default, it equals to ${yarn.resourcemanager.max-completed-applications}. This ensures that the applications kept in the state store are consistent with the applications remembered in RM memory. Any values larger than ${yarn.resourcemanager.max-completed-applications} will be reset to ${yarn.resourcemanager.max-completed-applications}. Note that this value impacts the RM recovery performance.Typically, a smaller value indicates better performance on RM recovery.
yarn.resourcemanager.zk-address		Host:Port of the ZooKeeper server to be used by the RM. This must be supplied when using the ZooKeeper based implementation of the RM state store and/or embedded automatic failover in a HA setting.
yarn.resourcemanager.zk-num-retries	1000	Number of times RM tries to connect to ZooKeeper.
yarn.resourcemanager.zk-retry-interval-ms	1000	Retry interval in milliseconds when connecting to ZooKeeper. When HA is enabled, the value here is NOT used. It is generated automatically from yarn.resourcemanager.zk-timeout-ms and yarn.resourcemanager.zk-num-retries.
yarn.resourcemanager.zk-state-store.parent-path	/rmstore	Full path of the ZooKeeper znode where RM state will be stored. This must be supplied when using org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore as the value for yarn.resourcemanager.store.class
yarn.resourcemanager.zk-timeout-ms	10000	ZooKeeper session timeout in milliseconds. Session expiration is managed by the ZooKeeper cluster itself, not by the client. This value is used by the cluster to determine when the client’s session expires. Expirations happens when the cluster does not hear from the client within the specified session timeout period (i.e. no heartbeat).
yarn.resourcemanager.zk-acl	world:anyone:rwcda	ACL’s to be used for ZooKeeper znodes.
yarn.resourcemanager.zk-state-store.root-node.acl		ACLs to be used for the root znode when using ZKRMStateStore in a HA scenario for fencing. ZKRMStateStore supports implicit fencing to allow a single ResourceManager write-access to the store. For fencing, the ResourceManagers in the cluster share read-write-admin privileges on the root node, but the Active ResourceManager claims exclusive create-delete permissions. By default, when this property is not set, we use the ACLs from yarn.resourcemanager.zk-acl for shared admin access and rm-address:random-number for username-based exclusive create-delete access. This property allows users to set ACLs of their choice instead of using the default mechanism. For fencing to work, the ACLs should be carefully set differently on each ResourceManger such that all the ResourceManagers have shared admin access and the Active ResourceManger takes over (exclusively) the create-delete access.
yarn.resourcemanager.zk-auth		Specify the auths to be used for the ACL’s specified in both the yarn.resourcemanager.zk-acl and yarn.resourcemanager.zk-state-store.root-node.acl properties. This takes a comma-separated list of authentication mechanisms, each of the form ‘scheme:auth’ (the same syntax used for the ‘addAuth’ command in the ZK CLI).
yarn.resourcemanager.fs.state-store.uri	${hadoop.tmp.dir}/yarn/system/rmstore	URI pointing to the location of the FileSystem path where RM state will be stored. This must be supplied when using org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore as the value for yarn.resourcemanager.store.class
yarn.resourcemanager.fs.state-store.retry-policy-spec	2000, 500	hdfs client retry policy specification. hdfs client retry is always enabled. Specified in pairs of sleep-time and number-of-retries and (t0, n0), (t1, n1), …, the first n0 retries sleep t0 milliseconds on average, the following n1 retries sleep t1 milliseconds on average, and so on.
yarn.resourcemanager.ha.enabled	false	启用RM高可用性。 如果启用 (1) RM启动时默认处于Standby（备用模式）,当被提示时转换到活动模式。 (2) RM集合中的节点在yarn.resourcemanager.ha.rm-ids中列出。 (3) 每个RM的id都来自于yarn.resourcemanager.ha.id。如果yarn.resourcemanager.ha.id是明确指定的，或者可以通过用本机地址匹配yarn.resourcemanager.address.{id}找到。 (4) 实际的物理地址来自于the pattern - {rpc-config}.{id}的配置。
yarn.resourcemanager.ha.automatic-failover.enabled	true	Enable automatic failover. By default, it is enabled only when HA is enabled
yarn.resourcemanager.ha.automatic-failover.embedded	true	Enable embedded automatic failover. By default, it is enabled only when HA is enabled. The embedded elector relies on the RM state store to handle fencing, and is primarily intended to be used in conjunction with ZKRMStateStore.
yarn.resourcemanager.ha.automatic-failover.zk-base-path	/yarn-leader-election	The base znode path to use for storing leader information, when using ZooKeeper based leader election.
yarn.resourcemanager.cluster-id		Name of the cluster. In a HA setting, this is used to ensure the RM participates in leader election for this cluster and ensures it does not affect other clusters
yarn.resourcemanager.ha.rm-ids		The list of RM nodes in the cluster when HA is enabled. See description of yarn.resourcemanager.ha .enabled for full details on how this is used.
yarn.resourcemanager.ha.id		The id (string) of the current RM. When HA is enabled, this is an optional config. The id of current RM can be set by explicitly specifying yarn.resourcemanager.ha.id or figured out by matching yarn.resourcemanager.address.{id} with local address See description of yarn.resourcemanager.ha.enabled for full details on how this is used.
yarn.client.failover-proxy-provider	org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider	When HA is enabled, the class to be used by Clients, AMs and NMs to failover to the Active RM. It should extend org.apache.hadoop.yarn.client.RMFailoverProxyProvider
yarn.client.failover-max-attempts		When HA is enabled, the max number of times FailoverProxyProvider should attempt failover. When set, this overrides the yarn.resourcemanager.connect.max-wait.ms. When not set, this is inferred from yarn.resourcemanager.connect.max-wait.ms.
yarn.client.failover-sleep-base-ms		When HA is enabled, the sleep base (in milliseconds) to be used for calculating the exponential delay between failovers. When set, this overrides the yarn.resourcemanager.connect.* settings. When not set, yarn.resourcemanager.connect.retry-interval.ms is used instead.
yarn.client.failover-sleep-max-ms		When HA is enabled, the maximum sleep time (in milliseconds) between failovers. When set, this overrides the yarn.resourcemanager.connect.* settings. When not set, yarn.resourcemanager.connect.retry-interval.ms is used instead.
yarn.client.failover-retries	0	When HA is enabled, the number of retries per attempt to connect to a ResourceManager. In other words, it is the ipc.client.connect.max.retries to be used during failover attempts
yarn.client.failover-retries-on-socket-timeouts	0	When HA is enabled, the number of retries per attempt to connect to a ResourceManager on socket timeouts. In other words, it is the ipc.client.connect.max.retries.on.timeouts to be used during failover attempts
yarn.resourcemanager.max-completed-applications	10000	The maximum number of completed applications RM keeps.
yarn.resourcemanager.delayed.delegation-token.removal-interval-ms	30000	Interval at which the delayed token removal thread runs
yarn.resourcemanager.proxy-user-privileges.enabled	false	如果是true, ResourceManager will have proxy-user privileges. Use case: In a secure cluster, YARN requires the user hdfs delegation-tokens to do localization and log-aggregation on behalf of the user. If this is set to true, ResourceManager is able to request new hdfs delegation tokens on behalf of the user. This is needed by long-running-service, because the hdfs tokens will eventually expire and YARN requires new valid tokens to do localization and log-aggregation. Note that to enable this use case, the corresponding HDFS NameNode has to configure ResourceManager as the proxy-user so that ResourceManager can itself ask for new tokens on behalf of the user when tokens are past their max-life-time.
yarn.resourcemanager.am-rm-tokens.master-key-rolling-interval-secs	86400	Interval for the roll over for the master key used to generate application tokens
yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs	86400	Interval for the roll over for the master key used to generate container tokens. It is expected to be much greater than yarn.nm.liveness-monitor.expiry-interval-ms and yarn.rm.container-allocation.expiry-interval-ms. Otherwise the behavior is undefined.
yarn.resourcemanager.nodemanagers.heartbeat-interval-ms	1000	The heart-beat interval in milliseconds for every NodeManager in the cluster.
yarn.resourcemanager.nodemanager.minimum.version	NONE	The minimum allowed version of a connecting nodemanager. The valid values are NONE (no version checking), EqualToRM (the nodemanager’s version is equal to or greater than the RM version), or a Version String.
yarn.resourcemanager.scheduler.monitor.enable	false	Enable a set of periodic monitors (specified in yarn.resourcemanager.scheduler.monitor.policies) that affect the scheduler.
yarn.resourcemanager.scheduler.monitor.policies	org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy	The list of SchedulingEditPolicy classes that interact with the scheduler. A particular module may be incompatible with the scheduler, other policies, or a configuration of either.
yarn.resourcemanager.configuration.provider-class	org.apache.hadoop.yarn.LocalConfigurationProvider	The class to use as the configuration provider. If org.apache.hadoop.yarn.LocalConfigurationProvider is used, the local configuration will be loaded. If org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider is used, the configuration which will be loaded should be uploaded to remote File system first.
yarn.resourcemanager.system-metrics-publisher.enabled	false	The setting that controls whether yarn system metrics is published on the timeline server or not by RM.
yarn.resourcemanager.system-metrics-publisher.dispatcher.pool-size	10	Number of worker threads that send the yarn system metrics data.
yarn.resourcemanager.max-log-aggregation-diagnostics-in-memory	10	Number of diagnostics/failure messages can be saved in RM for log aggregation. It also defines the number of diagnostics/failure messages can be shown in log aggregation web ui.
yarn.nodemanager.hostname	0.0.0.0	The hostname of the NM.
yarn.nodemanager.address	${yarn.nodemanager.hostname}:0	The address of the container manager in the NM.
yarn.nodemanager.bind-host		The actual address the server will bind to. If this optional address is set, the RPC and webapp servers will bind to this address and the port specified in yarn.nodemanager.address and yarn.nodemanager.webapp.address, respectively. This is most useful for making NM listen to all interfaces by setting to 0.0.0.0.
yarn.nodemanager.admin-env	MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX	Environment variables that should be forwarded from the NodeManager’s environment to the container’s.
yarn.nodemanager.env-whitelist	JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME	Environment variables that containers may override rather than use NodeManager’s default.
yarn.nodemanager.container-executor.class	org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor	who will execute(launch) the containers.
yarn.nodemanager.container-manager.thread-count	20	Number of threads container manager uses.
yarn.nodemanager.delete.thread-count	4	Number of threads used in cleanup.
yarn.nodemanager.delete.debug-delay-sec	0	Number of seconds after an application finishes before the nodemanager’s DeletionService will delete the application’s localized file directory and log directory. To diagnose Yarn application problems, set this property’s value large enough (for example, to 600 = 10 minutes) to permit examination of these directories. After changing the property’s value, you must restart the nodemanager in order for it to have an effect. The roots of Yarn applications’ work directories is configurable with the yarn.nodemanager.local-dirs property (see below), and the roots of the Yarn applications’ log directories is configurable with the yarn.nodemanager.log-dirs property (see also below).
yarn.nodemanager.keytab	/etc/krb5.keytab	Keytab for NM.
yarn.nodemanager.local-dirs	${hadoop.tmp.dir}/nm-local-dir	List of directories to store localized files in. An application’s localized file directory will be found in: ${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}. Individual containers’ work directories, called container_${contid}, will be subdirectories of this.
yarn.nodemanager.local-cache.max-files-per-directory	8192	It limits the maximum number of files which will be localized in a single local directory. If the limit is reached then sub-directories will be created and new files will be localized in them. If it is set to a value less than or equal to 36 [which are sub-directories (0-9 and then a-z)] then NodeManager will fail to start. For example; [for public cache] if this is configured with a value of 40 ( 4 files + 36 sub-directories) and the local-dir is “/tmp/local-dir1” then it will allow 4 files to be created directly inside “/tmp/local-dir1/filecache”. For files that are localized further it will create a sub-directory “0” inside “/tmp/local-dir1/filecache” and will localize files inside it until it becomes full. If a file is removed from a sub-directory that is marked full, then that sub-directory will be used back again to localize files.
yarn.nodemanager.localizer.address	${yarn.nodemanager.hostname}:8040	Address where the localizer IPC is.
yarn.nodemanager.localizer.cache.cleanup.interval-ms	600000	Interval in between cache cleanups.
yarn.nodemanager.localizer.cache.target-size-mb	10240	Target size of localizer cache in MB, per nodemanager. It is a target retention size that only includes resources with PUBLIC and PRIVATE visibility and excludes resources with APPLICATION visibility
yarn.nodemanager.localizer.client.thread-count	5	Number of threads to handle localization requests.
yarn.nodemanager.localizer.fetch.thread-count	4	Number of threads to use for localization fetching.
yarn.nodemanager.log-dirs	${yarn.log.dir}/userlogs	Where to store container logs. An application’s localized log directory will be found in ${yarn.nodemanager.log-dirs}/application_${appid}. Individual containers’ log directories will be below this, in directories named container_{$contid}. Each container directory will contain the files stderr, stdin, and syslog generated by that container.
yarn.log-aggregation-enable	false	Whether to enable log aggregation. Log aggregation collects each container’s logs and moves these logs onto a file-system, for e.g. HDFS, after the application completes. Users can configure the “yarn.nodemanager.remote-app-log-dir” and “yarn.nodemanager.remote-app-log-dir-suffix” properties to determine where these logs are moved to. Users can access the logs via the Application Timeline Server.
yarn.log-aggregation.retain-seconds	-1	How long to keep aggregation logs before deleting them. -1 disables. Be careful set this too small and you will spam the name node.
yarn.log-aggregation.retain-check-interval-seconds	-1	How long to wait between aggregated log retention checks. If set to 0 or a negative value then the value is computed as one-tenth of the aggregated log retention time. Be careful set this too small and you will spam the name node.
yarn.nodemanager.log.retain-seconds	10800	Time in seconds to retain user logs. Only applicable if log aggregation is disabled
yarn.nodemanager.remote-app-log-dir	/tmp/logs	Where to aggregate logs to.
yarn.nodemanager.remote-app-log-dir-suffix	logs	The remote log dir will be created at {yarn.nodemanager.remote-app-log-dir}/${user}/{thisParam}
*yarn.nodemanager.resource.memory-mb	8192	物理内存的数量，以MB为单位，分配给容器, 可以设置为物理机内存的80%。
yarn.nodemanager.pmem-check-enabled	true	Whether physical memory limits will be enforced for containers.
yarn.nodemanager.vmem-check-enabled	false	Whether virtual memory limits will be enforced for containers.
yarn.nodemanager.vmem-pmem-ratio	2.1	设置内存限制容器时虚拟内存和物理内存的比。 容器分配物理内存方面的表达,并且虚拟内存使用不得超过这个分配比例。 Container allocations are expressed in terms of physical memory, and virtual memory usage is allowed to exceed this allocation by this ratio.
yarn.nodemanager.resource.cpu-vcores	8	Number of vcores that can be allocated for containers. This is used by the RM scheduler when allocating resources for containers. This is not used to limit the number of physical cores used by YARN containers.
yarn.nodemanager.logaggregation.threadpool-size-max	100	Thread pool size for LogAggregationService in Node Manager.
yarn.nodemanager.resource.percentage-physical-cpu-limit	100	Percentage of CPU that can be allocated for containers. This setting allows users to limit the amount of CPU that YARN containers use. Currently functional only on Linux using cgroups. The default is to use 100% of CPU.
yarn.nodemanager.webapp.address	${yarn.nodemanager.hostname}:8042	NM Webapp address.
yarn.nodemanager.container-monitor.interval-ms	3000	How often to monitor containers.
yarn.nodemanager.container-monitor.resource-calculator.class		Class that calculates containers current resource utilization.
yarn.nodemanager.health-checker.interval-ms	600000	Frequency of running node health script.
yarn.nodemanager.health-checker.script.timeout-ms	1200000	Script time out period.
yarn.nodemanager.health-checker.script.path		The health check script to run.
yarn.nodemanager.health-checker.script.opts		The arguments to pass to the health check script.
yarn.nodemanager.disk-health-checker.interval-ms	120000	Frequency of running disk health checker code.
yarn.nodemanager.disk-health-checker.min-healthy-disks	0.25	The minimum fraction of number of disks to be healthy for the nodemanager to launch new containers. This correspond to both yarn-nodemanager.local-dirs and yarn.nodemanager.log-dirs. i.e. If there are less number of healthy local-dirs (or log-dirs) available, then new containers will not be launched on this node.
yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage	90.0	The maximum percentage of disk space utilization allowed after which a disk is marked as bad. Values can range from 0.0 to 100.0. If the value is greater than or equal to 100, the nodemanager will check for full disk. This applies to yarn-nodemanager.local-dirs and yarn.nodemanager.log-dirs.
yarn.nodemanager.disk-health-checker.disk-utilization-watermark-low-per-disk-percentage		The low threshold percentage of disk space used when a bad disk is marked as good. Values can range from 0.0 to 100.0. This applies to yarn-nodemanager.local-dirs and yarn.nodemanager.log-dirs. Note that if its value is more than yarn.nodemanager.disk-health-checker. max-disk-utilization-per-disk-percentage or not set, it will be set to the same value as yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage.
yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb	0	The minimum space that must be available on a disk for it to be used. This applies to yarn-nodemanager.local-dirs and yarn.nodemanager.log-dirs.
yarn.nodemanager.linux-container-executor.path		The path to the Linux container executor.
yarn.nodemanager.linux-container-executor.resources-handler.class	org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler	The class which should help the LCE handle resources.
yarn.nodemanager.linux-container-executor.cgroups.hierarchy	/hadoop-yarn	The cgroups hierarchy under which to place YARN proccesses (cannot contain commas). If yarn.nodemanager.linux-container-executor.cgroups.mount is false (that is, if cgroups have been pre-configured), then this cgroups hierarchy must already exist and be writable by the NodeManager user, otherwise the NodeManager may fail. Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler.
yarn.nodemanager.linux-container-executor.cgroups.mount	false	Whether the LCE should attempt to mount cgroups if not found. Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler.
yarn.nodemanager.linux-container-executor.cgroups.mount-path		Where the LCE should attempt to mount cgroups if not found. Common locations include /sys/fs/cgroup and /cgroup; the default location can vary depending on the Linux distribution in use. This path must exist before the NodeManager is launched. Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler, and yarn.nodemanager.linux-container-executor.cgroups.mount is true.
yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users	true	This determines which of the two modes that LCE should use on a non-secure cluster. If this value is set to true, then all containers will be launched as the user specified in yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user. If this value is set to false, then containers will run as the user who submitted the application.
yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user	nobody	The UNIX user that containers will run as when Linux-container-executor is used in nonsecure mode (a use case for this is using cgroups) if the yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users is set to true.
yarn.nodemanager.linux-container-executor.nonsecure-mode.user-pattern	^[.A-Za-z0-9][-@.A-Za-z0-9]{0,255}?[$]?$	The allowed pattern for UNIX user names enforced by Linux-container-executor when used in nonsecure mode (use case for this is using cgroups). The default value is taken from /usr/sbin/adduser
yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage	false	This flag determines whether apps should run with strict resource limits or be allowed to consume spare resources if they need them. For example, turning the flag on will restrict apps to use only their share of CPU, even if the node has spare CPU cycles. The default value is false i.e. use available resources. Please note that turning this flag on may reduce job throughput on the cluster.
yarn.nodemanager.log-aggregation.compression-type	none	T-file compression types used to compress aggregated logs.
yarn.nodemanager.principal		The kerberos principal for the node manager.
yarn.nodemanager.aux-services		有效的服务名称应该只包含a-zA-Z0-9_并且不能以数字开头。
yarn.nodemanager.sleep-delay-before-sigkill.ms	250	No. of ms to wait between sending a SIGTERM and SIGKILL to a container
yarn.nodemanager.process-kill-wait.ms	2000	Max time to wait for a process to come up when trying to cleanup a container
yarn.nodemanager.resourcemanager.minimum.version	NONE	The minimum allowed version of a resourcemanager that a nodemanager will connect to. The valid values are NONE (no version checking), EqualToNM (the resourcemanager’s version is equal to or greater than the NM version), or a Version String.
yarn.client.nodemanager-client-async.thread-pool-max-size	500	Max number of threads in NMClientAsync to process container management events
yarn.client.nodemanager-connect.max-wait-ms	900000	Max time to wait to establish a connection to NM
yarn.client.nodemanager-connect.retry-interval-ms	10000	Time interval between each attempt to connect to NM
yarn.client.max-cached-nodemanagers-proxies	0	Maximum number of proxy connections to cache for node managers. If set to a value greater than zero then the cache is enabled and the NMClient and MRAppMaster will cache the specified number of node manager proxies. There will be at max one proxy per node manager. Ex. configuring it to a value of 5 will make sure that client will at max have 5 proxies cached with 5 different node managers. These connections for these proxies will be timed out if idle for more than the system wide idle timeout period. Note that this could cause issues on large clusters as many connections could linger simultaneously and lead to a large number of connection threads. The token used for authentication will be used only at connection creation time. If a new token is received then the earlier connection should be closed in order to use the new token. This and (yarn.client.nodemanager-client-async.thread-pool-max-size) are related and should be in sync (no need for them to be equal). If the value of this property is zero then the connection cache is disabled and connections will use a zero idle timeout to prevent too many connection threads on large clusters.
yarn.nodemanager.recovery.enabled	false	Enable the node manager to recover after starting
yarn.nodemanager.recovery.dir	${hadoop.tmp.dir}/yarn-nm-recovery	The local filesystem directory in which the node manager will store state when recovery is enabled.
yarn.nodemanager.container-metrics.unregister-delay-ms	10000	The delay time ms to unregister container metrics after completion.
yarn.nodemanager.docker-container-executor.exec-name	/usr/bin/docker	Name or path to the Docker client.
yarn.nodemanager.aux-services.mapreduce_shuffle.class	org.apache.hadoop.mapred.ShuffleHandler
mapreduce.job.jar
mapreduce.job.hdfs-servers	${fs.defaultFS}
yarn.web-proxy.principal		The kerberos principal for the proxy, if the proxy is not running as part of the RM.
yarn.web-proxy.keytab		Keytab for WebAppProxy, if the proxy is not running as part of the RM.
yarn.web-proxy.address		The address for the web proxy as HOST:PORT, if this is not given then the proxy will run as part of the RM
yarn.application.classpath		CLASSPATH for YARN applications. A comma-separated list of CLASSPATH entries. When this value is empty, the following default CLASSPATH for YARN applications would be used. For Linux: $HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/share/hadoop/common/*, $HADOOP_COMMON_HOME/share/hadoop/common/lib/*, $HADOOP_HDFS_HOME/share/hadoop/hdfs/*, $HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*, $HADOOP_YARN_HOME/share/hadoop/yarn/*, $HADOOP_YARN_HOME/share/hadoop/yarn/lib/* For Windows: %HADOOP_CONF_DIR%, %HADOOP_COMMON_HOME%/share/hadoop/common/*, %HADOOP_COMMON_HOME%/share/hadoop/common/lib/*, %HADOOP_HDFS_HOME%/share/hadoop/hdfs/*, %HADOOP_HDFS_HOME%/share/hadoop/hdfs/lib/*, %HADOOP_YARN_HOME%/share/hadoop/yarn/*, %HADOOP_YARN_HOME%/share/hadoop/yarn/lib/*
yarn.timeline-service.enabled	false	Indicate to clients whether timeline service is enabled or not. If enabled, clients will put entities and events to the timeline server.
yarn.timeline-service.hostname	0.0.0.0	The hostname of the timeline service web application.
yarn.timeline-service.address	${yarn.timeline-service.hostname}:10200	This is default address for the timeline server to start the RPC server.
yarn.timeline-service.webapp.address	${yarn.timeline-service.hostname}:8188	The http address of the timeline service web application.
yarn.timeline-service.webapp.https.address	${yarn.timeline-service.hostname}:8190	The https address of the timeline service web application.
yarn.timeline-service.bind-host		The actual address the server will bind to. If this optional address is set, the RPC and webapp servers will bind to this address and the port specified in yarn.timeline-service.address and yarn.timeline-service.webapp.address, respectively. This is most useful for making the service listen to all interfaces by setting to 0.0.0.0.
yarn.timeline-service.store-class	org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore	Store class name for timeline store.
yarn.timeline-service.ttl-enable	true	Enable age off of timeline store data.
yarn.timeline-service.ttl-ms	604800000	Time to live for timeline store data in milliseconds.
yarn.timeline-service.leveldb-timeline-store.path	${hadoop.tmp.dir}/yarn/timeline	Store file name for leveldb timeline store.
yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms	300000	Length of time to wait between deletion cycles of leveldb timeline store in milliseconds.
yarn.timeline-service.leveldb-timeline-store.read-cache-size	104857600	Size of read cache for uncompressed blocks for leveldb timeline store in bytes.
yarn.timeline-service.leveldb-timeline-store.start-time-read-cache-size	10000	Size of cache for recently read entity start times for leveldb timeline store in number of entities.
yarn.timeline-service.leveldb-timeline-store.start-time-write-cache-size	10000	Size of cache for recently written entity start times for leveldb timeline store in number of entities.
yarn.timeline-service.handler-thread-count	10	Handler thread count to serve the client RPC requests.
yarn.timeline-service.http-authentication.type	simple	Defines authentication used for the timeline server HTTP endpoint. Supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME#
yarn.timeline-service.http-authentication.simple.anonymous.allowed	true	Indicates if anonymous requests are allowed by the timeline server when using ‘simple’ authentication.
yarn.timeline-service.principal		The Kerberos principal for the timeline server.
yarn.timeline-service.keytab	/etc/krb5.keytab	The Kerberos keytab for the timeline server.
yarn.timeline-service.client.max-retries	30	Default maximum number of retires for timeline servive client.
yarn.timeline-service.client.retry-interval-ms	1000	Default retry time interval for timeline servive client.
yarn.client.application-client-protocol.poll-interval-ms	200	The interval that the yarn client library uses to poll the completion status of the asynchronous API of application client protocol.
yarn.nodemanager.container-monitor.procfs-tree.smaps-based-rss.enabled	false	RSS usage of a process computed via /proc/pid/stat is not very accurate as it includes shared pages of a process. /proc/pid/smaps provides useful information like Private_Dirty, Private_Clean, Shared_Dirty, Shared_Clean which can be used for computing more accurate RSS. When this flag is enabled, RSS is computed as Min(Shared_Dirty, Pss) + Private_Clean + Private_Dirty. It excludes read-only shared mappings in RSS computation.
hadoop.registry.rm.enabled	false	Is the registry enabled: does the RM start it up, create the user and system paths, and purge service records when containers, application attempts and applications complete
hadoop.registry.zk.root	/registry
hadoop.registry.zk.session.timeout.ms	60000	Zookeeper session timeout in milliseconds
hadoop.registry.zk.connection.timeout.ms	15000	Zookeeper session timeout in milliseconds
hadoop.registry.zk.retry.times	5	Zookeeper connection retry count before failing
hadoop.registry.zk.retry.interval.ms	1000
hadoop.registry.zk.retry.ceiling.ms	60000	Zookeeper retry limit in milliseconds, during exponential backoff: {@value} This places a limit even if the retry times and interval limit, combined with the backoff policy, result in a long retry period
hadoop.registry.zk.quorum	localhost:2181	List of hostname:port pairs defining the zookeeper quorum binding for the registry
hadoop.registry.secure	false	Key to set if the registry is secure. Turning it on changes the permissions policy from “open access” to restrictions on kerberos with the option of a user adding one or more auth key pairs down their own tree.
hadoop.registry.system.acls	sasl:yarn@, sasl:mapred@, sasl:mapred@hdfs@	A comma separated list of Zookeeper ACL identifiers with system access to the registry in a secure cluster. These are given full access to all entries. If there is an “@” at the end of a SASL entry it instructs the registry client to append the default kerberos domain.
hadoop.registry.kerberos.realm		The kerberos realm: used to set the realm of system principals which do not declare their realm, and any other accounts that need the value. If empty, the default realm of the running process is used. If neither are known and the realm is needed, then the registry service/client will fail.
hadoop.registry.jaas.context	Client	Key to define the JAAS context. Used in secure mode
yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds	-1	Defines how often NMs wake up to upload log files. The default value is -1. By default, the logs will be uploaded when the application is finished. By setting this configure, logs can be uploaded periodically when the application is running. The minimum rolling-interval-seconds can be set is 3600.
yarn.am.blacklisting.enabled	true	Enable/disable blacklisting of hosts for AM based on AM failures on those hosts.
yarn.am.blacklisting.disable-failure-threshold	0.8f	Threshold of ratio number of NodeManager hosts that are allowed to be blacklisted for AM. Beyond this ratio there is no blacklisting to avoid danger of blacklisting the entire cluster.