Account Not In Store
Your aaccount is not valid for use in the U.S.store You must switch to
the Chinese store before
purchasing 的翻译是：帐户不在存储您的 aaccount 不是有效的使用在 U.S.store 你在购买之前必须切换到中国店 中文翻译英文意思，翻译英语
请在下面的文本框内输入文字，然后点击开始翻译按钮进行翻译，如果您看不到结果，请重新翻译！
Account Not In Store
Your aaccount is not valid for use in the U.S.store You must switch to
the Chinese store before
purchasing
选择语言：从
罗马尼亚语
罗马尼亚语
正在翻译，请等待...
考虑到存储库中您不因犯不适用于在美国使用存储您必须切换到之前的中国商店购买
帐户不在存储您的 aaccount 不是有效的使用在 U.S.store 你在购买之前必须切换到中国店
正在翻译，请等待...
帐户不在商店您的aaccount是无效的用于您必须以前交换对中国商店的U.S.store 购买
相关内容&aAccident category 正在翻译，请等待... & aneural predictors of bias were selective to the reward phase 偏心的神经系统的预报因子是有选择性的到奖励阶段 & acompleteness. 完整性。 & aThe results criteria covers “what an organisation achieves” namely: customer results, people results, society results and key performance results. 结果标准盖子“组织达到”即： 顾客结果、人结果，社会结果和主要绩效发生。 & aA secret history of clay :from Gauguin to Gormley. 黏土的秘密历史：从Gauguin到Gormley。 & aThe conditional statistics have been widely employed in the analysis of both non-reacting and reacting turbulent flows. Fig. 8a suggests that there is considerable difference between laminar and turbulent flames. 有条件统计在对非起反应和起反应湍流的分析广泛被使用了。 。 8a建议有层流和动荡火焰之间的可观的区别。 & anow,how many of you want to be policewomen? 现在，多少您想要是女警？ & aWhen we got to the top it was raining really hard 当我们有上面真正地艰苦下雨 & aHand link 手链接 & aA microprocessor can take all the information
surrounding it, have it be unbiased, have it react faster than the human can. 微处理器比人能可能采取围拢它的所有信息，安排它是公正的，安排它快速地起反应。 & ayou can't imagine how excited they were when they received these nice Christmas presents 您不可能想象多么激动他们是他们接受了这些好的圣诞节礼物 & ai've got a really big cock bud and i shoot a lot too, want to see? 我有真正地大公鸡芽，并且我很多射击，想要看？ & aWish I
service for the company 愿望I服务为公司 & alet's break for ten minutes and have dinner 正在翻译，请等待... & aUniversity Degree in Electrical Engineering (EE) or related disciplines 大学学位在电机工程 (EE) 或相关的学科 & aParticipants were 18 to 38 years old(M
=19.8, SD
=2.1), more often in a relationship(54.8%), and predominantly female (80.3%) and white (92.6%). 参加者经常是18到(38年M
=19.8， SD)
=2.1，在关系(54.8%)和主要地女性 (80.3%) 和白色 (92.6%)。 & a6300 正在翻译，请等待... & aSend me a picture please and share with me the joy of being a mom. 请送我一张图片并且与我分享喜悦是妈妈。 & akind of plan 种类计划 & aThe original price 原价 & aRemember° Remember° & aBack hard 后面坚硬 & adiminishing 减少 & asoftening lotionlotion lotion ultradouceur 正在翻译，请等待... & ahis daughter will inherit
the money 他的女儿将继承金钱 & aReplace a new back cover 替换新的封底 & ain the event of his death his daughter will inherit
the money 在他的死亡情形下他的女儿将继承金钱 & aAlumni 校友 & aAccount Not In Store
Your aaccount is not valid for use in the U.S.store You must switch to
the Chinese store before
purchasing 帐户不在商店您的aaccount是无效的用于您必须以前交换对中国商店的U.S.store 购买 &Hadoop 0.23.1 Release Notes
Hadoop 0.23.1 Release Notes
These release notes include new developer and user-facing incompatibilities, features, and major improvements.
Major improvement reported by xiexianshan and fixed by xiexianshan (fs)
Modify the option of FsShell getmerge from [addnl] to [-nl] for more comprehensive
The 'fs -getmerge' tool now uses a -nl flag to determine if adding a newline at end of each file is required, in favor of the 'addnl' boolean flag that was used earlier.
Major bug reported by bmahe and fixed by bmahe
Hadoop scripts unconditionally source &$bin&/../libexec/hadoop-config.sh.
Here is a patch to enable this behavior
Blocker bug reported by tgraves and fixed by sseth
test failures: TestViewFileSystemWithAuthorityLocalFileSystem and TestViewFileSystemLocalFileSystem
Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem*
Major bug reported by mahadev and fixed by mahadev
Add config for History Server protocol in hadoop-policy for service level authorization.
Adding config for MapReduce History Server protocol in hadoop-policy.xml for service level authorization.
Minor bug reported by karims and fixed by sho.shimauchi
dfs.blocksize accepts only absolute value
The default blocksize property 'dfs.blocksize' now accepts unit symbols to be used instead of byte length. Values such as &10k&, &128m&, &1g& are now OK to provide instead of just no. of bytes as was before.
Major sub-task reported by tlipcon and fixed by tlipcon (hdfs client, performance)
Simplify BlockReader to not inherit from FSInputChecker
BlockReader has been reimplemented to use direct byte buffers. If you use a custom socket factory, it must generate sockets that have associated Channels.
Major sub-task reported by tlipcon and fixed by tlipcon (hdfs client)
Switch default checksum to CRC32C
The default checksum algorithm used on HDFS is now CRC32C. Data from previous versions of Hadoop can still be read backwards-compatibly.
Major improvement reported by sanjay.radia and fixed by jnp
Shortcut a local client reads to a Datanodes files directly
1. New configurations
a. dfs.block.local-path-access.user is the key in datanode configuration to specify the user allowed to do short circuit read.
b. dfs.client.read.shortcircuit is the key to enable short circuit read at the client side configuration.
c. dfs.client.read.shortcircuit.skip.checksum is the key to bypass checksum check at the client side.
2. By default none of the above are enabled and short circuit read will not kick in.
3. If security is on, the feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general.
Major new feature reported by szetszwo and fixed by szetszwo
[umbrella] WebHDFS: a complete FileSystem implementation for accessing HDFS over HTTP
Provide WebHDFS as a complete FileSystem implementation for accessing HDFS over HTTP.
Previous hftp feature was a read-only FileSystem and does not provide &write& accesses.
Major new feature reported by hong.tang and fixed by amar_kamat (tools/rumen)
[Rumen] Need a standalone JobHistory log anonymizer
Added an anonymizer tool to Rumen. Anonymizer takes a Rumen trace file and/or topology as input. It supports persistence and plugins to override the default behavior.
Major task reported by vinaythota and fixed by vinaythota
Gridmix v3 cpu emulation system tests.
Adds system tests for the CPU emulation feature in Gridmix3.
Major new feature reported by mithun and fixed by mithun (distcp, mrv2)
DistCp Rewrite
DistCpV2 added to hadoop-tools.
Major bug reported by amar_kamat and fixed by amar_kamat (contrib/gridmix)
[Gridmix] TestGridmixSummary fails with NPE when run in DEBUG mode.
Fixed bugs in ExecutionSummarizer and ResourceUsageMatcher.
Blocker improvement reported by acmurthy and fixed by tgraves (mrv2, nodemanager, resourcemanager)
Support web-services for RM & NM
Support for web-services in YARN and MR components.
Major bug reported by amar_kamat and fixed by ravidotg (contrib/gridmix)
[Gridmix] TestUserResolve fails in trunk
Fixes bug in TestUserResolve.
Major sub-task reported by vinodkv and fixed by hitesh (mrv2, security)
NodeManager should fail fast with wrong configuration or permissions for LinuxContainerExecutor
Changed NodeManager to fail fast when LinuxContainerExecutor has wrong configuration or permissions.
Minor sub-task reported by hitesh and fixed by hitesh (mrv2)
org.apache.hadoop.mapreduce.TestNoJobSetupCleanup failing on trunk
Reneabled and fixed bugs in the failing test TestNoJobSetupCleanup.
Minor sub-task reported by hitesh and fixed by devaraj.k (mrv2, test)
ant test TestAuditLogger fails on trunk
Reenabled and fixed bugs in the failing ant test TestAuditLogger.
Minor sub-task reported by hitesh and fixed by hitesh (mrv2, test)
ant test TestDelegationToken failing on trunk
Reenabled and fixed bugs in the failing test TestDelegationToken.
Minor sub-task reported by hitesh and fixed by devaraj.k (mrv2, test)
ant test TestSubmitJob failing on trunk
Fixed a bug in TestSubmitJob.
Major bug reported by vinodkv and fixed by vinodkv (applicationmaster, mrv2)
MR AM should not read the username from configuration
Removed the unnecessary job user-name configuration in mapred-site.xml.
Major task reported by sseth and fixed by sseth (mrv2)
Move Log Related components from yarn-server-nodemanager to yarn-common
Moved log related components into yarn-common so that HistoryServer and clients can use them without depending on the yarn-server-nodemanager module.
Minor improvement reported by sseth and fixed by jeagles (mrv2)
Add AMInfo table to the AM job page
Added AMInfo table to the MR AM job pages to list all the job-attempts when AM restarts and recovers.
Major bug reported by revans2 and fixed by revans2 (mrv2)
Make MR AM not send a stopContainer w/o corresponding start container
Modified MR AM to not send a stop-container request for a container that isn't launched at all.
Major improvement reported by tgraves and fixed by tgraves (mrv2)
Improvements to CapacityScheduler doc
document changes only.
Blocker bug reported by vinodkv and fixed by vinodkv (applicationmaster, mrv2)
MR AM for sort-job going out of memory
Fixed bugs in ContainerLauncher of MR AppMaster due to which per-container connections to NodeManager were lingering long enough to hit the ulimits on number of processes.
Blocker bug reported by ramgopalnaali and fixed by sseth (mrv2)
Job is getting hanged indefinitely,if the child processes are killed on the NM.
KILL_CONTAINER eventtype is continuosly sent to the containers that are not existing
Fixed MR AM to stop considering node blacklisting after the number of nodes blacklisted crosses a threshold.
Critical bug reported by tgraves and fixed by jeagles (jobhistoryserver, mrv2)
JobHistoryServer doesn&t show job queue
Fixed JobHistoryServer to also show the job's queue name.
Major bug reported by vinodkv and fixed by hitesh (mrv2, resourcemanager)
Race condition in ResourceManager causing TestContainerManagerSecurity to fail sometimes
Fixed a race condition in ResourceManager that was causing TestContainerManagerSecurity to fail sometimes.
Blocker bug reported by vinodkv and fixed by amar_kamat (mrv2)
No rack-name logged in JobHistory for unsuccessful tasks
Unsuccessful tasks now log hostname and rackname to job history.
Blocker bug reported by vinodkv and fixed by vinodkv (applicationmaster, mrv2)
AM scheduling hangs frequently with sort job on 350 nodes
Fixed MR AM's ContainerLauncher to handle node-command timeouts correctly.
Critical improvement reported by kam_iitkgp and fixed by kamesh (mrv2)
Provide information about lost nodes in the UI.
Added information about lost/rebooted/decommissioned nodes on the webapps.
Critical bug reported by rramya and fixed by hitesh (build, mrv2)
compile-mapred-test fails
Fixed ant test compilation.
Major task reported by vinaythota and fixed by vinaythota
Memory Emulation system tests.
Added system tests to test the memory emulation feature in Gridmix.
Major bug reported by sseth and fixed by sseth (mrv2, nodemanager)
LocalResourceTracker should not tracking deleted cache entries
Fixed LocalResourceTracker in NodeManager to remove deleted cache entries correctly.
Critical bug reported by vinodkv and fixed by raviprak (applicationmaster, mrv2)
Network ACLs can prevent AMs to ping the Job-end notification URL
Enhanced MR AM to use a proxy to ping the job-end notification URL.
Critical bug reported by revans2 and fixed by revans2 (mrv2)
A tracking URL of N/A before the app master is launched breaks oozie
Fixed AM's tracking URL to always go through the proxy, even before the job started, so that it works properly with oozie throughout the job execution.
Blocker sub-task reported by johnvijoe and fixed by johnvijoe
Cluster.getDelegationToken() throws NPE if client.getDelegationToken() returns null.
Fixed Cluster's getDelegationToken's API to return null when there isn't a supported token.
Blocker bug reported by sseth and fixed by sseth (mrv2, nodemanager)
Log Aggregation broken in Secure Mode
Fixed log aggregation to work correctly in secure mode. Contributed by Siddharth Seth.
Blocker sub-task reported by sseth and fixed by sseth (mrv2, nodemanager)
ContainerLocalizer should request new resources after completing the current one
Modified ContainerLocalizer to send a heartbeat to NM immediately after downloading a resource instead of always waiting for a second.
Critical bug reported by patwhitey2007 and fixed by eepayne (job submission, mrv2)
Speculative Execution: speculative map tasks launched even if -Dmapreduce.map.speculative=false
Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces.
Minor bug reported by hitesh and fixed by hitesh (mrv2)
Wrong jar getting used in TestMR*Jobs* for MiniMRYarnCluster
Fixed pom files to refer to the correct MR app-jar needed by the integration tests.
Major bug reported by amar_kamat and fixed by amar_kamat
&ant docs& is broken
Fixes 'ant docs' by removing stale references to capacity-scheduler docs.
Blocker bug reported by tgraves and fixed by jeagles (mrv2)
job access controls not working app master and job history UI&s
Fixed job-access-controls to work with MR AM and JobHistoryServer web-apps.
Blocker sub-task reported by hitesh and fixed by hitesh (mrv2)
uber-jobs tried to write outputs into wrong dir
Fixed MR AM in uber mode to write map intermediate outputs in the correct directory to work properly in secure mode.
Blocker bug reported by amar_kamat and fixed by raviprak (mrv2, test)
Job submission failing in JUnit tests
Fixed failing JUnit tests in Gridmix.
Major improvement reported by amar_kamat and fixed by amar_kamat (contrib/gridmix)
[Gridmix] Improve STRESS mode locking
Modified Gridmix STRESS mode locking structure. The submitted thread and the polling thread now run simultaneously without blocking each other.
Major bug reported by raviprak and fixed by raviprak (mr-am, mrv2)
JobEndNotifier is getting interrupted before completing all its retries.
Fixed JobEndNotifier to not get interrupted before completing all its retries.
Critical bug reported by tgraves and fixed by jlowe (mrv2)
jobhistory web ui task counters no longer links to singletakecounter page
Fixed JobHistory web-UI to display links to single task's counters' page.
Blocker bug reported by sseth and fixed by sharadag (mr-am, mrv2)
RMContainerAllocator counts failed maps towards Reduce ramp up
Fixed MapReduce AM to count failed maps also towards Reduce ramp up.
Blocker sub-task reported by sseth and fixed by vinodkv (mr-am, mrv2)
Counters occupy a good part of AM heap
Removed a multitude of cloned/duplicate counters in the AM thereby reducing the AM heap size and preventing full GCs.
Blocker sub-task reported by sseth and fixed by sseth (mr-am, mrv2)
Batch jobHistory disk flushes
Batching JobHistory flushing to DFS so that we don't flush for every event slowing down AM.
Blocker sub-task reported by ravidotg and fixed by ravidotg (mrv2, nodemanager)
Deadlock in LocalDirsHandlerService and ShuffleHandler
Fixed a deadlock in NodeManager LocalDirectories's handling service.
Major bug reported by sseth and fixed by sseth (mr-am, mrv2)
The task timeout check interval should be configurable independent of mapreduce.task.timeout
Fixed TaskHeartBeatHandler to use a new configuration for the thread loop interval separate from task-timeout configuration property.
Blocker bug reported by karams and fixed by acmurthy (mrv2, resourcemanager, scheduler)
Sometimes NODE_UPDATE to the scheduler throws an NPE causing the scheduling to stop
Fixed an NPE occuring during scheduling in the ResourceManager.
Critical bug reported by karams and fixed by kamesh (mrv2, nodemanager)
When 0 is provided as port number in yarn.nodemanager.webapp.address, NMs webserver component picks up random port, NM keeps on Reporting 0 port to RM
Modified NM to report correct http address when an ephemeral web port is configured.
Blocker bug reported by tgraves and fixed by tgraves (mrv2)
write api documentation for web service apis for RM, NM, mapreduce app master, and job history server
new files added: A
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WebServicesIntro.apt.vm
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/NodeManagerRest.apt.vm
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/MapredAppMasterRest.apt.vm
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
The hadoop-project/src/site/site.xml is split into separate patch.
Blocker bug reported by mahadev and fixed by sseth (mrv2)
TestStagingCleanup and TestJobEndNotifier are failing on trunk.
Fixed failures in TestStagingCleanup and TestJobEndNotifier tests.
Critical sub-task reported by vinodkv and fixed by vinodkv (mr-am, mrv2, performance)
Optimize Job&s progress calculations in MR AM
Optimized Job's progress calculations in MR AM.
Blocker bug reported by vinodkv and fixed by vinodkv (mr-am, mrv2)
Lots of AMs hanging around in PIG testing
Modified CompositeService to avoid duplicate stop operations thereby solving race conditions in MR AM shutdown.
Major improvement reported by ravidotg and fixed by ravidotg (tools/rumen)
Provide a way to access other info of history file from Rumentool
Rumen now provides {{Parsed*}} objects. These objects provide extra information that are not provided by {{Logged*}} objects.
Major sub-task reported by sseth and fixed by sseth (mrv2, performance)
TaskHeartbeatHandler holds a global lock for all task-updates
Fixed TaskHeartbeatHandler to not hold a global lock for all task-updates.
Critical task reported by amolkekre and fixed by mahadev (mrv2)
NullPointerException running teragen
Committed to trunk and branch-0.23. Thanks Mahadev.
Blocker bug reported by sseth and fixed by sseth (mrv2)
TokenCache likely broken for FileSystems which don&t issue delegation tokens
Fixed TokenCache to work with absent FileSystem canonical service-names.
Blocker sub-task reported by acmurthy and fixed by acmurthy (mrv2, scheduler)
CapacityScheduler should be more conservative assigning off-switch requests
Making CapacityScheduler more conservative so as to assign only one off-switch container in a single scheduling iteration.
Blocker bug reported by karams and fixed by sseth (applicationmaster, mrv2, resourcemanager)
Sort job on 350 scale is consistently failing with latest MRV2 code
Fixed a race condition in MR AM which is failing the sort benchmark consistently.
Major bug reported by vinodkv and fixed by hitesh (mrv2)
Default RPC handlers are very low for YARN servers
Increased RPC handlers for all YARN servers to reasonable values for working at scale.
Critical bug reported by eepayne and fixed by eepayne (mrv2, resourcemanager)
ResourceManager should provide node lists in JMX output
New JMX Bean in ResourceManager to provide list of live node managers:
Hadoop:service=ResourceManager,name=RMNMInfo LiveNodeManagers
Major bug reported by sseth and fixed by sseth (mrv1, mrv2)
last split generated by FileInputFormat.getSplits may not have the best locality
Improved FileInputFormat to return better locality for the last split.
Blocker sub-task reported by sseth and fixed by revans2 (mrv2)
AppMaster recovery for Medium to large jobs take long time
Fixed MR AM recovery so that only single selected task output is recovered and thus reduce the unnecessarily bloated recovery time.
Blocker bug reported by sseth and fixed by acmurthy (mrv2, resourcemanager)
Incorrect headroom reported to jobs
Fixed the way head-room is allocated to applications by CapacityScheduler so that it deducts current-usage per user and not per-application.
Blocker bug reported by vinodkv and fixed by vinodkv (mrv2, task)
Reduce hangs in a corner case
Fixed EventFetcher and Fetcher threads to shut-down properly so that reducers don't hang in corner cases.
Blocker bug reported by jeagles and fixed by jeagles (mrv2)
java.io.File.createTempFile fails in map/reduce tasks
Fixing YARN+MR to allow MR jobs to be able to use java.io.File.createTempFile to create temporary files as part of their tasks.
Major bug reported by vinodkv and fixed by vinodkv (client, mrv2)
Command line listJobs should not visit each AM
Changed bin/mapred job -list to not print job-specific information not available at RM.
Very minor incompatibility in cmd-line output, inevitable due to MRv2 architecture.
Blocker bug reported by acmurthy and fixed by acmurthy (mrv2, resourcemanager, scheduler)
CS should only use &activeUsers with pending requests& for computing user-limits
Modified CapacityScheduler to use only users with pending requests for computing user-limits.
Blocker bug reported by acmurthy and fixed by acmurthy (mrv2)
Headroom should be capped by queue max-cap
Modified application limits to include queue max-capacities besides the usual user limits.
Major bug reported by vinodkv and fixed by vinodkv (mrv2, webapps)
RM webapp should have pages filtered based on App-state
Modified RM UI to filter applications based on state of the applications.
Major bug reported by rramya and fixed by vinodkv (mrv2)
Blacklisted NMs should not appear in Active nodes list
Changed active nodes list to not contain unhealthy nodes on the webUI and metrics.
Major bug reported by mahadev and fixed by mahadev (mrv2)
yarn-default.xml should be moved to hadoop-yarn-common.
Major bug reported by rramya and fixed by acmurthy (mrv2)
maxActiveApplications(|PerUser) per queue is too low for small clusters
Fixed CapacityScheduler so that maxActiveApplication and maxActiveApplicationsPerUser per queue are not too low for small clusters.
Major bug reported by davet and fixed by davet (jobhistoryserver, mrv2, resourcemanager)
yarn webapp interface vulnerable to cross scripting attacks
fix cross scripting attacks vulnerability through webapp interface.
Blocker bug reported by sseth and fixed by revans2 (mrv2)
NPE in FileOutputCommitter when running a 0 reduce job
Fixed an NPE in FileOutputCommitter for jobs with maps but no reduces.
Critical sub-task reported by sseth and fixed by sseth (mrv2)
Data Locality suffers if the AM asks for containers using IPs instead of hostnames
Fixed MR AM to always use hostnames and never IPs when requesting containers so that scheduler can give off data local containers correctly.
Other Jiras (describe bug fixes and minor changes)
Minor improvement reported by abagri and fixed by sho.shimauchi
conf.getBoolean must be case insensitive
Currently, if xx is set to &TRUE&, conf.getBoolean(&xx&, false) would return false. conf.getBoolean should do an equalsIgnoreCase() instead of equals()I am marking the change as incompatible because it does change semantics as pointed by Steve in HADOOP-4416
Minor bug reported by zshao and fixed by umamaheswararao (fs)
Path.normalize should use StringUtils.replace in favor of String.replace
in our environment, we are seeing that the JobClient is going out of memory because Path.normalizePath(String) is called several tens of thousands of times, and each time it calls &String.replace& twice.java.lang.String.replace compiles a regex to do the job which is very costly.We should use org.apache.commons.lang.StringUtils.replace which is much faster and consumes almost no extra memory.
Minor improvement reported by stevel@apache.org and fixed by jmhsieh (util)
RunJar should provide more diags when it can&t create a temp file
When you see a stack trace about permissions, it is better if the trace included the file/directory at fault:{code}Exception in thread &main& java.io.IOException: Permission denied at java.io.UnixFileSystem.createFileExclusively(Native Method) at java.io.File.checkAndCreate(File.java:1704) at java.io.File.createTempFile(File.java:1792) at org.apache.hadoop.util.RunJar.main(RunJar.java:147){code}As it is, you need to go into the code, discover that it&s {{${hadoop.tmp.dir}/hadoop-unja...
Minor improvement reported by nspiegelberg and fixed by nspiegelberg (fs, io)
Support non-recursive create() in FileSystem & SequenceFile.Writer
The proposed solution for HBASE-2312 requires the sequence file to handle a non-recursive create.
This is already supported by HDFS, but needs to have an equivalent FileSystem & SequenceFile.Writer API.
Minor improvement reported by nspiegelberg and fixed by nspiegelberg (fs)
LocalFileSystem Needs createNonRecursive API
While running sanity check tests for HBASE-2312, I noticed that HDFS-617 did not include createNonRecursive() support for the LocalFileSystem.
This is a problem for HBase, which allows the user to run over the LocalFS instead of HDFS for local cluster testing.
I think this only affects 0.20-append, but may affect the trunk based upon how exactly FileContext handles non-recursive creates.
Major improvement reported by eli and fixed by umamaheswararao
Log an error if the topology script doesn&t handle multiple args
ScriptBasedMapping#resolve currently warns and returns null if it passes n arguments to the topology script and gets back a different number of resolutions. This indicates a bug in the topology script (or it&s input) and therefore should be an error.{code}// invalid number of entries returned by the scriptLOG.warn(&Script & + scriptName + & returned &
+ Integer.toString(m.size()) + & values when &
+ Integer.toString(names.size()) + & were expected.&);{code}There&s on...
Minor improvement reported by stevel@apache.org and fixed by enis (util)
move up to Jackson 1.8.8
I see that hadoop-core still depends on Jackson 1.0.1 -but that project is now up to 1.8.2 in releases. Upgrading will make it easier for other Jackson-using apps that are more up to date to keep their classpath consistent.The patch would be updating the ivy file to pull
Trivial improvement reported by eli and fixed by qwertymaniac (metrics)
hadoop-metrics.properties missing some Ganglia31 options
The &jvm&, &rpc&, and &ugi& sections of hadoop-metrics.properties should have Ganglia31 options like &dfs& and &mapred&
Trivial improvement reported by xiexianshan and fixed by xiexianshan (fs)
Improvement for FSshell -stat
Add two optional formats for FSshell -stat, one is %G for group name of owner and the other is %U for user name.
Major sub-task reported by tucu00 and fixed by tucu00 (build)
Mavenize streaming and MR examples
MR1 code is still available in MR2 for testing contribs.While this is a temporary until contribs tests are ported to MR2.As a follow up the contrib projects themselves should be mavenized.
Major improvement reported by mrbsd and fixed by decster
Add support for LZ4 compression
According to several benchmark sites, LZ4 seems to overtake other fast compression algorithms, especially in the decompression speed area. The interface is also trivial to integrate (http://code.google.com/p/lz4/source/browse/trunk/lz4.h) and there is no license issue.
Trivial improvement reported by qwertymaniac and fixed by qwertymaniac (fs)
Remove duplicate call of Path#normalizePath during initialization.
Found during code reading on HADOOP-6490, there seems to be an unnecessary call of {{normalizePath(...)}} being made in the constructor {{Path(Path, Path)}}. Since {{initialize(...)}} normalizes its received path string already, its unnecessary to do it to the path parameter in the constructor&s call of the same.
Major improvement reported by tucu00 and fixed by tucu00 (fs)
Make GlobFilter class public
Currently the GlobFilter class is package private.As a generic filter it is quite useful (and I&ve found myself doing cut&paste of it a few times)
Major improvement reported by tlipcon and fixed by tlipcon (io, performance, util)
Improve performance of raw comparisons
Guava has a nice implementation of lexicographical byte-array comparison that uses sun.misc.Unsafe to compare unsigned byte arrays long-at-a-time. Their benchmarks show it as being 2x more CPU-efficient than the equivalent pure-Java implementation. We can easily integrate this into WritableComparator.compareBytes to improve CPU performance in the shuffle.
Major improvement reported by stevel@apache.org and fixed by stevel@apache.org (util)
Implement a base class for DNSToSwitchMapping implementations that can offer extra topology information
HDFS-2492 has identified a need for DNSToSwitchMapping implementations to provide a bit more topology information (e.g. whether or not there are multiple switches). This could be done by writing an extended interface, querying its methods if present and coming up with a default action if there is no extended interface. Alternatively, we have a base class that all the standard mappings implement, with a boolean isMultiRack() all the standard subclasses would extend this, as could any...
Major bug reported by bmahe and fixed by bmahe (build)
Make source tarball use conventional name.
When building binary and source tarballs, I get the following artifacts:Binary tarball: hadoop-0.23.0-SNAPSHOT.tar.gz Source tarball: hadoop-dist-0.23.0-SNAPSHOT-src.tar.gzNotice the &-dist& right between &hadoop& and the version in the source tarball name.
Major bug reported by bmahe and fixed by bmahe (build)
HADOOP_PREFIX cannot be overriden
hadoop-config.sh forces HADOOP_prefix to a specific value:export HADOOP_PREFIX=`dirname &$this&`/..It would be nice to make this overridable.
Major improvement reported by arpitgupta and fixed by arpitgupta (conf)
enable hadoop config generator to set dfs.block.local-path-access.user to enable short circuit read
we have a new config that allows to select which user can have access for short circuit read. We should make that configurable through the config generator scripts.
Major new feature reported by daryn and fixed by daryn (fs, security)
Port token service changes from 205
Need to merge the 205 token bug fixes and the feature to enable hostname-based tokens.
Blocker bug reported by johnvijoe and fixed by johnvijoe
move hadoop archive to core from tools
&The HadoopArchieves classes are included in the $HADOOP_HOME/hadoop_tools.jar, but this file is not found in `hadoop classpath`.A Pig script using HCatalog&s dynamic partitioning with HAR enabled will therefore fail if a jar with HAR is not included in the pig call&s &-cp& and &-Dpig.additional.jars& arguments.&I am not aware of any reason to not include hadoop-tools.jar in &hadoop classpath&. Will attach a patch soon.
Major bug reported by jeagles and fixed by jeagles (security, test)
TestUserGroupInformation#testGetServerSideGroups test fails in chroot
It is common when running in chroot to have root&s group vector preserved when running as your self.For example# Enter chroot$ sudo chroot /myroot# still root$ whoamiroot# switch to user preserving root&s group vector$ sudo -u user -P -s# root&s groups$ groups roota b c# user&s real groups$ groups userd e f# user&s effective groups$ groupsa b c d e f-------------------------------
Major bug reported by stevel@apache.org and fixed by eli (conf)
no NullAppender in the log4j config
running sbin/start-dfs.sh gives me a telling off about no null appender -should one be in the log4j config file.Full trace (failure expected, but full output not as expected){code}./start-dfs.sh log4j:ERROR Could not find value for key log4j.appender.NullAppenderlog4j:ERROR Could not instantiate appender named &NullAppender&.Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.Starting namenodes on []cat: /Users/slo/J...
Major bug reported by johnvijoe and fixed by johnvijoe
compilation failing because workDir not initialized in RunJar.java
Compilation is failing on 0.23 and trunk because workDir is not initialized in RunJar.java
Blocker bug reported by daryn and fixed by daryn (security)
multiple javax security configurations cause conflicts
Both UGI and the SPNEGO KerberosAuthenticator set the global javax security configuration.
SPNEGO stomps on UGI&s security config which leads to kerberos/SASL authentication errors.
Critical bug reported by daryn and fixed by daryn (security)
UGI getCurrentUser is not synchronized
Sporadic {{ConcurrentModificationExceptions}} are originating from {{UGI.getCurrentUser}} when it needs to create a new instance.
The problem was specifically observed in a JT under heavy load when a post-job cleanup is accessing the UGI while a new job is being processed.
Trivial improvement reported by tlipcon and fixed by tlipcon
Drop some info logging to DEBUG level in IPC, metrics, and HTTP
Our info level logs have gotten noisier and noisier over time, which is annoying both for users and when looking at unit tests. I&d like to drop a few of the less useful INFO level messages down to DEBUG.
Major bug reported by eli and fixed by eli (fs)
TestViewFsHdfs.testgetFileLinkStatus is failing an assert
Probably introduced by HADOOP-7783. I&ll fix it.{noformat}java.lang.AssertionError at org.apache.hadoop.fs.FileContext.qualifySymlinkTarget(FileContext.java:1111) at org.apache.hadoop.fs.FileContext.access$000(FileContext.java:170) at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1142) at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1137) at org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2327) at org.apache.hadoop.fs.FileContext.getF...
Major bug reported by abayer and fixed by abayer (build)
Building mvn site with Maven & 3.0.2 causes OOM errors
If you try to run mvn site with Maven 3.0.0 (and possibly 3.0.1 - haven&t actually tested that), you get hit with unavoidable OOM errors. Switching to Maven 3.0.2 or later fixes this. The enforcer should require 3.0.2 for builds.
Major bug reported by jmhsieh and fixed by jmhsieh
fix SequenceFile#createWriter with boolean createParent arg to respect createParent.
After HBASE-6840, one set of calls to createNonRecursive(...) seems fishy - the new boolean createParent variable from the signature isn&t used at all.
public static Writer+
createWriter(FileSystem fs, Configuration conf, Path name,+
Class keyClass, Class valClass, int bufferSize,+
short replication, long blockSize, boolean createParent,+
CompressionType compressionType, CompressionCodec codec,+
Metadata meta...
Major bug reported by tucu00 and fixed by tucu00 (build)
native libs should be under lib/native/ dir
Currently common and hdfs SO files end up under lib/ dir with all JARs, they should end up under lib/native.In addition, the hadoop-config.sh script needs some cleanup when comes to native lib handling:* it is using lib/native/${JAVA_PLATFORM} for the java.library.path, when it should use lib/native.* it is looking for build/lib/native, this is from the old ant build, not applicable anymore.* it is looking for the libhdfs.a and adding to the java.librar.path, this is not correct.
Major task reported by szetszwo and fixed by szetszwo (documentation)
Federation: update Balancer documentation
Update Balancer documentation for the new balancing policy and CLI.
Minor bug reported by stevel@apache.org and fixed by stevel@apache.org (util)
Regression HADOOP-7777 switch changes break HDFS tests when the isSingleSwitch() predicate is used
This doesn&t show up until you apply the HDFS-2492 patch, but the attempt to make the {{StaticMapping}} topology clever by deciding if it is single rack or multi rack based on its rack-&node mapping breaks the HDFS {{TestBlocksWithNotEnoughRacks}} test. Why? Because the racks go in after the switch topology is cached by the {{BlockManager}}, which assumes the system is always single-switch.Fix: default to assuming multi- remove the intelligence, add a setter for anyone who really wan...
Critical bug reported by tucu00 and fixed by tucu00 (security)
KerberosAuthenticatorHandler is not setting KerberosName name rules from configuration
While the KerberosAuthenticatorHandler defines the name rules property, it does not set it in KerberosName.
Trivial improvement reported by knoguchi and fixed by knoguchi (scripts)
Redirect hadoop script&s deprecation message to stderr
$ hadoop dfs -lsDEPRECATED: Use of this script to execute hdfs command is deprecated.Instead use the hdfs command for it....If we&re still letting the command run, I think we should redirect the deprecation message to stderr in case users have a script taking the output from stdout.
Minor bug reported by sureshms and fixed by sureshms (security)
Fix javadoc warnings in AuthenticationToken.java
Fix the following javadoc warning:[WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationToken.java:33: warning - Tag @link: reference not found: HttpServletRequest[WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationToken.java...
Major bug reported by szetszwo and fixed by tucu00
skipping name rules setting (if already set) should be done on UGI initialization only
Both TestDelegationToken and TestOfflineEditsViewer are currently failing.
Blocker bug reported by tucu00 and fixed by tucu00 (build)
hadoop-tools JARs are not part of the distro
After mavenizing streaming, the hadoop-streaming JAR is not part of the final tar.
Minor improvement reported by sho.shimauchi and fixed by sho.shimauchi (conf)
add configuration methods to handle human readable size values
It&s better to have a new configuration methods which handle human readable size values.For example, see HDFS-1314.
Major bug reported by revans2 and fixed by revans2 (build)
test-patch should run eclipse:eclipse to verify that it does not break again
Recently the eclipse:eclipse build was broken.
If we are going to document this on the wiki and have many developers use it we should verify that it always works.
Major bug reported by szetszwo and fixed by szetszwo (build)
duplicate declaration of hadoop-hdfs test-jar
[WARNING] Some problems were encountered while building the effective model for org.apache.hadoop:hadoop-common-project:pom:0.24.0-SNAPSHOT[WARNING] &dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)& must be unique: org.apache.hadoop:hadoop-hdfs:test-jar -& duplicate declaration of version ${project.version} @ org.apache.hadoop:hadoop-project:0.24.0-SNAPSHOT, /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-project/pom.xml, line 140, ...
Major bug reported by tucu00 and fixed by tucu00 (build)
compilation of protobuf files fails in windows/cygwin
HADOOP-7899 & HDFS-2511 introduced compilation of proto files as part of the build.Such compilation is failing in windows/cygwin
Trivial improvement reported by qwertymaniac and fixed by qwertymaniac (documentation)
[Doc] Remove hadoop.logfile.* properties.
The following only resides in core-default.xml and doesn&t look like its used anywhere at all. At least a grep of the prop name and parts of it does not give me back anything at all.These settings are now configurable via generic Log4J opts, via the shipped log4j.properties file in the distributions.{code}137 &!--- logging properties --&138 139 &property&140
&name&hadoop.logfile.size&/name&141
&value&&/value&142
&description&The max size of each log file&/description&...
Critical bug reported by sseth and fixed by sseth
Viewfs changes for MAPREDUCE-3529
ViewFs.getDelegationTokens returns a list of tokens for the associated namenodes. Credentials serializes these tokens using the service name for the actual namenodes. Effectively, tokens are not cached for viewfs (some more details in MR 3529). Affects any job which uses the TokenCache in tasks along with viewfs (some Pig jobs).Talk to Jitendra about this, some options1. Change Credentials.getAllTokens to return the key, instead of just a token list (associate the viewfs canonical name wit...
Critical improvement reported by tucu00 and fixed by tucu00 (build)
Normalize dependencies versions across all modules
Move all dependencies versions to the dependencyManagement section in the hadoop-project POMMove all plugin versions to the dependencyManagement section in the hadoop-project POM
Major bug reported by eli and fixed by tucu00 (build)
There&s a Hoop README in the root dir of the tarball
The Hoop README.txt is now in the root dir of the tarball.{noformat}hadoop-trunk1 $ tar xvzf hadoop-dist/target/hadoop-0.24.0-SNAPSHOT.tar.gz
-C /tmp/..hadoop-trunk1 $ head -n3 /tmp/hadoop-0.24.0-SNAPSHOT/README.txt -----------------------------------------------------------------------------HttpFS - Hadoop HDFS over HTTP{noformat}
Major improvement reported by rvs and fixed by rvs (build, conf, documentation, scripts)
Improve Hadoop subcomponent integration in Hadoop 0.23
h1. IntroductionFor the rest of this proposal it is assumed that the current setof Hadoop subcomponents is: * hadoop-common * hadoop-hdfs * hadoop-yarn * hadoop-mapreduceIt must be noted that this is an open ended list, though. For example,implementations of additional frameworks on top of yarn (e.g. MPI) wouldalso be considered a subcomponent.h1. Problem statementCurrently there&s an unfortunate coupling and hard-coding present at thelevel of launcher scripts, configuration s...
Minor bug reported by cim_michajlomatijkiw and fixed by cim_michajlomatijkiw (build)
Shell scripts created by hadoop-dist/pom.xml to build tar do not properly propagate failure
The run() function, as defined in dist-layout-stitching.sh and dist-tar-stitching, created in hadoop-dist/pom.xml, does not properly propagate the error code of a failing command.
See the following:{code}
# call fails with non-zero exit code
if [ $? != 0 ]; then
echo &Failed!&
# $?=result of echo above, likely 0, thus exit with code 0
Trivial bug reported by eli and fixed by eli (ipc)
Updated maxIdleTime default in the code to match core-default.xml
HADOOP-2909 intended to set the server max idle time for a connection to twice the client value. (&The server-side max idle time should be greater than the client-side max idle time, for example, twice of the client-side max idle time.&) This way when a server times out a connection it&s due a crashed client and not an inactive client so we don&t close client connections with outstanding requests (by setting 2x the client value on the server side the client should time out the connection firs...
Blocker bug reported by kihwal and fixed by daryn (security, util)
Deadlock in class init.
After HADOOP-7808, client-side commands hang occasionally. There are cyclic dependencies in NetUtils and SecurityUtil class initialization. Upon initial look at the stack trace, two threads deadlock when they hit the either of class init the same time.
Blocker bug reported by tgraves and fixed by prashant_
hadoop &job/queue/pipes& removed - should be added back, but deprecated
The mapred subcommands (mradmin|jobtracker|tasktracker|pipes|job|queue) were removed from the /bin/hadoop command. I believe for backwards compatibility at least some of these should have stayed along with the deprecated warnings.
Major bug reported by eli and fixed by qwertymaniac (fs)
TestViewFsTrash incorrectly determines the user&s home directory
HADOOP-7284 added a test called TestViewFsTrash which contains the following code to determine the user&s home directory. It only works if the user&s directory is one level deep, and breaks if the home directory is more than one level deep (eg user hudson, who&s home dir might be /usr/lib/hudson instead of /home/hudson).{code}
// create a link for home directory so that trash path works
// set up viewfs&s home dir root to point to home dir root on target
// But home dir is diffe...
Minor bug reported by qwertymaniac and fixed by qwertymaniac
Add entry to XML defaults for new LZ4 codec
HADOOP-7657 added in a new LZ4 codec, but failed to extend the io.compression.codecs list which MR/etc. use up to load codecs.We should add an entry to the core-default XML for this new codec, just as we did with Snappy.
Major bug reported by jeagles and fixed by jeagles (io)
Improve documentation for org.apache.hadoop.io.compress.Decompressor.getRemaining
Major bug reported by tlipcon and fixed by tlipcon (security)
UserGroupInformation fails to login if thread&s context classloader can&t load HadoopLoginModule
In a few hard-to-reproduce situations, we&ve seen a problem where the UGI login call causes a failure to login exception with the following cause:Caused by: javax.security.auth.login.LoginException: unable to find LoginModule class: org.apache.hadoop.security.UserGroupInformation $HadoopLoginModuleAfter a bunch of debugging, I determined that this happens when the login occurs in a thread whose Context ClassLoader has been set to null.
Major improvement reported by devaraj and fixed by jnp (security)
Support setting the run-as user in unsecure mode
Some applications need to be able to perform actions (such as launch MR jobs) from map or reduce tasks. In earlier unsecure versions of hadoop (20.x), it was possible to do this by setting user.name in the configuration. But in 20.205 and 1.0, when running in unsecure mode, this does not work. (In secure mode, you can do this using the kerberos credentials).
Major bug reported by jnp and fixed by jnp
Upper case in hostname part of the principals doesn&t work with kerberos.
Kerberos doesn&t like upper case in the hostname part of the principals.This issue has been seen in 23 as well as 1.0.
Major bug reported by anupamseth and fixed by anupamseth (conf)
Hadoop ignores old-style config options for enabling compressed output
Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress andmapred.output.compression.codec- settings that work on 0.20 but not on 0.23mapred.output.compress=truemapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec- settings that work on 0.23mapreduce.output.fileoutputformat.compress=truemapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2CodecThis breaks bac...
Major bug reported by gchanan and fixed by gchanan (io)
SequenceFile.createWriter(...createParent...) no longer works on existing file
SequenceFile.createWriter no longer works on an existing file, because old version specified OVEWRITE by default and new version does not.
This breaks some HBase tests.Tested against trunk.Patch with test to follow.
Major bug reported by daryn and fixed by daryn (fs)
CheckFileSystem does not correctly honor setVerifyChecksum
Regardless of the verify checksum flag, {{ChecksumFileSystem#open}} will instantiate a {{ChecksumFSInputChecker}} instead of a normal stream.
Critical bug reported by jlowe and fixed by jlowe (scripts)
&hadoop archive& fails with ClassNotFoundException
Running &hadoop archive& from a command prompt results in this error:Exception in thread &main& java.lang.NoClassDefFoundError: org/apache/hadoop/tools/HadoopArchivesCaused by: java.lang.ClassNotFoundException: org.apache.hadoop.tools.HadoopArchives at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) ...
Critical bug reported by arpitgupta and fixed by arpitgupta
fetchdt command not available in bin/hadoop
fetchdt command needs to be added to bin/hadoop to allow for backwards compatibility.
Major bug reported by daryn and fixed by daryn (fs)
ChecksumFileSystem&s rename doesn&t correctly handle checksum files
Rename will move the src file and its crc *if present* to the destination.
If the src file has no crc, but the destination already exists with a crc, then src will be associated with the old file&s crc.
Subsequent access to the file will fail with checksum errors.
Major bug reported by arpitgupta and fixed by arpitgupta
SecurityUtil acquired token message should be a debug rather than info
Major bug reported by umamaheswararao and fixed by daryn (fs)
TestFSInputChecker is failing in trunk.
Trunk build number 939 failed with TestFSInputChecker.https://builds.apache.org/job/Hadoop-Hdfs-trunk/939/junit.framework.AssertionFailedError: expected:&10& but was:&0& at junit.framework.Assert.fail(Assert.java:47) at junit.framework.Assert.failNotEquals(Assert.java:283) at junit.framework.Assert.assertEquals(Assert.java:64) at junit.framework.Assert.assertEquals(Assert.java:130) at junit.framework.Assert.assertEquals(Assert.java:136) at org.apache.hadoop.hdfs.TestFSInputChecker.ch...
Critical improvement reported by tucu00 and fixed by tucu00 (build)
Create hadoop-client and hadoop-minicluster artifacts for downstream projects
Using Hadoop from projects like Pig/Hive/Sqoop/Flume/Oozie or any in-house system that interacts with Hadoop is quite challenging for the following reasons:* *Different versions of Hadoop produce different artifacts:* Before Hadoop 0.23 there was a single artifact hadoop-core, starting with Hadoop 0.23 there are several (common, hdfs, mapred*, yarn*)* *There are no &client& artifacts:* Current artifacts include all JARs needed to run the services, thus bringing into clients several JARs t...
Minor bug reported by rvs and fixed by rvs (scripts)
hadoop-daemon.sh and yarn-daemon.sh are trying to mkdir and chow log/pid dirs which can fail
Here&s what I see when using Hadoop in Bigtop:{noformat}$ sudo /sbin/service hadoop-hdfs-namenode startStarting Hadoop namenode daemon (hadoop-namenode): chown: changing ownership of `/var/log/hadoop&: Operation not permittedstarting namenode, logging to /var/log/hadoop/hadoop-hdfs-namenode-centos5.out{noformat}This is a cosmetic issue, but it would be nice to fix it.
Major bug reported by daryn and fixed by daryn (fs)
ViewFileSystem does not honor setVerifyChecksum
{{ViewFileSystem#setVerifyChecksum}} is a no-op.
It should call {{setVerifyChecksum}} on the mount points.
Major improvement reported by daryn and fixed by daryn (fs)
ChRootFileSystem should extend FilterFileSystem
{{ChRootFileSystem}} simply extends {{FileSystem}}, and attempts to delegate some methods to the underlying mount point.
It is essentially the same as {{FilterFileSystem}} but it mangles the paths to include the chroot path.
Unfortunately {{ChRootFileSystem}} is not delegating some methods that should be delegated.
Changing the inheritance will prevent a copy-n-paste of code for HADOOP-8013 and HADOOP-8014 into both {{ChRootFileSystem}} and {{FilterFileSystem}}.
Major bug reported by mattf and fixed by jeagles (build, test)
Hudson auto test for HDFS has started throwing javadoc: warning - Error fetching URL: http://java.sun.com/javase/6/docs/api/package-list
Hudson automated testing has started failing with one javadoc warning message, consisting ofjavadoc: warning - Error fetching URL: http://java.sun.com/javase/6/docs/api/package-listThis may be due to Oracle&s decommissioning of the sun.com domain.
If one tries to access it manually, it is redirected to http://download.oracle.com/javase/6/docs/api/package-listSo it looks like a build script needs to be updated.
Minor improvement reported by qwertymaniac and fixed by atm (metrics)
Visiting /jmx on the daemon web interfaces may print unnecessary error in logs
Logs that follow a {{/jmx}} servlet visit:{code}11/11/22 12:09:52 ERROR jmx.JMXJsonServlet: getting attribute UsageThreshold of java.lang:type=MemoryPool,name=Par Eden Space threw an exceptionjavax.management.RuntimeMBeanException: java.lang.UnsupportedOperationException: Usage threshold is not supported at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:856)...{code}
Minor bug reported by raviphulari and fixed by qwertymaniac
Improve dfsadmin command line help
Enhance dfsadmin command line help informing &A quota of one forces a directory to remain empty&
Major improvement reported by szetszwo and fixed by umamaheswararao (name-node)
FSEditLog should not writes long and short as UTF8 and should not use ArrayWritable for writing non-array items
In FSEditLog, - long and short are first converted to String and are further converted to UTF8- For some non-array items, it first create an ArrayWritable object to hold all the items and then writes the ArrayWritable object.These result creating many intermediate objects which affects Namenode CPU performance and Namenode restart.
Minor bug reported by rramya and fixed by qwertymaniac (test)
dfsthroughput in test.jar throws NPE
On running hadoop jar hadoop-test.jar dfsthroughput OR hadoop org.apache.hadoop.hdfs.BenchmarkThroughput, we get NullPointerException. Below is the stacktrace:{noformat}Exception in thread &main& java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:394)
at java.util.Properties.setProperty(Properties.java:143)
at java.lang.System.setProperty(System.java:731)
at org.apache.hadoop.hdfs.BenchmarkThroughput.run(BenchmarkThroughput.java:198)
Minor improvement reported by stevel@apache.org and fixed by qwertymaniac (name-node)
BlockInfo.ensureCapacity may get a speedup from System.arraycopy()
BlockInfo.ensureCapacity() uses a for() loop to copy the old array data into the expanded array.
{{System.arraycopy()}} is generally much faster for this, as it can do a bulk memory copy. There is also the typesafe Java6 {{Arrays.copyOf()}} to consider, though here it offers no tangible benefit.
Major improvement reported by tucu00 and fixed by tucu00
HttpFS - a read/write Hadoop file system proxy
We&d like to contribute Hoop to Hadoop HDFS as a replacement (an improvement) for HDFS Proxy.Hoop provides access to all Hadoop Distributed File System (HDFS) operations (read and write) over HTTP/S.The Hoop server component is a REST HTTP gateway to HDFS supporting all file system operations. It can be accessed using standard HTTP tools (i.e. curl and wget), HTTP libraries from different programing languages (i.e. Perl, Java Script) as well as using the Hoop client. The Hoop server compo...
Major improvement reported by eli and fixed by umamaheswararao (data-node, name-node)
DataNodeCluster and NNStorage always pull fresh entropy
Jira for giving DataNodeCluster and NNStorage the same treatment as HDFS-1835. They&re not truly cryptographic uses as well. We should also factor this out to a utility method, seems like the three uses are slightly different, eg one uses DFSUtil.getRandom and the other creates a new Random object.
Trivial improvement reported by qwertymaniac and fixed by qwertymaniac (data-node)
DN should log a WARN, not an INFO when it detects a corruption during block transfer
Currently, in DataNode.java, we have:{code}
LOG.info(&Can&t replicate block & + block
+ & because on-disk length & + onDiskLength
+ & is shorter than NameNode recorded length & + block.getNumBytes());{code}This log is better off as a WARN as it indicates (and also reports) a corruption.
Major improvement reported by tlipcon and fixed by eli (name-node)
Undeprecate SecondaryNameNode
I would like to consider un-deprecating the SecondaryNameNode for 0.23, and amending the documentation to indicate that it is still the most trust-worthy way to run checkpoints, and while CN/BN may have some advantages, they&re not battle hardened as of yet. The test coverage for the 2NN is far superior to the CheckpointNode or BackupNode, and people have a lot more production experience. Indicating that it is deprecated before we have expanded test coverage of the CN/BN won&t send the right ...
Minor improvement reported by umamaheswararao and fixed by qwertymaniac (data-node)
Move maxXceiverCount check to before starting the thread in dataXceiver
We can hoist the maxXceiverCount out of DataXceiverServer#run, there&s no need to check each time we accept a connection, we can accept when we create a thread.
Minor improvement reported by eli and fixed by qwertymaniac (documentation)
hdfs-default.xml should include dfs.name.dir.restore
Minor improvement reported by tlipcon and fixed by tucu00 (build)
Add dev script to generate HDFS protobufs
Would like to add a simple shell script to re-generate the protobuf code in HDFS -- just easier than remembering the right syntax.
Minor improvement reported by tlipcon and fixed by tlipcon (data-node, performance)
Remove needless synchronization on FSDataSet.getBlockFile
HDFS-1148 discusses lock contention issues in FSDataset. It provides a more comprehensive fix, converting it all to RWLocks, etc. This JIRA is for one very specific fix which gives a decent performance improvement for TestParallelRead: getBlockFile() currently holds the lock which is completely unnecessary.
Trivial improvement reported by atm and fixed by qwertymaniac (name-node)
Remove unused imports
Looks like it has 11 unused imports by my count.
Major bug reported by qwertymaniac and fixed by qwertymaniac (data-node)
For a sufficiently large value of blocks, the DN Scanner may request a random number with a negative seed value.
Running off 0.20-security, I noticed that one could get the following exception when scanners are used:{code}DataXceiver java.lang.IllegalArgumentException: n must be positive at java.util.Random.nextInt(Random.java:250) at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.getNewBlockScanTime(DataBlockScanner.java:251) at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.addBlock(DataBlockScanner.java:268) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(Da...
Major bug reported by bmahe and fixed by bmahe (scripts)
HADOOP_PREFIX cannot be overriden
hadoop-config.sh forces HADOOP_prefix to a specific value:export HADOOP_PREFIX=`dirname &$this&`/..It would be nice to make this overridable.
Major bug reported by bmahe and fixed by bmahe (scripts)
Hadoop scripts unconditionally source &$bin&/../libexec/hadoop-config.sh.
It would be nice to be able to specify some other location for hadoop-config.sh
Major bug reported by szetszwo and fixed by szetszwo
Webhdfs: Support multiple namenodes in federation
DatanodeWebHdfsMethods only talks to the default namenode.
It won&t work if there are multiple namenodes in federation.
Major task reported by szetszwo and fixed by szetszwo (documentation)
Add WebHdfs Forrest doc
Critical bug reported by tlipcon and fixed by umamaheswararao (data-node)
BlockPoolSliceScanner spinning in loop
Playing with trunk, I managed to get a DataNode in a situation where the BlockPoolSliceScanner is spinning in the following loop, using 100% CPU:
at org.apache.hadoop.hdfs.server.datanode.DataNode$BPOfferService.isAlive(DataNode.java:820)
at org.apache.hadoop.hdfs.server.datanode.DataNode.isBPServiceAlive(DataNode.java:2962)
at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scan(BlockPoolSliceScanner.java:625)
at org.apache.hadoop.hdfs.server.data...
Major improvement reported by tlipcon and fixed by tlipcon (data-node)
Refactor BPOfferService to be a static inner class
Currently BPOfferService is a non-static inner class of DataNode. For HA we are adding another inner class inside of this, which makes the scope very hard to understand when reading the code (and has resulted in subtle bugs like HDFS-2529 where a variable is referenced from the wrong scope. Making it a static inner class with a reference to the DN has two advantages: a) scope is now explicit, and b) enables unit testing of the BPOS against a mocked-out DN.
Minor improvement reported by tlipcon and fixed by tlipcon (data-node)
Refactor DN configuration variables out of DataNode class
Right now there are many member variables in DataNode.java which are just read from configuration when the DN is started. Similar to what we did with DFSClient, we should refactor them into a new DNConf class which can be passed around - the motivation is to remove the many references we have throughout the code that read package-protected members of DataNode and reduce the number of members in DataNode itself.
Major improvement reported by tlipcon and fixed by tlipcon (data-node)
Some cleanup in BPOfferService
BPOfferService is currently rather difficult to follow and not really commented. This JIRA is to clean up the code a bit, add javadocs/comments where necessary, and improve the formatting of the log messages.
Minor improvement reported by tlipcon and fixed by tlipcon (data-node)
Move BPOfferService to be a non-inner class
Rounding out the cleanup of BPOfferService, it would be good to move it to its own file, so it&s no longer an inner class. DataNode.java is really large and hard to navigate. BPOfferService itself is ~700 lines, so seems like a large enough unit to merit its own file.
Major bug reported by qwertymaniac and fixed by qwertymaniac (name-node)
When 0 DNs are available, show a proper error when trying to browse DFS via web UI
Trace:{code}HTTP ERROR 500Problem accessing /nn_browsedfscontent.jsp. Reason:
n must be positiveCaused by:java.lang.IllegalArgumentException: n must be positive at java.util.Random.nextInt(Random.java:250) at org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:556) at org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:524) at org.apache.hadoop.hdfs.server.namenode.NamenodeJspHelper.getRandomDatanode(NamenodeJspHelper.java:372) at org....
Trivial improvement reported by qwertymaniac and fixed by qwertymaniac (data-node)
Use a set to manage child sockets in XceiverServer
Found while reading up for HDFS-2454, currently we maintain childSockets in a DataXceiverServer as a Map&Socket,Socket&. This can very well be a Set&Socket& data structure -- since the goal is easy removals.
Trivial improvement reported by eli and fixed by eli (documentation)
Add descriptions for dfs.*.https.address in hdfs-default.xml
Let&s add descriptions for dfs.*.https.address in hdfs-default.xml.
Trivial improvement reported by qwertymaniac and fixed by qwertymaniac (data-node)
Unnecessary double-check in DN#getHostName
We do a double config.get unnecessarily inside DN#getHostName(...). Can be removed by this patch.
Trivial task reported by joecrobak and fixed by joecrobak (documentation)
remove references to deprecated properties in hdfs-site.xml template and hdfs-default.xml
Some examples: hadoop-hdfs/src/main/packages/templates/conf/hdfs-site.xml contains an entry for dfs.name.dir rather than dfs.namenode.name.dir and hdfs-default.xml references dfs.name.dir twice in &description& tags rather than using dfs.namenode.name.dir.List of deprecated properties is here: http://hadoop.apache.org/common/docs/r0.23.0/hadoop-project-dist/hadoop-common/DeprecatedProperties.html
Minor bug reported by tlipcon and fixed by tlipcon (test)
DFSTestUtil may create empty files
DFSTestUtil creates files with random sizes, but there is no minimum size. So, sometimes, it can make a file with length 0. This will cause tests that use this functionality to fail - for example, TestListCorruptFileBlocks assumes that each of the created files has at least one block. We should add a minSize parameter to prevent this.
Major task reported by szetszwo and fixed by szetszwo (documentation)
Add WebHDFS apt doc
This issue is to add a WebHDFS doc in apt format in additional to the forrest doc (HDFS-2552).
Trivial bug reported by davevr and fixed by davevr (scripts)
hdfs jsp pages missing DOCTYPE [post-split branches]
Some jsp pages in the UI are missing a DOCTYPE declaration. This causes the pages to render incorrectly on some browsers, such as IE9.
Please see parent bug HADOOP-7827 for details and patch.
Major bug reported by szetszwo and fixed by szetszwo (documentation)
Some links in WebHDFS forrest doc do not work
Some links are pointing to DistributedFileSystem javadoc but the javadoc of DistributedFileSystem is not generated by default.
Critical bug reported by tucu00 and fixed by szetszwo
webhdfs HTTP API should implement getDelegationTokens() instead getDelegationToken()
The current API returns a single delegation token, that method from the FileSystem API is deprecated in favor of the one that returns a list of tokens. The HTTP API should implement the new/undeprecated signature getDelegationTokens().
Major bug reported by eli and fixed by eli (data-node, test)
TestDirectoryScanner doesn&t test parallel scans
The code from HDFS-854 below doesn&t run the test with parallel scanning. They probably intended &parallelism & 3&.{code}+
public void testDirectoryScanner() throws Exception {+
// Run the test with and without parallel scanning+
for (int parallelism = 1; parallelism & 2; parallelism++) {+
runTest(parallelism);+
Minor improvement reported by szetszwo and fixed by szetszwo (data-node, documentation, name-node)
Add a log message to show if WebHDFS is enabled
WebHDFS can be enabled/disabled by the conf key {{dfs.webhdfs.enabled}