Hadoop Source code change in replication policy of hdfs

Cancelled

Job #: 9812 Post Date: 09.07.2017 14:05

General information for the business: hadoop replication policy Kind of development: New program from scratch Description of every module: reports from large Internet companies have indicated that 10% files has been hitted by 90% users, which means there is a strong skew about the popularity of files distribution. Unfortunately, distributed file systems or databases like HDFS or Hbase use the static and same replication parameter for every file or table. This results in some servers are hot with high bandwidth consumption and MapReduce tasks or Hbase query running on, while other servers are relatively cold, and thus hurt user's experience and data-center's utilization. Goal: a dynamic, online approach to analyze which block in HDFS is hot, and where should be the new replication copy placed on. Description of requirements/functionality: reports from large Internet companies have indicated that 10% files has been hitted by 90% users, which means there is a strong skew about the popularity of files distribution. Unfortunately, distributed file systems or databases like HDFS or Hbase use the static and same replication parameter for every file or table. This results in some servers are hot with high bandwidth consumption and MapReduce tasks or Hbase query running on, while other servers are relatively cold, and thus hurt user's experience and data-center's utilization. Goal: a dynamic, online approach to analyze which block in HDFS is hot, and where should be the new replication copy placed on. OS requirements: Linux Extra notes:

Posted by

Required skills

JavaXMLJavaScript

Get the job done

Budget:
$30
-$49
Job type:
One-time
Hiring from:

?Bidding for first time? Click here to learn how StartAJob platform works.

?Want to do the job or have questions for the customer? Write then in the " Write a note or message to the customer"

?You already have the estimated cost and time to complete the work yuo are in the fields "Execution time" and "Payment requested" or leave them empty and can be changed later.

Invite