hadoop - Where Does the HDFS Account for Triple Replication in Usage Reports? -


in latest version of hadoop distributions, hdfs usage reports seem report on space without accounting replication factor, correct?

when 1 looks @ namenode web ui and/or runs 'hadoop dfsadmin -report' command, 1 can see report looks this:

configured capacity: 247699161084 (230.69 gb) present capacity: 233972113408 (217.9 gb) dfs remaining: 162082414592 (150.95 gb) dfs used: 71889698816 (66.95 gb) dfs used%: 30.73% under replicated blocks: 40 blocks corrupt replicas: 6 missing blocks: 0

based on machine sizes of cluster, seems report not account triple replication... i.e. if place file on hdfs, should account triple replication myself.

for example, if placed 50gb file on hdfs, hdfs dangerously close full (since seems file replicated 3 times, using 150gb remain)?

let define each of these terms mean.

  1. configured capacity: total capacity available hdfs storage. if have 4 nodes , each node has 50 gb capacity, configured capacity 200 gb. replication factor irrelevant in case of configured capacity.

  2. dfs used: amount of storage space has been used hdfs. divide dfs used replication factor actual size of files stored without replication. if dfs used 60 gb, , replication factor 3, actual size of files 60/3 = 20 gb.

  3. dfs remaining: amount of storage space still available hdfs. if have 150 gb remaining storage space, mean can store upto 150/3 = 50 gb of files without exceeding configured capacity (assuming replication factor = 3).

  4. present capacity: amount of storage space available storing user files after allocating space metadata. difference:(configured capacity - present capacity) used storing file system metadata. , inode information.

hope clears up.


Comments

Post a Comment

Popular posts from this blog

curl - PHP fsockopen help required -

i trying work on cloudflare api , have found in documentation. unfortunately have no idea how can use in fsockopen or through curl. here example of post request. need know how can make request below data post using curl or fsockopen curl https://www.cloudflare.com/api_json.html -d 'a=cache_lvl' \ -d 'tkn=8afbe6dea02407989af4dd4c97bb6e25' \ -d 'email=sample@example.com' \ -d 'z=example.com' \ -d 'v=agg' here link cloudflare https://www.cloudflare.com/docs/client-api.html#s4.2 try one: $ch = curl_init("https://www.cloudflare.com/api_json.html"); curl_setopt($ch, curlopt_returntransfer,1); curl_setopt($ch, curlopt_followlocation, 1); curl_setopt($ch, curlopt_ssl_verifypeer,false); // handling of -d curl parameter here. $param = array( 'a' => 'cache_lvl', 'tkn' => '8afbe6dea02407989af4dd4c97bb6e25', 'email' => 'sample@example.com', 'z' =...
Read more

HTTP/1.0 407 Proxy Authentication Required PHP -

i working in restricted network/limited access internet. since have use proxy make web service call. here php code calling web service , response . $client = new soapclient("http: //www.xyz.com/abc/wsdls/login.wsdl", array("trace" => 1, "exceptions" => 1, 'proxy_host' => $proxy_host, 'proxy_port' => $proxy_port, 'proxy_login' => $proxy_login, 'proxy_password' => $proxy_password)); response is fatal error: uncaught soapfault exception: [wsdl] soap-error: parsing wsdl: couldn't load ' http://www.xyz.com/abc/wsdls/login.wsdl ' : failed load external entity if copy url error message , paste in browser(with same proxy settings) works. 2nd scenario is, if pass values instead of variables giving me diffarant response $client = new soapclient("http: //www.xyz.com/abc/wsdls/login.wsdl", array("trace" => 1, "exceptions" => 1, 'proxy_host' ...
Read more

c# - Resource not found error -

i have problem accessing method on homecontroller. show code of method : [httpget] public actionresult decriptidentifiantetredirige(string login_crypter, string mdp_crypter) { string loginacrypter = _globalmanager.protegemotdepasse(login_crypter); string mdpacrypter = _globalmanager.protegemotdepasse(mdp_crypter); user userapp = new models.user(login_crypter, mdp_crypter); if (userapp.authentificationvalidee(userapp.userlogin, userapp.password)) { session["name"] = userapp.userlogin; return redirecttoaction("accueil", "home"); } else { return redirecttoaction("validerauthentification", "home"); } } then in routeconfig.cs wrote route : routes.maproute( name: "authentificationapresdecryptage", url: "{controller}/{action}/{login_crypter}/{mdp_crypter}", defaults: new...
Read more