hadoop - Unable to find large files in Linux using "du -h" -


i have hadoop cluster running out of space recently. try clean logs disk space. run command df -h, shows:

/dev/sda1        22g   20g  1.9g  92% / /dev/sda3       1.8t  747g  960g  44% /data/1 /dev/sdb1       1.8t  755g  986g  44% /data/2 /dev/sdc1       1.8t  754g  987g  44% /data/3 /dev/sdd1       1.8t  745g  996g  43% /data/4

the hdfs under dir /data fine. root dir / has little space left. used tool ncdu can calculate rapidly disk usage of dir, shows:

  2.9tib [##########] /data   1.5gib [          ] /home 800.9mib [          ] /usr 716.3mib [          ] /var 349.1mib [          ] /lib 293.8mib [          ] /opt

there no large directory. tried command such du -a | sort -n -r | head, still unable find invisible dir or file. know other way find out problem is?

thanks

i find answer.

cause have deleted large log file didn't reclaim space in filesystem , it's still taking disk space. that's why result of commands du -h , df -h not match.

solution

root@linux:# lsof / | grep deleted java    15393    hdfs    114w    reg    8,1    4463704    655506    /var/log/hadoop/hdfs/hadoop-hdfs-datanode-hslave.log (deleted)

the command lsof list open files. file opened when deleted it. second column shows pid of process opened file. 

root@linux:# cd /proc/15393/fd/ root@linux:# ls -l | grep deleted l-wx------ 1 hdfs hadoop 64 jul 25 17:55 114 -> /var/log/hadoop/hdfs/hadoop-hdfs-datanode-hslave.log (deleted) root@linux:# > 114

type file descriptor shows in previous line (114). space reclaimed.


Comments

Post a Comment

Popular posts from this blog

curl - PHP fsockopen help required -

i trying work on cloudflare api , have found in documentation. unfortunately have no idea how can use in fsockopen or through curl. here example of post request. need know how can make request below data post using curl or fsockopen curl https://www.cloudflare.com/api_json.html -d 'a=cache_lvl' \ -d 'tkn=8afbe6dea02407989af4dd4c97bb6e25' \ -d 'email=sample@example.com' \ -d 'z=example.com' \ -d 'v=agg' here link cloudflare https://www.cloudflare.com/docs/client-api.html#s4.2 try one: $ch = curl_init("https://www.cloudflare.com/api_json.html"); curl_setopt($ch, curlopt_returntransfer,1); curl_setopt($ch, curlopt_followlocation, 1); curl_setopt($ch, curlopt_ssl_verifypeer,false); // handling of -d curl parameter here. $param = array( 'a' => 'cache_lvl', 'tkn' => '8afbe6dea02407989af4dd4c97bb6e25', 'email' => 'sample@example.com', 'z' =...
Read more

HTTP/1.0 407 Proxy Authentication Required PHP -

i working in restricted network/limited access internet. since have use proxy make web service call. here php code calling web service , response . $client = new soapclient("http: //www.xyz.com/abc/wsdls/login.wsdl", array("trace" => 1, "exceptions" => 1, 'proxy_host' => $proxy_host, 'proxy_port' => $proxy_port, 'proxy_login' => $proxy_login, 'proxy_password' => $proxy_password)); response is fatal error: uncaught soapfault exception: [wsdl] soap-error: parsing wsdl: couldn't load ' http://www.xyz.com/abc/wsdls/login.wsdl ' : failed load external entity if copy url error message , paste in browser(with same proxy settings) works. 2nd scenario is, if pass values instead of variables giving me diffarant response $client = new soapclient("http: //www.xyz.com/abc/wsdls/login.wsdl", array("trace" => 1, "exceptions" => 1, 'proxy_host' ...
Read more

c# - Resource not found error -

i have problem accessing method on homecontroller. show code of method : [httpget] public actionresult decriptidentifiantetredirige(string login_crypter, string mdp_crypter) { string loginacrypter = _globalmanager.protegemotdepasse(login_crypter); string mdpacrypter = _globalmanager.protegemotdepasse(mdp_crypter); user userapp = new models.user(login_crypter, mdp_crypter); if (userapp.authentificationvalidee(userapp.userlogin, userapp.password)) { session["name"] = userapp.userlogin; return redirecttoaction("accueil", "home"); } else { return redirecttoaction("validerauthentification", "home"); } } then in routeconfig.cs wrote route : routes.maproute( name: "authentificationapresdecryptage", url: "{controller}/{action}/{login_crypter}/{mdp_crypter}", defaults: new...
Read more