Hadoop Errors

List of Hadoop Errors

A :

 Incompatible clusterIDs in Hadoop :  Some time you get below error in your logs , in this case you will not find datanode as running instance.

Error Stack - Incompatible clusterIDs in /usr/local/hadoop_store/hdfs/datanode: namenode clusterID = CID-52c3b330-d913-461d-8a5a-03c1659390bd; datanode clusterID = CID-bfea0588-2680-43cb-aa69-d5464168ca88
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:646)
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:320)

                                                           Fix : 

ClusterIDs of datanodes and namenodes should match, then only can datanodes communicate with namenode. 

When first time you run your hadoop instance (using start-all.sh or other way) you can see datanode is in running state, but after that if you format your name node then  new ClusterID will be assigned for namenodes  and ClusterIDs in datanodes won't match.

You can locate a VERSION files in your /usr/local/hadoop_store/hdfs/datanode/current/ (datanode directory ).

You can fix above issue by two way

1)-  If you dont want to delete data of datanode then go to  /usr/local/hadoop_store/hdfs/datanode/current/  open version file and update clusterID value with updated one (as mention in error trace).

2)- If you are not ok with above apporoach you can execute below commands to generate new clusterId.

   a)- Stop all HDFS services.

   b)- rm -rf /usr/local/hadoop_store/hdfs/datanode/* 

   c)- rm -rf /usr/local/hadoop_store/hdfs/namenode/*

   d)-format hdfs again (hadoop namenode -format )