Historically Haweli Kharagpur was the centre of the medieval Kharagpur Raj chieftaincy. It has references in Mahabharata when Pandavas had to face an exile with disguise. Then Pandavas came to hide in the hills of Jamalpur which is on the southern side of Haveli Kharagpur. People who go to Baidyanath Temple during the holy month of Shraavana start from Sultanganj and go via Tarapur but while returning they come via Jamui and Haveli Kharagpur. It is customary to visit the Panchbadan Temple while returning.
Haveli Kharagpur has a rich history and was one of the central points of administration during and pre-British Raj. It was a coveted place for the kings who built Mansion. According to the Bihar and Orissa District Gazetteers Haveli Kharagpur was surrounded by Mahalat Kharagpur (an extensive estate on the revenue roll of Bhagalpur). When Raja Rahmat Ali Khan fell into arrears of revenue in 1840, he sold Haveli Kharagpur along with Mahalat Kharagpur to Raja Bidyanand Singh. Raja Bidyanand Singh was the grandfather of the contemporary proprietors of Raj Darbhanga.