The owners of the old Maccess building off Spen Lane believes more should have been done by Kirklees Council to secure the site at Spen Lane.
Castlerock Capital in London, which owns the building, has been sent a bill by Kirklees Council for securing the site, which caught fire last month.
But owner of Castlerock Mark Whitehead believes it should have been the responsibility of the council to make sure the site was secured in the first place.
He said: “I have been working closely with the council over the last few years. I am disappointed that the local authorities have been unable to protect our premises. This has been at a considerable cost to us.
“I am working with the council on a potential solution but have no further comment at this stage.”
People living near the building had said it was a hang-out for young people, and Kirklees Council insisted it had no other choice than to secure the site itself.
A spokeswoman from the council’s pollution and noise control department said: “We were informed by the fire service that the building was open to access when the fire took place last month.
“We have now secured the site ourselves and have billed the owners for the work.”
The building was served with a demolition order last week, with work due to start by February 7. The company did not say whether it would appeal the order.
Meanwhile, the Northgate Mills site on Serpentine road could come down by the end of the month. Cleckheaton councillor Kath Pinnock revealed she received an email from council officers which said they believed work to start demolishing the site could start in the last week of January.