I grew up in Danbury in the 50s, when the hatting industry was beginning to die but was still quite active. A couple of the biggest hat shops were located railside: Mallory's on the Maybook Line and Lee's on the Norwalk Line. Both of those had coal trestle sidings (Mallory's, stoutly built of steel and concrete, is still there!). A couple of large fur shops, Empire and Federal, were located alongside the Norwalk line but oddly neither had sidings.
Other industries on the Norwalk Line were: McCarthy Coal Co. located just around the loop from the passenger station (and exactly on the site of the current Metro North station parking lot) was a huge coal shed and at one time had been two coal sheds; Sperry Rail Service --- perhaps not a rail customer in the usual sense but they did and still do maintain a siding; Preferred Utilities, right across the track from Sperry; Swift Premium meats reached by a long siding off the station loop track.
Other industries on the Maybrook west of the station were: Omaha Beef, still in business but has not used rail for years; a grocery wholesaler whose name escapes me located just west of the Main St. crossing; and --- let's not forget --- the Great Danbury Fair which received live livestock (for display purposes!!) during annual fair week as well as an occasional passenger special move; Danbury Block & Supply Co. west of Fair Grounds (on the site of present Stop & Shop supermarket). There may have been customers out in Mill Plain that I can't recall.
North of the station on the Berkshire was Stavola Scrap Iron (on the site of current Waste Management) and one of the McLachlan Hat shops just across the tracks from the engine terminal area.
As you yourself noted there were active team tracks serving most hat and fur shops that were not located railside.
Those are my thoughts at the moment. I have not mentioned Bethel (3 miles south of Danbury going toward Norwalk) which also had some healthy freight business then.