The usual objections to a new intermodal terminal have to do with truck traffic in and out rather than rail operations themselves. From what I recall of the area it is residential, with no real arterial highways nearby, and while it can be argued that railroad activity (including floodlights and noise) could have been anticipated by people who moved into the area, it can also be argued that long lines of tractor-trailers coming out of and waiting to get into what will amount to a major truck terminal could not. (I don't know whether road access is possible without clogging streets never intended for heavy truck traffic.) On the other hand, saying that the railroad was there first is putting it mildly -- wasn't Mount Clare the first railroad station in Baltimore? I suppose the site was chosen in the first place because it was on the outskirts in the 1830's. Solomon may have to decide this one.
