I would vote for Old Saybrook (OSB). In the morning and afternoon, you have Shore Line East trains for New Haven, sometimes powered by a "New Haven-painted" GP40(etc etc) or a leased Amtrak P40. There's a fair amount of Amtrak trains passing through as well, led by AEM7s or HHP-8s, plus the Acelas, which go through at nearly 100mph without stopping. P&W usually has two freights which meet and switch cars on the wye on weekdays as well. Plus you may see Amtrak work trains, led by grey locomotives. There are a couple high-level pltfms, one of which is in the center so you get good views from the ends down the line. In the freight station next to the platforms is Pizzaworks, a pretty good restaurant with a railroad-theme.
If you wanted to hit up New Haven, you subtract the P&W activity, but add lots and lots of Metro North trains, plus a few more Amtraks including the shuttles. There are a couple Amtraks which change motive power there as well. CSX does have a daily freight and P&W runs stone trains in the warmer months.
I've never been hassled by anyone at either station, as long as you stay out of the way of passengers on the platform (ie-no tripods, etc) and stay in public areas (of course). OSB seems to attract more railfans since its more laid back, while NHV is a beehive of activity.