The OL can support tighter headways within the existing signal system with more cars. I don't know precisely how much because I don't know how the signal blocks are arranged, but Red Line is a good measuring stick and the OL currently runs much longer headways than Red. I don't see any reason why they couldn't pack them in the same, as it's a shorter single line that doesn't throw any traffic management curveballs like Columbia Jct. or Harvard curve and is generally arrow-straight on the portions rebuilt in the last 35 years. The 01200 replacement order is for 146 cars, a +26 increase from the 120 currently available. That's 4 more potential rush hour consists plus a spare set.
ATO is a mixed bag as a signal system. On the one hand it's safer and more reliable than trip arms, which are mechanical equipment that outdoors has to have heaters attached to work in bad weather. ATO is just a solid-state track circuit. For maintenance and reliability solid-state hardware is always better bet than mechanical. The only heavy-rail collisions the T has experienced in the agency's lifetime outside of yard limits have been operator error because trip stops can't enforce speed limits, only stop at a block. The downside is that ATO is still a pretty "dumb" system, being just a 1-bit one-way pulse signal. The operator still has to control the vehicle within the speed limit or get a stop penalty with everyone getting thrown from their seats when the brakes get suddenly applied. And staying within that small margin of error at the speed limit and reacting in the 5 seconds the operator has to slow down before getting the penalty means a herky-jerky ride and much wider train spacing because of all the unpredictable penalty stops. This isn't as much a problem on the longer-headway and car fleet-poorer OL as it is on the RL where those stops really start to add up and hose rush hour schedules. The T compounded the issue by not taking into account the traffic loads and platform dwell times downtown with tighter signal block spacing, instead opting for pretty much uniform blocks. That kills it from the Park St. approach to past Broadway when everything gets backed up due to long unloading/loading dwell times at the Park and DTX platforms. There's no leeway to catch up when it takes a minute longer than scheduled to close the doors because the platform is packed like there would be if they chopped into smaller blocks.
The future is Communication Based Train Control (CTBC) signaling, which is basically "smart" 2-way high-bandwidth ATO between computers controlling the signals and an onboard computer on the trains. Allows the trains to communicate back to traffic control instead of just getting that rigid 1-bit signal, and the track circuit pulses can convey much wider variety of information computer-to-computer than it can "dumb" system to "dumb" system. Allows the train to make much more fine-tuned speed adjustments, and the central control computers to make signaling decisions based on the whole line's traffic load because it can real-time track positions of every train on the line (i.e. no need for a GPS unit because the cab signals do it themselves). Operators still have to manually obey the speed limits or go into penalty braking, but it reduces their responsibility a bit more and isn't nearly as all-or-nothing as an ATO speed penalty. All of these are the reasons why this system can allow much tighter signal spacing...makes more nimble adjustments, doesn't have to over-space as ridiculously much for penalty buffers on the schedule.
This is already in use on I think 5 worldwide metro systems and is getting to be pretty mature technology. NYC is testing CTBC on the L train as a trial, and the T's got studies programmed for the Green Line and Blue Line. This is basically the system that would replace the Green Line's full manual block signals in the subway and on the grade-separated branches (surface branches still obeying traffic lights as before), as CTBC does have light rail applications with even tighter spacing in exchange for a little more operator leeway. Goal is to actually get something in place in some future when the T actually has money. For Blue they're just doing the research and haven't committed to building anything. What would force a decision there is if the Lynn extension got built, as it would almost double the above-ground route mileage. They'd be reluctant to double the number of trip arms and trip arm heaters and probably would bite on CTBC from Day 1. Retrofitting the rest of the line probably wouldn't touch the subway until some later date (OL operated for 20 years with half-ATO/half-wayside), but would allow them to decomission all those weather-beaten trip arms and heaters above-ground, likely with the switchover happening at Airport simultaneous with the 3rd rail/overhead switch. And then they'd tackle the subway later where the climate control doesn't wear out the trip arms as fast and the route miles are very small.
Some costs (in 2003 $$$) for upgrading:
http://www.bostonmpo.org/bostonmpo/pmt-old/PMT-2.pdf. It's expensive mainly because it involves replacing 100% of the signal wiring with fiber, and no line currently has existing fiber signal wiring except for the spanking-new OL ATO Haymarket-Oak Grove and GL North Station-Lechmere (+ planned for the extension). Blue would cost $228 mil to retrofit, Orange would cost $367 mil, and Red a whopping $789 mil. Reason why they're only studying it for Blue, even though Red needs it extremely badly just to keep up with current traffic growth. CTBC would allow minimum 2-minute headways vs. 3-1/2 minutes on Red and Blue and 4-1/2 minutes on Orange, which also requires substantially expanding the vehicle fleets on each line to fill in those headways. You can see from that MPO doc what kind of ridership impact that has, though. +8000 to +11,000 new riders on each existing line with no extensions, and attracting anywhere from +2700 to +4500 brand new transit riders per line who aren't using any T service. Bigtime stuff. Subway signal replacement is...almost invisibly...one of the single biggest revenue boosters the T can undertake. In case you wanted another reason to hate how much time and money they're wasting pursuing frivolous outer-suburb commuter rail disaster projects. Every single one of these individual CTBC installs would convincingly outhaul South Coast Rail in new fare-paying riders.