335i is a beast in it's own right ... Easy to mod, can be adapted to use M parts like control arms and etc, torque spreads all over the rev range plus newer electronics like DTC ... However M cars like the e46 are track ready beast right out from the factory ... well special factory within a factory :4:
like you say it's a formula ... everything is made to complement each part of the car the engine, the software, the chassis, suspensions, bushings, the drivetrain and LSD. So what you have is one cohesive package that makes it a bit more special.
I enjoy driving a non M and a M car ... both have their own strength. My 330 feels faster around town than my e36 due to the way the torque delivers from 1.5k rpm and above however on track or on highway the M feels much more stable at higher speeds and gives more confidence even though there is no electronics stability control.
Oh and the noise ... the wonderful mechanical clanking of the M engine. It sings horribly at low revs but once the revs rises past 5.5k rpm up to the top ... Then it starts to sing beautifully ... the true voice and soul of M cars sits atop at the rev range.
M cars especially older ones like mine is not fast by today's standard nor the easiest to own but the driving experience makes it all worth it.