I have been running a top-mount starter on a 3-point bell housing with a flexplate flawlessly since 91. Have also run the same setup with a flywheel in others with no issues. The distance from the block to the ring gear should be the same no matter what. You should not need any shims between the crankshaft and flywheel/flex plate. Is the flex plate backwards?
The vibration is another issue. I can't remember seeing an aftermarket flex plate without holes and all OE flex plates have holes. It is also possible that the holes be smaller or odd-shaped on the side where the counter weight would be.
The flex plate for a 454+ ci is typically externally balanced. Those for </=427 are internally balanced. The difference is that the external balanced flex plate will have a counter weight welded on to compensate for an improperly designed crankshaft counter weight.
Steel flywheels could have a counter weight welded on or the side opposite could have holes drilled to remove an equal amount of mass. Cast flywheels typically have the counter weight cast in but could also have holes.
Did this issue start when you changed to the flex plate? I bet you don't have a crankshaft dowel pin. Can you verify that the flex plate and crankshaft are clocked correctly and that you do indeed have a counter weight present?
If that is correct, I would check the driveline and then the damper to ensure the rubber on the outer ring is still good.
Good luck,
Joe