What if I just plugged the weep hole. What would happen?
You're not fixing the problem of course. What could happen is that leak is now internal, could force the water into the main bearing and cause damage by washing out grease/lubricant. I'm not exactly sure what will happen but at some point I think this leak may show in another place which could make for a larger problem. How much water are we talking? I don't want to give you info like add another bilge pump or plug the leak cause you would just prolong the inevitable and put yourself in a potentially unsafe situation.
Think about this,
i just spent about $500 w/o impeller and a few other internals to freshen my energized YJ, I did the labor. You are probably only looking at $50-$100 max in parts to fix your issue right now if you put in the labor. I would consider the savings and take care of it before you have to replace many more parts. Probably a one day deal if you had all the parts and tools on hand, then a few beers while you watched the silicone set up after installing back into your boat!