You can actually run without seals. On the exhaust, it is not a problem because the port is pressurized. The intake port, operating under vacuum, will consume more oil. Oil in the combustion chamber will cause detonation.
What would concerne me more is the debris floating around your new engine.
Why are the seals coming lose? Is it that the guide diameter is undersized and the seal is falling off? Or are the seals coming apart?
If the retainers are hitting the seals, you are just one over-rev from serious carnage.