Deadly End Phantoms

You can enable or disable their spawning in End Midlands and End Highlands, and adjust their spawn weight and their minimum/maximum groups spawn size. You can also disable vanilla Phantoms entirely, making the Specter the only "Monster of the Night Skies".

Invisibility potions and smart bridge building are the way to not get constantly wrecked when crossing areas in the End now. They won't be able to detect you at ground level even if you're wearing armor, but if you're in an End City and you float up to their level you will most likely be in their detection range.

Have fun getting levitated by Shulkers and then swarmed by Specters in the End Cities, that's absolutely intended and will not be disabled. The main goal in making this mod was to make ascending the buildings of the cities harder as you can cheat your way up using the Shulker's levitation effect in vanilla.

Specters do not spawn in The End based on insomnia, they spawn naturally like Endermen. Since you can't sleep in The End, it's pointless to make them only spawn based on sleep, so they just spawn consistently at what is intended to be a manageable rate. Carry a shield and maybe a backup shield as well if you're not already using invisibility potions to avoid them.

The End should be a challenge as it's literally the end area of the game. If you're good enough to defeat the Ender Dragon without cheating, you can certainly handle a few flappy bois on your way to getting your Elytra.