Boss Attacks
As Nightmare 1 and 2's, except:
Throw-to-the-ceiling (Gloopy form only) - If you put even a toe into Nightmare's gloop, he will instantly fly up to the ceiling. He will then send several guided missiles, one after the other, after you - running around the edges of the arena will throw them off your track. If you go to the centre of the arena, a wiremesh cage will drop down and imprison you. If you don't escape from from the cage, Nightmare will drop down on you for humungous damage (1 and 3/4's of your 2 lifebars in DMD mode). To escape, attack one of the corners with melee attacks such as sword slashes or Rolling-Blazed jumps - a hole will appear, which you can go through. Once free, go to the edges of the arena and wait until Nightmare has dropped back down - don't step into the gloop for a few seconds after he has landed, as he can still hurt you even after landing. Afterwards, you can safely step on Nightmare's gloop, as it will take him a long time to do another throw-to-the-ceiling.
Yellow Bolts (red lifebar only) - These lame-o bolts only appear late on in the battle. They do minimal damage on their own, but can stun you and set you up for heavier damage from another attack. A yellow trace appears for 1-2 seconds first, giving you enough time to jump away from it. If you're attacking a core, then these bolts won't target you - you'd think that would be the best time, eh...? >_>
Underworld:
Once again, this is the same as the Underworld you get in the Nightmare 1 and 2 fights, but the Shadow boss is Nelo Angelo. Unlike with the previous two bosses, he won't appear until you've defeated the last Sargasso - if you know exactly where he drops down you can start attacking him straight away, but this takes a lot of experience to learn. Anyway, Shadow Nelo has the same moves as Nelo 1 - no Phantom Swords, thankfully. Although his attacks are strong, his defense is considerably weakened and he will go down very quickly to DTed sword combos or Ifrit. If this isn't a new game and you have the Bangle of Time, you can use it to defeat Nelo cheaply: freeze time whilst he is taunting or stunned, then wail on him non-stop with a grenade -> Kick-13 combo.
The Final Core:
Once Nightmare 3's lifebar is emptied, instead of just doing the decent thing and dying, so that you can get on with the game, he regains 25% of his life and brings out his third and most dangerous core by far - the final core. It's red but has the defense of a blue or green core. Nightmare meanwhile becomes a hybrid of his gloopy and solid self. In this form, everything but the kitchen sink is slung at you: lasers, ice slugs, yellow bolts, missiles, and gloopy swirls - oh, and the final core drains your DT on a regular basis.
But luckily you have Air Raid. And Vortex Lvl 2 :p
Trying to melee the final core in such chaos is sheer madness, so just fly around and Air Raid it instead. On Normal/Hard modes, Air Raid will be good enough. On DMD mode, though, Vortex Lvl 2 might be a better choice. Whereas 10 runes of Air Raid can't finish the final core off on DMD mode, 10 runes of Vortex just about can. Admittedly, it requires some luck though. To Vortex effectively, aim the analogue stick at the core and button-mash as fast as humanly possible. If you still run out of DT before the end, get to one of the sides ASAP and grenade-roll through the lasers, etc., until you've built some DT up, then resume Vortexing/Air Raiding.
Nightmare 3's Patterns/Mechanics:
See the sections on Nightmare 1 and 2's Patterns/Mechanics. It's all covered there.