The three big secret bosses need endgame gear and full exploration abilities unlocked first, so don't try these early. Simon lurks down in the abyss after you've got esquie's movement skills, just drop past some mini-bosses and off that big cliff to find him. Clea shows up in flying manor once you can properly fly around, and her whole thing is countering attacks, miss those and she just heals back up which is pretty rough. Painted love rounds out the tough trio in act 3, all giving some really good story bits if you can take them down.

For smaller challenges, the chromatic chapelier appears in the crows area west of visages, but only after you've shot every single crow there, so keep track as you go through. They're tough against light and dark damage, and those silence masks are pretty nasty if you're not ready. Maybe interesting that the lampmaster eveque goblu and dualliste aren't actually secret at all, they're just regular story bosses people sometimes miss.

Really important note, all these fights need top-tier equipment and good party setups, so definitely sort that out first. The rewards are worth it though, some of the best gear in the game plus achievements if you're into that. Lots of character backstory stuff too if you're following the lore.