To get yagoro on your side, eliminate all 12 shinbakufu members in the main story first. Complete naoe's quest "secrets of the blade" which opens after finishing wolves and foxes in act 2, this unlocks the brotherhood chambers. Make your way to engyoji in harima north of maniden hall, where you'll find a corpse with blade wounds. Follow the blood trail to hideyoshi camp, engage and win the duel with yagoro. Take out the lead samurai at jujiin courtyard, and when talking afterward, choose "you have the potential" to seal the deal.

Tip: get your dojo to veteran rank before recruiting, it'll maximize his assassination abilities. Once he joins, you'll get access to chain assassinations and improved stealth kills that work on tougher enemies. Worth noting that while he's not a romance option, his questline "past and present" shows some interesting character development as he adapts to the shinobi way of life.