Let me explain what you need to know about rare seeds - you actually can't get them directly on the island, but there's an interesting reason for that. The main source is going to be the traveling cart, which shows up in cindersap forest on fridays and sundays (this is by design, as it makes these valuable seeds feel more special when you find them). During spring and summer, you're guaranteed to find them for 1,000g each, while in other seasons there's about a 1% chance to spot them at varying prices.

Here's something neat - while you can't buy the seeds on the island, its special climate means you can grow them any time of year once you have them. This is actually really powerful, because normally these crops are season-restricted. You might also want to know about using the seed maker, which lets you turn your harvested sweet gem berries into 1-3 new seeds (though you'll make more money just selling the berries and buying new seeds from the cart).

A little tip that might save you some time - don't bother trying to process sweet gem berries into artisan goods, it won't work. The game balances this by making them incredibly valuable on their own. And while exploring caves and fighting monsters is fun, you won't find rare seeds as drops anywhere - stick to the cart and seed maker strategy, and you'll be swimming in sweet gem berries before you know it.