Do I have to buy a ticket to visit just the gardens?

Yes, the standard entrance ticket includes both the gardens and the interiors of the Villa