It's actually something I've thought about quite a bit. (I was already mulling over cultural appropriation before it was cool, e.g. in Four British Fantasists, a book published a full year before the founding of Tumblr.) One conclusion I've come to is that each case needs to be looked at individually. Tedious as that may seem, it kind of goes with the territory when you're talking about respect.
Here are some of the considerations.
What is the traditional role of the garment/object/style? Is it sacred or profane? Is it traditionally reserved to certain classes of people?
In the case of the yukata, it's simply a light, summer kimono. Like the kimono it's simply old-fashioned clothing, not sacred garb, although it's still worn at summer festivals and the like. From that point of view, it's no more cultural appropriation than is Japanese people wearing Western-style suits. But that comparison leads to the second possible objection, namely...
Does a European taking up that custom perpetuate a historical relationship of exploitation through slavery or colonialism with the originating culture?
In the case of Japan, there really is no such history with my own culture. (For a Brit, this is a rare treat.)
How do people in the country concerned feel about it?
This is hard to answer, and obviously I can't speak for every Japanese person, but from my own experience, reading what I can on the subject, and watching many a Youtube voxpop (e.g. this one), I have yet to encounter a single Japanese person in Japan* who has anything negative to say about Westerners wearing yukata. In fact, from my admittedly limited observation I'd say that cultural appropriation as a concept is alien to Japanese culture. Arguably, to apply the discourse of cultural appropriation to Japan is in itself a kind of cultural imperialism, but that's a rabbit hole I don't intend to go down here.
This doesn't mean that there are no ways to be offensive in Japanese culture. For a selection of ways to do that, I give you the example of Logan Paul, whose career has just ended as a result of it.
* People of Japanese descent brought up in Western countries may be a different matter.