How do you do baggy in the right way?

I see some people wear a baggy t-shirt over slacks or jeans and it can look so good, same thing with big knit sweaters or a men's button down. It looks so artsy and nonchalant.

Other times it can look a bit awkward or just not stylish. What's the "right" way to wear baggy clothes?