I think about the cost of clothing in the context of lifetime cost. If I buy a pair of jeans, how long can I wear those jeans for? I have no problem spending a bit more if it will last a lot longer. That brings the lifetime cost down. In doing that, it also decreases significantly the environmental toll. In most cases it means better craftsmanship also, which usually comes from better paid employees that take pride in their work.
It also means I wear the same clothing a lot longer than most people do, so I'm not keeping pace with fast fashion at all.