Is it just your wardrobe though, or does this habit exist elsewhere in your life? What's your desk like - is it clear of papers, or scattered with them, and if scattered, why is that? Is it because you are bored and you want to make yourself feel busy or look important? Or is it that you really can't be bothered to make the effort?
Whatever the reason, do you realise that you are projecting an image of yourself - and is it the image you want to project? More than that, how will you feel with a tidy office that you can find things in, and that you are proud to show people into?
Making that change is not so hard - it starts with a decision. Once you've decided you are ready to get tidy, work out what you need to make it happen. Practical things like plastic sacks to put the rubbish in, boxes to put things in to give to your local charity shop and maybe a good friend to help you make it fun and make it happen! Then simply make a date in your diary - just tackle one area at a time - you'll be amazed at what a difference it makes and how good you feel.
Don't stop until it's done, but do give yourself a reward at the end of it!