I think balance is the key (as almost always). Don't feel bad about enjoying your coffee the way you like it.
I made it a personal rule to not drink my calories in my everyday life (parties and special occasions is a different story). I mostly drink white or green tea and water. I don't have a caffeine habit, but I do enjoy the occasional cup of joe with milk in it.