I wouldn't say it's dumb not to believe in the Big Bang. But with a lot of people, the alternative to the big bang is believing in creationism which I guess is why people get called dumb for not believing in the Big Bang

The Big Bang is just a physical theory like any other, and some of what you said is correct, the theory will probably be proven to be incorrect, or at least refined and the theory made better.
But maybe it is a little dumb not to at least acknowledge that it is our best understanding of how we started, so far. At least it is built upon mathematical foundations. If you don't believe in the Big Bang or religious creation, then what do you believe? If the answer is nothing, then that is just boring, have an opinion :)