The problem you have is a rare case but if one testicle is small then there is no need to worry. I hope you might know that one testes is always bigger than the other one. But it does not means that you have a problem. If one testicle loses or not grown then the other one is still enough to make sperm.
The small one grows a bit to increase sperm output. If you don’t know then two testicles have tow functions. One is for producing testosterone and the other one is to produce sperm. So if testes is small then no need to worry as it can still produce testosterone but not in large amount.
If you go to the fact then having a normal sperm count is really a difficult one as most people suffers from low sperm count inspite of having normal testicles. And this means fertility problems.
So my friend, if your one testes has not grown then the other one is still in normal that is enough to help you to make babies.
If you still have doubt and need further help then you can contact or visit to your doctor. He/she cam better guide you further.