Main drawbacks I see for using SSL are:
1 – the cost. If you want to do it right you'd have to go and buy an expensive certificate.
2 – you need a static IP address. Well you could do it with a dynamic one but then the costs would go up even higher.
Either you have to deal with those two issues, or setup your own trust server, which makes things way more complicated then you really want as well. I had somethings like this a while back with one of my applications. We took an entirely different approach: compiling on demand.
the process worked something like this:
1 – you go to the website and create an account.
2 – the system creates your account and with that account it creates an encryption key specific for your account.
3 – you login to your account on the website
4 – you press the download button.
5 – at this point, the system generates an encription.jar which contains all the encription stuff. This is done on the fly because your unique encription key is compiled into this jar.
6 – your download starts.
What this does is it creates an encription key that is linked to your login. Granted this has some problems as well, but for us it worked gret because for one the application was always installed on just one system at a time. So we could figure out which key to use that way. For your situation you could use something similar I guess but then this generated key would be the public key and you would use it to verify the client is who they say they are.
Oh and I put in one more thing to make things a little harder to hackers. Each time I sent an encripted packet from the server to the client, part of that packet was a new encription key. So the encription key is only ever valid for one exchange.