Hi jondx,
yes, it's not necessary: as long as you stay in the same subnet, router doesn't matter...
Anyways i'm still trying to understand your network configuration. Ok, you have an Ubuntu server you want to connect to, running in a virtual machine on a PC... So you have an OS hosting an Ubuntu guest, and that's almost sure.
Now, BackBox is on a different PC, or it's another virtual guest on the same machine? What do you mean with "host"? The BackBox system? Are you trying to access the server from a different physical machine? Please, specify in detail the topology of your physical and virtual devices.
Also try to post the output of:
ssh -v <your_username>@<ssh_server>
Have fun!