Here is a handy application that I use fairly regularly when working on Windows systems. PuTTY has now been packaged as a portable app but, in all fairness the non-portable version worked just fine from a USB drive. The crux of the conversation is that it will now smoothly integrate with the PortableApps.com suite. That and there is no information stored in the registry.
PuTTY is a lightweight telnet and SSH client for Windows, along with an xterm terminal emulator. It allows you to securely connect to your systems while on the go.