NIS itself stores passwords and does not have limitations on the password implementation and on the used ciphers. Password authentification happens at login time on a client, at the imap server when connecting (if the imap server uses the NIS database) etc - and on the NIS server when changing the password (more below).
All systems that do authentication, do the password check locally and thus all systems, need to implement the same ciphers for encryption. The common lowest denominator is DES which only supports 8 characters. But if you know that all systems support e.g. SHA-512, you can use that one.
To change the cipher used for encryption on an openSUSE or SUSE Linux Enterprise 11 system, there are two different ways, depending on whether you use pam_unix2 or pam_unix in /etc/pam.d/common-password. pam_unix will be the default in new openSUSE 13.1 installations, pam_unix2 has been setup so far by default on openSUSE and SUSE Linux Enterprise. If you use pam_unix2, edit /etc/default/passwd and change CRYPT_YP to the desired cipher. If you use pam_unix, edit /etc/login.defs and edit ENCRYPT_METHOD - and then run "passwd".
Regarding compatibility of systems, here's some uncomplete information on ciphers available besides DES that I quickly gathered:
- SLES 10: blowfish
- SLES 11: md5, sha-256, sha-512, blowfish
- openSUSE 13.1: md5, sha-256, sha-512, blowfish
- Solaris 11: sha-256, md5
- RHEL 6: sha-256, sha-512, md5
Testing this, I stumpled upon one problem: When changing the NIS password, the old password is verified locally and then the old password is send unencrypted together with the encrypted new password to the NIS server. The NIS server then tests that the old password is correct (it cannot trust your client) and saves the encrypted new password in its database. When you now specify a cipher for encryption that is not known to the server, it will happily save the new encyrpted password - but once you change the next time, it will fail since it does not understand the new cipher and thus cannot check that the old password is correct. So, time to ask your admins to reset your NIS password...
Thanks for my colleague - and Linux NIS developer - Thorsten Kukuk for shedding some lights on these questions.