Hi Min,

There are several algorithms that ship with Liferay that you can configure (that it already supports). These are the properties (with defaults and options for overriding)

    #
    # Set this to the appropriate encryption algorithm to be used for
    # company level encryption algorithms (except password encryption which is
    # defined via the property "passwords.encryption.algorithm").
    #
    # If you change this setting, you may need to also change the property
    # "company.encryption.key.size" to configure the desired encryption key
    # size.
    #
    # For more details about encryption keys, see the Java Cryptography
    # Extension documentation.
    #
    company.encryption.algorithm=AES
    #company.encryption.algorithm=Blowfish
    #company.encryption.algorithm=DES
    #company.encryption.algorithm=DESede

    #
    # Set this to define the size used for the company wide encryption key. If
    # you change this setting, you may need to also change the property
    # "company.encryption.algorithm" to ensure the algorithm supports your key
    # size. Most encryption algorithm implementations do not support key sizes
    # stronger than 128. Only those with unlimited strength cryptography support
    # stronger keys.
    #
    # For more details about supported key sizes for encryption keys, see
    # the Java Cryptography Extension documentation.
    #
    #company.encryption.key.size=56
    company.encryption.key.size=128
    #company.encryption.key.size=168
    #company.encryption.key.size=256
    #company.encryption.key.size=384
    #company.encryption.key.size=512

There is also an entire section with options specifically around passwords --

##
## Passwords
##

    #
    # Set the following encryption algorithm to designate the password
    # encryption algorithm to migrate from. This is only necessary if
    # upgrading from a release prior to 6.2 or if you wish to switch password
    # encryption algorithms when there are existing users with passwords in the
    # portal.
    #
    #passwords.encryption.algorithm.legacy=BCRYPT
    #passwords.encryption.algorithm.legacy=MD2
    #passwords.encryption.algorithm.legacy=MD5
    #passwords.encryption.algorithm.legacy=NONE
    #passwords.encryption.algorithm.legacy=SHA
    #passwords.encryption.algorithm.legacy=SHA-256
    #passwords.encryption.algorithm.legacy=SHA-384
    #passwords.encryption.algorithm.legacy=SSHA
    #passwords.encryption.algorithm.legacy=UFC-CRYPT

    #
    # Set the following encryption algorithm to encrypt passwords. The default
    # algorithm is PBKDF2WithHmacSHA1/160/128000, generating 160 bit hashes
    # using 128,000 rounds.
    #
    # It is possible to decrease the rounds value in case of slow production
    # hardware, but be aware that 128,000 is the recommended value for 2014
    # by OWASP.
    #
    # If set to NONE, passwords are stored in the database as plain text.
    #
    #passwords.encryption.algorithm=BCRYPT/10
    #passwords.encryption.algorithm=MD2
    #passwords.encryption.algorithm=MD5
    #passwords.encryption.algorithm=NONE
    passwords.encryption.algorithm=PBKDF2WithHmacSHA1/160/128000
    #passwords.encryption.algorithm=SHA
    #passwords.encryption.algorithm=SHA-256
    #passwords.encryption.algorithm=SHA-384
    #passwords.encryption.algorithm=SSHA
    #passwords.encryption.algorithm=UFC-CRYPT

    #
    # Digested passwords are encoded via base64 or hex encoding. The default is
    # base64.
    #
    passwords.digest.encoding=base64
    #passwords.digest.encoding=hex

    #
    # Input a class name that implements
    # com.liferay.portal.security.pwd.Toolkit. This class will be called to
    # generate and validate passwords.
    #
    passwords.toolkit=com.liferay.portal.security.pwd.PasswordPolicyToolkit
    #passwords.toolkit=com.liferay.portal.security.pwd.RegExpToolkit

    #
    # If you choose to use com.liferay.portal.security.pwd.PasswordPolicyToolkit
    # as your password toolkit, you can choose either static or dynamic password
    # generation. Static is set through the property
    # "passwords.passwordpolicytoolkit.static" and dynamic uses the class
    # com.liferay.util.PwdGenerator to generate the password. If you are using
    # LDAP password syntax checking, you will also have to use the static
    # generator so that you can guarantee that passwords obey its rules.
    #
    #passwords.passwordpolicytoolkit.generator=static
    passwords.passwordpolicytoolkit.generator=dynamic
    passwords.passwordpolicytoolkit.static=iheartliferay

    #
    # Set the character sets for password validation.
    #
    passwords.passwordpolicytoolkit.charset.lowercase=abcdefghjkmnpqrstuvwxyz
    passwords.passwordpolicytoolkit.charset.numbers=0123456789
    passwords.passwordpolicytoolkit.charset.symbols=_.!@$*=-?
    passwords.passwordpolicytoolkit.charset.uppercase=ABCDEFGHJKLMNPQRSTUVWXYZ

    #
    # If you choose to use com.liferay.portal.security.pwd.RegExpToolkit as
    # your password toolkit, set the regular expression pattern that will be
    # used to generate and validate passwords.
    #
    # Note that \ is replaced with \\ to work in Java.
    #
    # The first pattern ensures that passwords must have at least 4 valid
    # characters consisting of digits or letters.
    #
    # The second pattern ensures that passwords must have at least 8 valid
    # characters consisting of digits or letters.
    #
    passwords.regexptoolkit.pattern=(?=.{4})(?:[a-zA-Z0-9]*)
    #passwords.regexptoolkit.pattern=(?=.{8})(?:[a-zA-Z0-9]*)

    #
    # Set the length and key for generating passwords.
    #
    #passwords.regexptoolkit.charset=0123456789
    passwords.regexptoolkit.charset=0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz
    #passwords.regexptoolkit.length=4
    passwords.regexptoolkit.length=8

    #
    # Set the properties of the default password policy.
    #
    passwords.default.policy.allow.dictionary.words=true
    passwords.default.policy.changeable=true
    passwords.default.policy.change.required=true
    passwords.default.policy.check.syntax=false
    passwords.default.policy.expireable=false
    passwords.default.policy.grace.limit=0
    passwords.default.policy.history=false
    passwords.default.policy.history.count=6
    passwords.default.policy.lockout=false
    passwords.default.policy.lockout.duration=0
    passwords.default.policy.max.age=8640000
    passwords.default.policy.max.failure=3
    passwords.default.policy.min.age=0
    passwords.default.policy.min.alphanumeric=0
    passwords.default.policy.min.length=6
    passwords.default.policy.min.lowercase=0
    passwords.default.policy.min.numbers=1
    passwords.default.policy.min.symbols=0
    passwords.default.policy.min.uppercase=1
    passwords.default.policy.name=Default Password Policy
    passwords.default.policy.regex=(?=.{4})(?:[a-zA-Z0-9]*)
    passwords.default.policy.reset.failure.count=600
    passwords.default.policy.reset.ticket.max.age=86400
    passwords.default.policy.warning.time=86400

Are you trying to use an algorithm other than those?