When you use secure mode, VCS does not store any passwords as authentication is handled by the O/S, not VCS - this is why it is secure as storing passwords encypted in a text file (main.cf) is not that secure.
The admin user in main.cf will need to be in the format admin@domain, so ideally the domain should be a central domain like LDAP. You can use unixpwd authentication and then the domain is the name of the host.
This is how secure clusters worked in 5.0 and 5.1 - I have not used secure cluster in 6.x, but I know 6.x uses embedded authentication as oppose to a more central model in 5.x, but I don't know if anything else has changed.
If you are still having issues, please post the initial section of you main.cf showing cluster and system definitions and also post the user, domain, broker and domain type you are using in java GUI and also the output of the following if it works "vssat showallbrokerdomains" (this command may not work in 6.x)
Mike