I'm not sure if it's exactly the same on Macs, but with the Windows version of the Java GUI I got load balancing to work just fine. In the settings file I entered a line like this:
<SID> : /M/<message server>/S/3600
Then in the SAP GUI for Java I created a new connection. In the System field my SID showed up in the drop-down, and selecting it auto-filled the Message Server field with the string from above. In the Group/Server field I saw the available load balancing groups as well as the individual application servers for that system, and selected the appropriate one.
Once I saved that, it created a new line in my connections file like so:
<SID> [<group>]:conn=/R/<SID>/G/<group>&systemName=<SID>
It works beautifully.