I don't have direct experience of testing this but I believe that working with Support you previously determined that the traffic recorded was UDP and TCP which didn't look feasible to test at this level.
Recommended options are
1- GUI-Level testing using Silk Test
2- Use our framework (e.g. JAVA) to hand script calls to external Test Drivers written by the application developers
I'm interested in what makes you now think that it should work on the java RMI level as I can't find any supporting documentation?
If correct the following will provide you with details on how to configure - community.microfocus.com/.../8656.how-do-i-set-up-an-application-profile-to-record-my-java-application.aspx