How does one link receivables credit memo transaction type to a Order Management transaction type

                 This is the intended functionality. The credit memo transactions will not be available in OM transaction type form. OM transaction type form has invoice which will refer the credit memo type defined in that transaction type.
                 AR creates invoice / credit memo based on the AR transaction type that is attached to the order line. Confirm that the receivables transaction type defined for credit memo is attached to the following:

1) OM Line Type. The navigation path to find this is:
Order Management Super User responsibility
Setup->Transaction Types->define
Query the 'Transaction type'. click on Finance tab, note down the Receivables transaction type (this we call as invoice transaction type)

2) OM Order Type. Check if there is a credit memo transaction type setup for this invoice transaction type. The navigation path to check transaction type setup is:

Receivables responsibility
Setup->transactions->transaction types
Query the '(Invoice) Transaction type' defined in step 1
Check 'Credit Memo Type' is defined for this (Invoice) Transaction type.

If there is no receivables transaction type attached at OM transaction type level then check the profile option
OM: Credit memo Transaction Type.

Explanation how this functionality works:
-----------------------------------------

When an RMA is created or copied from the standard order then invoice id (type) is referred in the RMA from the original sales order. When Auto invoicing is done then application checks for the invoice number and then creates a credit memo against the Invoice. So the Credit Memo type attached to that invoice type is taken precedence over the receivable transaction type selected in the transaction type (RMA) So that it can be referred from either side.

When creating the RMA without referring the standard order then while running the auto invoice program system could not find the invoice id (type) against which it can create a Credit Memo. While creating the Credit Memo since it could not find the referred invoice type, It checks the default receivable transaction type and credit memo type attached to it.

This can be verified with from the following example

When creating the RMA by referring the standard sales order then the RMA will check the Receivable transaction type attached to the standard sales order type and corresponding Credit Memo type will be selected in AR because this Credit Memo is attached to the invoice type referred in the field Receivable Transaction type of the OM Transaction type Form.
When creating the RMA without referring the standard sales order then the Credit Memo type is defaulted from the Receivable Transaction type attached to OM Transaction type (RMA). Because it cannot not find any value to refer as the invoice type. It checks the default value and selects the Credit Memo attached to it.