I don't know of any good documentation out there, so I'll write a brief synopsis here.
TRX is a common firmware image format. Basically, it's a small header that defines the offsets of up to 3 'partitions' of the firmware image (usually loader, vmlinuz, OS), along with a signature, checksum, version, and length of the image.
struct trx_header {
uint32_t magic; /* "HDR0" */
uint32_t len; /* Length of file including header */
uint32_t crc32; /* 32-bit CRC from flag_version to end of file */
uint32_t flag_version; /* 0:15 flags, 16:31 version */
uint32_t offsets[3]; /* Offsets of partitions from start of header */
};
TRX files can be created with the TRX or ASUSTRX tools you'll find in any firmare source for this platform.
Sometimes, vendors who use the TRX format also pre-pend their own header to this and require that header to be present. An example is the addpattern tool for the WRT54G(S) and other Linksys devices.
So, that's what a TRX file is. Whether or not the original firmware supports this format depends on the particular device and firmware. OpenWrt should support images that only have a TRX header, and also those that have the extended addpattern type header.