An RBXM file is a 3D Model file format used in Roblox Studio IDE for storing 3D objects of online multiplayer Roblox game. It contains 3D objects created for the game using the IDE. Examples of 3D objects in an RBXM file include shirts, pants, hat, buildings, vehicles, animals, aeroplanes, cars, or plant. RBMX files can be converted to the OBJ 3D file format using the Roblox IDE. Roblox IDE can be used to open RBMX files.
Another popular Roblox file format is RBXL which is a location file format and contains location defined in 3D word inside the Roblox game.
RBMX File Format - More Information
RBXM files are created and saved in binary file format and their internal file format specifications are not available publicly. This is different from the RBXMX file format that is based on the XML file format. There are plenty of tools/repositories available on Github that can be used to process Roblox RBXM files.