What is an FM file?
An FM file is a spreadsheet file created with FileMaker or FileMaker Pro that is a cross-platform relational database application. It is relatively older file format and in order to load this in FileMaker Pro, it needs to be converted to the new .fmp12 file format. This new .fmp12 is the only file format that can be hosted on FileMaker Server and FileMaker Server Advanced. FM files can be opened with FileMaker Pro Advanced.
FM File Format - More Information
Not much information is available about the internal file format and structure of FM files. The National Archives have hosted the PRONOM about FM File Format for reference though.
How to Convert Older FileMaker Pro Files to .fmp12 File Format?
FileMaker Pro can directly convert FM to .fmp12 file format. A step-by-step guide for converting FM files to .fmp12 can be referred for this purpose.