A .vgm is neither a score nor a recording. It is every byte a game wrote to its sound chips, in order, with the time between writes, so that a player holding the same chips (emulated) plays it back exactly. Three parts: a header of chip clocks, where a non-zero clock is the only thing that says a chip is present; a command stream, one opcode per write or wait; and a GD3 tag at the end, in UTF-16, saying what the tune is. The format has grown a chip at a time since 2001 and the version byte says how much of the header exists. A .vgz is the same file inside gzip and nothing more.
Every offset in the header is relative to its own field: the end-of-file value is added to 0x04, the GD3 offset to 0x14, the loop point to 0x1C, the stream offset to 0x34. A reader that adds them to zero lands 4 to 52 bytes early, every time.
The version decides what the header holds. 1.00 has the two clocks drawn above and nothing else; 1.10 adds YM2612 and YM2151 at 0x2C; 1.50 adds the stream offset at 0x34 (before it, the stream begins at 0x40); 1.51 adds a page of chip clocks from 0x38 to 0x7F; 1.61 another from 0x80 to 0xB7; 1.71 more to 0xE3. A field the file's version predates is not a field, whatever bytes are there. From 1.51 the top bit of a clock means two of that chip.
One opcode, then its operands. A chip write names the chip by its opcode and
carries the register and value; a wait carries a sample count at 44,100 Hz. The two most
common waits have their own one-byte opcodes, 0x62 for one NTSC frame (735
samples) and 0x63 for one PAL frame (882), and 0x70-0x7F wait
1 to 16 samples in the low nibble.
Reserved ranges have lengths. The spec fixes the operand count of
every opcode it has not yet assigned (0x30-0x3F one, 0x40-0x4E two, 0xC0-0xDF three,
0xE0-0xFF four) so that an old reader can skip a command a newer file uses. A stream can
therefore always be walked to its 0x66, and in a well-formed file that lands
exactly on the GD3 tag.
Chips with sample memory (the YM2612's DAC stream, the RF5C68's RAM, an OKI's ROM) get it from data blocks in the stream, each typed by a byte: 0x00-0x3F are uncompressed streams, 0x40-0x7E compressed, 0x80-0xBF ROM images, 0xC0-0xDF RAM writes. A block is the one part of a VGM that is audio rather than instructions, and the one part a reader can lift out.
Four bytes of magic, a version, a byte length, then eleven strings each ended by a 16-bit NUL: title, title in Japanese, game, game in Japanese, system, system in Japanese, author, author in Japanese, release date, who made the file, notes. An empty string is a lone NUL. The tag is the last thing in the file.
Four bytes at zero, with a trailing space. A .vgz is a gzip stream whose first inflated bytes are the same four; the extension says nothing and a reader confirms what is inside.