The reason it was red only is that red LEDs were cheap, blue and green weren't. At the time. They fell way down in price just a few years later.

If they waited, it could have been a full color optional peripheral for the N64. That was its other major drawback, not enough hardware power to do satisfying 3D. Red Alarm was about the best it got.

Pair a full color Virtual Boy with the N64 and suddenly you've got something worthwhile. Yes, you'd have to reduce poly count to make N64 games run smoothly when rendering 2 views. But most N64 games were already designed to do that for splitscreen multiplayer.

Also, the N64's 320*240 resolution was pretty close to the VB's 384*224 resolution (technically 1*224 since it was a vertical row of precision timed LEDs plus an oscillating mirror that created the horizontal resolution)

If they were planned to work together from the start, with that resolution, a framerate of 60fps is achieveable providing the game is designed around those constraints. Although if you cannot turn your head, a lower framerate isn't as noticeable.