When your music arrived at the earpieces everything was on time, in the correct order and each note was correct in transients, amplitude, time and phase.
Crossovers are, however, determine how the loudspeaker will reproduce the original music signal.
Once the music has passed through the crossover (capacitors and resistors) it emerges with changes to the transients, amplitude time and phase. Every original key part of the musical structure has been altered or moved from its original position. As drivers are highly conductive their phase characteristic is not constant but it varies depending on the frequency, resulting in overlapping or cancelling frequencies, disjoint output signal, and drastically distorted music.
And as Impedance and Capacitance varies with frequencies, the lower it is, the higher distortion the capacitor generates, which in some cases can become audible in a complicated network.
Typically a 3-5 unit employs 2-4 tantalum capacitors in the circuit, together with a series of resistors to create low pass and high pass filters. Such combination when combined with the inductive nature of Balanced Armature will introduce a significant amount of distortion as compared to Crossoverless.