A stunning AGO retrospective demonstrates why and how a collective trio from Toronto turned the art world on its stuffy head