This is another instance of Swift's sacrificing strict grammatical accuracy to terseness of expression. ' When I returned ' would, of course, be more correct.