We know in Mubahala the holy Prophet got with him only Imam Ali, Lady Fatima, Imam Hassan and Hussain. The end part of this narration says the Prophet gathered these personalities and said these are my family (Arabic word AHLI is used which means my Ahlalbait)

For more information please read our article Who are the central figures of Islam after Prophet Mohammad (pbuh).