When your family outgrows a home, what becomes the hardest thing to leave behind?

My husband & I have been talking about moving because we're running out of space now that our little one is growing. Surprisingly, I'm not worried about the house but rather our neighbors, nearby parks, and familiar routines. I wonder what's hardest for you to leaving behind?