Joyce Loretta Richardson was raised in Washington, D.C. She attended a competitive boarding school on a scholarship from the National Scholarship Fund for Negro Students, before joining the Pembroke Class of 1963. There, she studied American Civilization and participated in the Newman Club, Baseball team, and Catholic Interracial Council. After graduation, she completed graduate studies in social work in order to supplement her career in the field.