Harvey was born Charles Felix Harvey, III in Kinston in 1920. After his father died when he was 13, he was raised by his grandmother, Lula Gill, the widow of a Civil War Veteran. He credited her with helping him to develop the self-discipline that was instrumental in his success. Harvey recalled in a recent interview that she would tell him in his teen years to: “make me love you more by honoring my instructions.”
Harvey attended Kinston Public Schools, Woodberry Forest School for his high school years and he graduated from The University of North Carolina – Chapel Hill in 1943 with a B. S. in Commerce. Shortly thereafter he entered active service with the United States Navy. He was commissioned an Ensign and served two years in PT Boat Combat Duty in the Southwest Pacific, serving under command of Douglas Fairbanks, Jr., the movie star. Later he was transferred to duty at Ocracoke, North Carolina.
Harvey returned to Kinston and took over the family business, making it one of the most successful in Kinston history.