THE AFRICAN-AMERICAN TROOPS NOT ONLY BROUGHT themselves to Camp Nelson, they often brought their families as well. Families were brought for many reasons, including fear of retaliation from angry slave holders and anticipation of better opportunities or even emancipation for the entire family. Questions about the safety of the families of the new recruits were prominent at all recruitment centers, but were finally resolved with the events at Camp Nelson. The presence of these women and children in the camp posed difficult problems for army officials. No clear army policy existed for the treatment of refugees, and it was not even clear whether the families should be allowed to stay within the camp. In July 1864, Adjutant General Lorenzo Thomas finally ordered Camp commanders to eject the African-American dependents. This order was quickly rescinded, however, when irate reports reached Washington.

The situation was once again ambiguous, and each commander had to set his own policy. In the summer and fall of 1864, Brig. General Speed S. Fry, the commander of Camp Nelson, began harassing and expelling refugees from the camp and cooperating with slave owners to return their slaves. New refugees kept arriving, however, and old ones kept returning. Finally in November 1864, Fry decided to expel all the refugees and, to prevent their return, destroy their shanties. Hundreds of refugees died of exposure or disease in the cold November weather or soon after as a result of this action. Fry was severely criticized by the northern press, the U.S. Sanitary Commission and by the missionary to the refugees, Rev. John G. Fee. Fry's actions also enraged the African-American recruits and undermined the recruitment of African-Americans in Kentucky. Because of the complaints and reactions, Washington directed Fry to establish a camp for the refugees within Camp Nelson.

A direct result of Fry's actions at Camp Nelson and the uproar which followed was the passage into law, in February 1865, of the act which freed the wives and the children of the ex-slave enlistees. This act resulted in an increase in the enlistment of enslaved African-Americans in Kentucky and other border states. Assistant Quartermaster Captain Theron E. Hall was appointed superintendent for the refugees and immediately began building barracks to house them in the southwestern part of Camp Nelson. The barracks were eventually found to be inadequate and, with the encouragement of Rev. John G. Fee in the Spring of 1865, duplex cottages were constructed and designated the "Home for Colored Refugees." By June 1865, the refugee camp contained 97 cottages and numerous tents and shacks and provided housing for 3,060 people, primarily women and children. The refugee camp also included a school house, a hospital, a mess hall, a laundry, a lime kiln, teacher's quarters and offices. Missionaries from the American Missionary Association assisted the army in caring for the refugee families. They provided teachers for the school, ran church services, provided clothing and other supplies, and generally helped administer the camp.

The African-American soldiers also attended the school and church services at the refugee camp. Some African-American soldiers, most notably Gabriel Burdett from Garrard County, also helped teach in the school. The earliest missionary that served the refugee camp was the Rev. John G. Fee, who also had the longest term. Fee was a well known abolitionist and passionately believed in the equality of the races. Fee envisioned the refugee camp as a place to educate and train the freedmen to become independent, self-reliant members of an integrated American society. Rev. Fee felt a strong calling to help these people. As Fee stated:
"Here are thousands of noble men, made in the image of God, just emerging from the restraints of slavery into the liberties and responsibilities of free men, and of soldiers. I find them manifesting an almost universal desire to learn' and that they do make rapid progress...I feel that it is blessed to labor with such people...."
After the War ended, Fee split his time between Camp Nelson, where he founded Ariel College and Berea, Kentucky, where he founded and ran Berea College, one of the first, if not the first, integrated schools in the South. Fee also encouraged the educated African-American soldiers and family members from Camp Nelson to move to Berea and attend the college.

Other missionaries to the refugee camp were Abisha Scofield and Leonard Williams. All missionaries and paid teachers in the camp were European-American except for Belle Mitchell, who was an African-American teacher brought in by Fee. Ms. Mitchell only stayed a short while, however, and was dismissed when Fee was away in Berea.