The house, though, was falling into disrepair. Donna Kucholtz, who resided in Ennis during the time Miss Rafael was in the nursing home, recalls the house well. She said it was slowly deteriorating, as houses do when they are long neglected. Dust was all over the furniture and cobwebs hung on all  the walls and in the doorways, giving the appearance of a macabre movie set of a deserted house. Everything has been left exactly as it was when Julia left. Even her breakfast dishes were still on the table.

By 1983 Julia was about all out of cash and decided to sell the house. Finally, the place was put up for sale, and some of her siblings and their offspring showed it to prospective buyers. At this time much of the furniture was sold, and some had apparently been stolen. It was evident that long neglect had taken its toll on the once proud mansion. Docia Schultz Williams wrote the following in her book Best Tales of Texas Ghosts:

Dana Cody was an Ennis girl who had gone away to college and moved to Austin, where she was pursuing a career as a buyer in a department store. She came home for a visit, and she and her mother decided to go see the old Rafael House, which was up for sale. The two women just went out of curiosity, but Danna fell in love with the place immediately. She could see beyond the chipped paint and fading carpets to what could be done. She thought the place would make the perfect bed and breakfast inn. Finally, after researching the house she decided that she was up to doing all the work to make the aging house a Bed and Breakfast, and decided to buy the house. She spent months scrubbing, sorting, cleaning, and restoring. Contractors had to be called in to we do the electric wiring and plumbing. Lots of time and money had to be spent in reviving the tired old mansion. There were books, linens, dishes, and boxes and boxes of receipts and all the other things Julia never threw away. All of this had to be sorted out. Some of the Rafael furniture was still in the house. Danna was able to track down other pieces that had been sold, and was able to buy them back. The pieces were all turn of the century, and made in America, which was considered prestigious at the time. Most of the bedroom furnishings, several chairs, a china cabinet, and a buffet all found their way back into the house. Danna used impeccable taste in bringing life back into the old building, and numerous people in Ennis got behind her efforts to restore the house. Some of them even gave her family heirloom furniture, dishes, and linens, to help lend the proper period look to the place.

Soon after Danna purchased the house, Julia Rafael (1989) passed away. Some members of the Rafael family tried to tell Danna the house was haunted, but Danna didn't believe in ghosts, and if she had, she really didn't care. She wanted the house badly enough that she said if she had to put up with a few ghosts it really wouldn't matter to her.

The Dallas Morning News has written a few articles about the ghosts at the Rafael House, the first was in 1993 during the Halloween season. In that article readers learned of the garden that Raymond Rafael once loved long ago. Flowers bloomed in places where no flowers had been planted even after  the landscaping around the house had been extensively renovated. Other haunting events related to the house, are the same as  many haunted houses in Texas, moving objects, voices, unexplainable sounds, and smells that shouldn't be there.