In a small mixing bowl, combine brown sugar and brandy. Stir in walnuts. Place brie on top of oven-proof serving platter and spoon walnut mixture over top, covering completely. Wrap platter in plastic and refrigerate for at least two hours.
Preheat oven to 400*F (205*C). Remove brie from refrigerator, unwrap and discard plastic. Bake for 10 to 15 minutes or until cheese begins to melt. Cool slightly and serve with crackers.