Place sliced kumquats in a heavy saucepan and cover with water. Bring to a boil over high heat and drain. Transfer to a bowl of ice water; drain. Bring 2/3 cup of sugar and orange juice to a boil in a saucepan, lower heat to a simmer and add fruit. Simmer for 30 minutes, or until liquid is evaporated and fruit is glazed. Stir occasionally to make sure fruit is not sticking to the bottom of pan. Remove kumquats and reserve on a plate lined with parchment paper.
Preheat oven to 375°F.
Combine graham crackers, almonds, melted butter, 3 tablespoons sugar, cinnamon and nutmeg in a mixing bowl and press firmly into the bottom and sides of a 9-inch pie pan.
In another mixing bowl, whisk egg yolks until thick and pale yellow, about 5 minutes. Pour in condensed milk and candied kumquats. Slowly add lime juice, whisking constantly. Pour filling into prepared crust and bake for 10 to 12 minutes. Remove from oven and cool for 1 hour. Chill in refrigerator until ready to serve.