Preheat the oven to 350˚F and butter the bottom and sides of 9-inch springform pan. Cut a piece of parchment paper to fit the bottom, press it on the bottom, and butter it as well. Cut a long length of parchment paper and press it around the sides of the pan. This is a moist cake because of all the fruit, and it can be difficult to release from the pan. The parchment paper prevents this problem.
Cut the nectarines into bite sized pieces. Toss with nutmeg, allspice and 2 tablespoons of sugar.
Using a standing mixer fitted with the paddle attachment, cream together the butter and remaining sugar until fluffy, about 5 - 6 minutes. Mix in the egg, buttermilk and vanilla extract. In a separate medium-sized bowl, combine the flour, baking powder, baking soda and salt. Whisk it several times to evenly disperse the chemical leaveners throughout the flour. Pro-Tip: Weigh the flour for the best, most consistent result. I find that even carefully measured flour is more flour than weighed. Adding too much flour results in a denser cake.
Shift the speed of the mixer to low and add the dry ingredients to the butter mixture. Mix only until the flour is mostly incorporated, and finish with a rubber spatula. Over-mixing flour into a batter causes a tougher cake. The batter will be very thick and somewhat lumpy.
Scrape the batter into the prepared pan using a rubber spatula. Use the spatula to evenly spread it across the bottom of the pan. Spill the nectarines and blueberries onto the top of the batter, including any juices, and gently press in. Don't worry if the fruit is on top of each other in places. As the cake bakes, the batter will rise up and around it, and the nectarine chunks will settle across the surface and into the cake. Sprinkle turbinado sugar over the top, if using.
Bake for 10 minutes, then reduce the oven to 325˚F. Bake for an additional 55 - 60 minutes, or until a toothpick in the center comes out clean. The center will be the last part to cook, so only check around that area.
Place the pan on a cooling rack, and cool for 15 minutes before removing the side of the springform pan. Gently peel off the parchment paper from the sides. This is a tender cake, so you might not be able to peel off the bottom parchment paper. Just slide the cake from the base of the springform pan.
Serve slightly warm or at room temperature with a scoop of vanilla ice cream.