Preheat oven to 350°F. Lightly butter the sides of a 9-inch round cake pan. Line the bottom with a circle of parchment paper and butter the parchment too - this ensures your fruit releases cleanly.

Melt 4 tablespoons butter in a small saucepan over medium heat. Stir in brown sugar and cook until bubbling and slightly darkened, about 2 minutes. Pour into the prepared pan and tilt to spread evenly. Arrange peach and plum slices in concentric circles, overlapping slightly. Set aside.

In a medium bowl, whisk together flour, baking powder, and salt. This prevents any salty or leavening surprises in your finished cake.

In a large bowl with a hand mixer (or stand mixer with paddle attachment), beat 8 tablespoons butter and granulated sugar on medium-high speed until pale and fluffy, about 3 minutes. Scrape down the bowl once or twice - unmixed butter streaks will sink in the oven.

Beat in eggs one at a time, mixing well after each addition. Add almond extract and vanilla extract and beat until combined. The batter will look slightly curdled - that is normal and fixes itself in the next step.

With mixer on low speed, add flour mixture in three additions alternating with milk in two additions, beginning and ending with flour. Mix just until no dry streaks remain - overmixing makes the cake tough.

Spoon batter over the fruit in the pan and smooth the top with an offset spatula. Bake until the top is golden brown and a toothpick inserted in the center comes out clean, 40 to 45 minutes. The cake should spring back when lightly pressed.

Let cake cool in pan on a wire rack for exactly 10 minutes - any longer and the fruit may stick; any sooner and it falls apart. Run a thin knife around the edge, place a serving plate over the pan, and flip confidently. Let stand 1 minute, then lift off the pan. Peel away parchment if it stuck. Serve warm or at room temperature.