1. Preheat the oven to 180 ̊C, gas mark 4. For the sponge cake, grease and line a 15cm square cake tin with baking parchment. Beat the butter and sugar in a large bowl, until light and fluffy. Beat in the flour, eggs, milk and a pinch of salt, until well combined, then scrape into the tin. Bake for 25-30 minutes, until risen and golden. Leave to cool for 5 minutes in the tin, then transfer to a wire rack to cool completely.
2. For the roast plums, put the fruit in a deep, parchment-lined baking tray. Sprinkle over the stem ginger, syrup, sugar and 3 tbsp cold water. Cover with foil and bake for 25-30 minutes, until the plums are just soft but still holding their shape. Set aside to cool and reserve any syrupy juices.
3. For the custard, put the egg yolks, sugar and corn flour in a large bowl and mix well. Heat the milk, cream and vanilla in a large saucepan until just before boiling, then take off the heat and slowly pour it over the sugar and egg mixture, whisking continuously. Return to the pan and whisk over a medium-low heat for 2-3 minutes, until thickened. Whisk out any lumps that appear. Pour into a large bowl and cover the surface with clingfilm. Leave to cool completely, then chill until needed.
4. To assemble, cut the cake into fingers or squares and arrange over the base of a trifle dish. Drizzle over the sherry, then arrange the plums around the side of the dish. Drizzle with the plum syrup, then spoon over the custard. Finally, lightly whip the cream with the sugar and sherry, then spoon over the top. Scatter with the crystallised ginger to serve.