You may want to try a thinner, lower viscosity silicone for the first layer. Then, follow that up with a thicker, higher viscosity layer. That may reduce the number of bubbles/voids that end up in the finished silicone mold and leave bumps in the plaster.