In a heavy-bottomed saucepan, add in about 4 cups of vegetable oil. You want ¾ inch of oil in the bottom of the pan. Attach a candy thermometer and let it heat to up 350 degrees. I found that this can take quite a while so I like to do this first.
In a small bowl, add in ¼ cup of sugar and 1 teaspoon of nutmeg. Set aside.
In a large mixing bowl, add in the bananas, buttermilk, remaining ¼ cup sugar, and 3 teaspoons of grated nutmeg. Stir together to combine.
Add in the flour, baking powder and salt. Carefully fold together with a spatula to avoid breaking up the bananas more. Set aside.
In a separate large mixing bowl, add in the egg whites. Using a mixer, beat on high until stiff peaks form. It will take about 2 minutes.
Fold the egg whites into the mixture in 2 parts. The first part will loosen up the batter and lighten it. The second addition will give it the airiness.
Line a large plate with paper towels to drain the fritters over frying. Set aside.
Using a spoon or cookie scoop, drop rounded tablespoons into the hot oil. I added three at a time because I didn't want to crowd the fritters or drop the heat on the oil and make my fritters oily. Let the fritters puff up and become golden brown and then flip. It will take about 2 minutes.
Let fry on the other side until it is also golden brown and puffy. That will take another 2 minutes. Use a slotted spoon, remove the fritters and place on the dry paper towel.
Let drain about 30 seconds and roll into the nutmeg and sugar you made earlier. Roll around until coated and serve up. Eat 'em while they're hot!