Those are real bees featured in the episode. Jensen Ackles was the first to get stung which amused Jared Padalecki enormously. Ackles got his revenge shortly after when Padalecki sat down on a bee and was stung on his rear.
For this episode, all the actors who worked around bees were given costumes with cuffs sewed on the inside of their pants and sleeves to help prevent the insects from crawling into their clothing. It helped, but everyone still got stung.
Approximately 60,000 bees were used in the attic scene. Unfortunately, they were unseen on camera so CGI was still used in order to see the bees on screen.
Alongside Hook Man (2005), this is one of Eric Kripke's least favorite episodes of the first season.