Diacetyl is a member of the ketone family of chemicals & is known formally as butane-2,3-dione, with the chemical formula (CH3CO)2.
It is a flavour compound found in butter along with acetoin, both introduced artificially to margarine to give a buttery flavour. Diacetyl is also added to popcorn to give a buttery flavour.
Brewing yeast forms diacetyl during its life-cycle & imparts a buttery or butterscotch flavour which is mostly - but not exclusively - seen as an off-flavour. Chardonnay wine has diacetyl as a key flavour for the style. English style ales can have a higher diacetyl content as part of the overall flavour style, but generally it is kept to a minimum particularly in lagers & cleaner-tasting beers. Diacetyl can be tasted at concentrations less than 1ppm in beer.
The Krebs Cycle is how many cells produce energy. This cycle produces acetolactate in primary fermentation by brewers yeast, which can pass through the cell membrane into the fermenting wort. This acetolactate oxidises to diacetyl & is later reabsorbed into the yeast cell & metabolised into acetoin & 2,3-butanediol. These have a higher taste threshold, so impart less flavour to the beer than diacetyl.
Ales – fermented at around 20°C – reabsorb diacetyl much better than lagers – fermented at around 10°C – and these usually have a diacetyl rest at the end of primary fermentation. This involves raising the temperature to that of a fermenting ale for a couple of days to promote absorption & conversion of the diacetyl to the lower taste threshold molecules.
Many wild yeasts & souring bacteria produce diacetyl, but they are actively discouraged in regular beers.