By-products represent a major disposal problem for the food industry, but they are also promising sources of bioactive compounds. Olive pomace, one of the main by-products of olive oil production, is a potential low-cost, phenol-rich ingredient for the formulation of functional food. In this study, bakery products enriched with defatted olive pomace powder and their conventional counterparts were chemical characterized and in vitro digested. The bioaccessible fractions were supplemented to cultured human intestinal cells exposed to an inflammatory stimulus, and the anti-inflammatory effect and metabolome modification were evaluated. Although in all bakery products the enrichment with olive pomace significantly increased the total phenolic content, this increase was paralleled by an enhanced anti-inflammatory activity only in conventionally fermented bread. Therefore, while confirming olive oil by-products as functional ingredients for bakery food enrichment, our data highlight that changes in chemical composition cannot predict changes in functionality. Functionality should be first evaluated in biological in vitro systems, and then confirmed in human intervention studies.