In a high-speed blender, blend the ice, cold brew, milk, 2 tablespoon caramel sauce, and xanthan gum until smooth and creamy. Set aside.
Press the whipped cream into the bottom of a tall glass with a spoon. Pour in the remaining 2 tablespoon caramel sauce and add a layer of 1 tablespoon toffee bits.
Pour in the caramel frappe, then top with more whipped cream, caramel sauce, and/or toffee bits if desired. Drink up!