Why can't I set the ItemTemplate?

I think the problem is that you do not understand very well the difference between TabControl.ItemTemplate and TabItem.HeaderTemplate.
Judging by the template you are trying to add a button to close the tab in the header of the header.
If so, then for this purpose use the following code:

<grid>
<grid.resources>
 <datatemplate x:key="ClosableTabItemHeaderTemplate">
<datatemplate.resources>
 <booleantovisibilityconverter x:key="B2V">
</booleantovisibilityconverter></datatemplate.resources>
 <grid background="Transparent" horizontalalignment="Stretch" minwidth="100" tooltip="{Binding}">
<grid.columndefinitions>
<columndefinition>
<columndefinition>
</columndefinition></columndefinition></grid.columndefinitions>

 <contentpresenter content="{Binding}" grid.column="0" horizontalalignment="Center">
 </contentpresenter></grid></datatemplate></grid.resources></grid>

<datatemplate.triggers>
 <trigger property="IsMouseOver" value="True">
 <setter targetname="CLOSE" property="Visibility" value="Visible">
</setter></trigger>
</datatemplate.triggers>


 <style targettype="TabItem">
 <Setter Property="HeaderTemplate" Value="{StaticResource ClosableTabItemHeaderTemplate}" /> 
</style>



<tabcontrol>
<tabcontrol.items>
 <tabitem header="This is test tab item with longer name." width="100">
 <tabitem header="Short name" width="100">
</tabitem></tabitem></tabcontrol.items>
</tabcontrol>