Search for charged Higgs bosons in top quark decays

We present a search for charged Higgs bosons in top quark decays. We analyze the \eplus, \muplus, $ee$, $e\mu$, $\mu\mu$, \etau and \mutau final states from top quark pair production events, using data from about 1${\text{fb}}^{-1}$ of integrated luminosity recorded by the \dzero experiment at the Fermilab Tevatron Collider. We consider different scenarios of possible charged Higgs boson decays, one where the charged Higgs boson decays purely hadronically into a charm and a strange quark, another where it decays into a $\tau$ lepton and a $\tau$ neutrino and a third one where both decays appear. We extract limits on the branching ratio $B(t\to H^+ b)$ for all these models. We use two methods, one where the $t\bar{t}$ production cross section is fixed, and one where the cross section is fitted simultaneously with $B(t\to H^+b)$. Based on the extracted limits, we exclude regions in the charged Higgs boson mass and $\tan \beta$ parameter space for different scenarios of the minimal supersymmetric standard model.

We present a search for charged Higgs bosons in top quark decays. We analyze the e+jets, µ+jets, ee, eµ, µµ, τ e and τ µ final states from top quark pair production events, using data from about 1 fb −1 of integrated luminosity recorded by the D0 experiment at the Fermilab Tevatron Collider. We consider different scenarios of possible charged Higgs boson decays, one where the charged Higgs boson decays purely hadronically into a charm and a strange quark, another where it decays into a τ lepton and a τ neutrino and a third one where both decays appear. We extract limits on the branching ratio B(t → H + b) for all these models. We use two methods, one where the tt production cross section is fixed, and one where the cross section is fitted simultaneously with B(t → H + b). Based on the extracted limits, we exclude regions in the charged Higgs boson mass and tan β parameter space for different scenarios of the minimal supersymmetric standard model.

I. INTRODUCTION
In many extensions of the standard model (SM), including supersymmetry (SUSY) and grand unified theories, the existence of an additional Higgs doublet is required. Such models predict multiple physical Higgs particles, including three neutral and two charged Higgs bosons (H ± ) [1]. If the charged Higgs boson is sufficiently light, it can appear in top quark decays t → H + b [2].
Within the SM, the top quark decay into a W boson and a b quark occurs with almost 100% probability. The tt final state signatures are fully determined by the W boson decay modes. Measurements of top quark pair production cross sections σ tt in various channels [3] are potentially sensitive to the decay of top quarks to charged Higgs bosons. The presence of a light charged Higgs boson would result in a different distribution of tt events between different final states than expected in the SM.
In this Letter we compare the number of predicted and observed events in various tt final states and derive 95% confidence level (CL) limits on the production of charged Higgs bosons from top quark decays. The analysis is based on data collected with the D0 detector between August 2002 and February 2006 at the Fermilab Tevatron pp Collider at √ s = 1.96 TeV. The analyzed datasets correspond to an integrated luminosity of about 1 fb −1 .
The decay modes of the charged Higgs boson depend on the ratio of the vacuum expectation values of the two Higgs doublets, tan β. For small values of tan β it is dominated by the decay to quarks, while for larger values of tan β it is dominated by the decay to a τ lepton and a neutrino. We consider three models for the charged Higgs boson decay: a purely leptophobic model, where the charged Higgs boson decays into a charm and a strange quark, a purely tauonic model, where the charged Higgs boson decays exclusively into a τ lepton and a neutrino, and a model where both decays can occur. In all models we fix the tt cross section to the theoretical value within the SM and extract B(t → H + b). In the case of the tauonic model, in addition we extract σ tt and B(t → H + b) simultaneously, thus yielding a limit without assuming a particular value of the tt cross section.
A scenario in which the charged Higgs boson decays exclusively into quarks can be realized, for instance, in a general multi-Higgs-doublet model (MHDM) [4]. It has been demonstrated that such leptophobic charged Higgs bosons with a mass of about 80 GeV could lead to noticeable effects at the Tevatron if tan β ≤ 3.5 [5]. Moreover, large radiative corrections from SUSY-breaking effects can lead to a suppression of H + → τ + ν compared to H + → cs [6]. In that case, for small tan β, hadronic charged Higgs decays can become large in both the two-Higgs-doublet (2HDM) [5] and the minimal supersymmetric standard model (MSSM).
For values of tan β ≥ 20 we consider different models leading to different branching ratios. Values of B(H + → cs) close to one are predicted in specific CPviolating benchmark scenarios (CPX) with large threshold corrections [7]. For other models, the tauonic decays of the charged Higgs boson dominate at high tan β, for example, in the m max h benchmark scenario [8] where B(H + → τ + ν) can be close to one.

II. EVENT SELECTION AND ANALYSIS METHOD
This search for charged Higgs bosons is based on the following tt final states: the dilepton (ℓℓ) channel where both charged bosons (W + or H + ) decay into a light charged lepton (ℓ = e or µ) either directly or through the leptonic decay of a τ , the τ +lepton (τ ℓ) channel where one charged boson decays to a light charged lepton and the other one to a τ -lepton decaying hadronically, and the lepton plus jets (ℓ+jets) channel where one charged boson decays to a light charged lepton and the other decays into hadrons. We select events to create 14 subchannels: (i ) ee (µµ) subchannel with two isolated high transverse momentum (p T ) electrons (muons) and at least two high p T jets; (ii ) eµ subchannels with one isolated high p T electron and one muon and exactly one or at least two jets; (iii ) τ e (τ µ ) subchannel with one high p T hadronically decaying τ , one electron (muon) and at least two high p T jets one of which is identified as a b jet; (iv ) ℓ+jets subchannels with one isolated high p T electron (muon), exactly three or at least four high p T jets, further split into subsamples with one or at least two b-tagged jets. Details of the event selection and object reconstruction in the dilepton and τ ℓ channels can be found in Ref. [9]; a more detailed description of the ℓ+jets channel and the combination are given in Ref. [3]. All event samples are constructed to be mutually exclusive.
In the ℓ+jets channel the main background consists of W +jets production, with smaller contributions from multijet, single top quark and diboson production. The background contribution in the τ ℓ channel is dominated by multijet events, while the most important background in the ℓℓ channel emerges from Z+jets production. The sample composition of all 14 subchannels, assuming B(t → W + b) = 1 (hence B(t → H + b)=0), is given in Ref. [3].
The simulation of the W +jets and Z+jets backgrounds as well as the tt signal with no charged Higgs boson decay is performed using alpgen [10] for the matrix element calculation, followed by pythia [11] for parton showering and hadronization. Diboson samples are generated using pythia, while single top quark events are simulated using the singletop [12] generator. The generated events are processed through a geant-based [13] simulation of the D0 detector and the same reconstruction programs used for the data.
We simulate the signal containing charged Higgs bosons with the pythia Monte Carlo event generator [11], separately for the decays tt → W + bH −b (and its charge conjugate) and tt → H + bH −b . The total signal selection efficiency is calculated as a function of B ≡ B(t → H + b) as given by: yielding the number of tt events as a function of B. The efficiencies ǫ tt→W + bH −b and ǫ tt→H + bH −b are evaluated for the assumed H + decay modes. Figure 1 shows   tries with one and two b-tags represent the sum of four ℓ+jets subchannels each (with different light lepton flavor and = 3 and ≥ 4 jets). The dilepton contribution corresponds to the sum of the ee, µµ and two eµ subchannels, and the τ +lepton one shows the sum of the τ e and τ µ subchannels. For a non-zero branching ratio B(t → H + b → csb) the number of events decreases in the ℓ+jets, ℓℓ and τ ℓ final states. In case of a nonzero branching ratio B(t → H + b → τ + νb) the number of predicted events increases in the τ ℓ channel while it decreases in all other channels. The latter are often called disappearance channels.
The extraction of B(t → H + b) is done by calculating the predicted number of events in 14 search subchannels for various charged Higgs boson masses and branching ratios, and performing a maximum likelihood fit to the number of observed events in data. We constrain the multijet background determined from control samples in the ℓ+jets and τ ℓ channels by including Poisson terms in the likelihood function. We account for systematic uncertainties in the fit by modeling each independent source of systematic uncertainty as a Gaussian probability density function G with zero mean and width corresponding to one standard deviation (SD) of the parameter representing the systematic uncertainty. Correlations of systematic uncertainties between channels are naturally taken into account by using the same parameter for the same source of systematic uncertainty. The parameter for each systematic uncertainty is allowed to float during the likelihood fit. We maximize the likelihood function with P(n, m) representing the Poisson probability to observe n events when m events are expected. The product runs over the subsamples i, and multijet background samples j. K is the total number of independent sources of systematic uncertainty, with ν k being the corresponding nuisance parameter. The predicted number of events in each channel is the sum of the predicted background and the expected tt events, which depends on B(t → H + b).
During the fit, the tt cross section is set to 7.48 +0.55 −0.72 pb, corresponding to an approximation to the next-to-nextto-leading-order (NNLO) QCD cross section that includes all next-to-next-to-leading logarithms (NNLL) relevant in NNLO QCD [14] at the world average top quark mass of 173.1 GeV [15]. The uncertainty on the theoretical cross section includes the uncertainty on the world average top quark mass.
Since we find no evidence for a charged Higgs boson, we extract upper limits on B(t → H + b), assuming that B(H + → cs) = 1, or B(H + → τ + ν) = 1, or a mixture of both. The limit setting procedure follows the likelihood ratio ordering principle of Feldman and Cousins [16]. The determination of the limits requires the generation of pseudo-datasets. We generate ensembles of 10,000 pseudo-datasets with B(t → H + b) varied between zero and one in steps of 0.05, fully taking into account the systematic uncertainties and their correlations. Table I shows an example of the uncertainties on B(t → H + b) for M H + = 80 GeV in the tauonic and leptophobic charged Higgs boson models. We consider systematic uncertainties originating from electron, muon, τ and jet identification, τ and jet energy calibration, b-jet identification, limited statistics of data or Monte Carlo samples, modeling of triggers, signal and background, and integrated luminosity. To evaluate the signal modeling uncertainty we replace the SM tt sample generated with alpgen by the one generated with pythia and take the difference in acceptance as systematic uncertainty. For both the tauonic and leptophobic model, the two main sources of systematic uncertainty on B(t → H + b) are the uncertainty on the luminosity of 6.1% and the tt cross section, followed by the non-negligible uncertainties on signal modeling, b jet identification and jet energy scale. The former two are approximately of the same size as the statistical uncertainty. Since in the tauonic model we consider both appearance and disappearance channels, some uncertainties affecting the signal and background normalization cancel. Therefore, uncertainties on signal modeling, the tt cross section, lepton identification and luminosity are reduced in the tauonic model compared to the leptophobic model.    Figure 2 shows the expected and observed upper limits on B(t → H + b) assuming B(H + → cs) = 1 or B(H + → τ + ν) = 1 as a function of M H + along with the one standard deviation band around the expected limit. Table II lists   The CDF Collaboration reported a search for charged Higgs bosons using different tt decay channels with a data set of about 200 pb −1 [17], resulting in B(t → H + b) < 0.4 within the tauonic model. Recently, D0 reported limits on B(t → H + b) for the tauonic and leptophobic models extracted from cross section ratios [3] and for the tauonic model based on a measurement of the tt cross section in ℓ+jets channel using topological event information [18]. Exploring the full set of channels as presented here improves the limits derived in the cross section ratio method for the leptophobic and for the tauonic model in the high M H + region.
We also extract upper limits on B(t → H + b) mixing the tauonic and leptophobic models under the assumption B(H + → τ + ν) + B(H + → cs) = 1. We repeat the extraction of upper limits on B(t → H + b) in the range of 0 ≤ B(H + → τ + ν) ≤ 1 in steps of 0.1. For each assumed M H + we parametrize the expected and observed limits dependent on the mixture between tauonic and leptophobic decays. Figure 3 shows upper limits on B(t → H + b) as a function of B(H + → cs). As expected, the upper limit decreases with increasing tauonic decay fraction.

IV. SIMULTANEOUS EXTRACTION OF
The search for charged Higgs bosons in top quark decays is based on the distribution of tt events between the various final states. Naturally, it is also sensitive to the total number of tt events. This results in a large systematic uncertainty due to the theoretical uncertainty in the tt cross section calculations. If σ tt and B(t → H + b) are measured simultaneously the limit becomes independent of the assumed theoretical tt cross section. Furthermore, the luminosity uncertainty and other systematic uncertainties affecting the signal normalization are partially absorbed by the fitted cross section.
We perform a simultaneous fit of σ tt and B(t → H + b) for the tauonic model. The fitting and limit setting procedure is the same as described in Sec. III, with two free parameters instead of one. Table III shows the uncertainties on B(t → H + b) and σ tt for M H + = 80 GeV. The correlation between the two fitted quantities is about 70% for M H + up to 130 GeV and it reaches 90% for M H + = 155 GeV. For high charged Higgs boson masses, where the correlation becomes high, the sensitivity degrades compared to the case where the tt cross section is fixed.
The tt cross section is set to the measured value in the generation of pseudo-datasets for the limit setting procedure. For the fit to the pseudo-data, σ tt and B(t → H + b) are allowed to float. In Table II the expected and observed upper limits on B(t → H + b) are listed together with the simultaneous measurement of the tt cross section for a top quark mass of 170 GeV. Within uncertainties, the obtained cross section for all masses of the charged Higgs boson agrees with σ tt = 8.18 +0.98 −0.87 pb, which was measured on the same data set assuming B(t → W + b) = 1 [3].
In Fig. 4 the upper limits on B(t → H + b) for M H + from 80 to 155 GeV are shown. For small M H + , the simultaneous fit provides an improvement of the sensitivity of more than 20% compared to the case where the tt cross section is fixed. Furthermore, the tt cross section measured here represents a measurement independent of the assumption B(t → W + b) = 1.
The simultaneous fit requires a reasonably small correlation between the two fitted observables. Since at present we have only included disappearance channels for the leptophobic model, the correlation between B(t → H + b) and σ tt is large (≈ 90%) for all charged Higgs boson masses, and thus we have not used the simultaneous fit method there.  Direct searches for charged Higgs bosons have been performed by the LEP experiments resulting into limits of M H + < 79.3 GeV in the framework of 2HDM [19]. Indirect bounds on M H + in the region of tan β < 40 were obtained for several MSSM scenarios [20], two of which are identical to the ones presented in Sect.V C and V D of this Letter.

A. Leptophobic model
A leptophobic model with a branching ratio of B(H + → cs) = 1 is possible in MHDM [4,5]. Here we calculate the branching ratio B(t → H + b) as a function of tan β, and the charged Higgs boson mass including higher order QCD corrections [21] using FeynHiggs [22].   and an additional mass hierarchy between the first two and the third generation of sfermions which is introduced as follows: whereX collectively represents the chiral multiplet for the left-handed doublet squarksQ, the right-handed uptype (down-type) squarksŨ (D), the left-handed doublet sleptonsL or the right-handed charged sleptonsẼ. Tak

are large MQ
3,D3 = 2MŨ 3,L3,Ẽ3 = 2 TeV, we calculate the branching ratios B(t → H + b) including higher order QCD and higher order MSSM corrections using the CPX gh MSSM parameters in Table IV. The calculation is performed with the program CPsuperH [23].

C. No-mixing scenario
In the CP-conserving no-mixing scenario, the stop mixing parameter X t is set to zero, giving rise to a relatively restricted MSSM parameter space. In the [tan β, M H + ] parameter space analyzed here the branching ratio is B(H + → τ + ν) > 0.99 except for very low values of tan β and M H + where B(H + → τ + ν) > 0.95. We interpret the results derived in the tauonic model using the simultaneous fit in the framework of the no-mixing sce- nario. The branching ratios B(t → H + b) are calculated including higher order QCD and higher order MSSM corrections using the no-mixing MSSM parameters as given in Table IV. The calculation is performed with Feyn-Higgs [22]. In the CP-conserving m h -max scenario the stop mixing parameter is set to a large value, X t = 2M SUSY . The theoretical upper bound on the lighter CP-even neutral scalar, m h , for a given value of tan β and fixed m t and M SUSY is designed to be maximal. Therefore the model provides the largest parameter space in m h and as a consequence, less restrictive exclusion limits on tan β than the other models. In the investigated [tan β, M H + ] parameter space, B(H + → τ + ν) > 0.99 holds except for low values of tan β and M H + , where B(H + → τ + ν) > 0.97. Thus we use the simultaneous fit results within the tauonic model to derive constraints on the m h -max scenario. The branching ratios B(t → H + b) are calculated using FeynHiggs [22] including higher order QCD and higher order MSSM corrections. The m h -max MSSM parameters are given in Table IV.