Searches for exclusive Higgs boson decays into $D^*\gamma$ and $Z$ boson decays into $D^0\gamma$ and $K^0_s\gamma$ in $pp$ collisions at $\sqrt{s} = 13$ TeV with the ATLAS detector

Searches for the exclusive decays of the Higgs boson into $D^*\gamma$ and of the $Z$ boson into $D^0\gamma$ and $K^0_s\gamma$ can probe flavour-violating Higgs and $Z$ boson couplings to light quarks. Searches for these decays are performed with a $pp$ collision data sample corresponding to an integrated luminosity of $136.3$ fb$^{-1}$ collected at $\sqrt{s}=13$ TeV between 2016-2018 with the ATLAS detector at the CERN Large Hadron Collider. In the $D^*\gamma$ and $D^0\gamma$ channels, the observed (expected) 95$\%$ confidence-level upper limits on the respective branching fractions are ${\cal B}(H\rightarrow D^*\gamma)<1.0 (1.2)\times 10^{-3}$, ${\cal B}(Z\rightarrow D^0\gamma)<4.0 (3.4)\times 10^{-6}$, while the corresponding results in the $K^0_s\gamma$ channel are ${\cal B}(Z\rightarrow K^0_s\gamma)<3.1 (3.0)\times 10^{-6}$.


Introduction
After the observation of the Higgs boson () with a mass of 125 GeV by the ATLAS [1] and CMS [2] Collaborations [3,4], many studies were performed to measure its properties which, so far, are consistent with Standard Model (SM) expectations [5,6].These have confirmed its role in the spontaneous breaking of electroweak symmetry and the mass generation of the massive vector bosons [7,8].A complete observation of the Higgs boson Yukawa couplings to third-generation charged fermions was achieved by the ATLAS and CMS collaborations through the observation of the decays  →  +  − [9,10] and  →  b [11,12], and the production of Higgs bosons with top-quark pairs [13,14].Evidence was also reported for the decay  →  +  − [15, 16], and direct searches for  →  c [17,18] and  →  +  − decays [19,20] were performed, but there is no further experimental evidence for the Higgs boson couplings to the first and second generations of fermions.Instead, the light (, , ) quark couplings to the Higgs boson are loosely constrained by existing data on the total Higgs boson's width and combined measurements of Higgs boson production and decays [5,6].
The ATLAS and CMS Collaborations have also investigated potential beyond-the-SM (BSM) couplings of the Higgs boson, including searches for the lepton-flavour-violating decays  → ,  →  and  →  [19,[21][22][23] and for flavour-changing neutral currents via the -quark decays  →  and  →  [24 -27].An overview of BSM theories which allow flavour-violating couplings of the Higgs boson to quarks is presented in Ref. [28].These include BSM scenarios such as the minimal flavour violation framework [29] and the Giudice-Lebedev Higgs boson dependent Yukawa couplings model [30].
Rare exclusive decays of the Higgs boson into a meson and a photon probe both the potential flavour-violating Higgs boson couplings and the Yukawa couplings of the SM [31][32][33][34][35][36][37][38][39].Analogous decays of the  boson into a meson and a photon offer unique tests of the factorisation approach in quantum chromodynamics, QCD [40][41][42], and in particular, probes of potential flavour-changing neutral current interactions of the  boson via such decays are discussed in Ref. [41].Searches for these exclusive decays were performed by the ATLAS and CMS Collaborations [43][44][45][46][47][48][49][50], and the ATLAS constraints on them, with corresponding SM predictions, are summarised in Ref. [51]. Figure 1 shows an illustrative Feynman diagram of the  () → M process (where M denotes a meson) which proceeds via a flavour-violating coupling that can be probed with searches for exclusive Higgs bosons and  boson decays into a meson and a photon, as done previously in the ATLAS search for  →  *  [48].Motivated by the potential for flavour-violating interactions in the couplings of the Higgs and  bosons to quarks, searches are described for the decays  →  * ,  →  0  (and the corresponding chargeconjugate decays  →  *  and  →  0 ) and  →  0  , which use 136.3 fb −1 of ATLAS   collision data collected at √  = 13 TeV.For the  * ,  0  and  0   final states, only the three previously mentioned specific decay channels are allowed by angular momentum conservation.Hereafter,  * includes the  * and its anti-particle  * , and  0 includes the  0 and its anti-particle  0 .
In the SM, the  *  decay arises only from loop contributions.The SM branching fraction for Higgs boson decays into c or ū has been calculated to be B ( → c/ ū) = 5 × 10 −20 [52], while the branching fraction for  →  *  is expected to be much smaller, estimated at 7 × 10 −27 [53].The corresponding SM branching fractions for the  boson decays are similarly small [41,53], estimated at 1.4 × 10 −25 and 3.3 × 10 −20 for the  →  0  and  →  0   decays, respectively [53].The decays  →  *  and  →  0  probe for flavour-violating couplings of the Higgs and  bosons to -and -quarks, and the decay  →  0   probes for flavour-violating couplings of the  boson to -and -quarks.The LHCb experiment has searched for the decay  →  0  using 2.0 fb −1 of data collected at √  = 13 TeV, yielding a 95% CL upper limit of 2.1 × 10 −3 [54], but there are no further experimental constraints on these decays.Given that the expected SM branching fractions for these decays are vanishingly small, an observation could imply physics beyond the SM, such as the existence of these flavour-violating couplings.
The signatures of the exclusive decays into a meson and a photon include a high-energy photon and a meson appearing approximately back-to-back in the detector following the decay of the  or  boson, where there is a resonance in di-track mass to reconstruct the meson and in three-body mass to reconstruct the initial boson [43][44][45][46][47][48][49][50].The decays considered in this paper have an additional feature of a displaced decay vertex, either through the decay of the  0 in  →  *  and  →  0 , or the decay of the  0  in  →  0  .This provides a particularly distinct signature as requirements on the vertex displacement are used to reject multĳet events, the dominant contribution to the background.The backgrounds in these searches are considered inclusively, and originate primarily from multĳet and +jet events involving the production of the meson or a non-resonant di-track system near its mass.The two primary decay channels of the  * are considered, into  0  0 and  0 , where the  0 and photon are soft and are not explicitly reconstructed.The searches for  →  *  and  →  0  are grouped into a single event selection which targets the decay  0 →  −  + , while the search for  →  0   has a separate selection which targets the decay  0  →  +  − .

ATLAS detector
The ATLAS detector [1] is a multipurpose particle physics detector with an approximately forwardbackward symmetric cylindrical geometry and near 4 coverage in solid angle. 1 It consists of an inner tracking detector surrounded by a thin superconducting solenoid, electromagnetic and hadronic calorimeters, and a muon spectrometer.The inner tracking detector (ID) covers the pseudorapidity range of || < 2.5, and is surrounded by a thin superconducting solenoid providing a 2 T magnetic field.At small radii, a high-granularity silicon pixel detector covers the vertex region and typically provides three measurements per track.A new innermost pixel-detector layer, the insertable B-layer, was added before 13 TeV data-taking began in 2015 and provides an additional measurement at a radius of about 33 mm around a new and thinner beam pipe [55,56].The pixel detectors are followed by a silicon microstrip tracker, which typically provides four space-point measurements per track.The silicon detectors are complemented by a gas-filled straw-tube transition radiation tracker, which enables radially extended track reconstruction up to || = 2.0, with typically 35 measurements per track.A two-level trigger and data acquisition system is used to provide online selection and record events for offline analysis [57].The level-1 trigger is implemented in hardware and uses a subset of detector information to reduce the event rate to 100 kHz or less from the maximum LHC collision rate of 40 MHz.It is followed by a software-based high-level trigger which filters events using the full detector information and records events for detailed offline analysis at an average rate of 1 kHz.An extensive software suite [58] is used in data simulation, in the reconstruction and analysis of real and simulated data, in detector operations, and in the trigger and data acquisition systems of the experiment.

Data and simulation
The searches are performed with a sample of   collision data collected at a centre-of-mass energy √  = 13 TeV from 2016-2018.The total integrated luminosity available is 136.3 fb −1 for each final state, following the requirement that events must be collected under stable LHC beam conditions with all relevant detector components in good operating condition [59].
The data samples were recorded by a combination of dedicated triggers that were available from May 2016 and remained operational until the end of data taking in 2018.The search for  →  *  and  →  0  uses triggers which target  0 →  −  + decays whilst the search for  →  0   uses triggers which target  0  →  +  − decays.These decay channels were selected as they each involve exactly two charged hadrons, such that the meson candidates can be fully reconstructed, thus providing favourable information for triggering.Each trigger requires a photon at the level-1 trigger with  T > 24 GeV.At the high-level trigger, an isolated photon with a transverse momentum   T > 35 GeV [60] is required for the 2016 data taking period, and with a reduced threshold of   T > 25 GeV throughout 2017-2018.For the ID tracks required at the high-level trigger, modified versions of the -lepton trigger algorithms [61] are used.Each trigger requires a pair of tracks that is matched to a topological cluster of calorimeter cells [62] with a transverse energy greater than 25 GeV.Within the pair, one track is required to have  T greater than 15 GeV.Different requirements on the invariant mass of the pair of tracks are applied, depending on the targeted meson decay.For  0 →  −  + an invariant mass of the pair of tracks in the range 1800-1930 MeV is required under the  ±  ∓ hypothesis, whilst for  0  →  +  − an invariant mass in the range 460-538 MeV is required under the charged-pion hypothesis.
The generation and normalisation of simulated signal samples follow the methods used in the search for  →  *  and  () →  [48] and are summarised here.Higgs boson production through gluon () and vector-boson fusion (VBF) processes was modelled up to next-to-leading-order (NLO) in  s using the Powheg Box v4 Monte Carlo (MC) event generator [63][64][65][66][67]. Powheg Box v4 was interfaced with the Pythia 8.244 MC event generator [68] to model the parton shower, hadronisation and underlying event, with parameter values set according to the AZNLO set of tuned parameters ("tune") [69] and using CTEQ6L1 parton distribution functions (PDFs) [70].Additional contributions from the production of a Higgs boson with a  ± or  boson (denoted by   and  , respectively) were also modelled with Powheg Box v4, but interfaced to Pythia 8.244 with NNPDF2.3loPDFs [71] and the A14 tune [72] for hadronisation and the underlying event.Higgs boson production with top quarks ( t) was modelled at NLO using the event generator aMC@NLO [73] interfaced to Pythia 8.244, again with the NNPDF2.3loPDFs and A14 tune.Events were generated assuming a Higgs boson mass of   = 125 GeV, but were normalised to cross-sections associated with   = 125.09GeV.These were obtained from Ref. [74] and are summarised below.The  production rate was normalised such that it reproduces the total cross-section predicted by a next-to-next-to-next-to-leading-order QCD calculation with NLO electroweak corrections applied [75,76].The VBF production rate was normalised to an approximate next-to-next-to-leading-order (NNLO) QCD cross-section with NLO electroweak corrections applied [77][78][79].The   and   production rates were normalised to cross-sections calculated at NNLO in QCD with NLO electroweak corrections [80] including the NLO QCD corrections for  →  .Powheg Box v4 was also used to model inclusive  boson production with CT10 PDFs [81].Pythia 8.244 with CTEQ6L1 PDFs [70] and the AZNLO tune was used to simulate the parton showering and hadronisation.The prediction is normalised to the total cross-section obtained from the measurement in Ref. [82].
The Higgs and  boson decays were simulated as a cascade of two-body decays.The branching fractions for  * →  0  0 and  * →  0  are (64.7±0.9)% and (35.3±0.9)%respectively [83].The branching fraction for the decay  0 →  −  + is (3.947 ± 0.030)% and the branching fraction for the decay  0  →  +  − is (69.20 ± 0.05)%.The natural lifetime of the  0 meson is (4.103 ± 0.010) × 10 −13 s and for the  0  meson is (8.954 ± 0.004) × 10 −11 s.The simulated events were passed through a detailed Geant4 simulation of the ATLAS detector [84,85] and processed with the same software used to reconstruct the data.The generation of the simulated event samples includes the effect of multiple   interactions per bunch crossing, and the effect on the detector response due to interactions from bunch crossings before or after the one containing the hard interaction.

Physics objects and event selection
In addition to the trigger and data-quality requirements, several selection criteria are defined to retain events for further analysis.The searches are split into two distinct event selections that share several common requirements.The first selection, henceforth called the  0  selection, covers the search for both the  →  *  and  →  0  decays and targets the reconstruction of the  0 meson.The additional soft daughter pion or photon from the decay of the  * is not explicitly reconstructed; this does not deteriorate the sensitivity to the  →  *  signal.The second selection, the  0   selection, covers the search for the  →  0   signal and targets the reconstruction of the  0  meson.Events with a   interaction vertex reconstructed from at least two ID tracks with  T > 500 MeV are considered in the analysis.Within an event, the primary vertex is defined as the reconstructed vertex with the largest  2 T of associated ID tracks.Common to each event selection are the photon and charged-hadron requirements.Photons are reconstructed from clusters of energy in the electromagnetic calorimeter, and clusters which match ID tracks consistent with the hypothesis of a photon conversion into  +  − are classified as converted photon candidates, whilst clusters which have no matching ID tracks are classified as unconverted photon candidates [86].Photon candidates are required to satisfy 'tight' photon identification criteria [86], and have the kinematic and geometric requirements of   T > 35 GeV and |  | < 2.37, excluding the barrel-to-endcap calorimeter transition region 1.37 < |  | < 1.52.Trackand calorimeter-isolation requirements are imposed to further suppress contamination from jets.For the track isolation, the sum of the transverse momenta of all tracks within Δ = √︁ (Δ) 2 + (Δ) 2 = 0.2 of the photon direction, excluding those matched to the reconstructed photon itself, is required to be less than 5% of   T .To mitigate the effects of multiple   interactions in the same or neighbouring bunch crossings (pile-up events), only ID tracks consistent with originating from the primary vertex are considered [87].For the calorimeter isolation, the sum of the transverse momenta of all calorimeter energy deposits within Δ = 0.4 of the photon direction, excluding those matched to the reconstructed photon, is required to be less than 2.45 GeV + 0.022 ×   T [GeV].The effects of the underlying event and pile-up events are also accounted for on an event-by-event basis using an average underlying-event energy density determined from data [86].
Charged-hadron candidates are reconstructed from ID tracks and are required to satisfy 'loose' track identification criteria [88], and require  T > 5 GeV and || < 2.5.Pairs of charged-hadron candidates are combined to form the meson candidate M for each selection, and must have opposite charges.For the  0  final state, M is a  0 meson candidate, and for the  0   final state, M is a  0  meson candidate.The charged-hadron candidate in a pair with the higher  T , the leading track, must satisfy  T > 20 GeV.Meson candidates must satisfy the isolation requirement that the sum of the  T of the reconstructed ID tracks originating from the primary vertex within Δ = 0.2 of the leading charged-hadron candidate, excluding the pair of tracks which define the meson candidate itself, must be less than 10% of the  T of the meson candidate.
Combinations of meson and photon candidates must also satisfy Δ > /2, where Δ is the difference between the azimuthal angle of the two candidates.If an event has multiple photon candidates, the candidate with highest  T is chosen; if the event has multiple meson candidates, the candidate with mass closest to the target meson is selected.However, these situations arise in a small fraction of data events: fewer than 5% of events which satisfy the trigger have more than one meson candidate, and fewer than 0.5% of events have more than one photon candidate.
The differences between the two event selections arise from requirements on M mass and  T , and requirements on the meson candidate vertex.In the  0   channel, both the tracks are assigned the charged pion mass to calculate the M candidate mass.In the  0  final state, given the absence of particle identification information in the relevant momentum range, the assignment of the charged kaon or pion masses which gives a value of the M mass closest to the  0 mass is chosen.The  0  final state requires that the meson candidate satisfies 1800 MeV <  M < 1930 MeV and  M T > 39 GeV; the  0   final state requires 460 MeV <  M < 538 MeV and  M T > 38 GeV.To reject contributions from events with prompt vertices, requirements on    /    , the    significance of the meson candidate vertex, are imposed.
Here    is the signed projection of the vector leading from the primary vertex to the meson candidate vertex onto the direction of the M candidate's  T , and its corresponding uncertainty is     .The  0  final state requires    /    > 3, whilst the  0   final state requires    /    > 5. To remove events from particle interactions in the detector material, the radius of the meson candidate vertex must satisfy  < 15 mm for the  0  selection, inside the beampipe, and  < 65 mm for the  0   selection, between the second and third layers of the pixel detector.While many  0  decays are expected to occur beyond this region, the trigger algorithm described in Section 3 is very inefficient for such events and offline reconstructed vertices with  > 65 mm are dominated by material interactions not matched to the trigger signature.These selection criteria define the nominal 'Signal Region' (SR) for each final state, and events which satisfy these criteria are retained for further analysis.

Signal modelling
Figure 2 shows the generator-level  T distributions of the two tracks and the photon reconstructed in each of the signal decays before and after implementing the nominal SR event selection for each final state.
The trigger efficiency after the offline selection is 66% for  →  * , 69% for  →  0 , and 39% for  →  0   and is primarily due to differences between online and offline tracking performance.For the  →  *  signals, the total signal efficiency, including kinematic acceptance, trigger and reconstruction efficiencies, is approximately 9%.The corresponding total efficiency for  →  0  is 3% and for  →  0   is 0.2%.Overall, the lower efficiency to reconstruct the  boson decays is due to the softer  T distributions of the decay products.The particularly small total efficiency for the  →  0   signal is related to the large lifetime of the  0  meson, as most decays occur beyond the inner layers of the ID, where the track reconstruction efficiency is significantly reduced.  simulated signal events.The hatched histograms denote the full event selection while the dashed histograms show the events at generator level that fall in the geometric acceptance (both the tracks are required to have || < 2.5 while the photon is required to have |  | < 2.37, excluding the region 1.37 < |  | < 1.52).The dashed histograms are normalised to unity, and the relative difference between the two sets of distributions corresponds to the effects of reconstruction, trigger, and event selection efficiencies.The leading track is denoted by  trk1 T and the subleading candidate by  trk2 T for  →  *  and  →  0 .For  →  0   these are labelled  1 T and  2 T .
The Higgs boson mass distribution for  →  *  is modelled with a sum of two Gaussian probability density functions (pdf) with a common mean value.The mass resolution achieved is 2.2%, and the mean is shifted to approximately 121 GeV without the reconstruction of the additional daughter  0 or photon from the decay of the  * .For the  boson signals the  boson mass distribution is modelled with a Voigtian pdf, a convolution of relativistic Breit-Wigner and Gaussian pdfs, corrected with a mass-dependent efficiency factor which accounts for the turn on in signal efficiency versus  boson mass due to the kinematic requirements of the event selection.The mass resolutions achieved are 1.9% for  →  0  and 2.3% for  →  0  , and for each signal the mean of the Voigtian is approximately 91 GeV.

Background modelling and validation
For each final state, the main sources of background are events involving inclusive multĳet or  + jet processes, where a meson candidate is reconstructed from ID tracks originating from a jet, and these show a non-resonant kinematic structure in the three-body mass distribution.The background processes are modelled with a non-parametric data-driven approach using very finely binned templates to describe the Higgs and  boson mass distributions [89].The procedure captures the correlations between the kinematic and isolation variables, thus both the multĳet and  + jet processes can be modelled inclusively.This technique was also employed in previous ATLAS searches for exclusive decays of the Higgs and  bosons into a meson and a photon [43][44][45][46][47][48], and the specific implementation in this search is summarised below.
A loose, background-dominated selection is defined, denoted the 'generation region' (GR), where the nominal isolation requirements from the SR are removed and the meson candidate  T requirement is relaxed to  T > 25 GeV.The GR contains approximately 9700 events for the  0  final states, and 1600 events for the  0   final state.From the events in the GR, pdfs are constructed to describe the distributions of the relevant kinematic and isolation variables and their most important correlations.Pseudo-events, composed of a four momentum vector and isolation variable for both the meson and photon, are generated from these pdfs using the ancestral sampling procedure described in Figure 3.In this diagram, the labels '1D' and '2D' refer to the dimensionality of the pdfs used to draw the associated variables.Where two boxes in the diagram share a border, the values of the corresponding variables are sampled simultaneously from a combined pdf, and arrows leading into boxes show that the variable is drawn from a pdf described in bins of the input variable generated in a previous stage in the sampling.Vertices in the diagram labelled 'Sum' show that the output variable, denoted by the arrow leading out of the vertex, is calculated as a sum of the input variables, denoted by the dashed lines leading in to the vertex.The nominal SR requirements are imposed on the ensemble of pseudo-events generated by this scheme, and the surviving pseudo-events are used to construct finely binned templates for the  M distributions in each search, the discriminating variable used in the likelihood fit to the data.These templates are smoothed using Gaussian kernel density estimation [90].
The possibility of residual mismodelling of the background is addressed in the context of the maximumlikelihood fit described in Section 8.The normalisation of the background model is determined directly from a fit to the observed data in the SR.To allow the model to adjust its shape in the context of the fit to the  M distributions of the data, three variations to the nominal model are derived which show notably distinct shapes in the  M distribution.The first two alternative models are associated with distortions of the model's internal histograms for two kinematic variables, namely the photon candidate  T and the angle Δ between the pseudocandidate photon and meson.The form of the chosen distortions is motivated to introduce a large enough shape variation to the resulting  M template, ensuring the flexibility of the model to accommodate even large residual shape differences with the data.The third alternative model is obtained by a direct transformation of the  M template, based on a reweighting function with a linear dependence on  M .This form is motivated by the general shape of the small residual pre-fit mismodelling observed in some earlier applications of the method [43][44][45][46][47][48].The technique of moment morphing is applied to parameterise the adjustment of the background prediction's shape between that of the nominal model and each variation [91].Each of the three shape variations has a dedicated nuisance parameter.To help stabilize the fit described in Section 8, the two nuisance parameters associated with the Δ and photon  T distortions are each constrained by a separate Gaussian term in the likelihood.However, the data in the SR has enough statistical power that all three nuisance parameters are well constrained in the fit to values associated with shape variations much milder than those of the three alternative models.
The just described model was validated through the definition of three intermediate event selection requirements, denoted validation regions (VR).Each validation region is defined by the loose GR selection with one key additional requirement taken from the tight SR selection, as outlined in Table 1.The background modelling process was used to derive predictions for each of the three VRs, which were found to provide a good description of corresponding data samples, as shown in Figure 4 for the  0  final state and Figure 5 for the  0   final state.To further validate the model, the background modelling process was  (38) GeV Tight Tight repeated with three additional independent validation regions, built using the same sampling scheme and including the shape variations.These validation regions were identical to the nominal SR up to a single change in one of the requirements, resulting in event samples orthogonal to the SR.One pair of validation regions, defined for both the  0  and  0   final states, involved a prompt-vertex selection region, in which the    significance requirement is inverted.Another pair of validation regions considered events in which the two ID tracks that form the meson candidate are required to have the same sign charge, rather than be oppositely charged.One further validation region was defined for the  0   final state which selected  0  candidates in a sideband of the mass peak, by requiring 538 MeV <   0  < 1075 MeV.In all cases, the background modelling procedure, when applied to these validation regions, exhibited good agreement with the data and served as a futher validation of the procedure.Studies indicate that potential signal contamination in the GR would not be expected to result in a significant bias in the background prediction in the SR [89]., where the meson candidate is required to satisfy  T > 38 GeV, in addition to the GR requirements.VR2 is shown in (b), where the meson-candidate isolation requirement used in the SR is applied, in addition to the GR requirements.VR3 is shown in (c), where the photon-candidate isolation requirements used in the SR are applied, in addition to the GR requirements.In all cases, the ratio of the data to the prediction of the background model is shown in the lower panel.

Systematic uncertainties
Systematic uncertainties in the signal yield and inferred branching fraction of the Higgs and  boson decays are considered.Uncertainties in the Higgs boson production cross sections from the QCD scale total 5.0% and from the PDFs and strong coupling constant,  s , total 3.2% [28,92].
The corresponding uncertainty in the  boson production cross-section is 2.9% [82].The estimated integrated luminosity has an uncertainty of 0.84%, which is calculated using the method described in Ref. [93], and uses the LUCID-2 detector [94] for the primary luminosity measurements, complemented by measurements using the inner detector and calorimeters.
The uncertainty in the acceptance of the Higgs boson signal is estimated by varying the QCD normalisation and factorisation scales,  s , PDFs, set of tuned parameters for the underlying event, and parton showering at generator level.The total uncertainty in the  →  *  signal acceptance is estimated to be 2.0%.For the  boson, the respective signal acceptance uncertainty is determined to be 0.40%-0.52%by comparing the  boson kinematic distributions in simulated events with measurements in data [95].
Trigger efficiencies for photons are determined from samples enriched with  →  +  − events in data [60].The photon trigger efficiency is estimated to contribute a systematic uncertainty of 0.66%-0.72%to each of the expected signal yields [60,96].The uncertainty in the track component of the trigger is derived by varying the leading track  T / T distribution based on comparisons between data measurements and simulated events in Ref. [97], and is 2.9% for  →  * , 3.9% for  →  0 , and 6.0% for  →  0  .Photon identification efficiencies are determined using the enriched  →  +  − event samples, and inclusive photon events and  → ℓ + ℓ −  events [86].The photon identification efficiency uncertainties are 1.6%-1.9%for the Higgs and  boson signals.The photon energy scale uncertainty, determined from  →  +  − events and validated using  → ℓ + ℓ −  events [86], is propagated through the simulated samples as a function of   and   T .The track reconstruction efficiency uncertainties total 1.0%.The uncertainty in the photon energy scale and in the resolution in the simulation has a 0.12%-0.44%effect on the Higgs and  boson signal yields.To assess any effect on the expected signal yield from imperfect modelling of pile-up, the average number of pile-up interactions is varied in the simulation; the corresponding uncertainty is 2.9%-3.1%.The uncertainty in the efficiency to reconstruct decays with displaced vertices is estimated by using the studies in Ref. [98], and is 1.5% for the  0 mesons in  →  *  and  →  0 , and 3.5% for the  0  mesons in  →  0  .Combining the individual sources of uncertainty gives a total uncertainty in the signal yields equal to 7.9% for the  →  *  signal, 6.4% for  →  0 , and 8.5% for  →  0  .The uncertainty in the shape from each of the considered signal systematic uncertainties and from the statistical uncertainty in the simulated samples was found to be negligible.Systematic uncertainties in the shape of the background model are also considered, as described in Section 6.

Results
In total, 2243 events are observed in the signal region for the  0  and  *  final states and 283 events are observed in the signal region for the  0   final state.The selected data are compared with background and signal predictions using an unbinned maximum-likelihood fit to the  M distribution in the range of 70 GeV <  M < 300 GeV.The parameters of interest are the Higgs and  boson signal normalisations, and systematic uncertainties are modelled using additional nuisance parameters in the fit.The background normalisation is a free parameter in the model.Upper limits are set on the branching fractions for each of the Higgs and  boson signal decays using the CL s modified frequentist formalism [99] with the profile-likelihood-ratio test statistic and the asymptotic approximations derived in Ref. [100].For  0  and  *  final states, limits on the  →  *  signal are obtained while profiling the signal normalisation parameter of the  →  0  process (and vice versa).The SM production cross-section is assumed for the Higgs boson while the ATLAS measurement of the inclusive  boson cross-section is used for the  boson signals, as discussed in Section 5.
The results of the background-only fits for each search are shown in Figure 6, where the signal distributions shown correspond to the extracted 95% confidence-level (CL) branching fraction upper limits.The expected and observed numbers of background events in the three-body mass ranges relevant to the Higgs and  boson signals are given in Table 2, where the expected backgrounds are obtained from fits to these backgrounds.Table 2 also shows the expected number of signal events for reference branching fractions near the sensitivity of the analysis: 10 −3 for  →  *  and 10 −6 for  →  0  and  →  0  .No significant signal is observed in any of the search channels and the -values for the background only hypothesis are 0.24, 0.23 and 0.43 in the  →  * ,  →  0  and  →  0   channels, respectively.Figure 6: Comparison between data and background prediction for the  M distribution after the background-only fit ("Post-Fit") in the signal region for the (a)  *  and  0  final states and for the (b)  0   final state.The unbinned background pdf is shown with a yellow band that represents the uncertainty in the fit arising from the constrained background shape systematic uncertainties.This uncertainty is largest in the region  M < 100 GeV, where the gradient of the distributions vary most.The choice of bin width is motivated only to best visualise the data and is independent of the likelihood definition.The lower panels show the ratio between the data and the background prediction.The expected signal distributions are shown normalised to a branching fraction corresponding to the observed 95% CL upper limit.
The expected and observed 95% CL upper limits on the branching fractions for Higgs and  boson decays into each final state are presented in Table 3, along with the observed upper limits in terms of Higgs and Table 2: Numbers of observed and expected background events for the  M ranges of interest.Each expected background and the corresponding uncertainty is obtained by integrating the total pdf after a background-only fit to the data, where the uncertainty does not take into account statistical fluctuations in each mass range.Expected Higgs and  boson signal contributions, with their corresponding total systematic uncertainty, are shown for reference branching fractions of 10 −3 and 10 −6 , respectively.Entries are marked with a dash when there is no signal of that type in the specified range.

Channel
Mass  boson production cross-section times branching fraction for each decay.The systematic uncertainties in the signal normalisation and background shape described respectively in Sections 5 and 6 result in a 2% increase of the expected 95% CL upper limit on the branching fraction of the  →  *  decay, a 9% increase for the  →  0  decay, and a 10% increase for the  →  0   decay, mostly due to the systematic uncertainty in the background shape. , and a factor of approximately 500 improvement on the  →  0  limit set by LHCb. [

Figure 1 :
Figure 1: An illustrative Feynman diagram depicting the flavour-violating  → M and  → M processes considered in this search, where M is a flavoured meson.For Higgs boson decays, M =  * ; for  boson decays, M =  0 ,  0  .The indices  and  refer to the flavour of the quark, and  ≠ .

Figure 2 :
Figure 2: Generator-level transverse momentum ( T ) distributions of the photon and of the tracks, ordered in  T , for (a)  →  * , (b)  →  0 , and (c)  →  0  simulated signal events.The hatched histograms denote the full event selection while the dashed histograms show the events at generator level that fall in the geometric acceptance (both the tracks are required to have || < 2.5 while the photon is required to have |  | < 2.37, excluding the region 1.37 < |  | < 1.52).The dashed histograms are normalised to unity, and the relative difference between the two sets of distributions corresponds to the effects of reconstruction, trigger, and event selection efficiencies.The leading track is denoted by  trk1T and the subleading candidate by  trk2 T for  →  *  and  →  0 .For  →  0   these are labelled  1T and  2 T .

Figure 3 :
Figure 3: The ancestral sampling scheme used in the non-parametric data-driven model for each final state, see text for details.Solid arrows indicate that the output variable is sampled in bins of the input variable, whilst dashed arrows indicate that the output variable is calculated as a sum of the input variables.

Figure 4 :
Figure4: The   ±  ∓  distribution of the data and pre-fit background model prediction of validation regions for the  0  final state.VR1 is shown in (a), where the meson candidate is required to satisfy  T > 39 GeV in addition to the GR requirements.VR2 is shown in (b), where the meson-candidate isolation requirement used in the SR is applied, in addition to the GR requirements.VR3 is shown in (c), where the photon-candidate isolation requirements used in the SR are applied, in addition to the GR requirements.In all cases, the ratio of the data to the prediction of the background model is shown in the lower panel.

Figure 5 :
Figure 5: The   +  −  distribution of the data and pre-fit background model prediction of validation regions for the  0  final state.VR1 is shown in (a), where the meson candidate is required to satisfy  T > 38 GeV, in addition to the GR requirements.VR2 is shown in (b), where the meson-candidate isolation requirement used in the SR is applied, in addition to the GR requirements.VR3 is shown in (c), where the photon-candidate isolation requirements used in the SR are applied, in addition to the GR requirements.In all cases, the ratio of the data to the prediction of the background model is shown in the lower panel.

Table 3 :
Observed and expected (with the corresponding ±1 intervals) 95% CL upper limits on the branching fractions for  →  * ,  →  0  and  →  0  .Standard Model production of the Higgs boson is assumed.The corresponding upper limits on the production cross-section times branching fraction  × B are also shown.95% CL upper limits Branching Fraction  × B [fb] Channel Observed Expected Observed Expected  →  *  1.0 × 10 −3 1.2 Searches for the flavour-violating exclusive decays of the Higgs boson  →  * ,  →  0  and  →  0   were made using 136.3 fb −1 of   collision data at √  = 13 TeV collected with the ATLAS detector at the LHC.Each of these decay channels involve a displaced vertex, either from the decay of a  0 meson or from a  0  meson.The observed data are compatible with the expected backgrounds.The observed 95% CL upper limits are B ( →  * ) < 1.0 × 10 −3 , B ( →  0 ) < 4.0 × 10 −6 , and B ( →  0  ) < 3.1×10 −6 .The corresponding expected 95% CL upper limits on the branching fractions of each decay are B ( →  * ) < 1.2 × 10 −3 , B ( →  0 ) < 3.4 × 10 −6 , and B ( →  0  ) < 3.0 × 10 −6 , where Standard Model Higgs boson production is assumed.These results represent the first limits set on the decays  →  *  and  →  0
The calorimeter system covers the pseudorapidity range of || < 4.9.A high-granularity lead/liquid-argon (LAr) sampling electromagnetic calorimeter covers the region || < 3.2, with an additional thin LAr presampler covering || < 1.8 to correct for energy losses upstream.The electromagnetic calorimeter is divided into a barrel section covering || < 1.475 and two endcap sections covering 1.375 < || < 3.2.For || < 2.5 it is divided into three layers in depth, which are finely segmented in  and .A steel/scintillator-tile calorimeter provides hadronic calorimetry in the range of || < 1.7, while in the endcap region, 1.5 < || < 3.2, a copper/LAr calorimeter is used.The solid-angle coverage is completed with forward copper/LAr and tungsten/LAr calorimeter modules in 3.1 < || < 4.9, optimised for electromagnetic and hadronic measurements, respectively.The muon spectrometer surrounds the calorimeters and comprises separate trigger and high-precision tracking chambers measuring the deflection of muons in a magnetic field provided by three air-core superconducting toroids.

Table 1 :
Definition of validation regions, each of which is based on the GR selection with a single additional requirement associated with the SR selection.In the case of the meson  T requirement, the value for the  0   final state is shown in parentheses, while the value for the  0  final state is not.The isolation requirements of all regions are common to both the  0  and  0   final states.