Introduction

When the indirect band gap of a material is continuously reduced from positive to negative, band theory predicts a semiconductor-to-semimetal Lifshitz transition. However, such a transition may never be reached in the presence of electron-hole Coulomb attraction. It is postulated that poor screening in low carrier density semimetals and strong electron-hole binding in narrow gap semiconductors can both result in the spontaneous formation of excitons1,2. The condensation of excitons results in an insulating ground state, the so-called excitonic insulator (EI), which is separated from the normal state by a dome-shaped phase boundary peaked around the Lifshitz point (see Fig. 1a)3. This simple setup became a primary platform to search for condensed excitons and other exotic correlated phases, as well as potential solid-state BCS-BEC crossover phenomena4,5,6,7. For example, adding electronic anisotropy can further narrow the EI dome due to reduced Coulomb attraction8,9, and sufficiently heavy holes may cause Wigner crystallization10. Moreover, band-gap-tuned BCS-BEC crossover has been recently suggested in 2D moiré systems11. However, interband electron-phonon interaction can also lead to a gapped charge-density-wave state that occurs concurrently with the exciton condensation4. It remains unclear how the excitonic instability and the density-wave instability interact with each other, and whether it is possible at all to distinguish them. Since no super-transport (charge or heat) is associated with the excitonic insulator phase, and its analogy to BCS superconductivity is only formal10, an effective experimental method to distinguish the Coulomb and lattice channels is to identify a tuning parameter that gives differentiating predictions of the phase diagram.

Fig. 1: Band-gap-tuned semimetal-to-semiconductor transition in narrow band gap systems.
figure 1

a Archetypal phase diagrams in indirect band gap controlled excitonic insulator (EI) across the semimetal-to-semiconductor transition. b Normal state electrical resistivity evolution with S-doping in Ta2Ni(Se,S)5. c Structure motif of quasi-1D Ta2Ni(Se,S)5 system below and above transition temperature Ts (upper panel). The lattice distortion in the monoclinic phase is exaggerated. Symbolic sketch of a minimal lattice model of the system used to describe the low-energy interactions in the system (lower panel). \({\epsilon }_{k}^{c}\) and \({\epsilon }_{k}^{v}\) are kinetic energies of conduction and valence electrons. ωq is the chain-shearing B2g phonon frequency. g is the interband electron-phonon coupling vertex. V is the electron-hole Coulomb interaction potential.

Full size image

Band-gap-controlled materials that undergo semimetal-to-semiconductor transitions provide an ideal platform to investigate this issue. However, the experimental realization of such a transition in bulk systems has been challenging. Divalent metals under hydrostatic pressure and uniaxial strain were first proposed as promising candidates12, but experimental evidence was indirect and scarce without clear ways to separate Coulomb and electron-phonon interaction effects. Recent experiments suggest a potential exciton condensation in 1T-TiSe213, where the band gap tunability is limited. Quasi-one-dimensional (quasi-1D) ternary chalcogenide Ta2NiSe5 has recently emerged as another leading EI candidate, exhibiting an orthorhombic-to-monoclinic structural transition at Ts = 329 K, concomitant with a putative exciton condensation14,15. The system consists of parallel quasi-1D Ta and Ni chains (see Fig. 1c), where long-lived electron-hole pairs are supposed to dwell on Ta 5\({d}_{{x}^{2}-{y}^{2}}\) and Ni 3dxy orbitals. On the one hand, the sizable ground-state single-particle gap16,17,18,19, minute structural distortion14,20, and a “dome-like” temperature-pressure phase diagram10,15 are all ostensibly consistent with the predictions in a band-gap-tuned EI. On the other hand, strong electron-phonon coupling effects are also found to coexist with correlation effects20,21,22,23,24,25,26,27,28,29,30,31,32. With the presence of both interactions, the band structure alone no longer sufficiently dictates the excitonic insulator phase region, and it is unclear how the low-temperature broken symmetry state would evolve across the Lifshitz transition point. In Ta2Ni(Se,S)5, this question can be addressed since the value of the direct band gap can be effectively tuned by isovalent sulfur substitution of selenium15,33. However, the phase diagram of Ta2Ni(Se,S)5 – especially where the broken symmetry phase ends and where the normal state crosses from a gapless to a gapped electronic structure with S-doping – has seen vastly contradicting results14,15,27, mainly due to the lack of direct probes of the single-particle band gap and the order parameter of the broken symmetry.

In this work, using angle-resolved photoemission spectroscopy (ARPES) and high-resolution synchrotron x-ray diffraction (XRD), we directly probe the electronic and lattice structure of Ta2Ni(Se,S)5 system and map out its phase diagram across the semimetal-to-semiconductor. A broken symmetry phase is found to be continuously suppressed from the semimetal side to the semiconductor side, contradicting the anticipated maximal excitonic instability around the Lifshitz transition. Bolstered by first-principles and model calculations, we reveal the crucial role of strong interband electron-phonon coupling behind the phase diagram.

Results

The general impact of S-substitution on the low-energy electronic structure can be revealed through the normal-state (defined here as the high-temperature symmetric structure phase) resistivity measured along the chain direction, which slowly increases until an abrupt upturn around 70% doping (see Fig. 1b). Such an upturn indicates a potential Lifshitz transition on the normal-state band structures. Hence, S-doped Ta2NiSe5 constitutes an ideal platform to study band-gap-tuned semimetal-to-semiconductor transition in an electron-phonon coupled correlated system. We first track the evolution of the broken symmetry phase boundary in Ta2Ni(Se,S)5with direct measurements of the structural order parameter. In bulk excitonic insulators, exciton condensation is always tied to a simultaneous lattice distortion10. Employing high-resolution single-crystal synchrotron XRD, Fig. 2a characterizes the temperature- and S-substitution-dependent evolution of the lattice order parameter β (also see Supplementary Note 1), which is defined as the angle between the lattice a and c axes shown in Fig. 1c. We find the structural phase transition remains second-order throughout the entire S-doping range. The ground-state order parameter β(T = 0) monotonically decreases with increasing S-doping (Fig. 2b), and the structural transition temperature only approaches zero near full S-substitution (Ta2NiS5) (bottom panel in Fig. 2b). This is notably different from the electronic phase diagram inferred from resistivity (see Fig. 2b and Supplementary Note 1), which instead suggests an abrupt collapse of the broken symmetry phase at an intermediate S-doping level15. Therefore, it is imperative to obtain a direct electronic structure view to determine the normal state semimetal-to-semiconductor transition point pL along the S-doping axis.

Fig. 2: Structural order parameter of Ta2Ni(Se,S)5.
figure 2

a Change of the inter- and intra-chain Ta-Ta bond angle β − 90∘ as a function of temperature and S-doping. Error bars are determined based on the fitting error of the Bragg peak positions. b Top: experimental and DFT calculated β − 90∘ at zero temperature. Bottom: structural transition temperature Ts as a function of S-doping level, determined from XRD and resistivity measurement.

Full size image

Previous studies have reported conflicting values of pL from resistivity, Raman measurements15,27. Here, we conduct high-resolution ARPES measurement, which is a direct probe of electronic structure, to reveal the evolution of band overlap/gap of Ta2Ni(Se,S)5 across the semimetal-semiconductor phase transition. The high statistics of the measurements enable electronic state restoration up to 4.5 kBT above EF (see Methods), and the orbital selectivity through polarized X-ray beam, allows the direct comparison with density functional theory (DFT) calculations, both of which help us precisely locate pL. Figure 3 shows the photoemission spectra in both the normal and the broken symmetry state (also see Supplementary Note 1). As the S-doping level increases, the Ta 5\({d}_{{x}^{2}-{y}^{2}}\) orbital component (blue) gradually withdraws from the valence band top, which becomes dominated by the Ni 3dxy orbital (red). Pseudogap states, the strong depletion of spectra intensity around Fermi level19,20, are also observed, which account for the high-temperature insulating behavior in these semimetallic samples (see ref. 20 for detailed description). Beyond pL ( ~ 70% S-doping), the normal-state conduction and valence bands are completely separated in energy (see Fig. 3a) and, simultaneously in the broken-symmetry-state, the Ta orbital component largely disappears from the valence band top (see Fig. 3b). Fitting the normal-state band dispersions (see Supplementary Note 2 for details) quantifies the band gap (or band overlap in the case of negative values), which changes sign across pL (Fig. 3c). In addition, revealed through the “M-shape” dispersion of the valence band top in the broken symmetry state, the band back-bending momentum kF also approaches Γ near pL (Fig. 3d). Both results are consistent with a concurrent normal state semimetal-to-semiconductor transition. Our DFT results further confirm such a Lifshitz transition near pL via the band-gap-sign flip and the kF reduction at the single-particle level, without involving ad hoc electron-hole Coulomb attraction (see Fig. 3c, d and Supplementary Note 3).

Fig. 3: Electronic Lifshitz transition in Ta2Ni(Se,S)5.
figure 3

ARPES spectra along X − Γ − X direction for samples with different S-doping levels. a High-temperature orthorhombic phase. b Low-temperature monoclinic phase (except for 100% S-doping, where no structural transition was evidenced down to 10 K). Spectra from linear horizontal (LH, blue) and linear vertical (LV, red) polarization of incident light are overlaid, highlighting Ta 5\({d}_{{x}^{2}-{y}^{2}}\) conduction band and Ni 3dxy valence bands respectively. VBT - valence band top. c DFT calculation of 22% S-doped compound in high-temperature orthorhombic phase, and the evolution of Eg extracted from DFT calculation and photoemission spectra as a function of S-doping. d DFT calculation of 22% S-doped compound in low-temperature monoclinic phase, and the evolution of 2kF extracted from DFT calculation and photoemission spectra as a function of S-doping. Error bars are determined based on (c) the variation between dispersion fitting models and (d) the fitting error of the MDC peak position.

Full size image

The ARPES measurement of the semimetal-to-semiconductor transition doping is consistent with that deduced from resistivity measurement (Figs. 1b and 2b). However, the structural phase transition temperature Ts and the structural order parameter (approximated as \({\beta }_{\max }-9{0}^{\circ }\)) are found to decrease monotonically with increasing S-doping through our XRD measurements (Fig. 2b). This poses a marked departure from the archetypal band-gap-tuned excitonic insulator phase diagram, where the excitonic instability maximizes around pL (Fig. 1a). To highlight this contrast, Fig. 4a shows the evolution of both the measured structural transition temperature Ts and the experimentally determined Ta atom displacement ∣X∣ in Ta2Ni(Se,S)5. Both ∣X∣ and Ts appear insensitive to the system crossing pL, which leads to considerations beyond weak electron-hole Coulomb interaction. One possibility is a heavy-hole Wigner crystal phase10; but the electron and hole effective masses are both near the free electron mass in Ta2Ni(Se,S)533. Alternatively, strong Coulomb interaction may push the phase boundary towards the semiconductor side34, but the Hartree component will undo this effect20 (see also Supplementary Note 4). Considering the fact that strong electron-phonon coupling effect is revealed in both equilibrium and ultrafast pump-probe experiments in Ta2Ni(Se,S)520,21,22,23,24,25,26,27,28, we hereby investigate an experimentally informed minimal model with strong electron-phonon coupling, shown in Fig. 1c and Eq. (1) of Methods. This model contains both electron-hole Coulomb interactions and an interband electron-phonon coupling, which can well describe both the spectral properties of Ta2NiSe5 below and above Ts20. To examine the impact of strong electron-phonon coupling on the band-gap-tuned transition as a matter-of-principle question, we first set the inter-band Coulomb interaction to zero. The role of this Coulomb interaction primarily lies in a Hartree shift between two bands and can be mapped to a correction of Eg (see detailed discussions in Supplementary Note 4). The average phonon displacement ∣xph∣ at T = 0 is employed to approximate the order parameters ∣X∣ measured in the experiment (Fig. 4a). With the quantized phonon modes, a zero-point fluctuation leads to the zero-coupling baseline at ∣xph∣ = 1.

Fig. 4: Band gap tuned phase diagram with electron-phonon coupling.
figure 4

a Evolution of structural transition temperature Ts and the maximum Ta atom displacement ∣X∣ as a function of S-doping in Ta2Ni(Se,S)5. The electronic band structures within different regions are illustrated in the insets. Error bars are determined based on the fitting error of the Bragg peak positions. b The calculated average lattice displacement \(| {x}_{{{{{{{{\rm{ph}}}}}}}}}|=\sqrt{\langle {\hat{x}}_{{{{{{{{\rm{ph}}}}}}}}}^{2}\rangle }\) as a function of the band gap Eg at various electron-phonon coupling strengths. In the quantized phonon model, the zero-point fluctuation gives a finite ∣xph∣ = 1 in the decoupled limit. c Sketched phase diagrams of band-gap-tuned semimetal-to-semiconductor transition in the strong (top) and weak (bottom) coupling limits. Solid lines mark the phase boundaries, above which the shades show the order fluctuations20. Red shade on the bands illustrates the electronic states that participate in the phase transition. The horizontal dashed lines denote EF.

Full size image

Solved with exact diagonalization, ∣xph∣ for various Eg and coupling strengths are shown in Fig. 4b. In the weak-coupling case (e.g. g = 14 meV), the lattice order parameter displays a dome-like structure similar to the EI phase diagram derived based on Fermi-surface instability in Fig. 1a. With increasing coupling strengths, the dome gradually evolves into a monotonic crossover. This evolution exists not only for the many-body measure of fluctuations, but also for order parameters when symmetry breaking is allowed (see Supplementary Note 4). The monotonic crossover driven by strong, non-perturbative interactions reflects the irrelevance of Fermi-surface instability (top panel in Fig. 4c). The coupling strength in Ta2NiSe5, estimated from both ARPES spectral fitting and first-principles deformation potentials20, is 50 ~ 60 meV, placing it in the strong-coupling regime. While more complicated models with multiple bands may be necessary to quantitatively reproduce the order parameter evolution, strong electron-phonon coupling qualitatively describes the evolution of the broken-symmetry phase across pL in experiments. Moreover, strong coupling also expands the momentum range over which the lattice becomes unstable35, opening up the opportunity for a q = 0 transition when 2kF is small.

We demonstrate that strong electron-phonon interaction can greatly enhance the structural symmetry breaking in a semimetal, and the evolution of the broken-symmetry phase boundary can be indifferent to the semimetal-to-semiconductor transition. On the specific debate over the nature of the symmetry breaking in Ta2Ni(Se,S)5 systems, our results suggest a substantial deviation from the electron-hole Coulomb attraction-based EI phase (bottom panel in Fig. 4c). Instead, a strong electron-phonon-coupling induced structural transition can explain the observed symmetry breaking. This conclusion is consistent with recent experimental evidence of the crucial role of phonons in the equilibrium and nonequilibrium properties of Ta2NiSe522,23. Meanwhile, it has been suggested recently that additional Coulomb and acoustic-phonon effects can also cooperatively contribute to the transition27. On the high-temperature fluctuations, previous investigations have identified a normal-state spectral “pseudogap”, characterized by an anomalous spectral weight accumulation near the conduction band bottom. This phenomenon is interpreted as the signature of preformed excitons due to strong electronic correlation or electron coupling to lattice fluctuations20,36. We find that such a spectral anomaly ubiquitously exists in the normal state on the semimetal side of Ta2Ni(Se,S)5, but rapidly disappears on the semiconductor side (Fig. 3a). Intriguingly, the boundary of the pseudogap, or preformed excitons, is almost temperature-independent, similar to the abruptly terminated pseudogap in hole-doped superconducting cuprates37,38. Last but not least, the heavily S-substituted Ta2Ni(Se,S)5 system provides a promising platform to study both solid state BCS-BEC crossover phenomenon39 and critical fluctuations of low-energy phonons. Intriguingly, a potential competing order is observed in Fig. 2a for S-dopings above pL, similar to the situation in iron-based high Tc superconductors, where multiple orders are found competing with each other at low temperatures, including nematicity, spin-density wave, and superconductivity40,41.

Methods

Sample synthesis and resistivity measurements

Single crystals of Ta2Ni(Se,S)5 were grown by the chemical vapor transfer method with iodine (I2) as the transport agent. Starting materials composed of Ta powder (99.99%), Ni powder (99.99%), Se powder (99.99%), and S powder (99.99%) with a nominal molar ratio 2:1:5(1-x):5x (x is the nominal doping level of S) were fully ground and mixed together inside the glovebox. About 50 mg of iodine (I2) was added to the mixture of the starting powder. The mixture was then vacuumed, back-filled with 1/3 Argon, and sealed inside a quartz tube with an inner diameter of 8 mm, an outer diameter of 12 mm, and a length of about 120 mm. The sealed quartz tube was placed horizontally inside a muffle furnace during the growth. The hot end reaction temperature was set to 950 °C, and the cold end was left in the air with the temperature stabilized at 850 °C. Long and thin single crystals were harvested by quenching the furnace in the air after one week of reaction. Excess iodine was removed from the surfaces of the crystals with ethanol. Electrical resistivity measurements were carried out on a commercial PPMS (Quantum Design) by the four-probe method with the current applied along the a-axis of the Ta2Ni(Se,S)5 single crystals.

Angle-resolved photoemission spectroscopy (ARPES)

Synchrotron-based ARPES measurements were performed at the beamline BL5-2 of Stanford Synchrotron Radiation Laboratory (SSRL), SLAC, USA, and the BL 7.0.2 of the Advanced Light Source (ALS), USA. The samples were cleaved in situ and measured under the ultra-high vacuum below 3 × 10−11 Torr. Data was collected by the DA30L analyzer. The total energy and angle resolutions were 10 meV and 0.2°, respectively. High statistics measurements were done with a signal-to-noise ratio of 100 ~ 150 at the Fermi level on the energy distribution curve taken at the Fermi momentum. This enables electronic structure restoration up to 4.5 ~ 5 kBT after Fermi function division38,42. Orbital content selection is based on the dipole transition matrix element effect, following the experimental geometry described in ref. 20.

Single crystal x-ray scattering

Hard X-ray single-crystal diffraction is carried out at the energy of 44 keV at the beamline QM2 of the Cornell High Energy Synchrotron Source (CHESS). The needle-like sample is chosen with a typical lateral dimension of 100 microns, then mounted with GE Varnish on a rotating pin before being placed in the beam. A Pilatus 6M 2D area detector is used to collect the diffraction pattern with the sample rotated 365° around three different axes at a 0.1° step and 0.1s/frame data rate at each temperature. The full 3D intensity cube is stacked and indexed with the beamline software.

First-principles DFT calculation

Ab initio calculations are performed using the Quantum ESPRESSO package43,44. The crystal structure relaxations are performed using the r2SCAN functional45 with a semiempirical Grimme’s DFT-D2 van der Waals correction46. S-doped compounds are simulated by the virtual crystal approximation (VCA), where the pseudopotentials of Se and S atoms are linearly interpolated (i.e. mixed together) according to the chemical composition. A 30 × 30 × 15 k-mesh and a 100 Ry wavefunction energy cut-off were used. The electronic structure calculations are performed using the same r2SCAN functional with the relaxed structures.

Two-band model for many-body simulations

We consider the two-band model with inter-band e-ph couplings to describe the physics in the Ta2Ni(Se,S)5 systems, whose Hamiltonian reads as47:

$${{{{{{{\mathcal{H}}}}}}}}({E}_{g})= \mathop{\sum}\limits_{k\sigma }{\varepsilon }_{k}^{c}({E}_{g}){c}_{k\sigma }^{{{{\dagger}}} }{c}_{k\sigma }+\mathop{\sum}\limits_{k\sigma }{\varepsilon }_{k}^{v}({E}_{g}){f}_{k\sigma }^{{{{\dagger}}} }{f}_{k\sigma }+V\mathop{\sum}\limits_{i,\sigma,{\sigma }^{{\prime} }}\big({n}_{i\sigma }^{c}+{n}_{i+1\sigma }^{c}\big){n}_{i{\sigma }^{{\prime} }}^{f}\\ +\mathop{\sum}\limits_{kq\sigma }\frac{{g}_{q}}{\sqrt{N}}\left[\big({a}_{q}+{a}_{-q}^{{{{\dagger}}} }\big){c}_{k+q\sigma }^{{{{\dagger}}} }\, {f}_{k\sigma }+{{{{{{{\rm{H.c.}}}}}}}}\right]+\mathop{\sum}\limits_{q}{\omega }_{q}{a}_{q}^{{{{\dagger}}} }{a}_{q}$$
(1)

where \({c}_{k\sigma }^{{{{\dagger}}} }\) (ckσ) creates (annihilates) an electron at the conduction band (primarily Ta 5d) for momentum k and spin σ, with dispersion given by \({\varepsilon }_{k}^{c}\); and the \({f}_{k\sigma }^{{{{\dagger}}} }\) (fkσ) creates (annihilates) an electron at the valence band (primarily Ni 3d), with dispersion given by \({\varepsilon }_{k}^{v}\) (Fig. 1c). The \({n}_{i\sigma }^{c}\) and \({n}_{i\sigma }^{f}\) are the density operators for the conduction and valence band, respectively. We employ the band structures fitted from the Ta2NiSe5 experiments, as reported in ref. 20, which determine the valence and conduction band dispersion for a negative band gap Eg = − 0.3 eV. Due to the difficulty of fitting the full dispersions in the gapped Ta2Ni(Se,S)5 materials, we simplify the model using a rigid band separation controlled by Eg. That being said, the conduction and valence band structures for Ta2Ni(Se,S)5 read as

$${\varepsilon }_{k}^{c}({E}_{g})=3.25-1.8\,\cos (k)-0.9\,\cos (2k)-0.6\,\cos (3k)+{E}_{g}/2$$
(2)
$${\varepsilon }_{k}^{v}({E}_{g})=-1.95+1.5\,\cos (k)+0.3\,\cos (2k)+0.1\,\cos (3k)-{E}_{g}/2\,.$$
(3)

Here, Eg = 0 indicates the Lifshitz transition where \({\varepsilon }_{k=0}^{c}={\varepsilon }_{k=0}^{v}\), while Eg = 0.3 eV reflects the situation of Ta2NiS5.

To approximate the lattice distortion in a finite-size simulation, we employ the average displacement of phonons

$$| {x}_{{{{{{{{\rm{ph}}}}}}}}}|=\sqrt{{{{{{{{\rm{Tr}}}}}}}}\left[\rho {\hat{x}}_{{{{{{{{\rm{ph}}}}}}}}}^{2}\right]}\,,$$
(4)

where ρ is the density matrix of the e-ph system and \({\hat{x}}_{{{{{{{{\rm{ph}}}}}}}}}=({a}_{0}^{{{{\dagger}}} }+{a}_{0})/N\) is the uniform phonon displacement operator (In a finite-size system, the exact ground state does not spontaneously break symmetries, so that \(\langle {\hat{x}}_{{{{{{{{\rm{ph}}}}}}}}}\rangle \equiv 0\)).