Instanton effects in rank deformed superconformal Chern-Simons theories from topological strings

In the so-called (2, 2) theory, which is the U(N)4 circular quiver superconformal Chern-Simons theory with levels (k, 0, −k, 0), it was known that the instanton effects are described by the free energy of topological strings whose Gopakumar-Vafa invariants coincide with those of the local D5 del Pezzo geometry. By considering two types of one-parameter rank deformations U(N)×U(N + M)×U(N + 2M)×U(N + M) and U(N + M)×U(N)×U(N + M)×U(N), we classify the known diagonal BPS indices by degrees. Together with other two types of one-parameter deformations, we further propose the topological string expression when both of the above two deformations are turned on.


Introduction
M-theory is a mysterious theory. From the AdS/CFT correspondence, it was conjectured [1] from the gravity side that the free energy of M2-branes, the degrees of freedom of fundamental excitations in M-theory, is N 3/2 in the large N limit. After the proposal [2][3][4] that the N = 6 superconformal Chern-Simons theory with gauge group U(N 1 ) k ×U(N 2 ) −k and two pairs of bifundamental matters describes min(N 1 , N 2 ) M2-branes and |N 2 − N 1 | fractional M2-branes on the target geometry C 4 /Z k , this conjecture was confirmed from the gauge theory side. Namely, after using the localization technique [5], the partition function of the ABJM theory on S 3 , which is originally defined by the infinite-dimensional path integral, is reduced to a finite-dimensional matrix integration. Then, the 't Hooft expansion 1 of the matrix model is applicable [8][9][10]. Due to the peculiar power 3/2, it is natural to ask what the corrections in the large N limit are. The perturbative corrections were found to be summed up to the Airy function [10], which turned out to be just a starting point of the full exploration. One of the celebrating results in this matrix model is the correspondence to the topological string theory. After the discovery of the Fermi gas formalism [11] which rewrites the partition function into that of a Fermi gas system with N particles, more information on the matrix model was obtained. Besides the 't Hooft expansion, we can perform the WKB small k expansion [11,12] or study the numerical fitting from the exact values of the partition function [13][14][15][16]. Finally, the non-perturbative effects of the matrix model in the grand canonical ensemble are given explicitly by the free energy of the topological string theory on the local P 1 × P 1 geometry [8,9,17]. Although this result was originally found for the case of equal ranks N 2 = N 1 , it was soon generalized into the case of different ranks N 2 = N 1 [18,19].
If we consider general ranks, this class turns out to have a rich structure. Namely, the theories with different sequences of {s a } are expected to be connected and to form a non-trivial moduli space as a whole. In fact, since the ordering of s a = ±1 corresponds to the ordering of the quantum operators [21] in the one-particle density matrix in the Fermi gas formalism, the theories which are different only in the ordering are equivalent in the classical limit. This may suggest that these theories correspond to the topological string theory on the same Calabi-Yau manifold [35]. Furthermore, the N = 4 theory with the circular quiver can be translated to a type IIB brane system by replacing the edges s a = ±1 with (1, k)5-/NS5-branes and the vertices with N D3-branes stretching between two 5-branes. In this context the ordering of {s a } is the ordering of the (1, k)5-/NS5branes. This suggests that the theories with different ordering indeed should be unified if we consider general ranks, as a pair of 5-branes can be exchanged at the expense of creation/annihilation of the D3-branes, which is known as Hanany-Witten transitions [36].
In this paper we consider the (2, 2) model with general ranks. In fact, the (2, 2) model is the simplest one which is essentially distinctive from the ABJM theory U(N 1 ) k ×U(N 2 ) −k due to the following observations. Firstly, without rank deformations, the grand potential of the (2, 2) model was found to have the structure of the free energy of topological strings [23]. We actually found that the diagonal Gopakumar-Vafa invariants of the (2, 2) model match completely with those of the local D 5 del Pezzo geometry (table 6 in [37]). Secondly, the (2, 2) model allows a rank deformation with maximally three parameters (besides the overall rank N ), while the local D 5 del Pezzo geometry has 5 degrees of freedom for the Kähler parameters (besides the overall scaling), which implies a richer structure between the ranks JHEP08(2017)003 and the Kähler parameters. Thirdly, the (2, 2) model is the theory with the smallest number of 5-branes which has a non-trivial dependence on the ordering. Therefore, although so far a direct relation to the topological string theory was not found for any other N = 4 theories, it is worthwhile to study the rank deformations of the (2, 2) model thoroughly to understand the relation between the N = 4 theories and the topological string theories.
Among the general rank deformations, we find that the following two one-parameter rank deformations are particularly tractable: U(N ) k ×U(N + M ) 0 ×U(N + 2M ) −k ×U(N + M ) 0 and U(N + M ) k ×U(N ) 0 ×U(N + M ) −k ×U(N ) 0 . Unlike the situation with the general R charges [38], the FI terms completely cancel in these deformations. By using the technique [18] called the open string formalism, we can factorize the grand canonical partition function of these models as the product of the grand partition function of M = 0 and the determinant of a matrix whose matrix elements are expressed with a generalization of the two functions φ m (q) and ψ m (q) introduced in [23]. Although these functions were originally introduced from a technical reason to compute the grand canonical partition function of M = 0, it is interesting to find that these functions appear naturally in the rank deformations as well. Moreover, from the computation we can give a clear reason for the appearance of the two functions: φ m (q) and ψ m (q) are associated to the vertices with non-zero levels and zero levels respectively.
The result again has the structure of the free energy of topological strings. By matching the table of the BPS indices of the local D 5 del Pezzo geometry (the tables in section 5.4 of [39]) with the instanton coefficients obtained from the numerical studies of these two rank deformations, we can classify the BPS indices by various degrees. Interestingly, we have obtained different classifications of the BPS indices for the two one-parameter rank deformations, where the BPS indices in the latter case are a further classification of those in the former case.
After studying the two one-parameter rank deformations of the (2, 2) model, we propose a unification of them by a two-parameter deformation. As a non-trivial check of our unification and our classification of the BPS indices, we turn to another N = 4 theory, the (1, 1, 1, 1) model, whose rank deformations are connected to those of the (2, 2) model. We discuss the Hanany-Witten transition to see how these two models are related and study the rank deformations of the (1, 1, 1, 1) model as well. We have found that all of our classifications of the BPS indices are consistent with the rank deformations on the (1, 1, 1, 1) side. This paper is organized as follows. In the next section, we shall review the (2, 2) model without rank deformations. As we see in section 2.2.3, already by revisiting the result we find a lot of information to split the BPS indices, if we assume the existence of two Kähler parameters and compare the BPS indices with the table in [39]. In section 3.1, we study the first deformation U(N ) k ×U(N +M ) 0 ×U(N +2M ) −k ×U(N +M ) 0 . Though the open string formalism resembles to that of the ABJM case, it also contains a new indefinite feature, which requires a careful regularization. After this regularization, we are able to give a clear reason for the appearance of the two functions φ m (q) and ψ m (q). In section 3.2, we study the second deformation U(N + M ) k ×U(N ) 0 ×U(N + M ) −k ×U(N ) 0 . The non-perturbative effects are much complicated and we explain that the result can be given by the free energy of topological strings if we introduce six Kähler parameters. After that, in section 4 we JHEP08(2017)003 study other two rank deformations by utilizing the Hanany-Witten transition and propose to unify the four one-parameter rank deformations with a two-parameter deformation. Finally in section 5, we summarize and discuss future directions. The paper also contains four appendices. Appendix A and appendix B provide some useful formulas for the open string formalism. The exact values of the partition function and the grand potential for the four one-parameter rank deformations are summarized in appendix C. These results are crucial in the determination of the Kähler parameters and the BPS indices. In appendix D we provide the closed string formalism for U where the duality to the (1, 1, 1, 1) model at M = k/2 is manifest.

ABJM theory and (2, 2) model
Since we will discuss the topological string description of the non-perturbative effects in the (2, 2) model with rank deformations, we shall start by reviewing the Fermi gas formalism and the non-perturbative effects for the standard ABJM theory with the rank deformation and the (2, 2) model without rank deformations. We shall see that the topological string description of the non-perturbative effects in the ABJM theory is successful, though there is room for improvement for that in the (2, 2) model. Most of the contents in this section are reviews, though in section 2.1.3 and section 2.2.3 we will also raise some questions and clarify some points which we believe are not so trivial even to the experts but important for our later analysis. The main references of the review part are [17,18,23] (see also [40,41]).

Open string formalism
After applying the localization technique [5], the partition function of the ABJM theory on S 3 reduces to a matrix model Without loss of generality, we often set M = N 2 − N 1 ≥ 0 for the ABJM matrix model. Otherwise, we can apply the complex conjugation. Hereafter we set (N 1 , N 2 ) = (N, N +M ). One of the standard techniques to study this ABJM model with generally non-equal ranks M ≥ 0 is to utilize the following two essentially same determinant formulas [18] N m<m ′ 2 sinh which are obtained by combining the Vandermonde determinant and the Cauchy determinant. By multiplying these two determinant formulas, the extra exponential factors e ± M 2 m µm and e ± M 2 n νn cancel among themselves and we can perform the ν integrations by using a determinant formula in [18] (which is further generalized in appendix B) Here we regard Q(µ, ν), P (ν, µ) and E l (ν), E a (ν) respectively as matrices and vectors with continuous indices and the multiplications are given by contracting these continuous indices with the integrations in (2.2). Furthermore if we define the grand canonical partition function as and apply another determinant formula in [18], we arrive at the expression Here both the Fredholm determinant Det = exp Tr log and the matrix element H l,a (z) are defined in the small z expansion, consisting of traces and matrix elements of the powers of P Q.
To evaluate the exact values of the partition function, it is convenient to introduce the coordinate operator q and the momentum operator p satisfying the canonical commutation relation [ q, p] = i with the identification = 2πk. We also introduce the coordinate eigenstate |q and the momentum eigenstate |p as well as summarize some useful formulas in appendix A. In fact, after the Fourier transformation and the similarity transformation, Ξ ABJM (2.9)

JHEP08(2017)003
Notice that the density matrix ρ has the following structure [14,42] [ M , ρ] = E|0 0| E, This structure allows us to decompose the powers q 1 | ρ n |q 2 as with φ m (q) = q| E −1 ρ m E|0 , which can be computed recursively as This expression reduces the computation of q 1 | ρ n |q 2 to a much simpler computation of multiplying ρ to E|0 subsequently. By rewriting E l and E a with the momentum eigenstates 2πil| and |−2πia we can also reduce the computation of H l,a (z) to the computation of the functions obeying the same recurrence relation (2.13) as that of φ m (q). Indeed these functions are a straightforward generalization of φ m (q) by replacing the initial state |0 with |−2πi(a − 1 2 ) .

Instanton effects
The main strategy in studying the instanton effects in the ABJM matrix model with rank deformations is to evaluate the exact values of the partition function, read off the instanton coefficients numerically and interpolate each instanton coefficient to a function of (k, M ). If we define the grand potential J by adding infinite numbers of replicas from the beginning. Though it is ambiguous to decide which one in the replicas to be the reduced grand potential, we can avoid the ambiguity in the definition by choosing the real one satisfying (J ABJM k,M (µ)) * = J ABJM k,M (µ * ). The concept of introducing the reduced grand potential appears repetitively in the following for various JHEP08(2017)003 theories other than the ABJM theory, though we always define the reduced grand potential in the same way as (2.15).
Then, the reduced grand potential is expressed in terms of the free energy of the topological string theory on the local P 1 × P 1 geometry. First if we separate the grand potential into the perturbative part and the non-perturbative part in the large µ limit, (µ), the perturbative part is given by and A ABJM k [43,44] for integral k. For the non-perturbative part there appear bound states of worldsheet instantons e − 4µ k and membrane instantons e −2µ . It turns out that if we re-expand J ABJM k,M (µ) by the effective chemical potential µ eff which is defined by with (s L = 2j L + 1, s R = 2j R + 1) s R sin 2πg s ns L n(2 sin πg s n) 2 sin 2πg s n e −nd·T ,

JHEP08(2017)003
Here the two Kähler parameters T = (T + ; T − ) and the string coupling constant g s are (2.22) while N d j L ,j R are the BPS indices of the local P 1 × P 1 geometry. In this way the degrees of freedom for the rank deformations of two gauge groups reproduce the two degrees of freedom for the Kähler parameters of local P 1 × P 1 .

Comment on definition of grand potential
Although we have noted that without loss of generality we can consider M = N 2 − N 1 ≥ 0, it is interesting to ask how the grand potential changes across M = 0. From the inverse transformation of (2.15), we find From this we expect that if we define the grand potential as the grand potential coincides with the original one with negative M . In fact, we can check that J ABJM k,M (µ) +M µ withM ≥ 0, when expressed with the same effective chemical potential µ eff , coincides with the free energy of the topological string theory with the Kähler parameters T ± = 4µ eff /k ± πi(1 + 2M /k). Note that in (2.25) the power of e µ has to match with the same argument of the partition function regardless of the relative magnitude of the ranks, though the lower bound of the sum is not vey important. This observation is crucial later in our discussion of the rank deformation with multiple parameters.

(2, 2) model without rank deformations
Now let us turn to the review of the (2, 2) model without rank deformations. It turns out that most of the techniques and the results of the ABJM matrix model apply similarly to the (2, 2) model.

Fermi gas formalism
With the localization technique, the partition function of the (2, 2) model on S 3 reduces to the following matrix model As in the case of the ABJM theory [11], it was found that the grand canonical partition (2.28) A systematic method to compute these traces is established again by noticing the following structure of the matrix element of ρ [23] [ which we can compute recursively in m [23]. With this method we can compute many exact values for the (2, 2) model as well to study the non-perturbative effects. Interestingly, compared with the ABJM theory, here we need to introduce two functions (2.32) to accomplish this analysis. In section 3.1.2 we will comment on the origin of the two functions, which becomes apparent after we introduce the rank deformations.

Instanton effects
To summarize the result for the reduced grand potential (2.15) of the (2, 2) model, it is again convenient to introduce the effective chemical potential which is now given as Then, J k (µ) is split into the perturbative part and the non-perturbative part where C k , B k and A k are given as [21] given previously in (2.18). As in the ABJM case, due to the redefinition of the chemical potential (2.33), significant simplifications happen in the non-perturbative effects. Namely, the bound states of the worldsheet instantons e − µ k and the membrane instantons e −µ are taken care of by the pure worldsheet instantons and the non-perturbative part is simply given as with the pure worldsheet instantons 38) and the pure membrane instantons The instanton coefficients simplify in the following multi-covering structure where the first few coefficients are given by

JHEP08(2017)003
and β 1 (k) = − 2 sin 2πk π sin 2 πk , β 2 (k) = 8 sin 2πk + sin 4πk 2π sin 2 πk , β 3 (k) = − 6 sin 2πk + 6 sin 4πk π sin 2 πk , β 4 (k) = 9 sin 2πk + 30 sin 4πk + 9 sin 6πk π sin 2 πk , β 5 (k) = − 20 sin 2πk + 100 sin 4πk + 100 sin 6πk + 20 sin 8πk π sin 2 πk . (2.42) In the previous paper [23] it was found that, as in the ABJM model, J np k (µ eff ) of the (2, 2) model is consistent with the expression of the free energy of topological strings (2.21), if we identify the Kähler parameter and the string coupling as since the worldsheet and membrane instanton effects are respectively given by e − µ eff k and e −µ eff . Here by "consistent" we mean that a set of the numbers N d j L ,j R exists (not necessarily uniquely), with which the expression (2.21) reproduces the instanton coefficients δ d (k) and β d (k) determined by fitting. Note that in [23] we assumed boldly the absence of the discrete B-field effect from the simple multi-covering structure without sign factors (2.40) (unlike the ABJM case (2.22) with the effect of discrete B-field). Otherwise, the determination of the BPS indices was impossible with too many unknowns.
Though the identification of the (diagonal) BPS indices 3 called the (diagonal) Gopakumar-Vafa invariants, can be straightforwardly read off from the worldsheet instanton coefficients. As a result we found that they completely coincide with the Gopakumar-Vafa invariants of the local D 5 del Pezzo geometry (table 6 in [37]). After noticing that the inverse of the density matrix ρ −1 in the classical limit [ q, p] → 0, )} of the local D 5 del Pezzo geometry (see polygon 15 in [39]), this coincidence is probably not so surprising. Rather it can be regarded as a non-trivial check for the conjecture on the quantization of spectral curves 4 [35,45].
We also found, however, that the membrane instantons are inconsistent with the topological string expression (2.21) with the BPS indices on the local D 5 del Pezzo geometry (tables in section 5.4 of [39]). Since the match of the Gopakumar-Vafa invariants is highly non-trivial, we are strongly confident of the relation with the local D 5 del Pezzo geometry. Although originally we adopt the ansatz (2.43) to determine the BPS indices to reduce the 3 Here and in the following, the norm |d| stands for the sum of all components of d = (d1, d2, · · · ), |d| = i di, and the summation |d|=d stands for that under this constraint Unless mentioned explicitly, we often abbreviate |d| simply as d. 4 We thank Alba Grassi for valuable discussions.

JHEP08(2017)003
unknowns, after being confident with the relation to the local D 5 del Pezzo geometry, we can borrow the diagonal BPS indices directly from the table in [39] and ask what is the correct identification of the Kähler parameters and the BPS indices. Below, we shall start from the identification.

Splitting of Kähler parameters and BPS indices
Here let us see how to resolve the discrepancy in the BPS indices. We note that already at this point without rank deformations we have much information about the BPS indices. First we introduce a minor change in (2.21). In [15,17] it was noticed that for the BPS indices on the local P 1 × P 1 geometry satisfying 2j are cancelled among the worldsheet instantons and the membrane instantons. When the constraint is violated, the cancellation of the divergences does not work for (2.21) any more. Indeed the BPS indices listed in section 5.4 of [39] do not satisfy the constraint 2j L + 2j R − 1 ∈ 2Z.
To restore the cancellation of the divergence, we need to modify the expression of the worldsheet instantons slightly while keeping the expression of the membrane instantons, Indeed, with (2.45) we can show by an explicit regularization k → k + ǫ that the divergent coefficients at each e −nd·T add up to the following finite quadratic polynomial in the Kähler parameters Here | j L ,j R ,d,n stands for the summand in (2.45) with the indicated quantum numbers. Although the pole cancellation works by the above change, this may jeopardize the multi-covering structure (2.40) of the worldsheet instanton in the Gopakumar-Vafa interpretation. Looking more closely at the table of the BPS index in [39], however, we find the BPS indices still satisfy a modified constraint 2j L + 2j R − 1 − d ∈ 2Z. The modified constraint allows us to replace the sign factor (−1) (2j L +2j R −1)n with (−1) dn , which can be compensated if we define the Kähler parameters as Then, the multi-covering structure is preserved by defining (2.48)

JHEP08(2017)003
These modifications keep the worldsheet instantons, while change the membrane instantons. For simplicity, here let us assume that there are two Kähler parameters, each of which is shifted by ±πi, as in the case of the ABJM theory [19]. For the membrane instanton the multi-covering Now we try to find the BPS indices N which are consistent with the actual membrane instanton coefficients (2.42). Since we are already confident of the relation to the D 5 del Pezzo geometry, we start with the assumption that the diagonal BPS indices are given by the table in section 5.4 of [39]. Then, assuming the positivity of the BPS indices In where each term in the first factor can be interpreted as the contribution from (d + ; d − ) = (2; 0), (0; 2) and (1; 1) respectively.

Two rank deformations
In the previous section we have solved the mismatch of the BPS indices by introducing two Kähler parameters (2.49) and classifying the known diagonal BPS indices. Of course it is reasonable to doubt whether this ad hoc classification is meaningful. In the following JHEP08(2017)003 we shall consider the partition function of the (2, 2) model with the rank deformation with the integrations defined in (2.27). Surprisingly, we find that the BPS indices identified in the previous section actually works for one deformation.
Our strategy is the same as before. For the open string formalism to work, we would like to rewrite the partition function Z k (N 1 , N 2 , N 3 , N 4 ) using the combined determinant formula (2.3) of the Vandermonde determinant and the Cauchy determinant. In general the extra exponential factors e ± M 2 m µm and e ± M 2 n νn cause difficulties in the analysis, hence we hope them to cancel among themselves. Eventually we have found that the following two one-parameter deformations do not require extra cares: For later convenience, among the general rank deformations, let us denote the two-parameter deformation as the deformation (M 1 , M 2 ). We shall define the grand canonical partition function Ξ k,(M 1 ,M 2 ) (z) and the reduced grand potential J k, with Note that we have correlated the power of e µ , N + M 2 , to the first entry of the partition function (3.4). This is partially motivated by the discussions of reversing the rank sizes in the ABJM theory in section 2.1.3. The correlation of the power of e µ with the entry of the partition function is very important. Otherwise, we would encounter difficulties later in expressing the grand potential with the two-parameter rank deformation J k,(M 1 ,M 2 ) (µ) in terms of the topological string theory.
In this notation Ξ k (z) and J k (µ) without deformations appearing in section 2.2 should be denoted as Ξ k,(0,0) (z) and J k,(0,0) (µ). Also, the above two one-parameter deformations should be denoted respectively as (M, 0) and (0, M ) and the corresponding reduced grand potentials are J k,(M,0) (µ) and J k,(0,M ) (µ), which are the main subjects in this section. See figure 1 for the schematic picture of the analysis in this section. where JHEP08 (2017)003 with l, l ′ = M − 1 2 , M − 3 2 , · · · , 3 2 , 1 2 and a, a ′ = −M + 1 2 , −M + 3 2 , · · · , − 3 2 , − 1 2 (appearing in the matrices in this order). Note that Q, Q ′ , P ′ and P are identical functions of two variables, except the origin of the two variables. This redundant notation is helpful as it encodes the types of the two arguments and makes the matrix multiplication clearer. Using the formula in appendix A of [18] (or a special case of our appendix B), we can combine the four determinants into one (3.7) With the use of the formula in appendix B of [18], the grand canonical partition function becomes which can further be computed as with Ξ k (z) = Det(1+zP ′ P QQ ′ ) being the original grand canonical partition function (2.28) without rank deformations. The only remaining task is the computation of the matrix elements in (3.9). However, a naive evaluation leads us to the problems of infinity, E l Q ′ = E l 2 cosh πil −1 = ∞ and On the other hand, if we factor out the divergent factors there remain four identical blocks which makes the determinant vanishing. This indicates that we can remove the divergence by appropriate elementary row/column operations. Below we show this by an explicit regularization.

Regularization
To get rid of the indefinite property, let us regularize the expression (3.9) by changing E l and E a as

JHEP08(2017)003
which implies where we have used Then, (3.9) can be regularized as Now if we expand the regulator R as we find that all the divergent terms in the matrix elements can be eliminated by the elementary row/column operations, and we obtain (3.16) It is interesting to find that two functions (E l ′ ), (E l q) or (E a ′ ), (qE a ) appear, which correspond to φ m (q) and ψ m (q) introduced in [23] to study the (2, 2) model without rank deformations. Note that in [23] we introduced these two functions technically from the Fourier transformation of the powers of the hyperbolic secant function (2.29) without knowing the deep meaning of its appearance. After introducing the rank deformations, we see that there are in fact two functions E l ′ (ν), E l (κ) or E a ′ (ν), E a (λ) appearing in the determinant formula (3.5) which have distinct origins from the beginning: one is associated to the vertices of non-vanishing levels in the quiver diagram, while the other is associated to those of vanishing levels. Due to the vanishing levels we end up with an indefinite expression, which requires a regularization. The function ψ m (q) turns out to be the regularized form of E l (κ) or E a (λ) associated to the vertices of vanishing levels.
Using these exact values we can apply the numerical fitting to find the coefficients C k , B k,(M,0) , A k,(M,0) for the perturbative part and the instanton coefficients for the nonperturbative part. For the perturbative part, we find that the coefficients fit the exact values well. For the non-perturbative part the instanton coefficients are listed in appendix C.1.2.
As we have reviewed in section 2.2.2, the reduced grand potential has the structure of the free energy of topological strings only after we introduce the effective chemical potential µ eff . Here µ eff is determined such that the coefficient of e −nµ eff proportional to π −2 is given by ∼ ((nµ eff ) 2 /2 + nµ eff + 1)/π 2 as in (2.46). Since the terms proportional to π −2 in appendix C.1.2 is simply a sign modification from the undeformed case, the expression of µ eff turns out to be a sign modification from (2.33), Using this effective chemical potential, we can express the reduced grand potential as in appendix C.1.3. Now let us study the instanton effects in this reduced grand potential. As before we expect that after the rewriting with the effective chemical potential, the bound states of worldsheet instantons and membrane instantons are incorporated simply in the pure worldsheet instanton terms. Hence we can use all of the data without the pure membrane instanton effects to study the worldsheet instanton effects. We find that the worldsheet instantons (2.38) satisfy the same multi-covering structure as (2.40) (3.20) Note that of course it is impossible to fix infinitely many coefficients of the worldsheet instanton from the finite data. Our criterion for determining the coefficients is that the set of finite data fixes the next coefficient to vanish.

JHEP08(2017)003
As a simplest attempt, we stick to the choice of two Kähler parameters. The expression of the coefficient B in (3.17) may indicate that the two Kähler parameters introduced in (2.49) are modified into From this expression we can classify the BPS indices by the degrees by comparing the worldsheet instantons (3.20) with the general formula (2.45). Surprisingly, the result fits exactly with table 1 obtained from the membrane instanton without rank deformations. Besides, after determining the BPS indices, we can proceed to study the effects e −nµ eff which contain both worldsheet instantons and membrane instantons. We continue to find the exact match. This result indicates that, with the identification of the where Q, Q ′ , P , P ′ and E are given in (3.6). E ′ is the same as E. We give them different notations by indicating that E is contracted with e ik 4π µ 2 while E ′ is contracted with e − ik 4π ν 2 . To combine the determinants the determinant formula in appendix A of [18] is not enough. We have proved a generalized version of the determinant formula in appendix B, so that we can combine all of the determinants into one Applying the formula in [18] we finally obtain

JHEP08(2017)003
Each element of the 2M ×2M matrix can be computed in the small z expansion as in (3.9). This time we do not need any regularizations.

Instanton effects
In this case we have obtained the exact values for (k, M, N max ) = (2, 1, 13), (3, 1, 6), (4, 1, 7), (4, 2, 7), (6, 1, 6), (6, 2, 5) and (6,3,5), which are given in appendix C.2.1. From these exact values we can start the numerical fitting, We have not figured out the general expression of the constant map A k,(0,M ) except For the other cases we obtain it may suggest that the two Kähler parameters in the previous case further split into six As we see later in section 3.2.3 these Kähler parameters are indeed consistent with the instanton coefficients. As before, from the numerical analysis we can read off the reduced grand potential J np k,(0,M ) (µ) and rewrite it in terms of the effective chemical potential J np k,(0,M ) (µ eff ). The results are given respectively in appendix C.2.2 and appendix C.2.3. This time from the signs of the terms proportional to π −2 we define the effective chemical potential as with the total Gopakumar-Vafa invariants preserved, (3.32) Let us first discuss the validity of the identification of the Kähler parameters (3.29) along with the BPS indices of d = 1. From the reality of the reduced grand potential, we expect that each Kähler parameter is accompanied with its complex conjugate and that the BPS indices are symmetric under the exchange of the degrees for all the complex-conjugate pairs (3.34) Under these assumptions, we shall match the topological string expression with the first worldsheet instanton. Since we only have non-vanishing BPS indices for (j L , j R ) = (0, 0) in d = 1, we shall match J WS k (µ eff ) in (2.45) with these substitutions against δ 1 (k, M ) in (3.31)

JHEP08(2017)003
This also implies which will be useful below. Next, let us turn to the match of the quadratic polynomial in the instanton effects where we have both the first membrane instanton and other worldsheet instantons. By matching the constant part without the proportionality π −2 in (2.46) obtained after cancelling the apparent singularities with n = 1 and (j L , j R ) = (0, 0) substituted, we obtain the equality of After fixing the Kähler parameters to be (3.29) we can proceed further to higher instantons. Since we have generated six Kähler parameters (3.29) from a single parameter M , there are some essential ambiguities. In fact, due to the relations , (3.43) we cannot distinguish which degrees the BPS indices belong to. For example, for the degrees (d + ; d − ) = (2, 0), the independent degrees are while for the degrees (d + ; d − ) = (1, 1), the independent degrees are If we match the free energy of topological strings with the unknown BPS indices against (3.31) with the help of the e −2µ eff term in (k, M ) = (2, 1), we can uniquely fix the BPS indices as in table 2 for d = 2, aside from the essential ambiguity (3.43). For d = 3 strictly speaking we cannot split the BPS indices in the first row and in the third row. We fix the BPS indices by imposing an additional symmetry of exchanging T ± 1 and T ± 3 . In section 4, we shall see that actually the separation is consistent by introducing new deformations.

Attempt for general (M 1 , M 2 )
In the above two subsections, we have studied two specific rank deformations with only one of the deformation parameters (M 1 , M 2 ) turned on. This is basically because these two The non-perturbative part may be more restrictive from the expression of the free energy of topological strings. If we expect that the non-perturbative part is expressed by the topological string free energy (2.45), the only possibility is to change the number of the Kähler parameters or the identification of these geometrical parameters with the gauge theory parameters. The simplest choice to combine the above two subsections would be Unfortunately, note that the Kähler parameters in (3.47) deformed by two parameters (M 1 , M 2 ) still cannot lift the degeneracies of the degrees in (3.43). So far we have proposed a natural interpolation of the result of the above two specific rank deformations. To really trust this interpolation we need some checks. In the following we shall point out through the brane construction that there are two other tractable oneparameter deformations which can be used to check our proposal.

Two additional rank deformations
In the previous section, after studying the two rank deformations (M 1 , 0) and (0, M 2 ), we have tried to guess the unified expression of the reduced grand potential J k,(M 1 ,M 2 ) (µ eff ) for the rank deformation (3.2) with general two deformation parameters (M 1 , M 2 ). We expect that the non-perturbative effects are given by the free energy of topological strings with the six Kähler parameters (3.47). There are, however, not enough data to check the validity of (3.47).
In this section, we point out that the same theory can be investigated from its fieldtheoretical dual. Namely, as reviewed in section 4.1, by utilizing the Hanany-Witten transition in the brane picture, we can map the rank deformed (2, 2) model with gauge group (4.1) to the rank deformed (1, 1, 1, 1) model with gauge group In particular, with the duality between (4.1) and (4.2), we find that the two one-parameter rank deformations (M 1 , M 2 ) = (k/2 − M, k/2) and (M 1 , M 2 ) = (k/2, k/2 − M ) of the (2, 2) model correspond respectively to the rank deformations of the (1, 1, 1, 1) model (N ′ = N +k/2−M ), which are tractable in the exact computation of the partition function. Below we first explain briefly the duality between (4.1) and (4.2) using the Hanany-Witten transition and continue to study the two additional rank deformations. The schematic picture of the analysis in this section is given in figure 2.

Dualities from Hanany-Witten transitions
Here we shall explain the duality from the brane configurations. It was known [46,47] that the quiver superconformal Chern-Simons theories are realized by NS5-branes, (1, k)5-branes as well as some D3-branes in the brane configurations. Here the (1, k)5-brane and the NS5brane correspond respectively to s a = +1 and s a = −1 which characterize the levels by It is famous that the exchange of 5-branes will add/subtract the number of D3-branes in-between, called Hanany-Witten transition. Though the original brane system studied in [36] consists of a D3-brane (in 0123 plane) stretched between a NS5-brane (in 012456 plane) and a D5-brane (in 012789 plane), we can generalize the argument to the system with N D3-branes stretched between two types of general (q, p)5-branes [46]. In the case of our interest, we obtain the following equivalence (1,k)5 JHEP08(2017)003 by the direct substitution. According to the Hanany-Witten transition (4.2), we can rewrite the partition function into that of the (1, 1, 1, 1) model which is (up to complex conjugation) nothing but the orbifold theory of the rank deformed ABJM theory U(N ) k ×U(N + M ) −k . Hence we can skip the numerical computation by applying the technique established in [20] to generate the explicit expression of the grand potential directly from that of the U(N ) k ×U(N + M ) −k theory J ABJM k,M (µ) obtained in [18,19] (4.10) For the perturbative part we obtain We can introduce the effective chemical potential µ eff from the same criterion (2.46) as in the (M, 0) case, which we find for k = 2, 4, 6 while for k = 3. In appendix C.3.1 and appendix C.3.2 we list the non-perturbative part of the reduced grand potential in terms of µ and µ eff respectively. First we notice a non-trivial consistency check. In section 3.1 we have studied the deformation (M 1 , M 2 ) = (0, M ) of the (2, 2) model and in the current section we have studied 6 Though in [20] the reduced grand potential is defined from the partition function without taking the absolute values, the derivation of (4.10) works for our convention as well. Indeed the derivation is based only on the relation between the density matrices in the closed string formalism of Z ABJM

JHEP08(2017)003
the deformation (M 1 , M 2 ) = (k/2 − M, k/2) through the duality. Though in general these two deformations are complement with each other, both of the analyses are applicable at the point M = k/2. We have found a consistency check of these two analyses. In fact, we have compared the results of the exact values of the partition functions for (k, M ) = (2, 1), (4, 2), (6, 3) and find them completely identical. The same result is rederived for general even integers k using a different method called closed string formalism in appendix D.
If we express the worldsheet instanton effects by we find that, surprisingly, all of the worldsheet instantons with odd instanton numbers m are identically vanishing. The first few even worldsheet instantons are given by This time we find the coefficients C and B are given by for k = 2, 4, 6 while 20) for (k, M ) = (3, 1). We find that the comments found in the last subsection also apply here. In fact, at the point M = k/2, the analysis of the deformation (M 1 , M 2 ) = (k/2, k/2 − M ) in this subsection is consistent with that of the deformation (M 1 , M 2 ) = (M, 0) in section 3.1. Also, the odd worldsheet instanton effects are identically vanishing again.

Unifications
In the previous section, we have considered two rank deformations (M 1 , 0) and (0, M 2 ). From the rank deformation (M 1 , 0), we have found the BPS indices are separated by the two Kähler parameters (3.21). Furthermore from the rank deformation (0, M 2 ) we can further split the BPS indices by the six Kähler parameters (3.29). By combining these two deformations we have introduced (3.47) by interpolations. Here with further rank deformations, we can confirm the consistency of these interpolations.
We have observed that the coefficient C k is independent of (M 1 , M 2 ). This is rather trivial, however, as C k appears in the strict large N limit of the free energy which depends only on the volume of the dual geometry. The first non-trivial unification appears in the coefficient B k,(M 1 ,M 2 ) . From the four deformations, if we require B k,(M 1 ,M 2 ) to be given by a linear combination of six terms 1/k, M 1 , M 2 , M 2 1 /k, M 1 M 2 /k and M 2 2 /k, we uniquely find that The (M 1 , M 2 )-dependence of this B can be expressed by quadratic terms in the imaginary part of the Kähler parameters (3.47) This fact implies that the perturbative part of the grand potential J pert k,(M 1 ,M 2 ) (µ eff ) can be expressed as a cubic polynomial in the Kähler parameters but without quadratic terms (i, j, l = 1, 2, · · · , 6) where (T 1 , T 2 , T 3 , T 4 , T 5 , T 6 ) = (T + 1 , T + 2 , T + 3 , T − 1 , T − 2 , T − 3 ) and c ijl , c i are some constants (decomposition is not unique due to the identity (3.43)). This observation is consistent JHEP08(2017)003  with the Yukawa coupling [19,48] and would be an additional positive evidence for the unification. Also, it seems that the coefficient A k,(M 1 ,M 2 ) depends only on the argument We can also check the BPS indices identified in table 2. Interestingly, we find that our proposal reproduces correctly the vanishing odd worldsheet instantons. For example, for the first worldsheet instanton, we find that it works for both the deformations (M 1 , M 2 ) = (k/2 − M, k/2) and (M 1 , M 2 ) = (k/2, k/2 − M )

JHEP08(2017)003
After having a non-trivial check for our proposal of describing the reduced grand potential in terms of the free energy of topological strings, we can now turn to the BPS indices with higher degrees d = 4. If we again assume the symmetry (3.46), we can determine the BPS indices completely as in table 3. Since there is no other way to check the result, we leave the BPS indices in table 3 as a conjecture.
Finally, we stress that we have unified the four one-parameter deformations as a deformation with two parameters. In the unification, it is important that in the definition of the reduced grand potential (3.3) we have correlated the power of e µ with the argument of the partition function (3.4) as partially motivated by the discussions in section 2.1.3. In fact, if we change the definition into, for example, that with the power of e µ being simply N or min(N + M 1 , N + M 2 ), the unification never occurs.

Summary and conclusions
In this paper we have studied the (2, 2) model with the rank deformation of the two parameters (M 1 , M 2 ). We start with the two one-parameter deformations by setting M 1 = 0 or M 2 = 0. One of them splits the diagonal BPS indices with two Kähler parameters and the other further splits the BPS indices with six Kähler parameters. Then we propose an expression with the linear interpolation of these two deformations. Using the Hanany-Witten transition we can further check the validity of our proposal from the (1, 1, 1, 1) model with rank deformations. To conclude, we have found that the reduced grand potential of a quiver superconformal Chern-Simons theory with the two-parameter rank deformation is expressed by the free energy of the topological string theory. Compared with the previous works of rather trivial geometrical structures, local P 1 × P 1 or local P 2 , our model has a much richer structure.
There are apparently many questions related to our work.
Although we have claimed that we find a non-trivial check for our proposal of the topological string theory, we can only check the "boundary" of the moduli space (see figure 2). It is of course desirable to move into the "bulk" where both of M 1 and M 2 are neither zero nor k/2. So far we are not aware of any good methods for this purpose.
From our analysis of the BPS indices using the (2, 2) model, we give a prediction of how the diagonal BPS indices on the local D 5 del Pezzo geometry in [39] are split by various Kähler parameters. We hope that these can be understood from the geometrical method. It will be very interesting to compare our BPS indices with the results in [49] obtained by the refined topological vertex formalism [50,51] where the computation was done for the general six-parameter deformation.
It is surprising that all of the odd worldsheet instantons vanish when we study from the (1, 1, 1, 1) model side. In other words, if we study the same rank deformation completely from the (1, 1, 1, 1) model side, we would never arrive at the correct Kähler parameters. The vanishing is due to a fine cancellation happening with the specific Kähler parameters and the specific BPS indices. Probably this reflects an unknown symmetry of the local D 5 del Pezzo geometry.

JHEP08(2017)003
We have utilized the so-called open string formalism [18] for the numerical analysis. It is also desirable to establish a closed string formalism for all of four boundary deformations. This may encode the target Calabi-Yau threefold of the topological string theory in a direct way [52]. It may also enable us to study the membrane instantons from the WKB expansion [53].
We have considered the rank deformations U(N + At present it is unclear whether the open string formalism is applicable to other rank deformations. For example if we consider the deformation U(N ) k ×U(N + M ) 0 ×U(N ) −k ×U(N + M ) 0 or U(N + M ) k ×U(N ) 0 ×U(N + M ) −k ×U(N + 2M ) 0 we apparently encounter a divergence where our regularization does not work. We are not sure whether these divergences are solvable with technical improvements or essential to the system.
Probably it is not difficult to extend our work to the orthosymplectic groups [3,4,[53][54][55]. It is interesting to see whether the results of these theories are given by the chiral projection [56][57][58] or not. The study would give us a deeper understanding of the instanton effects.
It is also not difficult to study the instanton effects of the vacuum expectation values of the BPS Wilson loop in the (2, 2) model [59,60]. The Giambelli compatibility proved for the ABJM theory [61,62] seems to work parallelly.
Of course it is interesting to ask whether general (p, q) models are all described by the free energy of the topological string theory. From our experience of the vanishing of the odd worldsheet instantons in the (1, 1, 1, 1) model, the possibility has even enhanced.
We hope to answer these questions in the future.

B Determinantal formula
In this appendix we shall prove a determinantal formula generalizing that proved in appendix A of [18].
Proof. The case of M 2 = 0 was already proved in [18]. We can rederive it by assuming the case of M 1 = M 2 = 0 using the Laplace expansion.
Here in the Laplace expansion on the second line we have partitioned the set (N + M 1 ) = {1, · · · , N, N + 1, · · · , N + M 1 } by two disjoint sets (N ) and (M 1 ) which contain N and M 1 elements respectively and ε (N ),(M 1 ) denotes the sign of the permutation determined by these two disjoint sets.

JHEP08(2017)003
Similarly we can derive the general case using the Laplace expansion if we assume the case of M 2 = 0.
The final expression is nothing but the Laplace expansion of the desired result (B.1),

C Exact values and reduced grand potentials
In this appendix we list the first few absolute values of the exact values, the nonperturbative part of the reduced grand potential J np k,(M 1 ,M 2 ) (µ) obtained from the numerical fitting, and the expression of the reduced grand potential as a function of the effective chemical potential J np k,(M 1 ,M 2 ) (µ eff ).

JHEP08(2017)003
First, let us rewrite the partition function (3.22) into the bra/ket notation as follows

JHEP08(2017)003
Here, as explained previously, for the first determinant, since all of the components become the delta functions 8 as λ i | 1 2 cosh q 2 |µ j = 2π 2 cosh λ i 2 δ(µ j − λ i ) and 2πil|µ j = 2πδ(µ j − 2πil), we can perform the µ integrations by trivialization and substitution. For the second determinant, after extracting the factor e πia 2 k out of the determinant, we can decompose it into the product again by using the Cauchy-Vandermonde determinant in the opposite way. Similarly, for the third line we find (D.10) After combining the two determinants by using the formula in appendix A in [18], we obtain (D.1).
The Hanany-Witten duality (4.8) at M = k/2 is manifest in the closed string formalism. Indeed, with the help of the following identity (D.12) Hence the density matrix is ρ k,(0,M ) = ρ 2 ABJM , which is same as the density matrix of the orbifold ABJM theory [20].
Open Access. This article is distributed under the terms of the Creative Commons Attribution License (CC-BY 4.0), which permits any use, distribution and reproduction in any medium, provided the original author(s) and source are credited. 8 Notice that we have to deform the integration contour of µj (j > N ) in the pure-imaginary direction to process the delta functions from 2πil|µj . This deformation is valid only for M ≤ k/2; otherwise the contour crosses the poles of µj| 1 2 cosh p 2 |κi in the complex plane during the deformation, which we would also have to take into account [57].