Improving sentiment analysis via sentence type classification using BiLSTM-CRF and CNN

Different types of sentences express sentiment in very different ways. Traditional sentence-level sentiment classification research focuses on one-technique-fits-all solution or only centers on one special type of sentences. In this paper, we propose a divide-and-conquer approach which first classifies sentences into different types, then performs sentiment analysis separately on sentences from each type. Specifically, we find that sentences tend to be more complex if they contain more sentiment targets. Thus, we propose to first apply a neural network based sequence model to classify opinionated sentences into three types according to the number of targets appeared in a sentence. Each group of sentences is then fed into a one-dimensional convolutional neural network separately for sentiment classification. Our approach has been evaluated on four sentiment classification datasets and compared with a wide range of baselines. Experimental results show that: (1) sentence type classification can improve the performance of sentence-level sentiment analysis; (2) the proposed approach achieves state-of-the-art results on several benchmarking datasets. © 2016 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license ( http://creativecommons.org/licenses/by/4.0/ ).


Introduction
Sentiment analysis is the field of study that analyzes people's opinions, sentiments, appraisals, attitudes, and emotions toward entities and their attributes expressed in written text ( Liu, 2015 ). With the rapid growth of social media on the web, such as reviews, forum discussions, blogs, news, and comments, more and more people share their views and opinions online. As such, this fascinating problem is increasingly important in business and society.
One of the main directions of sentiment analysis is sentencelevel sentiment analysis. Much of the existing research on this topic focused on identifying the polarity of a sentence (e.g. positive, negative, neutral) based on the language clues extracted from the textual content of sentences ( Liu, 2012;Pang & Lee, 2004;Turney, 2002 ). They solved this task as a general problem without considering different sentence types. However, different types of sentences express sentiment in very different ways. For example, for the sentence "It is good. ", the sentiment polarity is definitely . Others dealt with mining opinions in comparative sentences, which is to determinate the degree of positivity surround the analysis of comparative sentences ( Ganapathibhotla & Liu, 2008;Jindal & Liu, 2006b;Yang & Ko, 2011 ). There has also been work focusing on sentiment analysis of conditional sentences ( Narayanan et al., 2009 ), or sentences with modality, which have some special characteristics that make it hard for a system to determine sentiment orientations ( Liu, Yu, Chen, & Liu, 2013 ).
In this paper, we propose a different way in dealing with different sentence types. In particular, we investigate the relationship between the number of opinion targets expressed in a sentence and the sentiment expressed in this sentence; propose a novel framework for improving sentiment analysis via sentence type classification. Opinion target (hereafter, target for short) can be any entity or aspect of the entity on which an opinion has been expressed ( Liu, 2015 ). An opinionated sentence can express sentiments without a mention of any target, or towards one target, two or more targets. We define three types of sentences: non-target sentences, one-target sentences and multi-target sentences , respectively. Consider the following examples from the movie review sentence polarity dataset v1.0 (hereafter, MR dataset for short) ( Pang & Lee, 2005 )  Example 2. If you sometimes like to go to the movies to have fun, Wasabi is a good place to start.
Example 3. Director Kapur is a filmmaker with a real flair for epic landscapes and adventure, and this is a better film than his earlier English-language movie, the overpraised Elizabeth.
Example 1 is a non-target sentence. In order to infer its target, we need to know its context. Example 2 is a one-target sentence, in which the sentiment polarity of the target Wasabi is positive. Example 3 is a multi-target sentence, in which there are three targets: Director Kapur, film and his earlier English-language movie, the overpraised Elizabeth . We can observe that sentences tend to be more complex with more opinion targets, and sentiment detection is more difficult for sentences containing more targets.
Based on this observation, we apply a deep neural network sequence model, which is a bidirectional long short-term memory with conditional random fields (henceforth BiLSTM-CRF) ( Lample, Ballesteros, Subramanian, Kawakami, & Dyer, 2016 ), to extract target expressions in opinionated sentences. Based on the targets extracted, we classify sentences into three groups: non-target, onetarget and multi-target. Then, one-dimensional convolutional neural networks (1d-CNNs) ( Kim, 2014 ) are trained for sentiment classification on each group separately. Finally, the sentiment polarity of each input sentence is predicted by one of the three 1d-CNNs.
The main contributions of our work are summarized below: • We propose a novel two-step pipeline framework for sentencelevel sentiment classification by first classifying sentences into different types based on the number of opinion targets they contain, and then training 1d-CNNs separately for sentences in each type for sentiment detection; • While conventional sentiment analysis methods largely ignore different sentence types, we have validated in our experiments that learning a sentiment classifier tailored to each sentence type would result in performance gains in sentence-level sentiment classification.
The rest of this article is organized as follows: we review related work in Section 2 ; and then present our approach in Section 3 ; experimental setup, evaluation results and discussions are reported in Section 4 ; finally, Section 5 concludes the paper and outlines future research directions.

Sentence type classification for sentiment analysis
Since early 20 0 0, sentiment analysis has grown to be one of the most active research areas in natural language processing (NLP) ( Liu, 2015;Ravi & Ravi, 2015 ). It is a multifaceted problem with many challenging and interrelated sub-problems, including sentence-level sentiment classification. Many researchers realized that different type of sentence need different treatment for sentiment analysis. Models of different sentence types, including subjective sentences, target-dependent sentences, comparative sentences, negation sentences, conditional sentences, sarcastic sentences, have been proposed for sentiment analysis.
Subjectivity classification distinguishes sentences that express opinions (called subjective sentences) from sentences that express factual information (called objective sentences) ( Liu, 2015 ). Although some objective sentences can imply sentiments or opinions and some subjective sentences may not express any opinion or sentiment, many researchers regard subjectivity and sentiment as the same concept ( Hatzivassiloglou & Wiebe, 20 0 0;Wiebe et al., 1999 ), i.e., subjective sentences express opinions and objective sentences express fact. Riloff and Wiebe (2003) presented a bootstrapping process to learn linguistically rich extraction patterns for subjective expressions from a large unannotated data. Rill, Reinel, Scheidt, and Zicari (2014) presented a system to detect emerging political topics on twitter and the impact on concept-level sentiment analysis. Appel, Chiclana, Carter, and Fujita (2016) proposed a hybrid approach using SentiWordNet ( Baccianella, Esuli, & Sebastiani, 2010 ) and fuzzy sets to estimate the semantic orientation polarity and intensity of sentiment words, before computing the sentence level sentiments. Muhammad, Wiratunga, and Lothian (2016) introduced a lexicon-based sentiment classification system for social media genres, which captures contextual polarity from both local and global context. Fernández-Gavilanes, Álvarez-López, Juncal-Martínez, Costa-Montenegro, and González-Castaño (2016) proposed a novel approach to predict sentiment in online texts based on an unsupervised dependency parsing-based text classification method.
Most previous target related works assumed targets have been given before performing sentiment classification ( Dong et al., 2014;Jiang et al., 2011;Mitchell et al., 2013;. Little research has been conducted on classifying sentence by the target number although there is a large body of work focusing on opinion target extraction from text. A comparative opinion sentence expresses a relation of similarities or differences between two or more entities and/or a preference of the opinion holder based on some shared aspects of the entities. Jindal and Liu (2006a) showed that almost every comparative sentence had a keyword (a word or phrase) indicating comparison, and identified comparative sentences by using class sequential rules based on human compiled keywords as features for a naive Bayes classifier. Ganapathibhotla and Liu (2008) reported they were the first work for mining opinions in comparative sentences. They solved the problem by using linguistic rules and a large external corpus of Pros and Cons from product reviews to determine whether the aspect and sentiment context were more associated with each other in Pros or in Cons. Kessler and Kuhn (2014) presented a corpus of comparison sentences from English camera reviews. Park and Yuan (2015) proposed two linguistic knowledge-driven approaches for Chinese comparative elements extraction.
Negation sentences occur fairly frequently in sentiment analysis corpus. Many researchers considered the impact of negation words or phrases as part of their works ( Hu & Liu, 2004;Pang, Lee, & Vaithyanathan, 2002 ); a few researchers investigated negation words identification and/or negative sentence processing as a single topic. Jia, Yu, and Meng (2009) studied the effect of negation on sentiment analysis, including negation term and its scope identification, by using a parse tree, typed dependencies and special linguistic rules. Zhang, Ferrari, and Enjalbert (2012) proposed a compositional model to detect valence shifters, such as negations, which contribute to the interpretation of the polarity and the intensity of opinion expressions. Carrillo-de Albornoz and Plaza (2013) studied the effect of modifiers on the emotions affected by negation, intensifiers and modality.
Conditional sentences are another commonly used language constructs in text. Such a sentence typically contains two clauses: the condition clause and the consequent clause. Their relationship has significant impact on the sentiment orientation of the sentence ( Liu, 2015 ). Narayanan et al. (2009) first presented a linguistic analysis of conditional sentences, and built some supervised learning models to determine if sentiments expressed on different topics in a conditional sentence are positive, negative or neutral. Liu (2015) listed a set of interesting patterns in conditional sentences that often indicate sentiment, which was particularly useful for reviews, online discussions, and blogs about products.
Sarcasm is a sophisticated form of speech act widely used in online communities. In the context of sentiment analysis, it means that when one says something positive, one actually means negative, and vice versa. Tsur, Davidov, and Rappoport (2010) presented a novel semi-supervised algorithm for sarcasm identification that recognized sarcastic sentences in product reviews. González-Ibánez, Muresan, and Wacholder (2011) reported on a method for constructing a corpus of sarcastic Twitter messages, and used this corpus to investigate the impact of lexical and pragmatic factors on machine learning effectiveness for identifying sarcastic utterances. Riloff et al. (2013) presented a bootstrapping algorithm for sarcasm recognition that automatically learned lists of positive sentiment phrases and negative situation phrases from sarcastic tweets.
Adversative and concessive structures, as another kind of linguistical feature, are constructions express antithetical circumstances ( Crystal, 2011 ). A adversative or a concessive clause is usually in clear opposition to the main clause about the fact or event commented. Fernández-Gavilanes et al. (2016) treated the constructions as an extension of intensification propagation, where the sentiment formulated could be diminished or intensified, depending on both adversative/concessive and main clauses. Hu and Liu (2004) used frequent nouns and noun phrases as feature candidates for opinion target extraction. Qiu, Liu, Bu, and Chen (2011) proposed a bootstrapping method where a dependency parser was used to identify syntactic relations that linked opinion words and targets for opinion target extraction. Popescu and Etzioni (2005) considered product features to be concepts forming certain relationships with the product and sought to identify the features connected with the product name by computing the point wise mutual information (PMI) score between the phrase and class-specific discriminators through a web search. Stoyanov and Cardie (2008) treated target extraction as a topic coreference resolution problem and proposed to train a classifier to judge if two opinions were on the same target. Liu, Xu, and Zhao (2014) constructed a heterogeneous graph to model semantic relations and opinion relations, and proposed a co-ranking algorithm to estimate the confidence of each candidate. The candidates with higher confidence would be extracted as opinion targets. Poria, Cambria, and Gelbukh (2016) presented the first deep learning approach to aspect extraction in opinion mining using a 7-layer CNN and a set of linguistic patterns to tag each word in sentences. Mitchell et al. (2013) modeled sentiment detection as a sequence tagging problem, extracted named entities and their sentiment classes jointly. They referred this kind of approach open domain targeted sentiment detection. Zhang, Zhang, and Vo (2015) followed Mitchell et al.'s work, studied the effect of word embeddings and automatic feature combinations on the task by extending a CRF baseline using neural networks.

Deep learning for sentiment classification
Deep learning approaches are able to automatically capture, to some extent, the syntactic and semantic features from text without feature engineering, which is labor intensive and time consuming. They attract much research interest in recent years, and achieve state-of-the-art performances in many fields of NLP, including sentiment classification. Socher et al. (2011) introduced semi-supervised recursive autoencoders for predicting sentiment distributions without using any pre-defined sentiment lexica or polarity shifting rules. Socher et al. (2013) proposed a family of recursive neural network, including recursive neural tensor network (RNTN), to learn the compositional semantic of variable-length phrases and sentences over a human annotated sentiment treebank. Kalchbrenner et al. (2014) and Kim (2014) proposed different CNN models for sentiment classification, respectively. Both of them can handle the input sentences with varying length and capture short and longrange relations. Kim (2014) 's model has little hyper parameter tuning and can be trained on pre-trained word vectors. Irsoy and Cardie (2014a) presented a deep recursive neural network (DRNN) constructed by stacking multiple recursive layers for compositionality in Language and evaluated the proposed model on sentiment classification tasks. Tai, Socher, and Manning (2015) introduced a tree long short-term memory (LSTM) for improving semantic representations, which outperforms many existing systems and strong LSTM baselines on sentiment classification. Tang et al. (2015c) proposed a joint segmentation and classification framework for sentence-level sentiment classification. Liu, Qiu, and Huang (2016) used a recurrent neural network (RNN) based multitask learning framework to jointly learn across multiple related tasks. Chaturvedi, Ong, Tsang, Welsch, and Cambria (2016) proposed a deep recurrent belief network with distributed time delays for learning word dependencies in text which uses Gaussian networks with time-delays to initialize the weights of each hidden neuron. Tang, Qin, and Liu (2015b) gave a survey on this topic.  Table 1 An example sentence with labels in IOB format. The target is the act , the label B indicates the beginning of a target, I indicates that the word is inside a target, and O indicates a word belongs to no target.

Words:
Yet the act is still charming here .

Methodology
We present our approach for improving sentiment analysis via sentence type classification in this section. An overview of the approach is shown in Fig. 1 . We first introduce the BiLSTM-CRF model which extracts target expressions from input opinionated sentences, and classifies each sentence according to the number of target explicitly expressed in it ( Section 3.1 ). Then, we describe the 1d-CNN sentiment classification model which predicts sentiment polarity for non-target sentences, one-target sentences and multi-target sentences, separately ( Section 3.2 ).

Sequence model for sentence type classification
We describe our approach for target extraction and sentence type classification with BiLSTM-CRF. Target extraction is similar to the classic problem of named entity recognition (NER), which views a sentence as a sequence of tokens usually labeled with IOB format (short for Inside, Outside, Beginning). Table 1 shows an example sentence with the appropriate labels in this format.
Deep neural sequence models have shown promising success in NER ( Lample et al., 2016 ), sequence tagging ( Huang, Xu, & Yu, 2015 ) and fine-grained opinion analysis ( Irsoy & Cardie, 2014b ). BiLSTM-CRF is one of deep neural sequence models, where a bidirectional long short-term memory (BiLSTM) layer ( Graves, Mohamed, & Hinton, 2013 ) and a conditional random fields (CRF) layer ( Lafferty, McCallum, & Pereira, 2001 ) are stacked together for sequence learning, as shown in Fig. 2 . BiLSTM incorporates a forward long short-term memory (LSTM) layer and a backward LSTM layer in order to learn information from preceding as well as following tokens. LSTM ( Hochreiter & Schmidhuber, 1997 ) is a kind of recurrent neural network (RNN) architecture with long shortterm memory units as hidden units. Next we briefly describe RNN, L STM, BiL STM and BiLSTM-CRF.
RNN ( Elman, 1990 ) is a class of artificial neural sequence model, where connections between units form a directed cycle. It takes arbitrary embedding sequences x = (x 1 , . . . , x T ) as input, uses its internal memory network to exhibit dynamic temporal behavior. It consisting of a hidden unit h and an optional output y. T is the last time step. It is also the length of input sentence in this text sequence learning task. At each time step t , the hidden state h t of the RNN is computed based on the previous hidden state h t−1 and the input at the current step x t : where U and W are weight matrices of the network; g( · ) is a non-linear activation function, such as an element-wise logistic sigmoid function. The output at time step t is computed as where V is another weight parameter of the network, softmax is an activation function often implemented at the final layer of a network.
LSTM is a variant of RNN designed to deal with vanishing gradients problem ( Hochreiter & Schmidhuber, 1997 ). The LSTM used in the BiLSTM-CRF ( Lample et al., 2016 ) has two gates (an input gate i t , an output gate o t ) and a cell activation vectors c t .
BiL STM uses two L STMs to learn each token of the sequence based on both the past and the future context of the token. As shown in Fig. 2 , one LSTM processes the sequence from left to right, the other one from right to left. At each time step t , a hidden forward layer with hidden unit function − → h is computed based on the previous hidden state − → h t−1 and the input at the current step x t and a hidden backward layer with hidden unit function ← − h is computed based on the future hidden state ← − h t+1 and the input at the current step x t . The forward and backward context representations, generated by − → h t and ← − h t respectively, are concatenated into a long vector. The combined outputs are the predictions of teacher-given target signals.
As another widely used sequence model, conditional random fields (CRF) is a type of discriminative undirected probabilistic graphical model, which represents a single log-linear distributions over structured outputs as a function of a particular observation input sequence.
Given observations variables X whose values are observed, random variables Y whose values the task requires the model to predict, and a undirected graph G where Y are connected by undirected edges indicating dependencies. CRF defines the conditional probability of a set of output values y ∈ Y given a set of input values x ∈ X to be proportional to the product of potential functions on cliques of the graph ( McCallum, 2003 ), where Z x is a normalization factor overall output values, S ( y, x ) is the set of cliques of G, s ( y s , x s ) is the clique potential on clique s .
Afterwards, in the BiLSTM-CRF model, a softmax over all possible tag sequences yields a probability for the sequence y . The prediction of the output sequence is computed as follows: where σ ( X, y ) is the score function defined as follows: where A is a matrix of transition scores, A y i ,y i +1 represents the score of a transition from the tag y i to y i +1 . n is the length of a sentence, P is the matrix of scores output by the BiLSTM network, P i,y i is the score of the y th i tag of the i th word in a sentence. As shown in Fig. 2 , dropout technique is used after the input layer of BiLSTM-CRF to reduce overfitting on the training data. This technique is firstly introduced by Hinton, Srivastava, Krizhevsky, Sutskever, and Salakhutdinov (2012) for preventing complex coadaptations on the training data. It has given big improvements on many tasks.
After target extraction by BiLSTM-CRF, all opinionated sentences are classified into non-target sentences, one-target sentences and multi-target sentences, according to the number of targets extracted from them.

1d-CNN for sentiment classification on each sentence type
1d-CNN, firstly proposed by Kim (2014) , takes sentences of varying lengths as input and produces fixed-length vectors as output. Before training, word embeddings for each word in the glossary of all input sentences are generated. All the word embeddings are stacked in a matrix M . In the input layer, embeddings of words comprising current training sentence are taken from M . The maximum length of sentences that the network handles is set. Longer sentences are cut; shorter sentences are padded with zero vectors. Then, dropout regularization is used to control over-fitting.
In the convolution layer, multiple filters with different window size move on the word embeddings to perform one-dimensional convolution. As the filter moves on, many sequences, which capture the syntactic and semantic features in the filtered n -gram, are generated. Many feature sequences are combined into a feature map. In the pooling layer, a max-overtime pooling operation ( Collobert et al., 2011 ) is applied to capture the most useful local features from feature maps. Activation functions are added to incorporate element-wise non-linearity. The outputs of multiple filters are concatenated in the merge layer. After another dropout process, a fully connected softmax layer output the probability distribution over labels from multiple classes.
CNN is one of most commonly used connectionism model for classification. Connectionism models focus on learning from environmental stimuli and storing this information in a form of connections between neurons. The weights in a neural network are adjusted according to the training data by some learning algorithm. It is the greater the difference in the training data, the more difficult for the learning algorithm to adapt the training data, and the worse classification results. Dividing opinionated sentences into different types according to the number of targets expressed in them can reduce the differences of training data in each group, therefore, improve overall classification accuracy.

Experiment
We conduct experiments to evaluate the performance of the proposed approach for sentence-level sentiment classification on various benchmarking datasets. In this section, we describe the experimental setup and baseline methods followed by the discussion of results.

Experimental setup
For training BiLSTM-CRF for target extraction and sentence type classification, we use the MPQA opinion corpus v2.0 (MPQA dataset for short) provided by Wiebe, Wilson, and Cardie (2005) 4 since it contains a diverse range of sentences with various numbers of opinion targets. It contains 14,492 sentences from a wide variety of news sources manually annotated with opinion target at the phrase level (7,026 targets). All the sentences are used to train BiLSTM-CRF.
For sentiment classification with 1d-CNN, we test our approach on different datasets: • MR : Movie review sentence polarity dataset v1.0. It contains 5331 positive snippets and 5331 negative snippets extracted from Rotten Tomatoes web site pages where reviews marked with "fresh" are labeled as positive, and reviews marked with "rotten" are labeled as negative. 10-fold cross validation was used for testing. • SST-1 : Stanford sentiment treebank contains 11,855 sentences also extracted from the original pool of Rotten Tomatoes page files. These sentences are split into 8544/1101/2210 for train/dev/test. Each of them is fine-grained labeled (very positive, positive, neutral, negative, very negative). • SST-2 : Binary labeled version of Stanford sentiment treebank, in which neutral reviews are removed, very positive and positive reviews are labeled as positive, negative and very negative reviews are labeled as negative ( Kim, 2014 ). It contains 9613 sentences split into 6920/872/1821 for train/dev/test. • CR : Customer reviews of 5 digital products contains 3771 sentences extracted from amazon.com, including 2405 positive sentences and 1366 negative sentences. 10-fold cross validation was used for testing.
Following Kim (2014) 's work, we use accuracy as the evaluation metric to measure the overall sentiment classification performance.
During training a BiLSTM-CRF for target extraction in a sentence, the input sequence x t is set to the t -th word embedding (a distributed representation for a word ( Bengio, Ducharme, Vincent, & Jauvin, 2003 )) in a input sentence. Publicly available word vectors trained from Google News 5 are used as pre-trained word embeddings. The size of these embeddings is 300. U, W, V and h 0 are initialized to a random vector of small values, h t+1 are initialized to a copy of h t recursively. A back-propagation algorithm with Adam stochastic optimization method is used to train the network through time with learning rate of 0.05. After each training epoch, the network is tested on validation data. The log-likelihood of validation data is computed for convergence detection.
For training CNN, we use: CNN-non-static model, ReLU as activation function, Adadelta decay parameter of 0.95, dropout rate of 0.5, the size of initial word vectors of 300. We use different filter windows and feature maps for different tar get classes. For nontarget sentences, we use filter windows of 3, 4, 5 with 100 feature maps each; For one-target sentences, we use filter windows of 3, 4, 5, 6 with 100 feature maps each; For multi-target sentences, we use filter windows of 3, 4, 5, 6, 7 with 200 feature maps each.

Baseline methods
We benchmark the following baseline methods for sentencelevel sentiment classification, some of them have been previously used in Kim (2014) : • MNB : Multinomial naive Bayes with uni-bigrams. • NBSVM : SVM variant using naive Bayes log-count ratios as feature values proposed by Wang and Manning (2012) . • Tree-CRF : Dependency tree based method for sentiment classification using CRF with hidden variables proposed by Nakagawa et al. (2010) . • RAE : Semi-supervised recursive autoencoders with pre-trained word vectors from Wikipedia proposed by Socher et al. (2011) . • MV-RNN : Recursive neural network using a vector and a matrix on every node in a parse tree for semantic compositionality proposed by Socher, Huval, Manning, and Ng (2012) . • RNTN : Recursive deep neural network for semantic compositionality over a sentiment treebank using tensor-based feature function proposed by Socher et al. (2013) . • Paragraph-Vec : An unsupervised algorithm learning distributed feature representations from sentences and documents proposed by Le and Mikolov (2014) . • DCNN : Dynamic convolutional neural network with dynamic kmax pooling operation proposed by Kalchbrenner et al. (2014) . • CNN-non-static : 1d-CNN with pre-trained word embeddings and fine-tuning optimizing strategy proposed by Kim (2014) . • CNN-multichannel : 1d-CNN with two sets of pre-trained word embeddings proposed by Kim (2014) . • DRNN : Deep recursive neural networks with stacked multiple recursive layers proposed by Irsoy and Cardie (2014a) . • Multi-task LSTM : A multi-task learning framework using LSTM to jointly learn across multiple related tasks proposed by Liu et al. (2016) . • Tree LSTM : A generalization of LSTM to tree structured network topologies proposed by Tai et al. (2015) . • Sentic patterns : A concept-level sentiment analysis approach using dependency-based rules proposed by Poria, Cambria, Winterstein, and Huang (2014) .

Qualitative evaluations
In Table 2 , we show the example sentences in each target class of Stanford sentiment treebank. It is observed that many nontarget sentences are small imperative sentences, have direct subjective expressions (DSEs) which consist of explicit mentions of private states or speech events expressing private states ( Irsoy & Cardie, 2014b ), e.g., funny and enjoyable in s 1, dark and disturbing in s 2. For some non-target sentences, it is difficult to detect its sentiment without context, e.g., it is unclear whether the word shower in s 3 conveys positive or negative sentiment. Non-target sentences tend to be short comparing with two other types of sentences. Many one-target sentences are simple sentences, which contain basic constituent elements forming a sentence. The subject is mostly the opinionated target in a one-target sentence, e.g., the act in s 4, Mr. Ratliff in s 5 and C.H.O. in s 6. Almost all the multitarget sentences are compound/complex/compound-complex sentences, which have two or more clauses, and are very complex in expressions. Many of them have coordinating or subordinating conjunctions, which make it difficult to identify the sentiment of a whole sentence, e.g., but in s 7, because and and in s 8, while in s 9. Table 3 Experimental results of sentiment classification accuracy. % is omitted. The best results are highlighted in bold face. The results of the top 10 approaches have been previously reported by Kim (2014) . The top 3 approaches are conventional machine learning approaches with handcrafted features. Sentic patterns is rule based approach. Other 11 approaches, including our approach, are deep neural network (DNN) approaches, which can automatically extract features from input data for classifier training without feature engineering. Overall, as the result of the qualitative evaluations, the difficulty degree of sentiment classification on each sentence type is T 2+ > T 1 > T 0 , i.e., multi-target sentences are most difficult, while non-target sentences are much easier for sentiment classification. The experimental results listed in the next subsection validate this observation. Table 3 shows the results achieved on the MR, SST-1, SST-2 and CR datasets. It is observed that comparing with three handcrafted features based methods, although RAE and MV-RNN perform worse on MR dataset, two CNN based methods gives better results on both MR and CR datasets. This indicates the effectiveness of DNN approaches. Among 11 DNN approaches, our approach outperforms other baselines on all the datasets except SST-1, i.e., our approach gives relative improvements of 0.98% compared to CNN-non-static on MR dataset, 0.23% and 0.47% relative improvements compared to CNN-multichannel on SST-2 and CR dataset, respectively. Comparing with two CNN based methods, our sentence type classification based approach gives superior performance on all the four datasets (including SST-1 dataset). These validate the influences of sentence type classification in terms of sentence-level sentiment analysis. Table 4 shows the statistics and comparison of each target class on the MR, SST-1, SST-2 and CR datasets. The relative improvement ratio Δ calculates as follows:

Comparison on each target class
It is obvious that the performance for every target class is improved using sentence type classification. Yet, the improvement for the multi-target sentences (T2+) is more significant than other two target classes on three of the four dataset, e.g. the relative improvement ratio of T2+ class on the SST-1 and CR datasets are 4.75% and 5.26%, respectively, which are about twice higher than the relative improvement ratio of T1 class. Table 4 is a clear indication that the proposed sentence type classification based sentiment classification approach is very effective for complex sentences. Both the Acc CNN and Acc our use 1d-CNN (non-static) and pretrained Google News word embedding, our approach achieves bet- Table 4 The class-by-class classification results using sentence type classification as well as without using sentence type classification on the four datasets. #train and #test are the word number of sentences in training and test dataset, respectively; l max and l avg are max and average word length of sentences, respectively; Acc CNN is the experimental result that we do sentiment classification directly on the four datasets using 1d-CNN (non-static) without sentence type classification, and statistic the accuracy on each target class the same with the target class recognized by BiLSTM-CRF. Acc our is the experimental result of our approach on each target class, which using both sentence type classification and 1d-CNN (non-static). Δ is the relative improvement ratio calculates. In the Acc CNN , Acc our and Δ columns, % is omitted for conciseness. ter performance because the divide-and-conquer approach, which first classifies sentences into different types, then optimize the sentiment classifier separately on sentences from each type.

Comparison with different sequence models
We have also experimented with different sequence models, including CRF, LSTM, BiRNN ( Schuster & Paliwal, 1997 ), BiRNN-CRF, BiL STM and BiL STM-CRF, for sentence type classification. For CRF, we use CRFSuite ( Okazaki, 2007 ) with word, Part-Of-Speech tag, prefix, suffix and a sentiment dictionary as features. For LSTM, BiRNN, BiRNN-CRF and BiLSTM, we also use Google News word embeddings as pre-trained word embeddings. For other parameters, we use default parameter settings. Table 5 shows the experimental results on the MR, SST-1, SST-2 and CR datasets. It can be observed that BiLSTM-CRF outperforms all the other approaches on all the four datasets. It is because BiLSTM-CRF has more complicated hidden units, and offers better composition capability than other DNN approaches. CRF with hand-crafted features gives comparable performance to LSTM, but lower performance than more complex DNN models. BiRNN and BiLSTM gives better performance compared to LSTM because they can learn each token of the sequence based on both the past and the future context of the token, while LSTM only use the past context of the token. Comparing BiRNN and BiLSTM with BiRNN-CRF and BiLSTM-CRF, respectively, it is observed that combining CRF and DNN models can improve the performance of DNN approaches.

Evaluation on opinion target extraction with BiLSTM-CRF
One unavoidable problem for every multi-step approach is the propagation of errors. In our approach, we use a BiLSTM-CRF/1d-CNN pipeline for sentiment analysis. It is interesting to see how Table 6 Experimental results of target extraction with BiLSTM-CRF on SemEval16 task 5 aspect based sentiment analysis dataset subtask 1 slot 2. Best System refers to the participation system with best performance submitted to SemEval16 task 5. Baseline refers to baseline model provided by the organizers; C refers to the model only uses the provided training data; U refers to the model uses other resources (e.g., publicly lexica) and additional data for training; "-" refers to no submissions were made. % is omitted for conciseness. The best results are highlighted in bold face. We have also conducted experiments with BiLSTM-CRF using the SemEval-2016 task 5 aspect based sentiment analysis dataset ( Pontiki et al., 2016 ). There are 3 subtasks in this task, each subtask contains several slots. We have conducted experiments on subtask 1 slot 2: sentence-level opinion target expression extraction, on the restaurants domain. F1 score is used as metric. The experimental results are shown in Table 6 .
In this table, for English, the best systems are NLANG ( Toh & Su, 2016 )  It is observed that BiLSTM-CRF achieves the best performance on all the dataset using different languages, and outperforms the others by a good margin in 5 out of 6 languages. It indicates that BiLSTM-CRF is effective in opinion target expression extraction.
We have also evaluated the performance of BiLSTM-CRF on the MPQA dataset described in Section 4.1 . We randomly select 90% sentences in MPQA dataset for training and the remaining 10% sentences for testing. BiLSTM-CRF achieves 20.73 F1 score on opinion target extraction. This is due to the complex nature of the data that many opinion targets are not simple named entities such as person, organization and location in typical NER tasks. Rather, the opinion targets could be events, abstract nouns or multi-word phrases. For example, "overview of Johnson's eccentric career " in sentence "An engaging overview of Johnson 's eccentric career. ". Target number classification is much easier. It achieves 65.83% accuracy, when we classify the test sentences into 3 groups by the target numbers extracted from them. These results show that even though the performance of the first step of our approach is not very high, our pipeline approach still achieves the state-of-the-art results on most benchmarking datasets. If we can improve the performance of the sequence model for opinion target extraction, the final sentiment classification performance of our approach may be further improved.
We have also considered using other existing opinion target detection systems, which are specifically trained for this task. Unfortunately, it is not very easy to find an applicable one. Some opinion target detection systems, such as Liu et al. (2014) , can also be regard as NER models.

Error analysis for sentence type classification
We have also done error analysis for sentence type classification. In this section, we list some result examples from the Stanford sentiment treebank. The __O, __B and __I concatenated after each word are the label predicted by BiLSTM-CRF. It is observed that sentences with basic constituent elements ( Easy example 1 ), even if a litter long in length ( Easy example 2 ), are relatively easier for target extraction with BiLSTM-CRF. One reason is that in these two sentences, the targets ( the art and the movie ) are commonly used nouns; Another reason is that the MPQA dataset, used for training BiLSTM-CRF model, is obtained from news sources. News text is usually more structured than the text from other sources, such as web reviews. Small imperative sentence ( Easy example 3 ) is also relatively easier for target extraction, because many of them are non-target sentences.
Sentences containing pronouns, such as you and it in Difficult example 1 and this in Difficult example 2 , are relatively more difficult for target extraction with BiLSTM-CRF. Moreover, complex target, such as overview of Johnson's eccentric career in Difficult example 3 , is also very difficult. We have also analyzed examples in which BiLSTM-CRF detects opinion targets better than CRF. As shown above, CRF can only identify a partial opinion target ( faces ), while BiLSTM-CRF can identify the whole opinion target more accurately ( their computeranimated faces ).

Conclusion
This paper has presented a novel approach to improve sentence-level sentiment analysis via sentence type classification. The approach employs BiLSTM-CRF to extract target expression in opinionated sentences, and classifies these sentences into three types according to the number of targets extracted from them. These three types of sentences are then used to train separate 1d-CNNs for sentiment classification. We have conducted extensive experiments on four sentence-level sentiment analysis datasets in comparison with 11 other approaches. Empirical results show that our approach achieves state-of-the-art performance on three of the four datasets. We have found that separating sentences containing different opinion targets boosts the performance of sentence-level sentiment analysis.
In future work, we plan to explore other sequence learning models for target expression detection and further evaluate our approach on other languages and other domains.