BUEES: a bottom-up event extraction system

Traditional event extraction systems focus mainly on event type identification and event participant extraction based on pre-specified event type paradigms and manually annotated corpora. However, different domains have different event type paradigms. When transferring to a new domain, we have to build a new event type paradigm and annotate a new corpus from scratch. This kind of conventional event extraction system requires massive human effort, and hence prevents event extraction from being widely applicable. In this paper, we present BUEES, a bottom-up event extraction system, which extracts events from the web in a completely unsupervised way. The system automatically builds an event type paradigm in the input corpus, and then proceeds to extract a large number of instance patterns of these events. Subsequently, the system extracts event arguments according to these patterns. By conducting a series of experiments, we demonstrate the good performance of BUEES and compare it to a state-of-the-art Chinese event extraction system, i.e., a supervised event extraction system. Experimental results show that BUEES performs comparably (5% higher F-measure in event type identification and 3% higher F-measure in event argument extraction), but without any human effort.


Introduction
Information Extraction (IE) is the task of identifying factual description (entities, relations and events) from unstructured natural language text and extracting information related to those descriptions Grishman (1997).Event extraction remains the most challenging task, because a larger field of view is often needed to understand how facts tie together, and it is situated at the end of an IE pipeline thus suffers from propagation of errors from word segmentation, named entity recognition, coreference resolution, etc. Ahn (2006).Although event extraction is a challenging problem, it has been widely used in several different specific domains, such as musical reports Xiao et al. (2011), financial analysis Lee et al. (2003), biomedical investigation Pham et al. (2013) and legal documents Schilder (2007).The main approaches used by most event extraction systems are based on knowledge engineering technology or machine learning technology.The knowledge engineering based event extraction systems use extraction patterns or rules to identify and extract the relevant information Riloff (1996); Soderland (1999); Yangarber et al. (2000).Most of these systems use annotated training data to learn pattern matching rules, based on lexical, syntactic, or semantic information.These systems traditionally were the top performers in most event extraction benchmarks, such as MUC Chinchor et al. (1993) and ACE Yeh et al. (2002).In the machine learning approach, domain experts label instances of the target concepts in a set of documents Miwa et al. (2010); Hong et al. (2011); Ritter et al. (2012).The system then learns a model of the extraction task, which can be applied to new documents automatically.
Both of these approaches require substantial hu-man effort, and hence prevent event extraction system become domain adaptation and more widely applicable.Recently, there appear many semisupervised event extraction systems that aim to reduce the annotated data required, ideally to a set of seed instances of the target events.One such system is Liao and Grishman Liao and Grishman (2010).They use two state-of-the-art bootstrapping-based event extraction systems, then rank their candidate patterns and accept the top-ranked patterns in each iteration.
However, for all such approaches it is still necessary to specify the target events in advance.In this paper, we explore the possibility to construct a completely unsupervised and bottom-up event extraction system, which does not need to pre-specify the target event types.
Although the task is important and emergent, the challenge of it at least lies into two aspects.
• How to automatically build event type paradigm.As pre-specifying interested event types in a domain needs rich background knowledge, event type paradigm is traditionally built by domain experts.It is a costly work.
• How to extract event arguments in a totally unsupervised way is a challenging problem.All of event extraction systems reported in the literature more or less need manual efforts.
To address the above challenges, in this paper, we design and develop a bootstrappingbased BUEES (a Bottom-Up Event Extraction System).The system automatically builds event type paradigm from scratch, based on the definition of event trigger: the words that most clearly expresses an event's occurrence, and our key observations: triggers are the most important lexical units to represent events.A set of triggers with similar meaning or usage represents the same event type.Event types can be discovered based on trigger clustering.
When the target event types are available, the next step is to learn a set of patterns to extract event arguments from the web documents.The system takes as input a small set of event seeds.It then uses these seeds to search the web to get more documents that contain the event seeds.The extraction patterns will be learned from these documents.Finally, we can extract event arguments by using these useful patterns.The major contributions of the work presented in this paper are as follows.
• To the best of our knowledge, our work is the first to propose the Bottom-Up Event Extraction System.We automatically build event type paradigm.Based on the paradigm, we proceed to implement traditional event extraction tasks.
• Our system is completely unsupervised.To our knowledge, all of the bootstrapping-based semisupervised event extraction systems need the manually constructed seeds in advance.However, our system can generate and select event seeds automatically.
The rest of this paper is organized as follows.Section 2 describes the task.Section 3 introduces the architecture of the system and then describes each component in detail.Section 4 evaluates the proposed method.We review related work of event extraction in Section 5, and finally conclude this paper in Section 6.

ACE Event Extraction Task
The event extraction task we addressing is that of the Automatic Content Extraction (ACE) evaluations LDC (2005), where an event is defined as a specific occurrence involving participants.Event extraction task requires that certain specified types of events should be detected.We first introduce some ACE terminology to understand this task more easily: • Entity: an object or a set of objects in one of the semantic categories of interest • Entity mention: a reference to an entity (typically, a noun phrase) • Event trigger: the main word which most clearly expresses an event occurrence • Event arguments: the entity mentions that are involved in an event • Event mention: a phrase or sentence within which an event is described, including trigger and arguments • Event type: a particular event category, such as "Conflict/Attack", "Life/Die", etc.
The ACE 2005 evaluation has 8 types of events, with 33 subtypes; for the purpose of this paper, we will treat these simply as 33 separate event types and do not consider the hierarchical structure among them.Besides, the ACE evaluation plan defines the following standards to determine the correctness of an event extraction: • A trigger is correctly labeled if its event type and offset (viz., the position of the trigger word in text) match a reference trigger.
• An argument is correctly identified if its event type and offsets match any of the reference argument mentions, in other word, correctly recognizing participants in an event.
• An argument is correctly classified if its role matches any of the reference argument mentions.
Figure 1 shows an example of an ACE event, where "born" is the trigger word.Its event type is "Life" and the subtype is "Be-Born".This event consists of three arguments, namely, "Mao Zedong", "1893", "Xiangtan, Hunan Province", which corresponds to three role labels in the Life/Be-Born event template of "Person", "Time-Within" and "Place", respectively.

New Task for Event Extraction
In addition to traditional event extraction tasks introduced above, we propose a new task of building event type paradigm.ACE manually annotates 8 types and 33 subtypes of events and construct the event type paradigm that are shown in Table 1.However, building an event type paradigm in this way not only requires massive human effort but also tends to be very data dependent.As a result, it may prevent

Description of BUEES
The goal of BUEES is to extract instances of events without any human supervision.The system is built based on the framework of bootstrapping and its architecture is shown in Figure 2. Traditional bootstrapping-based semi-supervised event extraction systems use manual construction of seed examples to learn extraction patterns, and then identify event types and recognize event arguments.Since the number of seeds is limited, the quality and coverage of seeds highly affect the performance of extraction patterns.However, in this paper, we propose to automatically construct the set of seeds (Section 3.1 -3.3) and explore a novel extraction pattern learning algorithm (Section 3.4 -3.5).
The system works in three stages.During the first stage, the system builds the event type paradigm and prepares seed instances.During the second stage, the system learns extraction patterns based on the seed set.During the third stage, the system identifies event types and extracts event arguments based on the learned patterns.In the following sections, we introduce each component of BUEES in detail.

Event Type Building and Seed Extractor
Since event trigger is the word that most clearly expresses an event's occurrence, the key idea of this paper is to automatically construct an event type paradigm by clustering event triggers.For example, in the ACE corpus, a set of event triggers {倒闭, 闭 门, 关 闭, 停 业, 解 散} ({bankrupt, shut down, close, close down, dismiss}) represents the sense of the event type "Business/End-Org".In addition to extract the event trigger in the sentence, we also extract its subject and object as the seed instance (subject, trigger, object).The event seeds are used in two ways.First, event type paradigm is built by clustering seeds.Second, the seeds are used for learning extraction patterns (refers to Section 3.4) Sudo et al. (2003) summarized three classical models for representing events.All of these three models rely on the syntactic tree structure and the trigger is specified as a predicate in this structure.In order to accurately extract event seeds, we employ the predicate-argument model Yangarber et al. (2000); Ding et al. (2013) which is based on a direct syntactic relation between a predicate and its arguments.We extract the syntactic relation for predicate-argument model by means of the HIT (Harbin Institute of Technology) Dependency Parser Che et al. (2009).Based on the predicate-argument model, we propose a seed extraction algorithm (SE).The details are shown in Algorithm 1.
Take the following sentence as an example: The HIT Chinese Dependency Parser dependencies are: → (Hunan Province-5, in-4) where each atomic formula represents a binary dependence from the governor (the first token) to the dependent (the second token).The SBV relation, which stands for the subject-predicate structure, means that the head is a predicate verb and the dependent is a subject of the predicate verb; the VOB dependency relation, which stands for the verbobject structure, means that the head is a verb and the dependent is an object of the verb; the ADV relation, which stands for the adverbial structure, means that the head is a verb and the dependent is an adverb of the verb; the POB relation, which stands for the prep-object structure, means that the head is an object and the dependent is a preposition of the object.
Since V SBV = V VOB = V t = 出生(born) in this case, based on the predicate-argument model, the word "出生(born)" should be extracted as a candidate event trigger and (Mao Zedong, born, Xiangtan Hunan) should be extracted as a candidate event seed instance.

Input:
Raw corpus D Output: Candidate seeds 1: for document d in raw corpus D do end for 13: end for

Seed Cluster
As we discuss above, a set of triggers with the same meaning and usage represents the same event type.We propose to cluster event seeds (i.e., event trigger and its corresponding subject and object) based on their semantic distances, and each of these clusters represents a type of event.Details are shown in Algorithm 2.
For every two seeds p i and p j in Algorithm 2, the similarity function Sim(p i , p j ) is calculated using semantic information provided by HowNet Dong and Dong (2006) as. (1) where S s and O s denotes the number of identical sememes in the DEF s (the concept definition in HowNet) of Sub i and Sub j , Obj i and Obj j ; Si and Sj denote the number of sememes in the DEF s of Sub i and Sub j , respectively; O i and O j denote the number of sememes in the DEF s of Obj i and Obj j , respectively.HowNet uses sememes to interpret concepts.Sememes are regarded as the basic unit of the meaning.For example, "paper" can be viewed as a concept, and its sememes are "white", "thin", "soft", "flammable", etc.A group of event seeds are aggregated to a seed cluster according to their semantic distance, and we view each seed cluster as one kind of event type.Then all these event types are finally employed to construct an event type paradigm.

Algorithm 2 Seed cluster algorithm (SC)
Input: Candidate seeds (sub, Vt, obj) the Threshold θ Output: Event clusters EC 1: EC ← [ ] 2: for seed p in the set of seeds P do 3: Compute the similarity (Sim) between p and the rest of other seeds, using function 1 and 2

4:
if Sim ≥ θ then end if 10: end for

Seed Filter
Although we obtain some useful candidate seeds, certain meaningless candidate seeds come along in the results of the seed extractor as well.Therefore, we introduce a seed filter which uses heuristic rule and ranking algorithm to filter out these less informative antecedent candidates.
Since event trigger words are extracted based on the predicate-argument model, most of these candidate trigger words are verb terms.However, not all of verb terms can be used as trigger words.For example, the copular verb (e.g."is") rarely acts as the event trigger.To investigate which categories of verbs can serve as event triggers, we classify Chinese verbs into eight subclasses listed in Table 2.Such classification makes each subclass function as one grammatical role.For example, a modal verb will never be the predicate of a sentence and a nominal verb will always function as a noun.
We perform the verb sub-classification model based on the work by Liu et al. (2007)  Figure 3 The procedure of pattern learning sure the accuracy of event seed instances, we stress that the trigger word in candidate event seed must be general verb or nominal verb.

Instance Collector and Pattern Learner
The Instance Collector is currently implemented in a very simple way.It takes as input event seeds, and then a search engine is used to get documents that contain at least one of the seed words.
Figure 3 depicts the general procedure of pattern learning.Firstly, the collected sentences are used to generate the event instances which are tagged with part of speech (POS) tagging and named entity (NE) tagging.Then the NE labels are replaced by [SLOT] marks.
For example, assuming there is a seed (Mao Zedong, born, Xiangtan Hunan), we can get the related sentence from search engine as shown in Example 1.The event instance patterns are generated as mentioned above.

Soft-Pattern Learner
In order to improve the generalization ability of the extraction pattern, we employ soft-pattern of sequential pattern as the final output of the pattern learner.
In our system, the soft-pattern is constituted by the following symbols.
Soft-Patterns are generalized from the set of event instance patterns.We exploit Soft-Pattern Learning algorithm (SPL) as shown in Algorithm 3. end for 10: end for The core algorithm in Generalization function is Best Match algorithm Friedman et al. (1977) which is based on LCS (Longest Common Sequence) algorithm Hirschberg (1977).We modify LCS algorithm with matching cost: (1) Two match units are identical: cost=0; (2) Two match units share the same NE type but different NE value: cost=5; (3) If both of two match units are noun, verb, adjective or adverb, compare their labels in a thesaurus TongYiCi CiLin Jiaju et al. (1983) (expansion version)2 .If there are some overlapping labels between two match units, the cost is 5, else cost=10; (4) No match of two units: cost=10.
After the best match is found, the event instance is converted into a soft-pattern by copying matched identical elements, adding skips and slots.For example, assume that we get the following two sentences by feeding the event seed into the search engine.

周杰伦本年度新专辑《魔杰座》将于10月9日全 亚洲同步发行。
→ This year, Jay Chou's new album "Mo Jie Zuo" will be released all over the Asia on October 9th.Event instances can be generated as follows by using the approach introduced in Section 3.4.
The best match can be found based on the LCS algorithm.Then, the soft-pattern can be generated as follows.
Note that the numbers ("Eb28A01") before "新" etc. are synonym labels from TongYiCi CiLin (expansion version).According to the costs defined above, the Soft-Pattern Learning algorithm is able to find the best generalization of any two event instance patterns.The example of the Best Match algorithm is shown in Table 3.

Experiments
To evaluate the effectiveness of our BUEES, we design a set of experiments.We first evaluate the performance of the proposed event type paradigm building approach.Then, we compare our event extraction approach with a state-of-the-art baseline.

Data Description
We use ACE 2005 corpus for our experiment which is totally the same as the baseline system.The corpus contains 633 Chinese documents which are categorized by three genres: Newswire, Broadcast News and Weblog.We randomly select 558 documents for event type paradigm building and 66 documents as test set for event extraction.We use ACE 2005 event type paradigm as the gold standard paradigm to evaluate our proposed approach.
To evaluate the robust of our approach, we also use two specific domain data sets: Financial News3 and Musical News4 , collected by ourselves.The domain specific corpus contains 6000 sentences from financial news and 6000 sentences from musical news, respectively.We carefully conducted user studies into two specific domain corpora.For each sentence in the data, two annotators were asked to label and cluster all potential triggers.The agreement between our two annotators, measured using Cohen's kappa coefficient, is substantial (kappa = 0.75).We asked the third annotator to adjudicate the trigger clusters on which the former tow annotators disagreed.Each trigger cluster is used to represent one type of event.All these events construct our final event type paradigm.

Event Type Paradigm Building
We first propose the task of building event type paradigm in this paper.To evaluate the effectiveness of our approach, we explore several reasonable evaluation metrics and implement a natural baseline method.The detailed introductions are as follows.

Evaluation Metrics.
We adopt F-Measure (F ) and Purity used by Halkidi et al. (2001) to determine the correctness of an event cluster: where i is the gold standard event seed cluster, and r is the event seed cluster which has the most identical seeds with i.So n i is the number of seeds in cluster i; n r is the number of seeds in cluster r; n is the number of all seeds; and n(i, r) is the number of identical seeds between i and r.For every cluster we first compute p(i, r), r(i, r) and f (i, r), then we obtain F-Measure and Purity for the whole clustering result.Note that the evaluation is based on word instances rather than word types.Baseline Method.
The task of building a bottom-up event extraction system is first proposed by this paper.There is no existing work to compare with.We build event type paradigm based on clustering event seeds that consist of event trigger, and its corresponding subject and object.A natural baseline method for this problem is only clustering event triggers.A group of triggers are aggregated to a trigger cluster according to their semantic distance, and we view each trigger cluster as one kind of event type.Then all these event types are finally employed to construct an event type paradigm.

Results and Analysis.
We first evaluate the task of event type paradigm building.All the evaluation results are shown in Table 4.There is no previous work on this problem, so the comparison experiments are implemented on our two different approaches.Baseline method only clusters trigger words, however, our approach clusters event seeds.
Table 4 shows that the F-Measure score is boosted from 63.21% to 69.57% and the Purity score is boosted from 60.17% to 70.24% by using our approach compared to the baseline method.We analyze the reasons are as follows.
Trigger word itself is not enough for representing event.The trigger and its corresponding subject and object play an important role in the event type discovery algorithm.As referred in Section 3.1, most of trigger words are verb terms.Polysemic verbs are a major issue in Natural Language Processing (NLP) community, such as "to fire a gun" and "to fire a manager", where "fire" has two different meanings.The state-of-the-art verb sense disambiguation approach Wagner et al. (2009) stresses that verbs which agree on their selectional preferences belong to a common semantic class.For example, "to arrest the suspect" and "to capture the suspect".Hence, our approach can achieve better performance than the baseline method.
We also run the comparison experiment using three different corpora (ACE 05, Financial News and Musical News) to evaluate the robustness and domain adaptiveness of our system.The performances on the specific domain corpora are better than that on the ACE corpus (about 5% absolute improvement on F-Measure and 6%-10% on Purity).The main reason is that the events in specific domain are more specific.In addition, the experiment results on both specific domain corpora can achieve good performance.This indicates that our system is domain independent.Analysis of Experimental Errors.We first inspect the errors produced by our approach.The errors are mainly caused by the sparse event triggers in corpus.Table 7 shows the distribution of the errors in detail.
After error analysis, we find that the most number of errors are caused by trigger extraction.The main reasons are as follows.First, not all of event triggers are verbs, such as "婚姻(marriage)" for "Life/Marry" event.Although it is reasonable to assume that event triggers are verbs because on average, there are more than 95% event triggers are verbs in ACE 2005 corpus.Second, since only verbs with subject and object are extracted, non-predicate verbs and the verbs without subject/object will not be extracted as candidate triggers.However, the coverage of possible triggers by our trigger extraction algorithm is reasonable good (more than 85%), because most of the trigger words appear repeatedly in the corpus, and their usages are varied.As long as one of their usages is fit for our extraction algorithm, they can be extracted as candidate triggers.Note that the goal of this paper is to build an event type paradigm for new domains.We concern more on the coverage of event type rather than event triggers.The event triggers extracted by us can cover all of event types.We will exploit more effective trigger extraction algorithm in future work.
Trigger ambiguity also accounts for a big proportion of the errors.As discussed above, we cannot judge the event type only by the trigger itself, such as "撤(withdraw/dismiss)" for both "Personnel/End-Position" event and "Movement/Transport" event.This kind of errors can be partially fixed by the PAC model.For example, we cluster "撤职务(dismiss duties)" for "Personnel/End-Position" event and "撤 军 队(withdraw troop)" for "Movement/Transport" event.These examples indicate that selectional preferences seem to be a reasonable feature even for highly ambiguous verbs like "撤" which encourages to improve argument extraction.
There are still some errors caused by trigger filter.This is mainly due to the fact that not all of triggers are general verb or nominal verb.More effective filter rules will be exploited in future.
Some other errors are caused by NLP tools, such as word segmentation, part-of-speech tagging and dependency parsing.We believe that our algorithms can be improved with the improvement of these NLP tools.In addition, there are about 10% of good event triggers extracted but put into the wrong cluster by trigger cluster. in Algorithm 2 can dramatically affect the performance of event seed clustering.We experiment with different values of event seed clustering threshold to find the best value.Figure 4 presents the effect on F-Measure of varying the threshold for seed clustering.This figure shows that the best performance of seed clustering can be obtained by selecting the threshold 0.6 for the ACE corpus, 0.7 for the Financial News corpus and 0.9 for the Musical ACE 05 corpus.Figure 4 also suggests that the performance of seed clustering do not dramatically change with the volatility of the threshold from 0.5 to 0.8.Hence, we can firstly set the threshold = 0.6 for new domains.

Event Type Identification and Argument Recognition
Based on the event type paradigm automatically built by us, we proceed to implement traditional event extraction tasks i.e., event type identification and event argument recognition.We adopt Precision (P ), Recall (R) and F-Measure (F ) to evaluate the effectiveness of our approach, and compare it with a state-of-the-art event extraction system.Baseline Method.
We use a state-of-the-art Chinese Event Extraction System as our baseline which is developed by Chen and Ji (2009).This system extracts events with annotated corpus.Its training and testing procedures are follows.
The system combines word-based classifier with character-based classifier.The event types are specified in advance.For every event mention in the ACE training corpus, features are extracted according to some language specific issues.In addition, a set of Maximum Entropy based classifiers are trained: • Event Type Identification: to distinguish event mentions from non-event-mentions, to classify

h h h h h h h h h h h h h h h
System/Human event mentions by type; • Event Argument Recognition: to distinguish event arguments from non-arguments.
In the testing procedure, each document is scanned for instances of triggers from the training corpus.If an instance is found by trigger classifier, the system tries to assign some of the mentions in the sentence as arguments of a potential event mention.The argument classifier is applied to the remaining mentions in the sentence, for any argument passing that classifier; the role classifier will assign a role to it.At last, the system will report the event with type and arguments.

Results and Analysis.
Table 6 shows the overall Precision (P ), Recall (R) and F-Measure (F ) scores of the baseline system and our BUEES.The table also lists the performance of two human annotators from Chen and Ji (2009).
BUEES outperforms the baseline without annotated corpus.The performance of event type identification is 2.3% higher than the baseline system, and the performance of event argument recognition is 5.4% higher.Several conclusions can be drawn from Table 6.
(1) BUEES does not use any labeled corpus and achieves comparable performance with the baseline system.
(2) Our approach on event type identification enhances the precision (7%) with little loss (0.2%) in recall compared to the baseline method.This recall loss is caused by the limited number of the event seeds.The precision of event type identification is only 2.5% worse than one human annotator, which indicates that the precision of our approach is reasonable good.As our approach is pattern-based, the recall of it needs to be improved.
(3) Our approach on event argument recognition enhances the F-Measure (5%) compared to the baseline method.It indicates that our pattern learning algorithm is effective.We also find that the performance of human annotation on event argument recognition is not high enough (59.7% in F-Measure), and hence it is a difficult problem requires further study.
(4) Table 6 also shows that BUEES system improves the recall (9.7%) performance of event argument recognition over the baseline system.It shows that unsupervised event extraction system can also achieve comparable or better performance with supervised system.However, it is obvious that the system recall is substantially lower than the system precision.Though bootstrapping-based approach can improve recall performance of the soft-pattern by increasing the number of iteration.When the number of iterations increased to a certain value, the recall will not be improved and the precision may be decrease.It is mainly because the more number of iterations the more noise information will be involved in our system.
We should also note that the result of human annotators use the perfect entity mentions, but our system extracts named entities automatically.So the gap is also partially due to wrong named entities.

Related Work
Our system is designed to address two issues of event extraction: event type paradigm building and traditional task of event extraction (i.e., event type identification and event argument recognition).The approach of event type paradigm building is related to some prior work on word cluster discovery (e.g.Barzilay and McKeown (2001); Lin and Pantel (2001); Ibrahim et al. (2003); Pang et al. (2003) Miller et al. (2004) Hasegawa et al. (2004) Rosenfeld and Feldman (2006)).Most of these works are based on machine translation techniques to solve paraphrase extraction problem.However, several recent researches have stressed the benefits of using word clusters to improve the performance of infor-mation extraction tasks.For example, Miller et al. Miller et al. (2004) proved that word clusters could significantly improve English name tagging performance.In the same vein, some studies work on the problem of relation extraction (Chambers andJurafsky (2009, 2011); Poon andDomingos (2008, 2009); Yates and Etzioni (2009)).In these works, "relation words" were extracted and clustered.In this paper, our work confirmed that event seed clusters are also effective for event type paradigm building.The problem of event seeds extraction and clustering is also a challenge problem.
The approach of event extraction is related to a weakly supervised pattern learning algorithm.Yangarber et al. (2000) used bootstrapping based method learning simple surface patterns for extracting information.Stevenson and Greenwood (2005) proposed similarity-centric bootstrapping which tried to find patterns with high lexical similarities.Liao and Grishman (2010) filtered ranking for bootstrapping in event extraction.They used two state-of-the-art bootstrapping-based event extraction systems then rank their candidate patterns and accept the top-ranked patterns at each iteration.The BUEES is similar to these systems in the general approach, but its surface patterns allow gaps that can be matched by any sequences of tokens, which make the patterns much more general, and allows to recognize more instances than the simple surface patterns.
Some English event extraction systems based on pattern or machine learning have been reported by researchers (Patwardhan and Riloff (2006); Yangarber et al. (2000); Grishman (2001); Ji and Grishman (2008)).However, to our knowledge, the non-English event extraction has rarely been reported by earlier researchers.The baseline system is based on ACE Chinese events.Its contribution is to exploit language specific feature for Chinese event extraction.However, the reported precision of the results was lower than English event extraction.In contrast, the performance of BUEES which is absolutely unsupervised is better than the baseline system and not lower than the state-of-the-art English system.
Web-scale information extraction has received considerable attention in the last few years.Preemptive Information Extraction and Open Information Extraction (Open IE) are the first paradigms that relax the restriction of a given vocabulary of relations and scale to all relation phrases expressed in text (Shinyama and Sekine (2006); Banko et al. (2007); Banko et al. (2008); Wu and Weld (2010); Etzioni et al. (2011);Fader et al. (2011)).Preemptive IE relies on document and entity clustering, which is too costly for Webscale IE.Open IE favors speed over deeper processing, which aids in scaling to Web-scale corpora.Comparing with Pre-emptive Information Extraction and Open Information Extraction, the main differences of this paper are on the following aspects.First, the previous work mainly focuses on relation extraction, however, this paper aims to extracts events from web corpus.Second, Open IE cannot give the event (or relation) type paradigm which is useful for application.

Conclusion and Future Work
We have presented the BUEES system which discovers interesting events; learns extraction patterns and extracts the event instances from the Web.BUEES neither relies on manually produced extraction patterns nor on manually annotated training corpus.
BUEES performs by clustering event seeds, generating seed instances and bootstrapping its pattern.Based on general sequential pattern, this paper proposed soft-pattern learning algorithm.Soft-pattern has much greater generalization ability and can reach a high performance for successful bootstrapping.
One of the future work we would like to explore more perfect patterns.Patterns in this paper are very simple that just use several cost values and rules to generate soft-pattern.We want to see if we can achieve higher performance with more complex and perfect patterns.

c
⃝Zhejiang University and Springer-Verlag Berlin Heidelberg 2013

add
Vt to the related event type ETre ∪ {Vt} 6: else if Sim < θ then 7: set up a new event type ETnew 8: EC ← ETnew 9:

Figure 4
Figure 4 Experimental results of different values of event seed clustering threshold 1. Statistically, about 94% of ACE Chinese event triggers are general verbs or nominal verbs and other types of verbs are rarely as trigger words.In order to en-

Table 3
The example of Best Match algorithm

Table 4
Experimental results of event type paradigm building

Table 5
Experimental errors of event type paradigm building

Table 6
Overall experimental results