Publications
2026
Abstract
The LLM-as-a-judge (LLMaJ) framework has emerged as a promising solution for cheap, reproducible, reference-free Natural Language Generation (NLG) evaluation. Prior work seeks to improve LLMaJ by decomposing evaluation tasks into simpler sub-tasks. In this work, we systematically compare LLMaJ methods with and without decomposition on multiple NLG datasets. We find no evidence that LLMaJ with task decomposition leads to performance gains over a fair baseline that does not use decomposition. Instead, we find that previously reported performance gains in decomposition-based LLMaJ stem from using human labels as training data, and not task decomposition itself. Also, we find that, when human labels are available, LLMaJ without using task decomposition can perform comparably to human annotators.
BibTeX
@inproceedings{steindl-etal-2026-task,
title = "Does task decomposition improve automatic {NLG} evaluation?",
author = "Steindl, Sebastian and
Voskarides, Nikos and
Gasparin, Alberto and
Marcheggiani, Diego",
editor = "Lee, Hung-Yi and
Dev, Sunipa and
Li, Junyi Jessy and
Elliott, Desmond",
booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
month = oct,
year = "2026",
address = "Budapest, Hungary",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.emnlp-main.447/",
doi = "10.18653/2026.emnlp-main.447",
pages = "10288--10301",
ISBN = "979-8-89176-459-0"
}
Abstract
The LLM-as-a-judge (LLMaJ) framework offers a cost-effective and reproducible solution for automatic evaluation. However, current evaluation practices typically compare LLMaJ judgments against aggregated ground-truth labels, overlooking the valuable information contained in Human Label Variation (HLV). Inspired by an increasing line of work that proposes to leverage HLV, we systematically study LLMaJ performance on predicting both a single, aggregated ground truth hard-label and unaggregated soft-labels that represent Human Judgment Distributions (HJD). Our results across five diverse datasets reveal that while LLMs achieve near human-level performance at hard-label prediction on most tasks, they exhibit poor performance when predicting soft-labels. To address this limitation, we propose NAPHA (eNtropy-Aware Post-Hoc Alignment), a simple yet effective lightweight post-hoc alignment method that matches the LLM distribution to the HJD by first assigning an instance to a discrete entropy class and then routing it to specialized, trained alignment models. We find that NAPHA consistently improves soft-labels prediction across base LLM models and datasets, with particularly strong gains on high-entropy instances where capturing diverse human perspectives is most critical. We also show via oracle experiments that improving entropy class prediction can substantially enhance NAPHA’s practical effectiveness.
BibTeX
@inproceedings{steindl-etal-2026-post,
title = "Post-hoc Alignment of {LLM}-judges to Human Judgment Distribution",
author = "Steindl, Sebastian and
Voskarides, Nikos and
Gasparin, Alberto and
Marcheggiani, Diego",
editor = "Lee, Hung-Yi and
Dev, Sunipa and
Li, Junyi Jessy and
Elliott, Desmond",
booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
month = oct,
year = "2026",
address = "Budapest, Hungary",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.emnlp-main.436/",
doi = "10.18653/2026.emnlp-main.436",
pages = "10014--10037",
ISBN = "979-8-89176-459-0"
}
2023
Abstract
Sentiment analysis (SA) systems are used in many products and hundreds of languages. Gender and racial biases are well-studied in English SA systems, but understudied in other languages, with few resources for such studies. To remedy this, we build a counterfactual evaluation corpus for gender and racial/migrant bias in four languages. We demonstrate its usefulness by answering a simple but important question that an engineer might need to answer when deploying a system: What biases do systems import from pre-trained models when compared to a baseline with no pre-training? Our evaluation corpus, by virtue of being counterfactual, not only reveals which models have less bias, but also pinpoints changes in model bias behaviour, which enables more targeted mitigation strategies. We release our code and evaluation corpora to facilitate future research.
BibTeX
@inproceedings{goldfarb-tarrant-etal-2023-bias,
title = "Bias Beyond {E}nglish: Counterfactual Tests for Bias in Sentiment Analysis in Four Languages",
author = "Goldfarb-Tarrant, Seraphina and
Lopez, Adam and
Blanco, Roi and
Marcheggiani, Diego",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
year = "2023",
url = "https://aclanthology.org/2023.findings-acl.272",
doi = "10.18653/v1/2023.findings-acl.272",
pages = "4458--4468",
abstract = "Sentiment analysis (SA) systems are used in many products and hundreds of languages. Gender and racial biases are well-studied in English SA systems, but understudied in other languages, with few resources for such studies. To remedy this, we build a counterfactual evaluation corpus for gender and racial/migrant bias in four languages. We demonstrate its usefulness by answering a simple but important question that an engineer might need to answer when deploying a system: What biases do systems import from pre-trained models when compared to a baseline with no pre-training? Our evaluation corpus, by virtue of being counterfactual, not only reveals which models have less bias, but also pinpoints changes in model bias behaviour, which enables more targeted mitigation strategies. We release our code and evaluation corpora to facilitate future research.",
}2020
Abstract
Semantic role labeling (SRL) is the task of identifying predicates and labeling argument spans with semantic roles. Even though most semantic-role formalisms are built upon constituent syntax, and only syntactic constituents can be labeled as arguments (e.g., FrameNet and PropBank), all the recent work on syntax-aware SRL relies on dependency representations of syntax. In contrast, we show how graph convolutional networks (GCNs) can be used to encode constituent structures and inform an SRL system. Nodes in our SpanGCN correspond to constituents. The computation is done in 3 stages. First, initial node representations are produced by ‘composing’ word representations of the first and last words in the constituent. Second, graph convolutions relying on the constituent tree are performed, yielding syntactically-informed constituent representations. Finally, the constituent representations are ‘decomposed’ back into word representations, which are used as input to the SRL classifier. We evaluate SpanGCN against alternatives, including a model using GCNs over dependency trees, and show its effectiveness on standard English SRL benchmarks CoNLL-2005, CoNLL-2012, and FrameNet.
BibTeX
@inproceedings{marcheggiani-titov-2020-graph,
title = "Graph Convolutions over Constituent Trees for Syntax-Aware Semantic Role Labeling",
author = "Marcheggiani, Diego and
Titov, Ivan",
booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
month = nov,
year = "2020",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/2020.emnlp-main.322",
pages = "3915--3928",
abstract = "Semantic role labeling (SRL) is the task of identifying predicates and labeling argument spans with semantic roles. Even though most semantic-role formalisms are built upon constituent syntax, and only syntactic constituents can be labeled as arguments (e.g., FrameNet and PropBank), all the recent work on syntax-aware SRL relies on dependency representations of syntax. In contrast, we show how graph convolutional networks (GCNs) can be used to encode constituent structures and inform an SRL system. Nodes in our SpanGCN correspond to constituents. The computation is done in 3 stages. First, initial node representations are produced by {`}composing{'} word representations of the first and last words in the constituent. Second, graph convolutions relying on the constituent tree are performed, yielding syntactically-informed constituent representations. Finally, the constituent representations are {`}decomposed{'} back into word representations, which are used as input to the SRL classifier. We evaluate SpanGCN against alternatives, including a model using GCNs over dependency trees, and show its effectiveness on standard English SRL benchmarks CoNLL-2005, CoNLL-2012, and FrameNet.",
}2019
Abstract
We present a system for answering questions based on the full text of books (BookQA), which first selects book passages given a question at hand, and then uses a memory network to reason and predict an answer. To improve generalization, we pretrain our memory network using artificial questions generated from book sentences. We experiment with the recently published NarrativeQA corpus, on the subset of Who questions, which expect book characters as answers. We experimentally show that BERT-based retrieval and pretraining improve over baseline results significantly. At the same time, we confirm that NarrativeQA is a highly challenging data set, and that there is need for novel research in order to achieve high-precision BookQA results. We analyze some of the bottlenecks of the current approach, and we argue that more research is needed on text representation, retrieval of relevant passages, and reasoning, including commonsense knowledge.
BibTeX
@inproceedings{angelidis-etal-2019-book,
title = "Book {QA}: Stories of Challenges and Opportunities",
author = "Angelidis, Stefanos and
Frermann, Lea and
Marcheggiani, Diego and
Blanco, Roi and
M{\`a}rquez, Llu{\'\i}s",
booktitle = "Proceedings of the 2nd Workshop on Machine Reading for Question Answering",
month = nov,
year = "2019",
address = "Hong Kong, China",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/D19-5811",
doi = "10.18653/v1/D19-5811",
pages = "78--85",
}
You Shall Know a User by the Company It Keeps: Dynamic Representations for Social Media Users in NLP
Abstract
Information about individuals can help to better understand what they say, particularly in social media where texts are short. Current approaches to modelling social media users pay attention to their social connections, but exploit this information in a static way, treating all connections uniformly. This ignores the fact, well known in sociolinguistics, that an individual may be part of several communities which are not equally relevant in all communicative situations. We present a model based on Graph Attention Networks that captures this observation. It dynamically explores the social graph of a user, computes a user representation given the most relevant connections for a target task, and combines it with linguistic information to make a prediction. We apply our model to three different tasks, evaluate it against alternative models, and analyse the results extensively, showing that it significantly outperforms other current methods.
BibTeX
@inproceedings{del-tredici-etal-2019-shall,
title = "You Shall Know a User by the Company It Keeps: Dynamic Representations for Social Media Users in {NLP}",
author = "Del Tredici, Marco and
Marcheggiani, Diego and
Schulte im Walde, Sabine and
Fern{\'a}ndez, Raquel",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)",
month = nov,
year = "2019",
address = "Hong Kong, China",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/D19-1477",
doi = "10.18653/v1/D19-1477",
pages = "4701--4711",
}
2018
Abstract
Most previous work on neural text generation from graph-structured data relies on standard sequence-to-sequence methods. These approaches linearise the input graph to be fed to a recurrent neural network. In this paper, we propose an alternative encoder based on graph convolutional networks that directly exploits the input structure. We report results on two graph-to-sequence datasets that empirically show the benefits of explicitly encoding the input graph structure.
BibTeX
@InProceedings{W18-6501,
author = "Marcheggiani, Diego and Perez-Beltrachini, Laura",
title = "Deep Graph Convolutional Encoders for Structured Data to Text Generation",
booktitle = "Proceedings of the 11th International Conference on Natural Language Generation",
year = "2018",
publisher = "Association for Computational Linguistics",
pages = "1--9",
location = "Tilburg University, The Netherlands",
url = "http://aclweb.org/anthology/W18-6501"
}
Abstract
Semantic representations have long been argued as potentially useful for enforcing meaning preservation and improving generalization performance of machine translation methods. In this work, we are the first to incorporate information about predicate-argument structure of source sentences (namely, semantic-role representations) into neural machine translation. We use Graph Convolutional Networks (GCNs) to inject a semantic bias into sentence encoders and achieve improvements in BLEU scores over the linguistic-agnostic and syntax-aware versions on the English--German language pair.
BibTeX
@InProceedings{marcheggiani-bastings-titov:2018:NAACL,
author = {Marcheggiani, Diego and Bastings, Jasmijn and Titov, Ivan},
title = {Exploiting Semantics in Neural Machine Translation with Graph Convolutional Networks},
booktitle = {Proceedings of the the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2018)},
month = {June},
year = {2018},
address = {New Orleans, US},
publisher = {Association for Computational Linguistics}
}
2017
Abstract
Semantic role labeling (SRL) is the task of identifying the predicate-argument structure of a sentence. It is typically regarded as an important step in the standard natural language processing pipeline, providing information to downstream tasks such as information extraction and question answering. As the semantic representations are closely related to syntactic ones, we exploit syntactic information in our model. We propose a version of graph convolutional networks (GCNs), a recent class of multilayer neural networks operating on graphs, suited to modeling syntactic dependency graphs. GCNs over syntactic dependency trees are used as sentence encoders, producing latent feature representations of words in a sentence and capturing information relevant to predicting the semantic representations. We observe that GCN layers are complementary to LSTM ones: when we stack both GCN and LSTM layers, we obtain a substantial improvement over an already state-of-the-art LSTM SRL model, resulting in the best reported scores on the standard benchmark (CoNLL-2009) both for Chinese and English.
BibTeX
@InProceedings{marcheggiani-titov:2017:EMNLP2017,
author = {Marcheggiani, Diego and Titov, Ivan},
title = {Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling},
booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
month = {September},
year = {2017},
address = {Copenhagen, Denmark},
publisher = {Association for Computational Linguistics},
pages = {1507--1516},
url = {https://www.aclweb.org/anthology/D17-1159}
}
Abstract
We present a simple and effective approach to incorporating syntactic structure into neural attention-based encoder-decoder models for machine translation. We rely on graph-convolutional networks (GCNs), a recent class of neural networks developed for modeling graph-structured data. Our GCNs use predicted syntactic dependency trees of source sentences to produce representations of words (i.e. hidden states of the encoder) that are sensitive to their syntactic neighborhoods. GCNs take word representations as input and produce word representations as output, so they can easily be incorporated as layers into standard encoders (e.g., on top of bidirectional RNNs or convolutional neural networks). We evaluate their effectiveness with English-German and English-Czech translation experiments for different types of encoders and observe substantial improvements over their syntax-agnostic versions in all the considered setups.
BibTeX
@InProceedings{bastings-EtAl:2017:EMNLP2017,
author = {Bastings, Jasmijn and Titov, Ivan and Aziz, Wilker and Marcheggiani, Diego and Simaan, Khalil},
title = {Graph Convolutional Encoders for Syntax-aware Neural Machine Translation},
booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
month = {September},
year = {2017},
address = {Copenhagen, Denmark},
publisher = {Association for Computational Linguistics},
pages = {1947--1957},
url = {https://www.aclweb.org/anthology/D17-1209}
}
Abstract
We introduce a simple and accurate neural model for dependency-based semantic role labeling. Our model predicts predicate-argument dependencies relying on states of a bidirectional LSTM encoder. The semantic role labeler achieves respectable performance on English even without any kind of syntactic information and only using local inference. However, when automatically predicted part-of-speech tags are provided as input, it substantially outperforms all previous local models and approaches the best reported results on the CoNLL-2009 dataset. Syntactic parsers are unreliable on out-of-domain data, so standard (i.e. syntactically-informed) SRL models are hindered when tested in this setting. Our syntax-agnostic model appears more robust, resulting in the best reported results on the standard out-of-domain test set.
BibTeX
@InProceedings{marcheggiani-frolov-titov:2017:CoNLL,
author = {Marcheggiani, Diego and Frolov, Anton and Titov, Ivan},
title = {A Simple and Accurate Syntax-Agnostic Neural Model for Dependency-based Semantic Role Labeling},
booktitle = {Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)},
month = {August},
year = {2017},
address = {Vancouver, Canada},
publisher = {Association for Computational Linguistics},
pages = {411--420},
url = {http://aclweb.org/anthology/K17-1041}
}
Abstract
In the last five years there has been a flurry of work on information extraction from clinical documents, i.e., on algorithms capable of extracting, from the informal and unstructured texts that are generated during everyday clinical practice, mentions of concepts relevant to such practice. Many of these research works are about methods based on supervised learning, i.e., methods for training an information extraction system from manually annotated examples. While a lot of work has been devoted to devising learning methods that generate more and more accurate information extractors, no work has been devoted to investigating the effect of the quality of training data on the learning process for the clinical domain. Low quality in training data often derives from the fact that the person who has annotated the data is different from the one against whose judgment the automatically annotated data must be evaluated. In this paper we test the impact of such data quality issues on the accuracy of information extraction systems as applied to the clinical domain. We do this by comparing the accuracy deriving from training data annotated by the authoritative coder (i.e., the one who has also annotated the test data, and by whose judgment we must abide), with the accuracy deriving from training data annotated by a different coder, equally expert in the subject matter. The results indicate that, although the disagreement between the two coders (as measured on the training set) is substantial, the difference is (surprisingly enough) not always statistically significant. While the dataset used in the present work originated in a clinical context, the issues we study in this work are of more general interest.
BibTeX
@article{Marcheggiani:2017:ELT:3139489.3106235,
author = {Marcheggiani, Diego and Sebastiani, Fabrizio},
title = {On the Effects of Low-Quality Training Data on Information Extraction from Clinical Reports},
journal = {ACM Journal of Data and Information Quality},
issue_date = {September 2017},
volume = {9},
number = {1},
month = sep,
year = {2017},
issn = {1936-1955},
pages = {1:1--1:25},
articleno = {1},
numpages = {25},
url = {http://doi.acm.org/10.1145/3106235},
doi = {10.1145/3106235},
acmid = {3106235},
publisher = {ACM},
address = {New York, NY, USA},
keywords = {Information extraction, annotation quality, clinical narratives, machine learning, medical reports, radiology reports},
}
2016
Abstract
We present a method for unsupervised open-domain relation discovery. In contrast to previous (mostly generative and agglomerative clustering) approaches, our model relies on rich contextual features and makes minimal independence assumptions. The model is composed of two parts: a feature-rich relation extractor, which predicts a semantic relation between two entities, and a factorization model, which reconstructs arguments (i.e., the entities) relying on the predicted relation. The two components are estimated jointly so as to minimize errors in recovering arguments. We study factorization models inspired by previous work in relation factorization and selectional preference modeling. Our models substantially outperform the generative and agglomerative-clustering counterparts and achieve state-of-the-art performance.
BibTeX
@article{DBLP:journals/tacl/MarcheggianiT16,
author = {Diego Marcheggiani and
Ivan Titov},
title = {Discrete-State Variational Autoencoders for Joint Discovery and Factorization
of Relations},
journal = {{TACL}},
volume = {4},
pages = {231--244},
year = {2016},
url = {https://tacl2013.cs.columbia.edu/ojs/index.php/tacl/article/view/761},
}
BibTeX
@inproceedings{marcheggianiKT16,
author = {Diego Marcheggiani, Ehsan Khoddam, Ivan Titov},
title = {Unsupervised Induction of Shallow Semantic Representations with Feature Rich Models},
booktitle = {Book of Abstracts of the 9th International Conference on Construction Grammar},
month = {October},
year = {2016},
address = {Juiz de Fora, Brazil},
url = {http://www.ufjf.br/iccg9/files/2015/02/ICCG9_Book_of_Abstracts_v1.pdf}
}
2015
Abstract
Microblogging is a model of content sharing in which the temporal locality of posts with respect to important events, either of foreseeable or unforeseeable nature, makes applications of real-time filtering of great practical interest. We propose the use of Entity Linking (EL) in order to improve the retrieval effectiveness, by enriching the representation of microblog posts and filtering queries. EL is the process of recognizing in an unstructured text the mention of relevant entities described in a knowledge base. EL of short pieces of text is a difficult task, but it is also a scenario in which the information EL adds to the text can have a substantial impact on the retrieval process. We implement a start-of-the-art filtering method, based on the best systems from the TREC Microblog track real-time adhoc retrieval and filtering tasks, and extend it with a Wikipedia-based EL method. Results show that the use of EL significantly improves over non-EL based versions of the filtering methods.
BibTeX
@inproceedings{berardiCEM15,
author = {Giacomo Berardi and
Diego Ceccarelli and
Andrea Esuli and
Diego Marcheggiani},
title = {On the impact of entity linking in microblog real-time filtering},
booktitle = {Proceedings of the 30th Annual {ACM} Symposium on Applied Computing,
Salamanca, Spain, April 13-17, 2015},
pages = {1066--1071},
year = {2015},
}
Abstract
In this paper we present some preliminary results on the generation of word embeddings for the Italian language. We compare two popular word representation models, word2vec and GloVe, and train them on two datasets with different stylistic properties. We test the generated word embeddings on a word analogy test derived from the one originally proposed for word2vec, adapted to capture some of the linguistic aspects that are specific of Italian. Results show that the tested models are able to create syntactically and semantically meaningful word embeddings despite the higher morphological complexity of Italian with respect to English. Moreover, we have found that the stylistic properties of the training dataset plays a relevant role in the type of information captured by the produced vectors.
BibTeX
@inproceedings{berardiEM15,
author = {Giacomo Berardi and
Andrea Esuli and
Diego Marcheggiani},
title = {Word Embeddings Go to Italy: {A} Comparison of Models and Training
Datasets},
booktitle = {Proceedings of the 6th Italian Information Retrieval Workshop, Cagliari,
Italy, May 25-26, 2015.},
year = {2015},
url = {http://ceur-ws.org/Vol-1404/paper_11.pdf}
}
Abstract
We present the Trip-MAML dataset, a Multi-Lingual dataset of hotel reviews that have been manually annotated at the sentence-level with Multi-Aspect sentiment labels. This dataset has been built as an extension of an existent English-only dataset, adding documents written in Italian and Spanish. We detail the dataset construction process, covering the data gathering, selection, and annotation. We present inter-annotator agreement figures and baseline experimental results, comparing the three languages. Trip-MAML is a multi-lingual dataset for aspect-oriented opinion mining that enables researchers (i) to face the problem on languages other than English and (ii) to the experiment the application of cross-lingual learning methods to the task.
BibTeX
@inproceedings{ZafraBEMMF15,
author = {Salud M. Jim{\'{e}}nez Zafra and
Giacomo Berardi and
Andrea Esuli and
Diego Marcheggiani and
Maria Teresa Mart{\'{\i}}n{-}Valdivia and
Alejandro Moreo Fern{\'{a}}ndez},
title = {A Multi-lingual Annotated Dataset for Aspect-Oriented Opinion Mining},
booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural
Language Processing, {EMNLP} 2015, Lisbon, Portugal, September 17-21,
2015},
pages = {2533--2538},
year = {2015},
url = {http://aclweb.org/anthology/D/D15/D15-1302.pdf}
}2014
Abstract
Active learning (AL) consists of asking human annotators to annotate automatically selected data that are assumed to bring the most benefit in the creation of a classifier. AL allows to learn accurate systems with much less annotated data than what is required by pure supervised learning algorithms, hence limiting the tedious effort of annotating a large collection of data. We experimentally investigate the behavior of several AL strategies for sequence labeling tasks (in a partially-labeled scenario) tailored on Partially-Labeled Conditional Random Fields, on four sequence labeling tasks: phrase chunking, part-of-speech tagging, named-entity recognition, and bio-entity recognition.
BibTeX
@inproceedings{MarcheggianiA14,
author = {Diego Marcheggiani and
Thierry Arti{\`{e}}res},
title = {An Experimental Comparison of Active Learning Strategies for Partially
Labeled Sequences},
booktitle = {Proceedings of the 2014 Conference on Empirical Methods in Natural
Language Processing, {EMNLP} 2014, October 25-29, 2014, Doha, Qatar,
{A} meeting of SIGDAT, a Special Interest Group of the {ACL}},
pages = {898--906},
year = {2014}
}
Abstract
A common feature of many online review sites is the use of an overall rating that summarizes the opinions expressed in a review. Unfortunately, these document-level ratings do not provide any information about the opinions contained in the review that concern a specific aspect (e.g., cleanliness) of the product being reviewed (e.g., a hotel). In this paper we study the finer-grained problem of aspect-oriented opinion mining at the sentence level, which consists of predicting, for all sentences in the review, whether the sentence expresses a positive, neutral, or negative opinion (or no opinion at all) about a specific aspect of the product. For this task we propose a set of increasingly powerful models based on conditional random fields (CRFs), including a hierarchical multi-label CRFs scheme that jointly models the overall opinion expressed in the review and the set of aspect-specific opinions expressed in each of its sentences. We evaluate the proposed models against a dataset of hotel reviews (which we here make publicly available) in which the set of aspects and the opinions expressed concerning them are manually annotated at the sentence level. We find that both hierarchical and multi-label factors lead to improved predictions of aspect-oriented opinions.
BibTeX
@inproceedings{DBLP:conf/ecir/MarcheggianiTE014,
author = {Diego Marcheggiani and
Oscar T{\"{a}}ckstr{\"{o}}m and
Andrea Esuli and
Fabrizio Sebastiani},
title = {Hierarchical Multi-label Conditional Random Fields for Aspect-Oriented
Opinion Mining},
booktitle = {Advances in Information Retrieval - 36th European Conference on {IR}
Research, {ECIR} 2014, Amsterdam, The Netherlands, April 13-16, 2014.
Proceedings},
pages = {273--285},
year = {2014}
}
2013
Abstract
We discuss the problem of performing information extraction from free-text radiology reports via supervised learning. In this task, segments of text (not necessarily coinciding with entire sentences, and possibly crossing sentence boundaries) need to be annotated with tags representing concepts of interest in the radiological domain. In this paper we present two novel approaches to IE for radiology reports: (i) a cascaded, two-stage method based on pipelining two taggers generated via the well known linear-chain conditional random fields (LC-CRFs) learner and (ii) a confidence-weighted ensemble method that combines standard LC-CRFs and the proposed two-stage method. We also report on the use of “positional features”, a novel type of feature intended to aid in the automatic annotation of texts in which the instances of a given concept may be hypothesized to systematically occur in specific areas of the text. We present experiments on a dataset of mammography reports in which the proposed ensemble is shown to outperform a traditional, single-stage CRFs system in two different, applicatively interesting scenarios.
BibTeX
@article{EsuliMS13,
author = {Andrea Esuli and
Diego Marcheggiani and
Fabrizio Sebastiani},
title = {An enhanced CRFs-based system for information extraction from radiology
reports},
journal = {Journal of Biomedical Informatics},
volume = {46},
number = {3},
pages = {425--435},
year = {2013},
}
2012
Abstract
We demonstrate a metadata enrichment system for the Europeana digital library. The system allows different institutions which provide to Europeana pointers (in the form of metadata records - MRs) to their content to enrich their MRs by classifying them under a classification scheme of their choice, and to extract/highlight entities of significant interest within the MRs themselves. The use of a supervised learning metaphor allows each content provider (CP) to generate classifiers and extractors tailored to the CP’s specific needs, thus allowing the tool to be effectively available to the multitude (2000+) of Europeana CPs.
BibTeX
@inproceedings{BerardiEGMS12,
author = {Giacomo Berardi and
Andrea Esuli and
Sergiu Gordea and
Diego Marcheggiani and
Fabrizio Sebastiani},
title = {Metadata Enrichment Services for the Europeana Digital Library},
booktitle = {Theory and Practice of Digital Libraries - Second International Conference,
{TPDL} 2012, Paphos, Cyprus, September 23-27, 2012. Proceedings},
pages = {508--511},
year = {2012}
}
Abstract
Our approach to the microblog filtering task is based on learning a relevance classifier from an initial training set of relevant and non relevant tweets, generated by using a simple retrieval method. The classifier is then retrained using the (simulated) user feedback collected during the training process, in order to improve its accuracy as the filtering process goes on. In the official runs the system scored low effectiveness values, suffering a strong imbalance toward recall.
BibTeX
@inproceedings{DBLP:conf/trec/BerardiEM12,
author = {Giacomo Berardi and
Andrea Esuli and
Diego Marcheggiani},
title = {ISTI@TREC Microblog Track 2012: Real-Time Filtering Through Supervised
Learning},
booktitle = {Proceedings of The Twenty-First Text REtrieval Conference, {TREC}
2012, Gaithersburg, Maryland, USA, November 6-9, 2012},
year = {2012},
url = {http://trec.nist.gov/pubs/trec21/papers/NEMIS_ISTI_CNR.microblog.final.pdf},
}
2011
Abstract
In the first year of the TREC Micro Blog track, our participation has focused on building from scratch an IR system based on the Whoosh IR library. Though the design of our system (CipCipPy) is pretty standard it includes three ad-hoc solutions for the track (i) a dedicated indexing function for hashtags that automatically recognizes the distinct words composing an hashtag, (ii) expansion of tweets based on the title of any referred Web page, and (iii) a tweet ranking function that ranks tweets in results by their content quality, which is compared against a reference corpus of Reuters news. In this preliminary paper we describe all the components of our system, and the efficacy scored by our runs. The CipCipPy system is available under a GPL license.
BibTeX
@inproceedings{BerardiEMS11,
author = {Giacomo Berardi and
Andrea Esuli and
Diego Marcheggiani and
Fabrizio Sebastiani},
title = {ISTI@TREC Microblog Track 2011 Exploring the Use of Hashtag Segmentation
and Text Quality Ranking},
booktitle = {Proceedings of The Twentieth Text REtrieval Conference, ({TREC})},
year = {2011}
}
2010
Abstract
Given a classifier trained on relatively few training examples, active learning (AL) consists in ranking a set of unlabeled examples in terms of how informative they would be, if manually labeled, for retraining a (hopefully) better classifier. An important text learning task in which AL is potentially useful is information extraction (IE), namely, the task of identifying within a text the expressions that instantiate a given concept. We contend that, unlike in other text learning tasks, IE is unique in that it does not make sense to rank individual items (i.e., word occurrences) for annotation, and that the minimal unit of text that is presented to the annotator should be an entire sentence. In this paper we propose a range of active learning strategies for IE that are based on ranking individual sentences, and experimentally compare them on a standard dataset for named entity extraction.
BibTeX
@inproceedings{esuliMS10,
author = {Andrea Esuli and
Diego Marcheggiani and
Fabrizio Sebastiani},
title = {Sentence-Based Active Learning Strategies for Information Extraction},
booktitle = {Proceedings of the First Italian Information Retrieval
Workshop, ({IIR})},
pages = {41--45},
year = {2010}
}
Abstract
We describe a boosting-based supervised learning approach to the Multi-Way Classification of Semantic Relations between Pairs of Nominals task 8 of SemEval-2. Participants were asked to determine which relation, from a set of nine relations plus “Other”, exists between two nominals, and also to determine the roles of the two nominals in the relation. Our participation has focused, rather than on the choice of a rich set of features, on the classification model adopted to determine the correct assignment of relation and roles.
BibTeX
@inproceedings{EsuliM010,
author = {Andrea Esuli and
Diego Marcheggiani and
Fabrizio Sebastiani},
title = {ISTI@SemEval-2 Task 8 Boosting-Based Multiway Relation Classification},
booktitle = {Proceedings of the 5th International Workshop on Semantic Evaluation,
SemEval@ACL 2010, Uppsala University, Uppsala, Sweden, July 15-16,
2010},
pages = {218--221},
year = {2010},
url = {http://aclweb.org/anthology/S/S10/S10-1048.pdf}
}