CS65 Lab 8

Due 11:59pm Wednesday Wednesday, November 19

For this week's lab, you have to read at least three papers and implement some new functionality in your sentiment classifier.

  1. First, read the SemEval-2014 Task 9 "task description" paper. This paper describes last year's SemEval task, which is quite similar to the task you have been working on as part of SemEval-2015's Task 10. This paper will give you an overview of the task and briefly summarize all of the systems that were submitted as part of the task. This will help give you an idea of what worked and what did not.

    The task description paper, describing the task and summarizing the submissions, is here:
    S14-2009: Sara Rosenthal; Alan Ritter; Preslav Nakov; Veselin Stoyanov, SemEval-2014 Task 9: Sentiment Analysis in Twitter


  2. Next, read at least one paper describing a top scoring system. As some of you have already seen in your system runs, sometimes one system is better than another system just by some odd parameter settings that happened to work. For that reason, there is no "winner" of the task, just some system that had the highest score. Also, there were multiple subtasks, including using a subset of tweets with sarcasm and a subset of LiveJournal entries in place of tweets. The ordering of the systems relative to those data sets is not the same as the ordering on the generic task as shown below. All that said, the systems that finished with the highest scores are more likely to be better examples for you to model than those systems that didn't do as well.

  3. Next, read at least one of the papers describing some other paper in the list, choosing one whose title sounds interesting or one that did well in the sarcasm or LiveJournal subtask, or one that had the highest average across the subtasks, or just randomly. If you know other people in the class, ask them what they are reading and pick something different. (This won't eliminate all duplication, but it will help a little.)

  4. Write a one paragraph summary of each of the three papers you just read. Write just enough so that it's helpful for both of us: for me to know you read and understood it, and for you as a reference.

  5. Next, add something to your system that you read about. For example, you can add more features that others have used. Or you could add another classifier. (You don't have to write that classifier by hand if you don't want to, but if you want to use another classifier, you will likely have to find the software and install it yourself. Ask me for help if you need it.) There are many choices here and no right or wrong choice. Ideally, you'd like to make a choice that improves your system the most, so try not to pick something that you think is trivial with no impact. However, if what you end up picking really doesn't improve your system (or makes it worse), that's fine! Be sure to cite which paper gave you the idea(s) that you used to augment your system.
  6. Finally, change your system so that you can run it in one of two ways:
    1. python3 classifier.py training_file n
    2. python3 classifier.py training_file test_file
    Running it the first way will do n-fold cross-validation on the training file specified. Running it the second way will use the first file as training and the second file as testing: no cross-validation.

Here are the system papers describing the systems submitted for Task 9. They are sorted below by their score on Task B. (See the task paper for more details on the ranking.)

  1. S14-2111: Yasuhide Miura; Shigeyuki Sakaki; Keigo Hattori; Tomoko Ohkuma, TeamX: A Sentiment Analyzer with Enhanced Lexicon Mapping and Weighting Scheme for Unbalanced Data
  2. S14-2033: Duyu Tang; Furu Wei; Bing Qin; Ting Liu; Ming Zhou, Coooolll: A Deep Learning System for Twitter Sentiment Classification
  3. S14-2086: Tobias Gunther; Jean Vancoppenolle; Richard Johansson, RTRGO: Enhancing the GU-MLT-LT System for Sentiment Analysis of Short Messages
  4. S14-2077: Xiaodan Zhu; Svetlana Kiritchenko; Saif Mohammad, NRC-Canada-2014: Recent Improvements in the Sentiment Analysis of Tweets
  5. S14-2120: Silvio Amir; Miguel B. Almeida; Bruno Martins; Joao Filgueiras; Mario J. Silva, TUGAS: Exploiting unlabelled data for Twitter sentiment analysis
  6. S14-2025: Joao Leal; Sara Pinto; Ana Bento; Hugo Goncalo Oliveira; Paulo Gomes, CISUC-KIS: Tackling Message Polarity Classification with a Large and Diverse Set of Features
  7. S14-2089: Nikolaos Malandrakis; Michael Falcone; Colin Vaz; Jesse James Bisogni; Alexandros Potamianos; Shrikanth Narayanan, SAIL: Sentiment Analysis using Semantic Similarity and Contrast Features
  8. S14-2015: Martin Jaggi; Fatih Uzdilli; Mark Cieliebak, Swiss-Chocolate: Sentiment Detection using Sparse SVMs and Part-Of-Speech n-Grams
  9. S14-2106: Alexandre Denis; Samuel Cruz-Lara; Nadia Bellalem; Lotfi Bellalem, Synalp-Empathic: A Valence Shifting Hybrid System for Sentiment Analysis
  10. S14-2115: Cicero dos Santos, Think Positive: Towards Twitter Sentiment Analysis from Scratch
  11. S14-2096: Stefan Evert; Thomas Proisl; Paul Greiner; Besim Kabashi, SentiKLUE: Updating a Polarity Classifier in 48 Hours
  12. S14-2062: Oliver Durr; Fatih Uzdilli; Mark Cieliebak, JOINT_FORCES: Unite Competing Sentiment Classifiers with Random Forest
  13. Paper missing
  14. S14-2015: Rafael-Michael Karampatsis; John Pavlopoulos; Prodromos Malakasiotis, AUEB: Two Stage Sentiment Analysis of Social Network Messages
  15. S14-2029: Sabih Bin Wasi; Rukhsar Neyaz; Houda Bouamor; Behrang Mohit, CMUQ$@$Qatar:Using Rich Lexical Features for Sentiment Analysis on Twitter
  16. S14-2070: Cynthia Van Hee; Marjan Van de Kauter; Orphee De Clercq; Els Lefever; Veronique Hoste, LT3: Sentiment Classification in User-Generated Content Using a Rich Feature Set
  17. S14-2031: Sara Rosenthal; Kathy McKeown; Apoorv Agarwal, Columbia NLP: Sentiment Detection of Sentences and Subjective Phrases in Social Media
  18. S14-2071: David Vilares; Miguel Hermo; Miguel A. Alonso; Carlos Gomez-Rodriguez; Yerai Doval, LyS: Porting a Twitter Sentiment Analysis Approach from Spanish to English
  19. S14-2074: Pedro Balage Filho; Lucas Avanco; Thiago Pardo; Maria das Gracas Volpe Nunes, NILC USP: An Improved Hybrid System for Sentiment Analysis in Twitter Messages
  20. S14-2095: Jose Saias, Senti.ue: Tweet Overall Sentiment Classification Approach for SemEval-2014 Task 9
  21. S14-2126: Lucie Flekova; Oliver Ferschke; Iryna Gurevych, UKPDIPF: Lexical Semantic Approach to Sentiment Polarity Prediction in Twitter Data
  22. Same as 21
  23. S14-2103: Boris Velichkov; Borislav Kapukaranov; Ivan Grozev; Jeni Karanesheva; Todor Mihaylov; Yasen Kiprov; Preslav Nakov; Ivan Koychev; Georgi Georgiev, SU-FMI: System Description for SemEval-2014 Task 9 on Sentiment Analysis in Twitter
  24. S14-2042: Jiang Zhao; Man Lan; Tiantian Zhu, ECNU: Expression- and Message-level Sentiment Orientation Classification in Twitter Using Multiple Effective Features
  25. Same as 24
  26. Paper missing
  27. S14-2026: Pablo Gamallo; Marcos Garcia, Citius: A Naive-Bayes Strategy for Sentiment Analysis on English Tweets
  28. S14-2028: Kamla Al-Mannai; Hanan Alshikhabobakr; Sabih Bin Wasi; Rukhsar Neyaz; Houda Bouamor; Behrang Mohit, CMUQ-Hybrid: Sentiment Classification By Feature Engineering and Parameter Tuning
  29. Same as 27
  30. S14-2067: Beakal Gizachew Assefa, KUNLPLab:Sentiment Analysis on Twitter Data
  31. Same as 20
  32. Paper missing
  33. S14-2017: Nadia Silva; Estevam Hruschka; Eduardo Hruschka, Biocom Usp: Tweet Sentiment Analysis with Adaptive Boosting Ensemble
  34. S14-2035: Julio Villena-Roman; Janine Garcia-Morera; Jose Carlos Gonzalez-Cristobal, DAEDALUS at SemEval-2014 Task 9: Comparing Approaches for Sentiment Analysis in Twitter
  35. (35)S14-2057: VIKRAM SINGH; Arif Md. Khan; Asif Ekbal, Indian Institute of Technology-Patna: Sentiment Analysis in Twitter
  36. Paper missing
  37. S14-2048: Javi Fernandez; Yoan Gutierrez; Jose Manuel Gomez; Patricio Martinez-Barco, GPLSI: Supervised Sentiment Analysis in Twitter using Skipgrams
  38. S14-2023: David Pinto; Darnes Vilarino; Saul Leon; Miguel Jasso; Cupertino Lucero, BUAP: Polarity Classification of Short Texts
  39. S14-2091: Akriti Vij; Nishta Malhotra; Naveen Nandan; Daniel Dahlmeier, SAP-RI: Twitter Sentiment Analysis in Two Days
  40. S14-2130: Pedro Aniel Sanchez-Mirabal; Ya relis Ruano Torres; Suilen Hernandez Alvarado; Yoan Gutierrez; Andres Montoyo; Rafael Munoz, UMCC DLSI: Sentiment Analysis in Twitter using Polirity Lexicons and Tweet Similarity
  41. Paper missing
  42. Paper missing
  43. S14-2113: Hussam Hamdan; Patrice Bellot; Frederic Bechet, The Impact of Z-score on Twitter Sentiment Analysis
  44. Paper missing
  45. S14-2100: Eugenio Martinez-Camara; Salud Maria Jimenez-Zafra; Maite Martin; L. Alfonso Urena Lopez, SINAI: Voting System for Twitter Sentiment Analysis
  46. S14-2054: Raja Selvarajan; Asif Ekbal, IITPatna: Supervised Approach for Sentiment Analysis in Twitter
  47. S14-2136: Richard Townsend; Aaron Kalair; Ojas Kulkarni; Rob Procter; Maria Liakata, University of Warwick: SENTIADAPTRON - A Domain Adaptable Sentiment Analyser for Tweets - Meets SemEval
  48. Same as 40
  49. Same as 47
  50. Same as 34

Note: I downloaded and ordered the papers myself and I may have accidentally omitted, misordered, or misrepresented one of them. If you think I did any of these things, let me know so I can fix it.

These systems only participated in Task A: