Can we measure sentiment changes towards policy through social media?
The overarching objective is to reveal trends and changes in response to relevant events or policies in the broader prosecutorial justice sector.
Conventional methods are labor-intensive, time-lagged, and not easily scaled up. Alternatively, this research investigates the viability and strategies of leveraging Twitter posts (tweets) and Natural Language Processing (NLP) to assess public sentiments in an efficient, timely, scalable, and more consistent way. With this tool, we hope to assist stakeholders gather massive responses online and respond to real-time events promptly.
Community support plays a crucial role in the effectiveness of prosecutorial justice sector and law enforcement. However, measuring public sentiments towards relevant policy implementations is difficult. Conventional methods are labor-intensive, time-lagged, and not scalable. Alternatively, we investigate the viability and strategies of leveraging social network services to assess public sentiments in a massive, timely, and effective manner.
Focusing on Twitter, we first evaluate strategies to harvest relevant tweets and mitigate contamination. This includes keywords selection and streaming pipeline modification. Then, using the collected tweets regarding prosecutorial justice departments at both local and national levels, we assess whether a sentiment baseline exists based on the positive, neutral, or negative sentiments perceived by existing Natural Language Processing models. Finally, we investigate geographical and longitudinal variations in sentiments.
The results show that measuring such sentiments is challenging.
- First, Twitter’s free Streaming API is not ideal qualitatively nor quantitatively for collecting the desired data.
- Second, the data is highly contaminated due to the inevitably ambiguous keywords.
- Finally, most tweets appear either neutral or slightly negative in the two models respectively.
However, it is possible to observe and compare longitudinal trends and changes instead of absolute values. Recommendations for similar research pipeline are then concluded.
This tool demonstrates the longitudinal sentiment scores of tweets related to preosecutorial justice sector of different states. The sentiments are assessed by TextBlob and CoreNLP Python modules respectively, with lower scores indicating more negative overall sentiment of the tweet.
- New York
- District of Columbia
Based on our results, concrete baselines for public sentiments towards prosecutorial justice sector are currently unavailable through tweets. However, changes in public sentiment can be observed. This may serve the needs of assessing community responses to events and policies. While our results showed most changes were within the margin of error, this does not rule out the possibility that statistically significant changes in sentiment could be detected.
One limitation we faced was that we could not acquire a search API from Twitter, limiting use in three ways. One, we were limited in the number of tweets to 1 per thousand. Two, we could not set restrictions to tweets containing a combination of search terms. Three, we could not search historical tweets, limiting our sample to an 8-week period. With a search API or firehose, it would be possible to get a larger and better quality sample that would reduce the error range of the sample and having a longer time period of data would increase the probability of some change in sentiment occuring during that time.
We suggest following recommendations for conducting a sentiment analysis in similar topics. First, the strategy can vary by the type of Twitter API available. However, unless the API is unlimited, one may optimize the query using a Streaming API before consuming the quota. The keywords associated with this topic are inevitably general. One needs to check the retrieved tweets and update the keywords addressing major contamination. If operating with Streaming API, one may use more specific keywords (e.g., names of the program or event) to mitigate excessive irrelevant tweets. With a purchased Search API, one has the privilege to use the intersection operator and geotag information for more precise queries.
Gaurav Bhardwaj, Yuwen Chang, Scott Smith, and Yukun Wan are masters student's at New York University's Center for Urban Science + Progress (CUSP). Federica Bianco is a Senior Research Scientist at CUSP and Assistant Professor of Informatics at NYU. BetaGov applies pragmatic randomized controlled studies in order to assess the effectiveness of public sector initiatives in areas such as education, healthcare, and criminal justice.
This is a collaborative project between CUSP and the sponsor BetaGov. BetaGov has been working with prosecutorial justice departments across the United States to implement innovative solutions. One example is the police-to-peace initiative that includes ”peace officer” on patrol cars, expecting to improve community sentiments towards law enforcement units. The arising question is to know if these implementations are taking effect as expected.