====== 6.0 Dev Cycle Outline  ======



Outline of Development Items (Click on the 'Asana' items to see the task in Asana)

\\


1. Improve the shingler system based on the de-duplication work that Tom and Loretta did for the project.

  * Research to see if there is a way to identify more duplicates without compromising processing efficiency.  If there is a solution, we will try to implement it.  ([[https://app.asana.com/0/1200147074266926/1202347518830353|Asana]])
  * Develop a stand-alone article de-duplication tool that Cline Center staff can use on a specified set of articles ([[https://app.asana.com/0/1200147074266926/1201908431870085|Asana]])
  * **Who it impacts**:  Analyst Ops, Cyber-infrastructure, Analytics and Data Management

2. Identify a solution to store duplicates without taking up so much disk space.  ([[https://app.asana.com/0/1200147074266926/1201188747166962|Asana]]) 

  * Articles that are exact duplicates will have their content stripped.
  * Content from articles that are ‘near-duplicates’ is retained and stored in the MongoDB dedicated for duplicate articles.
  * Feature request: Is there a way we can make this information useful? (Maybe count the number of ‘duplicates’ and near-duplicates that are associated with an article 
  * **Who it impacts**: Cyber-infrastructure

3. Lodestar bug fixes ([[https://app.asana.com/0/1200147074266926/1201782310775351|Asana]]) (Already been addressed)

  * Fix login issue (or lack of login ability) from GNC 5.1.1 ([[https://app.asana.com/0/1200147074266926/1201710481263356|Asana]])
  * Fix the stats reporting issue from GNC 5.1.1 ([[https://app.asana.com/0/1200147074266926/1201751733582156|Asana]])
  * Schedule an early deployment
  * **Who it impacts**: Cyber-infrastructure and Analyst Ops

4. Clean up the Gradle builds [[https://app.asana.com/0/1200147074266926/1201669321047905|(Asana]]) (This has already been addressed)

  * **Who it impacts**: Cyber-infrastructure

5. Addressing GNI Features and Bugs

Fix issues with the GNI

Fix issues identified to clean text of content ([[https://app.asana.com/0/1200147074266926/1200513047131467|Asana]])

Better organize and present Bulk LexisNexis data (Asana).

Remove source_name “LexisNexis” ([[https://app.asana.com/0/1200147074266926/1202357876842158|Asana]])

Resolve issues with the processing of new BLN data

Resolve issues with empty values

  * Empty web content ([[https://app.asana.com/0/1200147074266926/1202357876842160|Asana]])
  * Url_host blank
  * Missing URL values for web ([[https://app.asana.com/0/1200147074266926/1202357876842156|Asana]])
  * Missing title values

Add new features to GNI  

Add a feature to add new BLN sources ([[https://app.asana.com/0/1200147074266926/1202326478875380|Asana]])

Add word count (or single word token counts) variables for Content and Title ([[https://app.asana.com/0/1200147074266926/1201922300180157|Asana]])

Document-term matrices as an output? ([[https://app.asana.com/0/1200147074266926/1201922300180159|Asana]])

  * Question: Do we want the DTM to be a single token or multiple tokens?
  * User-defined? With limits?

Add text complexity variables?  If so, what? ([[https://app.asana.com/0/1200147074266926/1202357876842166|Asana]])

  * Lexical Richness (aka Type Token Ratio): example: [[https://pypi.org/project/lexicalrichness/|https:%%//%%pypi.org/project/lexicalrichness/]] 
  * Hapax Richness (# Words Occurring Once / # of Total Words)
  * Flesch’s reading ease test:  The formula for Flesch reading ease score is: 206.835 - 1.015 × (total words ÷ total sentences) - 84.6 × (total syllables ÷ total words)
  * Flesch-Kincaid equation: Flesch-Kincaid grade level formula: 0.39 x (words/sentences) + 11.8 x (syllables/words) - 15.59.

Update Moral Foundation sentiment using Diesner’s ‘Expanded Moral Foundations’ dictionary ([[https://app.asana.com/0/1200147074266926/1202357876842168|Asana]]): [[https://dl.acm.org/doi/abs/10.1145/3465336.3475112|https:%%//%%dl.acm.org/doi/abs/10.1145/3465336.3475112]] 

Extracted Quotes and speaker variables (assuming we use Stanford CoreNLP?) - Nice to have but not required. ([[https://app.asana.com/0/1200147074266926/1202357876842178|Asana]])

  * Maybe save the speaker of the quote.
  * Save the parse tree and store it in a separate mongo.

Web

Explore adding new sources to the web crawler.  If we can add them, then add them. ([[https://app.asana.com/0/1200147074266926/1202391286800748|Asana]])

  * Closed captions from news sources?
  * Pull articles from fact-checking sites

Add a new variable (or maybe use the ‘publisher’ field) to denote the source of web content instead of using simply the URL field. ([[https://app.asana.com/0/1200147074266926/1202357876842176|Asana]])

Improve the performance of the SPEED Classifier (Not required).  Probably need to focus on this in the next dev cycle) ([[https://app.asana.com/0/1200147074266926/1202357876842170|Asana]])

Add new content:  ([[https://app.asana.com/0/1200147074266926/1202357876842172|Asana]])

  * Eastview FBIS
  * Newsreel data

Add URLs for BulkLexisNexis_v2 sources (for users who do not have access to the content) ([[https://app.asana.com/0/1200147074266926/1202357876842174|Asana]])

**Who it impacts**: Cyber-infrastructure, Analyst Ops, Analytics and Data Management, External Engagement, Users of Cline Center GNC data

7. Transition from Illinois NER over to Stanford CoreNLP or some other alternative (i.e. spaCy). ([[https://app.asana.com/0/1200147074266926/1202357876842152|Asana]])

  * Question: Before we transition, what kind of license does CoreNLP have?  Can we commercialize it? License is [[http://www.gnu.org/licenses/gpl.html|GNU General Public License V3]] so it is ok to proceed
  * **Who it impacts**: Cyber-infrastructure, Analyst Ops, Analytics and Data Management, External Engagement, Users of Cline Center GNC data

8. Improve Geocoder based on findings from Event Detection work that Buddy and Loretta are working on ([[https://app.asana.com/0/1200147074266926/1201041045198981|Asana]])

Update the Geonames database that the geocoder uses (done)

Implement pattern-finding approaches to identify geolocations

Improve the list of potential geolocations that the geocoder provides for a location entity that has been identified in a text

  * Remove low probability locations using a standard deviation.

Create a new field with the top choice (based on score) for a predicted location.

Access how well the geo-coder does use the Universal Gold Standard event data and against other off-the-shelf geocoders available

  * [[https://cliff.mediacloud.org/|CLIFF/CLAVIN]]
  * [[https://github.com/openeventdata/mordecai|Mordecai ]]

**Who it impacts**: Cyber-infrastructure, Analyst Ops, Analytics and Data Management, External Engagement, Users of Cline Center GNC data

9. Archer Updates

Migrate Archer UI to new Vaadin ([[https://app.asana.com/0/1200147074266926/1201188747166962|Asana]])

Archer UI Updates

Feature Requests

Required

  * Incorporate Google Analytics ([[https://app.asana.com/0/1200147074266926/1199589601366934|Asana]])

Not required but would be nice to have ([[https://app.asana.com/0/1200147074266926/1202391286800750|Asana]])

  * Implementing Shibboleth authentication during login?
  * Edit Snapshot names
  * Ability to plot average sentiment over time
  * Ability to plot multiple items on a timeline
  * Ability to show Solr’s relevance score in the dashboard and enable a user to download it and sort with it
  * Ability to view, plot, and download word counts for Content and Title
  * Use a different delimiter for delimiting values within the Dashboard

Bug fixes

  * Can only download the results from the first query I download data from ([[https://app.asana.com/0/1200147074266926/1167622169026657|Asana]])
  * Add content_lenth and title_length to items that users can view in the dashboard, download, and plot.  Can we plot these two measures as well? ([[https://app.asana.com/0/1200147074266926/1202391286800759|Asana]])

Archer API Features ([[https://app.asana.com/0/1183004794810859/1201132265181561|Asana]]) (Already been addressed)

  * Term Queries
  * Stats Queries
  * Debug Query Feature
  * GroupBy Query

**Who it impacts**: Cyber-infrastructure, Analyst Ops, Analytics and Data Management, External Engagement, Users of Cline Center GNC data

10. MongoDB Items

  * Add authentication for MongoDB backend access ([[https://app.asana.com/0/1200147074266926/1201089022380727|Asana]])
  * Evaluate Mongo Read performance and try to improve on it ([[https://app.asana.com/0/1200147074266926/1201385068285917|Asana]])
  * **Who it impacts**: Cyber-infrastructure

11. Storm Cluster items

  * Move Storm UI and command-line off of the cluster machines and move them to dedicated virtual machines. (Nice to have if possible) ([[https://app.asana.com/0/1200147074266926/1200449602478702|Asana]])
  * Upgrade the Apache Storm System to 2.2.0 ([[https://app.asana.com/0/1200147074266926/1202015361856736|Asana]])
  * **Who it impacts**: Cyber-infrastructure

12. Retire ‘production’ VM ([[https://app.asana.com/0/1200147074266926/1201831854657609|Asana]])

Resolve CCP portal.  We need to do one of the following:

  * Migrate to new VM 
  * Hand off the portal to CCP

Figure out where to store Presidential Campaigns Data

Migrate the MySQL database that is on this VM

**Who it impacts**: Cyber-infrastructure, CCP, Scott Althaus, Analytics and Data management

\\


**Expected Deployment Date**: May 2023

\\


VMs that will be retired and what they will be replaced by

  - Staging-5-prod -> Staging-6-prod
  - Solr-5-prod -> Solr-6-prod 
  - Nexus-5-prod ->Nexus-6-prod
  - Mysql-5-prod -> Mysql-6-prod 
  - Mongo-5-prod -> Mongo-6-prod 
  - Gis-5-prod -> Gis-6-prod
  - Duplicates-5-prod -> Duplicates-6-prod
  - Production -> TBD if other VMs will need to be created

\\


Systems that will be impacted/updated:

  - Scout
  - Lodestar (again despite the recent update)
  - Archer
  - Archer API
  - Storm