Categorygithub.com/github-vet/botscmdtrack-bot
package
0.0.0-20210109214533-c7609e90cd61
Repository: https://github.com/github-vet/bots.git
Documentation: pkg.go.dev
Overview
Versions
1
Dependencies
17
Dependents
0
Files
828 SLOC

# README

TrackBot

TrackBot is responsible for automating the crowd-sourcing workflow.

Overview

Once an issue is found, crowd-sourcing is used to classify it into one of several categories. This classification is handled via a combination of expert opinion and community effort. Gophers can provide their assessment of each issue's classification by adding emoji reactions to the issues.

TrackBot periodically scans every issue in the repository, checking the issue reactions and the issue labels. In the course of a scan, it does a few things.

  1. Assess Expert Opinion
  2. Assess Community Opinion
  3. Detect and Alert on Expert Disagreement
  4. Close Issues

1. Assess Expert Opinion

TrackBot maintains a list of the GitHub usernames of experts who are trusted to render a careful opinion. When TrackBot finds that an expert has reacted to an issue, a few things happen.

  1. A label is added to the issue to indicate the expert opinion.
  2. The reliability of community members is updated based on whether or not they agree with the expert assessment.
  3. If 2 or more experts agree, the issue is closed.

2. Assess Community Opinion

TrackBot uses the reactions left by non-experts to update a community opinion. TrackBot keeps track of how frequently each community member has made an assessment that concurs with the assessment left by an expert. It uses this information to compute an overall reliability score, and sums this score over all users who have reacted to the issue. If the community score is high enough, a few things happen.

  1. If a baseline reliability threshold is met and more than 80% of the community votes (weighted by reliability) are assigned to a single classification, a label is added to the issue to indicate the community opinion.
  2. If no single classification receives more than 80% of the total community votes (weighted by reliability), a label is added to indicate the issue is confusing and may warrant higher scrutiny by an expert.
  3. If the community reliability score exceeds a high reliability threshold, the issue is labeled as 'reliable'.

3. Assess and Alert on Expert Disagreement

TrackBot notices when experts leave conflicting opinions and takes action by leaving a comment to alert them to the issue using an @ mention. Experts are expected to discuss and resolve any disagreement. TrackBot also labels the issue to indicate expert confusion in this case. Any confusion is not resolved until all experts can agree.

4. Close Issues

TrackBot also closes issues it finds which have the test or vendored label, and scans the file path to apply these labels if they are not already present.

# Functions

AddLabel adds the provided label to the issue, if it is present.
GetAllReactions retrieves the set of reactions on an issue, paging through the API as needed to make sure they are all retrieved.
HandleExpertAgreement handles the case where all the experts who have weighed in on the issue agree.
HandleExpertDisagreement handles the case where experts who have weighed in on the issue do not agree.
HasLabel returns true if the issue has a matching label.
MaybeCloseIssue closes the issue if the number of experts who have provided their assessment exceeds the threshold.
MaybeCloseIssueByLabel closes the issue if it does not need to be considered.
No description provided by the author
ProcessAllIssues processes all pages of issues found in the provided repository and looks for updates.
ProcessIssuePage processes one page of issues from GitHub.
ReadExpertsFile opens the provided file and parses the contents into a map of Experts, keyed by username.
ReadGophersFile opens the file at the provided path and reads it into a map of Gophers, keyed by username.
ReadIssuesFile reads a map of issues keyed by issue ID from the provided file.
RemoveLabel removes the provided label from the issue, if it is present.
Score retrieves the score of the Gopher from the map.
SetCommunityLabel adds or overwrites the community label associated with the provided assessment.
SetExpertLabel adds or overwrites the expert label associated with the provided assessment.
ThrottleExperts posts a comment on the issue mentioning the experts to draw attention to their disagreement and start a transparent conversation.
UpdateCommunityAssessment updates the overall community assessment based on the reliability of all the users involved.
UpdateIssueReactions updates the set of reactions associated with a single issue.
WriteExpertsFile writes the provided list of experts to the provided path, truncating whatever file may exist.
WriteGophersFile writes the provided map of gophers to the given file, truncating its contents.
WriteIssuesFile writes the map of issues to the provided file, truncating its contents in the process.

# Constants

AnonymousScore is the score of a gopher who has never had one of their votes assessed.
CommunityScoreThreshold marks the minimum reliability score needed on an issue before it will have a community label applied.
ConfusionThreshold marks the fraction of total score applied to an issue needed before the community is considered 'confused'.
DisagreementTemplate is the template used to comment whenever experts disagree on the outcome of an issue.
HighCommunityScoreThreshold marks the threshold needed before the 'reliable' label is applied.
MinExpertsNeededToClose controls the number of experts who must react before the issue is marked as closed.

# Variables

CloseTestIssues is a flag used to close any issues found which are in a test file.
ValidReactions lists the set of reactions which 'count' as input for the purpose of analysis.

# Structs

DisagreementData describes data for the Disagreement template.
Expert describes a GitHub user marked as an 'expert' for the purpose of crowd sourcing.
Gopher represents a GitHub user not marked as an 'expert' for the purpose of crowdsourcing.
Issue is a local record of an issue being tracked by this bot.
OptSchema defines a configuration option which can come either from the command-line or environment variables.
TrackBot stores all relevant state needed to run the TrackBot.