# README
DevStats code
This is a code reposotory for DevStats used to display CNCF projects dashboards, CDF projects dashboards, GraphQL projects dashboards and example Kubernetes/helm deployment.
Authors: Łukasz Gryglicki [email protected], Justyna Gryglicka [email protected], Josh Berkus [email protected].
Building and installing
- Follow this guide to see how to deploy on Kubernetes using Helm.
- Follow this guide to see GraphQL deployment using Kubernetes & Helm.
- Follow this guide for installing on bare metal.
- Follow this guide to deploy your oiwn project on bare metal (this example deployes Homebrew statistics).
- Fetch dependency libraries.
makethenmake testfinallymake install.
# Functions
ActorHit - are we intereste din this actor?.
ActorIDOrNil - return Actor ID from pointer or nil.
ActorLoginOrNil - return Actor Login from pointer or nil.
AddNIntervals adds (using nextIntervalStart) or subtracts (using prevIntervalStart) N itervals to the given date Functions Next/Prev can use Hour, Day, Week, Month, Quarter, Year functions (defined in this module) or other custom defined functions With `func(time.Time) time.Time` signature.
AddTSPoint add single point to the batch.
ArtificialEvent - create artificial API event (but from the past).
ArtificialPREvent - create artificial API event (PR state for now()).
BoolOrNil - return either nil or value of boolPtr.
CleanUTF8 - clean UTF8 string to containg only Pq allowed runes.
ClearDBLogs clears logs older by defined period (in context.go) It clears logs on `devstats` database.
CommentIDOrNil - return Comment ID from pointer or nil.
CompareFloat64Ptr - compares two string pointers returns true if both pointers are null or both are non null and point to floats with the same value (with epsilon).
CompareIntPtr - compares two int pointers returns true if both pointers are null or both are non null and point to integers with the same value.
CompareStringPtr - compares two string pointers returns true if both pointers are null or both are non null and point to strings with the same content.
ComputePeriodAtThisDate - for some longer periods, only recalculate them on specific dates/times see: time_test.go.
CreateDatabaseIfNeeded - creates requested database if not exists Returns true if database was not existing existed and created dropped.
CreateDatabaseIfNeededExtended - creates requested database if not exists Returns true if database was not existing existed and created dropped Allows specifying additional database parameters.
CreateTable is used to replace DB specific parts of Create Table SQL statement.
DatabaseExists - checks if database stored in context exists If closeConn is true - then it closes connection after checking if database exists If closeConn is false, then it returns open connection to default database "postgres".
DayStart - return time rounded to current day start.
DeleteArtificialEvent - create artificial API event (but from the past).
DeleteArtificialPREvent - create artificial API event (but from the past).
DescriblePeriodInHours - return string description of a time period given in hours.
DropDatabaseIfExists - drops requested database if exists Returns true if database existed and was dropped.
EnvReplace - replace all environment variables starting with "prefix" with contents of variables with "suffix" added - if defined If prefix is "DB_" and suffix is "_SRC" then: So if there is "DB_HOST_SRC" variable defined - it will replace "DB_HOST" and so on.
EnvRestore - restores all environment variables given in the map.
ESConn Connects to ElasticSearch.
ExecCommand - execute command given by array of strings with eventual environment map.
ExecSQL executes given SQL on Postgres DB (and return single state result, that doesn't need to be closed).
ExecSQLTx executes given SQL on Postgres DB (and return single state result, that doesn't need to be closed) It is for running inside transaction.
ExecSQLTxWithErr wrapper to ExecSQLTx that exists on error It is for running inside transaction.
ExecSQLWithErr wrapper to ExecSQL that exists on error.
Fatalf - it will call FatalOnError using fmt.Errorf with args provided.
FatalNoLog displays error message (if error present) and exits program, should be used for very early init state.
FatalOnError displays error message (if error present) and exits program.
FirstIntOrNil - return either nil or value of intPtr.
ForkeeIDOrNil - return Forkee ID from pointer or nil.
ForkeeNameOrNil - return Forkee Name from pointer or nil.
ForkeeOldIDOrNil - return ForkeeOld ID from pointer or nil.
GetAnnotations queries uses `git` to get `orgRepo` all tags list for all tags and returns those matching `annoRegexp`.
GetDateAgo returns date: 'from' - 'n hours/days' etc.
GetFakeAnnotations - returns 'startDate - joinDate' and 'joinDate - now' annotations.
GetFloatFromInterface if an interface is of numeric type, return its value as float64.
GetHidden - return list of shas to replace.
GetIntervalFunctions - return interval name, interval number, interval start, next, prev function from interval abbr: h|d2|w3|m4|q|y w3 = 3 weeks, q2 = 2 quarters, y = year (1), d7 = 7 days (not the same as w), m3 = 3 months (not the same as q).
GetProjectsList return list of projects to process It sorts them according to projects.yaml order Handles disabled/enabled projects Handles ONLY="project1 project2 ..
GetRateLimits - returns all and remaining API points and duration to wait for reset when core=true - returns Core limits, when core=false returns Search limits.
GetRecentRepos - get list of repos active last day.
GetTagValues returns tag values for a given key.
GetThreadsNum returns the number of available CPUs If environment variable GHA2DB_ST is set it retuns 1 It can be used to debug single threaded verion It runs on 95% CPU power by default.
GHClient - get GitHub client.
HandlePossibleError - display error specific message, detect rate limit and abuse.
HashArray takes []interface{} and returns hash string from given keys from map.
HashObject takes map[string]interface{} and keys from []string and returns hash string from given keys from map.
HashStrings - returns unique Hash for strings array This value is supposed to be used as ID (negative) to mark it was artificially generated.
HourStart - return time rounded to current hour start.
InsertIgnore - will return insert statement with ignore option specific for DB.
IntOrNil - return either nil or value of intPtr.
IsProjectDisabled - checks if project is disabled or not: fullName comes from makeOldRepoName for pre-2015 data! yamlDisabled (this is from projects.yaml - can be true or false) it also checks context (which can override `disabled: true` from projects.yaml) +pro1,-pro2 creates map {"pro1":true, "pro2":false}.
IssueIDOrNil - return Issue ID from pointer or nil.
MakeOldRepoName - before 2015 repository name should be Organization/Name (if Organization present) or just Name.
MakeUniqueSort - make string array unique & sorted.
MapFromString - returns maps from string formatted as map[k1:v1 k2:v2 k3:v3 ...].
MaybeHideFunc - use closure as a data storage.
Mgetc waits for single key press and return character pressed.
MilestoneIDOrNil - return Milestone ID from pointer or nil.
MonthStart - return time rounded to current month start.
NegatedBoolOrNil - return either nil or negated value of boolPtr.
NewTSPoint returns new point as specified by args.
NextDayStart - return time rounded to next day start.
NextHourStart - return time rounded to next hour start.
NextMonthStart - return time rounded to next month start.
NextQuarterStart - return time rounded to next quarter start.
NextWeekStart - return time rounded to next week start.
NextYearStart - return time rounded to next year start.
NormalizeName - clean DB string from ", ', -, /, ., " ",), (, ], [ trim leading and trailing space, lowercase Normalize Unicode characters.
NValue will return $n.
NValues will return values($1, $2, .., $n).
ObjectToJSON - serialize given object as JSON.
ObjectToYAML - serialize given object as YAML.
OrgIDOrNil - return Org ID from pointer or nil.
OrgLoginOrNil - return Org ID from pointer or nil.
PeriodParse - tries to parse period.
PgConn Connects to Postgres database.
PgConnDB Connects to Postgres database (with specific DB name) uses database 'dbname' instead of 'PgDB'.
PrepareQuickRangeQuery Perpares query using either ready `period` string or using `from` and `to` strings Values to replace are specially encoded {{period:alias.column}} Can either replace with: (alias.column >= now() - 'period'::interval) Or (alias.column >= 'from' and alias.column < 'to').
PrettyPrintJSON - pretty formats raw JSON bytes.
PrevDayStart - return time rounded to prev day start.
PrevHourStart - return time rounded to prev hour start.
PrevMonthStart - return time rounded to prev month start.
PrevQuarterStart - return time rounded to prev quarter start.
PrevWeekStart - return time rounded to prev week start.
PrevYearStart - return time rounded to prev year start.
Printf is a wrapper around Printf(...) that supports logging.
ProcessAnnotations Creates IfluxDB annotations and quick_series.
ProcessTag - insert given Tag into Postgres TSDB.
ProgressInfo display info about progress: i/n if current time >= last + period If displayed info, update last.
PullRequestIDOrNil - return PullRequest ID from pointer or nil.
QuarterStart - return time rounded to current month start.
QueryRowSQL executes given SQL on Postgres DB (and returns single row).
QueryRowSQLTx executes given SQL on Postgres DB (and returns single row).
QuerySQL executes given SQL on Postgres DB (and returns rowset that needs to be closed).
QuerySQLTx executes given SQL on Postgres DB (and returns rowset that needs to be closed) It is for running inside transaction.
QuerySQLTxWithErr wrapper to QuerySQLTx that exists on error It is for running inside transaction.
QuerySQLWithErr wrapper to QuerySQL that exists on error.
ReadFile tries to read any filename, but have a fallback it attempts to replace current project name with shared: /proj/ -> /shared/ This is to allow reading files that can be shared between projects.
ReleaseIDOrNil - return Release ID from pointer or nil.
RepoHit - are we interested in this org/repo ?.
RepoIDOrNil - return Repo ID from pointer or nil.
RepoNameOrNil - return Repo Name from pointer or nil.
SkipEmpty - skip one element arrays contining only empty string This is what strings.Split() returns for empty input We expect empty array or empty map returned in such cases.
Slugify replace all whitespace with "-", remove all non-word letters downcase.
StringOrNil - return either nil or value of strPtr.
StringsMapToArray this is a function that calls given function for all array items and returns array of items processed by this func Example call: lib.StringsMapToArray(func(x string) string { return strings.TrimSpace(x) }, []string{" a", " b ", "c "}).
StringsMapToSet this is a function that calls given function for all array items and returns set of items processed by this func Example call: lib.StringsMapToSet(func(x string) string { return strings.TrimSpace(x) }, []string{" a", " b ", "c "}).
StringsSetKeys - returns all keys from string map.
StripUnicode strip non-unicode and control characters from a string From: https://rosettacode.org/wiki/Strip_control_codes_and_extended_characters_from_a_string#Go.
Structure creates full database structure, indexes, views/summary tables etc.
SyncIssuesState synchonizes issues states manual: false: normal devstats sync cron mode using 'ghapi2db' tool true: manual sync using 'sync_issues' tool.
TableColumnExists - checks if a given table's has a given column.
TableExists - checks if a given table exists.
TimeOrNil - return either nil or value of timePtr.
TimeParseAny - attempts to parse time from string YYYY-MM-DD HH:MI:SS Skipping parts from right until only YYYY id left.
ToGHADate - return time formatted as YYYY-MM-DD-H.
ToYMDDate - return time formatted as YYYY-MM-DD.
ToYMDHDate - return time formatted as YYYY-MM-DD HH.
ToYMDHMSDate - return time formatted as YYYY-MM-DD HH:MI:SS.
TruncStringOrNil - return either nil or value of strPtr truncated to maxLen chars.
TruncToBytes - truncates text to <= size bytes (note that this can be a lot less UTF-8 runes).
WeekStart - return time rounded to current week start Assumes first week day is Sunday.
WriteTSPoints write batch of points to postgresql use mergeSeries = "name" to put all series in "name" table, and create "series" column that conatins all point names.
YearStart - return time rounded to current month start.
# Constants
Abuse - common constant string.
All - common constant string.
DefaultDataDir - common constant string.
Devstats - common constant string.
DevstatsCode - common constant string.
EngineIsClosedError - common constant string.
GHA - common constant string.
GHAAdmin - common constant string.
HideCfgFile - common constant string.
InvalidCatalogName - common constant string.
Kubernetes - common constant string.
LocalGitScripts - common constant string.
Localhost - common constant string.
Merged - common constant string.
Metrics - common constant string.
Nil - common constant string.
NotFound - common constant string.
Now - common constant string.
Null - common constant string.
Password - common constant string.
Quarter - common constant string.
Retry - common constant string.
TimeCol - common constant string.
TimeoutError - common constant string.
Today - common constant string.
Unset - common constant string.
# Structs
Actor - GHA Actor structure Name is unexported and not used by JSON load/save But is used when importing affiliations from cncf/gitdm:github_users.json.
AllProjects contain all projects data.
Annotation contain each annotation data.
Annotations contain list of annotations.
Asset - GHA Asset structure.
Author - GHA Commit Author structure.
Branch - GHA Branch structure.
Comment - GHA Comment structure.
Commit - GHA Commit structure.
Ctx - environment context packed in structure.
Dummy - structure with no data pointer to this struct is used to test if such field was present in JSON or not.
ES - ElasticSearch connection client, context and default mapping.
ESBulks keeps array of bulk services to add/delete each delete/add but can hold 10 items.
ESDataObject internal JSON data for stored documents.
Event - full GHA (GitHub Archive) event structure.
EventOld - full GHA (GitHub Archive) event structure, before 2015.
Forkee - GHA Forkee structure.
ForkeeOld - GHA Forkee structure (from before 2015) Handle missing 4 last properties (including two non-nulls!).
Issue - GHA Issue structure.
IssueConfig - holds issue data.
Label - GHA Label structure.
Milestone - GHA Milestone structure.
Org - GHA Org structure.
Page - GHA Page structure.
Payload - GHA Payload structure.
PayloadOld - GHA Payload structure (from before 2015).
Project contain mapping from project name to its command line used to sync it.
PullRequest - GHA Pull Request structure.
Release - GHA Release structure.
Repo - GHA Repo structure.
SkipDatesList holds a list of GHA hours to skip.
Tag contain each TSDB tag data.
Tags contain list of TSDB tags.
Team - GHA Team structure (only used before 2015).
TSPoint keeps single time series point.
# Type aliases
AnnotationsByDate annotations Sort interface.
AnyArray - holds array of interface{} - just a shortcut.
Int64Ary - sortable Int64 array.
IssueConfigAry - allows sorting IssueConfig array by IssueID annd then event creation date.
TSPoints keeps batch of TSPoint values to write.