dbhub-common

AddDatabase is handles database upload processing.

AddDefaultLicences adds the default licences to the PostgreSQL database.

AddDefaultUser adds the default user to the system, so the referential integrity of licence user_id 0 works.

AddUser adds a user to the system.

AnalysisRecordUserStorage adds a record to the backend database containing the amount of storage space used by a user.

AnalysisUsersWithDBs returns the list of users with at least one database.

ApiCallLog records an API call operation.

APIKeyDelete deletes an existing API key from the PostgreSQL database.

APIKeyGenerate generates a random API key and saves it in the database.

APIKeySave saves a new API key to the PostgreSQL database.

AuthorizerLive is a SQLite authorizer callback intended to allow almost anything.

AuthorizerSelect is a SQLite authorizer callback which only allows SELECT queries and their needed sub-operations to run.

BranchListResponse returns the branch list for a database.

CacheData caches data in Memcached.

CheckDBExists checks if a database exists.

CheckDBID checks if a given database ID is available, and returns its name so the caller can determine if it has been renamed.

CheckDBLive checks if the given database is a live database.

CheckDBPermissions checks if a database exists and can be accessed by the given user.

CheckDBStarred check if a database has been starred by a given user.

CheckDBWatched checks if a database is being watched by a given user.

CheckEmailExists checks if an email address already exists in our system.

CheckLicenceExists checks if a given licence exists in our system.

CheckUnicode checks if a given string is unicode, and safe for using in SQLite queries (eg no SQLite control characters).

CheckUserExists checks if a username already exists in our system.

ClearCache removes all items currently cached by Memcached, so it's like a newly started server.

CloseMQChannel closes an open AMQP channel.

CloseMQConnection closes an open AMQP connection.

CommitLicenceSHA returns the licence used by the database in a given commit.

CommitPublicFlag returns the public flag of a given commit.

ConnectCache connects to the Memcached server.

ConnectMinio parses the Minio configuration, to ensure it seems workable Note - this doesn't actually open a connection to the Minio server.

ConnectPostgreSQL creates a connection pool to the PostgreSQL server.

ConnectQueue creates the connections to the backend queue server.

CreateCommitID generate a stable SHA256 for a commit.

CreateDBTreeID generate the SHA256 for a tree Tree entry structure is: * [ entry type ] [ licence sha256] [ file sha256 ] [ file name ] [ last modified (timestamp) ] [ file size (bytes) ].

CreateRelease is used for creating a release when running tests.

CreateTag is used for creating a tag when running tests.

CypressSeed empties the backend database, then adds pre-defined test data (PostgreSQL and Minio).

DataValuesMatch compares two slices of DataValue objects.

DB4SDefaultList returns a list of 1) users with public databases, 2) along with the logged in users' most recently modified database (including their private one(s)).

DBDetails returns the details for a specific database.

DBDiff generates the differences between the two database files in dbA and dbB.

DBStars returns the star count for a given database.

DBWatchers returns the watchers count for a given database.

DefaultCommit returns the default commit ID for a specific database.

DeleteBranchHistory safely removes the commit history for a branch, from the head of the branch back to (but not including) the specified commit.

DeleteComment deletes a specific comment from a discussion.

DeleteDatabase deletes a database from PostgreSQL Note that we leave a stub/placeholder entry for all uploaded databases in PG, so our stats don't miss data over time and so the dependant table data doesn't go weird.

DeleteLicence removes a (user supplied) database licence from the system.

Diff generates the differences between the two commits commitA and commitB of the two databases specified in the other parameters.

DisconnectPostgreSQL disconnects the PostgreSQL database connections.

DiscussionComments returns the list of comments for a given discussion If a non-0 comID value is passed, it will only return the details for that specific comment in the discussion.

Discussions returns the list of discussions or MRs for a given database If a non-0 discID value is passed, it will only return the details for that specific discussion/MR.

DownloadDatabase returns the SQLite database file to the requester.

EnvProd changes the running environment to be "production" NOTE - This route to call this is only available when the server is _started_ in the "test" environment.

EnvTest changes the running environment to be "test" NOTE - This route to call this is only available when the server is _started_ in the "test" environment.

EscapeId puts an SQL identifier in quote characters and escapes any quote characters it contains, making it safe for use in SQL queries.

EscapeIds does the same as EscapeId but for a slice of identifiers.

EscapeValue formats, quotes and escapes a DataValue for use in SQL queries.

EscapeValues does the same as EscapeValue but for a slice of DataValues.

FlushViewCount periodically flushes the database view count from Memcache to PostgreSQL.

ForkDatabase forks the PostgreSQL entry for a SQLite database from one user to another.

ForkedFrom checks if the given database was forked from another, and if so returns that one's owner and database name.

ForkParent returns the parent of a database, if there is one (and it's accessible to the logged in user).

ForkTree returns the complete fork tree for a given database.

GenerateClientCert generates a new DBHub.io client certificate for the given user.

GetActivityStats returns the latest activity stats.

GetAPIKeys returns the list of API keys for a user.

GetAPIKeyUser returns the owner of a given API key.

GetBranches load the branch heads for a database TODO: It might be better to have the default branch name be returned as part of this list, by indicating in the list TODO which of the branches is the default.

GetCachedData retrieves cached data from Memcached.

GetCommitList returns the full commit list for a database.

GetCommonAncestorCommits determines the common ancestor commit (if any) between a source and destination branch.

GetCurrentFunctionName returns the name of the function this was called from.

GetDatabase extracts a database name from GET or POST/PUT data.

GetDefaultBranchName returns the default branch name for a database.

GetDefaultTableName returns the default table name for a database.

GetDiscussionAndMRCount returns the discussion and merge request counts for a database TODO: The only reason this function exists atm, is because we're incorrectly caching the discussion and MR data in TODO a way that makes invalidating it correctly hard/impossible.

GetFormBranch return the requested branch name, from get or post data.

GetFormCommit returns the requested database commit, from form data.

GetFormLicence returns the licence name (if any) present in the form data.

GetFormLive returns the "live" value (if any) present in the form data.

GetFormODC returns the database owner, database name, and commit (if any) present in the form data.

GetFormOwner returns the database owner present in the GET or POST/PUT data.

GetFormRelease returns the requested release name, from get or post data.

GetFormSourceURL returns the source URL (if any) present in the form data.

GetFormTable returns the table name present in the GET or POST/PUT data.

GetFormTag returns the requested tag name, from get or post data.

GetFormUDC returns the username, database, and commit (if any) present in the form data.

GetLicence returns the text for a given licence.

GetLicenceInfoFromSha256 returns the friendly name + licence URL for the licence matching a given sha256 Note - When user defined licence has the same sha256 as a default one we return the user defined licences' friendly name.

GetLicences returns the list of licences available to a user.

GetLicenceSha256FromName returns the sha256 for a given licence.

GetOD returns the requested database owner and database name.

GetODC returns the requested database owner, database name, and commit revision.

GetODT returns the requested database owner, database name, and table name.

GetODTC returns the requested database owner, database name, table name, and commit string.

GetPrimaryKeyAndOtherColumns figures out the primary key columns and the other columns of a table.

GetPub returns the requested "public" variable, if present in the form data If something goes wrong, it defaults to "false".

GetReleases returns the list of releases for a database.

GetShares returns a map with all users for which the given database is shared as key and their permissions as value.

GetSharesForUser returns a list of all the databases shared with the given user, and their permissions.

GetSQLiteRowCount returns the number of rows in a SQLite table.

GetTable returns the requested table name (if any).

GetTags returns the tags for a database.

GetUFD returns the username, folder, and database name (if any) present in the form data.

GetUsername returns the username (if any) present in the GET or POST/PUT data.

GetUsernameFromEmail returns the username associated with an email address.

GetViewCount retrieves the view count in Memcached for a database.

GetVisualisations returns the saved visualisations for a given database.

HttpErrorLog is used to filter out the copious 'TLS handshake error' messages we're getting.

IncrementDownloadCount increments the download count for a database.

IncrementViewCount increments the view counter in Memcached for a database.

InvalidateCacheEntry invalidate Memcache data for a database entry or entries.

IsCommitInBranchHistory checks if a given commit ID is in the history of the given branch.

JobQueueCheck checks if newly submitted work is available for processing.

JobQueueCreateDatabase creates a database on a live node.

JobQueueGetSize returns the on disk size of a database on a live node.

JobQueueListen listens for database notify events indicating newly submitted jobs.

JobSubmit submits job details to our PostgreSQL based job queue.

LiveAddDatabasePG adds the details for a live database to PostgreSQL.

LiveBackup asks the job queue backend to store the given database back into Minio.

LiveColumns requests the job queue backend to return a list of all columns of the given table.

LiveCreateDB requests the job queue backend create a new live SQLite database.

LiveDelete asks our job queue backend to delete a database.

LiveExecute asks our job queue backend to execute a SQL statement on a database.

LiveGenerateMinioNames generates Minio bucket and object names for a live database.

LiveGetMinioNames retrieves the Minio bucket and object names for a live database.

LiveIndexes asks our job queue backend to provide the list of indexes in a database.

LiveQuery sends a SQLite query to a live database on its hosting node.

LiveRetrieveDatabaseMinio retrieves a live SQLite database from Minio, and places it on the local filesystem.

LiveRowData asks our job queue backend to send us the SQLite table data for a given range of rows.

LiveSize asks our job queue backend for the file size of a database.

LiveSqlHistoryAdd adds a new record to the history of recently executed SQL statements.

LiveSqlHistoryDeleteOld deletes all saved SQL statements in the SQL history table, except for the most recent ones.

LiveSqlHistoryGet returns the list of recently executed SQL statement for a user and database.

LiveStoreDatabaseMinio stores a live SQLite database in Minio.

LiveTables asks our job queue backend to provide the list of tables (not including views!) in a database.

LiveTablesAndViews asks our job queue backend to provide the list of tables and views in a database.

LiveUserDBs returns the list of live databases owned by the user.

LiveViews asks our job queue backend to provide the list of views (not including tables!) in a database.

LogDB4SConnect creates a DB4S default browse list entry.

LogDownload creates a download log entry.

LogSQLiteQueryAfter adds memory allocation stats for the execution run of a user supplied SQLite query.

LogSQLiteQueryBefore logs the basic info for a user supplied SQLite query.

LogUpload creates an upload log entry.

MemcacheHandle returns the Memcached handle.

Merge merges the commits in commitDiffList into the destination branch destBranch of the given database.

MetadataCacheKey generates a predictable cache key for metadata information.

MetadataResponse returns the metadata for a database.

MinioDeleteDatabase deletes a database file from Minio.

MinioHandle gets a handle from Minio for a SQLite database object.

MinioHandleClose closes a Minio object handle.

MinioLocation returns the Minio bucket and ID for a given database.

MQCreateDBQueue creates a queue on the MQ server for "create database" messages.

MQCreateQueryQueue creates a queue on the MQ server for sending database queries to.

MQCreateResponse sends a success/failure response back.

MQRequest is the main function used for sending requests to our AMQP backend.

MQResponse sends an AMQP response back to its requester.

NewEvent adds an event entry to PostgreSQL.

NewResponseQueue is the constructor function for creating a new ResponseReceivers.

OpenSQLiteDatabase retrieves a SQLite database from Minio, opens it, then returns the connection handle.

OpenSQLiteDatabaseDefensive is similar to OpenSQLiteDatabase(), but opens the database Read Only and implements the recommended defensive precautions for potentially malicious user provided SQL queries: https://www.sqlite.org/security.html.

OpenSQLiteDatabaseLive is similar to OpenSQLiteDatabase(), but opens the a live SQLite database and implements the recommended defensive precautions for potentially malicious user provided SQL queries: https://www.sqlite.org/security.html TODO: De-duplicate/refactor the common code in this function and OpenSQLiteDatabaseDefensive() above, as they're TODO mostly the same.

PrefUserMaxRows returns the user's preference for maximum number of SQLite rows to display.

RandomString generates a random alphanumeric string of the desired length.

ReadConfig reads the server configuration file.

ReadSQLiteDB reads up to maxRows number of rows from a given SQLite database table.

ReadSQLiteDBCols reads up to maxRows # of rows from a SQLite database.

ReadSQLiteDBCSV is a specialised variation of the ReadSQLiteDB() function, just for our CSV exporting code.

RecordWebLogin records the start time of a user login session, for stats purposes.

RemoveLiveDB deletes a live database from the local node.

RenameDatabase renames a SQLite database.

ReservedUsernamesCheck checks a username against the list of reserved ones.

ResetDB resets the database to its default state.

ResponseComplete marks a response as processed.

ResponseQueueCheck checks if a newly submitted response is available for processing.

ResponseQueueListen listens for database notify events with responses from the other DBHub.io daemons.

ResponseSubmit adds a response to the job_responses table.

RetrieveDatabaseFile retrieves a SQLite database file from Minio.

SaveDBSettings saves updated database settings to PostgreSQL.

SendEmails sends status update emails to people watching databases.

SetClientCert stores a certificate for a given client.

SetUserPreferences sets the user's preference for maximum number of SQLite rows to display.

SetUserStatusUpdates increments the view counter in Memcached for a database.

SignalHandler is a background goroutine that exists to catch *nix termination signals then shut the daemon down cleanly.

SocialStats returns the latest social stats for a given database.

SQLiteBackupLive is used by our job queue backend nodes to refresh a live SQLite database back into Minio.

SQLiteExecuteQueryLive is used by our job queue backend infrastructure to execute a user provided SQLite statement.

SQLiteGetColumnsLive is used by our job queue backend nodes to retrieve the list of columns from a SQLite database.

SQLiteGetIndexesLive is used by our job queue backend nodes to retrieve the list of indexes from a SQLite database.

SQLiteGetTablesLive is used by our job queue backend nodes to retrieve the list of tables in a SQLite database.

SQLiteGetViewsLive is used by our job queue backend nodes to retrieve the list of views in a SQLite database.

SQLiteReadDatabasePage opens a SQLite database (locally) and returns a "page" of rows from it, for display in the database view page.

SQLiteRunQuery runs a SQLite query.

SQLiteRunQueryDefensive runs a user provided SQLite query, using our "defensive" mode.

SQLiteRunQueryLive is used by our job queue backend infrastructure to run a user provided SQLite query.

SQLiteSanityCheck performs basic sanity checks of an uploaded database.

SQLiteVersionNumber returns the version number of the available SQLite library, in 300X00Y format.

StatusUpdateCheck checks if a status update for the user exists for a given discussion or MR, and if so then removes it.

StatusUpdates returns the list of outstanding status updates for a user.

StatusUpdatesLoop periodically generates status updates (alert emails TBD) from the event queue.

StoreBranches updates the branches list for a database.

StoreComment adds a comment to a discussion.

StoreCommits updates the commit list for a database.

StoreDatabase stores database details in PostgreSQL, and the database data itself in Minio.

StoreDatabaseFile stores a database file in Minio.

StoreDefaultBranchName stores the default branch name for a database.

StoreDefaultTableName stores the default table name for a database.

StoreDiscussion stores a new discussion for a database.

StoreLicence stores a licence.

StoreReleases stores the releases for a database.

StoreShares stores the shares of a database.

StoreStatusUpdates stores the status updates list for a user.

StoreTags stores the tags for a database.

SwitchDefault changes the logged in user to be the user "default".

SwitchFirst changes the logged in user to be the test user "first".

SwitchSecond changes the logged in user to be the test user "second".

SwitchThird changes the logged in user to be the test user "third".

TableRowsCacheKey generates a predictable cache key for SQLite row data.

Tables returns the list of tables in the SQLite database.

TablesAndViews returns the list of tables and views in the SQLite database.

TestLogout logs out the user for test runs.

ToggleDBStar toggles the starring of a database by a user.

ToggleDBWatch toggles the watch status of a database by a user.

UpdateAvatarURL updates the Avatar URL for a user.

UpdateComment updates the text for a comment.

UpdateContributorsCount updates the contributors count for a database.

UpdateDiscussion updates the text for a discussion.

UpdateMergeRequestCommits updates the commit list for a Merge Request.

UpdateModified is a simple function to change the 'last modified' timestamp for a database to now().

UploadResponse validates incoming upload requests from the db4s and api daemons, then processes the upload.

User returns details for a user.

UserDBs returns the list of databases for a user.

UserNameFromAuth0ID returns the username for a given Auth0 ID.

UsersStarredDB returns the list of users who starred a database.

UserStarredDBs returns the list of databases starred by a user.

UserStatusUpdates returns the number of status updates outstanding for a user.

UsersWatchingDB returns the list of users watching a database.

UserWatchingDBs returns the list of databases watched by a user.

ValidateBranchName validates the provided branch, release, or tag name.

ValidateCommitID validates the provided commit ID.

ValidateDB validates the database name.

ValidateDiscussionTitle validates the provided discussion or merge request title.

ValidateDisplayName validates a provided full name.

ValidateEmail validates the provided email address.

ValidateFieldName validates the SQLite field name.

ValidateLicence validates the provided licence name (ID).

ValidateLicenceFullName validate the provided licence full name.

ValidateMarkdown validates the provided markdown.

ValidatePGTable validates the provided PostgreSQL table name.

ValidateUser validates the provided username.

ValidateUserDB validates the provided user and database name.

ValidateUserDBTable validates the provided user, database, and table name.

ValidateUserEmail validates the provided username and email address.

ValidateVisualisationName validates the provided name of a saved visualisation query.

ViewCount returns the view counter for a specific database.

Views returns the list of views in the SQLite database.

VisualisationDeleteParams deletes a set of visualisation parameters.

VisualisationRename renames an existing saved visualisation.

VisualisationSaveData saves visualisation result data for later retrieval.

VisualisationSaveParams saves a set of visualisation parameters for later retrieval.

WaitForResponse waits for the job queue server to provide a response for a given job id.

WriteDBtoDisk gets an uploaded database file from the user's incoming request, and writes it to a local temporary file.

ActionAdd is used for inserted rows and created objects.

ActionDelete is used for deleted rows and dropped objects.

ActionModify is used for updated rows and altered objects.

DefaultNumDisplayRows is the number of rows to display by default on the database page.

These are not iota, as it would be seriously bad for these numbers to change.

These are not iota, as it would be seriously bad for these numbers to change.

MaxDatabaseSize is the maximum database size accepted for upload (in MB).

MaxLicenceSize is the maximum licence size accepted for upload (in MB).

MinioFolderChars is the number of leading characters of a files' sha256 used as the Minio folder name eg: When set to 6, then "34f4255a737156147fbd0a44323a895d18ade79d4db521564d1b0dbb8764cbbc" -> Minio folder: "34f425" -> Minio filename: "5a737156147fbd0a44323a895d18ade79d4db521564d1b0dbb8764cbbc".

NewPkMerge produces SQL statements which generate new values for the primary key columns when executed.

NoMerge removes any SQL statements for merging from the diff results.

These are not iota, as it would be seriously bad for these numbers to change.

PreservePkMerge produces SQL statements which preserve the values of the primary key columns.

SqliteDebug displays some SQLite related debugging information (on the backend), when set to a non-0 value.

AmqpChan is the AMQP channel handle we use for communication with our AMQP backend.

CheckJobQueue is used by the live daemons for triggering a check of the job queue.

CheckResponsesQueue is used by the non-live daemons for triggering a check of the job responses queue.

Conf holds our configuration info.

JobListenConn is the PG server connection used for receiving PG notifications.

JobQueueConn is the PG server connection used for submitting and retrieving jobs.

JobQueueDebug tells the daemons whether or not to output debug messages while running job queue code Mostly useful for development / debugging purposes.

ResponseQueue is used to direct job queue responses back to the appropriate callers.

SQLiteFunctions lists the function we allow SQL queries to run.

SubmitterInstance is a random string generated at server start for identification purposes.

UseAMQP switches between running in AMQP mode (true) or job queue server mode (false).

Validate is used for input validation.

ApiInfo contains configuration info for the API daemon.

APIJSONColumn is a copy of the Column type from github.com/gwenn/gosqlite, but including JSON field name info.

APIJSONIndex holds the details of an index for a SQLite database.

APIJSONIndexColumn holds the details of one column of a SQLite database index.

APIKey is an internal structure used for passing around user API keys.

Auth0Info contains the Auth0 connection info used authenticating webUI users.

BranchListResponseContainer holds the response to a client request for the database branch list.

DataDiff stores a single change in the data of a table, i.e.

DB4SInfo contains configuration info for the DB4S end point daemon.

DiffObjectChangeset stores all the differences between two objects in a database, for example two tables.

Diffs is able to store all the differences between two databases.

DiskCacheInfo contains the path to the root of the local disk cache.

EnvInfo holds information about the purpose of the running server.

EventProcessingInfo hold configuration for the event processing loop.

ExecuteResponseContainer is used by our job queue backend, to return information in response to an Execute() call on a live database.

FilteringErrorLogWriter is a custom error logger for our http servers, to filter out the copious 'TLS handshake error' messages we're getting.

JobRequest holds the fields used for sending requests to our job request backend.

JobRequestRows holds the data used when making a rows request to our job queue backend.

JobResponseDBColumns holds the fields used for receiving column list responses from our job queue backend.

JobResponseDBCreate holds the fields used for receiving database creation responses from our job queue backend.

JobResponseDBError holds the structure used when our job queue backend only needs to response with an error field (empty or not).

JobResponseDBExecute holds the fields used for receiving the database execute response from our job queue backend.

JobResponseDBIndexes holds the fields used for receiving the database index list from our job queue backend.

JobResponseDBQuery holds the fields used for receiving database query results from our job queue backend.

JobResponseDBRows holds the fields used for receiving table row data from our job queue backend.

JobResponseDBSize holds the fields used for receiving database size responses from our job queue backend.

JobResponseDBTables holds the fields used for receiving the database table list from our job queue backend.

JobResponseDBViews holds the fields used for receiving the database views list from our job queue backend.

JsonError holds the details of an error.

LicenceInfo -> LicenceDir holds the path to the licence files.

LiveDBColumnsResponse holds the fields used for receiving column list responses from our AMQP backend.

LiveDBErrorResponse holds just the node name and any error message used in responses by our AMQP backend It's useful for error message, and other responses where no other fields are needed.

LiveDBExecuteResponse returns the number of rows changed by an Execute() call.

LiveDBIndexesResponse holds the fields used for receiving index list responses from our AMQP backend.

LiveDBQueryResponse holds the fields used for receiving query responses from our AMQP backend.

LiveDBRequest holds the fields used for sending requests to our AMQP backend.

LiveDBResponse holds the fields used for receiving (non-query) responses from our AMQP backend.

LiveDBRowsRequest holds the data used when making an AMQP rows request.

LiveDBRowsResponse holds the fields used for receiving database page row responses from our AMQP backend.

LiveDBSizeResponse holds the fields used for receiving database size responses from our AMQP backend.

LiveDBTablesResponse holds the fields used for receiving table list responses from our AMQP backend.

LiveDBViewsResponse holds the fields used for receiving view list responses from our AMQP backend.

LiveInfo holds configuration info for the Live database daemon.

MemcacheInfo contains the Memcached configuration parameters.

MetadataResponseContainer holds the response to a client request for database metadata.

MinioInfo contains the Minio connection parameters.

PGInfo contains the PostgreSQL connection parameters.

ResponseInfo holds job queue responses.

ResponseReceivers is a simple structure used for matching up job queue responses to the caller who submitted the job.

SchemaDiff describes the changes to the schema of a database object, i.e.

ShareDatabasePermissionsOthers contains a list of user permissions for a given database.

ShareDatabasePermissionsUser contains a list of shared database permissions for a given user.

SigningInfo contains the info used for signing DB4S client certificates.

StatusResponseContainer holds a general status message response to a client request.

TomlConfig is a top level structure containing the server configuration information.

UploadResponseContainer holds the response to a client upload.

UserMgmtInfo contains the various settings for specific users, or groups of users.

VisGetFields is used when validating input.

WebInfo contains configuration info for the webUI daemon.

WebpageResponseContainer holds the response to a client request for the database webUI URL.

AccessType is whether a database is private, or public, or both.

DiffType specifies the type of change in a row or object.

MergeStrategy specifies the type of SQL statements included in the diff results.

QuerySource is used internally to help choose the output format from a SQL query.

SetAccessType is used for setting the public flag of a database.

SetDBType is used for setting what type of database we're working with.

UserInfoSlice is used for sorting a UserInfo list by Last Modified date descending.

ValType indicates the type of data in a field returned from a SQLite query.