go-engine

WARNING: This library is a WIP and is not production ready.

This library is currently a learning project that implements the source{d} engine in Go using Gleam (Go implementation of MapReduce).

The heavy lifting of working with Git repositories is done by go-git.

To-do

• Split remotes properly in the repositories reader; for siva into seperate repos
• Implement the queries from QuerySetApp
• Generalize the filter function
• Improve the siva reading to turn rooted repositories into individual ones
• Add a Babelfish deployment to k8s
• UDF's:
  • readBlob read the content of a blob based on its hash
  • classifyLanguage implements enry to classify the programming language of the blobs content
  • extractUAST parses a blob using Babelfish

Future ideas:

• Research how to add named columns to Gleam
• Update gleam to use both inner & outer IP's so that binaries can be sent to agents from any IP
• Research on adding bitmap reader to go-git:
  • https://kscherer.github.io/git/2015/05/15/git-and-bitmaps
  • https://githubengineering.com/counting-objects/