modulepackage
0.0.0-20230509225123-53cef4e428e2
Repository: https://github.com/ciscocsirt/gosint.git
Documentation: pkg.go.dev
# README
GOSINT - Open Source Threat Intelligence Gathering and Processing Framework
The GOSINT framework is a project used for collecting, processing, and exporting high quality indicators of compromise (IOCs). GOSINT allows a security analyst to collect and standardize structured and unstructured threat intelligence. Applying threat intelligence to security operations enriches alert data with additional confidence, context, and co-occurrence. This means that you apply research from third parties to security event data to identify similar, or identical, indicators of malicious behavior. The framework is written in Go with a JavaScript frontend.
Installation
Please find the installation procedure at http://gosint.readthedocs.io/en/latest/installation.html
There are three ways to get up and running:
Updates
Updating is simple and encouraged as bugs are reported and fixed or new features are added. To update your instance of GOSINT, pull the latest version of GOSINT from the repository and re-run the build command to compile the updated binary.
godep go build -o gosint
Configuration
GOSINT needs some quick initial configuration to start making use of the framework features. All the settings you will need to specify can be found under the "Settings" tab.
Please find the configuration procedure at http://gosint.readthedocs.io/en/latest/configuration.html
Use
Please find the instructions for use at http://gosint.readthedocs.io/en/latest/use.html
# Functions
AlienvaultFeedCron is a function to setup the frequency in which the Alienvault API is crawled for new indicators.
CallOrkaMon is a function for re-initiating the orchestration code via the web API.
DeleteSinglePost gathers a single in the post-processing collection and deletes it.
DetermineIndicatorType is a function to decide what type of indicator was found.
DumpCSV is a function to recieve HTTP POST requests containing a list of GUID objects or choose all and dump them to CSV.
EditFeeds is a handler for viewing and modifying configured threat feeds to use for parsing.
EditSettings is a function for viewing and modifying the backend configuration through the web API.
No description provided by the author
GatherFeeds looks in the configured settings for feeds and dispatches all of them in cron jobs.
No description provided by the author
GetAlienvault is a function for gathering pulses from an Alienvault OTX account configured with subscriptions.
GetAllPre gathers all documents in the pre-processing collection and structures them in a JSON API response.
GetCSVResource is the function that supplied with the parsed sources retrieves the intel from each feed.
GetMetrics is a function to pull the latest metrics based off what is in the database.
GetOpenDNSWhois is a wrapper for calling the Umbrella API for whois on behalf of the frontend.
GetSinglePre gathers a single in the pre-processing collection and structures it in a JSON API response or deletes it based of method use.
GetVTDomain is a wrapper for calling the VT API for a domain on behalf of the frontend.
GetVTHash is a wrapper for calling the VT API for a hash based file report on behalf of the frontend.
GetVTIP is a wrapper for calling the VT API for an IP on behalf of the frontend.
GetVTURL is a wrapper for calling the VT API for a URL on behalf of the frontend.
GoGet is a function to run ad-hoc analysis of a web resource for indicators.
Has is a function for determining bytes seen in a byte array.
No description provided by the author
InsertRaw is used for inserting new indicators in the pre-processing MongoDB collection.
Legit is a function for determining if a hash has a highly statistical probability of being real by checking how unique it is.
OrkaDelete is a function for deleting recipes in the db.
OrkaMon is a function for tracking which plays exist and then running them sequentually.
OrkaPlay is a function for interfacing with plays via the API.
OrkaToCrits is a gadget for the orchestration to be used for sending indicators to CRITs.
ParseIndicators is a function to use the xurls library to pull all IP, URL, and domain indicators from a string and send them for DB insertion.
ParseSources parses the structures of sources and sends them to be retrieved.
PlayRun is a function for running a playbook entry received from OrkaMon.
PrintLogo is a simple function for printing the framework name called upon start.
Processed is a function to receive HTTP POST requests containing the processed indicator or a GET request to list all records.
Sanitize is a function to help de-fang and repair broken IOCs.
SendToCrits is a function to recieve HTTP POST requests containing a list of GUID objects or choose all and send them to CRITs.
SendToPost is a gadget for the orchestration to be used for sending indicators to post-processing.
StartAlienvault determines if an Alienvault OTX API key is configured and if so it sends a signal to kill any running Alienvault cron feed and restarts a new one with the new settings.
StartFeeds determines if feeds are running and if so it sends a signal to kill any running feed cron jobs and restarts the feeds cron job with the updated settings.
StartTwitter determines if Twitter API keys are configured and if so it sends a signal to kill any running parser goroutines and restarts a new one with the new settings.
TwitterParser creates a Twitter Stream and parses incoming tweets for indicators.
# Variables
No description provided by the author
No description provided by the author
No description provided by the author
# Structs
AdHoc is a structure to receive the parameters for an ad-hoc resource consumption.
AlienvaultIndicators is a structure to hold an Alienvault OTX pulse in.
AllResults is a structure for the JSON API response containing all indicators contained in a collection.
AllResultsPost is a structure for the JSON API response containing all indicators contained in the post-processing collection.
APIResponse is a generic struct to use for an API call status return.
No description provided by the author
No description provided by the author
Count is a struct used for the Metrics struct.
Dump is a structure for to receive the requested format and records to be dumped.
FeedArray is a structure for passing an array of sources.
Hash is a struct to hold and track parsed hashes.
Metrics structure for the framework.
MetricSource is a struct used for the Metrics struct.
MetricType is a struct used for the Metrics struct.
OTXResponse is a structure to return the Alienvault OTX API to.
OTXSeen is a struct used to hold information for tracking Alienvault pulse IDs that have already been ingested.
Playbook entry for Orka to use.
ProcessedIndicator is a structure for a processed indicator for insertion in the processed collection of the DB.
RawIndicators is a structure for a raw indicator for insertion in the pre_processing collection of the DB.
ReceipesConfigured is a struct to hold all the configured recipes in for an API response.
RecordToDump is a structure for an individual record to be dumped.
Settings structure for the framework.
Source is a structure created from parsing the specified sources which is used for retrieving the resource.
VTURL is a structure to receive the parameters for a VT URL API call.