modulepackage
0.0.0-20221220194752-0bfbb337257e
Repository: https://github.com/pendo-io/mapreduce.git
Documentation: pkg.go.dev
# README
Go Mapreduce for Appengine
This is a feature-complete (though not performance or reliability complete)
implementation of map/reduce for the go language on the appengine platform. It
make heavy use of task queues to provide load scaling, and a sample application
is available from http://github.com/pendo-io/mapreduce-sample.
This is an BETA release of this package. While it is being used for production use it certainly still has shortcomings and bugs.
# Functions
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
MapReduceHandler returns an http.Handler which is responsible for all of the urls pertaining to the mapreduce job.
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
# Constants
Datastore entity kinds for jobs and tasks.
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
TaskTypes defines the type of task, map or reduce.
TaskTypes defines the type of task, map or reduce.
# Structs
FatalError wraps an error.
No description provided by the author
IgnoreTaskStatusChange is an implementation of TaskStatusChange which ignores the call.
Int64KeyHandler provides a KeyHandler for int64 keys.
Int64ValueHandler provides a ValueHandler for int values.
JobInfo is the entity stored in the datastore defining the MapReduce Job.
JobTask is the entity stored in the datastore defining a single MapReduce task.
No description provided by the author
MappedData items are key/value pairs returned from the Map stage.
MapReduceJob defines a complete map reduce job, which is the pipeline and the parameters the job needs.
NilOutputWriter collects output and throws it away.
No description provided by the author
No description provided by the author
StringKeyHandler provides a KeyHandler for string keys.
StringValueHandler provides a ValueHandler for string values.
# Interfaces
InputReader is responsible for providing unique names for each of the input sources for a job, and creating individual SingleInputReader objects from those unique names.
IntermediateStorage defines how intermediare results are saved and read.
No description provided by the author
KeyHandler must be implemented for each key type to enable shuffling and storing of map keys.
No description provided by the author
Mapper defines a map function; it is passed an item from the input and returns a list of mapped items.
MapReducePipeline defines the complete pipeline for a map reduce job (but not the job itself).
No description provided by the author
Reducer defines the reduce function; it is called once for each key and is given a list of all of the values for that key.
No description provided by the author
this overlaps a great deal with SingleOutputWriter; they often share an implementation.
No description provided by the author
TaskInterface defines how the map and reduce tasks and controlled, and how they report their status.
TaskStatusChange allows the map reduce framework to notify tasks when their status has changed to RUNNING or DONE.
ValueHandler converts values from a map step to []byte and back again.
# Type aliases
No description provided by the author
StatusUpdateFunc functions are passed into Map and Reduce handlers to allow those handlers to post arbitrary status messages which are stored in the datastore.
No description provided by the author
No description provided by the author