As engineers we often find ourselves parsing through large amounts of data to uncover bugs in a service and to learn more about how a service operates. This process can be tedious and time consuming.

jEdit is a tool that can be used to quickly process, filter, and modify large JSON datasets so you can get back to focusing on feature work, opposed to wasting time looking through logs.

This tool was written in golang and can be used via the commandline. Continue reading to determine how to get started!

(back to top)

Built With

Golang
Cobra

(back to top)

Getting Started

Use the following instructions to learn how to clone the jEdit repository and to build the jEdit binary.

Prerequisites

Install Golang
Validate golang version is 1.7.2 or higher
```
go version
```

Installation

Clone the repo

git clone https://github.com/benmcgit/jedit.git

Build jEdit binary (in root directory)
```
./scripts/build.sh
```
Validate the binary is available
```
./jedit --help
```
Validate the binary is working (this should result in non-error output in your terminal). This script covers the following scenario:
- Add a new key value pair to only to records where "team" equal to "team-x"
- New key = "incident_id", New value = "6502"
```
./scripts/test.sh
```

(back to top)

Usage

Now that jEdit is installed, lets try it out!

There are two ways jEdit can be consumed

Command Line
Public API

See below examples of how to use each below.

Command Line

There is test data that we can execute these examples with. This test data is located in the testdata directory.

Reducing

cat testdata/yesterday.json | ./jedit query <FLAGS>

Lets say we only want to return the records where the key "team" contains the value "team-x". Try out this...
```
cat testdata/yesterday.json | ./jedit query --filter "team == team-x"
```
Maybe we want to include all records except ones containing "team" equal to "team-x". Then we can use this...
```
cat testdata/yesterday.json | ./jedit query --filter "team != team-x"
```
We can use more than one filter to reduce the filter set even further. In this example we return all values with "team" equal to "team-x" and "severity" with a value of 4 or higher...
```
cat testdata/yesterday.json | ./jedit query --filter "team == team-x" --filter "severity >= 4"
```

The following filter operators can be used to reduce the dataset. These operators can be used on strings along with digits.

cat testdata/yesterday.json | ./jedit query --filter "team < team-x" --filter "team != team-a" --filter "severity > 0" --filter "severity <= 4"

If the key does not match any key in the record set, 0 entries will be returned...

cat testdata/yesterday.json | ./jedit query --filter "thisdoesnotexist == team-x"

Sorting

cat testdata/yesterday.json | ./jedit sort <KEY> <FLAGS>

Use the "sort" command to sort the data by providing a key as an argument. Example:
```
cat testdata/yesterday.json | ./jedit sort severity
```
By default, the sort command will sort the dataset in descending order. Use the "asc" flag to sort in ascending order. Example:
```
cat testdata/yesterday.json | ./jedit sort severity --asc
```
You may also sort the dataset on non-numerical based key-value pairs. Example:
```
cat testdata/yesterday.json | ./jedit sort team
```

Adding a new key-value pair

cat testdata/yesterday.json | ./jedit addKey <NEW_KEY> <NEW_VALUE> <FLAGS>

There may be a reason to add a new key to each item in your dataset. You can accomplish this by providing the new key and new value as arguments to the "addKey" command. Example:
```
cat testdata/yesterday.json | ./jedit addKey newKey newValue
```
Its possible that you only want to add the key to specific items in the dataset. You can use filters to accomplish this. You may use as many filters as you'd like. Example:
```
cat testdata/yesterday.json | ./jedit addKey newKey newValue --filter "team == team-x" --filter "severity > 3"
```
What happens if the new key already exists? We may want to replace that data with a new value. To replace existing values we can use the "replace" flag. Example:
```
cat testdata/yesterday.json | ./jedit addKey severity 10 --replace
```
By default, if the new key already exists in the dataset, the old value for that key will be retained. In this example, the output will not be modified:
```
cat testdata/yesterday.json | ./jedit addKey severity 10
```

Removing a key-value pair

cat testdata/yesterday.json | ./jedit removeKey <KEY> <FLAGS>

We may want to remove a key-value pair to reduce complexity in our dataset even further. To accomplish this, we can use the "remove" command by supplying a key. Example:
```
cat testdata/yesterday.json | ./jedit removeKey team
```
We can use filters to selectively remove objects from our dataset. If no filter is supplied, the key will be removed from all entries. Example:
```
cat testdata/yesterday.json | ./jedit removeKey team --filter "team == team-a"
```

Modifying a key

cat testdata/yesterday.json | ./jedit modifyKey <KEY> <NEW_KEY_NAME> <FLAGS>

There may be a situation where we want to add a prefix, suffix, or modify the name of a key altogether. In this scenario we can use the "modifyKey" command by supplying a key and new key as arguments. Example:
```
cat testdata/yesterday.json | ./jedit modifyKey team my_super_awesome_team
```

Like the addKey, removeKey, and query commands, we can use filters to determine which objects in the dataset to modify. Example:

cat testdata/yesterday.json | ./jedit modifyKey team my_super_awesome_team --filter "team == team-x" --filter "ts > 1642415085"

Providing input

There are two ways a user can provide input.

Piping via Stdin

cat testdata/yesterday.json | ./jedit modifyKey ...

Using the --input flag

./jedit modifyKey ... --input testdata/yesterday.json

Defining output location

By default, jEdit will print its output to Stdout. Optianlly you can use the --output to create a new file and write to it.

cat testdata/yesterday.json | ./jedit modifyKey ... --output my_output_file.json

Need more help?

cat testdata/yesterday.json | ./jedit --help

Parsing and editing JSON in bulk can be time-consuming
and difficult. jEdit aims to help engineers save time by providing them 
a tool that can reduce, filter, and modify their existing JSON dataset.

Usage:
  jedit [command]

Available Commands:
  addKey      Adds an additional key to object(s) in your dataset
  completion  Generate the autocompletion script for the specified shell
  help        Help about any command
  modifyKey   Modifies existing keys on object(s) in your dataset
  query       Reduces the amount of object(s) in your dataset
  removeKey   Removes existing keys on object(s) in your dataset
  sort        Sorts objects in your dataset base on a user-provided key

Flags:
  -h, --help            help for jedit
      --input string    Path to JSON file to be parsed
      --output string   Path to file to write resulting JSON to. If not existent, it will be created.

Use "jedit [command] --help" for more information about a command.

Public API

Please see the documentation for how to use jedit within your golang project here.

IMPORTANT: jEdit must be consumed as a go module (required after go 1.17).You can create a go-module by running go mod init <YOUR_MODULE_NAME> (Please be sure to run go get -v to update your dependencies).

Example:

package main

import (
	"log"

	"github.com/benmcgit/jedit/pkg/jedit"
)

func main() {
	filtersStr := []string{"severity >= 4"}
	filePath := "yesterday_reduced.json"

	// validates input and creates an instance of Logs
	logs, err := jedit.ParseFile(filePath)
	if err != nil {
		log.Fatal(err)
	}

	// validates input and creates an array of Filter instances
	filters, err := jedit.ParseFilters(filtersStr)
	if err != nil {
		log.Fatal(err)
	}

	// apply commands on the logs
	logs.Add("urgency", "HIGH", false, filters)

	logs.Print()
}

(back to top)

Roadmap

Nice-to-haves

Preformance Improvements
- Use concurrent go-threads to parse stdin
Allow user to OR filters together (opposed to ANDing them together)

Required for first release

See the open issues for a full list of proposed features (and known issues).

(back to top)

Project Takeaways

Besides creating a tool that worked functionally, I put a lot of thought into two non-functional requirements - Ease of use, preformace.

To make sure jEdit was easy to use, I went through a few design iterations to determine a way I could use a similar filtering process across different commands. This led me to the design where I use a the same filter flag across the majority of my commands (Ex. --filter "key == value"). I hoped that this would allow for my users to learn the format once, and be able to apply the same logic across the query, addKey, removeKey, and modifyKey commands. The tradoff here is a good amount of validation for each filter.

One consideration I was debating on is whether I should AND all the filters together, or OR all of the filters together. I decided AND was more natural from a users perspective so I went with that. I think a cool new feature would be to allow for OR using a flag so the user can decide. I added this to the future improvements section.

As far as preformance, I believe this can be improved. My current logic is to first read in the JSON file and store each record as a type of "Log". After this, I continue to fullfill the customers requested command. There is an opportunity to filter or modify each record as its being read into momory. If preformance became an issue, this is where I would start investigating for potential improvements.

This project was my first experiance using a tool like Cobra. This tool allowed me to build commands that contain arguments and flags with simplicity. Along with being easy, it saved me a lot of time when it came to generating the jEdit user manual (./jedit --help). The next time you build a commandline tool using Golang, I would highly recommend checking out Cobra.

While I have a lot of experiance with CI tooling, I have not used GitHuib Actions very much. It was really easy to setup for a small project like this. Take a look at the Actions Tab to learn more. Right now, my CI pipeline is setup to run after every commit to master.

For fun, I used a free online service that suggested and generated a logo for jEdit. I was pleasently suprised how easy it was to use so I added those images to the images directory. If given more time I would try to come up with a logo myself, but I appreciate Hatchful for making my life a little easier and jEdit a little more official :).

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

Fork the Project
Create your Feature Branch (git checkout -b feature/AmazingFeature)
Commit your Changes (git commit -m 'Add some AmazingFeature')
Push to the Branch (git push origin feature/AmazingFeature)
Open a Pull Request

(back to top)

License

Distributed under the MIT License. See LICENSE.txt for more information.

(back to top)

Contact

Ben McAdams - @benwmcadams - [email protected]

Project Link: https://github.com/benmcgit/jedit

(back to top)

Acknowledgments

(back to top)

# Packages

cmd

pkg

No description provided by the author