Websites Content Analysis Using Multithreading

Terminal based program to analyze the metadata of the 1000 most visited websites around the world, implementing Go programming language, web scraping with crawlerclub, multithreading using goroutines and chart libraries with go-echarts.

The data processed and the keywords dataset are stored in JSON files, the top 1000 websites is extracted from htmlstrip.com and stored in a CSV file.

Main Menu

Obtained Chart Example

The found keywords are stored in a folder with the name of the web page, in this folder the chart is also located.

Download the project

$ git clone https://github.com/Johanx22x/multicore-project.git 
$ cd multicore-project

Running instructions

Dependencies

• go

Run the program

$ go run main.go

Warning

Do not compile the source code, there is a null pointer error if you try to run the binary file.

Instead use go run main.go to run the program.