shadow server api

In this blog, we will be using the api provided by the shadow server and obtain information. We will use the official API to pull data from the Shadow Server systems. Previously this has been delivered manually via email but it be manual process where we can configure the intelmq to fetch the email from the email server obtain the attachment and delete the file. But from my perspective, this seems ineffective and not the proper way to do it.

Use the below command to clone the official repo.

git clone https://github.com/The-Shadowserver-Foundation/api_utils.git

As provided in the guide, if we want to use the API we will have to enter the following information in the ~/.shadowserver.api file.

[api]
key = <<API-KEY>>
secret = <<SECRET>>
uri = https://transform.shadowserver.org/api2/

When I try to run it, I get the following message

reza@tools:~/Desktop/shadowserver/api_utils$ python3 report-manager.py 
Usage: report_manager.py /path/to/config.ini

After all the information has been entered into the appropriate locations, we can now run the script so that we can download the CSV file containing report for last two days.

reza@tools:~/Desktop/shadowserver/api_utils$ python3 report-manager.py report-manager-config.ini 

This command might take a while to execute. After this commands successful execution, we can see a lot of file downloaded into the path specifies in the config.ini file.

Some sample configuration has been given below:

[reports]
directory = /var/tmp/reports
min_disk_free = 512
notifier = none
url_prefix = http://myserver/reports/

[stomp]
server = 127.0.0.1
port = 61613
user = guest
password = guest
queue = /queue/mytest

[redis]
server = 127.0.0.1
port = 6379
;password = guest
queue = mytest

[kafka]
server = 127.0.0.1
port = 9092
queue = mytest

The above configuration is for all the every notifier how ever we will be only using redis in our current environment. So we will change it to:

[reports]
directory = /var/tmp/reports
min_disk_free = 512
notifier = redis
url_prefix = http://myserver/reports/

[redis]
server = 127.0.0.1
port = 6379
;password = guest
queue = ss-report-manager

Here you can see that change has been performed in the notifier’s value. Now lets re-execute the with updated configuration.

$ python3 report-manager.py report-manager-config.ini

After a while we should be able to see some data in db0 under the queue ss-report-manger

The downloaded files are now stored in /var/tmp/reports. Basically this path is considered as the root path for the web server. From here, you can build a webserver and instruct other systems to pull data from here.

Lets jump back to the IntelMQ system and start all the bots