Monitoring Mining Operations with AWS Lambda and Initial State

A while back, I published a post on monitoring a mining farm with Initial State. I mentioned in the post that I’d eventually publish some code on how I achieved this. However, after becoming so focused on expanding my monitoring service to a fully-fledged monitoring, configuration, and initialization platform for mining rigs I never got around to publishing just some monitoring code! I’ve been testing and expanding this now for over a year and will likely be releasing it as a hosted service soon for those who are interested. However, before I get to that point, I figured I’d release some of the original monitoring code for those who want to do this functionality themselves.

N.B.: This tutorial will assume a pre-existing understanding of AWS. I will not cover creating an AWS account, explaining CloudFormation or SAM (Serverless Application Model) or talk about setting up the deployment pipelines for everything. For readers who want a more full-service implementation, that’s what the hosted version will do for you (coming soon!). I plan to publish more tutorials on some of those specifics later.

Monitoring Endpoints with AWS Lambda

#

AWS Lambda will provide the fundamental basic that we’ll use in monitoring through this post. The lambda function is written in Node.js and deployed using a SAM CloudFormation template. A fantastic feature of the lambda event source ecosystem is the ability to setup lambda functions that are triggered on recurring CloudWatch timing events. This provides a cron-like functionality without having to have a continuous running cron server or using another AWS service to perform the trigger.

Take me to the code!

#

This repository contains everything you need to deploy a basic rig, network and pool monitor system. It will also send the information it collects to a series of Initial State buckets that will allow you to do the complex analysis done in posts like Analysing Performance with Initial State.

To deploy the code, you simply need your AWS CLI correctly setup and authorized with your AWS account. Then run the following commands:

$ aws cloudformation package --template-file sam.yaml --s3-bucket <some_writeable_bucket> --output-template-file sam-output.yaml
$ aws cloudformation deploy --template-file sam-output.yaml --stack-name mining-monitor --parameter-overrides EthosDistroUrl="http://<your_panel>.ethosdistro.com?json=yes" AccessKey=<your_initialstate_access_key> BucketKey=rig-monitor-v1 EtherAddress=<your_ether_wallet_address> --capabilities CAPABILITY_IAM

Note: replace <some_writeable_bucket> with a bucket that you’re CLI IAM user has access to upload to. Also, replace <your_panel>, <your_initialstate_access_key>, and <your_ether_wallet_address> with the appropriate values.

Once the stack is up and running, it’ll start collecting data and sending to Initial State into specific bucket keys. The beauty of Initial State is that you can stream to bucket keys that don’t yet exist in the platform, then you can go to your Initial State account and create a bucket with the same bucket key and access key and you’ll see all the data that’s been streaming into your account.

The bucket key formulas of all the buckets this stack will use will be some derivative of the MAIN_BUCKET_KEY environment variable that gets set by the BucketKey parameter in the sam.yaml

The bucket keys that will be used for this stack are as follows (with MAIN_BUCKET_KEY replaced as prescribed):

• {MAIN_BUCKET_KEY}: This will be the primary summary bucket for the rig, network, and pool data
• {MAIN_BUCKET_KEY}-sum: This will be a bucket of just the amount paid from nanopool over time
• {MAIN_BUCKET_KEY}-{rig_name}: This will be a number of buckets prefixed with the MAIN_BUCKET_KEY with the particular rig appended. This bucket won’t have any data unless the sendRigData lambda function has some of it’s code un-commented. Note that it’s currently commented to prevent overuse of your Initial State account.

To see this data in Initial State, just log into your account and create a real-time streaming bucket and override the generated Bucket Key with one of the bucket keys from the list above.

File a GitHub issue if you have any troubles, and enjoy!