Building a GPU Mining Rig (Part 3: GPU Installation, BIOS Config and First Boot)


Introduction

If you haven’t already checked out Part 1: Equipment or Part 2: Base Computer Assembly I recommend going back and reviewing those posts in order to build the context for this GPU Installation, BIOS Config and First Boot post. Here we’ll continue our rig assembly all the way to first boot. In a future post, I’ll discuss tunning techniques and challenges as well as a bit on handling energy and heat constraints on scale out.

Preparing for GPU Assembly.

Up until this point, sans the frame, we’ve pretty much been building a vanilla computer. The GPUs and GPU assembly is what will make this vanilla computer a GPU hash-mining machine. First, you’ll want to unpackage all of your GPU risers and go ahead and assemble them (attaching the Molex-to-SATA power connectors and the USB to PCIe x1 interfaces)

Assembled GPU risers

Here I’m using 6 risers made by 3 different manufacturers. I’ve tested quite a few, and the risers that I’ve linked in the Part 1: Equipment post have been generally reliable for the price. That said, like most custom assembled hardware, manufacturing and supply constraints cause potential shortages and as such, I’ve expanded to a few other GPU Risers as well. In general, most GPU risers are created equal, so long as they have on-board power connectors. The biggest source of quality issues are in the interface mounts for USB as well as the PCIe x16 and power management components. Reviews are your friend here. Make sure to purchase spare parts as you can expect some failures. You may have DoA components or you may experience failures over time, so, having a few in your spare components chest will serve you well.

Next, we’ll go ahead and unbox all of our GPUs. Keep in mind, the GPUs collectively make up the most expensive component to a GPU mining rig. They also are potentially re-usable if treated carefully and not over-worked. Because of their re-purpose-able value, you’ll want to keep the static bags, boxes, packing material, etc so you can either re-purpose yourself or sell should you wish or should GPU mining become infeasible in the future. In my experience, out of all the GPUs I’ve tested, the MSI Gaming line have a side benefit of being the best packaged GPUs as well. This adds additional value in recyclability and time-to-unpack.

In this step, we’ll want to set the boxes aside and prop the GPUs on the rig frame leaving all the protective coverings on the GPUs.

GPUs sitting on rig frame

Note that I’ve removed all the supplied GPU mounting screws from the rig frame in advance of staging the GPUs on the rails. I’ll keep the GPUs safely on the rails while I take each GPU off, one-by-one and assemble them with their respective GPU riser.

Prepping GPU for installation into riser

In order to connect the GPU with the riser, we’ll need to remove the red protective PCIe x16 cover from the GPU, unlock the PCIe slot (if the slot has a lock, some do not, and many are different) and then line up the key holes.

GPU lined up with riser socket

Slide the GPU into the socket and secure the locking mechanism

Note: If the GPU riser has a slider like pictured, locking it is just a matter of sliding the slider towards the socket to lock. I’ve seen other locking mechanisms like the motherboard memory slot locks that need to be depressed to unlock and will automatically lock when the GPU is slid into place or can be lifted to lock the GPU.

Once the GPU is locked it should look something like the following figure:

GPU mounted onto riser and locked

This figure is also a good time to point out the foam covering the solder points on the bottom of the GPU riser board. This is a great feature and necessary for rig frames where the underside of these GPU risers could potential come into contact with a metal part of the frame. Without protection on the underside of the board, solder pins could potentially come in conductive contact with each other and short out and/or permanently damage the GPU. That said, these tend to be stuck on and fall off over time given airflow and heat. These should either be removed, secured another way, or monitored during the first hours, days and months of operation to ensure they haven’t fallen off and obstructed any fans on the motherboard or power supply below.

Mounting GPUs with Risers back to rig frame

Once you’ve completed the above step with each GPU, you should be re-staging them on the rig frame with the cords dangling below. I recommend plugging the PCIe x1 sides of the GPU riser USBs in to the PCIe slots on the motherboard with the exception of ensuring that a well-known GPU (for instance, the far left GPU when looking at the display interface side of the rig) is plugged into the primary GPU slot on the motherboard. Typically, motherboards will dedicate two PCIe slots to GPUs, and the primary slot will be the topmost PCIe x16 slot (closest to the CPU). The ordering of the rest of the GPUs doesn’t particularly matter. We’ll get to why in a later post about troubleshooting and optimizing.

Main PCIe Slot for GPU

It’s necessary to do this because this will be the default PCIe lane used by the motherboard for providing a display for configuring BIOS and the ethOS OS.

The GPUs will be able to be plugged into the board one-by-one matching PCIe slots like the following figure

GPU risers plugged in to motherboard

Don’t screw anything in yet, leave the GPUs loosely staged so that we can reach everything and optimize spacing when everything is plugged in.

Providing Power to GPUs and SSD

The GPUs will take two kinds of power input. First, they’ll need the GPU riser to be powered through the SATA to Molex connector. This allows the PCIe interface to use reference power from a location other than power pulled from the board. While PCIe is capable of powering from the board through the PCIe interface, the amount of power we’d be drawing from the board for all six GPUs isn’t feasible. As such, we’ll need to get SATA power cables out and attach them to the back of the power supply to the SATA ports and then to the six SATA tails coming from the six GPU risers as well as one additional for the SSD that we mounted earlier in the process. These cables add a good bit of clutter, so do your best to keep them neat.

Here is an image pre plugging in the SATA power cables. You can see in this image that our SATA connectors are hanging down ready to be connected.

SATA Power Connectors for GPU Risers

Each of the SATA cables provided with the selected power supply provide three interfaces, so we’ll need three cables to service the seven power demand locations (6 GPUs and 1 SSD).

SATA Power Connections

Most modern, high performance GPUs need direct power supply power in addition to the reference power for PCIe. For this additional power, we’ll use the VGA cables provided by the power supply.

Note: It’s a good idea to go ahead and evenly space the GPUs and securely screw them to the rig frame at this point so that they don’t move, slip or float while plugging in the heftier 8-pin VGA cables. Also note, that if you’re using the same GPUs as me, you may need to remove the protective insert from the topmost Display Port on the GPU so that it will fit flush with the rig frame.

6-pin to 9-pin adapter connected

Most of these cables are one-to-one but there is at least one cable included that has two ends from one port on the power supply. However, this cable is an 8-pin on one outlet and a 6-pin on the other. So, we’ll need something to convert the 6-pin to 8-pin interface like the below image.

6-pin to 8-pin GPU power connector 6-pin to 8-pin connected

We don’t actually need to use this as the sourced power supply has 6 VGA connections and includes 6 cables. However, I’ve recently started doing this to reduce the sheer mass of cables to reduce airflow constraints.

When connecting the VGA cables, I recommend starting by connecting all the GPU-side connectors first, then plugging the power supply sides in one-by-one next. This allows for a much easier cord organization planning.

GPU Power cables connected

Once the GPU sides are connected, we’ll plug them into the VGA power slots on the power supply.

VGA Power Supply ports VGA cables plugged in

We can then take another Velcro strap from the power supply box and use it to help pull all the VGA cables together and off to the side.

VGA power cable management

It’s not the prettiest, but it’ll do.

Plugging in the rest

Now on to the simple stuff. We need to plug the power supply into an outlet, plug an Ethernet cable in to the motherboard, and then plug in a keyboard and display to boot and configure.

Power source plugged into power supply

Note that the power supply is still switched off until we have everything ready for first boot.

Ethernet plugged into motherboard

We’ll use the primary display GPU as the GPU to serve as our configuration display.

HDMI in to primary display GPU

First boot and BIOS Configuration

Before we initiate our first boot, we’ll want to make sure that we have no loose connections, everything is properly secured, and there are no open power ports on GPUs, GPU risers or motherboard.

After we’ve confirmed everything is ready for power on, we can flip the power supply switch, man the keyboard and hit that big fancy power button. If the machine POSTs successfully, you’ll see the Gigabyte splash screen with key commands at the bottom of your display.

We’ll want to go straight in to BIOS configuration mode, which is enabled by pressing the <DEL> key while the POST is happening. When you successfully enter into BIOS configuration mode, you should see a screen like the following:

BIOS configuration screen

Navigate to Advanced Frequency Settings and then to Advanced CPU Core Features

Advanced GPU Core Features

You’ll want to enable CPU Unlock, disable Cool & Quiet, disable C1E, and disable SVM.

Then, navigate to peripherals and disable the on-board audio (since we certainly won’t need this feature and can free up the board resources for it).

Lastly, you’ll want to navigate to Power Management and set the computer to power back on during any power loss. This helps ensure the rig reboots should a power outage ever occur then power return. This is so we don’t loose precious mining time if we’re physically away from the machines to reboot them.

Once our configurations are properly set, we can navigate to Save & Exit and the rig will reboot.

This time, let the rig fully finish the POST screen and - if all is well - we’ll see ethOS start booting. Pay attention to the messages that flash across the screen during this process. You should note that ethOS should identify 6 GPUs for mining and everything should boot smoothly. If it hangs at any screen, watch the drive LED indicator we setup earlier and ensure it’s still flashing (indicating activity) if it’s still flashing, just be patient. Otherwise, if you’ve been patient enough and the flashing has ceased, try rebooting. If the problem persists, check all your equipment and connections.

Once ethOS has fully and successfully booted, we should see a screen that looks like this:

ethOS main booted screen

The first thing we’ll want to do once ethOS boots successfully is disallow mining, upgrade ethOS to the latest distro, and then setup our configurations.

# minestop && disallow

This will stop the miner from mining and disallow it from automatically restarting while we do a touch of maintenance.

Then, we’ll perform the update to ethOS

# sudo ethos-update && sleep 5 && r

Note that we’re performing the update, then sleeping for 5 seconds before calling r which is shortened alias in ethOS for rebooting the rig. The 5 second sleep gives us an opportunity to visually inspect the final results for any errors. Then the rig will reboot into the newest ethOS (which at the time of writing this article is version 1.2.2)

Changing the default ethOS password

ethOS comes preconfigured out-of-the-box with the primary user ethos and password live. We’ll change the password to something more meaningful so we can remotely access the rig for easier configuration in the future. Since ethOS is based on Ubuntu, you can change the password the same way you would Ubuntu:

# passwd ethos

You will then be prompted to put in the current password live and then your own password after that.

Note: If your rig is on a network that you don’t control, I recommend using a private key instead of password authentication. Check out this article for more information.

Once your password has been configured, you’ll be able to access the rig from within your network using SSH (mac/linux) or PuTTY (windows)

$ ssh ethos@x.x.x.x

Where x.x.x.x is the IP address of your rig. ethOS prints this to the final boot screen if you’re not sure what it is.

Configuring the mining rig for mining Ethereum

For example, I am going to to show configuring using the default pool configurations for Ethereum and ethOS (which at the time of publication is nanopool). However, I highly recommend that you thoughtfully consider which mining pool you implement. For the health of the network, pools should be competitive and decentralized.

If you checked out my introductory post, I briefly discussed the configuration scheme of ethOS (one of the powerfully simple reasons why I choose this as the mining OS). This configuration scheme essentially boils down to two configuration files: remote.conf and local.conf. For the sake of building a scalable farm operation with remote management, we’ll be leveraging the remote.conf and ignoring the local.conf for everything but debugging.

The remote.conf file located at /home/ethos/remote.conf should contain a simple, one-line url for fetching a configuration file. I recommend using a secured url that has in-url authentication so that ethOS can retrieve the url’s contents simply.

I use GitHub to manage the secure storage of my configurations (and my monitoring applications) and build an authenticated url with the following scheme:

https://<<PERSONAL_ACCESS_TOKEN:x-oauth-basic@raw.githubusercontent.com/<<GITHUB_USER>>/<<GITHUB_REPO>>/master/miner.conf

Once you’ve filled in the blanks of the above url, you can paste this into the remote.conf (I recommend you do this via ssh into the rig).

With the remote.conf file configured with the GitHub CDN url, you can now make sure that your GitHub repo is up-to-date with your latest miner.conf file. ethOS recommends using this as a template: http://ethosdistro.com/pool.txt

I recommend the following custom config for this rig specifically (I’ve removed the unnecessary comments from ethOS recommended config, but you should reference that config for further customizations/explanations):

#custompanel details
custompanel publicsecret # change this to your own custom panel replace

# global defaults ----------------------------------------

globaldriver amdgpu
maxgputemp 78
autoreboot true

globalminer sgminer-gm

# Ethereum Configuratiosn
stratumproxy enabled
sgminer-gm=proxywallet 0xYOUR_ETH_ADDRESS_HERE
sgminer-gm=proxypool1 pool-usa.ethosdistro.com:5001
sgminer-gm=proxypool2 pool-eu.ethosdistro.com:5001
#flags --cl-global-work 8192 --farm-recheck 200

#---------------------------------------------------------

# per rig configs ----------------------------------------

## Mac Address (aa:bb:cc:11:22:33) <- replace with your mac address for ref
## IP Address x.x.x.x <- replace with a static IP if you've configured for ref
# 112233 rx480s [gaming x]
cor 112233 1340 1340 1340 1340 1340 1340
mem 112233 2250 2250 2250 2250 2250 2250
fan 112233 100 100 100 100 100 100
pwr 112233 7 7 7 7 7 7
reb 112233 0
loc 112233 00
miner 112233 sgminer-gm


#---------------------------------------------------------

NOTE: before you set the per rig config section to the overclocking I’ve specified above, I recommend you put x’s in place of each number to let the default clocks take place while you stabilize your mining. These numbers will vary by GPU model, manufacturer and even by similar card based on manufacturing differences. You’ll need to tune these to find the best values.

Once your config is committed and pushed to your master branch in GitHub, you’ll be ready to roll! Check to see if the GitHub CDN has propogated your config by performing the following command from within the ethOS prompt:

# curl `cat /home/ethos/remote.conf`

This should return the config you’ve committed to your master branch in GitHub. If it is not, wait a minute or so and try again. Once you can successfully pull down a config curling the contents of the remote.conf you’ll be ready to allow your miners and reboot the rig.

# putconf && allow && r

Once the rig successfully reboots, watch to see the state of your miner and GPUs go from idle to mining. They should start posting hashrates within 3-5 minutes of booting.

If you’ve setup your mining monitor applications, you’ll be able to go to Initial State and see your rigs getting work done and starting to report earnings!

Final rig

David Sulpy

I am a computer scientist that loves to learn. I grew up building hardware and software for fun and I've since turned it into a lifestyle. I am passionate about evolving my understanding of security, software development, scalable architecture, automation, and brain science. I have an undergraduate degree in computer science and a graduate degree in information security.

Nashville, TN https://bitform.at/