Poor Man’s GPU
A good friend of mine at UofT, Gennady Pekhimenko, told me about the ML profiling tool — Skyline that his students developed. The original implementation presents the UI as Atom editor plugin. However, they’re working on porting it to a VSCode plugin. He asked me to give it a test run.
Currently, Skyline supports models written on PyTorch and trained on Nvidia GPUs. This makes sense since PyTorch is increasingly becoming the most popular framework, and Nvidia dominates the ML training world because of the CUDA Toolkit.
The first problem I have is that I don’t own Nvidia GPU. Even if I bought a card with Nvidia GPU and wanted to use it as eGPU, I wouldn’t be able to do that because Nvidia doesn’t ship drivers for OSX. This is due to some sort of long feud between the two companies.
Traditional cloud providers are pretty expensive. AWS on-demand P3 instance as of this writing costs $3.06/hr. There are services like Paperspace where you can rent a cloud GPU for $0.45/hr, which is a significant improvement. However, my eye fell into this new company called vast.ai, where they built a platform that lets owners of the GPU hardware lend their hardware and earn money. As a result, the prices are noticeably lower than even Paperspace.
The second problem I have is that to test the VSCode plugin, I need to run the VSCode in the desktop environment with cloud GPUs being headless Linux instances. So I spent a weekend exploring that, and it turns out it is possible. Below is the documentation of what I had to do to get it working.
First things first. Register with vast.ai. Add a little bit of money to your account. Generate SSH key pair and upload the public key to your account to be able to connect to the VM in the cloud.
cs ~/.ssh
ssh-keygen -t rsa -f vastai
cat vastai.pub | pbcopy Rent a cheap GPU with a good network
SSH to the machine using the command provided by the console and install all the dependencies. What we really need are three things:
- x11vnc — VNC server
- Xvfb — virtual framebuffer
- A desktop environment. I chose MATE, but Xfce works as well.
# Your connection string will be different
ssh -p 12345 root@ssh5.vast.ai -L 8080:localhost:8080
apt-get update
apt-get install x11vnc xvfb mate-core mate-desktop-environment mate-
notification-daemonCreate a virtual frame buffer using Xvfb and start the desktop environment.
export DISPLAY=:0
Xvfb $DISPLAY -screen 0 1024x768x16 &
mate-session &Start the VNC server
x11vnc -display $DISPLAY -bg -forever -nopw -quiet -listen localhost -xkbOn your local machine. Setup SSH tunnel.
VASTAI_PORT=12345 # Replace with SSH port supplied by Vast AI
ssh -N -T -L 5900:localhost:5900 -p $VASTAI_PORT root@ssh5.vast.ai &Point your VNC Viewer to the localhost:5900. I used Real VNC Viewer.
Profit
Now I can test the plugin on many different GPU hardware!
As a next step, I need to package the above into a container or a setup script to speed up the environment setup. After that, I plan to test this on Paperspace, which presumably has a more reliable and uniform cluster. It would be cool to get this setup running on an iPad. Brian Linkletter wrote an interesting blog post on the topic some time ago.
Overall, this opens interesting possibilities. For example, you could run CI with UI tests on cheap cloud GPUs.
Some useful links that I found as I was going through this journey:
Originally published at https://nurlybayev.family.
