Pycharm professional remote Ubuntu 18.04 desktop setup for GPU usage in Macbook Pro 2020 [Best Practices]
In the last 1 month, I tried numerous times to configure my Macbook pro to utilize remote GPUs to train some Machine Learning models. There are many good articles about the same on the internet, but I found there are some very basic things, which nobody has mentioned which were actually creating a problem in my case. Hence I thought to write a new one including any future comments on any new issue while configuring PyCharm 2020.1.1 Professional Edition [Student Version] on MacBook Pro 2020 Edition to connect a Ubuntu 18.04 Remote Desktop hosting 2 GPUs [Tesla K40c and Titan Xp] with root access.
It’s better to start fresh. Let’s say you have your code/project(say HelloGPU) available in GitHub ready to be run or if you have it somewhere, you are ready to reconfigure it on both machines.
Steps to be followed on Remote Machine [UBUNTU]:
- Git clone HelloGPU to a fresh directory named $HOME/remote_projects.
- Create a fresh Conda/Python environment to run the project and install all project dependencies in that environment. This step is important as we will be using this fresh environment as a remote interpreter while configuring the local machine i.e. the Mac. If you will use any old environment, one problem I faced was it had lots of cached configuration and problems which were interfering. So it’s better to start fresh. To create a Conda environment with the name “remotenv” and with python version of 3.7, follow below step. Once the fresh environment is active, run the third command to have the address of the environment. we will need it while configuring remote interpreter in local machine.
- conda create -n remotenv python=3.7
- conda activate remotenv
- which python
3. Check if “SSH” is enabled or not by using “sudo systemctl status ssh”. You should see something like Active: active (running). If “SSH” is not active or not installed on your remote machine. Install it by using
- sudo apt update
- sudo apt install openssh-server
Check the status again. This is important as this will open up the remote machine to be connected via the internet and also enabled port 22, which we generally use to connect remote machines.
4. Lastly, run “ifconfig” on remote to know the IP address of the remote and “whoami” to know the exact username that we will use. These pieces of information will be needed while configuring PyCharm professional in local machine.
Steps to be followed on Local Machine [MAC]:
- Similar to the remote machine, we will clone the project in the local machine as well. Git clone HelloGPU to a fresh directory named $HOME/local_projects.
- Install the PyCharm Professional IDE and open the HelloGPU project.
- Go To PyCharm -> Preferences -> Select Project: HelloGPU -> Go to Project Interpreter
Select the setting icon on the right of the opened window and choose “Add”.
Select the “SSH Interpreter” option. In the Host field, put the IP address of your remote machine and in Username, put the remote user. These pieces of information you got from Step 4 in the Remote setup process.
Click Next, It will ask for password, Input the password you use to login to the remote machine. Click Next.
This is an important window, you need to change the default interpreter to point to the fresh environment you created in remote desktop i.e. “remotenv”. You have the address of the environment from step 2, third command of the Remote setup process. Check “Execute code using this interpreter with root privileges via sudo” Sync folders should be from local $HOME/local_projects to $HOME/remote_projects. Click Finish. Apply, Ok.
4. Once you are finished with Project Interpreter setup, go to PyCharm -> Preferences -> Deployment.
Check if connecting to remote by using the Test Connection button. Click on Autodetect for Root Path. Leave everything else as it is.
Go to Mappings, map local to remote project path here, it should point to HelloGPU project in local to HelloGPU project in the remote.
Click apply and/or OK. You are done.
5. Go to PyCharm -> Preferences -> Deployment -> Options. Check if Create empty Directories have been checked or not. If not, check it.
6. Go to Tools -> Deployment -> Select Automatic Upload (Always)
7. To run the code in local and emulating it in remote, use run/debug configuration on the top right corner of PyCharm. Set the script path with a file that to be executed, but choose the remote interpreter that has been configured to run/execute the project/file. PyCharm will automatically run the remote file that is mapped to the local file that you want to execute.
Good To Have:
There might be some other issues like remote project dependency error or something else as we configured a fresh python interpreter for the project. The best way is to open a terminal in PyCharm itself, connect to the remote machine using ssh “ssh USERNAME@IPADDRESS” for example “ssh jyotirmay@192.168.0.1” and then remote user password. once you are in, activate the python environment first by executing, ‘conda activate remotenv’, and install missing dependencies.
Another issue I faced is having very old GPU driver installed in remote, whereas they worked fine when I run the project manually in the remote machine. I used the 2nd method i.e. “Automatic Install using PPA repository to install Nvidia Beta drivers” mentioned in the article below to update the driver. Once updated, you need to restart the remote machine.
Let me know in the comment section if you still face any issues. I will try to cover it in this article. So that no one has to waste their time configuring this again.
SETTING UP THE SAME WITH VS-CODE:
I found this cool video about setting up a remote environment and working from your local machine using freely available one of the awesome IDE, i.e. Visual Studio Code here:
I think he has explained it pretty nicely and would be helpful for many who does not have access to the Pycharm professional. ENJOY.
