Using a virtualenv for IDAPython

IDA General
1

Since I just migrated to a new IDA installation, I thought I’d write up my current setup for working with Python virtual environments. Do you all have any other tricks for this?

Original post: Using a virtualenv for IDAPython

Using a virtualenv for IDAPython

I like to use a separate virtual environment for each of my Python projects and workspaces. This lets me isolate the packages and dependencies that I install, preventing versioning conflicts and generally keeping things more tidy. When I use IDA, I’d also like IDAPython to use packages from a dedicated virtualenv, rather than the system-wide configuration.

First, I create a virtualenv in a well-known location; I use $IDAUSR as a common place to put my IDA stuff:

$ python -m virtualenv ~/.idapro/venv
# or:
$ uv venv --seed ~/.idapro/venv

Now we need IDA to load this environment, rather than the global one.

Option 1: Open IDA with an already activated environment

If you lauch IDA with an environment already activated, it will stay in effect:

$ source ~/.idapro/venv/bin/activate.fish
$ ~/software/ida/ida

Then you’ll see the following in the IDA output pane:

IDAPython: Requested to use virtual environment interpreter at ~/.idapro/venv/bin/python3

The downside is that you have to remember to activate the environment every time, and that this doesn’t work with starting the application via the system menu, etc.

Alternatively, you can also set the $VIRTUAL_ENV environment variable prior to launching IDA, but it has the same problems as activating the environment: you need to do it within a console meant only for IDA (and not globally).

Option 2: Use IDAPYTHON_VENV_EXECUTABLE

Since at least 9.1, IDA supports IDAPYTHON_VENV_EXECUTABLE as a way to specify the path to the Python interpreter within a virtual environment.

This is a good fit: we can globally set IDAPYTHON_VENV_EXECUTABLE to so that IDA will always use a specific virtual environment, without overriding all Python environments on the system (which is what would happen if you set VIRTUAL_ENV globally).

Option 3: Activate the environment from idapythonrc.py

idapythonrc.py is an IDAPython-specific configuration file loaded when Python is first initialized upon IDA startup. Therefore, we can also use this file to activate the virtual environment.

We can add the following to $IDAUSR/idapythonrc.py:

import sys
import builtins
from pathlib import Path

import idaapi

_SENTINEL = "_idapython_venv_initialized"

def activate_virtualenv(virtualenv_path: Path):
    # already initialized in this process
    if getattr(builtins, _SENTINEL, False):
        return

    # already effectively active
    if Path(sys.prefix).resolve() == virtualenv_path.resolve():
        setattr(builtins, _SENTINEL, True)
        return

    for bindir in ("Scripts", "bin"):
        activate_this_path = virtualenv_path / bindir / "activate_this.py"

        if not activate_this_path.exists():
            continue

        if not activate_this_path.is_file():
            continue

        exec(activate_this_path.read_text(), dict(__file__=str(activate_this_path)))
        setattr(builtins, _SENTINEL, True)
        print("activated virtual environment: " + str(virtualenv_path))
        break

    else:
        print('Could not find "activate_this.py" in ' + str(virtualenv_path) + ". Activating site-packages directly.")
        ver = f"{sys.version_info.major}.{sys.version_info.minor}"
        site_packages = venv / "lib" / f"python{ver}" / "site-packages"
        if site_packages.exists():
            site.addsitedir(str(site_packages))

        sys.prefix = str(virtualenv_path)
        setattr(builtins, _SENTINEL, True)


IDAUSR = Path(idaapi.get_user_idadir())
activate_virtualenv(IDAUSR / "venv")

Remember, ($IDAUSR is usually ~/.idapro).

The activate_this.py file is documented here: Using Virtualenv without bin/python:

Luckily, it’s easy. You must use the custom Python interpreter to install libraries. But to use libraries, you just have to be sure the path is correct. A script is available to correct the path. You can setup the environment like:

activate_this = '/path/to/env/bin/activate_this.py'

exec(open(activate_this).read(), {‘file’: activate_this})


This will change sys.path and even change sys.prefix, but also allow you to use an existing interpreter. Items in your environment will show up first on sys.path, before global items. However, global items will always be accessible (as if the –system-site-packages flag had been used in creating the environment, whether it was or not). Also, this cannot undo the activation of other environments, or modules that have been imported. You shouldn’t try to, for instance, activate an environment before a web request; you should activate one environment as early as possible, and not do it again in that process.

I think you should also try to match the Python version used by the virtual environment with the IDAPython python version, so there’s no weirdness with native libraries or interpreter incompatibilities or something:

$ uv python install 3.12
$ uv python list
...
cpython-3.12.7-macos-aarch64-none    ~/.local/share/uv/python/cpython-3.12.7/bin/python3.12
...
$ idapyswitch --force-path ~/.local/share/uv/python/cpython-3.12.7/lib/libpython3.12.dylib

I found I was not able to use idapyswitch --force-path directly with the virtual environment contents, because IDA wants to reference the Python interpreter shared library, but the virtualenv doesn’t have a copy of it.

Anyways, with this onetime setup of either IDAPYTHON_VENV_EXECUTABLE or your idapythonrc.py, you can keep an isolated space for IDAPython. To install packages, you can either activate the environment (source ~/.idapro/venv/bin/activate.fish) or invoke pip directly: ~/.idapro/venv/bin/pip install foo.

References:

2

Thanks a lot for the post! Lots of interesting info!

You can check how it’s done in the source code. Basically, we’re mimicking what site.py does, but we do support an override with IDAPYTHON_VENV_EXECUTABLE so perhaps it can be used in your situation.

The activate_this.py trick is interesting, but seems to be specific to the virtualenv module, while we try to support the built-in venv too, which doesn’t seem to create such a script.

3

P.S. to figure out what library path to pass to idapyswitch, you can use GitHub - ktbarrett/find_libpython: Finds the libpython associated with the current Python environment, wherever it may be hiding. (some adjustment for macOS may be required).

4

Great write-up! Inspired me to push this, with a clear credit to you:

tl;dr you can’t use uv but you can use micromamba with even less resources to force IDA to respect it!

5

I am using basically the same setup as you described here. However, if I have IDA pro open and is working and go to File → Open and pick another I64 to work. I then get a problem there my idapythonrc.py is called, sys.prefix is set to the correct one but my imports from that virtual env are failing. Do anyone else get the same problems?

6

I don’t quite follow, but I’m happy to try to reproduce and triage.

I suppose if idapythonrc is manipulating sys.path then perhaps it might get updated multiple times, but I’m just guessing here. Can you share some more specific details?

7

I recorded a video

8

Any luck with reproduce my problem @williballenthin ?

9 
from pathlib import Path
import builtins
import os
import sys
import site

IDAUSR = Path(idaapi.get_user_idadir())
VENV = IDAUSR / "venv"

_SENTINEL = "_hcli_idapython_venv_initialized"

def activate_venv_once(venv: Path):
    # already initialized in this process
    if getattr(builtins, _SENTINEL, False):
        return

    # already effectively active
    if Path(sys.prefix).resolve() == venv.resolve():
        setattr(builtins, _SENTINEL, True)
        return

    ver = f"{sys.version_info.major}.{sys.version_info.minor}"
    site_packages = venv / "lib" / f"python{ver}" / "site-packages"
    if site_packages.exists():
        site.addsitedir(str(site_packages))

    sys.prefix = str(venv)
    setattr(builtins, _SENTINEL, True)

activate_venv_once(VENV)

I know this thread is ancient, but this is (most likely) the solution to the issue where the initialization script gets executed multiple times.