Using a virtualenv for IDAPython

I am using basically the same setup as you described here. However, if I have IDA pro open and is working and go to File → Open and pick another I64 to work. I then get a problem there my idapythonrc.py is called, sys.prefix is set to the correct one but my imports from that virtual env are failing. Do anyone else get the same problems?