Because sometimes python is not complex enough ….
Recently I had a task to connect to an existing server to get some data via python so I had two options. Handle all connections myself, communicate with the database in python, cross fingers and hope I would not break anything.
ORRR find a way to call C++ from python and use the existing libraries.
I opted to look into the second solution as it gave me a nice opportunity to research something new. The main thing for me was:
1) Easy to understand. (I am not a hardcore c++ coder)
2) No external dependencies. Just standard python and standard C++.
3) My brain should not explode while trying to get this done !!( Very important and very underestimated requirement)
Seeing as there were only a few beginner resources about this online I thought I’d make a step by step tutorial about it now that I kind of figured it out :). I can’t show how I implemented it with existing code but hopefully the example should make you understand how to do the same.
*side note:
seeing as I’m not a C++ expert I’d like to note that my C++ code might not be perfect but hopefully it will help you understand.
If you want to start learning more on C++ I recommend this resource (most corny website name ever but their basic tutorials on c++ are great):
http://www.gametutorials.com/
Their opening line usually is something like: “Talk to me like I am a 3 year old” which is why I’m finally starting to understand C++ 😛
Tools I used:
Python 2.7.3 x64
Visual Studio 2012 (any other version should do as far as I know)
Python tools for visual studio. (Ptvs)
What we will do:
- Create a c++ and pythonproject.
- Setup debugging
- Add some code that can later be called from python.
- Call the c++ dll with the ctypes module in python
Stage 1 – setup: (Skip it if you know how to setup a visual studio project for a DLL)
1.1) Open visual studio.
1.2) Create a new c++ project. Select Empty Project. (I like starting from a clean project)
1.3) Give it a name and location and click ok.
1.4) In the Solution Explorer right click on your project and go to “properties”
1.5) in the Configuration Properties -> General -> Configuration Type set it to “Dynamic Library (.dll)”
1.6) in the target extension change it to “.dll” instead of “.exe”
*Note that in some tutorials / online resources they refer to this as a .so (shared object / shared library).
Also one thing to recommend.
If you build release and debug the folder locations change. This makes it annoying for python as you have to swap between folder paths.
But you can also just change the build location for the debug and release to be the same folder path.
So change the output directory to:
$(SolutionDir)\dll\
For debug and release.
1.7) To be consistent with good coding practices lets make a header and cpp file.
Right click the “Header Files” Folder in the solution Explorer and click
“Add -> new Item”
Select the “header file (.h)” and give it a logical name for example:
RingRingPythonCalling.h
And if you want to be tidy create a proper “headers” folder in your project windows folder. Click ok and now you should have a .h file.
Lets do the same in the Source Files but now for a .cpp file
Please make sure that whatever you called your .h file that you call the .cpp file the same.
This will make it easier to manage our files and code in c++.
1.8) Creating your python project.
1) Right mouse button click on your solution and goto
Add -> New Project -> select a Python application.
2) Give it a name and put it in a logical location.
(preferably in the same folder as your c++ project )
Mine is: 
3) Now your solution should look something like this.
4) Now right click the project and click.
Set as StartUp project.
This makes sure that when you press F5 on your keyboard this python file is run. Else it will try to run the C++ file which is not possible as it’s a dll and not an executable
1.9) Settings for debugging.
Now in order to be easily able to debug our code in python and c++ at the same time do the following.
1) Right mouse click on your python project and click properties
2) Now goto debug and tick”Enable native code debugging”
3) This now allows the c++ code to be debug able.
4) Now goto
https://github.com/Microsoft/PTVS/wiki/Symbols-for-Python-mixed-mode-debugging
And download your corresponding pdb files for your python version.
5) Create a folder called symbols in your python install folder and put the pdb files in there. 
6) Now goto the menu bar
Tools -> Options
Debugging -> Symbols. .
Click on the folder icon and add the folderpath to the symbols there.
7) You might also need to click “Load All Symbols” Once for it to work.
8) Do note that enabling this makes running your python file slower.
So sometimes its recommended to just disable it. Especially once the c++ work is done or you are not making modifications in the c++ file.
Stage 2:
1.0) Creating some example code:
Open your header file by double clicking on it.
Now I wanted to find a use case for this tutorial instead of just doing a hello world.
So the code I have written here would be the basis of doing a c++ matrix or array class but focusing on the communication between c++ and python.
For example. Imagine you are taking skin cluster data from maya and you want to do some special smoothing operations on the skin cluster data.
The data that comes out of maya is usually a list of lists. So per vertex per influence
So on every row you have a vertex. And every column corresponds to every influence.
In python iterating over all these numbers can be quite slow. So if you send the vertex weight information over to c++ and do your math magic there it could be a lot faster.
So keeping this in mind I made this tutorial
In your header file write the following: *note I tried to comment the c++ to the best of my knowledge however I can not guarantee it’s a 100% correct 😀
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 |
// c++ Note: The ifndef, this is useless for python but can become usefull if the extension becomes a lot bigger. // c++ Note: This basically prevents the header from being compiled more than once if its used in multiple area's of a project. // c++ Note: If (if) this word is not(n) defined (def) #ifndef PYTHONEXTENSIONS_RINGRING_H // c++ Note: define the this word. #define PYTHONEXTENSIONS_RINGRING_H // Name spaces are a good habit to get into as it makes it easier to extend your project. // Also makes your code more readable. // Note that opinions differ on this. Do a search on stack overflow to give you months of reading material on namespaces haha. namespace PythonExtensions { // So here we define a class called PythonCalls class PythonCalls { // It has the following public members public: bool valid; // Valid boolean to keep track if the class is still valid to be called from python. // The first function here is the constructor. This gets called when the class is initialized. like a python __init__ PythonCalls(); // This is a deconstructor. it is the same name as the class but with a tilde ~ infront of it. // This is called when the class is deleted. ~PythonCalls(); // Lets create a double pointer pointer to use as an example of passing arguments around. /* A double pointer pointer can be SEEN as an list of lists in python. Its basically a list of pointers to other pointers. */ double ** array2D; // We need the info of the array because in c++ we can not query how big an array of pointers is. // So we just store the information as a member. (in python you can see this as self.rows and self.columns etc) int rows; // This will hold the info on how manny rows the array has int columns; // This will hold the info on how manny collumns the array has. // As you can see all functions return something. Even though most functions do not need to. // However there seems to be a problem in release mode where python starts garbage collecting the class object. // So returning a value prevents that from happening. int CreateArray( double ** pythonArray,int rows, int columns); int CreateArray( int rows, int columns); int PrintArray( ); int SmoothArray(int percentage); // Percentage is in 0 to 100 bool IsValid(); // just an example of a simple return boolean double ** GetArray(); // actually getting the array back again but the actual pointer. double ** GetArrayCopy(); // actually getting the array back again but this time a copy. private: }; } // Here we end our if. So the compiler knows where to end. #endif |
Now for the cpp file
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 |
#include <stdio.h> // Needed for the print function. #include <windows.h> #include "RingRingPythonCalling.h" // We need to include our header as else it does not know how the class is build up. namespace PythonExtensions { /* Here we put all our class functions and normal c++ functions. Python will not be able to call these functions directly. My guess is there is a way but I have not found it yet. See the C wrapper how we actually solve this problem. */ PythonCalls::PythonCalls( ) : rows(0), columns(0),valid(true)// We initialize our rows and columns to 0. // This is the constructor. Its like the __init__ in a python class. { // We set the valid bool to true so we can check from python if the object still lives in memory. } PythonCalls::~PythonCalls( ) // This is the deconstructor. It gets called when the object is deleted from memory. { // Here we set our boolean to false so we can check from python if we can still use the class. valid = false; // In here we should also delete any other objects i.e. our array. // But I'll leave that for now. } int PythonCalls::CreateArray( double ** pythonArray,int rowsSize, int columnsSize ) { // Creates an array from the python data. rows = (int)rowsSize; columns = (int)columnsSize; // First we create an empty array with the correct size. // We could skip this step and it would still work in this case ... // However then we would most likely be overwriting some chunk of memory somewhere // which is bad m'kay :P CreateArray(rows,columns); // And now we can assing our python array to our array member. array2D = pythonArray; return 1; }; int PythonCalls::CreateArray(int rows, int columns) { // Creates an empty array with the set size. array2D = new double *[rows]; for (int y=0;y<rows;y++) { array2D[y] = new double[columns]; for (int x=0;x<columns;x++) { array2D[y][x] = 0; } } return 1; } int PythonCalls::PrintArray( ) { printf("\nPrinting from c++. \n"); // So printing it we just itterate over the rows and columns. for (int y=0;y<rows;y++) { for (int x=0;x<columns;x++) { printf("\t%.3f \t", array2D[y][x]); } printf("\n"); } printf("End c++ print\n\n"); return 1; } int PythonCalls::SmoothArray(int percentage) { // A simple smoothing function. // Im sure there are 1000 ways to do this better but you get the idea. double total = 0; double mean = 0; double perc = (double)percentage *0.01; //printf("rows %d, columns %d",rows,columns ); for (int y=0;y<rows;y++) { total = 0; for (int x=0;x<columns;x++) { total += array2D[y][x]; } mean = total / (double)columns; for (int x=0;x<columns;x++) { if (array2D[y][x] > mean) { array2D[y][x] = array2D[y][x]- ((array2D[y][x]-mean)*perc); } else if (array2D[y][x] < mean) { array2D[y][x] = array2D[y][x] + ((mean - array2D[y][x])*perc); } } } Sleep((DWORD)0.00001); // Required for python. return 1; } bool PythonCalls::IsValid() { // This function is not really correct as if the object gets deleted so does this function. // But python will still hold a version of it and thus be able to call it and know that its no longer valid. return valid; } double ** PythonCalls::GetArray() { return array2D; } double ** PythonCalls::GetArrayCopy() { double ** arrayCopy = new double *[rows]; for (int y=0;y<rows;y++) { arrayCopy[y] = new double[columns]; for (int x=0;x<columns;x++) { arrayCopy[y][x] = array2D[y][x]; } } return arrayCopy; } ////////////////////////////////////////////////////////////////////////// // C Wrapper ////////////////////////////////////////////////////////////////////////// // Wrapper in C so python can access the functions extern "C" { __declspec(dllexport) PythonExtensions::PythonCalls * PythonCallsNew() { return new PythonExtensions::PythonCalls(); } __declspec(dllexport) int create_array(PythonExtensions::PythonCalls * pythonCallsClass, double ** pythonArray, int rows, int columns ) { return pythonCallsClass->CreateArray(pythonArray, rows, columns); } __declspec(dllexport) int create_empty_array(PythonExtensions::PythonCalls * pythonCallsClass, int rows, int columns ) { return pythonCallsClass->CreateArray( rows, columns); } __declspec(dllexport) int print_array(PythonExtensions::PythonCalls * pythonCallsClass) { return pythonCallsClass->PrintArray(); } __declspec(dllexport) int smooth_array(PythonExtensions::PythonCalls * pythonCallsClass, int percentage) { return pythonCallsClass->SmoothArray(percentage); } __declspec(dllexport) bool is_valid(PythonExtensions::PythonCalls * pythonCallsClass) { return pythonCallsClass->IsValid(); } __declspec(dllexport) double ** get_array(PythonExtensions::PythonCalls * pythonCallsClass) { return pythonCallsClass->GetArray(); } __declspec(dllexport) double ** get_array_copy(PythonExtensions::PythonCalls * pythonCallsClass) { return pythonCallsClass->GetArrayCopy(); } } } |
And last but not least the python file
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 |
''' This is a ctypes python example. It shows how to communicate with a c++ compiled dll. The reason why we do this is mainly performance or to make python communicate with other libraries. In this example we will create a 2d array (matrix) in python and send it to c++. In c++ we will smooth it and print it and send it back as example. ''' ''' Also i descided to keep the example flat in just one function. and not put it into a python class so its easier to understand. ''' import ctypes import time ## Define the size of the matrix we will make. Columns = 4 Rows = 4 def PythonToCPP(): ####################################### ## Initialisation ####################################### ## Define our libraray path libPath = r"..\c_pp_python\dll\c_pp_python.dll" ## I have seen some resources where people prefer to do C types in caps. ## I kinda like it too so Im doing the same here. Its just a personal preference. DOUBLE = ctypes.c_double PDOUBLE = ctypes.POINTER(DOUBLE) PPDOUBLE = ctypes.POINTER(PDOUBLE) INT = ctypes.c_int ## Makes a list of n doubles ## This variable can be send to a c function that takes " double * " DOUBLE_LIST = DOUBLE*Columns ## [1.0,1.0,1.0, ... ] ## Makes a array of n double pointers ## So every pointer points to a list of doubles. ## This variable can be send to a c function that takes " double ** " (note the double pointer star) PDOUBLE_Array = PDOUBLE * Rows ## [[1.0,1.0,1.0, ... ],[1.0,1.0,1.0, ... ],...] ## Now we initialize our array with doubles doublePointer = PDOUBLE_Array() ## Initialize the array with some values for i in xrange(Rows): doublePointer[i] = DOUBLE_LIST() for ii in xrange(Columns): doublePointer[i][ii] = i+ii ############################# ## Load our compiled library ############################# lib = ctypes.cdll.LoadLibrary( libPath ) ## Initialise our c++ class and store it in a variable. ## This object we pass in every time we call a function. You can kind of see it as the " self " in a python class. def __init__(self) obj = lib.PythonCallsNew(); ## Lets check if our object is still valid. print "1) Are we good to go? ", "Yes " if lib.is_valid(obj) == 1 else " No " print "Press enter to continue!" raw_input() ## Lets create the array in c++. We need to also inform c++ how big our array is when we created it. ## Note that we are putting the return value into a result. This somehow prevents python from garbage collecting. ## now the garbage collection should only happen when a result goes out of scope but this seems to do the trick. result = lib.create_array(obj,doublePointer,Rows,Columns) print "2) We created an array in c++" ## Lets print it from c++. result = lib.print_array(obj) ## We smooth our array by 50% result = lib.smooth_array(obj, 50) print "3) We smoothed the array by 50 % " ## Now lets print it from the python side. print "\nPrinting from Python" for i in xrange(Rows): for ii in xrange(Columns): print "\t",doublePointer[i][ii] , print print "\tEnd printing from python." print "\n4)Isnt this nice even though we changed it in c++ we still have access to the values from python!!" print "Press enter to continue!" raw_input() ## But what if we want to get a copy of the data so it doesnt change... ## We have a function that gets us a copy of the data from c++ (of course we can do this in python too but for example sake :) ) ## However if we just run the function from python we will get garbage data. (pointer value) ## i.e 2821472 instead of a python object like<__main__.LP_LP_c_double object at 0x0000000002580AC8> ## So we need to tell python ctypes what kind of data we are expecting lib.get_array_copy.restype = PPDOUBLE print "5)We made a copy of the array " arrayCopy = lib.get_array_copy(obj) ## We smooth our array by 50% result = lib.smooth_array(obj, 5) print "6)We smoothed the array by 50 % again " print "\nPrinting copy from Python" for i in xrange(Rows): for ii in xrange(Columns): print "\t",arrayCopy[i][ii] , print print "\tEnd printing copy from python." print "7) As you can see the copied array did not change after the smooth" print "Press enter to continue!" raw_input() ## Lets print it from c++ again result = lib.print_array(obj) print "8) But our c++ array did of course :) \n" print "Press enter to continue!" raw_input() ####################################### ## Speed testing ####################################### print "Now i also did some speed comparissons so you get an idea of the performance increase." print "Please do note that there are a LOT of optimisations that can be done on the Python and C++ side." print "Lets reset the values in the arrays and lets create a python equivalent with a list." ## Initialize the array with some values speedTestPythonArray = [] for i in xrange(Rows): #doublePointer[i] = DOUBLE_LIST() speedTestPythonArray.append( [0]*Columns) for ii in xrange(Columns): doublePointer[i][ii] = i+ii speedTestPythonArray[i][ii] = i+ii print "Press enter to start the performance test!" raw_input() cppStartTime = time.time() print "Calling the c++ smooth function 100.000 times" for x in xrange( 100000 ): result = lib.smooth_array(obj, 1) cppEndTime = time.time() - cppStartTime print "\nPrinting the c++ array" for i in xrange(Rows): for ii in xrange(Columns): print "\t",doublePointer[i][ii] , print pythonStartTime = time.time() print "Calling the python smooth function 100.000 times" for x in xrange( 100000): speedTestPythonArray = pythonSmooth(speedTestPythonArray, 1) pythonEndTime = time.time() - pythonStartTime print "\nPrinting the Python list" for i in xrange(Rows): for ii in xrange(Columns): print "\t",speedTestPythonArray[i][ii] , print print "Duration c++ ",cppEndTime print "Duration python",pythonEndTime print """So as you can see there is a clear speed improvement C++ will most likely still be below 1 second (depending on your machine). Ony my machine in release c++ is about 15 times faster than python. Duration c++ 0.0380001068115 Duration python 0.676999807358 Note that running it in debug is slower than in release. Build the code in release and press Ctrl-F5 The higher of numbers of rows anc colums the bigger the difference becomes. :) Try setting it to 20 or 40 Thats all folks. Hope it helps :) """ ## The python smooth function. def pythonSmooth(arrList, percentage): total = 0 mean = 0 perc = percentage *0.01; for y in xrange(Rows): total = 0; for x in xrange(Columns): total += arrList[y][x]; mean = total / float(Columns) for x in xrange(Columns): if arrList[y][x] > mean: arrList[y][x] = arrList[y][x]- ((arrList[y][x]-mean)*perc); elif arrList[y][x] < mean: arrList[y][x] = arrList[y][x] + ((mean - arrList[y][x])*perc); return arrList PythonToCPP() print "Press enter to exit" raw_input() |
Now the code should speak for itself and its comments.
As for running it.
Step3: Building the code.
1) Build the code in Debug or Release
If you press F7 it will build the project into a DLL.
alternatively you can go to Build –> Build solution.
Because a python file doesn’t need to be precompiled visual studio will only compile the DLL.
Scroll down to the Problems and solutions if you run into any problems or errors.
2) Now press F5.
Seeing as you set the python file to be the startup file it will run the script.
Here are the project files that you can download and test yourself.
Just running CallingCPP.py if you have python installed should work out of the box. Or you can compile it the dll yourself 🙂
Feel free to post any comments if you run into any problems
Problems and solutions:
1) When compiling you get this error:
fatal error C1083: Cannot open include file: ‘RingRingPythonCalling.h’: No such file or directory
a: Check that you wrote the name correctly of the header file.
b: Right mouse button click on your project and go to properties
Goto configuration properties -> c/c++ -> general
At Additional Include directories add the folder where your header file is located.
That should do the trick.
2) Debugging pops up the “Python Symbols Required” window.
You need to add the python symbols of the correct version.
Click download symbols it will take you to this link.
① https://github.com/Microsoft/PTVS/wiki/Symbols-for-Python-mixed-mode-debugging
② MAKE SURE YOU GET THE RIGHT VERSION.
③ To find your version do the following:
Run your python executable.
1. For me that path is “C:\Python27\python.exe”
2. Then you will see the python window appear with the version in the header. 
(note your window will most likely look black)
Now my version is 2.7.3 so make sure you also download the pdb for your corresponding version. Else the “Python Symbols required” dialog will keep popping up.
3) Python command window pops up and closes instantly.
This sometimes happens with the symbols messing with the python tools for visual studio.
The easiest workaround is to run the script without debugging.
So instead of pressing F5 press Ctrl-F5 untill you fix your bug.
4) WindowsError: exception: access violation reading 0x000007FE004148D0
There are a million reasons why you could get this error. However the one I ran into the most is:
① If you get this error the object is most likely being garbage collected by python.
② So Make sure you store the object in a variable returning from a c++ function. If your c++ function does not need to return anything just give it a int as a return value.
Additionally. I noticed that there is a more severe underlying bug that I could not find the source of.
① However I did find a solution. (Though very hacky)
② Just add a Sleep(0.0001) at the end of the function. As seen in the smoothArray function.
③ My guess is python is doing its own memory management after the function is called and moves the objects around. But the function returns to soon so python cant call the function again. This mainly happens in release and not in debug so it’s a nightmare to find.
in Edit Deformers.