f



real-time monitoring of propriety system: embedding python in C or embedding C in python?

Hi Group,

at work, we are thinking to replace some legacy application, which is a hom=
e-grown scripting language for monitoring and controlling a large experimen=
t. It is able to read live data from sensors, do some simple logic and calc=
ulations, send commands to other subsystems and finally generate some new s=
ignals. The way it is implemented is that it gets a chunk of 1 second of da=
ta (thousands of signals at sample rates from 1Hz to several kHz), does som=
e simple calculations on selected signals, does some simple logic, sends so=
me commands and finally computes some 1Hz output signals, all before the ne=
xt chunk of data arrives. The purpose is mainly to monitor other fast proce=
sses and adjust things like process gains and set-points, like in a SCADA s=
ystem. (I know about systems like Epics and Tango, but I cannot use those i=
n the near future.) It can be considered soft-real time: it is desirable th=
at the computation finishes within the next second most of the time, but if=
 the deadline is missed occasionally, nothing bad should happen. The curren=
t system is hard to maintain and is limited in capabilities (no advanced ma=
th, no sub-functions, ...).

I hope I don't have to convince you that python would be the perfect langua=
ge to replace such a home-grown scripting language, especially since you th=
an get all the power of tools like numpy, logging and interface to database=
s for free. Convincing my colleagues might cost some more effort, so I want=
 to write a quick (and dirty?) demonstration project. Since all the functio=
ns I have to interface with (read and write of live data, sending commands,=
 ...) are implemented in C, the solution will require writing both C and py=
thon. I have to choose between two architectures:

A) Implement the main program in C. In a loop, get a chunk of data using di=
rect call of C functions, convert data to python variables and call an embe=
dded python interpreter that runs one iteration of the user's algorithm. Wh=
en the script finishes, you read some variables from the interpreter and th=
en call some other C-function to write the results.

B) Implement the main loop in python. At the beginning of the loop, you cal=
l an embedded C function to get new data (using ctypes?), make the result r=
eadable from python (memoryview?), do the user's calculation and finally ca=
ll another C function to write the result.

Are there any advantages for using one method over the other? Note that I h=
ave more experience with python than with C.

Thanks,
Bas
0
wegwerp (104)
2/5/2013 3:10:43 PM
comp.lang.python 77058 articles. 6 followers. Post Follow

4 Replies
1374 Views

Similar Articles

[PageSpeed] 45

Bas <wegwerp@gmail.com> wrote:

> A) Implement the main program in C. In a loop, get a chunk of data
> using direct call of C functions, convert data to python variables and
> call an embedded python interpreter that runs one iteration of the
> user's algorithm. When the script finishes, you read some variables
> from the interpreter and then call some other C-function to write the
> results. 
> 
> B) Implement the main loop in python. At the beginning of the loop,
> you call an embedded C function to get new data (using ctypes?), make
> the result readable from python (memoryview?), do the user's
> calculation and finally call another C function to write the result. 
> 
> Are there any advantages for using one method over the other? Note
> that I have more experience with python than with C. 

Option B sounds like it makes your life simpler. Just turn the external 
code into a library, use ctypes to call the library and you're done. That 
also means reading command line arguments and/or config files can be done 
in Python and keep the C code simpler.

Embedding Python isn't hard but it sounds more complex than needed here.

You can of course mix the two. If it's more convenient put the main loop in 
Python but use callbacks from the library to handle the values as they 
appear, but again that probably just complicates things.

-- 
Duncan Booth http://kupuguy.blogspot.com
0
duncan.booth (1734)
2/5/2013 3:23:49 PM
On Feb 5, 8:10=A0pm, Bas <wegw...@gmail.com> wrote:
> Since all the functions I have to interface with (read and write of live =
data, sending
> commands, ...) are implemented in C, the solution will require writing bo=
th C and python.

Standard embedding/extending is ok when the interface is 'thin' ie the
number of functions going from the C-world to the python-world is not
large.

If you are dealing with a larger bunch of functions (as it seems you
are) you may want to look at swig:
http://www.swig.org/tutorial.html
0
rustompmody (868)
2/5/2013 4:14:52 PM
Bas, 05.02.2013 16:10:
> at work, we are thinking to replace some legacy application, which is a
> home-grown scripting language for monitoring and controlling a large
> experiment. It is able to read live data from sensors, do some simple
> logic and calculations, send commands to other subsystems and finally
> generate some new signals. The way it is implemented is that it gets a
> chunk of 1 second of data (thousands of signals at sample rates from 1Hz
> to several kHz), does some simple calculations on selected signals, does
> some simple logic, sends some commands and finally computes some 1Hz
> output signals, all before the next chunk of data arrives. The purpose
> is mainly to monitor other fast processes and adjust things like process
> gains and set-points, like in a SCADA system. (I know about systems like
> Epics and Tango, but I cannot use those in the near future.) It can be
> considered soft-real time: it is desirable that the computation finishes
> within the next second most of the time, but if the deadline is missed 
> occasionally, nothing bad should happen. The current system is hard to
> maintain and is limited in capabilities (no advanced math, no
> sub-functions, ...).
> 
> I hope I don't have to convince you that python would be the perfect
> language to replace such a home-grown scripting language, especially
> since you than get all the power of tools like numpy, logging and
> interface to databases for free. Convincing my colleagues might cost
> some more effort, so I want to write a quick (and dirty?) demonstration
> project. Since all the functions I have to interface with (read and
> write of live data, sending commands, ...) are implemented in C, the
> solution will require writing both C and python.

Or Cython and Python. Cython is a Python dialect that compiles down to C,
so you get native C/C++ interfacing and speed without leaving the wonderful
world of Python or the CPython runtime. It also comes with a lot of cool
features for efficient data processing that interface efficiently with NumPy.

And, yes, you'll most likely end up using NumPy in one way or another. It's
basically Python's data integration layer for high-performance and large
data computation.


> I have to choose between two architectures:
> 
> A) Implement the main program in C. In a loop, get a chunk of data using
> direct call of C functions, convert data to python variables and call an
> embedded python interpreter that runs one iteration of the user's
> algorithm. When the script finishes, you read some variables from the
> interpreter and then call some other C-function to write the results.
> 
> B) Implement the main loop in python. At the beginning of the loop, you
> call an embedded C function to get new data (using ctypes?), make the
> result readable from python (memoryview?), do the user's calculation and
> finally call another C function to write the result.
> 
> Are there any advantages for using one method over the other? Note that
> I have more experience with python than with C.

When it comes to interfacing, there is no difference between both. The only
question is really what starts up the application. This tends to be a bit
simpler if it's Python, but otherwise, it's just a one-time boilerplate
code writing thing.

The usual way to go about it is to write an extension module (commonly in
Cython) and import that from within the running Python runtime. In that
module (a shared library) you implement a wrapper around your C-level API,
often including some more high-level functionality that simplifies and
beautifies the underlying C-ish API for you. Importing that module can be
done using the normal "import" statement from Python code if the Python
runtime is running the app, and is more commonly registered at the C level
if you start Python from C.

That being said, I'd encourage you to go for the "Python runs it all"
approach first, because it tends to be less overhead and less fiddling to
get it working.

Stefan

0
stefan_ml (868)
2/5/2013 6:19:08 PM
On 2/5/2013 10:23 AM, Duncan Booth wrote:
> Bas <wegwerp@gmail.com> wrote:
>
>> A) Implement the main program in C. In a loop, get a chunk of data
>> using direct call of C functions, convert data to python variables and
>> call an embedded python interpreter that runs one iteration of the
>> user's algorithm. When the script finishes, you read some variables
>> from the interpreter and then call some other C-function to write the
>> results.
>>
>> B) Implement the main loop in python. At the beginning of the loop,
>> you call an embedded C function to get new data (using ctypes?), make
>> the result readable from python (memoryview?), do the user's
>> calculation and finally call another C function to write the result.
>>
>> Are there any advantages for using one method over the other? Note
>> that I have more experience with python than with C.
>
> Option B sounds like it makes your life simpler. Just turn the external
> code into a library, use ctypes to call the library and you're done. That
> also means reading command line arguments and/or config files can be done
> in Python and keep the C code simpler.

This is exactly how I would start. If this is not fast enough for 
production, Cython may help.

-- 
Terry Jan Reedy

0
tjreedy (5460)
2/5/2013 7:33:28 PM
Reply:

Similar Artilces:

Embedding Python in C/C++
Not a Python newb but a bit out of my depth here... We are trying to use the Python interpreter from within a C/C++ aplication to execute Python code which is in an external file. The idea is to share a dictionary between the C app and the python script. After Py_Initialize() we do something like (ignoring checks) p_module = PyImport_ImportModule("moduleX"); p_dict = PyDict_New(); Then, to add the dictionary we use PyModule_AddObject(p_module, "dictName", p_dict); Which fails. I was expecting to see an empty dictionary called "dictName" in the list of variable...

Real Time Embedded Systems Monitor in Python?
Hey Everyone! I've got a question regarding the capabilities of python in a real time environment. I'll start by saying I'm a little bit flaky when it comes to terminology, so please correct me or ask where it seems I'm not beings specific or using the wrong wording. I am looking into a project for the company I work for. Essentially it involves setting up a real time monitor / signal injector in between a CPU board and a system controller. The system controller sends signals (message packets) to the CPU board. We would like to create an environment where we can modify sign...

C binding to transfer matrices from C to Python and from Python to C
Hi all, Has someone written a C binding to transfer matrices from C to Python and vice versa ? Any pointer would be appreciated. Nils Nils Wagner <nwagner@mecha.uni-stuttgart.de> writes: > Hi all, > > Has someone written a C binding to transfer matrices from C to Python > and vice versa ? Is your question more along the lines of "I have an algorithm written in C that operates on matrices; how do I call it from Python?" I presume you want Numeric or numarray matrices. From easy to hard (and less to more powerful :-): - f2py will wrap C code. However, las...

Accessing members of Python class from C++ (Embedded Python)????
Hello everyone, Does anybody know about, have documentation on, or have any code samples on how to access class members from a python class in C++. Say I have a simple python script: --------------------------- class Animal: NumLegs = 5 Size = 4.5 --------------------------- How exactly do you access these members from C++. I know how to get the member if I know its name, ie. pkObject = PyDict_GetItemString(pkDict, "Animal"); int uiNumLegs = PyInt_AsLong(PyObject_GetAttrString(pkObject,"NumLegs")); But how do you iterate through all the class members and pri...

Embedding Python in Python
Hi, I've been programming in Python for a couple of weeks now and am quite impressed with the language - it was very easy to learn (it actually reminds me a lot of programming in BASIC many many years ago - it's very fast to hack together a couple lines of code in interactive mode). I'm throwing together a simple little game (sort of a MUD-type thing) as my first 'big' project, and I thought it would be cool to be able to script the game in Python. Basically, I want to be able to type scripts in from within the game and attach them to objects, monsters, rooms, whatever. N...

C\C++ Python embedding linking problem
Hi all, I am new to Python and trying to embed it into a c/c++ application. I started with examples from the documentation pages and go to the Pure Embedding example (http://docs.python.org/ext/pure-embedding.html). I can compile and run this example on wxDevC++ with a mingwin compiler. However when I tried this with Borland C++ Builder 5 I get linker errors. The code is the same exactly (the same main.c file), the only difference is the library I am using, as Borland requires OMF format library (I have converted it from). I am using the same binary distribution of Python on win XP. As far a...

Good examples for Embedding python in C, C++
Hallo I am looking for good examples on how to call a python script from C, C++ and pass parameters (list of strings) to the python script. The aim is to ultimately allow the end-user to modify the python script, without any code recompilation, to suit the needs of the end-user. The C, C++ part will remain intact. I could not find good examples, that could help me with this, in the references from python.org. I would appreciate your help. thanks Henko I did a presentation on this very topic at PyCon this year: http://elmer.sourceforge.net/PyCon04/elmer_pycon04.html ...and her...

Windows
I've been busy porting a Linux code to Windows, and originally decided to g= o for the mingw-w64 toolchain set, with gcc-4.8 and win32 threading model. I'm embedding Python in C++ code, and I need to be able to import specific = modules, like PySide, to eventually embed an IPython qtconsole in a C++ Qt = window. This all works under Linux, so the approach is sound. And everyth= ing compiled properly under Windows too. PySide was installed via pip. Now, from a normal Python/IPython console on the Windows system, I can impo= rt what I want: import PySide.QtCore No p...

Embedding Python in an embedded system
Howdy. I'm looking at embedding python in a little embedded system. The device (a linksys wrt54g router, popular hack object since it runs linux), has limited resources. Just 4MB flash and 16MB memory. I'm interested in Python because I need to be more agile with developing an application for this platform. Currently I am using C++ but it takes just too much time to develop/debug code. Does anyone have experience doing something like this? S. Stefan Arentz wrote: > Howdy. > > I'm looking at embedding python in a little embedded system. The device > (a linksys wr...

Embedded python on systems without python installed
I have python successfully embedded in a program I wrote. What files do I need and where do I need to put them so that it can run on systems that don't have python installed? KillSwitch wrote: > I have python successfully embedded in a program I wrote. > > What files do I need and where do I need to put them so that it can > run on systems that don't have python installed? > I embed python in Delphi apps, and the only thing I add is python24.dll, which I put in the same directory as the Delphi executable, (but i you want a less clean install, you can put the dll als...

Embedding Python in C++
Here is what I have: .... pName = PyString_FromString("btdownloadheadless"); pargs = PyString_FromString("argv = 'foo.torrent'"); PyImport = PyImport_ImportModuleEx ("btdownloadheadless",pargs, pargs, 0); .... I have a feeling my pargs is wrong. How do I make it so that it is the equivalent of typing btdownloadheadless.py --responsefile=foo.torrent in the command line ...

Embedding Python in C
Hi, I'm attempting to embedded Python23 in C. However , I seem to be missing some header files from my PC. Any idea where on the web I could locate the files below ? io.h sys/stat.h wchar.h Thanks in advance :) youngdubliner@hotmail.com wrote: > I'm attempting to embedded Python23 in C. > However , I seem to be missing some header files from my PC. > Any idea where on the web I could locate the files below ? > > io.h > > sys/stat.h > > wchar.h They are provided with Visual C. Regards, Martin ...

Python Embedding in C++
--=_IS_MIME_Boundary content-class: urn:content-classes:message Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C3BA4F.D1F66256" This is a multi-part message in MIME format. ------_=_NextPart_001_01C3BA4F.D1F66256 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi, Does it is possible to have a Python embedded C++ sample that invokes a Python method by sending my own C++ class. Into the Python script a sample of how to work with the received C++ class (Invoke the object methods or manipulate the class attrib...

C++ and Embedded Python
Is there a method to allow a pointer to an object in local memory to be shared with embedded Python within a C++ executable? The reason I require this is I have a C++ instance of a class that has data memers that are themeselves pointers to other object types. It is a fairly complicated object. In an ideal world I would just pass this pointer into an instance of the Python interpreter. This would allow the python program access and change some properties of this object instance, then pass it back to the calling C++ program. I would use this same system to pass messages between the C++ (callin...

embedding python in python #3
Hi, anyone had any experiences in embedding python in python? I've tried to do this but it doesn't work. eval("from Tkinter import *") Thanks maurice Use exec. On Wed, Sep 29, 2004 at 09:23:28AM +0000, Maurice LING wrote: > Hi, > > anyone had any experiences in embedding python in python? > > I've tried to do this but it doesn't work. > > eval("from Tkinter import *") Maurice LING <mauriceling@acm.org> wrote in message news:<415a7f0b$1@news.unimelb.edu.au>... > Hi, > > anyone had any experiences in embeddi...

Embedding Python in Python #2
Anyone know a good way to embed python within python? Now before you tell me that's silly, let me explain what I'd like to do. I'd like to allow user-defined scriptable objects. I'd like to give them access to modify pieces of my classes. I'd like to disallow access to pretty much the rest of the modules. Any ideas/examples? -Robey You probably want something like this: globalDict = {} exec(stringOfPythonCodeFromUser, globalDict) globalDict is now the global namespace of whatever was in stringOfPythonCodeFromUser, so you can grab values from that and selectivly im...

Python multithreading on cluster system? Embedding python in PVM?
Hi guys.I have read that one cannot perform true multithreading in python due to global interpreter lock mechanism.Suppose i have to implement a crawler on a say cluster system like clusterknoppix so that i can use parallel virtual machine (PVM)for programming in multiprocessor environment or say open MPI.Can i integrate python with PVM or MPI.Can i embed python into C for programming in multiprocessor environment.Is there any way of embedding python in PVM or MPI so that i can implement a true cluster based search engine? Any help would be very kind.Thanks. abhinav wrote: > Hi guys.I hav...

Calling the C API from Python and Python program from same C API
Hi Guys, I am having problems in the following C API program where myOtim_system is callable from python function. The module is listed below - static PyObject * myOptim_system(PyObject *self, PyObject *args) { const char *command; double u0, v0, u1, v1, u2, v2, u3, v3; int sts; /* variables */ PyObject *pstr, *pmod, *pdict, *pfunc, *pargs; char * cstr; char *dummy = "Test"; char *x = "x = "; if (!PyArg_ParseTuple(args, "s", &command)) return NULL; /* convert them to the doubles */ sscanf(command, "%lf %lf %lf %lf %lf %l...

c/c++ and python
Hi to all! I have a little problem. I want to develop an application in c/c++ that creates a window with gtk+ accordinly to the information on a xml file. The funcions that are called for manage the event should be written in python. I don't know how to do it, can you help me? Is it possible? Thanks a lot! Mattia Adami wrote: > Hi to all! > I have a little problem. I want to develop an application in c/c++ that > creates a window with gtk+ accordinly to the information on a xml file. > The funcions that are called for manage the event should be written in > python. I don&...

Let python call a C function pointer passed from the C Python API
I have C++ a void function pointer stored in a variable call. The pointed function takes an int and a char* as arguments. I have a python module containing this function: def yeah(x): x(int(0),"text argument") return "pointer called" As you can see I'm trying to use the argument x of the function like a method object. In the C++ side I'm doing following (note that I only paste relevant parts of code because this system is working fine, C++ is able to call function yeah and get its return value, but only with a string- oriented test): .... PyObject *pValue; // t...

Embedded python (use of modules) on system w/o Python installed
Hi there, I try to run a python script from a C program. The script uses the re (regex) module. The resulting executable must run on a system that has no lib and python script module, i.e no re.py How should I write my C program (I'd like to avoid any python file installation on target system) ? Now it is: --------- C prg ---------- #include <python2.2/Python.h> #define PYTHON_SCRIPT "test.py" int main(int argc, char * argv[]) { FILE *fd = fopen(PYTHON_SCRIPT, "r"); Py_Initialize(); PyRun_SimpleFile(fd, PYTHON_SCRIPT); Py_Finalize(); return 0; } -----...

Question about embedding python in C++
When extending python in c/c++ after you register a module is there a way in c/c++ to check if they were correctly registered? Cause I'm having the problem where when I execute the the python script it finds the module but none of the functions. TheShadow wrote: > When extending python in c/c++ after you register a module is there a > way in c/c++ to check if they were correctly registered? > > Cause I'm having the problem where when I execute the the python > script it finds the module but none of the functions. > Are you calling Py_InitModule(...) in your init...

extending and embedding python with C#
does anybody know anything about pythons ability to act as a wrapper for C#? I'm having a hard time finding out about this. Any clues are appreciated. "Luke" <webworldL@yahoo.com> wrote in news:mailman.57.1066944193.702.python-list@python.org: > does anybody know anything about pythons ability to act as a wrapper > for C#? I'm having a hard time finding out about this. Any clues are > appreciated. Depends on what you want to do. There is no out-of-the-box solution... yet. -- bjorn Bjorn Pettersen <bjorn.pettersen@comcast.net> writes: > &q...

ImportError while Embedding python in C
Hi All, I have a simple python script saved to "test.py" as import os import base64 def Testfunction(): print "Hello World" return Testfunction() I am trying to invoke this from a C program as follows int main(int argc, char* argv[]) { Py_Initialize(); PyObject* main_module = PyImport_AddModule("__main__"); PyObject* main_dict = PyModule_GetDict(main_module); FILE* file_1 = fopen(TestFile, "r"); PyRun_AnyFile(file_1, TestFile); Py_Finalize(); return 0; } This fails with the error Traceback (most recent call last): File &qu...

Web resources about - real-time monitoring of propriety system: embedding python in C or embedding C in python? - comp.lang.python

European Monitoring Centre for Drugs and Drug Addiction - Wikipedia, the free encyclopedia
The European Monitoring Centre for Drugs and Drug Addiction ( EMCDDA ) is an agency of the European Union . Established in 1993, the EMCDDA is ...

Business Wire Adds Nuvi Social Media Monitoring Reports
... how their content performed on Facebook and other social media platforms with the media-distribution company’s addition of Social Media Monitoring ...

Monitoring cache with Claspin - Facebook
Facebook Engineering hat eine Notiz mit dem Titel Monitoring cache with Claspin geschrieben. Du kannst den vollständigen Text hier lesen.

Monitoring App Quality with Insights
We’re introducing two new features in Insights : App Ratings and Negative Feedback. They both help you monitor positive and negative user signals. ...

Social Media Monitoring
Events have been a premier feature for Facebook even in its earliest iterations. A great tool for community organizing and brand promotion, Facebook ...

Announcing Bosun, our new open source monitoring & alerting system
... all the time, for no good reason. - Deep Thoughts, by Jack Handey A big part of scaling up an engineering team is getting serious about monitoring ...

Credit Karma - Free Credit Scores, Reports & Monitoring on the App Store
Read reviews, compare customer ratings, see screenshots, and learn more about Credit Karma - Free Credit Scores, Reports & Monitoring. Download ...

Apple may let users know if boss is monitoring phone
Reports say a new feature in iOS 9.3 will reveal if a phone is being tracked

Pregnancy monitoring goes high tech
A new pregnancy test tied to a health app helps users manage their stress as it monitors hormone levels.

LG has a Rolling Bot Robot for Home Monitoring, Cat Torturing
... from today surrounding the G5, they also announced the LG Rolling Bot . Think a bit of BB-8 from Star Wars, only as a reality and a home monitoring ...

Resources last updated: 3/10/2016 9:00:12 AM