py: Update and rework build system for including external C modules.

How to use this feature is documented in docs/develop/cmodules.rst.
This commit is contained in:
Andrew Leech 2018-12-12 16:50:55 +11:00 committed by Damien George
parent 2e516074da
commit 89ff506513
12 changed files with 205 additions and 139 deletions

163
docs/develop/cmodules.rst Normal file
View File

@ -0,0 +1,163 @@
MicroPython external C modules
==============================
When developing modules for use with MicroPython you may find you run into
limitations with the Python environment, often due to an inability to access
certain hardware resources or Python speed limitations.
If your limitations can't be resolved with suggestions in :ref:`speed_python`,
writing some or all of your module in C is a viable option.
If your module is designed to access or work with commonly available
hardware or libraries please consider implementing it inside the MicroPython
source tree alongside similar modules and submitting it as a pull request.
If however you're targeting obscure or proprietary systems it may make
more sense to keep this external to the main MicroPython repository.
This chapter describes how to compile such external modules into the
MicroPython executable or firmware image.
Structure of an external C module
---------------------------------
A MicroPython user C module is a directory with the following files:
* ``*.c`` and/or ``*.h`` source code files for your module.
These will typically include the low level functionality being implemented and
the MicroPython binding functions to expose the functions and module(s).
Currently the best reference for writing these functions/modules is
to find similar modules within the MicroPython tree and use them as examples.
* ``micropython.mk`` contains the Makefile fragment for this module.
``$(USERMOD_DIR)`` is available in ``micropython.mk`` as the path to your
module directory. As it's redefined for each c module, is should be expanded
in your ``micropython.mk`` to a local make variable,
eg ``EXAMPLE_MOD_DIR := $(USERMOD_DIR)``
Your ``micropython.mk`` must add your modules C files relative to your
expanded copy of ``$(USERMOD_DIR)`` to ``SRC_USERMOD``, eg
``SRC_USERMOD += $(EXAMPLE_MOD_DIR)/example.c``
If you have custom ``CFLAGS`` settings or include folders to define, these
should be added to ``CFLAGS_USERMOD``.
See below for full usage example.
Basic Example
-------------
This simple module named ``example`` provides a single function
``example.add_ints(a, b)`` which adds the two integer args together and returns
the result.
Directory::
example/
├── example.c
└── micropython.mk
``example.c``
.. code-block:: c
// Include required definitions first.
#include "py/obj.h"
#include "py/runtime.h"
#include "py/builtin.h"
#define MODULE_EXAMPLE_ENABLED (1)
// This is the function which will be called from Python as example.add_ints(a, b).
STATIC mp_obj_t example_add_ints(mp_obj_t a_obj, mp_obj_tab_obj) {
// Extract the ints from the micropython input objects
int a = mp_obj_get_int(a_obj);
int b = mp_obj_get_int(b_obj);
// Calculate the addition and convert to MicroPython object.
return mp_obj_new_int(a + b);
}
// Define a Python reference to the function above
STATIC MP_DEFINE_CONST_FUN_OBJ_1(example_add_ints_obj, example_add_ints);
// Define all properties of the example module.
// Table entries are key/value pairs of the attribute name (a string)
// and the MicroPython object reference.
// All identifiers and strings are written as MP_QSTR_xxx and will be
// optimized to word-sized integers by the build system (interned strings).
STATIC const mp_rom_map_elem_t example_module_globals_table[] = {
{ MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_example) },
{ MP_ROM_QSTR(MP_QSTR_add_ints), MP_ROM_PTR(&example_add_ints_obj) },
};
STATIC MP_DEFINE_CONST_DICT(example_module_globals, example_module_globals_table);
// Define module object.
const mp_obj_module_t example_user_cmodule = {
.base = { &mp_type_module },
.globals = (mp_obj_dict_t*)&example_module_globals,
};
// Register the module to make it available in Python
MP_REGISTER_MODULE(MP_QSTR_example, example_user_cmodule, MODULE_EXAMPLE_ENABLED);
``micropython.mk``
.. code-block:: make
EXAMPLE_MOD_DIR := $(USERMOD_DIR)
# Add all C files to SRC_USERMOD.
SRC_USERMOD += $(EXAMPLE_MOD_DIR)/example.c
# We can add our module folder to include paths if needed
# This is not actually needed in this example.
CFLAGS_USERMOD += -I$(EXAMPLE_MOD_DIR)
Compiling the cmodule into MicroPython
--------------------------------------
To build such a module, compile MicroPython (see `getting started
<https://github.com/micropython/micropython/wiki/Getting-Started>`_) with an
extra ``make`` flag named ``USER_C_MODULES`` set to the directory containing
all modules you want included (not to the module itself). For example:
Directory::
my_project/
├── modules/
│ └──example/
│ ├──example.c
│ └──micropython.mk
└── micropython/
├──ports/
... ├──stm32/
...
Building for stm32 port:
.. code-block:: bash
cd my_project/micropython/ports/stm32
make USER_C_MODULES=../../../modules all
Module usage in MicroPython
---------------------------
Once built into your copy of MicroPython, the module implemented
in ``example.c`` above can now be accessed in Python just
like any other builtin module, eg
.. code-block:: python
import example
print(example.add_ints(1, 3))
# should display 4

15
docs/develop/index.rst Normal file
View File

@ -0,0 +1,15 @@
Developing and building MicroPython
===================================
This chapter describes modules (function and class libraries) which are built
into MicroPython. There are a few categories of such modules:
This chapter describes some options for extending MicroPython in C. Note
that it doesn't aim to be a complete guide for developing with MicroPython.
See the `getting started guide
<https://github.com/micropython/micropython/wiki/Getting-Started>`_ for further information.
.. toctree::
:maxdepth: 1
cmodules.rst

View File

@ -6,6 +6,7 @@ MicroPython documentation and references
library/index.rst
reference/index.rst
genrst/index.rst
develop/index.rst
license.rst
pyboard/quickref.rst
esp8266/quickref.rst

View File

@ -1,86 +0,0 @@
Extending MicroPython with C
============================
Some specialized code would be unacceptably slow or needs to access hardware in
a way that cannot be done from MicroPython. Therefore, it supports a way of
extending the language with custom modules written in C. But before you consider
writing a module in C, please take a look at :ref:`speed_python`.
`Unlike CPython <https://docs.python.org/3/extending/building.html>`_, these
modules are (currently) embedded directly in the program image instead of being
dynamically loaded. This requires a `custom build of MicroPython
<https://github.com/micropython/micropython/wiki/Getting-Started>`_.
Writing a module
----------------
A module is a directory with the following files:
* ``micropython.mk``, which contains the Makefile fragment for this module.
* All C files you would like included.
Put the required build commands in ``micropython.mk``. For a simple module, you
will only have to add the file paths to ``SRC_MOD``, which will include these C
files in the build:
.. highlight:: make
.. code::
# Add all C files to SRC_MOD.
SRC_MOD += $(USER_C_MODULES)/example/example.c
This is a very bare bones module named ``example`` that provides
``example.double(x)``. Note that the name of the module must be equal to the
directory name and is also used in the name of the ``mp_obj_module_t`` object at
the bottom.
.. highlight:: c
.. code::
// Include required definitions first.
#include "py/obj.h"
#include "py/runtime.h"
// This is the function you will call using example.double(n).
STATIC mp_obj_t example_double(mp_obj_t x_obj) {
// Check input value and convert it to a C type.
if (!MP_OBJ_IS_SMALL_INT(x_obj)) {
mp_raise_ValueError("x is not a small int");
}
int x = mp_obj_int_get_truncated(x_obj);
// Calculate the double, and convert back to MicroPython object.
return mp_obj_new_int(x + x);
}
STATIC MP_DEFINE_CONST_FUN_OBJ_1(example_double_obj, example_double);
// Define all properties of the example module, which currently are the name (a
// string) and a function.
// All identifiers and strings are written as MP_QSTR_xxx and will be
// optimized to word-sized integers by the build system (interned strings).
STATIC const mp_rom_map_elem_t example_module_globals_table[] = {
{ MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_example) },
{ MP_ROM_QSTR(MP_QSTR_double), MP_ROM_PTR(&example_double_obj) },
};
STATIC MP_DEFINE_CONST_DICT(example_module_globals, example_module_globals_table);
// Define module object.
const mp_obj_module_t example_user_cmodule = {
.base = { &mp_type_module },
.globals = (mp_obj_dict_t*)&example_module_globals,
};
Using a module
--------------
To build such a module, compile MicroPython (see `getting started
<https://github.com/micropython/micropython/wiki/Getting-Started>`_) with an
extra ``make`` flag named ``USER_C_MODULES`` set to the directory containing
all modules you want included (not to the module itself!). For example:
.. highlight:: shell
.. code::
$ make USER_C_MODULES=path-to-modules-folder all

View File

@ -26,11 +26,3 @@ implementation and the best practices to use them.
constrained.rst
packages.rst
asm_thumb2_index.rst
cmodules.rst
.. only:: port_pyboard
.. toctree::
:maxdepth: 1
asm_thumb2_index.rst

View File

@ -9,6 +9,8 @@ override undefine MICROPY_FORCE_32BIT
override undefine CROSS_COMPILE
override undefine FROZEN_DIR
override undefine FROZEN_MPY_DIR
override undefine USER_C_MODULES
override undefine SRC_MOD
override undefine BUILD
override undefine PROG
endif

View File

@ -494,7 +494,7 @@ $(BUILD)/firmware.hex: $(BUILD)/firmware.elf
$(BUILD)/firmware.elf: $(OBJ)
$(ECHO) "LINK $@"
$(Q)$(LD) $(LDFLAGS) -o $@ $^ $(LIBS)
$(Q)$(LD) $(LDFLAGS) -o $@ $^ $(LDFLAGS_MOD) $(LIBS)
$(Q)$(SIZE) $@
PLLVALUES = boards/pllvalues.py

View File

@ -61,7 +61,6 @@ endif
MAKE_FROZEN = $(PYTHON) $(TOP)/tools/make-frozen.py
MPY_CROSS = $(TOP)/mpy-cross/mpy-cross
MPY_TOOL = $(PYTHON) $(TOP)/tools/mpy-tool.py
GEN_CMODULES = $(PYTHON) $(TOP)/tools/gen-cmodules.py
all:
.PHONY: all

View File

@ -20,12 +20,12 @@ endif
# can be located. By following this scheme, it allows a single build rule
# to be used to compile all .c files.
vpath %.S . $(TOP)
vpath %.S . $(TOP) $(USER_C_MODULES)
$(BUILD)/%.o: %.S
$(ECHO) "CC $<"
$(Q)$(CC) $(CFLAGS) -c -o $@ $<
vpath %.s . $(TOP)
vpath %.s . $(TOP) $(USER_C_MODULES)
$(BUILD)/%.o: %.s
$(ECHO) "AS $<"
$(Q)$(AS) -o $@ $<
@ -42,14 +42,14 @@ $(Q)$(CC) $(CFLAGS) -c -MD -o $@ $<
$(RM) -f $(@:.o=.d)
endef
vpath %.c . $(TOP)
vpath %.c . $(TOP) $(USER_C_MODULES)
$(BUILD)/%.o: %.c
$(call compile_c)
QSTR_GEN_EXTRA_CFLAGS += -DNO_QSTR
QSTR_GEN_EXTRA_CFLAGS += -I$(BUILD)/tmp
vpath %.c . $(TOP)
vpath %.c . $(TOP) $(USER_C_MODULES)
$(BUILD)/%.pp: %.c
$(ECHO) "PreProcess $<"
@ -105,7 +105,7 @@ endif
ifneq ($(FROZEN_MPY_DIR),)
# to build the MicroPython cross compiler
$(TOP)/mpy-cross/mpy-cross: $(TOP)/py/*.[ch] $(TOP)/mpy-cross/*.[ch] $(TOP)/ports/windows/fmode.c
$(Q)$(MAKE) -C $(TOP)/mpy-cross USER_C_MODULES=
$(Q)$(MAKE) -C $(TOP)/mpy-cross
# make a list of all the .py files that need compiling and freezing
FROZEN_MPY_PY_FILES := $(shell find -L $(FROZEN_MPY_DIR) -type f -name '*.py' | $(SED) -e 's=^$(FROZEN_MPY_DIR)/==')
@ -123,13 +123,6 @@ $(BUILD)/frozen_mpy.c: $(FROZEN_MPY_MPY_FILES) $(BUILD)/genhdr/qstrdefs.generate
$(Q)$(MPY_TOOL) -f -q $(BUILD)/genhdr/qstrdefs.preprocessed.h $(FROZEN_MPY_MPY_FILES) > $@
endif
# to build a list of modules for py/objmodule.c.
ifneq ($(USER_C_MODULES),)
$(BUILD)/genhdr/cmodules.h: | $(HEADER_BUILD)/mpversion.h
@$(ECHO) "GEN $@"
$(Q)$(GEN_CMODULES) $(USER_C_MODULES) > $@
endif
ifneq ($(PROG),)
# Build a standalone executable (unix does this)

View File

@ -31,6 +31,8 @@
#include "py/runtime.h"
#include "py/builtin.h"
#include "genhdr/moduledefs.h"
STATIC void module_print(const mp_print_t *print, mp_obj_t self_in, mp_print_kind_t kind) {
(void)kind;
mp_obj_module_t *self = MP_OBJ_TO_PTR(self_in);

View File

@ -131,9 +131,20 @@ endif
# External modules written in C.
ifneq ($(USER_C_MODULES),)
CFLAGS_MOD += -DMICROPY_CMODULES_INCLUDE_H='"genhdr/cmodules.h"'
include $(USER_C_MODULES)/*/micropython.mk
SRC_QSTR += $(BUILD)/genhdr/cmodules.h
# pre-define USERMOD variables as expanded so that variables are immediate
# expanded as they're added to them
SRC_USERMOD :=
CFLAGS_USERMOD :=
LDFLAGS_USERMOD :=
$(foreach module, $(wildcard $(USER_C_MODULES)/*/micropython.mk), \
$(eval USERMOD_DIR = $(patsubst %/,%,$(dir $(module))))\
$(info Including User C Module from $(USERMOD_DIR))\
$(eval include $(module))\
)
SRC_MOD += $(patsubst $(USER_C_MODULES)/%.c,%.c,$(SRC_USERMOD))
CFLAGS_MOD += $(CFLAGS_USERMOD)
LDFLAGS_MOD += $(LDFLAGS_USERMOD)
endif
# py object files
@ -335,6 +346,8 @@ $(HEADER_BUILD)/moduledefs.h: $(SRC_QSTR) $(QSTR_GLOBAL_DEPENDENCIES) | $(HEADER
@$(ECHO) "GEN $@"
$(Q)$(PYTHON) $(PY_SRC)/makemoduledefs.py --vpath="., $(TOP), $(USER_C_MODULES)" $(SRC_QSTR) > $@
SRC_QSTR += $(HEADER_BUILD)/moduledefs.h
# Force nlr code to always be compiled with space-saving optimisation so
# that the function preludes are of a minimal and predictable form.
$(PY_BUILD)/nlr%.o: CFLAGS += -Os

View File

@ -1,28 +0,0 @@
#!/usr/bin/env python
# Generate genhdr/cmodules.h for inclusion in py/objmodule.c.
from __future__ import print_function
import sys
import os
from glob import glob
def update_modules(path):
modules = []
for module in sorted(os.listdir(path)):
if not os.path.isfile('%s/%s/micropython.mk' % (path, module)):
continue # not a module
modules.append(module)
# Print header file for all external modules.
print('// Automatically generated by genmodules.py.\n')
for module in modules:
print('extern const struct _mp_obj_module_t %s_user_cmodule;' % module)
print('\n#define MICROPY_EXTRA_BUILTIN_MODULES \\')
for module in modules:
print(' { MP_ROM_QSTR(MP_QSTR_%s), MP_ROM_PTR(&%s_user_cmodule) }, \\' % (module, module))
print()
if __name__ == '__main__':
update_modules(sys.argv[1])