Commit Graph

239 Commits

Author SHA1 Message Date
Damien George 9adedce42e py: Add new Xtensa-Windowed arch for native emitter.
Enabled via the configuration MICROPY_EMIT_XTENSAWIN.
2019-10-05 13:44:53 +10:00
Damien George 306ec5369a py/emitnative: Add support for archs that cannot read executable data.
In which case place the native function prelude in a bytes object, linked
from the const_table of that function.  An architecture should define
N_PRELUDE_AS_BYTES_OBJ to 1 before including py/emitnative.c to emit
correct machine code, then enable MICROPY_EMIT_NATIVE_PRELUDE_AS_BYTES_OBJ
so the runtime can correctly handle the prelude being in a bytes object.
2019-10-05 13:42:39 +10:00
Damien George 3504edc804 py/emitnative: Add support for using setjmp with native emitter.
To enable this feature the N_NLR_SETJMP macro should be set to 1 before
including py/emitnative.c.
2019-10-05 13:41:58 +10:00
Damien George 4107597b84 py/emitnative: Add support for archs with windowed registers.
Such that args/return regs for the parent are different to args/return regs
for child calls.  For an architecture to use this feature it should define
the REG_PARENT_xxx macros before including py/emitnative.c.
2019-10-05 13:41:14 +10:00
Damien George c8c0fd4ca3 py: Rework and compress second part of bytecode prelude.
This patch compresses the second part of the bytecode prelude which
contains the source file name, function name, source-line-number mapping
and cell closure information.  This part of the prelude now begins with a
single varible length unsigned integer which encodes 2 numbers, being the
byte-size of the following 2 sections in the header: the "source info
section" and the "closure section".  After decoding this variable unsigned
integer it's possible to skip over one or both of these sections very
easily.

This scheme saves about 2 bytes for most functions compared to the original
format: one in the case that there are no closure cells, and one because
padding was eliminated.
2019-10-01 12:26:22 +10:00
Damien George b5ebfadbd6 py: Compress first part of bytecode prelude.
The start of the bytecode prelude contains 6 numbers telling the amount of
stack needed for the Python values and exceptions, and the signature of the
function.  Prior to this patch these numbers were all encoded one after the
other (2x variable unsigned integers, then 4x bytes), but using so many
bytes is unnecessary.

An entropy analysis of around 150,000 bytecode functions from the CPython
standard library showed that the optimal Shannon coding would need about
7.1 bits on average to encode these 6 numbers, compared to the existing 48
bits.

This patch attempts to get close to this optimal value by packing the 6
numbers into a single, varible-length unsigned integer via bit-wise
interleaving.  The interleaving scheme is chosen to minimise the average
number of bytes needed, and at the same time keep the scheme simple enough
so it can be implemented without too much overhead in code size or speed.
The scheme requires about 10.5 bits on average to store the 6 numbers.

As a result most functions which originally took 6 bytes to encode these 6
numbers now need only 1 byte (in 80% of cases).
2019-10-01 12:26:22 +10:00
Damien George 81d04a0200 py: Add n_state to mp_code_state_t struct.
This value is used often enough that it is better to cache it instead of
decode it each time.
2019-10-01 12:26:22 +10:00
Damien George b596638b9b mpy-cross: Set number of registers in nlr_buf_t based on native arch.
Fixes #5059.  Done in collaboration with Jim Mussared.
2019-09-26 15:53:05 +10:00
Damien George 74503107a7 py/emitnative: Factor sizeof/offsetof calculations to macros. 2019-09-26 15:52:19 +10:00
Damien George a4f1d82757 py/nativeglue: Remove dependency on mp_fun_table in dyn-compiler mode.
mpy-cross uses MICROPY_DYNAMIC_COMPILER and MICROPY_EMIT_NATIVE but does
not actually need to execute native functions, and does not need
mp_fun_table.  This commit makes it so mp_fun_table and all its entries are
not built when MICROPY_DYNAMIC_COMPILER is enabled, significantly reducing
the size of the mpy-cross executable and allowing it to be built on more
machines/OS's.
2019-05-29 21:17:29 +10:00
Jun Wu 089c9b71d1 py: remove "if (0)" and "if (false)" branches.
Prior to this commit, building the unix port with `DEBUG=1` and
`-finstrument-functions` the compilation would fail with an error like
"control reaches end of non-void function".  This change fixes this by
removing the problematic "if (0)" branches.  Not all branches affect
compilation, but they are all removed for consistency.
2019-05-06 18:28:28 +10:00
Damien George 5ea38e4d74 py/native: Improve support for bool type in viper functions.
Variables with type bool now act more like an int, and there is proper
casting to/from Python objects.
2019-05-03 23:18:30 +10:00
Damien George d9d92f27d7 py/compile: Add support to select the native emitter at runtime. 2019-03-14 12:22:25 +11:00
Damien George 3b973a5658 py: Move mp_native_type_from_qstr() from emitnative.c to nativeglue.c. 2019-03-14 12:22:25 +11:00
Damien George 1396a026be py: Add support to save native, viper and asm code to .mpy files.
This commit adds support for saving and loading .mpy files that contain
native code (native, viper and inline-asm).  A lot of the ground work was
already done for this in the form of removing pointers from generated
native code.  The changes here are mainly to link in qstr values to the
native code, and change the format of .mpy files to contain native code
blocks (possibly mixed with bytecode).

A top-level summary:

- @micropython.native, @micropython.viper and @micropython.asm_thumb/
  asm_xtensa are now allowed in .py files when compiling to .mpy, and they
  work transparently to the user.

- Entire .py files can be compiled to native via mpy-cross -X emit=native
  and for the most part the generated .mpy files should work the same as
  their bytecode version.

- The .mpy file format is changed to 1) specify in the header if the file
  contains native code and if so the architecture (eg x86, ARMV7M, Xtensa);
  2) for each function block the kind of code is specified (bytecode,
  native, viper, asm).

- When native code is loaded from a .mpy file the native code must be
  modified (in place) to link qstr values in, just like bytecode (see
  py/persistentcode.c:arch_link_qstr() function).

In addition, this now defines a public, native ABI for dynamically loadable
native code generated by other languages, like C.
2019-03-08 15:53:05 +11:00
Damien George 3986820912 py/emitnative: Adjust accounting of size of const_table.
n_obj no longer includes a count for mp_fun_table to make it a bit simpler.
2019-03-08 15:53:04 +11:00
Damien George 205edb4305 py/emitnative: Provide concentrated points of qstr emit. 2019-03-08 15:53:04 +11:00
Damien George 01a1f31f67 py/emitnative: Consolidate where HASCONSTS is set to load-const-obj fun.
Simplifies the code and fixes handling of the Ellipsis const in native code
generation (which also needs the constant table so must set this flag).
2019-03-08 15:53:04 +11:00
Damien George 02cc288edb py: Add independent config for debugging sentinel object values.
The new compile-time option is MICROPY_DEBUG_MP_OBJ_SENTINELS, disabled by
default.  This is to allow finer control of whether this debugging feature
is enabled or not (because, for example, this setting must be the same for
mpy-cross and the MicroPython main code when using native code generation).
2019-03-08 15:53:04 +11:00
Damien George 5a2599d962 py: Replace POP_BLOCK and POP_EXCEPT opcodes with POP_EXCEPT_JUMP.
POP_BLOCK and POP_EXCEPT are now the same, and are always followed by a
JUMP.  So this optimisation reduces code size, and RAM usage of bytecode by
two bytes for each try-except handler.
2019-03-05 16:09:58 +11:00
Damien George 4ee2c2a4cd py: Eliminate warnings about unused arguments when debugging disabled. 2019-02-25 14:52:36 +11:00
Damien George de71035e02 py/emitnative: Put None/False/True in global native const table.
So these constant objects can be loaded by dereferencing the REG_FUN_TABLE
pointer instead of loading immediate values.  This reduces the size of
generated native code (when such constants are used), and means that
pointers to these constants are no longer stored in the assembly code.
2018-10-15 00:20:49 +11:00
Damien George 6c6050ca43 py/emitnative: Push internal None rather than const obj where possible.
This shifts the work of loading the constant None object on to
load_reg_stack_imm(), making the handling of None more centralised.
2018-10-15 00:20:49 +11:00
Damien George 7c16bc0406 py/emitnative: Simplify viper mode handling in emit_native_import_name. 2018-10-15 00:20:49 +11:00
Damien George 175739cd37 py/emitnative: Consolidate use of stacked immediate values to one func.
This commit adds the helper function load_reg_stack_imm() which deals with
constant immediate values and converting them to Python objects if needed.
2018-10-15 00:20:49 +11:00
Damien George 6bda951d4d py/emitnative: Remove unused ptr argument from ASM_CALL_IND macro. 2018-10-13 15:16:33 +11:00
Damien George 25571800fc py/asmthumb: Remove unused fun_ptr arg from asm_thumb_bl_ind function. 2018-10-13 15:16:33 +11:00
Damien George 5f1dd5b86b py/asmarm: Simplify asm_arm_bl_ind to only load via index, not literal.
The maximum index into mp_fun_table is currently less than 1024 and should
stay that way to keep things efficient for all architectures, so there is
no need to handle loading the pointer directly via a literal in this
function.
2018-10-13 15:16:33 +11:00
Damien George 006671056d py/emitnative: Load native fun table ptr from const table for all archs.
All architectures now have a dedicated register to hold the pointer to the
native function table mp_fun_table, and so they all need to load this
register at the start of the native function.  This commit makes the
loading of this register uniform across architectures by passing the
pointer in the constant table for the native function, and then loading the
register from the constant table.  Doing it this way means that the pointer
is not stored in the assembly code, helping to make the code more portable.
2018-10-13 15:16:33 +11:00
Damien George 355eb8eafb py/asmx86: Change indirect calls to load fun ptr from the native table.
Instead of storing the function pointer directly in the assembly code.
This makes the generated code more independent of the runtime (so easier to
relocate the code), and reduces the generated code size.
2018-10-13 15:16:33 +11:00
Damien George 8e4b4bac70 py/asmx64: Change indirect calls to load fun ptr from the native table.
Instead of storing the function pointer directly in the assembly code.
This makes the generated code more independent of the runtime (so easier to
relocate the code), and reduces the generated code size.
2018-10-13 15:16:33 +11:00
Damien George 34af10d2ef py/emitnative: Clean up unused macro and forward function declarations. 2018-10-02 15:01:56 +10:00
Damien George cc2bd63c57 py/emitnative: Implement yield and yield-from in native emitter.
This commit adds first class support for yield and yield-from in the native
emitter, including send and throw support, and yields enclosed in exception
handlers (which requires pulling down the NLR stack before yielding, then
rebuilding it when resuming).

This has been fully tested and is working on unix x86 and x86-64, and
stm32.  Also basic tests have been done with the esp8266 port.  Performance
of existing native code is unchanged.
2018-10-01 13:31:11 +10:00
Damien George 8fec6f5434 py/emitnative: Reorder native state on C stack so nlr_buf_t is first.
The nlr_buf_t doesn't need to be part of the Python value stack (as it was
before this commit), it's simpler to have it separated as auxiliary state
that lives on the C stack.  This will help adding yield support because in
that case the nlr_buf_t and Python value stack live in separate memory
areas (C stack and heap respectively).
2018-10-01 12:36:21 +10:00
Damien George e9012a20f7 py/emitnative: Change type of const_table from uintptr_t to mp_uint_t.
This matches how bytecode does it, and matches the signature of
mp_emit_glue_assign_native.  Since the native emitter doesn't support
nan-boxing uintptr_t and mp_uint_t are anyway the same bit-width.
2018-09-28 00:04:10 +10:00
Damien George 7d4b6cc868 py/emitnative: Place const objs for native code in separate const table.
This commit changes native code to handle constant objects like bytecode:
instead of storing the pointers inside the native code they are now stored
in a separate constant table (such pointers include objects like bignum,
bytes, and raw code for nested functions).  This removes the need for the
GC to scan native code for root pointers, and takes a step towards making
native code independent of the runtime (eg so it can be compiled offline by
mpy-cross).

Note that the changes to the struct scope_t did not increase its size: on a
32-bit architecture it is still 48 bytes, and on a 64-bit architecture it
decreased from 80 to 72 bytes.
2018-09-27 23:39:08 +10:00
Damien George 30a45360e7 py/asmxtensa: Make indirect calls using func table, not raw pointers.
Loading a pointer by indexing into the native function table mp_fun_table,
rather than loading an immediate value (via a PC-relative load), uses less
code space.
2018-09-16 00:43:24 +10:00
Damien George 93d71c5436 py/emitnative: Make viper funcs run with their correct globals context.
Viper functions will now capture the globals at the point they were defined
and use these globals when executing.
2018-09-15 22:39:27 +10:00
Damien George f12e039c2b py/emitnative: Use macros instead of raw offsetof for slot locations.
Old globals are now stored in the second slot (ip in mp_code_state_t) to
make things simpler for viper.
2018-09-15 22:39:27 +10:00
Damien George a676b5acf6 py/emitnative: Support arbitrary number of arguments to viper functions. 2018-09-15 22:39:27 +10:00
Damien George 43f1848bfa py: Make viper functions have the same entry signature as native.
This commit makes viper functions have the same signature as native
functions, at the level of the emitter/assembler.  This means that viper
functions can now be wrapped in the same uPy object as native functions.

Viper functions are now responsible for parsing their arguments (before it
was done by the runtime), and this makes calling them more efficient (in
most cases) because the viper entry code can be custom generated to suit
the signature of the function.

This change also opens the way forward for viper functions to take
arbitrary numbers of arguments, and for them to handle globals correctly,
among other things.
2018-09-15 22:39:27 +10:00
Damien George 460954734e py/emitnative: Reuse mp_native_type_from_qstr when searching for a cast. 2018-09-15 13:52:58 +10:00
Damien George 80db30a510 py/emit: Completely remove set_native_type, arg type is set in compiler.
In viper mode, the type of the argument is now stored in id_info->flags.
2018-09-15 13:00:11 +10:00
Damien George 07caf4f969 py/emit: Remove need to call set_native_type to set viper return type.
Instead this return type is now stored in the scope_flags.
2018-09-15 12:41:25 +10:00
Damien George 1d7c221b30 py/emit: Remove need to call set_native_type to set native/viper mode.
The native emitter can easily determine the mode via scope->emit_options.
2018-09-15 12:17:14 +10:00
Damien George 4f3d9429b5 py: Fix native functions so they run with their correct globals context.
Prior to this commit a function compiled with the native decorator
@micropython.native would not work correctly when accessing global
variables, because the globals dict was not being set upon function entry.

This commit fixes this problem by, upon function entry, setting as the
current globals dict the globals dict context the function was defined
within, as per normal Python semantics, and as bytecode does.  Upon
function exit the original globals dict is restored.

In order to restore the globals dict when an exception is raised the native
function must guard its internals with an nlr_push/nlr_pop pair.  Because
this push/pop is relatively expensive, in both C stack usage for the
nlr_buf_t and CPU execution time, the implementation here optimises things
as much as possible.  First, the compiler keeps track of whether a function
even needs to access global variables.  Using this information the native
emitter then generates three different kinds of code:

1. no globals used, no exception handlers: no nlr handling code and no
   setting of the globals dict.

2. globals used, no exception handlers: an nlr_buf_t is allocated on the
   C stack but it is not used if the globals dict is unchanged, saving
   execution time because nlr_push/nlr_pop don't need to run.

3. function has exception handlers, may use globals: an nlr_buf_t is
   allocated and nlr_push/nlr_pop are always called.

In the end, native functions that don't access globals and don't have
exception handlers will run more efficiently than those that do.

Fixes issue #1573.
2018-09-13 22:47:20 +10:00
Damien George f2de9d60f7 py/emitnative: Fix try-finally in outer scope, so finally is cancelled. 2018-09-11 15:33:25 +10:00
Damien George 4ae7111573 py/emitnative: Add support for return/break/continue in try and with.
This patch adds full support for unwinding jumps to the native emitter.
This means that return/break/continue can be used in try-except,
try-finally and with statements.  For code that doesn't use unwinding jumps
there is almost no overhead added to the generated code.
2018-09-04 14:31:28 +10:00
Damien George 3cd2c281d7 py/emitnative: Cancel caught exception once handled to prevent reraise.
The native emitter keeps the current exception in a slot in its C stack
(instead of on its Python value stack), so when it catches an exception it
must explicitly clear that slot so the same exception is not reraised later
on.
2018-09-03 17:41:02 +10:00
Damien George a0a29724c8 py/emitnative: Fix bug with store of 16 and 32 values in viper ARM mode. 2018-08-17 14:11:37 +10:00