Commit Graph

133 Commits

Author SHA1 Message Date
Maarten van der Schrieck 3bca93b2d0 ports: Fix sys.stdout.buffer.write() return value.
MicroPython code may rely on the return value of sys.stdout.buffer.write()
to reflect the number of bytes actually written. While in most scenarios a
write() operation is successful, there are cases where it fails, leading to
data loss. This problem arises because, currently, write() merely returns
the number of bytes it was supposed to write, without indication of
failure.

One scenario where write() might fail, is where USB is used and the
receiving end doesn't read quickly enough to empty the receive buffer. In
that case, write() on the MicroPython side can timeout, resulting in the
loss of data without any indication, a behavior observed notably in
communication between a Pi Pico as a client and a Linux host using the ACM
driver.

A complex issue arises with mp_hal_stdout_tx_strn() when it involves
multiple outputs, such as USB, dupterm and hardware UART. The challenge is
in handling cases where writing to one output is successful, but another
fails, either fully or partially. This patch implements the following
solution:

mp_hal_stdout_tx_strn() attempts to write len bytes to all of the possible
destinations for that data, and returns the minimum successful write
length.

The implementation of this is complicated by several factors:
- multiple outputs may be enabled or disabled at compiled time
- multiple outputs may be enabled or disabled at runtime
- mp_os_dupterm_tx_strn() is one such output, optionally containing
  multiple additional outputs
- each of these outputs may or may not be able to report success
- each of these outputs may or may not be able to report partial writes

As a result, there's no single strategy that fits all ports, necessitating
unique logic for each instance of mp_hal_stdout_tx_strn().

Note that addressing sys.stdout.write() is more complex due to its data
modification process ("cooked" output), and it remains unchanged in this
patch. Developers who are concerned about accurate return values from
write operations should use sys.stdout.buffer.write().

This patch might disrupt some existing code, but it's also expected to
resolve issues, considering that the peculiar return value behavior of
sys.stdout.buffer.write() is not well-documented and likely not widely
known. Therefore, it's improbable that much existing code relies on the
previous behavior.

Signed-off-by: Maarten van der Schrieck <maarten@thingsconnected.nl>
2023-12-22 10:32:46 +11:00
Angus Gratton c393f5c123 windows: Implement MICROPY_INTERNAL_WFE() macro.
This should be the equivalent of the previous event poll hook macro.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2023-12-08 12:49:54 +11:00
Damien George e68aa40d2a extmod/modmachine: Add MICROPY_PY_MACHINE_PIN_BASE option.
And use it in qemu-arm, unix and windows ports to enable PinBase.

Signed-off-by: Damien George <damien@micropython.org>
2023-11-30 16:11:11 +11:00
Damien George 7d39db2503 extmod/modmachine: Factor ports' machine module dict to common code.
This is a code factoring to have the dict for the machine module in one
location, and all the ports use that same dict.  The machine.soft_reset()
function implementation is also factored because it's the same for all
ports that did already implement it.  Eventually more functions/bindings
can be factored.

All ports remain functionally the same, except:
- cc3200 port: gains soft_reset, mem8, mem16, mem32, Signal; loses POWER_ON
  (which was a legacy constant, replaced long ago by PWRON_RESET)
- nrf port: gains Signal
- qemu-arm port: gains soft_reset
- unix port: gains soft_reset
- zephyr port: gains soft_reset, mem8, mem16, mem32

Signed-off-by: Damien George <damien@micropython.org>
2023-11-30 16:11:11 +11:00
stijn 958c6d917d windows: Use the MicroPython logo as application icon.
Add a .ico file with common icon image size, created from
vector-logo-2.png, and embed it into the resulting executable.

Signed-off-by: stijn <stijn@ignitron.net>
2023-11-07 17:22:52 +11:00
Jim Mussared 1ee5731122 ports: Remove SRC_QSTR_AUTO_DEPS from all ports' Makefiles.
It's unused.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-11-03 14:04:17 +11:00
Jim Mussared 3533924c36 extmod/moddeflate: Add deflate module providing the DeflateIO class.
This provides similar functionality to the former zlib.DecompIO and
especially CPython's gzip.GzipFile for both compression and decompression.

This class can be used directly, and also can be used from Python to
implement (via io.BytesIO) zlib.decompress and zlib.compress, as well as
gzip.GzipFile.

Enable/disable this on all ports/boards that zlib was previously configured
for.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-07-21 19:32:40 +10:00
Jim Mussared add1200343 all: Remove the zlib module.
This will be replaced with a new deflate module providing the same
functionality, with an optional frozen Python wrapper providing a
replacement zlib module.

binascii.crc32 is temporarily disabled.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-07-21 18:48:29 +10:00
Jim Mussared 2fbc08c462 extmod/asyncio: Rename uasyncio to asyncio.
The asyncio module now has much better CPython compatibility and
deserves to be just called "asyncio".

This will avoid people having to write `from uasyncio import asyncio`.

Renames all files, and updates port manifests to use the new path. Also
renames the built-in _uasyncio to _asyncio.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-19 17:33:03 +10:00
Damien George f01d5fb657 py/mkrules.mk: Automatically configure frozen options when manifest set.
Following how mkrules.cmake works.  This makes it easy for a port to enable
frozen code, by defining FROZEN_MANIFEST in its Makefile.

Signed-off-by: Damien George <damien@micropython.org>
2023-06-08 23:12:56 +10:00
Jim Mussared a1fbb1980c extmod/modtimeq: Remove timeq module.
This is a MicroPython-specific module that existed to support the old
version of uasyncio.  It's undocumented and not enabled on all ports and
takes up code size unnecessarily.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:28 +10:00
Jim Mussared 45ac651d1a all: Rename *umodule*.c to remove the "u" prefix.
Updates any includes, and references from Makefiles/CMake.

This essentially reverts what was done long ago in commit
136b5cbd76

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:17 +10:00
Jim Mussared f5f9edf645 all: Rename UMODULE to MODULE in preprocessor/Makefile vars.
This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:11 +10:00
Jim Mussared dfe232d000 py/builtinimport: Remove weak links.
In order to keep "import umodule" working, the existing mechanism is
replaced with a simple fallback to drop the "u".

This makes importing of built-ins no longer touch the filesystem, which
makes a typical built-in import take ~0.15ms rather than 3-5ms.

(Weak links were added in c14a81662c)

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:04 +10:00
David Lechner 6a4a9bc147 windows/Makefile: Allow variants to add QSTR_DEFS.
This modifies the windows port Makefile to use += for QSTR_DEFS and
QSTR_GLOBAL_DEPENDENCIES so that variants can add additional files if
needed (similar to stm32 port).

Signed-off-by: David Lechner <david@pybricks.com>
2023-05-11 14:34:15 +10:00
Damien George 127fd170c8 unix/modutime: Use extmod version of time module.
No API or functional change.

Signed-off-by: Damien George <damien@micropython.org>
2023-04-27 15:11:52 +10:00
Damien George 9955553001 extmod/modutime: Provide a generic time module.
Based on extmod/utime_mphal.c, with:
- a globals dict added
- time.localtime wrapper added
- time.time wrapper added
- time.time_ns function added

New configuration options are added for this module:
- MICROPY_PY_UTIME (enabled at basic features level)
- MICROPY_PY_UTIME_GMTIME_LOCALTIME_MKTIME
- MICROPY_PY_UTIME_TIME_TIME_NS

Signed-off-by: Damien George <damien@micropython.org>
2023-04-27 15:09:56 +10:00
Jim Mussared 69b93527d5 ports: Make BOARD default from BOARD_DIR in Makefile's.
This allows:

    $ make BOARD_DIR=path/to/board

to infer BOARD=board, rather than the previous behavior that required
additionally setting BOARD explicitly.

Also makes the same change for VARIANT_DIR -> VARIANT on Unix.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
2023-02-23 09:47:08 +11:00
stijn 6abf03b5d0 windows/msvc: Fix module freezing.
Make this more generally useful and in line with what the mingw
and unix ports do: 16bit dig size to work on 32bit ports, a
self-contained qstrdefs.preprocessed.h because makemanifest.py
uses that, and a dev variant which effectively puts this to use:
previously the uasyncio module wasn't frozen but instead tests
ran by importing it from the extmod/ directory.
2023-02-01 13:10:00 +11:00
stijn e145318a81 windows/msvc: Fix qstr generation dependency.
The mpversion.h file must exist before py/ source can be preprocessed,
but this went unnoticed because micropython.vcxproj always calls
MakeVersionHdr before MakeQstrDefs.
2023-02-01 13:10:00 +11:00
stijn 65941ea0e5 windows/msvc: Make mpy-cross independent of micropython variant.
The variant.props may have incompatible build options which break
the mpy-cross build and in any case mpy-cross has nothing to do
with variant support.
2023-02-01 13:10:00 +11:00
stijn 55a76f4edd windows/msvc: Remove variant suffix from executable filename.
This is in line with the change made for other ports in d53c3b6a: since
the default output directory already includes the variant name in it
there's no need to add it to the executable as well.
2023-02-01 13:10:00 +11:00
stijn f6f177807f windows/.gitignore: Simplify by removing build artefacts.
Since all output is now in the build-<variant>/ directory, which is already
excluded by the root .gitignore, we don't need to repeat that.
2022-12-13 17:22:11 +11:00
stijn 9c7ff87643 all: Keep msvc build output in build/ directories.
This follow the change made for Makefile-based projects in b2e82402.
2022-12-13 17:18:53 +11:00
stijn fbc96009f8 windows: Make project file read-only for IDE.
Avoids the 'warning: Wildcards in project items are not supported'
message from the C++ project system in Visual Studio, while otherwise
remaining completely functional.
2022-11-15 16:37:31 +11:00
Jim Mussared d6d8722558 extmod: Make extmod.mk self-contained.
This makes it so that all a port needs to do is set the relevant variables
and "include extmod.mk" and doesn't need to worry about adding anything to
OBJ, CFLAGS, SRC_QSTR, etc.

Make all extmod variables (src, flags, etc) private to extmod.mk.

Also move common/shared, extmod-related fragments (e.g. wiznet, cyw43,
bluetooth) into extmod.mk.

Now that SRC_MOD, CFLAGS_MOD, CXXFLAGS_MOD are unused by both extmod.mk
(and user-C-modules in a previous commit), remove all uses of them from
port makefiles.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-11 23:31:49 +11:00
Jim Mussared 17f2783e4a all: Use += rather than = everywhere for CFLAGS/LDFLAGS/LIBS.
This avoids a surprise where an = can cancel out an earlier +=.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-11 23:17:41 +11:00
Jim Mussared 203dae41fb all: Update all manifest.py files to use new features.
Changes in this commit:
- Manifest include's now use the directory path where possible (no longer
  necessary to include the manifest.py file explicitly).
- Add manifest.py for all drivers and components that are referenced by
  port/board manifests.
- Replace all uses of freeze() with package()/module(), except for port and
  board modules.
- Use opt=3 everywhere, for consistency and to reduce code size.
- Use require() instead of include() for all micropython-lib references.
- Remove support for optional board-level manifest.py in mimxrt port, to
  make it behave the same as other ports (the board must set
  FROZEN_MANIFEST to a custom manifest.py, which can optionally include the
  default, port-level manifest).
- Also reinstates modules that were accidentally removed from the esp8266
  512k build in fbe9417b90.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
2022-09-05 18:43:18 +10:00
Damien George 8f4c108025 all: Remove MICROPY_PY_IO_FILEIO config option.
Since commit e65d1e69e8 there is no longer an
io.FileIO class, so this option is no longer needed.

This option also controlled whether or not files supported being opened in
binary mode (eg 'rb'), and could, if disabled, lead to confusion as to why
opening a file in binary mode silently did the wrong thing (it would just
open in text mode if MICROPY_PY_IO_FILEIO was disabled).

The various VFS implementations (POSIX, FAT, LFS) were the only places
where enabling this option made a difference, and in almost all cases where
one of these filesystems were enabled, MICROPY_PY_IO_FILEIO was also
enabled.  So it makes sense to just unconditionally enable this feature
(ability to open a file in binary mode) in all cases, and so just remove
this config option altogether.  That makes configuration simpler and means
binary file support always exists (and opening a file in binary mode is
arguably more fundamental than opening in text mode, so if anything should
be configurable then it should be the ability to open in text mode).

Signed-off-by: Damien George <damien@micropython.org>
2022-08-18 11:54:17 +10:00
Jim Mussared 28aaab9590 py/objstr: Add hex/fromhex to bytes/memoryview/bytearray.
These were added in Python 3.5.

Enabled via MICROPY_PY_BUILTINS_BYTES_HEX, and enabled by default for all
ports that currently have ubinascii.

Rework ubinascii to use the implementation of these methods.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-12 12:44:30 +10:00
Jim Mussared 94a19f1062 windows/Makefile: Update test dependency on $(PROG).
PR #9012 (b2e8240268) changed the output to
$(BUILD)/$(PROG) but the tests are still looking for $(PROG).

Also remove the now-unnecessary override of $(PROG) in the standard
variant.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-11 16:53:43 +10:00
Damien George d53c3b6ade unix/variants: Remove variant suffix from executable filename.
The executable now lives in the build directory, and since the build
directory already contains the variant name there is no need to also add
it to the executable.

Signed-off-by: Damien George <damien@micropython.org>
2022-08-11 13:34:34 +10:00
Jim Mussared 579f330508 py/mkenv.mk: Use micropython-lib from submodule by default.
Also adds micropython-lib to 'make submodules' when using a frozen manifest
(for make and cmake).

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-03 00:08:41 +10:00
Jim Mussared 092784da19 ports: Remove unused mp_type_{fileio/textio} macros in mpconfigport.h.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-07-26 18:07:22 +10:00
stijn d05377c060 windows/msvc: Support compressed ROM text for error messages.
Enable it in the dev variant as well for consistency with the
makefile-based dev variant.
2022-07-18 23:25:38 +10:00
stijn c4adeb2e08 windows: Provide a definition for MP_ALWAYSINLINE. 2022-07-18 23:24:46 +10:00
David Lechner 81dbea1ce3 shared/readline: Use MP_REGISTER_ROOT_POINTER().
This uses MP_REGISTER_ROOT_POINTER() to register the readline_history root
pointer array used by shared/readline.c and removes the registration from
all mpconfigport.h files.

This also required adding a new MICROPY_READLINE_HISTORY_SIZE config option
since not all ports used the same sized array.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-18 13:48:49 +10:00
David Lechner fc3d7ae11b py/make_root_pointers: Add MP_REGISTER_ROOT_POINTER parser/generator.
This adds new compile-time infrastructure to parse source code files for
`MP_REGISTER_ROOT_POINTER()` and generates a new `root_pointers.h` header
file containing the collected declarations.  This works the same as the
existing `MP_REGISTER_MODULE()` feature.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-18 13:48:23 +10:00
David Lechner c118b5d0e4 extmod/extmod.mk: Separate out extmod file list from py.mk to extmod.mk.
This separates extmod source files from `py.mk`.  Previously, `py.mk`
assumed that every consumer of the py/ directory also wanted to include
extmod/.  However, this is not the case.  For example, building mpy-cross
uses py/ but doesn't need extmod/.

This commit moves all extmod-specific items from `py.mk` to `extmod.mk` and
explicitly includes `extmod.mk` in ports that use it.

Signed-off-by: David Lechner <david@pybricks.com>
2022-06-21 00:14:34 +10:00
Damien George d2f018bff1 unix,windows: Factor out code that generates random bytes to a new func.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 23:41:30 +10:00
stijn 5bb2a85d74 windows: Use BCryptGenRandom to implement os.urandom.
Fix urandom not working on windows (there's no /dev/urandom) by using
a proper cryptographic random function (same one as CPython >= 3.11).
2022-06-07 23:32:30 +10:00
Damien George 47f634300c py: Change makemoduledefs process so it uses output of qstr extraction.
This cleans up the parsing of MP_REGISTER_MODULE() and generation of
genhdr/moduledefs.h so that it uses the same process as compressed error
string messages, using the output of qstr extraction.

This makes sure all MP_REGISTER_MODULE()'s that are part of the build are
correctly picked up.  Previously the extraction would miss some (eg if you
had a mod.c file in the board directory for an stm32 board).

Build speed is more or less unchanged.

Thanks to @stinos for the ports/windows/msvc/genhdr.targets changes.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-02 16:29:53 +10:00
Damien George 6e71cde6aa ports: Use default VFS config for import_stat and builtin_open.
For ports with MICROPY_VFS and MICROPY_PY_IO enabled their configuration
can now be simplified to use the defaults for mp_import_stat and
mp_builtin_open.

This commit makes no functional change, except for the following minor
points:
- the built-in "open" is removed from the minimal port (it previously did
  nothing)
- the duplicate built-in "input" is removed from the esp32 port
- qemu-arm now delegates to VFS import/open

Signed-off-by: Damien George <damien@micropython.org>
2022-05-25 13:04:45 +10:00
Jim Mussared 63431a5536 unix: Make port-specific modules use MP_REGISTER_MODULE.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-05-18 20:57:09 +10:00
stijn 20028c7c80 windows: Avoid busy loop when using scheduler. 2022-05-04 10:22:29 +10:00
stijn 8aa79c95bd windows: Consolidate all sleep-related functions into windows_mphal.c.
Replace the timer-based sleep with the standard win32 call since the former
has no benefits: even though it allows specifying the time in 100uSec
chunks, the actual resolution is still limited by the OS and is never
better than 1mSec.

For clarity move all of this next to the mp_hal_delay_ms definition so all
related functions are in one place.
2022-05-04 10:22:09 +10:00
Damien George 0149cd6b8b windows: Switch to VFS subsystem and use VfsPosix.
Following the unix port.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-10 00:26:36 +11:00
stijn c2e8a5acd2 windows: Unify project file headers.
The xmlns attribute is required for older msbuild version (e.g. for
VS2015).  Add it where needed, and reorder the attributes so all
files look the same.
2022-02-25 16:41:11 +11:00
stijn c14f47faa3 windows/Makefile: Specify CXXFLAGS in the Makefile.
Enables building user modules which use C++ code, like the unix port.
2022-02-18 14:40:36 +11:00
David Lechner 28cb573b89 windows/appveyor: Fix printing of test failures.
In the `after_test` section, the current directory is `ports/windows` when
tests are run, so running `run-tests.py` without changing the directory or
specifying a path causes a file not found error.

This commit fixes the problem by changing the directory before calling
`run-tests.py`.

Signed-off-by: David Lechner <david@pybricks.com>
2022-02-18 14:31:52 +11:00