Add OWN_GIL mode for true parallel Python execution#31
Merged
Conversation
Each OWN_GIL context gets a dedicated pthread with its own GIL, enabling true parallel CPU-bound execution (4x speedup with 4 cores). - Extend py_context_t with OWN_GIL fields - Implement owngil_context_thread_main() and dispatch_to_owngil_thread() - Register erlang module in OWN_GIL subinterpreters - Add owngil mode to py_context.erl - Add test suite and benchmark Requires Python 3.12+.
- Create py_owngil_features_SUITE.erl with 42 tests across 7 groups:
channels, buffers, reentrant callbacks, pid_send, reactor, async_task, asyncio
- Implement OWN_GIL reactor dispatch for true parallel Python execution:
- Add CTX_REQ_REACTOR_ON_READ_READY, CTX_REQ_REACTOR_ON_WRITE_READY,
CTX_REQ_REACTOR_INIT_CONNECTION request types
- Add reactor_buffer_ptr field to py_context_t for buffer passing
- Implement owngil_reactor_on_read_ready/on_write_ready/init_connection
- Add dispatch_reactor_read/write/init_to_owngil functions
- Modify reactor NIFs to dispatch to OWN_GIL thread when uses_own_gil=true
- Test results: 39 passed, 3 skipped (py_reactor_context integration)
Fix reactor tests failing in OWN_GIL mode by registering the py_event_loop module during OWN_GIL thread initialization. The reactor functions call get_module_state() to access the reactor cache, which requires the py_event_loop module to exist. Without this, get_module_state() returns NULL and reactor operations fail. Enable the previously skipped py_reactor_context tests now that OWN_GIL reactor dispatch works correctly.
Enable _with_env NIFs to work with OWN_GIL mode by dispatching requests to the dedicated OWN_GIL thread. Changes: - Add request types for env variants (CALL/EVAL/EXEC/CREATE_LOCAL_ENV) - Add local_env_ptr field to py_context_t for passing env resources - Add execute functions for env variants in OWN_GIL thread - Add dispatch functions for env variants - Update _with_env NIFs to dispatch for OWN_GIL mode - Update destructor to skip DECREF for OWN_GIL envs - Add get_nif_ref/1 to py_context for direct NIF access - Add local_env tests to py_owngil_features_SUITE
Use per-interpreter module state instead of global state for async callbacks. Each subinterpreter now gets its own pipe and futures dict. Changes: - Add erlang_module_state_t struct with pipe, futures dict, and mutex - Update ErlangModuleDef to use sizeof(erlang_module_state_t) for m_size - Add get_erlang_module_state() accessor function - Add erlang_module_free() for cleanup on module deallocation - Update async_callback_init(), process_async_callback_response(), get_async_callback_fd(), send_async_callback_request(), and register_async_future() to use module state - Initialize module state in create_erlang_module()
benoitc
force-pushed
the
feature/own-gil-context
branch
from
b98b9e9 to
a79b522
Compare
- Remove subinterpreter skip from py_asyncio_compat_SUITE - Fix test_create_unix_server_existing_path to work with both ErlangEventLoop (auto-unlinks) and asyncio (manual unlink)
- Set running loop early in run_until_complete() so task factories work correctly before run_forever() is called - Remove deprecated loop= parameter from asyncio.ensure_future() - Update test_task_factory to use modern asyncio.Task API with eager_start=False parameter
Set Python event loop in thread-local storage before processing async tasks. process_ready_tasks runs on dirty NIF scheduler threads (named 'Dummy-X'), not the main thread, and Python's asyncio uses thread-local storage for event loops. The fix imports asyncio.events and sets: - The current event loop via asyncio.set_event_loop() - The running loop via events._set_running_loop() This mirrors what Python's asyncio.run() does internally. The original context is restored before releasing the GIL.
- Add test_thread_local_event_loop to verify the fix works - Refactor tests to use stdlib modules instead of __main__ - This avoids context/interpreter isolation issues where functions defined via py:exec may not be visible to the event loop worker - Fix test_timeout to use shorter sleep to avoid blocking other tests
The eager_start parameter for asyncio.Task was introduced in Python 3.12. Use version check to fall back to loop parameter on Python 3.10-3.11.
Add eager_start=False when creating tasks in ErlangEventLoop.create_task to prevent eager execution which can cause test failures.
The eager_start parameter for asyncio.Task was introduced in Python 3.12. Use version check to fall back to loop parameter on Python 3.10-3.11. Also include loop parameter in Python 3.12+ for proper task association.
Replace deprecated erlang.install() + asyncio.run() pattern with erlang.run() in py_async_e2e_SUITE tests.
Use try/except instead of check-then-pop for thread-safety in free-threaded Python. The pool check and pop are not atomic.
- Fix TOCTOU race in async_callback_init() with mutex protection - Add 30s timeout to OWN_GIL dispatch functions to prevent deadlock - Add log_and_clear_python_error() helper for debugging - Document intentional leak-vs-crash tradeoff in destructors
- Fix mutex leak in erlang_module_free: always destroy async_futures_mutex regardless of pipe_initialized flag since mutex is always initialized - Fix ABBA deadlock in event_loop_down and event_loop_destructor: acquire GIL before namespaces_mutex to match normal execution path lock ordering - Add interp_id validation in owngil_execute_*_with_env functions to detect env resources from wrong interpreter, preventing dangling pointer access - Document OWN_GIL callback re-entry limitation: erlang.call() uses thread_worker_call rather than suspension/resume protocol
- CHANGELOG: Add OWN_GIL safety fixes section for 2.2.0 - owngil_internals.md: Add Safety Mechanisms section covering interp_id validation, lock ordering (ABBA prevention), callback re-entry limitation - event_loop_architecture.md: Add Per-Process Namespace Management section with lock ordering and cleanup behavior documentation - process-bound-envs.md: Add Interpreter ID Validation and Cleanup Safety sections explaining cross-interpreter protection
CHANGELOG.md: - Add OWN_GIL Context Mode with feature list - Add Process-Local Environments for OWN_GIL - Add Per-Process Event Loop Namespaces - Add OWN_GIL Test Suites section - Add Changed section for asyncio compatibility fixes docs/owngil_internals.md: - Add Quick Start section with usage examples - Add Feature Compatibility table - Add Benchmarking section with example output docs/scalability.md: - Add OWN_GIL to mode comparison table - Add OWN_GIL Mode section with architecture, usage, process-local envs - Update subinterp section to clarify shared-GIL behavior - Add "When to use OWN_GIL" guidance
docs/process-bound-envs.md: - Add OWN_GIL Mode section with explicit environment creation - Add Sharing Context, Isolating State examples - Add When to Use Explicit vs Implicit table - Add Event Loop Environments section with examples - Add event_loop_exec/eval usage for defining async functions - Update See Also with OWN_GIL internals link docs/event_loop_architecture.md: - Add Usage section with practical examples - Add Evaluating Expressions examples - Add Process Isolation examples showing namespace independence
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary