Modalities

Each capability is an independent, asynchronous modality with its own gRPC service, C# service/bridge pair, and Python client. Modalities do not depend on each other — you can use any subset.

Matrix

Modality	Direction	RPC style	Python client	What it does
Connection	request/response	unary	ConnectionClient	Liveness check (ping).
Info	request/response	unary	get_server_info	Mod/engine version, OS platform, Wine flag, engine/renderer host PIDs.
Camera	Resonite → Python	server-streaming	CameraClient	RGBA frames from the headset/view.
Speaker	Resonite → Python	server-streaming	SpeakerClient	Audio rendered by Resonite (engine output tap).
Microphone	Python → Resonite	client-streaming	MicrophoneClient	Push audio into Resonite as a virtual mic.
Locomotion	Python → Resonite	client-streaming	LocomotionClient	Drive movement commands; reset.
Grabber	request/response	unary	GrabberClient	Grab at the desktop cursor ray hit point / release, then operate the held / equipped item: use / unuse hold a button down with an analog strength (primary press pressure, e.g. BrushTool/Pen; left-click aligns / activates a tool, Pen-style press-drag-release), equip / dequip handle tools (desktop mode only).
Display	request/response	unary	DisplayClient	Read display info.
World	request/response	unary	WorldClient	List/open worlds, sessions, and records.
ContextMenu	request/response	unary	ContextMenuClient	Open/select the radial context menu.
Dash	request/response	unary	DashClient	Drive the ESC dash overlay (Userspace).
Inventory	request/response	unary	InventoryClient	Browse, spawn, and fetch thumbnail images of inventory items.
Session	request/response	unary	SessionClient	Configure the connected session: settings (get/partial apply), users (kick/ban/silence/respawn/role), and permission roles.
Contact	request/response	unary	ContactClient	Manage the dash Contacts tab: list/search/add/accept/remove friends, with presence.
Auth	Python → Resonite	unary	AuthClient	Resonite cloud login/logout/status (gh-auth-like).
Cursor	request/response	unary	CursorClient	Set/hold/release/get the desktop cursor in normalized coords.
Lifecycle	Python → Resonite	unary	LifecycleClient	Ask the engine to quit gracefully (shutdown); resoio.shutdown wraps it with engine-PID reporting (resoio.terminate is the deprecated former name).

Each modality also ships a minimal runnable script under python/examples/, and the matching API Reference page links it.

Direction conventions

Audio is split by direction on purpose: Speaker carries Resonite → Python audio (what Resonite plays out), and Microphone carries Python → Resonite audio (what you push in as a virtual mic). The same split logic gives Camera (out) and the input modalities (Locomotion, Microphone) their streaming directions.

Mirrored structure

The C# and Python sides mirror each other per modality. Adding a new one means a <Modality>Service + I<Modality>Bridge in Core, a FrooxEngine<Modality>Bridge in the mod, and a <Modality>Client in Python — see the C# Mod page and the project's add-new-modality workflow.