Tulpa

LLM-powered AI companions for Minecraft

Summon a companion, then just talk to it. A real player's body — driven by a mind that runs on your machine, on your own API key.

Features · Quick start · Usage · How it works · Tools · Config · Build

Tell your companion "go mine me a stack of iron," "build a hut where I'm standing," "follow me and kill anything hostile," or "smelt these and put the bars in that chest" — a large language model plans the work and drives the companion through it: pathfinding across terrain, mining, building, fighting, crafting, and using containers, entirely on its own. Talk to it in any language your model speaks.

Features

• 🗣️ Just talk to it. Natural-language requests become real, multi-step work in the world — no scripts, no GUIs full of toggles.
• 🧠 The brain runs on your machine. The agent loop calls the LLM from the owner's client with the owner's API key. No server-side LLM, no shared bill — each player pays only for their own companion's thinking.
• 🧍 The body is a real player, not a custom mob. Companions are server-side fake players, so they interact with the world through genuine player code — they "just work" with other mods, redstone, mob AI, and containers instead of needing per-feature adapters.
• 🧭 Baritone-grade pathfinding, from scratch. A self-contained A* pathfinder reimplements Baritone's cost model and movement set 1:1 (walk, diagonal, ascend/descend, jump, parkour, pillar-up, dig-down, scaffold-bridging) — no Baritone dependency.
• 🛠️ 27 tools. Mine, build, fight (native melee + bow), craft by hand-placing into the grid, use any container, scan the world, locate structures/biomes, manage inventory, and plan with a live to-do list.
• 🔌 Bring your own model. OpenAI, DeepSeek, Kimi/Moonshot, MiniMax, Doubao, and Qwen out of the box — any OpenAI-compatible endpoint via a custom base URL.
• 🪶 No third-party SDKs. LLM transport is the JDK's built-in HttpClient + Gson. That's it.
• 🧩 One codebase, both loaders. Ships for Fabric and NeoForge (MC 26.1.2, Java 25).

Quick start

1. Install the mod (plus Fabric API if you're on Fabric) and launch the game once to generate the config.
2. Add your API key. Press G → Settings (or run /tulpa settings), pick your provider, and paste your key.
3. Summon a companion: /tulpa player summon <name>
4. Press G to open the chat and tell it what to do.

You:   挖一组铁矿回来 / go grab me a stack of iron
Tulpa: On it. Heading underground to find iron.
       ▸ 4 steps · locate_biome · move_to · auto_mine · collect_items   ✔
Tulpa: Got 64 raw iron — want me to smelt it?

Usage

Open a companion with G (it goes straight to chat if you only have one; otherwise it shows the roster). The panel has three tabs:

A left-edge HUD shows a small avatar per companion: idle ones tuck away to a thin gold sliver; when one speaks, its avatar and a speech bubble slide out together.

Commands

How it works

   You (owner's client)                          Server
   ─────────────────────                         ──────────────────────────
   chat box ─▶ agent loop ──▶ LLM API
                   │  (your key, your model)
                   │  tool calls
                   ▼
             ExecuteToolPayload ───────────▶  task queue on the fake player
                   ▲                               │  per-tick dispatcher
                   │   results                     ▼
             TaskResultPayload ◀──────────   pathfinding · mining · combat ·
                                             crafting · container use …

1. You type a message; the client-side agent loop sends the conversation + tool schemas to your chosen LLM and streams the reply.
2. Read-only tools (status, scans, recipe lookups) answer instantly; world-action tools are shipped to the server, queued on the companion's body, and run by a per-tick task dispatcher.
3. Results flow back into the conversation, so the model sees what happened and decides the next step. It can chain many actions per request — and you can hit Stop anytime.
4. History is auto-compacted (summarized) as it approaches the context window, so long sessions keep working.

The companion

• Persists like a player — state lives in a vanilla playerdata/*.dat (position, inventory, health, owner); companions return on login.
• Owned by you, across dimensions — ownership is stored and checked by UUID (vanilla getOwner() is level-scoped and breaks across dimensions, so Tulpa avoids it).
• Recoverable death — vanilla death runs normally (drops, keepInventory, grave mods all work); the agent pauses and the body respawns at the owner after a short delay.
• Shows up as a player in the tab list and renders with its own skin.

What it can do

The model expresses intent — the companion figures out how (pathing, tool choice, reach) on its own.

All 27 tools

Movement & navigation

• move_to — pathfind to coordinates (bridges, digs, jumps as needed)
• locate_structure — find the nearest structure (village, stronghold, …)
• locate_biome — find the nearest biome

Mining & building

• auto_mine — "gather N of these blocks" — finds, paths to, mines, repeats
• break_block — remove one exact block
• place_block — place a block at an exact spot (optional facing)

Combat (native player combat — real cooldowns, weapon mods, crits)

• hunt — melee N mobs of given types
• shoot — kill N mobs with a bow (charges, leads moving targets)

Inventory & items

• collect_items — pick up nearby drops
• drop_items — drop items (hand off to you / shed junk)
• equip_item — wear/wield gear
• eat_item — eat food to heal

Containers & interaction

• interact_at — aim at and use a block (open chests, furnaces, doors, levers…)
• interact_entity — follow and use a moving entity (villagers, animals…)
• transfer — move items between slots of an open GUI by intent
• inspect_gui — read the open container's slots & machine progress
• close_gui — close the container

Crafting & recipes

• lookup_recipe — JEI-style recipe lookup (crafting / smelting / stonecutter / smithing)
• (crafting itself is done by hand-placing items into the grid via transfer + interact_at)

Perception & status

• get_self_status · get_owner_status · get_world_info
• scan_blocks · scan_nearby_entities · inspect_block

Planning & meta

• todowrite — the model's own to-do list (shown live in the plan panel)
• load_skill — load a Markdown "skill" (a saved workflow) on demand
• wait — idle deliberately (smelting, nightfall, timers) instead of burning tokens

Skills are Markdown workflows in config/tulpa/skills/<name>/SKILL.md. They're advertised to the model as a table of contents and loaded full-text only when relevant, keeping the prompt lean.

Configuration

Set everything in-game (G → Settings), or edit the config file directly.

Config lives in config/tulpa.json (Fabric) or config/tulpa-common.toml (NeoForge).

Building from source

A standard Gradle multiloader project: shared code in common/, thin loader modules in fabric/ and neoforge/. The common module reaches loader-specific functionality through a small Services abstraction (config, networking, platform) resolved via ServiceLoader.

./gradlew build                 # build both loaders
./gradlew :fabric:runClient     # run the Fabric client
./gradlew :neoforge:runClient   # run the NeoForge client

Stack: Java 25 · Minecraft 26.1.2 · Fabric Loader 0.19.2 / Fabric API 0.148.2 · NeoForge 26.1.2.50-beta · Mixin + MixinExtras · LLM transport via java.net.http.HttpClient + Gson (no third-party HTTP/LLM libraries). Only three small mixins are used — the design leans on the fake-player approach instead of invasive patches.

License

Released under the MIT License — do whatever you like (use, modify, distribute, even commercially); just keep the copyright notice.