Cross-compiling NetHack 3.7 Last edit: February 18, 2023 The NetHack 3.7 build process differs from the build process of previous versions in some important ways. Those differences make it possible to use a cross-compiler running on one platform (the "host" platform of the build) to produce a binary NetHack package that can execute on an entirely different platform. Part A Contents: A1. Why cross-compile? A2. Building NetHack 3.6 (before) A3. Building NetHack 3.7 (going forward) A4. How was the build procedure reduced to 5 steps? A5. How can I help with the cross-compiling initiative? Part B Contents: B1. Two sets of compiles and procedures B2. What needs to be built and executed on the HOST? B3. What needs to be built for the TARGET? B4. Case sample: msdos B5. Case sample: amiga (started but incomplete) B6. Case sample: Web Assembly, libnh -------------------------------------------------------------------------------- Part A - Cross-compiling NetHack -------------------------------------------------------------------------------- +--------------------------+ | A1. Why cross-compile? | +--------------------------+ By using cross-compilers on host platforms with fast processors, plenty of RAM and storage resources, and an available cross-compiler, it may be possible to keep or resurrect a working version of NetHack on platforms that are now too constrained to carry out the build process natively on the platform anymore. Some of the constraints in carrying out a native build on the desired target may include, but not necessarily be limited to, any of the following: o Access: Somebody with a working knowledge of the NetHack build process may not have the desired target build platform available to them. Conversely, somebody with a keen knowledge of the target platform, and access to it, may not be all that familiar with the NetHack build process. o Resources: Address space limitations, insufficient RAM, low amounts of disk storage, slow processor performance, may impede the ability to execute the compile process on the target platform. o Compilers: Some of the native compilers on historical platforms may only support the dialect of C that was popular when the platform and compiler were in their prime. Another useful potential result of cross-compiling, is that it paves the way for carrying out test and production builds of NetHack for multiple target platforms through automated steps carried out on the host platform(s). +-------------------------------------+ | A2. Building NetHack 3.6 (before) | +-------------------------------------+ Very generally, the build of NetHack in past versions required the following steps to be carried out: 1. Compile and link util/makedefs. 2. Run makedefs repeatedly with different command line options to produce several output files that are required for: (a) additional build steps to follow, including some header files: pm.h, onames.h, date.h. (b) creation of files, containing information required by, or about the game during its execution, that are stored in a portable, platform-independent way, that need to be inserted into the game package. (c) creation of files containing information required by, or about the game during its execution, that are stored in an architecture and/or platform and/or operating system dependent way, that need to be inserted into the game package (the quest text format is one example). 3. Compile and link the level compiler. This step needs to execute work-alike tools to lex and yacc, or needs to build pre-built lex and yacc output (.c, .h files) that are provided in the sys/share part of the NetHack source code tree. 4. Execute the level compiler to read dat/*.des files and create a set of binary output files that are architecture and/or operating system dependent on the build platform, for use by the game during its execution. 5. Compile and link the dungeon compiler. Like the level compiler, this step needs to execute work-alike tools to lex and yacc, or needs to build pre-built lex and yacc output (.c, .h files) that are provided in the sys/share part of the NetHack source code tree. 6. Execute the dungeon compiler to read dat/dungeon.def and create a set of binary output files that are architecture and/or operating system dependent on the build platform, for use by the game during its execution. 7. Compile and link several less critical utilities such as uudecode, tile-generation utilities, and so forth, all of which need to execute on the build platform during the build process to produce output files for use during the game, that are reasonably portable (not architecture and/or operating system dependent; only the output of the utilities becomes part of the game package, not the executable utilities themselves. 8. Compile and link the game itself. 9. Package the game and its required files including the output from previous steps 2b, 2c, 4, 6, 7 and 8 above. Steps 1, 2a, 2b, 7, and 9 above are not impediments to cross-compiling NetHack. Steps 2c, 3, 4, 5, 6 and 8 above are impediments to cross-compiling NetHack. That's because the files that those steps produce are very much tied to the platform where the build of NetHack is being carried out. Variations between platforms (such as 32-bit vs 64-bit, integer sizes, pointer sizes, processor byte order, data alignment requirements, struct padding and the way bitfields are stored) impact the portability of those data files between different platforms and operating systems. If all those things happen to match, the files might, just might, be usable across platforms, but the chances are against it, and that certainly cannot be counted on. +------------------------------------------+ | A3. Building NetHack 3.7 (going forward) | +------------------------------------------+ Again, very generally, the build of NetHack in 3.7 requires the following steps to be carried out: 1. Compile and link util/makedefs. 2. Run makedefs repeatedly with different command line options to produce several required output files that contain information required by the game, or contain information about the game during its execution, that are stored in a portable, platform-independent way, that need to be inserted into the game package (makedefs -d, -o, -r, -h, -s). 3. Compile and link several less critical utilities such as uudecode, tile-generation utilities, and so forth, all of which need to execute on the build platform during the build process to produce output files for use during the game, that are reasonably portable (not architecture and/or operating system dependent; only the output of the utilities becomes part of the game package, not the executable utilities themselves. 4. Compile and link the game components for the TARGET; that includes NetHack itself, Lua, and any optional regular-expression or window port libraries that you plan to link into the NetHack game executable. 5. Package the game and its required files including the output from previous steps 2, 3 and 4 above. Step 4 is now the only impediment to cross-compiling NetHack. That impediment is resolved by executing step 4 using a cross-compiler that runs on the build (host) platform to produce a resulting binary for the target platform, instead of executing the native compiler. +-----------------------------------------------------+ | A4. How was the build procedure reduced to 5 steps? | +-----------------------------------------------------+ The following are among several design changes planned in NetHack 3.7, and these specific changes are what altered the build process to make cross-compiling possible: o There is no creation of platform-dependent files, such as the quest text files, by makedefs during the build process. Instead, the quest text files have been converted to Lua and are inserted into the game package for processing by the embedded Lua during execution of NetHack. o There is no build-time level compiler involved. Instead, the level descriptions have been converted to Lua and are inserted into the game package for processing by the embedded Lua during execution of NetHack. o There is no build-time dungeon compiler involved. Instead, the dungeon description has been converted to Lua and is inserted into the game package for processing by the embedded Lua during execution of NetHack. o Some of the build and option information that was formerly produced during build time by makedefs, and contained information about the build-platform specifically, is now produced at runtime within the game under a cross-compiled build. As such, it now produces information applicable to the target NetHack environment, not the build environment. +------------------------------------------------------------+ | A5. How can I help with the cross-compiling initiative? | +------------------------------------------------------------+ o If you have a favourite target platform (let's call it XX-Platform for example purposes) that you'd like to see NetHack be able to run on, do some research to find out if a cross-compiler exists that: - produces output for XX-Platform. - executes on a platform that you use and love (Linux, Windows, macOS are some examples of platforms that have cross-compilers for other targets available) o Then, make the community, devteam, and others aware that you're starting a cross-compile of NetHack for XX-Platform. You might need to ask some "starting out" questions initially, and as you get deeper into it, you might need to ask some tougher questions. o Perhaps consider forking from NetHack on GitHub, and do the cross-compiler work there in your fork. Strive to get it to a point where it's ready to play-test on XX-Platform, or perhaps even use an emulator of XX-Platform if one is available. We live in a time where plenty do. Doing your work on a GitHub fork has the following advantages: - It will make it really simple to integrate your work back into the NetHack source tree if that's one of your goals. - It will make it possible and straightforward to merge upstream NetHack changes into your work for the XX-Platform cross-compile so that it stays current with the game as it evolves. - You may get help from others in the form of suggestions, or pull-requests, or offers to join the development. Chances are, you aren't the only person out there that would like to establish/resurrect/maintain NetHack on XX-Platform. Have fun! ----------------------------------------------------------------------------- Part B - Cross-compiling details ----------------------------------------------------------------------------- Part B Contents: B1. Two sets of compiles and procedures B2. What needs to be built and executed on the HOST? B3. What needs to be built for the TARGET? B4. Case sample: msdos +-----------------------------------------+ | B1. Two sets of compiles and procedures | +-----------------------------------------+ The HOST is the platform/place that you're running the build procedures using the native compiler/linker, and the cross-compiler/linker that runs on the HOST to build the game for a TARGET platform. You have to: 1. Build mandatory utilities on the HOST. 2. Execute the mandatory utilities to generate components that will be used during step 4 to build the game itself, or that will generate components that will become part of the TARGET game package. 3. Build optional or enhancing utilities on the HOST, execute those optional or enhancing utilities on the HOST to generate components that will become part of the TARGET game package. 4. Execute a supported cross-compiler to compile the rest of the game components like NetHack, Lua, and any optional libraries (the word "supported", in this sense, means a compiler that runs on your HOST build platform, and generates output executable files for the TARGET platform). It should be mentioned that you can execute the cross-compile build approach to generate binaries for the same platform as the host, where the HOST compiler and the TARGET compiler are the same. +------------------------------------------------------+ | B2. What needs to be built and executed on the HOST? | +------------------------------------------------------+ On the HOST, here are the mandatory things that have to be built. a) Using the HOST native compiler, build HOST native utility makedefs Compile and link the following with these compiler switches: -DCROSSCOMPILE from sources: util/makedefs.c, src/mdlib.c, src/monst.c, src/objects.c src/date.c, src/alloc.c b) Execute HOST native makedefs utility, util/makedefs, as follows: Required for complete packaging of the game, but not the C source game compile: util/makedefs -d util/makedefs -r util/makedefs -h util/makedefs -s c) Using the HOST native compiler, build these additional utilities if your target platform requires components that they produce. It is important to note that all of the required source files need to be compiled to native obj files for linking into the HOST-side utility. Some of the source files (src/monst.c, src/objects.c) were likely already compiled as native HOST-side obj files in order to build the native HOST utility 'makedefs' HOST utility above, and you don't need to compile them again for use in linking other HOST utilities if the HOST-native obj files produced from them are still around. NOTE: There are some source files that need to be compiled for linking into utilities that are needed on the HOST side, but that are also needed again later as TARGET obj files for linking into the TARGET executable, and therefore must be compiled twice. They include: src/drawing.c, src/objects.c, src/monst.c, src/alloc.c, util/panic.c. It is important to keep the compiled HOST-side obj files produced by the HOST native compiler distinct from the TARGET-side obj files produced by the cross-compiler. That can be accomplished either by naming the generated object files a little differently (perhaps with a suffix) between the HOST-side and the TARGET-side, or by placing the HOST-side obj files and the TARGET-side obj files into distinct, different directories during the build process. Use whatever method works best for your cross-compile effort and tools. util/dlb from sources: src/dlb.c, src/dlb_main.c, src/alloc.c, util/panic.c purpose: For packaging up many files that are required components of the TARGET game into a single nhdat or nhdat370 combined file util/uudecode from sources: sys/share/uudecode.c purpose: convert some binary files, that are distributed in the NetHack sources in uuencoded format, back into their original binary state util/tilemap from sources: win/share/tilemap.c purpose: produce output file src/tile.c that is required for building TARGET packages with tile support util/tile2bmp from sources: win/share/tile2bmp.c, win/share/tiletext.c, win/share/tilemap.c, src/drawing.c, src/monst.c, src/objects.c purpose: Read win/share/monsters.txt, win/share/objects.txt and win/share/other.txt files and produce a .bmp file. util/gif2txt from sources: win/share/gifread.c, win/share/tiletext.c, win/share/tilemap.c, src/drawing.c, src/monst.c, src/objects.c, src/alloc.c, util/panic.c purpose: Read win/share/monsters.txt, win/share/objects.txt and win/share/other.txt files and produce a .gif file. util/ppmwrite from sources: win/share/ppmwrite.c, win/share/tiletext.c, win/share/tilemap.c, src/drawing.c, src/monst.c, src/objects.c, src/alloc.c, util/panic.c purpose: Read win/share/monsters.txt, win/share/objects.txt and win/share/other.txt files and produce a .ppm file. +--------------------------------------------+ | B3. What needs to be built for the TARGET? | +--------------------------------------------+ For the TARGET side, here are the mandatory things that have to be built via the HOST-executed cross-compiler that generates code for the TARGET platform. Using the cross-compiler, build the following targets: a) NetHack sources (core is mandatory) With the cross-compiler and linker for the TARGET platform, cross-compile and link with these compiler switches: -DCROSSCOMPILE and -DCROSSCOMPILE_TARGET core sources (2019): src/allmain.c, src/alloc.c, src/apply.c, src/artifact.c, src/attrib.c, src/ball.c, src/bones.c, src/botl.c, src/cmd.c, src/dbridge.c, src/decl.c, src/detect.c, src/dig.c, src/display.c, src/dlb.c, src/do.c, src/do_name.c, src/do_wear.c, src/dog.c, src/dogmove.c, src/dokick.c, src/dothrow.c, src/drawing.c, src/dungeon.c, src/eat.c, src/end.c, src/engrave.c, src/exper.c, src/explode.c, src/extralev.c, src/files.c, src/fountain.c, src/hack.c, src/hacklib.c, src/insight.c, src/invent.c, src/isaac64.c, src/light.c, src/lock.c, src/mail.c, src/makemon.c, src/mcastu.c, src/mdlib.c, src/mhitm.c, src/mhitu.c, src/minion.c, src/mklev.c, src/mkmap.c, src/mkmaze.c, src/mkobj.c, src/mkroom.c, src/mon.c, src/mondata.c, src/monmove.c, src/monst.c, src/mplayer.c, src/mthrowu.c, src/muse.c, src/music.c, src/nhlsel.c, src/nhlua.c, src/nhlobj.c, src/o_init.c, src/objects.c, src/objnam.c, src/options.c, src/pager.c, src/pickup.c, src/pline.c, src/polyself.c, src/potion.c, src/pray.c, src/priest.c, src/quest.c, src/questpgr.c, src/read.c, src/rect.c, src/region.c, src/restore.c, src/rip.c, src/rnd.c, src/role.c, src/rumors.c, src/save.c, src/sfstruct.c, src/shk.c, src/shknam.c, src/sit.c, src/sounds.c, src/sp_lev.c, src/spell.c, src/steal.c, src/steed.c, src/symbols.c, src/sys.c, src/teleport.c, src/timeout.c, src/topten.c, src/track.c, src/trap.c, src/u_init.c, src/uhitm.c, src/vault.c, src/version.c, src/vision.c, src/weapon.c, src/were.c, src/wield.c, src/windows.c, src/wizard.c, src/worm.c, src/worn.c, src/write.c, src/zap.c, sys/share/cppregex.cpp tty sources: win/tty/getline.c, win/tty/termcap.c, win/tty/topl.c, win/tty/wintty.c generated (if req'd): src/tile.c plus your platform-specific source files that contain main, typically *main.c, and unix support in *unix.c, tty support in *tty.c, and other system support in *sys.c as well as others sources pertaining to your specific target platform(s). b) Lua (mandatory in 3.7) lib/lua-5.4.6/src from sources: lua.c, lapi.c, lauxlib.c, lbaselib.c, lcode.c, lcorolib.c, lctype.c, ldblib.c, ldebug.c, ldo.c, ldump.c, lfunc.c, lgc.c, linit.c, liolib.c, llex.c, lmathlib.c, lmem.c, loadlib.c, lobject.c, lopcodes.c, loslib.c, lparser.c, lstate.c, lstring.c, lstrlib.c, ltable.c, ltablib.c, ltm.c, lundump.c, lutf8lib.c, lvm.c, lzio.c, lbitlib.c purpose: links into the game executable to interpret lua level description files, lua dungeon description files, and a lua quest text file. d) recover (optional if desired/required; some targets have recover functionality built into NetHack itself) c) Additional optional library packages/obj files as required lib/pdcurses/... or lib/pdcursesmod/... from sources: addch.c, addchstr.c, addstr.c, attr.c, beep.c, bkgd.c, border.c, clear.c, color.c, delch.c, deleteln.c, getch.c, getstr.c, getyx.c, inch.c, inchstr.c, initscr.c, inopts.c, insch.c, insstr.c, instr.c, kernel.c, keyname.c, mouse.c, move.c, outopts.c, overlay.c, pad.c, panel.c, printw.c, refresh.c, scanw.c, scr_dump.c, scroll.c, slk.c, termattr.c, touch.c, util.c, window.c, debug.c, pdcclip.c, pdcdisp.c, pdcgetsc.c, pdckbd.c, pdcutil.c purpose: underlying curses platform support for some target platforms where inclusion of the NetHack curses window port in win/curses is desired +-------------------------+ | B4. Case sample: msdos | +-------------------------+ Cross-compiler used: Andrew Wu's djgpp cross-compiler Cross-compiler url: https://github.com/andrewwutw/build-djgpp Cross-compiler pre-built binary downloads: https://github.com/andrewwutw/build-djgpp/releases Cross-compiler bits tested: https://github.com/andrewwutw/build-djgpp and the pre-built binary for your platform from: https://github.com/andrewwutw/build-djgpp/releases and a DOS-extender (for including in msdos packaging) from http://sandmann.dotster.com/cwsdpmi/csdpmi7b.zip and pdcurses from: https://github.com/wmcbrine/PDCurses.git or pdcursesmod from: https://github.com/Bill-Gray/PDCursesMod.git - A shell script to download that djgpp cross-compiler and associated pieces for either linux or macOS is available: sh sys/msdos/fetch-cross-compiler.sh That script won't install anything, it just does file fetches and stores them in subfolders of lib. The linux.370 and macOS.370 hints files are configured to find the cross-compiler there if you add CROSS_TO_MSDOS=1 on your make command line. Note: Both the fetch-cross-compiler.sh script and and the msdos cross-compile and package procedures require unzip and zip to be available on your host build system. On your linux host: cd sys/unix ; sh setup.sh hints/linux.370 ; cd ../.. make fetch-lua On your macOS host: cd sys/unix ; sh setup.sh hints/macOS.370 ; cd ../.. make fetch-lua The MSDOS cross-compile can then be carried out by specifying CROSS_TO_MSDOS=1 on the make command line: make CROSS_TO_MSDOS=1 all make CROSS_TO_MSDOS=1 package You can explicitly include tty and curses support if desired. The default you'll end up with is a tty-only cross-compile build: make WANT_WIN_TTY=1 WANT_WIN_CURSES=1 CROSS_TO_MSDOS=1 package Result: The "make package" target will bundle all of the necessary components to run NetHack on msdos into a folder: targets/msdos/pkg and then it zips the contents of that folder into: targets/msdos/nh370dos.zip Also note that building the msdos targets using the make command above, does not preclude you from building local linux or macOS targets as well. Just drop the CROSS_TO_MSDOS=1 from the make command line. That's because the cross-compiler hints additions are enclosed inside ifdef sections and won't interfere with the non-cross-compile build in that case. +-------------------------+ | B5. Case sample: amiga | +-------------------------+ Disclaimer: This is a minimal recipe, just to help someone else get started if they have a desire to get a full cross-compile of NetHack going for the Amiga. See CAVEATS below. Cross-compiler used: bebbo's amiga-gcc Cross-compiler url: https://github.com/bebbo/amiga-gcc To our knowledge, a pre-built copy of the cross-compiler isn't available, so you will likely have to obtain the cross-compiler sources via git and build it on your system. The build prerequisite packages for building the compiler on Ubuntu can be easily obtained: sudo apt install make wget git gcc g++ lhasa libgmp-dev \ libmpfr-dev libmpc-dev flex bison gettext texinfo ncurses-dev \ autoconf rsync The build prerequisite packages for macOS via homebrew are documented but not tested by us any of us to date. brew install bash wget make lhasa gmp mpfr libmpc flex gettext \ texinfo gcc make autoconf After installing the prerequite packages and the cross-compiler it was a straightforward build: git clone https://github.com/bebbo/amiga-gcc.git cd amiga-gcc make update [Note that you may have to take ownership of the files in the bebbo repo via chown before succesfully carrying out the next steps] make clean make clean-prefix date; make all -j3 >&b.log; date The compiler pieces are installed in /opt/amiga by default. If you prefer, you can alter the prefix before you build if you want. The instructions for doing so were spelled out at the time of this writing at: https://github.com/bebbo/amiga-gcc On your linux host: cd sys/unix ; sh setup.sh hints/linux.370 ; cd ../.. make fetch-lua On your macOS host: cd sys/unix ; sh setup.sh hints/macOS.370 ; cd ../.. make fetch-lua The Amiga cross-compile can then be carried out by specifying CROSS_TO_AMIGA=1 on the make command line: make CROSS_TO_AMIGA=1 all make CROSS_TO_AMIGA=1 package You can explicitly include tty and curses support if desired, otherwise you'll end up with a tty-only cross-compile build. The SDL1 pdcurses support has not been tested. make WANT_WIN_TTY=1 WANT_WIN_CURSES=1 CROSS_TO_AMIGA=1 all Result: The "make package" target will bundle the (hopefully) necessary components to run NetHack on msdos into a folder: targets/amiga/pkg and then it zips the contents of that folder into: targets/amiga/nh370ami.zip Also note that building the amiga targets using the make command above, does not preclude you from building local linux or macOS targets as well. Just drop the CROSS_TO_AMIGA=1 from the make command line. The cross-compiler hints additions are enclosed inside ifdef sections and won't interfere with the non-cross-compile build in that case. CAVEATS: The original NetHack Amiga build steps included the source for some utilities that were built and executed on the amiga: txt2iff and xpm2iff as part of the NetHack build procedure on amiga. Those did not compile out-of-the-box on the linux host. They will either have to be: - ported to build and run on the linux or macOS cross-compile host or - their functionality will have to be rolled into amiga NetHack itself and executed on the target Amiga the first time the game is run, perhaps. If you make headway, or are successful getting a working copy of NetHack going on the amiga, drop us a note at devteam@nethack.org. +--------------------------------+ | B6. Case sample: Web Assembly | +--------------------------------+ Credit: The initial Web Assembly cross compile was found in a pull request: https://github.com/NetHack/NetHack/pull/385 by apowers313. The pull request was merged with some accompanying NetHack source tree integration changes in early October 2020. Cross-compiler used: emscripten Cross-compiler url: https://emscripten.org/docs/getting_started/downloads.html Here's a brief guide to obtaining the cross-compiler sources via git and building it on your system. For Ubuntu, the build prerequisite packages for building the compiler can be easily obtained: sudo apt-get install python3 cmake default-jre For macOS, you will need to install Xcode, git, cmake, Python 3.5 or new (at time of this writing). After installing the prerequite packages above, obtain the cross-compiler via git and build it from the directory of your choice using steps similar to these: git clone https://github.com/emscripten-core/emsdk.git cd emsdk git pull ./emsdk install latest ./emsdk activate latest source ./emsdk_env.sh The steps above reflect what was outlined at this url at the time of writing: https://emscripten.org/docs/getting_started/downloads.html That is the definitive source and trumps anything documented here. On your linux host, prepare to cross-compile NetHack as follows: cd sys/unix ; sh setup.sh hints/linux.370 ; cd ../.. make fetch-lua On your macOS host, prepare to cross-compile NetHack as follows: cd sys/unix ; sh setup.sh hints/macOS.370 ; cd ../.. make fetch-lua Then, cross-compile to targets/wasm as follows: make CROSS_TO_WASM=1 You can build src/libnh.a from pull request 385 as follows: make WANT_LIBNH=1 Do not add any additional windowport interfaces to your build (such as WANT_WIN_TTY=1 WANT_WIN_CURSES=1 WANT_WIN_X11=1 or WANT_WIN_QT=1) as those aren't applicable to the Web Assembly or nethacklib builds. A "shim" pseudo-windowport is included from pull request 385. Result: As mentioned, the wasm cross-compile will end up in targets/wasm and the nethacklib.a will end up in src. The cross-compiler hints additions are enclosed inside ifdef sections and shouldn't interfere with the non-cross-compile builds using hints/linux.370 or hints/macOS.370. ---