CadEditor. History of the project
Repository
https://github.com/spiiin/CadEditor
CadEditor is the universal level editor for NES/SMD/SNES/Game Boy games, the introduction of the project was here
Introduction
This is the article about the history of the project. CadEditor is my pet project started in 2012 as the simple level editor for one game, but now it has powerful config script system, that allows it to work with a large number of games (~200 games supported now). The article has a lot of programming terms, but I tried to write it as simply as possible
Post Body
0. My romhacking experience
I got into game research almost immediately after I discovered the opportunity to emulate old console games with the computer. I skipped period when emulators were developed, and popular games were dumped to ROM-files - they were already available on the internet.
At first, I was interested in secrets in games - unused content stored inside ROM-files, codes for a switch to debug mode for testing, password systems, little-known for gamers bonuses and secret places. Most of this material already was found by authors of dumps and romhackers. But after some research, I came across with that not all have been researched yet.
There are some of my first researches (articles in Russian):
Tecmo World Cup password system
Goal! Two password system
Adventure Island 2 random level select system
Battletoads regional version differences
Addams Family doors system
My first real interesting discovery was hidden prizes in New Ghostbusters 2 [NES]. It can be received only if the player will catch the ghost in a certain order. I found it when research code related to enemies behavior. I found the item ("bag with money"), that contains a property with an array of other items. That was a chain of ghost types needed to catch, for an appearance of the bag with money. There is a video with collect all bags:
One of my ideas of game research was building maps of game levels by binary data stored in ROM-file. It was a hard task for me, and I spent several weeks, but I achieved my goal and I was able to get maps for Jungle Book [NES]. But during the research, I understand that I can repeat this process with other games.
I was mostly interested in building level map only, but I this time, I met romhacker Lomax who created level hacks for classic Capcom NES games. He used hex-editor for changing levels. I wrote to him, that I can create a visual editor for changing levels of Chip & Dale Rescue Rangers[NES] (by that moment I already partially researched this game), and he interested in this idea.
Thus, I started working on a prototype CadEditor, in order to give the opportunity to change levels in the game Chip & Dale for him.
1. Chip and Dale and Darkwing Duck Editor
I distributed the first versions of the program through the Russian forum "emu-land", and received many feedbacks with proposals for improvements and found bugs. I gradually added the necessary features - block editor, build graphics from binary data as console PPU does it, object editor and level parameters editor. At the same time, I discovered that Darkwing Duck game [NES] used almost same level format, and even many data are stored at same offsets, so I add this game in the editor too.
At this point, users of editor created hacks for these two games:
2. Capcom games editor
I was interested: if Chip and Dale and Darkwing Duck games so similar, what about other Capcom games?
I researched level format for Duck Tales 1&2, Tale Spin, and Little Mermaid and found out that they are fairly similar also. But, all of them (except Tale Spin – it was identical to Darkwing Duck), has minor differences, which prevent to add it to the CadEditor.
I was thinking about how to add these games to the editor so as to be able to expand it to new games. Obviously, it was necessary to put the data addresses in the config file. But what about different ways to load data, when the format was unknown yet?
One of the possible solution - to embed script language interpreter to the editor, so config files for new games can be written in this language.
I wrote CadEditor in C#, and chose CSScript library for scripting. This library allows writing a script in C# also. It not a maximum short or simple language, but I can move code from editor to config scripts and back without any changes. Of course, first versions of my script system not allowed to describe any games, and part of loading logic was hardcoded inside CadEditor itself.
But using this approach, I added all games listed above to CadEditor:
3. Games editor with prepared block images
I tried to explore the games from other developers and figure out how the levels are described in them. One of the general formulations was - almost all games use blocks (described in various ways) and an array with a game map that contains the indexes of these blocks. This approach could be transferred to the editor - if I do not understand how blocks were built, but simply prepare screenshots of each block at the level, I still can display a map from prepared pictures and allow to change it. In addition, this approach allowed porting game configs from other level editors.
During editor updates I prepared images of blocks from several games from different developers - Jungle Book, Tiny Toon Adventures, Tom & Jerry, Battletoads.
After that, I had the idea of creating an editor that could work with ANY game using configs. For this, it was necessary to study the format of the levels of a large number of games, without understanding into the detailed structure of each. Next versions of the editor were added new descriptive features to the editor.
4. Universal block editor
I gradually added to the editor the ability to describe various block and screen formats. In versions 1.9-1.9.5, I added the ability to describe games in which screens are not described line by line but described per columns. In addition, the I added ability to use not only square blocks but also rectangular. This allowed for version 2.0 to add 32 games to the editor (one level from each for the tests).
Next, I discovered, that with CadEditor using prepared images of blocks, I can display game levels for the Sega Mega Drive platform and add several levels for it (Tiny Toon, Contra Hard Corps, QuackShot). Further, the development of the editor proceeded through research and the addition of new games, most of which did not require any additional efforts — all that was needed was to prepare block images.
However, it was a routine work that required a lot of time, so by this time I had written several scripts to automate actions. I called the project Autocorrupter – a script that allows you to get pictures of each block on the level. This script has also evolved and improved over time. This required writing the code and making changes to the NES Fceux emulator itself (changes are currently included in the latest version of the emulator).
At this point, the interest of users to the editor faded a little, because I was completely absorbed in adding and researching new games and was not engaged in expanding support for already explored games. However, some users shared my ideas and helped me. To version 3.0, the hacker with the nickname Mephisto took the source code of my editor and ported it to create the editor for Thunder Force 3 for the game Sega Mega Drive. He did not use the editor with pictures but wrote code for building blocks for Sega games. I ported his code back to my editor and this gave me the opportunity to describe the blocks for Sega Mega Drive games.
5. Plugin system
The editor at this point was overgrown with a huge amount of code (partially obsolete - written for specific games, and not for a generic editor), and I realized that the project had become so complex that I could not control it soon.
I started to think about how to simplify the code and break the project into separate parts.
The games for NES and SMD used a separate code for building graphics and editing blocks (there are some platform-specific differences - so editing happens a little differently). Therefore, I decided to add a plugin system to the editor - the common code used for any game remained in the editor's core, and things specific to the platform and the game were put into separate assemblies (in the form of DLL files), which the editor loaded after the game settings file was opened.
For example, when opening the Chip & Dale [NES] game settings, the editor loaded the plug-ins for building NES graphics, as well as the door parameter editing plug-in specific for this game.
For Contra Hard Corps [SMD] – the editor loaded the plugin for building SMD graphics, and a special background editor, which is present for all SMD games:
At the same time, the editors of screens, objects, and blocks always remained in the editor's core. This allowed to improve the architecture and reduce the amount of code in the main project.
6. Checking of different NES-games.
I also found a large class of games that use the same block format of 17 bytes - 16 tiles and 1 byte of the palette. Later, I realized that this is a natural description for the NES platform. I added support for building such blocks, which allowed us to once again go through the games that were described with pictures of blocks, and rewrite them using this blocks. Here are some examples of configuring games with such blocks.
(in Russian)
Teenage Mutant Ninja Turtles 2
Power Blade 2
Description of this types of blocks
Here is a list with some games, using blocks 16+1:
- Power Blade 1-2 (Natsume)
- Adventure Island 2-3 (Hudson soft)
- Yo-Noid (Now production)
- Jackie Chan Action Kung Fu (Now production)
- Contra Force(Konami)
- Batman Returns (Konami)
- Super C (Konami)
- Monster in My Pocket (Konami)
- Bucky O'Hare (Konami)
- Jackal (Konami)
- Teenage Mutant Ninja Turtles 1-3 (Konami)
- Alien 3 (Probe entertaiment)
- Battletoads (Rare)
- Battletoads & Double Dragon (Rare)
- Darkman (Ocean software)
- Many others
To search for blocks in the ROM file, I came up with an algorithm that allowed the search for standard blocks automatically, analyzing the video memory in the emulator. This has greatly accelerated the search process. This algorithm is implemented in the form of a Lua script called BlockFinder for the Fceux emulator. In a pair of scripts, Autocorrupter and BlockFinder allowed to adding support for new games to the editor very quickly.
With the help of new tools and update of the editor for the description of the blocks, a large number of games have been added. User Lancuster connected to the project, and he added all levels for those games, I created only one test config. He found more than 100 games with formats similar to those already added and described them. The total number of games that support the editing of screens and blocks is close to 200.
Resources
https://spiiin.github.io/CadEditor/
I collected ~60 posts and articles about game research and CadEditor (mostly in Russian).
Patreon
Itch.io
Emu-land forum theme related to CadEditor
Romhacking.net forum theme related to CadEditor
Roadmap
Future
The system of configs and plugins of the CadEditor, as well as tools for searching for blocks in ROM files, allows users to describe almost any game now.
However, in addition to the modules of the editor of screens and blocks, in order to conveniently change the games, it is also necessary to add an object editor. The editor supports a generic description for lists of objects, but most games are still waiting to be added to it.
In addition, most of the research concerned only NES games, although similar search principles can be applied to any platform with tile graphics — Sega Mega Drive, SNES, Game Boy, Game Boy Advance, Sega Master System, old PC games.
An interesting feature is the use of an editor with fully rebuild the game from sources (working with asm files). The plug-in editor system allows you to implement an assembly system using external utilities, including assembler, export, and data compression.
But my main goal, above all, is to preserve in the editor the history and culture of old games, as well as research and documenting the solutions used by developers. That is why I spend my time on this project.
Hello, @pinkwonder, and thank you for another contribution!
I like this series of yours so much. The amount of effort you have put into this project so far is astonishing, and I am happy you took my advice and shared a few words about the project's history.
Your post is comprehensive, rather informative and well-illustrated with videos. Furthermore, you managed to make this post personal by sharing your experience of working on this project, as well as the way you overcame several obstacles in the process of developing CadEditor. I also enjoyed the extensive information regarding all the research that needed to be done for the addition of those features mentioned in your post; you did a great job.
Once again, I spotted a few errors of style and proofreading, along with minor punctuation mistakes. Regardless, this is a very nice post and I look forward to seeing more awesome content from you in the future.
Your contribution has been evaluated according to Utopian policies and guidelines, as well as a predefined set of questions pertaining to the category.
To view those questions and the relevant answers related to your post, click here.
Need help? Write a ticket on https://support.utopian.io/.
Chat with us on Discord.
[utopian-moderator]
Thank you for your review, @lordneroo! Keep up the good work!
Thanks, you give a very good hint for the theme of this post ^_^
Thank you for sharing you game development post with us! You have received an upvote! Please visit our page @steemgg to learn more about how you can use our platform Steemgg, the first html5 gaming platform built on the Steem Blockchain to get more out of game development.
Vote for bobdos Witness
I already saw steemgg, it looks pretty cool. I'll explore games on it, and thin if I can use it for my projects.
Hi @pinkwonder!
Your post was upvoted by @steem-ua, new Steem dApp, using UserAuthority for algorithmic post curation!
Your post is eligible for our upvote, thanks to our collaboration with @utopian-io!
Feel free to join our @steem-ua Discord server
Hey, @pinkwonder!
Thanks for contributing on Utopian.
We’re already looking forward to your next contribution!
Get higher incentives and support Utopian.io!
Simply set @utopian.pay as a 5% (or higher) payout beneficiary on your contribution post (via SteemPlus or Steeditor).
Want to chat? Join us on Discord https://discord.gg/h52nFrV.
Vote for Utopian Witness!
Congratulations! This post has been upvoted from the communal account, @minnowsupport, by spiiin from the Minnow Support Project. It's a witness project run by aggroed, ausbitbank, teamsteem, someguy123, neoxian, followbtcnews, and netuoso. The goal is to help Steemit grow by supporting Minnows. Please find us at the Peace, Abundance, and Liberty Network (PALnet) Discord Channel. It's a completely public and open space to all members of the Steemit community who voluntarily choose to be there.
If you would like to delegate to the Minnow Support Project you can do so by clicking on the following links: 50SP, 100SP, 250SP, 500SP, 1000SP, 5000SP.
Be sure to leave at least 50SP undelegated on your account.
Congratulations,
you just received a 11.25% upvote from @steemhq - Community Bot!
Wanna join and receive free upvotes yourself?
Vote for
steemhq.witness
on Steemit or directly on SteemConnect and join the Community Witness.This service was brought to you by SteemHQ.com