The Atari 7800 (American version only) used an encryption algorithm which was used to determine which mode to run in: 2600 mode or 7800 mode. I've been able to collect quite a bit of information regarding this encryption algorithm (but not the actual encryption key - this is unknown).

This encryption algorithm is NOT a problem in terms of creating an emulator for two reasons. First, the actual program code and data is NOT the part that's been encrypted. In fact, only 960 bits are actually encrypted and their only purpose was authenticating whether or not the cartridge was an officially licensed 7800 game. Any ROM (at least the American versions) will already have the correct encryption signature. Second, the emulator can even be designed so that it simply bypasses this step. - Jim Pragit

"Did some reading of old stuff, and found a little about the encryption.  In NTSC 7800s and games, there is a 960 bit signature at the beginning of the game - it is being processed during the Atari logo startup screen.  If you don't have the signature, you get stuck in 2600 mode.  Interestingly, PAL systems and games (they're incompatible) do not have that startup screen, and thus I assume they don't have the signature. What they do to kick into 7800 mode then, I haven't a clue."

"What this means in practice is that every cartridge had a checksum which allowed the 7800 console to confirm that the cartridge was a genuine, Atari-authorized piece of software.  It was perceived that Atari had suffered financially from the release of some very low quality 2600 software towards the end of that system's life, and by authenticating cartridges, they could control the quality of the software available.  The Atari Lynx did exactly the same thing, BTW.

"If someone wanted to write an emulator, this authentication scheme is a non-issue.  Just leave it in there: it will still authenticate cartridges.  If you wish to write your own software, patch the boot ROM.

"I repeat what I said in the original post: there is NO ENCRYPTION OF THE BULK DATA.

"Each cartridge has the unencrypted program data, and somewhere in the cartridge, there is also a checksum which is calculated using a special mathematical function known as a trapdoor function. These functions are cryptographic in nature (specifically hashes and public key functions), but it is not correct to say that the cartridge is encrypted itself.  Only a tiny little checksum is.

"So what is a trapdoor function?  It is a mathematical operation which is trivially easy to calculate, but hideously difficult to invert (i.e.. like a trapdoor - it's easy to go one way, but difficult to come back the other).  Putting this mathematically:

 f() is our trapdoor function, it is easy to calculate.
 Let's assume that f-1() is the inverse, such that:

  x = f(y)
  y = f-1(x)

 f-1() will be hideously difficult to calculate.

"There are a number of trapdoor functions available.  Cryptographic hashes (MD5, SHA-1 etc.) are examples of these.

"However, there are classes of functions which are trapdoor functions which have specific properties, in that they are reversible, if you know certain things.  This is more commonly known as public key cryptography.

"Let me illustrate this mathematically:

"Our encryption function is called e(), our decryption function is called d().

 x = e(y, p1)
 y = d(x, p2)

 x = ciphertext (encrypted text)
 y = plaintext (unencrypted text)
 p1 = public key
 p2 = private key

"So, IF AND ONLY IF I possess key p1 (the public key), I can encrypt plaintext into ciphertext.  IF AND ONLY IF I have key p2 (the private key), I can decrypt ciphertext into plaintext.

"Note: in theory the functions could be keyless.  This is most unlikely in practice.

"Fine, so how does this work?

"I have a load of program code, and I want to ensure three things:

1. That the program code has not been modified in some way.   For example, I might wish to ensure that my Atari 7800 version of Tomb Raider :)  has Lara wearing her clothes, as a Nude Raider version might sell well, but would also reflect poorly on the console producing company. A real world example of this was Custer's Revenge, which cost Atari a *lot* of bad press, although they didn't write or authorize the game.
2. I want to ensure that any company producing games for my console actually has a legal agreement to pay me royalties.
3. That I enforce regional controls.  For example, I might wish to ensure that games sold in Japan won't work in a US system, and vise versa.

"The company then gives me a very royalty big check, which I cash in my bank account, then give them the security header.  They can then go off and make thousands of cartridges.

"When a use plugs a cartridge into the console, the boot ROM runs, and it pulls the security header out of the cartridge, and runs it through the decryption function using the "public" key.

"Important point: the "private" key is NOT IN THE CONSOLE. The encryption function is not in the console.  Only the "public" key and "decryption" function are in the console. Remember that mathematically, it is very difficult (read computationally impossible) to derive the encryption function and/or private key  from the decryption function and/or public key.  People familiar with cryptography will understand that I am generalizing a little here, but we can discuss that offline.

"The console then has the unencrypted hash of the original text.  It then runs through the actual data on the cartridge and calculates it's own hash, then compares the two hashes.  If the match, then the cartridge is allowed to run.

"So how does this help:

1. If the cartridge program has been modified, the hashes won't match, and the cartridge won't run.
2. If the company doesn't pay me royalties, they cannot produce the security header.
3. I would vary my public and private keys from region to region, so if people try importing cartridges, they have different keys, and won't run.

1. If cartridge images are made, the security header gets copied too, as will the boot ROM containing the checking code. As such, the security system will function perfectly for Atari authorized cartridges.
2. If you wish to produce your own cart's, simple patch the checking routines out of the boot ROM.  If memory serves - and I might be wrong here - the 7800's boot ROM is only 2K in size.  That's an afternoon's work.