december adventure 2023
the goal is to work on something small every day in december.
my goal: get unix extensions to uxn's console device working with uxn11
in particular here are some of the things i plan to work on:
* making sure the console.txt spec is clear and complete
* getting uxn11 (and uxncli) correctly implementing the spec
* writing demo and test programs
* updating determ (term.tal) to use the console device
* updating femto to use the console device
* implement uxntal bindings for some useful tools:
- ispell
- netcat / stunnel
- links / hget
- mpv / mpg123
my uxn11 branch is here
≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡
december 2023 calendar
day 1 day 9 day 17 day 25
day 2 day 10 day 18 day 26
day 3 day 11 day 19 day 27
day 4 day 12 day 20 day 28
day 5 day 13 day 21 day 29
day 6 day 14 day 22 day 30
day 7 day 15 day 23 day 31
day 8 day 16 day 24
≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡
dec 1: fixing bugs
i fixed some bugs preventing the make demo from running correctly.
additionally i started looking at fleshing out the ispell demo.
dec 2: more bugs and a spec change
i spent a couple of hours hitting some issues with the ispell demo.
this is the first test that actually reads/writes data so i expected some
problems. it turns out there were numerous issues:
- delayed initialization was losing process information
- file descriptor issues
- writing to the wrong port (0x2 instead of 0x4)
one thing i realized is that everything should use the stdin port (0x2).
proc-get (0x4) and host-get (0x5) don't really have a point, since they
all happen on the same vector (0x2) and the type port (0x7) already makes
it clear what type of data we have.
so i'm going to update the spec so that all reads are from stdin (0x2).
that will simplify the console.c code, reduce user errors, and
also simplify porting determ a bit.
(but all those updates will have to wait until tomorrow.)
dec 3: ispell is working!
after finding even more bugs (shocking, i know) i have the ispell demo
completely working:
ispell.tal
the basic idea is that it reads a line from stdin and passes it to ispell.
for example:
$ uxncli ispell.rom
>hello
hello 1
>helo
helo 0
>
note that is is demo quality code. if you type a bunch of words on one line
it may behave weirdly (e.g. messed up formatting). but it does show off
bi-directional pipes from within a uxn process. hurrah!
i also combined the stdin, proc-get, and host-get ports as i mentioned,
and updated the spec:
console.txt
anyway, that's an end to my adventures for the day. be well!
dec 4: snow day
i was pretty tired so i didn't do too much.
dec 5: femto
i ported the femto editor to the subprocess code. even though it runs
using uxncli it benefits from the changes. in particular, we previously
used a wrapper to run stty raw -echo to put the terminal in raw mode
before launching the ROM. now we can make the same call as a child process
which removes the need to use any wrapper.
(i also fixed a small femto bug.)
one of the goals of using subprocesses from uxn is to eliminate the need
for external wrappers and launchers. now that femto is working, the next
step will be porting determ and getting that to run in a stand-alone
way (as well as features like using the SHELL environment variable).
beyond terminal emulation, this would also make it possible for programs
to do various kinds of networking without needing an external wrapper
or driver program. (they would still need some kind of program such as
netcat but they could launch it themselves.)
dec 6: determ
the determ terminal emulator (term.tal) is working! this ROM provided
the biggest impetus for implementing subprocess support, since it requires
a lot of things that varvara can't usually do:
- allocate a PTY
- manage terminal attributes
- fork a child process
- set environment variables
most of the real work was porting the ROM to use Console/proc-get instead
of Console/stdin. after spending awhile tracking down one last bug we
got the terminal launching!
one thing i noticed yesterday is that we could probably compress the range
of Console/type values we are using. currently we use up to 0x83
but it would be easy to reduce that to 0x1b for example.
another thing i need to do is decide whether i need to add any host actions
to set more terminal attributes. it's possible that running stty will
be enough but i'm not totally sure -- it might be more straightforward to
do it in code; for one thing it would make it easier for the terminal to be
resized dynamically. we'll see.
beyond that i think we're ready to start making a patch for upstream uxn11!
be well!
dec 7: bug hunt
for awhile now there have been some highlighting bugs in determ when you
use the pointer to select text. i'm not sure exactly when they started, but
they were exacerbated when the blending modes were changed.
after looking into it a bit more, i'm pretty sure i see what is going on:
- everything is drawn to the fg layer (contradicting my memory)
- clearing and redrawing the pointer wipes out overlapping selections
- there's a 4 pixel inaccuracy with some addressing.
at one point i was experimenting with "backgrounds" for determ and i think
maybe i inadvertently changed how drawing works and forgot to revert. that
would explain why the mouse expects to be able to safely clear and redraw.
i think i will probably try to go back to doing most drawing on the bg layer
and reserve the fg for pointers, menus, etc.
dec 8: rest day
other than giving a bit of advice on the #DecemberAdventure hashtag
i didn't to any adventuring today.
dec 9: lots of small fixes
i spend a bit of time fixing some quality of life issues:
- added -2x and -3x arguments to uxn11 to start zoomed in
- fixed the selection but that had been plaguing determ
- added code in uxn11 to update width/height when windows resize
that last one would allow uxn ROMs to notice that the screen has changed
sizes by reading Screen/width and Screen/height and comparing them to
previous values. i got the idea after accidentally resizing determ and
wishing it actually "worked" properly.
(update: the resize detection code has some problems. it seems to break
rescaling. i'm not too worried -- i think i'm just using Xlib incorrectly.
it should be easy to fix later.)
i need to finalize the console spec and send neauoire a patch but i'm
waffling. Console/type uses 256 distinct sentinel values and the spec is
adding a bunch of new ranges:
- 0x05 - 0x07 (host response, host response end, stdin end)
- 0x20 - 0x23 (child N sent data)
- 0x40 - 0x43 (child N data end)
- 0x80 - 0x83 (child N exited)
there are some convenient aspects to this but arguably just compressing
the last ranges down into 0x10 - 0x1c would be a bit kinder to future
extensions. if i want to change it then now is the time to do it.
i'm just feeling indecisive.
i wanted to produce a cool demo video but it turns out that currently
femto can't run within determ. i think the issue is that the pty that
determ runs in is having trouble being put into "raw" mode by femto.
dec 10: another break
nothing significant to report today.
dec 11: mpg123 detour
i got interested in the idea of writing an mp3 player for varvara, and
started investigating options. it turns out that mpg123 has a really
nice generic command interface using stdin and stdout.
basically you enter a command (terminated by a newline), and it emits
one or more lines, each of which starts with @. here's an example
session from running mpg123 -R. input to the process starts with >
and output from the process starts with <.
< @R MPG123 (ThOr) v10 ; startup banner
> L spectra.mp3 ; load command
< @I { ; start id3
< @I ID3v2.title:Spectra ; song title
< @I ID3v2.artist:Chipzel ; artist name
< @I ID3v2.album:Spectra ; album title
< @I ID3v2.year:2013 ; song year
< @I ... ; other info possible
< @I } ; end id3
< @P 2 ; song playing
< @S 1.0 3 44100 Joint-Stereo 2 1044 2 0 0 0 320 0 1 ; sound info
< @F 0 9052 0.00 236.46 ; first frame output
< @F ... ; lots of frames of output
< @F 9051 1 236.43 0.03 ; last frame output
< @P 3 ; song ended
< @P 0 ; stopped
i got far enough to convince myself this was possible and it would work.
that said it's a bit of a distraction so it's probably something i will
be coming back to later.
dec 12: no idea?
if i did anything on this day i have no recollection of it.
dec 13: preparing a patch against uxn11
i took some time to create a draft patch against the mainline uxn11.
looking at the diff as a whole exposed some things that needed a bit
more clean up or documentation. that said, i was overall happy with
how the patch looked:
5 paths 17 groups, +204 lines (+353 -149)
(this doesn't include the design doc, demo programs, or my own ROMs.)
dec 14: it's looking ready
this was a big one!
i did most of the clean up tasks i noticed the previous day, and also
fixed a bunch of bugs across console.c, term.tal, and femto.tal:
- getting and setting environment variables was broken
- calling stty from subshells was not reliable
- killing a non-running subprocess could have strange effects
- determ was failing to respond to DSR queries correctly
- femto was not able to run inside determ
- femto was not exiting cleanly
at this point running determ and femto via uxn11 and uxncli
is just as good (or better) than using the old wrapper scripts.
obligatory screenshots and video:
uxn11-15.mp4 (9.5 MB, 1:28, has music)
(it's worth noting that there are some graphical glitches in the video.
when i started determ i aimed to just implement TERM=ansi, but it turns
out many programs aren't 100% compatible with "pure" ansi terminals.
so i pivoted to quasi-vt100 compatibility but am still missing some
fairly critical vt-100 features.)
dec 15: draft patch for uxn11
all my uxn11 changes in one place.
mostly i wanted to see all the changes in one place and get a sense
of how reasonable (or unreasonable) it might look. i was pretty
pleased with the result!
dec 17: took saturday off
wasn't feeling motivated, took a break.
dec 17: also took sunday off
still wasn't feeling motivated. continued my break.
dec 18: uuid-based extensions for uxn11
one problem with using an enhanced console only available in uxn11
is that it is incompatible with the current approach of using stdin
and stdout with wrapper scripts. this means i will either need to
drop support for uxnemu or else maintain two versions of my ROMs.
one idea i had was to include support for both approaches, and then
choose at runtime based on the emulator version. however, there is
currently not a good way for ROMs to figure this out.
while talking with neauoire about this they reminded me of the
extensions proposed by zzo38. these use the expansion port to
send and receive information about optional device features.
perfect!
currently only the uxn38 emulator supports UUID-based extensions,
but adding them to uxn11 ended up being pretty easy. i have also
updated the specification to describe their use with the console.
dec 19: another pause
took a bit of a break.
dec 20: still paused
still was not feeling adventurous.
dec 21: uxntal support for linguist
one weird thing about respositories with uxntal code is that sites
like github/codeberg/etc. don't know about the language, so the
language summary ends up listing other languages like shell, lua,
python, etc. instead.
i finally got tired of this state of affairs and submitted a PR to
the linguist project to add uxntal support. this mostly involved
reading contributor guidelines, looking at other PRs, and installing
a bunch of linguist dependencies.
linguist uses textmate grammars for syntax highlighting but
fortunately there's already an implementation for uxn.
anyway, the pull request is here.
dec 22: pr updates
took care of some minor issues noted on the PR.
i expect nothing will be merged before 2024 but i'm hopeful that
soon we'll have uxntal support in a linguist release.
dec 23: top-secret project
i paused my adventure to work on a top secret project.
dec 24: top-secret project (continued)
the top secret project continued to take all my time.
dec 25: top-secret project revealed
without sharing any actual details about it, the top-secret project
was related to a christmas gift for kate.
dec 26: post-holiday break
after all the chaos of the holidays it was nice to take a break.
dec 27: break
still taking a break (i was hoping to restore some energy for future
work but (spoiler alert) i ended up getting sick and not
doing anything else for the rest of the month.
dec 28: illness
i got sick today.
dec 29: very sick
even more sick today.
dec 30: still very sick
still very sick today.
dec 31: starting to recover
starting to feel better, but still sick.
so in conclusion: my adventure ended in an incomplete state,
partially due to holiday chaos and partially due to illness.
still, i had a good time and did get a lot done! thanks, eli!