december adventure 2023

  the goal is to work on something small every day in december.

  my goal: get unix extensions to uxn's console device working with uxn11

  in particular here are some of the things i plan to work on:

    * making sure the console.txt spec is clear and complete
    * getting uxn11 (and uxncli) correctly implementing the spec
    * writing demo and test programs
    * updating determ (term.tal) to use the console device
    * updating femto to use the console device
    * implement uxntal bindings for some useful tools:
      - ispell
      - netcat / stunnel
      - links / hget
      - mpv / mpg123

  my uxn11 branch is here


  december 2023 calendar

  day  1      day  9      day  17      day  25
  day  2      day 10      day  18      day  26
  day  3      day 11      day  19      day  27
  day  4      day 12      day  20      day  28
  day  5      day 13      day  21      day  29
  day  6      day 14      day  22      day  30
  day  7      day 15      day  23      day  31
  day  8      day 16      day  24

dec 1: fixing bugs

  i fixed some bugs preventing the make demo from running correctly.
  additionally i started looking at fleshing out the ispell demo.

dec 2: more bugs and a spec change

  i spent a couple of hours hitting some issues with the ispell demo.
  this is the first test that actually reads/writes data so i expected some
  problems. it turns out there were numerous issues:

    - delayed initialization was losing process information
    - file descriptor issues
    - writing to the wrong port (0x2 instead of 0x4)

  one thing i realized is that everything should use the stdin port (0x2).
  proc-get (0x4) and host-get (0x5) don't really have a point, since they
  all happen on the same vector (0x2) and the type port (0x7) already makes
  it clear what type of data we have.

  so i'm going to update the spec so that all reads are from stdin (0x2).
  that will simplify the console.c code, reduce user errors, and
  also simplify porting determ a bit.

  (but all those updates will have to wait until tomorrow.)

dec 3: ispell is working!

  after finding even more bugs (shocking, i know) i have the ispell demo
  completely working:


  the basic idea is that it reads a line from stdin and passes it to ispell.
  for example:

      $ uxncli ispell.rom
      hello 1
      helo 0

  note that is is demo quality code. if you type a bunch of words on one line
  it may behave weirdly (e.g. messed up formatting). but it does show off
  bi-directional pipes from within a uxn process. hurrah!

  i also combined the stdin, proc-get, and host-get ports as i mentioned,
  and updated the spec:  

  anyway, that's an end to my adventures for the day. be well!

dec 4: snow day

  i was pretty tired so i didn't do too much.

dec 5: femto

  i ported the femto editor to the subprocess code. even though it runs
  using uxncli it benefits from the changes. in particular, we previously
  used a wrapper to run stty raw -echo to put the terminal in raw mode
  before launching the ROM. now we can make the same call as a child process
  which removes the need to use any wrapper.

  (i also fixed a small femto bug.)


  one of the goals of using subprocesses from uxn is to eliminate the need
  for external wrappers and launchers. now that femto is working, the next
  step will be porting determ and getting that to run in a stand-alone
  way (as well as features like using the SHELL environment variable).

  beyond terminal emulation, this would also make it possible for programs
  to do various kinds of networking without needing an external wrapper
  or driver program. (they would still need some kind of program such as
  netcat but they could launch it themselves.)

dec 6: determ

  the determ terminal emulator (term.tal) is working! this ROM provided
  the biggest impetus for implementing subprocess support, since it requires
  a lot of things that varvara can't usually do:

    - allocate a PTY
    - manage terminal attributes
    - fork a child process
    - set environment variables

  most of the real work was porting the ROM to use Console/proc-get instead
  of Console/stdin. after spending awhile tracking down one last bug we
  got the terminal launching!


  one thing i noticed yesterday is that we could probably compress the range
  of Console/type values we are using. currently we use up to 0x83
  but it would be easy to reduce that to 0x1b for example.

  another thing i need to do is decide whether i need to add any host actions
  to set more terminal attributes. it's possible that running stty will
  be enough but i'm not totally sure -- it might be more straightforward to
  do it in code; for one thing it would make it easier for the terminal to be
  resized dynamically. we'll see.

  beyond that i think we're ready to start making a patch for upstream uxn11!

  be well!

dec 7: bug hunt

  for awhile now there have been some highlighting bugs in determ when you
  use the pointer to select text. i'm not sure exactly when they started, but
  they were exacerbated when the blending modes were changed.

  after looking into it a bit more, i'm pretty sure i see what is going on:

    - everything is drawn to the fg layer (contradicting my memory)
    - clearing and redrawing the pointer wipes out overlapping selections
    - there's a 4 pixel inaccuracy with some addressing.

  at one point i was experimenting with "backgrounds" for determ and i think
  maybe i inadvertently changed how drawing works and forgot to revert. that
  would explain why the mouse expects to be able to safely clear and redraw.

  i think i will probably try to go back to doing most drawing on the bg layer
  and reserve the fg for pointers, menus, etc.

dec 8: rest day

  other than giving a bit of advice on the #DecemberAdventure hashtag
  i didn't to any adventuring today.

dec 9: lots of small fixes

  i spend a bit of time fixing some quality of life issues:

    - added -2x and -3x arguments to uxn11 to start zoomed in
    - fixed the selection but that had been plaguing determ
    - added code in uxn11 to update width/height when windows resize

  that last one would allow uxn ROMs to notice that the screen has changed
  sizes by reading Screen/width and Screen/height and comparing them to
  previous values. i got the idea after accidentally resizing determ and
  wishing it actually "worked" properly.

  (update: the resize detection code has some problems. it seems to break
  rescaling. i'm not too worried -- i think i'm just using Xlib incorrectly.
  it should be easy to fix later.)
  i need to finalize the console spec and send neauoire a patch but i'm
  waffling. Console/type uses 256 distinct sentinel values and the spec is
  adding a bunch of new ranges:

    - 0x05 - 0x07  (host response, host response end, stdin end)
    - 0x20 - 0x23  (child N sent data)
    - 0x40 - 0x43  (child N data end)
    - 0x80 - 0x83  (child N exited)

  there are some convenient aspects to this but arguably just compressing
  the last ranges down into 0x10 - 0x1c would be a bit kinder to future
  extensions. if i want to change it then now is the time to do it.
  i'm just feeling indecisive.

  i wanted to produce a cool demo video but it turns out that currently
  femto can't run within determ. i think the issue is that the pty that
  determ runs in is having trouble being put into "raw" mode by femto.

dec 10: another break

  nothing significant to report today.
dec 11: mpg123 detour

  i got interested in the idea of writing an mp3 player for varvara, and
  started investigating options. it turns out that mpg123 has a really
  nice generic command interface using stdin and stdout.

  basically you enter a command (terminated by a newline), and it emits
  one or more lines, each of which starts with @. here's an example
  session from running mpg123 -R. input to the process starts with >
  and output from the process starts with <.

    < @R MPG123 (ThOr) v10                               ; startup banner
    > L spectra.mp3                                      ; load command
    < @I {                                               ; start id3
    < @I ID3v2.title:Spectra                             ; song title
    < @I ID3v2.artist:Chipzel                            ; artist name
    < @I ID3v2.album:Spectra                             ; album title
    < @I ID3v2.year:2013                                 ; song year
    < @I ...                                             ; other info possible
    < @I }                                               ; end id3
    < @P 2                                               ; song playing
    < @S 1.0 3 44100 Joint-Stereo 2 1044 2 0 0 0 320 0 1 ; sound info
    < @F 0 9052 0.00 236.46                              ; first frame output
    < @F ...                                             ; lots of frames of output
    < @F 9051 1 236.43 0.03                              ; last frame output
    < @P 3                                               ; song ended
    < @P 0                                               ; stopped

  i got far enough to convince myself this was possible and it would work.
  that said it's a bit of a distraction so it's probably something i will
  be coming back to later.

dec 12: no idea?

  if i did anything on this day i have no recollection of it.

dec 13: preparing a patch against uxn11

  i took some time to create a draft patch against the mainline uxn11.
  looking at the diff as a whole exposed some things that needed a bit
  more clean up or documentation. that said, i was overall happy with
  how the patch looked:

      5 paths  17 groups, +204 lines (+353 -149)

  (this doesn't include the design doc, demo programs, or my own ROMs.)

dec 14: it's looking ready 

  this was a big one!

  i did most of the clean up tasks i noticed the previous day, and also
  fixed a bunch of bugs across console.c, term.tal, and femto.tal:

    - getting and setting environment variables was broken
    - calling stty from subshells was not reliable
    - killing a non-running subprocess could have strange effects
    - determ was failing to respond to DSR queries correctly
    - femto was not able to run inside determ
    - femto was not exiting cleanly

  at this point running determ and femto via uxn11 and uxncli
  is just as good (or better) than using the old wrapper scripts.

  obligatory screenshots and video:

  uxn11-15.mp4 (9.5 MB, 1:28, has music)

  (it's worth noting that there are some graphical glitches in the video.
  when i started determ i aimed to just implement TERM=ansi, but it turns
  out many programs aren't 100% compatible with "pure" ansi terminals.
  so i pivoted to quasi-vt100 compatibility but am still missing some
  fairly critical vt-100 features.)


dec 15: draft patch for uxn11

  all my uxn11 changes in one place.

  mostly i wanted to see all the changes in one place and get a sense
  of how reasonable (or unreasonable) it might look. i was pretty
  pleased with the result!
dec 17: took saturday off

  wasn't feeling motivated, took a break.

dec 17: also took sunday off

  still wasn't feeling motivated. continued my break.

dec 18: uuid-based extensions for uxn11 

  one problem with using an enhanced console only available in uxn11
  is that it is incompatible with the current approach of using stdin
  and stdout with wrapper scripts. this means i will either need to
  drop support for uxnemu or else maintain two versions of my ROMs.

  one idea i had was to include support for both approaches, and then
  choose at runtime based on the emulator version. however, there is
  currently not a good way for ROMs to figure this out.

  while talking with neauoire about this they reminded me of the
  extensions proposed by zzo38. these use the expansion port to
  send and receive information about optional device features.

  currently only the uxn38 emulator supports UUID-based extensions,
  but adding them to uxn11 ended up being pretty easy. i have also
  updated the specification to describe their use with the console.

dec 19: another pause

  took a bit of a break.

dec 20: still paused

  still was not feeling adventurous.

dec 21: uxntal support for linguist 

  one weird thing about respositories with uxntal code is that sites
  like github/codeberg/etc. don't know about the language, so the
  language summary ends up listing other languages like shell, lua,
  python, etc. instead.

  i finally got tired of this state of affairs and submitted a PR to
  the linguist project to add uxntal support. this mostly involved
  reading contributor guidelines, looking at other PRs, and installing
  a bunch of linguist dependencies.

  linguist uses textmate grammars for syntax highlighting but
  fortunately there's already an implementation for uxn.

  anyway, the pull request is here.

dec 22: pr updates 

  took care of some minor issues noted on the PR.

  i expect nothing will be merged before 2024 but i'm hopeful that
  soon we'll have uxntal support in a linguist release.

dec 23: top-secret project

  i paused my adventure to work on a top secret project.

dec 24: top-secret project (continued)

  the top secret project continued to take all my time.

dec 25: top-secret project revealed

  without sharing any actual details about it, the top-secret project
  was related to a christmas gift for kate.

dec 26: post-holiday break

  after all the chaos of the holidays it was nice to take a break.

dec 27: break 

  still taking a break (i was hoping to restore some energy for future
  work but (spoiler alert) i ended up getting sick and not
  doing anything else for the rest of the month.

dec 28: illness

  i got sick today.

dec 29: very sick

  even more sick today.

dec 30: still very sick

  still very sick today.

dec 31: starting to recover

  starting to feel better, but still sick.

  so in conclusion: my adventure ended in an incomplete state,
  partially due to holiday chaos and partially due to illness.

  still, i had a good time and did get a lot done! thanks, eli!