Added notes about emacs config migration to org-mode

content/blog/2024-07-28-returning-to-emacs-init-in-org.md

@@ -0,0 +1,267 @@
++++
+title = "Refactoring Emacs Config using Org"
+date = "2024-07-28"
+author = "Peter Tillemans"
+email = "pti@snamellit.com"
+[taxonomies]
+tags = ["programming", "emacs"]
+categories = ["linux"]
++++
+
+# Refactoring my Emacs Config
+
+It is time to work on my Emacs config. Not because it is hopelessly broken, but to prevent that from happening. A long time ago I had a literate config I had to declare Emacs bankrupcy on because of the amount of cruft and lack of TLC. It had grown to a 4000 line 100kB+ inconsistent mess. It was nice to have evrything in a single file with documentation in place. 
+
+To reduce my config and leverage newer packages I then tried some of the big distros, Spacemacs and Doomemacs. This reduced the size I had to maintain to about 30k allowing me to keep on top of it. My personal modifications were then in a config.org file.
+
+A few years ago I wanted to get better on using Windows and switched my main machine to a Windows desktop. I immediately noticed Doom Emacs being slow. Then a search started to find the right version of Emacs on Windows : msvc version, cygwin, msys2, wsl, ..., which all had different integration issues. I finally figured out that there is something in Doom which caused the sluggishness, and it is far less present in much more minimal config. 
+*
+Hence I started relatively fresh using Rational Emacs, later Crafted Emacs as a base and organizing core. This is a regular elisp modular configuration. I split my personal configuration settings in different custom modules: pti-ui-XXX, pti-ide-XXX, pti-org-XXX, pti-evil-XXX.  
+
+# Goals to achieve 
+
+I like to jot down some things I want to achieve to help me decide when is done and guide some decision along the way to avoid me getting stuck in the middle of a bramble forrest. 
+
+I came up with:
+
+- Manage *init.el* as an org-file
+- Use a simple robust tangling solution which works together with
+  Crafted Emacs
+- Leverage inclusion of ~use-package~ in Emacs29 for configuration and
+  loading
+- Reduce dependencies : read evaluate the value a package brings
+  before including it
+- Refactor existing configuration
+
+In typical fashion the first goal is not a goal but what I will be doing, but judging from other _Business Analysis_ efforts, this is part for the course. *What is the goal of the project? Do the project on time and in budget!* (real helpful...)
+
+I actually added this when I notice the path getting all tangly and thorny and I remembered I forgot to list what I wanted to do. Well better late than never.
+
+# Approach
+
+There are couple of constraints and probable consequences.
+
+- likely editing the generated file instead of the source file
+## Navigation Between Functions
+
+Normally in Emacs Lisp buffers navigation between functions is magic. Who knows how this will pan out in org-babel. 
+
+That being said I found that in configuring emacs previously I use little homebrew functions which are usually close to where I use them and for the other functions `C-h f` is a good friend.
+
+## Editing the Generated File iso the Org Sources ##
+
+That is a *me* problem. I am *going* to be editing the generated file. So I need early feedback for this.
+
+## Tangling and Git 
+
+Adding generated files in git is a no-no. I will have to add the generated file to the gitignore.
+
+Then there is the issue that the config source file ideally is close to the generated file (see above). If I tangle the file from the git location into *~/.config/emacs* then I'll have to navigate between the two locations. Probably there is an easy solution for this, but I don't have it now.
+
+If I symbolic link the file from the git location to *~/.config/emacs* then it is unclear if emacs will compare the creation date of the symbolic link or the file it points to. Probably this is well defined but I remember being bitten by this in the past.
+
+But of course no-one is mandating that the emacs configuration has to be part of the *dotfiles* repo. I want to refactor that shortly and if I remove all the emacs stuff from the dotfiles repo then I can just clone the emacs config repo to *~/.config/emacs* and be done with it. And I have less to migrate in the dotfiles. So that is a win in my book.
+
+## Do I want to factor out Crafted Emacs?
+
+I am not sure. I like the idea of having a base configuration which is well maintained and I can just add my personal configuration to. The upstream moves forward with the times, I can pull in the changes and the (mild) breakage over time will push me to keep my configuration up to date.
+
+But I also like the idea of having a configuration which is mine and mine alone. I can do whatever I want with it. I can add a package, remove a package, change a package. I can do whatever I want. 
+
+However it is not on the goals list. I will keep it in mind and see how it goes.
+
+## Do I want to factor out the custom modules?
+
+Yes. In the past I was pretty happy with a single org file. Before I declared emacs bankruptcy and moved to Spacemacs and Doom Emacs I had a single org file. I liked it. I could search for things easily. I could see the context of the configuration I could add documentation in context. I liked it. 
+
+What I did not like was technical debt and conflicting modules which were integrated at different times and not necessarily work well together. The experience was inconsistent and I felt I was running behind the current state of the art. Hence the need to explore the big distros.
+
+After the big distros I moved to a modular configuration based on *Rational Emacs*, later renamed to *Crafted Emacs*. This was a good move. I could keep up with the changes in the emacs world and I could keep my configuration in check. It was all emacs lisp and this worked well for me.
+
+I added a number of custom modules to the configuration. This works well bt I think the tools in org-mode can do better for this use-case. An Emacs config is not just code to do features. Emacs is also an integration platform where workflows are at least as important as the code. Having a place to document the workflows and the code in context is a big win. Also the code blocks can generate other files which integrate with the Emacs modules. And being able to run code blocks in the org file is a big win to show running examples or have ad-hoc tools available. Now this is also possible with comments in the source code, but I was brainwashed that comments are bad and code is good so I not comfortable with large comments in the source code. (Although that is changing slowly).
+
+
+#  Approach
+
+## Recover the Infrastructure from the Old Configuration
+
+The tangling code proved to be very robust and will work fine with Crafted Emacs.
+
+## Use Elpaca as Package Manager
+
+I used *straight* as package manager with Crafted Emacs. I will switch to *elpaca* as one of the co-maintainers of *straight* started *elpaca* as a more user-friendly package manager.
+
+The integration with *use-package* is more seamless and the configuration is more concise compared to *straight*.
+
+I had to add *quelpa* in the mix for some reason. Part of the refactor is to remove both *straight* and *quelpa* and use *elpaca*.
+
+It also seems to be more in line with the modern package managers in *neovim*.
+
+## Use Org-Babel for Configuration
+
+Yeah, Duh. This is the whole point of the exercise.
+
+But it also for maintenance support, running examples and (if needed) generating support scripts.
+
+## Use *use-package* for Configuration
+
+Crafted Emacs leverages the customize infrastructure, including *package-install-selected-packages* to install packages. This means that adding the packages and configuring them is a two-step process and they end up in different modules. 
+
+Also there are subtle issues with state being persisted in the saved *custom.el* file. I maintain half a dozen systems which I want to be consistent. I have the feeling that *use-package* is better supported by the upstream package and I can steal more configs from the respective README's. Time will tell.
+
+However it is a new tool to learn and I never used it at this scale.
+
+## No new features
+
+My config suits me so I need no new features. I just want the same behavior after as before. I can quickly compare behavior on different machines. I will wait to update my VPS until I have a baseline on the desktop of the migration.
+
+## Do incremental changes
+
+I kept my *init.el* relatively small and moved big chunks to separate modules. I can repeatedly copy a file in an org babel block and tangle it to the *init.el* file. Then the resulting config has to be the same
+
+- I will start with the *init.el* as an org file
+- Verify it works and behaves the same
+- Split the block in logical pieces give them an Org Header.
+- Evaluate if they are in the right place and move if needed.
+- Migrate the config to use *use-package*
+- Verify it works and behaves the same
+- Repeat until done
+
+When a file is migrated, I baseline to the master branch and push it to the remote. 
+
+Then the next file.
+
+# Doing it.
+
+I will spare the grind of the actual migration. Actually migrating was no problem. Dealing with the long tail of issues was.
+
+## Issues
+
+### Use-Package defer and loading behavior.
+
+I faced quite some issues with the loading behavior of the packages. I started with `:defer nil` to get things working. If Emacs took 15s to load then that was not a priority to fix. This bit me in the backend afterwards.
+
+    I misundertood the documentation and made the classic mistake of attributing *magic* properties to things as `:commands`, `:hook` and `:bind`. Although it is documented that these can only define to functions in the package, I made the mistake that other functions would trigger loading the package. Since this auto loading is a thin veneer over the standard Emacs loading mechanism, this is not the case. I formulated the rule that you can only refer to a thing in the package from a thing which is already loaded. That helped debugging and avoiding this issue.
+    
+    I put a delay between 3 and 10s for modules I would classify as *creature comforts*. This is a good compromise between loading time and usability. I can always lower the delay if I feel the need for speed.
+    
+    It took some time to get this directed acyclic graph sorted out. 
+    
+### Use-Package and :custom
+
+Since Crafted Emacs uses the customize infrastructure, and I adopted that it would make sense to use the `:custom` keyword in `use-package`. Of course the penny dropped after I moved ton to `setq` forms in the `:init` blocks. This in turn caused issues with getting passwords with the *pass* command which is slow and requires sometimes a pin-entry.
+
+Using `:custom` ensures the variables are still defined on loading the package so it can bootstrap and and the password will be unlocked at that time, which is probably when I do something that actually needs it. 
+
+### Pinentry in Emacs
+
+I did not have this issue before but Emacs seemed to hang when started from the i3 launcher. From comparing to starting on the terminal this is likely because it was waiting for the pinentry to be entered when the *emacs --daemon* was started which has no GUI frame. This resulted in multiple `emacs --daemon` processes in the process list, which surprised me as I never experienced this before.
+
+I disabled the emacs pinentry support and then it pops up a GUI window so I could enter the password. This is not ideal but it works. In the mean time this is fixed as no more passwords are asked before Emacs is fully started. But it is likely this issue will return.
+
+### Elpaca and Dev Releases from Emacs
+
+Elpaca needs the build date of emacs to compare to package versions or
+something. However it does not support all dev versions. It has a
+table to map released versions to date codes to compare to the build
+date of emacs with the date of potential packages. (or something
+similar).
+
+This caused some headscratching how to get this and I finally settled on the 
+birthdate of the .drv package installed by *guix*.
+
+For guix emacs-next packages you can find the date with: ( <C-c C-c>
+in the source block below:
+
+    #+BEGIN_SRC shell
+    stat /gnu/store/*emacs-next-[23]*.drv | rg Birth | cut -d' ' -f3 | tr -d '-'
+    #+END_SRC
+
+    #+RESULTS:
+    | 20240727 |
+    | 20240727 |
+
+It is possible there are more so probably the most recent one is the
+one to use. 
+
+I override the elpaca generated one with the one I found in *early-init.el*.
+
+
+    (setq elpaca-core-date "20240727")
+
+This is manual for now. Maybe I should trigger this on `guix home reconfigure`.
+
+# Results 
+
+I now like visiting my config more 
+
+![config screenshot](emacs-refactor/first-things.png)
+
+
+Here is an example with workflow notes and executable sample code :
+
+![example of workflow notes and sample code](emacs-refactor/pass-sample.png)
+
+Looking at the git log:
+
+    commit 008f173ef49e2dfe175c4d84e6c87425442d1468
+    Date:   Tue Jul 30 16:51:18 2024 +0200
+
+    switch to new emacs config, burn old bridges
+
+    commit 2b44a918653890950b0985a4a5e8d65be22507c7
+    Author: Peter Tillemans <pti@snamellit.com>
+    Date:   Tue Jul 30 16:47:28 2024 +0200
+
+    save crafted emacs before deleting;
+
+    commit f79ab7ec6d8cd7af35253db997b291029174c38e
+    Author: Peter Tillemans <pti@snamellit.com>
+    Date:   Tue Jul 30 00:49:19 2024 +0200
+
+    rework and testing of the new config
+
+    commit acf915e24b43d4eb3950931b13121a037b77a659
+    Author: Peter Tillemans <pti@snamellit.com>
+    Date:   Mon Jul 29 17:51:19 2024 +0200
+
+    finished porting emacs init to org-mode
+
+    commit a1cd08f6e157e918f5979e57f0ee50b01688b98f
+    Author: Peter Tillemans <pti@snamellit.com>
+    Date:   Mon Jul 29 11:25:56 2024 +0200
+
+    remove dependency on general to define evil keybindings
+
+    commit 739c067427c6944c96ba5cf1990788a99beb9386
+    Author: Peter Tillemans <pti@snamellit.com>
+    Date:   Mon Jul 29 10:04:56 2024 +0200
+
+    remove migrated config
+
+    commit 4010de3d8eb8a7621d9e20bb7bc8af663059a527
+    Author: Peter Tillemans <pti@snamellit.com>
+    Date:   Mon Jul 29 10:01:17 2024 +0200
+
+    fix issues with starting emacs.
+
+    - migrated pti-org-packages and pti-org-config
+    - migrated last requires
+    - resolved some existing conflicts
+
+    commit 99c83bb248a896c99700cca6025777a938d36701
+    Author: Peter Tillemans <pti@snamellit.com>
+    Date:   Mon Jul 29 01:26:08 2024 +0200
+
+    Start refactor emacs config to org-mode again.
+
+
+I started on July 27th at 01:30 and the initial migration was done on July 19th 18:00 about. At that point the emacs config moved to a new repo. 
+
+![Gru evaluates before and after](emacs-refactor/grus-plan.jpg)
+
+comparing the results we see that the files reduced from 12 to 3, total lines is the same (1851 vs 1871). However total size increased  from 67kB to about 85kB. 
+
+I am pretty happy with the change. I removed some duplication, removed some packages I hardly used, found some configs in the wrong place and I have held everything in my hands for a late spring cleaning. 
+
+Is it better than my before emacs lisp configuration? I don't know. It is just a tradeoff, exchanging some consequences for other consequences. I learned a ton which will help maintain it in the future. Maybe I change my mind in a few years and migrate back, which will be another great opportunity for a spring cleaning.