Now I am using org-roam as my notes management app. It is great and it works on top of org-mode and of course Emacs. Not everyone feels comfortable with this since it requires to be okay with technical bits and bytes. I just love it…

I started using v1.0 which did not supported to map org headings but only file top headers. This was brilliant already enough in itself. With v2.0 org-roam included the ability to map headers also. What was good became even better! The only caveat at that time is that org-roam-server stopped working. That is understandable. Now there is a new project called org-roam-ui which is basically org-roam-server for v2.0 with headings support.

Recently I trimmed my DB, I am trying to find a balance between using headings and not using headings. My Emacs default config used to creat UUIDs for each heading. This was good at first but I had to adapt this for org-roam v2.0… I had +9000 notes, which is unpractical to be honest… So I cleared the UUIDs of the headers of several files that were not needed and now I have +3000 notes. It is still too much IMO, so further trimming is needed but that cleanup needs more thought.

Now lets preview this thing… Most screenshots have been anonimized. At the time of this writing this is how my notes looks like. Keep in mind that out of all of my notes, only a few selected ones are shared here on my website.

I am going to show first a little video sample

Figure 1 is how the exobrain looks considering all notes

Figure 2 is how the exobrain looks only considering the file headers and subheadings.

Figure 2 shows my Emacs configuration file cluster and how it relates to other notes.

Figure 4 shows my parashot study notes.

Figure 5 shows my beekeeping notes and log, which is quite extensive by the way.

Figure 6 shows my testing trading notes. I developed my own tool to document my simulated tradings and tests.

Figure 6 shows my simulated trading log. I integrated my own python tool with yasnippets, so Emacs could get all the goodies out of it. The result is a very detailed (and possibly overkill) simulated trade log. This is going to probably be decoupled from my org-roam database. I don’t really need to search on all of that.

Figure 8 shows my research notes on automated trading strategies.

Figure 9 shows my research notes on agroforestry.

Figure 10 shows my journal of the current year 2022. TODO’s tasks and how they integrate with the rest of the notes.

Figure 11 shows my homesteading notes and all the ongoing projects.

• How?
• Assumptions and limitations
• Interesting findings
• Experiments organized by sections
• Experiments organized by books
• Conclusion

The other day I was thinking about some practical way of learning hebrew, something dynamic that tickles my scientist curiosity. And I thought, wouldn’t it be cool if I could parse one of the hebrew codexes and get the top 20 most frequent words of a chapter or a book? Well this is an attempt to answer these type of questions.

How?

So I rolled my sleeves and I thought: I will google the Aleppo codex in plain text so I can download it and later use it for analysis. The problem is, I couldn’t find it…

The next challenge then is, how can I produce a plain text version of the Aleppo codex? I thought about parsing a mysword module, which are in reality a sqlite database so that should be easy… but then when I was doing something totally unrelated (probably around the bees or my first batch of mead) it hit me… I have several bibles from the Sword project already installed on my notebook and is all OSS, so… there is probably a python wrapper around diatheke or similar… and sure it is =)

If you want the technicalities, go read my note on How to parse the Aleppo codex and analyze its content in python.

But if you are not a geekie human, you probably are interested only in the results rather than the bits and bytes. This is why I splitted the whole thing in two notes, one more IT related and one more Hebrew related. So here we go, brace yourselves this is probably going to be long.

I am not sure on how to “slice” the codex so the chunks makes sense for an analysis, for instance: per chapter? Per book? Per “stories”? Or even other slicing like Torah, Neviim and Ketuvim. Since it is not clear I will start slicing by books at first.

Assumptions and limitations

Hebrew has some peculiarities one of them being the vowel pointing. This brings a ton of challenges. For the sake of simplicity I had to stick to a codex that does not include niqqud whatsoever. I am not sure if this is a good approach or not because two different words without niqqud can render the same writing yet have totally different meaning.

Another problem is that there are words that are trivially known like לא (h2834) or אל (h3882); or words that are not translated yet used very much like את (h7073). I have implemented a filter to have the possibility to skip these words. Not because they are not relevant, the Aleph-Tav has a ton of secrets and importance. But there are mixed together some irrelevant tokens considered as words, like opening and closing brackets and parenthesis or similar. So I implemented a list to ignore these words that are rather known or really irrelevant.

Interesting findings

It was rather interesting to me how fast the repetitions decline on unfiltered words. For instance the most used word is the את (h854) with 7073 occurrences. Yet 30 words later in a row, there is a 90% decline in occurrences, משה (h) with 704 occurrences.

Experiments organized by sections

Full Tanakh - top 20 words - unfiltered

Full Tanakh - Top 50 words - unfiltered

The words from above as the top 20, plus the following words

Torah

Experiments organized by books

Genesis

Conclusion

The more I try to know, the less I feel I know… This coding-linguistic area is fascinating and intreaguing. I feel like a taxonomy of this area is needed.

This will probaly take a lot of time, which I don’t have. So I want to try to balance between being effective and being efficient. I cannot afford to spend much time with this project but on the other hand the Hebrew language is really really apealing to me. I have yet to find a balance on how to proceed.

As part of documenting things I am going to include some very interesting resources that I have found.

From the time I spent researching I would probably go with the BHSA DB. This is a proposed roadmap:

• Check the ETCBC/course materials for an introduction
• Check the BHSA/bigTables for Pandas examples and learn how to use it
• Reevaluate then

Also, if focus is more in learning Hebrew, check the Parabible website which looks really lean and straigthforward.

Bible in Hebrew Demo

• Source: https://www.bibleinhebrew.com/bih/BiH_demo.php?uid=10
• Title: Bible in Hebrew Demo
• Captured on: [2022-06-06 Mon]

See, Hear, and Read the Bible in Hebrew - BibleinHebrew.com

• Source: https://www.bibleinhebrew.com/bih/
• Title: See, Hear, and Read the Bible in Hebrew - BibleinHebrew.com
• Captured on: [2022-06-06 Mon]

Parabible | Genesis 1

• Source: https://parabible.com/Genesis/1
• Title: Parabible | Genesis 1
• Captured on: [2022-06-06 Mon]

GitHub - jcuenod/awesome-bible-data: 😎 A curated list of generously licensed Bible data.

• Source: https://github.com/jcuenod/awesome-bible-data
• Title: GitHub - jcuenod/awesome-bible-data: 😎 A curated list of generously licensed Bible data.
• Captured on: [2022-06-05 Sun]

Text-Fabric versus SHEBANQ - BHSA

• Source: https://etcbc.github.io/bhsa/mql/
• Title: Text-Fabric versus SHEBANQ - BHSA
• Captured on: [2022-06-06 Mon]

GitHub - ETCBC/course_materials: Contains scripts to learn to work with the ETCBC database

• Source: https://github.com/ETCBC/course_materials
• Title: GitHub - ETCBC/course_materials: Contains scripts to learn to work with the ETCBC database
• Captured on: [2022-06-06 Mon]

bhsa/bigTablesP.ipynb at master · ETCBC/bhsa · GitHub

• Source: https://github.com/ETCBC/bhsa/blob/master/programs/bigTablesP.ipynb
• Title: bhsa/bigTablesP.ipynb at master · ETCBC/bhsa · GitHub
• Captured on: [2022-06-06 Mon]

Shebanq / Words Words

• Source: https://shebanq.ancient-data.org/hebrew/words
• Title: Words
• Captured on: [2022-06-05 Sun]

Shebanq / text [2017] Genesis 30:1

• Source: https://shebanq.ancient-data.org/hebrew/text
• Title: [2017] Genesis 30:1
• Captured on: [2022-06-05 Sun]

Jupyter Notebook Viewer

• Source: https://nbviewer.org/github/etcbc/bhsa/blob/master/tutorial/search.ipynb
• Title: Jupyter Notebook Viewer
• Captured on: [2022-06-06 Mon]

OSHB Read

• Source: https://hb.openscriptures.org/read/
• Title: OSHB Read
• Captured on: [2022-06-05 Sun]

OSHB Lexicon

• Source: http://openscriptures.github.io/HebrewLexicon/HomeFiles/Lexicon.html
• Title: OSHB Lexicon
• Captured on: [2022-06-05 Sun]

Bible Online Learner

• Source: https://booge.eu/
• Title: Bible Online Learner
• Captured on: [2022-06-05 Sun]

J. Ted Blakley — Online Hebrew Resources

• Source: https://www.blakleycreative.com/jtb/HebrewOnline.htm
• Title: J. Ted Blakley — Online Hebrew Resources
• Captured on: [2022-06-06 Mon]

Tanach.us text files webservice Text files

• Source: https://www.tanach.us/Pages/TextFiles.html
• Title: Text files
• Captured on: [2022-06-06 Mon]
• Command: wget –user-agent=" Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" “http://tanach.us/Server.txt?Deuteronomy26:1-1&layout=Text-only&content=Consonants”

Removing Vowels from Hebrew Unicode Text · GitHub

• Source: https://gist.github.com/yakovsh/345a71d841871cc3d375
• Title: Removing Vowels from Hebrew Unicode Text · GitHub
• Captured on: [2022-06-05 Sun]

shoroshim.pdf

• Source: https://halakhah.com/rst/shoroshim.pdf
• Title:
• Captured on: [2022-06-05 Sun]

unfoldingWord® Hebrew Grammar — unfoldingWord® Hebrew Grammar 1 documentation

• Source: https://uhg.readthedocs.io/en/latest/front.html
• Title: unfoldingWord® Hebrew Grammar — unfoldingWord® Hebrew Grammar 1 documentation
• Captured on: [2022-06-05 Sun]

• (OPTIONAL) Create a boot flash disk using android - EtchDroid USB writer | F-Droid - Free and Open Source Android App Repository
• Mounting and chrooting the system
• Edit the /etc/default/grub file
• Config and install GRUB and create the ramdisk
• References
  • [SOLVED] Grub with lvm on LUKS -> rescue mode. Incomplete grub.cfg? / Installation / Arch Linux Forums

Recently an upgrade on my notebook went bad and broke something. I wasn’t sure what went wrong but since I had just one system at that time, I had to figure out how to fix it with the help of my phone. Basically what I did is install EtchDroid to be able to download an iso and flash it to a flash drive on my phone directly. Later I could boot form that flash drive, mount the LUKS/LVM partitions and fix the bootloader. Here is a short how to.

(OPTIONAL) Create a boot flash disk using android

EtchDroid USB writer | F-Droid - Free and Open Source Android App Repository

• Source: https://f-droid.org/en/packages/eu.depau.etchdroid/
• Title: EtchDroid USB writer | F-Droid - Free and Open Source Android App Repository
• Captured on: [2021-08-05 Thu]

Mounting and chrooting the system

#+begin_src sh

cryptsetup luksOpen /dev/disk/by-partlabel/cryptlvm lvm

mount /dev/storage/root /mnt mount /dev/storage/home /mnt/home

mount /dev/sda1 /mnt/boot

cd mnt mount -t proc /proc proc #mount -t sysfs sys sys #mount -o bind dev dev mount -o bind run run

arch-chroot /mnt

dhcpcd eth0 #+end_src sh

Edit the /etc/default/grub file

#+begin_src sh

GRUB_CMDLINE_LINUX=“cryptdevice=/dev/sda2:lvm” GRUB_PRELOAD_MODULES=“part_gpt part_msdos cryptodisk luks” #+end_src sh

Config and install GRUB and create the ramdisk

#+begin_src sh

grub-mkconfig > /boot/grub/grub.cfg grub-install –efi-directory=/boot –target=x86_64-efi /dev/sda

mkinitcpio -p linux #+end_src sh

References

[SOLVED] Grub with lvm on LUKS -> rescue mode. Incomplete grub.cfg? / Installation / Arch Linux Forums

• Source: https://bbs.archlinux.org/viewtopic.php?id=246628
• Title: [SOLVED] Grub with lvm on LUKS -> rescue mode. Incomplete grub.cfg? / Installation / Arch Linux Forums
• Captured on: [2021-08-04 Wed]

My first couple of blogs were on blogspot around 2005 or so, later I switched to dokuwiki, which is awesome but it didn’t integrated well on my workflow. So I tried some extensions to try to write directly in markdown but it didn’t work well for me. Then I moved to Hyde, which a Python static website generator. I invested quite some time customizing everything to my needs and while it was functional, I had to write in markdown (which I don’t like…). I tried to write in RST at that time without success.

Then I moved from Vim to Emacs and discovered org-mode… And yeah, everything changed… I love Emacs+Org-Mode and I used quite a lot (and I still use Vim occasionally).

I wanted to write directly in Org-Mode for my new blog to see if content flows this time (hopefully) and I can achieve the integration I was looking for all these years.

The version of this blog uses plain Org-Mode and Hugo with ox-hugo as a middle helper. So far, so good, I just migrated my old posts which are few in number.

What I am looking forward is to create a mix of a Blog and a digital garden for my notes (also known as second brain or exobrain).

The rest of the file is actually a sandbox of general Org-Mode formatting to check out how it renders on Hugo itself.

This is H1

There is paragraph under h1

H2

• H3

  Something

• H3 with a tag sometag

Some basic test

This is bold, italic, code, verbatim and strike text.

• However bold and italic doesn’t play well when used together like in markdown.
• However bold and italic doesn’t play well when used together like in markdown.

List

• Bullet
• Another bullet
  • child
    • deep

Other style

• Bullet
• Another bullet
  • child
    • deep

Other style

1. Bullet
2. Another bullet
   1. child
      1. deep

Style * isn’t supported.

Links

link to org mode homepage

Check List [1/3] [33%]

• Item
• Item
• Checked item

Heading and has special class however <ul> and <li> are plain.

Task List

TODO some to-do

DONE done to-do

Items are added with special class.

Tables

<tr> has even and odd classes.

Source Code

Emacs Lisp:

(defun negate (x)
    "Negate the value of x."
    (- x))

(print
    (negate 10))

There are interesting classes like sourceCode and example. Also there html5 attributes prefixed with rundoc-.

Haskell:

factorial :: Int -> Int
factorial 0 = 1
factorial n = n * factorial (n - 1)

LaTeX

• Characters: α β → ↑ \or \and \implies π ∞
• Inline Math: \(f(x) = x^2\)
• More complex: \(\frac{x^2}{2}\)

LaTeX characters are wrapped in <em> and Math inside <span class="math inline">.

\mathscr{Hello!}

\begin{align*} 8 * 3 &= 8 + 8 \\ &= 24 \end{align*}

NOTE: There is standard LaTeX embeded above which is skipped during compilation to HTML.

This is using MathJax

\[\sum_{i=0}^n i^2 = \frac{(n^2+n)(2n+1)}{6}\]

Deadline

Tagged tag

Tags are not visible in render

Block Quote

Org mode is amazing. So is Hakyll & Pandoc.

Image

http://media.riffsy.com/images/f8534774b678ad1932b379a03460680b/raw

Images has to have extension like:

then it can be loaded even from other origin..

Description List

Frodo

The hobbit ringbearer

Aragorn

The human ranger, true kind of Gondor

Gandalf

The Grey Wizard

creddits to nihilmancer

I use Arch Linux and I prepared a repository with my custom install method based on archinstaller. Basically I have a custom partitioning based on LVM on top of LUKS and some variables to start he installing method.

Here is an overview of the parts on this tutorial:

Tutorial overview

1. Create a bootable USB and boot the broken system with it (AKA recovery mode**
2. Chroot into the system
3. Fix the issue

Assumptions

I will assume that the error is similar as mine. After opening the LUKS device, I had the following problem: ERROR: device /dev/mapper/storage-root not found.

Create a bootable USB

To create a bootable USB you need to download the latest Arch Linux iso and then us dd to dump it to a flash drive.

dd bs=4M if=/path/to/archlinux.iso of=/dev/sdx status=progress && sync

After completion, use the same flash drive to boot the broken computer. You generally need to hit a key on the F5 - F6 range to choose the boot method.

Chroot into the system

Once the system has booted, now we need to mount everything the system would mount on a boot stage. Adapt the LUKS label and LVM devices according to your needs.

# Open the LUKS device.
cryptsetup luksOpen /dev/disk/by-partlabel/cryptlvm lvm

# Mount the root partition
mount /dev/storage/root /mnt

# Mount the boot partition
mount /dev/sda1 /mnt/boot

# Mount proc, sys and dev
mount -t proc proc proc/
mount -t sysfs sys sys/
mount -o bind /dev dev/

# Switch to bash
bash

# Chroot
chroot /mnt

# Bring up network if needed
dhcpcd eth0

At this point you should have shell with a recovery environment set to fix the issue, so let’s fix it.

Fix the issue

To fix the issue we basically are going to perform an upgrade an recreate the initial ramdisk

# Update the packages database
pacman -Syy

# Upgrade the packages
pacman -Syu

# Update udev package
pacman -S udev

# Update mkinitcpio package
pacman -S mkinitcpio

# Recreate the initial ramdisk
mkinitcpio -p linux

# Exit the chroot environment
exit

# Reboot
reboot

If everything went well, you now should be able to boot the system properly.

Conclusions

I was quite worried about how to solve this issue due to the several layers of complexity: GPT, LUKS and LVM, however everything went out quite smoothly. The key part is to use a USB flash drive and then mount the boot and root partitions to recreate the ramdisk.

Thanks for reading. Spot an error or want to explain something better, feel free to send me a PR.

One of the problems of keyservers (in my opinion), is the problem of deleting old keys. I pretty much like the idea of PGP keyserver which verifies periodically by sending an email if the address is still in use and therefore maintain that uid of the key. The only problem is that the PGP server is centralized and it does not use FLOSS standards.

I understand that deleting a key on a distributed environment is hard and probably maintaining a history of deletions as done actually on most of keyservers is still a good solution. However since I don’t use GPG very often and I am not publishing my key yet, I wanted to try PKA.

This tutorial is made of two basic parts, first creating a TXT DNS record and then verifying that the key gets downloaded properly.

Tutorial overview

1. Creating the TXT DNS record
2. Testing the PKA record

Assumptions

I assume that you have gpg installed and that you know the basic idea on how it works, that you know what a private and a public keys are for and how to use them.

Creating the TXT DNS record

To create the TXT DNS record you will need to know the fingerprint of the key. To do so:

$ gpg --list-keys vonpupp@keybase.io
pub   rsa2048/0x536814BF4871A220 2016-11-12 [SC] [expires: 2018-11-12]
      Key fingerprint = F0B9 B3FB 25E9 1209 728E  4844 5368 14BF 4871 A220
      uid                   [ultimate] Albert De La Fuente <vonpupp@keybase.io>
      uid                   [ultimate] Albert De La Fuente (Social email address) <vonpupp@gmail.com>
      uid                   [ultimate] Albert De La Fuente (Main email address) <mail@albertdelafuente.com>
      uid                   [ultimate] Albert De La Fuente (Haevas email address) <albert@haevas.com>
      uid                   [ultimate] Albert De La Fuente (Academic email address) <albert@ime.usp.br>
      sub   rsa2048/0xE2977BF3F82AB971 2016-11-12 [E] [expires: 2018-11-12]

In my case, the fingerprint is F0B9 B3FB 25E9 1209 728E 4844 5368 14BF 4871 A220, or 0x536814BF4871A220 for short.

Then you need to export the key with:

$ gpg --export -a 0x536814BF4871A220 > public-0x536814BF4871A220.asc

Then create a TXT record where:

The Name part:

Is composed of mailbox._pka.albertdelafuente.com. So for instance if my email is long-anti-spam-email-address@albertdelafuentedotcom, then the name part should be long-anti-spam-email-address._pka.albertdelafuente.com.

The Text data:

Contains the fingerprint and the URL where to download the key. In my case:

  "v=pka1;fpr=F0B9B3FB25E91209728E4844536814BF4871A220;uri=http://albertdelafuente.com/media/files/public-0x536814BF4871A220.asc"

Do not forget to upload the key to match the uri as on the TXT record. In my case: http://albertdelafuente.com/media/files/public-0x536814BF4871A220.asc

Testing the PKA record

Once done that you can download the key (just for testing purposes) on another computer or VM as follows:

$ echo "Test message" | gpg --auto-key-locate pka -ear mail@albertdelafuente.com

In my case this will prompt you with a confirmation since my primary uid does not match the email. This is made in purpose because I have read that some spammers are using keyservers to get valid email addresses, therefore I always use keybase as my primary id.

You will see something similar to this as the output:

gpg: directory =/home/vagrant/.gnupg' created
gpg: new configuration file =/home/vagrant/.gnupg/gpg.conf' created
gpg: WARNING: options in =/home/vagrant/.gnupg/gpg.conf' are not yet active during this run
gpg: keyring =/home/vagrant/.gnupg/secring.gpg' created
gpg: keyring =/home/vagrant/.gnupg/pubring.gpg' created
gpg: requesting key 4871A220 from http server albertdelafuente.com
gpg: /home/vagrant/.gnupg/trustdb.gpg: trustdb created
gpg: key 4871A220: public key "Albert De La Fuente <vonpupp@keybase.io>" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
gpg: automatically retrieved =mail@albertdelafuente.com' via PKA
gpg: F82AB971: There is no assurance this key belongs to the named user

pub  2048R/F82AB971 2016-11-12 Albert De La Fuente <vonpupp@keybase.io>
 Primary key fingerprint: F0B9 B3FB 25E9 1209 728E  4844 5368 14BF 4871 A220
      Subkey fingerprint: 7A10 07B4 3F49 5317 5DE0  52E8 E297 7BF3 F82A B971

It is NOT certain that the key belongs to the person named
in the user ID.  If you *really* know what you are doing,
you may answer the next question with yes.

Use this key anyway? (y/N) y
-----BEGIN PGP MESSAGE-----
Version: GnuPG v1

hQEMA+KXe/P4KrlxAQf/d1yxYFBSPs0RKHJ98w+s82jK25R/IXCiFNe6BkX+oyp+
uh+4AObx93SuJ/ryHlthHQmnpid4BQWmhmCksiAH+xD1xrlrCDIsNQfJ5+aPQXjz
+Z6iKrWy8Lk13i6u3wgMZuk2eKN9Z1ppi15arXhFc93cta5p5K5tAH7CwMd5zP93
r7wgI2Jff+x3erN0zbkJ2PZgDrHZVLVWyOnwgRBw12N8El3L8i6JFbNY+g25AMUm
MMCPSTit8ILsFoPtkrJEOdq5p5aCw3dvIVSzmxflMJEsgqO+Per+KxtMehaBF5qX
I2TzltcgjlisSJ3rcBtjpm12rSVJrPs4BG2UKz0w6tJIAbF0FLlWXe8zMJMK1E3Q
BQ7y/gjTduiuuD++qyIxqWCoLCgHixvP4WiTPbbKvoXl4BP8Bf1ED9M/0Cyss2NI
tW7vVlLcXRQb
=gdKa
-----END PGP MESSAGE-----

Conclusions

As you can see it is not that hard to publish a public key via DNS. You need to export your key to a file and upload it and then create a TXT record relating the fingerprint and the location of the key. You may also publish your key on a keyserver also since not everybody retrieve the keys over DNS.

Thanks for reading. Spot an error or want to explain something better, feel free to send me a PR.

I have been using LVM on top of LUKS for more than 2 years without problems. However since I haven’t had the chance to troubleshoot these technologies in the past I was quite concerned of what would happen when I would have problems. Today my main computer didn’t boot up and I had to fix it manually. This post shows the steps I performed to recover my system.

Intro

cryptsetup luksOpen /dev/sda2 cryptlvm

e2fsck -f /dev/mapper/storage-home

resize2fs -p /dev/mapper/storage-home 200g

e2fsck -f /dev/mapper/storage-home

(lvdisplay)
lvreduce -L -15.68G /dev/storage/home
(lvdisplay)

[lvresize -l +100%FREE /dev/storate/root]

e2fsck -f /dev/mapper/storage-root

resize2fs /dev/mapper/storage-root

cryptsetup luksClose /dev/sda2 cryptlvm


### NOT NEEDED

(pvdisplay)
pvresize --sephysicalvolumesize 218G /dev/mapper/cryptlvm

/dev/mapper/cryptlvm is active and is in use
  type:    LUKS1
  cipher:  aes-xts-plain64
  keysize: 256 bits
  device:  /dev/sda2
  offset:  4096 sectors
  size:    487981391 sectors
  mode:    read/write

NEW_SECTORS = EXISTING_SECTORS * NEW_SIZE_IN_GB / EXISTING_SIZE_IN_GB
487981391 * 200.7 / 232.7   = 420876085
(487981391 - 15*1024*1024*2 = 456524111)

cryptsetup -b 420876085 resize cryptlvm

(cryptsetup status cryptlvm)

parted /dev/sda

resize

{# Local Variables: #} {# mode: markdown #} {# End: #}

Configuring Pacserve and Powerpill on Arch Linux

The documentation of Pacserve explains how it works. This image summarizes well the behavior.

Pacserve by itself seems very interesting however I could not make it work properly. I think it has something to do my router UDP multicast forwarding support (see openwrt-multicast). Nevertheless, I could use it within Powerpill which is a pacman wrapper to allow parallel and segmented downloads through Aria2 and Reflector. This combo sounds even better, so I will focus on how to install and use the tree tools together.

Installation

# Just in case the testdb does not exists
gpg --list-key

# Receive xyne gpg key
gpg --recv-keys 1D1F0DC78F173680

# Install the packages
yaourt -Sy --noconfirm reflector pacserve powerpill

Starting the services (both the Master and Slave(s) host(s))

After everything is installed:

# Enable opening the ports
sudo systemctl enable pacserve-ports.service

# Enable pacserve
sudo systemctl enable pacserve.service

# Start pacserve
sudo systemctl start pacserve.service

For troubleshooting purposes the services can be started manually. See the Troubleshooting section if you have problems.

Configuring Powerpill to use the Master server (Slave(s) hosts only)

Edit the file /etc/powerpill/powerpill.json to include the `pacserve`.`serve` attribute:

{
  "aria2": {
    "args": [
      "--allow-overwrite=true",
      "--always-resume=false",
      "--auto-file-renaming=false",
      "--check-integrity=true",
      "--conditional-get=true",
      "--continue=true",
      "--file-allocation=none",
      "--log-level=error",
      "--max-concurrent-downloads=100",
      "--max-connection-per-server=5",
      "--min-split-size=5M",
      "--remote-time=true",
      "--show-console-readout=true"
    ],
    "path": "/usr/bin/aria2c"
  },
  "pacman": {
    "config": "/etc/pacman.conf",
    "path": "/usr/bin/pacman"
  },
  "pacserve": {
    "server": "http://192.168.56.6:15678"
  },
  "powerpill": {
    "ask": true,
    "reflect databases": false
  },
  "reflector": {
    "args.unused": [
      "--protocol",
      "http",
      "--latest",
      "50"
    ]
  },
  "rsync": {
    "args": [
      "--no-motd",
      "--progress"
    ],
    "db only": true,
    "path": "/usr/bin/rsync",
    "servers": []
  }
}

In this example I configured two VMs using Vagrant with an internal network. The server runs on 192.168.56.6 on port `15678` (default port).

After this we can use Powerpill as a replacement of Pacman as follows

# System upgrade
sudo powerpill -Syu

# Install a package
sudo powerpill -S wget

Conclusions

When powerpill is used..

TODOs

• Check how to have only one Master DB and all the Slaves also sync the DB from the master
• Use reflector commands within Powerpill
• What happens if different architectures are on the same network and UDP multicast works?

Troubleshooting Pacserve (without Powerpill)

As I said before I could not make it working using multicast, so I had to manually start the Pacserve Master host as:

Input:

pacserve --multicast

Output:

PacserveServer
  PID                  1048
  Addresses
    lo:     127.0.0.1
    eth0:   10.0.2.15
    enp0s9: 192.168.56.6
  Port                 15678
  Multicast Address    all interfaces
  Multicast Port       15679
  Multicast Group      224.3.45.67
  Multicast Interval   5m
  Multicast Interfaces all
  Filelist             None
  Filterlist           None
  MOTD                 None
  Upload Directory     None
  Paths                None
  Static Peers         None

Press ctrl+C to exit.
[2015-10-23 05:54:54 AEDT] INFO: announcing presence by multicast (group: 224.3.45.67)
[2015-10-23 05:54:54 AEDT] INFO: announcement sent via all interfaces

Assuming the server is running with the IP 192.168.56.6, you can manually start Pacserve on the Slave(s) hosts as:

Input:

pacserve --multicast --peer "http://192.168.56.6:15678"

Output:

PacserveServer
  PID                  1102
  Addresses
    lo:     127.0.0.1
    eth0:   10.0.2.15
    enp0s9: 192.168.56.7
  Port                 15678
  Multicast Address    all interfaces
  Multicast Port       15679
  Multicast Group      224.3.45.67
  Multicast Interval   5m
  Multicast Interfaces all
  Filelist             None
  Filterlist           None
  MOTD                 None
  Upload Directory     None
  Paths                None
  Static Peers         http://192.168.56.6:15678

Press ctrl+C to exit.
[2015-10-23 05:56:00 AEDT] INFO: announcing presence via POST to http://192.168.56.6:15678
[2015-10-23 05:56:00 AEDT] INFO: POSTing to http://192.168.56.6:15678/ [type: nudge]
[2015-10-23 05:56:01 AEDT] INFO: announcing presence by multicast (group: 224.3.45.67)
[2015-10-23 05:56:01 AEDT] INFO: announcement sent via all interfaces

After starting the Slave host you should get the following message on the Master console:

[2015-10-23 05:56:00 AEDT] INFO: added http://192.168.56.7:15678/ (POST)
[2015-10-23 05:56:00 AEDT] INFO: 192.168.56.7 "POST / HTTP/1.1" 200 -

Since I’m not very familiarized with the use case data itself I looked up for a way on how to apply for the same concepts on other areas, and then I came up making some experiments with my Facebook and LinkedIn networks. I learned many things and the results were really interesting.

I first started with my Facebook contacts and this was the initial result

Afterward I applied a community detection algorithm (Louvain) with a specific parameter and I could identify 63 groups / communities.

Can you see them? Well… neither do I… So I applied a atlas layout algorithm so it’s visually friendly, and this is the result

It’s interesting to notice that about 46% of my contacts are within the three biggest groups (20.77%, 14.46% and 12.52%). There are also contacts (that multicolored group in the middle) that I couldn’t retrieve their connections (probably because a privacy setting). Of course the graphics generated are anonymized, but I’m able to identify each group, for instance the bottom middle blue group are my last job work European colleagues. The top right cyan group are some friends from FLOSS communities. Pretty cool, huh?

On the other hand getting the Linkedin results wasn’t that straightforward I had to deal with oauth and the LinkedIn API webservice directly within python. This are the results of my LinkedIn network.

Some notable groups at first were: bottom left cyan group are again my last work European colleagues, and green my last work Latin America colleagues.

Conclusions

This is an amazingly powerful analytics tool that could be used in many areas, probably the most notable (nowadays) are marketing and social networking. At the same time it’s quite scary to see how our privacy goes away, I don’t think it would be hard to track somebody having the right information. If I could do this by just examining the relationships imagine what could be possible with some extra data.

He commented that patents are like walking in a mine field, but worst because a mine ones it blows up it doesn’t blows again, patents keeps blowing on and on. He later highlighted some different scenarios.

A patent for your idea not to be “stolen”**. This is the worst case scenario

You might think that you had a great idea and you want to patent it so no one will stole your idea. Later a big company comes in and develops something around your idea. Then something similar will happen…

> (You) You go there and say: Hey this idea is mine, and I have the patent so you cannot use it!

>> (Big company) Ohh, it is a shame… But let me remind you that within your idea you used some ideas similar to these three patents of us…

> (oh oh)

>> …and we can find other patents (among our thousands) of us, you might violated with your idea as well…

> ehmmm

>> …so in order to no one get hurts, lets make a deal: You give us the right to use your patent and we wont sue you

> Well… You evaluate (in silence) and remember that they have some good lawyers and a lot of money as well… And you end by saying: Okay you can use it

>> Great. Thanks =D

Conclusion: The big company wins.

A patent to protect yourself from others attacking you

The truth is that it won’t protect you unless you find another infringement on the other side that is covered by your patent. A minor company wont sue you anyway since they don’t have the resources, and a big one will have very good lawyers to do so and to also defend themselves, so it’s a endless legal battle and unnecessary waste of money.

He also pointed that in fact a patent could be also be harmful if you try to defend yourself you don’t know how to use it properly, or something is wrong within the patent.

Conclusion

A never ending legal story if someones attacks you, or in the best scenario nobody will attack you so you won’t really need the patent in first place.

Later he also made a symbolism about software and music which personally I found very interesting. “Imagine that when Beethoven was alive the whole patent idea arose and somebody patented parts of musical pieces. Then some beautiful compositions would never been made. It’s easier to compose for the beauty of doing so than composing with the worry of not getting sued”.

The whole speech makes a lot of sense while patents in software don’t.