TIL: Python tips and tricks collection

This blog post is a collection of Python tips and tricks that I have found useful over the years. I will keep updating this entry as I learn more.

  • How to add a custom CA Root certificate to the CA Store used by pip in Windows?

    pip config set global.cert path/to/ca-bundle.crt
    pip config list
  • Generate requiements.txt
    Use pipreqs

    pip install pipreqs
    pipreqs /path/to/project

    or pip freeze. But, freeze saves all packages in the environment including those that you don´t use in your current project!

    pip freeze

Counter Strike: Source won’t start on NixOS

After the latest update of my NixOS machine, Counter Strike: Source wont start. Starting Steam from console shows the following error message

[...]
src/tcmalloc.cc:278] Attempt to free invalid pointer 0x94d1af0 
/home/user/.local/share/Steam/steamapps/common/Counter-Strike Source/hl2.sh: line 73: 14550 Aborted                 (core dumped) ${GAME_DEBUGGER} "${GAMEROOT}"/${GAMEEXE} "$@

I could fixed the problem by

  • copy the libmimalloc.so from Half-Life 2 bin-folder (/home/user/.local/share/Steam/steamapps/common/Half-Life 2/bin/libmimalloc.so) to Conter-Strike: Source bin-folder
  • Rename existing libtcmalloc_minimal.so.4 to libtcmalloc_minimal.so.4~ or similar
  • rename libmimalloc.so to ibtcmalloc_minimal.so.4

The game now starts 🙂

Synology HyperBackup to Hetzner Storage Box

Its possible to use Hetzner Storage Box as HyperBackup target with the following settings.

Preparations

  • Log in to StorageBox Administration from Hetzner
  • Select StorageBox
  • Create SubAccount (Optional)
    • Activate: Allow SSH
    • Activate: Allow external accessibility
    • User name and password will be displayed after saving (ONLY ONCE!)

Setup

  • Install HyperBackup package
  • Backup destination > File server > rsync
    • ServerType: rsync-compatible server
    • Server name: <uXXXX.your-storagebox.de>
    • Transmission encryption: On
    • Port: 23
    • Username: uXXX-subXXX
    • Password: XXX
    • Backup module: /home/
    • Directory: <backup name>

Logging from PHP to Docker logs (stdout)

If you run your PHP application inside a docker container, you could write (debug) output to the docker log. This is useful if you want to see the output of your application in the docker logs.

Commands

$out = fopen('php://stdout', 'w'); //output handler
fputs($out, "Output goes here...."); //writing output operation
fclose($out); //closing handler

Example

function _log($msg) {
    $msg = "myApp - " . date("c") . ": " . $msg."\n";
    $out = fopen('php://stdout', 'w');
    fputs($out, $msg);
    fclose($out);
}

Fixing Windows Hello PIN on Windows 11 (0x80090016)

This is the second time, my Lenovo X13s (Windows-on-ARM) machine wont boot and ask for the BitLocker key. After using the recovery key, and login with password, i can't set a new Windows PIN which is needed for biometric (fingerprint or face recognition) login. I get the error message Something went wrong. Try again later (0x80090016).

tl;dr

To fix this, you need to delete the folder C:\Windows\ServiceProfiles\LocalService\AppData\Local\Microsoft\NGC and reboot the system. After that, i was able to set a new PIN.

Steps

  1. takeown /f C:\Windows\ServiceProfiles\LocalService\AppData\Local\Microsoft\NGC /r /d y
  2. icacls C:\Windows\ServiceProfiles\LocalService\AppData\Local\Microsoft\NGC /grant administrators:F /t
  3. Delete C:\Windows\ServiceProfiles\LocalService\AppData\Local\Microsoft\NGC
  4. Reboot

MikroTik SMS to Telegram – A SMS Gateway Forwarder Script

I needed a way to forward SMS messages from my Mikrotik router with modem. The easiest way was to forward the SMS to a Telegram chat.

The script retrieves incoming SMS messages, extracts essential information such as the sender, message content, and timestamp, and forwards them to the Telegram Bot API. It also includes basic error handling and provides feedback on the success or failure of the forwarding process. Multiple chat IDs are also possible.

Basic setup

  1. Setup modem and test sending and reciving without the script
  2. Register Telegram Bot and get API Token. I don't want to explain this here, there are enough tutorials on the internet.
  3. Get Chat ID. Send a message for example to @chatIDrobot and get the Chat ID.

Script setup

  1. Create a new script forward-incoming-sms in the Mikrotik router and paste the script code:

# ------------------------------------------------- #
# SMS to Telegram - A SMS Gateway Forwarder Script  #
# ------------------------------------------------- #

# Description
# This script will forward all SMS messages to a Telegram chat.

# Author
# 2024-01-11 foorschtbar
# https://blog.spaps.de/

# Credits
# http://blog.redax.hu/2021/02/mikrotik-sms-to-sms-forwarding.html
# https://medium.com/@dedanirungu/forwarding-sms-messages-with-mikrotik-to-website-url-via-modem-12d926615834
# https://github.com/eworm-de/routeros-scripts/blob/main/sms-forward.rsc

# Configuration
:local token "12345678:AAABBCCC...XXXYYYZZZZ"
:local chatids {"12345678";"12345678"}

:put "== Starting SMS forwarder script =="

# Check if receiving is enabled
:if ([ /tool/sms/get receive-enabled ] = false) do={
  :log warning ("Receiving of SMS is not enabled.")
  :error ("exit script");
}

# Check if the modem is in running state
:local Settings [ /tool/sms/get ];
:if ([ /interface/lte/get ($Settings->"port") running ] != true) do={
  :log warning ("The LTE interface is not in running state, skipping.")
  :error ("exit script");
}

# forward SMS in a loop
:local smsCount [ :len [ /tool/sms/inbox/find ] ]
:put ("Found ".$smsCount." SMS to process")
:local index 0
:foreach sms in=[ /tool/sms/inbox/find ] do={
    :set index ($index + 1)
    :put ("> Processing ".$index." of ".$smsCount)

    :local smsVal [ /tool/sms/inbox/get $sms ];
    :local smsPhone ($smsVal->"phone")
    :local smsType ($smsVal->"type")
    :local smsMessage ($smsVal->"message")
    :local smsTime ($smsVal->"timestamp")

    :local logmsg ("SMS from ".$smsPhone." on ".$smsTime." (".$smsType."):\n".$smsMessage)
    :put ($logmsg);
    :log info ("Forwarding ". $logmsg);

    # URL safe message
  :local urlMessage ""
    :for i from=0 to=([:len $logmsg] - 1) do={ 
      :local char [:pick $logmsg $i]

      :if ($char = "\n") do={
        :set $char "%0A";
      }

      :if ($char = " ") do={
        :set $char "%20";
      }

      :if ($char = "-") do={
        :set $char "%2D";
      }

      :if ($char = "\?") do={
        :set $char "%3F";
      }

      :if ($char = "!") do={
        :set $char "%21";
      }

      :if ($char = "+") do={
        :set $char "%2B";
      }

      :if ($char = "%") do={
        :set $char "%22";
      }

      :if ($char = "'") do={
        :set $char "%27";
      }

      :if ($char = "(") do={
        :set $char "%28";
      }

      :if ($char = ")") do={
        :set $char "%29";
      }

      :if ($char = ",") do={
        :set $char "%2C";
      }

      :if ($char = ".") do={
        :set $char "%2E";
      }

      :if ($char = ":") do={
        :set $char "%3A";
      }

      :if ($char = ";") do={
        :set $char "%3B";
      }

      :if ($char = "=") do={
        :set $char "%3D";
      }

      :if ($char = "&") do={
        :set $char "%26";
      }

      :if ($char = "*") do={
        :set $char "%2A";
      }

      :if ($char = "/") do={
        :set $char "%2F";
      } 

        :set urlMessage ($urlMessage . $char);

    }

    # send POST
    :local noerror true
    :local chatIdx 1
    :local chatsTotal [ :len $chatids ]
    :foreach chatid in=$chatids do={
        :put ("> Sending HTTP request ". $chatIdx . " of " . $chatsTotal."...")
        :local url ("https://api.telegram.org/bot" .$token . "/sendMessage")
        :local parameters ("?chat_id=" . $chatid . "&text=" . $urlMessage)
        :local fullurl ($url . $parameters)
        :local responseStr [/tool fetch url=$fullurl http-method=get as-value output=user]
        :put ("Status: ".$responseStr->"status")
        :put ("Data: ".$responseStr->"data")
        :if ($responseStr->"status" = "finished") do={
            :put ("Successfully forwarded message ID: " . $index. " to chat ID: " . $chatid)
            :log info ("Successfully forwarded message ID: " . $index. " to chat ID: " . $chatid)
        } else={
            :put ("Failed to forward message ID: " . $index. " to chat ID: " . $chatid)
            :log error ("Failed to forward message ID: " . $index. " to chat ID: " . $chatid)
            :set noerror false
        }
        :set chatIdx ($chatIdx + 1)
    }

    :if ($noerror) do={
        /tool sms inbox remove $sms
        :put ("Deleted message ID: " . $index)
    }
}

:put "== Finished SMS forwarder script =="
  1. Change the configuration variables in the script: tokenand chatids
  2. Test the script by running it manually /system script run forward-incoming-sms
  3. Create a new scheduler forward-incoming-sms. You can use the following command or create it in the Webfig
    /system scheduler add name=forward-incoming-sms interval=10s on-event="/system script run forward-incoming-sms" start-time=startup

Temporarily access NixOS with encrypted LUKS from Ubuntu Live

Setup

  • Boot Ubuntu Live as usual
  • Change keyboard layout with setxkbmap <lang>
  • Install useful packages with sudo apt install vim nix-bin

Open crypt device

  • Idenitify LUKS device with lsblk --fs. Look for crypto_LUKS:
    nvme0n1
    ├─nvme0n1p1   vfat        FAT32
    └─nvme0n1p2   crypto_LUKS 2
  • Open encrypted device with cryptsetup luksOpen /dev/<root partition (sda2 or similar)> secure
  • Check sucessfull open with lslbk. You should now see the crypt partition(s):
    nvme0n1       259:0    0 476.9G  0 disk
    ├─nvme0n1p1   259:1    0   549M  0 part
    └─nvme0n1p2   259:2    0 476.4G  0 part
    └─enc-pv    253:0    0 476.4G  0 crypt
      ├─vg-swap 253:1    0    16G  0 lvm
      └─vg-root 253:2    0 460.4G  0 lvm

Mount partitions

  • lvchange -a y /dev/vg/swap
  • lvchange -a y /dev/vg/root
  • mount /dev/vg/root /mnt
  • mount /dev/<boot partition (sda1 or similar)> /mnt/boot
  • swapon /dev/vg/swap

Temporarily access NixOS

  • Access with nixos-enter
  • Optional:
    • Add additional channel if needed
      nix-channel --add https://github.com/NixOS/mobile-nixos/archive/refs/heads/master.tar.gz mobile-nixos
    • Update channels nix-channel --update
  • Change config
  • Rebuild system nixos-rebuild boot

Enable sshd for root on Ubuntu Live

  • Set keyboard layout with setxkbmap <lang>
  • Update package repo with sudo apt update
  • Install sshd and vim with sudo apt install openssh-server vim
  • Update sshd config with sudo vim /etc/ssh/sshd_config
    • Set PermitRootLogin yes
    • Set PasswordAuthentication yes
  • Set password for root with sudo passwd root
  • Restart SSHD service sudo systemctl restart ssh.service
  • Login via ssh to machine

Resize a LUKS encrypted root partition

  1. Resizing the partition used by the encrypted volume
parted /dev/sda

(parted) print
[...]
Number  Start   End    Size   File system  Name                  Flags
 1      1049kB  525MB  524MB  fat16        EFI system partition  boot, esp
 2      525MB   256GB  256GB               Linux filesystem

(parted) resizepart
Partition number? 2
End?  [512GB]? '100%'

(parted) print
[...]
Number  Start   End    Size   File system  Name                  Flags
 1      1049kB  525MB  524MB  fat16        EFI system partition  boot, esp
 2      525MB   512GB  512GB               Linux filesystem

(parted) q
  1. Boot machine from a bootable USB linux system (Arch Linux-, Ubuntu or similar USB installer)
  2. Open and resizing the encrypted LUKS volume
cryptsetup open /dev/sdb1 sdb1_crypt
cryptsetup resize sdb1_crypt
  1. Resize the physical device
pvresize /dev/mapper/sdb1_crypt
  1. Resize the logical device
lvextend -l +100%FREE /dev/mapper/vg-root
  1. Resizing the file-system of the volume
resize2fs /dev/mapper/vg-root
  1. Reboot. You are done.